Haoyu Jia
'Courage is grace under pressure.'
Resume
Research Topics
- LLM-driven Robot Agents
- Software Infrastructures for LLM Agents
- Robot Software Engineering
Tags: LLM Agent; Software Engineering; Robot;
Education
| Period | University | Degree | Major | Remarks |
|---|---|---|---|---|
| 2024~Present | The University of Tokyo | Master's Degree Candidate | Creative Informatics | • Working on LLM-driven robot agents in JSK Robotics Laboratory. |
| 2018~2023 | China University of Petroleum (East China) | Bachelor's Degree | Data Science | • Average grade: 87.67/100. • Received the Outstanding Graduation Thesis Award. |
Publications
| Title | Authors | Status | Role |
|---|---|---|---|
| Mockingbird: How does LLM perform in general machine learning tasks? | Haoyu Jia, Yoshiki Obinata, Kento Kawaharazuka, Kei Okada | Under Review | Lead Author |
| Remote Life Support Robot Interface System for Global Task Planning and Local Action Expansion Using Foundation Models | Yoshiki Obinata, Haoyu Jia, Kento Kawaharazuka, Naoaki Kanazawa, Kei Okada | Accepted, International Conference on Humanoid Robots (Humanoids) | Developer for Augmented Reality Client |
| Dexterous grasp data augmentation based on grasp synthesis with fingertip workspace cloud and contact-aware sampling | Liqi Wu, Haoyu Jia, Kento Kawaharazuka, Hirokazu Ishida, Kei Okada | Accepted, Advanced Robotics, 39(12), 730–747. | Developer for Teleoperation Client |
Competitions
| Year | Name | Award | Rule |
|---|---|---|---|
| 2021 | National College Robot Competition (RoboMaster) | National First Prize | Leader of Computer Vision Systems Group |
Projects
| Year | Name | Description |
|---|---|---|
| 2021 | Object detection system based on OpenCV | • Contains an innovative CUDA filter to reduce the time consumption from 82ms to 4ms. • Based on Gaia Framework. • Contributed to the 2021 RoboMaster National First Prize. • |
| 2021 | GaiaFramework: robot software development platform | • ROS-like platform, based on micro-service model, provides UI utilities support. Use Redis as configuration server and message queue. • Boost regular robot system development period from 3 moths to 1 month. • |
| 2021 | Object detection system for industrial robot arms | • Detection system for industrial arms to customized the pipeline operations with a visualized control panel. • |
| 2022 | Prism: a dynamic code weaving framework for C# | • This library provides a framework for run-time IL weaving, and implementation modules of AOP, Dependency Injection, RPC based on this framework. • Received more than 12.6k downloads in total on NuGet. • • |
| 2022 | Nebula: robot software development libraries set | • Introduces most frequent used techniques in web development (e.g., auto dependency injection) into robot software development. • Received more than 11.5k downloads in total on NuGet. • • |
| 2022 | ShadowCode: an Android application system for Covid-19 exposure alert | • Use Bluetooth-BLE to scan and record exposure history. Data processing server is built on Microsoft Azure micro-service platform. • Winner of the Outstanding Prize in 2022 Shandong Province College Software Development Competition. • |
| 2024 | PilotHelmet: a teleoperation client for Meta Quest 3 | • Built with Unity, this client displays camera video on the helmet screens and transmit hand movement data to ROS topics. • |
| 2024 | MyCobotSharp: an unofficial .NET client for MyCobot robot arms | • Support TCP or serial port communication with MyCobot robot arms. • |
| 2024 | BigBrother-Mobile: a robot arm agent that follows users' voice commands | • Implemented as an Android application; it controls MyCobot robot arm through MyCobotSharp. • Status and functions of robot arms is mapped to the LLM-driven agent. • |
| 2025 | Thumbs!: a mobile application that can recommend users based on face identification results | • Frontend application is built with Flutter. • Backend is designed with microservice structure. • |
| 2025 | BigBrother: an extendible agent | • Action module is implemented as an external action server; actions can be configured in MongoDB. • Users are identified by voice embeddings; users' personal information is injected into the context so that the agent 'know about' who it is talking to. • |
| 2025 | PromptCompose: primitive blocks for LLM workflow | • This framework provides primitive blocks for building LLM workflows. • In addition to basic blocks such as Text and Image, advanced blocks such as Predicate and Selection are also provided. • Inspired by Flutter, unchanged blocks are cached to reduce the inference time for workflows. • |
Ongoing projects will be appended to this list after several milestones.