Haoyu Jia
'The New World Opens Only to Fast Learners.'
Resume
Research Topics
LLM-driven Agentic Systems for Robots
- Improving the task-solving capabilities of LLM-powered agents for robotic systems, with a focus on systematic methods for enhancing reasoning reliability and long-term memory.
- Investigating system architectures and design paradigms that leverage LLMs not only to automate predefined tasks, but also to adapt to unforeseen tasks and operate under uncertainties.
Software Engineering for LLM-driven Robot Systems
- Exploring designing principles and frameworks for LLM-driven systems, with a focus on efficient and robust integration, and other software engineering topics including extensibility and observability.
Keywords: LLM Agent; Software Engineering; Robot;
Education
| Period | University | Degree | Major | Remarks |
|---|---|---|---|---|
| 2019~2023 | China University of Petroleum (East China) | Bachelor's Degree | Data Science |
|
| 2024~2026 | The University of Tokyo | Master's Degree Candidate | Creative Informatics |
|
Selected Publications
| Title | Authors | Status | Contribution |
|---|---|---|---|
| Mockingbird: How does LLM perform in general machine learning tasks? | Haoyu Jia, Yoshiki Obinata, Kento Kawaharazuka, Kei Okada | Under Review | Lead Author; Designing and implementing the self-reflection framework to leverage the in-context learning ability of LLMs in conventional machine learning problems. |
| Remote Life Support Robot Interface System for Global Task Planning and Local Action Expansion Using Foundation Models | Yoshiki Obinata, Haoyu Jia, Kento Kawaharazuka, Naoaki Kanazawa, Kei Okada | Accepted, International Conference on Humanoid Robots (Humanoids 2024) | Designing and implementing an AR interaction interface that enables users to issue commands by dragging and manipulating virtual objects. |
| Dexterous grasp data augmentation based on grasp synthesis with fingertip workspace cloud and contact-aware sampling | Liqi Wu, Haoyu Jia, Kento Kawaharazuka, Hirokazu Ishida, Kei Okada | Accepted, Advanced Robotics, 39(12), 730–747. | Developing a VR application that captures hand-tracking data and synthesizes real-time VR views from camera images for immersive robot teleoperation. |
Competitions
| Year | Name | Award | Role |
|---|---|---|---|
| 2021 | National College Robot Competition (RoboMaster) | National First Prize | Leader of Computer Vision Systems Group |
Technical Skills
- Languages: C#, C++, Python, Java, SQL
- Frameworks:
- 3D, VR, AR and MR: Unity, Unreal Engine
- Desktop Apps: WPF, Qt, WinForms
- Mobile Apps: Flutter, Android SDK
- Web: ASP.NET, Blazor
- Machine Learning: Keras, PyTorch, TensorFlow, Gymnasium
- Robotics: ROS, ROS2, OpenCV, CUDA
- Code Generation: Roslyn, Emit
- Data Analysis: SciPy, Scikit-Learn, Pandas
- Middlewares: Redis, MongoDB, ActiveMQ
Selected Projects
Infrastructure for LLM Agents
These projects are developed for improving the integration of LLMs into software systems (typically agentic systems for robots). By providing APIs and mechanisms that are designed to suit the nature of LLMs, these projects together serve for the goal of unleashing more potential of LLMs, rather than simply using them to automate predefined tasks.
| Name | Status | Description |
|---|---|---|
| Released | • This framework provides a high-level API to emit IL (.NET bytecode) instructions at run-time to augment the program. • Suitable to be used as compiler for custom domain-specific languages. | |
| Released | • This framework dynamically synthesizes serializers at runtime via an LLM‑augmented expert system. • Trait-based ruleset covers standard types; unresolved edge cases are adaptively completed by the LLM. • It uses EmitToolbox as its bytecode generation engine. | |
| Released | • This framework generate dependency injectors that can inject an object at a constant time cost. • Compared to official DI framework provided by Microsoft, it allows dynamically add and remove items in injection containers. • It uses EmitToolbox as its bytecode generation engine. | |
| Released | • This framework provides primitive blocks for building LLM workflows. • In addition to basic blocks such as Text and Image, advanced blocks such as Predicate and Selection are also provided. • Inspired by Flutter, unchanged blocks are cached to reduce the inference time for workflows. | |
| Released | • Implemented with Unity, this simulator provides a visualized environment with physical simulation for monkey-banana problems. | |
| Developing | • This framework provides domain-specific language to allows LLMs to specify 'intention' equations on controllable properties, and the solver generate action sequence to achieve the equations. |
Ecosystem for Robot Systems
These projects are contributing to the software engineering efforts for robotic software development by introducing higher-level abstractions, models and patterns, rather than the de facto manner of simply wrapping the data communication protocols.
Some projects are also part of a broader effort to strengthen the C# ecosystem for robotic software development. Currently, C# is mainly deployed in the industrial robotics industry, mainly for commanding and monitoring embedded systems, and these C# projects attempt to uplift it to a more comprehensive role to fully leverage the ecosystem of .NET.
| Name | Status | Description |
|---|---|---|
| Released |
| |
| Released |
| |
| Released |
| |
| Deprecated |
| |
| Deprecated |
|
Miscellaneous Projects & Applications
| Year | Name | Description |
|---|---|---|
| 2025 | Thumbs!: a mobile application that can recommend users based on face identification results. | • Frontend application is built with Flutter. • Backend is designed with microservice structure. • |
| 2025 | BigBrother: an agent that has personalized behaviors based on who is currently speaking to it. | • Action module is implemented as an external action server; actions can be configured in MongoDB. • Users are identified by voice embeddings; users' personal information is injected into the context so that the agent 'know about' who it is talking to. • |
| 2024 | BigBrother-Mobile: a robot arm agent that follows users' voice commands. | • Implemented as an Android application; it controls MyCobot robot arm through MyCobotSharp. • Status and functions of robot arms is mapped to the LLM-driven agent. • |
| 2022 | ShadowCode: an Android application system for Covid-19 exposure alert. | • Use Bluetooth-BLE to scan and record exposure history. Data processing server is built on Microsoft Azure micro-service platform. • Winner of the Outstanding Prize in 2022 Shandong Province College Software Development Competition. • |
| 2022 | Prism: a dynamic code weaving framework for C#. | • This library provides a framework for run-time IL weaving, and implementation modules of AOP, Dependency Injection, RPC based on this framework. • Received more than 13.2k downloads in total on NuGet. • • • Deprecated. Use 'EmitToolbox' instead. |
| 2021 | Object detection system based on OpenCV. | • Contains an innovative CUDA filter to reduce the time consumption from 82ms to 4ms. • Based on Gaia Framework. • Contributed to the 2021 RoboMaster National First Prize. • |
| 2021 | Object detection system for industrial robot arms. | • Detection system for industrial arms to customized the pipeline operations with a visualized control panel. • |
Ongoing projects will be appended to this list after several milestones.
Appendix
Packages
Total Packages
Total Downloads
All-time Total
Recent Downloads
Latest Version Only
Your browser fetches these statistics in real-time from NuGet.org.Refresh the page to get the latest numbers.
Projects
Rows with the '' icon are expandable;click the icon to view project details.
Year | Name | Status | Description | Tags | |
|---|---|---|---|---|---|
| 2024 |  |
| C# LLM Infrastructure Dynamic Code Generation | ||
| 2024 |
| C# Infrastructure LLM Robotics LLM × Robot | |||
| 2024 |  |
| C# Infrastructure LLM Robotics Dynamic Code Generation | ||
| 2024 |
| C# Infrastructure LLM Framework | |||
| 2024 |
| C# Unity Engine Simulator LLM | |||
| 2024 |
| C# Infrastructure Robotics Framework | |||
| 2023 |  |
| C# Infrastructure Robotics SDK | ||
| 2023 |  |
| C# Infrastructure Robotics SDK | ||
| 2024 |
| C# Unity Engine Robotics VR ROS Teleoperation | |||
| 2022 |  |
| C# Robotics Infrastructure Framework |
18 items
10 / page
- 1
- 2