Trustworthy Robots in the AI Era

This article is excerpted from an internal report for 'trustworthy AI systems for robots'.

Times Have Changed...

The era of ROS1/2 is coming to an end. They were at vital importance back to the age when we needed to efficiently glue different modules and sensors together, therefore we could stand all of its drawbacks such as intrinsic performance issues and lack of structural design. However, the situation of robotics research has changed since the renaissance of neural-network-based technologies.

Today's emphasis of robotics research has transferred to more complicated tasks and systems where AI infrastructures are usually involved. For researchers with a focus on the software part of robotics, it is no longer necessary to build customized robots, for they are hard to be innovative and competitive compared to mature commercial whole robot platforms, such as legged robot Boston Dynamics Spot, and humanoid robot Unitree G1. The decreasing price trend of these commercial robot platforms has made these products affordable to more and more research labs, and they all provide their own software development kits which is not built upon ROS.

[Talk on International GameDev Conference, 2024]

Modern languages don't help solve real hard problems. ... people talk all the time about it's more important not to leak memory, so you got all these quite complex systems put into place to make sure you free your memory ... for us, the much harder problem than freeing memory is to know how the game should behave and what should this system be like. ... What are the real problems? I don't understand my program, it's too big and complicated. Sometimes when I go to debug it, it takes too long and I don't know what the problem is. Sometimes I have to write big systems that are a little bit error prone, because the languages don't have features to help me out.

-- Jonathan Blow

As the talk of Jonathan, the same story happens in robotics: ROS1/2 do not help solve real hard problems of current robotics research focus.

For robot manufactures, the ability of ROS1/2 to adapt a wide range of sensors is not needed. Sensors and modules of their robot platforms has a limited number and range. The philosophy of ROS1/2 is to treat every robot system as designed for academic research, hence many underlying infrastructures are developed with the lack of consideration, which of course cannot meet the requirement for commercial products. As for these robot manufactures, they can not benefit from the advantage of ROS1/2 to easily develop quick yet thoughtless prototypes, but they have to be legally responsible for internal bugs and security issues within ROS1/2. To gain fully control over the whole system platform, and also to better optimize the software for their hardware, these famous manufactures choose to gradually adapt to their exclusive SDKs after they have enough loyal customers. Despite the intention of tightly binding customers to their products, ROS1/2 cannot help robot manufactures with a commercial standard to meet.

For current researchers, the days of easily publishing papers on assembling robots have gone forever. Spending weeks on assembling a robot whose functions are provided by most commercial robot platform will do no good for robotics researchers of software or algorithms. Also, the implicitness of support for messages which are usually predefined as broader than usually needed in third-party ROS1/2 nodes will cause more mental burden for researchers. There are still fierce debates about the pros and cons of distributed micro-service structure, but it is widely acknowledged that a communication interface cannot guarantee micro-services (or nodes) to be robust and reusable: it depends on how the whole system is split into these services (or nodes), and it is sure that distributed structure will takes more tools such as logs gathering and more effort to debug. And as for the real difficulties where complicated systems where AI infrastructures such as neural networks or complicated decision making systems are used, ROS1/2 seems not to be helpful at all, whereas only adds the difficulties to design and debug into the development-validation loops.

...But Not That Much.

It is undeniable that the technology researchers used in these days are much more advanced than previous, however the ultimate problem that we try to solve remains still: to make robots more widely used in society. Robots have already embraced huge success in industry, and incoming success in academic. According to a government report of China, in the year of 2020, more than 212,000 sets of industrial robots had been produced, and the robot application density had increased to 24.6 sets per thousand of people. People have witnessed the success of industrial robot arms especially in the field of high-end automobile manufacturing, but no one can guarantee that would be the same story for robots in other fields.

Overbuilding things the world doesn't have use for, or is not ready for, typically ends badly.

-- Jim Covello, Analyst of Goldman Sachs Inc.

Both of the society and robots are probably not ready for the large-scale deployment of autonomous robots in non-high-end fields. No one would argue against the future when we will have a tremendous number of robots working for us, but the only question is when. For developers who work with robot software systems, there is an eternal question: who will buy your robot (software systems) for what purpose? Unitree gives an answer: laboratories, for research. Here is a disorted joke about this:

Current business of legged robots or humanoid robots is like a Ponzi scheme: many students rush into robotics, for they think there is a huge business opportunity awaiting them; and after they graduate and start their companies, the only customer is students who recently rushed into robotics.

-- Disorted Joke

This joke is not completely true, and judging from what is going on in China, where more than 5.461 billion CNY is publicly disclosed as invested in robotics since 2023, there are many investment scams. According to real cases happened among my friends, many old money in China are recruiting oversea PhDs with high annual salaries to start companies quickly. Their only goal is not to develop useful robots but to go public as soon as possible. These oversea PhDs will be their CEO, or technical leader, but are actually used as golden brands to convince investors to buy their stocks. Their target is not the potential undeveloped market, but the money of their investors -- if they cannot answer the question of 'who and for what', how can they see the market?

The situation has gradually gotten out of control. Here is an example. According to recent news, a 23-years-old PhD candidate young boy created an robot company U---x AI (name is hidden). He is a second year PhD student and there is still three years left for him to graduate. His research focus is visual-touch in his bachelor and PhD study, but his company is a whole robot platform manufacturer. Its 100% capacity of 10 million CNY comes from the company Bo---e Intelligent Manufacturing (founded in June, 2023), whose 100% of capacity comes from the company An---n Company Management(founded in July, 2022), whose boss is also a member of the board of directors of U---x, Bo---e, and another fully controlled company Ag---a Intelligent Manufacturing (founded in October, 2023).

How can this affect we researchers and developers? The most direct influence is that after this large-scale farce is subsided, researchers or developers will have much more difficulty to get investment for macro goals; they must specifically answer the eternal question of 'who will buy your robots and for what purpose'.

Who and for What

The essence of the robot’s mission is to replace humans. This sounds terrifying to irrelevant people, but what more terrifying to some of researchers is the doubt that whether their robots are capable and competitive enough to substitute humans, and for now, the answer is usually a stuttering and hesitant yet undeniable no.

Household service robot is a huge potential market, but the fact is that it is unreachable in recent years. Everyone would like a robot to do the housework and cook delicious meals for them, but no one would expect a robot which may fall down if it step on children's toy and could possibly hit into furniture and break the glass of television while it trying to restore its balance; nor a robot which may drop vegetables on the ground or get on fire as it trying to cook. This story sounds pessimistic but probably it is a optimistic expectation for robots in the near future.

Despite functions, there is also a vital factor which cannot be ignored---price. Months ago, a visitor to our laboratory summarized the goal of their laboratory as 'making robots faster, higher, and stronger'. I personally held negative attitude towards this kind of 'Robot Olympics'. Even though many robotics researchers disdain engineering, but robotics itself is actually a subject of engineering rather than science--if they seriously consider about conventional science, they would find that there is no 'undiscovered ultimate truth of the universe' within robots, which are fully artificial creations, designed, built, and improved by humans. To think about this, even mathematics is not categorized as 'science'. As a subject of engineering, customers nor the society cannot really benefit from 'Robot Olympics'. Because, if we learned something from the tragic failure of Concorde supersonic airplane, then we would know: engineering is about weighing the pros and cons, and it is not enough to get accepted by the public for just being 'faster'; customers always care about more aspects, such as price.

After taking the factor of price into consideration, currently there is not too many areas that robots are competitive enough compared to human labor. Even though the price of robots is keeping going down, there are many geographic areas where human labor is at a extreme low price, such as Guangzhou, China, where workers in electronic assembling factories usually are paid 0.98~1.96 USD (about 140~280 JPY) per hour. It is nearly impossible for robots to be efficient and cheap enough to function at such a low price in the near future. Therefore, robots can only embrace success in geographic areas or business areas with high enough salaries.

There is a clue for finding these areas. Though human labor is cheap somewhere, but human lives are expensive everywhere. A soldier in Russia has a month salary of 1,402 USD and get a compensation money of 137,030 USD if he or her sacrificed for duty; meanwhile, the whole-procedure price of a Unitree legged robot Go2 is 1,403 USD, which is only the salary of a month for a human soldier. Even if we skip the discussion in these sensitive areas, people will not be hesitate to pay for affordable things which can truly guarantee their safety, especially in areas with special social situation, such as America. Currently there is already attempts of selling robots as police equipment, but that is more like a financial support from the local government to the young robot companies whose orders have a characteristic of 'low demand on numbers and high price per robot'. In contrast, there will be a large number of families who live in the rural area are willing to replace 'guns under pillows' with security robots in the yard, no matter they are legged or humanoids, only if their prices are affordable. And the functions required for robots are just patrolling, identifying, alarming, and using non-lethal equipment such as pepper spray, or simply calling the police, which are already proved as fully doable in laboratories.

In conclusion, there are profitable yet untouched areas awaiting challengers in robotics, even for the technology level of today.

Overlooked Reliability.

It takes 10 years of Baidu to put autonomous driving from laboratories to society as Apollo Autonomous-Driving Taxi, but for now, these robot taxis still cannot understand the command of traffic polices, and cause chaos when they encounter bicycle riders who do not obey traffic rules. Usually, we do not see to many research applied in society, for the effort to make these systems resilient enough to function under social context is usually equal or more than the effort to invent or develop these systems. The reliability of robots, which means that the ability of robots handle most of the possible incidents that hinder them from carrying out their tasks, and perform proper and necessary operations when it stuck in unrecoverable states. In a nutshell, the ability to obey these laws:

• Law 1: A robot may not injure a human being or, through inaction, allow a human being to come to harm.
• Law 2: A robot must obey the orders given it by human beings except where such orders would conflict with the Law 1.
• Law 3: A robot must protect its own existence as long as such protection does not conflict with the Law 1 or Law 2.

Indeed they are Asimov's Three Laws of Robotics. It is very intuitive that trustworthy robots are which obeying these laws. Actually there can be more local laws made by their owners, but these are fundamental laws for robots to be applied in society. Sadly, current robot systems can not guarantee obeying these laws; and what more sad is the fact that judging from the topics of published papers, robot researchers are not interested in this direction in recent years.

In conclusion, which sounds awful, current robots are not trustworthy, and most of current robots researchers do not mean to make them trustworthy. In November 8^th 2023, a 40-years-old worker in South Korea is crushed by a industrial robot arm, which confused him as a box of vegetables, and crushed his face and chest against the conveyor belt. This incident did not affect the operation of the company who developed this robot system, but that night was surely a long night for the family awaiting this worker at home; and at that night, many researchers were still too busy with their 'Robot Olympics', to make their robots (injure people) 'faster, higher, and stronger'. It is fair that we cannot expect too much from them, for they cannot easily publish papers on the topic of trustworthy robots (or caring about 'human lives'). Reliability is a vital yet ignored prerequisite.

AI Makes Robots More Unreliable.

Probability theory based neural networks cannot guarantee the same output for the same input. Some people may expect using AI to regulate robots, but according to the behavior of current LLMs and other neural network based AI services (such as object detection), AI is also an nondeterministic factor. On July 18^th, news is reported that LLMs including ChatGPT-4o, Gemini Advanced, Claude 3.5 and more, could not give the correct answer of 'which is bigger, 9.9 or 9.11?' And someone also noticed that if positions of these two numbers are switched, then these LLMs could output the correct answer. This uncovered the disappointing fact that LLMs only recognized these tow numbers as two tokens and never learned to do math. There is a bad trend of worshiping LLMs as some kind of 'digital gods', and regard any research where LLMs is involved as only a wrapper of LLMs based on the false belief that LLMs have unleashed all the magic. In face, according to my experiment, even the 'common sense' within LLMs is not as stable and correct as we usually believe; levels of details of the answers varies according to the language that the question is asked in; and sometimes they are confident to forgery answers to questions they do not understand. Many of researchers take this as 'natural and normal', yet it is 'unnatural and abnormal' to see them still worshiping LLMs as 'digital gods' who knows everything and can do everything right.

Therefore, when it comes to AI (especially LLM) embedded robot systems, more measures must be taken to ensure their reliability, for not only the systems themselves should obey Asimov's Three Laws of Robotics, but also ignore the commands from the AI which does not meet the requirement of Three Laws.