| Project Reference: | ITP/067/23LP |
| Project Title: | Voice-Controlled Mobile Manipulator Powered by Large Language Model |
| Hosting Institution: | LSCM R&D Centre (LSCM) |
| Abstract: | The number of the elderly who lives alone in society keeps growing, which leads to an increasing demand for robots used to care for the elderly. This seed project aims to use human natural language to control mobile manipulator to complete operations such as moving and grabbing various kinds of objects, so that traditional robots have the ability to understand natural language instructions, enhance the control of robots and expand their usage. The R&D methodology involves the comprehensive application of natural language processing, computer vision and multi-modal, visual navigation, robot grasping and other technologies. Human natural language instructions will be understood and analyzed through large language model, converting them into low-level action instructions and codes that robots can specifically execute. Computer vision and multi-modal technology will use vision language models to obtain environmental information such as object classifications and locations around the robot as external information input to decompose natural language instructions. Visual navigation is used to identify the spatial location of target items and guide the robot to the target location. Vision-based robot grasping is used to locate the target objects and control the manipulator to grasp the targets. This R&D work can greatly improve the existing control interface of robots, making them more widely used in home, elderly/disabled care, logistics and other industries. |
| Project Coordinator: | Dr Jun Luo |
| Approved Funding Amount: | HK$ 2.79M |
| Project Period: | 1 Mar 2024 - 31 Aug 2025 |
