Gesture-Controlled Aerial Robot Formation for Human-Swarm Interaction in Safety Monitoring Applications
Ioannis Pitas, and Martin Saska
AI summary
Problem
Current human-UAV interaction systems lack integrated, real-time onboard processing for gesture control and dynamic formation adaptation, often relying on offboard computation or lacking robust safety features for complex environments.
Approach
A leader-follower UAV team uses an onboard pipeline to detect a worker, estimate their 3D position, and recognize hand gestures to dynamically adjust the formation’s shape and camera angles in real time.
Key results
- Fully onboard gesture recognition and human tracking pipeline
- Dynamic formation control with adaptive observation angles and distances
- Multi-modal human position estimation fusing camera, stereo, and UWB data
- Successful real-world validation with three UAVs in outdoor mock-up scenarios
Why it matters
This system empowers high-risk workers to safely and intuitively control aerial monitoring teams in real-time, reducing reliance on remote operators and improving situational awareness for infrastructure maintenance.
Abstract
This paper presents a formation control approach for contactless gesture-based Human-Swarm Interaction (HSI) between a team of multi-rotor Unmanned Aerial Vehicles (UAVs) and a human worker. The approach is designed to monitor the safety of human workers, particularly those operating at heights. In the proposed dynamic formation scheme, one UAV acts as the formation leader, equipped with sensors for detecting human workers and recognizing gestures. The follower UAVs maintain a predetermined formation relative to the worker’s position, providing additional perspectives of the monitored scene. Hand gestures enable the human worker to specify movement and action commands for the UAV team and to initiate other mission- related tasks without requiring additional communication chan- nels or specific markers. Combined with a novel unified human detection and tracking algorithm, a human position estimation method, and a gesture detection pipeline, the proposed approach represents the first instance of an HSI system incorporating all these modules onboard real-world UAVs. Simulations and field experiments involving three UAVs and a human worker in a mock-up scenario demonstrate the effectiveness and responsive- ness of the proposed approach.