Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation
Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharov, Vitor Guizilini, YUE WANG
AI summary
Problem
Existing humanoid datasets lack task diversity, full-body locomotion, and human-robot interaction, while the field lacks standardized evaluation frameworks for fair policy comparison.
Approach
The authors engineered a low-latency teleoperation pipeline to collect a multimodal dataset of 260 diverse humanoid tasks and introduced a cloud-based platform for standardized real-world policy benchmarking.
Key results
- A multimodal dataset of 10.3k trajectories across 260 tasks and 7 categories
- A teleoperation pipeline reducing control delay to 2ms and halving data collection time
- Policy analysis showing imitation and VLA models struggle with high-dimensional action spaces and loco-manipulation
- A cloud-based evaluation platform enabling standardized real-world humanoid policy benchmarking
Why it matters
It equips the robotics community with a standardized benchmark and rich data to accelerate the development of robust, general-purpose humanoid manipulation policies.
Abstract
From loco-motion to dextrous manipulation, hu- manoid robots have made remarkable strides in demonstrating complex full-body capabilities. However, the majority of current robot learning datasets and benchmarks mainly focus on stationary robot arms, and the few existing humanoid datasets are either confined to fixed environments or limited in task diversity, often lacking human-humanoid interaction and lower- body locomotion. Moreover, there are a few standardized evaluation platforms for benchmarking learning-based policies on humanoid data. In this work, we present Humanoid Everyday, a large-scale and diverse humanoid manipulation dataset characterized by extensive task variety involving dextrous object manipulation, human-humanoid interaction, locomotion- integrated actions, and more. Leveraging a highly efficient human-supervised teleoperation pipeline, Humanoid Everyday aggregates high-quality multimodal sensory data—including RGB, depth, LiDAR, and tactile inputs—together with natural language annotations, comprising 10.3k trajectories and over 3 million frames of data across 260 tasks across 7 broad categories. * Equal contribution. † Equal advising. In addition, we conduct an analysis of representative policy learning methods on our dataset, providing insights into their strengths and limitations across different task categories. For standardized evaluation, we introduce a cloud-based evaluation platform that allows researchers to seamlessly deploy their policies in our controlled setting and receive performance feedback. By releasing Humanoid Everyday along with our policy learning analysis and a standardized cloud-based evaluation platform, we intend to advance research in general-purpose humanoid manipulation and lay the groundwork for more capable and embodied robotic agents in real-world scenarios. Our dataset, data collection code, and cloud evaluation website are made publicly available on our project website: https: //humanoideveryday.github.io