C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning
Guanting Ye, Qiyan Zhao, Wenhao Yu, Xiaofeng Zhang, Jianmin Ji, Yanyong Zhang, and Ka-Veng Yuen,∗
AI summary
Problem
Inheriting standard Rotary Position Embedding (RoPE) from language models disrupts the spatial continuity of visual features and causes long-term attention decay, leading to spatial locality loss and visual token neglect in 3D Large Multimodal Models.
Approach
C2RoPE replaces standard RoPE with a spatio-temporal continuous positional encoding that fuses 1D temporal indices with 2D Cartesian coordinates using a frequency allocation strategy, and applies Chebyshev Causal Masking to enforce locality-aware attention during decoding.
Key results
- Identifies spatial locality loss and visual token neglect as critical RoPE limitations in 3D LMMs
- Introduces triplet hybrid positional indexing with frequency allocation for spatio-temporal encoding
- Achieves +4.3 EM@1 on ScanQA and +1.2 EM@1/EM@R on SQA3D over LLaVA-3D
- Demonstrates consistent cross-benchmark improvements in 3D scene reasoning and visual question answering
Why it matters
Provides a foundational positional encoding upgrade that enhances 3D scene understanding and cross-modal reasoning for robotics, navigation, and human-robot interaction systems.
Abstract
Recent advances in 3D Large Multimodal Mod- els (LMMs) built on Large Language Models (LLMs) have established the alignment of 3D visual features with LLM representations as the dominant paradigm. However, the inher- ited Rotary Position Embedding (RoPE) introduces limitations for multimodal processing. Specifically, applying 1D temporal positional indices disrupts the continuity of visual features along the column dimension, resulting in spatial locality loss. Moreover, RoPE follows the prior that temporally closer image tokens are more causally related, leading to long-term decay in attention allocation and causing the model to progressively neglect earlier visual tokens as the sequence length increases. To address these issues, we propose C2RoPE, an improved RoPE that explicitly models local spatial Continuity and spatial Causal relationships for visual processing. C2RoPE introduces a spatio-temporal continuous positional embedding mechanism for visual tokens. It first integrates 1D temporal positions with Cartesian-based spatial coordinates to construct a triplet hybrid positional index, and then employs a frequency alloca- tion strategy to encode spatio-temporal positional information across the three index components. Additionally, we introduce Chebyshev Causal Masking, which determines causal depen- dencies by computing the Chebyshev distance of image tokens in 2D space. Evaluation results across various benchmarks, including 3D scene reasoning and 3D visual question answering, demonstrate C2RoPE’s effectiveness. The code is be available at https://github.com/ErikZ719/C2RoPE.