CaRe-Ego: Contact-aware Relationship Modeling for Egocentric Interactive Hand-object Segmentation

Yuejiao Su   Yi Wang   Lap-Pui Chau

The Hong Kong Polytechnic University

Video Demonstrations

We validate the effectiveness of the CaRe-Ego on several out-of-distribution videos from THU-READ dataset.

Abstract

Egocentric Interactive hand-object segmentation (EgoIHOS) requires the segmentation of hands and interacting objects in egocentric images, which is crucial for understanding human behavior in assistive systems. Previous methods typically recognize hands and interacting objects as distinct semantic categories based solely on visual features, or simply use hand predictions as auxiliary cues for object segmentation. Despite the promising progress achieved by these methods, they fail to adequately model the interactive relationships between hands and objects while ignoring the coupled physical relationships among object categories, ultimately constraining their segmentation performance. To make up for the shortcomings of existing methods, we propose a novel method called CaRe-Ego that achieves state-of-the-art performance by emphasizing the contact between hands and objects from two aspects. First, we introduce a Hand-guided Object Feature Enhancer (HOFE) to establish the hand-object interactive relationships to extract more contact-relevant and discriminative object features. Second, we design the Contact-centric Object Decoupling Strategy (CODS) to explicitly model and disentangle coupling relationships among object categories, thereby emphasizing contact-aware feature learning. Experiments on various in-domain and out-of-domain test sets show that Care-Ego significantly outperforms existing methods with robust generalization capability.


CaRe-Ego Framework

Illuatration of the proposed CaRe-Ego. The method comprises four main components: an encoder, a multi-branch decoder, a hand-guided object feature enhancer (HOFE), and a contact-centric object decoupling strategy (CODS). Rep. in this figure denotes representations.

Qualitative Results

Comparison results on the EgoHOS in-domain test set.
Comparison results on the EgoHOS out-of-domain test set.
Comparison results on the out-of-distribution mini-HOI4D dataset.

Citation

@misc{su2025careegocontactawarerelationshipmodeling,
      title={CaRe-Ego: Contact-aware Relationship Modeling for Egocentric Interactive Hand-object Segmentation}, 
      author={Yuejiao Su and Yi Wang and Lap-Pui Chau},
      year={2025},
      eprint={2407.05576},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.05576}, 
  }
  

Data

News: The mini-HOI4D dataset and the checkpoint of the best model have been released.

mini-HOI4D. Google Drive.

Model. Google Drive.