LLM-enhanced Scene Graph Learning for Household Rearrangement

Wenhao Li1,*       Zhiyuan Yu1,*       Qijin She1,*       Zhinan Yu1       Yuqing Lan1       Chenyang Zhu1       Ruizhen Hu2,†       Kai Xu1,†      
1 National University of Defense Technology       2Shenzhen University *Indicates Equal Contribution,  Corresponding Author
Description of Image

We propose a LLM-enhanced scene graph learning method to construct affordance-enhanced graph (AEG) and perform task planning for house-tidying rearrangement.

Abstract

The household rearrangement task involves spotting misplaced objects in a scene and accommodate them with proper places. It depends both on common-sense knowledge on the objective side and human user preference on the subjective side.

In achieving such task, we propose to mine object functionality with user preference alignment directly from the scene itself, without relying on human intervention. To do so, we work with scene graph representation and propose LLM-enhanced scene graph learning which transforms the input scene graph into an affordance-enhanced graph (AEG) with information-enhanced nodes and newly discovered edges (relations). In AEG, the nodes corresponding to the receptacle objects are augmented with context-induced affordance which encodes what kind of carriable objects can be placed on it. New edges are discovered with newly discovered non-local relations.

With AEG, we perform task planning for scene rearrangement by detecting misplaced carriables and determining a proper placement for each of them. We test our method by implementing a tiding robot in simulator and perform evaluation on a new benchmark we build. Extensive evalua- tions demonstrate that our method achieves state-of-the-art performance on misplacement detection and the following rearrangement planning.

Pipeline

Description of Image

Given a scene graph (SG) with key frames, we utilize Large Language Model (LLM) to perform context-induced affordance analysis for all objects in the scene. These affordances are incorporated into the SG, updating the nodes and edges to construct an Affordance Enhanced Graph (AEG). We then evaluate the appropriateness of the current placements of all carriable objects in the AEG based on the affordance information of the objects and their receptacles, identifying misplaced items. For each misplaced carriable, we rate the suitability of each receptacle in the AEG as a placement target. The top k suitable receptacles are selected as candidates, and their affordances are retrieved as prompt to the LLM to generate the placement decision.

Visual Results

Simulated Environment

Description of Image

Real Scene

Description of Image

BibTeX

@article{li2024llm,
      title={LLM-enhanced Scene Graph Learning for Household Rearrangement},
      author={Li, Wenhao and Yu, Zhiyuan and She, Qijin and Yu, Zhinan and Lan, Yuqing and Zhu, Chenyang and Hu, Ruizhen and Xu, Kai},
      journal={arXiv preprint arXiv:2408.12093},
      year={2024}
    }