Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided, Self-Consistent MLLMs for Food Preparation Task Planning

Yu-Hong Shen*1, Chuan-Yu Wu*2, Yi-Ru Yang1, Yen-Ling Tai1, Yi-Ting Chen1
1Department of Computer Science, National Yang Ming Chiao Tung University
2Department of Computer Science and Information Engineering, National Taiwan University
*Indicates Equal Contribution
Paper Code & Dataset arXiv

We identify four categories of failures when using Multimodal Large Language Models (MLLM) with in-context learning for food preparation task planning. MLLMs fail to compare quantities between bowls, identify which bowls need repositioning before scooping, recognize spatial relationships between objects, and consider moving objects to avoid collisions. As a result, the robots may not follow the instructions properly and might even spill the bowl.

Abstract

We study Multimodal Large Language Models (MLLMs) with in-context learning for food preparation task planning. In this context, we identify two key challenges: cross-modal distraction and geometric feasibility. Cross-modal distraction occurs when the inclusion of visual input degrades the reasoning performance ofa MLLM. Geometric feasibility refers to the ability of MLLMs to ensure that the selected skills are physically executable in the environment. To address these issues, we adapt Chain of Thought (CoT) with Self-Consistency to mitigate reasoning loss from cross-modal distractions and use affordance predictor as skill preconditions to guide MLLM on geometric feasibility. We construct a dataset to evaluate the ability of MLLMs on quantity estimation, reachability analysis, relative positioning and collision avoidance. We conducted a detailed evaluation to identify issues among different baselines and analyze the reasons for improvement, providing insights into each approach. Our method reaches a success rate of 76.7% on the entire dataset, showing a substantial improvement over the CoT baseline at 36.7%.

Method

Video Presentation

BibTeX

@misc{shen2025mitigatingcrossmodaldistractionensuring,
    title={Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided, Self-Consistent MLLMs for Food Preparation Task Planning}, 
    author={Yu-Hong Shen and Chuan-Yu Wu and Yi-Ru Yang and Yen-Ling Tai and Yi-Ting Chen},
    year={2025},
    eprint={2503.13055},
    archivePrefix={arXiv},
    primaryClass={cs.RO},
    url={https://arxiv.org/abs/2503.13055}, 
}