Some collaborators and I recently released a first version of a benchmark we think highlights a critical gap in recent AI models in understanding causality in the real-world, beyond a physics focus.

Everyday environments are rich in tangible control interfaces (TCIs), like, light switches, appliance panels, and embedded GUIs, that are designed for humans and demand commonsense and physics reasoning, but also causal prediction and outcome verification in time and space (e.g., delayed heating, remote lights).

Paper: SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

Data and leaderboard in HuggingFace.

Feedback, suggestions, and collaborators are very welcome!