Multi-object inference
Beliefs are maintained over the locations of the mug, teabag and kettle rather than a single target.
Embodied AI · Active inference · World models
Find the mug.
Make the tea.
The first demonstration showed an agent using semantic priors and negative evidence to locate a hidden mug. The new Habitat-Sim task carries the same reasoning into multi-object search, hidden-state inference and an 18-action plan that ends with tea served at the dining table.
3
objects reasoned about jointly
18
dependent task actions
Visible
beliefs, policies and hidden states
New demonstration · July 2026
From uncertain search to making a cup of tea
The agent must locate three objects, reason about what is currently known, satisfy action dependencies and track state changes from an empty kettle to a served cup of tea.
Multi-object task · full decision trace
Task relevance
The search policy can value an object that is useful downstream, even when another search node is currently active.
Hidden object states
The model tracks whether water is present and hot, whether tea is brewed and whether it has been served.
Dependent action sequence
Eighteen graph-linked actions connect locating and moving objects to filling, boiling, pouring, steeping and serving.
Decision-making logic
One model decides what to inspect and what to do next
Search and task execution are coupled. Evidence changes beliefs; beliefs change which actions are available and valuable; successful actions change the world state that the next decision is conditioned on.
Rank object–site hypotheses
The policy compares candidate searches across all unresolved task-relevant objects, balancing current probability, expected information gain and travel cost.
Use absence as evidence
When an object is not visible at an inspected site, probability mass is redistributed over the remaining hypotheses and the search order changes online.
Respect dependencies
Actions become eligible only after their graph dependencies are complete. The teabag cannot be inserted before the mug is placed, and pouring waits for both hot water and the teabag.
Update state beliefs
Inspecting, filling, boiling, pouring and steeping alter beliefs about hidden states. The task finishes only when the model is sufficiently confident that the tea has been brewed and served.
Search→
Locate→
Move objects→
Inspect kettle→
Fill & boil→
Pour & steep→
Serve
Foundational demonstration
Belief-guided search in a realistic 3D home
The original 32-second demo isolates the core inference loop: begin with a semantic prior, inspect useful locations, treat absence as evidence and replan until the cup is found.
The scientific idea
A world model that can expose its reasoning
The central object is not a fixed action policy. It is a structured belief about hidden states of the world, together with a model of what each possible action is expected to reveal.
World
A realistic ReplicaCAD apartment is rendered in Habitat-Sim. The agent receives egocentric observations and travels between collision-free inspection viewpoints on the navigation mesh.
Belief
The model maintains categorical beliefs over the locations of multiple objects, together with confidence about task-relevant states such as whether the kettle contains water, whether the water is hot and whether the tea has brewed.
Action
Candidate inspections are scored by travel distance, probability, expected information gain and task relevance. A dependency graph then exposes the next executable action once its prerequisites and belief preconditions are satisfied.
1
Start with structured priors
→
Objects and hidden states begin with explicit, inspectable probabilities.
2
Evaluate candidate policies
→
Which observation or action is useful, informative, relevant and reachable?
3
Act and observe
→
The agent navigates, inspects or manipulates and gathers new evidence.
4
Revise state and replan
Observations and action effects update beliefs, task eligibility and the next policy.
Accessible maths
The search policy in three ideas
The location-search component uses deliberately transparent equations. The tea-making demo applies this logic across object–site pairs, then hands resolved objects to the stateful task graph.
Update beliefs with evidence
\[
q_{t+1}(L_j)
=
\frac{p(o_{t+1}\mid L_j)\,q_t(L_j)}
{\sum_k p(o_{t+1}\mid L_k)\,q_t(L_k)}
\]
\(L_j\) is a possible location for the object currently being evaluated. When an inspected site is empty, its likelihood is reduced while alternatives remain plausible. Probability mass therefore moves elsewhere rather than simply disappearing.
Value information
\[
\mathrm{IG}_i
=
H[q_t(L)]
-
\mathbb{E}_{o\mid \pi_i}
\left[H\!\left(q_{t+1}(L\mid o,\pi_i)\right)\right]
\]
Entropy \(H[q]=-\sum_j q(L_j)\log q(L_j)\) measures uncertainty. An inspection is informative when it is expected to make the posterior sharper, whether the mug is found or ruled out.
Select the next inspection
\[
G_i
=
0.20\,d_i
-
3.00\,q_t(L_i)
-
1.50\,\mathrm{IG}_i
\]
\(d_i\) is the navigation distance to inspection site \(i\). The second term favours locations where the target object is currently likely. The third favours actions that reduce uncertainty. Lower \(G_i\) is preferred. In the tea-making task, candidate object–site pairs are also weighted by their relevance to the remaining task graph.
In plain language:
go somewhere plausible and informative, while accounting for the effort needed
to get there.
How this relates to expected free energy
In the broader active inference formulation, policies are evaluated using expected free energy:
\[
G(\pi)
\approx
\underbrace{\mathbb{E}[-\log p(o\mid C)]}_{\text{goal alignment}}
+
\underbrace{\mathbb{E}[H(p(o\mid s))]}_{\text{ambiguity}}
-
\underbrace{I(s;o\mid\pi)}_{\text{information gain}}
\]
Both demonstrations use a compact, interpretable approximation at the semantic planning level. The tea-making extension evaluates searches across multiple task-relevant objects and then gates downstream actions through dependencies and probabilistic state preconditions. The decision remains visible rather than being hidden inside a learned policy network.
Why it matters
Reasoning continues beyond finding the object
A useful world model must decide what to look for, learn from what is absent, compose actions over time and keep track of how its own interventions change the world. The tea-making task makes that full loop visible.
Beliefs, uncertainty and candidate actions remain visible throughout.
Search order and task execution change online as evidence changes the posterior.
Dependencies connect object search to a coherent sequence of goal-directed actions.
The model represents how inspection and manipulation change task-relevant hidden states.
Current scope
A controlled proof of concept, with a clear route forward
These demonstrations are designed to make the inference and task loop legible. They show meaningful progress from search to stateful action, without claiming general household autonomy.
Hand-specified semantic priors
Initial object-location and state probabilities encode ordinary task knowledge.
Idealised perception and inspection
Semantic simulator channels provide object detections and observations of selected hidden states.
Task graph and manipulation primitives
Active inference selects semantic searches and task actions while Habitat handles navigation and simulator-level interactions.
Learned perception and state estimation
Replace idealised channels with vision models that return graded, fallible object and state evidence.
Unseen homes and richer tasks
Test transfer across layouts, object categories, longer horizons and learned semantic relationships.
Learned grasping and lower-level inference
Replace scripted interaction primitives with grasping, locomotion and continuous active inference control.
Read the paper
Active Inference World Models: From Embodied Control to General-Purpose Adaptive Intelligence
The preprint develops the wider argument for world models grounded in active inference, linking embodied control, semantic priors, adaptive decision-making and general-purpose intelligence.
Shaw, A. D. & Berndt, L. C. S. (2026). Active Inference World Models: From Embodied Control to General-Purpose Adaptive Intelligence. Zenodo.
Alexander D. Shaw · AI & Technical CV · Consulting & collaboration