RoomMemory

What does a small AI model need to understand and edit a 3D world?

Status: Work in progress
Tools: Unreal Engine / Qwen3-VL-2B

RoomMemory prototype. In-game capture.

The prototype

Starting small.

I started with a small room, a plant, and a spell that changes the world around it. I wanted to explore a game where a player can describe a change in their own words, while the world remembers what they placed and what should stay.

The first version is playable. I can place and rotate the plant, transform the room into prepared jungle or underwater environments, and return. The plant even has a little bounce when clicked.

The prototype uses Qwen3-VL-2B locally with Unreal. Written spells produce a proposal that I can apply or dismiss. The environments and assets already exist; Unreal's rules preserve objects and their support relationships.

That gave me a concrete place to ask a harder question: can the model choose the right change, especially when the room has a history?

Early observations

What the first tests showed.

01 Text / history / tools

The lamp that changed back

In turn 9 of a ten-turn dialogue, the request was to undo an older lamp change while preserving a later one. The model chose the wrong edit: the lamp went from lavender back to gold. It then claimed it had preserved the later change.

Expected

Keep the later lavender appearance

Actual

Reverted to gold while claiming success

Schematic of the reported result, not a screenshot.

The operation was valid, but its target was wrong. This editing test used text, history and tools, not images. It made the gap between a successful tool call and a successful request very clear.

02 One image / one description

A separate visual check

For a different test, I used one unchanged room screenshot and only the question, “What do you see in this image?” No object names, scene description, scene JSON or conversation history were added.

The run review reports that Qwen identified the main furniture and the plant on the table, but may have miscounted the objects on a shelf. That possible error still needs a side-by-side check of the image and raw response. This was one description, not an editing or undo test.

Questions to explore

What I want to explore next.

Context from the engine

How little context can I give a small model without losing what matters? One idea is to combine an image with a short, relevant set of facts from Unreal: object identities, properties and relationships. Facts supplied by the engine would be assistance, not evidence that the model inferred them visually.

Task difficulty and prompt help

I am also considering a difficulty dial, varying scene complexity and task demands. That would need to stay separate from how much help the prompt provides. Neither idea is implemented or validated yet.

Back to Lab