Programasweights lets you control 3D avatars with plain English
Describe actions like 'wave while walking' and the avatar executes instantly.
Programasweights, created by developer yuntiandeng, introduces a 3D avatar that users can control with plain English instead of predefined buttons or scripts. At its core, the system uses a 'director' neural program—compiled from earlier work on programasweights—that at runtime converts a user's sentence into a tiny action program. This program includes loops, holds, and parallel tracks, allowing sequences like 'wave while walking, then jump a couple times' that would be cumbersome to hardcode. The action program executes locally in the browser after an initial download of the small model. A debug panel (enabled via ?dbg=1) shows the exact action program written for each command, offering transparency into the compilation process.
The implications for gaming are significant: instead of NPCs following fixed, hand-authored recipes, they could improvise behavior from user chats and emotions, with the model writing action programs on the fly. The inference and runtime code is already released on GitHub, and a paper/tweet provides background on the approach. An uncleaned version of the full code is also available, with a cleaned release planned. This system makes avatar control intuitive and opens the door to dynamic, language-driven interactions in virtual worlds.
- Directly type natural language commands to control a 3D avatar; the system compiles sentences into action programs using loop, hold, and parallel constructs.
- The 'director' neural program runs locally in the browser after an initial download, enabling offline execution with minimal latency.
- Open-source inference code is available on GitHub, and the approach is detailed in a tweet/paper—aimed at enabling dynamic NPC behavior in games.
Why It Matters
Replaces rigid button-based avatar controls with intuitive language commands, and could enable dynamic, improvised NPC behaviors in games.