Andrea Ricci managed to get Claude Sonnet to play DOOM by transmitting images captured from the game screen. To do this, Ricci ported the game to SCINTIX P4 and sent the images to the model via WebSockets; the model then analyzed what it saw and issued movement and firing commands.
The game runs on an ESP32-P4 with a 1024×600 MIPI DSI panel. The system uses depth scanning within the field of view to detect walls and an ASCII automap to track previously visited areas. Claude Sonnet advances through corridors, opens doors, and fires at enemies without additional assistance.
Why it matters
This experiment shows that a language model is not limited to generating text, but can also process visual inputs and direct actions in a physical game system. For developers, the main value lies in establishing communication between the model and embedded hardware through a connection such as WebSockets, as well as using auxiliary layers that make it easier to understand the game world. This approach offers a concrete example of how AI models can be connected to existing software and devices. However, what remains unclear is to what extent the model could sustain this performance in different games, with more complex interfaces, or in less structured visual environments; the current demonstration alone does not establish that scope.
Background
Claude is not a new name in the FikirPilot archive: we published a news article mentioning the name in the last 90 days; that article was dated August 26, 2026. This development is added alongside those records in our archive; readers can follow earlier articles bearing the same name through the archive to see the timeline as a whole.