Prompt-driven texture changes
Send descriptive text to modify textures instantly.
StyleShift
Real-time texture generation with Stable Diffusion
This prototype explores how generative AI can be used to dynamically transform 3D environments in real time. Starting with a single object, a cup, I experimented with applying different styles using text prompts via Stable Diffusion. These included materials like ceramic, glass, and artistic styles like sketch or watercolor.
After validating the concept, I extended it to the full room. The walls, floor, and objects all change textures based on the selected prompt, giving the user the feeling of being instantly transported to a different environment. This allows for immersive mood or theme changes without scene transitions or reloading.
Concept
StyleShift treats text prompts as a live environment control. Instead of pre-authored art direction or hard-coded material swaps, the user can reshape how a space looks and feels through language — from a single object to an entire room.
The prototype was developed as part of early exploration for 8pm and the Cat, testing whether real-time generative visuals could support adaptive mood and atmosphere in VR storytelling.
System Design
The system enables real-time texture transformation based on user-defined prompts. A descriptive text prompt (e.g., "wooden texture", "futuristic metal", "cozy hand-drawn") is sent from Unity to the Stable Diffusion Web UI API, along with a conditional image using the object's UV map (via ControlNet). The resulting AI-generated texture is decoded and applied directly to 3D models within the Unity scene.
Pipeline
Prompt
Stable Diffusion (ControlNet)
Texture Generation
Scene Update
Key Features
Real-time AI integration
Uses Stable Diffusion to generate and apply textures during runtime.