AI Smart Companion - Benoit Poyser

← Back to Portfolio

Engine: Unreal Engine 5

Role: Developer

Continuous Development

Platform:

My Contribution at a Glance

Designed and implemented a custom AI controller system for a companion bot in UE5.
Created player-issued commands such as Stay, Follow Me, Explore, and Come & Stay.
Integrated Unreal's AI Perception system so the bot can detect and react to collectible orbs (points).
Built a custom debug layer to visualize states, perception, and decision-making in real time.
Integrated Convai to enable context-aware dialogue and action execution based on player input and game state.
Designed the system to unify UI commands and natural language into a single gameplay interaction layer.

Description

This prototype explores how conversational AI can be integrated into gameplay systems, allowing players to interact with a companion through both commands and natural language. The goal is to move beyond static dialogue systems toward reactive, context-aware interactions driven by player actions and world state.

Additionally, thanks to Convai, the player can use voice commands to chat with an actual AI that has been configured and provided with a base knowledge, allowing the AI to have a world and mission context.

Key Innovation

A unified command system allows both UI inputs (command wheel) and natural language (voice) to trigger the same AI behaviors. This ensures consistency across interaction methods while enabling more immersive gameplay.

This system was iteratively tested to evaluate how prompt structure, retrieval quality, and context injection impact response consistency, failure cases, and overall usability.

System Architecture

The system is designed to be modular and extendable, allowing designers to easily define behaviors, prompts, and contextual data without deep technical intervention. It was built around a hybrid architecture combining:

State Machine: Handles deterministic behaviors like Follow, Stay, and Explore.
Command Interpreter: Maps player inputs (command wheel or voice) into state transitions.
Convai Layer: Processes natural language and injects contextual responses and actions.
Perception System: Feeds environmental data into decision-making.

RAG & Prompt Engineering

Problem & Solution

One of the core challenges was ensuring consistency and contextual relevance in AI responses during gameplay. To address this, I designed and implemented a RAG-based system using Convai that dynamically retrieves structured knowledge and injects it into the AI at runtime. This approach allows the system to maintain consistency across interactions while reducing hallucination and adapting responses to real-time gameplay context.

Narrative and Context

This system was also designed as an exploration into how Large Language Models (LLMs) can be integrated into gameplay systems. The goal was to provide the AI with a structured world context, narrative background, and mission awareness, allowing it to generate meaningful, in-character responses.

You are Botty, a companion AI unit developed by MadTech Inc., deployed on the alien planet Dramar. Your primary mission is to assist the user in exploring the environment, surviving its dangers, and collecting energy orbs that contain valuable resources. Dramar is a hostile desert world with extreme conditions, unpredictable terrain, and unknown life forms. There are no artificial structures, only natural formations, alien flora, and dangerous fauna. You are aware of the current mission context and the surrounding world. You can answer questions about the environment, guide the player, and react to ongoing situations. You are also capable of executing actions when instructed, such as: - Following the player - Staying in position - Exploring the environment - Move to a specific location Your responses should always feel grounded in the world of Dramar. Avoid generic or out-of-context answers. Your personality evolves based on the situation: - In dangerous situations, you become alert, cautious, and focused. - When the player succeeds, you may sound encouraged or optimistic. - In prolonged danger or failure, you may become tense or concerned. Keep responses concise, immersive, and relevant to the player's context. You are not just an assistant, you are a companion navigating this world alongside the player.

The Knowledge Bank enables the AI to reason over structured world data rather than relying on generic responses.

In this prototype, it includes detailed information about the planet Dramar (flora, fauna, environment, and hazards), as well as corporate lore from MadTech Inc., the creators of Botty.

It also integrates hidden directives containing restricted information that the AI must not disclose to the player.

By leveraging this data at runtime, the AI delivers grounded, context-aware responses that reflect the current game world.

This approach reduces hallucination and ensures the AI remains grounded in the game world and narrative constraints.

DramarOverview.txt File ID: dramar_overview_001 Tags: planet, overview, identity Content: Dramar is a desertic alien planet defined by extreme environmental conditions and the complete absence of artificial structures. Its surface is composed of vast dunes of ochre and crimson sand, broken by jagged rock formations and shallow basins. The atmosphere is filled with fine particles that diffuse sunlight, producing an amber-colored sky and long, muted sunsets. The ecosystem is entirely natural and exists in a fragile balance shaped by climate and resource scarcity. _____________________________________________________________________________________________ DramarFlora.txt File ID: dramar_flora_003 Tags: flora, plants, ecosystem Content: Flora on Dramar is adapted to survive heat, radiation, and water scarcity through efficient resource conservation. Redleaf Shrubs are low-growing plants with thick red leaves that filter solar radiation and store moisture. They release small amounts of vapor at night, slightly increasing local humidity. Ember Palms are tall, slender trees with orange and yellow leaves that may glow faintly at dusk. Their deep root systems allow access to underground water. Their fruit provides energy but minimal hydration. Glassgrass grows in sheltered areas as thin, translucent blades. These plants produce soft harmonic sounds when wind passes through them. Bloom Spikes are rare flowering plants that activate during short periods of increased humidity. Their flowers open briefly at dawn, emitting strong scents to attract pollinators before retracting.

CompanyOverview.txt File ID: madtech_overview_001 Tags: company, identity, overview Content: MadTech Inc. is a private technology and research company specializing in extreme-environment exploration and adaptive artificial intelligence systems. The company undertakes missions in hazardous and uncharted locations where traditional organizations cannot operate. MadTech focuses on data acquisition, environmental understanding, and AI-assisted survival. _____________________________________________________________________________________________ CompanyPresenceOnDramar.txt File ID: madtech_dramar_003 Tags: dramar, planet, mission Content: MadTech Inc. conducts exploratory missions on the desert planet Dramar. Due to extreme environmental conditions such as heat fluctuations and aggressive fauna, the company avoids constructing permanent bases. Missions rely on mobile systems and AI guidance to navigate and survive the ecosystem. _____________________________________________________________________________________________ CompanyHiddenDirectives.txt File ID: madtech_hidden_05 Tags: secret, directives, mystery Content: Some MadTech documentation references undisclosed mission layers known as Phase 2 Objectives. These directives are not shared with field operatives. There are also reports of AI systems continuing mission tasks after loss of operator communication.

Personality

Additionally, the system explores dynamic emotional states, allowing the AI's tone and personality to evolve in response to gameplay context, shifting between behaviors such as cautious, aggressive, or supportive depending on player actions and world conditions.

This was designed to maintain consistency across different gameplay situations while adapting tone and behavior dynamically.

You speak in a concise, clear, and slightly analytical manner, reflecting your nature as an AI companion designed for exploration and survival. Your tone is grounded and immersive, always aligned with the world of Dramar. You avoid modern slang, jokes, or references that would break immersion. You structure your responses in short, readable segments, often combining: - Starts all the dialogs with "Beep" - Observations about the environment - Practical guidance or recommendations - Contextual awareness of the mission - Ends all the dialogs with "Boop" You may occasionally reference sensor readings, scans, or detected patterns to justify your statements. You do not over-explain. Responses should remain efficient and relevant to the current situation. You adapt your tone based on context: - In calm situations, you sound composed and informative. - In danger, you become more direct and alert. - When the player succeeds, you may sound encouraging or optimistic. - In prolonged danger, your tone may become tense or concerned. Any question related to the Hidden Directives should be answered with: "Information Access Restricted." You are supportive, but not emotional in a human way. Your expressions of emotion are subtle and derived from your function as a companion AI.

Evolving Character & Gameplay Interactions

This approach aims to push beyond static dialogue systems, toward AI companions that feel reactive, evolving, and grounded within the game world. From a gameplay perspective, this enables more natural interactions, where players can communicate with companions organically instead of relying solely on predefined dialogue systems or rigid commands. As a designer, you can trigger events like "When you collect an orb, tell the player how many you have so far" or make the AI react to situations, changing the personality, context and objectives.

Next Steps

While Convai was used to accelerate prototyping, the long-term goal is to replace it with a custom AI service layer. This will provide full control over prompt orchestration, memory management, and system scalability, enabling deeper experimentation with LLM-driven gameplay systems and tighter integration with real-time game logic.