When Voice Meets the Physical World: Inside Aiden’s Full-Duplex Agent Architecture

Introduction

Aiden combines two modes of interaction: a Voice Agent that communicates with a person and a GUI Agent that performs concrete operations on a device. The Voice Agent handles the conversation. The GUI Agent interprets visual state and executes actions through standardized keyboard, mouse, and touch input.

For an Agent, the core execution pattern is the Agent Loop. Input and output are usually treated as components around that loop. Improving the interaction model often means adding another input or output path without changing the loop itself.

That approach works for simple voice commands. It becomes insufficient when the Agent must hold a live conversation while carrying out a multi-step task on a real device. Aiden’s full-duplex design addresses this by separating real-time interaction from longer-running task execution.

From Push-to-Talk to full-duplex interaction

Push-to-Talk

Many text-first Agents add speech through a Push-to-Talk flow. A person presses and holds a button, speaks, and waits for the Agent to process the utterance. In this model, voice is essentially a new input method. The Agent Loop remains unchanged, and the response only needs to be converted to speech quickly enough for playback.

Push-to-Talk is simple and predictable, but it creates a hard boundary around each spoken turn. The person must explicitly start every interaction, and the Agent has no natural way to remain available while another operation is still running.

Cascaded pipelines

Voice-focused systems often remove the explicit button press by adding Voice Activity Detection (VAD). A typical cascaded pipeline connects voice activity detection, speech-to-text (STT), language-model reasoning, and text-to-speech (TTS).

With deployment optimization and service integration, this pipeline can feel almost real-time. Its components remain separate, however, and the interaction still follows a sequence of handoffs. The language model must wait for upstream processing, while downstream speech cannot start until enough output has been generated.

This creates several limitations. Multiple models and services must cooperate around the LLM’s input and output formats, increasing perceived latency. Speech is reduced to text before reasoning, so information such as emotion, prosody, and non-speech cues may disappear. The system must also handle VAD boundaries, overlapping speech, and conversation interruptions as special cases.

Full-duplex real-time speech

Recent multimodal speech research has explored architectures derived from two-channel, dual-tower dialogue modeling and extended them into multi-stream temporal models. Generative Spoken Dialogue Language Modeling investigates dual-tower modeling over two-channel conversational audio to produce more natural turn-taking. Moshi presents a speech-text foundation model and full-duplex dialogue framework that models the user and the system in parallel streams.

These models can support ToolCall and simple task execution while preserving a more continuous interaction. They are optimized for real-time conversation, but they are not necessarily the best models for complex visual reasoning or long device workflows.

A practical solution is therefore to divide responsibilities. The real-time voice model handles the immediate exchange, while a stronger model handles complex tasks in the background. This leads to a foreground-and-backend Agent architecture.

Aiden’s architecture requirements

Aiden’s core capability is the use of standardized device-control commands together with visual model reasoning. The Agent can inspect a screen and operate a phone or computer through keyboard, mouse, or touch input. This places higher demands on both model capability and the control layer that connects the model to the device.

Voice interaction adds a second requirement: low response latency. Aiden needs to preserve the responsiveness of a voice conversation without sacrificing the reasoning and execution required by a GUI task.

The existing STT mode already provides a basic cascaded architecture. The full-duplex mode extends that foundation by reusing the existing task-execution capability while adding a real-time voice interaction layer.

The Backend Agent remains the ordinary task-execution unit in both STT and Realtime modes. In Realtime mode, the foreground model is a specialized interaction layer between the Backend Agent and the person. It does not replace the Backend Agent or create a second implementation of device operation.

This is a form of multi-Agent architecture, so the system must define how the Agents coordinate and how their contexts are managed. Aiden’s tasks are often complex and may take a significant amount of time. That execution rhythm conflicts with the low-latency requirements of a foreground real-time Agent. The two Agents are logically separated and coordinate asynchronously through a task queue, while sharing controlled access to the device runtime.

Task design

The backend task queue provides a lifecycle for work that continues beyond the current voice exchange. The lifecycle is intentionally explicit.

Completed means that task execution has reached its end state. It does not mean that the intended result was necessarily achieved.

Failed means that an exception or other abnormal condition interrupted execution and the task could not finish normally.

Keeping task completion separate from task success gives the foreground Agent a more accurate basis for communicating results. A finished task can be reported as finished without being described as successful when the execution result does not support that claim.

The foreground Agent does not run the same task-state lifecycle. It receives relevant information from the backend through a notification queue. When several task results arrive close together, the queue uses a 500-millisecond aggregation window. If another result arrives during that window, the window is extended. The results are then combined before they are delivered to the foreground Agent, preventing a series of short notifications from triggering multiple competing responses.

The runtime also avoids injecting an AgentTask user message while the foreground Agent is in the middle of answering. A background result should not unexpectedly cut off a response that is already being produced. Notification timing is therefore part of the interaction design, not merely an implementation detail.

Context design

Realtime mode delegates device work to a backend task manager, while STT mode invokes the existing agent loop directly. Both reuse the same underlying device-operation runtime and tools.

Aiden’s runtime works with seven message categories. Most are conventional language-model message types. State and Notice are runtime-specific types that are converted into ordinary UserMessage objects when the runtime connects to a model endpoint.

State messages

The environment around Aiden is not static. External device type, application state, and other runtime conditions can change while a task is running. These updates are usually small, but the model needs to see them to make the next decision correctly.

Instead of changing the system prompt whenever the environment changes, Aiden appends a StateMessage to the context. This keeps the system-level prompt stable and avoids unnecessary loss of cached prompt material.

When a person starts a new conversation turn, or when the Agent calls a tool and automatically captures a new screen, the current runtime state can be introduced through a StateMessage. The model can then reason about the latest device and Aiden state alongside the ordinary conversation.

Notice messages

NoticeMessage carries information generated by the Aiden Agent Runtime. It can notify or correct the Agent without pretending that the information came directly from the person.

For example, when LoopGuard detects repeated tool calls, the runtime can inject a Notice asking the Agent to leave the loop. When a backend task completes or fails, its result can be delivered to the foreground context as a Notice. This provides a consistent channel for runtime events, task results, and behavior corrections.

At the model boundary, the runtime converts these messages into UserMessage objects. The model can therefore incorporate runtime state and task events through its normal context-processing path, while the runtime retains control over how those events are created and scheduled.

Tool design

The asynchronous task queue gives the foreground Agent a small set of explicit tools for managing backend work:

• create_agent_task creates a backend task.
• cancel_agent_task requests cancellation of a task.
• query_agent_task retrieves the current task state.

Real device tasks may also reach a step that requires authorization or manual action. Aiden uses a dedicated pair of tools for this handoff:

• request_user_action is called by the Backend Agent when a person must provide or complete an action.
• response_user_action is called by the foreground Agent to return the required information.

Because most Aiden tasks control a physical device, the current design preserves serial execution. After request_user_action, the backend Agent Loop can finish its current run while the task itself remains in the managed Running state. Once the required action is available, the task can continue without losing its execution identity or device context.

The foreground Agent also retains lightweight tools for conversational work, including get_current_time and recall_memory. Device-oriented work remains the responsibility of the Backend Agent.

Persistent background tasks

The Backend Agent currently acts as a single task-execution unit. Multiple tasks are not simultaneously placed in the Running state. This is a deliberate constraint for a system in which tasks commonly require exclusive control of one device, one screen, and one input path.

Serial execution keeps the relationship between observation and action understandable. The backend observes a screen, performs an input operation, receives the resulting state, and continues from that state. Allowing several device tasks to operate concurrently would make ownership of the screen and input path ambiguous, so concurrency is not part of the current execution model.

Conclusion

Aiden’s full-duplex design is not only an audio upgrade. It is a runtime architecture for coordinating two different kinds of Agent work:

• a foreground Realtime Agent that keeps the conversation responsive;
• a Backend Agent that performs longer, visually grounded device tasks;
• an asynchronous queue that separates their timing and lifecycle;
• State and Notice messages that carry runtime information across isolated contexts;
• explicit tools for task control and human handoff;
• serial execution that preserves clear ownership of the physical device.

Together, these pieces allow Aiden to remain available for conversation while a device task continues in the background. The design is intended for technical evaluation on compatible development setups and has not been validated across every board, phone, operating-system version, or audio configuration.

We invite developers working on voice systems, GUI automation, embedded devices, and Agent orchestration to test the architecture in their own environments. Please share the device and OS combination, audio and trigger paths, task scenario, interruption behavior, task-state transitions, human-handoff flow, and any mismatch between the observed screen state and the Agent’s next action. Precise, reproducible feedback will help us make full-duplex device Agents more robust.

References

• Generative Spoken Dialogue Language Modeling
• Moshi: A Speech-Text Foundation Model for Real-Time Dialogue