Voice Capabilities: VAD / STT / TTS

The Go Agent supports device-side voice interaction, primarily consisting of internal/agent/audio_client.go, audio_dialog.go, vad.go, stt.go, and the tts/ provider.

Architecture

┌─────────────┐
│   daemon    │
└──────┬──────┘
       ├─────────────────────────► HTTP Server / Web UI (all input modes)
       │
       └─ input_mode=stt ────────► Audio Dialog Loop
                                   │
                                   ├─ AudioServiceClient
                                   ├─ VAD
                                   ├─ STT
                                   ├─ LLM
                                   └─ TTS

Components

Component	Files	Description
Audio client	audio_client.go	Connects to audio_service, starts recording/playback sessions, reads/writes PCM chunks
VAD	vad.go + /oem/usr/bin/rknn_vad or /oem/usr/bin/cpu_vad	Silero VAD inference; input is fixed at 16 kHz, 512 samples/32 ms, with state maintained in helper
STT	stt.go, provider-specific clients	OpenAI Whisper, OpenRouter, Tencent Cloud ASR, Qwen ASR, and Google Cloud STT; tencent / tencent_asr remain compatibility aliases for tencent-asr
TTS	tts/, tts_helpers.go	Pluggable TTS provider, outputs PCM for the configured playback backend; automatically resamples to the target playback sample rate when necessary
Dialog manager	audio_dialog.go	Orchestrates recording, VAD, STT/LLM/TTS flow
Voice notifications	voice_notification.go, tts_fallback.go	Adds a persistent response tail or final-turn failure replacement before TTS, with bundled local WAV fallback when final TTS is unavailable

Input Modes

input_mode = "text"

Runs the HTTP server and Web UI without starting the device-side audio loop. Browser clients can still upload or record audio:

• If [stt] is configured, browser audio is transcribed to text first;
• Otherwise audio is passed as model attachment.

input_mode = "stt"

Runs the device-side audio loop alongside the HTTP server and Web UI:

1. Records PCM via audio_service;
2. VAD determines sentence boundaries;
3. Sends WAV to STT;
4. STT text goes to Agent runtime;
5. TTS synthesizes the reply and plays it through the configured playback backend.

Before final non-streaming TTS, the runtime passes the spoken reply through the shared Voice Notification manager. A successful turn may receive one short persistent tail. A final LLM failure may use a fixed replacement. These changes affect spoken text only, not assistant history or UI response text.

Trigger Modes

trigger_mode = "manual"

Press Enter to start recording, press Enter again to stop, or let VAD determine silence end.

trigger_mode = "wakeup"

Waits for GPIO 33 or GPIO 32 falling edge trigger to start recording; both trigger paths enter the same wakeup flow. When input_mode = "stt" and voice_followup_enabled = true, wakeup opens a continuous voice session: the first turn still requires GPIO, but after the Agent replies it will continue listening for follow-up questions within the voice_followup_timeout_ms window. voice_first_turn_timeout_ms controls the first-turn waiting window, and voice_max_turns controls the maximum number of turns per session. Repeated wakeup triggers during listening or recording are merged or ignored and will not restart recording or discard already-recorded audio; whether triggering wakeup again during thinking cancels the current LLM request is controlled by voice_interrupt_on_wakeup; the microphone is not open by default during TTS playback, and recording continues only after playback ends; triggering wakeup again during playback will interrupt the current turn and immediately start recording. Requires Linux GPIO sysfs to be available and hardware wiring to be completed.

Configuration Snippet

input_mode = "stt"
trigger_mode = "wakeup"
vad_backend = "rknn"
vad_model_path = "/oem/usr/model/silero_vad_6_2_encoder_rv1106_w8a8_v1.rknn"
vad_helper_path = "/oem/usr/bin/rknn_vad"
vad_speech_threshold = 0.5
silence_ms = 550
min_speech_ms = 300
voice_followup_enabled = false
voice_followup_timeout_ms = 5000
voice_first_turn_timeout_ms = 10000
voice_max_turns = 0
voice_interrupt_on_wakeup = true
voice_streaming_tts_enabled = true
voice_tool_call_speech = true
voice_max_response_tokens = 300

[audio]
socket = "/run/audio_service/audio_service.sock"
sample_rate = 16000
channels = 1
bit_width = 16
playback_backend = "auto"

[stt_providers.openai-main]
type = "openai-whisper"
api_key = "$OPENAI_API_KEY"
model = "whisper-1"

[stt]
provider = "openai-main"

# OpenRouter alternative:
# [stt_providers.openrouter-main]
# type = "openrouter"
# api_key = "$OPENROUTER_API_KEY"
# model = "qwen/qwen3-asr-flash-2026-02-10"
# Set [stt].provider = "openrouter-main" to select it.

[tts_providers.alicloud-main]
type = "alicloud"
api_key = "$DASHSCOPE_API_KEY"
model = "qwen3-tts-flash-realtime"
voice_id = "Cherry"
emotion = "happy"

[tts]
provider = "alicloud-main"
speed = 1.0

Tool Call Speech

When voice_tool_call_speech = true, the runtime will asynchronously TTS and play the content of tool call events when they arrive. This content comes only from the assistant content in the same LLM tool-call response, not from tool parameters.

If the LLM does not generate assistant content in the same response as the tool call, that tool call remains silent. The runtime does not derive short speech announcements from speech, description, or other context fields in tool parameters.

TTS Provider Usage

Common fields in [tts] are provider, api_key, model, voice_id, emotion, speed, and reference_id. Different providers interpret fields differently; see Agent Configuration Reference for full details. The following examples omit api_key and only show adapter behavior-related configuration.

All TTS providers are called through a unified streaming session: the Agent writes LLM output fragments to the adapter, and the adapter decides when to send to the backend. Fish Audio, Alicloud, and Volcengine are true streaming WebSocket links; the Minimax WebSocket adapter buffers internally at sentence boundaries before sending, so the upper layer doesn't need to distinguish between “true streaming” or “sentence-level streaming”. The runtime can switch providers via POST /api/settings/tts; playback that has already started will continue using the old provider, and subsequent requests will use the new provider.

Every assistant response uses one protocol: its first bytes are exactly one leading <tts>...</tts> block, followed by the user-facing text. This applies both to final answers and to progress content emitted with tool calls. The spoken and visible forms do not need to be identical.

Only the first leading TTS block in each LLM response is streamed. When its closing </tts> tag arrives, the runtime immediately flushes the provider so short MiniMax speech does not wait for the remaining visible response. At a tool-call response boundary, the streamed block is finalized before the tool event is handled, preventing the same progress speech from being played again through the tool-event path while still preserving voice_tool_call_speech and tool-specific suppression such as wait_for_wakeup. Trailing tags, extra TTS blocks, and missing closing tags remain runtime compatibility fallbacks for older or malformed model output, but they are not valid prompt output. If the model omits the closing tag, the response boundary acts as an implicit </tts> and flushes the remaining speech instead of dropping it.

# Minimax WebSocket
[tts]
provider = "minimax"
model = "speech-2.8-hd"
voice_id = "male-qn-qingse"
emotion = "happy"
speed = 1.0

# Fish Audio WebSocket
[tts]
provider = "fish-audio"
model = "s2-pro"
reference_id = "98655a12fa944e26b274c535e5e03842"
speed = 1.0

# Alicloud Qwen-TTS Realtime
[tts]
provider = "alicloud"
model = "qwen3-tts-flash-realtime"
voice_id = "Cherry"
speed = 1.0

# Volcengine WebSocket Bidirectional Streaming V3
[tts]
provider = "volcengine"
model = "seed-tts-2.0"
voice_id = "zh_female_vv_uranus_bigtts"
speed = 1.0

Dependencies

• audio_service must be running for board-side recording/playback;
• TTS adapter outputs PCM and writes to the configured playback backend. audio.playback_backend = "auto" uses audio_service on board and the local OS player in desktop/PC Agent mode through ADB input backend or environment bridge;
• rknn_vad / cpu_vad helper must be executable; when vad_backend="rknn", vad_model_path points to a converted encoder RKNN; when vad_backend="cpu", helper defaults to /oem/usr/bin/cpu_vad;
• STT/TTS require external API keys;
• wakeup mode requires GPIO 33 or GPIO 32 hardware trigger condition.

VAD helper can be verified directly on the board:

/oem/usr/bin/rknn_vad --model /oem/usr/model/silero_vad_6_2_encoder_rv1106_w8a8_v1.rknn --weights /oem/usr/model/silero_vad_6_2_lstm_decoder_weights.bin --self-test
/oem/usr/bin/cpu_vad --weights /oem/usr/model/silero_vad_6_2_lstm_decoder_weights.bin --self-test

On success, it will output a line P <probability>; if there are still RKNN input configuration issues, it will directly output ERR ....

Known Limitations

• Web UI and device voice interactions share one Agent runtime, so only one Agent run can execute at a time;