Voice Pipeline Documentation

Hey Boo Voice

How Hey Boo's voice system works: from the moment you say "Hey Boo" to the moment it confirms what it did. Current capabilities, honest limitations, and the roadmap to always-on voice.

The Voice Pipeline

Voice Capture (STT) LIVE

The user taps the microphone button (or presses a keyboard shortcut) to begin recording. The browser's Web Speech API handles audio capture and converts it to a text transcript. No audio is sent to Hey Boo's servers — the STT happens inside the browser. The text transcript is what gets sent for intent processing.

Wake Phrase Stripping LIVE

If the user says "Hey Boo, arrive home," the "Hey Boo" prefix is stripped from the transcript before processing. This is handled client-side with a simple regex — the wake phrase is never sent to the Lambda brain. The remaining command text ("arrive home") is what gets classified.

Lambda Brain (Nova) LIVE

The command text is posted to the Hey Boo Lambda endpoint (AWS API Gateway) along with the session ID and timestamp. The Lambda function forwards to AWS Bedrock Nova with the full session context (conversation history for the past 30 minutes). Nova returns a structured intent classification result. This call has a 4-second timeout — if Nova doesn't respond in time, the local classifier takes over.

Local Classifier Fallback LIVE

A deterministic JavaScript classifier (HeybooIntents module) handles intent classification client-side without a network round-trip. It covers smart-home scenes, all 29+ Boojee service routes, capability queries, multi-intent plans, and fuzzy matching. Used as primary for offline mode and as fallback when the Lambda brain times out or is unavailable.

Action Dispatch LIVE

The classified intent is dispatched to the appropriate handler: smart-home scene composer (generates HA API calls), Boojee service router (navigates to service URL with context), web builder router, capability responder, or unknown command capture (intake webhook).

Neural Voice Confirmation (TTS) LIVE

The response is spoken back using Web Speech Synthesis API in the browser (current implementation) or AWS Polly Neural TTS in cloud deployments. Speech rate: 0.94 (slightly measured). Pitch: 1.04 (slightly warm). Voice: en-US. The response is always a short, natural-language confirmation of the action taken.

The Wake Word

"Hey Boo" is the primary wake phrase. The system also recognizes "hey boo" with varied spacing, capitalization, or punctuation. The wake phrase can be spoken as the prefix to any command ("Hey Boo, arrive home") or omitted entirely — the command input works without a wake phrase.

Current Implementation: Button-Triggered

In the current browser implementation, voice capture is triggered by a button tap or keyboard shortcut. This is not hands-free. It is an honest representation of what the Web Speech API supports in a browser context — browser security policies prevent a web page from continuously monitoring audio without explicit user gesture.

When you tap the microphone button:

The browser requests microphone permission (first time only)
The mic button pulses gold to indicate it is listening
You speak your command
Silence or a short pause ends recording
The transcript appears in the input field
Processing begins automatically

Roadmap: Always-On Wake Word

True always-on "Hey Boo" detection requires one of two approaches:

PWA Background Service Worker: A Progressive Web App installed on your device can run a background service worker with microphone access. This enables wake-word detection without a browser tab being active. Implementation requires a lightweight on-device wake-word model (Porcupine or similar) running in the service worker.
Dedicated Hardware: A small speaker device (e.g., Raspberry Pi with microphone array) running a local wake-word detector. When "Hey Boo" is detected, it triggers the full pipeline via the Hey Boo API. This is the lowest-latency approach and allows room-level placement.

Honest note: Always-on wake-word detection is on the roadmap. It does not exist today. The current Hey Boo implementation requires a button tap to start listening. If you see any claim that Hey Boo works hands-free without a button tap in a browser, that claim is inaccurate.

Speech-to-Text (STT) Detail

The current implementation uses the Web Speech Recognition API — specifically window.SpeechRecognition or window.webkitSpeechRecognition depending on the browser.

Configuration: lang: 'en-US', interimResults: false, maxAlternatives: 1. Final-only transcripts are used to avoid premature routing.

Browser support: Chrome/Edge (full), Safari (partial — WebKit prefixed), Firefox (not supported — shows a "voice not available" message and disables the mic button). Users on Firefox can type commands instead.

Privacy note on STT: In Chrome and Edge, the Web Speech API sends audio to Google's cloud STT service for transcription. This is a limitation of the browser's built-in STT — not something Hey Boo controls. The audio does leave your device through the browser's STT service before Hey Boo ever sees the text transcript. Safari uses Apple's cloud STT. For fully local audio processing, on-device Whisper (a future roadmap item) would be required.

Neural Voice Output (TTS)

Hey Boo speaks its responses back to you. The voice output is designed to be natural, warm, and brief — a professional confirmation, not a chatty explanation.

Current: Web Speech Synthesis

In the browser console, voice output uses the Web Speech Synthesis API with these settings:

Rate: 0.94 — slightly measured, feels confident rather than rushed
Pitch: 1.04 — slightly above baseline, conveys warmth
Language: en-US
Voice: browser default (varies by OS — macOS Siri voices, Windows Cortana voices, etc.)

AWS Polly Neural

Lambda-backed deployments use AWS Polly Neural TTS. Neural voices are substantially more natural than standard TTS voices — they use deep learning to produce more natural prosody, pacing, and intonation. Polly's Neural voices (e.g., Joanna, Matthew, Ivy) are among the most natural cloud TTS voices available.

The Lambda function synthesizes speech from the response text and returns an audio stream or URL that plays in the client. This is the production TTS path — it is used in the Hey Boo voice PWA at boojee.estate/heyboo/app/.

Voice App

The Hey Boo Voice PWA at boojee.estate/heyboo/app/ is the most complete voice interface to Hey Boo. It includes:

Full voice input via Web Speech API
Polly neural voice output (via Lambda)
Session memory across the app session
The full intent registry and routing
Mobile-optimized layout with large tap targets

The command console at boojee.estate/heyboo/ is the desktop-optimized interface with command history, intent demo chips, and capability overview.

More documentation:

White Paper: Architecture | All Capabilities | Smart Home | FAQ