💪 I had an idea: a voice assistant that talks like a real person, one you can interrupt mid-sentence, ask about anything and give tasks to in real time. Works on phone, tablet and computer. Zero load on the system. Like in "Iron Man". A week ago I realized the market already gives you all the tools you need - and built it in 24 hours
What changed in the stack - OpenAI Realtime API:
A year ago this would have been a pipeline of four separate services: voice detector, STT transcription, LLM with lag, TTS voiceover. Every link added 200-400ms - 1 to 2 seconds per reply in total. It felt like pressing a button, not having a conversation. Now OpenAI Realtime API is an audio-native model: it takes the raw audio stream directly and outputs audio with no transcription in between. Latency around 300ms - human reaction level. The most important feature: VAD (Voice Activity Detection). The assistant detects that you've started talking - and instantly cuts off its own answer. That's exactly what makes it feel like a live dialogue, not waiting in line for a reply. All the compute is on the server, the client is in the browser - zero load on the system at all
Architecture:
⚡️ Wake word - Porcupine/Picovoice listens locally with no load, activates only on the keyword
⚡️ Realtime API - two-way audio + tool use right in the conversation: search, browser, system commands, API
⚡️ WebSocket/WebRTC - all the heavy compute on the server, the client - a browser on any device
⚡️ Context - remembers the conversation topic and tasks between sessions
Cross-platform came out natively: one WebRTC client in the browser works the same on iPhone, Android, iPad and Mac. Not three different apps - one interface everywhere
What Jarvis can do right now:
💎 Interruption with no pause - VAD stops the answer instantly the moment you open your mouth
💎 Live requests - search, data, calculations right inside the conversation via tool use
💎 Voice commands - launch, open, send without leaving the dialogue
💎 Minimalism - one screen, no apps to install, works anywhere there's a browser
The hard part wasn't the code - Realtime API spins up in a few hours from the official Python and Node.js examples. The real work is UX: the right VAD threshold so it doesn't react to background noise, echo cancellation when you talk through the speaker, context management in long conversations. That's where the time goes when you build something actually convenient, not just working
The tooling market for a stack like this took shape literally over the last year: ElevenLabs Conversational AI gives a similar architecture with latency around 500ms on any LLM backend, Google Gemini Flash Live - realtime voice in 90+ languages. This is already a mature stack, not an experiment
⭐️ Sam Altman, CEO OpenAI ("The Intelligence Age", 2024):
"Everyone will have access to a smart friend with the knowledge of a doctor, a lawyer, a financial advisor - and an expert in any field you need"
💭 I'd been putting this idea off for half a year - it seemed to need complex infrastructure, a month of work, a separate server. While I was putting it off, the industry completely rebuilt the stack under my feet. The challenge turned out not to be how to build it - the challenge is how to tune the UX to your own scenario. I'll show a demo soon and walk you through all the architecture details. Do you already have your own voice assistant - or hasn't this topic come up yet? 👀