MatIntellect

I can't shake the thought of making my system work by voice. There are pros and cons here

Pros - of course, I want to build a system like in the movie "Iron Man", where in the morning I say "Hey, Jarvis", and my system automatically starts up, gets to work and answers me by voice - fast, clear and so on 🤖

And the cons. Can you really orchestrate a system well by voice, when you don't see text, don't see data and only hear a voice answering you through the speakers? More no than yes 🤔

So in the near future I'll build a system that will interact with me by voice, where I'll talk to it like to a real person. But the main product development for my clients I'll still keep in text, with a big context window and lots of details. Otherwise it'll all turn into a total dump and be completely unusable

How this thought turned out

Since then voice has appeared in my system, but exactly where it fits. I've long been writing to my personal agent Jes both in text and with voice notes in Telegram: speech recognition turns a voice note into a command, and the answer comes back as text I can reread. Jes has been running for 134 days, handles 10 scheduled jobs on its own and hands tasks to 7 other bots. Saying "remind me tomorrow at 10:00 to check the requests" by voice is easier than typing, and checking that it understood correctly is easier with your eyes

That's how the rule formed: I speak by voice, I read in text. Speaking is faster than typing, especially on the go. But making decisions about data, numbers and drafts is easier when everything is in front of you

A voice entrance for a clinic

In the end I built a real "Jarvis" not only for myself. The owner of an aesthetic medicine clinic had several AI bots, each with its own chat, and every request meant hunting for the right one. He needed one assistant he could simply talk to

I built a voice agent that became one way into all his bots. It runs on the clinic's server, and the owner connected his own ChatGPT subscription himself. It works in two layers: a fast model carries the live voice conversation, while real tasks go to an agent with tools. So the conversation flows without pauses, and the work is done as carefully as in a text chat. This agent doesn't answer strangers, and its memory is shared with Telegram: whatever is said by voice is visible in the chat too. A voice agent like this is built on top of an existing system in about 2 weeks

Where voice works and where it doesn't

Voice is good for short commands, questions on the go and moments when your hands are busy. It's also good for capturing thoughts: I transcribe long calls on my own computer, and a 57-minute conversation becomes text in about 10 minutes

But complex work with lots of details I still do in text. My worry from this note turned out to be right: if you run everything through voice, you lose control over the details. Voice is a convenient door into the system, not a replacement for the screen

Where to start with voice

You don't need to build a talking robot right away. The first step is almost free: teach your bot to understand voice notes. You dictate a task like you would to a colleague, the bot turns it into text and does it, and sends the answer back as text. That alone saves a lot of time for people who are on their feet all day: doctors, realtors, cafe owners, and anyone running things from a phone between meetings

The second step is a live voice conversation where the agent answers by voice right away. You need it when you already have several bots and finding the right chat has become a hassle, or when the system will be used by someone who finds talking easier than typing. In both cases one rule matters: anything risky the agent does, it shows first and waits for confirmation

And what about you, what are your fantasies? What do you want to build? 💭

Share:

No comments yet

Leave a Comment

Fields marked with an asterisk (*) are required