MatIntellect

My attempt to rethink the voice assistant led me to build a multi-functional tool that's controlled by voice - meaning it's not just Siri or Alice, where you ask something simple or have it do some basic function

It's a full-on conductor of your system that can manage other agents, take real actions, plan, be your one-hundred-percent copilot - connected to the shared memory of your entire system, knowing everything that's going on in it, and ready at any moment, anywhere, from any device to pick up work on the projects you're busy with right now 🎙️

In my case the voice assistant is connected directly to Hermes - an agent running on a ChatGPT subscription

But on top of that, the voice assistant has vision. I can show it something on camera or send it an image. I can do the same from my phone 👁️

And a nice bonus - a live face with live facial expressions that makes it feel like you're working with a real person

What do you think?

Watch the video to see what came out - waiting for your comments and questions 🤩

How it works inside

The assistant is built in two layers. The first layer is a fast model that carries the live voice conversation: it answers without pauses, you can interrupt it, and it sees the camera image. The second layer is an agent with tools, in my case Hermes. When I ask it to do something rather than just answer, the task goes to that agent: it gives tasks to other bots, searches, calculates and brings the result back into the conversation

Thanks to this split, the voice doesn't slow down because of heavy work, and the heavy work doesn't suffer from the rush of conversation. The fast model handles talking, the agent handles doing

The voice shares memory with Telegram. Whatever I say out loud shows up in the chat, and whatever we discussed in text in the morning, the voice knows in the evening. So it doesn't matter which device or method I connect with: the work picks up where it left off

Where it already works for a client

I built the same voice entrance for a clinic owner. He had several AI bots, each with its own chat, and every request meant hunting for the right one. Now he has one voice entrance to all of them: the agent runs on the clinic's server, the owner connected his own ChatGPT subscription himself, and the agent doesn't answer strangers. On top of an existing system a voice agent like this takes about 2 weeks to build

What matters when building this for a business

First, boundaries. Giving a command by voice is easy, so anything risky has to wait for confirmation. My personal agent Jes has a simple rule: deleting data, spending money, messaging clients and publishing happen only after my yes. Jes has been running for 134 days, handles 10 scheduled jobs and hands tasks to 7 other bots, and this rule has never got in the way of the work

Second, voice belongs where typing is inconvenient. It's ideal for short tasks on the go, while complex decisions about numbers and drafts are easier to make with your eyes, so the agent also sends answers with data as text

Why a voice needs vision and a face

Vision removes a whole class of explanations. Instead of describing an error on screen, a receipt or a document in words, I just show it to the camera or send a photo, and the assistant immediately understands what I mean. On a phone this is especially handy: you see something on the go, point the camera, ask and get an answer right away

The face isn't decoration. When the one you talk to looks at you and reacts with expressions, the conversation feels more natural, and you want to use the assistant more often. For a business owner who doesn't like texting bots, this often decides whether the system lives or gets abandoned a week later

For a version built for your team, see the voice AI agent service

Share:

No comments yet

Leave a Comment

Fields marked with an asterisk (*) are required