AI Tinkerers Agents Everywhere

I built AgentTalkie at Agents, Everywhere, the AI Tinkerers global hackathon, in San Francisco on September 12, 2026.

The problem I chose

I run Grokbot, OpenClaw, Hermes, Claude Code, and Codex, and the list keeps growing. They span cloud and local machines, each with its own session and memory. There is no central place to coordinate the fleet or unblock the work that needs my decision.

Agents work well by themselves. The fleet is not coordinated. I wanted one voice surface where work could move between specialized tools without hiding execution from the operator.

What shipped at the event

The build started from the MIT-licensed CopilotKit Agents Everywhere starter. On top of that baseline, I built the voice coordination flow, direct-tool approval path, durable job runner, artifact lineage, coding-agent handoff, and AgentTalkie interface.

The final recorded run moves through Ambiguous AI, Exa, Codex, and Claude Code in one conversation. It creates a real Ambiguous document and reads it back, returns source-backed research with timing and cost, and hands the exact artifact from Codex to Claude Code for revision. Every step leaves a tool call, result, and receipt.

Toolset

• GPT-Live and WebRTC: the voice layer. Client delegation lets voice hand work to other agents instead of answering from its own context.
• CopilotKit and AG-UI: the Agents Everywhere starter, plus typed events for every action, result and receipt.
• Ambiguous AI: one MCP across 17 agent-first apps, used for documents.
• Exa: source-backed research inside the same thread.
• Ori, Codex and Claude Code: local coding agents with native output and exact-artifact handoffs.
• Next.js, React and Postgres: the operator workspace, durable jobs and receipts.
• Docker: bounded runners for coding jobs.

What was hard

• GPT-Live isn’t Realtime. New endpoint, new model, WebRTC and client delegation. My existing Realtime code didn’t port over.
• Silence reads as failure. A 40-second deep-reasoning search inside a voice turn felt hung. Long work had to leave the audio loop and become a durable job with visible progress.
• Local runners inherit too much. Agent CLIs pick up installed auth, MCP servers and skills. Isolation and routing had to be explicit before a job started.
• “Done” needs proof. Every write had to be validated against the live schema first and read back after.

What I learned

• Typed events beat parsed logs. CopilotKit and AG-UI turn the action trail into data.
• Native tool output builds trust. A generic “done” message doesn’t.
• When one agent hands work to another, exact artifact identity matters more than a polished summary.
• The operator needs one place for the conversation, the progress, the results and the next decision.

What I would take into the next build

Start with one unmistakable cross-tool handoff, then make every state around it honest. Add integrations only when each one changes the workflow, not to grow the logo row. Keep the last good artifact visible, make uncertain writes explicit, and rehearse the complete path with the same provider access used on stage.

The product architecture and current limits are documented in the AgentTalkie project case study. The longer story is in Building AgentTalkie.