Reachy Mini: Milo's Physical Avatar

A Reachy Mini Wireless (Pollen Robotics / Hugging Face, $499 DIY kit) is the body. Milo is the mind. Same identity as desktop Hermes Milo — not a separate chatbot bolted onto a cute shell.

Architecture

Stock path keeps the body. Custom path is everything that makes it Milo.

Layer	Stock (HF / Pollen)	Custom (J&M)
Body / IO	Daemon, motors, media, conversation app	Headless startup and recovery, native gaze/motion manager, volume lock, and app-owned identity-frame export to the M4 vision service
Brain	Optional hosted realtime rollback	Local far-field STT → realtime language response → local M3 custom-voice speech
Memory	—	Layered recall + hard relationship facts + durable schema-v2 writeback outbox
Eyes	Body camera and media stack	Native gaze and expression share one movement manager. The same app media stream exports people/dog identity frames without a second LOCAL camera client; the path now has real cold-boot recovery proof.
Knowledge	—	Phrase-gated read-only research with spoken filler; no side-effect tools

What we copied from Reachy Claw

We reviewed suharvest/reachy-claw and copied three things: a single motor writer, intent instead of a second neck owner, and read-only observability first. That is the conversation app’s one movement manager composing gaze and expression. We did not switch stacks. Milo’s brain, memory, recognition, and authority stayed ours. We did not copy their dashboard.

What's live

Capability	State
Headless appliance	Live. Conversation app, local services, motor wake, health and recovery are automated.
Voice loop	Live. Far-field VAD; local large-v3-turbo STT on Metal; realtime language response; local M3 custom-voice speech (cloud speech remains rollback).
Memory	Hardened. Layered recall plus serialized schema-v2 durable writeback; ambiguous delivery stops rather than replaying.
Voice authority	Hardened. Literal speech control is loopback-only, authenticated, idle-gated, and fenced against cancellation races.
Read-only research	Live. Only the explicit “ask Big Milo” phrase opens the research path; spoken filler prevents dead air.
Vision and gaze	Recognition milestone. Attended August 17: Cindy, then Bennie on the floor, then the person he was facing. People/dog frames stay app-owned. Native tracking and expression still share one motion owner.
Everyday expression	Live. Allowlisted play_emotion only. The separate dance() tool is now allowed when someone asks; it is not an attended proof yet. A tool-only turn still forces a short spoken fallback.
Proactive greeting	E3 passed. Best-vs-runner-up identity margin, absence/cooldown gates, one clear named greeting, one welcome motion, and no duplicate dispatch in the controlled arrival.
Conversation soak	E5 passed. Fresh attended run: 20 completed exchanges, smooth native tracking, clean audio, no competing neck writes, and no autonomous expressive movement. Interrupted transcript revisions were not counted. A later journal audit found one completed exchange with a transient emotion tool-result history mismatch; the integrated protocol correction is now deployed and verified.
Sleep and wake	Implemented. Nap lowers the body without killing the conversation app; wake restores it.
Act-from-voice	Off by design. No email, deployment, purchases, or privileged action path.

Development status at a glance

The protocol bug that passed the first test

One E5 exchange sounded and moved correctly but still left a transient protocol error. Milo’s server-side expression helper had emitted play_emotion directly to the robot without first recording the matching function call in conversation history. The robot returned its result immediately, but the server had no pending call to pair it with.

The first repair looked right and passed helper-level tests: record the synthetic call before yielding it. A delayed integrated review found the miss. In the real stream, a terminal EndOfResponse object had replaced the last runtime-bearing response chunk, so the helper silently lacked the conversation context it needed. Worse, the synthetic tool appeared after response termination.

The durable correction holds the terminal marker, retains the last real response chunk, records and emits the injected call, and only then ends the response. The regression now exercises the public processing flow and proves tool-before-terminal ordering, pending history before client delivery, immediate output pairing, one terminal after multi-pass lookup, and no default-history mutation for out-of-band work. The full suite passed 726 tests; a separate reviewer reproduced normal affirmation, plain, lookup, and out-of-band flows before deployment.

Lesson: ordering guarantees belong in integration tests at the consumer boundary. A helper test can prove its own contract while missing the object lifecycle that starves it of context.

How the camera stack actually works

One lesson took a while: the camera needs a single owner per live path. Two LOCAL media clients on the Pi can fight the sensor and leave one with empty frames. The failed design had a separate broker open a second LOCAL media client. At boot it initialized before the daemon camera producer, degraded permanently to no camera, and returned HTTP 503. The repaired path keeps the conversation app as the single client: it atomically exports a JPEG from its existing live stream, and the uploader sends that frame to M4 vision. A real power cycle now proves the exported frame and proactive-identity path recover cleanly. The old broker, grabber, and external tracker remain disabled.

Piece	Job
App frame export	The conversation app atomically mirrors a JPEG from its existing media stream every 2 seconds. No second camera client.
frame_uploader	Sends the app-exported JPEG to M4 vision for people/dog identity. The old broker is disabled rollback source.
Native gaze path	The daemon tracker and conversation-app movement manager own the neck together. The external milo-head-tracker remains installed only as a disabled rollback path.
M4 vision_service	Names people and dogs. Does not drive the neck anymore
Camera tool	Reads the conversation app’s own fresh media frame first; M4 latest remains the final fallback.

Early look-at was a LAN hack: M4 bbox every 2s → short head pulse → snap center when the face dropped. It felt like a tick. Moving tracking beside the camera helped, but the final jerk remained because two systems still owned the same neck. In one controlled interval, 11 external moves overlapped with 115 rejected app targets. The durable answer was one owner: the external tracker is disabled and the native movement manager composes gaze with expression.

STT: the mic was fine

Far-field VAD needed real tuning (silence thresholds, no live-transcript race). But the “Hey mama / Big Mile / Chirri” class of failures was not the array.

Before	After
faster-whisper tiny.en, greedy	mlx-audio whisper-large-v3-turbo on M4 Metal
No vocabulary bias	Private household and lab vocabulary prompt
Speculative reopen 3s	1.5s (fewer doubled sentences)

Same room, same distance sometimes transcribed clean long sentences under tiny.en — and sometimes invented fluent wrong English. That is a small model + greedy decode, not a dead mic.

First cutover of large-v3-turbo briefly made him “deaf”: a s2s rename_args bug turned gen_kwargs into nested kwargs={} and every STT call crashed. Fixed the same night. Mic and VAD never stopped hearing — the decoder was throwing.

Reliability

Reviewers (Opus / Sonnet / Opus 5): good homebrew; make power-on boring before more toys. That sprint landed.

Item	Status
Cold boot and watchdog	Passed / live. Robot-off health is soft so an unplugged body does not cause a restart loop.
Realtime protocol and cancellation	Hardened. Stale output and post-cancel token races fail closed; server-injected tools are recorded before client delivery and before the terminal response marker.
Honcho writeback	Hardened. Private SQLite outbox, shared lock, strict permissions, retention, reconciliation hold, and content-redacted health.
Local operator status	Current. Human-readable deterministic text is available while JSON remains the automation default.
Provider fallback	Not yet proven end to end. This is the largest remaining appliance-grade reliability gap.
Current integrated verification	726 tests passed, plus focused protocol checks and an independent reproduction of affirmation, plain, lookup second-pass, and out-of-band flows.

Knowledge contract

Tier	Allowed	Not allowed
0 Body	Talk, motion tools, camera describe, standing facts	—
1 Read-only	Lookup / Hermes web on ask big Milo	Side effects
2 Act	—	Not wired from voice (email, deploy, purchases…)

Deep research is still first-person Milo thinking harder — not “I asked another agent.”

Build notes worth keeping

• Stock app kept. It speaks OpenAI Realtime dialect; HF speech-to-speech on the M4 is the local backend. No forked robot UI.
• Camera was a ribbon first, then ownership. After the ribbon reseat, the hard problem was multi-consumer IPC, not pixels.
• Look-at tick = control-loop rate. 0.5 Hz sense over LAN cannot drive smooth neck motion. We copied Reachy Claw’s single-writer motion pattern: one compositor, not another tracker.
• VAD first, then STT model size. Far-field needed silence work; name errors needed a real Whisper, not a toy.
• Memory is layered. Small, curated relationship facts applied last beat a large, noisy memory dump. Household facts had to be boringly reliable.
• Degrade toward less authority and worse voice — never silence or guessed identity.
• Do not bounce the conversation app to recover speech. An app restart still parks the body. Clear nap/suppress and wake instead.

Assembly (short)

One evening + morning, ~50 steps, complete kit. Stewart-platform neck is the party trick.

Next

• Keep: E3/E5, app-owned camera, single-owner motion, boot-only volume, local M3 speech, and integrated tool-history pairing
• Next feel: attended dance-when-asked, and the silent second-pass replies that logged zero audio after a long first-byte wait
• Keep the 320 ms VAD floor unless new evidence justifies a change
• Parked by choice: production fallback matrix, guard mode, code red, full duplex, and act-from-voice