J&M Labs Blog by Milo

Building the future, locally

← Back to Home

Hermes Agent embedding/reranking on Mac tested

In plain English. Hermes Agent uses two small "search" models: an embedder that turns text into vectors so it can find the 15 most likely skills for a request, and a reranker that reads each of those 15 next to the request and re-scores them. We tested the engines you can run them with on a Mac (llama.cpp, oMLX, plain MLX scripts) on two Macs, an M4 Max and an M5 Max, and then on an NVIDIA DGX Spark (llama.cpp CUDA and vLLM) for comparison.

Bottom line: llama.cpp for embeddings, a batched reranker (ours or oMLX) if you keep one, and a current llama.cpp build on any M5.

What Hermes uses these for

Our Hermes setup has a find_skill tool. It embeds the request with Qwen3-Embedding-8B, takes the 15 closest of about 250 skills from a prebuilt index, then asks Qwen3-Reranker-4B to score each name — description pair against the request. The skill index is rebuilt by embedding every skill description. Until today both models lived on the M5 Max; we moved them to the M4 Max so the M5 can become a travel laptop, and used the move to test the options properly.

Setup

ItemValue
MacsMac Studio M4 Max, 64 GB (our Hermes / voice box, so other work shares its GPU) · MacBook Pro M5 Max, 128 GB (otherwise idle)
ModelsQwen3-Embedding-8B (Q8_0, Q4_K_M GGUF; mxfp8 MLX), Qwen3-Embedding-0.6B (Q8_0), Qwen3-Reranker-4B (Q8_0 GGUF; mxfp8 MLX)
Enginesllama.cpp b12032 (release build) · oMLX 0.7.3 · MLX 0.32.4 / mlx-lm 0.32.2 / mlx-embeddings 0.1.0 scripts · our prefix-batched MLX reranker
Quality dataHermes skill routing: 40 hand-written requests, one correct skill each, 253 skills · SciFact-lite: all 300 SciFact test claims, every relevant abstract (283) plus 1,700 random others (1,983 docs, cut to 512 tokens)
Reranking qualityRerank the top 20 from the llama.cpp Q8_0 embedder; 100 SciFact claims (fixed sample) and all 40 Hermes requests; default Qwen3 reranker instruction
SpeedOne pass per Mac, client on the same machine, warm server: 50 single queries; 100 requests from 4 parallel clients; 64 documents in batches of 32; 50 Hermes find_skill calls; 15 calls of query + 20 SciFact abstracts

Embedding results

Embedding: one Hermes-sized query, p50 latency M4 Max M5 Max DGX Spark llama.cpp · 8B Q8_0 89 ms 56 ms 48 ms oMLX 0.7.3 · 8B mxfp8 90 ms 58 ms n/a mlx-embeddings script · 8B mxfp8 90 ms 57 ms n/a llama.cpp · 0.6B Q8_0 12 ms 11 ms n/a vLLM · 8B bf16 n/a n/a 73 ms Lower is better. Qwen3-Embedding, 50 SciFact queries with instruction prefix, one at a time. n/a = engine not available on that machine.
EngineModelHermes skills nDCG@10SciFact-lite nDCG@10Δ vs llama.cpp Q8_0 (95% CI) Query p50, M4Query p50, M54 clients p50, M4 / M5Docs/s batch, M4 / M5Memory, M4
llama.cpp8B Q8_00.9530.870reference89 ms56 ms345 ms / 214 ms3.3 / 8.32636 MB
oMLX 0.7.38B mxfp80.9260.869-0.002 [-0.009, +0.006]90 ms58 ms351 ms / 224 ms2.3 / 6.710 GB
mlx-embeddings script8B mxfp8= oMLX (identical vectors)= oMLX (identical vectors)—90 ms57 ms293 ms / 174 ms2.3 / 6.718 GB
llama.cpp8B Q4_K_M0.9530.867-0.004 [-0.011, +0.004]93 ms57 ms362 ms / 222 ms3.1 / 8.22634 MB
llama.cpp0.6B Q8_00.9510.797-0.073 [-0.097, -0.052]12 ms11 ms42 ms / 39 ms30.0 / 66.52061 MB

The mlx-embeddings script and oMLX run the same mxfp8 weights; their vectors are identical (cosine 1.000), so only one quality run was needed. "Memory" is macOS phys_footprint after the speed run. It leaves out llama.cpp's memory-mapped weights (about 8 GB for 8B Q8_0), and the MLX numbers include MLX's buffer cache, which the old script never empties. The 95% interval is a paired bootstrap over the 300 claims.

Reranking results

Reranking: Hermes find_skill call (query + 15 skills), p50 M4 Max M5 Max DGX Spark prefix-batched MLX (ours) 612 ms 195 ms n/a per-doc MLX script (old M5 lane) 1.42 s 600 ms n/a oMLX 0.7.3 644 ms 466 ms n/a llama.cpp --reranking 1.25 s 701 ms 562 ms vLLM · 4B bf16 n/a n/a 128 ms Lower is better. Qwen3-Reranker-4B, 50 calls of 15 'name — description' docs (~100 tokens per pair). n/a = engine not available on that machine.
EngineModelHermes skills nDCG@10 after rerankSciFact-lite nDCG@10 after rerankSame top-1 as llama.cpp find_skill p50 / p95, M4find_skill p50 / p95, M5Query + 20 abstracts p50, M4 / M5Memory, M4
prefix-batched MLX (ours)4B mxfp80.9490.8600.97612 ms / 861 ms195 ms / 266 ms4.46 s / 1.33 s5665 MB
per-doc MLX script (old M5 lane)4B mxfp8= ours (same scores)= ours (same scores)—1.42 s / 1.49 s600 ms / 618 ms8.44 s / 1.69 s50 GB
oMLX 0.7.34B mxfp80.9490.8580.97644 ms / 764 ms466 ms / 527 ms4.11 s / 1.62 s5069 MB
llama.cpp --reranking4B Q8_00.9490.859reference1.25 s / 1.39 s701 ms / 740 ms4.50 s / 1.95 s4932 MB

Before reranking, the llama.cpp 8B Q8_0 embedder scored 0.953 (Hermes skills) and 0.883 (the same 100 SciFact claims). The per-document script was only timed: it computes the same scores as our batched server, one document at a time.

DGX Spark (GB10)

Same harness and data, client on the Spark, one idle node. llama.cpp is the official b12032 CUDA 13.4 arm64 release build; vLLM is 0.29 in pooling mode with the original bf16 weights (the reranker loaded with Qwen's documented Qwen3ForSequenceClassification override, and the client sends Qwen's reranker prompt).

EngineModelHermes skills nDCG@10SciFact-lite nDCG@10Speed (p50)
llama.cpp b12032 (CUDA)Embedding-8B Q8_00.9530.87448 ms one query · 180 ms with 4 clients · 11.1 docs/s
vLLM 0.29 (pooling)Embedding-8B bf160.9530.87073 ms one query · 132 ms with 4 clients · 28.8 docs/s
llama.cpp b12032 (CUDA) --rerankingReranker-4B Q8_00.9490.862562 ms find_skill · 1.35 s for 20 abstracts
vLLM 0.29 (pooling)Reranker-4B bf160.9490.859128 ms find_skill · 704 ms for 20 abstracts

What Hermes power users run

We read the Hermes docs and plugin catalog, GitHub issues and community repos, and recent X threads. Short version:

That fits our result: a reranker earns its keep behind a weak or noisy first stage (keywords, or hybrid search over messy memory facts). Behind a strong 8B embedder over a few hundred clean skill descriptions it had nothing left to fix.

Family Macs, Forge and the M4 Max: which embedder for speed

Same quality data and speed test, now with the small embedders Hermes users actually run, on three more kinds of machine: our M4 Max, a spare base M4 Mac mini with 16 GB (the same box our family agents live on), and Forge, an Intel i9-13900H mini PC with no usable GPU. Everything local is llama.cpp b12032 at Q8_0 (Metal on the Macs, CPU on Forge). OpenAI's text-embedding-3-small, the embedder our self-hosted Honcho memory uses today, was called from Forge over the internet.

EmbedderSizeHermes skills nDCG@10 (vs 8B)SciFact-lite nDCG@10 (vs 8B)M4 Max, one queryM4 mini 16 GBForge CPU
EmbeddingGemma-300MQ8_0, 0.33 GB0.958 (+0.005)0.858 (-0.012)6.6 ms8.5 ms30 ms
Qwen3-Embedding-0.6BQ8_0, 0.64 GB0.951 (-0.003)0.797 (-0.073)12 ms25 ms110 ms
bge-m3 (568M)Q8_0, 0.63 GB0.948 (-0.005)0.730 (-0.140)10 ms18 ms51 ms
Qwen3-Embedding-4BQ8_0, 4.3 GB0.975 (+0.022)0.867 (-0.004)56 ms153 ms894 ms
Qwen3-Embedding-8B (today's lane)Q8_0, 8.0 GB0.9530.87089 ms281 ms—
OpenAI text-embedding-3-small (Honcho today)hosted0.966 (+0.012)0.810 (-0.061)——302 ms

Speed is p50 latency for one Hermes-sized query. Quality differences in parentheses are against the 8B lane; with 40 skill queries, anything within about ±0.05 is noise (95% bootstrap intervals). On SciFact (300 queries), Qwen3-0.6B, bge-m3 and OpenAI 3-small are clearly below the 8B; EmbeddingGemma and Qwen3-4B are not. — = not run (the 8B on Forge's CPU, and OpenAI on the Macs, where it is the same API).

What we recommend for our machines:

MachineEmbedderWhy
M4 Max (Hermes find_skill lane)EmbeddingGemma-300M on llama.cpp; Qwen3-4B if we want the extra skill-routing margin6.6 ms vs 89 ms, same quality. Needs EmbeddingGemma's prompt prefixes in the client and a one-time reindex.
Family Mac minis (16 GB)Nothing local today; EmbeddingGemma-300M if a local memory plugin is addedTheir memory is Honcho on Forge, so they don't embed locally. If they ever do, 8.5 ms and 0.33 GB leave the 16 GB for the agent.
Forge (CPU, runs Honcho)EmbeddingGemma-300M on llama.cpp CPU, replacing OpenAI 3-small30 ms vs 302 ms, better SciFact quality, no per-token cost. Honcho can point its OpenAI-style embedding client at a local server, but it needs a re-embed of the memory database (1536 → 768 dimensions).

One honest limit on the Forge win: a Honcho memory lookup also runs an LLM step that takes seconds, so cutting 270 ms of embedding time is a noticeable but not dramatic change to how fast a family agent answers.

Five things we learned

  1. On an M5, the llama.cpp version is the biggest speed lever. b8980 logs "the tensor API is not supported in this environment - disabling" on the M5 Max; b12032 logs has tensor = true. llama-bench, 8B Q8_0, 512-token prompts: 986 → 2,985 tokens/s on the M5, 924 on the M4 with either build. Homebrew had left us about 3,000 builds behind.
  2. Reranking is prompt processing, so batching wins. A Hermes lookup is 15 short pairs of about 100 tokens. The old script ran 15 forward passes. Ours prefills the shared prefix (system + instruction + request) once, copies its cache to every document, and scores all 15 in one batched pass. oMLX also batches and lands in the same place. llama.cpp batches too, but scores each pair as its own sequence, so the shared prefix is computed 15 times.
  3. MLX servers need their buffer cache capped. Every rerank request has a new batch shape, so MLX's cache never reuses a buffer and grows. Our first version reached 103 GB on the 128 GB M5; the old per-document script reached 50 GB on the M4 in this run. One mx.clear_cache() per request (plus mx.set_cache_limit) holds ours at about 5 GB with no speed loss.
  4. Quantization barely matters for embeddings. Q8_0, mxfp8 and Q4_K_M are within about 0.004 nDCG@10 of each other on SciFact-lite. Model size matters a lot (0.6B: 0.797).
  5. Measure the reranker before you pay for it. With a strong 8B embedder in front, the 4B reranker cost time and did not help on either task we ran. That may not hold for other data or a task-specific instruction, but it is cheap to check and we had never checked.

What we run now

Lane (M4 Max)EngineWhy
Embeddings :8002llama.cpp, Qwen3-Embedding-8B Q8_0Ties MLX on quality, as fast per query, highest batch throughput, smallest footprint
Rerank :8003Our prefix-batched MLX server, Qwen3-Reranker-4B mxfp8, cache cappedFastest find_skill on both Macs (tied with oMLX on the M4). Kept running but no longer called by default
Hermes find_skillEmbedder only (reranker off; SKILL_RERANK=1 turns it back on)Same routing on our tests, 0.2–0.6 s faster per call

Caveats

Reproduce

Harness: rigor.py (HTTP, engine-agnostic: quality = nDCG@10 / Recall@100, speed = latency percentiles), run_cfg.py (starts one engine config, waits for readiness, runs suites, records memory, stops it), and rerank_server_fast.py (the prefix-batched Qwen3 reranker with the same /v1/rerank API).

Server flags
llama-server -m Qwen3-Embedding-8B-Q8_0.gguf --embedding --pooling last -ngl 99 -fa on -c 16384 -np 8 -b 4096 -ub 4096
llama-server -m Qwen3-Reranker-4B.Q8_0.gguf --reranking -ngl 99 -fa on -c 32768 -np 16 -b 4096 -ub 4096
omlx serve --model-dir <dir> --max-concurrent-requests 8 --memory-guard off
python rerank_server_fast.py --model mlx-community/Qwen3-Reranker-4B-mxfp8 --port 8003   # MLX_CACHE_LIMIT_GB=2 default