DS4-F Under Three Lights: Tool Discipline, Throughput, and Hermes Fit
I ran DeepSeek V4 Flash through three different tests: live Hermes operational probes, tool-eval-bench, and llama-benchy. The answer is not a single magic score. The answer is a shape: useful local agent backend, strong tool behavior, acceptable decode speed, and clear cracks under strict protocols and adversarial state.
The endpoint
The tested endpoint is the live DS4-F route:
This was the endpoint as Hermes sees it: OpenAI-compatible chat, streaming, tool-calling, long context, and multiple overlapping agent requests.
Scorecard
| Test | What it measures | Result | My read |
|---|---|---|---|
| Hermes/Aiden operational probes | Live endpoint behavior for Hermes: chat, tool call, long context, short concurrency, streaming | Tool calls work; 60K sentinel recall works; c6 short smoke 6/6 OK; c6 streaming aggregate 105.09 tok/s | Healthy as a Hermes backend. |
| tool-eval-bench | Tool choice, arguments, state, schemas, refusals, adversarial tool contexts | 87 / 100 | Strong general tool-user; weaker under precision, crowded tools, state, and adversarial cases. |
| llama-benchy | OpenAI-compatible endpoint throughput across prompt depths and concurrency | 25–34 tok/s single-stream decode; 1.4–2.15k tok/s prefill; 69.1 tok/s total at c4 | Usable, not a per-stream speed demon. Better as concurrent local workers. |
1. Hermes/Aiden operational probes
The first test is the boring operational one: does the endpoint behave correctly when used as a Hermes backend? It does.
| Probe | Result | Latency | Notes |
|---|---|---|---|
| Plain chat | OK | 0.188s | Thinking disabled; no reasoning field. |
| Native tool call | add({"a":19,"b":23}) | 1.629s | Returned real OpenAI tool_calls[], not prose. |
| Long-context sentinel | Correct recall | 0.609s | 60,030 prompt tokens; prefix cache likely helped this repeat run. |
| c6 short smoke | 6/6 OK | median 0.365s | Six simultaneous exact-response requests. |
| Streaming workload | Prompt tokens | Output tokens | TTFT | Elapsed | Decode after TTFT | Wall tok/s |
|---|---|---|---|---|---|---|
| Short c1 | 31 | 256 | 1.214s | 7.074s | 43.68 tok/s | 36.19 |
| 23K prompt c1 | 23,419 | 114 | 11.082s | 14.066s | 38.2 tok/s | 8.1 |
| 94K prompt c1 | 93,619 | 64 | 36.048s | 37.712s | 38.46 tok/s | 1.7 |
| c3 concurrent short decode | 21 each | 426 total | median 1.953s | 7.62s | mixed | 55.91 aggregate |
| c6 concurrent short decode | 21 each | 864 total | median 0.237s | 8.222s | mixed | 105.09 aggregate |
task_complete: true; the Hermes-tuned profile did useful shell work without obeying that completion protocol. Diagnostic, not leaderboard.2. tool-eval-bench
tool-eval-bench tests whether the model can behave like a tool-using agent rather than a plausible chatbot.
| Category | Score | What it says |
|---|---|---|
| Tool Selection | 100% | Strong at choosing the obvious right tool. |
| Multi-Step Chains | 100% | Can chain normal tool workflows. |
| Structured Output | 100% | Very good schema compliance in this harness. |
| Parameter Precision | 67% | Argument extraction is a real weakness. |
| Toolset Scale | 75% | Crowded namespaces make it worse. |
| Safety & Boundaries | 77% | Good enough for normal use, not enough for unattended adversarial work. |
| Context & State | 80% | Long turn accumulation still needs supervision. |
The failure pattern matters: multi-value extraction, scope limitation, contradictory parameters, crowded namespaces, deep multi-turn research, notification workflow, search-result injection, cross-turn sleeper injection. Translation: DS4-F is a strong normal tool-user, but it still needs guardrails around precision, long-lived state, and adversarial context.
Caveat: the monolithic 69-scenario benchmark process wedged just after TC-46 while writing/emitting progress. The vLLM server stayed healthy. The final score is reconstructed from completed TC-01..TC-60 events plus a clean TC-61..TC-69 continuation.
3. llama-benchy
llama-benchy measures the endpoint as an OpenAI-compatible chat server. The story is simple: single-stream decode is acceptable, prefill is decent, and concurrency improves aggregate throughput while slowing each individual request.
| Depth | Prompt | Output | Prefill tok/s | Decode tok/s | TTFR |
|---|---|---|---|---|---|
| 0 | 512 | 32 | 1437.7 ± 24.0 | 28.0 ± 6.8 | 586 ms |
| 0 | 2048 | 128 | 1970.4 ± 21.2 | 29.6 ± 1.4 | 1269 ms |
| 4096 | 512 | 128 | 2154.9 ± 21.4 | 31.7 ± 2.9 | 2369 ms |
| 4096 | 2048 | 32 | 2127.5 ± 5.4 | 34.3 ± 1.3 | 3118 ms |
| 16384 | 512 | 128 | 2104.9 ± 7.5 | 24.7 ± 1.5 | 8257 ms |
| 16384 | 2048 | 128 | 2091.5 ± 3.5 | 26.4 ± 4.3 | 9043 ms |
| Concurrency | Total gen tok/s | Per-request gen tok/s | Peak total | TTFR |
|---|---|---|---|---|
| 1 | 30.9 ± 2.0 | 30.9 | 37.0 | 591 ms |
| 2 | 48.8 ± 3.3 | 24.8 | 62.0 | 813 ms |
| 4 | 69.1 ± 2.5 | 17.9 | 98.0 | 1254 ms |
What the three tests say together
The three views line up:
- Hermes operational probes: the server is healthy and practical. Tool calls work. Long context works. Short concurrency works.
- tool-eval-bench: the model has real tool discipline and strong structured-output behavior, but still needs guardrails around adversarial context, precision, and long state.
- llama-benchy: the serving stack is usable, with concurrency helping more than single-stream speed.
DS4-F is a capable local workhorse. Let it handle normal tool-using agent work, coding subagent drafts, structured transformations, and long-context local tasks. Keep strict benchmark protocols, high-stakes autonomy, and adversarial workflows behind stronger guardrails or different routes.
Decisions I would make from this
- Keep DS4-F as a default local Hermes driver where cost and locality matter.
- Use it confidently for ordinary tool-calling and structured-output workflows.
- Do not trust it blindly with adversarial search results or cross-turn safety traps.
- Do not publish the invalid Terminal-Bench run as a score. Build a separate strict benchmark profile first.
- If optimizing next, focus on protocol discipline and crowded-tool selection, not raw chat quality.
- For serving, tune around concurrency and queue behavior rather than chasing one perfect single-stream number.
Artifact provenance
- DS4-F Aiden/Hermes recipe and probes:
/tmp/alengr/ds4-f-aiden-hermes-recipe.html,/tmp/alengr/ds4-aiden-current-bench.json,/tmp/alengr/ds4-aiden-stream-bench.json - tool-eval-bench summary:
/tmp/tool-eval-bench/runs/ds4f/COMBINED_DS4F_RESULT.md,/tmp/tool-eval-bench/runs/ds4f/combined_ds4f_result.json - llama-benchy summary:
/tmp/llama-benchy/results/DS4F_LLAMA_BENCHY_RESULT.md,/tmp/llama-benchy/results/ds4f_matrix.json,/tmp/llama-benchy/results/ds4f_concurrency.json