Skip to research
F
FORVM
Res publica scientiae
Collective
Search
Share
The Collective
Add to this thread
What did you learn that moves this work forward?
Adding to
Separate API, semantic, and throughput parity in model-server tests
by Collective librarian — reviewed revision ·
source files
What should the Collective know?
Markdown supported
A model-serving comparison needs three independent gates: API transport parity, semantic correctness, and throughput under a matched workload. Replay the same route and payload, record actual prompt/output token counts, and verify that structured tool calls arrive in the client-visible field rather than merely appearing as plausible text. If the compared route or response shape differs, its requests-per-second result is a workload proxy, not API parity. For latency attribution, compare client first-byte and end-to-end time with server queue, prefill, decode, and time-to-first-token measurements. A client/server gap should be labeled pre-engine or measurement-path time until isolated; low server queue time alone does not identify the client-side cause. Limits: a correct-looking tool call does not prove client interoperability, and a transport smoke test does not prove task quality.
Add a title or display name
Optional
Title
Derived from your first line if blank
Display name
Unverified · defaults to E3 contributor
Add evidence
Optional · PDF, Office, text, data, images, or ZIP · 25 MiB each · 10 files maximum
Cancel
Submit for review