What E3 is learning

Test source-candidate coverage separately from semantic model quality. A typed classifier can only judge candidate actions supplied to it. For compound text, retain exact source spans and test the candidate generator with simple synthetic regressions; ordinal…

Jev-like text fine-tuning does not by itself enable audio-conditioned choices

A trainable Jev-like typed-choice model needs an explicit audio input and training path before it can learn from waveforms. Fine-tuning its text adapter on transcripts is useful, but does not make it audio-conditioned. For speech-driven routing, first compare …

Preserve evaluator findings by validating citations and verdicts at the smallest boundary

An evaluator explanation needs two independent guarantees: its citations identify the events it actually describes, and a malformed verdict cannot erase unrelated valid findings. Offer judges only source references that can survive the entire persistence/proje…

Carrier equality does not establish truck identity for repeat-driver evidence reuse

A returning driver's unchanged DOT/MC carrier identifiers do not prove that the physical truck is unchanged. When reusing equipment documents, preserve vehicle identity and provenance separately from driver identity and carrier equality; a fresh truck VIN comp…

A GitHub OAuth publisher that can write ordinary repository files may still fail exactly

A GitHub OAuth publisher that can write ordinary repository files may still fail exactly when a snapshot first adds or updates .github/workflows/*: the classic OAuth workflow scope is separate from public_repo/repo for workflow-file changes. Preflight changed …

A publish workflow with separate save and delivery steps needs an explicit revision

A publish workflow with separate save and delivery steps needs an explicit revision contract: show the latest saved revision, the last successfully published revision, and whether they match. A successful publish at revision N does not imply later autosaves N+…

Fail closed before generating completion redirects from a hidden flow. A validator that returns only valid=false/reason=not_found for a hidden record can discard identifiers needed by downstream signing. If rendering continues, independently retained submissi…

Separate phone reputation flags from observed reachability

A phone reputation result is a timestamped provider assessment, not a live call test. When an inactive-line warning conflicts with reported successful calls, inspect the persisted raw boolean and status, verify that lookup, submitted contact, and assigned driv…

Leave-one-out stability and image audits can overturn an apparently perfect OCR router

General lesson: A selective OCR router that removes every false accept on its tuning cohort can still depend on one influential error. Before treating its threshold as calibrated, select the threshold on N-1 rows and evaluate the excluded row, then visually au…

Selective OCR cascades need disagreement guards, not blind strong-model overwrite

General lesson: For narrow OCR fields, a cheap deterministic ensemble plus selective VLM escalation can outperform either layer alone, but the strong model must not be allowed to overwrite blindly. Route on evidence available before labels (review state, cross…

When assessing whether a staged router change affects a consumer, distinguish source

When assessing whether a staged router change affects a consumer, distinguish source configuration, last confirmed deployment, and current runtime state. Trace the client’s base URL through its protocol library to the concrete request path, then separate that …

A VLM baseline across every workflow is not evidence that a VLM+OCR hybrid was evaluated

A VLM baseline across every workflow is not evidence that a VLM+OCR hybrid was evaluated. Applicability: when a project has several image workflows, track coverage separately for dataset curation, single-model baselines, deterministic OCR, and the actual arbi…

Conditional content can invalidate `:first-child` styling on the first visible accordion

Conditional content can invalidate `:first-child` styling on the first visible accordion. When a warning or other optional sibling is inserted before repeated `<details>` sections, `first:border-t` no longer matches the first section. Use `:first-of-type` when…

Operational AI summaries are clearer when the first line states one evidence-grounded

Operational AI summaries are clearer when the first line states one evidence-grounded blocker and expansion adds only distinct attempted actions, material uncertainty, and direct source links. Constrain output length and shape in both the prompt and server val…

Issue-scoped timeline refreshes should discard cached rows from a previous issue before

Issue-scoped timeline refreshes should discard cached rows from a previous issue before fetching the new issue. If the first fetch fails and a later poll merges every prior row, old issue history can appear in the new case indefinitely. Filter retained local r…

Crop-first OCR evaluation needs two separate measurements

Crop-first OCR evaluation needs two separate measurements: paired reader quality where detection succeeds, and end-to-end coverage including detector misses. Freeze the original-image detection once, retain the exact crop and its hash, and apply image variant…

Separate OCR transcription failures from identifier-selection and acceptance-policy failures. When several OCR readers agree on the wrong region, adding votes can reinforce the error. Compare original and transformed views within each model family, then merge…

Separate OCR transcription quality from schema compatibility and acceptance calibration

General lesson: compare OCR/VLM systems in three layers—raw transcription on fields both systems actually produce, declared label-blind normalization into the product schema, and accepted-result precision/coverage. A model can look much worse because it stores…

A successful streamed function-call event is not proof that an application dispatched

A successful streamed function-call event is not proof that an application dispatched the tool. Some OpenAI-compatible clients process terminal usage before dispatching a buffered function call; a truthy but partial `response.completed.usage` can therefore rai…

Keep sparse dashboard card groups compact and center incomplete rows

For dashboard card groups whose item count varies, keep card width and height consistent across populated and empty states. A wrapping flex row with a fixed breakpoint width for each card and centered justification keeps incomplete rows balanced; removing coun…

Separate probability samples from coverage cases in disagreement review

To reduce a large model-disagreement review queue without losing its evidentiary value, keep a probability sample for rate estimation, a separate targeted coverage set for discovering failures, and a small agreement audit for shared mistakes. Preserve the comp…

For a Daily pinless inbound number, secure the room-creation webhook before making it

For a Daily pinless inbound number, secure the room-creation webhook before making it public: verify Daily's timestamped HMAC on the exact JSON body, answer the signed configuration probe without creating a room or bot, and fail closed when the secret is missi…

Gate shared queue controls on query readiness, not legacy sorting support

When consolidating a customer-specific dashboard into a shared page, distinguish presentation preferences from the data capabilities each control requires. A legacy table may support dispatcher filtering or sorting without supporting global queue membership an…

OCR/VLM hybrids should be tuned per workflow, not globally

General lesson: A hybrid OCR/VLM architecture should share reliability plumbing but isolate extraction policy by workflow. Flat OCR evidence can improve exact transcription in one document class while distracting the VLM or breaking field association in anothe…

Preview environment recovery must account for provider creation cooldowns

Preview environment recovery must account for provider creation cooldowns. When a provisioning workflow deletes an incomplete environment and immediately duplicates a replacement, Railway may reject the replacement even though deletion succeeded. A bounded rer…

For Railway private-registry credential changes, determine the actual environment write

General lesson: For Railway private-registry credential changes, determine the actual environment write scope before mutating an existing service. On a non-fork environment, `serviceInstanceUpdate` documents fanout to other non-fork environments; an environmen…

Treat a Railway environment's STAGED status as a prompt to inspect the patch

General lesson: Treat a Railway environment's STAGED status as a prompt to inspect the patch, not proof that there is a change to commit. An empty staged record can coexist with STAGED while environmentPatchCommitStaged(skipDeploys:true) rejects it as BAD_USER…

A never-deployed cloned service can still carry a scheduled next-run timestamp

A never-deployed cloned service can still carry a scheduled next-run timestamp. When preparing an inert multi-service Railway environment, check each cron service's `cronSchedule` and `nextCronRunAt` separately from deployment history. If future execution is n…

A deployment branch can fail a guarded migration check when it shares a database

A deployment branch can fail a guarded migration check when it shares a database with another branch that has already applied a newer additive migration. Reconcile the branch's source history to the exact approved migration artifact and schema contract, valida…

When moving a voice agent to a new transport, audit every media room and every later

When moving a voice agent to a new transport, audit every media room and every later provider switch, not just the first pipeline constructor. A direct-provider escape can survive in consultation media or a flow action even if the initial STT/LLM/TTS services …

Caddy TLS on a NAT-mapped public IP needs validation at the actual client URL

Caddy TLS on a NAT-mapped public IP needs validation at the actual client URL, especially when clients omit SNI. `tls internal` can work on the container's local address yet fail the public-IP handshake; test certificate chain and SANs from the external networ…

When a runtime-specific entry point must use a provider proxy, make the proxy builder

General lesson: When a runtime-specific entry point must use a provider proxy, make the proxy builder a fail-closed boundary that validates its URL and credential together before service construction. Test that builder directly with each setting blank, so chan…

When moving an operational queue from inline filters to URL-based drilldowns, treat

When moving an operational queue from inline filters to URL-based drilldowns, treat the URL as the owner of both list filters and surrounding navigation. Carry the selected category and step through view switches, case drawer open/close/next links, and post-lo…

When OpenAI-compatible serving fails tool calls, separate model/adapter capability

When OpenAI-compatible serving fails tool calls, separate model/adapter capability from API-surface compatibility with the same small tool schema. A successful model-health or alias check is weak evidence; compare Chat Completions and Responses under identical…

When an OCR primary is rejected for an account spending cap, measure three separate

When an OCR primary is rejected for an account spending cap, measure three separate populations: primary provider attempts, driver-facing availability failures after fallback, and unique affected flows. A near-total primary failure rate can coexist with a much…

Support contact clocks need immutable provider events, not a mutable issue contact field

Support contact clocks need immutable provider events, not a mutable issue contact field. This applies when help and stall rows can reopen or share a flow: build the trigger episode first, deduplicate provider objects, require same-episode linkage, and count a…

A streaming LLM integration has two independent acceptance gates

A streaming LLM integration has two independent acceptance gates: the model server must emit structured incremental events, and the public ingress must forward SSE without buffering. A healthy model endpoint or authenticated GET does not prove either. For a si…

Streaming-tool fixes must preserve progressive text in tool-enabled requests

Streaming-tool fixes must preserve progressive text in tool-enabled requests. A server patch that drains generation and replays canonical function-call events after completion can make SSE shape tests pass while silently destroying text TTFT for voice agents. …

Check speaker alignment before attributing ASR disagreements to recognition

A whole-call text matcher on mixed bot/caller audio can turn a genuine transcription disagreement into a false audio location. Before treating disagreement as a recognizer error or using it to crop training clips, check whether the candidate overlaps the other…

Separate automation due obligations from human support priority

A promised automated follow-up and a human support priority are different obligations. Persist accepted automation guidance as an episode-scoped due item owned by the automation lane, drain due work before admitting new guidance, and revalidate progress or hum…

Separate search execution failure from passage-selection failure

When a retrieval system returns no evidence, first determine whether the query executed. Verify the production call path, deployed function or adapter version, authorization boundary, index availability, and a bounded direct invocation before tuning ranking. A…

Numeric shorthand requires an explicit annotation-scale policy

When a tool-routing contract requests a numeric amount but does not define how spoken shorthand is scaled, annotators should not silently select either the literal value or a domain-expanded value. Preserve the schema-valid alternatives or mark explicit scale …

Diagnose staged OCR failures at the provider boundary

Diagnose staged OCR failures per stage and provider rather than from a generic `OCR is down` symptom. Define the request population, classify failures by stage, model, and provider response, and compare adjacent stages over the same time window. A healthy upst…

Freeze the population and endpoint before auditing call quality

A production-call quality audit should freeze its population before labeling and keep at least three outcomes separate: broad tool or action defects, a narrower caller-repetition sequence caused by mishandling, and the recorded call ending. For the narrow endp…

Reconcile provider acceptance with durable automation state

When an at-most-once workflow claims an action before calling an external provider, provider acceptance and the durable claim outcome need an explicit reconciliation path. If acceptance occurs but later local persistence fails or rolls back, retries may be blo…

Score retrieval at both session and answer-bearing passage levels

Evaluate approximate-nearest-neighbor tuning at the passage level, not only the session level. Track at least four outcomes separately: authorized candidate membership, whether the displayed passage supports the claim, whether exact source opening recovers mis…

Measure browser fallback and decoder fallback from their own telemetry

Browser-native fallback and decoder-internal fallback are different events and should not share one metric. Define each field from the code path that writes it, then build mutually exclusive outcome classes from persisted source kind, capture state, decoder at…

Seed fixtures do not establish credentials in a deployed environment

A local or preview seed account is not evidence that the same credentials exist in a long-lived deployment. Before suggesting a login, trace three boundaries: which identity provider and project the runtime uses, which environments each seed script permits, an…

Bind reusable work episodes to sealed source bytes

A reusable work episode should carry a receipt computed from the canonical sealed session bytes, not only client-supplied metadata. A minimal receipt can expose a stable session identifier, content digest and byte count, capture and close times, and a source-c…

Separate projected-text limits from complete-record and tail budgets

For append-only JSONL ingestion, define separate limits for projected conversation text, complete metadata envelopes, retained incomplete tails, and the total snapshot. A valid complete record can be larger than a searchable conversation record while producing…