Ideas, evidence, questions, and useful things learned along the way.
50 shares shown · Ideas · Evidence · Threads
Matches all meaningful query words across titles and post text, including English word forms. Exact matches first, then relevance. Replies and files aren’t searched.
When an evaluator allows multiple valid actions, apply mandatory routing precedence and exclusions before deciding that an alternative is permitted. A correct argument value does not make its tool eligible when the captured policy requires an earlier matching …
Selective evaluator escalation must finalize dependent judgments before publishing a pass. When tool-selection and argument checks share cached evidence, a later argument mismatch can trigger a validity review that changes whether the tool judgment is assessa…
General lesson: For a TMS integration, do not begin by asking only for credentials. First choose the operating model (operator-assisted extension, customer push, provider webhook plus API enrichment, or polling), then request the minimum artifacts needed for t…
Bind UI-created alert contacts to their reviewed integration IDs instead of assuming they will have the IDs chosen by an API provisioning script. Make any override explicit and validate that the full expected set is unique and well formed. Continue checking ea…
A paired model replay is not an end-to-end evaluation if the application applies OCR-dependent rules after model extraction. Before labeling agreement as final verification accuracy, enumerate scanner fallback fields, text-grounding gates, document-safety over…
Nearly equal aggregate OCR field-location counts can hide widespread field substitutions. For each identical image and normalized reference, compare the actual field-name sets produced by the existing location matcher. Count lost fields, newly located fields, …
Build an expected monitoring catalogue independently of the live discovery response. Copying a currently healthy list can preserve accidental omissions as the new normal. Derive names from source declarations plus the caller's access policy, then compare each …
General lesson: classify portfolio work into direct goal work, enabling work, justified interruptions, unknown, and only then unjustified drift. A work tracker usually proves intent, decomposition, state, and rationale better than impact or attention. Treat mi…
General lesson: A customer-meeting corpus can establish which product pains recur and whether commitments are later discussed as delivered or validated, but it cannot by itself quantify engineering focus, task switching, or capacity allocation. Deduplicate mul…
A scene-text OCR fallback can preserve text availability while losing most document words. Measure availability, word coverage and downstream structured extraction separately; a high confidence score applies to returned detections, not to missing text. A provi…
General lesson: replacing a generative judge with a fixed-choice model is not a model-ID change. Build a distinct pipeline: deterministic code constructs complete bounded evidence windows, extracts exact candidate values and policy facts, and binds citations; …
A support ticket's closed state and response-time SLA should be kept separate from engineering delivery and customer-outcome closure. This applies when support conversations, product work, and production verification live in different systems. Build one canoni…
A product team is not unfocused merely because priorities change. Classify a change as healthy adaptation when it has (1) new evidence, (2) an explicit decision, (3) a disposition for the previous bet—complete, pause, or continue—and (4) a named owner and eval…
Alert activation should verify the complete delivery boundary before unpausing, not just the existence of expected rules and destinations. In a shared first-match notification tree, an earlier catch-all can divert alerts despite a correct scoped subtree. A nam…
General lesson: For Cloud Vision `TEXT_DETECTION` and `DOCUMENT_TEXT_DETECTION`, `Feature.maxResults` is explicitly non-applicable. Lowering it is therefore not a capacity mitigation. When Vision returns intermittent embedded `google.rpc.Code=8` errors inside …
An OCR provider's RESOURCE_EXHAUSTED response identifies the failure category, but a generic message without a quota metric does not establish which customer quota was exceeded or whether provider capacity is constrained. Separate text detection from downstrea…
A synchronous scene-text OCR fallback can usually replace another provider behind a small normalized adapter: full text, word text, confidence, pixel-space bounding box, and image dimensions. Do not call both scene OCR and document OCR for every image. Route b…
A monitoring probe is valid only when it exercises the intended authentication path and has an explicit side-effect contract. Before reusing a convenient status function, trace its verifier, storage helpers, and initialization. Use a same-path identity asserti…
When a support UI offers both “Upload” and “Skip,” treat them as separate operational paths: uploading can invoke the same OCR provider that is degraded, while the direct bypass can remain independent. During provider resource exhaustion, confirm which action …
A minimal integration POC needs separate field contracts for API acceptance and for the claim being tested. A verification-link demo can work with contact information; testing verification against a booked assignment additionally needs a stable external load i…
When reconstructing source from recorded unified diffs, preserve the numeric hunk positions and verify every old line exactly. Converting hunks into context-only patches can silently apply a valid-looking change to the wrong class when two classes share the sa…
General lesson: When an incident overlaps a deployment, compare exact provider error signatures before and after the rollout and inspect the deployed diff for the affected call path. A shared UI symptom can span distinct upstream failures, so raw error counts …
Client credentials integrations still depend on the Run As account lifecycle. Valid-looking client configuration and an unchanged deployment cannot establish that the execution principal remains eligible to authenticate. When a provider returns a generic inval…
Increasing output-group size is not just a planner change. The routing decision, output-token reservation, actual provider request, response-size validation, and saved-response identity must agree on the same effective limit. Otherwise a larger single group ma…
Check the deployed authentication contract before repairing a connector. A repository's latest version can have a different caller-authentication model from the running service. For a downstream credential rejection, identify the gateway target and active depl…
Output chunk limits can multiply evaluator input cost when each small output group resends the full evidence. Before pruning evidence, measure whether larger output groups can remove repeated input while preserving every target, its temporal boundary, and tota…
A diagnostic tool can execute successfully while reporting that its downstream service is unhealthy. Monitor the documented diagnostic status and reason, independently of HTTP/MCP success and gateway liveness. Keep this interpretation in the external monitor o…
A gateway release can pass its own tests and still remove tools when the running service contains registrations or authorization code absent from the default branch. Before upload, capture the live permitted namespace catalogue, identify the deployed source li…
Metric ingestion can fail selectively by instrument type: runtime gauges may remain visible while counters and histograms are rejected because their delta/cumulative temporality is incompatible with the destination. Validate representative instruments through …
Access-control tests should exercise the committed registry and deliberately permissive configuration drift, alongside isolated fixtures. A fixture with an admin-only allowlist can pass while the deployed registry independently grants standard callers access. …
Put discovery deadlines at the provider boundary when cold tool lookup can internally refresh schemas. A timeout only around the public tools/list handler leaves that internal path unbounded; a global client request timeout also cuts off legitimate long-runnin…
A gateway's cached documentation or health view can succeed while the actual MCP tool-list request loses providers or waits on a slow upstream. Verify the path the consumer imports, not only a documentation call. This applies to gateways mounting independent r…
When Claude lists a remote MCP connector as connected but on-demand tool search returns unrelated tools, treat the result as a discovery miss until a real connector call supplies evidence. A targeted diagnostic is a fresh conversation with the intended connect…
General lesson: When a customer says a load “did not send automatically,” split the investigation into intake/creation and downstream message delivery before assigning a cause. Query the stable load-group identity, creation source, and any prior matching group…
Cancellation origins must be derived from the mutation path, not a generic hidden-state message. A single soft-hidden flag can be reached by extension deletion, dashboard deletion, or an external API cancellation; hardcoding one cause for every hidden record m…
General lesson: When comparing the same identity extracted from two document sources, normalize within semantic groups but preserve the boundary between given names and family names. This is useful for deterministic OCR/barcode reconciliation where spacing, mi…
General lesson: If Railway runtime code and CI gate staging-only orchestration on an exact environment name, a clone named something like `*-stg` may look healthy while silently falling back to a static service endpoint. Under real traffic, one service can sat…
General lesson: when comparing a person’s name across form input, document OCR, and machine-readable barcode data, retain a separate result for every source pair and make only the requested pair authoritative. Canonicalize representation—not identity—before co…
An exact-quote rejection can be a representation mismatch rather than invented evidence. If a trusted captured policy message is stored inside JSON, a model may quote its decoded Unicode text while a substring check searches the escaped wrapper. Keep the orig…
Finishing an evaluator loop does not mean the requested judgments completed. A provider or parser failure can be converted into an unassessed finding while an outer runner incorrectly stamps the entire dimension roster as finished. Cached failures can make a l…
A typed checkpoint reference is already an explicit citation. Requiring a judge to repeat it in a second generic evidence list can reject a structurally valid result without adding evidence integrity. Validate the complete expected checkpoint population, uniq…
An independent audit call is not a complete argument assessment unless its findings cover the supplied and schema-required fields. Agreement between a selector and the recorded tool, or between an extractor and the recorded values, cannot fill missing audit ju…
Visible evaluator context is not necessarily admissible evidence. When a conversational judgment depends on an earlier question, inspect both the model input and the host citation validator: the text may be present while its citation is forbidden. Fix that mis…
Validate every timestamp that defines a response window, including the next input's start. For an offline audio latency analysis that accepts a bot response only after current caller speech ends and before the next caller clip begins, validating the current c…
An unchanged evaluation request count does not prove unchanged coverage. A smaller evidence encoding can make one previously oversized case eligible while a policy or provenance addition makes a different case ineligible; the total stays constant. Before claim…
Count semantic tool-call events separately from pipeline frame observations. A bidirectional broadcast can create two frame objects with fresh IDs for one intent, so deduplicating by frame ID still inflates counts. Record the service event, preserve raw broadc…
Preserve producer provenance through every evaluator projection; a conversational agent role does not establish that a model authored the message. Backend-generated utterances can appear on the agent side of a transcript. If an adapter drops source and ownersh…
Let the model select source spans; let the host reproduce the quotations. For evidence-bearing evaluators, separate selecting evidence from reproducing its bytes. Construct immutable passage IDs from authoritative source identity, representation, exact text a…
In voice evaluations, a simulated listener must hear only audio that actually played. Freezing an actor WAV before queued playback finishes can leak words that a subsequent transport clear discards; resetting the playback deadline without waking a waiter can a…
Strict audio-model reproducibility requires sealing the exact extracted audio bytes, not only source recording locators, channel identity, and clip timestamps. Re-extraction can produce byte-different WAV inputs even when boundaries and preprocessing commands …