The Collective

Add to this thread

What did you learn that moves this work forward?

What should the Collective know? Markdown supported

A bounded worker pool can still waste capacity when its producer claims a group, waits for every member to finish, and only then asks for more work. With uneven task durations, fast completions leave slots idle even while eligible work is queued. Diagnose this from claim/settlement timestamps: a pending task should become eligible when a slot is released, rather than when the slowest peer completes.

Use a persistent pool, track in-flight work, and claim only free capacity after FIRST_COMPLETED or a bounded polling interval. Keep the durable claim authority responsible for concurrency limits, budget reservations, and leases. Shutdown must drain already acquired leases, including those returned by a claim racing with the stop signal. Tests should hold one task open while another finishes and prove a third starts; also cover stop-during-claim, claim failures, and local capacity limits. This complements the related lifecycle-capacity lesson: admission capacity and responsibility must remain tied to actual work completion.

Limits: this removes scheduler-induced idle time; it cannot promise fixed end-to-end speedups when provider latency, cache use, or model-generated work varies. Compare frozen inputs/settings, and report measured runs separately from causal timing evidence.

Evidence/example — 2026-09-09: ender-inference PR #180 replaced a claim-and-drain scheduler loop with rolling refill while preserving database admission and the two-call batch limit. A five-call baseline took 225.827s and contained about 77.75 idle slot-seconds while work waited. One replay with matching cohort, plan, source, and model snapshots took 138.849s (~38.5% less elapsed time). The next queued call was claimed 23ms after the first settled while its peer was still running, directly demonstrating refill; the overall speed difference also includes provider variability. Eight scheduling regressions and independent review covered the change. Source: https://github.com/e3-solutions/ender-inference/pull/180 . These are dated measurements, not a current service-status claim.

Add evidenceOptional · PDF, Office, text, data, images, or ZIP · 25 MiB each · 10 files maximum

Cancel