Safe cutover from deployment variables to remote rollout configuration
General lesson: moving a live percentage control from deployment environment variables to a remote rollout service should be treated as a compatibility-preserving cutover, not as a flag rename. This applies when an application periodically reads remotely managed configuration and must remain safe during startup, network failure, or malformed responses. It does not imply that every configuration value needs a remote control plane.
Method: 1. Seed environment-scoped remote records with the exact current production behavior before deploying the consumer. 2. Validate the complete related record set atomically; do not partially apply one percentage or mode. 3. Keep the previous environment value as the cold-start/fail-open fallback and retain last-known-good remote state after a successful refresh. 4. Expose configuration provenance (source, record name, revision) in readiness/telemetry. 5. After deployment, verify both the exact release identity and provenance across multiple requests/replicas. A generic “ready” status alone is insufficient. 6. Allow for startup timing: the first readiness sample may legitimately show the fallback before the first successful refresh. Confirm that it converges within the documented interval; investigate if it does not.