Digital release integrity — Highlight Report Addendum

The Project Office › Master Highlight Report › Digital release integrity

Highlight Report Addendum — Digital release integrity

This addendum records a production-delivery incident affecting the independence and currency of Cedwyn webapp releases. It does not describe a confirmed Cloudflare platform outage. The Cloudflare zone remains active, and the application-specific source branches and Pages projects remain present. The weakness is that some production requests and releases converge through shared components.

1. Recap

Incident and service position
  1. On 2–3 November, the production release path was reviewed after concern that one agent's update could replace or obscure another webapp's work. The review covered the active zone, Pages projects, Worker scripts and routes, audit events, telemetry, Git branches, pinned section revisions and the live production build history.
  2. Cloudflare returned an active zone state. The account contains 91 Pages project records, with separately configured production branches. This confirms that the source and Pages project separation has not simply been deleted.
  3. That separation is not complete at the public delivery layer. The Worker bfe-apex-router owns catch-all routes for bucklerfamily.estate/* and www.bucklerfamily.estate/*; its source proxies most application paths to the central Cedwyn Pages project. The central project publishes from the shared site branch.
  4. No Git branch overwrite between the Research and Forensic section branches was identified. Their newer source commits remained available. The incident was a stale assembled release, not proof that either branch's work had been erased.

2. Performance

Confirmed production and release findings
  1. Stale release confirmed. The production build published at approximately 04:39 UTC on 3 November was assembled from main commit 4a3c448. At that checkout, sections.lock still pinned Research to 3cd867a and Forensic to 9480ff3. Newer section commits had arrived at approximately 04:05 and 04:28 UTC respectively; those newer pins were committed to main at 04:31 UTC. The older release therefore omitted those newer revisions while leaving their source commits intact.
  2. Follow-on build in progress at last verification. Build run 41578577096 (run 502), using main commit aca84bd, contains the updated Research and Forensic pins. Its job was still in progress at the last GitHub status check (3 November, 04:40:45 UTC). Its successful completion and public deployment are not recorded here as confirmed.
  3. Separate Worker storage, shared releases. Worker bfe-analytics serves both the Cloudflare usage dashboard and the Analytics application. Their Durable Object data identities are separate, but the executable Worker and its assets are shared. A deployment intended for either application can change both.
  4. Additional shared route surface. Worker cedwyn-online serves multiple Wales and Argentina application paths. This is another shared deployment boundary that needs explicit ownership and release controls.
  5. Malformed API requests. Telemetry recorded 80 requests to duplicated paths including /cloudflare/cloudflare/api/ and /analytics/analytics/api/; these returned HTTP 404. Some sampled requests used internal workboard and burndown-check user-agent labels. The dashboard's current frontend code has the correct single-prefix API base, so the originating caller remains to be traced.
  6. Operational activity. The 2–3 November Cloudflare audit window recorded 89 Worker deployments, 36 route creations, 22 route updates and 30 DNS record changes. Audit entries generally resolve to the account owner identity; this evidence does not identify which agent initiated each action.
  7. Projects publication boundary. The old standalone Pages project bucklerfamily-projects still points to the Datro repository and its projects branch; its last successful production deployment was 27 October. The Cedwyn repository also has a current projects branch, which is the source used for the central Cedwyn build's /projects/ output. This report update is being released through the Cedwyn source path, not by restoring the old Datro branch or publishing to that standalone project.

3. Issues, risks and concerns

New control findings; no unsupported outage claim
  1. R6 — Production isolation and release integrity (AMBER, open). A catch-all apex router and shared central Pages build mean independently maintained source branches do not guarantee independently released application versions. A stale build can publish after newer section work is committed.
  2. R7 — Shared Worker blast radius (AMBER, open). Multi-application Workers can replace the code and assets serving more than one app. Separate database namespaces reduce data collision risk but do not provide code-release isolation.
  3. R8 — Snapshot reproducibility (AMBER, open). The inspected build report shows some sections assembled through live-site crawl or fallback crawl rather than a pinned source revision. The same source commit can therefore produce different output at different times. Missing pinned source must fail closed or produce an explicit, reviewed exception.
  4. R9 — Release attribution (AMBER, open). The Cloudflare account audit history does not provide enough agent-level attribution to determine which automation initiated every route, DNS or Worker action. Deployment records need to include the responsible agent, source commit, target application, reason and previous version.
  5. Service impact boundary. The evidence supports a production consistency and routing risk. It does not establish a platform-wide outage, widespread human visitor failure, data loss, or that one named agent deliberately overwrote another's work.
  6. Governance baseline. The programme's overall RAG remains AMBER pending an approved tolerance baseline and Executive decision. This addendum does not invent a time, cost or quality tolerance, nor infer an exception decision.

4. Moving forward

Corrective actions and approvals required
  1. Release verification. Confirm run 502 completes, deploys to the central cedwyn Pages project and serves the updated Research and Forensic revisions. Record the deployed commit and cache-busted page checks before closing the stale-release incident.
  2. Restore application-level delivery boundaries. Each application should publish from its own controlled source and deployment target. The apex router should resolve the application's assigned production target rather than proxying every path to one shared central build. Preserve the root homepage as a separately controlled product.
  3. Split multi-app Workers. Give the Cloudflare usage dashboard and Analytics their own deployable Workers. Review cedwyn-online and other shared route handlers for the same cross-application risk. Keep each app's own database and secrets scoped to its Worker.
  4. Enforce source freshness. A build must verify the current main commit and all pinned section SHAs immediately before publication. A pin that is missing, rejected or stale must block release; live crawl fallbacks must not silently replace a pinned source.
  5. Agent release contract. Every write and deployment must carry a target app, source branch and commit, agent identity, reason, expected routes/resources and previous version. Agents should not modify another app's Worker, route or binding unless the change is explicitly authorised as shared infrastructure.
  6. Monitoring repair. Trace the callers producing duplicated API prefixes, fix the common URL construction, and add tests that assert each webapp API base is prefixed exactly once. Separate application 4xx responses from Worker exceptions and Cloudflare platform errors in the dashboard.
  7. Projects release. Publish this addendum from the Cedwyn projects branch through the central Cedwyn build. Do not push it to the Datro-backed standalone Pages project or change domains, DNS or Worker routes as part of this release.

Decisions requested: approve the target architecture for independent app delivery; approve separate Worker ownership where code is currently shared; confirm which agent or service may authorise shared routing changes; and approve a fail-closed release rule for missing or stale section pins.

No DNS records, Pages custom domains, Worker routes, Worker bindings or production routing rules were changed in preparing this report. Source publication is being handled through the Cedwyn Projects branch and central build workflow.

Return to the master highlight report · Programme plan · Programme structure and risks

Small print: dotted links explain a word (see the glossary); amber-shaded links list every name the records use for the same thing.