Irrational Solution
Sample report
Production Triage report
Covers one system: the GreenRoom V1 to V2 migration, reviewed before cutover.
SampleNot a client engagement. This report reviews GreenRoom V2, a professional network for the music industry built by GreenRoom Technologies Inc.
The findings describe the migration plan as it stood at review, and section 5 shows the fixes built since. V1 is still live and the cutover is pending. Vendors are named generically, as in any report shared beyond its client. Client reports follow this structure, scoped to your system and your decision.
Summary
Do not cut over on the plan as it stood at review.
The target design, a Postgres graph in four layers, is sound. The way there was not. Two critical risks put data that cannot be re-created at stake: sign-in continuity rested on an identity bridge nobody had checked, and the ETL could not safely be re-run.
The fix plan has four items. F1 to F3 must land before cutover. F4 ships alongside them, so discovery, the feature V2 is built around, is protected from day one.
Risks by severity
- 2critical
- 2high
- 1medium
System snapshot
Two systems run side by side until cutover.
V1 runs on serverless services with a document store, a hosted auth provider and a payment processor. V2 runs services over a Postgres graph. An ETL moves V1 data to Postgres, and an identity bridge from the V1 auth provider to V2 auth was assumed but not checked.
V1 (live)
Serverless API
Document storeidentity, payments, bookings
Hosted auth provider
Payments
ETLno deterministic keys
Identity bridgeassumed, not checked
V2 (private beta release)
Web app and API
Postgres graphfour layers, entities to intent
V2 auth
Discoverynatural-language search over profile data
Account history and sign-in continuity both have to survive the cutover.
Findings, ranked by risk
Five risks, ranked by blast radius and likelihood. In a client report each one cites file paths, queries and run output; here the evidence is summarized.
| ID | Risk | Severity | Fixed by |
|---|---|---|---|
| R1 | Sign-in continuity could break at cutover | Critical | Fixed by F1 |
| R2 | The ETL was not safe to re-run | Critical | Fixed by F2 |
| R3 | No rollback once V2 accepts writes | High | Fixed by F3 |
| R4 | Discovery has no evaluation harness | High | Fixed by F4 |
| R5 | No per-stage visibility in the migration | Medium | Fixed by F2 |
Sign-in continuity could break at cutover
CriticalWhere it lives Identity bridge, from the hosted auth provider to V2 auth
Evidence
The plan assumed the auth provider's sessions and user IDs would carry over for legacy V1 accounts. Nothing checked that mapping, and those sessions do not transfer to the V2 auth system.
Blast radius
Legacy users locked out of their paid history, a support backlog, and lost trust in the cohort V2 most needs to keep.
The ETL was not safe to re-run
CriticalWhere it lives ETL, from the document store to Postgres
Evidence
Rows were inserted without deterministic keys, so a partial failure followed by a re-run could write a second copy of the same record.
Blast radius
Duplicate or conflicting records across every table the ETL writes, and manual clean-up of each one.
No rollback once V2 accepts writes
HighWhere it lives Cutover plan
Evidence
No reverse migration was designed, and nothing prevented V1 from changing underneath a load.
Blast radius
Any defect found after cutover becomes a forward-fix emergency instead of a controlled revert.
Fixed by F3: Freeze V1 during loads, with manifest-based rollback
Discovery has no evaluation harness
HighWhere it lives Discovery, the natural-language search layer over the graph
Evidence
No fixed set of queries with expected matches existed, so a change to a prompt or model could not be compared with a baseline.
Blast radius
The feature V2 is built around gets worse without anyone noticing until users stop trusting it.
No per-stage visibility in the migration
MediumWhere it lives ETL pipeline
Evidence
Success was inferred from the absence of errors, not from checks showing that the loaded data matched the source.
Blast radius
Silent data loss found weeks later, when it is expensive to trace.
Fix plan
Four fixes, about 15 days of work in all. Each is written so any senior engineer could carry it out, not only the one who wrote it.
Run a read-only pass over every legacy V1 account against the V2 identity data, and produce an exceptions list before any cutover.
Use a lazy credential bridge: a V1 user signs in with their existing password and the V2 account is created at first sign-in. Accounts that fail mapping get a re-link flow designed now, not during an incident.
Acceptance check
Every legacy account either maps cleanly or appears on the exceptions list with a re-link path. Conflicting rows are held back and reported, not loaded.
Give every migrated record a deterministic uuid5 key derived from its source identity, and write with ON CONFLICT so a re-run changes nothing.
Add a dry-run mode and a read-only verify suite that reports counts per stage, so each run can be audited and resumed.
Acceptance check
Field-level parity: after running the full ETL twice, every field of every migrated row matches the source mapping, and the second run changes no rows. Row counts alone do not pass.
Hold V1 read-only for the length of each load so the source cannot move underneath it. Record a manifest of every row a load writes, so a rollback removes exactly those rows and nothing else.
Acceptance check
A rollback rehearsed on a copy of the data returns it to its pre-load state, confirmed by a field-level comparison with no differences.
Freeze a set of real discovery queries, each with its expected profile matches, into a regression suite that runs on every merge.
Pin the model version, set temperature to 0, and cache retrieval results, so a change in score means the system changed and not the noise around it. This is a tripwire, not a research project.
Acceptance check
The suite fails the build only when the pass rate drops by more than 5 percentage points against the frozen baseline, averaged across 3 runs.
Where the system stands
The migration is not done: the cutover is still pending. Since this review, the fixes for the migration itself have been built:
- F1, the credential bridge: a V1 user signs in with their existing password, and accounts that do not map cleanly are held back, not loaded.
- F2, safe re-runs: every record has a deterministic uuid5 key and every write uses ON CONFLICT, so running the load twice changes nothing. A dry-run mode previews each load, and a field-level audit checks the loaded data against the source.
- F3, freeze and rollback: V1 is read-only during each load, and a rollback removes exactly the rows the load wrote.
F4, the evaluation tripwire for discovery, is being brought online.