This sequence revisits the methodolgy of Evolutionary Database Design, twenty years later. A key constraint to database-changes-as-code has all the time been round shared database assets. With copy-on-write branching in Databricks Lakebase, a one-second, zero-storage-at-creation department of a terabyte-scale manufacturing database is now an O(1) operation, and the constraint that stored Follow #4 (all people will get their very own database occasion) aspirational has lifted. On this sequence, the authors describes what adjustments when the constraint lifts: not the methodology, that holds, however the practices that emerge for the primary time, the team-scale governance that turns into automated, the function evolution for the DBA, and the brand new functionality that brokers share with their human counterparts.
Jen is the developer character from Evolutionary Database Design. In that essay she applied a database refactoring, splitting an inventory_code subject into location_code, batch_number, and serial_number, as a routine consumer story, illustrating that DBAs and builders can collaborate, schemas can evolve in small increments, and migrations carry the change ahead safely.
The sequence picks up with Jen twenty years later. The methodology she follows is identical one she adopted in 2003. What’s new is the technical functionality beneath her workflow, enabled by the lakebase structure: copy-on-write database branching, which makes the practices she has been studying about operationally actual at manufacturing scale. Throughout the three components of this sequence she is identical Jen at three scopes, her day (Half 1), her new playbook (Half 2), and her group (Half 3).
Half 1 walked Jen by means of one characteristic. The practices she adopted have been described within the 2003 Evolutionary Database Design essay, expanded within the 2006 Refactoring Databases e book , and introduced into the CI/CD pipeline within the 2010 Steady Supply e book (Chapter 12).
Unique Seven Practices
The 2003 essay named seven practices. 5 of the seven had limitations of their utility till 2026.
- DBAs collaborate carefully with builders.
- All database artifacts are model managed with utility code.
- All database adjustments are migrations.
- All people will get their very own database occasion.
- Builders repeatedly combine database adjustments.
- All database adjustments are database refactorings.
- Builders can replace their databases on demand.
Limitations of their utility
- Follow #1 (DBA collaboration). Each schema change had production-scale penalties if it acquired unfastened, so DBA assessment remained synchronous and gating. Collaboration was constrained by the DBA’s calendar.
- Follow #4 (All people will get their very own database occasion). Licensing prices, infrastructure prices, DBA time. Aspirational on most groups. Most groups fell again to shared improvement databases and accepted the rivalry.
- Follow #5 (Steady integration of database adjustments). The 2010 Steady Supply wave introduced migrations into the pipeline, however the pipeline ran migrations towards shared goal databases. Per-pipeline isolation was lacking.
- Follow #6 (All database adjustments are refactorings). Making use of every refactoring required follow areas (check databases) that almost all groups didn’t have at PR granularity.
- Follow #7 (Builders replace on demand). Builders may run migrations towards shared environments on demand, however couldn’t safely experiment, as a result of their experiments would have an effect on others.
What Modified
The know-how launched by Databricks Lakebase removes the roadblocks to the implementation of the above practices. Databricks Lakebase is a managed Postgres database that makes use of the identical object storage layer (the info lake) that the remainder of the Databricks lakehouse runs on. The database’s knowledge lives in shared, sturdy storage – successfully S3 buckets; the Postgres engine runs as a separate compute layer above it. Compute and storage scale independently. The engine can scale up below load, down when visitors drops, and to zero when idle. Lakebase is built-in with the unity catalog permitting for unified governance throughout a number of environments.
Copy-on-write database branching is what the decoupled structure makes sensible at scale. A department creates a brand new pointer into the identical shared storage with a divergence marker. Till the department writes, it shares all pages with its guardian. When the department writes, solely the modified pages diverge; the guardian stays untouched. Branching is a metadata operation, no knowledge copy required, finishing in roughly one second no matter guardian dimension. The branches preserve knowledge within the modified pages solely.
When the technical price of a department is decoupled from the scale of the info inside it, the constraint behind Follow #4 (all people will get their very own database occasion) is unblocked. Per-developer, per-PR, per-experiment branches change into routine. The compensating layer above can come out. Mocks come out of the check loop, changed by actual Postgres on a per-test department. Shared staging stops being the one place to check schema adjustments. In-memory database substitutes (H2, SQLite) come out of the unit-test layer. The devops tax to create docker based mostly containers to run native databases will not be mandatory, DBA ticket queues for provisioning shrink as a result of branches are self-service.
The know-how is what allows methodology optimizations and completes the objective of the practices behind the unique 2003 publish.
Rising practices for 2026
Lakebase copy-on-write database branching lifts the earlier limitations on the unique practices and allows 4 further ones for elaboration.
- DBAs collaborate carefully with builders. With the schema diff posted on each PR, the DBA evaluations async, like every other code reviewer. For the reason that provisioning tax is negligible the DBA’s now have the bandwidth to assessment and possibly even work with the builders to create the options within the first place as an alternative of reviewing publish implementation.
- All database artifacts are model managed with utility code. The schema diff, the database migrations and migration check outcomes now be part of the artifact set.
- All database adjustments are migrations. Plus a brand new authorship rule: idempotency. Permitting all merged migrations to be deployed to downstream environments like QA, staging and Manufacturing robotically.
- All people will get their very own database occasion. Operational at per-developer, per-PR, and per-experiment granularity. Permitting builders to experiment on a number of options and never simply accept the primary resolution that works. This freedom will end in optimum options being developed for manufacturing scale.
- Builders repeatedly combine database adjustments. Operational at PR granularity. Each PR runs by means of CI by itself department.
- All database adjustments are database refactorings. The 2006 catalog nonetheless applies; branches provide you with an inexpensive rehearsal area to strive the refactorings on numerous sized environments, migrations that labored in improvement databases nearly all the time didn’t carry out in manufacturing, now you’ll be able to strive migrations in bigger dimension databases to make sure they’re performant.
- Builders can replace their databases on demand. “On demand” now means one second, remoted, towards production-shaped knowledge.
- Harmful testing as a default choice. Blast radius is zero; reset prices one second. Don’t have to fret about, “Will my check pollute knowledge for others?”, since now after each check run I can get a brand new department.
- A/B variant prototyping on the database stage. Construct two designs on parallel branches; work with the bigger group and the DBA’s to debate the answer and do a present and inform, then hold the winner.
- With the built-in unity catalog, governance is designed as soon as, inherited by all of the branches. Insurance policies observe every department robotically, we’ll talk about this intimately in Half 3.
- Agent-as-practitioner with the identical branching functionality. Brokers get branches, not manufacturing, we’ll talk about this intimately in Half 3.
How the workflow runs in CI
The 2 practices that get improved probably the most due to the aptitude supplied by Lakebase are #4 (all people will get their very own database occasion) and #5 (builders repeatedly combine database adjustments). Not simply everybody will get a database, each PR or each department or a number of databases per department may be had with negligible price and time. The mechanism may be automated utilizing two GitHub Actions workflow templates that may be scaffolded into each undertaking.
Per-PR department creation. pr.yml triggers on pull_request: [opened, synchronize] and creates ci-pr- forked from the PR’s base department:
The identical job applies migrations towards the CI department and runs the applying’s check suite towards actual Postgres. Full instance of the workflow may be discovered right here: pr.yml
Schema diff posted on the PR. The identical pr.yml job dumps the schema of each branches, codecs the diff, and posts it as a PR remark. That is what lets the DBA assessment async like every code reviewer (Follow #1, re-cast):
The DBA, the code reviewer, the group and any agent creator of the migration see the identical artifact on the PR.
Department cleanup on merge. merge.yml destroys the CI Pull Request Department ci-pr- and the linked characteristic department’s Lakebase department the second the PR merges:
Full instance of the merge workflow may be discovered right here: merge.yml With so many branches floating round it could be value having a weekly cleanup script that removes orphaned and unused branches, an instance workflow may be discovered right here: cleanup-orphans.yml. Lastly, if the merge is right into a ‘tiered’ department, e.g. used for staging or predominant/manufacturing (this idea can be additional elaborated partially 3), the workflow could also be orchestrated to chop a recent department from the tier to check migrations towards it earlier than making use of the ultimate migration to any setting the place customers are discovered stay and continually including knowledge.
Collectively these workflows implement each PR will get its personal database and branches are ephemeral as properties of the pipeline, not developer disciplines.
The brand new playbook for Evolutionary Database Improvement
Follow #1: DBAs collaborate carefully with builders
Rule. The DBA collaborates with builders all through the characteristic, not simply at gate-review time. The collaboration is asynchronous, inline with the PR, the way in which different code reviewers take part.
Why is that this a sturdy behavior now? The schema diff and migration check outcomes land on each PR robotically (see How the workflow runs in CI above). The DBA evaluations a concrete artifact on their very own schedule. The migration has already run towards a real-data CI department, so the DBA doesn’t must mentally simulate the change.
Mechanics:
- The DBA is a CODEOWNER on schema-affecting paths (
migrations/,db/, schema check directories). - The DBA evaluations on the PR like every other reviewer, async.
- Evaluation focus shifts from will this break the database to is that this the proper design? Has it been applied the proper method? Subjects: knowledge integrity guidelines, indexing technique, design cohesion, future extensibility, long-term maintainability.
Anti-pattern. Not together with the DBA within the PR circulate with there being professional artifacts to assessment.
The DBA function additionally features new duties in coverage administration and governance at group scale. Half 3 covers these.
Follow #2: All database artifacts are model managed with utility code
Rule. Each SQL file, migration script, and schema check lives in the identical repository as the applying code. The schema diff and migration check outcomes be part of the artifact set as PR-time outputs.
Why is that this a sturdy behavior now? Branching provides two new artifacts to the set: the per-PR schema diff and the per-PR schema migration check outcomes. Each are generated by CI from the precise department state. Each land within the PR as concrete proof about what was modified and the way the change migration script carried out.
Mechanics:
- Migration information in a versioned listing (
migrations/,db/migration/,alembic/variations/, framework-dependent). - Schema-affecting exams within the check tree alongside utility exams.
- Schema diff generated per PR by CI, posted as a PR remark.
- Migration check ends in the CI run abstract and PR remark.
Anti-pattern. Producing the schema diff exterior the PR circulate (a separate dashboard the reviewer has to open). The artifact has to stay the place the assessment occurs. For the reason that schema adjustments are tied to the code adjustments and breaking this dependency creates downstream issues for deployment.
Follow #3: All database adjustments are migrations
Rule. No handbook ALTER TABLE towards any setting. Each schema change is a versioned, checked-in migration script. Migrations are idempotent.
Why is that this a sturdy behavior now? The migration-as-artifact rule is unchanged from 2003. What’s new is the authorship self-discipline of idempotency. The identical migration runs towards many branches over the lifetime of a transition, so it has to behave the identical method each time. A migration that fails on re-apply is a bug.
Mechanics:
- Use migration frameworks like
flyway, liquibase, Knex, Alembicor others, these frameworks hold observe of which migrations have been run and which haven’t been, this permits the group to use a command likeflyway migratewhich simply applies the adjustments that haven’t been utilized (by preserving observe of the adjustments in a metadata desk) - It’s higher to separate irreversible work throughout migrations. For instance a migration that provides new columns and drops the supply column in a single shot makes rollback more durable than they should be, so utilizing the broaden first and contract later technique, gives many extra choices after a deploy cycle confirms no stay readers reference it.
- The 2006 database refactoring catalog names which refactorings are reversible and which aren’t. Use it.
Anti-pattern. A migration that is dependent upon the schema being in a particular intermediate state due to native adjustments made within the department. The migration should apply appropriately towards any guardian department that features prior migrations.
Follow #4: All people will get their very own database occasion
Rule. Each developer, each PR, each experiment, each check run will get its personal Lakebase department.
Why is that this a sturdy behavior now? The additional effort to create docker containers, to put in native database servers, purchase licenses, hydrate empty databases with current schema and check knowledge is not required. Only a easy create-branch Lakebase command branches a 1TB database in the identical one second as a 1MB database. No knowledge is copied at creation; solely modified pages eat storage. Per-developer, per-PR, and per-experiment cases are routine.
Mechanics:
- Per-developer branches: created on demand by way of
databricks postgres create-branchor the SCM extension’s branch-create circulate. - Per-PR branches: created robotically by CI on PR open (
pr.yml), destroyed on merge or shut (merge.yml). See How the workflow runs in CI above for the PR and merge snippets. - Per-experiment branches: forked off staging or manufacturing for design exploration; discarded after the experiment.
Anti-pattern. Sharing a improvement database throughout the group “for comfort.” The contention-driven serialization Half 1 named comes again the second the branches collapse.
The place Jen’s instance extends. Her per-developer department was forked off staging at characteristic begin. The CI department was forked off staging on PR open. Her A/B exploration branches (Follow #9) have been forked off staging in parallel. 4 branches throughout one characteristic, all in seconds, all remoted.
Follow #5: Builders repeatedly combine database adjustments
Rule. Each PR runs by means of CI towards a recent Lakebase department, with migrations utilized and exams run towards actual Postgres.
Why is that this a sturdy behavior now? The CI pipeline has had migration self-discipline since 2010. What’s new is per-pipeline isolation: every PR will get its personal department, so integration runs towards real-shaped knowledge with out rivalry.
Mechanics:
- CI creates a department on PR open. See How the workflow runs in CI above for the PR snippet.
- Migrations utilized towards the department by way of
lakebase-migrate apply. - Software exams run towards the migrated department by means of the ORM, no mocks.
- Schema diff posted on the PR.
- Department destroyed on merge or shut.
Anti-pattern. Working PR validation towards shared staging. The serialization comes again; the per-PR isolation property is misplaced.
Follow #6: All database adjustments are database refactorings
Rule. Schema adjustments observe named refactoring patterns: Cut up Column, Rename Column, Transfer Column, Substitute Sort, and so forth. Every has specific transition mechanics (hold previous plus new in parallel, populate from previous, swap readers, drop previous).
Why is that this a sturdy behavior now? The 2006 catalog at databaserefactoring.com names 70+ refactorings with labored examples. What’s new in 2026 is an inexpensive place to rehearse the transition mechanics: a developer department absorbs the rehearsal; the CI department verifies; manufacturing sees solely the verified end result.
Mechanics:
- Establish the refactoring by title. Look it up within the catalog.
- Apply the named transition mechanics on a per-developer department. Validate towards production-shaped knowledge.
- Open the PR. CI runs the migration by itself department and posts the diff.
- Merge as soon as the diff and the check outcomes land approval.
Anti-pattern. A one-off schema change that doesn’t map to a named refactoring. The 70+ catalog covers the widespread instances; if you end up exterior it, you’re seemingly combining a number of refactorings into one migration and may break up.
The place Jen’s instance extends. Her V87 migration is the Cut up Column refactoring: splitting inventory_code into location_code, batch_number, and serial_number. The catalog web page at databaserefactoring.com/SplitColumns.html names the transition mechanics. Her department was the rehearsal area; the PR’s CI run was the verification.
Follow #7: Builders can replace their databases on demand
Rule. A developer can refresh their department’s database state on demand: reset to the guardian’s present state, fork a recent department off manufacturing, discard an experimental department, share a department with a teammate. All in seconds.
Why is that this a sturdy behavior now? “On demand” in 2026 means one second, remoted, towards production-shaped knowledge. None of those operations seek the advice of ops calendars or DBA queues.
Mechanics:
- Reset: delete and recreate the department from its guardian.
- Fork off manufacturing:
databricks postgres create-branch --source manufacturing. - Discard:
databricks postgres delete-branch. - Share: hand the department endpoint to a teammate for a pairing session.
Anti-pattern. Treating any department as sturdy past its objective. The migration is the sturdy artifact; the department is the workspace.
Follow #8: Harmful testing as a default choice
Rule. When the blast radius of a damaging motion is zero, damaging testing turns into a every day choice quite than a quarterly train.
Why is that this a sturdy behavior now? A department resets in a single second. Something you do to a department may be undone by making a recent one off the identical guardian. Harmful exams cease needing ops calendars and approval gates.
Issues that now match inside a standard characteristic cycle:
- “What occurs if my migration fails midway by means of the UPDATE assertion?” Run it. Kill the method at 50%. Confirm the rollback works. Reset.
- “What occurs if a backup-restore is mid-way by means of when a failover triggers?” Simulate the partial state. Confirm the applying’s habits. Reset.
- “What’s the precise time-to-recover for our DR runbook?” Run the runbook. Measure. Reset.
- “Does this migration run efficiently towards manufacturing form knowledge or dimension” confirm it earlier than c
Cultural impact. When reset prices nothing, groups cease treating the check database as a treasured useful resource. Assessments may be aggressive. Cleanup may be skipped, as a result of the following department begins recent.
The place Jen’s instance extends. Earlier than opening her PR, she took the production-shaped knowledge on her department and intentionally corrupted round one p.c of the inventory_code values to appear like edge instances: lacking digits, embedded areas, trailing whitespace. The sorts of artifacts that historic knowledge accumulates. She ran her migration. Two rows failed her substring math. She mounted the script and re-ran. The department absorbed the damaging check. Manufacturing by no means noticed it.
Follow #9: A/B variant prototyping on the database stage
Rule. When two designs are in rivalry, construct them on parallel branches, evaluate towards production-shaped knowledge, and hold the higher resolution.
Why is that this a sturdy behavior now? Per-branch price is close to zero. Exploring two schema designs not requires choosing one prematurely, and it not requires a provisioning course of most groups is not going to undertake for an exploratory query.
Mechanics:
- Create two branches off the identical guardian.
- Apply Design A’s migration to 1 department, Design B’s migration to the opposite.
- Run the applying towards every. Measure what issues: question latency on the widespread learn path, migration time at manufacturing quantity, index footprint, lock rivalry below simulated load.
- Choose the higher resolution. Discard the sub optimum resolution. Doc the choice within the PR description so the following one who has to increase the schema is aware of what was thought-about and why this design was picked.
Anti-pattern. Working A/B prototypes with out writing down the choice and the reasoning. The branches are gone in a second; the design resolution ought to be everlasting.
The place Jen’s instance extends. She thought-about two designs for the situation/batch/serial characteristic: three new columns on the present stock desk, or a separate inventory_attributes lookup desk keyed by inventory_id, anticipating that extra attributes can be added later. She constructed each on parallel branches off staging. She ran the applying’s learn path towards every, measured question efficiency towards production-shaped knowledge, and checked out how every migration would scale to manufacturing volumes. The lookup-table model carried out worse on the widespread learn path as a result of each stock show required a be part of. She shipped the columns model, threw away the lookup-table department, and left a word within the PR description: Thought-about lookup-table model; rejected as a result of the widespread learn path turns into a be part of. Revisit if greater than 5 attributes accumulate.
What Jen’s New Playbook Exhibits
We’ve named the seven practices from 2003 with the restrictions that stored 5 of them aspirational, and re-cast them for 2026 as soon as branching landed, and added 4 new practices that branching allows. Eleven practices whole within the new playbook for Evolutionary Database Improvement, 9 of that are defined above.
In Half 1 – Jen’s story: one characteristic, one database we noticed Jen work by means of one characteristic: she paired a code department with a Lakebase department, ran an actual migration towards production-shaped knowledge in seconds, examined with out mocks, opened a PR with the schema diff posted inline, and merged with the migration utilized and the ephemeral branches cleaned up. Database change turned a part of regular improvement.
In Half 3 – Jen’s Workforce at Scale, we take a look at the playbook at fifty builders, the DBA’s advanced duties in coverage administration and governance, and the brokers creating branches alongside Jen. Practices #10 and #11 get their full therapy there.
The Companion: Plugin Walkthrough covers the Lakebase SCM Extension for VS Code and Cursor.
A Lakebase App Dev Package for brokers, with a companion e-book for human practitioners, can be launched as a follow-on.