Models, migrations and seeding at scale

D10 from chapter twenty six was "changing the schema is the scary part: the migration that half ran, with no way back". This chapter is how that is answered, and it is more interesting than it sounds, because the usual answer is a lie.

The model

A generated model is a struct with tags:

type Invoice struct {
    ID        uint         `gorm:"primarykey" json:"id"`
    Number    string       `gorm:"size:50;uniqueIndex" json:"number"`
    Total     money.Amount `gorm:"not null" json:"total"`
    Due       jsontime.Date `json:"due"`
    ClientID  uint         `gorm:"index" json:"client_id"`
    Client    *Client      `json:"client,omitempty"`
    Version   int          `gorm:"default:1" json:"version"`
    CreatedAt time.Time    `json:"created_at"`
    UpdatedAt time.Time    `json:"updated_at"`
    DeletedAt gorm.DeletedAt `gorm:"index" json:"-"`
}

Three of those fields are the framework's rather than yours.

Version is optimistic locking, and chapter thirty three explains the 409 it produces. DeletedAt is soft delete: a DELETE sets the timestamp and every query excludes it automatically, which is how the Trash page works.CreatedAt and UpdatedAt are maintained by GORM.

Migrations, and the honest problem

grit migrate

It reads your models and makes the database match. No migration files to write.

That convenience has a well-known cost and this is where most frameworks are quiet: GORM's AutoMigrate has no down scripts. It can add a column. It cannot tell you what it added last Tuesday, and it cannot undo it.

So the answer here is not a pretend rollback. It is a record:

grit migrate            # apply, and record what this run changed
grit migrate --status   # what previous runs did
grit migrate --down     # undo the last run, using that record

Each run writes down what it actually added. --down reads that record and drops exactly those things. It is not a general rollback, it is an undo ofrecorded runs, and the distinction matters: a change made by someone typing SQL into a console was never recorded, so it cannot be undone this way.

The refusal that protects you

grit migrate --down

Dropping a column destroys data. The command asks, and a non-interactive run needs --yes, deliberately, so a CI pipeline cannot quietly delete a column because somebody renamed a struct field.

A rename is the case to understand. Changing Name to FullName in a struct is not a rename to AutoMigrate: it is an unknown column appearing and a known one no longer being mentioned. The safe sequence is to add the new column, backfill it, then remove the old one in a later change, which is the same expand-and-contract dance every schema tool requires.

Seeding, and why it is not optional

D9 was "nothing to develop against".

grit seed                        # every seeder
grit seed Invoice --count 50000  # one, at scale

The generator writes a factory per resource that produces realistic values for each field type: a real-looking name for string, a valid address for email, an amount in range for money.

At ten rows everything works. The sort is instant, pagination has one page, and an N+1 costs ten queries instead of ten thousand. Every list-page bug you have is invisible until the data arrives, which is why the habit is to seed beforeyou look at a screen rather than after a user complains.

A million rows, with the heap flat

The seeder is the pipeline from chapter fourteen, in production:

build rows in chunks  ->  bounded channel  ->  batch writers  ->  database

Measured on a laptop: *999,800 rows in 59 seconds, with the heap flat at 10 MB the whole time*.

The number that matters is the third one. Sixteen thousand rows a second is a fine figure and it is not the point. The point is that memory is a property of the pipeline rather than of how many rows you asked for: the channel holds a fixed number of batches in flight, so ten thousand rows and ten million look the same to the heap.

The seeder is also resumable: it counts what is in the table and inserts only the difference. Run it twice and the second run does nothing. Kill it halfway through a million rows and run it again, and it continues.

Backups, and the half that counts

grit backup           # write an archive
grit restore <file>   # replay one

cmd/restore existing beside cmd/backup is the point. B5 asked "is any of this backed up", and the honest reading of that question is "has anyone ever restored one".

A backup you have never restored is a rumour. The archive carries a row count per table, so a restore can be checked rather than assumed, and the restore is a command rather than a runbook page nobody has followed.

grit new migrations-lab --api
cd migrations-lab
grit generate resource Product \
  --fields "name:string, price:money, sku:string"
grit migrate
grit seed Product --count 1000

grit migrate --status

grit generate field Product weight:float
grit migrate
grit migrate --status      # a new run, with the new column
grit migrate --down        # and gone again

time grit seed Product --count 200000

grit backup
grit seed Product --count 10          # change the data
grit restore <the archive>

Assignment

Commit your work with the message Migrations - Assignment One, exactly as written, so the leaderboard can score it.

grit seed Reading --count 100000