fix: delete rows the county has superseded

rssnyder commented

2026-08-06 10:45:19 -05:00

Owner

The county re-records amended deeds — a misspelled name, the wrong buyer —
under a new book/page. Because book/page is part of the row key, the
correction arrives as a new row rather than an update, and the loaders only
ever INSERT OR REPLACE, so the superseded version was kept forever. Polk
currently has 145 such rows: users see sales that never happened (parcel
32001167000000 shows the same 2026-03-13 sale to both ROBERSTON and
ROBERTSON), and the inflated sale counts would raise watchlist notifications
for what are really corrections.

Treats the CSVs as the county's current state and deletes what is no longer
in them: SQLite is reconciled against the CSVs, then PostgreSQL against
SQLite. Converging the two copies also lets the existing row-count check
skip unchanged tables again, which is most of the nightly runtime.

Both passes refuse to delete more than 1% of a table (minimum 100 rows), and
treat an empty or unreadable source as a failed download rather than an
instruction to delete everything. ingest_log is deliberately not reconciled:
it is our own history, not the county's, and the staging copy legitimately
lacks rows PostgreSQL has.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

The county re-records amended deeds — a misspelled name, the wrong buyer — under a new book/page. Because book/page is part of the row key, the correction arrives as a new row rather than an update, and the loaders only ever INSERT OR REPLACE, so the superseded version was kept forever. Polk currently has 145 such rows: users see sales that never happened (parcel 32001167000000 shows the same 2026-03-13 sale to both ROBERSTON and ROBERTSON), and the inflated sale counts would raise watchlist notifications for what are really corrections. Treats the CSVs as the county's current state and deletes what is no longer in them: SQLite is reconciled against the CSVs, then PostgreSQL against SQLite. Converging the two copies also lets the existing row-count check skip unchanged tables again, which is most of the nightly runtime. Both passes refuse to delete more than 1% of a table (minimum 100 rows), and treat an empty or unreadable source as a failed download rather than an instruction to delete everything. ingest_log is deliberately not reconciled: it is our own history, not the county's, and the staging copy legitimately lacks rows PostgreSQL has. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

rssnyder added 1 commit

2026-08-06 10:45:19 -05:00

fix: delete rows the county has superseded 8c30b1ab07

The county re-records amended deeds — a misspelled name, the wrong buyer —
under a new book/page. Because book/page is part of the row key, the
correction arrives as a new row rather than an update, and the loaders only
ever INSERT OR REPLACE, so the superseded version was kept forever. Polk
currently has 145 such rows: users see sales that never happened (parcel
32001167000000 shows the same 2026-03-13 sale to both ROBERSTON and
ROBERTSON), and the inflated sale counts would raise watchlist notifications
for what are really corrections.

Treats the CSVs as the county's current state and deletes what is no longer
in them: SQLite is reconciled against the CSVs, then PostgreSQL against
SQLite. Converging the two copies also lets the existing row-count check
skip unchanged tables again, which is most of the nightly runtime.

Both passes refuse to delete more than 1% of a table (minimum 100 rows), and
treat an empty or unreadable source as a failed download rather than an
instruction to delete everything. ingest_log is deliberately not reconciled:
it is our own history, not the county's, and the staging copy legitimately
lacks rows PostgreSQL has.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

rssnyder merged commit 09f07de919 into main

2026-08-06 10:45:37 -05:00

rssnyder deleted branch fix/reconcile-superseded-rows

2026-08-06 10:45:38 -05:00

rssnyder referenced this pull request from a commit

2026-08-06 10:45:39 -05:00

Merge pull request 'fix: delete rows the county has superseded' (#5) from fix/reconcile-superseded-rows into main

Rows
Columns

fix: delete rows the county has superseded #5