feat: hash downloads and push when sales land #6

Merged

rssnyder merged 1 commit from feat/etl-hash-checks-and-pushover into main

2026-08-11 21:45:08 -05:00

rssnyder commented

2026-08-11 21:44:57 -05:00

Owner

Three changes to the nightly load, all prompted by a fortnight of clean runs
that turned out to be indistinguishable from a broken feed.

Change detection no longer trusts file size alone. The county rebuilds its
exports most nights, usually with identical contents, and the old check
compared Content-Length against the local file: any edit that left the byte
count unchanged — a corrected buyer name, a fixed spelling — was skipped
forever. The server's ETag is Apache's inode-size-mtime, so it moves on every
rebuild and can't decide the question either. The two signals now do separate
jobs: validators from a HEAD decide whether to spend the bandwidth, and a
SHA-256 of the bytes decides whether anything actually changed. State lives in
raw/.download_state.json; deleting it costs one extra round of downloads, not
correctness. A run that re-fetches an unchanged file says so.

The current-year sales file is no longer skipped outright when the local copy
is under 24 hours old. That saved one HEAD per run and cost correctness: the
county publishes late in the evening, so a file written by an off-schedule run
could hide a same-day update until the following day.

A Freshness line reports the newest sale_date across the sales tables and how
long it has sat there, warning after POLK_STALE_AFTER_DAYS (default 7). Every
other signal in the log reports on our own machinery and reads the same
whether the county is quiet or the feed is broken; this is the one check on
their side of the line.

Finally, an optional Pushover notification when a run adds sales, enabled only
when PUSHOVER_TOKEN and PUSHOVER_USER are both set. Counts are taken after
reconciliation, so a re-recorded deed — a new row that retires an old one —
nets to zero rather than announcing a sale that never happened.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Three changes to the nightly load, all prompted by a fortnight of clean runs that turned out to be indistinguishable from a broken feed. Change detection no longer trusts file size alone. The county rebuilds its exports most nights, usually with identical contents, and the old check compared Content-Length against the local file: any edit that left the byte count unchanged — a corrected buyer name, a fixed spelling — was skipped forever. The server's ETag is Apache's inode-size-mtime, so it moves on every rebuild and can't decide the question either. The two signals now do separate jobs: validators from a HEAD decide whether to spend the bandwidth, and a SHA-256 of the bytes decides whether anything actually changed. State lives in raw/.download_state.json; deleting it costs one extra round of downloads, not correctness. A run that re-fetches an unchanged file says so. The current-year sales file is no longer skipped outright when the local copy is under 24 hours old. That saved one HEAD per run and cost correctness: the county publishes late in the evening, so a file written by an off-schedule run could hide a same-day update until the following day. A Freshness line reports the newest sale_date across the sales tables and how long it has sat there, warning after POLK_STALE_AFTER_DAYS (default 7). Every other signal in the log reports on our own machinery and reads the same whether the county is quiet or the feed is broken; this is the one check on their side of the line. Finally, an optional Pushover notification when a run adds sales, enabled only when PUSHOVER_TOKEN and PUSHOVER_USER are both set. Counts are taken after reconciliation, so a re-recorded deed — a new row that retires an old one — nets to zero rather than announcing a sale that never happened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

rssnyder added 1 commit

2026-08-11 21:44:57 -05:00

feat: hash downloads and push when sales land 61fb704587

Three changes to the nightly load, all prompted by a fortnight of clean runs
that turned out to be indistinguishable from a broken feed.

Change detection no longer trusts file size alone. The county rebuilds its
exports most nights, usually with identical contents, and the old check
compared Content-Length against the local file: any edit that left the byte
count unchanged — a corrected buyer name, a fixed spelling — was skipped
forever. The server's ETag is Apache's inode-size-mtime, so it moves on every
rebuild and can't decide the question either. The two signals now do separate
jobs: validators from a HEAD decide whether to spend the bandwidth, and a
SHA-256 of the bytes decides whether anything actually changed. State lives in
raw/.download_state.json; deleting it costs one extra round of downloads, not
correctness. A run that re-fetches an unchanged file says so.

The current-year sales file is no longer skipped outright when the local copy
is under 24 hours old. That saved one HEAD per run and cost correctness: the
county publishes late in the evening, so a file written by an off-schedule run
could hide a same-day update until the following day.

A Freshness line reports the newest sale_date across the sales tables and how
long it has sat there, warning after POLK_STALE_AFTER_DAYS (default 7). Every
other signal in the log reports on our own machinery and reads the same
whether the county is quiet or the feed is broken; this is the one check on
their side of the line.

Finally, an optional Pushover notification when a run adds sales, enabled only
when PUSHOVER_TOKEN and PUSHOVER_USER are both set. Counts are taken after
reconciliation, so a re-recorded deed — a new row that retires an old one —
nets to zero rather than announcing a sale that never happened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

rssnyder merged commit fbfc8229e4 into main

2026-08-11 21:45:08 -05:00

rssnyder deleted branch feat/etl-hash-checks-and-pushover

2026-08-11 21:45:08 -05:00

rssnyder referenced this pull request from a commit

2026-08-11 21:45:09 -05:00

Merge pull request 'feat: hash downloads and push when sales land' (#6) from feat/etl-hash-checks-and-pushover into main