rssnyder/polk-county-iowa-property-search
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

feat: run the data load on a systemd timer #4

Merged
rssnyder merged 1 commit from feat/etl-systemd-timer into main 2026-08-04 22:30:32 -05:00
Conversation 0 Commits 1 Files changed 5 +130 -15
rssnyder commented 2026-08-04 22:30:25 -05:00
Owner
Copy link

Adds the unit files, an example environment file, and setup notes for
running update_polk_county_db.py on the app host. systemd rather than cron
because cron isn't installed there, and because EnvironmentFile= reads the
credentials directly — under cron that needs a wrapper to source them.

Also streams the PostgreSQL sync instead of materializing each table. The
previous fetchall(), plus the second copy that building one batch list
creates, held two complete copies of a 170k-400k row table in memory and was
OOM-killed on the 464MB app host during the first real run.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Adds the unit files, an example environment file, and setup notes for running update_polk_county_db.py on the app host. systemd rather than cron because cron isn't installed there, and because EnvironmentFile= reads the credentials directly — under cron that needs a wrapper to source them. Also streams the PostgreSQL sync instead of materializing each table. The previous fetchall(), plus the second copy that building one batch list creates, held two complete copies of a 170k-400k row table in memory and was OOM-killed on the 464MB app host during the first real run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the unit files, an example environment file, and setup notes for
running update_polk_county_db.py on the app host. systemd rather than cron
because cron isn't installed there, and because EnvironmentFile= reads the
credentials directly — under cron that needs a wrapper to source them.

Also streams the PostgreSQL sync instead of materializing each table. The
previous fetchall(), plus the second copy that building one batch list
creates, held two complete copies of a 170k-400k row table in memory and was
OOM-killed on the 464MB app host during the first real run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rssnyder deleted branch feat/etl-systemd-timer 2026-08-04 22:30:32 -05:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
rssnyder/polk-county-iowa-property-search!4
No description provided.