wallet: lock the ingest, pair on time and app, skip notices, ECB fallback
ci / lint-test (push) Successful in 54s

Four defects found reviewing a month of overseas wallet rows:

- Race: n8n posts a poll's items concurrently and both halves of a tap
  usually share a poll, so each ran findPairedRow before the other inserted.
  11 taps became 22 rows (A$3,841.40), identical created_at on every pair.
  One advisory lock now covers re-check -> pair -> adopt -> insert, on the
  transaction's own connection (a second pooled one per request deadlocks a
  burst). Mutation-tested: without the lock the concurrent test fails 3/3.
- Pair window: amount within +/-1 DAY folded the UNICEF $50 donation and an
  unrelated $50 Amex charge 17h later into one row and renamed it. A pair is
  now minutes apart AND from two different apps. Needs when/where a
  notification arrived: migration 0039 adds expense_metadata.notified_at /
  notif_app, backfilled from capture.wallet_events (144 rows). A merged half
  now records wallet_evt:<id>, so a replay cannot merge it twice.
- Not purchases: a CommBank overdraft notice became a $21.21 purchase named
  "To avoid fees and interest," (the Westpac fixed-width parser ran on a
  CommBank body). Account notices and transit-card balance drawdowns are
  skipped; the descriptor parser only runs on Westpac.
- FX: with no ledger history (Hong Kong) every tap stored A$0.00. The ECB
  rate via Frankfurter is the fallback, flagged rate_ecb; the ledger median
  stays first because it carries the card's FX fee.

Also: dates in Australia/Melbourne like the statements (UTC slicing filed 28
of 134 rows a day early), and test:setup now actually rebuilds personal_test
-- replaying the dump over it never applied a migration.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-10-03 16:55:07 +10:00
co-authored by Claude Opus 5.5
parent 22fde30b26
commit a3e4e1102a
8 changed files with 450 additions and 146 deletions
+14 -5
View File
@@ -19,14 +19,23 @@ if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
exit 1
fi
# Create test database (ignore error if it already exists)
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres \
-c "CREATE DATABASE $TEST_DB;" 2>/dev/null || true
# Drop and recreate, so the schema is rebuilt rather than replayed. Replaying
# the dump over an existing database was a silent no-op for every migration:
# each CREATE TABLE failed with "already exists" and the old shape survived, so
# a new column (0039's expense_metadata.notified_at, 2026-10-03) never reached
# the test database. The guard is because this now DROPs: it refuses anything
# not named *_test, whatever TEST_DB was set to.
case "$TEST_DB" in
*_test) ;;
*) echo "ERROR: refusing to drop '$TEST_DB' -- the test database must be named *_test"; exit 1 ;;
esac
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres -q \
-c "DROP DATABASE IF EXISTS $TEST_DB WITH (FORCE);" \
-c "CREATE DATABASE $TEST_DB;"
# Wipe and rebuild schema from production (schema only, no data)
echo "Copying schema from $PG_USER to $TEST_DB..."
docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q -v ON_ERROR_STOP=1
echo "Schema ready."