From 3e826a317b49d207b5ede70b277617129537eca4 Mon Sep 17 00:00:00 2001 From: siddharthd Date: Thu, 13 Aug 2026 11:29:08 +1000 Subject: [PATCH] CLAUDE.md: document the Frollo account feed and the new reconcile exclusion Two things a future reader would otherwise have to reverse-engineer. The reconcile queue now excludes account-feed rows for a reason unrelated to payment method: a feed row is the account's own ledger entry, not a receipt awaiting a statement line, because the importer only covers accounts whose statements are deliberately not imported. Measured, the queue went from 8 to 558 without it. And the de-duplication rule needs its history attached, because the obvious version is wrong in a way that passes tests: keeping close-id rows as genuine repeats survived 29 tests and a clean dry run while still doubling three salary payments. In-scope duplicate pairs have id gaps from 58 to 260 million; real repeats sat at 1-4. --- CLAUDE.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 59 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a094364..c576559 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -131,7 +131,7 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/< ### Key Tables - `statements` — one row per billing period per bank account -- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown +- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent ### Cash and reconciliation @@ -142,9 +142,19 @@ within 3 days and 1% on amount — and accepting one is silently destructive: reconciled manual rows are filtered out of every query, so the cash spend disappears while the card transaction it matched claims to be that same spend. -Only cash is excluded. Bank transfers *do* appear on a statement now that -transaction accounts are imported, and NULL means unknown — both stay -candidates, preserving the behaviour of every pre-existing row. +Only cash is excluded on payment method. Bank transfers *do* appear on a +statement now that transaction accounts are imported, and NULL means unknown — +both stay candidates, preserving the behaviour of every pre-existing row. + +**Account-feed rows are excluded separately, and for a different reason** +(`awaitsStatementLine()`, added with the Frollo importer). A feed row is the +account's own ledger entry, not a receipt awaiting a statement line: the importer +only covers accounts whose statements are deliberately *not* imported. Without +the exclusion those ~600 rows a year sit in the queue forever and bury the +receipts that genuinely need a decision — measured, the queue went from 8 to 558. +It is scoped to a named list (`ACCOUNT_FEED_SOURCES`) rather than to +`source IS NOT NULL`, so a future source that *does* await a statement is not +swept up by it. ATM withdrawals stay categorised as spend rather than `transfers`. Treating them as transfers only works if every cash purchase is logged; with partial logging @@ -617,6 +627,51 @@ deleted, and deleting a rule must not cascade away the audit trail. changed by something else since the run are flagged, because reverting restores the pre-run value and discards the later edit. +### Frollo account feed (`source = 'frollo'`) + +Covers the accounts whose statements arrive every 182 to 460 days — AMP including +the loan, ANZ Access, Wise including the income account, Up, ING, the small +transaction accounts. **Credit cards are excluded on purpose**: their monthly +statements already cover them and are authoritative, and a second producer for +the same rows would only create reconciliation work. + +- Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and + `src/lib/frollo-ingest.ts` (report + insert). 42 unit tests. +- Two callers share that module so they cannot drift: `scripts/import-frollo.mts` + (dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by + the n8n `Frollo Import` workflow). The DB is reached through an injected + executor because one runs under Prisma and the other under a bare `pg` client. +- **Export with pending EXCLUDED.** A pending row changes both its id *and* its + description when it settles, so importing one guarantees a duplicate next run. + +**The one thing to understand before touching this: a CDR re-consent makes Frollo +re-ingest an account's whole history under fresh transaction ids while the +originals survive**, and consents expire annually. On one export 385 of 1,989 rows +were such twins and every salary payment appeared twice — a blind import doubles +reported income. `dedupe()` collapses each natural-key group to its **lowest** id +(lowest because old ids survive re-ingestion, which keeps `source_ref` stable and +makes a re-import insert nothing). + +An earlier version kept close-id rows on a 10,000 threshold, reasoning that +genuine same-day repeats have consecutive ids. It passed 29 tests and a clean dry +run and was still wrong: in-scope duplicate pairs have id gaps from **58 to 260 +million**, while real repeats sat at 1–4. Nothing separates them. It now collapses +unconditionally and *flags* anything within 10 for review, because the errors are +asymmetric — keeping a twin doubles income, collapsing a real repeat understates +by one row. + +`category` is not imported: Frollo assigns its own and the owner reports they are +often wrong. Note a trigger rewrites NULL to `'other'` on insert, which is +equivalent for display and leaves the rules engine (which writes +`transaction_overrides.category_override`) free to decide. + +Foreign rows carry only the native figure — there is no AUD equivalent anywhere in +the export — so they take order-ingestion's shape: `amount` is native, +`foreign_currency_code` names it, `amount_aud` stays NULL, and +`AMOUNT_UNCONVERTED` already reports the balance as incomplete. + +Full history and the reasoning: `DECISIONS.md` ING-11 in the smarthome repo. + ### Trusting extracted statement data **Balance assertions are the check that works.** `getStatements` computes