CLAUDE.md: document the Frollo account feed and the new reconcile exclusion
ci / lint-test (push) Successful in 39s
ci / lint-test (push) Successful in 39s
Two things a future reader would otherwise have to reverse-engineer. The reconcile queue now excludes account-feed rows for a reason unrelated to payment method: a feed row is the account's own ledger entry, not a receipt awaiting a statement line, because the importer only covers accounts whose statements are deliberately not imported. Measured, the queue went from 8 to 558 without it. And the de-duplication rule needs its history attached, because the obvious version is wrong in a way that passes tests: keeping close-id rows as genuine repeats survived 29 tests and a clean dry run while still doubling three salary payments. In-scope duplicate pairs have id gaps from 58 to 260 million; real repeats sat at 1-4.
This commit is contained in:
@@ -131,7 +131,7 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
|
||||
### Key Tables
|
||||
|
||||
- `statements` — one row per billing period per bank account
|
||||
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown
|
||||
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
|
||||
|
||||
### Cash and reconciliation
|
||||
|
||||
@@ -142,9 +142,19 @@ within 3 days and 1% on amount — and accepting one is silently destructive:
|
||||
reconciled manual rows are filtered out of every query, so the cash spend
|
||||
disappears while the card transaction it matched claims to be that same spend.
|
||||
|
||||
Only cash is excluded. Bank transfers *do* appear on a statement now that
|
||||
transaction accounts are imported, and NULL means unknown — both stay
|
||||
candidates, preserving the behaviour of every pre-existing row.
|
||||
Only cash is excluded on payment method. Bank transfers *do* appear on a
|
||||
statement now that transaction accounts are imported, and NULL means unknown —
|
||||
both stay candidates, preserving the behaviour of every pre-existing row.
|
||||
|
||||
**Account-feed rows are excluded separately, and for a different reason**
|
||||
(`awaitsStatementLine()`, added with the Frollo importer). A feed row is the
|
||||
account's own ledger entry, not a receipt awaiting a statement line: the importer
|
||||
only covers accounts whose statements are deliberately *not* imported. Without
|
||||
the exclusion those ~600 rows a year sit in the queue forever and bury the
|
||||
receipts that genuinely need a decision — measured, the queue went from 8 to 558.
|
||||
It is scoped to a named list (`ACCOUNT_FEED_SOURCES`) rather than to
|
||||
`source IS NOT NULL`, so a future source that *does* await a statement is not
|
||||
swept up by it.
|
||||
|
||||
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
|
||||
as transfers only works if every cash purchase is logged; with partial logging
|
||||
@@ -617,6 +627,51 @@ deleted, and deleting a rule must not cascade away the audit trail.
|
||||
changed by something else since the run are flagged, because reverting restores
|
||||
the pre-run value and discards the later edit.
|
||||
|
||||
### Frollo account feed (`source = 'frollo'`)
|
||||
|
||||
Covers the accounts whose statements arrive every 182 to 460 days — AMP including
|
||||
the loan, ANZ Access, Wise including the income account, Up, ING, the small
|
||||
transaction accounts. **Credit cards are excluded on purpose**: their monthly
|
||||
statements already cover them and are authoritative, and a second producer for
|
||||
the same rows would only create reconciliation work.
|
||||
|
||||
- Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and
|
||||
`src/lib/frollo-ingest.ts` (report + insert). 42 unit tests.
|
||||
- Two callers share that module so they cannot drift: `scripts/import-frollo.mts`
|
||||
(dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by
|
||||
the n8n `Frollo Import` workflow). The DB is reached through an injected
|
||||
executor because one runs under Prisma and the other under a bare `pg` client.
|
||||
- **Export with pending EXCLUDED.** A pending row changes both its id *and* its
|
||||
description when it settles, so importing one guarantees a duplicate next run.
|
||||
|
||||
**The one thing to understand before touching this: a CDR re-consent makes Frollo
|
||||
re-ingest an account's whole history under fresh transaction ids while the
|
||||
originals survive**, and consents expire annually. On one export 385 of 1,989 rows
|
||||
were such twins and every salary payment appeared twice — a blind import doubles
|
||||
reported income. `dedupe()` collapses each natural-key group to its **lowest** id
|
||||
(lowest because old ids survive re-ingestion, which keeps `source_ref` stable and
|
||||
makes a re-import insert nothing).
|
||||
|
||||
An earlier version kept close-id rows on a 10,000 threshold, reasoning that
|
||||
genuine same-day repeats have consecutive ids. It passed 29 tests and a clean dry
|
||||
run and was still wrong: in-scope duplicate pairs have id gaps from **58 to 260
|
||||
million**, while real repeats sat at 1–4. Nothing separates them. It now collapses
|
||||
unconditionally and *flags* anything within 10 for review, because the errors are
|
||||
asymmetric — keeping a twin doubles income, collapsing a real repeat understates
|
||||
by one row.
|
||||
|
||||
`category` is not imported: Frollo assigns its own and the owner reports they are
|
||||
often wrong. Note a trigger rewrites NULL to `'other'` on insert, which is
|
||||
equivalent for display and leaves the rules engine (which writes
|
||||
`transaction_overrides.category_override`) free to decide.
|
||||
|
||||
Foreign rows carry only the native figure — there is no AUD equivalent anywhere in
|
||||
the export — so they take order-ingestion's shape: `amount` is native,
|
||||
`foreign_currency_code` names it, `amount_aud` stays NULL, and
|
||||
`AMOUNT_UNCONVERTED` already reports the balance as incomplete.
|
||||
|
||||
Full history and the reasoning: `DECISIONS.md` ING-11 in the smarthome repo.
|
||||
|
||||
### Trusting extracted statement data
|
||||
|
||||
**Balance assertions are the check that works.** `getStatements` computes
|
||||
|
||||
Reference in New Issue
Block a user