CLAUDE.md: document the Frollo account feed and the new reconcile exclusion
ci / lint-test (push) Successful in 39s
ci / lint-test (push) Successful in 39s
Two things a future reader would otherwise have to reverse-engineer. The reconcile queue now excludes account-feed rows for a reason unrelated to payment method: a feed row is the account's own ledger entry, not a receipt awaiting a statement line, because the importer only covers accounts whose statements are deliberately not imported. Measured, the queue went from 8 to 558 without it. And the de-duplication rule needs its history attached, because the obvious version is wrong in a way that passes tests: keeping close-id rows as genuine repeats survived 29 tests and a clean dry run while still doubling three salary payments. In-scope duplicate pairs have id gaps from 58 to 260 million; real repeats sat at 1-4.
This commit is contained in:
@@ -131,7 +131,7 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
|
|||||||
### Key Tables
|
### Key Tables
|
||||||
|
|
||||||
- `statements` — one row per billing period per bank account
|
- `statements` — one row per billing period per bank account
|
||||||
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown
|
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
|
||||||
|
|
||||||
### Cash and reconciliation
|
### Cash and reconciliation
|
||||||
|
|
||||||
@@ -142,9 +142,19 @@ within 3 days and 1% on amount — and accepting one is silently destructive:
|
|||||||
reconciled manual rows are filtered out of every query, so the cash spend
|
reconciled manual rows are filtered out of every query, so the cash spend
|
||||||
disappears while the card transaction it matched claims to be that same spend.
|
disappears while the card transaction it matched claims to be that same spend.
|
||||||
|
|
||||||
Only cash is excluded. Bank transfers *do* appear on a statement now that
|
Only cash is excluded on payment method. Bank transfers *do* appear on a
|
||||||
transaction accounts are imported, and NULL means unknown — both stay
|
statement now that transaction accounts are imported, and NULL means unknown —
|
||||||
candidates, preserving the behaviour of every pre-existing row.
|
both stay candidates, preserving the behaviour of every pre-existing row.
|
||||||
|
|
||||||
|
**Account-feed rows are excluded separately, and for a different reason**
|
||||||
|
(`awaitsStatementLine()`, added with the Frollo importer). A feed row is the
|
||||||
|
account's own ledger entry, not a receipt awaiting a statement line: the importer
|
||||||
|
only covers accounts whose statements are deliberately *not* imported. Without
|
||||||
|
the exclusion those ~600 rows a year sit in the queue forever and bury the
|
||||||
|
receipts that genuinely need a decision — measured, the queue went from 8 to 558.
|
||||||
|
It is scoped to a named list (`ACCOUNT_FEED_SOURCES`) rather than to
|
||||||
|
`source IS NOT NULL`, so a future source that *does* await a statement is not
|
||||||
|
swept up by it.
|
||||||
|
|
||||||
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
|
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
|
||||||
as transfers only works if every cash purchase is logged; with partial logging
|
as transfers only works if every cash purchase is logged; with partial logging
|
||||||
@@ -617,6 +627,51 @@ deleted, and deleting a rule must not cascade away the audit trail.
|
|||||||
changed by something else since the run are flagged, because reverting restores
|
changed by something else since the run are flagged, because reverting restores
|
||||||
the pre-run value and discards the later edit.
|
the pre-run value and discards the later edit.
|
||||||
|
|
||||||
|
### Frollo account feed (`source = 'frollo'`)
|
||||||
|
|
||||||
|
Covers the accounts whose statements arrive every 182 to 460 days — AMP including
|
||||||
|
the loan, ANZ Access, Wise including the income account, Up, ING, the small
|
||||||
|
transaction accounts. **Credit cards are excluded on purpose**: their monthly
|
||||||
|
statements already cover them and are authoritative, and a second producer for
|
||||||
|
the same rows would only create reconciliation work.
|
||||||
|
|
||||||
|
- Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and
|
||||||
|
`src/lib/frollo-ingest.ts` (report + insert). 42 unit tests.
|
||||||
|
- Two callers share that module so they cannot drift: `scripts/import-frollo.mts`
|
||||||
|
(dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by
|
||||||
|
the n8n `Frollo Import` workflow). The DB is reached through an injected
|
||||||
|
executor because one runs under Prisma and the other under a bare `pg` client.
|
||||||
|
- **Export with pending EXCLUDED.** A pending row changes both its id *and* its
|
||||||
|
description when it settles, so importing one guarantees a duplicate next run.
|
||||||
|
|
||||||
|
**The one thing to understand before touching this: a CDR re-consent makes Frollo
|
||||||
|
re-ingest an account's whole history under fresh transaction ids while the
|
||||||
|
originals survive**, and consents expire annually. On one export 385 of 1,989 rows
|
||||||
|
were such twins and every salary payment appeared twice — a blind import doubles
|
||||||
|
reported income. `dedupe()` collapses each natural-key group to its **lowest** id
|
||||||
|
(lowest because old ids survive re-ingestion, which keeps `source_ref` stable and
|
||||||
|
makes a re-import insert nothing).
|
||||||
|
|
||||||
|
An earlier version kept close-id rows on a 10,000 threshold, reasoning that
|
||||||
|
genuine same-day repeats have consecutive ids. It passed 29 tests and a clean dry
|
||||||
|
run and was still wrong: in-scope duplicate pairs have id gaps from **58 to 260
|
||||||
|
million**, while real repeats sat at 1–4. Nothing separates them. It now collapses
|
||||||
|
unconditionally and *flags* anything within 10 for review, because the errors are
|
||||||
|
asymmetric — keeping a twin doubles income, collapsing a real repeat understates
|
||||||
|
by one row.
|
||||||
|
|
||||||
|
`category` is not imported: Frollo assigns its own and the owner reports they are
|
||||||
|
often wrong. Note a trigger rewrites NULL to `'other'` on insert, which is
|
||||||
|
equivalent for display and leaves the rules engine (which writes
|
||||||
|
`transaction_overrides.category_override`) free to decide.
|
||||||
|
|
||||||
|
Foreign rows carry only the native figure — there is no AUD equivalent anywhere in
|
||||||
|
the export — so they take order-ingestion's shape: `amount` is native,
|
||||||
|
`foreign_currency_code` names it, `amount_aud` stays NULL, and
|
||||||
|
`AMOUNT_UNCONVERTED` already reports the balance as incomplete.
|
||||||
|
|
||||||
|
Full history and the reasoning: `DECISIONS.md` ING-11 in the smarthome repo.
|
||||||
|
|
||||||
### Trusting extracted statement data
|
### Trusting extracted statement data
|
||||||
|
|
||||||
**Balance assertions are the check that works.** `getStatements` computes
|
**Balance assertions are the check that works.** `getStatements` computes
|
||||||
|
|||||||
Reference in New Issue
Block a user