From 2738213a236315a3da87f6f42495be1d90f89fb9 Mon Sep 17 00:00:00 2001 From: siddharthd Date: Thu, 13 Aug 2026 12:24:59 +1000 Subject: [PATCH] docs: record the ledger-duplicate guard and why awaitsStatementLine is gone --- CLAUDE.md | 66 ++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 56 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index c576559..ea9e2b6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -146,15 +146,24 @@ Only cash is excluded on payment method. Bank transfers *do* appear on a statement now that transaction accounts are imported, and NULL means unknown — both stay candidates, preserving the behaviour of every pre-existing row. -**Account-feed rows are excluded separately, and for a different reason** -(`awaitsStatementLine()`, added with the Frollo importer). A feed row is the -account's own ledger entry, not a receipt awaiting a statement line: the importer -only covers accounts whose statements are deliberately *not* imported. Without -the exclusion those ~600 rows a year sit in the queue forever and bury the -receipts that genuinely need a decision — measured, the queue went from 8 to 558. -It is scoped to a named list (`ACCOUNT_FEED_SOURCES`) rather than to -`source IS NOT NULL`, so a future source that *does* await a statement is not -swept up by it. +**Account-feed rows are NOT excluded, and the removed exclusion is worth +knowing about.** There was an `awaitsStatementLine()` predicate here, added with +the Frollo importer on the premise that a feed row is the account's own ledger +entry with no statement line coming. That premise was asserted and never tested. +It was false for almost every account: 422 of the first 550 imported rows already +had a statement twin. + +What matters is how it got in. The queue jumped from 8 to 558 the moment the feed +landed, and that jump was read as noise and filtered out. **The queue was right** — +those rows genuinely were provisional entries awaiting statement lines — and +filtering it removed the only mechanism that would ever have collapsed them, so +the duplicates became permanent instead of transient and stayed invisible until a +human saw one salary payment listed twice in two currencies. + +A feed row belongs in the queue. Queue volume is solved **upstream**, by not +importing rows the ledger already holds (`LEDGER_MATCH_DAYS`), never downstream +by hiding the ones that are there. If a genuinely statement-less feed is ever +added, give it its own predicate and prove the premise with a query first. ATM withdrawals stay categorised as spend rather than `transfers`. Treating them as transfers only works if every cash purchase is logged; with partial logging @@ -636,7 +645,44 @@ statements already cover them and are authoritative, and a second producer for the same rows would only create reconciliation work. - Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and - `src/lib/frollo-ingest.ts` (report + insert). 42 unit tests. + `src/lib/frollo-ingest.ts` (report + insert + the ledger-duplicate guard). + 53 unit tests across `frollo-csv.test.ts` and `frollo-ingest.test.ts`. + +**There are TWO de-duplications and they solve different problems.** `dedupe()` +compares the file against itself (CDR re-consent twins, below). The +ledger-duplicate guard compares the file against `transactions`, and its absence +is what made the first import write 422 duplicates out of 550 — $1,023,824.63 +counted twice — because "these accounts issue no statements" had been asserted +rather than queried. The check that would have caught it is one SQL statement. +The check run instead compared each row's date to the statement's min–max window, +which for accounts whose statements span 182 to 460 days swallows a year and +cannot distinguish covered from uncovered at all. + +The guard matches on **amount + direction, within `LEDGER_MATCH_DAYS` (3)**, and +consumes each ledger row once so a genuine repeat survives. Three details, each +of which was wrong first and each caught only by rehearsing against the real +database rather than fixtures: + +- **`pg` returns a DATE as a JS `Date`; Prisma and the CSV give strings.** + `String(date).slice(0, 10)` is `"Wed Mar 10"` → NaN → every comparison skipped, + silently. The first dry run reported 550 to insert against a ledger holding 422 + of them. `dayMs()` accepts both, and both shapes are tested because both + drivers are in use. +- **Direction must be in the key.** This ledger is full of internal transfers + between the owner's own accounts and the feed carries *both legs*: 2026-05-18 + has +3076.04 into ANZ and −3076.04 out of AMP. On amount alone the credit leg + consumed the ledger's debit row and the real duplicate was written — 20 rows. +- **`refund` is money in.** Ledger types are debit | credit | payment | fee | + interest | refund; the feed calls a reversed account fee a `credit` while the + statement importer types it `refund`. Classifying it as an outflow left every + ANZ servicing-fee reversal behind. + +Residual after all three: **4 rows / $15.01**, all sub-$5 account fees where +several identical amounts fall in overlapping windows and greedy consumption +picks the wrong one. Left alone deliberately at this scale. + +The count is printed by the CLI even when zero. A number you have to go looking +for is a number nobody looks at. - Two callers share that module so they cannot drift: `scripts/import-frollo.mts` (dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by the n8n `Frollo Import` workflow). The DB is reached through an injected