docs: record the ledger-duplicate guard and why awaitsStatementLine is gone
ci / lint-test (push) Successful in 40s

This commit is contained in:
2026-08-13 12:24:59 +10:00
parent bb8a009e02
commit 2738213a23
+56 -10
View File
@@ -146,15 +146,24 @@ Only cash is excluded on payment method. Bank transfers *do* appear on a
statement now that transaction accounts are imported, and NULL means unknown — statement now that transaction accounts are imported, and NULL means unknown —
both stay candidates, preserving the behaviour of every pre-existing row. both stay candidates, preserving the behaviour of every pre-existing row.
**Account-feed rows are excluded separately, and for a different reason** **Account-feed rows are NOT excluded, and the removed exclusion is worth
(`awaitsStatementLine()`, added with the Frollo importer). A feed row is the knowing about.** There was an `awaitsStatementLine()` predicate here, added with
account's own ledger entry, not a receipt awaiting a statement line: the importer the Frollo importer on the premise that a feed row is the account's own ledger
only covers accounts whose statements are deliberately *not* imported. Without entry with no statement line coming. That premise was asserted and never tested.
the exclusion those ~600 rows a year sit in the queue forever and bury the It was false for almost every account: 422 of the first 550 imported rows already
receipts that genuinely need a decision — measured, the queue went from 8 to 558. had a statement twin.
It is scoped to a named list (`ACCOUNT_FEED_SOURCES`) rather than to
`source IS NOT NULL`, so a future source that *does* await a statement is not What matters is how it got in. The queue jumped from 8 to 558 the moment the feed
swept up by it. landed, and that jump was read as noise and filtered out. **The queue was right**
those rows genuinely were provisional entries awaiting statement lines — and
filtering it removed the only mechanism that would ever have collapsed them, so
the duplicates became permanent instead of transient and stayed invisible until a
human saw one salary payment listed twice in two currencies.
A feed row belongs in the queue. Queue volume is solved **upstream**, by not
importing rows the ledger already holds (`LEDGER_MATCH_DAYS`), never downstream
by hiding the ones that are there. If a genuinely statement-less feed is ever
added, give it its own predicate and prove the premise with a query first.
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
as transfers only works if every cash purchase is logged; with partial logging as transfers only works if every cash purchase is logged; with partial logging
@@ -636,7 +645,44 @@ statements already cover them and are authoritative, and a second producer for
the same rows would only create reconciliation work. the same rows would only create reconciliation work.
- Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and - Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and
`src/lib/frollo-ingest.ts` (report + insert). 42 unit tests. `src/lib/frollo-ingest.ts` (report + insert + the ledger-duplicate guard).
53 unit tests across `frollo-csv.test.ts` and `frollo-ingest.test.ts`.
**There are TWO de-duplications and they solve different problems.** `dedupe()`
compares the file against itself (CDR re-consent twins, below). The
ledger-duplicate guard compares the file against `transactions`, and its absence
is what made the first import write 422 duplicates out of 550 — $1,023,824.63
counted twice — because "these accounts issue no statements" had been asserted
rather than queried. The check that would have caught it is one SQL statement.
The check run instead compared each row's date to the statement's minmax window,
which for accounts whose statements span 182 to 460 days swallows a year and
cannot distinguish covered from uncovered at all.
The guard matches on **amount + direction, within `LEDGER_MATCH_DAYS` (3)**, and
consumes each ledger row once so a genuine repeat survives. Three details, each
of which was wrong first and each caught only by rehearsing against the real
database rather than fixtures:
- **`pg` returns a DATE as a JS `Date`; Prisma and the CSV give strings.**
`String(date).slice(0, 10)` is `"Wed Mar 10"` → NaN → every comparison skipped,
silently. The first dry run reported 550 to insert against a ledger holding 422
of them. `dayMs()` accepts both, and both shapes are tested because both
drivers are in use.
- **Direction must be in the key.** This ledger is full of internal transfers
between the owner's own accounts and the feed carries *both legs*: 2026-05-18
has +3076.04 into ANZ and 3076.04 out of AMP. On amount alone the credit leg
consumed the ledger's debit row and the real duplicate was written — 20 rows.
- **`refund` is money in.** Ledger types are debit | credit | payment | fee |
interest | refund; the feed calls a reversed account fee a `credit` while the
statement importer types it `refund`. Classifying it as an outflow left every
ANZ servicing-fee reversal behind.
Residual after all three: **4 rows / $15.01**, all sub-$5 account fees where
several identical amounts fall in overlapping windows and greedy consumption
picks the wrong one. Left alone deliberately at this scale.
The count is printed by the CLI even when zero. A number you have to go looking
for is a number nobody looks at.
- Two callers share that module so they cannot drift: `scripts/import-frollo.mts` - Two callers share that module so they cannot drift: `scripts/import-frollo.mts`
(dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by (dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by
the n8n `Frollo Import` workflow). The DB is reached through an injected the n8n `Frollo Import` workflow). The DB is reached through an injected