docs: record the ledger-duplicate guard and why awaitsStatementLine is gone
ci / lint-test (push) Successful in 40s
ci / lint-test (push) Successful in 40s
This commit is contained in:
@@ -146,15 +146,24 @@ Only cash is excluded on payment method. Bank transfers *do* appear on a
|
||||
statement now that transaction accounts are imported, and NULL means unknown —
|
||||
both stay candidates, preserving the behaviour of every pre-existing row.
|
||||
|
||||
**Account-feed rows are excluded separately, and for a different reason**
|
||||
(`awaitsStatementLine()`, added with the Frollo importer). A feed row is the
|
||||
account's own ledger entry, not a receipt awaiting a statement line: the importer
|
||||
only covers accounts whose statements are deliberately *not* imported. Without
|
||||
the exclusion those ~600 rows a year sit in the queue forever and bury the
|
||||
receipts that genuinely need a decision — measured, the queue went from 8 to 558.
|
||||
It is scoped to a named list (`ACCOUNT_FEED_SOURCES`) rather than to
|
||||
`source IS NOT NULL`, so a future source that *does* await a statement is not
|
||||
swept up by it.
|
||||
**Account-feed rows are NOT excluded, and the removed exclusion is worth
|
||||
knowing about.** There was an `awaitsStatementLine()` predicate here, added with
|
||||
the Frollo importer on the premise that a feed row is the account's own ledger
|
||||
entry with no statement line coming. That premise was asserted and never tested.
|
||||
It was false for almost every account: 422 of the first 550 imported rows already
|
||||
had a statement twin.
|
||||
|
||||
What matters is how it got in. The queue jumped from 8 to 558 the moment the feed
|
||||
landed, and that jump was read as noise and filtered out. **The queue was right** —
|
||||
those rows genuinely were provisional entries awaiting statement lines — and
|
||||
filtering it removed the only mechanism that would ever have collapsed them, so
|
||||
the duplicates became permanent instead of transient and stayed invisible until a
|
||||
human saw one salary payment listed twice in two currencies.
|
||||
|
||||
A feed row belongs in the queue. Queue volume is solved **upstream**, by not
|
||||
importing rows the ledger already holds (`LEDGER_MATCH_DAYS`), never downstream
|
||||
by hiding the ones that are there. If a genuinely statement-less feed is ever
|
||||
added, give it its own predicate and prove the premise with a query first.
|
||||
|
||||
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
|
||||
as transfers only works if every cash purchase is logged; with partial logging
|
||||
@@ -636,7 +645,44 @@ statements already cover them and are authoritative, and a second producer for
|
||||
the same rows would only create reconciliation work.
|
||||
|
||||
- Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and
|
||||
`src/lib/frollo-ingest.ts` (report + insert). 42 unit tests.
|
||||
`src/lib/frollo-ingest.ts` (report + insert + the ledger-duplicate guard).
|
||||
53 unit tests across `frollo-csv.test.ts` and `frollo-ingest.test.ts`.
|
||||
|
||||
**There are TWO de-duplications and they solve different problems.** `dedupe()`
|
||||
compares the file against itself (CDR re-consent twins, below). The
|
||||
ledger-duplicate guard compares the file against `transactions`, and its absence
|
||||
is what made the first import write 422 duplicates out of 550 — $1,023,824.63
|
||||
counted twice — because "these accounts issue no statements" had been asserted
|
||||
rather than queried. The check that would have caught it is one SQL statement.
|
||||
The check run instead compared each row's date to the statement's min–max window,
|
||||
which for accounts whose statements span 182 to 460 days swallows a year and
|
||||
cannot distinguish covered from uncovered at all.
|
||||
|
||||
The guard matches on **amount + direction, within `LEDGER_MATCH_DAYS` (3)**, and
|
||||
consumes each ledger row once so a genuine repeat survives. Three details, each
|
||||
of which was wrong first and each caught only by rehearsing against the real
|
||||
database rather than fixtures:
|
||||
|
||||
- **`pg` returns a DATE as a JS `Date`; Prisma and the CSV give strings.**
|
||||
`String(date).slice(0, 10)` is `"Wed Mar 10"` → NaN → every comparison skipped,
|
||||
silently. The first dry run reported 550 to insert against a ledger holding 422
|
||||
of them. `dayMs()` accepts both, and both shapes are tested because both
|
||||
drivers are in use.
|
||||
- **Direction must be in the key.** This ledger is full of internal transfers
|
||||
between the owner's own accounts and the feed carries *both legs*: 2026-05-18
|
||||
has +3076.04 into ANZ and −3076.04 out of AMP. On amount alone the credit leg
|
||||
consumed the ledger's debit row and the real duplicate was written — 20 rows.
|
||||
- **`refund` is money in.** Ledger types are debit | credit | payment | fee |
|
||||
interest | refund; the feed calls a reversed account fee a `credit` while the
|
||||
statement importer types it `refund`. Classifying it as an outflow left every
|
||||
ANZ servicing-fee reversal behind.
|
||||
|
||||
Residual after all three: **4 rows / $15.01**, all sub-$5 account fees where
|
||||
several identical amounts fall in overlapping windows and greedy consumption
|
||||
picks the wrong one. Left alone deliberately at this scale.
|
||||
|
||||
The count is printed by the CLI even when zero. A number you have to go looking
|
||||
for is a number nobody looks at.
|
||||
- Two callers share that module so they cannot drift: `scripts/import-frollo.mts`
|
||||
(dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by
|
||||
the n8n `Frollo Import` workflow). The DB is reached through an injected
|
||||
|
||||
Reference in New Issue
Block a user