frollo: don't re-import what the ledger already has, and stop hiding the queue
ci / lint-test (push) Successful in 44s

The first import wrote 550 rows on 2026-08-13. 422 of them (77%) were second
copies of transactions the ledger already held from statements — $1,023,824.63
of movement counted twice. The owner found it by opening the transactions view
and seeing one HDR Global salary listed twice, once as A$15,518.53 from the
statement and once as US$10,782.00 from the feed.

The currency was never the defect. toLedgerRow already left amount_aud NULL and
named the currency in foreign_currency_code, which is the documented contract for
a row whose AUD value is unknown, and the transactions page labels it. What made
it look wrong was the duplicate sitting beside it.

Two changes.

Upstream, ingestFrolloCsv now drops rows the ledger already holds, matching on
amount + direction within LEDGER_MATCH_DAYS (3). Three things had to be right and
the first two were not, each caught only by rehearsing against real data rather
than fixtures:

  - pg returns a DATE as a JS Date while Prisma and the CSV give strings.
    String(date).slice(0,10) is "Wed Mar 10", which parses to NaN, so the first
    dry run reported 550 to insert and zero duplicates. dayMs() takes both.
  - Direction has to be in the key. This ledger is full of internal transfers
    between the owner's own accounts and the feed carries both legs: 2026-05-18
    has +3076.04 into ANZ and -3076.04 out of AMP. Matching on amount alone let
    the credit leg consume the ledger's debit row, so the real duplicate was
    written — 20 rows got in that way.
  - 'refund' is money in. The feed calls a reversed account fee a credit and the
    statement importer types it 'refund'; classifying it as an outflow left every
    ANZ servicing-fee reversal behind.

Downstream, awaitsStatementLine() is removed. Its premise — feed rows never await
a statement line — was asserted, never tested, and false for almost every
account. Worse is how it got there: the reconcile queue jumped 8 -> 558 when the
feed landed, that jump was read as noise and filtered away, and filtering it
removed the only mechanism that would ever have collapsed the duplicates. The
queue was right. A feed row IS a row awaiting its statement line.

Re-imported: 375 dropped as already-on-ledger, 175 inserted. Residual duplicates
4 rows / $15.01, all sub-$5 account fees where several identical amounts fall in
overlapping windows and greedy consumption picks the wrong one; not chased
further at this scale.

The CLI prints the already-on-ledger count even when zero — a number you have to
go looking for is a number nobody looks at.
This commit is contained in:
2026-08-13 12:23:01 +10:00
parent 3e826a317b
commit bb8a009e02
4 changed files with 337 additions and 19 deletions
+5 -1
View File
@@ -25,7 +25,7 @@
*/
import { readFileSync } from "node:fs";
import pg from "pg";
import { ingestFrolloCsv, type SqlExecutor } from "../src/lib/frollo-ingest.ts";
import { ingestFrolloCsv, LEDGER_MATCH_DAYS, type SqlExecutor } from "../src/lib/frollo-ingest.ts";
function arg(name: string, fallback?: string): string | undefined {
const i = process.argv.indexOf(`--${name}`);
@@ -99,6 +99,10 @@ console.log();
console.log("LEDGER");
console.log(` already imported : ${String(report.alreadyImported).padStart(5)}`);
// Printed even when zero. This is the count that was missing on 2026-08-13,
// when the whole file was written on top of a ledger that already held 77% of
// it; a number you have to go looking for is a number nobody looks at.
console.log(` already on ledger: ${String(report.ledgerDuplicates).padStart(5)} (statement twin within ${LEDGER_MATCH_DAYS} days)`);
console.log(` new to insert : ${String(report.toInsert).padStart(5)}`);
if (report.anomalies.length > 0) {