frollo: don't import what the statements already cover
ci / lint-test (push) Successful in 45s

"We should not be importing from Frollo what we already have from statements"
(owner). The amount+direction guard could not deliver that, because the two
sources decompose the same event differently: Frollo bundles the Wise fee into
the transfer (10001.13) where the statement itemises it (10000.00 + 1.13). So 38
Wise USD rows passed the amount guard as new while being the same money, and were
the reason the transactions view showed a USD figure where every neighbouring row
showed AUD.

A statement's billing_end_date is a hard watermark: everything on that account up
to that date is already in the ledger, itemised and converted. Guard 1 now drops
any feed row at or before its account's newest statement. Guard 2 (amount +
direction) stays as the net for accounts that have no statement at all.

Note this is the coverage test the first import needed and got wrong. That one
asked whether a row's date fell inside a statement's min-max window, which for
periods spanning 182 to 460 days swallows a year and answers nothing. The
watermark asks a question that has an answer: up to what date is this account
complete?

Matching is on last4, verified against the live statement set — the eight
in-scope accounts with statements each map to one bank, no cross-bank collision.
The watermarks are printed by the CLI so a wrong boundary is visible rather than
inferred from a row count.

Re-imported from empty: 425 covered by statement, 15 amount twins, 110 inserted.
Exactly one row now carries a foreign currency with no AUD figure — the
2026-08-12 HDR salary, which is the genuinely-new pre-statement row this feed
exists for. Was 39.
This commit is contained in:
2026-08-13 12:36:08 +10:00
parent afd75d3f09
commit b82c4570bd
3 changed files with 133 additions and 22 deletions
+41 -1
View File
@@ -46,9 +46,13 @@ const csv = (...lines: string[]) => [HEADER, ...lines].join("\n");
* which is crude but keeps the test free of a live connection — and the guard
* is pure logic over what comes back.
*/
function exec(prior: { transaction_date: string; amount: string; transaction_type?: string }[]): SqlExecutor {
function exec(
prior: { transaction_date: string; amount: string; transaction_type?: string }[],
coverage: { last4: string; covered_to: string }[] = []
): SqlExecutor {
return (async (sql: string) => {
if (sql.includes("WHERE source = $1")) return [];
if (sql.includes("FROM statements")) return coverage;
if (sql.includes("superseded_by_id IS NULL")) return prior.map((p) => ({ transaction_type: "debit", ...p }));
return [];
}) as SqlExecutor;
@@ -162,6 +166,42 @@ describe("ledger-duplicate guard", () => {
expect(r.toInsert).toBe(0);
});
it("drops anything a statement already covers, whatever the amount", async () => {
// The primary guard. Amount-matching cannot see this case: the two sources
// decompose the same event differently — Frollo bundles the Wise fee into
// the transfer (10001.13) where the statement itemises it (10000.00 + 1.13)
// — so 38 Wise USD rows survived the amount guard while being the same
// money. A statement's billing_end_date is a hard watermark instead.
const r = await ingestFrolloCsv(
csv(
line({ id: "1", desc: "Interactive Brokers LLC", amount: "-10001.13", date: "2026-07-25" }),
line({ id: "2", desc: "HDR Global Services", amount: "10782.00", date: "2026-08-12" })
),
exec([], [{ last4: "1910", covered_to: "2026-07-26" }])
);
expect(r.coveredByStatement).toBe(1);
expect(r.toInsert).toBe(1); // only the post-watermark row survives
expect(r.statementWatermarks).toEqual([{ last4: "1910", coveredTo: "2026-07-26" }]);
});
it("keeps everything for an account that has no statement at all", async () => {
const r = await ingestFrolloCsv(
csv(line({ id: "1", amount: "-42.50", date: "2020-01-01" })),
exec([], [{ last4: "9999", covered_to: "2026-07-26" }])
);
expect(r.coveredByStatement).toBe(0);
expect(r.toInsert).toBe(1);
});
it("treats the watermark date itself as covered", async () => {
const r = await ingestFrolloCsv(
csv(line({ id: "1", amount: "-42.50", date: "2026-07-26" })),
exec([], [{ last4: "1910", covered_to: "2026-07-26" }])
);
expect(r.coveredByStatement).toBe(1);
expect(r.toInsert).toBe(0);
});
it("keeps everything when the ledger is empty", async () => {
const r = await ingestFrolloCsv(
csv(