Import Frollo account feeds for the accounts statements don't cover

Credit cards keep arriving as monthly statements and stay the source of
truth for them. This covers the other fourteen accounts, whose statements
arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise
including the income account, Up, ING, and the small transaction accounts.
About 50 rows a month, where the alternative is downloading each statement
by hand.

The file needs three defences, all found by diffing three real exports
(smarthome DECISIONS.md ING-11):

A CDR re-consent makes Frollo re-ingest an account's whole history under
fresh transaction ids while the originals survive, and consents expire
annually. On this export 112 rows were such twins, and every HDR salary
payment appeared twice — importing blind doubles reported income. dedupe()
collapses each natural-key group to its lowest id, lowest because old ids
were a strict subset of new across two exports, so source_ref stays stable
and a re-import inserts nothing.

An earlier version of that rule kept close-id rows on a 10,000 threshold,
reasoning that genuine same-day repeats have consecutive ids. Verifying it
against the income rows killed it: in-scope duplicate pairs have id gaps
from 58 to 260 million, so no threshold separates them from the gaps of 1-4
that real repeats showed. It now collapses unconditionally and flags
anything within 10 for review — the errors are asymmetric, and nothing in
scope has ever tripped the flag.

A lapsed consent removes an account from the export silently, with no error
and no marker; the row count just drops. So the import asserts the account
roster and refuses to run when a configured account contributes nothing.

Also holds these rows out of the pending-reconciliation queue. A feed row is
the account's own ledger entry, not a receipt awaiting a statement line —
these accounts' statements are deliberately not imported — so without the
exclusion 550 rows a year would bury the receipts that need a decision. The
queue stays at 8 instead of 558.

Foreign rows follow order-ingestion's existing shape: amount is the native
figure, foreign_currency_code names it, amount_aud stays NULL rather than
asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as
incomplete.

Dry run by default. Verified against the real export before applying:
550 rows inserted, 14 accounts, re-run inserts 0.

src/__tests__/unit/frollo-csv.test.ts

@@ -0,0 +1,309 @@
+import { describe, it, expect } from "vitest";
+import { parseCSVRows } from "@/lib/csv-parser";
+import {
+  ACCOUNTS,
+  SUSPECT_REPEAT_GAP,
+  accountFor,
+  accountReport,
+  dedupe,
+  last4,
+  naturalKey,
+  readRows,
+  skipReason,
+  toLedgerRow,
+  unknownAccounts,
+  type FrolloRow,
+} from "@/lib/frollo-csv";
+
+const HEADER =
+  "transaction_id,description,user_description,amount,currency,transaction_date,posted_date," +
+  "account_number,account_name,credit_debit,transaction_type,provider_name,merchant_name," +
+  "budget_category,category_name,user_tags,notes,included";
+
+function csv(...lines: string[]): FrolloRow[] {
+  return readRows(parseCSVRows([HEADER, ...lines].join("\n")));
+}
+
+/** Shapes a line in the export's real column order. */
+function line(o: {
+  id: string;
+  desc?: string;
+  amount: string;
+  currency?: string;
+  date?: string;
+  posted?: string;
+  account?: string;
+  accountName?: string;
+  cd?: string;
+  type?: string;
+  merchant?: string;
+}): string {
+  return [
+    o.id,
+    o.desc ?? "SOME MERCHANT",
+    "",
+    o.amount,
+    o.currency ?? "AUD",
+    o.date ?? "2026-03-01",
+    o.posted ?? "",
+    o.account ?? "xxx-xxx xxxx1910",
+    o.accountName ?? "Smart Access",
+    o.cd ?? (o.amount.startsWith("-") ? "debit" : "credit"),
+    o.type ?? "payment",
+    "CommBank",
+    o.merchant ?? "Some Merchant",
+    "lifestyle",
+    "Groceries",
+    '""',
+    "",
+    "true",
+  ].join(",");
+}
+
+describe("readRows", () => {
+  it("maps by header name, not position", () => {
+    const rows = csv(line({ id: "1", amount: "-12.34", desc: "COFFEE" }));
+    expect(rows).toHaveLength(1);
+    expect(rows[0]).toMatchObject({
+      transactionId: "1",
+      description: "COFFEE",
+      amount: -12.34,
+      currency: "AUD",
+      accountName: "Smart Access",
+    });
+  });
+
+  it("throws when a required column is missing rather than reading undefined", () => {
+    const broken = "transaction_id,description,amount\n1,X,-1.00";
+    expect(() => readRows(parseCSVRows(broken))).toThrow(/missing expected column/i);
+  });
+
+  it("keeps a quoted description containing commas intact", () => {
+    const desc = '"DIDI_NZ DidiNZ pending   Auckland    NZL, ##0626           8.77 NZ DOLLAR"';
+    const rows = csv(line({ id: "1", amount: "-7.23", desc }));
+    expect(rows[0].description).toContain("NZL, ##0626");
+  });
+
+  it("treats an absent posted_date as null, not empty string", () => {
+    expect(csv(line({ id: "1", amount: "-1.00" }))[0].postedDate).toBeNull();
+  });
+});
+
+describe("account scoping", () => {
+  it("matches on the last four digits regardless of masking format", () => {
+    expect(last4("xxx-xxx xxxx1910")).toBe("1910");
+    expect(last4("xxxxxx xxxxx6264")).toBe("6264");
+    expect(last4("x xxxx 0887  ")).toBe("0887");
+    expect(last4("XXXXXXXXXXXX0351")).toBe("0351");
+  });
+
+  it("has no duplicate last4 among configured accounts", () => {
+    const keys = ACCOUNTS.map((a) => a.last4);
+    expect(new Set(keys).size).toBe(keys.length);
+  });
+
+  it("excludes the credit cards, which statements already cover", () => {
+    for (const card of ["3893", "8032", "1000", "0351", "6227"]) {
+      expect(ACCOUNTS.find((a) => a.last4 === card)).toBeUndefined();
+    }
+  });
+
+  it("recognises the loan and income accounts", () => {
+    expect(accountFor("xxx-xxx xxxxx0682")?.label).toMatch(/loan/i);
+    expect(accountFor("xxxxxxxxxxxx4830")?.label).toMatch(/income/i);
+  });
+});
+
+describe("skipReason", () => {
+  it("skips a zero-amount fee waiver", () => {
+    const row = csv(
+      line({ id: "1", amount: "0.0", desc: "INTNL TRANSACTION FEE, $ 41.58 FEE SAV" })
+    )[0];
+    expect(skipReason(row)).toBe("zero_amount");
+  });
+
+  it("skips a credit-card row", () => {
+    const row = csv(line({ id: "1", amount: "-50.00", account: "xxxx xxxx xxxx 3893" }))[0];
+    expect(skipReason(row)).toBe("out_of_scope_account");
+  });
+
+  it("skips a Westpac pending row on its description prefix", () => {
+    const row = csv(
+      line({ id: "1", amount: "-13.50", desc: "Pending - Breadworld Bakery Cafe", account: "xxx-xxx xx4878" })
+    )[0];
+    expect(skipReason(row)).toBe("pending");
+  });
+
+  it("accepts an ordinary in-scope row", () => {
+    expect(skipReason(csv(line({ id: "1", amount: "-29.17" }))[0])).toBeNull();
+  });
+
+  it("skips rather than imports an unparseable amount", () => {
+    const row = csv(line({ id: "1", amount: "n/a" }))[0];
+    expect(skipReason(row)).toBe("unparseable_amount");
+  });
+});
+
+describe("dedupe", () => {
+  it("collapses a re-ingest twin to the earlier id", () => {
+    const rows = csv(
+      line({ id: "1200000000", amount: "-61.03", desc: "AMAZON MARKETPLACE" }),
+      line({ id: "1326000000", amount: "-61.03", desc: "AMAZON MARKETPLACE" })
+    );
+    const { kept, dropped } = dedupe(rows);
+    expect(kept).toHaveLength(1);
+    expect(kept[0].transactionId).toBe("1200000000");
+    expect(dropped).toHaveLength(1);
+    expect(dropped[0].duplicateOf.transactionId).toBe("1200000000");
+  });
+
+  it("collapses close-id repeats too, but flags them for review", () => {
+    // Five identical $1.00 Apple charges with consecutive ids. On a card
+    // account, so out of scope in practice — but if such a group ever reaches
+    // dedupe it must be surfaced, not silently decided either way.
+    const rows = csv(
+      ...["1326128755", "1326128756", "1326128757", "1326128758", "1326128759"].map((id) =>
+        line({ id, amount: "-1.0", desc: "APPLE.COM/BILL SYDNEY", date: "2026-07-28" })
+      )
+    );
+    const { kept, dropped } = dedupe(rows);
+    expect(kept).toHaveLength(1);
+    expect(dropped).toHaveLength(4);
+    expect(dropped.every((d) => d.suspectGenuineRepeat)).toBe(true);
+  });
+
+  it("does not flag a far-apart twin as a possible repeat", () => {
+    const rows = csv(
+      line({ id: "1101410977", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830" }),
+      line({ id: "1101415475", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830" })
+    );
+    const { kept, dropped } = dedupe(rows);
+    expect(kept).toHaveLength(1);
+    expect(dropped[0].suspectGenuineRepeat).toBe(false);
+    expect(dropped[0].idGap).toBe(4498);
+  });
+
+  it("collapses a real income twin — the 100%-overstatement case", () => {
+    // Every HDR salary payment appears twice in every export, id gaps ranging
+    // from 58 to 79 million. Keeping both doubled reported income.
+    const rows = csv(
+      line({ id: "1126624097", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830", desc: "HDR Global Services" }),
+      line({ id: "1126624242", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830", desc: "HDR Global Services" })
+    );
+    const { kept } = dedupe(rows);
+    expect(kept).toHaveLength(1);
+    expect(kept[0].transactionId).toBe("1126624097");
+  });
+
+  it("does not merge rows that differ in amount, date, description or account", () => {
+    const rows = csv(
+      line({ id: "1000000001", amount: "-10.00" }),
+      line({ id: "1326000001", amount: "-10.01" }),
+      line({ id: "1326000002", amount: "-10.00", date: "2026-03-02" }),
+      line({ id: "1326000003", amount: "-10.00", desc: "OTHER" }),
+      line({ id: "1326000004", amount: "-10.00", account: "xxx-xxx xx4878" })
+    );
+    expect(dedupe(rows).kept).toHaveLength(5);
+  });
+
+  it("keeps the lowest id, because old ids survive a re-ingest", () => {
+    const rows = csv(
+      line({ id: "1326000001", amount: "-5.00" }),
+      line({ id: "1000000000", amount: "-5.00" }),
+      line({ id: "1200000000", amount: "-5.00" })
+    );
+    const { kept } = dedupe(rows);
+    expect(kept).toHaveLength(1);
+    expect(kept[0].transactionId).toBe("1000000000");
+  });
+
+  it("reports the id gap so a suspect group can be reviewed", () => {
+    const near = csv(
+      line({ id: "1000000000", amount: "-5.00" }),
+      line({ id: String(1000000000 + SUSPECT_REPEAT_GAP - 1), amount: "-5.00" })
+    );
+    expect(dedupe(near).dropped[0].suspectGenuineRepeat).toBe(true);
+    const far = csv(
+      line({ id: "1000000000", amount: "-5.00" }),
+      line({ id: String(1000000000 + SUSPECT_REPEAT_GAP), amount: "-5.00" })
+    );
+    expect(dedupe(far).dropped[0].suspectGenuineRepeat).toBe(false);
+  });
+
+  it("is idempotent", () => {
+    const rows = csv(
+      line({ id: "1000000001", amount: "-61.03" }),
+      line({ id: "1326000001", amount: "-61.03" })
+    );
+    const once = dedupe(rows).kept;
+    expect(dedupe(once).kept).toEqual(once);
+  });
+});
+
+describe("naturalKey", () => {
+  it("is stable across masking differences on the same account", () => {
+    const a = csv(line({ id: "1", amount: "-1.00", account: "xxx-xxx xxxx1910" }))[0];
+    const b = csv(line({ id: "2", amount: "-1.00", account: "xx-xxx xxxxxx1910" }))[0];
+    expect(naturalKey(a)).toBe(naturalKey(b));
+  });
+});
+
+describe("toLedgerRow", () => {
+  it("stores a debit as a positive amount with a direction", () => {
+    const r = toLedgerRow(csv(line({ id: "1", amount: "-29.17" }))[0]);
+    expect(r.amount).toBe(29.17);
+    expect(r.transactionType).toBe("debit");
+  });
+
+  it("marks a positive amount as a credit", () => {
+    const r = toLedgerRow(csv(line({ id: "1", amount: "302.70" }))[0]);
+    expect(r.amount).toBe(302.7);
+    expect(r.transactionType).toBe("credit");
+  });
+
+  it("leaves a foreign row's amount native and names the currency", () => {
+    const r = toLedgerRow(
+      csv(line({ id: "1", amount: "-10001.13", currency: "USD", account: "xxxxxxxxxxxx4830" }))[0]
+    );
+    expect(r.amount).toBe(10001.13);
+    expect(r.foreignCurrencyCode).toBe("USD");
+    expect(r.foreignCurrencyAmount).toBe(10001.13);
+  });
+
+  it("carries no foreign fields for an AUD row", () => {
+    const r = toLedgerRow(csv(line({ id: "1", amount: "-5.00" }))[0]);
+    expect(r.foreignCurrencyCode).toBeNull();
+    expect(r.foreignCurrencyAmount).toBeNull();
+  });
+
+  it("drops Frollo's literal 'Unknown' merchant rather than showing it", () => {
+    const r = toLedgerRow(csv(line({ id: "1", amount: "-5.00", merchant: "Unknown" }))[0]);
+    expect(r.merchantName).toBeNull();
+  });
+
+  it("keeps a real merchant", () => {
+    const r = toLedgerRow(csv(line({ id: "1", amount: "-5.00", merchant: "Solar Victoria" }))[0]);
+    expect(r.merchantName).toBe("Solar Victoria");
+  });
+
+  it("carries the provider's id as source_ref for idempotency", () => {
+    const r = toLedgerRow(csv(line({ id: "986846710", amount: "-5.00" }))[0]);
+    expect(r.sourceRef).toBe("986846710");
+  });
+});
+
+describe("accountReport / unknownAccounts", () => {
+  it("reports a configured account with no rows as absent — a lapsed consent", () => {
+    const rows = csv(line({ id: "1", amount: "-5.00", account: "xxx-xxx xxxx1910" }));
+    const report = accountReport(rows);
+    expect(report.find((r) => r.spec.last4 === "1910")?.present).toBe(true);
+    expect(report.find((r) => r.spec.last4 === "4830")?.present).toBe(false);
+  });
+
+  it("surfaces an account in the file that the importer does not know", () => {
+    const rows = csv(line({ id: "1", amount: "-5.00", account: "xxxx xxxx xxxx 9999", accountName: "New Card" }));
+    const unknown = unknownAccounts(rows);
+    expect(unknown).toHaveLength(1);
+    expect(unknown[0]).toMatchObject({ accountName: "New Card", rows: 1 });
+  });
+});