csv import: exclude what the statements already cover, and delete the Frollo importer
ci / lint-test (push) Successful in 43s
ci / lint-test (push) Successful in 43s
"This is becoming too complex... Frollo should be done through that [the manual
CSV import]" (owner). It was right: a bespoke importer, an API route, a CLI, two
scheduled n8n workflows and a shared secret existed to do what the CSV import
modal already did, minus one rule.
That rule is statement coverage, and it turns out to be the whole thing. Applying
each account's newest billing_end_date as a watermark takes the real 2,607-row
Frollo export down to 171 rows — with no account allowlist, no credit-card
exclusion and no Frollo-specific scoping at all. Cards drop out on their own
because their statements are current; the 46 card rows that survive are genuinely
post-statement. Every bit of the bespoke apparatus was doing by hand what one
query does generically.
Deleted: src/lib/frollo-csv.ts, src/lib/frollo-ingest.ts, scripts/import-frollo.mts,
src/app/api/frollo/, both test files, the FROLLO_INGEST_TOKEN wiring, and the n8n
Frollo Import + Frollo Freshness Check workflows.
Added to the shared CSV path, so every import benefits:
- getStatementCoverage() + /api/import/statement-coverage. The review step
leaves out rows an account's statements already cover and says how many, with
the rows one click away. Only applies when an account column is mapped and
that account has statements — a row is never dropped on a guess.
- Optional Account and Row ID columns in the mapper. Account drives the
watermark and is stored as source_account; Row ID becomes source_ref.
- An in-file duplicate warning. A CDR re-consent re-exports history under
fresh ids, so source_ref cannot see it — 385 twins in one 2,563-row export
doubled every salary payment, and that is not visible by eye in a review
table.
Two pre-existing bugs in that path, both of which this plan depends on:
- The category chosen in the review step was accepted by
batchInsertCSVTransactions and then left out of the INSERT column list, so it
was silently discarded and the trigger wrote 'other'. Not cosmetic: an
uncategorised credit is admitted by NET_SPEND_ROWS and negated by
SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of spend while
counting as no income.
- The path had no idempotency whatsoever. row_index is assigned MAX+1 on every
run, which makes uq_transaction_identity structurally unable to fire, so a
second import of the same file duplicated all of it. Now writes source +
source_ref with ON CONFLICT DO NOTHING.
awaitsStatementLine()'s removal note is kept but rewritten: it no longer points
at a deleted file, and the lesson stands — the queue jump from 8 to 558 was the
measurement, not the noise.
This commit is contained in:
@@ -1,127 +0,0 @@
|
||||
/**
|
||||
* Imports a Frollo transaction export into `transactions`.
|
||||
*
|
||||
* node --experimental-strip-types scripts/import-frollo.mts --file <csv>
|
||||
* node --experimental-strip-types scripts/import-frollo.mts --file <csv> --apply
|
||||
*
|
||||
* Dry run by default: it prints exactly what it would do and touches nothing.
|
||||
* Rehearse first — the failure mode here is doubling reported income, and the
|
||||
* de-duplication rule that shipped second only survived because a dry run was
|
||||
* read against the salary rows.
|
||||
*
|
||||
* Export the file with PENDING TRANSACTIONS EXCLUDED. A pending row changes both
|
||||
* its id and its description when it settles, so importing one guarantees a
|
||||
* duplicate on the next run; every pending row observed has been on a credit
|
||||
* card, which this importer does not cover anyway. See DECISIONS.md ING-11.
|
||||
*
|
||||
* All parsing, scoping, de-duplication and insertion live in
|
||||
* `src/lib/frollo-ingest.ts`, shared with POST /api/frollo/ingest so the manual
|
||||
* and automatic paths cannot drift.
|
||||
*
|
||||
* Needs DATABASE_URL. postgres-personal publishes no host port, so from the host:
|
||||
* export DATABASE_URL="postgresql://personal:<pw>@$(docker inspect postgres-personal \
|
||||
* --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'):5432/personal"
|
||||
* The container IP changes on every recreate.
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import pg from "pg";
|
||||
import { ingestFrolloCsv, LEDGER_MATCH_DAYS, type SqlExecutor } from "../src/lib/frollo-ingest.ts";
|
||||
|
||||
function arg(name: string, fallback?: string): string | undefined {
|
||||
const i = process.argv.indexOf(`--${name}`);
|
||||
if (i >= 0 && process.argv[i + 1] && !process.argv[i + 1].startsWith("--")) return process.argv[i + 1];
|
||||
return fallback;
|
||||
}
|
||||
const has = (name: string) => process.argv.includes(`--${name}`);
|
||||
|
||||
const file = arg("file");
|
||||
if (!file) {
|
||||
console.error("usage: --file <csv> [--apply] [--force] [--owner <id>]");
|
||||
process.exit(2);
|
||||
}
|
||||
if (!process.env.DATABASE_URL) {
|
||||
console.error("DATABASE_URL is not set — cannot compare against the ledger.");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const money = (n: number) => n.toLocaleString("en-AU", { minimumFractionDigits: 2, maximumFractionDigits: 2 });
|
||||
|
||||
const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
|
||||
await client.connect();
|
||||
const exec: SqlExecutor = async <T,>(sql: string, params: unknown[]) =>
|
||||
(await client.query(sql, params)).rows as T[];
|
||||
|
||||
let report;
|
||||
try {
|
||||
report = await ingestFrolloCsv(readFileSync(file, "utf8"), exec, {
|
||||
ownerId: Number(arg("owner", "1")),
|
||||
apply: has("apply"),
|
||||
force: has("force"),
|
||||
});
|
||||
} finally {
|
||||
// Left open, the process hangs after an exception and looks like a slow query.
|
||||
if (!has("keep-open")) await client.end().catch(() => {});
|
||||
}
|
||||
|
||||
console.log(`\n${file}`);
|
||||
console.log(` ${report.totalRows} rows in file\n`);
|
||||
|
||||
console.log("SCOPE");
|
||||
for (const [why, n] of Object.entries(report.skipped).sort((a, b) => b[1]! - a[1]!)) {
|
||||
console.log(` skipped ${String(n).padStart(5)} ${why}`);
|
||||
}
|
||||
console.log(` in scope ${String(report.inScope).padStart(4)}\n`);
|
||||
|
||||
console.log("ACCOUNTS");
|
||||
for (const a of report.accounts) {
|
||||
console.log(
|
||||
` ${a.present ? " " : "!"} ${a.spec.last4} ${a.spec.label.padEnd(24)} ${String(a.rows).padStart(5)} rows`
|
||||
);
|
||||
}
|
||||
if (report.unknownAccounts.length > 0) {
|
||||
console.log("\n accounts in the file this importer does not know (skipped):");
|
||||
for (const u of report.unknownAccounts) {
|
||||
console.log(` ${u.accountNumber} ${u.accountName} ${u.rows} rows`);
|
||||
}
|
||||
}
|
||||
console.log();
|
||||
|
||||
console.log("DE-DUPLICATION (re-ingest twins from CDR re-consent)");
|
||||
console.log(` ${report.duplicatesDropped} dropped`);
|
||||
if (report.suspectRepeats.length > 0) {
|
||||
console.log(`\n REVIEW: ${report.suspectRepeats.length} collapsed row(s) had near-consecutive ids`);
|
||||
console.log(" and may be real repeats rather than duplicates:");
|
||||
for (const s of report.suspectRepeats.slice(0, 10)) {
|
||||
console.log(` ${s.transactionDate} ${money(s.amount).padStart(11)} ${s.description.slice(0, 40)} (gap ${s.idGap})`);
|
||||
}
|
||||
}
|
||||
console.log();
|
||||
|
||||
console.log("LEDGER");
|
||||
console.log(` already imported : ${String(report.alreadyImported).padStart(5)}`);
|
||||
// Printed even when zero. These are the counts that were missing on
|
||||
// 2026-08-13, when the whole file was written on top of a ledger that already
|
||||
// held 77% of it; a number you have to go looking for is a number nobody looks
|
||||
// at. The watermarks are printed too, so a wrong coverage boundary is visible
|
||||
// rather than inferred from a row count.
|
||||
console.log(` covered by stmt : ${String(report.coveredByStatement).padStart(5)} (dated on or before the account's newest statement)`);
|
||||
console.log(` amount twin : ${String(report.ledgerDuplicates).padStart(5)} (same amount + direction within ${LEDGER_MATCH_DAYS} days)`);
|
||||
if (report.statementWatermarks.length > 0) {
|
||||
console.log(" statements cover:");
|
||||
for (const w of report.statementWatermarks) console.log(` ${w.last4} up to ${w.coveredTo}`);
|
||||
}
|
||||
console.log(` new to insert : ${String(report.toInsert).padStart(5)}`);
|
||||
|
||||
if (report.anomalies.length > 0) {
|
||||
console.log("\nANOMALIES");
|
||||
for (const a of report.anomalies) console.log(` ! ${a}`);
|
||||
}
|
||||
|
||||
if (report.applied) {
|
||||
console.log(`\nInserted ${report.inserted} row(s).\n`);
|
||||
} else if (report.anomalies.length > 0 && has("apply")) {
|
||||
console.log("\nNOT APPLIED — anomalies above. Read them, then re-run with --force.\n");
|
||||
process.exit(1);
|
||||
} else {
|
||||
console.log("\nDRY RUN — nothing written. Re-run with --apply to insert.\n");
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
applyMapping,
|
||||
inFileDuplicates,
|
||||
last4,
|
||||
splitByCoverage,
|
||||
type ColumnMapping,
|
||||
type ParsedTransaction,
|
||||
} from "@/lib/csv-parser";
|
||||
|
||||
/**
|
||||
* The statement-coverage rule, which replaced a whole Frollo-specific importer.
|
||||
*
|
||||
* The first aggregator import wrote 422 duplicate rows out of 550 — $1,023,824
|
||||
* of movement counted twice — because "these accounts issue no statements" was
|
||||
* asserted rather than queried. Amount-matching could not fix it either: the two
|
||||
* sources decompose the same event differently, bundling a transfer fee where
|
||||
* the statement itemises it.
|
||||
*
|
||||
* A statement's end date answers the question that actually has an answer: up
|
||||
* to what date is this account complete? On the real 2026-08-13 export it takes
|
||||
* 2,607 rows down to 171, with no account allowlist and no card exclusions.
|
||||
*/
|
||||
|
||||
const tx = (o: Partial<ParsedTransaction> & { date: string }): ParsedTransaction => ({
|
||||
description: "SOMETHING",
|
||||
amount: 10,
|
||||
transaction_type: "debit",
|
||||
...o,
|
||||
});
|
||||
|
||||
describe("last4", () => {
|
||||
it("reads the same account written four different ways", () => {
|
||||
// Every one of these forms appears in the real data.
|
||||
expect(last4("xxxxxxxxxxxx2176")).toBe("2176");
|
||||
expect(last4("235242176")).toBe("2176");
|
||||
expect(last4("xxx-xxx xx4878")).toBe("4878");
|
||||
expect(last4("4085-56264")).toBe("6264");
|
||||
});
|
||||
|
||||
it("returns empty for an unusable identifier rather than guessing", () => {
|
||||
expect(last4("N/A")).toBe("");
|
||||
expect(last4("12")).toBe("");
|
||||
expect(last4("")).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("splitByCoverage", () => {
|
||||
const coverage = [{ last4: "2176", coveredTo: "2026-07-26" }];
|
||||
|
||||
it("excludes a row on or before the account's newest statement", () => {
|
||||
const { keep, covered } = splitByCoverage(
|
||||
[tx({ date: "2026-07-25", account: "xxxxxxxxxxxx2176" })],
|
||||
coverage
|
||||
);
|
||||
expect(covered).toHaveLength(1);
|
||||
expect(keep).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("treats the watermark date itself as covered", () => {
|
||||
const { covered } = splitByCoverage(
|
||||
[tx({ date: "2026-07-26", account: "xxxxxxxxxxxx2176" })],
|
||||
coverage
|
||||
);
|
||||
expect(covered).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("keeps a row after the watermark — the whole point of the feed", () => {
|
||||
// The 2026-08-12 salary: the statement has not arrived, so this row is the
|
||||
// only record of it and must survive.
|
||||
const { keep } = splitByCoverage(
|
||||
[tx({ date: "2026-08-12", account: "xxxxxxxxxxxx2176", amount: 10782 })],
|
||||
coverage
|
||||
);
|
||||
expect(keep).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("keeps everything for an account with no statements", () => {
|
||||
const { keep, covered } = splitByCoverage(
|
||||
[tx({ date: "2020-01-01", account: "xxx-xxx xx4878" })],
|
||||
coverage
|
||||
);
|
||||
expect(keep).toHaveLength(1);
|
||||
expect(covered).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("keeps everything when no account column was mapped", () => {
|
||||
// A single-account bank export has no account column, and a row must never
|
||||
// be dropped on a guess about which account it belongs to.
|
||||
const { keep, covered } = splitByCoverage([tx({ date: "2020-01-01" })], coverage);
|
||||
expect(keep).toHaveLength(1);
|
||||
expect(covered).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("applies each account's own watermark, not a global one", () => {
|
||||
// The real export spans watermarks from 2026-03-31 to 2026-07-26. A single
|
||||
// date would either import duplicates or discard real rows.
|
||||
const { keep, covered } = splitByCoverage(
|
||||
[
|
||||
tx({ date: "2026-05-01", account: "xxxx0887" }), // covered to 2026-03-31 → keep
|
||||
tx({ date: "2026-05-01", account: "xxxx2176" }), // covered to 2026-07-26 → drop
|
||||
],
|
||||
[
|
||||
{ last4: "0887", coveredTo: "2026-03-31" },
|
||||
{ last4: "2176", coveredTo: "2026-07-26" },
|
||||
]
|
||||
);
|
||||
expect(keep).toHaveLength(1);
|
||||
expect(keep[0].account).toBe("xxxx0887");
|
||||
expect(covered).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("inFileDuplicates", () => {
|
||||
it("counts twins that a row id cannot catch", () => {
|
||||
// A CDR re-consent re-exports an account's whole history under fresh ids
|
||||
// while the originals survive, so source_ref sees two distinct rows. 385 of
|
||||
// these were in one 2,563-row export and doubled every salary payment.
|
||||
const rows = [
|
||||
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
|
||||
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
|
||||
];
|
||||
expect(inFileDuplicates(rows)).toBe(1);
|
||||
});
|
||||
|
||||
it("does not flag the same amount on different days", () => {
|
||||
expect(
|
||||
inFileDuplicates([
|
||||
tx({ date: "2026-07-07", amount: 5, description: "COFFEE" }),
|
||||
tx({ date: "2026-07-08", amount: 5, description: "COFFEE" }),
|
||||
])
|
||||
).toBe(0);
|
||||
});
|
||||
|
||||
it("does not flag the same amount on different accounts", () => {
|
||||
// Both legs of an internal transfer: same day, same amount, different
|
||||
// accounts, and both are real.
|
||||
expect(
|
||||
inFileDuplicates([
|
||||
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "6264" }),
|
||||
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "9940" }),
|
||||
])
|
||||
).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("applyMapping with account and row-id columns", () => {
|
||||
const labels = ["transaction_id", "description", "amount", "transaction_date", "account_number"];
|
||||
const mapping: ColumnMapping = {
|
||||
dateCol: "transaction_date",
|
||||
descriptionCol: "description",
|
||||
amountMode: "single",
|
||||
amountCol: "amount",
|
||||
accountCol: "account_number",
|
||||
sourceRefCol: "transaction_id",
|
||||
};
|
||||
|
||||
it("carries the account and row id through", () => {
|
||||
const out = applyMapping(
|
||||
[["1326131276", "HDR Global Services", "10782.00", "2026-08-12", "xxxxxxxxxxxx4830"]],
|
||||
labels,
|
||||
mapping,
|
||||
"YYYY-MM-DD"
|
||||
);
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0].account).toBe("xxxxxxxxxxxx4830");
|
||||
expect(out[0].source_ref).toBe("1326131276");
|
||||
// A positive single-column amount is money in.
|
||||
expect(out[0].transaction_type).toBe("credit");
|
||||
});
|
||||
|
||||
it("leaves both undefined when the columns are not mapped", () => {
|
||||
const out = applyMapping(
|
||||
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830"]],
|
||||
labels,
|
||||
{ dateCol: "transaction_date", descriptionCol: "description", amountMode: "single", amountCol: "amount" },
|
||||
"YYYY-MM-DD"
|
||||
);
|
||||
expect(out[0].account).toBeUndefined();
|
||||
expect(out[0].source_ref).toBeUndefined();
|
||||
expect(out[0].transaction_type).toBe("debit");
|
||||
});
|
||||
});
|
||||
@@ -1,388 +0,0 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { parseCSVRows } from "@/lib/csv-parser";
|
||||
import {
|
||||
ACCOUNTS,
|
||||
EXCLUDED_ACCOUNTS,
|
||||
isKnownExcluded,
|
||||
SUSPECT_REPEAT_GAP,
|
||||
accountFor,
|
||||
accountReport,
|
||||
dedupe,
|
||||
last4,
|
||||
naturalKey,
|
||||
readRows,
|
||||
skipReason,
|
||||
toLedgerRow,
|
||||
unknownAccounts,
|
||||
type FrolloRow,
|
||||
} from "@/lib/frollo-csv";
|
||||
import { LARGE_BATCH, findAnomalies } from "@/lib/frollo-ingest";
|
||||
|
||||
const HEADER =
|
||||
"transaction_id,description,user_description,amount,currency,transaction_date,posted_date," +
|
||||
"account_number,account_name,credit_debit,transaction_type,provider_name,merchant_name," +
|
||||
"budget_category,category_name,user_tags,notes,included";
|
||||
|
||||
function csv(...lines: string[]): FrolloRow[] {
|
||||
return readRows(parseCSVRows([HEADER, ...lines].join("\n")));
|
||||
}
|
||||
|
||||
/** Shapes a line in the export's real column order. */
|
||||
function line(o: {
|
||||
id: string;
|
||||
desc?: string;
|
||||
amount: string;
|
||||
currency?: string;
|
||||
date?: string;
|
||||
posted?: string;
|
||||
account?: string;
|
||||
accountName?: string;
|
||||
cd?: string;
|
||||
type?: string;
|
||||
merchant?: string;
|
||||
}): string {
|
||||
return [
|
||||
o.id,
|
||||
o.desc ?? "SOME MERCHANT",
|
||||
"",
|
||||
o.amount,
|
||||
o.currency ?? "AUD",
|
||||
o.date ?? "2026-03-01",
|
||||
o.posted ?? "",
|
||||
o.account ?? "xxx-xxx xxxx1910",
|
||||
o.accountName ?? "Smart Access",
|
||||
o.cd ?? (o.amount.startsWith("-") ? "debit" : "credit"),
|
||||
o.type ?? "payment",
|
||||
"CommBank",
|
||||
o.merchant ?? "Some Merchant",
|
||||
"lifestyle",
|
||||
"Groceries",
|
||||
'""',
|
||||
"",
|
||||
"true",
|
||||
].join(",");
|
||||
}
|
||||
|
||||
describe("readRows", () => {
|
||||
it("maps by header name, not position", () => {
|
||||
const rows = csv(line({ id: "1", amount: "-12.34", desc: "COFFEE" }));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({
|
||||
transactionId: "1",
|
||||
description: "COFFEE",
|
||||
amount: -12.34,
|
||||
currency: "AUD",
|
||||
accountName: "Smart Access",
|
||||
});
|
||||
});
|
||||
|
||||
it("throws when a required column is missing rather than reading undefined", () => {
|
||||
const broken = "transaction_id,description,amount\n1,X,-1.00";
|
||||
expect(() => readRows(parseCSVRows(broken))).toThrow(/missing expected column/i);
|
||||
});
|
||||
|
||||
it("keeps a quoted description containing commas intact", () => {
|
||||
const desc = '"DIDI_NZ DidiNZ pending Auckland NZL, ##0626 8.77 NZ DOLLAR"';
|
||||
const rows = csv(line({ id: "1", amount: "-7.23", desc }));
|
||||
expect(rows[0].description).toContain("NZL, ##0626");
|
||||
});
|
||||
|
||||
it("treats an absent posted_date as null, not empty string", () => {
|
||||
expect(csv(line({ id: "1", amount: "-1.00" }))[0].postedDate).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("account scoping", () => {
|
||||
it("matches on the last four digits regardless of masking format", () => {
|
||||
expect(last4("xxx-xxx xxxx1910")).toBe("1910");
|
||||
expect(last4("xxxxxx xxxxx6264")).toBe("6264");
|
||||
expect(last4("x xxxx 0887 ")).toBe("0887");
|
||||
expect(last4("XXXXXXXXXXXX0351")).toBe("0351");
|
||||
});
|
||||
|
||||
it("has no duplicate last4 among configured accounts", () => {
|
||||
const keys = ACCOUNTS.map((a) => a.last4);
|
||||
expect(new Set(keys).size).toBe(keys.length);
|
||||
});
|
||||
|
||||
it("excludes the credit cards, which statements already cover", () => {
|
||||
for (const card of ["3893", "8032", "1000", "0351", "6227"]) {
|
||||
expect(ACCOUNTS.find((a) => a.last4 === card)).toBeUndefined();
|
||||
}
|
||||
});
|
||||
|
||||
it("recognises the loan and income accounts", () => {
|
||||
expect(accountFor("xxx-xxx xxxxx0682")?.label).toMatch(/loan/i);
|
||||
expect(accountFor("xxxxxxxxxxxx4830")?.label).toMatch(/income/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe("skipReason", () => {
|
||||
it("skips a zero-amount fee waiver", () => {
|
||||
const row = csv(
|
||||
line({ id: "1", amount: "0.0", desc: "INTNL TRANSACTION FEE, $ 41.58 FEE SAV" })
|
||||
)[0];
|
||||
expect(skipReason(row)).toBe("zero_amount");
|
||||
});
|
||||
|
||||
it("skips a credit-card row", () => {
|
||||
const row = csv(line({ id: "1", amount: "-50.00", account: "xxxx xxxx xxxx 3893" }))[0];
|
||||
expect(skipReason(row)).toBe("out_of_scope_account");
|
||||
});
|
||||
|
||||
it("skips a Westpac pending row on its description prefix", () => {
|
||||
const row = csv(
|
||||
line({ id: "1", amount: "-13.50", desc: "Pending - Breadworld Bakery Cafe", account: "xxx-xxx xx4878" })
|
||||
)[0];
|
||||
expect(skipReason(row)).toBe("pending");
|
||||
});
|
||||
|
||||
it("accepts an ordinary in-scope row", () => {
|
||||
expect(skipReason(csv(line({ id: "1", amount: "-29.17" }))[0])).toBeNull();
|
||||
});
|
||||
|
||||
it("skips rather than imports an unparseable amount", () => {
|
||||
const row = csv(line({ id: "1", amount: "n/a" }))[0];
|
||||
expect(skipReason(row)).toBe("unparseable_amount");
|
||||
});
|
||||
});
|
||||
|
||||
describe("dedupe", () => {
|
||||
it("collapses a re-ingest twin to the earlier id", () => {
|
||||
const rows = csv(
|
||||
line({ id: "1200000000", amount: "-61.03", desc: "AMAZON MARKETPLACE" }),
|
||||
line({ id: "1326000000", amount: "-61.03", desc: "AMAZON MARKETPLACE" })
|
||||
);
|
||||
const { kept, dropped } = dedupe(rows);
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(kept[0].transactionId).toBe("1200000000");
|
||||
expect(dropped).toHaveLength(1);
|
||||
expect(dropped[0].duplicateOf.transactionId).toBe("1200000000");
|
||||
});
|
||||
|
||||
it("collapses close-id repeats too, but flags them for review", () => {
|
||||
// Five identical $1.00 Apple charges with consecutive ids. On a card
|
||||
// account, so out of scope in practice — but if such a group ever reaches
|
||||
// dedupe it must be surfaced, not silently decided either way.
|
||||
const rows = csv(
|
||||
...["1326128755", "1326128756", "1326128757", "1326128758", "1326128759"].map((id) =>
|
||||
line({ id, amount: "-1.0", desc: "APPLE.COM/BILL SYDNEY", date: "2026-07-28" })
|
||||
)
|
||||
);
|
||||
const { kept, dropped } = dedupe(rows);
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(dropped).toHaveLength(4);
|
||||
expect(dropped.every((d) => d.suspectGenuineRepeat)).toBe(true);
|
||||
});
|
||||
|
||||
it("does not flag a far-apart twin as a possible repeat", () => {
|
||||
const rows = csv(
|
||||
line({ id: "1101410977", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830" }),
|
||||
line({ id: "1101415475", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830" })
|
||||
);
|
||||
const { kept, dropped } = dedupe(rows);
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(dropped[0].suspectGenuineRepeat).toBe(false);
|
||||
expect(dropped[0].idGap).toBe(4498);
|
||||
});
|
||||
|
||||
it("collapses a real income twin — the 100%-overstatement case", () => {
|
||||
// Every HDR salary payment appears twice in every export, id gaps ranging
|
||||
// from 58 to 79 million. Keeping both doubled reported income.
|
||||
const rows = csv(
|
||||
line({ id: "1126624097", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830", desc: "HDR Global Services" }),
|
||||
line({ id: "1126624242", amount: "10790.8", currency: "USD", account: "xxxxxxxxxxxx4830", desc: "HDR Global Services" })
|
||||
);
|
||||
const { kept } = dedupe(rows);
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(kept[0].transactionId).toBe("1126624097");
|
||||
});
|
||||
|
||||
it("does not merge rows that differ in amount, date, description or account", () => {
|
||||
const rows = csv(
|
||||
line({ id: "1000000001", amount: "-10.00" }),
|
||||
line({ id: "1326000001", amount: "-10.01" }),
|
||||
line({ id: "1326000002", amount: "-10.00", date: "2026-03-02" }),
|
||||
line({ id: "1326000003", amount: "-10.00", desc: "OTHER" }),
|
||||
line({ id: "1326000004", amount: "-10.00", account: "xxx-xxx xx4878" })
|
||||
);
|
||||
expect(dedupe(rows).kept).toHaveLength(5);
|
||||
});
|
||||
|
||||
it("keeps the lowest id, because old ids survive a re-ingest", () => {
|
||||
const rows = csv(
|
||||
line({ id: "1326000001", amount: "-5.00" }),
|
||||
line({ id: "1000000000", amount: "-5.00" }),
|
||||
line({ id: "1200000000", amount: "-5.00" })
|
||||
);
|
||||
const { kept } = dedupe(rows);
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(kept[0].transactionId).toBe("1000000000");
|
||||
});
|
||||
|
||||
it("reports the id gap so a suspect group can be reviewed", () => {
|
||||
const near = csv(
|
||||
line({ id: "1000000000", amount: "-5.00" }),
|
||||
line({ id: String(1000000000 + SUSPECT_REPEAT_GAP - 1), amount: "-5.00" })
|
||||
);
|
||||
expect(dedupe(near).dropped[0].suspectGenuineRepeat).toBe(true);
|
||||
const far = csv(
|
||||
line({ id: "1000000000", amount: "-5.00" }),
|
||||
line({ id: String(1000000000 + SUSPECT_REPEAT_GAP), amount: "-5.00" })
|
||||
);
|
||||
expect(dedupe(far).dropped[0].suspectGenuineRepeat).toBe(false);
|
||||
});
|
||||
|
||||
it("is idempotent", () => {
|
||||
const rows = csv(
|
||||
line({ id: "1000000001", amount: "-61.03" }),
|
||||
line({ id: "1326000001", amount: "-61.03" })
|
||||
);
|
||||
const once = dedupe(rows).kept;
|
||||
expect(dedupe(once).kept).toEqual(once);
|
||||
});
|
||||
});
|
||||
|
||||
describe("naturalKey", () => {
|
||||
it("is stable across masking differences on the same account", () => {
|
||||
const a = csv(line({ id: "1", amount: "-1.00", account: "xxx-xxx xxxx1910" }))[0];
|
||||
const b = csv(line({ id: "2", amount: "-1.00", account: "xx-xxx xxxxxx1910" }))[0];
|
||||
expect(naturalKey(a)).toBe(naturalKey(b));
|
||||
});
|
||||
});
|
||||
|
||||
describe("toLedgerRow", () => {
|
||||
it("stores a debit as a positive amount with a direction", () => {
|
||||
const r = toLedgerRow(csv(line({ id: "1", amount: "-29.17" }))[0]);
|
||||
expect(r.amount).toBe(29.17);
|
||||
expect(r.transactionType).toBe("debit");
|
||||
});
|
||||
|
||||
it("marks a positive amount as a credit", () => {
|
||||
const r = toLedgerRow(csv(line({ id: "1", amount: "302.70" }))[0]);
|
||||
expect(r.amount).toBe(302.7);
|
||||
expect(r.transactionType).toBe("credit");
|
||||
});
|
||||
|
||||
it("leaves a foreign row's amount native and names the currency", () => {
|
||||
const r = toLedgerRow(
|
||||
csv(line({ id: "1", amount: "-10001.13", currency: "USD", account: "xxxxxxxxxxxx4830" }))[0]
|
||||
);
|
||||
expect(r.amount).toBe(10001.13);
|
||||
expect(r.foreignCurrencyCode).toBe("USD");
|
||||
expect(r.foreignCurrencyAmount).toBe(10001.13);
|
||||
});
|
||||
|
||||
it("carries no foreign fields for an AUD row", () => {
|
||||
const r = toLedgerRow(csv(line({ id: "1", amount: "-5.00" }))[0]);
|
||||
expect(r.foreignCurrencyCode).toBeNull();
|
||||
expect(r.foreignCurrencyAmount).toBeNull();
|
||||
});
|
||||
|
||||
it("drops Frollo's literal 'Unknown' merchant rather than showing it", () => {
|
||||
const r = toLedgerRow(csv(line({ id: "1", amount: "-5.00", merchant: "Unknown" }))[0]);
|
||||
expect(r.merchantName).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps a real merchant", () => {
|
||||
const r = toLedgerRow(csv(line({ id: "1", amount: "-5.00", merchant: "Solar Victoria" }))[0]);
|
||||
expect(r.merchantName).toBe("Solar Victoria");
|
||||
});
|
||||
|
||||
it("carries the provider's id as source_ref for idempotency", () => {
|
||||
const r = toLedgerRow(csv(line({ id: "986846710", amount: "-5.00" }))[0]);
|
||||
expect(r.sourceRef).toBe("986846710");
|
||||
});
|
||||
});
|
||||
|
||||
describe("accountReport / unknownAccounts", () => {
|
||||
it("reports a configured account with no rows as absent — a lapsed consent", () => {
|
||||
const rows = csv(line({ id: "1", amount: "-5.00", account: "xxx-xxx xxxx1910" }));
|
||||
const report = accountReport(rows);
|
||||
expect(report.find((r) => r.spec.last4 === "1910")?.present).toBe(true);
|
||||
expect(report.find((r) => r.spec.last4 === "4830")?.present).toBe(false);
|
||||
});
|
||||
|
||||
it("surfaces an account in the file that the importer does not know", () => {
|
||||
const rows = csv(line({ id: "1", amount: "-5.00", account: "xxxx xxxx xxxx 9999", accountName: "New Card" }));
|
||||
const unknown = unknownAccounts(rows);
|
||||
expect(unknown).toHaveLength(1);
|
||||
expect(unknown[0]).toMatchObject({ accountName: "New Card", rows: 1 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("findAnomalies", () => {
|
||||
const base = { missingAccounts: [], unknownAccounts: [], suspectRepeats: [], toInsert: 10, skipped: {} };
|
||||
|
||||
it("passes a boring run", () => {
|
||||
expect(findAnomalies(base)).toEqual([]);
|
||||
});
|
||||
|
||||
it("flags a configured account that contributed nothing — a lapsed consent", () => {
|
||||
const out = findAnomalies({ ...base, missingAccounts: ["4830 Wise (income)"] });
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0]).toMatch(/consent may have lapsed/i);
|
||||
});
|
||||
|
||||
it("flags an account that appeared in the file unannounced", () => {
|
||||
const out = findAnomalies({
|
||||
...base,
|
||||
unknownAccounts: [{ accountNumber: "xxxx9999", accountName: "New Everyday", rows: 4 }],
|
||||
});
|
||||
expect(out[0]).toMatch(/unknown account/i);
|
||||
});
|
||||
|
||||
it("flags a de-duplication that might have collapsed a real repeat", () => {
|
||||
const out = findAnomalies({
|
||||
...base,
|
||||
suspectRepeats: [{ description: "X", transactionDate: "2026-01-01", amount: -1, idGap: 2 }],
|
||||
});
|
||||
expect(out[0]).toMatch(/genuine repeats/i);
|
||||
});
|
||||
|
||||
it("flags a batch far larger than an ordinary month", () => {
|
||||
expect(findAnomalies({ ...base, toInsert: LARGE_BATCH + 1 })).toHaveLength(1);
|
||||
expect(findAnomalies({ ...base, toInsert: LARGE_BATCH })).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("flags an export taken with pending included", () => {
|
||||
const out = findAnomalies({ ...base, skipped: { pending: 3 } });
|
||||
expect(out[0]).toMatch(/pending/i);
|
||||
});
|
||||
|
||||
it("reports every anomaly at once rather than the first", () => {
|
||||
expect(
|
||||
findAnomalies({ ...base, missingAccounts: ["a"], toInsert: LARGE_BATCH + 1, skipped: { pending: 1 } })
|
||||
).toHaveLength(3);
|
||||
});
|
||||
});
|
||||
|
||||
describe("excluded vs unknown accounts", () => {
|
||||
it("does not report a deliberately excluded credit card as unknown", () => {
|
||||
// An alert that fires on every run is an alert nobody reads.
|
||||
const rows = csv(line({ id: "1", amount: "-50.00", account: "xxxx xxxx xxxx 3893", accountName: "Ultimate Awards credit card" }));
|
||||
expect(unknownAccounts(rows)).toEqual([]);
|
||||
expect(isKnownExcluded("xxxx xxxx xxxx 3893")).toBe(true);
|
||||
});
|
||||
|
||||
it("still reports a genuinely new account", () => {
|
||||
const rows = csv(line({ id: "1", amount: "-50.00", account: "xxxx xxxx xxxx 9999", accountName: "Brand New" }));
|
||||
expect(unknownAccounts(rows)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("keeps the imported and excluded rosters disjoint", () => {
|
||||
const a = new Set(ACCOUNTS.map((x) => x.last4));
|
||||
for (const e of EXCLUDED_ACCOUNTS) expect(a.has(e.last4)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("pending detection precedes account scoping", () => {
|
||||
it("reports a pending credit-card row as pending, not out of scope", () => {
|
||||
// Otherwise the wrong-export-option signal never fires, since every pending
|
||||
// row observed so far has been on a card.
|
||||
const row = csv(
|
||||
line({ id: "1", amount: "-13.50", desc: "Pending - Breadworld Bakery Cafe", account: "xxxx xxxx xxxx 8032" })
|
||||
)[0];
|
||||
expect(skipReason(row)).toBe("pending");
|
||||
});
|
||||
});
|
||||
@@ -1,229 +0,0 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { ingestFrolloCsv, LEDGER_MATCH_DAYS, type SqlExecutor } from "@/lib/frollo-ingest";
|
||||
|
||||
/**
|
||||
* Tests for the ledger-duplicate guard.
|
||||
*
|
||||
* This is the check whose absence caused the 2026-08-13 first import to write
|
||||
* 422 duplicate rows out of 550 — every one of them a second copy of a payment
|
||||
* the ledger already held from a statement. It passed 29 unit tests and a clean
|
||||
* dry run at the time, because every one of those tests compared the CSV against
|
||||
* itself. Nothing compared it against the database.
|
||||
*/
|
||||
|
||||
const HEADER =
|
||||
"transaction_id,description,user_description,amount,currency,transaction_date,posted_date," +
|
||||
"account_number,account_name,credit_debit,transaction_type,provider_name,merchant_name," +
|
||||
"budget_category,category_name,user_tags,notes,included";
|
||||
|
||||
function line(o: { id: string; desc?: string; amount: string; date: string }): string {
|
||||
return [
|
||||
o.id,
|
||||
o.desc ?? "SOME MERCHANT",
|
||||
"",
|
||||
o.amount,
|
||||
"AUD",
|
||||
o.date,
|
||||
"",
|
||||
"xxx-xxx xxxx1910",
|
||||
"Smart Access",
|
||||
o.amount.startsWith("-") ? "debit" : "credit",
|
||||
"payment",
|
||||
"CommBank",
|
||||
"Some Merchant",
|
||||
"lifestyle",
|
||||
"Groceries",
|
||||
"",
|
||||
"",
|
||||
"true",
|
||||
].join(",");
|
||||
}
|
||||
|
||||
const csv = (...lines: string[]) => [HEADER, ...lines].join("\n");
|
||||
|
||||
/**
|
||||
* Stands in for the database. The two queries are told apart by their text,
|
||||
* which is crude but keeps the test free of a live connection — and the guard
|
||||
* is pure logic over what comes back.
|
||||
*/
|
||||
function exec(
|
||||
prior: { transaction_date: string; amount: string; transaction_type?: string }[],
|
||||
coverage: { last4: string; covered_to: string }[] = []
|
||||
): SqlExecutor {
|
||||
return (async (sql: string) => {
|
||||
if (sql.includes("WHERE source = $1")) return [];
|
||||
if (sql.includes("FROM statements")) return coverage;
|
||||
if (sql.includes("superseded_by_id IS NULL")) return prior.map((p) => ({ transaction_type: "debit", ...p }));
|
||||
return [];
|
||||
}) as SqlExecutor;
|
||||
}
|
||||
|
||||
describe("ledger-duplicate guard", () => {
|
||||
it("drops a row the ledger already holds on the same day", async () => {
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2026-03-10" })),
|
||||
exec([{ transaction_date: "2026-03-10", amount: "42.50" }])
|
||||
);
|
||||
expect(r.ledgerDuplicates).toBe(1);
|
||||
expect(r.toInsert).toBe(0);
|
||||
});
|
||||
|
||||
it("drops a row dated within the match window", async () => {
|
||||
// The feed dates a transaction when the provider posted it and a statement
|
||||
// when the bank did; a weekend puts two days between them.
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2026-03-10" })),
|
||||
exec([{ transaction_date: "2026-03-13", amount: "42.50" }])
|
||||
);
|
||||
expect(LEDGER_MATCH_DAYS).toBe(3);
|
||||
expect(r.ledgerDuplicates).toBe(1);
|
||||
expect(r.toInsert).toBe(0);
|
||||
});
|
||||
|
||||
it("keeps a row beyond the match window", async () => {
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2026-03-10" })),
|
||||
exec([{ transaction_date: "2026-03-14", amount: "42.50" }])
|
||||
);
|
||||
expect(r.ledgerDuplicates).toBe(0);
|
||||
expect(r.toInsert).toBe(1);
|
||||
});
|
||||
|
||||
it("keeps a row whose amount differs", async () => {
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2026-03-10" })),
|
||||
exec([{ transaction_date: "2026-03-10", amount: "42.51" }])
|
||||
);
|
||||
expect(r.ledgerDuplicates).toBe(0);
|
||||
expect(r.toInsert).toBe(1);
|
||||
});
|
||||
|
||||
it("consumes each ledger row once, so a genuine repeat survives", async () => {
|
||||
// Two real charges of the same amount in the same week, one of which the
|
||||
// ledger already has. Matching without consuming would drop both and lose a
|
||||
// transaction that never existed anywhere else.
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(
|
||||
line({ id: "1", desc: "COFFEE ONE", amount: "-5.00", date: "2026-03-10" }),
|
||||
line({ id: "2", desc: "COFFEE TWO", amount: "-5.00", date: "2026-03-11" })
|
||||
),
|
||||
exec([{ transaction_date: "2026-03-10", amount: "5.00" }])
|
||||
);
|
||||
expect(r.ledgerDuplicates).toBe(1);
|
||||
expect(r.toInsert).toBe(1);
|
||||
});
|
||||
|
||||
it("matches when the driver returns Date objects, not strings", async () => {
|
||||
// `pg` maps a Postgres DATE to a JS Date; Prisma and the CSV give strings.
|
||||
// The first cut of this guard read one as the other, and because the failure
|
||||
// is a silent NaN it reported 550 rows to insert against a ledger holding
|
||||
// 422 of them. Both shapes are tested because both drivers are in use: the
|
||||
// CLI runs on pg, the API route on Prisma.
|
||||
const asDate = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2026-03-10" })),
|
||||
exec([{ transaction_date: new Date(2026, 2, 10) as unknown as string, amount: "42.50" }])
|
||||
);
|
||||
expect(asDate.ledgerDuplicates).toBe(1);
|
||||
expect(asDate.toInsert).toBe(0);
|
||||
});
|
||||
|
||||
it("never matches a credit against a debit", async () => {
|
||||
// Internal transfers between the owner's own accounts put both legs in the
|
||||
// feed: 2026-05-18 carries +3076.04 into ANZ and -3076.04 out of AMP. On
|
||||
// amount alone the credit leg consumed the ledger's debit row and the real
|
||||
// duplicate was written. 20 rows got in this way before direction was part
|
||||
// of the key.
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(
|
||||
line({ id: "1", desc: "PAYMENT FROM SELF", amount: "3076.04", date: "2026-05-18" }),
|
||||
line({ id: "2", desc: "Transfer to Self", amount: "-3076.04", date: "2026-05-18" })
|
||||
),
|
||||
exec([{ transaction_date: "2026-05-18", amount: "3076.04", transaction_type: "debit" }])
|
||||
);
|
||||
// The debit leg is the duplicate; the credit leg is a genuinely new row.
|
||||
expect(r.ledgerDuplicates).toBe(1);
|
||||
expect(r.toInsert).toBe(1);
|
||||
});
|
||||
|
||||
it("treats a statement 'refund' as money in", async () => {
|
||||
// The feed calls a reversed account fee a credit; the statement importer
|
||||
// types it 'refund'. Classifying refund as an outflow left every ANZ
|
||||
// servicing-fee reversal in the file as a duplicate.
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", desc: "REVERSAL OF ACCOUNT SERVICING FEE", amount: "5.00", date: "2026-02-27" })),
|
||||
exec([{ transaction_date: "2026-02-27", amount: "5.00", transaction_type: "refund" }])
|
||||
);
|
||||
expect(r.ledgerDuplicates).toBe(1);
|
||||
expect(r.toInsert).toBe(0);
|
||||
});
|
||||
|
||||
it("treats a statement 'fee' as money out", async () => {
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", desc: "ACCOUNT SERVICING FEE", amount: "-5.00", date: "2026-03-31" })),
|
||||
exec([{ transaction_date: "2026-03-31", amount: "5.00", transaction_type: "fee" }])
|
||||
);
|
||||
expect(r.ledgerDuplicates).toBe(1);
|
||||
expect(r.toInsert).toBe(0);
|
||||
});
|
||||
|
||||
it("drops anything a statement already covers, whatever the amount", async () => {
|
||||
// The primary guard. Amount-matching cannot see this case: the two sources
|
||||
// decompose the same event differently — Frollo bundles the Wise fee into
|
||||
// the transfer (10001.13) where the statement itemises it (10000.00 + 1.13)
|
||||
// — so 38 Wise USD rows survived the amount guard while being the same
|
||||
// money. A statement's billing_end_date is a hard watermark instead.
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(
|
||||
line({ id: "1", desc: "Interactive Brokers LLC", amount: "-10001.13", date: "2026-07-25" }),
|
||||
line({ id: "2", desc: "HDR Global Services", amount: "10782.00", date: "2026-08-12" })
|
||||
),
|
||||
exec([], [{ last4: "1910", covered_to: "2026-07-26" }])
|
||||
);
|
||||
expect(r.coveredByStatement).toBe(1);
|
||||
expect(r.toInsert).toBe(1); // only the post-watermark row survives
|
||||
expect(r.statementWatermarks).toEqual([{ last4: "1910", coveredTo: "2026-07-26" }]);
|
||||
});
|
||||
|
||||
it("keeps everything for an account that has no statement at all", async () => {
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2020-01-01" })),
|
||||
exec([], [{ last4: "9999", covered_to: "2026-07-26" }])
|
||||
);
|
||||
expect(r.coveredByStatement).toBe(0);
|
||||
expect(r.toInsert).toBe(1);
|
||||
});
|
||||
|
||||
it("treats the watermark date itself as covered", async () => {
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2026-07-26" })),
|
||||
exec([], [{ last4: "1910", covered_to: "2026-07-26" }])
|
||||
);
|
||||
expect(r.coveredByStatement).toBe(1);
|
||||
expect(r.toInsert).toBe(0);
|
||||
});
|
||||
|
||||
it("keeps everything when the ledger is empty", async () => {
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(
|
||||
line({ id: "1", desc: "A", amount: "-5.00", date: "2026-03-10" }),
|
||||
line({ id: "2", desc: "B", amount: "-6.00", date: "2026-03-11" })
|
||||
),
|
||||
exec([])
|
||||
);
|
||||
expect(r.ledgerDuplicates).toBe(0);
|
||||
expect(r.toInsert).toBe(2);
|
||||
});
|
||||
|
||||
it("reports duplicates rather than silently discarding them", async () => {
|
||||
// The owner's standing requirement: a row that disappears must be counted,
|
||||
// because a missing transaction can mean several things and one of them is
|
||||
// worth a follow-up.
|
||||
const r = await ingestFrolloCsv(
|
||||
csv(line({ id: "1", amount: "-42.50", date: "2026-03-10" })),
|
||||
exec([{ transaction_date: "2026-03-10", amount: "42.50" }])
|
||||
);
|
||||
expect(r).toHaveProperty("ledgerDuplicates");
|
||||
expect(r.inScope).toBe(1);
|
||||
expect(r.ledgerDuplicates + r.toInsert).toBe(r.inScope);
|
||||
});
|
||||
});
|
||||
@@ -1,62 +0,0 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { queryRaw } from "@/lib/db";
|
||||
import { ingestFrolloCsv } from "@/lib/frollo-ingest";
|
||||
|
||||
/**
|
||||
* Machine ingest endpoint for a Frollo transaction export.
|
||||
*
|
||||
* Sibling to /api/orders/ingest and /api/receipts/ingest, shaped like them. Auth
|
||||
* is a shared secret rather than the Traefik `x-forwarded-user` header, because
|
||||
* n8n calls this app-to-app with no browser session to forward.
|
||||
*
|
||||
* Its own token so it can be rotated without touching the order or receipt
|
||||
* flows, each of which runs on a schedule nobody is watching.
|
||||
*
|
||||
* The body is the CSV itself, as text. The caller does not parse anything —
|
||||
* parsing, scoping, de-duplication and the insert all live in one place
|
||||
* (`frollo-ingest.ts`), shared with `scripts/import-frollo.mts`.
|
||||
*/
|
||||
function authorised(req: NextRequest): boolean {
|
||||
const expected = process.env.FROLLO_INGEST_TOKEN;
|
||||
if (!expected) return false; // fail closed when unconfigured
|
||||
const got = req.headers.get("x-ingest-token");
|
||||
return !!got && got === expected;
|
||||
}
|
||||
|
||||
export async function POST(req: NextRequest) {
|
||||
if (!authorised(req)) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
|
||||
|
||||
let body: { csv?: string; dryRun?: boolean; force?: boolean };
|
||||
try {
|
||||
body = await req.json();
|
||||
} catch {
|
||||
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
|
||||
}
|
||||
if (!body?.csv || typeof body.csv !== "string") {
|
||||
return NextResponse.json({ error: "csv (string) is required" }, { status: 400 });
|
||||
}
|
||||
|
||||
try {
|
||||
const report = await ingestFrolloCsv(body.csv, queryRaw, {
|
||||
apply: !body.dryRun,
|
||||
force: body.force === true,
|
||||
});
|
||||
|
||||
// An automatic caller must not be able to write through an anomaly by
|
||||
// accident: the run reports 409 and inserts nothing, so a human decides.
|
||||
// Today's session is the argument — the de-duplication rule passed 29 tests
|
||||
// and a clean dry run while still doubling three salary payments.
|
||||
if (!report.applied && report.anomalies.length > 0 && !body.dryRun) {
|
||||
return NextResponse.json({ kind: "needs_review", ...report }, { status: 409 });
|
||||
}
|
||||
return NextResponse.json({ kind: "frollo", ...report });
|
||||
} catch (e) {
|
||||
// A missing or renamed column throws rather than importing blank rows, and
|
||||
// that is worth surfacing as loudly as an unparseable receipt.
|
||||
const message = e instanceof Error ? e.message : String(e);
|
||||
if (/missing expected column|empty CSV/i.test(message)) {
|
||||
return NextResponse.json({ kind: "rejected", reason: message }, { status: 422 });
|
||||
}
|
||||
return NextResponse.json({ error: message }, { status: 500 });
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,12 @@ export async function POST(req: NextRequest) {
|
||||
|
||||
const body = await req.json() as {
|
||||
bank_name: string;
|
||||
/**
|
||||
* Optional provenance label, stored on every row as `source`. With a
|
||||
* `source_ref` per row it makes a re-import a no-op; without it this path
|
||||
* has no idempotency at all.
|
||||
*/
|
||||
source?: string;
|
||||
transactions: {
|
||||
date: string;
|
||||
description: string;
|
||||
@@ -17,6 +23,8 @@ export async function POST(req: NextRequest) {
|
||||
foreign_currency_amount?: number;
|
||||
foreign_currency_code?: string;
|
||||
category?: string;
|
||||
account?: string;
|
||||
source_ref?: string;
|
||||
}[];
|
||||
};
|
||||
|
||||
@@ -25,7 +33,7 @@ export async function POST(req: NextRequest) {
|
||||
}
|
||||
|
||||
const tagId = await ensureTag("csv-import", "#8b5cf6");
|
||||
const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId);
|
||||
const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId, body.source);
|
||||
|
||||
return NextResponse.json({ inserted }, { status: 201 });
|
||||
}
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { getCurrentUser } from "@/lib/auth";
|
||||
import { getStatementCoverage } from "@/lib/queries";
|
||||
|
||||
/**
|
||||
* How far each account's statements already reach.
|
||||
*
|
||||
* Read by the CSV import modal so it can drop rows the ledger already holds.
|
||||
* The first Frollo import went in without this and wrote 422 duplicates out of
|
||||
* 550 — the same payments the statements already carried, itemised and
|
||||
* converted. On the 2026-08-13 export the rule takes 2,607 rows down to 171.
|
||||
*/
|
||||
export async function GET(req: NextRequest) {
|
||||
const user = await getCurrentUser(req);
|
||||
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||
return NextResponse.json({ coverage: await getStatementCoverage() });
|
||||
}
|
||||
@@ -2,10 +2,10 @@
|
||||
|
||||
import { useState, useRef, useEffect } from "react";
|
||||
import { CATEGORIES, formatCategory } from "@/lib/categories";
|
||||
import { useImportCSV } from "@/lib/hooks";
|
||||
import { useImportCSV, useStatementCoverage } from "@/lib/hooks";
|
||||
import {
|
||||
parseCSVRows, detectHasHeaders, getColumnLabels, getDataRows, applyMapping,
|
||||
saveBankPreset, loadBankPresets,
|
||||
saveBankPreset, loadBankPresets, splitByCoverage, inFileDuplicates,
|
||||
type DateFormat, type ColumnMapping, type ParsedTransaction, type BankPreset,
|
||||
} from "@/lib/csv-parser";
|
||||
|
||||
@@ -14,6 +14,8 @@ const TX_TYPES = ["debit", "credit", "payment", "refund", "fee", "interest", "tr
|
||||
|
||||
type Step = "upload" | "map" | "review" | "done";
|
||||
|
||||
|
||||
|
||||
function ColSelect({
|
||||
label, value, onChange, options, required,
|
||||
}: {
|
||||
@@ -54,6 +56,9 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
||||
const [error, setError] = useState("");
|
||||
const [presets, setPresets] = useState<BankPreset[]>([]);
|
||||
const [insertedCount, setInsertedCount] = useState(0);
|
||||
const [coveredRows, setCoveredRows] = useState<ParsedTransaction[]>([]);
|
||||
const [showCovered, setShowCovered] = useState(false);
|
||||
const coverage = useStatementCoverage();
|
||||
|
||||
useEffect(() => { setPresets(loadBankPresets()); }, []);
|
||||
|
||||
@@ -111,7 +116,9 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
||||
if (savePreset) {
|
||||
saveBankPreset({ bankName: bankName.trim(), mapping, dateFormat });
|
||||
}
|
||||
setEditedRows(parsed);
|
||||
const { keep, covered } = splitByCoverage(parsed, coverage.data ?? []);
|
||||
setEditedRows(keep);
|
||||
setCoveredRows(covered);
|
||||
setStep("review");
|
||||
}
|
||||
|
||||
@@ -120,7 +127,13 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
||||
const valid = editedRows.filter((r) => r.date && r.amount > 0 && r.description);
|
||||
if (!valid.length) { setError("No valid rows to import"); return; }
|
||||
try {
|
||||
const result = await importCSV.mutateAsync({ bank_name: bankName, transactions: valid });
|
||||
const result = await importCSV.mutateAsync({
|
||||
bank_name: bankName,
|
||||
// Provenance, so a second import of the same file inserts nothing —
|
||||
// paired with each row's source_ref where the file carries an id.
|
||||
source: bankName.trim().toLowerCase().replace(/\s+/g, "-"),
|
||||
transactions: valid,
|
||||
});
|
||||
setInsertedCount(result.inserted);
|
||||
setStep("done");
|
||||
} catch (e) {
|
||||
@@ -286,7 +299,14 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
||||
<div className="grid grid-cols-2 gap-3">
|
||||
<ColSelect label="Merchant Column (optional)" value={mapping.merchantCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, merchantCol: v || undefined }))} options={columnLabels} />
|
||||
<ColSelect label="Category Column (optional)" value={mapping.categoryCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, categoryCol: v || undefined }))} options={columnLabels} />
|
||||
<ColSelect label="Account Column (optional)" value={mapping.accountCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, accountCol: v || undefined }))} options={columnLabels} />
|
||||
<ColSelect label="Row ID Column (optional)" value={mapping.sourceRefCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, sourceRefCol: v || undefined }))} options={columnLabels} />
|
||||
</div>
|
||||
<p className="text-[11px] text-zinc-500 leading-relaxed">
|
||||
Map <b>Account</b> when the file covers more than one account: rows already
|
||||
covered by that account's statements are then left out. Map <b>Row ID</b> to
|
||||
make re-importing the same file do nothing.
|
||||
</p>
|
||||
|
||||
<label className="flex items-center gap-2 text-sm cursor-pointer text-zinc-400">
|
||||
<input type="checkbox" checked={savePreset} onChange={(e) => setSavePreset(e.target.checked)} className="accent-indigo-500" />
|
||||
@@ -300,9 +320,52 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
||||
{/* Step 3: Review */}
|
||||
{step === "review" && (
|
||||
<div>
|
||||
<p className="text-xs text-zinc-500 mb-3">
|
||||
{editedRows.length} transactions parsed. Edit or remove rows before importing.
|
||||
<p className="text-xs text-zinc-500 mb-2">
|
||||
{editedRows.length} transactions to import. Edit or remove rows before importing.
|
||||
</p>
|
||||
|
||||
{/*
|
||||
Reported, never silent. A row that disappears can mean several
|
||||
things and one of them is worth following up, so the count is
|
||||
always shown and the rows themselves are one click away.
|
||||
*/}
|
||||
{coveredRows.length > 0 && (
|
||||
<div className="mb-3 rounded border border-zinc-800 bg-zinc-800/30 px-3 py-2">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setShowCovered((v) => !v)}
|
||||
className="text-xs text-zinc-300 hover:text-white text-left w-full"
|
||||
>
|
||||
{coveredRows.length} row{coveredRows.length === 1 ? "" : "s"} left out — already covered by
|
||||
a statement for that account.{" "}
|
||||
<span className="text-indigo-400">{showCovered ? "hide" : "show"}</span>
|
||||
</button>
|
||||
{showCovered && (
|
||||
<>
|
||||
<ul className="mt-2 max-h-40 overflow-y-auto text-[11px] text-zinc-500 font-mono space-y-0.5">
|
||||
{coveredRows.slice(0, 200).map((r, i) => (
|
||||
<li key={i}>{r.date} {r.amount.toFixed(2).padStart(10)} {r.description.slice(0, 44)}</li>
|
||||
))}
|
||||
</ul>
|
||||
{coveredRows.length > 200 && (
|
||||
<p className="mt-1 text-[11px] text-zinc-600">…and {coveredRows.length - 200} more.</p>
|
||||
)}
|
||||
<p className="mt-2 text-[11px] text-zinc-500">
|
||||
Statements cover:{" "}
|
||||
{(coverage.data ?? []).map((c) => `${c.last4} to ${c.coveredTo}`).join(" · ")}
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{inFileDuplicates(editedRows) > 0 && (
|
||||
<p className="mb-3 rounded border border-amber-900/50 bg-amber-950/30 px-3 py-2 text-xs text-amber-300">
|
||||
{inFileDuplicates(editedRows)} row{inFileDuplicates(editedRows) === 1 ? " duplicates another row" : "s duplicate other rows"} in
|
||||
this same file (same date, amount and description). A re-consent can make an
|
||||
aggregator re-export history under new ids — check before importing.
|
||||
</p>
|
||||
)}
|
||||
<div className="overflow-x-auto rounded border border-zinc-800">
|
||||
<table className="w-full text-xs">
|
||||
<thead className="border-b border-zinc-800">
|
||||
|
||||
@@ -9,6 +9,22 @@ export interface ColumnMapping {
|
||||
creditCol?: string;
|
||||
merchantCol?: string;
|
||||
categoryCol?: string;
|
||||
/**
|
||||
* Column holding the account number, when the file covers more than one.
|
||||
*
|
||||
* Optional and usually absent — a bank's own export is one account per file.
|
||||
* An aggregator export (Frollo) is not, and without this every row would be
|
||||
* measured against the same statement coverage.
|
||||
*/
|
||||
accountCol?: string;
|
||||
/**
|
||||
* Column holding the provider's own row id, when it has one.
|
||||
*
|
||||
* Stored as `source_ref` so re-importing the same file inserts nothing. The
|
||||
* importer has no other idempotency: `row_index` is assigned MAX+1 on every
|
||||
* run, which makes `uq_transaction_identity` structurally unable to fire.
|
||||
*/
|
||||
sourceRefCol?: string;
|
||||
}
|
||||
|
||||
export interface BankPreset {
|
||||
@@ -24,6 +40,10 @@ export interface ParsedTransaction {
|
||||
transaction_type: string;
|
||||
merchant_name?: string;
|
||||
category?: string;
|
||||
/** Raw account identifier from the file, when a column was mapped. */
|
||||
account?: string;
|
||||
/** Provider row id, when a column was mapped. */
|
||||
source_ref?: string;
|
||||
}
|
||||
|
||||
export function parseCSVRows(text: string): string[][] {
|
||||
@@ -108,6 +128,8 @@ export function applyMapping(
|
||||
const descIdx = idx(mapping.descriptionCol);
|
||||
const merchantIdx = mapping.merchantCol ? idx(mapping.merchantCol) : -1;
|
||||
const categoryIdx = mapping.categoryCol ? idx(mapping.categoryCol) : -1;
|
||||
const accountIdx = mapping.accountCol ? idx(mapping.accountCol) : -1;
|
||||
const sourceRefIdx = mapping.sourceRefCol ? idx(mapping.sourceRefCol) : -1;
|
||||
|
||||
const results: ParsedTransaction[] = [];
|
||||
for (const row of dataRows) {
|
||||
@@ -138,6 +160,8 @@ export function applyMapping(
|
||||
const tx: ParsedTransaction = { date, description, amount, transaction_type };
|
||||
if (merchantIdx >= 0 && row[merchantIdx]) tx.merchant_name = row[merchantIdx].trim();
|
||||
if (categoryIdx >= 0 && row[categoryIdx]) tx.category = row[categoryIdx].trim();
|
||||
if (accountIdx >= 0 && row[accountIdx]) tx.account = row[accountIdx].trim();
|
||||
if (sourceRefIdx >= 0 && row[sourceRefIdx]) tx.source_ref = row[sourceRefIdx].trim();
|
||||
results.push(tx);
|
||||
}
|
||||
return results;
|
||||
@@ -154,3 +178,57 @@ export function loadBankPresets(): BankPreset[] {
|
||||
try { return JSON.parse(localStorage.getItem("csv-presets") || "[]"); }
|
||||
catch { return []; }
|
||||
}
|
||||
|
||||
/** Last four digits of an account identifier, however it is written. */
|
||||
export function last4(s: string): string {
|
||||
const d = s.replace(/\D/g, "");
|
||||
return d.length >= 4 ? d.slice(-4) : "";
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits parsed rows into what to import and what the statements already have.
|
||||
*
|
||||
* A statement's end date is a hard watermark: everything on that account up to
|
||||
* it is already in the ledger, itemised and converted. Without this the first
|
||||
* aggregator import wrote 422 duplicates out of 550 rows; with it the same
|
||||
* 2,607-row export offers 171.
|
||||
*
|
||||
* Rows are only excluded when an account column was mapped AND that account has
|
||||
* statements. Anything else is imported — a row is never dropped on a guess.
|
||||
*/
|
||||
export function splitByCoverage(
|
||||
rows: ParsedTransaction[],
|
||||
coverage: { last4: string; coveredTo: string }[]
|
||||
): { keep: ParsedTransaction[]; covered: ParsedTransaction[] } {
|
||||
const wm = new Map(coverage.map((c) => [c.last4, c.coveredTo]));
|
||||
const keep: ParsedTransaction[] = [];
|
||||
const covered: ParsedTransaction[] = [];
|
||||
for (const r of rows) {
|
||||
const to = r.account ? wm.get(last4(r.account)) : undefined;
|
||||
// Both are YYYY-MM-DD, so a string compare is a date compare.
|
||||
if (to && r.date <= to) covered.push(r);
|
||||
else keep.push(r);
|
||||
}
|
||||
return { keep, covered };
|
||||
}
|
||||
|
||||
/**
|
||||
* Rows that duplicate another row in the SAME file.
|
||||
*
|
||||
* Distinct from the coverage check, and not catchable by `source_ref`: a CDR
|
||||
* re-consent makes the provider re-export an account's whole history under
|
||||
* fresh ids while the originals survive, so the twins are identical in
|
||||
* everything except the id. That put 385 duplicates in one 2,563-row export and
|
||||
* doubled every salary payment. 385 rows is not something you spot by eye in a
|
||||
* review table, so it is counted here.
|
||||
*/
|
||||
export function inFileDuplicates(rows: ParsedTransaction[]): number {
|
||||
const seen = new Set<string>();
|
||||
let n = 0;
|
||||
for (const r of rows) {
|
||||
const key = [r.date, r.description, r.amount, r.transaction_type, r.account ?? ""].join("|");
|
||||
if (seen.has(key)) n += 1;
|
||||
else seen.add(key);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
@@ -1,382 +0,0 @@
|
||||
/**
|
||||
* Frollo transaction-export parsing and de-duplication.
|
||||
*
|
||||
* Frollo is a CDR (Open Banking) aggregator. Its app emails a CSV of up to 12
|
||||
* months of transactions. This module turns that file into rows ready for
|
||||
* `transactions`, and — mostly — decides which of them are real.
|
||||
*
|
||||
* Scope is deliberately narrow: credit cards are excluded because they already
|
||||
* arrive as monthly statements, which are authoritative. What Frollo is for is
|
||||
* the accounts whose statements arrive every 182 to 460 days (see
|
||||
* ACCOUNTS below), where the alternative is downloading each one by hand.
|
||||
*
|
||||
* The file has three properties that any consumer has to handle, all learned by
|
||||
* diffing three real exports (smarthome DECISIONS.md ING-11):
|
||||
*
|
||||
* 1. A CDR re-consent makes Frollo re-ingest the account's whole history under
|
||||
* fresh transaction ids, while the originals survive. So the same purchase
|
||||
* appears twice with different ids, and consents expire annually — this is
|
||||
* not a one-off. On one 1,989-row export, 385 rows were such twins, and Wise
|
||||
* credits were overstated by exactly 100%.
|
||||
* 2. `transaction_id` is otherwise stable: across two exports 12 days apart,
|
||||
* 1,462 of 1,532 shared rows kept identical ids and none were replaced. It
|
||||
* is therefore a good `source_ref` — it just cannot detect a twin.
|
||||
* 3. The export cannot express "pending", and the tell differs per bank
|
||||
* (Westpac prefixes the description; Amex emits no `posted_date` or
|
||||
* `transaction_type` at all, so its pending rows are indistinguishable from
|
||||
* its posted ones). The export's own option is the only reliable control.
|
||||
* Every pending row observed was on a credit card, i.e. out of scope, so
|
||||
* exporting with pending EXCLUDED costs nothing here — and it avoids the
|
||||
* re-key problem, since a settling row changes both its id and its
|
||||
* description.
|
||||
*/
|
||||
|
||||
/** A row as it appears in the CSV, after header mapping. */
|
||||
export interface FrolloRow {
|
||||
transactionId: string;
|
||||
description: string;
|
||||
amount: number;
|
||||
currency: string;
|
||||
transactionDate: string;
|
||||
postedDate: string | null;
|
||||
accountNumber: string;
|
||||
accountName: string;
|
||||
creditDebit: string;
|
||||
transactionType: string | null;
|
||||
providerName: string;
|
||||
merchantName: string | null;
|
||||
}
|
||||
|
||||
/** A row shaped for insertion into `transactions`. */
|
||||
export interface LedgerRow {
|
||||
sourceRef: string;
|
||||
sourceAccount: string;
|
||||
transactionDate: string;
|
||||
description: string;
|
||||
/** Always positive; direction lives in `transactionType`. */
|
||||
amount: number;
|
||||
transactionType: "debit" | "credit";
|
||||
merchantName: string | null;
|
||||
foreignCurrencyAmount: number | null;
|
||||
foreignCurrencyCode: string | null;
|
||||
}
|
||||
|
||||
export interface AccountSpec {
|
||||
/** Last four characters of the digits in the masked account number. */
|
||||
last4: string;
|
||||
label: string;
|
||||
provider: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Accounts this importer owns.
|
||||
*
|
||||
* Keyed on the last four digits rather than the masked string, because the
|
||||
* masking format varies by institution (`xxx-xxx xxxx1910`, `xxxxxx xxxxx6264`,
|
||||
* `x xxxx 0887`) and is a presentation choice Frollo could change. All fourteen
|
||||
* are distinct, as are the five excluded cards.
|
||||
*
|
||||
* Credit cards are absent on purpose — monthly statements already cover them and
|
||||
* are the source of truth. Adding one here would create a second producer for
|
||||
* rows the reconcile queue would then have to match against themselves.
|
||||
*/
|
||||
export const ACCOUNTS: AccountSpec[] = [
|
||||
{ last4: "6264", label: "ANZ Access Advantage", provider: "ANZ" },
|
||||
{ last4: "9940", label: "AMP (mortgage)", provider: "AMP" },
|
||||
{ last4: "8109", label: "AMP", provider: "AMP" },
|
||||
{ last4: "0682", label: "AMP (loan)", provider: "AMP" },
|
||||
{ last4: "4830", label: "Wise (income)", provider: "Wise" },
|
||||
{ last4: "2176", label: "Wise", provider: "Wise" },
|
||||
{ last4: "0052", label: "Up Spending", provider: "Up" },
|
||||
{ last4: "0078", label: "Up Savings", provider: "Up" },
|
||||
{ last4: "1910", label: "CommBank Smart Access", provider: "CommBank" },
|
||||
{ last4: "4878", label: "Westpac Choice", provider: "Westpac" },
|
||||
{ last4: "4758", label: "Westpac Business One", provider: "Westpac" },
|
||||
{ last4: "0887", label: "ING Orange Everyday", provider: "ING" },
|
||||
{ last4: "9336", label: "ING Savings Maximiser", provider: "ING" },
|
||||
{ last4: "9807", label: "ANZ Online Saver", provider: "ANZ" },
|
||||
];
|
||||
|
||||
/**
|
||||
* Accounts knowingly left out, so "excluded" is distinguishable from "unknown".
|
||||
*
|
||||
* Without this the five credit cards read as unrecognised accounts on every
|
||||
* single run, which would raise the new-account anomaly every time and make an
|
||||
* automatic import permanently refuse. An alert that always fires is an alert
|
||||
* nobody reads.
|
||||
*
|
||||
* Anything in neither list is genuinely new — a freshly connected account, or a
|
||||
* re-masked number — and does deserve a look before its rows are ignored.
|
||||
*/
|
||||
export const EXCLUDED_ACCOUNTS: AccountSpec[] = [
|
||||
{ last4: "3893", label: "CommBank Ultimate Awards (card)", provider: "CommBank" },
|
||||
{ last4: "8032", label: "Westpac Altitude Qantas Black (card)", provider: "Westpac" },
|
||||
{ last4: "1000", label: "Amex Platinum Business (card)", provider: "American Express" },
|
||||
{ last4: "0351", label: "HSBC Star Alliance (card)", provider: "HSBC" },
|
||||
{ last4: "6227", label: "MyCard Prestige (card)", provider: "Mycard" },
|
||||
];
|
||||
|
||||
export function isKnownExcluded(accountNumber: string): boolean {
|
||||
const key = last4(accountNumber);
|
||||
return EXCLUDED_ACCOUNTS.some((a) => a.last4 === key);
|
||||
}
|
||||
|
||||
/**
|
||||
* Below this id gap, two rows sharing a natural key are *probably* a genuine
|
||||
* same-day repeat rather than a re-ingest twin — and are reported rather than
|
||||
* silently collapsed.
|
||||
*
|
||||
* This is a review trigger, not a decision. An earlier version used a 10,000
|
||||
* threshold to *keep* close-id rows, and it was wrong: measured across two real
|
||||
* exports, in-scope duplicate pairs have gaps from **58 to 260 million**, while
|
||||
* the only observed genuine repeats (five identical $1.00 Apple charges) had
|
||||
* gaps of 1 to 4. A threshold anywhere in between is a magic number, and the
|
||||
* error it enables is asymmetric — keeping a twin doubled reported income by
|
||||
* 100%, whereas collapsing a real repeat understates by one row.
|
||||
*
|
||||
* So dedupe collapses unconditionally and flags anything this close for a human
|
||||
* to look at. On the accounts in scope, nothing has ever tripped it: every
|
||||
* duplicate group is a pair and the minimum gap is 58.
|
||||
*/
|
||||
export const SUSPECT_REPEAT_GAP = 10;
|
||||
|
||||
export function last4(accountNumber: string): string {
|
||||
return accountNumber.replace(/\D/g, "").slice(-4);
|
||||
}
|
||||
|
||||
export function accountFor(accountNumber: string): AccountSpec | undefined {
|
||||
const key = last4(accountNumber);
|
||||
return ACCOUNTS.find((a) => a.last4 === key);
|
||||
}
|
||||
|
||||
const REQUIRED_COLUMNS = [
|
||||
"transaction_id",
|
||||
"description",
|
||||
"amount",
|
||||
"currency",
|
||||
"transaction_date",
|
||||
"account_number",
|
||||
"account_name",
|
||||
"credit_debit",
|
||||
];
|
||||
|
||||
/**
|
||||
* Maps a parsed CSV grid to rows, by header name rather than position.
|
||||
*
|
||||
* Throws on a missing required column rather than reading undefined into a
|
||||
* transaction: a column rename upstream should stop the import loudly, not
|
||||
* silently produce rows with blank descriptions.
|
||||
*/
|
||||
export function readRows(grid: string[][]): FrolloRow[] {
|
||||
if (grid.length === 0) throw new Error("empty CSV");
|
||||
const header = grid[0].map((h) => h.trim());
|
||||
const missing = REQUIRED_COLUMNS.filter((c) => !header.includes(c));
|
||||
if (missing.length > 0) {
|
||||
throw new Error(`Frollo CSV missing expected column(s): ${missing.join(", ")}`);
|
||||
}
|
||||
const at = (row: string[], col: string): string => {
|
||||
const i = header.indexOf(col);
|
||||
return i >= 0 ? (row[i] ?? "").trim() : "";
|
||||
};
|
||||
const out: FrolloRow[] = [];
|
||||
for (const row of grid.slice(1)) {
|
||||
if (row.every((f) => f === "")) continue;
|
||||
const amount = Number(at(row, "amount"));
|
||||
out.push({
|
||||
transactionId: at(row, "transaction_id"),
|
||||
description: at(row, "description"),
|
||||
amount: Number.isFinite(amount) ? amount : NaN,
|
||||
currency: at(row, "currency") || "AUD",
|
||||
transactionDate: at(row, "transaction_date"),
|
||||
postedDate: at(row, "posted_date") || null,
|
||||
accountNumber: at(row, "account_number"),
|
||||
accountName: at(row, "account_name"),
|
||||
creditDebit: at(row, "credit_debit"),
|
||||
transactionType: at(row, "transaction_type") || null,
|
||||
providerName: at(row, "provider_name"),
|
||||
merchantName: at(row, "merchant_name") || null,
|
||||
});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export type SkipReason =
|
||||
| "zero_amount"
|
||||
| "unparseable_amount"
|
||||
| "out_of_scope_account"
|
||||
| "pending";
|
||||
|
||||
/**
|
||||
* Why a row is not imported, or null if it should be.
|
||||
*
|
||||
* `zero_amount` is mostly CommBank fee *waivers* — 574 of them in one export,
|
||||
* genuinely $0.00 with the notional fee written into the description text
|
||||
* (`INTNL TRANSACTION FEE, $ 41.58 FEE SAV`). They are not transactions.
|
||||
*
|
||||
* `pending` is a belt-and-braces check on top of the export option. It only
|
||||
* catches Westpac's description prefix; Amex pending rows are undetectable here,
|
||||
* which is why the option matters and this does not replace it.
|
||||
*/
|
||||
export function skipReason(row: FrolloRow): SkipReason | null {
|
||||
if (!Number.isFinite(row.amount)) return "unparseable_amount";
|
||||
if (row.amount === 0) return "zero_amount";
|
||||
// Pending is tested before account scope on purpose. Every pending row seen so
|
||||
// far has been on a credit card, so scoping first would classify them all as
|
||||
// out-of-scope and the "this export was taken with pending INCLUDED" signal
|
||||
// would never fire — even though it is worth acting on wherever it appears,
|
||||
// because the option makes successive exports incomparable.
|
||||
if (/^\s*pending\s*[-:]/i.test(row.description)) return "pending";
|
||||
if (!accountFor(row.accountNumber)) return "out_of_scope_account";
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Natural key: what makes two rows the same purchase, ignoring ids. */
|
||||
export function naturalKey(row: FrolloRow): string {
|
||||
return [
|
||||
last4(row.accountNumber),
|
||||
row.transactionDate,
|
||||
row.amount.toFixed(2),
|
||||
row.description,
|
||||
row.currency,
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
export interface DroppedRow {
|
||||
row: FrolloRow;
|
||||
duplicateOf: FrolloRow;
|
||||
idGap: number;
|
||||
/** Ids close enough that this might be a real repeat, not a twin. Review it. */
|
||||
suspectGenuineRepeat: boolean;
|
||||
}
|
||||
|
||||
export interface DedupeResult {
|
||||
kept: FrolloRow[];
|
||||
dropped: DroppedRow[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapses each natural-key group to its lowest-id row.
|
||||
*
|
||||
* Lowest because old ids survive a re-ingestion — verified across two exports
|
||||
* twelve days apart, where the earlier ids were a strict subset of the later
|
||||
* ones. Keeping the lowest therefore keeps `source_ref` stable from run to run,
|
||||
* so a re-import updates nothing and inserts nothing.
|
||||
*
|
||||
* Everything collapsed is returned in `dropped`, and anything whose ids sit
|
||||
* within SUSPECT_REPEAT_GAP is flagged: that is the shape a genuine same-day
|
||||
* repeat would take, and it should be looked at rather than assumed.
|
||||
*/
|
||||
export function dedupe(rows: FrolloRow[], suspectGap = SUSPECT_REPEAT_GAP): DedupeResult {
|
||||
const groups = new Map<string, FrolloRow[]>();
|
||||
for (const row of rows) {
|
||||
const k = naturalKey(row);
|
||||
const g = groups.get(k);
|
||||
if (g) g.push(row);
|
||||
else groups.set(k, [row]);
|
||||
}
|
||||
|
||||
const kept: FrolloRow[] = [];
|
||||
const dropped: DroppedRow[] = [];
|
||||
|
||||
for (const group of groups.values()) {
|
||||
if (group.length === 1) {
|
||||
kept.push(group[0]);
|
||||
continue;
|
||||
}
|
||||
const sorted = [...group].sort(
|
||||
(a, b) => Number(a.transactionId) - Number(b.transactionId)
|
||||
);
|
||||
const winner = sorted[0];
|
||||
kept.push(winner);
|
||||
for (let i = 1; i < sorted.length; i++) {
|
||||
const idGap = Number(sorted[i].transactionId) - Number(sorted[i - 1].transactionId);
|
||||
dropped.push({
|
||||
row: sorted[i],
|
||||
duplicateOf: winner,
|
||||
idGap,
|
||||
suspectGenuineRepeat: idGap < suspectGap,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return { kept, dropped };
|
||||
}
|
||||
|
||||
/**
|
||||
* Shapes a row for `transactions`.
|
||||
*
|
||||
* `category` is deliberately not set. Frollo assigns its own categories and the
|
||||
* owner reports they are often wrong; finance-app's rules engine plus
|
||||
* `category_override` already decide this, and a third untrusted source
|
||||
* competing with both is worse than none.
|
||||
*
|
||||
* For a non-AUD row the file gives only the foreign figure — there is no AUD
|
||||
* equivalent anywhere in the export. That is the same position order-ingestion
|
||||
* is in, so it takes the same shape: `amount` IS the native figure,
|
||||
* `foreign_currency_code` names it, and `amount_aud` is left NULL rather than
|
||||
* asserting a rate we do not have. `AMOUNT_UNCONVERTED` in analytics-sql then
|
||||
* counts these and the UI reports the balance as incomplete.
|
||||
*/
|
||||
export function toLedgerRow(row: FrolloRow): LedgerRow {
|
||||
const foreign = row.currency !== "AUD";
|
||||
const magnitude = Math.abs(row.amount);
|
||||
return {
|
||||
sourceRef: row.transactionId,
|
||||
sourceAccount: row.accountNumber,
|
||||
transactionDate: row.transactionDate,
|
||||
description: row.description,
|
||||
amount: magnitude,
|
||||
transactionType: row.amount < 0 ? "debit" : "credit",
|
||||
// Frollo writes "Unknown" where it could not resolve a merchant. Storing
|
||||
// that would put the literal word on the screen; NULL lets the existing
|
||||
// merchant fallbacks work.
|
||||
merchantName: row.merchantName && row.merchantName !== "Unknown" ? row.merchantName : null,
|
||||
foreignCurrencyAmount: foreign ? magnitude : null,
|
||||
foreignCurrencyCode: foreign ? row.currency : null,
|
||||
};
|
||||
}
|
||||
|
||||
export interface AccountReport {
|
||||
spec: AccountSpec;
|
||||
rows: number;
|
||||
present: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Which configured accounts the file actually contains.
|
||||
*
|
||||
* A lapsed CDR consent does not error — the account simply stops appearing, and
|
||||
* the row count drops. Nothing else in the pipeline would notice, so the import
|
||||
* asserts the roster rather than trusting it.
|
||||
*/
|
||||
export function accountReport(rows: FrolloRow[]): AccountReport[] {
|
||||
const counts = new Map<string, number>();
|
||||
for (const r of rows) {
|
||||
const k = last4(r.accountNumber);
|
||||
counts.set(k, (counts.get(k) ?? 0) + 1);
|
||||
}
|
||||
return ACCOUNTS.map((spec) => ({
|
||||
spec,
|
||||
rows: counts.get(spec.last4) ?? 0,
|
||||
present: (counts.get(spec.last4) ?? 0) > 0,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Accounts in the file that are neither imported nor knowingly excluded.
|
||||
*
|
||||
* Empty is the normal state. A name appearing here means a new account reached
|
||||
* the export and its rows are being dropped on the floor — worth a look before
|
||||
* that becomes the status quo.
|
||||
*/
|
||||
export function unknownAccounts(rows: FrolloRow[]): { accountNumber: string; accountName: string; rows: number }[] {
|
||||
const seen = new Map<string, { accountNumber: string; accountName: string; rows: number }>();
|
||||
for (const r of rows) {
|
||||
if (accountFor(r.accountNumber) || isKnownExcluded(r.accountNumber)) continue;
|
||||
const k = last4(r.accountNumber);
|
||||
const e = seen.get(k);
|
||||
if (e) e.rows++;
|
||||
else seen.set(k, { accountNumber: r.accountNumber, accountName: r.accountName, rows: 1 });
|
||||
}
|
||||
return [...seen.values()].sort((a, b) => b.rows - a.rows);
|
||||
}
|
||||
@@ -1,384 +0,0 @@
|
||||
import {
|
||||
accountReport,
|
||||
dedupe,
|
||||
last4,
|
||||
readRows,
|
||||
skipReason,
|
||||
toLedgerRow,
|
||||
unknownAccounts,
|
||||
type AccountReport,
|
||||
type FrolloRow,
|
||||
type SkipReason,
|
||||
} from "./frollo-csv.ts";
|
||||
import { parseCSVRows } from "./csv-parser.ts";
|
||||
|
||||
/**
|
||||
* Turns a Frollo CSV into ledger rows and (optionally) writes them.
|
||||
*
|
||||
* Shared by `scripts/import-frollo.mts` and `POST /api/frollo/ingest` so the
|
||||
* hand-run and automatic paths cannot drift. They previously would have been two
|
||||
* implementations of the same insert, which is exactly how the pantry healthcheck
|
||||
* came to be fixed in one repo and left broken in the other.
|
||||
*
|
||||
* The database is reached through an injected executor rather than imported
|
||||
* directly: the API route runs inside Next with Prisma, while the CLI runs under
|
||||
* `node --experimental-strip-types` with a plain `pg` client and no path aliases.
|
||||
*/
|
||||
// The .ts extensions above are load-bearing: this module is imported both by
|
||||
// Next (which resolves either form) and by scripts/import-frollo.mts running
|
||||
// under `node --experimental-strip-types`, whose ESM resolver requires the
|
||||
// extension written out. tsconfig sets allowImportingTsExtensions for this.
|
||||
export type SqlExecutor = <T = unknown>(sql: string, params: unknown[]) => Promise<T[]>;
|
||||
|
||||
export const SOURCE = "frollo";
|
||||
|
||||
/**
|
||||
* Above this many new rows, an automatic run stops and asks.
|
||||
*
|
||||
* Steady state is ~50 rows a month. A batch several times that means something
|
||||
* changed — a re-consent duplicating history, a longer export window, or a new
|
||||
* account — and none of those should land unseen. The first (backfill) import
|
||||
* was 550 and was run by hand with `force`.
|
||||
*/
|
||||
export const LARGE_BATCH = 200;
|
||||
|
||||
/**
|
||||
* How far a Frollo row may sit from a ledger row and still be the same event.
|
||||
*
|
||||
* The feed dates a transaction when the provider posted it; a statement dates it
|
||||
* when the bank did. Those differ by a day or two around weekends. Three days is
|
||||
* wide enough to catch that and narrow enough that two genuinely different
|
||||
* charges for the identical amount in the same week are rare — and when they do
|
||||
* collide, the cost is one missing row in a feed whose whole job is provisional
|
||||
* visibility, against a permanent double-count the other way.
|
||||
*/
|
||||
export const LEDGER_MATCH_DAYS = 3;
|
||||
|
||||
/**
|
||||
* Ledger transaction types that mean money arriving.
|
||||
*
|
||||
* The whole set in use is debit | credit | payment | fee | interest | refund.
|
||||
* `refund` is the one that bites: the feed calls a reversed account fee a
|
||||
* `credit` while the statement importer types it `refund`, so classifying it as
|
||||
* an outflow left five real duplicates behind — every ANZ servicing-fee reversal
|
||||
* in the file. Small money, but it is the same shape of mismatch that would
|
||||
* matter on a large refund.
|
||||
*/
|
||||
const INFLOW_TYPES = new Set(["credit", "refund"]);
|
||||
|
||||
/**
|
||||
* A date column, as midnight UTC, from whatever the driver handed back.
|
||||
*
|
||||
* `pg` maps a Postgres DATE to a JS `Date` in local time, while Prisma and the
|
||||
* CSV both yield `"YYYY-MM-DD"` strings. Reading one as the other is silent:
|
||||
* `String(new Date()).slice(0, 10)` is `"Wed Mar 10"`, which parses to NaN, and
|
||||
* a guard built on it skips every row it was meant to compare — the first dry
|
||||
* run of this code reported 550 rows to insert and zero duplicates against a
|
||||
* ledger holding 422 of them.
|
||||
*/
|
||||
function dayMs(v: unknown): number {
|
||||
if (v instanceof Date) return Date.UTC(v.getFullYear(), v.getMonth(), v.getDate());
|
||||
const m = /^(\d{4})-(\d{2})-(\d{2})/.exec(String(v));
|
||||
return m ? Date.UTC(Number(m[1]), Number(m[2]) - 1, Number(m[3])) : NaN;
|
||||
}
|
||||
|
||||
export interface IngestOptions {
|
||||
ownerId?: number;
|
||||
/** Write. Without it, everything is computed and nothing is inserted. */
|
||||
apply?: boolean;
|
||||
/** Proceed despite anomalies. For a human who has read them. */
|
||||
force?: boolean;
|
||||
largeBatch?: number;
|
||||
}
|
||||
|
||||
export interface IngestReport {
|
||||
totalRows: number;
|
||||
inScope: number;
|
||||
skipped: Partial<Record<SkipReason, number>>;
|
||||
accounts: AccountReport[];
|
||||
missingAccounts: string[];
|
||||
unknownAccounts: { accountNumber: string; accountName: string; rows: number }[];
|
||||
duplicatesDropped: number;
|
||||
suspectRepeats: { description: string; transactionDate: string; amount: number; idGap: number }[];
|
||||
alreadyImported: number;
|
||||
/**
|
||||
* Rows dropped because a statement already covers that account up to that
|
||||
* date. The primary guard — see `statementWatermarks`.
|
||||
*/
|
||||
coveredByStatement: number;
|
||||
/** Per-account coverage boundary used above, for the report. */
|
||||
statementWatermarks: { last4: string; coveredTo: string }[];
|
||||
/**
|
||||
* Rows dropped because the ledger holds the same amount, same direction,
|
||||
* within a few days. The secondary guard, for accounts with no statement.
|
||||
*
|
||||
* Both counts are reported, never silent: a Frollo row vanishing can mean
|
||||
* several things and one of them is worth following up.
|
||||
*/
|
||||
ledgerDuplicates: number;
|
||||
toInsert: number;
|
||||
inserted: number;
|
||||
/** Non-empty means an automatic run should stop and a human should look. */
|
||||
anomalies: string[];
|
||||
applied: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reasons an automatic import should stop rather than write.
|
||||
*
|
||||
* Pure, so it is testable without a database. Every entry is a thing that has
|
||||
* either happened or come one verification away from happening: a lapsed CDR
|
||||
* consent silently shrinking the file, a new account appearing unnoticed, a
|
||||
* de-duplication that might have collapsed a real repeat, and a batch far larger
|
||||
* than the account activity can explain.
|
||||
*/
|
||||
export function findAnomalies(
|
||||
report: Pick<IngestReport, "missingAccounts" | "unknownAccounts" | "suspectRepeats" | "toInsert" | "skipped">,
|
||||
largeBatch = LARGE_BATCH
|
||||
): string[] {
|
||||
const out: string[] = [];
|
||||
if (report.missingAccounts.length > 0) {
|
||||
out.push(
|
||||
`${report.missingAccounts.length} configured account(s) contributed no rows ` +
|
||||
`(${report.missingAccounts.join(", ")}) — a CDR consent may have lapsed`
|
||||
);
|
||||
}
|
||||
if (report.unknownAccounts.length > 0) {
|
||||
out.push(
|
||||
`${report.unknownAccounts.length} unknown account(s) in the file, skipped ` +
|
||||
`(${report.unknownAccounts.map((a) => a.accountName).join(", ")})`
|
||||
);
|
||||
}
|
||||
if (report.suspectRepeats.length > 0) {
|
||||
out.push(
|
||||
`${report.suspectRepeats.length} de-duplicated row(s) had near-consecutive ids ` +
|
||||
`and may be genuine repeats rather than twins`
|
||||
);
|
||||
}
|
||||
if (report.toInsert > largeBatch) {
|
||||
out.push(`${report.toInsert} new rows is more than the expected ${largeBatch}`);
|
||||
}
|
||||
if ((report.skipped.pending ?? 0) > 0) {
|
||||
out.push(
|
||||
`${report.skipped.pending} pending row(s) present — export was taken with pending ` +
|
||||
`INCLUDED; they are skipped, but re-export with it excluded`
|
||||
);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export async function ingestFrolloCsv(
|
||||
csvText: string,
|
||||
exec: SqlExecutor,
|
||||
opts: IngestOptions = {}
|
||||
): Promise<IngestReport> {
|
||||
const ownerId = opts.ownerId ?? 1;
|
||||
const rows = readRows(parseCSVRows(csvText));
|
||||
|
||||
const skipped: Partial<Record<SkipReason, number>> = {};
|
||||
const inScope: FrolloRow[] = [];
|
||||
for (const r of rows) {
|
||||
const why = skipReason(r);
|
||||
if (why) skipped[why] = (skipped[why] ?? 0) + 1;
|
||||
else inScope.push(r);
|
||||
}
|
||||
|
||||
const accounts = accountReport(inScope);
|
||||
const missingAccounts = accounts.filter((a) => !a.present).map((a) => `${a.spec.last4} ${a.spec.label}`);
|
||||
const unknown = unknownAccounts(rows);
|
||||
|
||||
const { kept, dropped } = dedupe(inScope);
|
||||
const suspectRepeats = dropped
|
||||
.filter((d) => d.suspectGenuineRepeat)
|
||||
.map((d) => ({
|
||||
description: d.row.description,
|
||||
transactionDate: d.row.transactionDate,
|
||||
amount: d.row.amount,
|
||||
idGap: d.idGap,
|
||||
}));
|
||||
|
||||
const ledger = kept.map(toLedgerRow);
|
||||
const existing = await exec<{ source_ref: string }>(
|
||||
"SELECT source_ref FROM transactions WHERE source = $1",
|
||||
[SOURCE]
|
||||
);
|
||||
const seen = new Set(existing.map((r) => r.source_ref));
|
||||
const notYetImported = ledger.filter((r) => !seen.has(r.sourceRef));
|
||||
|
||||
// GUARD 1 — statement coverage. Do not import what the statements already
|
||||
// have.
|
||||
//
|
||||
// A statement's `billing_end_date` is a hard watermark: everything on that
|
||||
// account up to that date is already in the ledger, itemised and converted.
|
||||
// Amount-matching cannot replace this, because the two sources decompose the
|
||||
// same event differently — Frollo bundles the Wise fee into the transfer
|
||||
// (10001.13) where the statement itemises it (10000.00 + 1.13), so 38 Wise USD
|
||||
// rows survived the amount guard as "new" while being the same money.
|
||||
//
|
||||
// This is the coverage test the first import needed and got wrong. It asked
|
||||
// whether a row's date fell inside a statement's min–max *window*, which for
|
||||
// periods spanning 182 to 460 days swallows a year and answers nothing. The
|
||||
// watermark asks a different and answerable question: up to what date is this
|
||||
// account complete?
|
||||
//
|
||||
// Matching is on last4 alone. Verified against the live statement set: the
|
||||
// eight in-scope accounts with statements each map to exactly one bank, no
|
||||
// cross-bank collision. The per-account watermarks are reported so a wrong one
|
||||
// is visible rather than inferred from a row count.
|
||||
const coverage = await exec<{ last4: string; covered_to: unknown }>(
|
||||
`SELECT right(regexp_replace(account_number, '[^0-9]', '', 'g'), 4) AS last4,
|
||||
MAX(billing_end_date) AS covered_to
|
||||
FROM statements
|
||||
WHERE account_number IS NOT NULL
|
||||
GROUP BY 1`,
|
||||
[]
|
||||
);
|
||||
const watermark = new Map<string, number>();
|
||||
for (const c of coverage) {
|
||||
if (!c.last4) continue; // Wise files one statement as account_number 'N/A'
|
||||
const t = dayMs(c.covered_to);
|
||||
if (!Number.isFinite(t)) continue;
|
||||
watermark.set(c.last4, Math.max(watermark.get(c.last4) ?? -Infinity, t));
|
||||
}
|
||||
|
||||
let coveredByStatement = 0;
|
||||
const uncovered = notYetImported.filter((r) => {
|
||||
const w = watermark.get(last4(r.sourceAccount));
|
||||
if (w === undefined) return true; // no statement for this account, ever
|
||||
if (dayMs(r.transactionDate) <= w) {
|
||||
coveredByStatement += 1;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
const statementWatermarks = [...watermark.entries()]
|
||||
.filter(([l4]) => notYetImported.some((r) => last4(r.sourceAccount) === l4))
|
||||
.map(([l4, t]) => ({ last4: l4, coveredTo: new Date(t).toISOString().slice(0, 10) }))
|
||||
.sort((a, b) => a.last4.localeCompare(b.last4));
|
||||
|
||||
// GUARD 2 — drop rows the ledger already holds from a statement.
|
||||
//
|
||||
// This guard is the whole lesson of the 2026-08-13 first import. The feed was
|
||||
// scoped to "accounts that issue no monthly statement", but that was asserted,
|
||||
// never tested against the ledger — and it was wrong for almost every account:
|
||||
// 422 of the 550 rows imported (77%) had a statement twin within three days,
|
||||
// $1,023,824.63 of movement counted twice. The HDR Global salary showed it
|
||||
// most plainly, appearing as both A$15,518.53 (statement, converted) and
|
||||
// US$10,782.00 (feed, native) for the same July payment.
|
||||
//
|
||||
// The check that would have caught it is one query and takes a second. The
|
||||
// check that was run instead compared each row's date against the statement's
|
||||
// min-max window, which for accounts whose statements span 182 to 460 days
|
||||
// swallows a year and cannot distinguish covered from uncovered at all.
|
||||
const dates = uncovered.map((r) => dayMs(r.transactionDate)).filter(Number.isFinite);
|
||||
let ledgerDuplicates = 0;
|
||||
let fresh = uncovered;
|
||||
if (dates.length > 0) {
|
||||
const pad = LEDGER_MATCH_DAYS * 86_400_000;
|
||||
const from = new Date(Math.min(...dates) - pad).toISOString().slice(0, 10);
|
||||
const to = new Date(Math.max(...dates) + pad).toISOString().slice(0, 10);
|
||||
|
||||
// Superseded rows are excluded: one has already been replaced by the row
|
||||
// that supersedes it, so matching against both would hide a genuine gap.
|
||||
const priorRows = await exec<{ transaction_date: string; amount: string; transaction_type: string }>(
|
||||
`SELECT transaction_date, amount, transaction_type
|
||||
FROM transactions
|
||||
WHERE (source IS NULL OR source <> $1)
|
||||
AND superseded_by_id IS NULL
|
||||
AND transaction_date BETWEEN $2::date AND $3::date`,
|
||||
[SOURCE, from, to]
|
||||
);
|
||||
|
||||
// Keyed on amount AND direction.
|
||||
//
|
||||
// Amount is the field both sides agree on exactly — descriptions do not
|
||||
// survive the trip (the feed writes "HDR Global Services (Bermuda)" where
|
||||
// the statement writes "Received money from HDR Global Services (Bermuda)
|
||||
// with reference ...") and dates drift by a day or two.
|
||||
//
|
||||
// Direction has to be in the key because this ledger is full of internal
|
||||
// transfers between the owner's own accounts, and both legs are in the feed:
|
||||
// 2026-05-18 carries a +3076.04 credit into ANZ and a -3076.04 debit out of
|
||||
// AMP. On amount alone the credit leg consumed the ledger's debit row and
|
||||
// the genuine duplicate was written — 20 rows got in this way on the first
|
||||
// corrected run. A credit is never the duplicate of a debit.
|
||||
const key = (amount: number, inflow: boolean) => `${Math.abs(amount).toFixed(2)}:${inflow ? "in" : "out"}`;
|
||||
const byAmount = new Map<string, number[]>();
|
||||
for (const p of priorRows) {
|
||||
const k = key(Number(p.amount), INFLOW_TYPES.has(p.transaction_type));
|
||||
const t = dayMs(p.transaction_date);
|
||||
if (!Number.isFinite(t)) continue;
|
||||
const list = byAmount.get(k);
|
||||
if (list) list.push(t);
|
||||
else byAmount.set(k, [t]);
|
||||
}
|
||||
|
||||
fresh = uncovered.filter((r) => {
|
||||
const candidates = byAmount.get(key(r.amount, r.transactionType === "credit"));
|
||||
if (!candidates) return true;
|
||||
const t = dayMs(r.transactionDate);
|
||||
if (!Number.isFinite(t)) return true;
|
||||
const hit = candidates.findIndex((c) => Math.abs(c - t) <= pad);
|
||||
if (hit === -1) return true;
|
||||
// Consume the match so two feed rows cannot both claim one ledger row —
|
||||
// a real pair of identical charges must not collapse into one.
|
||||
candidates.splice(hit, 1);
|
||||
ledgerDuplicates += 1;
|
||||
return false;
|
||||
});
|
||||
}
|
||||
|
||||
const base = {
|
||||
totalRows: rows.length,
|
||||
inScope: inScope.length,
|
||||
skipped,
|
||||
accounts,
|
||||
missingAccounts,
|
||||
unknownAccounts: unknown,
|
||||
duplicatesDropped: dropped.length,
|
||||
suspectRepeats,
|
||||
alreadyImported: ledger.length - notYetImported.length,
|
||||
coveredByStatement,
|
||||
statementWatermarks,
|
||||
ledgerDuplicates,
|
||||
toInsert: fresh.length,
|
||||
};
|
||||
const anomalies = findAnomalies(base, opts.largeBatch);
|
||||
|
||||
if (!opts.apply || (anomalies.length > 0 && !opts.force)) {
|
||||
return { ...base, inserted: 0, anomalies, applied: false };
|
||||
}
|
||||
|
||||
let inserted = 0;
|
||||
for (const r of fresh) {
|
||||
// ON CONFLICT against uq_transaction_source_ref: belt and braces over the
|
||||
// source_ref check above, which is not transactional.
|
||||
const res = await exec<{ id: number }>(
|
||||
`INSERT INTO transactions
|
||||
(statement_id, owner_id, transaction_date, description, amount, transaction_type,
|
||||
merchant_name, foreign_currency_amount, foreign_currency_code,
|
||||
source, source_ref, source_account)
|
||||
VALUES (NULL, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
|
||||
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL
|
||||
DO NOTHING
|
||||
RETURNING id`,
|
||||
[
|
||||
ownerId,
|
||||
r.transactionDate,
|
||||
r.description,
|
||||
r.amount,
|
||||
r.transactionType,
|
||||
r.merchantName,
|
||||
r.foreignCurrencyAmount,
|
||||
r.foreignCurrencyCode,
|
||||
SOURCE,
|
||||
r.sourceRef,
|
||||
r.sourceAccount,
|
||||
]
|
||||
);
|
||||
inserted += res.length;
|
||||
}
|
||||
|
||||
return { ...base, inserted, anomalies, applied: true };
|
||||
}
|
||||
+20
-1
@@ -1053,9 +1053,11 @@ export function useImportCSV() {
|
||||
return useMutation({
|
||||
mutationFn: async (body: {
|
||||
bank_name: string;
|
||||
source?: string;
|
||||
transactions: {
|
||||
date: string; description: string; amount: number; transaction_type: string;
|
||||
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string; category?: string;
|
||||
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string;
|
||||
category?: string; account?: string; source_ref?: string;
|
||||
}[];
|
||||
}) => {
|
||||
const res = await fetch("/api/import/csv", {
|
||||
@@ -1257,3 +1259,20 @@ export function useOrderDetail(entityKey: string | null) {
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* How far each account's statements already reach, for the CSV import modal.
|
||||
*
|
||||
* Fetched once when the modal opens: the review step uses it to drop rows the
|
||||
* ledger already holds rather than offering them for import.
|
||||
*/
|
||||
export function useStatementCoverage() {
|
||||
return useQuery({
|
||||
queryKey: ["statement-coverage"],
|
||||
queryFn: async () => {
|
||||
const res = await fetch("/api/import/statement-coverage");
|
||||
if (!res.ok) throw new Error("Could not load statement coverage");
|
||||
return (await res.json()).coverage as { last4: string; coveredTo: string }[];
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
+77
-25
@@ -670,8 +670,11 @@ export async function batchInsertCSVTransactions(
|
||||
foreign_currency_amount?: number;
|
||||
foreign_currency_code?: string;
|
||||
category?: string;
|
||||
account?: string;
|
||||
source_ref?: string;
|
||||
}[],
|
||||
tagId: number
|
||||
tagId: number,
|
||||
source?: string
|
||||
): Promise<number> {
|
||||
if (rows.length === 0) return 0;
|
||||
|
||||
@@ -685,13 +688,32 @@ export async function batchInsertCSVTransactions(
|
||||
const params: unknown[] = [ownerId];
|
||||
let p = 2;
|
||||
rows.forEach((r, i) => {
|
||||
valueClauses.push(`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`);
|
||||
params.push(r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null, r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null);
|
||||
valueClauses.push(
|
||||
`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`
|
||||
);
|
||||
params.push(
|
||||
r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null,
|
||||
r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null,
|
||||
// The category the reviewer chose. It used to be accepted here and then
|
||||
// left out of the column list, so every category picked in the review
|
||||
// step was silently discarded and the trigger wrote 'other'. That is not
|
||||
// cosmetic: an uncategorised credit is admitted by NET_SPEND_ROWS and
|
||||
// negated by SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of
|
||||
// spend and none of them counted as income.
|
||||
r.category ?? null,
|
||||
source ?? null,
|
||||
// source + source_ref is the only idempotency this path has. Without it a
|
||||
// second import of the same file duplicates every row, because row_index
|
||||
// is assigned MAX+1 and uq_transaction_identity can never fire.
|
||||
r.source_ref ?? null,
|
||||
r.account ?? null
|
||||
);
|
||||
});
|
||||
|
||||
const txIds = await queryRaw<{ id: number }>(
|
||||
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, row_index)
|
||||
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, category, source, source_ref, source_account, row_index)
|
||||
VALUES ${valueClauses.join(", ")}
|
||||
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL DO NOTHING
|
||||
RETURNING id`,
|
||||
params
|
||||
);
|
||||
@@ -733,30 +755,24 @@ export const needsCardMatch = (alias = "t") =>
|
||||
`(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`;
|
||||
|
||||
/**
|
||||
* REMOVED 2026-08-13, and the reason is worth more than the code was.
|
||||
* REMOVED 2026-08-13. There used to be an `awaitsStatementLine()` predicate
|
||||
* here, excluding account-feed rows from the pending-reconciliation queue.
|
||||
*
|
||||
* There used to be an `awaitsStatementLine()` predicate here excluding
|
||||
* account-feed rows from the pending-reconciliation queue. Its premise — "the
|
||||
* Frollo importer deliberately covers only accounts whose statements are *not*
|
||||
* imported, so no statement line is coming for these, ever" — was asserted and
|
||||
* never tested. It was false for almost every account: of the 550 rows the first
|
||||
* import wrote, 422 already had a statement twin.
|
||||
* Its premise — a feed row is the account's own ledger entry, so no statement
|
||||
* line is coming — was asserted and never tested. It was false for almost every
|
||||
* account: 422 of the first 550 imported rows already had a statement twin.
|
||||
*
|
||||
* What makes this worth recording is *how* it was introduced. The queue jumped
|
||||
* from 8 to 558 the moment the feed landed, and that jump was read as noise and
|
||||
* filtered away. The queue was right. Those 550 rows genuinely were provisional
|
||||
* entries awaiting their statement lines, and suppressing the count removed the
|
||||
* only mechanism that would ever have collapsed them — so the duplicates became
|
||||
* permanent instead of transient, and stayed invisible until a human noticed the
|
||||
* same salary payment listed twice in two currencies.
|
||||
* How it got in matters more than what it did. The queue jumped from 8 to 558
|
||||
* the moment the feed landed, and that jump was read as noise and filtered
|
||||
* away. The queue was right — those rows genuinely were provisional entries
|
||||
* awaiting statement lines — and hiding them removed the only mechanism that
|
||||
* would ever have collapsed them, so a transient overlap became a permanent
|
||||
* double-count and stayed invisible until a human saw one salary payment listed
|
||||
* twice in two currencies.
|
||||
*
|
||||
* A feed row IS a row awaiting a statement line. It belongs in the queue. The
|
||||
* volume problem is solved upstream, by not importing rows the ledger already
|
||||
* holds (`LEDGER_MATCH_DAYS` in frollo-ingest.ts) — not downstream by hiding
|
||||
* the ones that are.
|
||||
*
|
||||
* If a genuinely statement-less feed is ever added, give it a predicate of its
|
||||
* own and prove the premise with a query first.
|
||||
* An imported row that has no statement belongs in the queue. Volume is solved
|
||||
* upstream, by not importing what the statements already cover
|
||||
* (`getStatementCoverage`), never downstream by hiding what is there.
|
||||
*/
|
||||
|
||||
/**
|
||||
@@ -1528,3 +1544,39 @@ export async function getTagTransactionIds(tagId: number): Promise<number[]> {
|
||||
);
|
||||
return rows.map((r) => r.transaction_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* The last date each account's statements cover, keyed by the account's last
|
||||
* four digits.
|
||||
*
|
||||
* This is the coverage test that matters, and it is not the one that was tried
|
||||
* first. The first attempt asked whether a row's date fell inside a statement's
|
||||
* min-max *window*, which for accounts whose statements span 182 to 460 days
|
||||
* swallows a year and answers nothing — 422 duplicates out of 550 rows got in
|
||||
* that way. `billing_end_date` asks a question that has an answer: up to what
|
||||
* date is this account complete?
|
||||
*
|
||||
* Amount matching cannot substitute for it. Two sources decompose the same
|
||||
* event differently — an aggregator bundles the Wise transfer fee into the
|
||||
* transfer (10001.13) where the statement itemises it (10000.00 + 1.13) — so
|
||||
* the rows are the same money with different numbers.
|
||||
*
|
||||
* Keyed on last4 because account numbers are written differently everywhere
|
||||
* ("235242176", "xxxxxxxxxxxx2176", "4085-56264"). Where two banks share a
|
||||
* last4 the later date wins, which errs towards excluding a row rather than
|
||||
* duplicating one; the caller shows every watermark so a wrong one is visible.
|
||||
*/
|
||||
export async function getStatementCoverage(): Promise<{ last4: string; coveredTo: string }[]> {
|
||||
const rows = await queryRaw<{ last4: string; covered_to: string }>(
|
||||
`SELECT right(regexp_replace(account_number, '[^0-9]', '', 'g'), 4) AS last4,
|
||||
MAX(billing_end_date)::text AS covered_to
|
||||
FROM statements
|
||||
WHERE account_number IS NOT NULL
|
||||
GROUP BY 1`,
|
||||
[]
|
||||
);
|
||||
return rows
|
||||
.filter((r) => r.last4 && r.last4.length === 4)
|
||||
.map((r) => ({ last4: r.last4, coveredTo: r.covered_to }))
|
||||
.sort((a, b) => a.last4.localeCompare(b.last4));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user