siddharthd/finance-app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

csv import: exclude what the statements already cover, and delete the Frollo importer
ci / lint-test (push) Successful in 43s
Details

Browse Source 
"This is becoming too complex... Frollo should be done through that [the manual
CSV import]" (owner). It was right: a bespoke importer, an API route, a CLI, two
scheduled n8n workflows and a shared secret existed to do what the CSV import
modal already did, minus one rule.

That rule is statement coverage, and it turns out to be the whole thing. Applying
each account's newest billing_end_date as a watermark takes the real 2,607-row
Frollo export down to 171 rows — with no account allowlist, no credit-card
exclusion and no Frollo-specific scoping at all. Cards drop out on their own
because their statements are current; the 46 card rows that survive are genuinely
post-statement. Every bit of the bespoke apparatus was doing by hand what one
query does generically.

Deleted: src/lib/frollo-csv.ts, src/lib/frollo-ingest.ts, scripts/import-frollo.mts,
src/app/api/frollo/, both test files, the FROLLO_INGEST_TOKEN wiring, and the n8n
Frollo Import + Frollo Freshness Check workflows.

Added to the shared CSV path, so every import benefits:

  - getStatementCoverage() + /api/import/statement-coverage. The review step
    leaves out rows an account's statements already cover and says how many, with
    the rows one click away. Only applies when an account column is mapped and
    that account has statements — a row is never dropped on a guess.
  - Optional Account and Row ID columns in the mapper. Account drives the
    watermark and is stored as source_account; Row ID becomes source_ref.
  - An in-file duplicate warning. A CDR re-consent re-exports history under
    fresh ids, so source_ref cannot see it — 385 twins in one 2,563-row export
    doubled every salary payment, and that is not visible by eye in a review
    table.

Two pre-existing bugs in that path, both of which this plan depends on:

  - The category chosen in the review step was accepted by
    batchInsertCSVTransactions and then left out of the INSERT column list, so it
    was silently discarded and the trigger wrote 'other'. Not cosmetic: an
    uncategorised credit is admitted by NET_SPEND_ROWS and negated by
    SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of spend while
    counting as no income.
  - The path had no idempotency whatsoever. row_index is assigned MAX+1 on every
    run, which makes uq_transaction_identity structurally unable to fire, so a
    second import of the same file duplicated all of it. Now writes source +
    source_ref with ON CONFLICT DO NOTHING.

awaitsStatementLine()'s removal note is kept but rewritten: it no longer points
at a deleted file, and the lesson stands — the queue jump from 8 to 558 was the
measurement, not the noise.
This commit is contained in:
siddharthd
2026-08-13 12:58:06 +10:00
parent b82c4570bd
commit 461c021e7a
13 changed files with 453 additions and 1605 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/import-frollo.mts
-127
View File
@@ -1,127 +0,0 @@
/**
* Imports a Frollo transaction export into `transactions`.
*
* node --experimental-strip-types scripts/import-frollo.mts --file <csv>
* node --experimental-strip-types scripts/import-frollo.mts --file <csv> --apply
*
* Dry run by default: it prints exactly what it would do and touches nothing.
* Rehearse first — the failure mode here is doubling reported income, and the
* de-duplication rule that shipped second only survived because a dry run was
* read against the salary rows.
*
* Export the file with PENDING TRANSACTIONS EXCLUDED. A pending row changes both
* its id and its description when it settles, so importing one guarantees a
* duplicate on the next run; every pending row observed has been on a credit
* card, which this importer does not cover anyway. See DECISIONS.md ING-11.
*
* All parsing, scoping, de-duplication and insertion live in
* `src/lib/frollo-ingest.ts`, shared with POST /api/frollo/ingest so the manual
* and automatic paths cannot drift.
*
* Needs DATABASE_URL. postgres-personal publishes no host port, so from the host:
* export DATABASE_URL="postgresql://personal:<pw>@$(docker inspect postgres-personal \
* --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'):5432/personal"
* The container IP changes on every recreate.
*/
import { readFileSync } from "node:fs";
import pg from "pg";
import { ingestFrolloCsv, LEDGER_MATCH_DAYS, type SqlExecutor } from "../src/lib/frollo-ingest.ts";
function arg(name: string, fallback?: string): string | undefined {
const i = process.argv.indexOf(`--${name}`);
if (i >= 0 && process.argv[i + 1] && !process.argv[i + 1].startsWith("--")) return process.argv[i + 1];
return fallback;
}
const has = (name: string) => process.argv.includes(`--${name}`);
const file = arg("file");
if (!file) {
console.error("usage: --file <csv> [--apply] [--force] [--owner <id>]");
process.exit(2);
}
if (!process.env.DATABASE_URL) {
console.error("DATABASE_URL is not set — cannot compare against the ledger.");
process.exit(2);
}
const money = (n: number) => n.toLocaleString("en-AU", { minimumFractionDigits: 2, maximumFractionDigits: 2 });
const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
await client.connect();
const exec: SqlExecutor = async <T,>(sql: string, params: unknown[]) =>
(await client.query(sql, params)).rows as T[];
let report;
try {
report = await ingestFrolloCsv(readFileSync(file, "utf8"), exec, {
ownerId: Number(arg("owner", "1")),
apply: has("apply"),
force: has("force"),
});
} finally {
// Left open, the process hangs after an exception and looks like a slow query.
if (!has("keep-open")) await client.end().catch(() => {});
}
console.log(`\n${file}`);
console.log(` ${report.totalRows} rows in file\n`);
console.log("SCOPE");
for (const [why, n] of Object.entries(report.skipped).sort((a, b) => b[1]! - a[1]!)) {
console.log(` skipped ${String(n).padStart(5)} ${why}`);
}
console.log(` in scope ${String(report.inScope).padStart(4)}\n`);
console.log("ACCOUNTS");
for (const a of report.accounts) {
console.log(
` ${a.present ? " " : "!"} ${a.spec.last4} ${a.spec.label.padEnd(24)} ${String(a.rows).padStart(5)} rows`
);
}
if (report.unknownAccounts.length > 0) {
console.log("\n accounts in the file this importer does not know (skipped):");
for (const u of report.unknownAccounts) {
console.log(` ${u.accountNumber} ${u.accountName} ${u.rows} rows`);
}
}
console.log();
console.log("DE-DUPLICATION (re-ingest twins from CDR re-consent)");
console.log(` ${report.duplicatesDropped} dropped`);
if (report.suspectRepeats.length > 0) {
console.log(`\n REVIEW: ${report.suspectRepeats.length} collapsed row(s) had near-consecutive ids`);
console.log(" and may be real repeats rather than duplicates:");
for (const s of report.suspectRepeats.slice(0, 10)) {
console.log(` ${s.transactionDate} ${money(s.amount).padStart(11)} ${s.description.slice(0, 40)} (gap ${s.idGap})`);
}
}
console.log();
console.log("LEDGER");
console.log(` already imported : ${String(report.alreadyImported).padStart(5)}`);
// Printed even when zero. These are the counts that were missing on
// 2026-08-13, when the whole file was written on top of a ledger that already
// held 77% of it; a number you have to go looking for is a number nobody looks
// at. The watermarks are printed too, so a wrong coverage boundary is visible
// rather than inferred from a row count.
console.log(` covered by stmt : ${String(report.coveredByStatement).padStart(5)} (dated on or before the account's newest statement)`);
console.log(` amount twin : ${String(report.ledgerDuplicates).padStart(5)} (same amount + direction within ${LEDGER_MATCH_DAYS} days)`);
if (report.statementWatermarks.length > 0) {
console.log(" statements cover:");
for (const w of report.statementWatermarks) console.log(` ${w.last4} up to ${w.coveredTo}`);
}
console.log(` new to insert : ${String(report.toInsert).padStart(5)}`);
if (report.anomalies.length > 0) {
console.log("\nANOMALIES");
for (const a of report.anomalies) console.log(` ! ${a}`);
}
if (report.applied) {
console.log(`\nInserted ${report.inserted} row(s).\n`);
} else if (report.anomalies.length > 0 && has("apply")) {
console.log("\nNOT APPLIED — anomalies above. Read them, then re-run with --force.\n");
process.exit(1);
} else {
console.log("\nDRY RUN — nothing written. Re-run with --apply to insert.\n");
}