Import Frollo account feeds for the accounts statements don't cover
ci / lint-test (push) Successful in 47s
ci / lint-test (push) Successful in 47s
Credit cards keep arriving as monthly statements and stay the source of truth for them. This covers the other fourteen accounts, whose statements arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise including the income account, Up, ING, and the small transaction accounts. About 50 rows a month, where the alternative is downloading each statement by hand. The file needs three defences, all found by diffing three real exports (smarthome DECISIONS.md ING-11): A CDR re-consent makes Frollo re-ingest an account's whole history under fresh transaction ids while the originals survive, and consents expire annually. On this export 112 rows were such twins, and every HDR salary payment appeared twice — importing blind doubles reported income. dedupe() collapses each natural-key group to its lowest id, lowest because old ids were a strict subset of new across two exports, so source_ref stays stable and a re-import inserts nothing. An earlier version of that rule kept close-id rows on a 10,000 threshold, reasoning that genuine same-day repeats have consecutive ids. Verifying it against the income rows killed it: in-scope duplicate pairs have id gaps from 58 to 260 million, so no threshold separates them from the gaps of 1-4 that real repeats showed. It now collapses unconditionally and flags anything within 10 for review — the errors are asymmetric, and nothing in scope has ever tripped the flag. A lapsed consent removes an account from the export silently, with no error and no marker; the row count just drops. So the import asserts the account roster and refuses to run when a configured account contributes nothing. Also holds these rows out of the pending-reconciliation queue. A feed row is the account's own ledger entry, not a receipt awaiting a statement line — these accounts' statements are deliberately not imported — so without the exclusion 550 rows a year would bury the receipts that need a decision. The queue stays at 8 instead of 558. Foreign rows follow order-ingestion's existing shape: amount is the native figure, foreign_currency_code names it, amount_aud stays NULL rather than asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as incomplete. Dry run by default. Verified against the real export before applying: 550 rows inserted, 14 accounts, re-run inserts 0.
This commit is contained in:
@@ -0,0 +1,259 @@
|
||||
/**
|
||||
* Imports a Frollo transaction export into `transactions`.
|
||||
*
|
||||
* node --experimental-strip-types scripts/import-frollo.mts --file <csv>
|
||||
* node --experimental-strip-types scripts/import-frollo.mts --file <csv> --apply
|
||||
*
|
||||
* Dry run by default: it prints exactly what it would do and touches nothing.
|
||||
* Rehearse first — every irreversible step in this repo that skipped that has
|
||||
* cost something, and the failure mode here is doubling reported income.
|
||||
*
|
||||
* Export the file with PENDING TRANSACTIONS EXCLUDED. A pending row changes both
|
||||
* its id and its description when it settles, so importing one guarantees a
|
||||
* duplicate on the next run; and every pending row observed so far has been on a
|
||||
* credit card, which this importer does not cover anyway. See DECISIONS.md
|
||||
* ING-11 in the smarthome repo.
|
||||
*
|
||||
* Needs DATABASE_URL. postgres-personal publishes no host port, so from the host:
|
||||
* export DATABASE_URL="postgresql://personal:<pw>@$(docker inspect postgres-personal \
|
||||
* --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'):5432/personal"
|
||||
* The container IP changes on every recreate.
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import pg from "pg";
|
||||
import { parseCSVRows } from "../src/lib/csv-parser.ts";
|
||||
import {
|
||||
accountFor,
|
||||
accountReport,
|
||||
dedupe,
|
||||
readRows,
|
||||
skipReason,
|
||||
toLedgerRow,
|
||||
unknownAccounts,
|
||||
type FrolloRow,
|
||||
type SkipReason,
|
||||
} from "../src/lib/frollo-csv.ts";
|
||||
|
||||
const SOURCE = "frollo";
|
||||
|
||||
function arg(name: string, fallback?: string): string | undefined {
|
||||
const i = process.argv.indexOf(`--${name}`);
|
||||
if (i >= 0 && process.argv[i + 1] && !process.argv[i + 1].startsWith("--")) {
|
||||
return process.argv[i + 1];
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
const has = (name: string) => process.argv.includes(`--${name}`);
|
||||
|
||||
const file = arg("file");
|
||||
const apply = has("apply");
|
||||
const ownerId = Number(arg("owner", "1"));
|
||||
const allowMissing = has("allow-missing-accounts");
|
||||
|
||||
if (!file) {
|
||||
console.error("usage: --file <csv> [--apply] [--owner <id>] [--allow-missing-accounts]");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const money = (n: number) => n.toLocaleString("en-AU", { minimumFractionDigits: 2, maximumFractionDigits: 2 });
|
||||
|
||||
const rows = readRows(parseCSVRows(readFileSync(file, "utf8")));
|
||||
console.log(`\n${file}`);
|
||||
console.log(` ${rows.length} rows in file\n`);
|
||||
|
||||
// ---- 1. scope ------------------------------------------------------------
|
||||
const skipped = new Map<SkipReason, FrolloRow[]>();
|
||||
const inScope: FrolloRow[] = [];
|
||||
for (const r of rows) {
|
||||
const why = skipReason(r);
|
||||
if (why) {
|
||||
const list = skipped.get(why);
|
||||
if (list) list.push(r);
|
||||
else skipped.set(why, [r]);
|
||||
} else inScope.push(r);
|
||||
}
|
||||
console.log("SCOPE");
|
||||
for (const [why, list] of [...skipped.entries()].sort((a, b) => b[1].length - a[1].length)) {
|
||||
console.log(` skipped ${String(list.length).padStart(5)} ${why}`);
|
||||
}
|
||||
console.log(` in scope ${String(inScope.length).padStart(4)}\n`);
|
||||
|
||||
if (skipped.has("pending")) {
|
||||
console.log(
|
||||
` NOTE: ${skipped.get("pending")!.length} pending row(s) present — the export was taken\n` +
|
||||
` with pending INCLUDED. They are skipped, but re-export with pending\n` +
|
||||
` excluded so successive files are comparable.\n`
|
||||
);
|
||||
}
|
||||
|
||||
// ---- 2. accounts ---------------------------------------------------------
|
||||
// A lapsed CDR consent removes an account from the export silently; the row
|
||||
// count just drops. Assert the roster rather than trusting it.
|
||||
console.log("ACCOUNTS");
|
||||
const report = accountReport(inScope);
|
||||
for (const a of report) {
|
||||
const mark = a.present ? " " : "!";
|
||||
console.log(` ${mark} ${a.spec.last4} ${a.spec.label.padEnd(24)} ${String(a.rows).padStart(5)} rows`);
|
||||
}
|
||||
const missing = report.filter((a) => !a.present);
|
||||
const unknown = unknownAccounts(rows);
|
||||
if (unknown.length > 0) {
|
||||
console.log("\n accounts in the file this importer does not know (skipped):");
|
||||
for (const u of unknown) console.log(` ${u.accountNumber} ${u.accountName} ${u.rows} rows`);
|
||||
}
|
||||
if (missing.length > 0) {
|
||||
console.log(`\n ${missing.length} configured account(s) contributed NO rows:`);
|
||||
for (const m of missing) console.log(` ${m.spec.last4} ${m.spec.label} (${m.spec.provider})`);
|
||||
console.log(" A CDR consent may have lapsed — check before treating this file as complete.");
|
||||
if (!allowMissing) {
|
||||
console.log(" Refusing to continue. Re-run with --allow-missing-accounts if this is expected.\n");
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
console.log();
|
||||
|
||||
// ---- 3. dedupe -----------------------------------------------------------
|
||||
const { kept, dropped } = dedupe(inScope);
|
||||
console.log("DE-DUPLICATION (re-ingest twins from CDR re-consent)");
|
||||
console.log(` ${dropped.length} dropped, ${kept.length} kept`);
|
||||
if (dropped.length > 0) {
|
||||
const byAccount = new Map<string, number>();
|
||||
for (const d of dropped) {
|
||||
const label = accountFor(d.row.accountNumber)?.label ?? "?";
|
||||
byAccount.set(label, (byAccount.get(label) ?? 0) + 1);
|
||||
}
|
||||
for (const [label, n] of [...byAccount.entries()].sort((a, b) => b[1] - a[1])) {
|
||||
console.log(` ${String(n).padStart(4)} ${label}`);
|
||||
}
|
||||
console.log(" examples:");
|
||||
for (const d of dropped.slice(0, 3)) {
|
||||
console.log(
|
||||
` ${d.row.transactionDate} ${money(d.row.amount).padStart(11)} ${d.row.description.slice(0, 34)}` +
|
||||
` id ${d.row.transactionId} duplicates ${d.duplicateOf.transactionId} (gap ${d.idGap})`
|
||||
);
|
||||
}
|
||||
// Ids this close look like a genuine same-day repeat rather than a twin.
|
||||
// Nothing in scope has ever tripped it; if something does, look before
|
||||
// trusting the collapse.
|
||||
const suspect = dropped.filter((d) => d.suspectGenuineRepeat);
|
||||
if (suspect.length > 0) {
|
||||
console.log(`\n REVIEW: ${suspect.length} collapsed row(s) had near-consecutive ids and may be`);
|
||||
console.log(" real repeats rather than duplicates:");
|
||||
for (const d of suspect.slice(0, 10)) {
|
||||
console.log(
|
||||
` ${d.row.transactionDate} ${money(d.row.amount).padStart(11)} ${d.row.description.slice(0, 40)}` +
|
||||
` ids ${d.duplicateOf.transactionId}/${d.row.transactionId} gap ${d.idGap}`
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
console.log();
|
||||
|
||||
// ---- 4. what would change ------------------------------------------------
|
||||
const ledger = kept.map(toLedgerRow);
|
||||
if (!process.env.DATABASE_URL) {
|
||||
console.error("DATABASE_URL is not set — cannot compare against the ledger.");
|
||||
process.exit(2);
|
||||
}
|
||||
const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
|
||||
await client.connect();
|
||||
|
||||
const existing = await client.query<{ source_ref: string }>(
|
||||
"SELECT source_ref FROM transactions WHERE source = $1",
|
||||
[SOURCE]
|
||||
);
|
||||
const seen = new Set(existing.rows.map((r) => r.source_ref));
|
||||
const fresh = ledger.filter((r) => !seen.has(r.sourceRef));
|
||||
const already = ledger.length - fresh.length;
|
||||
|
||||
console.log("LEDGER");
|
||||
console.log(` already imported : ${String(already).padStart(5)}`);
|
||||
console.log(` new to insert : ${String(fresh.length).padStart(5)}`);
|
||||
const debits = fresh.filter((r) => r.transactionType === "debit");
|
||||
const credits = fresh.filter((r) => r.transactionType === "credit");
|
||||
const aud = (rs: typeof fresh) => rs.filter((r) => !r.foreignCurrencyCode);
|
||||
console.log(
|
||||
` of which AUD : ${aud(debits).length} debits ${money(aud(debits).reduce((s, r) => s + r.amount, 0))}` +
|
||||
` / ${aud(credits).length} credits ${money(aud(credits).reduce((s, r) => s + r.amount, 0))}`
|
||||
);
|
||||
const fx = fresh.filter((r) => r.foreignCurrencyCode);
|
||||
if (fx.length > 0) {
|
||||
const byCcy = new Map<string, number>();
|
||||
for (const r of fx) byCcy.set(r.foreignCurrencyCode!, (byCcy.get(r.foreignCurrencyCode!) ?? 0) + 1);
|
||||
console.log(` foreign : ${[...byCcy].map(([c, n]) => `${n} ${c}`).join(", ")} (amount_aud left NULL)`);
|
||||
}
|
||||
if (fresh.length > 0) {
|
||||
console.log("\n first rows to insert:");
|
||||
for (const r of fresh.slice(0, 8)) {
|
||||
console.log(
|
||||
` ${r.transactionDate} ${r.transactionType.padEnd(6)} ${money(r.amount).padStart(11)}` +
|
||||
` ${(r.foreignCurrencyCode ?? "AUD").padEnd(4)} ${r.description.slice(0, 40)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- 5. apply ------------------------------------------------------------
|
||||
if (!apply) {
|
||||
console.log("\nDRY RUN — nothing written. Re-run with --apply to insert.\n");
|
||||
await client.end();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (fresh.length === 0) {
|
||||
console.log("\nNothing to insert.\n");
|
||||
await client.end();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
// row_index is left NULL: it exists to order statement lines, and the
|
||||
// uniqueness this import relies on is uq_transaction_source_ref, not
|
||||
// uq_transaction_identity.
|
||||
//
|
||||
// category is not set, because Frollo's own categories are not trustworthy. Note
|
||||
// what actually lands: trg_transactions_normalize_category rewrites NULL to
|
||||
// 'other' on insert, so these rows arrive categorised 'other' rather than
|
||||
// uncategorised. That is equivalent for display and for the rules engine, which
|
||||
// writes transaction_overrides.category_override and is read ahead of
|
||||
// t.category by EFFECTIVE_CATEGORY — so a rule still wins.
|
||||
let inserted = 0;
|
||||
try {
|
||||
await client.query("BEGIN");
|
||||
for (const r of fresh) {
|
||||
const res = await client.query(
|
||||
`INSERT INTO transactions
|
||||
(statement_id, owner_id, transaction_date, description, amount, transaction_type,
|
||||
merchant_name, foreign_currency_amount, foreign_currency_code,
|
||||
source, source_ref, source_account)
|
||||
VALUES (NULL, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
|
||||
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL
|
||||
DO NOTHING`,
|
||||
[
|
||||
ownerId,
|
||||
r.transactionDate,
|
||||
r.description,
|
||||
r.amount,
|
||||
r.transactionType,
|
||||
r.merchantName,
|
||||
r.foreignCurrencyAmount,
|
||||
r.foreignCurrencyCode,
|
||||
SOURCE,
|
||||
r.sourceRef,
|
||||
r.sourceAccount,
|
||||
]
|
||||
);
|
||||
inserted += res.rowCount ?? 0;
|
||||
}
|
||||
await client.query("COMMIT");
|
||||
} catch (e) {
|
||||
await client.query("ROLLBACK");
|
||||
console.error("\nROLLED BACK:", e);
|
||||
await client.end();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const after = await client.query<{ n: string }>(
|
||||
"SELECT count(*) AS n FROM transactions WHERE source = $1",
|
||||
[SOURCE]
|
||||
);
|
||||
console.log(`\nInserted ${inserted} row(s). ${after.rows[0].n} Frollo rows now in the ledger.\n`);
|
||||
await client.end();
|
||||
Reference in New Issue
Block a user