siddharthd/finance-app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

csv import: exclude what the statements already cover, and delete the Frollo importer
ci / lint-test (push) Successful in 43s
Details

Browse Source 
"This is becoming too complex... Frollo should be done through that [the manual
CSV import]" (owner). It was right: a bespoke importer, an API route, a CLI, two
scheduled n8n workflows and a shared secret existed to do what the CSV import
modal already did, minus one rule.

That rule is statement coverage, and it turns out to be the whole thing. Applying
each account's newest billing_end_date as a watermark takes the real 2,607-row
Frollo export down to 171 rows — with no account allowlist, no credit-card
exclusion and no Frollo-specific scoping at all. Cards drop out on their own
because their statements are current; the 46 card rows that survive are genuinely
post-statement. Every bit of the bespoke apparatus was doing by hand what one
query does generically.

Deleted: src/lib/frollo-csv.ts, src/lib/frollo-ingest.ts, scripts/import-frollo.mts,
src/app/api/frollo/, both test files, the FROLLO_INGEST_TOKEN wiring, and the n8n
Frollo Import + Frollo Freshness Check workflows.

Added to the shared CSV path, so every import benefits:

  - getStatementCoverage() + /api/import/statement-coverage. The review step
    leaves out rows an account's statements already cover and says how many, with
    the rows one click away. Only applies when an account column is mapped and
    that account has statements — a row is never dropped on a guess.
  - Optional Account and Row ID columns in the mapper. Account drives the
    watermark and is stored as source_account; Row ID becomes source_ref.
  - An in-file duplicate warning. A CDR re-consent re-exports history under
    fresh ids, so source_ref cannot see it — 385 twins in one 2,563-row export
    doubled every salary payment, and that is not visible by eye in a review
    table.

Two pre-existing bugs in that path, both of which this plan depends on:

  - The category chosen in the review step was accepted by
    batchInsertCSVTransactions and then left out of the INSERT column list, so it
    was silently discarded and the trigger wrote 'other'. Not cosmetic: an
    uncategorised credit is admitted by NET_SPEND_ROWS and negated by
    SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of spend while
    counting as no income.
  - The path had no idempotency whatsoever. row_index is assigned MAX+1 on every
    run, which makes uq_transaction_identity structurally unable to fire, so a
    second import of the same file duplicated all of it. Now writes source +
    source_ref with ON CONFLICT DO NOTHING.

awaitsStatementLine()'s removal note is kept but rewritten: it no longer points
at a deleted file, and the lesson stands — the queue jump from 8 to 558 was the
measurement, not the noise.
This commit is contained in:
siddharthd
2026-08-13 12:58:06 +10:00
parent b82c4570bd
commit 461c021e7a
13 changed files with 453 additions and 1605 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lib/csv-parser.ts
+78
View File
@@ -9,6 +9,22 @@ export interface ColumnMapping {
creditCol?: string;
merchantCol?: string;
categoryCol?: string;
/**
* Column holding the account number, when the file covers more than one.
*
* Optional and usually absent — a bank's own export is one account per file.
* An aggregator export (Frollo) is not, and without this every row would be
* measured against the same statement coverage.
*/
accountCol?: string;
/**
* Column holding the provider's own row id, when it has one.
*
* Stored as `source_ref` so re-importing the same file inserts nothing. The
* importer has no other idempotency: `row_index` is assigned MAX+1 on every
* run, which makes `uq_transaction_identity` structurally unable to fire.
*/
sourceRefCol?: string;
}
export interface BankPreset {
@@ -24,6 +40,10 @@ export interface ParsedTransaction {
transaction_type: string;
merchant_name?: string;
category?: string;
/** Raw account identifier from the file, when a column was mapped. */
account?: string;
/** Provider row id, when a column was mapped. */
source_ref?: string;
}
export function parseCSVRows(text: string): string[][] {
@@ -108,6 +128,8 @@ export function applyMapping(
const descIdx = idx(mapping.descriptionCol);
const merchantIdx = mapping.merchantCol ? idx(mapping.merchantCol) : -1;
const categoryIdx = mapping.categoryCol ? idx(mapping.categoryCol) : -1;
const accountIdx = mapping.accountCol ? idx(mapping.accountCol) : -1;
const sourceRefIdx = mapping.sourceRefCol ? idx(mapping.sourceRefCol) : -1;
const results: ParsedTransaction[] = [];
for (const row of dataRows) {
@@ -138,6 +160,8 @@ export function applyMapping(
const tx: ParsedTransaction = { date, description, amount, transaction_type };
if (merchantIdx >= 0 && row[merchantIdx]) tx.merchant_name = row[merchantIdx].trim();
if (categoryIdx >= 0 && row[categoryIdx]) tx.category = row[categoryIdx].trim();
if (accountIdx >= 0 && row[accountIdx]) tx.account = row[accountIdx].trim();
if (sourceRefIdx >= 0 && row[sourceRefIdx]) tx.source_ref = row[sourceRefIdx].trim();
results.push(tx);
}
return results;
@@ -154,3 +178,57 @@ export function loadBankPresets(): BankPreset[] {
try { return JSON.parse(localStorage.getItem("csv-presets") || "[]"); }
catch { return []; }
}
/** Last four digits of an account identifier, however it is written. */
export function last4(s: string): string {
const d = s.replace(/\D/g, "");
return d.length >= 4 ? d.slice(-4) : "";
}
/**
* Splits parsed rows into what to import and what the statements already have.
*
* A statement's end date is a hard watermark: everything on that account up to
* it is already in the ledger, itemised and converted. Without this the first
* aggregator import wrote 422 duplicates out of 550 rows; with it the same
* 2,607-row export offers 171.
*
* Rows are only excluded when an account column was mapped AND that account has
* statements. Anything else is imported — a row is never dropped on a guess.
*/
export function splitByCoverage(
rows: ParsedTransaction[],
coverage: { last4: string; coveredTo: string }[]
): { keep: ParsedTransaction[]; covered: ParsedTransaction[] } {
const wm = new Map(coverage.map((c) => [c.last4, c.coveredTo]));
const keep: ParsedTransaction[] = [];
const covered: ParsedTransaction[] = [];
for (const r of rows) {
const to = r.account ? wm.get(last4(r.account)) : undefined;
// Both are YYYY-MM-DD, so a string compare is a date compare.
if (to && r.date <= to) covered.push(r);
else keep.push(r);
}
return { keep, covered };
}
/**
* Rows that duplicate another row in the SAME file.
*
* Distinct from the coverage check, and not catchable by `source_ref`: a CDR
* re-consent makes the provider re-export an account's whole history under
* fresh ids while the originals survive, so the twins are identical in
* everything except the id. That put 385 duplicates in one 2,563-row export and
* doubled every salary payment. 385 rows is not something you spot by eye in a
* review table, so it is counted here.
*/
export function inFileDuplicates(rows: ParsedTransaction[]): number {
const seen = new Set<string>();
let n = 0;
for (const r of rows) {
const key = [r.date, r.description, r.amount, r.transaction_type, r.account ?? ""].join("|");
if (seen.has(key)) n += 1;
else seen.add(key);
}
return n;
}
src/lib/frollo-csv.ts
-382
View File
@@ -1,382 +0,0 @@
/**
* Frollo transaction-export parsing and de-duplication.
*
* Frollo is a CDR (Open Banking) aggregator. Its app emails a CSV of up to 12
* months of transactions. This module turns that file into rows ready for
* `transactions`, and — mostly — decides which of them are real.
*
* Scope is deliberately narrow: credit cards are excluded because they already
* arrive as monthly statements, which are authoritative. What Frollo is for is
* the accounts whose statements arrive every 182 to 460 days (see
* ACCOUNTS below), where the alternative is downloading each one by hand.
*
* The file has three properties that any consumer has to handle, all learned by
* diffing three real exports (smarthome DECISIONS.md ING-11):
*
* 1. A CDR re-consent makes Frollo re-ingest the account's whole history under
* fresh transaction ids, while the originals survive. So the same purchase
* appears twice with different ids, and consents expire annually — this is
* not a one-off. On one 1,989-row export, 385 rows were such twins, and Wise
* credits were overstated by exactly 100%.
* 2. `transaction_id` is otherwise stable: across two exports 12 days apart,
* 1,462 of 1,532 shared rows kept identical ids and none were replaced. It
* is therefore a good `source_ref` — it just cannot detect a twin.
* 3. The export cannot express "pending", and the tell differs per bank
* (Westpac prefixes the description; Amex emits no `posted_date` or
* `transaction_type` at all, so its pending rows are indistinguishable from
* its posted ones). The export's own option is the only reliable control.
* Every pending row observed was on a credit card, i.e. out of scope, so
* exporting with pending EXCLUDED costs nothing here — and it avoids the
* re-key problem, since a settling row changes both its id and its
* description.
*/
/** A row as it appears in the CSV, after header mapping. */
export interface FrolloRow {
transactionId: string;
description: string;
amount: number;
currency: string;
transactionDate: string;
postedDate: string | null;
accountNumber: string;
accountName: string;
creditDebit: string;
transactionType: string | null;
providerName: string;
merchantName: string | null;
}
/** A row shaped for insertion into `transactions`. */
export interface LedgerRow {
sourceRef: string;
sourceAccount: string;
transactionDate: string;
description: string;
/** Always positive; direction lives in `transactionType`. */
amount: number;
transactionType: "debit" | "credit";
merchantName: string | null;
foreignCurrencyAmount: number | null;
foreignCurrencyCode: string | null;
}
export interface AccountSpec {
/** Last four characters of the digits in the masked account number. */
last4: string;
label: string;
provider: string;
}
/**
* Accounts this importer owns.
*
* Keyed on the last four digits rather than the masked string, because the
* masking format varies by institution (`xxx-xxx xxxx1910`, `xxxxxx xxxxx6264`,
* `x xxxx 0887`) and is a presentation choice Frollo could change. All fourteen
* are distinct, as are the five excluded cards.
*
* Credit cards are absent on purpose — monthly statements already cover them and
* are the source of truth. Adding one here would create a second producer for
* rows the reconcile queue would then have to match against themselves.
*/
export const ACCOUNTS: AccountSpec[] = [
{ last4: "6264", label: "ANZ Access Advantage", provider: "ANZ" },
{ last4: "9940", label: "AMP (mortgage)", provider: "AMP" },
{ last4: "8109", label: "AMP", provider: "AMP" },
{ last4: "0682", label: "AMP (loan)", provider: "AMP" },
{ last4: "4830", label: "Wise (income)", provider: "Wise" },
{ last4: "2176", label: "Wise", provider: "Wise" },
{ last4: "0052", label: "Up Spending", provider: "Up" },
{ last4: "0078", label: "Up Savings", provider: "Up" },
{ last4: "1910", label: "CommBank Smart Access", provider: "CommBank" },
{ last4: "4878", label: "Westpac Choice", provider: "Westpac" },
{ last4: "4758", label: "Westpac Business One", provider: "Westpac" },
{ last4: "0887", label: "ING Orange Everyday", provider: "ING" },
{ last4: "9336", label: "ING Savings Maximiser", provider: "ING" },
{ last4: "9807", label: "ANZ Online Saver", provider: "ANZ" },
];
/**
* Accounts knowingly left out, so "excluded" is distinguishable from "unknown".
*
* Without this the five credit cards read as unrecognised accounts on every
* single run, which would raise the new-account anomaly every time and make an
* automatic import permanently refuse. An alert that always fires is an alert
* nobody reads.
*
* Anything in neither list is genuinely new — a freshly connected account, or a
* re-masked number — and does deserve a look before its rows are ignored.
*/
export const EXCLUDED_ACCOUNTS: AccountSpec[] = [
{ last4: "3893", label: "CommBank Ultimate Awards (card)", provider: "CommBank" },
{ last4: "8032", label: "Westpac Altitude Qantas Black (card)", provider: "Westpac" },
{ last4: "1000", label: "Amex Platinum Business (card)", provider: "American Express" },
{ last4: "0351", label: "HSBC Star Alliance (card)", provider: "HSBC" },
{ last4: "6227", label: "MyCard Prestige (card)", provider: "Mycard" },
];
export function isKnownExcluded(accountNumber: string): boolean {
const key = last4(accountNumber);
return EXCLUDED_ACCOUNTS.some((a) => a.last4 === key);
}
/**
* Below this id gap, two rows sharing a natural key are *probably* a genuine
* same-day repeat rather than a re-ingest twin — and are reported rather than
* silently collapsed.
*
* This is a review trigger, not a decision. An earlier version used a 10,000
* threshold to *keep* close-id rows, and it was wrong: measured across two real
* exports, in-scope duplicate pairs have gaps from **58 to 260 million**, while
* the only observed genuine repeats (five identical $1.00 Apple charges) had
* gaps of 1 to 4. A threshold anywhere in between is a magic number, and the
* error it enables is asymmetric — keeping a twin doubled reported income by
* 100%, whereas collapsing a real repeat understates by one row.
*
* So dedupe collapses unconditionally and flags anything this close for a human
* to look at. On the accounts in scope, nothing has ever tripped it: every
* duplicate group is a pair and the minimum gap is 58.
*/
export const SUSPECT_REPEAT_GAP = 10;
export function last4(accountNumber: string): string {
return accountNumber.replace(/\D/g, "").slice(-4);
}
export function accountFor(accountNumber: string): AccountSpec | undefined {
const key = last4(accountNumber);
return ACCOUNTS.find((a) => a.last4 === key);
}
const REQUIRED_COLUMNS = [
"transaction_id",
"description",
"amount",
"currency",
"transaction_date",
"account_number",
"account_name",
"credit_debit",
];
/**
* Maps a parsed CSV grid to rows, by header name rather than position.
*
* Throws on a missing required column rather than reading undefined into a
* transaction: a column rename upstream should stop the import loudly, not
* silently produce rows with blank descriptions.
*/
export function readRows(grid: string[][]): FrolloRow[] {
if (grid.length === 0) throw new Error("empty CSV");
const header = grid[0].map((h) => h.trim());
const missing = REQUIRED_COLUMNS.filter((c) => !header.includes(c));
if (missing.length > 0) {
throw new Error(`Frollo CSV missing expected column(s): ${missing.join(", ")}`);
}
const at = (row: string[], col: string): string => {
const i = header.indexOf(col);
return i >= 0 ? (row[i] ?? "").trim() : "";
};
const out: FrolloRow[] = [];
for (const row of grid.slice(1)) {
if (row.every((f) => f === "")) continue;
const amount = Number(at(row, "amount"));
out.push({
transactionId: at(row, "transaction_id"),
description: at(row, "description"),
amount: Number.isFinite(amount) ? amount : NaN,
currency: at(row, "currency") || "AUD",
transactionDate: at(row, "transaction_date"),
postedDate: at(row, "posted_date") || null,
accountNumber: at(row, "account_number"),
accountName: at(row, "account_name"),
creditDebit: at(row, "credit_debit"),
transactionType: at(row, "transaction_type") || null,
providerName: at(row, "provider_name"),
merchantName: at(row, "merchant_name") || null,
});
}
return out;
}
export type SkipReason =
| "zero_amount"
| "unparseable_amount"
| "out_of_scope_account"
| "pending";
/**
* Why a row is not imported, or null if it should be.
*
* `zero_amount` is mostly CommBank fee *waivers* — 574 of them in one export,
* genuinely $0.00 with the notional fee written into the description text
* (`INTNL TRANSACTION FEE, $ 41.58 FEE SAV`). They are not transactions.
*
* `pending` is a belt-and-braces check on top of the export option. It only
* catches Westpac's description prefix; Amex pending rows are undetectable here,
* which is why the option matters and this does not replace it.
*/
export function skipReason(row: FrolloRow): SkipReason | null {
if (!Number.isFinite(row.amount)) return "unparseable_amount";
if (row.amount === 0) return "zero_amount";
// Pending is tested before account scope on purpose. Every pending row seen so
// far has been on a credit card, so scoping first would classify them all as
// out-of-scope and the "this export was taken with pending INCLUDED" signal
// would never fire — even though it is worth acting on wherever it appears,
// because the option makes successive exports incomparable.
if (/^\s*pending\s*[-:]/i.test(row.description)) return "pending";
if (!accountFor(row.accountNumber)) return "out_of_scope_account";
return null;
}
/** Natural key: what makes two rows the same purchase, ignoring ids. */
export function naturalKey(row: FrolloRow): string {
return [
last4(row.accountNumber),
row.transactionDate,
row.amount.toFixed(2),
row.description,
row.currency,
].join("");
}
export interface DroppedRow {
row: FrolloRow;
duplicateOf: FrolloRow;
idGap: number;
/** Ids close enough that this might be a real repeat, not a twin. Review it. */
suspectGenuineRepeat: boolean;
}
export interface DedupeResult {
kept: FrolloRow[];
dropped: DroppedRow[];
}
/**
* Collapses each natural-key group to its lowest-id row.
*
* Lowest because old ids survive a re-ingestion — verified across two exports
* twelve days apart, where the earlier ids were a strict subset of the later
* ones. Keeping the lowest therefore keeps `source_ref` stable from run to run,
* so a re-import updates nothing and inserts nothing.
*
* Everything collapsed is returned in `dropped`, and anything whose ids sit
* within SUSPECT_REPEAT_GAP is flagged: that is the shape a genuine same-day
* repeat would take, and it should be looked at rather than assumed.
*/
export function dedupe(rows: FrolloRow[], suspectGap = SUSPECT_REPEAT_GAP): DedupeResult {
const groups = new Map<string, FrolloRow[]>();
for (const row of rows) {
const k = naturalKey(row);
const g = groups.get(k);
if (g) g.push(row);
else groups.set(k, [row]);
}
const kept: FrolloRow[] = [];
const dropped: DroppedRow[] = [];
for (const group of groups.values()) {
if (group.length === 1) {
kept.push(group[0]);
continue;
}
const sorted = [...group].sort(
(a, b) => Number(a.transactionId) - Number(b.transactionId)
);
const winner = sorted[0];
kept.push(winner);
for (let i = 1; i < sorted.length; i++) {
const idGap = Number(sorted[i].transactionId) - Number(sorted[i - 1].transactionId);
dropped.push({
row: sorted[i],
duplicateOf: winner,
idGap,
suspectGenuineRepeat: idGap < suspectGap,
});
}
}
return { kept, dropped };
}
/**
* Shapes a row for `transactions`.
*
* `category` is deliberately not set. Frollo assigns its own categories and the
* owner reports they are often wrong; finance-app's rules engine plus
* `category_override` already decide this, and a third untrusted source
* competing with both is worse than none.
*
* For a non-AUD row the file gives only the foreign figure — there is no AUD
* equivalent anywhere in the export. That is the same position order-ingestion
* is in, so it takes the same shape: `amount` IS the native figure,
* `foreign_currency_code` names it, and `amount_aud` is left NULL rather than
* asserting a rate we do not have. `AMOUNT_UNCONVERTED` in analytics-sql then
* counts these and the UI reports the balance as incomplete.
*/
export function toLedgerRow(row: FrolloRow): LedgerRow {
const foreign = row.currency !== "AUD";
const magnitude = Math.abs(row.amount);
return {
sourceRef: row.transactionId,
sourceAccount: row.accountNumber,
transactionDate: row.transactionDate,
description: row.description,
amount: magnitude,
transactionType: row.amount < 0 ? "debit" : "credit",
// Frollo writes "Unknown" where it could not resolve a merchant. Storing
// that would put the literal word on the screen; NULL lets the existing
// merchant fallbacks work.
merchantName: row.merchantName && row.merchantName !== "Unknown" ? row.merchantName : null,
foreignCurrencyAmount: foreign ? magnitude : null,
foreignCurrencyCode: foreign ? row.currency : null,
};
}
export interface AccountReport {
spec: AccountSpec;
rows: number;
present: boolean;
}
/**
* Which configured accounts the file actually contains.
*
* A lapsed CDR consent does not error — the account simply stops appearing, and
* the row count drops. Nothing else in the pipeline would notice, so the import
* asserts the roster rather than trusting it.
*/
export function accountReport(rows: FrolloRow[]): AccountReport[] {
const counts = new Map<string, number>();
for (const r of rows) {
const k = last4(r.accountNumber);
counts.set(k, (counts.get(k) ?? 0) + 1);
}
return ACCOUNTS.map((spec) => ({
spec,
rows: counts.get(spec.last4) ?? 0,
present: (counts.get(spec.last4) ?? 0) > 0,
}));
}
/**
* Accounts in the file that are neither imported nor knowingly excluded.
*
* Empty is the normal state. A name appearing here means a new account reached
* the export and its rows are being dropped on the floor — worth a look before
* that becomes the status quo.
*/
export function unknownAccounts(rows: FrolloRow[]): { accountNumber: string; accountName: string; rows: number }[] {
const seen = new Map<string, { accountNumber: string; accountName: string; rows: number }>();
for (const r of rows) {
if (accountFor(r.accountNumber) || isKnownExcluded(r.accountNumber)) continue;
const k = last4(r.accountNumber);
const e = seen.get(k);
if (e) e.rows++;
else seen.set(k, { accountNumber: r.accountNumber, accountName: r.accountName, rows: 1 });
}
return [...seen.values()].sort((a, b) => b.rows - a.rows);
}
src/lib/frollo-ingest.ts
-384
View File
@@ -1,384 +0,0 @@
import {
accountReport,
dedupe,
last4,
readRows,
skipReason,
toLedgerRow,
unknownAccounts,
type AccountReport,
type FrolloRow,
type SkipReason,
} from "./frollo-csv.ts";
import { parseCSVRows } from "./csv-parser.ts";
/**
* Turns a Frollo CSV into ledger rows and (optionally) writes them.
*
* Shared by `scripts/import-frollo.mts` and `POST /api/frollo/ingest` so the
* hand-run and automatic paths cannot drift. They previously would have been two
* implementations of the same insert, which is exactly how the pantry healthcheck
* came to be fixed in one repo and left broken in the other.
*
* The database is reached through an injected executor rather than imported
* directly: the API route runs inside Next with Prisma, while the CLI runs under
* `node --experimental-strip-types` with a plain `pg` client and no path aliases.
*/
// The .ts extensions above are load-bearing: this module is imported both by
// Next (which resolves either form) and by scripts/import-frollo.mts running
// under `node --experimental-strip-types`, whose ESM resolver requires the
// extension written out. tsconfig sets allowImportingTsExtensions for this.
export type SqlExecutor = <T = unknown>(sql: string, params: unknown[]) => Promise<T[]>;
export const SOURCE = "frollo";
/**
* Above this many new rows, an automatic run stops and asks.
*
* Steady state is ~50 rows a month. A batch several times that means something
* changed — a re-consent duplicating history, a longer export window, or a new
* account — and none of those should land unseen. The first (backfill) import
* was 550 and was run by hand with `force`.
*/
export const LARGE_BATCH = 200;
/**
* How far a Frollo row may sit from a ledger row and still be the same event.
*
* The feed dates a transaction when the provider posted it; a statement dates it
* when the bank did. Those differ by a day or two around weekends. Three days is
* wide enough to catch that and narrow enough that two genuinely different
* charges for the identical amount in the same week are rare — and when they do
* collide, the cost is one missing row in a feed whose whole job is provisional
* visibility, against a permanent double-count the other way.
*/
export const LEDGER_MATCH_DAYS = 3;
/**
* Ledger transaction types that mean money arriving.
*
* The whole set in use is debit | credit | payment | fee | interest | refund.
* `refund` is the one that bites: the feed calls a reversed account fee a
* `credit` while the statement importer types it `refund`, so classifying it as
* an outflow left five real duplicates behind — every ANZ servicing-fee reversal
* in the file. Small money, but it is the same shape of mismatch that would
* matter on a large refund.
*/
const INFLOW_TYPES = new Set(["credit", "refund"]);
/**
* A date column, as midnight UTC, from whatever the driver handed back.
*
* `pg` maps a Postgres DATE to a JS `Date` in local time, while Prisma and the
* CSV both yield `"YYYY-MM-DD"` strings. Reading one as the other is silent:
* `String(new Date()).slice(0, 10)` is `"Wed Mar 10"`, which parses to NaN, and
* a guard built on it skips every row it was meant to compare — the first dry
* run of this code reported 550 rows to insert and zero duplicates against a
* ledger holding 422 of them.
*/
function dayMs(v: unknown): number {
if (v instanceof Date) return Date.UTC(v.getFullYear(), v.getMonth(), v.getDate());
const m = /^(\d{4})-(\d{2})-(\d{2})/.exec(String(v));
return m ? Date.UTC(Number(m[1]), Number(m[2]) - 1, Number(m[3])) : NaN;
}
export interface IngestOptions {
ownerId?: number;
/** Write. Without it, everything is computed and nothing is inserted. */
apply?: boolean;
/** Proceed despite anomalies. For a human who has read them. */
force?: boolean;
largeBatch?: number;
}
export interface IngestReport {
totalRows: number;
inScope: number;
skipped: Partial<Record<SkipReason, number>>;
accounts: AccountReport[];
missingAccounts: string[];
unknownAccounts: { accountNumber: string; accountName: string; rows: number }[];
duplicatesDropped: number;
suspectRepeats: { description: string; transactionDate: string; amount: number; idGap: number }[];
alreadyImported: number;
/**
* Rows dropped because a statement already covers that account up to that
* date. The primary guard — see `statementWatermarks`.
*/
coveredByStatement: number;
/** Per-account coverage boundary used above, for the report. */
statementWatermarks: { last4: string; coveredTo: string }[];
/**
* Rows dropped because the ledger holds the same amount, same direction,
* within a few days. The secondary guard, for accounts with no statement.
*
* Both counts are reported, never silent: a Frollo row vanishing can mean
* several things and one of them is worth following up.
*/
ledgerDuplicates: number;
toInsert: number;
inserted: number;
/** Non-empty means an automatic run should stop and a human should look. */
anomalies: string[];
applied: boolean;
}
/**
* Reasons an automatic import should stop rather than write.
*
* Pure, so it is testable without a database. Every entry is a thing that has
* either happened or come one verification away from happening: a lapsed CDR
* consent silently shrinking the file, a new account appearing unnoticed, a
* de-duplication that might have collapsed a real repeat, and a batch far larger
* than the account activity can explain.
*/
export function findAnomalies(
report: Pick<IngestReport, "missingAccounts" | "unknownAccounts" | "suspectRepeats" | "toInsert" | "skipped">,
largeBatch = LARGE_BATCH
): string[] {
const out: string[] = [];
if (report.missingAccounts.length > 0) {
out.push(
`${report.missingAccounts.length} configured account(s) contributed no rows ` +
`(${report.missingAccounts.join(", ")}) — a CDR consent may have lapsed`
);
}
if (report.unknownAccounts.length > 0) {
out.push(
`${report.unknownAccounts.length} unknown account(s) in the file, skipped ` +
`(${report.unknownAccounts.map((a) => a.accountName).join(", ")})`
);
}
if (report.suspectRepeats.length > 0) {
out.push(
`${report.suspectRepeats.length} de-duplicated row(s) had near-consecutive ids ` +
`and may be genuine repeats rather than twins`
);
}
if (report.toInsert > largeBatch) {
out.push(`${report.toInsert} new rows is more than the expected ${largeBatch}`);
}
if ((report.skipped.pending ?? 0) > 0) {
out.push(
`${report.skipped.pending} pending row(s) present — export was taken with pending ` +
`INCLUDED; they are skipped, but re-export with it excluded`
);
}
return out;
}
export async function ingestFrolloCsv(
csvText: string,
exec: SqlExecutor,
opts: IngestOptions = {}
): Promise<IngestReport> {
const ownerId = opts.ownerId ?? 1;
const rows = readRows(parseCSVRows(csvText));
const skipped: Partial<Record<SkipReason, number>> = {};
const inScope: FrolloRow[] = [];
for (const r of rows) {
const why = skipReason(r);
if (why) skipped[why] = (skipped[why] ?? 0) + 1;
else inScope.push(r);
}
const accounts = accountReport(inScope);
const missingAccounts = accounts.filter((a) => !a.present).map((a) => `${a.spec.last4} ${a.spec.label}`);
const unknown = unknownAccounts(rows);
const { kept, dropped } = dedupe(inScope);
const suspectRepeats = dropped
.filter((d) => d.suspectGenuineRepeat)
.map((d) => ({
description: d.row.description,
transactionDate: d.row.transactionDate,
amount: d.row.amount,
idGap: d.idGap,
}));
const ledger = kept.map(toLedgerRow);
const existing = await exec<{ source_ref: string }>(
"SELECT source_ref FROM transactions WHERE source = $1",
[SOURCE]
);
const seen = new Set(existing.map((r) => r.source_ref));
const notYetImported = ledger.filter((r) => !seen.has(r.sourceRef));
// GUARD 1 — statement coverage. Do not import what the statements already
// have.
//
// A statement's `billing_end_date` is a hard watermark: everything on that
// account up to that date is already in the ledger, itemised and converted.
// Amount-matching cannot replace this, because the two sources decompose the
// same event differently — Frollo bundles the Wise fee into the transfer
// (10001.13) where the statement itemises it (10000.00 + 1.13), so 38 Wise USD
// rows survived the amount guard as "new" while being the same money.
//
// This is the coverage test the first import needed and got wrong. It asked
// whether a row's date fell inside a statement's min–max *window*, which for
// periods spanning 182 to 460 days swallows a year and answers nothing. The
// watermark asks a different and answerable question: up to what date is this
// account complete?
//
// Matching is on last4 alone. Verified against the live statement set: the
// eight in-scope accounts with statements each map to exactly one bank, no
// cross-bank collision. The per-account watermarks are reported so a wrong one
// is visible rather than inferred from a row count.
const coverage = await exec<{ last4: string; covered_to: unknown }>(
`SELECT right(regexp_replace(account_number, '[^0-9]', '', 'g'), 4) AS last4,
MAX(billing_end_date) AS covered_to
FROM statements
WHERE account_number IS NOT NULL
GROUP BY 1`,
[]
);
const watermark = new Map<string, number>();
for (const c of coverage) {
if (!c.last4) continue; // Wise files one statement as account_number 'N/A'
const t = dayMs(c.covered_to);
if (!Number.isFinite(t)) continue;
watermark.set(c.last4, Math.max(watermark.get(c.last4) ?? -Infinity, t));
}
let coveredByStatement = 0;
const uncovered = notYetImported.filter((r) => {
const w = watermark.get(last4(r.sourceAccount));
if (w === undefined) return true; // no statement for this account, ever
if (dayMs(r.transactionDate) <= w) {
coveredByStatement += 1;
return false;
}
return true;
});
const statementWatermarks = [...watermark.entries()]
.filter(([l4]) => notYetImported.some((r) => last4(r.sourceAccount) === l4))
.map(([l4, t]) => ({ last4: l4, coveredTo: new Date(t).toISOString().slice(0, 10) }))
.sort((a, b) => a.last4.localeCompare(b.last4));
// GUARD 2 — drop rows the ledger already holds from a statement.
//
// This guard is the whole lesson of the 2026-08-13 first import. The feed was
// scoped to "accounts that issue no monthly statement", but that was asserted,
// never tested against the ledger — and it was wrong for almost every account:
// 422 of the 550 rows imported (77%) had a statement twin within three days,
// $1,023,824.63 of movement counted twice. The HDR Global salary showed it
// most plainly, appearing as both A$15,518.53 (statement, converted) and
// US$10,782.00 (feed, native) for the same July payment.
//
// The check that would have caught it is one query and takes a second. The
// check that was run instead compared each row's date against the statement's
// min-max window, which for accounts whose statements span 182 to 460 days
// swallows a year and cannot distinguish covered from uncovered at all.
const dates = uncovered.map((r) => dayMs(r.transactionDate)).filter(Number.isFinite);
let ledgerDuplicates = 0;
let fresh = uncovered;
if (dates.length > 0) {
const pad = LEDGER_MATCH_DAYS * 86_400_000;
const from = new Date(Math.min(...dates) - pad).toISOString().slice(0, 10);
const to = new Date(Math.max(...dates) + pad).toISOString().slice(0, 10);
// Superseded rows are excluded: one has already been replaced by the row
// that supersedes it, so matching against both would hide a genuine gap.
const priorRows = await exec<{ transaction_date: string; amount: string; transaction_type: string }>(
`SELECT transaction_date, amount, transaction_type
FROM transactions
WHERE (source IS NULL OR source <> $1)
AND superseded_by_id IS NULL
AND transaction_date BETWEEN $2::date AND $3::date`,
[SOURCE, from, to]
);
// Keyed on amount AND direction.
//
// Amount is the field both sides agree on exactly — descriptions do not
// survive the trip (the feed writes "HDR Global Services (Bermuda)" where
// the statement writes "Received money from HDR Global Services (Bermuda)
// with reference ...") and dates drift by a day or two.
//
// Direction has to be in the key because this ledger is full of internal
// transfers between the owner's own accounts, and both legs are in the feed:
// 2026-05-18 carries a +3076.04 credit into ANZ and a -3076.04 debit out of
// AMP. On amount alone the credit leg consumed the ledger's debit row and
// the genuine duplicate was written — 20 rows got in this way on the first
// corrected run. A credit is never the duplicate of a debit.
const key = (amount: number, inflow: boolean) => `${Math.abs(amount).toFixed(2)}:${inflow ? "in" : "out"}`;
const byAmount = new Map<string, number[]>();
for (const p of priorRows) {
const k = key(Number(p.amount), INFLOW_TYPES.has(p.transaction_type));
const t = dayMs(p.transaction_date);
if (!Number.isFinite(t)) continue;
const list = byAmount.get(k);
if (list) list.push(t);
else byAmount.set(k, [t]);
}
fresh = uncovered.filter((r) => {
const candidates = byAmount.get(key(r.amount, r.transactionType === "credit"));
if (!candidates) return true;
const t = dayMs(r.transactionDate);
if (!Number.isFinite(t)) return true;
const hit = candidates.findIndex((c) => Math.abs(c - t) <= pad);
if (hit === -1) return true;
// Consume the match so two feed rows cannot both claim one ledger row —
// a real pair of identical charges must not collapse into one.
candidates.splice(hit, 1);
ledgerDuplicates += 1;
return false;
});
}
const base = {
totalRows: rows.length,
inScope: inScope.length,
skipped,
accounts,
missingAccounts,
unknownAccounts: unknown,
duplicatesDropped: dropped.length,
suspectRepeats,
alreadyImported: ledger.length - notYetImported.length,
coveredByStatement,
statementWatermarks,
ledgerDuplicates,
toInsert: fresh.length,
};
const anomalies = findAnomalies(base, opts.largeBatch);
if (!opts.apply || (anomalies.length > 0 && !opts.force)) {
return { ...base, inserted: 0, anomalies, applied: false };
}
let inserted = 0;
for (const r of fresh) {
// ON CONFLICT against uq_transaction_source_ref: belt and braces over the
// source_ref check above, which is not transactional.
const res = await exec<{ id: number }>(
`INSERT INTO transactions
(statement_id, owner_id, transaction_date, description, amount, transaction_type,
merchant_name, foreign_currency_amount, foreign_currency_code,
source, source_ref, source_account)
VALUES (NULL, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL
DO NOTHING
RETURNING id`,
[
ownerId,
r.transactionDate,
r.description,
r.amount,
r.transactionType,
r.merchantName,
r.foreignCurrencyAmount,
r.foreignCurrencyCode,
SOURCE,
r.sourceRef,
r.sourceAccount,
]
);
inserted += res.length;
}
return { ...base, inserted, anomalies, applied: true };
}
src/lib/hooks.ts
+20 -1
View File
@@ -1053,9 +1053,11 @@ export function useImportCSV() {
return useMutation({
mutationFn: async (body: {
bank_name: string;
source?: string;
transactions: {
date: string; description: string; amount: number; transaction_type: string;
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string; category?: string;
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string;
category?: string; account?: string; source_ref?: string;
}[];
}) => {
const res = await fetch("/api/import/csv", {
@@ -1257,3 +1259,20 @@ export function useOrderDetail(entityKey: string | null) {
},
});
}
/**
* How far each account's statements already reach, for the CSV import modal.
*
* Fetched once when the modal opens: the review step uses it to drop rows the
* ledger already holds rather than offering them for import.
*/
export function useStatementCoverage() {
return useQuery({
queryKey: ["statement-coverage"],
queryFn: async () => {
const res = await fetch("/api/import/statement-coverage");
if (!res.ok) throw new Error("Could not load statement coverage");
return (await res.json()).coverage as { last4: string; coveredTo: string }[];
},
});
}
src/lib/queries.ts
+77 -25
View File
@@ -670,8 +670,11 @@ export async function batchInsertCSVTransactions(
foreign_currency_amount?: number;
foreign_currency_code?: string;
category?: string;
account?: string;
source_ref?: string;
}[],
tagId: number
tagId: number,
source?: string
): Promise<number> {
if (rows.length === 0) return 0;
@@ -685,13 +688,32 @@ export async function batchInsertCSVTransactions(
const params: unknown[] = [ownerId];
let p = 2;
rows.forEach((r, i) => {
valueClauses.push(`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`);
params.push(r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null, r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null);
valueClauses.push(
`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`
);
params.push(
r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null,
r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null,
// The category the reviewer chose. It used to be accepted here and then
// left out of the column list, so every category picked in the review
// step was silently discarded and the trigger wrote 'other'. That is not
// cosmetic: an uncategorised credit is admitted by NET_SPEND_ROWS and
// negated by SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of
// spend and none of them counted as income.
r.category ?? null,
source ?? null,
// source + source_ref is the only idempotency this path has. Without it a
// second import of the same file duplicates every row, because row_index
// is assigned MAX+1 and uq_transaction_identity can never fire.
r.source_ref ?? null,
r.account ?? null
);
});
const txIds = await queryRaw<{ id: number }>(
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, row_index)
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, category, source, source_ref, source_account, row_index)
VALUES ${valueClauses.join(", ")}
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL DO NOTHING
RETURNING id`,
params
);
@@ -733,30 +755,24 @@ export const needsCardMatch = (alias = "t") =>
`(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`;
/**
* REMOVED 2026-08-13, and the reason is worth more than the code was.
* REMOVED 2026-08-13. There used to be an `awaitsStatementLine()` predicate
* here, excluding account-feed rows from the pending-reconciliation queue.
*
* There used to be an `awaitsStatementLine()` predicate here excluding
* account-feed rows from the pending-reconciliation queue. Its premise — "the
* Frollo importer deliberately covers only accounts whose statements are *not*
* imported, so no statement line is coming for these, ever" — was asserted and
* never tested. It was false for almost every account: of the 550 rows the first
* import wrote, 422 already had a statement twin.
* Its premise — a feed row is the account's own ledger entry, so no statement
* line is coming — was asserted and never tested. It was false for almost every
* account: 422 of the first 550 imported rows already had a statement twin.
*
* What makes this worth recording is *how* it was introduced. The queue jumped
* from 8 to 558 the moment the feed landed, and that jump was read as noise and
* filtered away. The queue was right. Those 550 rows genuinely were provisional
* entries awaiting their statement lines, and suppressing the count removed the
* only mechanism that would ever have collapsed them — so the duplicates became
* permanent instead of transient, and stayed invisible until a human noticed the
* same salary payment listed twice in two currencies.
* How it got in matters more than what it did. The queue jumped from 8 to 558
* the moment the feed landed, and that jump was read as noise and filtered
* away. The queue was right — those rows genuinely were provisional entries
* awaiting statement lines — and hiding them removed the only mechanism that
* would ever have collapsed them, so a transient overlap became a permanent
* double-count and stayed invisible until a human saw one salary payment listed
* twice in two currencies.
*
* A feed row IS a row awaiting a statement line. It belongs in the queue. The
* volume problem is solved upstream, by not importing rows the ledger already
* holds (`LEDGER_MATCH_DAYS` in frollo-ingest.ts) — not downstream by hiding
* the ones that are.
*
* If a genuinely statement-less feed is ever added, give it a predicate of its
* own and prove the premise with a query first.
* An imported row that has no statement belongs in the queue. Volume is solved
* upstream, by not importing what the statements already cover
* (`getStatementCoverage`), never downstream by hiding what is there.
*/
/**
@@ -1528,3 +1544,39 @@ export async function getTagTransactionIds(tagId: number): Promise<number[]> {
);
return rows.map((r) => r.transaction_id);
}
/**
* The last date each account's statements cover, keyed by the account's last
* four digits.
*
* This is the coverage test that matters, and it is not the one that was tried
* first. The first attempt asked whether a row's date fell inside a statement's
* min-max *window*, which for accounts whose statements span 182 to 460 days
* swallows a year and answers nothing — 422 duplicates out of 550 rows got in
* that way. `billing_end_date` asks a question that has an answer: up to what
* date is this account complete?
*
* Amount matching cannot substitute for it. Two sources decompose the same
* event differently — an aggregator bundles the Wise transfer fee into the
* transfer (10001.13) where the statement itemises it (10000.00 + 1.13) — so
* the rows are the same money with different numbers.
*
* Keyed on last4 because account numbers are written differently everywhere
* ("235242176", "xxxxxxxxxxxx2176", "4085-56264"). Where two banks share a
* last4 the later date wins, which errs towards excluding a row rather than
* duplicating one; the caller shows every watermark so a wrong one is visible.
*/
export async function getStatementCoverage(): Promise<{ last4: string; coveredTo: string }[]> {
const rows = await queryRaw<{ last4: string; covered_to: string }>(
`SELECT right(regexp_replace(account_number, '[^0-9]', '', 'g'), 4) AS last4,
MAX(billing_end_date)::text AS covered_to
FROM statements
WHERE account_number IS NOT NULL
GROUP BY 1`,
[]
);
return rows
.filter((r) => r.last4 && r.last4.length === 4)
.map((r) => ({ last4: r.last4, coveredTo: r.covered_to }))
.sort((a, b) => a.last4.localeCompare(b.last4));
}