Import Frollo account feeds for the accounts statements don't cover
ci / lint-test (push) Successful in 47s
ci / lint-test (push) Successful in 47s
Credit cards keep arriving as monthly statements and stay the source of truth for them. This covers the other fourteen accounts, whose statements arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise including the income account, Up, ING, and the small transaction accounts. About 50 rows a month, where the alternative is downloading each statement by hand. The file needs three defences, all found by diffing three real exports (smarthome DECISIONS.md ING-11): A CDR re-consent makes Frollo re-ingest an account's whole history under fresh transaction ids while the originals survive, and consents expire annually. On this export 112 rows were such twins, and every HDR salary payment appeared twice — importing blind doubles reported income. dedupe() collapses each natural-key group to its lowest id, lowest because old ids were a strict subset of new across two exports, so source_ref stays stable and a re-import inserts nothing. An earlier version of that rule kept close-id rows on a 10,000 threshold, reasoning that genuine same-day repeats have consecutive ids. Verifying it against the income rows killed it: in-scope duplicate pairs have id gaps from 58 to 260 million, so no threshold separates them from the gaps of 1-4 that real repeats showed. It now collapses unconditionally and flags anything within 10 for review — the errors are asymmetric, and nothing in scope has ever tripped the flag. A lapsed consent removes an account from the export silently, with no error and no marker; the row count just drops. So the import asserts the account roster and refuses to run when a configured account contributes nothing. Also holds these rows out of the pending-reconciliation queue. A feed row is the account's own ledger entry, not a receipt awaiting a statement line — these accounts' statements are deliberately not imported — so without the exclusion 550 rows a year would bury the receipts that need a decision. The queue stays at 8 instead of 558. Foreign rows follow order-ingestion's existing shape: amount is the native figure, foreign_currency_code names it, amount_aud stays NULL rather than asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as incomplete. Dry run by default. Verified against the real export before applying: 550 rows inserted, 14 accounts, re-run inserts 0.
This commit is contained in:
@@ -0,0 +1,347 @@
|
||||
/**
|
||||
* Frollo transaction-export parsing and de-duplication.
|
||||
*
|
||||
* Frollo is a CDR (Open Banking) aggregator. Its app emails a CSV of up to 12
|
||||
* months of transactions. This module turns that file into rows ready for
|
||||
* `transactions`, and — mostly — decides which of them are real.
|
||||
*
|
||||
* Scope is deliberately narrow: credit cards are excluded because they already
|
||||
* arrive as monthly statements, which are authoritative. What Frollo is for is
|
||||
* the accounts whose statements arrive every 182 to 460 days (see
|
||||
* ACCOUNTS below), where the alternative is downloading each one by hand.
|
||||
*
|
||||
* The file has three properties that any consumer has to handle, all learned by
|
||||
* diffing three real exports (smarthome DECISIONS.md ING-11):
|
||||
*
|
||||
* 1. A CDR re-consent makes Frollo re-ingest the account's whole history under
|
||||
* fresh transaction ids, while the originals survive. So the same purchase
|
||||
* appears twice with different ids, and consents expire annually — this is
|
||||
* not a one-off. On one 1,989-row export, 385 rows were such twins, and Wise
|
||||
* credits were overstated by exactly 100%.
|
||||
* 2. `transaction_id` is otherwise stable: across two exports 12 days apart,
|
||||
* 1,462 of 1,532 shared rows kept identical ids and none were replaced. It
|
||||
* is therefore a good `source_ref` — it just cannot detect a twin.
|
||||
* 3. The export cannot express "pending", and the tell differs per bank
|
||||
* (Westpac prefixes the description; Amex emits no `posted_date` or
|
||||
* `transaction_type` at all, so its pending rows are indistinguishable from
|
||||
* its posted ones). The export's own option is the only reliable control.
|
||||
* Every pending row observed was on a credit card, i.e. out of scope, so
|
||||
* exporting with pending EXCLUDED costs nothing here — and it avoids the
|
||||
* re-key problem, since a settling row changes both its id and its
|
||||
* description.
|
||||
*/
|
||||
|
||||
/** A row as it appears in the CSV, after header mapping. */
|
||||
export interface FrolloRow {
|
||||
transactionId: string;
|
||||
description: string;
|
||||
amount: number;
|
||||
currency: string;
|
||||
transactionDate: string;
|
||||
postedDate: string | null;
|
||||
accountNumber: string;
|
||||
accountName: string;
|
||||
creditDebit: string;
|
||||
transactionType: string | null;
|
||||
providerName: string;
|
||||
merchantName: string | null;
|
||||
}
|
||||
|
||||
/** A row shaped for insertion into `transactions`. */
|
||||
export interface LedgerRow {
|
||||
sourceRef: string;
|
||||
sourceAccount: string;
|
||||
transactionDate: string;
|
||||
description: string;
|
||||
/** Always positive; direction lives in `transactionType`. */
|
||||
amount: number;
|
||||
transactionType: "debit" | "credit";
|
||||
merchantName: string | null;
|
||||
foreignCurrencyAmount: number | null;
|
||||
foreignCurrencyCode: string | null;
|
||||
}
|
||||
|
||||
export interface AccountSpec {
|
||||
/** Last four characters of the digits in the masked account number. */
|
||||
last4: string;
|
||||
label: string;
|
||||
provider: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Accounts this importer owns.
|
||||
*
|
||||
* Keyed on the last four digits rather than the masked string, because the
|
||||
* masking format varies by institution (`xxx-xxx xxxx1910`, `xxxxxx xxxxx6264`,
|
||||
* `x xxxx 0887`) and is a presentation choice Frollo could change. All fourteen
|
||||
* are distinct, as are the five excluded cards.
|
||||
*
|
||||
* Credit cards are absent on purpose — monthly statements already cover them and
|
||||
* are the source of truth. Adding one here would create a second producer for
|
||||
* rows the reconcile queue would then have to match against themselves.
|
||||
*/
|
||||
export const ACCOUNTS: AccountSpec[] = [
|
||||
{ last4: "6264", label: "ANZ Access Advantage", provider: "ANZ" },
|
||||
{ last4: "9940", label: "AMP (mortgage)", provider: "AMP" },
|
||||
{ last4: "8109", label: "AMP", provider: "AMP" },
|
||||
{ last4: "0682", label: "AMP (loan)", provider: "AMP" },
|
||||
{ last4: "4830", label: "Wise (income)", provider: "Wise" },
|
||||
{ last4: "2176", label: "Wise", provider: "Wise" },
|
||||
{ last4: "0052", label: "Up Spending", provider: "Up" },
|
||||
{ last4: "0078", label: "Up Savings", provider: "Up" },
|
||||
{ last4: "1910", label: "CommBank Smart Access", provider: "CommBank" },
|
||||
{ last4: "4878", label: "Westpac Choice", provider: "Westpac" },
|
||||
{ last4: "4758", label: "Westpac Business One", provider: "Westpac" },
|
||||
{ last4: "0887", label: "ING Orange Everyday", provider: "ING" },
|
||||
{ last4: "9336", label: "ING Savings Maximiser", provider: "ING" },
|
||||
{ last4: "9807", label: "ANZ Online Saver", provider: "ANZ" },
|
||||
];
|
||||
|
||||
/**
|
||||
* Below this id gap, two rows sharing a natural key are *probably* a genuine
|
||||
* same-day repeat rather than a re-ingest twin — and are reported rather than
|
||||
* silently collapsed.
|
||||
*
|
||||
* This is a review trigger, not a decision. An earlier version used a 10,000
|
||||
* threshold to *keep* close-id rows, and it was wrong: measured across two real
|
||||
* exports, in-scope duplicate pairs have gaps from **58 to 260 million**, while
|
||||
* the only observed genuine repeats (five identical $1.00 Apple charges) had
|
||||
* gaps of 1 to 4. A threshold anywhere in between is a magic number, and the
|
||||
* error it enables is asymmetric — keeping a twin doubled reported income by
|
||||
* 100%, whereas collapsing a real repeat understates by one row.
|
||||
*
|
||||
* So dedupe collapses unconditionally and flags anything this close for a human
|
||||
* to look at. On the accounts in scope, nothing has ever tripped it: every
|
||||
* duplicate group is a pair and the minimum gap is 58.
|
||||
*/
|
||||
export const SUSPECT_REPEAT_GAP = 10;
|
||||
|
||||
export function last4(accountNumber: string): string {
|
||||
return accountNumber.replace(/\D/g, "").slice(-4);
|
||||
}
|
||||
|
||||
export function accountFor(accountNumber: string): AccountSpec | undefined {
|
||||
const key = last4(accountNumber);
|
||||
return ACCOUNTS.find((a) => a.last4 === key);
|
||||
}
|
||||
|
||||
const REQUIRED_COLUMNS = [
|
||||
"transaction_id",
|
||||
"description",
|
||||
"amount",
|
||||
"currency",
|
||||
"transaction_date",
|
||||
"account_number",
|
||||
"account_name",
|
||||
"credit_debit",
|
||||
];
|
||||
|
||||
/**
|
||||
* Maps a parsed CSV grid to rows, by header name rather than position.
|
||||
*
|
||||
* Throws on a missing required column rather than reading undefined into a
|
||||
* transaction: a column rename upstream should stop the import loudly, not
|
||||
* silently produce rows with blank descriptions.
|
||||
*/
|
||||
export function readRows(grid: string[][]): FrolloRow[] {
|
||||
if (grid.length === 0) throw new Error("empty CSV");
|
||||
const header = grid[0].map((h) => h.trim());
|
||||
const missing = REQUIRED_COLUMNS.filter((c) => !header.includes(c));
|
||||
if (missing.length > 0) {
|
||||
throw new Error(`Frollo CSV missing expected column(s): ${missing.join(", ")}`);
|
||||
}
|
||||
const at = (row: string[], col: string): string => {
|
||||
const i = header.indexOf(col);
|
||||
return i >= 0 ? (row[i] ?? "").trim() : "";
|
||||
};
|
||||
const out: FrolloRow[] = [];
|
||||
for (const row of grid.slice(1)) {
|
||||
if (row.every((f) => f === "")) continue;
|
||||
const amount = Number(at(row, "amount"));
|
||||
out.push({
|
||||
transactionId: at(row, "transaction_id"),
|
||||
description: at(row, "description"),
|
||||
amount: Number.isFinite(amount) ? amount : NaN,
|
||||
currency: at(row, "currency") || "AUD",
|
||||
transactionDate: at(row, "transaction_date"),
|
||||
postedDate: at(row, "posted_date") || null,
|
||||
accountNumber: at(row, "account_number"),
|
||||
accountName: at(row, "account_name"),
|
||||
creditDebit: at(row, "credit_debit"),
|
||||
transactionType: at(row, "transaction_type") || null,
|
||||
providerName: at(row, "provider_name"),
|
||||
merchantName: at(row, "merchant_name") || null,
|
||||
});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export type SkipReason =
|
||||
| "zero_amount"
|
||||
| "unparseable_amount"
|
||||
| "out_of_scope_account"
|
||||
| "pending";
|
||||
|
||||
/**
|
||||
* Why a row is not imported, or null if it should be.
|
||||
*
|
||||
* `zero_amount` is mostly CommBank fee *waivers* — 574 of them in one export,
|
||||
* genuinely $0.00 with the notional fee written into the description text
|
||||
* (`INTNL TRANSACTION FEE, $ 41.58 FEE SAV`). They are not transactions.
|
||||
*
|
||||
* `pending` is a belt-and-braces check on top of the export option. It only
|
||||
* catches Westpac's description prefix; Amex pending rows are undetectable here,
|
||||
* which is why the option matters and this does not replace it.
|
||||
*/
|
||||
export function skipReason(row: FrolloRow): SkipReason | null {
|
||||
if (!Number.isFinite(row.amount)) return "unparseable_amount";
|
||||
if (row.amount === 0) return "zero_amount";
|
||||
if (!accountFor(row.accountNumber)) return "out_of_scope_account";
|
||||
if (/^\s*pending\s*[-:]/i.test(row.description)) return "pending";
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Natural key: what makes two rows the same purchase, ignoring ids. */
|
||||
export function naturalKey(row: FrolloRow): string {
|
||||
return [
|
||||
last4(row.accountNumber),
|
||||
row.transactionDate,
|
||||
row.amount.toFixed(2),
|
||||
row.description,
|
||||
row.currency,
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
export interface DroppedRow {
|
||||
row: FrolloRow;
|
||||
duplicateOf: FrolloRow;
|
||||
idGap: number;
|
||||
/** Ids close enough that this might be a real repeat, not a twin. Review it. */
|
||||
suspectGenuineRepeat: boolean;
|
||||
}
|
||||
|
||||
export interface DedupeResult {
|
||||
kept: FrolloRow[];
|
||||
dropped: DroppedRow[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapses each natural-key group to its lowest-id row.
|
||||
*
|
||||
* Lowest because old ids survive a re-ingestion — verified across two exports
|
||||
* twelve days apart, where the earlier ids were a strict subset of the later
|
||||
* ones. Keeping the lowest therefore keeps `source_ref` stable from run to run,
|
||||
* so a re-import updates nothing and inserts nothing.
|
||||
*
|
||||
* Everything collapsed is returned in `dropped`, and anything whose ids sit
|
||||
* within SUSPECT_REPEAT_GAP is flagged: that is the shape a genuine same-day
|
||||
* repeat would take, and it should be looked at rather than assumed.
|
||||
*/
|
||||
export function dedupe(rows: FrolloRow[], suspectGap = SUSPECT_REPEAT_GAP): DedupeResult {
|
||||
const groups = new Map<string, FrolloRow[]>();
|
||||
for (const row of rows) {
|
||||
const k = naturalKey(row);
|
||||
const g = groups.get(k);
|
||||
if (g) g.push(row);
|
||||
else groups.set(k, [row]);
|
||||
}
|
||||
|
||||
const kept: FrolloRow[] = [];
|
||||
const dropped: DroppedRow[] = [];
|
||||
|
||||
for (const group of groups.values()) {
|
||||
if (group.length === 1) {
|
||||
kept.push(group[0]);
|
||||
continue;
|
||||
}
|
||||
const sorted = [...group].sort(
|
||||
(a, b) => Number(a.transactionId) - Number(b.transactionId)
|
||||
);
|
||||
const winner = sorted[0];
|
||||
kept.push(winner);
|
||||
for (let i = 1; i < sorted.length; i++) {
|
||||
const idGap = Number(sorted[i].transactionId) - Number(sorted[i - 1].transactionId);
|
||||
dropped.push({
|
||||
row: sorted[i],
|
||||
duplicateOf: winner,
|
||||
idGap,
|
||||
suspectGenuineRepeat: idGap < suspectGap,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return { kept, dropped };
|
||||
}
|
||||
|
||||
/**
|
||||
* Shapes a row for `transactions`.
|
||||
*
|
||||
* `category` is deliberately not set. Frollo assigns its own categories and the
|
||||
* owner reports they are often wrong; finance-app's rules engine plus
|
||||
* `category_override` already decide this, and a third untrusted source
|
||||
* competing with both is worse than none.
|
||||
*
|
||||
* For a non-AUD row the file gives only the foreign figure — there is no AUD
|
||||
* equivalent anywhere in the export. That is the same position order-ingestion
|
||||
* is in, so it takes the same shape: `amount` IS the native figure,
|
||||
* `foreign_currency_code` names it, and `amount_aud` is left NULL rather than
|
||||
* asserting a rate we do not have. `AMOUNT_UNCONVERTED` in analytics-sql then
|
||||
* counts these and the UI reports the balance as incomplete.
|
||||
*/
|
||||
export function toLedgerRow(row: FrolloRow): LedgerRow {
|
||||
const foreign = row.currency !== "AUD";
|
||||
const magnitude = Math.abs(row.amount);
|
||||
return {
|
||||
sourceRef: row.transactionId,
|
||||
sourceAccount: row.accountNumber,
|
||||
transactionDate: row.transactionDate,
|
||||
description: row.description,
|
||||
amount: magnitude,
|
||||
transactionType: row.amount < 0 ? "debit" : "credit",
|
||||
// Frollo writes "Unknown" where it could not resolve a merchant. Storing
|
||||
// that would put the literal word on the screen; NULL lets the existing
|
||||
// merchant fallbacks work.
|
||||
merchantName: row.merchantName && row.merchantName !== "Unknown" ? row.merchantName : null,
|
||||
foreignCurrencyAmount: foreign ? magnitude : null,
|
||||
foreignCurrencyCode: foreign ? row.currency : null,
|
||||
};
|
||||
}
|
||||
|
||||
export interface AccountReport {
|
||||
spec: AccountSpec;
|
||||
rows: number;
|
||||
present: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Which configured accounts the file actually contains.
|
||||
*
|
||||
* A lapsed CDR consent does not error — the account simply stops appearing, and
|
||||
* the row count drops. Nothing else in the pipeline would notice, so the import
|
||||
* asserts the roster rather than trusting it.
|
||||
*/
|
||||
export function accountReport(rows: FrolloRow[]): AccountReport[] {
|
||||
const counts = new Map<string, number>();
|
||||
for (const r of rows) {
|
||||
const k = last4(r.accountNumber);
|
||||
counts.set(k, (counts.get(k) ?? 0) + 1);
|
||||
}
|
||||
return ACCOUNTS.map((spec) => ({
|
||||
spec,
|
||||
rows: counts.get(spec.last4) ?? 0,
|
||||
present: (counts.get(spec.last4) ?? 0) > 0,
|
||||
}));
|
||||
}
|
||||
|
||||
/** Accounts in the file that this importer does not know about. */
|
||||
export function unknownAccounts(rows: FrolloRow[]): { accountNumber: string; accountName: string; rows: number }[] {
|
||||
const seen = new Map<string, { accountNumber: string; accountName: string; rows: number }>();
|
||||
for (const r of rows) {
|
||||
if (accountFor(r.accountNumber)) continue;
|
||||
const k = last4(r.accountNumber);
|
||||
const e = seen.get(k);
|
||||
if (e) e.rows++;
|
||||
else seen.set(k, { accountNumber: r.accountNumber, accountName: r.accountName, rows: 1 });
|
||||
}
|
||||
return [...seen.values()].sort((a, b) => b.rows - a.rows);
|
||||
}
|
||||
@@ -732,6 +732,25 @@ export async function batchInsertCSVTransactions(
|
||||
export const needsCardMatch = (alias = "t") =>
|
||||
`(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`;
|
||||
|
||||
/**
|
||||
* Rows for which a statement line could still arrive.
|
||||
*
|
||||
* A row imported from an account feed is that account's own ledger entry, not a
|
||||
* receipt waiting to be matched against one. The Frollo importer deliberately
|
||||
* covers only accounts whose statements are *not* imported — credit cards are
|
||||
* excluded from it precisely because theirs are — so no statement line is coming
|
||||
* for these, ever. Same shape as a credits-funded order, different reason.
|
||||
*
|
||||
* Without this, roughly 600 rows a year would sit in the pending-reconciliation
|
||||
* queue forever and bury the receipts that genuinely need a decision. Scoped to
|
||||
* feeds rather than to `source IS NOT NULL`, so a future source that *does* await
|
||||
* a statement is not silently swept up by it.
|
||||
*/
|
||||
export const ACCOUNT_FEED_SOURCES = ["frollo"] as const;
|
||||
|
||||
export const awaitsStatementLine = (alias = "t") =>
|
||||
`(${alias}.source IS NULL OR ${alias}.source NOT IN (${ACCOUNT_FEED_SOURCES.map((s) => `'${s}'`).join(", ")}))`;
|
||||
|
||||
/**
|
||||
* Bank label for a transaction. A row with no statement was not imported from
|
||||
* one, and the label has to say *why*: "Manual" reads as "hand-entered, still
|
||||
@@ -784,6 +803,7 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
|
||||
) txn_splits ON true
|
||||
WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL
|
||||
AND ${needsCardMatch("t")}
|
||||
AND ${awaitsStatementLine("t")}
|
||||
ORDER BY t.transaction_date DESC, t.row_index ASC`,
|
||||
[ownerId]
|
||||
);
|
||||
@@ -826,6 +846,7 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
|
||||
AND m.owner_id = $1
|
||||
AND m.reconciled_with_id IS NULL
|
||||
AND ${needsCardMatch("m")}
|
||||
AND ${awaitsStatementLine("m")}
|
||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM transactions mt WHERE mt.reconciled_with_id = t.id
|
||||
|
||||
Reference in New Issue
Block a user