diff --git a/CLAUDE.md b/CLAUDE.md index 4249332..c350c9f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -865,6 +865,53 @@ Card'`) and the DB normalises it on write. The raw extracted value is preserved The TypeScript mirror is `src/lib/statement-types.ts` — keep the list, the SQL function, and the CHECK constraint in sync when adding a type. +### Bank names are normalised on write (migration 0031) + +**`bank_name` is a component of the duplicate check.** `uq_statement_identity` +is `(bank_name, account_number, billing_end_date)`, so two spellings of one +institution mean the same statement can be ingested twice without the index ever +firing. Gemini reads the name off whatever the PDF prints, so the spelling varies +per *document*, not per account. + +Zip exposed it: account `2705256` arrived under four names in sixteen minutes — +`ZipMoney Payments Pty Ltd`, `ZipMoney Payments Pty Limited`, `Zip Pay`, +`ZipPay` — with ten more in the queue. It was not unique. 17 `bank_name` values +represented 13 institutions, and every split pair shared an account number: +Amex (14/1), Citibank/Citi (8/4), NAB (3/1). Fragmentation also splits the +statements and transactions bank filters and `by_bank` in `/api/analytics/fees`. + +`normalize_bank_name()` + `trg_statements_normalize_bank_name` (BEFORE INSERT OR +UPDATE OF bank_name) mirror the 0013 `statement_type` pattern, with **one +deliberate difference: the vocabulary is open.** An unrecognised name passes +through tidied, never collapsed to a fallback, and there is **no CHECK +constraint** — a bank this household has never used must be able to arrive +without a migration, and destroying its name on first contact is worse than +leaving it unmapped. + +Matching is by **prefix, not by enumerated spelling** (`zip%`, `citi%`, +`%american express%`, `%national australia bank%`), so unseen variants like +`Zip Co Australia Pty Ltd` normalise with no code change. Branches for Westpac, +ANZ, HSBC, ING, AMP, Up and CommBank are no-ops today that map *to* the spelling +already in use — they exist to catch the legal-entity variant a future PDF might +print. + +Wise is deliberately unmapped: `Wise Australia Pty Ltd.` is a single spelling, so +shortening it would be a rename nobody asked for rather than a merge. + +**The residual gap and its tell.** The function cannot know that two names for a +*new* bank are one institution. That always shows up the same way, so check it +rather than trusting the map: + +```sql +SELECT account_number, array_agg(DISTINCT bank_name) +FROM statements WHERE account_number IS NOT NULL +GROUP BY 1 HAVING count(DISTINCT bank_name) > 1; +``` + +`account_owner_mappings` also keys on `(bank_name, account_number)` with its own +UNIQUE constraint, so any future rename must update it too or strand its rows. +It is empty today; the migration handles it anyway. + ### Loans A loan repayment is **not** an expense. It is part principal (equity, a diff --git a/prisma/migrations/0031_normalize_bank_name/migration.sql b/prisma/migrations/0031_normalize_bank_name/migration.sql new file mode 100644 index 0000000..222d8eb --- /dev/null +++ b/prisma/migrations/0031_normalize_bank_name/migration.sql @@ -0,0 +1,148 @@ +-- Normalise statements.bank_name so one institution is one name. +-- +-- WHY THIS IS NOT COSMETIC: `uq_statement_identity` is +-- (bank_name, account_number, billing_end_date). The bank name is a *component +-- of the duplicate check*, so two spellings of one institution mean the same +-- statement can be ingested twice without the index ever firing. Gemini reads +-- the name off whatever the PDF happens to print, so the spelling varies per +-- document, not per account. +-- +-- Zip made this visible. One account (2705256) arrived under four names in +-- sixteen minutes — 'ZipMoney Payments Pty Ltd', 'ZipMoney Payments Pty +-- Limited', 'Zip Pay', 'ZipPay' — with ten or more still in the ingestion +-- queue. It was not unique: 17 bank_name values represented 13 institutions, +-- and every split pair shared an identical account number. +-- +-- American Express (14) / AMERICAN EXPRESS (1) -> XXXX-XXXXXX-01000 +-- Citibank (8) / Citi (4) -> 5258073330340253 +-- National Australia Bank Limited (3) / ... (NAB) (1) -> 1015001000116227 +-- +-- Fragmentation also splits the statements-page bank filter, the transactions +-- bank filter, and by_bank in /api/analytics/fees. +-- +-- THE VOCABULARY IS OPEN, unlike statement_type in 0013. An unrecognised name +-- passes through tidied, never collapsed to a fallback: a bank this household +-- has never used must be able to arrive without a migration, and destroying its +-- name on first contact is far worse than leaving it unmapped. That is also why +-- there is deliberately NO CHECK constraint here. +-- +-- Aliases for banks that only ever arrive one way (Westpac, ANZ, HSBC, ING, +-- AMP, Up, CommBank) are no-ops today and map TO the spelling already in use, +-- so they cannot rename anything. They exist to catch the legal-entity variant +-- the next PDF might print — which is exactly how Zip got here. +-- +-- Wise is deliberately left alone: 'Wise Australia Pty Ltd.' is a single +-- spelling, so shortening it would be a rename nobody asked for rather than a +-- merge. Add a branch if a second spelling ever shows up. +-- +-- Idempotent: safe to re-run. + +CREATE OR REPLACE FUNCTION normalize_bank_name(raw TEXT) +RETURNS TEXT AS $$ +DECLARE + v TEXT; +BEGIN + IF raw IS NULL THEN + RETURN NULL; + END IF; + + -- Collapse whitespace before matching so 'Zip Pay' and 'Zip Pay' agree. + v := lower(regexp_replace(trim(raw), '\s+', ' ', 'g')); + + IF v = '' THEN + -- An empty string is not a name. NULL is how the schema already spells + -- "unknown": bankLabel() renders it 'Manual' and uq_statement_identity + -- excludes it. + RETURN NULL; + END IF; + + -- ---- Merges: institutions seen under more than one spelling ---- + + -- 'Zip Pay', 'ZipPay', 'ZipMoney Payments Pty Ltd', 'ZipMoney Payments Pty + -- Limited'. Anchored so a merchant that merely contains "zip" cannot match. + IF v LIKE 'zip%' THEN + RETURN 'Zip'; + END IF; + + IF v LIKE '%american express%' OR v LIKE 'amex%' THEN + RETURN 'American Express'; + END IF; + + -- Citi, Citibank, Citigroup — one card, 5258073330340253. + IF v LIKE 'citi%' THEN + RETURN 'Citibank'; + END IF; + + IF v LIKE '%national australia bank%' OR v = 'nab' THEN + RETURN 'NAB'; + END IF; + + -- ---- Defensive: map future legal-entity variants onto the name in use ---- + + IF v LIKE '%westpac%' THEN + RETURN 'Westpac'; + END IF; + + IF v = 'anz' OR v LIKE 'anz %' OR v LIKE '%australia and new zealand banking%' THEN + RETURN 'ANZ'; + END IF; + + IF v LIKE '%commonwealth bank%' OR v LIKE 'commbank%' OR v = 'cba' THEN + RETURN 'Commonwealth Bank'; + END IF; + + IF v LIKE 'hsbc%' THEN + RETURN 'HSBC'; + END IF; + + IF v = 'ing' OR v LIKE 'ing %' THEN + RETURN 'ING'; + END IF; + + IF v = 'amp' OR v LIKE 'amp %' THEN + RETURN 'AMP Bank'; + END IF; + + IF v = 'up' OR v LIKE 'up bank%' THEN + RETURN 'Up'; + END IF; + + -- Unknown institution: keep exactly what was extracted, tidied only. + RETURN regexp_replace(trim(raw), '\s+', ' ', 'g'); +END; +$$ LANGUAGE plpgsql IMMUTABLE; + +-- Normalise on write so the N8N workflow keeps sending whatever the PDF prints. +CREATE OR REPLACE FUNCTION statements_normalize_bank_name_trigger() +RETURNS TRIGGER AS $$ +BEGIN + NEW.bank_name := normalize_bank_name(NEW.bank_name); + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS trg_statements_normalize_bank_name ON statements; +CREATE TRIGGER trg_statements_normalize_bank_name + BEFORE INSERT OR UPDATE OF bank_name ON statements + FOR EACH ROW EXECUTE FUNCTION statements_normalize_bank_name_trigger(); + +-- Backfill. Verified collision-free against uq_statement_identity before +-- writing this: no two statements share (normalised name, account_number, +-- billing_end_date). +UPDATE statements +SET bank_name = normalize_bank_name(bank_name) +WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name); + +-- account_owner_mappings keys on (bank_name, account_number) with its own +-- UNIQUE constraint, so a rename on `statements` would strand its rows. The +-- table is empty today; this keeps it aligned if it is ever populated before +-- this migration runs somewhere else. ON CONFLICT because two spellings of one +-- bank could otherwise merge into one key. +UPDATE account_owner_mappings +SET bank_name = normalize_bank_name(bank_name) +WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name) + AND NOT EXISTS ( + SELECT 1 FROM account_owner_mappings other + WHERE other.bank_name = normalize_bank_name(account_owner_mappings.bank_name) + AND other.account_number = account_owner_mappings.account_number + );