bank_name is a component of uq_statement_identity (bank_name, account_number, billing_end_date), so two spellings of one institution mean the same statement can be ingested twice without the index ever firing. Gemini reads the name off whatever the PDF prints, so the spelling varies per document, not per account. Zip exposed it: account 2705256 arrived under four names in sixteen minutes — ZipMoney Payments Pty Ltd, ZipMoney Payments Pty Limited, Zip Pay, ZipPay — with ten more in the ingestion queue. Not unique: 17 bank_name values represented 13 institutions, every split pair sharing an account number (Amex 14/1, Citibank/Citi 8/4, NAB 3/1). Fragmentation also splits both bank filters and by_bank in /api/analytics/fees. normalize_bank_name() + a BEFORE INSERT OR UPDATE trigger, mirroring the 0013 statement_type pattern with one deliberate difference: the vocabulary is OPEN. An unrecognised name passes through tidied rather than collapsing to a fallback, and there is no CHECK constraint — a bank never used before must be able to arrive without a migration, and destroying its name on first contact is worse than leaving it unmapped. Matching is by prefix, not by enumerated spelling, so unseen variants normalise with no code change. Branches for Westpac, ANZ, HSBC, ING, AMP, Up and CommBank are no-ops that map to the spelling already in use, to catch the legal-entity variant a future PDF might print. Wise is left alone: one spelling is a rename, not a merge. Backfill verified collision-free against uq_statement_identity first. 17 names -> 12. account_owner_mappings keys on (bank_name, account_number) with its own UNIQUE constraint and is updated too; it is empty today.
This commit is contained in:
@@ -865,6 +865,53 @@ Card'`) and the DB normalises it on write. The raw extracted value is preserved
|
||||
The TypeScript mirror is `src/lib/statement-types.ts` — keep the list, the SQL
|
||||
function, and the CHECK constraint in sync when adding a type.
|
||||
|
||||
### Bank names are normalised on write (migration 0031)
|
||||
|
||||
**`bank_name` is a component of the duplicate check.** `uq_statement_identity`
|
||||
is `(bank_name, account_number, billing_end_date)`, so two spellings of one
|
||||
institution mean the same statement can be ingested twice without the index ever
|
||||
firing. Gemini reads the name off whatever the PDF prints, so the spelling varies
|
||||
per *document*, not per account.
|
||||
|
||||
Zip exposed it: account `2705256` arrived under four names in sixteen minutes —
|
||||
`ZipMoney Payments Pty Ltd`, `ZipMoney Payments Pty Limited`, `Zip Pay`,
|
||||
`ZipPay` — with ten more in the queue. It was not unique. 17 `bank_name` values
|
||||
represented 13 institutions, and every split pair shared an account number:
|
||||
Amex (14/1), Citibank/Citi (8/4), NAB (3/1). Fragmentation also splits the
|
||||
statements and transactions bank filters and `by_bank` in `/api/analytics/fees`.
|
||||
|
||||
`normalize_bank_name()` + `trg_statements_normalize_bank_name` (BEFORE INSERT OR
|
||||
UPDATE OF bank_name) mirror the 0013 `statement_type` pattern, with **one
|
||||
deliberate difference: the vocabulary is open.** An unrecognised name passes
|
||||
through tidied, never collapsed to a fallback, and there is **no CHECK
|
||||
constraint** — a bank this household has never used must be able to arrive
|
||||
without a migration, and destroying its name on first contact is worse than
|
||||
leaving it unmapped.
|
||||
|
||||
Matching is by **prefix, not by enumerated spelling** (`zip%`, `citi%`,
|
||||
`%american express%`, `%national australia bank%`), so unseen variants like
|
||||
`Zip Co Australia Pty Ltd` normalise with no code change. Branches for Westpac,
|
||||
ANZ, HSBC, ING, AMP, Up and CommBank are no-ops today that map *to* the spelling
|
||||
already in use — they exist to catch the legal-entity variant a future PDF might
|
||||
print.
|
||||
|
||||
Wise is deliberately unmapped: `Wise Australia Pty Ltd.` is a single spelling, so
|
||||
shortening it would be a rename nobody asked for rather than a merge.
|
||||
|
||||
**The residual gap and its tell.** The function cannot know that two names for a
|
||||
*new* bank are one institution. That always shows up the same way, so check it
|
||||
rather than trusting the map:
|
||||
|
||||
```sql
|
||||
SELECT account_number, array_agg(DISTINCT bank_name)
|
||||
FROM statements WHERE account_number IS NOT NULL
|
||||
GROUP BY 1 HAVING count(DISTINCT bank_name) > 1;
|
||||
```
|
||||
|
||||
`account_owner_mappings` also keys on `(bank_name, account_number)` with its own
|
||||
UNIQUE constraint, so any future rename must update it too or strand its rows.
|
||||
It is empty today; the migration handles it anyway.
|
||||
|
||||
### Loans
|
||||
|
||||
A loan repayment is **not** an expense. It is part principal (equity, a
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
-- Normalise statements.bank_name so one institution is one name.
|
||||
--
|
||||
-- WHY THIS IS NOT COSMETIC: `uq_statement_identity` is
|
||||
-- (bank_name, account_number, billing_end_date). The bank name is a *component
|
||||
-- of the duplicate check*, so two spellings of one institution mean the same
|
||||
-- statement can be ingested twice without the index ever firing. Gemini reads
|
||||
-- the name off whatever the PDF happens to print, so the spelling varies per
|
||||
-- document, not per account.
|
||||
--
|
||||
-- Zip made this visible. One account (2705256) arrived under four names in
|
||||
-- sixteen minutes — 'ZipMoney Payments Pty Ltd', 'ZipMoney Payments Pty
|
||||
-- Limited', 'Zip Pay', 'ZipPay' — with ten or more still in the ingestion
|
||||
-- queue. It was not unique: 17 bank_name values represented 13 institutions,
|
||||
-- and every split pair shared an identical account number.
|
||||
--
|
||||
-- American Express (14) / AMERICAN EXPRESS (1) -> XXXX-XXXXXX-01000
|
||||
-- Citibank (8) / Citi (4) -> 5258073330340253
|
||||
-- National Australia Bank Limited (3) / ... (NAB) (1) -> 1015001000116227
|
||||
--
|
||||
-- Fragmentation also splits the statements-page bank filter, the transactions
|
||||
-- bank filter, and by_bank in /api/analytics/fees.
|
||||
--
|
||||
-- THE VOCABULARY IS OPEN, unlike statement_type in 0013. An unrecognised name
|
||||
-- passes through tidied, never collapsed to a fallback: a bank this household
|
||||
-- has never used must be able to arrive without a migration, and destroying its
|
||||
-- name on first contact is far worse than leaving it unmapped. That is also why
|
||||
-- there is deliberately NO CHECK constraint here.
|
||||
--
|
||||
-- Aliases for banks that only ever arrive one way (Westpac, ANZ, HSBC, ING,
|
||||
-- AMP, Up, CommBank) are no-ops today and map TO the spelling already in use,
|
||||
-- so they cannot rename anything. They exist to catch the legal-entity variant
|
||||
-- the next PDF might print — which is exactly how Zip got here.
|
||||
--
|
||||
-- Wise is deliberately left alone: 'Wise Australia Pty Ltd.' is a single
|
||||
-- spelling, so shortening it would be a rename nobody asked for rather than a
|
||||
-- merge. Add a branch if a second spelling ever shows up.
|
||||
--
|
||||
-- Idempotent: safe to re-run.
|
||||
|
||||
CREATE OR REPLACE FUNCTION normalize_bank_name(raw TEXT)
|
||||
RETURNS TEXT AS $$
|
||||
DECLARE
|
||||
v TEXT;
|
||||
BEGIN
|
||||
IF raw IS NULL THEN
|
||||
RETURN NULL;
|
||||
END IF;
|
||||
|
||||
-- Collapse whitespace before matching so 'Zip Pay' and 'Zip Pay' agree.
|
||||
v := lower(regexp_replace(trim(raw), '\s+', ' ', 'g'));
|
||||
|
||||
IF v = '' THEN
|
||||
-- An empty string is not a name. NULL is how the schema already spells
|
||||
-- "unknown": bankLabel() renders it 'Manual' and uq_statement_identity
|
||||
-- excludes it.
|
||||
RETURN NULL;
|
||||
END IF;
|
||||
|
||||
-- ---- Merges: institutions seen under more than one spelling ----
|
||||
|
||||
-- 'Zip Pay', 'ZipPay', 'ZipMoney Payments Pty Ltd', 'ZipMoney Payments Pty
|
||||
-- Limited'. Anchored so a merchant that merely contains "zip" cannot match.
|
||||
IF v LIKE 'zip%' THEN
|
||||
RETURN 'Zip';
|
||||
END IF;
|
||||
|
||||
IF v LIKE '%american express%' OR v LIKE 'amex%' THEN
|
||||
RETURN 'American Express';
|
||||
END IF;
|
||||
|
||||
-- Citi, Citibank, Citigroup — one card, 5258073330340253.
|
||||
IF v LIKE 'citi%' THEN
|
||||
RETURN 'Citibank';
|
||||
END IF;
|
||||
|
||||
IF v LIKE '%national australia bank%' OR v = 'nab' THEN
|
||||
RETURN 'NAB';
|
||||
END IF;
|
||||
|
||||
-- ---- Defensive: map future legal-entity variants onto the name in use ----
|
||||
|
||||
IF v LIKE '%westpac%' THEN
|
||||
RETURN 'Westpac';
|
||||
END IF;
|
||||
|
||||
IF v = 'anz' OR v LIKE 'anz %' OR v LIKE '%australia and new zealand banking%' THEN
|
||||
RETURN 'ANZ';
|
||||
END IF;
|
||||
|
||||
IF v LIKE '%commonwealth bank%' OR v LIKE 'commbank%' OR v = 'cba' THEN
|
||||
RETURN 'Commonwealth Bank';
|
||||
END IF;
|
||||
|
||||
IF v LIKE 'hsbc%' THEN
|
||||
RETURN 'HSBC';
|
||||
END IF;
|
||||
|
||||
IF v = 'ing' OR v LIKE 'ing %' THEN
|
||||
RETURN 'ING';
|
||||
END IF;
|
||||
|
||||
IF v = 'amp' OR v LIKE 'amp %' THEN
|
||||
RETURN 'AMP Bank';
|
||||
END IF;
|
||||
|
||||
IF v = 'up' OR v LIKE 'up bank%' THEN
|
||||
RETURN 'Up';
|
||||
END IF;
|
||||
|
||||
-- Unknown institution: keep exactly what was extracted, tidied only.
|
||||
RETURN regexp_replace(trim(raw), '\s+', ' ', 'g');
|
||||
END;
|
||||
$$ LANGUAGE plpgsql IMMUTABLE;
|
||||
|
||||
-- Normalise on write so the N8N workflow keeps sending whatever the PDF prints.
|
||||
CREATE OR REPLACE FUNCTION statements_normalize_bank_name_trigger()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
NEW.bank_name := normalize_bank_name(NEW.bank_name);
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS trg_statements_normalize_bank_name ON statements;
|
||||
CREATE TRIGGER trg_statements_normalize_bank_name
|
||||
BEFORE INSERT OR UPDATE OF bank_name ON statements
|
||||
FOR EACH ROW EXECUTE FUNCTION statements_normalize_bank_name_trigger();
|
||||
|
||||
-- Backfill. Verified collision-free against uq_statement_identity before
|
||||
-- writing this: no two statements share (normalised name, account_number,
|
||||
-- billing_end_date).
|
||||
UPDATE statements
|
||||
SET bank_name = normalize_bank_name(bank_name)
|
||||
WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name);
|
||||
|
||||
-- account_owner_mappings keys on (bank_name, account_number) with its own
|
||||
-- UNIQUE constraint, so a rename on `statements` would strand its rows. The
|
||||
-- table is empty today; this keeps it aligned if it is ever populated before
|
||||
-- this migration runs somewhere else. ON CONFLICT because two spellings of one
|
||||
-- bank could otherwise merge into one key.
|
||||
UPDATE account_owner_mappings
|
||||
SET bank_name = normalize_bank_name(bank_name)
|
||||
WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM account_owner_mappings other
|
||||
WHERE other.bank_name = normalize_bank_name(account_owner_mappings.bank_name)
|
||||
AND other.account_number = account_owner_mappings.account_number
|
||||
);
|
||||
Reference in New Issue
Block a user