bank_name is a component of uq_statement_identity (bank_name, account_number, billing_end_date), so two spellings of one institution mean the same statement can be ingested twice without the index ever firing. Gemini reads the name off whatever the PDF prints, so the spelling varies per document, not per account. Zip exposed it: account 2705256 arrived under four names in sixteen minutes — ZipMoney Payments Pty Ltd, ZipMoney Payments Pty Limited, Zip Pay, ZipPay — with ten more in the ingestion queue. Not unique: 17 bank_name values represented 13 institutions, every split pair sharing an account number (Amex 14/1, Citibank/Citi 8/4, NAB 3/1). Fragmentation also splits both bank filters and by_bank in /api/analytics/fees. normalize_bank_name() + a BEFORE INSERT OR UPDATE trigger, mirroring the 0013 statement_type pattern with one deliberate difference: the vocabulary is OPEN. An unrecognised name passes through tidied rather than collapsing to a fallback, and there is no CHECK constraint — a bank never used before must be able to arrive without a migration, and destroying its name on first contact is worse than leaving it unmapped. Matching is by prefix, not by enumerated spelling, so unseen variants normalise with no code change. Branches for Westpac, ANZ, HSBC, ING, AMP, Up and CommBank are no-ops that map to the spelling already in use, to catch the legal-entity variant a future PDF might print. Wise is left alone: one spelling is a rename, not a merge. Backfill verified collision-free against uq_statement_identity first. 17 names -> 12. account_owner_mappings keys on (bank_name, account_number) with its own UNIQUE constraint and is updated too; it is empty today.
This commit is contained in:
@@ -865,6 +865,53 @@ Card'`) and the DB normalises it on write. The raw extracted value is preserved
|
||||
The TypeScript mirror is `src/lib/statement-types.ts` — keep the list, the SQL
|
||||
function, and the CHECK constraint in sync when adding a type.
|
||||
|
||||
### Bank names are normalised on write (migration 0031)
|
||||
|
||||
**`bank_name` is a component of the duplicate check.** `uq_statement_identity`
|
||||
is `(bank_name, account_number, billing_end_date)`, so two spellings of one
|
||||
institution mean the same statement can be ingested twice without the index ever
|
||||
firing. Gemini reads the name off whatever the PDF prints, so the spelling varies
|
||||
per *document*, not per account.
|
||||
|
||||
Zip exposed it: account `2705256` arrived under four names in sixteen minutes —
|
||||
`ZipMoney Payments Pty Ltd`, `ZipMoney Payments Pty Limited`, `Zip Pay`,
|
||||
`ZipPay` — with ten more in the queue. It was not unique. 17 `bank_name` values
|
||||
represented 13 institutions, and every split pair shared an account number:
|
||||
Amex (14/1), Citibank/Citi (8/4), NAB (3/1). Fragmentation also splits the
|
||||
statements and transactions bank filters and `by_bank` in `/api/analytics/fees`.
|
||||
|
||||
`normalize_bank_name()` + `trg_statements_normalize_bank_name` (BEFORE INSERT OR
|
||||
UPDATE OF bank_name) mirror the 0013 `statement_type` pattern, with **one
|
||||
deliberate difference: the vocabulary is open.** An unrecognised name passes
|
||||
through tidied, never collapsed to a fallback, and there is **no CHECK
|
||||
constraint** — a bank this household has never used must be able to arrive
|
||||
without a migration, and destroying its name on first contact is worse than
|
||||
leaving it unmapped.
|
||||
|
||||
Matching is by **prefix, not by enumerated spelling** (`zip%`, `citi%`,
|
||||
`%american express%`, `%national australia bank%`), so unseen variants like
|
||||
`Zip Co Australia Pty Ltd` normalise with no code change. Branches for Westpac,
|
||||
ANZ, HSBC, ING, AMP, Up and CommBank are no-ops today that map *to* the spelling
|
||||
already in use — they exist to catch the legal-entity variant a future PDF might
|
||||
print.
|
||||
|
||||
Wise is deliberately unmapped: `Wise Australia Pty Ltd.` is a single spelling, so
|
||||
shortening it would be a rename nobody asked for rather than a merge.
|
||||
|
||||
**The residual gap and its tell.** The function cannot know that two names for a
|
||||
*new* bank are one institution. That always shows up the same way, so check it
|
||||
rather than trusting the map:
|
||||
|
||||
```sql
|
||||
SELECT account_number, array_agg(DISTINCT bank_name)
|
||||
FROM statements WHERE account_number IS NOT NULL
|
||||
GROUP BY 1 HAVING count(DISTINCT bank_name) > 1;
|
||||
```
|
||||
|
||||
`account_owner_mappings` also keys on `(bank_name, account_number)` with its own
|
||||
UNIQUE constraint, so any future rename must update it too or strand its rows.
|
||||
It is empty today; the migration handles it anyway.
|
||||
|
||||
### Loans
|
||||
|
||||
A loan repayment is **not** an expense. It is part principal (equity, a
|
||||
|
||||
Reference in New Issue
Block a user