statements: derive account identity, fix the new-bank alert (0033)
ci / lint-test (push) Successful in 42s

account_number fragments exactly like bank_name did, from the same cause:
ANZ's Access Advantage arrived as both '4085-56264' (4 statements) and
'408556264' (1). uq_statement_identity keyed on it, so re-importing one
period under the other spelling evaded the duplicate check.

account_number_key is GENERATED ALWAYS — the number with separators
stripped — and the unique index moves onto it. Derived rather than
rewritten because the raw number is the readable one and some of it is
structure: Up stores '633-123 / 176540052', a BSB and an account number,
and flattening it would lose a distinction a human reads at a glance to
fix a machine problem. The Amex mask and case are preserved; 'X' records
which digits were redacted.

Not the cause of the documented 31-row / $42,040.68 ANZ duplication —
107/142/143 overlap on different end dates, which that index cannot catch
at any spelling.

Adds statement_identity_drift: one account under two bank names, or one
bank under two spellings of one number. Non-empty means fragmentation the
normaliser could not know about. Currently one row, the ANZ pair above,
now unified by key.

Also documents the N8N regression this series caused. Check Known Bank
compared the raw Gemini name against the canonical column before the
insert, so nothing ever matched and every Zip statement was tagged
pending and held for Slack approval — a stall, not just a noisy alert.
Fixed in workflow FysADdFwEtwONQl4 by comparing like with like through
normalize_bank_name(), passing currency so a genuinely new national
entity still alerts. Verified live: 12s after vs 30-53s before.
This commit is contained in:
2026-08-15 17:03:13 +10:00
parent c3b623052d
commit 2cfaf8e8c8
4 changed files with 151 additions and 15 deletions
+64 -8
View File
@@ -898,20 +898,76 @@ print.
Wise is deliberately unmapped: `Wise Australia Pty Ltd.` is a single spelling, so
shortening it would be a rename nobody asked for rather than a merge.
**The residual gap and its tell.** The function cannot know that two names for a
*new* bank are one institution. That always shows up the same way, so check it
rather than trusting the map:
**One institution's national entities stay apart (migration 0032).** A prefix
generalises past the evidence it was built on: every 0031 merge was provably one
account, but `citi%` would have flattened Citibank India into Citibank Australia
on arrival. `hsbc%` and `%american express%` had the same defect. The country now
comes from the name when the name states it and from the currency otherwise, with
AUD as home taking no suffix — so nothing renamed (`UPDATE 0`).
```sql
SELECT account_number, array_agg(DISTINCT bank_name)
FROM statements WHERE account_number IS NOT NULL
GROUP BY 1 HAVING count(DISTINCT bank_name) > 1;
```
Currency alone cannot be the discriminator: **Wise holds AUD, EUR and USD
accounts under one provider**, so "non-AUD means a different bank" would shatter
it into three. That is why Wise must stay unmapped, and why mapping any
multi-currency provider through this function would be wrong.
`N.A.` is deliberately not a country marker — it means "National Association", a
US legal form printed on Citibank letterhead worldwide including India.
**The residual gap and its tell.** The function cannot know that two names for a
*new* bank are one institution. That always shows up the same way, so check the
`statement_identity_drift` view (migration 0033) rather than trusting the map.
Non-empty means an account is fragmented, by name or by number.
### Account identity is derived, not the raw number (migration 0033)
`account_number` fragments exactly like `bank_name` did, from the same cause —
Gemini copies what the PDF prints. ANZ's Access Advantage arrived as both
`4085-56264` (4 statements) and `408556264` (1). Since `uq_statement_identity`
keyed on it, re-importing one period under the other spelling evaded the
duplicate check.
`account_number_key` is a **GENERATED ALWAYS** column — `account_number` with
separators stripped — and the unique index is now
`(bank_name, account_number_key, billing_end_date)`. **Never write to it;
display `account_number`.**
Derived rather than rewritten because the raw number is the readable one and
some of it is structure: Up stores `633-123 / 176540052`, a BSB *and* an account
number, and flattening it would lose a distinction a human reads at a glance to
fix a machine problem. Case and the Amex mask (`XXXX-XXXXXX-01000`) are
preserved — `X` records which digits were redacted.
This is **not** what caused the documented 31-row / $42,040.68 ANZ duplication.
Statements 107/142/143 overlap on *different* end dates, which that index cannot
catch at any spelling. Different problem, same table.
`account_owner_mappings` also keys on `(bank_name, account_number)` with its own
UNIQUE constraint, so any future rename must update it too or strand its rows.
It is empty today; the migration handles it anyway.
**Normalising `bank_name` broke the N8N new-bank alert, and the fix lives in the
workflow.** `Check Known Bank` ran
`SELECT COUNT(id) FROM statements WHERE bank_name = '<raw Gemini name>'`,
comparing the raw extraction against the now-canonical column *before* the insert
— so the trigger had not fired yet and nothing ever matched. That branch tags the
document **pending and holds it for Slack approval**, so every Zip statement
stalled awaiting a manual click, not merely a noisy alert. It was a regression
made worse by normalisation: previously a repeat of the same spelling at least
matched.
Fixed 2026-08-15 by comparing like with like:
```sql
SELECT COUNT(id) as count FROM statements
WHERE bank_name = normalize_bank_name('{{ ...summary.bank_name }}', '{{ ...summary.currency }}')
```
Currency is passed so a genuinely new national entity (Citibank India) still
alerts. Verified live: the run after the fix took 12s against 3053s for the
approval-branch runs before it. Workflow `FysADdFwEtwONQl4` in the smarthome
repo — **any future change to `normalize_bank_name()` must keep that node in
step**, since it is the one caller outside this codebase.
### Loans
A loan repayment is **not** an expense. It is part principal (equity, a