statements: derive account identity, fix the new-bank alert (0033)
ci / lint-test (push) Successful in 42s
ci / lint-test (push) Successful in 42s
account_number fragments exactly like bank_name did, from the same cause: ANZ's Access Advantage arrived as both '4085-56264' (4 statements) and '408556264' (1). uq_statement_identity keyed on it, so re-importing one period under the other spelling evaded the duplicate check. account_number_key is GENERATED ALWAYS — the number with separators stripped — and the unique index moves onto it. Derived rather than rewritten because the raw number is the readable one and some of it is structure: Up stores '633-123 / 176540052', a BSB and an account number, and flattening it would lose a distinction a human reads at a glance to fix a machine problem. The Amex mask and case are preserved; 'X' records which digits were redacted. Not the cause of the documented 31-row / $42,040.68 ANZ duplication — 107/142/143 overlap on different end dates, which that index cannot catch at any spelling. Adds statement_identity_drift: one account under two bank names, or one bank under two spellings of one number. Non-empty means fragmentation the normaliser could not know about. Currently one row, the ANZ pair above, now unified by key. Also documents the N8N regression this series caused. Check Known Bank compared the raw Gemini name against the canonical column before the insert, so nothing ever matched and every Zip statement was tagged pending and held for Slack approval — a stall, not just a noisy alert. Fixed in workflow FysADdFwEtwONQl4 by comparing like with like through normalize_bank_name(), passing currency so a genuinely new national entity still alerts. Verified live: 12s after vs 30-53s before.
This commit is contained in:
@@ -898,20 +898,76 @@ print.
|
||||
Wise is deliberately unmapped: `Wise Australia Pty Ltd.` is a single spelling, so
|
||||
shortening it would be a rename nobody asked for rather than a merge.
|
||||
|
||||
**The residual gap and its tell.** The function cannot know that two names for a
|
||||
*new* bank are one institution. That always shows up the same way, so check it
|
||||
rather than trusting the map:
|
||||
**One institution's national entities stay apart (migration 0032).** A prefix
|
||||
generalises past the evidence it was built on: every 0031 merge was provably one
|
||||
account, but `citi%` would have flattened Citibank India into Citibank Australia
|
||||
on arrival. `hsbc%` and `%american express%` had the same defect. The country now
|
||||
comes from the name when the name states it and from the currency otherwise, with
|
||||
AUD as home taking no suffix — so nothing renamed (`UPDATE 0`).
|
||||
|
||||
```sql
|
||||
SELECT account_number, array_agg(DISTINCT bank_name)
|
||||
FROM statements WHERE account_number IS NOT NULL
|
||||
GROUP BY 1 HAVING count(DISTINCT bank_name) > 1;
|
||||
```
|
||||
Currency alone cannot be the discriminator: **Wise holds AUD, EUR and USD
|
||||
accounts under one provider**, so "non-AUD means a different bank" would shatter
|
||||
it into three. That is why Wise must stay unmapped, and why mapping any
|
||||
multi-currency provider through this function would be wrong.
|
||||
|
||||
`N.A.` is deliberately not a country marker — it means "National Association", a
|
||||
US legal form printed on Citibank letterhead worldwide including India.
|
||||
|
||||
**The residual gap and its tell.** The function cannot know that two names for a
|
||||
*new* bank are one institution. That always shows up the same way, so check the
|
||||
`statement_identity_drift` view (migration 0033) rather than trusting the map.
|
||||
Non-empty means an account is fragmented, by name or by number.
|
||||
|
||||
### Account identity is derived, not the raw number (migration 0033)
|
||||
|
||||
`account_number` fragments exactly like `bank_name` did, from the same cause —
|
||||
Gemini copies what the PDF prints. ANZ's Access Advantage arrived as both
|
||||
`4085-56264` (4 statements) and `408556264` (1). Since `uq_statement_identity`
|
||||
keyed on it, re-importing one period under the other spelling evaded the
|
||||
duplicate check.
|
||||
|
||||
`account_number_key` is a **GENERATED ALWAYS** column — `account_number` with
|
||||
separators stripped — and the unique index is now
|
||||
`(bank_name, account_number_key, billing_end_date)`. **Never write to it;
|
||||
display `account_number`.**
|
||||
|
||||
Derived rather than rewritten because the raw number is the readable one and
|
||||
some of it is structure: Up stores `633-123 / 176540052`, a BSB *and* an account
|
||||
number, and flattening it would lose a distinction a human reads at a glance to
|
||||
fix a machine problem. Case and the Amex mask (`XXXX-XXXXXX-01000`) are
|
||||
preserved — `X` records which digits were redacted.
|
||||
|
||||
This is **not** what caused the documented 31-row / $42,040.68 ANZ duplication.
|
||||
Statements 107/142/143 overlap on *different* end dates, which that index cannot
|
||||
catch at any spelling. Different problem, same table.
|
||||
|
||||
`account_owner_mappings` also keys on `(bank_name, account_number)` with its own
|
||||
UNIQUE constraint, so any future rename must update it too or strand its rows.
|
||||
It is empty today; the migration handles it anyway.
|
||||
|
||||
**Normalising `bank_name` broke the N8N new-bank alert, and the fix lives in the
|
||||
workflow.** `Check Known Bank` ran
|
||||
`SELECT COUNT(id) FROM statements WHERE bank_name = '<raw Gemini name>'`,
|
||||
comparing the raw extraction against the now-canonical column *before* the insert
|
||||
— so the trigger had not fired yet and nothing ever matched. That branch tags the
|
||||
document **pending and holds it for Slack approval**, so every Zip statement
|
||||
stalled awaiting a manual click, not merely a noisy alert. It was a regression
|
||||
made worse by normalisation: previously a repeat of the same spelling at least
|
||||
matched.
|
||||
|
||||
Fixed 2026-08-15 by comparing like with like:
|
||||
|
||||
```sql
|
||||
SELECT COUNT(id) as count FROM statements
|
||||
WHERE bank_name = normalize_bank_name('{{ ...summary.bank_name }}', '{{ ...summary.currency }}')
|
||||
```
|
||||
|
||||
Currency is passed so a genuinely new national entity (Citibank India) still
|
||||
alerts. Verified live: the run after the fix took 12s against 30–53s for the
|
||||
approval-branch runs before it. Workflow `FysADdFwEtwONQl4` in the smarthome
|
||||
repo — **any future change to `normalize_bank_name()` must keep that node in
|
||||
step**, since it is the one caller outside this codebase.
|
||||
|
||||
### Loans
|
||||
|
||||
A loan repayment is **not** an expense. It is part principal (equity, a
|
||||
|
||||
Reference in New Issue
Block a user