statements: derive account identity, fix the new-bank alert (0033)
ci / lint-test (push) Successful in 42s

account_number fragments exactly like bank_name did, from the same cause:
ANZ's Access Advantage arrived as both '4085-56264' (4 statements) and
'408556264' (1). uq_statement_identity keyed on it, so re-importing one
period under the other spelling evaded the duplicate check.

account_number_key is GENERATED ALWAYS — the number with separators
stripped — and the unique index moves onto it. Derived rather than
rewritten because the raw number is the readable one and some of it is
structure: Up stores '633-123 / 176540052', a BSB and an account number,
and flattening it would lose a distinction a human reads at a glance to
fix a machine problem. The Amex mask and case are preserved; 'X' records
which digits were redacted.

Not the cause of the documented 31-row / $42,040.68 ANZ duplication —
107/142/143 overlap on different end dates, which that index cannot catch
at any spelling.

Adds statement_identity_drift: one account under two bank names, or one
bank under two spellings of one number. Non-empty means fragmentation the
normaliser could not know about. Currently one row, the ANZ pair above,
now unified by key.

Also documents the N8N regression this series caused. Check Known Bank
compared the raw Gemini name against the canonical column before the
insert, so nothing ever matched and every Zip statement was tagged
pending and held for Slack approval — a stall, not just a noisy alert.
Fixed in workflow FysADdFwEtwONQl4 by comparing like with like through
normalize_bank_name(), passing currency so a genuinely new national
entity still alerts. Verified live: 12s after vs 30-53s before.
This commit is contained in:
2026-08-15 17:03:13 +10:00
parent c3b623052d
commit 2cfaf8e8c8
4 changed files with 151 additions and 15 deletions
@@ -0,0 +1,71 @@
-- One account, one identity — without destroying how the number reads.
--
-- `account_number` carries the same fragmentation `bank_name` had, from the
-- same cause: Gemini copies whatever the PDF prints, and the formatting varies
-- per document.
--
-- ANZ '4085-56264' 4 statements ANZ ACCESS ADVANTAGE
-- ANZ '408556264' 1 statement ACCESS ADVANTAGE <- same account
--
-- That matters because `uq_statement_identity` is
-- (bank_name, account_number, billing_end_date). Re-importing one period under
-- the other spelling evades the duplicate check entirely.
--
-- (It is NOT what caused the documented 31-row / $42,040.68 ANZ duplication.
-- Statements 107/142/143 overlap on *different* end dates, which that index
-- cannot catch at any spelling. Different problem, same table.)
--
-- WHY A GENERATED COLUMN RATHER THAN REWRITING THE VALUE: the raw number is the
-- readable one and some of it is structure, not noise. Up stores
-- '633-123 / 176540052' — a BSB and an account number — and flattening that to
-- '633123176540052' would lose a distinction a human reads at a glance, to fix
-- a machine problem. So the raw text stays exactly as extracted and the
-- comparison key is derived beside it. Same reason the app derives trip
-- participation instead of storing it: two records of one fact drift.
--
-- Separators only. Case and the Amex mask ('XXXX-XXXXXX-01000') are preserved,
-- because 'X' is real information about which digits were redacted, and
-- case-folding an alphanumeric account id could merge two genuinely different
-- ones.
--
-- Verified collision-free before rebuilding the index: no two statements share
-- (bank_name, stripped account_number, billing_end_date).
--
-- Idempotent: safe to re-run.
ALTER TABLE statements
ADD COLUMN IF NOT EXISTS account_number_key TEXT
GENERATED ALWAYS AS (
NULLIF(regexp_replace(coalesce(account_number, ''), '[^0-9A-Za-z]', '', 'g'), '')
) STORED;
COMMENT ON COLUMN statements.account_number_key IS
'Derived from account_number, separators stripped. The identity used by '
'uq_statement_identity so one account under two formats is one account. '
'Never write to it — it is GENERATED. Display account_number instead.';
-- Rebuild the identity index on the derived key. Dropped and recreated rather
-- than added alongside: leaving the old one in place would keep admitting the
-- duplicate it exists to stop.
DROP INDEX IF EXISTS uq_statement_identity;
CREATE UNIQUE INDEX uq_statement_identity
ON statements (bank_name, account_number_key, billing_end_date)
WHERE bank_name IS NOT NULL AND billing_end_date IS NOT NULL;
-- The fragmentation detector, now able to see both halves: one account under
-- two bank names, or one bank under two spellings of one number.
CREATE OR REPLACE VIEW statement_identity_drift AS
SELECT account_number_key,
array_agg(DISTINCT bank_name) AS bank_names,
array_agg(DISTINCT account_number) AS account_numbers,
count(*) AS statements
FROM statements
WHERE account_number_key IS NOT NULL
GROUP BY account_number_key
HAVING count(DISTINCT bank_name) > 1
OR count(DISTINCT account_number) > 1;
COMMENT ON VIEW statement_identity_drift IS
'Non-empty means an account is fragmented: normalize_bank_name() cannot know '
'that two names for a bank it has never seen are one institution, and this is '
'how that always shows up. Check it after loading a new bank.';