statements: keep one bank's national entities apart (migration 0032)
ci / lint-test (push) Successful in 44s
ci / lint-test (push) Successful in 44s
Fixes 0031. Every merge it made was provably one account — each split pair shared an account_number — but the PREFIX generalises past that evidence. Citibank Australia and Citibank India are different banks with different accounts and currency, and `citi%` flattens them into one. Older Citi India statements are queued to load, so this would have merged them on arrival. Not Citi-specific: `hsbc%` and `%american express%` carry the identical defect. Currency alone cannot be the discriminator: Wise holds AUD, EUR and USD accounts under one provider, so a blanket "non-AUD is a different bank" rule would shatter it into three. Wise stays unmapped for exactly that reason, now a documented property rather than an accident. The country comes from the name when the name states it and from the currency otherwise, with AUD as home taking no suffix — so nothing renames: the backfill touched 0 rows. 'Citibank' + AUD -> 'Citibank' (unchanged) 'Citibank India' + INR -> 'Citibank India' 'Citibank N.A.' + INR -> 'Citibank India' (currency carries it) 'CITIBANK INDIA' + AUD -> 'Citibank India' (name beats a wrong currency) 'N.A.' is deliberately not read as a country marker: it means "National Association", a US legal form printed on Citibank letterhead worldwide, including India. Reading it as US is precisely the mistake this fixes. The trigger now fires on UPDATE OF bank_name, currency, since the country is partly derived from the currency.
This commit is contained in:
@@ -0,0 +1,164 @@
|
|||||||
|
-- Keep one institution's national entities apart. Fixes 0031.
|
||||||
|
--
|
||||||
|
-- 0031 merged spellings by prefix — `citi%` -> 'Citibank'. Every merge it made
|
||||||
|
-- was provably one account (each split pair shared an account_number), but the
|
||||||
|
-- PREFIX generalises past that evidence: Citibank Australia and Citibank India
|
||||||
|
-- are different banks, different accounts, different currency, and `citi%`
|
||||||
|
-- flattens them into one. Older Citi India statements are queued to load, so
|
||||||
|
-- this would have merged them on arrival.
|
||||||
|
--
|
||||||
|
-- It is not a Citi problem. `hsbc%` and `%american express%` carry the identical
|
||||||
|
-- defect — all three are global banks operating as separate national entities.
|
||||||
|
--
|
||||||
|
-- WHY NOT DISCRIMINATE ON CURRENCY ALONE: Wise holds AUD, EUR and USD accounts
|
||||||
|
-- under one provider, so a blanket "non-AUD means a different bank" rule would
|
||||||
|
-- shatter it into three. Wise stays unmapped for exactly this reason — mapping
|
||||||
|
-- a genuinely multi-currency provider through this function would be wrong, and
|
||||||
|
-- that is now a documented property rather than an accident.
|
||||||
|
--
|
||||||
|
-- THE RULE: the country comes from the name when the name states it, and from
|
||||||
|
-- the currency otherwise. AUD is home and takes no suffix, so nothing renames
|
||||||
|
-- today — all 21 existing statements keep the name 0031 gave them.
|
||||||
|
--
|
||||||
|
-- 'Citibank' + AUD -> 'Citibank' (unchanged)
|
||||||
|
-- 'Citi' + AUD -> 'Citibank' (unchanged)
|
||||||
|
-- 'Citibank Australia'+ AUD -> 'Citibank'
|
||||||
|
-- 'Citibank India' + INR -> 'Citibank India'
|
||||||
|
-- 'Citibank N.A.' + INR -> 'Citibank India' (currency carries it)
|
||||||
|
-- 'HSBC' + AUD -> 'HSBC' (unchanged)
|
||||||
|
--
|
||||||
|
-- Degradation is safe but not silent: an India statement whose currency was
|
||||||
|
-- extracted as AUD and whose name omits the country WILL merge. The tell is a
|
||||||
|
-- second account_number appearing under one bank_name with a different
|
||||||
|
-- currency — see the check in CLAUDE.md.
|
||||||
|
--
|
||||||
|
-- Idempotent: safe to re-run.
|
||||||
|
|
||||||
|
-- 'N.A.' is deliberately absent: it means "National Association", a US legal
|
||||||
|
-- form that appears on Citibank letterhead worldwide, including India. Reading
|
||||||
|
-- it as a country marker is precisely the mistake this migration exists to fix.
|
||||||
|
CREATE OR REPLACE FUNCTION bank_country_suffix(v TEXT, cur TEXT)
|
||||||
|
RETURNS TEXT AS $$
|
||||||
|
DECLARE
|
||||||
|
c TEXT := upper(trim(coalesce(cur, '')));
|
||||||
|
BEGIN
|
||||||
|
-- The name wins when it states a country: it is the source's own assertion,
|
||||||
|
-- and it survives a currency that failed to extract.
|
||||||
|
IF v LIKE '%australia%' OR v LIKE '% au%' THEN RETURN ''; END IF; -- home
|
||||||
|
IF v LIKE '%india%' THEN RETURN ' India'; END IF;
|
||||||
|
IF v LIKE '%singapore%' THEN RETURN ' Singapore'; END IF;
|
||||||
|
IF v LIKE '%hong kong%' THEN RETURN ' Hong Kong'; END IF;
|
||||||
|
IF v LIKE '%new zealand%' THEN RETURN ' NZ'; END IF;
|
||||||
|
IF v LIKE '%united kingdom%' OR v LIKE '% uk%' THEN RETURN ' UK'; END IF;
|
||||||
|
IF v LIKE '%emirates%' OR v LIKE '% uae%' THEN RETURN ' UAE'; END IF;
|
||||||
|
|
||||||
|
-- Otherwise infer from the statement's own currency. AUD is home.
|
||||||
|
IF c = '' OR c = 'AUD' THEN RETURN ''; END IF;
|
||||||
|
|
||||||
|
IF c = 'INR' THEN RETURN ' India'; END IF;
|
||||||
|
IF c = 'SGD' THEN RETURN ' Singapore'; END IF;
|
||||||
|
IF c = 'HKD' THEN RETURN ' Hong Kong'; END IF;
|
||||||
|
IF c = 'NZD' THEN RETURN ' NZ'; END IF;
|
||||||
|
IF c = 'GBP' THEN RETURN ' UK'; END IF;
|
||||||
|
IF c = 'AED' THEN RETURN ' UAE'; END IF;
|
||||||
|
IF c = 'USD' THEN RETURN ' US'; END IF;
|
||||||
|
|
||||||
|
-- EUR spans countries and cannot name one, so fall back to the code itself.
|
||||||
|
-- A distinguishable name beats a wrong one.
|
||||||
|
RETURN ' ' || c;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql IMMUTABLE;
|
||||||
|
|
||||||
|
-- `currency` is a second parameter with a default so any existing single-arg
|
||||||
|
-- call keeps working; the trigger passes NEW.currency.
|
||||||
|
CREATE OR REPLACE FUNCTION normalize_bank_name(raw TEXT, cur TEXT DEFAULT NULL)
|
||||||
|
RETURNS TEXT AS $$
|
||||||
|
DECLARE
|
||||||
|
v TEXT;
|
||||||
|
BEGIN
|
||||||
|
IF raw IS NULL THEN
|
||||||
|
RETURN NULL;
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
v := lower(regexp_replace(trim(raw), '\s+', ' ', 'g'));
|
||||||
|
|
||||||
|
IF v = '' THEN
|
||||||
|
RETURN NULL;
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
-- ---- Merges: institutions seen under more than one spelling ----
|
||||||
|
-- Each carries a country suffix, empty for AUD, so one bank's national
|
||||||
|
-- entities stay apart while its spellings collapse.
|
||||||
|
|
||||||
|
IF v LIKE 'zip%' THEN
|
||||||
|
RETURN 'Zip' || bank_country_suffix(v, cur);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v LIKE '%american express%' OR v LIKE 'amex%' THEN
|
||||||
|
RETURN 'American Express' || bank_country_suffix(v, cur);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v LIKE 'citi%' THEN
|
||||||
|
RETURN 'Citibank' || bank_country_suffix(v, cur);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v LIKE '%national australia bank%' OR v = 'nab' THEN
|
||||||
|
RETURN 'NAB';
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
-- ---- Defensive: map future legal-entity variants onto the name in use ----
|
||||||
|
|
||||||
|
IF v LIKE '%westpac%' THEN
|
||||||
|
RETURN 'Westpac' || bank_country_suffix(v, cur);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v = 'anz' OR v LIKE 'anz %' OR v LIKE '%australia and new zealand banking%' THEN
|
||||||
|
RETURN 'ANZ' || bank_country_suffix(v, cur);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v LIKE '%commonwealth bank%' OR v LIKE 'commbank%' OR v = 'cba' THEN
|
||||||
|
RETURN 'Commonwealth Bank';
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v LIKE 'hsbc%' THEN
|
||||||
|
RETURN 'HSBC' || bank_country_suffix(v, cur);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v = 'ing' OR v LIKE 'ing %' THEN
|
||||||
|
RETURN 'ING' || bank_country_suffix(v, cur);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v = 'amp' OR v LIKE 'amp %' THEN
|
||||||
|
RETURN 'AMP Bank';
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF v = 'up' OR v LIKE 'up bank%' THEN
|
||||||
|
RETURN 'Up';
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
-- Unknown institution: keep exactly what was extracted, tidied only.
|
||||||
|
-- Wise reaches here, which is what keeps its AUD/EUR/USD accounts together.
|
||||||
|
RETURN regexp_replace(trim(raw), '\s+', ' ', 'g');
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql IMMUTABLE;
|
||||||
|
|
||||||
|
CREATE OR REPLACE FUNCTION statements_normalize_bank_name_trigger()
|
||||||
|
RETURNS TRIGGER AS $$
|
||||||
|
BEGIN
|
||||||
|
NEW.bank_name := normalize_bank_name(NEW.bank_name, NEW.currency);
|
||||||
|
RETURN NEW;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql;
|
||||||
|
|
||||||
|
-- Re-point the trigger: it must fire when `currency` changes too, since the
|
||||||
|
-- country is now partly derived from it.
|
||||||
|
DROP TRIGGER IF EXISTS trg_statements_normalize_bank_name ON statements;
|
||||||
|
CREATE TRIGGER trg_statements_normalize_bank_name
|
||||||
|
BEFORE INSERT OR UPDATE OF bank_name, currency ON statements
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION statements_normalize_bank_name_trigger();
|
||||||
|
|
||||||
|
-- Backfill. Expected to touch nothing: every existing statement is AUD except
|
||||||
|
-- Wise, which is unmapped.
|
||||||
|
UPDATE statements
|
||||||
|
SET bank_name = normalize_bank_name(bank_name, currency)
|
||||||
|
WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name, currency);
|
||||||
Reference in New Issue
Block a user