diff --git a/CLAUDE.md b/CLAUDE.md index 6b799e8..69df4bd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -268,6 +268,56 @@ by the `rn` window and ordered by amount closeness, so a looser band changes live queue at the time of the change: candidate counts went **0 → 0** on all 12 rows, i.e. no new noise. **The auto-matcher stays exact** — see `findAutoMatches()`. +### Upcoming annual charges (`GET /api/annual-charges`, migration 0038) + +Card annual fees, loan package fees and annual subscriptions, derived from +**payments alone** — independent of the email lane, because a bank announces an +annual fee on the statement and nowhere else. + +**`transaction_type = 'fee'` is the universe, not a description match.** The +extractor already classifies fees structurally (131 rows), and *every* card and +loan annual fee on this ledger is typed that way. The first version of this +filtered `transaction_type = 'debit'` and returned **none** of them. The fees are +not debits. + +**Two signals, kept apart because they are not equally trustworthy:** + +| confidence | basis | +|---|---| +| `confirmed` | typed a fee **and** repeated on an annual cadence | +| `likely` | typed a fee, seen once, and *named* annual/card/package — the kind is certain, the cadence inferred | +| `possible` | no fee wording; a merchant whose every gap is annual **and** whose amount is stable | + +**A naive "two charges ~365 days apart" is useless** — measured on this ledger it +ranks a bakery first with **202** such pairs, then a donation, Patreon, Coles. +Frequency is the confound. Even "every gap is annual" admits a Target visited +twice ($30, then $198). The `possible` tier therefore also requires amount +stability (≤20%, which absorbs VicRoads' ~3%/yr CPI drift) and ≥2 observations. +It still carries real false positives (H&M, Superdry) — that is what the label is +for, and why it must never render like a bank fact. + +**`expected_not_seen` is the more interesting status.** The estimate passed and +nothing landed, which means the account stopped being imported, or closed, or the +charge arrived under a different label. `account_days_stale` is the +discriminator: Westpac at 0 days stale is trustworthy; ANZ at 392 is simply +missing data. Never report these as upcoming. + +**Account continuity (`account_links`, migration 0038).** One card can appear +under two identities: Citi Prestige `5258…0253` (to 2025-11-13) became NAB MyCard +`1015…6227` (from 2025-11-14), same 14th-to-13th cycle, no gap. Without the link +the $700 fee reads as two dead one-offs instead of one due again ~Nov 2026. The +table is **hand-asserted with evidence** (`source_documents` ids), not inferred — +a contiguous-cycle heuristic would join two genuinely different accounts as +readily as one renamed one. The resolver walks it forward recursively; +`uq_account_links_from` forbids a fork and `account_links_no_self` a self-link, so +it terminates. **Display the resolved current bank**, not the one that printed the +last fee — naming the retired brand sends you to the wrong bank. + +Known limitation: the merchant signal fragments on label drift. VicRoads rego was +paid 2024/2025 as `VICROADS` and 2026 as `VICROADS ONLINE PAYM`, so the series +splits and the 2025 estimate shows as `expected_not_seen`. Not fuzzy-matched on +purpose — see the rejected pg_trgm merchant grouping. + ### Cash and reconciliation `payment_method` in (`cash`, `credits`) excludes a transaction from diff --git a/prisma/migrations/0038_account_links/migration.sql b/prisma/migrations/0038_account_links/migration.sql new file mode 100644 index 0000000..daf72c6 --- /dev/null +++ b/prisma/migrations/0038_account_links/migration.sql @@ -0,0 +1,76 @@ +-- Account continuity across a rebrand, reissue or issuer change. +-- +-- WHY THIS EXISTS: one card can appear in `statements` under two completely +-- different identities. The live case is the Citi Prestige card, which moved to +-- NAB and was rebranded MyCard: +-- +-- Citibank 5258073330340253 statements ... 2025-11-13 +-- NAB 1015001000116227 statements 2025-11-14 ... +-- +-- Same 14th-to-13th billing cycle, no gap, one continuous product. Nothing in +-- the schema linked them, so anything grouping by (bank_name, account) sees a +-- dead account and a new one with no history. For the annual-fee detector that +-- is not cosmetic: the $700 fee charged 2024-12-13 and 2025-11-13 reads as +-- "stopped" when it is due again around November 2026 under the new name. +-- +-- WHY A TABLE AND NOT AN INFERENCE: contiguous-cycle heuristics would join two +-- genuinely different accounts as readily as one renamed one, and a wrong join +-- merges two real fee series. There are only a handful of these events in a +-- lifetime, each announced by mail, so each link is asserted once with the +-- document that proves it. `evidence` is not decoration -- it is how a future +-- reader checks the claim instead of trusting it. +-- +-- NOT `account_owner_mappings`: that answers "whose account is this" and keys on +-- the raw (bank_name, account_number). This answers "is this the same account as +-- that one", and keys on the normalised form. + +CREATE TABLE IF NOT EXISTS account_links ( + id bigserial PRIMARY KEY, + -- Normalised the same way statements.account_number_key is generated + -- (non-alphanumerics stripped), so these join without per-call cleanup. + -- Amex-style masks keep their X's: they record which digits were redacted. + from_bank_name text NOT NULL, + from_account_key text NOT NULL, + to_bank_name text NOT NULL, + to_account_key text NOT NULL, + -- When the successor took over. The predecessor's last statement and the + -- successor's first are the observable boundary; this is the announced date + -- where mail gives one. + effective_date date, + reason text NOT NULL + CHECK (reason IN ('rebrand', 'reissue', 'issuer_change', 'migration')), + -- Provenance for the claim, e.g. 'source_documents:32919'. Free text because + -- the evidence is not always a document -- sometimes it is a statement pair. + evidence text, + notes text, + created_at timestamptz NOT NULL DEFAULT now(), + + -- One successor per predecessor: a chain, never a fork. A fork would make + -- "the current identity of this account" ambiguous, and the resolver walks + -- this forward assuming it is not. + CONSTRAINT uq_account_links_from UNIQUE (from_bank_name, from_account_key), + -- A link to itself would make that walk non-terminating. + CONSTRAINT account_links_no_self CHECK ( + from_bank_name IS DISTINCT FROM to_bank_name + OR from_account_key IS DISTINCT FROM to_account_key + ) +); + +COMMENT ON TABLE account_links IS + 'Predecessor -> successor account identity across rebrands/reissues. Hand-asserted, each row carrying the evidence for its claim.'; + +CREATE INDEX IF NOT EXISTS account_links_to + ON account_links (to_bank_name, to_account_key); + +-- The one link the mail record proves today. +-- source_documents 32919 (2025-10-01) "your Citi Prestige Card account is moving to MyCard" +-- source_documents 32618 (2025-11-10) "MyCard is coming soon" +-- source_documents 32539 (2025-11-24) "The new MyCard App and MyCard Online are now live" +-- Statement boundary: Citibank ends 2025-11-13, NAB begins 2025-11-14. +INSERT INTO account_links + (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason, evidence, notes) +VALUES + ('Citibank', '5258073330340253', 'NAB', '1015001000116227', '2025-11-14', 'issuer_change', + 'source_documents:32919,32618,32539', + 'Citi Prestige -> MyCard Prestige. Billing cycle (14th-13th) continues unbroken across the boundary.') +ON CONFLICT (from_bank_name, from_account_key) DO NOTHING; diff --git a/src/__tests__/integration/annual-charges.test.ts b/src/__tests__/integration/annual-charges.test.ts new file mode 100644 index 0000000..4b3619c --- /dev/null +++ b/src/__tests__/integration/annual-charges.test.ts @@ -0,0 +1,170 @@ +import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest"; +import { Pool } from "pg"; +import { createPool, resetDB, seedParticipants, mockDbWithPool } from "./helpers"; + +/** + * Upcoming annual charges from payments alone. + * + * The exclusions carry this file. Anything can be made to find a card annual + * fee; the difficulty is not also reporting the monthly account-servicing fee, + * the per-transfer Wise charge, and the restaurant visited twice a year apart. + * Measured on the real ledger, a naive "two charges ~365 days apart" ranks a + * bakery first with 202 such pairs. + */ + +let pool: Pool; +let ownerId: number; +let getUpcomingAnnualCharges: typeof import("../../lib/annual-charges").getUpcomingAnnualCharges; + +let acctSeq = 0; + +/** A statement plus one transaction on it, for a named account. */ +async function charge( + bank: string, + acct: string, + date: string, + description: string, + amount: number, + type = "fee" +) { + const key = `${bank}:${acct}`; + let stId = stmtCache.get(`${key}:${date}`); + if (!stId) { + const st = await pool.query( + `INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id) + VALUES ($4, $1, $2, $3::date - 20, $3::date + 10, $5) RETURNING id`, + [acct, bank, date, `s${acctSeq++}.pdf`, ownerId] + ); + stId = st.rows[0].id as number; + stmtCache.set(`${key}:${date}`, stId); + } + await pool.query( + `INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id) + VALUES ($1,$2,$3,$4,$5,$6)`, + [stId, date, description, amount, type, ownerId] + ); +} +let stmtCache = new Map(); + +/** A statementless merchant purchase, for the non-fee signal. */ +async function purchase(date: string, merchant: string, amount: number) { + await pool.query( + `INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id, merchant_normalized, row_index) + VALUES (NULL,$1,$2,$3,'debit',$4,$2,0)`, + [date, merchant, amount, ownerId] + ); +} + +const find = (rows: T[], label: string) => + rows.find((r) => r.label === label); + +beforeAll(async () => { + pool = createPool(); + mockDbWithPool(pool); + ({ getUpcomingAnnualCharges } = await import("../../lib/annual-charges")); +}); + +beforeEach(async () => { + await resetDB(pool); + await pool.query("DELETE FROM account_links"); + stmtCache = new Map(); + ({ ownerId } = await seedParticipants(pool)); +}); + +describe("what it finds", () => { + it("confirms a card annual fee that repeated a year later", async () => { + await charge("American Express", "1000", "2025-06-10", "ANNUAL CARD FEE", 875); + await charge("American Express", "1000", "2026-06-10", "ANNUAL CARD FEE", 1750); + const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CARD FEE"); + expect(row?.confidence).toBe("confirmed"); + expect(row?.observations).toBe(2); + // The amount is NOT averaged: this fee doubled, and the next one to plan for + // is the latest, not the mean of a rise. + expect(Number(row?.last_amount)).toBe(1750); + expect(row?.next_due_estimate.startsWith("2027-06")).toBe(true); + }); + + it("accepts a fee seen only once when the bank names it annual", async () => { + await charge("Westpac", "8032", "2025-10-21", "CARD FEE", 295); + const row = find(await getUpcomingAnnualCharges(ownerId), "CARD FEE"); + // One charge cannot show a cadence, but the kind is not in doubt. + expect(row?.confidence).toBe("likely"); + expect(row?.next_due_estimate).toBe("2026-10-21"); + }); + + it("finds an annual merchant charge with no fee wording, if the amount is stable", async () => { + await purchase("2025-02-03", "AMAZON PRIME", 79); + await purchase("2026-02-01", "AMAZON PRIME", 79); + const row = find(await getUpcomingAnnualCharges(ownerId), "AMAZON PRIME"); + expect(row?.confidence).toBe("possible"); + }); +}); + +describe("what it must NOT report", () => { + it("ignores a monthly account fee", async () => { + for (const d of ["2026-01-31", "2026-02-28", "2026-03-31", "2026-04-30"]) { + await charge("ANZ", "5626", d, "ACCOUNT SERVICING FEE", 5); + } + expect(find(await getUpcomingAnnualCharges(ownerId), "ACCOUNT SERVICING FEE")).toBeUndefined(); + }); + + it("ignores per-transaction fees, which never repeat under one label", async () => { + await charge("Wise Australia Pty Ltd.", "9001", "2026-05-20", "WISE CHARGES FOR: TRANSFER-2143509822", 30.67); + await charge("Wise Australia Pty Ltd.", "9001", "2026-06-04", "WISE CHARGES FOR: TRANSFER-2171403032", 19.51); + const rows = await getUpcomingAnnualCharges(ownerId); + expect(rows.filter((r) => r.label.startsWith("WISE CHARGES"))).toHaveLength(0); + }); + + it("ignores a merchant visited twice a year apart for wildly different amounts", async () => { + // The exact shape that defeats a pure cadence test: Target, $30 then $198. + await purchase("2025-07-21", "TARGET", 30); + await purchase("2026-07-11", "TARGET", 198); + expect(find(await getUpcomingAnnualCharges(ownerId), "TARGET")).toBeUndefined(); + }); + + it("ignores a merchant seen only once", async () => { + await purchase("2026-04-03", "UBER ONE MEMBERSHIP", 96); + expect(find(await getUpcomingAnnualCharges(ownerId), "UBER ONE MEMBERSHIP")).toBeUndefined(); + }); +}); + +describe("account identity and honesty about gaps", () => { + it("treats a renamed account as one continuous fee series", async () => { + // Citi Prestige -> NAB MyCard: same card, two bank names, two numbers. + await charge("Citibank", "5258073330340253", "2024-12-13", "Annual Fee", 700); + await charge("Citibank", "5258073330340253", "2025-11-13", "Annual Fee", 700); + await pool.query( + `INSERT INTO account_links (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason) + VALUES ('Citibank','5258073330340253','NAB','1015001000116227','2025-11-14','issuer_change')` + ); + // A statement under the NEW identity, so the link has somewhere to resolve to. + await charge("NAB", "1015001000116227", "2026-08-13", "Purchase", 29.99, "debit"); + + const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL FEE"); + expect(row?.observations).toBe(2); + expect(row?.confidence).toBe("confirmed"); + // Names the account's CURRENT bank -- the next fee comes from there, and + // sending someone to the retired brand is the bug this link exists to stop. + expect(row?.bank_name).toBe("NAB"); + }); + + it("marks a passed estimate as expected_not_seen rather than upcoming", async () => { + await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450); + await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450); + const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CHARGE"); + expect(row?.status).toBe("expected_not_seen"); + // And it reports how stale the account is, which is what separates + // "it stopped being charged" from "we stopped importing the statements". + expect(row?.account_days_stale).toBeGreaterThan(300); + }); + + it("sorts upcoming ahead of expected_not_seen", async () => { + await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450); + await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450); + await charge("Westpac", "8032", "2026-08-01", "CARD FEE", 295); + const rows = await getUpcomingAnnualCharges(ownerId); + const firstMissing = rows.findIndex((r) => r.status === "expected_not_seen"); + const lastUpcoming = rows.map((r) => r.status).lastIndexOf("upcoming"); + expect(lastUpcoming).toBeLessThan(firstMissing); + }); +}); diff --git a/src/app/api/annual-charges/route.ts b/src/app/api/annual-charges/route.ts new file mode 100644 index 0000000..c9a1910 --- /dev/null +++ b/src/app/api/annual-charges/route.ts @@ -0,0 +1,44 @@ +import { NextRequest, NextResponse } from "next/server"; +import { getCurrentUser } from "@/lib/auth"; +import { getUpcomingAnnualCharges } from "@/lib/annual-charges"; + +/** + * Upcoming annual charges, from payments alone. + * + * Lives here rather than in home-dashboard so the detection sits beside the + * ledger it queries, is covered by this repo's integration suite, and can be + * reused by a future Slack nudge without moving. home-dashboard reads it over + * networks_internal the way its other widgets read their sources. + * + * Machine access uses the same shared secret as the dashboard's other reads; + * a browser session scopes to that user instead. + */ +export async function GET(req: NextRequest) { + // Its OWN token, with no fallback to WALLET_INGEST_TOKEN. home-dashboard's + // compose states the rule plainly -- "this app never holds write creds" -- and + // WALLET_INGEST_TOKEN unlocks POST /api/wallet/ingest and the estate-wide + // reconcile sweep. Handing that to a read-only surface to save minting a + // secret would quietly convert a dashboard into a writer. Unset means closed. + const token = process.env.DASH_READ_TOKEN; + const supplied = req.headers.get("x-ingest-token"); + const machine = !!token && !!supplied && supplied === token; + + let ownerId: number | null = null; + if (machine) { + // A machine caller names the owner explicitly rather than defaulting to 1: + // silently assuming an owner is how an ingested order ended up invisible in + // every scoped view while sitting in the table. + const q = req.nextUrl.searchParams.get("owner_id"); + ownerId = q !== null && Number.isInteger(Number(q)) ? Number(q) : null; + if (ownerId === null) { + return NextResponse.json({ error: "owner_id is required for machine access" }, { status: 400 }); + } + } else { + const user = await getCurrentUser(req); + if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 }); + ownerId = user.id; + } + + const charges = await getUpcomingAnnualCharges(ownerId); + return NextResponse.json({ charges, count: charges.length }); +} diff --git a/src/lib/annual-charges.ts b/src/lib/annual-charges.ts new file mode 100644 index 0000000..0abc719 --- /dev/null +++ b/src/lib/annual-charges.ts @@ -0,0 +1,274 @@ +import { queryRaw } from "@/lib/db"; + +/** + * Upcoming annual charges, derived from payments alone. + * + * Independent of the email lane on purpose: a recurring CHARGE is visible in the + * ledger whether or not anything was ever emailed about it, which is the only + * way a card annual fee or a loan package fee shows up at all -- banks announce + * those on the statement, not in a renewal notice. + * + * WHAT DOES NOT WORK, measured before this was written: "two charges about 365 + * days apart" is useless on its own. Ranked that way the top hit on this ledger + * is a bakery with 202 such pairs, then a donation, Patreon, Coles -- merchants + * used often enough that some pair is inevitably a year apart. Frequency is the + * confound. Even "every gap is annual" still surfaces a Target visited twice + * ($30, then $198) and a Specsavers ($24, then $224). Two visits a year apart + * are not a fee. + * + * So there are two signals, and they are kept apart because they are not equally + * trustworthy -- a guess must never render beside a certainty as if it were one: + * + * confirmed the bank called it a fee AND it has repeated on an annual cadence + * likely the bank called it a fee, seen once -- the kind is certain, the + * cadence is inferred (an annual fee charged once is still annual) + * possible no fee wording; a merchant whose every gap is annual AND whose + * amount is stable. Catches Amazon Prime, Uber One, VicRoads rego, + * none of which carry fee wording. Also where false positives live. + */ + +export type Confidence = "confirmed" | "likely" | "possible"; + +/** + * `expected_not_seen` is not a lesser version of `upcoming` -- it is the more + * interesting row. The estimate has passed and no charge landed, which means one + * of three things, and `account_days_stale` is what tells them apart: + * the account stopped being imported (HSBC: ledger ends 2026-01-07, while the + * mail shows statements to 2026-02-07) + * the account genuinely closed + * the charge landed under a different label (VicRoads paid 2026-03-16 as + * "VICROADS ONLINE PAYM", not "VICROADS") + * Reporting it as upcoming would be a lie; hiding it would lose the signal. + */ +export type ChargeStatus = "upcoming" | "expected_not_seen"; + +export type AnnualCharge = { + label: string; + status: ChargeStatus; + bank_name: string | null; + account_number: string | null; + confidence: Confidence; + observations: number; + last_charged: string; + last_amount: string; + amount_low: string; + amount_high: string; + median_gap_days: number | null; + next_due_estimate: string; + /** Ledger liveness for the account: a fee that "stopped" may just be an + * account that stopped being imported. Null for merchant-signal rows, which + * are not tied to one account. */ + account_last_statement: string | null; + account_days_stale: number | null; +}; + +/** + * `transaction_type = 'fee'` is the universe, not a description match. + * + * The statement extractor already classifies fees structurally -- 131 rows + * across the ledger -- and every card and loan annual fee on this ledger is + * typed that way. An earlier version of this filtered `transaction_type = + * 'debit'` and returned NONE of them, which is the whole reason this comment + * exists: the fees are not debits. + * + * That universe is broader than "annual", though. It also holds ANZ's monthly + * ACCOUNT SERVICING FEE, Zip's MONTHLY ACCOUNT SERVICE FEE, CommBank's MONTHLY + * FEE, Citibank LATE PAYMENT FEE, HSBC OVERSEAS TRANSACTION FEE and every Wise + * per-transfer charge. Cadence separates them: an annual fee repeats about a + * year apart, the rest repeat monthly, on an event, or never (Wise labels each + * transfer uniquely, so they can never group to two observations at all). + */ +const FEE_TYPE = "fee"; + +/** + * Wording that establishes the KIND when there is only one observation. + * + * A single charge cannot demonstrate a cadence, but a bank calling something an + * annual fee has already told us what it is -- Westpac's CARD FEE $295 and AMP's + * Professional Pack Annual Fee each appear once and are certainly annual. + * Deliberately narrow: bare "fee" would readmit every per-transaction charge. + */ +const ANNUAL_WORDING = + "(annual|yearly)[[:space:]]*(card[[:space:]]*)?(fee|charge)" + + "|card[[:space:]]+fee" + + "|(package|membership|facility)[[:space:]]+fee"; + +/** + * Resolve every statement to a canonical account identity, following + * account_links forward. One card can appear under two bank names and two + * numbers after an issuer change (Citi Prestige -> NAB MyCard); without this the + * $700 fee charged 2024-12-13 and 2025-11-13 reads as two dead one-offs. + * + * Recursive because the table permits a chain (A->B->C); `uq_account_links_from` + * forbids a fork, and `account_links_no_self` a self-link, so the walk + * terminates. The depth guard is belt-and-braces against a cycle someone inserts + * by hand (A->B, B->A), which no constraint can catch. + */ +const RESOLVED_ACCOUNTS = ` + WITH RECURSIVE chain AS ( + SELECT s.id AS statement_id, s.bank_name, s.account_number, + s.bank_name AS cur_bank, s.account_number_key AS cur_key, 0 AS depth + FROM statements s + UNION ALL + SELECT c.statement_id, c.bank_name, c.account_number, + l.to_bank_name, l.to_account_key, c.depth + 1 + FROM chain c + JOIN account_links l + ON l.from_bank_name = c.cur_bank AND l.from_account_key = c.cur_key + WHERE c.depth < 10 + ), + resolved AS ( + SELECT DISTINCT ON (statement_id) + statement_id, bank_name, account_number, cur_bank AS acct_bank, cur_key AS acct_key + FROM chain ORDER BY statement_id, depth DESC + )`; + +export async function getUpcomingAnnualCharges(ownerId: number): Promise { + // Signal 1 -- the bank's own fee wording, grouped by resolved account identity + // and by the fee's wording so two different fees on one card stay apart + // (ANZ charges "ANNUAL FEE" $120 and "ADDITIONAL CARD FEE" $10 on the same + // statement, and they are not the same obligation). + const feeRows = await queryRaw( + `${RESOLVED_ACCOUNTS}, + fees AS ( + SELECT r.acct_bank, r.acct_key, r.bank_name, r.account_number, + upper(trim(regexp_replace(t.description, '[[:space:]]+', ' ', 'g'))) AS label, + t.transaction_date, t.amount + FROM transactions t + JOIN resolved r ON r.statement_id = t.statement_id + JOIN statements s ON s.id = t.statement_id + WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL + AND t.amount > 0 + AND COALESCE(t.owner_id, s.owner_id) = $1 + -- Structural first, wording only as a widening: a bank that types a + -- fee as a plain debit still gets picked up if it names it annual. + AND (t.transaction_type = $3 OR t.description ~* $2) + ), + gapped AS ( + SELECT f.*, + transaction_date - lag(transaction_date) + OVER (PARTITION BY acct_key, label ORDER BY transaction_date) AS gap + FROM fees f + ), + agg AS ( + SELECT acct_key, acct_bank, label, + count(*)::int AS observations, + max(transaction_date) AS last_charged, + min(amount) AS amount_low, max(amount) AS amount_high, + percentile_cont(0.5) WITHIN GROUP (ORDER BY gap) + FILTER (WHERE gap IS NOT NULL) AS median_gap, + count(*) FILTER (WHERE gap IS NOT NULL) AS gaps, + count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps, + (array_agg(bank_name ORDER BY transaction_date DESC))[1] AS bank_name, + (array_agg(account_number ORDER BY transaction_date DESC))[1] AS account_number, + (array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount + FROM gapped GROUP BY acct_key, acct_bank, label + ), + liveness AS ( + SELECT r.acct_key, max(s.billing_end_date) AS last_statement + FROM statements s JOIN resolved r ON r.statement_id = s.id + GROUP BY r.acct_key + ) + SELECT a.label, + -- The account's CURRENT identity, not the bank that happened to + -- print the last fee. That $700 fee last appeared under "Citibank" + -- in Nov 2025, but the account is NAB/MyCard now and the next one + -- will come from there; naming the dead brand would send you to the + -- wrong bank. + a.acct_bank AS bank_name, a.account_number, + CASE WHEN (a.last_charged + COALESCE(round(a.median_gap)::int, 365)) >= CURRENT_DATE + THEN 'upcoming' ELSE 'expected_not_seen' END AS status, + -- One observation cannot prove a cadence, but the bank naming it an + -- annual fee already tells us the kind. Repeated on an annual gap it + -- is confirmed; seen once it is likely and says so. + CASE WHEN a.gaps > 0 AND a.annual_gaps = a.gaps THEN 'confirmed' + ELSE 'likely' END AS confidence, + a.observations, + a.last_charged::text AS last_charged, + a.last_amount::text AS last_amount, + a.amount_low::text AS amount_low, + a.amount_high::text AS amount_high, + round(a.median_gap)::int AS median_gap_days, + (a.last_charged + COALESCE(round(a.median_gap)::int, 365))::text AS next_due_estimate, + l.last_statement::text AS account_last_statement, + (CURRENT_DATE - l.last_statement)::int AS account_days_stale + FROM agg a LEFT JOIN liveness l ON l.acct_key = a.acct_key + -- Two ways to earn a place, and nothing else gets one. Filtered here, not + -- in the UI, so the API can never be read wrongly by a future caller. + WHERE + -- Repeated on an annual cadence. Excludes ANZ's monthly ACCOUNT + -- SERVICING FEE and Zip's MONTHLY ACCOUNT SERVICE FEE, whose gaps are + -- ~30 days, and Citibank's LATE PAYMENT FEE, which is event-driven. + (a.gaps > 0 AND a.annual_gaps = a.gaps AND a.median_gap >= 300) + -- Or seen once, but named as an annual/card/package fee. Westpac's CARD + -- FEE and AMP's Professional Pack Annual Fee are each a single charge and + -- unmistakably annual. A Wise per-transfer fee is also a single charge + -- and says nothing of the sort, which is what keeps it out. + OR (a.gaps = 0 AND a.label ~* $2)`, + [ownerId, ANNUAL_WORDING, FEE_TYPE] + ); + + // Signal 2 -- merchant-charged annual items with no fee wording. Requires + // BOTH that every gap is annual AND that the amount is stable: VicRoads rego + // rises about 3% a year ($825.23 -> $852.88 -> $875.48), so equality is too + // strict, while no amount test at all readmits the twice-visited restaurant. + // Keyed on merchant, never on card -- that rego was paid on three different + // banks, and a card-keyed view would see three unrelated one-offs. + const merchantRows = await queryRaw( + `WITH t AS ( + SELECT upper(trim(COALESCE(t.merchant_normalized, t.merchant_name, t.description))) AS label, + t.transaction_date, t.amount + FROM transactions t + LEFT JOIN statements s ON s.id = t.statement_id + WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL + AND t.transaction_type = 'debit' AND t.amount > 0 + AND COALESCE(t.owner_id, s.owner_id) = $1 + AND t.transaction_type = 'debit' AND t.description !~* $2 + ), + gapped AS ( + SELECT t.*, transaction_date - lag(transaction_date) + OVER (PARTITION BY label ORDER BY transaction_date) AS gap + FROM t + ), + agg AS ( + SELECT label, count(*)::int AS observations, + max(transaction_date) AS last_charged, + min(amount) AS amount_low, max(amount) AS amount_high, + percentile_cont(0.5) WITHIN GROUP (ORDER BY gap) + FILTER (WHERE gap IS NOT NULL) AS median_gap, + count(*) FILTER (WHERE gap IS NOT NULL) AS gaps, + count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps, + (array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount + FROM gapped GROUP BY label + ) + SELECT label, NULL::text AS bank_name, NULL::text AS account_number, + CASE WHEN (last_charged + round(median_gap)::int) >= CURRENT_DATE + THEN 'upcoming' ELSE 'expected_not_seen' END AS status, + 'possible' AS confidence, observations, + last_charged::text AS last_charged, + last_amount::text AS last_amount, + amount_low::text AS amount_low, + amount_high::text AS amount_high, + round(median_gap)::int AS median_gap_days, + (last_charged + round(median_gap)::int)::text AS next_due_estimate, + NULL::text AS account_last_statement, + NULL::int AS account_days_stale + FROM agg + WHERE gaps >= 1 AND annual_gaps = gaps + -- Amount stability is what separates a subscription from a restaurant. + -- 20% absorbs CPI drift and an FX-priced renewal without admitting a + -- $30-then-$198 pair. + AND amount_high <= amount_low * 1.20 + -- Two observations is the floor for a cadence claim, and this lane has no + -- wording to fall back on the way the fee lane does. + AND observations >= 2`, + [ownerId, ANNUAL_WORDING] + ); + + // Upcoming first and soonest-first within that, so the thing you can still + // act on is at the top; the expected-but-missing rows follow as a review list. + const rank = (c: AnnualCharge) => (c.status === "upcoming" ? 0 : 1); + return [...feeRows, ...merchantRows].sort( + (a, b) => rank(a) - rank(b) || a.next_due_estimate.localeCompare(b.next_due_estimate) + ); +}