annual charges: surface upcoming card/loan fees from payments alone
ci / lint-test (push) Successful in 56s

Answers "what annual payments are coming?" without touching the email lane --
a bank announces an annual fee on the statement and nowhere else.

WHAT DOES NOT WORK, measured before writing it: "two charges ~365 days apart"
ranks a BAKERY first on this ledger, with 202 such pairs, then a donation,
Patreon, Coles. Frequency is the confound. Tightening to "every gap is annual"
still admits a Target visited twice ($30, then $198).

transaction_type='fee' is the universe, not a description match. The extractor
already classifies fees structurally (131 rows) and every card and loan annual
fee here is typed that way -- the first cut of this filtered
transaction_type='debit' and returned NONE of them. The fees are not debits.

Three confidence tiers, kept apart so a guess never renders like a bank fact:
  confirmed  typed a fee AND repeated on an annual cadence
  likely     typed a fee, seen once, and NAMED annual/card/package -- one charge
             cannot show a cadence but the kind is not in doubt
  possible   no fee wording; merchant whose every gap is annual AND whose amount
             is stable (<=20%, absorbing VicRoads' ~3%/yr drift). Catches Amazon
             Prime and rego; also where the false positives live, hence the label.

expected_not_seen is the more interesting status: the estimate passed and
nothing landed. account_days_stale is the discriminator -- Westpac at 0 days is
trustworthy, ANZ at 392 is simply missing data. Reporting those as upcoming
would be a lie.

account_links (migration 0038) fixes account continuity. Citi Prestige
5258...0253 became NAB MyCard 1015...6227 -- same card, same 14th-to-13th cycle,
no gap -- so the $700 fee charged 2024-12-13 and 2025-11-13 read as two dead
one-offs instead of one due again ~Nov 2026. Hand-asserted with source_documents
ids as evidence rather than inferred: a contiguous-cycle heuristic would join two
genuinely different accounts as readily as one renamed one. Display resolves to
the CURRENT bank; naming the retired brand sends you to the wrong bank.

The GET takes its own DASH_READ_TOKEN with no fallback to WALLET_INGEST_TOKEN --
that one unlocks wallet ingest and the estate-wide reconcile sweep, and
home-dashboard's compose states the rule: it never holds a write credential.

Known limit: the merchant signal fragments on label drift (VicRoads paid as
VICROADS then VICROADS ONLINE PAYM). Not fuzzy-matched, per the rejected pg_trgm
merchant grouping.

339 integration + 196 unit green, tsc clean. Board #265.
This commit is contained in:
2026-08-23 21:44:17 +10:00
parent 10cf0aaf21
commit 5076f27f68
5 changed files with 614 additions and 0 deletions
+50
View File
@@ -268,6 +268,56 @@ by the `rn` window and ordered by amount closeness, so a looser band changes
live queue at the time of the change: candidate counts went **0 → 0** on all 12
rows, i.e. no new noise. **The auto-matcher stays exact** — see `findAutoMatches()`.
### Upcoming annual charges (`GET /api/annual-charges`, migration 0038)
Card annual fees, loan package fees and annual subscriptions, derived from
**payments alone** — independent of the email lane, because a bank announces an
annual fee on the statement and nowhere else.
**`transaction_type = 'fee'` is the universe, not a description match.** The
extractor already classifies fees structurally (131 rows), and *every* card and
loan annual fee on this ledger is typed that way. The first version of this
filtered `transaction_type = 'debit'` and returned **none** of them. The fees are
not debits.
**Two signals, kept apart because they are not equally trustworthy:**
| confidence | basis |
|---|---|
| `confirmed` | typed a fee **and** repeated on an annual cadence |
| `likely` | typed a fee, seen once, and *named* annual/card/package — the kind is certain, the cadence inferred |
| `possible` | no fee wording; a merchant whose every gap is annual **and** whose amount is stable |
**A naive "two charges ~365 days apart" is useless** — measured on this ledger it
ranks a bakery first with **202** such pairs, then a donation, Patreon, Coles.
Frequency is the confound. Even "every gap is annual" admits a Target visited
twice ($30, then $198). The `possible` tier therefore also requires amount
stability (≤20%, which absorbs VicRoads' ~3%/yr CPI drift) and ≥2 observations.
It still carries real false positives (H&M, Superdry) — that is what the label is
for, and why it must never render like a bank fact.
**`expected_not_seen` is the more interesting status.** The estimate passed and
nothing landed, which means the account stopped being imported, or closed, or the
charge arrived under a different label. `account_days_stale` is the
discriminator: Westpac at 0 days stale is trustworthy; ANZ at 392 is simply
missing data. Never report these as upcoming.
**Account continuity (`account_links`, migration 0038).** One card can appear
under two identities: Citi Prestige `5258…0253` (to 2025-11-13) became NAB MyCard
`1015…6227` (from 2025-11-14), same 14th-to-13th cycle, no gap. Without the link
the $700 fee reads as two dead one-offs instead of one due again ~Nov 2026. The
table is **hand-asserted with evidence** (`source_documents` ids), not inferred —
a contiguous-cycle heuristic would join two genuinely different accounts as
readily as one renamed one. The resolver walks it forward recursively;
`uq_account_links_from` forbids a fork and `account_links_no_self` a self-link, so
it terminates. **Display the resolved current bank**, not the one that printed the
last fee — naming the retired brand sends you to the wrong bank.
Known limitation: the merchant signal fragments on label drift. VicRoads rego was
paid 2024/2025 as `VICROADS` and 2026 as `VICROADS ONLINE PAYM`, so the series
splits and the 2025 estimate shows as `expected_not_seen`. Not fuzzy-matched on
purpose — see the rejected pg_trgm merchant grouping.
### Cash and reconciliation
`payment_method` in (`cash`, `credits`) excludes a transaction from
@@ -0,0 +1,76 @@
-- Account continuity across a rebrand, reissue or issuer change.
--
-- WHY THIS EXISTS: one card can appear in `statements` under two completely
-- different identities. The live case is the Citi Prestige card, which moved to
-- NAB and was rebranded MyCard:
--
-- Citibank 5258073330340253 statements ... 2025-11-13
-- NAB 1015001000116227 statements 2025-11-14 ...
--
-- Same 14th-to-13th billing cycle, no gap, one continuous product. Nothing in
-- the schema linked them, so anything grouping by (bank_name, account) sees a
-- dead account and a new one with no history. For the annual-fee detector that
-- is not cosmetic: the $700 fee charged 2024-12-13 and 2025-11-13 reads as
-- "stopped" when it is due again around November 2026 under the new name.
--
-- WHY A TABLE AND NOT AN INFERENCE: contiguous-cycle heuristics would join two
-- genuinely different accounts as readily as one renamed one, and a wrong join
-- merges two real fee series. There are only a handful of these events in a
-- lifetime, each announced by mail, so each link is asserted once with the
-- document that proves it. `evidence` is not decoration -- it is how a future
-- reader checks the claim instead of trusting it.
--
-- NOT `account_owner_mappings`: that answers "whose account is this" and keys on
-- the raw (bank_name, account_number). This answers "is this the same account as
-- that one", and keys on the normalised form.
CREATE TABLE IF NOT EXISTS account_links (
id bigserial PRIMARY KEY,
-- Normalised the same way statements.account_number_key is generated
-- (non-alphanumerics stripped), so these join without per-call cleanup.
-- Amex-style masks keep their X's: they record which digits were redacted.
from_bank_name text NOT NULL,
from_account_key text NOT NULL,
to_bank_name text NOT NULL,
to_account_key text NOT NULL,
-- When the successor took over. The predecessor's last statement and the
-- successor's first are the observable boundary; this is the announced date
-- where mail gives one.
effective_date date,
reason text NOT NULL
CHECK (reason IN ('rebrand', 'reissue', 'issuer_change', 'migration')),
-- Provenance for the claim, e.g. 'source_documents:32919'. Free text because
-- the evidence is not always a document -- sometimes it is a statement pair.
evidence text,
notes text,
created_at timestamptz NOT NULL DEFAULT now(),
-- One successor per predecessor: a chain, never a fork. A fork would make
-- "the current identity of this account" ambiguous, and the resolver walks
-- this forward assuming it is not.
CONSTRAINT uq_account_links_from UNIQUE (from_bank_name, from_account_key),
-- A link to itself would make that walk non-terminating.
CONSTRAINT account_links_no_self CHECK (
from_bank_name IS DISTINCT FROM to_bank_name
OR from_account_key IS DISTINCT FROM to_account_key
)
);
COMMENT ON TABLE account_links IS
'Predecessor -> successor account identity across rebrands/reissues. Hand-asserted, each row carrying the evidence for its claim.';
CREATE INDEX IF NOT EXISTS account_links_to
ON account_links (to_bank_name, to_account_key);
-- The one link the mail record proves today.
-- source_documents 32919 (2025-10-01) "your Citi Prestige Card account is moving to MyCard"
-- source_documents 32618 (2025-11-10) "MyCard is coming soon"
-- source_documents 32539 (2025-11-24) "The new MyCard App and MyCard Online are now live"
-- Statement boundary: Citibank ends 2025-11-13, NAB begins 2025-11-14.
INSERT INTO account_links
(from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason, evidence, notes)
VALUES
('Citibank', '5258073330340253', 'NAB', '1015001000116227', '2025-11-14', 'issuer_change',
'source_documents:32919,32618,32539',
'Citi Prestige -> MyCard Prestige. Billing cycle (14th-13th) continues unbroken across the boundary.')
ON CONFLICT (from_bank_name, from_account_key) DO NOTHING;
@@ -0,0 +1,170 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, mockDbWithPool } from "./helpers";
/**
* Upcoming annual charges from payments alone.
*
* The exclusions carry this file. Anything can be made to find a card annual
* fee; the difficulty is not also reporting the monthly account-servicing fee,
* the per-transfer Wise charge, and the restaurant visited twice a year apart.
* Measured on the real ledger, a naive "two charges ~365 days apart" ranks a
* bakery first with 202 such pairs.
*/
let pool: Pool;
let ownerId: number;
let getUpcomingAnnualCharges: typeof import("../../lib/annual-charges").getUpcomingAnnualCharges;
let acctSeq = 0;
/** A statement plus one transaction on it, for a named account. */
async function charge(
bank: string,
acct: string,
date: string,
description: string,
amount: number,
type = "fee"
) {
const key = `${bank}:${acct}`;
let stId = stmtCache.get(`${key}:${date}`);
if (!stId) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ($4, $1, $2, $3::date - 20, $3::date + 10, $5) RETURNING id`,
[acct, bank, date, `s${acctSeq++}.pdf`, ownerId]
);
stId = st.rows[0].id as number;
stmtCache.set(`${key}:${date}`, stId);
}
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1,$2,$3,$4,$5,$6)`,
[stId, date, description, amount, type, ownerId]
);
}
let stmtCache = new Map<string, number>();
/** A statementless merchant purchase, for the non-fee signal. */
async function purchase(date: string, merchant: string, amount: number) {
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id, merchant_normalized, row_index)
VALUES (NULL,$1,$2,$3,'debit',$4,$2,0)`,
[date, merchant, amount, ownerId]
);
}
const find = <T extends { label: string }>(rows: T[], label: string) =>
rows.find((r) => r.label === label);
beforeAll(async () => {
pool = createPool();
mockDbWithPool(pool);
({ getUpcomingAnnualCharges } = await import("../../lib/annual-charges"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM account_links");
stmtCache = new Map();
({ ownerId } = await seedParticipants(pool));
});
describe("what it finds", () => {
it("confirms a card annual fee that repeated a year later", async () => {
await charge("American Express", "1000", "2025-06-10", "ANNUAL CARD FEE", 875);
await charge("American Express", "1000", "2026-06-10", "ANNUAL CARD FEE", 1750);
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CARD FEE");
expect(row?.confidence).toBe("confirmed");
expect(row?.observations).toBe(2);
// The amount is NOT averaged: this fee doubled, and the next one to plan for
// is the latest, not the mean of a rise.
expect(Number(row?.last_amount)).toBe(1750);
expect(row?.next_due_estimate.startsWith("2027-06")).toBe(true);
});
it("accepts a fee seen only once when the bank names it annual", async () => {
await charge("Westpac", "8032", "2025-10-21", "CARD FEE", 295);
const row = find(await getUpcomingAnnualCharges(ownerId), "CARD FEE");
// One charge cannot show a cadence, but the kind is not in doubt.
expect(row?.confidence).toBe("likely");
expect(row?.next_due_estimate).toBe("2026-10-21");
});
it("finds an annual merchant charge with no fee wording, if the amount is stable", async () => {
await purchase("2025-02-03", "AMAZON PRIME", 79);
await purchase("2026-02-01", "AMAZON PRIME", 79);
const row = find(await getUpcomingAnnualCharges(ownerId), "AMAZON PRIME");
expect(row?.confidence).toBe("possible");
});
});
describe("what it must NOT report", () => {
it("ignores a monthly account fee", async () => {
for (const d of ["2026-01-31", "2026-02-28", "2026-03-31", "2026-04-30"]) {
await charge("ANZ", "5626", d, "ACCOUNT SERVICING FEE", 5);
}
expect(find(await getUpcomingAnnualCharges(ownerId), "ACCOUNT SERVICING FEE")).toBeUndefined();
});
it("ignores per-transaction fees, which never repeat under one label", async () => {
await charge("Wise Australia Pty Ltd.", "9001", "2026-05-20", "WISE CHARGES FOR: TRANSFER-2143509822", 30.67);
await charge("Wise Australia Pty Ltd.", "9001", "2026-06-04", "WISE CHARGES FOR: TRANSFER-2171403032", 19.51);
const rows = await getUpcomingAnnualCharges(ownerId);
expect(rows.filter((r) => r.label.startsWith("WISE CHARGES"))).toHaveLength(0);
});
it("ignores a merchant visited twice a year apart for wildly different amounts", async () => {
// The exact shape that defeats a pure cadence test: Target, $30 then $198.
await purchase("2025-07-21", "TARGET", 30);
await purchase("2026-07-11", "TARGET", 198);
expect(find(await getUpcomingAnnualCharges(ownerId), "TARGET")).toBeUndefined();
});
it("ignores a merchant seen only once", async () => {
await purchase("2026-04-03", "UBER ONE MEMBERSHIP", 96);
expect(find(await getUpcomingAnnualCharges(ownerId), "UBER ONE MEMBERSHIP")).toBeUndefined();
});
});
describe("account identity and honesty about gaps", () => {
it("treats a renamed account as one continuous fee series", async () => {
// Citi Prestige -> NAB MyCard: same card, two bank names, two numbers.
await charge("Citibank", "5258073330340253", "2024-12-13", "Annual Fee", 700);
await charge("Citibank", "5258073330340253", "2025-11-13", "Annual Fee", 700);
await pool.query(
`INSERT INTO account_links (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason)
VALUES ('Citibank','5258073330340253','NAB','1015001000116227','2025-11-14','issuer_change')`
);
// A statement under the NEW identity, so the link has somewhere to resolve to.
await charge("NAB", "1015001000116227", "2026-08-13", "Purchase", 29.99, "debit");
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL FEE");
expect(row?.observations).toBe(2);
expect(row?.confidence).toBe("confirmed");
// Names the account's CURRENT bank -- the next fee comes from there, and
// sending someone to the retired brand is the bug this link exists to stop.
expect(row?.bank_name).toBe("NAB");
});
it("marks a passed estimate as expected_not_seen rather than upcoming", async () => {
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CHARGE");
expect(row?.status).toBe("expected_not_seen");
// And it reports how stale the account is, which is what separates
// "it stopped being charged" from "we stopped importing the statements".
expect(row?.account_days_stale).toBeGreaterThan(300);
});
it("sorts upcoming ahead of expected_not_seen", async () => {
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
await charge("Westpac", "8032", "2026-08-01", "CARD FEE", 295);
const rows = await getUpcomingAnnualCharges(ownerId);
const firstMissing = rows.findIndex((r) => r.status === "expected_not_seen");
const lastUpcoming = rows.map((r) => r.status).lastIndexOf("upcoming");
expect(lastUpcoming).toBeLessThan(firstMissing);
});
});
+44
View File
@@ -0,0 +1,44 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { getUpcomingAnnualCharges } from "@/lib/annual-charges";
/**
* Upcoming annual charges, from payments alone.
*
* Lives here rather than in home-dashboard so the detection sits beside the
* ledger it queries, is covered by this repo's integration suite, and can be
* reused by a future Slack nudge without moving. home-dashboard reads it over
* networks_internal the way its other widgets read their sources.
*
* Machine access uses the same shared secret as the dashboard's other reads;
* a browser session scopes to that user instead.
*/
export async function GET(req: NextRequest) {
// Its OWN token, with no fallback to WALLET_INGEST_TOKEN. home-dashboard's
// compose states the rule plainly -- "this app never holds write creds" -- and
// WALLET_INGEST_TOKEN unlocks POST /api/wallet/ingest and the estate-wide
// reconcile sweep. Handing that to a read-only surface to save minting a
// secret would quietly convert a dashboard into a writer. Unset means closed.
const token = process.env.DASH_READ_TOKEN;
const supplied = req.headers.get("x-ingest-token");
const machine = !!token && !!supplied && supplied === token;
let ownerId: number | null = null;
if (machine) {
// A machine caller names the owner explicitly rather than defaulting to 1:
// silently assuming an owner is how an ingested order ended up invisible in
// every scoped view while sitting in the table.
const q = req.nextUrl.searchParams.get("owner_id");
ownerId = q !== null && Number.isInteger(Number(q)) ? Number(q) : null;
if (ownerId === null) {
return NextResponse.json({ error: "owner_id is required for machine access" }, { status: 400 });
}
} else {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
ownerId = user.id;
}
const charges = await getUpcomingAnnualCharges(ownerId);
return NextResponse.json({ charges, count: charges.length });
}
+274
View File
@@ -0,0 +1,274 @@
import { queryRaw } from "@/lib/db";
/**
* Upcoming annual charges, derived from payments alone.
*
* Independent of the email lane on purpose: a recurring CHARGE is visible in the
* ledger whether or not anything was ever emailed about it, which is the only
* way a card annual fee or a loan package fee shows up at all -- banks announce
* those on the statement, not in a renewal notice.
*
* WHAT DOES NOT WORK, measured before this was written: "two charges about 365
* days apart" is useless on its own. Ranked that way the top hit on this ledger
* is a bakery with 202 such pairs, then a donation, Patreon, Coles -- merchants
* used often enough that some pair is inevitably a year apart. Frequency is the
* confound. Even "every gap is annual" still surfaces a Target visited twice
* ($30, then $198) and a Specsavers ($24, then $224). Two visits a year apart
* are not a fee.
*
* So there are two signals, and they are kept apart because they are not equally
* trustworthy -- a guess must never render beside a certainty as if it were one:
*
* confirmed the bank called it a fee AND it has repeated on an annual cadence
* likely the bank called it a fee, seen once -- the kind is certain, the
* cadence is inferred (an annual fee charged once is still annual)
* possible no fee wording; a merchant whose every gap is annual AND whose
* amount is stable. Catches Amazon Prime, Uber One, VicRoads rego,
* none of which carry fee wording. Also where false positives live.
*/
export type Confidence = "confirmed" | "likely" | "possible";
/**
* `expected_not_seen` is not a lesser version of `upcoming` -- it is the more
* interesting row. The estimate has passed and no charge landed, which means one
* of three things, and `account_days_stale` is what tells them apart:
* the account stopped being imported (HSBC: ledger ends 2026-01-07, while the
* mail shows statements to 2026-02-07)
* the account genuinely closed
* the charge landed under a different label (VicRoads paid 2026-03-16 as
* "VICROADS ONLINE PAYM", not "VICROADS")
* Reporting it as upcoming would be a lie; hiding it would lose the signal.
*/
export type ChargeStatus = "upcoming" | "expected_not_seen";
export type AnnualCharge = {
label: string;
status: ChargeStatus;
bank_name: string | null;
account_number: string | null;
confidence: Confidence;
observations: number;
last_charged: string;
last_amount: string;
amount_low: string;
amount_high: string;
median_gap_days: number | null;
next_due_estimate: string;
/** Ledger liveness for the account: a fee that "stopped" may just be an
* account that stopped being imported. Null for merchant-signal rows, which
* are not tied to one account. */
account_last_statement: string | null;
account_days_stale: number | null;
};
/**
* `transaction_type = 'fee'` is the universe, not a description match.
*
* The statement extractor already classifies fees structurally -- 131 rows
* across the ledger -- and every card and loan annual fee on this ledger is
* typed that way. An earlier version of this filtered `transaction_type =
* 'debit'` and returned NONE of them, which is the whole reason this comment
* exists: the fees are not debits.
*
* That universe is broader than "annual", though. It also holds ANZ's monthly
* ACCOUNT SERVICING FEE, Zip's MONTHLY ACCOUNT SERVICE FEE, CommBank's MONTHLY
* FEE, Citibank LATE PAYMENT FEE, HSBC OVERSEAS TRANSACTION FEE and every Wise
* per-transfer charge. Cadence separates them: an annual fee repeats about a
* year apart, the rest repeat monthly, on an event, or never (Wise labels each
* transfer uniquely, so they can never group to two observations at all).
*/
const FEE_TYPE = "fee";
/**
* Wording that establishes the KIND when there is only one observation.
*
* A single charge cannot demonstrate a cadence, but a bank calling something an
* annual fee has already told us what it is -- Westpac's CARD FEE $295 and AMP's
* Professional Pack Annual Fee each appear once and are certainly annual.
* Deliberately narrow: bare "fee" would readmit every per-transaction charge.
*/
const ANNUAL_WORDING =
"(annual|yearly)[[:space:]]*(card[[:space:]]*)?(fee|charge)" +
"|card[[:space:]]+fee" +
"|(package|membership|facility)[[:space:]]+fee";
/**
* Resolve every statement to a canonical account identity, following
* account_links forward. One card can appear under two bank names and two
* numbers after an issuer change (Citi Prestige -> NAB MyCard); without this the
* $700 fee charged 2024-12-13 and 2025-11-13 reads as two dead one-offs.
*
* Recursive because the table permits a chain (A->B->C); `uq_account_links_from`
* forbids a fork, and `account_links_no_self` a self-link, so the walk
* terminates. The depth guard is belt-and-braces against a cycle someone inserts
* by hand (A->B, B->A), which no constraint can catch.
*/
const RESOLVED_ACCOUNTS = `
WITH RECURSIVE chain AS (
SELECT s.id AS statement_id, s.bank_name, s.account_number,
s.bank_name AS cur_bank, s.account_number_key AS cur_key, 0 AS depth
FROM statements s
UNION ALL
SELECT c.statement_id, c.bank_name, c.account_number,
l.to_bank_name, l.to_account_key, c.depth + 1
FROM chain c
JOIN account_links l
ON l.from_bank_name = c.cur_bank AND l.from_account_key = c.cur_key
WHERE c.depth < 10
),
resolved AS (
SELECT DISTINCT ON (statement_id)
statement_id, bank_name, account_number, cur_bank AS acct_bank, cur_key AS acct_key
FROM chain ORDER BY statement_id, depth DESC
)`;
export async function getUpcomingAnnualCharges(ownerId: number): Promise<AnnualCharge[]> {
// Signal 1 -- the bank's own fee wording, grouped by resolved account identity
// and by the fee's wording so two different fees on one card stay apart
// (ANZ charges "ANNUAL FEE" $120 and "ADDITIONAL CARD FEE" $10 on the same
// statement, and they are not the same obligation).
const feeRows = await queryRaw<AnnualCharge>(
`${RESOLVED_ACCOUNTS},
fees AS (
SELECT r.acct_bank, r.acct_key, r.bank_name, r.account_number,
upper(trim(regexp_replace(t.description, '[[:space:]]+', ' ', 'g'))) AS label,
t.transaction_date, t.amount
FROM transactions t
JOIN resolved r ON r.statement_id = t.statement_id
JOIN statements s ON s.id = t.statement_id
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
AND t.amount > 0
AND COALESCE(t.owner_id, s.owner_id) = $1
-- Structural first, wording only as a widening: a bank that types a
-- fee as a plain debit still gets picked up if it names it annual.
AND (t.transaction_type = $3 OR t.description ~* $2)
),
gapped AS (
SELECT f.*,
transaction_date - lag(transaction_date)
OVER (PARTITION BY acct_key, label ORDER BY transaction_date) AS gap
FROM fees f
),
agg AS (
SELECT acct_key, acct_bank, label,
count(*)::int AS observations,
max(transaction_date) AS last_charged,
min(amount) AS amount_low, max(amount) AS amount_high,
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
FILTER (WHERE gap IS NOT NULL) AS median_gap,
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
(array_agg(bank_name ORDER BY transaction_date DESC))[1] AS bank_name,
(array_agg(account_number ORDER BY transaction_date DESC))[1] AS account_number,
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
FROM gapped GROUP BY acct_key, acct_bank, label
),
liveness AS (
SELECT r.acct_key, max(s.billing_end_date) AS last_statement
FROM statements s JOIN resolved r ON r.statement_id = s.id
GROUP BY r.acct_key
)
SELECT a.label,
-- The account's CURRENT identity, not the bank that happened to
-- print the last fee. That $700 fee last appeared under "Citibank"
-- in Nov 2025, but the account is NAB/MyCard now and the next one
-- will come from there; naming the dead brand would send you to the
-- wrong bank.
a.acct_bank AS bank_name, a.account_number,
CASE WHEN (a.last_charged + COALESCE(round(a.median_gap)::int, 365)) >= CURRENT_DATE
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
-- One observation cannot prove a cadence, but the bank naming it an
-- annual fee already tells us the kind. Repeated on an annual gap it
-- is confirmed; seen once it is likely and says so.
CASE WHEN a.gaps > 0 AND a.annual_gaps = a.gaps THEN 'confirmed'
ELSE 'likely' END AS confidence,
a.observations,
a.last_charged::text AS last_charged,
a.last_amount::text AS last_amount,
a.amount_low::text AS amount_low,
a.amount_high::text AS amount_high,
round(a.median_gap)::int AS median_gap_days,
(a.last_charged + COALESCE(round(a.median_gap)::int, 365))::text AS next_due_estimate,
l.last_statement::text AS account_last_statement,
(CURRENT_DATE - l.last_statement)::int AS account_days_stale
FROM agg a LEFT JOIN liveness l ON l.acct_key = a.acct_key
-- Two ways to earn a place, and nothing else gets one. Filtered here, not
-- in the UI, so the API can never be read wrongly by a future caller.
WHERE
-- Repeated on an annual cadence. Excludes ANZ's monthly ACCOUNT
-- SERVICING FEE and Zip's MONTHLY ACCOUNT SERVICE FEE, whose gaps are
-- ~30 days, and Citibank's LATE PAYMENT FEE, which is event-driven.
(a.gaps > 0 AND a.annual_gaps = a.gaps AND a.median_gap >= 300)
-- Or seen once, but named as an annual/card/package fee. Westpac's CARD
-- FEE and AMP's Professional Pack Annual Fee are each a single charge and
-- unmistakably annual. A Wise per-transfer fee is also a single charge
-- and says nothing of the sort, which is what keeps it out.
OR (a.gaps = 0 AND a.label ~* $2)`,
[ownerId, ANNUAL_WORDING, FEE_TYPE]
);
// Signal 2 -- merchant-charged annual items with no fee wording. Requires
// BOTH that every gap is annual AND that the amount is stable: VicRoads rego
// rises about 3% a year ($825.23 -> $852.88 -> $875.48), so equality is too
// strict, while no amount test at all readmits the twice-visited restaurant.
// Keyed on merchant, never on card -- that rego was paid on three different
// banks, and a card-keyed view would see three unrelated one-offs.
const merchantRows = await queryRaw<AnnualCharge>(
`WITH t AS (
SELECT upper(trim(COALESCE(t.merchant_normalized, t.merchant_name, t.description))) AS label,
t.transaction_date, t.amount
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
AND t.transaction_type = 'debit' AND t.amount > 0
AND COALESCE(t.owner_id, s.owner_id) = $1
AND t.transaction_type = 'debit' AND t.description !~* $2
),
gapped AS (
SELECT t.*, transaction_date - lag(transaction_date)
OVER (PARTITION BY label ORDER BY transaction_date) AS gap
FROM t
),
agg AS (
SELECT label, count(*)::int AS observations,
max(transaction_date) AS last_charged,
min(amount) AS amount_low, max(amount) AS amount_high,
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
FILTER (WHERE gap IS NOT NULL) AS median_gap,
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
FROM gapped GROUP BY label
)
SELECT label, NULL::text AS bank_name, NULL::text AS account_number,
CASE WHEN (last_charged + round(median_gap)::int) >= CURRENT_DATE
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
'possible' AS confidence, observations,
last_charged::text AS last_charged,
last_amount::text AS last_amount,
amount_low::text AS amount_low,
amount_high::text AS amount_high,
round(median_gap)::int AS median_gap_days,
(last_charged + round(median_gap)::int)::text AS next_due_estimate,
NULL::text AS account_last_statement,
NULL::int AS account_days_stale
FROM agg
WHERE gaps >= 1 AND annual_gaps = gaps
-- Amount stability is what separates a subscription from a restaurant.
-- 20% absorbs CPI drift and an FX-priced renewal without admitting a
-- $30-then-$198 pair.
AND amount_high <= amount_low * 1.20
-- Two observations is the floor for a cadence claim, and this lane has no
-- wording to fall back on the way the fee lane does.
AND observations >= 2`,
[ownerId, ANNUAL_WORDING]
);
// Upcoming first and soonest-first within that, so the thing you can still
// act on is at the top; the expected-but-missing rows follow as a review list.
const rank = (c: AnnualCharge) => (c.status === "upcoming" ? 0 : 1);
return [...feeRows, ...merchantRows].sort(
(a, b) => rank(a) - rank(b) || a.next_due_estimate.localeCompare(b.next_due_estimate)
);
}