annual charges: surface upcoming card/loan fees from payments alone
ci / lint-test (push) Successful in 56s
ci / lint-test (push) Successful in 56s
Answers "what annual payments are coming?" without touching the email lane --
a bank announces an annual fee on the statement and nowhere else.
WHAT DOES NOT WORK, measured before writing it: "two charges ~365 days apart"
ranks a BAKERY first on this ledger, with 202 such pairs, then a donation,
Patreon, Coles. Frequency is the confound. Tightening to "every gap is annual"
still admits a Target visited twice ($30, then $198).
transaction_type='fee' is the universe, not a description match. The extractor
already classifies fees structurally (131 rows) and every card and loan annual
fee here is typed that way -- the first cut of this filtered
transaction_type='debit' and returned NONE of them. The fees are not debits.
Three confidence tiers, kept apart so a guess never renders like a bank fact:
confirmed typed a fee AND repeated on an annual cadence
likely typed a fee, seen once, and NAMED annual/card/package -- one charge
cannot show a cadence but the kind is not in doubt
possible no fee wording; merchant whose every gap is annual AND whose amount
is stable (<=20%, absorbing VicRoads' ~3%/yr drift). Catches Amazon
Prime and rego; also where the false positives live, hence the label.
expected_not_seen is the more interesting status: the estimate passed and
nothing landed. account_days_stale is the discriminator -- Westpac at 0 days is
trustworthy, ANZ at 392 is simply missing data. Reporting those as upcoming
would be a lie.
account_links (migration 0038) fixes account continuity. Citi Prestige
5258...0253 became NAB MyCard 1015...6227 -- same card, same 14th-to-13th cycle,
no gap -- so the $700 fee charged 2024-12-13 and 2025-11-13 read as two dead
one-offs instead of one due again ~Nov 2026. Hand-asserted with source_documents
ids as evidence rather than inferred: a contiguous-cycle heuristic would join two
genuinely different accounts as readily as one renamed one. Display resolves to
the CURRENT bank; naming the retired brand sends you to the wrong bank.
The GET takes its own DASH_READ_TOKEN with no fallback to WALLET_INGEST_TOKEN --
that one unlocks wallet ingest and the estate-wide reconcile sweep, and
home-dashboard's compose states the rule: it never holds a write credential.
Known limit: the merchant signal fragments on label drift (VicRoads paid as
VICROADS then VICROADS ONLINE PAYM). Not fuzzy-matched, per the rejected pg_trgm
merchant grouping.
339 integration + 196 unit green, tsc clean. Board #265.
This commit is contained in:
@@ -268,6 +268,56 @@ by the `rn` window and ordered by amount closeness, so a looser band changes
|
||||
live queue at the time of the change: candidate counts went **0 → 0** on all 12
|
||||
rows, i.e. no new noise. **The auto-matcher stays exact** — see `findAutoMatches()`.
|
||||
|
||||
### Upcoming annual charges (`GET /api/annual-charges`, migration 0038)
|
||||
|
||||
Card annual fees, loan package fees and annual subscriptions, derived from
|
||||
**payments alone** — independent of the email lane, because a bank announces an
|
||||
annual fee on the statement and nowhere else.
|
||||
|
||||
**`transaction_type = 'fee'` is the universe, not a description match.** The
|
||||
extractor already classifies fees structurally (131 rows), and *every* card and
|
||||
loan annual fee on this ledger is typed that way. The first version of this
|
||||
filtered `transaction_type = 'debit'` and returned **none** of them. The fees are
|
||||
not debits.
|
||||
|
||||
**Two signals, kept apart because they are not equally trustworthy:**
|
||||
|
||||
| confidence | basis |
|
||||
|---|---|
|
||||
| `confirmed` | typed a fee **and** repeated on an annual cadence |
|
||||
| `likely` | typed a fee, seen once, and *named* annual/card/package — the kind is certain, the cadence inferred |
|
||||
| `possible` | no fee wording; a merchant whose every gap is annual **and** whose amount is stable |
|
||||
|
||||
**A naive "two charges ~365 days apart" is useless** — measured on this ledger it
|
||||
ranks a bakery first with **202** such pairs, then a donation, Patreon, Coles.
|
||||
Frequency is the confound. Even "every gap is annual" admits a Target visited
|
||||
twice ($30, then $198). The `possible` tier therefore also requires amount
|
||||
stability (≤20%, which absorbs VicRoads' ~3%/yr CPI drift) and ≥2 observations.
|
||||
It still carries real false positives (H&M, Superdry) — that is what the label is
|
||||
for, and why it must never render like a bank fact.
|
||||
|
||||
**`expected_not_seen` is the more interesting status.** The estimate passed and
|
||||
nothing landed, which means the account stopped being imported, or closed, or the
|
||||
charge arrived under a different label. `account_days_stale` is the
|
||||
discriminator: Westpac at 0 days stale is trustworthy; ANZ at 392 is simply
|
||||
missing data. Never report these as upcoming.
|
||||
|
||||
**Account continuity (`account_links`, migration 0038).** One card can appear
|
||||
under two identities: Citi Prestige `5258…0253` (to 2025-11-13) became NAB MyCard
|
||||
`1015…6227` (from 2025-11-14), same 14th-to-13th cycle, no gap. Without the link
|
||||
the $700 fee reads as two dead one-offs instead of one due again ~Nov 2026. The
|
||||
table is **hand-asserted with evidence** (`source_documents` ids), not inferred —
|
||||
a contiguous-cycle heuristic would join two genuinely different accounts as
|
||||
readily as one renamed one. The resolver walks it forward recursively;
|
||||
`uq_account_links_from` forbids a fork and `account_links_no_self` a self-link, so
|
||||
it terminates. **Display the resolved current bank**, not the one that printed the
|
||||
last fee — naming the retired brand sends you to the wrong bank.
|
||||
|
||||
Known limitation: the merchant signal fragments on label drift. VicRoads rego was
|
||||
paid 2024/2025 as `VICROADS` and 2026 as `VICROADS ONLINE PAYM`, so the series
|
||||
splits and the 2025 estimate shows as `expected_not_seen`. Not fuzzy-matched on
|
||||
purpose — see the rejected pg_trgm merchant grouping.
|
||||
|
||||
### Cash and reconciliation
|
||||
|
||||
`payment_method` in (`cash`, `credits`) excludes a transaction from
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
-- Account continuity across a rebrand, reissue or issuer change.
|
||||
--
|
||||
-- WHY THIS EXISTS: one card can appear in `statements` under two completely
|
||||
-- different identities. The live case is the Citi Prestige card, which moved to
|
||||
-- NAB and was rebranded MyCard:
|
||||
--
|
||||
-- Citibank 5258073330340253 statements ... 2025-11-13
|
||||
-- NAB 1015001000116227 statements 2025-11-14 ...
|
||||
--
|
||||
-- Same 14th-to-13th billing cycle, no gap, one continuous product. Nothing in
|
||||
-- the schema linked them, so anything grouping by (bank_name, account) sees a
|
||||
-- dead account and a new one with no history. For the annual-fee detector that
|
||||
-- is not cosmetic: the $700 fee charged 2024-12-13 and 2025-11-13 reads as
|
||||
-- "stopped" when it is due again around November 2026 under the new name.
|
||||
--
|
||||
-- WHY A TABLE AND NOT AN INFERENCE: contiguous-cycle heuristics would join two
|
||||
-- genuinely different accounts as readily as one renamed one, and a wrong join
|
||||
-- merges two real fee series. There are only a handful of these events in a
|
||||
-- lifetime, each announced by mail, so each link is asserted once with the
|
||||
-- document that proves it. `evidence` is not decoration -- it is how a future
|
||||
-- reader checks the claim instead of trusting it.
|
||||
--
|
||||
-- NOT `account_owner_mappings`: that answers "whose account is this" and keys on
|
||||
-- the raw (bank_name, account_number). This answers "is this the same account as
|
||||
-- that one", and keys on the normalised form.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS account_links (
|
||||
id bigserial PRIMARY KEY,
|
||||
-- Normalised the same way statements.account_number_key is generated
|
||||
-- (non-alphanumerics stripped), so these join without per-call cleanup.
|
||||
-- Amex-style masks keep their X's: they record which digits were redacted.
|
||||
from_bank_name text NOT NULL,
|
||||
from_account_key text NOT NULL,
|
||||
to_bank_name text NOT NULL,
|
||||
to_account_key text NOT NULL,
|
||||
-- When the successor took over. The predecessor's last statement and the
|
||||
-- successor's first are the observable boundary; this is the announced date
|
||||
-- where mail gives one.
|
||||
effective_date date,
|
||||
reason text NOT NULL
|
||||
CHECK (reason IN ('rebrand', 'reissue', 'issuer_change', 'migration')),
|
||||
-- Provenance for the claim, e.g. 'source_documents:32919'. Free text because
|
||||
-- the evidence is not always a document -- sometimes it is a statement pair.
|
||||
evidence text,
|
||||
notes text,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
|
||||
-- One successor per predecessor: a chain, never a fork. A fork would make
|
||||
-- "the current identity of this account" ambiguous, and the resolver walks
|
||||
-- this forward assuming it is not.
|
||||
CONSTRAINT uq_account_links_from UNIQUE (from_bank_name, from_account_key),
|
||||
-- A link to itself would make that walk non-terminating.
|
||||
CONSTRAINT account_links_no_self CHECK (
|
||||
from_bank_name IS DISTINCT FROM to_bank_name
|
||||
OR from_account_key IS DISTINCT FROM to_account_key
|
||||
)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE account_links IS
|
||||
'Predecessor -> successor account identity across rebrands/reissues. Hand-asserted, each row carrying the evidence for its claim.';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS account_links_to
|
||||
ON account_links (to_bank_name, to_account_key);
|
||||
|
||||
-- The one link the mail record proves today.
|
||||
-- source_documents 32919 (2025-10-01) "your Citi Prestige Card account is moving to MyCard"
|
||||
-- source_documents 32618 (2025-11-10) "MyCard is coming soon"
|
||||
-- source_documents 32539 (2025-11-24) "The new MyCard App and MyCard Online are now live"
|
||||
-- Statement boundary: Citibank ends 2025-11-13, NAB begins 2025-11-14.
|
||||
INSERT INTO account_links
|
||||
(from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason, evidence, notes)
|
||||
VALUES
|
||||
('Citibank', '5258073330340253', 'NAB', '1015001000116227', '2025-11-14', 'issuer_change',
|
||||
'source_documents:32919,32618,32539',
|
||||
'Citi Prestige -> MyCard Prestige. Billing cycle (14th-13th) continues unbroken across the boundary.')
|
||||
ON CONFLICT (from_bank_name, from_account_key) DO NOTHING;
|
||||
@@ -0,0 +1,170 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants, mockDbWithPool } from "./helpers";
|
||||
|
||||
/**
|
||||
* Upcoming annual charges from payments alone.
|
||||
*
|
||||
* The exclusions carry this file. Anything can be made to find a card annual
|
||||
* fee; the difficulty is not also reporting the monthly account-servicing fee,
|
||||
* the per-transfer Wise charge, and the restaurant visited twice a year apart.
|
||||
* Measured on the real ledger, a naive "two charges ~365 days apart" ranks a
|
||||
* bakery first with 202 such pairs.
|
||||
*/
|
||||
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let getUpcomingAnnualCharges: typeof import("../../lib/annual-charges").getUpcomingAnnualCharges;
|
||||
|
||||
let acctSeq = 0;
|
||||
|
||||
/** A statement plus one transaction on it, for a named account. */
|
||||
async function charge(
|
||||
bank: string,
|
||||
acct: string,
|
||||
date: string,
|
||||
description: string,
|
||||
amount: number,
|
||||
type = "fee"
|
||||
) {
|
||||
const key = `${bank}:${acct}`;
|
||||
let stId = stmtCache.get(`${key}:${date}`);
|
||||
if (!stId) {
|
||||
const st = await pool.query(
|
||||
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||
VALUES ($4, $1, $2, $3::date - 20, $3::date + 10, $5) RETURNING id`,
|
||||
[acct, bank, date, `s${acctSeq++}.pdf`, ownerId]
|
||||
);
|
||||
stId = st.rows[0].id as number;
|
||||
stmtCache.set(`${key}:${date}`, stId);
|
||||
}
|
||||
await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||
VALUES ($1,$2,$3,$4,$5,$6)`,
|
||||
[stId, date, description, amount, type, ownerId]
|
||||
);
|
||||
}
|
||||
let stmtCache = new Map<string, number>();
|
||||
|
||||
/** A statementless merchant purchase, for the non-fee signal. */
|
||||
async function purchase(date: string, merchant: string, amount: number) {
|
||||
await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id, merchant_normalized, row_index)
|
||||
VALUES (NULL,$1,$2,$3,'debit',$4,$2,0)`,
|
||||
[date, merchant, amount, ownerId]
|
||||
);
|
||||
}
|
||||
|
||||
const find = <T extends { label: string }>(rows: T[], label: string) =>
|
||||
rows.find((r) => r.label === label);
|
||||
|
||||
beforeAll(async () => {
|
||||
pool = createPool();
|
||||
mockDbWithPool(pool);
|
||||
({ getUpcomingAnnualCharges } = await import("../../lib/annual-charges"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM account_links");
|
||||
stmtCache = new Map();
|
||||
({ ownerId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("what it finds", () => {
|
||||
it("confirms a card annual fee that repeated a year later", async () => {
|
||||
await charge("American Express", "1000", "2025-06-10", "ANNUAL CARD FEE", 875);
|
||||
await charge("American Express", "1000", "2026-06-10", "ANNUAL CARD FEE", 1750);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CARD FEE");
|
||||
expect(row?.confidence).toBe("confirmed");
|
||||
expect(row?.observations).toBe(2);
|
||||
// The amount is NOT averaged: this fee doubled, and the next one to plan for
|
||||
// is the latest, not the mean of a rise.
|
||||
expect(Number(row?.last_amount)).toBe(1750);
|
||||
expect(row?.next_due_estimate.startsWith("2027-06")).toBe(true);
|
||||
});
|
||||
|
||||
it("accepts a fee seen only once when the bank names it annual", async () => {
|
||||
await charge("Westpac", "8032", "2025-10-21", "CARD FEE", 295);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "CARD FEE");
|
||||
// One charge cannot show a cadence, but the kind is not in doubt.
|
||||
expect(row?.confidence).toBe("likely");
|
||||
expect(row?.next_due_estimate).toBe("2026-10-21");
|
||||
});
|
||||
|
||||
it("finds an annual merchant charge with no fee wording, if the amount is stable", async () => {
|
||||
await purchase("2025-02-03", "AMAZON PRIME", 79);
|
||||
await purchase("2026-02-01", "AMAZON PRIME", 79);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "AMAZON PRIME");
|
||||
expect(row?.confidence).toBe("possible");
|
||||
});
|
||||
});
|
||||
|
||||
describe("what it must NOT report", () => {
|
||||
it("ignores a monthly account fee", async () => {
|
||||
for (const d of ["2026-01-31", "2026-02-28", "2026-03-31", "2026-04-30"]) {
|
||||
await charge("ANZ", "5626", d, "ACCOUNT SERVICING FEE", 5);
|
||||
}
|
||||
expect(find(await getUpcomingAnnualCharges(ownerId), "ACCOUNT SERVICING FEE")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores per-transaction fees, which never repeat under one label", async () => {
|
||||
await charge("Wise Australia Pty Ltd.", "9001", "2026-05-20", "WISE CHARGES FOR: TRANSFER-2143509822", 30.67);
|
||||
await charge("Wise Australia Pty Ltd.", "9001", "2026-06-04", "WISE CHARGES FOR: TRANSFER-2171403032", 19.51);
|
||||
const rows = await getUpcomingAnnualCharges(ownerId);
|
||||
expect(rows.filter((r) => r.label.startsWith("WISE CHARGES"))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("ignores a merchant visited twice a year apart for wildly different amounts", async () => {
|
||||
// The exact shape that defeats a pure cadence test: Target, $30 then $198.
|
||||
await purchase("2025-07-21", "TARGET", 30);
|
||||
await purchase("2026-07-11", "TARGET", 198);
|
||||
expect(find(await getUpcomingAnnualCharges(ownerId), "TARGET")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores a merchant seen only once", async () => {
|
||||
await purchase("2026-04-03", "UBER ONE MEMBERSHIP", 96);
|
||||
expect(find(await getUpcomingAnnualCharges(ownerId), "UBER ONE MEMBERSHIP")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("account identity and honesty about gaps", () => {
|
||||
it("treats a renamed account as one continuous fee series", async () => {
|
||||
// Citi Prestige -> NAB MyCard: same card, two bank names, two numbers.
|
||||
await charge("Citibank", "5258073330340253", "2024-12-13", "Annual Fee", 700);
|
||||
await charge("Citibank", "5258073330340253", "2025-11-13", "Annual Fee", 700);
|
||||
await pool.query(
|
||||
`INSERT INTO account_links (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason)
|
||||
VALUES ('Citibank','5258073330340253','NAB','1015001000116227','2025-11-14','issuer_change')`
|
||||
);
|
||||
// A statement under the NEW identity, so the link has somewhere to resolve to.
|
||||
await charge("NAB", "1015001000116227", "2026-08-13", "Purchase", 29.99, "debit");
|
||||
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL FEE");
|
||||
expect(row?.observations).toBe(2);
|
||||
expect(row?.confidence).toBe("confirmed");
|
||||
// Names the account's CURRENT bank -- the next fee comes from there, and
|
||||
// sending someone to the retired brand is the bug this link exists to stop.
|
||||
expect(row?.bank_name).toBe("NAB");
|
||||
});
|
||||
|
||||
it("marks a passed estimate as expected_not_seen rather than upcoming", async () => {
|
||||
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
|
||||
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CHARGE");
|
||||
expect(row?.status).toBe("expected_not_seen");
|
||||
// And it reports how stale the account is, which is what separates
|
||||
// "it stopped being charged" from "we stopped importing the statements".
|
||||
expect(row?.account_days_stale).toBeGreaterThan(300);
|
||||
});
|
||||
|
||||
it("sorts upcoming ahead of expected_not_seen", async () => {
|
||||
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
|
||||
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
|
||||
await charge("Westpac", "8032", "2026-08-01", "CARD FEE", 295);
|
||||
const rows = await getUpcomingAnnualCharges(ownerId);
|
||||
const firstMissing = rows.findIndex((r) => r.status === "expected_not_seen");
|
||||
const lastUpcoming = rows.map((r) => r.status).lastIndexOf("upcoming");
|
||||
expect(lastUpcoming).toBeLessThan(firstMissing);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,44 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { getCurrentUser } from "@/lib/auth";
|
||||
import { getUpcomingAnnualCharges } from "@/lib/annual-charges";
|
||||
|
||||
/**
|
||||
* Upcoming annual charges, from payments alone.
|
||||
*
|
||||
* Lives here rather than in home-dashboard so the detection sits beside the
|
||||
* ledger it queries, is covered by this repo's integration suite, and can be
|
||||
* reused by a future Slack nudge without moving. home-dashboard reads it over
|
||||
* networks_internal the way its other widgets read their sources.
|
||||
*
|
||||
* Machine access uses the same shared secret as the dashboard's other reads;
|
||||
* a browser session scopes to that user instead.
|
||||
*/
|
||||
export async function GET(req: NextRequest) {
|
||||
// Its OWN token, with no fallback to WALLET_INGEST_TOKEN. home-dashboard's
|
||||
// compose states the rule plainly -- "this app never holds write creds" -- and
|
||||
// WALLET_INGEST_TOKEN unlocks POST /api/wallet/ingest and the estate-wide
|
||||
// reconcile sweep. Handing that to a read-only surface to save minting a
|
||||
// secret would quietly convert a dashboard into a writer. Unset means closed.
|
||||
const token = process.env.DASH_READ_TOKEN;
|
||||
const supplied = req.headers.get("x-ingest-token");
|
||||
const machine = !!token && !!supplied && supplied === token;
|
||||
|
||||
let ownerId: number | null = null;
|
||||
if (machine) {
|
||||
// A machine caller names the owner explicitly rather than defaulting to 1:
|
||||
// silently assuming an owner is how an ingested order ended up invisible in
|
||||
// every scoped view while sitting in the table.
|
||||
const q = req.nextUrl.searchParams.get("owner_id");
|
||||
ownerId = q !== null && Number.isInteger(Number(q)) ? Number(q) : null;
|
||||
if (ownerId === null) {
|
||||
return NextResponse.json({ error: "owner_id is required for machine access" }, { status: 400 });
|
||||
}
|
||||
} else {
|
||||
const user = await getCurrentUser(req);
|
||||
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||
ownerId = user.id;
|
||||
}
|
||||
|
||||
const charges = await getUpcomingAnnualCharges(ownerId);
|
||||
return NextResponse.json({ charges, count: charges.length });
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
import { queryRaw } from "@/lib/db";
|
||||
|
||||
/**
|
||||
* Upcoming annual charges, derived from payments alone.
|
||||
*
|
||||
* Independent of the email lane on purpose: a recurring CHARGE is visible in the
|
||||
* ledger whether or not anything was ever emailed about it, which is the only
|
||||
* way a card annual fee or a loan package fee shows up at all -- banks announce
|
||||
* those on the statement, not in a renewal notice.
|
||||
*
|
||||
* WHAT DOES NOT WORK, measured before this was written: "two charges about 365
|
||||
* days apart" is useless on its own. Ranked that way the top hit on this ledger
|
||||
* is a bakery with 202 such pairs, then a donation, Patreon, Coles -- merchants
|
||||
* used often enough that some pair is inevitably a year apart. Frequency is the
|
||||
* confound. Even "every gap is annual" still surfaces a Target visited twice
|
||||
* ($30, then $198) and a Specsavers ($24, then $224). Two visits a year apart
|
||||
* are not a fee.
|
||||
*
|
||||
* So there are two signals, and they are kept apart because they are not equally
|
||||
* trustworthy -- a guess must never render beside a certainty as if it were one:
|
||||
*
|
||||
* confirmed the bank called it a fee AND it has repeated on an annual cadence
|
||||
* likely the bank called it a fee, seen once -- the kind is certain, the
|
||||
* cadence is inferred (an annual fee charged once is still annual)
|
||||
* possible no fee wording; a merchant whose every gap is annual AND whose
|
||||
* amount is stable. Catches Amazon Prime, Uber One, VicRoads rego,
|
||||
* none of which carry fee wording. Also where false positives live.
|
||||
*/
|
||||
|
||||
export type Confidence = "confirmed" | "likely" | "possible";
|
||||
|
||||
/**
|
||||
* `expected_not_seen` is not a lesser version of `upcoming` -- it is the more
|
||||
* interesting row. The estimate has passed and no charge landed, which means one
|
||||
* of three things, and `account_days_stale` is what tells them apart:
|
||||
* the account stopped being imported (HSBC: ledger ends 2026-01-07, while the
|
||||
* mail shows statements to 2026-02-07)
|
||||
* the account genuinely closed
|
||||
* the charge landed under a different label (VicRoads paid 2026-03-16 as
|
||||
* "VICROADS ONLINE PAYM", not "VICROADS")
|
||||
* Reporting it as upcoming would be a lie; hiding it would lose the signal.
|
||||
*/
|
||||
export type ChargeStatus = "upcoming" | "expected_not_seen";
|
||||
|
||||
export type AnnualCharge = {
|
||||
label: string;
|
||||
status: ChargeStatus;
|
||||
bank_name: string | null;
|
||||
account_number: string | null;
|
||||
confidence: Confidence;
|
||||
observations: number;
|
||||
last_charged: string;
|
||||
last_amount: string;
|
||||
amount_low: string;
|
||||
amount_high: string;
|
||||
median_gap_days: number | null;
|
||||
next_due_estimate: string;
|
||||
/** Ledger liveness for the account: a fee that "stopped" may just be an
|
||||
* account that stopped being imported. Null for merchant-signal rows, which
|
||||
* are not tied to one account. */
|
||||
account_last_statement: string | null;
|
||||
account_days_stale: number | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* `transaction_type = 'fee'` is the universe, not a description match.
|
||||
*
|
||||
* The statement extractor already classifies fees structurally -- 131 rows
|
||||
* across the ledger -- and every card and loan annual fee on this ledger is
|
||||
* typed that way. An earlier version of this filtered `transaction_type =
|
||||
* 'debit'` and returned NONE of them, which is the whole reason this comment
|
||||
* exists: the fees are not debits.
|
||||
*
|
||||
* That universe is broader than "annual", though. It also holds ANZ's monthly
|
||||
* ACCOUNT SERVICING FEE, Zip's MONTHLY ACCOUNT SERVICE FEE, CommBank's MONTHLY
|
||||
* FEE, Citibank LATE PAYMENT FEE, HSBC OVERSEAS TRANSACTION FEE and every Wise
|
||||
* per-transfer charge. Cadence separates them: an annual fee repeats about a
|
||||
* year apart, the rest repeat monthly, on an event, or never (Wise labels each
|
||||
* transfer uniquely, so they can never group to two observations at all).
|
||||
*/
|
||||
const FEE_TYPE = "fee";
|
||||
|
||||
/**
|
||||
* Wording that establishes the KIND when there is only one observation.
|
||||
*
|
||||
* A single charge cannot demonstrate a cadence, but a bank calling something an
|
||||
* annual fee has already told us what it is -- Westpac's CARD FEE $295 and AMP's
|
||||
* Professional Pack Annual Fee each appear once and are certainly annual.
|
||||
* Deliberately narrow: bare "fee" would readmit every per-transaction charge.
|
||||
*/
|
||||
const ANNUAL_WORDING =
|
||||
"(annual|yearly)[[:space:]]*(card[[:space:]]*)?(fee|charge)" +
|
||||
"|card[[:space:]]+fee" +
|
||||
"|(package|membership|facility)[[:space:]]+fee";
|
||||
|
||||
/**
|
||||
* Resolve every statement to a canonical account identity, following
|
||||
* account_links forward. One card can appear under two bank names and two
|
||||
* numbers after an issuer change (Citi Prestige -> NAB MyCard); without this the
|
||||
* $700 fee charged 2024-12-13 and 2025-11-13 reads as two dead one-offs.
|
||||
*
|
||||
* Recursive because the table permits a chain (A->B->C); `uq_account_links_from`
|
||||
* forbids a fork, and `account_links_no_self` a self-link, so the walk
|
||||
* terminates. The depth guard is belt-and-braces against a cycle someone inserts
|
||||
* by hand (A->B, B->A), which no constraint can catch.
|
||||
*/
|
||||
const RESOLVED_ACCOUNTS = `
|
||||
WITH RECURSIVE chain AS (
|
||||
SELECT s.id AS statement_id, s.bank_name, s.account_number,
|
||||
s.bank_name AS cur_bank, s.account_number_key AS cur_key, 0 AS depth
|
||||
FROM statements s
|
||||
UNION ALL
|
||||
SELECT c.statement_id, c.bank_name, c.account_number,
|
||||
l.to_bank_name, l.to_account_key, c.depth + 1
|
||||
FROM chain c
|
||||
JOIN account_links l
|
||||
ON l.from_bank_name = c.cur_bank AND l.from_account_key = c.cur_key
|
||||
WHERE c.depth < 10
|
||||
),
|
||||
resolved AS (
|
||||
SELECT DISTINCT ON (statement_id)
|
||||
statement_id, bank_name, account_number, cur_bank AS acct_bank, cur_key AS acct_key
|
||||
FROM chain ORDER BY statement_id, depth DESC
|
||||
)`;
|
||||
|
||||
export async function getUpcomingAnnualCharges(ownerId: number): Promise<AnnualCharge[]> {
|
||||
// Signal 1 -- the bank's own fee wording, grouped by resolved account identity
|
||||
// and by the fee's wording so two different fees on one card stay apart
|
||||
// (ANZ charges "ANNUAL FEE" $120 and "ADDITIONAL CARD FEE" $10 on the same
|
||||
// statement, and they are not the same obligation).
|
||||
const feeRows = await queryRaw<AnnualCharge>(
|
||||
`${RESOLVED_ACCOUNTS},
|
||||
fees AS (
|
||||
SELECT r.acct_bank, r.acct_key, r.bank_name, r.account_number,
|
||||
upper(trim(regexp_replace(t.description, '[[:space:]]+', ' ', 'g'))) AS label,
|
||||
t.transaction_date, t.amount
|
||||
FROM transactions t
|
||||
JOIN resolved r ON r.statement_id = t.statement_id
|
||||
JOIN statements s ON s.id = t.statement_id
|
||||
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
|
||||
AND t.amount > 0
|
||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||
-- Structural first, wording only as a widening: a bank that types a
|
||||
-- fee as a plain debit still gets picked up if it names it annual.
|
||||
AND (t.transaction_type = $3 OR t.description ~* $2)
|
||||
),
|
||||
gapped AS (
|
||||
SELECT f.*,
|
||||
transaction_date - lag(transaction_date)
|
||||
OVER (PARTITION BY acct_key, label ORDER BY transaction_date) AS gap
|
||||
FROM fees f
|
||||
),
|
||||
agg AS (
|
||||
SELECT acct_key, acct_bank, label,
|
||||
count(*)::int AS observations,
|
||||
max(transaction_date) AS last_charged,
|
||||
min(amount) AS amount_low, max(amount) AS amount_high,
|
||||
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
|
||||
FILTER (WHERE gap IS NOT NULL) AS median_gap,
|
||||
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
|
||||
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
|
||||
(array_agg(bank_name ORDER BY transaction_date DESC))[1] AS bank_name,
|
||||
(array_agg(account_number ORDER BY transaction_date DESC))[1] AS account_number,
|
||||
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
|
||||
FROM gapped GROUP BY acct_key, acct_bank, label
|
||||
),
|
||||
liveness AS (
|
||||
SELECT r.acct_key, max(s.billing_end_date) AS last_statement
|
||||
FROM statements s JOIN resolved r ON r.statement_id = s.id
|
||||
GROUP BY r.acct_key
|
||||
)
|
||||
SELECT a.label,
|
||||
-- The account's CURRENT identity, not the bank that happened to
|
||||
-- print the last fee. That $700 fee last appeared under "Citibank"
|
||||
-- in Nov 2025, but the account is NAB/MyCard now and the next one
|
||||
-- will come from there; naming the dead brand would send you to the
|
||||
-- wrong bank.
|
||||
a.acct_bank AS bank_name, a.account_number,
|
||||
CASE WHEN (a.last_charged + COALESCE(round(a.median_gap)::int, 365)) >= CURRENT_DATE
|
||||
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
|
||||
-- One observation cannot prove a cadence, but the bank naming it an
|
||||
-- annual fee already tells us the kind. Repeated on an annual gap it
|
||||
-- is confirmed; seen once it is likely and says so.
|
||||
CASE WHEN a.gaps > 0 AND a.annual_gaps = a.gaps THEN 'confirmed'
|
||||
ELSE 'likely' END AS confidence,
|
||||
a.observations,
|
||||
a.last_charged::text AS last_charged,
|
||||
a.last_amount::text AS last_amount,
|
||||
a.amount_low::text AS amount_low,
|
||||
a.amount_high::text AS amount_high,
|
||||
round(a.median_gap)::int AS median_gap_days,
|
||||
(a.last_charged + COALESCE(round(a.median_gap)::int, 365))::text AS next_due_estimate,
|
||||
l.last_statement::text AS account_last_statement,
|
||||
(CURRENT_DATE - l.last_statement)::int AS account_days_stale
|
||||
FROM agg a LEFT JOIN liveness l ON l.acct_key = a.acct_key
|
||||
-- Two ways to earn a place, and nothing else gets one. Filtered here, not
|
||||
-- in the UI, so the API can never be read wrongly by a future caller.
|
||||
WHERE
|
||||
-- Repeated on an annual cadence. Excludes ANZ's monthly ACCOUNT
|
||||
-- SERVICING FEE and Zip's MONTHLY ACCOUNT SERVICE FEE, whose gaps are
|
||||
-- ~30 days, and Citibank's LATE PAYMENT FEE, which is event-driven.
|
||||
(a.gaps > 0 AND a.annual_gaps = a.gaps AND a.median_gap >= 300)
|
||||
-- Or seen once, but named as an annual/card/package fee. Westpac's CARD
|
||||
-- FEE and AMP's Professional Pack Annual Fee are each a single charge and
|
||||
-- unmistakably annual. A Wise per-transfer fee is also a single charge
|
||||
-- and says nothing of the sort, which is what keeps it out.
|
||||
OR (a.gaps = 0 AND a.label ~* $2)`,
|
||||
[ownerId, ANNUAL_WORDING, FEE_TYPE]
|
||||
);
|
||||
|
||||
// Signal 2 -- merchant-charged annual items with no fee wording. Requires
|
||||
// BOTH that every gap is annual AND that the amount is stable: VicRoads rego
|
||||
// rises about 3% a year ($825.23 -> $852.88 -> $875.48), so equality is too
|
||||
// strict, while no amount test at all readmits the twice-visited restaurant.
|
||||
// Keyed on merchant, never on card -- that rego was paid on three different
|
||||
// banks, and a card-keyed view would see three unrelated one-offs.
|
||||
const merchantRows = await queryRaw<AnnualCharge>(
|
||||
`WITH t AS (
|
||||
SELECT upper(trim(COALESCE(t.merchant_normalized, t.merchant_name, t.description))) AS label,
|
||||
t.transaction_date, t.amount
|
||||
FROM transactions t
|
||||
LEFT JOIN statements s ON s.id = t.statement_id
|
||||
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
|
||||
AND t.transaction_type = 'debit' AND t.amount > 0
|
||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||
AND t.transaction_type = 'debit' AND t.description !~* $2
|
||||
),
|
||||
gapped AS (
|
||||
SELECT t.*, transaction_date - lag(transaction_date)
|
||||
OVER (PARTITION BY label ORDER BY transaction_date) AS gap
|
||||
FROM t
|
||||
),
|
||||
agg AS (
|
||||
SELECT label, count(*)::int AS observations,
|
||||
max(transaction_date) AS last_charged,
|
||||
min(amount) AS amount_low, max(amount) AS amount_high,
|
||||
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
|
||||
FILTER (WHERE gap IS NOT NULL) AS median_gap,
|
||||
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
|
||||
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
|
||||
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
|
||||
FROM gapped GROUP BY label
|
||||
)
|
||||
SELECT label, NULL::text AS bank_name, NULL::text AS account_number,
|
||||
CASE WHEN (last_charged + round(median_gap)::int) >= CURRENT_DATE
|
||||
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
|
||||
'possible' AS confidence, observations,
|
||||
last_charged::text AS last_charged,
|
||||
last_amount::text AS last_amount,
|
||||
amount_low::text AS amount_low,
|
||||
amount_high::text AS amount_high,
|
||||
round(median_gap)::int AS median_gap_days,
|
||||
(last_charged + round(median_gap)::int)::text AS next_due_estimate,
|
||||
NULL::text AS account_last_statement,
|
||||
NULL::int AS account_days_stale
|
||||
FROM agg
|
||||
WHERE gaps >= 1 AND annual_gaps = gaps
|
||||
-- Amount stability is what separates a subscription from a restaurant.
|
||||
-- 20% absorbs CPI drift and an FX-priced renewal without admitting a
|
||||
-- $30-then-$198 pair.
|
||||
AND amount_high <= amount_low * 1.20
|
||||
-- Two observations is the floor for a cadence claim, and this lane has no
|
||||
-- wording to fall back on the way the fee lane does.
|
||||
AND observations >= 2`,
|
||||
[ownerId, ANNUAL_WORDING]
|
||||
);
|
||||
|
||||
// Upcoming first and soonest-first within that, so the thing you can still
|
||||
// act on is at the top; the expected-but-missing rows follow as a review list.
|
||||
const rank = (c: AnnualCharge) => (c.status === "upcoming" ? 0 : 1);
|
||||
return [...feeRows, ...merchantRows].sort(
|
||||
(a, b) => rank(a) - rank(b) || a.next_due_estimate.localeCompare(b.next_due_estimate)
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user