From a4ab543a6c832d43f759f4438bd7bbcabf3bd63f Mon Sep 17 00:00:00 2001 From: siddharthd Date: Mon, 27 Jul 2026 17:10:27 +1000 Subject: [PATCH] fix(analytics): make the displayed numbers mean what they say MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six metric-integrity defects from the UI/IA review, plus two found while verifying the review's own claims against the code. The reconciled-row exclusion existed only in queries.ts. Every analytics route counted the superseded manual rows as spend — 48 rows, $4,474.79 of double count, invisible precisely because the transaction list looked right. It is now one fragment both sides import. The spend-pace chart computed its own totals in the browser: gross amounts, debits only, no personal share, no refunds, fees, interest or itemised loan repayments. On live data it ended July at $4,747.31 under a headline reading $3,597.10 — and its own baseline line was drawn from the split-adjusted monthly totals, so the two series in one chart disagreed with each other. Both now come from /api/analytics/daily, built from the same fragments as the headline. Fees aggregated every statement ever imported with no date filter, under a heading with no period, so a lifetime figure read as a current one and grew forever. Now bounded, labelled, and selectable. Comparisons no longer measure a month in progress against complete ones: the in-progress month is out of every baseline, and a selected current month is compared through the same day. Two the review did not catch: - Every analytics window was a day early. toISOString() on a local-midnight Date converts backwards through UTC. Surfaced only once fees started reporting the range it had used. - /monthly rounded per category, /daily per category-day, so the pace chart ended a few cents off the headline above it. Shared currency needed amending rather than applying. Reading s.currency would have labelled every order row AUD, since an order receipt has no statement and carries its own currency — the opposite convention from a foreign charge on an AUD statement, where amount IS AUD. NATIVE_CURRENCY's COALESCE order keeps the two apart. Balances also now count rows whose AUD value is genuinely unknown instead of netting a foreign figure against AUD ones. Latent today: no foreign transaction is currently split. Tag-filtered balance cards no longer claim "owes you". With a filter on, payments are deliberately not subtracted, so the figure is a split total and settling against it would record a payment for a debt that never was. Split-coverage warnings deliberately omitted (user decision). --- docs/ui-information-architecture-review.md | 56 +++++++- .../integration/analytics-sql.test.ts | 126 +++++++++++++++++ src/app/api/analytics/daily/route.ts | 87 ++++++++++++ src/app/api/analytics/fees/route.ts | 42 +++++- .../analytics/merchants/[merchant]/route.ts | 6 +- src/app/api/analytics/merchants/route.ts | 6 +- src/app/api/analytics/monthly/route.ts | 13 +- src/app/api/analytics/subscriptions/route.ts | 3 +- src/app/budget/page.tsx | 133 ++++++++++++------ src/app/insights/page.tsx | 53 ++++++- src/app/shared/page.tsx | 59 ++++++-- src/lib/analytics-sql.ts | 69 ++++++++- src/lib/hooks.ts | 40 +++++- src/lib/queries.ts | 40 ++++-- 14 files changed, 643 insertions(+), 90 deletions(-) create mode 100644 src/__tests__/integration/analytics-sql.test.ts create mode 100644 src/app/api/analytics/daily/route.ts diff --git a/docs/ui-information-architecture-review.md b/docs/ui-information-architecture-review.md index ca94b95..f396fa5 100644 --- a/docs/ui-information-architecture-review.md +++ b/docs/ui-information-architecture-review.md @@ -1,7 +1,61 @@ # UI and information architecture review **Date:** 2026-07-26 -**Status:** Review and redesign proposal — nothing implemented +**Status:** Priority 0 implemented 2026-07-27 (see below). Priorities 1–4 remain +proposals. + +## Implementation status — Priority 0 (2026-07-27) + +All six Priority 0 items landed, with three amendments found while verifying the +proposals against the code: + +1. **Reconciled source rows** — the exclusion was missing from *all five* + analytics routes, not only `/monthly`. It is now one fragment + (`EXCLUDE_RECONCILED_SOURCE`) that `queries.ts` also imports, so the two + halves cannot drift apart again. Real effect: 48 rows, **$4,474.79** of + double-counted spend removed from every category total, mover, Pareto and + merchant ranking. +2. **Spend pace** — now served by `/api/analytics/daily`, built from the same + fragments as the headline. Measured on live data, the old client-side series + ended July at **$4,747.31** against a headline of **$3,597.10** — a 32% + overstatement of the number directly above it. +3. **Fees and interest** — bounded by an explicit period (default 12 months, + `months=0` for all time), with the range shown and selectable. The unbounded + figure was overstating the last 12 months by roughly **$2,700 of fees**. +4. **Split-coverage warning** — *deliberately not implemented* (user decision, + 2026-07-27). +5. **Shared foreign currency** — amended. The obvious fix, reading `s.currency`, + would have mislabelled every order row as AUD, because an order receipt has + no statement and carries its own currency. Sourcing is now + `NATIVE_CURRENCY = COALESCE(s.currency, t.foreign_currency_code, 'AUD')`, + whose COALESCE order keeps two opposite denomination conventions apart. Note + this change is **latent on today's data**: no foreign transaction is + currently split, so nothing on Shared looks different yet. +6. **Partial-month comparisons** — the hero average, the top movers and the pace + baseline now exclude the in-progress month, and compare through the same day + of the month when the selected month is the current one. + +Also fixed while in here, both found by checking rather than by proposal: + +- **Every analytics window was a day early.** `toISOString()` on a local-midnight + `Date` converts backwards through UTC in any timezone east of Greenwich. Now + `toDateStr()`. This was pre-existing in `/monthly` and `/merchants`. +- **Rounding grain.** `/monthly` rounded per category and `/daily` per + category-day, so the pace chart ended the month a few cents off its own + headline. Both now carry 4dp and round once, at display. + +Guarded by `src/__tests__/integration/analytics-sql.test.ts`. + +The doc's characterisation of `REGULAR_CATEGORIES` (Insights section) is also +slightly off: the set has 13 members including rent, utilities, insurance and +subscriptions, not the 8 listed. The case for replacing it stands — a flat +binary cannot express obligation — but that is the reason, not arbitrary +membership. Note too that the proposed Fixed/Essential/Lifestyle model needs a +commitment dimension that does not exist yet: `fees` cannot be split into +avoidable versus known-annual, `subscriptions` cannot be split into contractual +versus cancellable, and the contracted loan repayment is not in the spend stream +at all (`SPEND_BASE` keeps only the interest portion). That is a data-model +change, not an Insights rework. ## Executive summary diff --git a/src/__tests__/integration/analytics-sql.test.ts b/src/__tests__/integration/analytics-sql.test.ts new file mode 100644 index 0000000..97f078f --- /dev/null +++ b/src/__tests__/integration/analytics-sql.test.ts @@ -0,0 +1,126 @@ +import { describe, it, expect } from "vitest"; +import { queryRaw, queryRow } from "../../lib/db"; +import { + EXCLUDE_RECONCILED_SOURCE, + NATIVE_CURRENCY, + AMOUNT_UNCONVERTED, +} from "../../lib/analytics-sql"; + +/** + * These fragments are the ones that drifted. + * + * The reconciled-row exclusion lived only in `queries.ts` for months while every + * analytics route counted the superseded manual rows as spend — 48 rows, $4,474 + * of double count, invisible because the transaction list (which did exclude + * them) looked right. The currency expression has the same shape of risk: it is + * read by two call sites with two different denomination conventions. + * + * Assertions are per-row against known fixtures rather than against aggregate + * totals, so a change in unrelated data cannot mask a regression. + */ + +async function scratchTxn(cols: string, vals: string, params: unknown[] = []) { + const row = await queryRow<{ id: number }>( + `INSERT INTO transactions (${cols}) VALUES (${vals}) RETURNING id`, + params + ); + return row!.id; +} + +/** Does this row survive the predicate? */ +async function passes(predicate: string, id: number): Promise { + const rows = await queryRaw( + `SELECT t.id FROM transactions t + LEFT JOIN statements s ON s.id = t.statement_id + WHERE t.id = $1 AND (${predicate})`, + [id] + ); + return rows.length === 1; +} + +describe("EXCLUDE_RECONCILED_SOURCE", () => { + it("drops a manual row that a statement line has superseded", async () => { + const survivor = await scratchTxn( + "transaction_date, description, amount, transaction_type", + "'2026-03-01','Analytics fixture — survivor', 10.00, 'debit'" + ); + const superseded = await scratchTxn( + "transaction_date, description, amount, transaction_type, reconciled_with_id", + "'2026-03-01','Analytics fixture — superseded', 10.00, 'debit', $1", + [survivor] + ); + + expect(await passes(EXCLUDE_RECONCILED_SOURCE, superseded)).toBe(false); + }); + + it("keeps an ordinary manual row that was never reconciled", async () => { + const id = await scratchTxn( + "transaction_date, description, amount, transaction_type", + "'2026-03-01','Analytics fixture — unreconciled', 10.00, 'debit'" + ); + expect(await passes(EXCLUDE_RECONCILED_SOURCE, id)).toBe(true); + }); + + it("keeps a credits order row — nothing ever sets reconciled_with_id on one", async () => { + // The order slice records the card match in expense_metadata, not on the + // transaction, and needsCardMatch() holds these out of the reconcile queue. + // If that ever changes, this exclusion would start eating real spend. + const id = await scratchTxn( + "transaction_date, description, amount, transaction_type, payment_method", + "'2026-03-01','Order - Analytics fixture', 25.00, 'debit', 'credits'" + ); + expect(await passes(EXCLUDE_RECONCILED_SOURCE, id)).toBe(true); + }); +}); + +describe("NATIVE_CURRENCY", () => { + async function currencyOf(id: number) { + const row = await queryRow<{ ccy: string; unconverted: boolean }>( + `SELECT ${NATIVE_CURRENCY} AS ccy, ${AMOUNT_UNCONVERTED} AS unconverted + FROM transactions t LEFT JOIN statements s ON s.id = t.statement_id + WHERE t.id = $1`, + [id] + ); + return row!; + } + + it("reads a statement-less order row's own currency, not 'AUD'", async () => { + // The bug this guards: sourcing currency from s.currency alone labelled + // every foreign order row AUD, because an order has no statement. + const id = await scratchTxn( + "transaction_date, description, amount, transaction_type, payment_method, foreign_currency_amount, foreign_currency_code", + "'2026-03-01','Order - Foreign fixture', 3500.00, 'debit', 'credits', 3500.00, 'LKR'" + ); + const { ccy, unconverted } = await currencyOf(id); + expect(ccy).toBe("LKR"); + // No amount_aud: the ingest path refuses to assert an FX rate it lacks, so + // this row's AUD value is genuinely unknown and must be reported as such. + expect(unconverted).toBe(true); + }); + + it("prefers the statement's currency over the foreign-charge record", async () => { + // Opposite convention: on an AUD statement, `amount` is AUD and + // foreign_currency_code merely notes what was originally charged. Reading + // the foreign code here would mislabel an AUD row as USD. + const stmt = await queryRow<{ id: number }>( + `INSERT INTO statements (bank_name, account_number, billing_end_date, currency, filename) + VALUES ('Analytics Fixture Bank','0000','2026-03-31','AUD','analytics-fixture.pdf') RETURNING id` + ); + const id = await scratchTxn( + "transaction_date, description, amount, amount_aud, transaction_type, statement_id, foreign_currency_amount, foreign_currency_code", + "'2026-03-01','Overseas purchase fixture', 45.00, 45.00, 'debit', $1, 30.00, 'USD'", + [stmt!.id] + ); + const { ccy, unconverted } = await currencyOf(id); + expect(ccy).toBe("AUD"); + expect(unconverted).toBe(false); + }); + + it("defaults a plain manual row to AUD", async () => { + const id = await scratchTxn( + "transaction_date, description, amount, transaction_type", + "'2026-03-01','Plain manual fixture', 12.00, 'debit'" + ); + expect((await currencyOf(id)).ccy).toBe("AUD"); + }); +}); diff --git a/src/app/api/analytics/daily/route.ts b/src/app/api/analytics/daily/route.ts new file mode 100644 index 0000000..4cb2225 --- /dev/null +++ b/src/app/api/analytics/daily/route.ts @@ -0,0 +1,87 @@ +import { NextRequest, NextResponse } from "next/server"; +import { getCurrentUser } from "@/lib/auth"; +import { queryRaw } from "@/lib/db"; +import { + OWNER_SCOPE, + STATEMENTS_JOIN, + EXCLUDE_NON_SPEND, + EXCLUDE_RECONCILED_SOURCE, + EFFECTIVE_CATEGORY, + NET_SPEND_ROWS, + SPEND_SIGNED, + mySplitOf, + toDateStr, +} from "@/lib/analytics-sql"; + +/** + * Daily net spend, by month, day-of-month and category. + * + * This exists so the spend-pace chart stops computing its own totals. It used to + * sum gross `amount_aud ?? amount` over `transaction_type = 'debit'` in the + * browser, which meant it ignored personal share, refunds, fees, interest, and + * itemised loan repayments — every rule the headline applies. The two numbers + * could disagree while both were labelled "spend", and the chart's own baseline + * line was drawn from the split-adjusted monthly totals, so the two series in + * one chart were on different bases. + * + * Day-of-month granularity is also what lets the page compare a partial current + * month against prior months *through the same day*, instead of against their + * full-month totals — which always made a month in progress look thrifty. + * + * Same fragments as /api/analytics/monthly. If that route's semantics change, + * this one changes with it. + */ +export async function GET(req: NextRequest) { + const user = await getCurrentUser(req); + if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 }); + + const { searchParams } = new URL(req.url); + const monthCount = Math.min(Math.max(Number(searchParams.get("months") || "12"), 1), 24); + + const now = new Date(); + const endDate = new Date(now.getFullYear(), now.getMonth() + 1, 1); + const startDate = new Date(now.getFullYear(), now.getMonth() - monthCount + 1, 1); + + const rows = await queryRaw<{ month: string; day: number; category: string; spent: string }>( + `SELECT + TO_CHAR(DATE_TRUNC('month', t.transaction_date::date), 'YYYY-MM') as month, + EXTRACT(DAY FROM t.transaction_date::date)::int as day, + ${EFFECTIVE_CATEGORY} as category, + -- 4dp, not 2. This is grouped finer than /monthly (by day as well as + -- category), so rounding each bucket to cents and summing accumulates a + -- different error than rounding per category does — the pace chart ended + -- the month a few cents off the headline it sits under. Round once, at + -- display time. + SUM(${mySplitOf(SPEND_SIGNED)})::numeric(14,4) as spent + FROM transactions t + LEFT JOIN transaction_overrides o ON o.transaction_id = t.id + LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1 + ${STATEMENTS_JOIN} + WHERE ${OWNER_SCOPE} = $1 + AND ${NET_SPEND_ROWS} + AND ${EXCLUDE_NON_SPEND} + AND ${EXCLUDE_RECONCILED_SOURCE} + AND t.transaction_date >= $2 + AND t.transaction_date < $3 + GROUP BY 1, 2, 3 + ORDER BY 1, 2`, + [user.id, toDateStr(startDate), toDateStr(endDate)] + ); + + // Sparse by design — a day with no spend has no entry, and the client treats + // a missing day as zero. Emitting 31 zeroes per month per category would + // dominate the payload. + const daily: Record> = {}; + const byCategory: Record>> = {}; + + for (const r of rows) { + const spent = Number(r.spent); + const m = (daily[r.month] ??= {}); + m[r.day] = (m[r.day] ?? 0) + spent; + + const c = ((byCategory[r.month] ??= {})[r.category] ??= {}); + c[r.day] = (c[r.day] ?? 0) + spent; + } + + return NextResponse.json({ daily, byCategory }); +} diff --git a/src/app/api/analytics/fees/route.ts b/src/app/api/analytics/fees/route.ts index 57657ad..06ad48b 100644 --- a/src/app/api/analytics/fees/route.ts +++ b/src/app/api/analytics/fees/route.ts @@ -1,12 +1,35 @@ import { NextRequest, NextResponse } from "next/server"; import { getCurrentUser } from "@/lib/auth"; import { queryRaw } from "@/lib/db"; -import { OWNER_SCOPE, STATEMENTS_JOIN, mySplitOf } from "@/lib/analytics-sql"; +import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_RECONCILED_SOURCE, mySplitOf, toDateStr } from "@/lib/analytics-sql"; +/** + * Fees and interest over an explicit window. + * + * This used to aggregate every statement ever imported with no date filter, and + * the UI printed the result with no period label — so a lifetime-to-date total + * read as a current-period one, and grew forever. `months=0` asks for all time + * deliberately, which is a different claim from asking for it by accident. + */ export async function GET(req: NextRequest) { const user = await getCurrentUser(req); if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 }); + const { searchParams } = new URL(req.url); + const monthsParam = Number(searchParams.get("months") ?? "12"); + const months = Number.isFinite(monthsParam) ? Math.min(Math.max(monthsParam, 0), 120) : 12; + const allTime = months === 0; + + const now = new Date(); + const from = new Date(now.getFullYear(), now.getMonth() - months + 1, 1); + const fromStr = toDateStr(from); + const toStr = toDateStr(new Date(now.getFullYear(), now.getMonth() + 1, 1)); + + // A statement is dated by the period it covers, not by when it was imported. + const stmtWindow = allTime ? "" : `AND billing_end_date >= $2 AND billing_end_date < $3`; + const txnWindow = allTime ? "" : `AND t.transaction_date >= $2 AND t.transaction_date < $3`; + const windowParams = allTime ? [] : [fromStr, toStr]; + // Statement-level fees and interest (aggregated by Gemini from the PDF) const stmtRows = await queryRaw<{ bank_name: string; @@ -19,10 +42,11 @@ export async function GET(req: NextRequest) { SUM(COALESCE(interest_charged, 0))::numeric(12,2) AS interest FROM statements WHERE owner_id = $1 + ${stmtWindow} GROUP BY bank_name HAVING SUM(COALESCE(fees_charged, 0)) + SUM(COALESCE(interest_charged, 0)) > 0 ORDER BY (SUM(COALESCE(fees_charged, 0)) + SUM(COALESCE(interest_charged, 0))) DESC`, - [user.id] + [user.id, ...windowParams] ); // Transaction-level fee and interest line items (split-adjusted) @@ -49,8 +73,10 @@ export async function GET(req: NextRequest) { ${STATEMENTS_JOIN} WHERE ${OWNER_SCOPE} = $1 AND t.transaction_type IN ('fee', 'interest') + AND ${EXCLUDE_RECONCILED_SOURCE} + ${txnWindow} ORDER BY t.transaction_date DESC`, - [user.id] + [user.id, ...windowParams] ); const by_bank = stmtRows.map((r) => ({ @@ -69,5 +95,13 @@ export async function GET(req: NextRequest) { const total_fees = by_bank.reduce((s, r) => s + r.fees, 0); const total_interest = by_bank.reduce((s, r) => s + r.interest, 0); - return NextResponse.json({ by_bank, transactions, total_fees, total_interest }); + return NextResponse.json({ + by_bank, + transactions, + total_fees, + total_interest, + // The period is part of the answer — the client must be able to say what + // window these totals cover rather than implying "now". + period: { months, from: allTime ? null : fromStr, to: allTime ? null : toStr, all_time: allTime }, + }); } diff --git a/src/app/api/analytics/merchants/[merchant]/route.ts b/src/app/api/analytics/merchants/[merchant]/route.ts index 9abac5d..c1b734f 100644 --- a/src/app/api/analytics/merchants/[merchant]/route.ts +++ b/src/app/api/analytics/merchants/[merchant]/route.ts @@ -1,7 +1,8 @@ import { NextRequest, NextResponse } from "next/server"; import { getCurrentUser } from "@/lib/auth"; import { queryRaw } from "@/lib/db"; -import { OWNER_SCOPE, STATEMENTS_JOIN, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql"; +import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql"; +import { bankLabel } from "@/lib/queries"; const MY_AMOUNT = mySplitOf(`COALESCE(t.amount_aud, t.amount)`); @@ -39,7 +40,7 @@ export async function GET( END::numeric(10,2) as my_amount, t.transaction_type, ${EFFECTIVE_CATEGORY} as category, - COALESCE(s.bank_name, 'Manual') as bank_name, + ${bankLabel()} as bank_name, t.statement_id FROM transactions t ${STATEMENTS_JOIN} @@ -48,6 +49,7 @@ export async function GET( WHERE ${OWNER_SCOPE} = $1 AND t.transaction_type IN ('debit', 'fee', 'interest', 'refund', 'credit') AND COALESCE(o.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) = $2 + AND ${EXCLUDE_RECONCILED_SOURCE} ORDER BY t.transaction_date DESC LIMIT 500 `, [user.id, decoded]); diff --git a/src/app/api/analytics/merchants/route.ts b/src/app/api/analytics/merchants/route.ts index dffc6cf..6137e31 100644 --- a/src/app/api/analytics/merchants/route.ts +++ b/src/app/api/analytics/merchants/route.ts @@ -1,7 +1,7 @@ import { NextRequest, NextResponse } from "next/server"; import { getCurrentUser } from "@/lib/auth"; import { queryRaw } from "@/lib/db"; -import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql"; +import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf, toDateStr } from "@/lib/analytics-sql"; // Split-adjusted amount helper (positive for spend, negative for refunds) const MY_AMOUNT = mySplitOf(`COALESCE(t.amount_aud, t.amount)`); @@ -21,7 +21,7 @@ export async function GET(req: NextRequest) { const cutoff = new Date(); cutoff.setMonth(cutoff.getMonth() - months); - const fromDate = cutoff.toISOString().slice(0, 10); + const fromDate = toDateStr(cutoff); // Merchant aggregates — net spend (debits + fees - refunds/credits) const rows = await queryRaw<{ @@ -69,6 +69,7 @@ export async function GET(req: NextRequest) { AND t.transaction_type IN ('debit', 'fee', 'interest', 'refund', 'credit') AND t.transaction_date >= $2 AND ${EXCLUDE_NON_SPEND} + AND ${EXCLUDE_RECONCILED_SOURCE} GROUP BY 1 HAVING SUM(${SPEND_EXPR}) > 0 ORDER BY net_spend DESC @@ -95,6 +96,7 @@ export async function GET(req: NextRequest) { AND t.transaction_date >= $2 AND COALESCE(o.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) = ANY($3) AND ${EXCLUDE_NON_SPEND} + AND ${EXCLUDE_RECONCILED_SOURCE} GROUP BY 1, 2 ORDER BY 1, 2 `, [user.id, fromDate, topMerchants]); diff --git a/src/app/api/analytics/monthly/route.ts b/src/app/api/analytics/monthly/route.ts index 5652617..4dd53c5 100644 --- a/src/app/api/analytics/monthly/route.ts +++ b/src/app/api/analytics/monthly/route.ts @@ -6,9 +6,11 @@ import { STATEMENTS_JOIN, EFFECTIVE_CATEGORY, EXCLUDE_NON_SPEND, + EXCLUDE_RECONCILED_SOURCE, NET_SPEND_ROWS, SPEND_SIGNED, mySplitOf, + toDateStr, } from "@/lib/analytics-sql"; export async function GET(req: NextRequest) { @@ -22,8 +24,8 @@ export async function GET(req: NextRequest) { const endDate = new Date(now.getFullYear(), now.getMonth() + 1, 1); const startDate = new Date(now.getFullYear(), now.getMonth() - monthCount + 1, 1); - const startStr = startDate.toISOString().slice(0, 10); - const endStr = endDate.toISOString().slice(0, 10); + const startStr = toDateStr(startDate); + const endStr = toDateStr(endDate); // Expenses: debits excluding transfers and investments, split-adjusted const spendRows = await queryRaw<{ @@ -35,7 +37,9 @@ export async function GET(req: NextRequest) { `SELECT TO_CHAR(DATE_TRUNC('month', t.transaction_date::date), 'YYYY-MM') as month, ${EFFECTIVE_CATEGORY} as category, - SUM(${mySplitOf(SPEND_SIGNED)})::numeric(12,2) as total_spent, + -- 4dp so the month total is summed from unrounded parts; every consumer + -- rounds for display. See the note in /api/analytics/daily. + SUM(${mySplitOf(SPEND_SIGNED)})::numeric(14,4) as total_spent, COUNT(*)::int as transaction_count FROM transactions t LEFT JOIN transaction_overrides o ON o.transaction_id = t.id @@ -44,6 +48,7 @@ export async function GET(req: NextRequest) { WHERE ${OWNER_SCOPE} = $1 AND ${NET_SPEND_ROWS} AND ${EXCLUDE_NON_SPEND} + AND ${EXCLUDE_RECONCILED_SOURCE} AND t.transaction_date >= $2 AND t.transaction_date < $3 GROUP BY 1, 2 @@ -67,6 +72,7 @@ export async function GET(req: NextRequest) { WHERE ${OWNER_SCOPE} = $1 AND t.transaction_type IN ('credit', 'payment') AND ${EFFECTIVE_CATEGORY} = 'income' + AND ${EXCLUDE_RECONCILED_SOURCE} AND t.transaction_date >= $2 AND t.transaction_date < $3 GROUP BY 1 @@ -89,6 +95,7 @@ export async function GET(req: NextRequest) { ${STATEMENTS_JOIN} WHERE ${OWNER_SCOPE} = $1 AND ${EFFECTIVE_CATEGORY} = 'investment' + AND ${EXCLUDE_RECONCILED_SOURCE} AND t.transaction_date >= $2 AND t.transaction_date < $3 GROUP BY 1 diff --git a/src/app/api/analytics/subscriptions/route.ts b/src/app/api/analytics/subscriptions/route.ts index 335430f..46ab938 100644 --- a/src/app/api/analytics/subscriptions/route.ts +++ b/src/app/api/analytics/subscriptions/route.ts @@ -1,7 +1,7 @@ import { NextRequest, NextResponse } from "next/server"; import { getCurrentUser } from "@/lib/auth"; import { queryRaw } from "@/lib/db"; -import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql"; +import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql"; export async function GET(req: NextRequest) { const user = await getCurrentUser(req); @@ -31,6 +31,7 @@ export async function GET(req: NextRequest) { WHERE ${OWNER_SCOPE} = $1 AND t.transaction_type IN ('debit', 'fee') AND ${EXCLUDE_NON_SPEND} + AND ${EXCLUDE_RECONCILED_SOURCE} AND COALESCE(o.merchant_normalized, t.merchant_normalized, t.merchant_name) IS NOT NULL ), merchant_with_lag AS ( diff --git a/src/app/budget/page.tsx b/src/app/budget/page.tsx index de19410..7fe8a06 100644 --- a/src/app/budget/page.tsx +++ b/src/app/budget/page.tsx @@ -16,7 +16,7 @@ import { ReferenceLine, } from "recharts"; import { useQueryClient } from "@tanstack/react-query"; -import { useMonthlyAnalytics, useTransactions, useUpdateTransaction } from "@/lib/hooks"; +import { useMonthlyAnalytics, useDailySpend, useTransactions, useUpdateTransaction } from "@/lib/hooks"; import { formatCategory, CATEGORIES } from "@/lib/categories"; import { CATEGORY_COLORS, CHART, TOOLTIP_STYLE } from "@/lib/category-colors"; @@ -37,6 +37,26 @@ function formatShortMonth(m: string): string { const [year, month] = m.split("-"); return new Date(Number(year), Number(month) - 1, 1).toLocaleString("default", { month: "short" }); } +function daysInMonthOf(m: string): number { + const [year, month] = m.split("-").map(Number); + return new Date(year, month, 0).getDate(); +} +/** + * How much of `month` has actually happened. A month in progress is only + * complete up to today; every earlier month is complete. + */ +function elapsedDays(m: string): number { + return m === currentMonthStr() ? new Date().getDate() : daysInMonthOf(m); +} +/** Spend in `month` from day 1 through `throughDay` inclusive. */ +function spendThrough(days: Record | undefined, throughDay: number): number { + if (!days) return 0; + let sum = 0; + for (const [d, v] of Object.entries(days)) { + if (Number(d) <= throughDay) sum += v; + } + return sum; +} function fmt(n: number): string { return `$${Math.round(n).toLocaleString()}`; } function fmtExact(n: number): string { return `$${n.toFixed(2)}`; } function fmtSigned(n: number): string { return `${n >= 0 ? "+" : "−"}$${Math.abs(n) >= 100 ? Math.round(Math.abs(n)).toLocaleString() : Math.abs(n).toFixed(0)}`; } @@ -195,12 +215,14 @@ export default function AnalyticsPage() { // eslint-disable-next-line react-hooks/exhaustive-deps }, [months]); - // Cumulative chart: fetch this month's transactions - const smFrom = `${selectedMonth}-01`; - const [smYear, smMonth] = selectedMonth.split("-").map(Number); - const smNextDate = new Date(smYear, smMonth, 1); - const smTo = `${smNextDate.getFullYear()}-${String(smNextDate.getMonth() + 1).padStart(2, "0")}-01`; - const { data: monthTxData } = useTransactions({ from: smFrom, to: smTo, limit: 1000 }); + // Day-of-month spend, split-adjusted server-side by the same rules as the + // headline. Also what makes the comparisons below like-for-like. + const { data: dailyData } = useDailySpend(12); + + // A month in progress is only comparable to prior months through the same day. + // Comparing 27 days of July against a full June always flattered July. + const compareDay = elapsedDays(selectedMonth); + const selectedIsPartial = selectedMonth === currentMonthStr(); // Category rows for selected month const categoryRows = useMemo(() => { @@ -226,22 +248,27 @@ export default function AnalyticsPage() { .slice(0, 8); }, [analytics, months, selectedMonth]); - // Top movers vs previous month + // Top movers vs previous month, compared through the same day of the month so + // a month in progress is not measured against a complete one. const movers = useMemo(() => { if (!analytics) return []; const pm = prevMonth(selectedMonth); if (!months.includes(pm)) return []; - return analytics.rows - .map((r) => ({ - category: r.category, - delta: (r.spent[selectedMonth] || 0) - (r.spent[pm] || 0), - now: r.spent[selectedMonth] || 0, - before: r.spent[pm] || 0, - })) + + const catsNow = dailyData?.byCategory?.[selectedMonth] ?? {}; + const catsBefore = dailyData?.byCategory?.[pm] ?? {}; + const categories = new Set([...Object.keys(catsNow), ...Object.keys(catsBefore)]); + + return Array.from(categories) + .map((category) => { + const now = spendThrough(catsNow[category], compareDay); + const before = spendThrough(catsBefore[category], compareDay); + return { category, delta: now - before, now, before }; + }) .filter((r) => Math.abs(r.delta) >= 1) .sort((a, b) => Math.abs(b.delta) - Math.abs(a.delta)) .slice(0, 6); - }, [analytics, months, selectedMonth]); + }, [analytics, dailyData, months, selectedMonth, compareDay]); // Pareto chart data const paretoData = useMemo(() => { @@ -258,39 +285,40 @@ export default function AnalyticsPage() { }); }, [categoryRows]); - // Cumulative spend chart data + // Cumulative spend chart data. + // + // Both series now come from the same server-side spend definition as the + // headline. The typical line is also a real averaged curve rather than the + // month total spread evenly — spending is lumpy (rent on the 1st, a shop on + // the weekend), so a straight line made ordinary months look erratic. const cumulativeData = useMemo(() => { - const daysInMonth = new Date(smYear, smMonth, 0).getDate(); - const isCurrentMonth = selectedMonth === currentMonthStr(); - const today = new Date(); - const lastDay = isCurrentMonth ? today.getDate() : daysInMonth; + const daysInMonth = daysInMonthOf(selectedMonth); + const lastDay = elapsedDays(selectedMonth); - const daily: Record = {}; - (monthTxData?.data ?? []) - .filter((tx) => tx.transaction_type === "debit" && - !["transfers", "investment"].includes(tx.effective_category) && - !tx.tags?.some((t: any) => (typeof t === "string" ? t : t.name) === "family")) - .forEach((tx) => { - const day = new Date(tx.transaction_date).getDate(); - daily[day] = (daily[day] || 0) + Number(tx.amount_aud ?? tx.amount); - }); - - const priorMonths = analytics?.months.filter((m) => m !== selectedMonth) ?? []; - const priorAvg = priorMonths.length > 0 - ? priorMonths.reduce((s, m) => s + (analytics?.totals[m]?.spent || 0), 0) / priorMonths.length - : 0; + const daily = dailyData?.daily?.[selectedMonth] ?? {}; + // Only complete months with data form the baseline. A month still in + // progress has no spend recorded past today, so including it would pull the + // typical curve down by however much of it has not happened yet. + const priorMonths = (analytics?.months ?? []).filter( + (m) => m !== selectedMonth && m !== currentMonthStr() && (analytics?.totals[m]?.spent || 0) > 0 + ); let cum = 0; return Array.from({ length: daysInMonth }, (_, i) => { const day = i + 1; if (day <= lastDay) cum += daily[day] || 0; + + const typical = priorMonths.length + ? priorMonths.reduce((s, m) => s + spendThrough(dailyData?.daily?.[m], day), 0) / priorMonths.length + : 0; + return { day, actual: day <= lastDay ? Math.round(cum * 100) / 100 : null, - typical: Math.round((priorAvg * day / daysInMonth) * 100) / 100, + typical: Math.round(typical * 100) / 100, }; }); - }, [monthTxData, analytics, selectedMonth, smYear, smMonth]); + }, [dailyData, analytics, selectedMonth]); if (isLoading || !analytics) { return ( @@ -305,16 +333,30 @@ export default function AnalyticsPage() { const hasIncome = months.some((m) => (analytics.totals[m]?.income || 0) > 0); const hasInvestments = months.some((m) => (analytics.totals[m]?.investments || 0) > 0); - // Hero delta vs the average of the other months that have data - const otherMonths = months.filter((m) => m !== selectedMonth && (analytics.totals[m]?.spent || 0) > 0); + // Hero delta vs the average of the other *complete* months that have data. + // + // Two partial-month traps here. The month in progress never belongs in the + // baseline, because most of it has not happened. And when the month in + // progress is the one selected, its running total has to be measured against + // the same slice of each prior month, not against their full totals. + const otherMonths = months.filter( + (m) => m !== selectedMonth && m !== currentMonthStr() && (analytics.totals[m]?.spent || 0) > 0 + ); + const comparableSpend = selectedIsPartial + ? spendThrough(dailyData?.daily?.[selectedMonth], compareDay) + : totals.spent; const avgSpend = otherMonths.length - ? otherMonths.reduce((s, m) => s + (analytics.totals[m]?.spent || 0), 0) / otherMonths.length + ? otherMonths.reduce( + (s, m) => s + (selectedIsPartial ? spendThrough(dailyData?.daily?.[m], compareDay) : analytics.totals[m]?.spent || 0), + 0 + ) / otherMonths.length : 0; - const avgDeltaPct = avgSpend > 0 ? Math.round(((totals.spent - avgSpend) / avgSpend) * 100) : 0; + const avgDeltaPct = avgSpend > 0 ? Math.round(((comparableSpend - avgSpend) / avgSpend) * 100) : 0; + const throughQualifier = selectedIsPartial ? ` through day ${compareDay}` : ""; const heroSentence = avgSpend === 0 ? "" : - Math.abs(avgDeltaPct) <= 3 ? `in line with your ${otherMonths.length}-month average` : - `${Math.abs(avgDeltaPct)}% ${avgDeltaPct > 0 ? "above" : "below"} your ${otherMonths.length}-month average of ${fmt(avgSpend)}`; + Math.abs(avgDeltaPct) <= 3 ? `in line with your ${otherMonths.length}-month average${throughQualifier}` : + `${Math.abs(avgDeltaPct)}% ${avgDeltaPct > 0 ? "above" : "below"} your ${otherMonths.length}-month average of ${fmt(avgSpend)}${throughQualifier}`; const pareto80idx = paretoData.findIndex((r) => r.cumulative >= 80); const tableMonths = analytics.months.slice(0, 6); // newest-first, last 6 @@ -373,7 +415,10 @@ export default function AnalyticsPage() { {movers.length > 0 && (

What changed

-

Biggest category moves vs {formatShortMonth(prevMonth(selectedMonth))}

+

+ Biggest category moves vs {formatShortMonth(prevMonth(selectedMonth))} + {selectedIsPartial && `, both through day ${compareDay}`} +

{movers.map((m) => (
diff --git a/src/app/insights/page.tsx b/src/app/insights/page.tsx index cfde3ee..b742b3f 100644 --- a/src/app/insights/page.tsx +++ b/src/app/insights/page.tsx @@ -10,6 +10,18 @@ import { CHART, TOOLTIP_STYLE } from "@/lib/category-colors"; const SPEND_TYPES = new Set(["debit", "fee", "interest"]); +/** + * The API returns an exclusive upper bound (first day of the month after the + * window). Showing that date verbatim would claim a month the totals exclude, + * so `exclusive` steps back a day for display. + */ +function formatPeriodBound(iso: string | null, exclusive = false): string { + if (!iso) return "—"; + const d = new Date(`${iso}T00:00:00`); + if (exclusive) d.setDate(d.getDate() - 1); + return d.toLocaleDateString("default", { month: "short", year: "numeric" }); +} + function fmt(n: number) { return new Intl.NumberFormat("en-AU", { style: "currency", currency: "AUD", maximumFractionDigits: 0 }).format(n); } @@ -41,10 +53,13 @@ const FREQ_LABEL: Record = { }; // ─── Section wrapper ──────────────────────────────────────────────── -function Section({ title, children }: { title: string; children: React.ReactNode }) { +function Section({ title, aside, children }: { title: string; aside?: React.ReactNode; children: React.ReactNode }) { return (
-

{title}

+
+

{title}

+ {aside} +
{children}
); @@ -298,7 +313,8 @@ export default function InsightsPage() { const { data: analytics } = useMonthlyAnalytics(12); const { data: analytics6 } = useMonthlyAnalytics(6); const { data: subData } = useSubscriptions(); - const { data: feesData } = useFees(); + const [feeMonths, setFeeMonths] = useState(12); + const { data: feesData } = useFees(feeMonths); // Build regular/occasional chart data const chartData = useMemo(() => { @@ -431,11 +447,38 @@ export default function InsightsPage() { {/* ── 4. Fees & Interest ── */} -
+
+ {feesData?.period && ( + + {feesData.period.all_time + ? "All time" + : `${formatPeriodBound(feesData.period.from)} – ${formatPeriodBound(feesData.period.to, true)}`} + + )} + +
+ } + > {!feesData ? (

Loading...

) : feesData.by_bank.length === 0 && feesData.transactions.length === 0 ? ( -

No fees or interest recorded across your statements.

+

+ No fees or interest recorded {feesData.period?.all_time ? "on any statement" : "in this period"}. +

) : (
{feesData.by_bank.length > 0 && ( diff --git a/src/app/shared/page.tsx b/src/app/shared/page.tsx index c40ca08..95de71e 100644 --- a/src/app/shared/page.tsx +++ b/src/app/shared/page.tsx @@ -22,8 +22,12 @@ function formatDate(d: string) { const SPEND_TYPES = new Set(["debit", "fee", "interest"]); -function formatAmount(n: number, type?: string) { - const formatted = `$${Number(n).toFixed(2)}`; +function formatAmount(n: number, type?: string, currency?: string) { + // A bare "$" on a non-AUD row was the visible half of the problem: the row + // read as dollars while the participant balances converted to AUD, so the + // two disagreed on screen with nothing to explain why. + const value = Number(n).toFixed(2); + const formatted = !currency || currency === "AUD" ? `$${value}` : `${currency} ${value}`; return type && !SPEND_TYPES.has(type) ? `+${formatted}` : formatted; } @@ -298,7 +302,17 @@ export default function SharedPage() { return {sortDir === "desc" ? "↓" : "↑"}; } const { data: balances = [], isLoading: balLoading } = useParticipantBalances(realTagIds); + const { data: allTags = [] } = useTags(); const { data: me } = useCurrentUser(); + + // Names the tag scope when one is active. Non-empty means the cards below are + // split totals rather than payable balances. + const tagScopeLabel = + realTagIds.length === 0 + ? null + : realTagIds.length === 1 + ? (allTags.find((t) => String(t.id) === realTagIds[0])?.name ?? "this tag") + : `${realTagIds.length} tags`; const [addingParticipant, setAddingParticipant] = useState(false); const [paymentModal, setPaymentModal] = useState<{ id: number; name: string; balance: number } | null>(null); const [showHistory, setShowHistory] = useState(null); @@ -351,23 +365,37 @@ export default function SharedPage() {

{b.name}

- {settled ? "all square" : theyOweMe ? `owes you` : "you owe"} + {/* With a tag filter on, payments are deliberately not + subtracted — so this is a split total, not a payable + balance, and must not claim to be one. */} + {tagScopeLabel + ? `split total in ${tagScopeLabel}` + : settled ? "all square" : theyOweMe ? "owes you" : "you owe"}

-

+

${net.toFixed(2)}

+ {b.unconverted_count > 0 && ( +

+ approx · {b.unconverted_count} unconverted +

+ )}
- + {/* Settling against a tag-scoped total would record a payment + for a figure that never was the debt. */} + {!tagScopeLabel && ( + + )}