fix(analytics): make the displayed numbers mean what they say
ci / lint-test (push) Failing after 44s

Six metric-integrity defects from the UI/IA review, plus two found while
verifying the review's own claims against the code.

The reconciled-row exclusion existed only in queries.ts. Every analytics
route counted the superseded manual rows as spend — 48 rows, $4,474.79 of
double count, invisible precisely because the transaction list looked
right. It is now one fragment both sides import.

The spend-pace chart computed its own totals in the browser: gross
amounts, debits only, no personal share, no refunds, fees, interest or
itemised loan repayments. On live data it ended July at $4,747.31 under a
headline reading $3,597.10 — and its own baseline line was drawn from the
split-adjusted monthly totals, so the two series in one chart disagreed
with each other. Both now come from /api/analytics/daily, built from the
same fragments as the headline.

Fees aggregated every statement ever imported with no date filter, under
a heading with no period, so a lifetime figure read as a current one and
grew forever. Now bounded, labelled, and selectable.

Comparisons no longer measure a month in progress against complete ones:
the in-progress month is out of every baseline, and a selected current
month is compared through the same day.

Two the review did not catch:

- Every analytics window was a day early. toISOString() on a
  local-midnight Date converts backwards through UTC. Surfaced only once
  fees started reporting the range it had used.
- /monthly rounded per category, /daily per category-day, so the pace
  chart ended a few cents off the headline above it.

Shared currency needed amending rather than applying. Reading s.currency
would have labelled every order row AUD, since an order receipt has no
statement and carries its own currency — the opposite convention from a
foreign charge on an AUD statement, where amount IS AUD. NATIVE_CURRENCY's
COALESCE order keeps the two apart. Balances also now count rows whose AUD
value is genuinely unknown instead of netting a foreign figure against AUD
ones. Latent today: no foreign transaction is currently split.

Tag-filtered balance cards no longer claim "owes you". With a filter on,
payments are deliberately not subtracted, so the figure is a split total
and settling against it would record a payment for a debt that never was.

Split-coverage warnings deliberately omitted (user decision).
This commit is contained in:
2026-07-27 17:10:27 +10:00
parent 5ee5ee24cf
commit a4ab543a6c
14 changed files with 643 additions and 90 deletions
+68 -1
View File
@@ -1,7 +1,7 @@
// Shared SQL fragments for analytics queries, so spend/income semantics stay
// identical across routes.
//
// Two rules every analytics query must follow:
// Three rules every analytics query must follow:
//
// 1. Join `statements` with LEFT JOIN and scope on COALESCE(t.owner_id, s.owner_id).
// An INNER JOIN silently drops every manual/CSV transaction (statement_id IS
@@ -11,6 +11,23 @@
// bank account and again as the underlying purchases on the card statement.
// Categorising the money movement as `transfers` and excluding it here is what
// stops the double count. Investments are a balance-sheet move, not spend.
// 3. Apply EXCLUDE_RECONCILED_SOURCE to every row-level query. The transaction
// queries have always done this (`queries.ts`); analytics never did, which is
// the other half of the same double count.
/**
* `YYYY-MM-DD` for a Date, read in local time.
*
* `toISOString().slice(0, 10)` is the obvious thing and it is wrong here: these
* are calendar boundaries built with `new Date(y, m, 1)`, which is local
* midnight. In any timezone east of UTC that converts to the *previous* day, so
* every window silently started and ended a day early — visible once the fees
* endpoint began reporting the range it had used ("2026-04-30" for a window
* meant to open on 1 May).
*/
export function toDateStr(d: Date): string {
return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}-${String(d.getDate()).padStart(2, "0")}`;
}
/** Owner scoping that works for both statement-linked and manual transactions. */
export const OWNER_SCOPE = `COALESCE(t.owner_id, s.owner_id)`;
@@ -18,6 +35,56 @@ export const OWNER_SCOPE = `COALESCE(t.owner_id, s.owner_id)`;
/** Join clause to pair with OWNER_SCOPE. */
export const STATEMENTS_JOIN = `LEFT JOIN statements s ON s.id = t.statement_id`;
/**
* Drops the manual/CSV row that a statement line has superseded.
*
* Reconciliation keeps both rows: the manual one the user entered and the
* statement line it turned out to be. Only the statement line should count, or
* the same purchase is spent twice. `queries.ts` has always applied this; the
* analytics routes did not, so every reconciled row was double-counted in the
* category totals, movers, Pareto, and merchant rankings.
*
* Scoped to `statement_id IS NULL` deliberately: the *source* row is the manual
* one. A statement line pointing at something else is the survivor, not the
* duplicate.
*
* Order-receipt rows (`payment_method = 'credits'`) are unaffected — they are
* inserted with `reconciled_with_id` NULL and are held out of the reconcile
* queue by `needsCardMatch()`, so nothing ever sets it. If one is reconciled by
* hand against a card line, this is what stops it double-counting.
*/
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)`;
/**
* The currency `t.amount` is actually denominated in.
*
* Two different conventions meet here and the COALESCE order is what keeps them
* apart:
*
* - A statement row is denominated in its statement's currency. If that row is
* an overseas purchase on an AUD statement, `amount` is still AUD and
* `foreign_currency_code` merely records what was originally charged — so
* `s.currency` must win.
* - An order-receipt row has no statement. There, `amount` IS the native
* figure and `foreign_currency_code` names it (order-ingestion.ts leaves
* `amount_aud` NULL rather than asserting an FX rate it does not have).
*
* Reading `s.currency` alone labels every foreign order row as AUD.
* `s` must be the statements alias in scope.
*/
export const NATIVE_CURRENCY = `COALESCE(s.currency, t.foreign_currency_code, 'AUD')`;
/**
* True when a row's AUD value is unknown: it is denominated in something other
* than AUD and carries no converted figure.
*
* Every settlement total uses `COALESCE(t.amount_aud, t.amount)`, which for such
* a row silently nets a foreign figure against AUD ones. Rather than drop the
* row (which changes a balance with no trace) or convert it (with no rate),
* count these and let the UI say the balance is incomplete.
*/
export const AMOUNT_UNCONVERTED = `(t.amount_aud IS NULL AND ${NATIVE_CURRENCY} <> 'AUD')`;
/** Transaction types that represent money going out. */
export const SPEND_TYPES = `('debit', 'fee', 'interest')`;
+35 -5
View File
@@ -1,7 +1,7 @@
"use client";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import type { TransactionRow, StatementRow, TagRow, TripRow, TripAnalytics } from "./queries";
import type { TransactionRow, StatementRow, TagRow, TripRow, TripAnalytics, ParticipantBalance } from "./queries";
export type { TripRow, TripAnalytics };
import type { CurrentUser } from "./auth";
@@ -216,7 +216,7 @@ export function useParticipants() {
}
export function useParticipantBalances(tagIds?: string[]) {
return useQuery<{ id: number; name: string; total_owed: number; unsettled_count: number }[]>({
return useQuery<ParticipantBalance[]>({
queryKey: ["participant-balances", tagIds],
queryFn: async () => {
const params = tagIds?.length ? `?tag_ids=${tagIds.join(",")}` : "";
@@ -775,6 +775,27 @@ export function useMonthlyAnalytics(months?: number) {
});
}
/**
* Sparse by day-of-month; a missing day means zero.
* daily → { "2026-07": { 3: 42.10 } }
* byCategory → { "2026-07": { dining: { 3: 42.10 } } }
*/
export interface DailySpend {
daily: Record<string, Record<number, number>>;
byCategory: Record<string, Record<string, Record<number, number>>>;
}
export function useDailySpend(months?: number) {
const m = months || 12;
return useQuery<DailySpend>({
queryKey: ["analytics", "daily", m],
queryFn: async () => {
const res = await fetch(`/api/analytics/daily?months=${m}`);
return res.json();
},
});
}
export interface SubscriptionRow {
merchant: string;
category: string;
@@ -815,16 +836,25 @@ export interface FeeTxnRow {
bank_name: string;
}
export function useFees() {
export interface FeePeriod {
months: number;
from: string | null;
to: string | null;
all_time: boolean;
}
/** `months = 0` means all time. */
export function useFees(months = 12) {
return useQuery<{
by_bank: FeeBankRow[];
transactions: FeeTxnRow[];
total_fees: number;
total_interest: number;
period: FeePeriod;
}>({
queryKey: ["analytics", "fees"],
queryKey: ["analytics", "fees", months],
queryFn: async () => {
const res = await fetch("/api/analytics/fees");
const res = await fetch(`/api/analytics/fees?months=${months}`);
return res.json();
},
});
+29 -11
View File
@@ -1,4 +1,5 @@
import { queryRaw } from "./db";
import { EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY, AMOUNT_UNCONVERTED } from "./analytics-sql";
export interface RoutePointRow {
label: string;
@@ -49,9 +50,13 @@ export interface TransactionRow {
my_amount: number;
// statement context (null for manual transactions)
bank_name: string;
// Native currency of the statement this row came from ('AUD' for manual rows).
// `amount` is in this currency; `amount_aud` is the converted figure.
// The currency `amount` is denominated in; `amount_aud` is the converted
// figure where one exists. Usually the statement's currency, but an
// order-receipt row has no statement and carries its own — see
// NATIVE_CURRENCY. Not simply 'AUD' for every statement-less row.
currency: string;
/** True when `amount` is non-AUD and no converted figure exists. */
amount_unconverted: boolean;
owner_id: number;
owner_name: string;
// tags
@@ -124,7 +129,7 @@ interface TransactionFilters {
export async function getTransactions(ownerId: number, filters: TransactionFilters) {
const conditions: string[] = [
`(COALESCE(t.owner_id, s.owner_id) = $1 OR EXISTS (SELECT 1 FROM transaction_splits ts_me WHERE ts_me.transaction_id = t.id AND ts_me.participant_id = $1))`,
`NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)`,
EXCLUDE_RECONCILED_SOURCE,
];
const params: unknown[] = [ownerId];
let paramIdx = 2;
@@ -227,7 +232,8 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
COALESCE(o.category_override, t.category) as effective_category,
COALESCE(o.merchant_normalized, t.merchant_normalized, t.merchant_name) as effective_merchant,
${bankLabel()} as bank_name,
COALESCE(s.currency, 'AUD') as currency,
${NATIVE_CURRENCY} as currency,
${AMOUNT_UNCONVERTED} as amount_unconverted,
-- My share, resolved the same way analytics does it (see myShare in
-- analytics-sql.ts): explicit split row, then override, then whatever is
-- left after everyone else. Computed here so the UI cannot drift from
@@ -427,6 +433,8 @@ export interface ParticipantBalance {
name: string;
total_owed: number;
unsettled_count: number;
/** Splits counted at a non-AUD figure because no converted amount exists. */
unconverted_count: number;
}
export async function getParticipantBalances(ownerId: number, tagIds?: number[]) {
@@ -456,7 +464,11 @@ export async function getParticipantBalances(ownerId: number, tagIds?: number[])
SELECT p.id, p.name,
COALESCE(SUM(splits.signed_amount), 0)::numeric(12,2)
${paymentsSelect} AS total_owed,
COALESCE(SUM(splits.split_count), 0)::int AS unsettled_count
COALESCE(SUM(splits.split_count), 0)::int AS unsettled_count,
-- Splits whose AUD value is unknown. They are still summed above (as
-- their native figure), so a non-zero count means this balance is
-- approximate and the UI has to say so.
COALESCE(SUM(splits.unconverted_count), 0)::int AS unconverted_count
FROM participants p
LEFT JOIN (
@@ -465,12 +477,13 @@ export async function getParticipantBalances(ownerId: number, tagIds?: number[])
-- currency, so splitting on it nets a USD figure against AUD ones.
SELECT ts.participant_id AS pid,
(CASE WHEN t.transaction_type IN ('debit', 'fee', 'interest') THEN COALESCE(t.amount_aud, t.amount) ELSE -COALESCE(t.amount_aud, t.amount) END) * ts.share_percent / 100 AS signed_amount,
1 AS split_count
1 AS split_count,
(CASE WHEN ${AMOUNT_UNCONVERTED} THEN 1 ELSE 0 END) AS unconverted_count
FROM transaction_splits ts
JOIN transactions t ON t.id = ts.transaction_id
LEFT JOIN statements s ON s.id = t.statement_id
WHERE COALESCE(t.owner_id, s.owner_id) = $1 AND ts.participant_id != $1
AND NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
AND ${EXCLUDE_RECONCILED_SOURCE}
${tagFilter}
UNION ALL
@@ -478,12 +491,13 @@ export async function getParticipantBalances(ownerId: number, tagIds?: number[])
-- I owe them: my splits on transactions they own
SELECT COALESCE(t.owner_id, s.owner_id) AS pid,
-((CASE WHEN t.transaction_type IN ('debit', 'fee', 'interest') THEN COALESCE(t.amount_aud, t.amount) ELSE -COALESCE(t.amount_aud, t.amount) END) * ts.share_percent / 100) AS signed_amount,
0 AS split_count
0 AS split_count,
(CASE WHEN ${AMOUNT_UNCONVERTED} THEN 1 ELSE 0 END) AS unconverted_count
FROM transaction_splits ts
JOIN transactions t ON t.id = ts.transaction_id
LEFT JOIN statements s ON s.id = t.statement_id
WHERE ts.participant_id = $1 AND COALESCE(t.owner_id, s.owner_id) != $1
AND NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
AND ${EXCLUDE_RECONCILED_SOURCE}
${tagFilter}
) splits ON splits.pid = p.id
${paymentsJoin}
@@ -742,6 +756,10 @@ export async function getSharedTransactions(ownerId: number, tagIds?: number[],
COALESCE(o.category_override, t.category) as effective_category,
COALESCE(o.merchant_normalized, t.merchant_normalized, t.merchant_name) as effective_merchant,
${bankLabel()} as bank_name,
-- The table renders t.amount, which is not always AUD. Without these the
-- rows visibly disagreed with the participant balances, which do convert.
${NATIVE_CURRENCY} as currency,
${AMOUNT_UNCONVERTED} as amount_unconverted,
COALESCE(t.owner_id, s.owner_id) as owner_id,
p_owner.name as owner_name,
COALESCE(src.created_at, t.created_at) as created_at,
@@ -768,10 +786,10 @@ export async function getSharedTransactions(ownerId: number, tagIds?: number[],
AND EXISTS (SELECT 1 FROM transaction_splits ts_me WHERE ts_me.transaction_id = t.id AND ts_me.participant_id = $1)
)
)
AND NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
AND ${EXCLUDE_RECONCILED_SOURCE}
${tagClause}
${participantClause}
GROUP BY t.id, o.category_override, o.merchant_normalized, o.notes, s.bank_name, s.owner_id, p_owner.name, src.created_at
GROUP BY t.id, o.category_override, o.merchant_normalized, o.notes, s.bank_name, s.currency, s.owner_id, p_owner.name, src.created_at
ORDER BY t.transaction_date DESC
`, params);