40 Commits
Author SHA1 Message Date
siddharthdandClaude Opus 5.5 c25c82932d void: a statementless row for a charge that never posted (migration 0040)
ci / lint-test (push) Successful in 54s
A card notification is sent for an authorisation; a released hold never
reaches a statement. Txn 7951 (a $1,040.41 Airbnb hold, booking cancelled
the next day) sat in the reconcile queue forever and counted as spend.
Neither reconcile (needs a statement twin) nor supersede (needs a survivor)
can express it.

- transactions.voided_at + void_reason; CHECKs: reason mandatory,
  statementless only
- EXCLUDE_RECONCILED_SOURCE excludes voided rows; reconcile queue,
  findAutoMatches, both adoption helpers, annual charges and the manual
  reconcile POST filter it; merge refuses a voided row on either side
- POST/DELETE /api/transactions/<id>/void; refuses reconciled, superseded
  and settled-split rows
- "Never posted" button on /reconcile
- wallet-ingestion deliberately still sees voided rows, so a late twin or a
  prompt_version re-post cannot rebook the hold

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-12 18:05:05 +11:00
siddharthd a1c2eb736f rules: run history timestamps in en-AU, not the browser's locale (d/m, never m/d)
ci / lint-test (push) Successful in 1m4s
2026-10-11 00:48:20 +11:00
siddharthd 069034951e reconcile queue: offer a foreign row's exact native twin regardless of provisional AUD
ci / lint-test (push) Successful in 1m4s
Wallet INR taps were priced at a provisional 0.0180 AUD/INR; CommBank charged
~0.0149, so every rupee twin sat ~20% outside the picker's +-5% band. The
auto-matcher pairs on the native figure but refuses ambiguity (INR 112 twice in
three days, INR 175 three times), which left those rows offered nothing at all.
2026-10-11 00:35:50 +11:00
siddharthdandClaude Opus 5.5 e956f39ffc order-parse: document that both order lanes must send the RFC Message-ID
ci / lint-test (push) Successful in 1m1s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 18:59:56 +10:00
siddharthdandClaude Opus 5.5 d75589e128 docs: the wallet lane -- storage convention, pairing, lock, skips, replay
ci / lint-test (push) Successful in 48s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 17:30:48 +10:00
siddharthdandClaude Opus 5.5 ce0bf0a775 reconcile queue: never list or offer a superseded row
ci / lint-test (push) Successful in 58s
getPendingReconciliations filtered neither the manual side nor the candidate
statement side on superseded_by_id. After the 2026-10-03 wallet cleanup the
queue still showed duplicate half 7037 (superseded under 7038) beside its
twin, with the same Digidirect statement line offered to both -- accepting it
on the hidden row would have spent the line and stranded the real one. The
candidate side could likewise offer one of the 31 superseded ANZ rows.
findAutoMatches() always had both filters. First tests for this query;
both fail against the old SQL.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 17:21:31 +10:00
siddharthdandClaude Opus 5.5 a3e4e1102a wallet: lock the ingest, pair on time and app, skip notices, ECB fallback
ci / lint-test (push) Successful in 54s
Four defects found reviewing a month of overseas wallet rows:

- Race: n8n posts a poll's items concurrently and both halves of a tap
  usually share a poll, so each ran findPairedRow before the other inserted.
  11 taps became 22 rows (A$3,841.40), identical created_at on every pair.
  One advisory lock now covers re-check -> pair -> adopt -> insert, on the
  transaction's own connection (a second pooled one per request deadlocks a
  burst). Mutation-tested: without the lock the concurrent test fails 3/3.
- Pair window: amount within +/-1 DAY folded the UNICEF $50 donation and an
  unrelated $50 Amex charge 17h later into one row and renamed it. A pair is
  now minutes apart AND from two different apps. Needs when/where a
  notification arrived: migration 0039 adds expense_metadata.notified_at /
  notif_app, backfilled from capture.wallet_events (144 rows). A merged half
  now records wallet_evt:<id>, so a replay cannot merge it twice.
- Not purchases: a CommBank overdraft notice became a $21.21 purchase named
  "To avoid fees and interest," (the Westpac fixed-width parser ran on a
  CommBank body). Account notices and transit-card balance drawdowns are
  skipped; the descriptor parser only runs on Westpac.
- FX: with no ledger history (Hong Kong) every tap stored A$0.00. The ECB
  rate via Frankfurter is the fallback, flagged rate_ecb; the ledger median
  stays first because it carries the card's FX fee.

Also: dates in Australia/Melbourne like the statements (UTC slicing filed 28
of 134 rows a day early), and test:setup now actually rebuilds personal_test
-- replaying the dump over it never applied a migration.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 16:55:07 +10:00
siddharthdandClaude Opus 5.5 22fde30b26 currency: a wallet row's amount is AUD, so show its native figure, not the AUD one under a foreign label
ci / lint-test (push) Successful in 1m41s
Wallet rows store the provisional AUD figure in amount and amount_aud and the
native figure in foreign_currency_amount. NATIVE_CURRENCY labelled every
statement-less row with foreign_currency_code (the order-lane convention,
where amount IS native and amount_aud is NULL), so an INR 524 tap rendered as
"$9.42 / INR 9.42" (txn 8162).

amount_aud being present now marks amount as AUD. Measured live: exactly the
38 foreign wallet rows change label, nothing else. The transactions list's
second line prints foreign_currency_amount where one exists, which also
surfaces the native figure on statement rows that carry one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 16:38:33 +10:00
siddharthdandClaude Opus 5 48c6b2395f wallet: never store a native figure as AUD; rate from statement rows only
ci / lint-test (push) Successful in 51s
A CommBank approval prompt for 672006 VND had no VND rate history, and the
no_rate_reference branch left the native figure in transactions.amount -- the
ledger carried A$672,006 for six days (txn 7316, repaired to 35.95).

- no rate history: amount = 0, still flagged aud_provisional/no_rate_reference;
  reconcile matches on the native pair, so the statement supplies the AUD
- medianRateToAud reads statement rows only; a wallet row's amount is itself
  a guess and a 1:1 row would have poisoned the next estimate
- foreign halves pair on the native pair, since several can share amount 0
- currencyFromAmountText reads ISO codes around the figure ("672006 VND")

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 12:43:25 +10:00
siddharthd 5076f27f68 annual charges: surface upcoming card/loan fees from payments alone
ci / lint-test (push) Successful in 56s
Answers "what annual payments are coming?" without touching the email lane --
a bank announces an annual fee on the statement and nowhere else.

WHAT DOES NOT WORK, measured before writing it: "two charges ~365 days apart"
ranks a BAKERY first on this ledger, with 202 such pairs, then a donation,
Patreon, Coles. Frequency is the confound. Tightening to "every gap is annual"
still admits a Target visited twice ($30, then $198).

transaction_type='fee' is the universe, not a description match. The extractor
already classifies fees structurally (131 rows) and every card and loan annual
fee here is typed that way -- the first cut of this filtered
transaction_type='debit' and returned NONE of them. The fees are not debits.

Three confidence tiers, kept apart so a guess never renders like a bank fact:
  confirmed  typed a fee AND repeated on an annual cadence
  likely     typed a fee, seen once, and NAMED annual/card/package -- one charge
             cannot show a cadence but the kind is not in doubt
  possible   no fee wording; merchant whose every gap is annual AND whose amount
             is stable (<=20%, absorbing VicRoads' ~3%/yr drift). Catches Amazon
             Prime and rego; also where the false positives live, hence the label.

expected_not_seen is the more interesting status: the estimate passed and
nothing landed. account_days_stale is the discriminator -- Westpac at 0 days is
trustworthy, ANZ at 392 is simply missing data. Reporting those as upcoming
would be a lie.

account_links (migration 0038) fixes account continuity. Citi Prestige
5258...0253 became NAB MyCard 1015...6227 -- same card, same 14th-to-13th cycle,
no gap -- so the $700 fee charged 2024-12-13 and 2025-11-13 read as two dead
one-offs instead of one due again ~Nov 2026. Hand-asserted with source_documents
ids as evidence rather than inferred: a contiguous-cycle heuristic would join two
genuinely different accounts as readily as one renamed one. Display resolves to
the CURRENT bank; naming the retired brand sends you to the wrong bank.

The GET takes its own DASH_READ_TOKEN with no fallback to WALLET_INGEST_TOKEN --
that one unlocks wallet ingest and the estate-wide reconcile sweep, and
home-dashboard's compose states the rule: it never holds a write credential.

Known limit: the merchant signal fragments on label drift (VicRoads paid as
VICROADS then VICROADS ONLINE PAYM). Not fuzzy-matched, per the rejected pg_trgm
merchant grouping.

339 integration + 196 unit green, tsc clean. Board #265.
2026-08-23 21:44:17 +10:00
siddharthd 10cf0aaf21 transactions: merge/supersede a duplicate by hand, and widen the picker band
ci / lint-test (push) Successful in 55s
Two gaps, found because SP PEAKDESIGN was sitting duplicated with no way to
resolve it.

1. THE PICKER COULD NOT OFFER THE RIGHT ANSWER. The manual reconcile queue's
   candidate band was `amount * 0.99 .. 1.01`. A foreign card purchase is
   charged the notified amount PLUS the card's foreign transaction fee -- 3% on
   the Westpac Altitude -- so a wallet row and its statement line can never
   match exactly: notified $42.44, charged $43.71, x1.03. The one row that WAS
   its twin was the single row outside the band. Widened to +-5%. Safe because
   the list is capped at 5 by the rn window and ordered by amount closeness, so
   a looser band changes which five appear, never how many, and a person
   confirms each. Measured on the live queue: candidate counts 0 -> 0 on all 12
   rows, i.e. no new noise. The auto-matcher stays exact.

2. THERE WAS NO MANUAL MERGE OR DELETE AT ALL. `superseded_by_id` was read
   everywhere (analytics and auto-reconcile both exclude it) and written
   NOWHERE in application code -- set exactly once, by migration 0023.

POST /api/transactions/<id>/merge takes {other_id, mode}: `reconcile` folds a
statementless row into its statement twin, `supersede` handles two rows on the
same footing (a statement imported twice). Deliberately NO delete -- children
are ON DELETE CASCADE, so deleting destroys the splits, tags and metadata that
make a row legible. Both modes keep the row and merely stop it counting.

Both move curation onto the survivor FIRST, through one extracted
moveCurationWithin() rather than a second copy: reconcile.ts's own header warns
that a duplicate would eventually move a different set, and a split stranded on
a hidden row stops counting in silence.

Refuses, all tested: hiding a statement row behind a statementless one (the
bank's record is the side that survives -- burying it under a wallet row leaves
the ledger holding the pre-fee amount); either side already reconciled or
superseded (409, since re-pointing strands the curation moved on the first
pass); burying under an already-superseded row; self-merge; wrong mode for the
pair; someone else's row (ownership via COALESCE(t.owner_id, s.owner_id),
because a statement row carries owner_id on the statement).

UI: a collapsed "Mark as duplicate..." section at the bottom of the edit modal,
debounced search over the existing transactions endpoint, statement-vs-
statementless badges, mode chosen automatically and stated in plain language,
and the wrong-direction case explained with the button disabled rather than
letting it 400.

Also: resetDB now truncates transaction_audit -- the 0037 triggers fire on every
insert the suite makes, so without it the test DB grew ~1,100 rows per full run,
forever (1,125 -> 2).

329 integration + 196 unit green, tsc clean, lint error-free, build compiles.
Board #260.
2026-08-23 19:00:57 +10:00
siddharthd 2251060ac6 db: append-only change log for the ledger (migration 0037)
ci / lint-test (push) Successful in 57s
`transaction_audit` records every INSERT/UPDATE/DELETE on transactions,
transaction_splits, transaction_tags, transaction_overrides and
expense_metadata, via AFTER row triggers.

WHY: on 2026-08-23 the n8n CC Statement Processor stamped `statement_id` onto
seven statementless wallet/pantry rows without setting `reconciled_with_id`,
double-counting $245.90 and hiding those rows from the reconcile queue
(`WHERE statement_id IS NULL`). Answering "was anything lost?" required
restoring the 02:30 pg_dump into a scratch database and diffing it column by
column, because nothing in the estate recorded what changed.

Trigger rather than application code because the ledger has at least five
independent writers -- this app (Prisma), n8n (raw SQL), pantry-app, the
ingestion runner (psycopg) and ad-hoc `docker exec psql`. That bug was an n8n
raw-SQL UPDATE; an app-level audit would not have seen it.

Not Postgres' own logs: wal_level=replica, wal_keep_size=0, archive_mode=off,
log_statement=none, track_commit_timestamp=off, no pgaudit -- verified live.
Postgres was recording nothing usable, and has no built-in audit tables.

UPDATEs store only the columns that changed (a whole-row copy buries a
one-column edit, and line_items can be kilobytes); no-op UPDATEs write nothing;
txid groups the rows of one logical edit, which matters because
applyReconcileMatch moves four children in a single transaction. No FK to
transactions -- the log must outlive the row. TRUNCATE does not fire row
triggers, so the test suite's wipes are silent.

Rehearsed against a scratch restore first: changed-column-only diffing, no-op
suppression, composite PK (transaction_tags), and the
matched_transaction_id fallback (a card-settled order has only that) all
verified before applying. Applied to personal and personal_test. Board #259.
2026-08-23 15:57:44 +10:00
siddharthd 2c05f10aed wallet+intake: adopt the row that is already there, and keep a gift card out of the queue
ci / lint-test (push) Successful in 53s
Three defects found by the 2026-08-18 Woolworths shop reaching the ledger
twice — txn 6858 (pantry receipt, 23 items) and txn 6927 (wallet) were one
$103.30 purchase, and the second parked in the reconcile queue waiting for a
statement line the first had already claimed.

The design says signals for a purchase arrive in sequence — wallet first,
receipt second, statement last — and the receipt lane honours it
(findAdoptableTransaction). Nothing else did.

1. The wallet writer inserted unconditionally. It now adopts a row a receipt
   got to first (findWalletAdoptableTransaction). Deliberately NOT the same
   predicate as the receipt direction: that one refuses a row carrying line
   items, because for a receipt such a row is a different purchase sharing an
   amount; here an itemised row IS the receipt being folded into. Losing that
   guard means the merchant supplies it, so adoption requires merchant
   agreement — a tap that vanishes into an unrelated purchase is worse than a
   visible duplicate. Idempotent across a prompt_version replay via a
   wallet_evt:<id> metadata flag, since an adopted row keeps no wallet
   source_ref of its own.

2. intake_create booked a second row for a tap the wallet lane had already
   recorded. matchTransaction cannot see those rows — its key includes the
   platform and a notification has none — so an Uber Eats screenshot never
   matched the wallet row for the very tap that paid for it. That is the
   NORMAL arrival order, not an edge case.

3. A gift-card tap was written payment_method 'card', which means "a statement
   line is coming"; the queue and findAutoMatches filter on exactly that, so
   it parked forever. The instrument's name survives only in the notification
   title ("Visa Gift Card") — the interpreter reduces it to issuer "Visa",
   which a plain Visa debit card also produces — so notif_title is now carried
   through and a stored-value tap is booked 'credits'. 3 of the first 21 rows.

Also: pantry reports "this receipt is already on my board" in the 200 body and
the forwarder dropped the field, so the one gate that DID fire on the duplicate
was silent (captures 66/69). It now becomes the card. And ambiguity — two
ledger rows for one shop, the exact state behind this bug — no longer reads as
"no match, finance will book it".
2026-08-22 18:10:08 +10:00
siddharthd 9b59edcf48 reconcile: machine auth for the scheduled sweep, across all owners
ci / lint-test (push) Successful in 46s
I built the auto-reconcile endpoint and then left nothing calling it —
exactly the gap I had criticised reconcilePendingOrders for. Found by
tracing the pipeline end to end rather than by anything failing.

PATCH now takes two callers. A person in the browser reconciles their own
queue via the session; the scheduled sweep has no session, carries the
machine token, and sweeps EVERY owner with a queue. That distinction is
load-bearing: 169 of the 177 rows outstanding belong to the second
participant, so a sweep scoped to the session user would leave almost all
of the queue untouched.

A wrong machine token does not fail the request outright — it falls
through to session auth, so the all-owner sweep stays gated while a
browser caller is unaffected. Verified live: 403 with a bad token and no
session, dry-run OK with the real one.

3 new tests. Two test-harness bugs fixed while adding them: the request
stub had no `headers` at all, and the statement helper reused one account
number, so two owners collided on uq_statement_identity rather than
modelling two real cards.
2026-08-22 16:50:40 +10:00
siddharthd 3d0d75519a wallet: the deterministic parse must beat the model's location
ci / lint-test (push) Successful in 46s
Found by reading the backfilled rows rather than by a test. The precedence
was subtitle -> MODEL -> descriptor, so the model's answer won wherever it
had one, which is always. That defeats the entire point of parsing the
descriptor positionally: the ledger got "SAN FRANCISOUS" for the only
foreign merchant in the capture window, plus the padded "MELBOURNE   AU",
and the bare "Sydney"/"Perth" with their country codes dropped.

Correct order is subtitle -> deterministic descriptor -> model. After the
fix those rows read SAN FRANCISO US, MELBOURNE AU, Sydney AU, Perth AU,
and a subtitle-derived place still wins where one exists.

The test that should have caught this passed because its fixture omitted
`location`, so the fallback chain was never exercised — every real
interpretation carries one. Fixture now includes the model's actual
mangled answer for that row, so the test fails if the order regresses.

Re-ran the backfill on the corrected code: 21 transactions, $6,284.37
unchanged, 13 with a location. The 21 previously-written rows were deleted
first rather than superseded, which is normally forbidden on transactions
— checked before doing it that they carried zero splits, tags and
overrides, being machine-written minutes earlier by this same lane with
nothing yet attached.
2026-08-22 16:33:56 +10:00
siddharthd 8d1875225f wallet: a transaction for every tap (board #252, ING-11)
ci / lint-test (push) Successful in 52s
Step 3, and the last app-side piece. POST /api/wallet/ingest takes an
interpreted notification and writes a statementless card transaction plus
one expense_metadata row. Sibling to /api/orders/ingest and
/api/receipts/ingest, same shared-secret shape, its own token so the
wallet credential rotates without touching the order flow.

Nothing here knows about statements, and that is the point: statementless
+ payment_method 'card' is exactly the reconcile queue's predicate, so a
wallet row enters it unaided and the auto-matcher confirms it later.

THE PAIR IS MERGED, NEVER DEDUPED. One tap emits two notifications ~2-8s
apart: the issuer app has the card tail, Apple Wallet has the merchant and
the suburb, and neither alone is complete. Matched on amount and date
because the two share no identifier. My first version COALESCEd every
field, which made the result depend on arrival order — the issuer half
usually lands first, so the truncated all-caps "WYNDHAM VAL AU" beat the
subtitle's "Wyndham Vale, VIC". A subtitle-derived place now overwrites; a
descriptor-derived one only fills a gap. The test that caught this asserts
both arrival orders.

CURRENCY IS NEVER ASSUMED. Only Amex states one ("A$5,103.49"); Westpac,
CommBank and Apple Wallet write a bare "$", which is 14 of 21 captured
rows. Recorded as unknown and flagged, not assumed AUD — harmless at home,
a 23x error on a THB tap abroad. When a currency IS stated and is not AUD,
the NATIVE figure goes to foreign_currency_amount, because the AUD amount
is the bank's own conversion and a notification cannot predict it, while
the native pair is exact on both sides and is what the statement will
match on. transactions.amount is NOT NULL so it takes a provisional
conversion from the ledger-derived median rate, flagged aud_provisional,
overwritten at reconcile. No rate history means no_rate_reference rather
than an invented number.

wallet-descriptor.ts moves the positional parsing out of the model, which
handled ONE fixed-width format three ways on the same day's data: kept the
country code on "WYNDHAM VAL AU", stripped it on "Sydney" and "Perth", and
swallowed "SAN FRANCISOUS" whole — that being "SAN FRANCISO" truncated to
the 12-char city field plus "US", on the only foreign merchant in the
entire capture window. Every unit-test input is a real captured string.

14 unit + 12 integration tests. npm run test:all green (196 + 304), lint
clean.
2026-08-22 16:24:28 +10:00
siddharthd 8e66e527ae receipts: adopt the wallet row instead of creating a second one (board #252)
ci / lint-test (push) Successful in 51s
Step 2 of wallet-first ingestion. Signals for one purchase arrive in
sequence, not in competition: the notification within seconds of the tap,
the receipt hours or days later. Until now the receipt path always
INSERTed its own transaction, so the same shop became two rows.

The case this is built from is measured, not hypothetical. Woolworths
$103.30 on 2026-08-18: the wallet event landed 09:00:14 on the 18th and
nothing created a transaction until 20:25:06 on the 19th -- 35h25m later,
by this path, from scratch -- discarding what only the notification knew
(the instrument, and the suburb).

findAdoptableTransaction() is deliberately as narrow as findAutoMatches():
exact amount, a few days either side, unclaimed, and UNIQUE. Two
candidates means two plausible purchases, and guessing between them moves
one shop's line items onto another shop's row. It also refuses a row that
already carries line items, because a fully-described row is a different
purchase that happens to share an amount. Returning null is always safe --
the caller falls back to creating its own row, which is the old behaviour.

On adoption, both the transaction UPDATE and the expense_metadata UPDATE
have a column list that IS the ownership boundary, following
order_transaction_bridge's UPGRADE pass: the receipt read the register
tape so it wins on what was bought, and `location` is structurally ABSENT
from the update rather than conditionally skipped, so the notification's
suburb cannot be erased by a bug here. expense_metadata.transaction_id is
UNIQUE, so an adopted row that already has metadata is enriched in place
rather than raising.

5 new integration tests including both refusals. Existing receipt tests
unchanged and passing. npm run test:all green.
2026-08-22 16:17:38 +10:00
siddharthd f31f12799a reconcile: automatic pass for unambiguous statement matches (board #252)
ci / lint-test (push) Successful in 56s
Prerequisite for the wallet lane writing a transaction per tap. The queue
is worked by hand and already ~177 deep; wallet adds ~63 rows a month,
~767 a year. Filtering those rows back out is explicitly ruled out by the
awaitsStatementLine() removal note: "an imported row that has no statement
belongs in the queue; volume is solved upstream, never downstream by
hiding what is there."

The rule is deliberately stricter than the queue's own candidate ranking
(top 5 within +/-1% and +/-3 days, for a human to choose between). A
machine gets one shot and no judgement, so: exact amount, never a
tolerance band; and unique BOTH ways, so two identical $13.50 bakery taps
in one window are refused rather than paired arbitrarily. Foreign rows
match on the native pair, because `amount` is the bank's own conversion
and a wallet row cannot predict it.

VALIDATED AGAINST REAL DECISIONS, and that overturned my reasoning. I
first required both sides to be non-foreign, arguing a currency mismatch
is ambiguous. Checked against the 48 reconciliations a human has actually
made: that version matched ZERO of them. All 48 are exactly the shape it
refused — a manual row carrying no currency, matched on exact AUD against
a statement row that does carry one (EUR, CHF) — because overseas receipts
and orders record what was charged in AUD, and only the wallet lane
records natively. Reverted; the rule now reproduces all 48 and proposes
nothing they did not. The coincidence it was meant to guard against is
already covered by unique-both-ways.

Extracted the merge into src/lib/reconcile.ts so the manual route and the
automatic pass cannot drift — there must be exactly one implementation of
something that moves curation off a row reconciliation is about to hide.

Fixed a latent bug in that merge while extracting it: splits were
recreated WITHOUT `settled`/`settled_at`, and the column defaults to
false, so reconciling a settled split silently marked a paid debt owed
again. No queue row carries one today, but this path is about to run
unattended, where that surfaces as money rather than as an exception.

PATCH ?dry=1 reports without writing. Each pair is its own transaction, so
one bad pair leaves the rest untouched and in the queue.

11 new integration tests, most of them about refusal. Live dry-run
proposes 0 pairs for either owner, which is correct: nothing in the queue
has an exact statement twin yet. npm run test:all green (182 + 287).
2026-08-22 16:13:28 +10:00
siddharthd 5e355c7e85 currency: normalise foreign_currency_code to ISO-4217 (board #254)
ci / lint-test (push) Successful in 49s
Exact-code equality is the whole basis of FX reconciliation —
order_transaction_bridge matches on foreign_currency_code = %(ccy)s — so a
row labelled 'BAHT' could never match a THB order and a row labelled
'null' could never match anything. 226 of 1,632 coded rows (14%) were
unmatchable for this reason alone, silently degrading a shipped job.

187 rows carried the literal 4-character STRING 'null' rather than SQL
NULL, 42 of those also carrying foreign_currency_amount = -1.00 exactly.
Root cause is a truthy guard in the CC processors' Parse node —
't.foreign_currency_code || null' passes the string "null" straight
through, because a non-empty string is truthy. Not historical: 155 rows in
2026-03 but also 8 in May, 3 in July and 21 in August. Fixed at source in
the same change (smarthome cc-statement-processor-paperless.json).

Mappings were verified against IMPLIED RATE, not trusted from the label,
because a row whose rate contradicts its currency is a broken ledger row
rather than a mislabelled one — the trap order_transaction_bridge's rate
guard exists for. All six agreed: BAHT 0.0433 vs THB 0.0470, INDIAN RUPEE
0.0183 vs INR 0.0180, U.A.E. DIRHAM 0.4287 vs AED 0.4184, US DOLLAR 1.4054
vs USD 1.5513, NZ DOLLAR 0.8243 vs NZD 0.9311, EURO 1.6342 vs EUR 1.6964.
foreign_currency_amount is untouched by the relabels — only the code was
ever wrong, so this is a relabel, never a recompute.

Applied row counts matched the pre-flight exactly: 42, 187, 13, 10, 6, 4,
4, 2. Postcondition asserted inside the transaction: 0 non-ISO codes
remain. Verified after: 0 'null' strings, 0 -1.00 sentinels, 1445 coded
rows (1632 - 187).
2026-08-22 16:03:06 +10:00
siddharthd 669f9119b9 enrichment readers: resolve one metadata row, deterministically (board #253)
ci / lint-test (push) Successful in 53s
Four readers of expense_metadata that only the 1:1 UNIQUE constraints keep
correct, found while designing wallet-first ingestion. Each fails silently
rather than erroring, so none of them would announce itself.

merchantVerdict (order-reviews.ts) was a genuine fan-out: a plain JOIN on
transaction_id OR matched_transaction_id, no per-transaction cap, feeding
`for (const r of rows) counts[r.rating] += 1`. One review joined to N
metadata rows is counted N times, inflating the loved/never tallies that
drive the pre-order warning — and its LIMIT 50 applies to the fanned-out
set, so duplicates would also push older reviews out of the window. Now a
LATERAL, so the merchant matched is the merchant the UI shows.

nudgeState (slack/interactive) had no LIMIT and no ORDER BY, feeding
queryRow, which returns rows[0]. Postgres guarantees no ordering, so the
merchant and currency on every Slack card refresh could flip at random.
It was the only "resolve both links" site missing the convention its
siblings all carry.

getSharedTransactions had LATERAL LIMIT 1 with no ORDER BY — bounded, but
arbitrary, and free to disagree with the main transaction list about the
same transaction.

transaction_search (migration 0035): NOT the fan-out I first claimed. It
joined only on em.transaction_id, which is UNIQUE, so it could not
multiply rows and sums over it were never inflated. The real defect is
narrower and worse: it OMITTED matched_transaction_id, so every
card-settled order was invisible to it — 148 transactions (Glovo Milan,
Uber Eats, all source='order-bridge') reporting NULL payment_method /
order_reference / line_items while the data sat in the table. This view is
the agent read surface (Hermes' MCP selects those columns from it) and
home-dashboard reads it via dash_ro, so the omission was estate-wide.
Broadening the join is what introduces fan-out risk — transaction_id and
matched_transaction_id are two SEPARATE unique indexes, so two different
rows can legally point at one transaction — hence LATERAL rather than a
second OR. Verified before applying: 4760 rows unchanged, debit sum
unchanged at 1514412.89, only the 148 rows differ. Grants re-granted and
re-verified, since CREATE OR REPLACE VIEW drops them.

Also fixes an integration assertion 258fe3b left stale: it changed the
needs-header card text and updated the unit tests but not intake.test.ts.
The new assertion checks both halves of needsHeaderCard — that it names
what is actually missing, and that a non-in-store source gets the
share-more-of-the-screen advice.

npm run test:all: 182 unit + 276 integration, all passing.
2026-08-22 15:57:31 +10:00
siddharthd 258fe3b617 intake: the needs-header card names what is missing and gives the right advice for it
ci / lint-test (push) Successful in 49s
A Woolworths eReceipt SCREENSHOT has no date anywhere on it — store, total,
items, payment, savings, not one date (2026-08-22, batch 60ed68be…) — while
the app's own Share button sends the raw receipt with the date line. The card
said 'share the top of the screen too', which is delivery-app advice and sends
the person back for a shot that cannot exist. Now: the missing field by name,
and for a dateless in-store receipt, the Share button or pantry's own upload.
2026-08-22 12:35:09 +10:00
siddharthd 0823244e4a intake: the word "Image" is not a receipt, and the prompt must not be data
ci / lint-test (push) Successful in 52s
Batch 3d2ffbe8 arrived as one text part whose entire content was the five
characters `Image` — an iOS Shortcut coerced the shared photo to text. The
model was asked anyway and answered out of its prompt: Burgertory
(Braybrook), two "The Pride" lines at $22.80, a $41.15 total. Nothing on it
existed. Burgertory was in ORDER_ITEM_PROMPT as the example of how to print
a branch in parentheses. Only the missing date kept it out of the matcher;
the invented header and items were stored on the batch regardless.

Three layers, none of them a model:

1. textCouldBeReceipt() — deterministic, before any model call. A TEXT part
   with fewer than two words, no digit, or under 40 characters and no money
   figure cannot be a receipt. Length is NOT the test on its own: `TOTAL
   $17.00` is twelve characters and is exactly what this door accepts. A
   rejected part is dropped like an unreadable file type, so one junk part
   beside a good screenshot still goes through; if nothing usable is left
   the card quotes what the phone actually sent. PDFs and images untouched.

2. The prompt names no real shop — merchant examples are
   PROMPT_PLACEHOLDER_MERCHANTS, exported from intake-rules so the prompt
   and the guard cannot drift — and it is told what to return when the input
   is not a receipt at all.

3. confabulationReason() — the belt. Reachable where the pre-check cannot
   look: a PDF that renders to a scrap of junk still reaches the model. Also
   rejects any answer whose merchant IS a placeholder.

All three file `unsupported`, never `needs_header`. A confabulated header is
not a partial one, and the needs_header path stores what it is given.

Same shape as the ingestion engine's 2026-08-05 empty-body defect: a stage
that can be handed nothing has to say so, because the model downstream will
always answer.

Tests: 16 unit cases over the two pure functions and the prompt; three
integration cases — `Image` is declined with zero model calls and nothing
but `unsupported` on the batch row, a junk part beside a real receipt still
enriches, and a junk PDF is unsupported rather than needs_header. The first
and third fail on the tree before this commit.
2026-08-22 11:46:05 +10:00
siddharthd d88ca2a24c intake: a duplicate card that refuses a purchase must say what to do instead
ci / lint-test (push) Successful in 1m5s
The header (merchant, date, total) is a purchase's identity here, so a genuine
second identical purchase — two of the same coffee, one shop, one day — reads
as a re-share and is refused. That default is right; a re-share is far commoner
than a repeat. But the card said only "Nothing was changed", which is a dead
end: the money is real and the ledger silently never gets it. Say to add it in
the app.
2026-08-21 18:27:53 +10:00
siddharthd 91586ce4a4 test(intake): the second review pass, and CLAUDE.md for the rules it set
ci / lint-test (push) Successful in 45s
Six of the nine fixes have a test that fails against the previous tree: the
metadata row's currency, the Create-path hold, two concurrent batches booking
once, mrkdwn escaping including plain_text, the escaped currency, and
candidate-row currency.

Three do not, and the reason is the same each time — the window needs a crash
or a lost race that a test cannot force. Asserted instead: a stale metadataId
falls back to the live row rather than reporting a phantom write; two writes
racing to create a metadata row both resolve to one; the create path records a
baseline at all. All three passed before as well.
2026-08-21 18:23:52 +10:00
siddharthd 4b09d4136f intake: lock the purchase, not the batch, and hold on both write paths
Two different unresolved batches for one purchase could each create a
transaction. ON CONFLICT (source, source_ref) cannot stop it — their source_refs
differ — and both taps clear their match and duplicate checks before either
inserts. The create now takes a transaction-scoped advisory lock keyed on
(platform, merchant, date, total), re-checks inside it, and inserts; the loser
reports `duplicate` against the winner's transaction, the same answer the
sequential path gives.

The lock has to be inside a real transaction, which this app did not have.
queryRaw is $queryRawUnsafe on a POOLED client, so every call is its own
implicit transaction and pg_advisory_xact_lock taken in a statement of its own
is released before the next statement runs — the lock would have looked right
and held nothing. withTransaction() in db.ts is the scope; the integration
helper mirrors it with BEGIN/COMMIT on one pooled connection.

Also: the "hold rather than replace an itemised receipt with an unpriced
upload" guard lived only in routeFood, and createFromBatch's re-match walked
straight past it — the same upload destroyed the same receipt depending on
which button reached it. One function, both callers.
2026-08-21 18:23:51 +10:00
siddharthd 4f0f0dd285 intake: escape the currency and the button label too
money() interpolated the extracted currency raw, and parseExtraction accepts
any string for it — so "<@U123>" would have been a live mention on every card
the batch produced. And plain_text disables *formatting*, not entity decoding:
&, < and > are control characters everywhere in a Slack payload, so the Create
button's label is escaped like any other value we did not write. My earlier
note claiming the opposite was wrong.

Candidate rows also carry their transaction's currency (NATIVE_CURRENCY) rather
than always printing "$" under a header reading "USD 22.40".
2026-08-21 18:23:51 +10:00
siddharthd b788b4e8fa line-items: snapshot before writing, and stop reporting writes that did not happen
Three faults on the same function, all of them "the write is assumed to have
worked".

- The create path snapshotted the undo baseline AFTER its INSERT. A crash in
  between left the retry recording the intake's own write as "previous", so
  undo restored the thing it was meant to reverse — the exact failure
  previous_line_items exists to prevent. Both paths now snapshot first.
- An UPDATE against a row deleted between the select and the write affects
  nothing and still returned a metadata_id, so the batch was marked enriched
  against a row nobody had written to. The write is now checked and falls
  through to the ordered lookup, then to a create.
- Two calls for one transaction could both find no row and both INSERT;
  expense_metadata.transaction_id is UNIQUE, so one lost with a constraint
  error that reached the user as an error card. ON CONFLICT (transaction_id)
  DO UPDATE, with (xmax = 0) so created_metadata stays truthful.

And the new row's currency comes from the batch instead of a hard-coded 'AUD'.
OrderDetails prefers expense_metadata.currency, so a USD order — booked
correctly on the transaction with foreign_currency_code — printed "$".
2026-08-21 18:23:32 +10:00
siddharthd b5569b8ccc schema: previous_line_items on the model; say what the finish_reason rule is
Migration 0034 has the column and the Prisma model did not, so a regenerated
client and the database disagreed.

The extraction comment claimed only a finished reply is accepted while the code
accepts an absent finish_reason. The code is right — failing every call on a
provider that omits an optional field would be a certain outage in exchange for
a case that also requires syntactically complete JSON, and the harm a truncated
reply does is closed on the write side. The prose now states the rule as it is:
a reason that is PRESENT must be `stop`.
2026-08-21 18:23:32 +10:00
siddharthd d4ff68363a expense_metadata: every reader resolves the row a write lands on
getTransactions' provenance LATERAL, merchantForTransaction and the Slack
details modal all select on transaction_id = $1 OR matched_transaction_id = $1
with a bare LIMIT 1, so they could read a different row than replaceLineItems
writes. Same ORDER BY as the match, the write and the order-detail GET.

Defensive, not a live bug: measured 2026-08-21, expense_metadata holds 667
rows, 147 of them pointing-only, and ZERO transactions resolve to more than one
row by either link. This changes nothing today. It is here so the readers still
agree on the day multiplicity appears, which is the only condition under which
any of them was ever wrong.
2026-08-21 18:23:31 +10:00
siddharthd 631e75e58f test(intake): cover the review findings, and CLAUDE.md for the rules they set
ci / lint-test (push) Successful in 59s
Unit: word-boundary merchant matching (Bigals Pizza is not an IGA), mrkdwn
escaping vs the plain_text button label, a foreign total on a card, and the
finish_reason gate.

Integration: the NULL-platform wildcard on both the food and grocery paths,
candidates crossing platforms, EXCLUDE_RECONCILED_SOURCE, the pinned metadata
row, the panel reading the row the write landed on, the once-per-batch undo
baseline, the priceless-upload hold, Create re-matching, two pending cards
booking once, a pantry-resolved sibling, and a USD receipt.

Eleven of the thirteen fail against the pre-fix tree; the exceptions are the
no-batch-id baseline guard (a no-change assertion) and the concurrent-tap one,
which is a race and passed either way.
2026-08-21 17:53:06 +10:00
siddharthd c25db5211b intake: a NULL platform is "no platform", and Create re-checks before it inserts
Six fixes in the router, all of them cases where a stale or over-wide answer
was written rather than re-asked.

- ($3 IS NULL OR em.platform = $3) made an in-store receipt a wildcard: a till
  receipt from the Coles tap matched the Uber Eats Coles order for the same
  day and money, and overwrote its items. In-store now matches only rows with
  no platform of their own.
- findCandidates loses its platform filter entirely. That card is a human
  choice, shown precisely because the exact key missed — the platform is one
  of the things that may be wrong, so the list must widen there, not narrow.
- createFromBatch re-runs the match. The card is minutes old by the time it is
  tapped and the email/order bridge books orders on its own schedule; the
  match it was drawn from is not current.
- ...and checks for a resolved sibling batch. findDuplicateBatch ignores
  unresolved batches by design, so two pending cards for one order each booked
  it.
- INSERT ... ON CONFLICT (source, source_ref) DO NOTHING with a re-select,
  rather than SELECT-then-INSERT, which loses a double tap.
- A non-AUD receipt is booked as order-ingestion.ts books one: amount native,
  amount_aud NULL, foreign_currency_* set. No FX rate exists at intake.

Also: EXCLUDE_RECONCILED_SOURCE in both queries instead of a bare
superseded_by_id check — a reconciled manual twin at the exact key read as an
ambiguity, which is the same spend counted twice. And routeFood refuses to
replace an itemised receipt with an extraction carrying no prices at all: the
reconciliation check passes those (there is nothing to check), so without this
a priceless upload silently destroys a good record. It holds instead.

RouteContext.subtotal was written and never read; it is replaced by currency,
which the cards need.
2026-08-21 17:53:06 +10:00
siddharthd 9a442b7355 intake: escape mrkdwn, and stop printing every total as dollars
Merchant names, transaction descriptions and model output are none of ours:
Slack reads &, < and > as markup, so "Ben & Jerry's" and a description
carrying <http://…|text> rendered as something the card never meant to show.
esc() on every interpolated value in mrkdwn — never on our own literal *bold*,
and never on a button's plain_text label, where entities would print verbatim.

money() also hard-coded "$" while the order nudge already prints "USD 15.00".
The cards now carry the batch's currency.
2026-08-21 17:52:47 +10:00
siddharthd f9638cc5e5 line-items: one metadata row, agreed on by the write and the read
Three defects on one surface, all invisible while a transaction has exactly
one expense_metadata row — which is almost always.

1. ORDER BY (transaction_id = $1) DESC did the opposite of its comment.
   Postgres orders DESC as NULLS FIRST, and the row that merely POINTS at the
   transaction has a NULL transaction_id, so the clause written to prefer the
   owned row preferred the other one — in exactly the case it exists for.
   matchTransaction already spelled out NULLS LAST; now so do this and
   GET /api/transactions/[id]/order, which had no ORDER BY at all and could
   read a different row from the one an enrichment had just written.

2. replaceLineItems takes an optional metadataId. The match resolves a row
   through a platform-filtered join; re-picking independently here can land
   the write on a row the match ruled out.

3. previous_line_items on intake_batches is the undo baseline, recorded once
   per batch under COALESCE and read back. Returning whatever was just read
   hands a pantry retry its own write as the "previous" state, so undo
   restores the thing it undid. Migration 0034 is unapplied anywhere, so the
   column is added there rather than in a 0035.
2026-08-21 17:52:47 +10:00
siddharthd 7424d13858 intake: only a reply that finished is a reply the schema constrained
liteLLMExtractor rejected finish_reason "length" and accepted everything else,
so a content_filter or tool_calls stop was parsed as if the strict json_schema
had held. Reject any reason present and not "stop", naming it.
2026-08-21 17:52:25 +10:00
siddharthd be3d10d4f1 intake: match merchants on whole words, not substrings
merchantKey("Bigals Pizza") contains "iga", so GROCERY_PATTERN filed a pizza
shop as a grocer and merchantMatches matched it against every IGA on the same
date and total. Short names in a substring list collide with ordinary words,
and the collision reads as a correct classification rather than a bug.

GROCERY_PATTERN is now a word-boundary alternation; merchantMatches asks
whether one name's words appear as a contiguous run inside the other's, which
keeps Coles <-> Coles (Wyndham Vale) and Order - Coles (Wyndham Vale).

Also collapses the third PLATFORM_LABEL copy: order-ingestion's was exported
and read nowhere, order-details.tsx had its own, and the intake one — the only
live copy — was missing `uber`. One map, in intake-rules.
2026-08-21 17:52:25 +10:00
siddharthd a4e5f540cc docs: intake router routes, env vars, migration 0034
ci / lint-test (push) Successful in 48s
2026-08-21 12:10:06 +10:00
siddharthd 32a47bf8ee test(intake): rules, PDF text, C2 idempotency, enrich-vs-candidates, pantry hand-off, Slack taps
Unit: merchant pattern, header validation, reconciliation tolerance, verbatim
line items, one-call message shape, card action ids; unpdf over the two real
Uber Eats exports (Coles $85.74 / 15 lines, Burgertory $22.40 / Michael +
Joel). Integration: fetch stubbed at the LiteLLM and pantry boundaries so the
PDF path, persistence, match and writes run for real against personal_test.
intake-live.test.ts is opt-in (INTAKE_LIVE_TEST=1) and was run green against
cloud-budget on both PDFs.
2026-08-21 12:10:06 +10:00
siddharthd c7ea516c34 slack: dispatch the four intake taps on action_id (plan §11 C5)
intake_use_candidate / intake_create / intake_grocery / intake_meal carry JSON
{batch_id, ...} rather than <id>:<verb>, so they branch before the order-nudge
parse. They act on the persisted batch: enrich the chosen transaction, create
ING-10-style (statement_id NULL, owner_id from shared_by, source/source_ref
make a second press a no-op), or remember the merchant's kind and route. A tap
on an already-resolved batch returns its current card and writes nothing. Card
updates go to response_url, as the existing handlers do.
2026-08-21 12:10:05 +10:00
siddharthd db5c9a02a7 intake: POST /api/intake router + PUT /api/transactions/[id]/line-items (plan §11 C1, C2)
One upload point, every applicable step. Persist the batch, classify each
file, ONE LiteLLM call (cloud-budget, temperature 0, strict json_schema; PDFs
as unpdf text, never pixels) with an order-item prompt that keeps names
verbatim, header (merchant, date, total), merchant pattern
(coles|woolworths|woolies|aldi|iga|foodworks|costco) -> grocery | food | ask,
exact (platform, merchant, date, total) match against transactions +
expense_metadata, then act:

- food + unique match  -> replace line_items via the C2 path, 'enriched'
- food + miss/ambiguity -> 'candidates' (same merchant ±3d) with
  intake_use_candidate / intake_create buttons; never auto-creates
- grocery              -> original files to pantry POST /api/capture with
  finance_target_transaction_id + finance_action enrich|create (C3);
  409 -> 'duplicate'
- ask / needs_header / held / duplicate / unsupported as replies, not writes

Reconciliation is against the printed item subtotal (a delivery total is
items + fees - promotions: 18.94 of Coles lines settled at 5.74),  or
2%, else 'held'. Originals are kept under INTAKE_FILE_DIR by sha256 so a later
'Groceries' tap can still forward them.

C2 is an idempotent replace on transaction_id = :id OR
matched_transaction_id = :id, accepts INTAKE_TOKEN or RECEIPT_INGEST_TOKEN,
creates the expense_metadata row when absent, 404 when the transaction does
not exist, returns {previous_line_items, line_items} so pantry's undo is the
same call with the previous list.
2026-08-21 12:10:05 +10:00
siddharthd a538376638 db(0034): intake_batches + intake_merchant_kinds for the intake router
One row per upload through POST /api/intake, written BEFORE extraction so a
Slack tap minutes later has something to act on and a re-share (same sha256
set, or the same (platform, merchant, date, total) header already resolved)
is detected. Columns are the §11 C1 contract in smarthome
plans/intake-router-slice-plan.md. intake_merchant_kinds holds the remembered
'Groceries or a meal?' answer per normalised merchant (plan §2, Codex #6).

Not applied to production — finance has no migrate service; apply by hand at
deploy, then npm run test:setup.
2026-08-21 12:10:05 +10:00
56 changed files with 6101 additions and 246 deletions
+319 -1
View File
@@ -65,7 +65,7 @@ docker logs finance -f
npm test # unit — pure logic, no database npm test # unit — pure logic, no database
npm run test:integration # database-backed; resolves the DSN itself (see below) npm run test:integration # database-backed; resolves the DSN itself (see below)
npm run test:all # both npm run test:all # both
npm run test:setup # once, and after any migration: creates/refreshes personal_test npm run test:setup # once, and after any migration: drops and rebuilds personal_test
npm run lint # blocking in CI since #79 — 0 errors, warnings allowed npm run lint # blocking in CI since #79 — 0 errors, warnings allowed
npx tsc --noEmit # blocking in CI; nothing else compiles the test files npx tsc --noEmit # blocking in CI; nothing else compiles the test files
``` ```
@@ -86,6 +86,13 @@ had pointed the suite at **postgres-pantry** — whose database it would have
TRUNCATEd had the credentials happened to match. A loopback address cannot be TRUNCATEd had the credentials happened to match. A loopback address cannot be
reassigned to another container, which is why the published port wins. reassigned to another container, which is why the published port wins.
**`test:setup` really rebuilds now (2026-10-03).** It used to replay the
schema dump over the existing database, where every `CREATE TABLE` failed with
"already exists" and the old shape survived — so no migration ever reached
`personal_test`, and new-column code failed every test with a misleading
`column does not exist`. It now `DROP DATABASE … WITH (FORCE)`s first, refusing
any name not ending `_test`, and loads with `ON_ERROR_STOP`.
**Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to **Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to
truncate anything whose `current_database()` is not `personal_test`, and the truncate anything whose `current_database()` is not `personal_test`, and the
suite runs `fileParallelism: false` because every file shares that one database suite runs `fileParallelism: false` because every file shares that one database
@@ -176,6 +183,215 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
- `statements` — one row per billing period per bank account - `statements` — one row per billing period per bank account
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent - `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
### The change log (migration 0037)
`transaction_audit` is an **append-only** row-level record of every change to
`transactions`, `transaction_splits`, `transaction_tags`, `transaction_overrides`
and `expense_metadata`. Never `UPDATE` or `DELETE` from it. There is deliberately
**no foreign key** to `transactions` — the log has to outlive the row it describes.
```sql
-- everything that ever happened to one transaction
SELECT changed_at, table_name, op, old_data, new_data, app_name
FROM transaction_audit WHERE transaction_id = 6961 ORDER BY changed_at;
-- one logical edit, all the rows it touched (a reconcile moves 4 children at once)
SELECT * FROM transaction_audit WHERE txid = <txid> ORDER BY id;
```
**Why it exists.** On 2026-08-23 the n8n CC Statement Processor stamped
`statement_id` onto seven statementless wallet/pantry rows without setting
`reconciled_with_id`, double-counting $245.90 and hiding those rows from the
reconcile queue (`WHERE statement_id IS NULL`). Answering *"was anything lost?"*
took restoring the 02:30 `pg_dump` into a scratch database and diffing it column
by column, because nothing recorded what changed.
**Why it is a trigger and not application code.** The ledger has at least five
independent writers — this app (Prisma), n8n (raw SQL in a Postgres node),
pantry-app, the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`.
That bug was an n8n raw-SQL `UPDATE`; an app-level audit would not have seen it.
**Why not Postgres' own logs.** Checked live: `wal_level=replica` (WAL cannot be
decoded into row changes), `wal_keep_size=0` and `archive_mode=off` (recycled
almost immediately), `log_statement=none`, `track_commit_timestamp=off`, no
`pgaudit`. Postgres records nothing usable and has no built-in audit or temporal
tables.
Three details worth knowing before relying on it:
- **`old_data`/`new_data` hold only the columns that actually changed** on an
`UPDATE`. A whole-row copy would bury a one-column edit, and
`expense_metadata.line_items` alone can be kilobytes.
- **A no-op `UPDATE` writes nothing** (the upsert paths issue plenty).
- **Attribution is partial and that is understood.** Every client connects as the
same `personal` role, so `db_user` never discriminates. `app_name` is only as
good as what each client sets; `client_addr` separates containers and is NULL
for `docker exec` over the unix socket — itself a useful signal. `txid` is the
reliable grouping key.
`TRUNCATE` does not fire row-level triggers, so the integration suite's table
wipes produce no audit rows.
### Resolving a duplicate by hand (`POST /api/transactions/<id>/merge`)
`<id>` is the row that gets **hidden**; `other_id` is the one that **survives**.
| mode | requires | sets | for |
|---|---|---|---|
| `reconcile` | loser statementless, survivor a statement row | `reconciled_with_id` | the everyday case |
| `supersede` | nothing | `superseded_by_id` | two rows on the same footing, e.g. a statement imported twice |
**There is no delete, deliberately.** Every child of `transactions` is
`ON DELETE CASCADE`, so removing a row destroys the splits, tags and metadata
that make it legible. Both modes keep the row and its children and merely stop
it counting. Do not add a delete endpoint.
Both modes **move curation onto the survivor first**, through the single
`moveCurationWithin()` in `lib/reconcile.ts` — because both hide the source row
from every figure, and a split stranded on a hidden row stops counting in
silence. That is also why the two callers share one implementation: the file
header warns that a second copy would eventually move a different set.
Refusals (all deliberate, all tested in `merge-supersede.test.ts`): a row already
reconciled or superseded on **either** side → 409 (re-pointing it strands the
curation moved on the first pass); superseding under a row that is itself
superseded → 409; a row that is not the caller's → 404. Ownership is checked with
`COALESCE(t.owner_id, s.owner_id)` — a statement row carries `owner_id` on the
statement, so checking `t.owner_id` alone rejects every statement row as "not
found".
**Why it had to exist.** `superseded_by_id` was read everywhere (analytics and
auto-reconcile both exclude it) but written **nowhere** in application code — set
exactly once, by migration 0023. The reconcile queue was the only merge path, and
it only offers candidates inside an amount band.
**The band is ±5%, not ±1% (2026-08-23).** A foreign card purchase is charged the
notified amount **plus the card's foreign transaction fee** — 3% on the Westpac
Altitude — so a wallet row and its statement line can never match exactly.
`SP PEAKDESIGN` was notified at **$42.44** and charged **$43.71** (×1.03), which
put the one row that *was* its twin outside the old band: the picker could not
offer the only correct answer. Widening is safe because the list is capped at 5
by the `rn` window and ordered by amount closeness, so a looser band changes
*which* five appear, never how many, and a person confirms each. Measured on the
live queue at the time of the change: candidate counts went **0 → 0** on all 12
rows, i.e. no new noise. **The auto-matcher stays exact** — see `findAutoMatches()`.
### A charge that never posted (`POST /api/transactions/<id>/void`, migration 0040)
The third way a row stops counting, for a case with **no survivor**: a card
notification is sent for an *authorisation*, and when the merchant releases the
hold no statement line ever arrives. Txn 7951 — a $1,040.41 Airbnb hold on Amex,
booking cancelled the next day — sat in the reconcile queue with nothing to
match and counted as spend everywhere. `reconcile` needs a statement twin,
`supersede` needs a survivor (and `transactions_no_self_supersede` blocks
pointing a row at itself), so it gets its own columns: `voided_at` +
`void_reason` (reason mandatory, by CHECK). Body `{reason}`; `DELETE` undoes it.
UI: **Never posted** on every row of `/reconcile`.
- **In `EXCLUDE_RECONCILED_SOURCE`**, so every "count each purchase once" query
drops it without being edited; the queue, `findAutoMatches`, both adoption
helpers, annual charges and the manual reconcile POST filter it explicitly.
- **Statementless only** — `transactions_void_statementless` enforces it in the
database. A statement line is the bank's record of a posted charge; a wrong
one is a supersede.
- **Refused** on a reconciled row (it posted), a superseded row, and a row with a
*settled* split (someone already paid their share — voiding would quietly turn
that into a credit). `merge` refuses a voided row on either side.
- **The wallet lane deliberately still SEES voided rows.** `alreadyIngested()` and
the issuer/Wallet pairing in `wallet-ingestion.ts` do not filter `voided_at`:
the voided row is the sink that stops a late twin notification, or the next
`prompt_version` re-post, from booking the hold again. Same for the intake
router's idempotency lookup. Do not "fix" those.
- **Outside this repo, nothing filters it**: Hermes' `agent_ro` views
(`cashflow_monthly` etc.) and ingestion-engine's `instalment_matcher` already
ignore `superseded_by_id`/`reconciled_with_id` too, so a voided row is no worse
there than any other hidden row.
### Upcoming annual charges (`GET /api/annual-charges`, migration 0038)
Card annual fees, loan package fees and annual subscriptions, derived from
**payments alone** — independent of the email lane, because a bank announces an
annual fee on the statement and nowhere else.
**`transaction_type = 'fee'` is the universe, not a description match.** The
extractor already classifies fees structurally (131 rows), and *every* card and
loan annual fee on this ledger is typed that way. The first version of this
filtered `transaction_type = 'debit'` and returned **none** of them. The fees are
not debits.
**Two signals, kept apart because they are not equally trustworthy:**
| confidence | basis |
|---|---|
| `confirmed` | typed a fee **and** repeated on an annual cadence |
| `likely` | typed a fee, seen once, and *named* annual/card/package — the kind is certain, the cadence inferred |
| `possible` | no fee wording; a merchant whose every gap is annual **and** whose amount is stable |
**A naive "two charges ~365 days apart" is useless** — measured on this ledger it
ranks a bakery first with **202** such pairs, then a donation, Patreon, Coles.
Frequency is the confound. Even "every gap is annual" admits a Target visited
twice ($30, then $198). The `possible` tier therefore also requires amount
stability (≤20%, which absorbs VicRoads' ~3%/yr CPI drift) and ≥2 observations.
It still carries real false positives (H&M, Superdry) — that is what the label is
for, and why it must never render like a bank fact.
**`expected_not_seen` is the more interesting status.** The estimate passed and
nothing landed, which means the account stopped being imported, or closed, or the
charge arrived under a different label. `account_days_stale` is the
discriminator: Westpac at 0 days stale is trustworthy; ANZ at 392 is simply
missing data. Never report these as upcoming.
**Account continuity (`account_links`, migration 0038).** One card can appear
under two identities: Citi Prestige `5258…0253` (to 2025-11-13) became NAB MyCard
`1015…6227` (from 2025-11-14), same 14th-to-13th cycle, no gap. Without the link
the $700 fee reads as two dead one-offs instead of one due again ~Nov 2026. The
table is **hand-asserted with evidence** (`source_documents` ids), not inferred —
a contiguous-cycle heuristic would join two genuinely different accounts as
readily as one renamed one. The resolver walks it forward recursively;
`uq_account_links_from` forbids a fork and `account_links_no_self` a self-link, so
it terminates. **Display the resolved current bank**, not the one that printed the
last fee — naming the retired brand sends you to the wrong bank.
Known limitation: the merchant signal fragments on label drift. VicRoads rego was
paid 2024/2025 as `VICROADS` and 2026 as `VICROADS ONLINE PAYM`, so the series
splits and the 2025 estimate shows as `expected_not_seen`. Not fuzzy-matched on
purpose — see the rejected pg_trgm merchant grouping.
### The wallet lane (`/api/wallet/ingest`, `lib/wallet-ingestion.ts`)
iOS notifications → `capture.wallet_events` → n8n **Wallet Interpret**
(`CAp1s5cuzJk2mDaI`, local gemma, every 15 min) → this route. A row per tap,
statementless, `source='wallet'`, `source_ref='evt:<id>'`; the statement later
confirms it through the reconcile queue. Fixed 2026-10-03 after a month abroad
(ticket #266); the rules below are each the fix for something that broke.
- **Storage convention differs from the order lane.** `amount` = `amount_aud` =
a *provisional AUD* figure; the native figure is `foreign_currency_amount`.
`NATIVE_CURRENCY` reads `amount_aud IS NOT NULL` as "amount is AUD" — without
that an INR 524 tap displayed as "INR 9.42".
- **Provisional rate:** ledger median over *statement* rows (≥5, carries the
card's FX fee) → else ECB via Frankfurter (`rate_ecb`) → else `0` with
`no_rate_reference`. Never the native figure as AUD (672006 VND once became
A$672,006).
- **One tap = two notifications** (issuer app + Wallet, 2–8 s apart), merged.
A pair must be **≤10 minutes apart AND from different apps**
(`expense_metadata.notified_at` / `notif_app`, migration 0039). The old
amount-within-±1-day rule merged the UNICEF $50 into an unrelated $50 charge.
- **The whole check→pair→adopt→insert runs under one advisory lock**
(`hashtextextended('wallet-ingest',0)`), on the transaction's own connection.
n8n posts a poll's items concurrently; without it 11 taps became 22 rows.
Prisma cannot deserialise `void` — the lock SELECT needs `::text`.
- **A merged half records `wallet_evt:<id>` in the survivor's flags**, so a
replay is idempotent. Reconciled wallet rows carry **no** `expense_metadata`
(reconcile moves it to the statement twin) — the flag check follows it there.
- **Skipped, not booked:** account notices (`overdrawn`, `to avoid fees`, …) and
transit-card drawdowns (`Your new/current balance is` — the Octopus *load* is
the spend). The fixed-width descriptor parser runs on Westpac bodies only.
- **Dated in Australia/Melbourne**, like the statements (UTC slicing filed 28 of
134 rows a day early).
- **Replay a few events by relabelling their interpretations**
(`UPDATE capture.wallet_interpretations SET prompt_version='v1-<why>' WHERE …`),
not by bumping `prompt_version`, which re-posts every event ever captured.
### Cash and reconciliation ### Cash and reconciliation
`payment_method` in (`cash`, `credits`) excludes a transaction from `payment_method` in (`cash`, `credits`) excludes a transaction from
@@ -622,6 +838,108 @@ and one tap must not flatten it. It says which case it refused on.
Slack reaches this route through an n8n webhook, not directly — see the Slack reaches this route through an n8n webhook, not directly — see the
smarthome repo's CLAUDE.md and the `slack-interactive-via-n8n` memory. smarthome repo's CLAUDE.md and the `slack-interactive-via-n8n` memory.
### Intake router — one upload, every applicable step (2026-08-21)
`POST /api/intake` (token `INTAKE_TOKEN`, multipart: `source`, `shared_by`,
`batch_id`, 1–6 `file`s) is the finance half of the share-sheet intake; design
and the cross-repo contract are in smarthome `plans/intake-router-slice-plan.md`
§11 (C1, C2, C5 live here; C3/C4 are pantry-app; C6 is n8n + compose).
- **Order of operations is the plan's and is load-bearing:** persist
`intake_batches` row → classify each file → ONE LiteLLM call for the whole
batch (`cloud-budget`, `temperature 0`, strict `json_schema`; PDFs go in as
`unpdf` text, never pixels) → header `(merchant, date, total)` → merchant
kind → duplicate check → reconciliation → match → act. `src/lib/intake-rules.ts`
holds the pure decisions, `intake-router.ts` the orchestration.
- **Match is `(platform, merchant, date, total)`, exact, never widened.** A
miss or an ambiguity returns candidates (same merchant ±3 days) with
`intake_use_candidate` / `intake_create` buttons; a transaction is created
only on the explicit tap, ING-10-style (`statement_id NULL`, `owner_id` from
`shared_by` → `participants.email`, else `DEFAULT_OWNER_ID`;
`source='intake'`, `source_ref=batch_id` so a second press is a no-op —
written as `ON CONFLICT … DO NOTHING`, because a SELECT-then-INSERT loses a
double tap). A non-AUD receipt is booked the way `order-ingestion.ts` books
one: `amount` native, `amount_aud` **NULL**, `foreign_currency_*` set — no FX
rate exists at intake and inventing one is worse than admitting there is none.
- **A NULL platform means "no platform", not "any platform".** An in-store tap
(`coles`, `woolworths`, `other`) matches only rows whose `expense_metadata`
carries no platform of its own. The wildcard reading let a till receipt match
the Uber Eats order for the same shop on the same day and overwrite its items.
The **candidates** query is the opposite and deliberately so: it has no
platform filter at all, because the card is a human choice shown precisely
because the exact key missed — the platform is one of the things that may be
wrong, so the list must widen there, not narrow.
- **The Create tap re-runs the match before it inserts**, and checks for a
resolved sibling batch. The card is minutes old by the time it is pressed and
the email/order bridge books orders on its own schedule; `findDuplicateBatch`
ignores *unresolved* batches by design, so two pending cards for one order
would otherwise each book it.
- **One `expense_metadata` row, agreed on by every query that resolves one.** A
transaction can own a row *and* be the `matched_transaction_id` of another;
`matchTransaction` returns the row it resolved through and `replaceLineItems`
writes to exactly that one. Every reader orders by
`(transaction_id = $1) DESC **NULLS LAST**, id` — the match, the write, the
order-detail GET, `getTransactions`' provenance LATERAL, `merchantForTransaction`
and the Slack details modal. The `NULLS LAST` is load-bearing: Postgres orders
`DESC` as `NULLS FIRST` and the row that merely *points* at the transaction has
a NULL `transaction_id`, so the clause meant to prefer the owned row preferred
the other one. Measured 2026-08-21: 667 rows, 147 pointing-only, **zero**
transactions resolving to more than one row — the ordering changes nothing
today and exists so the readers still agree when multiplicity appears.
- **The create is guarded by a transaction-scoped advisory lock on the
PURCHASE**, not the batch. `ON CONFLICT (source, source_ref)` handles a double
tap on one card; two *different* batches for one purchase have different
`source_ref`s and both clear their match and duplicate checks before either
inserts. The lock has to live inside `withTransaction` (`src/lib/db.ts`):
`queryRaw` is `$queryRawUnsafe` on a **pooled** client, so every call is its
own implicit transaction and `pg_advisory_xact_lock` taken in a statement of
its own is released before the next statement runs. The loser reports
`duplicate` pointing at the winner's transaction — the same answer the
sequential path gives.
- **`replaceLineItems` snapshots the undo baseline BEFORE it writes, on both
paths**, and treats an UPDATE that touched no row as a failure rather than a
success. The metadata INSERT is `ON CONFLICT (transaction_id) DO UPDATE` and
carries the batch's **currency** — a hard-coded `'AUD'` made `OrderDetails`
print `$` on a USD order, because it prefers `expense_metadata.currency`.
- **`intake_batches.previous_line_items` is the undo baseline, written once.**
Returning whatever the write just read makes a pantry retry get its own write
back as the "previous" state, so undo restores the thing it undid. Recorded
under `COALESCE(previous_line_items, …)` and read back.
- **Grocery never writes finance here.** The ORIGINAL files go to pantry
`POST /api/capture` with `intake.finance_target_transaction_id` +
`finance_action: enrich|create`; pantry's confirm calls
`PUT /api/transactions/[id]/line-items` back (C4), which is why that route
accepts `RECEIPT_INGEST_TOKEN` too. Files are kept under `INTAKE_FILE_DIR`
keyed by sha256 so a later "Groceries" tap can still forward them.
- **Reconciliation is against the printed item subtotal, not the total.** A
delivery order's total is items + fees − promotions ($118.94 of Coles lines
became $85.74), so checking lines against the total would hold every
delivery order. Tolerance $1 or 2%; a failure is `held`, nothing written.
- **Nothing that cannot be a receipt reaches the model, and the prompt names no
real shop.** On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so
a batch arrived as the five characters `Image` — and the model returned a
whole receipt: `Burgertory (Braybrook)`, two "The Pride" lines, $41.15. None
of it existed; "Burgertory" was in `ORDER_ITEM_PROMPT` as the example of how
to print a branch in parentheses. Only the absent date kept it out of the
matcher, and the invented header and items were still stored on the batch as
`needs_header`. Three layers now, in order: `textCouldBeReceipt()` rejects a
TEXT part with fewer than two words, no digit, or under 40 characters with no
money figure (`TOTAL $17.00` passes — length alone is *not* the test)
**before** any model call; the prompt's merchant examples are
`PROMPT_PLACEHOLDER_MERCHANTS` format placeholders and it is told to return
nulls for a non-receipt; and `confabulationReason()` catches what the
pre-check cannot see — a PDF that renders to junk — plus any answer echoing a
placeholder. All three file `unsupported`, never `needs_header`: a
confabulated header is not a partial one. Same shape as the ingestion
engine's 2026-08-05 empty-body defect — **a stage that can be handed nothing
must say so, because the model downstream always answers, and it answers out
of its prompt.**
- **Names stay verbatim.** Burgertory calls its burgers "Michael" and "Joel";
the prompt forbids canonicalising and the stored shape is the order lane's
(`qty`, `description`, `amount`, `options`) so `order-details.tsx` renders it.
- Statuses beyond the contract list: `ask` (the "Groceries or a meal?" holding
state), `held`, `error` (502 — model or pantry down; the batch row says so).
### The shared loan ### The shared loan
The loan is a **separate ledger**, not a shared expense and not a settlement The loan is a **separate ledger**, not a shared expense and not a settlement
+14
View File
@@ -282,6 +282,18 @@ All routes require authentication via `X-Forwarded-User` header (set by Traefik)
| GET / POST | `/api/tags` | List or create tags | | GET / POST | `/api/tags` | List or create tags |
| PATCH / DELETE | `/api/tags/[id]` | Update or delete a tag | | PATCH / DELETE | `/api/tags/[id]` | Update or delete a tag |
**Machine routes** — shared-secret `x-ingest-token` header instead of `X-Forwarded-User`; each fails closed when its token is unset:
| Method | Route | Token | Description |
|--------|-------|-------|-------------|
| POST / PATCH | `/api/orders/ingest` | `ORDER_INGEST_TOKEN` | n8n order receipts → transactions (see smarthome CLAUDE.md "Order Processor") |
| POST | `/api/receipts/ingest` | `RECEIPT_INGEST_TOKEN` | pantry-app confirmed grocery receipts → one transaction per tender leg (ING-10) |
| POST | `/api/intake` | `INTAKE_TOKEN` | **Intake router** (2026-08-21): multipart `source` (`uber_eats\|doordash\|coles\|woolworths\|other`), `shared_by`, `batch_id` (uuid), 1–6 `file`s (image / PDF / text). Persists `intake_batches`, extracts once via LiteLLM `cloud-budget` (PDF → text via `unpdf`), matches `(platform, merchant, date, total)` exactly, then enriches the matched order's `line_items` (food), forwards the original files to pantry `POST /api/capture` (grocery), or replies with candidates / "Groceries or a meal?" / needs-header / held / duplicate. Returns `{kind, status, transaction_id?, capture_id?, candidates?, slack_blocks, text}` |
| PUT | `/api/transactions/[id]/line-items` | `INTAKE_TOKEN` or `RECEIPT_INGEST_TOKEN` | Idempotent replace of `expense_metadata.line_items` for `transaction_id = :id OR matched_transaction_id = :id`; body `{line_items, source: 'intake'\|'pantry', batch_id?, platform?, replace: true}`; creates the metadata row if absent, never a transaction (404). Returns `{previous_line_items, line_items}` — pantry calls it on every confirm and with `previous_line_items` on undo |
| POST | `/api/slack/interactive` | `ORDER_INGEST_TOKEN` + Slack signature | Order-nudge buttons and the intake taps (`intake_use_candidate`, `intake_create`, `intake_grocery`, `intake_meal`) |
Env for the intake router: `INTAKE_TOKEN`, `LITELLM_BASE_URL` + `LITELLM_KEY` (OpenAI-compatible, alias `cloud-budget`), `PANTRY_BASE_URL` (+ the existing `RECEIPT_INGEST_TOKEN` as pantry's machine auth), optional `INTAKE_FILE_DIR` (original files by sha256, default `/app/data/intake` — a later "Groceries" tap re-sends them) and `FINANCE_PUBLIC_URL` (the card's View link, default `https://finance.bosecamp.com`). Design: smarthome `plans/intake-router-slice-plan.md` §11.
--- ---
## Ingestion Pipeline ## Ingestion Pipeline
@@ -347,6 +359,8 @@ docker exec postgres-personal psql -U personal -d personal \
| `0009_split_payments` | `split_payments` | | `0009_split_payments` | `split_payments` |
| `0010_csv_import_reconcile` | `reconciled_with_id`, CSV import support | | `0010_csv_import_reconcile` | `reconciled_with_id`, CSV import support |
| `0011_trips` | `trips`, `trip_id` on `transaction_overrides` (catch-up — was applied directly) | | `0011_trips` | `trips`, `trip_id` on `transaction_overrides` (catch-up — was applied directly) |
| … | |
| `0034_intake_batches` | `intake_batches` (one row per `/api/intake` upload, written before extraction) and `intake_merchant_kinds` (the remembered "Groceries or a meal?" answer per merchant) |
> `paperless_doc_id` on statements and the `uq_statements_paperless_doc_id` index were added directly (not tracked in a migration file). > `paperless_doc_id` on statements and the `uq_statements_paperless_doc_id` index were added directly (not tracked in a migration file).
> `owner_id` on transactions and `statement_id` made nullable were applied directly (March 2026) to support manual transaction entry without a fake statement. > `owner_id` on transactions and `statement_id` made nullable were applied directly (March 2026) to support manual transaction entry without a fake statement.
@@ -32,10 +32,20 @@ CREATE TABLE IF NOT EXISTS intake_batches (
status text NOT NULL, status text NOT NULL,
transaction_id integer, transaction_id integer,
capture_id integer, capture_id integer,
-- The undo baseline: what `expense_metadata.line_items` held before this
-- batch first replaced it. Recorded here rather than returned from the write,
-- because pantry re-calls the enrich endpoint on every confirm and after a
-- timeout — a caller that hands back whatever it just read gives a retry its
-- OWN write as the "previous" state, and undo then restores the thing it was
-- meant to reverse. Written under COALESCE, so only the first snapshot sticks.
previous_line_items jsonb,
created_at timestamptz NOT NULL DEFAULT now(), created_at timestamptz NOT NULL DEFAULT now(),
resolved_at timestamptz resolved_at timestamptz
); );
-- For a database that already ran an earlier draft of this file.
ALTER TABLE intake_batches ADD COLUMN IF NOT EXISTS previous_line_items jsonb;
COMMENT ON TABLE intake_batches IS COMMENT ON TABLE intake_batches IS
'One row per upload batch through POST /api/intake. Persisted before ' 'One row per upload batch through POST /api/intake. Persisted before '
'extraction so Slack taps can act on it later and re-shares are detected. ' 'extraction so Slack taps can act on it later and re-shares are detected. '
@@ -0,0 +1,73 @@
-- transaction_search: resolve expense_metadata by BOTH links, without fanning out.
--
-- THE DEFECT (measured 2026-08-22): the view joined only on
-- `em.transaction_id = t.id`, so it silently omitted every card-settled order,
-- which carries only `matched_transaction_id`. 148 transactions are affected --
-- Glovo orders in Milan, Uber Eats, all `source='order-bridge'` -- and for each
-- of them the view reports NULL payment_method / order_reference / line_items
-- while the data sits in the table.
--
-- That is not cosmetic. This view is the agent read surface: Hermes'
-- postgres-personal MCP selects payment_method and order_reference from it
-- (personal-agent-gateway/mcp-servers/postgres-personal/queries.py), and
-- home-dashboard reads it via dash_ro. CLAUDE.md states the rule the view broke:
-- "That lookup must match transaction_id OR matched_transaction_id: a
-- card-settled order has only the latter."
--
-- NOT a fan-out fix, contrary to a first reading. The original join CANNOT fan
-- out: expense_metadata_transaction_id_key is UNIQUE on transaction_id, so it
-- matched at most one row by construction. Sums taken over this view were never
-- inflated.
--
-- But broadening the join is exactly what INTRODUCES that risk, which is why
-- LATERAL is used rather than a second OR'd condition. transaction_id and
-- matched_transaction_id are enforced by two SEPARATE unique indexes
-- (expense_metadata_transaction_id_key, uq_expense_matched_txn), so one
-- transaction can legally be pointed at by two DIFFERENT rows -- one via each
-- link. Zero such transactions exist today; the LATERAL means that stays a
-- non-event rather than a silent duplicate in the agent's spending totals.
--
-- Tie-break matches every other reader (getTransactions, replaceLineItems,
-- matchTransaction, /api/transactions/[id]/order), so the row this view resolves
-- is the row the UI shows.
--
-- Verified before applying: row count unchanged at 4760 (= count(transactions)),
-- sum(amount) over debits unchanged at 1514412.89 -- amount comes from
-- transactions, never from the metadata row -- and the only difference is the
-- 148 rows that gain the metadata they always should have had.
--
-- CREATE OR REPLACE VIEW DROPS GRANTS. dash_ro (home-dashboard) and agent_ro
-- (Hermes) both read this view and losing either breaks a surface with no error
-- at replace time, so both are re-granted below.
CREATE OR REPLACE VIEW transaction_search AS
SELECT t.id,
t.transaction_date,
t.description,
COALESCE(t.merchant_normalized, t.merchant_name) AS merchant,
t.category,
t.transaction_type,
COALESCE(t.amount_aud, t.amount) AS amount,
t.foreign_currency_amount,
t.foreign_currency_code,
t.location,
s.bank_name,
s.card_name,
s.account_number,
s.statement_type,
s.billing_end_date,
em.payment_method,
em.order_reference,
em.line_items
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN LATERAL (
SELECT em2.payment_method, em2.order_reference, em2.line_items
FROM expense_metadata em2
WHERE em2.transaction_id = t.id OR em2.matched_transaction_id = t.id
ORDER BY (em2.transaction_id = t.id) DESC NULLS LAST, em2.id
LIMIT 1
) em ON true;
GRANT SELECT ON transaction_search TO dash_ro;
GRANT SELECT ON transaction_search TO agent_ro;
@@ -0,0 +1,73 @@
-- Normalise foreign_currency_code to ISO-4217, and clear two sentinel values.
--
-- WHY THIS MATTERS: exact-code equality is the whole basis of FX reconciliation.
-- ingestion-engine's order_transaction_bridge matches foreign orders with
-- AND t.foreign_currency_code = %(ccy)s
-- AND t.foreign_currency_amount = %(total)s
-- so a row labelled 'BAHT' can never match a THB order, and a row labelled
-- 'null' can never match anything. 226 of 1,632 coded rows (14%) were
-- unmatchable for this reason alone, which silently degrades a shipped job.
--
-- THE SENTINELS. 187 rows carry the literal 4-character STRING 'null' rather
-- than SQL NULL, and 42 of those also carry foreign_currency_amount = -1.00
-- exactly. Neither is data: they are "no value" markers that survived a truthy
-- guard. Root cause is in the CC statement processors' Parse node, which does
-- foreign_currency_code: t.foreign_currency_code || null
-- and a non-empty string is truthy, so a model that answers with the STRING
-- "null" (rather than JSON null) passes straight through. Same for -1. Fixed
-- workflow-side in the same change; this migration cleans what already landed.
-- Not purely historical -- 155 rows in 2026-03, but also 8 in May, 3 in July
-- and 21 in August 2026.
--
-- MAPPINGS WERE VERIFIED AGAINST IMPLIED RATE, NOT TRUSTED FROM THE LABEL,
-- because a row whose rate contradicts its claimed currency is a broken ledger
-- row rather than a mislabelled one (the trap order_transaction_bridge's rate
-- guard exists for: 2 of its first 73 matches were broken rows, not bad
-- matches). Median amount/foreign_currency_amount per raw code vs the ISO
-- median over statement rows (n>=5):
-- BAHT 0.0433 vs THB 0.0470 INDIAN RUPEE 0.0183 vs INR 0.0180
-- U.A.E. DIRHAM 0.4287 vs AED 0.4184 US DOLLAR 1.4054 vs USD 1.5513
-- NZ DOLLAR 0.8243 vs NZD 0.9311 EURO 1.6342 vs EUR 1.6964
-- Every label agrees with its rate. foreign_currency_amount is untouched by the
-- relabels -- only the code was ever wrong, so this is a relabel, never a
-- recompute.
--
-- EXPECTED ROW COUNTS (verified pre-flight, includes superseded rows):
-- code 'null' -> NULL 187 BAHT -> THB 13
-- amount -1.00 -> NULL 42 U.A.E. DIRHAM -> AED 10
-- NZ DOLLAR -> NZD 6 US DOLLAR -> USD 4
-- INDIAN RUPEE -> INR 4 EURO -> EUR 2
-- Pre-flight also confirmed: no -1.00 amount exists outside the 'null'-code
-- rows, and no other non-ISO code exists.
BEGIN;
-- Sentinels first: the -1.00 amounts live only on 'null'-code rows, so clear
-- them before that code is nulled and the set becomes unfindable.
UPDATE transactions SET foreign_currency_amount = NULL
WHERE foreign_currency_amount = -1.00 AND foreign_currency_code = 'null';
UPDATE transactions SET foreign_currency_code = NULL
WHERE foreign_currency_code = 'null';
-- Word-form relabels.
UPDATE transactions SET foreign_currency_code = 'THB' WHERE foreign_currency_code = 'BAHT';
UPDATE transactions SET foreign_currency_code = 'AED' WHERE foreign_currency_code = 'U.A.E. DIRHAM';
UPDATE transactions SET foreign_currency_code = 'NZD' WHERE foreign_currency_code = 'NZ DOLLAR';
UPDATE transactions SET foreign_currency_code = 'USD' WHERE foreign_currency_code = 'US DOLLAR';
UPDATE transactions SET foreign_currency_code = 'INR' WHERE foreign_currency_code = 'INDIAN RUPEE';
UPDATE transactions SET foreign_currency_code = 'EUR' WHERE foreign_currency_code = 'EURO';
-- Postcondition: every remaining code is ISO-4217 shaped. Fails the whole
-- transaction rather than leaving a partial normalisation behind.
DO $$
DECLARE bad int;
BEGIN
SELECT count(*) INTO bad FROM transactions
WHERE foreign_currency_code IS NOT NULL AND foreign_currency_code !~ '^[A-Z]{3}$';
IF bad <> 0 THEN
RAISE EXCEPTION 'normalisation incomplete: % non-ISO codes remain', bad;
END IF;
END $$;
COMMIT;
@@ -0,0 +1,152 @@
-- Append-only change log for the ledger and its curation children.
--
-- WHY THIS EXISTS: on 2026-08-23 the CC Statement Processor's `Run Reconciliation`
-- node stamped `statement_id` onto seven statementless wallet/pantry rows without
-- setting `reconciled_with_id` or superseding the statement's own extracted line.
-- The purchase counted twice, and because finance-app's reconcile queue is
-- `WHERE statement_id IS NULL`, the stamp also hid those rows from the only thing
-- that could repair them. Answering "was anything lost?" required restoring the
-- 02:30 pg_dump into a scratch database and diffing it column by column, because
-- NOTHING in this estate records what changed. That is the gap this closes.
--
-- WHY AT THE DATABASE AND NOT IN THE APP: the ledger has at least five independent
-- writers -- finance-app (Prisma), n8n (raw SQL in a Postgres node), pantry-app,
-- the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`. The 4a bug
-- was an n8n raw-SQL UPDATE; an application-level audit would not have seen it at
-- all. A trigger cannot be bypassed by choice of client.
--
-- WHY NOT POSTGRES' OWN LOGS: checked live on this instance -- wal_level=replica
-- (so WAL cannot be logically decoded into row changes), wal_keep_size=0 and
-- archive_mode=off (so WAL is recycled almost immediately), log_statement=none,
-- logging_collector=off, track_commit_timestamp=off, and pgaudit is not installed.
-- Postgres was recording nothing usable. It has no built-in audit or temporal
-- tables; this always has to be built.
CREATE TABLE IF NOT EXISTS transaction_audit (
id bigserial PRIMARY KEY,
changed_at timestamptz NOT NULL DEFAULT now(),
-- Groups every row touched by one logical edit. `applyReconcileMatch` moves
-- overrides, tags, splits and metadata in a single transaction; without this
-- they read as four unrelated events.
txid bigint NOT NULL DEFAULT txid_current(),
table_name text NOT NULL,
op text NOT NULL CHECK (op IN ('INSERT','UPDATE','DELETE')),
-- Full primary key as an object, because transaction_tags is composite
-- (transaction_id, tag_id) and the others are a bare `id`.
row_key jsonb NOT NULL,
-- Denormalised so "everything that ever happened to transaction N" is one
-- index scan rather than a five-way union.
transaction_id bigint,
-- On UPDATE these hold ONLY the columns that actually changed. Storing whole
-- rows would bury a one-column edit in forty unchanged fields, and
-- expense_metadata.line_items alone can be kilobytes.
old_data jsonb,
new_data jsonb,
-- Attribution is partial and that is understood: every client connects as the
-- same `personal` role. app_name is only as good as what each client sets;
-- client_addr distinguishes containers and is NULL for `docker exec` over the
-- unix socket, which is itself a useful signal.
db_user text NOT NULL DEFAULT current_user,
app_name text,
client_addr inet
);
COMMENT ON TABLE transaction_audit IS
'Append-only row-level change log for transactions + curation children. Never UPDATE or DELETE from this table.';
CREATE INDEX IF NOT EXISTS transaction_audit_txn_time
ON transaction_audit (transaction_id, changed_at DESC);
CREATE INDEX IF NOT EXISTS transaction_audit_time
ON transaction_audit (changed_at DESC);
CREATE INDEX IF NOT EXISTS transaction_audit_txid
ON transaction_audit (txid);
CREATE INDEX IF NOT EXISTS transaction_audit_table_time
ON transaction_audit (table_name, changed_at DESC);
-- TG_ARGV[0]: comma-separated candidate columns for the owning transaction id;
-- the first non-null wins (expense_metadata carries the id in
-- transaction_id OR matched_transaction_id -- a card-settled order
-- only ever has the latter).
-- TG_ARGV[1]: comma-separated primary key columns.
CREATE OR REPLACE FUNCTION log_transaction_change() RETURNS trigger
LANGUAGE plpgsql SECURITY DEFINER
SET search_path = public, pg_temp
AS $fn$
DECLARE
v_full jsonb;
v_old jsonb;
v_new jsonb;
v_key jsonb := '{}'::jsonb;
v_txn bigint;
k text;
BEGIN
IF TG_OP = 'INSERT' THEN
v_new := to_jsonb(NEW);
v_full := v_new;
ELSIF TG_OP = 'DELETE' THEN
v_old := to_jsonb(OLD);
v_full := v_old;
ELSE
v_old := to_jsonb(OLD);
v_new := to_jsonb(NEW);
v_full := v_new;
-- Reduce to the changed columns only. IS DISTINCT FROM so a NULL->NULL
-- comparison counts as unchanged rather than evaluating to NULL.
FOR k IN SELECT jsonb_object_keys(v_old) LOOP
IF (v_old -> k) IS NOT DISTINCT FROM (v_new -> k) THEN
v_old := v_old - k;
v_new := v_new - k;
END IF;
END LOOP;
-- A no-op UPDATE is noise, and finance-app's upserts issue plenty of them.
IF v_old = '{}'::jsonb THEN
RETURN NULL;
END IF;
END IF;
FOREACH k IN ARRAY string_to_array(TG_ARGV[1], ',') LOOP
v_key := v_key || jsonb_build_object(k, v_full -> k);
END LOOP;
FOREACH k IN ARRAY string_to_array(TG_ARGV[0], ',') LOOP
IF v_txn IS NULL AND (v_full ->> k) IS NOT NULL THEN
v_txn := (v_full ->> k)::bigint;
END IF;
END LOOP;
INSERT INTO transaction_audit
(table_name, op, row_key, transaction_id, old_data, new_data, app_name, client_addr)
VALUES
(TG_TABLE_NAME, TG_OP, v_key, v_txn, v_old, v_new,
NULLIF(current_setting('application_name', true), ''), inet_client_addr());
RETURN NULL; -- AFTER trigger; return value is ignored
END
$fn$;
-- AFTER, so a statement that raises never leaves a phantom audit row; the audit
-- write is in the same transaction as the change and rolls back with it.
DROP TRIGGER IF EXISTS trg_audit_transactions ON transactions;
CREATE TRIGGER trg_audit_transactions
AFTER INSERT OR UPDATE OR DELETE ON transactions
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('id', 'id');
DROP TRIGGER IF EXISTS trg_audit_transaction_splits ON transaction_splits;
CREATE TRIGGER trg_audit_transaction_splits
AFTER INSERT OR UPDATE OR DELETE ON transaction_splits
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
DROP TRIGGER IF EXISTS trg_audit_transaction_tags ON transaction_tags;
CREATE TRIGGER trg_audit_transaction_tags
AFTER INSERT OR UPDATE OR DELETE ON transaction_tags
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'transaction_id,tag_id');
DROP TRIGGER IF EXISTS trg_audit_transaction_overrides ON transaction_overrides;
CREATE TRIGGER trg_audit_transaction_overrides
AFTER INSERT OR UPDATE OR DELETE ON transaction_overrides
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
DROP TRIGGER IF EXISTS trg_audit_expense_metadata ON expense_metadata;
CREATE TRIGGER trg_audit_expense_metadata
AFTER INSERT OR UPDATE OR DELETE ON expense_metadata
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id,matched_transaction_id', 'id');
@@ -0,0 +1,76 @@
-- Account continuity across a rebrand, reissue or issuer change.
--
-- WHY THIS EXISTS: one card can appear in `statements` under two completely
-- different identities. The live case is the Citi Prestige card, which moved to
-- NAB and was rebranded MyCard:
--
-- Citibank 5258073330340253 statements ... 2025-11-13
-- NAB 1015001000116227 statements 2025-11-14 ...
--
-- Same 14th-to-13th billing cycle, no gap, one continuous product. Nothing in
-- the schema linked them, so anything grouping by (bank_name, account) sees a
-- dead account and a new one with no history. For the annual-fee detector that
-- is not cosmetic: the $700 fee charged 2024-12-13 and 2025-11-13 reads as
-- "stopped" when it is due again around November 2026 under the new name.
--
-- WHY A TABLE AND NOT AN INFERENCE: contiguous-cycle heuristics would join two
-- genuinely different accounts as readily as one renamed one, and a wrong join
-- merges two real fee series. There are only a handful of these events in a
-- lifetime, each announced by mail, so each link is asserted once with the
-- document that proves it. `evidence` is not decoration -- it is how a future
-- reader checks the claim instead of trusting it.
--
-- NOT `account_owner_mappings`: that answers "whose account is this" and keys on
-- the raw (bank_name, account_number). This answers "is this the same account as
-- that one", and keys on the normalised form.
CREATE TABLE IF NOT EXISTS account_links (
id bigserial PRIMARY KEY,
-- Normalised the same way statements.account_number_key is generated
-- (non-alphanumerics stripped), so these join without per-call cleanup.
-- Amex-style masks keep their X's: they record which digits were redacted.
from_bank_name text NOT NULL,
from_account_key text NOT NULL,
to_bank_name text NOT NULL,
to_account_key text NOT NULL,
-- When the successor took over. The predecessor's last statement and the
-- successor's first are the observable boundary; this is the announced date
-- where mail gives one.
effective_date date,
reason text NOT NULL
CHECK (reason IN ('rebrand', 'reissue', 'issuer_change', 'migration')),
-- Provenance for the claim, e.g. 'source_documents:32919'. Free text because
-- the evidence is not always a document -- sometimes it is a statement pair.
evidence text,
notes text,
created_at timestamptz NOT NULL DEFAULT now(),
-- One successor per predecessor: a chain, never a fork. A fork would make
-- "the current identity of this account" ambiguous, and the resolver walks
-- this forward assuming it is not.
CONSTRAINT uq_account_links_from UNIQUE (from_bank_name, from_account_key),
-- A link to itself would make that walk non-terminating.
CONSTRAINT account_links_no_self CHECK (
from_bank_name IS DISTINCT FROM to_bank_name
OR from_account_key IS DISTINCT FROM to_account_key
)
);
COMMENT ON TABLE account_links IS
'Predecessor -> successor account identity across rebrands/reissues. Hand-asserted, each row carrying the evidence for its claim.';
CREATE INDEX IF NOT EXISTS account_links_to
ON account_links (to_bank_name, to_account_key);
-- The one link the mail record proves today.
-- source_documents 32919 (2025-10-01) "your Citi Prestige Card account is moving to MyCard"
-- source_documents 32618 (2025-11-10) "MyCard is coming soon"
-- source_documents 32539 (2025-11-24) "The new MyCard App and MyCard Online are now live"
-- Statement boundary: Citibank ends 2025-11-13, NAB begins 2025-11-14.
INSERT INTO account_links
(from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason, evidence, notes)
VALUES
('Citibank', '5258073330340253', 'NAB', '1015001000116227', '2025-11-14', 'issuer_change',
'source_documents:32919,32618,32539',
'Citi Prestige -> MyCard Prestige. Billing cycle (14th-13th) continues unbroken across the boundary.')
ON CONFLICT (from_bank_name, from_account_key) DO NOTHING;
@@ -0,0 +1,26 @@
-- When, and from which app, a wallet notification arrived.
--
-- WHY: one tap emits two notifications 2-8 seconds apart -- the issuer app
-- (card tail) and Apple Wallet (merchant, suburb) -- and wallet-ingestion.ts
-- merges them. It used to recognise the pair by amount alone, within +/-1 DAY,
-- which is wide enough to fold two different purchases together: on 2026-10-01
-- an Amex "$50.00 at A MEDIUM" notification merged into the UNICEF $50 donation
-- from 17 hours earlier and renamed it. A real pair is seconds apart and comes
-- from two DIFFERENT apps; two notifications from one app are two charges.
-- Neither fact was stored anywhere on the ledger side, so the matcher could not
-- ask. These columns let it.
--
-- Backfilled from capture.wallet_events through source_message_id ('evt:<id>'),
-- which every wallet-written metadata row carries.
ALTER TABLE expense_metadata
ADD COLUMN IF NOT EXISTS notified_at timestamptz,
ADD COLUMN IF NOT EXISTS notif_app text;
UPDATE expense_metadata em
SET notified_at = e.received_at,
notif_app = e.notif_app
FROM capture.wallet_events e
WHERE em.source = 'wallet'
AND em.source_message_id = 'evt:' || e.id
AND em.notified_at IS NULL;
@@ -0,0 +1,43 @@
-- A statementless row for a charge that never posted.
--
-- WHY: the wallet lane books every card notification as a purchase, and a card
-- notification is sent for an AUTHORISATION, not a settlement. When the
-- merchant releases the hold the bank never posts the charge, so no statement
-- line ever arrives: the row sits in the reconcile queue forever and counts as
-- spend in every figure. Txn 7951: Airbnb HMXAHAW49Y, $1,040.41 held on Amex
-- 1000 on 2026-09-24, booking cancelled 25 Sep inside the free window, hold
-- released ("We removed the $1,040.41 (AUD) payment hold"). Statement 219
-- (11 Sep - 10 Oct) has no such line and reconciles to its header without it.
--
-- Neither existing exclusion fits. reconciled_with_id needs a statement twin
-- and superseded_by_id needs a surviving row (and may not point at itself,
-- transactions_no_self_supersede). There is no survivor: nothing happened.
--
-- Not a DELETE: every child of transactions cascades, and the row is also the
-- sink the wallet lane checks before re-posting an event. Deleting it would let
-- the next prompt_version bump re-create it.
ALTER TABLE transactions
ADD COLUMN IF NOT EXISTS voided_at timestamptz,
ADD COLUMN IF NOT EXISTS void_reason text;
COMMENT ON COLUMN transactions.voided_at IS
'Set when this statementless row records a charge that never posted (e.g. a released authorisation hold). Excluded from every figure and from the reconcile queue, kept for its children and audit trail. NULL = live.';
-- A reason is mandatory: a void with no explanation is indistinguishable from
-- a mistake six months later.
ALTER TABLE transactions
ADD CONSTRAINT transactions_void_has_reason
CHECK ((voided_at IS NULL AND void_reason IS NULL)
OR (voided_at IS NOT NULL AND btrim(coalesce(void_reason, '')) <> ''));
-- Statement rows cannot be voided. A statement line is the bank's own record
-- of a posted charge, and the statement's rows must keep summing to its
-- header; a wrong statement row is a supersede, not a void.
ALTER TABLE transactions
ADD CONSTRAINT transactions_void_statementless
CHECK (voided_at IS NULL OR statement_id IS NULL);
CREATE INDEX IF NOT EXISTS idx_transactions_voided
ON transactions (voided_at)
WHERE voided_at IS NOT NULL;
+8 -1
View File
@@ -190,7 +190,9 @@ model transactions {
owner_id Int? owner_id Int?
reconciled_with_id Int? reconciled_with_id Int?
superseded_by_id Int? superseded_by_id Int?
principal_amount Decimal? @db.Decimal(12, 2) voided_at DateTime? @db.Timestamptz(6) // a charge that never posted (migration 0040)
void_reason String?
principal_amount Decimal? @db.Decimal(12, 2)
interest_amount Decimal? @db.Decimal(12, 2) interest_amount Decimal? @db.Decimal(12, 2)
source String? // feed this row came from, e.g. "frollo"; NULL = statement/manual (migration 0028) source String? // feed this row came from, e.g. "frollo"; NULL = statement/manual (migration 0028)
source_ref String? // the provider's own transaction id — idempotency key for re-imports source_ref String? // the provider's own transaction id — idempotency key for re-imports
@@ -237,6 +239,9 @@ model expense_metadata {
tender_raw String? tender_raw String?
receipt_sha256 String? receipt_sha256 String?
receipt_group String? receipt_group String?
// 0039 — when/where a wallet notification arrived; the pair merge needs both.
notified_at DateTime? @db.Timestamptz(6)
notif_app String?
transaction transactions? @relation(fields: [transaction_id], references: [id], onDelete: Cascade) transaction transactions? @relation(fields: [transaction_id], references: [id], onDelete: Cascade)
@@unique([source, order_reference], name: "uq_expense_source_order") @@unique([source, order_reference], name: "uq_expense_source_order")
@@ -283,6 +288,8 @@ model intake_batches {
status String status String
transaction_id Int? transaction_id Int?
capture_id Int? capture_id Int?
// The undo baseline, written once per batch (see 0034 and line-items.ts).
previous_line_items Json?
created_at DateTime @default(now()) @db.Timestamptz(6) created_at DateTime @default(now()) @db.Timestamptz(6)
resolved_at DateTime? @db.Timestamptz(6) resolved_at DateTime? @db.Timestamptz(6)
} }
+14 -5
View File
@@ -19,14 +19,23 @@ if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
exit 1 exit 1
fi fi
# Create test database (ignore error if it already exists) # Drop and recreate, so the schema is rebuilt rather than replayed. Replaying
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres \ # the dump over an existing database was a silent no-op for every migration:
-c "CREATE DATABASE $TEST_DB;" 2>/dev/null || true # each CREATE TABLE failed with "already exists" and the old shape survived, so
# a new column (0039's expense_metadata.notified_at, 2026-10-03) never reached
# the test database. The guard is because this now DROPs: it refuses anything
# not named *_test, whatever TEST_DB was set to.
case "$TEST_DB" in
*_test) ;;
*) echo "ERROR: refusing to drop '$TEST_DB' -- the test database must be named *_test"; exit 1 ;;
esac
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres -q \
-c "DROP DATABASE IF EXISTS $TEST_DB WITH (FORCE);" \
-c "CREATE DATABASE $TEST_DB;"
# Wipe and rebuild schema from production (schema only, no data)
echo "Copying schema from $PG_USER to $TEST_DB..." echo "Copying schema from $PG_USER to $TEST_DB..."
docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \ docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q | docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q -v ON_ERROR_STOP=1
echo "Schema ready." echo "Schema ready."
@@ -105,6 +105,19 @@ describe("NATIVE_CURRENCY", () => {
expect(unconverted).toBe(true); expect(unconverted).toBe(true);
}); });
it("labels a statement-less wallet row AUD, because its amount already is", async () => {
// Wallet rows store a provisional AUD figure in amount AND amount_aud, and
// the native figure in foreign_currency_amount. Reading the foreign code
// here printed "INR 9.42" for an INR 524 tap (txn 8162).
const id = await scratchTxn(
"transaction_date, description, amount, amount_aud, transaction_type, payment_method, source, source_ref, foreign_currency_amount, foreign_currency_code",
"'2026-10-03','Wallet foreign fixture', 9.42, 9.42, 'debit', 'card', 'wallet', 'evt:fixture-native-ccy', 524.00, 'INR'"
);
const { ccy, unconverted } = await currencyOf(id);
expect(ccy).toBe("AUD");
expect(unconverted).toBe(false);
});
it("prefers the statement's currency over the foreign-charge record", async () => { it("prefers the statement's currency over the foreign-charge record", async () => {
// Opposite convention: on an AUD statement, `amount` is AUD and // Opposite convention: on an AUD statement, `amount` is AUD and
// foreign_currency_code merely notes what was originally charged. Reading // foreign_currency_code merely notes what was originally charged. Reading
@@ -0,0 +1,170 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, mockDbWithPool } from "./helpers";
/**
* Upcoming annual charges from payments alone.
*
* The exclusions carry this file. Anything can be made to find a card annual
* fee; the difficulty is not also reporting the monthly account-servicing fee,
* the per-transfer Wise charge, and the restaurant visited twice a year apart.
* Measured on the real ledger, a naive "two charges ~365 days apart" ranks a
* bakery first with 202 such pairs.
*/
let pool: Pool;
let ownerId: number;
let getUpcomingAnnualCharges: typeof import("../../lib/annual-charges").getUpcomingAnnualCharges;
let acctSeq = 0;
/** A statement plus one transaction on it, for a named account. */
async function charge(
bank: string,
acct: string,
date: string,
description: string,
amount: number,
type = "fee"
) {
const key = `${bank}:${acct}`;
let stId = stmtCache.get(`${key}:${date}`);
if (!stId) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ($4, $1, $2, $3::date - 20, $3::date + 10, $5) RETURNING id`,
[acct, bank, date, `s${acctSeq++}.pdf`, ownerId]
);
stId = st.rows[0].id as number;
stmtCache.set(`${key}:${date}`, stId);
}
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1,$2,$3,$4,$5,$6)`,
[stId, date, description, amount, type, ownerId]
);
}
let stmtCache = new Map<string, number>();
/** A statementless merchant purchase, for the non-fee signal. */
async function purchase(date: string, merchant: string, amount: number) {
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id, merchant_normalized, row_index)
VALUES (NULL,$1,$2,$3,'debit',$4,$2,0)`,
[date, merchant, amount, ownerId]
);
}
const find = <T extends { label: string }>(rows: T[], label: string) =>
rows.find((r) => r.label === label);
beforeAll(async () => {
pool = createPool();
mockDbWithPool(pool);
({ getUpcomingAnnualCharges } = await import("../../lib/annual-charges"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM account_links");
stmtCache = new Map();
({ ownerId } = await seedParticipants(pool));
});
describe("what it finds", () => {
it("confirms a card annual fee that repeated a year later", async () => {
await charge("American Express", "1000", "2025-06-10", "ANNUAL CARD FEE", 875);
await charge("American Express", "1000", "2026-06-10", "ANNUAL CARD FEE", 1750);
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CARD FEE");
expect(row?.confidence).toBe("confirmed");
expect(row?.observations).toBe(2);
// The amount is NOT averaged: this fee doubled, and the next one to plan for
// is the latest, not the mean of a rise.
expect(Number(row?.last_amount)).toBe(1750);
expect(row?.next_due_estimate.startsWith("2027-06")).toBe(true);
});
it("accepts a fee seen only once when the bank names it annual", async () => {
await charge("Westpac", "8032", "2025-10-21", "CARD FEE", 295);
const row = find(await getUpcomingAnnualCharges(ownerId), "CARD FEE");
// One charge cannot show a cadence, but the kind is not in doubt.
expect(row?.confidence).toBe("likely");
expect(row?.next_due_estimate).toBe("2026-10-21");
});
it("finds an annual merchant charge with no fee wording, if the amount is stable", async () => {
await purchase("2025-02-03", "AMAZON PRIME", 79);
await purchase("2026-02-01", "AMAZON PRIME", 79);
const row = find(await getUpcomingAnnualCharges(ownerId), "AMAZON PRIME");
expect(row?.confidence).toBe("possible");
});
});
describe("what it must NOT report", () => {
it("ignores a monthly account fee", async () => {
for (const d of ["2026-01-31", "2026-02-28", "2026-03-31", "2026-04-30"]) {
await charge("ANZ", "5626", d, "ACCOUNT SERVICING FEE", 5);
}
expect(find(await getUpcomingAnnualCharges(ownerId), "ACCOUNT SERVICING FEE")).toBeUndefined();
});
it("ignores per-transaction fees, which never repeat under one label", async () => {
await charge("Wise Australia Pty Ltd.", "9001", "2026-05-20", "WISE CHARGES FOR: TRANSFER-2143509822", 30.67);
await charge("Wise Australia Pty Ltd.", "9001", "2026-06-04", "WISE CHARGES FOR: TRANSFER-2171403032", 19.51);
const rows = await getUpcomingAnnualCharges(ownerId);
expect(rows.filter((r) => r.label.startsWith("WISE CHARGES"))).toHaveLength(0);
});
it("ignores a merchant visited twice a year apart for wildly different amounts", async () => {
// The exact shape that defeats a pure cadence test: Target, $30 then $198.
await purchase("2025-07-21", "TARGET", 30);
await purchase("2026-07-11", "TARGET", 198);
expect(find(await getUpcomingAnnualCharges(ownerId), "TARGET")).toBeUndefined();
});
it("ignores a merchant seen only once", async () => {
await purchase("2026-04-03", "UBER ONE MEMBERSHIP", 96);
expect(find(await getUpcomingAnnualCharges(ownerId), "UBER ONE MEMBERSHIP")).toBeUndefined();
});
});
describe("account identity and honesty about gaps", () => {
it("treats a renamed account as one continuous fee series", async () => {
// Citi Prestige -> NAB MyCard: same card, two bank names, two numbers.
await charge("Citibank", "5258073330340253", "2024-12-13", "Annual Fee", 700);
await charge("Citibank", "5258073330340253", "2025-11-13", "Annual Fee", 700);
await pool.query(
`INSERT INTO account_links (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason)
VALUES ('Citibank','5258073330340253','NAB','1015001000116227','2025-11-14','issuer_change')`
);
// A statement under the NEW identity, so the link has somewhere to resolve to.
await charge("NAB", "1015001000116227", "2026-08-13", "Purchase", 29.99, "debit");
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL FEE");
expect(row?.observations).toBe(2);
expect(row?.confidence).toBe("confirmed");
// Names the account's CURRENT bank -- the next fee comes from there, and
// sending someone to the retired brand is the bug this link exists to stop.
expect(row?.bank_name).toBe("NAB");
});
it("marks a passed estimate as expected_not_seen rather than upcoming", async () => {
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CHARGE");
expect(row?.status).toBe("expected_not_seen");
// And it reports how stale the account is, which is what separates
// "it stopped being charged" from "we stopped importing the statements".
expect(row?.account_days_stale).toBeGreaterThan(300);
});
it("sorts upcoming ahead of expected_not_seen", async () => {
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
await charge("Westpac", "8032", "2026-08-01", "CARD FEE", 295);
const rows = await getUpcomingAnnualCharges(ownerId);
const firstMissing = rows.findIndex((r) => r.status === "expected_not_seen");
const lastUpcoming = rows.map((r) => r.status).lastIndexOf("upcoming");
expect(lastUpcoming).toBeLessThan(firstMissing);
});
});
@@ -0,0 +1,271 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
/**
* The automatic pass over the reconcile queue.
*
* This exists because the wallet lane writes a transaction for every tap
* (DECISIONS.md ING-11), adding ~63 rows a month to a queue already ~177 deep
* and worked by hand. Hiding those rows instead is explicitly ruled out --
* see the awaitsStatementLine() removal note in queries.ts.
*
* The cases that must NOT match matter more than the ones that must: this runs
* unattended, and a wrong pairing merges one purchase's curation onto another
* purchase's row. So most of what follows is about refusal.
*/
const MACHINE_TOKEN = "test-wallet-token";
let pool: Pool;
let ownerId: number;
let autoReconcile: typeof import("../../app/api/transactions/reconcile/route").PATCH;
/** Browser-session caller: no machine token, auth comes from the mocked session. */
const userReq = (url = "http://x/api/transactions/reconcile") =>
({ url, headers: { get: () => null } }) as never;
/** Scheduled caller: machine token, no session, sweeps every owner. */
const machineReq = (url = "http://x/api/transactions/reconcile") =>
({ url, headers: { get: (h: string) => (h === "x-ingest-token" ? MACHINE_TOKEN : null) } }) as never;
async function statementLine(
amount: number,
date: string,
description: string,
opts: { owner?: number; type?: string; fxAmount?: number; fxCode?: string } = {}
) {
const owner = opts.owner ?? ownerId;
const st = await pool.query(
// Account number varies by owner: uq_statement_identity is
// (bank_name, account_number, billing_end_date), so two owners sharing one
// account collide rather than modelling two real cards.
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, owner, `1234-5678-9012-${String(owner).padStart(4, "0")}`]
);
const t = await pool.query(
`INSERT INTO transactions
(statement_id, transaction_date, description, amount, transaction_type, owner_id,
foreign_currency_amount, foreign_currency_code)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id`,
[st.rows[0].id, date, description, amount, opts.type ?? "debit", owner,
opts.fxAmount ?? null, opts.fxCode ?? null]
);
return t.rows[0].id as number;
}
async function walletRow(
amount: number,
date: string,
opts: { fxAmount?: number; fxCode?: string; paymentMethod?: string } = {}
) {
const id = await insertTransaction(pool, ownerId, {
amount, transaction_date: date, description: "Wallet tap",
});
await pool.query(
`UPDATE transactions
SET foreign_currency_amount = $2, foreign_currency_code = $3, payment_method = $4
WHERE id = $1`,
[id, opts.fxAmount ?? null, opts.fxCode ?? null, opts.paymentMethod ?? null]
);
return id;
}
const reconciledWith = async (id: number) =>
(await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [id]))
.rows[0].reconciled_with_id as number | null;
beforeAll(async () => {
process.env.WALLET_INGEST_TOKEN = MACHINE_TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ PATCH: autoReconcile } = await import("../../app/api/transactions/reconcile/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
describe("pairs it should apply", () => {
it("matches an exact amount two days apart and moves the link", async () => {
const manual = await walletRow(13.5, "2026-08-13");
const stmt = await statementLine(13.5, "2026-08-15", "BREADWORLD BAKERY");
const body = await (await autoReconcile(userReq())).json();
expect(body.reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("matches a foreign row on the NATIVE pair, not on the AUD amount", async () => {
// The bank converts at its own rate, so the wallet row cannot predict the
// AUD figure. It can only know THB 500. Deliberately give the statement a
// wildly different AUD amount to prove `amount` is not what matched.
const manual = await walletRow(999.99, "2026-08-13", { fxAmount: 500, fxCode: "THB" });
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
fxAmount: 500, fxCode: "THB",
});
const body = await (await autoReconcile(userReq())).json();
expect(body.reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("carries a SETTLED split across without resurrecting the debt", async () => {
const manual = await walletRow(50, "2026-08-13");
const stmt = await statementLine(50, "2026-08-14", "SHARED DINNER");
await pool.query(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled, settled_at)
VALUES ($1, $2, 50, true, now())`,
[manual, ownerId]
);
expect((await autoReconcile(userReq())).status ?? 200).toBe(200);
const moved = await pool.query(
`SELECT settled, settled_at FROM transaction_splits WHERE transaction_id = $1`, [stmt]
);
expect(moved.rowCount).toBe(1);
// The whole point: a delete-and-recreate defaults `settled` to false, which
// silently marks a paid debt owed again. Nothing errors when it regresses.
expect(moved.rows[0].settled).toBe(true);
expect(moved.rows[0].settled_at).not.toBeNull();
});
});
describe("pairs it must refuse", () => {
it("refuses two identical amounts in the same window (ambiguous both ways)", async () => {
// Two $13.50 bakery taps in one week is the ordinary case, not an edge one.
const a = await walletRow(13.5, "2026-08-13");
const b = await walletRow(13.5, "2026-08-14");
await statementLine(13.5, "2026-08-14", "BREADWORLD ONE");
await statementLine(13.5, "2026-08-15", "BREADWORLD TWO");
const body = await (await autoReconcile(userReq())).json();
expect(body.reconciled).toBe(0);
expect(await reconciledWith(a)).toBeNull();
expect(await reconciledWith(b)).toBeNull();
});
it("refuses a near-miss amount — a person should look at that", async () => {
const manual = await walletRow(13.5, "2026-08-13");
await statementLine(13.55, "2026-08-14", "BREADWORLD BAKERY");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
it("refuses a match outside the day window", async () => {
const manual = await walletRow(13.5, "2026-08-13");
await statementLine(13.5, "2026-08-25", "BREADWORLD BAKERY");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
it("refuses a statement row already claimed by another manual row", async () => {
const claimed = await statementLine(40, "2026-08-14", "ALREADY TAKEN");
const first = await walletRow(40, "2026-08-13");
await pool.query(`UPDATE transactions SET reconciled_with_id = $2 WHERE id = $1`, [first, claimed]);
const second = await walletRow(40, "2026-08-14");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(second)).toBeNull();
});
it("refuses a credits-funded row — no card leg is ever coming", async () => {
const manual = await walletRow(30, "2026-08-13", { paymentMethod: "credits" });
await statementLine(30, "2026-08-14", "COINCIDENTAL AMOUNT");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
});
describe("an AUD row against a foreign statement line", () => {
it("DOES match on the AUD figure — this is what every real pair looks like", async () => {
// Checked against live data: all 48 reconciliations a human has made are
// this exact shape -- manual row with no currency, statement row carrying
// EUR or CHF, paired on exact AUD. Overseas receipts and orders record what
// was charged in AUD; only the wallet lane records natively. An earlier
// version of this rule required both sides to be non-foreign, on the
// reasoning that a currency mismatch is ambiguous. It matched 0 of the 48.
const manual = await walletRow(22.85, "2026-08-13");
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
fxAmount: 500, fxCode: "THB",
});
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("still refuses when a second candidate makes it ambiguous", async () => {
// The coincidence this guards against -- a domestic tap whose amount equals
// a foreign charge's AUD figure -- is caught by unique-both-ways, not by a
// currency test.
const manual = await walletRow(22.85, "2026-08-13");
await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", { fxAmount: 500, fxCode: "THB" });
await statementLine(22.85, "2026-08-13", "MELBOURNE CAFE");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
});
describe("dry run", () => {
it("reports what it would do and writes nothing", async () => {
const manual = await walletRow(13.5, "2026-08-13");
await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
const body = await (await autoReconcile(userReq("http://x/api/transactions/reconcile?dry=1"))).json();
expect(body.dry_run).toBe(true);
expect(body.would_reconcile).toBe(1);
expect(await reconciledWith(manual)).toBeNull();
});
});
describe("the scheduled sweep", () => {
it("runs on the machine token with no session at all", async () => {
const manual = await walletRow(13.5, "2026-08-13");
const stmt = await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
const body = await (await autoReconcile(machineReq())).json();
expect(body.reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("covers EVERY owner, not just one", async () => {
// 169 of the 177 rows outstanding belong to the second participant, so a
// sweep scoped to a single user would leave almost all of the queue alone.
const other = (await pool.query(
`INSERT INTO participants (name) VALUES ('Carol') RETURNING id`
)).rows[0].id as number;
const mine = await walletRow(11, "2026-08-13");
const theirTx = (await pool.query(
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description,
amount, transaction_type, category, row_index)
VALUES ($1, NULL, '2026-08-13', 'Theirs', 22, 'debit', 'other', 0) RETURNING id`, [other]
)).rows[0].id as number;
await statementLine(11, "2026-08-14", "MINE");
await statementLine(22, "2026-08-14", "THEIRS", { owner: other });
const body = await (await autoReconcile(machineReq())).json();
expect(body.reconciled).toBe(2);
expect(await reconciledWith(mine)).not.toBeNull();
expect(await reconciledWith(theirTx)).not.toBeNull();
});
it("rejects a wrong machine token and falls through to session auth", async () => {
const bad = ({ url: "http://x/", headers: { get: () => "not-the-token" } }) as never;
// Session is mocked, so this still succeeds -- but as the session user only,
// proving the bad token did not grant the all-owner sweep.
const body = await (await autoReconcile(bad)).json();
expect(body.reconciled).toBe(0);
});
});
+24
View File
@@ -23,6 +23,25 @@ export function mockDbWithPool(p: Pool) {
const result = await p.query(sql, params); const result = await p.query(sql, params);
return result.rows[0] ?? null; return result.rows[0] ?? null;
}, },
// Mirrors withTransaction: one connection, BEGIN/COMMIT around the callback,
// so pg_advisory_xact_lock actually holds across the statements inside it.
withTransaction: async (fn: (q: unknown) => Promise<unknown>) => {
const client = await p.connect();
try {
await client.query("BEGIN");
const result = await fn({
queryRaw: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows,
queryRow: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows[0] ?? null,
});
await client.query("COMMIT");
return result;
} catch (e) {
await client.query("ROLLBACK");
throw e;
} finally {
client.release();
}
},
prisma: p, prisma: p,
})); }));
} }
@@ -65,6 +84,11 @@ export async function resetDB(pool: Pool) {
await assertTestDatabase(pool); await assertTestDatabase(pool);
await pool.query(` await pool.query(`
TRUNCATE TRUNCATE
-- The audit triggers (migration 0037) fire on every INSERT the suite makes.
-- TRUNCATE does not fire row-level triggers, so nothing here is logged --
-- but without this line the log is never cleared either, and the table
-- grows by roughly a thousand rows per full run, forever.
transaction_audit,
split_payments, split_payments,
transaction_splits, transaction_splits,
transaction_tags, transaction_tags,
+416 -3
View File
@@ -49,6 +49,7 @@ let fileDir: string;
let intakePOST: (req: NextRequest) => Promise<Response>; let intakePOST: (req: NextRequest) => Promise<Response>;
let lineItemsPUT: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>; let lineItemsPUT: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
let slackPOST: (req: NextRequest) => Promise<Response>; let slackPOST: (req: NextRequest) => Promise<Response>;
let orderGET: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
/** What the next LiteLLM call answers, and what the next pantry call answers. */ /** What the next LiteLLM call answers, and what the next pantry call answers. */
let modelReply: ExtractedOrder | Error = burgertory; let modelReply: ExtractedOrder | Error = burgertory;
@@ -104,6 +105,21 @@ async function seedOrder(over: { merchant: string; date: string; amount: number;
} }
return id; return id;
} }
/** A row shaped the way the wallet lane writes one: statementless, no metadata
* at all, no platform, and named by the notification rather than the receipt. */
async function seedWalletRow(over: { merchant: string; date: string; amount: number; owner?: number }) {
const t = await pool.query(
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description, amount, amount_aud, transaction_type, category, merchant_name, merchant_normalized, payment_method, location, source, source_ref, row_index)
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', $3, $3, 'card', 'Werribee, VIC', 'wallet', $5, 0) RETURNING id`,
[over.owner ?? ownerId, over.date, over.merchant, over.amount, `evt:${Math.floor(over.amount * 100)}`]
);
return t.rows[0].id as number;
}
/** What `components/order-details.tsx` would actually render for a transaction. */
const orderPanel = async (txnId: number) =>
(await (await orderGET({ headers: { get: () => null } } as unknown as NextRequest, { params: Promise.resolve({ id: String(txnId) }) })).json()) as { line_items?: unknown[] } | null;
const metaRows = async () => (await pool.query(`SELECT id, platform, line_items FROM expense_metadata ORDER BY id`)).rows as { id: number; platform: string | null; line_items: { description: string }[] }[];
const lineItemsOf = async (txnId: number) => const lineItemsOf = async (txnId: number) =>
(await pool.query(`SELECT line_items FROM expense_metadata WHERE transaction_id = $1 OR matched_transaction_id = $1`, [txnId])).rows[0]?.line_items; (await pool.query(`SELECT line_items FROM expense_metadata WHERE transaction_id = $1 OR matched_transaction_id = $1`, [txnId])).rows[0]?.line_items;
const batch = async (id: string) => (await pool.query(`SELECT * FROM intake_batches WHERE id = $1`, [id])).rows[0]; const batch = async (id: string) => (await pool.query(`SELECT * FROM intake_batches WHERE id = $1`, [id])).rows[0];
@@ -127,9 +143,17 @@ beforeAll(async () => {
participantForSlackUser: () => 1, participantForSlackUser: () => 1,
slackUserForParticipant: () => null, slackUserForParticipant: () => null,
})); }));
// The order-detail panel is the surface an enrichment has to appear on, so
// the read is exercised beside the write rather than assumed.
vi.doMock("@/lib/auth", () => ({ getCurrentUser: async () => ({ id: 1, name: "Owner", email: "o@x" }) }));
vi.doMock("@/lib/queries", async (importOriginal) => ({
...(await importOriginal<Record<string, unknown>>()),
canAccessTransactions: async () => true,
}));
({ POST: intakePOST } = await import("../../app/api/intake/route")); ({ POST: intakePOST } = await import("../../app/api/intake/route"));
({ PUT: lineItemsPUT } = await import("../../app/api/transactions/[id]/line-items/route")); ({ PUT: lineItemsPUT } = await import("../../app/api/transactions/[id]/line-items/route"));
({ POST: slackPOST } = await import("../../app/api/slack/interactive/route")); ({ POST: slackPOST } = await import("../../app/api/slack/interactive/route"));
({ GET: orderGET } = await import("../../app/api/transactions/[id]/order/route"));
stubFetch(); stubFetch();
}); });
@@ -210,6 +234,82 @@ describe("PUT /api/transactions/[id]/line-items (C2)", () => {
expect(em).toEqual({ source: "intake", platform: "ubereats", order_reference: "intake:b1", merchant_normalized: "Coles", amount: "85.74" }); expect(em).toEqual({ source: "intake", platform: "ubereats", order_reference: "intake:b1", merchant_normalized: "Coles", amount: "85.74" });
}); });
it("the undo baseline is recorded ONCE per batch: a retry gets the original, not its own write", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const original = [{ qty: 1, description: "what the order already had", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(original)]);
const id = uuid();
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
const first = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
expect(first.previous_line_items).toEqual(original);
// Pantry re-calls on every confirm and after a timeout. Before, the retry
// was handed its OWN write back, so undo restored the thing it undid.
const retry = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
expect(retry.previous_line_items).toEqual(original);
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual(original);
await put(txn, { line_items: retry.previous_line_items, source: "pantry", batch_id: id, replace: true });
expect(await lineItemsOf(txn)).toEqual(original);
});
it("without a batch id the baseline is still the row as read — nothing else changes", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const first = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
expect(first.previous_line_items).toEqual([]);
const second = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
expect(second.previous_line_items).toEqual(items);
});
it("a pinned row that no longer exists falls back to the live one, rather than reporting a phantom write", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const live = (await pool.query(`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].id as number;
const { replaceLineItems } = await import("../../lib/line-items");
const r = await replaceLineItems(txn, items, { source: "intake", metadataId: live + 999 });
expect(r.metadata_id).toBe(live);
expect(await lineItemsOf(txn)).toHaveLength(2);
});
it("two writes racing to create the metadata row both resolve to the same one", async () => {
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
const both = await Promise.all([
put(txn, { line_items: items, source: "intake", replace: true }),
put(txn, { line_items: items, source: "intake", replace: true }),
]);
expect(both.map((r) => r.status)).toEqual([200, 200]);
expect((await pool.query(`SELECT count(*)::int AS n FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].n).toBe(1);
});
it("the create path records an undo baseline too, before it writes", async () => {
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
const id = uuid();
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
await put(txn, { line_items: items, source: "intake", batch_id: id, currency: "AUD", replace: true });
// [] rather than NULL: there was nothing there, and "nothing" is the state
// undo has to be able to restore.
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual([]);
});
it("the panel reads the row the write landed on, whichever order the rows were inserted in", async () => {
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
// Inserted first: the delivery receipt pointing at the statement line.
await pool.query(
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, '[]'::jsonb)`,
[t, uuid()]
);
// Inserted second: a row the transaction owns. An unordered LIMIT 1 reads
// the first one, the write takes the second, and the enrichment reports
// success against a row nothing renders.
await pool.query(
`INSERT INTO expense_metadata (transaction_id, source, order_reference, line_items) VALUES ($1, 'pantry', $2, '[]'::jsonb)`,
[t, uuid()]
);
const written = await (await put(t, { line_items: items, source: "intake", replace: true })).json();
const panel = await orderPanel(t);
expect(panel?.line_items).toHaveLength(2);
expect((await metaRows()).find((r) => r.id === written.metadata_id)!.line_items).toHaveLength(2);
});
it("finds the row through matched_transaction_id — a card-settled order has only that", async () => { it("finds the row through matched_transaction_id — a card-settled order has only that", async () => {
const statementLine = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false }); const statementLine = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
await pool.query( await pool.query(
@@ -250,6 +350,64 @@ describe("POST /api/intake — gates", () => {
expect((await batch(id)).status).toBe("unsupported"); expect((await batch(id)).status).toBe("unsupported");
expect(calls).toHaveLength(0); expect(calls).toHaveLength(0);
}); });
// 2026-08-22, live batch 3d2ffbe8…: an iOS Shortcut coerced the shared photo
// to text, so the whole upload was the five characters `Image`. The model was
// asked anyway and returned Burgertory (Braybrook), two "The Pride" lines and
// a $41.15 total — none of which existed. Only the absent date kept it out of
// the matcher, and the invented header was stored on the batch as
// `needs_header`. The model must never be asked.
it("a text part that cannot be a receipt is declined BEFORE any model call", async () => {
const id = uuid();
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)", order_date: null, total: 41.15 };
const res = await intakePOST(
multipart({ source: "other", batch_id: id }, [{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") }])
);
const body = await res.json();
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
expect(body.text).toMatch(/single word/);
expect(body.text).toMatch(/"Image"/);
// The model was never asked — the only thing that makes confabulation impossible.
expect(calls).toHaveLength(0);
const row = await batch(id);
expect(row.status).toBe("unsupported");
expect(row.kind).toBe("unsupported");
expect(row.header).toBeNull();
expect(row.items).toBeNull();
expect(row.transaction_id).toBeNull();
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
});
it("a junk text part beside a real receipt does not sink the batch", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const body = await (
await intakePOST(
multipart({ source: "uber_eats" }, [
{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") },
{ name: "Receipt_19Aug2026.pdf", type: "application/pdf", bytes: BURGERTORY },
])
)
).json();
expect(body.status).toBe("enriched");
expect(await lineItemsOf(txn)).toHaveLength(2);
});
it("an order extracted from an unreadable PDF is unsupported, not needs_header", async () => {
const id = uuid();
// A PDF that renders to a scrap of junk still reaches the model — the
// pre-check only inspects text parts. `confabulationReason` is the belt.
vi.spyOn(await import("../../lib/intake-extract"), "extractPdfText").mockResolvedValueOnce("Image");
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)" };
const body = await (
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [{ name: "x.pdf", type: "application/pdf", bytes: BURGERTORY }]))
).json();
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
expect(body.text).toMatch(/nothing shared could be read/);
const row = await batch(id);
expect(row.status).toBe("unsupported");
expect(row.header).toBeNull();
expect(row.items).toBeNull();
});
}); });
describe("POST /api/intake — food (enrich vs candidates)", () => { describe("POST /api/intake — food (enrich vs candidates)", () => {
@@ -285,10 +443,82 @@ describe("POST /api/intake — food (enrich vs candidates)", () => {
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1); expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
}); });
it("the platform is part of the key: the same merchant/date/total on DoorDash is not the match", async () => { it("the platform is part of the key: the same merchant/date/total on DoorDash is not the match — but it IS offered", async () => {
await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "doordash" }); const doordash = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "doordash" });
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json(); const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body.status).toBe("candidates"); expect(body.status).toBe("candidates");
// The candidates card is a human choice, not a decision. The platform is one
// of the things that may be wrong, so it must not narrow the list — a
// platform-filtered card showed nothing at all here.
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([doordash]);
});
it("a NULL platform is 'no platform', not a wildcard — an in-store share does not overwrite the delivery order", async () => {
await pool.query(`INSERT INTO intake_merchant_kinds (merchant_key, kind) VALUES ('burgertory werribee', 'food')`);
const delivery = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "ubereats" });
const already = [{ qty: 1, description: "what the delivery order already had", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [delivery, JSON.stringify(already)]);
// `other` implies no platform; the delivery row carries one, so it is not
// the match — it is only a candidate.
const body = await (await intakePOST(multipart({ source: "other" }, [pdf]))).json();
expect(body.status).toBe("candidates");
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([delivery]);
expect(await lineItemsOf(delivery)).toEqual(already);
});
it("a reconciled manual row is neither matched nor offered — EXCLUDE_RECONCILED_SOURCE, as everywhere else", async () => {
const live = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const reconciled = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, reconciled]);
const near = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-17", amount: 19.9 });
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, near]);
// Two rows at the exact key would have read as an ambiguity; one of them is
// the same spend counted twice.
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ status: "enriched", transaction_id: live });
});
it("the write lands on the row the MATCH resolved through, not on whichever row sorts first", async () => {
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
// The statement line owns a receipt row of its own (a pantry confirm) …
const grocery = [{ qty: 1, description: "a grocery shop", amount: 22.4 }];
const own = (await pool.query(
`INSERT INTO expense_metadata (transaction_id, source, order_reference, merchant_normalized, line_items) VALUES ($1, 'pantry', $2, 'Woolworths', $3::jsonb) RETURNING id`,
[t, uuid(), JSON.stringify(grocery)]
)).rows[0].id as number;
// … and is also the card leg of the Uber Eats order being enriched.
const delivery = (await pool.query(
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, merchant_normalized, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, 'Burgertory (Werribee)', '[]'::jsonb) RETURNING id`,
[t, uuid()]
)).rows[0].id as number;
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ status: "enriched", transaction_id: t });
const rows = await metaRows();
expect(rows.find((r) => r.id === delivery)!.line_items.map((i) => i.description)).toEqual(["Michael", "Joel"]);
expect(rows.find((r) => r.id === own)!.line_items).toEqual(grocery);
});
it("an upload with no prices never replaces a receipt that already has items", async () => {
const priceless = { ...burgertory, subtotal: null, total: 22.4, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
modelReply = priceless;
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ kind: "held", status: "held" });
expect(body.text).toMatch(/left the existing items alone/);
expect(await lineItemsOf(txn)).toEqual(already);
// …but an EMPTY receipt is exactly what this slice exists to fill.
const empty = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-20", amount: 22.4 });
modelReply = { ...priceless, order_date: "2026-08-20" };
const second = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(second).toMatchObject({ status: "enriched", transaction_id: empty });
expect((await lineItemsOf(empty)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
}); });
it("food + no match → candidates (same merchant ±3 days, any amount) with Use-this / Create buttons; NEVER auto-creates", async () => { it("food + no match → candidates (same merchant ±3 days, any amount) with Use-this / Create buttons; NEVER auto-creates", async () => {
@@ -322,7 +552,13 @@ describe("POST /api/intake — food (enrich vs candidates)", () => {
const id = uuid(); const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json(); const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("needs_header"); expect(body.status).toBe("needs_header");
expect(body.text).toMatch(/share the top of the screen/); // Asserts BOTH halves of needsHeaderCard(): that it names what is actually
// missing (total, not a blanket "total or date"), and that a uber_eats
// source gets the share-more-of-the-screen advice rather than the in-store
// Share-button advice. 258fe3b changed this text and updated the unit tests
// but not this one.
expect(body.text).toMatch(/I can see items but no total/);
expect(body.text).toMatch(/share the part of the screen that shows it/);
expect(await lineItemsOf(txn)).toEqual([]); expect(await lineItemsOf(txn)).toEqual([]);
expect((await batch(id)).status).toBe("needs_header"); expect((await batch(id)).status).toBe("needs_header");
}); });
@@ -391,6 +627,19 @@ describe("POST /api/intake — grocery → pantry", () => {
expect((await batch(id))).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn }); expect((await batch(id))).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn });
}); });
it("an in-store tap does not hand pantry the delivery order as its finance target", async () => {
modelReply = coles;
// The Uber Eats Coles order the email pipeline already booked.
const delivery = await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: "ubereats" });
// The same shop, same day, same money — but shared from the Coles app, so
// it carries no platform and is a different purchase.
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
const pantryCall = calls.find((c) => c.url === "http://pantry.test/api/capture")!.body as { intake: Record<string, unknown> };
expect(pantryCall.intake).toMatchObject({ finance_action: "create", finance_target_transaction_id: null, platform: null });
expect(await lineItemsOf(delivery)).toEqual([]);
});
it("grocery + no match → pantry with finance_action create (ING-10 as today)", async () => { it("grocery + no match → pantry with finance_action create (ING-10 as today)", async () => {
modelReply = coles; modelReply = coles;
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json(); const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
@@ -400,6 +649,33 @@ describe("POST /api/intake — grocery → pantry", () => {
expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" }); expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" });
}); });
it("pantry saying the shop is already on its board is SHOWN, not swallowed", async () => {
// Pantry accepts a second capture of one receipt on purpose (the first may
// have been a bad crop) and reports it rather than refusing. Dropping that
// field turned "reported" into silence — and a duplicate nobody sees is a
// duplicate that gets confirmed, which counts the same groceries twice
// because pantry's addItem accumulates. Seen live on captures 66/69.
modelReply = coles;
pantryReply = { status: 200, body: { captureId: 902, items: [1, 2, 3], alreadyCaptured: { captureId: 66, confirmed: true } } };
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
expect(body.text).toMatch(/Already on the pantry board/);
expect(body.text).toMatch(/capture 66 \(already confirmed\)/);
expect(body.text).toMatch(/capture 902/);
});
it("two ledger rows for one shop are reported, not treated as no match", async () => {
// Ambiguity is not a miss. This is the exact state that produced the
// 2026-08-18 Woolworths double-book: the shop was in the ledger twice, the
// matcher returned nothing, and the card said finance would book it.
modelReply = coles;
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
expect(body.text).toMatch(/found more than one row it could be/);
});
it("pantry 409 → duplicate", async () => { it("pantry 409 → duplicate", async () => {
modelReply = coles; modelReply = coles;
pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } }; pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } };
@@ -457,6 +733,143 @@ describe("Slack taps (C5)", () => {
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1); expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
}); });
it("intake_create re-runs the match: an order booked between the card and the tap is enriched, not duplicated", async () => {
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
// The email/order bridge books the same purchase while the card sits in
// Slack. The card's "no match" is minutes stale by the time it is pressed.
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: txn });
expect((await lineItemsOf(txn)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
expect(lastSlackUpdate().text).toMatch(/^Added 2 items/);
});
it("intake_create adopts the wallet row for the same tap instead of booking it twice", async () => {
// The normal arrival order: the notification lands within seconds of the
// tap, the screenshot is shared later. matchTransaction cannot see that row
// — its key includes the platform and a notification has none — so before
// this, Create booked the purchase a second time.
const wallet = await seedWalletRow({ merchant: "Burgertory", date: "2026-08-19", amount: 22.4 });
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: wallet });
expect((await lineItemsOf(wallet)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
// The suburb the notification supplied is the one thing nothing else knows.
expect((await pool.query(`SELECT location FROM transactions WHERE id = $1`, [wallet])).rows[0].location).toBe("Werribee, VIC");
});
it("intake_create still creates when a same-amount row already carries a receipt", async () => {
// A row that another source fully described is a DIFFERENT purchase that
// happens to share an amount — adopting it would move one order's items
// onto another order's row.
const other = await seedOrder({ merchant: "Someone Else", date: "2026-08-19", amount: 22.4, platform: null });
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [other, JSON.stringify([{ description: "theirs", qty: 1, amount: 22.4 }])]);
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(2);
});
it("two pending cards for one order book it ONCE", async () => {
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
expect((await batch(a)).status).toBe("candidates");
expect((await batch(b)).status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: a }));
await slackPOST(slackPress("intake_create", { batch_id: b }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect((await batch(b)).transaction_id).toBe((await batch(a)).transaction_id);
});
it("a sibling already resolved to the pantry board makes Create a duplicate, not a second row", async () => {
// Two shares of one receipt, resolved out of order: nothing the second card
// can match on, because the first one never produced a transaction.
modelReply = { ...coles, merchant: "Romeo's Food Hall" };
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "other", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
await intakePOST(multipart({ source: "other", batch_id: a }, [{ name: "Receipt_18Aug2026.pdf", type: "application/pdf", bytes: COLES }]));
expect((await batch(a)).status).toBe("ask");
await slackPOST(slackPress("intake_grocery", { batch_id: a }));
expect((await batch(a)).status).toBe("forwarded_to_pantry");
await slackPOST(slackPress("intake_meal", { batch_id: b }));
expect((await batch(b)).status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: b }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
expect(await batch(b)).toMatchObject({ status: "duplicate", capture_id: 901 });
expect(lastSlackUpdate().text).toMatch(/^Already handled/);
});
it("two taps that land together still book one transaction, and neither reports an error", async () => {
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: id })), slackPOST(slackPress("intake_create", { batch_id: id }))]);
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
const texts = calls.filter((c) => c.url.includes("hooks.slack.com")).map((c) => (c.body as { text: string }).text);
expect(texts.every((t) => !/Couldn't process/.test(t))).toBe(true);
});
it("a foreign receipt is booked in its own currency, with no invented AUD figure", async () => {
modelReply = { ...burgertory, currency: "USD" };
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await slackPOST(slackPress("intake_create", { batch_id: id }));
const t = (await pool.query(`SELECT amount::text, amount_aud, foreign_currency_amount::text, foreign_currency_code FROM transactions`)).rows[0];
expect(t).toEqual({ amount: "22.40", amount_aud: null, foreign_currency_amount: "22.40", foreign_currency_code: "USD" });
expect(lastSlackUpdate().text).toMatch(/USD 22\.40/);
// OrderDetails prefers expense_metadata.currency, so a hard-coded 'AUD'
// here printed "$" on a USD order's receipt panel.
expect((await pool.query(`SELECT currency FROM expense_metadata`)).rows[0].currency).toBe("USD");
});
it("the hold applies to the Create tap's re-match, not only to the upload path", async () => {
modelReply = { ...burgertory, subtotal: null, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
// Booked WITH its items between the card and the tap. The re-match walked
// past the guard routeFood applies, so the same upload destroyed the same
// receipt depending on which button reached it.
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect(await lineItemsOf(txn)).toEqual(already);
expect(await batch(id)).toMatchObject({ status: "held" });
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
});
it("two DIFFERENT batches for one purchase, tapped together, book it once", async () => {
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
// Their source_refs differ, so ON CONFLICT cannot save this; both taps
// clear their match and duplicate checks before either inserts.
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: a })), slackPOST(slackPress("intake_create", { batch_id: b }))]);
const rows = (await pool.query(`SELECT id FROM transactions`)).rows;
expect(rows).toHaveLength(1);
expect((await batch(a)).transaction_id).toBe(rows[0].id);
expect((await batch(b)).transaction_id).toBe(rows[0].id);
});
it("unknown shared_by falls back to the default owner — never a NULL owner", async () => { it("unknown shared_by falls back to the default owner — never a NULL owner", async () => {
const id = uuid(); const id = uuid();
await intakePOST(multipart({ source: "uber_eats", shared_by: "nobody@example.test", batch_id: id }, [pdf])); await intakePOST(multipart({ source: "uber_eats", shared_by: "nobody@example.test", batch_id: id }, [pdf]));
@@ -0,0 +1,175 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
/**
* Manually resolving a duplicate the automation cannot see.
*
* Until this route existed there was no way to merge or remove a transaction at
* all: `superseded_by_id` was read everywhere and written nowhere in
* application code, having been set exactly once by migration 0023. The only
* merge path was the reconcile queue, whose candidate list is bounded by an
* amount band -- so SP PEAKDESIGN (notified $42.44, charged $43.71, the card's
* 3% foreign transaction fee) had no offerable twin.
*
* The refusals matter more than the happy paths, exactly as in
* auto-reconcile.test.ts: this writes to columns that hide a row from every
* figure, so a wrong call does not error, it quietly stops counting money.
*/
let pool: Pool;
let ownerId: number;
let otherId: number;
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
const call = (id: number, body: unknown) => merge(req(body), { params: Promise.resolve({ id: String(id) }) });
// uq_statement_identity is (bank_name, account_number, billing_end_date), so
// every statement in a test needs its own account or the second insert collides.
let acctSeq = 0;
async function statementLine(amount: number, date: string, description: string) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, ownerId, `4444-0000-0000-${String(1000 + acctSeq++)}`]
);
const t = await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
[st.rows[0].id, date, description, amount, ownerId]
);
return t.rows[0].id as number;
}
beforeAll(async () => {
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId, otherId } = await seedParticipants(pool));
});
describe("reconcile mode", () => {
it("folds a wallet row into its statement twin even though the amounts differ by the FX fee", async () => {
const wallet = await insertTransaction(pool, ownerId, {
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
});
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
const res = await call(wallet, { other_id: stmt, mode: "reconcile" });
expect(res.status ?? 200).toBe(200);
const r = await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [wallet]);
expect(r.rows[0].reconciled_with_id).toBe(stmt);
});
it("refuses when the target is not a statement row", async () => {
const a = await insertTransaction(pool, ownerId, { amount: 10 });
const b = await insertTransaction(pool, ownerId, { amount: 10 });
const res = await call(a, { other_id: b, mode: "reconcile" });
expect(res.status).toBe(400);
});
it("refuses when the source is itself a statement row", async () => {
const s1 = await statementLine(10, "2026-08-01", "A");
const s2 = await statementLine(10, "2026-08-02", "B");
const res = await call(s1, { other_id: s2, mode: "reconcile" });
expect(res.status).toBe(400);
});
});
describe("supersede mode", () => {
it("carries splits onto the survivor before hiding the loser", async () => {
// The whole reason supersede moves curation: `superseded_by_id` removes the
// row from every figure, so a split left behind stops counting silently and
// a settled debt reappears as owed.
const loser = await insertTransaction(pool, ownerId, { amount: 80, description: "DUP" });
const survivor = await insertTransaction(pool, ownerId, { amount: 80, description: "KEEP" });
await pool.query(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
VALUES ($1,$2,50,true), ($1,$3,50,false)`,
[loser, ownerId, otherId]
);
const res = await call(loser, { other_id: survivor, mode: "supersede" });
expect(res.status ?? 200).toBe(200);
const onLoser = await pool.query(`SELECT count(*)::int c FROM transaction_splits WHERE transaction_id=$1`, [loser]);
const onSurvivor = await pool.query(
`SELECT participant_id, share_percent, settled FROM transaction_splits
WHERE transaction_id=$1 ORDER BY participant_id`, [survivor]
);
expect(onLoser.rows[0].c).toBe(0);
expect(onSurvivor.rows).toHaveLength(2);
// settled must survive the move -- a delete-and-recreate defaults it to false.
expect(onSurvivor.rows.map((r) => r.settled)).toContain(true);
const marked = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [loser]);
expect(marked.rows[0].superseded_by_id).toBe(survivor);
});
it("works between two statement rows, which reconcile cannot express", async () => {
const s1 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
const s2 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
const res = await call(s1, { other_id: s2, mode: "supersede" });
expect(res.status ?? 200).toBe(200);
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [s1]);
expect(r.rows[0].superseded_by_id).toBe(s2);
});
it("refuses to hide a statement row behind a statementless one", async () => {
// The wrong direction, and silently wrong: a wallet row holds the amount
// BEFORE the card's foreign transaction fee, so burying the statement line
// under it leaves the ledger holding $42.44 for a $43.71 charge.
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
const wallet = await insertTransaction(pool, ownerId, {
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
});
const res = await call(stmt, { other_id: wallet, mode: "supersede" });
expect(res.status).toBe(400);
// and the statement row is untouched
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [stmt]);
expect(r.rows[0].superseded_by_id).toBeNull();
});
it("refuses to bury a row under one that is already superseded", async () => {
const a = await insertTransaction(pool, ownerId, { amount: 5 });
const b = await insertTransaction(pool, ownerId, { amount: 5 });
const c = await insertTransaction(pool, ownerId, { amount: 5 });
await call(b, { other_id: c, mode: "supersede" }); // b -> c
const res = await call(a, { other_id: b, mode: "supersede" }); // a -> b, but b is buried
expect(res.status).toBe(409);
});
});
describe("refusals common to both modes", () => {
it("refuses a transaction merging into itself", async () => {
const a = await insertTransaction(pool, ownerId, { amount: 1 });
const res = await call(a, { other_id: a, mode: "supersede" });
expect(res.status).toBe(400);
});
it("refuses a row that is already reconciled", async () => {
const wallet = await insertTransaction(pool, ownerId, { amount: 12, transaction_date: "2026-08-10" });
const stmt = await statementLine(12, "2026-08-10", "X");
await call(wallet, { other_id: stmt, mode: "reconcile" });
const other = await insertTransaction(pool, ownerId, { amount: 12 });
const res = await call(wallet, { other_id: other, mode: "supersede" });
expect(res.status).toBe(409);
});
it("refuses a transaction belonging to someone else", async () => {
const mine = await insertTransaction(pool, ownerId, { amount: 7 });
const theirs = await insertTransaction(pool, otherId, { amount: 7 });
const res = await call(mine, { other_id: theirs, mode: "supersede" });
expect(res.status).toBe(404);
});
});
@@ -0,0 +1,153 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants } from "./helpers";
/**
* A receipt arriving after the wallet notification must ENRICH that row, not
* create a second one.
*
* The case this is built from is real. Woolworths $103.30 on 2026-08-18: the
* wallet event landed at 09:00:14 on the 18th, and nothing created a
* transaction until 20:25:06 on the 19th -- 35h25m later -- at which point the
* receipt path made its own row from scratch. One shop, two rows, and the
* notification's contribution (the instrument, and the suburb) thrown away.
*
* Sources for one purchase arrive in sequence, not in competition, so the rule
* is: the receipt read the register tape and wins on what was bought; anything
* only the notification knew survives untouched.
*/
const TOKEN = "test-receipt-token";
let pool: Pool;
let ownerId: number;
let ingest: typeof import("../../app/api/receipts/ingest/route").POST;
const ingestReq = (body: unknown) =>
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
/** A row shaped the way the wallet lane writes one: statementless, no line
* items, carrying the suburb the notification supplied. */
async function walletRow(amount: number, date: string, merchant: string, location: string) {
const t = await pool.query(
`INSERT INTO transactions
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
transaction_type, category, payment_method, merchant_name, merchant_normalized,
location, source, source_ref, row_index)
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', 'card', $3, $3, $5, 'wallet', $6, 0)
RETURNING id`,
[ownerId, date, merchant, amount, location, `evt-${amount}-${date}`]
);
const id = t.rows[0].id as number;
await pool.query(
`INSERT INTO expense_metadata
(transaction_id, source, merchant_normalized, amount, transaction_date,
currency, line_items, payment_method_detail, flags)
VALUES ($1, 'wallet', $2, $3, $4, 'AUD', '[]'::jsonb, 'Visa', '[]'::jsonb)`,
[id, merchant, amount, date]
);
return id;
}
const receipt = (amount: number, date: string) => ({
receipt_uid: `woolworths:3345:${amount}:2026-08-18`,
capture_event_id: 900,
merchant_name: "Woolworths",
store_detail: "Wyndham Vale",
transaction_date: date,
total: amount,
tender_legs: [
{ leg_index: 1, amount, card_last4: "6078", card_product: "VISA PREPAID", class: "gift_card" },
],
line_items: [
{ description: "Milk 2L", qty: 1, unit: "ea", amount: 3.5, category: "dairy" },
{ description: "Bread", qty: 1, unit: "ea", amount: 4.0, category: "bakery" },
],
});
beforeAll(async () => {
process.env.RECEIPT_INGEST_TOKEN = TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: ingest } = await import("../../app/api/receipts/ingest/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
const txCount = async () =>
Number((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n);
describe("a receipt meeting the wallet row for the same tap", () => {
it("adopts it instead of creating a second transaction", async () => {
const walletId = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
expect(await txCount()).toBe(1);
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
expect(await txCount()).toBe(1);
const meta = await pool.query(
`SELECT transaction_id, jsonb_array_length(line_items) AS items, source,
payment_method_detail, card_last4
FROM expense_metadata`
);
expect(meta.rowCount).toBe(1);
expect(meta.rows[0].transaction_id).toBe(walletId);
expect(meta.rows[0].items).toBe(2);
expect(meta.rows[0].source).toBe("pantry");
// The receipt read the card off the tape; the notification only saw "Visa".
expect(meta.rows[0].payment_method_detail).toBe("VISA PREPAID");
expect(meta.rows[0].card_last4).toBe("6078");
});
it("keeps the suburb only the notification knew", async () => {
await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
const t = await pool.query(`SELECT location, merchant_normalized, category FROM transactions`);
// `location` is absent from the adoption UPDATE's column list on purpose:
// the receipt has no suburb, so it must not be able to erase one.
expect(t.rows[0].location).toBe("Wyndham Vale, VIC");
// ...while what the receipt does know wins.
expect(t.rows[0].merchant_normalized).toBe("Woolworths");
expect(t.rows[0].category).toBe("groceries");
});
it("creates its own row when there is no wallet row to adopt", async () => {
await ingest(ingestReq(receipt(58.4, "2026-08-18")));
expect(await txCount()).toBe(1);
const meta = await pool.query(`SELECT source, jsonb_array_length(line_items) AS items FROM expense_metadata`);
expect(meta.rows[0].source).toBe("pantry");
expect(meta.rows[0].items).toBe(2);
});
it("refuses to adopt when two candidates share the amount", async () => {
// Two taps of the same value in the window: adopting either would move this
// shop's line items onto a purchase it did not describe.
await walletRow(103.3, "2026-08-17", "Woolworths", "Wyndham Vale, VIC");
await walletRow(103.3, "2026-08-18", "Coles", "Manor Lakes, VIC");
expect(await txCount()).toBe(2);
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
// Falls back to creating its own row rather than guessing.
expect(await txCount()).toBe(3);
});
it("refuses to adopt a row that already carries line items", async () => {
const other = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
await pool.query(
`UPDATE expense_metadata SET line_items = '[{"description":"Something else","amount":1}]'::jsonb
WHERE transaction_id = $1`, [other]
);
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
// A fully-described row is a different purchase that happens to share an amount.
expect(await txCount()).toBe(2);
});
});
@@ -0,0 +1,86 @@
import { describe, it, expect, beforeAll, beforeEach } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
import { getPendingReconciliations } from "../../lib/queries";
/**
* The reconcile queue must not offer what is already resolved.
*
* A superseded row is hidden from every figure (EXCLUDE_RECONCILED_SOURCE), so
* listing it is not merely noise: on 2026-10-03 the queue showed the duplicate
* wallet half 7037, superseded under its twin 7038, as a second "Digidirect"
* with the same statement line offered as its match. Accepting it there would
* have spent that line on a hidden row and left the real one unmatched forever.
* findAutoMatches() always had both filters; this query had neither.
*/
let pool: Pool;
let ownerId: number;
let acctSeq = 0;
async function statementLine(amount: number, date: string, description: string) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('q.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
);
const t = await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
[st.rows[0].id, date, description, amount, ownerId]
);
return t.rows[0].id as number;
}
beforeAll(() => { pool = createPool(); });
beforeEach(async () => {
await resetDB(pool);
({ ownerId } = await seedParticipants(pool));
});
describe("getPendingReconciliations", () => {
it("does not list a superseded manual row", async () => {
const survivor = await insertTransaction(pool, ownerId, { description: "DIGIDIRECT", amount: 1180.05, transaction_date: "2026-08-25" });
const hidden = await insertTransaction(pool, ownerId, { description: "Digidirect", amount: 1180.05, transaction_date: "2026-08-25" });
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [survivor, hidden]);
const line = await statementLine(1180.05, "2026-08-25", "SP DIGIDIRECT STRATHFIELD");
const queue = await getPendingReconciliations(ownerId);
const ids = queue.map((r) => r.id);
expect(ids).toContain(survivor);
expect(ids).not.toContain(hidden);
// The survivor is still offered its line.
const row = queue.find((r) => r.id === survivor)!;
expect(row.matches.map((m) => m.id)).toContain(line);
});
it("does not offer a superseded statement row as a match", async () => {
const manual = await insertTransaction(pool, ownerId, { description: "Grocer", amount: 42.0, transaction_date: "2026-08-10" });
const live = await statementLine(42.0, "2026-08-10", "GROCER");
const dupe = await statementLine(42.0, "2026-08-10", "GROCER");
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [live, dupe]);
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
const offered = row.matches.map((m) => m.id);
expect(offered).toContain(live);
expect(offered).not.toContain(dupe);
});
it("offers the exact native twin of a foreign row even when its provisional AUD is far off", async () => {
// Live shape, 2026-10-09: wallet INR 112 at a provisional A$2.01; CommBank
// charged A$1.68 -- 16% under, outside the +-5% band.
const manual = await insertTransaction(pool, ownerId, { description: "SWIGGY LIMITED", amount: 2.01, transaction_date: "2026-10-04" });
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [manual]);
const twin = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
const otherRupees = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [twin]);
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 113 WHERE id = $1`, [otherRupees]);
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
const offered = row.matches.map((m) => m.id);
expect(offered).toContain(twin);
// A different native figure at the same AUD is not admitted by this path.
expect(offered).not.toContain(otherRupees);
});
});
+167
View File
@@ -0,0 +1,167 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
import { EXCLUDE_RECONCILED_SOURCE } from "@/lib/analytics-sql";
/**
* A charge that never posted (migration 0040).
*
* Txn 7951: a $1,040.41 Airbnb authorisation hold on Amex, booking cancelled
* the next day, hold released. The wallet lane booked the notification as a
* purchase and no statement line ever arrived, so it sat in the reconcile queue
* forever and counted as spend everywhere. A void has no survivor, which is why
* neither reconcile nor supersede could express it.
*
* As in merge-supersede.test.ts, the refusals matter as much as the happy path:
* a void quietly stops money counting.
*/
let pool: Pool;
let ownerId: number;
let otherId: number;
let voidRoute: typeof import("../../app/api/transactions/[id]/void/route");
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
const params = (id: number) => ({ params: Promise.resolve({ id: String(id) }) });
const doVoid = (id: number, reason: unknown = "hold released, booking cancelled") =>
voidRoute.POST(req({ reason }), params(id));
let acctSeq = 0;
async function statementLine(amount: number, date: string, description: string) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
);
const t = await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
[st.rows[0].id, date, description, amount, ownerId]
);
return t.rows[0].id as number;
}
/** What every "count each purchase once" query sees. */
async function countedIds(): Promise<number[]> {
const r = await pool.query(
`SELECT t.id FROM transactions t WHERE ${EXCLUDE_RECONCILED_SOURCE} ORDER BY t.id`
);
return r.rows.map((x) => x.id);
}
beforeAll(async () => {
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
voidRoute = await import("../../app/api/transactions/[id]/void/route");
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId, otherId } = await seedParticipants(pool));
});
describe("voiding a charge that never posted", () => {
it("drops the row out of every figure and the reconcile queue, but keeps it", async () => {
const { getPendingReconciliations } = await import("@/lib/queries");
const hold = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 1040.41 });
const real = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 529.05 });
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).toContain(hold);
const res = await doVoid(hold);
expect(res.status ?? 200).toBe(200);
expect(await countedIds()).toEqual([real]);
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).not.toContain(hold);
const r = await pool.query(`SELECT voided_at, void_reason FROM transactions WHERE id = $1`, [hold]);
expect(r.rows[0].voided_at).not.toBeNull();
expect(r.rows[0].void_reason).toBe("hold released, booking cancelled");
});
it("can be undone", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 1040.41 });
await doVoid(hold);
const res = await voidRoute.DELETE(req({}), params(hold));
expect(res.status ?? 200).toBe(200);
expect(await countedIds()).toEqual([hold]);
});
it("requires a reason", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 10 });
expect((await doVoid(hold, " ")).status).toBe(400);
expect((await doVoid(hold, null)).status).toBe(400);
expect(await countedIds()).toEqual([hold]);
});
it("refuses a statement line -- that charge posted", async () => {
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
expect((await doVoid(stmt)).status).toBe(409);
expect(await countedIds()).toEqual([stmt]);
});
it("refuses a reconciled row -- it found its statement line, so it posted", async () => {
const wallet = await insertTransaction(pool, ownerId, { amount: 50, transaction_date: "2026-09-10" });
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [stmt, wallet]);
expect((await doVoid(wallet)).status).toBe(409);
});
it("refuses a row with a settled split -- someone already paid their share", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 80 });
await pool.query(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
VALUES ($1, $2, 50, true)`,
[hold, otherId]
);
expect((await doVoid(hold)).status).toBe(409);
expect(await countedIds()).toEqual([hold]);
});
it("is enforced by the database, not only the route", async () => {
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
await expect(
pool.query(`UPDATE transactions SET voided_at = now(), void_reason = 'x' WHERE id = $1`, [stmt])
).rejects.toThrow(/transactions_void_statementless/);
const manual = await insertTransaction(pool, ownerId, { amount: 5 });
await expect(
pool.query(`UPDATE transactions SET voided_at = now() WHERE id = $1`, [manual])
).rejects.toThrow(/transactions_void_has_reason/);
});
});
describe("the matchers never use a voided row", () => {
it("auto-reconcile does not pair a voided row with a statement line of the same amount", async () => {
const { findAutoMatches } = await import("@/lib/auto-reconcile");
const hold = await insertTransaction(pool, ownerId, { amount: 120, transaction_date: "2026-09-10" });
await statementLine(120, "2026-09-11", "SOMETHING");
expect((await findAutoMatches(ownerId)).map((m) => m.manual_id)).toContain(hold);
await doVoid(hold);
expect(await findAutoMatches(ownerId)).toEqual([]);
});
it("a receipt for a rebooking at the same amount adopts the live row, not the released hold", async () => {
const { findAdoptableTransaction } = await import("@/lib/auto-reconcile");
const hold = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-10" });
const rebooked = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-11" });
// Two candidates: the adopter refuses rather than guess.
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBeNull();
await doVoid(hold);
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBe(rebooked);
});
it("merge refuses a voided row on either side", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 70, transaction_date: "2026-09-10" });
const stmt = await statementLine(70, "2026-09-10", "SOMETHING");
await doVoid(hold);
const res = await merge(req({ other_id: stmt, mode: "reconcile" }), params(hold));
expect(res.status).toBe(409);
});
});
@@ -0,0 +1,176 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants } from "./helpers";
/**
* The mirror of receipt-adopts-wallet-row: a wallet notification arriving AFTER
* something else already booked the purchase must fold into that row.
*
* Built from the failure, not from the design. The 2026-08-18 Woolworths shop
* ($103.30, Visa gift card) reached the ledger twice: txn 6858 from the pantry
* receipt on the 19th, and txn 6927 from the wallet lane on the 22nd, when the
* first 21 wallet events were backfilled days after their taps. The sequencing
* the lane is designed around -- wallet first, receipt second -- is a tendency,
* and every writer has to survive it not holding.
*
* Two rules are load-bearing here and are tested separately below:
* - the merchant must agree, because losing the "no line items" guard that
* protects the other direction means amount+date alone could swallow a tap
* into an unrelated purchase of the same value
* - a gift-card tap is `credits`, because `card` means "a statement line is
* coming" and the reconcile queue filters on exactly that
*/
const TOKEN = "test-wallet-token";
let pool: Pool;
let ownerId: number;
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
const req = (body: unknown) =>
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
/** A row shaped the way the pantry receipt lane writes one: statementless,
* itemised, no location (nothing but a notification supplies a suburb). */
async function receiptRow(amount: number, date: string, merchant: string, opts: { items?: number; meta?: boolean } = {}) {
const { items = 3, meta = true } = opts;
const t = await pool.query(
`INSERT INTO transactions
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
transaction_type, category, payment_method, merchant_name, merchant_normalized, row_index)
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'groceries', 'credits', $3, $3, 0)
RETURNING id`,
[ownerId, date, merchant, amount]
);
const id = t.rows[0].id as number;
if (meta) {
const lineItems = Array.from({ length: items }, (_, i) => ({ description: `item ${i}`, qty: 1, amount: 1 }));
await pool.query(
`INSERT INTO expense_metadata
(transaction_id, source, merchant_normalized, amount, transaction_date, currency, line_items, flags)
VALUES ($1, 'pantry', $2, $3, $4, 'AUD', $5::jsonb, '[]'::jsonb)`,
[id, merchant, amount, date, JSON.stringify(lineItems)]
);
}
return id;
}
/** The real event 80: the Apple Wallet half of a Visa gift-card tap. */
const woolworthsGiftCard = {
event_id: 80, kind: "transaction", direction: "unknown", amount: 103.3,
amount_text: "$103.30", currency: "$?", merchant: "Woolworths", issuer: "Visa",
notif_app: "Wallet", notif_title: "Visa Gift Card",
notif_subtitle: "Woolworths, Wyndham Vale, VIC",
occurred_at: "2026-08-18T09:00:14+10:00",
};
const rows = async () =>
(await pool.query(
`SELECT id, amount, merchant_normalized, location, source, source_ref, payment_method
FROM transactions ORDER BY id`
)).rows;
beforeAll(async () => {
process.env.WALLET_INGEST_TOKEN = TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
describe("a wallet notification for a purchase the ledger already holds", () => {
it("adopts the receipt's row instead of creating a second one", async () => {
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
const body = await (await ingest(req(woolworthsGiftCard))).json();
expect(body.skipped).toBe("adopted");
expect(body.transactionId).toBe(existing);
expect(await rows()).toHaveLength(1);
});
it("gives the adopted row the suburb, and leaves what the receipt read alone", async () => {
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
await ingest(req(woolworthsGiftCard));
const [t] = await rows();
// Wallet's own field: nothing else on the estate writes a suburb.
expect(t.location).toBe("Wyndham Vale, VIC");
// The receipt read the register tape. The notification does not get to
// rename the merchant to its own shorter string.
expect(t.merchant_normalized).toBe("WOOLWORTHS 3345");
expect(t.id).toBe(existing);
const [m] = (await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
expect(m.source).toBe("pantry");
expect(m.line_items).toHaveLength(3);
expect(m.payment_method_detail).toBe("Visa");
expect(m.flags).toContain("wallet_evt:80");
});
it("is idempotent when the interpreter replays every event", async () => {
// Bumping prompt_version re-interprets and re-posts the whole capture
// history on purpose -- that is the replay mechanism. An adopted row keeps
// no wallet source_ref of its own, so the flag is what makes replay safe.
await ingest(req(woolworthsGiftCard));
const second = await (await ingest(req(woolworthsGiftCard))).json();
expect(second.skipped).toBe("already_ingested");
expect(await rows()).toHaveLength(1);
});
it("writes its own row when the merchant disagrees", async () => {
// Same money, same day, different shop. Folding into it would make the tap
// vanish -- worse than a duplicate, which is at least visible in the queue.
await receiptRow(103.3, "2026-08-18", "Bunnings Werribee");
const body = await (await ingest(req(woolworthsGiftCard))).json();
expect(body.skipped).toBeUndefined();
expect(await rows()).toHaveLength(2);
});
it("writes its own row when two candidates share the amount", async () => {
await receiptRow(103.3, "2026-08-18", "Woolworths Wyndham Vale");
await receiptRow(103.3, "2026-08-18", "Woolworths Point Cook");
const body = await (await ingest(req(woolworthsGiftCard))).json();
expect(body.skipped).toBeUndefined();
expect(await rows()).toHaveLength(3);
});
it("creates the metadata row when the adopted transaction has none", async () => {
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345", { meta: false });
await ingest(req(woolworthsGiftCard));
const meta = (await pool.query(`SELECT * FROM expense_metadata`)).rows;
expect(meta).toHaveLength(1);
expect(meta[0].transaction_id).toBe(existing);
expect(meta[0].card_last4).toBeNull();
expect(meta[0].flags).toContain("wallet_evt:80");
});
});
describe("a stored-value instrument", () => {
it("is booked as credits, so it never enters the reconcile queue", async () => {
const body = await (await ingest(req(woolworthsGiftCard))).json();
const [t] = await rows();
expect(body.transactionId).toBeTruthy();
// 'card' would park it forever: findAutoMatches and the queue both exclude
// 'credits' and 'cash' precisely because no statement line is coming.
expect(t.payment_method).toBe("credits");
expect(body.flags).toContain("stored_value_instrument");
});
it("leaves an ordinary card tap as card", async () => {
const body = await (await ingest(req({ ...woolworthsGiftCard, notif_title: "Payment", issuer: "Mastercard", notif_app: "Westpac" }))).json();
const [t] = await rows();
expect(t.payment_method).toBe("card");
expect(body.flags).not.toContain("stored_value_instrument");
});
});
@@ -0,0 +1,403 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants } from "./helpers";
/**
* Wallet notifications becoming transactions.
*
* Every fixture is a real captured event from capture.wallet_events
* (2026-08-13 .. 08-22). The rules being tested are the ones the design turned
* on, so each has a reason rather than being a shape check:
*
* - a transaction for EVERY tap, because overseas is where live spend matters
* - the pair is MERGED, never deduped: the issuer half has the card tail, the
* Wallet half has the merchant and suburb, neither alone is complete
* - the NATIVE amount is what gets stored for a foreign charge, because the
* AUD figure is the bank's own conversion and cannot be predicted
* - a bare "$" is recorded as unknown, never assumed AUD
*/
const TOKEN = "test-wallet-token";
let pool: Pool;
let ownerId: number;
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
const req = (body: unknown) =>
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
const noAuth = (body: unknown) =>
({ headers: { get: () => null }, json: async () => body }) as never;
// --- real captured events -------------------------------------------------
const breadworldBank = {
event_id: 64, kind: "transaction", direction: "debit", amount: 13.5,
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
card_tail: "8032", issuer: "Mastercard", notif_app: "Westpac",
notif_body: "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU",
occurred_at: "2026-08-13T08:01:28+10:00",
};
const breadworldWallet = {
event_id: 65, kind: "transaction", direction: "unknown", amount: 13.5,
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
notif_app: "Wallet", notif_subtitle: "Breadworld Bakery Cafe, Wyndham Vale, VIC",
occurred_at: "2026-08-13T08:01:35+10:00",
};
const amexRewardPay = {
event_id: 87, kind: "transaction", direction: "debit", amount: 5103.49,
amount_text: "A$5,103.49", currency: "AUD", merchant: "REWARDPAY",
card_tail: "01000", issuer: "American Express", notif_app: "Amex AU",
notif_body: "You have made a A$5,103.49 charge on your American Express® Card ending in 01000 at REWARDPAY.",
occurred_at: "2026-08-20T11:54:43+10:00",
};
const ebayOrderPlaced = {
event_id: 66, kind: "not_transaction", direction: "unknown", amount: null,
notif_app: "Wallet", notif_body: "eBay has received your order.",
occurred_at: "2026-08-13T09:45:55+10:00",
};
/** Only the rows this lane wrote -- some tests seed statement history for the
* rate reference, and those must not be mistaken for wallet output. */
const txns = async () =>
(await pool.query(
`SELECT id, amount, amount_aud, merchant_normalized, location, source, source_ref,
foreign_currency_amount, foreign_currency_code, payment_method, statement_id
FROM transactions WHERE source = 'wallet' ORDER BY id`
)).rows;
const metas = async () =>
(await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
beforeAll(async () => {
process.env.WALLET_INGEST_TOKEN = TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
});
beforeEach(async () => {
// No network in tests. The ECB fallback treats any fetch failure as "no
// rate", which is what every pre-existing currency test assumes; the one test
// about the fallback stubs a rate of its own.
vi.stubGlobal("fetch", async () => { throw new Error("no network in tests"); });
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
describe("auth", () => {
it("fails closed without the token", async () => {
expect((await ingest(noAuth(breadworldBank))).status).toBe(401);
expect(await txns()).toHaveLength(0);
});
});
describe("what reaches the ledger", () => {
it("creates a statementless card transaction for a tap", async () => {
const body = await (await ingest(req(breadworldBank))).json();
expect(body.transactionId).toBeTruthy();
const [t] = await txns();
expect(Number(t.amount)).toBe(13.5);
expect(t.source).toBe("wallet");
expect(t.source_ref).toBe("evt:64");
// Statementless + card is exactly what puts it in the reconcile queue, with
// no extra wiring: that is how the statement eventually confirms it.
expect(t.statement_id).toBeNull();
expect(t.payment_method).toBe("card");
});
it("ignores an eBay order-lifecycle notification", async () => {
// 10 of the 13 non-transactions in the capture window are these.
const body = await (await ingest(req(ebayOrderPlaced))).json();
expect(body.skipped).toBe("not_a_transaction");
expect(await txns()).toHaveLength(0);
});
it("is idempotent on the event id", async () => {
await ingest(req(breadworldBank));
const second = await (await ingest(req(breadworldBank))).json();
expect(second.skipped).toBe("already_ingested");
expect(await txns()).toHaveLength(1);
});
});
describe("the notification pair", () => {
it("merges the two halves into ONE transaction, keeping both contributions", async () => {
await ingest(req(breadworldBank)); // has the card tail
await ingest(req(breadworldWallet)); // has the suburb
const rows = await txns();
expect(rows).toHaveLength(1);
// The Wallet half supplies the suburb; the issuer half never had one.
expect(rows[0].location).toBe("Wyndham Vale, VIC");
const [m] = await metas();
// ...and the issuer half's card tail survives the merge.
expect(m.card_last4).toBe("8032");
expect(m.payment_method_detail).toBe("Mastercard");
});
it("merges in either arrival order", async () => {
await ingest(req(breadworldWallet));
await ingest(req(breadworldBank));
expect(await txns()).toHaveLength(1);
const [m] = await metas();
expect(m.card_last4).toBe("8032");
const [t] = await txns();
expect(t.location).toBe("Wyndham Vale, VIC");
});
});
describe("the notification pair is recognised by time and app, not amount alone", () => {
it("makes ONE row when both halves are posted at the same moment", async () => {
// n8n posts a poll's items concurrently, and both halves of a tap usually
// land in the same poll. Before the lock, each ran its pair lookup before
// the other inserted: 11 taps became 22 rows, identical created_at on each.
await Promise.all([ingest(req(breadworldBank)), ingest(req(breadworldWallet))]);
expect(await txns()).toHaveLength(1);
const [m] = await metas();
expect(m.card_last4).toBe("8032");
});
it("keeps two same-amount charges hours apart as two purchases", async () => {
// Event 275 ($50 at A MEDIUM) merged into the UNICEF $50 donation from 17
// hours earlier and renamed it, because the old window was +/-1 day.
await ingest(req({
...amexRewardPay, event_id: 267, amount: 50, amount_text: "A$50.00", merchant: "UNICEF",
notif_body: "You have made a A$50.00 charge on your American Express® Card ending in 01000 at UNICEF.",
occurred_at: "2026-10-01T11:02:40+10:00",
}));
await ingest(req({
...amexRewardPay, event_id: 275, amount: 50, amount_text: "$50.00", merchant: "A MEDIUM",
notif_body: "You have made a $50.00 charge on your American Express® Card ending in 01000 at A MEDIUM.",
occurred_at: "2026-10-02T04:06:49+10:00",
}));
const rows = await txns();
expect(rows).toHaveLength(2);
expect(rows.map((r) => r.merchant_normalized).sort()).toEqual(["A MEDIUM", "UNICEF"]);
});
it("never pairs two notifications from the SAME app, however close", async () => {
// A pair is the issuer half plus the Wallet half. Two from one app are two
// charges -- two $20 coffees bought together, say.
await ingest(req(breadworldBank));
await ingest(req({ ...breadworldBank, event_id: 640, occurred_at: "2026-08-13T08:01:40+10:00" }));
expect(await txns()).toHaveLength(2);
});
it("records a merged half, so a replay cannot merge it again", async () => {
await ingest(req(breadworldBank));
await ingest(req(breadworldWallet));
const again = await (await ingest(req(breadworldWallet))).json();
expect(again.skipped).toBe("already_ingested");
const [m] = await metas();
expect(m.flags).toContain("wallet_evt:65");
});
});
describe("notifications that name an amount but are not purchases", () => {
it("skips an overdraft notice", async () => {
// Event 235 became a $21.21 purchase named "To avoid fees and interest,".
const body = await (await ingest(req({
event_id: 235, kind: "transaction", direction: "debit", amount: 21.21,
amount_text: "$21.21", currency: "$?", merchant: null, notif_app: "CommBank",
notif_body: "Your Smart Access 1910 is overdrawn by $21.21. To avoid fees and interest, bring your account to positive by midnight 21 Sep (Syd/Melb time).",
occurred_at: "2026-09-22T09:00:18+10:00",
}))).json();
expect(body.skipped).toBe("account_notice");
expect(await txns()).toHaveLength(0);
});
it("skips a transit-card fare, because the card's load is the spend", async () => {
// Event 200: the Octopus load is on the bank statement; each fare draws it
// down. 11 of 13 fares were already not_transaction; this makes the other 2 agree.
const body = await (await ingest(req({
event_id: 200, kind: "transaction", direction: "debit", amount: 105,
amount_text: "HKD 105.00", currency: "HKD", merchant: "Octopus", notif_app: "Wallet",
notif_title: "Octopus", notif_subtitle: "HKD 105.00 for Train",
notif_body: "Your new balance is -HKD 12.60.",
occurred_at: "2026-09-14T02:00:37+10:00",
}))).json();
expect(body.skipped).toBe("stored_value_drawdown");
expect(await txns()).toHaveLength(0);
});
it("does not read a Westpac descriptor out of another bank's message", async () => {
await ingest(req({
event_id: 2350, kind: "transaction", direction: "debit", amount: 9.5,
amount_text: "$9.50", currency: "$?", merchant: null, notif_app: "CommBank",
notif_body: "Payment of $9.50 made. Thanks for banking with us.",
occurred_at: "2026-09-22T09:00:18+10:00",
}));
const [t] = await txns();
expect(t.merchant_normalized).toBe("Card purchase");
});
});
describe("dating", () => {
it("dates a tap in Melbourne local time, as the statements do", async () => {
// 21:30 UTC on 30 Sep is 07:30 on 1 Oct in Melbourne. Slicing the ISO
// string filed it a day early.
await ingest(req({ ...breadworldBank, occurred_at: "2026-09-30T21:30:00Z" }));
const { rows } = await pool.query(
`SELECT transaction_date::text AS d FROM transactions WHERE source = 'wallet'`
);
expect(rows[0].d).toBe("2026-10-01");
});
});
describe("currency", () => {
it("falls back to the ECB rate when the ledger has no history", async () => {
// Hong Kong: no HKD statement row existed, so every tap stored A$0.00.
const calls: string[] = [];
vi.stubGlobal("fetch", async (url: string) => {
calls.push(String(url));
return { ok: true, json: async () => ({ rates: { AUD: 0.19 } }) };
});
await ingest(req({
event_id: 167, kind: "transaction", direction: "debit", amount: 310.1,
amount_text: "HKD 310.10", currency: "HKD", merchant: "Wonder Taxi", notif_app: "Wallet",
notif_subtitle: "Wonder Taxi, Tsim Sha Tsui East, Kowloon",
occurred_at: "2026-09-10T08:54:00+10:00",
}));
const [t] = await txns();
expect(Number(t.amount)).toBeCloseTo(58.92, 2);
expect(Number(t.foreign_currency_amount)).toBe(310.1);
expect(calls[0]).toContain("/2026-09-10?from=HKD&to=AUD");
const [m] = await metas();
// Flagged as the mid-rate, not the bank's: it lacks the card's FX fee.
expect(m.flags).toEqual(expect.arrayContaining(["aud_provisional", "rate_ecb"]));
expect(m.flags).not.toContain("no_rate_reference");
});
it("records AUD when Amex states it explicitly", async () => {
await ingest(req(amexRewardPay));
const [t] = await txns();
expect(t.foreign_currency_code).toBeNull();
expect(Number(t.amount)).toBe(5103.49);
const [m] = await metas();
expect(m.currency).toBe("AUD");
expect(m.flags).not.toContain("currency_unverified");
});
it("flags a bare dollar sign rather than assuming AUD", async () => {
// 14 of 21 captured transaction rows are a bare "$". Assuming AUD is
// harmless at home and a 23x error on a THB tap abroad.
await ingest(req(breadworldBank));
const [m] = await metas();
expect(m.currency).toBeNull();
expect(m.flags).toContain("currency_unverified");
});
it("stores a foreign charge on the NATIVE pair, with a provisional AUD", async () => {
// Seed enough history for the median rate to exist (n >= 5).
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf','1','Test Bank','2026-01-01','2026-01-31',$1) RETURNING id`, [ownerId]
);
for (let i = 0; i < 6; i++) {
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount,
transaction_type, owner_id, foreign_currency_amount, foreign_currency_code)
VALUES ($1,'2026-01-10','BANGKOK',4.57,'debit',$2,100,'THB')`, [st.rows[0].id, ownerId]
);
}
await ingest(req({
event_id: 999, kind: "transaction", direction: "debit", amount: 500,
amount_text: "฿500", merchant: "7-Eleven", notif_app: "Wallet",
notif_subtitle: "7-Eleven, Bangkok", occurred_at: "2026-08-20T12:00:00+10:00",
}));
const [t] = await txns();
expect(Number(t.foreign_currency_amount)).toBe(500);
expect(t.foreign_currency_code).toBe("THB");
// The native figure is exact on both sides, so it is what the statement
// will match on. amount is only ever provisional here.
expect(Number(t.amount)).toBeCloseTo(22.85, 1);
const [m] = await metas();
expect(m.flags).toContain("aud_provisional");
});
it("still records a foreign charge when there is no rate history", async () => {
await ingest(req({
event_id: 998, kind: "transaction", direction: "debit", amount: 42,
amount_text: "€42.00", merchant: "Somewhere", notif_app: "Wallet",
occurred_at: "2026-08-20T12:00:00+10:00",
}));
const [t] = await txns();
expect(t.foreign_currency_code).toBe("EUR");
const [m] = await metas();
// Says so, rather than inventing a rate.
expect(m.flags).toContain("no_rate_reference");
// And never stores the native figure as AUD (672006 VND became A$672,006).
expect(Number(t.amount)).toBe(0);
expect(Number(t.foreign_currency_amount)).toBe(42);
});
it("derives the rate from statement rows only, never from wallet rows", async () => {
// Six statementless rows at a nonsense 1:1 rate must not count as history.
for (let i = 0; i < 6; i++) {
await pool.query(
`INSERT INTO transactions (transaction_date, description, amount, transaction_type,
owner_id, foreign_currency_amount, foreign_currency_code, source, source_ref)
VALUES ('2026-01-10','VN',1000,'debit',$1,1000,'VND','wallet',$2)`,
[ownerId, `evt:seed${i}`]
);
}
await ingest(req({
event_id: 997, kind: "transaction", direction: "debit", amount: 672006,
amount_text: "672006 VND", currency: "VND", merchant: "EVISA VIETNAM", notif_app: "CommBank",
occurred_at: "2026-09-05T11:22:00+10:00",
}));
const t = (await txns()).find((r) => r.source_ref === "evt:997")!;
expect(Number(t.amount)).toBe(0);
const [m] = (await metas()).filter((r) => r.source_message_id === "evt:997");
expect(m.flags).toContain("no_rate_reference");
});
it("pairs foreign halves on the native figure, not the placeholder amount", async () => {
await ingest(req({
event_id: 996, kind: "transaction", direction: "debit", amount: 42,
amount_text: "€42.00", merchant: "Cafe A", notif_app: "Wallet",
occurred_at: "2026-08-20T12:00:00+10:00",
}));
await ingest(req({
event_id: 995, kind: "transaction", direction: "debit", amount: 17,
amount_text: "€17.00", merchant: "Cafe B", notif_app: "Wallet",
occurred_at: "2026-08-20T12:05:00+10:00",
}));
// Both have amount 0; they are different purchases and must stay two rows.
expect((await txns()).filter((r) => r.foreign_currency_code === "EUR")).toHaveLength(2);
});
});
describe("location", () => {
it("takes the suburb from the Wallet subtitle", async () => {
await ingest(req(breadworldWallet));
const [t] = await txns();
expect(t.location).toBe("Wyndham Vale, VIC");
});
it("falls back to the fixed-width descriptor, country included", async () => {
// The issuer half has no subtitle; the country code is positional.
await ingest(req({
...breadworldBank, event_id: 75, amount: 42.44,
merchant: "SP PEAKDESIGN",
// The model's own answer for this row, included because every real
// interpretation carries one -- omitting it is what let a wrong
// precedence order pass this test the first time, and ship
// "SAN FRANCISOUS" into the ledger.
location: "SAN FRANCISOUS",
notif_body: "$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS",
}));
const [t] = await txns();
expect(t.location).toBe("SAN FRANCISO US");
const [m] = await metas();
// The only foreign merchant in the whole capture window — worth surfacing.
expect(m.flags).toContain("foreign_country_US");
});
});
+197 -2
View File
@@ -3,17 +3,22 @@ import { readFileSync } from "fs";
import { resolve } from "path"; import { resolve } from "path";
import { import {
classifyInput, classifyInput,
confabulationReason,
GROCERY_PATTERN,
merchantKey, merchantKey,
merchantMatches, merchantMatches,
platformForSource, platformForSource,
PROMPT_PLACEHOLDER_MERCHANTS,
quoteShared,
reconcileItems, reconcileItems,
resolveMerchantKind, resolveMerchantKind,
textCouldBeReceipt,
toLineItems, toLineItems,
validateHeader, validateHeader,
type ExtractedOrder, type ExtractedOrder,
} from "../../lib/intake-rules"; } from "../../lib/intake-rules";
import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT } from "../../lib/intake-extract"; import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT, ExtractionError, liteLLMExtractor, ORDER_ITEM_PROMPT } from "../../lib/intake-extract";
import { askCard, candidatesCard, enrichedCard } from "../../lib/intake-blocks"; import { askCard, candidatesCard, enrichedCard, type Block } from "../../lib/intake-blocks";
/** /**
* The intake router's pure rules, against the strings real receipts print. * The intake router's pure rules, against the strings real receipts print.
@@ -85,6 +90,17 @@ describe("merchant pattern — grocery | food | ask", () => {
expect(merchantMatches("Burgertory (Werribee)", "Coles")).toBe(false); expect(merchantMatches("Burgertory (Werribee)", "Coles")).toBe(false);
expect(merchantMatches(null, "Coles")).toBe(false); expect(merchantMatches(null, "Coles")).toBe(false);
}); });
it("whole words, not substrings: `Bigals Pizza` contains `iga` and is neither a grocer nor an IGA", () => {
expect(GROCERY_PATTERN.test(merchantKey("Bigals Pizza"))).toBe(false);
expect(resolveMerchantKind("uber_eats", "Bigals Pizza")).toBe("food");
expect(merchantMatches("Bigals Pizza", "IGA")).toBe(false);
// and the grocers a boundary must not cost us
expect(resolveMerchantKind("uber_eats", "IGA Wyndham Vale")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Woolworths Group Limited")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Coles (Wyndham Vale)")).toBe("grocery");
expect(merchantMatches("IGA Wyndham Vale", "IGA")).toBe(true);
});
}); });
describe("validateHeader — nothing is written from a half-read upload", () => { describe("validateHeader — nothing is written from a half-read upload", () => {
@@ -221,4 +237,183 @@ describe("cards carry the C5 action ids and JSON values", () => {
expect(ids).toContain('"action_id":"intake_grocery"'); expect(ids).toContain('"action_id":"intake_grocery"');
expect(ids).toContain('"action_id":"intake_meal"'); expect(ids).toContain('"action_id":"intake_meal"');
}); });
it("every value the card did not write is escaped, plain_text included", () => {
const hostile = { ...c, merchant: "Ben & Jerry's <fake>" };
const { text, blocks } = candidatesCard(hostile, [
{ transaction_id: 42, description: "SQ *BEN & JERRY <AU>", transaction_date: "2026-08-17", amount: "19.90" },
]);
const sections = JSON.stringify(blocks.filter((b) => (b as Block).type === "section"));
expect(sections).toContain("Ben &amp; Jerry's &lt;fake&gt;");
expect(sections).toContain("SQ *BEN &amp; JERRY &lt;AU&gt;");
expect(text).toContain("Ben &amp; Jerry's &lt;fake&gt;");
// plain_text disables formatting, not entity decoding — &, < and > are
// control characters everywhere in a Slack payload.
const whole = JSON.stringify(blocks);
expect(whole).not.toContain("<fake>");
expect(whole).not.toContain("<AU>");
});
it("the currency is model output too, so it is escaped like any other", () => {
// parseExtraction accepts any string for `currency`; unescaped, `<@U123>`
// would be a live mention on every card the batch produces.
const { text } = enrichedCard({ ...c, currency: "<@U123>", transactionId: 7 });
expect(text).toContain("&lt;@U123&gt; 22.40");
expect(text).not.toContain("<@U123>");
});
it("a foreign receipt is not printed as dollars — header and candidate rows alike", () => {
expect(enrichedCard({ ...c, currency: "USD", transactionId: 7 }).text).toBe(
"Added 2 items to your USD 22.40 Uber Eats (Burgertory (Werribee)) order ✓"
);
expect(enrichedCard({ ...c, currency: "AUD", transactionId: 7 }).text).toContain("$22.40");
expect(enrichedCard({ ...c, transactionId: 7 }).text).toContain("$22.40");
// A candidate row is a ledger transaction and carries its own currency; it
// used to print "$" beneath a header reading "USD 22.40".
const { blocks } = candidatesCard({ ...c, currency: "USD" }, [
{ transaction_id: 42, description: "Order - Burgertory", transaction_date: "2026-08-17", amount: "19.90", currency: "USD" },
]);
expect(JSON.stringify(blocks)).toContain("USD 19.90");
});
});
describe("liteLLMExtractor — a finish_reason that is present must be `stop`", () => {
const good = { merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4, subtotal: 28, currency: "AUD", items: [], adjustments: [] };
const extractor = (finish_reason?: string) =>
liteLLMExtractor({
baseUrl: "http://litellm.test",
key: "k",
fetchImpl: (async () =>
Response.json({
choices: [{ message: { content: JSON.stringify(good) }, ...(finish_reason ? { finish_reason } : {}) }],
})) as unknown as typeof fetch,
});
it("accepts `stop`, and — on purpose — a reply that reports no reason at all", async () => {
expect((await extractor("stop")([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
expect((await extractor()([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
});
it("rejects any other reason, naming it — the strict schema only held for a reply that finished", async () => {
// An absent reason is NOT rejected: see the comment in intake-extract.ts.
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(ExtractionError);
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/too long/);
await expect(extractor("content_filter")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/content_filter/);
await expect(extractor("tool_calls")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/tool_calls/);
});
});
describe("textCouldBeReceipt — nothing that cannot be a receipt reaches the model", () => {
const why = (t: string | null | undefined) => {
const r = textCouldBeReceipt(t);
return r.ok ? null : r.reason;
};
it("the live defect: the phone sent the WORD `Image`, not the image", () => {
expect(textCouldBeReceipt("Image").ok).toBe(false);
expect(why("Image")).toMatch(/single word/);
expect(why("Image\n")).toMatch(/single word/);
});
it("a filename is not a receipt, digits in it notwithstanding", () => {
expect(why("IMG_0042.jpeg")).toMatch(/single word/);
expect(why("Receipt_19Aug2026.pdf")).toMatch(/single word/);
});
it("empty, blank and whitespace-only", () => {
expect(why("")).toMatch(/empty/);
expect(why(" \n\t ")).toMatch(/empty/);
expect(why(null)).toMatch(/empty/);
expect(why(undefined)).toMatch(/empty/);
});
it("prose with no numbers cannot be a receipt", () => {
expect(why("Shared from Photos")).toMatch(/no numbers/);
expect(why("here is the receipt for dinner last night, sorry it took so long")).toMatch(/no numbers/);
});
it("short but with a printed amount goes through — this door accepts a pasted line", () => {
expect(textCouldBeReceipt("TOTAL $17.00").ok).toBe(true);
expect(textCouldBeReceipt("Total 17.00").ok).toBe(true);
expect(textCouldBeReceipt("Gesamt 12,50 EUR").ok).toBe(true);
});
it("short with digits but no money figure does not — a screenshot name is not a purchase", () => {
expect(why("Screenshot 2026 08 22")).toMatch(/too short/);
expect(why("2 items")).toMatch(/too short/);
});
it("a real receipt body passes", () => {
expect(
textCouldBeReceipt("Burgertory (Werribee)\n19 Aug 2026\n1x Michael 14.00\n1x Joel 14.00\nSubtotal 28.00\nTotal 22.40").ok
).toBe(true);
});
it("quoteShared flattens and truncates what goes on the card", () => {
expect(quoteShared("Image")).toBe("Image");
expect(quoteShared(" a \n b ")).toBe("a b");
expect(quoteShared("x".repeat(200))).toHaveLength(80);
expect(quoteShared("x".repeat(200)).endsWith("…")).toBe(true);
});
});
describe("confabulationReason — the belt to that pre-check", () => {
const nothing = { textParts: ["Image"], hasImages: false };
const readable = { textParts: ["Total $22.40 Burgertory (Werribee) 2026-08-19"], hasImages: false };
const order = { merchant: "Burgertory (Braybrook)", items: [{ description: "The Pride" }] };
it("flags an order extracted from nothing readable", () => {
expect(confabulationReason(nothing, order)).toMatch(/nothing shared could be read/);
});
it("does not flag one extracted from something readable", () => {
expect(confabulationReason(readable, order)).toBeNull();
});
it("does not flag an image share — the pixels are what we asked for", () => {
expect(confabulationReason({ textParts: [], hasImages: true }, order)).toBeNull();
});
it("an empty answer to an unreadable share is honest, not a confabulation", () => {
expect(confabulationReason(nothing, { merchant: null, items: [] })).toBeNull();
});
it("the prompt's own placeholder coming back as a merchant is always a confabulation", () => {
for (const p of PROMPT_PLACEHOLDER_MERCHANTS) {
expect(confabulationReason(readable, { merchant: p, items: [] })).toMatch(/prompt's own example/);
expect(confabulationReason({ textParts: [], hasImages: true }, { merchant: p, items: [] })).toMatch(/prompt's own example/);
}
});
it("a real merchant that happens to be named in the prompt's platform list is NOT flagged", () => {
expect(confabulationReason(readable, { merchant: "Coles (Wyndham Vale)", items: [] })).toBeNull();
expect(confabulationReason(readable, { merchant: "Woolworths", items: [] })).toBeNull();
});
});
describe("the prompt carries no real merchant to echo", () => {
it("names no shop it could hand back as data", () => {
expect(ORDER_ITEM_PROMPT).not.toMatch(/Burgertory/);
expect(ORDER_ITEM_PROMPT).not.toMatch(/Wyndham Vale/);
expect(ORDER_ITEM_PROMPT).toContain(PROMPT_PLACEHOLDER_MERCHANTS[0]);
});
it("tells the model what to return when the input is not a receipt", () => {
expect(ORDER_ITEM_PROMPT).toMatch(/NOT a receipt/);
expect(ORDER_ITEM_PROMPT).toMatch(/Do not construct a plausible order/);
});
});
describe("needsHeaderCard", () => {
it("names the missing field and sends a dateless in-store screenshot to the app's Share button", async () => {
const { needsHeaderCard } = await import("@/lib/intake-blocks");
const dateless = needsHeaderCard(["date"], "woolworths").text;
expect(dateless).toContain("no date");
expect(dateless).toContain("Share button");
expect(dateless).not.toContain("total");
const delivery = needsHeaderCard(["total"], "uber_eats").text;
expect(delivery).toContain("no total");
expect(delivery).toContain("part of the screen");
expect(needsHeaderCard().text).toContain("total or date");
});
}); });
@@ -0,0 +1,109 @@
import { describe, it, expect } from "vitest";
import {
parseWestpacDescriptor,
parseWalletSubtitle,
currencyFromAmountText,
} from "@/lib/wallet-descriptor";
/**
* Every input below is a real captured notification body or subtitle from
* capture.wallet_events (2026-08-13 .. 08-22), not an invented fixture. The
* model handled this one fixed-width format three different ways on this exact
* data, which is why the parsing moved into code.
*/
describe("parseWestpacDescriptor", () => {
it("reads merchant, city and country from the fixed-width block", () => {
expect(parseWestpacDescriptor(
"$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
)).toEqual({ merchant: "Breadworld Bakery Cafe", city: "WYNDHAM VAL", country: "AU" });
});
it("handles a city padded to the full field width", () => {
expect(parseWestpacDescriptor(
"$54.50 paid from Mastercard ending...8032. PLA DONATION MELBOURNE AU"
)).toEqual({ merchant: "PLA DONATION", city: "MELBOURNE", country: "AU" });
});
it("splits a 12-char city that runs into the country code with no space", () => {
// The case that defeated the model: "SAN FRANCISO" exactly fills the city
// field, so "US" follows immediately. Read as one token it becomes the
// nonsense location "SAN FRANCISOUS" and the country is lost -- on the only
// foreign merchant in the whole capture window.
expect(parseWestpacDescriptor(
"$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS"
)).toEqual({ merchant: "SP PEAKDESIGN", city: "SAN FRANCISO", country: "US" });
});
it("reads mixed-case cities the model inconsistently stripped", () => {
expect(parseWestpacDescriptor(
"$20.99 paid from Mastercard ending...8032. Spotify P45AE035CB Sydney AU"
)).toEqual({ merchant: "Spotify P45AE035CB", city: "Sydney", country: "AU" });
});
it("degrades to nulls rather than guessing", () => {
expect(parseWestpacDescriptor(null)).toEqual({ merchant: null, city: null, country: null });
expect(parseWestpacDescriptor("")).toEqual({ merchant: null, city: null, country: null });
// No sentence terminator: not this format at all.
expect(parseWestpacDescriptor("You have a new notification"))
.toEqual({ merchant: null, city: null, country: null });
});
it("does not invent a country from a short merchant-only tail", () => {
// "...IT" here is the end of a word, not Italy. A wrong country on a
// foreign transaction is worse than no country.
const r = parseWestpacDescriptor("$9.00 paid from Mastercard ending...8032. SOME MERCHANT");
expect(r.country).toBeNull();
});
});
describe("parseWalletSubtitle", () => {
it("separates merchant from suburb and state", () => {
expect(parseWalletSubtitle("Breadworld Bakery Cafe, Wyndham Vale, VIC"))
.toEqual({ merchant: "Breadworld Bakery Cafe", location: "Wyndham Vale, VIC" });
});
it("returns no location for an online merchant", () => {
// 2 of the 10 Wallet subtitles captured have no place at all.
expect(parseWalletSubtitle("Seatsaero")).toEqual({ merchant: "Seatsaero", location: null });
expect(parseWalletSubtitle("Prism Accounting Bookkeeping Services"))
.toEqual({ merchant: "Prism Accounting Bookkeeping Services", location: null });
});
it("keeps a merchant name containing its own comma out of the location", () => {
expect(parseWalletSubtitle("Big Daddy`s Fresh Ma, Manor Lakes, VIC").location)
.toBe("Manor Lakes, VIC");
});
it("handles absent input", () => {
expect(parseWalletSubtitle(null)).toEqual({ merchant: null, location: null });
expect(parseWalletSubtitle(" ")).toEqual({ merchant: null, location: null });
});
});
describe("currencyFromAmountText", () => {
it("reads Amex's explicit marker", () => {
expect(currencyFromAmountText("A$5,103.49")).toBe("AUD");
expect(currencyFromAmountText("A$13.50")).toBe("AUD");
});
it("refuses to guess a bare dollar sign", () => {
// 14 of 21 captured transaction rows are this. Assuming AUD is harmless at
// home and a 23x error on a THB tap abroad, so the row goes to the
// reconcile queue with no currency instead.
expect(currencyFromAmountText("$13.50")).toBeNull();
expect(currencyFromAmountText("$103.30")).toBeNull();
});
it("reads unambiguous symbols when a card does state one", () => {
expect(currencyFromAmountText("€24.00")).toBe("EUR");
expect(currencyFromAmountText("฿500")).toBe("THB");
expect(currencyFromAmountText("S$18.20")).toBe("SGD");
expect(currencyFromAmountText("NZ$40.00")).toBe("NZD");
});
it("handles absent input", () => {
expect(currencyFromAmountText(null)).toBeNull();
expect(currencyFromAmountText("")).toBeNull();
});
});
+44
View File
@@ -0,0 +1,44 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { getUpcomingAnnualCharges } from "@/lib/annual-charges";
/**
* Upcoming annual charges, from payments alone.
*
* Lives here rather than in home-dashboard so the detection sits beside the
* ledger it queries, is covered by this repo's integration suite, and can be
* reused by a future Slack nudge without moving. home-dashboard reads it over
* networks_internal the way its other widgets read their sources.
*
* Machine access uses the same shared secret as the dashboard's other reads;
* a browser session scopes to that user instead.
*/
export async function GET(req: NextRequest) {
// Its OWN token, with no fallback to WALLET_INGEST_TOKEN. home-dashboard's
// compose states the rule plainly -- "this app never holds write creds" -- and
// WALLET_INGEST_TOKEN unlocks POST /api/wallet/ingest and the estate-wide
// reconcile sweep. Handing that to a read-only surface to save minting a
// secret would quietly convert a dashboard into a writer. Unset means closed.
const token = process.env.DASH_READ_TOKEN;
const supplied = req.headers.get("x-ingest-token");
const machine = !!token && !!supplied && supplied === token;
let ownerId: number | null = null;
if (machine) {
// A machine caller names the owner explicitly rather than defaulting to 1:
// silently assuming an owner is how an ingested order ended up invisible in
// every scoped view while sitting in the table.
const q = req.nextUrl.searchParams.get("owner_id");
ownerId = q !== null && Number.isInteger(Number(q)) ? Number(q) : null;
if (ownerId === null) {
return NextResponse.json({ error: "owner_id is required for machine access" }, { status: 400 });
}
} else {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
ownerId = user.id;
}
const charges = await getUpcomingAnnualCharges(ownerId);
return NextResponse.json({ charges, count: charges.length });
}
+23 -4
View File
@@ -234,6 +234,16 @@ async function buildDetailsModal(transactionId: number, participantId: number) {
`SELECT merchant_normalized AS merchant, line_items `SELECT merchant_normalized AS merchant, line_items
FROM expense_metadata FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[transactionId] [transactionId]
); );
@@ -377,12 +387,21 @@ async function nudgeState(transactionId: number) {
currency: string | null; currency: string | null;
amount: string; amount: string;
}>( }>(
`SELECT em.merchant_normalized AS merchant, // LATERAL + the shared tie-break, not a bare join: `queryRow` returns
COALESCE(em.currency, 'AUD') AS currency, // rows[0] and Postgres guarantees no ordering, so a second metadata row on
// one transaction would flip the merchant and currency on every card
// refresh at random. Every sibling reader already resolves this way.
`SELECT order_ctx.merchant_normalized AS merchant,
COALESCE(order_ctx.currency, 'AUD') AS currency,
t.amount t.amount
FROM transactions t FROM transactions t
LEFT JOIN expense_metadata em LEFT JOIN LATERAL (
ON em.transaction_id = t.id OR em.matched_transaction_id = t.id SELECT em.merchant_normalized, em.currency
FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
LIMIT 1
) order_ctx ON true
WHERE t.id = $1`, WHERE t.id = $1`,
[transactionId] [transactionId]
); );
@@ -0,0 +1,155 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { applyReconcileMatch, supersedeTransaction } from "@/lib/reconcile";
/**
* Manually resolve a duplicate pair the automation cannot see.
*
* WHY THIS EXISTS: until now there was no way to merge or remove a transaction
* at all. `superseded_by_id` was READ everywhere -- analytics excludes it,
* auto-reconcile excludes it -- but written nowhere in application code; it had
* only ever been set once, by migration 0023. The reconcile queue was the sole
* merge path, and it only offers candidates within a narrow amount band, so a
* pair outside it (SP PEAKDESIGN: notified $42.44, charged $43.71 -- the card's
* 3% foreign transaction fee) was unresolvable from the UI.
*
* There is deliberately NO delete. Every child of `transactions` is
* ON DELETE CASCADE, so removing a row destroys its splits, tags and metadata
* curation. Both modes here keep the row and its children and merely stop it
* counting:
*
* reconcile -- a statementLESS row folded into its statement twin
* (`reconciled_with_id`). The everyday case.
* supersede -- two rows on the same footing, e.g. a statement imported twice
* (`superseded_by_id`). No constraint on either side's origin.
*
* Both move curation onto the survivor FIRST, via one shared implementation in
* lib/reconcile.ts, because both hide the source row from every figure and
* splits stranded on a hidden row silently stop counting.
*/
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
const { id } = await params;
const loserId = Number(id);
if (!Number.isInteger(loserId)) {
return NextResponse.json({ error: "invalid transaction id" }, { status: 400 });
}
let body: { other_id?: number; mode?: string };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
}
const otherId = Number(body.other_id);
const mode = body.mode ?? "reconcile";
if (!Number.isInteger(otherId)) {
return NextResponse.json({ error: "other_id is required" }, { status: 400 });
}
if (loserId === otherId) {
return NextResponse.json({ error: "a transaction cannot merge into itself" }, { status: 400 });
}
if (mode !== "reconcile" && mode !== "supersede") {
return NextResponse.json({ error: "mode must be 'reconcile' or 'supersede'" }, { status: 400 });
}
// Both rows must be the caller's. A statement row carries owner_id on the
// statement rather than the transaction, which is why this reads the same
// COALESCE every scoped query uses -- checking t.owner_id alone would reject
// every statement row as "not found".
const rows = await queryRaw<{
id: number;
statement_id: number | null;
reconciled_with_id: number | null;
superseded_by_id: number | null;
voided_at: Date | null;
}>(
`SELECT t.id, t.statement_id, t.reconciled_with_id, t.superseded_by_id, t.voided_at
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
WHERE t.id = ANY($1::int[]) AND COALESCE(t.owner_id, s.owner_id) = $2`,
[[loserId, otherId], user.id]
);
if (rows.length !== 2) {
return NextResponse.json({ error: "one or both transactions not found" }, { status: 404 });
}
const loser = rows.find((r) => r.id === loserId)!;
const other = rows.find((r) => r.id === otherId)!;
// Refuse a row that is already folded into something else, in either
// direction. Re-pointing it would strand the curation moved on the first
// pass -- and silently, since both columns hide the row from every figure.
for (const [label, row] of [["source", loser], ["target", other]] as const) {
if (row.reconciled_with_id !== null) {
return NextResponse.json(
{ error: `${label} transaction ${row.id} is already reconciled with ${row.reconciled_with_id}` },
{ status: 409 }
);
}
if (row.superseded_by_id !== null) {
return NextResponse.json(
{ error: `${label} transaction ${row.id} is already superseded by ${row.superseded_by_id}` },
{ status: 409 }
);
}
// A voided row is a charge that never posted. As a source it is already
// out of every figure; as a survivor it would take the other row out too.
if (row.voided_at !== null) {
return NextResponse.json(
{ error: `${label} transaction ${row.id} is voided (a charge that never posted)` },
{ status: 409 }
);
}
}
try {
if (mode === "reconcile") {
// applyReconcileMatch's contract, enforced here so a misuse is a 400 and
// not a row quietly linked the wrong way round.
if (loser.statement_id !== null) {
return NextResponse.json(
{ error: "reconcile expects the source to be a statementless row; use mode 'supersede'" },
{ status: 400 }
);
}
if (other.statement_id === null) {
return NextResponse.json(
{ error: "reconcile expects the target to be a statement row; use mode 'supersede'" },
{ status: 400 }
);
}
await applyReconcileMatch(loserId, otherId);
} else {
// Never hide a statement row behind a statementless one. The statement is
// the bank's own record; the statementless row is a notification, a
// receipt or a hand entry, and is the side that can be wrong -- a wallet
// row carries the amount BEFORE the card's foreign transaction fee, so
// burying the statement line under it would leave the ledger holding
// $42.44 for a purchase the bank charged $43.71 for, with the statement
// no longer reconciling against its own rows. The correct move is the
// other direction, which is exactly what `reconcile` does.
//
// Statement-under-statement stays allowed: that is the imported-twice
// pair migration 0023 existed for, and neither side is more true.
if (loser.statement_id !== null && other.statement_id === null) {
return NextResponse.json(
{
error:
"Refusing to hide a statement row behind a statementless one — the statement is the bank's record. " +
`Open transaction ${otherId} instead and fold it into this one.`,
},
{ status: 400 }
);
}
await supersedeTransaction(loserId, otherId);
}
} catch (e) {
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 400 });
}
return NextResponse.json({ ok: true, mode, source: loserId, target: otherId });
}
@@ -34,6 +34,14 @@ export async function GET(
-- through matched_transaction_id. Both directions have to resolve or the -- through matched_transaction_id. Both directions have to resolve or the
-- detail is missing on exactly the orders that were paid by card. -- detail is missing on exactly the orders that were paid by card.
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
-- The same order as replaceLineItems in line-items.ts and matchTransaction
-- in intake-router.ts. Both directions can resolve at once — a statement
-- line owning its own row while a delivery receipt points at it — and an
-- unordered LIMIT 1 then reads a different row from the one an enrichment
-- wrote, so a PUT that reported success showed nothing on the screen.
-- NULLS LAST because DESC is NULLS FIRST and the pointing row's
-- transaction_id is NULL.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[Number(id)] [Number(id)]
); );
@@ -0,0 +1,64 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { unvoidTransaction, voidTransaction } from "@/lib/reconcile";
/**
* POST /api/transactions/<id>/void { reason } -- a charge that never posted
* DELETE /api/transactions/<id>/void -- undo it
*
* For a statementless row whose charge never reached a statement: an
* authorisation hold the merchant released (txn 7951, a cancelled Airbnb
* booking). The row stays, with its children and audit trail, and drops out of
* every figure and the reconcile queue. Rules and refusals: `voidTransaction`
* in lib/reconcile.ts.
*/
async function ownedRow(req: NextRequest, idParam: string) {
const user = await getCurrentUser(req);
if (!user) return { error: NextResponse.json({ error: "Unauthorized" }, { status: 403 }) };
const id = Number(idParam);
if (!Number.isInteger(id)) {
return { error: NextResponse.json({ error: "invalid transaction id" }, { status: 400 }) };
}
// Same owner scope as every other route; a statement row carries owner_id on
// the statement. (Statement rows are refused later, but must 400, not 404.)
const rows = await queryRaw<{ id: number }>(
`SELECT t.id FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
WHERE t.id = $1 AND COALESCE(t.owner_id, s.owner_id) = $2`,
[id, user.id]
);
if (rows.length === 0) {
return { error: NextResponse.json({ error: "transaction not found" }, { status: 404 }) };
}
return { id };
}
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
const owned = await ownedRow(req, (await params).id);
if (owned.error) return owned.error;
let body: { reason?: unknown };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
}
const reason = typeof body.reason === "string" ? body.reason.trim() : "";
if (!reason) return NextResponse.json({ error: "reason is required" }, { status: 400 });
try {
await voidTransaction(owned.id, reason);
} catch (e) {
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 409 });
}
return NextResponse.json({ ok: true, id: owned.id, voided: true });
}
export async function DELETE(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
const owned = await ownedRow(req, (await params).id);
if (owned.error) return owned.error;
await unvoidTransaction(owned.id);
return NextResponse.json({ ok: true, id: owned.id, voided: false });
}
+66 -90
View File
@@ -1,6 +1,8 @@
import { NextRequest, NextResponse } from "next/server"; import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth"; import { getCurrentUser } from "@/lib/auth";
import { prisma, queryRaw } from "@/lib/db"; import { queryRaw } from "@/lib/db";
import { applyReconcileMatch } from "@/lib/reconcile";
import { findAutoMatches } from "@/lib/auto-reconcile";
export async function POST(req: NextRequest) { export async function POST(req: NextRequest) {
const user = await getCurrentUser(req); const user = await getCurrentUser(req);
@@ -17,7 +19,7 @@ export async function POST(req: NextRequest) {
// Verify all manual_ids belong to this user // Verify all manual_ids belong to this user
const manualIds = body.matches.map((m) => m.manual_id); const manualIds = body.matches.map((m) => m.manual_id);
const owned = await queryRaw<{ id: number }>( const owned = await queryRaw<{ id: number }>(
`SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND owner_id = $2`, `SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND voided_at IS NULL AND owner_id = $2`,
[manualIds, user.id] [manualIds, user.id]
); );
if (owned.length !== manualIds.length) { if (owned.length !== manualIds.length) {
@@ -25,97 +27,71 @@ export async function POST(req: NextRequest) {
} }
let reconciled = 0; let reconciled = 0;
for (const { manual_id, statement_tx_id } of body.matches) { for (const { manual_id, statement_tx_id } of body.matches) {
await prisma.$transaction(async (tx) => { await applyReconcileMatch(manual_id, statement_tx_id);
// Copy overrides: manual → statement tx
const override = await tx.transaction_overrides.findUnique({
where: { transaction_id: manual_id },
});
if (override) {
await tx.transaction_overrides.upsert({
where: { transaction_id: statement_tx_id },
update: {
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
updated_at: new Date(),
},
create: {
transaction_id: statement_tx_id,
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
},
});
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
}
// Move tags: manual → statement tx
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
if (tags.length) {
await tx.transaction_tags.createMany({
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
skipDuplicates: true,
});
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
}
// Move splits: manual → statement tx
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
if (splits.length) {
await tx.transaction_splits.createMany({
data: splits.map((s) => ({
transaction_id: statement_tx_id,
participant_id: s.participant_id,
share_percent: s.share_percent,
})),
skipDuplicates: true,
});
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
}
// Move provenance: manual → statement tx.
//
// Overrides, tags and splits above were always carried across; expense_metadata was
// the one child left behind, which did not matter while every metadata row came from
// an email that had created its own transaction. It matters now: a scanned grocery
// receipt puts its line items here, and reconciliation hides the manual row from
// every figure — so without this the shop's contents disappear at exactly the moment
// the statement line appears, and `COLES 0556 MANOR LAKES` stays as unreadable as it
// was before the receipt was ever scanned.
//
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
// or Paperless copy got there first) keeps it. The pantry row stays attached to the
// reconciled manual transaction and is flagged, rather than raising a constraint
// violation or silently overwriting the other source.
const moved = await tx.$executeRawUnsafe(
`UPDATE expense_metadata SET transaction_id = $1
WHERE transaction_id = $2
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
statement_tx_id,
manual_id
);
if (moved === 0) {
await tx.$executeRawUnsafe(
`UPDATE expense_metadata
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
WHERE transaction_id = $1`,
manual_id
);
}
// Mark manual tx as reconciled (link to statement tx)
await tx.$executeRawUnsafe(
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
statement_tx_id,
manual_id
);
});
reconciled++; reconciled++;
} }
return NextResponse.json({ reconciled }); return NextResponse.json({ reconciled });
} }
/**
* Automatic pass over the reconcile queue.
*
* Applies only pairs that are unambiguous by the strict rule in
* `findAutoMatches` -- exact amount (or exact native pair for a foreign row),
* within a few days, and unique in BOTH directions. Everything else is left in
* the queue for a person, which is the point: this exists to keep the queue
* workable as the wallet lane adds ~63 rows a month, not to clear it.
*
* Validated against the 48 reconciliations a human had already made: the rule
* reproduces all 48 and proposes nothing they did not.
*
* `?dry=1` returns what it WOULD do without writing, which is how this should
* be run the first time after any change to the matching rule.
*/
export async function PATCH(req: NextRequest) {
// Two callers, two auth modes. A person in the browser reconciles their own
// queue; the scheduled sweep has no session and carries the machine token,
// and must cover EVERY owner with a queue -- 169 of the 177 rows outstanding
// belong to the second participant, so a sweep scoped to one user would leave
// almost all of it untouched.
const machineToken = req.headers.get("x-ingest-token");
const machineOk = !!machineToken && machineToken === process.env.WALLET_INGEST_TOKEN;
let ownerIds: number[];
if (machineOk) {
ownerIds = (await queryRaw<{ owner_id: number }>(
`SELECT DISTINCT owner_id FROM transactions
WHERE statement_id IS NULL AND owner_id IS NOT NULL
AND reconciled_with_id IS NULL AND superseded_by_id IS NULL`
)).map((r) => r.owner_id);
} else {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
ownerIds = [user.id];
}
const dry = new URL(req.url).searchParams.get("dry") === "1";
const matches = (await Promise.all(ownerIds.map((id) => findAutoMatches(id)))).flat();
if (dry) {
return NextResponse.json({ dry_run: true, would_reconcile: matches.length, matches });
}
let reconciled = 0;
const failed: { manual_id: number; error: string }[] = [];
for (const m of matches) {
try {
await applyReconcileMatch(m.manual_id, m.statement_tx_id);
reconciled++;
} catch (e) {
// One bad pair must not abandon the rest -- each merge is its own
// transaction, so a failure here leaves that pair untouched and in the
// queue rather than half-applied.
failed.push({ manual_id: m.manual_id, error: e instanceof Error ? e.message : String(e) });
}
}
return NextResponse.json({ reconciled, examined: matches.length, failed });
}
+49
View File
@@ -0,0 +1,49 @@
import { NextRequest, NextResponse } from "next/server";
import { processWalletIngestion, type WalletEvent } from "@/lib/wallet-ingestion";
/**
* Machine ingest endpoint for interpreted iOS Wallet notifications.
*
* Sibling to /api/orders/ingest and /api/receipts/ingest, and deliberately
* shaped like them: a shared secret rather than a browser session, because n8n
* calls this app-to-app.
*
* Its own token so the wallet lane's credential can be rotated without touching
* the order flow, which runs on a schedule nobody is watching.
*
* n8n is transport only. The interpretation already happened (Wallet Interpret,
* CAp1s5cuzJk2mDaI, local model on Unraid because card tails are PII); this
* endpoint owns every decision about what reaches the ledger.
*/
function authorised(req: NextRequest): boolean {
const expected = process.env.WALLET_INGEST_TOKEN;
if (!expected) return false; // fail closed when unconfigured
const got = req.headers.get("x-ingest-token");
return !!got && got === expected;
}
export async function POST(req: NextRequest) {
if (!authorised(req)) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
let body: WalletEvent;
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
}
if (!Number.isInteger(body?.event_id) || typeof body?.kind !== "string") {
return NextResponse.json({ error: "event_id (int) and kind are required" }, { status: 400 });
}
try {
const result = await processWalletIngestion(body);
return NextResponse.json(result);
} catch (e) {
// A 500 here alerts #log-alerts through the workflow error handler, which
// is what should happen: a write that fails is a template or schema change,
// not something to swallow.
const message = e instanceof Error ? e.message : String(e);
return NextResponse.json({ error: message }, { status: 500 });
}
}
+34 -1
View File
@@ -1,7 +1,7 @@
"use client"; "use client";
import { useState } from "react"; import { useState } from "react";
import { usePendingReconciliations, useReconcile } from "@/lib/hooks"; import { usePendingReconciliations, useReconcile, useVoidTransaction } from "@/lib/hooks";
import type { ManualTxWithMatches, PotentialMatch } from "@/lib/queries"; import type { ManualTxWithMatches, PotentialMatch } from "@/lib/queries";
function formatDate(d: string | Date) { function formatDate(d: string | Date) {
@@ -156,6 +156,7 @@ export default function ReconcilePage() {
<span key={tag.id} className="w-2 h-2 rounded-full" style={{ backgroundColor: tag.color }} title={tag.name} /> <span key={tag.id} className="w-2 h-2 rounded-full" style={{ backgroundColor: tag.color }} title={tag.name} />
))} ))}
</div> </div>
<VoidButton id={tx.id} amount={tx.amount} onError={setError} />
</div> </div>
))} ))}
</div> </div>
@@ -243,6 +244,7 @@ function ReconcileRow({
Clear Clear
</button> </button>
)} )}
<VoidButton id={tx.id} amount={tx.amount} />
{!isSkipped && ( {!isSkipped && (
<button onClick={onSkip} className="text-xs text-zinc-500 hover:text-zinc-300 px-2 py-1 rounded hover:bg-zinc-800"> <button onClick={onSkip} className="text-xs text-zinc-500 hover:text-zinc-300 px-2 py-1 rounded hover:bg-zinc-800">
Skip (no match) Skip (no match)
@@ -296,3 +298,34 @@ function MatchRow({
</button> </button>
); );
} }
/**
* "Never posted": a released authorisation hold, e.g. a booking cancelled
* before the charge settled. Drops the row out of every figure and this queue
* (migration 0040); the row and its history are kept.
*/
function VoidButton({ id, amount, onError }: { id: number; amount: number; onError?: (msg: string) => void }) {
const voidTx = useVoidTransaction();
async function handle() {
const reason = window.prompt(
`Mark this $${Number(amount).toFixed(2)} charge as never posted? It will stop counting everywhere.\n\nReason (e.g. "hold released, booking cancelled"):`
);
if (!reason?.trim()) return;
try {
await voidTx.mutateAsync({ id, reason });
} catch (e) {
const msg = e instanceof Error ? e.message : "Void failed";
if (onError) onError(msg); else window.alert(msg);
}
}
return (
<button
onClick={handle}
disabled={voidTx.isPending}
title="The charge never posted (e.g. a released hold) — stop counting it"
className="text-xs text-zinc-500 hover:text-red-300 px-2 py-1 rounded hover:bg-zinc-800 disabled:opacity-50 flex-shrink-0"
>
{voidTx.isPending ? "Voiding…" : "Never posted"}
</button>
);
}
+2 -2
View File
@@ -463,7 +463,7 @@ export default function RulesPage() {
> >
{expandedRun === run.id ? "▾" : "▸"} {expandedRun === run.id ? "▾" : "▸"}
</button> </button>
<span className="text-zinc-300">{new Date(run.applied_at).toLocaleString()}</span> <span className="text-zinc-300">{new Date(run.applied_at).toLocaleString("en-AU")}</span>
{/* Which rule ran matters most: a merchant rename and a {/* Which rule ran matters most: a merchant rename and a
50/50 split of everything look identical as counts. */} 50/50 split of everything look identical as counts. */}
<span className="text-zinc-200"> <span className="text-zinc-200">
@@ -480,7 +480,7 @@ export default function RulesPage() {
)} )}
</div> </div>
{run.reverted_at ? ( {run.reverted_at ? (
<span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString()}</span> <span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString("en-AU")}</span>
) : ( ) : (
<button <button
onClick={() => { onClick={() => {
+12 -1
View File
@@ -1085,7 +1085,18 @@ function TransactionsContent() {
) : ( ) : (
<> <>
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)} {formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
{t.currency && t.currency !== "AUD" && ( {/*
The native figure: `foreign_currency_amount` where one
was recorded (a statement or wallet row, whose `amount`
is AUD), else `amount` on a non-AUD statement. Printing
`amount` under the foreign code put the AUD figure
behind an INR label on every wallet row.
*/}
{t.foreign_currency_code && t.foreign_currency_code !== "AUD" && t.foreign_currency_amount !== null ? (
<div className="text-[10px] text-zinc-500 mt-0.5">
{formatAmount(t.foreign_currency_amount, t.transaction_type, t.foreign_currency_code)}
</div>
) : t.currency && t.currency !== "AUD" && (
<div className="text-[10px] text-zinc-500 mt-0.5"> <div className="text-[10px] text-zinc-500 mt-0.5">
{formatAmount(t.amount, t.transaction_type, t.currency)} {formatAmount(t.amount, t.transaction_type, t.currency)}
</div> </div>
+208 -1
View File
@@ -1,8 +1,9 @@
"use client"; "use client";
import { useState } from "react"; import { useEffect, useState } from "react";
import { import {
useUpdateTransaction, useUpdateTransaction,
useMergeTransaction,
useTags, useTags,
useAddTransactionTag, useAddTransactionTag,
useRemoveTransactionTag, useRemoveTransactionTag,
@@ -93,6 +94,210 @@ function InlineTags({ transactionId, initialTags }: { transactionId: number; ini
); );
} }
/**
* Determines which merge mode the backend will accept for (loser, candidate).
* Mirrors the constraint enforced server-side in the merge route: reconcile
* requires a statementless loser folding into a statement survivor; every
* other pairing is a supersede.
*/
function mergeModeFor(loser: TransactionRow, candidate: TransactionRow): "reconcile" | "supersede" {
return loser.statement_id === null && candidate.statement_id !== null ? "reconcile" : "supersede";
}
/**
* Collapsed by default, at the bottom, visually separate — this is the one
* control on the modal that changes which rows this transaction can ever be
* again. `transaction` is always the row that will be hidden (the loser);
* the row picked here is the survivor (`other_id`).
*/
function DuplicateSection({ transaction, onMerged }: { transaction: TransactionRow; onMerged: () => void }) {
const [expanded, setExpanded] = useState(false);
const [query, setQuery] = useState("");
const [results, setResults] = useState<TransactionRow[]>([]);
const [searching, setSearching] = useState(false);
const [selected, setSelected] = useState<TransactionRow | null>(null);
const mergeTxn = useMergeTransaction();
useEffect(() => {
if (!expanded || selected) return;
const q = query.trim();
// No synchronous setState here: an empty box is handled at render time, and
// `searching` is raised inside the timeout. Clearing state here instead
// cascades a render on every keystroke (react-hooks lint catches it).
if (q.length === 0) return;
const handle = setTimeout(() => {
setSearching(true);
fetch(`/api/transactions?search=${encodeURIComponent(q)}&limit=8`)
.then((res) => res.json())
.then((body: { data?: TransactionRow[] }) => {
setResults((body.data ?? []).filter((r) => r.id !== transaction.id));
})
.catch(() => setResults([]))
.finally(() => setSearching(false));
}, 300);
return () => clearTimeout(handle);
}, [query, expanded, selected, transaction.id]);
function reset() {
setExpanded(false);
setQuery("");
setResults([]);
setSelected(null);
mergeTxn.reset();
}
async function handleConfirm() {
if (!selected) return;
try {
await mergeTxn.mutateAsync({
id: transaction.id,
other_id: selected.id,
mode: mergeModeFor(transaction, selected),
});
onMerged();
} catch {
// Surfaced below via mergeTxn.error — section stays open so they can retry.
}
}
const mode = selected ? mergeModeFor(transaction, selected) : null;
// Hiding a statement row behind a statementless one is always backwards — the
// bank's record is the side that survives. The API refuses it too; catching it
// here just means the button explains itself instead of returning a 400.
const wrongDirection =
selected !== null && transaction.statement_id !== null && selected.statement_id === null;
return (
<div className="pt-4 border-t border-zinc-800">
{!expanded ? (
<button
type="button"
onClick={() => setExpanded(true)}
className="text-xs text-zinc-500 hover:text-zinc-300"
>
Mark as duplicate…
</button>
) : (
<div className="space-y-2.5">
<div className="flex items-center justify-between">
<p className="font-semibold text-xs text-zinc-400">Mark as duplicate</p>
<button
type="button"
onClick={reset}
className="text-xs text-zinc-500 hover:text-zinc-300"
>
Cancel
</button>
</div>
{!selected ? (
<>
<input
type="text"
value={query}
onChange={(e) => setQuery(e.target.value)}
placeholder="Search for the surviving transaction…"
autoFocus
className="w-full bg-zinc-800 border border-zinc-700 rounded px-2 py-1.5 text-sm"
/>
{searching && <p className="text-xs text-zinc-500">Searching…</p>}
{!searching && query.trim().length > 0 && results.length === 0 && (
<p className="text-xs text-zinc-600 italic">No matches</p>
)}
{/* Gated on the query too: results are no longer cleared when the
box empties, so without this the last search's rows linger. */}
{query.trim().length > 0 && results.length > 0 && (
<div className="space-y-1 max-h-48 overflow-y-auto">
{results.map((r) => (
<button
key={r.id}
type="button"
onClick={() => setSelected(r)}
className="w-full text-left px-2 py-1.5 rounded bg-zinc-800/50 hover:bg-zinc-800 border border-zinc-700/50"
>
<div className="flex items-center justify-between gap-2">
<span className="text-sm text-zinc-200 truncate">{r.description}</span>
<span
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(r.transaction_type) ? "text-red-400" : "text-green-400"}`}
>
{formatAmount(r.amount, r.transaction_type)}
</span>
</div>
<div className="flex items-center justify-between mt-0.5">
<span className="text-xs text-zinc-500">
{new Date(r.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
</span>
<span className="text-xs text-zinc-500">
{r.statement_id !== null ? "Statement row" : "Statementless"}
</span>
</div>
</button>
))}
</div>
)}
</>
) : (
<div className="space-y-2">
<div className="px-2 py-1.5 rounded bg-zinc-800/50 border border-zinc-700/50">
<div className="flex items-center justify-between gap-2">
<span className="text-sm text-zinc-200 truncate">{selected.description}</span>
<span
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(selected.transaction_type) ? "text-red-400" : "text-green-400"}`}
>
{formatAmount(selected.amount, selected.transaction_type)}
</span>
</div>
<div className="flex items-center justify-between mt-0.5">
<span className="text-xs text-zinc-500">
{new Date(selected.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
</span>
<span className="text-xs text-zinc-500">
{selected.statement_id !== null ? "Statement row" : "Statementless"}
</span>
</div>
</div>
{wrongDirection ? (
<p className="text-xs text-red-400">
This is the wrong way round — the statement is the bank&apos;s record, so it
has to be the row that survives. Close this and open{" "}
<span className="text-zinc-300">{selected.description}</span> instead, then
fold that one into this.
</p>
) : (
<p className="text-xs text-amber-400/90">
{mode === "reconcile"
? "This row will be folded into the statement row above."
: "This row will be marked a duplicate of the row above."}
</p>
)}
{mergeTxn.error && (
<p className="text-red-400 text-xs">{(mergeTxn.error as Error).message}</p>
)}
<div className="flex gap-2">
<button
type="button"
onClick={() => setSelected(null)}
className="px-3 py-1.5 bg-zinc-800 hover:bg-zinc-700 text-zinc-300 rounded-lg text-xs"
>
Choose a different row
</button>
<button
type="button"
onClick={handleConfirm}
disabled={mergeTxn.isPending || wrongDirection}
className="px-3 py-1.5 bg-amber-700 hover:bg-amber-600 disabled:opacity-50 text-white rounded-lg text-xs"
>
{mergeTxn.isPending ? "Merging…" : mode === "reconcile" ? "Fold into statement row" : "Mark as duplicate"}
</button>
</div>
</div>
)}
</div>
)}
</div>
);
}
export function EditTransactionModal({ export function EditTransactionModal({
transaction, transaction,
onClose, onClose,
@@ -416,6 +621,8 @@ export function EditTransactionModal({
<OrderDetails transactionId={transaction.id} currency={transaction.currency ?? null} /> <OrderDetails transactionId={transaction.id} currency={transaction.currency ?? null} />
<DuplicateSection transaction={transaction} onMerged={onClose} />
</div> </div>
{/* Footer */} {/* Footer */}
+4 -7
View File
@@ -13,12 +13,9 @@ import {
type OrderReceipt, type OrderReceipt,
type OrderRating, type OrderRating,
} from "@/lib/hooks"; } from "@/lib/hooks";
// One platform vocabulary, shared with the intake cards — three copies had
const PLATFORM_LABEL: Record<string, string> = { // already drifted apart (the intake one was missing `uber`).
doordash: "DoorDash", import { platformLabel } from "@/lib/intake-rules";
ubereats: "Uber Eats",
uber: "Uber",
};
/** /**
* Who records verdicts. A two-person household with one primary user: the * Who records verdicts. A two-person household with one primary user: the
@@ -99,7 +96,7 @@ export function OrderDetails({
<p className="text-xs text-zinc-500"> <p className="text-xs text-zinc-500">
Order details Order details
{receipt.platform && ( {receipt.platform && (
<span className="ml-1.5 text-zinc-400">{PLATFORM_LABEL[receipt.platform] ?? receipt.platform}</span> <span className="ml-1.5 text-zinc-400">{platformLabel(receipt.platform)}</span>
)} )}
</p> </p>
{receipt.card_last4 && ( {receipt.card_last4 && (
+17 -2
View File
@@ -71,9 +71,17 @@ export const STATEMENTS_JOIN = `LEFT JOIN statements s ON s.id = t.statement_id`
* without being edited. Anything summing transactions without this fragment * without being edited. Anything summing transactions without this fragment
* still double-counts — that is the same gap that let reconciled rows into the * still double-counts — that is the same gap that let reconciled rows into the
* analytics routes in the first place. * analytics routes in the first place.
*
* **3. The charge that never posted** (`voided_at`, migration 0040).
*
* A card notification is sent for an authorisation, and a released hold never
* reaches a statement (txn 7951: a $1,040.41 Airbnb hold on a booking cancelled
* the next day). There is no survivor to point at, so it gets its own column —
* and it lives in this fragment for the same reason case 2 does.
*/ */
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL) export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
AND t.superseded_by_id IS NULL`; AND t.superseded_by_id IS NULL
AND t.voided_at IS NULL`;
/** /**
* The currency `t.amount` is actually denominated in. * The currency `t.amount` is actually denominated in.
@@ -89,10 +97,17 @@ export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reco
* figure and `foreign_currency_code` names it (order-ingestion.ts leaves * figure and `foreign_currency_code` names it (order-ingestion.ts leaves
* `amount_aud` NULL rather than asserting an FX rate it does not have). * `amount_aud` NULL rather than asserting an FX rate it does not have).
* *
* - A wallet row also has no statement, but follows the STATEMENT convention:
* `amount` = `amount_aud` = a provisional AUD figure, and the native figure
* lives in `foreign_currency_amount` (wallet-ingestion.ts). `amount_aud`
* being present is what says `amount` is already AUD. Without that arm, an
* INR 524 tap stored as A$9.42 rendered as "INR 9.42" -- the AUD number
* under the foreign label (txn 8162, 2026-10-03).
*
* Reading `s.currency` alone labels every foreign order row as AUD. * Reading `s.currency` alone labels every foreign order row as AUD.
* `s` must be the statements alias in scope. * `s` must be the statements alias in scope.
*/ */
export const NATIVE_CURRENCY = `COALESCE(s.currency, t.foreign_currency_code, 'AUD')`; export const NATIVE_CURRENCY = `COALESCE(s.currency, CASE WHEN t.amount_aud IS NULL THEN t.foreign_currency_code END, 'AUD')`;
/** /**
* True when a row's AUD value is unknown: it is denominated in something other * True when a row's AUD value is unknown: it is denominated in something other
+275
View File
@@ -0,0 +1,275 @@
import { queryRaw } from "@/lib/db";
/**
* Upcoming annual charges, derived from payments alone.
*
* Independent of the email lane on purpose: a recurring CHARGE is visible in the
* ledger whether or not anything was ever emailed about it, which is the only
* way a card annual fee or a loan package fee shows up at all -- banks announce
* those on the statement, not in a renewal notice.
*
* WHAT DOES NOT WORK, measured before this was written: "two charges about 365
* days apart" is useless on its own. Ranked that way the top hit on this ledger
* is a bakery with 202 such pairs, then a donation, Patreon, Coles -- merchants
* used often enough that some pair is inevitably a year apart. Frequency is the
* confound. Even "every gap is annual" still surfaces a Target visited twice
* ($30, then $198) and a Specsavers ($24, then $224). Two visits a year apart
* are not a fee.
*
* So there are two signals, and they are kept apart because they are not equally
* trustworthy -- a guess must never render beside a certainty as if it were one:
*
* confirmed the bank called it a fee AND it has repeated on an annual cadence
* likely the bank called it a fee, seen once -- the kind is certain, the
* cadence is inferred (an annual fee charged once is still annual)
* possible no fee wording; a merchant whose every gap is annual AND whose
* amount is stable. Catches Amazon Prime, Uber One, VicRoads rego,
* none of which carry fee wording. Also where false positives live.
*/
export type Confidence = "confirmed" | "likely" | "possible";
/**
* `expected_not_seen` is not a lesser version of `upcoming` -- it is the more
* interesting row. The estimate has passed and no charge landed, which means one
* of three things, and `account_days_stale` is what tells them apart:
* the account stopped being imported (HSBC: ledger ends 2026-01-07, while the
* mail shows statements to 2026-02-07)
* the account genuinely closed
* the charge landed under a different label (VicRoads paid 2026-03-16 as
* "VICROADS ONLINE PAYM", not "VICROADS")
* Reporting it as upcoming would be a lie; hiding it would lose the signal.
*/
export type ChargeStatus = "upcoming" | "expected_not_seen";
export type AnnualCharge = {
label: string;
status: ChargeStatus;
bank_name: string | null;
account_number: string | null;
confidence: Confidence;
observations: number;
last_charged: string;
last_amount: string;
amount_low: string;
amount_high: string;
median_gap_days: number | null;
next_due_estimate: string;
/** Ledger liveness for the account: a fee that "stopped" may just be an
* account that stopped being imported. Null for merchant-signal rows, which
* are not tied to one account. */
account_last_statement: string | null;
account_days_stale: number | null;
};
/**
* `transaction_type = 'fee'` is the universe, not a description match.
*
* The statement extractor already classifies fees structurally -- 131 rows
* across the ledger -- and every card and loan annual fee on this ledger is
* typed that way. An earlier version of this filtered `transaction_type =
* 'debit'` and returned NONE of them, which is the whole reason this comment
* exists: the fees are not debits.
*
* That universe is broader than "annual", though. It also holds ANZ's monthly
* ACCOUNT SERVICING FEE, Zip's MONTHLY ACCOUNT SERVICE FEE, CommBank's MONTHLY
* FEE, Citibank LATE PAYMENT FEE, HSBC OVERSEAS TRANSACTION FEE and every Wise
* per-transfer charge. Cadence separates them: an annual fee repeats about a
* year apart, the rest repeat monthly, on an event, or never (Wise labels each
* transfer uniquely, so they can never group to two observations at all).
*/
const FEE_TYPE = "fee";
/**
* Wording that establishes the KIND when there is only one observation.
*
* A single charge cannot demonstrate a cadence, but a bank calling something an
* annual fee has already told us what it is -- Westpac's CARD FEE $295 and AMP's
* Professional Pack Annual Fee each appear once and are certainly annual.
* Deliberately narrow: bare "fee" would readmit every per-transaction charge.
*/
const ANNUAL_WORDING =
"(annual|yearly)[[:space:]]*(card[[:space:]]*)?(fee|charge)" +
"|card[[:space:]]+fee" +
"|(package|membership|facility)[[:space:]]+fee";
/**
* Resolve every statement to a canonical account identity, following
* account_links forward. One card can appear under two bank names and two
* numbers after an issuer change (Citi Prestige -> NAB MyCard); without this the
* $700 fee charged 2024-12-13 and 2025-11-13 reads as two dead one-offs.
*
* Recursive because the table permits a chain (A->B->C); `uq_account_links_from`
* forbids a fork, and `account_links_no_self` a self-link, so the walk
* terminates. The depth guard is belt-and-braces against a cycle someone inserts
* by hand (A->B, B->A), which no constraint can catch.
*/
const RESOLVED_ACCOUNTS = `
WITH RECURSIVE chain AS (
SELECT s.id AS statement_id, s.bank_name, s.account_number,
s.bank_name AS cur_bank, s.account_number_key AS cur_key, 0 AS depth
FROM statements s
UNION ALL
SELECT c.statement_id, c.bank_name, c.account_number,
l.to_bank_name, l.to_account_key, c.depth + 1
FROM chain c
JOIN account_links l
ON l.from_bank_name = c.cur_bank AND l.from_account_key = c.cur_key
WHERE c.depth < 10
),
resolved AS (
SELECT DISTINCT ON (statement_id)
statement_id, bank_name, account_number, cur_bank AS acct_bank, cur_key AS acct_key
FROM chain ORDER BY statement_id, depth DESC
)`;
export async function getUpcomingAnnualCharges(ownerId: number): Promise<AnnualCharge[]> {
// Signal 1 -- the bank's own fee wording, grouped by resolved account identity
// and by the fee's wording so two different fees on one card stay apart
// (ANZ charges "ANNUAL FEE" $120 and "ADDITIONAL CARD FEE" $10 on the same
// statement, and they are not the same obligation).
const feeRows = await queryRaw<AnnualCharge>(
`${RESOLVED_ACCOUNTS},
fees AS (
SELECT r.acct_bank, r.acct_key, r.bank_name, r.account_number,
upper(trim(regexp_replace(t.description, '[[:space:]]+', ' ', 'g'))) AS label,
t.transaction_date, t.amount
FROM transactions t
JOIN resolved r ON r.statement_id = t.statement_id
JOIN statements s ON s.id = t.statement_id
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
AND t.amount > 0
AND COALESCE(t.owner_id, s.owner_id) = $1
-- Structural first, wording only as a widening: a bank that types a
-- fee as a plain debit still gets picked up if it names it annual.
AND (t.transaction_type = $3 OR t.description ~* $2)
),
gapped AS (
SELECT f.*,
transaction_date - lag(transaction_date)
OVER (PARTITION BY acct_key, label ORDER BY transaction_date) AS gap
FROM fees f
),
agg AS (
SELECT acct_key, acct_bank, label,
count(*)::int AS observations,
max(transaction_date) AS last_charged,
min(amount) AS amount_low, max(amount) AS amount_high,
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
FILTER (WHERE gap IS NOT NULL) AS median_gap,
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
(array_agg(bank_name ORDER BY transaction_date DESC))[1] AS bank_name,
(array_agg(account_number ORDER BY transaction_date DESC))[1] AS account_number,
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
FROM gapped GROUP BY acct_key, acct_bank, label
),
liveness AS (
SELECT r.acct_key, max(s.billing_end_date) AS last_statement
FROM statements s JOIN resolved r ON r.statement_id = s.id
GROUP BY r.acct_key
)
SELECT a.label,
-- The account's CURRENT identity, not the bank that happened to
-- print the last fee. That $700 fee last appeared under "Citibank"
-- in Nov 2025, but the account is NAB/MyCard now and the next one
-- will come from there; naming the dead brand would send you to the
-- wrong bank.
a.acct_bank AS bank_name, a.account_number,
CASE WHEN (a.last_charged + COALESCE(round(a.median_gap)::int, 365)) >= CURRENT_DATE
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
-- One observation cannot prove a cadence, but the bank naming it an
-- annual fee already tells us the kind. Repeated on an annual gap it
-- is confirmed; seen once it is likely and says so.
CASE WHEN a.gaps > 0 AND a.annual_gaps = a.gaps THEN 'confirmed'
ELSE 'likely' END AS confidence,
a.observations,
a.last_charged::text AS last_charged,
a.last_amount::text AS last_amount,
a.amount_low::text AS amount_low,
a.amount_high::text AS amount_high,
round(a.median_gap)::int AS median_gap_days,
(a.last_charged + COALESCE(round(a.median_gap)::int, 365))::text AS next_due_estimate,
l.last_statement::text AS account_last_statement,
(CURRENT_DATE - l.last_statement)::int AS account_days_stale
FROM agg a LEFT JOIN liveness l ON l.acct_key = a.acct_key
-- Two ways to earn a place, and nothing else gets one. Filtered here, not
-- in the UI, so the API can never be read wrongly by a future caller.
WHERE
-- Repeated on an annual cadence. Excludes ANZ's monthly ACCOUNT
-- SERVICING FEE and Zip's MONTHLY ACCOUNT SERVICE FEE, whose gaps are
-- ~30 days, and Citibank's LATE PAYMENT FEE, which is event-driven.
(a.gaps > 0 AND a.annual_gaps = a.gaps AND a.median_gap >= 300)
-- Or seen once, but named as an annual/card/package fee. Westpac's CARD
-- FEE and AMP's Professional Pack Annual Fee are each a single charge and
-- unmistakably annual. A Wise per-transfer fee is also a single charge
-- and says nothing of the sort, which is what keeps it out.
OR (a.gaps = 0 AND a.label ~* $2)`,
[ownerId, ANNUAL_WORDING, FEE_TYPE]
);
// Signal 2 -- merchant-charged annual items with no fee wording. Requires
// BOTH that every gap is annual AND that the amount is stable: VicRoads rego
// rises about 3% a year ($825.23 -> $852.88 -> $875.48), so equality is too
// strict, while no amount test at all readmits the twice-visited restaurant.
// Keyed on merchant, never on card -- that rego was paid on three different
// banks, and a card-keyed view would see three unrelated one-offs.
const merchantRows = await queryRaw<AnnualCharge>(
`WITH t AS (
SELECT upper(trim(COALESCE(t.merchant_normalized, t.merchant_name, t.description))) AS label,
t.transaction_date, t.amount
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
AND t.voided_at IS NULL
AND t.transaction_type = 'debit' AND t.amount > 0
AND COALESCE(t.owner_id, s.owner_id) = $1
AND t.transaction_type = 'debit' AND t.description !~* $2
),
gapped AS (
SELECT t.*, transaction_date - lag(transaction_date)
OVER (PARTITION BY label ORDER BY transaction_date) AS gap
FROM t
),
agg AS (
SELECT label, count(*)::int AS observations,
max(transaction_date) AS last_charged,
min(amount) AS amount_low, max(amount) AS amount_high,
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
FILTER (WHERE gap IS NOT NULL) AS median_gap,
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
FROM gapped GROUP BY label
)
SELECT label, NULL::text AS bank_name, NULL::text AS account_number,
CASE WHEN (last_charged + round(median_gap)::int) >= CURRENT_DATE
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
'possible' AS confidence, observations,
last_charged::text AS last_charged,
last_amount::text AS last_amount,
amount_low::text AS amount_low,
amount_high::text AS amount_high,
round(median_gap)::int AS median_gap_days,
(last_charged + round(median_gap)::int)::text AS next_due_estimate,
NULL::text AS account_last_statement,
NULL::int AS account_days_stale
FROM agg
WHERE gaps >= 1 AND annual_gaps = gaps
-- Amount stability is what separates a subscription from a restaurant.
-- 20% absorbs CPI drift and an FX-priced renewal without admitting a
-- $30-then-$198 pair.
AND amount_high <= amount_low * 1.20
-- Two observations is the floor for a cadence claim, and this lane has no
-- wording to fall back on the way the fee lane does.
AND observations >= 2`,
[ownerId, ANNUAL_WORDING]
);
// Upcoming first and soonest-first within that, so the thing you can still
// act on is at the top; the expected-but-missing rows follow as a review list.
const rank = (c: AnnualCharge) => (c.status === "upcoming" ? 0 : 1);
return [...feeRows, ...merchantRows].sort(
(a, b) => rank(a) - rank(b) || a.next_due_estimate.localeCompare(b.next_due_estimate)
);
}
+235
View File
@@ -0,0 +1,235 @@
import { queryRaw, type Queryable } from "@/lib/db";
import { merchantMatches } from "@/lib/intake-rules";
/**
* Unambiguous statement matches for statementless ("manual") transactions.
*
* WHY THIS EXISTS
* ---------------
* The reconcile queue is worked by hand and is already ~177 deep. The wallet
* lane writes a transaction for every tap (DECISIONS.md ING-11, 2026-08-22),
* which at the observed rate adds ~63 rows a month, ~767 a year. Without an
* automatic pass for the easy cases that queue stops being workable, and the
* tempting alternative -- filtering the new rows back out of it -- is
* explicitly ruled out by the `awaitsStatementLine()` removal note in
* queries.ts: "an imported row that has no statement belongs in the queue;
* volume is solved upstream, never downstream by hiding what is there."
*
* WHAT COUNTS AS UNAMBIGUOUS
* --------------------------
* Deliberately much stricter than the queue's own candidate ranking, which
* offers the top 5 within +/-1% and +/-3 days for a human to choose between.
* A machine gets one shot and no judgement, so:
*
* - EXACT amount equality, not the queue's +/-1% band. A near-miss is exactly
* the case a person should look at.
* - Same owner, and the statement row must not already be reconciled against.
* - UNIQUE BOTH WAYS: the manual row must have exactly one candidate, AND
* that candidate must be claimed by exactly one manual row. This is the
* order_transaction_bridge rule, and it is what stops two identical $13.50
* bakery taps in one window from being paired arbitrarily.
*
* Foreign rows match on the NATIVE pair (foreign_currency_amount +
* foreign_currency_code), never on `amount`: `amount` is the AUD the bank
* charged after applying its own rate, so a wallet row cannot predict it, while
* the native figure is exact on both sides. Same reasoning as
* order_transaction_bridge's FX path.
*
* This function only PROPOSES. Applying a pair goes through the same merge the
* manual route uses, so overrides/tags/splits/metadata move identically.
*/
export type AutoMatch = {
manual_id: number;
statement_tx_id: number;
amount: string;
manual_date: string;
statement_date: string;
description: string;
basis: "aud" | "native";
};
/** Days either side of the manual row's date to consider. Authorisation and
* settlement are rarely the same day; 3 matches the queue's own window and
* the wallet reconciliation check's default. */
export const AUTO_MATCH_WINDOW_DAYS = 3;
export async function findAutoMatches(
ownerId: number,
windowDays: number = AUTO_MATCH_WINDOW_DAYS
): Promise<AutoMatch[]> {
return queryRaw<AutoMatch>(
`WITH pairs AS (
SELECT m.id AS manual_id,
t.id AS statement_tx_id,
t.amount::text AS amount,
m.transaction_date::text AS manual_date,
t.transaction_date::text AS statement_date,
t.description AS description,
CASE WHEN m.foreign_currency_code IS NOT NULL
AND m.foreign_currency_amount IS NOT NULL
THEN 'native' ELSE 'aud' END AS basis
FROM transactions m
JOIN transactions t
ON t.statement_id IS NOT NULL
AND t.transaction_type = 'debit'
AND t.transaction_date BETWEEN m.transaction_date - $2::int
AND m.transaction_date + $2::int
AND (
-- Foreign: compare what both sides actually recorded natively.
(m.foreign_currency_code IS NOT NULL
AND m.foreign_currency_amount IS NOT NULL
AND t.foreign_currency_code = m.foreign_currency_code
AND t.foreign_currency_amount = m.foreign_currency_amount)
OR -- AUD side: exact equality, never a tolerance band. The
-- statement row MAY be foreign here, and that is not a mistake:
-- all 48 reconciliations a human has made on this ledger are
-- exactly this shape -- a manual row carrying no currency,
-- matched on exact AUD against a statement row that does carry
-- one (EUR, CHF). Overseas receipts and orders record what was
-- charged in AUD; only the wallet lane records natively, which
-- is why the branch above exists. Requiring both sides to be
-- non-foreign was tried and rejected: it matched 0 of those 48.
-- The coincidence risk (a domestic tap whose amount happens to
-- equal a foreign charge's AUD figure) is already covered by
-- unique-both-ways, which refuses as soon as a second candidate
-- exists.
(m.foreign_currency_code IS NULL
AND t.amount = m.amount)
)
JOIN statements s ON s.id = t.statement_id
WHERE m.statement_id IS NULL
AND m.owner_id = $1
AND m.reconciled_with_id IS NULL
AND m.superseded_by_id IS NULL
AND m.voided_at IS NULL
AND (m.payment_method IS NULL OR m.payment_method NOT IN ('cash', 'credits'))
AND t.superseded_by_id IS NULL
AND COALESCE(t.owner_id, s.owner_id) = $1
-- The statement row must be unclaimed.
AND NOT EXISTS (SELECT 1 FROM transactions mt WHERE mt.reconciled_with_id = t.id)
)
SELECT manual_id, statement_tx_id, amount, manual_date, statement_date, description, basis
FROM pairs p
WHERE (SELECT count(*) FROM pairs q WHERE q.manual_id = p.manual_id) = 1
AND (SELECT count(*) FROM pairs q WHERE q.statement_tx_id = p.statement_tx_id) = 1
ORDER BY manual_id`,
[ownerId, windowDays]
);
}
/**
* An existing statementless transaction that a newly-arriving receipt or order
* describes -- i.e. the row the wallet lane already created for this purchase.
*
* WHY: signals for one purchase arrive in sequence, not in competition. The
* wallet notification lands within seconds of the tap; the receipt is shared
* hours or days later. Measured on txn 6858: the wallet event arrived 35h25m
* before anything created a transaction, and pantry then created one from
* scratch -- so the same Woolworths shop would have had two rows. The receipt
* must adopt the row that already exists and enrich it.
*
* Deliberately narrow, for the same reason findAutoMatches() is: exact amount,
* a few days either side, unclaimed, and UNIQUE. Two candidates means two
* plausible purchases, and inventing a link between the wrong pair moves one
* shop's line items onto another shop's row. Returning null is always safe --
* the caller falls back to creating its own row, which is today's behaviour.
*/
/**
* The row a newly-arriving WALLET notification describes -- the mirror of
* findAdoptableTransaction, for the case where the signals arrived in the other
* order.
*
* WHY: the sequencing this estate is built on -- wallet first, receipt second,
* statement last -- is a tendency, not a guarantee. The wallet lane can be
* parked (its first 21 rows were backfilled days late), the interpreter runs
* every 15 minutes, the phone can be offline. Whenever a receipt reaches the
* ledger before the notification for the same tap, the wallet writer used to
* insert unconditionally: measured on the 2026-08-18 Woolworths shop, txn 6858
* (pantry, 23 items) and txn 6927 (wallet) were the same $103.30 purchase, and
* the second sat in the reconcile queue waiting for a statement line that the
* first had already claimed.
*
* Deliberately NOT the same predicate as findAdoptableTransaction, in the one
* way that matters: that one refuses a row which already carries line items,
* because for a receipt such a row is a DIFFERENT purchase that happens to
* share an amount. Here it is the opposite -- an itemised row is the receipt we
* are trying to fold into, and refusing it is the bug.
*
* Losing that guard means the merchant has to supply it. A wallet row adopts
* only when the merchant agrees (the same whole-word containment the intake
* matcher uses: "Woolworths" vs "WOOLWORTHS 3345"), because amount-and-date
* alone would let a notification silently disappear into an unrelated purchase
* of the same value -- and a wallet row that is never written is a tap with no
* trace, which is worse than a visible duplicate. When in doubt, return null:
* the caller then writes its own row, which is the pre-existing behaviour.
*/
export async function findWalletAdoptableTransaction(
ownerId: number,
amount: number,
date: string,
merchant: string | null,
windowDays: number = AUTO_MATCH_WINDOW_DAYS,
// wallet-ingestion calls this while holding its advisory lock, and must pass
// that transaction's connection: borrowing a second pooled one per request
// is how a burst of concurrent posts exhausts the pool and deadlocks.
db: Pick<Queryable, "queryRaw"> = { queryRaw }
): Promise<{ id: number; metadata_id: number | null } | null> {
// No usable merchant, no adoption. "Card purchase" is wallet-ingestion's own
// fallback for a notification that named no merchant at all.
if (!merchant || merchant === "Card purchase") return null;
const rows = await db.queryRaw<{ id: number; metadata_id: number | null; merchant: string | null }>(
`SELECT t.id, em.id AS metadata_id,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant
FROM transactions t
LEFT JOIN expense_metadata em ON em.transaction_id = t.id
WHERE t.statement_id IS NULL
AND t.owner_id = $1
AND t.amount = $2::numeric
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
AND t.reconciled_with_id IS NULL
AND t.superseded_by_id IS NULL
-- A voided row is a charge that never posted; a real charge for the same
-- amount (a rebooking after a released hold) must not fold into it.
AND t.voided_at IS NULL
-- A wallet row is the OTHER HALF of this tap, not a purchase to fold
-- into; findPairedRow() merges those, with different rules.
AND COALESCE(t.source, '') <> 'wallet'
LIMIT 5`,
[ownerId, amount, date, windowDays]
);
const hits = rows.filter((r) => merchantMatches(r.merchant, merchant));
return hits.length === 1 ? { id: hits[0].id, metadata_id: hits[0].metadata_id } : null;
}
export async function findAdoptableTransaction(
ownerId: number,
amount: number,
date: string,
windowDays: number = AUTO_MATCH_WINDOW_DAYS
): Promise<number | null> {
const rows = await queryRaw<{ id: number }>(
`SELECT t.id
FROM transactions t
WHERE t.statement_id IS NULL
AND t.owner_id = $1
AND t.amount = $2::numeric
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
AND t.reconciled_with_id IS NULL
AND t.superseded_by_id IS NULL
-- A voided row is a charge that never posted; a real charge for the same
-- amount (a rebooking after a released hold) must not fold into it.
AND t.voided_at IS NULL
-- Never adopt a row that another source already fully described; that
-- is a different purchase that happens to share an amount.
AND NOT EXISTS (
SELECT 1 FROM expense_metadata em
WHERE (em.transaction_id = t.id OR em.matched_transaction_id = t.id)
AND jsonb_typeof(em.line_items) = 'array'
AND jsonb_array_length(em.line_items) > 0
)
LIMIT 2`,
[ownerId, amount, date, windowDays]
);
return rows.length === 1 ? rows[0].id : null;
}
+35
View File
@@ -20,3 +20,38 @@ export async function queryRow<T>(sql: string, params: unknown[] = []): Promise<
const rows = await prisma.$queryRawUnsafe<T[]>(sql, ...params); const rows = await prisma.$queryRawUnsafe<T[]>(sql, ...params);
return rows.length > 0 ? rows[0] : null; return rows.length > 0 ? rows[0] : null;
} }
/** The same two calls, scoped to one connection inside one transaction. */
export type Queryable = {
queryRaw<T>(sql: string, params?: unknown[]): Promise<T[]>;
queryRow<T>(sql: string, params?: unknown[]): Promise<T | null>;
};
/**
* Run several statements as one transaction, on one connection.
*
* The module-level `queryRaw`/`queryRow` are `$queryRawUnsafe` on a POOLED
* client, so each call is its own implicit transaction on whichever connection
* the pool hands out. That makes a check-then-write pair non-atomic, and it
* makes `pg_advisory_xact_lock` useless outside this helper — the lock is
* released the instant the statement that took it commits, which is before the
* next statement runs. Anything that needs a lock to still be held on its next
* query has to be in here.
*/
export async function withTransaction<T>(
fn: (q: Queryable) => Promise<T>,
// Prisma's defaults (maxWait 2s, timeout 5s) count time spent WAITING on an
// advisory lock, so a caller that serialises a burst on one needs more.
opts?: { maxWait?: number; timeout?: number }
): Promise<T> {
return prisma.$transaction(async (tx) => {
const q: Queryable = {
queryRaw: <R>(sql: string, params: unknown[] = []) => tx.$queryRawUnsafe<R[]>(sql, ...params),
queryRow: async <R>(sql: string, params: unknown[] = []) => {
const rows = await tx.$queryRawUnsafe<R[]>(sql, ...params);
return rows.length > 0 ? rows[0] : null;
},
};
return fn(q);
}, opts);
}
+55
View File
@@ -177,6 +177,40 @@ export function useUpdateTransaction() {
}); });
} }
export function useMergeTransaction() {
const qc = useQueryClient();
return useMutation({
mutationFn: async ({
id,
other_id,
mode,
}: {
id: number;
other_id: number;
mode: "reconcile" | "supersede";
}) => {
const res = await fetch(`/api/transactions/${id}/merge`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ other_id, mode }),
});
if (!res.ok) {
const body = await res.json().catch(() => ({}));
throw new Error(body.error || "Failed to merge transaction");
}
return res.json();
},
onSuccess: () => {
qc.invalidateQueries({ queryKey: ["transactions"] });
qc.invalidateQueries({ queryKey: ["transaction"] });
qc.invalidateQueries({ queryKey: ["splits"] });
qc.invalidateQueries({ queryKey: ["analytics"] });
qc.invalidateQueries({ queryKey: ["participant-balances"] });
qc.invalidateQueries({ queryKey: ["shared-transactions"] });
},
});
}
export function useBulkAction() { export function useBulkAction() {
const qc = useQueryClient(); const qc = useQueryClient();
return useMutation({ return useMutation({
@@ -1128,6 +1162,27 @@ export function useReconcile() {
}); });
} }
/** Mark a statementless row as a charge that never posted (migration 0040). */
export function useVoidTransaction() {
const qc = useQueryClient();
return useMutation({
mutationFn: async ({ id, reason }: { id: number; reason: string }) => {
const res = await fetch(`/api/transactions/${id}/void`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ reason }),
});
if (!res.ok) throw new Error((await res.json()).error || "Void failed");
return res.json();
},
onSuccess: () => {
qc.invalidateQueries({ queryKey: ["transactions"] });
qc.invalidateQueries({ queryKey: ["reconcile-pending"] });
qc.invalidateQueries({ queryKey: ["analytics"] });
},
});
}
export function useMerchantTransactions(merchant: string | null) { export function useMerchantTransactions(merchant: string | null) {
return useQuery<{ transactions: MerchantTxnRow[] }>({ return useQuery<{ transactions: MerchantTxnRow[] }>({
queryKey: ["analytics", "merchant-txns", merchant], queryKey: ["analytics", "merchant-txns", merchant],
+3 -2
View File
@@ -58,6 +58,7 @@ const ctxFor = (batch: BatchRow): RouteContext | null =>
platform: batch.header.platform, platform: batch.header.platform,
header: batch.header, header: batch.header,
items: batch.items ?? [], items: batch.items ?? [],
currency: batch.header.currency,
} }
: null; : null;
@@ -79,13 +80,13 @@ export async function handleIntakeAction(
if (actionId === "intake_use_candidate") { if (actionId === "intake_use_candidate") {
if (!Number.isInteger(value.transaction_id)) return note("That button carried no transaction id."); if (!Number.isInteger(value.transaction_id)) return note("That button carried no transaction id.");
const txnId = value.transaction_id as number; const txnId = value.transaction_id as number;
await replaceLineItems(txnId, ctx.items, { source: "intake", platform: ctx.platform, batchId: batch.id }); await replaceLineItems(txnId, ctx.items, { source: "intake", platform: ctx.platform, batchId: batch.id, currency: ctx.currency });
await queryRaw( await queryRaw(
`UPDATE intake_batches SET status = 'enriched', kind = COALESCE(kind, 'food'), transaction_id = $2, resolved_at = now() WHERE id = $1::uuid`, `UPDATE intake_batches SET status = 'enriched', kind = COALESCE(kind, 'food'), transaction_id = $2, resolved_at = now() WHERE id = $1::uuid`,
[batch.id, txnId] [batch.id, txnId]
); );
return { return {
...enrichedCard({ batchId: batch.id, merchant: ctx.header.merchant, total: ctx.header.total, orderDate: ctx.header.order_date, platform: ctx.platform, itemCount: ctx.items.length, transactionId: txnId }), ...enrichedCard({ batchId: batch.id, merchant: ctx.header.merchant, total: ctx.header.total, orderDate: ctx.header.order_date, platform: ctx.platform, itemCount: ctx.items.length, currency: ctx.currency, transactionId: txnId }),
replace: true, replace: true,
}; };
} }
+90 -20
View File
@@ -12,7 +12,7 @@ import { platformLabel } from "@/lib/intake-rules";
export type Block = Record<string, unknown>; export type Block = Record<string, unknown>;
export type Candidate = { transaction_id: number; description: string | null; transaction_date: string; amount: string | number }; export type Candidate = { transaction_id: number; description: string | null; transaction_date: string; amount: string | number; currency?: string | null };
export function financePublicUrl(): string { export function financePublicUrl(): string {
return (process.env.FINANCE_PUBLIC_URL || "https://finance.bosecamp.com").replace(/\/$/, ""); return (process.env.FINANCE_PUBLIC_URL || "https://finance.bosecamp.com").replace(/\/$/, "");
@@ -23,7 +23,29 @@ export function transactionLink(merchant: string | null): string {
return `${financePublicUrl()}/transactions?q=${encodeURIComponent(merchant ?? "")}`; return `${financePublicUrl()}/transactions?q=${encodeURIComponent(merchant ?? "")}`;
} }
const money = (n: number | string) => `$${Number(n).toFixed(2)}`; /**
* Slack renders `&`, `<` and `>` as markup in mrkdwn (and in the `text`
* fallback). Merchant names, transaction descriptions and model output are
* none of ours — `Ben & Jerry's` must not arrive as `Ben &amp; Jerry's`, and a
* description containing `<http://…|click>` must not become a link the card
* never meant to show. Applied to interpolated values only; our own literal
* `*bold*` and `:emoji:` stay markup.
*/
export function esc(value: string | null | undefined): string {
return (value ?? "").replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
}
/**
* `$` is only right for AUD. An intake batch carries the currency the receipt
* printed, and the order nudge already renders `USD 15.00` rather than
* pretending a foreign order was dollars.
*/
const money = (n: number | string, currency?: string | null) => {
const amount = Number(n).toFixed(2);
// The currency comes off the model (`parseExtraction` accepts any string), so
// an unescaped `<@U123>` here would be a mention on every card.
return !currency || currency === "AUD" ? `$${amount}` : `${esc(currency)} ${amount}`;
};
const section = (text: string): Block => ({ type: "section", text: { type: "mrkdwn", text } }); const section = (text: string): Block => ({ type: "section", text: { type: "mrkdwn", text } });
const button = (action_id: string, text: string, value: unknown, style?: "primary" | "danger"): Block => ({ const button = (action_id: string, text: string, value: unknown, style?: "primary" | "danger"): Block => ({
type: "button", type: "button",
@@ -40,10 +62,12 @@ export type CardInput = {
orderDate: string | null; orderDate: string | null;
platform: string | null; platform: string | null;
itemCount: number; itemCount: number;
currency?: string | null;
}; };
export function enrichedCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } { export function enrichedCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
const what = [c.total !== null ? money(c.total) : null, platformLabel(c.platform) || c.merchant, c.platform && c.merchant ? `(${c.merchant})` : null] const merchant = esc(c.merchant);
const what = [c.total !== null ? money(c.total, c.currency) : null, esc(platformLabel(c.platform)) || merchant, c.platform && merchant ? `(${merchant})` : null]
.filter(Boolean) .filter(Boolean)
.join(" "); .join(" ");
const text = `Added ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} to your ${what} order ✓`; const text = `Added ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} to your ${what} order ✓`;
@@ -60,7 +84,7 @@ export function enrichedCard(c: CardInput & { transactionId: number }): { blocks
} }
export function createdCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } { export function createdCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
const text = `Created a ${c.total !== null ? money(c.total) + " " : ""}${c.merchant ?? "new"} transaction with ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} ✓`; const text = `Created a ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant) || "new"} transaction with ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} ✓`;
return { return {
text, text,
blocks: [ blocks: [
@@ -74,42 +98,83 @@ export function createdCard(c: CardInput & { transactionId: number }): { blocks:
} }
export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks: Block[]; text: string } { export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks: Block[]; text: string } {
const label = platformLabel(c.platform) || ""; const label = esc(platformLabel(c.platform));
const head = `No matching ${label ? label + " " : ""}order for *${c.merchant ?? "this merchant"}* ${c.total !== null ? money(c.total) : ""} on ${c.orderDate ?? "?"}.`; const head = `No matching ${label ? label + " " : ""}order for *${esc(c.merchant) || "this merchant"}* ${c.total !== null ? money(c.total, c.currency) : ""} on ${esc(c.orderDate) || "?"}.`;
const text = `${head.replace(/\*/g, "")} ${candidates.length ? "Pick one below or create it." : "Create it?"}`; const text = `${head.replace(/\*/g, "")} ${candidates.length ? "Pick one below or create it." : "Create it?"}`;
const blocks: Block[] = [section(`:mag: ${head}${candidates.length ? "\nIs it one of these?" : ""}`)]; const blocks: Block[] = [section(`:mag: ${head}${candidates.length ? "\nIs it one of these?" : ""}`)];
for (const cand of candidates) { for (const cand of candidates) {
blocks.push({ blocks.push({
type: "section", type: "section",
text: { type: "mrkdwn", text: `${cand.transaction_date} · ${cand.description ?? "—"} · ${money(cand.amount)}` }, text: { type: "mrkdwn", text: `${esc(cand.transaction_date)} · ${esc(cand.description) || "—"} · ${money(cand.amount, cand.currency)}` },
accessory: button("intake_use_candidate", "Use this", { batch_id: c.batchId, transaction_id: cand.transaction_id }), accessory: button("intake_use_candidate", "Use this", { batch_id: c.batchId, transaction_id: cand.transaction_id }),
}); });
} }
blocks.push({ blocks.push({
type: "actions", type: "actions",
block_id: "intake_create", block_id: "intake_create",
elements: [button("intake_create", `Create new ${c.total !== null ? money(c.total) + " " : ""}${c.merchant ?? ""} transaction`.replace(/\s+/g, " ").trim(), { batch_id: c.batchId }, "primary")], // plain_text disables *formatting*, not entity decoding: `&`, `<` and `>`
// are control characters everywhere in a Slack payload, so the label is
// escaped like any other value we did not write.
elements: [button("intake_create", `Create new ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant)} transaction`.replace(/\s+/g, " ").trim(), { batch_id: c.batchId }, "primary")],
}); });
return { blocks, text }; return { blocks, text };
} }
export function forwardedCard(c: CardInput & { captureId: number; enriching: boolean }): { blocks: Block[]; text: string } { export function forwardedCard(
const text = `${c.merchant ?? "Grocery receipt"} ${c.total !== null ? money(c.total) : ""} → pantry board (${c.itemCount} item${c.itemCount === 1 ? "" : "s"}) — finance ${c.enriching ? "updates" : "books it"} on confirm`.replace(/\s+/g, " "); c: CardInput & {
captureId: number;
enriching: boolean;
/** Pantry already holds this exact receipt (same barcode tuple). */
alreadyCaptured?: { captureId: number; confirmed: boolean } | null;
/** Finance found more than one row this could be, so it picked none. */
ambiguous?: boolean;
}
): { blocks: Block[]; text: string } {
const what = `${esc(c.merchant) || "Grocery receipt"} ${c.total !== null ? money(c.total, c.currency) : ""}`.trim();
const items = `${c.itemCount} item${c.itemCount === 1 ? "" : "s"}`;
// The warning comes FIRST and the emoji changes, because this card is read at
// a glance on a phone: a duplicate warning appended to a cheerful "→ pantry
// board" line is a duplicate warning nobody reads. Confirming both copies
// counts the same groceries twice — pantry's addItem accumulates — so the
// card has to say which capture to dismiss, not merely that there are two.
if (c.alreadyCaptured) {
const state = c.alreadyCaptured.confirmed ? "already confirmed" : "still waiting to be confirmed";
const text = `Already on the pantry board — ${what} is capture ${c.alreadyCaptured.captureId} (${state}). This upload is capture ${c.captureId}; dismiss it unless it replaces the first, because confirming both counts the same groceries twice.`.replace(/\s+/g, " ");
return { text, blocks: [section(`:repeat: ${text}`)] };
}
const tail = c.ambiguous
? "— finance found more than one row it could be, so it picked none; reconcile it there"
: `— finance ${c.enriching ? "updates" : "books it"} on confirm`;
const text = `${what} → pantry board (${items}) ${tail}`.replace(/\s+/g, " ");
return { text, blocks: [section(`:shopping_trolley: ${text}`)] }; return { text, blocks: [section(`:shopping_trolley: ${text}`)] };
} }
export function needsHeaderCard(): { blocks: Block[]; text: string } { /**
const text = "I can see items but no order total or date — share the top of the screen too so I can match it."; * Names what is missing, because the advice differs. A delivery-app screenshot
* that lost its header needs the top of the screen; a Woolworths eReceipt
* SCREENSHOT has no date anywhere on it (seen 2026-08-22 — store, total, items,
* payment, and not one date), while the app's own Share button sends the raw
* receipt WITH the date. So for a dateless in-store receipt the right advice is
* the Share button, or pantry's own upload — not "share more of the screen".
*/
export function needsHeaderCard(missing: string[] = ["total", "date"], source?: string | null): { blocks: Block[]; text: string } {
const what = missing.length ? missing.join(" or ") : "total or date";
const inStore = source === "coles" || source === "woolworths";
const hint =
missing.length === 1 && missing[0] === "date" && inStore
? "an app screenshot carries no date — use the app's Share button instead, or upload the picture to pantry directly"
: "share the part of the screen that shows it";
const text = `I can see items but no ${esc(what)} — ${hint}, so I can match it.`;
return { text, blocks: [section(`:warning: ${text}`)] }; return { text, blocks: [section(`:warning: ${text}`)] };
} }
export function heldCard(reason: string): { blocks: Block[]; text: string } { export function heldCard(reason: string, headline = "Items don't add up to the total — is a shot missing?"): { blocks: Block[]; text: string } {
const text = `Items don't add up to the total — is a shot missing? (${reason})`; const text = `${headline} (${esc(reason)})`;
return { text, blocks: [section(`:warning: ${text}`)] }; return { text, blocks: [section(`:warning: ${text}`)] };
} }
export function askCard(c: CardInput): { blocks: Block[]; text: string } { export function askCard(c: CardInput): { blocks: Block[]; text: string } {
const text = `${c.merchant ?? "This upload"} ${c.total !== null ? money(c.total) : ""} — groceries or a meal?`.replace(/\s+/g, " "); const text = `${esc(c.merchant) || "This upload"} ${c.total !== null ? money(c.total, c.currency) : ""} — groceries or a meal?`.replace(/\s+/g, " ");
return { return {
text, text,
blocks: [ blocks: [
@@ -126,18 +191,23 @@ export function askCard(c: CardInput): { blocks: Block[]; text: string } {
}; };
} }
export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null }): { blocks: Block[]; text: string } { export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null; currency?: string | null }): { blocks: Block[]; text: string } {
const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${c.batchId}` : "an earlier upload"; const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${esc(c.batchId)}` : "an earlier upload";
const text = `Already handled — ${c.merchant ?? "this order"} ${c.total !== null ? money(c.total) : ""} is ${where}. Nothing was changed.`.replace(/\s+/g, " "); // The header (merchant, date, total) IS the identity of a purchase here, so a
// genuine SECOND identical purchase — two of the same coffee, one day, one shop
// — reads as a re-share and is refused. That is the right default (a re-share is
// far commoner than a repeat) but it is a dead end without this sentence: the
// card must say what to do instead, or the ledger quietly loses the second one.
const text = `Already handled — ${esc(c.merchant) || "this order"} ${c.total !== null ? money(c.total, c.currency) : ""} is ${where}. Nothing was changed — if this really is a second, separate purchase, add it in the app.`.replace(/\s+/g, " ");
return { text, blocks: [section(`:repeat: ${text}`)] }; return { text, blocks: [section(`:repeat: ${text}`)] };
} }
export function unsupportedCard(detail?: string): { blocks: Block[]; text: string } { export function unsupportedCard(detail?: string): { blocks: Block[]; text: string } {
const text = `I can read receipts, statements and order screens — not this.${detail ? ` (${detail})` : ""}`; const text = `I can read receipts, statements and order screens — not this.${detail ? ` (${esc(detail)})` : ""}`;
return { text, blocks: [section(`:no_entry_sign: ${text}`)] }; return { text, blocks: [section(`:no_entry_sign: ${text}`)] };
} }
export function errorCard(detail: string): { blocks: Block[]; text: string } { export function errorCard(detail: string): { blocks: Block[]; text: string } {
const text = `Couldn't process that upload: ${detail}`; const text = `Couldn't process that upload: ${esc(detail)}`;
return { text, blocks: [section(`:x: ${text}`)] }; return { text, blocks: [section(`:x: ${text}`)] };
} }
+31 -5
View File
@@ -1,5 +1,5 @@
import { extractText, getDocumentProxy } from "unpdf"; import { extractText, getDocumentProxy } from "unpdf";
import type { ExtractedOrder, ExtractedItem } from "@/lib/intake-rules"; import { PROMPT_PLACEHOLDER_MERCHANTS, type ExtractedOrder, type ExtractedItem } from "@/lib/intake-rules";
/** /**
* Extraction for the intake router: one joint model call per batch. * Extraction for the intake router: one joint model call per batch.
@@ -31,13 +31,22 @@ export async function extractPdfText(bytes: Buffer): Promise<string> {
return (Array.isArray(text) ? text.join("\n") : text).trim(); return (Array.isArray(text) ? text.join("\n") : text).trim();
} }
/**
* The merchant examples are FORMAT PLACEHOLDERS, never real shops
* (`PROMPT_PLACEHOLDER_MERCHANTS`). Given an input with nothing in it, a model
* still answers, and it answers out of its prompt: on 2026-08-22 the word
* `Image` produced a complete Burgertory receipt because the prompt named
* Burgertory. The "not a receipt" rule below and the placeholder names are the
* two halves of closing that.
*/
export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text. export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text.
Return ONE JSON object with exactly these keys: Return ONE JSON object with exactly these keys:
{ "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] } { "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] }
Rules: Rules:
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (e.g. "Coles (Wyndham Vale)", "Burgertory (Werribee)"). Never the delivery platform. null if not shown. - If what you are given is NOT a receipt or an order — it is blank, a filename, a single word such as "Image", an app screen with no purchase on it, or anything else you cannot read a purchase off — return { "merchant": null, "order_date": null, "total": null, "subtotal": null, "currency": null, "items": [], "adjustments": [] }. Do not construct a plausible order. Nothing in these instructions is data: never return an example from this prompt as if it were printed on the input.
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (format: "${PROMPT_PLACEHOLDER_MERCHANTS[0]}"). Never the delivery platform. null if not shown.
- order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown. - order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown.
- total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown. - total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown.
- subtotal: the printed item subtotal before fees and discounts, as a number, or null. - subtotal: the printed item subtotal before fees and discounts, as a number, or null.
@@ -185,9 +194,26 @@ export function liteLLMExtractor(opts: {
}); });
if (!response.ok) throw new ExtractionError(`LiteLLM returned ${response.status}`); if (!response.ok) throw new ExtractionError(`LiteLLM returned ${response.status}`);
const raw = (await response.json()) as { choices?: { message?: { content?: string }; finish_reason?: string }[] }; const raw = (await response.json()) as { choices?: { message?: { content?: string }; finish_reason?: string }[] };
// A truncated reply can still parse once the JSON is closed, which quietly // The rule is: a finish_reason that is PRESENT must be `stop`. An absent
// drops the tail of a long receipt. Losing lines silently is worse than failing. // one is accepted, deliberately — providers differ on whether they send the
if (raw.choices?.[0]?.finish_reason === "length") throw new ExtractionError("The receipt was too long to read in one pass."); // field, and failing every call on a provider that omits it would be a
// certain outage in exchange for a case that also requires the model to
// emit syntactically complete JSON. The harm a truncated reply does — a
// partial item list replacing a good one — is closed on the write side
// instead, by the guard in `routeFood`/`createFromBatch`.
//
// `length` is the common one: a truncated reply still parses once the JSON
// happens to close, quietly dropping the tail of a long receipt. Any other
// reason (`content_filter`, `tool_calls`) is a different reply shape that
// the strict schema never constrained.
const finish = raw.choices?.[0]?.finish_reason;
if (finish && finish !== "stop") {
throw new ExtractionError(
finish === "length"
? "The receipt was too long to read in one pass."
: `The model stopped before it finished (${finish}).`
);
}
const content = raw.choices?.[0]?.message?.content; const content = raw.choices?.[0]?.message?.content;
if (!content) throw new ExtractionError("LiteLLM returned no content"); if (!content) throw new ExtractionError("LiteLLM returned no content");
try { try {
+28 -3
View File
@@ -20,7 +20,21 @@ export type PantryForwardArgs = {
financeAction: "enrich" | "create"; financeAction: "enrich" | "create";
}; };
export type PantryForwardResult = { captureId: number; itemCount: number | null }; /**
* `alreadyCaptured` is pantry telling us this exact receipt is already on its
* board -- same store, register, receipt number and date, which is the tuple
* both Coles and Woolworths encode in their barcode. Pantry deliberately still
* ACCEPTS the second capture (the first may have been a bad crop being
* replaced) and says so instead of refusing; dropping the field on the floor
* turned "said so" into silence, and a duplicate that is not shown is a
* duplicate that gets confirmed -- which double-counts the groceries, because
* pantry's addItem accumulates.
*/
export type PantryForwardResult = {
captureId: number;
itemCount: number | null;
alreadyCaptured: { captureId: number; confirmed: boolean } | null;
};
export class PantryDuplicateError extends Error { export class PantryDuplicateError extends Error {
constructor( constructor(
@@ -65,12 +79,23 @@ export function pantryForwarder(opts: { baseUrl?: string; token?: string; fetchI
} catch (e) { } catch (e) {
throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`); throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`);
} }
const body = (await response.json().catch(() => ({}))) as { captureId?: number; error?: string; items?: unknown[]; reason?: string }; const body = (await response.json().catch(() => ({}))) as {
captureId?: number; error?: string; items?: unknown[]; reason?: string;
alreadyCaptured?: { captureId?: number; confirmed?: boolean } | null;
};
if (response.status === 409) { if (response.status === 409) {
throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured"); throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured");
} }
if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim()); if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim());
if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId"); if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId");
return { captureId: body.captureId, itemCount: Array.isArray(body.items) ? body.items.length : null }; const prior = body.alreadyCaptured;
return {
captureId: body.captureId,
itemCount: Array.isArray(body.items) ? body.items.length : null,
alreadyCaptured:
prior && typeof prior.captureId === "number" && prior.captureId !== body.captureId
? { captureId: prior.captureId, confirmed: prior.confirmed === true }
: null,
};
}; };
} }
+273 -47
View File
@@ -1,9 +1,14 @@
import { randomUUID } from "node:crypto"; import { randomUUID } from "node:crypto";
import { queryRaw, queryRow } from "@/lib/db"; import { queryRaw, queryRow, withTransaction } from "@/lib/db";
import { EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY } from "@/lib/analytics-sql";
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion"; import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
import { import {
classifyInput, classifyInput,
confabulationReason,
merchantKey, merchantKey,
quoteShared,
textCouldBeReceipt,
merchantMatches, merchantMatches,
platformForSource, platformForSource,
reconcileItems, reconcileItems,
@@ -207,15 +212,33 @@ export async function findDuplicateBatch(
return rows.find((r) => merchantMatches(r.header?.merchant, header.merchant)) ?? null; return rows.find((r) => merchantMatches(r.header?.merchant, header.merchant)) ?? null;
} }
type MatchRow = { transaction_id: number; description: string | null; transaction_date: string; amount: string; merchant: string | null; platform: string | null }; type MatchRow = {
transaction_id: number;
description: string | null;
transaction_date: string;
amount: string;
merchant: string | null;
platform: string | null;
/** What `amount` is denominated in — a candidate row is not always dollars. */
currency: string;
/** The `expense_metadata` row the match resolved through — null when there is none. */
metadata_id: number | null;
/** Does that row already carry a receipt? */
has_line_items: boolean;
};
/** /**
* The match rule (plan §5, D2): `(platform, merchant, date, total)`, amount and * The match rule (plan §5, D2): `(platform, merchant, date, total)`, amount and
* date EXACT, never widened silently. Platform is required when the source * date EXACT, never widened silently. The merchant compares normalised, with
* implies one; an in-store receipt has none and matches on the other three. * whole-word containment ("Coles" vs "Coles (Wyndham Vale)") — that is
* The merchant compares normalised, with containment ("Coles" vs "Coles * spelling, not widening. A unique hit is a match; two hits are an ambiguity
* (Wyndham Vale)") — that is spelling, not widening. A unique hit is a match; * and go to the candidates card like a miss.
* two hits are an ambiguity and go to the candidates card like a miss. *
* A NULL platform means "no platform", not "any platform". An in-store Coles
* receipt carries none, and `$3 IS NULL OR em.platform = $3` let it match an
* Uber Eats Coles order on the same day for the same money — and then overwrite
* that order's items with the till receipt's. The in-store side matches rows
* with no platform of their own; the delivery side still requires its own.
*/ */
export async function matchTransaction(header: OrderHeader, platform: string | null): Promise<{ match: MatchRow | null; ambiguous: boolean }> { export async function matchTransaction(header: OrderHeader, platform: string | null): Promise<{ match: MatchRow | null; ambiguous: boolean }> {
if (!header.merchant) return { match: null, ambiguous: false }; if (!header.merchant) return { match: null, ambiguous: false };
@@ -223,13 +246,15 @@ export async function matchTransaction(header: OrderHeader, platform: string | n
`SELECT DISTINCT ON (t.id) `SELECT DISTINCT ON (t.id)
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text, t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant, COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
em.platform em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency,
COALESCE(jsonb_typeof(em.line_items) = 'array' AND jsonb_array_length(em.line_items) > 0, false) AS has_line_items
FROM transactions t FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
WHERE t.superseded_by_id IS NULL WHERE ${EXCLUDE_RECONCILED_SOURCE}
AND t.transaction_date = $1::date AND t.transaction_date = $1::date
AND t.amount = $2::numeric AND t.amount = $2::numeric
AND ($3::text IS NULL OR em.platform = $3) AND (($3::text IS NULL AND em.platform IS NULL) OR em.platform = $3)
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`, ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
[header.order_date, header.total, platform] [header.order_date, header.total, platform]
); );
@@ -238,21 +263,29 @@ export async function matchTransaction(header: OrderHeader, platform: string | n
return { match: null, ambiguous: hits.length > 1 }; return { match: null, ambiguous: hits.length > 1 };
} }
/** Nearest candidates for the card: same platform and merchant, ±3 days, any amount. */ /**
export async function findCandidates(header: OrderHeader, platform: string | null): Promise<Candidate[]> { * Nearest candidates for the card: same merchant, ±3 days, any amount.
*
* Deliberately NOT filtered by platform. The card is a human choice, not a
* decision — the whole reason it is shown is that the exact key missed, so the
* platform is one of the things that may be wrong. A filter here narrows the
* list at exactly the moment it needs to widen, and the row the user was
* looking for is simply absent with nothing saying why.
*/
export async function findCandidates(header: OrderHeader): Promise<Candidate[]> {
if (!header.merchant) return []; if (!header.merchant) return [];
const rows = await queryRaw<MatchRow>( const rows = await queryRaw<MatchRow>(
`SELECT DISTINCT ON (t.id) `SELECT DISTINCT ON (t.id)
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text, t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant, COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
em.platform em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency, false AS has_line_items
FROM transactions t FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
WHERE t.superseded_by_id IS NULL WHERE ${EXCLUDE_RECONCILED_SOURCE}
AND t.transaction_date BETWEEN $1::date - 3 AND $1::date + 3 AND t.transaction_date BETWEEN $1::date - 3 AND $1::date + 3
AND ($2::text IS NULL OR em.platform = $2)
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`, ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
[header.order_date, platform] [header.order_date]
); );
return rows return rows
.filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant)) .filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant))
@@ -263,7 +296,7 @@ export async function findCandidates(header: OrderHeader, platform: string | nul
return Math.abs(Number(a.amount) - header.total) - Math.abs(Number(b.amount) - header.total); return Math.abs(Number(a.amount) - header.total) - Math.abs(Number(b.amount) - header.total);
}) })
.slice(0, 5) .slice(0, 5)
.map((r) => ({ transaction_id: r.transaction_id, description: r.description, transaction_date: r.transaction_date, amount: r.amount })); .map((r) => ({ transaction_id: r.transaction_id, description: r.description, transaction_date: r.transaction_date, amount: r.amount, currency: r.currency }));
} }
// ─── the two destinations ─────────────────────────────────────────────────── // ─── the two destinations ───────────────────────────────────────────────────
@@ -275,7 +308,7 @@ export type RouteContext = {
platform: string | null; platform: string | null;
header: OrderHeader; header: OrderHeader;
items: StoredLineItem[]; items: StoredLineItem[];
subtotal?: number | null; currency?: string | null;
}; };
const cardInput = (ctx: RouteContext): CardInput => ({ const cardInput = (ctx: RouteContext): CardInput => ({
@@ -285,6 +318,7 @@ const cardInput = (ctx: RouteContext): CardInput => ({
orderDate: ctx.header.order_date, orderDate: ctx.header.order_date,
platform: ctx.platform, platform: ctx.platform,
itemCount: ctx.items.length, itemCount: ctx.items.length,
currency: ctx.currency,
}); });
const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, extra: Partial<IntakeResult> = {}): IntakeResult => ({ const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, extra: Partial<IntakeResult> = {}): IntakeResult => ({
@@ -299,22 +333,56 @@ const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card:
...extra, ...extra,
}); });
/**
* Are these items unverifiable — no line carrying any money at all?
*
* That is `reconcileItems`' `basis: "none"`: pasted text with no prices, or an
* extraction that read the names and none of the figures. The reconciliation
* check passes it, correctly, because there is nothing to check. What must not
* follow is replacing a receipt that IS itemised with one nothing verified.
*/
const unverifiable = (items: StoredLineItem[]) => items.every((i) => i.amount === null);
/**
* The hold, shared by EVERY path that enriches a matched transaction.
*
* It lived only in `routeFood` and `createFromBatch`'s re-match walked straight
* past it — the same upload destroyed the same receipt depending on which
* button reached it. Returns null when there is nothing to protect.
*/
async function holdRatherThanReplace(ctx: RouteContext, match: MatchRow): Promise<IntakeResult | null> {
if (!unverifiable(ctx.items) || !match.has_line_items) return null;
await updateBatch(ctx.batchId, { status: "held", items: ctx.items });
const card = heldCard(
"this upload carries no prices, so nothing checked it against the total — and that order already has its items",
"I left the existing items alone"
);
return finish(ctx, "held", "held", card, { transaction_id: match.transaction_id });
}
/** Food: enrich the matched order's line_items, or show candidates. NEVER auto-creates. */ /** Food: enrich the matched order's line_items, or show candidates. NEVER auto-creates. */
export async function routeFood(ctx: RouteContext): Promise<IntakeResult> { export async function routeFood(ctx: RouteContext): Promise<IntakeResult> {
const { match } = await matchTransaction(ctx.header, ctx.platform); const { match } = await matchTransaction(ctx.header, ctx.platform);
if (match) { if (match) {
await replaceLineItems(match.transaction_id, ctx.items, { source: "intake", platform: ctx.platform, batchId: ctx.batchId }); const held = await holdRatherThanReplace(ctx, match);
if (held) return held;
await replaceLineItems(match.transaction_id, ctx.items, { source: "intake", platform: ctx.platform, batchId: ctx.batchId, metadataId: match.metadata_id, currency: ctx.currency });
await updateBatch(ctx.batchId, { kind: "food", status: "enriched", transaction_id: match.transaction_id, items: ctx.items }); await updateBatch(ctx.batchId, { kind: "food", status: "enriched", transaction_id: match.transaction_id, items: ctx.items });
return finish(ctx, "food", "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id }); return finish(ctx, "food", "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
} }
const candidates = await findCandidates(ctx.header, ctx.platform); const candidates = await findCandidates(ctx.header);
await updateBatch(ctx.batchId, { kind: "food", status: "candidates", items: ctx.items }); await updateBatch(ctx.batchId, { kind: "food", status: "candidates", items: ctx.items });
return finish(ctx, "food", "candidates", candidatesCard(cardInput(ctx), candidates), { candidates }); return finish(ctx, "food", "candidates", candidatesCard(cardInput(ctx), candidates), { candidates });
} }
/** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */ /** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */
export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> { export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> {
const { match } = await matchTransaction(ctx.header, ctx.platform); // `ambiguous` is not a miss. Two ledger rows for one shop is precisely the
// state that produced the 2026-08-18 Woolworths double-book, and forwarding
// with financeAction 'create' as though nothing matched is what turns it into
// a third row on confirm. The groceries still go to the board — they have to,
// it is where they are worked — but the card must say finance could not pick.
const { match, ambiguous } = await matchTransaction(ctx.header, ctx.platform);
try { try {
const r = await pantry({ const r = await pantry({
files, files,
@@ -325,12 +393,19 @@ export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantr
financeAction: match ? "enrich" : "create", financeAction: match ? "enrich" : "create",
}); });
await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items }); await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items });
const card = forwardedCard({ ...cardInput(ctx), itemCount: r.itemCount ?? ctx.items.length, captureId: r.captureId, enriching: !!match }); const card = forwardedCard({
...cardInput(ctx),
itemCount: r.itemCount ?? ctx.items.length,
captureId: r.captureId,
enriching: !!match,
alreadyCaptured: r.alreadyCaptured,
ambiguous,
});
return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id }); return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id });
} catch (e) { } catch (e) {
if (e instanceof PantryDuplicateError) { if (e instanceof PantryDuplicateError) {
await updateBatch(ctx.batchId, { kind: "grocery", status: "duplicate", capture_id: e.captureId, items: ctx.items }); await updateBatch(ctx.batchId, { kind: "grocery", status: "duplicate", capture_id: e.captureId, items: ctx.items });
return finish(ctx, "grocery", "duplicate", duplicateCard({ merchant: ctx.header.merchant, total: ctx.header.total, transactionId: match?.transaction_id ?? null, captureId: e.captureId, batchId: null }), { return finish(ctx, "grocery", "duplicate", duplicateCard({ merchant: ctx.header.merchant, total: ctx.header.total, currency: ctx.currency, transactionId: match?.transaction_id ?? null, captureId: e.captureId, batchId: null }), {
capture_id: e.captureId ?? undefined, capture_id: e.captureId ?? undefined,
}); });
} }
@@ -345,38 +420,144 @@ export async function createFromBatch(batch: BatchRow, kind: "grocery" | "food")
if (!batch.header) throw new Error("batch has no header"); if (!batch.header) throw new Error("batch has no header");
const header = batch.header; const header = batch.header;
const items = batch.items ?? []; const items = batch.items ?? [];
const ctx: RouteContext = {
batchId: batch.id,
source: batch.source,
sharedBy: batch.shared_by,
platform: header.platform,
header,
items,
currency: header.currency,
};
// The card was built minutes ago and the tap arrives now. In between, the
// email/order bridge books orders on its own schedule — so re-run the match
// rather than trusting the "no match" the card was drawn from. Creating a
// second row for an order the ledger already holds is the one outcome this
// whole slice exists to avoid.
const { match } = await matchTransaction(header, header.platform);
if (match) {
const held = await holdRatherThanReplace(ctx, match);
if (held) return held;
await replaceLineItems(match.transaction_id, items, { source: "intake", platform: header.platform, batchId: batch.id, metadataId: match.metadata_id, currency: header.currency });
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: match.transaction_id, items });
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
}
// A sibling batch of the same order — the PDF after the screenshot — may have
// been resolved since. `findDuplicateBatch` deliberately ignores unresolved
// batches, so two pending cards for one order both reach here; without this
// the second tap books the order twice.
const twin = await findDuplicateBatch(batch.id, batch.file_sha256, header);
if (twin) {
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: twin.transaction_id, capture_id: twin.capture_id });
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: twin.transaction_id, captureId: twin.capture_id, batchId: twin.id });
return finish(ctx, kind, "duplicate", card, {
transaction_id: twin.transaction_id ?? undefined,
capture_id: twin.capture_id ?? undefined,
duplicate_of: { batch_id: twin.id, transaction_id: twin.transaction_id, capture_id: twin.capture_id },
});
}
const ownerId = await ownerForEmail(batch.shared_by); const ownerId = await ownerForEmail(batch.shared_by);
// Before minting a row: is one already here that nothing has described yet?
//
// matchTransaction above cannot see it. Its key includes the platform, and a
// wallet notification has none — so an Uber Eats screenshot never matches the
// wallet row for the very tap that paid for it, and the Create button books
// the purchase a second time. That is the normal arrival order, not an edge
// case: the notification lands within seconds of the tap and the screenshot
// is shared later.
//
// findAdoptableTransaction is the same helper the receipt lane uses and
// carries the same guards — exact amount, ±3 days, unclaimed, UNIQUE, and
// never a row that already carries line items. Its uniqueness check runs
// outside the advisory lock below, so two simultaneous taps could both adopt;
// the second would then find the items in place and fall through to the lock,
// which is where duplicates are actually resolved.
const adopted = await findAdoptableTransaction(ownerId, header.total, header.order_date);
if (adopted) {
await replaceLineItems(adopted, items, { source: "intake", platform: header.platform, batchId: batch.id, currency: header.currency });
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: adopted, items });
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: adopted }), { transaction_id: adopted });
}
const merchant = header.merchant ?? "Unknown merchant"; const merchant = header.merchant ?? "Unknown merchant";
// Not `orderDescription()`: it always prefixes "Order - ", which is right for
// a delivery receipt and wrong for the in-store till receipt this path also
// handles. Not `resolveCategory()` either — it re-derives grocery-vs-food
// from a merchant pattern, and `kind` is the answer the router already
// reached, remembered merchant and Slack tap included.
const description = header.platform ? `Order - ${merchant}` : merchant; const description = header.platform ? `Order - ${merchant}` : merchant;
const category = kind === "grocery" ? "groceries" : "dining"; const category = kind === "grocery" ? "groceries" : "dining";
// source/source_ref make a second press of the same card a no-op rather than a second row. // `amount` is the figure printed on the receipt, in its own currency. No FX
const existing = await queryRow<{ id: number }>(`SELECT id FROM transactions WHERE source = 'intake' AND source_ref = $1`, [batch.id]); // rate exists at intake, so a foreign order leaves `amount_aud` NULL rather
const txnId = // than asserting a conversion — the same shape order-ingestion.ts writes.
existing?.id ?? const currency = header.currency ?? "AUD";
( const isAud = currency === "AUD";
await queryRow<{ id: number }>( // Check and insert under one lock, on one connection.
`INSERT INTO transactions ( //
transaction_date, description, amount, amount_aud, category, payment_method, // `source_ref = batch.id` makes a second press of THIS card a no-op, and
merchant_name, merchant_normalized, transaction_type, owner_id, source, source_ref // ON CONFLICT makes that safe against a double tap. It cannot help two
) VALUES ($1::date, $2, $3::numeric, $3::numeric, $4, NULL, $5, $5, 'debit', $6, 'intake', $7) // DIFFERENT batches for one purchase — their keys differ — and both taps run
RETURNING id`, // their match and duplicate checks before either inserts, so both can find
[header.order_date, description, header.total, category, merchant, ownerId, batch.id] // nothing. The lock is keyed on the purchase, not the batch, which is the
) // only thing the two share. It has to be transaction-scoped and therefore
)!.id; // inside withTransaction: taken by a statement of its own it would be
await replaceLineItems(txnId, items, { source: "intake", platform: header.platform, batchId: batch.id }); // released before the next one ran (see db.ts).
const purchaseKey = [header.platform ?? "", merchantKey(merchant), header.order_date, header.total.toFixed(2)].join("|");
const outcome = await withTransaction(async (q) => {
await q.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, [purchaseKey]);
// Mine (a double tap) or another batch's (the race). The ORDER BY is what
// tells them apart when both exist.
const existing = await q.queryRow<{ id: number; source_ref: string }>(
`SELECT id, source_ref FROM transactions
WHERE source = 'intake' AND superseded_by_id IS NULL
AND (source_ref = $1
OR (transaction_date = $2::date AND amount = $3::numeric
AND lower(merchant_normalized) = lower($4)))
ORDER BY (source_ref = $1) DESC NULLS LAST, id
LIMIT 1`,
[batch.id, header.order_date, header.total, merchant]
);
if (existing) return { id: existing.id, mine: existing.source_ref === batch.id };
const row = await q.queryRow<{ id: number }>(
`INSERT INTO transactions (
transaction_date, description, amount, amount_aud, category, payment_method,
merchant_name, merchant_normalized, transaction_type, owner_id, source, source_ref,
foreign_currency_amount, foreign_currency_code
) VALUES ($1::date, $2, $3::numeric, $4::numeric, $5, NULL, $6, $6, 'debit', $7, 'intake', $8, $9::numeric, $10)
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL
DO NOTHING
RETURNING id`,
[header.order_date, description, header.total, isAud ? header.total : null, category, merchant, ownerId, batch.id, isAud ? null : header.total, isAud ? null : currency]
);
return { id: row!.id, mine: true };
});
// Another batch booked this purchase in the moment between the two taps. Say
// the same thing the sequential case says one second later, and write nothing.
if (!outcome.mine) {
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: outcome.id });
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: outcome.id, captureId: null, batchId: null });
return finish(ctx, kind, "duplicate", card, { transaction_id: outcome.id });
}
const txnId = outcome.id;
await replaceLineItems(txnId, items, { source: "intake", platform: header.platform, batchId: batch.id, currency });
await updateBatch(batch.id, { kind, status: "created", transaction_id: txnId }); await updateBatch(batch.id, { kind, status: "created", transaction_id: txnId });
const ctx: RouteContext = { batchId: batch.id, source: batch.source, sharedBy: batch.shared_by, platform: header.platform, header, items };
return finish(ctx, kind, "created", createdCard({ ...cardInput(ctx), transactionId: txnId }), { transaction_id: txnId }); return finish(ctx, kind, "created", createdCard({ ...cardInput(ctx), transactionId: txnId }), { transaction_id: txnId });
} }
/** The card for a batch as it stands — what a second press of a resolved card gets back. */ /** The card for a batch as it stands — what a second press of a resolved card gets back. */
export function resolvedCard(batch: BatchRow): { blocks: Block[]; text: string } { export function resolvedCard(batch: BatchRow): { blocks: Block[]; text: string } {
const h = batch.header; const h = batch.header;
const base: CardInput = { batchId: batch.id, merchant: h?.merchant ?? null, total: h?.total ?? null, orderDate: h?.order_date ?? null, platform: h?.platform ?? null, itemCount: batch.items?.length ?? 0 }; const base: CardInput = { batchId: batch.id, merchant: h?.merchant ?? null, total: h?.total ?? null, orderDate: h?.order_date ?? null, platform: h?.platform ?? null, itemCount: batch.items?.length ?? 0, currency: h?.currency };
if (batch.status === "enriched" && batch.transaction_id) return enrichedCard({ ...base, transactionId: batch.transaction_id }); if (batch.status === "enriched" && batch.transaction_id) return enrichedCard({ ...base, transactionId: batch.transaction_id });
if (batch.status === "created" && batch.transaction_id) return createdCard({ ...base, transactionId: batch.transaction_id }); if (batch.status === "created" && batch.transaction_id) return createdCard({ ...base, transactionId: batch.transaction_id });
if (batch.status === "forwarded_to_pantry" && batch.capture_id) return forwardedCard({ ...base, captureId: batch.capture_id, enriching: !!batch.transaction_id }); if (batch.status === "forwarded_to_pantry" && batch.capture_id) return forwardedCard({ ...base, captureId: batch.capture_id, enriching: !!batch.transaction_id });
return duplicateCard({ merchant: base.merchant, total: base.total, transactionId: batch.transaction_id, captureId: batch.capture_id, batchId: batch.id }); return duplicateCard({ merchant: base.merchant, total: base.total, currency: base.currency, transactionId: batch.transaction_id, captureId: batch.capture_id, batchId: batch.id });
} }
// ─── the router ───────────────────────────────────────────────────────────── // ─── the router ─────────────────────────────────────────────────────────────
@@ -406,6 +587,8 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
// 1. classify each file and build the parts, in the order shared. // 1. classify each file and build the parts, in the order shared.
const parts: InputPart[] = []; const parts: InputPart[] = [];
const usable: StoredFile[] = []; const usable: StoredFile[] = [];
/** Text parts dropped before the model saw them — see `textCouldBeReceipt`. */
const rejectedText: { name: string; text: string; reason: string }[] = [];
for (const f of input.files) { for (const f of input.files) {
const kind = classifyInput(f); const kind = classifyInput(f);
if (kind === "pdf") { if (kind === "pdf") {
@@ -423,7 +606,20 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} else if (kind === "image") { } else if (kind === "image") {
parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name }); parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name });
} else if (kind === "text") { } else if (kind === "text") {
parts.push({ kind: "text", text: f.bytes.toString("utf-8"), name: f.name }); // Nothing that cannot be a receipt reaches the model. A share that
// arrives as the word "Image" (the phone coerced the photo to text) is
// not a thin receipt, and asking anyway produces a whole invented one —
// 2026-08-22, `textCouldBeReceipt`. Dropped like an unreadable file type
// rather than failing the batch, so one junk text part beside a good
// screenshot still goes through; if nothing usable is left the card below
// quotes what the phone actually sent.
const text = f.bytes.toString("utf-8");
const check = textCouldBeReceipt(text);
if (!check.ok) {
rejectedText.push({ name: f.name, text, reason: check.reason });
continue;
}
parts.push({ kind: "text", text, name: f.name });
} else { } else {
continue; continue;
} }
@@ -431,14 +627,23 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} }
if (!parts.length) { if (!parts.length) {
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" }); await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
return fail("unsupported", "unsupported", unsupportedCard()); const dropped = rejectedText[0];
return fail(
"unsupported",
"unsupported",
unsupportedCard(
dropped
? `the share arrived as text and ${dropped.reason}: "${quoteShared(dropped.text)}" — send the picture or the PDF itself`
: undefined
)
);
} }
// A re-share of the very same files, before spending a model call on it. // A re-share of the very same files, before spending a model call on it.
const dupBySha = await findDuplicateBatch(batchId, shas, null); const dupBySha = await findDuplicateBatch(batchId, shas, null);
if (dupBySha) { if (dupBySha) {
await updateBatch(batchId, { status: "duplicate", transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id }); await updateBatch(batchId, { status: "duplicate", transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id });
return fail("unsupported", "duplicate", duplicateCard({ merchant: dupBySha.header?.merchant ?? null, total: dupBySha.header?.total ?? null, transactionId: dupBySha.transaction_id, captureId: dupBySha.capture_id, batchId: dupBySha.id }), 200, { return fail("unsupported", "duplicate", duplicateCard({ merchant: dupBySha.header?.merchant ?? null, total: dupBySha.header?.total ?? null, currency: dupBySha.header?.currency, transactionId: dupBySha.transaction_id, captureId: dupBySha.capture_id, batchId: dupBySha.id }), 200, {
kind: (dupBySha.kind as IntakeKind) ?? "food", kind: (dupBySha.kind as IntakeKind) ?? "food",
duplicate_of: { batch_id: dupBySha.id, transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id }, duplicate_of: { batch_id: dupBySha.id, transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id },
}); });
@@ -462,11 +667,32 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} }
const items = toLineItems(extracted.items); const items = toLineItems(extracted.items);
// 2b. did it answer out of the upload, or out of its prompt?
const confab = confabulationReason(
{
textParts: parts.filter((p): p is Extract<InputPart, { kind: "text" }> => p.kind === "text").map((p) => p.text),
hasImages: parts.some((p) => p.kind === "image"),
},
extracted
);
if (confab) {
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
return fail("unsupported", "unsupported", unsupportedCard(confab));
}
// 3. header. // 3. header.
//
// This path stores the partial header and the items on the batch row, which
// is right for a genuine screenshot missing its top — the next tap has
// something to work from. It is wrong for a confabulation, and on 2026-08-22
// it stored one: an invented Burgertory header with two invented lines, held
// back from the matcher only by an absent date. Nothing here could tell the
// two apart, which is why the pre-check above and 2b now keep confabulations
// from reaching this point at all rather than trying to filter them here.
const headerCheck = validateHeader(extracted); const headerCheck = validateHeader(extracted);
if (!headerCheck.ok) { if (!headerCheck.ok) {
await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" }); await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" });
return fail("held", "needs_header", needsHeaderCard(), 200, { items }); return fail("held", "needs_header", needsHeaderCard(headerCheck.missing, input.source), 200, { items });
} }
const header = headerCheck.header; const header = headerCheck.header;
const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" }; const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" };
@@ -478,7 +704,7 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
const dup = await findDuplicateBatch(batchId, shas, storedHeader); const dup = await findDuplicateBatch(batchId, shas, storedHeader);
if (dup) { if (dup) {
await updateBatch(batchId, { header: storedHeader, items, kind, status: "duplicate", transaction_id: dup.transaction_id, capture_id: dup.capture_id }); await updateBatch(batchId, { header: storedHeader, items, kind, status: "duplicate", transaction_id: dup.transaction_id, capture_id: dup.capture_id });
return fail(kind, "duplicate", duplicateCard({ merchant: header.merchant, total: header.total, transactionId: dup.transaction_id, captureId: dup.capture_id, batchId: dup.id }), 200, { return fail(kind, "duplicate", duplicateCard({ merchant: header.merchant, total: header.total, currency: storedHeader.currency, transactionId: dup.transaction_id, captureId: dup.capture_id, batchId: dup.id }), 200, {
header, header,
items, items,
duplicate_of: { batch_id: dup.id, transaction_id: dup.transaction_id, capture_id: dup.capture_id }, duplicate_of: { batch_id: dup.id, transaction_id: dup.transaction_id, capture_id: dup.capture_id },
@@ -493,7 +719,7 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} }
await updateBatch(batchId, { header: storedHeader, items, kind, status: "received" }); await updateBatch(batchId, { header: storedHeader, items, kind, status: "received" });
const ctx: RouteContext = { batchId, source: input.source, sharedBy: input.sharedBy, platform, header, items, subtotal: extracted.subtotal }; const ctx: RouteContext = { batchId, source: input.source, sharedBy: input.sharedBy, platform, header, items, currency: storedHeader.currency };
// 7. act. // 7. act.
if (kind === "ask") { if (kind === "ask") {
+123 -8
View File
@@ -43,6 +43,7 @@ export function platformForSource(source: IntakeSource): string | null {
export const PLATFORM_LABEL: Record<string, string> = { export const PLATFORM_LABEL: Record<string, string> = {
ubereats: "Uber Eats", ubereats: "Uber Eats",
doordash: "DoorDash", doordash: "DoorDash",
uber: "Uber",
}; };
export function platformLabel(platform: string | null): string { export function platformLabel(platform: string | null): string {
@@ -51,10 +52,16 @@ export function platformLabel(platform: string | null): string {
/** /**
* A small pattern list, not an exact lookup (plan §2, Codex #6). Uber Eats prints * A small pattern list, not an exact lookup (plan §2, Codex #6). Uber Eats prints
* `Coles (Wyndham Vale)`, a PDF may print `Woolworths Group Limited`; substring on * `Coles (Wyndham Vale)`, a PDF may print `Woolworths Group Limited`; a
* the normalised name covers both without a merchant master-data subsystem. * word-boundary alternation on the normalised name covers both without a
* merchant master-data subsystem.
*
* The boundaries are load-bearing, not tidiness: a bare substring made
* `Bigals Pizza` a grocer, because "b-IGA-ls" contains `iga`. Short names in a
* substring list collide with ordinary words, and the collision reads as a
* correct classification.
*/ */
export const GROCERY_PATTERN = /coles|woolworths|woolies|aldi|iga|foodworks|costco/i; export const GROCERY_PATTERN = /\b(coles|woolworths|woolies|aldi|iga|foodworks|costco)\b/i;
/** /**
* The normalised merchant used for matching and for the remembered-kind table. * The normalised merchant used for matching and for the remembered-kind table.
@@ -219,10 +226,118 @@ export function toLineItems(items: ExtractedItem[]): StoredLineItem[] {
}); });
} }
/** Does the stored merchant name denote the extracted one? Exact key, or one contains the other. */ /** Is `needle` a contiguous run of whole words inside `hay`? */
function containsWordRun(hay: string[], needle: string[]): boolean {
if (!needle.length || needle.length > hay.length) return false;
for (let i = 0; i + needle.length <= hay.length; i++) {
if (needle.every((w, j) => hay[i + j] === w)) return true;
}
return false;
}
/**
* Does the stored merchant name denote the extracted one? One name's words must
* appear as a contiguous run inside the other's — so `Coles` matches
* `Coles (Wyndham Vale)` and `Order - Coles (Wyndham Vale)`, which is spelling,
* not widening.
*
* Whole words, never a raw substring: `merchantKey("Bigals Pizza")` contains
* `iga`, so a pizza shop matched every IGA on the same date and total.
*/
export function merchantMatches(stored: string | null | undefined, extracted: string | null | undefined): boolean { export function merchantMatches(stored: string | null | undefined, extracted: string | null | undefined): boolean {
const a = merchantKey(stored); const a = merchantKey(stored).split(" ").filter(Boolean);
const b = merchantKey(extracted); const b = merchantKey(extracted).split(" ").filter(Boolean);
if (!a || !b) return false; if (!a.length || !b.length) return false;
return a === b || a.includes(b) || b.includes(a); return containsWordRun(a, b) || containsWordRun(b, a);
}
// ─── is this text even a receipt? (2026-08-22) ───────────────────────────────
/**
* The merchant names the extraction prompt uses to show the FORMAT. They are
* placeholders rather than real shops on purpose.
*
* On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so the intake
* router received one text part whose entire content was the five characters
* `Image`. The model was handed that, and returned a whole receipt:
* `Burgertory (Braybrook)`, two "The Pride" lines at $22.80, a $41.15 total.
* Nothing on the receipt existed — "Burgertory" was in the prompt, as the
* example of how to print a branch in parentheses. Only the absent date kept it
* out of the matcher; the invented header and items were stored on the batch.
*
* Same shape as the ingestion engine's 2026-08-05 empty-body defect: a stage
* that can be handed nothing has to say so, because the model downstream will
* always answer, and it answers out of its prompt.
*
* Exported from here rather than written into the prompt string so the prompt
* and the guard that rejects an echo of it cannot drift apart.
*/
export const PROMPT_PLACEHOLDER_MERCHANTS = ["<Store> (<Suburb>)"] as const;
/**
* Below this a text part needs a money figure to be believable. NOT a floor on
* its own: `TOTAL $17.00` is twelve characters and is exactly the kind of
* pasted line this door exists to accept.
*/
export const MIN_RECEIPT_TEXT_CHARS = 40;
/** `$17.00`, `17.00`, `12,50` — a printed amount, not merely a digit. */
const MONEY_SHAPE = /(?:[$€£¥]\s*\d|\d[.,]\d{2}(?!\d))/;
export type TextCheck = { ok: true } | { ok: false; reason: string };
/**
* Can this text possibly be a receipt? Deterministic, before any model call.
*
* A receipt has more than one word and at least one number. Everything the
* phone substitutes when a share goes wrong — the literal word `Image`, a bare
* filename like `IMG_0042.jpeg`, an empty body — fails one of those, and none
* of them fails in a way a model will admit to.
*
* Applies to TEXT parts only. A PDF is checked for readable text separately and
* an image is the thing we actually asked for.
*/
export function textCouldBeReceipt(raw: string | null | undefined): TextCheck {
const text = (raw ?? "").trim();
if (!text) return { ok: false, reason: "it was empty" };
if (!/\s/.test(text)) return { ok: false, reason: "it is a single word" };
if (!/\d/.test(text)) return { ok: false, reason: "it has no numbers in it" };
if (text.length < MIN_RECEIPT_TEXT_CHARS && !MONEY_SHAPE.test(text)) {
return { ok: false, reason: "it is too short to be a receipt" };
}
return { ok: true };
}
/** What the phone actually sent, short enough to put on a card. */
export function quoteShared(raw: string, max = 80): string {
const flat = (raw ?? "").replace(/\s+/g, " ").trim();
return flat.length > max ? `${flat.slice(0, max - 1)}…` : flat;
}
/**
* Belt to `textCouldBeReceipt`'s braces: did the extractor answer out of its
* prompt rather than out of the upload?
*
* Two tells, both cheap and both deterministic:
* - the merchant IS one of the prompt's format placeholders; or
* - nothing readable was sent at all (no images, and no text part that could
* be a receipt) and yet a merchant or items came back.
*
* The second is reachable even with the pre-check in place, because the
* pre-check only inspects text parts: a PDF that extracts to a few characters
* of junk still reaches the model. This is `unsupported`, never `needs_header`
* — a confabulated header is not a partial one.
*/
export function confabulationReason(
input: { textParts: string[]; hasImages: boolean },
extracted: { merchant: string | null; items: unknown[] }
): string | null {
const key = merchantKey(extracted.merchant);
if (key && PROMPT_PLACEHOLDER_MERCHANTS.some((p) => merchantKey(p) === key)) {
return "the merchant came back as the prompt's own example rather than a name off the receipt";
}
if (input.hasImages) return null;
if (input.textParts.some((t) => textCouldBeReceipt(t).ok)) return null;
if (!extracted.merchant && !extracted.items.length) return null;
return "nothing shared could be read as a receipt, yet an order came back";
} }
+97 -21
View File
@@ -1,4 +1,4 @@
import { queryRow, queryRaw } from "@/lib/db"; import { queryRow } from "@/lib/db";
/** /**
* Replace the itemised receipt behind a transaction (plan §11 C2). * Replace the itemised receipt behind a transaction (plan §11 C2).
@@ -15,6 +15,12 @@ import { queryRow, queryRaw } from "@/lib/db";
* transaction has no row at all (a manual transaction, a statement line), one * transaction has no row at all (a manual transaction, a statement line), one
* is created so the items have somewhere to live; a transaction is NEVER * is created so the items have somewhere to live; a transaction is NEVER
* created here. * created here.
*
* `opts.metadataId` pins WHICH row when the caller already resolved one. A
* statement transaction can own its own `expense_metadata` row AND be the
* `matched_transaction_id` of a delivery row: the match query and this write
* then re-pick independently and can land on different rows, so the enrichment
* reports success against a row nobody reads.
*/ */
export type LineItemsSource = "intake" | "pantry"; export type LineItemsSource = "intake" | "pantry";
@@ -55,10 +61,35 @@ export type ReplaceResult = {
line_items: LineItemRecord[]; line_items: LineItemRecord[];
}; };
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
/**
* The undo baseline, recorded once per batch.
*
* Returning the row that was just read makes a RETRY destroy the thing undo
* exists to restore: pantry re-calls this on every confirm and after a timeout,
* and the second call would hand back its own write as the "previous" state.
* The first snapshot a batch ever takes is the only one worth keeping, so it is
* written under COALESCE and read back rather than trusted.
*/
async function undoBaseline(batchId: string | null | undefined, snapshot: unknown[]): Promise<unknown[]> {
// Pantry passes back whatever batch id it was given; only an intake batch has
// a row to record against, and a non-uuid would fail the cast.
if (!batchId || !UUID.test(batchId)) return snapshot;
const row = await queryRow<{ previous_line_items: unknown }>(
`UPDATE intake_batches
SET previous_line_items = COALESCE(previous_line_items, $2::jsonb)
WHERE id = $1::uuid
RETURNING previous_line_items`,
[batchId, JSON.stringify(snapshot)]
);
return Array.isArray(row?.previous_line_items) ? row.previous_line_items : snapshot;
}
export async function replaceLineItems( export async function replaceLineItems(
transactionId: number, transactionId: number,
lineItems: LineItemRecord[], lineItems: LineItemRecord[],
opts: { source: LineItemsSource; platform?: string | null; batchId?: string | null } opts: { source: LineItemsSource; platform?: string | null; batchId?: string | null; metadataId?: number | null; currency?: string | null }
): Promise<ReplaceResult> { ): Promise<ReplaceResult> {
const txn = await queryRow<{ const txn = await queryRow<{
id: number; id: number;
@@ -73,36 +104,78 @@ export async function replaceLineItems(
); );
if (!txn) throw new TransactionNotFoundError(`transaction ${transactionId} does not exist`); if (!txn) throw new TransactionNotFoundError(`transaction ${transactionId} does not exist`);
// Prefer the row that IS this transaction over one that merely points at it — // The caller's row wins where it still exists; otherwise prefer the row that
// a statement line can be the `matched_transaction_id` of a receipt row and // IS this transaction over one that merely points at it — a statement line
// carry its own row too, and the items belong on the receipt. // can be the `matched_transaction_id` of a receipt row and carry its own row
// too, and the items belong on the receipt.
const pinned = opts.metadataId
? await queryRow<{ id: number; line_items: unknown }>(
`SELECT id, line_items FROM expense_metadata
WHERE id = $1 AND (transaction_id = $2 OR matched_transaction_id = $2)`,
[opts.metadataId, transactionId]
)
: null;
const json = JSON.stringify(lineItems);
/**
* Snapshot the baseline, then write. In that order on BOTH paths: the create
* path used to snapshot after its INSERT, so a crash in between left a retry
* recording the intake's own write as "previous" — undo restoring the thing
* it was meant to reverse, which is the failure `previous_line_items` exists
* to prevent.
*
* Returns null when the row is gone between the read and the write. That
* UPDATE reported success and the batch was marked enriched against a row
* nothing had written to.
*/
const write = async (row: { id: number; line_items: unknown }): Promise<ReplaceResult | null> => {
const previous = await undoBaseline(opts.batchId, Array.isArray(row.line_items) ? row.line_items : []);
const updated = await queryRow<{ id: number }>(
`UPDATE expense_metadata SET line_items = $2::jsonb WHERE id = $1 RETURNING id`,
[row.id, json]
);
if (!updated) return null;
return { transaction_id: transactionId, metadata_id: row.id, created_metadata: false, previous_line_items: previous, line_items: lineItems };
};
if (pinned) {
const done = await write(pinned);
if (done) return done;
}
const existing = await queryRow<{ id: number; line_items: unknown }>( const existing = await queryRow<{ id: number; line_items: unknown }>(
`SELECT id, line_items FROM expense_metadata `SELECT id, line_items FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
ORDER BY (transaction_id = $1) DESC, id -- NULLS LAST is load-bearing, and its absence inverted this clause.
-- Postgres orders DESC as NULLS FIRST, and the row that merely POINTS at
-- the transaction has a NULL transaction_id — so the comparison meant to
-- prefer the owned row preferred the other one, silently and in exactly
-- the case it was written for. matchTransaction spells it out; this did
-- not, and the two therefore disagreed.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[transactionId] [transactionId]
); );
const json = JSON.stringify(lineItems);
if (existing) { if (existing) {
await queryRaw(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE id = $1`, [existing.id, json]); const done = await write(existing);
return { if (done) return done;
transaction_id: transactionId,
metadata_id: existing.id,
created_metadata: false,
previous_line_items: Array.isArray(existing.line_items) ? existing.line_items : [],
line_items: lineItems,
};
} }
// Nothing to update. Snapshot the empty baseline BEFORE creating the row.
const previous = await undoBaseline(opts.batchId, []);
const reference = opts.batchId ? `intake:${opts.batchId}` : `intake:txn:${transactionId}`; const reference = opts.batchId ? `intake:${opts.batchId}` : `intake:txn:${transactionId}`;
const created = await queryRow<{ id: number }>( // ON CONFLICT because two calls for one transaction can both find no row and
// both INSERT; expense_metadata.transaction_id is UNIQUE, so one of them lost
// with a constraint error that reached the user as an error card. `xmax = 0`
// is how a plain insert is told from a conflict-update, so the reported
// `created_metadata` stays true.
const created = await queryRow<{ id: number; inserted: boolean }>(
`INSERT INTO expense_metadata ( `INSERT INTO expense_metadata (
transaction_id, source, platform, order_reference, line_items, transaction_id, source, platform, order_reference, line_items,
merchant_normalized, amount, transaction_date, currency, flags, extraction_model merchant_normalized, amount, transaction_date, currency, flags, extraction_model
) VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::numeric, $8::date, 'AUD', '[]'::jsonb, 'cloud-budget') ) VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::numeric, $8::date, $9, '[]'::jsonb, 'cloud-budget')
RETURNING id`, ON CONFLICT (transaction_id) DO UPDATE SET line_items = EXCLUDED.line_items
RETURNING id, (xmax = 0) AS inserted`,
[ [
transactionId, transactionId,
opts.source, opts.source,
@@ -112,13 +185,16 @@ export async function replaceLineItems(
txn.merchant_normalized ?? txn.merchant_name ?? null, txn.merchant_normalized ?? txn.merchant_name ?? null,
txn.amount, txn.amount,
txn.transaction_date, txn.transaction_date,
// Not hard-coded 'AUD': the transaction may carry a foreign figure, and
// OrderDetails prefers this column, so a USD order printed "$".
opts.currency || "AUD",
] ]
); );
return { return {
transaction_id: transactionId, transaction_id: transactionId,
metadata_id: created!.id, metadata_id: created!.id,
created_metadata: true, created_metadata: created!.inserted,
previous_line_items: [], previous_line_items: previous,
line_items: lineItems, line_items: lineItems,
}; };
} }
-7
View File
@@ -12,13 +12,6 @@ export const CUTOVER_DATE = "2026-01-09";
*/ */
export const DEFAULT_OWNER_ID = 1; export const DEFAULT_OWNER_ID = 1;
/** Human labels for the platform a receipt came from. */
export const PLATFORM_LABEL: Record<ParsedOrder["platform"], string> = {
doordash: "DoorDash",
ubereats: "Uber Eats",
uber: "Uber",
};
/** /**
* Transaction description. * Transaction description.
* *
+5 -1
View File
@@ -526,7 +526,11 @@ export function parseOrderHTML(html: string, meta: MessageMeta): ParsedOrder {
} else { } else {
// DoorDash carries no order id anywhere in the receipt, so the provider // DoorDash carries no order id anywhere in the receipt, so the provider
// message id is the only stable identity available. That is correct for // message id is the only stable identity available. That is correct for
// ingestion idempotency: one receipt is one order. // ingestion idempotency: one receipt is one order — PROVIDED every caller
// sends the same id for the same email. Two lanes post here (n8n Order
// Processor and ingestion-engine order_handoff); both must send the bare
// RFC Message-ID. n8n sent Gmail's own id until 2026-10-03 and one order
// became txns 7268 + 7269.
if (!meta.messageId) { if (!meta.messageId) {
throw new OrderParseError("no order id in body and no messageId supplied"); throw new OrderParseError("no order id in body and no messageId supplied");
} }
+28 -3
View File
@@ -119,6 +119,16 @@ export async function merchantForTransaction(
const row = await queryRow<{ merchant_normalized: string | null }>( const row = await queryRow<{ merchant_normalized: string | null }>(
`SELECT merchant_normalized FROM expense_metadata `SELECT merchant_normalized FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[transactionId] [transactionId]
); );
@@ -157,9 +167,24 @@ export async function merchantVerdict(
FROM order_reviews r FROM order_reviews r
JOIN transactions t ON t.id = r.transaction_id JOIN transactions t ON t.id = r.transaction_id
JOIN participants p ON p.id = r.participant_id JOIN participants p ON p.id = r.participant_id
JOIN expense_metadata em -- (No backticks in here: this SQL lives in a TS template literal and a
ON em.transaction_id = r.transaction_id -- backtick ends the string -- TS1005 on a line that looks like a comment.)
OR em.matched_transaction_id = r.transaction_id -- LATERAL, not a plain JOIN. A plain join on
-- transaction_id OR matched_transaction_id fans out: one review
-- joined to N metadata rows appears N times in the result, and the loops
-- below count every appearance -- inflating the loved/never tallies
-- that drive the pre-order warning. LIMIT 50 applies to the fanned-out
-- set too, so duplicates would also push older reviews out of the
-- window. Same tie-break as every other reader, so the merchant matched
-- here is the merchant the UI shows.
JOIN LATERAL (
SELECT em2.merchant_normalized
FROM expense_metadata em2
WHERE em2.transaction_id = r.transaction_id
OR em2.matched_transaction_id = r.transaction_id
ORDER BY (em2.transaction_id = r.transaction_id) DESC NULLS LAST, em2.id
LIMIT 1
) em ON true
-- Case-folded: the platforms capitalise the same restaurant differently -- Case-folded: the platforms capitalise the same restaurant differently
-- ("TEG Kebabs & Biryani" on Uber Eats, "TEG KEBABS & BIRYANI" on -- ("TEG Kebabs & Biryani" on Uber Eats, "TEG KEBABS & BIRYANI" on
-- DoorDash). An exact match split one restaurant's history in two, so a -- DoorDash). An exact match split one restaurant's history in two, so a
+52 -1
View File
@@ -367,6 +367,16 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
SELECT em.route, em.platform SELECT em.route, em.platform
FROM expense_metadata em FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
LIMIT 1 LIMIT 1
) order_ctx ON true ) order_ctx ON true
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
@@ -841,6 +851,15 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
WHERE ts.transaction_id = t.id WHERE ts.transaction_id = t.id
) txn_splits ON true ) txn_splits ON true
WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL
-- A superseded row is already resolved, just not by reconciling. Without
-- this the queue kept offering it: a duplicate wallet half hidden under
-- its twin (txn 7037, 2026-10-03) sat beside that twin with the same
-- statement line as its match, and accepting it on the hidden row would
-- have used the line up and stranded the real one.
AND t.superseded_by_id IS NULL
-- A voided row is resolved too: the charge never posted, so no statement
-- line will ever arrive for it (migration 0040).
AND t.voided_at IS NULL
AND ${needsCardMatch("t")} AND ${needsCardMatch("t")}
ORDER BY t.transaction_date DESC, t.row_index ASC`, ORDER BY t.transaction_date DESC, t.row_index ASC`,
[ownerId] [ownerId]
@@ -877,12 +896,39 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
FROM transactions m FROM transactions m
JOIN transactions t ON t.statement_id IS NOT NULL JOIN transactions t ON t.statement_id IS NOT NULL
AND t.transaction_date BETWEEN m.transaction_date - 3 AND m.transaction_date + 3 AND t.transaction_date BETWEEN m.transaction_date - 3 AND m.transaction_date + 3
AND t.amount BETWEEN m.amount * 0.99 AND m.amount * 1.01 -- +-5%, NOT +-1%. A foreign card purchase is charged at the notified
-- amount PLUS the card's foreign transaction fee (3% on the Westpac
-- Altitude), so the wallet row and its statement line can never match
-- exactly and the old +-1% band excluded the statement row outright:
-- SP PEAKDESIGN was notified at $42.44 and charged $43.71 (x1.03),
-- so the one row that WAS its twin was the one the picker could not
-- offer. Widening is safe here because this list is capped at 5 by
-- the rn window and ordered by amount closeness; a looser band changes
-- which five appear, never how many, and a person confirms each one.
-- The AUTO-matcher stays exact on purpose; see findAutoMatches().
AND (t.amount BETWEEN m.amount * 0.95 AND m.amount * 1.05
-- Or the exact native pair, whatever the AUD figures say. A
-- wallet row's AUD is provisional: in Oct 2026 it priced INR at
-- 0.0180 while CommBank charged ~0.0149, 20% under, so the band
-- excluded every rupee twin. findAutoMatches() matches on this
-- pair already, but refuses when two lines share it inside its
-- window (INR 112 on 10/4 and again on 10/7) -- and those are
-- exactly the rows that land here for a person to choose.
OR (m.foreign_currency_code IS NOT NULL
AND t.foreign_currency_code = m.foreign_currency_code
AND t.foreign_currency_amount = m.foreign_currency_amount))
JOIN statements s ON s.id = t.statement_id JOIN statements s ON s.id = t.statement_id
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
WHERE m.statement_id IS NULL WHERE m.statement_id IS NULL
AND m.owner_id = $1 AND m.owner_id = $1
AND m.reconciled_with_id IS NULL AND m.reconciled_with_id IS NULL
AND m.superseded_by_id IS NULL
AND m.voided_at IS NULL
-- Nor offer a superseded STATEMENT row as the match: the ANZ
-- statements 107/142/143 left 31 of those, and a manual row
-- reconciled to one inherits its exclusion from every figure.
-- findAutoMatches() has had both filters all along.
AND t.superseded_by_id IS NULL
AND ${needsCardMatch("m")} AND ${needsCardMatch("m")}
AND COALESCE(t.owner_id, s.owner_id) = $1 AND COALESCE(t.owner_id, s.owner_id) = $1
AND NOT EXISTS ( AND NOT EXISTS (
@@ -980,6 +1026,11 @@ export async function getSharedTransactions(ownerId: number, tagIds?: number[],
SELECT em.platform SELECT em.platform
FROM expense_metadata em FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
-- Same tie-break as getTransactions / replaceLineItems / matchTransaction,
-- so every reader resolves the row a write lands on. Without the ORDER BY
-- this LIMIT 1 is bounded but arbitrary: it would disagree with the main
-- transaction list for the same transaction the day multiplicity appears.
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
LIMIT 1 LIMIT 1
) order_ctx ON true ) order_ctx ON true
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
+68 -8
View File
@@ -1,5 +1,6 @@
import { queryRaw, queryRow } from "@/lib/db"; import { queryRaw, queryRow } from "@/lib/db";
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion"; import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
/** /**
* Grocery receipts scanned in pantry-app, arriving as candidate spend. * Grocery receipts scanned in pantry-app, arriving as candidate spend.
@@ -154,16 +155,75 @@ export async function processReceiptIngestion(receipt: ParsedReceipt): Promise<R
const paymentMethod = paymentMethodFor(leg.class); const paymentMethod = paymentMethodFor(leg.class);
const amount = round2(Number(leg.amount)); const amount = round2(Number(leg.amount));
const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name; const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name;
const txn = await queryRow<{ id: number }>(
`INSERT INTO transactions ( // Adopt the row the wallet lane already created for this tap, if there is
transaction_date, description, amount, amount_aud, category, payment_method, // one. Signals for a purchase arrive in sequence -- the notification within
merchant_name, merchant_normalized, transaction_type, owner_id // seconds, the receipt hours later -- so creating a second transaction here
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`, // is how one shop ends up as two rows. Measured on txn 6858: the wallet
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID] // event arrived 35h25m before this path created its own row for the same
); // $103.30 Woolworths shop. Null means no unambiguous candidate, and the
// fallback is exactly the old behaviour.
const adoptedId = await findAdoptableTransaction(DEFAULT_OWNER_ID, amount, receipt.transaction_date);
const txn = adoptedId
? await queryRow<{ id: number }>(
// The receipt read the register tape; the notification read a push
// banner. So the receipt wins on what was bought and where, and the
// column list here IS the ownership boundary -- `location` is absent
// because the wallet notification's suburb is the better value and
// must survive.
`UPDATE transactions
SET description = $2, merchant_name = $3, merchant_normalized = $3,
category = $4, payment_method = $5
WHERE id = $1 RETURNING id`,
[adoptedId, description, receipt.merchant_name, RECEIPT_CATEGORY, paymentMethod]
)
: await queryRow<{ id: number }>(
`INSERT INTO transactions (
transaction_date, description, amount, amount_aud, category, payment_method,
merchant_name, merchant_normalized, transaction_type, owner_id
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
);
const carriesLines = leg.leg_index === cardLeg?.leg_index; const carriesLines = leg.leg_index === cardLeg?.leg_index;
const meta = await queryRow<{ id: number }>(
// An adopted row may already carry a metadata row (the wallet lane writes
// one). expense_metadata.transaction_id is UNIQUE, so a second INSERT would
// raise -- and must not, because the two describe the same purchase.
// Enrich in place instead, with the same ownership split as the transaction
// update above: everything the receipt read wins, and anything only the
// notification knew (issuer, and the transaction's location) is absent from
// the column list rather than conditionally skipped, so it cannot be
// clobbered by a bug here. Mirrors order_transaction_bridge's UPGRADE pass.
const priorMeta = adoptedId
? await queryRow<{ id: number }>(
`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [adoptedId]
)
: null;
const meta = priorMeta
? await queryRow<{ id: number }>(
`UPDATE expense_metadata
SET source = 'pantry', order_reference = $2, line_items = $3::jsonb,
subtotal = $4, amount = $5, merchant_normalized = $6,
transaction_date = $7, card_last4 = COALESCE($8, card_last4),
currency = 'AUD', flags = flags || $9::jsonb,
payment_method = $10, payment_method_detail = $11,
tender_raw = COALESCE($12, tender_raw),
receipt_sha256 = $13, receipt_group = $14,
extraction_model = 'cloud-budget'
WHERE id = $1 RETURNING id`,
[
priorMeta.id, reference,
JSON.stringify(carriesLines ? receipt.line_items ?? [] : []),
receipt.tax_amount ?? null, amount, receipt.merchant_name,
receipt.transaction_date, leg.card_last4 ?? null,
JSON.stringify(carriesLines ? [...flags, "enriched_prior_row"] : [...flags, "line_items_on_card_leg", "enriched_prior_row"]),
paymentMethod, leg.card_product ?? null, receipt.tender_raw ?? null,
receipt.image_sha256 ?? null, group,
]
)
: await queryRow<{ id: number }>(
`INSERT INTO expense_metadata ( `INSERT INTO expense_metadata (
transaction_id, source, order_reference, line_items, subtotal, amount, transaction_id, source, order_reference, line_items, subtotal, amount,
merchant_normalized, transaction_date, card_last4, currency, flags, merchant_normalized, transaction_date, card_last4, currency, flags,
+248
View File
@@ -0,0 +1,248 @@
import { prisma } from "@/lib/db";
/**
* Merge a statementless ("manual") transaction into its statement twin.
*
* Extracted from POST /api/transactions/reconcile so the manual queue and the
* automatic pass cannot drift apart. There must be exactly one implementation
* of this: it moves curation (overrides, tags, splits) and provenance
* (expense_metadata) off a row that reconciliation is about to hide from every
* figure, and a second copy of that logic would eventually move a different set.
*
* The manual row is NOT deleted -- it is linked via `reconciled_with_id`.
* Deleting it would cascade its children away; superseding/linking keeps the
* history and lets the row stay visible as provenance.
*/
export async function applyReconcileMatch(
manual_id: number,
statement_tx_id: number
): Promise<void> {
await prisma.$transaction(async (tx) => {
await moveCurationWithin(tx, manual_id, statement_tx_id);
// Mark manual tx as reconciled (link to statement tx)
await tx.$executeRawUnsafe(
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
statement_tx_id,
manual_id
);
});
}
/**
* Move overrides, tags, splits and expense_metadata from one transaction to
* another, inside a caller-supplied transaction.
*
* Callers: `applyReconcileMatch` (then sets `reconciled_with_id`) and
* `supersedeTransaction` (then sets `superseded_by_id`). Both hide the source
* row from every figure, so both must carry its curation across first; keeping
* one implementation is what the file header insists on.
*/
async function moveCurationWithin(
tx: Parameters<Parameters<typeof prisma.$transaction>[0]>[0],
manual_id: number,
statement_tx_id: number
): Promise<void> {
{
// Copy overrides: manual → statement tx
const override = await tx.transaction_overrides.findUnique({
where: { transaction_id: manual_id },
});
if (override) {
await tx.transaction_overrides.upsert({
where: { transaction_id: statement_tx_id },
update: {
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
updated_at: new Date(),
},
create: {
transaction_id: statement_tx_id,
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
},
});
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
}
// Move tags: manual → statement tx
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
if (tags.length) {
await tx.transaction_tags.createMany({
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
skipDuplicates: true,
});
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
}
// Move splits: manual → statement tx.
//
// `settled` and `settled_at` MUST be carried. This is a delete-and-recreate,
// and the column defaults to false — so omitting it silently marks every
// settled split unsettled again and resurrects a debt that was already
// paid. Nothing errors and nothing in the UI says the flag was reset; it
// just reappears as owed. No queue row carries a settled split today, but
// the auto-matcher runs this path unattended, so a latent version of this
// would surface as money, not as an exception.
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
if (splits.length) {
await tx.transaction_splits.createMany({
data: splits.map((s) => ({
transaction_id: statement_tx_id,
participant_id: s.participant_id,
share_percent: s.share_percent,
settled: s.settled ?? false,
settled_at: s.settled_at,
})),
skipDuplicates: true,
});
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
}
// Move provenance: manual → statement tx.
//
// Overrides, tags and splits above were always carried across; expense_metadata was
// the one child left behind, which did not matter while every metadata row came from
// an email that had created its own transaction. It matters now: a scanned grocery
// receipt puts its line items here, and reconciliation hides the manual row from
// every figure — so without this the shop's contents disappear at exactly the moment
// the statement line appears, and COLES 0556 MANOR LAKES stays as unreadable as it
// was before the receipt was ever scanned.
//
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
// or Paperless copy got there first) keeps it. The manual row's metadata stays
// attached to the reconciled manual transaction and is flagged, rather than raising a
// constraint violation or silently overwriting the other source.
const moved = await tx.$executeRawUnsafe(
`UPDATE expense_metadata SET transaction_id = $1
WHERE transaction_id = $2
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
statement_tx_id,
manual_id
);
if (moved === 0) {
await tx.$executeRawUnsafe(
`UPDATE expense_metadata
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
WHERE transaction_id = $1`,
manual_id
);
}
}
}
/**
* Manually mark `loser_id` as superseded by `survivor_id`.
*
* The escape hatch for a duplicate that `applyReconcileMatch` cannot express:
* that one merges a statementLESS row into its statement twin, so it has
* nothing to say about two rows on the SAME footing -- the pair left by a
* statement imported twice (migration 0023 superseded 31 such pairs, $42,040.68,
* by hand), or two feed rows for one purchase.
*
* Curation moves FIRST, using the same block `applyReconcileMatch` runs, because
* `superseded_by_id` hides the row from every figure (`analytics-sql.ts`) and
* splits or tags left on it would silently stop counting. That is why this lives
* beside its sibling and calls into the same movement rather than re-deriving
* it: the header of this file warns that a second copy would eventually move a
* different set, and a supersede that forgot splits would resurrect a debt in
* exactly the way `applyReconcileMatch`'s own comment describes.
*
* NOT a delete. Every child of `transactions` is ON DELETE CASCADE, so removing
* the row would destroy the splits, tags and metadata that make it legible --
* and a duplicate is evidence of how the ledger was assembled, which is worth
* keeping. `transactions_no_self_supersede` blocks a row pointing at itself; the
* survivor is re-checked here anyway so the error is a 400 and not a 500.
*/
export async function supersedeTransaction(
loser_id: number,
survivor_id: number
): Promise<void> {
if (loser_id === survivor_id) throw new Error("A transaction cannot supersede itself");
await prisma.$transaction(async (tx) => {
// Refuse to bury a row underneath one that is itself already buried, which
// would make the curation we just moved unreachable in a second hop.
const survivor = await tx.transactions.findUnique({
where: { id: survivor_id },
select: { superseded_by_id: true },
});
if (!survivor) throw new Error(`Survivor transaction ${survivor_id} not found`);
if (survivor.superseded_by_id !== null) {
throw new Error(`Transaction ${survivor_id} is itself superseded; pick the surviving row`);
}
await moveCurationWithin(tx, loser_id, survivor_id);
await tx.$executeRawUnsafe(
`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`,
survivor_id,
loser_id
);
});
}
/**
* Mark a statementless row as a charge that never posted (migration 0040).
*
* The case neither sibling above can express: a card notification is sent for
* an AUTHORISATION, and when the merchant releases the hold no statement line
* ever arrives. Txn 7951 was a $1,040.41 Airbnb hold on a booking cancelled the
* next day -- it sat in the reconcile queue with nothing to match and counted as
* spend in every figure. There is no survivor to move curation onto, so nothing
* moves: the row and its children stay, and `EXCLUDE_RECONCILED_SOURCE` stops
* them counting.
*
* Refusals, all deliberate:
* - a statement row: that is the bank's record of a posted charge, and the
* statement must keep summing to its header (`transactions_void_statementless`
* enforces this in the database too). A wrong statement row is a supersede.
* - a row already reconciled or superseded: it is already out of every figure,
* and a reconciled one evidently DID post.
* - a row with a SETTLED split: someone has already paid their share of it.
* Voiding would silently turn that payment into a credit on their balance;
* that needs a person to look at it, not a side effect.
*/
export async function voidTransaction(id: number, reason: string): Promise<void> {
const why = reason.trim();
if (!why) throw new Error("A void needs a reason");
await prisma.$transaction(async (tx) => {
const rows = await tx.$queryRawUnsafe<
{ statement_id: number | null; reconciled_with_id: number | null; superseded_by_id: number | null; settled: number }[]
>(
`SELECT t.statement_id, t.reconciled_with_id, t.superseded_by_id,
(SELECT count(*)::int FROM transaction_splits ts
WHERE ts.transaction_id = t.id AND ts.settled) AS settled
FROM transactions t WHERE t.id = $1 FOR UPDATE`,
id
);
const row = rows[0];
if (!row) throw new Error(`Transaction ${id} not found`);
if (row.statement_id !== null) {
throw new Error(`Transaction ${id} is a statement line; a posted charge cannot be voided (supersede it instead)`);
}
if (row.reconciled_with_id !== null) {
throw new Error(`Transaction ${id} is reconciled with ${row.reconciled_with_id}, so it posted`);
}
if (row.superseded_by_id !== null) {
throw new Error(`Transaction ${id} is already superseded by ${row.superseded_by_id}`);
}
if (row.settled > 0) {
throw new Error(`Transaction ${id} has ${row.settled} settled split(s); resolve those before voiding`);
}
await tx.$executeRawUnsafe(
`UPDATE transactions SET voided_at = now(), void_reason = $1 WHERE id = $2 AND voided_at IS NULL`,
why,
id
);
});
}
/** Undo `voidTransaction` -- e.g. the "released" hold posted after all. */
export async function unvoidTransaction(id: number): Promise<void> {
await prisma.$executeRawUnsafe(
`UPDATE transactions SET voided_at = NULL, void_reason = NULL WHERE id = $1`,
id
);
}
+120
View File
@@ -0,0 +1,120 @@
/**
* Deterministic parsing of the bits of a wallet notification that have a fixed
* shape.
*
* Everything here was previously left to the interpreting model, which handled
* ONE fixed-width format three different ways on the same day's data: it kept
* the country code on "WYNDHAM VAL AU", stripped it on "Sydney" and "Perth",
* and swallowed "SAN FRANCISOUS" whole into the location field. A fixed-width
* field is not a judgement call, and the same argument that put the order
* parsers in code rather than in a Gemini call applies: a regex fails loudly on
* a template change, where a model returns a plausible wrong answer.
*
* The model still decides what the amount and merchant ARE -- it is better at
* that than a pattern, because five cards emit three shapes. This only handles
* the parts that are positional.
*/
/**
* Westpac writes the merchant/city/country block as a fixed-width statement
* descriptor: 27-char merchant, 12-char city, 2-char ISO country. Verified
* across every Westpac event captured:
*
* Breadworld Bakery Cafe WYNDHAM VAL AU
* PLA DONATION MELBOURNE AU
* SP PEAKDESIGN SAN FRANCISOUS <- "SAN FRANCISO" + "US"
*
* The last example is why this is positional rather than a split on whitespace:
* a 12-character city fills its field exactly and runs straight into the
* country code with no separator.
*/
export type WestpacDescriptor = {
merchant: string | null;
city: string | null;
/** ISO-3166 alpha-2, upper-case. Null when the tail is not a country code. */
country: string | null;
};
const MERCHANT_WIDTH = 27;
const CITY_WIDTH = 12;
/**
* `body` is the whole notification body, e.g.
* "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
* Returns nulls rather than throwing: a template change must degrade to "we
* don't know", never to a wrong country on a foreign transaction.
*/
export function parseWestpacDescriptor(body: string | null | undefined): WestpacDescriptor {
const none: WestpacDescriptor = { merchant: null, city: null, country: null };
if (!body) return none;
// The descriptor is whatever follows the first sentence ("... ending...8032. ").
// [\s\S] rather than the `s` flag: the tsconfig target predates es2018.
const m = body.match(/^[\s\S]*?\.\s([\s\S]*)$/);
if (!m) return none;
const tail = m[1].replace(/\s+$/, "");
if (!tail) return none;
const merchant = tail.slice(0, MERCHANT_WIDTH).trim() || null;
const rest = tail.slice(MERCHANT_WIDTH);
if (!rest) return { merchant, city: null, country: null };
// Country is the final two characters of the fixed block, but only when the
// block is actually long enough to contain one -- a short tail is a merchant
// name with no location, not a merchant whose last two letters are a country.
const block = rest.slice(0, CITY_WIDTH + 2);
if (block.length < CITY_WIDTH + 2) {
return { merchant, city: block.trim() || null, country: null };
}
const city = block.slice(0, CITY_WIDTH).trim() || null;
const tailTwo = block.slice(CITY_WIDTH).trim().toUpperCase();
const country = /^[A-Z]{2}$/.test(tailTwo) ? tailTwo : null;
return { merchant, city, country: country };
}
/**
* The Apple Wallet notification's subtitle, which is comma-delimited and clean:
* "Breadworld Bakery Cafe, Wyndham Vale, VIC" -> location "Wyndham Vale, VIC"
* "Seatsaero" -> no location (online merchant)
*
* Only the first comma separates merchant from place; the rest is the place.
*/
export function parseWalletSubtitle(subtitle: string | null | undefined): {
merchant: string | null;
location: string | null;
} {
if (!subtitle || !subtitle.trim()) return { merchant: null, location: null };
const parts = subtitle.split(",").map((p) => p.trim()).filter(Boolean);
if (parts.length <= 1) return { merchant: parts[0] ?? null, location: null };
return { merchant: parts[0], location: parts.slice(1).join(", ") };
}
/**
* Currency marker, if the notification states one.
*
* Only Amex does: it writes "A$5,103.49". Westpac, CommBank and Apple Wallet
* write a bare "$", which is honestly recorded as unknown rather than assumed
* to be AUD -- 14 of 21 captured transaction rows. Domestically that assumption
* would be harmless; abroad it is a 23x error on a THB tap, so it is not made
* here. The statement settles it within weeks, and the reconcile queue holds
* the row until then.
*/
export function currencyFromAmountText(amountText: string | null | undefined): string | null {
if (!amountText) return null;
const t = amountText.trim();
if (/^A\$/i.test(t)) return "AUD";
if (/^(NZ\$)/i.test(t)) return "NZD";
if (/^US\$/i.test(t)) return "USD";
if (/^€/.test(t)) return "EUR";
if (/^£/.test(t)) return "GBP";
if (/^¥/.test(t)) return "JPY";
if (/^฿/.test(t)) return "THB";
if (/^S\$/i.test(t)) return "SGD";
if (/^₹/.test(t)) return "INR";
// CommBank's approval prompts write the ISO code after the figure
// ("672006 VND", "1340.00 INR"); some issuers put it first ("USD 10.00").
const iso = t.match(/^[\d.,\s]+([A-Z]{3})$/) ?? t.match(/^([A-Z]{3})\s*[\d.,]+$/);
if (iso) return iso[1];
// A bare "$" is genuinely ambiguous -- do not guess.
return null;
}
+481
View File
@@ -0,0 +1,481 @@
import { queryRow, withTransaction, type Queryable } from "@/lib/db";
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
import { findWalletAdoptableTransaction } from "@/lib/auto-reconcile";
import {
parseWestpacDescriptor,
parseWalletSubtitle,
currencyFromAmountText,
} from "@/lib/wallet-descriptor";
/**
* Wallet notifications -> transactions.
*
* A transaction is created for EVERY tap (DECISIONS.md ING-11, 2026-08-22),
* because overseas trips are where live spend matters most: the statement lag
* is the same 6-35 days abroad but knowing what you have spent matters far more
* in an unfamiliar currency.
*
* SEQUENCING IS THE WHOLE DESIGN. Wallet is the FIRST signal, not a late one --
* measured on txn 6858, the notification arrived 35h25m before anything created
* a transaction. So this creates the row, the receipt enriches it
* (receipt-ingestion.ts adopts it), and the statement confirms it via the
* reconcile queue, which a wallet row enters unaided: it is statementless with
* payment_method 'card'. Nothing here needs to know about statements.
*
* But that sequence is a TENDENCY, not a guarantee -- these first 21 rows were
* themselves backfilled days after the taps -- so this also adopts a row a
* receipt got to first (findWalletAdoptableTransaction), and a stored-value
* instrument is written as 'credits' so it never enters a queue no statement
* line will ever resolve.
*/
export type WalletEvent = {
/** capture.wallet_events.id -- the idempotency key. */
event_id: number;
/** 'transaction' | 'not_transaction' | 'unsure' from the interpreter. */
kind: string;
direction?: string | null;
amount: number | null;
/** The exact substring the model read the amount from, e.g. "A$5,103.49". */
amount_text?: string | null;
currency?: string | null;
merchant?: string | null;
card_tail?: string | null;
issuer?: string | null;
location?: string | null;
notif_app?: string | null;
/** The notification's title line, e.g. "Visa Gift Card" or "Payment". */
notif_title?: string | null;
notif_body?: string | null;
notif_subtitle?: string | null;
occurred_at?: string | null;
};
export type WalletIngestResult = {
transactionId: number | null;
metadataId: number | null;
skipped?:
| "not_a_transaction" | "already_ingested" | "no_amount" | "merged_pair" | "adopted"
| "account_notice" | "stored_value_drawdown";
flags: string[];
};
/**
* How far apart the two halves of ONE tap can arrive. Measured 2-8 seconds on
* every captured pair; ten minutes leaves room for a phone that was offline,
* and is still nowhere near the +/-1 DAY the matcher used to allow -- which
* folded the UNICEF $50 donation and an unrelated $50 bar charge 17 hours later
* into one row (event 275 into txn 8091, 2026-10-01).
*/
const PAIR_WINDOW_MINUTES = 10;
/** The ledger is dated the way the bank statements are: Melbourne local. */
const LEDGER_TZ = "Australia/Melbourne";
/**
* The calendar date of an instant, in LEDGER_TZ.
*
* Slicing the ISO string took whatever offset n8n serialised -- UTC in
* practice -- so a Melbourne-evening tap landed on the previous day (28 of the
* first 134 rows). Statements, which this row is reconciled against, date in
* Australian local time.
*/
export function ledgerDate(occurredAt: string | null | undefined, now: Date = new Date()): string {
const d = occurredAt ? new Date(occurredAt) : now;
const at = Number.isNaN(d.getTime()) ? now : d;
// en-CA formats as YYYY-MM-DD.
return new Intl.DateTimeFormat("en-CA", {
timeZone: LEDGER_TZ, year: "numeric", month: "2-digit", day: "2-digit",
}).format(at);
}
/**
* A ledger-derived AUD rate for a currency: the median of what the bank
* actually charged divided by the native figure, over statement rows.
*
* Same source order_transaction_bridge uses for its plausibility guard. This is
* only ever a PROVISIONAL value -- transactions.amount is NOT NULL and always
* AUD, and a notification cannot know the AUD figure because the bank sets the
* rate at settlement. The statement overwrites it at reconcile time.
* Returns null when there is not enough history to be worth anything.
*/
async function medianRateToAud(code: string): Promise<number | null> {
const row = await queryRow<{ rate: string | null }>(
`SELECT percentile_cont(0.5) WITHIN GROUP (
ORDER BY amount / NULLIF(foreign_currency_amount, 0)
)::text AS rate
FROM transactions
WHERE foreign_currency_code = $1
AND foreign_currency_amount > 0
AND superseded_by_id IS NULL
-- Statement rows only: a wallet row's amount is itself a provisional
-- guess, and one stored before this filter (txn 7316, 672006 VND at a
-- 1:1 "rate") would otherwise feed the next estimate.
AND statement_id IS NOT NULL
HAVING count(*) >= 5`,
[code]
);
const rate = row?.rate ? Number(row.rate) : null;
return rate && Number.isFinite(rate) && rate > 0 ? rate : null;
}
/**
* The ECB reference rate for the day, via Frankfurter (free, no key; the same
* source the statement processor converts with).
*
* The fallback when the ledger has no history for a currency. Hong Kong was the
* first trip to need it: no HKD statement row existed, so every HKD tap stored
* A$0.00 and the trip read as free until the statement landed weeks later. The
* ledger median is still preferred -- it carries the card's ~3% foreign fee,
* which the ECB mid-rate does not -- so this one is flagged as such.
*
* Any failure is null, never a throw: no rate is a state this lane already
* handles (a flagged 0), and a slow FX API must not lose the purchase.
*/
async function ecbRateToAud(code: string, date: string): Promise<number | null> {
try {
const res = await fetch(
`https://api.frankfurter.dev/v1/${date}?from=${encodeURIComponent(code)}&to=AUD`,
{ signal: AbortSignal.timeout(5000) }
);
if (!res.ok) return null;
const body = (await res.json()) as { rates?: { AUD?: number } };
const rate = body?.rates?.AUD;
return typeof rate === "number" && Number.isFinite(rate) && rate > 0 ? rate : null;
} catch {
return null;
}
}
/**
* The other half of a paired notification, if it has already been ingested.
*
* One tap emits TWO notifications ~2-8s apart: the issuer app (which has the
* card tail) and Apple Wallet (which has the better merchant and the suburb).
* Neither alone is complete, so they must be MERGED, never deduped -- dropping
* either loses a field nothing else supplies. The notifications share no
* identifier, so a pair is recognised by what a pair IS: the same amount,
* minutes apart, from two different apps. Two notifications from one app are
* two charges, however close together.
*/
async function findPairedRow(
db: Queryable,
amount: number,
notifiedAt: string,
notifApp: string | null,
fx: { amount: number; code: string } | null = null
) {
// A foreign row pairs on its native pair: its `amount` is a provisional
// conversion (or a 0 placeholder), so two unrelated foreign taps could share it.
const rows = await db.queryRaw<{ id: number; meta_id: number }>(
`SELECT t.id, em.id AS meta_id
FROM transactions t
JOIN expense_metadata em ON em.transaction_id = t.id
WHERE t.source = 'wallet'
AND t.statement_id IS NULL
AND t.superseded_by_id IS NULL
AND CASE WHEN $4::text IS NULL
THEN t.amount = $1::numeric AND t.foreign_currency_code IS NULL
ELSE t.foreign_currency_code = $4::text
AND t.foreign_currency_amount = $5::numeric END
AND em.notified_at BETWEEN $2::timestamptz - make_interval(mins => $3::int)
AND $2::timestamptz + make_interval(mins => $3::int)
AND em.notif_app IS DISTINCT FROM $6::text
LIMIT 2`,
[amount, notifiedAt, PAIR_WINDOW_MINUTES, fx?.code ?? null, fx?.amount ?? null, notifApp]
);
return rows.length === 1 ? rows[0] : null;
}
/**
* A stored-value instrument: a gift card, a prepaid travel card, a store card.
*
* It matters because payment_method decides queue membership. 'card' means "a
* statement line is coming"; findAutoMatches and the reconcile queue both
* exclude 'credits' and 'cash' for exactly that reason. A $103.30 Woolworths
* tap on a Visa gift card written as 'card' parks in the queue forever, being
* offered as an unmatched row against statements that can never contain it --
* three of the first 21 wallet rows (events 80, 92, 101) were gift-card taps.
*
* The evidence is the notification TITLE, which is where Apple Wallet prints
* the instrument's own name ("Visa Gift Card"); the interpreter reduces that to
* issuer "Visa", which a plain Visa debit card also produces, so the title is
* the only place the distinction survives. Read as a prior, not a contract: a
* miss leaves today's behaviour, which is the queue.
*/
const STORED_VALUE = /gift\s*card|prepaid|travel\s*(?:money|card)|store\s*card/i;
/**
* A transit card reporting a drawdown of its own balance ("HKD 7.40 for Train"
* / "Your new balance is HKD 169.50.").
*
* Unlike a gift card, a transit card is loaded FROM a bank card -- that load is
* on the statement and is the spend. Each fare is that money being used, so
* booking it again counts it twice. The interpreter already called 11 of 13
* Octopus fares not_transaction; the two it did not (events 191, 200) became
* rows, so the outcome depended on the model's mood. This makes it a rule. The
* body wording is the signal because it is what every such card says; the
* issuer name is not (Octopus, Opal, myki, Suica each name themselves).
*/
const BALANCE_DRAWDOWN = /\byour (?:new|current) balance is\b/i;
/**
* Account notices that state an amount without money moving: "Your Smart Access
* 1910 is overdrawn by $21.21. To avoid fees and interest, ..." (event 235)
* became a $21.21 purchase named "To avoid fees and interest,". The model is
* right that the text names an amount; it is not a payment.
*/
const ACCOUNT_NOTICE = /\boverdrawn\b|\bto avoid fees\b|\bavailable balance\b|\blow balance\b/i;
/** How an adopted or merged row records which wallet event folded into it. */
const adoptedFlag = (eventId: number) => `wallet_evt:${eventId}`;
export function isStoredValueInstrument(ev: Pick<WalletEvent, "notif_title" | "issuer" | "notif_app">): boolean {
return [ev.notif_title, ev.issuer, ev.notif_app].some((v) => !!v && STORED_VALUE.test(v));
}
/** Already landed: a row this lane wrote, or one it adopted or merged into. */
async function alreadyIngested(db: Pick<Queryable, "queryRow">, eventId: number) {
return db.queryRow<{ id: number }>(
`SELECT id FROM transactions WHERE source = 'wallet' AND source_ref = $1
UNION ALL
SELECT em.transaction_id AS id FROM expense_metadata em
JOIN transactions t ON t.id = em.transaction_id
WHERE em.flags @> $2::jsonb AND t.superseded_by_id IS NULL
LIMIT 1`,
[`evt:${eventId}`, JSON.stringify([adoptedFlag(eventId)])]
);
}
export async function processWalletIngestion(ev: WalletEvent): Promise<WalletIngestResult> {
const flags: string[] = [];
// An open-world classifier feeding a closed-world writer drops the unknown
// and calls it a refusal (ING-14). `unsure` is a real answer and must not
// silently become a transaction.
if (ev.kind !== "transaction") {
return { transactionId: null, metadataId: null, skipped: "not_a_transaction", flags };
}
if (ev.amount === null || !Number.isFinite(ev.amount) || ev.amount <= 0) {
return { transactionId: null, metadataId: null, skipped: "no_amount", flags };
}
const text = [ev.notif_subtitle, ev.notif_body].filter(Boolean).join("\n");
if (ACCOUNT_NOTICE.test(text)) {
return { transactionId: null, metadataId: null, skipped: "account_notice", flags };
}
if (BALANCE_DRAWDOWN.test(ev.notif_body ?? "")) {
return { transactionId: null, metadataId: null, skipped: "stored_value_drawdown", flags };
}
// Two places this event may already have landed: a row this lane wrote (which
// carries the ref natively), or a row it ADOPTED or MERGED into, whose
// source_ref belongs to whoever created it -- marked in the metadata flags
// rather than by overwriting a field another lane owns. Both must be checked,
// because bumping the interpreter's prompt_version deliberately re-interprets
// and re-posts every event ever captured -- that is the replay mechanism, and
// it must not mint a second row for a purchase that has one. Checked here
// without the lock to skip the FX lookup on a replay, and again under it.
const seen = await alreadyIngested({ queryRow }, ev.event_id);
if (seen) return { transactionId: seen.id, metadataId: null, skipped: "already_ingested", flags };
const sourceRef = `evt:${ev.event_id}`;
const notifiedAt = ev.occurred_at ?? new Date().toISOString();
const date = ledgerDate(ev.occurred_at);
// Deterministic parsing of the positional fields. The model decides what the
// amount and merchant ARE; it does not get to parse a fixed-width block.
// Only Westpac writes that block: run over a CommBank overdraft notice it
// produced the merchant "To avoid fees and interest," (event 235).
const desc = /westpac/i.test(ev.notif_app ?? "")
? parseWestpacDescriptor(ev.notif_body)
: { merchant: null, city: null, country: null };
const sub = parseWalletSubtitle(ev.notif_subtitle);
const merchant = sub.merchant ?? ev.merchant ?? desc.merchant ?? "Card purchase";
// Two sources for a place, and they are not equal. The Apple Wallet subtitle
// gives "Wyndham Vale, VIC"; the issuer's fixed-width descriptor gives
// "WYNDHAM VAL AU" -- a 12-character truncation in caps. The subtitle wins
// outright, which matters for the pair merge below: whichever half arrives
// second must be able to REPLACE a descriptor-derived place, or the result
// depends on arrival order.
const subtitleLocation = sub.location ?? null;
// Precedence: the Wallet subtitle, then the DETERMINISTIC descriptor parse,
// and only then the model's own answer. Getting this order wrong defeats the
// point of parsing positionally at all -- the model returned the mangled
// "SAN FRANCISOUS" and the padded "MELBOURNE AU" for exactly these rows,
// which is what wallet-descriptor.ts exists to correct.
const descriptorLocation = desc.city && desc.country
? `${desc.city} ${desc.country}`
: desc.city;
const location = subtitleLocation ?? descriptorLocation ?? ev.location ?? null;
// Currency: only ever what the notification actually states. A bare "$" is
// recorded as unknown, never assumed AUD -- harmless domestically, a 23x
// error on a THB tap abroad.
const stated = currencyFromAmountText(ev.amount_text)
?? (ev.currency && /^[A-Z]{3}$/.test(ev.currency) ? ev.currency : null);
if (!stated) flags.push("currency_unverified");
if (desc.country && desc.country !== "AU") flags.push(`foreign_country_${desc.country}`);
// Foreign: the native figure is what both sides can agree on exactly, so it
// is what gets stored. `amount` still needs a value (NOT NULL) and takes a
// provisional conversion, flagged, which the statement overwrites. Resolved
// BEFORE the lock: the ECB fallback is a network call, and the lock
// serialises every wallet post behind whatever runs inside it.
let amountAud = ev.amount;
let fxAmount: number | null = null;
let fxCode: string | null = null;
if (stated && stated !== "AUD") {
fxAmount = ev.amount;
fxCode = stated;
const ledgerRate = await medianRateToAud(stated);
const rate = ledgerRate ?? (await ecbRateToAud(stated, date));
if (rate) {
amountAud = Number((ev.amount * rate).toFixed(2));
flags.push("aud_provisional");
if (!ledgerRate) flags.push("rate_ecb");
} else {
// No rate from either source. The native figure must NOT fall through as
// AUD -- 672006 VND once landed as A$672,006. Zero is a flagged
// placeholder; the statement supplies the real AUD at reconcile time,
// which matches on the native pair, not on this column.
amountAud = 0;
flags.push("aud_provisional", "no_rate_reference");
}
}
// A gift card has no statement line coming, so it must not enter the
// reconcile queue (which filters on exactly this column).
const storedValue = isStoredValueInstrument(ev);
const paymentMethod = storedValue ? "credits" : "card";
if (storedValue) flags.push("stored_value_instrument");
// Everything from the idempotency re-check to the INSERT holds one lock.
// n8n's HTTP node posts a poll's items CONCURRENTLY, and both halves of a tap
// usually land in the same 15-minute poll -- so each ran findPairedRow before
// the other had inserted, neither found its partner, and one tap became two
// rows: 11 pairs, A$3,841.40, every pair with an identical created_at. A
// single lock for the whole lane, not one per amount: wallet volume is a few
// dozen a day, and a coarse lock has no key to get wrong.
return withTransaction(async (db) => {
await db.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended('wallet-ingest', 0))::text`);
const existing = await alreadyIngested(db, ev.event_id);
if (existing) {
return { transactionId: existing.id, metadataId: null, skipped: "already_ingested" as const, flags };
}
// Merge the paired notification rather than creating a second row for one
// tap. The issuer half brings the card tail; the Wallet half brings the
// merchant and suburb. COALESCE so whichever arrives second only ever fills
// gaps -- it can never blank a field the first half supplied.
const pair = await findPairedRow(
db, amountAud, notifiedAt, ev.notif_app ?? null,
fxCode && fxAmount !== null ? { amount: fxAmount, code: fxCode } : null
);
if (pair) {
await db.queryRaw(
`UPDATE transactions
SET merchant_name = COALESCE(NULLIF($2,'Card purchase'), merchant_name),
merchant_normalized = COALESCE(NULLIF($2,'Card purchase'), merchant_normalized),
-- A subtitle-derived place overwrites; anything else only fills a
-- gap. Without the $4 branch the merged row depends on which
-- notification happened to arrive first, and the issuer half
-- usually wins by a few seconds -- keeping the worse string.
location = CASE WHEN $4::boolean THEN COALESCE($3, location)
ELSE COALESCE(location, $3) END
WHERE id = $1`,
[pair.id, merchant, location, subtitleLocation !== null]
);
// The merged half records itself, or a replay finds nothing and merges
// it again (or, once the partner is superseded, mints a new row).
await db.queryRaw(
`UPDATE expense_metadata
SET card_last4 = COALESCE(card_last4, $2),
payment_method_detail = COALESCE(payment_method_detail, $3),
merchant_normalized = COALESCE(NULLIF($4,'Card purchase'), merchant_normalized),
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
WHERE id = $1`,
[pair.meta_id, ev.card_tail ?? null, ev.issuer ?? null, merchant, JSON.stringify([adoptedFlag(ev.event_id)])]
);
return { transactionId: pair.id, metadataId: pair.meta_id, skipped: "merged_pair" as const, flags };
}
// The receipt got here first. Fold into its row instead of writing a second
// one for the same purchase (see findWalletAdoptableTransaction). The column
// list IS the ownership boundary, the mirror of receipt-ingestion.ts's: the
// receipt read the register tape, so merchant, description, category and the
// items are its business and are absent here; `location` is wallet's own --
// nothing else writes a suburb -- and payment_method is filled only when the
// other lane left it unknown.
const adopted = await findWalletAdoptableTransaction(DEFAULT_OWNER_ID, amountAud, date, merchant, undefined, db);
if (adopted) {
await db.queryRaw(
`UPDATE transactions
SET location = COALESCE($2, location),
payment_method = COALESCE(payment_method, $3)
WHERE id = $1`,
[adopted.id, location, paymentMethod]
);
let metadataId = adopted.metadata_id;
if (metadataId) {
await db.queryRaw(
`UPDATE expense_metadata
SET card_last4 = COALESCE(card_last4, $2),
payment_method_detail = COALESCE(payment_method_detail, $3),
currency = COALESCE(currency, $4),
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
WHERE id = $1`,
[metadataId, ev.card_tail ?? null, ev.issuer ?? null, stated, JSON.stringify([...flags, adoptedFlag(ev.event_id)])]
);
} else {
// expense_metadata.transaction_id is UNIQUE, so this branch only runs
// when the adopted row genuinely has none -- and it must produce one, or
// the card tail and the flag that makes this idempotent have nowhere to
// live.
const meta = await db.queryRow<{ id: number }>(
`INSERT INTO expense_metadata (
transaction_id, source, source_message_id, merchant_normalized, amount,
transaction_date, card_last4, currency, payment_method,
payment_method_detail, line_items, flags, extraction_model,
notified_at, notif_app
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
RETURNING id`,
[
adopted.id, sourceRef, merchant, ev.amount, date, ev.card_tail ?? null, stated,
paymentMethod, ev.issuer ?? null, JSON.stringify([...flags, adoptedFlag(ev.event_id)]),
notifiedAt, ev.notif_app ?? null,
]
);
metadataId = meta!.id;
}
return { transactionId: adopted.id, metadataId, skipped: "adopted" as const, flags };
}
const txn = await db.queryRow<{ id: number }>(
`INSERT INTO transactions (
transaction_date, description, amount, amount_aud, category, payment_method,
merchant_name, merchant_normalized, transaction_type, owner_id, location,
foreign_currency_amount, foreign_currency_code, source, source_ref, row_index
) VALUES ($1,$2,$3,$3,'other',$10,$4,$4,'debit',$5,$6,$7,$8,'wallet',$9,0)
RETURNING id`,
[date, merchant, amountAud, merchant, DEFAULT_OWNER_ID, location, fxAmount, fxCode, sourceRef, paymentMethod]
);
const meta = await db.queryRow<{ id: number }>(
`INSERT INTO expense_metadata (
transaction_id, source, source_message_id, merchant_normalized, amount,
transaction_date, card_last4, currency, payment_method,
payment_method_detail, line_items, flags, extraction_model,
notified_at, notif_app
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
RETURNING id`,
[
txn!.id, sourceRef, merchant, ev.amount, date,
ev.card_tail ?? null, stated, paymentMethod, ev.issuer ?? null, JSON.stringify(flags),
notifiedAt, ev.notif_app ?? null,
]
);
return { transactionId: txn!.id, metadataId: meta!.id, flags };
}, { maxWait: 10_000, timeout: 30_000 });
}