1 Commits
Author SHA1 Message Date
siddharthd 057595f4ac wip(intake): router, enrich endpoint, Slack handlers, migration 0034 — UNTESTED checkpoint
ci / lint-test (push) Successful in 44s
Agent was terminated by the session usage limit before running lint/tests or
writing its report. Files are complete drafts against plan §11 (C1, C2, C5):
src/app/api/intake/, src/app/api/transactions/[id]/line-items/, src/lib/intake-*.ts,
src/lib/line-items.ts, prisma/migrations/0034_intake_batches, fixtures + tests.
Next: npm ci && npx prisma generate && npm run lint && npm test && npm run test:integration.
2026-08-21 12:03:48 +10:00
56 changed files with 246 additions and 6101 deletions
+1 -319
View File
@@ -65,7 +65,7 @@ docker logs finance -f
npm test # unit — pure logic, no database npm test # unit — pure logic, no database
npm run test:integration # database-backed; resolves the DSN itself (see below) npm run test:integration # database-backed; resolves the DSN itself (see below)
npm run test:all # both npm run test:all # both
npm run test:setup # once, and after any migration: drops and rebuilds personal_test npm run test:setup # once, and after any migration: creates/refreshes personal_test
npm run lint # blocking in CI since #79 — 0 errors, warnings allowed npm run lint # blocking in CI since #79 — 0 errors, warnings allowed
npx tsc --noEmit # blocking in CI; nothing else compiles the test files npx tsc --noEmit # blocking in CI; nothing else compiles the test files
``` ```
@@ -86,13 +86,6 @@ had pointed the suite at **postgres-pantry** — whose database it would have
TRUNCATEd had the credentials happened to match. A loopback address cannot be TRUNCATEd had the credentials happened to match. A loopback address cannot be
reassigned to another container, which is why the published port wins. reassigned to another container, which is why the published port wins.
**`test:setup` really rebuilds now (2026-10-03).** It used to replay the
schema dump over the existing database, where every `CREATE TABLE` failed with
"already exists" and the old shape survived — so no migration ever reached
`personal_test`, and new-column code failed every test with a misleading
`column does not exist`. It now `DROP DATABASE … WITH (FORCE)`s first, refusing
any name not ending `_test`, and loads with `ON_ERROR_STOP`.
**Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to **Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to
truncate anything whose `current_database()` is not `personal_test`, and the truncate anything whose `current_database()` is not `personal_test`, and the
suite runs `fileParallelism: false` because every file shares that one database suite runs `fileParallelism: false` because every file shares that one database
@@ -183,215 +176,6 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
- `statements` — one row per billing period per bank account - `statements` — one row per billing period per bank account
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent - `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
### The change log (migration 0037)
`transaction_audit` is an **append-only** row-level record of every change to
`transactions`, `transaction_splits`, `transaction_tags`, `transaction_overrides`
and `expense_metadata`. Never `UPDATE` or `DELETE` from it. There is deliberately
**no foreign key** to `transactions` — the log has to outlive the row it describes.
```sql
-- everything that ever happened to one transaction
SELECT changed_at, table_name, op, old_data, new_data, app_name
FROM transaction_audit WHERE transaction_id = 6961 ORDER BY changed_at;
-- one logical edit, all the rows it touched (a reconcile moves 4 children at once)
SELECT * FROM transaction_audit WHERE txid = <txid> ORDER BY id;
```
**Why it exists.** On 2026-08-23 the n8n CC Statement Processor stamped
`statement_id` onto seven statementless wallet/pantry rows without setting
`reconciled_with_id`, double-counting $245.90 and hiding those rows from the
reconcile queue (`WHERE statement_id IS NULL`). Answering *"was anything lost?"*
took restoring the 02:30 `pg_dump` into a scratch database and diffing it column
by column, because nothing recorded what changed.
**Why it is a trigger and not application code.** The ledger has at least five
independent writers — this app (Prisma), n8n (raw SQL in a Postgres node),
pantry-app, the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`.
That bug was an n8n raw-SQL `UPDATE`; an app-level audit would not have seen it.
**Why not Postgres' own logs.** Checked live: `wal_level=replica` (WAL cannot be
decoded into row changes), `wal_keep_size=0` and `archive_mode=off` (recycled
almost immediately), `log_statement=none`, `track_commit_timestamp=off`, no
`pgaudit`. Postgres records nothing usable and has no built-in audit or temporal
tables.
Three details worth knowing before relying on it:
- **`old_data`/`new_data` hold only the columns that actually changed** on an
`UPDATE`. A whole-row copy would bury a one-column edit, and
`expense_metadata.line_items` alone can be kilobytes.
- **A no-op `UPDATE` writes nothing** (the upsert paths issue plenty).
- **Attribution is partial and that is understood.** Every client connects as the
same `personal` role, so `db_user` never discriminates. `app_name` is only as
good as what each client sets; `client_addr` separates containers and is NULL
for `docker exec` over the unix socket — itself a useful signal. `txid` is the
reliable grouping key.
`TRUNCATE` does not fire row-level triggers, so the integration suite's table
wipes produce no audit rows.
### Resolving a duplicate by hand (`POST /api/transactions/<id>/merge`)
`<id>` is the row that gets **hidden**; `other_id` is the one that **survives**.
| mode | requires | sets | for |
|---|---|---|---|
| `reconcile` | loser statementless, survivor a statement row | `reconciled_with_id` | the everyday case |
| `supersede` | nothing | `superseded_by_id` | two rows on the same footing, e.g. a statement imported twice |
**There is no delete, deliberately.** Every child of `transactions` is
`ON DELETE CASCADE`, so removing a row destroys the splits, tags and metadata
that make it legible. Both modes keep the row and its children and merely stop
it counting. Do not add a delete endpoint.
Both modes **move curation onto the survivor first**, through the single
`moveCurationWithin()` in `lib/reconcile.ts` — because both hide the source row
from every figure, and a split stranded on a hidden row stops counting in
silence. That is also why the two callers share one implementation: the file
header warns that a second copy would eventually move a different set.
Refusals (all deliberate, all tested in `merge-supersede.test.ts`): a row already
reconciled or superseded on **either** side → 409 (re-pointing it strands the
curation moved on the first pass); superseding under a row that is itself
superseded → 409; a row that is not the caller's → 404. Ownership is checked with
`COALESCE(t.owner_id, s.owner_id)` — a statement row carries `owner_id` on the
statement, so checking `t.owner_id` alone rejects every statement row as "not
found".
**Why it had to exist.** `superseded_by_id` was read everywhere (analytics and
auto-reconcile both exclude it) but written **nowhere** in application code — set
exactly once, by migration 0023. The reconcile queue was the only merge path, and
it only offers candidates inside an amount band.
**The band is ±5%, not ±1% (2026-08-23).** A foreign card purchase is charged the
notified amount **plus the card's foreign transaction fee** — 3% on the Westpac
Altitude — so a wallet row and its statement line can never match exactly.
`SP PEAKDESIGN` was notified at **$42.44** and charged **$43.71** (×1.03), which
put the one row that *was* its twin outside the old band: the picker could not
offer the only correct answer. Widening is safe because the list is capped at 5
by the `rn` window and ordered by amount closeness, so a looser band changes
*which* five appear, never how many, and a person confirms each. Measured on the
live queue at the time of the change: candidate counts went **0 → 0** on all 12
rows, i.e. no new noise. **The auto-matcher stays exact** — see `findAutoMatches()`.
### A charge that never posted (`POST /api/transactions/<id>/void`, migration 0040)
The third way a row stops counting, for a case with **no survivor**: a card
notification is sent for an *authorisation*, and when the merchant releases the
hold no statement line ever arrives. Txn 7951 — a $1,040.41 Airbnb hold on Amex,
booking cancelled the next day — sat in the reconcile queue with nothing to
match and counted as spend everywhere. `reconcile` needs a statement twin,
`supersede` needs a survivor (and `transactions_no_self_supersede` blocks
pointing a row at itself), so it gets its own columns: `voided_at` +
`void_reason` (reason mandatory, by CHECK). Body `{reason}`; `DELETE` undoes it.
UI: **Never posted** on every row of `/reconcile`.
- **In `EXCLUDE_RECONCILED_SOURCE`**, so every "count each purchase once" query
drops it without being edited; the queue, `findAutoMatches`, both adoption
helpers, annual charges and the manual reconcile POST filter it explicitly.
- **Statementless only** — `transactions_void_statementless` enforces it in the
database. A statement line is the bank's record of a posted charge; a wrong
one is a supersede.
- **Refused** on a reconciled row (it posted), a superseded row, and a row with a
*settled* split (someone already paid their share — voiding would quietly turn
that into a credit). `merge` refuses a voided row on either side.
- **The wallet lane deliberately still SEES voided rows.** `alreadyIngested()` and
the issuer/Wallet pairing in `wallet-ingestion.ts` do not filter `voided_at`:
the voided row is the sink that stops a late twin notification, or the next
`prompt_version` re-post, from booking the hold again. Same for the intake
router's idempotency lookup. Do not "fix" those.
- **Outside this repo, nothing filters it**: Hermes' `agent_ro` views
(`cashflow_monthly` etc.) and ingestion-engine's `instalment_matcher` already
ignore `superseded_by_id`/`reconciled_with_id` too, so a voided row is no worse
there than any other hidden row.
### Upcoming annual charges (`GET /api/annual-charges`, migration 0038)
Card annual fees, loan package fees and annual subscriptions, derived from
**payments alone** — independent of the email lane, because a bank announces an
annual fee on the statement and nowhere else.
**`transaction_type = 'fee'` is the universe, not a description match.** The
extractor already classifies fees structurally (131 rows), and *every* card and
loan annual fee on this ledger is typed that way. The first version of this
filtered `transaction_type = 'debit'` and returned **none** of them. The fees are
not debits.
**Two signals, kept apart because they are not equally trustworthy:**
| confidence | basis |
|---|---|
| `confirmed` | typed a fee **and** repeated on an annual cadence |
| `likely` | typed a fee, seen once, and *named* annual/card/package — the kind is certain, the cadence inferred |
| `possible` | no fee wording; a merchant whose every gap is annual **and** whose amount is stable |
**A naive "two charges ~365 days apart" is useless** — measured on this ledger it
ranks a bakery first with **202** such pairs, then a donation, Patreon, Coles.
Frequency is the confound. Even "every gap is annual" admits a Target visited
twice ($30, then $198). The `possible` tier therefore also requires amount
stability (≤20%, which absorbs VicRoads' ~3%/yr CPI drift) and ≥2 observations.
It still carries real false positives (H&M, Superdry) — that is what the label is
for, and why it must never render like a bank fact.
**`expected_not_seen` is the more interesting status.** The estimate passed and
nothing landed, which means the account stopped being imported, or closed, or the
charge arrived under a different label. `account_days_stale` is the
discriminator: Westpac at 0 days stale is trustworthy; ANZ at 392 is simply
missing data. Never report these as upcoming.
**Account continuity (`account_links`, migration 0038).** One card can appear
under two identities: Citi Prestige `5258…0253` (to 2025-11-13) became NAB MyCard
`1015…6227` (from 2025-11-14), same 14th-to-13th cycle, no gap. Without the link
the $700 fee reads as two dead one-offs instead of one due again ~Nov 2026. The
table is **hand-asserted with evidence** (`source_documents` ids), not inferred —
a contiguous-cycle heuristic would join two genuinely different accounts as
readily as one renamed one. The resolver walks it forward recursively;
`uq_account_links_from` forbids a fork and `account_links_no_self` a self-link, so
it terminates. **Display the resolved current bank**, not the one that printed the
last fee — naming the retired brand sends you to the wrong bank.
Known limitation: the merchant signal fragments on label drift. VicRoads rego was
paid 2024/2025 as `VICROADS` and 2026 as `VICROADS ONLINE PAYM`, so the series
splits and the 2025 estimate shows as `expected_not_seen`. Not fuzzy-matched on
purpose — see the rejected pg_trgm merchant grouping.
### The wallet lane (`/api/wallet/ingest`, `lib/wallet-ingestion.ts`)
iOS notifications → `capture.wallet_events` → n8n **Wallet Interpret**
(`CAp1s5cuzJk2mDaI`, local gemma, every 15 min) → this route. A row per tap,
statementless, `source='wallet'`, `source_ref='evt:<id>'`; the statement later
confirms it through the reconcile queue. Fixed 2026-10-03 after a month abroad
(ticket #266); the rules below are each the fix for something that broke.
- **Storage convention differs from the order lane.** `amount` = `amount_aud` =
a *provisional AUD* figure; the native figure is `foreign_currency_amount`.
`NATIVE_CURRENCY` reads `amount_aud IS NOT NULL` as "amount is AUD" — without
that an INR 524 tap displayed as "INR 9.42".
- **Provisional rate:** ledger median over *statement* rows (≥5, carries the
card's FX fee) → else ECB via Frankfurter (`rate_ecb`) → else `0` with
`no_rate_reference`. Never the native figure as AUD (672006 VND once became
A$672,006).
- **One tap = two notifications** (issuer app + Wallet, 2–8 s apart), merged.
A pair must be **≤10 minutes apart AND from different apps**
(`expense_metadata.notified_at` / `notif_app`, migration 0039). The old
amount-within-±1-day rule merged the UNICEF $50 into an unrelated $50 charge.
- **The whole check→pair→adopt→insert runs under one advisory lock**
(`hashtextextended('wallet-ingest',0)`), on the transaction's own connection.
n8n posts a poll's items concurrently; without it 11 taps became 22 rows.
Prisma cannot deserialise `void` — the lock SELECT needs `::text`.
- **A merged half records `wallet_evt:<id>` in the survivor's flags**, so a
replay is idempotent. Reconciled wallet rows carry **no** `expense_metadata`
(reconcile moves it to the statement twin) — the flag check follows it there.
- **Skipped, not booked:** account notices (`overdrawn`, `to avoid fees`, …) and
transit-card drawdowns (`Your new/current balance is` — the Octopus *load* is
the spend). The fixed-width descriptor parser runs on Westpac bodies only.
- **Dated in Australia/Melbourne**, like the statements (UTC slicing filed 28 of
134 rows a day early).
- **Replay a few events by relabelling their interpretations**
(`UPDATE capture.wallet_interpretations SET prompt_version='v1-<why>' WHERE …`),
not by bumping `prompt_version`, which re-posts every event ever captured.
### Cash and reconciliation ### Cash and reconciliation
`payment_method` in (`cash`, `credits`) excludes a transaction from `payment_method` in (`cash`, `credits`) excludes a transaction from
@@ -838,108 +622,6 @@ and one tap must not flatten it. It says which case it refused on.
Slack reaches this route through an n8n webhook, not directly — see the Slack reaches this route through an n8n webhook, not directly — see the
smarthome repo's CLAUDE.md and the `slack-interactive-via-n8n` memory. smarthome repo's CLAUDE.md and the `slack-interactive-via-n8n` memory.
### Intake router — one upload, every applicable step (2026-08-21)
`POST /api/intake` (token `INTAKE_TOKEN`, multipart: `source`, `shared_by`,
`batch_id`, 1–6 `file`s) is the finance half of the share-sheet intake; design
and the cross-repo contract are in smarthome `plans/intake-router-slice-plan.md`
§11 (C1, C2, C5 live here; C3/C4 are pantry-app; C6 is n8n + compose).
- **Order of operations is the plan's and is load-bearing:** persist
`intake_batches` row → classify each file → ONE LiteLLM call for the whole
batch (`cloud-budget`, `temperature 0`, strict `json_schema`; PDFs go in as
`unpdf` text, never pixels) → header `(merchant, date, total)` → merchant
kind → duplicate check → reconciliation → match → act. `src/lib/intake-rules.ts`
holds the pure decisions, `intake-router.ts` the orchestration.
- **Match is `(platform, merchant, date, total)`, exact, never widened.** A
miss or an ambiguity returns candidates (same merchant ±3 days) with
`intake_use_candidate` / `intake_create` buttons; a transaction is created
only on the explicit tap, ING-10-style (`statement_id NULL`, `owner_id` from
`shared_by` → `participants.email`, else `DEFAULT_OWNER_ID`;
`source='intake'`, `source_ref=batch_id` so a second press is a no-op —
written as `ON CONFLICT … DO NOTHING`, because a SELECT-then-INSERT loses a
double tap). A non-AUD receipt is booked the way `order-ingestion.ts` books
one: `amount` native, `amount_aud` **NULL**, `foreign_currency_*` set — no FX
rate exists at intake and inventing one is worse than admitting there is none.
- **A NULL platform means "no platform", not "any platform".** An in-store tap
(`coles`, `woolworths`, `other`) matches only rows whose `expense_metadata`
carries no platform of its own. The wildcard reading let a till receipt match
the Uber Eats order for the same shop on the same day and overwrite its items.
The **candidates** query is the opposite and deliberately so: it has no
platform filter at all, because the card is a human choice shown precisely
because the exact key missed — the platform is one of the things that may be
wrong, so the list must widen there, not narrow.
- **The Create tap re-runs the match before it inserts**, and checks for a
resolved sibling batch. The card is minutes old by the time it is pressed and
the email/order bridge books orders on its own schedule; `findDuplicateBatch`
ignores *unresolved* batches by design, so two pending cards for one order
would otherwise each book it.
- **One `expense_metadata` row, agreed on by every query that resolves one.** A
transaction can own a row *and* be the `matched_transaction_id` of another;
`matchTransaction` returns the row it resolved through and `replaceLineItems`
writes to exactly that one. Every reader orders by
`(transaction_id = $1) DESC **NULLS LAST**, id` — the match, the write, the
order-detail GET, `getTransactions`' provenance LATERAL, `merchantForTransaction`
and the Slack details modal. The `NULLS LAST` is load-bearing: Postgres orders
`DESC` as `NULLS FIRST` and the row that merely *points* at the transaction has
a NULL `transaction_id`, so the clause meant to prefer the owned row preferred
the other one. Measured 2026-08-21: 667 rows, 147 pointing-only, **zero**
transactions resolving to more than one row — the ordering changes nothing
today and exists so the readers still agree when multiplicity appears.
- **The create is guarded by a transaction-scoped advisory lock on the
PURCHASE**, not the batch. `ON CONFLICT (source, source_ref)` handles a double
tap on one card; two *different* batches for one purchase have different
`source_ref`s and both clear their match and duplicate checks before either
inserts. The lock has to live inside `withTransaction` (`src/lib/db.ts`):
`queryRaw` is `$queryRawUnsafe` on a **pooled** client, so every call is its
own implicit transaction and `pg_advisory_xact_lock` taken in a statement of
its own is released before the next statement runs. The loser reports
`duplicate` pointing at the winner's transaction — the same answer the
sequential path gives.
- **`replaceLineItems` snapshots the undo baseline BEFORE it writes, on both
paths**, and treats an UPDATE that touched no row as a failure rather than a
success. The metadata INSERT is `ON CONFLICT (transaction_id) DO UPDATE` and
carries the batch's **currency** — a hard-coded `'AUD'` made `OrderDetails`
print `$` on a USD order, because it prefers `expense_metadata.currency`.
- **`intake_batches.previous_line_items` is the undo baseline, written once.**
Returning whatever the write just read makes a pantry retry get its own write
back as the "previous" state, so undo restores the thing it undid. Recorded
under `COALESCE(previous_line_items, …)` and read back.
- **Grocery never writes finance here.** The ORIGINAL files go to pantry
`POST /api/capture` with `intake.finance_target_transaction_id` +
`finance_action: enrich|create`; pantry's confirm calls
`PUT /api/transactions/[id]/line-items` back (C4), which is why that route
accepts `RECEIPT_INGEST_TOKEN` too. Files are kept under `INTAKE_FILE_DIR`
keyed by sha256 so a later "Groceries" tap can still forward them.
- **Reconciliation is against the printed item subtotal, not the total.** A
delivery order's total is items + fees − promotions ($118.94 of Coles lines
became $85.74), so checking lines against the total would hold every
delivery order. Tolerance $1 or 2%; a failure is `held`, nothing written.
- **Nothing that cannot be a receipt reaches the model, and the prompt names no
real shop.** On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so
a batch arrived as the five characters `Image` — and the model returned a
whole receipt: `Burgertory (Braybrook)`, two "The Pride" lines, $41.15. None
of it existed; "Burgertory" was in `ORDER_ITEM_PROMPT` as the example of how
to print a branch in parentheses. Only the absent date kept it out of the
matcher, and the invented header and items were still stored on the batch as
`needs_header`. Three layers now, in order: `textCouldBeReceipt()` rejects a
TEXT part with fewer than two words, no digit, or under 40 characters with no
money figure (`TOTAL $17.00` passes — length alone is *not* the test)
**before** any model call; the prompt's merchant examples are
`PROMPT_PLACEHOLDER_MERCHANTS` format placeholders and it is told to return
nulls for a non-receipt; and `confabulationReason()` catches what the
pre-check cannot see — a PDF that renders to junk — plus any answer echoing a
placeholder. All three file `unsupported`, never `needs_header`: a
confabulated header is not a partial one. Same shape as the ingestion
engine's 2026-08-05 empty-body defect — **a stage that can be handed nothing
must say so, because the model downstream always answers, and it answers out
of its prompt.**
- **Names stay verbatim.** Burgertory calls its burgers "Michael" and "Joel";
the prompt forbids canonicalising and the stored shape is the order lane's
(`qty`, `description`, `amount`, `options`) so `order-details.tsx` renders it.
- Statuses beyond the contract list: `ask` (the "Groceries or a meal?" holding
state), `held`, `error` (502 — model or pantry down; the batch row says so).
### The shared loan ### The shared loan
The loan is a **separate ledger**, not a shared expense and not a settlement The loan is a **separate ledger**, not a shared expense and not a settlement
-14
View File
@@ -282,18 +282,6 @@ All routes require authentication via `X-Forwarded-User` header (set by Traefik)
| GET / POST | `/api/tags` | List or create tags | | GET / POST | `/api/tags` | List or create tags |
| PATCH / DELETE | `/api/tags/[id]` | Update or delete a tag | | PATCH / DELETE | `/api/tags/[id]` | Update or delete a tag |
**Machine routes** — shared-secret `x-ingest-token` header instead of `X-Forwarded-User`; each fails closed when its token is unset:
| Method | Route | Token | Description |
|--------|-------|-------|-------------|
| POST / PATCH | `/api/orders/ingest` | `ORDER_INGEST_TOKEN` | n8n order receipts → transactions (see smarthome CLAUDE.md "Order Processor") |
| POST | `/api/receipts/ingest` | `RECEIPT_INGEST_TOKEN` | pantry-app confirmed grocery receipts → one transaction per tender leg (ING-10) |
| POST | `/api/intake` | `INTAKE_TOKEN` | **Intake router** (2026-08-21): multipart `source` (`uber_eats\|doordash\|coles\|woolworths\|other`), `shared_by`, `batch_id` (uuid), 1–6 `file`s (image / PDF / text). Persists `intake_batches`, extracts once via LiteLLM `cloud-budget` (PDF → text via `unpdf`), matches `(platform, merchant, date, total)` exactly, then enriches the matched order's `line_items` (food), forwards the original files to pantry `POST /api/capture` (grocery), or replies with candidates / "Groceries or a meal?" / needs-header / held / duplicate. Returns `{kind, status, transaction_id?, capture_id?, candidates?, slack_blocks, text}` |
| PUT | `/api/transactions/[id]/line-items` | `INTAKE_TOKEN` or `RECEIPT_INGEST_TOKEN` | Idempotent replace of `expense_metadata.line_items` for `transaction_id = :id OR matched_transaction_id = :id`; body `{line_items, source: 'intake'\|'pantry', batch_id?, platform?, replace: true}`; creates the metadata row if absent, never a transaction (404). Returns `{previous_line_items, line_items}` — pantry calls it on every confirm and with `previous_line_items` on undo |
| POST | `/api/slack/interactive` | `ORDER_INGEST_TOKEN` + Slack signature | Order-nudge buttons and the intake taps (`intake_use_candidate`, `intake_create`, `intake_grocery`, `intake_meal`) |
Env for the intake router: `INTAKE_TOKEN`, `LITELLM_BASE_URL` + `LITELLM_KEY` (OpenAI-compatible, alias `cloud-budget`), `PANTRY_BASE_URL` (+ the existing `RECEIPT_INGEST_TOKEN` as pantry's machine auth), optional `INTAKE_FILE_DIR` (original files by sha256, default `/app/data/intake` — a later "Groceries" tap re-sends them) and `FINANCE_PUBLIC_URL` (the card's View link, default `https://finance.bosecamp.com`). Design: smarthome `plans/intake-router-slice-plan.md` §11.
--- ---
## Ingestion Pipeline ## Ingestion Pipeline
@@ -359,8 +347,6 @@ docker exec postgres-personal psql -U personal -d personal \
| `0009_split_payments` | `split_payments` | | `0009_split_payments` | `split_payments` |
| `0010_csv_import_reconcile` | `reconciled_with_id`, CSV import support | | `0010_csv_import_reconcile` | `reconciled_with_id`, CSV import support |
| `0011_trips` | `trips`, `trip_id` on `transaction_overrides` (catch-up — was applied directly) | | `0011_trips` | `trips`, `trip_id` on `transaction_overrides` (catch-up — was applied directly) |
| … | |
| `0034_intake_batches` | `intake_batches` (one row per `/api/intake` upload, written before extraction) and `intake_merchant_kinds` (the remembered "Groceries or a meal?" answer per merchant) |
> `paperless_doc_id` on statements and the `uq_statements_paperless_doc_id` index were added directly (not tracked in a migration file). > `paperless_doc_id` on statements and the `uq_statements_paperless_doc_id` index were added directly (not tracked in a migration file).
> `owner_id` on transactions and `statement_id` made nullable were applied directly (March 2026) to support manual transaction entry without a fake statement. > `owner_id` on transactions and `statement_id` made nullable were applied directly (March 2026) to support manual transaction entry without a fake statement.
@@ -32,20 +32,10 @@ CREATE TABLE IF NOT EXISTS intake_batches (
status text NOT NULL, status text NOT NULL,
transaction_id integer, transaction_id integer,
capture_id integer, capture_id integer,
-- The undo baseline: what `expense_metadata.line_items` held before this
-- batch first replaced it. Recorded here rather than returned from the write,
-- because pantry re-calls the enrich endpoint on every confirm and after a
-- timeout — a caller that hands back whatever it just read gives a retry its
-- OWN write as the "previous" state, and undo then restores the thing it was
-- meant to reverse. Written under COALESCE, so only the first snapshot sticks.
previous_line_items jsonb,
created_at timestamptz NOT NULL DEFAULT now(), created_at timestamptz NOT NULL DEFAULT now(),
resolved_at timestamptz resolved_at timestamptz
); );
-- For a database that already ran an earlier draft of this file.
ALTER TABLE intake_batches ADD COLUMN IF NOT EXISTS previous_line_items jsonb;
COMMENT ON TABLE intake_batches IS COMMENT ON TABLE intake_batches IS
'One row per upload batch through POST /api/intake. Persisted before ' 'One row per upload batch through POST /api/intake. Persisted before '
'extraction so Slack taps can act on it later and re-shares are detected. ' 'extraction so Slack taps can act on it later and re-shares are detected. '
@@ -1,73 +0,0 @@
-- transaction_search: resolve expense_metadata by BOTH links, without fanning out.
--
-- THE DEFECT (measured 2026-08-22): the view joined only on
-- `em.transaction_id = t.id`, so it silently omitted every card-settled order,
-- which carries only `matched_transaction_id`. 148 transactions are affected --
-- Glovo orders in Milan, Uber Eats, all `source='order-bridge'` -- and for each
-- of them the view reports NULL payment_method / order_reference / line_items
-- while the data sits in the table.
--
-- That is not cosmetic. This view is the agent read surface: Hermes'
-- postgres-personal MCP selects payment_method and order_reference from it
-- (personal-agent-gateway/mcp-servers/postgres-personal/queries.py), and
-- home-dashboard reads it via dash_ro. CLAUDE.md states the rule the view broke:
-- "That lookup must match transaction_id OR matched_transaction_id: a
-- card-settled order has only the latter."
--
-- NOT a fan-out fix, contrary to a first reading. The original join CANNOT fan
-- out: expense_metadata_transaction_id_key is UNIQUE on transaction_id, so it
-- matched at most one row by construction. Sums taken over this view were never
-- inflated.
--
-- But broadening the join is exactly what INTRODUCES that risk, which is why
-- LATERAL is used rather than a second OR'd condition. transaction_id and
-- matched_transaction_id are enforced by two SEPARATE unique indexes
-- (expense_metadata_transaction_id_key, uq_expense_matched_txn), so one
-- transaction can legally be pointed at by two DIFFERENT rows -- one via each
-- link. Zero such transactions exist today; the LATERAL means that stays a
-- non-event rather than a silent duplicate in the agent's spending totals.
--
-- Tie-break matches every other reader (getTransactions, replaceLineItems,
-- matchTransaction, /api/transactions/[id]/order), so the row this view resolves
-- is the row the UI shows.
--
-- Verified before applying: row count unchanged at 4760 (= count(transactions)),
-- sum(amount) over debits unchanged at 1514412.89 -- amount comes from
-- transactions, never from the metadata row -- and the only difference is the
-- 148 rows that gain the metadata they always should have had.
--
-- CREATE OR REPLACE VIEW DROPS GRANTS. dash_ro (home-dashboard) and agent_ro
-- (Hermes) both read this view and losing either breaks a surface with no error
-- at replace time, so both are re-granted below.
CREATE OR REPLACE VIEW transaction_search AS
SELECT t.id,
t.transaction_date,
t.description,
COALESCE(t.merchant_normalized, t.merchant_name) AS merchant,
t.category,
t.transaction_type,
COALESCE(t.amount_aud, t.amount) AS amount,
t.foreign_currency_amount,
t.foreign_currency_code,
t.location,
s.bank_name,
s.card_name,
s.account_number,
s.statement_type,
s.billing_end_date,
em.payment_method,
em.order_reference,
em.line_items
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN LATERAL (
SELECT em2.payment_method, em2.order_reference, em2.line_items
FROM expense_metadata em2
WHERE em2.transaction_id = t.id OR em2.matched_transaction_id = t.id
ORDER BY (em2.transaction_id = t.id) DESC NULLS LAST, em2.id
LIMIT 1
) em ON true;
GRANT SELECT ON transaction_search TO dash_ro;
GRANT SELECT ON transaction_search TO agent_ro;
@@ -1,73 +0,0 @@
-- Normalise foreign_currency_code to ISO-4217, and clear two sentinel values.
--
-- WHY THIS MATTERS: exact-code equality is the whole basis of FX reconciliation.
-- ingestion-engine's order_transaction_bridge matches foreign orders with
-- AND t.foreign_currency_code = %(ccy)s
-- AND t.foreign_currency_amount = %(total)s
-- so a row labelled 'BAHT' can never match a THB order, and a row labelled
-- 'null' can never match anything. 226 of 1,632 coded rows (14%) were
-- unmatchable for this reason alone, which silently degrades a shipped job.
--
-- THE SENTINELS. 187 rows carry the literal 4-character STRING 'null' rather
-- than SQL NULL, and 42 of those also carry foreign_currency_amount = -1.00
-- exactly. Neither is data: they are "no value" markers that survived a truthy
-- guard. Root cause is in the CC statement processors' Parse node, which does
-- foreign_currency_code: t.foreign_currency_code || null
-- and a non-empty string is truthy, so a model that answers with the STRING
-- "null" (rather than JSON null) passes straight through. Same for -1. Fixed
-- workflow-side in the same change; this migration cleans what already landed.
-- Not purely historical -- 155 rows in 2026-03, but also 8 in May, 3 in July
-- and 21 in August 2026.
--
-- MAPPINGS WERE VERIFIED AGAINST IMPLIED RATE, NOT TRUSTED FROM THE LABEL,
-- because a row whose rate contradicts its claimed currency is a broken ledger
-- row rather than a mislabelled one (the trap order_transaction_bridge's rate
-- guard exists for: 2 of its first 73 matches were broken rows, not bad
-- matches). Median amount/foreign_currency_amount per raw code vs the ISO
-- median over statement rows (n>=5):
-- BAHT 0.0433 vs THB 0.0470 INDIAN RUPEE 0.0183 vs INR 0.0180
-- U.A.E. DIRHAM 0.4287 vs AED 0.4184 US DOLLAR 1.4054 vs USD 1.5513
-- NZ DOLLAR 0.8243 vs NZD 0.9311 EURO 1.6342 vs EUR 1.6964
-- Every label agrees with its rate. foreign_currency_amount is untouched by the
-- relabels -- only the code was ever wrong, so this is a relabel, never a
-- recompute.
--
-- EXPECTED ROW COUNTS (verified pre-flight, includes superseded rows):
-- code 'null' -> NULL 187 BAHT -> THB 13
-- amount -1.00 -> NULL 42 U.A.E. DIRHAM -> AED 10
-- NZ DOLLAR -> NZD 6 US DOLLAR -> USD 4
-- INDIAN RUPEE -> INR 4 EURO -> EUR 2
-- Pre-flight also confirmed: no -1.00 amount exists outside the 'null'-code
-- rows, and no other non-ISO code exists.
BEGIN;
-- Sentinels first: the -1.00 amounts live only on 'null'-code rows, so clear
-- them before that code is nulled and the set becomes unfindable.
UPDATE transactions SET foreign_currency_amount = NULL
WHERE foreign_currency_amount = -1.00 AND foreign_currency_code = 'null';
UPDATE transactions SET foreign_currency_code = NULL
WHERE foreign_currency_code = 'null';
-- Word-form relabels.
UPDATE transactions SET foreign_currency_code = 'THB' WHERE foreign_currency_code = 'BAHT';
UPDATE transactions SET foreign_currency_code = 'AED' WHERE foreign_currency_code = 'U.A.E. DIRHAM';
UPDATE transactions SET foreign_currency_code = 'NZD' WHERE foreign_currency_code = 'NZ DOLLAR';
UPDATE transactions SET foreign_currency_code = 'USD' WHERE foreign_currency_code = 'US DOLLAR';
UPDATE transactions SET foreign_currency_code = 'INR' WHERE foreign_currency_code = 'INDIAN RUPEE';
UPDATE transactions SET foreign_currency_code = 'EUR' WHERE foreign_currency_code = 'EURO';
-- Postcondition: every remaining code is ISO-4217 shaped. Fails the whole
-- transaction rather than leaving a partial normalisation behind.
DO $$
DECLARE bad int;
BEGIN
SELECT count(*) INTO bad FROM transactions
WHERE foreign_currency_code IS NOT NULL AND foreign_currency_code !~ '^[A-Z]{3}$';
IF bad <> 0 THEN
RAISE EXCEPTION 'normalisation incomplete: % non-ISO codes remain', bad;
END IF;
END $$;
COMMIT;
@@ -1,152 +0,0 @@
-- Append-only change log for the ledger and its curation children.
--
-- WHY THIS EXISTS: on 2026-08-23 the CC Statement Processor's `Run Reconciliation`
-- node stamped `statement_id` onto seven statementless wallet/pantry rows without
-- setting `reconciled_with_id` or superseding the statement's own extracted line.
-- The purchase counted twice, and because finance-app's reconcile queue is
-- `WHERE statement_id IS NULL`, the stamp also hid those rows from the only thing
-- that could repair them. Answering "was anything lost?" required restoring the
-- 02:30 pg_dump into a scratch database and diffing it column by column, because
-- NOTHING in this estate records what changed. That is the gap this closes.
--
-- WHY AT THE DATABASE AND NOT IN THE APP: the ledger has at least five independent
-- writers -- finance-app (Prisma), n8n (raw SQL in a Postgres node), pantry-app,
-- the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`. The 4a bug
-- was an n8n raw-SQL UPDATE; an application-level audit would not have seen it at
-- all. A trigger cannot be bypassed by choice of client.
--
-- WHY NOT POSTGRES' OWN LOGS: checked live on this instance -- wal_level=replica
-- (so WAL cannot be logically decoded into row changes), wal_keep_size=0 and
-- archive_mode=off (so WAL is recycled almost immediately), log_statement=none,
-- logging_collector=off, track_commit_timestamp=off, and pgaudit is not installed.
-- Postgres was recording nothing usable. It has no built-in audit or temporal
-- tables; this always has to be built.
CREATE TABLE IF NOT EXISTS transaction_audit (
id bigserial PRIMARY KEY,
changed_at timestamptz NOT NULL DEFAULT now(),
-- Groups every row touched by one logical edit. `applyReconcileMatch` moves
-- overrides, tags, splits and metadata in a single transaction; without this
-- they read as four unrelated events.
txid bigint NOT NULL DEFAULT txid_current(),
table_name text NOT NULL,
op text NOT NULL CHECK (op IN ('INSERT','UPDATE','DELETE')),
-- Full primary key as an object, because transaction_tags is composite
-- (transaction_id, tag_id) and the others are a bare `id`.
row_key jsonb NOT NULL,
-- Denormalised so "everything that ever happened to transaction N" is one
-- index scan rather than a five-way union.
transaction_id bigint,
-- On UPDATE these hold ONLY the columns that actually changed. Storing whole
-- rows would bury a one-column edit in forty unchanged fields, and
-- expense_metadata.line_items alone can be kilobytes.
old_data jsonb,
new_data jsonb,
-- Attribution is partial and that is understood: every client connects as the
-- same `personal` role. app_name is only as good as what each client sets;
-- client_addr distinguishes containers and is NULL for `docker exec` over the
-- unix socket, which is itself a useful signal.
db_user text NOT NULL DEFAULT current_user,
app_name text,
client_addr inet
);
COMMENT ON TABLE transaction_audit IS
'Append-only row-level change log for transactions + curation children. Never UPDATE or DELETE from this table.';
CREATE INDEX IF NOT EXISTS transaction_audit_txn_time
ON transaction_audit (transaction_id, changed_at DESC);
CREATE INDEX IF NOT EXISTS transaction_audit_time
ON transaction_audit (changed_at DESC);
CREATE INDEX IF NOT EXISTS transaction_audit_txid
ON transaction_audit (txid);
CREATE INDEX IF NOT EXISTS transaction_audit_table_time
ON transaction_audit (table_name, changed_at DESC);
-- TG_ARGV[0]: comma-separated candidate columns for the owning transaction id;
-- the first non-null wins (expense_metadata carries the id in
-- transaction_id OR matched_transaction_id -- a card-settled order
-- only ever has the latter).
-- TG_ARGV[1]: comma-separated primary key columns.
CREATE OR REPLACE FUNCTION log_transaction_change() RETURNS trigger
LANGUAGE plpgsql SECURITY DEFINER
SET search_path = public, pg_temp
AS $fn$
DECLARE
v_full jsonb;
v_old jsonb;
v_new jsonb;
v_key jsonb := '{}'::jsonb;
v_txn bigint;
k text;
BEGIN
IF TG_OP = 'INSERT' THEN
v_new := to_jsonb(NEW);
v_full := v_new;
ELSIF TG_OP = 'DELETE' THEN
v_old := to_jsonb(OLD);
v_full := v_old;
ELSE
v_old := to_jsonb(OLD);
v_new := to_jsonb(NEW);
v_full := v_new;
-- Reduce to the changed columns only. IS DISTINCT FROM so a NULL->NULL
-- comparison counts as unchanged rather than evaluating to NULL.
FOR k IN SELECT jsonb_object_keys(v_old) LOOP
IF (v_old -> k) IS NOT DISTINCT FROM (v_new -> k) THEN
v_old := v_old - k;
v_new := v_new - k;
END IF;
END LOOP;
-- A no-op UPDATE is noise, and finance-app's upserts issue plenty of them.
IF v_old = '{}'::jsonb THEN
RETURN NULL;
END IF;
END IF;
FOREACH k IN ARRAY string_to_array(TG_ARGV[1], ',') LOOP
v_key := v_key || jsonb_build_object(k, v_full -> k);
END LOOP;
FOREACH k IN ARRAY string_to_array(TG_ARGV[0], ',') LOOP
IF v_txn IS NULL AND (v_full ->> k) IS NOT NULL THEN
v_txn := (v_full ->> k)::bigint;
END IF;
END LOOP;
INSERT INTO transaction_audit
(table_name, op, row_key, transaction_id, old_data, new_data, app_name, client_addr)
VALUES
(TG_TABLE_NAME, TG_OP, v_key, v_txn, v_old, v_new,
NULLIF(current_setting('application_name', true), ''), inet_client_addr());
RETURN NULL; -- AFTER trigger; return value is ignored
END
$fn$;
-- AFTER, so a statement that raises never leaves a phantom audit row; the audit
-- write is in the same transaction as the change and rolls back with it.
DROP TRIGGER IF EXISTS trg_audit_transactions ON transactions;
CREATE TRIGGER trg_audit_transactions
AFTER INSERT OR UPDATE OR DELETE ON transactions
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('id', 'id');
DROP TRIGGER IF EXISTS trg_audit_transaction_splits ON transaction_splits;
CREATE TRIGGER trg_audit_transaction_splits
AFTER INSERT OR UPDATE OR DELETE ON transaction_splits
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
DROP TRIGGER IF EXISTS trg_audit_transaction_tags ON transaction_tags;
CREATE TRIGGER trg_audit_transaction_tags
AFTER INSERT OR UPDATE OR DELETE ON transaction_tags
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'transaction_id,tag_id');
DROP TRIGGER IF EXISTS trg_audit_transaction_overrides ON transaction_overrides;
CREATE TRIGGER trg_audit_transaction_overrides
AFTER INSERT OR UPDATE OR DELETE ON transaction_overrides
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
DROP TRIGGER IF EXISTS trg_audit_expense_metadata ON expense_metadata;
CREATE TRIGGER trg_audit_expense_metadata
AFTER INSERT OR UPDATE OR DELETE ON expense_metadata
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id,matched_transaction_id', 'id');
@@ -1,76 +0,0 @@
-- Account continuity across a rebrand, reissue or issuer change.
--
-- WHY THIS EXISTS: one card can appear in `statements` under two completely
-- different identities. The live case is the Citi Prestige card, which moved to
-- NAB and was rebranded MyCard:
--
-- Citibank 5258073330340253 statements ... 2025-11-13
-- NAB 1015001000116227 statements 2025-11-14 ...
--
-- Same 14th-to-13th billing cycle, no gap, one continuous product. Nothing in
-- the schema linked them, so anything grouping by (bank_name, account) sees a
-- dead account and a new one with no history. For the annual-fee detector that
-- is not cosmetic: the $700 fee charged 2024-12-13 and 2025-11-13 reads as
-- "stopped" when it is due again around November 2026 under the new name.
--
-- WHY A TABLE AND NOT AN INFERENCE: contiguous-cycle heuristics would join two
-- genuinely different accounts as readily as one renamed one, and a wrong join
-- merges two real fee series. There are only a handful of these events in a
-- lifetime, each announced by mail, so each link is asserted once with the
-- document that proves it. `evidence` is not decoration -- it is how a future
-- reader checks the claim instead of trusting it.
--
-- NOT `account_owner_mappings`: that answers "whose account is this" and keys on
-- the raw (bank_name, account_number). This answers "is this the same account as
-- that one", and keys on the normalised form.
CREATE TABLE IF NOT EXISTS account_links (
id bigserial PRIMARY KEY,
-- Normalised the same way statements.account_number_key is generated
-- (non-alphanumerics stripped), so these join without per-call cleanup.
-- Amex-style masks keep their X's: they record which digits were redacted.
from_bank_name text NOT NULL,
from_account_key text NOT NULL,
to_bank_name text NOT NULL,
to_account_key text NOT NULL,
-- When the successor took over. The predecessor's last statement and the
-- successor's first are the observable boundary; this is the announced date
-- where mail gives one.
effective_date date,
reason text NOT NULL
CHECK (reason IN ('rebrand', 'reissue', 'issuer_change', 'migration')),
-- Provenance for the claim, e.g. 'source_documents:32919'. Free text because
-- the evidence is not always a document -- sometimes it is a statement pair.
evidence text,
notes text,
created_at timestamptz NOT NULL DEFAULT now(),
-- One successor per predecessor: a chain, never a fork. A fork would make
-- "the current identity of this account" ambiguous, and the resolver walks
-- this forward assuming it is not.
CONSTRAINT uq_account_links_from UNIQUE (from_bank_name, from_account_key),
-- A link to itself would make that walk non-terminating.
CONSTRAINT account_links_no_self CHECK (
from_bank_name IS DISTINCT FROM to_bank_name
OR from_account_key IS DISTINCT FROM to_account_key
)
);
COMMENT ON TABLE account_links IS
'Predecessor -> successor account identity across rebrands/reissues. Hand-asserted, each row carrying the evidence for its claim.';
CREATE INDEX IF NOT EXISTS account_links_to
ON account_links (to_bank_name, to_account_key);
-- The one link the mail record proves today.
-- source_documents 32919 (2025-10-01) "your Citi Prestige Card account is moving to MyCard"
-- source_documents 32618 (2025-11-10) "MyCard is coming soon"
-- source_documents 32539 (2025-11-24) "The new MyCard App and MyCard Online are now live"
-- Statement boundary: Citibank ends 2025-11-13, NAB begins 2025-11-14.
INSERT INTO account_links
(from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason, evidence, notes)
VALUES
('Citibank', '5258073330340253', 'NAB', '1015001000116227', '2025-11-14', 'issuer_change',
'source_documents:32919,32618,32539',
'Citi Prestige -> MyCard Prestige. Billing cycle (14th-13th) continues unbroken across the boundary.')
ON CONFLICT (from_bank_name, from_account_key) DO NOTHING;
@@ -1,26 +0,0 @@
-- When, and from which app, a wallet notification arrived.
--
-- WHY: one tap emits two notifications 2-8 seconds apart -- the issuer app
-- (card tail) and Apple Wallet (merchant, suburb) -- and wallet-ingestion.ts
-- merges them. It used to recognise the pair by amount alone, within +/-1 DAY,
-- which is wide enough to fold two different purchases together: on 2026-10-01
-- an Amex "$50.00 at A MEDIUM" notification merged into the UNICEF $50 donation
-- from 17 hours earlier and renamed it. A real pair is seconds apart and comes
-- from two DIFFERENT apps; two notifications from one app are two charges.
-- Neither fact was stored anywhere on the ledger side, so the matcher could not
-- ask. These columns let it.
--
-- Backfilled from capture.wallet_events through source_message_id ('evt:<id>'),
-- which every wallet-written metadata row carries.
ALTER TABLE expense_metadata
ADD COLUMN IF NOT EXISTS notified_at timestamptz,
ADD COLUMN IF NOT EXISTS notif_app text;
UPDATE expense_metadata em
SET notified_at = e.received_at,
notif_app = e.notif_app
FROM capture.wallet_events e
WHERE em.source = 'wallet'
AND em.source_message_id = 'evt:' || e.id
AND em.notified_at IS NULL;
@@ -1,43 +0,0 @@
-- A statementless row for a charge that never posted.
--
-- WHY: the wallet lane books every card notification as a purchase, and a card
-- notification is sent for an AUTHORISATION, not a settlement. When the
-- merchant releases the hold the bank never posts the charge, so no statement
-- line ever arrives: the row sits in the reconcile queue forever and counts as
-- spend in every figure. Txn 7951: Airbnb HMXAHAW49Y, $1,040.41 held on Amex
-- 1000 on 2026-09-24, booking cancelled 25 Sep inside the free window, hold
-- released ("We removed the $1,040.41 (AUD) payment hold"). Statement 219
-- (11 Sep - 10 Oct) has no such line and reconciles to its header without it.
--
-- Neither existing exclusion fits. reconciled_with_id needs a statement twin
-- and superseded_by_id needs a surviving row (and may not point at itself,
-- transactions_no_self_supersede). There is no survivor: nothing happened.
--
-- Not a DELETE: every child of transactions cascades, and the row is also the
-- sink the wallet lane checks before re-posting an event. Deleting it would let
-- the next prompt_version bump re-create it.
ALTER TABLE transactions
ADD COLUMN IF NOT EXISTS voided_at timestamptz,
ADD COLUMN IF NOT EXISTS void_reason text;
COMMENT ON COLUMN transactions.voided_at IS
'Set when this statementless row records a charge that never posted (e.g. a released authorisation hold). Excluded from every figure and from the reconcile queue, kept for its children and audit trail. NULL = live.';
-- A reason is mandatory: a void with no explanation is indistinguishable from
-- a mistake six months later.
ALTER TABLE transactions
ADD CONSTRAINT transactions_void_has_reason
CHECK ((voided_at IS NULL AND void_reason IS NULL)
OR (voided_at IS NOT NULL AND btrim(coalesce(void_reason, '')) <> ''));
-- Statement rows cannot be voided. A statement line is the bank's own record
-- of a posted charge, and the statement's rows must keep summing to its
-- header; a wrong statement row is a supersede, not a void.
ALTER TABLE transactions
ADD CONSTRAINT transactions_void_statementless
CHECK (voided_at IS NULL OR statement_id IS NULL);
CREATE INDEX IF NOT EXISTS idx_transactions_voided
ON transactions (voided_at)
WHERE voided_at IS NOT NULL;
+1 -8
View File
@@ -190,9 +190,7 @@ model transactions {
owner_id Int? owner_id Int?
reconciled_with_id Int? reconciled_with_id Int?
superseded_by_id Int? superseded_by_id Int?
voided_at DateTime? @db.Timestamptz(6) // a charge that never posted (migration 0040) principal_amount Decimal? @db.Decimal(12, 2)
void_reason String?
principal_amount Decimal? @db.Decimal(12, 2)
interest_amount Decimal? @db.Decimal(12, 2) interest_amount Decimal? @db.Decimal(12, 2)
source String? // feed this row came from, e.g. "frollo"; NULL = statement/manual (migration 0028) source String? // feed this row came from, e.g. "frollo"; NULL = statement/manual (migration 0028)
source_ref String? // the provider's own transaction id — idempotency key for re-imports source_ref String? // the provider's own transaction id — idempotency key for re-imports
@@ -239,9 +237,6 @@ model expense_metadata {
tender_raw String? tender_raw String?
receipt_sha256 String? receipt_sha256 String?
receipt_group String? receipt_group String?
// 0039 — when/where a wallet notification arrived; the pair merge needs both.
notified_at DateTime? @db.Timestamptz(6)
notif_app String?
transaction transactions? @relation(fields: [transaction_id], references: [id], onDelete: Cascade) transaction transactions? @relation(fields: [transaction_id], references: [id], onDelete: Cascade)
@@unique([source, order_reference], name: "uq_expense_source_order") @@unique([source, order_reference], name: "uq_expense_source_order")
@@ -288,8 +283,6 @@ model intake_batches {
status String status String
transaction_id Int? transaction_id Int?
capture_id Int? capture_id Int?
// The undo baseline, written once per batch (see 0034 and line-items.ts).
previous_line_items Json?
created_at DateTime @default(now()) @db.Timestamptz(6) created_at DateTime @default(now()) @db.Timestamptz(6)
resolved_at DateTime? @db.Timestamptz(6) resolved_at DateTime? @db.Timestamptz(6)
} }
+5 -14
View File
@@ -19,23 +19,14 @@ if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
exit 1 exit 1
fi fi
# Drop and recreate, so the schema is rebuilt rather than replayed. Replaying # Create test database (ignore error if it already exists)
# the dump over an existing database was a silent no-op for every migration: docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres \
# each CREATE TABLE failed with "already exists" and the old shape survived, so -c "CREATE DATABASE $TEST_DB;" 2>/dev/null || true
# a new column (0039's expense_metadata.notified_at, 2026-10-03) never reached
# the test database. The guard is because this now DROPs: it refuses anything
# not named *_test, whatever TEST_DB was set to.
case "$TEST_DB" in
*_test) ;;
*) echo "ERROR: refusing to drop '$TEST_DB' -- the test database must be named *_test"; exit 1 ;;
esac
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres -q \
-c "DROP DATABASE IF EXISTS $TEST_DB WITH (FORCE);" \
-c "CREATE DATABASE $TEST_DB;"
# Wipe and rebuild schema from production (schema only, no data)
echo "Copying schema from $PG_USER to $TEST_DB..." echo "Copying schema from $PG_USER to $TEST_DB..."
docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \ docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q -v ON_ERROR_STOP=1 | docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q
echo "Schema ready." echo "Schema ready."
@@ -105,19 +105,6 @@ describe("NATIVE_CURRENCY", () => {
expect(unconverted).toBe(true); expect(unconverted).toBe(true);
}); });
it("labels a statement-less wallet row AUD, because its amount already is", async () => {
// Wallet rows store a provisional AUD figure in amount AND amount_aud, and
// the native figure in foreign_currency_amount. Reading the foreign code
// here printed "INR 9.42" for an INR 524 tap (txn 8162).
const id = await scratchTxn(
"transaction_date, description, amount, amount_aud, transaction_type, payment_method, source, source_ref, foreign_currency_amount, foreign_currency_code",
"'2026-10-03','Wallet foreign fixture', 9.42, 9.42, 'debit', 'card', 'wallet', 'evt:fixture-native-ccy', 524.00, 'INR'"
);
const { ccy, unconverted } = await currencyOf(id);
expect(ccy).toBe("AUD");
expect(unconverted).toBe(false);
});
it("prefers the statement's currency over the foreign-charge record", async () => { it("prefers the statement's currency over the foreign-charge record", async () => {
// Opposite convention: on an AUD statement, `amount` is AUD and // Opposite convention: on an AUD statement, `amount` is AUD and
// foreign_currency_code merely notes what was originally charged. Reading // foreign_currency_code merely notes what was originally charged. Reading
@@ -1,170 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, mockDbWithPool } from "./helpers";
/**
* Upcoming annual charges from payments alone.
*
* The exclusions carry this file. Anything can be made to find a card annual
* fee; the difficulty is not also reporting the monthly account-servicing fee,
* the per-transfer Wise charge, and the restaurant visited twice a year apart.
* Measured on the real ledger, a naive "two charges ~365 days apart" ranks a
* bakery first with 202 such pairs.
*/
let pool: Pool;
let ownerId: number;
let getUpcomingAnnualCharges: typeof import("../../lib/annual-charges").getUpcomingAnnualCharges;
let acctSeq = 0;
/** A statement plus one transaction on it, for a named account. */
async function charge(
bank: string,
acct: string,
date: string,
description: string,
amount: number,
type = "fee"
) {
const key = `${bank}:${acct}`;
let stId = stmtCache.get(`${key}:${date}`);
if (!stId) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ($4, $1, $2, $3::date - 20, $3::date + 10, $5) RETURNING id`,
[acct, bank, date, `s${acctSeq++}.pdf`, ownerId]
);
stId = st.rows[0].id as number;
stmtCache.set(`${key}:${date}`, stId);
}
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1,$2,$3,$4,$5,$6)`,
[stId, date, description, amount, type, ownerId]
);
}
let stmtCache = new Map<string, number>();
/** A statementless merchant purchase, for the non-fee signal. */
async function purchase(date: string, merchant: string, amount: number) {
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id, merchant_normalized, row_index)
VALUES (NULL,$1,$2,$3,'debit',$4,$2,0)`,
[date, merchant, amount, ownerId]
);
}
const find = <T extends { label: string }>(rows: T[], label: string) =>
rows.find((r) => r.label === label);
beforeAll(async () => {
pool = createPool();
mockDbWithPool(pool);
({ getUpcomingAnnualCharges } = await import("../../lib/annual-charges"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM account_links");
stmtCache = new Map();
({ ownerId } = await seedParticipants(pool));
});
describe("what it finds", () => {
it("confirms a card annual fee that repeated a year later", async () => {
await charge("American Express", "1000", "2025-06-10", "ANNUAL CARD FEE", 875);
await charge("American Express", "1000", "2026-06-10", "ANNUAL CARD FEE", 1750);
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CARD FEE");
expect(row?.confidence).toBe("confirmed");
expect(row?.observations).toBe(2);
// The amount is NOT averaged: this fee doubled, and the next one to plan for
// is the latest, not the mean of a rise.
expect(Number(row?.last_amount)).toBe(1750);
expect(row?.next_due_estimate.startsWith("2027-06")).toBe(true);
});
it("accepts a fee seen only once when the bank names it annual", async () => {
await charge("Westpac", "8032", "2025-10-21", "CARD FEE", 295);
const row = find(await getUpcomingAnnualCharges(ownerId), "CARD FEE");
// One charge cannot show a cadence, but the kind is not in doubt.
expect(row?.confidence).toBe("likely");
expect(row?.next_due_estimate).toBe("2026-10-21");
});
it("finds an annual merchant charge with no fee wording, if the amount is stable", async () => {
await purchase("2025-02-03", "AMAZON PRIME", 79);
await purchase("2026-02-01", "AMAZON PRIME", 79);
const row = find(await getUpcomingAnnualCharges(ownerId), "AMAZON PRIME");
expect(row?.confidence).toBe("possible");
});
});
describe("what it must NOT report", () => {
it("ignores a monthly account fee", async () => {
for (const d of ["2026-01-31", "2026-02-28", "2026-03-31", "2026-04-30"]) {
await charge("ANZ", "5626", d, "ACCOUNT SERVICING FEE", 5);
}
expect(find(await getUpcomingAnnualCharges(ownerId), "ACCOUNT SERVICING FEE")).toBeUndefined();
});
it("ignores per-transaction fees, which never repeat under one label", async () => {
await charge("Wise Australia Pty Ltd.", "9001", "2026-05-20", "WISE CHARGES FOR: TRANSFER-2143509822", 30.67);
await charge("Wise Australia Pty Ltd.", "9001", "2026-06-04", "WISE CHARGES FOR: TRANSFER-2171403032", 19.51);
const rows = await getUpcomingAnnualCharges(ownerId);
expect(rows.filter((r) => r.label.startsWith("WISE CHARGES"))).toHaveLength(0);
});
it("ignores a merchant visited twice a year apart for wildly different amounts", async () => {
// The exact shape that defeats a pure cadence test: Target, $30 then $198.
await purchase("2025-07-21", "TARGET", 30);
await purchase("2026-07-11", "TARGET", 198);
expect(find(await getUpcomingAnnualCharges(ownerId), "TARGET")).toBeUndefined();
});
it("ignores a merchant seen only once", async () => {
await purchase("2026-04-03", "UBER ONE MEMBERSHIP", 96);
expect(find(await getUpcomingAnnualCharges(ownerId), "UBER ONE MEMBERSHIP")).toBeUndefined();
});
});
describe("account identity and honesty about gaps", () => {
it("treats a renamed account as one continuous fee series", async () => {
// Citi Prestige -> NAB MyCard: same card, two bank names, two numbers.
await charge("Citibank", "5258073330340253", "2024-12-13", "Annual Fee", 700);
await charge("Citibank", "5258073330340253", "2025-11-13", "Annual Fee", 700);
await pool.query(
`INSERT INTO account_links (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason)
VALUES ('Citibank','5258073330340253','NAB','1015001000116227','2025-11-14','issuer_change')`
);
// A statement under the NEW identity, so the link has somewhere to resolve to.
await charge("NAB", "1015001000116227", "2026-08-13", "Purchase", 29.99, "debit");
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL FEE");
expect(row?.observations).toBe(2);
expect(row?.confidence).toBe("confirmed");
// Names the account's CURRENT bank -- the next fee comes from there, and
// sending someone to the retired brand is the bug this link exists to stop.
expect(row?.bank_name).toBe("NAB");
});
it("marks a passed estimate as expected_not_seen rather than upcoming", async () => {
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CHARGE");
expect(row?.status).toBe("expected_not_seen");
// And it reports how stale the account is, which is what separates
// "it stopped being charged" from "we stopped importing the statements".
expect(row?.account_days_stale).toBeGreaterThan(300);
});
it("sorts upcoming ahead of expected_not_seen", async () => {
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
await charge("Westpac", "8032", "2026-08-01", "CARD FEE", 295);
const rows = await getUpcomingAnnualCharges(ownerId);
const firstMissing = rows.findIndex((r) => r.status === "expected_not_seen");
const lastUpcoming = rows.map((r) => r.status).lastIndexOf("upcoming");
expect(lastUpcoming).toBeLessThan(firstMissing);
});
});
@@ -1,271 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
/**
* The automatic pass over the reconcile queue.
*
* This exists because the wallet lane writes a transaction for every tap
* (DECISIONS.md ING-11), adding ~63 rows a month to a queue already ~177 deep
* and worked by hand. Hiding those rows instead is explicitly ruled out --
* see the awaitsStatementLine() removal note in queries.ts.
*
* The cases that must NOT match matter more than the ones that must: this runs
* unattended, and a wrong pairing merges one purchase's curation onto another
* purchase's row. So most of what follows is about refusal.
*/
const MACHINE_TOKEN = "test-wallet-token";
let pool: Pool;
let ownerId: number;
let autoReconcile: typeof import("../../app/api/transactions/reconcile/route").PATCH;
/** Browser-session caller: no machine token, auth comes from the mocked session. */
const userReq = (url = "http://x/api/transactions/reconcile") =>
({ url, headers: { get: () => null } }) as never;
/** Scheduled caller: machine token, no session, sweeps every owner. */
const machineReq = (url = "http://x/api/transactions/reconcile") =>
({ url, headers: { get: (h: string) => (h === "x-ingest-token" ? MACHINE_TOKEN : null) } }) as never;
async function statementLine(
amount: number,
date: string,
description: string,
opts: { owner?: number; type?: string; fxAmount?: number; fxCode?: string } = {}
) {
const owner = opts.owner ?? ownerId;
const st = await pool.query(
// Account number varies by owner: uq_statement_identity is
// (bank_name, account_number, billing_end_date), so two owners sharing one
// account collide rather than modelling two real cards.
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, owner, `1234-5678-9012-${String(owner).padStart(4, "0")}`]
);
const t = await pool.query(
`INSERT INTO transactions
(statement_id, transaction_date, description, amount, transaction_type, owner_id,
foreign_currency_amount, foreign_currency_code)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id`,
[st.rows[0].id, date, description, amount, opts.type ?? "debit", owner,
opts.fxAmount ?? null, opts.fxCode ?? null]
);
return t.rows[0].id as number;
}
async function walletRow(
amount: number,
date: string,
opts: { fxAmount?: number; fxCode?: string; paymentMethod?: string } = {}
) {
const id = await insertTransaction(pool, ownerId, {
amount, transaction_date: date, description: "Wallet tap",
});
await pool.query(
`UPDATE transactions
SET foreign_currency_amount = $2, foreign_currency_code = $3, payment_method = $4
WHERE id = $1`,
[id, opts.fxAmount ?? null, opts.fxCode ?? null, opts.paymentMethod ?? null]
);
return id;
}
const reconciledWith = async (id: number) =>
(await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [id]))
.rows[0].reconciled_with_id as number | null;
beforeAll(async () => {
process.env.WALLET_INGEST_TOKEN = MACHINE_TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ PATCH: autoReconcile } = await import("../../app/api/transactions/reconcile/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
describe("pairs it should apply", () => {
it("matches an exact amount two days apart and moves the link", async () => {
const manual = await walletRow(13.5, "2026-08-13");
const stmt = await statementLine(13.5, "2026-08-15", "BREADWORLD BAKERY");
const body = await (await autoReconcile(userReq())).json();
expect(body.reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("matches a foreign row on the NATIVE pair, not on the AUD amount", async () => {
// The bank converts at its own rate, so the wallet row cannot predict the
// AUD figure. It can only know THB 500. Deliberately give the statement a
// wildly different AUD amount to prove `amount` is not what matched.
const manual = await walletRow(999.99, "2026-08-13", { fxAmount: 500, fxCode: "THB" });
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
fxAmount: 500, fxCode: "THB",
});
const body = await (await autoReconcile(userReq())).json();
expect(body.reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("carries a SETTLED split across without resurrecting the debt", async () => {
const manual = await walletRow(50, "2026-08-13");
const stmt = await statementLine(50, "2026-08-14", "SHARED DINNER");
await pool.query(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled, settled_at)
VALUES ($1, $2, 50, true, now())`,
[manual, ownerId]
);
expect((await autoReconcile(userReq())).status ?? 200).toBe(200);
const moved = await pool.query(
`SELECT settled, settled_at FROM transaction_splits WHERE transaction_id = $1`, [stmt]
);
expect(moved.rowCount).toBe(1);
// The whole point: a delete-and-recreate defaults `settled` to false, which
// silently marks a paid debt owed again. Nothing errors when it regresses.
expect(moved.rows[0].settled).toBe(true);
expect(moved.rows[0].settled_at).not.toBeNull();
});
});
describe("pairs it must refuse", () => {
it("refuses two identical amounts in the same window (ambiguous both ways)", async () => {
// Two $13.50 bakery taps in one week is the ordinary case, not an edge one.
const a = await walletRow(13.5, "2026-08-13");
const b = await walletRow(13.5, "2026-08-14");
await statementLine(13.5, "2026-08-14", "BREADWORLD ONE");
await statementLine(13.5, "2026-08-15", "BREADWORLD TWO");
const body = await (await autoReconcile(userReq())).json();
expect(body.reconciled).toBe(0);
expect(await reconciledWith(a)).toBeNull();
expect(await reconciledWith(b)).toBeNull();
});
it("refuses a near-miss amount — a person should look at that", async () => {
const manual = await walletRow(13.5, "2026-08-13");
await statementLine(13.55, "2026-08-14", "BREADWORLD BAKERY");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
it("refuses a match outside the day window", async () => {
const manual = await walletRow(13.5, "2026-08-13");
await statementLine(13.5, "2026-08-25", "BREADWORLD BAKERY");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
it("refuses a statement row already claimed by another manual row", async () => {
const claimed = await statementLine(40, "2026-08-14", "ALREADY TAKEN");
const first = await walletRow(40, "2026-08-13");
await pool.query(`UPDATE transactions SET reconciled_with_id = $2 WHERE id = $1`, [first, claimed]);
const second = await walletRow(40, "2026-08-14");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(second)).toBeNull();
});
it("refuses a credits-funded row — no card leg is ever coming", async () => {
const manual = await walletRow(30, "2026-08-13", { paymentMethod: "credits" });
await statementLine(30, "2026-08-14", "COINCIDENTAL AMOUNT");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
});
describe("an AUD row against a foreign statement line", () => {
it("DOES match on the AUD figure — this is what every real pair looks like", async () => {
// Checked against live data: all 48 reconciliations a human has made are
// this exact shape -- manual row with no currency, statement row carrying
// EUR or CHF, paired on exact AUD. Overseas receipts and orders record what
// was charged in AUD; only the wallet lane records natively. An earlier
// version of this rule required both sides to be non-foreign, on the
// reasoning that a currency mismatch is ambiguous. It matched 0 of the 48.
const manual = await walletRow(22.85, "2026-08-13");
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
fxAmount: 500, fxCode: "THB",
});
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("still refuses when a second candidate makes it ambiguous", async () => {
// The coincidence this guards against -- a domestic tap whose amount equals
// a foreign charge's AUD figure -- is caught by unique-both-ways, not by a
// currency test.
const manual = await walletRow(22.85, "2026-08-13");
await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", { fxAmount: 500, fxCode: "THB" });
await statementLine(22.85, "2026-08-13", "MELBOURNE CAFE");
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
expect(await reconciledWith(manual)).toBeNull();
});
});
describe("dry run", () => {
it("reports what it would do and writes nothing", async () => {
const manual = await walletRow(13.5, "2026-08-13");
await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
const body = await (await autoReconcile(userReq("http://x/api/transactions/reconcile?dry=1"))).json();
expect(body.dry_run).toBe(true);
expect(body.would_reconcile).toBe(1);
expect(await reconciledWith(manual)).toBeNull();
});
});
describe("the scheduled sweep", () => {
it("runs on the machine token with no session at all", async () => {
const manual = await walletRow(13.5, "2026-08-13");
const stmt = await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
const body = await (await autoReconcile(machineReq())).json();
expect(body.reconciled).toBe(1);
expect(await reconciledWith(manual)).toBe(stmt);
});
it("covers EVERY owner, not just one", async () => {
// 169 of the 177 rows outstanding belong to the second participant, so a
// sweep scoped to a single user would leave almost all of the queue alone.
const other = (await pool.query(
`INSERT INTO participants (name) VALUES ('Carol') RETURNING id`
)).rows[0].id as number;
const mine = await walletRow(11, "2026-08-13");
const theirTx = (await pool.query(
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description,
amount, transaction_type, category, row_index)
VALUES ($1, NULL, '2026-08-13', 'Theirs', 22, 'debit', 'other', 0) RETURNING id`, [other]
)).rows[0].id as number;
await statementLine(11, "2026-08-14", "MINE");
await statementLine(22, "2026-08-14", "THEIRS", { owner: other });
const body = await (await autoReconcile(machineReq())).json();
expect(body.reconciled).toBe(2);
expect(await reconciledWith(mine)).not.toBeNull();
expect(await reconciledWith(theirTx)).not.toBeNull();
});
it("rejects a wrong machine token and falls through to session auth", async () => {
const bad = ({ url: "http://x/", headers: { get: () => "not-the-token" } }) as never;
// Session is mocked, so this still succeeds -- but as the session user only,
// proving the bad token did not grant the all-owner sweep.
const body = await (await autoReconcile(bad)).json();
expect(body.reconciled).toBe(0);
});
});
-24
View File
@@ -23,25 +23,6 @@ export function mockDbWithPool(p: Pool) {
const result = await p.query(sql, params); const result = await p.query(sql, params);
return result.rows[0] ?? null; return result.rows[0] ?? null;
}, },
// Mirrors withTransaction: one connection, BEGIN/COMMIT around the callback,
// so pg_advisory_xact_lock actually holds across the statements inside it.
withTransaction: async (fn: (q: unknown) => Promise<unknown>) => {
const client = await p.connect();
try {
await client.query("BEGIN");
const result = await fn({
queryRaw: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows,
queryRow: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows[0] ?? null,
});
await client.query("COMMIT");
return result;
} catch (e) {
await client.query("ROLLBACK");
throw e;
} finally {
client.release();
}
},
prisma: p, prisma: p,
})); }));
} }
@@ -84,11 +65,6 @@ export async function resetDB(pool: Pool) {
await assertTestDatabase(pool); await assertTestDatabase(pool);
await pool.query(` await pool.query(`
TRUNCATE TRUNCATE
-- The audit triggers (migration 0037) fire on every INSERT the suite makes.
-- TRUNCATE does not fire row-level triggers, so nothing here is logged --
-- but without this line the log is never cleared either, and the table
-- grows by roughly a thousand rows per full run, forever.
transaction_audit,
split_payments, split_payments,
transaction_splits, transaction_splits,
transaction_tags, transaction_tags,
+3 -416
View File
@@ -49,7 +49,6 @@ let fileDir: string;
let intakePOST: (req: NextRequest) => Promise<Response>; let intakePOST: (req: NextRequest) => Promise<Response>;
let lineItemsPUT: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>; let lineItemsPUT: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
let slackPOST: (req: NextRequest) => Promise<Response>; let slackPOST: (req: NextRequest) => Promise<Response>;
let orderGET: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
/** What the next LiteLLM call answers, and what the next pantry call answers. */ /** What the next LiteLLM call answers, and what the next pantry call answers. */
let modelReply: ExtractedOrder | Error = burgertory; let modelReply: ExtractedOrder | Error = burgertory;
@@ -105,21 +104,6 @@ async function seedOrder(over: { merchant: string; date: string; amount: number;
} }
return id; return id;
} }
/** A row shaped the way the wallet lane writes one: statementless, no metadata
* at all, no platform, and named by the notification rather than the receipt. */
async function seedWalletRow(over: { merchant: string; date: string; amount: number; owner?: number }) {
const t = await pool.query(
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description, amount, amount_aud, transaction_type, category, merchant_name, merchant_normalized, payment_method, location, source, source_ref, row_index)
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', $3, $3, 'card', 'Werribee, VIC', 'wallet', $5, 0) RETURNING id`,
[over.owner ?? ownerId, over.date, over.merchant, over.amount, `evt:${Math.floor(over.amount * 100)}`]
);
return t.rows[0].id as number;
}
/** What `components/order-details.tsx` would actually render for a transaction. */
const orderPanel = async (txnId: number) =>
(await (await orderGET({ headers: { get: () => null } } as unknown as NextRequest, { params: Promise.resolve({ id: String(txnId) }) })).json()) as { line_items?: unknown[] } | null;
const metaRows = async () => (await pool.query(`SELECT id, platform, line_items FROM expense_metadata ORDER BY id`)).rows as { id: number; platform: string | null; line_items: { description: string }[] }[];
const lineItemsOf = async (txnId: number) => const lineItemsOf = async (txnId: number) =>
(await pool.query(`SELECT line_items FROM expense_metadata WHERE transaction_id = $1 OR matched_transaction_id = $1`, [txnId])).rows[0]?.line_items; (await pool.query(`SELECT line_items FROM expense_metadata WHERE transaction_id = $1 OR matched_transaction_id = $1`, [txnId])).rows[0]?.line_items;
const batch = async (id: string) => (await pool.query(`SELECT * FROM intake_batches WHERE id = $1`, [id])).rows[0]; const batch = async (id: string) => (await pool.query(`SELECT * FROM intake_batches WHERE id = $1`, [id])).rows[0];
@@ -143,17 +127,9 @@ beforeAll(async () => {
participantForSlackUser: () => 1, participantForSlackUser: () => 1,
slackUserForParticipant: () => null, slackUserForParticipant: () => null,
})); }));
// The order-detail panel is the surface an enrichment has to appear on, so
// the read is exercised beside the write rather than assumed.
vi.doMock("@/lib/auth", () => ({ getCurrentUser: async () => ({ id: 1, name: "Owner", email: "o@x" }) }));
vi.doMock("@/lib/queries", async (importOriginal) => ({
...(await importOriginal<Record<string, unknown>>()),
canAccessTransactions: async () => true,
}));
({ POST: intakePOST } = await import("../../app/api/intake/route")); ({ POST: intakePOST } = await import("../../app/api/intake/route"));
({ PUT: lineItemsPUT } = await import("../../app/api/transactions/[id]/line-items/route")); ({ PUT: lineItemsPUT } = await import("../../app/api/transactions/[id]/line-items/route"));
({ POST: slackPOST } = await import("../../app/api/slack/interactive/route")); ({ POST: slackPOST } = await import("../../app/api/slack/interactive/route"));
({ GET: orderGET } = await import("../../app/api/transactions/[id]/order/route"));
stubFetch(); stubFetch();
}); });
@@ -234,82 +210,6 @@ describe("PUT /api/transactions/[id]/line-items (C2)", () => {
expect(em).toEqual({ source: "intake", platform: "ubereats", order_reference: "intake:b1", merchant_normalized: "Coles", amount: "85.74" }); expect(em).toEqual({ source: "intake", platform: "ubereats", order_reference: "intake:b1", merchant_normalized: "Coles", amount: "85.74" });
}); });
it("the undo baseline is recorded ONCE per batch: a retry gets the original, not its own write", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const original = [{ qty: 1, description: "what the order already had", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(original)]);
const id = uuid();
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
const first = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
expect(first.previous_line_items).toEqual(original);
// Pantry re-calls on every confirm and after a timeout. Before, the retry
// was handed its OWN write back, so undo restored the thing it undid.
const retry = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
expect(retry.previous_line_items).toEqual(original);
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual(original);
await put(txn, { line_items: retry.previous_line_items, source: "pantry", batch_id: id, replace: true });
expect(await lineItemsOf(txn)).toEqual(original);
});
it("without a batch id the baseline is still the row as read — nothing else changes", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const first = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
expect(first.previous_line_items).toEqual([]);
const second = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
expect(second.previous_line_items).toEqual(items);
});
it("a pinned row that no longer exists falls back to the live one, rather than reporting a phantom write", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const live = (await pool.query(`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].id as number;
const { replaceLineItems } = await import("../../lib/line-items");
const r = await replaceLineItems(txn, items, { source: "intake", metadataId: live + 999 });
expect(r.metadata_id).toBe(live);
expect(await lineItemsOf(txn)).toHaveLength(2);
});
it("two writes racing to create the metadata row both resolve to the same one", async () => {
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
const both = await Promise.all([
put(txn, { line_items: items, source: "intake", replace: true }),
put(txn, { line_items: items, source: "intake", replace: true }),
]);
expect(both.map((r) => r.status)).toEqual([200, 200]);
expect((await pool.query(`SELECT count(*)::int AS n FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].n).toBe(1);
});
it("the create path records an undo baseline too, before it writes", async () => {
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
const id = uuid();
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
await put(txn, { line_items: items, source: "intake", batch_id: id, currency: "AUD", replace: true });
// [] rather than NULL: there was nothing there, and "nothing" is the state
// undo has to be able to restore.
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual([]);
});
it("the panel reads the row the write landed on, whichever order the rows were inserted in", async () => {
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
// Inserted first: the delivery receipt pointing at the statement line.
await pool.query(
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, '[]'::jsonb)`,
[t, uuid()]
);
// Inserted second: a row the transaction owns. An unordered LIMIT 1 reads
// the first one, the write takes the second, and the enrichment reports
// success against a row nothing renders.
await pool.query(
`INSERT INTO expense_metadata (transaction_id, source, order_reference, line_items) VALUES ($1, 'pantry', $2, '[]'::jsonb)`,
[t, uuid()]
);
const written = await (await put(t, { line_items: items, source: "intake", replace: true })).json();
const panel = await orderPanel(t);
expect(panel?.line_items).toHaveLength(2);
expect((await metaRows()).find((r) => r.id === written.metadata_id)!.line_items).toHaveLength(2);
});
it("finds the row through matched_transaction_id — a card-settled order has only that", async () => { it("finds the row through matched_transaction_id — a card-settled order has only that", async () => {
const statementLine = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false }); const statementLine = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
await pool.query( await pool.query(
@@ -350,64 +250,6 @@ describe("POST /api/intake — gates", () => {
expect((await batch(id)).status).toBe("unsupported"); expect((await batch(id)).status).toBe("unsupported");
expect(calls).toHaveLength(0); expect(calls).toHaveLength(0);
}); });
// 2026-08-22, live batch 3d2ffbe8…: an iOS Shortcut coerced the shared photo
// to text, so the whole upload was the five characters `Image`. The model was
// asked anyway and returned Burgertory (Braybrook), two "The Pride" lines and
// a $41.15 total — none of which existed. Only the absent date kept it out of
// the matcher, and the invented header was stored on the batch as
// `needs_header`. The model must never be asked.
it("a text part that cannot be a receipt is declined BEFORE any model call", async () => {
const id = uuid();
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)", order_date: null, total: 41.15 };
const res = await intakePOST(
multipart({ source: "other", batch_id: id }, [{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") }])
);
const body = await res.json();
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
expect(body.text).toMatch(/single word/);
expect(body.text).toMatch(/"Image"/);
// The model was never asked — the only thing that makes confabulation impossible.
expect(calls).toHaveLength(0);
const row = await batch(id);
expect(row.status).toBe("unsupported");
expect(row.kind).toBe("unsupported");
expect(row.header).toBeNull();
expect(row.items).toBeNull();
expect(row.transaction_id).toBeNull();
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
});
it("a junk text part beside a real receipt does not sink the batch", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const body = await (
await intakePOST(
multipart({ source: "uber_eats" }, [
{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") },
{ name: "Receipt_19Aug2026.pdf", type: "application/pdf", bytes: BURGERTORY },
])
)
).json();
expect(body.status).toBe("enriched");
expect(await lineItemsOf(txn)).toHaveLength(2);
});
it("an order extracted from an unreadable PDF is unsupported, not needs_header", async () => {
const id = uuid();
// A PDF that renders to a scrap of junk still reaches the model — the
// pre-check only inspects text parts. `confabulationReason` is the belt.
vi.spyOn(await import("../../lib/intake-extract"), "extractPdfText").mockResolvedValueOnce("Image");
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)" };
const body = await (
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [{ name: "x.pdf", type: "application/pdf", bytes: BURGERTORY }]))
).json();
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
expect(body.text).toMatch(/nothing shared could be read/);
const row = await batch(id);
expect(row.status).toBe("unsupported");
expect(row.header).toBeNull();
expect(row.items).toBeNull();
});
}); });
describe("POST /api/intake — food (enrich vs candidates)", () => { describe("POST /api/intake — food (enrich vs candidates)", () => {
@@ -443,82 +285,10 @@ describe("POST /api/intake — food (enrich vs candidates)", () => {
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1); expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
}); });
it("the platform is part of the key: the same merchant/date/total on DoorDash is not the match — but it IS offered", async () => { it("the platform is part of the key: the same merchant/date/total on DoorDash is not the match", async () => {
const doordash = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "doordash" }); await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "doordash" });
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json(); const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body.status).toBe("candidates"); expect(body.status).toBe("candidates");
// The candidates card is a human choice, not a decision. The platform is one
// of the things that may be wrong, so it must not narrow the list — a
// platform-filtered card showed nothing at all here.
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([doordash]);
});
it("a NULL platform is 'no platform', not a wildcard — an in-store share does not overwrite the delivery order", async () => {
await pool.query(`INSERT INTO intake_merchant_kinds (merchant_key, kind) VALUES ('burgertory werribee', 'food')`);
const delivery = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "ubereats" });
const already = [{ qty: 1, description: "what the delivery order already had", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [delivery, JSON.stringify(already)]);
// `other` implies no platform; the delivery row carries one, so it is not
// the match — it is only a candidate.
const body = await (await intakePOST(multipart({ source: "other" }, [pdf]))).json();
expect(body.status).toBe("candidates");
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([delivery]);
expect(await lineItemsOf(delivery)).toEqual(already);
});
it("a reconciled manual row is neither matched nor offered — EXCLUDE_RECONCILED_SOURCE, as everywhere else", async () => {
const live = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const reconciled = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, reconciled]);
const near = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-17", amount: 19.9 });
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, near]);
// Two rows at the exact key would have read as an ambiguity; one of them is
// the same spend counted twice.
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ status: "enriched", transaction_id: live });
});
it("the write lands on the row the MATCH resolved through, not on whichever row sorts first", async () => {
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
// The statement line owns a receipt row of its own (a pantry confirm) …
const grocery = [{ qty: 1, description: "a grocery shop", amount: 22.4 }];
const own = (await pool.query(
`INSERT INTO expense_metadata (transaction_id, source, order_reference, merchant_normalized, line_items) VALUES ($1, 'pantry', $2, 'Woolworths', $3::jsonb) RETURNING id`,
[t, uuid(), JSON.stringify(grocery)]
)).rows[0].id as number;
// … and is also the card leg of the Uber Eats order being enriched.
const delivery = (await pool.query(
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, merchant_normalized, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, 'Burgertory (Werribee)', '[]'::jsonb) RETURNING id`,
[t, uuid()]
)).rows[0].id as number;
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ status: "enriched", transaction_id: t });
const rows = await metaRows();
expect(rows.find((r) => r.id === delivery)!.line_items.map((i) => i.description)).toEqual(["Michael", "Joel"]);
expect(rows.find((r) => r.id === own)!.line_items).toEqual(grocery);
});
it("an upload with no prices never replaces a receipt that already has items", async () => {
const priceless = { ...burgertory, subtotal: null, total: 22.4, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
modelReply = priceless;
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ kind: "held", status: "held" });
expect(body.text).toMatch(/left the existing items alone/);
expect(await lineItemsOf(txn)).toEqual(already);
// …but an EMPTY receipt is exactly what this slice exists to fill.
const empty = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-20", amount: 22.4 });
modelReply = { ...priceless, order_date: "2026-08-20" };
const second = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(second).toMatchObject({ status: "enriched", transaction_id: empty });
expect((await lineItemsOf(empty)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
}); });
it("food + no match → candidates (same merchant ±3 days, any amount) with Use-this / Create buttons; NEVER auto-creates", async () => { it("food + no match → candidates (same merchant ±3 days, any amount) with Use-this / Create buttons; NEVER auto-creates", async () => {
@@ -552,13 +322,7 @@ describe("POST /api/intake — food (enrich vs candidates)", () => {
const id = uuid(); const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json(); const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("needs_header"); expect(body.status).toBe("needs_header");
// Asserts BOTH halves of needsHeaderCard(): that it names what is actually expect(body.text).toMatch(/share the top of the screen/);
// missing (total, not a blanket "total or date"), and that a uber_eats
// source gets the share-more-of-the-screen advice rather than the in-store
// Share-button advice. 258fe3b changed this text and updated the unit tests
// but not this one.
expect(body.text).toMatch(/I can see items but no total/);
expect(body.text).toMatch(/share the part of the screen that shows it/);
expect(await lineItemsOf(txn)).toEqual([]); expect(await lineItemsOf(txn)).toEqual([]);
expect((await batch(id)).status).toBe("needs_header"); expect((await batch(id)).status).toBe("needs_header");
}); });
@@ -627,19 +391,6 @@ describe("POST /api/intake — grocery → pantry", () => {
expect((await batch(id))).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn }); expect((await batch(id))).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn });
}); });
it("an in-store tap does not hand pantry the delivery order as its finance target", async () => {
modelReply = coles;
// The Uber Eats Coles order the email pipeline already booked.
const delivery = await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: "ubereats" });
// The same shop, same day, same money — but shared from the Coles app, so
// it carries no platform and is a different purchase.
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
const pantryCall = calls.find((c) => c.url === "http://pantry.test/api/capture")!.body as { intake: Record<string, unknown> };
expect(pantryCall.intake).toMatchObject({ finance_action: "create", finance_target_transaction_id: null, platform: null });
expect(await lineItemsOf(delivery)).toEqual([]);
});
it("grocery + no match → pantry with finance_action create (ING-10 as today)", async () => { it("grocery + no match → pantry with finance_action create (ING-10 as today)", async () => {
modelReply = coles; modelReply = coles;
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json(); const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
@@ -649,33 +400,6 @@ describe("POST /api/intake — grocery → pantry", () => {
expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" }); expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" });
}); });
it("pantry saying the shop is already on its board is SHOWN, not swallowed", async () => {
// Pantry accepts a second capture of one receipt on purpose (the first may
// have been a bad crop) and reports it rather than refusing. Dropping that
// field turned "reported" into silence — and a duplicate nobody sees is a
// duplicate that gets confirmed, which counts the same groceries twice
// because pantry's addItem accumulates. Seen live on captures 66/69.
modelReply = coles;
pantryReply = { status: 200, body: { captureId: 902, items: [1, 2, 3], alreadyCaptured: { captureId: 66, confirmed: true } } };
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
expect(body.text).toMatch(/Already on the pantry board/);
expect(body.text).toMatch(/capture 66 \(already confirmed\)/);
expect(body.text).toMatch(/capture 902/);
});
it("two ledger rows for one shop are reported, not treated as no match", async () => {
// Ambiguity is not a miss. This is the exact state that produced the
// 2026-08-18 Woolworths double-book: the shop was in the ledger twice, the
// matcher returned nothing, and the card said finance would book it.
modelReply = coles;
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
expect(body.text).toMatch(/found more than one row it could be/);
});
it("pantry 409 → duplicate", async () => { it("pantry 409 → duplicate", async () => {
modelReply = coles; modelReply = coles;
pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } }; pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } };
@@ -733,143 +457,6 @@ describe("Slack taps (C5)", () => {
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1); expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
}); });
it("intake_create re-runs the match: an order booked between the card and the tap is enriched, not duplicated", async () => {
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
// The email/order bridge books the same purchase while the card sits in
// Slack. The card's "no match" is minutes stale by the time it is pressed.
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: txn });
expect((await lineItemsOf(txn)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
expect(lastSlackUpdate().text).toMatch(/^Added 2 items/);
});
it("intake_create adopts the wallet row for the same tap instead of booking it twice", async () => {
// The normal arrival order: the notification lands within seconds of the
// tap, the screenshot is shared later. matchTransaction cannot see that row
// — its key includes the platform and a notification has none — so before
// this, Create booked the purchase a second time.
const wallet = await seedWalletRow({ merchant: "Burgertory", date: "2026-08-19", amount: 22.4 });
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: wallet });
expect((await lineItemsOf(wallet)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
// The suburb the notification supplied is the one thing nothing else knows.
expect((await pool.query(`SELECT location FROM transactions WHERE id = $1`, [wallet])).rows[0].location).toBe("Werribee, VIC");
});
it("intake_create still creates when a same-amount row already carries a receipt", async () => {
// A row that another source fully described is a DIFFERENT purchase that
// happens to share an amount — adopting it would move one order's items
// onto another order's row.
const other = await seedOrder({ merchant: "Someone Else", date: "2026-08-19", amount: 22.4, platform: null });
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [other, JSON.stringify([{ description: "theirs", qty: 1, amount: 22.4 }])]);
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(2);
});
it("two pending cards for one order book it ONCE", async () => {
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
expect((await batch(a)).status).toBe("candidates");
expect((await batch(b)).status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: a }));
await slackPOST(slackPress("intake_create", { batch_id: b }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect((await batch(b)).transaction_id).toBe((await batch(a)).transaction_id);
});
it("a sibling already resolved to the pantry board makes Create a duplicate, not a second row", async () => {
// Two shares of one receipt, resolved out of order: nothing the second card
// can match on, because the first one never produced a transaction.
modelReply = { ...coles, merchant: "Romeo's Food Hall" };
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "other", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
await intakePOST(multipart({ source: "other", batch_id: a }, [{ name: "Receipt_18Aug2026.pdf", type: "application/pdf", bytes: COLES }]));
expect((await batch(a)).status).toBe("ask");
await slackPOST(slackPress("intake_grocery", { batch_id: a }));
expect((await batch(a)).status).toBe("forwarded_to_pantry");
await slackPOST(slackPress("intake_meal", { batch_id: b }));
expect((await batch(b)).status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: b }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
expect(await batch(b)).toMatchObject({ status: "duplicate", capture_id: 901 });
expect(lastSlackUpdate().text).toMatch(/^Already handled/);
});
it("two taps that land together still book one transaction, and neither reports an error", async () => {
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: id })), slackPOST(slackPress("intake_create", { batch_id: id }))]);
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
const texts = calls.filter((c) => c.url.includes("hooks.slack.com")).map((c) => (c.body as { text: string }).text);
expect(texts.every((t) => !/Couldn't process/.test(t))).toBe(true);
});
it("a foreign receipt is booked in its own currency, with no invented AUD figure", async () => {
modelReply = { ...burgertory, currency: "USD" };
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await slackPOST(slackPress("intake_create", { batch_id: id }));
const t = (await pool.query(`SELECT amount::text, amount_aud, foreign_currency_amount::text, foreign_currency_code FROM transactions`)).rows[0];
expect(t).toEqual({ amount: "22.40", amount_aud: null, foreign_currency_amount: "22.40", foreign_currency_code: "USD" });
expect(lastSlackUpdate().text).toMatch(/USD 22\.40/);
// OrderDetails prefers expense_metadata.currency, so a hard-coded 'AUD'
// here printed "$" on a USD order's receipt panel.
expect((await pool.query(`SELECT currency FROM expense_metadata`)).rows[0].currency).toBe("USD");
});
it("the hold applies to the Create tap's re-match, not only to the upload path", async () => {
modelReply = { ...burgertory, subtotal: null, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
// Booked WITH its items between the card and the tap. The re-match walked
// past the guard routeFood applies, so the same upload destroyed the same
// receipt depending on which button reached it.
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect(await lineItemsOf(txn)).toEqual(already);
expect(await batch(id)).toMatchObject({ status: "held" });
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
});
it("two DIFFERENT batches for one purchase, tapped together, book it once", async () => {
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
// Their source_refs differ, so ON CONFLICT cannot save this; both taps
// clear their match and duplicate checks before either inserts.
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: a })), slackPOST(slackPress("intake_create", { batch_id: b }))]);
const rows = (await pool.query(`SELECT id FROM transactions`)).rows;
expect(rows).toHaveLength(1);
expect((await batch(a)).transaction_id).toBe(rows[0].id);
expect((await batch(b)).transaction_id).toBe(rows[0].id);
});
it("unknown shared_by falls back to the default owner — never a NULL owner", async () => { it("unknown shared_by falls back to the default owner — never a NULL owner", async () => {
const id = uuid(); const id = uuid();
await intakePOST(multipart({ source: "uber_eats", shared_by: "nobody@example.test", batch_id: id }, [pdf])); await intakePOST(multipart({ source: "uber_eats", shared_by: "nobody@example.test", batch_id: id }, [pdf]));
@@ -1,175 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
/**
* Manually resolving a duplicate the automation cannot see.
*
* Until this route existed there was no way to merge or remove a transaction at
* all: `superseded_by_id` was read everywhere and written nowhere in
* application code, having been set exactly once by migration 0023. The only
* merge path was the reconcile queue, whose candidate list is bounded by an
* amount band -- so SP PEAKDESIGN (notified $42.44, charged $43.71, the card's
* 3% foreign transaction fee) had no offerable twin.
*
* The refusals matter more than the happy paths, exactly as in
* auto-reconcile.test.ts: this writes to columns that hide a row from every
* figure, so a wrong call does not error, it quietly stops counting money.
*/
let pool: Pool;
let ownerId: number;
let otherId: number;
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
const call = (id: number, body: unknown) => merge(req(body), { params: Promise.resolve({ id: String(id) }) });
// uq_statement_identity is (bank_name, account_number, billing_end_date), so
// every statement in a test needs its own account or the second insert collides.
let acctSeq = 0;
async function statementLine(amount: number, date: string, description: string) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, ownerId, `4444-0000-0000-${String(1000 + acctSeq++)}`]
);
const t = await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
[st.rows[0].id, date, description, amount, ownerId]
);
return t.rows[0].id as number;
}
beforeAll(async () => {
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId, otherId } = await seedParticipants(pool));
});
describe("reconcile mode", () => {
it("folds a wallet row into its statement twin even though the amounts differ by the FX fee", async () => {
const wallet = await insertTransaction(pool, ownerId, {
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
});
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
const res = await call(wallet, { other_id: stmt, mode: "reconcile" });
expect(res.status ?? 200).toBe(200);
const r = await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [wallet]);
expect(r.rows[0].reconciled_with_id).toBe(stmt);
});
it("refuses when the target is not a statement row", async () => {
const a = await insertTransaction(pool, ownerId, { amount: 10 });
const b = await insertTransaction(pool, ownerId, { amount: 10 });
const res = await call(a, { other_id: b, mode: "reconcile" });
expect(res.status).toBe(400);
});
it("refuses when the source is itself a statement row", async () => {
const s1 = await statementLine(10, "2026-08-01", "A");
const s2 = await statementLine(10, "2026-08-02", "B");
const res = await call(s1, { other_id: s2, mode: "reconcile" });
expect(res.status).toBe(400);
});
});
describe("supersede mode", () => {
it("carries splits onto the survivor before hiding the loser", async () => {
// The whole reason supersede moves curation: `superseded_by_id` removes the
// row from every figure, so a split left behind stops counting silently and
// a settled debt reappears as owed.
const loser = await insertTransaction(pool, ownerId, { amount: 80, description: "DUP" });
const survivor = await insertTransaction(pool, ownerId, { amount: 80, description: "KEEP" });
await pool.query(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
VALUES ($1,$2,50,true), ($1,$3,50,false)`,
[loser, ownerId, otherId]
);
const res = await call(loser, { other_id: survivor, mode: "supersede" });
expect(res.status ?? 200).toBe(200);
const onLoser = await pool.query(`SELECT count(*)::int c FROM transaction_splits WHERE transaction_id=$1`, [loser]);
const onSurvivor = await pool.query(
`SELECT participant_id, share_percent, settled FROM transaction_splits
WHERE transaction_id=$1 ORDER BY participant_id`, [survivor]
);
expect(onLoser.rows[0].c).toBe(0);
expect(onSurvivor.rows).toHaveLength(2);
// settled must survive the move -- a delete-and-recreate defaults it to false.
expect(onSurvivor.rows.map((r) => r.settled)).toContain(true);
const marked = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [loser]);
expect(marked.rows[0].superseded_by_id).toBe(survivor);
});
it("works between two statement rows, which reconcile cannot express", async () => {
const s1 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
const s2 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
const res = await call(s1, { other_id: s2, mode: "supersede" });
expect(res.status ?? 200).toBe(200);
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [s1]);
expect(r.rows[0].superseded_by_id).toBe(s2);
});
it("refuses to hide a statement row behind a statementless one", async () => {
// The wrong direction, and silently wrong: a wallet row holds the amount
// BEFORE the card's foreign transaction fee, so burying the statement line
// under it leaves the ledger holding $42.44 for a $43.71 charge.
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
const wallet = await insertTransaction(pool, ownerId, {
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
});
const res = await call(stmt, { other_id: wallet, mode: "supersede" });
expect(res.status).toBe(400);
// and the statement row is untouched
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [stmt]);
expect(r.rows[0].superseded_by_id).toBeNull();
});
it("refuses to bury a row under one that is already superseded", async () => {
const a = await insertTransaction(pool, ownerId, { amount: 5 });
const b = await insertTransaction(pool, ownerId, { amount: 5 });
const c = await insertTransaction(pool, ownerId, { amount: 5 });
await call(b, { other_id: c, mode: "supersede" }); // b -> c
const res = await call(a, { other_id: b, mode: "supersede" }); // a -> b, but b is buried
expect(res.status).toBe(409);
});
});
describe("refusals common to both modes", () => {
it("refuses a transaction merging into itself", async () => {
const a = await insertTransaction(pool, ownerId, { amount: 1 });
const res = await call(a, { other_id: a, mode: "supersede" });
expect(res.status).toBe(400);
});
it("refuses a row that is already reconciled", async () => {
const wallet = await insertTransaction(pool, ownerId, { amount: 12, transaction_date: "2026-08-10" });
const stmt = await statementLine(12, "2026-08-10", "X");
await call(wallet, { other_id: stmt, mode: "reconcile" });
const other = await insertTransaction(pool, ownerId, { amount: 12 });
const res = await call(wallet, { other_id: other, mode: "supersede" });
expect(res.status).toBe(409);
});
it("refuses a transaction belonging to someone else", async () => {
const mine = await insertTransaction(pool, ownerId, { amount: 7 });
const theirs = await insertTransaction(pool, otherId, { amount: 7 });
const res = await call(mine, { other_id: theirs, mode: "supersede" });
expect(res.status).toBe(404);
});
});
@@ -1,153 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants } from "./helpers";
/**
* A receipt arriving after the wallet notification must ENRICH that row, not
* create a second one.
*
* The case this is built from is real. Woolworths $103.30 on 2026-08-18: the
* wallet event landed at 09:00:14 on the 18th, and nothing created a
* transaction until 20:25:06 on the 19th -- 35h25m later -- at which point the
* receipt path made its own row from scratch. One shop, two rows, and the
* notification's contribution (the instrument, and the suburb) thrown away.
*
* Sources for one purchase arrive in sequence, not in competition, so the rule
* is: the receipt read the register tape and wins on what was bought; anything
* only the notification knew survives untouched.
*/
const TOKEN = "test-receipt-token";
let pool: Pool;
let ownerId: number;
let ingest: typeof import("../../app/api/receipts/ingest/route").POST;
const ingestReq = (body: unknown) =>
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
/** A row shaped the way the wallet lane writes one: statementless, no line
* items, carrying the suburb the notification supplied. */
async function walletRow(amount: number, date: string, merchant: string, location: string) {
const t = await pool.query(
`INSERT INTO transactions
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
transaction_type, category, payment_method, merchant_name, merchant_normalized,
location, source, source_ref, row_index)
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', 'card', $3, $3, $5, 'wallet', $6, 0)
RETURNING id`,
[ownerId, date, merchant, amount, location, `evt-${amount}-${date}`]
);
const id = t.rows[0].id as number;
await pool.query(
`INSERT INTO expense_metadata
(transaction_id, source, merchant_normalized, amount, transaction_date,
currency, line_items, payment_method_detail, flags)
VALUES ($1, 'wallet', $2, $3, $4, 'AUD', '[]'::jsonb, 'Visa', '[]'::jsonb)`,
[id, merchant, amount, date]
);
return id;
}
const receipt = (amount: number, date: string) => ({
receipt_uid: `woolworths:3345:${amount}:2026-08-18`,
capture_event_id: 900,
merchant_name: "Woolworths",
store_detail: "Wyndham Vale",
transaction_date: date,
total: amount,
tender_legs: [
{ leg_index: 1, amount, card_last4: "6078", card_product: "VISA PREPAID", class: "gift_card" },
],
line_items: [
{ description: "Milk 2L", qty: 1, unit: "ea", amount: 3.5, category: "dairy" },
{ description: "Bread", qty: 1, unit: "ea", amount: 4.0, category: "bakery" },
],
});
beforeAll(async () => {
process.env.RECEIPT_INGEST_TOKEN = TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: ingest } = await import("../../app/api/receipts/ingest/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
const txCount = async () =>
Number((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n);
describe("a receipt meeting the wallet row for the same tap", () => {
it("adopts it instead of creating a second transaction", async () => {
const walletId = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
expect(await txCount()).toBe(1);
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
expect(await txCount()).toBe(1);
const meta = await pool.query(
`SELECT transaction_id, jsonb_array_length(line_items) AS items, source,
payment_method_detail, card_last4
FROM expense_metadata`
);
expect(meta.rowCount).toBe(1);
expect(meta.rows[0].transaction_id).toBe(walletId);
expect(meta.rows[0].items).toBe(2);
expect(meta.rows[0].source).toBe("pantry");
// The receipt read the card off the tape; the notification only saw "Visa".
expect(meta.rows[0].payment_method_detail).toBe("VISA PREPAID");
expect(meta.rows[0].card_last4).toBe("6078");
});
it("keeps the suburb only the notification knew", async () => {
await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
const t = await pool.query(`SELECT location, merchant_normalized, category FROM transactions`);
// `location` is absent from the adoption UPDATE's column list on purpose:
// the receipt has no suburb, so it must not be able to erase one.
expect(t.rows[0].location).toBe("Wyndham Vale, VIC");
// ...while what the receipt does know wins.
expect(t.rows[0].merchant_normalized).toBe("Woolworths");
expect(t.rows[0].category).toBe("groceries");
});
it("creates its own row when there is no wallet row to adopt", async () => {
await ingest(ingestReq(receipt(58.4, "2026-08-18")));
expect(await txCount()).toBe(1);
const meta = await pool.query(`SELECT source, jsonb_array_length(line_items) AS items FROM expense_metadata`);
expect(meta.rows[0].source).toBe("pantry");
expect(meta.rows[0].items).toBe(2);
});
it("refuses to adopt when two candidates share the amount", async () => {
// Two taps of the same value in the window: adopting either would move this
// shop's line items onto a purchase it did not describe.
await walletRow(103.3, "2026-08-17", "Woolworths", "Wyndham Vale, VIC");
await walletRow(103.3, "2026-08-18", "Coles", "Manor Lakes, VIC");
expect(await txCount()).toBe(2);
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
// Falls back to creating its own row rather than guessing.
expect(await txCount()).toBe(3);
});
it("refuses to adopt a row that already carries line items", async () => {
const other = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
await pool.query(
`UPDATE expense_metadata SET line_items = '[{"description":"Something else","amount":1}]'::jsonb
WHERE transaction_id = $1`, [other]
);
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
// A fully-described row is a different purchase that happens to share an amount.
expect(await txCount()).toBe(2);
});
});
@@ -1,86 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
import { getPendingReconciliations } from "../../lib/queries";
/**
* The reconcile queue must not offer what is already resolved.
*
* A superseded row is hidden from every figure (EXCLUDE_RECONCILED_SOURCE), so
* listing it is not merely noise: on 2026-10-03 the queue showed the duplicate
* wallet half 7037, superseded under its twin 7038, as a second "Digidirect"
* with the same statement line offered as its match. Accepting it there would
* have spent that line on a hidden row and left the real one unmatched forever.
* findAutoMatches() always had both filters; this query had neither.
*/
let pool: Pool;
let ownerId: number;
let acctSeq = 0;
async function statementLine(amount: number, date: string, description: string) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('q.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
);
const t = await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
[st.rows[0].id, date, description, amount, ownerId]
);
return t.rows[0].id as number;
}
beforeAll(() => { pool = createPool(); });
beforeEach(async () => {
await resetDB(pool);
({ ownerId } = await seedParticipants(pool));
});
describe("getPendingReconciliations", () => {
it("does not list a superseded manual row", async () => {
const survivor = await insertTransaction(pool, ownerId, { description: "DIGIDIRECT", amount: 1180.05, transaction_date: "2026-08-25" });
const hidden = await insertTransaction(pool, ownerId, { description: "Digidirect", amount: 1180.05, transaction_date: "2026-08-25" });
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [survivor, hidden]);
const line = await statementLine(1180.05, "2026-08-25", "SP DIGIDIRECT STRATHFIELD");
const queue = await getPendingReconciliations(ownerId);
const ids = queue.map((r) => r.id);
expect(ids).toContain(survivor);
expect(ids).not.toContain(hidden);
// The survivor is still offered its line.
const row = queue.find((r) => r.id === survivor)!;
expect(row.matches.map((m) => m.id)).toContain(line);
});
it("does not offer a superseded statement row as a match", async () => {
const manual = await insertTransaction(pool, ownerId, { description: "Grocer", amount: 42.0, transaction_date: "2026-08-10" });
const live = await statementLine(42.0, "2026-08-10", "GROCER");
const dupe = await statementLine(42.0, "2026-08-10", "GROCER");
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [live, dupe]);
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
const offered = row.matches.map((m) => m.id);
expect(offered).toContain(live);
expect(offered).not.toContain(dupe);
});
it("offers the exact native twin of a foreign row even when its provisional AUD is far off", async () => {
// Live shape, 2026-10-09: wallet INR 112 at a provisional A$2.01; CommBank
// charged A$1.68 -- 16% under, outside the +-5% band.
const manual = await insertTransaction(pool, ownerId, { description: "SWIGGY LIMITED", amount: 2.01, transaction_date: "2026-10-04" });
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [manual]);
const twin = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
const otherRupees = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [twin]);
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 113 WHERE id = $1`, [otherRupees]);
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
const offered = row.matches.map((m) => m.id);
expect(offered).toContain(twin);
// A different native figure at the same AUD is not admitted by this path.
expect(offered).not.toContain(otherRupees);
});
});
-167
View File
@@ -1,167 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
import { EXCLUDE_RECONCILED_SOURCE } from "@/lib/analytics-sql";
/**
* A charge that never posted (migration 0040).
*
* Txn 7951: a $1,040.41 Airbnb authorisation hold on Amex, booking cancelled
* the next day, hold released. The wallet lane booked the notification as a
* purchase and no statement line ever arrived, so it sat in the reconcile queue
* forever and counted as spend everywhere. A void has no survivor, which is why
* neither reconcile nor supersede could express it.
*
* As in merge-supersede.test.ts, the refusals matter as much as the happy path:
* a void quietly stops money counting.
*/
let pool: Pool;
let ownerId: number;
let otherId: number;
let voidRoute: typeof import("../../app/api/transactions/[id]/void/route");
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
const params = (id: number) => ({ params: Promise.resolve({ id: String(id) }) });
const doVoid = (id: number, reason: unknown = "hold released, booking cancelled") =>
voidRoute.POST(req({ reason }), params(id));
let acctSeq = 0;
async function statementLine(amount: number, date: string, description: string) {
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
);
const t = await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
[st.rows[0].id, date, description, amount, ownerId]
);
return t.rows[0].id as number;
}
/** What every "count each purchase once" query sees. */
async function countedIds(): Promise<number[]> {
const r = await pool.query(
`SELECT t.id FROM transactions t WHERE ${EXCLUDE_RECONCILED_SOURCE} ORDER BY t.id`
);
return r.rows.map((x) => x.id);
}
beforeAll(async () => {
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
voidRoute = await import("../../app/api/transactions/[id]/void/route");
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId, otherId } = await seedParticipants(pool));
});
describe("voiding a charge that never posted", () => {
it("drops the row out of every figure and the reconcile queue, but keeps it", async () => {
const { getPendingReconciliations } = await import("@/lib/queries");
const hold = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 1040.41 });
const real = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 529.05 });
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).toContain(hold);
const res = await doVoid(hold);
expect(res.status ?? 200).toBe(200);
expect(await countedIds()).toEqual([real]);
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).not.toContain(hold);
const r = await pool.query(`SELECT voided_at, void_reason FROM transactions WHERE id = $1`, [hold]);
expect(r.rows[0].voided_at).not.toBeNull();
expect(r.rows[0].void_reason).toBe("hold released, booking cancelled");
});
it("can be undone", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 1040.41 });
await doVoid(hold);
const res = await voidRoute.DELETE(req({}), params(hold));
expect(res.status ?? 200).toBe(200);
expect(await countedIds()).toEqual([hold]);
});
it("requires a reason", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 10 });
expect((await doVoid(hold, " ")).status).toBe(400);
expect((await doVoid(hold, null)).status).toBe(400);
expect(await countedIds()).toEqual([hold]);
});
it("refuses a statement line -- that charge posted", async () => {
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
expect((await doVoid(stmt)).status).toBe(409);
expect(await countedIds()).toEqual([stmt]);
});
it("refuses a reconciled row -- it found its statement line, so it posted", async () => {
const wallet = await insertTransaction(pool, ownerId, { amount: 50, transaction_date: "2026-09-10" });
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [stmt, wallet]);
expect((await doVoid(wallet)).status).toBe(409);
});
it("refuses a row with a settled split -- someone already paid their share", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 80 });
await pool.query(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
VALUES ($1, $2, 50, true)`,
[hold, otherId]
);
expect((await doVoid(hold)).status).toBe(409);
expect(await countedIds()).toEqual([hold]);
});
it("is enforced by the database, not only the route", async () => {
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
await expect(
pool.query(`UPDATE transactions SET voided_at = now(), void_reason = 'x' WHERE id = $1`, [stmt])
).rejects.toThrow(/transactions_void_statementless/);
const manual = await insertTransaction(pool, ownerId, { amount: 5 });
await expect(
pool.query(`UPDATE transactions SET voided_at = now() WHERE id = $1`, [manual])
).rejects.toThrow(/transactions_void_has_reason/);
});
});
describe("the matchers never use a voided row", () => {
it("auto-reconcile does not pair a voided row with a statement line of the same amount", async () => {
const { findAutoMatches } = await import("@/lib/auto-reconcile");
const hold = await insertTransaction(pool, ownerId, { amount: 120, transaction_date: "2026-09-10" });
await statementLine(120, "2026-09-11", "SOMETHING");
expect((await findAutoMatches(ownerId)).map((m) => m.manual_id)).toContain(hold);
await doVoid(hold);
expect(await findAutoMatches(ownerId)).toEqual([]);
});
it("a receipt for a rebooking at the same amount adopts the live row, not the released hold", async () => {
const { findAdoptableTransaction } = await import("@/lib/auto-reconcile");
const hold = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-10" });
const rebooked = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-11" });
// Two candidates: the adopter refuses rather than guess.
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBeNull();
await doVoid(hold);
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBe(rebooked);
});
it("merge refuses a voided row on either side", async () => {
const hold = await insertTransaction(pool, ownerId, { amount: 70, transaction_date: "2026-09-10" });
const stmt = await statementLine(70, "2026-09-10", "SOMETHING");
await doVoid(hold);
const res = await merge(req({ other_id: stmt, mode: "reconcile" }), params(hold));
expect(res.status).toBe(409);
});
});
@@ -1,176 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants } from "./helpers";
/**
* The mirror of receipt-adopts-wallet-row: a wallet notification arriving AFTER
* something else already booked the purchase must fold into that row.
*
* Built from the failure, not from the design. The 2026-08-18 Woolworths shop
* ($103.30, Visa gift card) reached the ledger twice: txn 6858 from the pantry
* receipt on the 19th, and txn 6927 from the wallet lane on the 22nd, when the
* first 21 wallet events were backfilled days after their taps. The sequencing
* the lane is designed around -- wallet first, receipt second -- is a tendency,
* and every writer has to survive it not holding.
*
* Two rules are load-bearing here and are tested separately below:
* - the merchant must agree, because losing the "no line items" guard that
* protects the other direction means amount+date alone could swallow a tap
* into an unrelated purchase of the same value
* - a gift-card tap is `credits`, because `card` means "a statement line is
* coming" and the reconcile queue filters on exactly that
*/
const TOKEN = "test-wallet-token";
let pool: Pool;
let ownerId: number;
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
const req = (body: unknown) =>
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
/** A row shaped the way the pantry receipt lane writes one: statementless,
* itemised, no location (nothing but a notification supplies a suburb). */
async function receiptRow(amount: number, date: string, merchant: string, opts: { items?: number; meta?: boolean } = {}) {
const { items = 3, meta = true } = opts;
const t = await pool.query(
`INSERT INTO transactions
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
transaction_type, category, payment_method, merchant_name, merchant_normalized, row_index)
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'groceries', 'credits', $3, $3, 0)
RETURNING id`,
[ownerId, date, merchant, amount]
);
const id = t.rows[0].id as number;
if (meta) {
const lineItems = Array.from({ length: items }, (_, i) => ({ description: `item ${i}`, qty: 1, amount: 1 }));
await pool.query(
`INSERT INTO expense_metadata
(transaction_id, source, merchant_normalized, amount, transaction_date, currency, line_items, flags)
VALUES ($1, 'pantry', $2, $3, $4, 'AUD', $5::jsonb, '[]'::jsonb)`,
[id, merchant, amount, date, JSON.stringify(lineItems)]
);
}
return id;
}
/** The real event 80: the Apple Wallet half of a Visa gift-card tap. */
const woolworthsGiftCard = {
event_id: 80, kind: "transaction", direction: "unknown", amount: 103.3,
amount_text: "$103.30", currency: "$?", merchant: "Woolworths", issuer: "Visa",
notif_app: "Wallet", notif_title: "Visa Gift Card",
notif_subtitle: "Woolworths, Wyndham Vale, VIC",
occurred_at: "2026-08-18T09:00:14+10:00",
};
const rows = async () =>
(await pool.query(
`SELECT id, amount, merchant_normalized, location, source, source_ref, payment_method
FROM transactions ORDER BY id`
)).rows;
beforeAll(async () => {
process.env.WALLET_INGEST_TOKEN = TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
});
beforeEach(async () => {
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
describe("a wallet notification for a purchase the ledger already holds", () => {
it("adopts the receipt's row instead of creating a second one", async () => {
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
const body = await (await ingest(req(woolworthsGiftCard))).json();
expect(body.skipped).toBe("adopted");
expect(body.transactionId).toBe(existing);
expect(await rows()).toHaveLength(1);
});
it("gives the adopted row the suburb, and leaves what the receipt read alone", async () => {
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
await ingest(req(woolworthsGiftCard));
const [t] = await rows();
// Wallet's own field: nothing else on the estate writes a suburb.
expect(t.location).toBe("Wyndham Vale, VIC");
// The receipt read the register tape. The notification does not get to
// rename the merchant to its own shorter string.
expect(t.merchant_normalized).toBe("WOOLWORTHS 3345");
expect(t.id).toBe(existing);
const [m] = (await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
expect(m.source).toBe("pantry");
expect(m.line_items).toHaveLength(3);
expect(m.payment_method_detail).toBe("Visa");
expect(m.flags).toContain("wallet_evt:80");
});
it("is idempotent when the interpreter replays every event", async () => {
// Bumping prompt_version re-interprets and re-posts the whole capture
// history on purpose -- that is the replay mechanism. An adopted row keeps
// no wallet source_ref of its own, so the flag is what makes replay safe.
await ingest(req(woolworthsGiftCard));
const second = await (await ingest(req(woolworthsGiftCard))).json();
expect(second.skipped).toBe("already_ingested");
expect(await rows()).toHaveLength(1);
});
it("writes its own row when the merchant disagrees", async () => {
// Same money, same day, different shop. Folding into it would make the tap
// vanish -- worse than a duplicate, which is at least visible in the queue.
await receiptRow(103.3, "2026-08-18", "Bunnings Werribee");
const body = await (await ingest(req(woolworthsGiftCard))).json();
expect(body.skipped).toBeUndefined();
expect(await rows()).toHaveLength(2);
});
it("writes its own row when two candidates share the amount", async () => {
await receiptRow(103.3, "2026-08-18", "Woolworths Wyndham Vale");
await receiptRow(103.3, "2026-08-18", "Woolworths Point Cook");
const body = await (await ingest(req(woolworthsGiftCard))).json();
expect(body.skipped).toBeUndefined();
expect(await rows()).toHaveLength(3);
});
it("creates the metadata row when the adopted transaction has none", async () => {
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345", { meta: false });
await ingest(req(woolworthsGiftCard));
const meta = (await pool.query(`SELECT * FROM expense_metadata`)).rows;
expect(meta).toHaveLength(1);
expect(meta[0].transaction_id).toBe(existing);
expect(meta[0].card_last4).toBeNull();
expect(meta[0].flags).toContain("wallet_evt:80");
});
});
describe("a stored-value instrument", () => {
it("is booked as credits, so it never enters the reconcile queue", async () => {
const body = await (await ingest(req(woolworthsGiftCard))).json();
const [t] = await rows();
expect(body.transactionId).toBeTruthy();
// 'card' would park it forever: findAutoMatches and the queue both exclude
// 'credits' and 'cash' precisely because no statement line is coming.
expect(t.payment_method).toBe("credits");
expect(body.flags).toContain("stored_value_instrument");
});
it("leaves an ordinary card tap as card", async () => {
const body = await (await ingest(req({ ...woolworthsGiftCard, notif_title: "Payment", issuer: "Mastercard", notif_app: "Westpac" }))).json();
const [t] = await rows();
expect(t.payment_method).toBe("card");
expect(body.flags).not.toContain("stored_value_instrument");
});
});
@@ -1,403 +0,0 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import { Pool } from "pg";
import { createPool, resetDB, seedParticipants } from "./helpers";
/**
* Wallet notifications becoming transactions.
*
* Every fixture is a real captured event from capture.wallet_events
* (2026-08-13 .. 08-22). The rules being tested are the ones the design turned
* on, so each has a reason rather than being a shape check:
*
* - a transaction for EVERY tap, because overseas is where live spend matters
* - the pair is MERGED, never deduped: the issuer half has the card tail, the
* Wallet half has the merchant and suburb, neither alone is complete
* - the NATIVE amount is what gets stored for a foreign charge, because the
* AUD figure is the bank's own conversion and cannot be predicted
* - a bare "$" is recorded as unknown, never assumed AUD
*/
const TOKEN = "test-wallet-token";
let pool: Pool;
let ownerId: number;
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
const req = (body: unknown) =>
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
const noAuth = (body: unknown) =>
({ headers: { get: () => null }, json: async () => body }) as never;
// --- real captured events -------------------------------------------------
const breadworldBank = {
event_id: 64, kind: "transaction", direction: "debit", amount: 13.5,
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
card_tail: "8032", issuer: "Mastercard", notif_app: "Westpac",
notif_body: "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU",
occurred_at: "2026-08-13T08:01:28+10:00",
};
const breadworldWallet = {
event_id: 65, kind: "transaction", direction: "unknown", amount: 13.5,
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
notif_app: "Wallet", notif_subtitle: "Breadworld Bakery Cafe, Wyndham Vale, VIC",
occurred_at: "2026-08-13T08:01:35+10:00",
};
const amexRewardPay = {
event_id: 87, kind: "transaction", direction: "debit", amount: 5103.49,
amount_text: "A$5,103.49", currency: "AUD", merchant: "REWARDPAY",
card_tail: "01000", issuer: "American Express", notif_app: "Amex AU",
notif_body: "You have made a A$5,103.49 charge on your American Express® Card ending in 01000 at REWARDPAY.",
occurred_at: "2026-08-20T11:54:43+10:00",
};
const ebayOrderPlaced = {
event_id: 66, kind: "not_transaction", direction: "unknown", amount: null,
notif_app: "Wallet", notif_body: "eBay has received your order.",
occurred_at: "2026-08-13T09:45:55+10:00",
};
/** Only the rows this lane wrote -- some tests seed statement history for the
* rate reference, and those must not be mistaken for wallet output. */
const txns = async () =>
(await pool.query(
`SELECT id, amount, amount_aud, merchant_normalized, location, source, source_ref,
foreign_currency_amount, foreign_currency_code, payment_method, statement_id
FROM transactions WHERE source = 'wallet' ORDER BY id`
)).rows;
const metas = async () =>
(await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
beforeAll(async () => {
process.env.WALLET_INGEST_TOKEN = TOKEN;
pool = createPool();
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
}));
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
});
beforeEach(async () => {
// No network in tests. The ECB fallback treats any fetch failure as "no
// rate", which is what every pre-existing currency test assumes; the one test
// about the fallback stubs a rate of its own.
vi.stubGlobal("fetch", async () => { throw new Error("no network in tests"); });
await resetDB(pool);
await pool.query("DELETE FROM expense_metadata");
({ ownerId } = await seedParticipants(pool));
});
describe("auth", () => {
it("fails closed without the token", async () => {
expect((await ingest(noAuth(breadworldBank))).status).toBe(401);
expect(await txns()).toHaveLength(0);
});
});
describe("what reaches the ledger", () => {
it("creates a statementless card transaction for a tap", async () => {
const body = await (await ingest(req(breadworldBank))).json();
expect(body.transactionId).toBeTruthy();
const [t] = await txns();
expect(Number(t.amount)).toBe(13.5);
expect(t.source).toBe("wallet");
expect(t.source_ref).toBe("evt:64");
// Statementless + card is exactly what puts it in the reconcile queue, with
// no extra wiring: that is how the statement eventually confirms it.
expect(t.statement_id).toBeNull();
expect(t.payment_method).toBe("card");
});
it("ignores an eBay order-lifecycle notification", async () => {
// 10 of the 13 non-transactions in the capture window are these.
const body = await (await ingest(req(ebayOrderPlaced))).json();
expect(body.skipped).toBe("not_a_transaction");
expect(await txns()).toHaveLength(0);
});
it("is idempotent on the event id", async () => {
await ingest(req(breadworldBank));
const second = await (await ingest(req(breadworldBank))).json();
expect(second.skipped).toBe("already_ingested");
expect(await txns()).toHaveLength(1);
});
});
describe("the notification pair", () => {
it("merges the two halves into ONE transaction, keeping both contributions", async () => {
await ingest(req(breadworldBank)); // has the card tail
await ingest(req(breadworldWallet)); // has the suburb
const rows = await txns();
expect(rows).toHaveLength(1);
// The Wallet half supplies the suburb; the issuer half never had one.
expect(rows[0].location).toBe("Wyndham Vale, VIC");
const [m] = await metas();
// ...and the issuer half's card tail survives the merge.
expect(m.card_last4).toBe("8032");
expect(m.payment_method_detail).toBe("Mastercard");
});
it("merges in either arrival order", async () => {
await ingest(req(breadworldWallet));
await ingest(req(breadworldBank));
expect(await txns()).toHaveLength(1);
const [m] = await metas();
expect(m.card_last4).toBe("8032");
const [t] = await txns();
expect(t.location).toBe("Wyndham Vale, VIC");
});
});
describe("the notification pair is recognised by time and app, not amount alone", () => {
it("makes ONE row when both halves are posted at the same moment", async () => {
// n8n posts a poll's items concurrently, and both halves of a tap usually
// land in the same poll. Before the lock, each ran its pair lookup before
// the other inserted: 11 taps became 22 rows, identical created_at on each.
await Promise.all([ingest(req(breadworldBank)), ingest(req(breadworldWallet))]);
expect(await txns()).toHaveLength(1);
const [m] = await metas();
expect(m.card_last4).toBe("8032");
});
it("keeps two same-amount charges hours apart as two purchases", async () => {
// Event 275 ($50 at A MEDIUM) merged into the UNICEF $50 donation from 17
// hours earlier and renamed it, because the old window was +/-1 day.
await ingest(req({
...amexRewardPay, event_id: 267, amount: 50, amount_text: "A$50.00", merchant: "UNICEF",
notif_body: "You have made a A$50.00 charge on your American Express® Card ending in 01000 at UNICEF.",
occurred_at: "2026-10-01T11:02:40+10:00",
}));
await ingest(req({
...amexRewardPay, event_id: 275, amount: 50, amount_text: "$50.00", merchant: "A MEDIUM",
notif_body: "You have made a $50.00 charge on your American Express® Card ending in 01000 at A MEDIUM.",
occurred_at: "2026-10-02T04:06:49+10:00",
}));
const rows = await txns();
expect(rows).toHaveLength(2);
expect(rows.map((r) => r.merchant_normalized).sort()).toEqual(["A MEDIUM", "UNICEF"]);
});
it("never pairs two notifications from the SAME app, however close", async () => {
// A pair is the issuer half plus the Wallet half. Two from one app are two
// charges -- two $20 coffees bought together, say.
await ingest(req(breadworldBank));
await ingest(req({ ...breadworldBank, event_id: 640, occurred_at: "2026-08-13T08:01:40+10:00" }));
expect(await txns()).toHaveLength(2);
});
it("records a merged half, so a replay cannot merge it again", async () => {
await ingest(req(breadworldBank));
await ingest(req(breadworldWallet));
const again = await (await ingest(req(breadworldWallet))).json();
expect(again.skipped).toBe("already_ingested");
const [m] = await metas();
expect(m.flags).toContain("wallet_evt:65");
});
});
describe("notifications that name an amount but are not purchases", () => {
it("skips an overdraft notice", async () => {
// Event 235 became a $21.21 purchase named "To avoid fees and interest,".
const body = await (await ingest(req({
event_id: 235, kind: "transaction", direction: "debit", amount: 21.21,
amount_text: "$21.21", currency: "$?", merchant: null, notif_app: "CommBank",
notif_body: "Your Smart Access 1910 is overdrawn by $21.21. To avoid fees and interest, bring your account to positive by midnight 21 Sep (Syd/Melb time).",
occurred_at: "2026-09-22T09:00:18+10:00",
}))).json();
expect(body.skipped).toBe("account_notice");
expect(await txns()).toHaveLength(0);
});
it("skips a transit-card fare, because the card's load is the spend", async () => {
// Event 200: the Octopus load is on the bank statement; each fare draws it
// down. 11 of 13 fares were already not_transaction; this makes the other 2 agree.
const body = await (await ingest(req({
event_id: 200, kind: "transaction", direction: "debit", amount: 105,
amount_text: "HKD 105.00", currency: "HKD", merchant: "Octopus", notif_app: "Wallet",
notif_title: "Octopus", notif_subtitle: "HKD 105.00 for Train",
notif_body: "Your new balance is -HKD 12.60.",
occurred_at: "2026-09-14T02:00:37+10:00",
}))).json();
expect(body.skipped).toBe("stored_value_drawdown");
expect(await txns()).toHaveLength(0);
});
it("does not read a Westpac descriptor out of another bank's message", async () => {
await ingest(req({
event_id: 2350, kind: "transaction", direction: "debit", amount: 9.5,
amount_text: "$9.50", currency: "$?", merchant: null, notif_app: "CommBank",
notif_body: "Payment of $9.50 made. Thanks for banking with us.",
occurred_at: "2026-09-22T09:00:18+10:00",
}));
const [t] = await txns();
expect(t.merchant_normalized).toBe("Card purchase");
});
});
describe("dating", () => {
it("dates a tap in Melbourne local time, as the statements do", async () => {
// 21:30 UTC on 30 Sep is 07:30 on 1 Oct in Melbourne. Slicing the ISO
// string filed it a day early.
await ingest(req({ ...breadworldBank, occurred_at: "2026-09-30T21:30:00Z" }));
const { rows } = await pool.query(
`SELECT transaction_date::text AS d FROM transactions WHERE source = 'wallet'`
);
expect(rows[0].d).toBe("2026-10-01");
});
});
describe("currency", () => {
it("falls back to the ECB rate when the ledger has no history", async () => {
// Hong Kong: no HKD statement row existed, so every tap stored A$0.00.
const calls: string[] = [];
vi.stubGlobal("fetch", async (url: string) => {
calls.push(String(url));
return { ok: true, json: async () => ({ rates: { AUD: 0.19 } }) };
});
await ingest(req({
event_id: 167, kind: "transaction", direction: "debit", amount: 310.1,
amount_text: "HKD 310.10", currency: "HKD", merchant: "Wonder Taxi", notif_app: "Wallet",
notif_subtitle: "Wonder Taxi, Tsim Sha Tsui East, Kowloon",
occurred_at: "2026-09-10T08:54:00+10:00",
}));
const [t] = await txns();
expect(Number(t.amount)).toBeCloseTo(58.92, 2);
expect(Number(t.foreign_currency_amount)).toBe(310.1);
expect(calls[0]).toContain("/2026-09-10?from=HKD&to=AUD");
const [m] = await metas();
// Flagged as the mid-rate, not the bank's: it lacks the card's FX fee.
expect(m.flags).toEqual(expect.arrayContaining(["aud_provisional", "rate_ecb"]));
expect(m.flags).not.toContain("no_rate_reference");
});
it("records AUD when Amex states it explicitly", async () => {
await ingest(req(amexRewardPay));
const [t] = await txns();
expect(t.foreign_currency_code).toBeNull();
expect(Number(t.amount)).toBe(5103.49);
const [m] = await metas();
expect(m.currency).toBe("AUD");
expect(m.flags).not.toContain("currency_unverified");
});
it("flags a bare dollar sign rather than assuming AUD", async () => {
// 14 of 21 captured transaction rows are a bare "$". Assuming AUD is
// harmless at home and a 23x error on a THB tap abroad.
await ingest(req(breadworldBank));
const [m] = await metas();
expect(m.currency).toBeNull();
expect(m.flags).toContain("currency_unverified");
});
it("stores a foreign charge on the NATIVE pair, with a provisional AUD", async () => {
// Seed enough history for the median rate to exist (n >= 5).
const st = await pool.query(
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
VALUES ('s.pdf','1','Test Bank','2026-01-01','2026-01-31',$1) RETURNING id`, [ownerId]
);
for (let i = 0; i < 6; i++) {
await pool.query(
`INSERT INTO transactions (statement_id, transaction_date, description, amount,
transaction_type, owner_id, foreign_currency_amount, foreign_currency_code)
VALUES ($1,'2026-01-10','BANGKOK',4.57,'debit',$2,100,'THB')`, [st.rows[0].id, ownerId]
);
}
await ingest(req({
event_id: 999, kind: "transaction", direction: "debit", amount: 500,
amount_text: "฿500", merchant: "7-Eleven", notif_app: "Wallet",
notif_subtitle: "7-Eleven, Bangkok", occurred_at: "2026-08-20T12:00:00+10:00",
}));
const [t] = await txns();
expect(Number(t.foreign_currency_amount)).toBe(500);
expect(t.foreign_currency_code).toBe("THB");
// The native figure is exact on both sides, so it is what the statement
// will match on. amount is only ever provisional here.
expect(Number(t.amount)).toBeCloseTo(22.85, 1);
const [m] = await metas();
expect(m.flags).toContain("aud_provisional");
});
it("still records a foreign charge when there is no rate history", async () => {
await ingest(req({
event_id: 998, kind: "transaction", direction: "debit", amount: 42,
amount_text: "€42.00", merchant: "Somewhere", notif_app: "Wallet",
occurred_at: "2026-08-20T12:00:00+10:00",
}));
const [t] = await txns();
expect(t.foreign_currency_code).toBe("EUR");
const [m] = await metas();
// Says so, rather than inventing a rate.
expect(m.flags).toContain("no_rate_reference");
// And never stores the native figure as AUD (672006 VND became A$672,006).
expect(Number(t.amount)).toBe(0);
expect(Number(t.foreign_currency_amount)).toBe(42);
});
it("derives the rate from statement rows only, never from wallet rows", async () => {
// Six statementless rows at a nonsense 1:1 rate must not count as history.
for (let i = 0; i < 6; i++) {
await pool.query(
`INSERT INTO transactions (transaction_date, description, amount, transaction_type,
owner_id, foreign_currency_amount, foreign_currency_code, source, source_ref)
VALUES ('2026-01-10','VN',1000,'debit',$1,1000,'VND','wallet',$2)`,
[ownerId, `evt:seed${i}`]
);
}
await ingest(req({
event_id: 997, kind: "transaction", direction: "debit", amount: 672006,
amount_text: "672006 VND", currency: "VND", merchant: "EVISA VIETNAM", notif_app: "CommBank",
occurred_at: "2026-09-05T11:22:00+10:00",
}));
const t = (await txns()).find((r) => r.source_ref === "evt:997")!;
expect(Number(t.amount)).toBe(0);
const [m] = (await metas()).filter((r) => r.source_message_id === "evt:997");
expect(m.flags).toContain("no_rate_reference");
});
it("pairs foreign halves on the native figure, not the placeholder amount", async () => {
await ingest(req({
event_id: 996, kind: "transaction", direction: "debit", amount: 42,
amount_text: "€42.00", merchant: "Cafe A", notif_app: "Wallet",
occurred_at: "2026-08-20T12:00:00+10:00",
}));
await ingest(req({
event_id: 995, kind: "transaction", direction: "debit", amount: 17,
amount_text: "€17.00", merchant: "Cafe B", notif_app: "Wallet",
occurred_at: "2026-08-20T12:05:00+10:00",
}));
// Both have amount 0; they are different purchases and must stay two rows.
expect((await txns()).filter((r) => r.foreign_currency_code === "EUR")).toHaveLength(2);
});
});
describe("location", () => {
it("takes the suburb from the Wallet subtitle", async () => {
await ingest(req(breadworldWallet));
const [t] = await txns();
expect(t.location).toBe("Wyndham Vale, VIC");
});
it("falls back to the fixed-width descriptor, country included", async () => {
// The issuer half has no subtitle; the country code is positional.
await ingest(req({
...breadworldBank, event_id: 75, amount: 42.44,
merchant: "SP PEAKDESIGN",
// The model's own answer for this row, included because every real
// interpretation carries one -- omitting it is what let a wrong
// precedence order pass this test the first time, and ship
// "SAN FRANCISOUS" into the ledger.
location: "SAN FRANCISOUS",
notif_body: "$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS",
}));
const [t] = await txns();
expect(t.location).toBe("SAN FRANCISO US");
const [m] = await metas();
// The only foreign merchant in the whole capture window — worth surfacing.
expect(m.flags).toContain("foreign_country_US");
});
});
+2 -197
View File
@@ -3,22 +3,17 @@ import { readFileSync } from "fs";
import { resolve } from "path"; import { resolve } from "path";
import { import {
classifyInput, classifyInput,
confabulationReason,
GROCERY_PATTERN,
merchantKey, merchantKey,
merchantMatches, merchantMatches,
platformForSource, platformForSource,
PROMPT_PLACEHOLDER_MERCHANTS,
quoteShared,
reconcileItems, reconcileItems,
resolveMerchantKind, resolveMerchantKind,
textCouldBeReceipt,
toLineItems, toLineItems,
validateHeader, validateHeader,
type ExtractedOrder, type ExtractedOrder,
} from "../../lib/intake-rules"; } from "../../lib/intake-rules";
import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT, ExtractionError, liteLLMExtractor, ORDER_ITEM_PROMPT } from "../../lib/intake-extract"; import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT } from "../../lib/intake-extract";
import { askCard, candidatesCard, enrichedCard, type Block } from "../../lib/intake-blocks"; import { askCard, candidatesCard, enrichedCard } from "../../lib/intake-blocks";
/** /**
* The intake router's pure rules, against the strings real receipts print. * The intake router's pure rules, against the strings real receipts print.
@@ -90,17 +85,6 @@ describe("merchant pattern — grocery | food | ask", () => {
expect(merchantMatches("Burgertory (Werribee)", "Coles")).toBe(false); expect(merchantMatches("Burgertory (Werribee)", "Coles")).toBe(false);
expect(merchantMatches(null, "Coles")).toBe(false); expect(merchantMatches(null, "Coles")).toBe(false);
}); });
it("whole words, not substrings: `Bigals Pizza` contains `iga` and is neither a grocer nor an IGA", () => {
expect(GROCERY_PATTERN.test(merchantKey("Bigals Pizza"))).toBe(false);
expect(resolveMerchantKind("uber_eats", "Bigals Pizza")).toBe("food");
expect(merchantMatches("Bigals Pizza", "IGA")).toBe(false);
// and the grocers a boundary must not cost us
expect(resolveMerchantKind("uber_eats", "IGA Wyndham Vale")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Woolworths Group Limited")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Coles (Wyndham Vale)")).toBe("grocery");
expect(merchantMatches("IGA Wyndham Vale", "IGA")).toBe(true);
});
}); });
describe("validateHeader — nothing is written from a half-read upload", () => { describe("validateHeader — nothing is written from a half-read upload", () => {
@@ -237,183 +221,4 @@ describe("cards carry the C5 action ids and JSON values", () => {
expect(ids).toContain('"action_id":"intake_grocery"'); expect(ids).toContain('"action_id":"intake_grocery"');
expect(ids).toContain('"action_id":"intake_meal"'); expect(ids).toContain('"action_id":"intake_meal"');
}); });
it("every value the card did not write is escaped, plain_text included", () => {
const hostile = { ...c, merchant: "Ben & Jerry's <fake>" };
const { text, blocks } = candidatesCard(hostile, [
{ transaction_id: 42, description: "SQ *BEN & JERRY <AU>", transaction_date: "2026-08-17", amount: "19.90" },
]);
const sections = JSON.stringify(blocks.filter((b) => (b as Block).type === "section"));
expect(sections).toContain("Ben &amp; Jerry's &lt;fake&gt;");
expect(sections).toContain("SQ *BEN &amp; JERRY &lt;AU&gt;");
expect(text).toContain("Ben &amp; Jerry's &lt;fake&gt;");
// plain_text disables formatting, not entity decoding — &, < and > are
// control characters everywhere in a Slack payload.
const whole = JSON.stringify(blocks);
expect(whole).not.toContain("<fake>");
expect(whole).not.toContain("<AU>");
});
it("the currency is model output too, so it is escaped like any other", () => {
// parseExtraction accepts any string for `currency`; unescaped, `<@U123>`
// would be a live mention on every card the batch produces.
const { text } = enrichedCard({ ...c, currency: "<@U123>", transactionId: 7 });
expect(text).toContain("&lt;@U123&gt; 22.40");
expect(text).not.toContain("<@U123>");
});
it("a foreign receipt is not printed as dollars — header and candidate rows alike", () => {
expect(enrichedCard({ ...c, currency: "USD", transactionId: 7 }).text).toBe(
"Added 2 items to your USD 22.40 Uber Eats (Burgertory (Werribee)) order ✓"
);
expect(enrichedCard({ ...c, currency: "AUD", transactionId: 7 }).text).toContain("$22.40");
expect(enrichedCard({ ...c, transactionId: 7 }).text).toContain("$22.40");
// A candidate row is a ledger transaction and carries its own currency; it
// used to print "$" beneath a header reading "USD 22.40".
const { blocks } = candidatesCard({ ...c, currency: "USD" }, [
{ transaction_id: 42, description: "Order - Burgertory", transaction_date: "2026-08-17", amount: "19.90", currency: "USD" },
]);
expect(JSON.stringify(blocks)).toContain("USD 19.90");
});
});
describe("liteLLMExtractor — a finish_reason that is present must be `stop`", () => {
const good = { merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4, subtotal: 28, currency: "AUD", items: [], adjustments: [] };
const extractor = (finish_reason?: string) =>
liteLLMExtractor({
baseUrl: "http://litellm.test",
key: "k",
fetchImpl: (async () =>
Response.json({
choices: [{ message: { content: JSON.stringify(good) }, ...(finish_reason ? { finish_reason } : {}) }],
})) as unknown as typeof fetch,
});
it("accepts `stop`, and — on purpose — a reply that reports no reason at all", async () => {
expect((await extractor("stop")([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
expect((await extractor()([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
});
it("rejects any other reason, naming it — the strict schema only held for a reply that finished", async () => {
// An absent reason is NOT rejected: see the comment in intake-extract.ts.
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(ExtractionError);
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/too long/);
await expect(extractor("content_filter")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/content_filter/);
await expect(extractor("tool_calls")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/tool_calls/);
});
});
describe("textCouldBeReceipt — nothing that cannot be a receipt reaches the model", () => {
const why = (t: string | null | undefined) => {
const r = textCouldBeReceipt(t);
return r.ok ? null : r.reason;
};
it("the live defect: the phone sent the WORD `Image`, not the image", () => {
expect(textCouldBeReceipt("Image").ok).toBe(false);
expect(why("Image")).toMatch(/single word/);
expect(why("Image\n")).toMatch(/single word/);
});
it("a filename is not a receipt, digits in it notwithstanding", () => {
expect(why("IMG_0042.jpeg")).toMatch(/single word/);
expect(why("Receipt_19Aug2026.pdf")).toMatch(/single word/);
});
it("empty, blank and whitespace-only", () => {
expect(why("")).toMatch(/empty/);
expect(why(" \n\t ")).toMatch(/empty/);
expect(why(null)).toMatch(/empty/);
expect(why(undefined)).toMatch(/empty/);
});
it("prose with no numbers cannot be a receipt", () => {
expect(why("Shared from Photos")).toMatch(/no numbers/);
expect(why("here is the receipt for dinner last night, sorry it took so long")).toMatch(/no numbers/);
});
it("short but with a printed amount goes through — this door accepts a pasted line", () => {
expect(textCouldBeReceipt("TOTAL $17.00").ok).toBe(true);
expect(textCouldBeReceipt("Total 17.00").ok).toBe(true);
expect(textCouldBeReceipt("Gesamt 12,50 EUR").ok).toBe(true);
});
it("short with digits but no money figure does not — a screenshot name is not a purchase", () => {
expect(why("Screenshot 2026 08 22")).toMatch(/too short/);
expect(why("2 items")).toMatch(/too short/);
});
it("a real receipt body passes", () => {
expect(
textCouldBeReceipt("Burgertory (Werribee)\n19 Aug 2026\n1x Michael 14.00\n1x Joel 14.00\nSubtotal 28.00\nTotal 22.40").ok
).toBe(true);
});
it("quoteShared flattens and truncates what goes on the card", () => {
expect(quoteShared("Image")).toBe("Image");
expect(quoteShared(" a \n b ")).toBe("a b");
expect(quoteShared("x".repeat(200))).toHaveLength(80);
expect(quoteShared("x".repeat(200)).endsWith("…")).toBe(true);
});
});
describe("confabulationReason — the belt to that pre-check", () => {
const nothing = { textParts: ["Image"], hasImages: false };
const readable = { textParts: ["Total $22.40 Burgertory (Werribee) 2026-08-19"], hasImages: false };
const order = { merchant: "Burgertory (Braybrook)", items: [{ description: "The Pride" }] };
it("flags an order extracted from nothing readable", () => {
expect(confabulationReason(nothing, order)).toMatch(/nothing shared could be read/);
});
it("does not flag one extracted from something readable", () => {
expect(confabulationReason(readable, order)).toBeNull();
});
it("does not flag an image share — the pixels are what we asked for", () => {
expect(confabulationReason({ textParts: [], hasImages: true }, order)).toBeNull();
});
it("an empty answer to an unreadable share is honest, not a confabulation", () => {
expect(confabulationReason(nothing, { merchant: null, items: [] })).toBeNull();
});
it("the prompt's own placeholder coming back as a merchant is always a confabulation", () => {
for (const p of PROMPT_PLACEHOLDER_MERCHANTS) {
expect(confabulationReason(readable, { merchant: p, items: [] })).toMatch(/prompt's own example/);
expect(confabulationReason({ textParts: [], hasImages: true }, { merchant: p, items: [] })).toMatch(/prompt's own example/);
}
});
it("a real merchant that happens to be named in the prompt's platform list is NOT flagged", () => {
expect(confabulationReason(readable, { merchant: "Coles (Wyndham Vale)", items: [] })).toBeNull();
expect(confabulationReason(readable, { merchant: "Woolworths", items: [] })).toBeNull();
});
});
describe("the prompt carries no real merchant to echo", () => {
it("names no shop it could hand back as data", () => {
expect(ORDER_ITEM_PROMPT).not.toMatch(/Burgertory/);
expect(ORDER_ITEM_PROMPT).not.toMatch(/Wyndham Vale/);
expect(ORDER_ITEM_PROMPT).toContain(PROMPT_PLACEHOLDER_MERCHANTS[0]);
});
it("tells the model what to return when the input is not a receipt", () => {
expect(ORDER_ITEM_PROMPT).toMatch(/NOT a receipt/);
expect(ORDER_ITEM_PROMPT).toMatch(/Do not construct a plausible order/);
});
});
describe("needsHeaderCard", () => {
it("names the missing field and sends a dateless in-store screenshot to the app's Share button", async () => {
const { needsHeaderCard } = await import("@/lib/intake-blocks");
const dateless = needsHeaderCard(["date"], "woolworths").text;
expect(dateless).toContain("no date");
expect(dateless).toContain("Share button");
expect(dateless).not.toContain("total");
const delivery = needsHeaderCard(["total"], "uber_eats").text;
expect(delivery).toContain("no total");
expect(delivery).toContain("part of the screen");
expect(needsHeaderCard().text).toContain("total or date");
});
}); });
@@ -1,109 +0,0 @@
import { describe, it, expect } from "vitest";
import {
parseWestpacDescriptor,
parseWalletSubtitle,
currencyFromAmountText,
} from "@/lib/wallet-descriptor";
/**
* Every input below is a real captured notification body or subtitle from
* capture.wallet_events (2026-08-13 .. 08-22), not an invented fixture. The
* model handled this one fixed-width format three different ways on this exact
* data, which is why the parsing moved into code.
*/
describe("parseWestpacDescriptor", () => {
it("reads merchant, city and country from the fixed-width block", () => {
expect(parseWestpacDescriptor(
"$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
)).toEqual({ merchant: "Breadworld Bakery Cafe", city: "WYNDHAM VAL", country: "AU" });
});
it("handles a city padded to the full field width", () => {
expect(parseWestpacDescriptor(
"$54.50 paid from Mastercard ending...8032. PLA DONATION MELBOURNE AU"
)).toEqual({ merchant: "PLA DONATION", city: "MELBOURNE", country: "AU" });
});
it("splits a 12-char city that runs into the country code with no space", () => {
// The case that defeated the model: "SAN FRANCISO" exactly fills the city
// field, so "US" follows immediately. Read as one token it becomes the
// nonsense location "SAN FRANCISOUS" and the country is lost -- on the only
// foreign merchant in the whole capture window.
expect(parseWestpacDescriptor(
"$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS"
)).toEqual({ merchant: "SP PEAKDESIGN", city: "SAN FRANCISO", country: "US" });
});
it("reads mixed-case cities the model inconsistently stripped", () => {
expect(parseWestpacDescriptor(
"$20.99 paid from Mastercard ending...8032. Spotify P45AE035CB Sydney AU"
)).toEqual({ merchant: "Spotify P45AE035CB", city: "Sydney", country: "AU" });
});
it("degrades to nulls rather than guessing", () => {
expect(parseWestpacDescriptor(null)).toEqual({ merchant: null, city: null, country: null });
expect(parseWestpacDescriptor("")).toEqual({ merchant: null, city: null, country: null });
// No sentence terminator: not this format at all.
expect(parseWestpacDescriptor("You have a new notification"))
.toEqual({ merchant: null, city: null, country: null });
});
it("does not invent a country from a short merchant-only tail", () => {
// "...IT" here is the end of a word, not Italy. A wrong country on a
// foreign transaction is worse than no country.
const r = parseWestpacDescriptor("$9.00 paid from Mastercard ending...8032. SOME MERCHANT");
expect(r.country).toBeNull();
});
});
describe("parseWalletSubtitle", () => {
it("separates merchant from suburb and state", () => {
expect(parseWalletSubtitle("Breadworld Bakery Cafe, Wyndham Vale, VIC"))
.toEqual({ merchant: "Breadworld Bakery Cafe", location: "Wyndham Vale, VIC" });
});
it("returns no location for an online merchant", () => {
// 2 of the 10 Wallet subtitles captured have no place at all.
expect(parseWalletSubtitle("Seatsaero")).toEqual({ merchant: "Seatsaero", location: null });
expect(parseWalletSubtitle("Prism Accounting Bookkeeping Services"))
.toEqual({ merchant: "Prism Accounting Bookkeeping Services", location: null });
});
it("keeps a merchant name containing its own comma out of the location", () => {
expect(parseWalletSubtitle("Big Daddy`s Fresh Ma, Manor Lakes, VIC").location)
.toBe("Manor Lakes, VIC");
});
it("handles absent input", () => {
expect(parseWalletSubtitle(null)).toEqual({ merchant: null, location: null });
expect(parseWalletSubtitle(" ")).toEqual({ merchant: null, location: null });
});
});
describe("currencyFromAmountText", () => {
it("reads Amex's explicit marker", () => {
expect(currencyFromAmountText("A$5,103.49")).toBe("AUD");
expect(currencyFromAmountText("A$13.50")).toBe("AUD");
});
it("refuses to guess a bare dollar sign", () => {
// 14 of 21 captured transaction rows are this. Assuming AUD is harmless at
// home and a 23x error on a THB tap abroad, so the row goes to the
// reconcile queue with no currency instead.
expect(currencyFromAmountText("$13.50")).toBeNull();
expect(currencyFromAmountText("$103.30")).toBeNull();
});
it("reads unambiguous symbols when a card does state one", () => {
expect(currencyFromAmountText("€24.00")).toBe("EUR");
expect(currencyFromAmountText("฿500")).toBe("THB");
expect(currencyFromAmountText("S$18.20")).toBe("SGD");
expect(currencyFromAmountText("NZ$40.00")).toBe("NZD");
});
it("handles absent input", () => {
expect(currencyFromAmountText(null)).toBeNull();
expect(currencyFromAmountText("")).toBeNull();
});
});
-44
View File
@@ -1,44 +0,0 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { getUpcomingAnnualCharges } from "@/lib/annual-charges";
/**
* Upcoming annual charges, from payments alone.
*
* Lives here rather than in home-dashboard so the detection sits beside the
* ledger it queries, is covered by this repo's integration suite, and can be
* reused by a future Slack nudge without moving. home-dashboard reads it over
* networks_internal the way its other widgets read their sources.
*
* Machine access uses the same shared secret as the dashboard's other reads;
* a browser session scopes to that user instead.
*/
export async function GET(req: NextRequest) {
// Its OWN token, with no fallback to WALLET_INGEST_TOKEN. home-dashboard's
// compose states the rule plainly -- "this app never holds write creds" -- and
// WALLET_INGEST_TOKEN unlocks POST /api/wallet/ingest and the estate-wide
// reconcile sweep. Handing that to a read-only surface to save minting a
// secret would quietly convert a dashboard into a writer. Unset means closed.
const token = process.env.DASH_READ_TOKEN;
const supplied = req.headers.get("x-ingest-token");
const machine = !!token && !!supplied && supplied === token;
let ownerId: number | null = null;
if (machine) {
// A machine caller names the owner explicitly rather than defaulting to 1:
// silently assuming an owner is how an ingested order ended up invisible in
// every scoped view while sitting in the table.
const q = req.nextUrl.searchParams.get("owner_id");
ownerId = q !== null && Number.isInteger(Number(q)) ? Number(q) : null;
if (ownerId === null) {
return NextResponse.json({ error: "owner_id is required for machine access" }, { status: 400 });
}
} else {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
ownerId = user.id;
}
const charges = await getUpcomingAnnualCharges(ownerId);
return NextResponse.json({ charges, count: charges.length });
}
+4 -23
View File
@@ -234,16 +234,6 @@ async function buildDetailsModal(transactionId: number, participantId: number) {
`SELECT merchant_normalized AS merchant, line_items `SELECT merchant_normalized AS merchant, line_items
FROM expense_metadata FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[transactionId] [transactionId]
); );
@@ -387,21 +377,12 @@ async function nudgeState(transactionId: number) {
currency: string | null; currency: string | null;
amount: string; amount: string;
}>( }>(
// LATERAL + the shared tie-break, not a bare join: `queryRow` returns `SELECT em.merchant_normalized AS merchant,
// rows[0] and Postgres guarantees no ordering, so a second metadata row on COALESCE(em.currency, 'AUD') AS currency,
// one transaction would flip the merchant and currency on every card
// refresh at random. Every sibling reader already resolves this way.
`SELECT order_ctx.merchant_normalized AS merchant,
COALESCE(order_ctx.currency, 'AUD') AS currency,
t.amount t.amount
FROM transactions t FROM transactions t
LEFT JOIN LATERAL ( LEFT JOIN expense_metadata em
SELECT em.merchant_normalized, em.currency ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
LIMIT 1
) order_ctx ON true
WHERE t.id = $1`, WHERE t.id = $1`,
[transactionId] [transactionId]
); );
@@ -1,155 +0,0 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { applyReconcileMatch, supersedeTransaction } from "@/lib/reconcile";
/**
* Manually resolve a duplicate pair the automation cannot see.
*
* WHY THIS EXISTS: until now there was no way to merge or remove a transaction
* at all. `superseded_by_id` was READ everywhere -- analytics excludes it,
* auto-reconcile excludes it -- but written nowhere in application code; it had
* only ever been set once, by migration 0023. The reconcile queue was the sole
* merge path, and it only offers candidates within a narrow amount band, so a
* pair outside it (SP PEAKDESIGN: notified $42.44, charged $43.71 -- the card's
* 3% foreign transaction fee) was unresolvable from the UI.
*
* There is deliberately NO delete. Every child of `transactions` is
* ON DELETE CASCADE, so removing a row destroys its splits, tags and metadata
* curation. Both modes here keep the row and its children and merely stop it
* counting:
*
* reconcile -- a statementLESS row folded into its statement twin
* (`reconciled_with_id`). The everyday case.
* supersede -- two rows on the same footing, e.g. a statement imported twice
* (`superseded_by_id`). No constraint on either side's origin.
*
* Both move curation onto the survivor FIRST, via one shared implementation in
* lib/reconcile.ts, because both hide the source row from every figure and
* splits stranded on a hidden row silently stop counting.
*/
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
const { id } = await params;
const loserId = Number(id);
if (!Number.isInteger(loserId)) {
return NextResponse.json({ error: "invalid transaction id" }, { status: 400 });
}
let body: { other_id?: number; mode?: string };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
}
const otherId = Number(body.other_id);
const mode = body.mode ?? "reconcile";
if (!Number.isInteger(otherId)) {
return NextResponse.json({ error: "other_id is required" }, { status: 400 });
}
if (loserId === otherId) {
return NextResponse.json({ error: "a transaction cannot merge into itself" }, { status: 400 });
}
if (mode !== "reconcile" && mode !== "supersede") {
return NextResponse.json({ error: "mode must be 'reconcile' or 'supersede'" }, { status: 400 });
}
// Both rows must be the caller's. A statement row carries owner_id on the
// statement rather than the transaction, which is why this reads the same
// COALESCE every scoped query uses -- checking t.owner_id alone would reject
// every statement row as "not found".
const rows = await queryRaw<{
id: number;
statement_id: number | null;
reconciled_with_id: number | null;
superseded_by_id: number | null;
voided_at: Date | null;
}>(
`SELECT t.id, t.statement_id, t.reconciled_with_id, t.superseded_by_id, t.voided_at
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
WHERE t.id = ANY($1::int[]) AND COALESCE(t.owner_id, s.owner_id) = $2`,
[[loserId, otherId], user.id]
);
if (rows.length !== 2) {
return NextResponse.json({ error: "one or both transactions not found" }, { status: 404 });
}
const loser = rows.find((r) => r.id === loserId)!;
const other = rows.find((r) => r.id === otherId)!;
// Refuse a row that is already folded into something else, in either
// direction. Re-pointing it would strand the curation moved on the first
// pass -- and silently, since both columns hide the row from every figure.
for (const [label, row] of [["source", loser], ["target", other]] as const) {
if (row.reconciled_with_id !== null) {
return NextResponse.json(
{ error: `${label} transaction ${row.id} is already reconciled with ${row.reconciled_with_id}` },
{ status: 409 }
);
}
if (row.superseded_by_id !== null) {
return NextResponse.json(
{ error: `${label} transaction ${row.id} is already superseded by ${row.superseded_by_id}` },
{ status: 409 }
);
}
// A voided row is a charge that never posted. As a source it is already
// out of every figure; as a survivor it would take the other row out too.
if (row.voided_at !== null) {
return NextResponse.json(
{ error: `${label} transaction ${row.id} is voided (a charge that never posted)` },
{ status: 409 }
);
}
}
try {
if (mode === "reconcile") {
// applyReconcileMatch's contract, enforced here so a misuse is a 400 and
// not a row quietly linked the wrong way round.
if (loser.statement_id !== null) {
return NextResponse.json(
{ error: "reconcile expects the source to be a statementless row; use mode 'supersede'" },
{ status: 400 }
);
}
if (other.statement_id === null) {
return NextResponse.json(
{ error: "reconcile expects the target to be a statement row; use mode 'supersede'" },
{ status: 400 }
);
}
await applyReconcileMatch(loserId, otherId);
} else {
// Never hide a statement row behind a statementless one. The statement is
// the bank's own record; the statementless row is a notification, a
// receipt or a hand entry, and is the side that can be wrong -- a wallet
// row carries the amount BEFORE the card's foreign transaction fee, so
// burying the statement line under it would leave the ledger holding
// $42.44 for a purchase the bank charged $43.71 for, with the statement
// no longer reconciling against its own rows. The correct move is the
// other direction, which is exactly what `reconcile` does.
//
// Statement-under-statement stays allowed: that is the imported-twice
// pair migration 0023 existed for, and neither side is more true.
if (loser.statement_id !== null && other.statement_id === null) {
return NextResponse.json(
{
error:
"Refusing to hide a statement row behind a statementless one — the statement is the bank's record. " +
`Open transaction ${otherId} instead and fold it into this one.`,
},
{ status: 400 }
);
}
await supersedeTransaction(loserId, otherId);
}
} catch (e) {
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 400 });
}
return NextResponse.json({ ok: true, mode, source: loserId, target: otherId });
}
@@ -34,14 +34,6 @@ export async function GET(
-- through matched_transaction_id. Both directions have to resolve or the -- through matched_transaction_id. Both directions have to resolve or the
-- detail is missing on exactly the orders that were paid by card. -- detail is missing on exactly the orders that were paid by card.
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
-- The same order as replaceLineItems in line-items.ts and matchTransaction
-- in intake-router.ts. Both directions can resolve at once — a statement
-- line owning its own row while a delivery receipt points at it — and an
-- unordered LIMIT 1 then reads a different row from the one an enrichment
-- wrote, so a PUT that reported success showed nothing on the screen.
-- NULLS LAST because DESC is NULLS FIRST and the pointing row's
-- transaction_id is NULL.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[Number(id)] [Number(id)]
); );
@@ -1,64 +0,0 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { unvoidTransaction, voidTransaction } from "@/lib/reconcile";
/**
* POST /api/transactions/<id>/void { reason } -- a charge that never posted
* DELETE /api/transactions/<id>/void -- undo it
*
* For a statementless row whose charge never reached a statement: an
* authorisation hold the merchant released (txn 7951, a cancelled Airbnb
* booking). The row stays, with its children and audit trail, and drops out of
* every figure and the reconcile queue. Rules and refusals: `voidTransaction`
* in lib/reconcile.ts.
*/
async function ownedRow(req: NextRequest, idParam: string) {
const user = await getCurrentUser(req);
if (!user) return { error: NextResponse.json({ error: "Unauthorized" }, { status: 403 }) };
const id = Number(idParam);
if (!Number.isInteger(id)) {
return { error: NextResponse.json({ error: "invalid transaction id" }, { status: 400 }) };
}
// Same owner scope as every other route; a statement row carries owner_id on
// the statement. (Statement rows are refused later, but must 400, not 404.)
const rows = await queryRaw<{ id: number }>(
`SELECT t.id FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
WHERE t.id = $1 AND COALESCE(t.owner_id, s.owner_id) = $2`,
[id, user.id]
);
if (rows.length === 0) {
return { error: NextResponse.json({ error: "transaction not found" }, { status: 404 }) };
}
return { id };
}
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
const owned = await ownedRow(req, (await params).id);
if (owned.error) return owned.error;
let body: { reason?: unknown };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
}
const reason = typeof body.reason === "string" ? body.reason.trim() : "";
if (!reason) return NextResponse.json({ error: "reason is required" }, { status: 400 });
try {
await voidTransaction(owned.id, reason);
} catch (e) {
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 409 });
}
return NextResponse.json({ ok: true, id: owned.id, voided: true });
}
export async function DELETE(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
const owned = await ownedRow(req, (await params).id);
if (owned.error) return owned.error;
await unvoidTransaction(owned.id);
return NextResponse.json({ ok: true, id: owned.id, voided: false });
}
+90 -66
View File
@@ -1,8 +1,6 @@
import { NextRequest, NextResponse } from "next/server"; import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth"; import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db"; import { prisma, queryRaw } from "@/lib/db";
import { applyReconcileMatch } from "@/lib/reconcile";
import { findAutoMatches } from "@/lib/auto-reconcile";
export async function POST(req: NextRequest) { export async function POST(req: NextRequest) {
const user = await getCurrentUser(req); const user = await getCurrentUser(req);
@@ -19,7 +17,7 @@ export async function POST(req: NextRequest) {
// Verify all manual_ids belong to this user // Verify all manual_ids belong to this user
const manualIds = body.matches.map((m) => m.manual_id); const manualIds = body.matches.map((m) => m.manual_id);
const owned = await queryRaw<{ id: number }>( const owned = await queryRaw<{ id: number }>(
`SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND voided_at IS NULL AND owner_id = $2`, `SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND owner_id = $2`,
[manualIds, user.id] [manualIds, user.id]
); );
if (owned.length !== manualIds.length) { if (owned.length !== manualIds.length) {
@@ -27,71 +25,97 @@ export async function POST(req: NextRequest) {
} }
let reconciled = 0; let reconciled = 0;
for (const { manual_id, statement_tx_id } of body.matches) { for (const { manual_id, statement_tx_id } of body.matches) {
await applyReconcileMatch(manual_id, statement_tx_id); await prisma.$transaction(async (tx) => {
// Copy overrides: manual → statement tx
const override = await tx.transaction_overrides.findUnique({
where: { transaction_id: manual_id },
});
if (override) {
await tx.transaction_overrides.upsert({
where: { transaction_id: statement_tx_id },
update: {
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
updated_at: new Date(),
},
create: {
transaction_id: statement_tx_id,
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
},
});
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
}
// Move tags: manual → statement tx
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
if (tags.length) {
await tx.transaction_tags.createMany({
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
skipDuplicates: true,
});
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
}
// Move splits: manual → statement tx
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
if (splits.length) {
await tx.transaction_splits.createMany({
data: splits.map((s) => ({
transaction_id: statement_tx_id,
participant_id: s.participant_id,
share_percent: s.share_percent,
})),
skipDuplicates: true,
});
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
}
// Move provenance: manual → statement tx.
//
// Overrides, tags and splits above were always carried across; expense_metadata was
// the one child left behind, which did not matter while every metadata row came from
// an email that had created its own transaction. It matters now: a scanned grocery
// receipt puts its line items here, and reconciliation hides the manual row from
// every figure — so without this the shop's contents disappear at exactly the moment
// the statement line appears, and `COLES 0556 MANOR LAKES` stays as unreadable as it
// was before the receipt was ever scanned.
//
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
// or Paperless copy got there first) keeps it. The pantry row stays attached to the
// reconciled manual transaction and is flagged, rather than raising a constraint
// violation or silently overwriting the other source.
const moved = await tx.$executeRawUnsafe(
`UPDATE expense_metadata SET transaction_id = $1
WHERE transaction_id = $2
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
statement_tx_id,
manual_id
);
if (moved === 0) {
await tx.$executeRawUnsafe(
`UPDATE expense_metadata
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
WHERE transaction_id = $1`,
manual_id
);
}
// Mark manual tx as reconciled (link to statement tx)
await tx.$executeRawUnsafe(
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
statement_tx_id,
manual_id
);
});
reconciled++; reconciled++;
} }
return NextResponse.json({ reconciled }); return NextResponse.json({ reconciled });
} }
/**
* Automatic pass over the reconcile queue.
*
* Applies only pairs that are unambiguous by the strict rule in
* `findAutoMatches` -- exact amount (or exact native pair for a foreign row),
* within a few days, and unique in BOTH directions. Everything else is left in
* the queue for a person, which is the point: this exists to keep the queue
* workable as the wallet lane adds ~63 rows a month, not to clear it.
*
* Validated against the 48 reconciliations a human had already made: the rule
* reproduces all 48 and proposes nothing they did not.
*
* `?dry=1` returns what it WOULD do without writing, which is how this should
* be run the first time after any change to the matching rule.
*/
export async function PATCH(req: NextRequest) {
// Two callers, two auth modes. A person in the browser reconciles their own
// queue; the scheduled sweep has no session and carries the machine token,
// and must cover EVERY owner with a queue -- 169 of the 177 rows outstanding
// belong to the second participant, so a sweep scoped to one user would leave
// almost all of it untouched.
const machineToken = req.headers.get("x-ingest-token");
const machineOk = !!machineToken && machineToken === process.env.WALLET_INGEST_TOKEN;
let ownerIds: number[];
if (machineOk) {
ownerIds = (await queryRaw<{ owner_id: number }>(
`SELECT DISTINCT owner_id FROM transactions
WHERE statement_id IS NULL AND owner_id IS NOT NULL
AND reconciled_with_id IS NULL AND superseded_by_id IS NULL`
)).map((r) => r.owner_id);
} else {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
ownerIds = [user.id];
}
const dry = new URL(req.url).searchParams.get("dry") === "1";
const matches = (await Promise.all(ownerIds.map((id) => findAutoMatches(id)))).flat();
if (dry) {
return NextResponse.json({ dry_run: true, would_reconcile: matches.length, matches });
}
let reconciled = 0;
const failed: { manual_id: number; error: string }[] = [];
for (const m of matches) {
try {
await applyReconcileMatch(m.manual_id, m.statement_tx_id);
reconciled++;
} catch (e) {
// One bad pair must not abandon the rest -- each merge is its own
// transaction, so a failure here leaves that pair untouched and in the
// queue rather than half-applied.
failed.push({ manual_id: m.manual_id, error: e instanceof Error ? e.message : String(e) });
}
}
return NextResponse.json({ reconciled, examined: matches.length, failed });
}
-49
View File
@@ -1,49 +0,0 @@
import { NextRequest, NextResponse } from "next/server";
import { processWalletIngestion, type WalletEvent } from "@/lib/wallet-ingestion";
/**
* Machine ingest endpoint for interpreted iOS Wallet notifications.
*
* Sibling to /api/orders/ingest and /api/receipts/ingest, and deliberately
* shaped like them: a shared secret rather than a browser session, because n8n
* calls this app-to-app.
*
* Its own token so the wallet lane's credential can be rotated without touching
* the order flow, which runs on a schedule nobody is watching.
*
* n8n is transport only. The interpretation already happened (Wallet Interpret,
* CAp1s5cuzJk2mDaI, local model on Unraid because card tails are PII); this
* endpoint owns every decision about what reaches the ledger.
*/
function authorised(req: NextRequest): boolean {
const expected = process.env.WALLET_INGEST_TOKEN;
if (!expected) return false; // fail closed when unconfigured
const got = req.headers.get("x-ingest-token");
return !!got && got === expected;
}
export async function POST(req: NextRequest) {
if (!authorised(req)) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
let body: WalletEvent;
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
}
if (!Number.isInteger(body?.event_id) || typeof body?.kind !== "string") {
return NextResponse.json({ error: "event_id (int) and kind are required" }, { status: 400 });
}
try {
const result = await processWalletIngestion(body);
return NextResponse.json(result);
} catch (e) {
// A 500 here alerts #log-alerts through the workflow error handler, which
// is what should happen: a write that fails is a template or schema change,
// not something to swallow.
const message = e instanceof Error ? e.message : String(e);
return NextResponse.json({ error: message }, { status: 500 });
}
}
+1 -34
View File
@@ -1,7 +1,7 @@
"use client"; "use client";
import { useState } from "react"; import { useState } from "react";
import { usePendingReconciliations, useReconcile, useVoidTransaction } from "@/lib/hooks"; import { usePendingReconciliations, useReconcile } from "@/lib/hooks";
import type { ManualTxWithMatches, PotentialMatch } from "@/lib/queries"; import type { ManualTxWithMatches, PotentialMatch } from "@/lib/queries";
function formatDate(d: string | Date) { function formatDate(d: string | Date) {
@@ -156,7 +156,6 @@ export default function ReconcilePage() {
<span key={tag.id} className="w-2 h-2 rounded-full" style={{ backgroundColor: tag.color }} title={tag.name} /> <span key={tag.id} className="w-2 h-2 rounded-full" style={{ backgroundColor: tag.color }} title={tag.name} />
))} ))}
</div> </div>
<VoidButton id={tx.id} amount={tx.amount} onError={setError} />
</div> </div>
))} ))}
</div> </div>
@@ -244,7 +243,6 @@ function ReconcileRow({
Clear Clear
</button> </button>
)} )}
<VoidButton id={tx.id} amount={tx.amount} />
{!isSkipped && ( {!isSkipped && (
<button onClick={onSkip} className="text-xs text-zinc-500 hover:text-zinc-300 px-2 py-1 rounded hover:bg-zinc-800"> <button onClick={onSkip} className="text-xs text-zinc-500 hover:text-zinc-300 px-2 py-1 rounded hover:bg-zinc-800">
Skip (no match) Skip (no match)
@@ -298,34 +296,3 @@ function MatchRow({
</button> </button>
); );
} }
/**
* "Never posted": a released authorisation hold, e.g. a booking cancelled
* before the charge settled. Drops the row out of every figure and this queue
* (migration 0040); the row and its history are kept.
*/
function VoidButton({ id, amount, onError }: { id: number; amount: number; onError?: (msg: string) => void }) {
const voidTx = useVoidTransaction();
async function handle() {
const reason = window.prompt(
`Mark this $${Number(amount).toFixed(2)} charge as never posted? It will stop counting everywhere.\n\nReason (e.g. "hold released, booking cancelled"):`
);
if (!reason?.trim()) return;
try {
await voidTx.mutateAsync({ id, reason });
} catch (e) {
const msg = e instanceof Error ? e.message : "Void failed";
if (onError) onError(msg); else window.alert(msg);
}
}
return (
<button
onClick={handle}
disabled={voidTx.isPending}
title="The charge never posted (e.g. a released hold) — stop counting it"
className="text-xs text-zinc-500 hover:text-red-300 px-2 py-1 rounded hover:bg-zinc-800 disabled:opacity-50 flex-shrink-0"
>
{voidTx.isPending ? "Voiding…" : "Never posted"}
</button>
);
}
+2 -2
View File
@@ -463,7 +463,7 @@ export default function RulesPage() {
> >
{expandedRun === run.id ? "▾" : "▸"} {expandedRun === run.id ? "▾" : "▸"}
</button> </button>
<span className="text-zinc-300">{new Date(run.applied_at).toLocaleString("en-AU")}</span> <span className="text-zinc-300">{new Date(run.applied_at).toLocaleString()}</span>
{/* Which rule ran matters most: a merchant rename and a {/* Which rule ran matters most: a merchant rename and a
50/50 split of everything look identical as counts. */} 50/50 split of everything look identical as counts. */}
<span className="text-zinc-200"> <span className="text-zinc-200">
@@ -480,7 +480,7 @@ export default function RulesPage() {
)} )}
</div> </div>
{run.reverted_at ? ( {run.reverted_at ? (
<span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString("en-AU")}</span> <span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString()}</span>
) : ( ) : (
<button <button
onClick={() => { onClick={() => {
+1 -12
View File
@@ -1085,18 +1085,7 @@ function TransactionsContent() {
) : ( ) : (
<> <>
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)} {formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
{/* {t.currency && t.currency !== "AUD" && (
The native figure: `foreign_currency_amount` where one
was recorded (a statement or wallet row, whose `amount`
is AUD), else `amount` on a non-AUD statement. Printing
`amount` under the foreign code put the AUD figure
behind an INR label on every wallet row.
*/}
{t.foreign_currency_code && t.foreign_currency_code !== "AUD" && t.foreign_currency_amount !== null ? (
<div className="text-[10px] text-zinc-500 mt-0.5">
{formatAmount(t.foreign_currency_amount, t.transaction_type, t.foreign_currency_code)}
</div>
) : t.currency && t.currency !== "AUD" && (
<div className="text-[10px] text-zinc-500 mt-0.5"> <div className="text-[10px] text-zinc-500 mt-0.5">
{formatAmount(t.amount, t.transaction_type, t.currency)} {formatAmount(t.amount, t.transaction_type, t.currency)}
</div> </div>
+1 -208
View File
@@ -1,9 +1,8 @@
"use client"; "use client";
import { useEffect, useState } from "react"; import { useState } from "react";
import { import {
useUpdateTransaction, useUpdateTransaction,
useMergeTransaction,
useTags, useTags,
useAddTransactionTag, useAddTransactionTag,
useRemoveTransactionTag, useRemoveTransactionTag,
@@ -94,210 +93,6 @@ function InlineTags({ transactionId, initialTags }: { transactionId: number; ini
); );
} }
/**
* Determines which merge mode the backend will accept for (loser, candidate).
* Mirrors the constraint enforced server-side in the merge route: reconcile
* requires a statementless loser folding into a statement survivor; every
* other pairing is a supersede.
*/
function mergeModeFor(loser: TransactionRow, candidate: TransactionRow): "reconcile" | "supersede" {
return loser.statement_id === null && candidate.statement_id !== null ? "reconcile" : "supersede";
}
/**
* Collapsed by default, at the bottom, visually separate — this is the one
* control on the modal that changes which rows this transaction can ever be
* again. `transaction` is always the row that will be hidden (the loser);
* the row picked here is the survivor (`other_id`).
*/
function DuplicateSection({ transaction, onMerged }: { transaction: TransactionRow; onMerged: () => void }) {
const [expanded, setExpanded] = useState(false);
const [query, setQuery] = useState("");
const [results, setResults] = useState<TransactionRow[]>([]);
const [searching, setSearching] = useState(false);
const [selected, setSelected] = useState<TransactionRow | null>(null);
const mergeTxn = useMergeTransaction();
useEffect(() => {
if (!expanded || selected) return;
const q = query.trim();
// No synchronous setState here: an empty box is handled at render time, and
// `searching` is raised inside the timeout. Clearing state here instead
// cascades a render on every keystroke (react-hooks lint catches it).
if (q.length === 0) return;
const handle = setTimeout(() => {
setSearching(true);
fetch(`/api/transactions?search=${encodeURIComponent(q)}&limit=8`)
.then((res) => res.json())
.then((body: { data?: TransactionRow[] }) => {
setResults((body.data ?? []).filter((r) => r.id !== transaction.id));
})
.catch(() => setResults([]))
.finally(() => setSearching(false));
}, 300);
return () => clearTimeout(handle);
}, [query, expanded, selected, transaction.id]);
function reset() {
setExpanded(false);
setQuery("");
setResults([]);
setSelected(null);
mergeTxn.reset();
}
async function handleConfirm() {
if (!selected) return;
try {
await mergeTxn.mutateAsync({
id: transaction.id,
other_id: selected.id,
mode: mergeModeFor(transaction, selected),
});
onMerged();
} catch {
// Surfaced below via mergeTxn.error — section stays open so they can retry.
}
}
const mode = selected ? mergeModeFor(transaction, selected) : null;
// Hiding a statement row behind a statementless one is always backwards — the
// bank's record is the side that survives. The API refuses it too; catching it
// here just means the button explains itself instead of returning a 400.
const wrongDirection =
selected !== null && transaction.statement_id !== null && selected.statement_id === null;
return (
<div className="pt-4 border-t border-zinc-800">
{!expanded ? (
<button
type="button"
onClick={() => setExpanded(true)}
className="text-xs text-zinc-500 hover:text-zinc-300"
>
Mark as duplicate…
</button>
) : (
<div className="space-y-2.5">
<div className="flex items-center justify-between">
<p className="font-semibold text-xs text-zinc-400">Mark as duplicate</p>
<button
type="button"
onClick={reset}
className="text-xs text-zinc-500 hover:text-zinc-300"
>
Cancel
</button>
</div>
{!selected ? (
<>
<input
type="text"
value={query}
onChange={(e) => setQuery(e.target.value)}
placeholder="Search for the surviving transaction…"
autoFocus
className="w-full bg-zinc-800 border border-zinc-700 rounded px-2 py-1.5 text-sm"
/>
{searching && <p className="text-xs text-zinc-500">Searching…</p>}
{!searching && query.trim().length > 0 && results.length === 0 && (
<p className="text-xs text-zinc-600 italic">No matches</p>
)}
{/* Gated on the query too: results are no longer cleared when the
box empties, so without this the last search's rows linger. */}
{query.trim().length > 0 && results.length > 0 && (
<div className="space-y-1 max-h-48 overflow-y-auto">
{results.map((r) => (
<button
key={r.id}
type="button"
onClick={() => setSelected(r)}
className="w-full text-left px-2 py-1.5 rounded bg-zinc-800/50 hover:bg-zinc-800 border border-zinc-700/50"
>
<div className="flex items-center justify-between gap-2">
<span className="text-sm text-zinc-200 truncate">{r.description}</span>
<span
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(r.transaction_type) ? "text-red-400" : "text-green-400"}`}
>
{formatAmount(r.amount, r.transaction_type)}
</span>
</div>
<div className="flex items-center justify-between mt-0.5">
<span className="text-xs text-zinc-500">
{new Date(r.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
</span>
<span className="text-xs text-zinc-500">
{r.statement_id !== null ? "Statement row" : "Statementless"}
</span>
</div>
</button>
))}
</div>
)}
</>
) : (
<div className="space-y-2">
<div className="px-2 py-1.5 rounded bg-zinc-800/50 border border-zinc-700/50">
<div className="flex items-center justify-between gap-2">
<span className="text-sm text-zinc-200 truncate">{selected.description}</span>
<span
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(selected.transaction_type) ? "text-red-400" : "text-green-400"}`}
>
{formatAmount(selected.amount, selected.transaction_type)}
</span>
</div>
<div className="flex items-center justify-between mt-0.5">
<span className="text-xs text-zinc-500">
{new Date(selected.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
</span>
<span className="text-xs text-zinc-500">
{selected.statement_id !== null ? "Statement row" : "Statementless"}
</span>
</div>
</div>
{wrongDirection ? (
<p className="text-xs text-red-400">
This is the wrong way round — the statement is the bank&apos;s record, so it
has to be the row that survives. Close this and open{" "}
<span className="text-zinc-300">{selected.description}</span> instead, then
fold that one into this.
</p>
) : (
<p className="text-xs text-amber-400/90">
{mode === "reconcile"
? "This row will be folded into the statement row above."
: "This row will be marked a duplicate of the row above."}
</p>
)}
{mergeTxn.error && (
<p className="text-red-400 text-xs">{(mergeTxn.error as Error).message}</p>
)}
<div className="flex gap-2">
<button
type="button"
onClick={() => setSelected(null)}
className="px-3 py-1.5 bg-zinc-800 hover:bg-zinc-700 text-zinc-300 rounded-lg text-xs"
>
Choose a different row
</button>
<button
type="button"
onClick={handleConfirm}
disabled={mergeTxn.isPending || wrongDirection}
className="px-3 py-1.5 bg-amber-700 hover:bg-amber-600 disabled:opacity-50 text-white rounded-lg text-xs"
>
{mergeTxn.isPending ? "Merging…" : mode === "reconcile" ? "Fold into statement row" : "Mark as duplicate"}
</button>
</div>
</div>
)}
</div>
)}
</div>
);
}
export function EditTransactionModal({ export function EditTransactionModal({
transaction, transaction,
onClose, onClose,
@@ -621,8 +416,6 @@ export function EditTransactionModal({
<OrderDetails transactionId={transaction.id} currency={transaction.currency ?? null} /> <OrderDetails transactionId={transaction.id} currency={transaction.currency ?? null} />
<DuplicateSection transaction={transaction} onMerged={onClose} />
</div> </div>
{/* Footer */} {/* Footer */}
+7 -4
View File
@@ -13,9 +13,12 @@ import {
type OrderReceipt, type OrderReceipt,
type OrderRating, type OrderRating,
} from "@/lib/hooks"; } from "@/lib/hooks";
// One platform vocabulary, shared with the intake cards — three copies had
// already drifted apart (the intake one was missing `uber`). const PLATFORM_LABEL: Record<string, string> = {
import { platformLabel } from "@/lib/intake-rules"; doordash: "DoorDash",
ubereats: "Uber Eats",
uber: "Uber",
};
/** /**
* Who records verdicts. A two-person household with one primary user: the * Who records verdicts. A two-person household with one primary user: the
@@ -96,7 +99,7 @@ export function OrderDetails({
<p className="text-xs text-zinc-500"> <p className="text-xs text-zinc-500">
Order details Order details
{receipt.platform && ( {receipt.platform && (
<span className="ml-1.5 text-zinc-400">{platformLabel(receipt.platform)}</span> <span className="ml-1.5 text-zinc-400">{PLATFORM_LABEL[receipt.platform] ?? receipt.platform}</span>
)} )}
</p> </p>
{receipt.card_last4 && ( {receipt.card_last4 && (
+2 -17
View File
@@ -71,17 +71,9 @@ export const STATEMENTS_JOIN = `LEFT JOIN statements s ON s.id = t.statement_id`
* without being edited. Anything summing transactions without this fragment * without being edited. Anything summing transactions without this fragment
* still double-counts — that is the same gap that let reconciled rows into the * still double-counts — that is the same gap that let reconciled rows into the
* analytics routes in the first place. * analytics routes in the first place.
*
* **3. The charge that never posted** (`voided_at`, migration 0040).
*
* A card notification is sent for an authorisation, and a released hold never
* reaches a statement (txn 7951: a $1,040.41 Airbnb hold on a booking cancelled
* the next day). There is no survivor to point at, so it gets its own column —
* and it lives in this fragment for the same reason case 2 does.
*/ */
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL) export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
AND t.superseded_by_id IS NULL AND t.superseded_by_id IS NULL`;
AND t.voided_at IS NULL`;
/** /**
* The currency `t.amount` is actually denominated in. * The currency `t.amount` is actually denominated in.
@@ -97,17 +89,10 @@ export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reco
* figure and `foreign_currency_code` names it (order-ingestion.ts leaves * figure and `foreign_currency_code` names it (order-ingestion.ts leaves
* `amount_aud` NULL rather than asserting an FX rate it does not have). * `amount_aud` NULL rather than asserting an FX rate it does not have).
* *
* - A wallet row also has no statement, but follows the STATEMENT convention:
* `amount` = `amount_aud` = a provisional AUD figure, and the native figure
* lives in `foreign_currency_amount` (wallet-ingestion.ts). `amount_aud`
* being present is what says `amount` is already AUD. Without that arm, an
* INR 524 tap stored as A$9.42 rendered as "INR 9.42" -- the AUD number
* under the foreign label (txn 8162, 2026-10-03).
*
* Reading `s.currency` alone labels every foreign order row as AUD. * Reading `s.currency` alone labels every foreign order row as AUD.
* `s` must be the statements alias in scope. * `s` must be the statements alias in scope.
*/ */
export const NATIVE_CURRENCY = `COALESCE(s.currency, CASE WHEN t.amount_aud IS NULL THEN t.foreign_currency_code END, 'AUD')`; export const NATIVE_CURRENCY = `COALESCE(s.currency, t.foreign_currency_code, 'AUD')`;
/** /**
* True when a row's AUD value is unknown: it is denominated in something other * True when a row's AUD value is unknown: it is denominated in something other
-275
View File
@@ -1,275 +0,0 @@
import { queryRaw } from "@/lib/db";
/**
* Upcoming annual charges, derived from payments alone.
*
* Independent of the email lane on purpose: a recurring CHARGE is visible in the
* ledger whether or not anything was ever emailed about it, which is the only
* way a card annual fee or a loan package fee shows up at all -- banks announce
* those on the statement, not in a renewal notice.
*
* WHAT DOES NOT WORK, measured before this was written: "two charges about 365
* days apart" is useless on its own. Ranked that way the top hit on this ledger
* is a bakery with 202 such pairs, then a donation, Patreon, Coles -- merchants
* used often enough that some pair is inevitably a year apart. Frequency is the
* confound. Even "every gap is annual" still surfaces a Target visited twice
* ($30, then $198) and a Specsavers ($24, then $224). Two visits a year apart
* are not a fee.
*
* So there are two signals, and they are kept apart because they are not equally
* trustworthy -- a guess must never render beside a certainty as if it were one:
*
* confirmed the bank called it a fee AND it has repeated on an annual cadence
* likely the bank called it a fee, seen once -- the kind is certain, the
* cadence is inferred (an annual fee charged once is still annual)
* possible no fee wording; a merchant whose every gap is annual AND whose
* amount is stable. Catches Amazon Prime, Uber One, VicRoads rego,
* none of which carry fee wording. Also where false positives live.
*/
export type Confidence = "confirmed" | "likely" | "possible";
/**
* `expected_not_seen` is not a lesser version of `upcoming` -- it is the more
* interesting row. The estimate has passed and no charge landed, which means one
* of three things, and `account_days_stale` is what tells them apart:
* the account stopped being imported (HSBC: ledger ends 2026-01-07, while the
* mail shows statements to 2026-02-07)
* the account genuinely closed
* the charge landed under a different label (VicRoads paid 2026-03-16 as
* "VICROADS ONLINE PAYM", not "VICROADS")
* Reporting it as upcoming would be a lie; hiding it would lose the signal.
*/
export type ChargeStatus = "upcoming" | "expected_not_seen";
export type AnnualCharge = {
label: string;
status: ChargeStatus;
bank_name: string | null;
account_number: string | null;
confidence: Confidence;
observations: number;
last_charged: string;
last_amount: string;
amount_low: string;
amount_high: string;
median_gap_days: number | null;
next_due_estimate: string;
/** Ledger liveness for the account: a fee that "stopped" may just be an
* account that stopped being imported. Null for merchant-signal rows, which
* are not tied to one account. */
account_last_statement: string | null;
account_days_stale: number | null;
};
/**
* `transaction_type = 'fee'` is the universe, not a description match.
*
* The statement extractor already classifies fees structurally -- 131 rows
* across the ledger -- and every card and loan annual fee on this ledger is
* typed that way. An earlier version of this filtered `transaction_type =
* 'debit'` and returned NONE of them, which is the whole reason this comment
* exists: the fees are not debits.
*
* That universe is broader than "annual", though. It also holds ANZ's monthly
* ACCOUNT SERVICING FEE, Zip's MONTHLY ACCOUNT SERVICE FEE, CommBank's MONTHLY
* FEE, Citibank LATE PAYMENT FEE, HSBC OVERSEAS TRANSACTION FEE and every Wise
* per-transfer charge. Cadence separates them: an annual fee repeats about a
* year apart, the rest repeat monthly, on an event, or never (Wise labels each
* transfer uniquely, so they can never group to two observations at all).
*/
const FEE_TYPE = "fee";
/**
* Wording that establishes the KIND when there is only one observation.
*
* A single charge cannot demonstrate a cadence, but a bank calling something an
* annual fee has already told us what it is -- Westpac's CARD FEE $295 and AMP's
* Professional Pack Annual Fee each appear once and are certainly annual.
* Deliberately narrow: bare "fee" would readmit every per-transaction charge.
*/
const ANNUAL_WORDING =
"(annual|yearly)[[:space:]]*(card[[:space:]]*)?(fee|charge)" +
"|card[[:space:]]+fee" +
"|(package|membership|facility)[[:space:]]+fee";
/**
* Resolve every statement to a canonical account identity, following
* account_links forward. One card can appear under two bank names and two
* numbers after an issuer change (Citi Prestige -> NAB MyCard); without this the
* $700 fee charged 2024-12-13 and 2025-11-13 reads as two dead one-offs.
*
* Recursive because the table permits a chain (A->B->C); `uq_account_links_from`
* forbids a fork, and `account_links_no_self` a self-link, so the walk
* terminates. The depth guard is belt-and-braces against a cycle someone inserts
* by hand (A->B, B->A), which no constraint can catch.
*/
const RESOLVED_ACCOUNTS = `
WITH RECURSIVE chain AS (
SELECT s.id AS statement_id, s.bank_name, s.account_number,
s.bank_name AS cur_bank, s.account_number_key AS cur_key, 0 AS depth
FROM statements s
UNION ALL
SELECT c.statement_id, c.bank_name, c.account_number,
l.to_bank_name, l.to_account_key, c.depth + 1
FROM chain c
JOIN account_links l
ON l.from_bank_name = c.cur_bank AND l.from_account_key = c.cur_key
WHERE c.depth < 10
),
resolved AS (
SELECT DISTINCT ON (statement_id)
statement_id, bank_name, account_number, cur_bank AS acct_bank, cur_key AS acct_key
FROM chain ORDER BY statement_id, depth DESC
)`;
export async function getUpcomingAnnualCharges(ownerId: number): Promise<AnnualCharge[]> {
// Signal 1 -- the bank's own fee wording, grouped by resolved account identity
// and by the fee's wording so two different fees on one card stay apart
// (ANZ charges "ANNUAL FEE" $120 and "ADDITIONAL CARD FEE" $10 on the same
// statement, and they are not the same obligation).
const feeRows = await queryRaw<AnnualCharge>(
`${RESOLVED_ACCOUNTS},
fees AS (
SELECT r.acct_bank, r.acct_key, r.bank_name, r.account_number,
upper(trim(regexp_replace(t.description, '[[:space:]]+', ' ', 'g'))) AS label,
t.transaction_date, t.amount
FROM transactions t
JOIN resolved r ON r.statement_id = t.statement_id
JOIN statements s ON s.id = t.statement_id
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
AND t.amount > 0
AND COALESCE(t.owner_id, s.owner_id) = $1
-- Structural first, wording only as a widening: a bank that types a
-- fee as a plain debit still gets picked up if it names it annual.
AND (t.transaction_type = $3 OR t.description ~* $2)
),
gapped AS (
SELECT f.*,
transaction_date - lag(transaction_date)
OVER (PARTITION BY acct_key, label ORDER BY transaction_date) AS gap
FROM fees f
),
agg AS (
SELECT acct_key, acct_bank, label,
count(*)::int AS observations,
max(transaction_date) AS last_charged,
min(amount) AS amount_low, max(amount) AS amount_high,
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
FILTER (WHERE gap IS NOT NULL) AS median_gap,
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
(array_agg(bank_name ORDER BY transaction_date DESC))[1] AS bank_name,
(array_agg(account_number ORDER BY transaction_date DESC))[1] AS account_number,
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
FROM gapped GROUP BY acct_key, acct_bank, label
),
liveness AS (
SELECT r.acct_key, max(s.billing_end_date) AS last_statement
FROM statements s JOIN resolved r ON r.statement_id = s.id
GROUP BY r.acct_key
)
SELECT a.label,
-- The account's CURRENT identity, not the bank that happened to
-- print the last fee. That $700 fee last appeared under "Citibank"
-- in Nov 2025, but the account is NAB/MyCard now and the next one
-- will come from there; naming the dead brand would send you to the
-- wrong bank.
a.acct_bank AS bank_name, a.account_number,
CASE WHEN (a.last_charged + COALESCE(round(a.median_gap)::int, 365)) >= CURRENT_DATE
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
-- One observation cannot prove a cadence, but the bank naming it an
-- annual fee already tells us the kind. Repeated on an annual gap it
-- is confirmed; seen once it is likely and says so.
CASE WHEN a.gaps > 0 AND a.annual_gaps = a.gaps THEN 'confirmed'
ELSE 'likely' END AS confidence,
a.observations,
a.last_charged::text AS last_charged,
a.last_amount::text AS last_amount,
a.amount_low::text AS amount_low,
a.amount_high::text AS amount_high,
round(a.median_gap)::int AS median_gap_days,
(a.last_charged + COALESCE(round(a.median_gap)::int, 365))::text AS next_due_estimate,
l.last_statement::text AS account_last_statement,
(CURRENT_DATE - l.last_statement)::int AS account_days_stale
FROM agg a LEFT JOIN liveness l ON l.acct_key = a.acct_key
-- Two ways to earn a place, and nothing else gets one. Filtered here, not
-- in the UI, so the API can never be read wrongly by a future caller.
WHERE
-- Repeated on an annual cadence. Excludes ANZ's monthly ACCOUNT
-- SERVICING FEE and Zip's MONTHLY ACCOUNT SERVICE FEE, whose gaps are
-- ~30 days, and Citibank's LATE PAYMENT FEE, which is event-driven.
(a.gaps > 0 AND a.annual_gaps = a.gaps AND a.median_gap >= 300)
-- Or seen once, but named as an annual/card/package fee. Westpac's CARD
-- FEE and AMP's Professional Pack Annual Fee are each a single charge and
-- unmistakably annual. A Wise per-transfer fee is also a single charge
-- and says nothing of the sort, which is what keeps it out.
OR (a.gaps = 0 AND a.label ~* $2)`,
[ownerId, ANNUAL_WORDING, FEE_TYPE]
);
// Signal 2 -- merchant-charged annual items with no fee wording. Requires
// BOTH that every gap is annual AND that the amount is stable: VicRoads rego
// rises about 3% a year ($825.23 -> $852.88 -> $875.48), so equality is too
// strict, while no amount test at all readmits the twice-visited restaurant.
// Keyed on merchant, never on card -- that rego was paid on three different
// banks, and a card-keyed view would see three unrelated one-offs.
const merchantRows = await queryRaw<AnnualCharge>(
`WITH t AS (
SELECT upper(trim(COALESCE(t.merchant_normalized, t.merchant_name, t.description))) AS label,
t.transaction_date, t.amount
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
AND t.voided_at IS NULL
AND t.transaction_type = 'debit' AND t.amount > 0
AND COALESCE(t.owner_id, s.owner_id) = $1
AND t.transaction_type = 'debit' AND t.description !~* $2
),
gapped AS (
SELECT t.*, transaction_date - lag(transaction_date)
OVER (PARTITION BY label ORDER BY transaction_date) AS gap
FROM t
),
agg AS (
SELECT label, count(*)::int AS observations,
max(transaction_date) AS last_charged,
min(amount) AS amount_low, max(amount) AS amount_high,
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
FILTER (WHERE gap IS NOT NULL) AS median_gap,
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
FROM gapped GROUP BY label
)
SELECT label, NULL::text AS bank_name, NULL::text AS account_number,
CASE WHEN (last_charged + round(median_gap)::int) >= CURRENT_DATE
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
'possible' AS confidence, observations,
last_charged::text AS last_charged,
last_amount::text AS last_amount,
amount_low::text AS amount_low,
amount_high::text AS amount_high,
round(median_gap)::int AS median_gap_days,
(last_charged + round(median_gap)::int)::text AS next_due_estimate,
NULL::text AS account_last_statement,
NULL::int AS account_days_stale
FROM agg
WHERE gaps >= 1 AND annual_gaps = gaps
-- Amount stability is what separates a subscription from a restaurant.
-- 20% absorbs CPI drift and an FX-priced renewal without admitting a
-- $30-then-$198 pair.
AND amount_high <= amount_low * 1.20
-- Two observations is the floor for a cadence claim, and this lane has no
-- wording to fall back on the way the fee lane does.
AND observations >= 2`,
[ownerId, ANNUAL_WORDING]
);
// Upcoming first and soonest-first within that, so the thing you can still
// act on is at the top; the expected-but-missing rows follow as a review list.
const rank = (c: AnnualCharge) => (c.status === "upcoming" ? 0 : 1);
return [...feeRows, ...merchantRows].sort(
(a, b) => rank(a) - rank(b) || a.next_due_estimate.localeCompare(b.next_due_estimate)
);
}
-235
View File
@@ -1,235 +0,0 @@
import { queryRaw, type Queryable } from "@/lib/db";
import { merchantMatches } from "@/lib/intake-rules";
/**
* Unambiguous statement matches for statementless ("manual") transactions.
*
* WHY THIS EXISTS
* ---------------
* The reconcile queue is worked by hand and is already ~177 deep. The wallet
* lane writes a transaction for every tap (DECISIONS.md ING-11, 2026-08-22),
* which at the observed rate adds ~63 rows a month, ~767 a year. Without an
* automatic pass for the easy cases that queue stops being workable, and the
* tempting alternative -- filtering the new rows back out of it -- is
* explicitly ruled out by the `awaitsStatementLine()` removal note in
* queries.ts: "an imported row that has no statement belongs in the queue;
* volume is solved upstream, never downstream by hiding what is there."
*
* WHAT COUNTS AS UNAMBIGUOUS
* --------------------------
* Deliberately much stricter than the queue's own candidate ranking, which
* offers the top 5 within +/-1% and +/-3 days for a human to choose between.
* A machine gets one shot and no judgement, so:
*
* - EXACT amount equality, not the queue's +/-1% band. A near-miss is exactly
* the case a person should look at.
* - Same owner, and the statement row must not already be reconciled against.
* - UNIQUE BOTH WAYS: the manual row must have exactly one candidate, AND
* that candidate must be claimed by exactly one manual row. This is the
* order_transaction_bridge rule, and it is what stops two identical $13.50
* bakery taps in one window from being paired arbitrarily.
*
* Foreign rows match on the NATIVE pair (foreign_currency_amount +
* foreign_currency_code), never on `amount`: `amount` is the AUD the bank
* charged after applying its own rate, so a wallet row cannot predict it, while
* the native figure is exact on both sides. Same reasoning as
* order_transaction_bridge's FX path.
*
* This function only PROPOSES. Applying a pair goes through the same merge the
* manual route uses, so overrides/tags/splits/metadata move identically.
*/
export type AutoMatch = {
manual_id: number;
statement_tx_id: number;
amount: string;
manual_date: string;
statement_date: string;
description: string;
basis: "aud" | "native";
};
/** Days either side of the manual row's date to consider. Authorisation and
* settlement are rarely the same day; 3 matches the queue's own window and
* the wallet reconciliation check's default. */
export const AUTO_MATCH_WINDOW_DAYS = 3;
export async function findAutoMatches(
ownerId: number,
windowDays: number = AUTO_MATCH_WINDOW_DAYS
): Promise<AutoMatch[]> {
return queryRaw<AutoMatch>(
`WITH pairs AS (
SELECT m.id AS manual_id,
t.id AS statement_tx_id,
t.amount::text AS amount,
m.transaction_date::text AS manual_date,
t.transaction_date::text AS statement_date,
t.description AS description,
CASE WHEN m.foreign_currency_code IS NOT NULL
AND m.foreign_currency_amount IS NOT NULL
THEN 'native' ELSE 'aud' END AS basis
FROM transactions m
JOIN transactions t
ON t.statement_id IS NOT NULL
AND t.transaction_type = 'debit'
AND t.transaction_date BETWEEN m.transaction_date - $2::int
AND m.transaction_date + $2::int
AND (
-- Foreign: compare what both sides actually recorded natively.
(m.foreign_currency_code IS NOT NULL
AND m.foreign_currency_amount IS NOT NULL
AND t.foreign_currency_code = m.foreign_currency_code
AND t.foreign_currency_amount = m.foreign_currency_amount)
OR -- AUD side: exact equality, never a tolerance band. The
-- statement row MAY be foreign here, and that is not a mistake:
-- all 48 reconciliations a human has made on this ledger are
-- exactly this shape -- a manual row carrying no currency,
-- matched on exact AUD against a statement row that does carry
-- one (EUR, CHF). Overseas receipts and orders record what was
-- charged in AUD; only the wallet lane records natively, which
-- is why the branch above exists. Requiring both sides to be
-- non-foreign was tried and rejected: it matched 0 of those 48.
-- The coincidence risk (a domestic tap whose amount happens to
-- equal a foreign charge's AUD figure) is already covered by
-- unique-both-ways, which refuses as soon as a second candidate
-- exists.
(m.foreign_currency_code IS NULL
AND t.amount = m.amount)
)
JOIN statements s ON s.id = t.statement_id
WHERE m.statement_id IS NULL
AND m.owner_id = $1
AND m.reconciled_with_id IS NULL
AND m.superseded_by_id IS NULL
AND m.voided_at IS NULL
AND (m.payment_method IS NULL OR m.payment_method NOT IN ('cash', 'credits'))
AND t.superseded_by_id IS NULL
AND COALESCE(t.owner_id, s.owner_id) = $1
-- The statement row must be unclaimed.
AND NOT EXISTS (SELECT 1 FROM transactions mt WHERE mt.reconciled_with_id = t.id)
)
SELECT manual_id, statement_tx_id, amount, manual_date, statement_date, description, basis
FROM pairs p
WHERE (SELECT count(*) FROM pairs q WHERE q.manual_id = p.manual_id) = 1
AND (SELECT count(*) FROM pairs q WHERE q.statement_tx_id = p.statement_tx_id) = 1
ORDER BY manual_id`,
[ownerId, windowDays]
);
}
/**
* An existing statementless transaction that a newly-arriving receipt or order
* describes -- i.e. the row the wallet lane already created for this purchase.
*
* WHY: signals for one purchase arrive in sequence, not in competition. The
* wallet notification lands within seconds of the tap; the receipt is shared
* hours or days later. Measured on txn 6858: the wallet event arrived 35h25m
* before anything created a transaction, and pantry then created one from
* scratch -- so the same Woolworths shop would have had two rows. The receipt
* must adopt the row that already exists and enrich it.
*
* Deliberately narrow, for the same reason findAutoMatches() is: exact amount,
* a few days either side, unclaimed, and UNIQUE. Two candidates means two
* plausible purchases, and inventing a link between the wrong pair moves one
* shop's line items onto another shop's row. Returning null is always safe --
* the caller falls back to creating its own row, which is today's behaviour.
*/
/**
* The row a newly-arriving WALLET notification describes -- the mirror of
* findAdoptableTransaction, for the case where the signals arrived in the other
* order.
*
* WHY: the sequencing this estate is built on -- wallet first, receipt second,
* statement last -- is a tendency, not a guarantee. The wallet lane can be
* parked (its first 21 rows were backfilled days late), the interpreter runs
* every 15 minutes, the phone can be offline. Whenever a receipt reaches the
* ledger before the notification for the same tap, the wallet writer used to
* insert unconditionally: measured on the 2026-08-18 Woolworths shop, txn 6858
* (pantry, 23 items) and txn 6927 (wallet) were the same $103.30 purchase, and
* the second sat in the reconcile queue waiting for a statement line that the
* first had already claimed.
*
* Deliberately NOT the same predicate as findAdoptableTransaction, in the one
* way that matters: that one refuses a row which already carries line items,
* because for a receipt such a row is a DIFFERENT purchase that happens to
* share an amount. Here it is the opposite -- an itemised row is the receipt we
* are trying to fold into, and refusing it is the bug.
*
* Losing that guard means the merchant has to supply it. A wallet row adopts
* only when the merchant agrees (the same whole-word containment the intake
* matcher uses: "Woolworths" vs "WOOLWORTHS 3345"), because amount-and-date
* alone would let a notification silently disappear into an unrelated purchase
* of the same value -- and a wallet row that is never written is a tap with no
* trace, which is worse than a visible duplicate. When in doubt, return null:
* the caller then writes its own row, which is the pre-existing behaviour.
*/
export async function findWalletAdoptableTransaction(
ownerId: number,
amount: number,
date: string,
merchant: string | null,
windowDays: number = AUTO_MATCH_WINDOW_DAYS,
// wallet-ingestion calls this while holding its advisory lock, and must pass
// that transaction's connection: borrowing a second pooled one per request
// is how a burst of concurrent posts exhausts the pool and deadlocks.
db: Pick<Queryable, "queryRaw"> = { queryRaw }
): Promise<{ id: number; metadata_id: number | null } | null> {
// No usable merchant, no adoption. "Card purchase" is wallet-ingestion's own
// fallback for a notification that named no merchant at all.
if (!merchant || merchant === "Card purchase") return null;
const rows = await db.queryRaw<{ id: number; metadata_id: number | null; merchant: string | null }>(
`SELECT t.id, em.id AS metadata_id,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant
FROM transactions t
LEFT JOIN expense_metadata em ON em.transaction_id = t.id
WHERE t.statement_id IS NULL
AND t.owner_id = $1
AND t.amount = $2::numeric
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
AND t.reconciled_with_id IS NULL
AND t.superseded_by_id IS NULL
-- A voided row is a charge that never posted; a real charge for the same
-- amount (a rebooking after a released hold) must not fold into it.
AND t.voided_at IS NULL
-- A wallet row is the OTHER HALF of this tap, not a purchase to fold
-- into; findPairedRow() merges those, with different rules.
AND COALESCE(t.source, '') <> 'wallet'
LIMIT 5`,
[ownerId, amount, date, windowDays]
);
const hits = rows.filter((r) => merchantMatches(r.merchant, merchant));
return hits.length === 1 ? { id: hits[0].id, metadata_id: hits[0].metadata_id } : null;
}
export async function findAdoptableTransaction(
ownerId: number,
amount: number,
date: string,
windowDays: number = AUTO_MATCH_WINDOW_DAYS
): Promise<number | null> {
const rows = await queryRaw<{ id: number }>(
`SELECT t.id
FROM transactions t
WHERE t.statement_id IS NULL
AND t.owner_id = $1
AND t.amount = $2::numeric
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
AND t.reconciled_with_id IS NULL
AND t.superseded_by_id IS NULL
-- A voided row is a charge that never posted; a real charge for the same
-- amount (a rebooking after a released hold) must not fold into it.
AND t.voided_at IS NULL
-- Never adopt a row that another source already fully described; that
-- is a different purchase that happens to share an amount.
AND NOT EXISTS (
SELECT 1 FROM expense_metadata em
WHERE (em.transaction_id = t.id OR em.matched_transaction_id = t.id)
AND jsonb_typeof(em.line_items) = 'array'
AND jsonb_array_length(em.line_items) > 0
)
LIMIT 2`,
[ownerId, amount, date, windowDays]
);
return rows.length === 1 ? rows[0].id : null;
}
-35
View File
@@ -20,38 +20,3 @@ export async function queryRow<T>(sql: string, params: unknown[] = []): Promise<
const rows = await prisma.$queryRawUnsafe<T[]>(sql, ...params); const rows = await prisma.$queryRawUnsafe<T[]>(sql, ...params);
return rows.length > 0 ? rows[0] : null; return rows.length > 0 ? rows[0] : null;
} }
/** The same two calls, scoped to one connection inside one transaction. */
export type Queryable = {
queryRaw<T>(sql: string, params?: unknown[]): Promise<T[]>;
queryRow<T>(sql: string, params?: unknown[]): Promise<T | null>;
};
/**
* Run several statements as one transaction, on one connection.
*
* The module-level `queryRaw`/`queryRow` are `$queryRawUnsafe` on a POOLED
* client, so each call is its own implicit transaction on whichever connection
* the pool hands out. That makes a check-then-write pair non-atomic, and it
* makes `pg_advisory_xact_lock` useless outside this helper — the lock is
* released the instant the statement that took it commits, which is before the
* next statement runs. Anything that needs a lock to still be held on its next
* query has to be in here.
*/
export async function withTransaction<T>(
fn: (q: Queryable) => Promise<T>,
// Prisma's defaults (maxWait 2s, timeout 5s) count time spent WAITING on an
// advisory lock, so a caller that serialises a burst on one needs more.
opts?: { maxWait?: number; timeout?: number }
): Promise<T> {
return prisma.$transaction(async (tx) => {
const q: Queryable = {
queryRaw: <R>(sql: string, params: unknown[] = []) => tx.$queryRawUnsafe<R[]>(sql, ...params),
queryRow: async <R>(sql: string, params: unknown[] = []) => {
const rows = await tx.$queryRawUnsafe<R[]>(sql, ...params);
return rows.length > 0 ? rows[0] : null;
},
};
return fn(q);
}, opts);
}
-55
View File
@@ -177,40 +177,6 @@ export function useUpdateTransaction() {
}); });
} }
export function useMergeTransaction() {
const qc = useQueryClient();
return useMutation({
mutationFn: async ({
id,
other_id,
mode,
}: {
id: number;
other_id: number;
mode: "reconcile" | "supersede";
}) => {
const res = await fetch(`/api/transactions/${id}/merge`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ other_id, mode }),
});
if (!res.ok) {
const body = await res.json().catch(() => ({}));
throw new Error(body.error || "Failed to merge transaction");
}
return res.json();
},
onSuccess: () => {
qc.invalidateQueries({ queryKey: ["transactions"] });
qc.invalidateQueries({ queryKey: ["transaction"] });
qc.invalidateQueries({ queryKey: ["splits"] });
qc.invalidateQueries({ queryKey: ["analytics"] });
qc.invalidateQueries({ queryKey: ["participant-balances"] });
qc.invalidateQueries({ queryKey: ["shared-transactions"] });
},
});
}
export function useBulkAction() { export function useBulkAction() {
const qc = useQueryClient(); const qc = useQueryClient();
return useMutation({ return useMutation({
@@ -1162,27 +1128,6 @@ export function useReconcile() {
}); });
} }
/** Mark a statementless row as a charge that never posted (migration 0040). */
export function useVoidTransaction() {
const qc = useQueryClient();
return useMutation({
mutationFn: async ({ id, reason }: { id: number; reason: string }) => {
const res = await fetch(`/api/transactions/${id}/void`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ reason }),
});
if (!res.ok) throw new Error((await res.json()).error || "Void failed");
return res.json();
},
onSuccess: () => {
qc.invalidateQueries({ queryKey: ["transactions"] });
qc.invalidateQueries({ queryKey: ["reconcile-pending"] });
qc.invalidateQueries({ queryKey: ["analytics"] });
},
});
}
export function useMerchantTransactions(merchant: string | null) { export function useMerchantTransactions(merchant: string | null) {
return useQuery<{ transactions: MerchantTxnRow[] }>({ return useQuery<{ transactions: MerchantTxnRow[] }>({
queryKey: ["analytics", "merchant-txns", merchant], queryKey: ["analytics", "merchant-txns", merchant],
+2 -3
View File
@@ -58,7 +58,6 @@ const ctxFor = (batch: BatchRow): RouteContext | null =>
platform: batch.header.platform, platform: batch.header.platform,
header: batch.header, header: batch.header,
items: batch.items ?? [], items: batch.items ?? [],
currency: batch.header.currency,
} }
: null; : null;
@@ -80,13 +79,13 @@ export async function handleIntakeAction(
if (actionId === "intake_use_candidate") { if (actionId === "intake_use_candidate") {
if (!Number.isInteger(value.transaction_id)) return note("That button carried no transaction id."); if (!Number.isInteger(value.transaction_id)) return note("That button carried no transaction id.");
const txnId = value.transaction_id as number; const txnId = value.transaction_id as number;
await replaceLineItems(txnId, ctx.items, { source: "intake", platform: ctx.platform, batchId: batch.id, currency: ctx.currency }); await replaceLineItems(txnId, ctx.items, { source: "intake", platform: ctx.platform, batchId: batch.id });
await queryRaw( await queryRaw(
`UPDATE intake_batches SET status = 'enriched', kind = COALESCE(kind, 'food'), transaction_id = $2, resolved_at = now() WHERE id = $1::uuid`, `UPDATE intake_batches SET status = 'enriched', kind = COALESCE(kind, 'food'), transaction_id = $2, resolved_at = now() WHERE id = $1::uuid`,
[batch.id, txnId] [batch.id, txnId]
); );
return { return {
...enrichedCard({ batchId: batch.id, merchant: ctx.header.merchant, total: ctx.header.total, orderDate: ctx.header.order_date, platform: ctx.platform, itemCount: ctx.items.length, currency: ctx.currency, transactionId: txnId }), ...enrichedCard({ batchId: batch.id, merchant: ctx.header.merchant, total: ctx.header.total, orderDate: ctx.header.order_date, platform: ctx.platform, itemCount: ctx.items.length, transactionId: txnId }),
replace: true, replace: true,
}; };
} }
+20 -90
View File
@@ -12,7 +12,7 @@ import { platformLabel } from "@/lib/intake-rules";
export type Block = Record<string, unknown>; export type Block = Record<string, unknown>;
export type Candidate = { transaction_id: number; description: string | null; transaction_date: string; amount: string | number; currency?: string | null }; export type Candidate = { transaction_id: number; description: string | null; transaction_date: string; amount: string | number };
export function financePublicUrl(): string { export function financePublicUrl(): string {
return (process.env.FINANCE_PUBLIC_URL || "https://finance.bosecamp.com").replace(/\/$/, ""); return (process.env.FINANCE_PUBLIC_URL || "https://finance.bosecamp.com").replace(/\/$/, "");
@@ -23,29 +23,7 @@ export function transactionLink(merchant: string | null): string {
return `${financePublicUrl()}/transactions?q=${encodeURIComponent(merchant ?? "")}`; return `${financePublicUrl()}/transactions?q=${encodeURIComponent(merchant ?? "")}`;
} }
/** const money = (n: number | string) => `$${Number(n).toFixed(2)}`;
* Slack renders `&`, `<` and `>` as markup in mrkdwn (and in the `text`
* fallback). Merchant names, transaction descriptions and model output are
* none of ours — `Ben & Jerry's` must not arrive as `Ben &amp; Jerry's`, and a
* description containing `<http://…|click>` must not become a link the card
* never meant to show. Applied to interpolated values only; our own literal
* `*bold*` and `:emoji:` stay markup.
*/
export function esc(value: string | null | undefined): string {
return (value ?? "").replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
}
/**
* `$` is only right for AUD. An intake batch carries the currency the receipt
* printed, and the order nudge already renders `USD 15.00` rather than
* pretending a foreign order was dollars.
*/
const money = (n: number | string, currency?: string | null) => {
const amount = Number(n).toFixed(2);
// The currency comes off the model (`parseExtraction` accepts any string), so
// an unescaped `<@U123>` here would be a mention on every card.
return !currency || currency === "AUD" ? `$${amount}` : `${esc(currency)} ${amount}`;
};
const section = (text: string): Block => ({ type: "section", text: { type: "mrkdwn", text } }); const section = (text: string): Block => ({ type: "section", text: { type: "mrkdwn", text } });
const button = (action_id: string, text: string, value: unknown, style?: "primary" | "danger"): Block => ({ const button = (action_id: string, text: string, value: unknown, style?: "primary" | "danger"): Block => ({
type: "button", type: "button",
@@ -62,12 +40,10 @@ export type CardInput = {
orderDate: string | null; orderDate: string | null;
platform: string | null; platform: string | null;
itemCount: number; itemCount: number;
currency?: string | null;
}; };
export function enrichedCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } { export function enrichedCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
const merchant = esc(c.merchant); const what = [c.total !== null ? money(c.total) : null, platformLabel(c.platform) || c.merchant, c.platform && c.merchant ? `(${c.merchant})` : null]
const what = [c.total !== null ? money(c.total, c.currency) : null, esc(platformLabel(c.platform)) || merchant, c.platform && merchant ? `(${merchant})` : null]
.filter(Boolean) .filter(Boolean)
.join(" "); .join(" ");
const text = `Added ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} to your ${what} order ✓`; const text = `Added ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} to your ${what} order ✓`;
@@ -84,7 +60,7 @@ export function enrichedCard(c: CardInput & { transactionId: number }): { blocks
} }
export function createdCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } { export function createdCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
const text = `Created a ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant) || "new"} transaction with ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} ✓`; const text = `Created a ${c.total !== null ? money(c.total) + " " : ""}${c.merchant ?? "new"} transaction with ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} ✓`;
return { return {
text, text,
blocks: [ blocks: [
@@ -98,83 +74,42 @@ export function createdCard(c: CardInput & { transactionId: number }): { blocks:
} }
export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks: Block[]; text: string } { export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks: Block[]; text: string } {
const label = esc(platformLabel(c.platform)); const label = platformLabel(c.platform) || "";
const head = `No matching ${label ? label + " " : ""}order for *${esc(c.merchant) || "this merchant"}* ${c.total !== null ? money(c.total, c.currency) : ""} on ${esc(c.orderDate) || "?"}.`; const head = `No matching ${label ? label + " " : ""}order for *${c.merchant ?? "this merchant"}* ${c.total !== null ? money(c.total) : ""} on ${c.orderDate ?? "?"}.`;
const text = `${head.replace(/\*/g, "")} ${candidates.length ? "Pick one below or create it." : "Create it?"}`; const text = `${head.replace(/\*/g, "")} ${candidates.length ? "Pick one below or create it." : "Create it?"}`;
const blocks: Block[] = [section(`:mag: ${head}${candidates.length ? "\nIs it one of these?" : ""}`)]; const blocks: Block[] = [section(`:mag: ${head}${candidates.length ? "\nIs it one of these?" : ""}`)];
for (const cand of candidates) { for (const cand of candidates) {
blocks.push({ blocks.push({
type: "section", type: "section",
text: { type: "mrkdwn", text: `${esc(cand.transaction_date)} · ${esc(cand.description) || "—"} · ${money(cand.amount, cand.currency)}` }, text: { type: "mrkdwn", text: `${cand.transaction_date} · ${cand.description ?? "—"} · ${money(cand.amount)}` },
accessory: button("intake_use_candidate", "Use this", { batch_id: c.batchId, transaction_id: cand.transaction_id }), accessory: button("intake_use_candidate", "Use this", { batch_id: c.batchId, transaction_id: cand.transaction_id }),
}); });
} }
blocks.push({ blocks.push({
type: "actions", type: "actions",
block_id: "intake_create", block_id: "intake_create",
// plain_text disables *formatting*, not entity decoding: `&`, `<` and `>` elements: [button("intake_create", `Create new ${c.total !== null ? money(c.total) + " " : ""}${c.merchant ?? ""} transaction`.replace(/\s+/g, " ").trim(), { batch_id: c.batchId }, "primary")],
// are control characters everywhere in a Slack payload, so the label is
// escaped like any other value we did not write.
elements: [button("intake_create", `Create new ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant)} transaction`.replace(/\s+/g, " ").trim(), { batch_id: c.batchId }, "primary")],
}); });
return { blocks, text }; return { blocks, text };
} }
export function forwardedCard( export function forwardedCard(c: CardInput & { captureId: number; enriching: boolean }): { blocks: Block[]; text: string } {
c: CardInput & { const text = `${c.merchant ?? "Grocery receipt"} ${c.total !== null ? money(c.total) : ""} → pantry board (${c.itemCount} item${c.itemCount === 1 ? "" : "s"}) — finance ${c.enriching ? "updates" : "books it"} on confirm`.replace(/\s+/g, " ");
captureId: number;
enriching: boolean;
/** Pantry already holds this exact receipt (same barcode tuple). */
alreadyCaptured?: { captureId: number; confirmed: boolean } | null;
/** Finance found more than one row this could be, so it picked none. */
ambiguous?: boolean;
}
): { blocks: Block[]; text: string } {
const what = `${esc(c.merchant) || "Grocery receipt"} ${c.total !== null ? money(c.total, c.currency) : ""}`.trim();
const items = `${c.itemCount} item${c.itemCount === 1 ? "" : "s"}`;
// The warning comes FIRST and the emoji changes, because this card is read at
// a glance on a phone: a duplicate warning appended to a cheerful "→ pantry
// board" line is a duplicate warning nobody reads. Confirming both copies
// counts the same groceries twice — pantry's addItem accumulates — so the
// card has to say which capture to dismiss, not merely that there are two.
if (c.alreadyCaptured) {
const state = c.alreadyCaptured.confirmed ? "already confirmed" : "still waiting to be confirmed";
const text = `Already on the pantry board — ${what} is capture ${c.alreadyCaptured.captureId} (${state}). This upload is capture ${c.captureId}; dismiss it unless it replaces the first, because confirming both counts the same groceries twice.`.replace(/\s+/g, " ");
return { text, blocks: [section(`:repeat: ${text}`)] };
}
const tail = c.ambiguous
? "— finance found more than one row it could be, so it picked none; reconcile it there"
: `— finance ${c.enriching ? "updates" : "books it"} on confirm`;
const text = `${what} → pantry board (${items}) ${tail}`.replace(/\s+/g, " ");
return { text, blocks: [section(`:shopping_trolley: ${text}`)] }; return { text, blocks: [section(`:shopping_trolley: ${text}`)] };
} }
/** export function needsHeaderCard(): { blocks: Block[]; text: string } {
* Names what is missing, because the advice differs. A delivery-app screenshot const text = "I can see items but no order total or date — share the top of the screen too so I can match it.";
* that lost its header needs the top of the screen; a Woolworths eReceipt
* SCREENSHOT has no date anywhere on it (seen 2026-08-22 — store, total, items,
* payment, and not one date), while the app's own Share button sends the raw
* receipt WITH the date. So for a dateless in-store receipt the right advice is
* the Share button, or pantry's own upload — not "share more of the screen".
*/
export function needsHeaderCard(missing: string[] = ["total", "date"], source?: string | null): { blocks: Block[]; text: string } {
const what = missing.length ? missing.join(" or ") : "total or date";
const inStore = source === "coles" || source === "woolworths";
const hint =
missing.length === 1 && missing[0] === "date" && inStore
? "an app screenshot carries no date — use the app's Share button instead, or upload the picture to pantry directly"
: "share the part of the screen that shows it";
const text = `I can see items but no ${esc(what)} — ${hint}, so I can match it.`;
return { text, blocks: [section(`:warning: ${text}`)] }; return { text, blocks: [section(`:warning: ${text}`)] };
} }
export function heldCard(reason: string, headline = "Items don't add up to the total — is a shot missing?"): { blocks: Block[]; text: string } { export function heldCard(reason: string): { blocks: Block[]; text: string } {
const text = `${headline} (${esc(reason)})`; const text = `Items don't add up to the total — is a shot missing? (${reason})`;
return { text, blocks: [section(`:warning: ${text}`)] }; return { text, blocks: [section(`:warning: ${text}`)] };
} }
export function askCard(c: CardInput): { blocks: Block[]; text: string } { export function askCard(c: CardInput): { blocks: Block[]; text: string } {
const text = `${esc(c.merchant) || "This upload"} ${c.total !== null ? money(c.total, c.currency) : ""} — groceries or a meal?`.replace(/\s+/g, " "); const text = `${c.merchant ?? "This upload"} ${c.total !== null ? money(c.total) : ""} — groceries or a meal?`.replace(/\s+/g, " ");
return { return {
text, text,
blocks: [ blocks: [
@@ -191,23 +126,18 @@ export function askCard(c: CardInput): { blocks: Block[]; text: string } {
}; };
} }
export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null; currency?: string | null }): { blocks: Block[]; text: string } { export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null }): { blocks: Block[]; text: string } {
const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${esc(c.batchId)}` : "an earlier upload"; const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${c.batchId}` : "an earlier upload";
// The header (merchant, date, total) IS the identity of a purchase here, so a const text = `Already handled — ${c.merchant ?? "this order"} ${c.total !== null ? money(c.total) : ""} is ${where}. Nothing was changed.`.replace(/\s+/g, " ");
// genuine SECOND identical purchase — two of the same coffee, one day, one shop
// — reads as a re-share and is refused. That is the right default (a re-share is
// far commoner than a repeat) but it is a dead end without this sentence: the
// card must say what to do instead, or the ledger quietly loses the second one.
const text = `Already handled — ${esc(c.merchant) || "this order"} ${c.total !== null ? money(c.total, c.currency) : ""} is ${where}. Nothing was changed — if this really is a second, separate purchase, add it in the app.`.replace(/\s+/g, " ");
return { text, blocks: [section(`:repeat: ${text}`)] }; return { text, blocks: [section(`:repeat: ${text}`)] };
} }
export function unsupportedCard(detail?: string): { blocks: Block[]; text: string } { export function unsupportedCard(detail?: string): { blocks: Block[]; text: string } {
const text = `I can read receipts, statements and order screens — not this.${detail ? ` (${esc(detail)})` : ""}`; const text = `I can read receipts, statements and order screens — not this.${detail ? ` (${detail})` : ""}`;
return { text, blocks: [section(`:no_entry_sign: ${text}`)] }; return { text, blocks: [section(`:no_entry_sign: ${text}`)] };
} }
export function errorCard(detail: string): { blocks: Block[]; text: string } { export function errorCard(detail: string): { blocks: Block[]; text: string } {
const text = `Couldn't process that upload: ${esc(detail)}`; const text = `Couldn't process that upload: ${detail}`;
return { text, blocks: [section(`:x: ${text}`)] }; return { text, blocks: [section(`:x: ${text}`)] };
} }
+5 -31
View File
@@ -1,5 +1,5 @@
import { extractText, getDocumentProxy } from "unpdf"; import { extractText, getDocumentProxy } from "unpdf";
import { PROMPT_PLACEHOLDER_MERCHANTS, type ExtractedOrder, type ExtractedItem } from "@/lib/intake-rules"; import type { ExtractedOrder, ExtractedItem } from "@/lib/intake-rules";
/** /**
* Extraction for the intake router: one joint model call per batch. * Extraction for the intake router: one joint model call per batch.
@@ -31,22 +31,13 @@ export async function extractPdfText(bytes: Buffer): Promise<string> {
return (Array.isArray(text) ? text.join("\n") : text).trim(); return (Array.isArray(text) ? text.join("\n") : text).trim();
} }
/**
* The merchant examples are FORMAT PLACEHOLDERS, never real shops
* (`PROMPT_PLACEHOLDER_MERCHANTS`). Given an input with nothing in it, a model
* still answers, and it answers out of its prompt: on 2026-08-22 the word
* `Image` produced a complete Burgertory receipt because the prompt named
* Burgertory. The "not a receipt" rule below and the placeholder names are the
* two halves of closing that.
*/
export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text. export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text.
Return ONE JSON object with exactly these keys: Return ONE JSON object with exactly these keys:
{ "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] } { "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] }
Rules: Rules:
- If what you are given is NOT a receipt or an order — it is blank, a filename, a single word such as "Image", an app screen with no purchase on it, or anything else you cannot read a purchase off — return { "merchant": null, "order_date": null, "total": null, "subtotal": null, "currency": null, "items": [], "adjustments": [] }. Do not construct a plausible order. Nothing in these instructions is data: never return an example from this prompt as if it were printed on the input. - merchant: the store or restaurant name exactly as printed, including any branch in parentheses (e.g. "Coles (Wyndham Vale)", "Burgertory (Werribee)"). Never the delivery platform. null if not shown.
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (format: "${PROMPT_PLACEHOLDER_MERCHANTS[0]}"). Never the delivery platform. null if not shown.
- order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown. - order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown.
- total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown. - total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown.
- subtotal: the printed item subtotal before fees and discounts, as a number, or null. - subtotal: the printed item subtotal before fees and discounts, as a number, or null.
@@ -194,26 +185,9 @@ export function liteLLMExtractor(opts: {
}); });
if (!response.ok) throw new ExtractionError(`LiteLLM returned ${response.status}`); if (!response.ok) throw new ExtractionError(`LiteLLM returned ${response.status}`);
const raw = (await response.json()) as { choices?: { message?: { content?: string }; finish_reason?: string }[] }; const raw = (await response.json()) as { choices?: { message?: { content?: string }; finish_reason?: string }[] };
// The rule is: a finish_reason that is PRESENT must be `stop`. An absent // A truncated reply can still parse once the JSON is closed, which quietly
// one is accepted, deliberately — providers differ on whether they send the // drops the tail of a long receipt. Losing lines silently is worse than failing.
// field, and failing every call on a provider that omits it would be a if (raw.choices?.[0]?.finish_reason === "length") throw new ExtractionError("The receipt was too long to read in one pass.");
// certain outage in exchange for a case that also requires the model to
// emit syntactically complete JSON. The harm a truncated reply does — a
// partial item list replacing a good one — is closed on the write side
// instead, by the guard in `routeFood`/`createFromBatch`.
//
// `length` is the common one: a truncated reply still parses once the JSON
// happens to close, quietly dropping the tail of a long receipt. Any other
// reason (`content_filter`, `tool_calls`) is a different reply shape that
// the strict schema never constrained.
const finish = raw.choices?.[0]?.finish_reason;
if (finish && finish !== "stop") {
throw new ExtractionError(
finish === "length"
? "The receipt was too long to read in one pass."
: `The model stopped before it finished (${finish}).`
);
}
const content = raw.choices?.[0]?.message?.content; const content = raw.choices?.[0]?.message?.content;
if (!content) throw new ExtractionError("LiteLLM returned no content"); if (!content) throw new ExtractionError("LiteLLM returned no content");
try { try {
+3 -28
View File
@@ -20,21 +20,7 @@ export type PantryForwardArgs = {
financeAction: "enrich" | "create"; financeAction: "enrich" | "create";
}; };
/** export type PantryForwardResult = { captureId: number; itemCount: number | null };
* `alreadyCaptured` is pantry telling us this exact receipt is already on its
* board -- same store, register, receipt number and date, which is the tuple
* both Coles and Woolworths encode in their barcode. Pantry deliberately still
* ACCEPTS the second capture (the first may have been a bad crop being
* replaced) and says so instead of refusing; dropping the field on the floor
* turned "said so" into silence, and a duplicate that is not shown is a
* duplicate that gets confirmed -- which double-counts the groceries, because
* pantry's addItem accumulates.
*/
export type PantryForwardResult = {
captureId: number;
itemCount: number | null;
alreadyCaptured: { captureId: number; confirmed: boolean } | null;
};
export class PantryDuplicateError extends Error { export class PantryDuplicateError extends Error {
constructor( constructor(
@@ -79,23 +65,12 @@ export function pantryForwarder(opts: { baseUrl?: string; token?: string; fetchI
} catch (e) { } catch (e) {
throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`); throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`);
} }
const body = (await response.json().catch(() => ({}))) as { const body = (await response.json().catch(() => ({}))) as { captureId?: number; error?: string; items?: unknown[]; reason?: string };
captureId?: number; error?: string; items?: unknown[]; reason?: string;
alreadyCaptured?: { captureId?: number; confirmed?: boolean } | null;
};
if (response.status === 409) { if (response.status === 409) {
throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured"); throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured");
} }
if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim()); if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim());
if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId"); if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId");
const prior = body.alreadyCaptured; return { captureId: body.captureId, itemCount: Array.isArray(body.items) ? body.items.length : null };
return {
captureId: body.captureId,
itemCount: Array.isArray(body.items) ? body.items.length : null,
alreadyCaptured:
prior && typeof prior.captureId === "number" && prior.captureId !== body.captureId
? { captureId: prior.captureId, confirmed: prior.confirmed === true }
: null,
};
}; };
} }
+47 -273
View File
@@ -1,14 +1,9 @@
import { randomUUID } from "node:crypto"; import { randomUUID } from "node:crypto";
import { queryRaw, queryRow, withTransaction } from "@/lib/db"; import { queryRaw, queryRow } from "@/lib/db";
import { EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY } from "@/lib/analytics-sql";
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion"; import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
import { import {
classifyInput, classifyInput,
confabulationReason,
merchantKey, merchantKey,
quoteShared,
textCouldBeReceipt,
merchantMatches, merchantMatches,
platformForSource, platformForSource,
reconcileItems, reconcileItems,
@@ -212,33 +207,15 @@ export async function findDuplicateBatch(
return rows.find((r) => merchantMatches(r.header?.merchant, header.merchant)) ?? null; return rows.find((r) => merchantMatches(r.header?.merchant, header.merchant)) ?? null;
} }
type MatchRow = { type MatchRow = { transaction_id: number; description: string | null; transaction_date: string; amount: string; merchant: string | null; platform: string | null };
transaction_id: number;
description: string | null;
transaction_date: string;
amount: string;
merchant: string | null;
platform: string | null;
/** What `amount` is denominated in — a candidate row is not always dollars. */
currency: string;
/** The `expense_metadata` row the match resolved through — null when there is none. */
metadata_id: number | null;
/** Does that row already carry a receipt? */
has_line_items: boolean;
};
/** /**
* The match rule (plan §5, D2): `(platform, merchant, date, total)`, amount and * The match rule (plan §5, D2): `(platform, merchant, date, total)`, amount and
* date EXACT, never widened silently. The merchant compares normalised, with * date EXACT, never widened silently. Platform is required when the source
* whole-word containment ("Coles" vs "Coles (Wyndham Vale)") — that is * implies one; an in-store receipt has none and matches on the other three.
* spelling, not widening. A unique hit is a match; two hits are an ambiguity * The merchant compares normalised, with containment ("Coles" vs "Coles
* and go to the candidates card like a miss. * (Wyndham Vale)") — that is spelling, not widening. A unique hit is a match;
* * two hits are an ambiguity and go to the candidates card like a miss.
* A NULL platform means "no platform", not "any platform". An in-store Coles
* receipt carries none, and `$3 IS NULL OR em.platform = $3` let it match an
* Uber Eats Coles order on the same day for the same money — and then overwrite
* that order's items with the till receipt's. The in-store side matches rows
* with no platform of their own; the delivery side still requires its own.
*/ */
export async function matchTransaction(header: OrderHeader, platform: string | null): Promise<{ match: MatchRow | null; ambiguous: boolean }> { export async function matchTransaction(header: OrderHeader, platform: string | null): Promise<{ match: MatchRow | null; ambiguous: boolean }> {
if (!header.merchant) return { match: null, ambiguous: false }; if (!header.merchant) return { match: null, ambiguous: false };
@@ -246,15 +223,13 @@ export async function matchTransaction(header: OrderHeader, platform: string | n
`SELECT DISTINCT ON (t.id) `SELECT DISTINCT ON (t.id)
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text, t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant, COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency, em.platform
COALESCE(jsonb_typeof(em.line_items) = 'array' AND jsonb_array_length(em.line_items) > 0, false) AS has_line_items
FROM transactions t FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
WHERE ${EXCLUDE_RECONCILED_SOURCE} WHERE t.superseded_by_id IS NULL
AND t.transaction_date = $1::date AND t.transaction_date = $1::date
AND t.amount = $2::numeric AND t.amount = $2::numeric
AND (($3::text IS NULL AND em.platform IS NULL) OR em.platform = $3) AND ($3::text IS NULL OR em.platform = $3)
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`, ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
[header.order_date, header.total, platform] [header.order_date, header.total, platform]
); );
@@ -263,29 +238,21 @@ export async function matchTransaction(header: OrderHeader, platform: string | n
return { match: null, ambiguous: hits.length > 1 }; return { match: null, ambiguous: hits.length > 1 };
} }
/** /** Nearest candidates for the card: same platform and merchant, ±3 days, any amount. */
* Nearest candidates for the card: same merchant, ±3 days, any amount. export async function findCandidates(header: OrderHeader, platform: string | null): Promise<Candidate[]> {
*
* Deliberately NOT filtered by platform. The card is a human choice, not a
* decision — the whole reason it is shown is that the exact key missed, so the
* platform is one of the things that may be wrong. A filter here narrows the
* list at exactly the moment it needs to widen, and the row the user was
* looking for is simply absent with nothing saying why.
*/
export async function findCandidates(header: OrderHeader): Promise<Candidate[]> {
if (!header.merchant) return []; if (!header.merchant) return [];
const rows = await queryRaw<MatchRow>( const rows = await queryRaw<MatchRow>(
`SELECT DISTINCT ON (t.id) `SELECT DISTINCT ON (t.id)
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text, t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant, COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency, false AS has_line_items em.platform
FROM transactions t FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
WHERE ${EXCLUDE_RECONCILED_SOURCE} WHERE t.superseded_by_id IS NULL
AND t.transaction_date BETWEEN $1::date - 3 AND $1::date + 3 AND t.transaction_date BETWEEN $1::date - 3 AND $1::date + 3
AND ($2::text IS NULL OR em.platform = $2)
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`, ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
[header.order_date] [header.order_date, platform]
); );
return rows return rows
.filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant)) .filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant))
@@ -296,7 +263,7 @@ export async function findCandidates(header: OrderHeader): Promise<Candidate[]>
return Math.abs(Number(a.amount) - header.total) - Math.abs(Number(b.amount) - header.total); return Math.abs(Number(a.amount) - header.total) - Math.abs(Number(b.amount) - header.total);
}) })
.slice(0, 5) .slice(0, 5)
.map((r) => ({ transaction_id: r.transaction_id, description: r.description, transaction_date: r.transaction_date, amount: r.amount, currency: r.currency })); .map((r) => ({ transaction_id: r.transaction_id, description: r.description, transaction_date: r.transaction_date, amount: r.amount }));
} }
// ─── the two destinations ─────────────────────────────────────────────────── // ─── the two destinations ───────────────────────────────────────────────────
@@ -308,7 +275,7 @@ export type RouteContext = {
platform: string | null; platform: string | null;
header: OrderHeader; header: OrderHeader;
items: StoredLineItem[]; items: StoredLineItem[];
currency?: string | null; subtotal?: number | null;
}; };
const cardInput = (ctx: RouteContext): CardInput => ({ const cardInput = (ctx: RouteContext): CardInput => ({
@@ -318,7 +285,6 @@ const cardInput = (ctx: RouteContext): CardInput => ({
orderDate: ctx.header.order_date, orderDate: ctx.header.order_date,
platform: ctx.platform, platform: ctx.platform,
itemCount: ctx.items.length, itemCount: ctx.items.length,
currency: ctx.currency,
}); });
const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, extra: Partial<IntakeResult> = {}): IntakeResult => ({ const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, extra: Partial<IntakeResult> = {}): IntakeResult => ({
@@ -333,56 +299,22 @@ const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card:
...extra, ...extra,
}); });
/**
* Are these items unverifiable — no line carrying any money at all?
*
* That is `reconcileItems`' `basis: "none"`: pasted text with no prices, or an
* extraction that read the names and none of the figures. The reconciliation
* check passes it, correctly, because there is nothing to check. What must not
* follow is replacing a receipt that IS itemised with one nothing verified.
*/
const unverifiable = (items: StoredLineItem[]) => items.every((i) => i.amount === null);
/**
* The hold, shared by EVERY path that enriches a matched transaction.
*
* It lived only in `routeFood` and `createFromBatch`'s re-match walked straight
* past it — the same upload destroyed the same receipt depending on which
* button reached it. Returns null when there is nothing to protect.
*/
async function holdRatherThanReplace(ctx: RouteContext, match: MatchRow): Promise<IntakeResult | null> {
if (!unverifiable(ctx.items) || !match.has_line_items) return null;
await updateBatch(ctx.batchId, { status: "held", items: ctx.items });
const card = heldCard(
"this upload carries no prices, so nothing checked it against the total — and that order already has its items",
"I left the existing items alone"
);
return finish(ctx, "held", "held", card, { transaction_id: match.transaction_id });
}
/** Food: enrich the matched order's line_items, or show candidates. NEVER auto-creates. */ /** Food: enrich the matched order's line_items, or show candidates. NEVER auto-creates. */
export async function routeFood(ctx: RouteContext): Promise<IntakeResult> { export async function routeFood(ctx: RouteContext): Promise<IntakeResult> {
const { match } = await matchTransaction(ctx.header, ctx.platform); const { match } = await matchTransaction(ctx.header, ctx.platform);
if (match) { if (match) {
const held = await holdRatherThanReplace(ctx, match); await replaceLineItems(match.transaction_id, ctx.items, { source: "intake", platform: ctx.platform, batchId: ctx.batchId });
if (held) return held;
await replaceLineItems(match.transaction_id, ctx.items, { source: "intake", platform: ctx.platform, batchId: ctx.batchId, metadataId: match.metadata_id, currency: ctx.currency });
await updateBatch(ctx.batchId, { kind: "food", status: "enriched", transaction_id: match.transaction_id, items: ctx.items }); await updateBatch(ctx.batchId, { kind: "food", status: "enriched", transaction_id: match.transaction_id, items: ctx.items });
return finish(ctx, "food", "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id }); return finish(ctx, "food", "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
} }
const candidates = await findCandidates(ctx.header); const candidates = await findCandidates(ctx.header, ctx.platform);
await updateBatch(ctx.batchId, { kind: "food", status: "candidates", items: ctx.items }); await updateBatch(ctx.batchId, { kind: "food", status: "candidates", items: ctx.items });
return finish(ctx, "food", "candidates", candidatesCard(cardInput(ctx), candidates), { candidates }); return finish(ctx, "food", "candidates", candidatesCard(cardInput(ctx), candidates), { candidates });
} }
/** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */ /** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */
export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> { export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> {
// `ambiguous` is not a miss. Two ledger rows for one shop is precisely the const { match } = await matchTransaction(ctx.header, ctx.platform);
// state that produced the 2026-08-18 Woolworths double-book, and forwarding
// with financeAction 'create' as though nothing matched is what turns it into
// a third row on confirm. The groceries still go to the board — they have to,
// it is where they are worked — but the card must say finance could not pick.
const { match, ambiguous } = await matchTransaction(ctx.header, ctx.platform);
try { try {
const r = await pantry({ const r = await pantry({
files, files,
@@ -393,19 +325,12 @@ export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantr
financeAction: match ? "enrich" : "create", financeAction: match ? "enrich" : "create",
}); });
await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items }); await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items });
const card = forwardedCard({ const card = forwardedCard({ ...cardInput(ctx), itemCount: r.itemCount ?? ctx.items.length, captureId: r.captureId, enriching: !!match });
...cardInput(ctx),
itemCount: r.itemCount ?? ctx.items.length,
captureId: r.captureId,
enriching: !!match,
alreadyCaptured: r.alreadyCaptured,
ambiguous,
});
return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id }); return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id });
} catch (e) { } catch (e) {
if (e instanceof PantryDuplicateError) { if (e instanceof PantryDuplicateError) {
await updateBatch(ctx.batchId, { kind: "grocery", status: "duplicate", capture_id: e.captureId, items: ctx.items }); await updateBatch(ctx.batchId, { kind: "grocery", status: "duplicate", capture_id: e.captureId, items: ctx.items });
return finish(ctx, "grocery", "duplicate", duplicateCard({ merchant: ctx.header.merchant, total: ctx.header.total, currency: ctx.currency, transactionId: match?.transaction_id ?? null, captureId: e.captureId, batchId: null }), { return finish(ctx, "grocery", "duplicate", duplicateCard({ merchant: ctx.header.merchant, total: ctx.header.total, transactionId: match?.transaction_id ?? null, captureId: e.captureId, batchId: null }), {
capture_id: e.captureId ?? undefined, capture_id: e.captureId ?? undefined,
}); });
} }
@@ -420,144 +345,38 @@ export async function createFromBatch(batch: BatchRow, kind: "grocery" | "food")
if (!batch.header) throw new Error("batch has no header"); if (!batch.header) throw new Error("batch has no header");
const header = batch.header; const header = batch.header;
const items = batch.items ?? []; const items = batch.items ?? [];
const ctx: RouteContext = {
batchId: batch.id,
source: batch.source,
sharedBy: batch.shared_by,
platform: header.platform,
header,
items,
currency: header.currency,
};
// The card was built minutes ago and the tap arrives now. In between, the
// email/order bridge books orders on its own schedule — so re-run the match
// rather than trusting the "no match" the card was drawn from. Creating a
// second row for an order the ledger already holds is the one outcome this
// whole slice exists to avoid.
const { match } = await matchTransaction(header, header.platform);
if (match) {
const held = await holdRatherThanReplace(ctx, match);
if (held) return held;
await replaceLineItems(match.transaction_id, items, { source: "intake", platform: header.platform, batchId: batch.id, metadataId: match.metadata_id, currency: header.currency });
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: match.transaction_id, items });
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
}
// A sibling batch of the same order — the PDF after the screenshot — may have
// been resolved since. `findDuplicateBatch` deliberately ignores unresolved
// batches, so two pending cards for one order both reach here; without this
// the second tap books the order twice.
const twin = await findDuplicateBatch(batch.id, batch.file_sha256, header);
if (twin) {
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: twin.transaction_id, capture_id: twin.capture_id });
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: twin.transaction_id, captureId: twin.capture_id, batchId: twin.id });
return finish(ctx, kind, "duplicate", card, {
transaction_id: twin.transaction_id ?? undefined,
capture_id: twin.capture_id ?? undefined,
duplicate_of: { batch_id: twin.id, transaction_id: twin.transaction_id, capture_id: twin.capture_id },
});
}
const ownerId = await ownerForEmail(batch.shared_by); const ownerId = await ownerForEmail(batch.shared_by);
// Before minting a row: is one already here that nothing has described yet?
//
// matchTransaction above cannot see it. Its key includes the platform, and a
// wallet notification has none — so an Uber Eats screenshot never matches the
// wallet row for the very tap that paid for it, and the Create button books
// the purchase a second time. That is the normal arrival order, not an edge
// case: the notification lands within seconds of the tap and the screenshot
// is shared later.
//
// findAdoptableTransaction is the same helper the receipt lane uses and
// carries the same guards — exact amount, ±3 days, unclaimed, UNIQUE, and
// never a row that already carries line items. Its uniqueness check runs
// outside the advisory lock below, so two simultaneous taps could both adopt;
// the second would then find the items in place and fall through to the lock,
// which is where duplicates are actually resolved.
const adopted = await findAdoptableTransaction(ownerId, header.total, header.order_date);
if (adopted) {
await replaceLineItems(adopted, items, { source: "intake", platform: header.platform, batchId: batch.id, currency: header.currency });
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: adopted, items });
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: adopted }), { transaction_id: adopted });
}
const merchant = header.merchant ?? "Unknown merchant"; const merchant = header.merchant ?? "Unknown merchant";
// Not `orderDescription()`: it always prefixes "Order - ", which is right for
// a delivery receipt and wrong for the in-store till receipt this path also
// handles. Not `resolveCategory()` either — it re-derives grocery-vs-food
// from a merchant pattern, and `kind` is the answer the router already
// reached, remembered merchant and Slack tap included.
const description = header.platform ? `Order - ${merchant}` : merchant; const description = header.platform ? `Order - ${merchant}` : merchant;
const category = kind === "grocery" ? "groceries" : "dining"; const category = kind === "grocery" ? "groceries" : "dining";
// `amount` is the figure printed on the receipt, in its own currency. No FX // source/source_ref make a second press of the same card a no-op rather than a second row.
// rate exists at intake, so a foreign order leaves `amount_aud` NULL rather const existing = await queryRow<{ id: number }>(`SELECT id FROM transactions WHERE source = 'intake' AND source_ref = $1`, [batch.id]);
// than asserting a conversion — the same shape order-ingestion.ts writes. const txnId =
const currency = header.currency ?? "AUD"; existing?.id ??
const isAud = currency === "AUD"; (
// Check and insert under one lock, on one connection. await queryRow<{ id: number }>(
// `INSERT INTO transactions (
// `source_ref = batch.id` makes a second press of THIS card a no-op, and transaction_date, description, amount, amount_aud, category, payment_method,
// ON CONFLICT makes that safe against a double tap. It cannot help two merchant_name, merchant_normalized, transaction_type, owner_id, source, source_ref
// DIFFERENT batches for one purchase — their keys differ — and both taps run ) VALUES ($1::date, $2, $3::numeric, $3::numeric, $4, NULL, $5, $5, 'debit', $6, 'intake', $7)
// their match and duplicate checks before either inserts, so both can find RETURNING id`,
// nothing. The lock is keyed on the purchase, not the batch, which is the [header.order_date, description, header.total, category, merchant, ownerId, batch.id]
// only thing the two share. It has to be transaction-scoped and therefore )
// inside withTransaction: taken by a statement of its own it would be )!.id;
// released before the next one ran (see db.ts). await replaceLineItems(txnId, items, { source: "intake", platform: header.platform, batchId: batch.id });
const purchaseKey = [header.platform ?? "", merchantKey(merchant), header.order_date, header.total.toFixed(2)].join("|");
const outcome = await withTransaction(async (q) => {
await q.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, [purchaseKey]);
// Mine (a double tap) or another batch's (the race). The ORDER BY is what
// tells them apart when both exist.
const existing = await q.queryRow<{ id: number; source_ref: string }>(
`SELECT id, source_ref FROM transactions
WHERE source = 'intake' AND superseded_by_id IS NULL
AND (source_ref = $1
OR (transaction_date = $2::date AND amount = $3::numeric
AND lower(merchant_normalized) = lower($4)))
ORDER BY (source_ref = $1) DESC NULLS LAST, id
LIMIT 1`,
[batch.id, header.order_date, header.total, merchant]
);
if (existing) return { id: existing.id, mine: existing.source_ref === batch.id };
const row = await q.queryRow<{ id: number }>(
`INSERT INTO transactions (
transaction_date, description, amount, amount_aud, category, payment_method,
merchant_name, merchant_normalized, transaction_type, owner_id, source, source_ref,
foreign_currency_amount, foreign_currency_code
) VALUES ($1::date, $2, $3::numeric, $4::numeric, $5, NULL, $6, $6, 'debit', $7, 'intake', $8, $9::numeric, $10)
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL
DO NOTHING
RETURNING id`,
[header.order_date, description, header.total, isAud ? header.total : null, category, merchant, ownerId, batch.id, isAud ? null : header.total, isAud ? null : currency]
);
return { id: row!.id, mine: true };
});
// Another batch booked this purchase in the moment between the two taps. Say
// the same thing the sequential case says one second later, and write nothing.
if (!outcome.mine) {
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: outcome.id });
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: outcome.id, captureId: null, batchId: null });
return finish(ctx, kind, "duplicate", card, { transaction_id: outcome.id });
}
const txnId = outcome.id;
await replaceLineItems(txnId, items, { source: "intake", platform: header.platform, batchId: batch.id, currency });
await updateBatch(batch.id, { kind, status: "created", transaction_id: txnId }); await updateBatch(batch.id, { kind, status: "created", transaction_id: txnId });
const ctx: RouteContext = { batchId: batch.id, source: batch.source, sharedBy: batch.shared_by, platform: header.platform, header, items };
return finish(ctx, kind, "created", createdCard({ ...cardInput(ctx), transactionId: txnId }), { transaction_id: txnId }); return finish(ctx, kind, "created", createdCard({ ...cardInput(ctx), transactionId: txnId }), { transaction_id: txnId });
} }
/** The card for a batch as it stands — what a second press of a resolved card gets back. */ /** The card for a batch as it stands — what a second press of a resolved card gets back. */
export function resolvedCard(batch: BatchRow): { blocks: Block[]; text: string } { export function resolvedCard(batch: BatchRow): { blocks: Block[]; text: string } {
const h = batch.header; const h = batch.header;
const base: CardInput = { batchId: batch.id, merchant: h?.merchant ?? null, total: h?.total ?? null, orderDate: h?.order_date ?? null, platform: h?.platform ?? null, itemCount: batch.items?.length ?? 0, currency: h?.currency }; const base: CardInput = { batchId: batch.id, merchant: h?.merchant ?? null, total: h?.total ?? null, orderDate: h?.order_date ?? null, platform: h?.platform ?? null, itemCount: batch.items?.length ?? 0 };
if (batch.status === "enriched" && batch.transaction_id) return enrichedCard({ ...base, transactionId: batch.transaction_id }); if (batch.status === "enriched" && batch.transaction_id) return enrichedCard({ ...base, transactionId: batch.transaction_id });
if (batch.status === "created" && batch.transaction_id) return createdCard({ ...base, transactionId: batch.transaction_id }); if (batch.status === "created" && batch.transaction_id) return createdCard({ ...base, transactionId: batch.transaction_id });
if (batch.status === "forwarded_to_pantry" && batch.capture_id) return forwardedCard({ ...base, captureId: batch.capture_id, enriching: !!batch.transaction_id }); if (batch.status === "forwarded_to_pantry" && batch.capture_id) return forwardedCard({ ...base, captureId: batch.capture_id, enriching: !!batch.transaction_id });
return duplicateCard({ merchant: base.merchant, total: base.total, currency: base.currency, transactionId: batch.transaction_id, captureId: batch.capture_id, batchId: batch.id }); return duplicateCard({ merchant: base.merchant, total: base.total, transactionId: batch.transaction_id, captureId: batch.capture_id, batchId: batch.id });
} }
// ─── the router ───────────────────────────────────────────────────────────── // ─── the router ─────────────────────────────────────────────────────────────
@@ -587,8 +406,6 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
// 1. classify each file and build the parts, in the order shared. // 1. classify each file and build the parts, in the order shared.
const parts: InputPart[] = []; const parts: InputPart[] = [];
const usable: StoredFile[] = []; const usable: StoredFile[] = [];
/** Text parts dropped before the model saw them — see `textCouldBeReceipt`. */
const rejectedText: { name: string; text: string; reason: string }[] = [];
for (const f of input.files) { for (const f of input.files) {
const kind = classifyInput(f); const kind = classifyInput(f);
if (kind === "pdf") { if (kind === "pdf") {
@@ -606,20 +423,7 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} else if (kind === "image") { } else if (kind === "image") {
parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name }); parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name });
} else if (kind === "text") { } else if (kind === "text") {
// Nothing that cannot be a receipt reaches the model. A share that parts.push({ kind: "text", text: f.bytes.toString("utf-8"), name: f.name });
// arrives as the word "Image" (the phone coerced the photo to text) is
// not a thin receipt, and asking anyway produces a whole invented one —
// 2026-08-22, `textCouldBeReceipt`. Dropped like an unreadable file type
// rather than failing the batch, so one junk text part beside a good
// screenshot still goes through; if nothing usable is left the card below
// quotes what the phone actually sent.
const text = f.bytes.toString("utf-8");
const check = textCouldBeReceipt(text);
if (!check.ok) {
rejectedText.push({ name: f.name, text, reason: check.reason });
continue;
}
parts.push({ kind: "text", text, name: f.name });
} else { } else {
continue; continue;
} }
@@ -627,23 +431,14 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} }
if (!parts.length) { if (!parts.length) {
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" }); await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
const dropped = rejectedText[0]; return fail("unsupported", "unsupported", unsupportedCard());
return fail(
"unsupported",
"unsupported",
unsupportedCard(
dropped
? `the share arrived as text and ${dropped.reason}: "${quoteShared(dropped.text)}" — send the picture or the PDF itself`
: undefined
)
);
} }
// A re-share of the very same files, before spending a model call on it. // A re-share of the very same files, before spending a model call on it.
const dupBySha = await findDuplicateBatch(batchId, shas, null); const dupBySha = await findDuplicateBatch(batchId, shas, null);
if (dupBySha) { if (dupBySha) {
await updateBatch(batchId, { status: "duplicate", transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id }); await updateBatch(batchId, { status: "duplicate", transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id });
return fail("unsupported", "duplicate", duplicateCard({ merchant: dupBySha.header?.merchant ?? null, total: dupBySha.header?.total ?? null, currency: dupBySha.header?.currency, transactionId: dupBySha.transaction_id, captureId: dupBySha.capture_id, batchId: dupBySha.id }), 200, { return fail("unsupported", "duplicate", duplicateCard({ merchant: dupBySha.header?.merchant ?? null, total: dupBySha.header?.total ?? null, transactionId: dupBySha.transaction_id, captureId: dupBySha.capture_id, batchId: dupBySha.id }), 200, {
kind: (dupBySha.kind as IntakeKind) ?? "food", kind: (dupBySha.kind as IntakeKind) ?? "food",
duplicate_of: { batch_id: dupBySha.id, transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id }, duplicate_of: { batch_id: dupBySha.id, transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id },
}); });
@@ -667,32 +462,11 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} }
const items = toLineItems(extracted.items); const items = toLineItems(extracted.items);
// 2b. did it answer out of the upload, or out of its prompt?
const confab = confabulationReason(
{
textParts: parts.filter((p): p is Extract<InputPart, { kind: "text" }> => p.kind === "text").map((p) => p.text),
hasImages: parts.some((p) => p.kind === "image"),
},
extracted
);
if (confab) {
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
return fail("unsupported", "unsupported", unsupportedCard(confab));
}
// 3. header. // 3. header.
//
// This path stores the partial header and the items on the batch row, which
// is right for a genuine screenshot missing its top — the next tap has
// something to work from. It is wrong for a confabulation, and on 2026-08-22
// it stored one: an invented Burgertory header with two invented lines, held
// back from the matcher only by an absent date. Nothing here could tell the
// two apart, which is why the pre-check above and 2b now keep confabulations
// from reaching this point at all rather than trying to filter them here.
const headerCheck = validateHeader(extracted); const headerCheck = validateHeader(extracted);
if (!headerCheck.ok) { if (!headerCheck.ok) {
await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" }); await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" });
return fail("held", "needs_header", needsHeaderCard(headerCheck.missing, input.source), 200, { items }); return fail("held", "needs_header", needsHeaderCard(), 200, { items });
} }
const header = headerCheck.header; const header = headerCheck.header;
const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" }; const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" };
@@ -704,7 +478,7 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
const dup = await findDuplicateBatch(batchId, shas, storedHeader); const dup = await findDuplicateBatch(batchId, shas, storedHeader);
if (dup) { if (dup) {
await updateBatch(batchId, { header: storedHeader, items, kind, status: "duplicate", transaction_id: dup.transaction_id, capture_id: dup.capture_id }); await updateBatch(batchId, { header: storedHeader, items, kind, status: "duplicate", transaction_id: dup.transaction_id, capture_id: dup.capture_id });
return fail(kind, "duplicate", duplicateCard({ merchant: header.merchant, total: header.total, currency: storedHeader.currency, transactionId: dup.transaction_id, captureId: dup.capture_id, batchId: dup.id }), 200, { return fail(kind, "duplicate", duplicateCard({ merchant: header.merchant, total: header.total, transactionId: dup.transaction_id, captureId: dup.capture_id, batchId: dup.id }), 200, {
header, header,
items, items,
duplicate_of: { batch_id: dup.id, transaction_id: dup.transaction_id, capture_id: dup.capture_id }, duplicate_of: { batch_id: dup.id, transaction_id: dup.transaction_id, capture_id: dup.capture_id },
@@ -719,7 +493,7 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
} }
await updateBatch(batchId, { header: storedHeader, items, kind, status: "received" }); await updateBatch(batchId, { header: storedHeader, items, kind, status: "received" });
const ctx: RouteContext = { batchId, source: input.source, sharedBy: input.sharedBy, platform, header, items, currency: storedHeader.currency }; const ctx: RouteContext = { batchId, source: input.source, sharedBy: input.sharedBy, platform, header, items, subtotal: extracted.subtotal };
// 7. act. // 7. act.
if (kind === "ask") { if (kind === "ask") {
+8 -123
View File
@@ -43,7 +43,6 @@ export function platformForSource(source: IntakeSource): string | null {
export const PLATFORM_LABEL: Record<string, string> = { export const PLATFORM_LABEL: Record<string, string> = {
ubereats: "Uber Eats", ubereats: "Uber Eats",
doordash: "DoorDash", doordash: "DoorDash",
uber: "Uber",
}; };
export function platformLabel(platform: string | null): string { export function platformLabel(platform: string | null): string {
@@ -52,16 +51,10 @@ export function platformLabel(platform: string | null): string {
/** /**
* A small pattern list, not an exact lookup (plan §2, Codex #6). Uber Eats prints * A small pattern list, not an exact lookup (plan §2, Codex #6). Uber Eats prints
* `Coles (Wyndham Vale)`, a PDF may print `Woolworths Group Limited`; a * `Coles (Wyndham Vale)`, a PDF may print `Woolworths Group Limited`; substring on
* word-boundary alternation on the normalised name covers both without a * the normalised name covers both without a merchant master-data subsystem.
* merchant master-data subsystem.
*
* The boundaries are load-bearing, not tidiness: a bare substring made
* `Bigals Pizza` a grocer, because "b-IGA-ls" contains `iga`. Short names in a
* substring list collide with ordinary words, and the collision reads as a
* correct classification.
*/ */
export const GROCERY_PATTERN = /\b(coles|woolworths|woolies|aldi|iga|foodworks|costco)\b/i; export const GROCERY_PATTERN = /coles|woolworths|woolies|aldi|iga|foodworks|costco/i;
/** /**
* The normalised merchant used for matching and for the remembered-kind table. * The normalised merchant used for matching and for the remembered-kind table.
@@ -226,118 +219,10 @@ export function toLineItems(items: ExtractedItem[]): StoredLineItem[] {
}); });
} }
/** Is `needle` a contiguous run of whole words inside `hay`? */ /** Does the stored merchant name denote the extracted one? Exact key, or one contains the other. */
function containsWordRun(hay: string[], needle: string[]): boolean {
if (!needle.length || needle.length > hay.length) return false;
for (let i = 0; i + needle.length <= hay.length; i++) {
if (needle.every((w, j) => hay[i + j] === w)) return true;
}
return false;
}
/**
* Does the stored merchant name denote the extracted one? One name's words must
* appear as a contiguous run inside the other's — so `Coles` matches
* `Coles (Wyndham Vale)` and `Order - Coles (Wyndham Vale)`, which is spelling,
* not widening.
*
* Whole words, never a raw substring: `merchantKey("Bigals Pizza")` contains
* `iga`, so a pizza shop matched every IGA on the same date and total.
*/
export function merchantMatches(stored: string | null | undefined, extracted: string | null | undefined): boolean { export function merchantMatches(stored: string | null | undefined, extracted: string | null | undefined): boolean {
const a = merchantKey(stored).split(" ").filter(Boolean); const a = merchantKey(stored);
const b = merchantKey(extracted).split(" ").filter(Boolean); const b = merchantKey(extracted);
if (!a.length || !b.length) return false; if (!a || !b) return false;
return containsWordRun(a, b) || containsWordRun(b, a); return a === b || a.includes(b) || b.includes(a);
}
// ─── is this text even a receipt? (2026-08-22) ───────────────────────────────
/**
* The merchant names the extraction prompt uses to show the FORMAT. They are
* placeholders rather than real shops on purpose.
*
* On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so the intake
* router received one text part whose entire content was the five characters
* `Image`. The model was handed that, and returned a whole receipt:
* `Burgertory (Braybrook)`, two "The Pride" lines at $22.80, a $41.15 total.
* Nothing on the receipt existed — "Burgertory" was in the prompt, as the
* example of how to print a branch in parentheses. Only the absent date kept it
* out of the matcher; the invented header and items were stored on the batch.
*
* Same shape as the ingestion engine's 2026-08-05 empty-body defect: a stage
* that can be handed nothing has to say so, because the model downstream will
* always answer, and it answers out of its prompt.
*
* Exported from here rather than written into the prompt string so the prompt
* and the guard that rejects an echo of it cannot drift apart.
*/
export const PROMPT_PLACEHOLDER_MERCHANTS = ["<Store> (<Suburb>)"] as const;
/**
* Below this a text part needs a money figure to be believable. NOT a floor on
* its own: `TOTAL $17.00` is twelve characters and is exactly the kind of
* pasted line this door exists to accept.
*/
export const MIN_RECEIPT_TEXT_CHARS = 40;
/** `$17.00`, `17.00`, `12,50` — a printed amount, not merely a digit. */
const MONEY_SHAPE = /(?:[$€£¥]\s*\d|\d[.,]\d{2}(?!\d))/;
export type TextCheck = { ok: true } | { ok: false; reason: string };
/**
* Can this text possibly be a receipt? Deterministic, before any model call.
*
* A receipt has more than one word and at least one number. Everything the
* phone substitutes when a share goes wrong — the literal word `Image`, a bare
* filename like `IMG_0042.jpeg`, an empty body — fails one of those, and none
* of them fails in a way a model will admit to.
*
* Applies to TEXT parts only. A PDF is checked for readable text separately and
* an image is the thing we actually asked for.
*/
export function textCouldBeReceipt(raw: string | null | undefined): TextCheck {
const text = (raw ?? "").trim();
if (!text) return { ok: false, reason: "it was empty" };
if (!/\s/.test(text)) return { ok: false, reason: "it is a single word" };
if (!/\d/.test(text)) return { ok: false, reason: "it has no numbers in it" };
if (text.length < MIN_RECEIPT_TEXT_CHARS && !MONEY_SHAPE.test(text)) {
return { ok: false, reason: "it is too short to be a receipt" };
}
return { ok: true };
}
/** What the phone actually sent, short enough to put on a card. */
export function quoteShared(raw: string, max = 80): string {
const flat = (raw ?? "").replace(/\s+/g, " ").trim();
return flat.length > max ? `${flat.slice(0, max - 1)}…` : flat;
}
/**
* Belt to `textCouldBeReceipt`'s braces: did the extractor answer out of its
* prompt rather than out of the upload?
*
* Two tells, both cheap and both deterministic:
* - the merchant IS one of the prompt's format placeholders; or
* - nothing readable was sent at all (no images, and no text part that could
* be a receipt) and yet a merchant or items came back.
*
* The second is reachable even with the pre-check in place, because the
* pre-check only inspects text parts: a PDF that extracts to a few characters
* of junk still reaches the model. This is `unsupported`, never `needs_header`
* — a confabulated header is not a partial one.
*/
export function confabulationReason(
input: { textParts: string[]; hasImages: boolean },
extracted: { merchant: string | null; items: unknown[] }
): string | null {
const key = merchantKey(extracted.merchant);
if (key && PROMPT_PLACEHOLDER_MERCHANTS.some((p) => merchantKey(p) === key)) {
return "the merchant came back as the prompt's own example rather than a name off the receipt";
}
if (input.hasImages) return null;
if (input.textParts.some((t) => textCouldBeReceipt(t).ok)) return null;
if (!extracted.merchant && !extracted.items.length) return null;
return "nothing shared could be read as a receipt, yet an order came back";
} }
+21 -97
View File
@@ -1,4 +1,4 @@
import { queryRow } from "@/lib/db"; import { queryRow, queryRaw } from "@/lib/db";
/** /**
* Replace the itemised receipt behind a transaction (plan §11 C2). * Replace the itemised receipt behind a transaction (plan §11 C2).
@@ -15,12 +15,6 @@ import { queryRow } from "@/lib/db";
* transaction has no row at all (a manual transaction, a statement line), one * transaction has no row at all (a manual transaction, a statement line), one
* is created so the items have somewhere to live; a transaction is NEVER * is created so the items have somewhere to live; a transaction is NEVER
* created here. * created here.
*
* `opts.metadataId` pins WHICH row when the caller already resolved one. A
* statement transaction can own its own `expense_metadata` row AND be the
* `matched_transaction_id` of a delivery row: the match query and this write
* then re-pick independently and can land on different rows, so the enrichment
* reports success against a row nobody reads.
*/ */
export type LineItemsSource = "intake" | "pantry"; export type LineItemsSource = "intake" | "pantry";
@@ -61,35 +55,10 @@ export type ReplaceResult = {
line_items: LineItemRecord[]; line_items: LineItemRecord[];
}; };
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
/**
* The undo baseline, recorded once per batch.
*
* Returning the row that was just read makes a RETRY destroy the thing undo
* exists to restore: pantry re-calls this on every confirm and after a timeout,
* and the second call would hand back its own write as the "previous" state.
* The first snapshot a batch ever takes is the only one worth keeping, so it is
* written under COALESCE and read back rather than trusted.
*/
async function undoBaseline(batchId: string | null | undefined, snapshot: unknown[]): Promise<unknown[]> {
// Pantry passes back whatever batch id it was given; only an intake batch has
// a row to record against, and a non-uuid would fail the cast.
if (!batchId || !UUID.test(batchId)) return snapshot;
const row = await queryRow<{ previous_line_items: unknown }>(
`UPDATE intake_batches
SET previous_line_items = COALESCE(previous_line_items, $2::jsonb)
WHERE id = $1::uuid
RETURNING previous_line_items`,
[batchId, JSON.stringify(snapshot)]
);
return Array.isArray(row?.previous_line_items) ? row.previous_line_items : snapshot;
}
export async function replaceLineItems( export async function replaceLineItems(
transactionId: number, transactionId: number,
lineItems: LineItemRecord[], lineItems: LineItemRecord[],
opts: { source: LineItemsSource; platform?: string | null; batchId?: string | null; metadataId?: number | null; currency?: string | null } opts: { source: LineItemsSource; platform?: string | null; batchId?: string | null }
): Promise<ReplaceResult> { ): Promise<ReplaceResult> {
const txn = await queryRow<{ const txn = await queryRow<{
id: number; id: number;
@@ -104,78 +73,36 @@ export async function replaceLineItems(
); );
if (!txn) throw new TransactionNotFoundError(`transaction ${transactionId} does not exist`); if (!txn) throw new TransactionNotFoundError(`transaction ${transactionId} does not exist`);
// The caller's row wins where it still exists; otherwise prefer the row that // Prefer the row that IS this transaction over one that merely points at it —
// IS this transaction over one that merely points at it — a statement line // a statement line can be the `matched_transaction_id` of a receipt row and
// can be the `matched_transaction_id` of a receipt row and carry its own row // carry its own row too, and the items belong on the receipt.
// too, and the items belong on the receipt.
const pinned = opts.metadataId
? await queryRow<{ id: number; line_items: unknown }>(
`SELECT id, line_items FROM expense_metadata
WHERE id = $1 AND (transaction_id = $2 OR matched_transaction_id = $2)`,
[opts.metadataId, transactionId]
)
: null;
const json = JSON.stringify(lineItems);
/**
* Snapshot the baseline, then write. In that order on BOTH paths: the create
* path used to snapshot after its INSERT, so a crash in between left a retry
* recording the intake's own write as "previous" — undo restoring the thing
* it was meant to reverse, which is the failure `previous_line_items` exists
* to prevent.
*
* Returns null when the row is gone between the read and the write. That
* UPDATE reported success and the batch was marked enriched against a row
* nothing had written to.
*/
const write = async (row: { id: number; line_items: unknown }): Promise<ReplaceResult | null> => {
const previous = await undoBaseline(opts.batchId, Array.isArray(row.line_items) ? row.line_items : []);
const updated = await queryRow<{ id: number }>(
`UPDATE expense_metadata SET line_items = $2::jsonb WHERE id = $1 RETURNING id`,
[row.id, json]
);
if (!updated) return null;
return { transaction_id: transactionId, metadata_id: row.id, created_metadata: false, previous_line_items: previous, line_items: lineItems };
};
if (pinned) {
const done = await write(pinned);
if (done) return done;
}
const existing = await queryRow<{ id: number; line_items: unknown }>( const existing = await queryRow<{ id: number; line_items: unknown }>(
`SELECT id, line_items FROM expense_metadata `SELECT id, line_items FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
-- NULLS LAST is load-bearing, and its absence inverted this clause. ORDER BY (transaction_id = $1) DESC, id
-- Postgres orders DESC as NULLS FIRST, and the row that merely POINTS at
-- the transaction has a NULL transaction_id — so the comparison meant to
-- prefer the owned row preferred the other one, silently and in exactly
-- the case it was written for. matchTransaction spells it out; this did
-- not, and the two therefore disagreed.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[transactionId] [transactionId]
); );
const json = JSON.stringify(lineItems);
if (existing) { if (existing) {
const done = await write(existing); await queryRaw(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE id = $1`, [existing.id, json]);
if (done) return done; return {
transaction_id: transactionId,
metadata_id: existing.id,
created_metadata: false,
previous_line_items: Array.isArray(existing.line_items) ? existing.line_items : [],
line_items: lineItems,
};
} }
// Nothing to update. Snapshot the empty baseline BEFORE creating the row.
const previous = await undoBaseline(opts.batchId, []);
const reference = opts.batchId ? `intake:${opts.batchId}` : `intake:txn:${transactionId}`; const reference = opts.batchId ? `intake:${opts.batchId}` : `intake:txn:${transactionId}`;
// ON CONFLICT because two calls for one transaction can both find no row and const created = await queryRow<{ id: number }>(
// both INSERT; expense_metadata.transaction_id is UNIQUE, so one of them lost
// with a constraint error that reached the user as an error card. `xmax = 0`
// is how a plain insert is told from a conflict-update, so the reported
// `created_metadata` stays true.
const created = await queryRow<{ id: number; inserted: boolean }>(
`INSERT INTO expense_metadata ( `INSERT INTO expense_metadata (
transaction_id, source, platform, order_reference, line_items, transaction_id, source, platform, order_reference, line_items,
merchant_normalized, amount, transaction_date, currency, flags, extraction_model merchant_normalized, amount, transaction_date, currency, flags, extraction_model
) VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::numeric, $8::date, $9, '[]'::jsonb, 'cloud-budget') ) VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::numeric, $8::date, 'AUD', '[]'::jsonb, 'cloud-budget')
ON CONFLICT (transaction_id) DO UPDATE SET line_items = EXCLUDED.line_items RETURNING id`,
RETURNING id, (xmax = 0) AS inserted`,
[ [
transactionId, transactionId,
opts.source, opts.source,
@@ -185,16 +112,13 @@ export async function replaceLineItems(
txn.merchant_normalized ?? txn.merchant_name ?? null, txn.merchant_normalized ?? txn.merchant_name ?? null,
txn.amount, txn.amount,
txn.transaction_date, txn.transaction_date,
// Not hard-coded 'AUD': the transaction may carry a foreign figure, and
// OrderDetails prefers this column, so a USD order printed "$".
opts.currency || "AUD",
] ]
); );
return { return {
transaction_id: transactionId, transaction_id: transactionId,
metadata_id: created!.id, metadata_id: created!.id,
created_metadata: created!.inserted, created_metadata: true,
previous_line_items: previous, previous_line_items: [],
line_items: lineItems, line_items: lineItems,
}; };
} }
+7
View File
@@ -12,6 +12,13 @@ export const CUTOVER_DATE = "2026-01-09";
*/ */
export const DEFAULT_OWNER_ID = 1; export const DEFAULT_OWNER_ID = 1;
/** Human labels for the platform a receipt came from. */
export const PLATFORM_LABEL: Record<ParsedOrder["platform"], string> = {
doordash: "DoorDash",
ubereats: "Uber Eats",
uber: "Uber",
};
/** /**
* Transaction description. * Transaction description.
* *
+1 -5
View File
@@ -526,11 +526,7 @@ export function parseOrderHTML(html: string, meta: MessageMeta): ParsedOrder {
} else { } else {
// DoorDash carries no order id anywhere in the receipt, so the provider // DoorDash carries no order id anywhere in the receipt, so the provider
// message id is the only stable identity available. That is correct for // message id is the only stable identity available. That is correct for
// ingestion idempotency: one receipt is one order — PROVIDED every caller // ingestion idempotency: one receipt is one order.
// sends the same id for the same email. Two lanes post here (n8n Order
// Processor and ingestion-engine order_handoff); both must send the bare
// RFC Message-ID. n8n sent Gmail's own id until 2026-10-03 and one order
// became txns 7268 + 7269.
if (!meta.messageId) { if (!meta.messageId) {
throw new OrderParseError("no order id in body and no messageId supplied"); throw new OrderParseError("no order id in body and no messageId supplied");
} }
+3 -28
View File
@@ -119,16 +119,6 @@ export async function merchantForTransaction(
const row = await queryRow<{ merchant_normalized: string | null }>( const row = await queryRow<{ merchant_normalized: string | null }>(
`SELECT merchant_normalized FROM expense_metadata `SELECT merchant_normalized FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1 WHERE transaction_id = $1 OR matched_transaction_id = $1
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`, LIMIT 1`,
[transactionId] [transactionId]
); );
@@ -167,24 +157,9 @@ export async function merchantVerdict(
FROM order_reviews r FROM order_reviews r
JOIN transactions t ON t.id = r.transaction_id JOIN transactions t ON t.id = r.transaction_id
JOIN participants p ON p.id = r.participant_id JOIN participants p ON p.id = r.participant_id
-- (No backticks in here: this SQL lives in a TS template literal and a JOIN expense_metadata em
-- backtick ends the string -- TS1005 on a line that looks like a comment.) ON em.transaction_id = r.transaction_id
-- LATERAL, not a plain JOIN. A plain join on OR em.matched_transaction_id = r.transaction_id
-- transaction_id OR matched_transaction_id fans out: one review
-- joined to N metadata rows appears N times in the result, and the loops
-- below count every appearance -- inflating the loved/never tallies
-- that drive the pre-order warning. LIMIT 50 applies to the fanned-out
-- set too, so duplicates would also push older reviews out of the
-- window. Same tie-break as every other reader, so the merchant matched
-- here is the merchant the UI shows.
JOIN LATERAL (
SELECT em2.merchant_normalized
FROM expense_metadata em2
WHERE em2.transaction_id = r.transaction_id
OR em2.matched_transaction_id = r.transaction_id
ORDER BY (em2.transaction_id = r.transaction_id) DESC NULLS LAST, em2.id
LIMIT 1
) em ON true
-- Case-folded: the platforms capitalise the same restaurant differently -- Case-folded: the platforms capitalise the same restaurant differently
-- ("TEG Kebabs & Biryani" on Uber Eats, "TEG KEBABS & BIRYANI" on -- ("TEG Kebabs & Biryani" on Uber Eats, "TEG KEBABS & BIRYANI" on
-- DoorDash). An exact match split one restaurant's history in two, so a -- DoorDash). An exact match split one restaurant's history in two, so a
+1 -52
View File
@@ -367,16 +367,6 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
SELECT em.route, em.platform SELECT em.route, em.platform
FROM expense_metadata em FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
LIMIT 1 LIMIT 1
) order_ctx ON true ) order_ctx ON true
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
@@ -851,15 +841,6 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
WHERE ts.transaction_id = t.id WHERE ts.transaction_id = t.id
) txn_splits ON true ) txn_splits ON true
WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL
-- A superseded row is already resolved, just not by reconciling. Without
-- this the queue kept offering it: a duplicate wallet half hidden under
-- its twin (txn 7037, 2026-10-03) sat beside that twin with the same
-- statement line as its match, and accepting it on the hidden row would
-- have used the line up and stranded the real one.
AND t.superseded_by_id IS NULL
-- A voided row is resolved too: the charge never posted, so no statement
-- line will ever arrive for it (migration 0040).
AND t.voided_at IS NULL
AND ${needsCardMatch("t")} AND ${needsCardMatch("t")}
ORDER BY t.transaction_date DESC, t.row_index ASC`, ORDER BY t.transaction_date DESC, t.row_index ASC`,
[ownerId] [ownerId]
@@ -896,39 +877,12 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
FROM transactions m FROM transactions m
JOIN transactions t ON t.statement_id IS NOT NULL JOIN transactions t ON t.statement_id IS NOT NULL
AND t.transaction_date BETWEEN m.transaction_date - 3 AND m.transaction_date + 3 AND t.transaction_date BETWEEN m.transaction_date - 3 AND m.transaction_date + 3
-- +-5%, NOT +-1%. A foreign card purchase is charged at the notified AND t.amount BETWEEN m.amount * 0.99 AND m.amount * 1.01
-- amount PLUS the card's foreign transaction fee (3% on the Westpac
-- Altitude), so the wallet row and its statement line can never match
-- exactly and the old +-1% band excluded the statement row outright:
-- SP PEAKDESIGN was notified at $42.44 and charged $43.71 (x1.03),
-- so the one row that WAS its twin was the one the picker could not
-- offer. Widening is safe here because this list is capped at 5 by
-- the rn window and ordered by amount closeness; a looser band changes
-- which five appear, never how many, and a person confirms each one.
-- The AUTO-matcher stays exact on purpose; see findAutoMatches().
AND (t.amount BETWEEN m.amount * 0.95 AND m.amount * 1.05
-- Or the exact native pair, whatever the AUD figures say. A
-- wallet row's AUD is provisional: in Oct 2026 it priced INR at
-- 0.0180 while CommBank charged ~0.0149, 20% under, so the band
-- excluded every rupee twin. findAutoMatches() matches on this
-- pair already, but refuses when two lines share it inside its
-- window (INR 112 on 10/4 and again on 10/7) -- and those are
-- exactly the rows that land here for a person to choose.
OR (m.foreign_currency_code IS NOT NULL
AND t.foreign_currency_code = m.foreign_currency_code
AND t.foreign_currency_amount = m.foreign_currency_amount))
JOIN statements s ON s.id = t.statement_id JOIN statements s ON s.id = t.statement_id
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
WHERE m.statement_id IS NULL WHERE m.statement_id IS NULL
AND m.owner_id = $1 AND m.owner_id = $1
AND m.reconciled_with_id IS NULL AND m.reconciled_with_id IS NULL
AND m.superseded_by_id IS NULL
AND m.voided_at IS NULL
-- Nor offer a superseded STATEMENT row as the match: the ANZ
-- statements 107/142/143 left 31 of those, and a manual row
-- reconciled to one inherits its exclusion from every figure.
-- findAutoMatches() has had both filters all along.
AND t.superseded_by_id IS NULL
AND ${needsCardMatch("m")} AND ${needsCardMatch("m")}
AND COALESCE(t.owner_id, s.owner_id) = $1 AND COALESCE(t.owner_id, s.owner_id) = $1
AND NOT EXISTS ( AND NOT EXISTS (
@@ -1026,11 +980,6 @@ export async function getSharedTransactions(ownerId: number, tagIds?: number[],
SELECT em.platform SELECT em.platform
FROM expense_metadata em FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
-- Same tie-break as getTransactions / replaceLineItems / matchTransaction,
-- so every reader resolves the row a write lands on. Without the ORDER BY
-- this LIMIT 1 is bounded but arbitrary: it would disagree with the main
-- transaction list for the same transaction the day multiplicity appears.
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
LIMIT 1 LIMIT 1
) order_ctx ON true ) order_ctx ON true
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
+8 -68
View File
@@ -1,6 +1,5 @@
import { queryRaw, queryRow } from "@/lib/db"; import { queryRaw, queryRow } from "@/lib/db";
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion"; import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
/** /**
* Grocery receipts scanned in pantry-app, arriving as candidate spend. * Grocery receipts scanned in pantry-app, arriving as candidate spend.
@@ -155,75 +154,16 @@ export async function processReceiptIngestion(receipt: ParsedReceipt): Promise<R
const paymentMethod = paymentMethodFor(leg.class); const paymentMethod = paymentMethodFor(leg.class);
const amount = round2(Number(leg.amount)); const amount = round2(Number(leg.amount));
const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name; const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name;
const txn = await queryRow<{ id: number }>(
// Adopt the row the wallet lane already created for this tap, if there is `INSERT INTO transactions (
// one. Signals for a purchase arrive in sequence -- the notification within transaction_date, description, amount, amount_aud, category, payment_method,
// seconds, the receipt hours later -- so creating a second transaction here merchant_name, merchant_normalized, transaction_type, owner_id
// is how one shop ends up as two rows. Measured on txn 6858: the wallet ) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
// event arrived 35h25m before this path created its own row for the same [receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
// $103.30 Woolworths shop. Null means no unambiguous candidate, and the );
// fallback is exactly the old behaviour.
const adoptedId = await findAdoptableTransaction(DEFAULT_OWNER_ID, amount, receipt.transaction_date);
const txn = adoptedId
? await queryRow<{ id: number }>(
// The receipt read the register tape; the notification read a push
// banner. So the receipt wins on what was bought and where, and the
// column list here IS the ownership boundary -- `location` is absent
// because the wallet notification's suburb is the better value and
// must survive.
`UPDATE transactions
SET description = $2, merchant_name = $3, merchant_normalized = $3,
category = $4, payment_method = $5
WHERE id = $1 RETURNING id`,
[adoptedId, description, receipt.merchant_name, RECEIPT_CATEGORY, paymentMethod]
)
: await queryRow<{ id: number }>(
`INSERT INTO transactions (
transaction_date, description, amount, amount_aud, category, payment_method,
merchant_name, merchant_normalized, transaction_type, owner_id
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
);
const carriesLines = leg.leg_index === cardLeg?.leg_index; const carriesLines = leg.leg_index === cardLeg?.leg_index;
const meta = await queryRow<{ id: number }>(
// An adopted row may already carry a metadata row (the wallet lane writes
// one). expense_metadata.transaction_id is UNIQUE, so a second INSERT would
// raise -- and must not, because the two describe the same purchase.
// Enrich in place instead, with the same ownership split as the transaction
// update above: everything the receipt read wins, and anything only the
// notification knew (issuer, and the transaction's location) is absent from
// the column list rather than conditionally skipped, so it cannot be
// clobbered by a bug here. Mirrors order_transaction_bridge's UPGRADE pass.
const priorMeta = adoptedId
? await queryRow<{ id: number }>(
`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [adoptedId]
)
: null;
const meta = priorMeta
? await queryRow<{ id: number }>(
`UPDATE expense_metadata
SET source = 'pantry', order_reference = $2, line_items = $3::jsonb,
subtotal = $4, amount = $5, merchant_normalized = $6,
transaction_date = $7, card_last4 = COALESCE($8, card_last4),
currency = 'AUD', flags = flags || $9::jsonb,
payment_method = $10, payment_method_detail = $11,
tender_raw = COALESCE($12, tender_raw),
receipt_sha256 = $13, receipt_group = $14,
extraction_model = 'cloud-budget'
WHERE id = $1 RETURNING id`,
[
priorMeta.id, reference,
JSON.stringify(carriesLines ? receipt.line_items ?? [] : []),
receipt.tax_amount ?? null, amount, receipt.merchant_name,
receipt.transaction_date, leg.card_last4 ?? null,
JSON.stringify(carriesLines ? [...flags, "enriched_prior_row"] : [...flags, "line_items_on_card_leg", "enriched_prior_row"]),
paymentMethod, leg.card_product ?? null, receipt.tender_raw ?? null,
receipt.image_sha256 ?? null, group,
]
)
: await queryRow<{ id: number }>(
`INSERT INTO expense_metadata ( `INSERT INTO expense_metadata (
transaction_id, source, order_reference, line_items, subtotal, amount, transaction_id, source, order_reference, line_items, subtotal, amount,
merchant_normalized, transaction_date, card_last4, currency, flags, merchant_normalized, transaction_date, card_last4, currency, flags,
-248
View File
@@ -1,248 +0,0 @@
import { prisma } from "@/lib/db";
/**
* Merge a statementless ("manual") transaction into its statement twin.
*
* Extracted from POST /api/transactions/reconcile so the manual queue and the
* automatic pass cannot drift apart. There must be exactly one implementation
* of this: it moves curation (overrides, tags, splits) and provenance
* (expense_metadata) off a row that reconciliation is about to hide from every
* figure, and a second copy of that logic would eventually move a different set.
*
* The manual row is NOT deleted -- it is linked via `reconciled_with_id`.
* Deleting it would cascade its children away; superseding/linking keeps the
* history and lets the row stay visible as provenance.
*/
export async function applyReconcileMatch(
manual_id: number,
statement_tx_id: number
): Promise<void> {
await prisma.$transaction(async (tx) => {
await moveCurationWithin(tx, manual_id, statement_tx_id);
// Mark manual tx as reconciled (link to statement tx)
await tx.$executeRawUnsafe(
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
statement_tx_id,
manual_id
);
});
}
/**
* Move overrides, tags, splits and expense_metadata from one transaction to
* another, inside a caller-supplied transaction.
*
* Callers: `applyReconcileMatch` (then sets `reconciled_with_id`) and
* `supersedeTransaction` (then sets `superseded_by_id`). Both hide the source
* row from every figure, so both must carry its curation across first; keeping
* one implementation is what the file header insists on.
*/
async function moveCurationWithin(
tx: Parameters<Parameters<typeof prisma.$transaction>[0]>[0],
manual_id: number,
statement_tx_id: number
): Promise<void> {
{
// Copy overrides: manual → statement tx
const override = await tx.transaction_overrides.findUnique({
where: { transaction_id: manual_id },
});
if (override) {
await tx.transaction_overrides.upsert({
where: { transaction_id: statement_tx_id },
update: {
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
updated_at: new Date(),
},
create: {
transaction_id: statement_tx_id,
category_override: override.category_override,
merchant_normalized: override.merchant_normalized,
notes: override.notes,
my_share_percent: override.my_share_percent,
},
});
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
}
// Move tags: manual → statement tx
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
if (tags.length) {
await tx.transaction_tags.createMany({
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
skipDuplicates: true,
});
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
}
// Move splits: manual → statement tx.
//
// `settled` and `settled_at` MUST be carried. This is a delete-and-recreate,
// and the column defaults to false — so omitting it silently marks every
// settled split unsettled again and resurrects a debt that was already
// paid. Nothing errors and nothing in the UI says the flag was reset; it
// just reappears as owed. No queue row carries a settled split today, but
// the auto-matcher runs this path unattended, so a latent version of this
// would surface as money, not as an exception.
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
if (splits.length) {
await tx.transaction_splits.createMany({
data: splits.map((s) => ({
transaction_id: statement_tx_id,
participant_id: s.participant_id,
share_percent: s.share_percent,
settled: s.settled ?? false,
settled_at: s.settled_at,
})),
skipDuplicates: true,
});
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
}
// Move provenance: manual → statement tx.
//
// Overrides, tags and splits above were always carried across; expense_metadata was
// the one child left behind, which did not matter while every metadata row came from
// an email that had created its own transaction. It matters now: a scanned grocery
// receipt puts its line items here, and reconciliation hides the manual row from
// every figure — so without this the shop's contents disappear at exactly the moment
// the statement line appears, and COLES 0556 MANOR LAKES stays as unreadable as it
// was before the receipt was ever scanned.
//
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
// or Paperless copy got there first) keeps it. The manual row's metadata stays
// attached to the reconciled manual transaction and is flagged, rather than raising a
// constraint violation or silently overwriting the other source.
const moved = await tx.$executeRawUnsafe(
`UPDATE expense_metadata SET transaction_id = $1
WHERE transaction_id = $2
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
statement_tx_id,
manual_id
);
if (moved === 0) {
await tx.$executeRawUnsafe(
`UPDATE expense_metadata
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
WHERE transaction_id = $1`,
manual_id
);
}
}
}
/**
* Manually mark `loser_id` as superseded by `survivor_id`.
*
* The escape hatch for a duplicate that `applyReconcileMatch` cannot express:
* that one merges a statementLESS row into its statement twin, so it has
* nothing to say about two rows on the SAME footing -- the pair left by a
* statement imported twice (migration 0023 superseded 31 such pairs, $42,040.68,
* by hand), or two feed rows for one purchase.
*
* Curation moves FIRST, using the same block `applyReconcileMatch` runs, because
* `superseded_by_id` hides the row from every figure (`analytics-sql.ts`) and
* splits or tags left on it would silently stop counting. That is why this lives
* beside its sibling and calls into the same movement rather than re-deriving
* it: the header of this file warns that a second copy would eventually move a
* different set, and a supersede that forgot splits would resurrect a debt in
* exactly the way `applyReconcileMatch`'s own comment describes.
*
* NOT a delete. Every child of `transactions` is ON DELETE CASCADE, so removing
* the row would destroy the splits, tags and metadata that make it legible --
* and a duplicate is evidence of how the ledger was assembled, which is worth
* keeping. `transactions_no_self_supersede` blocks a row pointing at itself; the
* survivor is re-checked here anyway so the error is a 400 and not a 500.
*/
export async function supersedeTransaction(
loser_id: number,
survivor_id: number
): Promise<void> {
if (loser_id === survivor_id) throw new Error("A transaction cannot supersede itself");
await prisma.$transaction(async (tx) => {
// Refuse to bury a row underneath one that is itself already buried, which
// would make the curation we just moved unreachable in a second hop.
const survivor = await tx.transactions.findUnique({
where: { id: survivor_id },
select: { superseded_by_id: true },
});
if (!survivor) throw new Error(`Survivor transaction ${survivor_id} not found`);
if (survivor.superseded_by_id !== null) {
throw new Error(`Transaction ${survivor_id} is itself superseded; pick the surviving row`);
}
await moveCurationWithin(tx, loser_id, survivor_id);
await tx.$executeRawUnsafe(
`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`,
survivor_id,
loser_id
);
});
}
/**
* Mark a statementless row as a charge that never posted (migration 0040).
*
* The case neither sibling above can express: a card notification is sent for
* an AUTHORISATION, and when the merchant releases the hold no statement line
* ever arrives. Txn 7951 was a $1,040.41 Airbnb hold on a booking cancelled the
* next day -- it sat in the reconcile queue with nothing to match and counted as
* spend in every figure. There is no survivor to move curation onto, so nothing
* moves: the row and its children stay, and `EXCLUDE_RECONCILED_SOURCE` stops
* them counting.
*
* Refusals, all deliberate:
* - a statement row: that is the bank's record of a posted charge, and the
* statement must keep summing to its header (`transactions_void_statementless`
* enforces this in the database too). A wrong statement row is a supersede.
* - a row already reconciled or superseded: it is already out of every figure,
* and a reconciled one evidently DID post.
* - a row with a SETTLED split: someone has already paid their share of it.
* Voiding would silently turn that payment into a credit on their balance;
* that needs a person to look at it, not a side effect.
*/
export async function voidTransaction(id: number, reason: string): Promise<void> {
const why = reason.trim();
if (!why) throw new Error("A void needs a reason");
await prisma.$transaction(async (tx) => {
const rows = await tx.$queryRawUnsafe<
{ statement_id: number | null; reconciled_with_id: number | null; superseded_by_id: number | null; settled: number }[]
>(
`SELECT t.statement_id, t.reconciled_with_id, t.superseded_by_id,
(SELECT count(*)::int FROM transaction_splits ts
WHERE ts.transaction_id = t.id AND ts.settled) AS settled
FROM transactions t WHERE t.id = $1 FOR UPDATE`,
id
);
const row = rows[0];
if (!row) throw new Error(`Transaction ${id} not found`);
if (row.statement_id !== null) {
throw new Error(`Transaction ${id} is a statement line; a posted charge cannot be voided (supersede it instead)`);
}
if (row.reconciled_with_id !== null) {
throw new Error(`Transaction ${id} is reconciled with ${row.reconciled_with_id}, so it posted`);
}
if (row.superseded_by_id !== null) {
throw new Error(`Transaction ${id} is already superseded by ${row.superseded_by_id}`);
}
if (row.settled > 0) {
throw new Error(`Transaction ${id} has ${row.settled} settled split(s); resolve those before voiding`);
}
await tx.$executeRawUnsafe(
`UPDATE transactions SET voided_at = now(), void_reason = $1 WHERE id = $2 AND voided_at IS NULL`,
why,
id
);
});
}
/** Undo `voidTransaction` -- e.g. the "released" hold posted after all. */
export async function unvoidTransaction(id: number): Promise<void> {
await prisma.$executeRawUnsafe(
`UPDATE transactions SET voided_at = NULL, void_reason = NULL WHERE id = $1`,
id
);
}
-120
View File
@@ -1,120 +0,0 @@
/**
* Deterministic parsing of the bits of a wallet notification that have a fixed
* shape.
*
* Everything here was previously left to the interpreting model, which handled
* ONE fixed-width format three different ways on the same day's data: it kept
* the country code on "WYNDHAM VAL AU", stripped it on "Sydney" and "Perth",
* and swallowed "SAN FRANCISOUS" whole into the location field. A fixed-width
* field is not a judgement call, and the same argument that put the order
* parsers in code rather than in a Gemini call applies: a regex fails loudly on
* a template change, where a model returns a plausible wrong answer.
*
* The model still decides what the amount and merchant ARE -- it is better at
* that than a pattern, because five cards emit three shapes. This only handles
* the parts that are positional.
*/
/**
* Westpac writes the merchant/city/country block as a fixed-width statement
* descriptor: 27-char merchant, 12-char city, 2-char ISO country. Verified
* across every Westpac event captured:
*
* Breadworld Bakery Cafe WYNDHAM VAL AU
* PLA DONATION MELBOURNE AU
* SP PEAKDESIGN SAN FRANCISOUS <- "SAN FRANCISO" + "US"
*
* The last example is why this is positional rather than a split on whitespace:
* a 12-character city fills its field exactly and runs straight into the
* country code with no separator.
*/
export type WestpacDescriptor = {
merchant: string | null;
city: string | null;
/** ISO-3166 alpha-2, upper-case. Null when the tail is not a country code. */
country: string | null;
};
const MERCHANT_WIDTH = 27;
const CITY_WIDTH = 12;
/**
* `body` is the whole notification body, e.g.
* "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
* Returns nulls rather than throwing: a template change must degrade to "we
* don't know", never to a wrong country on a foreign transaction.
*/
export function parseWestpacDescriptor(body: string | null | undefined): WestpacDescriptor {
const none: WestpacDescriptor = { merchant: null, city: null, country: null };
if (!body) return none;
// The descriptor is whatever follows the first sentence ("... ending...8032. ").
// [\s\S] rather than the `s` flag: the tsconfig target predates es2018.
const m = body.match(/^[\s\S]*?\.\s([\s\S]*)$/);
if (!m) return none;
const tail = m[1].replace(/\s+$/, "");
if (!tail) return none;
const merchant = tail.slice(0, MERCHANT_WIDTH).trim() || null;
const rest = tail.slice(MERCHANT_WIDTH);
if (!rest) return { merchant, city: null, country: null };
// Country is the final two characters of the fixed block, but only when the
// block is actually long enough to contain one -- a short tail is a merchant
// name with no location, not a merchant whose last two letters are a country.
const block = rest.slice(0, CITY_WIDTH + 2);
if (block.length < CITY_WIDTH + 2) {
return { merchant, city: block.trim() || null, country: null };
}
const city = block.slice(0, CITY_WIDTH).trim() || null;
const tailTwo = block.slice(CITY_WIDTH).trim().toUpperCase();
const country = /^[A-Z]{2}$/.test(tailTwo) ? tailTwo : null;
return { merchant, city, country: country };
}
/**
* The Apple Wallet notification's subtitle, which is comma-delimited and clean:
* "Breadworld Bakery Cafe, Wyndham Vale, VIC" -> location "Wyndham Vale, VIC"
* "Seatsaero" -> no location (online merchant)
*
* Only the first comma separates merchant from place; the rest is the place.
*/
export function parseWalletSubtitle(subtitle: string | null | undefined): {
merchant: string | null;
location: string | null;
} {
if (!subtitle || !subtitle.trim()) return { merchant: null, location: null };
const parts = subtitle.split(",").map((p) => p.trim()).filter(Boolean);
if (parts.length <= 1) return { merchant: parts[0] ?? null, location: null };
return { merchant: parts[0], location: parts.slice(1).join(", ") };
}
/**
* Currency marker, if the notification states one.
*
* Only Amex does: it writes "A$5,103.49". Westpac, CommBank and Apple Wallet
* write a bare "$", which is honestly recorded as unknown rather than assumed
* to be AUD -- 14 of 21 captured transaction rows. Domestically that assumption
* would be harmless; abroad it is a 23x error on a THB tap, so it is not made
* here. The statement settles it within weeks, and the reconcile queue holds
* the row until then.
*/
export function currencyFromAmountText(amountText: string | null | undefined): string | null {
if (!amountText) return null;
const t = amountText.trim();
if (/^A\$/i.test(t)) return "AUD";
if (/^(NZ\$)/i.test(t)) return "NZD";
if (/^US\$/i.test(t)) return "USD";
if (/^€/.test(t)) return "EUR";
if (/^£/.test(t)) return "GBP";
if (/^¥/.test(t)) return "JPY";
if (/^฿/.test(t)) return "THB";
if (/^S\$/i.test(t)) return "SGD";
if (/^₹/.test(t)) return "INR";
// CommBank's approval prompts write the ISO code after the figure
// ("672006 VND", "1340.00 INR"); some issuers put it first ("USD 10.00").
const iso = t.match(/^[\d.,\s]+([A-Z]{3})$/) ?? t.match(/^([A-Z]{3})\s*[\d.,]+$/);
if (iso) return iso[1];
// A bare "$" is genuinely ambiguous -- do not guess.
return null;
}
-481
View File
@@ -1,481 +0,0 @@
import { queryRow, withTransaction, type Queryable } from "@/lib/db";
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
import { findWalletAdoptableTransaction } from "@/lib/auto-reconcile";
import {
parseWestpacDescriptor,
parseWalletSubtitle,
currencyFromAmountText,
} from "@/lib/wallet-descriptor";
/**
* Wallet notifications -> transactions.
*
* A transaction is created for EVERY tap (DECISIONS.md ING-11, 2026-08-22),
* because overseas trips are where live spend matters most: the statement lag
* is the same 6-35 days abroad but knowing what you have spent matters far more
* in an unfamiliar currency.
*
* SEQUENCING IS THE WHOLE DESIGN. Wallet is the FIRST signal, not a late one --
* measured on txn 6858, the notification arrived 35h25m before anything created
* a transaction. So this creates the row, the receipt enriches it
* (receipt-ingestion.ts adopts it), and the statement confirms it via the
* reconcile queue, which a wallet row enters unaided: it is statementless with
* payment_method 'card'. Nothing here needs to know about statements.
*
* But that sequence is a TENDENCY, not a guarantee -- these first 21 rows were
* themselves backfilled days after the taps -- so this also adopts a row a
* receipt got to first (findWalletAdoptableTransaction), and a stored-value
* instrument is written as 'credits' so it never enters a queue no statement
* line will ever resolve.
*/
export type WalletEvent = {
/** capture.wallet_events.id -- the idempotency key. */
event_id: number;
/** 'transaction' | 'not_transaction' | 'unsure' from the interpreter. */
kind: string;
direction?: string | null;
amount: number | null;
/** The exact substring the model read the amount from, e.g. "A$5,103.49". */
amount_text?: string | null;
currency?: string | null;
merchant?: string | null;
card_tail?: string | null;
issuer?: string | null;
location?: string | null;
notif_app?: string | null;
/** The notification's title line, e.g. "Visa Gift Card" or "Payment". */
notif_title?: string | null;
notif_body?: string | null;
notif_subtitle?: string | null;
occurred_at?: string | null;
};
export type WalletIngestResult = {
transactionId: number | null;
metadataId: number | null;
skipped?:
| "not_a_transaction" | "already_ingested" | "no_amount" | "merged_pair" | "adopted"
| "account_notice" | "stored_value_drawdown";
flags: string[];
};
/**
* How far apart the two halves of ONE tap can arrive. Measured 2-8 seconds on
* every captured pair; ten minutes leaves room for a phone that was offline,
* and is still nowhere near the +/-1 DAY the matcher used to allow -- which
* folded the UNICEF $50 donation and an unrelated $50 bar charge 17 hours later
* into one row (event 275 into txn 8091, 2026-10-01).
*/
const PAIR_WINDOW_MINUTES = 10;
/** The ledger is dated the way the bank statements are: Melbourne local. */
const LEDGER_TZ = "Australia/Melbourne";
/**
* The calendar date of an instant, in LEDGER_TZ.
*
* Slicing the ISO string took whatever offset n8n serialised -- UTC in
* practice -- so a Melbourne-evening tap landed on the previous day (28 of the
* first 134 rows). Statements, which this row is reconciled against, date in
* Australian local time.
*/
export function ledgerDate(occurredAt: string | null | undefined, now: Date = new Date()): string {
const d = occurredAt ? new Date(occurredAt) : now;
const at = Number.isNaN(d.getTime()) ? now : d;
// en-CA formats as YYYY-MM-DD.
return new Intl.DateTimeFormat("en-CA", {
timeZone: LEDGER_TZ, year: "numeric", month: "2-digit", day: "2-digit",
}).format(at);
}
/**
* A ledger-derived AUD rate for a currency: the median of what the bank
* actually charged divided by the native figure, over statement rows.
*
* Same source order_transaction_bridge uses for its plausibility guard. This is
* only ever a PROVISIONAL value -- transactions.amount is NOT NULL and always
* AUD, and a notification cannot know the AUD figure because the bank sets the
* rate at settlement. The statement overwrites it at reconcile time.
* Returns null when there is not enough history to be worth anything.
*/
async function medianRateToAud(code: string): Promise<number | null> {
const row = await queryRow<{ rate: string | null }>(
`SELECT percentile_cont(0.5) WITHIN GROUP (
ORDER BY amount / NULLIF(foreign_currency_amount, 0)
)::text AS rate
FROM transactions
WHERE foreign_currency_code = $1
AND foreign_currency_amount > 0
AND superseded_by_id IS NULL
-- Statement rows only: a wallet row's amount is itself a provisional
-- guess, and one stored before this filter (txn 7316, 672006 VND at a
-- 1:1 "rate") would otherwise feed the next estimate.
AND statement_id IS NOT NULL
HAVING count(*) >= 5`,
[code]
);
const rate = row?.rate ? Number(row.rate) : null;
return rate && Number.isFinite(rate) && rate > 0 ? rate : null;
}
/**
* The ECB reference rate for the day, via Frankfurter (free, no key; the same
* source the statement processor converts with).
*
* The fallback when the ledger has no history for a currency. Hong Kong was the
* first trip to need it: no HKD statement row existed, so every HKD tap stored
* A$0.00 and the trip read as free until the statement landed weeks later. The
* ledger median is still preferred -- it carries the card's ~3% foreign fee,
* which the ECB mid-rate does not -- so this one is flagged as such.
*
* Any failure is null, never a throw: no rate is a state this lane already
* handles (a flagged 0), and a slow FX API must not lose the purchase.
*/
async function ecbRateToAud(code: string, date: string): Promise<number | null> {
try {
const res = await fetch(
`https://api.frankfurter.dev/v1/${date}?from=${encodeURIComponent(code)}&to=AUD`,
{ signal: AbortSignal.timeout(5000) }
);
if (!res.ok) return null;
const body = (await res.json()) as { rates?: { AUD?: number } };
const rate = body?.rates?.AUD;
return typeof rate === "number" && Number.isFinite(rate) && rate > 0 ? rate : null;
} catch {
return null;
}
}
/**
* The other half of a paired notification, if it has already been ingested.
*
* One tap emits TWO notifications ~2-8s apart: the issuer app (which has the
* card tail) and Apple Wallet (which has the better merchant and the suburb).
* Neither alone is complete, so they must be MERGED, never deduped -- dropping
* either loses a field nothing else supplies. The notifications share no
* identifier, so a pair is recognised by what a pair IS: the same amount,
* minutes apart, from two different apps. Two notifications from one app are
* two charges, however close together.
*/
async function findPairedRow(
db: Queryable,
amount: number,
notifiedAt: string,
notifApp: string | null,
fx: { amount: number; code: string } | null = null
) {
// A foreign row pairs on its native pair: its `amount` is a provisional
// conversion (or a 0 placeholder), so two unrelated foreign taps could share it.
const rows = await db.queryRaw<{ id: number; meta_id: number }>(
`SELECT t.id, em.id AS meta_id
FROM transactions t
JOIN expense_metadata em ON em.transaction_id = t.id
WHERE t.source = 'wallet'
AND t.statement_id IS NULL
AND t.superseded_by_id IS NULL
AND CASE WHEN $4::text IS NULL
THEN t.amount = $1::numeric AND t.foreign_currency_code IS NULL
ELSE t.foreign_currency_code = $4::text
AND t.foreign_currency_amount = $5::numeric END
AND em.notified_at BETWEEN $2::timestamptz - make_interval(mins => $3::int)
AND $2::timestamptz + make_interval(mins => $3::int)
AND em.notif_app IS DISTINCT FROM $6::text
LIMIT 2`,
[amount, notifiedAt, PAIR_WINDOW_MINUTES, fx?.code ?? null, fx?.amount ?? null, notifApp]
);
return rows.length === 1 ? rows[0] : null;
}
/**
* A stored-value instrument: a gift card, a prepaid travel card, a store card.
*
* It matters because payment_method decides queue membership. 'card' means "a
* statement line is coming"; findAutoMatches and the reconcile queue both
* exclude 'credits' and 'cash' for exactly that reason. A $103.30 Woolworths
* tap on a Visa gift card written as 'card' parks in the queue forever, being
* offered as an unmatched row against statements that can never contain it --
* three of the first 21 wallet rows (events 80, 92, 101) were gift-card taps.
*
* The evidence is the notification TITLE, which is where Apple Wallet prints
* the instrument's own name ("Visa Gift Card"); the interpreter reduces that to
* issuer "Visa", which a plain Visa debit card also produces, so the title is
* the only place the distinction survives. Read as a prior, not a contract: a
* miss leaves today's behaviour, which is the queue.
*/
const STORED_VALUE = /gift\s*card|prepaid|travel\s*(?:money|card)|store\s*card/i;
/**
* A transit card reporting a drawdown of its own balance ("HKD 7.40 for Train"
* / "Your new balance is HKD 169.50.").
*
* Unlike a gift card, a transit card is loaded FROM a bank card -- that load is
* on the statement and is the spend. Each fare is that money being used, so
* booking it again counts it twice. The interpreter already called 11 of 13
* Octopus fares not_transaction; the two it did not (events 191, 200) became
* rows, so the outcome depended on the model's mood. This makes it a rule. The
* body wording is the signal because it is what every such card says; the
* issuer name is not (Octopus, Opal, myki, Suica each name themselves).
*/
const BALANCE_DRAWDOWN = /\byour (?:new|current) balance is\b/i;
/**
* Account notices that state an amount without money moving: "Your Smart Access
* 1910 is overdrawn by $21.21. To avoid fees and interest, ..." (event 235)
* became a $21.21 purchase named "To avoid fees and interest,". The model is
* right that the text names an amount; it is not a payment.
*/
const ACCOUNT_NOTICE = /\boverdrawn\b|\bto avoid fees\b|\bavailable balance\b|\blow balance\b/i;
/** How an adopted or merged row records which wallet event folded into it. */
const adoptedFlag = (eventId: number) => `wallet_evt:${eventId}`;
export function isStoredValueInstrument(ev: Pick<WalletEvent, "notif_title" | "issuer" | "notif_app">): boolean {
return [ev.notif_title, ev.issuer, ev.notif_app].some((v) => !!v && STORED_VALUE.test(v));
}
/** Already landed: a row this lane wrote, or one it adopted or merged into. */
async function alreadyIngested(db: Pick<Queryable, "queryRow">, eventId: number) {
return db.queryRow<{ id: number }>(
`SELECT id FROM transactions WHERE source = 'wallet' AND source_ref = $1
UNION ALL
SELECT em.transaction_id AS id FROM expense_metadata em
JOIN transactions t ON t.id = em.transaction_id
WHERE em.flags @> $2::jsonb AND t.superseded_by_id IS NULL
LIMIT 1`,
[`evt:${eventId}`, JSON.stringify([adoptedFlag(eventId)])]
);
}
export async function processWalletIngestion(ev: WalletEvent): Promise<WalletIngestResult> {
const flags: string[] = [];
// An open-world classifier feeding a closed-world writer drops the unknown
// and calls it a refusal (ING-14). `unsure` is a real answer and must not
// silently become a transaction.
if (ev.kind !== "transaction") {
return { transactionId: null, metadataId: null, skipped: "not_a_transaction", flags };
}
if (ev.amount === null || !Number.isFinite(ev.amount) || ev.amount <= 0) {
return { transactionId: null, metadataId: null, skipped: "no_amount", flags };
}
const text = [ev.notif_subtitle, ev.notif_body].filter(Boolean).join("\n");
if (ACCOUNT_NOTICE.test(text)) {
return { transactionId: null, metadataId: null, skipped: "account_notice", flags };
}
if (BALANCE_DRAWDOWN.test(ev.notif_body ?? "")) {
return { transactionId: null, metadataId: null, skipped: "stored_value_drawdown", flags };
}
// Two places this event may already have landed: a row this lane wrote (which
// carries the ref natively), or a row it ADOPTED or MERGED into, whose
// source_ref belongs to whoever created it -- marked in the metadata flags
// rather than by overwriting a field another lane owns. Both must be checked,
// because bumping the interpreter's prompt_version deliberately re-interprets
// and re-posts every event ever captured -- that is the replay mechanism, and
// it must not mint a second row for a purchase that has one. Checked here
// without the lock to skip the FX lookup on a replay, and again under it.
const seen = await alreadyIngested({ queryRow }, ev.event_id);
if (seen) return { transactionId: seen.id, metadataId: null, skipped: "already_ingested", flags };
const sourceRef = `evt:${ev.event_id}`;
const notifiedAt = ev.occurred_at ?? new Date().toISOString();
const date = ledgerDate(ev.occurred_at);
// Deterministic parsing of the positional fields. The model decides what the
// amount and merchant ARE; it does not get to parse a fixed-width block.
// Only Westpac writes that block: run over a CommBank overdraft notice it
// produced the merchant "To avoid fees and interest," (event 235).
const desc = /westpac/i.test(ev.notif_app ?? "")
? parseWestpacDescriptor(ev.notif_body)
: { merchant: null, city: null, country: null };
const sub = parseWalletSubtitle(ev.notif_subtitle);
const merchant = sub.merchant ?? ev.merchant ?? desc.merchant ?? "Card purchase";
// Two sources for a place, and they are not equal. The Apple Wallet subtitle
// gives "Wyndham Vale, VIC"; the issuer's fixed-width descriptor gives
// "WYNDHAM VAL AU" -- a 12-character truncation in caps. The subtitle wins
// outright, which matters for the pair merge below: whichever half arrives
// second must be able to REPLACE a descriptor-derived place, or the result
// depends on arrival order.
const subtitleLocation = sub.location ?? null;
// Precedence: the Wallet subtitle, then the DETERMINISTIC descriptor parse,
// and only then the model's own answer. Getting this order wrong defeats the
// point of parsing positionally at all -- the model returned the mangled
// "SAN FRANCISOUS" and the padded "MELBOURNE AU" for exactly these rows,
// which is what wallet-descriptor.ts exists to correct.
const descriptorLocation = desc.city && desc.country
? `${desc.city} ${desc.country}`
: desc.city;
const location = subtitleLocation ?? descriptorLocation ?? ev.location ?? null;
// Currency: only ever what the notification actually states. A bare "$" is
// recorded as unknown, never assumed AUD -- harmless domestically, a 23x
// error on a THB tap abroad.
const stated = currencyFromAmountText(ev.amount_text)
?? (ev.currency && /^[A-Z]{3}$/.test(ev.currency) ? ev.currency : null);
if (!stated) flags.push("currency_unverified");
if (desc.country && desc.country !== "AU") flags.push(`foreign_country_${desc.country}`);
// Foreign: the native figure is what both sides can agree on exactly, so it
// is what gets stored. `amount` still needs a value (NOT NULL) and takes a
// provisional conversion, flagged, which the statement overwrites. Resolved
// BEFORE the lock: the ECB fallback is a network call, and the lock
// serialises every wallet post behind whatever runs inside it.
let amountAud = ev.amount;
let fxAmount: number | null = null;
let fxCode: string | null = null;
if (stated && stated !== "AUD") {
fxAmount = ev.amount;
fxCode = stated;
const ledgerRate = await medianRateToAud(stated);
const rate = ledgerRate ?? (await ecbRateToAud(stated, date));
if (rate) {
amountAud = Number((ev.amount * rate).toFixed(2));
flags.push("aud_provisional");
if (!ledgerRate) flags.push("rate_ecb");
} else {
// No rate from either source. The native figure must NOT fall through as
// AUD -- 672006 VND once landed as A$672,006. Zero is a flagged
// placeholder; the statement supplies the real AUD at reconcile time,
// which matches on the native pair, not on this column.
amountAud = 0;
flags.push("aud_provisional", "no_rate_reference");
}
}
// A gift card has no statement line coming, so it must not enter the
// reconcile queue (which filters on exactly this column).
const storedValue = isStoredValueInstrument(ev);
const paymentMethod = storedValue ? "credits" : "card";
if (storedValue) flags.push("stored_value_instrument");
// Everything from the idempotency re-check to the INSERT holds one lock.
// n8n's HTTP node posts a poll's items CONCURRENTLY, and both halves of a tap
// usually land in the same 15-minute poll -- so each ran findPairedRow before
// the other had inserted, neither found its partner, and one tap became two
// rows: 11 pairs, A$3,841.40, every pair with an identical created_at. A
// single lock for the whole lane, not one per amount: wallet volume is a few
// dozen a day, and a coarse lock has no key to get wrong.
return withTransaction(async (db) => {
await db.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended('wallet-ingest', 0))::text`);
const existing = await alreadyIngested(db, ev.event_id);
if (existing) {
return { transactionId: existing.id, metadataId: null, skipped: "already_ingested" as const, flags };
}
// Merge the paired notification rather than creating a second row for one
// tap. The issuer half brings the card tail; the Wallet half brings the
// merchant and suburb. COALESCE so whichever arrives second only ever fills
// gaps -- it can never blank a field the first half supplied.
const pair = await findPairedRow(
db, amountAud, notifiedAt, ev.notif_app ?? null,
fxCode && fxAmount !== null ? { amount: fxAmount, code: fxCode } : null
);
if (pair) {
await db.queryRaw(
`UPDATE transactions
SET merchant_name = COALESCE(NULLIF($2,'Card purchase'), merchant_name),
merchant_normalized = COALESCE(NULLIF($2,'Card purchase'), merchant_normalized),
-- A subtitle-derived place overwrites; anything else only fills a
-- gap. Without the $4 branch the merged row depends on which
-- notification happened to arrive first, and the issuer half
-- usually wins by a few seconds -- keeping the worse string.
location = CASE WHEN $4::boolean THEN COALESCE($3, location)
ELSE COALESCE(location, $3) END
WHERE id = $1`,
[pair.id, merchant, location, subtitleLocation !== null]
);
// The merged half records itself, or a replay finds nothing and merges
// it again (or, once the partner is superseded, mints a new row).
await db.queryRaw(
`UPDATE expense_metadata
SET card_last4 = COALESCE(card_last4, $2),
payment_method_detail = COALESCE(payment_method_detail, $3),
merchant_normalized = COALESCE(NULLIF($4,'Card purchase'), merchant_normalized),
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
WHERE id = $1`,
[pair.meta_id, ev.card_tail ?? null, ev.issuer ?? null, merchant, JSON.stringify([adoptedFlag(ev.event_id)])]
);
return { transactionId: pair.id, metadataId: pair.meta_id, skipped: "merged_pair" as const, flags };
}
// The receipt got here first. Fold into its row instead of writing a second
// one for the same purchase (see findWalletAdoptableTransaction). The column
// list IS the ownership boundary, the mirror of receipt-ingestion.ts's: the
// receipt read the register tape, so merchant, description, category and the
// items are its business and are absent here; `location` is wallet's own --
// nothing else writes a suburb -- and payment_method is filled only when the
// other lane left it unknown.
const adopted = await findWalletAdoptableTransaction(DEFAULT_OWNER_ID, amountAud, date, merchant, undefined, db);
if (adopted) {
await db.queryRaw(
`UPDATE transactions
SET location = COALESCE($2, location),
payment_method = COALESCE(payment_method, $3)
WHERE id = $1`,
[adopted.id, location, paymentMethod]
);
let metadataId = adopted.metadata_id;
if (metadataId) {
await db.queryRaw(
`UPDATE expense_metadata
SET card_last4 = COALESCE(card_last4, $2),
payment_method_detail = COALESCE(payment_method_detail, $3),
currency = COALESCE(currency, $4),
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
WHERE id = $1`,
[metadataId, ev.card_tail ?? null, ev.issuer ?? null, stated, JSON.stringify([...flags, adoptedFlag(ev.event_id)])]
);
} else {
// expense_metadata.transaction_id is UNIQUE, so this branch only runs
// when the adopted row genuinely has none -- and it must produce one, or
// the card tail and the flag that makes this idempotent have nowhere to
// live.
const meta = await db.queryRow<{ id: number }>(
`INSERT INTO expense_metadata (
transaction_id, source, source_message_id, merchant_normalized, amount,
transaction_date, card_last4, currency, payment_method,
payment_method_detail, line_items, flags, extraction_model,
notified_at, notif_app
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
RETURNING id`,
[
adopted.id, sourceRef, merchant, ev.amount, date, ev.card_tail ?? null, stated,
paymentMethod, ev.issuer ?? null, JSON.stringify([...flags, adoptedFlag(ev.event_id)]),
notifiedAt, ev.notif_app ?? null,
]
);
metadataId = meta!.id;
}
return { transactionId: adopted.id, metadataId, skipped: "adopted" as const, flags };
}
const txn = await db.queryRow<{ id: number }>(
`INSERT INTO transactions (
transaction_date, description, amount, amount_aud, category, payment_method,
merchant_name, merchant_normalized, transaction_type, owner_id, location,
foreign_currency_amount, foreign_currency_code, source, source_ref, row_index
) VALUES ($1,$2,$3,$3,'other',$10,$4,$4,'debit',$5,$6,$7,$8,'wallet',$9,0)
RETURNING id`,
[date, merchant, amountAud, merchant, DEFAULT_OWNER_ID, location, fxAmount, fxCode, sourceRef, paymentMethod]
);
const meta = await db.queryRow<{ id: number }>(
`INSERT INTO expense_metadata (
transaction_id, source, source_message_id, merchant_normalized, amount,
transaction_date, card_last4, currency, payment_method,
payment_method_detail, line_items, flags, extraction_model,
notified_at, notif_app
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
RETURNING id`,
[
txn!.id, sourceRef, merchant, ev.amount, date,
ev.card_tail ?? null, stated, paymentMethod, ev.issuer ?? null, JSON.stringify(flags),
notifiedAt, ev.notif_app ?? null,
]
);
return { transactionId: txn!.id, metadataId: meta!.id, flags };
}, { maxWait: 10_000, timeout: 30_000 });
}