Compare commits
23
Commits
91586ce4a4
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c25c82932d | ||
|
|
a1c2eb736f | ||
|
|
069034951e | ||
|
|
e956f39ffc | ||
|
|
d75589e128 | ||
|
|
ce0bf0a775 | ||
|
|
a3e4e1102a | ||
|
|
22fde30b26 | ||
|
|
48c6b2395f | ||
|
|
5076f27f68 | ||
|
|
10cf0aaf21 | ||
|
|
2251060ac6 | ||
|
|
2c05f10aed | ||
|
|
9b59edcf48 | ||
|
|
3d0d75519a | ||
|
|
8d1875225f | ||
|
|
8e66e527ae | ||
|
|
f31f12799a | ||
|
|
5e355c7e85 | ||
|
|
669f9119b9 | ||
|
|
258fe3b617 | ||
|
|
0823244e4a | ||
|
|
d88ca2a24c |
@@ -65,7 +65,7 @@ docker logs finance -f
|
||||
npm test # unit — pure logic, no database
|
||||
npm run test:integration # database-backed; resolves the DSN itself (see below)
|
||||
npm run test:all # both
|
||||
npm run test:setup # once, and after any migration: creates/refreshes personal_test
|
||||
npm run test:setup # once, and after any migration: drops and rebuilds personal_test
|
||||
npm run lint # blocking in CI since #79 — 0 errors, warnings allowed
|
||||
npx tsc --noEmit # blocking in CI; nothing else compiles the test files
|
||||
```
|
||||
@@ -86,6 +86,13 @@ had pointed the suite at **postgres-pantry** — whose database it would have
|
||||
TRUNCATEd had the credentials happened to match. A loopback address cannot be
|
||||
reassigned to another container, which is why the published port wins.
|
||||
|
||||
**`test:setup` really rebuilds now (2026-10-03).** It used to replay the
|
||||
schema dump over the existing database, where every `CREATE TABLE` failed with
|
||||
"already exists" and the old shape survived — so no migration ever reached
|
||||
`personal_test`, and new-column code failed every test with a misleading
|
||||
`column does not exist`. It now `DROP DATABASE … WITH (FORCE)`s first, refusing
|
||||
any name not ending `_test`, and loads with `ON_ERROR_STOP`.
|
||||
|
||||
**Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to
|
||||
truncate anything whose `current_database()` is not `personal_test`, and the
|
||||
suite runs `fileParallelism: false` because every file shares that one database
|
||||
@@ -176,6 +183,215 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
|
||||
- `statements` — one row per billing period per bank account
|
||||
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
|
||||
|
||||
### The change log (migration 0037)
|
||||
|
||||
`transaction_audit` is an **append-only** row-level record of every change to
|
||||
`transactions`, `transaction_splits`, `transaction_tags`, `transaction_overrides`
|
||||
and `expense_metadata`. Never `UPDATE` or `DELETE` from it. There is deliberately
|
||||
**no foreign key** to `transactions` — the log has to outlive the row it describes.
|
||||
|
||||
```sql
|
||||
-- everything that ever happened to one transaction
|
||||
SELECT changed_at, table_name, op, old_data, new_data, app_name
|
||||
FROM transaction_audit WHERE transaction_id = 6961 ORDER BY changed_at;
|
||||
|
||||
-- one logical edit, all the rows it touched (a reconcile moves 4 children at once)
|
||||
SELECT * FROM transaction_audit WHERE txid = <txid> ORDER BY id;
|
||||
```
|
||||
|
||||
**Why it exists.** On 2026-08-23 the n8n CC Statement Processor stamped
|
||||
`statement_id` onto seven statementless wallet/pantry rows without setting
|
||||
`reconciled_with_id`, double-counting $245.90 and hiding those rows from the
|
||||
reconcile queue (`WHERE statement_id IS NULL`). Answering *"was anything lost?"*
|
||||
took restoring the 02:30 `pg_dump` into a scratch database and diffing it column
|
||||
by column, because nothing recorded what changed.
|
||||
|
||||
**Why it is a trigger and not application code.** The ledger has at least five
|
||||
independent writers — this app (Prisma), n8n (raw SQL in a Postgres node),
|
||||
pantry-app, the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`.
|
||||
That bug was an n8n raw-SQL `UPDATE`; an app-level audit would not have seen it.
|
||||
|
||||
**Why not Postgres' own logs.** Checked live: `wal_level=replica` (WAL cannot be
|
||||
decoded into row changes), `wal_keep_size=0` and `archive_mode=off` (recycled
|
||||
almost immediately), `log_statement=none`, `track_commit_timestamp=off`, no
|
||||
`pgaudit`. Postgres records nothing usable and has no built-in audit or temporal
|
||||
tables.
|
||||
|
||||
Three details worth knowing before relying on it:
|
||||
- **`old_data`/`new_data` hold only the columns that actually changed** on an
|
||||
`UPDATE`. A whole-row copy would bury a one-column edit, and
|
||||
`expense_metadata.line_items` alone can be kilobytes.
|
||||
- **A no-op `UPDATE` writes nothing** (the upsert paths issue plenty).
|
||||
- **Attribution is partial and that is understood.** Every client connects as the
|
||||
same `personal` role, so `db_user` never discriminates. `app_name` is only as
|
||||
good as what each client sets; `client_addr` separates containers and is NULL
|
||||
for `docker exec` over the unix socket — itself a useful signal. `txid` is the
|
||||
reliable grouping key.
|
||||
|
||||
`TRUNCATE` does not fire row-level triggers, so the integration suite's table
|
||||
wipes produce no audit rows.
|
||||
|
||||
### Resolving a duplicate by hand (`POST /api/transactions/<id>/merge`)
|
||||
|
||||
`<id>` is the row that gets **hidden**; `other_id` is the one that **survives**.
|
||||
|
||||
| mode | requires | sets | for |
|
||||
|---|---|---|---|
|
||||
| `reconcile` | loser statementless, survivor a statement row | `reconciled_with_id` | the everyday case |
|
||||
| `supersede` | nothing | `superseded_by_id` | two rows on the same footing, e.g. a statement imported twice |
|
||||
|
||||
**There is no delete, deliberately.** Every child of `transactions` is
|
||||
`ON DELETE CASCADE`, so removing a row destroys the splits, tags and metadata
|
||||
that make it legible. Both modes keep the row and its children and merely stop
|
||||
it counting. Do not add a delete endpoint.
|
||||
|
||||
Both modes **move curation onto the survivor first**, through the single
|
||||
`moveCurationWithin()` in `lib/reconcile.ts` — because both hide the source row
|
||||
from every figure, and a split stranded on a hidden row stops counting in
|
||||
silence. That is also why the two callers share one implementation: the file
|
||||
header warns that a second copy would eventually move a different set.
|
||||
|
||||
Refusals (all deliberate, all tested in `merge-supersede.test.ts`): a row already
|
||||
reconciled or superseded on **either** side → 409 (re-pointing it strands the
|
||||
curation moved on the first pass); superseding under a row that is itself
|
||||
superseded → 409; a row that is not the caller's → 404. Ownership is checked with
|
||||
`COALESCE(t.owner_id, s.owner_id)` — a statement row carries `owner_id` on the
|
||||
statement, so checking `t.owner_id` alone rejects every statement row as "not
|
||||
found".
|
||||
|
||||
**Why it had to exist.** `superseded_by_id` was read everywhere (analytics and
|
||||
auto-reconcile both exclude it) but written **nowhere** in application code — set
|
||||
exactly once, by migration 0023. The reconcile queue was the only merge path, and
|
||||
it only offers candidates inside an amount band.
|
||||
|
||||
**The band is ±5%, not ±1% (2026-08-23).** A foreign card purchase is charged the
|
||||
notified amount **plus the card's foreign transaction fee** — 3% on the Westpac
|
||||
Altitude — so a wallet row and its statement line can never match exactly.
|
||||
`SP PEAKDESIGN` was notified at **$42.44** and charged **$43.71** (×1.03), which
|
||||
put the one row that *was* its twin outside the old band: the picker could not
|
||||
offer the only correct answer. Widening is safe because the list is capped at 5
|
||||
by the `rn` window and ordered by amount closeness, so a looser band changes
|
||||
*which* five appear, never how many, and a person confirms each. Measured on the
|
||||
live queue at the time of the change: candidate counts went **0 → 0** on all 12
|
||||
rows, i.e. no new noise. **The auto-matcher stays exact** — see `findAutoMatches()`.
|
||||
|
||||
### A charge that never posted (`POST /api/transactions/<id>/void`, migration 0040)
|
||||
|
||||
The third way a row stops counting, for a case with **no survivor**: a card
|
||||
notification is sent for an *authorisation*, and when the merchant releases the
|
||||
hold no statement line ever arrives. Txn 7951 — a $1,040.41 Airbnb hold on Amex,
|
||||
booking cancelled the next day — sat in the reconcile queue with nothing to
|
||||
match and counted as spend everywhere. `reconcile` needs a statement twin,
|
||||
`supersede` needs a survivor (and `transactions_no_self_supersede` blocks
|
||||
pointing a row at itself), so it gets its own columns: `voided_at` +
|
||||
`void_reason` (reason mandatory, by CHECK). Body `{reason}`; `DELETE` undoes it.
|
||||
UI: **Never posted** on every row of `/reconcile`.
|
||||
|
||||
- **In `EXCLUDE_RECONCILED_SOURCE`**, so every "count each purchase once" query
|
||||
drops it without being edited; the queue, `findAutoMatches`, both adoption
|
||||
helpers, annual charges and the manual reconcile POST filter it explicitly.
|
||||
- **Statementless only** — `transactions_void_statementless` enforces it in the
|
||||
database. A statement line is the bank's record of a posted charge; a wrong
|
||||
one is a supersede.
|
||||
- **Refused** on a reconciled row (it posted), a superseded row, and a row with a
|
||||
*settled* split (someone already paid their share — voiding would quietly turn
|
||||
that into a credit). `merge` refuses a voided row on either side.
|
||||
- **The wallet lane deliberately still SEES voided rows.** `alreadyIngested()` and
|
||||
the issuer/Wallet pairing in `wallet-ingestion.ts` do not filter `voided_at`:
|
||||
the voided row is the sink that stops a late twin notification, or the next
|
||||
`prompt_version` re-post, from booking the hold again. Same for the intake
|
||||
router's idempotency lookup. Do not "fix" those.
|
||||
- **Outside this repo, nothing filters it**: Hermes' `agent_ro` views
|
||||
(`cashflow_monthly` etc.) and ingestion-engine's `instalment_matcher` already
|
||||
ignore `superseded_by_id`/`reconciled_with_id` too, so a voided row is no worse
|
||||
there than any other hidden row.
|
||||
|
||||
### Upcoming annual charges (`GET /api/annual-charges`, migration 0038)
|
||||
|
||||
Card annual fees, loan package fees and annual subscriptions, derived from
|
||||
**payments alone** — independent of the email lane, because a bank announces an
|
||||
annual fee on the statement and nowhere else.
|
||||
|
||||
**`transaction_type = 'fee'` is the universe, not a description match.** The
|
||||
extractor already classifies fees structurally (131 rows), and *every* card and
|
||||
loan annual fee on this ledger is typed that way. The first version of this
|
||||
filtered `transaction_type = 'debit'` and returned **none** of them. The fees are
|
||||
not debits.
|
||||
|
||||
**Two signals, kept apart because they are not equally trustworthy:**
|
||||
|
||||
| confidence | basis |
|
||||
|---|---|
|
||||
| `confirmed` | typed a fee **and** repeated on an annual cadence |
|
||||
| `likely` | typed a fee, seen once, and *named* annual/card/package — the kind is certain, the cadence inferred |
|
||||
| `possible` | no fee wording; a merchant whose every gap is annual **and** whose amount is stable |
|
||||
|
||||
**A naive "two charges ~365 days apart" is useless** — measured on this ledger it
|
||||
ranks a bakery first with **202** such pairs, then a donation, Patreon, Coles.
|
||||
Frequency is the confound. Even "every gap is annual" admits a Target visited
|
||||
twice ($30, then $198). The `possible` tier therefore also requires amount
|
||||
stability (≤20%, which absorbs VicRoads' ~3%/yr CPI drift) and ≥2 observations.
|
||||
It still carries real false positives (H&M, Superdry) — that is what the label is
|
||||
for, and why it must never render like a bank fact.
|
||||
|
||||
**`expected_not_seen` is the more interesting status.** The estimate passed and
|
||||
nothing landed, which means the account stopped being imported, or closed, or the
|
||||
charge arrived under a different label. `account_days_stale` is the
|
||||
discriminator: Westpac at 0 days stale is trustworthy; ANZ at 392 is simply
|
||||
missing data. Never report these as upcoming.
|
||||
|
||||
**Account continuity (`account_links`, migration 0038).** One card can appear
|
||||
under two identities: Citi Prestige `5258…0253` (to 2025-11-13) became NAB MyCard
|
||||
`1015…6227` (from 2025-11-14), same 14th-to-13th cycle, no gap. Without the link
|
||||
the $700 fee reads as two dead one-offs instead of one due again ~Nov 2026. The
|
||||
table is **hand-asserted with evidence** (`source_documents` ids), not inferred —
|
||||
a contiguous-cycle heuristic would join two genuinely different accounts as
|
||||
readily as one renamed one. The resolver walks it forward recursively;
|
||||
`uq_account_links_from` forbids a fork and `account_links_no_self` a self-link, so
|
||||
it terminates. **Display the resolved current bank**, not the one that printed the
|
||||
last fee — naming the retired brand sends you to the wrong bank.
|
||||
|
||||
Known limitation: the merchant signal fragments on label drift. VicRoads rego was
|
||||
paid 2024/2025 as `VICROADS` and 2026 as `VICROADS ONLINE PAYM`, so the series
|
||||
splits and the 2025 estimate shows as `expected_not_seen`. Not fuzzy-matched on
|
||||
purpose — see the rejected pg_trgm merchant grouping.
|
||||
|
||||
### The wallet lane (`/api/wallet/ingest`, `lib/wallet-ingestion.ts`)
|
||||
|
||||
iOS notifications → `capture.wallet_events` → n8n **Wallet Interpret**
|
||||
(`CAp1s5cuzJk2mDaI`, local gemma, every 15 min) → this route. A row per tap,
|
||||
statementless, `source='wallet'`, `source_ref='evt:<id>'`; the statement later
|
||||
confirms it through the reconcile queue. Fixed 2026-10-03 after a month abroad
|
||||
(ticket #266); the rules below are each the fix for something that broke.
|
||||
|
||||
- **Storage convention differs from the order lane.** `amount` = `amount_aud` =
|
||||
a *provisional AUD* figure; the native figure is `foreign_currency_amount`.
|
||||
`NATIVE_CURRENCY` reads `amount_aud IS NOT NULL` as "amount is AUD" — without
|
||||
that an INR 524 tap displayed as "INR 9.42".
|
||||
- **Provisional rate:** ledger median over *statement* rows (≥5, carries the
|
||||
card's FX fee) → else ECB via Frankfurter (`rate_ecb`) → else `0` with
|
||||
`no_rate_reference`. Never the native figure as AUD (672006 VND once became
|
||||
A$672,006).
|
||||
- **One tap = two notifications** (issuer app + Wallet, 2–8 s apart), merged.
|
||||
A pair must be **≤10 minutes apart AND from different apps**
|
||||
(`expense_metadata.notified_at` / `notif_app`, migration 0039). The old
|
||||
amount-within-±1-day rule merged the UNICEF $50 into an unrelated $50 charge.
|
||||
- **The whole check→pair→adopt→insert runs under one advisory lock**
|
||||
(`hashtextextended('wallet-ingest',0)`), on the transaction's own connection.
|
||||
n8n posts a poll's items concurrently; without it 11 taps became 22 rows.
|
||||
Prisma cannot deserialise `void` — the lock SELECT needs `::text`.
|
||||
- **A merged half records `wallet_evt:<id>` in the survivor's flags**, so a
|
||||
replay is idempotent. Reconciled wallet rows carry **no** `expense_metadata`
|
||||
(reconcile moves it to the statement twin) — the flag check follows it there.
|
||||
- **Skipped, not booked:** account notices (`overdrawn`, `to avoid fees`, …) and
|
||||
transit-card drawdowns (`Your new/current balance is` — the Octopus *load* is
|
||||
the spend). The fixed-width descriptor parser runs on Westpac bodies only.
|
||||
- **Dated in Australia/Melbourne**, like the statements (UTC slicing filed 28 of
|
||||
134 rows a day early).
|
||||
- **Replay a few events by relabelling their interpretations**
|
||||
(`UPDATE capture.wallet_interpretations SET prompt_version='v1-<why>' WHERE …`),
|
||||
not by bumping `prompt_version`, which re-posts every event ever captured.
|
||||
|
||||
### Cash and reconciliation
|
||||
|
||||
`payment_method` in (`cash`, `credits`) excludes a transaction from
|
||||
@@ -699,6 +915,25 @@ and the cross-repo contract are in smarthome `plans/intake-router-slice-plan.md`
|
||||
delivery order's total is items + fees − promotions ($118.94 of Coles lines
|
||||
became $85.74), so checking lines against the total would hold every
|
||||
delivery order. Tolerance $1 or 2%; a failure is `held`, nothing written.
|
||||
- **Nothing that cannot be a receipt reaches the model, and the prompt names no
|
||||
real shop.** On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so
|
||||
a batch arrived as the five characters `Image` — and the model returned a
|
||||
whole receipt: `Burgertory (Braybrook)`, two "The Pride" lines, $41.15. None
|
||||
of it existed; "Burgertory" was in `ORDER_ITEM_PROMPT` as the example of how
|
||||
to print a branch in parentheses. Only the absent date kept it out of the
|
||||
matcher, and the invented header and items were still stored on the batch as
|
||||
`needs_header`. Three layers now, in order: `textCouldBeReceipt()` rejects a
|
||||
TEXT part with fewer than two words, no digit, or under 40 characters with no
|
||||
money figure (`TOTAL $17.00` passes — length alone is *not* the test)
|
||||
**before** any model call; the prompt's merchant examples are
|
||||
`PROMPT_PLACEHOLDER_MERCHANTS` format placeholders and it is told to return
|
||||
nulls for a non-receipt; and `confabulationReason()` catches what the
|
||||
pre-check cannot see — a PDF that renders to junk — plus any answer echoing a
|
||||
placeholder. All three file `unsupported`, never `needs_header`: a
|
||||
confabulated header is not a partial one. Same shape as the ingestion
|
||||
engine's 2026-08-05 empty-body defect — **a stage that can be handed nothing
|
||||
must say so, because the model downstream always answers, and it answers out
|
||||
of its prompt.**
|
||||
- **Names stay verbatim.** Burgertory calls its burgers "Michael" and "Joel";
|
||||
the prompt forbids canonicalising and the stored shape is the order lane's
|
||||
(`qty`, `description`, `amount`, `options`) so `order-details.tsx` renders it.
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
-- transaction_search: resolve expense_metadata by BOTH links, without fanning out.
|
||||
--
|
||||
-- THE DEFECT (measured 2026-08-22): the view joined only on
|
||||
-- `em.transaction_id = t.id`, so it silently omitted every card-settled order,
|
||||
-- which carries only `matched_transaction_id`. 148 transactions are affected --
|
||||
-- Glovo orders in Milan, Uber Eats, all `source='order-bridge'` -- and for each
|
||||
-- of them the view reports NULL payment_method / order_reference / line_items
|
||||
-- while the data sits in the table.
|
||||
--
|
||||
-- That is not cosmetic. This view is the agent read surface: Hermes'
|
||||
-- postgres-personal MCP selects payment_method and order_reference from it
|
||||
-- (personal-agent-gateway/mcp-servers/postgres-personal/queries.py), and
|
||||
-- home-dashboard reads it via dash_ro. CLAUDE.md states the rule the view broke:
|
||||
-- "That lookup must match transaction_id OR matched_transaction_id: a
|
||||
-- card-settled order has only the latter."
|
||||
--
|
||||
-- NOT a fan-out fix, contrary to a first reading. The original join CANNOT fan
|
||||
-- out: expense_metadata_transaction_id_key is UNIQUE on transaction_id, so it
|
||||
-- matched at most one row by construction. Sums taken over this view were never
|
||||
-- inflated.
|
||||
--
|
||||
-- But broadening the join is exactly what INTRODUCES that risk, which is why
|
||||
-- LATERAL is used rather than a second OR'd condition. transaction_id and
|
||||
-- matched_transaction_id are enforced by two SEPARATE unique indexes
|
||||
-- (expense_metadata_transaction_id_key, uq_expense_matched_txn), so one
|
||||
-- transaction can legally be pointed at by two DIFFERENT rows -- one via each
|
||||
-- link. Zero such transactions exist today; the LATERAL means that stays a
|
||||
-- non-event rather than a silent duplicate in the agent's spending totals.
|
||||
--
|
||||
-- Tie-break matches every other reader (getTransactions, replaceLineItems,
|
||||
-- matchTransaction, /api/transactions/[id]/order), so the row this view resolves
|
||||
-- is the row the UI shows.
|
||||
--
|
||||
-- Verified before applying: row count unchanged at 4760 (= count(transactions)),
|
||||
-- sum(amount) over debits unchanged at 1514412.89 -- amount comes from
|
||||
-- transactions, never from the metadata row -- and the only difference is the
|
||||
-- 148 rows that gain the metadata they always should have had.
|
||||
--
|
||||
-- CREATE OR REPLACE VIEW DROPS GRANTS. dash_ro (home-dashboard) and agent_ro
|
||||
-- (Hermes) both read this view and losing either breaks a surface with no error
|
||||
-- at replace time, so both are re-granted below.
|
||||
|
||||
CREATE OR REPLACE VIEW transaction_search AS
|
||||
SELECT t.id,
|
||||
t.transaction_date,
|
||||
t.description,
|
||||
COALESCE(t.merchant_normalized, t.merchant_name) AS merchant,
|
||||
t.category,
|
||||
t.transaction_type,
|
||||
COALESCE(t.amount_aud, t.amount) AS amount,
|
||||
t.foreign_currency_amount,
|
||||
t.foreign_currency_code,
|
||||
t.location,
|
||||
s.bank_name,
|
||||
s.card_name,
|
||||
s.account_number,
|
||||
s.statement_type,
|
||||
s.billing_end_date,
|
||||
em.payment_method,
|
||||
em.order_reference,
|
||||
em.line_items
|
||||
FROM transactions t
|
||||
LEFT JOIN statements s ON s.id = t.statement_id
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT em2.payment_method, em2.order_reference, em2.line_items
|
||||
FROM expense_metadata em2
|
||||
WHERE em2.transaction_id = t.id OR em2.matched_transaction_id = t.id
|
||||
ORDER BY (em2.transaction_id = t.id) DESC NULLS LAST, em2.id
|
||||
LIMIT 1
|
||||
) em ON true;
|
||||
|
||||
GRANT SELECT ON transaction_search TO dash_ro;
|
||||
GRANT SELECT ON transaction_search TO agent_ro;
|
||||
@@ -0,0 +1,73 @@
|
||||
-- Normalise foreign_currency_code to ISO-4217, and clear two sentinel values.
|
||||
--
|
||||
-- WHY THIS MATTERS: exact-code equality is the whole basis of FX reconciliation.
|
||||
-- ingestion-engine's order_transaction_bridge matches foreign orders with
|
||||
-- AND t.foreign_currency_code = %(ccy)s
|
||||
-- AND t.foreign_currency_amount = %(total)s
|
||||
-- so a row labelled 'BAHT' can never match a THB order, and a row labelled
|
||||
-- 'null' can never match anything. 226 of 1,632 coded rows (14%) were
|
||||
-- unmatchable for this reason alone, which silently degrades a shipped job.
|
||||
--
|
||||
-- THE SENTINELS. 187 rows carry the literal 4-character STRING 'null' rather
|
||||
-- than SQL NULL, and 42 of those also carry foreign_currency_amount = -1.00
|
||||
-- exactly. Neither is data: they are "no value" markers that survived a truthy
|
||||
-- guard. Root cause is in the CC statement processors' Parse node, which does
|
||||
-- foreign_currency_code: t.foreign_currency_code || null
|
||||
-- and a non-empty string is truthy, so a model that answers with the STRING
|
||||
-- "null" (rather than JSON null) passes straight through. Same for -1. Fixed
|
||||
-- workflow-side in the same change; this migration cleans what already landed.
|
||||
-- Not purely historical -- 155 rows in 2026-03, but also 8 in May, 3 in July
|
||||
-- and 21 in August 2026.
|
||||
--
|
||||
-- MAPPINGS WERE VERIFIED AGAINST IMPLIED RATE, NOT TRUSTED FROM THE LABEL,
|
||||
-- because a row whose rate contradicts its claimed currency is a broken ledger
|
||||
-- row rather than a mislabelled one (the trap order_transaction_bridge's rate
|
||||
-- guard exists for: 2 of its first 73 matches were broken rows, not bad
|
||||
-- matches). Median amount/foreign_currency_amount per raw code vs the ISO
|
||||
-- median over statement rows (n>=5):
|
||||
-- BAHT 0.0433 vs THB 0.0470 INDIAN RUPEE 0.0183 vs INR 0.0180
|
||||
-- U.A.E. DIRHAM 0.4287 vs AED 0.4184 US DOLLAR 1.4054 vs USD 1.5513
|
||||
-- NZ DOLLAR 0.8243 vs NZD 0.9311 EURO 1.6342 vs EUR 1.6964
|
||||
-- Every label agrees with its rate. foreign_currency_amount is untouched by the
|
||||
-- relabels -- only the code was ever wrong, so this is a relabel, never a
|
||||
-- recompute.
|
||||
--
|
||||
-- EXPECTED ROW COUNTS (verified pre-flight, includes superseded rows):
|
||||
-- code 'null' -> NULL 187 BAHT -> THB 13
|
||||
-- amount -1.00 -> NULL 42 U.A.E. DIRHAM -> AED 10
|
||||
-- NZ DOLLAR -> NZD 6 US DOLLAR -> USD 4
|
||||
-- INDIAN RUPEE -> INR 4 EURO -> EUR 2
|
||||
-- Pre-flight also confirmed: no -1.00 amount exists outside the 'null'-code
|
||||
-- rows, and no other non-ISO code exists.
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- Sentinels first: the -1.00 amounts live only on 'null'-code rows, so clear
|
||||
-- them before that code is nulled and the set becomes unfindable.
|
||||
UPDATE transactions SET foreign_currency_amount = NULL
|
||||
WHERE foreign_currency_amount = -1.00 AND foreign_currency_code = 'null';
|
||||
|
||||
UPDATE transactions SET foreign_currency_code = NULL
|
||||
WHERE foreign_currency_code = 'null';
|
||||
|
||||
-- Word-form relabels.
|
||||
UPDATE transactions SET foreign_currency_code = 'THB' WHERE foreign_currency_code = 'BAHT';
|
||||
UPDATE transactions SET foreign_currency_code = 'AED' WHERE foreign_currency_code = 'U.A.E. DIRHAM';
|
||||
UPDATE transactions SET foreign_currency_code = 'NZD' WHERE foreign_currency_code = 'NZ DOLLAR';
|
||||
UPDATE transactions SET foreign_currency_code = 'USD' WHERE foreign_currency_code = 'US DOLLAR';
|
||||
UPDATE transactions SET foreign_currency_code = 'INR' WHERE foreign_currency_code = 'INDIAN RUPEE';
|
||||
UPDATE transactions SET foreign_currency_code = 'EUR' WHERE foreign_currency_code = 'EURO';
|
||||
|
||||
-- Postcondition: every remaining code is ISO-4217 shaped. Fails the whole
|
||||
-- transaction rather than leaving a partial normalisation behind.
|
||||
DO $$
|
||||
DECLARE bad int;
|
||||
BEGIN
|
||||
SELECT count(*) INTO bad FROM transactions
|
||||
WHERE foreign_currency_code IS NOT NULL AND foreign_currency_code !~ '^[A-Z]{3}$';
|
||||
IF bad <> 0 THEN
|
||||
RAISE EXCEPTION 'normalisation incomplete: % non-ISO codes remain', bad;
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
COMMIT;
|
||||
@@ -0,0 +1,152 @@
|
||||
-- Append-only change log for the ledger and its curation children.
|
||||
--
|
||||
-- WHY THIS EXISTS: on 2026-08-23 the CC Statement Processor's `Run Reconciliation`
|
||||
-- node stamped `statement_id` onto seven statementless wallet/pantry rows without
|
||||
-- setting `reconciled_with_id` or superseding the statement's own extracted line.
|
||||
-- The purchase counted twice, and because finance-app's reconcile queue is
|
||||
-- `WHERE statement_id IS NULL`, the stamp also hid those rows from the only thing
|
||||
-- that could repair them. Answering "was anything lost?" required restoring the
|
||||
-- 02:30 pg_dump into a scratch database and diffing it column by column, because
|
||||
-- NOTHING in this estate records what changed. That is the gap this closes.
|
||||
--
|
||||
-- WHY AT THE DATABASE AND NOT IN THE APP: the ledger has at least five independent
|
||||
-- writers -- finance-app (Prisma), n8n (raw SQL in a Postgres node), pantry-app,
|
||||
-- the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`. The 4a bug
|
||||
-- was an n8n raw-SQL UPDATE; an application-level audit would not have seen it at
|
||||
-- all. A trigger cannot be bypassed by choice of client.
|
||||
--
|
||||
-- WHY NOT POSTGRES' OWN LOGS: checked live on this instance -- wal_level=replica
|
||||
-- (so WAL cannot be logically decoded into row changes), wal_keep_size=0 and
|
||||
-- archive_mode=off (so WAL is recycled almost immediately), log_statement=none,
|
||||
-- logging_collector=off, track_commit_timestamp=off, and pgaudit is not installed.
|
||||
-- Postgres was recording nothing usable. It has no built-in audit or temporal
|
||||
-- tables; this always has to be built.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS transaction_audit (
|
||||
id bigserial PRIMARY KEY,
|
||||
changed_at timestamptz NOT NULL DEFAULT now(),
|
||||
-- Groups every row touched by one logical edit. `applyReconcileMatch` moves
|
||||
-- overrides, tags, splits and metadata in a single transaction; without this
|
||||
-- they read as four unrelated events.
|
||||
txid bigint NOT NULL DEFAULT txid_current(),
|
||||
table_name text NOT NULL,
|
||||
op text NOT NULL CHECK (op IN ('INSERT','UPDATE','DELETE')),
|
||||
-- Full primary key as an object, because transaction_tags is composite
|
||||
-- (transaction_id, tag_id) and the others are a bare `id`.
|
||||
row_key jsonb NOT NULL,
|
||||
-- Denormalised so "everything that ever happened to transaction N" is one
|
||||
-- index scan rather than a five-way union.
|
||||
transaction_id bigint,
|
||||
-- On UPDATE these hold ONLY the columns that actually changed. Storing whole
|
||||
-- rows would bury a one-column edit in forty unchanged fields, and
|
||||
-- expense_metadata.line_items alone can be kilobytes.
|
||||
old_data jsonb,
|
||||
new_data jsonb,
|
||||
-- Attribution is partial and that is understood: every client connects as the
|
||||
-- same `personal` role. app_name is only as good as what each client sets;
|
||||
-- client_addr distinguishes containers and is NULL for `docker exec` over the
|
||||
-- unix socket, which is itself a useful signal.
|
||||
db_user text NOT NULL DEFAULT current_user,
|
||||
app_name text,
|
||||
client_addr inet
|
||||
);
|
||||
|
||||
COMMENT ON TABLE transaction_audit IS
|
||||
'Append-only row-level change log for transactions + curation children. Never UPDATE or DELETE from this table.';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS transaction_audit_txn_time
|
||||
ON transaction_audit (transaction_id, changed_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS transaction_audit_time
|
||||
ON transaction_audit (changed_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS transaction_audit_txid
|
||||
ON transaction_audit (txid);
|
||||
CREATE INDEX IF NOT EXISTS transaction_audit_table_time
|
||||
ON transaction_audit (table_name, changed_at DESC);
|
||||
|
||||
-- TG_ARGV[0]: comma-separated candidate columns for the owning transaction id;
|
||||
-- the first non-null wins (expense_metadata carries the id in
|
||||
-- transaction_id OR matched_transaction_id -- a card-settled order
|
||||
-- only ever has the latter).
|
||||
-- TG_ARGV[1]: comma-separated primary key columns.
|
||||
CREATE OR REPLACE FUNCTION log_transaction_change() RETURNS trigger
|
||||
LANGUAGE plpgsql SECURITY DEFINER
|
||||
SET search_path = public, pg_temp
|
||||
AS $fn$
|
||||
DECLARE
|
||||
v_full jsonb;
|
||||
v_old jsonb;
|
||||
v_new jsonb;
|
||||
v_key jsonb := '{}'::jsonb;
|
||||
v_txn bigint;
|
||||
k text;
|
||||
BEGIN
|
||||
IF TG_OP = 'INSERT' THEN
|
||||
v_new := to_jsonb(NEW);
|
||||
v_full := v_new;
|
||||
ELSIF TG_OP = 'DELETE' THEN
|
||||
v_old := to_jsonb(OLD);
|
||||
v_full := v_old;
|
||||
ELSE
|
||||
v_old := to_jsonb(OLD);
|
||||
v_new := to_jsonb(NEW);
|
||||
v_full := v_new;
|
||||
-- Reduce to the changed columns only. IS DISTINCT FROM so a NULL->NULL
|
||||
-- comparison counts as unchanged rather than evaluating to NULL.
|
||||
FOR k IN SELECT jsonb_object_keys(v_old) LOOP
|
||||
IF (v_old -> k) IS NOT DISTINCT FROM (v_new -> k) THEN
|
||||
v_old := v_old - k;
|
||||
v_new := v_new - k;
|
||||
END IF;
|
||||
END LOOP;
|
||||
-- A no-op UPDATE is noise, and finance-app's upserts issue plenty of them.
|
||||
IF v_old = '{}'::jsonb THEN
|
||||
RETURN NULL;
|
||||
END IF;
|
||||
END IF;
|
||||
|
||||
FOREACH k IN ARRAY string_to_array(TG_ARGV[1], ',') LOOP
|
||||
v_key := v_key || jsonb_build_object(k, v_full -> k);
|
||||
END LOOP;
|
||||
|
||||
FOREACH k IN ARRAY string_to_array(TG_ARGV[0], ',') LOOP
|
||||
IF v_txn IS NULL AND (v_full ->> k) IS NOT NULL THEN
|
||||
v_txn := (v_full ->> k)::bigint;
|
||||
END IF;
|
||||
END LOOP;
|
||||
|
||||
INSERT INTO transaction_audit
|
||||
(table_name, op, row_key, transaction_id, old_data, new_data, app_name, client_addr)
|
||||
VALUES
|
||||
(TG_TABLE_NAME, TG_OP, v_key, v_txn, v_old, v_new,
|
||||
NULLIF(current_setting('application_name', true), ''), inet_client_addr());
|
||||
|
||||
RETURN NULL; -- AFTER trigger; return value is ignored
|
||||
END
|
||||
$fn$;
|
||||
|
||||
-- AFTER, so a statement that raises never leaves a phantom audit row; the audit
|
||||
-- write is in the same transaction as the change and rolls back with it.
|
||||
DROP TRIGGER IF EXISTS trg_audit_transactions ON transactions;
|
||||
CREATE TRIGGER trg_audit_transactions
|
||||
AFTER INSERT OR UPDATE OR DELETE ON transactions
|
||||
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('id', 'id');
|
||||
|
||||
DROP TRIGGER IF EXISTS trg_audit_transaction_splits ON transaction_splits;
|
||||
CREATE TRIGGER trg_audit_transaction_splits
|
||||
AFTER INSERT OR UPDATE OR DELETE ON transaction_splits
|
||||
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
|
||||
|
||||
DROP TRIGGER IF EXISTS trg_audit_transaction_tags ON transaction_tags;
|
||||
CREATE TRIGGER trg_audit_transaction_tags
|
||||
AFTER INSERT OR UPDATE OR DELETE ON transaction_tags
|
||||
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'transaction_id,tag_id');
|
||||
|
||||
DROP TRIGGER IF EXISTS trg_audit_transaction_overrides ON transaction_overrides;
|
||||
CREATE TRIGGER trg_audit_transaction_overrides
|
||||
AFTER INSERT OR UPDATE OR DELETE ON transaction_overrides
|
||||
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
|
||||
|
||||
DROP TRIGGER IF EXISTS trg_audit_expense_metadata ON expense_metadata;
|
||||
CREATE TRIGGER trg_audit_expense_metadata
|
||||
AFTER INSERT OR UPDATE OR DELETE ON expense_metadata
|
||||
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id,matched_transaction_id', 'id');
|
||||
@@ -0,0 +1,76 @@
|
||||
-- Account continuity across a rebrand, reissue or issuer change.
|
||||
--
|
||||
-- WHY THIS EXISTS: one card can appear in `statements` under two completely
|
||||
-- different identities. The live case is the Citi Prestige card, which moved to
|
||||
-- NAB and was rebranded MyCard:
|
||||
--
|
||||
-- Citibank 5258073330340253 statements ... 2025-11-13
|
||||
-- NAB 1015001000116227 statements 2025-11-14 ...
|
||||
--
|
||||
-- Same 14th-to-13th billing cycle, no gap, one continuous product. Nothing in
|
||||
-- the schema linked them, so anything grouping by (bank_name, account) sees a
|
||||
-- dead account and a new one with no history. For the annual-fee detector that
|
||||
-- is not cosmetic: the $700 fee charged 2024-12-13 and 2025-11-13 reads as
|
||||
-- "stopped" when it is due again around November 2026 under the new name.
|
||||
--
|
||||
-- WHY A TABLE AND NOT AN INFERENCE: contiguous-cycle heuristics would join two
|
||||
-- genuinely different accounts as readily as one renamed one, and a wrong join
|
||||
-- merges two real fee series. There are only a handful of these events in a
|
||||
-- lifetime, each announced by mail, so each link is asserted once with the
|
||||
-- document that proves it. `evidence` is not decoration -- it is how a future
|
||||
-- reader checks the claim instead of trusting it.
|
||||
--
|
||||
-- NOT `account_owner_mappings`: that answers "whose account is this" and keys on
|
||||
-- the raw (bank_name, account_number). This answers "is this the same account as
|
||||
-- that one", and keys on the normalised form.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS account_links (
|
||||
id bigserial PRIMARY KEY,
|
||||
-- Normalised the same way statements.account_number_key is generated
|
||||
-- (non-alphanumerics stripped), so these join without per-call cleanup.
|
||||
-- Amex-style masks keep their X's: they record which digits were redacted.
|
||||
from_bank_name text NOT NULL,
|
||||
from_account_key text NOT NULL,
|
||||
to_bank_name text NOT NULL,
|
||||
to_account_key text NOT NULL,
|
||||
-- When the successor took over. The predecessor's last statement and the
|
||||
-- successor's first are the observable boundary; this is the announced date
|
||||
-- where mail gives one.
|
||||
effective_date date,
|
||||
reason text NOT NULL
|
||||
CHECK (reason IN ('rebrand', 'reissue', 'issuer_change', 'migration')),
|
||||
-- Provenance for the claim, e.g. 'source_documents:32919'. Free text because
|
||||
-- the evidence is not always a document -- sometimes it is a statement pair.
|
||||
evidence text,
|
||||
notes text,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
|
||||
-- One successor per predecessor: a chain, never a fork. A fork would make
|
||||
-- "the current identity of this account" ambiguous, and the resolver walks
|
||||
-- this forward assuming it is not.
|
||||
CONSTRAINT uq_account_links_from UNIQUE (from_bank_name, from_account_key),
|
||||
-- A link to itself would make that walk non-terminating.
|
||||
CONSTRAINT account_links_no_self CHECK (
|
||||
from_bank_name IS DISTINCT FROM to_bank_name
|
||||
OR from_account_key IS DISTINCT FROM to_account_key
|
||||
)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE account_links IS
|
||||
'Predecessor -> successor account identity across rebrands/reissues. Hand-asserted, each row carrying the evidence for its claim.';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS account_links_to
|
||||
ON account_links (to_bank_name, to_account_key);
|
||||
|
||||
-- The one link the mail record proves today.
|
||||
-- source_documents 32919 (2025-10-01) "your Citi Prestige Card account is moving to MyCard"
|
||||
-- source_documents 32618 (2025-11-10) "MyCard is coming soon"
|
||||
-- source_documents 32539 (2025-11-24) "The new MyCard App and MyCard Online are now live"
|
||||
-- Statement boundary: Citibank ends 2025-11-13, NAB begins 2025-11-14.
|
||||
INSERT INTO account_links
|
||||
(from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason, evidence, notes)
|
||||
VALUES
|
||||
('Citibank', '5258073330340253', 'NAB', '1015001000116227', '2025-11-14', 'issuer_change',
|
||||
'source_documents:32919,32618,32539',
|
||||
'Citi Prestige -> MyCard Prestige. Billing cycle (14th-13th) continues unbroken across the boundary.')
|
||||
ON CONFLICT (from_bank_name, from_account_key) DO NOTHING;
|
||||
@@ -0,0 +1,26 @@
|
||||
-- When, and from which app, a wallet notification arrived.
|
||||
--
|
||||
-- WHY: one tap emits two notifications 2-8 seconds apart -- the issuer app
|
||||
-- (card tail) and Apple Wallet (merchant, suburb) -- and wallet-ingestion.ts
|
||||
-- merges them. It used to recognise the pair by amount alone, within +/-1 DAY,
|
||||
-- which is wide enough to fold two different purchases together: on 2026-10-01
|
||||
-- an Amex "$50.00 at A MEDIUM" notification merged into the UNICEF $50 donation
|
||||
-- from 17 hours earlier and renamed it. A real pair is seconds apart and comes
|
||||
-- from two DIFFERENT apps; two notifications from one app are two charges.
|
||||
-- Neither fact was stored anywhere on the ledger side, so the matcher could not
|
||||
-- ask. These columns let it.
|
||||
--
|
||||
-- Backfilled from capture.wallet_events through source_message_id ('evt:<id>'),
|
||||
-- which every wallet-written metadata row carries.
|
||||
|
||||
ALTER TABLE expense_metadata
|
||||
ADD COLUMN IF NOT EXISTS notified_at timestamptz,
|
||||
ADD COLUMN IF NOT EXISTS notif_app text;
|
||||
|
||||
UPDATE expense_metadata em
|
||||
SET notified_at = e.received_at,
|
||||
notif_app = e.notif_app
|
||||
FROM capture.wallet_events e
|
||||
WHERE em.source = 'wallet'
|
||||
AND em.source_message_id = 'evt:' || e.id
|
||||
AND em.notified_at IS NULL;
|
||||
@@ -0,0 +1,43 @@
|
||||
-- A statementless row for a charge that never posted.
|
||||
--
|
||||
-- WHY: the wallet lane books every card notification as a purchase, and a card
|
||||
-- notification is sent for an AUTHORISATION, not a settlement. When the
|
||||
-- merchant releases the hold the bank never posts the charge, so no statement
|
||||
-- line ever arrives: the row sits in the reconcile queue forever and counts as
|
||||
-- spend in every figure. Txn 7951: Airbnb HMXAHAW49Y, $1,040.41 held on Amex
|
||||
-- 1000 on 2026-09-24, booking cancelled 25 Sep inside the free window, hold
|
||||
-- released ("We removed the $1,040.41 (AUD) payment hold"). Statement 219
|
||||
-- (11 Sep - 10 Oct) has no such line and reconciles to its header without it.
|
||||
--
|
||||
-- Neither existing exclusion fits. reconciled_with_id needs a statement twin
|
||||
-- and superseded_by_id needs a surviving row (and may not point at itself,
|
||||
-- transactions_no_self_supersede). There is no survivor: nothing happened.
|
||||
--
|
||||
-- Not a DELETE: every child of transactions cascades, and the row is also the
|
||||
-- sink the wallet lane checks before re-posting an event. Deleting it would let
|
||||
-- the next prompt_version bump re-create it.
|
||||
|
||||
ALTER TABLE transactions
|
||||
ADD COLUMN IF NOT EXISTS voided_at timestamptz,
|
||||
ADD COLUMN IF NOT EXISTS void_reason text;
|
||||
|
||||
COMMENT ON COLUMN transactions.voided_at IS
|
||||
'Set when this statementless row records a charge that never posted (e.g. a released authorisation hold). Excluded from every figure and from the reconcile queue, kept for its children and audit trail. NULL = live.';
|
||||
|
||||
-- A reason is mandatory: a void with no explanation is indistinguishable from
|
||||
-- a mistake six months later.
|
||||
ALTER TABLE transactions
|
||||
ADD CONSTRAINT transactions_void_has_reason
|
||||
CHECK ((voided_at IS NULL AND void_reason IS NULL)
|
||||
OR (voided_at IS NOT NULL AND btrim(coalesce(void_reason, '')) <> ''));
|
||||
|
||||
-- Statement rows cannot be voided. A statement line is the bank's own record
|
||||
-- of a posted charge, and the statement's rows must keep summing to its
|
||||
-- header; a wrong statement row is a supersede, not a void.
|
||||
ALTER TABLE transactions
|
||||
ADD CONSTRAINT transactions_void_statementless
|
||||
CHECK (voided_at IS NULL OR statement_id IS NULL);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_transactions_voided
|
||||
ON transactions (voided_at)
|
||||
WHERE voided_at IS NOT NULL;
|
||||
@@ -190,7 +190,9 @@ model transactions {
|
||||
owner_id Int?
|
||||
reconciled_with_id Int?
|
||||
superseded_by_id Int?
|
||||
principal_amount Decimal? @db.Decimal(12, 2)
|
||||
voided_at DateTime? @db.Timestamptz(6) // a charge that never posted (migration 0040)
|
||||
void_reason String?
|
||||
principal_amount Decimal? @db.Decimal(12, 2)
|
||||
interest_amount Decimal? @db.Decimal(12, 2)
|
||||
source String? // feed this row came from, e.g. "frollo"; NULL = statement/manual (migration 0028)
|
||||
source_ref String? // the provider's own transaction id — idempotency key for re-imports
|
||||
@@ -237,6 +239,9 @@ model expense_metadata {
|
||||
tender_raw String?
|
||||
receipt_sha256 String?
|
||||
receipt_group String?
|
||||
// 0039 — when/where a wallet notification arrived; the pair merge needs both.
|
||||
notified_at DateTime? @db.Timestamptz(6)
|
||||
notif_app String?
|
||||
transaction transactions? @relation(fields: [transaction_id], references: [id], onDelete: Cascade)
|
||||
|
||||
@@unique([source, order_reference], name: "uq_expense_source_order")
|
||||
|
||||
@@ -19,14 +19,23 @@ if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create test database (ignore error if it already exists)
|
||||
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres \
|
||||
-c "CREATE DATABASE $TEST_DB;" 2>/dev/null || true
|
||||
# Drop and recreate, so the schema is rebuilt rather than replayed. Replaying
|
||||
# the dump over an existing database was a silent no-op for every migration:
|
||||
# each CREATE TABLE failed with "already exists" and the old shape survived, so
|
||||
# a new column (0039's expense_metadata.notified_at, 2026-10-03) never reached
|
||||
# the test database. The guard is because this now DROPs: it refuses anything
|
||||
# not named *_test, whatever TEST_DB was set to.
|
||||
case "$TEST_DB" in
|
||||
*_test) ;;
|
||||
*) echo "ERROR: refusing to drop '$TEST_DB' -- the test database must be named *_test"; exit 1 ;;
|
||||
esac
|
||||
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres -q \
|
||||
-c "DROP DATABASE IF EXISTS $TEST_DB WITH (FORCE);" \
|
||||
-c "CREATE DATABASE $TEST_DB;"
|
||||
|
||||
# Wipe and rebuild schema from production (schema only, no data)
|
||||
echo "Copying schema from $PG_USER to $TEST_DB..."
|
||||
docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \
|
||||
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q
|
||||
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q -v ON_ERROR_STOP=1
|
||||
|
||||
echo "Schema ready."
|
||||
|
||||
|
||||
@@ -105,6 +105,19 @@ describe("NATIVE_CURRENCY", () => {
|
||||
expect(unconverted).toBe(true);
|
||||
});
|
||||
|
||||
it("labels a statement-less wallet row AUD, because its amount already is", async () => {
|
||||
// Wallet rows store a provisional AUD figure in amount AND amount_aud, and
|
||||
// the native figure in foreign_currency_amount. Reading the foreign code
|
||||
// here printed "INR 9.42" for an INR 524 tap (txn 8162).
|
||||
const id = await scratchTxn(
|
||||
"transaction_date, description, amount, amount_aud, transaction_type, payment_method, source, source_ref, foreign_currency_amount, foreign_currency_code",
|
||||
"'2026-10-03','Wallet foreign fixture', 9.42, 9.42, 'debit', 'card', 'wallet', 'evt:fixture-native-ccy', 524.00, 'INR'"
|
||||
);
|
||||
const { ccy, unconverted } = await currencyOf(id);
|
||||
expect(ccy).toBe("AUD");
|
||||
expect(unconverted).toBe(false);
|
||||
});
|
||||
|
||||
it("prefers the statement's currency over the foreign-charge record", async () => {
|
||||
// Opposite convention: on an AUD statement, `amount` is AUD and
|
||||
// foreign_currency_code merely notes what was originally charged. Reading
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants, mockDbWithPool } from "./helpers";
|
||||
|
||||
/**
|
||||
* Upcoming annual charges from payments alone.
|
||||
*
|
||||
* The exclusions carry this file. Anything can be made to find a card annual
|
||||
* fee; the difficulty is not also reporting the monthly account-servicing fee,
|
||||
* the per-transfer Wise charge, and the restaurant visited twice a year apart.
|
||||
* Measured on the real ledger, a naive "two charges ~365 days apart" ranks a
|
||||
* bakery first with 202 such pairs.
|
||||
*/
|
||||
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let getUpcomingAnnualCharges: typeof import("../../lib/annual-charges").getUpcomingAnnualCharges;
|
||||
|
||||
let acctSeq = 0;
|
||||
|
||||
/** A statement plus one transaction on it, for a named account. */
|
||||
async function charge(
|
||||
bank: string,
|
||||
acct: string,
|
||||
date: string,
|
||||
description: string,
|
||||
amount: number,
|
||||
type = "fee"
|
||||
) {
|
||||
const key = `${bank}:${acct}`;
|
||||
let stId = stmtCache.get(`${key}:${date}`);
|
||||
if (!stId) {
|
||||
const st = await pool.query(
|
||||
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||
VALUES ($4, $1, $2, $3::date - 20, $3::date + 10, $5) RETURNING id`,
|
||||
[acct, bank, date, `s${acctSeq++}.pdf`, ownerId]
|
||||
);
|
||||
stId = st.rows[0].id as number;
|
||||
stmtCache.set(`${key}:${date}`, stId);
|
||||
}
|
||||
await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||
VALUES ($1,$2,$3,$4,$5,$6)`,
|
||||
[stId, date, description, amount, type, ownerId]
|
||||
);
|
||||
}
|
||||
let stmtCache = new Map<string, number>();
|
||||
|
||||
/** A statementless merchant purchase, for the non-fee signal. */
|
||||
async function purchase(date: string, merchant: string, amount: number) {
|
||||
await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id, merchant_normalized, row_index)
|
||||
VALUES (NULL,$1,$2,$3,'debit',$4,$2,0)`,
|
||||
[date, merchant, amount, ownerId]
|
||||
);
|
||||
}
|
||||
|
||||
const find = <T extends { label: string }>(rows: T[], label: string) =>
|
||||
rows.find((r) => r.label === label);
|
||||
|
||||
beforeAll(async () => {
|
||||
pool = createPool();
|
||||
mockDbWithPool(pool);
|
||||
({ getUpcomingAnnualCharges } = await import("../../lib/annual-charges"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM account_links");
|
||||
stmtCache = new Map();
|
||||
({ ownerId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("what it finds", () => {
|
||||
it("confirms a card annual fee that repeated a year later", async () => {
|
||||
await charge("American Express", "1000", "2025-06-10", "ANNUAL CARD FEE", 875);
|
||||
await charge("American Express", "1000", "2026-06-10", "ANNUAL CARD FEE", 1750);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CARD FEE");
|
||||
expect(row?.confidence).toBe("confirmed");
|
||||
expect(row?.observations).toBe(2);
|
||||
// The amount is NOT averaged: this fee doubled, and the next one to plan for
|
||||
// is the latest, not the mean of a rise.
|
||||
expect(Number(row?.last_amount)).toBe(1750);
|
||||
expect(row?.next_due_estimate.startsWith("2027-06")).toBe(true);
|
||||
});
|
||||
|
||||
it("accepts a fee seen only once when the bank names it annual", async () => {
|
||||
await charge("Westpac", "8032", "2025-10-21", "CARD FEE", 295);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "CARD FEE");
|
||||
// One charge cannot show a cadence, but the kind is not in doubt.
|
||||
expect(row?.confidence).toBe("likely");
|
||||
expect(row?.next_due_estimate).toBe("2026-10-21");
|
||||
});
|
||||
|
||||
it("finds an annual merchant charge with no fee wording, if the amount is stable", async () => {
|
||||
await purchase("2025-02-03", "AMAZON PRIME", 79);
|
||||
await purchase("2026-02-01", "AMAZON PRIME", 79);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "AMAZON PRIME");
|
||||
expect(row?.confidence).toBe("possible");
|
||||
});
|
||||
});
|
||||
|
||||
describe("what it must NOT report", () => {
|
||||
it("ignores a monthly account fee", async () => {
|
||||
for (const d of ["2026-01-31", "2026-02-28", "2026-03-31", "2026-04-30"]) {
|
||||
await charge("ANZ", "5626", d, "ACCOUNT SERVICING FEE", 5);
|
||||
}
|
||||
expect(find(await getUpcomingAnnualCharges(ownerId), "ACCOUNT SERVICING FEE")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores per-transaction fees, which never repeat under one label", async () => {
|
||||
await charge("Wise Australia Pty Ltd.", "9001", "2026-05-20", "WISE CHARGES FOR: TRANSFER-2143509822", 30.67);
|
||||
await charge("Wise Australia Pty Ltd.", "9001", "2026-06-04", "WISE CHARGES FOR: TRANSFER-2171403032", 19.51);
|
||||
const rows = await getUpcomingAnnualCharges(ownerId);
|
||||
expect(rows.filter((r) => r.label.startsWith("WISE CHARGES"))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("ignores a merchant visited twice a year apart for wildly different amounts", async () => {
|
||||
// The exact shape that defeats a pure cadence test: Target, $30 then $198.
|
||||
await purchase("2025-07-21", "TARGET", 30);
|
||||
await purchase("2026-07-11", "TARGET", 198);
|
||||
expect(find(await getUpcomingAnnualCharges(ownerId), "TARGET")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores a merchant seen only once", async () => {
|
||||
await purchase("2026-04-03", "UBER ONE MEMBERSHIP", 96);
|
||||
expect(find(await getUpcomingAnnualCharges(ownerId), "UBER ONE MEMBERSHIP")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("account identity and honesty about gaps", () => {
|
||||
it("treats a renamed account as one continuous fee series", async () => {
|
||||
// Citi Prestige -> NAB MyCard: same card, two bank names, two numbers.
|
||||
await charge("Citibank", "5258073330340253", "2024-12-13", "Annual Fee", 700);
|
||||
await charge("Citibank", "5258073330340253", "2025-11-13", "Annual Fee", 700);
|
||||
await pool.query(
|
||||
`INSERT INTO account_links (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason)
|
||||
VALUES ('Citibank','5258073330340253','NAB','1015001000116227','2025-11-14','issuer_change')`
|
||||
);
|
||||
// A statement under the NEW identity, so the link has somewhere to resolve to.
|
||||
await charge("NAB", "1015001000116227", "2026-08-13", "Purchase", 29.99, "debit");
|
||||
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL FEE");
|
||||
expect(row?.observations).toBe(2);
|
||||
expect(row?.confidence).toBe("confirmed");
|
||||
// Names the account's CURRENT bank -- the next fee comes from there, and
|
||||
// sending someone to the retired brand is the bug this link exists to stop.
|
||||
expect(row?.bank_name).toBe("NAB");
|
||||
});
|
||||
|
||||
it("marks a passed estimate as expected_not_seen rather than upcoming", async () => {
|
||||
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
|
||||
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
|
||||
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CHARGE");
|
||||
expect(row?.status).toBe("expected_not_seen");
|
||||
// And it reports how stale the account is, which is what separates
|
||||
// "it stopped being charged" from "we stopped importing the statements".
|
||||
expect(row?.account_days_stale).toBeGreaterThan(300);
|
||||
});
|
||||
|
||||
it("sorts upcoming ahead of expected_not_seen", async () => {
|
||||
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
|
||||
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
|
||||
await charge("Westpac", "8032", "2026-08-01", "CARD FEE", 295);
|
||||
const rows = await getUpcomingAnnualCharges(ownerId);
|
||||
const firstMissing = rows.findIndex((r) => r.status === "expected_not_seen");
|
||||
const lastUpcoming = rows.map((r) => r.status).lastIndexOf("upcoming");
|
||||
expect(lastUpcoming).toBeLessThan(firstMissing);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,271 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||
|
||||
/**
|
||||
* The automatic pass over the reconcile queue.
|
||||
*
|
||||
* This exists because the wallet lane writes a transaction for every tap
|
||||
* (DECISIONS.md ING-11), adding ~63 rows a month to a queue already ~177 deep
|
||||
* and worked by hand. Hiding those rows instead is explicitly ruled out --
|
||||
* see the awaitsStatementLine() removal note in queries.ts.
|
||||
*
|
||||
* The cases that must NOT match matter more than the ones that must: this runs
|
||||
* unattended, and a wrong pairing merges one purchase's curation onto another
|
||||
* purchase's row. So most of what follows is about refusal.
|
||||
*/
|
||||
|
||||
const MACHINE_TOKEN = "test-wallet-token";
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let autoReconcile: typeof import("../../app/api/transactions/reconcile/route").PATCH;
|
||||
|
||||
/** Browser-session caller: no machine token, auth comes from the mocked session. */
|
||||
const userReq = (url = "http://x/api/transactions/reconcile") =>
|
||||
({ url, headers: { get: () => null } }) as never;
|
||||
|
||||
/** Scheduled caller: machine token, no session, sweeps every owner. */
|
||||
const machineReq = (url = "http://x/api/transactions/reconcile") =>
|
||||
({ url, headers: { get: (h: string) => (h === "x-ingest-token" ? MACHINE_TOKEN : null) } }) as never;
|
||||
|
||||
async function statementLine(
|
||||
amount: number,
|
||||
date: string,
|
||||
description: string,
|
||||
opts: { owner?: number; type?: string; fxAmount?: number; fxCode?: string } = {}
|
||||
) {
|
||||
const owner = opts.owner ?? ownerId;
|
||||
const st = await pool.query(
|
||||
// Account number varies by owner: uq_statement_identity is
|
||||
// (bank_name, account_number, billing_end_date), so two owners sharing one
|
||||
// account collide rather than modelling two real cards.
|
||||
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||
[date, owner, `1234-5678-9012-${String(owner).padStart(4, "0")}`]
|
||||
);
|
||||
const t = await pool.query(
|
||||
`INSERT INTO transactions
|
||||
(statement_id, transaction_date, description, amount, transaction_type, owner_id,
|
||||
foreign_currency_amount, foreign_currency_code)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id`,
|
||||
[st.rows[0].id, date, description, amount, opts.type ?? "debit", owner,
|
||||
opts.fxAmount ?? null, opts.fxCode ?? null]
|
||||
);
|
||||
return t.rows[0].id as number;
|
||||
}
|
||||
|
||||
async function walletRow(
|
||||
amount: number,
|
||||
date: string,
|
||||
opts: { fxAmount?: number; fxCode?: string; paymentMethod?: string } = {}
|
||||
) {
|
||||
const id = await insertTransaction(pool, ownerId, {
|
||||
amount, transaction_date: date, description: "Wallet tap",
|
||||
});
|
||||
await pool.query(
|
||||
`UPDATE transactions
|
||||
SET foreign_currency_amount = $2, foreign_currency_code = $3, payment_method = $4
|
||||
WHERE id = $1`,
|
||||
[id, opts.fxAmount ?? null, opts.fxCode ?? null, opts.paymentMethod ?? null]
|
||||
);
|
||||
return id;
|
||||
}
|
||||
|
||||
const reconciledWith = async (id: number) =>
|
||||
(await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [id]))
|
||||
.rows[0].reconciled_with_id as number | null;
|
||||
|
||||
beforeAll(async () => {
|
||||
process.env.WALLET_INGEST_TOKEN = MACHINE_TOKEN;
|
||||
pool = createPool();
|
||||
vi.doMock("@/lib/auth", () => ({
|
||||
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||
}));
|
||||
({ PATCH: autoReconcile } = await import("../../app/api/transactions/reconcile/route"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM expense_metadata");
|
||||
({ ownerId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("pairs it should apply", () => {
|
||||
it("matches an exact amount two days apart and moves the link", async () => {
|
||||
const manual = await walletRow(13.5, "2026-08-13");
|
||||
const stmt = await statementLine(13.5, "2026-08-15", "BREADWORLD BAKERY");
|
||||
|
||||
const body = await (await autoReconcile(userReq())).json();
|
||||
expect(body.reconciled).toBe(1);
|
||||
expect(await reconciledWith(manual)).toBe(stmt);
|
||||
});
|
||||
|
||||
it("matches a foreign row on the NATIVE pair, not on the AUD amount", async () => {
|
||||
// The bank converts at its own rate, so the wallet row cannot predict the
|
||||
// AUD figure. It can only know THB 500. Deliberately give the statement a
|
||||
// wildly different AUD amount to prove `amount` is not what matched.
|
||||
const manual = await walletRow(999.99, "2026-08-13", { fxAmount: 500, fxCode: "THB" });
|
||||
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
|
||||
fxAmount: 500, fxCode: "THB",
|
||||
});
|
||||
|
||||
const body = await (await autoReconcile(userReq())).json();
|
||||
expect(body.reconciled).toBe(1);
|
||||
expect(await reconciledWith(manual)).toBe(stmt);
|
||||
});
|
||||
|
||||
it("carries a SETTLED split across without resurrecting the debt", async () => {
|
||||
const manual = await walletRow(50, "2026-08-13");
|
||||
const stmt = await statementLine(50, "2026-08-14", "SHARED DINNER");
|
||||
await pool.query(
|
||||
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled, settled_at)
|
||||
VALUES ($1, $2, 50, true, now())`,
|
||||
[manual, ownerId]
|
||||
);
|
||||
|
||||
expect((await autoReconcile(userReq())).status ?? 200).toBe(200);
|
||||
|
||||
const moved = await pool.query(
|
||||
`SELECT settled, settled_at FROM transaction_splits WHERE transaction_id = $1`, [stmt]
|
||||
);
|
||||
expect(moved.rowCount).toBe(1);
|
||||
// The whole point: a delete-and-recreate defaults `settled` to false, which
|
||||
// silently marks a paid debt owed again. Nothing errors when it regresses.
|
||||
expect(moved.rows[0].settled).toBe(true);
|
||||
expect(moved.rows[0].settled_at).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("pairs it must refuse", () => {
|
||||
it("refuses two identical amounts in the same window (ambiguous both ways)", async () => {
|
||||
// Two $13.50 bakery taps in one week is the ordinary case, not an edge one.
|
||||
const a = await walletRow(13.5, "2026-08-13");
|
||||
const b = await walletRow(13.5, "2026-08-14");
|
||||
await statementLine(13.5, "2026-08-14", "BREADWORLD ONE");
|
||||
await statementLine(13.5, "2026-08-15", "BREADWORLD TWO");
|
||||
|
||||
const body = await (await autoReconcile(userReq())).json();
|
||||
expect(body.reconciled).toBe(0);
|
||||
expect(await reconciledWith(a)).toBeNull();
|
||||
expect(await reconciledWith(b)).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses a near-miss amount — a person should look at that", async () => {
|
||||
const manual = await walletRow(13.5, "2026-08-13");
|
||||
await statementLine(13.55, "2026-08-14", "BREADWORLD BAKERY");
|
||||
|
||||
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||
expect(await reconciledWith(manual)).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses a match outside the day window", async () => {
|
||||
const manual = await walletRow(13.5, "2026-08-13");
|
||||
await statementLine(13.5, "2026-08-25", "BREADWORLD BAKERY");
|
||||
|
||||
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||
expect(await reconciledWith(manual)).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses a statement row already claimed by another manual row", async () => {
|
||||
const claimed = await statementLine(40, "2026-08-14", "ALREADY TAKEN");
|
||||
const first = await walletRow(40, "2026-08-13");
|
||||
await pool.query(`UPDATE transactions SET reconciled_with_id = $2 WHERE id = $1`, [first, claimed]);
|
||||
const second = await walletRow(40, "2026-08-14");
|
||||
|
||||
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||
expect(await reconciledWith(second)).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses a credits-funded row — no card leg is ever coming", async () => {
|
||||
const manual = await walletRow(30, "2026-08-13", { paymentMethod: "credits" });
|
||||
await statementLine(30, "2026-08-14", "COINCIDENTAL AMOUNT");
|
||||
|
||||
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||
expect(await reconciledWith(manual)).toBeNull();
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
describe("an AUD row against a foreign statement line", () => {
|
||||
it("DOES match on the AUD figure — this is what every real pair looks like", async () => {
|
||||
// Checked against live data: all 48 reconciliations a human has made are
|
||||
// this exact shape -- manual row with no currency, statement row carrying
|
||||
// EUR or CHF, paired on exact AUD. Overseas receipts and orders record what
|
||||
// was charged in AUD; only the wallet lane records natively. An earlier
|
||||
// version of this rule required both sides to be non-foreign, on the
|
||||
// reasoning that a currency mismatch is ambiguous. It matched 0 of the 48.
|
||||
const manual = await walletRow(22.85, "2026-08-13");
|
||||
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
|
||||
fxAmount: 500, fxCode: "THB",
|
||||
});
|
||||
|
||||
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(1);
|
||||
expect(await reconciledWith(manual)).toBe(stmt);
|
||||
});
|
||||
|
||||
it("still refuses when a second candidate makes it ambiguous", async () => {
|
||||
// The coincidence this guards against -- a domestic tap whose amount equals
|
||||
// a foreign charge's AUD figure -- is caught by unique-both-ways, not by a
|
||||
// currency test.
|
||||
const manual = await walletRow(22.85, "2026-08-13");
|
||||
await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", { fxAmount: 500, fxCode: "THB" });
|
||||
await statementLine(22.85, "2026-08-13", "MELBOURNE CAFE");
|
||||
|
||||
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||
expect(await reconciledWith(manual)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("dry run", () => {
|
||||
it("reports what it would do and writes nothing", async () => {
|
||||
const manual = await walletRow(13.5, "2026-08-13");
|
||||
await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
|
||||
|
||||
const body = await (await autoReconcile(userReq("http://x/api/transactions/reconcile?dry=1"))).json();
|
||||
expect(body.dry_run).toBe(true);
|
||||
expect(body.would_reconcile).toBe(1);
|
||||
expect(await reconciledWith(manual)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("the scheduled sweep", () => {
|
||||
it("runs on the machine token with no session at all", async () => {
|
||||
const manual = await walletRow(13.5, "2026-08-13");
|
||||
const stmt = await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
|
||||
|
||||
const body = await (await autoReconcile(machineReq())).json();
|
||||
expect(body.reconciled).toBe(1);
|
||||
expect(await reconciledWith(manual)).toBe(stmt);
|
||||
});
|
||||
|
||||
it("covers EVERY owner, not just one", async () => {
|
||||
// 169 of the 177 rows outstanding belong to the second participant, so a
|
||||
// sweep scoped to a single user would leave almost all of the queue alone.
|
||||
const other = (await pool.query(
|
||||
`INSERT INTO participants (name) VALUES ('Carol') RETURNING id`
|
||||
)).rows[0].id as number;
|
||||
|
||||
const mine = await walletRow(11, "2026-08-13");
|
||||
const theirTx = (await pool.query(
|
||||
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description,
|
||||
amount, transaction_type, category, row_index)
|
||||
VALUES ($1, NULL, '2026-08-13', 'Theirs', 22, 'debit', 'other', 0) RETURNING id`, [other]
|
||||
)).rows[0].id as number;
|
||||
|
||||
await statementLine(11, "2026-08-14", "MINE");
|
||||
await statementLine(22, "2026-08-14", "THEIRS", { owner: other });
|
||||
|
||||
const body = await (await autoReconcile(machineReq())).json();
|
||||
expect(body.reconciled).toBe(2);
|
||||
expect(await reconciledWith(mine)).not.toBeNull();
|
||||
expect(await reconciledWith(theirTx)).not.toBeNull();
|
||||
});
|
||||
|
||||
it("rejects a wrong machine token and falls through to session auth", async () => {
|
||||
const bad = ({ url: "http://x/", headers: { get: () => "not-the-token" } }) as never;
|
||||
// Session is mocked, so this still succeeds -- but as the session user only,
|
||||
// proving the bad token did not grant the all-owner sweep.
|
||||
const body = await (await autoReconcile(bad)).json();
|
||||
expect(body.reconciled).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -84,6 +84,11 @@ export async function resetDB(pool: Pool) {
|
||||
await assertTestDatabase(pool);
|
||||
await pool.query(`
|
||||
TRUNCATE
|
||||
-- The audit triggers (migration 0037) fire on every INSERT the suite makes.
|
||||
-- TRUNCATE does not fire row-level triggers, so nothing here is logged --
|
||||
-- but without this line the log is never cleared either, and the table
|
||||
-- grows by roughly a thousand rows per full run, forever.
|
||||
transaction_audit,
|
||||
split_payments,
|
||||
transaction_splits,
|
||||
transaction_tags,
|
||||
|
||||
@@ -105,6 +105,17 @@ async function seedOrder(over: { merchant: string; date: string; amount: number;
|
||||
}
|
||||
return id;
|
||||
}
|
||||
/** A row shaped the way the wallet lane writes one: statementless, no metadata
|
||||
* at all, no platform, and named by the notification rather than the receipt. */
|
||||
async function seedWalletRow(over: { merchant: string; date: string; amount: number; owner?: number }) {
|
||||
const t = await pool.query(
|
||||
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description, amount, amount_aud, transaction_type, category, merchant_name, merchant_normalized, payment_method, location, source, source_ref, row_index)
|
||||
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', $3, $3, 'card', 'Werribee, VIC', 'wallet', $5, 0) RETURNING id`,
|
||||
[over.owner ?? ownerId, over.date, over.merchant, over.amount, `evt:${Math.floor(over.amount * 100)}`]
|
||||
);
|
||||
return t.rows[0].id as number;
|
||||
}
|
||||
|
||||
/** What `components/order-details.tsx` would actually render for a transaction. */
|
||||
const orderPanel = async (txnId: number) =>
|
||||
(await (await orderGET({ headers: { get: () => null } } as unknown as NextRequest, { params: Promise.resolve({ id: String(txnId) }) })).json()) as { line_items?: unknown[] } | null;
|
||||
@@ -339,6 +350,64 @@ describe("POST /api/intake — gates", () => {
|
||||
expect((await batch(id)).status).toBe("unsupported");
|
||||
expect(calls).toHaveLength(0);
|
||||
});
|
||||
|
||||
// 2026-08-22, live batch 3d2ffbe8…: an iOS Shortcut coerced the shared photo
|
||||
// to text, so the whole upload was the five characters `Image`. The model was
|
||||
// asked anyway and returned Burgertory (Braybrook), two "The Pride" lines and
|
||||
// a $41.15 total — none of which existed. Only the absent date kept it out of
|
||||
// the matcher, and the invented header was stored on the batch as
|
||||
// `needs_header`. The model must never be asked.
|
||||
it("a text part that cannot be a receipt is declined BEFORE any model call", async () => {
|
||||
const id = uuid();
|
||||
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)", order_date: null, total: 41.15 };
|
||||
const res = await intakePOST(
|
||||
multipart({ source: "other", batch_id: id }, [{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") }])
|
||||
);
|
||||
const body = await res.json();
|
||||
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
|
||||
expect(body.text).toMatch(/single word/);
|
||||
expect(body.text).toMatch(/"Image"/);
|
||||
// The model was never asked — the only thing that makes confabulation impossible.
|
||||
expect(calls).toHaveLength(0);
|
||||
const row = await batch(id);
|
||||
expect(row.status).toBe("unsupported");
|
||||
expect(row.kind).toBe("unsupported");
|
||||
expect(row.header).toBeNull();
|
||||
expect(row.items).toBeNull();
|
||||
expect(row.transaction_id).toBeNull();
|
||||
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
|
||||
});
|
||||
|
||||
it("a junk text part beside a real receipt does not sink the batch", async () => {
|
||||
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||
const body = await (
|
||||
await intakePOST(
|
||||
multipart({ source: "uber_eats" }, [
|
||||
{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") },
|
||||
{ name: "Receipt_19Aug2026.pdf", type: "application/pdf", bytes: BURGERTORY },
|
||||
])
|
||||
)
|
||||
).json();
|
||||
expect(body.status).toBe("enriched");
|
||||
expect(await lineItemsOf(txn)).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("an order extracted from an unreadable PDF is unsupported, not needs_header", async () => {
|
||||
const id = uuid();
|
||||
// A PDF that renders to a scrap of junk still reaches the model — the
|
||||
// pre-check only inspects text parts. `confabulationReason` is the belt.
|
||||
vi.spyOn(await import("../../lib/intake-extract"), "extractPdfText").mockResolvedValueOnce("Image");
|
||||
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)" };
|
||||
const body = await (
|
||||
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [{ name: "x.pdf", type: "application/pdf", bytes: BURGERTORY }]))
|
||||
).json();
|
||||
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
|
||||
expect(body.text).toMatch(/nothing shared could be read/);
|
||||
const row = await batch(id);
|
||||
expect(row.status).toBe("unsupported");
|
||||
expect(row.header).toBeNull();
|
||||
expect(row.items).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("POST /api/intake — food (enrich vs candidates)", () => {
|
||||
@@ -483,7 +552,13 @@ describe("POST /api/intake — food (enrich vs candidates)", () => {
|
||||
const id = uuid();
|
||||
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
|
||||
expect(body.status).toBe("needs_header");
|
||||
expect(body.text).toMatch(/share the top of the screen/);
|
||||
// Asserts BOTH halves of needsHeaderCard(): that it names what is actually
|
||||
// missing (total, not a blanket "total or date"), and that a uber_eats
|
||||
// source gets the share-more-of-the-screen advice rather than the in-store
|
||||
// Share-button advice. 258fe3b changed this text and updated the unit tests
|
||||
// but not this one.
|
||||
expect(body.text).toMatch(/I can see items but no total/);
|
||||
expect(body.text).toMatch(/share the part of the screen that shows it/);
|
||||
expect(await lineItemsOf(txn)).toEqual([]);
|
||||
expect((await batch(id)).status).toBe("needs_header");
|
||||
});
|
||||
@@ -574,6 +649,33 @@ describe("POST /api/intake — grocery → pantry", () => {
|
||||
expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" });
|
||||
});
|
||||
|
||||
it("pantry saying the shop is already on its board is SHOWN, not swallowed", async () => {
|
||||
// Pantry accepts a second capture of one receipt on purpose (the first may
|
||||
// have been a bad crop) and reports it rather than refusing. Dropping that
|
||||
// field turned "reported" into silence — and a duplicate nobody sees is a
|
||||
// duplicate that gets confirmed, which counts the same groceries twice
|
||||
// because pantry's addItem accumulates. Seen live on captures 66/69.
|
||||
modelReply = coles;
|
||||
pantryReply = { status: 200, body: { captureId: 902, items: [1, 2, 3], alreadyCaptured: { captureId: 66, confirmed: true } } };
|
||||
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
|
||||
expect(body.status).toBe("forwarded_to_pantry");
|
||||
expect(body.text).toMatch(/Already on the pantry board/);
|
||||
expect(body.text).toMatch(/capture 66 \(already confirmed\)/);
|
||||
expect(body.text).toMatch(/capture 902/);
|
||||
});
|
||||
|
||||
it("two ledger rows for one shop are reported, not treated as no match", async () => {
|
||||
// Ambiguity is not a miss. This is the exact state that produced the
|
||||
// 2026-08-18 Woolworths double-book: the shop was in the ledger twice, the
|
||||
// matcher returned nothing, and the card said finance would book it.
|
||||
modelReply = coles;
|
||||
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
|
||||
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
|
||||
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
|
||||
expect(body.status).toBe("forwarded_to_pantry");
|
||||
expect(body.text).toMatch(/found more than one row it could be/);
|
||||
});
|
||||
|
||||
it("pantry 409 → duplicate", async () => {
|
||||
modelReply = coles;
|
||||
pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } };
|
||||
@@ -647,6 +749,37 @@ describe("Slack taps (C5)", () => {
|
||||
expect(lastSlackUpdate().text).toMatch(/^Added 2 items/);
|
||||
});
|
||||
|
||||
it("intake_create adopts the wallet row for the same tap instead of booking it twice", async () => {
|
||||
// The normal arrival order: the notification lands within seconds of the
|
||||
// tap, the screenshot is shared later. matchTransaction cannot see that row
|
||||
// — its key includes the platform and a notification has none — so before
|
||||
// this, Create booked the purchase a second time.
|
||||
const wallet = await seedWalletRow({ merchant: "Burgertory", date: "2026-08-19", amount: 22.4 });
|
||||
const id = uuid();
|
||||
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
|
||||
expect(body.status).toBe("candidates");
|
||||
|
||||
await slackPOST(slackPress("intake_create", { batch_id: id }));
|
||||
|
||||
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: wallet });
|
||||
expect((await lineItemsOf(wallet)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
|
||||
// The suburb the notification supplied is the one thing nothing else knows.
|
||||
expect((await pool.query(`SELECT location FROM transactions WHERE id = $1`, [wallet])).rows[0].location).toBe("Werribee, VIC");
|
||||
});
|
||||
|
||||
it("intake_create still creates when a same-amount row already carries a receipt", async () => {
|
||||
// A row that another source fully described is a DIFFERENT purchase that
|
||||
// happens to share an amount — adopting it would move one order's items
|
||||
// onto another order's row.
|
||||
const other = await seedOrder({ merchant: "Someone Else", date: "2026-08-19", amount: 22.4, platform: null });
|
||||
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [other, JSON.stringify([{ description: "theirs", qty: 1, amount: 22.4 }])]);
|
||||
const id = uuid();
|
||||
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
|
||||
await slackPOST(slackPress("intake_create", { batch_id: id }));
|
||||
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(2);
|
||||
});
|
||||
|
||||
it("two pending cards for one order book it ONCE", async () => {
|
||||
const a = uuid();
|
||||
const b = uuid();
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||
|
||||
/**
|
||||
* Manually resolving a duplicate the automation cannot see.
|
||||
*
|
||||
* Until this route existed there was no way to merge or remove a transaction at
|
||||
* all: `superseded_by_id` was read everywhere and written nowhere in
|
||||
* application code, having been set exactly once by migration 0023. The only
|
||||
* merge path was the reconcile queue, whose candidate list is bounded by an
|
||||
* amount band -- so SP PEAKDESIGN (notified $42.44, charged $43.71, the card's
|
||||
* 3% foreign transaction fee) had no offerable twin.
|
||||
*
|
||||
* The refusals matter more than the happy paths, exactly as in
|
||||
* auto-reconcile.test.ts: this writes to columns that hide a row from every
|
||||
* figure, so a wrong call does not error, it quietly stops counting money.
|
||||
*/
|
||||
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let otherId: number;
|
||||
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
|
||||
|
||||
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
|
||||
const call = (id: number, body: unknown) => merge(req(body), { params: Promise.resolve({ id: String(id) }) });
|
||||
|
||||
// uq_statement_identity is (bank_name, account_number, billing_end_date), so
|
||||
// every statement in a test needs its own account or the second insert collides.
|
||||
let acctSeq = 0;
|
||||
|
||||
async function statementLine(amount: number, date: string, description: string) {
|
||||
const st = await pool.query(
|
||||
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||
[date, ownerId, `4444-0000-0000-${String(1000 + acctSeq++)}`]
|
||||
);
|
||||
const t = await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
|
||||
[st.rows[0].id, date, description, amount, ownerId]
|
||||
);
|
||||
return t.rows[0].id as number;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
pool = createPool();
|
||||
vi.doMock("@/lib/auth", () => ({
|
||||
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||
}));
|
||||
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM expense_metadata");
|
||||
({ ownerId, otherId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("reconcile mode", () => {
|
||||
it("folds a wallet row into its statement twin even though the amounts differ by the FX fee", async () => {
|
||||
const wallet = await insertTransaction(pool, ownerId, {
|
||||
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
|
||||
});
|
||||
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
|
||||
|
||||
const res = await call(wallet, { other_id: stmt, mode: "reconcile" });
|
||||
expect(res.status ?? 200).toBe(200);
|
||||
|
||||
const r = await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [wallet]);
|
||||
expect(r.rows[0].reconciled_with_id).toBe(stmt);
|
||||
});
|
||||
|
||||
it("refuses when the target is not a statement row", async () => {
|
||||
const a = await insertTransaction(pool, ownerId, { amount: 10 });
|
||||
const b = await insertTransaction(pool, ownerId, { amount: 10 });
|
||||
const res = await call(a, { other_id: b, mode: "reconcile" });
|
||||
expect(res.status).toBe(400);
|
||||
});
|
||||
|
||||
it("refuses when the source is itself a statement row", async () => {
|
||||
const s1 = await statementLine(10, "2026-08-01", "A");
|
||||
const s2 = await statementLine(10, "2026-08-02", "B");
|
||||
const res = await call(s1, { other_id: s2, mode: "reconcile" });
|
||||
expect(res.status).toBe(400);
|
||||
});
|
||||
});
|
||||
|
||||
describe("supersede mode", () => {
|
||||
it("carries splits onto the survivor before hiding the loser", async () => {
|
||||
// The whole reason supersede moves curation: `superseded_by_id` removes the
|
||||
// row from every figure, so a split left behind stops counting silently and
|
||||
// a settled debt reappears as owed.
|
||||
const loser = await insertTransaction(pool, ownerId, { amount: 80, description: "DUP" });
|
||||
const survivor = await insertTransaction(pool, ownerId, { amount: 80, description: "KEEP" });
|
||||
await pool.query(
|
||||
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
|
||||
VALUES ($1,$2,50,true), ($1,$3,50,false)`,
|
||||
[loser, ownerId, otherId]
|
||||
);
|
||||
|
||||
const res = await call(loser, { other_id: survivor, mode: "supersede" });
|
||||
expect(res.status ?? 200).toBe(200);
|
||||
|
||||
const onLoser = await pool.query(`SELECT count(*)::int c FROM transaction_splits WHERE transaction_id=$1`, [loser]);
|
||||
const onSurvivor = await pool.query(
|
||||
`SELECT participant_id, share_percent, settled FROM transaction_splits
|
||||
WHERE transaction_id=$1 ORDER BY participant_id`, [survivor]
|
||||
);
|
||||
expect(onLoser.rows[0].c).toBe(0);
|
||||
expect(onSurvivor.rows).toHaveLength(2);
|
||||
// settled must survive the move -- a delete-and-recreate defaults it to false.
|
||||
expect(onSurvivor.rows.map((r) => r.settled)).toContain(true);
|
||||
|
||||
const marked = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [loser]);
|
||||
expect(marked.rows[0].superseded_by_id).toBe(survivor);
|
||||
});
|
||||
|
||||
it("works between two statement rows, which reconcile cannot express", async () => {
|
||||
const s1 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
|
||||
const s2 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
|
||||
const res = await call(s1, { other_id: s2, mode: "supersede" });
|
||||
expect(res.status ?? 200).toBe(200);
|
||||
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [s1]);
|
||||
expect(r.rows[0].superseded_by_id).toBe(s2);
|
||||
});
|
||||
|
||||
it("refuses to hide a statement row behind a statementless one", async () => {
|
||||
// The wrong direction, and silently wrong: a wallet row holds the amount
|
||||
// BEFORE the card's foreign transaction fee, so burying the statement line
|
||||
// under it leaves the ledger holding $42.44 for a $43.71 charge.
|
||||
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
|
||||
const wallet = await insertTransaction(pool, ownerId, {
|
||||
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
|
||||
});
|
||||
const res = await call(stmt, { other_id: wallet, mode: "supersede" });
|
||||
expect(res.status).toBe(400);
|
||||
// and the statement row is untouched
|
||||
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [stmt]);
|
||||
expect(r.rows[0].superseded_by_id).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses to bury a row under one that is already superseded", async () => {
|
||||
const a = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||
const b = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||
const c = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||
await call(b, { other_id: c, mode: "supersede" }); // b -> c
|
||||
const res = await call(a, { other_id: b, mode: "supersede" }); // a -> b, but b is buried
|
||||
expect(res.status).toBe(409);
|
||||
});
|
||||
});
|
||||
|
||||
describe("refusals common to both modes", () => {
|
||||
it("refuses a transaction merging into itself", async () => {
|
||||
const a = await insertTransaction(pool, ownerId, { amount: 1 });
|
||||
const res = await call(a, { other_id: a, mode: "supersede" });
|
||||
expect(res.status).toBe(400);
|
||||
});
|
||||
|
||||
it("refuses a row that is already reconciled", async () => {
|
||||
const wallet = await insertTransaction(pool, ownerId, { amount: 12, transaction_date: "2026-08-10" });
|
||||
const stmt = await statementLine(12, "2026-08-10", "X");
|
||||
await call(wallet, { other_id: stmt, mode: "reconcile" });
|
||||
const other = await insertTransaction(pool, ownerId, { amount: 12 });
|
||||
const res = await call(wallet, { other_id: other, mode: "supersede" });
|
||||
expect(res.status).toBe(409);
|
||||
});
|
||||
|
||||
it("refuses a transaction belonging to someone else", async () => {
|
||||
const mine = await insertTransaction(pool, ownerId, { amount: 7 });
|
||||
const theirs = await insertTransaction(pool, otherId, { amount: 7 });
|
||||
const res = await call(mine, { other_id: theirs, mode: "supersede" });
|
||||
expect(res.status).toBe(404);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,153 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants } from "./helpers";
|
||||
|
||||
/**
|
||||
* A receipt arriving after the wallet notification must ENRICH that row, not
|
||||
* create a second one.
|
||||
*
|
||||
* The case this is built from is real. Woolworths $103.30 on 2026-08-18: the
|
||||
* wallet event landed at 09:00:14 on the 18th, and nothing created a
|
||||
* transaction until 20:25:06 on the 19th -- 35h25m later -- at which point the
|
||||
* receipt path made its own row from scratch. One shop, two rows, and the
|
||||
* notification's contribution (the instrument, and the suburb) thrown away.
|
||||
*
|
||||
* Sources for one purchase arrive in sequence, not in competition, so the rule
|
||||
* is: the receipt read the register tape and wins on what was bought; anything
|
||||
* only the notification knew survives untouched.
|
||||
*/
|
||||
|
||||
const TOKEN = "test-receipt-token";
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let ingest: typeof import("../../app/api/receipts/ingest/route").POST;
|
||||
|
||||
const ingestReq = (body: unknown) =>
|
||||
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
|
||||
|
||||
/** A row shaped the way the wallet lane writes one: statementless, no line
|
||||
* items, carrying the suburb the notification supplied. */
|
||||
async function walletRow(amount: number, date: string, merchant: string, location: string) {
|
||||
const t = await pool.query(
|
||||
`INSERT INTO transactions
|
||||
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
|
||||
transaction_type, category, payment_method, merchant_name, merchant_normalized,
|
||||
location, source, source_ref, row_index)
|
||||
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', 'card', $3, $3, $5, 'wallet', $6, 0)
|
||||
RETURNING id`,
|
||||
[ownerId, date, merchant, amount, location, `evt-${amount}-${date}`]
|
||||
);
|
||||
const id = t.rows[0].id as number;
|
||||
await pool.query(
|
||||
`INSERT INTO expense_metadata
|
||||
(transaction_id, source, merchant_normalized, amount, transaction_date,
|
||||
currency, line_items, payment_method_detail, flags)
|
||||
VALUES ($1, 'wallet', $2, $3, $4, 'AUD', '[]'::jsonb, 'Visa', '[]'::jsonb)`,
|
||||
[id, merchant, amount, date]
|
||||
);
|
||||
return id;
|
||||
}
|
||||
|
||||
const receipt = (amount: number, date: string) => ({
|
||||
receipt_uid: `woolworths:3345:${amount}:2026-08-18`,
|
||||
capture_event_id: 900,
|
||||
merchant_name: "Woolworths",
|
||||
store_detail: "Wyndham Vale",
|
||||
transaction_date: date,
|
||||
total: amount,
|
||||
tender_legs: [
|
||||
{ leg_index: 1, amount, card_last4: "6078", card_product: "VISA PREPAID", class: "gift_card" },
|
||||
],
|
||||
line_items: [
|
||||
{ description: "Milk 2L", qty: 1, unit: "ea", amount: 3.5, category: "dairy" },
|
||||
{ description: "Bread", qty: 1, unit: "ea", amount: 4.0, category: "bakery" },
|
||||
],
|
||||
});
|
||||
|
||||
beforeAll(async () => {
|
||||
process.env.RECEIPT_INGEST_TOKEN = TOKEN;
|
||||
pool = createPool();
|
||||
vi.doMock("@/lib/auth", () => ({
|
||||
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||
}));
|
||||
({ POST: ingest } = await import("../../app/api/receipts/ingest/route"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM expense_metadata");
|
||||
({ ownerId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
const txCount = async () =>
|
||||
Number((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n);
|
||||
|
||||
describe("a receipt meeting the wallet row for the same tap", () => {
|
||||
it("adopts it instead of creating a second transaction", async () => {
|
||||
const walletId = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
|
||||
expect(await txCount()).toBe(1);
|
||||
|
||||
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||
|
||||
expect(await txCount()).toBe(1);
|
||||
const meta = await pool.query(
|
||||
`SELECT transaction_id, jsonb_array_length(line_items) AS items, source,
|
||||
payment_method_detail, card_last4
|
||||
FROM expense_metadata`
|
||||
);
|
||||
expect(meta.rowCount).toBe(1);
|
||||
expect(meta.rows[0].transaction_id).toBe(walletId);
|
||||
expect(meta.rows[0].items).toBe(2);
|
||||
expect(meta.rows[0].source).toBe("pantry");
|
||||
// The receipt read the card off the tape; the notification only saw "Visa".
|
||||
expect(meta.rows[0].payment_method_detail).toBe("VISA PREPAID");
|
||||
expect(meta.rows[0].card_last4).toBe("6078");
|
||||
});
|
||||
|
||||
it("keeps the suburb only the notification knew", async () => {
|
||||
await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
|
||||
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||
|
||||
const t = await pool.query(`SELECT location, merchant_normalized, category FROM transactions`);
|
||||
// `location` is absent from the adoption UPDATE's column list on purpose:
|
||||
// the receipt has no suburb, so it must not be able to erase one.
|
||||
expect(t.rows[0].location).toBe("Wyndham Vale, VIC");
|
||||
// ...while what the receipt does know wins.
|
||||
expect(t.rows[0].merchant_normalized).toBe("Woolworths");
|
||||
expect(t.rows[0].category).toBe("groceries");
|
||||
});
|
||||
|
||||
it("creates its own row when there is no wallet row to adopt", async () => {
|
||||
await ingest(ingestReq(receipt(58.4, "2026-08-18")));
|
||||
expect(await txCount()).toBe(1);
|
||||
const meta = await pool.query(`SELECT source, jsonb_array_length(line_items) AS items FROM expense_metadata`);
|
||||
expect(meta.rows[0].source).toBe("pantry");
|
||||
expect(meta.rows[0].items).toBe(2);
|
||||
});
|
||||
|
||||
it("refuses to adopt when two candidates share the amount", async () => {
|
||||
// Two taps of the same value in the window: adopting either would move this
|
||||
// shop's line items onto a purchase it did not describe.
|
||||
await walletRow(103.3, "2026-08-17", "Woolworths", "Wyndham Vale, VIC");
|
||||
await walletRow(103.3, "2026-08-18", "Coles", "Manor Lakes, VIC");
|
||||
expect(await txCount()).toBe(2);
|
||||
|
||||
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||
|
||||
// Falls back to creating its own row rather than guessing.
|
||||
expect(await txCount()).toBe(3);
|
||||
});
|
||||
|
||||
it("refuses to adopt a row that already carries line items", async () => {
|
||||
const other = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
|
||||
await pool.query(
|
||||
`UPDATE expense_metadata SET line_items = '[{"description":"Something else","amount":1}]'::jsonb
|
||||
WHERE transaction_id = $1`, [other]
|
||||
);
|
||||
|
||||
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||
|
||||
// A fully-described row is a different purchase that happens to share an amount.
|
||||
expect(await txCount()).toBe(2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,86 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||
import { getPendingReconciliations } from "../../lib/queries";
|
||||
|
||||
/**
|
||||
* The reconcile queue must not offer what is already resolved.
|
||||
*
|
||||
* A superseded row is hidden from every figure (EXCLUDE_RECONCILED_SOURCE), so
|
||||
* listing it is not merely noise: on 2026-10-03 the queue showed the duplicate
|
||||
* wallet half 7037, superseded under its twin 7038, as a second "Digidirect"
|
||||
* with the same statement line offered as its match. Accepting it there would
|
||||
* have spent that line on a hidden row and left the real one unmatched forever.
|
||||
* findAutoMatches() always had both filters; this query had neither.
|
||||
*/
|
||||
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let acctSeq = 0;
|
||||
|
||||
async function statementLine(amount: number, date: string, description: string) {
|
||||
const st = await pool.query(
|
||||
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||
VALUES ('q.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
|
||||
);
|
||||
const t = await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
|
||||
[st.rows[0].id, date, description, amount, ownerId]
|
||||
);
|
||||
return t.rows[0].id as number;
|
||||
}
|
||||
|
||||
beforeAll(() => { pool = createPool(); });
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
({ ownerId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("getPendingReconciliations", () => {
|
||||
it("does not list a superseded manual row", async () => {
|
||||
const survivor = await insertTransaction(pool, ownerId, { description: "DIGIDIRECT", amount: 1180.05, transaction_date: "2026-08-25" });
|
||||
const hidden = await insertTransaction(pool, ownerId, { description: "Digidirect", amount: 1180.05, transaction_date: "2026-08-25" });
|
||||
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [survivor, hidden]);
|
||||
const line = await statementLine(1180.05, "2026-08-25", "SP DIGIDIRECT STRATHFIELD");
|
||||
|
||||
const queue = await getPendingReconciliations(ownerId);
|
||||
const ids = queue.map((r) => r.id);
|
||||
expect(ids).toContain(survivor);
|
||||
expect(ids).not.toContain(hidden);
|
||||
// The survivor is still offered its line.
|
||||
const row = queue.find((r) => r.id === survivor)!;
|
||||
expect(row.matches.map((m) => m.id)).toContain(line);
|
||||
});
|
||||
|
||||
it("does not offer a superseded statement row as a match", async () => {
|
||||
const manual = await insertTransaction(pool, ownerId, { description: "Grocer", amount: 42.0, transaction_date: "2026-08-10" });
|
||||
const live = await statementLine(42.0, "2026-08-10", "GROCER");
|
||||
const dupe = await statementLine(42.0, "2026-08-10", "GROCER");
|
||||
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [live, dupe]);
|
||||
|
||||
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
|
||||
const offered = row.matches.map((m) => m.id);
|
||||
expect(offered).toContain(live);
|
||||
expect(offered).not.toContain(dupe);
|
||||
});
|
||||
|
||||
it("offers the exact native twin of a foreign row even when its provisional AUD is far off", async () => {
|
||||
// Live shape, 2026-10-09: wallet INR 112 at a provisional A$2.01; CommBank
|
||||
// charged A$1.68 -- 16% under, outside the +-5% band.
|
||||
const manual = await insertTransaction(pool, ownerId, { description: "SWIGGY LIMITED", amount: 2.01, transaction_date: "2026-10-04" });
|
||||
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [manual]);
|
||||
const twin = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
|
||||
const otherRupees = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
|
||||
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [twin]);
|
||||
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 113 WHERE id = $1`, [otherRupees]);
|
||||
|
||||
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
|
||||
const offered = row.matches.map((m) => m.id);
|
||||
expect(offered).toContain(twin);
|
||||
// A different native figure at the same AUD is not admitted by this path.
|
||||
expect(offered).not.toContain(otherRupees);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,167 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||
import { EXCLUDE_RECONCILED_SOURCE } from "@/lib/analytics-sql";
|
||||
|
||||
/**
|
||||
* A charge that never posted (migration 0040).
|
||||
*
|
||||
* Txn 7951: a $1,040.41 Airbnb authorisation hold on Amex, booking cancelled
|
||||
* the next day, hold released. The wallet lane booked the notification as a
|
||||
* purchase and no statement line ever arrived, so it sat in the reconcile queue
|
||||
* forever and counted as spend everywhere. A void has no survivor, which is why
|
||||
* neither reconcile nor supersede could express it.
|
||||
*
|
||||
* As in merge-supersede.test.ts, the refusals matter as much as the happy path:
|
||||
* a void quietly stops money counting.
|
||||
*/
|
||||
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let otherId: number;
|
||||
let voidRoute: typeof import("../../app/api/transactions/[id]/void/route");
|
||||
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
|
||||
|
||||
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
|
||||
const params = (id: number) => ({ params: Promise.resolve({ id: String(id) }) });
|
||||
const doVoid = (id: number, reason: unknown = "hold released, booking cancelled") =>
|
||||
voidRoute.POST(req({ reason }), params(id));
|
||||
|
||||
let acctSeq = 0;
|
||||
async function statementLine(amount: number, date: string, description: string) {
|
||||
const st = await pool.query(
|
||||
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
|
||||
);
|
||||
const t = await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
|
||||
[st.rows[0].id, date, description, amount, ownerId]
|
||||
);
|
||||
return t.rows[0].id as number;
|
||||
}
|
||||
|
||||
/** What every "count each purchase once" query sees. */
|
||||
async function countedIds(): Promise<number[]> {
|
||||
const r = await pool.query(
|
||||
`SELECT t.id FROM transactions t WHERE ${EXCLUDE_RECONCILED_SOURCE} ORDER BY t.id`
|
||||
);
|
||||
return r.rows.map((x) => x.id);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
pool = createPool();
|
||||
vi.doMock("@/lib/auth", () => ({
|
||||
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||
}));
|
||||
voidRoute = await import("../../app/api/transactions/[id]/void/route");
|
||||
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM expense_metadata");
|
||||
({ ownerId, otherId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("voiding a charge that never posted", () => {
|
||||
it("drops the row out of every figure and the reconcile queue, but keeps it", async () => {
|
||||
const { getPendingReconciliations } = await import("@/lib/queries");
|
||||
const hold = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 1040.41 });
|
||||
const real = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 529.05 });
|
||||
|
||||
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).toContain(hold);
|
||||
|
||||
const res = await doVoid(hold);
|
||||
expect(res.status ?? 200).toBe(200);
|
||||
|
||||
expect(await countedIds()).toEqual([real]);
|
||||
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).not.toContain(hold);
|
||||
const r = await pool.query(`SELECT voided_at, void_reason FROM transactions WHERE id = $1`, [hold]);
|
||||
expect(r.rows[0].voided_at).not.toBeNull();
|
||||
expect(r.rows[0].void_reason).toBe("hold released, booking cancelled");
|
||||
});
|
||||
|
||||
it("can be undone", async () => {
|
||||
const hold = await insertTransaction(pool, ownerId, { amount: 1040.41 });
|
||||
await doVoid(hold);
|
||||
const res = await voidRoute.DELETE(req({}), params(hold));
|
||||
expect(res.status ?? 200).toBe(200);
|
||||
expect(await countedIds()).toEqual([hold]);
|
||||
});
|
||||
|
||||
it("requires a reason", async () => {
|
||||
const hold = await insertTransaction(pool, ownerId, { amount: 10 });
|
||||
expect((await doVoid(hold, " ")).status).toBe(400);
|
||||
expect((await doVoid(hold, null)).status).toBe(400);
|
||||
expect(await countedIds()).toEqual([hold]);
|
||||
});
|
||||
|
||||
it("refuses a statement line -- that charge posted", async () => {
|
||||
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
|
||||
expect((await doVoid(stmt)).status).toBe(409);
|
||||
expect(await countedIds()).toEqual([stmt]);
|
||||
});
|
||||
|
||||
it("refuses a reconciled row -- it found its statement line, so it posted", async () => {
|
||||
const wallet = await insertTransaction(pool, ownerId, { amount: 50, transaction_date: "2026-09-10" });
|
||||
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
|
||||
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [stmt, wallet]);
|
||||
expect((await doVoid(wallet)).status).toBe(409);
|
||||
});
|
||||
|
||||
it("refuses a row with a settled split -- someone already paid their share", async () => {
|
||||
const hold = await insertTransaction(pool, ownerId, { amount: 80 });
|
||||
await pool.query(
|
||||
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
|
||||
VALUES ($1, $2, 50, true)`,
|
||||
[hold, otherId]
|
||||
);
|
||||
expect((await doVoid(hold)).status).toBe(409);
|
||||
expect(await countedIds()).toEqual([hold]);
|
||||
});
|
||||
|
||||
it("is enforced by the database, not only the route", async () => {
|
||||
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
|
||||
await expect(
|
||||
pool.query(`UPDATE transactions SET voided_at = now(), void_reason = 'x' WHERE id = $1`, [stmt])
|
||||
).rejects.toThrow(/transactions_void_statementless/);
|
||||
|
||||
const manual = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||
await expect(
|
||||
pool.query(`UPDATE transactions SET voided_at = now() WHERE id = $1`, [manual])
|
||||
).rejects.toThrow(/transactions_void_has_reason/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the matchers never use a voided row", () => {
|
||||
it("auto-reconcile does not pair a voided row with a statement line of the same amount", async () => {
|
||||
const { findAutoMatches } = await import("@/lib/auto-reconcile");
|
||||
const hold = await insertTransaction(pool, ownerId, { amount: 120, transaction_date: "2026-09-10" });
|
||||
await statementLine(120, "2026-09-11", "SOMETHING");
|
||||
expect((await findAutoMatches(ownerId)).map((m) => m.manual_id)).toContain(hold);
|
||||
|
||||
await doVoid(hold);
|
||||
expect(await findAutoMatches(ownerId)).toEqual([]);
|
||||
});
|
||||
|
||||
it("a receipt for a rebooking at the same amount adopts the live row, not the released hold", async () => {
|
||||
const { findAdoptableTransaction } = await import("@/lib/auto-reconcile");
|
||||
const hold = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-10" });
|
||||
const rebooked = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-11" });
|
||||
// Two candidates: the adopter refuses rather than guess.
|
||||
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBeNull();
|
||||
|
||||
await doVoid(hold);
|
||||
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBe(rebooked);
|
||||
});
|
||||
|
||||
it("merge refuses a voided row on either side", async () => {
|
||||
const hold = await insertTransaction(pool, ownerId, { amount: 70, transaction_date: "2026-09-10" });
|
||||
const stmt = await statementLine(70, "2026-09-10", "SOMETHING");
|
||||
await doVoid(hold);
|
||||
const res = await merge(req({ other_id: stmt, mode: "reconcile" }), params(hold));
|
||||
expect(res.status).toBe(409);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,176 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants } from "./helpers";
|
||||
|
||||
/**
|
||||
* The mirror of receipt-adopts-wallet-row: a wallet notification arriving AFTER
|
||||
* something else already booked the purchase must fold into that row.
|
||||
*
|
||||
* Built from the failure, not from the design. The 2026-08-18 Woolworths shop
|
||||
* ($103.30, Visa gift card) reached the ledger twice: txn 6858 from the pantry
|
||||
* receipt on the 19th, and txn 6927 from the wallet lane on the 22nd, when the
|
||||
* first 21 wallet events were backfilled days after their taps. The sequencing
|
||||
* the lane is designed around -- wallet first, receipt second -- is a tendency,
|
||||
* and every writer has to survive it not holding.
|
||||
*
|
||||
* Two rules are load-bearing here and are tested separately below:
|
||||
* - the merchant must agree, because losing the "no line items" guard that
|
||||
* protects the other direction means amount+date alone could swallow a tap
|
||||
* into an unrelated purchase of the same value
|
||||
* - a gift-card tap is `credits`, because `card` means "a statement line is
|
||||
* coming" and the reconcile queue filters on exactly that
|
||||
*/
|
||||
|
||||
const TOKEN = "test-wallet-token";
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
|
||||
|
||||
const req = (body: unknown) =>
|
||||
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
|
||||
|
||||
/** A row shaped the way the pantry receipt lane writes one: statementless,
|
||||
* itemised, no location (nothing but a notification supplies a suburb). */
|
||||
async function receiptRow(amount: number, date: string, merchant: string, opts: { items?: number; meta?: boolean } = {}) {
|
||||
const { items = 3, meta = true } = opts;
|
||||
const t = await pool.query(
|
||||
`INSERT INTO transactions
|
||||
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
|
||||
transaction_type, category, payment_method, merchant_name, merchant_normalized, row_index)
|
||||
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'groceries', 'credits', $3, $3, 0)
|
||||
RETURNING id`,
|
||||
[ownerId, date, merchant, amount]
|
||||
);
|
||||
const id = t.rows[0].id as number;
|
||||
if (meta) {
|
||||
const lineItems = Array.from({ length: items }, (_, i) => ({ description: `item ${i}`, qty: 1, amount: 1 }));
|
||||
await pool.query(
|
||||
`INSERT INTO expense_metadata
|
||||
(transaction_id, source, merchant_normalized, amount, transaction_date, currency, line_items, flags)
|
||||
VALUES ($1, 'pantry', $2, $3, $4, 'AUD', $5::jsonb, '[]'::jsonb)`,
|
||||
[id, merchant, amount, date, JSON.stringify(lineItems)]
|
||||
);
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
/** The real event 80: the Apple Wallet half of a Visa gift-card tap. */
|
||||
const woolworthsGiftCard = {
|
||||
event_id: 80, kind: "transaction", direction: "unknown", amount: 103.3,
|
||||
amount_text: "$103.30", currency: "$?", merchant: "Woolworths", issuer: "Visa",
|
||||
notif_app: "Wallet", notif_title: "Visa Gift Card",
|
||||
notif_subtitle: "Woolworths, Wyndham Vale, VIC",
|
||||
occurred_at: "2026-08-18T09:00:14+10:00",
|
||||
};
|
||||
|
||||
const rows = async () =>
|
||||
(await pool.query(
|
||||
`SELECT id, amount, merchant_normalized, location, source, source_ref, payment_method
|
||||
FROM transactions ORDER BY id`
|
||||
)).rows;
|
||||
|
||||
beforeAll(async () => {
|
||||
process.env.WALLET_INGEST_TOKEN = TOKEN;
|
||||
pool = createPool();
|
||||
vi.doMock("@/lib/auth", () => ({
|
||||
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||
}));
|
||||
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM expense_metadata");
|
||||
({ ownerId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("a wallet notification for a purchase the ledger already holds", () => {
|
||||
it("adopts the receipt's row instead of creating a second one", async () => {
|
||||
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
|
||||
|
||||
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||
|
||||
expect(body.skipped).toBe("adopted");
|
||||
expect(body.transactionId).toBe(existing);
|
||||
expect(await rows()).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("gives the adopted row the suburb, and leaves what the receipt read alone", async () => {
|
||||
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
|
||||
await ingest(req(woolworthsGiftCard));
|
||||
|
||||
const [t] = await rows();
|
||||
// Wallet's own field: nothing else on the estate writes a suburb.
|
||||
expect(t.location).toBe("Wyndham Vale, VIC");
|
||||
// The receipt read the register tape. The notification does not get to
|
||||
// rename the merchant to its own shorter string.
|
||||
expect(t.merchant_normalized).toBe("WOOLWORTHS 3345");
|
||||
expect(t.id).toBe(existing);
|
||||
|
||||
const [m] = (await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
|
||||
expect(m.source).toBe("pantry");
|
||||
expect(m.line_items).toHaveLength(3);
|
||||
expect(m.payment_method_detail).toBe("Visa");
|
||||
expect(m.flags).toContain("wallet_evt:80");
|
||||
});
|
||||
|
||||
it("is idempotent when the interpreter replays every event", async () => {
|
||||
// Bumping prompt_version re-interprets and re-posts the whole capture
|
||||
// history on purpose -- that is the replay mechanism. An adopted row keeps
|
||||
// no wallet source_ref of its own, so the flag is what makes replay safe.
|
||||
await ingest(req(woolworthsGiftCard));
|
||||
const second = await (await ingest(req(woolworthsGiftCard))).json();
|
||||
expect(second.skipped).toBe("already_ingested");
|
||||
expect(await rows()).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("writes its own row when the merchant disagrees", async () => {
|
||||
// Same money, same day, different shop. Folding into it would make the tap
|
||||
// vanish -- worse than a duplicate, which is at least visible in the queue.
|
||||
await receiptRow(103.3, "2026-08-18", "Bunnings Werribee");
|
||||
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||
|
||||
expect(body.skipped).toBeUndefined();
|
||||
expect(await rows()).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("writes its own row when two candidates share the amount", async () => {
|
||||
await receiptRow(103.3, "2026-08-18", "Woolworths Wyndham Vale");
|
||||
await receiptRow(103.3, "2026-08-18", "Woolworths Point Cook");
|
||||
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||
|
||||
expect(body.skipped).toBeUndefined();
|
||||
expect(await rows()).toHaveLength(3);
|
||||
});
|
||||
|
||||
it("creates the metadata row when the adopted transaction has none", async () => {
|
||||
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345", { meta: false });
|
||||
await ingest(req(woolworthsGiftCard));
|
||||
|
||||
const meta = (await pool.query(`SELECT * FROM expense_metadata`)).rows;
|
||||
expect(meta).toHaveLength(1);
|
||||
expect(meta[0].transaction_id).toBe(existing);
|
||||
expect(meta[0].card_last4).toBeNull();
|
||||
expect(meta[0].flags).toContain("wallet_evt:80");
|
||||
});
|
||||
});
|
||||
|
||||
describe("a stored-value instrument", () => {
|
||||
it("is booked as credits, so it never enters the reconcile queue", async () => {
|
||||
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||
const [t] = await rows();
|
||||
|
||||
expect(body.transactionId).toBeTruthy();
|
||||
// 'card' would park it forever: findAutoMatches and the queue both exclude
|
||||
// 'credits' and 'cash' precisely because no statement line is coming.
|
||||
expect(t.payment_method).toBe("credits");
|
||||
expect(body.flags).toContain("stored_value_instrument");
|
||||
});
|
||||
|
||||
it("leaves an ordinary card tap as card", async () => {
|
||||
const body = await (await ingest(req({ ...woolworthsGiftCard, notif_title: "Payment", issuer: "Mastercard", notif_app: "Westpac" }))).json();
|
||||
const [t] = await rows();
|
||||
expect(t.payment_method).toBe("card");
|
||||
expect(body.flags).not.toContain("stored_value_instrument");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,403 @@
|
||||
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||
import { Pool } from "pg";
|
||||
import { createPool, resetDB, seedParticipants } from "./helpers";
|
||||
|
||||
/**
|
||||
* Wallet notifications becoming transactions.
|
||||
*
|
||||
* Every fixture is a real captured event from capture.wallet_events
|
||||
* (2026-08-13 .. 08-22). The rules being tested are the ones the design turned
|
||||
* on, so each has a reason rather than being a shape check:
|
||||
*
|
||||
* - a transaction for EVERY tap, because overseas is where live spend matters
|
||||
* - the pair is MERGED, never deduped: the issuer half has the card tail, the
|
||||
* Wallet half has the merchant and suburb, neither alone is complete
|
||||
* - the NATIVE amount is what gets stored for a foreign charge, because the
|
||||
* AUD figure is the bank's own conversion and cannot be predicted
|
||||
* - a bare "$" is recorded as unknown, never assumed AUD
|
||||
*/
|
||||
|
||||
const TOKEN = "test-wallet-token";
|
||||
let pool: Pool;
|
||||
let ownerId: number;
|
||||
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
|
||||
|
||||
const req = (body: unknown) =>
|
||||
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
|
||||
|
||||
const noAuth = (body: unknown) =>
|
||||
({ headers: { get: () => null }, json: async () => body }) as never;
|
||||
|
||||
// --- real captured events -------------------------------------------------
|
||||
const breadworldBank = {
|
||||
event_id: 64, kind: "transaction", direction: "debit", amount: 13.5,
|
||||
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
|
||||
card_tail: "8032", issuer: "Mastercard", notif_app: "Westpac",
|
||||
notif_body: "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU",
|
||||
occurred_at: "2026-08-13T08:01:28+10:00",
|
||||
};
|
||||
const breadworldWallet = {
|
||||
event_id: 65, kind: "transaction", direction: "unknown", amount: 13.5,
|
||||
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
|
||||
notif_app: "Wallet", notif_subtitle: "Breadworld Bakery Cafe, Wyndham Vale, VIC",
|
||||
occurred_at: "2026-08-13T08:01:35+10:00",
|
||||
};
|
||||
const amexRewardPay = {
|
||||
event_id: 87, kind: "transaction", direction: "debit", amount: 5103.49,
|
||||
amount_text: "A$5,103.49", currency: "AUD", merchant: "REWARDPAY",
|
||||
card_tail: "01000", issuer: "American Express", notif_app: "Amex AU",
|
||||
notif_body: "You have made a A$5,103.49 charge on your American Express® Card ending in 01000 at REWARDPAY.",
|
||||
occurred_at: "2026-08-20T11:54:43+10:00",
|
||||
};
|
||||
const ebayOrderPlaced = {
|
||||
event_id: 66, kind: "not_transaction", direction: "unknown", amount: null,
|
||||
notif_app: "Wallet", notif_body: "eBay has received your order.",
|
||||
occurred_at: "2026-08-13T09:45:55+10:00",
|
||||
};
|
||||
|
||||
/** Only the rows this lane wrote -- some tests seed statement history for the
|
||||
* rate reference, and those must not be mistaken for wallet output. */
|
||||
const txns = async () =>
|
||||
(await pool.query(
|
||||
`SELECT id, amount, amount_aud, merchant_normalized, location, source, source_ref,
|
||||
foreign_currency_amount, foreign_currency_code, payment_method, statement_id
|
||||
FROM transactions WHERE source = 'wallet' ORDER BY id`
|
||||
)).rows;
|
||||
|
||||
const metas = async () =>
|
||||
(await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
|
||||
|
||||
beforeAll(async () => {
|
||||
process.env.WALLET_INGEST_TOKEN = TOKEN;
|
||||
pool = createPool();
|
||||
vi.doMock("@/lib/auth", () => ({
|
||||
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||
}));
|
||||
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// No network in tests. The ECB fallback treats any fetch failure as "no
|
||||
// rate", which is what every pre-existing currency test assumes; the one test
|
||||
// about the fallback stubs a rate of its own.
|
||||
vi.stubGlobal("fetch", async () => { throw new Error("no network in tests"); });
|
||||
await resetDB(pool);
|
||||
await pool.query("DELETE FROM expense_metadata");
|
||||
({ ownerId } = await seedParticipants(pool));
|
||||
});
|
||||
|
||||
describe("auth", () => {
|
||||
it("fails closed without the token", async () => {
|
||||
expect((await ingest(noAuth(breadworldBank))).status).toBe(401);
|
||||
expect(await txns()).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("what reaches the ledger", () => {
|
||||
it("creates a statementless card transaction for a tap", async () => {
|
||||
const body = await (await ingest(req(breadworldBank))).json();
|
||||
expect(body.transactionId).toBeTruthy();
|
||||
|
||||
const [t] = await txns();
|
||||
expect(Number(t.amount)).toBe(13.5);
|
||||
expect(t.source).toBe("wallet");
|
||||
expect(t.source_ref).toBe("evt:64");
|
||||
// Statementless + card is exactly what puts it in the reconcile queue, with
|
||||
// no extra wiring: that is how the statement eventually confirms it.
|
||||
expect(t.statement_id).toBeNull();
|
||||
expect(t.payment_method).toBe("card");
|
||||
});
|
||||
|
||||
it("ignores an eBay order-lifecycle notification", async () => {
|
||||
// 10 of the 13 non-transactions in the capture window are these.
|
||||
const body = await (await ingest(req(ebayOrderPlaced))).json();
|
||||
expect(body.skipped).toBe("not_a_transaction");
|
||||
expect(await txns()).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("is idempotent on the event id", async () => {
|
||||
await ingest(req(breadworldBank));
|
||||
const second = await (await ingest(req(breadworldBank))).json();
|
||||
expect(second.skipped).toBe("already_ingested");
|
||||
expect(await txns()).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the notification pair", () => {
|
||||
it("merges the two halves into ONE transaction, keeping both contributions", async () => {
|
||||
await ingest(req(breadworldBank)); // has the card tail
|
||||
await ingest(req(breadworldWallet)); // has the suburb
|
||||
|
||||
const rows = await txns();
|
||||
expect(rows).toHaveLength(1);
|
||||
// The Wallet half supplies the suburb; the issuer half never had one.
|
||||
expect(rows[0].location).toBe("Wyndham Vale, VIC");
|
||||
|
||||
const [m] = await metas();
|
||||
// ...and the issuer half's card tail survives the merge.
|
||||
expect(m.card_last4).toBe("8032");
|
||||
expect(m.payment_method_detail).toBe("Mastercard");
|
||||
});
|
||||
|
||||
it("merges in either arrival order", async () => {
|
||||
await ingest(req(breadworldWallet));
|
||||
await ingest(req(breadworldBank));
|
||||
|
||||
expect(await txns()).toHaveLength(1);
|
||||
const [m] = await metas();
|
||||
expect(m.card_last4).toBe("8032");
|
||||
const [t] = await txns();
|
||||
expect(t.location).toBe("Wyndham Vale, VIC");
|
||||
});
|
||||
});
|
||||
|
||||
describe("the notification pair is recognised by time and app, not amount alone", () => {
|
||||
it("makes ONE row when both halves are posted at the same moment", async () => {
|
||||
// n8n posts a poll's items concurrently, and both halves of a tap usually
|
||||
// land in the same poll. Before the lock, each ran its pair lookup before
|
||||
// the other inserted: 11 taps became 22 rows, identical created_at on each.
|
||||
await Promise.all([ingest(req(breadworldBank)), ingest(req(breadworldWallet))]);
|
||||
expect(await txns()).toHaveLength(1);
|
||||
const [m] = await metas();
|
||||
expect(m.card_last4).toBe("8032");
|
||||
});
|
||||
|
||||
it("keeps two same-amount charges hours apart as two purchases", async () => {
|
||||
// Event 275 ($50 at A MEDIUM) merged into the UNICEF $50 donation from 17
|
||||
// hours earlier and renamed it, because the old window was +/-1 day.
|
||||
await ingest(req({
|
||||
...amexRewardPay, event_id: 267, amount: 50, amount_text: "A$50.00", merchant: "UNICEF",
|
||||
notif_body: "You have made a A$50.00 charge on your American Express® Card ending in 01000 at UNICEF.",
|
||||
occurred_at: "2026-10-01T11:02:40+10:00",
|
||||
}));
|
||||
await ingest(req({
|
||||
...amexRewardPay, event_id: 275, amount: 50, amount_text: "$50.00", merchant: "A MEDIUM",
|
||||
notif_body: "You have made a $50.00 charge on your American Express® Card ending in 01000 at A MEDIUM.",
|
||||
occurred_at: "2026-10-02T04:06:49+10:00",
|
||||
}));
|
||||
const rows = await txns();
|
||||
expect(rows).toHaveLength(2);
|
||||
expect(rows.map((r) => r.merchant_normalized).sort()).toEqual(["A MEDIUM", "UNICEF"]);
|
||||
});
|
||||
|
||||
it("never pairs two notifications from the SAME app, however close", async () => {
|
||||
// A pair is the issuer half plus the Wallet half. Two from one app are two
|
||||
// charges -- two $20 coffees bought together, say.
|
||||
await ingest(req(breadworldBank));
|
||||
await ingest(req({ ...breadworldBank, event_id: 640, occurred_at: "2026-08-13T08:01:40+10:00" }));
|
||||
expect(await txns()).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("records a merged half, so a replay cannot merge it again", async () => {
|
||||
await ingest(req(breadworldBank));
|
||||
await ingest(req(breadworldWallet));
|
||||
const again = await (await ingest(req(breadworldWallet))).json();
|
||||
expect(again.skipped).toBe("already_ingested");
|
||||
const [m] = await metas();
|
||||
expect(m.flags).toContain("wallet_evt:65");
|
||||
});
|
||||
});
|
||||
|
||||
describe("notifications that name an amount but are not purchases", () => {
|
||||
it("skips an overdraft notice", async () => {
|
||||
// Event 235 became a $21.21 purchase named "To avoid fees and interest,".
|
||||
const body = await (await ingest(req({
|
||||
event_id: 235, kind: "transaction", direction: "debit", amount: 21.21,
|
||||
amount_text: "$21.21", currency: "$?", merchant: null, notif_app: "CommBank",
|
||||
notif_body: "Your Smart Access 1910 is overdrawn by $21.21. To avoid fees and interest, bring your account to positive by midnight 21 Sep (Syd/Melb time).",
|
||||
occurred_at: "2026-09-22T09:00:18+10:00",
|
||||
}))).json();
|
||||
expect(body.skipped).toBe("account_notice");
|
||||
expect(await txns()).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("skips a transit-card fare, because the card's load is the spend", async () => {
|
||||
// Event 200: the Octopus load is on the bank statement; each fare draws it
|
||||
// down. 11 of 13 fares were already not_transaction; this makes the other 2 agree.
|
||||
const body = await (await ingest(req({
|
||||
event_id: 200, kind: "transaction", direction: "debit", amount: 105,
|
||||
amount_text: "HKD 105.00", currency: "HKD", merchant: "Octopus", notif_app: "Wallet",
|
||||
notif_title: "Octopus", notif_subtitle: "HKD 105.00 for Train",
|
||||
notif_body: "Your new balance is -HKD 12.60.",
|
||||
occurred_at: "2026-09-14T02:00:37+10:00",
|
||||
}))).json();
|
||||
expect(body.skipped).toBe("stored_value_drawdown");
|
||||
expect(await txns()).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("does not read a Westpac descriptor out of another bank's message", async () => {
|
||||
await ingest(req({
|
||||
event_id: 2350, kind: "transaction", direction: "debit", amount: 9.5,
|
||||
amount_text: "$9.50", currency: "$?", merchant: null, notif_app: "CommBank",
|
||||
notif_body: "Payment of $9.50 made. Thanks for banking with us.",
|
||||
occurred_at: "2026-09-22T09:00:18+10:00",
|
||||
}));
|
||||
const [t] = await txns();
|
||||
expect(t.merchant_normalized).toBe("Card purchase");
|
||||
});
|
||||
});
|
||||
|
||||
describe("dating", () => {
|
||||
it("dates a tap in Melbourne local time, as the statements do", async () => {
|
||||
// 21:30 UTC on 30 Sep is 07:30 on 1 Oct in Melbourne. Slicing the ISO
|
||||
// string filed it a day early.
|
||||
await ingest(req({ ...breadworldBank, occurred_at: "2026-09-30T21:30:00Z" }));
|
||||
const { rows } = await pool.query(
|
||||
`SELECT transaction_date::text AS d FROM transactions WHERE source = 'wallet'`
|
||||
);
|
||||
expect(rows[0].d).toBe("2026-10-01");
|
||||
});
|
||||
});
|
||||
|
||||
describe("currency", () => {
|
||||
it("falls back to the ECB rate when the ledger has no history", async () => {
|
||||
// Hong Kong: no HKD statement row existed, so every tap stored A$0.00.
|
||||
const calls: string[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string) => {
|
||||
calls.push(String(url));
|
||||
return { ok: true, json: async () => ({ rates: { AUD: 0.19 } }) };
|
||||
});
|
||||
await ingest(req({
|
||||
event_id: 167, kind: "transaction", direction: "debit", amount: 310.1,
|
||||
amount_text: "HKD 310.10", currency: "HKD", merchant: "Wonder Taxi", notif_app: "Wallet",
|
||||
notif_subtitle: "Wonder Taxi, Tsim Sha Tsui East, Kowloon",
|
||||
occurred_at: "2026-09-10T08:54:00+10:00",
|
||||
}));
|
||||
const [t] = await txns();
|
||||
expect(Number(t.amount)).toBeCloseTo(58.92, 2);
|
||||
expect(Number(t.foreign_currency_amount)).toBe(310.1);
|
||||
expect(calls[0]).toContain("/2026-09-10?from=HKD&to=AUD");
|
||||
const [m] = await metas();
|
||||
// Flagged as the mid-rate, not the bank's: it lacks the card's FX fee.
|
||||
expect(m.flags).toEqual(expect.arrayContaining(["aud_provisional", "rate_ecb"]));
|
||||
expect(m.flags).not.toContain("no_rate_reference");
|
||||
});
|
||||
|
||||
it("records AUD when Amex states it explicitly", async () => {
|
||||
await ingest(req(amexRewardPay));
|
||||
const [t] = await txns();
|
||||
expect(t.foreign_currency_code).toBeNull();
|
||||
expect(Number(t.amount)).toBe(5103.49);
|
||||
const [m] = await metas();
|
||||
expect(m.currency).toBe("AUD");
|
||||
expect(m.flags).not.toContain("currency_unverified");
|
||||
});
|
||||
|
||||
it("flags a bare dollar sign rather than assuming AUD", async () => {
|
||||
// 14 of 21 captured transaction rows are a bare "$". Assuming AUD is
|
||||
// harmless at home and a 23x error on a THB tap abroad.
|
||||
await ingest(req(breadworldBank));
|
||||
const [m] = await metas();
|
||||
expect(m.currency).toBeNull();
|
||||
expect(m.flags).toContain("currency_unverified");
|
||||
});
|
||||
|
||||
it("stores a foreign charge on the NATIVE pair, with a provisional AUD", async () => {
|
||||
// Seed enough history for the median rate to exist (n >= 5).
|
||||
const st = await pool.query(
|
||||
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||
VALUES ('s.pdf','1','Test Bank','2026-01-01','2026-01-31',$1) RETURNING id`, [ownerId]
|
||||
);
|
||||
for (let i = 0; i < 6; i++) {
|
||||
await pool.query(
|
||||
`INSERT INTO transactions (statement_id, transaction_date, description, amount,
|
||||
transaction_type, owner_id, foreign_currency_amount, foreign_currency_code)
|
||||
VALUES ($1,'2026-01-10','BANGKOK',4.57,'debit',$2,100,'THB')`, [st.rows[0].id, ownerId]
|
||||
);
|
||||
}
|
||||
|
||||
await ingest(req({
|
||||
event_id: 999, kind: "transaction", direction: "debit", amount: 500,
|
||||
amount_text: "฿500", merchant: "7-Eleven", notif_app: "Wallet",
|
||||
notif_subtitle: "7-Eleven, Bangkok", occurred_at: "2026-08-20T12:00:00+10:00",
|
||||
}));
|
||||
|
||||
const [t] = await txns();
|
||||
expect(Number(t.foreign_currency_amount)).toBe(500);
|
||||
expect(t.foreign_currency_code).toBe("THB");
|
||||
// The native figure is exact on both sides, so it is what the statement
|
||||
// will match on. amount is only ever provisional here.
|
||||
expect(Number(t.amount)).toBeCloseTo(22.85, 1);
|
||||
const [m] = await metas();
|
||||
expect(m.flags).toContain("aud_provisional");
|
||||
});
|
||||
|
||||
it("still records a foreign charge when there is no rate history", async () => {
|
||||
await ingest(req({
|
||||
event_id: 998, kind: "transaction", direction: "debit", amount: 42,
|
||||
amount_text: "€42.00", merchant: "Somewhere", notif_app: "Wallet",
|
||||
occurred_at: "2026-08-20T12:00:00+10:00",
|
||||
}));
|
||||
const [t] = await txns();
|
||||
expect(t.foreign_currency_code).toBe("EUR");
|
||||
const [m] = await metas();
|
||||
// Says so, rather than inventing a rate.
|
||||
expect(m.flags).toContain("no_rate_reference");
|
||||
// And never stores the native figure as AUD (672006 VND became A$672,006).
|
||||
expect(Number(t.amount)).toBe(0);
|
||||
expect(Number(t.foreign_currency_amount)).toBe(42);
|
||||
});
|
||||
|
||||
it("derives the rate from statement rows only, never from wallet rows", async () => {
|
||||
// Six statementless rows at a nonsense 1:1 rate must not count as history.
|
||||
for (let i = 0; i < 6; i++) {
|
||||
await pool.query(
|
||||
`INSERT INTO transactions (transaction_date, description, amount, transaction_type,
|
||||
owner_id, foreign_currency_amount, foreign_currency_code, source, source_ref)
|
||||
VALUES ('2026-01-10','VN',1000,'debit',$1,1000,'VND','wallet',$2)`,
|
||||
[ownerId, `evt:seed${i}`]
|
||||
);
|
||||
}
|
||||
await ingest(req({
|
||||
event_id: 997, kind: "transaction", direction: "debit", amount: 672006,
|
||||
amount_text: "672006 VND", currency: "VND", merchant: "EVISA VIETNAM", notif_app: "CommBank",
|
||||
occurred_at: "2026-09-05T11:22:00+10:00",
|
||||
}));
|
||||
const t = (await txns()).find((r) => r.source_ref === "evt:997")!;
|
||||
expect(Number(t.amount)).toBe(0);
|
||||
const [m] = (await metas()).filter((r) => r.source_message_id === "evt:997");
|
||||
expect(m.flags).toContain("no_rate_reference");
|
||||
});
|
||||
|
||||
it("pairs foreign halves on the native figure, not the placeholder amount", async () => {
|
||||
await ingest(req({
|
||||
event_id: 996, kind: "transaction", direction: "debit", amount: 42,
|
||||
amount_text: "€42.00", merchant: "Cafe A", notif_app: "Wallet",
|
||||
occurred_at: "2026-08-20T12:00:00+10:00",
|
||||
}));
|
||||
await ingest(req({
|
||||
event_id: 995, kind: "transaction", direction: "debit", amount: 17,
|
||||
amount_text: "€17.00", merchant: "Cafe B", notif_app: "Wallet",
|
||||
occurred_at: "2026-08-20T12:05:00+10:00",
|
||||
}));
|
||||
// Both have amount 0; they are different purchases and must stay two rows.
|
||||
expect((await txns()).filter((r) => r.foreign_currency_code === "EUR")).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("location", () => {
|
||||
it("takes the suburb from the Wallet subtitle", async () => {
|
||||
await ingest(req(breadworldWallet));
|
||||
const [t] = await txns();
|
||||
expect(t.location).toBe("Wyndham Vale, VIC");
|
||||
});
|
||||
|
||||
it("falls back to the fixed-width descriptor, country included", async () => {
|
||||
// The issuer half has no subtitle; the country code is positional.
|
||||
await ingest(req({
|
||||
...breadworldBank, event_id: 75, amount: 42.44,
|
||||
merchant: "SP PEAKDESIGN",
|
||||
// The model's own answer for this row, included because every real
|
||||
// interpretation carries one -- omitting it is what let a wrong
|
||||
// precedence order pass this test the first time, and ship
|
||||
// "SAN FRANCISOUS" into the ledger.
|
||||
location: "SAN FRANCISOUS",
|
||||
notif_body: "$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS",
|
||||
}));
|
||||
const [t] = await txns();
|
||||
expect(t.location).toBe("SAN FRANCISO US");
|
||||
const [m] = await metas();
|
||||
// The only foreign merchant in the whole capture window — worth surfacing.
|
||||
expect(m.flags).toContain("foreign_country_US");
|
||||
});
|
||||
});
|
||||
@@ -3,17 +3,21 @@ import { readFileSync } from "fs";
|
||||
import { resolve } from "path";
|
||||
import {
|
||||
classifyInput,
|
||||
confabulationReason,
|
||||
GROCERY_PATTERN,
|
||||
merchantKey,
|
||||
merchantMatches,
|
||||
platformForSource,
|
||||
PROMPT_PLACEHOLDER_MERCHANTS,
|
||||
quoteShared,
|
||||
reconcileItems,
|
||||
resolveMerchantKind,
|
||||
textCouldBeReceipt,
|
||||
toLineItems,
|
||||
validateHeader,
|
||||
type ExtractedOrder,
|
||||
} from "../../lib/intake-rules";
|
||||
import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT, ExtractionError, liteLLMExtractor } from "../../lib/intake-extract";
|
||||
import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT, ExtractionError, liteLLMExtractor, ORDER_ITEM_PROMPT } from "../../lib/intake-extract";
|
||||
import { askCard, candidatesCard, enrichedCard, type Block } from "../../lib/intake-blocks";
|
||||
|
||||
/**
|
||||
@@ -298,3 +302,118 @@ describe("liteLLMExtractor — a finish_reason that is present must be `stop`",
|
||||
await expect(extractor("tool_calls")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/tool_calls/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("textCouldBeReceipt — nothing that cannot be a receipt reaches the model", () => {
|
||||
const why = (t: string | null | undefined) => {
|
||||
const r = textCouldBeReceipt(t);
|
||||
return r.ok ? null : r.reason;
|
||||
};
|
||||
|
||||
it("the live defect: the phone sent the WORD `Image`, not the image", () => {
|
||||
expect(textCouldBeReceipt("Image").ok).toBe(false);
|
||||
expect(why("Image")).toMatch(/single word/);
|
||||
expect(why("Image\n")).toMatch(/single word/);
|
||||
});
|
||||
|
||||
it("a filename is not a receipt, digits in it notwithstanding", () => {
|
||||
expect(why("IMG_0042.jpeg")).toMatch(/single word/);
|
||||
expect(why("Receipt_19Aug2026.pdf")).toMatch(/single word/);
|
||||
});
|
||||
|
||||
it("empty, blank and whitespace-only", () => {
|
||||
expect(why("")).toMatch(/empty/);
|
||||
expect(why(" \n\t ")).toMatch(/empty/);
|
||||
expect(why(null)).toMatch(/empty/);
|
||||
expect(why(undefined)).toMatch(/empty/);
|
||||
});
|
||||
|
||||
it("prose with no numbers cannot be a receipt", () => {
|
||||
expect(why("Shared from Photos")).toMatch(/no numbers/);
|
||||
expect(why("here is the receipt for dinner last night, sorry it took so long")).toMatch(/no numbers/);
|
||||
});
|
||||
|
||||
it("short but with a printed amount goes through — this door accepts a pasted line", () => {
|
||||
expect(textCouldBeReceipt("TOTAL $17.00").ok).toBe(true);
|
||||
expect(textCouldBeReceipt("Total 17.00").ok).toBe(true);
|
||||
expect(textCouldBeReceipt("Gesamt 12,50 EUR").ok).toBe(true);
|
||||
});
|
||||
|
||||
it("short with digits but no money figure does not — a screenshot name is not a purchase", () => {
|
||||
expect(why("Screenshot 2026 08 22")).toMatch(/too short/);
|
||||
expect(why("2 items")).toMatch(/too short/);
|
||||
});
|
||||
|
||||
it("a real receipt body passes", () => {
|
||||
expect(
|
||||
textCouldBeReceipt("Burgertory (Werribee)\n19 Aug 2026\n1x Michael 14.00\n1x Joel 14.00\nSubtotal 28.00\nTotal 22.40").ok
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("quoteShared flattens and truncates what goes on the card", () => {
|
||||
expect(quoteShared("Image")).toBe("Image");
|
||||
expect(quoteShared(" a \n b ")).toBe("a b");
|
||||
expect(quoteShared("x".repeat(200))).toHaveLength(80);
|
||||
expect(quoteShared("x".repeat(200)).endsWith("…")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("confabulationReason — the belt to that pre-check", () => {
|
||||
const nothing = { textParts: ["Image"], hasImages: false };
|
||||
const readable = { textParts: ["Total $22.40 Burgertory (Werribee) 2026-08-19"], hasImages: false };
|
||||
const order = { merchant: "Burgertory (Braybrook)", items: [{ description: "The Pride" }] };
|
||||
|
||||
it("flags an order extracted from nothing readable", () => {
|
||||
expect(confabulationReason(nothing, order)).toMatch(/nothing shared could be read/);
|
||||
});
|
||||
|
||||
it("does not flag one extracted from something readable", () => {
|
||||
expect(confabulationReason(readable, order)).toBeNull();
|
||||
});
|
||||
|
||||
it("does not flag an image share — the pixels are what we asked for", () => {
|
||||
expect(confabulationReason({ textParts: [], hasImages: true }, order)).toBeNull();
|
||||
});
|
||||
|
||||
it("an empty answer to an unreadable share is honest, not a confabulation", () => {
|
||||
expect(confabulationReason(nothing, { merchant: null, items: [] })).toBeNull();
|
||||
});
|
||||
|
||||
it("the prompt's own placeholder coming back as a merchant is always a confabulation", () => {
|
||||
for (const p of PROMPT_PLACEHOLDER_MERCHANTS) {
|
||||
expect(confabulationReason(readable, { merchant: p, items: [] })).toMatch(/prompt's own example/);
|
||||
expect(confabulationReason({ textParts: [], hasImages: true }, { merchant: p, items: [] })).toMatch(/prompt's own example/);
|
||||
}
|
||||
});
|
||||
|
||||
it("a real merchant that happens to be named in the prompt's platform list is NOT flagged", () => {
|
||||
expect(confabulationReason(readable, { merchant: "Coles (Wyndham Vale)", items: [] })).toBeNull();
|
||||
expect(confabulationReason(readable, { merchant: "Woolworths", items: [] })).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("the prompt carries no real merchant to echo", () => {
|
||||
it("names no shop it could hand back as data", () => {
|
||||
expect(ORDER_ITEM_PROMPT).not.toMatch(/Burgertory/);
|
||||
expect(ORDER_ITEM_PROMPT).not.toMatch(/Wyndham Vale/);
|
||||
expect(ORDER_ITEM_PROMPT).toContain(PROMPT_PLACEHOLDER_MERCHANTS[0]);
|
||||
});
|
||||
|
||||
it("tells the model what to return when the input is not a receipt", () => {
|
||||
expect(ORDER_ITEM_PROMPT).toMatch(/NOT a receipt/);
|
||||
expect(ORDER_ITEM_PROMPT).toMatch(/Do not construct a plausible order/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("needsHeaderCard", () => {
|
||||
it("names the missing field and sends a dateless in-store screenshot to the app's Share button", async () => {
|
||||
const { needsHeaderCard } = await import("@/lib/intake-blocks");
|
||||
const dateless = needsHeaderCard(["date"], "woolworths").text;
|
||||
expect(dateless).toContain("no date");
|
||||
expect(dateless).toContain("Share button");
|
||||
expect(dateless).not.toContain("total");
|
||||
const delivery = needsHeaderCard(["total"], "uber_eats").text;
|
||||
expect(delivery).toContain("no total");
|
||||
expect(delivery).toContain("part of the screen");
|
||||
expect(needsHeaderCard().text).toContain("total or date");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
parseWestpacDescriptor,
|
||||
parseWalletSubtitle,
|
||||
currencyFromAmountText,
|
||||
} from "@/lib/wallet-descriptor";
|
||||
|
||||
/**
|
||||
* Every input below is a real captured notification body or subtitle from
|
||||
* capture.wallet_events (2026-08-13 .. 08-22), not an invented fixture. The
|
||||
* model handled this one fixed-width format three different ways on this exact
|
||||
* data, which is why the parsing moved into code.
|
||||
*/
|
||||
|
||||
describe("parseWestpacDescriptor", () => {
|
||||
it("reads merchant, city and country from the fixed-width block", () => {
|
||||
expect(parseWestpacDescriptor(
|
||||
"$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
|
||||
)).toEqual({ merchant: "Breadworld Bakery Cafe", city: "WYNDHAM VAL", country: "AU" });
|
||||
});
|
||||
|
||||
it("handles a city padded to the full field width", () => {
|
||||
expect(parseWestpacDescriptor(
|
||||
"$54.50 paid from Mastercard ending...8032. PLA DONATION MELBOURNE AU"
|
||||
)).toEqual({ merchant: "PLA DONATION", city: "MELBOURNE", country: "AU" });
|
||||
});
|
||||
|
||||
it("splits a 12-char city that runs into the country code with no space", () => {
|
||||
// The case that defeated the model: "SAN FRANCISO" exactly fills the city
|
||||
// field, so "US" follows immediately. Read as one token it becomes the
|
||||
// nonsense location "SAN FRANCISOUS" and the country is lost -- on the only
|
||||
// foreign merchant in the whole capture window.
|
||||
expect(parseWestpacDescriptor(
|
||||
"$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS"
|
||||
)).toEqual({ merchant: "SP PEAKDESIGN", city: "SAN FRANCISO", country: "US" });
|
||||
});
|
||||
|
||||
it("reads mixed-case cities the model inconsistently stripped", () => {
|
||||
expect(parseWestpacDescriptor(
|
||||
"$20.99 paid from Mastercard ending...8032. Spotify P45AE035CB Sydney AU"
|
||||
)).toEqual({ merchant: "Spotify P45AE035CB", city: "Sydney", country: "AU" });
|
||||
});
|
||||
|
||||
it("degrades to nulls rather than guessing", () => {
|
||||
expect(parseWestpacDescriptor(null)).toEqual({ merchant: null, city: null, country: null });
|
||||
expect(parseWestpacDescriptor("")).toEqual({ merchant: null, city: null, country: null });
|
||||
// No sentence terminator: not this format at all.
|
||||
expect(parseWestpacDescriptor("You have a new notification"))
|
||||
.toEqual({ merchant: null, city: null, country: null });
|
||||
});
|
||||
|
||||
it("does not invent a country from a short merchant-only tail", () => {
|
||||
// "...IT" here is the end of a word, not Italy. A wrong country on a
|
||||
// foreign transaction is worse than no country.
|
||||
const r = parseWestpacDescriptor("$9.00 paid from Mastercard ending...8032. SOME MERCHANT");
|
||||
expect(r.country).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseWalletSubtitle", () => {
|
||||
it("separates merchant from suburb and state", () => {
|
||||
expect(parseWalletSubtitle("Breadworld Bakery Cafe, Wyndham Vale, VIC"))
|
||||
.toEqual({ merchant: "Breadworld Bakery Cafe", location: "Wyndham Vale, VIC" });
|
||||
});
|
||||
|
||||
it("returns no location for an online merchant", () => {
|
||||
// 2 of the 10 Wallet subtitles captured have no place at all.
|
||||
expect(parseWalletSubtitle("Seatsaero")).toEqual({ merchant: "Seatsaero", location: null });
|
||||
expect(parseWalletSubtitle("Prism Accounting Bookkeeping Services"))
|
||||
.toEqual({ merchant: "Prism Accounting Bookkeeping Services", location: null });
|
||||
});
|
||||
|
||||
it("keeps a merchant name containing its own comma out of the location", () => {
|
||||
expect(parseWalletSubtitle("Big Daddy`s Fresh Ma, Manor Lakes, VIC").location)
|
||||
.toBe("Manor Lakes, VIC");
|
||||
});
|
||||
|
||||
it("handles absent input", () => {
|
||||
expect(parseWalletSubtitle(null)).toEqual({ merchant: null, location: null });
|
||||
expect(parseWalletSubtitle(" ")).toEqual({ merchant: null, location: null });
|
||||
});
|
||||
});
|
||||
|
||||
describe("currencyFromAmountText", () => {
|
||||
it("reads Amex's explicit marker", () => {
|
||||
expect(currencyFromAmountText("A$5,103.49")).toBe("AUD");
|
||||
expect(currencyFromAmountText("A$13.50")).toBe("AUD");
|
||||
});
|
||||
|
||||
it("refuses to guess a bare dollar sign", () => {
|
||||
// 14 of 21 captured transaction rows are this. Assuming AUD is harmless at
|
||||
// home and a 23x error on a THB tap abroad, so the row goes to the
|
||||
// reconcile queue with no currency instead.
|
||||
expect(currencyFromAmountText("$13.50")).toBeNull();
|
||||
expect(currencyFromAmountText("$103.30")).toBeNull();
|
||||
});
|
||||
|
||||
it("reads unambiguous symbols when a card does state one", () => {
|
||||
expect(currencyFromAmountText("€24.00")).toBe("EUR");
|
||||
expect(currencyFromAmountText("฿500")).toBe("THB");
|
||||
expect(currencyFromAmountText("S$18.20")).toBe("SGD");
|
||||
expect(currencyFromAmountText("NZ$40.00")).toBe("NZD");
|
||||
});
|
||||
|
||||
it("handles absent input", () => {
|
||||
expect(currencyFromAmountText(null)).toBeNull();
|
||||
expect(currencyFromAmountText("")).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,44 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { getCurrentUser } from "@/lib/auth";
|
||||
import { getUpcomingAnnualCharges } from "@/lib/annual-charges";
|
||||
|
||||
/**
|
||||
* Upcoming annual charges, from payments alone.
|
||||
*
|
||||
* Lives here rather than in home-dashboard so the detection sits beside the
|
||||
* ledger it queries, is covered by this repo's integration suite, and can be
|
||||
* reused by a future Slack nudge without moving. home-dashboard reads it over
|
||||
* networks_internal the way its other widgets read their sources.
|
||||
*
|
||||
* Machine access uses the same shared secret as the dashboard's other reads;
|
||||
* a browser session scopes to that user instead.
|
||||
*/
|
||||
export async function GET(req: NextRequest) {
|
||||
// Its OWN token, with no fallback to WALLET_INGEST_TOKEN. home-dashboard's
|
||||
// compose states the rule plainly -- "this app never holds write creds" -- and
|
||||
// WALLET_INGEST_TOKEN unlocks POST /api/wallet/ingest and the estate-wide
|
||||
// reconcile sweep. Handing that to a read-only surface to save minting a
|
||||
// secret would quietly convert a dashboard into a writer. Unset means closed.
|
||||
const token = process.env.DASH_READ_TOKEN;
|
||||
const supplied = req.headers.get("x-ingest-token");
|
||||
const machine = !!token && !!supplied && supplied === token;
|
||||
|
||||
let ownerId: number | null = null;
|
||||
if (machine) {
|
||||
// A machine caller names the owner explicitly rather than defaulting to 1:
|
||||
// silently assuming an owner is how an ingested order ended up invisible in
|
||||
// every scoped view while sitting in the table.
|
||||
const q = req.nextUrl.searchParams.get("owner_id");
|
||||
ownerId = q !== null && Number.isInteger(Number(q)) ? Number(q) : null;
|
||||
if (ownerId === null) {
|
||||
return NextResponse.json({ error: "owner_id is required for machine access" }, { status: 400 });
|
||||
}
|
||||
} else {
|
||||
const user = await getCurrentUser(req);
|
||||
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||
ownerId = user.id;
|
||||
}
|
||||
|
||||
const charges = await getUpcomingAnnualCharges(ownerId);
|
||||
return NextResponse.json({ charges, count: charges.length });
|
||||
}
|
||||
@@ -387,12 +387,21 @@ async function nudgeState(transactionId: number) {
|
||||
currency: string | null;
|
||||
amount: string;
|
||||
}>(
|
||||
`SELECT em.merchant_normalized AS merchant,
|
||||
COALESCE(em.currency, 'AUD') AS currency,
|
||||
// LATERAL + the shared tie-break, not a bare join: `queryRow` returns
|
||||
// rows[0] and Postgres guarantees no ordering, so a second metadata row on
|
||||
// one transaction would flip the merchant and currency on every card
|
||||
// refresh at random. Every sibling reader already resolves this way.
|
||||
`SELECT order_ctx.merchant_normalized AS merchant,
|
||||
COALESCE(order_ctx.currency, 'AUD') AS currency,
|
||||
t.amount
|
||||
FROM transactions t
|
||||
LEFT JOIN expense_metadata em
|
||||
ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT em.merchant_normalized, em.currency
|
||||
FROM expense_metadata em
|
||||
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
|
||||
LIMIT 1
|
||||
) order_ctx ON true
|
||||
WHERE t.id = $1`,
|
||||
[transactionId]
|
||||
);
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { getCurrentUser } from "@/lib/auth";
|
||||
import { queryRaw } from "@/lib/db";
|
||||
import { applyReconcileMatch, supersedeTransaction } from "@/lib/reconcile";
|
||||
|
||||
/**
|
||||
* Manually resolve a duplicate pair the automation cannot see.
|
||||
*
|
||||
* WHY THIS EXISTS: until now there was no way to merge or remove a transaction
|
||||
* at all. `superseded_by_id` was READ everywhere -- analytics excludes it,
|
||||
* auto-reconcile excludes it -- but written nowhere in application code; it had
|
||||
* only ever been set once, by migration 0023. The reconcile queue was the sole
|
||||
* merge path, and it only offers candidates within a narrow amount band, so a
|
||||
* pair outside it (SP PEAKDESIGN: notified $42.44, charged $43.71 -- the card's
|
||||
* 3% foreign transaction fee) was unresolvable from the UI.
|
||||
*
|
||||
* There is deliberately NO delete. Every child of `transactions` is
|
||||
* ON DELETE CASCADE, so removing a row destroys its splits, tags and metadata
|
||||
* curation. Both modes here keep the row and its children and merely stop it
|
||||
* counting:
|
||||
*
|
||||
* reconcile -- a statementLESS row folded into its statement twin
|
||||
* (`reconciled_with_id`). The everyday case.
|
||||
* supersede -- two rows on the same footing, e.g. a statement imported twice
|
||||
* (`superseded_by_id`). No constraint on either side's origin.
|
||||
*
|
||||
* Both move curation onto the survivor FIRST, via one shared implementation in
|
||||
* lib/reconcile.ts, because both hide the source row from every figure and
|
||||
* splits stranded on a hidden row silently stop counting.
|
||||
*/
|
||||
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
|
||||
const user = await getCurrentUser(req);
|
||||
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||
|
||||
const { id } = await params;
|
||||
const loserId = Number(id);
|
||||
if (!Number.isInteger(loserId)) {
|
||||
return NextResponse.json({ error: "invalid transaction id" }, { status: 400 });
|
||||
}
|
||||
|
||||
let body: { other_id?: number; mode?: string };
|
||||
try {
|
||||
body = await req.json();
|
||||
} catch {
|
||||
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
|
||||
}
|
||||
|
||||
const otherId = Number(body.other_id);
|
||||
const mode = body.mode ?? "reconcile";
|
||||
if (!Number.isInteger(otherId)) {
|
||||
return NextResponse.json({ error: "other_id is required" }, { status: 400 });
|
||||
}
|
||||
if (loserId === otherId) {
|
||||
return NextResponse.json({ error: "a transaction cannot merge into itself" }, { status: 400 });
|
||||
}
|
||||
if (mode !== "reconcile" && mode !== "supersede") {
|
||||
return NextResponse.json({ error: "mode must be 'reconcile' or 'supersede'" }, { status: 400 });
|
||||
}
|
||||
|
||||
// Both rows must be the caller's. A statement row carries owner_id on the
|
||||
// statement rather than the transaction, which is why this reads the same
|
||||
// COALESCE every scoped query uses -- checking t.owner_id alone would reject
|
||||
// every statement row as "not found".
|
||||
const rows = await queryRaw<{
|
||||
id: number;
|
||||
statement_id: number | null;
|
||||
reconciled_with_id: number | null;
|
||||
superseded_by_id: number | null;
|
||||
voided_at: Date | null;
|
||||
}>(
|
||||
`SELECT t.id, t.statement_id, t.reconciled_with_id, t.superseded_by_id, t.voided_at
|
||||
FROM transactions t
|
||||
LEFT JOIN statements s ON s.id = t.statement_id
|
||||
WHERE t.id = ANY($1::int[]) AND COALESCE(t.owner_id, s.owner_id) = $2`,
|
||||
[[loserId, otherId], user.id]
|
||||
);
|
||||
if (rows.length !== 2) {
|
||||
return NextResponse.json({ error: "one or both transactions not found" }, { status: 404 });
|
||||
}
|
||||
const loser = rows.find((r) => r.id === loserId)!;
|
||||
const other = rows.find((r) => r.id === otherId)!;
|
||||
|
||||
// Refuse a row that is already folded into something else, in either
|
||||
// direction. Re-pointing it would strand the curation moved on the first
|
||||
// pass -- and silently, since both columns hide the row from every figure.
|
||||
for (const [label, row] of [["source", loser], ["target", other]] as const) {
|
||||
if (row.reconciled_with_id !== null) {
|
||||
return NextResponse.json(
|
||||
{ error: `${label} transaction ${row.id} is already reconciled with ${row.reconciled_with_id}` },
|
||||
{ status: 409 }
|
||||
);
|
||||
}
|
||||
if (row.superseded_by_id !== null) {
|
||||
return NextResponse.json(
|
||||
{ error: `${label} transaction ${row.id} is already superseded by ${row.superseded_by_id}` },
|
||||
{ status: 409 }
|
||||
);
|
||||
}
|
||||
// A voided row is a charge that never posted. As a source it is already
|
||||
// out of every figure; as a survivor it would take the other row out too.
|
||||
if (row.voided_at !== null) {
|
||||
return NextResponse.json(
|
||||
{ error: `${label} transaction ${row.id} is voided (a charge that never posted)` },
|
||||
{ status: 409 }
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
if (mode === "reconcile") {
|
||||
// applyReconcileMatch's contract, enforced here so a misuse is a 400 and
|
||||
// not a row quietly linked the wrong way round.
|
||||
if (loser.statement_id !== null) {
|
||||
return NextResponse.json(
|
||||
{ error: "reconcile expects the source to be a statementless row; use mode 'supersede'" },
|
||||
{ status: 400 }
|
||||
);
|
||||
}
|
||||
if (other.statement_id === null) {
|
||||
return NextResponse.json(
|
||||
{ error: "reconcile expects the target to be a statement row; use mode 'supersede'" },
|
||||
{ status: 400 }
|
||||
);
|
||||
}
|
||||
await applyReconcileMatch(loserId, otherId);
|
||||
} else {
|
||||
// Never hide a statement row behind a statementless one. The statement is
|
||||
// the bank's own record; the statementless row is a notification, a
|
||||
// receipt or a hand entry, and is the side that can be wrong -- a wallet
|
||||
// row carries the amount BEFORE the card's foreign transaction fee, so
|
||||
// burying the statement line under it would leave the ledger holding
|
||||
// $42.44 for a purchase the bank charged $43.71 for, with the statement
|
||||
// no longer reconciling against its own rows. The correct move is the
|
||||
// other direction, which is exactly what `reconcile` does.
|
||||
//
|
||||
// Statement-under-statement stays allowed: that is the imported-twice
|
||||
// pair migration 0023 existed for, and neither side is more true.
|
||||
if (loser.statement_id !== null && other.statement_id === null) {
|
||||
return NextResponse.json(
|
||||
{
|
||||
error:
|
||||
"Refusing to hide a statement row behind a statementless one — the statement is the bank's record. " +
|
||||
`Open transaction ${otherId} instead and fold it into this one.`,
|
||||
},
|
||||
{ status: 400 }
|
||||
);
|
||||
}
|
||||
await supersedeTransaction(loserId, otherId);
|
||||
}
|
||||
} catch (e) {
|
||||
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 400 });
|
||||
}
|
||||
|
||||
return NextResponse.json({ ok: true, mode, source: loserId, target: otherId });
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { getCurrentUser } from "@/lib/auth";
|
||||
import { queryRaw } from "@/lib/db";
|
||||
import { unvoidTransaction, voidTransaction } from "@/lib/reconcile";
|
||||
|
||||
/**
|
||||
* POST /api/transactions/<id>/void { reason } -- a charge that never posted
|
||||
* DELETE /api/transactions/<id>/void -- undo it
|
||||
*
|
||||
* For a statementless row whose charge never reached a statement: an
|
||||
* authorisation hold the merchant released (txn 7951, a cancelled Airbnb
|
||||
* booking). The row stays, with its children and audit trail, and drops out of
|
||||
* every figure and the reconcile queue. Rules and refusals: `voidTransaction`
|
||||
* in lib/reconcile.ts.
|
||||
*/
|
||||
|
||||
async function ownedRow(req: NextRequest, idParam: string) {
|
||||
const user = await getCurrentUser(req);
|
||||
if (!user) return { error: NextResponse.json({ error: "Unauthorized" }, { status: 403 }) };
|
||||
const id = Number(idParam);
|
||||
if (!Number.isInteger(id)) {
|
||||
return { error: NextResponse.json({ error: "invalid transaction id" }, { status: 400 }) };
|
||||
}
|
||||
// Same owner scope as every other route; a statement row carries owner_id on
|
||||
// the statement. (Statement rows are refused later, but must 400, not 404.)
|
||||
const rows = await queryRaw<{ id: number }>(
|
||||
`SELECT t.id FROM transactions t
|
||||
LEFT JOIN statements s ON s.id = t.statement_id
|
||||
WHERE t.id = $1 AND COALESCE(t.owner_id, s.owner_id) = $2`,
|
||||
[id, user.id]
|
||||
);
|
||||
if (rows.length === 0) {
|
||||
return { error: NextResponse.json({ error: "transaction not found" }, { status: 404 }) };
|
||||
}
|
||||
return { id };
|
||||
}
|
||||
|
||||
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
|
||||
const owned = await ownedRow(req, (await params).id);
|
||||
if (owned.error) return owned.error;
|
||||
|
||||
let body: { reason?: unknown };
|
||||
try {
|
||||
body = await req.json();
|
||||
} catch {
|
||||
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
|
||||
}
|
||||
const reason = typeof body.reason === "string" ? body.reason.trim() : "";
|
||||
if (!reason) return NextResponse.json({ error: "reason is required" }, { status: 400 });
|
||||
|
||||
try {
|
||||
await voidTransaction(owned.id, reason);
|
||||
} catch (e) {
|
||||
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 409 });
|
||||
}
|
||||
return NextResponse.json({ ok: true, id: owned.id, voided: true });
|
||||
}
|
||||
|
||||
export async function DELETE(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
|
||||
const owned = await ownedRow(req, (await params).id);
|
||||
if (owned.error) return owned.error;
|
||||
await unvoidTransaction(owned.id);
|
||||
return NextResponse.json({ ok: true, id: owned.id, voided: false });
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { getCurrentUser } from "@/lib/auth";
|
||||
import { prisma, queryRaw } from "@/lib/db";
|
||||
import { queryRaw } from "@/lib/db";
|
||||
import { applyReconcileMatch } from "@/lib/reconcile";
|
||||
import { findAutoMatches } from "@/lib/auto-reconcile";
|
||||
|
||||
export async function POST(req: NextRequest) {
|
||||
const user = await getCurrentUser(req);
|
||||
@@ -17,7 +19,7 @@ export async function POST(req: NextRequest) {
|
||||
// Verify all manual_ids belong to this user
|
||||
const manualIds = body.matches.map((m) => m.manual_id);
|
||||
const owned = await queryRaw<{ id: number }>(
|
||||
`SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND owner_id = $2`,
|
||||
`SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND voided_at IS NULL AND owner_id = $2`,
|
||||
[manualIds, user.id]
|
||||
);
|
||||
if (owned.length !== manualIds.length) {
|
||||
@@ -25,97 +27,71 @@ export async function POST(req: NextRequest) {
|
||||
}
|
||||
|
||||
let reconciled = 0;
|
||||
|
||||
for (const { manual_id, statement_tx_id } of body.matches) {
|
||||
await prisma.$transaction(async (tx) => {
|
||||
// Copy overrides: manual → statement tx
|
||||
const override = await tx.transaction_overrides.findUnique({
|
||||
where: { transaction_id: manual_id },
|
||||
});
|
||||
if (override) {
|
||||
await tx.transaction_overrides.upsert({
|
||||
where: { transaction_id: statement_tx_id },
|
||||
update: {
|
||||
category_override: override.category_override,
|
||||
merchant_normalized: override.merchant_normalized,
|
||||
notes: override.notes,
|
||||
my_share_percent: override.my_share_percent,
|
||||
updated_at: new Date(),
|
||||
},
|
||||
create: {
|
||||
transaction_id: statement_tx_id,
|
||||
category_override: override.category_override,
|
||||
merchant_normalized: override.merchant_normalized,
|
||||
notes: override.notes,
|
||||
my_share_percent: override.my_share_percent,
|
||||
},
|
||||
});
|
||||
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
|
||||
}
|
||||
|
||||
// Move tags: manual → statement tx
|
||||
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
|
||||
if (tags.length) {
|
||||
await tx.transaction_tags.createMany({
|
||||
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
|
||||
skipDuplicates: true,
|
||||
});
|
||||
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
|
||||
}
|
||||
|
||||
// Move splits: manual → statement tx
|
||||
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
|
||||
if (splits.length) {
|
||||
await tx.transaction_splits.createMany({
|
||||
data: splits.map((s) => ({
|
||||
transaction_id: statement_tx_id,
|
||||
participant_id: s.participant_id,
|
||||
share_percent: s.share_percent,
|
||||
})),
|
||||
skipDuplicates: true,
|
||||
});
|
||||
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
|
||||
}
|
||||
|
||||
// Move provenance: manual → statement tx.
|
||||
//
|
||||
// Overrides, tags and splits above were always carried across; expense_metadata was
|
||||
// the one child left behind, which did not matter while every metadata row came from
|
||||
// an email that had created its own transaction. It matters now: a scanned grocery
|
||||
// receipt puts its line items here, and reconciliation hides the manual row from
|
||||
// every figure — so without this the shop's contents disappear at exactly the moment
|
||||
// the statement line appears, and `COLES 0556 MANOR LAKES` stays as unreadable as it
|
||||
// was before the receipt was ever scanned.
|
||||
//
|
||||
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
|
||||
// or Paperless copy got there first) keeps it. The pantry row stays attached to the
|
||||
// reconciled manual transaction and is flagged, rather than raising a constraint
|
||||
// violation or silently overwriting the other source.
|
||||
const moved = await tx.$executeRawUnsafe(
|
||||
`UPDATE expense_metadata SET transaction_id = $1
|
||||
WHERE transaction_id = $2
|
||||
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
|
||||
statement_tx_id,
|
||||
manual_id
|
||||
);
|
||||
if (moved === 0) {
|
||||
await tx.$executeRawUnsafe(
|
||||
`UPDATE expense_metadata
|
||||
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
|
||||
WHERE transaction_id = $1`,
|
||||
manual_id
|
||||
);
|
||||
}
|
||||
|
||||
// Mark manual tx as reconciled (link to statement tx)
|
||||
await tx.$executeRawUnsafe(
|
||||
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
|
||||
statement_tx_id,
|
||||
manual_id
|
||||
);
|
||||
});
|
||||
await applyReconcileMatch(manual_id, statement_tx_id);
|
||||
reconciled++;
|
||||
}
|
||||
|
||||
return NextResponse.json({ reconciled });
|
||||
}
|
||||
|
||||
/**
|
||||
* Automatic pass over the reconcile queue.
|
||||
*
|
||||
* Applies only pairs that are unambiguous by the strict rule in
|
||||
* `findAutoMatches` -- exact amount (or exact native pair for a foreign row),
|
||||
* within a few days, and unique in BOTH directions. Everything else is left in
|
||||
* the queue for a person, which is the point: this exists to keep the queue
|
||||
* workable as the wallet lane adds ~63 rows a month, not to clear it.
|
||||
*
|
||||
* Validated against the 48 reconciliations a human had already made: the rule
|
||||
* reproduces all 48 and proposes nothing they did not.
|
||||
*
|
||||
* `?dry=1` returns what it WOULD do without writing, which is how this should
|
||||
* be run the first time after any change to the matching rule.
|
||||
*/
|
||||
export async function PATCH(req: NextRequest) {
|
||||
// Two callers, two auth modes. A person in the browser reconciles their own
|
||||
// queue; the scheduled sweep has no session and carries the machine token,
|
||||
// and must cover EVERY owner with a queue -- 169 of the 177 rows outstanding
|
||||
// belong to the second participant, so a sweep scoped to one user would leave
|
||||
// almost all of it untouched.
|
||||
const machineToken = req.headers.get("x-ingest-token");
|
||||
const machineOk = !!machineToken && machineToken === process.env.WALLET_INGEST_TOKEN;
|
||||
|
||||
let ownerIds: number[];
|
||||
if (machineOk) {
|
||||
ownerIds = (await queryRaw<{ owner_id: number }>(
|
||||
`SELECT DISTINCT owner_id FROM transactions
|
||||
WHERE statement_id IS NULL AND owner_id IS NOT NULL
|
||||
AND reconciled_with_id IS NULL AND superseded_by_id IS NULL`
|
||||
)).map((r) => r.owner_id);
|
||||
} else {
|
||||
const user = await getCurrentUser(req);
|
||||
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||
ownerIds = [user.id];
|
||||
}
|
||||
|
||||
const dry = new URL(req.url).searchParams.get("dry") === "1";
|
||||
const matches = (await Promise.all(ownerIds.map((id) => findAutoMatches(id)))).flat();
|
||||
|
||||
if (dry) {
|
||||
return NextResponse.json({ dry_run: true, would_reconcile: matches.length, matches });
|
||||
}
|
||||
|
||||
let reconciled = 0;
|
||||
const failed: { manual_id: number; error: string }[] = [];
|
||||
for (const m of matches) {
|
||||
try {
|
||||
await applyReconcileMatch(m.manual_id, m.statement_tx_id);
|
||||
reconciled++;
|
||||
} catch (e) {
|
||||
// One bad pair must not abandon the rest -- each merge is its own
|
||||
// transaction, so a failure here leaves that pair untouched and in the
|
||||
// queue rather than half-applied.
|
||||
failed.push({ manual_id: m.manual_id, error: e instanceof Error ? e.message : String(e) });
|
||||
}
|
||||
}
|
||||
|
||||
return NextResponse.json({ reconciled, examined: matches.length, failed });
|
||||
}
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { processWalletIngestion, type WalletEvent } from "@/lib/wallet-ingestion";
|
||||
|
||||
/**
|
||||
* Machine ingest endpoint for interpreted iOS Wallet notifications.
|
||||
*
|
||||
* Sibling to /api/orders/ingest and /api/receipts/ingest, and deliberately
|
||||
* shaped like them: a shared secret rather than a browser session, because n8n
|
||||
* calls this app-to-app.
|
||||
*
|
||||
* Its own token so the wallet lane's credential can be rotated without touching
|
||||
* the order flow, which runs on a schedule nobody is watching.
|
||||
*
|
||||
* n8n is transport only. The interpretation already happened (Wallet Interpret,
|
||||
* CAp1s5cuzJk2mDaI, local model on Unraid because card tails are PII); this
|
||||
* endpoint owns every decision about what reaches the ledger.
|
||||
*/
|
||||
function authorised(req: NextRequest): boolean {
|
||||
const expected = process.env.WALLET_INGEST_TOKEN;
|
||||
if (!expected) return false; // fail closed when unconfigured
|
||||
const got = req.headers.get("x-ingest-token");
|
||||
return !!got && got === expected;
|
||||
}
|
||||
|
||||
export async function POST(req: NextRequest) {
|
||||
if (!authorised(req)) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
|
||||
|
||||
let body: WalletEvent;
|
||||
try {
|
||||
body = await req.json();
|
||||
} catch {
|
||||
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
|
||||
}
|
||||
|
||||
if (!Number.isInteger(body?.event_id) || typeof body?.kind !== "string") {
|
||||
return NextResponse.json({ error: "event_id (int) and kind are required" }, { status: 400 });
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await processWalletIngestion(body);
|
||||
return NextResponse.json(result);
|
||||
} catch (e) {
|
||||
// A 500 here alerts #log-alerts through the workflow error handler, which
|
||||
// is what should happen: a write that fails is a template or schema change,
|
||||
// not something to swallow.
|
||||
const message = e instanceof Error ? e.message : String(e);
|
||||
return NextResponse.json({ error: message }, { status: 500 });
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import { usePendingReconciliations, useReconcile } from "@/lib/hooks";
|
||||
import { usePendingReconciliations, useReconcile, useVoidTransaction } from "@/lib/hooks";
|
||||
import type { ManualTxWithMatches, PotentialMatch } from "@/lib/queries";
|
||||
|
||||
function formatDate(d: string | Date) {
|
||||
@@ -156,6 +156,7 @@ export default function ReconcilePage() {
|
||||
<span key={tag.id} className="w-2 h-2 rounded-full" style={{ backgroundColor: tag.color }} title={tag.name} />
|
||||
))}
|
||||
</div>
|
||||
<VoidButton id={tx.id} amount={tx.amount} onError={setError} />
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
@@ -243,6 +244,7 @@ function ReconcileRow({
|
||||
Clear
|
||||
</button>
|
||||
)}
|
||||
<VoidButton id={tx.id} amount={tx.amount} />
|
||||
{!isSkipped && (
|
||||
<button onClick={onSkip} className="text-xs text-zinc-500 hover:text-zinc-300 px-2 py-1 rounded hover:bg-zinc-800">
|
||||
Skip (no match)
|
||||
@@ -296,3 +298,34 @@ function MatchRow({
|
||||
</button>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* "Never posted": a released authorisation hold, e.g. a booking cancelled
|
||||
* before the charge settled. Drops the row out of every figure and this queue
|
||||
* (migration 0040); the row and its history are kept.
|
||||
*/
|
||||
function VoidButton({ id, amount, onError }: { id: number; amount: number; onError?: (msg: string) => void }) {
|
||||
const voidTx = useVoidTransaction();
|
||||
async function handle() {
|
||||
const reason = window.prompt(
|
||||
`Mark this $${Number(amount).toFixed(2)} charge as never posted? It will stop counting everywhere.\n\nReason (e.g. "hold released, booking cancelled"):`
|
||||
);
|
||||
if (!reason?.trim()) return;
|
||||
try {
|
||||
await voidTx.mutateAsync({ id, reason });
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : "Void failed";
|
||||
if (onError) onError(msg); else window.alert(msg);
|
||||
}
|
||||
}
|
||||
return (
|
||||
<button
|
||||
onClick={handle}
|
||||
disabled={voidTx.isPending}
|
||||
title="The charge never posted (e.g. a released hold) — stop counting it"
|
||||
className="text-xs text-zinc-500 hover:text-red-300 px-2 py-1 rounded hover:bg-zinc-800 disabled:opacity-50 flex-shrink-0"
|
||||
>
|
||||
{voidTx.isPending ? "Voiding…" : "Never posted"}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -463,7 +463,7 @@ export default function RulesPage() {
|
||||
>
|
||||
{expandedRun === run.id ? "▾" : "▸"}
|
||||
</button>
|
||||
<span className="text-zinc-300">{new Date(run.applied_at).toLocaleString()}</span>
|
||||
<span className="text-zinc-300">{new Date(run.applied_at).toLocaleString("en-AU")}</span>
|
||||
{/* Which rule ran matters most: a merchant rename and a
|
||||
50/50 split of everything look identical as counts. */}
|
||||
<span className="text-zinc-200">
|
||||
@@ -480,7 +480,7 @@ export default function RulesPage() {
|
||||
)}
|
||||
</div>
|
||||
{run.reverted_at ? (
|
||||
<span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString()}</span>
|
||||
<span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString("en-AU")}</span>
|
||||
) : (
|
||||
<button
|
||||
onClick={() => {
|
||||
|
||||
@@ -1085,7 +1085,18 @@ function TransactionsContent() {
|
||||
) : (
|
||||
<>
|
||||
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
|
||||
{t.currency && t.currency !== "AUD" && (
|
||||
{/*
|
||||
The native figure: `foreign_currency_amount` where one
|
||||
was recorded (a statement or wallet row, whose `amount`
|
||||
is AUD), else `amount` on a non-AUD statement. Printing
|
||||
`amount` under the foreign code put the AUD figure
|
||||
behind an INR label on every wallet row.
|
||||
*/}
|
||||
{t.foreign_currency_code && t.foreign_currency_code !== "AUD" && t.foreign_currency_amount !== null ? (
|
||||
<div className="text-[10px] text-zinc-500 mt-0.5">
|
||||
{formatAmount(t.foreign_currency_amount, t.transaction_type, t.foreign_currency_code)}
|
||||
</div>
|
||||
) : t.currency && t.currency !== "AUD" && (
|
||||
<div className="text-[10px] text-zinc-500 mt-0.5">
|
||||
{formatAmount(t.amount, t.transaction_type, t.currency)}
|
||||
</div>
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import { useEffect, useState } from "react";
|
||||
import {
|
||||
useUpdateTransaction,
|
||||
useMergeTransaction,
|
||||
useTags,
|
||||
useAddTransactionTag,
|
||||
useRemoveTransactionTag,
|
||||
@@ -93,6 +94,210 @@ function InlineTags({ transactionId, initialTags }: { transactionId: number; ini
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Determines which merge mode the backend will accept for (loser, candidate).
|
||||
* Mirrors the constraint enforced server-side in the merge route: reconcile
|
||||
* requires a statementless loser folding into a statement survivor; every
|
||||
* other pairing is a supersede.
|
||||
*/
|
||||
function mergeModeFor(loser: TransactionRow, candidate: TransactionRow): "reconcile" | "supersede" {
|
||||
return loser.statement_id === null && candidate.statement_id !== null ? "reconcile" : "supersede";
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapsed by default, at the bottom, visually separate — this is the one
|
||||
* control on the modal that changes which rows this transaction can ever be
|
||||
* again. `transaction` is always the row that will be hidden (the loser);
|
||||
* the row picked here is the survivor (`other_id`).
|
||||
*/
|
||||
function DuplicateSection({ transaction, onMerged }: { transaction: TransactionRow; onMerged: () => void }) {
|
||||
const [expanded, setExpanded] = useState(false);
|
||||
const [query, setQuery] = useState("");
|
||||
const [results, setResults] = useState<TransactionRow[]>([]);
|
||||
const [searching, setSearching] = useState(false);
|
||||
const [selected, setSelected] = useState<TransactionRow | null>(null);
|
||||
const mergeTxn = useMergeTransaction();
|
||||
|
||||
useEffect(() => {
|
||||
if (!expanded || selected) return;
|
||||
const q = query.trim();
|
||||
// No synchronous setState here: an empty box is handled at render time, and
|
||||
// `searching` is raised inside the timeout. Clearing state here instead
|
||||
// cascades a render on every keystroke (react-hooks lint catches it).
|
||||
if (q.length === 0) return;
|
||||
const handle = setTimeout(() => {
|
||||
setSearching(true);
|
||||
fetch(`/api/transactions?search=${encodeURIComponent(q)}&limit=8`)
|
||||
.then((res) => res.json())
|
||||
.then((body: { data?: TransactionRow[] }) => {
|
||||
setResults((body.data ?? []).filter((r) => r.id !== transaction.id));
|
||||
})
|
||||
.catch(() => setResults([]))
|
||||
.finally(() => setSearching(false));
|
||||
}, 300);
|
||||
return () => clearTimeout(handle);
|
||||
}, [query, expanded, selected, transaction.id]);
|
||||
|
||||
function reset() {
|
||||
setExpanded(false);
|
||||
setQuery("");
|
||||
setResults([]);
|
||||
setSelected(null);
|
||||
mergeTxn.reset();
|
||||
}
|
||||
|
||||
async function handleConfirm() {
|
||||
if (!selected) return;
|
||||
try {
|
||||
await mergeTxn.mutateAsync({
|
||||
id: transaction.id,
|
||||
other_id: selected.id,
|
||||
mode: mergeModeFor(transaction, selected),
|
||||
});
|
||||
onMerged();
|
||||
} catch {
|
||||
// Surfaced below via mergeTxn.error — section stays open so they can retry.
|
||||
}
|
||||
}
|
||||
|
||||
const mode = selected ? mergeModeFor(transaction, selected) : null;
|
||||
// Hiding a statement row behind a statementless one is always backwards — the
|
||||
// bank's record is the side that survives. The API refuses it too; catching it
|
||||
// here just means the button explains itself instead of returning a 400.
|
||||
const wrongDirection =
|
||||
selected !== null && transaction.statement_id !== null && selected.statement_id === null;
|
||||
|
||||
return (
|
||||
<div className="pt-4 border-t border-zinc-800">
|
||||
{!expanded ? (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setExpanded(true)}
|
||||
className="text-xs text-zinc-500 hover:text-zinc-300"
|
||||
>
|
||||
Mark as duplicate…
|
||||
</button>
|
||||
) : (
|
||||
<div className="space-y-2.5">
|
||||
<div className="flex items-center justify-between">
|
||||
<p className="font-semibold text-xs text-zinc-400">Mark as duplicate</p>
|
||||
<button
|
||||
type="button"
|
||||
onClick={reset}
|
||||
className="text-xs text-zinc-500 hover:text-zinc-300"
|
||||
>
|
||||
Cancel
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{!selected ? (
|
||||
<>
|
||||
<input
|
||||
type="text"
|
||||
value={query}
|
||||
onChange={(e) => setQuery(e.target.value)}
|
||||
placeholder="Search for the surviving transaction…"
|
||||
autoFocus
|
||||
className="w-full bg-zinc-800 border border-zinc-700 rounded px-2 py-1.5 text-sm"
|
||||
/>
|
||||
{searching && <p className="text-xs text-zinc-500">Searching…</p>}
|
||||
{!searching && query.trim().length > 0 && results.length === 0 && (
|
||||
<p className="text-xs text-zinc-600 italic">No matches</p>
|
||||
)}
|
||||
{/* Gated on the query too: results are no longer cleared when the
|
||||
box empties, so without this the last search's rows linger. */}
|
||||
{query.trim().length > 0 && results.length > 0 && (
|
||||
<div className="space-y-1 max-h-48 overflow-y-auto">
|
||||
{results.map((r) => (
|
||||
<button
|
||||
key={r.id}
|
||||
type="button"
|
||||
onClick={() => setSelected(r)}
|
||||
className="w-full text-left px-2 py-1.5 rounded bg-zinc-800/50 hover:bg-zinc-800 border border-zinc-700/50"
|
||||
>
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<span className="text-sm text-zinc-200 truncate">{r.description}</span>
|
||||
<span
|
||||
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(r.transaction_type) ? "text-red-400" : "text-green-400"}`}
|
||||
>
|
||||
{formatAmount(r.amount, r.transaction_type)}
|
||||
</span>
|
||||
</div>
|
||||
<div className="flex items-center justify-between mt-0.5">
|
||||
<span className="text-xs text-zinc-500">
|
||||
{new Date(r.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
|
||||
</span>
|
||||
<span className="text-xs text-zinc-500">
|
||||
{r.statement_id !== null ? "Statement row" : "Statementless"}
|
||||
</span>
|
||||
</div>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
) : (
|
||||
<div className="space-y-2">
|
||||
<div className="px-2 py-1.5 rounded bg-zinc-800/50 border border-zinc-700/50">
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<span className="text-sm text-zinc-200 truncate">{selected.description}</span>
|
||||
<span
|
||||
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(selected.transaction_type) ? "text-red-400" : "text-green-400"}`}
|
||||
>
|
||||
{formatAmount(selected.amount, selected.transaction_type)}
|
||||
</span>
|
||||
</div>
|
||||
<div className="flex items-center justify-between mt-0.5">
|
||||
<span className="text-xs text-zinc-500">
|
||||
{new Date(selected.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
|
||||
</span>
|
||||
<span className="text-xs text-zinc-500">
|
||||
{selected.statement_id !== null ? "Statement row" : "Statementless"}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
{wrongDirection ? (
|
||||
<p className="text-xs text-red-400">
|
||||
This is the wrong way round — the statement is the bank's record, so it
|
||||
has to be the row that survives. Close this and open{" "}
|
||||
<span className="text-zinc-300">{selected.description}</span> instead, then
|
||||
fold that one into this.
|
||||
</p>
|
||||
) : (
|
||||
<p className="text-xs text-amber-400/90">
|
||||
{mode === "reconcile"
|
||||
? "This row will be folded into the statement row above."
|
||||
: "This row will be marked a duplicate of the row above."}
|
||||
</p>
|
||||
)}
|
||||
{mergeTxn.error && (
|
||||
<p className="text-red-400 text-xs">{(mergeTxn.error as Error).message}</p>
|
||||
)}
|
||||
<div className="flex gap-2">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setSelected(null)}
|
||||
className="px-3 py-1.5 bg-zinc-800 hover:bg-zinc-700 text-zinc-300 rounded-lg text-xs"
|
||||
>
|
||||
Choose a different row
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={handleConfirm}
|
||||
disabled={mergeTxn.isPending || wrongDirection}
|
||||
className="px-3 py-1.5 bg-amber-700 hover:bg-amber-600 disabled:opacity-50 text-white rounded-lg text-xs"
|
||||
>
|
||||
{mergeTxn.isPending ? "Merging…" : mode === "reconcile" ? "Fold into statement row" : "Mark as duplicate"}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function EditTransactionModal({
|
||||
transaction,
|
||||
onClose,
|
||||
@@ -416,6 +621,8 @@ export function EditTransactionModal({
|
||||
|
||||
<OrderDetails transactionId={transaction.id} currency={transaction.currency ?? null} />
|
||||
|
||||
<DuplicateSection transaction={transaction} onMerged={onClose} />
|
||||
|
||||
</div>
|
||||
|
||||
{/* Footer */}
|
||||
|
||||
@@ -71,9 +71,17 @@ export const STATEMENTS_JOIN = `LEFT JOIN statements s ON s.id = t.statement_id`
|
||||
* without being edited. Anything summing transactions without this fragment
|
||||
* still double-counts — that is the same gap that let reconciled rows into the
|
||||
* analytics routes in the first place.
|
||||
*
|
||||
* **3. The charge that never posted** (`voided_at`, migration 0040).
|
||||
*
|
||||
* A card notification is sent for an authorisation, and a released hold never
|
||||
* reaches a statement (txn 7951: a $1,040.41 Airbnb hold on a booking cancelled
|
||||
* the next day). There is no survivor to point at, so it gets its own column —
|
||||
* and it lives in this fragment for the same reason case 2 does.
|
||||
*/
|
||||
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
|
||||
AND t.superseded_by_id IS NULL`;
|
||||
AND t.superseded_by_id IS NULL
|
||||
AND t.voided_at IS NULL`;
|
||||
|
||||
/**
|
||||
* The currency `t.amount` is actually denominated in.
|
||||
@@ -89,10 +97,17 @@ export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reco
|
||||
* figure and `foreign_currency_code` names it (order-ingestion.ts leaves
|
||||
* `amount_aud` NULL rather than asserting an FX rate it does not have).
|
||||
*
|
||||
* - A wallet row also has no statement, but follows the STATEMENT convention:
|
||||
* `amount` = `amount_aud` = a provisional AUD figure, and the native figure
|
||||
* lives in `foreign_currency_amount` (wallet-ingestion.ts). `amount_aud`
|
||||
* being present is what says `amount` is already AUD. Without that arm, an
|
||||
* INR 524 tap stored as A$9.42 rendered as "INR 9.42" -- the AUD number
|
||||
* under the foreign label (txn 8162, 2026-10-03).
|
||||
*
|
||||
* Reading `s.currency` alone labels every foreign order row as AUD.
|
||||
* `s` must be the statements alias in scope.
|
||||
*/
|
||||
export const NATIVE_CURRENCY = `COALESCE(s.currency, t.foreign_currency_code, 'AUD')`;
|
||||
export const NATIVE_CURRENCY = `COALESCE(s.currency, CASE WHEN t.amount_aud IS NULL THEN t.foreign_currency_code END, 'AUD')`;
|
||||
|
||||
/**
|
||||
* True when a row's AUD value is unknown: it is denominated in something other
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
import { queryRaw } from "@/lib/db";
|
||||
|
||||
/**
|
||||
* Upcoming annual charges, derived from payments alone.
|
||||
*
|
||||
* Independent of the email lane on purpose: a recurring CHARGE is visible in the
|
||||
* ledger whether or not anything was ever emailed about it, which is the only
|
||||
* way a card annual fee or a loan package fee shows up at all -- banks announce
|
||||
* those on the statement, not in a renewal notice.
|
||||
*
|
||||
* WHAT DOES NOT WORK, measured before this was written: "two charges about 365
|
||||
* days apart" is useless on its own. Ranked that way the top hit on this ledger
|
||||
* is a bakery with 202 such pairs, then a donation, Patreon, Coles -- merchants
|
||||
* used often enough that some pair is inevitably a year apart. Frequency is the
|
||||
* confound. Even "every gap is annual" still surfaces a Target visited twice
|
||||
* ($30, then $198) and a Specsavers ($24, then $224). Two visits a year apart
|
||||
* are not a fee.
|
||||
*
|
||||
* So there are two signals, and they are kept apart because they are not equally
|
||||
* trustworthy -- a guess must never render beside a certainty as if it were one:
|
||||
*
|
||||
* confirmed the bank called it a fee AND it has repeated on an annual cadence
|
||||
* likely the bank called it a fee, seen once -- the kind is certain, the
|
||||
* cadence is inferred (an annual fee charged once is still annual)
|
||||
* possible no fee wording; a merchant whose every gap is annual AND whose
|
||||
* amount is stable. Catches Amazon Prime, Uber One, VicRoads rego,
|
||||
* none of which carry fee wording. Also where false positives live.
|
||||
*/
|
||||
|
||||
export type Confidence = "confirmed" | "likely" | "possible";
|
||||
|
||||
/**
|
||||
* `expected_not_seen` is not a lesser version of `upcoming` -- it is the more
|
||||
* interesting row. The estimate has passed and no charge landed, which means one
|
||||
* of three things, and `account_days_stale` is what tells them apart:
|
||||
* the account stopped being imported (HSBC: ledger ends 2026-01-07, while the
|
||||
* mail shows statements to 2026-02-07)
|
||||
* the account genuinely closed
|
||||
* the charge landed under a different label (VicRoads paid 2026-03-16 as
|
||||
* "VICROADS ONLINE PAYM", not "VICROADS")
|
||||
* Reporting it as upcoming would be a lie; hiding it would lose the signal.
|
||||
*/
|
||||
export type ChargeStatus = "upcoming" | "expected_not_seen";
|
||||
|
||||
export type AnnualCharge = {
|
||||
label: string;
|
||||
status: ChargeStatus;
|
||||
bank_name: string | null;
|
||||
account_number: string | null;
|
||||
confidence: Confidence;
|
||||
observations: number;
|
||||
last_charged: string;
|
||||
last_amount: string;
|
||||
amount_low: string;
|
||||
amount_high: string;
|
||||
median_gap_days: number | null;
|
||||
next_due_estimate: string;
|
||||
/** Ledger liveness for the account: a fee that "stopped" may just be an
|
||||
* account that stopped being imported. Null for merchant-signal rows, which
|
||||
* are not tied to one account. */
|
||||
account_last_statement: string | null;
|
||||
account_days_stale: number | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* `transaction_type = 'fee'` is the universe, not a description match.
|
||||
*
|
||||
* The statement extractor already classifies fees structurally -- 131 rows
|
||||
* across the ledger -- and every card and loan annual fee on this ledger is
|
||||
* typed that way. An earlier version of this filtered `transaction_type =
|
||||
* 'debit'` and returned NONE of them, which is the whole reason this comment
|
||||
* exists: the fees are not debits.
|
||||
*
|
||||
* That universe is broader than "annual", though. It also holds ANZ's monthly
|
||||
* ACCOUNT SERVICING FEE, Zip's MONTHLY ACCOUNT SERVICE FEE, CommBank's MONTHLY
|
||||
* FEE, Citibank LATE PAYMENT FEE, HSBC OVERSEAS TRANSACTION FEE and every Wise
|
||||
* per-transfer charge. Cadence separates them: an annual fee repeats about a
|
||||
* year apart, the rest repeat monthly, on an event, or never (Wise labels each
|
||||
* transfer uniquely, so they can never group to two observations at all).
|
||||
*/
|
||||
const FEE_TYPE = "fee";
|
||||
|
||||
/**
|
||||
* Wording that establishes the KIND when there is only one observation.
|
||||
*
|
||||
* A single charge cannot demonstrate a cadence, but a bank calling something an
|
||||
* annual fee has already told us what it is -- Westpac's CARD FEE $295 and AMP's
|
||||
* Professional Pack Annual Fee each appear once and are certainly annual.
|
||||
* Deliberately narrow: bare "fee" would readmit every per-transaction charge.
|
||||
*/
|
||||
const ANNUAL_WORDING =
|
||||
"(annual|yearly)[[:space:]]*(card[[:space:]]*)?(fee|charge)" +
|
||||
"|card[[:space:]]+fee" +
|
||||
"|(package|membership|facility)[[:space:]]+fee";
|
||||
|
||||
/**
|
||||
* Resolve every statement to a canonical account identity, following
|
||||
* account_links forward. One card can appear under two bank names and two
|
||||
* numbers after an issuer change (Citi Prestige -> NAB MyCard); without this the
|
||||
* $700 fee charged 2024-12-13 and 2025-11-13 reads as two dead one-offs.
|
||||
*
|
||||
* Recursive because the table permits a chain (A->B->C); `uq_account_links_from`
|
||||
* forbids a fork, and `account_links_no_self` a self-link, so the walk
|
||||
* terminates. The depth guard is belt-and-braces against a cycle someone inserts
|
||||
* by hand (A->B, B->A), which no constraint can catch.
|
||||
*/
|
||||
const RESOLVED_ACCOUNTS = `
|
||||
WITH RECURSIVE chain AS (
|
||||
SELECT s.id AS statement_id, s.bank_name, s.account_number,
|
||||
s.bank_name AS cur_bank, s.account_number_key AS cur_key, 0 AS depth
|
||||
FROM statements s
|
||||
UNION ALL
|
||||
SELECT c.statement_id, c.bank_name, c.account_number,
|
||||
l.to_bank_name, l.to_account_key, c.depth + 1
|
||||
FROM chain c
|
||||
JOIN account_links l
|
||||
ON l.from_bank_name = c.cur_bank AND l.from_account_key = c.cur_key
|
||||
WHERE c.depth < 10
|
||||
),
|
||||
resolved AS (
|
||||
SELECT DISTINCT ON (statement_id)
|
||||
statement_id, bank_name, account_number, cur_bank AS acct_bank, cur_key AS acct_key
|
||||
FROM chain ORDER BY statement_id, depth DESC
|
||||
)`;
|
||||
|
||||
export async function getUpcomingAnnualCharges(ownerId: number): Promise<AnnualCharge[]> {
|
||||
// Signal 1 -- the bank's own fee wording, grouped by resolved account identity
|
||||
// and by the fee's wording so two different fees on one card stay apart
|
||||
// (ANZ charges "ANNUAL FEE" $120 and "ADDITIONAL CARD FEE" $10 on the same
|
||||
// statement, and they are not the same obligation).
|
||||
const feeRows = await queryRaw<AnnualCharge>(
|
||||
`${RESOLVED_ACCOUNTS},
|
||||
fees AS (
|
||||
SELECT r.acct_bank, r.acct_key, r.bank_name, r.account_number,
|
||||
upper(trim(regexp_replace(t.description, '[[:space:]]+', ' ', 'g'))) AS label,
|
||||
t.transaction_date, t.amount
|
||||
FROM transactions t
|
||||
JOIN resolved r ON r.statement_id = t.statement_id
|
||||
JOIN statements s ON s.id = t.statement_id
|
||||
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
|
||||
AND t.amount > 0
|
||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||
-- Structural first, wording only as a widening: a bank that types a
|
||||
-- fee as a plain debit still gets picked up if it names it annual.
|
||||
AND (t.transaction_type = $3 OR t.description ~* $2)
|
||||
),
|
||||
gapped AS (
|
||||
SELECT f.*,
|
||||
transaction_date - lag(transaction_date)
|
||||
OVER (PARTITION BY acct_key, label ORDER BY transaction_date) AS gap
|
||||
FROM fees f
|
||||
),
|
||||
agg AS (
|
||||
SELECT acct_key, acct_bank, label,
|
||||
count(*)::int AS observations,
|
||||
max(transaction_date) AS last_charged,
|
||||
min(amount) AS amount_low, max(amount) AS amount_high,
|
||||
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
|
||||
FILTER (WHERE gap IS NOT NULL) AS median_gap,
|
||||
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
|
||||
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
|
||||
(array_agg(bank_name ORDER BY transaction_date DESC))[1] AS bank_name,
|
||||
(array_agg(account_number ORDER BY transaction_date DESC))[1] AS account_number,
|
||||
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
|
||||
FROM gapped GROUP BY acct_key, acct_bank, label
|
||||
),
|
||||
liveness AS (
|
||||
SELECT r.acct_key, max(s.billing_end_date) AS last_statement
|
||||
FROM statements s JOIN resolved r ON r.statement_id = s.id
|
||||
GROUP BY r.acct_key
|
||||
)
|
||||
SELECT a.label,
|
||||
-- The account's CURRENT identity, not the bank that happened to
|
||||
-- print the last fee. That $700 fee last appeared under "Citibank"
|
||||
-- in Nov 2025, but the account is NAB/MyCard now and the next one
|
||||
-- will come from there; naming the dead brand would send you to the
|
||||
-- wrong bank.
|
||||
a.acct_bank AS bank_name, a.account_number,
|
||||
CASE WHEN (a.last_charged + COALESCE(round(a.median_gap)::int, 365)) >= CURRENT_DATE
|
||||
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
|
||||
-- One observation cannot prove a cadence, but the bank naming it an
|
||||
-- annual fee already tells us the kind. Repeated on an annual gap it
|
||||
-- is confirmed; seen once it is likely and says so.
|
||||
CASE WHEN a.gaps > 0 AND a.annual_gaps = a.gaps THEN 'confirmed'
|
||||
ELSE 'likely' END AS confidence,
|
||||
a.observations,
|
||||
a.last_charged::text AS last_charged,
|
||||
a.last_amount::text AS last_amount,
|
||||
a.amount_low::text AS amount_low,
|
||||
a.amount_high::text AS amount_high,
|
||||
round(a.median_gap)::int AS median_gap_days,
|
||||
(a.last_charged + COALESCE(round(a.median_gap)::int, 365))::text AS next_due_estimate,
|
||||
l.last_statement::text AS account_last_statement,
|
||||
(CURRENT_DATE - l.last_statement)::int AS account_days_stale
|
||||
FROM agg a LEFT JOIN liveness l ON l.acct_key = a.acct_key
|
||||
-- Two ways to earn a place, and nothing else gets one. Filtered here, not
|
||||
-- in the UI, so the API can never be read wrongly by a future caller.
|
||||
WHERE
|
||||
-- Repeated on an annual cadence. Excludes ANZ's monthly ACCOUNT
|
||||
-- SERVICING FEE and Zip's MONTHLY ACCOUNT SERVICE FEE, whose gaps are
|
||||
-- ~30 days, and Citibank's LATE PAYMENT FEE, which is event-driven.
|
||||
(a.gaps > 0 AND a.annual_gaps = a.gaps AND a.median_gap >= 300)
|
||||
-- Or seen once, but named as an annual/card/package fee. Westpac's CARD
|
||||
-- FEE and AMP's Professional Pack Annual Fee are each a single charge and
|
||||
-- unmistakably annual. A Wise per-transfer fee is also a single charge
|
||||
-- and says nothing of the sort, which is what keeps it out.
|
||||
OR (a.gaps = 0 AND a.label ~* $2)`,
|
||||
[ownerId, ANNUAL_WORDING, FEE_TYPE]
|
||||
);
|
||||
|
||||
// Signal 2 -- merchant-charged annual items with no fee wording. Requires
|
||||
// BOTH that every gap is annual AND that the amount is stable: VicRoads rego
|
||||
// rises about 3% a year ($825.23 -> $852.88 -> $875.48), so equality is too
|
||||
// strict, while no amount test at all readmits the twice-visited restaurant.
|
||||
// Keyed on merchant, never on card -- that rego was paid on three different
|
||||
// banks, and a card-keyed view would see three unrelated one-offs.
|
||||
const merchantRows = await queryRaw<AnnualCharge>(
|
||||
`WITH t AS (
|
||||
SELECT upper(trim(COALESCE(t.merchant_normalized, t.merchant_name, t.description))) AS label,
|
||||
t.transaction_date, t.amount
|
||||
FROM transactions t
|
||||
LEFT JOIN statements s ON s.id = t.statement_id
|
||||
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
|
||||
AND t.voided_at IS NULL
|
||||
AND t.transaction_type = 'debit' AND t.amount > 0
|
||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||
AND t.transaction_type = 'debit' AND t.description !~* $2
|
||||
),
|
||||
gapped AS (
|
||||
SELECT t.*, transaction_date - lag(transaction_date)
|
||||
OVER (PARTITION BY label ORDER BY transaction_date) AS gap
|
||||
FROM t
|
||||
),
|
||||
agg AS (
|
||||
SELECT label, count(*)::int AS observations,
|
||||
max(transaction_date) AS last_charged,
|
||||
min(amount) AS amount_low, max(amount) AS amount_high,
|
||||
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
|
||||
FILTER (WHERE gap IS NOT NULL) AS median_gap,
|
||||
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
|
||||
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
|
||||
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
|
||||
FROM gapped GROUP BY label
|
||||
)
|
||||
SELECT label, NULL::text AS bank_name, NULL::text AS account_number,
|
||||
CASE WHEN (last_charged + round(median_gap)::int) >= CURRENT_DATE
|
||||
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
|
||||
'possible' AS confidence, observations,
|
||||
last_charged::text AS last_charged,
|
||||
last_amount::text AS last_amount,
|
||||
amount_low::text AS amount_low,
|
||||
amount_high::text AS amount_high,
|
||||
round(median_gap)::int AS median_gap_days,
|
||||
(last_charged + round(median_gap)::int)::text AS next_due_estimate,
|
||||
NULL::text AS account_last_statement,
|
||||
NULL::int AS account_days_stale
|
||||
FROM agg
|
||||
WHERE gaps >= 1 AND annual_gaps = gaps
|
||||
-- Amount stability is what separates a subscription from a restaurant.
|
||||
-- 20% absorbs CPI drift and an FX-priced renewal without admitting a
|
||||
-- $30-then-$198 pair.
|
||||
AND amount_high <= amount_low * 1.20
|
||||
-- Two observations is the floor for a cadence claim, and this lane has no
|
||||
-- wording to fall back on the way the fee lane does.
|
||||
AND observations >= 2`,
|
||||
[ownerId, ANNUAL_WORDING]
|
||||
);
|
||||
|
||||
// Upcoming first and soonest-first within that, so the thing you can still
|
||||
// act on is at the top; the expected-but-missing rows follow as a review list.
|
||||
const rank = (c: AnnualCharge) => (c.status === "upcoming" ? 0 : 1);
|
||||
return [...feeRows, ...merchantRows].sort(
|
||||
(a, b) => rank(a) - rank(b) || a.next_due_estimate.localeCompare(b.next_due_estimate)
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,235 @@
|
||||
import { queryRaw, type Queryable } from "@/lib/db";
|
||||
import { merchantMatches } from "@/lib/intake-rules";
|
||||
|
||||
/**
|
||||
* Unambiguous statement matches for statementless ("manual") transactions.
|
||||
*
|
||||
* WHY THIS EXISTS
|
||||
* ---------------
|
||||
* The reconcile queue is worked by hand and is already ~177 deep. The wallet
|
||||
* lane writes a transaction for every tap (DECISIONS.md ING-11, 2026-08-22),
|
||||
* which at the observed rate adds ~63 rows a month, ~767 a year. Without an
|
||||
* automatic pass for the easy cases that queue stops being workable, and the
|
||||
* tempting alternative -- filtering the new rows back out of it -- is
|
||||
* explicitly ruled out by the `awaitsStatementLine()` removal note in
|
||||
* queries.ts: "an imported row that has no statement belongs in the queue;
|
||||
* volume is solved upstream, never downstream by hiding what is there."
|
||||
*
|
||||
* WHAT COUNTS AS UNAMBIGUOUS
|
||||
* --------------------------
|
||||
* Deliberately much stricter than the queue's own candidate ranking, which
|
||||
* offers the top 5 within +/-1% and +/-3 days for a human to choose between.
|
||||
* A machine gets one shot and no judgement, so:
|
||||
*
|
||||
* - EXACT amount equality, not the queue's +/-1% band. A near-miss is exactly
|
||||
* the case a person should look at.
|
||||
* - Same owner, and the statement row must not already be reconciled against.
|
||||
* - UNIQUE BOTH WAYS: the manual row must have exactly one candidate, AND
|
||||
* that candidate must be claimed by exactly one manual row. This is the
|
||||
* order_transaction_bridge rule, and it is what stops two identical $13.50
|
||||
* bakery taps in one window from being paired arbitrarily.
|
||||
*
|
||||
* Foreign rows match on the NATIVE pair (foreign_currency_amount +
|
||||
* foreign_currency_code), never on `amount`: `amount` is the AUD the bank
|
||||
* charged after applying its own rate, so a wallet row cannot predict it, while
|
||||
* the native figure is exact on both sides. Same reasoning as
|
||||
* order_transaction_bridge's FX path.
|
||||
*
|
||||
* This function only PROPOSES. Applying a pair goes through the same merge the
|
||||
* manual route uses, so overrides/tags/splits/metadata move identically.
|
||||
*/
|
||||
|
||||
export type AutoMatch = {
|
||||
manual_id: number;
|
||||
statement_tx_id: number;
|
||||
amount: string;
|
||||
manual_date: string;
|
||||
statement_date: string;
|
||||
description: string;
|
||||
basis: "aud" | "native";
|
||||
};
|
||||
|
||||
/** Days either side of the manual row's date to consider. Authorisation and
|
||||
* settlement are rarely the same day; 3 matches the queue's own window and
|
||||
* the wallet reconciliation check's default. */
|
||||
export const AUTO_MATCH_WINDOW_DAYS = 3;
|
||||
|
||||
export async function findAutoMatches(
|
||||
ownerId: number,
|
||||
windowDays: number = AUTO_MATCH_WINDOW_DAYS
|
||||
): Promise<AutoMatch[]> {
|
||||
return queryRaw<AutoMatch>(
|
||||
`WITH pairs AS (
|
||||
SELECT m.id AS manual_id,
|
||||
t.id AS statement_tx_id,
|
||||
t.amount::text AS amount,
|
||||
m.transaction_date::text AS manual_date,
|
||||
t.transaction_date::text AS statement_date,
|
||||
t.description AS description,
|
||||
CASE WHEN m.foreign_currency_code IS NOT NULL
|
||||
AND m.foreign_currency_amount IS NOT NULL
|
||||
THEN 'native' ELSE 'aud' END AS basis
|
||||
FROM transactions m
|
||||
JOIN transactions t
|
||||
ON t.statement_id IS NOT NULL
|
||||
AND t.transaction_type = 'debit'
|
||||
AND t.transaction_date BETWEEN m.transaction_date - $2::int
|
||||
AND m.transaction_date + $2::int
|
||||
AND (
|
||||
-- Foreign: compare what both sides actually recorded natively.
|
||||
(m.foreign_currency_code IS NOT NULL
|
||||
AND m.foreign_currency_amount IS NOT NULL
|
||||
AND t.foreign_currency_code = m.foreign_currency_code
|
||||
AND t.foreign_currency_amount = m.foreign_currency_amount)
|
||||
OR -- AUD side: exact equality, never a tolerance band. The
|
||||
-- statement row MAY be foreign here, and that is not a mistake:
|
||||
-- all 48 reconciliations a human has made on this ledger are
|
||||
-- exactly this shape -- a manual row carrying no currency,
|
||||
-- matched on exact AUD against a statement row that does carry
|
||||
-- one (EUR, CHF). Overseas receipts and orders record what was
|
||||
-- charged in AUD; only the wallet lane records natively, which
|
||||
-- is why the branch above exists. Requiring both sides to be
|
||||
-- non-foreign was tried and rejected: it matched 0 of those 48.
|
||||
-- The coincidence risk (a domestic tap whose amount happens to
|
||||
-- equal a foreign charge's AUD figure) is already covered by
|
||||
-- unique-both-ways, which refuses as soon as a second candidate
|
||||
-- exists.
|
||||
(m.foreign_currency_code IS NULL
|
||||
AND t.amount = m.amount)
|
||||
)
|
||||
JOIN statements s ON s.id = t.statement_id
|
||||
WHERE m.statement_id IS NULL
|
||||
AND m.owner_id = $1
|
||||
AND m.reconciled_with_id IS NULL
|
||||
AND m.superseded_by_id IS NULL
|
||||
AND m.voided_at IS NULL
|
||||
AND (m.payment_method IS NULL OR m.payment_method NOT IN ('cash', 'credits'))
|
||||
AND t.superseded_by_id IS NULL
|
||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||
-- The statement row must be unclaimed.
|
||||
AND NOT EXISTS (SELECT 1 FROM transactions mt WHERE mt.reconciled_with_id = t.id)
|
||||
)
|
||||
SELECT manual_id, statement_tx_id, amount, manual_date, statement_date, description, basis
|
||||
FROM pairs p
|
||||
WHERE (SELECT count(*) FROM pairs q WHERE q.manual_id = p.manual_id) = 1
|
||||
AND (SELECT count(*) FROM pairs q WHERE q.statement_tx_id = p.statement_tx_id) = 1
|
||||
ORDER BY manual_id`,
|
||||
[ownerId, windowDays]
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* An existing statementless transaction that a newly-arriving receipt or order
|
||||
* describes -- i.e. the row the wallet lane already created for this purchase.
|
||||
*
|
||||
* WHY: signals for one purchase arrive in sequence, not in competition. The
|
||||
* wallet notification lands within seconds of the tap; the receipt is shared
|
||||
* hours or days later. Measured on txn 6858: the wallet event arrived 35h25m
|
||||
* before anything created a transaction, and pantry then created one from
|
||||
* scratch -- so the same Woolworths shop would have had two rows. The receipt
|
||||
* must adopt the row that already exists and enrich it.
|
||||
*
|
||||
* Deliberately narrow, for the same reason findAutoMatches() is: exact amount,
|
||||
* a few days either side, unclaimed, and UNIQUE. Two candidates means two
|
||||
* plausible purchases, and inventing a link between the wrong pair moves one
|
||||
* shop's line items onto another shop's row. Returning null is always safe --
|
||||
* the caller falls back to creating its own row, which is today's behaviour.
|
||||
*/
|
||||
/**
|
||||
* The row a newly-arriving WALLET notification describes -- the mirror of
|
||||
* findAdoptableTransaction, for the case where the signals arrived in the other
|
||||
* order.
|
||||
*
|
||||
* WHY: the sequencing this estate is built on -- wallet first, receipt second,
|
||||
* statement last -- is a tendency, not a guarantee. The wallet lane can be
|
||||
* parked (its first 21 rows were backfilled days late), the interpreter runs
|
||||
* every 15 minutes, the phone can be offline. Whenever a receipt reaches the
|
||||
* ledger before the notification for the same tap, the wallet writer used to
|
||||
* insert unconditionally: measured on the 2026-08-18 Woolworths shop, txn 6858
|
||||
* (pantry, 23 items) and txn 6927 (wallet) were the same $103.30 purchase, and
|
||||
* the second sat in the reconcile queue waiting for a statement line that the
|
||||
* first had already claimed.
|
||||
*
|
||||
* Deliberately NOT the same predicate as findAdoptableTransaction, in the one
|
||||
* way that matters: that one refuses a row which already carries line items,
|
||||
* because for a receipt such a row is a DIFFERENT purchase that happens to
|
||||
* share an amount. Here it is the opposite -- an itemised row is the receipt we
|
||||
* are trying to fold into, and refusing it is the bug.
|
||||
*
|
||||
* Losing that guard means the merchant has to supply it. A wallet row adopts
|
||||
* only when the merchant agrees (the same whole-word containment the intake
|
||||
* matcher uses: "Woolworths" vs "WOOLWORTHS 3345"), because amount-and-date
|
||||
* alone would let a notification silently disappear into an unrelated purchase
|
||||
* of the same value -- and a wallet row that is never written is a tap with no
|
||||
* trace, which is worse than a visible duplicate. When in doubt, return null:
|
||||
* the caller then writes its own row, which is the pre-existing behaviour.
|
||||
*/
|
||||
export async function findWalletAdoptableTransaction(
|
||||
ownerId: number,
|
||||
amount: number,
|
||||
date: string,
|
||||
merchant: string | null,
|
||||
windowDays: number = AUTO_MATCH_WINDOW_DAYS,
|
||||
// wallet-ingestion calls this while holding its advisory lock, and must pass
|
||||
// that transaction's connection: borrowing a second pooled one per request
|
||||
// is how a burst of concurrent posts exhausts the pool and deadlocks.
|
||||
db: Pick<Queryable, "queryRaw"> = { queryRaw }
|
||||
): Promise<{ id: number; metadata_id: number | null } | null> {
|
||||
// No usable merchant, no adoption. "Card purchase" is wallet-ingestion's own
|
||||
// fallback for a notification that named no merchant at all.
|
||||
if (!merchant || merchant === "Card purchase") return null;
|
||||
const rows = await db.queryRaw<{ id: number; metadata_id: number | null; merchant: string | null }>(
|
||||
`SELECT t.id, em.id AS metadata_id,
|
||||
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant
|
||||
FROM transactions t
|
||||
LEFT JOIN expense_metadata em ON em.transaction_id = t.id
|
||||
WHERE t.statement_id IS NULL
|
||||
AND t.owner_id = $1
|
||||
AND t.amount = $2::numeric
|
||||
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
|
||||
AND t.reconciled_with_id IS NULL
|
||||
AND t.superseded_by_id IS NULL
|
||||
-- A voided row is a charge that never posted; a real charge for the same
|
||||
-- amount (a rebooking after a released hold) must not fold into it.
|
||||
AND t.voided_at IS NULL
|
||||
-- A wallet row is the OTHER HALF of this tap, not a purchase to fold
|
||||
-- into; findPairedRow() merges those, with different rules.
|
||||
AND COALESCE(t.source, '') <> 'wallet'
|
||||
LIMIT 5`,
|
||||
[ownerId, amount, date, windowDays]
|
||||
);
|
||||
const hits = rows.filter((r) => merchantMatches(r.merchant, merchant));
|
||||
return hits.length === 1 ? { id: hits[0].id, metadata_id: hits[0].metadata_id } : null;
|
||||
}
|
||||
|
||||
export async function findAdoptableTransaction(
|
||||
ownerId: number,
|
||||
amount: number,
|
||||
date: string,
|
||||
windowDays: number = AUTO_MATCH_WINDOW_DAYS
|
||||
): Promise<number | null> {
|
||||
const rows = await queryRaw<{ id: number }>(
|
||||
`SELECT t.id
|
||||
FROM transactions t
|
||||
WHERE t.statement_id IS NULL
|
||||
AND t.owner_id = $1
|
||||
AND t.amount = $2::numeric
|
||||
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
|
||||
AND t.reconciled_with_id IS NULL
|
||||
AND t.superseded_by_id IS NULL
|
||||
-- A voided row is a charge that never posted; a real charge for the same
|
||||
-- amount (a rebooking after a released hold) must not fold into it.
|
||||
AND t.voided_at IS NULL
|
||||
-- Never adopt a row that another source already fully described; that
|
||||
-- is a different purchase that happens to share an amount.
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM expense_metadata em
|
||||
WHERE (em.transaction_id = t.id OR em.matched_transaction_id = t.id)
|
||||
AND jsonb_typeof(em.line_items) = 'array'
|
||||
AND jsonb_array_length(em.line_items) > 0
|
||||
)
|
||||
LIMIT 2`,
|
||||
[ownerId, amount, date, windowDays]
|
||||
);
|
||||
return rows.length === 1 ? rows[0].id : null;
|
||||
}
|
||||
+7
-2
@@ -38,7 +38,12 @@ export type Queryable = {
|
||||
* next statement runs. Anything that needs a lock to still be held on its next
|
||||
* query has to be in here.
|
||||
*/
|
||||
export async function withTransaction<T>(fn: (q: Queryable) => Promise<T>): Promise<T> {
|
||||
export async function withTransaction<T>(
|
||||
fn: (q: Queryable) => Promise<T>,
|
||||
// Prisma's defaults (maxWait 2s, timeout 5s) count time spent WAITING on an
|
||||
// advisory lock, so a caller that serialises a burst on one needs more.
|
||||
opts?: { maxWait?: number; timeout?: number }
|
||||
): Promise<T> {
|
||||
return prisma.$transaction(async (tx) => {
|
||||
const q: Queryable = {
|
||||
queryRaw: <R>(sql: string, params: unknown[] = []) => tx.$queryRawUnsafe<R[]>(sql, ...params),
|
||||
@@ -48,5 +53,5 @@ export async function withTransaction<T>(fn: (q: Queryable) => Promise<T>): Prom
|
||||
},
|
||||
};
|
||||
return fn(q);
|
||||
});
|
||||
}, opts);
|
||||
}
|
||||
|
||||
@@ -177,6 +177,40 @@ export function useUpdateTransaction() {
|
||||
});
|
||||
}
|
||||
|
||||
export function useMergeTransaction() {
|
||||
const qc = useQueryClient();
|
||||
return useMutation({
|
||||
mutationFn: async ({
|
||||
id,
|
||||
other_id,
|
||||
mode,
|
||||
}: {
|
||||
id: number;
|
||||
other_id: number;
|
||||
mode: "reconcile" | "supersede";
|
||||
}) => {
|
||||
const res = await fetch(`/api/transactions/${id}/merge`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ other_id, mode }),
|
||||
});
|
||||
if (!res.ok) {
|
||||
const body = await res.json().catch(() => ({}));
|
||||
throw new Error(body.error || "Failed to merge transaction");
|
||||
}
|
||||
return res.json();
|
||||
},
|
||||
onSuccess: () => {
|
||||
qc.invalidateQueries({ queryKey: ["transactions"] });
|
||||
qc.invalidateQueries({ queryKey: ["transaction"] });
|
||||
qc.invalidateQueries({ queryKey: ["splits"] });
|
||||
qc.invalidateQueries({ queryKey: ["analytics"] });
|
||||
qc.invalidateQueries({ queryKey: ["participant-balances"] });
|
||||
qc.invalidateQueries({ queryKey: ["shared-transactions"] });
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function useBulkAction() {
|
||||
const qc = useQueryClient();
|
||||
return useMutation({
|
||||
@@ -1128,6 +1162,27 @@ export function useReconcile() {
|
||||
});
|
||||
}
|
||||
|
||||
/** Mark a statementless row as a charge that never posted (migration 0040). */
|
||||
export function useVoidTransaction() {
|
||||
const qc = useQueryClient();
|
||||
return useMutation({
|
||||
mutationFn: async ({ id, reason }: { id: number; reason: string }) => {
|
||||
const res = await fetch(`/api/transactions/${id}/void`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ reason }),
|
||||
});
|
||||
if (!res.ok) throw new Error((await res.json()).error || "Void failed");
|
||||
return res.json();
|
||||
},
|
||||
onSuccess: () => {
|
||||
qc.invalidateQueries({ queryKey: ["transactions"] });
|
||||
qc.invalidateQueries({ queryKey: ["reconcile-pending"] });
|
||||
qc.invalidateQueries({ queryKey: ["analytics"] });
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function useMerchantTransactions(merchant: string | null) {
|
||||
return useQuery<{ transactions: MerchantTxnRow[] }>({
|
||||
queryKey: ["analytics", "merchant-txns", merchant],
|
||||
|
||||
@@ -120,13 +120,51 @@ export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks:
|
||||
return { blocks, text };
|
||||
}
|
||||
|
||||
export function forwardedCard(c: CardInput & { captureId: number; enriching: boolean }): { blocks: Block[]; text: string } {
|
||||
const text = `${esc(c.merchant) || "Grocery receipt"} ${c.total !== null ? money(c.total, c.currency) : ""} → pantry board (${c.itemCount} item${c.itemCount === 1 ? "" : "s"}) — finance ${c.enriching ? "updates" : "books it"} on confirm`.replace(/\s+/g, " ");
|
||||
export function forwardedCard(
|
||||
c: CardInput & {
|
||||
captureId: number;
|
||||
enriching: boolean;
|
||||
/** Pantry already holds this exact receipt (same barcode tuple). */
|
||||
alreadyCaptured?: { captureId: number; confirmed: boolean } | null;
|
||||
/** Finance found more than one row this could be, so it picked none. */
|
||||
ambiguous?: boolean;
|
||||
}
|
||||
): { blocks: Block[]; text: string } {
|
||||
const what = `${esc(c.merchant) || "Grocery receipt"} ${c.total !== null ? money(c.total, c.currency) : ""}`.trim();
|
||||
const items = `${c.itemCount} item${c.itemCount === 1 ? "" : "s"}`;
|
||||
// The warning comes FIRST and the emoji changes, because this card is read at
|
||||
// a glance on a phone: a duplicate warning appended to a cheerful "→ pantry
|
||||
// board" line is a duplicate warning nobody reads. Confirming both copies
|
||||
// counts the same groceries twice — pantry's addItem accumulates — so the
|
||||
// card has to say which capture to dismiss, not merely that there are two.
|
||||
if (c.alreadyCaptured) {
|
||||
const state = c.alreadyCaptured.confirmed ? "already confirmed" : "still waiting to be confirmed";
|
||||
const text = `Already on the pantry board — ${what} is capture ${c.alreadyCaptured.captureId} (${state}). This upload is capture ${c.captureId}; dismiss it unless it replaces the first, because confirming both counts the same groceries twice.`.replace(/\s+/g, " ");
|
||||
return { text, blocks: [section(`:repeat: ${text}`)] };
|
||||
}
|
||||
const tail = c.ambiguous
|
||||
? "— finance found more than one row it could be, so it picked none; reconcile it there"
|
||||
: `— finance ${c.enriching ? "updates" : "books it"} on confirm`;
|
||||
const text = `${what} → pantry board (${items}) ${tail}`.replace(/\s+/g, " ");
|
||||
return { text, blocks: [section(`:shopping_trolley: ${text}`)] };
|
||||
}
|
||||
|
||||
export function needsHeaderCard(): { blocks: Block[]; text: string } {
|
||||
const text = "I can see items but no order total or date — share the top of the screen too so I can match it.";
|
||||
/**
|
||||
* Names what is missing, because the advice differs. A delivery-app screenshot
|
||||
* that lost its header needs the top of the screen; a Woolworths eReceipt
|
||||
* SCREENSHOT has no date anywhere on it (seen 2026-08-22 — store, total, items,
|
||||
* payment, and not one date), while the app's own Share button sends the raw
|
||||
* receipt WITH the date. So for a dateless in-store receipt the right advice is
|
||||
* the Share button, or pantry's own upload — not "share more of the screen".
|
||||
*/
|
||||
export function needsHeaderCard(missing: string[] = ["total", "date"], source?: string | null): { blocks: Block[]; text: string } {
|
||||
const what = missing.length ? missing.join(" or ") : "total or date";
|
||||
const inStore = source === "coles" || source === "woolworths";
|
||||
const hint =
|
||||
missing.length === 1 && missing[0] === "date" && inStore
|
||||
? "an app screenshot carries no date — use the app's Share button instead, or upload the picture to pantry directly"
|
||||
: "share the part of the screen that shows it";
|
||||
const text = `I can see items but no ${esc(what)} — ${hint}, so I can match it.`;
|
||||
return { text, blocks: [section(`:warning: ${text}`)] };
|
||||
}
|
||||
|
||||
@@ -155,7 +193,12 @@ export function askCard(c: CardInput): { blocks: Block[]; text: string } {
|
||||
|
||||
export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null; currency?: string | null }): { blocks: Block[]; text: string } {
|
||||
const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${esc(c.batchId)}` : "an earlier upload";
|
||||
const text = `Already handled — ${esc(c.merchant) || "this order"} ${c.total !== null ? money(c.total, c.currency) : ""} is ${where}. Nothing was changed.`.replace(/\s+/g, " ");
|
||||
// The header (merchant, date, total) IS the identity of a purchase here, so a
|
||||
// genuine SECOND identical purchase — two of the same coffee, one day, one shop
|
||||
// — reads as a re-share and is refused. That is the right default (a re-share is
|
||||
// far commoner than a repeat) but it is a dead end without this sentence: the
|
||||
// card must say what to do instead, or the ledger quietly loses the second one.
|
||||
const text = `Already handled — ${esc(c.merchant) || "this order"} ${c.total !== null ? money(c.total, c.currency) : ""} is ${where}. Nothing was changed — if this really is a second, separate purchase, add it in the app.`.replace(/\s+/g, " ");
|
||||
return { text, blocks: [section(`:repeat: ${text}`)] };
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { extractText, getDocumentProxy } from "unpdf";
|
||||
import type { ExtractedOrder, ExtractedItem } from "@/lib/intake-rules";
|
||||
import { PROMPT_PLACEHOLDER_MERCHANTS, type ExtractedOrder, type ExtractedItem } from "@/lib/intake-rules";
|
||||
|
||||
/**
|
||||
* Extraction for the intake router: one joint model call per batch.
|
||||
@@ -31,13 +31,22 @@ export async function extractPdfText(bytes: Buffer): Promise<string> {
|
||||
return (Array.isArray(text) ? text.join("\n") : text).trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* The merchant examples are FORMAT PLACEHOLDERS, never real shops
|
||||
* (`PROMPT_PLACEHOLDER_MERCHANTS`). Given an input with nothing in it, a model
|
||||
* still answers, and it answers out of its prompt: on 2026-08-22 the word
|
||||
* `Image` produced a complete Burgertory receipt because the prompt named
|
||||
* Burgertory. The "not a receipt" rule below and the placeholder names are the
|
||||
* two halves of closing that.
|
||||
*/
|
||||
export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text.
|
||||
|
||||
Return ONE JSON object with exactly these keys:
|
||||
{ "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] }
|
||||
|
||||
Rules:
|
||||
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (e.g. "Coles (Wyndham Vale)", "Burgertory (Werribee)"). Never the delivery platform. null if not shown.
|
||||
- If what you are given is NOT a receipt or an order — it is blank, a filename, a single word such as "Image", an app screen with no purchase on it, or anything else you cannot read a purchase off — return { "merchant": null, "order_date": null, "total": null, "subtotal": null, "currency": null, "items": [], "adjustments": [] }. Do not construct a plausible order. Nothing in these instructions is data: never return an example from this prompt as if it were printed on the input.
|
||||
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (format: "${PROMPT_PLACEHOLDER_MERCHANTS[0]}"). Never the delivery platform. null if not shown.
|
||||
- order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown.
|
||||
- total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown.
|
||||
- subtotal: the printed item subtotal before fees and discounts, as a number, or null.
|
||||
|
||||
@@ -20,7 +20,21 @@ export type PantryForwardArgs = {
|
||||
financeAction: "enrich" | "create";
|
||||
};
|
||||
|
||||
export type PantryForwardResult = { captureId: number; itemCount: number | null };
|
||||
/**
|
||||
* `alreadyCaptured` is pantry telling us this exact receipt is already on its
|
||||
* board -- same store, register, receipt number and date, which is the tuple
|
||||
* both Coles and Woolworths encode in their barcode. Pantry deliberately still
|
||||
* ACCEPTS the second capture (the first may have been a bad crop being
|
||||
* replaced) and says so instead of refusing; dropping the field on the floor
|
||||
* turned "said so" into silence, and a duplicate that is not shown is a
|
||||
* duplicate that gets confirmed -- which double-counts the groceries, because
|
||||
* pantry's addItem accumulates.
|
||||
*/
|
||||
export type PantryForwardResult = {
|
||||
captureId: number;
|
||||
itemCount: number | null;
|
||||
alreadyCaptured: { captureId: number; confirmed: boolean } | null;
|
||||
};
|
||||
|
||||
export class PantryDuplicateError extends Error {
|
||||
constructor(
|
||||
@@ -65,12 +79,23 @@ export function pantryForwarder(opts: { baseUrl?: string; token?: string; fetchI
|
||||
} catch (e) {
|
||||
throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
const body = (await response.json().catch(() => ({}))) as { captureId?: number; error?: string; items?: unknown[]; reason?: string };
|
||||
const body = (await response.json().catch(() => ({}))) as {
|
||||
captureId?: number; error?: string; items?: unknown[]; reason?: string;
|
||||
alreadyCaptured?: { captureId?: number; confirmed?: boolean } | null;
|
||||
};
|
||||
if (response.status === 409) {
|
||||
throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured");
|
||||
}
|
||||
if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim());
|
||||
if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId");
|
||||
return { captureId: body.captureId, itemCount: Array.isArray(body.items) ? body.items.length : null };
|
||||
const prior = body.alreadyCaptured;
|
||||
return {
|
||||
captureId: body.captureId,
|
||||
itemCount: Array.isArray(body.items) ? body.items.length : null,
|
||||
alreadyCaptured:
|
||||
prior && typeof prior.captureId === "number" && prior.captureId !== body.captureId
|
||||
? { captureId: prior.captureId, confirmed: prior.confirmed === true }
|
||||
: null,
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
@@ -2,9 +2,13 @@ import { randomUUID } from "node:crypto";
|
||||
import { queryRaw, queryRow, withTransaction } from "@/lib/db";
|
||||
import { EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY } from "@/lib/analytics-sql";
|
||||
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
||||
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
|
||||
import {
|
||||
classifyInput,
|
||||
confabulationReason,
|
||||
merchantKey,
|
||||
quoteShared,
|
||||
textCouldBeReceipt,
|
||||
merchantMatches,
|
||||
platformForSource,
|
||||
reconcileItems,
|
||||
@@ -373,7 +377,12 @@ export async function routeFood(ctx: RouteContext): Promise<IntakeResult> {
|
||||
|
||||
/** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */
|
||||
export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> {
|
||||
const { match } = await matchTransaction(ctx.header, ctx.platform);
|
||||
// `ambiguous` is not a miss. Two ledger rows for one shop is precisely the
|
||||
// state that produced the 2026-08-18 Woolworths double-book, and forwarding
|
||||
// with financeAction 'create' as though nothing matched is what turns it into
|
||||
// a third row on confirm. The groceries still go to the board — they have to,
|
||||
// it is where they are worked — but the card must say finance could not pick.
|
||||
const { match, ambiguous } = await matchTransaction(ctx.header, ctx.platform);
|
||||
try {
|
||||
const r = await pantry({
|
||||
files,
|
||||
@@ -384,7 +393,14 @@ export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantr
|
||||
financeAction: match ? "enrich" : "create",
|
||||
});
|
||||
await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items });
|
||||
const card = forwardedCard({ ...cardInput(ctx), itemCount: r.itemCount ?? ctx.items.length, captureId: r.captureId, enriching: !!match });
|
||||
const card = forwardedCard({
|
||||
...cardInput(ctx),
|
||||
itemCount: r.itemCount ?? ctx.items.length,
|
||||
captureId: r.captureId,
|
||||
enriching: !!match,
|
||||
alreadyCaptured: r.alreadyCaptured,
|
||||
ambiguous,
|
||||
});
|
||||
return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id });
|
||||
} catch (e) {
|
||||
if (e instanceof PantryDuplicateError) {
|
||||
@@ -444,6 +460,29 @@ export async function createFromBatch(batch: BatchRow, kind: "grocery" | "food")
|
||||
}
|
||||
|
||||
const ownerId = await ownerForEmail(batch.shared_by);
|
||||
|
||||
// Before minting a row: is one already here that nothing has described yet?
|
||||
//
|
||||
// matchTransaction above cannot see it. Its key includes the platform, and a
|
||||
// wallet notification has none — so an Uber Eats screenshot never matches the
|
||||
// wallet row for the very tap that paid for it, and the Create button books
|
||||
// the purchase a second time. That is the normal arrival order, not an edge
|
||||
// case: the notification lands within seconds of the tap and the screenshot
|
||||
// is shared later.
|
||||
//
|
||||
// findAdoptableTransaction is the same helper the receipt lane uses and
|
||||
// carries the same guards — exact amount, ±3 days, unclaimed, UNIQUE, and
|
||||
// never a row that already carries line items. Its uniqueness check runs
|
||||
// outside the advisory lock below, so two simultaneous taps could both adopt;
|
||||
// the second would then find the items in place and fall through to the lock,
|
||||
// which is where duplicates are actually resolved.
|
||||
const adopted = await findAdoptableTransaction(ownerId, header.total, header.order_date);
|
||||
if (adopted) {
|
||||
await replaceLineItems(adopted, items, { source: "intake", platform: header.platform, batchId: batch.id, currency: header.currency });
|
||||
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: adopted, items });
|
||||
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: adopted }), { transaction_id: adopted });
|
||||
}
|
||||
|
||||
const merchant = header.merchant ?? "Unknown merchant";
|
||||
// Not `orderDescription()`: it always prefixes "Order - ", which is right for
|
||||
// a delivery receipt and wrong for the in-store till receipt this path also
|
||||
@@ -548,6 +587,8 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
||||
// 1. classify each file and build the parts, in the order shared.
|
||||
const parts: InputPart[] = [];
|
||||
const usable: StoredFile[] = [];
|
||||
/** Text parts dropped before the model saw them — see `textCouldBeReceipt`. */
|
||||
const rejectedText: { name: string; text: string; reason: string }[] = [];
|
||||
for (const f of input.files) {
|
||||
const kind = classifyInput(f);
|
||||
if (kind === "pdf") {
|
||||
@@ -565,7 +606,20 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
||||
} else if (kind === "image") {
|
||||
parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name });
|
||||
} else if (kind === "text") {
|
||||
parts.push({ kind: "text", text: f.bytes.toString("utf-8"), name: f.name });
|
||||
// Nothing that cannot be a receipt reaches the model. A share that
|
||||
// arrives as the word "Image" (the phone coerced the photo to text) is
|
||||
// not a thin receipt, and asking anyway produces a whole invented one —
|
||||
// 2026-08-22, `textCouldBeReceipt`. Dropped like an unreadable file type
|
||||
// rather than failing the batch, so one junk text part beside a good
|
||||
// screenshot still goes through; if nothing usable is left the card below
|
||||
// quotes what the phone actually sent.
|
||||
const text = f.bytes.toString("utf-8");
|
||||
const check = textCouldBeReceipt(text);
|
||||
if (!check.ok) {
|
||||
rejectedText.push({ name: f.name, text, reason: check.reason });
|
||||
continue;
|
||||
}
|
||||
parts.push({ kind: "text", text, name: f.name });
|
||||
} else {
|
||||
continue;
|
||||
}
|
||||
@@ -573,7 +627,16 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
||||
}
|
||||
if (!parts.length) {
|
||||
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
|
||||
return fail("unsupported", "unsupported", unsupportedCard());
|
||||
const dropped = rejectedText[0];
|
||||
return fail(
|
||||
"unsupported",
|
||||
"unsupported",
|
||||
unsupportedCard(
|
||||
dropped
|
||||
? `the share arrived as text and ${dropped.reason}: "${quoteShared(dropped.text)}" — send the picture or the PDF itself`
|
||||
: undefined
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// A re-share of the very same files, before spending a model call on it.
|
||||
@@ -604,11 +667,32 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
||||
}
|
||||
const items = toLineItems(extracted.items);
|
||||
|
||||
// 2b. did it answer out of the upload, or out of its prompt?
|
||||
const confab = confabulationReason(
|
||||
{
|
||||
textParts: parts.filter((p): p is Extract<InputPart, { kind: "text" }> => p.kind === "text").map((p) => p.text),
|
||||
hasImages: parts.some((p) => p.kind === "image"),
|
||||
},
|
||||
extracted
|
||||
);
|
||||
if (confab) {
|
||||
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
|
||||
return fail("unsupported", "unsupported", unsupportedCard(confab));
|
||||
}
|
||||
|
||||
// 3. header.
|
||||
//
|
||||
// This path stores the partial header and the items on the batch row, which
|
||||
// is right for a genuine screenshot missing its top — the next tap has
|
||||
// something to work from. It is wrong for a confabulation, and on 2026-08-22
|
||||
// it stored one: an invented Burgertory header with two invented lines, held
|
||||
// back from the matcher only by an absent date. Nothing here could tell the
|
||||
// two apart, which is why the pre-check above and 2b now keep confabulations
|
||||
// from reaching this point at all rather than trying to filter them here.
|
||||
const headerCheck = validateHeader(extracted);
|
||||
if (!headerCheck.ok) {
|
||||
await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" });
|
||||
return fail("held", "needs_header", needsHeaderCard(), 200, { items });
|
||||
return fail("held", "needs_header", needsHeaderCard(headerCheck.missing, input.source), 200, { items });
|
||||
}
|
||||
const header = headerCheck.header;
|
||||
const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" };
|
||||
|
||||
@@ -250,3 +250,94 @@ export function merchantMatches(stored: string | null | undefined, extracted: st
|
||||
if (!a.length || !b.length) return false;
|
||||
return containsWordRun(a, b) || containsWordRun(b, a);
|
||||
}
|
||||
|
||||
// ─── is this text even a receipt? (2026-08-22) ───────────────────────────────
|
||||
|
||||
/**
|
||||
* The merchant names the extraction prompt uses to show the FORMAT. They are
|
||||
* placeholders rather than real shops on purpose.
|
||||
*
|
||||
* On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so the intake
|
||||
* router received one text part whose entire content was the five characters
|
||||
* `Image`. The model was handed that, and returned a whole receipt:
|
||||
* `Burgertory (Braybrook)`, two "The Pride" lines at $22.80, a $41.15 total.
|
||||
* Nothing on the receipt existed — "Burgertory" was in the prompt, as the
|
||||
* example of how to print a branch in parentheses. Only the absent date kept it
|
||||
* out of the matcher; the invented header and items were stored on the batch.
|
||||
*
|
||||
* Same shape as the ingestion engine's 2026-08-05 empty-body defect: a stage
|
||||
* that can be handed nothing has to say so, because the model downstream will
|
||||
* always answer, and it answers out of its prompt.
|
||||
*
|
||||
* Exported from here rather than written into the prompt string so the prompt
|
||||
* and the guard that rejects an echo of it cannot drift apart.
|
||||
*/
|
||||
export const PROMPT_PLACEHOLDER_MERCHANTS = ["<Store> (<Suburb>)"] as const;
|
||||
|
||||
/**
|
||||
* Below this a text part needs a money figure to be believable. NOT a floor on
|
||||
* its own: `TOTAL $17.00` is twelve characters and is exactly the kind of
|
||||
* pasted line this door exists to accept.
|
||||
*/
|
||||
export const MIN_RECEIPT_TEXT_CHARS = 40;
|
||||
|
||||
/** `$17.00`, `17.00`, `12,50` — a printed amount, not merely a digit. */
|
||||
const MONEY_SHAPE = /(?:[$€£¥]\s*\d|\d[.,]\d{2}(?!\d))/;
|
||||
|
||||
export type TextCheck = { ok: true } | { ok: false; reason: string };
|
||||
|
||||
/**
|
||||
* Can this text possibly be a receipt? Deterministic, before any model call.
|
||||
*
|
||||
* A receipt has more than one word and at least one number. Everything the
|
||||
* phone substitutes when a share goes wrong — the literal word `Image`, a bare
|
||||
* filename like `IMG_0042.jpeg`, an empty body — fails one of those, and none
|
||||
* of them fails in a way a model will admit to.
|
||||
*
|
||||
* Applies to TEXT parts only. A PDF is checked for readable text separately and
|
||||
* an image is the thing we actually asked for.
|
||||
*/
|
||||
export function textCouldBeReceipt(raw: string | null | undefined): TextCheck {
|
||||
const text = (raw ?? "").trim();
|
||||
if (!text) return { ok: false, reason: "it was empty" };
|
||||
if (!/\s/.test(text)) return { ok: false, reason: "it is a single word" };
|
||||
if (!/\d/.test(text)) return { ok: false, reason: "it has no numbers in it" };
|
||||
if (text.length < MIN_RECEIPT_TEXT_CHARS && !MONEY_SHAPE.test(text)) {
|
||||
return { ok: false, reason: "it is too short to be a receipt" };
|
||||
}
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
/** What the phone actually sent, short enough to put on a card. */
|
||||
export function quoteShared(raw: string, max = 80): string {
|
||||
const flat = (raw ?? "").replace(/\s+/g, " ").trim();
|
||||
return flat.length > max ? `${flat.slice(0, max - 1)}…` : flat;
|
||||
}
|
||||
|
||||
/**
|
||||
* Belt to `textCouldBeReceipt`'s braces: did the extractor answer out of its
|
||||
* prompt rather than out of the upload?
|
||||
*
|
||||
* Two tells, both cheap and both deterministic:
|
||||
* - the merchant IS one of the prompt's format placeholders; or
|
||||
* - nothing readable was sent at all (no images, and no text part that could
|
||||
* be a receipt) and yet a merchant or items came back.
|
||||
*
|
||||
* The second is reachable even with the pre-check in place, because the
|
||||
* pre-check only inspects text parts: a PDF that extracts to a few characters
|
||||
* of junk still reaches the model. This is `unsupported`, never `needs_header`
|
||||
* — a confabulated header is not a partial one.
|
||||
*/
|
||||
export function confabulationReason(
|
||||
input: { textParts: string[]; hasImages: boolean },
|
||||
extracted: { merchant: string | null; items: unknown[] }
|
||||
): string | null {
|
||||
const key = merchantKey(extracted.merchant);
|
||||
if (key && PROMPT_PLACEHOLDER_MERCHANTS.some((p) => merchantKey(p) === key)) {
|
||||
return "the merchant came back as the prompt's own example rather than a name off the receipt";
|
||||
}
|
||||
if (input.hasImages) return null;
|
||||
if (input.textParts.some((t) => textCouldBeReceipt(t).ok)) return null;
|
||||
if (!extracted.merchant && !extracted.items.length) return null;
|
||||
return "nothing shared could be read as a receipt, yet an order came back";
|
||||
}
|
||||
|
||||
@@ -526,7 +526,11 @@ export function parseOrderHTML(html: string, meta: MessageMeta): ParsedOrder {
|
||||
} else {
|
||||
// DoorDash carries no order id anywhere in the receipt, so the provider
|
||||
// message id is the only stable identity available. That is correct for
|
||||
// ingestion idempotency: one receipt is one order.
|
||||
// ingestion idempotency: one receipt is one order — PROVIDED every caller
|
||||
// sends the same id for the same email. Two lanes post here (n8n Order
|
||||
// Processor and ingestion-engine order_handoff); both must send the bare
|
||||
// RFC Message-ID. n8n sent Gmail's own id until 2026-10-03 and one order
|
||||
// became txns 7268 + 7269.
|
||||
if (!meta.messageId) {
|
||||
throw new OrderParseError("no order id in body and no messageId supplied");
|
||||
}
|
||||
|
||||
@@ -167,9 +167,24 @@ export async function merchantVerdict(
|
||||
FROM order_reviews r
|
||||
JOIN transactions t ON t.id = r.transaction_id
|
||||
JOIN participants p ON p.id = r.participant_id
|
||||
JOIN expense_metadata em
|
||||
ON em.transaction_id = r.transaction_id
|
||||
OR em.matched_transaction_id = r.transaction_id
|
||||
-- (No backticks in here: this SQL lives in a TS template literal and a
|
||||
-- backtick ends the string -- TS1005 on a line that looks like a comment.)
|
||||
-- LATERAL, not a plain JOIN. A plain join on
|
||||
-- transaction_id OR matched_transaction_id fans out: one review
|
||||
-- joined to N metadata rows appears N times in the result, and the loops
|
||||
-- below count every appearance -- inflating the loved/never tallies
|
||||
-- that drive the pre-order warning. LIMIT 50 applies to the fanned-out
|
||||
-- set too, so duplicates would also push older reviews out of the
|
||||
-- window. Same tie-break as every other reader, so the merchant matched
|
||||
-- here is the merchant the UI shows.
|
||||
JOIN LATERAL (
|
||||
SELECT em2.merchant_normalized
|
||||
FROM expense_metadata em2
|
||||
WHERE em2.transaction_id = r.transaction_id
|
||||
OR em2.matched_transaction_id = r.transaction_id
|
||||
ORDER BY (em2.transaction_id = r.transaction_id) DESC NULLS LAST, em2.id
|
||||
LIMIT 1
|
||||
) em ON true
|
||||
-- Case-folded: the platforms capitalise the same restaurant differently
|
||||
-- ("TEG Kebabs & Biryani" on Uber Eats, "TEG KEBABS & BIRYANI" on
|
||||
-- DoorDash). An exact match split one restaurant's history in two, so a
|
||||
|
||||
+42
-1
@@ -851,6 +851,15 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
|
||||
WHERE ts.transaction_id = t.id
|
||||
) txn_splits ON true
|
||||
WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL
|
||||
-- A superseded row is already resolved, just not by reconciling. Without
|
||||
-- this the queue kept offering it: a duplicate wallet half hidden under
|
||||
-- its twin (txn 7037, 2026-10-03) sat beside that twin with the same
|
||||
-- statement line as its match, and accepting it on the hidden row would
|
||||
-- have used the line up and stranded the real one.
|
||||
AND t.superseded_by_id IS NULL
|
||||
-- A voided row is resolved too: the charge never posted, so no statement
|
||||
-- line will ever arrive for it (migration 0040).
|
||||
AND t.voided_at IS NULL
|
||||
AND ${needsCardMatch("t")}
|
||||
ORDER BY t.transaction_date DESC, t.row_index ASC`,
|
||||
[ownerId]
|
||||
@@ -887,12 +896,39 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
|
||||
FROM transactions m
|
||||
JOIN transactions t ON t.statement_id IS NOT NULL
|
||||
AND t.transaction_date BETWEEN m.transaction_date - 3 AND m.transaction_date + 3
|
||||
AND t.amount BETWEEN m.amount * 0.99 AND m.amount * 1.01
|
||||
-- +-5%, NOT +-1%. A foreign card purchase is charged at the notified
|
||||
-- amount PLUS the card's foreign transaction fee (3% on the Westpac
|
||||
-- Altitude), so the wallet row and its statement line can never match
|
||||
-- exactly and the old +-1% band excluded the statement row outright:
|
||||
-- SP PEAKDESIGN was notified at $42.44 and charged $43.71 (x1.03),
|
||||
-- so the one row that WAS its twin was the one the picker could not
|
||||
-- offer. Widening is safe here because this list is capped at 5 by
|
||||
-- the rn window and ordered by amount closeness; a looser band changes
|
||||
-- which five appear, never how many, and a person confirms each one.
|
||||
-- The AUTO-matcher stays exact on purpose; see findAutoMatches().
|
||||
AND (t.amount BETWEEN m.amount * 0.95 AND m.amount * 1.05
|
||||
-- Or the exact native pair, whatever the AUD figures say. A
|
||||
-- wallet row's AUD is provisional: in Oct 2026 it priced INR at
|
||||
-- 0.0180 while CommBank charged ~0.0149, 20% under, so the band
|
||||
-- excluded every rupee twin. findAutoMatches() matches on this
|
||||
-- pair already, but refuses when two lines share it inside its
|
||||
-- window (INR 112 on 10/4 and again on 10/7) -- and those are
|
||||
-- exactly the rows that land here for a person to choose.
|
||||
OR (m.foreign_currency_code IS NOT NULL
|
||||
AND t.foreign_currency_code = m.foreign_currency_code
|
||||
AND t.foreign_currency_amount = m.foreign_currency_amount))
|
||||
JOIN statements s ON s.id = t.statement_id
|
||||
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
|
||||
WHERE m.statement_id IS NULL
|
||||
AND m.owner_id = $1
|
||||
AND m.reconciled_with_id IS NULL
|
||||
AND m.superseded_by_id IS NULL
|
||||
AND m.voided_at IS NULL
|
||||
-- Nor offer a superseded STATEMENT row as the match: the ANZ
|
||||
-- statements 107/142/143 left 31 of those, and a manual row
|
||||
-- reconciled to one inherits its exclusion from every figure.
|
||||
-- findAutoMatches() has had both filters all along.
|
||||
AND t.superseded_by_id IS NULL
|
||||
AND ${needsCardMatch("m")}
|
||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||
AND NOT EXISTS (
|
||||
@@ -990,6 +1026,11 @@ export async function getSharedTransactions(ownerId: number, tagIds?: number[],
|
||||
SELECT em.platform
|
||||
FROM expense_metadata em
|
||||
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||
-- Same tie-break as getTransactions / replaceLineItems / matchTransaction,
|
||||
-- so every reader resolves the row a write lands on. Without the ORDER BY
|
||||
-- this LIMIT 1 is bounded but arbitrary: it would disagree with the main
|
||||
-- transaction list for the same transaction the day multiplicity appears.
|
||||
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
|
||||
LIMIT 1
|
||||
) order_ctx ON true
|
||||
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { queryRaw, queryRow } from "@/lib/db";
|
||||
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
||||
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
|
||||
|
||||
/**
|
||||
* Grocery receipts scanned in pantry-app, arriving as candidate spend.
|
||||
@@ -154,16 +155,75 @@ export async function processReceiptIngestion(receipt: ParsedReceipt): Promise<R
|
||||
const paymentMethod = paymentMethodFor(leg.class);
|
||||
const amount = round2(Number(leg.amount));
|
||||
const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name;
|
||||
const txn = await queryRow<{ id: number }>(
|
||||
`INSERT INTO transactions (
|
||||
transaction_date, description, amount, amount_aud, category, payment_method,
|
||||
merchant_name, merchant_normalized, transaction_type, owner_id
|
||||
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
|
||||
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
|
||||
);
|
||||
|
||||
// Adopt the row the wallet lane already created for this tap, if there is
|
||||
// one. Signals for a purchase arrive in sequence -- the notification within
|
||||
// seconds, the receipt hours later -- so creating a second transaction here
|
||||
// is how one shop ends up as two rows. Measured on txn 6858: the wallet
|
||||
// event arrived 35h25m before this path created its own row for the same
|
||||
// $103.30 Woolworths shop. Null means no unambiguous candidate, and the
|
||||
// fallback is exactly the old behaviour.
|
||||
const adoptedId = await findAdoptableTransaction(DEFAULT_OWNER_ID, amount, receipt.transaction_date);
|
||||
const txn = adoptedId
|
||||
? await queryRow<{ id: number }>(
|
||||
// The receipt read the register tape; the notification read a push
|
||||
// banner. So the receipt wins on what was bought and where, and the
|
||||
// column list here IS the ownership boundary -- `location` is absent
|
||||
// because the wallet notification's suburb is the better value and
|
||||
// must survive.
|
||||
`UPDATE transactions
|
||||
SET description = $2, merchant_name = $3, merchant_normalized = $3,
|
||||
category = $4, payment_method = $5
|
||||
WHERE id = $1 RETURNING id`,
|
||||
[adoptedId, description, receipt.merchant_name, RECEIPT_CATEGORY, paymentMethod]
|
||||
)
|
||||
: await queryRow<{ id: number }>(
|
||||
`INSERT INTO transactions (
|
||||
transaction_date, description, amount, amount_aud, category, payment_method,
|
||||
merchant_name, merchant_normalized, transaction_type, owner_id
|
||||
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
|
||||
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
|
||||
);
|
||||
|
||||
const carriesLines = leg.leg_index === cardLeg?.leg_index;
|
||||
const meta = await queryRow<{ id: number }>(
|
||||
|
||||
// An adopted row may already carry a metadata row (the wallet lane writes
|
||||
// one). expense_metadata.transaction_id is UNIQUE, so a second INSERT would
|
||||
// raise -- and must not, because the two describe the same purchase.
|
||||
// Enrich in place instead, with the same ownership split as the transaction
|
||||
// update above: everything the receipt read wins, and anything only the
|
||||
// notification knew (issuer, and the transaction's location) is absent from
|
||||
// the column list rather than conditionally skipped, so it cannot be
|
||||
// clobbered by a bug here. Mirrors order_transaction_bridge's UPGRADE pass.
|
||||
const priorMeta = adoptedId
|
||||
? await queryRow<{ id: number }>(
|
||||
`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [adoptedId]
|
||||
)
|
||||
: null;
|
||||
|
||||
const meta = priorMeta
|
||||
? await queryRow<{ id: number }>(
|
||||
`UPDATE expense_metadata
|
||||
SET source = 'pantry', order_reference = $2, line_items = $3::jsonb,
|
||||
subtotal = $4, amount = $5, merchant_normalized = $6,
|
||||
transaction_date = $7, card_last4 = COALESCE($8, card_last4),
|
||||
currency = 'AUD', flags = flags || $9::jsonb,
|
||||
payment_method = $10, payment_method_detail = $11,
|
||||
tender_raw = COALESCE($12, tender_raw),
|
||||
receipt_sha256 = $13, receipt_group = $14,
|
||||
extraction_model = 'cloud-budget'
|
||||
WHERE id = $1 RETURNING id`,
|
||||
[
|
||||
priorMeta.id, reference,
|
||||
JSON.stringify(carriesLines ? receipt.line_items ?? [] : []),
|
||||
receipt.tax_amount ?? null, amount, receipt.merchant_name,
|
||||
receipt.transaction_date, leg.card_last4 ?? null,
|
||||
JSON.stringify(carriesLines ? [...flags, "enriched_prior_row"] : [...flags, "line_items_on_card_leg", "enriched_prior_row"]),
|
||||
paymentMethod, leg.card_product ?? null, receipt.tender_raw ?? null,
|
||||
receipt.image_sha256 ?? null, group,
|
||||
]
|
||||
)
|
||||
: await queryRow<{ id: number }>(
|
||||
`INSERT INTO expense_metadata (
|
||||
transaction_id, source, order_reference, line_items, subtotal, amount,
|
||||
merchant_normalized, transaction_date, card_last4, currency, flags,
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
import { prisma } from "@/lib/db";
|
||||
|
||||
/**
|
||||
* Merge a statementless ("manual") transaction into its statement twin.
|
||||
*
|
||||
* Extracted from POST /api/transactions/reconcile so the manual queue and the
|
||||
* automatic pass cannot drift apart. There must be exactly one implementation
|
||||
* of this: it moves curation (overrides, tags, splits) and provenance
|
||||
* (expense_metadata) off a row that reconciliation is about to hide from every
|
||||
* figure, and a second copy of that logic would eventually move a different set.
|
||||
*
|
||||
* The manual row is NOT deleted -- it is linked via `reconciled_with_id`.
|
||||
* Deleting it would cascade its children away; superseding/linking keeps the
|
||||
* history and lets the row stay visible as provenance.
|
||||
*/
|
||||
export async function applyReconcileMatch(
|
||||
manual_id: number,
|
||||
statement_tx_id: number
|
||||
): Promise<void> {
|
||||
await prisma.$transaction(async (tx) => {
|
||||
await moveCurationWithin(tx, manual_id, statement_tx_id);
|
||||
|
||||
// Mark manual tx as reconciled (link to statement tx)
|
||||
await tx.$executeRawUnsafe(
|
||||
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
|
||||
statement_tx_id,
|
||||
manual_id
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Move overrides, tags, splits and expense_metadata from one transaction to
|
||||
* another, inside a caller-supplied transaction.
|
||||
*
|
||||
* Callers: `applyReconcileMatch` (then sets `reconciled_with_id`) and
|
||||
* `supersedeTransaction` (then sets `superseded_by_id`). Both hide the source
|
||||
* row from every figure, so both must carry its curation across first; keeping
|
||||
* one implementation is what the file header insists on.
|
||||
*/
|
||||
async function moveCurationWithin(
|
||||
tx: Parameters<Parameters<typeof prisma.$transaction>[0]>[0],
|
||||
manual_id: number,
|
||||
statement_tx_id: number
|
||||
): Promise<void> {
|
||||
{
|
||||
// Copy overrides: manual → statement tx
|
||||
const override = await tx.transaction_overrides.findUnique({
|
||||
where: { transaction_id: manual_id },
|
||||
});
|
||||
if (override) {
|
||||
await tx.transaction_overrides.upsert({
|
||||
where: { transaction_id: statement_tx_id },
|
||||
update: {
|
||||
category_override: override.category_override,
|
||||
merchant_normalized: override.merchant_normalized,
|
||||
notes: override.notes,
|
||||
my_share_percent: override.my_share_percent,
|
||||
updated_at: new Date(),
|
||||
},
|
||||
create: {
|
||||
transaction_id: statement_tx_id,
|
||||
category_override: override.category_override,
|
||||
merchant_normalized: override.merchant_normalized,
|
||||
notes: override.notes,
|
||||
my_share_percent: override.my_share_percent,
|
||||
},
|
||||
});
|
||||
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
|
||||
}
|
||||
|
||||
// Move tags: manual → statement tx
|
||||
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
|
||||
if (tags.length) {
|
||||
await tx.transaction_tags.createMany({
|
||||
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
|
||||
skipDuplicates: true,
|
||||
});
|
||||
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
|
||||
}
|
||||
|
||||
// Move splits: manual → statement tx.
|
||||
//
|
||||
// `settled` and `settled_at` MUST be carried. This is a delete-and-recreate,
|
||||
// and the column defaults to false — so omitting it silently marks every
|
||||
// settled split unsettled again and resurrects a debt that was already
|
||||
// paid. Nothing errors and nothing in the UI says the flag was reset; it
|
||||
// just reappears as owed. No queue row carries a settled split today, but
|
||||
// the auto-matcher runs this path unattended, so a latent version of this
|
||||
// would surface as money, not as an exception.
|
||||
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
|
||||
if (splits.length) {
|
||||
await tx.transaction_splits.createMany({
|
||||
data: splits.map((s) => ({
|
||||
transaction_id: statement_tx_id,
|
||||
participant_id: s.participant_id,
|
||||
share_percent: s.share_percent,
|
||||
settled: s.settled ?? false,
|
||||
settled_at: s.settled_at,
|
||||
})),
|
||||
skipDuplicates: true,
|
||||
});
|
||||
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
|
||||
}
|
||||
|
||||
// Move provenance: manual → statement tx.
|
||||
//
|
||||
// Overrides, tags and splits above were always carried across; expense_metadata was
|
||||
// the one child left behind, which did not matter while every metadata row came from
|
||||
// an email that had created its own transaction. It matters now: a scanned grocery
|
||||
// receipt puts its line items here, and reconciliation hides the manual row from
|
||||
// every figure — so without this the shop's contents disappear at exactly the moment
|
||||
// the statement line appears, and COLES 0556 MANOR LAKES stays as unreadable as it
|
||||
// was before the receipt was ever scanned.
|
||||
//
|
||||
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
|
||||
// or Paperless copy got there first) keeps it. The manual row's metadata stays
|
||||
// attached to the reconciled manual transaction and is flagged, rather than raising a
|
||||
// constraint violation or silently overwriting the other source.
|
||||
const moved = await tx.$executeRawUnsafe(
|
||||
`UPDATE expense_metadata SET transaction_id = $1
|
||||
WHERE transaction_id = $2
|
||||
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
|
||||
statement_tx_id,
|
||||
manual_id
|
||||
);
|
||||
if (moved === 0) {
|
||||
await tx.$executeRawUnsafe(
|
||||
`UPDATE expense_metadata
|
||||
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
|
||||
WHERE transaction_id = $1`,
|
||||
manual_id
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Manually mark `loser_id` as superseded by `survivor_id`.
|
||||
*
|
||||
* The escape hatch for a duplicate that `applyReconcileMatch` cannot express:
|
||||
* that one merges a statementLESS row into its statement twin, so it has
|
||||
* nothing to say about two rows on the SAME footing -- the pair left by a
|
||||
* statement imported twice (migration 0023 superseded 31 such pairs, $42,040.68,
|
||||
* by hand), or two feed rows for one purchase.
|
||||
*
|
||||
* Curation moves FIRST, using the same block `applyReconcileMatch` runs, because
|
||||
* `superseded_by_id` hides the row from every figure (`analytics-sql.ts`) and
|
||||
* splits or tags left on it would silently stop counting. That is why this lives
|
||||
* beside its sibling and calls into the same movement rather than re-deriving
|
||||
* it: the header of this file warns that a second copy would eventually move a
|
||||
* different set, and a supersede that forgot splits would resurrect a debt in
|
||||
* exactly the way `applyReconcileMatch`'s own comment describes.
|
||||
*
|
||||
* NOT a delete. Every child of `transactions` is ON DELETE CASCADE, so removing
|
||||
* the row would destroy the splits, tags and metadata that make it legible --
|
||||
* and a duplicate is evidence of how the ledger was assembled, which is worth
|
||||
* keeping. `transactions_no_self_supersede` blocks a row pointing at itself; the
|
||||
* survivor is re-checked here anyway so the error is a 400 and not a 500.
|
||||
*/
|
||||
export async function supersedeTransaction(
|
||||
loser_id: number,
|
||||
survivor_id: number
|
||||
): Promise<void> {
|
||||
if (loser_id === survivor_id) throw new Error("A transaction cannot supersede itself");
|
||||
await prisma.$transaction(async (tx) => {
|
||||
// Refuse to bury a row underneath one that is itself already buried, which
|
||||
// would make the curation we just moved unreachable in a second hop.
|
||||
const survivor = await tx.transactions.findUnique({
|
||||
where: { id: survivor_id },
|
||||
select: { superseded_by_id: true },
|
||||
});
|
||||
if (!survivor) throw new Error(`Survivor transaction ${survivor_id} not found`);
|
||||
if (survivor.superseded_by_id !== null) {
|
||||
throw new Error(`Transaction ${survivor_id} is itself superseded; pick the surviving row`);
|
||||
}
|
||||
await moveCurationWithin(tx, loser_id, survivor_id);
|
||||
await tx.$executeRawUnsafe(
|
||||
`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`,
|
||||
survivor_id,
|
||||
loser_id
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark a statementless row as a charge that never posted (migration 0040).
|
||||
*
|
||||
* The case neither sibling above can express: a card notification is sent for
|
||||
* an AUTHORISATION, and when the merchant releases the hold no statement line
|
||||
* ever arrives. Txn 7951 was a $1,040.41 Airbnb hold on a booking cancelled the
|
||||
* next day -- it sat in the reconcile queue with nothing to match and counted as
|
||||
* spend in every figure. There is no survivor to move curation onto, so nothing
|
||||
* moves: the row and its children stay, and `EXCLUDE_RECONCILED_SOURCE` stops
|
||||
* them counting.
|
||||
*
|
||||
* Refusals, all deliberate:
|
||||
* - a statement row: that is the bank's record of a posted charge, and the
|
||||
* statement must keep summing to its header (`transactions_void_statementless`
|
||||
* enforces this in the database too). A wrong statement row is a supersede.
|
||||
* - a row already reconciled or superseded: it is already out of every figure,
|
||||
* and a reconciled one evidently DID post.
|
||||
* - a row with a SETTLED split: someone has already paid their share of it.
|
||||
* Voiding would silently turn that payment into a credit on their balance;
|
||||
* that needs a person to look at it, not a side effect.
|
||||
*/
|
||||
export async function voidTransaction(id: number, reason: string): Promise<void> {
|
||||
const why = reason.trim();
|
||||
if (!why) throw new Error("A void needs a reason");
|
||||
await prisma.$transaction(async (tx) => {
|
||||
const rows = await tx.$queryRawUnsafe<
|
||||
{ statement_id: number | null; reconciled_with_id: number | null; superseded_by_id: number | null; settled: number }[]
|
||||
>(
|
||||
`SELECT t.statement_id, t.reconciled_with_id, t.superseded_by_id,
|
||||
(SELECT count(*)::int FROM transaction_splits ts
|
||||
WHERE ts.transaction_id = t.id AND ts.settled) AS settled
|
||||
FROM transactions t WHERE t.id = $1 FOR UPDATE`,
|
||||
id
|
||||
);
|
||||
const row = rows[0];
|
||||
if (!row) throw new Error(`Transaction ${id} not found`);
|
||||
if (row.statement_id !== null) {
|
||||
throw new Error(`Transaction ${id} is a statement line; a posted charge cannot be voided (supersede it instead)`);
|
||||
}
|
||||
if (row.reconciled_with_id !== null) {
|
||||
throw new Error(`Transaction ${id} is reconciled with ${row.reconciled_with_id}, so it posted`);
|
||||
}
|
||||
if (row.superseded_by_id !== null) {
|
||||
throw new Error(`Transaction ${id} is already superseded by ${row.superseded_by_id}`);
|
||||
}
|
||||
if (row.settled > 0) {
|
||||
throw new Error(`Transaction ${id} has ${row.settled} settled split(s); resolve those before voiding`);
|
||||
}
|
||||
await tx.$executeRawUnsafe(
|
||||
`UPDATE transactions SET voided_at = now(), void_reason = $1 WHERE id = $2 AND voided_at IS NULL`,
|
||||
why,
|
||||
id
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/** Undo `voidTransaction` -- e.g. the "released" hold posted after all. */
|
||||
export async function unvoidTransaction(id: number): Promise<void> {
|
||||
await prisma.$executeRawUnsafe(
|
||||
`UPDATE transactions SET voided_at = NULL, void_reason = NULL WHERE id = $1`,
|
||||
id
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* Deterministic parsing of the bits of a wallet notification that have a fixed
|
||||
* shape.
|
||||
*
|
||||
* Everything here was previously left to the interpreting model, which handled
|
||||
* ONE fixed-width format three different ways on the same day's data: it kept
|
||||
* the country code on "WYNDHAM VAL AU", stripped it on "Sydney" and "Perth",
|
||||
* and swallowed "SAN FRANCISOUS" whole into the location field. A fixed-width
|
||||
* field is not a judgement call, and the same argument that put the order
|
||||
* parsers in code rather than in a Gemini call applies: a regex fails loudly on
|
||||
* a template change, where a model returns a plausible wrong answer.
|
||||
*
|
||||
* The model still decides what the amount and merchant ARE -- it is better at
|
||||
* that than a pattern, because five cards emit three shapes. This only handles
|
||||
* the parts that are positional.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Westpac writes the merchant/city/country block as a fixed-width statement
|
||||
* descriptor: 27-char merchant, 12-char city, 2-char ISO country. Verified
|
||||
* across every Westpac event captured:
|
||||
*
|
||||
* Breadworld Bakery Cafe WYNDHAM VAL AU
|
||||
* PLA DONATION MELBOURNE AU
|
||||
* SP PEAKDESIGN SAN FRANCISOUS <- "SAN FRANCISO" + "US"
|
||||
*
|
||||
* The last example is why this is positional rather than a split on whitespace:
|
||||
* a 12-character city fills its field exactly and runs straight into the
|
||||
* country code with no separator.
|
||||
*/
|
||||
export type WestpacDescriptor = {
|
||||
merchant: string | null;
|
||||
city: string | null;
|
||||
/** ISO-3166 alpha-2, upper-case. Null when the tail is not a country code. */
|
||||
country: string | null;
|
||||
};
|
||||
|
||||
const MERCHANT_WIDTH = 27;
|
||||
const CITY_WIDTH = 12;
|
||||
|
||||
/**
|
||||
* `body` is the whole notification body, e.g.
|
||||
* "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
|
||||
* Returns nulls rather than throwing: a template change must degrade to "we
|
||||
* don't know", never to a wrong country on a foreign transaction.
|
||||
*/
|
||||
export function parseWestpacDescriptor(body: string | null | undefined): WestpacDescriptor {
|
||||
const none: WestpacDescriptor = { merchant: null, city: null, country: null };
|
||||
if (!body) return none;
|
||||
|
||||
// The descriptor is whatever follows the first sentence ("... ending...8032. ").
|
||||
// [\s\S] rather than the `s` flag: the tsconfig target predates es2018.
|
||||
const m = body.match(/^[\s\S]*?\.\s([\s\S]*)$/);
|
||||
if (!m) return none;
|
||||
const tail = m[1].replace(/\s+$/, "");
|
||||
if (!tail) return none;
|
||||
|
||||
const merchant = tail.slice(0, MERCHANT_WIDTH).trim() || null;
|
||||
const rest = tail.slice(MERCHANT_WIDTH);
|
||||
if (!rest) return { merchant, city: null, country: null };
|
||||
|
||||
// Country is the final two characters of the fixed block, but only when the
|
||||
// block is actually long enough to contain one -- a short tail is a merchant
|
||||
// name with no location, not a merchant whose last two letters are a country.
|
||||
const block = rest.slice(0, CITY_WIDTH + 2);
|
||||
if (block.length < CITY_WIDTH + 2) {
|
||||
return { merchant, city: block.trim() || null, country: null };
|
||||
}
|
||||
const city = block.slice(0, CITY_WIDTH).trim() || null;
|
||||
const tailTwo = block.slice(CITY_WIDTH).trim().toUpperCase();
|
||||
const country = /^[A-Z]{2}$/.test(tailTwo) ? tailTwo : null;
|
||||
return { merchant, city, country: country };
|
||||
}
|
||||
|
||||
/**
|
||||
* The Apple Wallet notification's subtitle, which is comma-delimited and clean:
|
||||
* "Breadworld Bakery Cafe, Wyndham Vale, VIC" -> location "Wyndham Vale, VIC"
|
||||
* "Seatsaero" -> no location (online merchant)
|
||||
*
|
||||
* Only the first comma separates merchant from place; the rest is the place.
|
||||
*/
|
||||
export function parseWalletSubtitle(subtitle: string | null | undefined): {
|
||||
merchant: string | null;
|
||||
location: string | null;
|
||||
} {
|
||||
if (!subtitle || !subtitle.trim()) return { merchant: null, location: null };
|
||||
const parts = subtitle.split(",").map((p) => p.trim()).filter(Boolean);
|
||||
if (parts.length <= 1) return { merchant: parts[0] ?? null, location: null };
|
||||
return { merchant: parts[0], location: parts.slice(1).join(", ") };
|
||||
}
|
||||
|
||||
/**
|
||||
* Currency marker, if the notification states one.
|
||||
*
|
||||
* Only Amex does: it writes "A$5,103.49". Westpac, CommBank and Apple Wallet
|
||||
* write a bare "$", which is honestly recorded as unknown rather than assumed
|
||||
* to be AUD -- 14 of 21 captured transaction rows. Domestically that assumption
|
||||
* would be harmless; abroad it is a 23x error on a THB tap, so it is not made
|
||||
* here. The statement settles it within weeks, and the reconcile queue holds
|
||||
* the row until then.
|
||||
*/
|
||||
export function currencyFromAmountText(amountText: string | null | undefined): string | null {
|
||||
if (!amountText) return null;
|
||||
const t = amountText.trim();
|
||||
if (/^A\$/i.test(t)) return "AUD";
|
||||
if (/^(NZ\$)/i.test(t)) return "NZD";
|
||||
if (/^US\$/i.test(t)) return "USD";
|
||||
if (/^€/.test(t)) return "EUR";
|
||||
if (/^£/.test(t)) return "GBP";
|
||||
if (/^¥/.test(t)) return "JPY";
|
||||
if (/^฿/.test(t)) return "THB";
|
||||
if (/^S\$/i.test(t)) return "SGD";
|
||||
if (/^₹/.test(t)) return "INR";
|
||||
// CommBank's approval prompts write the ISO code after the figure
|
||||
// ("672006 VND", "1340.00 INR"); some issuers put it first ("USD 10.00").
|
||||
const iso = t.match(/^[\d.,\s]+([A-Z]{3})$/) ?? t.match(/^([A-Z]{3})\s*[\d.,]+$/);
|
||||
if (iso) return iso[1];
|
||||
// A bare "$" is genuinely ambiguous -- do not guess.
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,481 @@
|
||||
import { queryRow, withTransaction, type Queryable } from "@/lib/db";
|
||||
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
||||
import { findWalletAdoptableTransaction } from "@/lib/auto-reconcile";
|
||||
import {
|
||||
parseWestpacDescriptor,
|
||||
parseWalletSubtitle,
|
||||
currencyFromAmountText,
|
||||
} from "@/lib/wallet-descriptor";
|
||||
|
||||
/**
|
||||
* Wallet notifications -> transactions.
|
||||
*
|
||||
* A transaction is created for EVERY tap (DECISIONS.md ING-11, 2026-08-22),
|
||||
* because overseas trips are where live spend matters most: the statement lag
|
||||
* is the same 6-35 days abroad but knowing what you have spent matters far more
|
||||
* in an unfamiliar currency.
|
||||
*
|
||||
* SEQUENCING IS THE WHOLE DESIGN. Wallet is the FIRST signal, not a late one --
|
||||
* measured on txn 6858, the notification arrived 35h25m before anything created
|
||||
* a transaction. So this creates the row, the receipt enriches it
|
||||
* (receipt-ingestion.ts adopts it), and the statement confirms it via the
|
||||
* reconcile queue, which a wallet row enters unaided: it is statementless with
|
||||
* payment_method 'card'. Nothing here needs to know about statements.
|
||||
*
|
||||
* But that sequence is a TENDENCY, not a guarantee -- these first 21 rows were
|
||||
* themselves backfilled days after the taps -- so this also adopts a row a
|
||||
* receipt got to first (findWalletAdoptableTransaction), and a stored-value
|
||||
* instrument is written as 'credits' so it never enters a queue no statement
|
||||
* line will ever resolve.
|
||||
*/
|
||||
|
||||
export type WalletEvent = {
|
||||
/** capture.wallet_events.id -- the idempotency key. */
|
||||
event_id: number;
|
||||
/** 'transaction' | 'not_transaction' | 'unsure' from the interpreter. */
|
||||
kind: string;
|
||||
direction?: string | null;
|
||||
amount: number | null;
|
||||
/** The exact substring the model read the amount from, e.g. "A$5,103.49". */
|
||||
amount_text?: string | null;
|
||||
currency?: string | null;
|
||||
merchant?: string | null;
|
||||
card_tail?: string | null;
|
||||
issuer?: string | null;
|
||||
location?: string | null;
|
||||
notif_app?: string | null;
|
||||
/** The notification's title line, e.g. "Visa Gift Card" or "Payment". */
|
||||
notif_title?: string | null;
|
||||
notif_body?: string | null;
|
||||
notif_subtitle?: string | null;
|
||||
occurred_at?: string | null;
|
||||
};
|
||||
|
||||
export type WalletIngestResult = {
|
||||
transactionId: number | null;
|
||||
metadataId: number | null;
|
||||
skipped?:
|
||||
| "not_a_transaction" | "already_ingested" | "no_amount" | "merged_pair" | "adopted"
|
||||
| "account_notice" | "stored_value_drawdown";
|
||||
flags: string[];
|
||||
};
|
||||
|
||||
/**
|
||||
* How far apart the two halves of ONE tap can arrive. Measured 2-8 seconds on
|
||||
* every captured pair; ten minutes leaves room for a phone that was offline,
|
||||
* and is still nowhere near the +/-1 DAY the matcher used to allow -- which
|
||||
* folded the UNICEF $50 donation and an unrelated $50 bar charge 17 hours later
|
||||
* into one row (event 275 into txn 8091, 2026-10-01).
|
||||
*/
|
||||
const PAIR_WINDOW_MINUTES = 10;
|
||||
|
||||
/** The ledger is dated the way the bank statements are: Melbourne local. */
|
||||
const LEDGER_TZ = "Australia/Melbourne";
|
||||
|
||||
/**
|
||||
* The calendar date of an instant, in LEDGER_TZ.
|
||||
*
|
||||
* Slicing the ISO string took whatever offset n8n serialised -- UTC in
|
||||
* practice -- so a Melbourne-evening tap landed on the previous day (28 of the
|
||||
* first 134 rows). Statements, which this row is reconciled against, date in
|
||||
* Australian local time.
|
||||
*/
|
||||
export function ledgerDate(occurredAt: string | null | undefined, now: Date = new Date()): string {
|
||||
const d = occurredAt ? new Date(occurredAt) : now;
|
||||
const at = Number.isNaN(d.getTime()) ? now : d;
|
||||
// en-CA formats as YYYY-MM-DD.
|
||||
return new Intl.DateTimeFormat("en-CA", {
|
||||
timeZone: LEDGER_TZ, year: "numeric", month: "2-digit", day: "2-digit",
|
||||
}).format(at);
|
||||
}
|
||||
|
||||
/**
|
||||
* A ledger-derived AUD rate for a currency: the median of what the bank
|
||||
* actually charged divided by the native figure, over statement rows.
|
||||
*
|
||||
* Same source order_transaction_bridge uses for its plausibility guard. This is
|
||||
* only ever a PROVISIONAL value -- transactions.amount is NOT NULL and always
|
||||
* AUD, and a notification cannot know the AUD figure because the bank sets the
|
||||
* rate at settlement. The statement overwrites it at reconcile time.
|
||||
* Returns null when there is not enough history to be worth anything.
|
||||
*/
|
||||
async function medianRateToAud(code: string): Promise<number | null> {
|
||||
const row = await queryRow<{ rate: string | null }>(
|
||||
`SELECT percentile_cont(0.5) WITHIN GROUP (
|
||||
ORDER BY amount / NULLIF(foreign_currency_amount, 0)
|
||||
)::text AS rate
|
||||
FROM transactions
|
||||
WHERE foreign_currency_code = $1
|
||||
AND foreign_currency_amount > 0
|
||||
AND superseded_by_id IS NULL
|
||||
-- Statement rows only: a wallet row's amount is itself a provisional
|
||||
-- guess, and one stored before this filter (txn 7316, 672006 VND at a
|
||||
-- 1:1 "rate") would otherwise feed the next estimate.
|
||||
AND statement_id IS NOT NULL
|
||||
HAVING count(*) >= 5`,
|
||||
[code]
|
||||
);
|
||||
const rate = row?.rate ? Number(row.rate) : null;
|
||||
return rate && Number.isFinite(rate) && rate > 0 ? rate : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The ECB reference rate for the day, via Frankfurter (free, no key; the same
|
||||
* source the statement processor converts with).
|
||||
*
|
||||
* The fallback when the ledger has no history for a currency. Hong Kong was the
|
||||
* first trip to need it: no HKD statement row existed, so every HKD tap stored
|
||||
* A$0.00 and the trip read as free until the statement landed weeks later. The
|
||||
* ledger median is still preferred -- it carries the card's ~3% foreign fee,
|
||||
* which the ECB mid-rate does not -- so this one is flagged as such.
|
||||
*
|
||||
* Any failure is null, never a throw: no rate is a state this lane already
|
||||
* handles (a flagged 0), and a slow FX API must not lose the purchase.
|
||||
*/
|
||||
async function ecbRateToAud(code: string, date: string): Promise<number | null> {
|
||||
try {
|
||||
const res = await fetch(
|
||||
`https://api.frankfurter.dev/v1/${date}?from=${encodeURIComponent(code)}&to=AUD`,
|
||||
{ signal: AbortSignal.timeout(5000) }
|
||||
);
|
||||
if (!res.ok) return null;
|
||||
const body = (await res.json()) as { rates?: { AUD?: number } };
|
||||
const rate = body?.rates?.AUD;
|
||||
return typeof rate === "number" && Number.isFinite(rate) && rate > 0 ? rate : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The other half of a paired notification, if it has already been ingested.
|
||||
*
|
||||
* One tap emits TWO notifications ~2-8s apart: the issuer app (which has the
|
||||
* card tail) and Apple Wallet (which has the better merchant and the suburb).
|
||||
* Neither alone is complete, so they must be MERGED, never deduped -- dropping
|
||||
* either loses a field nothing else supplies. The notifications share no
|
||||
* identifier, so a pair is recognised by what a pair IS: the same amount,
|
||||
* minutes apart, from two different apps. Two notifications from one app are
|
||||
* two charges, however close together.
|
||||
*/
|
||||
async function findPairedRow(
|
||||
db: Queryable,
|
||||
amount: number,
|
||||
notifiedAt: string,
|
||||
notifApp: string | null,
|
||||
fx: { amount: number; code: string } | null = null
|
||||
) {
|
||||
// A foreign row pairs on its native pair: its `amount` is a provisional
|
||||
// conversion (or a 0 placeholder), so two unrelated foreign taps could share it.
|
||||
const rows = await db.queryRaw<{ id: number; meta_id: number }>(
|
||||
`SELECT t.id, em.id AS meta_id
|
||||
FROM transactions t
|
||||
JOIN expense_metadata em ON em.transaction_id = t.id
|
||||
WHERE t.source = 'wallet'
|
||||
AND t.statement_id IS NULL
|
||||
AND t.superseded_by_id IS NULL
|
||||
AND CASE WHEN $4::text IS NULL
|
||||
THEN t.amount = $1::numeric AND t.foreign_currency_code IS NULL
|
||||
ELSE t.foreign_currency_code = $4::text
|
||||
AND t.foreign_currency_amount = $5::numeric END
|
||||
AND em.notified_at BETWEEN $2::timestamptz - make_interval(mins => $3::int)
|
||||
AND $2::timestamptz + make_interval(mins => $3::int)
|
||||
AND em.notif_app IS DISTINCT FROM $6::text
|
||||
LIMIT 2`,
|
||||
[amount, notifiedAt, PAIR_WINDOW_MINUTES, fx?.code ?? null, fx?.amount ?? null, notifApp]
|
||||
);
|
||||
return rows.length === 1 ? rows[0] : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* A stored-value instrument: a gift card, a prepaid travel card, a store card.
|
||||
*
|
||||
* It matters because payment_method decides queue membership. 'card' means "a
|
||||
* statement line is coming"; findAutoMatches and the reconcile queue both
|
||||
* exclude 'credits' and 'cash' for exactly that reason. A $103.30 Woolworths
|
||||
* tap on a Visa gift card written as 'card' parks in the queue forever, being
|
||||
* offered as an unmatched row against statements that can never contain it --
|
||||
* three of the first 21 wallet rows (events 80, 92, 101) were gift-card taps.
|
||||
*
|
||||
* The evidence is the notification TITLE, which is where Apple Wallet prints
|
||||
* the instrument's own name ("Visa Gift Card"); the interpreter reduces that to
|
||||
* issuer "Visa", which a plain Visa debit card also produces, so the title is
|
||||
* the only place the distinction survives. Read as a prior, not a contract: a
|
||||
* miss leaves today's behaviour, which is the queue.
|
||||
*/
|
||||
const STORED_VALUE = /gift\s*card|prepaid|travel\s*(?:money|card)|store\s*card/i;
|
||||
|
||||
/**
|
||||
* A transit card reporting a drawdown of its own balance ("HKD 7.40 for Train"
|
||||
* / "Your new balance is HKD 169.50.").
|
||||
*
|
||||
* Unlike a gift card, a transit card is loaded FROM a bank card -- that load is
|
||||
* on the statement and is the spend. Each fare is that money being used, so
|
||||
* booking it again counts it twice. The interpreter already called 11 of 13
|
||||
* Octopus fares not_transaction; the two it did not (events 191, 200) became
|
||||
* rows, so the outcome depended on the model's mood. This makes it a rule. The
|
||||
* body wording is the signal because it is what every such card says; the
|
||||
* issuer name is not (Octopus, Opal, myki, Suica each name themselves).
|
||||
*/
|
||||
const BALANCE_DRAWDOWN = /\byour (?:new|current) balance is\b/i;
|
||||
|
||||
/**
|
||||
* Account notices that state an amount without money moving: "Your Smart Access
|
||||
* 1910 is overdrawn by $21.21. To avoid fees and interest, ..." (event 235)
|
||||
* became a $21.21 purchase named "To avoid fees and interest,". The model is
|
||||
* right that the text names an amount; it is not a payment.
|
||||
*/
|
||||
const ACCOUNT_NOTICE = /\boverdrawn\b|\bto avoid fees\b|\bavailable balance\b|\blow balance\b/i;
|
||||
|
||||
/** How an adopted or merged row records which wallet event folded into it. */
|
||||
const adoptedFlag = (eventId: number) => `wallet_evt:${eventId}`;
|
||||
|
||||
export function isStoredValueInstrument(ev: Pick<WalletEvent, "notif_title" | "issuer" | "notif_app">): boolean {
|
||||
return [ev.notif_title, ev.issuer, ev.notif_app].some((v) => !!v && STORED_VALUE.test(v));
|
||||
}
|
||||
|
||||
/** Already landed: a row this lane wrote, or one it adopted or merged into. */
|
||||
async function alreadyIngested(db: Pick<Queryable, "queryRow">, eventId: number) {
|
||||
return db.queryRow<{ id: number }>(
|
||||
`SELECT id FROM transactions WHERE source = 'wallet' AND source_ref = $1
|
||||
UNION ALL
|
||||
SELECT em.transaction_id AS id FROM expense_metadata em
|
||||
JOIN transactions t ON t.id = em.transaction_id
|
||||
WHERE em.flags @> $2::jsonb AND t.superseded_by_id IS NULL
|
||||
LIMIT 1`,
|
||||
[`evt:${eventId}`, JSON.stringify([adoptedFlag(eventId)])]
|
||||
);
|
||||
}
|
||||
|
||||
export async function processWalletIngestion(ev: WalletEvent): Promise<WalletIngestResult> {
|
||||
const flags: string[] = [];
|
||||
|
||||
// An open-world classifier feeding a closed-world writer drops the unknown
|
||||
// and calls it a refusal (ING-14). `unsure` is a real answer and must not
|
||||
// silently become a transaction.
|
||||
if (ev.kind !== "transaction") {
|
||||
return { transactionId: null, metadataId: null, skipped: "not_a_transaction", flags };
|
||||
}
|
||||
if (ev.amount === null || !Number.isFinite(ev.amount) || ev.amount <= 0) {
|
||||
return { transactionId: null, metadataId: null, skipped: "no_amount", flags };
|
||||
}
|
||||
const text = [ev.notif_subtitle, ev.notif_body].filter(Boolean).join("\n");
|
||||
if (ACCOUNT_NOTICE.test(text)) {
|
||||
return { transactionId: null, metadataId: null, skipped: "account_notice", flags };
|
||||
}
|
||||
if (BALANCE_DRAWDOWN.test(ev.notif_body ?? "")) {
|
||||
return { transactionId: null, metadataId: null, skipped: "stored_value_drawdown", flags };
|
||||
}
|
||||
|
||||
// Two places this event may already have landed: a row this lane wrote (which
|
||||
// carries the ref natively), or a row it ADOPTED or MERGED into, whose
|
||||
// source_ref belongs to whoever created it -- marked in the metadata flags
|
||||
// rather than by overwriting a field another lane owns. Both must be checked,
|
||||
// because bumping the interpreter's prompt_version deliberately re-interprets
|
||||
// and re-posts every event ever captured -- that is the replay mechanism, and
|
||||
// it must not mint a second row for a purchase that has one. Checked here
|
||||
// without the lock to skip the FX lookup on a replay, and again under it.
|
||||
const seen = await alreadyIngested({ queryRow }, ev.event_id);
|
||||
if (seen) return { transactionId: seen.id, metadataId: null, skipped: "already_ingested", flags };
|
||||
|
||||
const sourceRef = `evt:${ev.event_id}`;
|
||||
const notifiedAt = ev.occurred_at ?? new Date().toISOString();
|
||||
const date = ledgerDate(ev.occurred_at);
|
||||
|
||||
// Deterministic parsing of the positional fields. The model decides what the
|
||||
// amount and merchant ARE; it does not get to parse a fixed-width block.
|
||||
// Only Westpac writes that block: run over a CommBank overdraft notice it
|
||||
// produced the merchant "To avoid fees and interest," (event 235).
|
||||
const desc = /westpac/i.test(ev.notif_app ?? "")
|
||||
? parseWestpacDescriptor(ev.notif_body)
|
||||
: { merchant: null, city: null, country: null };
|
||||
const sub = parseWalletSubtitle(ev.notif_subtitle);
|
||||
const merchant = sub.merchant ?? ev.merchant ?? desc.merchant ?? "Card purchase";
|
||||
// Two sources for a place, and they are not equal. The Apple Wallet subtitle
|
||||
// gives "Wyndham Vale, VIC"; the issuer's fixed-width descriptor gives
|
||||
// "WYNDHAM VAL AU" -- a 12-character truncation in caps. The subtitle wins
|
||||
// outright, which matters for the pair merge below: whichever half arrives
|
||||
// second must be able to REPLACE a descriptor-derived place, or the result
|
||||
// depends on arrival order.
|
||||
const subtitleLocation = sub.location ?? null;
|
||||
// Precedence: the Wallet subtitle, then the DETERMINISTIC descriptor parse,
|
||||
// and only then the model's own answer. Getting this order wrong defeats the
|
||||
// point of parsing positionally at all -- the model returned the mangled
|
||||
// "SAN FRANCISOUS" and the padded "MELBOURNE AU" for exactly these rows,
|
||||
// which is what wallet-descriptor.ts exists to correct.
|
||||
const descriptorLocation = desc.city && desc.country
|
||||
? `${desc.city} ${desc.country}`
|
||||
: desc.city;
|
||||
const location = subtitleLocation ?? descriptorLocation ?? ev.location ?? null;
|
||||
|
||||
// Currency: only ever what the notification actually states. A bare "$" is
|
||||
// recorded as unknown, never assumed AUD -- harmless domestically, a 23x
|
||||
// error on a THB tap abroad.
|
||||
const stated = currencyFromAmountText(ev.amount_text)
|
||||
?? (ev.currency && /^[A-Z]{3}$/.test(ev.currency) ? ev.currency : null);
|
||||
if (!stated) flags.push("currency_unverified");
|
||||
if (desc.country && desc.country !== "AU") flags.push(`foreign_country_${desc.country}`);
|
||||
|
||||
// Foreign: the native figure is what both sides can agree on exactly, so it
|
||||
// is what gets stored. `amount` still needs a value (NOT NULL) and takes a
|
||||
// provisional conversion, flagged, which the statement overwrites. Resolved
|
||||
// BEFORE the lock: the ECB fallback is a network call, and the lock
|
||||
// serialises every wallet post behind whatever runs inside it.
|
||||
let amountAud = ev.amount;
|
||||
let fxAmount: number | null = null;
|
||||
let fxCode: string | null = null;
|
||||
if (stated && stated !== "AUD") {
|
||||
fxAmount = ev.amount;
|
||||
fxCode = stated;
|
||||
const ledgerRate = await medianRateToAud(stated);
|
||||
const rate = ledgerRate ?? (await ecbRateToAud(stated, date));
|
||||
if (rate) {
|
||||
amountAud = Number((ev.amount * rate).toFixed(2));
|
||||
flags.push("aud_provisional");
|
||||
if (!ledgerRate) flags.push("rate_ecb");
|
||||
} else {
|
||||
// No rate from either source. The native figure must NOT fall through as
|
||||
// AUD -- 672006 VND once landed as A$672,006. Zero is a flagged
|
||||
// placeholder; the statement supplies the real AUD at reconcile time,
|
||||
// which matches on the native pair, not on this column.
|
||||
amountAud = 0;
|
||||
flags.push("aud_provisional", "no_rate_reference");
|
||||
}
|
||||
}
|
||||
|
||||
// A gift card has no statement line coming, so it must not enter the
|
||||
// reconcile queue (which filters on exactly this column).
|
||||
const storedValue = isStoredValueInstrument(ev);
|
||||
const paymentMethod = storedValue ? "credits" : "card";
|
||||
if (storedValue) flags.push("stored_value_instrument");
|
||||
|
||||
// Everything from the idempotency re-check to the INSERT holds one lock.
|
||||
// n8n's HTTP node posts a poll's items CONCURRENTLY, and both halves of a tap
|
||||
// usually land in the same 15-minute poll -- so each ran findPairedRow before
|
||||
// the other had inserted, neither found its partner, and one tap became two
|
||||
// rows: 11 pairs, A$3,841.40, every pair with an identical created_at. A
|
||||
// single lock for the whole lane, not one per amount: wallet volume is a few
|
||||
// dozen a day, and a coarse lock has no key to get wrong.
|
||||
return withTransaction(async (db) => {
|
||||
await db.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended('wallet-ingest', 0))::text`);
|
||||
|
||||
const existing = await alreadyIngested(db, ev.event_id);
|
||||
if (existing) {
|
||||
return { transactionId: existing.id, metadataId: null, skipped: "already_ingested" as const, flags };
|
||||
}
|
||||
|
||||
// Merge the paired notification rather than creating a second row for one
|
||||
// tap. The issuer half brings the card tail; the Wallet half brings the
|
||||
// merchant and suburb. COALESCE so whichever arrives second only ever fills
|
||||
// gaps -- it can never blank a field the first half supplied.
|
||||
const pair = await findPairedRow(
|
||||
db, amountAud, notifiedAt, ev.notif_app ?? null,
|
||||
fxCode && fxAmount !== null ? { amount: fxAmount, code: fxCode } : null
|
||||
);
|
||||
if (pair) {
|
||||
await db.queryRaw(
|
||||
`UPDATE transactions
|
||||
SET merchant_name = COALESCE(NULLIF($2,'Card purchase'), merchant_name),
|
||||
merchant_normalized = COALESCE(NULLIF($2,'Card purchase'), merchant_normalized),
|
||||
-- A subtitle-derived place overwrites; anything else only fills a
|
||||
-- gap. Without the $4 branch the merged row depends on which
|
||||
-- notification happened to arrive first, and the issuer half
|
||||
-- usually wins by a few seconds -- keeping the worse string.
|
||||
location = CASE WHEN $4::boolean THEN COALESCE($3, location)
|
||||
ELSE COALESCE(location, $3) END
|
||||
WHERE id = $1`,
|
||||
[pair.id, merchant, location, subtitleLocation !== null]
|
||||
);
|
||||
// The merged half records itself, or a replay finds nothing and merges
|
||||
// it again (or, once the partner is superseded, mints a new row).
|
||||
await db.queryRaw(
|
||||
`UPDATE expense_metadata
|
||||
SET card_last4 = COALESCE(card_last4, $2),
|
||||
payment_method_detail = COALESCE(payment_method_detail, $3),
|
||||
merchant_normalized = COALESCE(NULLIF($4,'Card purchase'), merchant_normalized),
|
||||
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
|
||||
WHERE id = $1`,
|
||||
[pair.meta_id, ev.card_tail ?? null, ev.issuer ?? null, merchant, JSON.stringify([adoptedFlag(ev.event_id)])]
|
||||
);
|
||||
return { transactionId: pair.id, metadataId: pair.meta_id, skipped: "merged_pair" as const, flags };
|
||||
}
|
||||
|
||||
// The receipt got here first. Fold into its row instead of writing a second
|
||||
// one for the same purchase (see findWalletAdoptableTransaction). The column
|
||||
// list IS the ownership boundary, the mirror of receipt-ingestion.ts's: the
|
||||
// receipt read the register tape, so merchant, description, category and the
|
||||
// items are its business and are absent here; `location` is wallet's own --
|
||||
// nothing else writes a suburb -- and payment_method is filled only when the
|
||||
// other lane left it unknown.
|
||||
const adopted = await findWalletAdoptableTransaction(DEFAULT_OWNER_ID, amountAud, date, merchant, undefined, db);
|
||||
if (adopted) {
|
||||
await db.queryRaw(
|
||||
`UPDATE transactions
|
||||
SET location = COALESCE($2, location),
|
||||
payment_method = COALESCE(payment_method, $3)
|
||||
WHERE id = $1`,
|
||||
[adopted.id, location, paymentMethod]
|
||||
);
|
||||
let metadataId = adopted.metadata_id;
|
||||
if (metadataId) {
|
||||
await db.queryRaw(
|
||||
`UPDATE expense_metadata
|
||||
SET card_last4 = COALESCE(card_last4, $2),
|
||||
payment_method_detail = COALESCE(payment_method_detail, $3),
|
||||
currency = COALESCE(currency, $4),
|
||||
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
|
||||
WHERE id = $1`,
|
||||
[metadataId, ev.card_tail ?? null, ev.issuer ?? null, stated, JSON.stringify([...flags, adoptedFlag(ev.event_id)])]
|
||||
);
|
||||
} else {
|
||||
// expense_metadata.transaction_id is UNIQUE, so this branch only runs
|
||||
// when the adopted row genuinely has none -- and it must produce one, or
|
||||
// the card tail and the flag that makes this idempotent have nowhere to
|
||||
// live.
|
||||
const meta = await db.queryRow<{ id: number }>(
|
||||
`INSERT INTO expense_metadata (
|
||||
transaction_id, source, source_message_id, merchant_normalized, amount,
|
||||
transaction_date, card_last4, currency, payment_method,
|
||||
payment_method_detail, line_items, flags, extraction_model,
|
||||
notified_at, notif_app
|
||||
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
|
||||
RETURNING id`,
|
||||
[
|
||||
adopted.id, sourceRef, merchant, ev.amount, date, ev.card_tail ?? null, stated,
|
||||
paymentMethod, ev.issuer ?? null, JSON.stringify([...flags, adoptedFlag(ev.event_id)]),
|
||||
notifiedAt, ev.notif_app ?? null,
|
||||
]
|
||||
);
|
||||
metadataId = meta!.id;
|
||||
}
|
||||
return { transactionId: adopted.id, metadataId, skipped: "adopted" as const, flags };
|
||||
}
|
||||
|
||||
const txn = await db.queryRow<{ id: number }>(
|
||||
`INSERT INTO transactions (
|
||||
transaction_date, description, amount, amount_aud, category, payment_method,
|
||||
merchant_name, merchant_normalized, transaction_type, owner_id, location,
|
||||
foreign_currency_amount, foreign_currency_code, source, source_ref, row_index
|
||||
) VALUES ($1,$2,$3,$3,'other',$10,$4,$4,'debit',$5,$6,$7,$8,'wallet',$9,0)
|
||||
RETURNING id`,
|
||||
[date, merchant, amountAud, merchant, DEFAULT_OWNER_ID, location, fxAmount, fxCode, sourceRef, paymentMethod]
|
||||
);
|
||||
|
||||
const meta = await db.queryRow<{ id: number }>(
|
||||
`INSERT INTO expense_metadata (
|
||||
transaction_id, source, source_message_id, merchant_normalized, amount,
|
||||
transaction_date, card_last4, currency, payment_method,
|
||||
payment_method_detail, line_items, flags, extraction_model,
|
||||
notified_at, notif_app
|
||||
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
|
||||
RETURNING id`,
|
||||
[
|
||||
txn!.id, sourceRef, merchant, ev.amount, date,
|
||||
ev.card_tail ?? null, stated, paymentMethod, ev.issuer ?? null, JSON.stringify(flags),
|
||||
notifiedAt, ev.notif_app ?? null,
|
||||
]
|
||||
);
|
||||
|
||||
return { transactionId: txn!.id, metadataId: meta!.id, flags };
|
||||
}, { maxWait: 10_000, timeout: 30_000 });
|
||||
}
|
||||
Reference in New Issue
Block a user