Compare commits
35
Commits
a4e5f540cc
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c25c82932d | ||
|
|
a1c2eb736f | ||
|
|
069034951e | ||
|
|
e956f39ffc | ||
|
|
d75589e128 | ||
|
|
ce0bf0a775 | ||
|
|
a3e4e1102a | ||
|
|
22fde30b26 | ||
|
|
48c6b2395f | ||
|
|
5076f27f68 | ||
|
|
10cf0aaf21 | ||
|
|
2251060ac6 | ||
|
|
2c05f10aed | ||
|
|
9b59edcf48 | ||
|
|
3d0d75519a | ||
|
|
8d1875225f | ||
|
|
8e66e527ae | ||
|
|
f31f12799a | ||
|
|
5e355c7e85 | ||
|
|
669f9119b9 | ||
|
|
258fe3b617 | ||
|
|
0823244e4a | ||
|
|
d88ca2a24c | ||
|
|
91586ce4a4 | ||
|
|
4b09d4136f | ||
|
|
4f0f0dd285 | ||
|
|
b788b4e8fa | ||
|
|
b5569b8ccc | ||
|
|
d4ff68363a | ||
|
|
631e75e58f | ||
|
|
c25db5211b | ||
|
|
9a442b7355 | ||
|
|
f9638cc5e5 | ||
|
|
7424d13858 | ||
|
|
be3d10d4f1 |
@@ -65,7 +65,7 @@ docker logs finance -f
|
|||||||
npm test # unit — pure logic, no database
|
npm test # unit — pure logic, no database
|
||||||
npm run test:integration # database-backed; resolves the DSN itself (see below)
|
npm run test:integration # database-backed; resolves the DSN itself (see below)
|
||||||
npm run test:all # both
|
npm run test:all # both
|
||||||
npm run test:setup # once, and after any migration: creates/refreshes personal_test
|
npm run test:setup # once, and after any migration: drops and rebuilds personal_test
|
||||||
npm run lint # blocking in CI since #79 — 0 errors, warnings allowed
|
npm run lint # blocking in CI since #79 — 0 errors, warnings allowed
|
||||||
npx tsc --noEmit # blocking in CI; nothing else compiles the test files
|
npx tsc --noEmit # blocking in CI; nothing else compiles the test files
|
||||||
```
|
```
|
||||||
@@ -86,6 +86,13 @@ had pointed the suite at **postgres-pantry** — whose database it would have
|
|||||||
TRUNCATEd had the credentials happened to match. A loopback address cannot be
|
TRUNCATEd had the credentials happened to match. A loopback address cannot be
|
||||||
reassigned to another container, which is why the published port wins.
|
reassigned to another container, which is why the published port wins.
|
||||||
|
|
||||||
|
**`test:setup` really rebuilds now (2026-10-03).** It used to replay the
|
||||||
|
schema dump over the existing database, where every `CREATE TABLE` failed with
|
||||||
|
"already exists" and the old shape survived — so no migration ever reached
|
||||||
|
`personal_test`, and new-column code failed every test with a misleading
|
||||||
|
`column does not exist`. It now `DROP DATABASE … WITH (FORCE)`s first, refusing
|
||||||
|
any name not ending `_test`, and loads with `ON_ERROR_STOP`.
|
||||||
|
|
||||||
**Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to
|
**Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to
|
||||||
truncate anything whose `current_database()` is not `personal_test`, and the
|
truncate anything whose `current_database()` is not `personal_test`, and the
|
||||||
suite runs `fileParallelism: false` because every file shares that one database
|
suite runs `fileParallelism: false` because every file shares that one database
|
||||||
@@ -176,6 +183,215 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
|
|||||||
- `statements` — one row per billing period per bank account
|
- `statements` — one row per billing period per bank account
|
||||||
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
|
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
|
||||||
|
|
||||||
|
### The change log (migration 0037)
|
||||||
|
|
||||||
|
`transaction_audit` is an **append-only** row-level record of every change to
|
||||||
|
`transactions`, `transaction_splits`, `transaction_tags`, `transaction_overrides`
|
||||||
|
and `expense_metadata`. Never `UPDATE` or `DELETE` from it. There is deliberately
|
||||||
|
**no foreign key** to `transactions` — the log has to outlive the row it describes.
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- everything that ever happened to one transaction
|
||||||
|
SELECT changed_at, table_name, op, old_data, new_data, app_name
|
||||||
|
FROM transaction_audit WHERE transaction_id = 6961 ORDER BY changed_at;
|
||||||
|
|
||||||
|
-- one logical edit, all the rows it touched (a reconcile moves 4 children at once)
|
||||||
|
SELECT * FROM transaction_audit WHERE txid = <txid> ORDER BY id;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why it exists.** On 2026-08-23 the n8n CC Statement Processor stamped
|
||||||
|
`statement_id` onto seven statementless wallet/pantry rows without setting
|
||||||
|
`reconciled_with_id`, double-counting $245.90 and hiding those rows from the
|
||||||
|
reconcile queue (`WHERE statement_id IS NULL`). Answering *"was anything lost?"*
|
||||||
|
took restoring the 02:30 `pg_dump` into a scratch database and diffing it column
|
||||||
|
by column, because nothing recorded what changed.
|
||||||
|
|
||||||
|
**Why it is a trigger and not application code.** The ledger has at least five
|
||||||
|
independent writers — this app (Prisma), n8n (raw SQL in a Postgres node),
|
||||||
|
pantry-app, the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`.
|
||||||
|
That bug was an n8n raw-SQL `UPDATE`; an app-level audit would not have seen it.
|
||||||
|
|
||||||
|
**Why not Postgres' own logs.** Checked live: `wal_level=replica` (WAL cannot be
|
||||||
|
decoded into row changes), `wal_keep_size=0` and `archive_mode=off` (recycled
|
||||||
|
almost immediately), `log_statement=none`, `track_commit_timestamp=off`, no
|
||||||
|
`pgaudit`. Postgres records nothing usable and has no built-in audit or temporal
|
||||||
|
tables.
|
||||||
|
|
||||||
|
Three details worth knowing before relying on it:
|
||||||
|
- **`old_data`/`new_data` hold only the columns that actually changed** on an
|
||||||
|
`UPDATE`. A whole-row copy would bury a one-column edit, and
|
||||||
|
`expense_metadata.line_items` alone can be kilobytes.
|
||||||
|
- **A no-op `UPDATE` writes nothing** (the upsert paths issue plenty).
|
||||||
|
- **Attribution is partial and that is understood.** Every client connects as the
|
||||||
|
same `personal` role, so `db_user` never discriminates. `app_name` is only as
|
||||||
|
good as what each client sets; `client_addr` separates containers and is NULL
|
||||||
|
for `docker exec` over the unix socket — itself a useful signal. `txid` is the
|
||||||
|
reliable grouping key.
|
||||||
|
|
||||||
|
`TRUNCATE` does not fire row-level triggers, so the integration suite's table
|
||||||
|
wipes produce no audit rows.
|
||||||
|
|
||||||
|
### Resolving a duplicate by hand (`POST /api/transactions/<id>/merge`)
|
||||||
|
|
||||||
|
`<id>` is the row that gets **hidden**; `other_id` is the one that **survives**.
|
||||||
|
|
||||||
|
| mode | requires | sets | for |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `reconcile` | loser statementless, survivor a statement row | `reconciled_with_id` | the everyday case |
|
||||||
|
| `supersede` | nothing | `superseded_by_id` | two rows on the same footing, e.g. a statement imported twice |
|
||||||
|
|
||||||
|
**There is no delete, deliberately.** Every child of `transactions` is
|
||||||
|
`ON DELETE CASCADE`, so removing a row destroys the splits, tags and metadata
|
||||||
|
that make it legible. Both modes keep the row and its children and merely stop
|
||||||
|
it counting. Do not add a delete endpoint.
|
||||||
|
|
||||||
|
Both modes **move curation onto the survivor first**, through the single
|
||||||
|
`moveCurationWithin()` in `lib/reconcile.ts` — because both hide the source row
|
||||||
|
from every figure, and a split stranded on a hidden row stops counting in
|
||||||
|
silence. That is also why the two callers share one implementation: the file
|
||||||
|
header warns that a second copy would eventually move a different set.
|
||||||
|
|
||||||
|
Refusals (all deliberate, all tested in `merge-supersede.test.ts`): a row already
|
||||||
|
reconciled or superseded on **either** side → 409 (re-pointing it strands the
|
||||||
|
curation moved on the first pass); superseding under a row that is itself
|
||||||
|
superseded → 409; a row that is not the caller's → 404. Ownership is checked with
|
||||||
|
`COALESCE(t.owner_id, s.owner_id)` — a statement row carries `owner_id` on the
|
||||||
|
statement, so checking `t.owner_id` alone rejects every statement row as "not
|
||||||
|
found".
|
||||||
|
|
||||||
|
**Why it had to exist.** `superseded_by_id` was read everywhere (analytics and
|
||||||
|
auto-reconcile both exclude it) but written **nowhere** in application code — set
|
||||||
|
exactly once, by migration 0023. The reconcile queue was the only merge path, and
|
||||||
|
it only offers candidates inside an amount band.
|
||||||
|
|
||||||
|
**The band is ±5%, not ±1% (2026-08-23).** A foreign card purchase is charged the
|
||||||
|
notified amount **plus the card's foreign transaction fee** — 3% on the Westpac
|
||||||
|
Altitude — so a wallet row and its statement line can never match exactly.
|
||||||
|
`SP PEAKDESIGN` was notified at **$42.44** and charged **$43.71** (×1.03), which
|
||||||
|
put the one row that *was* its twin outside the old band: the picker could not
|
||||||
|
offer the only correct answer. Widening is safe because the list is capped at 5
|
||||||
|
by the `rn` window and ordered by amount closeness, so a looser band changes
|
||||||
|
*which* five appear, never how many, and a person confirms each. Measured on the
|
||||||
|
live queue at the time of the change: candidate counts went **0 → 0** on all 12
|
||||||
|
rows, i.e. no new noise. **The auto-matcher stays exact** — see `findAutoMatches()`.
|
||||||
|
|
||||||
|
### A charge that never posted (`POST /api/transactions/<id>/void`, migration 0040)
|
||||||
|
|
||||||
|
The third way a row stops counting, for a case with **no survivor**: a card
|
||||||
|
notification is sent for an *authorisation*, and when the merchant releases the
|
||||||
|
hold no statement line ever arrives. Txn 7951 — a $1,040.41 Airbnb hold on Amex,
|
||||||
|
booking cancelled the next day — sat in the reconcile queue with nothing to
|
||||||
|
match and counted as spend everywhere. `reconcile` needs a statement twin,
|
||||||
|
`supersede` needs a survivor (and `transactions_no_self_supersede` blocks
|
||||||
|
pointing a row at itself), so it gets its own columns: `voided_at` +
|
||||||
|
`void_reason` (reason mandatory, by CHECK). Body `{reason}`; `DELETE` undoes it.
|
||||||
|
UI: **Never posted** on every row of `/reconcile`.
|
||||||
|
|
||||||
|
- **In `EXCLUDE_RECONCILED_SOURCE`**, so every "count each purchase once" query
|
||||||
|
drops it without being edited; the queue, `findAutoMatches`, both adoption
|
||||||
|
helpers, annual charges and the manual reconcile POST filter it explicitly.
|
||||||
|
- **Statementless only** — `transactions_void_statementless` enforces it in the
|
||||||
|
database. A statement line is the bank's record of a posted charge; a wrong
|
||||||
|
one is a supersede.
|
||||||
|
- **Refused** on a reconciled row (it posted), a superseded row, and a row with a
|
||||||
|
*settled* split (someone already paid their share — voiding would quietly turn
|
||||||
|
that into a credit). `merge` refuses a voided row on either side.
|
||||||
|
- **The wallet lane deliberately still SEES voided rows.** `alreadyIngested()` and
|
||||||
|
the issuer/Wallet pairing in `wallet-ingestion.ts` do not filter `voided_at`:
|
||||||
|
the voided row is the sink that stops a late twin notification, or the next
|
||||||
|
`prompt_version` re-post, from booking the hold again. Same for the intake
|
||||||
|
router's idempotency lookup. Do not "fix" those.
|
||||||
|
- **Outside this repo, nothing filters it**: Hermes' `agent_ro` views
|
||||||
|
(`cashflow_monthly` etc.) and ingestion-engine's `instalment_matcher` already
|
||||||
|
ignore `superseded_by_id`/`reconciled_with_id` too, so a voided row is no worse
|
||||||
|
there than any other hidden row.
|
||||||
|
|
||||||
|
### Upcoming annual charges (`GET /api/annual-charges`, migration 0038)
|
||||||
|
|
||||||
|
Card annual fees, loan package fees and annual subscriptions, derived from
|
||||||
|
**payments alone** — independent of the email lane, because a bank announces an
|
||||||
|
annual fee on the statement and nowhere else.
|
||||||
|
|
||||||
|
**`transaction_type = 'fee'` is the universe, not a description match.** The
|
||||||
|
extractor already classifies fees structurally (131 rows), and *every* card and
|
||||||
|
loan annual fee on this ledger is typed that way. The first version of this
|
||||||
|
filtered `transaction_type = 'debit'` and returned **none** of them. The fees are
|
||||||
|
not debits.
|
||||||
|
|
||||||
|
**Two signals, kept apart because they are not equally trustworthy:**
|
||||||
|
|
||||||
|
| confidence | basis |
|
||||||
|
|---|---|
|
||||||
|
| `confirmed` | typed a fee **and** repeated on an annual cadence |
|
||||||
|
| `likely` | typed a fee, seen once, and *named* annual/card/package — the kind is certain, the cadence inferred |
|
||||||
|
| `possible` | no fee wording; a merchant whose every gap is annual **and** whose amount is stable |
|
||||||
|
|
||||||
|
**A naive "two charges ~365 days apart" is useless** — measured on this ledger it
|
||||||
|
ranks a bakery first with **202** such pairs, then a donation, Patreon, Coles.
|
||||||
|
Frequency is the confound. Even "every gap is annual" admits a Target visited
|
||||||
|
twice ($30, then $198). The `possible` tier therefore also requires amount
|
||||||
|
stability (≤20%, which absorbs VicRoads' ~3%/yr CPI drift) and ≥2 observations.
|
||||||
|
It still carries real false positives (H&M, Superdry) — that is what the label is
|
||||||
|
for, and why it must never render like a bank fact.
|
||||||
|
|
||||||
|
**`expected_not_seen` is the more interesting status.** The estimate passed and
|
||||||
|
nothing landed, which means the account stopped being imported, or closed, or the
|
||||||
|
charge arrived under a different label. `account_days_stale` is the
|
||||||
|
discriminator: Westpac at 0 days stale is trustworthy; ANZ at 392 is simply
|
||||||
|
missing data. Never report these as upcoming.
|
||||||
|
|
||||||
|
**Account continuity (`account_links`, migration 0038).** One card can appear
|
||||||
|
under two identities: Citi Prestige `5258…0253` (to 2025-11-13) became NAB MyCard
|
||||||
|
`1015…6227` (from 2025-11-14), same 14th-to-13th cycle, no gap. Without the link
|
||||||
|
the $700 fee reads as two dead one-offs instead of one due again ~Nov 2026. The
|
||||||
|
table is **hand-asserted with evidence** (`source_documents` ids), not inferred —
|
||||||
|
a contiguous-cycle heuristic would join two genuinely different accounts as
|
||||||
|
readily as one renamed one. The resolver walks it forward recursively;
|
||||||
|
`uq_account_links_from` forbids a fork and `account_links_no_self` a self-link, so
|
||||||
|
it terminates. **Display the resolved current bank**, not the one that printed the
|
||||||
|
last fee — naming the retired brand sends you to the wrong bank.
|
||||||
|
|
||||||
|
Known limitation: the merchant signal fragments on label drift. VicRoads rego was
|
||||||
|
paid 2024/2025 as `VICROADS` and 2026 as `VICROADS ONLINE PAYM`, so the series
|
||||||
|
splits and the 2025 estimate shows as `expected_not_seen`. Not fuzzy-matched on
|
||||||
|
purpose — see the rejected pg_trgm merchant grouping.
|
||||||
|
|
||||||
|
### The wallet lane (`/api/wallet/ingest`, `lib/wallet-ingestion.ts`)
|
||||||
|
|
||||||
|
iOS notifications → `capture.wallet_events` → n8n **Wallet Interpret**
|
||||||
|
(`CAp1s5cuzJk2mDaI`, local gemma, every 15 min) → this route. A row per tap,
|
||||||
|
statementless, `source='wallet'`, `source_ref='evt:<id>'`; the statement later
|
||||||
|
confirms it through the reconcile queue. Fixed 2026-10-03 after a month abroad
|
||||||
|
(ticket #266); the rules below are each the fix for something that broke.
|
||||||
|
|
||||||
|
- **Storage convention differs from the order lane.** `amount` = `amount_aud` =
|
||||||
|
a *provisional AUD* figure; the native figure is `foreign_currency_amount`.
|
||||||
|
`NATIVE_CURRENCY` reads `amount_aud IS NOT NULL` as "amount is AUD" — without
|
||||||
|
that an INR 524 tap displayed as "INR 9.42".
|
||||||
|
- **Provisional rate:** ledger median over *statement* rows (≥5, carries the
|
||||||
|
card's FX fee) → else ECB via Frankfurter (`rate_ecb`) → else `0` with
|
||||||
|
`no_rate_reference`. Never the native figure as AUD (672006 VND once became
|
||||||
|
A$672,006).
|
||||||
|
- **One tap = two notifications** (issuer app + Wallet, 2–8 s apart), merged.
|
||||||
|
A pair must be **≤10 minutes apart AND from different apps**
|
||||||
|
(`expense_metadata.notified_at` / `notif_app`, migration 0039). The old
|
||||||
|
amount-within-±1-day rule merged the UNICEF $50 into an unrelated $50 charge.
|
||||||
|
- **The whole check→pair→adopt→insert runs under one advisory lock**
|
||||||
|
(`hashtextextended('wallet-ingest',0)`), on the transaction's own connection.
|
||||||
|
n8n posts a poll's items concurrently; without it 11 taps became 22 rows.
|
||||||
|
Prisma cannot deserialise `void` — the lock SELECT needs `::text`.
|
||||||
|
- **A merged half records `wallet_evt:<id>` in the survivor's flags**, so a
|
||||||
|
replay is idempotent. Reconciled wallet rows carry **no** `expense_metadata`
|
||||||
|
(reconcile moves it to the statement twin) — the flag check follows it there.
|
||||||
|
- **Skipped, not booked:** account notices (`overdrawn`, `to avoid fees`, …) and
|
||||||
|
transit-card drawdowns (`Your new/current balance is` — the Octopus *load* is
|
||||||
|
the spend). The fixed-width descriptor parser runs on Westpac bodies only.
|
||||||
|
- **Dated in Australia/Melbourne**, like the statements (UTC slicing filed 28 of
|
||||||
|
134 rows a day early).
|
||||||
|
- **Replay a few events by relabelling their interpretations**
|
||||||
|
(`UPDATE capture.wallet_interpretations SET prompt_version='v1-<why>' WHERE …`),
|
||||||
|
not by bumping `prompt_version`, which re-posts every event ever captured.
|
||||||
|
|
||||||
### Cash and reconciliation
|
### Cash and reconciliation
|
||||||
|
|
||||||
`payment_method` in (`cash`, `credits`) excludes a transaction from
|
`payment_method` in (`cash`, `credits`) excludes a transaction from
|
||||||
@@ -640,7 +856,55 @@ and the cross-repo contract are in smarthome `plans/intake-router-slice-plan.md`
|
|||||||
`intake_use_candidate` / `intake_create` buttons; a transaction is created
|
`intake_use_candidate` / `intake_create` buttons; a transaction is created
|
||||||
only on the explicit tap, ING-10-style (`statement_id NULL`, `owner_id` from
|
only on the explicit tap, ING-10-style (`statement_id NULL`, `owner_id` from
|
||||||
`shared_by` → `participants.email`, else `DEFAULT_OWNER_ID`;
|
`shared_by` → `participants.email`, else `DEFAULT_OWNER_ID`;
|
||||||
`source='intake'`, `source_ref=batch_id` so a second press is a no-op).
|
`source='intake'`, `source_ref=batch_id` so a second press is a no-op —
|
||||||
|
written as `ON CONFLICT … DO NOTHING`, because a SELECT-then-INSERT loses a
|
||||||
|
double tap). A non-AUD receipt is booked the way `order-ingestion.ts` books
|
||||||
|
one: `amount` native, `amount_aud` **NULL**, `foreign_currency_*` set — no FX
|
||||||
|
rate exists at intake and inventing one is worse than admitting there is none.
|
||||||
|
- **A NULL platform means "no platform", not "any platform".** An in-store tap
|
||||||
|
(`coles`, `woolworths`, `other`) matches only rows whose `expense_metadata`
|
||||||
|
carries no platform of its own. The wildcard reading let a till receipt match
|
||||||
|
the Uber Eats order for the same shop on the same day and overwrite its items.
|
||||||
|
The **candidates** query is the opposite and deliberately so: it has no
|
||||||
|
platform filter at all, because the card is a human choice shown precisely
|
||||||
|
because the exact key missed — the platform is one of the things that may be
|
||||||
|
wrong, so the list must widen there, not narrow.
|
||||||
|
- **The Create tap re-runs the match before it inserts**, and checks for a
|
||||||
|
resolved sibling batch. The card is minutes old by the time it is pressed and
|
||||||
|
the email/order bridge books orders on its own schedule; `findDuplicateBatch`
|
||||||
|
ignores *unresolved* batches by design, so two pending cards for one order
|
||||||
|
would otherwise each book it.
|
||||||
|
- **One `expense_metadata` row, agreed on by every query that resolves one.** A
|
||||||
|
transaction can own a row *and* be the `matched_transaction_id` of another;
|
||||||
|
`matchTransaction` returns the row it resolved through and `replaceLineItems`
|
||||||
|
writes to exactly that one. Every reader orders by
|
||||||
|
`(transaction_id = $1) DESC **NULLS LAST**, id` — the match, the write, the
|
||||||
|
order-detail GET, `getTransactions`' provenance LATERAL, `merchantForTransaction`
|
||||||
|
and the Slack details modal. The `NULLS LAST` is load-bearing: Postgres orders
|
||||||
|
`DESC` as `NULLS FIRST` and the row that merely *points* at the transaction has
|
||||||
|
a NULL `transaction_id`, so the clause meant to prefer the owned row preferred
|
||||||
|
the other one. Measured 2026-08-21: 667 rows, 147 pointing-only, **zero**
|
||||||
|
transactions resolving to more than one row — the ordering changes nothing
|
||||||
|
today and exists so the readers still agree when multiplicity appears.
|
||||||
|
- **The create is guarded by a transaction-scoped advisory lock on the
|
||||||
|
PURCHASE**, not the batch. `ON CONFLICT (source, source_ref)` handles a double
|
||||||
|
tap on one card; two *different* batches for one purchase have different
|
||||||
|
`source_ref`s and both clear their match and duplicate checks before either
|
||||||
|
inserts. The lock has to live inside `withTransaction` (`src/lib/db.ts`):
|
||||||
|
`queryRaw` is `$queryRawUnsafe` on a **pooled** client, so every call is its
|
||||||
|
own implicit transaction and `pg_advisory_xact_lock` taken in a statement of
|
||||||
|
its own is released before the next statement runs. The loser reports
|
||||||
|
`duplicate` pointing at the winner's transaction — the same answer the
|
||||||
|
sequential path gives.
|
||||||
|
- **`replaceLineItems` snapshots the undo baseline BEFORE it writes, on both
|
||||||
|
paths**, and treats an UPDATE that touched no row as a failure rather than a
|
||||||
|
success. The metadata INSERT is `ON CONFLICT (transaction_id) DO UPDATE` and
|
||||||
|
carries the batch's **currency** — a hard-coded `'AUD'` made `OrderDetails`
|
||||||
|
print `$` on a USD order, because it prefers `expense_metadata.currency`.
|
||||||
|
- **`intake_batches.previous_line_items` is the undo baseline, written once.**
|
||||||
|
Returning whatever the write just read makes a pantry retry get its own write
|
||||||
|
back as the "previous" state, so undo restores the thing it undid. Recorded
|
||||||
|
under `COALESCE(previous_line_items, …)` and read back.
|
||||||
- **Grocery never writes finance here.** The ORIGINAL files go to pantry
|
- **Grocery never writes finance here.** The ORIGINAL files go to pantry
|
||||||
`POST /api/capture` with `intake.finance_target_transaction_id` +
|
`POST /api/capture` with `intake.finance_target_transaction_id` +
|
||||||
`finance_action: enrich|create`; pantry's confirm calls
|
`finance_action: enrich|create`; pantry's confirm calls
|
||||||
@@ -651,6 +915,25 @@ and the cross-repo contract are in smarthome `plans/intake-router-slice-plan.md`
|
|||||||
delivery order's total is items + fees − promotions ($118.94 of Coles lines
|
delivery order's total is items + fees − promotions ($118.94 of Coles lines
|
||||||
became $85.74), so checking lines against the total would hold every
|
became $85.74), so checking lines against the total would hold every
|
||||||
delivery order. Tolerance $1 or 2%; a failure is `held`, nothing written.
|
delivery order. Tolerance $1 or 2%; a failure is `held`, nothing written.
|
||||||
|
- **Nothing that cannot be a receipt reaches the model, and the prompt names no
|
||||||
|
real shop.** On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so
|
||||||
|
a batch arrived as the five characters `Image` — and the model returned a
|
||||||
|
whole receipt: `Burgertory (Braybrook)`, two "The Pride" lines, $41.15. None
|
||||||
|
of it existed; "Burgertory" was in `ORDER_ITEM_PROMPT` as the example of how
|
||||||
|
to print a branch in parentheses. Only the absent date kept it out of the
|
||||||
|
matcher, and the invented header and items were still stored on the batch as
|
||||||
|
`needs_header`. Three layers now, in order: `textCouldBeReceipt()` rejects a
|
||||||
|
TEXT part with fewer than two words, no digit, or under 40 characters with no
|
||||||
|
money figure (`TOTAL $17.00` passes — length alone is *not* the test)
|
||||||
|
**before** any model call; the prompt's merchant examples are
|
||||||
|
`PROMPT_PLACEHOLDER_MERCHANTS` format placeholders and it is told to return
|
||||||
|
nulls for a non-receipt; and `confabulationReason()` catches what the
|
||||||
|
pre-check cannot see — a PDF that renders to junk — plus any answer echoing a
|
||||||
|
placeholder. All three file `unsupported`, never `needs_header`: a
|
||||||
|
confabulated header is not a partial one. Same shape as the ingestion
|
||||||
|
engine's 2026-08-05 empty-body defect — **a stage that can be handed nothing
|
||||||
|
must say so, because the model downstream always answers, and it answers out
|
||||||
|
of its prompt.**
|
||||||
- **Names stay verbatim.** Burgertory calls its burgers "Michael" and "Joel";
|
- **Names stay verbatim.** Burgertory calls its burgers "Michael" and "Joel";
|
||||||
the prompt forbids canonicalising and the stored shape is the order lane's
|
the prompt forbids canonicalising and the stored shape is the order lane's
|
||||||
(`qty`, `description`, `amount`, `options`) so `order-details.tsx` renders it.
|
(`qty`, `description`, `amount`, `options`) so `order-details.tsx` renders it.
|
||||||
|
|||||||
@@ -32,10 +32,20 @@ CREATE TABLE IF NOT EXISTS intake_batches (
|
|||||||
status text NOT NULL,
|
status text NOT NULL,
|
||||||
transaction_id integer,
|
transaction_id integer,
|
||||||
capture_id integer,
|
capture_id integer,
|
||||||
|
-- The undo baseline: what `expense_metadata.line_items` held before this
|
||||||
|
-- batch first replaced it. Recorded here rather than returned from the write,
|
||||||
|
-- because pantry re-calls the enrich endpoint on every confirm and after a
|
||||||
|
-- timeout — a caller that hands back whatever it just read gives a retry its
|
||||||
|
-- OWN write as the "previous" state, and undo then restores the thing it was
|
||||||
|
-- meant to reverse. Written under COALESCE, so only the first snapshot sticks.
|
||||||
|
previous_line_items jsonb,
|
||||||
created_at timestamptz NOT NULL DEFAULT now(),
|
created_at timestamptz NOT NULL DEFAULT now(),
|
||||||
resolved_at timestamptz
|
resolved_at timestamptz
|
||||||
);
|
);
|
||||||
|
|
||||||
|
-- For a database that already ran an earlier draft of this file.
|
||||||
|
ALTER TABLE intake_batches ADD COLUMN IF NOT EXISTS previous_line_items jsonb;
|
||||||
|
|
||||||
COMMENT ON TABLE intake_batches IS
|
COMMENT ON TABLE intake_batches IS
|
||||||
'One row per upload batch through POST /api/intake. Persisted before '
|
'One row per upload batch through POST /api/intake. Persisted before '
|
||||||
'extraction so Slack taps can act on it later and re-shares are detected. '
|
'extraction so Slack taps can act on it later and re-shares are detected. '
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
-- transaction_search: resolve expense_metadata by BOTH links, without fanning out.
|
||||||
|
--
|
||||||
|
-- THE DEFECT (measured 2026-08-22): the view joined only on
|
||||||
|
-- `em.transaction_id = t.id`, so it silently omitted every card-settled order,
|
||||||
|
-- which carries only `matched_transaction_id`. 148 transactions are affected --
|
||||||
|
-- Glovo orders in Milan, Uber Eats, all `source='order-bridge'` -- and for each
|
||||||
|
-- of them the view reports NULL payment_method / order_reference / line_items
|
||||||
|
-- while the data sits in the table.
|
||||||
|
--
|
||||||
|
-- That is not cosmetic. This view is the agent read surface: Hermes'
|
||||||
|
-- postgres-personal MCP selects payment_method and order_reference from it
|
||||||
|
-- (personal-agent-gateway/mcp-servers/postgres-personal/queries.py), and
|
||||||
|
-- home-dashboard reads it via dash_ro. CLAUDE.md states the rule the view broke:
|
||||||
|
-- "That lookup must match transaction_id OR matched_transaction_id: a
|
||||||
|
-- card-settled order has only the latter."
|
||||||
|
--
|
||||||
|
-- NOT a fan-out fix, contrary to a first reading. The original join CANNOT fan
|
||||||
|
-- out: expense_metadata_transaction_id_key is UNIQUE on transaction_id, so it
|
||||||
|
-- matched at most one row by construction. Sums taken over this view were never
|
||||||
|
-- inflated.
|
||||||
|
--
|
||||||
|
-- But broadening the join is exactly what INTRODUCES that risk, which is why
|
||||||
|
-- LATERAL is used rather than a second OR'd condition. transaction_id and
|
||||||
|
-- matched_transaction_id are enforced by two SEPARATE unique indexes
|
||||||
|
-- (expense_metadata_transaction_id_key, uq_expense_matched_txn), so one
|
||||||
|
-- transaction can legally be pointed at by two DIFFERENT rows -- one via each
|
||||||
|
-- link. Zero such transactions exist today; the LATERAL means that stays a
|
||||||
|
-- non-event rather than a silent duplicate in the agent's spending totals.
|
||||||
|
--
|
||||||
|
-- Tie-break matches every other reader (getTransactions, replaceLineItems,
|
||||||
|
-- matchTransaction, /api/transactions/[id]/order), so the row this view resolves
|
||||||
|
-- is the row the UI shows.
|
||||||
|
--
|
||||||
|
-- Verified before applying: row count unchanged at 4760 (= count(transactions)),
|
||||||
|
-- sum(amount) over debits unchanged at 1514412.89 -- amount comes from
|
||||||
|
-- transactions, never from the metadata row -- and the only difference is the
|
||||||
|
-- 148 rows that gain the metadata they always should have had.
|
||||||
|
--
|
||||||
|
-- CREATE OR REPLACE VIEW DROPS GRANTS. dash_ro (home-dashboard) and agent_ro
|
||||||
|
-- (Hermes) both read this view and losing either breaks a surface with no error
|
||||||
|
-- at replace time, so both are re-granted below.
|
||||||
|
|
||||||
|
CREATE OR REPLACE VIEW transaction_search AS
|
||||||
|
SELECT t.id,
|
||||||
|
t.transaction_date,
|
||||||
|
t.description,
|
||||||
|
COALESCE(t.merchant_normalized, t.merchant_name) AS merchant,
|
||||||
|
t.category,
|
||||||
|
t.transaction_type,
|
||||||
|
COALESCE(t.amount_aud, t.amount) AS amount,
|
||||||
|
t.foreign_currency_amount,
|
||||||
|
t.foreign_currency_code,
|
||||||
|
t.location,
|
||||||
|
s.bank_name,
|
||||||
|
s.card_name,
|
||||||
|
s.account_number,
|
||||||
|
s.statement_type,
|
||||||
|
s.billing_end_date,
|
||||||
|
em.payment_method,
|
||||||
|
em.order_reference,
|
||||||
|
em.line_items
|
||||||
|
FROM transactions t
|
||||||
|
LEFT JOIN statements s ON s.id = t.statement_id
|
||||||
|
LEFT JOIN LATERAL (
|
||||||
|
SELECT em2.payment_method, em2.order_reference, em2.line_items
|
||||||
|
FROM expense_metadata em2
|
||||||
|
WHERE em2.transaction_id = t.id OR em2.matched_transaction_id = t.id
|
||||||
|
ORDER BY (em2.transaction_id = t.id) DESC NULLS LAST, em2.id
|
||||||
|
LIMIT 1
|
||||||
|
) em ON true;
|
||||||
|
|
||||||
|
GRANT SELECT ON transaction_search TO dash_ro;
|
||||||
|
GRANT SELECT ON transaction_search TO agent_ro;
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
-- Normalise foreign_currency_code to ISO-4217, and clear two sentinel values.
|
||||||
|
--
|
||||||
|
-- WHY THIS MATTERS: exact-code equality is the whole basis of FX reconciliation.
|
||||||
|
-- ingestion-engine's order_transaction_bridge matches foreign orders with
|
||||||
|
-- AND t.foreign_currency_code = %(ccy)s
|
||||||
|
-- AND t.foreign_currency_amount = %(total)s
|
||||||
|
-- so a row labelled 'BAHT' can never match a THB order, and a row labelled
|
||||||
|
-- 'null' can never match anything. 226 of 1,632 coded rows (14%) were
|
||||||
|
-- unmatchable for this reason alone, which silently degrades a shipped job.
|
||||||
|
--
|
||||||
|
-- THE SENTINELS. 187 rows carry the literal 4-character STRING 'null' rather
|
||||||
|
-- than SQL NULL, and 42 of those also carry foreign_currency_amount = -1.00
|
||||||
|
-- exactly. Neither is data: they are "no value" markers that survived a truthy
|
||||||
|
-- guard. Root cause is in the CC statement processors' Parse node, which does
|
||||||
|
-- foreign_currency_code: t.foreign_currency_code || null
|
||||||
|
-- and a non-empty string is truthy, so a model that answers with the STRING
|
||||||
|
-- "null" (rather than JSON null) passes straight through. Same for -1. Fixed
|
||||||
|
-- workflow-side in the same change; this migration cleans what already landed.
|
||||||
|
-- Not purely historical -- 155 rows in 2026-03, but also 8 in May, 3 in July
|
||||||
|
-- and 21 in August 2026.
|
||||||
|
--
|
||||||
|
-- MAPPINGS WERE VERIFIED AGAINST IMPLIED RATE, NOT TRUSTED FROM THE LABEL,
|
||||||
|
-- because a row whose rate contradicts its claimed currency is a broken ledger
|
||||||
|
-- row rather than a mislabelled one (the trap order_transaction_bridge's rate
|
||||||
|
-- guard exists for: 2 of its first 73 matches were broken rows, not bad
|
||||||
|
-- matches). Median amount/foreign_currency_amount per raw code vs the ISO
|
||||||
|
-- median over statement rows (n>=5):
|
||||||
|
-- BAHT 0.0433 vs THB 0.0470 INDIAN RUPEE 0.0183 vs INR 0.0180
|
||||||
|
-- U.A.E. DIRHAM 0.4287 vs AED 0.4184 US DOLLAR 1.4054 vs USD 1.5513
|
||||||
|
-- NZ DOLLAR 0.8243 vs NZD 0.9311 EURO 1.6342 vs EUR 1.6964
|
||||||
|
-- Every label agrees with its rate. foreign_currency_amount is untouched by the
|
||||||
|
-- relabels -- only the code was ever wrong, so this is a relabel, never a
|
||||||
|
-- recompute.
|
||||||
|
--
|
||||||
|
-- EXPECTED ROW COUNTS (verified pre-flight, includes superseded rows):
|
||||||
|
-- code 'null' -> NULL 187 BAHT -> THB 13
|
||||||
|
-- amount -1.00 -> NULL 42 U.A.E. DIRHAM -> AED 10
|
||||||
|
-- NZ DOLLAR -> NZD 6 US DOLLAR -> USD 4
|
||||||
|
-- INDIAN RUPEE -> INR 4 EURO -> EUR 2
|
||||||
|
-- Pre-flight also confirmed: no -1.00 amount exists outside the 'null'-code
|
||||||
|
-- rows, and no other non-ISO code exists.
|
||||||
|
|
||||||
|
BEGIN;
|
||||||
|
|
||||||
|
-- Sentinels first: the -1.00 amounts live only on 'null'-code rows, so clear
|
||||||
|
-- them before that code is nulled and the set becomes unfindable.
|
||||||
|
UPDATE transactions SET foreign_currency_amount = NULL
|
||||||
|
WHERE foreign_currency_amount = -1.00 AND foreign_currency_code = 'null';
|
||||||
|
|
||||||
|
UPDATE transactions SET foreign_currency_code = NULL
|
||||||
|
WHERE foreign_currency_code = 'null';
|
||||||
|
|
||||||
|
-- Word-form relabels.
|
||||||
|
UPDATE transactions SET foreign_currency_code = 'THB' WHERE foreign_currency_code = 'BAHT';
|
||||||
|
UPDATE transactions SET foreign_currency_code = 'AED' WHERE foreign_currency_code = 'U.A.E. DIRHAM';
|
||||||
|
UPDATE transactions SET foreign_currency_code = 'NZD' WHERE foreign_currency_code = 'NZ DOLLAR';
|
||||||
|
UPDATE transactions SET foreign_currency_code = 'USD' WHERE foreign_currency_code = 'US DOLLAR';
|
||||||
|
UPDATE transactions SET foreign_currency_code = 'INR' WHERE foreign_currency_code = 'INDIAN RUPEE';
|
||||||
|
UPDATE transactions SET foreign_currency_code = 'EUR' WHERE foreign_currency_code = 'EURO';
|
||||||
|
|
||||||
|
-- Postcondition: every remaining code is ISO-4217 shaped. Fails the whole
|
||||||
|
-- transaction rather than leaving a partial normalisation behind.
|
||||||
|
DO $$
|
||||||
|
DECLARE bad int;
|
||||||
|
BEGIN
|
||||||
|
SELECT count(*) INTO bad FROM transactions
|
||||||
|
WHERE foreign_currency_code IS NOT NULL AND foreign_currency_code !~ '^[A-Z]{3}$';
|
||||||
|
IF bad <> 0 THEN
|
||||||
|
RAISE EXCEPTION 'normalisation incomplete: % non-ISO codes remain', bad;
|
||||||
|
END IF;
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
@@ -0,0 +1,152 @@
|
|||||||
|
-- Append-only change log for the ledger and its curation children.
|
||||||
|
--
|
||||||
|
-- WHY THIS EXISTS: on 2026-08-23 the CC Statement Processor's `Run Reconciliation`
|
||||||
|
-- node stamped `statement_id` onto seven statementless wallet/pantry rows without
|
||||||
|
-- setting `reconciled_with_id` or superseding the statement's own extracted line.
|
||||||
|
-- The purchase counted twice, and because finance-app's reconcile queue is
|
||||||
|
-- `WHERE statement_id IS NULL`, the stamp also hid those rows from the only thing
|
||||||
|
-- that could repair them. Answering "was anything lost?" required restoring the
|
||||||
|
-- 02:30 pg_dump into a scratch database and diffing it column by column, because
|
||||||
|
-- NOTHING in this estate records what changed. That is the gap this closes.
|
||||||
|
--
|
||||||
|
-- WHY AT THE DATABASE AND NOT IN THE APP: the ledger has at least five independent
|
||||||
|
-- writers -- finance-app (Prisma), n8n (raw SQL in a Postgres node), pantry-app,
|
||||||
|
-- the ingestion-engine runner (psycopg), and ad-hoc `docker exec psql`. The 4a bug
|
||||||
|
-- was an n8n raw-SQL UPDATE; an application-level audit would not have seen it at
|
||||||
|
-- all. A trigger cannot be bypassed by choice of client.
|
||||||
|
--
|
||||||
|
-- WHY NOT POSTGRES' OWN LOGS: checked live on this instance -- wal_level=replica
|
||||||
|
-- (so WAL cannot be logically decoded into row changes), wal_keep_size=0 and
|
||||||
|
-- archive_mode=off (so WAL is recycled almost immediately), log_statement=none,
|
||||||
|
-- logging_collector=off, track_commit_timestamp=off, and pgaudit is not installed.
|
||||||
|
-- Postgres was recording nothing usable. It has no built-in audit or temporal
|
||||||
|
-- tables; this always has to be built.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS transaction_audit (
|
||||||
|
id bigserial PRIMARY KEY,
|
||||||
|
changed_at timestamptz NOT NULL DEFAULT now(),
|
||||||
|
-- Groups every row touched by one logical edit. `applyReconcileMatch` moves
|
||||||
|
-- overrides, tags, splits and metadata in a single transaction; without this
|
||||||
|
-- they read as four unrelated events.
|
||||||
|
txid bigint NOT NULL DEFAULT txid_current(),
|
||||||
|
table_name text NOT NULL,
|
||||||
|
op text NOT NULL CHECK (op IN ('INSERT','UPDATE','DELETE')),
|
||||||
|
-- Full primary key as an object, because transaction_tags is composite
|
||||||
|
-- (transaction_id, tag_id) and the others are a bare `id`.
|
||||||
|
row_key jsonb NOT NULL,
|
||||||
|
-- Denormalised so "everything that ever happened to transaction N" is one
|
||||||
|
-- index scan rather than a five-way union.
|
||||||
|
transaction_id bigint,
|
||||||
|
-- On UPDATE these hold ONLY the columns that actually changed. Storing whole
|
||||||
|
-- rows would bury a one-column edit in forty unchanged fields, and
|
||||||
|
-- expense_metadata.line_items alone can be kilobytes.
|
||||||
|
old_data jsonb,
|
||||||
|
new_data jsonb,
|
||||||
|
-- Attribution is partial and that is understood: every client connects as the
|
||||||
|
-- same `personal` role. app_name is only as good as what each client sets;
|
||||||
|
-- client_addr distinguishes containers and is NULL for `docker exec` over the
|
||||||
|
-- unix socket, which is itself a useful signal.
|
||||||
|
db_user text NOT NULL DEFAULT current_user,
|
||||||
|
app_name text,
|
||||||
|
client_addr inet
|
||||||
|
);
|
||||||
|
|
||||||
|
COMMENT ON TABLE transaction_audit IS
|
||||||
|
'Append-only row-level change log for transactions + curation children. Never UPDATE or DELETE from this table.';
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS transaction_audit_txn_time
|
||||||
|
ON transaction_audit (transaction_id, changed_at DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS transaction_audit_time
|
||||||
|
ON transaction_audit (changed_at DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS transaction_audit_txid
|
||||||
|
ON transaction_audit (txid);
|
||||||
|
CREATE INDEX IF NOT EXISTS transaction_audit_table_time
|
||||||
|
ON transaction_audit (table_name, changed_at DESC);
|
||||||
|
|
||||||
|
-- TG_ARGV[0]: comma-separated candidate columns for the owning transaction id;
|
||||||
|
-- the first non-null wins (expense_metadata carries the id in
|
||||||
|
-- transaction_id OR matched_transaction_id -- a card-settled order
|
||||||
|
-- only ever has the latter).
|
||||||
|
-- TG_ARGV[1]: comma-separated primary key columns.
|
||||||
|
CREATE OR REPLACE FUNCTION log_transaction_change() RETURNS trigger
|
||||||
|
LANGUAGE plpgsql SECURITY DEFINER
|
||||||
|
SET search_path = public, pg_temp
|
||||||
|
AS $fn$
|
||||||
|
DECLARE
|
||||||
|
v_full jsonb;
|
||||||
|
v_old jsonb;
|
||||||
|
v_new jsonb;
|
||||||
|
v_key jsonb := '{}'::jsonb;
|
||||||
|
v_txn bigint;
|
||||||
|
k text;
|
||||||
|
BEGIN
|
||||||
|
IF TG_OP = 'INSERT' THEN
|
||||||
|
v_new := to_jsonb(NEW);
|
||||||
|
v_full := v_new;
|
||||||
|
ELSIF TG_OP = 'DELETE' THEN
|
||||||
|
v_old := to_jsonb(OLD);
|
||||||
|
v_full := v_old;
|
||||||
|
ELSE
|
||||||
|
v_old := to_jsonb(OLD);
|
||||||
|
v_new := to_jsonb(NEW);
|
||||||
|
v_full := v_new;
|
||||||
|
-- Reduce to the changed columns only. IS DISTINCT FROM so a NULL->NULL
|
||||||
|
-- comparison counts as unchanged rather than evaluating to NULL.
|
||||||
|
FOR k IN SELECT jsonb_object_keys(v_old) LOOP
|
||||||
|
IF (v_old -> k) IS NOT DISTINCT FROM (v_new -> k) THEN
|
||||||
|
v_old := v_old - k;
|
||||||
|
v_new := v_new - k;
|
||||||
|
END IF;
|
||||||
|
END LOOP;
|
||||||
|
-- A no-op UPDATE is noise, and finance-app's upserts issue plenty of them.
|
||||||
|
IF v_old = '{}'::jsonb THEN
|
||||||
|
RETURN NULL;
|
||||||
|
END IF;
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
FOREACH k IN ARRAY string_to_array(TG_ARGV[1], ',') LOOP
|
||||||
|
v_key := v_key || jsonb_build_object(k, v_full -> k);
|
||||||
|
END LOOP;
|
||||||
|
|
||||||
|
FOREACH k IN ARRAY string_to_array(TG_ARGV[0], ',') LOOP
|
||||||
|
IF v_txn IS NULL AND (v_full ->> k) IS NOT NULL THEN
|
||||||
|
v_txn := (v_full ->> k)::bigint;
|
||||||
|
END IF;
|
||||||
|
END LOOP;
|
||||||
|
|
||||||
|
INSERT INTO transaction_audit
|
||||||
|
(table_name, op, row_key, transaction_id, old_data, new_data, app_name, client_addr)
|
||||||
|
VALUES
|
||||||
|
(TG_TABLE_NAME, TG_OP, v_key, v_txn, v_old, v_new,
|
||||||
|
NULLIF(current_setting('application_name', true), ''), inet_client_addr());
|
||||||
|
|
||||||
|
RETURN NULL; -- AFTER trigger; return value is ignored
|
||||||
|
END
|
||||||
|
$fn$;
|
||||||
|
|
||||||
|
-- AFTER, so a statement that raises never leaves a phantom audit row; the audit
|
||||||
|
-- write is in the same transaction as the change and rolls back with it.
|
||||||
|
DROP TRIGGER IF EXISTS trg_audit_transactions ON transactions;
|
||||||
|
CREATE TRIGGER trg_audit_transactions
|
||||||
|
AFTER INSERT OR UPDATE OR DELETE ON transactions
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('id', 'id');
|
||||||
|
|
||||||
|
DROP TRIGGER IF EXISTS trg_audit_transaction_splits ON transaction_splits;
|
||||||
|
CREATE TRIGGER trg_audit_transaction_splits
|
||||||
|
AFTER INSERT OR UPDATE OR DELETE ON transaction_splits
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
|
||||||
|
|
||||||
|
DROP TRIGGER IF EXISTS trg_audit_transaction_tags ON transaction_tags;
|
||||||
|
CREATE TRIGGER trg_audit_transaction_tags
|
||||||
|
AFTER INSERT OR UPDATE OR DELETE ON transaction_tags
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'transaction_id,tag_id');
|
||||||
|
|
||||||
|
DROP TRIGGER IF EXISTS trg_audit_transaction_overrides ON transaction_overrides;
|
||||||
|
CREATE TRIGGER trg_audit_transaction_overrides
|
||||||
|
AFTER INSERT OR UPDATE OR DELETE ON transaction_overrides
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id', 'id');
|
||||||
|
|
||||||
|
DROP TRIGGER IF EXISTS trg_audit_expense_metadata ON expense_metadata;
|
||||||
|
CREATE TRIGGER trg_audit_expense_metadata
|
||||||
|
AFTER INSERT OR UPDATE OR DELETE ON expense_metadata
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION log_transaction_change('transaction_id,matched_transaction_id', 'id');
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
-- Account continuity across a rebrand, reissue or issuer change.
|
||||||
|
--
|
||||||
|
-- WHY THIS EXISTS: one card can appear in `statements` under two completely
|
||||||
|
-- different identities. The live case is the Citi Prestige card, which moved to
|
||||||
|
-- NAB and was rebranded MyCard:
|
||||||
|
--
|
||||||
|
-- Citibank 5258073330340253 statements ... 2025-11-13
|
||||||
|
-- NAB 1015001000116227 statements 2025-11-14 ...
|
||||||
|
--
|
||||||
|
-- Same 14th-to-13th billing cycle, no gap, one continuous product. Nothing in
|
||||||
|
-- the schema linked them, so anything grouping by (bank_name, account) sees a
|
||||||
|
-- dead account and a new one with no history. For the annual-fee detector that
|
||||||
|
-- is not cosmetic: the $700 fee charged 2024-12-13 and 2025-11-13 reads as
|
||||||
|
-- "stopped" when it is due again around November 2026 under the new name.
|
||||||
|
--
|
||||||
|
-- WHY A TABLE AND NOT AN INFERENCE: contiguous-cycle heuristics would join two
|
||||||
|
-- genuinely different accounts as readily as one renamed one, and a wrong join
|
||||||
|
-- merges two real fee series. There are only a handful of these events in a
|
||||||
|
-- lifetime, each announced by mail, so each link is asserted once with the
|
||||||
|
-- document that proves it. `evidence` is not decoration -- it is how a future
|
||||||
|
-- reader checks the claim instead of trusting it.
|
||||||
|
--
|
||||||
|
-- NOT `account_owner_mappings`: that answers "whose account is this" and keys on
|
||||||
|
-- the raw (bank_name, account_number). This answers "is this the same account as
|
||||||
|
-- that one", and keys on the normalised form.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS account_links (
|
||||||
|
id bigserial PRIMARY KEY,
|
||||||
|
-- Normalised the same way statements.account_number_key is generated
|
||||||
|
-- (non-alphanumerics stripped), so these join without per-call cleanup.
|
||||||
|
-- Amex-style masks keep their X's: they record which digits were redacted.
|
||||||
|
from_bank_name text NOT NULL,
|
||||||
|
from_account_key text NOT NULL,
|
||||||
|
to_bank_name text NOT NULL,
|
||||||
|
to_account_key text NOT NULL,
|
||||||
|
-- When the successor took over. The predecessor's last statement and the
|
||||||
|
-- successor's first are the observable boundary; this is the announced date
|
||||||
|
-- where mail gives one.
|
||||||
|
effective_date date,
|
||||||
|
reason text NOT NULL
|
||||||
|
CHECK (reason IN ('rebrand', 'reissue', 'issuer_change', 'migration')),
|
||||||
|
-- Provenance for the claim, e.g. 'source_documents:32919'. Free text because
|
||||||
|
-- the evidence is not always a document -- sometimes it is a statement pair.
|
||||||
|
evidence text,
|
||||||
|
notes text,
|
||||||
|
created_at timestamptz NOT NULL DEFAULT now(),
|
||||||
|
|
||||||
|
-- One successor per predecessor: a chain, never a fork. A fork would make
|
||||||
|
-- "the current identity of this account" ambiguous, and the resolver walks
|
||||||
|
-- this forward assuming it is not.
|
||||||
|
CONSTRAINT uq_account_links_from UNIQUE (from_bank_name, from_account_key),
|
||||||
|
-- A link to itself would make that walk non-terminating.
|
||||||
|
CONSTRAINT account_links_no_self CHECK (
|
||||||
|
from_bank_name IS DISTINCT FROM to_bank_name
|
||||||
|
OR from_account_key IS DISTINCT FROM to_account_key
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
COMMENT ON TABLE account_links IS
|
||||||
|
'Predecessor -> successor account identity across rebrands/reissues. Hand-asserted, each row carrying the evidence for its claim.';
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS account_links_to
|
||||||
|
ON account_links (to_bank_name, to_account_key);
|
||||||
|
|
||||||
|
-- The one link the mail record proves today.
|
||||||
|
-- source_documents 32919 (2025-10-01) "your Citi Prestige Card account is moving to MyCard"
|
||||||
|
-- source_documents 32618 (2025-11-10) "MyCard is coming soon"
|
||||||
|
-- source_documents 32539 (2025-11-24) "The new MyCard App and MyCard Online are now live"
|
||||||
|
-- Statement boundary: Citibank ends 2025-11-13, NAB begins 2025-11-14.
|
||||||
|
INSERT INTO account_links
|
||||||
|
(from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason, evidence, notes)
|
||||||
|
VALUES
|
||||||
|
('Citibank', '5258073330340253', 'NAB', '1015001000116227', '2025-11-14', 'issuer_change',
|
||||||
|
'source_documents:32919,32618,32539',
|
||||||
|
'Citi Prestige -> MyCard Prestige. Billing cycle (14th-13th) continues unbroken across the boundary.')
|
||||||
|
ON CONFLICT (from_bank_name, from_account_key) DO NOTHING;
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
-- When, and from which app, a wallet notification arrived.
|
||||||
|
--
|
||||||
|
-- WHY: one tap emits two notifications 2-8 seconds apart -- the issuer app
|
||||||
|
-- (card tail) and Apple Wallet (merchant, suburb) -- and wallet-ingestion.ts
|
||||||
|
-- merges them. It used to recognise the pair by amount alone, within +/-1 DAY,
|
||||||
|
-- which is wide enough to fold two different purchases together: on 2026-10-01
|
||||||
|
-- an Amex "$50.00 at A MEDIUM" notification merged into the UNICEF $50 donation
|
||||||
|
-- from 17 hours earlier and renamed it. A real pair is seconds apart and comes
|
||||||
|
-- from two DIFFERENT apps; two notifications from one app are two charges.
|
||||||
|
-- Neither fact was stored anywhere on the ledger side, so the matcher could not
|
||||||
|
-- ask. These columns let it.
|
||||||
|
--
|
||||||
|
-- Backfilled from capture.wallet_events through source_message_id ('evt:<id>'),
|
||||||
|
-- which every wallet-written metadata row carries.
|
||||||
|
|
||||||
|
ALTER TABLE expense_metadata
|
||||||
|
ADD COLUMN IF NOT EXISTS notified_at timestamptz,
|
||||||
|
ADD COLUMN IF NOT EXISTS notif_app text;
|
||||||
|
|
||||||
|
UPDATE expense_metadata em
|
||||||
|
SET notified_at = e.received_at,
|
||||||
|
notif_app = e.notif_app
|
||||||
|
FROM capture.wallet_events e
|
||||||
|
WHERE em.source = 'wallet'
|
||||||
|
AND em.source_message_id = 'evt:' || e.id
|
||||||
|
AND em.notified_at IS NULL;
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
-- A statementless row for a charge that never posted.
|
||||||
|
--
|
||||||
|
-- WHY: the wallet lane books every card notification as a purchase, and a card
|
||||||
|
-- notification is sent for an AUTHORISATION, not a settlement. When the
|
||||||
|
-- merchant releases the hold the bank never posts the charge, so no statement
|
||||||
|
-- line ever arrives: the row sits in the reconcile queue forever and counts as
|
||||||
|
-- spend in every figure. Txn 7951: Airbnb HMXAHAW49Y, $1,040.41 held on Amex
|
||||||
|
-- 1000 on 2026-09-24, booking cancelled 25 Sep inside the free window, hold
|
||||||
|
-- released ("We removed the $1,040.41 (AUD) payment hold"). Statement 219
|
||||||
|
-- (11 Sep - 10 Oct) has no such line and reconciles to its header without it.
|
||||||
|
--
|
||||||
|
-- Neither existing exclusion fits. reconciled_with_id needs a statement twin
|
||||||
|
-- and superseded_by_id needs a surviving row (and may not point at itself,
|
||||||
|
-- transactions_no_self_supersede). There is no survivor: nothing happened.
|
||||||
|
--
|
||||||
|
-- Not a DELETE: every child of transactions cascades, and the row is also the
|
||||||
|
-- sink the wallet lane checks before re-posting an event. Deleting it would let
|
||||||
|
-- the next prompt_version bump re-create it.
|
||||||
|
|
||||||
|
ALTER TABLE transactions
|
||||||
|
ADD COLUMN IF NOT EXISTS voided_at timestamptz,
|
||||||
|
ADD COLUMN IF NOT EXISTS void_reason text;
|
||||||
|
|
||||||
|
COMMENT ON COLUMN transactions.voided_at IS
|
||||||
|
'Set when this statementless row records a charge that never posted (e.g. a released authorisation hold). Excluded from every figure and from the reconcile queue, kept for its children and audit trail. NULL = live.';
|
||||||
|
|
||||||
|
-- A reason is mandatory: a void with no explanation is indistinguishable from
|
||||||
|
-- a mistake six months later.
|
||||||
|
ALTER TABLE transactions
|
||||||
|
ADD CONSTRAINT transactions_void_has_reason
|
||||||
|
CHECK ((voided_at IS NULL AND void_reason IS NULL)
|
||||||
|
OR (voided_at IS NOT NULL AND btrim(coalesce(void_reason, '')) <> ''));
|
||||||
|
|
||||||
|
-- Statement rows cannot be voided. A statement line is the bank's own record
|
||||||
|
-- of a posted charge, and the statement's rows must keep summing to its
|
||||||
|
-- header; a wrong statement row is a supersede, not a void.
|
||||||
|
ALTER TABLE transactions
|
||||||
|
ADD CONSTRAINT transactions_void_statementless
|
||||||
|
CHECK (voided_at IS NULL OR statement_id IS NULL);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_transactions_voided
|
||||||
|
ON transactions (voided_at)
|
||||||
|
WHERE voided_at IS NOT NULL;
|
||||||
@@ -190,7 +190,9 @@ model transactions {
|
|||||||
owner_id Int?
|
owner_id Int?
|
||||||
reconciled_with_id Int?
|
reconciled_with_id Int?
|
||||||
superseded_by_id Int?
|
superseded_by_id Int?
|
||||||
principal_amount Decimal? @db.Decimal(12, 2)
|
voided_at DateTime? @db.Timestamptz(6) // a charge that never posted (migration 0040)
|
||||||
|
void_reason String?
|
||||||
|
principal_amount Decimal? @db.Decimal(12, 2)
|
||||||
interest_amount Decimal? @db.Decimal(12, 2)
|
interest_amount Decimal? @db.Decimal(12, 2)
|
||||||
source String? // feed this row came from, e.g. "frollo"; NULL = statement/manual (migration 0028)
|
source String? // feed this row came from, e.g. "frollo"; NULL = statement/manual (migration 0028)
|
||||||
source_ref String? // the provider's own transaction id — idempotency key for re-imports
|
source_ref String? // the provider's own transaction id — idempotency key for re-imports
|
||||||
@@ -237,6 +239,9 @@ model expense_metadata {
|
|||||||
tender_raw String?
|
tender_raw String?
|
||||||
receipt_sha256 String?
|
receipt_sha256 String?
|
||||||
receipt_group String?
|
receipt_group String?
|
||||||
|
// 0039 — when/where a wallet notification arrived; the pair merge needs both.
|
||||||
|
notified_at DateTime? @db.Timestamptz(6)
|
||||||
|
notif_app String?
|
||||||
transaction transactions? @relation(fields: [transaction_id], references: [id], onDelete: Cascade)
|
transaction transactions? @relation(fields: [transaction_id], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
@@unique([source, order_reference], name: "uq_expense_source_order")
|
@@unique([source, order_reference], name: "uq_expense_source_order")
|
||||||
@@ -283,6 +288,8 @@ model intake_batches {
|
|||||||
status String
|
status String
|
||||||
transaction_id Int?
|
transaction_id Int?
|
||||||
capture_id Int?
|
capture_id Int?
|
||||||
|
// The undo baseline, written once per batch (see 0034 and line-items.ts).
|
||||||
|
previous_line_items Json?
|
||||||
created_at DateTime @default(now()) @db.Timestamptz(6)
|
created_at DateTime @default(now()) @db.Timestamptz(6)
|
||||||
resolved_at DateTime? @db.Timestamptz(6)
|
resolved_at DateTime? @db.Timestamptz(6)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,14 +19,23 @@ if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Create test database (ignore error if it already exists)
|
# Drop and recreate, so the schema is rebuilt rather than replayed. Replaying
|
||||||
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres \
|
# the dump over an existing database was a silent no-op for every migration:
|
||||||
-c "CREATE DATABASE $TEST_DB;" 2>/dev/null || true
|
# each CREATE TABLE failed with "already exists" and the old shape survived, so
|
||||||
|
# a new column (0039's expense_metadata.notified_at, 2026-10-03) never reached
|
||||||
|
# the test database. The guard is because this now DROPs: it refuses anything
|
||||||
|
# not named *_test, whatever TEST_DB was set to.
|
||||||
|
case "$TEST_DB" in
|
||||||
|
*_test) ;;
|
||||||
|
*) echo "ERROR: refusing to drop '$TEST_DB' -- the test database must be named *_test"; exit 1 ;;
|
||||||
|
esac
|
||||||
|
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres -q \
|
||||||
|
-c "DROP DATABASE IF EXISTS $TEST_DB WITH (FORCE);" \
|
||||||
|
-c "CREATE DATABASE $TEST_DB;"
|
||||||
|
|
||||||
# Wipe and rebuild schema from production (schema only, no data)
|
|
||||||
echo "Copying schema from $PG_USER to $TEST_DB..."
|
echo "Copying schema from $PG_USER to $TEST_DB..."
|
||||||
docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \
|
docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \
|
||||||
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q
|
| docker exec -i "$PG_CONTAINER" psql -U "$PG_USER" -d "$TEST_DB" -q -v ON_ERROR_STOP=1
|
||||||
|
|
||||||
echo "Schema ready."
|
echo "Schema ready."
|
||||||
|
|
||||||
|
|||||||
@@ -105,6 +105,19 @@ describe("NATIVE_CURRENCY", () => {
|
|||||||
expect(unconverted).toBe(true);
|
expect(unconverted).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("labels a statement-less wallet row AUD, because its amount already is", async () => {
|
||||||
|
// Wallet rows store a provisional AUD figure in amount AND amount_aud, and
|
||||||
|
// the native figure in foreign_currency_amount. Reading the foreign code
|
||||||
|
// here printed "INR 9.42" for an INR 524 tap (txn 8162).
|
||||||
|
const id = await scratchTxn(
|
||||||
|
"transaction_date, description, amount, amount_aud, transaction_type, payment_method, source, source_ref, foreign_currency_amount, foreign_currency_code",
|
||||||
|
"'2026-10-03','Wallet foreign fixture', 9.42, 9.42, 'debit', 'card', 'wallet', 'evt:fixture-native-ccy', 524.00, 'INR'"
|
||||||
|
);
|
||||||
|
const { ccy, unconverted } = await currencyOf(id);
|
||||||
|
expect(ccy).toBe("AUD");
|
||||||
|
expect(unconverted).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
it("prefers the statement's currency over the foreign-charge record", async () => {
|
it("prefers the statement's currency over the foreign-charge record", async () => {
|
||||||
// Opposite convention: on an AUD statement, `amount` is AUD and
|
// Opposite convention: on an AUD statement, `amount` is AUD and
|
||||||
// foreign_currency_code merely notes what was originally charged. Reading
|
// foreign_currency_code merely notes what was originally charged. Reading
|
||||||
|
|||||||
@@ -0,0 +1,170 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants, mockDbWithPool } from "./helpers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Upcoming annual charges from payments alone.
|
||||||
|
*
|
||||||
|
* The exclusions carry this file. Anything can be made to find a card annual
|
||||||
|
* fee; the difficulty is not also reporting the monthly account-servicing fee,
|
||||||
|
* the per-transfer Wise charge, and the restaurant visited twice a year apart.
|
||||||
|
* Measured on the real ledger, a naive "two charges ~365 days apart" ranks a
|
||||||
|
* bakery first with 202 such pairs.
|
||||||
|
*/
|
||||||
|
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let getUpcomingAnnualCharges: typeof import("../../lib/annual-charges").getUpcomingAnnualCharges;
|
||||||
|
|
||||||
|
let acctSeq = 0;
|
||||||
|
|
||||||
|
/** A statement plus one transaction on it, for a named account. */
|
||||||
|
async function charge(
|
||||||
|
bank: string,
|
||||||
|
acct: string,
|
||||||
|
date: string,
|
||||||
|
description: string,
|
||||||
|
amount: number,
|
||||||
|
type = "fee"
|
||||||
|
) {
|
||||||
|
const key = `${bank}:${acct}`;
|
||||||
|
let stId = stmtCache.get(`${key}:${date}`);
|
||||||
|
if (!stId) {
|
||||||
|
const st = await pool.query(
|
||||||
|
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||||
|
VALUES ($4, $1, $2, $3::date - 20, $3::date + 10, $5) RETURNING id`,
|
||||||
|
[acct, bank, date, `s${acctSeq++}.pdf`, ownerId]
|
||||||
|
);
|
||||||
|
stId = st.rows[0].id as number;
|
||||||
|
stmtCache.set(`${key}:${date}`, stId);
|
||||||
|
}
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||||
|
VALUES ($1,$2,$3,$4,$5,$6)`,
|
||||||
|
[stId, date, description, amount, type, ownerId]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let stmtCache = new Map<string, number>();
|
||||||
|
|
||||||
|
/** A statementless merchant purchase, for the non-fee signal. */
|
||||||
|
async function purchase(date: string, merchant: string, amount: number) {
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id, merchant_normalized, row_index)
|
||||||
|
VALUES (NULL,$1,$2,$3,'debit',$4,$2,0)`,
|
||||||
|
[date, merchant, amount, ownerId]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const find = <T extends { label: string }>(rows: T[], label: string) =>
|
||||||
|
rows.find((r) => r.label === label);
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
pool = createPool();
|
||||||
|
mockDbWithPool(pool);
|
||||||
|
({ getUpcomingAnnualCharges } = await import("../../lib/annual-charges"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM account_links");
|
||||||
|
stmtCache = new Map();
|
||||||
|
({ ownerId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("what it finds", () => {
|
||||||
|
it("confirms a card annual fee that repeated a year later", async () => {
|
||||||
|
await charge("American Express", "1000", "2025-06-10", "ANNUAL CARD FEE", 875);
|
||||||
|
await charge("American Express", "1000", "2026-06-10", "ANNUAL CARD FEE", 1750);
|
||||||
|
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CARD FEE");
|
||||||
|
expect(row?.confidence).toBe("confirmed");
|
||||||
|
expect(row?.observations).toBe(2);
|
||||||
|
// The amount is NOT averaged: this fee doubled, and the next one to plan for
|
||||||
|
// is the latest, not the mean of a rise.
|
||||||
|
expect(Number(row?.last_amount)).toBe(1750);
|
||||||
|
expect(row?.next_due_estimate.startsWith("2027-06")).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts a fee seen only once when the bank names it annual", async () => {
|
||||||
|
await charge("Westpac", "8032", "2025-10-21", "CARD FEE", 295);
|
||||||
|
const row = find(await getUpcomingAnnualCharges(ownerId), "CARD FEE");
|
||||||
|
// One charge cannot show a cadence, but the kind is not in doubt.
|
||||||
|
expect(row?.confidence).toBe("likely");
|
||||||
|
expect(row?.next_due_estimate).toBe("2026-10-21");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("finds an annual merchant charge with no fee wording, if the amount is stable", async () => {
|
||||||
|
await purchase("2025-02-03", "AMAZON PRIME", 79);
|
||||||
|
await purchase("2026-02-01", "AMAZON PRIME", 79);
|
||||||
|
const row = find(await getUpcomingAnnualCharges(ownerId), "AMAZON PRIME");
|
||||||
|
expect(row?.confidence).toBe("possible");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("what it must NOT report", () => {
|
||||||
|
it("ignores a monthly account fee", async () => {
|
||||||
|
for (const d of ["2026-01-31", "2026-02-28", "2026-03-31", "2026-04-30"]) {
|
||||||
|
await charge("ANZ", "5626", d, "ACCOUNT SERVICING FEE", 5);
|
||||||
|
}
|
||||||
|
expect(find(await getUpcomingAnnualCharges(ownerId), "ACCOUNT SERVICING FEE")).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores per-transaction fees, which never repeat under one label", async () => {
|
||||||
|
await charge("Wise Australia Pty Ltd.", "9001", "2026-05-20", "WISE CHARGES FOR: TRANSFER-2143509822", 30.67);
|
||||||
|
await charge("Wise Australia Pty Ltd.", "9001", "2026-06-04", "WISE CHARGES FOR: TRANSFER-2171403032", 19.51);
|
||||||
|
const rows = await getUpcomingAnnualCharges(ownerId);
|
||||||
|
expect(rows.filter((r) => r.label.startsWith("WISE CHARGES"))).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores a merchant visited twice a year apart for wildly different amounts", async () => {
|
||||||
|
// The exact shape that defeats a pure cadence test: Target, $30 then $198.
|
||||||
|
await purchase("2025-07-21", "TARGET", 30);
|
||||||
|
await purchase("2026-07-11", "TARGET", 198);
|
||||||
|
expect(find(await getUpcomingAnnualCharges(ownerId), "TARGET")).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores a merchant seen only once", async () => {
|
||||||
|
await purchase("2026-04-03", "UBER ONE MEMBERSHIP", 96);
|
||||||
|
expect(find(await getUpcomingAnnualCharges(ownerId), "UBER ONE MEMBERSHIP")).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("account identity and honesty about gaps", () => {
|
||||||
|
it("treats a renamed account as one continuous fee series", async () => {
|
||||||
|
// Citi Prestige -> NAB MyCard: same card, two bank names, two numbers.
|
||||||
|
await charge("Citibank", "5258073330340253", "2024-12-13", "Annual Fee", 700);
|
||||||
|
await charge("Citibank", "5258073330340253", "2025-11-13", "Annual Fee", 700);
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO account_links (from_bank_name, from_account_key, to_bank_name, to_account_key, effective_date, reason)
|
||||||
|
VALUES ('Citibank','5258073330340253','NAB','1015001000116227','2025-11-14','issuer_change')`
|
||||||
|
);
|
||||||
|
// A statement under the NEW identity, so the link has somewhere to resolve to.
|
||||||
|
await charge("NAB", "1015001000116227", "2026-08-13", "Purchase", 29.99, "debit");
|
||||||
|
|
||||||
|
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL FEE");
|
||||||
|
expect(row?.observations).toBe(2);
|
||||||
|
expect(row?.confidence).toBe("confirmed");
|
||||||
|
// Names the account's CURRENT bank -- the next fee comes from there, and
|
||||||
|
// sending someone to the retired brand is the bug this link exists to stop.
|
||||||
|
expect(row?.bank_name).toBe("NAB");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("marks a passed estimate as expected_not_seen rather than upcoming", async () => {
|
||||||
|
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
|
||||||
|
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
|
||||||
|
const row = find(await getUpcomingAnnualCharges(ownerId), "ANNUAL CHARGE");
|
||||||
|
expect(row?.status).toBe("expected_not_seen");
|
||||||
|
// And it reports how stale the account is, which is what separates
|
||||||
|
// "it stopped being charged" from "we stopped importing the statements".
|
||||||
|
expect(row?.account_days_stale).toBeGreaterThan(300);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("sorts upcoming ahead of expected_not_seen", async () => {
|
||||||
|
await charge("HSBC", "0351", "2024-03-06", "ANNUAL CHARGE", 450);
|
||||||
|
await charge("HSBC", "0351", "2025-03-06", "ANNUAL CHARGE", 450);
|
||||||
|
await charge("Westpac", "8032", "2026-08-01", "CARD FEE", 295);
|
||||||
|
const rows = await getUpcomingAnnualCharges(ownerId);
|
||||||
|
const firstMissing = rows.findIndex((r) => r.status === "expected_not_seen");
|
||||||
|
const lastUpcoming = rows.map((r) => r.status).lastIndexOf("upcoming");
|
||||||
|
expect(lastUpcoming).toBeLessThan(firstMissing);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,271 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The automatic pass over the reconcile queue.
|
||||||
|
*
|
||||||
|
* This exists because the wallet lane writes a transaction for every tap
|
||||||
|
* (DECISIONS.md ING-11), adding ~63 rows a month to a queue already ~177 deep
|
||||||
|
* and worked by hand. Hiding those rows instead is explicitly ruled out --
|
||||||
|
* see the awaitsStatementLine() removal note in queries.ts.
|
||||||
|
*
|
||||||
|
* The cases that must NOT match matter more than the ones that must: this runs
|
||||||
|
* unattended, and a wrong pairing merges one purchase's curation onto another
|
||||||
|
* purchase's row. So most of what follows is about refusal.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const MACHINE_TOKEN = "test-wallet-token";
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let autoReconcile: typeof import("../../app/api/transactions/reconcile/route").PATCH;
|
||||||
|
|
||||||
|
/** Browser-session caller: no machine token, auth comes from the mocked session. */
|
||||||
|
const userReq = (url = "http://x/api/transactions/reconcile") =>
|
||||||
|
({ url, headers: { get: () => null } }) as never;
|
||||||
|
|
||||||
|
/** Scheduled caller: machine token, no session, sweeps every owner. */
|
||||||
|
const machineReq = (url = "http://x/api/transactions/reconcile") =>
|
||||||
|
({ url, headers: { get: (h: string) => (h === "x-ingest-token" ? MACHINE_TOKEN : null) } }) as never;
|
||||||
|
|
||||||
|
async function statementLine(
|
||||||
|
amount: number,
|
||||||
|
date: string,
|
||||||
|
description: string,
|
||||||
|
opts: { owner?: number; type?: string; fxAmount?: number; fxCode?: string } = {}
|
||||||
|
) {
|
||||||
|
const owner = opts.owner ?? ownerId;
|
||||||
|
const st = await pool.query(
|
||||||
|
// Account number varies by owner: uq_statement_identity is
|
||||||
|
// (bank_name, account_number, billing_end_date), so two owners sharing one
|
||||||
|
// account collide rather than modelling two real cards.
|
||||||
|
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||||
|
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||||
|
[date, owner, `1234-5678-9012-${String(owner).padStart(4, "0")}`]
|
||||||
|
);
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions
|
||||||
|
(statement_id, transaction_date, description, amount, transaction_type, owner_id,
|
||||||
|
foreign_currency_amount, foreign_currency_code)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id`,
|
||||||
|
[st.rows[0].id, date, description, amount, opts.type ?? "debit", owner,
|
||||||
|
opts.fxAmount ?? null, opts.fxCode ?? null]
|
||||||
|
);
|
||||||
|
return t.rows[0].id as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function walletRow(
|
||||||
|
amount: number,
|
||||||
|
date: string,
|
||||||
|
opts: { fxAmount?: number; fxCode?: string; paymentMethod?: string } = {}
|
||||||
|
) {
|
||||||
|
const id = await insertTransaction(pool, ownerId, {
|
||||||
|
amount, transaction_date: date, description: "Wallet tap",
|
||||||
|
});
|
||||||
|
await pool.query(
|
||||||
|
`UPDATE transactions
|
||||||
|
SET foreign_currency_amount = $2, foreign_currency_code = $3, payment_method = $4
|
||||||
|
WHERE id = $1`,
|
||||||
|
[id, opts.fxAmount ?? null, opts.fxCode ?? null, opts.paymentMethod ?? null]
|
||||||
|
);
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
const reconciledWith = async (id: number) =>
|
||||||
|
(await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [id]))
|
||||||
|
.rows[0].reconciled_with_id as number | null;
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
process.env.WALLET_INGEST_TOKEN = MACHINE_TOKEN;
|
||||||
|
pool = createPool();
|
||||||
|
vi.doMock("@/lib/auth", () => ({
|
||||||
|
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||||
|
}));
|
||||||
|
({ PATCH: autoReconcile } = await import("../../app/api/transactions/reconcile/route"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM expense_metadata");
|
||||||
|
({ ownerId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("pairs it should apply", () => {
|
||||||
|
it("matches an exact amount two days apart and moves the link", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
const stmt = await statementLine(13.5, "2026-08-15", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq())).json();
|
||||||
|
expect(body.reconciled).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("matches a foreign row on the NATIVE pair, not on the AUD amount", async () => {
|
||||||
|
// The bank converts at its own rate, so the wallet row cannot predict the
|
||||||
|
// AUD figure. It can only know THB 500. Deliberately give the statement a
|
||||||
|
// wildly different AUD amount to prove `amount` is not what matched.
|
||||||
|
const manual = await walletRow(999.99, "2026-08-13", { fxAmount: 500, fxCode: "THB" });
|
||||||
|
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
|
||||||
|
fxAmount: 500, fxCode: "THB",
|
||||||
|
});
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq())).json();
|
||||||
|
expect(body.reconciled).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("carries a SETTLED split across without resurrecting the debt", async () => {
|
||||||
|
const manual = await walletRow(50, "2026-08-13");
|
||||||
|
const stmt = await statementLine(50, "2026-08-14", "SHARED DINNER");
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled, settled_at)
|
||||||
|
VALUES ($1, $2, 50, true, now())`,
|
||||||
|
[manual, ownerId]
|
||||||
|
);
|
||||||
|
|
||||||
|
expect((await autoReconcile(userReq())).status ?? 200).toBe(200);
|
||||||
|
|
||||||
|
const moved = await pool.query(
|
||||||
|
`SELECT settled, settled_at FROM transaction_splits WHERE transaction_id = $1`, [stmt]
|
||||||
|
);
|
||||||
|
expect(moved.rowCount).toBe(1);
|
||||||
|
// The whole point: a delete-and-recreate defaults `settled` to false, which
|
||||||
|
// silently marks a paid debt owed again. Nothing errors when it regresses.
|
||||||
|
expect(moved.rows[0].settled).toBe(true);
|
||||||
|
expect(moved.rows[0].settled_at).not.toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("pairs it must refuse", () => {
|
||||||
|
it("refuses two identical amounts in the same window (ambiguous both ways)", async () => {
|
||||||
|
// Two $13.50 bakery taps in one week is the ordinary case, not an edge one.
|
||||||
|
const a = await walletRow(13.5, "2026-08-13");
|
||||||
|
const b = await walletRow(13.5, "2026-08-14");
|
||||||
|
await statementLine(13.5, "2026-08-14", "BREADWORLD ONE");
|
||||||
|
await statementLine(13.5, "2026-08-15", "BREADWORLD TWO");
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq())).json();
|
||||||
|
expect(body.reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(a)).toBeNull();
|
||||||
|
expect(await reconciledWith(b)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a near-miss amount — a person should look at that", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
await statementLine(13.55, "2026-08-14", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a match outside the day window", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
await statementLine(13.5, "2026-08-25", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a statement row already claimed by another manual row", async () => {
|
||||||
|
const claimed = await statementLine(40, "2026-08-14", "ALREADY TAKEN");
|
||||||
|
const first = await walletRow(40, "2026-08-13");
|
||||||
|
await pool.query(`UPDATE transactions SET reconciled_with_id = $2 WHERE id = $1`, [first, claimed]);
|
||||||
|
const second = await walletRow(40, "2026-08-14");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(second)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a credits-funded row — no card leg is ever coming", async () => {
|
||||||
|
const manual = await walletRow(30, "2026-08-13", { paymentMethod: "credits" });
|
||||||
|
await statementLine(30, "2026-08-14", "COINCIDENTAL AMOUNT");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("an AUD row against a foreign statement line", () => {
|
||||||
|
it("DOES match on the AUD figure — this is what every real pair looks like", async () => {
|
||||||
|
// Checked against live data: all 48 reconciliations a human has made are
|
||||||
|
// this exact shape -- manual row with no currency, statement row carrying
|
||||||
|
// EUR or CHF, paired on exact AUD. Overseas receipts and orders record what
|
||||||
|
// was charged in AUD; only the wallet lane records natively. An earlier
|
||||||
|
// version of this rule required both sides to be non-foreign, on the
|
||||||
|
// reasoning that a currency mismatch is ambiguous. It matched 0 of the 48.
|
||||||
|
const manual = await walletRow(22.85, "2026-08-13");
|
||||||
|
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
|
||||||
|
fxAmount: 500, fxCode: "THB",
|
||||||
|
});
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("still refuses when a second candidate makes it ambiguous", async () => {
|
||||||
|
// The coincidence this guards against -- a domestic tap whose amount equals
|
||||||
|
// a foreign charge's AUD figure -- is caught by unique-both-ways, not by a
|
||||||
|
// currency test.
|
||||||
|
const manual = await walletRow(22.85, "2026-08-13");
|
||||||
|
await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", { fxAmount: 500, fxCode: "THB" });
|
||||||
|
await statementLine(22.85, "2026-08-13", "MELBOURNE CAFE");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("dry run", () => {
|
||||||
|
it("reports what it would do and writes nothing", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq("http://x/api/transactions/reconcile?dry=1"))).json();
|
||||||
|
expect(body.dry_run).toBe(true);
|
||||||
|
expect(body.would_reconcile).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the scheduled sweep", () => {
|
||||||
|
it("runs on the machine token with no session at all", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
const stmt = await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(machineReq())).json();
|
||||||
|
expect(body.reconciled).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("covers EVERY owner, not just one", async () => {
|
||||||
|
// 169 of the 177 rows outstanding belong to the second participant, so a
|
||||||
|
// sweep scoped to a single user would leave almost all of the queue alone.
|
||||||
|
const other = (await pool.query(
|
||||||
|
`INSERT INTO participants (name) VALUES ('Carol') RETURNING id`
|
||||||
|
)).rows[0].id as number;
|
||||||
|
|
||||||
|
const mine = await walletRow(11, "2026-08-13");
|
||||||
|
const theirTx = (await pool.query(
|
||||||
|
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description,
|
||||||
|
amount, transaction_type, category, row_index)
|
||||||
|
VALUES ($1, NULL, '2026-08-13', 'Theirs', 22, 'debit', 'other', 0) RETURNING id`, [other]
|
||||||
|
)).rows[0].id as number;
|
||||||
|
|
||||||
|
await statementLine(11, "2026-08-14", "MINE");
|
||||||
|
await statementLine(22, "2026-08-14", "THEIRS", { owner: other });
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(machineReq())).json();
|
||||||
|
expect(body.reconciled).toBe(2);
|
||||||
|
expect(await reconciledWith(mine)).not.toBeNull();
|
||||||
|
expect(await reconciledWith(theirTx)).not.toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects a wrong machine token and falls through to session auth", async () => {
|
||||||
|
const bad = ({ url: "http://x/", headers: { get: () => "not-the-token" } }) as never;
|
||||||
|
// Session is mocked, so this still succeeds -- but as the session user only,
|
||||||
|
// proving the bad token did not grant the all-owner sweep.
|
||||||
|
const body = await (await autoReconcile(bad)).json();
|
||||||
|
expect(body.reconciled).toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -23,6 +23,25 @@ export function mockDbWithPool(p: Pool) {
|
|||||||
const result = await p.query(sql, params);
|
const result = await p.query(sql, params);
|
||||||
return result.rows[0] ?? null;
|
return result.rows[0] ?? null;
|
||||||
},
|
},
|
||||||
|
// Mirrors withTransaction: one connection, BEGIN/COMMIT around the callback,
|
||||||
|
// so pg_advisory_xact_lock actually holds across the statements inside it.
|
||||||
|
withTransaction: async (fn: (q: unknown) => Promise<unknown>) => {
|
||||||
|
const client = await p.connect();
|
||||||
|
try {
|
||||||
|
await client.query("BEGIN");
|
||||||
|
const result = await fn({
|
||||||
|
queryRaw: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows,
|
||||||
|
queryRow: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows[0] ?? null,
|
||||||
|
});
|
||||||
|
await client.query("COMMIT");
|
||||||
|
return result;
|
||||||
|
} catch (e) {
|
||||||
|
await client.query("ROLLBACK");
|
||||||
|
throw e;
|
||||||
|
} finally {
|
||||||
|
client.release();
|
||||||
|
}
|
||||||
|
},
|
||||||
prisma: p,
|
prisma: p,
|
||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
@@ -65,6 +84,11 @@ export async function resetDB(pool: Pool) {
|
|||||||
await assertTestDatabase(pool);
|
await assertTestDatabase(pool);
|
||||||
await pool.query(`
|
await pool.query(`
|
||||||
TRUNCATE
|
TRUNCATE
|
||||||
|
-- The audit triggers (migration 0037) fire on every INSERT the suite makes.
|
||||||
|
-- TRUNCATE does not fire row-level triggers, so nothing here is logged --
|
||||||
|
-- but without this line the log is never cleared either, and the table
|
||||||
|
-- grows by roughly a thousand rows per full run, forever.
|
||||||
|
transaction_audit,
|
||||||
split_payments,
|
split_payments,
|
||||||
transaction_splits,
|
transaction_splits,
|
||||||
transaction_tags,
|
transaction_tags,
|
||||||
|
|||||||
@@ -49,6 +49,7 @@ let fileDir: string;
|
|||||||
let intakePOST: (req: NextRequest) => Promise<Response>;
|
let intakePOST: (req: NextRequest) => Promise<Response>;
|
||||||
let lineItemsPUT: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
|
let lineItemsPUT: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
|
||||||
let slackPOST: (req: NextRequest) => Promise<Response>;
|
let slackPOST: (req: NextRequest) => Promise<Response>;
|
||||||
|
let orderGET: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
|
||||||
|
|
||||||
/** What the next LiteLLM call answers, and what the next pantry call answers. */
|
/** What the next LiteLLM call answers, and what the next pantry call answers. */
|
||||||
let modelReply: ExtractedOrder | Error = burgertory;
|
let modelReply: ExtractedOrder | Error = burgertory;
|
||||||
@@ -104,6 +105,21 @@ async function seedOrder(over: { merchant: string; date: string; amount: number;
|
|||||||
}
|
}
|
||||||
return id;
|
return id;
|
||||||
}
|
}
|
||||||
|
/** A row shaped the way the wallet lane writes one: statementless, no metadata
|
||||||
|
* at all, no platform, and named by the notification rather than the receipt. */
|
||||||
|
async function seedWalletRow(over: { merchant: string; date: string; amount: number; owner?: number }) {
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description, amount, amount_aud, transaction_type, category, merchant_name, merchant_normalized, payment_method, location, source, source_ref, row_index)
|
||||||
|
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', $3, $3, 'card', 'Werribee, VIC', 'wallet', $5, 0) RETURNING id`,
|
||||||
|
[over.owner ?? ownerId, over.date, over.merchant, over.amount, `evt:${Math.floor(over.amount * 100)}`]
|
||||||
|
);
|
||||||
|
return t.rows[0].id as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What `components/order-details.tsx` would actually render for a transaction. */
|
||||||
|
const orderPanel = async (txnId: number) =>
|
||||||
|
(await (await orderGET({ headers: { get: () => null } } as unknown as NextRequest, { params: Promise.resolve({ id: String(txnId) }) })).json()) as { line_items?: unknown[] } | null;
|
||||||
|
const metaRows = async () => (await pool.query(`SELECT id, platform, line_items FROM expense_metadata ORDER BY id`)).rows as { id: number; platform: string | null; line_items: { description: string }[] }[];
|
||||||
const lineItemsOf = async (txnId: number) =>
|
const lineItemsOf = async (txnId: number) =>
|
||||||
(await pool.query(`SELECT line_items FROM expense_metadata WHERE transaction_id = $1 OR matched_transaction_id = $1`, [txnId])).rows[0]?.line_items;
|
(await pool.query(`SELECT line_items FROM expense_metadata WHERE transaction_id = $1 OR matched_transaction_id = $1`, [txnId])).rows[0]?.line_items;
|
||||||
const batch = async (id: string) => (await pool.query(`SELECT * FROM intake_batches WHERE id = $1`, [id])).rows[0];
|
const batch = async (id: string) => (await pool.query(`SELECT * FROM intake_batches WHERE id = $1`, [id])).rows[0];
|
||||||
@@ -127,9 +143,17 @@ beforeAll(async () => {
|
|||||||
participantForSlackUser: () => 1,
|
participantForSlackUser: () => 1,
|
||||||
slackUserForParticipant: () => null,
|
slackUserForParticipant: () => null,
|
||||||
}));
|
}));
|
||||||
|
// The order-detail panel is the surface an enrichment has to appear on, so
|
||||||
|
// the read is exercised beside the write rather than assumed.
|
||||||
|
vi.doMock("@/lib/auth", () => ({ getCurrentUser: async () => ({ id: 1, name: "Owner", email: "o@x" }) }));
|
||||||
|
vi.doMock("@/lib/queries", async (importOriginal) => ({
|
||||||
|
...(await importOriginal<Record<string, unknown>>()),
|
||||||
|
canAccessTransactions: async () => true,
|
||||||
|
}));
|
||||||
({ POST: intakePOST } = await import("../../app/api/intake/route"));
|
({ POST: intakePOST } = await import("../../app/api/intake/route"));
|
||||||
({ PUT: lineItemsPUT } = await import("../../app/api/transactions/[id]/line-items/route"));
|
({ PUT: lineItemsPUT } = await import("../../app/api/transactions/[id]/line-items/route"));
|
||||||
({ POST: slackPOST } = await import("../../app/api/slack/interactive/route"));
|
({ POST: slackPOST } = await import("../../app/api/slack/interactive/route"));
|
||||||
|
({ GET: orderGET } = await import("../../app/api/transactions/[id]/order/route"));
|
||||||
stubFetch();
|
stubFetch();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -210,6 +234,82 @@ describe("PUT /api/transactions/[id]/line-items (C2)", () => {
|
|||||||
expect(em).toEqual({ source: "intake", platform: "ubereats", order_reference: "intake:b1", merchant_normalized: "Coles", amount: "85.74" });
|
expect(em).toEqual({ source: "intake", platform: "ubereats", order_reference: "intake:b1", merchant_normalized: "Coles", amount: "85.74" });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("the undo baseline is recorded ONCE per batch: a retry gets the original, not its own write", async () => {
|
||||||
|
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const original = [{ qty: 1, description: "what the order already had", amount: 22.4 }];
|
||||||
|
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(original)]);
|
||||||
|
const id = uuid();
|
||||||
|
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
|
||||||
|
|
||||||
|
const first = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
|
||||||
|
expect(first.previous_line_items).toEqual(original);
|
||||||
|
// Pantry re-calls on every confirm and after a timeout. Before, the retry
|
||||||
|
// was handed its OWN write back, so undo restored the thing it undid.
|
||||||
|
const retry = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
|
||||||
|
expect(retry.previous_line_items).toEqual(original);
|
||||||
|
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual(original);
|
||||||
|
|
||||||
|
await put(txn, { line_items: retry.previous_line_items, source: "pantry", batch_id: id, replace: true });
|
||||||
|
expect(await lineItemsOf(txn)).toEqual(original);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("without a batch id the baseline is still the row as read — nothing else changes", async () => {
|
||||||
|
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const first = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
|
||||||
|
expect(first.previous_line_items).toEqual([]);
|
||||||
|
const second = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
|
||||||
|
expect(second.previous_line_items).toEqual(items);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a pinned row that no longer exists falls back to the live one, rather than reporting a phantom write", async () => {
|
||||||
|
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const live = (await pool.query(`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].id as number;
|
||||||
|
const { replaceLineItems } = await import("../../lib/line-items");
|
||||||
|
const r = await replaceLineItems(txn, items, { source: "intake", metadataId: live + 999 });
|
||||||
|
expect(r.metadata_id).toBe(live);
|
||||||
|
expect(await lineItemsOf(txn)).toHaveLength(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("two writes racing to create the metadata row both resolve to the same one", async () => {
|
||||||
|
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
|
||||||
|
const both = await Promise.all([
|
||||||
|
put(txn, { line_items: items, source: "intake", replace: true }),
|
||||||
|
put(txn, { line_items: items, source: "intake", replace: true }),
|
||||||
|
]);
|
||||||
|
expect(both.map((r) => r.status)).toEqual([200, 200]);
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].n).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("the create path records an undo baseline too, before it writes", async () => {
|
||||||
|
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
|
||||||
|
const id = uuid();
|
||||||
|
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
|
||||||
|
await put(txn, { line_items: items, source: "intake", batch_id: id, currency: "AUD", replace: true });
|
||||||
|
// [] rather than NULL: there was nothing there, and "nothing" is the state
|
||||||
|
// undo has to be able to restore.
|
||||||
|
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("the panel reads the row the write landed on, whichever order the rows were inserted in", async () => {
|
||||||
|
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
|
||||||
|
// Inserted first: the delivery receipt pointing at the statement line.
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, '[]'::jsonb)`,
|
||||||
|
[t, uuid()]
|
||||||
|
);
|
||||||
|
// Inserted second: a row the transaction owns. An unordered LIMIT 1 reads
|
||||||
|
// the first one, the write takes the second, and the enrichment reports
|
||||||
|
// success against a row nothing renders.
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO expense_metadata (transaction_id, source, order_reference, line_items) VALUES ($1, 'pantry', $2, '[]'::jsonb)`,
|
||||||
|
[t, uuid()]
|
||||||
|
);
|
||||||
|
const written = await (await put(t, { line_items: items, source: "intake", replace: true })).json();
|
||||||
|
const panel = await orderPanel(t);
|
||||||
|
expect(panel?.line_items).toHaveLength(2);
|
||||||
|
expect((await metaRows()).find((r) => r.id === written.metadata_id)!.line_items).toHaveLength(2);
|
||||||
|
});
|
||||||
|
|
||||||
it("finds the row through matched_transaction_id — a card-settled order has only that", async () => {
|
it("finds the row through matched_transaction_id — a card-settled order has only that", async () => {
|
||||||
const statementLine = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
|
const statementLine = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
|
||||||
await pool.query(
|
await pool.query(
|
||||||
@@ -250,6 +350,64 @@ describe("POST /api/intake — gates", () => {
|
|||||||
expect((await batch(id)).status).toBe("unsupported");
|
expect((await batch(id)).status).toBe("unsupported");
|
||||||
expect(calls).toHaveLength(0);
|
expect(calls).toHaveLength(0);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// 2026-08-22, live batch 3d2ffbe8…: an iOS Shortcut coerced the shared photo
|
||||||
|
// to text, so the whole upload was the five characters `Image`. The model was
|
||||||
|
// asked anyway and returned Burgertory (Braybrook), two "The Pride" lines and
|
||||||
|
// a $41.15 total — none of which existed. Only the absent date kept it out of
|
||||||
|
// the matcher, and the invented header was stored on the batch as
|
||||||
|
// `needs_header`. The model must never be asked.
|
||||||
|
it("a text part that cannot be a receipt is declined BEFORE any model call", async () => {
|
||||||
|
const id = uuid();
|
||||||
|
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)", order_date: null, total: 41.15 };
|
||||||
|
const res = await intakePOST(
|
||||||
|
multipart({ source: "other", batch_id: id }, [{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") }])
|
||||||
|
);
|
||||||
|
const body = await res.json();
|
||||||
|
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
|
||||||
|
expect(body.text).toMatch(/single word/);
|
||||||
|
expect(body.text).toMatch(/"Image"/);
|
||||||
|
// The model was never asked — the only thing that makes confabulation impossible.
|
||||||
|
expect(calls).toHaveLength(0);
|
||||||
|
const row = await batch(id);
|
||||||
|
expect(row.status).toBe("unsupported");
|
||||||
|
expect(row.kind).toBe("unsupported");
|
||||||
|
expect(row.header).toBeNull();
|
||||||
|
expect(row.items).toBeNull();
|
||||||
|
expect(row.transaction_id).toBeNull();
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a junk text part beside a real receipt does not sink the batch", async () => {
|
||||||
|
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const body = await (
|
||||||
|
await intakePOST(
|
||||||
|
multipart({ source: "uber_eats" }, [
|
||||||
|
{ name: "shared.txt", type: "text/plain", bytes: Buffer.from("Image") },
|
||||||
|
{ name: "Receipt_19Aug2026.pdf", type: "application/pdf", bytes: BURGERTORY },
|
||||||
|
])
|
||||||
|
)
|
||||||
|
).json();
|
||||||
|
expect(body.status).toBe("enriched");
|
||||||
|
expect(await lineItemsOf(txn)).toHaveLength(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("an order extracted from an unreadable PDF is unsupported, not needs_header", async () => {
|
||||||
|
const id = uuid();
|
||||||
|
// A PDF that renders to a scrap of junk still reaches the model — the
|
||||||
|
// pre-check only inspects text parts. `confabulationReason` is the belt.
|
||||||
|
vi.spyOn(await import("../../lib/intake-extract"), "extractPdfText").mockResolvedValueOnce("Image");
|
||||||
|
modelReply = { ...burgertory, merchant: "Burgertory (Braybrook)" };
|
||||||
|
const body = await (
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [{ name: "x.pdf", type: "application/pdf", bytes: BURGERTORY }]))
|
||||||
|
).json();
|
||||||
|
expect(body).toMatchObject({ kind: "unsupported", status: "unsupported" });
|
||||||
|
expect(body.text).toMatch(/nothing shared could be read/);
|
||||||
|
const row = await batch(id);
|
||||||
|
expect(row.status).toBe("unsupported");
|
||||||
|
expect(row.header).toBeNull();
|
||||||
|
expect(row.items).toBeNull();
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("POST /api/intake — food (enrich vs candidates)", () => {
|
describe("POST /api/intake — food (enrich vs candidates)", () => {
|
||||||
@@ -285,10 +443,82 @@ describe("POST /api/intake — food (enrich vs candidates)", () => {
|
|||||||
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("the platform is part of the key: the same merchant/date/total on DoorDash is not the match", async () => {
|
it("the platform is part of the key: the same merchant/date/total on DoorDash is not the match — but it IS offered", async () => {
|
||||||
await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "doordash" });
|
const doordash = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "doordash" });
|
||||||
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
|
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
|
||||||
expect(body.status).toBe("candidates");
|
expect(body.status).toBe("candidates");
|
||||||
|
// The candidates card is a human choice, not a decision. The platform is one
|
||||||
|
// of the things that may be wrong, so it must not narrow the list — a
|
||||||
|
// platform-filtered card showed nothing at all here.
|
||||||
|
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([doordash]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a NULL platform is 'no platform', not a wildcard — an in-store share does not overwrite the delivery order", async () => {
|
||||||
|
await pool.query(`INSERT INTO intake_merchant_kinds (merchant_key, kind) VALUES ('burgertory werribee', 'food')`);
|
||||||
|
const delivery = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "ubereats" });
|
||||||
|
const already = [{ qty: 1, description: "what the delivery order already had", amount: 22.4 }];
|
||||||
|
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [delivery, JSON.stringify(already)]);
|
||||||
|
|
||||||
|
// `other` implies no platform; the delivery row carries one, so it is not
|
||||||
|
// the match — it is only a candidate.
|
||||||
|
const body = await (await intakePOST(multipart({ source: "other" }, [pdf]))).json();
|
||||||
|
expect(body.status).toBe("candidates");
|
||||||
|
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([delivery]);
|
||||||
|
expect(await lineItemsOf(delivery)).toEqual(already);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a reconciled manual row is neither matched nor offered — EXCLUDE_RECONCILED_SOURCE, as everywhere else", async () => {
|
||||||
|
const live = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const reconciled = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, reconciled]);
|
||||||
|
const near = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-17", amount: 19.9 });
|
||||||
|
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, near]);
|
||||||
|
|
||||||
|
// Two rows at the exact key would have read as an ambiguity; one of them is
|
||||||
|
// the same spend counted twice.
|
||||||
|
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
|
||||||
|
expect(body).toMatchObject({ status: "enriched", transaction_id: live });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("the write lands on the row the MATCH resolved through, not on whichever row sorts first", async () => {
|
||||||
|
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
|
||||||
|
// The statement line owns a receipt row of its own (a pantry confirm) …
|
||||||
|
const grocery = [{ qty: 1, description: "a grocery shop", amount: 22.4 }];
|
||||||
|
const own = (await pool.query(
|
||||||
|
`INSERT INTO expense_metadata (transaction_id, source, order_reference, merchant_normalized, line_items) VALUES ($1, 'pantry', $2, 'Woolworths', $3::jsonb) RETURNING id`,
|
||||||
|
[t, uuid(), JSON.stringify(grocery)]
|
||||||
|
)).rows[0].id as number;
|
||||||
|
// … and is also the card leg of the Uber Eats order being enriched.
|
||||||
|
const delivery = (await pool.query(
|
||||||
|
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, merchant_normalized, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, 'Burgertory (Werribee)', '[]'::jsonb) RETURNING id`,
|
||||||
|
[t, uuid()]
|
||||||
|
)).rows[0].id as number;
|
||||||
|
|
||||||
|
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
|
||||||
|
expect(body).toMatchObject({ status: "enriched", transaction_id: t });
|
||||||
|
const rows = await metaRows();
|
||||||
|
expect(rows.find((r) => r.id === delivery)!.line_items.map((i) => i.description)).toEqual(["Michael", "Joel"]);
|
||||||
|
expect(rows.find((r) => r.id === own)!.line_items).toEqual(grocery);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("an upload with no prices never replaces a receipt that already has items", async () => {
|
||||||
|
const priceless = { ...burgertory, subtotal: null, total: 22.4, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
|
||||||
|
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
|
||||||
|
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
|
||||||
|
|
||||||
|
modelReply = priceless;
|
||||||
|
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
|
||||||
|
expect(body).toMatchObject({ kind: "held", status: "held" });
|
||||||
|
expect(body.text).toMatch(/left the existing items alone/);
|
||||||
|
expect(await lineItemsOf(txn)).toEqual(already);
|
||||||
|
|
||||||
|
// …but an EMPTY receipt is exactly what this slice exists to fill.
|
||||||
|
const empty = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-20", amount: 22.4 });
|
||||||
|
modelReply = { ...priceless, order_date: "2026-08-20" };
|
||||||
|
const second = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
|
||||||
|
expect(second).toMatchObject({ status: "enriched", transaction_id: empty });
|
||||||
|
expect((await lineItemsOf(empty)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("food + no match → candidates (same merchant ±3 days, any amount) with Use-this / Create buttons; NEVER auto-creates", async () => {
|
it("food + no match → candidates (same merchant ±3 days, any amount) with Use-this / Create buttons; NEVER auto-creates", async () => {
|
||||||
@@ -322,7 +552,13 @@ describe("POST /api/intake — food (enrich vs candidates)", () => {
|
|||||||
const id = uuid();
|
const id = uuid();
|
||||||
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
|
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
|
||||||
expect(body.status).toBe("needs_header");
|
expect(body.status).toBe("needs_header");
|
||||||
expect(body.text).toMatch(/share the top of the screen/);
|
// Asserts BOTH halves of needsHeaderCard(): that it names what is actually
|
||||||
|
// missing (total, not a blanket "total or date"), and that a uber_eats
|
||||||
|
// source gets the share-more-of-the-screen advice rather than the in-store
|
||||||
|
// Share-button advice. 258fe3b changed this text and updated the unit tests
|
||||||
|
// but not this one.
|
||||||
|
expect(body.text).toMatch(/I can see items but no total/);
|
||||||
|
expect(body.text).toMatch(/share the part of the screen that shows it/);
|
||||||
expect(await lineItemsOf(txn)).toEqual([]);
|
expect(await lineItemsOf(txn)).toEqual([]);
|
||||||
expect((await batch(id)).status).toBe("needs_header");
|
expect((await batch(id)).status).toBe("needs_header");
|
||||||
});
|
});
|
||||||
@@ -391,6 +627,19 @@ describe("POST /api/intake — grocery → pantry", () => {
|
|||||||
expect((await batch(id))).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn });
|
expect((await batch(id))).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("an in-store tap does not hand pantry the delivery order as its finance target", async () => {
|
||||||
|
modelReply = coles;
|
||||||
|
// The Uber Eats Coles order the email pipeline already booked.
|
||||||
|
const delivery = await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: "ubereats" });
|
||||||
|
// The same shop, same day, same money — but shared from the Coles app, so
|
||||||
|
// it carries no platform and is a different purchase.
|
||||||
|
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
|
||||||
|
expect(body.status).toBe("forwarded_to_pantry");
|
||||||
|
const pantryCall = calls.find((c) => c.url === "http://pantry.test/api/capture")!.body as { intake: Record<string, unknown> };
|
||||||
|
expect(pantryCall.intake).toMatchObject({ finance_action: "create", finance_target_transaction_id: null, platform: null });
|
||||||
|
expect(await lineItemsOf(delivery)).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
it("grocery + no match → pantry with finance_action create (ING-10 as today)", async () => {
|
it("grocery + no match → pantry with finance_action create (ING-10 as today)", async () => {
|
||||||
modelReply = coles;
|
modelReply = coles;
|
||||||
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
|
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
|
||||||
@@ -400,6 +649,33 @@ describe("POST /api/intake — grocery → pantry", () => {
|
|||||||
expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" });
|
expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("pantry saying the shop is already on its board is SHOWN, not swallowed", async () => {
|
||||||
|
// Pantry accepts a second capture of one receipt on purpose (the first may
|
||||||
|
// have been a bad crop) and reports it rather than refusing. Dropping that
|
||||||
|
// field turned "reported" into silence — and a duplicate nobody sees is a
|
||||||
|
// duplicate that gets confirmed, which counts the same groceries twice
|
||||||
|
// because pantry's addItem accumulates. Seen live on captures 66/69.
|
||||||
|
modelReply = coles;
|
||||||
|
pantryReply = { status: 200, body: { captureId: 902, items: [1, 2, 3], alreadyCaptured: { captureId: 66, confirmed: true } } };
|
||||||
|
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
|
||||||
|
expect(body.status).toBe("forwarded_to_pantry");
|
||||||
|
expect(body.text).toMatch(/Already on the pantry board/);
|
||||||
|
expect(body.text).toMatch(/capture 66 \(already confirmed\)/);
|
||||||
|
expect(body.text).toMatch(/capture 902/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("two ledger rows for one shop are reported, not treated as no match", async () => {
|
||||||
|
// Ambiguity is not a miss. This is the exact state that produced the
|
||||||
|
// 2026-08-18 Woolworths double-book: the shop was in the ledger twice, the
|
||||||
|
// matcher returned nothing, and the card said finance would book it.
|
||||||
|
modelReply = coles;
|
||||||
|
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
|
||||||
|
await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: null });
|
||||||
|
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
|
||||||
|
expect(body.status).toBe("forwarded_to_pantry");
|
||||||
|
expect(body.text).toMatch(/found more than one row it could be/);
|
||||||
|
});
|
||||||
|
|
||||||
it("pantry 409 → duplicate", async () => {
|
it("pantry 409 → duplicate", async () => {
|
||||||
modelReply = coles;
|
modelReply = coles;
|
||||||
pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } };
|
pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } };
|
||||||
@@ -457,6 +733,143 @@ describe("Slack taps (C5)", () => {
|
|||||||
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("intake_create re-runs the match: an order booked between the card and the tap is enriched, not duplicated", async () => {
|
||||||
|
const id = uuid();
|
||||||
|
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
|
||||||
|
expect(body.status).toBe("candidates");
|
||||||
|
|
||||||
|
// The email/order bridge books the same purchase while the card sits in
|
||||||
|
// Slack. The card's "no match" is minutes stale by the time it is pressed.
|
||||||
|
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: id }));
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||||
|
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: txn });
|
||||||
|
expect((await lineItemsOf(txn)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
|
||||||
|
expect(lastSlackUpdate().text).toMatch(/^Added 2 items/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("intake_create adopts the wallet row for the same tap instead of booking it twice", async () => {
|
||||||
|
// The normal arrival order: the notification lands within seconds of the
|
||||||
|
// tap, the screenshot is shared later. matchTransaction cannot see that row
|
||||||
|
// — its key includes the platform and a notification has none — so before
|
||||||
|
// this, Create booked the purchase a second time.
|
||||||
|
const wallet = await seedWalletRow({ merchant: "Burgertory", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const id = uuid();
|
||||||
|
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
|
||||||
|
expect(body.status).toBe("candidates");
|
||||||
|
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: id }));
|
||||||
|
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||||
|
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: wallet });
|
||||||
|
expect((await lineItemsOf(wallet)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
|
||||||
|
// The suburb the notification supplied is the one thing nothing else knows.
|
||||||
|
expect((await pool.query(`SELECT location FROM transactions WHERE id = $1`, [wallet])).rows[0].location).toBe("Werribee, VIC");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("intake_create still creates when a same-amount row already carries a receipt", async () => {
|
||||||
|
// A row that another source fully described is a DIFFERENT purchase that
|
||||||
|
// happens to share an amount — adopting it would move one order's items
|
||||||
|
// onto another order's row.
|
||||||
|
const other = await seedOrder({ merchant: "Someone Else", date: "2026-08-19", amount: 22.4, platform: null });
|
||||||
|
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [other, JSON.stringify([{ description: "theirs", qty: 1, amount: 22.4 }])]);
|
||||||
|
const id = uuid();
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: id }));
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("two pending cards for one order book it ONCE", async () => {
|
||||||
|
const a = uuid();
|
||||||
|
const b = uuid();
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
|
||||||
|
expect((await batch(a)).status).toBe("candidates");
|
||||||
|
expect((await batch(b)).status).toBe("candidates");
|
||||||
|
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: a }));
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: b }));
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||||
|
expect((await batch(b)).transaction_id).toBe((await batch(a)).transaction_id);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a sibling already resolved to the pantry board makes Create a duplicate, not a second row", async () => {
|
||||||
|
// Two shares of one receipt, resolved out of order: nothing the second card
|
||||||
|
// can match on, because the first one never produced a transaction.
|
||||||
|
modelReply = { ...coles, merchant: "Romeo's Food Hall" };
|
||||||
|
const a = uuid();
|
||||||
|
const b = uuid();
|
||||||
|
await intakePOST(multipart({ source: "other", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
|
||||||
|
await intakePOST(multipart({ source: "other", batch_id: a }, [{ name: "Receipt_18Aug2026.pdf", type: "application/pdf", bytes: COLES }]));
|
||||||
|
expect((await batch(a)).status).toBe("ask");
|
||||||
|
|
||||||
|
await slackPOST(slackPress("intake_grocery", { batch_id: a }));
|
||||||
|
expect((await batch(a)).status).toBe("forwarded_to_pantry");
|
||||||
|
|
||||||
|
await slackPOST(slackPress("intake_meal", { batch_id: b }));
|
||||||
|
expect((await batch(b)).status).toBe("candidates");
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: b }));
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
|
||||||
|
expect(await batch(b)).toMatchObject({ status: "duplicate", capture_id: 901 });
|
||||||
|
expect(lastSlackUpdate().text).toMatch(/^Already handled/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("two taps that land together still book one transaction, and neither reports an error", async () => {
|
||||||
|
const id = uuid();
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
|
||||||
|
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: id })), slackPOST(slackPress("intake_create", { batch_id: id }))]);
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||||
|
const texts = calls.filter((c) => c.url.includes("hooks.slack.com")).map((c) => (c.body as { text: string }).text);
|
||||||
|
expect(texts.every((t) => !/Couldn't process/.test(t))).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a foreign receipt is booked in its own currency, with no invented AUD figure", async () => {
|
||||||
|
modelReply = { ...burgertory, currency: "USD" };
|
||||||
|
const id = uuid();
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: id }));
|
||||||
|
const t = (await pool.query(`SELECT amount::text, amount_aud, foreign_currency_amount::text, foreign_currency_code FROM transactions`)).rows[0];
|
||||||
|
expect(t).toEqual({ amount: "22.40", amount_aud: null, foreign_currency_amount: "22.40", foreign_currency_code: "USD" });
|
||||||
|
expect(lastSlackUpdate().text).toMatch(/USD 22\.40/);
|
||||||
|
// OrderDetails prefers expense_metadata.currency, so a hard-coded 'AUD'
|
||||||
|
// here printed "$" on a USD order's receipt panel.
|
||||||
|
expect((await pool.query(`SELECT currency FROM expense_metadata`)).rows[0].currency).toBe("USD");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("the hold applies to the Create tap's re-match, not only to the upload path", async () => {
|
||||||
|
modelReply = { ...burgertory, subtotal: null, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
|
||||||
|
const id = uuid();
|
||||||
|
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
|
||||||
|
expect(body.status).toBe("candidates");
|
||||||
|
|
||||||
|
// Booked WITH its items between the card and the tap. The re-match walked
|
||||||
|
// past the guard routeFood applies, so the same upload destroyed the same
|
||||||
|
// receipt depending on which button reached it.
|
||||||
|
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
|
||||||
|
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
|
||||||
|
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
|
||||||
|
|
||||||
|
await slackPOST(slackPress("intake_create", { batch_id: id }));
|
||||||
|
expect(await lineItemsOf(txn)).toEqual(already);
|
||||||
|
expect(await batch(id)).toMatchObject({ status: "held" });
|
||||||
|
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("two DIFFERENT batches for one purchase, tapped together, book it once", async () => {
|
||||||
|
const a = uuid();
|
||||||
|
const b = uuid();
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
|
||||||
|
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
|
||||||
|
// Their source_refs differ, so ON CONFLICT cannot save this; both taps
|
||||||
|
// clear their match and duplicate checks before either inserts.
|
||||||
|
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: a })), slackPOST(slackPress("intake_create", { batch_id: b }))]);
|
||||||
|
const rows = (await pool.query(`SELECT id FROM transactions`)).rows;
|
||||||
|
expect(rows).toHaveLength(1);
|
||||||
|
expect((await batch(a)).transaction_id).toBe(rows[0].id);
|
||||||
|
expect((await batch(b)).transaction_id).toBe(rows[0].id);
|
||||||
|
});
|
||||||
|
|
||||||
it("unknown shared_by falls back to the default owner — never a NULL owner", async () => {
|
it("unknown shared_by falls back to the default owner — never a NULL owner", async () => {
|
||||||
const id = uuid();
|
const id = uuid();
|
||||||
await intakePOST(multipart({ source: "uber_eats", shared_by: "nobody@example.test", batch_id: id }, [pdf]));
|
await intakePOST(multipart({ source: "uber_eats", shared_by: "nobody@example.test", batch_id: id }, [pdf]));
|
||||||
|
|||||||
@@ -0,0 +1,175 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Manually resolving a duplicate the automation cannot see.
|
||||||
|
*
|
||||||
|
* Until this route existed there was no way to merge or remove a transaction at
|
||||||
|
* all: `superseded_by_id` was read everywhere and written nowhere in
|
||||||
|
* application code, having been set exactly once by migration 0023. The only
|
||||||
|
* merge path was the reconcile queue, whose candidate list is bounded by an
|
||||||
|
* amount band -- so SP PEAKDESIGN (notified $42.44, charged $43.71, the card's
|
||||||
|
* 3% foreign transaction fee) had no offerable twin.
|
||||||
|
*
|
||||||
|
* The refusals matter more than the happy paths, exactly as in
|
||||||
|
* auto-reconcile.test.ts: this writes to columns that hide a row from every
|
||||||
|
* figure, so a wrong call does not error, it quietly stops counting money.
|
||||||
|
*/
|
||||||
|
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let otherId: number;
|
||||||
|
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
|
||||||
|
|
||||||
|
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
|
||||||
|
const call = (id: number, body: unknown) => merge(req(body), { params: Promise.resolve({ id: String(id) }) });
|
||||||
|
|
||||||
|
// uq_statement_identity is (bank_name, account_number, billing_end_date), so
|
||||||
|
// every statement in a test needs its own account or the second insert collides.
|
||||||
|
let acctSeq = 0;
|
||||||
|
|
||||||
|
async function statementLine(amount: number, date: string, description: string) {
|
||||||
|
const st = await pool.query(
|
||||||
|
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||||
|
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||||
|
[date, ownerId, `4444-0000-0000-${String(1000 + acctSeq++)}`]
|
||||||
|
);
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||||
|
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
|
||||||
|
[st.rows[0].id, date, description, amount, ownerId]
|
||||||
|
);
|
||||||
|
return t.rows[0].id as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
pool = createPool();
|
||||||
|
vi.doMock("@/lib/auth", () => ({
|
||||||
|
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||||
|
}));
|
||||||
|
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM expense_metadata");
|
||||||
|
({ ownerId, otherId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("reconcile mode", () => {
|
||||||
|
it("folds a wallet row into its statement twin even though the amounts differ by the FX fee", async () => {
|
||||||
|
const wallet = await insertTransaction(pool, ownerId, {
|
||||||
|
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
|
||||||
|
});
|
||||||
|
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
|
||||||
|
|
||||||
|
const res = await call(wallet, { other_id: stmt, mode: "reconcile" });
|
||||||
|
expect(res.status ?? 200).toBe(200);
|
||||||
|
|
||||||
|
const r = await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [wallet]);
|
||||||
|
expect(r.rows[0].reconciled_with_id).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses when the target is not a statement row", async () => {
|
||||||
|
const a = await insertTransaction(pool, ownerId, { amount: 10 });
|
||||||
|
const b = await insertTransaction(pool, ownerId, { amount: 10 });
|
||||||
|
const res = await call(a, { other_id: b, mode: "reconcile" });
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses when the source is itself a statement row", async () => {
|
||||||
|
const s1 = await statementLine(10, "2026-08-01", "A");
|
||||||
|
const s2 = await statementLine(10, "2026-08-02", "B");
|
||||||
|
const res = await call(s1, { other_id: s2, mode: "reconcile" });
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("supersede mode", () => {
|
||||||
|
it("carries splits onto the survivor before hiding the loser", async () => {
|
||||||
|
// The whole reason supersede moves curation: `superseded_by_id` removes the
|
||||||
|
// row from every figure, so a split left behind stops counting silently and
|
||||||
|
// a settled debt reappears as owed.
|
||||||
|
const loser = await insertTransaction(pool, ownerId, { amount: 80, description: "DUP" });
|
||||||
|
const survivor = await insertTransaction(pool, ownerId, { amount: 80, description: "KEEP" });
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
|
||||||
|
VALUES ($1,$2,50,true), ($1,$3,50,false)`,
|
||||||
|
[loser, ownerId, otherId]
|
||||||
|
);
|
||||||
|
|
||||||
|
const res = await call(loser, { other_id: survivor, mode: "supersede" });
|
||||||
|
expect(res.status ?? 200).toBe(200);
|
||||||
|
|
||||||
|
const onLoser = await pool.query(`SELECT count(*)::int c FROM transaction_splits WHERE transaction_id=$1`, [loser]);
|
||||||
|
const onSurvivor = await pool.query(
|
||||||
|
`SELECT participant_id, share_percent, settled FROM transaction_splits
|
||||||
|
WHERE transaction_id=$1 ORDER BY participant_id`, [survivor]
|
||||||
|
);
|
||||||
|
expect(onLoser.rows[0].c).toBe(0);
|
||||||
|
expect(onSurvivor.rows).toHaveLength(2);
|
||||||
|
// settled must survive the move -- a delete-and-recreate defaults it to false.
|
||||||
|
expect(onSurvivor.rows.map((r) => r.settled)).toContain(true);
|
||||||
|
|
||||||
|
const marked = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [loser]);
|
||||||
|
expect(marked.rows[0].superseded_by_id).toBe(survivor);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("works between two statement rows, which reconcile cannot express", async () => {
|
||||||
|
const s1 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
|
||||||
|
const s2 = await statementLine(25, "2026-08-01", "IMPORTED TWICE");
|
||||||
|
const res = await call(s1, { other_id: s2, mode: "supersede" });
|
||||||
|
expect(res.status ?? 200).toBe(200);
|
||||||
|
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [s1]);
|
||||||
|
expect(r.rows[0].superseded_by_id).toBe(s2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses to hide a statement row behind a statementless one", async () => {
|
||||||
|
// The wrong direction, and silently wrong: a wallet row holds the amount
|
||||||
|
// BEFORE the card's foreign transaction fee, so burying the statement line
|
||||||
|
// under it leaves the ledger holding $42.44 for a $43.71 charge.
|
||||||
|
const stmt = await statementLine(43.71, "2026-08-15", "SP PEAKDESIGN SAN FRANCISCO USA");
|
||||||
|
const wallet = await insertTransaction(pool, ownerId, {
|
||||||
|
description: "SP PEAKDESIGN", amount: 42.44, transaction_date: "2026-08-16",
|
||||||
|
});
|
||||||
|
const res = await call(stmt, { other_id: wallet, mode: "supersede" });
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
// and the statement row is untouched
|
||||||
|
const r = await pool.query(`SELECT superseded_by_id FROM transactions WHERE id=$1`, [stmt]);
|
||||||
|
expect(r.rows[0].superseded_by_id).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses to bury a row under one that is already superseded", async () => {
|
||||||
|
const a = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||||
|
const b = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||||
|
const c = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||||
|
await call(b, { other_id: c, mode: "supersede" }); // b -> c
|
||||||
|
const res = await call(a, { other_id: b, mode: "supersede" }); // a -> b, but b is buried
|
||||||
|
expect(res.status).toBe(409);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("refusals common to both modes", () => {
|
||||||
|
it("refuses a transaction merging into itself", async () => {
|
||||||
|
const a = await insertTransaction(pool, ownerId, { amount: 1 });
|
||||||
|
const res = await call(a, { other_id: a, mode: "supersede" });
|
||||||
|
expect(res.status).toBe(400);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a row that is already reconciled", async () => {
|
||||||
|
const wallet = await insertTransaction(pool, ownerId, { amount: 12, transaction_date: "2026-08-10" });
|
||||||
|
const stmt = await statementLine(12, "2026-08-10", "X");
|
||||||
|
await call(wallet, { other_id: stmt, mode: "reconcile" });
|
||||||
|
const other = await insertTransaction(pool, ownerId, { amount: 12 });
|
||||||
|
const res = await call(wallet, { other_id: other, mode: "supersede" });
|
||||||
|
expect(res.status).toBe(409);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a transaction belonging to someone else", async () => {
|
||||||
|
const mine = await insertTransaction(pool, ownerId, { amount: 7 });
|
||||||
|
const theirs = await insertTransaction(pool, otherId, { amount: 7 });
|
||||||
|
const res = await call(mine, { other_id: theirs, mode: "supersede" });
|
||||||
|
expect(res.status).toBe(404);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants } from "./helpers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A receipt arriving after the wallet notification must ENRICH that row, not
|
||||||
|
* create a second one.
|
||||||
|
*
|
||||||
|
* The case this is built from is real. Woolworths $103.30 on 2026-08-18: the
|
||||||
|
* wallet event landed at 09:00:14 on the 18th, and nothing created a
|
||||||
|
* transaction until 20:25:06 on the 19th -- 35h25m later -- at which point the
|
||||||
|
* receipt path made its own row from scratch. One shop, two rows, and the
|
||||||
|
* notification's contribution (the instrument, and the suburb) thrown away.
|
||||||
|
*
|
||||||
|
* Sources for one purchase arrive in sequence, not in competition, so the rule
|
||||||
|
* is: the receipt read the register tape and wins on what was bought; anything
|
||||||
|
* only the notification knew survives untouched.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const TOKEN = "test-receipt-token";
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let ingest: typeof import("../../app/api/receipts/ingest/route").POST;
|
||||||
|
|
||||||
|
const ingestReq = (body: unknown) =>
|
||||||
|
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
|
||||||
|
|
||||||
|
/** A row shaped the way the wallet lane writes one: statementless, no line
|
||||||
|
* items, carrying the suburb the notification supplied. */
|
||||||
|
async function walletRow(amount: number, date: string, merchant: string, location: string) {
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions
|
||||||
|
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
|
||||||
|
transaction_type, category, payment_method, merchant_name, merchant_normalized,
|
||||||
|
location, source, source_ref, row_index)
|
||||||
|
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'other', 'card', $3, $3, $5, 'wallet', $6, 0)
|
||||||
|
RETURNING id`,
|
||||||
|
[ownerId, date, merchant, amount, location, `evt-${amount}-${date}`]
|
||||||
|
);
|
||||||
|
const id = t.rows[0].id as number;
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO expense_metadata
|
||||||
|
(transaction_id, source, merchant_normalized, amount, transaction_date,
|
||||||
|
currency, line_items, payment_method_detail, flags)
|
||||||
|
VALUES ($1, 'wallet', $2, $3, $4, 'AUD', '[]'::jsonb, 'Visa', '[]'::jsonb)`,
|
||||||
|
[id, merchant, amount, date]
|
||||||
|
);
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
const receipt = (amount: number, date: string) => ({
|
||||||
|
receipt_uid: `woolworths:3345:${amount}:2026-08-18`,
|
||||||
|
capture_event_id: 900,
|
||||||
|
merchant_name: "Woolworths",
|
||||||
|
store_detail: "Wyndham Vale",
|
||||||
|
transaction_date: date,
|
||||||
|
total: amount,
|
||||||
|
tender_legs: [
|
||||||
|
{ leg_index: 1, amount, card_last4: "6078", card_product: "VISA PREPAID", class: "gift_card" },
|
||||||
|
],
|
||||||
|
line_items: [
|
||||||
|
{ description: "Milk 2L", qty: 1, unit: "ea", amount: 3.5, category: "dairy" },
|
||||||
|
{ description: "Bread", qty: 1, unit: "ea", amount: 4.0, category: "bakery" },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
process.env.RECEIPT_INGEST_TOKEN = TOKEN;
|
||||||
|
pool = createPool();
|
||||||
|
vi.doMock("@/lib/auth", () => ({
|
||||||
|
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||||
|
}));
|
||||||
|
({ POST: ingest } = await import("../../app/api/receipts/ingest/route"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM expense_metadata");
|
||||||
|
({ ownerId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
const txCount = async () =>
|
||||||
|
Number((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n);
|
||||||
|
|
||||||
|
describe("a receipt meeting the wallet row for the same tap", () => {
|
||||||
|
it("adopts it instead of creating a second transaction", async () => {
|
||||||
|
const walletId = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
|
||||||
|
expect(await txCount()).toBe(1);
|
||||||
|
|
||||||
|
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||||
|
|
||||||
|
expect(await txCount()).toBe(1);
|
||||||
|
const meta = await pool.query(
|
||||||
|
`SELECT transaction_id, jsonb_array_length(line_items) AS items, source,
|
||||||
|
payment_method_detail, card_last4
|
||||||
|
FROM expense_metadata`
|
||||||
|
);
|
||||||
|
expect(meta.rowCount).toBe(1);
|
||||||
|
expect(meta.rows[0].transaction_id).toBe(walletId);
|
||||||
|
expect(meta.rows[0].items).toBe(2);
|
||||||
|
expect(meta.rows[0].source).toBe("pantry");
|
||||||
|
// The receipt read the card off the tape; the notification only saw "Visa".
|
||||||
|
expect(meta.rows[0].payment_method_detail).toBe("VISA PREPAID");
|
||||||
|
expect(meta.rows[0].card_last4).toBe("6078");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps the suburb only the notification knew", async () => {
|
||||||
|
await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
|
||||||
|
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||||
|
|
||||||
|
const t = await pool.query(`SELECT location, merchant_normalized, category FROM transactions`);
|
||||||
|
// `location` is absent from the adoption UPDATE's column list on purpose:
|
||||||
|
// the receipt has no suburb, so it must not be able to erase one.
|
||||||
|
expect(t.rows[0].location).toBe("Wyndham Vale, VIC");
|
||||||
|
// ...while what the receipt does know wins.
|
||||||
|
expect(t.rows[0].merchant_normalized).toBe("Woolworths");
|
||||||
|
expect(t.rows[0].category).toBe("groceries");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("creates its own row when there is no wallet row to adopt", async () => {
|
||||||
|
await ingest(ingestReq(receipt(58.4, "2026-08-18")));
|
||||||
|
expect(await txCount()).toBe(1);
|
||||||
|
const meta = await pool.query(`SELECT source, jsonb_array_length(line_items) AS items FROM expense_metadata`);
|
||||||
|
expect(meta.rows[0].source).toBe("pantry");
|
||||||
|
expect(meta.rows[0].items).toBe(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses to adopt when two candidates share the amount", async () => {
|
||||||
|
// Two taps of the same value in the window: adopting either would move this
|
||||||
|
// shop's line items onto a purchase it did not describe.
|
||||||
|
await walletRow(103.3, "2026-08-17", "Woolworths", "Wyndham Vale, VIC");
|
||||||
|
await walletRow(103.3, "2026-08-18", "Coles", "Manor Lakes, VIC");
|
||||||
|
expect(await txCount()).toBe(2);
|
||||||
|
|
||||||
|
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||||
|
|
||||||
|
// Falls back to creating its own row rather than guessing.
|
||||||
|
expect(await txCount()).toBe(3);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses to adopt a row that already carries line items", async () => {
|
||||||
|
const other = await walletRow(103.3, "2026-08-18", "Woolworths", "Wyndham Vale, VIC");
|
||||||
|
await pool.query(
|
||||||
|
`UPDATE expense_metadata SET line_items = '[{"description":"Something else","amount":1}]'::jsonb
|
||||||
|
WHERE transaction_id = $1`, [other]
|
||||||
|
);
|
||||||
|
|
||||||
|
await ingest(ingestReq(receipt(103.3, "2026-08-18")));
|
||||||
|
|
||||||
|
// A fully-described row is a different purchase that happens to share an amount.
|
||||||
|
expect(await txCount()).toBe(2);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||||
|
import { getPendingReconciliations } from "../../lib/queries";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The reconcile queue must not offer what is already resolved.
|
||||||
|
*
|
||||||
|
* A superseded row is hidden from every figure (EXCLUDE_RECONCILED_SOURCE), so
|
||||||
|
* listing it is not merely noise: on 2026-10-03 the queue showed the duplicate
|
||||||
|
* wallet half 7037, superseded under its twin 7038, as a second "Digidirect"
|
||||||
|
* with the same statement line offered as its match. Accepting it there would
|
||||||
|
* have spent that line on a hidden row and left the real one unmatched forever.
|
||||||
|
* findAutoMatches() always had both filters; this query had neither.
|
||||||
|
*/
|
||||||
|
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let acctSeq = 0;
|
||||||
|
|
||||||
|
async function statementLine(amount: number, date: string, description: string) {
|
||||||
|
const st = await pool.query(
|
||||||
|
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||||
|
VALUES ('q.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||||
|
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
|
||||||
|
);
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||||
|
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
|
||||||
|
[st.rows[0].id, date, description, amount, ownerId]
|
||||||
|
);
|
||||||
|
return t.rows[0].id as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeAll(() => { pool = createPool(); });
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
({ ownerId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("getPendingReconciliations", () => {
|
||||||
|
it("does not list a superseded manual row", async () => {
|
||||||
|
const survivor = await insertTransaction(pool, ownerId, { description: "DIGIDIRECT", amount: 1180.05, transaction_date: "2026-08-25" });
|
||||||
|
const hidden = await insertTransaction(pool, ownerId, { description: "Digidirect", amount: 1180.05, transaction_date: "2026-08-25" });
|
||||||
|
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [survivor, hidden]);
|
||||||
|
const line = await statementLine(1180.05, "2026-08-25", "SP DIGIDIRECT STRATHFIELD");
|
||||||
|
|
||||||
|
const queue = await getPendingReconciliations(ownerId);
|
||||||
|
const ids = queue.map((r) => r.id);
|
||||||
|
expect(ids).toContain(survivor);
|
||||||
|
expect(ids).not.toContain(hidden);
|
||||||
|
// The survivor is still offered its line.
|
||||||
|
const row = queue.find((r) => r.id === survivor)!;
|
||||||
|
expect(row.matches.map((m) => m.id)).toContain(line);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not offer a superseded statement row as a match", async () => {
|
||||||
|
const manual = await insertTransaction(pool, ownerId, { description: "Grocer", amount: 42.0, transaction_date: "2026-08-10" });
|
||||||
|
const live = await statementLine(42.0, "2026-08-10", "GROCER");
|
||||||
|
const dupe = await statementLine(42.0, "2026-08-10", "GROCER");
|
||||||
|
await pool.query(`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`, [live, dupe]);
|
||||||
|
|
||||||
|
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
|
||||||
|
const offered = row.matches.map((m) => m.id);
|
||||||
|
expect(offered).toContain(live);
|
||||||
|
expect(offered).not.toContain(dupe);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("offers the exact native twin of a foreign row even when its provisional AUD is far off", async () => {
|
||||||
|
// Live shape, 2026-10-09: wallet INR 112 at a provisional A$2.01; CommBank
|
||||||
|
// charged A$1.68 -- 16% under, outside the +-5% band.
|
||||||
|
const manual = await insertTransaction(pool, ownerId, { description: "SWIGGY LIMITED", amount: 2.01, transaction_date: "2026-10-04" });
|
||||||
|
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [manual]);
|
||||||
|
const twin = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
|
||||||
|
const otherRupees = await statementLine(1.68, "2026-10-04", "Swiggy Limited Bengaluru");
|
||||||
|
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 112 WHERE id = $1`, [twin]);
|
||||||
|
await pool.query(`UPDATE transactions SET foreign_currency_code = 'INR', foreign_currency_amount = 113 WHERE id = $1`, [otherRupees]);
|
||||||
|
|
||||||
|
const [row] = (await getPendingReconciliations(ownerId)).filter((r) => r.id === manual);
|
||||||
|
const offered = row.matches.map((m) => m.id);
|
||||||
|
expect(offered).toContain(twin);
|
||||||
|
// A different native figure at the same AUD is not admitted by this path.
|
||||||
|
expect(offered).not.toContain(otherRupees);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||||
|
import { EXCLUDE_RECONCILED_SOURCE } from "@/lib/analytics-sql";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A charge that never posted (migration 0040).
|
||||||
|
*
|
||||||
|
* Txn 7951: a $1,040.41 Airbnb authorisation hold on Amex, booking cancelled
|
||||||
|
* the next day, hold released. The wallet lane booked the notification as a
|
||||||
|
* purchase and no statement line ever arrived, so it sat in the reconcile queue
|
||||||
|
* forever and counted as spend everywhere. A void has no survivor, which is why
|
||||||
|
* neither reconcile nor supersede could express it.
|
||||||
|
*
|
||||||
|
* As in merge-supersede.test.ts, the refusals matter as much as the happy path:
|
||||||
|
* a void quietly stops money counting.
|
||||||
|
*/
|
||||||
|
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let otherId: number;
|
||||||
|
let voidRoute: typeof import("../../app/api/transactions/[id]/void/route");
|
||||||
|
let merge: typeof import("../../app/api/transactions/[id]/merge/route").POST;
|
||||||
|
|
||||||
|
const req = (body: unknown) => ({ json: async () => body, headers: { get: () => null } }) as never;
|
||||||
|
const params = (id: number) => ({ params: Promise.resolve({ id: String(id) }) });
|
||||||
|
const doVoid = (id: number, reason: unknown = "hold released, booking cancelled") =>
|
||||||
|
voidRoute.POST(req({ reason }), params(id));
|
||||||
|
|
||||||
|
let acctSeq = 0;
|
||||||
|
async function statementLine(amount: number, date: string, description: string) {
|
||||||
|
const st = await pool.query(
|
||||||
|
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||||
|
VALUES ('s.pdf', $3, 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||||
|
[date, ownerId, `5555-0000-0000-${String(1000 + acctSeq++)}`]
|
||||||
|
);
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
|
||||||
|
VALUES ($1, $2, $3, $4, 'debit', $5) RETURNING id`,
|
||||||
|
[st.rows[0].id, date, description, amount, ownerId]
|
||||||
|
);
|
||||||
|
return t.rows[0].id as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What every "count each purchase once" query sees. */
|
||||||
|
async function countedIds(): Promise<number[]> {
|
||||||
|
const r = await pool.query(
|
||||||
|
`SELECT t.id FROM transactions t WHERE ${EXCLUDE_RECONCILED_SOURCE} ORDER BY t.id`
|
||||||
|
);
|
||||||
|
return r.rows.map((x) => x.id);
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
pool = createPool();
|
||||||
|
vi.doMock("@/lib/auth", () => ({
|
||||||
|
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||||
|
}));
|
||||||
|
voidRoute = await import("../../app/api/transactions/[id]/void/route");
|
||||||
|
({ POST: merge } = await import("../../app/api/transactions/[id]/merge/route"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM expense_metadata");
|
||||||
|
({ ownerId, otherId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("voiding a charge that never posted", () => {
|
||||||
|
it("drops the row out of every figure and the reconcile queue, but keeps it", async () => {
|
||||||
|
const { getPendingReconciliations } = await import("@/lib/queries");
|
||||||
|
const hold = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 1040.41 });
|
||||||
|
const real = await insertTransaction(pool, ownerId, { description: "AIRBNB", amount: 529.05 });
|
||||||
|
|
||||||
|
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).toContain(hold);
|
||||||
|
|
||||||
|
const res = await doVoid(hold);
|
||||||
|
expect(res.status ?? 200).toBe(200);
|
||||||
|
|
||||||
|
expect(await countedIds()).toEqual([real]);
|
||||||
|
expect((await getPendingReconciliations(ownerId)).map((t) => t.id)).not.toContain(hold);
|
||||||
|
const r = await pool.query(`SELECT voided_at, void_reason FROM transactions WHERE id = $1`, [hold]);
|
||||||
|
expect(r.rows[0].voided_at).not.toBeNull();
|
||||||
|
expect(r.rows[0].void_reason).toBe("hold released, booking cancelled");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("can be undone", async () => {
|
||||||
|
const hold = await insertTransaction(pool, ownerId, { amount: 1040.41 });
|
||||||
|
await doVoid(hold);
|
||||||
|
const res = await voidRoute.DELETE(req({}), params(hold));
|
||||||
|
expect(res.status ?? 200).toBe(200);
|
||||||
|
expect(await countedIds()).toEqual([hold]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("requires a reason", async () => {
|
||||||
|
const hold = await insertTransaction(pool, ownerId, { amount: 10 });
|
||||||
|
expect((await doVoid(hold, " ")).status).toBe(400);
|
||||||
|
expect((await doVoid(hold, null)).status).toBe(400);
|
||||||
|
expect(await countedIds()).toEqual([hold]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a statement line -- that charge posted", async () => {
|
||||||
|
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
|
||||||
|
expect((await doVoid(stmt)).status).toBe(409);
|
||||||
|
expect(await countedIds()).toEqual([stmt]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a reconciled row -- it found its statement line, so it posted", async () => {
|
||||||
|
const wallet = await insertTransaction(pool, ownerId, { amount: 50, transaction_date: "2026-09-10" });
|
||||||
|
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
|
||||||
|
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [stmt, wallet]);
|
||||||
|
expect((await doVoid(wallet)).status).toBe(409);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a row with a settled split -- someone already paid their share", async () => {
|
||||||
|
const hold = await insertTransaction(pool, ownerId, { amount: 80 });
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled)
|
||||||
|
VALUES ($1, $2, 50, true)`,
|
||||||
|
[hold, otherId]
|
||||||
|
);
|
||||||
|
expect((await doVoid(hold)).status).toBe(409);
|
||||||
|
expect(await countedIds()).toEqual([hold]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is enforced by the database, not only the route", async () => {
|
||||||
|
const stmt = await statementLine(50, "2026-09-10", "SOMETHING");
|
||||||
|
await expect(
|
||||||
|
pool.query(`UPDATE transactions SET voided_at = now(), void_reason = 'x' WHERE id = $1`, [stmt])
|
||||||
|
).rejects.toThrow(/transactions_void_statementless/);
|
||||||
|
|
||||||
|
const manual = await insertTransaction(pool, ownerId, { amount: 5 });
|
||||||
|
await expect(
|
||||||
|
pool.query(`UPDATE transactions SET voided_at = now() WHERE id = $1`, [manual])
|
||||||
|
).rejects.toThrow(/transactions_void_has_reason/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the matchers never use a voided row", () => {
|
||||||
|
it("auto-reconcile does not pair a voided row with a statement line of the same amount", async () => {
|
||||||
|
const { findAutoMatches } = await import("@/lib/auto-reconcile");
|
||||||
|
const hold = await insertTransaction(pool, ownerId, { amount: 120, transaction_date: "2026-09-10" });
|
||||||
|
await statementLine(120, "2026-09-11", "SOMETHING");
|
||||||
|
expect((await findAutoMatches(ownerId)).map((m) => m.manual_id)).toContain(hold);
|
||||||
|
|
||||||
|
await doVoid(hold);
|
||||||
|
expect(await findAutoMatches(ownerId)).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a receipt for a rebooking at the same amount adopts the live row, not the released hold", async () => {
|
||||||
|
const { findAdoptableTransaction } = await import("@/lib/auto-reconcile");
|
||||||
|
const hold = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-10" });
|
||||||
|
const rebooked = await insertTransaction(pool, ownerId, { amount: 300, transaction_date: "2026-09-11" });
|
||||||
|
// Two candidates: the adopter refuses rather than guess.
|
||||||
|
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBeNull();
|
||||||
|
|
||||||
|
await doVoid(hold);
|
||||||
|
expect(await findAdoptableTransaction(ownerId, 300, "2026-09-11")).toBe(rebooked);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("merge refuses a voided row on either side", async () => {
|
||||||
|
const hold = await insertTransaction(pool, ownerId, { amount: 70, transaction_date: "2026-09-10" });
|
||||||
|
const stmt = await statementLine(70, "2026-09-10", "SOMETHING");
|
||||||
|
await doVoid(hold);
|
||||||
|
const res = await merge(req({ other_id: stmt, mode: "reconcile" }), params(hold));
|
||||||
|
expect(res.status).toBe(409);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,176 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants } from "./helpers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The mirror of receipt-adopts-wallet-row: a wallet notification arriving AFTER
|
||||||
|
* something else already booked the purchase must fold into that row.
|
||||||
|
*
|
||||||
|
* Built from the failure, not from the design. The 2026-08-18 Woolworths shop
|
||||||
|
* ($103.30, Visa gift card) reached the ledger twice: txn 6858 from the pantry
|
||||||
|
* receipt on the 19th, and txn 6927 from the wallet lane on the 22nd, when the
|
||||||
|
* first 21 wallet events were backfilled days after their taps. The sequencing
|
||||||
|
* the lane is designed around -- wallet first, receipt second -- is a tendency,
|
||||||
|
* and every writer has to survive it not holding.
|
||||||
|
*
|
||||||
|
* Two rules are load-bearing here and are tested separately below:
|
||||||
|
* - the merchant must agree, because losing the "no line items" guard that
|
||||||
|
* protects the other direction means amount+date alone could swallow a tap
|
||||||
|
* into an unrelated purchase of the same value
|
||||||
|
* - a gift-card tap is `credits`, because `card` means "a statement line is
|
||||||
|
* coming" and the reconcile queue filters on exactly that
|
||||||
|
*/
|
||||||
|
|
||||||
|
const TOKEN = "test-wallet-token";
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
|
||||||
|
|
||||||
|
const req = (body: unknown) =>
|
||||||
|
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
|
||||||
|
|
||||||
|
/** A row shaped the way the pantry receipt lane writes one: statementless,
|
||||||
|
* itemised, no location (nothing but a notification supplies a suburb). */
|
||||||
|
async function receiptRow(amount: number, date: string, merchant: string, opts: { items?: number; meta?: boolean } = {}) {
|
||||||
|
const { items = 3, meta = true } = opts;
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions
|
||||||
|
(owner_id, statement_id, transaction_date, description, amount, amount_aud,
|
||||||
|
transaction_type, category, payment_method, merchant_name, merchant_normalized, row_index)
|
||||||
|
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'groceries', 'credits', $3, $3, 0)
|
||||||
|
RETURNING id`,
|
||||||
|
[ownerId, date, merchant, amount]
|
||||||
|
);
|
||||||
|
const id = t.rows[0].id as number;
|
||||||
|
if (meta) {
|
||||||
|
const lineItems = Array.from({ length: items }, (_, i) => ({ description: `item ${i}`, qty: 1, amount: 1 }));
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO expense_metadata
|
||||||
|
(transaction_id, source, merchant_normalized, amount, transaction_date, currency, line_items, flags)
|
||||||
|
VALUES ($1, 'pantry', $2, $3, $4, 'AUD', $5::jsonb, '[]'::jsonb)`,
|
||||||
|
[id, merchant, amount, date, JSON.stringify(lineItems)]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The real event 80: the Apple Wallet half of a Visa gift-card tap. */
|
||||||
|
const woolworthsGiftCard = {
|
||||||
|
event_id: 80, kind: "transaction", direction: "unknown", amount: 103.3,
|
||||||
|
amount_text: "$103.30", currency: "$?", merchant: "Woolworths", issuer: "Visa",
|
||||||
|
notif_app: "Wallet", notif_title: "Visa Gift Card",
|
||||||
|
notif_subtitle: "Woolworths, Wyndham Vale, VIC",
|
||||||
|
occurred_at: "2026-08-18T09:00:14+10:00",
|
||||||
|
};
|
||||||
|
|
||||||
|
const rows = async () =>
|
||||||
|
(await pool.query(
|
||||||
|
`SELECT id, amount, merchant_normalized, location, source, source_ref, payment_method
|
||||||
|
FROM transactions ORDER BY id`
|
||||||
|
)).rows;
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
process.env.WALLET_INGEST_TOKEN = TOKEN;
|
||||||
|
pool = createPool();
|
||||||
|
vi.doMock("@/lib/auth", () => ({
|
||||||
|
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||||
|
}));
|
||||||
|
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM expense_metadata");
|
||||||
|
({ ownerId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("a wallet notification for a purchase the ledger already holds", () => {
|
||||||
|
it("adopts the receipt's row instead of creating a second one", async () => {
|
||||||
|
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
|
||||||
|
|
||||||
|
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||||
|
|
||||||
|
expect(body.skipped).toBe("adopted");
|
||||||
|
expect(body.transactionId).toBe(existing);
|
||||||
|
expect(await rows()).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("gives the adopted row the suburb, and leaves what the receipt read alone", async () => {
|
||||||
|
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345");
|
||||||
|
await ingest(req(woolworthsGiftCard));
|
||||||
|
|
||||||
|
const [t] = await rows();
|
||||||
|
// Wallet's own field: nothing else on the estate writes a suburb.
|
||||||
|
expect(t.location).toBe("Wyndham Vale, VIC");
|
||||||
|
// The receipt read the register tape. The notification does not get to
|
||||||
|
// rename the merchant to its own shorter string.
|
||||||
|
expect(t.merchant_normalized).toBe("WOOLWORTHS 3345");
|
||||||
|
expect(t.id).toBe(existing);
|
||||||
|
|
||||||
|
const [m] = (await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
|
||||||
|
expect(m.source).toBe("pantry");
|
||||||
|
expect(m.line_items).toHaveLength(3);
|
||||||
|
expect(m.payment_method_detail).toBe("Visa");
|
||||||
|
expect(m.flags).toContain("wallet_evt:80");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is idempotent when the interpreter replays every event", async () => {
|
||||||
|
// Bumping prompt_version re-interprets and re-posts the whole capture
|
||||||
|
// history on purpose -- that is the replay mechanism. An adopted row keeps
|
||||||
|
// no wallet source_ref of its own, so the flag is what makes replay safe.
|
||||||
|
await ingest(req(woolworthsGiftCard));
|
||||||
|
const second = await (await ingest(req(woolworthsGiftCard))).json();
|
||||||
|
expect(second.skipped).toBe("already_ingested");
|
||||||
|
expect(await rows()).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("writes its own row when the merchant disagrees", async () => {
|
||||||
|
// Same money, same day, different shop. Folding into it would make the tap
|
||||||
|
// vanish -- worse than a duplicate, which is at least visible in the queue.
|
||||||
|
await receiptRow(103.3, "2026-08-18", "Bunnings Werribee");
|
||||||
|
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||||
|
|
||||||
|
expect(body.skipped).toBeUndefined();
|
||||||
|
expect(await rows()).toHaveLength(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("writes its own row when two candidates share the amount", async () => {
|
||||||
|
await receiptRow(103.3, "2026-08-18", "Woolworths Wyndham Vale");
|
||||||
|
await receiptRow(103.3, "2026-08-18", "Woolworths Point Cook");
|
||||||
|
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||||
|
|
||||||
|
expect(body.skipped).toBeUndefined();
|
||||||
|
expect(await rows()).toHaveLength(3);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("creates the metadata row when the adopted transaction has none", async () => {
|
||||||
|
const existing = await receiptRow(103.3, "2026-08-18", "WOOLWORTHS 3345", { meta: false });
|
||||||
|
await ingest(req(woolworthsGiftCard));
|
||||||
|
|
||||||
|
const meta = (await pool.query(`SELECT * FROM expense_metadata`)).rows;
|
||||||
|
expect(meta).toHaveLength(1);
|
||||||
|
expect(meta[0].transaction_id).toBe(existing);
|
||||||
|
expect(meta[0].card_last4).toBeNull();
|
||||||
|
expect(meta[0].flags).toContain("wallet_evt:80");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("a stored-value instrument", () => {
|
||||||
|
it("is booked as credits, so it never enters the reconcile queue", async () => {
|
||||||
|
const body = await (await ingest(req(woolworthsGiftCard))).json();
|
||||||
|
const [t] = await rows();
|
||||||
|
|
||||||
|
expect(body.transactionId).toBeTruthy();
|
||||||
|
// 'card' would park it forever: findAutoMatches and the queue both exclude
|
||||||
|
// 'credits' and 'cash' precisely because no statement line is coming.
|
||||||
|
expect(t.payment_method).toBe("credits");
|
||||||
|
expect(body.flags).toContain("stored_value_instrument");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves an ordinary card tap as card", async () => {
|
||||||
|
const body = await (await ingest(req({ ...woolworthsGiftCard, notif_title: "Payment", issuer: "Mastercard", notif_app: "Westpac" }))).json();
|
||||||
|
const [t] = await rows();
|
||||||
|
expect(t.payment_method).toBe("card");
|
||||||
|
expect(body.flags).not.toContain("stored_value_instrument");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,403 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants } from "./helpers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wallet notifications becoming transactions.
|
||||||
|
*
|
||||||
|
* Every fixture is a real captured event from capture.wallet_events
|
||||||
|
* (2026-08-13 .. 08-22). The rules being tested are the ones the design turned
|
||||||
|
* on, so each has a reason rather than being a shape check:
|
||||||
|
*
|
||||||
|
* - a transaction for EVERY tap, because overseas is where live spend matters
|
||||||
|
* - the pair is MERGED, never deduped: the issuer half has the card tail, the
|
||||||
|
* Wallet half has the merchant and suburb, neither alone is complete
|
||||||
|
* - the NATIVE amount is what gets stored for a foreign charge, because the
|
||||||
|
* AUD figure is the bank's own conversion and cannot be predicted
|
||||||
|
* - a bare "$" is recorded as unknown, never assumed AUD
|
||||||
|
*/
|
||||||
|
|
||||||
|
const TOKEN = "test-wallet-token";
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let ingest: typeof import("../../app/api/wallet/ingest/route").POST;
|
||||||
|
|
||||||
|
const req = (body: unknown) =>
|
||||||
|
({ headers: { get: (h: string) => (h === "x-ingest-token" ? TOKEN : null) }, json: async () => body }) as never;
|
||||||
|
|
||||||
|
const noAuth = (body: unknown) =>
|
||||||
|
({ headers: { get: () => null }, json: async () => body }) as never;
|
||||||
|
|
||||||
|
// --- real captured events -------------------------------------------------
|
||||||
|
const breadworldBank = {
|
||||||
|
event_id: 64, kind: "transaction", direction: "debit", amount: 13.5,
|
||||||
|
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
|
||||||
|
card_tail: "8032", issuer: "Mastercard", notif_app: "Westpac",
|
||||||
|
notif_body: "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU",
|
||||||
|
occurred_at: "2026-08-13T08:01:28+10:00",
|
||||||
|
};
|
||||||
|
const breadworldWallet = {
|
||||||
|
event_id: 65, kind: "transaction", direction: "unknown", amount: 13.5,
|
||||||
|
amount_text: "$13.50", currency: "$?", merchant: "Breadworld Bakery Cafe",
|
||||||
|
notif_app: "Wallet", notif_subtitle: "Breadworld Bakery Cafe, Wyndham Vale, VIC",
|
||||||
|
occurred_at: "2026-08-13T08:01:35+10:00",
|
||||||
|
};
|
||||||
|
const amexRewardPay = {
|
||||||
|
event_id: 87, kind: "transaction", direction: "debit", amount: 5103.49,
|
||||||
|
amount_text: "A$5,103.49", currency: "AUD", merchant: "REWARDPAY",
|
||||||
|
card_tail: "01000", issuer: "American Express", notif_app: "Amex AU",
|
||||||
|
notif_body: "You have made a A$5,103.49 charge on your American Express® Card ending in 01000 at REWARDPAY.",
|
||||||
|
occurred_at: "2026-08-20T11:54:43+10:00",
|
||||||
|
};
|
||||||
|
const ebayOrderPlaced = {
|
||||||
|
event_id: 66, kind: "not_transaction", direction: "unknown", amount: null,
|
||||||
|
notif_app: "Wallet", notif_body: "eBay has received your order.",
|
||||||
|
occurred_at: "2026-08-13T09:45:55+10:00",
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Only the rows this lane wrote -- some tests seed statement history for the
|
||||||
|
* rate reference, and those must not be mistaken for wallet output. */
|
||||||
|
const txns = async () =>
|
||||||
|
(await pool.query(
|
||||||
|
`SELECT id, amount, amount_aud, merchant_normalized, location, source, source_ref,
|
||||||
|
foreign_currency_amount, foreign_currency_code, payment_method, statement_id
|
||||||
|
FROM transactions WHERE source = 'wallet' ORDER BY id`
|
||||||
|
)).rows;
|
||||||
|
|
||||||
|
const metas = async () =>
|
||||||
|
(await pool.query(`SELECT * FROM expense_metadata ORDER BY id`)).rows;
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
process.env.WALLET_INGEST_TOKEN = TOKEN;
|
||||||
|
pool = createPool();
|
||||||
|
vi.doMock("@/lib/auth", () => ({
|
||||||
|
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||||
|
}));
|
||||||
|
({ POST: ingest } = await import("../../app/api/wallet/ingest/route"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
// No network in tests. The ECB fallback treats any fetch failure as "no
|
||||||
|
// rate", which is what every pre-existing currency test assumes; the one test
|
||||||
|
// about the fallback stubs a rate of its own.
|
||||||
|
vi.stubGlobal("fetch", async () => { throw new Error("no network in tests"); });
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM expense_metadata");
|
||||||
|
({ ownerId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("auth", () => {
|
||||||
|
it("fails closed without the token", async () => {
|
||||||
|
expect((await ingest(noAuth(breadworldBank))).status).toBe(401);
|
||||||
|
expect(await txns()).toHaveLength(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("what reaches the ledger", () => {
|
||||||
|
it("creates a statementless card transaction for a tap", async () => {
|
||||||
|
const body = await (await ingest(req(breadworldBank))).json();
|
||||||
|
expect(body.transactionId).toBeTruthy();
|
||||||
|
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(Number(t.amount)).toBe(13.5);
|
||||||
|
expect(t.source).toBe("wallet");
|
||||||
|
expect(t.source_ref).toBe("evt:64");
|
||||||
|
// Statementless + card is exactly what puts it in the reconcile queue, with
|
||||||
|
// no extra wiring: that is how the statement eventually confirms it.
|
||||||
|
expect(t.statement_id).toBeNull();
|
||||||
|
expect(t.payment_method).toBe("card");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores an eBay order-lifecycle notification", async () => {
|
||||||
|
// 10 of the 13 non-transactions in the capture window are these.
|
||||||
|
const body = await (await ingest(req(ebayOrderPlaced))).json();
|
||||||
|
expect(body.skipped).toBe("not_a_transaction");
|
||||||
|
expect(await txns()).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is idempotent on the event id", async () => {
|
||||||
|
await ingest(req(breadworldBank));
|
||||||
|
const second = await (await ingest(req(breadworldBank))).json();
|
||||||
|
expect(second.skipped).toBe("already_ingested");
|
||||||
|
expect(await txns()).toHaveLength(1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the notification pair", () => {
|
||||||
|
it("merges the two halves into ONE transaction, keeping both contributions", async () => {
|
||||||
|
await ingest(req(breadworldBank)); // has the card tail
|
||||||
|
await ingest(req(breadworldWallet)); // has the suburb
|
||||||
|
|
||||||
|
const rows = await txns();
|
||||||
|
expect(rows).toHaveLength(1);
|
||||||
|
// The Wallet half supplies the suburb; the issuer half never had one.
|
||||||
|
expect(rows[0].location).toBe("Wyndham Vale, VIC");
|
||||||
|
|
||||||
|
const [m] = await metas();
|
||||||
|
// ...and the issuer half's card tail survives the merge.
|
||||||
|
expect(m.card_last4).toBe("8032");
|
||||||
|
expect(m.payment_method_detail).toBe("Mastercard");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("merges in either arrival order", async () => {
|
||||||
|
await ingest(req(breadworldWallet));
|
||||||
|
await ingest(req(breadworldBank));
|
||||||
|
|
||||||
|
expect(await txns()).toHaveLength(1);
|
||||||
|
const [m] = await metas();
|
||||||
|
expect(m.card_last4).toBe("8032");
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(t.location).toBe("Wyndham Vale, VIC");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the notification pair is recognised by time and app, not amount alone", () => {
|
||||||
|
it("makes ONE row when both halves are posted at the same moment", async () => {
|
||||||
|
// n8n posts a poll's items concurrently, and both halves of a tap usually
|
||||||
|
// land in the same poll. Before the lock, each ran its pair lookup before
|
||||||
|
// the other inserted: 11 taps became 22 rows, identical created_at on each.
|
||||||
|
await Promise.all([ingest(req(breadworldBank)), ingest(req(breadworldWallet))]);
|
||||||
|
expect(await txns()).toHaveLength(1);
|
||||||
|
const [m] = await metas();
|
||||||
|
expect(m.card_last4).toBe("8032");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps two same-amount charges hours apart as two purchases", async () => {
|
||||||
|
// Event 275 ($50 at A MEDIUM) merged into the UNICEF $50 donation from 17
|
||||||
|
// hours earlier and renamed it, because the old window was +/-1 day.
|
||||||
|
await ingest(req({
|
||||||
|
...amexRewardPay, event_id: 267, amount: 50, amount_text: "A$50.00", merchant: "UNICEF",
|
||||||
|
notif_body: "You have made a A$50.00 charge on your American Express® Card ending in 01000 at UNICEF.",
|
||||||
|
occurred_at: "2026-10-01T11:02:40+10:00",
|
||||||
|
}));
|
||||||
|
await ingest(req({
|
||||||
|
...amexRewardPay, event_id: 275, amount: 50, amount_text: "$50.00", merchant: "A MEDIUM",
|
||||||
|
notif_body: "You have made a $50.00 charge on your American Express® Card ending in 01000 at A MEDIUM.",
|
||||||
|
occurred_at: "2026-10-02T04:06:49+10:00",
|
||||||
|
}));
|
||||||
|
const rows = await txns();
|
||||||
|
expect(rows).toHaveLength(2);
|
||||||
|
expect(rows.map((r) => r.merchant_normalized).sort()).toEqual(["A MEDIUM", "UNICEF"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("never pairs two notifications from the SAME app, however close", async () => {
|
||||||
|
// A pair is the issuer half plus the Wallet half. Two from one app are two
|
||||||
|
// charges -- two $20 coffees bought together, say.
|
||||||
|
await ingest(req(breadworldBank));
|
||||||
|
await ingest(req({ ...breadworldBank, event_id: 640, occurred_at: "2026-08-13T08:01:40+10:00" }));
|
||||||
|
expect(await txns()).toHaveLength(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("records a merged half, so a replay cannot merge it again", async () => {
|
||||||
|
await ingest(req(breadworldBank));
|
||||||
|
await ingest(req(breadworldWallet));
|
||||||
|
const again = await (await ingest(req(breadworldWallet))).json();
|
||||||
|
expect(again.skipped).toBe("already_ingested");
|
||||||
|
const [m] = await metas();
|
||||||
|
expect(m.flags).toContain("wallet_evt:65");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("notifications that name an amount but are not purchases", () => {
|
||||||
|
it("skips an overdraft notice", async () => {
|
||||||
|
// Event 235 became a $21.21 purchase named "To avoid fees and interest,".
|
||||||
|
const body = await (await ingest(req({
|
||||||
|
event_id: 235, kind: "transaction", direction: "debit", amount: 21.21,
|
||||||
|
amount_text: "$21.21", currency: "$?", merchant: null, notif_app: "CommBank",
|
||||||
|
notif_body: "Your Smart Access 1910 is overdrawn by $21.21. To avoid fees and interest, bring your account to positive by midnight 21 Sep (Syd/Melb time).",
|
||||||
|
occurred_at: "2026-09-22T09:00:18+10:00",
|
||||||
|
}))).json();
|
||||||
|
expect(body.skipped).toBe("account_notice");
|
||||||
|
expect(await txns()).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("skips a transit-card fare, because the card's load is the spend", async () => {
|
||||||
|
// Event 200: the Octopus load is on the bank statement; each fare draws it
|
||||||
|
// down. 11 of 13 fares were already not_transaction; this makes the other 2 agree.
|
||||||
|
const body = await (await ingest(req({
|
||||||
|
event_id: 200, kind: "transaction", direction: "debit", amount: 105,
|
||||||
|
amount_text: "HKD 105.00", currency: "HKD", merchant: "Octopus", notif_app: "Wallet",
|
||||||
|
notif_title: "Octopus", notif_subtitle: "HKD 105.00 for Train",
|
||||||
|
notif_body: "Your new balance is -HKD 12.60.",
|
||||||
|
occurred_at: "2026-09-14T02:00:37+10:00",
|
||||||
|
}))).json();
|
||||||
|
expect(body.skipped).toBe("stored_value_drawdown");
|
||||||
|
expect(await txns()).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not read a Westpac descriptor out of another bank's message", async () => {
|
||||||
|
await ingest(req({
|
||||||
|
event_id: 2350, kind: "transaction", direction: "debit", amount: 9.5,
|
||||||
|
amount_text: "$9.50", currency: "$?", merchant: null, notif_app: "CommBank",
|
||||||
|
notif_body: "Payment of $9.50 made. Thanks for banking with us.",
|
||||||
|
occurred_at: "2026-09-22T09:00:18+10:00",
|
||||||
|
}));
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(t.merchant_normalized).toBe("Card purchase");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("dating", () => {
|
||||||
|
it("dates a tap in Melbourne local time, as the statements do", async () => {
|
||||||
|
// 21:30 UTC on 30 Sep is 07:30 on 1 Oct in Melbourne. Slicing the ISO
|
||||||
|
// string filed it a day early.
|
||||||
|
await ingest(req({ ...breadworldBank, occurred_at: "2026-09-30T21:30:00Z" }));
|
||||||
|
const { rows } = await pool.query(
|
||||||
|
`SELECT transaction_date::text AS d FROM transactions WHERE source = 'wallet'`
|
||||||
|
);
|
||||||
|
expect(rows[0].d).toBe("2026-10-01");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("currency", () => {
|
||||||
|
it("falls back to the ECB rate when the ledger has no history", async () => {
|
||||||
|
// Hong Kong: no HKD statement row existed, so every tap stored A$0.00.
|
||||||
|
const calls: string[] = [];
|
||||||
|
vi.stubGlobal("fetch", async (url: string) => {
|
||||||
|
calls.push(String(url));
|
||||||
|
return { ok: true, json: async () => ({ rates: { AUD: 0.19 } }) };
|
||||||
|
});
|
||||||
|
await ingest(req({
|
||||||
|
event_id: 167, kind: "transaction", direction: "debit", amount: 310.1,
|
||||||
|
amount_text: "HKD 310.10", currency: "HKD", merchant: "Wonder Taxi", notif_app: "Wallet",
|
||||||
|
notif_subtitle: "Wonder Taxi, Tsim Sha Tsui East, Kowloon",
|
||||||
|
occurred_at: "2026-09-10T08:54:00+10:00",
|
||||||
|
}));
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(Number(t.amount)).toBeCloseTo(58.92, 2);
|
||||||
|
expect(Number(t.foreign_currency_amount)).toBe(310.1);
|
||||||
|
expect(calls[0]).toContain("/2026-09-10?from=HKD&to=AUD");
|
||||||
|
const [m] = await metas();
|
||||||
|
// Flagged as the mid-rate, not the bank's: it lacks the card's FX fee.
|
||||||
|
expect(m.flags).toEqual(expect.arrayContaining(["aud_provisional", "rate_ecb"]));
|
||||||
|
expect(m.flags).not.toContain("no_rate_reference");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("records AUD when Amex states it explicitly", async () => {
|
||||||
|
await ingest(req(amexRewardPay));
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(t.foreign_currency_code).toBeNull();
|
||||||
|
expect(Number(t.amount)).toBe(5103.49);
|
||||||
|
const [m] = await metas();
|
||||||
|
expect(m.currency).toBe("AUD");
|
||||||
|
expect(m.flags).not.toContain("currency_unverified");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("flags a bare dollar sign rather than assuming AUD", async () => {
|
||||||
|
// 14 of 21 captured transaction rows are a bare "$". Assuming AUD is
|
||||||
|
// harmless at home and a 23x error on a THB tap abroad.
|
||||||
|
await ingest(req(breadworldBank));
|
||||||
|
const [m] = await metas();
|
||||||
|
expect(m.currency).toBeNull();
|
||||||
|
expect(m.flags).toContain("currency_unverified");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("stores a foreign charge on the NATIVE pair, with a provisional AUD", async () => {
|
||||||
|
// Seed enough history for the median rate to exist (n >= 5).
|
||||||
|
const st = await pool.query(
|
||||||
|
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||||
|
VALUES ('s.pdf','1','Test Bank','2026-01-01','2026-01-31',$1) RETURNING id`, [ownerId]
|
||||||
|
);
|
||||||
|
for (let i = 0; i < 6; i++) {
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transactions (statement_id, transaction_date, description, amount,
|
||||||
|
transaction_type, owner_id, foreign_currency_amount, foreign_currency_code)
|
||||||
|
VALUES ($1,'2026-01-10','BANGKOK',4.57,'debit',$2,100,'THB')`, [st.rows[0].id, ownerId]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
await ingest(req({
|
||||||
|
event_id: 999, kind: "transaction", direction: "debit", amount: 500,
|
||||||
|
amount_text: "฿500", merchant: "7-Eleven", notif_app: "Wallet",
|
||||||
|
notif_subtitle: "7-Eleven, Bangkok", occurred_at: "2026-08-20T12:00:00+10:00",
|
||||||
|
}));
|
||||||
|
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(Number(t.foreign_currency_amount)).toBe(500);
|
||||||
|
expect(t.foreign_currency_code).toBe("THB");
|
||||||
|
// The native figure is exact on both sides, so it is what the statement
|
||||||
|
// will match on. amount is only ever provisional here.
|
||||||
|
expect(Number(t.amount)).toBeCloseTo(22.85, 1);
|
||||||
|
const [m] = await metas();
|
||||||
|
expect(m.flags).toContain("aud_provisional");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("still records a foreign charge when there is no rate history", async () => {
|
||||||
|
await ingest(req({
|
||||||
|
event_id: 998, kind: "transaction", direction: "debit", amount: 42,
|
||||||
|
amount_text: "€42.00", merchant: "Somewhere", notif_app: "Wallet",
|
||||||
|
occurred_at: "2026-08-20T12:00:00+10:00",
|
||||||
|
}));
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(t.foreign_currency_code).toBe("EUR");
|
||||||
|
const [m] = await metas();
|
||||||
|
// Says so, rather than inventing a rate.
|
||||||
|
expect(m.flags).toContain("no_rate_reference");
|
||||||
|
// And never stores the native figure as AUD (672006 VND became A$672,006).
|
||||||
|
expect(Number(t.amount)).toBe(0);
|
||||||
|
expect(Number(t.foreign_currency_amount)).toBe(42);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("derives the rate from statement rows only, never from wallet rows", async () => {
|
||||||
|
// Six statementless rows at a nonsense 1:1 rate must not count as history.
|
||||||
|
for (let i = 0; i < 6; i++) {
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transactions (transaction_date, description, amount, transaction_type,
|
||||||
|
owner_id, foreign_currency_amount, foreign_currency_code, source, source_ref)
|
||||||
|
VALUES ('2026-01-10','VN',1000,'debit',$1,1000,'VND','wallet',$2)`,
|
||||||
|
[ownerId, `evt:seed${i}`]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
await ingest(req({
|
||||||
|
event_id: 997, kind: "transaction", direction: "debit", amount: 672006,
|
||||||
|
amount_text: "672006 VND", currency: "VND", merchant: "EVISA VIETNAM", notif_app: "CommBank",
|
||||||
|
occurred_at: "2026-09-05T11:22:00+10:00",
|
||||||
|
}));
|
||||||
|
const t = (await txns()).find((r) => r.source_ref === "evt:997")!;
|
||||||
|
expect(Number(t.amount)).toBe(0);
|
||||||
|
const [m] = (await metas()).filter((r) => r.source_message_id === "evt:997");
|
||||||
|
expect(m.flags).toContain("no_rate_reference");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("pairs foreign halves on the native figure, not the placeholder amount", async () => {
|
||||||
|
await ingest(req({
|
||||||
|
event_id: 996, kind: "transaction", direction: "debit", amount: 42,
|
||||||
|
amount_text: "€42.00", merchant: "Cafe A", notif_app: "Wallet",
|
||||||
|
occurred_at: "2026-08-20T12:00:00+10:00",
|
||||||
|
}));
|
||||||
|
await ingest(req({
|
||||||
|
event_id: 995, kind: "transaction", direction: "debit", amount: 17,
|
||||||
|
amount_text: "€17.00", merchant: "Cafe B", notif_app: "Wallet",
|
||||||
|
occurred_at: "2026-08-20T12:05:00+10:00",
|
||||||
|
}));
|
||||||
|
// Both have amount 0; they are different purchases and must stay two rows.
|
||||||
|
expect((await txns()).filter((r) => r.foreign_currency_code === "EUR")).toHaveLength(2);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("location", () => {
|
||||||
|
it("takes the suburb from the Wallet subtitle", async () => {
|
||||||
|
await ingest(req(breadworldWallet));
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(t.location).toBe("Wyndham Vale, VIC");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("falls back to the fixed-width descriptor, country included", async () => {
|
||||||
|
// The issuer half has no subtitle; the country code is positional.
|
||||||
|
await ingest(req({
|
||||||
|
...breadworldBank, event_id: 75, amount: 42.44,
|
||||||
|
merchant: "SP PEAKDESIGN",
|
||||||
|
// The model's own answer for this row, included because every real
|
||||||
|
// interpretation carries one -- omitting it is what let a wrong
|
||||||
|
// precedence order pass this test the first time, and ship
|
||||||
|
// "SAN FRANCISOUS" into the ledger.
|
||||||
|
location: "SAN FRANCISOUS",
|
||||||
|
notif_body: "$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS",
|
||||||
|
}));
|
||||||
|
const [t] = await txns();
|
||||||
|
expect(t.location).toBe("SAN FRANCISO US");
|
||||||
|
const [m] = await metas();
|
||||||
|
// The only foreign merchant in the whole capture window — worth surfacing.
|
||||||
|
expect(m.flags).toContain("foreign_country_US");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -3,17 +3,22 @@ import { readFileSync } from "fs";
|
|||||||
import { resolve } from "path";
|
import { resolve } from "path";
|
||||||
import {
|
import {
|
||||||
classifyInput,
|
classifyInput,
|
||||||
|
confabulationReason,
|
||||||
|
GROCERY_PATTERN,
|
||||||
merchantKey,
|
merchantKey,
|
||||||
merchantMatches,
|
merchantMatches,
|
||||||
platformForSource,
|
platformForSource,
|
||||||
|
PROMPT_PLACEHOLDER_MERCHANTS,
|
||||||
|
quoteShared,
|
||||||
reconcileItems,
|
reconcileItems,
|
||||||
resolveMerchantKind,
|
resolveMerchantKind,
|
||||||
|
textCouldBeReceipt,
|
||||||
toLineItems,
|
toLineItems,
|
||||||
validateHeader,
|
validateHeader,
|
||||||
type ExtractedOrder,
|
type ExtractedOrder,
|
||||||
} from "../../lib/intake-rules";
|
} from "../../lib/intake-rules";
|
||||||
import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT } from "../../lib/intake-extract";
|
import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT, ExtractionError, liteLLMExtractor, ORDER_ITEM_PROMPT } from "../../lib/intake-extract";
|
||||||
import { askCard, candidatesCard, enrichedCard } from "../../lib/intake-blocks";
|
import { askCard, candidatesCard, enrichedCard, type Block } from "../../lib/intake-blocks";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The intake router's pure rules, against the strings real receipts print.
|
* The intake router's pure rules, against the strings real receipts print.
|
||||||
@@ -85,6 +90,17 @@ describe("merchant pattern — grocery | food | ask", () => {
|
|||||||
expect(merchantMatches("Burgertory (Werribee)", "Coles")).toBe(false);
|
expect(merchantMatches("Burgertory (Werribee)", "Coles")).toBe(false);
|
||||||
expect(merchantMatches(null, "Coles")).toBe(false);
|
expect(merchantMatches(null, "Coles")).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("whole words, not substrings: `Bigals Pizza` contains `iga` and is neither a grocer nor an IGA", () => {
|
||||||
|
expect(GROCERY_PATTERN.test(merchantKey("Bigals Pizza"))).toBe(false);
|
||||||
|
expect(resolveMerchantKind("uber_eats", "Bigals Pizza")).toBe("food");
|
||||||
|
expect(merchantMatches("Bigals Pizza", "IGA")).toBe(false);
|
||||||
|
// and the grocers a boundary must not cost us
|
||||||
|
expect(resolveMerchantKind("uber_eats", "IGA Wyndham Vale")).toBe("grocery");
|
||||||
|
expect(resolveMerchantKind("uber_eats", "Woolworths Group Limited")).toBe("grocery");
|
||||||
|
expect(resolveMerchantKind("uber_eats", "Coles (Wyndham Vale)")).toBe("grocery");
|
||||||
|
expect(merchantMatches("IGA Wyndham Vale", "IGA")).toBe(true);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("validateHeader — nothing is written from a half-read upload", () => {
|
describe("validateHeader — nothing is written from a half-read upload", () => {
|
||||||
@@ -221,4 +237,183 @@ describe("cards carry the C5 action ids and JSON values", () => {
|
|||||||
expect(ids).toContain('"action_id":"intake_grocery"');
|
expect(ids).toContain('"action_id":"intake_grocery"');
|
||||||
expect(ids).toContain('"action_id":"intake_meal"');
|
expect(ids).toContain('"action_id":"intake_meal"');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("every value the card did not write is escaped, plain_text included", () => {
|
||||||
|
const hostile = { ...c, merchant: "Ben & Jerry's <fake>" };
|
||||||
|
const { text, blocks } = candidatesCard(hostile, [
|
||||||
|
{ transaction_id: 42, description: "SQ *BEN & JERRY <AU>", transaction_date: "2026-08-17", amount: "19.90" },
|
||||||
|
]);
|
||||||
|
const sections = JSON.stringify(blocks.filter((b) => (b as Block).type === "section"));
|
||||||
|
expect(sections).toContain("Ben & Jerry's <fake>");
|
||||||
|
expect(sections).toContain("SQ *BEN & JERRY <AU>");
|
||||||
|
expect(text).toContain("Ben & Jerry's <fake>");
|
||||||
|
// plain_text disables formatting, not entity decoding — &, < and > are
|
||||||
|
// control characters everywhere in a Slack payload.
|
||||||
|
const whole = JSON.stringify(blocks);
|
||||||
|
expect(whole).not.toContain("<fake>");
|
||||||
|
expect(whole).not.toContain("<AU>");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("the currency is model output too, so it is escaped like any other", () => {
|
||||||
|
// parseExtraction accepts any string for `currency`; unescaped, `<@U123>`
|
||||||
|
// would be a live mention on every card the batch produces.
|
||||||
|
const { text } = enrichedCard({ ...c, currency: "<@U123>", transactionId: 7 });
|
||||||
|
expect(text).toContain("<@U123> 22.40");
|
||||||
|
expect(text).not.toContain("<@U123>");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a foreign receipt is not printed as dollars — header and candidate rows alike", () => {
|
||||||
|
expect(enrichedCard({ ...c, currency: "USD", transactionId: 7 }).text).toBe(
|
||||||
|
"Added 2 items to your USD 22.40 Uber Eats (Burgertory (Werribee)) order ✓"
|
||||||
|
);
|
||||||
|
expect(enrichedCard({ ...c, currency: "AUD", transactionId: 7 }).text).toContain("$22.40");
|
||||||
|
expect(enrichedCard({ ...c, transactionId: 7 }).text).toContain("$22.40");
|
||||||
|
// A candidate row is a ledger transaction and carries its own currency; it
|
||||||
|
// used to print "$" beneath a header reading "USD 22.40".
|
||||||
|
const { blocks } = candidatesCard({ ...c, currency: "USD" }, [
|
||||||
|
{ transaction_id: 42, description: "Order - Burgertory", transaction_date: "2026-08-17", amount: "19.90", currency: "USD" },
|
||||||
|
]);
|
||||||
|
expect(JSON.stringify(blocks)).toContain("USD 19.90");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("liteLLMExtractor — a finish_reason that is present must be `stop`", () => {
|
||||||
|
const good = { merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4, subtotal: 28, currency: "AUD", items: [], adjustments: [] };
|
||||||
|
const extractor = (finish_reason?: string) =>
|
||||||
|
liteLLMExtractor({
|
||||||
|
baseUrl: "http://litellm.test",
|
||||||
|
key: "k",
|
||||||
|
fetchImpl: (async () =>
|
||||||
|
Response.json({
|
||||||
|
choices: [{ message: { content: JSON.stringify(good) }, ...(finish_reason ? { finish_reason } : {}) }],
|
||||||
|
})) as unknown as typeof fetch,
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts `stop`, and — on purpose — a reply that reports no reason at all", async () => {
|
||||||
|
expect((await extractor("stop")([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
|
||||||
|
expect((await extractor()([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects any other reason, naming it — the strict schema only held for a reply that finished", async () => {
|
||||||
|
// An absent reason is NOT rejected: see the comment in intake-extract.ts.
|
||||||
|
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(ExtractionError);
|
||||||
|
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/too long/);
|
||||||
|
await expect(extractor("content_filter")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/content_filter/);
|
||||||
|
await expect(extractor("tool_calls")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/tool_calls/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("textCouldBeReceipt — nothing that cannot be a receipt reaches the model", () => {
|
||||||
|
const why = (t: string | null | undefined) => {
|
||||||
|
const r = textCouldBeReceipt(t);
|
||||||
|
return r.ok ? null : r.reason;
|
||||||
|
};
|
||||||
|
|
||||||
|
it("the live defect: the phone sent the WORD `Image`, not the image", () => {
|
||||||
|
expect(textCouldBeReceipt("Image").ok).toBe(false);
|
||||||
|
expect(why("Image")).toMatch(/single word/);
|
||||||
|
expect(why("Image\n")).toMatch(/single word/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a filename is not a receipt, digits in it notwithstanding", () => {
|
||||||
|
expect(why("IMG_0042.jpeg")).toMatch(/single word/);
|
||||||
|
expect(why("Receipt_19Aug2026.pdf")).toMatch(/single word/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("empty, blank and whitespace-only", () => {
|
||||||
|
expect(why("")).toMatch(/empty/);
|
||||||
|
expect(why(" \n\t ")).toMatch(/empty/);
|
||||||
|
expect(why(null)).toMatch(/empty/);
|
||||||
|
expect(why(undefined)).toMatch(/empty/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("prose with no numbers cannot be a receipt", () => {
|
||||||
|
expect(why("Shared from Photos")).toMatch(/no numbers/);
|
||||||
|
expect(why("here is the receipt for dinner last night, sorry it took so long")).toMatch(/no numbers/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("short but with a printed amount goes through — this door accepts a pasted line", () => {
|
||||||
|
expect(textCouldBeReceipt("TOTAL $17.00").ok).toBe(true);
|
||||||
|
expect(textCouldBeReceipt("Total 17.00").ok).toBe(true);
|
||||||
|
expect(textCouldBeReceipt("Gesamt 12,50 EUR").ok).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("short with digits but no money figure does not — a screenshot name is not a purchase", () => {
|
||||||
|
expect(why("Screenshot 2026 08 22")).toMatch(/too short/);
|
||||||
|
expect(why("2 items")).toMatch(/too short/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a real receipt body passes", () => {
|
||||||
|
expect(
|
||||||
|
textCouldBeReceipt("Burgertory (Werribee)\n19 Aug 2026\n1x Michael 14.00\n1x Joel 14.00\nSubtotal 28.00\nTotal 22.40").ok
|
||||||
|
).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("quoteShared flattens and truncates what goes on the card", () => {
|
||||||
|
expect(quoteShared("Image")).toBe("Image");
|
||||||
|
expect(quoteShared(" a \n b ")).toBe("a b");
|
||||||
|
expect(quoteShared("x".repeat(200))).toHaveLength(80);
|
||||||
|
expect(quoteShared("x".repeat(200)).endsWith("…")).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("confabulationReason — the belt to that pre-check", () => {
|
||||||
|
const nothing = { textParts: ["Image"], hasImages: false };
|
||||||
|
const readable = { textParts: ["Total $22.40 Burgertory (Werribee) 2026-08-19"], hasImages: false };
|
||||||
|
const order = { merchant: "Burgertory (Braybrook)", items: [{ description: "The Pride" }] };
|
||||||
|
|
||||||
|
it("flags an order extracted from nothing readable", () => {
|
||||||
|
expect(confabulationReason(nothing, order)).toMatch(/nothing shared could be read/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not flag one extracted from something readable", () => {
|
||||||
|
expect(confabulationReason(readable, order)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not flag an image share — the pixels are what we asked for", () => {
|
||||||
|
expect(confabulationReason({ textParts: [], hasImages: true }, order)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("an empty answer to an unreadable share is honest, not a confabulation", () => {
|
||||||
|
expect(confabulationReason(nothing, { merchant: null, items: [] })).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("the prompt's own placeholder coming back as a merchant is always a confabulation", () => {
|
||||||
|
for (const p of PROMPT_PLACEHOLDER_MERCHANTS) {
|
||||||
|
expect(confabulationReason(readable, { merchant: p, items: [] })).toMatch(/prompt's own example/);
|
||||||
|
expect(confabulationReason({ textParts: [], hasImages: true }, { merchant: p, items: [] })).toMatch(/prompt's own example/);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a real merchant that happens to be named in the prompt's platform list is NOT flagged", () => {
|
||||||
|
expect(confabulationReason(readable, { merchant: "Coles (Wyndham Vale)", items: [] })).toBeNull();
|
||||||
|
expect(confabulationReason(readable, { merchant: "Woolworths", items: [] })).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("the prompt carries no real merchant to echo", () => {
|
||||||
|
it("names no shop it could hand back as data", () => {
|
||||||
|
expect(ORDER_ITEM_PROMPT).not.toMatch(/Burgertory/);
|
||||||
|
expect(ORDER_ITEM_PROMPT).not.toMatch(/Wyndham Vale/);
|
||||||
|
expect(ORDER_ITEM_PROMPT).toContain(PROMPT_PLACEHOLDER_MERCHANTS[0]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("tells the model what to return when the input is not a receipt", () => {
|
||||||
|
expect(ORDER_ITEM_PROMPT).toMatch(/NOT a receipt/);
|
||||||
|
expect(ORDER_ITEM_PROMPT).toMatch(/Do not construct a plausible order/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("needsHeaderCard", () => {
|
||||||
|
it("names the missing field and sends a dateless in-store screenshot to the app's Share button", async () => {
|
||||||
|
const { needsHeaderCard } = await import("@/lib/intake-blocks");
|
||||||
|
const dateless = needsHeaderCard(["date"], "woolworths").text;
|
||||||
|
expect(dateless).toContain("no date");
|
||||||
|
expect(dateless).toContain("Share button");
|
||||||
|
expect(dateless).not.toContain("total");
|
||||||
|
const delivery = needsHeaderCard(["total"], "uber_eats").text;
|
||||||
|
expect(delivery).toContain("no total");
|
||||||
|
expect(delivery).toContain("part of the screen");
|
||||||
|
expect(needsHeaderCard().text).toContain("total or date");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,109 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import {
|
||||||
|
parseWestpacDescriptor,
|
||||||
|
parseWalletSubtitle,
|
||||||
|
currencyFromAmountText,
|
||||||
|
} from "@/lib/wallet-descriptor";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every input below is a real captured notification body or subtitle from
|
||||||
|
* capture.wallet_events (2026-08-13 .. 08-22), not an invented fixture. The
|
||||||
|
* model handled this one fixed-width format three different ways on this exact
|
||||||
|
* data, which is why the parsing moved into code.
|
||||||
|
*/
|
||||||
|
|
||||||
|
describe("parseWestpacDescriptor", () => {
|
||||||
|
it("reads merchant, city and country from the fixed-width block", () => {
|
||||||
|
expect(parseWestpacDescriptor(
|
||||||
|
"$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
|
||||||
|
)).toEqual({ merchant: "Breadworld Bakery Cafe", city: "WYNDHAM VAL", country: "AU" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("handles a city padded to the full field width", () => {
|
||||||
|
expect(parseWestpacDescriptor(
|
||||||
|
"$54.50 paid from Mastercard ending...8032. PLA DONATION MELBOURNE AU"
|
||||||
|
)).toEqual({ merchant: "PLA DONATION", city: "MELBOURNE", country: "AU" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("splits a 12-char city that runs into the country code with no space", () => {
|
||||||
|
// The case that defeated the model: "SAN FRANCISO" exactly fills the city
|
||||||
|
// field, so "US" follows immediately. Read as one token it becomes the
|
||||||
|
// nonsense location "SAN FRANCISOUS" and the country is lost -- on the only
|
||||||
|
// foreign merchant in the whole capture window.
|
||||||
|
expect(parseWestpacDescriptor(
|
||||||
|
"$42.44 paid from Mastercard ending...8032. SP PEAKDESIGN SAN FRANCISOUS"
|
||||||
|
)).toEqual({ merchant: "SP PEAKDESIGN", city: "SAN FRANCISO", country: "US" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reads mixed-case cities the model inconsistently stripped", () => {
|
||||||
|
expect(parseWestpacDescriptor(
|
||||||
|
"$20.99 paid from Mastercard ending...8032. Spotify P45AE035CB Sydney AU"
|
||||||
|
)).toEqual({ merchant: "Spotify P45AE035CB", city: "Sydney", country: "AU" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("degrades to nulls rather than guessing", () => {
|
||||||
|
expect(parseWestpacDescriptor(null)).toEqual({ merchant: null, city: null, country: null });
|
||||||
|
expect(parseWestpacDescriptor("")).toEqual({ merchant: null, city: null, country: null });
|
||||||
|
// No sentence terminator: not this format at all.
|
||||||
|
expect(parseWestpacDescriptor("You have a new notification"))
|
||||||
|
.toEqual({ merchant: null, city: null, country: null });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not invent a country from a short merchant-only tail", () => {
|
||||||
|
// "...IT" here is the end of a word, not Italy. A wrong country on a
|
||||||
|
// foreign transaction is worse than no country.
|
||||||
|
const r = parseWestpacDescriptor("$9.00 paid from Mastercard ending...8032. SOME MERCHANT");
|
||||||
|
expect(r.country).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("parseWalletSubtitle", () => {
|
||||||
|
it("separates merchant from suburb and state", () => {
|
||||||
|
expect(parseWalletSubtitle("Breadworld Bakery Cafe, Wyndham Vale, VIC"))
|
||||||
|
.toEqual({ merchant: "Breadworld Bakery Cafe", location: "Wyndham Vale, VIC" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns no location for an online merchant", () => {
|
||||||
|
// 2 of the 10 Wallet subtitles captured have no place at all.
|
||||||
|
expect(parseWalletSubtitle("Seatsaero")).toEqual({ merchant: "Seatsaero", location: null });
|
||||||
|
expect(parseWalletSubtitle("Prism Accounting Bookkeeping Services"))
|
||||||
|
.toEqual({ merchant: "Prism Accounting Bookkeeping Services", location: null });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps a merchant name containing its own comma out of the location", () => {
|
||||||
|
expect(parseWalletSubtitle("Big Daddy`s Fresh Ma, Manor Lakes, VIC").location)
|
||||||
|
.toBe("Manor Lakes, VIC");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("handles absent input", () => {
|
||||||
|
expect(parseWalletSubtitle(null)).toEqual({ merchant: null, location: null });
|
||||||
|
expect(parseWalletSubtitle(" ")).toEqual({ merchant: null, location: null });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("currencyFromAmountText", () => {
|
||||||
|
it("reads Amex's explicit marker", () => {
|
||||||
|
expect(currencyFromAmountText("A$5,103.49")).toBe("AUD");
|
||||||
|
expect(currencyFromAmountText("A$13.50")).toBe("AUD");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses to guess a bare dollar sign", () => {
|
||||||
|
// 14 of 21 captured transaction rows are this. Assuming AUD is harmless at
|
||||||
|
// home and a 23x error on a THB tap abroad, so the row goes to the
|
||||||
|
// reconcile queue with no currency instead.
|
||||||
|
expect(currencyFromAmountText("$13.50")).toBeNull();
|
||||||
|
expect(currencyFromAmountText("$103.30")).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reads unambiguous symbols when a card does state one", () => {
|
||||||
|
expect(currencyFromAmountText("€24.00")).toBe("EUR");
|
||||||
|
expect(currencyFromAmountText("฿500")).toBe("THB");
|
||||||
|
expect(currencyFromAmountText("S$18.20")).toBe("SGD");
|
||||||
|
expect(currencyFromAmountText("NZ$40.00")).toBe("NZD");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("handles absent input", () => {
|
||||||
|
expect(currencyFromAmountText(null)).toBeNull();
|
||||||
|
expect(currencyFromAmountText("")).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { getCurrentUser } from "@/lib/auth";
|
||||||
|
import { getUpcomingAnnualCharges } from "@/lib/annual-charges";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Upcoming annual charges, from payments alone.
|
||||||
|
*
|
||||||
|
* Lives here rather than in home-dashboard so the detection sits beside the
|
||||||
|
* ledger it queries, is covered by this repo's integration suite, and can be
|
||||||
|
* reused by a future Slack nudge without moving. home-dashboard reads it over
|
||||||
|
* networks_internal the way its other widgets read their sources.
|
||||||
|
*
|
||||||
|
* Machine access uses the same shared secret as the dashboard's other reads;
|
||||||
|
* a browser session scopes to that user instead.
|
||||||
|
*/
|
||||||
|
export async function GET(req: NextRequest) {
|
||||||
|
// Its OWN token, with no fallback to WALLET_INGEST_TOKEN. home-dashboard's
|
||||||
|
// compose states the rule plainly -- "this app never holds write creds" -- and
|
||||||
|
// WALLET_INGEST_TOKEN unlocks POST /api/wallet/ingest and the estate-wide
|
||||||
|
// reconcile sweep. Handing that to a read-only surface to save minting a
|
||||||
|
// secret would quietly convert a dashboard into a writer. Unset means closed.
|
||||||
|
const token = process.env.DASH_READ_TOKEN;
|
||||||
|
const supplied = req.headers.get("x-ingest-token");
|
||||||
|
const machine = !!token && !!supplied && supplied === token;
|
||||||
|
|
||||||
|
let ownerId: number | null = null;
|
||||||
|
if (machine) {
|
||||||
|
// A machine caller names the owner explicitly rather than defaulting to 1:
|
||||||
|
// silently assuming an owner is how an ingested order ended up invisible in
|
||||||
|
// every scoped view while sitting in the table.
|
||||||
|
const q = req.nextUrl.searchParams.get("owner_id");
|
||||||
|
ownerId = q !== null && Number.isInteger(Number(q)) ? Number(q) : null;
|
||||||
|
if (ownerId === null) {
|
||||||
|
return NextResponse.json({ error: "owner_id is required for machine access" }, { status: 400 });
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const user = await getCurrentUser(req);
|
||||||
|
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||||
|
ownerId = user.id;
|
||||||
|
}
|
||||||
|
|
||||||
|
const charges = await getUpcomingAnnualCharges(ownerId);
|
||||||
|
return NextResponse.json({ charges, count: charges.length });
|
||||||
|
}
|
||||||
@@ -234,6 +234,16 @@ async function buildDetailsModal(transactionId: number, participantId: number) {
|
|||||||
`SELECT merchant_normalized AS merchant, line_items
|
`SELECT merchant_normalized AS merchant, line_items
|
||||||
FROM expense_metadata
|
FROM expense_metadata
|
||||||
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
||||||
|
-- Ordered the same way as replaceLineItems (line-items.ts) and
|
||||||
|
-- matchTransaction (intake-router.ts), so every reader of this row
|
||||||
|
-- resolves the same one a write lands on. NULLS LAST because DESC is
|
||||||
|
-- NULLS FIRST and the row that merely POINTS at the transaction has a
|
||||||
|
-- NULL transaction_id.
|
||||||
|
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
|
||||||
|
-- pointing-only, and ZERO transactions resolving to more than one row by
|
||||||
|
-- either link — so this changes nothing today. It is here so the readers
|
||||||
|
-- still agree on the day multiplicity appears.
|
||||||
|
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
|
||||||
LIMIT 1`,
|
LIMIT 1`,
|
||||||
[transactionId]
|
[transactionId]
|
||||||
);
|
);
|
||||||
@@ -377,12 +387,21 @@ async function nudgeState(transactionId: number) {
|
|||||||
currency: string | null;
|
currency: string | null;
|
||||||
amount: string;
|
amount: string;
|
||||||
}>(
|
}>(
|
||||||
`SELECT em.merchant_normalized AS merchant,
|
// LATERAL + the shared tie-break, not a bare join: `queryRow` returns
|
||||||
COALESCE(em.currency, 'AUD') AS currency,
|
// rows[0] and Postgres guarantees no ordering, so a second metadata row on
|
||||||
|
// one transaction would flip the merchant and currency on every card
|
||||||
|
// refresh at random. Every sibling reader already resolves this way.
|
||||||
|
`SELECT order_ctx.merchant_normalized AS merchant,
|
||||||
|
COALESCE(order_ctx.currency, 'AUD') AS currency,
|
||||||
t.amount
|
t.amount
|
||||||
FROM transactions t
|
FROM transactions t
|
||||||
LEFT JOIN expense_metadata em
|
LEFT JOIN LATERAL (
|
||||||
ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
SELECT em.merchant_normalized, em.currency
|
||||||
|
FROM expense_metadata em
|
||||||
|
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||||
|
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
|
||||||
|
LIMIT 1
|
||||||
|
) order_ctx ON true
|
||||||
WHERE t.id = $1`,
|
WHERE t.id = $1`,
|
||||||
[transactionId]
|
[transactionId]
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { getCurrentUser } from "@/lib/auth";
|
||||||
|
import { queryRaw } from "@/lib/db";
|
||||||
|
import { applyReconcileMatch, supersedeTransaction } from "@/lib/reconcile";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Manually resolve a duplicate pair the automation cannot see.
|
||||||
|
*
|
||||||
|
* WHY THIS EXISTS: until now there was no way to merge or remove a transaction
|
||||||
|
* at all. `superseded_by_id` was READ everywhere -- analytics excludes it,
|
||||||
|
* auto-reconcile excludes it -- but written nowhere in application code; it had
|
||||||
|
* only ever been set once, by migration 0023. The reconcile queue was the sole
|
||||||
|
* merge path, and it only offers candidates within a narrow amount band, so a
|
||||||
|
* pair outside it (SP PEAKDESIGN: notified $42.44, charged $43.71 -- the card's
|
||||||
|
* 3% foreign transaction fee) was unresolvable from the UI.
|
||||||
|
*
|
||||||
|
* There is deliberately NO delete. Every child of `transactions` is
|
||||||
|
* ON DELETE CASCADE, so removing a row destroys its splits, tags and metadata
|
||||||
|
* curation. Both modes here keep the row and its children and merely stop it
|
||||||
|
* counting:
|
||||||
|
*
|
||||||
|
* reconcile -- a statementLESS row folded into its statement twin
|
||||||
|
* (`reconciled_with_id`). The everyday case.
|
||||||
|
* supersede -- two rows on the same footing, e.g. a statement imported twice
|
||||||
|
* (`superseded_by_id`). No constraint on either side's origin.
|
||||||
|
*
|
||||||
|
* Both move curation onto the survivor FIRST, via one shared implementation in
|
||||||
|
* lib/reconcile.ts, because both hide the source row from every figure and
|
||||||
|
* splits stranded on a hidden row silently stop counting.
|
||||||
|
*/
|
||||||
|
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
|
||||||
|
const user = await getCurrentUser(req);
|
||||||
|
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||||
|
|
||||||
|
const { id } = await params;
|
||||||
|
const loserId = Number(id);
|
||||||
|
if (!Number.isInteger(loserId)) {
|
||||||
|
return NextResponse.json({ error: "invalid transaction id" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
let body: { other_id?: number; mode?: string };
|
||||||
|
try {
|
||||||
|
body = await req.json();
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
const otherId = Number(body.other_id);
|
||||||
|
const mode = body.mode ?? "reconcile";
|
||||||
|
if (!Number.isInteger(otherId)) {
|
||||||
|
return NextResponse.json({ error: "other_id is required" }, { status: 400 });
|
||||||
|
}
|
||||||
|
if (loserId === otherId) {
|
||||||
|
return NextResponse.json({ error: "a transaction cannot merge into itself" }, { status: 400 });
|
||||||
|
}
|
||||||
|
if (mode !== "reconcile" && mode !== "supersede") {
|
||||||
|
return NextResponse.json({ error: "mode must be 'reconcile' or 'supersede'" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Both rows must be the caller's. A statement row carries owner_id on the
|
||||||
|
// statement rather than the transaction, which is why this reads the same
|
||||||
|
// COALESCE every scoped query uses -- checking t.owner_id alone would reject
|
||||||
|
// every statement row as "not found".
|
||||||
|
const rows = await queryRaw<{
|
||||||
|
id: number;
|
||||||
|
statement_id: number | null;
|
||||||
|
reconciled_with_id: number | null;
|
||||||
|
superseded_by_id: number | null;
|
||||||
|
voided_at: Date | null;
|
||||||
|
}>(
|
||||||
|
`SELECT t.id, t.statement_id, t.reconciled_with_id, t.superseded_by_id, t.voided_at
|
||||||
|
FROM transactions t
|
||||||
|
LEFT JOIN statements s ON s.id = t.statement_id
|
||||||
|
WHERE t.id = ANY($1::int[]) AND COALESCE(t.owner_id, s.owner_id) = $2`,
|
||||||
|
[[loserId, otherId], user.id]
|
||||||
|
);
|
||||||
|
if (rows.length !== 2) {
|
||||||
|
return NextResponse.json({ error: "one or both transactions not found" }, { status: 404 });
|
||||||
|
}
|
||||||
|
const loser = rows.find((r) => r.id === loserId)!;
|
||||||
|
const other = rows.find((r) => r.id === otherId)!;
|
||||||
|
|
||||||
|
// Refuse a row that is already folded into something else, in either
|
||||||
|
// direction. Re-pointing it would strand the curation moved on the first
|
||||||
|
// pass -- and silently, since both columns hide the row from every figure.
|
||||||
|
for (const [label, row] of [["source", loser], ["target", other]] as const) {
|
||||||
|
if (row.reconciled_with_id !== null) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: `${label} transaction ${row.id} is already reconciled with ${row.reconciled_with_id}` },
|
||||||
|
{ status: 409 }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (row.superseded_by_id !== null) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: `${label} transaction ${row.id} is already superseded by ${row.superseded_by_id}` },
|
||||||
|
{ status: 409 }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// A voided row is a charge that never posted. As a source it is already
|
||||||
|
// out of every figure; as a survivor it would take the other row out too.
|
||||||
|
if (row.voided_at !== null) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: `${label} transaction ${row.id} is voided (a charge that never posted)` },
|
||||||
|
{ status: 409 }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (mode === "reconcile") {
|
||||||
|
// applyReconcileMatch's contract, enforced here so a misuse is a 400 and
|
||||||
|
// not a row quietly linked the wrong way round.
|
||||||
|
if (loser.statement_id !== null) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "reconcile expects the source to be a statementless row; use mode 'supersede'" },
|
||||||
|
{ status: 400 }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (other.statement_id === null) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: "reconcile expects the target to be a statement row; use mode 'supersede'" },
|
||||||
|
{ status: 400 }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
await applyReconcileMatch(loserId, otherId);
|
||||||
|
} else {
|
||||||
|
// Never hide a statement row behind a statementless one. The statement is
|
||||||
|
// the bank's own record; the statementless row is a notification, a
|
||||||
|
// receipt or a hand entry, and is the side that can be wrong -- a wallet
|
||||||
|
// row carries the amount BEFORE the card's foreign transaction fee, so
|
||||||
|
// burying the statement line under it would leave the ledger holding
|
||||||
|
// $42.44 for a purchase the bank charged $43.71 for, with the statement
|
||||||
|
// no longer reconciling against its own rows. The correct move is the
|
||||||
|
// other direction, which is exactly what `reconcile` does.
|
||||||
|
//
|
||||||
|
// Statement-under-statement stays allowed: that is the imported-twice
|
||||||
|
// pair migration 0023 existed for, and neither side is more true.
|
||||||
|
if (loser.statement_id !== null && other.statement_id === null) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{
|
||||||
|
error:
|
||||||
|
"Refusing to hide a statement row behind a statementless one — the statement is the bank's record. " +
|
||||||
|
`Open transaction ${otherId} instead and fold it into this one.`,
|
||||||
|
},
|
||||||
|
{ status: 400 }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
await supersedeTransaction(loserId, otherId);
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
return NextResponse.json({ ok: true, mode, source: loserId, target: otherId });
|
||||||
|
}
|
||||||
@@ -34,6 +34,14 @@ export async function GET(
|
|||||||
-- through matched_transaction_id. Both directions have to resolve or the
|
-- through matched_transaction_id. Both directions have to resolve or the
|
||||||
-- detail is missing on exactly the orders that were paid by card.
|
-- detail is missing on exactly the orders that were paid by card.
|
||||||
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
||||||
|
-- The same order as replaceLineItems in line-items.ts and matchTransaction
|
||||||
|
-- in intake-router.ts. Both directions can resolve at once — a statement
|
||||||
|
-- line owning its own row while a delivery receipt points at it — and an
|
||||||
|
-- unordered LIMIT 1 then reads a different row from the one an enrichment
|
||||||
|
-- wrote, so a PUT that reported success showed nothing on the screen.
|
||||||
|
-- NULLS LAST because DESC is NULLS FIRST and the pointing row's
|
||||||
|
-- transaction_id is NULL.
|
||||||
|
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
|
||||||
LIMIT 1`,
|
LIMIT 1`,
|
||||||
[Number(id)]
|
[Number(id)]
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { getCurrentUser } from "@/lib/auth";
|
||||||
|
import { queryRaw } from "@/lib/db";
|
||||||
|
import { unvoidTransaction, voidTransaction } from "@/lib/reconcile";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* POST /api/transactions/<id>/void { reason } -- a charge that never posted
|
||||||
|
* DELETE /api/transactions/<id>/void -- undo it
|
||||||
|
*
|
||||||
|
* For a statementless row whose charge never reached a statement: an
|
||||||
|
* authorisation hold the merchant released (txn 7951, a cancelled Airbnb
|
||||||
|
* booking). The row stays, with its children and audit trail, and drops out of
|
||||||
|
* every figure and the reconcile queue. Rules and refusals: `voidTransaction`
|
||||||
|
* in lib/reconcile.ts.
|
||||||
|
*/
|
||||||
|
|
||||||
|
async function ownedRow(req: NextRequest, idParam: string) {
|
||||||
|
const user = await getCurrentUser(req);
|
||||||
|
if (!user) return { error: NextResponse.json({ error: "Unauthorized" }, { status: 403 }) };
|
||||||
|
const id = Number(idParam);
|
||||||
|
if (!Number.isInteger(id)) {
|
||||||
|
return { error: NextResponse.json({ error: "invalid transaction id" }, { status: 400 }) };
|
||||||
|
}
|
||||||
|
// Same owner scope as every other route; a statement row carries owner_id on
|
||||||
|
// the statement. (Statement rows are refused later, but must 400, not 404.)
|
||||||
|
const rows = await queryRaw<{ id: number }>(
|
||||||
|
`SELECT t.id FROM transactions t
|
||||||
|
LEFT JOIN statements s ON s.id = t.statement_id
|
||||||
|
WHERE t.id = $1 AND COALESCE(t.owner_id, s.owner_id) = $2`,
|
||||||
|
[id, user.id]
|
||||||
|
);
|
||||||
|
if (rows.length === 0) {
|
||||||
|
return { error: NextResponse.json({ error: "transaction not found" }, { status: 404 }) };
|
||||||
|
}
|
||||||
|
return { id };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function POST(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
|
||||||
|
const owned = await ownedRow(req, (await params).id);
|
||||||
|
if (owned.error) return owned.error;
|
||||||
|
|
||||||
|
let body: { reason?: unknown };
|
||||||
|
try {
|
||||||
|
body = await req.json();
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
const reason = typeof body.reason === "string" ? body.reason.trim() : "";
|
||||||
|
if (!reason) return NextResponse.json({ error: "reason is required" }, { status: 400 });
|
||||||
|
|
||||||
|
try {
|
||||||
|
await voidTransaction(owned.id, reason);
|
||||||
|
} catch (e) {
|
||||||
|
return NextResponse.json({ error: e instanceof Error ? e.message : String(e) }, { status: 409 });
|
||||||
|
}
|
||||||
|
return NextResponse.json({ ok: true, id: owned.id, voided: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function DELETE(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
|
||||||
|
const owned = await ownedRow(req, (await params).id);
|
||||||
|
if (owned.error) return owned.error;
|
||||||
|
await unvoidTransaction(owned.id);
|
||||||
|
return NextResponse.json({ ok: true, id: owned.id, voided: false });
|
||||||
|
}
|
||||||
@@ -1,6 +1,8 @@
|
|||||||
import { NextRequest, NextResponse } from "next/server";
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
import { getCurrentUser } from "@/lib/auth";
|
import { getCurrentUser } from "@/lib/auth";
|
||||||
import { prisma, queryRaw } from "@/lib/db";
|
import { queryRaw } from "@/lib/db";
|
||||||
|
import { applyReconcileMatch } from "@/lib/reconcile";
|
||||||
|
import { findAutoMatches } from "@/lib/auto-reconcile";
|
||||||
|
|
||||||
export async function POST(req: NextRequest) {
|
export async function POST(req: NextRequest) {
|
||||||
const user = await getCurrentUser(req);
|
const user = await getCurrentUser(req);
|
||||||
@@ -17,7 +19,7 @@ export async function POST(req: NextRequest) {
|
|||||||
// Verify all manual_ids belong to this user
|
// Verify all manual_ids belong to this user
|
||||||
const manualIds = body.matches.map((m) => m.manual_id);
|
const manualIds = body.matches.map((m) => m.manual_id);
|
||||||
const owned = await queryRaw<{ id: number }>(
|
const owned = await queryRaw<{ id: number }>(
|
||||||
`SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND owner_id = $2`,
|
`SELECT id FROM transactions WHERE id = ANY($1::int[]) AND statement_id IS NULL AND voided_at IS NULL AND owner_id = $2`,
|
||||||
[manualIds, user.id]
|
[manualIds, user.id]
|
||||||
);
|
);
|
||||||
if (owned.length !== manualIds.length) {
|
if (owned.length !== manualIds.length) {
|
||||||
@@ -25,97 +27,71 @@ export async function POST(req: NextRequest) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let reconciled = 0;
|
let reconciled = 0;
|
||||||
|
|
||||||
for (const { manual_id, statement_tx_id } of body.matches) {
|
for (const { manual_id, statement_tx_id } of body.matches) {
|
||||||
await prisma.$transaction(async (tx) => {
|
await applyReconcileMatch(manual_id, statement_tx_id);
|
||||||
// Copy overrides: manual → statement tx
|
|
||||||
const override = await tx.transaction_overrides.findUnique({
|
|
||||||
where: { transaction_id: manual_id },
|
|
||||||
});
|
|
||||||
if (override) {
|
|
||||||
await tx.transaction_overrides.upsert({
|
|
||||||
where: { transaction_id: statement_tx_id },
|
|
||||||
update: {
|
|
||||||
category_override: override.category_override,
|
|
||||||
merchant_normalized: override.merchant_normalized,
|
|
||||||
notes: override.notes,
|
|
||||||
my_share_percent: override.my_share_percent,
|
|
||||||
updated_at: new Date(),
|
|
||||||
},
|
|
||||||
create: {
|
|
||||||
transaction_id: statement_tx_id,
|
|
||||||
category_override: override.category_override,
|
|
||||||
merchant_normalized: override.merchant_normalized,
|
|
||||||
notes: override.notes,
|
|
||||||
my_share_percent: override.my_share_percent,
|
|
||||||
},
|
|
||||||
});
|
|
||||||
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Move tags: manual → statement tx
|
|
||||||
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
|
|
||||||
if (tags.length) {
|
|
||||||
await tx.transaction_tags.createMany({
|
|
||||||
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
|
|
||||||
skipDuplicates: true,
|
|
||||||
});
|
|
||||||
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Move splits: manual → statement tx
|
|
||||||
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
|
|
||||||
if (splits.length) {
|
|
||||||
await tx.transaction_splits.createMany({
|
|
||||||
data: splits.map((s) => ({
|
|
||||||
transaction_id: statement_tx_id,
|
|
||||||
participant_id: s.participant_id,
|
|
||||||
share_percent: s.share_percent,
|
|
||||||
})),
|
|
||||||
skipDuplicates: true,
|
|
||||||
});
|
|
||||||
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Move provenance: manual → statement tx.
|
|
||||||
//
|
|
||||||
// Overrides, tags and splits above were always carried across; expense_metadata was
|
|
||||||
// the one child left behind, which did not matter while every metadata row came from
|
|
||||||
// an email that had created its own transaction. It matters now: a scanned grocery
|
|
||||||
// receipt puts its line items here, and reconciliation hides the manual row from
|
|
||||||
// every figure — so without this the shop's contents disappear at exactly the moment
|
|
||||||
// the statement line appears, and `COLES 0556 MANOR LAKES` stays as unreadable as it
|
|
||||||
// was before the receipt was ever scanned.
|
|
||||||
//
|
|
||||||
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
|
|
||||||
// or Paperless copy got there first) keeps it. The pantry row stays attached to the
|
|
||||||
// reconciled manual transaction and is flagged, rather than raising a constraint
|
|
||||||
// violation or silently overwriting the other source.
|
|
||||||
const moved = await tx.$executeRawUnsafe(
|
|
||||||
`UPDATE expense_metadata SET transaction_id = $1
|
|
||||||
WHERE transaction_id = $2
|
|
||||||
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
|
|
||||||
statement_tx_id,
|
|
||||||
manual_id
|
|
||||||
);
|
|
||||||
if (moved === 0) {
|
|
||||||
await tx.$executeRawUnsafe(
|
|
||||||
`UPDATE expense_metadata
|
|
||||||
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
|
|
||||||
WHERE transaction_id = $1`,
|
|
||||||
manual_id
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Mark manual tx as reconciled (link to statement tx)
|
|
||||||
await tx.$executeRawUnsafe(
|
|
||||||
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
|
|
||||||
statement_tx_id,
|
|
||||||
manual_id
|
|
||||||
);
|
|
||||||
});
|
|
||||||
reconciled++;
|
reconciled++;
|
||||||
}
|
}
|
||||||
|
|
||||||
return NextResponse.json({ reconciled });
|
return NextResponse.json({ reconciled });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Automatic pass over the reconcile queue.
|
||||||
|
*
|
||||||
|
* Applies only pairs that are unambiguous by the strict rule in
|
||||||
|
* `findAutoMatches` -- exact amount (or exact native pair for a foreign row),
|
||||||
|
* within a few days, and unique in BOTH directions. Everything else is left in
|
||||||
|
* the queue for a person, which is the point: this exists to keep the queue
|
||||||
|
* workable as the wallet lane adds ~63 rows a month, not to clear it.
|
||||||
|
*
|
||||||
|
* Validated against the 48 reconciliations a human had already made: the rule
|
||||||
|
* reproduces all 48 and proposes nothing they did not.
|
||||||
|
*
|
||||||
|
* `?dry=1` returns what it WOULD do without writing, which is how this should
|
||||||
|
* be run the first time after any change to the matching rule.
|
||||||
|
*/
|
||||||
|
export async function PATCH(req: NextRequest) {
|
||||||
|
// Two callers, two auth modes. A person in the browser reconciles their own
|
||||||
|
// queue; the scheduled sweep has no session and carries the machine token,
|
||||||
|
// and must cover EVERY owner with a queue -- 169 of the 177 rows outstanding
|
||||||
|
// belong to the second participant, so a sweep scoped to one user would leave
|
||||||
|
// almost all of it untouched.
|
||||||
|
const machineToken = req.headers.get("x-ingest-token");
|
||||||
|
const machineOk = !!machineToken && machineToken === process.env.WALLET_INGEST_TOKEN;
|
||||||
|
|
||||||
|
let ownerIds: number[];
|
||||||
|
if (machineOk) {
|
||||||
|
ownerIds = (await queryRaw<{ owner_id: number }>(
|
||||||
|
`SELECT DISTINCT owner_id FROM transactions
|
||||||
|
WHERE statement_id IS NULL AND owner_id IS NOT NULL
|
||||||
|
AND reconciled_with_id IS NULL AND superseded_by_id IS NULL`
|
||||||
|
)).map((r) => r.owner_id);
|
||||||
|
} else {
|
||||||
|
const user = await getCurrentUser(req);
|
||||||
|
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||||
|
ownerIds = [user.id];
|
||||||
|
}
|
||||||
|
|
||||||
|
const dry = new URL(req.url).searchParams.get("dry") === "1";
|
||||||
|
const matches = (await Promise.all(ownerIds.map((id) => findAutoMatches(id)))).flat();
|
||||||
|
|
||||||
|
if (dry) {
|
||||||
|
return NextResponse.json({ dry_run: true, would_reconcile: matches.length, matches });
|
||||||
|
}
|
||||||
|
|
||||||
|
let reconciled = 0;
|
||||||
|
const failed: { manual_id: number; error: string }[] = [];
|
||||||
|
for (const m of matches) {
|
||||||
|
try {
|
||||||
|
await applyReconcileMatch(m.manual_id, m.statement_tx_id);
|
||||||
|
reconciled++;
|
||||||
|
} catch (e) {
|
||||||
|
// One bad pair must not abandon the rest -- each merge is its own
|
||||||
|
// transaction, so a failure here leaves that pair untouched and in the
|
||||||
|
// queue rather than half-applied.
|
||||||
|
failed.push({ manual_id: m.manual_id, error: e instanceof Error ? e.message : String(e) });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return NextResponse.json({ reconciled, examined: matches.length, failed });
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { processWalletIngestion, type WalletEvent } from "@/lib/wallet-ingestion";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Machine ingest endpoint for interpreted iOS Wallet notifications.
|
||||||
|
*
|
||||||
|
* Sibling to /api/orders/ingest and /api/receipts/ingest, and deliberately
|
||||||
|
* shaped like them: a shared secret rather than a browser session, because n8n
|
||||||
|
* calls this app-to-app.
|
||||||
|
*
|
||||||
|
* Its own token so the wallet lane's credential can be rotated without touching
|
||||||
|
* the order flow, which runs on a schedule nobody is watching.
|
||||||
|
*
|
||||||
|
* n8n is transport only. The interpretation already happened (Wallet Interpret,
|
||||||
|
* CAp1s5cuzJk2mDaI, local model on Unraid because card tails are PII); this
|
||||||
|
* endpoint owns every decision about what reaches the ledger.
|
||||||
|
*/
|
||||||
|
function authorised(req: NextRequest): boolean {
|
||||||
|
const expected = process.env.WALLET_INGEST_TOKEN;
|
||||||
|
if (!expected) return false; // fail closed when unconfigured
|
||||||
|
const got = req.headers.get("x-ingest-token");
|
||||||
|
return !!got && got === expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function POST(req: NextRequest) {
|
||||||
|
if (!authorised(req)) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
|
||||||
|
|
||||||
|
let body: WalletEvent;
|
||||||
|
try {
|
||||||
|
body = await req.json();
|
||||||
|
} catch {
|
||||||
|
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!Number.isInteger(body?.event_id) || typeof body?.kind !== "string") {
|
||||||
|
return NextResponse.json({ error: "event_id (int) and kind are required" }, { status: 400 });
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await processWalletIngestion(body);
|
||||||
|
return NextResponse.json(result);
|
||||||
|
} catch (e) {
|
||||||
|
// A 500 here alerts #log-alerts through the workflow error handler, which
|
||||||
|
// is what should happen: a write that fails is a template or schema change,
|
||||||
|
// not something to swallow.
|
||||||
|
const message = e instanceof Error ? e.message : String(e);
|
||||||
|
return NextResponse.json({ error: message }, { status: 500 });
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
"use client";
|
"use client";
|
||||||
|
|
||||||
import { useState } from "react";
|
import { useState } from "react";
|
||||||
import { usePendingReconciliations, useReconcile } from "@/lib/hooks";
|
import { usePendingReconciliations, useReconcile, useVoidTransaction } from "@/lib/hooks";
|
||||||
import type { ManualTxWithMatches, PotentialMatch } from "@/lib/queries";
|
import type { ManualTxWithMatches, PotentialMatch } from "@/lib/queries";
|
||||||
|
|
||||||
function formatDate(d: string | Date) {
|
function formatDate(d: string | Date) {
|
||||||
@@ -156,6 +156,7 @@ export default function ReconcilePage() {
|
|||||||
<span key={tag.id} className="w-2 h-2 rounded-full" style={{ backgroundColor: tag.color }} title={tag.name} />
|
<span key={tag.id} className="w-2 h-2 rounded-full" style={{ backgroundColor: tag.color }} title={tag.name} />
|
||||||
))}
|
))}
|
||||||
</div>
|
</div>
|
||||||
|
<VoidButton id={tx.id} amount={tx.amount} onError={setError} />
|
||||||
</div>
|
</div>
|
||||||
))}
|
))}
|
||||||
</div>
|
</div>
|
||||||
@@ -243,6 +244,7 @@ function ReconcileRow({
|
|||||||
Clear
|
Clear
|
||||||
</button>
|
</button>
|
||||||
)}
|
)}
|
||||||
|
<VoidButton id={tx.id} amount={tx.amount} />
|
||||||
{!isSkipped && (
|
{!isSkipped && (
|
||||||
<button onClick={onSkip} className="text-xs text-zinc-500 hover:text-zinc-300 px-2 py-1 rounded hover:bg-zinc-800">
|
<button onClick={onSkip} className="text-xs text-zinc-500 hover:text-zinc-300 px-2 py-1 rounded hover:bg-zinc-800">
|
||||||
Skip (no match)
|
Skip (no match)
|
||||||
@@ -296,3 +298,34 @@ function MatchRow({
|
|||||||
</button>
|
</button>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* "Never posted": a released authorisation hold, e.g. a booking cancelled
|
||||||
|
* before the charge settled. Drops the row out of every figure and this queue
|
||||||
|
* (migration 0040); the row and its history are kept.
|
||||||
|
*/
|
||||||
|
function VoidButton({ id, amount, onError }: { id: number; amount: number; onError?: (msg: string) => void }) {
|
||||||
|
const voidTx = useVoidTransaction();
|
||||||
|
async function handle() {
|
||||||
|
const reason = window.prompt(
|
||||||
|
`Mark this $${Number(amount).toFixed(2)} charge as never posted? It will stop counting everywhere.\n\nReason (e.g. "hold released, booking cancelled"):`
|
||||||
|
);
|
||||||
|
if (!reason?.trim()) return;
|
||||||
|
try {
|
||||||
|
await voidTx.mutateAsync({ id, reason });
|
||||||
|
} catch (e) {
|
||||||
|
const msg = e instanceof Error ? e.message : "Void failed";
|
||||||
|
if (onError) onError(msg); else window.alert(msg);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<button
|
||||||
|
onClick={handle}
|
||||||
|
disabled={voidTx.isPending}
|
||||||
|
title="The charge never posted (e.g. a released hold) — stop counting it"
|
||||||
|
className="text-xs text-zinc-500 hover:text-red-300 px-2 py-1 rounded hover:bg-zinc-800 disabled:opacity-50 flex-shrink-0"
|
||||||
|
>
|
||||||
|
{voidTx.isPending ? "Voiding…" : "Never posted"}
|
||||||
|
</button>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|||||||
@@ -463,7 +463,7 @@ export default function RulesPage() {
|
|||||||
>
|
>
|
||||||
{expandedRun === run.id ? "▾" : "▸"}
|
{expandedRun === run.id ? "▾" : "▸"}
|
||||||
</button>
|
</button>
|
||||||
<span className="text-zinc-300">{new Date(run.applied_at).toLocaleString()}</span>
|
<span className="text-zinc-300">{new Date(run.applied_at).toLocaleString("en-AU")}</span>
|
||||||
{/* Which rule ran matters most: a merchant rename and a
|
{/* Which rule ran matters most: a merchant rename and a
|
||||||
50/50 split of everything look identical as counts. */}
|
50/50 split of everything look identical as counts. */}
|
||||||
<span className="text-zinc-200">
|
<span className="text-zinc-200">
|
||||||
@@ -480,7 +480,7 @@ export default function RulesPage() {
|
|||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
{run.reverted_at ? (
|
{run.reverted_at ? (
|
||||||
<span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString()}</span>
|
<span className="text-xs text-zinc-500">reverted {new Date(run.reverted_at).toLocaleString("en-AU")}</span>
|
||||||
) : (
|
) : (
|
||||||
<button
|
<button
|
||||||
onClick={() => {
|
onClick={() => {
|
||||||
|
|||||||
@@ -1085,7 +1085,18 @@ function TransactionsContent() {
|
|||||||
) : (
|
) : (
|
||||||
<>
|
<>
|
||||||
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
|
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
|
||||||
{t.currency && t.currency !== "AUD" && (
|
{/*
|
||||||
|
The native figure: `foreign_currency_amount` where one
|
||||||
|
was recorded (a statement or wallet row, whose `amount`
|
||||||
|
is AUD), else `amount` on a non-AUD statement. Printing
|
||||||
|
`amount` under the foreign code put the AUD figure
|
||||||
|
behind an INR label on every wallet row.
|
||||||
|
*/}
|
||||||
|
{t.foreign_currency_code && t.foreign_currency_code !== "AUD" && t.foreign_currency_amount !== null ? (
|
||||||
|
<div className="text-[10px] text-zinc-500 mt-0.5">
|
||||||
|
{formatAmount(t.foreign_currency_amount, t.transaction_type, t.foreign_currency_code)}
|
||||||
|
</div>
|
||||||
|
) : t.currency && t.currency !== "AUD" && (
|
||||||
<div className="text-[10px] text-zinc-500 mt-0.5">
|
<div className="text-[10px] text-zinc-500 mt-0.5">
|
||||||
{formatAmount(t.amount, t.transaction_type, t.currency)}
|
{formatAmount(t.amount, t.transaction_type, t.currency)}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
"use client";
|
"use client";
|
||||||
|
|
||||||
import { useState } from "react";
|
import { useEffect, useState } from "react";
|
||||||
import {
|
import {
|
||||||
useUpdateTransaction,
|
useUpdateTransaction,
|
||||||
|
useMergeTransaction,
|
||||||
useTags,
|
useTags,
|
||||||
useAddTransactionTag,
|
useAddTransactionTag,
|
||||||
useRemoveTransactionTag,
|
useRemoveTransactionTag,
|
||||||
@@ -93,6 +94,210 @@ function InlineTags({ transactionId, initialTags }: { transactionId: number; ini
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Determines which merge mode the backend will accept for (loser, candidate).
|
||||||
|
* Mirrors the constraint enforced server-side in the merge route: reconcile
|
||||||
|
* requires a statementless loser folding into a statement survivor; every
|
||||||
|
* other pairing is a supersede.
|
||||||
|
*/
|
||||||
|
function mergeModeFor(loser: TransactionRow, candidate: TransactionRow): "reconcile" | "supersede" {
|
||||||
|
return loser.statement_id === null && candidate.statement_id !== null ? "reconcile" : "supersede";
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Collapsed by default, at the bottom, visually separate — this is the one
|
||||||
|
* control on the modal that changes which rows this transaction can ever be
|
||||||
|
* again. `transaction` is always the row that will be hidden (the loser);
|
||||||
|
* the row picked here is the survivor (`other_id`).
|
||||||
|
*/
|
||||||
|
function DuplicateSection({ transaction, onMerged }: { transaction: TransactionRow; onMerged: () => void }) {
|
||||||
|
const [expanded, setExpanded] = useState(false);
|
||||||
|
const [query, setQuery] = useState("");
|
||||||
|
const [results, setResults] = useState<TransactionRow[]>([]);
|
||||||
|
const [searching, setSearching] = useState(false);
|
||||||
|
const [selected, setSelected] = useState<TransactionRow | null>(null);
|
||||||
|
const mergeTxn = useMergeTransaction();
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!expanded || selected) return;
|
||||||
|
const q = query.trim();
|
||||||
|
// No synchronous setState here: an empty box is handled at render time, and
|
||||||
|
// `searching` is raised inside the timeout. Clearing state here instead
|
||||||
|
// cascades a render on every keystroke (react-hooks lint catches it).
|
||||||
|
if (q.length === 0) return;
|
||||||
|
const handle = setTimeout(() => {
|
||||||
|
setSearching(true);
|
||||||
|
fetch(`/api/transactions?search=${encodeURIComponent(q)}&limit=8`)
|
||||||
|
.then((res) => res.json())
|
||||||
|
.then((body: { data?: TransactionRow[] }) => {
|
||||||
|
setResults((body.data ?? []).filter((r) => r.id !== transaction.id));
|
||||||
|
})
|
||||||
|
.catch(() => setResults([]))
|
||||||
|
.finally(() => setSearching(false));
|
||||||
|
}, 300);
|
||||||
|
return () => clearTimeout(handle);
|
||||||
|
}, [query, expanded, selected, transaction.id]);
|
||||||
|
|
||||||
|
function reset() {
|
||||||
|
setExpanded(false);
|
||||||
|
setQuery("");
|
||||||
|
setResults([]);
|
||||||
|
setSelected(null);
|
||||||
|
mergeTxn.reset();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleConfirm() {
|
||||||
|
if (!selected) return;
|
||||||
|
try {
|
||||||
|
await mergeTxn.mutateAsync({
|
||||||
|
id: transaction.id,
|
||||||
|
other_id: selected.id,
|
||||||
|
mode: mergeModeFor(transaction, selected),
|
||||||
|
});
|
||||||
|
onMerged();
|
||||||
|
} catch {
|
||||||
|
// Surfaced below via mergeTxn.error — section stays open so they can retry.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const mode = selected ? mergeModeFor(transaction, selected) : null;
|
||||||
|
// Hiding a statement row behind a statementless one is always backwards — the
|
||||||
|
// bank's record is the side that survives. The API refuses it too; catching it
|
||||||
|
// here just means the button explains itself instead of returning a 400.
|
||||||
|
const wrongDirection =
|
||||||
|
selected !== null && transaction.statement_id !== null && selected.statement_id === null;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="pt-4 border-t border-zinc-800">
|
||||||
|
{!expanded ? (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => setExpanded(true)}
|
||||||
|
className="text-xs text-zinc-500 hover:text-zinc-300"
|
||||||
|
>
|
||||||
|
Mark as duplicate…
|
||||||
|
</button>
|
||||||
|
) : (
|
||||||
|
<div className="space-y-2.5">
|
||||||
|
<div className="flex items-center justify-between">
|
||||||
|
<p className="font-semibold text-xs text-zinc-400">Mark as duplicate</p>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={reset}
|
||||||
|
className="text-xs text-zinc-500 hover:text-zinc-300"
|
||||||
|
>
|
||||||
|
Cancel
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{!selected ? (
|
||||||
|
<>
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
value={query}
|
||||||
|
onChange={(e) => setQuery(e.target.value)}
|
||||||
|
placeholder="Search for the surviving transaction…"
|
||||||
|
autoFocus
|
||||||
|
className="w-full bg-zinc-800 border border-zinc-700 rounded px-2 py-1.5 text-sm"
|
||||||
|
/>
|
||||||
|
{searching && <p className="text-xs text-zinc-500">Searching…</p>}
|
||||||
|
{!searching && query.trim().length > 0 && results.length === 0 && (
|
||||||
|
<p className="text-xs text-zinc-600 italic">No matches</p>
|
||||||
|
)}
|
||||||
|
{/* Gated on the query too: results are no longer cleared when the
|
||||||
|
box empties, so without this the last search's rows linger. */}
|
||||||
|
{query.trim().length > 0 && results.length > 0 && (
|
||||||
|
<div className="space-y-1 max-h-48 overflow-y-auto">
|
||||||
|
{results.map((r) => (
|
||||||
|
<button
|
||||||
|
key={r.id}
|
||||||
|
type="button"
|
||||||
|
onClick={() => setSelected(r)}
|
||||||
|
className="w-full text-left px-2 py-1.5 rounded bg-zinc-800/50 hover:bg-zinc-800 border border-zinc-700/50"
|
||||||
|
>
|
||||||
|
<div className="flex items-center justify-between gap-2">
|
||||||
|
<span className="text-sm text-zinc-200 truncate">{r.description}</span>
|
||||||
|
<span
|
||||||
|
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(r.transaction_type) ? "text-red-400" : "text-green-400"}`}
|
||||||
|
>
|
||||||
|
{formatAmount(r.amount, r.transaction_type)}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div className="flex items-center justify-between mt-0.5">
|
||||||
|
<span className="text-xs text-zinc-500">
|
||||||
|
{new Date(r.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
|
||||||
|
</span>
|
||||||
|
<span className="text-xs text-zinc-500">
|
||||||
|
{r.statement_id !== null ? "Statement row" : "Statementless"}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<div className="space-y-2">
|
||||||
|
<div className="px-2 py-1.5 rounded bg-zinc-800/50 border border-zinc-700/50">
|
||||||
|
<div className="flex items-center justify-between gap-2">
|
||||||
|
<span className="text-sm text-zinc-200 truncate">{selected.description}</span>
|
||||||
|
<span
|
||||||
|
className={`text-sm font-mono shrink-0 ${SPEND_TYPES.has(selected.transaction_type) ? "text-red-400" : "text-green-400"}`}
|
||||||
|
>
|
||||||
|
{formatAmount(selected.amount, selected.transaction_type)}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div className="flex items-center justify-between mt-0.5">
|
||||||
|
<span className="text-xs text-zinc-500">
|
||||||
|
{new Date(selected.transaction_date).toLocaleDateString("en-AU", { day: "numeric", month: "short", year: "numeric" })}
|
||||||
|
</span>
|
||||||
|
<span className="text-xs text-zinc-500">
|
||||||
|
{selected.statement_id !== null ? "Statement row" : "Statementless"}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{wrongDirection ? (
|
||||||
|
<p className="text-xs text-red-400">
|
||||||
|
This is the wrong way round — the statement is the bank's record, so it
|
||||||
|
has to be the row that survives. Close this and open{" "}
|
||||||
|
<span className="text-zinc-300">{selected.description}</span> instead, then
|
||||||
|
fold that one into this.
|
||||||
|
</p>
|
||||||
|
) : (
|
||||||
|
<p className="text-xs text-amber-400/90">
|
||||||
|
{mode === "reconcile"
|
||||||
|
? "This row will be folded into the statement row above."
|
||||||
|
: "This row will be marked a duplicate of the row above."}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
{mergeTxn.error && (
|
||||||
|
<p className="text-red-400 text-xs">{(mergeTxn.error as Error).message}</p>
|
||||||
|
)}
|
||||||
|
<div className="flex gap-2">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => setSelected(null)}
|
||||||
|
className="px-3 py-1.5 bg-zinc-800 hover:bg-zinc-700 text-zinc-300 rounded-lg text-xs"
|
||||||
|
>
|
||||||
|
Choose a different row
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={handleConfirm}
|
||||||
|
disabled={mergeTxn.isPending || wrongDirection}
|
||||||
|
className="px-3 py-1.5 bg-amber-700 hover:bg-amber-600 disabled:opacity-50 text-white rounded-lg text-xs"
|
||||||
|
>
|
||||||
|
{mergeTxn.isPending ? "Merging…" : mode === "reconcile" ? "Fold into statement row" : "Mark as duplicate"}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
export function EditTransactionModal({
|
export function EditTransactionModal({
|
||||||
transaction,
|
transaction,
|
||||||
onClose,
|
onClose,
|
||||||
@@ -416,6 +621,8 @@ export function EditTransactionModal({
|
|||||||
|
|
||||||
<OrderDetails transactionId={transaction.id} currency={transaction.currency ?? null} />
|
<OrderDetails transactionId={transaction.id} currency={transaction.currency ?? null} />
|
||||||
|
|
||||||
|
<DuplicateSection transaction={transaction} onMerged={onClose} />
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Footer */}
|
{/* Footer */}
|
||||||
|
|||||||
@@ -13,12 +13,9 @@ import {
|
|||||||
type OrderReceipt,
|
type OrderReceipt,
|
||||||
type OrderRating,
|
type OrderRating,
|
||||||
} from "@/lib/hooks";
|
} from "@/lib/hooks";
|
||||||
|
// One platform vocabulary, shared with the intake cards — three copies had
|
||||||
const PLATFORM_LABEL: Record<string, string> = {
|
// already drifted apart (the intake one was missing `uber`).
|
||||||
doordash: "DoorDash",
|
import { platformLabel } from "@/lib/intake-rules";
|
||||||
ubereats: "Uber Eats",
|
|
||||||
uber: "Uber",
|
|
||||||
};
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Who records verdicts. A two-person household with one primary user: the
|
* Who records verdicts. A two-person household with one primary user: the
|
||||||
@@ -99,7 +96,7 @@ export function OrderDetails({
|
|||||||
<p className="text-xs text-zinc-500">
|
<p className="text-xs text-zinc-500">
|
||||||
Order details
|
Order details
|
||||||
{receipt.platform && (
|
{receipt.platform && (
|
||||||
<span className="ml-1.5 text-zinc-400">{PLATFORM_LABEL[receipt.platform] ?? receipt.platform}</span>
|
<span className="ml-1.5 text-zinc-400">{platformLabel(receipt.platform)}</span>
|
||||||
)}
|
)}
|
||||||
</p>
|
</p>
|
||||||
{receipt.card_last4 && (
|
{receipt.card_last4 && (
|
||||||
|
|||||||
@@ -71,9 +71,17 @@ export const STATEMENTS_JOIN = `LEFT JOIN statements s ON s.id = t.statement_id`
|
|||||||
* without being edited. Anything summing transactions without this fragment
|
* without being edited. Anything summing transactions without this fragment
|
||||||
* still double-counts — that is the same gap that let reconciled rows into the
|
* still double-counts — that is the same gap that let reconciled rows into the
|
||||||
* analytics routes in the first place.
|
* analytics routes in the first place.
|
||||||
|
*
|
||||||
|
* **3. The charge that never posted** (`voided_at`, migration 0040).
|
||||||
|
*
|
||||||
|
* A card notification is sent for an authorisation, and a released hold never
|
||||||
|
* reaches a statement (txn 7951: a $1,040.41 Airbnb hold on a booking cancelled
|
||||||
|
* the next day). There is no survivor to point at, so it gets its own column —
|
||||||
|
* and it lives in this fragment for the same reason case 2 does.
|
||||||
*/
|
*/
|
||||||
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
|
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
|
||||||
AND t.superseded_by_id IS NULL`;
|
AND t.superseded_by_id IS NULL
|
||||||
|
AND t.voided_at IS NULL`;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The currency `t.amount` is actually denominated in.
|
* The currency `t.amount` is actually denominated in.
|
||||||
@@ -89,10 +97,17 @@ export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reco
|
|||||||
* figure and `foreign_currency_code` names it (order-ingestion.ts leaves
|
* figure and `foreign_currency_code` names it (order-ingestion.ts leaves
|
||||||
* `amount_aud` NULL rather than asserting an FX rate it does not have).
|
* `amount_aud` NULL rather than asserting an FX rate it does not have).
|
||||||
*
|
*
|
||||||
|
* - A wallet row also has no statement, but follows the STATEMENT convention:
|
||||||
|
* `amount` = `amount_aud` = a provisional AUD figure, and the native figure
|
||||||
|
* lives in `foreign_currency_amount` (wallet-ingestion.ts). `amount_aud`
|
||||||
|
* being present is what says `amount` is already AUD. Without that arm, an
|
||||||
|
* INR 524 tap stored as A$9.42 rendered as "INR 9.42" -- the AUD number
|
||||||
|
* under the foreign label (txn 8162, 2026-10-03).
|
||||||
|
*
|
||||||
* Reading `s.currency` alone labels every foreign order row as AUD.
|
* Reading `s.currency` alone labels every foreign order row as AUD.
|
||||||
* `s` must be the statements alias in scope.
|
* `s` must be the statements alias in scope.
|
||||||
*/
|
*/
|
||||||
export const NATIVE_CURRENCY = `COALESCE(s.currency, t.foreign_currency_code, 'AUD')`;
|
export const NATIVE_CURRENCY = `COALESCE(s.currency, CASE WHEN t.amount_aud IS NULL THEN t.foreign_currency_code END, 'AUD')`;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* True when a row's AUD value is unknown: it is denominated in something other
|
* True when a row's AUD value is unknown: it is denominated in something other
|
||||||
|
|||||||
@@ -0,0 +1,275 @@
|
|||||||
|
import { queryRaw } from "@/lib/db";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Upcoming annual charges, derived from payments alone.
|
||||||
|
*
|
||||||
|
* Independent of the email lane on purpose: a recurring CHARGE is visible in the
|
||||||
|
* ledger whether or not anything was ever emailed about it, which is the only
|
||||||
|
* way a card annual fee or a loan package fee shows up at all -- banks announce
|
||||||
|
* those on the statement, not in a renewal notice.
|
||||||
|
*
|
||||||
|
* WHAT DOES NOT WORK, measured before this was written: "two charges about 365
|
||||||
|
* days apart" is useless on its own. Ranked that way the top hit on this ledger
|
||||||
|
* is a bakery with 202 such pairs, then a donation, Patreon, Coles -- merchants
|
||||||
|
* used often enough that some pair is inevitably a year apart. Frequency is the
|
||||||
|
* confound. Even "every gap is annual" still surfaces a Target visited twice
|
||||||
|
* ($30, then $198) and a Specsavers ($24, then $224). Two visits a year apart
|
||||||
|
* are not a fee.
|
||||||
|
*
|
||||||
|
* So there are two signals, and they are kept apart because they are not equally
|
||||||
|
* trustworthy -- a guess must never render beside a certainty as if it were one:
|
||||||
|
*
|
||||||
|
* confirmed the bank called it a fee AND it has repeated on an annual cadence
|
||||||
|
* likely the bank called it a fee, seen once -- the kind is certain, the
|
||||||
|
* cadence is inferred (an annual fee charged once is still annual)
|
||||||
|
* possible no fee wording; a merchant whose every gap is annual AND whose
|
||||||
|
* amount is stable. Catches Amazon Prime, Uber One, VicRoads rego,
|
||||||
|
* none of which carry fee wording. Also where false positives live.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type Confidence = "confirmed" | "likely" | "possible";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `expected_not_seen` is not a lesser version of `upcoming` -- it is the more
|
||||||
|
* interesting row. The estimate has passed and no charge landed, which means one
|
||||||
|
* of three things, and `account_days_stale` is what tells them apart:
|
||||||
|
* the account stopped being imported (HSBC: ledger ends 2026-01-07, while the
|
||||||
|
* mail shows statements to 2026-02-07)
|
||||||
|
* the account genuinely closed
|
||||||
|
* the charge landed under a different label (VicRoads paid 2026-03-16 as
|
||||||
|
* "VICROADS ONLINE PAYM", not "VICROADS")
|
||||||
|
* Reporting it as upcoming would be a lie; hiding it would lose the signal.
|
||||||
|
*/
|
||||||
|
export type ChargeStatus = "upcoming" | "expected_not_seen";
|
||||||
|
|
||||||
|
export type AnnualCharge = {
|
||||||
|
label: string;
|
||||||
|
status: ChargeStatus;
|
||||||
|
bank_name: string | null;
|
||||||
|
account_number: string | null;
|
||||||
|
confidence: Confidence;
|
||||||
|
observations: number;
|
||||||
|
last_charged: string;
|
||||||
|
last_amount: string;
|
||||||
|
amount_low: string;
|
||||||
|
amount_high: string;
|
||||||
|
median_gap_days: number | null;
|
||||||
|
next_due_estimate: string;
|
||||||
|
/** Ledger liveness for the account: a fee that "stopped" may just be an
|
||||||
|
* account that stopped being imported. Null for merchant-signal rows, which
|
||||||
|
* are not tied to one account. */
|
||||||
|
account_last_statement: string | null;
|
||||||
|
account_days_stale: number | null;
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `transaction_type = 'fee'` is the universe, not a description match.
|
||||||
|
*
|
||||||
|
* The statement extractor already classifies fees structurally -- 131 rows
|
||||||
|
* across the ledger -- and every card and loan annual fee on this ledger is
|
||||||
|
* typed that way. An earlier version of this filtered `transaction_type =
|
||||||
|
* 'debit'` and returned NONE of them, which is the whole reason this comment
|
||||||
|
* exists: the fees are not debits.
|
||||||
|
*
|
||||||
|
* That universe is broader than "annual", though. It also holds ANZ's monthly
|
||||||
|
* ACCOUNT SERVICING FEE, Zip's MONTHLY ACCOUNT SERVICE FEE, CommBank's MONTHLY
|
||||||
|
* FEE, Citibank LATE PAYMENT FEE, HSBC OVERSEAS TRANSACTION FEE and every Wise
|
||||||
|
* per-transfer charge. Cadence separates them: an annual fee repeats about a
|
||||||
|
* year apart, the rest repeat monthly, on an event, or never (Wise labels each
|
||||||
|
* transfer uniquely, so they can never group to two observations at all).
|
||||||
|
*/
|
||||||
|
const FEE_TYPE = "fee";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wording that establishes the KIND when there is only one observation.
|
||||||
|
*
|
||||||
|
* A single charge cannot demonstrate a cadence, but a bank calling something an
|
||||||
|
* annual fee has already told us what it is -- Westpac's CARD FEE $295 and AMP's
|
||||||
|
* Professional Pack Annual Fee each appear once and are certainly annual.
|
||||||
|
* Deliberately narrow: bare "fee" would readmit every per-transaction charge.
|
||||||
|
*/
|
||||||
|
const ANNUAL_WORDING =
|
||||||
|
"(annual|yearly)[[:space:]]*(card[[:space:]]*)?(fee|charge)" +
|
||||||
|
"|card[[:space:]]+fee" +
|
||||||
|
"|(package|membership|facility)[[:space:]]+fee";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Resolve every statement to a canonical account identity, following
|
||||||
|
* account_links forward. One card can appear under two bank names and two
|
||||||
|
* numbers after an issuer change (Citi Prestige -> NAB MyCard); without this the
|
||||||
|
* $700 fee charged 2024-12-13 and 2025-11-13 reads as two dead one-offs.
|
||||||
|
*
|
||||||
|
* Recursive because the table permits a chain (A->B->C); `uq_account_links_from`
|
||||||
|
* forbids a fork, and `account_links_no_self` a self-link, so the walk
|
||||||
|
* terminates. The depth guard is belt-and-braces against a cycle someone inserts
|
||||||
|
* by hand (A->B, B->A), which no constraint can catch.
|
||||||
|
*/
|
||||||
|
const RESOLVED_ACCOUNTS = `
|
||||||
|
WITH RECURSIVE chain AS (
|
||||||
|
SELECT s.id AS statement_id, s.bank_name, s.account_number,
|
||||||
|
s.bank_name AS cur_bank, s.account_number_key AS cur_key, 0 AS depth
|
||||||
|
FROM statements s
|
||||||
|
UNION ALL
|
||||||
|
SELECT c.statement_id, c.bank_name, c.account_number,
|
||||||
|
l.to_bank_name, l.to_account_key, c.depth + 1
|
||||||
|
FROM chain c
|
||||||
|
JOIN account_links l
|
||||||
|
ON l.from_bank_name = c.cur_bank AND l.from_account_key = c.cur_key
|
||||||
|
WHERE c.depth < 10
|
||||||
|
),
|
||||||
|
resolved AS (
|
||||||
|
SELECT DISTINCT ON (statement_id)
|
||||||
|
statement_id, bank_name, account_number, cur_bank AS acct_bank, cur_key AS acct_key
|
||||||
|
FROM chain ORDER BY statement_id, depth DESC
|
||||||
|
)`;
|
||||||
|
|
||||||
|
export async function getUpcomingAnnualCharges(ownerId: number): Promise<AnnualCharge[]> {
|
||||||
|
// Signal 1 -- the bank's own fee wording, grouped by resolved account identity
|
||||||
|
// and by the fee's wording so two different fees on one card stay apart
|
||||||
|
// (ANZ charges "ANNUAL FEE" $120 and "ADDITIONAL CARD FEE" $10 on the same
|
||||||
|
// statement, and they are not the same obligation).
|
||||||
|
const feeRows = await queryRaw<AnnualCharge>(
|
||||||
|
`${RESOLVED_ACCOUNTS},
|
||||||
|
fees AS (
|
||||||
|
SELECT r.acct_bank, r.acct_key, r.bank_name, r.account_number,
|
||||||
|
upper(trim(regexp_replace(t.description, '[[:space:]]+', ' ', 'g'))) AS label,
|
||||||
|
t.transaction_date, t.amount
|
||||||
|
FROM transactions t
|
||||||
|
JOIN resolved r ON r.statement_id = t.statement_id
|
||||||
|
JOIN statements s ON s.id = t.statement_id
|
||||||
|
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
|
||||||
|
AND t.amount > 0
|
||||||
|
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||||
|
-- Structural first, wording only as a widening: a bank that types a
|
||||||
|
-- fee as a plain debit still gets picked up if it names it annual.
|
||||||
|
AND (t.transaction_type = $3 OR t.description ~* $2)
|
||||||
|
),
|
||||||
|
gapped AS (
|
||||||
|
SELECT f.*,
|
||||||
|
transaction_date - lag(transaction_date)
|
||||||
|
OVER (PARTITION BY acct_key, label ORDER BY transaction_date) AS gap
|
||||||
|
FROM fees f
|
||||||
|
),
|
||||||
|
agg AS (
|
||||||
|
SELECT acct_key, acct_bank, label,
|
||||||
|
count(*)::int AS observations,
|
||||||
|
max(transaction_date) AS last_charged,
|
||||||
|
min(amount) AS amount_low, max(amount) AS amount_high,
|
||||||
|
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
|
||||||
|
FILTER (WHERE gap IS NOT NULL) AS median_gap,
|
||||||
|
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
|
||||||
|
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
|
||||||
|
(array_agg(bank_name ORDER BY transaction_date DESC))[1] AS bank_name,
|
||||||
|
(array_agg(account_number ORDER BY transaction_date DESC))[1] AS account_number,
|
||||||
|
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
|
||||||
|
FROM gapped GROUP BY acct_key, acct_bank, label
|
||||||
|
),
|
||||||
|
liveness AS (
|
||||||
|
SELECT r.acct_key, max(s.billing_end_date) AS last_statement
|
||||||
|
FROM statements s JOIN resolved r ON r.statement_id = s.id
|
||||||
|
GROUP BY r.acct_key
|
||||||
|
)
|
||||||
|
SELECT a.label,
|
||||||
|
-- The account's CURRENT identity, not the bank that happened to
|
||||||
|
-- print the last fee. That $700 fee last appeared under "Citibank"
|
||||||
|
-- in Nov 2025, but the account is NAB/MyCard now and the next one
|
||||||
|
-- will come from there; naming the dead brand would send you to the
|
||||||
|
-- wrong bank.
|
||||||
|
a.acct_bank AS bank_name, a.account_number,
|
||||||
|
CASE WHEN (a.last_charged + COALESCE(round(a.median_gap)::int, 365)) >= CURRENT_DATE
|
||||||
|
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
|
||||||
|
-- One observation cannot prove a cadence, but the bank naming it an
|
||||||
|
-- annual fee already tells us the kind. Repeated on an annual gap it
|
||||||
|
-- is confirmed; seen once it is likely and says so.
|
||||||
|
CASE WHEN a.gaps > 0 AND a.annual_gaps = a.gaps THEN 'confirmed'
|
||||||
|
ELSE 'likely' END AS confidence,
|
||||||
|
a.observations,
|
||||||
|
a.last_charged::text AS last_charged,
|
||||||
|
a.last_amount::text AS last_amount,
|
||||||
|
a.amount_low::text AS amount_low,
|
||||||
|
a.amount_high::text AS amount_high,
|
||||||
|
round(a.median_gap)::int AS median_gap_days,
|
||||||
|
(a.last_charged + COALESCE(round(a.median_gap)::int, 365))::text AS next_due_estimate,
|
||||||
|
l.last_statement::text AS account_last_statement,
|
||||||
|
(CURRENT_DATE - l.last_statement)::int AS account_days_stale
|
||||||
|
FROM agg a LEFT JOIN liveness l ON l.acct_key = a.acct_key
|
||||||
|
-- Two ways to earn a place, and nothing else gets one. Filtered here, not
|
||||||
|
-- in the UI, so the API can never be read wrongly by a future caller.
|
||||||
|
WHERE
|
||||||
|
-- Repeated on an annual cadence. Excludes ANZ's monthly ACCOUNT
|
||||||
|
-- SERVICING FEE and Zip's MONTHLY ACCOUNT SERVICE FEE, whose gaps are
|
||||||
|
-- ~30 days, and Citibank's LATE PAYMENT FEE, which is event-driven.
|
||||||
|
(a.gaps > 0 AND a.annual_gaps = a.gaps AND a.median_gap >= 300)
|
||||||
|
-- Or seen once, but named as an annual/card/package fee. Westpac's CARD
|
||||||
|
-- FEE and AMP's Professional Pack Annual Fee are each a single charge and
|
||||||
|
-- unmistakably annual. A Wise per-transfer fee is also a single charge
|
||||||
|
-- and says nothing of the sort, which is what keeps it out.
|
||||||
|
OR (a.gaps = 0 AND a.label ~* $2)`,
|
||||||
|
[ownerId, ANNUAL_WORDING, FEE_TYPE]
|
||||||
|
);
|
||||||
|
|
||||||
|
// Signal 2 -- merchant-charged annual items with no fee wording. Requires
|
||||||
|
// BOTH that every gap is annual AND that the amount is stable: VicRoads rego
|
||||||
|
// rises about 3% a year ($825.23 -> $852.88 -> $875.48), so equality is too
|
||||||
|
// strict, while no amount test at all readmits the twice-visited restaurant.
|
||||||
|
// Keyed on merchant, never on card -- that rego was paid on three different
|
||||||
|
// banks, and a card-keyed view would see three unrelated one-offs.
|
||||||
|
const merchantRows = await queryRaw<AnnualCharge>(
|
||||||
|
`WITH t AS (
|
||||||
|
SELECT upper(trim(COALESCE(t.merchant_normalized, t.merchant_name, t.description))) AS label,
|
||||||
|
t.transaction_date, t.amount
|
||||||
|
FROM transactions t
|
||||||
|
LEFT JOIN statements s ON s.id = t.statement_id
|
||||||
|
WHERE t.superseded_by_id IS NULL AND t.reconciled_with_id IS NULL
|
||||||
|
AND t.voided_at IS NULL
|
||||||
|
AND t.transaction_type = 'debit' AND t.amount > 0
|
||||||
|
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||||
|
AND t.transaction_type = 'debit' AND t.description !~* $2
|
||||||
|
),
|
||||||
|
gapped AS (
|
||||||
|
SELECT t.*, transaction_date - lag(transaction_date)
|
||||||
|
OVER (PARTITION BY label ORDER BY transaction_date) AS gap
|
||||||
|
FROM t
|
||||||
|
),
|
||||||
|
agg AS (
|
||||||
|
SELECT label, count(*)::int AS observations,
|
||||||
|
max(transaction_date) AS last_charged,
|
||||||
|
min(amount) AS amount_low, max(amount) AS amount_high,
|
||||||
|
percentile_cont(0.5) WITHIN GROUP (ORDER BY gap)
|
||||||
|
FILTER (WHERE gap IS NOT NULL) AS median_gap,
|
||||||
|
count(*) FILTER (WHERE gap IS NOT NULL) AS gaps,
|
||||||
|
count(*) FILTER (WHERE gap BETWEEN 300 AND 430) AS annual_gaps,
|
||||||
|
(array_agg(amount ORDER BY transaction_date DESC))[1] AS last_amount
|
||||||
|
FROM gapped GROUP BY label
|
||||||
|
)
|
||||||
|
SELECT label, NULL::text AS bank_name, NULL::text AS account_number,
|
||||||
|
CASE WHEN (last_charged + round(median_gap)::int) >= CURRENT_DATE
|
||||||
|
THEN 'upcoming' ELSE 'expected_not_seen' END AS status,
|
||||||
|
'possible' AS confidence, observations,
|
||||||
|
last_charged::text AS last_charged,
|
||||||
|
last_amount::text AS last_amount,
|
||||||
|
amount_low::text AS amount_low,
|
||||||
|
amount_high::text AS amount_high,
|
||||||
|
round(median_gap)::int AS median_gap_days,
|
||||||
|
(last_charged + round(median_gap)::int)::text AS next_due_estimate,
|
||||||
|
NULL::text AS account_last_statement,
|
||||||
|
NULL::int AS account_days_stale
|
||||||
|
FROM agg
|
||||||
|
WHERE gaps >= 1 AND annual_gaps = gaps
|
||||||
|
-- Amount stability is what separates a subscription from a restaurant.
|
||||||
|
-- 20% absorbs CPI drift and an FX-priced renewal without admitting a
|
||||||
|
-- $30-then-$198 pair.
|
||||||
|
AND amount_high <= amount_low * 1.20
|
||||||
|
-- Two observations is the floor for a cadence claim, and this lane has no
|
||||||
|
-- wording to fall back on the way the fee lane does.
|
||||||
|
AND observations >= 2`,
|
||||||
|
[ownerId, ANNUAL_WORDING]
|
||||||
|
);
|
||||||
|
|
||||||
|
// Upcoming first and soonest-first within that, so the thing you can still
|
||||||
|
// act on is at the top; the expected-but-missing rows follow as a review list.
|
||||||
|
const rank = (c: AnnualCharge) => (c.status === "upcoming" ? 0 : 1);
|
||||||
|
return [...feeRows, ...merchantRows].sort(
|
||||||
|
(a, b) => rank(a) - rank(b) || a.next_due_estimate.localeCompare(b.next_due_estimate)
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,235 @@
|
|||||||
|
import { queryRaw, type Queryable } from "@/lib/db";
|
||||||
|
import { merchantMatches } from "@/lib/intake-rules";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Unambiguous statement matches for statementless ("manual") transactions.
|
||||||
|
*
|
||||||
|
* WHY THIS EXISTS
|
||||||
|
* ---------------
|
||||||
|
* The reconcile queue is worked by hand and is already ~177 deep. The wallet
|
||||||
|
* lane writes a transaction for every tap (DECISIONS.md ING-11, 2026-08-22),
|
||||||
|
* which at the observed rate adds ~63 rows a month, ~767 a year. Without an
|
||||||
|
* automatic pass for the easy cases that queue stops being workable, and the
|
||||||
|
* tempting alternative -- filtering the new rows back out of it -- is
|
||||||
|
* explicitly ruled out by the `awaitsStatementLine()` removal note in
|
||||||
|
* queries.ts: "an imported row that has no statement belongs in the queue;
|
||||||
|
* volume is solved upstream, never downstream by hiding what is there."
|
||||||
|
*
|
||||||
|
* WHAT COUNTS AS UNAMBIGUOUS
|
||||||
|
* --------------------------
|
||||||
|
* Deliberately much stricter than the queue's own candidate ranking, which
|
||||||
|
* offers the top 5 within +/-1% and +/-3 days for a human to choose between.
|
||||||
|
* A machine gets one shot and no judgement, so:
|
||||||
|
*
|
||||||
|
* - EXACT amount equality, not the queue's +/-1% band. A near-miss is exactly
|
||||||
|
* the case a person should look at.
|
||||||
|
* - Same owner, and the statement row must not already be reconciled against.
|
||||||
|
* - UNIQUE BOTH WAYS: the manual row must have exactly one candidate, AND
|
||||||
|
* that candidate must be claimed by exactly one manual row. This is the
|
||||||
|
* order_transaction_bridge rule, and it is what stops two identical $13.50
|
||||||
|
* bakery taps in one window from being paired arbitrarily.
|
||||||
|
*
|
||||||
|
* Foreign rows match on the NATIVE pair (foreign_currency_amount +
|
||||||
|
* foreign_currency_code), never on `amount`: `amount` is the AUD the bank
|
||||||
|
* charged after applying its own rate, so a wallet row cannot predict it, while
|
||||||
|
* the native figure is exact on both sides. Same reasoning as
|
||||||
|
* order_transaction_bridge's FX path.
|
||||||
|
*
|
||||||
|
* This function only PROPOSES. Applying a pair goes through the same merge the
|
||||||
|
* manual route uses, so overrides/tags/splits/metadata move identically.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type AutoMatch = {
|
||||||
|
manual_id: number;
|
||||||
|
statement_tx_id: number;
|
||||||
|
amount: string;
|
||||||
|
manual_date: string;
|
||||||
|
statement_date: string;
|
||||||
|
description: string;
|
||||||
|
basis: "aud" | "native";
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Days either side of the manual row's date to consider. Authorisation and
|
||||||
|
* settlement are rarely the same day; 3 matches the queue's own window and
|
||||||
|
* the wallet reconciliation check's default. */
|
||||||
|
export const AUTO_MATCH_WINDOW_DAYS = 3;
|
||||||
|
|
||||||
|
export async function findAutoMatches(
|
||||||
|
ownerId: number,
|
||||||
|
windowDays: number = AUTO_MATCH_WINDOW_DAYS
|
||||||
|
): Promise<AutoMatch[]> {
|
||||||
|
return queryRaw<AutoMatch>(
|
||||||
|
`WITH pairs AS (
|
||||||
|
SELECT m.id AS manual_id,
|
||||||
|
t.id AS statement_tx_id,
|
||||||
|
t.amount::text AS amount,
|
||||||
|
m.transaction_date::text AS manual_date,
|
||||||
|
t.transaction_date::text AS statement_date,
|
||||||
|
t.description AS description,
|
||||||
|
CASE WHEN m.foreign_currency_code IS NOT NULL
|
||||||
|
AND m.foreign_currency_amount IS NOT NULL
|
||||||
|
THEN 'native' ELSE 'aud' END AS basis
|
||||||
|
FROM transactions m
|
||||||
|
JOIN transactions t
|
||||||
|
ON t.statement_id IS NOT NULL
|
||||||
|
AND t.transaction_type = 'debit'
|
||||||
|
AND t.transaction_date BETWEEN m.transaction_date - $2::int
|
||||||
|
AND m.transaction_date + $2::int
|
||||||
|
AND (
|
||||||
|
-- Foreign: compare what both sides actually recorded natively.
|
||||||
|
(m.foreign_currency_code IS NOT NULL
|
||||||
|
AND m.foreign_currency_amount IS NOT NULL
|
||||||
|
AND t.foreign_currency_code = m.foreign_currency_code
|
||||||
|
AND t.foreign_currency_amount = m.foreign_currency_amount)
|
||||||
|
OR -- AUD side: exact equality, never a tolerance band. The
|
||||||
|
-- statement row MAY be foreign here, and that is not a mistake:
|
||||||
|
-- all 48 reconciliations a human has made on this ledger are
|
||||||
|
-- exactly this shape -- a manual row carrying no currency,
|
||||||
|
-- matched on exact AUD against a statement row that does carry
|
||||||
|
-- one (EUR, CHF). Overseas receipts and orders record what was
|
||||||
|
-- charged in AUD; only the wallet lane records natively, which
|
||||||
|
-- is why the branch above exists. Requiring both sides to be
|
||||||
|
-- non-foreign was tried and rejected: it matched 0 of those 48.
|
||||||
|
-- The coincidence risk (a domestic tap whose amount happens to
|
||||||
|
-- equal a foreign charge's AUD figure) is already covered by
|
||||||
|
-- unique-both-ways, which refuses as soon as a second candidate
|
||||||
|
-- exists.
|
||||||
|
(m.foreign_currency_code IS NULL
|
||||||
|
AND t.amount = m.amount)
|
||||||
|
)
|
||||||
|
JOIN statements s ON s.id = t.statement_id
|
||||||
|
WHERE m.statement_id IS NULL
|
||||||
|
AND m.owner_id = $1
|
||||||
|
AND m.reconciled_with_id IS NULL
|
||||||
|
AND m.superseded_by_id IS NULL
|
||||||
|
AND m.voided_at IS NULL
|
||||||
|
AND (m.payment_method IS NULL OR m.payment_method NOT IN ('cash', 'credits'))
|
||||||
|
AND t.superseded_by_id IS NULL
|
||||||
|
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||||
|
-- The statement row must be unclaimed.
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM transactions mt WHERE mt.reconciled_with_id = t.id)
|
||||||
|
)
|
||||||
|
SELECT manual_id, statement_tx_id, amount, manual_date, statement_date, description, basis
|
||||||
|
FROM pairs p
|
||||||
|
WHERE (SELECT count(*) FROM pairs q WHERE q.manual_id = p.manual_id) = 1
|
||||||
|
AND (SELECT count(*) FROM pairs q WHERE q.statement_tx_id = p.statement_tx_id) = 1
|
||||||
|
ORDER BY manual_id`,
|
||||||
|
[ownerId, windowDays]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* An existing statementless transaction that a newly-arriving receipt or order
|
||||||
|
* describes -- i.e. the row the wallet lane already created for this purchase.
|
||||||
|
*
|
||||||
|
* WHY: signals for one purchase arrive in sequence, not in competition. The
|
||||||
|
* wallet notification lands within seconds of the tap; the receipt is shared
|
||||||
|
* hours or days later. Measured on txn 6858: the wallet event arrived 35h25m
|
||||||
|
* before anything created a transaction, and pantry then created one from
|
||||||
|
* scratch -- so the same Woolworths shop would have had two rows. The receipt
|
||||||
|
* must adopt the row that already exists and enrich it.
|
||||||
|
*
|
||||||
|
* Deliberately narrow, for the same reason findAutoMatches() is: exact amount,
|
||||||
|
* a few days either side, unclaimed, and UNIQUE. Two candidates means two
|
||||||
|
* plausible purchases, and inventing a link between the wrong pair moves one
|
||||||
|
* shop's line items onto another shop's row. Returning null is always safe --
|
||||||
|
* the caller falls back to creating its own row, which is today's behaviour.
|
||||||
|
*/
|
||||||
|
/**
|
||||||
|
* The row a newly-arriving WALLET notification describes -- the mirror of
|
||||||
|
* findAdoptableTransaction, for the case where the signals arrived in the other
|
||||||
|
* order.
|
||||||
|
*
|
||||||
|
* WHY: the sequencing this estate is built on -- wallet first, receipt second,
|
||||||
|
* statement last -- is a tendency, not a guarantee. The wallet lane can be
|
||||||
|
* parked (its first 21 rows were backfilled days late), the interpreter runs
|
||||||
|
* every 15 minutes, the phone can be offline. Whenever a receipt reaches the
|
||||||
|
* ledger before the notification for the same tap, the wallet writer used to
|
||||||
|
* insert unconditionally: measured on the 2026-08-18 Woolworths shop, txn 6858
|
||||||
|
* (pantry, 23 items) and txn 6927 (wallet) were the same $103.30 purchase, and
|
||||||
|
* the second sat in the reconcile queue waiting for a statement line that the
|
||||||
|
* first had already claimed.
|
||||||
|
*
|
||||||
|
* Deliberately NOT the same predicate as findAdoptableTransaction, in the one
|
||||||
|
* way that matters: that one refuses a row which already carries line items,
|
||||||
|
* because for a receipt such a row is a DIFFERENT purchase that happens to
|
||||||
|
* share an amount. Here it is the opposite -- an itemised row is the receipt we
|
||||||
|
* are trying to fold into, and refusing it is the bug.
|
||||||
|
*
|
||||||
|
* Losing that guard means the merchant has to supply it. A wallet row adopts
|
||||||
|
* only when the merchant agrees (the same whole-word containment the intake
|
||||||
|
* matcher uses: "Woolworths" vs "WOOLWORTHS 3345"), because amount-and-date
|
||||||
|
* alone would let a notification silently disappear into an unrelated purchase
|
||||||
|
* of the same value -- and a wallet row that is never written is a tap with no
|
||||||
|
* trace, which is worse than a visible duplicate. When in doubt, return null:
|
||||||
|
* the caller then writes its own row, which is the pre-existing behaviour.
|
||||||
|
*/
|
||||||
|
export async function findWalletAdoptableTransaction(
|
||||||
|
ownerId: number,
|
||||||
|
amount: number,
|
||||||
|
date: string,
|
||||||
|
merchant: string | null,
|
||||||
|
windowDays: number = AUTO_MATCH_WINDOW_DAYS,
|
||||||
|
// wallet-ingestion calls this while holding its advisory lock, and must pass
|
||||||
|
// that transaction's connection: borrowing a second pooled one per request
|
||||||
|
// is how a burst of concurrent posts exhausts the pool and deadlocks.
|
||||||
|
db: Pick<Queryable, "queryRaw"> = { queryRaw }
|
||||||
|
): Promise<{ id: number; metadata_id: number | null } | null> {
|
||||||
|
// No usable merchant, no adoption. "Card purchase" is wallet-ingestion's own
|
||||||
|
// fallback for a notification that named no merchant at all.
|
||||||
|
if (!merchant || merchant === "Card purchase") return null;
|
||||||
|
const rows = await db.queryRaw<{ id: number; metadata_id: number | null; merchant: string | null }>(
|
||||||
|
`SELECT t.id, em.id AS metadata_id,
|
||||||
|
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant
|
||||||
|
FROM transactions t
|
||||||
|
LEFT JOIN expense_metadata em ON em.transaction_id = t.id
|
||||||
|
WHERE t.statement_id IS NULL
|
||||||
|
AND t.owner_id = $1
|
||||||
|
AND t.amount = $2::numeric
|
||||||
|
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
|
||||||
|
AND t.reconciled_with_id IS NULL
|
||||||
|
AND t.superseded_by_id IS NULL
|
||||||
|
-- A voided row is a charge that never posted; a real charge for the same
|
||||||
|
-- amount (a rebooking after a released hold) must not fold into it.
|
||||||
|
AND t.voided_at IS NULL
|
||||||
|
-- A wallet row is the OTHER HALF of this tap, not a purchase to fold
|
||||||
|
-- into; findPairedRow() merges those, with different rules.
|
||||||
|
AND COALESCE(t.source, '') <> 'wallet'
|
||||||
|
LIMIT 5`,
|
||||||
|
[ownerId, amount, date, windowDays]
|
||||||
|
);
|
||||||
|
const hits = rows.filter((r) => merchantMatches(r.merchant, merchant));
|
||||||
|
return hits.length === 1 ? { id: hits[0].id, metadata_id: hits[0].metadata_id } : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function findAdoptableTransaction(
|
||||||
|
ownerId: number,
|
||||||
|
amount: number,
|
||||||
|
date: string,
|
||||||
|
windowDays: number = AUTO_MATCH_WINDOW_DAYS
|
||||||
|
): Promise<number | null> {
|
||||||
|
const rows = await queryRaw<{ id: number }>(
|
||||||
|
`SELECT t.id
|
||||||
|
FROM transactions t
|
||||||
|
WHERE t.statement_id IS NULL
|
||||||
|
AND t.owner_id = $1
|
||||||
|
AND t.amount = $2::numeric
|
||||||
|
AND t.transaction_date BETWEEN $3::date - $4::int AND $3::date + $4::int
|
||||||
|
AND t.reconciled_with_id IS NULL
|
||||||
|
AND t.superseded_by_id IS NULL
|
||||||
|
-- A voided row is a charge that never posted; a real charge for the same
|
||||||
|
-- amount (a rebooking after a released hold) must not fold into it.
|
||||||
|
AND t.voided_at IS NULL
|
||||||
|
-- Never adopt a row that another source already fully described; that
|
||||||
|
-- is a different purchase that happens to share an amount.
|
||||||
|
AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM expense_metadata em
|
||||||
|
WHERE (em.transaction_id = t.id OR em.matched_transaction_id = t.id)
|
||||||
|
AND jsonb_typeof(em.line_items) = 'array'
|
||||||
|
AND jsonb_array_length(em.line_items) > 0
|
||||||
|
)
|
||||||
|
LIMIT 2`,
|
||||||
|
[ownerId, amount, date, windowDays]
|
||||||
|
);
|
||||||
|
return rows.length === 1 ? rows[0].id : null;
|
||||||
|
}
|
||||||
@@ -20,3 +20,38 @@ export async function queryRow<T>(sql: string, params: unknown[] = []): Promise<
|
|||||||
const rows = await prisma.$queryRawUnsafe<T[]>(sql, ...params);
|
const rows = await prisma.$queryRawUnsafe<T[]>(sql, ...params);
|
||||||
return rows.length > 0 ? rows[0] : null;
|
return rows.length > 0 ? rows[0] : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** The same two calls, scoped to one connection inside one transaction. */
|
||||||
|
export type Queryable = {
|
||||||
|
queryRaw<T>(sql: string, params?: unknown[]): Promise<T[]>;
|
||||||
|
queryRow<T>(sql: string, params?: unknown[]): Promise<T | null>;
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run several statements as one transaction, on one connection.
|
||||||
|
*
|
||||||
|
* The module-level `queryRaw`/`queryRow` are `$queryRawUnsafe` on a POOLED
|
||||||
|
* client, so each call is its own implicit transaction on whichever connection
|
||||||
|
* the pool hands out. That makes a check-then-write pair non-atomic, and it
|
||||||
|
* makes `pg_advisory_xact_lock` useless outside this helper — the lock is
|
||||||
|
* released the instant the statement that took it commits, which is before the
|
||||||
|
* next statement runs. Anything that needs a lock to still be held on its next
|
||||||
|
* query has to be in here.
|
||||||
|
*/
|
||||||
|
export async function withTransaction<T>(
|
||||||
|
fn: (q: Queryable) => Promise<T>,
|
||||||
|
// Prisma's defaults (maxWait 2s, timeout 5s) count time spent WAITING on an
|
||||||
|
// advisory lock, so a caller that serialises a burst on one needs more.
|
||||||
|
opts?: { maxWait?: number; timeout?: number }
|
||||||
|
): Promise<T> {
|
||||||
|
return prisma.$transaction(async (tx) => {
|
||||||
|
const q: Queryable = {
|
||||||
|
queryRaw: <R>(sql: string, params: unknown[] = []) => tx.$queryRawUnsafe<R[]>(sql, ...params),
|
||||||
|
queryRow: async <R>(sql: string, params: unknown[] = []) => {
|
||||||
|
const rows = await tx.$queryRawUnsafe<R[]>(sql, ...params);
|
||||||
|
return rows.length > 0 ? rows[0] : null;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
return fn(q);
|
||||||
|
}, opts);
|
||||||
|
}
|
||||||
|
|||||||
@@ -177,6 +177,40 @@ export function useUpdateTransaction() {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function useMergeTransaction() {
|
||||||
|
const qc = useQueryClient();
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: async ({
|
||||||
|
id,
|
||||||
|
other_id,
|
||||||
|
mode,
|
||||||
|
}: {
|
||||||
|
id: number;
|
||||||
|
other_id: number;
|
||||||
|
mode: "reconcile" | "supersede";
|
||||||
|
}) => {
|
||||||
|
const res = await fetch(`/api/transactions/${id}/merge`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ other_id, mode }),
|
||||||
|
});
|
||||||
|
if (!res.ok) {
|
||||||
|
const body = await res.json().catch(() => ({}));
|
||||||
|
throw new Error(body.error || "Failed to merge transaction");
|
||||||
|
}
|
||||||
|
return res.json();
|
||||||
|
},
|
||||||
|
onSuccess: () => {
|
||||||
|
qc.invalidateQueries({ queryKey: ["transactions"] });
|
||||||
|
qc.invalidateQueries({ queryKey: ["transaction"] });
|
||||||
|
qc.invalidateQueries({ queryKey: ["splits"] });
|
||||||
|
qc.invalidateQueries({ queryKey: ["analytics"] });
|
||||||
|
qc.invalidateQueries({ queryKey: ["participant-balances"] });
|
||||||
|
qc.invalidateQueries({ queryKey: ["shared-transactions"] });
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function useBulkAction() {
|
export function useBulkAction() {
|
||||||
const qc = useQueryClient();
|
const qc = useQueryClient();
|
||||||
return useMutation({
|
return useMutation({
|
||||||
@@ -1128,6 +1162,27 @@ export function useReconcile() {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Mark a statementless row as a charge that never posted (migration 0040). */
|
||||||
|
export function useVoidTransaction() {
|
||||||
|
const qc = useQueryClient();
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: async ({ id, reason }: { id: number; reason: string }) => {
|
||||||
|
const res = await fetch(`/api/transactions/${id}/void`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ reason }),
|
||||||
|
});
|
||||||
|
if (!res.ok) throw new Error((await res.json()).error || "Void failed");
|
||||||
|
return res.json();
|
||||||
|
},
|
||||||
|
onSuccess: () => {
|
||||||
|
qc.invalidateQueries({ queryKey: ["transactions"] });
|
||||||
|
qc.invalidateQueries({ queryKey: ["reconcile-pending"] });
|
||||||
|
qc.invalidateQueries({ queryKey: ["analytics"] });
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function useMerchantTransactions(merchant: string | null) {
|
export function useMerchantTransactions(merchant: string | null) {
|
||||||
return useQuery<{ transactions: MerchantTxnRow[] }>({
|
return useQuery<{ transactions: MerchantTxnRow[] }>({
|
||||||
queryKey: ["analytics", "merchant-txns", merchant],
|
queryKey: ["analytics", "merchant-txns", merchant],
|
||||||
|
|||||||
@@ -58,6 +58,7 @@ const ctxFor = (batch: BatchRow): RouteContext | null =>
|
|||||||
platform: batch.header.platform,
|
platform: batch.header.platform,
|
||||||
header: batch.header,
|
header: batch.header,
|
||||||
items: batch.items ?? [],
|
items: batch.items ?? [],
|
||||||
|
currency: batch.header.currency,
|
||||||
}
|
}
|
||||||
: null;
|
: null;
|
||||||
|
|
||||||
@@ -79,13 +80,13 @@ export async function handleIntakeAction(
|
|||||||
if (actionId === "intake_use_candidate") {
|
if (actionId === "intake_use_candidate") {
|
||||||
if (!Number.isInteger(value.transaction_id)) return note("That button carried no transaction id.");
|
if (!Number.isInteger(value.transaction_id)) return note("That button carried no transaction id.");
|
||||||
const txnId = value.transaction_id as number;
|
const txnId = value.transaction_id as number;
|
||||||
await replaceLineItems(txnId, ctx.items, { source: "intake", platform: ctx.platform, batchId: batch.id });
|
await replaceLineItems(txnId, ctx.items, { source: "intake", platform: ctx.platform, batchId: batch.id, currency: ctx.currency });
|
||||||
await queryRaw(
|
await queryRaw(
|
||||||
`UPDATE intake_batches SET status = 'enriched', kind = COALESCE(kind, 'food'), transaction_id = $2, resolved_at = now() WHERE id = $1::uuid`,
|
`UPDATE intake_batches SET status = 'enriched', kind = COALESCE(kind, 'food'), transaction_id = $2, resolved_at = now() WHERE id = $1::uuid`,
|
||||||
[batch.id, txnId]
|
[batch.id, txnId]
|
||||||
);
|
);
|
||||||
return {
|
return {
|
||||||
...enrichedCard({ batchId: batch.id, merchant: ctx.header.merchant, total: ctx.header.total, orderDate: ctx.header.order_date, platform: ctx.platform, itemCount: ctx.items.length, transactionId: txnId }),
|
...enrichedCard({ batchId: batch.id, merchant: ctx.header.merchant, total: ctx.header.total, orderDate: ctx.header.order_date, platform: ctx.platform, itemCount: ctx.items.length, currency: ctx.currency, transactionId: txnId }),
|
||||||
replace: true,
|
replace: true,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
+90
-20
@@ -12,7 +12,7 @@ import { platformLabel } from "@/lib/intake-rules";
|
|||||||
|
|
||||||
export type Block = Record<string, unknown>;
|
export type Block = Record<string, unknown>;
|
||||||
|
|
||||||
export type Candidate = { transaction_id: number; description: string | null; transaction_date: string; amount: string | number };
|
export type Candidate = { transaction_id: number; description: string | null; transaction_date: string; amount: string | number; currency?: string | null };
|
||||||
|
|
||||||
export function financePublicUrl(): string {
|
export function financePublicUrl(): string {
|
||||||
return (process.env.FINANCE_PUBLIC_URL || "https://finance.bosecamp.com").replace(/\/$/, "");
|
return (process.env.FINANCE_PUBLIC_URL || "https://finance.bosecamp.com").replace(/\/$/, "");
|
||||||
@@ -23,7 +23,29 @@ export function transactionLink(merchant: string | null): string {
|
|||||||
return `${financePublicUrl()}/transactions?q=${encodeURIComponent(merchant ?? "")}`;
|
return `${financePublicUrl()}/transactions?q=${encodeURIComponent(merchant ?? "")}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
const money = (n: number | string) => `$${Number(n).toFixed(2)}`;
|
/**
|
||||||
|
* Slack renders `&`, `<` and `>` as markup in mrkdwn (and in the `text`
|
||||||
|
* fallback). Merchant names, transaction descriptions and model output are
|
||||||
|
* none of ours — `Ben & Jerry's` must not arrive as `Ben & Jerry's`, and a
|
||||||
|
* description containing `<http://…|click>` must not become a link the card
|
||||||
|
* never meant to show. Applied to interpolated values only; our own literal
|
||||||
|
* `*bold*` and `:emoji:` stay markup.
|
||||||
|
*/
|
||||||
|
export function esc(value: string | null | undefined): string {
|
||||||
|
return (value ?? "").replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `$` is only right for AUD. An intake batch carries the currency the receipt
|
||||||
|
* printed, and the order nudge already renders `USD 15.00` rather than
|
||||||
|
* pretending a foreign order was dollars.
|
||||||
|
*/
|
||||||
|
const money = (n: number | string, currency?: string | null) => {
|
||||||
|
const amount = Number(n).toFixed(2);
|
||||||
|
// The currency comes off the model (`parseExtraction` accepts any string), so
|
||||||
|
// an unescaped `<@U123>` here would be a mention on every card.
|
||||||
|
return !currency || currency === "AUD" ? `$${amount}` : `${esc(currency)} ${amount}`;
|
||||||
|
};
|
||||||
const section = (text: string): Block => ({ type: "section", text: { type: "mrkdwn", text } });
|
const section = (text: string): Block => ({ type: "section", text: { type: "mrkdwn", text } });
|
||||||
const button = (action_id: string, text: string, value: unknown, style?: "primary" | "danger"): Block => ({
|
const button = (action_id: string, text: string, value: unknown, style?: "primary" | "danger"): Block => ({
|
||||||
type: "button",
|
type: "button",
|
||||||
@@ -40,10 +62,12 @@ export type CardInput = {
|
|||||||
orderDate: string | null;
|
orderDate: string | null;
|
||||||
platform: string | null;
|
platform: string | null;
|
||||||
itemCount: number;
|
itemCount: number;
|
||||||
|
currency?: string | null;
|
||||||
};
|
};
|
||||||
|
|
||||||
export function enrichedCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
|
export function enrichedCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
|
||||||
const what = [c.total !== null ? money(c.total) : null, platformLabel(c.platform) || c.merchant, c.platform && c.merchant ? `(${c.merchant})` : null]
|
const merchant = esc(c.merchant);
|
||||||
|
const what = [c.total !== null ? money(c.total, c.currency) : null, esc(platformLabel(c.platform)) || merchant, c.platform && merchant ? `(${merchant})` : null]
|
||||||
.filter(Boolean)
|
.filter(Boolean)
|
||||||
.join(" ");
|
.join(" ");
|
||||||
const text = `Added ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} to your ${what} order ✓`;
|
const text = `Added ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} to your ${what} order ✓`;
|
||||||
@@ -60,7 +84,7 @@ export function enrichedCard(c: CardInput & { transactionId: number }): { blocks
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function createdCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
|
export function createdCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
|
||||||
const text = `Created a ${c.total !== null ? money(c.total) + " " : ""}${c.merchant ?? "new"} transaction with ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} ✓`;
|
const text = `Created a ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant) || "new"} transaction with ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} ✓`;
|
||||||
return {
|
return {
|
||||||
text,
|
text,
|
||||||
blocks: [
|
blocks: [
|
||||||
@@ -74,42 +98,83 @@ export function createdCard(c: CardInput & { transactionId: number }): { blocks:
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks: Block[]; text: string } {
|
export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks: Block[]; text: string } {
|
||||||
const label = platformLabel(c.platform) || "";
|
const label = esc(platformLabel(c.platform));
|
||||||
const head = `No matching ${label ? label + " " : ""}order for *${c.merchant ?? "this merchant"}* ${c.total !== null ? money(c.total) : ""} on ${c.orderDate ?? "?"}.`;
|
const head = `No matching ${label ? label + " " : ""}order for *${esc(c.merchant) || "this merchant"}* ${c.total !== null ? money(c.total, c.currency) : ""} on ${esc(c.orderDate) || "?"}.`;
|
||||||
const text = `${head.replace(/\*/g, "")} ${candidates.length ? "Pick one below or create it." : "Create it?"}`;
|
const text = `${head.replace(/\*/g, "")} ${candidates.length ? "Pick one below or create it." : "Create it?"}`;
|
||||||
const blocks: Block[] = [section(`:mag: ${head}${candidates.length ? "\nIs it one of these?" : ""}`)];
|
const blocks: Block[] = [section(`:mag: ${head}${candidates.length ? "\nIs it one of these?" : ""}`)];
|
||||||
for (const cand of candidates) {
|
for (const cand of candidates) {
|
||||||
blocks.push({
|
blocks.push({
|
||||||
type: "section",
|
type: "section",
|
||||||
text: { type: "mrkdwn", text: `${cand.transaction_date} · ${cand.description ?? "—"} · ${money(cand.amount)}` },
|
text: { type: "mrkdwn", text: `${esc(cand.transaction_date)} · ${esc(cand.description) || "—"} · ${money(cand.amount, cand.currency)}` },
|
||||||
accessory: button("intake_use_candidate", "Use this", { batch_id: c.batchId, transaction_id: cand.transaction_id }),
|
accessory: button("intake_use_candidate", "Use this", { batch_id: c.batchId, transaction_id: cand.transaction_id }),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
blocks.push({
|
blocks.push({
|
||||||
type: "actions",
|
type: "actions",
|
||||||
block_id: "intake_create",
|
block_id: "intake_create",
|
||||||
elements: [button("intake_create", `Create new ${c.total !== null ? money(c.total) + " " : ""}${c.merchant ?? ""} transaction`.replace(/\s+/g, " ").trim(), { batch_id: c.batchId }, "primary")],
|
// plain_text disables *formatting*, not entity decoding: `&`, `<` and `>`
|
||||||
|
// are control characters everywhere in a Slack payload, so the label is
|
||||||
|
// escaped like any other value we did not write.
|
||||||
|
elements: [button("intake_create", `Create new ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant)} transaction`.replace(/\s+/g, " ").trim(), { batch_id: c.batchId }, "primary")],
|
||||||
});
|
});
|
||||||
return { blocks, text };
|
return { blocks, text };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function forwardedCard(c: CardInput & { captureId: number; enriching: boolean }): { blocks: Block[]; text: string } {
|
export function forwardedCard(
|
||||||
const text = `${c.merchant ?? "Grocery receipt"} ${c.total !== null ? money(c.total) : ""} → pantry board (${c.itemCount} item${c.itemCount === 1 ? "" : "s"}) — finance ${c.enriching ? "updates" : "books it"} on confirm`.replace(/\s+/g, " ");
|
c: CardInput & {
|
||||||
|
captureId: number;
|
||||||
|
enriching: boolean;
|
||||||
|
/** Pantry already holds this exact receipt (same barcode tuple). */
|
||||||
|
alreadyCaptured?: { captureId: number; confirmed: boolean } | null;
|
||||||
|
/** Finance found more than one row this could be, so it picked none. */
|
||||||
|
ambiguous?: boolean;
|
||||||
|
}
|
||||||
|
): { blocks: Block[]; text: string } {
|
||||||
|
const what = `${esc(c.merchant) || "Grocery receipt"} ${c.total !== null ? money(c.total, c.currency) : ""}`.trim();
|
||||||
|
const items = `${c.itemCount} item${c.itemCount === 1 ? "" : "s"}`;
|
||||||
|
// The warning comes FIRST and the emoji changes, because this card is read at
|
||||||
|
// a glance on a phone: a duplicate warning appended to a cheerful "→ pantry
|
||||||
|
// board" line is a duplicate warning nobody reads. Confirming both copies
|
||||||
|
// counts the same groceries twice — pantry's addItem accumulates — so the
|
||||||
|
// card has to say which capture to dismiss, not merely that there are two.
|
||||||
|
if (c.alreadyCaptured) {
|
||||||
|
const state = c.alreadyCaptured.confirmed ? "already confirmed" : "still waiting to be confirmed";
|
||||||
|
const text = `Already on the pantry board — ${what} is capture ${c.alreadyCaptured.captureId} (${state}). This upload is capture ${c.captureId}; dismiss it unless it replaces the first, because confirming both counts the same groceries twice.`.replace(/\s+/g, " ");
|
||||||
|
return { text, blocks: [section(`:repeat: ${text}`)] };
|
||||||
|
}
|
||||||
|
const tail = c.ambiguous
|
||||||
|
? "— finance found more than one row it could be, so it picked none; reconcile it there"
|
||||||
|
: `— finance ${c.enriching ? "updates" : "books it"} on confirm`;
|
||||||
|
const text = `${what} → pantry board (${items}) ${tail}`.replace(/\s+/g, " ");
|
||||||
return { text, blocks: [section(`:shopping_trolley: ${text}`)] };
|
return { text, blocks: [section(`:shopping_trolley: ${text}`)] };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function needsHeaderCard(): { blocks: Block[]; text: string } {
|
/**
|
||||||
const text = "I can see items but no order total or date — share the top of the screen too so I can match it.";
|
* Names what is missing, because the advice differs. A delivery-app screenshot
|
||||||
|
* that lost its header needs the top of the screen; a Woolworths eReceipt
|
||||||
|
* SCREENSHOT has no date anywhere on it (seen 2026-08-22 — store, total, items,
|
||||||
|
* payment, and not one date), while the app's own Share button sends the raw
|
||||||
|
* receipt WITH the date. So for a dateless in-store receipt the right advice is
|
||||||
|
* the Share button, or pantry's own upload — not "share more of the screen".
|
||||||
|
*/
|
||||||
|
export function needsHeaderCard(missing: string[] = ["total", "date"], source?: string | null): { blocks: Block[]; text: string } {
|
||||||
|
const what = missing.length ? missing.join(" or ") : "total or date";
|
||||||
|
const inStore = source === "coles" || source === "woolworths";
|
||||||
|
const hint =
|
||||||
|
missing.length === 1 && missing[0] === "date" && inStore
|
||||||
|
? "an app screenshot carries no date — use the app's Share button instead, or upload the picture to pantry directly"
|
||||||
|
: "share the part of the screen that shows it";
|
||||||
|
const text = `I can see items but no ${esc(what)} — ${hint}, so I can match it.`;
|
||||||
return { text, blocks: [section(`:warning: ${text}`)] };
|
return { text, blocks: [section(`:warning: ${text}`)] };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function heldCard(reason: string): { blocks: Block[]; text: string } {
|
export function heldCard(reason: string, headline = "Items don't add up to the total — is a shot missing?"): { blocks: Block[]; text: string } {
|
||||||
const text = `Items don't add up to the total — is a shot missing? (${reason})`;
|
const text = `${headline} (${esc(reason)})`;
|
||||||
return { text, blocks: [section(`:warning: ${text}`)] };
|
return { text, blocks: [section(`:warning: ${text}`)] };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function askCard(c: CardInput): { blocks: Block[]; text: string } {
|
export function askCard(c: CardInput): { blocks: Block[]; text: string } {
|
||||||
const text = `${c.merchant ?? "This upload"} ${c.total !== null ? money(c.total) : ""} — groceries or a meal?`.replace(/\s+/g, " ");
|
const text = `${esc(c.merchant) || "This upload"} ${c.total !== null ? money(c.total, c.currency) : ""} — groceries or a meal?`.replace(/\s+/g, " ");
|
||||||
return {
|
return {
|
||||||
text,
|
text,
|
||||||
blocks: [
|
blocks: [
|
||||||
@@ -126,18 +191,23 @@ export function askCard(c: CardInput): { blocks: Block[]; text: string } {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null }): { blocks: Block[]; text: string } {
|
export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null; currency?: string | null }): { blocks: Block[]; text: string } {
|
||||||
const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${c.batchId}` : "an earlier upload";
|
const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${esc(c.batchId)}` : "an earlier upload";
|
||||||
const text = `Already handled — ${c.merchant ?? "this order"} ${c.total !== null ? money(c.total) : ""} is ${where}. Nothing was changed.`.replace(/\s+/g, " ");
|
// The header (merchant, date, total) IS the identity of a purchase here, so a
|
||||||
|
// genuine SECOND identical purchase — two of the same coffee, one day, one shop
|
||||||
|
// — reads as a re-share and is refused. That is the right default (a re-share is
|
||||||
|
// far commoner than a repeat) but it is a dead end without this sentence: the
|
||||||
|
// card must say what to do instead, or the ledger quietly loses the second one.
|
||||||
|
const text = `Already handled — ${esc(c.merchant) || "this order"} ${c.total !== null ? money(c.total, c.currency) : ""} is ${where}. Nothing was changed — if this really is a second, separate purchase, add it in the app.`.replace(/\s+/g, " ");
|
||||||
return { text, blocks: [section(`:repeat: ${text}`)] };
|
return { text, blocks: [section(`:repeat: ${text}`)] };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function unsupportedCard(detail?: string): { blocks: Block[]; text: string } {
|
export function unsupportedCard(detail?: string): { blocks: Block[]; text: string } {
|
||||||
const text = `I can read receipts, statements and order screens — not this.${detail ? ` (${detail})` : ""}`;
|
const text = `I can read receipts, statements and order screens — not this.${detail ? ` (${esc(detail)})` : ""}`;
|
||||||
return { text, blocks: [section(`:no_entry_sign: ${text}`)] };
|
return { text, blocks: [section(`:no_entry_sign: ${text}`)] };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function errorCard(detail: string): { blocks: Block[]; text: string } {
|
export function errorCard(detail: string): { blocks: Block[]; text: string } {
|
||||||
const text = `Couldn't process that upload: ${detail}`;
|
const text = `Couldn't process that upload: ${esc(detail)}`;
|
||||||
return { text, blocks: [section(`:x: ${text}`)] };
|
return { text, blocks: [section(`:x: ${text}`)] };
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { extractText, getDocumentProxy } from "unpdf";
|
import { extractText, getDocumentProxy } from "unpdf";
|
||||||
import type { ExtractedOrder, ExtractedItem } from "@/lib/intake-rules";
|
import { PROMPT_PLACEHOLDER_MERCHANTS, type ExtractedOrder, type ExtractedItem } from "@/lib/intake-rules";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Extraction for the intake router: one joint model call per batch.
|
* Extraction for the intake router: one joint model call per batch.
|
||||||
@@ -31,13 +31,22 @@ export async function extractPdfText(bytes: Buffer): Promise<string> {
|
|||||||
return (Array.isArray(text) ? text.join("\n") : text).trim();
|
return (Array.isArray(text) ? text.join("\n") : text).trim();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The merchant examples are FORMAT PLACEHOLDERS, never real shops
|
||||||
|
* (`PROMPT_PLACEHOLDER_MERCHANTS`). Given an input with nothing in it, a model
|
||||||
|
* still answers, and it answers out of its prompt: on 2026-08-22 the word
|
||||||
|
* `Image` produced a complete Burgertory receipt because the prompt named
|
||||||
|
* Burgertory. The "not a receipt" rule below and the placeholder names are the
|
||||||
|
* two halves of closing that.
|
||||||
|
*/
|
||||||
export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text.
|
export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text.
|
||||||
|
|
||||||
Return ONE JSON object with exactly these keys:
|
Return ONE JSON object with exactly these keys:
|
||||||
{ "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] }
|
{ "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] }
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (e.g. "Coles (Wyndham Vale)", "Burgertory (Werribee)"). Never the delivery platform. null if not shown.
|
- If what you are given is NOT a receipt or an order — it is blank, a filename, a single word such as "Image", an app screen with no purchase on it, or anything else you cannot read a purchase off — return { "merchant": null, "order_date": null, "total": null, "subtotal": null, "currency": null, "items": [], "adjustments": [] }. Do not construct a plausible order. Nothing in these instructions is data: never return an example from this prompt as if it were printed on the input.
|
||||||
|
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (format: "${PROMPT_PLACEHOLDER_MERCHANTS[0]}"). Never the delivery platform. null if not shown.
|
||||||
- order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown.
|
- order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown.
|
||||||
- total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown.
|
- total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown.
|
||||||
- subtotal: the printed item subtotal before fees and discounts, as a number, or null.
|
- subtotal: the printed item subtotal before fees and discounts, as a number, or null.
|
||||||
@@ -185,9 +194,26 @@ export function liteLLMExtractor(opts: {
|
|||||||
});
|
});
|
||||||
if (!response.ok) throw new ExtractionError(`LiteLLM returned ${response.status}`);
|
if (!response.ok) throw new ExtractionError(`LiteLLM returned ${response.status}`);
|
||||||
const raw = (await response.json()) as { choices?: { message?: { content?: string }; finish_reason?: string }[] };
|
const raw = (await response.json()) as { choices?: { message?: { content?: string }; finish_reason?: string }[] };
|
||||||
// A truncated reply can still parse once the JSON is closed, which quietly
|
// The rule is: a finish_reason that is PRESENT must be `stop`. An absent
|
||||||
// drops the tail of a long receipt. Losing lines silently is worse than failing.
|
// one is accepted, deliberately — providers differ on whether they send the
|
||||||
if (raw.choices?.[0]?.finish_reason === "length") throw new ExtractionError("The receipt was too long to read in one pass.");
|
// field, and failing every call on a provider that omits it would be a
|
||||||
|
// certain outage in exchange for a case that also requires the model to
|
||||||
|
// emit syntactically complete JSON. The harm a truncated reply does — a
|
||||||
|
// partial item list replacing a good one — is closed on the write side
|
||||||
|
// instead, by the guard in `routeFood`/`createFromBatch`.
|
||||||
|
//
|
||||||
|
// `length` is the common one: a truncated reply still parses once the JSON
|
||||||
|
// happens to close, quietly dropping the tail of a long receipt. Any other
|
||||||
|
// reason (`content_filter`, `tool_calls`) is a different reply shape that
|
||||||
|
// the strict schema never constrained.
|
||||||
|
const finish = raw.choices?.[0]?.finish_reason;
|
||||||
|
if (finish && finish !== "stop") {
|
||||||
|
throw new ExtractionError(
|
||||||
|
finish === "length"
|
||||||
|
? "The receipt was too long to read in one pass."
|
||||||
|
: `The model stopped before it finished (${finish}).`
|
||||||
|
);
|
||||||
|
}
|
||||||
const content = raw.choices?.[0]?.message?.content;
|
const content = raw.choices?.[0]?.message?.content;
|
||||||
if (!content) throw new ExtractionError("LiteLLM returned no content");
|
if (!content) throw new ExtractionError("LiteLLM returned no content");
|
||||||
try {
|
try {
|
||||||
|
|||||||
@@ -20,7 +20,21 @@ export type PantryForwardArgs = {
|
|||||||
financeAction: "enrich" | "create";
|
financeAction: "enrich" | "create";
|
||||||
};
|
};
|
||||||
|
|
||||||
export type PantryForwardResult = { captureId: number; itemCount: number | null };
|
/**
|
||||||
|
* `alreadyCaptured` is pantry telling us this exact receipt is already on its
|
||||||
|
* board -- same store, register, receipt number and date, which is the tuple
|
||||||
|
* both Coles and Woolworths encode in their barcode. Pantry deliberately still
|
||||||
|
* ACCEPTS the second capture (the first may have been a bad crop being
|
||||||
|
* replaced) and says so instead of refusing; dropping the field on the floor
|
||||||
|
* turned "said so" into silence, and a duplicate that is not shown is a
|
||||||
|
* duplicate that gets confirmed -- which double-counts the groceries, because
|
||||||
|
* pantry's addItem accumulates.
|
||||||
|
*/
|
||||||
|
export type PantryForwardResult = {
|
||||||
|
captureId: number;
|
||||||
|
itemCount: number | null;
|
||||||
|
alreadyCaptured: { captureId: number; confirmed: boolean } | null;
|
||||||
|
};
|
||||||
|
|
||||||
export class PantryDuplicateError extends Error {
|
export class PantryDuplicateError extends Error {
|
||||||
constructor(
|
constructor(
|
||||||
@@ -65,12 +79,23 @@ export function pantryForwarder(opts: { baseUrl?: string; token?: string; fetchI
|
|||||||
} catch (e) {
|
} catch (e) {
|
||||||
throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`);
|
throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`);
|
||||||
}
|
}
|
||||||
const body = (await response.json().catch(() => ({}))) as { captureId?: number; error?: string; items?: unknown[]; reason?: string };
|
const body = (await response.json().catch(() => ({}))) as {
|
||||||
|
captureId?: number; error?: string; items?: unknown[]; reason?: string;
|
||||||
|
alreadyCaptured?: { captureId?: number; confirmed?: boolean } | null;
|
||||||
|
};
|
||||||
if (response.status === 409) {
|
if (response.status === 409) {
|
||||||
throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured");
|
throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured");
|
||||||
}
|
}
|
||||||
if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim());
|
if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim());
|
||||||
if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId");
|
if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId");
|
||||||
return { captureId: body.captureId, itemCount: Array.isArray(body.items) ? body.items.length : null };
|
const prior = body.alreadyCaptured;
|
||||||
|
return {
|
||||||
|
captureId: body.captureId,
|
||||||
|
itemCount: Array.isArray(body.items) ? body.items.length : null,
|
||||||
|
alreadyCaptured:
|
||||||
|
prior && typeof prior.captureId === "number" && prior.captureId !== body.captureId
|
||||||
|
? { captureId: prior.captureId, confirmed: prior.confirmed === true }
|
||||||
|
: null,
|
||||||
|
};
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
+273
-47
@@ -1,9 +1,14 @@
|
|||||||
import { randomUUID } from "node:crypto";
|
import { randomUUID } from "node:crypto";
|
||||||
import { queryRaw, queryRow } from "@/lib/db";
|
import { queryRaw, queryRow, withTransaction } from "@/lib/db";
|
||||||
|
import { EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY } from "@/lib/analytics-sql";
|
||||||
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
||||||
|
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
|
||||||
import {
|
import {
|
||||||
classifyInput,
|
classifyInput,
|
||||||
|
confabulationReason,
|
||||||
merchantKey,
|
merchantKey,
|
||||||
|
quoteShared,
|
||||||
|
textCouldBeReceipt,
|
||||||
merchantMatches,
|
merchantMatches,
|
||||||
platformForSource,
|
platformForSource,
|
||||||
reconcileItems,
|
reconcileItems,
|
||||||
@@ -207,15 +212,33 @@ export async function findDuplicateBatch(
|
|||||||
return rows.find((r) => merchantMatches(r.header?.merchant, header.merchant)) ?? null;
|
return rows.find((r) => merchantMatches(r.header?.merchant, header.merchant)) ?? null;
|
||||||
}
|
}
|
||||||
|
|
||||||
type MatchRow = { transaction_id: number; description: string | null; transaction_date: string; amount: string; merchant: string | null; platform: string | null };
|
type MatchRow = {
|
||||||
|
transaction_id: number;
|
||||||
|
description: string | null;
|
||||||
|
transaction_date: string;
|
||||||
|
amount: string;
|
||||||
|
merchant: string | null;
|
||||||
|
platform: string | null;
|
||||||
|
/** What `amount` is denominated in — a candidate row is not always dollars. */
|
||||||
|
currency: string;
|
||||||
|
/** The `expense_metadata` row the match resolved through — null when there is none. */
|
||||||
|
metadata_id: number | null;
|
||||||
|
/** Does that row already carry a receipt? */
|
||||||
|
has_line_items: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The match rule (plan §5, D2): `(platform, merchant, date, total)`, amount and
|
* The match rule (plan §5, D2): `(platform, merchant, date, total)`, amount and
|
||||||
* date EXACT, never widened silently. Platform is required when the source
|
* date EXACT, never widened silently. The merchant compares normalised, with
|
||||||
* implies one; an in-store receipt has none and matches on the other three.
|
* whole-word containment ("Coles" vs "Coles (Wyndham Vale)") — that is
|
||||||
* The merchant compares normalised, with containment ("Coles" vs "Coles
|
* spelling, not widening. A unique hit is a match; two hits are an ambiguity
|
||||||
* (Wyndham Vale)") — that is spelling, not widening. A unique hit is a match;
|
* and go to the candidates card like a miss.
|
||||||
* two hits are an ambiguity and go to the candidates card like a miss.
|
*
|
||||||
|
* A NULL platform means "no platform", not "any platform". An in-store Coles
|
||||||
|
* receipt carries none, and `$3 IS NULL OR em.platform = $3` let it match an
|
||||||
|
* Uber Eats Coles order on the same day for the same money — and then overwrite
|
||||||
|
* that order's items with the till receipt's. The in-store side matches rows
|
||||||
|
* with no platform of their own; the delivery side still requires its own.
|
||||||
*/
|
*/
|
||||||
export async function matchTransaction(header: OrderHeader, platform: string | null): Promise<{ match: MatchRow | null; ambiguous: boolean }> {
|
export async function matchTransaction(header: OrderHeader, platform: string | null): Promise<{ match: MatchRow | null; ambiguous: boolean }> {
|
||||||
if (!header.merchant) return { match: null, ambiguous: false };
|
if (!header.merchant) return { match: null, ambiguous: false };
|
||||||
@@ -223,13 +246,15 @@ export async function matchTransaction(header: OrderHeader, platform: string | n
|
|||||||
`SELECT DISTINCT ON (t.id)
|
`SELECT DISTINCT ON (t.id)
|
||||||
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
|
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
|
||||||
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
|
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
|
||||||
em.platform
|
em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency,
|
||||||
|
COALESCE(jsonb_typeof(em.line_items) = 'array' AND jsonb_array_length(em.line_items) > 0, false) AS has_line_items
|
||||||
FROM transactions t
|
FROM transactions t
|
||||||
|
LEFT JOIN statements s ON s.id = t.statement_id
|
||||||
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||||
WHERE t.superseded_by_id IS NULL
|
WHERE ${EXCLUDE_RECONCILED_SOURCE}
|
||||||
AND t.transaction_date = $1::date
|
AND t.transaction_date = $1::date
|
||||||
AND t.amount = $2::numeric
|
AND t.amount = $2::numeric
|
||||||
AND ($3::text IS NULL OR em.platform = $3)
|
AND (($3::text IS NULL AND em.platform IS NULL) OR em.platform = $3)
|
||||||
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
|
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
|
||||||
[header.order_date, header.total, platform]
|
[header.order_date, header.total, platform]
|
||||||
);
|
);
|
||||||
@@ -238,21 +263,29 @@ export async function matchTransaction(header: OrderHeader, platform: string | n
|
|||||||
return { match: null, ambiguous: hits.length > 1 };
|
return { match: null, ambiguous: hits.length > 1 };
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Nearest candidates for the card: same platform and merchant, ±3 days, any amount. */
|
/**
|
||||||
export async function findCandidates(header: OrderHeader, platform: string | null): Promise<Candidate[]> {
|
* Nearest candidates for the card: same merchant, ±3 days, any amount.
|
||||||
|
*
|
||||||
|
* Deliberately NOT filtered by platform. The card is a human choice, not a
|
||||||
|
* decision — the whole reason it is shown is that the exact key missed, so the
|
||||||
|
* platform is one of the things that may be wrong. A filter here narrows the
|
||||||
|
* list at exactly the moment it needs to widen, and the row the user was
|
||||||
|
* looking for is simply absent with nothing saying why.
|
||||||
|
*/
|
||||||
|
export async function findCandidates(header: OrderHeader): Promise<Candidate[]> {
|
||||||
if (!header.merchant) return [];
|
if (!header.merchant) return [];
|
||||||
const rows = await queryRaw<MatchRow>(
|
const rows = await queryRaw<MatchRow>(
|
||||||
`SELECT DISTINCT ON (t.id)
|
`SELECT DISTINCT ON (t.id)
|
||||||
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
|
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
|
||||||
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
|
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
|
||||||
em.platform
|
em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency, false AS has_line_items
|
||||||
FROM transactions t
|
FROM transactions t
|
||||||
|
LEFT JOIN statements s ON s.id = t.statement_id
|
||||||
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||||
WHERE t.superseded_by_id IS NULL
|
WHERE ${EXCLUDE_RECONCILED_SOURCE}
|
||||||
AND t.transaction_date BETWEEN $1::date - 3 AND $1::date + 3
|
AND t.transaction_date BETWEEN $1::date - 3 AND $1::date + 3
|
||||||
AND ($2::text IS NULL OR em.platform = $2)
|
|
||||||
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
|
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
|
||||||
[header.order_date, platform]
|
[header.order_date]
|
||||||
);
|
);
|
||||||
return rows
|
return rows
|
||||||
.filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant))
|
.filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant))
|
||||||
@@ -263,7 +296,7 @@ export async function findCandidates(header: OrderHeader, platform: string | nul
|
|||||||
return Math.abs(Number(a.amount) - header.total) - Math.abs(Number(b.amount) - header.total);
|
return Math.abs(Number(a.amount) - header.total) - Math.abs(Number(b.amount) - header.total);
|
||||||
})
|
})
|
||||||
.slice(0, 5)
|
.slice(0, 5)
|
||||||
.map((r) => ({ transaction_id: r.transaction_id, description: r.description, transaction_date: r.transaction_date, amount: r.amount }));
|
.map((r) => ({ transaction_id: r.transaction_id, description: r.description, transaction_date: r.transaction_date, amount: r.amount, currency: r.currency }));
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── the two destinations ───────────────────────────────────────────────────
|
// ─── the two destinations ───────────────────────────────────────────────────
|
||||||
@@ -275,7 +308,7 @@ export type RouteContext = {
|
|||||||
platform: string | null;
|
platform: string | null;
|
||||||
header: OrderHeader;
|
header: OrderHeader;
|
||||||
items: StoredLineItem[];
|
items: StoredLineItem[];
|
||||||
subtotal?: number | null;
|
currency?: string | null;
|
||||||
};
|
};
|
||||||
|
|
||||||
const cardInput = (ctx: RouteContext): CardInput => ({
|
const cardInput = (ctx: RouteContext): CardInput => ({
|
||||||
@@ -285,6 +318,7 @@ const cardInput = (ctx: RouteContext): CardInput => ({
|
|||||||
orderDate: ctx.header.order_date,
|
orderDate: ctx.header.order_date,
|
||||||
platform: ctx.platform,
|
platform: ctx.platform,
|
||||||
itemCount: ctx.items.length,
|
itemCount: ctx.items.length,
|
||||||
|
currency: ctx.currency,
|
||||||
});
|
});
|
||||||
|
|
||||||
const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, extra: Partial<IntakeResult> = {}): IntakeResult => ({
|
const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, extra: Partial<IntakeResult> = {}): IntakeResult => ({
|
||||||
@@ -299,22 +333,56 @@ const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card:
|
|||||||
...extra,
|
...extra,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Are these items unverifiable — no line carrying any money at all?
|
||||||
|
*
|
||||||
|
* That is `reconcileItems`' `basis: "none"`: pasted text with no prices, or an
|
||||||
|
* extraction that read the names and none of the figures. The reconciliation
|
||||||
|
* check passes it, correctly, because there is nothing to check. What must not
|
||||||
|
* follow is replacing a receipt that IS itemised with one nothing verified.
|
||||||
|
*/
|
||||||
|
const unverifiable = (items: StoredLineItem[]) => items.every((i) => i.amount === null);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The hold, shared by EVERY path that enriches a matched transaction.
|
||||||
|
*
|
||||||
|
* It lived only in `routeFood` and `createFromBatch`'s re-match walked straight
|
||||||
|
* past it — the same upload destroyed the same receipt depending on which
|
||||||
|
* button reached it. Returns null when there is nothing to protect.
|
||||||
|
*/
|
||||||
|
async function holdRatherThanReplace(ctx: RouteContext, match: MatchRow): Promise<IntakeResult | null> {
|
||||||
|
if (!unverifiable(ctx.items) || !match.has_line_items) return null;
|
||||||
|
await updateBatch(ctx.batchId, { status: "held", items: ctx.items });
|
||||||
|
const card = heldCard(
|
||||||
|
"this upload carries no prices, so nothing checked it against the total — and that order already has its items",
|
||||||
|
"I left the existing items alone"
|
||||||
|
);
|
||||||
|
return finish(ctx, "held", "held", card, { transaction_id: match.transaction_id });
|
||||||
|
}
|
||||||
|
|
||||||
/** Food: enrich the matched order's line_items, or show candidates. NEVER auto-creates. */
|
/** Food: enrich the matched order's line_items, or show candidates. NEVER auto-creates. */
|
||||||
export async function routeFood(ctx: RouteContext): Promise<IntakeResult> {
|
export async function routeFood(ctx: RouteContext): Promise<IntakeResult> {
|
||||||
const { match } = await matchTransaction(ctx.header, ctx.platform);
|
const { match } = await matchTransaction(ctx.header, ctx.platform);
|
||||||
if (match) {
|
if (match) {
|
||||||
await replaceLineItems(match.transaction_id, ctx.items, { source: "intake", platform: ctx.platform, batchId: ctx.batchId });
|
const held = await holdRatherThanReplace(ctx, match);
|
||||||
|
if (held) return held;
|
||||||
|
await replaceLineItems(match.transaction_id, ctx.items, { source: "intake", platform: ctx.platform, batchId: ctx.batchId, metadataId: match.metadata_id, currency: ctx.currency });
|
||||||
await updateBatch(ctx.batchId, { kind: "food", status: "enriched", transaction_id: match.transaction_id, items: ctx.items });
|
await updateBatch(ctx.batchId, { kind: "food", status: "enriched", transaction_id: match.transaction_id, items: ctx.items });
|
||||||
return finish(ctx, "food", "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
|
return finish(ctx, "food", "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
|
||||||
}
|
}
|
||||||
const candidates = await findCandidates(ctx.header, ctx.platform);
|
const candidates = await findCandidates(ctx.header);
|
||||||
await updateBatch(ctx.batchId, { kind: "food", status: "candidates", items: ctx.items });
|
await updateBatch(ctx.batchId, { kind: "food", status: "candidates", items: ctx.items });
|
||||||
return finish(ctx, "food", "candidates", candidatesCard(cardInput(ctx), candidates), { candidates });
|
return finish(ctx, "food", "candidates", candidatesCard(cardInput(ctx), candidates), { candidates });
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */
|
/** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */
|
||||||
export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> {
|
export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> {
|
||||||
const { match } = await matchTransaction(ctx.header, ctx.platform);
|
// `ambiguous` is not a miss. Two ledger rows for one shop is precisely the
|
||||||
|
// state that produced the 2026-08-18 Woolworths double-book, and forwarding
|
||||||
|
// with financeAction 'create' as though nothing matched is what turns it into
|
||||||
|
// a third row on confirm. The groceries still go to the board — they have to,
|
||||||
|
// it is where they are worked — but the card must say finance could not pick.
|
||||||
|
const { match, ambiguous } = await matchTransaction(ctx.header, ctx.platform);
|
||||||
try {
|
try {
|
||||||
const r = await pantry({
|
const r = await pantry({
|
||||||
files,
|
files,
|
||||||
@@ -325,12 +393,19 @@ export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantr
|
|||||||
financeAction: match ? "enrich" : "create",
|
financeAction: match ? "enrich" : "create",
|
||||||
});
|
});
|
||||||
await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items });
|
await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items });
|
||||||
const card = forwardedCard({ ...cardInput(ctx), itemCount: r.itemCount ?? ctx.items.length, captureId: r.captureId, enriching: !!match });
|
const card = forwardedCard({
|
||||||
|
...cardInput(ctx),
|
||||||
|
itemCount: r.itemCount ?? ctx.items.length,
|
||||||
|
captureId: r.captureId,
|
||||||
|
enriching: !!match,
|
||||||
|
alreadyCaptured: r.alreadyCaptured,
|
||||||
|
ambiguous,
|
||||||
|
});
|
||||||
return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id });
|
return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id });
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
if (e instanceof PantryDuplicateError) {
|
if (e instanceof PantryDuplicateError) {
|
||||||
await updateBatch(ctx.batchId, { kind: "grocery", status: "duplicate", capture_id: e.captureId, items: ctx.items });
|
await updateBatch(ctx.batchId, { kind: "grocery", status: "duplicate", capture_id: e.captureId, items: ctx.items });
|
||||||
return finish(ctx, "grocery", "duplicate", duplicateCard({ merchant: ctx.header.merchant, total: ctx.header.total, transactionId: match?.transaction_id ?? null, captureId: e.captureId, batchId: null }), {
|
return finish(ctx, "grocery", "duplicate", duplicateCard({ merchant: ctx.header.merchant, total: ctx.header.total, currency: ctx.currency, transactionId: match?.transaction_id ?? null, captureId: e.captureId, batchId: null }), {
|
||||||
capture_id: e.captureId ?? undefined,
|
capture_id: e.captureId ?? undefined,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -345,38 +420,144 @@ export async function createFromBatch(batch: BatchRow, kind: "grocery" | "food")
|
|||||||
if (!batch.header) throw new Error("batch has no header");
|
if (!batch.header) throw new Error("batch has no header");
|
||||||
const header = batch.header;
|
const header = batch.header;
|
||||||
const items = batch.items ?? [];
|
const items = batch.items ?? [];
|
||||||
|
const ctx: RouteContext = {
|
||||||
|
batchId: batch.id,
|
||||||
|
source: batch.source,
|
||||||
|
sharedBy: batch.shared_by,
|
||||||
|
platform: header.platform,
|
||||||
|
header,
|
||||||
|
items,
|
||||||
|
currency: header.currency,
|
||||||
|
};
|
||||||
|
|
||||||
|
// The card was built minutes ago and the tap arrives now. In between, the
|
||||||
|
// email/order bridge books orders on its own schedule — so re-run the match
|
||||||
|
// rather than trusting the "no match" the card was drawn from. Creating a
|
||||||
|
// second row for an order the ledger already holds is the one outcome this
|
||||||
|
// whole slice exists to avoid.
|
||||||
|
const { match } = await matchTransaction(header, header.platform);
|
||||||
|
if (match) {
|
||||||
|
const held = await holdRatherThanReplace(ctx, match);
|
||||||
|
if (held) return held;
|
||||||
|
await replaceLineItems(match.transaction_id, items, { source: "intake", platform: header.platform, batchId: batch.id, metadataId: match.metadata_id, currency: header.currency });
|
||||||
|
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: match.transaction_id, items });
|
||||||
|
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
|
||||||
|
}
|
||||||
|
|
||||||
|
// A sibling batch of the same order — the PDF after the screenshot — may have
|
||||||
|
// been resolved since. `findDuplicateBatch` deliberately ignores unresolved
|
||||||
|
// batches, so two pending cards for one order both reach here; without this
|
||||||
|
// the second tap books the order twice.
|
||||||
|
const twin = await findDuplicateBatch(batch.id, batch.file_sha256, header);
|
||||||
|
if (twin) {
|
||||||
|
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: twin.transaction_id, capture_id: twin.capture_id });
|
||||||
|
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: twin.transaction_id, captureId: twin.capture_id, batchId: twin.id });
|
||||||
|
return finish(ctx, kind, "duplicate", card, {
|
||||||
|
transaction_id: twin.transaction_id ?? undefined,
|
||||||
|
capture_id: twin.capture_id ?? undefined,
|
||||||
|
duplicate_of: { batch_id: twin.id, transaction_id: twin.transaction_id, capture_id: twin.capture_id },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
const ownerId = await ownerForEmail(batch.shared_by);
|
const ownerId = await ownerForEmail(batch.shared_by);
|
||||||
|
|
||||||
|
// Before minting a row: is one already here that nothing has described yet?
|
||||||
|
//
|
||||||
|
// matchTransaction above cannot see it. Its key includes the platform, and a
|
||||||
|
// wallet notification has none — so an Uber Eats screenshot never matches the
|
||||||
|
// wallet row for the very tap that paid for it, and the Create button books
|
||||||
|
// the purchase a second time. That is the normal arrival order, not an edge
|
||||||
|
// case: the notification lands within seconds of the tap and the screenshot
|
||||||
|
// is shared later.
|
||||||
|
//
|
||||||
|
// findAdoptableTransaction is the same helper the receipt lane uses and
|
||||||
|
// carries the same guards — exact amount, ±3 days, unclaimed, UNIQUE, and
|
||||||
|
// never a row that already carries line items. Its uniqueness check runs
|
||||||
|
// outside the advisory lock below, so two simultaneous taps could both adopt;
|
||||||
|
// the second would then find the items in place and fall through to the lock,
|
||||||
|
// which is where duplicates are actually resolved.
|
||||||
|
const adopted = await findAdoptableTransaction(ownerId, header.total, header.order_date);
|
||||||
|
if (adopted) {
|
||||||
|
await replaceLineItems(adopted, items, { source: "intake", platform: header.platform, batchId: batch.id, currency: header.currency });
|
||||||
|
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: adopted, items });
|
||||||
|
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: adopted }), { transaction_id: adopted });
|
||||||
|
}
|
||||||
|
|
||||||
const merchant = header.merchant ?? "Unknown merchant";
|
const merchant = header.merchant ?? "Unknown merchant";
|
||||||
|
// Not `orderDescription()`: it always prefixes "Order - ", which is right for
|
||||||
|
// a delivery receipt and wrong for the in-store till receipt this path also
|
||||||
|
// handles. Not `resolveCategory()` either — it re-derives grocery-vs-food
|
||||||
|
// from a merchant pattern, and `kind` is the answer the router already
|
||||||
|
// reached, remembered merchant and Slack tap included.
|
||||||
const description = header.platform ? `Order - ${merchant}` : merchant;
|
const description = header.platform ? `Order - ${merchant}` : merchant;
|
||||||
const category = kind === "grocery" ? "groceries" : "dining";
|
const category = kind === "grocery" ? "groceries" : "dining";
|
||||||
// source/source_ref make a second press of the same card a no-op rather than a second row.
|
// `amount` is the figure printed on the receipt, in its own currency. No FX
|
||||||
const existing = await queryRow<{ id: number }>(`SELECT id FROM transactions WHERE source = 'intake' AND source_ref = $1`, [batch.id]);
|
// rate exists at intake, so a foreign order leaves `amount_aud` NULL rather
|
||||||
const txnId =
|
// than asserting a conversion — the same shape order-ingestion.ts writes.
|
||||||
existing?.id ??
|
const currency = header.currency ?? "AUD";
|
||||||
(
|
const isAud = currency === "AUD";
|
||||||
await queryRow<{ id: number }>(
|
// Check and insert under one lock, on one connection.
|
||||||
`INSERT INTO transactions (
|
//
|
||||||
transaction_date, description, amount, amount_aud, category, payment_method,
|
// `source_ref = batch.id` makes a second press of THIS card a no-op, and
|
||||||
merchant_name, merchant_normalized, transaction_type, owner_id, source, source_ref
|
// ON CONFLICT makes that safe against a double tap. It cannot help two
|
||||||
) VALUES ($1::date, $2, $3::numeric, $3::numeric, $4, NULL, $5, $5, 'debit', $6, 'intake', $7)
|
// DIFFERENT batches for one purchase — their keys differ — and both taps run
|
||||||
RETURNING id`,
|
// their match and duplicate checks before either inserts, so both can find
|
||||||
[header.order_date, description, header.total, category, merchant, ownerId, batch.id]
|
// nothing. The lock is keyed on the purchase, not the batch, which is the
|
||||||
)
|
// only thing the two share. It has to be transaction-scoped and therefore
|
||||||
)!.id;
|
// inside withTransaction: taken by a statement of its own it would be
|
||||||
await replaceLineItems(txnId, items, { source: "intake", platform: header.platform, batchId: batch.id });
|
// released before the next one ran (see db.ts).
|
||||||
|
const purchaseKey = [header.platform ?? "", merchantKey(merchant), header.order_date, header.total.toFixed(2)].join("|");
|
||||||
|
const outcome = await withTransaction(async (q) => {
|
||||||
|
await q.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, [purchaseKey]);
|
||||||
|
// Mine (a double tap) or another batch's (the race). The ORDER BY is what
|
||||||
|
// tells them apart when both exist.
|
||||||
|
const existing = await q.queryRow<{ id: number; source_ref: string }>(
|
||||||
|
`SELECT id, source_ref FROM transactions
|
||||||
|
WHERE source = 'intake' AND superseded_by_id IS NULL
|
||||||
|
AND (source_ref = $1
|
||||||
|
OR (transaction_date = $2::date AND amount = $3::numeric
|
||||||
|
AND lower(merchant_normalized) = lower($4)))
|
||||||
|
ORDER BY (source_ref = $1) DESC NULLS LAST, id
|
||||||
|
LIMIT 1`,
|
||||||
|
[batch.id, header.order_date, header.total, merchant]
|
||||||
|
);
|
||||||
|
if (existing) return { id: existing.id, mine: existing.source_ref === batch.id };
|
||||||
|
const row = await q.queryRow<{ id: number }>(
|
||||||
|
`INSERT INTO transactions (
|
||||||
|
transaction_date, description, amount, amount_aud, category, payment_method,
|
||||||
|
merchant_name, merchant_normalized, transaction_type, owner_id, source, source_ref,
|
||||||
|
foreign_currency_amount, foreign_currency_code
|
||||||
|
) VALUES ($1::date, $2, $3::numeric, $4::numeric, $5, NULL, $6, $6, 'debit', $7, 'intake', $8, $9::numeric, $10)
|
||||||
|
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL
|
||||||
|
DO NOTHING
|
||||||
|
RETURNING id`,
|
||||||
|
[header.order_date, description, header.total, isAud ? header.total : null, category, merchant, ownerId, batch.id, isAud ? null : header.total, isAud ? null : currency]
|
||||||
|
);
|
||||||
|
return { id: row!.id, mine: true };
|
||||||
|
});
|
||||||
|
|
||||||
|
// Another batch booked this purchase in the moment between the two taps. Say
|
||||||
|
// the same thing the sequential case says one second later, and write nothing.
|
||||||
|
if (!outcome.mine) {
|
||||||
|
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: outcome.id });
|
||||||
|
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: outcome.id, captureId: null, batchId: null });
|
||||||
|
return finish(ctx, kind, "duplicate", card, { transaction_id: outcome.id });
|
||||||
|
}
|
||||||
|
|
||||||
|
const txnId = outcome.id;
|
||||||
|
await replaceLineItems(txnId, items, { source: "intake", platform: header.platform, batchId: batch.id, currency });
|
||||||
await updateBatch(batch.id, { kind, status: "created", transaction_id: txnId });
|
await updateBatch(batch.id, { kind, status: "created", transaction_id: txnId });
|
||||||
const ctx: RouteContext = { batchId: batch.id, source: batch.source, sharedBy: batch.shared_by, platform: header.platform, header, items };
|
|
||||||
return finish(ctx, kind, "created", createdCard({ ...cardInput(ctx), transactionId: txnId }), { transaction_id: txnId });
|
return finish(ctx, kind, "created", createdCard({ ...cardInput(ctx), transactionId: txnId }), { transaction_id: txnId });
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The card for a batch as it stands — what a second press of a resolved card gets back. */
|
/** The card for a batch as it stands — what a second press of a resolved card gets back. */
|
||||||
export function resolvedCard(batch: BatchRow): { blocks: Block[]; text: string } {
|
export function resolvedCard(batch: BatchRow): { blocks: Block[]; text: string } {
|
||||||
const h = batch.header;
|
const h = batch.header;
|
||||||
const base: CardInput = { batchId: batch.id, merchant: h?.merchant ?? null, total: h?.total ?? null, orderDate: h?.order_date ?? null, platform: h?.platform ?? null, itemCount: batch.items?.length ?? 0 };
|
const base: CardInput = { batchId: batch.id, merchant: h?.merchant ?? null, total: h?.total ?? null, orderDate: h?.order_date ?? null, platform: h?.platform ?? null, itemCount: batch.items?.length ?? 0, currency: h?.currency };
|
||||||
if (batch.status === "enriched" && batch.transaction_id) return enrichedCard({ ...base, transactionId: batch.transaction_id });
|
if (batch.status === "enriched" && batch.transaction_id) return enrichedCard({ ...base, transactionId: batch.transaction_id });
|
||||||
if (batch.status === "created" && batch.transaction_id) return createdCard({ ...base, transactionId: batch.transaction_id });
|
if (batch.status === "created" && batch.transaction_id) return createdCard({ ...base, transactionId: batch.transaction_id });
|
||||||
if (batch.status === "forwarded_to_pantry" && batch.capture_id) return forwardedCard({ ...base, captureId: batch.capture_id, enriching: !!batch.transaction_id });
|
if (batch.status === "forwarded_to_pantry" && batch.capture_id) return forwardedCard({ ...base, captureId: batch.capture_id, enriching: !!batch.transaction_id });
|
||||||
return duplicateCard({ merchant: base.merchant, total: base.total, transactionId: batch.transaction_id, captureId: batch.capture_id, batchId: batch.id });
|
return duplicateCard({ merchant: base.merchant, total: base.total, currency: base.currency, transactionId: batch.transaction_id, captureId: batch.capture_id, batchId: batch.id });
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── the router ─────────────────────────────────────────────────────────────
|
// ─── the router ─────────────────────────────────────────────────────────────
|
||||||
@@ -406,6 +587,8 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
|||||||
// 1. classify each file and build the parts, in the order shared.
|
// 1. classify each file and build the parts, in the order shared.
|
||||||
const parts: InputPart[] = [];
|
const parts: InputPart[] = [];
|
||||||
const usable: StoredFile[] = [];
|
const usable: StoredFile[] = [];
|
||||||
|
/** Text parts dropped before the model saw them — see `textCouldBeReceipt`. */
|
||||||
|
const rejectedText: { name: string; text: string; reason: string }[] = [];
|
||||||
for (const f of input.files) {
|
for (const f of input.files) {
|
||||||
const kind = classifyInput(f);
|
const kind = classifyInput(f);
|
||||||
if (kind === "pdf") {
|
if (kind === "pdf") {
|
||||||
@@ -423,7 +606,20 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
|||||||
} else if (kind === "image") {
|
} else if (kind === "image") {
|
||||||
parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name });
|
parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name });
|
||||||
} else if (kind === "text") {
|
} else if (kind === "text") {
|
||||||
parts.push({ kind: "text", text: f.bytes.toString("utf-8"), name: f.name });
|
// Nothing that cannot be a receipt reaches the model. A share that
|
||||||
|
// arrives as the word "Image" (the phone coerced the photo to text) is
|
||||||
|
// not a thin receipt, and asking anyway produces a whole invented one —
|
||||||
|
// 2026-08-22, `textCouldBeReceipt`. Dropped like an unreadable file type
|
||||||
|
// rather than failing the batch, so one junk text part beside a good
|
||||||
|
// screenshot still goes through; if nothing usable is left the card below
|
||||||
|
// quotes what the phone actually sent.
|
||||||
|
const text = f.bytes.toString("utf-8");
|
||||||
|
const check = textCouldBeReceipt(text);
|
||||||
|
if (!check.ok) {
|
||||||
|
rejectedText.push({ name: f.name, text, reason: check.reason });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
parts.push({ kind: "text", text, name: f.name });
|
||||||
} else {
|
} else {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -431,14 +627,23 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
|||||||
}
|
}
|
||||||
if (!parts.length) {
|
if (!parts.length) {
|
||||||
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
|
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
|
||||||
return fail("unsupported", "unsupported", unsupportedCard());
|
const dropped = rejectedText[0];
|
||||||
|
return fail(
|
||||||
|
"unsupported",
|
||||||
|
"unsupported",
|
||||||
|
unsupportedCard(
|
||||||
|
dropped
|
||||||
|
? `the share arrived as text and ${dropped.reason}: "${quoteShared(dropped.text)}" — send the picture or the PDF itself`
|
||||||
|
: undefined
|
||||||
|
)
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// A re-share of the very same files, before spending a model call on it.
|
// A re-share of the very same files, before spending a model call on it.
|
||||||
const dupBySha = await findDuplicateBatch(batchId, shas, null);
|
const dupBySha = await findDuplicateBatch(batchId, shas, null);
|
||||||
if (dupBySha) {
|
if (dupBySha) {
|
||||||
await updateBatch(batchId, { status: "duplicate", transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id });
|
await updateBatch(batchId, { status: "duplicate", transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id });
|
||||||
return fail("unsupported", "duplicate", duplicateCard({ merchant: dupBySha.header?.merchant ?? null, total: dupBySha.header?.total ?? null, transactionId: dupBySha.transaction_id, captureId: dupBySha.capture_id, batchId: dupBySha.id }), 200, {
|
return fail("unsupported", "duplicate", duplicateCard({ merchant: dupBySha.header?.merchant ?? null, total: dupBySha.header?.total ?? null, currency: dupBySha.header?.currency, transactionId: dupBySha.transaction_id, captureId: dupBySha.capture_id, batchId: dupBySha.id }), 200, {
|
||||||
kind: (dupBySha.kind as IntakeKind) ?? "food",
|
kind: (dupBySha.kind as IntakeKind) ?? "food",
|
||||||
duplicate_of: { batch_id: dupBySha.id, transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id },
|
duplicate_of: { batch_id: dupBySha.id, transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id },
|
||||||
});
|
});
|
||||||
@@ -462,11 +667,32 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
|||||||
}
|
}
|
||||||
const items = toLineItems(extracted.items);
|
const items = toLineItems(extracted.items);
|
||||||
|
|
||||||
|
// 2b. did it answer out of the upload, or out of its prompt?
|
||||||
|
const confab = confabulationReason(
|
||||||
|
{
|
||||||
|
textParts: parts.filter((p): p is Extract<InputPart, { kind: "text" }> => p.kind === "text").map((p) => p.text),
|
||||||
|
hasImages: parts.some((p) => p.kind === "image"),
|
||||||
|
},
|
||||||
|
extracted
|
||||||
|
);
|
||||||
|
if (confab) {
|
||||||
|
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
|
||||||
|
return fail("unsupported", "unsupported", unsupportedCard(confab));
|
||||||
|
}
|
||||||
|
|
||||||
// 3. header.
|
// 3. header.
|
||||||
|
//
|
||||||
|
// This path stores the partial header and the items on the batch row, which
|
||||||
|
// is right for a genuine screenshot missing its top — the next tap has
|
||||||
|
// something to work from. It is wrong for a confabulation, and on 2026-08-22
|
||||||
|
// it stored one: an invented Burgertory header with two invented lines, held
|
||||||
|
// back from the matcher only by an absent date. Nothing here could tell the
|
||||||
|
// two apart, which is why the pre-check above and 2b now keep confabulations
|
||||||
|
// from reaching this point at all rather than trying to filter them here.
|
||||||
const headerCheck = validateHeader(extracted);
|
const headerCheck = validateHeader(extracted);
|
||||||
if (!headerCheck.ok) {
|
if (!headerCheck.ok) {
|
||||||
await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" });
|
await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" });
|
||||||
return fail("held", "needs_header", needsHeaderCard(), 200, { items });
|
return fail("held", "needs_header", needsHeaderCard(headerCheck.missing, input.source), 200, { items });
|
||||||
}
|
}
|
||||||
const header = headerCheck.header;
|
const header = headerCheck.header;
|
||||||
const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" };
|
const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" };
|
||||||
@@ -478,7 +704,7 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
|||||||
const dup = await findDuplicateBatch(batchId, shas, storedHeader);
|
const dup = await findDuplicateBatch(batchId, shas, storedHeader);
|
||||||
if (dup) {
|
if (dup) {
|
||||||
await updateBatch(batchId, { header: storedHeader, items, kind, status: "duplicate", transaction_id: dup.transaction_id, capture_id: dup.capture_id });
|
await updateBatch(batchId, { header: storedHeader, items, kind, status: "duplicate", transaction_id: dup.transaction_id, capture_id: dup.capture_id });
|
||||||
return fail(kind, "duplicate", duplicateCard({ merchant: header.merchant, total: header.total, transactionId: dup.transaction_id, captureId: dup.capture_id, batchId: dup.id }), 200, {
|
return fail(kind, "duplicate", duplicateCard({ merchant: header.merchant, total: header.total, currency: storedHeader.currency, transactionId: dup.transaction_id, captureId: dup.capture_id, batchId: dup.id }), 200, {
|
||||||
header,
|
header,
|
||||||
items,
|
items,
|
||||||
duplicate_of: { batch_id: dup.id, transaction_id: dup.transaction_id, capture_id: dup.capture_id },
|
duplicate_of: { batch_id: dup.id, transaction_id: dup.transaction_id, capture_id: dup.capture_id },
|
||||||
@@ -493,7 +719,7 @@ export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<I
|
|||||||
}
|
}
|
||||||
|
|
||||||
await updateBatch(batchId, { header: storedHeader, items, kind, status: "received" });
|
await updateBatch(batchId, { header: storedHeader, items, kind, status: "received" });
|
||||||
const ctx: RouteContext = { batchId, source: input.source, sharedBy: input.sharedBy, platform, header, items, subtotal: extracted.subtotal };
|
const ctx: RouteContext = { batchId, source: input.source, sharedBy: input.sharedBy, platform, header, items, currency: storedHeader.currency };
|
||||||
|
|
||||||
// 7. act.
|
// 7. act.
|
||||||
if (kind === "ask") {
|
if (kind === "ask") {
|
||||||
|
|||||||
+123
-8
@@ -43,6 +43,7 @@ export function platformForSource(source: IntakeSource): string | null {
|
|||||||
export const PLATFORM_LABEL: Record<string, string> = {
|
export const PLATFORM_LABEL: Record<string, string> = {
|
||||||
ubereats: "Uber Eats",
|
ubereats: "Uber Eats",
|
||||||
doordash: "DoorDash",
|
doordash: "DoorDash",
|
||||||
|
uber: "Uber",
|
||||||
};
|
};
|
||||||
|
|
||||||
export function platformLabel(platform: string | null): string {
|
export function platformLabel(platform: string | null): string {
|
||||||
@@ -51,10 +52,16 @@ export function platformLabel(platform: string | null): string {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* A small pattern list, not an exact lookup (plan §2, Codex #6). Uber Eats prints
|
* A small pattern list, not an exact lookup (plan §2, Codex #6). Uber Eats prints
|
||||||
* `Coles (Wyndham Vale)`, a PDF may print `Woolworths Group Limited`; substring on
|
* `Coles (Wyndham Vale)`, a PDF may print `Woolworths Group Limited`; a
|
||||||
* the normalised name covers both without a merchant master-data subsystem.
|
* word-boundary alternation on the normalised name covers both without a
|
||||||
|
* merchant master-data subsystem.
|
||||||
|
*
|
||||||
|
* The boundaries are load-bearing, not tidiness: a bare substring made
|
||||||
|
* `Bigals Pizza` a grocer, because "b-IGA-ls" contains `iga`. Short names in a
|
||||||
|
* substring list collide with ordinary words, and the collision reads as a
|
||||||
|
* correct classification.
|
||||||
*/
|
*/
|
||||||
export const GROCERY_PATTERN = /coles|woolworths|woolies|aldi|iga|foodworks|costco/i;
|
export const GROCERY_PATTERN = /\b(coles|woolworths|woolies|aldi|iga|foodworks|costco)\b/i;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The normalised merchant used for matching and for the remembered-kind table.
|
* The normalised merchant used for matching and for the remembered-kind table.
|
||||||
@@ -219,10 +226,118 @@ export function toLineItems(items: ExtractedItem[]): StoredLineItem[] {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Does the stored merchant name denote the extracted one? Exact key, or one contains the other. */
|
/** Is `needle` a contiguous run of whole words inside `hay`? */
|
||||||
|
function containsWordRun(hay: string[], needle: string[]): boolean {
|
||||||
|
if (!needle.length || needle.length > hay.length) return false;
|
||||||
|
for (let i = 0; i + needle.length <= hay.length; i++) {
|
||||||
|
if (needle.every((w, j) => hay[i + j] === w)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Does the stored merchant name denote the extracted one? One name's words must
|
||||||
|
* appear as a contiguous run inside the other's — so `Coles` matches
|
||||||
|
* `Coles (Wyndham Vale)` and `Order - Coles (Wyndham Vale)`, which is spelling,
|
||||||
|
* not widening.
|
||||||
|
*
|
||||||
|
* Whole words, never a raw substring: `merchantKey("Bigals Pizza")` contains
|
||||||
|
* `iga`, so a pizza shop matched every IGA on the same date and total.
|
||||||
|
*/
|
||||||
export function merchantMatches(stored: string | null | undefined, extracted: string | null | undefined): boolean {
|
export function merchantMatches(stored: string | null | undefined, extracted: string | null | undefined): boolean {
|
||||||
const a = merchantKey(stored);
|
const a = merchantKey(stored).split(" ").filter(Boolean);
|
||||||
const b = merchantKey(extracted);
|
const b = merchantKey(extracted).split(" ").filter(Boolean);
|
||||||
if (!a || !b) return false;
|
if (!a.length || !b.length) return false;
|
||||||
return a === b || a.includes(b) || b.includes(a);
|
return containsWordRun(a, b) || containsWordRun(b, a);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── is this text even a receipt? (2026-08-22) ───────────────────────────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The merchant names the extraction prompt uses to show the FORMAT. They are
|
||||||
|
* placeholders rather than real shops on purpose.
|
||||||
|
*
|
||||||
|
* On 2026-08-22 an iOS Shortcut coerced a shared photo to text, so the intake
|
||||||
|
* router received one text part whose entire content was the five characters
|
||||||
|
* `Image`. The model was handed that, and returned a whole receipt:
|
||||||
|
* `Burgertory (Braybrook)`, two "The Pride" lines at $22.80, a $41.15 total.
|
||||||
|
* Nothing on the receipt existed — "Burgertory" was in the prompt, as the
|
||||||
|
* example of how to print a branch in parentheses. Only the absent date kept it
|
||||||
|
* out of the matcher; the invented header and items were stored on the batch.
|
||||||
|
*
|
||||||
|
* Same shape as the ingestion engine's 2026-08-05 empty-body defect: a stage
|
||||||
|
* that can be handed nothing has to say so, because the model downstream will
|
||||||
|
* always answer, and it answers out of its prompt.
|
||||||
|
*
|
||||||
|
* Exported from here rather than written into the prompt string so the prompt
|
||||||
|
* and the guard that rejects an echo of it cannot drift apart.
|
||||||
|
*/
|
||||||
|
export const PROMPT_PLACEHOLDER_MERCHANTS = ["<Store> (<Suburb>)"] as const;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Below this a text part needs a money figure to be believable. NOT a floor on
|
||||||
|
* its own: `TOTAL $17.00` is twelve characters and is exactly the kind of
|
||||||
|
* pasted line this door exists to accept.
|
||||||
|
*/
|
||||||
|
export const MIN_RECEIPT_TEXT_CHARS = 40;
|
||||||
|
|
||||||
|
/** `$17.00`, `17.00`, `12,50` — a printed amount, not merely a digit. */
|
||||||
|
const MONEY_SHAPE = /(?:[$€£¥]\s*\d|\d[.,]\d{2}(?!\d))/;
|
||||||
|
|
||||||
|
export type TextCheck = { ok: true } | { ok: false; reason: string };
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Can this text possibly be a receipt? Deterministic, before any model call.
|
||||||
|
*
|
||||||
|
* A receipt has more than one word and at least one number. Everything the
|
||||||
|
* phone substitutes when a share goes wrong — the literal word `Image`, a bare
|
||||||
|
* filename like `IMG_0042.jpeg`, an empty body — fails one of those, and none
|
||||||
|
* of them fails in a way a model will admit to.
|
||||||
|
*
|
||||||
|
* Applies to TEXT parts only. A PDF is checked for readable text separately and
|
||||||
|
* an image is the thing we actually asked for.
|
||||||
|
*/
|
||||||
|
export function textCouldBeReceipt(raw: string | null | undefined): TextCheck {
|
||||||
|
const text = (raw ?? "").trim();
|
||||||
|
if (!text) return { ok: false, reason: "it was empty" };
|
||||||
|
if (!/\s/.test(text)) return { ok: false, reason: "it is a single word" };
|
||||||
|
if (!/\d/.test(text)) return { ok: false, reason: "it has no numbers in it" };
|
||||||
|
if (text.length < MIN_RECEIPT_TEXT_CHARS && !MONEY_SHAPE.test(text)) {
|
||||||
|
return { ok: false, reason: "it is too short to be a receipt" };
|
||||||
|
}
|
||||||
|
return { ok: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What the phone actually sent, short enough to put on a card. */
|
||||||
|
export function quoteShared(raw: string, max = 80): string {
|
||||||
|
const flat = (raw ?? "").replace(/\s+/g, " ").trim();
|
||||||
|
return flat.length > max ? `${flat.slice(0, max - 1)}…` : flat;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Belt to `textCouldBeReceipt`'s braces: did the extractor answer out of its
|
||||||
|
* prompt rather than out of the upload?
|
||||||
|
*
|
||||||
|
* Two tells, both cheap and both deterministic:
|
||||||
|
* - the merchant IS one of the prompt's format placeholders; or
|
||||||
|
* - nothing readable was sent at all (no images, and no text part that could
|
||||||
|
* be a receipt) and yet a merchant or items came back.
|
||||||
|
*
|
||||||
|
* The second is reachable even with the pre-check in place, because the
|
||||||
|
* pre-check only inspects text parts: a PDF that extracts to a few characters
|
||||||
|
* of junk still reaches the model. This is `unsupported`, never `needs_header`
|
||||||
|
* — a confabulated header is not a partial one.
|
||||||
|
*/
|
||||||
|
export function confabulationReason(
|
||||||
|
input: { textParts: string[]; hasImages: boolean },
|
||||||
|
extracted: { merchant: string | null; items: unknown[] }
|
||||||
|
): string | null {
|
||||||
|
const key = merchantKey(extracted.merchant);
|
||||||
|
if (key && PROMPT_PLACEHOLDER_MERCHANTS.some((p) => merchantKey(p) === key)) {
|
||||||
|
return "the merchant came back as the prompt's own example rather than a name off the receipt";
|
||||||
|
}
|
||||||
|
if (input.hasImages) return null;
|
||||||
|
if (input.textParts.some((t) => textCouldBeReceipt(t).ok)) return null;
|
||||||
|
if (!extracted.merchant && !extracted.items.length) return null;
|
||||||
|
return "nothing shared could be read as a receipt, yet an order came back";
|
||||||
}
|
}
|
||||||
|
|||||||
+97
-21
@@ -1,4 +1,4 @@
|
|||||||
import { queryRow, queryRaw } from "@/lib/db";
|
import { queryRow } from "@/lib/db";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Replace the itemised receipt behind a transaction (plan §11 C2).
|
* Replace the itemised receipt behind a transaction (plan §11 C2).
|
||||||
@@ -15,6 +15,12 @@ import { queryRow, queryRaw } from "@/lib/db";
|
|||||||
* transaction has no row at all (a manual transaction, a statement line), one
|
* transaction has no row at all (a manual transaction, a statement line), one
|
||||||
* is created so the items have somewhere to live; a transaction is NEVER
|
* is created so the items have somewhere to live; a transaction is NEVER
|
||||||
* created here.
|
* created here.
|
||||||
|
*
|
||||||
|
* `opts.metadataId` pins WHICH row when the caller already resolved one. A
|
||||||
|
* statement transaction can own its own `expense_metadata` row AND be the
|
||||||
|
* `matched_transaction_id` of a delivery row: the match query and this write
|
||||||
|
* then re-pick independently and can land on different rows, so the enrichment
|
||||||
|
* reports success against a row nobody reads.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
export type LineItemsSource = "intake" | "pantry";
|
export type LineItemsSource = "intake" | "pantry";
|
||||||
@@ -55,10 +61,35 @@ export type ReplaceResult = {
|
|||||||
line_items: LineItemRecord[];
|
line_items: LineItemRecord[];
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The undo baseline, recorded once per batch.
|
||||||
|
*
|
||||||
|
* Returning the row that was just read makes a RETRY destroy the thing undo
|
||||||
|
* exists to restore: pantry re-calls this on every confirm and after a timeout,
|
||||||
|
* and the second call would hand back its own write as the "previous" state.
|
||||||
|
* The first snapshot a batch ever takes is the only one worth keeping, so it is
|
||||||
|
* written under COALESCE and read back rather than trusted.
|
||||||
|
*/
|
||||||
|
async function undoBaseline(batchId: string | null | undefined, snapshot: unknown[]): Promise<unknown[]> {
|
||||||
|
// Pantry passes back whatever batch id it was given; only an intake batch has
|
||||||
|
// a row to record against, and a non-uuid would fail the cast.
|
||||||
|
if (!batchId || !UUID.test(batchId)) return snapshot;
|
||||||
|
const row = await queryRow<{ previous_line_items: unknown }>(
|
||||||
|
`UPDATE intake_batches
|
||||||
|
SET previous_line_items = COALESCE(previous_line_items, $2::jsonb)
|
||||||
|
WHERE id = $1::uuid
|
||||||
|
RETURNING previous_line_items`,
|
||||||
|
[batchId, JSON.stringify(snapshot)]
|
||||||
|
);
|
||||||
|
return Array.isArray(row?.previous_line_items) ? row.previous_line_items : snapshot;
|
||||||
|
}
|
||||||
|
|
||||||
export async function replaceLineItems(
|
export async function replaceLineItems(
|
||||||
transactionId: number,
|
transactionId: number,
|
||||||
lineItems: LineItemRecord[],
|
lineItems: LineItemRecord[],
|
||||||
opts: { source: LineItemsSource; platform?: string | null; batchId?: string | null }
|
opts: { source: LineItemsSource; platform?: string | null; batchId?: string | null; metadataId?: number | null; currency?: string | null }
|
||||||
): Promise<ReplaceResult> {
|
): Promise<ReplaceResult> {
|
||||||
const txn = await queryRow<{
|
const txn = await queryRow<{
|
||||||
id: number;
|
id: number;
|
||||||
@@ -73,36 +104,78 @@ export async function replaceLineItems(
|
|||||||
);
|
);
|
||||||
if (!txn) throw new TransactionNotFoundError(`transaction ${transactionId} does not exist`);
|
if (!txn) throw new TransactionNotFoundError(`transaction ${transactionId} does not exist`);
|
||||||
|
|
||||||
// Prefer the row that IS this transaction over one that merely points at it —
|
// The caller's row wins where it still exists; otherwise prefer the row that
|
||||||
// a statement line can be the `matched_transaction_id` of a receipt row and
|
// IS this transaction over one that merely points at it — a statement line
|
||||||
// carry its own row too, and the items belong on the receipt.
|
// can be the `matched_transaction_id` of a receipt row and carry its own row
|
||||||
|
// too, and the items belong on the receipt.
|
||||||
|
const pinned = opts.metadataId
|
||||||
|
? await queryRow<{ id: number; line_items: unknown }>(
|
||||||
|
`SELECT id, line_items FROM expense_metadata
|
||||||
|
WHERE id = $1 AND (transaction_id = $2 OR matched_transaction_id = $2)`,
|
||||||
|
[opts.metadataId, transactionId]
|
||||||
|
)
|
||||||
|
: null;
|
||||||
|
|
||||||
|
const json = JSON.stringify(lineItems);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Snapshot the baseline, then write. In that order on BOTH paths: the create
|
||||||
|
* path used to snapshot after its INSERT, so a crash in between left a retry
|
||||||
|
* recording the intake's own write as "previous" — undo restoring the thing
|
||||||
|
* it was meant to reverse, which is the failure `previous_line_items` exists
|
||||||
|
* to prevent.
|
||||||
|
*
|
||||||
|
* Returns null when the row is gone between the read and the write. That
|
||||||
|
* UPDATE reported success and the batch was marked enriched against a row
|
||||||
|
* nothing had written to.
|
||||||
|
*/
|
||||||
|
const write = async (row: { id: number; line_items: unknown }): Promise<ReplaceResult | null> => {
|
||||||
|
const previous = await undoBaseline(opts.batchId, Array.isArray(row.line_items) ? row.line_items : []);
|
||||||
|
const updated = await queryRow<{ id: number }>(
|
||||||
|
`UPDATE expense_metadata SET line_items = $2::jsonb WHERE id = $1 RETURNING id`,
|
||||||
|
[row.id, json]
|
||||||
|
);
|
||||||
|
if (!updated) return null;
|
||||||
|
return { transaction_id: transactionId, metadata_id: row.id, created_metadata: false, previous_line_items: previous, line_items: lineItems };
|
||||||
|
};
|
||||||
|
|
||||||
|
if (pinned) {
|
||||||
|
const done = await write(pinned);
|
||||||
|
if (done) return done;
|
||||||
|
}
|
||||||
const existing = await queryRow<{ id: number; line_items: unknown }>(
|
const existing = await queryRow<{ id: number; line_items: unknown }>(
|
||||||
`SELECT id, line_items FROM expense_metadata
|
`SELECT id, line_items FROM expense_metadata
|
||||||
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
||||||
ORDER BY (transaction_id = $1) DESC, id
|
-- NULLS LAST is load-bearing, and its absence inverted this clause.
|
||||||
|
-- Postgres orders DESC as NULLS FIRST, and the row that merely POINTS at
|
||||||
|
-- the transaction has a NULL transaction_id — so the comparison meant to
|
||||||
|
-- prefer the owned row preferred the other one, silently and in exactly
|
||||||
|
-- the case it was written for. matchTransaction spells it out; this did
|
||||||
|
-- not, and the two therefore disagreed.
|
||||||
|
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
|
||||||
LIMIT 1`,
|
LIMIT 1`,
|
||||||
[transactionId]
|
[transactionId]
|
||||||
);
|
);
|
||||||
|
|
||||||
const json = JSON.stringify(lineItems);
|
|
||||||
if (existing) {
|
if (existing) {
|
||||||
await queryRaw(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE id = $1`, [existing.id, json]);
|
const done = await write(existing);
|
||||||
return {
|
if (done) return done;
|
||||||
transaction_id: transactionId,
|
|
||||||
metadata_id: existing.id,
|
|
||||||
created_metadata: false,
|
|
||||||
previous_line_items: Array.isArray(existing.line_items) ? existing.line_items : [],
|
|
||||||
line_items: lineItems,
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Nothing to update. Snapshot the empty baseline BEFORE creating the row.
|
||||||
|
const previous = await undoBaseline(opts.batchId, []);
|
||||||
const reference = opts.batchId ? `intake:${opts.batchId}` : `intake:txn:${transactionId}`;
|
const reference = opts.batchId ? `intake:${opts.batchId}` : `intake:txn:${transactionId}`;
|
||||||
const created = await queryRow<{ id: number }>(
|
// ON CONFLICT because two calls for one transaction can both find no row and
|
||||||
|
// both INSERT; expense_metadata.transaction_id is UNIQUE, so one of them lost
|
||||||
|
// with a constraint error that reached the user as an error card. `xmax = 0`
|
||||||
|
// is how a plain insert is told from a conflict-update, so the reported
|
||||||
|
// `created_metadata` stays true.
|
||||||
|
const created = await queryRow<{ id: number; inserted: boolean }>(
|
||||||
`INSERT INTO expense_metadata (
|
`INSERT INTO expense_metadata (
|
||||||
transaction_id, source, platform, order_reference, line_items,
|
transaction_id, source, platform, order_reference, line_items,
|
||||||
merchant_normalized, amount, transaction_date, currency, flags, extraction_model
|
merchant_normalized, amount, transaction_date, currency, flags, extraction_model
|
||||||
) VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::numeric, $8::date, 'AUD', '[]'::jsonb, 'cloud-budget')
|
) VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::numeric, $8::date, $9, '[]'::jsonb, 'cloud-budget')
|
||||||
RETURNING id`,
|
ON CONFLICT (transaction_id) DO UPDATE SET line_items = EXCLUDED.line_items
|
||||||
|
RETURNING id, (xmax = 0) AS inserted`,
|
||||||
[
|
[
|
||||||
transactionId,
|
transactionId,
|
||||||
opts.source,
|
opts.source,
|
||||||
@@ -112,13 +185,16 @@ export async function replaceLineItems(
|
|||||||
txn.merchant_normalized ?? txn.merchant_name ?? null,
|
txn.merchant_normalized ?? txn.merchant_name ?? null,
|
||||||
txn.amount,
|
txn.amount,
|
||||||
txn.transaction_date,
|
txn.transaction_date,
|
||||||
|
// Not hard-coded 'AUD': the transaction may carry a foreign figure, and
|
||||||
|
// OrderDetails prefers this column, so a USD order printed "$".
|
||||||
|
opts.currency || "AUD",
|
||||||
]
|
]
|
||||||
);
|
);
|
||||||
return {
|
return {
|
||||||
transaction_id: transactionId,
|
transaction_id: transactionId,
|
||||||
metadata_id: created!.id,
|
metadata_id: created!.id,
|
||||||
created_metadata: true,
|
created_metadata: created!.inserted,
|
||||||
previous_line_items: [],
|
previous_line_items: previous,
|
||||||
line_items: lineItems,
|
line_items: lineItems,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,13 +12,6 @@ export const CUTOVER_DATE = "2026-01-09";
|
|||||||
*/
|
*/
|
||||||
export const DEFAULT_OWNER_ID = 1;
|
export const DEFAULT_OWNER_ID = 1;
|
||||||
|
|
||||||
/** Human labels for the platform a receipt came from. */
|
|
||||||
export const PLATFORM_LABEL: Record<ParsedOrder["platform"], string> = {
|
|
||||||
doordash: "DoorDash",
|
|
||||||
ubereats: "Uber Eats",
|
|
||||||
uber: "Uber",
|
|
||||||
};
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Transaction description.
|
* Transaction description.
|
||||||
*
|
*
|
||||||
|
|||||||
@@ -526,7 +526,11 @@ export function parseOrderHTML(html: string, meta: MessageMeta): ParsedOrder {
|
|||||||
} else {
|
} else {
|
||||||
// DoorDash carries no order id anywhere in the receipt, so the provider
|
// DoorDash carries no order id anywhere in the receipt, so the provider
|
||||||
// message id is the only stable identity available. That is correct for
|
// message id is the only stable identity available. That is correct for
|
||||||
// ingestion idempotency: one receipt is one order.
|
// ingestion idempotency: one receipt is one order — PROVIDED every caller
|
||||||
|
// sends the same id for the same email. Two lanes post here (n8n Order
|
||||||
|
// Processor and ingestion-engine order_handoff); both must send the bare
|
||||||
|
// RFC Message-ID. n8n sent Gmail's own id until 2026-10-03 and one order
|
||||||
|
// became txns 7268 + 7269.
|
||||||
if (!meta.messageId) {
|
if (!meta.messageId) {
|
||||||
throw new OrderParseError("no order id in body and no messageId supplied");
|
throw new OrderParseError("no order id in body and no messageId supplied");
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -119,6 +119,16 @@ export async function merchantForTransaction(
|
|||||||
const row = await queryRow<{ merchant_normalized: string | null }>(
|
const row = await queryRow<{ merchant_normalized: string | null }>(
|
||||||
`SELECT merchant_normalized FROM expense_metadata
|
`SELECT merchant_normalized FROM expense_metadata
|
||||||
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
WHERE transaction_id = $1 OR matched_transaction_id = $1
|
||||||
|
-- Ordered the same way as replaceLineItems (line-items.ts) and
|
||||||
|
-- matchTransaction (intake-router.ts), so every reader of this row
|
||||||
|
-- resolves the same one a write lands on. NULLS LAST because DESC is
|
||||||
|
-- NULLS FIRST and the row that merely POINTS at the transaction has a
|
||||||
|
-- NULL transaction_id.
|
||||||
|
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
|
||||||
|
-- pointing-only, and ZERO transactions resolving to more than one row by
|
||||||
|
-- either link — so this changes nothing today. It is here so the readers
|
||||||
|
-- still agree on the day multiplicity appears.
|
||||||
|
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
|
||||||
LIMIT 1`,
|
LIMIT 1`,
|
||||||
[transactionId]
|
[transactionId]
|
||||||
);
|
);
|
||||||
@@ -157,9 +167,24 @@ export async function merchantVerdict(
|
|||||||
FROM order_reviews r
|
FROM order_reviews r
|
||||||
JOIN transactions t ON t.id = r.transaction_id
|
JOIN transactions t ON t.id = r.transaction_id
|
||||||
JOIN participants p ON p.id = r.participant_id
|
JOIN participants p ON p.id = r.participant_id
|
||||||
JOIN expense_metadata em
|
-- (No backticks in here: this SQL lives in a TS template literal and a
|
||||||
ON em.transaction_id = r.transaction_id
|
-- backtick ends the string -- TS1005 on a line that looks like a comment.)
|
||||||
OR em.matched_transaction_id = r.transaction_id
|
-- LATERAL, not a plain JOIN. A plain join on
|
||||||
|
-- transaction_id OR matched_transaction_id fans out: one review
|
||||||
|
-- joined to N metadata rows appears N times in the result, and the loops
|
||||||
|
-- below count every appearance -- inflating the loved/never tallies
|
||||||
|
-- that drive the pre-order warning. LIMIT 50 applies to the fanned-out
|
||||||
|
-- set too, so duplicates would also push older reviews out of the
|
||||||
|
-- window. Same tie-break as every other reader, so the merchant matched
|
||||||
|
-- here is the merchant the UI shows.
|
||||||
|
JOIN LATERAL (
|
||||||
|
SELECT em2.merchant_normalized
|
||||||
|
FROM expense_metadata em2
|
||||||
|
WHERE em2.transaction_id = r.transaction_id
|
||||||
|
OR em2.matched_transaction_id = r.transaction_id
|
||||||
|
ORDER BY (em2.transaction_id = r.transaction_id) DESC NULLS LAST, em2.id
|
||||||
|
LIMIT 1
|
||||||
|
) em ON true
|
||||||
-- Case-folded: the platforms capitalise the same restaurant differently
|
-- Case-folded: the platforms capitalise the same restaurant differently
|
||||||
-- ("TEG Kebabs & Biryani" on Uber Eats, "TEG KEBABS & BIRYANI" on
|
-- ("TEG Kebabs & Biryani" on Uber Eats, "TEG KEBABS & BIRYANI" on
|
||||||
-- DoorDash). An exact match split one restaurant's history in two, so a
|
-- DoorDash). An exact match split one restaurant's history in two, so a
|
||||||
|
|||||||
+52
-1
@@ -367,6 +367,16 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
|
|||||||
SELECT em.route, em.platform
|
SELECT em.route, em.platform
|
||||||
FROM expense_metadata em
|
FROM expense_metadata em
|
||||||
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||||
|
-- Ordered the same way as replaceLineItems (line-items.ts) and
|
||||||
|
-- matchTransaction (intake-router.ts), so every reader of this row
|
||||||
|
-- resolves the same one a write lands on. NULLS LAST because DESC is
|
||||||
|
-- NULLS FIRST and the row that merely POINTS at the transaction has a
|
||||||
|
-- NULL transaction_id.
|
||||||
|
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
|
||||||
|
-- pointing-only, and ZERO transactions resolving to more than one row by
|
||||||
|
-- either link — so this changes nothing today. It is here so the readers
|
||||||
|
-- still agree on the day multiplicity appears.
|
||||||
|
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
|
||||||
LIMIT 1
|
LIMIT 1
|
||||||
) order_ctx ON true
|
) order_ctx ON true
|
||||||
LEFT JOIN LATERAL (
|
LEFT JOIN LATERAL (
|
||||||
@@ -841,6 +851,15 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
|
|||||||
WHERE ts.transaction_id = t.id
|
WHERE ts.transaction_id = t.id
|
||||||
) txn_splits ON true
|
) txn_splits ON true
|
||||||
WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL
|
WHERE t.statement_id IS NULL AND t.owner_id = $1 AND t.reconciled_with_id IS NULL
|
||||||
|
-- A superseded row is already resolved, just not by reconciling. Without
|
||||||
|
-- this the queue kept offering it: a duplicate wallet half hidden under
|
||||||
|
-- its twin (txn 7037, 2026-10-03) sat beside that twin with the same
|
||||||
|
-- statement line as its match, and accepting it on the hidden row would
|
||||||
|
-- have used the line up and stranded the real one.
|
||||||
|
AND t.superseded_by_id IS NULL
|
||||||
|
-- A voided row is resolved too: the charge never posted, so no statement
|
||||||
|
-- line will ever arrive for it (migration 0040).
|
||||||
|
AND t.voided_at IS NULL
|
||||||
AND ${needsCardMatch("t")}
|
AND ${needsCardMatch("t")}
|
||||||
ORDER BY t.transaction_date DESC, t.row_index ASC`,
|
ORDER BY t.transaction_date DESC, t.row_index ASC`,
|
||||||
[ownerId]
|
[ownerId]
|
||||||
@@ -877,12 +896,39 @@ export async function getPendingReconciliations(ownerId: number): Promise<Manual
|
|||||||
FROM transactions m
|
FROM transactions m
|
||||||
JOIN transactions t ON t.statement_id IS NOT NULL
|
JOIN transactions t ON t.statement_id IS NOT NULL
|
||||||
AND t.transaction_date BETWEEN m.transaction_date - 3 AND m.transaction_date + 3
|
AND t.transaction_date BETWEEN m.transaction_date - 3 AND m.transaction_date + 3
|
||||||
AND t.amount BETWEEN m.amount * 0.99 AND m.amount * 1.01
|
-- +-5%, NOT +-1%. A foreign card purchase is charged at the notified
|
||||||
|
-- amount PLUS the card's foreign transaction fee (3% on the Westpac
|
||||||
|
-- Altitude), so the wallet row and its statement line can never match
|
||||||
|
-- exactly and the old +-1% band excluded the statement row outright:
|
||||||
|
-- SP PEAKDESIGN was notified at $42.44 and charged $43.71 (x1.03),
|
||||||
|
-- so the one row that WAS its twin was the one the picker could not
|
||||||
|
-- offer. Widening is safe here because this list is capped at 5 by
|
||||||
|
-- the rn window and ordered by amount closeness; a looser band changes
|
||||||
|
-- which five appear, never how many, and a person confirms each one.
|
||||||
|
-- The AUTO-matcher stays exact on purpose; see findAutoMatches().
|
||||||
|
AND (t.amount BETWEEN m.amount * 0.95 AND m.amount * 1.05
|
||||||
|
-- Or the exact native pair, whatever the AUD figures say. A
|
||||||
|
-- wallet row's AUD is provisional: in Oct 2026 it priced INR at
|
||||||
|
-- 0.0180 while CommBank charged ~0.0149, 20% under, so the band
|
||||||
|
-- excluded every rupee twin. findAutoMatches() matches on this
|
||||||
|
-- pair already, but refuses when two lines share it inside its
|
||||||
|
-- window (INR 112 on 10/4 and again on 10/7) -- and those are
|
||||||
|
-- exactly the rows that land here for a person to choose.
|
||||||
|
OR (m.foreign_currency_code IS NOT NULL
|
||||||
|
AND t.foreign_currency_code = m.foreign_currency_code
|
||||||
|
AND t.foreign_currency_amount = m.foreign_currency_amount))
|
||||||
JOIN statements s ON s.id = t.statement_id
|
JOIN statements s ON s.id = t.statement_id
|
||||||
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
|
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
|
||||||
WHERE m.statement_id IS NULL
|
WHERE m.statement_id IS NULL
|
||||||
AND m.owner_id = $1
|
AND m.owner_id = $1
|
||||||
AND m.reconciled_with_id IS NULL
|
AND m.reconciled_with_id IS NULL
|
||||||
|
AND m.superseded_by_id IS NULL
|
||||||
|
AND m.voided_at IS NULL
|
||||||
|
-- Nor offer a superseded STATEMENT row as the match: the ANZ
|
||||||
|
-- statements 107/142/143 left 31 of those, and a manual row
|
||||||
|
-- reconciled to one inherits its exclusion from every figure.
|
||||||
|
-- findAutoMatches() has had both filters all along.
|
||||||
|
AND t.superseded_by_id IS NULL
|
||||||
AND ${needsCardMatch("m")}
|
AND ${needsCardMatch("m")}
|
||||||
AND COALESCE(t.owner_id, s.owner_id) = $1
|
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||||
AND NOT EXISTS (
|
AND NOT EXISTS (
|
||||||
@@ -980,6 +1026,11 @@ export async function getSharedTransactions(ownerId: number, tagIds?: number[],
|
|||||||
SELECT em.platform
|
SELECT em.platform
|
||||||
FROM expense_metadata em
|
FROM expense_metadata em
|
||||||
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||||
|
-- Same tie-break as getTransactions / replaceLineItems / matchTransaction,
|
||||||
|
-- so every reader resolves the row a write lands on. Without the ORDER BY
|
||||||
|
-- this LIMIT 1 is bounded but arbitrary: it would disagree with the main
|
||||||
|
-- transaction list for the same transaction the day multiplicity appears.
|
||||||
|
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
|
||||||
LIMIT 1
|
LIMIT 1
|
||||||
) order_ctx ON true
|
) order_ctx ON true
|
||||||
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
|
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import { queryRaw, queryRow } from "@/lib/db";
|
import { queryRaw, queryRow } from "@/lib/db";
|
||||||
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
||||||
|
import { findAdoptableTransaction } from "@/lib/auto-reconcile";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Grocery receipts scanned in pantry-app, arriving as candidate spend.
|
* Grocery receipts scanned in pantry-app, arriving as candidate spend.
|
||||||
@@ -154,16 +155,75 @@ export async function processReceiptIngestion(receipt: ParsedReceipt): Promise<R
|
|||||||
const paymentMethod = paymentMethodFor(leg.class);
|
const paymentMethod = paymentMethodFor(leg.class);
|
||||||
const amount = round2(Number(leg.amount));
|
const amount = round2(Number(leg.amount));
|
||||||
const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name;
|
const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name;
|
||||||
const txn = await queryRow<{ id: number }>(
|
|
||||||
`INSERT INTO transactions (
|
// Adopt the row the wallet lane already created for this tap, if there is
|
||||||
transaction_date, description, amount, amount_aud, category, payment_method,
|
// one. Signals for a purchase arrive in sequence -- the notification within
|
||||||
merchant_name, merchant_normalized, transaction_type, owner_id
|
// seconds, the receipt hours later -- so creating a second transaction here
|
||||||
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
|
// is how one shop ends up as two rows. Measured on txn 6858: the wallet
|
||||||
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
|
// event arrived 35h25m before this path created its own row for the same
|
||||||
);
|
// $103.30 Woolworths shop. Null means no unambiguous candidate, and the
|
||||||
|
// fallback is exactly the old behaviour.
|
||||||
|
const adoptedId = await findAdoptableTransaction(DEFAULT_OWNER_ID, amount, receipt.transaction_date);
|
||||||
|
const txn = adoptedId
|
||||||
|
? await queryRow<{ id: number }>(
|
||||||
|
// The receipt read the register tape; the notification read a push
|
||||||
|
// banner. So the receipt wins on what was bought and where, and the
|
||||||
|
// column list here IS the ownership boundary -- `location` is absent
|
||||||
|
// because the wallet notification's suburb is the better value and
|
||||||
|
// must survive.
|
||||||
|
`UPDATE transactions
|
||||||
|
SET description = $2, merchant_name = $3, merchant_normalized = $3,
|
||||||
|
category = $4, payment_method = $5
|
||||||
|
WHERE id = $1 RETURNING id`,
|
||||||
|
[adoptedId, description, receipt.merchant_name, RECEIPT_CATEGORY, paymentMethod]
|
||||||
|
)
|
||||||
|
: await queryRow<{ id: number }>(
|
||||||
|
`INSERT INTO transactions (
|
||||||
|
transaction_date, description, amount, amount_aud, category, payment_method,
|
||||||
|
merchant_name, merchant_normalized, transaction_type, owner_id
|
||||||
|
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
|
||||||
|
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
|
||||||
|
);
|
||||||
|
|
||||||
const carriesLines = leg.leg_index === cardLeg?.leg_index;
|
const carriesLines = leg.leg_index === cardLeg?.leg_index;
|
||||||
const meta = await queryRow<{ id: number }>(
|
|
||||||
|
// An adopted row may already carry a metadata row (the wallet lane writes
|
||||||
|
// one). expense_metadata.transaction_id is UNIQUE, so a second INSERT would
|
||||||
|
// raise -- and must not, because the two describe the same purchase.
|
||||||
|
// Enrich in place instead, with the same ownership split as the transaction
|
||||||
|
// update above: everything the receipt read wins, and anything only the
|
||||||
|
// notification knew (issuer, and the transaction's location) is absent from
|
||||||
|
// the column list rather than conditionally skipped, so it cannot be
|
||||||
|
// clobbered by a bug here. Mirrors order_transaction_bridge's UPGRADE pass.
|
||||||
|
const priorMeta = adoptedId
|
||||||
|
? await queryRow<{ id: number }>(
|
||||||
|
`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [adoptedId]
|
||||||
|
)
|
||||||
|
: null;
|
||||||
|
|
||||||
|
const meta = priorMeta
|
||||||
|
? await queryRow<{ id: number }>(
|
||||||
|
`UPDATE expense_metadata
|
||||||
|
SET source = 'pantry', order_reference = $2, line_items = $3::jsonb,
|
||||||
|
subtotal = $4, amount = $5, merchant_normalized = $6,
|
||||||
|
transaction_date = $7, card_last4 = COALESCE($8, card_last4),
|
||||||
|
currency = 'AUD', flags = flags || $9::jsonb,
|
||||||
|
payment_method = $10, payment_method_detail = $11,
|
||||||
|
tender_raw = COALESCE($12, tender_raw),
|
||||||
|
receipt_sha256 = $13, receipt_group = $14,
|
||||||
|
extraction_model = 'cloud-budget'
|
||||||
|
WHERE id = $1 RETURNING id`,
|
||||||
|
[
|
||||||
|
priorMeta.id, reference,
|
||||||
|
JSON.stringify(carriesLines ? receipt.line_items ?? [] : []),
|
||||||
|
receipt.tax_amount ?? null, amount, receipt.merchant_name,
|
||||||
|
receipt.transaction_date, leg.card_last4 ?? null,
|
||||||
|
JSON.stringify(carriesLines ? [...flags, "enriched_prior_row"] : [...flags, "line_items_on_card_leg", "enriched_prior_row"]),
|
||||||
|
paymentMethod, leg.card_product ?? null, receipt.tender_raw ?? null,
|
||||||
|
receipt.image_sha256 ?? null, group,
|
||||||
|
]
|
||||||
|
)
|
||||||
|
: await queryRow<{ id: number }>(
|
||||||
`INSERT INTO expense_metadata (
|
`INSERT INTO expense_metadata (
|
||||||
transaction_id, source, order_reference, line_items, subtotal, amount,
|
transaction_id, source, order_reference, line_items, subtotal, amount,
|
||||||
merchant_normalized, transaction_date, card_last4, currency, flags,
|
merchant_normalized, transaction_date, card_last4, currency, flags,
|
||||||
|
|||||||
@@ -0,0 +1,248 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Merge a statementless ("manual") transaction into its statement twin.
|
||||||
|
*
|
||||||
|
* Extracted from POST /api/transactions/reconcile so the manual queue and the
|
||||||
|
* automatic pass cannot drift apart. There must be exactly one implementation
|
||||||
|
* of this: it moves curation (overrides, tags, splits) and provenance
|
||||||
|
* (expense_metadata) off a row that reconciliation is about to hide from every
|
||||||
|
* figure, and a second copy of that logic would eventually move a different set.
|
||||||
|
*
|
||||||
|
* The manual row is NOT deleted -- it is linked via `reconciled_with_id`.
|
||||||
|
* Deleting it would cascade its children away; superseding/linking keeps the
|
||||||
|
* history and lets the row stay visible as provenance.
|
||||||
|
*/
|
||||||
|
export async function applyReconcileMatch(
|
||||||
|
manual_id: number,
|
||||||
|
statement_tx_id: number
|
||||||
|
): Promise<void> {
|
||||||
|
await prisma.$transaction(async (tx) => {
|
||||||
|
await moveCurationWithin(tx, manual_id, statement_tx_id);
|
||||||
|
|
||||||
|
// Mark manual tx as reconciled (link to statement tx)
|
||||||
|
await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
|
||||||
|
statement_tx_id,
|
||||||
|
manual_id
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Move overrides, tags, splits and expense_metadata from one transaction to
|
||||||
|
* another, inside a caller-supplied transaction.
|
||||||
|
*
|
||||||
|
* Callers: `applyReconcileMatch` (then sets `reconciled_with_id`) and
|
||||||
|
* `supersedeTransaction` (then sets `superseded_by_id`). Both hide the source
|
||||||
|
* row from every figure, so both must carry its curation across first; keeping
|
||||||
|
* one implementation is what the file header insists on.
|
||||||
|
*/
|
||||||
|
async function moveCurationWithin(
|
||||||
|
tx: Parameters<Parameters<typeof prisma.$transaction>[0]>[0],
|
||||||
|
manual_id: number,
|
||||||
|
statement_tx_id: number
|
||||||
|
): Promise<void> {
|
||||||
|
{
|
||||||
|
// Copy overrides: manual → statement tx
|
||||||
|
const override = await tx.transaction_overrides.findUnique({
|
||||||
|
where: { transaction_id: manual_id },
|
||||||
|
});
|
||||||
|
if (override) {
|
||||||
|
await tx.transaction_overrides.upsert({
|
||||||
|
where: { transaction_id: statement_tx_id },
|
||||||
|
update: {
|
||||||
|
category_override: override.category_override,
|
||||||
|
merchant_normalized: override.merchant_normalized,
|
||||||
|
notes: override.notes,
|
||||||
|
my_share_percent: override.my_share_percent,
|
||||||
|
updated_at: new Date(),
|
||||||
|
},
|
||||||
|
create: {
|
||||||
|
transaction_id: statement_tx_id,
|
||||||
|
category_override: override.category_override,
|
||||||
|
merchant_normalized: override.merchant_normalized,
|
||||||
|
notes: override.notes,
|
||||||
|
my_share_percent: override.my_share_percent,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move tags: manual → statement tx
|
||||||
|
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
|
||||||
|
if (tags.length) {
|
||||||
|
await tx.transaction_tags.createMany({
|
||||||
|
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
|
||||||
|
skipDuplicates: true,
|
||||||
|
});
|
||||||
|
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move splits: manual → statement tx.
|
||||||
|
//
|
||||||
|
// `settled` and `settled_at` MUST be carried. This is a delete-and-recreate,
|
||||||
|
// and the column defaults to false — so omitting it silently marks every
|
||||||
|
// settled split unsettled again and resurrects a debt that was already
|
||||||
|
// paid. Nothing errors and nothing in the UI says the flag was reset; it
|
||||||
|
// just reappears as owed. No queue row carries a settled split today, but
|
||||||
|
// the auto-matcher runs this path unattended, so a latent version of this
|
||||||
|
// would surface as money, not as an exception.
|
||||||
|
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
|
||||||
|
if (splits.length) {
|
||||||
|
await tx.transaction_splits.createMany({
|
||||||
|
data: splits.map((s) => ({
|
||||||
|
transaction_id: statement_tx_id,
|
||||||
|
participant_id: s.participant_id,
|
||||||
|
share_percent: s.share_percent,
|
||||||
|
settled: s.settled ?? false,
|
||||||
|
settled_at: s.settled_at,
|
||||||
|
})),
|
||||||
|
skipDuplicates: true,
|
||||||
|
});
|
||||||
|
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move provenance: manual → statement tx.
|
||||||
|
//
|
||||||
|
// Overrides, tags and splits above were always carried across; expense_metadata was
|
||||||
|
// the one child left behind, which did not matter while every metadata row came from
|
||||||
|
// an email that had created its own transaction. It matters now: a scanned grocery
|
||||||
|
// receipt puts its line items here, and reconciliation hides the manual row from
|
||||||
|
// every figure — so without this the shop's contents disappear at exactly the moment
|
||||||
|
// the statement line appears, and COLES 0556 MANOR LAKES stays as unreadable as it
|
||||||
|
// was before the receipt was ever scanned.
|
||||||
|
//
|
||||||
|
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
|
||||||
|
// or Paperless copy got there first) keeps it. The manual row's metadata stays
|
||||||
|
// attached to the reconciled manual transaction and is flagged, rather than raising a
|
||||||
|
// constraint violation or silently overwriting the other source.
|
||||||
|
const moved = await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE expense_metadata SET transaction_id = $1
|
||||||
|
WHERE transaction_id = $2
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
|
||||||
|
statement_tx_id,
|
||||||
|
manual_id
|
||||||
|
);
|
||||||
|
if (moved === 0) {
|
||||||
|
await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE expense_metadata
|
||||||
|
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
|
||||||
|
WHERE transaction_id = $1`,
|
||||||
|
manual_id
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Manually mark `loser_id` as superseded by `survivor_id`.
|
||||||
|
*
|
||||||
|
* The escape hatch for a duplicate that `applyReconcileMatch` cannot express:
|
||||||
|
* that one merges a statementLESS row into its statement twin, so it has
|
||||||
|
* nothing to say about two rows on the SAME footing -- the pair left by a
|
||||||
|
* statement imported twice (migration 0023 superseded 31 such pairs, $42,040.68,
|
||||||
|
* by hand), or two feed rows for one purchase.
|
||||||
|
*
|
||||||
|
* Curation moves FIRST, using the same block `applyReconcileMatch` runs, because
|
||||||
|
* `superseded_by_id` hides the row from every figure (`analytics-sql.ts`) and
|
||||||
|
* splits or tags left on it would silently stop counting. That is why this lives
|
||||||
|
* beside its sibling and calls into the same movement rather than re-deriving
|
||||||
|
* it: the header of this file warns that a second copy would eventually move a
|
||||||
|
* different set, and a supersede that forgot splits would resurrect a debt in
|
||||||
|
* exactly the way `applyReconcileMatch`'s own comment describes.
|
||||||
|
*
|
||||||
|
* NOT a delete. Every child of `transactions` is ON DELETE CASCADE, so removing
|
||||||
|
* the row would destroy the splits, tags and metadata that make it legible --
|
||||||
|
* and a duplicate is evidence of how the ledger was assembled, which is worth
|
||||||
|
* keeping. `transactions_no_self_supersede` blocks a row pointing at itself; the
|
||||||
|
* survivor is re-checked here anyway so the error is a 400 and not a 500.
|
||||||
|
*/
|
||||||
|
export async function supersedeTransaction(
|
||||||
|
loser_id: number,
|
||||||
|
survivor_id: number
|
||||||
|
): Promise<void> {
|
||||||
|
if (loser_id === survivor_id) throw new Error("A transaction cannot supersede itself");
|
||||||
|
await prisma.$transaction(async (tx) => {
|
||||||
|
// Refuse to bury a row underneath one that is itself already buried, which
|
||||||
|
// would make the curation we just moved unreachable in a second hop.
|
||||||
|
const survivor = await tx.transactions.findUnique({
|
||||||
|
where: { id: survivor_id },
|
||||||
|
select: { superseded_by_id: true },
|
||||||
|
});
|
||||||
|
if (!survivor) throw new Error(`Survivor transaction ${survivor_id} not found`);
|
||||||
|
if (survivor.superseded_by_id !== null) {
|
||||||
|
throw new Error(`Transaction ${survivor_id} is itself superseded; pick the surviving row`);
|
||||||
|
}
|
||||||
|
await moveCurationWithin(tx, loser_id, survivor_id);
|
||||||
|
await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE transactions SET superseded_by_id = $1 WHERE id = $2`,
|
||||||
|
survivor_id,
|
||||||
|
loser_id
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Mark a statementless row as a charge that never posted (migration 0040).
|
||||||
|
*
|
||||||
|
* The case neither sibling above can express: a card notification is sent for
|
||||||
|
* an AUTHORISATION, and when the merchant releases the hold no statement line
|
||||||
|
* ever arrives. Txn 7951 was a $1,040.41 Airbnb hold on a booking cancelled the
|
||||||
|
* next day -- it sat in the reconcile queue with nothing to match and counted as
|
||||||
|
* spend in every figure. There is no survivor to move curation onto, so nothing
|
||||||
|
* moves: the row and its children stay, and `EXCLUDE_RECONCILED_SOURCE` stops
|
||||||
|
* them counting.
|
||||||
|
*
|
||||||
|
* Refusals, all deliberate:
|
||||||
|
* - a statement row: that is the bank's record of a posted charge, and the
|
||||||
|
* statement must keep summing to its header (`transactions_void_statementless`
|
||||||
|
* enforces this in the database too). A wrong statement row is a supersede.
|
||||||
|
* - a row already reconciled or superseded: it is already out of every figure,
|
||||||
|
* and a reconciled one evidently DID post.
|
||||||
|
* - a row with a SETTLED split: someone has already paid their share of it.
|
||||||
|
* Voiding would silently turn that payment into a credit on their balance;
|
||||||
|
* that needs a person to look at it, not a side effect.
|
||||||
|
*/
|
||||||
|
export async function voidTransaction(id: number, reason: string): Promise<void> {
|
||||||
|
const why = reason.trim();
|
||||||
|
if (!why) throw new Error("A void needs a reason");
|
||||||
|
await prisma.$transaction(async (tx) => {
|
||||||
|
const rows = await tx.$queryRawUnsafe<
|
||||||
|
{ statement_id: number | null; reconciled_with_id: number | null; superseded_by_id: number | null; settled: number }[]
|
||||||
|
>(
|
||||||
|
`SELECT t.statement_id, t.reconciled_with_id, t.superseded_by_id,
|
||||||
|
(SELECT count(*)::int FROM transaction_splits ts
|
||||||
|
WHERE ts.transaction_id = t.id AND ts.settled) AS settled
|
||||||
|
FROM transactions t WHERE t.id = $1 FOR UPDATE`,
|
||||||
|
id
|
||||||
|
);
|
||||||
|
const row = rows[0];
|
||||||
|
if (!row) throw new Error(`Transaction ${id} not found`);
|
||||||
|
if (row.statement_id !== null) {
|
||||||
|
throw new Error(`Transaction ${id} is a statement line; a posted charge cannot be voided (supersede it instead)`);
|
||||||
|
}
|
||||||
|
if (row.reconciled_with_id !== null) {
|
||||||
|
throw new Error(`Transaction ${id} is reconciled with ${row.reconciled_with_id}, so it posted`);
|
||||||
|
}
|
||||||
|
if (row.superseded_by_id !== null) {
|
||||||
|
throw new Error(`Transaction ${id} is already superseded by ${row.superseded_by_id}`);
|
||||||
|
}
|
||||||
|
if (row.settled > 0) {
|
||||||
|
throw new Error(`Transaction ${id} has ${row.settled} settled split(s); resolve those before voiding`);
|
||||||
|
}
|
||||||
|
await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE transactions SET voided_at = now(), void_reason = $1 WHERE id = $2 AND voided_at IS NULL`,
|
||||||
|
why,
|
||||||
|
id
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Undo `voidTransaction` -- e.g. the "released" hold posted after all. */
|
||||||
|
export async function unvoidTransaction(id: number): Promise<void> {
|
||||||
|
await prisma.$executeRawUnsafe(
|
||||||
|
`UPDATE transactions SET voided_at = NULL, void_reason = NULL WHERE id = $1`,
|
||||||
|
id
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
/**
|
||||||
|
* Deterministic parsing of the bits of a wallet notification that have a fixed
|
||||||
|
* shape.
|
||||||
|
*
|
||||||
|
* Everything here was previously left to the interpreting model, which handled
|
||||||
|
* ONE fixed-width format three different ways on the same day's data: it kept
|
||||||
|
* the country code on "WYNDHAM VAL AU", stripped it on "Sydney" and "Perth",
|
||||||
|
* and swallowed "SAN FRANCISOUS" whole into the location field. A fixed-width
|
||||||
|
* field is not a judgement call, and the same argument that put the order
|
||||||
|
* parsers in code rather than in a Gemini call applies: a regex fails loudly on
|
||||||
|
* a template change, where a model returns a plausible wrong answer.
|
||||||
|
*
|
||||||
|
* The model still decides what the amount and merchant ARE -- it is better at
|
||||||
|
* that than a pattern, because five cards emit three shapes. This only handles
|
||||||
|
* the parts that are positional.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Westpac writes the merchant/city/country block as a fixed-width statement
|
||||||
|
* descriptor: 27-char merchant, 12-char city, 2-char ISO country. Verified
|
||||||
|
* across every Westpac event captured:
|
||||||
|
*
|
||||||
|
* Breadworld Bakery Cafe WYNDHAM VAL AU
|
||||||
|
* PLA DONATION MELBOURNE AU
|
||||||
|
* SP PEAKDESIGN SAN FRANCISOUS <- "SAN FRANCISO" + "US"
|
||||||
|
*
|
||||||
|
* The last example is why this is positional rather than a split on whitespace:
|
||||||
|
* a 12-character city fills its field exactly and runs straight into the
|
||||||
|
* country code with no separator.
|
||||||
|
*/
|
||||||
|
export type WestpacDescriptor = {
|
||||||
|
merchant: string | null;
|
||||||
|
city: string | null;
|
||||||
|
/** ISO-3166 alpha-2, upper-case. Null when the tail is not a country code. */
|
||||||
|
country: string | null;
|
||||||
|
};
|
||||||
|
|
||||||
|
const MERCHANT_WIDTH = 27;
|
||||||
|
const CITY_WIDTH = 12;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `body` is the whole notification body, e.g.
|
||||||
|
* "$13.50 paid from Mastercard ending...8032. Breadworld Bakery Cafe WYNDHAM VAL AU"
|
||||||
|
* Returns nulls rather than throwing: a template change must degrade to "we
|
||||||
|
* don't know", never to a wrong country on a foreign transaction.
|
||||||
|
*/
|
||||||
|
export function parseWestpacDescriptor(body: string | null | undefined): WestpacDescriptor {
|
||||||
|
const none: WestpacDescriptor = { merchant: null, city: null, country: null };
|
||||||
|
if (!body) return none;
|
||||||
|
|
||||||
|
// The descriptor is whatever follows the first sentence ("... ending...8032. ").
|
||||||
|
// [\s\S] rather than the `s` flag: the tsconfig target predates es2018.
|
||||||
|
const m = body.match(/^[\s\S]*?\.\s([\s\S]*)$/);
|
||||||
|
if (!m) return none;
|
||||||
|
const tail = m[1].replace(/\s+$/, "");
|
||||||
|
if (!tail) return none;
|
||||||
|
|
||||||
|
const merchant = tail.slice(0, MERCHANT_WIDTH).trim() || null;
|
||||||
|
const rest = tail.slice(MERCHANT_WIDTH);
|
||||||
|
if (!rest) return { merchant, city: null, country: null };
|
||||||
|
|
||||||
|
// Country is the final two characters of the fixed block, but only when the
|
||||||
|
// block is actually long enough to contain one -- a short tail is a merchant
|
||||||
|
// name with no location, not a merchant whose last two letters are a country.
|
||||||
|
const block = rest.slice(0, CITY_WIDTH + 2);
|
||||||
|
if (block.length < CITY_WIDTH + 2) {
|
||||||
|
return { merchant, city: block.trim() || null, country: null };
|
||||||
|
}
|
||||||
|
const city = block.slice(0, CITY_WIDTH).trim() || null;
|
||||||
|
const tailTwo = block.slice(CITY_WIDTH).trim().toUpperCase();
|
||||||
|
const country = /^[A-Z]{2}$/.test(tailTwo) ? tailTwo : null;
|
||||||
|
return { merchant, city, country: country };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The Apple Wallet notification's subtitle, which is comma-delimited and clean:
|
||||||
|
* "Breadworld Bakery Cafe, Wyndham Vale, VIC" -> location "Wyndham Vale, VIC"
|
||||||
|
* "Seatsaero" -> no location (online merchant)
|
||||||
|
*
|
||||||
|
* Only the first comma separates merchant from place; the rest is the place.
|
||||||
|
*/
|
||||||
|
export function parseWalletSubtitle(subtitle: string | null | undefined): {
|
||||||
|
merchant: string | null;
|
||||||
|
location: string | null;
|
||||||
|
} {
|
||||||
|
if (!subtitle || !subtitle.trim()) return { merchant: null, location: null };
|
||||||
|
const parts = subtitle.split(",").map((p) => p.trim()).filter(Boolean);
|
||||||
|
if (parts.length <= 1) return { merchant: parts[0] ?? null, location: null };
|
||||||
|
return { merchant: parts[0], location: parts.slice(1).join(", ") };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Currency marker, if the notification states one.
|
||||||
|
*
|
||||||
|
* Only Amex does: it writes "A$5,103.49". Westpac, CommBank and Apple Wallet
|
||||||
|
* write a bare "$", which is honestly recorded as unknown rather than assumed
|
||||||
|
* to be AUD -- 14 of 21 captured transaction rows. Domestically that assumption
|
||||||
|
* would be harmless; abroad it is a 23x error on a THB tap, so it is not made
|
||||||
|
* here. The statement settles it within weeks, and the reconcile queue holds
|
||||||
|
* the row until then.
|
||||||
|
*/
|
||||||
|
export function currencyFromAmountText(amountText: string | null | undefined): string | null {
|
||||||
|
if (!amountText) return null;
|
||||||
|
const t = amountText.trim();
|
||||||
|
if (/^A\$/i.test(t)) return "AUD";
|
||||||
|
if (/^(NZ\$)/i.test(t)) return "NZD";
|
||||||
|
if (/^US\$/i.test(t)) return "USD";
|
||||||
|
if (/^€/.test(t)) return "EUR";
|
||||||
|
if (/^£/.test(t)) return "GBP";
|
||||||
|
if (/^¥/.test(t)) return "JPY";
|
||||||
|
if (/^฿/.test(t)) return "THB";
|
||||||
|
if (/^S\$/i.test(t)) return "SGD";
|
||||||
|
if (/^₹/.test(t)) return "INR";
|
||||||
|
// CommBank's approval prompts write the ISO code after the figure
|
||||||
|
// ("672006 VND", "1340.00 INR"); some issuers put it first ("USD 10.00").
|
||||||
|
const iso = t.match(/^[\d.,\s]+([A-Z]{3})$/) ?? t.match(/^([A-Z]{3})\s*[\d.,]+$/);
|
||||||
|
if (iso) return iso[1];
|
||||||
|
// A bare "$" is genuinely ambiguous -- do not guess.
|
||||||
|
return null;
|
||||||
|
}
|
||||||
@@ -0,0 +1,481 @@
|
|||||||
|
import { queryRow, withTransaction, type Queryable } from "@/lib/db";
|
||||||
|
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
||||||
|
import { findWalletAdoptableTransaction } from "@/lib/auto-reconcile";
|
||||||
|
import {
|
||||||
|
parseWestpacDescriptor,
|
||||||
|
parseWalletSubtitle,
|
||||||
|
currencyFromAmountText,
|
||||||
|
} from "@/lib/wallet-descriptor";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wallet notifications -> transactions.
|
||||||
|
*
|
||||||
|
* A transaction is created for EVERY tap (DECISIONS.md ING-11, 2026-08-22),
|
||||||
|
* because overseas trips are where live spend matters most: the statement lag
|
||||||
|
* is the same 6-35 days abroad but knowing what you have spent matters far more
|
||||||
|
* in an unfamiliar currency.
|
||||||
|
*
|
||||||
|
* SEQUENCING IS THE WHOLE DESIGN. Wallet is the FIRST signal, not a late one --
|
||||||
|
* measured on txn 6858, the notification arrived 35h25m before anything created
|
||||||
|
* a transaction. So this creates the row, the receipt enriches it
|
||||||
|
* (receipt-ingestion.ts adopts it), and the statement confirms it via the
|
||||||
|
* reconcile queue, which a wallet row enters unaided: it is statementless with
|
||||||
|
* payment_method 'card'. Nothing here needs to know about statements.
|
||||||
|
*
|
||||||
|
* But that sequence is a TENDENCY, not a guarantee -- these first 21 rows were
|
||||||
|
* themselves backfilled days after the taps -- so this also adopts a row a
|
||||||
|
* receipt got to first (findWalletAdoptableTransaction), and a stored-value
|
||||||
|
* instrument is written as 'credits' so it never enters a queue no statement
|
||||||
|
* line will ever resolve.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type WalletEvent = {
|
||||||
|
/** capture.wallet_events.id -- the idempotency key. */
|
||||||
|
event_id: number;
|
||||||
|
/** 'transaction' | 'not_transaction' | 'unsure' from the interpreter. */
|
||||||
|
kind: string;
|
||||||
|
direction?: string | null;
|
||||||
|
amount: number | null;
|
||||||
|
/** The exact substring the model read the amount from, e.g. "A$5,103.49". */
|
||||||
|
amount_text?: string | null;
|
||||||
|
currency?: string | null;
|
||||||
|
merchant?: string | null;
|
||||||
|
card_tail?: string | null;
|
||||||
|
issuer?: string | null;
|
||||||
|
location?: string | null;
|
||||||
|
notif_app?: string | null;
|
||||||
|
/** The notification's title line, e.g. "Visa Gift Card" or "Payment". */
|
||||||
|
notif_title?: string | null;
|
||||||
|
notif_body?: string | null;
|
||||||
|
notif_subtitle?: string | null;
|
||||||
|
occurred_at?: string | null;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type WalletIngestResult = {
|
||||||
|
transactionId: number | null;
|
||||||
|
metadataId: number | null;
|
||||||
|
skipped?:
|
||||||
|
| "not_a_transaction" | "already_ingested" | "no_amount" | "merged_pair" | "adopted"
|
||||||
|
| "account_notice" | "stored_value_drawdown";
|
||||||
|
flags: string[];
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How far apart the two halves of ONE tap can arrive. Measured 2-8 seconds on
|
||||||
|
* every captured pair; ten minutes leaves room for a phone that was offline,
|
||||||
|
* and is still nowhere near the +/-1 DAY the matcher used to allow -- which
|
||||||
|
* folded the UNICEF $50 donation and an unrelated $50 bar charge 17 hours later
|
||||||
|
* into one row (event 275 into txn 8091, 2026-10-01).
|
||||||
|
*/
|
||||||
|
const PAIR_WINDOW_MINUTES = 10;
|
||||||
|
|
||||||
|
/** The ledger is dated the way the bank statements are: Melbourne local. */
|
||||||
|
const LEDGER_TZ = "Australia/Melbourne";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The calendar date of an instant, in LEDGER_TZ.
|
||||||
|
*
|
||||||
|
* Slicing the ISO string took whatever offset n8n serialised -- UTC in
|
||||||
|
* practice -- so a Melbourne-evening tap landed on the previous day (28 of the
|
||||||
|
* first 134 rows). Statements, which this row is reconciled against, date in
|
||||||
|
* Australian local time.
|
||||||
|
*/
|
||||||
|
export function ledgerDate(occurredAt: string | null | undefined, now: Date = new Date()): string {
|
||||||
|
const d = occurredAt ? new Date(occurredAt) : now;
|
||||||
|
const at = Number.isNaN(d.getTime()) ? now : d;
|
||||||
|
// en-CA formats as YYYY-MM-DD.
|
||||||
|
return new Intl.DateTimeFormat("en-CA", {
|
||||||
|
timeZone: LEDGER_TZ, year: "numeric", month: "2-digit", day: "2-digit",
|
||||||
|
}).format(at);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A ledger-derived AUD rate for a currency: the median of what the bank
|
||||||
|
* actually charged divided by the native figure, over statement rows.
|
||||||
|
*
|
||||||
|
* Same source order_transaction_bridge uses for its plausibility guard. This is
|
||||||
|
* only ever a PROVISIONAL value -- transactions.amount is NOT NULL and always
|
||||||
|
* AUD, and a notification cannot know the AUD figure because the bank sets the
|
||||||
|
* rate at settlement. The statement overwrites it at reconcile time.
|
||||||
|
* Returns null when there is not enough history to be worth anything.
|
||||||
|
*/
|
||||||
|
async function medianRateToAud(code: string): Promise<number | null> {
|
||||||
|
const row = await queryRow<{ rate: string | null }>(
|
||||||
|
`SELECT percentile_cont(0.5) WITHIN GROUP (
|
||||||
|
ORDER BY amount / NULLIF(foreign_currency_amount, 0)
|
||||||
|
)::text AS rate
|
||||||
|
FROM transactions
|
||||||
|
WHERE foreign_currency_code = $1
|
||||||
|
AND foreign_currency_amount > 0
|
||||||
|
AND superseded_by_id IS NULL
|
||||||
|
-- Statement rows only: a wallet row's amount is itself a provisional
|
||||||
|
-- guess, and one stored before this filter (txn 7316, 672006 VND at a
|
||||||
|
-- 1:1 "rate") would otherwise feed the next estimate.
|
||||||
|
AND statement_id IS NOT NULL
|
||||||
|
HAVING count(*) >= 5`,
|
||||||
|
[code]
|
||||||
|
);
|
||||||
|
const rate = row?.rate ? Number(row.rate) : null;
|
||||||
|
return rate && Number.isFinite(rate) && rate > 0 ? rate : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The ECB reference rate for the day, via Frankfurter (free, no key; the same
|
||||||
|
* source the statement processor converts with).
|
||||||
|
*
|
||||||
|
* The fallback when the ledger has no history for a currency. Hong Kong was the
|
||||||
|
* first trip to need it: no HKD statement row existed, so every HKD tap stored
|
||||||
|
* A$0.00 and the trip read as free until the statement landed weeks later. The
|
||||||
|
* ledger median is still preferred -- it carries the card's ~3% foreign fee,
|
||||||
|
* which the ECB mid-rate does not -- so this one is flagged as such.
|
||||||
|
*
|
||||||
|
* Any failure is null, never a throw: no rate is a state this lane already
|
||||||
|
* handles (a flagged 0), and a slow FX API must not lose the purchase.
|
||||||
|
*/
|
||||||
|
async function ecbRateToAud(code: string, date: string): Promise<number | null> {
|
||||||
|
try {
|
||||||
|
const res = await fetch(
|
||||||
|
`https://api.frankfurter.dev/v1/${date}?from=${encodeURIComponent(code)}&to=AUD`,
|
||||||
|
{ signal: AbortSignal.timeout(5000) }
|
||||||
|
);
|
||||||
|
if (!res.ok) return null;
|
||||||
|
const body = (await res.json()) as { rates?: { AUD?: number } };
|
||||||
|
const rate = body?.rates?.AUD;
|
||||||
|
return typeof rate === "number" && Number.isFinite(rate) && rate > 0 ? rate : null;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The other half of a paired notification, if it has already been ingested.
|
||||||
|
*
|
||||||
|
* One tap emits TWO notifications ~2-8s apart: the issuer app (which has the
|
||||||
|
* card tail) and Apple Wallet (which has the better merchant and the suburb).
|
||||||
|
* Neither alone is complete, so they must be MERGED, never deduped -- dropping
|
||||||
|
* either loses a field nothing else supplies. The notifications share no
|
||||||
|
* identifier, so a pair is recognised by what a pair IS: the same amount,
|
||||||
|
* minutes apart, from two different apps. Two notifications from one app are
|
||||||
|
* two charges, however close together.
|
||||||
|
*/
|
||||||
|
async function findPairedRow(
|
||||||
|
db: Queryable,
|
||||||
|
amount: number,
|
||||||
|
notifiedAt: string,
|
||||||
|
notifApp: string | null,
|
||||||
|
fx: { amount: number; code: string } | null = null
|
||||||
|
) {
|
||||||
|
// A foreign row pairs on its native pair: its `amount` is a provisional
|
||||||
|
// conversion (or a 0 placeholder), so two unrelated foreign taps could share it.
|
||||||
|
const rows = await db.queryRaw<{ id: number; meta_id: number }>(
|
||||||
|
`SELECT t.id, em.id AS meta_id
|
||||||
|
FROM transactions t
|
||||||
|
JOIN expense_metadata em ON em.transaction_id = t.id
|
||||||
|
WHERE t.source = 'wallet'
|
||||||
|
AND t.statement_id IS NULL
|
||||||
|
AND t.superseded_by_id IS NULL
|
||||||
|
AND CASE WHEN $4::text IS NULL
|
||||||
|
THEN t.amount = $1::numeric AND t.foreign_currency_code IS NULL
|
||||||
|
ELSE t.foreign_currency_code = $4::text
|
||||||
|
AND t.foreign_currency_amount = $5::numeric END
|
||||||
|
AND em.notified_at BETWEEN $2::timestamptz - make_interval(mins => $3::int)
|
||||||
|
AND $2::timestamptz + make_interval(mins => $3::int)
|
||||||
|
AND em.notif_app IS DISTINCT FROM $6::text
|
||||||
|
LIMIT 2`,
|
||||||
|
[amount, notifiedAt, PAIR_WINDOW_MINUTES, fx?.code ?? null, fx?.amount ?? null, notifApp]
|
||||||
|
);
|
||||||
|
return rows.length === 1 ? rows[0] : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A stored-value instrument: a gift card, a prepaid travel card, a store card.
|
||||||
|
*
|
||||||
|
* It matters because payment_method decides queue membership. 'card' means "a
|
||||||
|
* statement line is coming"; findAutoMatches and the reconcile queue both
|
||||||
|
* exclude 'credits' and 'cash' for exactly that reason. A $103.30 Woolworths
|
||||||
|
* tap on a Visa gift card written as 'card' parks in the queue forever, being
|
||||||
|
* offered as an unmatched row against statements that can never contain it --
|
||||||
|
* three of the first 21 wallet rows (events 80, 92, 101) were gift-card taps.
|
||||||
|
*
|
||||||
|
* The evidence is the notification TITLE, which is where Apple Wallet prints
|
||||||
|
* the instrument's own name ("Visa Gift Card"); the interpreter reduces that to
|
||||||
|
* issuer "Visa", which a plain Visa debit card also produces, so the title is
|
||||||
|
* the only place the distinction survives. Read as a prior, not a contract: a
|
||||||
|
* miss leaves today's behaviour, which is the queue.
|
||||||
|
*/
|
||||||
|
const STORED_VALUE = /gift\s*card|prepaid|travel\s*(?:money|card)|store\s*card/i;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A transit card reporting a drawdown of its own balance ("HKD 7.40 for Train"
|
||||||
|
* / "Your new balance is HKD 169.50.").
|
||||||
|
*
|
||||||
|
* Unlike a gift card, a transit card is loaded FROM a bank card -- that load is
|
||||||
|
* on the statement and is the spend. Each fare is that money being used, so
|
||||||
|
* booking it again counts it twice. The interpreter already called 11 of 13
|
||||||
|
* Octopus fares not_transaction; the two it did not (events 191, 200) became
|
||||||
|
* rows, so the outcome depended on the model's mood. This makes it a rule. The
|
||||||
|
* body wording is the signal because it is what every such card says; the
|
||||||
|
* issuer name is not (Octopus, Opal, myki, Suica each name themselves).
|
||||||
|
*/
|
||||||
|
const BALANCE_DRAWDOWN = /\byour (?:new|current) balance is\b/i;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Account notices that state an amount without money moving: "Your Smart Access
|
||||||
|
* 1910 is overdrawn by $21.21. To avoid fees and interest, ..." (event 235)
|
||||||
|
* became a $21.21 purchase named "To avoid fees and interest,". The model is
|
||||||
|
* right that the text names an amount; it is not a payment.
|
||||||
|
*/
|
||||||
|
const ACCOUNT_NOTICE = /\boverdrawn\b|\bto avoid fees\b|\bavailable balance\b|\blow balance\b/i;
|
||||||
|
|
||||||
|
/** How an adopted or merged row records which wallet event folded into it. */
|
||||||
|
const adoptedFlag = (eventId: number) => `wallet_evt:${eventId}`;
|
||||||
|
|
||||||
|
export function isStoredValueInstrument(ev: Pick<WalletEvent, "notif_title" | "issuer" | "notif_app">): boolean {
|
||||||
|
return [ev.notif_title, ev.issuer, ev.notif_app].some((v) => !!v && STORED_VALUE.test(v));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Already landed: a row this lane wrote, or one it adopted or merged into. */
|
||||||
|
async function alreadyIngested(db: Pick<Queryable, "queryRow">, eventId: number) {
|
||||||
|
return db.queryRow<{ id: number }>(
|
||||||
|
`SELECT id FROM transactions WHERE source = 'wallet' AND source_ref = $1
|
||||||
|
UNION ALL
|
||||||
|
SELECT em.transaction_id AS id FROM expense_metadata em
|
||||||
|
JOIN transactions t ON t.id = em.transaction_id
|
||||||
|
WHERE em.flags @> $2::jsonb AND t.superseded_by_id IS NULL
|
||||||
|
LIMIT 1`,
|
||||||
|
[`evt:${eventId}`, JSON.stringify([adoptedFlag(eventId)])]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function processWalletIngestion(ev: WalletEvent): Promise<WalletIngestResult> {
|
||||||
|
const flags: string[] = [];
|
||||||
|
|
||||||
|
// An open-world classifier feeding a closed-world writer drops the unknown
|
||||||
|
// and calls it a refusal (ING-14). `unsure` is a real answer and must not
|
||||||
|
// silently become a transaction.
|
||||||
|
if (ev.kind !== "transaction") {
|
||||||
|
return { transactionId: null, metadataId: null, skipped: "not_a_transaction", flags };
|
||||||
|
}
|
||||||
|
if (ev.amount === null || !Number.isFinite(ev.amount) || ev.amount <= 0) {
|
||||||
|
return { transactionId: null, metadataId: null, skipped: "no_amount", flags };
|
||||||
|
}
|
||||||
|
const text = [ev.notif_subtitle, ev.notif_body].filter(Boolean).join("\n");
|
||||||
|
if (ACCOUNT_NOTICE.test(text)) {
|
||||||
|
return { transactionId: null, metadataId: null, skipped: "account_notice", flags };
|
||||||
|
}
|
||||||
|
if (BALANCE_DRAWDOWN.test(ev.notif_body ?? "")) {
|
||||||
|
return { transactionId: null, metadataId: null, skipped: "stored_value_drawdown", flags };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Two places this event may already have landed: a row this lane wrote (which
|
||||||
|
// carries the ref natively), or a row it ADOPTED or MERGED into, whose
|
||||||
|
// source_ref belongs to whoever created it -- marked in the metadata flags
|
||||||
|
// rather than by overwriting a field another lane owns. Both must be checked,
|
||||||
|
// because bumping the interpreter's prompt_version deliberately re-interprets
|
||||||
|
// and re-posts every event ever captured -- that is the replay mechanism, and
|
||||||
|
// it must not mint a second row for a purchase that has one. Checked here
|
||||||
|
// without the lock to skip the FX lookup on a replay, and again under it.
|
||||||
|
const seen = await alreadyIngested({ queryRow }, ev.event_id);
|
||||||
|
if (seen) return { transactionId: seen.id, metadataId: null, skipped: "already_ingested", flags };
|
||||||
|
|
||||||
|
const sourceRef = `evt:${ev.event_id}`;
|
||||||
|
const notifiedAt = ev.occurred_at ?? new Date().toISOString();
|
||||||
|
const date = ledgerDate(ev.occurred_at);
|
||||||
|
|
||||||
|
// Deterministic parsing of the positional fields. The model decides what the
|
||||||
|
// amount and merchant ARE; it does not get to parse a fixed-width block.
|
||||||
|
// Only Westpac writes that block: run over a CommBank overdraft notice it
|
||||||
|
// produced the merchant "To avoid fees and interest," (event 235).
|
||||||
|
const desc = /westpac/i.test(ev.notif_app ?? "")
|
||||||
|
? parseWestpacDescriptor(ev.notif_body)
|
||||||
|
: { merchant: null, city: null, country: null };
|
||||||
|
const sub = parseWalletSubtitle(ev.notif_subtitle);
|
||||||
|
const merchant = sub.merchant ?? ev.merchant ?? desc.merchant ?? "Card purchase";
|
||||||
|
// Two sources for a place, and they are not equal. The Apple Wallet subtitle
|
||||||
|
// gives "Wyndham Vale, VIC"; the issuer's fixed-width descriptor gives
|
||||||
|
// "WYNDHAM VAL AU" -- a 12-character truncation in caps. The subtitle wins
|
||||||
|
// outright, which matters for the pair merge below: whichever half arrives
|
||||||
|
// second must be able to REPLACE a descriptor-derived place, or the result
|
||||||
|
// depends on arrival order.
|
||||||
|
const subtitleLocation = sub.location ?? null;
|
||||||
|
// Precedence: the Wallet subtitle, then the DETERMINISTIC descriptor parse,
|
||||||
|
// and only then the model's own answer. Getting this order wrong defeats the
|
||||||
|
// point of parsing positionally at all -- the model returned the mangled
|
||||||
|
// "SAN FRANCISOUS" and the padded "MELBOURNE AU" for exactly these rows,
|
||||||
|
// which is what wallet-descriptor.ts exists to correct.
|
||||||
|
const descriptorLocation = desc.city && desc.country
|
||||||
|
? `${desc.city} ${desc.country}`
|
||||||
|
: desc.city;
|
||||||
|
const location = subtitleLocation ?? descriptorLocation ?? ev.location ?? null;
|
||||||
|
|
||||||
|
// Currency: only ever what the notification actually states. A bare "$" is
|
||||||
|
// recorded as unknown, never assumed AUD -- harmless domestically, a 23x
|
||||||
|
// error on a THB tap abroad.
|
||||||
|
const stated = currencyFromAmountText(ev.amount_text)
|
||||||
|
?? (ev.currency && /^[A-Z]{3}$/.test(ev.currency) ? ev.currency : null);
|
||||||
|
if (!stated) flags.push("currency_unverified");
|
||||||
|
if (desc.country && desc.country !== "AU") flags.push(`foreign_country_${desc.country}`);
|
||||||
|
|
||||||
|
// Foreign: the native figure is what both sides can agree on exactly, so it
|
||||||
|
// is what gets stored. `amount` still needs a value (NOT NULL) and takes a
|
||||||
|
// provisional conversion, flagged, which the statement overwrites. Resolved
|
||||||
|
// BEFORE the lock: the ECB fallback is a network call, and the lock
|
||||||
|
// serialises every wallet post behind whatever runs inside it.
|
||||||
|
let amountAud = ev.amount;
|
||||||
|
let fxAmount: number | null = null;
|
||||||
|
let fxCode: string | null = null;
|
||||||
|
if (stated && stated !== "AUD") {
|
||||||
|
fxAmount = ev.amount;
|
||||||
|
fxCode = stated;
|
||||||
|
const ledgerRate = await medianRateToAud(stated);
|
||||||
|
const rate = ledgerRate ?? (await ecbRateToAud(stated, date));
|
||||||
|
if (rate) {
|
||||||
|
amountAud = Number((ev.amount * rate).toFixed(2));
|
||||||
|
flags.push("aud_provisional");
|
||||||
|
if (!ledgerRate) flags.push("rate_ecb");
|
||||||
|
} else {
|
||||||
|
// No rate from either source. The native figure must NOT fall through as
|
||||||
|
// AUD -- 672006 VND once landed as A$672,006. Zero is a flagged
|
||||||
|
// placeholder; the statement supplies the real AUD at reconcile time,
|
||||||
|
// which matches on the native pair, not on this column.
|
||||||
|
amountAud = 0;
|
||||||
|
flags.push("aud_provisional", "no_rate_reference");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A gift card has no statement line coming, so it must not enter the
|
||||||
|
// reconcile queue (which filters on exactly this column).
|
||||||
|
const storedValue = isStoredValueInstrument(ev);
|
||||||
|
const paymentMethod = storedValue ? "credits" : "card";
|
||||||
|
if (storedValue) flags.push("stored_value_instrument");
|
||||||
|
|
||||||
|
// Everything from the idempotency re-check to the INSERT holds one lock.
|
||||||
|
// n8n's HTTP node posts a poll's items CONCURRENTLY, and both halves of a tap
|
||||||
|
// usually land in the same 15-minute poll -- so each ran findPairedRow before
|
||||||
|
// the other had inserted, neither found its partner, and one tap became two
|
||||||
|
// rows: 11 pairs, A$3,841.40, every pair with an identical created_at. A
|
||||||
|
// single lock for the whole lane, not one per amount: wallet volume is a few
|
||||||
|
// dozen a day, and a coarse lock has no key to get wrong.
|
||||||
|
return withTransaction(async (db) => {
|
||||||
|
await db.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended('wallet-ingest', 0))::text`);
|
||||||
|
|
||||||
|
const existing = await alreadyIngested(db, ev.event_id);
|
||||||
|
if (existing) {
|
||||||
|
return { transactionId: existing.id, metadataId: null, skipped: "already_ingested" as const, flags };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Merge the paired notification rather than creating a second row for one
|
||||||
|
// tap. The issuer half brings the card tail; the Wallet half brings the
|
||||||
|
// merchant and suburb. COALESCE so whichever arrives second only ever fills
|
||||||
|
// gaps -- it can never blank a field the first half supplied.
|
||||||
|
const pair = await findPairedRow(
|
||||||
|
db, amountAud, notifiedAt, ev.notif_app ?? null,
|
||||||
|
fxCode && fxAmount !== null ? { amount: fxAmount, code: fxCode } : null
|
||||||
|
);
|
||||||
|
if (pair) {
|
||||||
|
await db.queryRaw(
|
||||||
|
`UPDATE transactions
|
||||||
|
SET merchant_name = COALESCE(NULLIF($2,'Card purchase'), merchant_name),
|
||||||
|
merchant_normalized = COALESCE(NULLIF($2,'Card purchase'), merchant_normalized),
|
||||||
|
-- A subtitle-derived place overwrites; anything else only fills a
|
||||||
|
-- gap. Without the $4 branch the merged row depends on which
|
||||||
|
-- notification happened to arrive first, and the issuer half
|
||||||
|
-- usually wins by a few seconds -- keeping the worse string.
|
||||||
|
location = CASE WHEN $4::boolean THEN COALESCE($3, location)
|
||||||
|
ELSE COALESCE(location, $3) END
|
||||||
|
WHERE id = $1`,
|
||||||
|
[pair.id, merchant, location, subtitleLocation !== null]
|
||||||
|
);
|
||||||
|
// The merged half records itself, or a replay finds nothing and merges
|
||||||
|
// it again (or, once the partner is superseded, mints a new row).
|
||||||
|
await db.queryRaw(
|
||||||
|
`UPDATE expense_metadata
|
||||||
|
SET card_last4 = COALESCE(card_last4, $2),
|
||||||
|
payment_method_detail = COALESCE(payment_method_detail, $3),
|
||||||
|
merchant_normalized = COALESCE(NULLIF($4,'Card purchase'), merchant_normalized),
|
||||||
|
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
|
||||||
|
WHERE id = $1`,
|
||||||
|
[pair.meta_id, ev.card_tail ?? null, ev.issuer ?? null, merchant, JSON.stringify([adoptedFlag(ev.event_id)])]
|
||||||
|
);
|
||||||
|
return { transactionId: pair.id, metadataId: pair.meta_id, skipped: "merged_pair" as const, flags };
|
||||||
|
}
|
||||||
|
|
||||||
|
// The receipt got here first. Fold into its row instead of writing a second
|
||||||
|
// one for the same purchase (see findWalletAdoptableTransaction). The column
|
||||||
|
// list IS the ownership boundary, the mirror of receipt-ingestion.ts's: the
|
||||||
|
// receipt read the register tape, so merchant, description, category and the
|
||||||
|
// items are its business and are absent here; `location` is wallet's own --
|
||||||
|
// nothing else writes a suburb -- and payment_method is filled only when the
|
||||||
|
// other lane left it unknown.
|
||||||
|
const adopted = await findWalletAdoptableTransaction(DEFAULT_OWNER_ID, amountAud, date, merchant, undefined, db);
|
||||||
|
if (adopted) {
|
||||||
|
await db.queryRaw(
|
||||||
|
`UPDATE transactions
|
||||||
|
SET location = COALESCE($2, location),
|
||||||
|
payment_method = COALESCE(payment_method, $3)
|
||||||
|
WHERE id = $1`,
|
||||||
|
[adopted.id, location, paymentMethod]
|
||||||
|
);
|
||||||
|
let metadataId = adopted.metadata_id;
|
||||||
|
if (metadataId) {
|
||||||
|
await db.queryRaw(
|
||||||
|
`UPDATE expense_metadata
|
||||||
|
SET card_last4 = COALESCE(card_last4, $2),
|
||||||
|
payment_method_detail = COALESCE(payment_method_detail, $3),
|
||||||
|
currency = COALESCE(currency, $4),
|
||||||
|
flags = COALESCE(flags, '[]'::jsonb) || $5::jsonb
|
||||||
|
WHERE id = $1`,
|
||||||
|
[metadataId, ev.card_tail ?? null, ev.issuer ?? null, stated, JSON.stringify([...flags, adoptedFlag(ev.event_id)])]
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
// expense_metadata.transaction_id is UNIQUE, so this branch only runs
|
||||||
|
// when the adopted row genuinely has none -- and it must produce one, or
|
||||||
|
// the card tail and the flag that makes this idempotent have nowhere to
|
||||||
|
// live.
|
||||||
|
const meta = await db.queryRow<{ id: number }>(
|
||||||
|
`INSERT INTO expense_metadata (
|
||||||
|
transaction_id, source, source_message_id, merchant_normalized, amount,
|
||||||
|
transaction_date, card_last4, currency, payment_method,
|
||||||
|
payment_method_detail, line_items, flags, extraction_model,
|
||||||
|
notified_at, notif_app
|
||||||
|
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
|
||||||
|
RETURNING id`,
|
||||||
|
[
|
||||||
|
adopted.id, sourceRef, merchant, ev.amount, date, ev.card_tail ?? null, stated,
|
||||||
|
paymentMethod, ev.issuer ?? null, JSON.stringify([...flags, adoptedFlag(ev.event_id)]),
|
||||||
|
notifiedAt, ev.notif_app ?? null,
|
||||||
|
]
|
||||||
|
);
|
||||||
|
metadataId = meta!.id;
|
||||||
|
}
|
||||||
|
return { transactionId: adopted.id, metadataId, skipped: "adopted" as const, flags };
|
||||||
|
}
|
||||||
|
|
||||||
|
const txn = await db.queryRow<{ id: number }>(
|
||||||
|
`INSERT INTO transactions (
|
||||||
|
transaction_date, description, amount, amount_aud, category, payment_method,
|
||||||
|
merchant_name, merchant_normalized, transaction_type, owner_id, location,
|
||||||
|
foreign_currency_amount, foreign_currency_code, source, source_ref, row_index
|
||||||
|
) VALUES ($1,$2,$3,$3,'other',$10,$4,$4,'debit',$5,$6,$7,$8,'wallet',$9,0)
|
||||||
|
RETURNING id`,
|
||||||
|
[date, merchant, amountAud, merchant, DEFAULT_OWNER_ID, location, fxAmount, fxCode, sourceRef, paymentMethod]
|
||||||
|
);
|
||||||
|
|
||||||
|
const meta = await db.queryRow<{ id: number }>(
|
||||||
|
`INSERT INTO expense_metadata (
|
||||||
|
transaction_id, source, source_message_id, merchant_normalized, amount,
|
||||||
|
transaction_date, card_last4, currency, payment_method,
|
||||||
|
payment_method_detail, line_items, flags, extraction_model,
|
||||||
|
notified_at, notif_app
|
||||||
|
) VALUES ($1,'wallet',$2,$3,$4,$5,$6,$7,$8,$9,'[]'::jsonb,$10::jsonb,'wallet-interpret',$11,$12)
|
||||||
|
RETURNING id`,
|
||||||
|
[
|
||||||
|
txn!.id, sourceRef, merchant, ev.amount, date,
|
||||||
|
ev.card_tail ?? null, stated, paymentMethod, ev.issuer ?? null, JSON.stringify(flags),
|
||||||
|
notifiedAt, ev.notif_app ?? null,
|
||||||
|
]
|
||||||
|
);
|
||||||
|
|
||||||
|
return { transactionId: txn!.id, metadataId: meta!.id, flags };
|
||||||
|
}, { maxWait: 10_000, timeout: 30_000 });
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user