Author SHA1 Message Date
siddharthd d88ca2a24c intake: a duplicate card that refuses a purchase must say what to do instead
ci / lint-test (push) Successful in 1m5s
The header (merchant, date, total) is a purchase's identity here, so a genuine
second identical purchase — two of the same coffee, one shop, one day — reads
as a re-share and is refused. That default is right; a re-share is far commoner
than a repeat. But the card said only "Nothing was changed", which is a dead
end: the money is real and the ledger silently never gets it. Say to add it in
the app.
2026-08-21 18:27:53 +10:00
siddharthd 91586ce4a4 test(intake): the second review pass, and CLAUDE.md for the rules it set
ci / lint-test (push) Successful in 45s
Six of the nine fixes have a test that fails against the previous tree: the
metadata row's currency, the Create-path hold, two concurrent batches booking
once, mrkdwn escaping including plain_text, the escaped currency, and
candidate-row currency.

Three do not, and the reason is the same each time — the window needs a crash
or a lost race that a test cannot force. Asserted instead: a stale metadataId
falls back to the live row rather than reporting a phantom write; two writes
racing to create a metadata row both resolve to one; the create path records a
baseline at all. All three passed before as well.
2026-08-21 18:23:52 +10:00
siddharthd 4b09d4136f intake: lock the purchase, not the batch, and hold on both write paths
Two different unresolved batches for one purchase could each create a
transaction. ON CONFLICT (source, source_ref) cannot stop it — their source_refs
differ — and both taps clear their match and duplicate checks before either
inserts. The create now takes a transaction-scoped advisory lock keyed on
(platform, merchant, date, total), re-checks inside it, and inserts; the loser
reports `duplicate` against the winner's transaction, the same answer the
sequential path gives.

The lock has to be inside a real transaction, which this app did not have.
queryRaw is $queryRawUnsafe on a POOLED client, so every call is its own
implicit transaction and pg_advisory_xact_lock taken in a statement of its own
is released before the next statement runs — the lock would have looked right
and held nothing. withTransaction() in db.ts is the scope; the integration
helper mirrors it with BEGIN/COMMIT on one pooled connection.

Also: the "hold rather than replace an itemised receipt with an unpriced
upload" guard lived only in routeFood, and createFromBatch's re-match walked
straight past it — the same upload destroyed the same receipt depending on
which button reached it. One function, both callers.
2026-08-21 18:23:51 +10:00
siddharthd 4f0f0dd285 intake: escape the currency and the button label too
money() interpolated the extracted currency raw, and parseExtraction accepts
any string for it — so "<@U123>" would have been a live mention on every card
the batch produced. And plain_text disables *formatting*, not entity decoding:
&, < and > are control characters everywhere in a Slack payload, so the Create
button's label is escaped like any other value we did not write. My earlier
note claiming the opposite was wrong.

Candidate rows also carry their transaction's currency (NATIVE_CURRENCY) rather
than always printing "$" under a header reading "USD 22.40".
2026-08-21 18:23:51 +10:00
siddharthd b788b4e8fa line-items: snapshot before writing, and stop reporting writes that did not happen
Three faults on the same function, all of them "the write is assumed to have
worked".

- The create path snapshotted the undo baseline AFTER its INSERT. A crash in
  between left the retry recording the intake's own write as "previous", so
  undo restored the thing it was meant to reverse — the exact failure
  previous_line_items exists to prevent. Both paths now snapshot first.
- An UPDATE against a row deleted between the select and the write affects
  nothing and still returned a metadata_id, so the batch was marked enriched
  against a row nobody had written to. The write is now checked and falls
  through to the ordered lookup, then to a create.
- Two calls for one transaction could both find no row and both INSERT;
  expense_metadata.transaction_id is UNIQUE, so one lost with a constraint
  error that reached the user as an error card. ON CONFLICT (transaction_id)
  DO UPDATE, with (xmax = 0) so created_metadata stays truthful.

And the new row's currency comes from the batch instead of a hard-coded 'AUD'.
OrderDetails prefers expense_metadata.currency, so a USD order — booked
correctly on the transaction with foreign_currency_code — printed "$".
2026-08-21 18:23:32 +10:00
siddharthd b5569b8ccc schema: previous_line_items on the model; say what the finish_reason rule is
Migration 0034 has the column and the Prisma model did not, so a regenerated
client and the database disagreed.

The extraction comment claimed only a finished reply is accepted while the code
accepts an absent finish_reason. The code is right — failing every call on a
provider that omits an optional field would be a certain outage in exchange for
a case that also requires syntactically complete JSON, and the harm a truncated
reply does is closed on the write side. The prose now states the rule as it is:
a reason that is PRESENT must be `stop`.
2026-08-21 18:23:32 +10:00
siddharthd d4ff68363a expense_metadata: every reader resolves the row a write lands on
getTransactions' provenance LATERAL, merchantForTransaction and the Slack
details modal all select on transaction_id = $1 OR matched_transaction_id = $1
with a bare LIMIT 1, so they could read a different row than replaceLineItems
writes. Same ORDER BY as the match, the write and the order-detail GET.

Defensive, not a live bug: measured 2026-08-21, expense_metadata holds 667
rows, 147 of them pointing-only, and ZERO transactions resolve to more than one
row by either link. This changes nothing today. It is here so the readers still
agree on the day multiplicity appears, which is the only condition under which
any of them was ever wrong.
2026-08-21 18:23:31 +10:00
siddharthd 631e75e58f test(intake): cover the review findings, and CLAUDE.md for the rules they set
ci / lint-test (push) Successful in 59s
Unit: word-boundary merchant matching (Bigals Pizza is not an IGA), mrkdwn
escaping vs the plain_text button label, a foreign total on a card, and the
finish_reason gate.

Integration: the NULL-platform wildcard on both the food and grocery paths,
candidates crossing platforms, EXCLUDE_RECONCILED_SOURCE, the pinned metadata
row, the panel reading the row the write landed on, the once-per-batch undo
baseline, the priceless-upload hold, Create re-matching, two pending cards
booking once, a pantry-resolved sibling, and a USD receipt.

Eleven of the thirteen fail against the pre-fix tree; the exceptions are the
no-batch-id baseline guard (a no-change assertion) and the concurrent-tap one,
which is a race and passed either way.
2026-08-21 17:53:06 +10:00
siddharthd c25db5211b intake: a NULL platform is "no platform", and Create re-checks before it inserts
Six fixes in the router, all of them cases where a stale or over-wide answer
was written rather than re-asked.

- ($3 IS NULL OR em.platform = $3) made an in-store receipt a wildcard: a till
  receipt from the Coles tap matched the Uber Eats Coles order for the same
  day and money, and overwrote its items. In-store now matches only rows with
  no platform of their own.
- findCandidates loses its platform filter entirely. That card is a human
  choice, shown precisely because the exact key missed — the platform is one
  of the things that may be wrong, so the list must widen there, not narrow.
- createFromBatch re-runs the match. The card is minutes old by the time it is
  tapped and the email/order bridge books orders on its own schedule; the
  match it was drawn from is not current.
- ...and checks for a resolved sibling batch. findDuplicateBatch ignores
  unresolved batches by design, so two pending cards for one order each booked
  it.
- INSERT ... ON CONFLICT (source, source_ref) DO NOTHING with a re-select,
  rather than SELECT-then-INSERT, which loses a double tap.
- A non-AUD receipt is booked as order-ingestion.ts books one: amount native,
  amount_aud NULL, foreign_currency_* set. No FX rate exists at intake.

Also: EXCLUDE_RECONCILED_SOURCE in both queries instead of a bare
superseded_by_id check — a reconciled manual twin at the exact key read as an
ambiguity, which is the same spend counted twice. And routeFood refuses to
replace an itemised receipt with an extraction carrying no prices at all: the
reconciliation check passes those (there is nothing to check), so without this
a priceless upload silently destroys a good record. It holds instead.

RouteContext.subtotal was written and never read; it is replaced by currency,
which the cards need.
2026-08-21 17:53:06 +10:00
siddharthd 9a442b7355 intake: escape mrkdwn, and stop printing every total as dollars
Merchant names, transaction descriptions and model output are none of ours:
Slack reads &, < and > as markup, so "Ben & Jerry's" and a description
carrying <http://…|text> rendered as something the card never meant to show.
esc() on every interpolated value in mrkdwn — never on our own literal *bold*,
and never on a button's plain_text label, where entities would print verbatim.

money() also hard-coded "$" while the order nudge already prints "USD 15.00".
The cards now carry the batch's currency.
2026-08-21 17:52:47 +10:00
siddharthd f9638cc5e5 line-items: one metadata row, agreed on by the write and the read
Three defects on one surface, all invisible while a transaction has exactly
one expense_metadata row — which is almost always.

1. ORDER BY (transaction_id = $1) DESC did the opposite of its comment.
   Postgres orders DESC as NULLS FIRST, and the row that merely POINTS at the
   transaction has a NULL transaction_id, so the clause written to prefer the
   owned row preferred the other one — in exactly the case it exists for.
   matchTransaction already spelled out NULLS LAST; now so do this and
   GET /api/transactions/[id]/order, which had no ORDER BY at all and could
   read a different row from the one an enrichment had just written.

2. replaceLineItems takes an optional metadataId. The match resolves a row
   through a platform-filtered join; re-picking independently here can land
   the write on a row the match ruled out.

3. previous_line_items on intake_batches is the undo baseline, recorded once
   per batch under COALESCE and read back. Returning whatever was just read
   hands a pantry retry its own write as the "previous" state, so undo
   restores the thing it undid. Migration 0034 is unapplied anywhere, so the
   column is added there rather than in a 0035.
2026-08-21 17:52:47 +10:00
siddharthd 7424d13858 intake: only a reply that finished is a reply the schema constrained
liteLLMExtractor rejected finish_reason "length" and accepted everything else,
so a content_filter or tool_calls stop was parsed as if the strict json_schema
had held. Reject any reason present and not "stop", naming it.
2026-08-21 17:52:25 +10:00
siddharthd be3d10d4f1 intake: match merchants on whole words, not substrings
merchantKey("Bigals Pizza") contains "iga", so GROCERY_PATTERN filed a pizza
shop as a grocer and merchantMatches matched it against every IGA on the same
date and total. Short names in a substring list collide with ordinary words,
and the collision reads as a correct classification rather than a bug.

GROCERY_PATTERN is now a word-boundary alternation; merchantMatches asks
whether one name's words appear as a contiguous run inside the other's, which
keeps Coles <-> Coles (Wyndham Vale) and Order - Coles (Wyndham Vale).

Also collapses the third PLATFORM_LABEL copy: order-ingestion's was exported
and read nowhere, order-details.tsx had its own, and the intake one — the only
live copy — was missing `uber`. One map, in intake-rules.
2026-08-21 17:52:25 +10:00
siddharthd a4e5f540cc docs: intake router routes, env vars, migration 0034
ci / lint-test (push) Successful in 48s
2026-08-21 12:10:06 +10:00
siddharthd 32a47bf8ee test(intake): rules, PDF text, C2 idempotency, enrich-vs-candidates, pantry hand-off, Slack taps
Unit: merchant pattern, header validation, reconciliation tolerance, verbatim
line items, one-call message shape, card action ids; unpdf over the two real
Uber Eats exports (Coles $85.74 / 15 lines, Burgertory $22.40 / Michael +
Joel). Integration: fetch stubbed at the LiteLLM and pantry boundaries so the
PDF path, persistence, match and writes run for real against personal_test.
intake-live.test.ts is opt-in (INTAKE_LIVE_TEST=1) and was run green against
cloud-budget on both PDFs.
2026-08-21 12:10:06 +10:00
siddharthd c7ea516c34 slack: dispatch the four intake taps on action_id (plan §11 C5)
intake_use_candidate / intake_create / intake_grocery / intake_meal carry JSON
{batch_id, ...} rather than <id>:<verb>, so they branch before the order-nudge
parse. They act on the persisted batch: enrich the chosen transaction, create
ING-10-style (statement_id NULL, owner_id from shared_by, source/source_ref
make a second press a no-op), or remember the merchant's kind and route. A tap
on an already-resolved batch returns its current card and writes nothing. Card
updates go to response_url, as the existing handlers do.
2026-08-21 12:10:05 +10:00
siddharthd db5c9a02a7 intake: POST /api/intake router + PUT /api/transactions/[id]/line-items (plan §11 C1, C2)
One upload point, every applicable step. Persist the batch, classify each
file, ONE LiteLLM call (cloud-budget, temperature 0, strict json_schema; PDFs
as unpdf text, never pixels) with an order-item prompt that keeps names
verbatim, header (merchant, date, total), merchant pattern
(coles|woolworths|woolies|aldi|iga|foodworks|costco) -> grocery | food | ask,
exact (platform, merchant, date, total) match against transactions +
expense_metadata, then act:

- food + unique match  -> replace line_items via the C2 path, 'enriched'
- food + miss/ambiguity -> 'candidates' (same merchant ±3d) with
  intake_use_candidate / intake_create buttons; never auto-creates
- grocery              -> original files to pantry POST /api/capture with
  finance_target_transaction_id + finance_action enrich|create (C3);
  409 -> 'duplicate'
- ask / needs_header / held / duplicate / unsupported as replies, not writes

Reconciliation is against the printed item subtotal (a delivery total is
items + fees - promotions: 18.94 of Coles lines settled at 5.74),  or
2%, else 'held'. Originals are kept under INTAKE_FILE_DIR by sha256 so a later
'Groceries' tap can still forward them.

C2 is an idempotent replace on transaction_id = :id OR
matched_transaction_id = :id, accepts INTAKE_TOKEN or RECEIPT_INGEST_TOKEN,
creates the expense_metadata row when absent, 404 when the transaction does
not exist, returns {previous_line_items, line_items} so pantry's undo is the
same call with the previous list.
2026-08-21 12:10:05 +10:00
siddharthd a538376638 db(0034): intake_batches + intake_merchant_kinds for the intake router
One row per upload through POST /api/intake, written BEFORE extraction so a
Slack tap minutes later has something to act on and a re-share (same sha256
set, or the same (platform, merchant, date, total) header already resolved)
is detected. Columns are the §11 C1 contract in smarthome
plans/intake-router-slice-plan.md. intake_merchant_kinds holds the remembered
'Groceries or a meal?' answer per normalised merchant (plan §2, Codex #6).

Not applied to production — finance has no migrate service; apply by hand at
deploy, then npm run test:setup.
2026-08-21 12:10:05 +10:00
siddharthd b958d05287 test(db): resolve the integration DSN at run time instead of storing an IP (#161)
ci / lint-test (push) Successful in 1m5s
setup-test-db.sh wrote .env.test once with the Postgres container's bridge IP
baked in, and Docker reassigns those on recreation. The file rotted silently:
its stored 172.22.0.18 belongs to **gitea** as of today, and an earlier drift
had the suite pointing its TRUNCATE at **postgres-pantry** — which it would
have wiped had the credentials happened to match.

Root-cause fix: stop persisting a host at all.

- scripts/resolve-test-db.sh re-derives the DSN from the live container and
  rewrites .env.test. Resolution order is TEST_DATABASE_URL, then the
  container's published host port, then the bridge IP. `docker port
  postgres-personal` reports 127.0.0.1:5432 since the 2026-08-17 PG17
  migration, so loopback is what gets used here — and a loopback address
  cannot be reassigned to another container, which is the whole point.
  (Note this supersedes the smarthome CLAUDE.md's "postgres-personal publishes
  no host port"; that is now stale.) The bridge-IP fallback is kept for a host
  where the port is not published, and the script never prints the password.
- Wired as `pretest:integration`, so npm runs it before every integration run.
  .env.test becomes a generated artifact rather than stored configuration.
- setup-test-db.sh now only creates the database and copies the schema, then
  delegates to the resolver.
- helpers.ts::assertTestDatabase stays and its message now points at the
  resolver rather than at `docker inspect`. It still earns its place: running
  vitest directly bypasses the pretest hook, and a stale .env.test would then
  still be read.

Docs: CLAUDE.md gains a Testing section carrying this, the prisma-generate
trap for a fresh checkout or worktree, and the frozen-clock rule for tests
whose subject is defined relative to "now". AGENTS.md updated for the same, and
for lint now being blocking.
2026-08-21 00:52:54 +10:00
siddharthd eaf5d5ac66 test(analytics): guard the three unguarded Priority 0 behaviours (#155)
docs/ui-information-architecture-review.md claimed Priority 0 was guarded by
analytics-sql.test.ts. That file only ever covered the three SQL *fragments*
(EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY, INVESTMENT_SIGNED). Items 2, 3 and
6 of the Priority 0 list had nothing, and all three fail silently — no error,
no empty screen, just a number that quietly disagrees with the one beside it.

New: src/__tests__/integration/analytics-priority0.test.ts, 18 tests.

- **Item 2, pace/headline parity.** /api/analytics/daily summed over a month
  must equal /api/analytics/monthly's headline to the cent, per category, and
  across every month in the window. Cents rather than exact equality is the
  assertion because the two routes group at different grains (month+category
  vs month+day+category) and both carry 4dp — the displayed grain is what has
  to agree, which is precisely the bug (the chart ending the month a few cents
  off the headline above it). Fixtures are only shapes where a naive gross
  debit sum disagrees: a 50% split, a refund, fee and interest rows, an
  itemised loan repayment, a reconciled source row, family-tagged spend,
  transfers/investment/income, and a row someone else paid that I hold a split
  on. A fourth test asserts the naive sum does NOT match — without it, parity
  between two queries identical by construction would pass on fixtures too
  plain to discriminate.
- **Item 3, fees date range.** Default 12 months with the window reported
  (from=2025-09-01, not 08-31 — toDateStr, not toISOString); the older tail
  excluded; months=0 all-time with a null range rather than a period claim the
  number does not make; a narrower months=3; the transaction-level fee lines
  bounded by the same window as the statement totals, since a bounded total
  beside an unbounded list is the same defect wearing the fix; and a nonsense
  months param clamped rather than producing an Invalid Date window.
- **Item 6, partial-month comparisons.** The in-progress month never enters a
  baseline; a partial month is compared through the same day of each prior
  month; a complete month keeps full totals; top movers truncate both sides;
  year boundary and leap February.

Item 6 was untestable by construction, which is why it had no test: the rules
were closures in budget/page.tsx reading `new Date()` inline. Extracted to
src/lib/partial-month.ts with an injectable clock, and budget/page.tsx now
imports them. The extraction also de-duplicates the baseline filter, which was
written out identically in two places.

The file freezes the clock (vi.useFakeTimers, toFake: ["Date"] only — pg needs
real timers for connection handling). Without it the fixtures drift out of the
routes' rolling windows over time and the assertions go vacuous, which is the
same class of silent failure the file exists to catch.

Mutation-checked, all three: breaking daily's spend semantics fails 3 of the
item-2 tests (the placebo correctly still passes); forcing the fees window to
all-time fails 5; letting the current month into the baseline fails 1; and
dropping the elapsed-day truncation fails 3.
2026-08-21 00:52:36 +10:00
siddharthd fc9ffaba25 chore(ci): clear the lint debt and make lint blocking (#79)
CI ran lint with continue-on-error because 23 errors predated it. All are
fixed, so the flag is gone and a typecheck step is added beside it — nothing
else in CI compiles the test files, so a type error in a fixture could only
ever be found on a developer's machine.

Real fixes, no blanket disables and no suppressions:

- react-hooks/set-state-in-effect (6): each was "reset or seed state when an
  input changes", so each becomes a state adjustment during render behind a
  previous-value guard — the documented React pattern for it. Two were also
  latent bugs. transactions/page.tsx depended on `others`, rebuilt by .filter()
  every render, so the effect already ran every render, one commit late.
  shared/page.tsx depended on the `tagIds` array identity, so the row window
  was reset more often than "changing any filter" claims — including after
  "Show more", the one interaction it must not undo; the key now joins by
  value. csv-import-modal reads localStorage once at mount, which is a lazy
  useState initializer, not an effect (SSR-guarded, though the modal is
  mounted from client state and never prerenders).
- react-hooks/static-components (3): SortIcon was declared inside SharedPage,
  making it a new component type on every render. Hoisted to module scope with
  the sort state passed as props.
- @typescript-eslint/no-explicit-any (9): the dynamically imported route
  handlers in two integration tests. Typed as their real signatures with one
  `as unknown as NextRequest` cast at the fixture boundary, so the call sites
  and response bodies get checked instead of opted out.
- react/no-unescaped-entities (5): exact-equivalent entities (&apos;, &quot;),
  so no rendered text changes.

Also fixes the one pre-existing tsc error (unit/order-ingestion.test.ts:511):
`options` is optional, and `?? 0` keeps the assertion failing on an absent
list rather than silencing it with a non-null bang.

9 warnings remain (no-unused-vars, exhaustive-deps). eslint exits 0 on
warnings, so lint blocks on errors only — deliberately, since none of the
remaining warnings should stop a push.
2026-08-21 00:52:14 +10:00
siddharthd 9f1be74330 transactions: let a gift card be a payment method you can pick
ci / lint-test (push) Successful in 43s
`credits` has been a valid payment_method since migration 0018, which added it
so the order pipeline could file gift-card-funded orders. It reached the
database, `needsCardMatch()` (which holds those rows out of reconciliation) and
`bankLabel()` (which renders them "Gift Card"). It never reached the two places
a human touches it.

So entering a gift-card purchase by hand was impossible: the dropdown stopped at
"Other", the row took the NULL default, and NULL is *reconcilable* — it joined
the pending queue to be offered card matches within 3 days and 1% that cannot
exist, because the balance already paid the bill. Accepting one is the
destructive case the cash exclusion documents: reconciled manual rows are
filtered out of every query, so the spend vanishes while an unrelated card row
claims to be it.

- `src/lib/payment-methods.ts` — one canonical list, mirroring the
  `statement-types.ts` pattern, plus the TS mirror of `needsCardMatch()`.
- `PATCH /api/transactions/[id]` — the `VALID_METHODS` literal omitted
  `credits`, so the API answered 400 for a value the CHECK constraint admits
  and 372 live rows already carry. It validates against the list now.
- `POST /api/transactions` — validated at all, so a bad value is a 400 naming
  the field rather than a 500 carrying a Postgres error.
- Add modal — offers every method, and explains the consequence when you pick
  one that suppresses reconciliation. Relabelled "Paid by" → "Payment method",
  which the edit modal uses for the *owner*; they are different questions.
- Edit modal — had no payment-method field at all, so an existing row wrongly
  in the queue could not be corrected. Manual rows only: the queue is
  `statement_id IS NULL`, so it decides nothing on a statement row.

The test asserts the list against `pg_get_constraintdef` and evaluates
`needsCardMatch()` as real SQL rather than re-implementing it, so the next
migration that widens the constraint fails the suite instead of drifting for
two months. Mutation-tested by dropping `credits`.

Unit 130 pass, integration 206 pass (the integration DB needed `npm run
test:setup` first — `.env.test` still named a stale container IP).
2026-08-17 18:14:58 +10:00
siddharthd 4f307e9517 docs: what importing a BNPL account changes, and the Zip data decisions
ci / lint-test (push) Successful in 41s
24 Zip statements landed (Dec 2020 - Jun 2026), all balance-asserting at
$0.00. Zip is a credit account, so three existing row types had to be
re-read. Data changes applied via transaction_overrides; this records why.

A ZIPPAY debit leaving the bank duplicates the Zip statement's purchases
only where a Zip statement covers that period. 2 of 8 do, and are now
transfers ($125.97). The other six stay spend: for those periods the bank
debit is the only record the purchase happened, so moving them would have
deleted $703.35 of real spend.

'Account Credit' is cashback, confirmed by the owner and consistent with
Zip labelling its other money-in types explicitly (Payment = repayment,
'Refund <merchant>' = merchant refund). 34 rows, $740, round amounts, no
matching bank debit. Gemini split the identical description 23 'other'
(subtracts from spend) / 11 'transfers' (excluded), so one thing did two
opposite things. Now all 34 shopping+credit, netting against spend the
way refunds already do. Income was rejected for the reason the Up item
sales note gives.

Prezzee gift cards ($1,580) stay spend — stored value only nets out if
every downstream purchase is logged, and for 2020 it is not. Same
reasoning as the ATM-withdrawal rule.

31 uncategorised debits mapped by merchant with merchant_normalized set
too, since the raw descriptions carry card numbers and order UUIDs. All
137 Zip rows are now categorised; $11,135.11 net spend, $10,085.11
transfers.

Flags statement 200 (Sept 2021): extracted without billing_end_date or
balances, so it is the one statement uq_statement_identity cannot cover
(the index is partial on billing_end_date IS NOT NULL) and a re-import
would duplicate it.
2026-08-15 19:11:50 +10:00
siddharthd 2cfaf8e8c8 statements: derive account identity, fix the new-bank alert (0033)
ci / lint-test (push) Successful in 42s
account_number fragments exactly like bank_name did, from the same cause:
ANZ's Access Advantage arrived as both '4085-56264' (4 statements) and
'408556264' (1). uq_statement_identity keyed on it, so re-importing one
period under the other spelling evaded the duplicate check.

account_number_key is GENERATED ALWAYS — the number with separators
stripped — and the unique index moves onto it. Derived rather than
rewritten because the raw number is the readable one and some of it is
structure: Up stores '633-123 / 176540052', a BSB and an account number,
and flattening it would lose a distinction a human reads at a glance to
fix a machine problem. The Amex mask and case are preserved; 'X' records
which digits were redacted.

Not the cause of the documented 31-row / $42,040.68 ANZ duplication —
107/142/143 overlap on different end dates, which that index cannot catch
at any spelling.

Adds statement_identity_drift: one account under two bank names, or one
bank under two spellings of one number. Non-empty means fragmentation the
normaliser could not know about. Currently one row, the ANZ pair above,
now unified by key.

Also documents the N8N regression this series caused. Check Known Bank
compared the raw Gemini name against the canonical column before the
insert, so nothing ever matched and every Zip statement was tagged
pending and held for Slack approval — a stall, not just a noisy alert.
Fixed in workflow FysADdFwEtwONQl4 by comparing like with like through
normalize_bank_name(), passing currency so a genuinely new national
entity still alerts. Verified live: 12s after vs 30-53s before.
2026-08-15 17:03:13 +10:00
siddharthd c3b623052d statements: keep one bank's national entities apart (migration 0032)
ci / lint-test (push) Successful in 44s
Fixes 0031. Every merge it made was provably one account — each split
pair shared an account_number — but the PREFIX generalises past that
evidence. Citibank Australia and Citibank India are different banks with
different accounts and currency, and `citi%` flattens them into one.
Older Citi India statements are queued to load, so this would have merged
them on arrival. Not Citi-specific: `hsbc%` and `%american express%`
carry the identical defect.

Currency alone cannot be the discriminator: Wise holds AUD, EUR and USD
accounts under one provider, so a blanket "non-AUD is a different bank"
rule would shatter it into three. Wise stays unmapped for exactly that
reason, now a documented property rather than an accident.

The country comes from the name when the name states it and from the
currency otherwise, with AUD as home taking no suffix — so nothing
renames: the backfill touched 0 rows.

  'Citibank'       + AUD -> 'Citibank'        (unchanged)
  'Citibank India' + INR -> 'Citibank India'
  'Citibank N.A.'  + INR -> 'Citibank India'  (currency carries it)
  'CITIBANK INDIA' + AUD -> 'Citibank India'  (name beats a wrong currency)

'N.A.' is deliberately not read as a country marker: it means "National
Association", a US legal form printed on Citibank letterhead worldwide,
including India. Reading it as US is precisely the mistake this fixes.

The trigger now fires on UPDATE OF bank_name, currency, since the country
is partly derived from the currency.
2026-08-15 16:52:14 +10:00
siddharthd 0bb4329249 statements: one institution, one bank name (migration 0031)
ci / lint-test (push) Successful in 46s
bank_name is a component of uq_statement_identity
(bank_name, account_number, billing_end_date), so two spellings of one
institution mean the same statement can be ingested twice without the
index ever firing. Gemini reads the name off whatever the PDF prints, so
the spelling varies per document, not per account.

Zip exposed it: account 2705256 arrived under four names in sixteen
minutes — ZipMoney Payments Pty Ltd, ZipMoney Payments Pty Limited,
Zip Pay, ZipPay — with ten more in the ingestion queue. Not unique:
17 bank_name values represented 13 institutions, every split pair sharing
an account number (Amex 14/1, Citibank/Citi 8/4, NAB 3/1). Fragmentation
also splits both bank filters and by_bank in /api/analytics/fees.

normalize_bank_name() + a BEFORE INSERT OR UPDATE trigger, mirroring the
0013 statement_type pattern with one deliberate difference: the
vocabulary is OPEN. An unrecognised name passes through tidied rather
than collapsing to a fallback, and there is no CHECK constraint — a bank
never used before must be able to arrive without a migration, and
destroying its name on first contact is worse than leaving it unmapped.

Matching is by prefix, not by enumerated spelling, so unseen variants
normalise with no code change. Branches for Westpac, ANZ, HSBC, ING, AMP,
Up and CommBank are no-ops that map to the spelling already in use, to
catch the legal-entity variant a future PDF might print. Wise is left
alone: one spelling is a rename, not a merge.

Backfill verified collision-free against uq_statement_identity first.
17 names -> 12. account_owner_mappings keys on (bank_name,
account_number) with its own UNIQUE constraint and is updated too; it is
empty today.
2026-08-15 16:45:04 +10:00
siddharthd 95d8544752 transactions: change who paid, on a row and on a statement
ci / lint-test (push) Successful in 46s
Owner was write-once for every ingestion path — a pantry receipt hardcodes
DEFAULT_OWNER_ID and there was not one `UPDATE ... SET owner_id` in src/ —
so a shop the other person paid for was permanently filed as yours.

PATCH /api/transactions/[id] now takes owner_id, for manual rows only. A
statement row returns 400 statement_owned and points at the statements
page: its effective owner is COALESCE(t.owner_id, s.owner_id), so writing
it there would either no-op or detach one row from the account it came
from.

PATCH /api/statements/[id] is new. The statements page has had an owner
dropdown since it was built, wired to a route with no PATCH handler —
every change 405'd, and because useUpdateStatement never checked res.ok
it failed silently and the select snapped back on refetch. It writes both
tables: 2,194 statement rows carry their own owner_id against 1,803 that
inherit, so updating `statements` alone moves less than half and splits
one account's history between two people.

The guard is the point. Access is "owner OR holds a split", so handing a
row over while holding no split removes it from your list and 404s every
route that could put it back — only the new owner can undo it. That is
409 would_lose_access, and the modal offers both ways forward: add my
split first, or give it away anyway. Taking a row onto your own ledger is
never blocked, and claiming a row you cannot see is a 404 before any
owner logic runs.

Splits are deliberately not rewritten. They record shares, not direction,
so a 50/50 flips from "they owe me" to "I owe them" untouched, settled
included.

Also adds the missing res.ok check to useUpdateTransaction, without which
every rejection resolved as success: the modal closed, the list
refetched, and the edit silently vanished.

14 new integration tests; 203 integration + 130 unit green.
2026-08-15 16:17:38 +10:00
siddharthd 6c14b493ef analytics: your share of what someone else paid is your spend
ci / lint-test (push) Successful in 1m25s
Every spend analytic gated on `OWNER_SCOPE = $1` and scaled by `mySplitOf`
*within* that gate, so ownership was a precondition for an expense being
yours. Half a grocery shop Sonu paid for counted as zero — in monthly,
daily, merchants, subscriptions, fees and the budget page. 167 rows /
$3,210.91 across Jan-Jul 2026, worst in April (+$1,354.83, the Europe
trips), while getParticipantBalances booked the matching debt correctly.
The app could say you owed her for a shop while insisting you had not
spent anything on it.

New MY_SPEND_SCOPE(): owner = me OR I hold a split. OWNER_SCOPE stays on
the things that measure an *account* rather than a person — the income
and investment lines, and the statement-level fee rollup.

myShare had to change with it, and widening the gate alone would have
been worse than the bug: its `100 - everyone else` fallback is the
payer's remainder, so on someone else's unsplit row it returns 100 and
moves their whole bill onto you. It now branches on ownership — my row
resolves as before; their row takes an explicit split row only, absent
meaning 0. That 0 is what makes the wider gate safe.

my_share_percent is deliberately not read on someone else's row: one
unscoped column, writable by anyone who can see the row, so "my" can
only mean the owner's. All 402 rows carrying one today are owner-side.

MY_SHARE_PCT mirrors myShare for the transactions list, which has no
viewer-scoped ts join; a test asserts the two agree across seven fixture
shapes.

No historical restatement — every non-owner split is 2026-dated, and the
1,266 pre-2026 SplitMyExpenses splits are all on rows you own.

Also fixes a latent failure in the NATIVE_CURRENCY test, which inserted a
statement relying on participant id 1 existing (owner_id is NOT NULL
DEFAULT 1 with an FK) and only passed when a sibling file had left one
behind. It now owns its fixture.

15 new integration tests; 189 integration + 130 unit green.
2026-08-15 15:37:32 +10:00
siddharthd 22c2349a47 docs: one CSV import path, and the statement-coverage rule that replaced the Frollo importer
ci / lint-test (push) Successful in 43s
2026-08-13 15:09:11 +10:00
siddharthd 3edcc27781 csv import: map a currency column, or foreign rows land as AUD
ci / lint-test (push) Successful in 41s
Deleting the Frollo importer dropped its currency handling and nothing replaced
it: ColumnMapping had no currency column, so applyMapping could never produce
foreign_currency_code and batchInsertCSVTransactions' support for it was
unreachable from the UI. The USD 10,782 salary imported as A$10,782 — about a
third under, sitting in a column of AUD figures looking entirely normal, which
is the same defect the owner spotted in the first place.

An optional Currency column now sets foreign_currency_code and
foreign_currency_amount when the cell is a three-letter code other than AUD, and
leaves amount_aud NULL rather than inventing a rate. That is the shape order
ingestion already uses and what AMOUNT_UNCONVERTED looks for, so the row renders
as its native figure with "no AUD rate" and the statement supplies the real
number when it arrives.

Caught by reading the imported row rather than the import summary: the summary
said 171 inserted and was right about everything it reported.
2026-08-13 15:08:50 +10:00
siddharthd 461c021e7a csv import: exclude what the statements already cover, and delete the Frollo importer
ci / lint-test (push) Successful in 43s
"This is becoming too complex... Frollo should be done through that [the manual
CSV import]" (owner). It was right: a bespoke importer, an API route, a CLI, two
scheduled n8n workflows and a shared secret existed to do what the CSV import
modal already did, minus one rule.

That rule is statement coverage, and it turns out to be the whole thing. Applying
each account's newest billing_end_date as a watermark takes the real 2,607-row
Frollo export down to 171 rows — with no account allowlist, no credit-card
exclusion and no Frollo-specific scoping at all. Cards drop out on their own
because their statements are current; the 46 card rows that survive are genuinely
post-statement. Every bit of the bespoke apparatus was doing by hand what one
query does generically.

Deleted: src/lib/frollo-csv.ts, src/lib/frollo-ingest.ts, scripts/import-frollo.mts,
src/app/api/frollo/, both test files, the FROLLO_INGEST_TOKEN wiring, and the n8n
Frollo Import + Frollo Freshness Check workflows.

Added to the shared CSV path, so every import benefits:

  - getStatementCoverage() + /api/import/statement-coverage. The review step
    leaves out rows an account's statements already cover and says how many, with
    the rows one click away. Only applies when an account column is mapped and
    that account has statements — a row is never dropped on a guess.
  - Optional Account and Row ID columns in the mapper. Account drives the
    watermark and is stored as source_account; Row ID becomes source_ref.
  - An in-file duplicate warning. A CDR re-consent re-exports history under
    fresh ids, so source_ref cannot see it — 385 twins in one 2,563-row export
    doubled every salary payment, and that is not visible by eye in a review
    table.

Two pre-existing bugs in that path, both of which this plan depends on:

  - The category chosen in the review step was accepted by
    batchInsertCSVTransactions and then left out of the INSERT column list, so it
    was silently discarded and the trigger wrote 'other'. Not cosmetic: an
    uncategorised credit is admitted by NET_SPEND_ROWS and negated by
    SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of spend while
    counting as no income.
  - The path had no idempotency whatsoever. row_index is assigned MAX+1 on every
    run, which makes uq_transaction_identity structurally unable to fire, so a
    second import of the same file duplicated all of it. Now writes source +
    source_ref with ON CONFLICT DO NOTHING.

awaitsStatementLine()'s removal note is kept but rewritten: it no longer points
at a deleted file, and the lesson stands — the queue jump from 8 to 558 was the
measurement, not the noise.
2026-08-13 12:58:06 +10:00
siddharthd b82c4570bd frollo: don't import what the statements already cover
ci / lint-test (push) Successful in 45s
"We should not be importing from Frollo what we already have from statements"
(owner). The amount+direction guard could not deliver that, because the two
sources decompose the same event differently: Frollo bundles the Wise fee into
the transfer (10001.13) where the statement itemises it (10000.00 + 1.13). So 38
Wise USD rows passed the amount guard as new while being the same money, and were
the reason the transactions view showed a USD figure where every neighbouring row
showed AUD.

A statement's billing_end_date is a hard watermark: everything on that account up
to that date is already in the ledger, itemised and converted. Guard 1 now drops
any feed row at or before its account's newest statement. Guard 2 (amount +
direction) stays as the net for accounts that have no statement at all.

Note this is the coverage test the first import needed and got wrong. That one
asked whether a row's date fell inside a statement's min-max window, which for
periods spanning 182 to 460 days swallows a year and answers nothing. The
watermark asks a question that has an answer: up to what date is this account
complete?

Matching is on last4, verified against the live statement set — the eight
in-scope accounts with statements each map to one bank, no cross-bank collision.
The watermarks are printed by the CLI so a wrong boundary is visible rather than
inferred from a row count.

Re-imported from empty: 425 covered by statement, 15 amount twins, 110 inserted.
Exactly one row now carries a foreign currency with no AUD figure — the
2026-08-12 HDR salary, which is the genuinely-new pre-statement row this feed
exists for. Was 39.
2026-08-13 12:36:08 +10:00
siddharthd afd75d3f09 transactions: never print a foreign amount as AUD
ci / lint-test (push) Successful in 48s
The amount cell rendered formatAmount(amount_aud ?? amount) with no currency
argument, so a row with no AUD figure fell back to its native amount and was
stamped with a dollar sign. A USD 10,782.00 Wise credit showed as "+$10,782.00"
in a column of AUD figures — the same payment from the statement, one row above,
correctly showed $15,518.53 over USD 10,782.00. So the feed row read as $10,782
AUD when the real value is about $15,500, and the sub-line just repeated the
same number correctly labelled.

amount_unconverted is already selected by the transactions query (queries.ts:298)
and was simply unused here. When set, the native figure becomes the headline and
the absence of a rate is stated rather than papered over.

Pre-existing rather than Frollo's: any row with amount_aud NULL and a foreign
currency hits it, including foreign order-receipt rows. Frollo made it visible
at scale — 39 rows.
2026-08-13 12:32:17 +10:00
siddharthd 2738213a23 docs: record the ledger-duplicate guard and why awaitsStatementLine is gone
ci / lint-test (push) Successful in 40s
2026-08-13 12:24:59 +10:00
siddharthd bb8a009e02 frollo: don't re-import what the ledger already has, and stop hiding the queue
ci / lint-test (push) Successful in 44s
The first import wrote 550 rows on 2026-08-13. 422 of them (77%) were second
copies of transactions the ledger already held from statements — $1,023,824.63
of movement counted twice. The owner found it by opening the transactions view
and seeing one HDR Global salary listed twice, once as A$15,518.53 from the
statement and once as US$10,782.00 from the feed.

The currency was never the defect. toLedgerRow already left amount_aud NULL and
named the currency in foreign_currency_code, which is the documented contract for
a row whose AUD value is unknown, and the transactions page labels it. What made
it look wrong was the duplicate sitting beside it.

Two changes.

Upstream, ingestFrolloCsv now drops rows the ledger already holds, matching on
amount + direction within LEDGER_MATCH_DAYS (3). Three things had to be right and
the first two were not, each caught only by rehearsing against real data rather
than fixtures:

  - pg returns a DATE as a JS Date while Prisma and the CSV give strings.
    String(date).slice(0,10) is "Wed Mar 10", which parses to NaN, so the first
    dry run reported 550 to insert and zero duplicates. dayMs() takes both.
  - Direction has to be in the key. This ledger is full of internal transfers
    between the owner's own accounts and the feed carries both legs: 2026-05-18
    has +3076.04 into ANZ and -3076.04 out of AMP. Matching on amount alone let
    the credit leg consume the ledger's debit row, so the real duplicate was
    written — 20 rows got in that way.
  - 'refund' is money in. The feed calls a reversed account fee a credit and the
    statement importer types it 'refund'; classifying it as an outflow left every
    ANZ servicing-fee reversal behind.

Downstream, awaitsStatementLine() is removed. Its premise — feed rows never await
a statement line — was asserted, never tested, and false for almost every
account. Worse is how it got there: the reconcile queue jumped 8 -> 558 when the
feed landed, that jump was read as noise and filtered away, and filtering it
removed the only mechanism that would ever have collapsed the duplicates. The
queue was right. A feed row IS a row awaiting its statement line.

Re-imported: 375 dropped as already-on-ledger, 175 inserted. Residual duplicates
4 rows / $15.01, all sub-$5 account fees where several identical amounts fall in
overlapping windows and greedy consumption picks the wrong one; not chased
further at this scale.

The CLI prints the already-on-ledger count even when zero — a number you have to
go looking for is a number nobody looks at.
2026-08-13 12:23:01 +10:00
siddharthd 3e826a317b CLAUDE.md: document the Frollo account feed and the new reconcile exclusion
ci / lint-test (push) Successful in 39s
Two things a future reader would otherwise have to reverse-engineer.

The reconcile queue now excludes account-feed rows for a reason unrelated to
payment method: a feed row is the account's own ledger entry, not a receipt
awaiting a statement line, because the importer only covers accounts whose
statements are deliberately not imported. Measured, the queue went from 8 to
558 without it.

And the de-duplication rule needs its history attached, because the obvious
version is wrong in a way that passes tests: keeping close-id rows as genuine
repeats survived 29 tests and a clean dry run while still doubling three
salary payments. In-scope duplicate pairs have id gaps from 58 to 260
million; real repeats sat at 1-4.
2026-08-13 11:29:08 +10:00
siddharthd 21e9e765a3 Add /api/frollo/ingest so the import can run unattended
ci / lint-test (push) Successful in 40s
Shares one module with the CLI rather than reimplementing the insert:
frollo-ingest.ts holds parsing, scoping, de-duplication and the write, and
both callers pass in their own SQL executor (Prisma in the route, a pg
client in the script). The alternative is two implementations of the same
insert, which is how the pantry healthcheck came to be fixed in one repo
and left broken in the other.

The route refuses rather than guesses. findAnomalies() returns every reason
an unattended run should stop - a configured account contributing no rows,
an unrecognised account, a near-consecutive-id collapse that might be a
real repeat, a batch over ~200 rows, or an export taken with pending
included - and the route answers 409 having written nothing.

Two defects the wiring surfaced. Deliberately excluded credit cards were
reported as unknown accounts, which would have raised the new-account
anomaly on every single run and left the automatic path permanently
refusing; EXCLUDED_ACCOUNTS now distinguishes excluded from unknown. And
pending was tested after account scope, so pending rows on cards - which is
all of them so far - classified as out-of-scope and the wrong-export-option
signal could never fire; pending is now tested first.
2026-08-13 11:19:27 +10:00
siddharthd 493ff6f631 Import Frollo account feeds for the accounts statements don't cover
ci / lint-test (push) Successful in 47s
Credit cards keep arriving as monthly statements and stay the source of
truth for them. This covers the other fourteen accounts, whose statements
arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise
including the income account, Up, ING, and the small transaction accounts.
About 50 rows a month, where the alternative is downloading each statement
by hand.

The file needs three defences, all found by diffing three real exports
(smarthome DECISIONS.md ING-11):

A CDR re-consent makes Frollo re-ingest an account's whole history under
fresh transaction ids while the originals survive, and consents expire
annually. On this export 112 rows were such twins, and every HDR salary
payment appeared twice — importing blind doubles reported income. dedupe()
collapses each natural-key group to its lowest id, lowest because old ids
were a strict subset of new across two exports, so source_ref stays stable
and a re-import inserts nothing.

An earlier version of that rule kept close-id rows on a 10,000 threshold,
reasoning that genuine same-day repeats have consecutive ids. Verifying it
against the income rows killed it: in-scope duplicate pairs have id gaps
from 58 to 260 million, so no threshold separates them from the gaps of 1-4
that real repeats showed. It now collapses unconditionally and flags
anything within 10 for review — the errors are asymmetric, and nothing in
scope has ever tripped the flag.

A lapsed consent removes an account from the export silently, with no error
and no marker; the row count just drops. So the import asserts the account
roster and refuses to run when a configured account contributes nothing.

Also holds these rows out of the pending-reconciliation queue. A feed row is
the account's own ledger entry, not a receipt awaiting a statement line —
these accounts' statements are deliberately not imported — so without the
exclusion 550 rows a year would bury the receipts that need a decision. The
queue stays at 8 instead of 558.

Foreign rows follow order-ingestion's existing shape: amount is the native
figure, foreign_currency_code names it, amount_aud stays NULL rather than
asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as
incomplete.

Dry run by default. Verified against the real export before applying:
550 rows inserted, 14 accounts, re-run inserts 0.
2026-08-13 10:49:35 +10:00
siddharthd 2c666236b2 orders: share the merchant tidy, and fix two bugs the corpus run found (board 212)
ci / lint-test (push) Successful in 41s
The tidy existed but lived inside the list page, so the DETAIL page never had
it: following a row through showed "Apple" becoming "Apple Pty Ltd." Moved to
src/lib/merchant-label.ts and applied on both surfaces.

Running it over all 1,451 distinct merchant names — rather than over examples —
turned up two real defects:

  Coburger & Co                    -> "Coburger &"
  eBay Commerce Australia Pty Ltd. -> "EBay Commerce Australia"

The first strips a suffix that is part of the brand and leaves a dangling
connector. The second is the capitalisation rule firing on "starts with a
lower-case letter" when what it means is "is a bare domain" — mangling a brand
that is deliberately lower-cased. Both are now guarded, and re-running over the
full corpus reports 170 names tidied and 0 defects.

PRESENTATION ONLY. It merges nothing and must never decide two rows are the
same merchant. Amazon.in stays Amazon.in — a different marketplace with
different currency and geography, not a name variant — and there is a test
asserting it can never equal amazon.com.au. Kogan.com, GOG.com, AliExpress.com
and Catch.com.au are real brand names containing a TLD and are left alone.
Real unification is the merchant-alias bridge, ticket 176, which has already
merged what can be merged safely.

The five names that read alike after tidying (ebay/eBay/eBay Inc.,
menulog/Menulog Pty Ltd, deliveroo/Deliveroo, grab/Grab, paypal/PayPal) are
genuinely one merchant each, so reading alike is correct. Lower-case platform
SLUGS are deliberately not capitalised: "ebay" -> "Ebay" is wrong, and getting
it right is the registry's job, not the stylesheet's.
2026-08-12 21:07:50 +10:00
siddharthd 215303161b orders: bring the lifecycle-only toggle's numbers up to date (board 210)
ci / lint-test (push) Successful in 42s
The count moved three times on 2026-08-12 and the history is the point:

  755  when the predicate was written
  904  after board 219 turned unstated zeros into NULLs — those 149 rows had
       been failing the "no amount" test on a technicality
  699  after 210a retired 205 Amazon "share your experience" mails, which were
       never lifecycle events at all
  620  after 210b joined 79 phantoms to their real parent, placebo-verified at
       a 1.9% error rate

What is left is the corpus coverage ceiling rather than a defect — a mail whose
parent is not in the corpus has nothing to join to — and the tooltip now says
that instead of describing the whole class as an open ingestion defect.
2026-08-12 20:38:43 +10:00
siddharthd 3b5e495ca4 orders phase 2: order_transaction_links, and move every reader to it (board 205)
ci / lint-test (push) Successful in 41s
Phase 1 read linkage out of expense_metadata, which is shaped as ONE ROW PER
TRANSACTION — transaction_id UNIQUE, matched_transaction_id partial-unique — so
every multiplicity it expresses is smuggled through a string key (0029 keys
split shipments <entity_key>#f<fact_id>). A BNPL plan needs four rows against
one order and has no such trick available.

Migration 0030 adds order_transaction_links (many-to-many, keyed on entity_key
rather than entities.id: finance-app does not model the spine and must not hold
an FK across a boundary a re-extraction can decompile) and backfills all 63
existing bridge rows.

THE UNIQUENESS RULE WAS WRONG FIRST TIME, in the most on-brand way available.
It read (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of
the same order — both leg_index NULL — collapsed to one key and ON CONFLICT DO
NOTHING dropped one SILENTLY. Caught only because the backfill reported 62
against 63 candidates. The row it ate was order_amazon_249-4859367-0690246's
$130.00 second shipment, the same order named in migration 0029's comment as
the reason split shipments need distinct keys at all. What identifies a leg
depends on its kind: an instalment by its INDEX, a shipment by its FACT, a
whole-order charge by neither.

ALL FOUR READERS MOVE TOGETHER, links first with expense_metadata as fallback:

  - LINK_LATERAL (list txn_count/first_txn_id, and the has_transaction facets
    that read it)
  - the detail page's transactions query
  - /api/transactions/[id]/order
  - the order_ctx lateral in queries.ts

Moving fewer is not a smaller change, it is an inconsistent one: the matcher's
four links for order_ebay_14-11714-95953 come with no expense_metadata row, so
a half-move would show four instalments on the detail page while the list said
txn_count = 0 and put the order on the wrong side of BOTH has_transaction
filters. Verified after: detail 4 legs, list txn_count 4, has_transaction=yes
includes it, =no excludes it.

UNION not UNION ALL on transaction_id — after the backfill the same charge is
legitimately in both stores and counting it twice would show "2 charges" on a
single-payment order.

order_platform is deliberately NOT coalesced with the link's platform. It gates
the receipt disclosure arrow on /transactions, and a BNPL leg has no receipt
behind it — filling it in put an arrow on four Afterpay rows that expand to
nothing, which is the exact promise the arrow exists to avoid making. Separate
leg fields carry the sub-line instead ("1 of 4 - DJI Air 3 Fly More Combo").
order-details.tsx now also requires a real receipt (platform present) before
rendering, because the endpoint can answer with a link alone.

Two invariant views, both empty and expected to stay so: order_link_orphans
(spine re-keying silently orphans a TEXT key — bridge links are rebuildable,
`manual` ones are lost curation) and order_link_drift (the two stores
disagreeing). The plan's suggested fix for drift — widening the bridge's NOT
EXISTS guard to "neither store has it" — was NOT taken: the expense_metadata
INSERT has no ON CONFLICT, so re-attempting an order that already has a receipt
row would duplicate it. Detecting is cheap; a non-idempotent re-write is not.

Unchanged: order_feed 6,265, order_spend AUD 4,895 / $442,651.80. Links 75.
2026-08-12 20:29:42 +10:00
siddharthd 6b9b5fe518 orders: badge the annual summary receipts (board 211)
ci / lint-test (push) Successful in 40s
Migration 033 adds entity_orders.summarises_period; order_spend drops those
rows and order_feed keeps them. Unbadged, a $2,376 annual tax receipt
restating twelve monthly donations renders as the largest single purchase of
the year, so both surfaces say what it is:

- /orders  — a "year summary" chip beside the auth badges
- /orders/<key> — a banner in the same shape as the settled-rail one, saying
  the monthly debits are in the ledger under their own dates

The amount is deliberately still shown. It is the year's giving total and the
reason the row is worth keeping; it just belongs to no total on the page. The
detail page already falls back to o.order_total when order_spend has no row,
which is exactly the case now — verified: gross_total null, order_total 2125.5.
2026-08-12 20:12:36 +10:00
siddharthd 231ef3411a orders: record why line items may not exempt a row from the phantom filter
ci / lint-test (push) Successful in 40s
Board 219 nulled 263 unstated zero totals in the spine, which moved 149 rows
into the lifecycle-only hide predicate — they had been failing its "no amount"
test on a technicality, an unstated 0 they should never have carried, while
matching the phantom signature on every other count.

Tried exempting rows that carry line items, on the theory that naming what was
bought proves a purchase. It surfaced 442 rows and the first window checked
showed why it is wrong: the Amazon "Ordered:" mail that mints a msg- phantom
carries the item list too, so Xelsluthe and YIWENTEC each rendered TWICE again
— the exact complaint this page was fixed for today. Reverted, and the reason
recorded in place so it is not re-attempted.

Items do not separate an orphan purchase from an item-carrying echo. Only
deduplication does, and that is board 210.

No behaviour change from the previous commit; comments and counts only.
2026-08-12 18:26:24 +10:00
80 changed files with 7159 additions and 319 deletions
+12 -4
View File
@@ -22,10 +22,18 @@ jobs:
# the normal colour.
- name: Generate Prisma client
run: npx prisma generate
# Advisory until the pre-existing lint debt is cleared (2026-07-19:
# ~20 errors across budget/insights/shared pages) — then make blocking.
- name: Lint (advisory)
# Blocking since #79. It was advisory from 2026-07-19 while 23 errors sat
# across the pages and the integration tests; those are now fixed, so a
# new one must not be able to slip in behind a green tick. Warnings are
# still allowed (eslint exits 0 on them) — there are 9 left, all
# `no-unused-vars` or `exhaustive-deps` advisories, none of which should
# stop a push.
- name: Lint
run: npm run lint
continue-on-error: true
# Typecheck is separate from `next build`: nothing else in CI compiles the
# test files, so a type error in a fixture could only ever be found on a
# developer's machine.
- name: Typecheck
run: npx tsc --noEmit
- name: Unit tests
run: npm test
+5 -4
View File
@@ -13,10 +13,11 @@ Keep data flow consistent: API routes call query functions in `src/lib/queries.t
- `npm ci` installs the locked dependency set (Node 22 is used in CI).
- `npm run dev` starts the local Next.js development server.
- `npm run build` creates a production build; `npm start` serves it.
- `npm run lint` runs the Next.js ESLint configuration. Existing lint debt makes CI lint advisory, but new code should pass.
- `npm run lint` runs the Next.js ESLint configuration. **Blocking in CI since #79** — the pre-existing debt is cleared, so it must stay at zero errors (warnings are still allowed).
- `npx tsc --noEmit` typechecks, including the test files. Also blocking in CI.
- `npm test` runs fast unit tests.
- `npm run test:setup` prepares the PostgreSQL test database using `.env.test`.
- `npm run test:integration` runs database-backed tests.
- `npm run test:setup` creates/refreshes the `personal_test` database (run it once, and after any migration).
- `npm run test:integration` runs database-backed tests. It re-resolves the DSN from the live container first (`scripts/resolve-test-db.sh`), so `.env.test` must never carry a hand-written host — see `CLAUDE.md` → Testing.
- `npm run test:all` runs both test suites.
## Coding Style & Naming Conventions
@@ -33,4 +34,4 @@ Add schema changes as the next numbered `prisma/migrations/NNNN_description/migr
## Commits & Pull Requests
History follows concise Conventional Commit-style subjects such as `feat(rules): preview rule changes`, `fix(trips): ...`, and `docs: ...`. Keep commits focused. Pull requests should explain behavior and data-model impact, link related issues, list validation commands, and include screenshots for UI changes. Ensure unit tests and the production build pass; call out any known lint warnings or migration steps.
History follows concise Conventional Commit-style subjects such as `feat(rules): preview rule changes`, `fix(trips): ...`, and `docs: ...`. Keep commits focused. Pull requests should explain behavior and data-model impact, link related issues, list validation commands, and include screenshots for UI changes. Ensure lint, typecheck, unit tests and the production build pass; call out any migration steps.
+467 -7
View File
@@ -59,6 +59,49 @@ docker exec postgres-personal psql -U personal -d personal
docker logs finance -f
```
## Testing
```bash
npm test # unit — pure logic, no database
npm run test:integration # database-backed; resolves the DSN itself (see below)
npm run test:all # both
npm run test:setup # once, and after any migration: creates/refreshes personal_test
npm run lint # blocking in CI since #79 — 0 errors, warnings allowed
npx tsc --noEmit # blocking in CI; nothing else compiles the test files
```
**The integration DSN is resolved at run time — never hand-write a host into
`.env.test`.** `scripts/resolve-test-db.sh` runs automatically as
`pretest:integration` and rewrites that file from the live container before
every run. It prefers the container's **published host port** (`docker port
postgres-personal` → `127.0.0.1:5432`, published since the 2026-08-17 PG17
migration) and falls back to the bridge IP only where that is absent. Set
`TEST_DATABASE_URL` to override it entirely.
This replaces the old "refresh the container IP" instruction, which described
the defect rather than a workaround (#161). `setup-test-db.sh` used to bake the
bridge IP into `.env.test` once, and Docker reassigns those on recreation: by
2026-08-21 the stored `172.22.0.18` belonged to **gitea**, and an earlier drift
had pointed the suite at **postgres-pantry** — whose database it would have
TRUNCATEd had the credentials happened to match. A loopback address cannot be
reassigned to another container, which is why the published port wins.
**Two guards, and both matter.** `helpers.ts::assertTestDatabase` refuses to
truncate anything whose `current_database()` is not `personal_test`, and the
suite runs `fileParallelism: false` because every file shares that one database
and `resetDB()` TRUNCATEs it.
**A fresh checkout or worktree needs `npx prisma generate` first.** The client is
generated into the gitignored `src/generated/prisma`, so without it every test
that reaches `src/lib/db.ts` fails on a missing package — which reads as a code
error rather than a setup one. CI does this explicitly for the same reason.
**Freeze the clock in any test whose subject is defined relative to "now".**
`analytics-priority0.test.ts` uses `vi.useFakeTimers({ toFake: ["Date"] })` —
only `Date`, because `pg` needs real timers for connection handling. Without it
the fixtures drift out of the routes' rolling windows over time and the
assertions quietly stop testing anything.
## Architecture
### Key Files
@@ -131,20 +174,59 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
### Key Tables
- `statements` — one row per billing period per bank account
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016, `credits` added by 0018) is `card | cash | bank_transfer | credits | other`, NULL = unknown, and the TypeScript mirror is `src/lib/payment-methods.ts`; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
### Cash and reconciliation
`payment_method = 'cash'` excludes a transaction from reconciliation via the
`notCash()` fragment in `queries.ts`. Cash never appears on a statement, so
`payment_method` in (`cash`, `credits`) excludes a transaction from
reconciliation via the `needsCardMatch()` fragment in `queries.ts` (named
`notCash()` when it only knew about cash). Cash never appears on a statement, so
without it a cash entry sits in the pending queue forever being offered matches
within 3 days and 1% on amount — and accepting one is silently destructive:
reconciled manual rows are filtered out of every query, so the cash spend
disappears while the card transaction it matched claims to be that same spend.
Only cash is excluded. Bank transfers *do* appear on a statement now that
transaction accounts are imported, and NULL means unknown — both stay
candidates, preserving the behaviour of every pre-existing row.
`credits` is the same shape for a different reason: a gift card or store balance
already paid the bill, so no card leg is ever coming. `bankLabel()` therefore
renders those rows **"Gift Card"** rather than "Manual" — "Manual" reads as
"still awaiting a card line", which is wrong here.
Only those two are excluded on payment method. Bank transfers *do* appear on a
statement now that transaction accounts are imported, and NULL means unknown —
both stay candidates, preserving the behaviour of every pre-existing row.
**The vocabulary lives in three places and drifted between them (fixed
2026-08-17).** Migration 0018 admitted `credits` to the CHECK constraint for the
order pipeline; the `VALID_METHODS` literal in `PATCH /api/transactions/[id]`
and the add-transaction dropdown were never updated. So the database held 372
gift-card rows carrying a value the UI could not offer and the API answered 400
for — a hand-entered gift-card purchase silently took the NULL default, and NULL
*is* reconcilable, so it joined the queue to be offered matches that cannot
exist. `src/lib/payment-methods.ts` is now the single list (mirroring
`statement-types.ts`), and both routes validate against it.
`src/__tests__/integration/payment-methods.test.ts` asserts it against
`pg_get_constraintdef` and evaluates `needsCardMatch()` in real SQL, so the next
migration that widens the constraint fails the suite instead of drifting
quietly.
**Account-feed rows are NOT excluded, and the removed exclusion is worth
knowing about.** There was an `awaitsStatementLine()` predicate here, added with
the Frollo importer on the premise that a feed row is the account's own ledger
entry with no statement line coming. That premise was asserted and never tested.
It was false for almost every account: 422 of the first 550 imported rows already
had a statement twin.
What matters is how it got in. The queue jumped from 8 to 558 the moment the feed
landed, and that jump was read as noise and filtered out. **The queue was right** —
those rows genuinely were provisional entries awaiting statement lines — and
filtering it removed the only mechanism that would ever have collapsed them, so
the duplicates became permanent instead of transient and stayed invisible until a
human saw one salary payment listed twice in two currencies.
A feed row belongs in the queue. Queue volume is solved **upstream**, by not
importing rows the ledger already holds (`LEDGER_MATCH_DAYS`), never downstream
by hiding the ones that are there. If a genuinely statement-less feed is ever
added, give it its own predicate and prove the premise with a query first.
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
as transfers only works if every cash purchase is logged; with partial logging
@@ -540,6 +622,89 @@ and one tap must not flatten it. It says which case it refused on.
Slack reaches this route through an n8n webhook, not directly — see the
smarthome repo's CLAUDE.md and the `slack-interactive-via-n8n` memory.
### Intake router — one upload, every applicable step (2026-08-21)
`POST /api/intake` (token `INTAKE_TOKEN`, multipart: `source`, `shared_by`,
`batch_id`, 1–6 `file`s) is the finance half of the share-sheet intake; design
and the cross-repo contract are in smarthome `plans/intake-router-slice-plan.md`
§11 (C1, C2, C5 live here; C3/C4 are pantry-app; C6 is n8n + compose).
- **Order of operations is the plan's and is load-bearing:** persist
`intake_batches` row → classify each file → ONE LiteLLM call for the whole
batch (`cloud-budget`, `temperature 0`, strict `json_schema`; PDFs go in as
`unpdf` text, never pixels) → header `(merchant, date, total)` → merchant
kind → duplicate check → reconciliation → match → act. `src/lib/intake-rules.ts`
holds the pure decisions, `intake-router.ts` the orchestration.
- **Match is `(platform, merchant, date, total)`, exact, never widened.** A
miss or an ambiguity returns candidates (same merchant ±3 days) with
`intake_use_candidate` / `intake_create` buttons; a transaction is created
only on the explicit tap, ING-10-style (`statement_id NULL`, `owner_id` from
`shared_by` → `participants.email`, else `DEFAULT_OWNER_ID`;
`source='intake'`, `source_ref=batch_id` so a second press is a no-op —
written as `ON CONFLICT … DO NOTHING`, because a SELECT-then-INSERT loses a
double tap). A non-AUD receipt is booked the way `order-ingestion.ts` books
one: `amount` native, `amount_aud` **NULL**, `foreign_currency_*` set — no FX
rate exists at intake and inventing one is worse than admitting there is none.
- **A NULL platform means "no platform", not "any platform".** An in-store tap
(`coles`, `woolworths`, `other`) matches only rows whose `expense_metadata`
carries no platform of its own. The wildcard reading let a till receipt match
the Uber Eats order for the same shop on the same day and overwrite its items.
The **candidates** query is the opposite and deliberately so: it has no
platform filter at all, because the card is a human choice shown precisely
because the exact key missed — the platform is one of the things that may be
wrong, so the list must widen there, not narrow.
- **The Create tap re-runs the match before it inserts**, and checks for a
resolved sibling batch. The card is minutes old by the time it is pressed and
the email/order bridge books orders on its own schedule; `findDuplicateBatch`
ignores *unresolved* batches by design, so two pending cards for one order
would otherwise each book it.
- **One `expense_metadata` row, agreed on by every query that resolves one.** A
transaction can own a row *and* be the `matched_transaction_id` of another;
`matchTransaction` returns the row it resolved through and `replaceLineItems`
writes to exactly that one. Every reader orders by
`(transaction_id = $1) DESC **NULLS LAST**, id` — the match, the write, the
order-detail GET, `getTransactions`' provenance LATERAL, `merchantForTransaction`
and the Slack details modal. The `NULLS LAST` is load-bearing: Postgres orders
`DESC` as `NULLS FIRST` and the row that merely *points* at the transaction has
a NULL `transaction_id`, so the clause meant to prefer the owned row preferred
the other one. Measured 2026-08-21: 667 rows, 147 pointing-only, **zero**
transactions resolving to more than one row — the ordering changes nothing
today and exists so the readers still agree when multiplicity appears.
- **The create is guarded by a transaction-scoped advisory lock on the
PURCHASE**, not the batch. `ON CONFLICT (source, source_ref)` handles a double
tap on one card; two *different* batches for one purchase have different
`source_ref`s and both clear their match and duplicate checks before either
inserts. The lock has to live inside `withTransaction` (`src/lib/db.ts`):
`queryRaw` is `$queryRawUnsafe` on a **pooled** client, so every call is its
own implicit transaction and `pg_advisory_xact_lock` taken in a statement of
its own is released before the next statement runs. The loser reports
`duplicate` pointing at the winner's transaction — the same answer the
sequential path gives.
- **`replaceLineItems` snapshots the undo baseline BEFORE it writes, on both
paths**, and treats an UPDATE that touched no row as a failure rather than a
success. The metadata INSERT is `ON CONFLICT (transaction_id) DO UPDATE` and
carries the batch's **currency** — a hard-coded `'AUD'` made `OrderDetails`
print `$` on a USD order, because it prefers `expense_metadata.currency`.
- **`intake_batches.previous_line_items` is the undo baseline, written once.**
Returning whatever the write just read makes a pantry retry get its own write
back as the "previous" state, so undo restores the thing it undid. Recorded
under `COALESCE(previous_line_items, …)` and read back.
- **Grocery never writes finance here.** The ORIGINAL files go to pantry
`POST /api/capture` with `intake.finance_target_transaction_id` +
`finance_action: enrich|create`; pantry's confirm calls
`PUT /api/transactions/[id]/line-items` back (C4), which is why that route
accepts `RECEIPT_INGEST_TOKEN` too. Files are kept under `INTAKE_FILE_DIR`
keyed by sha256 so a later "Groceries" tap can still forward them.
- **Reconciliation is against the printed item subtotal, not the total.** A
delivery order's total is items + fees − promotions ($118.94 of Coles lines
became $85.74), so checking lines against the total would hold every
delivery order. Tolerance $1 or 2%; a failure is `held`, nothing written.
- **Names stay verbatim.** Burgertory calls its burgers "Michael" and "Joel";
the prompt forbids canonicalising and the stored shape is the order lane's
(`qty`, `description`, `amount`, `options`) so `order-details.tsx` renders it.
- Statuses beyond the contract list: `ask` (the "Groceries or a meal?" holding
state), `held`, `error` (502 — model or pantry down; the batch row says so).
### The shared loan
The loan is a **separate ledger**, not a shared expense and not a settlement
@@ -617,6 +782,93 @@ deleted, and deleting a rule must not cascade away the audit trail.
changed by something else since the run are flagged, because reverting restores
the pre-run value and discards the later edit.
### Importing an aggregator/bank CSV (`/api/import/csv`)
There is **one** import path — the CSV import modal — and Frollo goes through it
like any other file. A bespoke Frollo importer, API route, CLI and two scheduled
n8n workflows existed for a day and were deleted on 2026-08-13: net −1,152 lines.
**The rule that made the rest unnecessary is statement coverage.** Each account's
newest `billing_end_date` is a watermark; a row on or before it is already in the
ledger. On the real 2,607-row export that leaves **171 rows** — with no account
allowlist and no credit-card exclusion, because cards have current statements and
drop out on their own. Every bit of the deleted apparatus was doing by hand what
`getStatementCoverage()` does generically.
Note what this replaced. The first attempt asked whether a row's date fell inside
a statement's min–max *window*, which for accounts whose statements span 182 to
460 days swallows a year and answers nothing — it let 422 duplicates into 550
rows. Amount-matching cannot substitute either: the two sources decompose the
same event differently, bundling a Wise transfer fee into the transfer (10001.13)
where the statement itemises it (10000.00 + 1.13).
**Map these optional columns or lose something silently:**
| Column | Without it |
|---|---|
| **Account** | nothing is excluded — you review the whole file |
| **Currency** | a foreign row is stored as if AUD (USD 10,782 → A$10,782) |
| **Row ID** | a re-import duplicates instead of no-opping |
Leave **Category** unmapped for an aggregator: its spend categories are not
trusted, and you set them per row in the review step.
**Two bugs fixed the day this shipped, both of which the flow depends on:**
`batchInsertCSVTransactions` accepted a `category` and then omitted it from the
INSERT column list, so every category chosen in review was discarded and the
trigger wrote `'other'` — not cosmetic, because an uncategorised credit is
admitted by `NET_SPEND_ROWS` and negated by `SPEND_SIGNED`, so 62 imported
transfers cancelled **$74,338** of spend while counting as no income. And the
path had no idempotency at all: `row_index` is assigned MAX+1 every run, making
`uq_transaction_identity` structurally unable to fire. It now writes
`source`/`source_ref` with `ON CONFLICT DO NOTHING`.
**The in-file duplicate warning is not redundant with `source_ref`.** A CDR
re-consent re-exports an account's whole history under fresh ids while the
originals survive — 385 twins in one 2,563-row export, doubling every salary
payment. Fresh ids mean `source_ref` sees new rows, and 385 is not visible by eye
in a review table. Consents expire annually, so expect it.
Undo an import: `DELETE FROM transactions WHERE source = '<source>'`.
### Zip, and the three things a BNPL account needs (2026-08-15)
24 statements, Dec 2020 → Jun 2026, all balance-asserting at $0.00. Zip is a
credit account, so importing it changes how three existing row types must read.
**1. A repayment leaving the bank is a duplicate ONLY where a Zip statement
covers the purchases.** Zip's own `Payment` rows are already `transfers`. On the
bank side, `ZIPPAY*` debits are the same money — but only 2 of 8 fall inside a
Zip statement period. Those two are now `transfers`; **the other six must stay as
spend**, because for those periods the bank debit is the only record the purchase
ever happened. Moving all eight would have deleted $703.35 of real spend. Same
statement-coverage rule as the CSV importer, and the same trap.
**2. `Account Credit` is cashback, and Zip labels its three money-in types
differently.** `Payment` = repayment from the bank. `Refund <merchant>` = a
genuine merchant refund. `Account Credit` = Zip cashback — 34 rows, $740, round
amounts ($20 ×20, $30 ×10, $10 ×4), with no matching debit from any bank account.
Gemini filed the identical description two ways — 23 as `other` (which *subtracts*
from spend) and 11 as `transfers` (excluded entirely) — so one thing was doing two
opposite things. All 34 are now `shopping` + `credit`, **netting against spend**:
cashback lowers what a purchase actually cost, which is how `NET_SPEND_ROWS` /
`SPEND_SIGNED` already treat a refund. Filing it as `income` was rejected for the
reason the Up item sales note gives — it flatters `net = income − spent` and mixes
non-earnings into the salary line. Ties to the **Cashback tracking (ING-6)** epic;
a dedicated `cashback` category would need the CHECK constraint, the TS mirror,
and a decision in every analytics fragment, so it was not done here.
**3. A gift card stays spend.** Prezzee is $1,580 of the history. Stored value is
tempting to call `transfers`, but that only works if every downstream purchase is
logged — it is not, for 2020. Identical reasoning to the ATM-withdrawal rule.
All 137 Zip rows are categorised; net spend is $11,135.11 against $10,085.11 of
`transfers`. **Statement 200 (Sept 2021) came through incomplete** — no
`billing_end_date`, no balances, 2 rows — so it is the one statement not covered
by `uq_statement_identity` (the index is partial on `billing_end_date IS NOT
NULL`) and a re-import would duplicate it. Worth re-processing paperless doc 296.
### Trusting extracted statement data
**Balance assertions are the check that works.** `getStatements` computes
@@ -692,6 +944,99 @@ rather than hand-rolling them. Two failure modes they exist to prevent:
Use the `EXCLUDE_NON_SPEND` fragment: a bare `category NOT IN (...)` evaluates
to NULL for uncategorised rows and drops them from totals.
### Spend is gated on MY_SPEND_SCOPE, not OWNER_SCOPE (2026-08-15)
**Ownership is not a precondition for an expense being yours.** Every spend
analytic used to gate on `OWNER_SCOPE = $1`, with `mySplitOf` scaling *within*
that gate — so your half of a shop Sonu paid for counted as **zero**, in monthly,
daily, merchants, subscriptions, fees and the budget page. 167 rows / **$3,210.91**
across Jan–Jul 2026, worst in April (+$1,354.83, the Europe trips). Meanwhile
`getParticipantBalances` booked the matching debt correctly, so the app could say
you owed her for a shop while insisting you had not spent anything on it.
Use `MY_SPEND_SCOPE()` — `owner = me OR I hold a split` — for anything measuring
what *I* spent. `OWNER_SCOPE` is still right for anything measuring an **account**:
the income and investment lines in `/analytics/monthly`, and the statement-level
fee rollup in `/analytics/fees` (which reads `statements`, where splits are
meaningless).
**`myShare` had to change with it, and widening the gate alone would have been
worse than the bug.** Its old third fallback, `100 - everyone else`, is the
*payer's* remainder — on someone else's unsplit row it returns 100 and would have
moved their entire bill onto you. It now branches on ownership:
- **My row** — unchanged: explicit split, then `my_share_percent`, then the
remainder.
- **Their row** — an explicit split row only; absent means **0**.
`my_share_percent` is deliberately not consulted on someone else's row. It is one
unscoped column on `transaction_overrides` writable by anyone who can see the row,
so "my" can only mean the owner's. All 402 rows carrying one today are owner-side.
That 0 is what makes the wider gate safe: admitting a row can never add more than
the share actually held. Tested both ways in
`src/__tests__/integration/analytics-sql.test.ts` — the discriminating cases are
all ones where you hold **no** split on someone else's row, because that is the
only place old and new disagree.
`MY_SHARE_PCT` in `queries.ts` mirrors `myShare` in subselect form for the
transactions list (which has no viewer-scoped `ts` join). A test asserts the two
agree row for row across seven fixture shapes — keep it that way, since the list
is the drill-down for these totals.
**No historical restatement:** every non-owner split is 2026-dated. The 1,266
pre-2026 SplitMyExpenses splits are all on rows you own, so nothing before the
cutover moves.
### Changing who paid (2026-08-15)
Owner was **write-once for every ingestion path** until now — a pantry receipt
hardcodes `DEFAULT_OWNER_ID` (`receipt-ingestion.ts`), and there was not one
`UPDATE ... SET owner_id` in `src/`. A shop the other person paid for was
permanently filed as yours.
Two routes, because the owner lives in two places:
- **`PATCH /api/transactions/[id]` with `owner_id`** — manual rows only
(`statement_id IS NULL`). A statement row returns 400 `statement_owned` and
points at the statements page; its effective owner is
`COALESCE(t.owner_id, s.owner_id)`, so writing it here would either no-op or
detach one row from the account it was extracted from.
- **`PATCH /api/statements/[id]` with `owner_id`** — the statements page has had
this dropdown since it was built, wired to a route with **no PATCH handler**.
Every change 405'd, and `useUpdateStatement` never checked `res.ok`, so it
failed silently and the select just snapped back.
**The statement route writes both tables.** 2,194 statement rows carry their own
`owner_id` against 1,803 that inherit, so updating `statements` alone moves less
than half and splits one account's history between two people. It updates rows
matching the *old* owner and returns `rows_moved`; all 2,194 agree today, and a
row that disagrees was set deliberately and is left alone.
**Reassignment is a one-way door, and the guard is the point.** Access is
`owner OR holds a split` (`canAccessTransactions`), so handing a row over while
holding no split removes it from your list and 404s every route that could put
it back — only the new owner can undo it. The route returns **409
`would_lose_access`** and the modal offers the two real ways forward: add your
split first, or "Give it away anyway" (`release: true`). Taking a row *onto*
your ledger is never blocked — there is no door to close behind you.
Reassignment is a correction to a row you already hold, never a way to reach one
you do not: `canAccessTransactions` runs first, so claiming a stranger's
transaction is a 404 before any owner logic runs.
**Splits are deliberately not rewritten.** They record shares, not direction —
`getParticipantBalances` derives who owes whom from ownership, so a 50/50 flips
from "they owe me" to "I owe them" untouched, `settled` included. Tested both
directions.
**Order matters when you do this by hand:** split first, then reassign. The
reverse locks you out, which is exactly what the 409 exists to stop.
`useUpdateTransaction` also gained the missing `res.ok` check. Without it every
rejection resolved as success — the modal closed, the list refetched, and the
edit silently vanished.
### Statement types
`statements.statement_type` is constrained to `credit_card | transaction |
@@ -704,6 +1049,109 @@ Card'`) and the DB normalises it on write. The raw extracted value is preserved
The TypeScript mirror is `src/lib/statement-types.ts` — keep the list, the SQL
function, and the CHECK constraint in sync when adding a type.
### Bank names are normalised on write (migration 0031)
**`bank_name` is a component of the duplicate check.** `uq_statement_identity`
is `(bank_name, account_number, billing_end_date)`, so two spellings of one
institution mean the same statement can be ingested twice without the index ever
firing. Gemini reads the name off whatever the PDF prints, so the spelling varies
per *document*, not per account.
Zip exposed it: account `2705256` arrived under four names in sixteen minutes —
`ZipMoney Payments Pty Ltd`, `ZipMoney Payments Pty Limited`, `Zip Pay`,
`ZipPay` — with ten more in the queue. It was not unique. 17 `bank_name` values
represented 13 institutions, and every split pair shared an account number:
Amex (14/1), Citibank/Citi (8/4), NAB (3/1). Fragmentation also splits the
statements and transactions bank filters and `by_bank` in `/api/analytics/fees`.
`normalize_bank_name()` + `trg_statements_normalize_bank_name` (BEFORE INSERT OR
UPDATE OF bank_name) mirror the 0013 `statement_type` pattern, with **one
deliberate difference: the vocabulary is open.** An unrecognised name passes
through tidied, never collapsed to a fallback, and there is **no CHECK
constraint** — a bank this household has never used must be able to arrive
without a migration, and destroying its name on first contact is worse than
leaving it unmapped.
Matching is by **prefix, not by enumerated spelling** (`zip%`, `citi%`,
`%american express%`, `%national australia bank%`), so unseen variants like
`Zip Co Australia Pty Ltd` normalise with no code change. Branches for Westpac,
ANZ, HSBC, ING, AMP, Up and CommBank are no-ops today that map *to* the spelling
already in use — they exist to catch the legal-entity variant a future PDF might
print.
Wise is deliberately unmapped: `Wise Australia Pty Ltd.` is a single spelling, so
shortening it would be a rename nobody asked for rather than a merge.
**One institution's national entities stay apart (migration 0032).** A prefix
generalises past the evidence it was built on: every 0031 merge was provably one
account, but `citi%` would have flattened Citibank India into Citibank Australia
on arrival. `hsbc%` and `%american express%` had the same defect. The country now
comes from the name when the name states it and from the currency otherwise, with
AUD as home taking no suffix — so nothing renamed (`UPDATE 0`).
Currency alone cannot be the discriminator: **Wise holds AUD, EUR and USD
accounts under one provider**, so "non-AUD means a different bank" would shatter
it into three. That is why Wise must stay unmapped, and why mapping any
multi-currency provider through this function would be wrong.
`N.A.` is deliberately not a country marker — it means "National Association", a
US legal form printed on Citibank letterhead worldwide including India.
**The residual gap and its tell.** The function cannot know that two names for a
*new* bank are one institution. That always shows up the same way, so check the
`statement_identity_drift` view (migration 0033) rather than trusting the map.
Non-empty means an account is fragmented, by name or by number.
### Account identity is derived, not the raw number (migration 0033)
`account_number` fragments exactly like `bank_name` did, from the same cause —
Gemini copies what the PDF prints. ANZ's Access Advantage arrived as both
`4085-56264` (4 statements) and `408556264` (1). Since `uq_statement_identity`
keyed on it, re-importing one period under the other spelling evaded the
duplicate check.
`account_number_key` is a **GENERATED ALWAYS** column — `account_number` with
separators stripped — and the unique index is now
`(bank_name, account_number_key, billing_end_date)`. **Never write to it;
display `account_number`.**
Derived rather than rewritten because the raw number is the readable one and
some of it is structure: Up stores `633-123 / 176540052`, a BSB *and* an account
number, and flattening it would lose a distinction a human reads at a glance to
fix a machine problem. Case and the Amex mask (`XXXX-XXXXXX-01000`) are
preserved — `X` records which digits were redacted.
This is **not** what caused the documented 31-row / $42,040.68 ANZ duplication.
Statements 107/142/143 overlap on *different* end dates, which that index cannot
catch at any spelling. Different problem, same table.
`account_owner_mappings` also keys on `(bank_name, account_number)` with its own
UNIQUE constraint, so any future rename must update it too or strand its rows.
It is empty today; the migration handles it anyway.
**Normalising `bank_name` broke the N8N new-bank alert, and the fix lives in the
workflow.** `Check Known Bank` ran
`SELECT COUNT(id) FROM statements WHERE bank_name = '<raw Gemini name>'`,
comparing the raw extraction against the now-canonical column *before* the insert
— so the trigger had not fired yet and nothing ever matched. That branch tags the
document **pending and holds it for Slack approval**, so every Zip statement
stalled awaiting a manual click, not merely a noisy alert. It was a regression
made worse by normalisation: previously a repeat of the same spelling at least
matched.
Fixed 2026-08-15 by comparing like with like:
```sql
SELECT COUNT(id) as count FROM statements
WHERE bank_name = normalize_bank_name('{{ ...summary.bank_name }}', '{{ ...summary.currency }}')
```
Currency is passed so a genuinely new national entity (Citibank India) still
alerts. Verified live: the run after the fix took 12s against 30–53s for the
approval-branch runs before it. Workflow `FysADdFwEtwONQl4` in the smarthome
repo — **any future change to `normalize_bank_name()` must keep that node in
step**, since it is the one caller outside this codebase.
### Loans
A loan repayment is **not** an expense. It is part principal (equity, a
@@ -810,7 +1258,19 @@ See `README.md` → **Known Gaps / TODOs** for full details.
drone, camera). Correct in that they are excluded from spend, but it mixes
asset disposals into the income line alongside salary.
- **`payment_method` is not shown in the transactions list** — settable on create
and edit only. Worth a column or filter if cash becomes routine.
and edit only. Worth a column or filter if cash becomes routine. A `credits`
row is at least visible indirectly: `bankLabel()` renders it "Gift Card"
rather than "Manual", and the bank filter has a chip for it.
- **219 manual rows carry a NULL `payment_method`**, so they sit in the pending
queue being offered card matches. NULL means *unknown*, not *unreconcilable* —
that is deliberate — but any of them paid by gift card or cash is being
offered a match that cannot exist. No bulk fix, because guessing the method
from a description would be a guess written into a column that suppresses
reconciliation.
- **Pantry receipts still land as yours.** `processReceiptIngestion` hardcodes
`DEFAULT_OWNER_ID`; the ingest route takes no owner. Correctable per row now
(see "Changing who paid"), but a "Paid by" step at capture time would stop the
correction being needed.
- **Raw statement exports live in `dump/`**, gitignored since `31a8177`. They were
committed by accident in `030490e` and remain in that commit's history; the repo
has no GitHub remote, so exposure is limited to the local Gitea. Purging history
+14
View File
@@ -282,6 +282,18 @@ All routes require authentication via `X-Forwarded-User` header (set by Traefik)
| GET / POST | `/api/tags` | List or create tags |
| PATCH / DELETE | `/api/tags/[id]` | Update or delete a tag |
**Machine routes** — shared-secret `x-ingest-token` header instead of `X-Forwarded-User`; each fails closed when its token is unset:
| Method | Route | Token | Description |
|--------|-------|-------|-------------|
| POST / PATCH | `/api/orders/ingest` | `ORDER_INGEST_TOKEN` | n8n order receipts → transactions (see smarthome CLAUDE.md "Order Processor") |
| POST | `/api/receipts/ingest` | `RECEIPT_INGEST_TOKEN` | pantry-app confirmed grocery receipts → one transaction per tender leg (ING-10) |
| POST | `/api/intake` | `INTAKE_TOKEN` | **Intake router** (2026-08-21): multipart `source` (`uber_eats\|doordash\|coles\|woolworths\|other`), `shared_by`, `batch_id` (uuid), 1–6 `file`s (image / PDF / text). Persists `intake_batches`, extracts once via LiteLLM `cloud-budget` (PDF → text via `unpdf`), matches `(platform, merchant, date, total)` exactly, then enriches the matched order's `line_items` (food), forwards the original files to pantry `POST /api/capture` (grocery), or replies with candidates / "Groceries or a meal?" / needs-header / held / duplicate. Returns `{kind, status, transaction_id?, capture_id?, candidates?, slack_blocks, text}` |
| PUT | `/api/transactions/[id]/line-items` | `INTAKE_TOKEN` or `RECEIPT_INGEST_TOKEN` | Idempotent replace of `expense_metadata.line_items` for `transaction_id = :id OR matched_transaction_id = :id`; body `{line_items, source: 'intake'\|'pantry', batch_id?, platform?, replace: true}`; creates the metadata row if absent, never a transaction (404). Returns `{previous_line_items, line_items}` — pantry calls it on every confirm and with `previous_line_items` on undo |
| POST | `/api/slack/interactive` | `ORDER_INGEST_TOKEN` + Slack signature | Order-nudge buttons and the intake taps (`intake_use_candidate`, `intake_create`, `intake_grocery`, `intake_meal`) |
Env for the intake router: `INTAKE_TOKEN`, `LITELLM_BASE_URL` + `LITELLM_KEY` (OpenAI-compatible, alias `cloud-budget`), `PANTRY_BASE_URL` (+ the existing `RECEIPT_INGEST_TOKEN` as pantry's machine auth), optional `INTAKE_FILE_DIR` (original files by sha256, default `/app/data/intake` — a later "Groceries" tap re-sends them) and `FINANCE_PUBLIC_URL` (the card's View link, default `https://finance.bosecamp.com`). Design: smarthome `plans/intake-router-slice-plan.md` §11.
---
## Ingestion Pipeline
@@ -347,6 +359,8 @@ docker exec postgres-personal psql -U personal -d personal \
| `0009_split_payments` | `split_payments` |
| `0010_csv_import_reconcile` | `reconciled_with_id`, CSV import support |
| `0011_trips` | `trips`, `trip_id` on `transaction_overrides` (catch-up — was applied directly) |
| … | |
| `0034_intake_batches` | `intake_batches` (one row per `/api/intake` upload, written before extraction) and `intake_merchant_kinds` (the remembered "Groceries or a meal?" answer per merchant) |
> `paperless_doc_id` on statements and the `uq_statements_paperless_doc_id` index were added directly (not tracked in a migration file).
> `owner_id` on transactions and `statement_id` made nullable were applied directly (March 2026) to support manual transaction entry without a fake statement.
+17 -7
View File
@@ -47,13 +47,23 @@ Also fixed while in here, both found by checking rather than by proposal:
category-day, so the pace chart ended the month a few cents off its own
headline. Both now carry 4dp and round once, at display.
Partially guarded by `src/__tests__/integration/analytics-sql.test.ts`, which
covers the SQL fragments — `EXCLUDE_RECONCILED_SOURCE`, `NATIVE_CURRENCY` and
`INVESTMENT_SIGNED` — and nothing else. **Items 2, 3 and 6 have no test.** Those
are the three where a regression is silent rather than loud: the pace chart would
simply go back to disagreeing with the headline printed directly above it, the
fees figure back to growing forever, and a partial month back to being measured
against complete ones — all without anything failing. Tracked as #155.
Guarded in two files. `src/__tests__/integration/analytics-sql.test.ts` covers
the SQL fragments — `EXCLUDE_RECONCILED_SOURCE`, `NATIVE_CURRENCY` and
`INVESTMENT_SIGNED`. Since 2026-08-21 `analytics-priority0.test.ts` covers items
2, 3 and 6 (#155): the three where a regression is silent rather than loud — the
pace chart simply going back to disagreeing with the headline printed directly
above it, the fees figure back to growing forever, and a partial month back to
being measured against complete ones, none of which fails anything.
Two things were needed to make item 6 testable at all, and both are the point
rather than incidental. The rules lived as closures inside `budget/page.tsx`
reading `new Date()` inline, so they now live in `src/lib/partial-month.ts` with
an injectable clock; and the tests freeze that clock, because a fixture defined
against the real "now" drifts out of the routes' rolling windows and the
assertion goes vacuous — the same class of silent failure, one level up. Item
2's suite carries a placebo assertion for the same reason: parity between two
queries that are identical by construction proves nothing unless some fixture
can actually tell the old and new spend definitions apart.
### Landed after this doc, in the same family (2026-08-02)
+19 -1
View File
@@ -16,7 +16,8 @@
"prisma": "^7.4.2",
"react": "19.2.3",
"react-dom": "19.2.3",
"recharts": "^3.8.0"
"recharts": "^3.8.0",
"unpdf": "^1.8.1"
},
"devDependencies": {
"@tailwindcss/postcss": "^4",
@@ -8494,6 +8495,23 @@
"dev": true,
"license": "MIT"
},
"node_modules/unpdf": {
"version": "1.8.1",
"resolved": "https://registry.npmjs.org/unpdf/-/unpdf-1.8.1.tgz",
"integrity": "sha512-xkURhy2SoGpOIH0a1gLHNkASPIQYonadDJs2AQwPEfUakafeD9EA1WTWWsaR++gfTCXJpV27W7tU1nXuk82UKQ==",
"license": "MIT",
"engines": {
"node": ">=22"
},
"peerDependencies": {
"@napi-rs/canvas": "^0.1.69 || ^1.0.0"
},
"peerDependenciesMeta": {
"@napi-rs/canvas": {
"optional": true
}
}
},
"node_modules/unrs-resolver": {
"version": "1.11.1",
"resolved": "https://registry.npmjs.org/unrs-resolver/-/unrs-resolver-1.11.1.tgz",
+3 -1
View File
@@ -10,6 +10,7 @@
"test": "vitest run --config vitest.config.ts",
"test:watch": "vitest --config vitest.config.ts",
"test:setup": "bash scripts/setup-test-db.sh",
"pretest:integration": "bash scripts/resolve-test-db.sh",
"test:integration": "vitest run --config vitest.integration.config.ts",
"test:all": "npm test && npm run test:integration"
},
@@ -22,7 +23,8 @@
"prisma": "^7.4.2",
"react": "19.2.3",
"react-dom": "19.2.3",
"recharts": "^3.8.0"
"recharts": "^3.8.0",
"unpdf": "^1.8.1"
},
"devDependencies": {
"@tailwindcss/postcss": "^4",
@@ -0,0 +1,127 @@
-- 0030 — order ↔ transaction, many-to-many (board 205, orders phase 2)
--
-- Phase 1 read linkage out of `expense_metadata`, which is shaped as ONE ROW
-- PER TRANSACTION: `transaction_id` is UNIQUE and `matched_transaction_id`
-- partial-unique. Every multiplicity it expresses today is smuggled through a
-- string key — migration 0029 keys split shipments `<entity_key>#f<fact_id>` —
-- and a BNPL plan needs four rows for one order with no such trick available.
--
-- The flagship case: order_ebay_14-11714-95953, a A$1,599 DJI drone paid in
-- four A$399.75 Afterpay legs (txns 2318, 2333, 1652, 1664). Phase 1 shows it
-- with zero linked transactions, which is correct and useless.
--
-- KEYED ON entity_key, NOT entities.id. finance-app does not model the spine
-- and must not carry an FK across an ownership boundary that a spine
-- re-extraction can decompile. The real cascade risk is NOT transaction
-- deletion (that is handled below) but spine RE-KEYING: supersede_stale()
-- decompiles an entity when a re-extraction yields a different entity_key, and
-- a TEXT key with no FK silently orphans. Bridge-sourced links are rebuildable;
-- `manual` ones are curation and are not. Hence the orphan view at the end.
CREATE TABLE IF NOT EXISTS order_transaction_links (
id SERIAL PRIMARY KEY,
entity_key TEXT NOT NULL,
transaction_id INTEGER NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
leg_kind TEXT NOT NULL CHECK (leg_kind IN ('charge','shipment','instalment','refund','fee')),
-- HUMAN position only: "2 of 4". Never an id.
--
-- The first draft stored a fact id here. Live shipment keys already carry
-- fact ids 22664 and 22711, max(extracted_facts.id) is 23,781 today, and
-- SMALLINT tops out at 32,767 — that ceiling arrives on ordinary corpus
-- growth and the insert dies with `smallint out of range`. Fact ids go in
-- source_fact_id, which is BIGINT.
leg_index SMALLINT,
leg_count SMALLINT,
source_fact_id BIGINT,
-- The LEG's own amount, signed: positive is money out, negative is a credit
-- coming back. Storing the order total on every leg would make four rows sum
-- to four times the purchase.
amount NUMERIC(12,2) NOT NULL,
currency TEXT NOT NULL DEFAULT 'AUD',
source TEXT NOT NULL CHECK (source IN ('order-bridge','instalment-matcher','manual')),
confidence TEXT NOT NULL DEFAULT 'exact' CHECK (confidence IN ('exact','derived','manual')),
evidence JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Every index needs IF NOT EXISTS, not just the table: a run that creates the
-- table and then fails during the backfill would otherwise abort on re-run
-- with "relation already exists" and need manual surgery.
--
-- uq_otl_transaction is load-bearing. Without it two matchers can each claim
-- txn 2318 and the order reads as paid twice. It is deliberately stricter than
-- reality for one case — a single card charge covering two orders — which then
-- surfaces as a NAMED REJECTED INSERT rather than as silent duplication.
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_transaction ON order_transaction_links (transaction_id);
-- WHAT IDENTIFIES A LEG DEPENDS ON ITS KIND, and the first version of this
-- index got it wrong in the most on-brand way available. It read
-- (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of the
-- same order — both with a NULL leg_index — collapsed to the same key and
-- ON CONFLICT DO NOTHING dropped one SILENTLY. Caught only because the
-- backfill reported 62 rows against 63 candidates.
--
-- The row it ate was order_amazon_249-4859367-0690246's $130.00 second
-- shipment (txn 3171) — the same order named in migration 0029's comment as
-- the reason split shipments need distinct keys at all, because Amazon mails
-- and CHARGES per shipment. A uniqueness rule that cannot hold two shipments
-- is the exact defect 0029 was written to fix.
--
-- So: an instalment leg is identified by its INDEX (1 of 4), a shipment leg by
-- its FACT (each shipment is its own extracted fact), and a whole-order charge
-- by neither — one per order, which the zeros express.
DROP INDEX IF EXISTS uq_otl_leg;
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_leg ON order_transaction_links
(entity_key, leg_kind, COALESCE(leg_index, 0), COALESCE(source_fact_id, 0));
CREATE INDEX IF NOT EXISTS idx_otl_entity ON order_transaction_links (entity_key);
-- Backfill the existing bridge rows so the table is not empty on day one and
-- the readers can switch over in the same change. `#f<fact_id>` splits into a
-- shipment leg carrying its fact id; everything else is a whole-order charge.
INSERT INTO order_transaction_links
(entity_key, transaction_id, leg_kind, leg_index, source_fact_id, amount, currency, source, confidence, evidence)
SELECT split_part(em.source_message_id, '#', 1) AS entity_key,
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
CASE WHEN em.source_message_id LIKE '%#f%' THEN 'shipment' ELSE 'charge' END,
NULL::smallint,
NULLIF(substring(em.source_message_id FROM '#f([0-9]+)$'), '')::bigint,
COALESCE(em.amount, 0),
COALESCE(em.currency, 'AUD'),
'order-bridge',
'exact',
jsonb_build_object('backfilled_from', 'expense_metadata',
'source_message_id', em.source_message_id)
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
ON CONFLICT DO NOTHING;
-- Spine re-keying orphans links silently. Bridge rows can be rebuilt by
-- re-running the bridge; a `manual` orphan is lost curation and is the row
-- that actually needs a human. Same idea as order_settlement_violations.
CREATE OR REPLACE VIEW order_link_orphans AS
SELECT l.id, l.entity_key, l.transaction_id, l.leg_kind, l.leg_index,
l.amount, l.currency, l.source, l.confidence, l.created_at
FROM order_transaction_links l
WHERE NOT EXISTS (SELECT 1 FROM entities e WHERE e.entity_key = l.entity_key);
-- Two stores that can disagree will disagree. The bridge now writes both in
-- ONE transaction, so a half-written pair should be impossible — this view
-- exists to prove that rather than to assume it, and it must stay empty.
--
-- The plan's suggested guard (widen the bridge's `NOT EXISTS expense_metadata`
-- to "neither store has it") was NOT taken: the expense_metadata INSERT has no
-- ON CONFLICT, so re-attempting an order that already has a receipt row would
-- duplicate it. Detecting drift is cheap; re-attempting a non-idempotent write
-- to fix a state that cannot occur is not.
CREATE OR REPLACE VIEW order_link_drift AS
SELECT em.source_message_id,
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
em.amount, em.currency, em.reconciled_at
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM order_transaction_links l
WHERE l.transaction_id = COALESCE(em.matched_transaction_id, em.transaction_id)
);
@@ -0,0 +1,148 @@
-- Normalise statements.bank_name so one institution is one name.
--
-- WHY THIS IS NOT COSMETIC: `uq_statement_identity` is
-- (bank_name, account_number, billing_end_date). The bank name is a *component
-- of the duplicate check*, so two spellings of one institution mean the same
-- statement can be ingested twice without the index ever firing. Gemini reads
-- the name off whatever the PDF happens to print, so the spelling varies per
-- document, not per account.
--
-- Zip made this visible. One account (2705256) arrived under four names in
-- sixteen minutes — 'ZipMoney Payments Pty Ltd', 'ZipMoney Payments Pty
-- Limited', 'Zip Pay', 'ZipPay' — with ten or more still in the ingestion
-- queue. It was not unique: 17 bank_name values represented 13 institutions,
-- and every split pair shared an identical account number.
--
-- American Express (14) / AMERICAN EXPRESS (1) -> XXXX-XXXXXX-01000
-- Citibank (8) / Citi (4) -> 5258073330340253
-- National Australia Bank Limited (3) / ... (NAB) (1) -> 1015001000116227
--
-- Fragmentation also splits the statements-page bank filter, the transactions
-- bank filter, and by_bank in /api/analytics/fees.
--
-- THE VOCABULARY IS OPEN, unlike statement_type in 0013. An unrecognised name
-- passes through tidied, never collapsed to a fallback: a bank this household
-- has never used must be able to arrive without a migration, and destroying its
-- name on first contact is far worse than leaving it unmapped. That is also why
-- there is deliberately NO CHECK constraint here.
--
-- Aliases for banks that only ever arrive one way (Westpac, ANZ, HSBC, ING,
-- AMP, Up, CommBank) are no-ops today and map TO the spelling already in use,
-- so they cannot rename anything. They exist to catch the legal-entity variant
-- the next PDF might print — which is exactly how Zip got here.
--
-- Wise is deliberately left alone: 'Wise Australia Pty Ltd.' is a single
-- spelling, so shortening it would be a rename nobody asked for rather than a
-- merge. Add a branch if a second spelling ever shows up.
--
-- Idempotent: safe to re-run.
CREATE OR REPLACE FUNCTION normalize_bank_name(raw TEXT)
RETURNS TEXT AS $$
DECLARE
v TEXT;
BEGIN
IF raw IS NULL THEN
RETURN NULL;
END IF;
-- Collapse whitespace before matching so 'Zip Pay' and 'Zip Pay' agree.
v := lower(regexp_replace(trim(raw), '\s+', ' ', 'g'));
IF v = '' THEN
-- An empty string is not a name. NULL is how the schema already spells
-- "unknown": bankLabel() renders it 'Manual' and uq_statement_identity
-- excludes it.
RETURN NULL;
END IF;
-- ---- Merges: institutions seen under more than one spelling ----
-- 'Zip Pay', 'ZipPay', 'ZipMoney Payments Pty Ltd', 'ZipMoney Payments Pty
-- Limited'. Anchored so a merchant that merely contains "zip" cannot match.
IF v LIKE 'zip%' THEN
RETURN 'Zip';
END IF;
IF v LIKE '%american express%' OR v LIKE 'amex%' THEN
RETURN 'American Express';
END IF;
-- Citi, Citibank, Citigroup — one card, 5258073330340253.
IF v LIKE 'citi%' THEN
RETURN 'Citibank';
END IF;
IF v LIKE '%national australia bank%' OR v = 'nab' THEN
RETURN 'NAB';
END IF;
-- ---- Defensive: map future legal-entity variants onto the name in use ----
IF v LIKE '%westpac%' THEN
RETURN 'Westpac';
END IF;
IF v = 'anz' OR v LIKE 'anz %' OR v LIKE '%australia and new zealand banking%' THEN
RETURN 'ANZ';
END IF;
IF v LIKE '%commonwealth bank%' OR v LIKE 'commbank%' OR v = 'cba' THEN
RETURN 'Commonwealth Bank';
END IF;
IF v LIKE 'hsbc%' THEN
RETURN 'HSBC';
END IF;
IF v = 'ing' OR v LIKE 'ing %' THEN
RETURN 'ING';
END IF;
IF v = 'amp' OR v LIKE 'amp %' THEN
RETURN 'AMP Bank';
END IF;
IF v = 'up' OR v LIKE 'up bank%' THEN
RETURN 'Up';
END IF;
-- Unknown institution: keep exactly what was extracted, tidied only.
RETURN regexp_replace(trim(raw), '\s+', ' ', 'g');
END;
$$ LANGUAGE plpgsql IMMUTABLE;
-- Normalise on write so the N8N workflow keeps sending whatever the PDF prints.
CREATE OR REPLACE FUNCTION statements_normalize_bank_name_trigger()
RETURNS TRIGGER AS $$
BEGIN
NEW.bank_name := normalize_bank_name(NEW.bank_name);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_statements_normalize_bank_name ON statements;
CREATE TRIGGER trg_statements_normalize_bank_name
BEFORE INSERT OR UPDATE OF bank_name ON statements
FOR EACH ROW EXECUTE FUNCTION statements_normalize_bank_name_trigger();
-- Backfill. Verified collision-free against uq_statement_identity before
-- writing this: no two statements share (normalised name, account_number,
-- billing_end_date).
UPDATE statements
SET bank_name = normalize_bank_name(bank_name)
WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name);
-- account_owner_mappings keys on (bank_name, account_number) with its own
-- UNIQUE constraint, so a rename on `statements` would strand its rows. The
-- table is empty today; this keeps it aligned if it is ever populated before
-- this migration runs somewhere else. ON CONFLICT because two spellings of one
-- bank could otherwise merge into one key.
UPDATE account_owner_mappings
SET bank_name = normalize_bank_name(bank_name)
WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name)
AND NOT EXISTS (
SELECT 1 FROM account_owner_mappings other
WHERE other.bank_name = normalize_bank_name(account_owner_mappings.bank_name)
AND other.account_number = account_owner_mappings.account_number
);
@@ -0,0 +1,164 @@
-- Keep one institution's national entities apart. Fixes 0031.
--
-- 0031 merged spellings by prefix — `citi%` -> 'Citibank'. Every merge it made
-- was provably one account (each split pair shared an account_number), but the
-- PREFIX generalises past that evidence: Citibank Australia and Citibank India
-- are different banks, different accounts, different currency, and `citi%`
-- flattens them into one. Older Citi India statements are queued to load, so
-- this would have merged them on arrival.
--
-- It is not a Citi problem. `hsbc%` and `%american express%` carry the identical
-- defect — all three are global banks operating as separate national entities.
--
-- WHY NOT DISCRIMINATE ON CURRENCY ALONE: Wise holds AUD, EUR and USD accounts
-- under one provider, so a blanket "non-AUD means a different bank" rule would
-- shatter it into three. Wise stays unmapped for exactly this reason — mapping
-- a genuinely multi-currency provider through this function would be wrong, and
-- that is now a documented property rather than an accident.
--
-- THE RULE: the country comes from the name when the name states it, and from
-- the currency otherwise. AUD is home and takes no suffix, so nothing renames
-- today — all 21 existing statements keep the name 0031 gave them.
--
-- 'Citibank' + AUD -> 'Citibank' (unchanged)
-- 'Citi' + AUD -> 'Citibank' (unchanged)
-- 'Citibank Australia'+ AUD -> 'Citibank'
-- 'Citibank India' + INR -> 'Citibank India'
-- 'Citibank N.A.' + INR -> 'Citibank India' (currency carries it)
-- 'HSBC' + AUD -> 'HSBC' (unchanged)
--
-- Degradation is safe but not silent: an India statement whose currency was
-- extracted as AUD and whose name omits the country WILL merge. The tell is a
-- second account_number appearing under one bank_name with a different
-- currency — see the check in CLAUDE.md.
--
-- Idempotent: safe to re-run.
-- 'N.A.' is deliberately absent: it means "National Association", a US legal
-- form that appears on Citibank letterhead worldwide, including India. Reading
-- it as a country marker is precisely the mistake this migration exists to fix.
CREATE OR REPLACE FUNCTION bank_country_suffix(v TEXT, cur TEXT)
RETURNS TEXT AS $$
DECLARE
c TEXT := upper(trim(coalesce(cur, '')));
BEGIN
-- The name wins when it states a country: it is the source's own assertion,
-- and it survives a currency that failed to extract.
IF v LIKE '%australia%' OR v LIKE '% au%' THEN RETURN ''; END IF; -- home
IF v LIKE '%india%' THEN RETURN ' India'; END IF;
IF v LIKE '%singapore%' THEN RETURN ' Singapore'; END IF;
IF v LIKE '%hong kong%' THEN RETURN ' Hong Kong'; END IF;
IF v LIKE '%new zealand%' THEN RETURN ' NZ'; END IF;
IF v LIKE '%united kingdom%' OR v LIKE '% uk%' THEN RETURN ' UK'; END IF;
IF v LIKE '%emirates%' OR v LIKE '% uae%' THEN RETURN ' UAE'; END IF;
-- Otherwise infer from the statement's own currency. AUD is home.
IF c = '' OR c = 'AUD' THEN RETURN ''; END IF;
IF c = 'INR' THEN RETURN ' India'; END IF;
IF c = 'SGD' THEN RETURN ' Singapore'; END IF;
IF c = 'HKD' THEN RETURN ' Hong Kong'; END IF;
IF c = 'NZD' THEN RETURN ' NZ'; END IF;
IF c = 'GBP' THEN RETURN ' UK'; END IF;
IF c = 'AED' THEN RETURN ' UAE'; END IF;
IF c = 'USD' THEN RETURN ' US'; END IF;
-- EUR spans countries and cannot name one, so fall back to the code itself.
-- A distinguishable name beats a wrong one.
RETURN ' ' || c;
END;
$$ LANGUAGE plpgsql IMMUTABLE;
-- `currency` is a second parameter with a default so any existing single-arg
-- call keeps working; the trigger passes NEW.currency.
CREATE OR REPLACE FUNCTION normalize_bank_name(raw TEXT, cur TEXT DEFAULT NULL)
RETURNS TEXT AS $$
DECLARE
v TEXT;
BEGIN
IF raw IS NULL THEN
RETURN NULL;
END IF;
v := lower(regexp_replace(trim(raw), '\s+', ' ', 'g'));
IF v = '' THEN
RETURN NULL;
END IF;
-- ---- Merges: institutions seen under more than one spelling ----
-- Each carries a country suffix, empty for AUD, so one bank's national
-- entities stay apart while its spellings collapse.
IF v LIKE 'zip%' THEN
RETURN 'Zip' || bank_country_suffix(v, cur);
END IF;
IF v LIKE '%american express%' OR v LIKE 'amex%' THEN
RETURN 'American Express' || bank_country_suffix(v, cur);
END IF;
IF v LIKE 'citi%' THEN
RETURN 'Citibank' || bank_country_suffix(v, cur);
END IF;
IF v LIKE '%national australia bank%' OR v = 'nab' THEN
RETURN 'NAB';
END IF;
-- ---- Defensive: map future legal-entity variants onto the name in use ----
IF v LIKE '%westpac%' THEN
RETURN 'Westpac' || bank_country_suffix(v, cur);
END IF;
IF v = 'anz' OR v LIKE 'anz %' OR v LIKE '%australia and new zealand banking%' THEN
RETURN 'ANZ' || bank_country_suffix(v, cur);
END IF;
IF v LIKE '%commonwealth bank%' OR v LIKE 'commbank%' OR v = 'cba' THEN
RETURN 'Commonwealth Bank';
END IF;
IF v LIKE 'hsbc%' THEN
RETURN 'HSBC' || bank_country_suffix(v, cur);
END IF;
IF v = 'ing' OR v LIKE 'ing %' THEN
RETURN 'ING' || bank_country_suffix(v, cur);
END IF;
IF v = 'amp' OR v LIKE 'amp %' THEN
RETURN 'AMP Bank';
END IF;
IF v = 'up' OR v LIKE 'up bank%' THEN
RETURN 'Up';
END IF;
-- Unknown institution: keep exactly what was extracted, tidied only.
-- Wise reaches here, which is what keeps its AUD/EUR/USD accounts together.
RETURN regexp_replace(trim(raw), '\s+', ' ', 'g');
END;
$$ LANGUAGE plpgsql IMMUTABLE;
CREATE OR REPLACE FUNCTION statements_normalize_bank_name_trigger()
RETURNS TRIGGER AS $$
BEGIN
NEW.bank_name := normalize_bank_name(NEW.bank_name, NEW.currency);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Re-point the trigger: it must fire when `currency` changes too, since the
-- country is now partly derived from it.
DROP TRIGGER IF EXISTS trg_statements_normalize_bank_name ON statements;
CREATE TRIGGER trg_statements_normalize_bank_name
BEFORE INSERT OR UPDATE OF bank_name, currency ON statements
FOR EACH ROW EXECUTE FUNCTION statements_normalize_bank_name_trigger();
-- Backfill. Expected to touch nothing: every existing statement is AUD except
-- Wise, which is unmapped.
UPDATE statements
SET bank_name = normalize_bank_name(bank_name, currency)
WHERE bank_name IS DISTINCT FROM normalize_bank_name(bank_name, currency);
@@ -0,0 +1,71 @@
-- One account, one identity — without destroying how the number reads.
--
-- `account_number` carries the same fragmentation `bank_name` had, from the
-- same cause: Gemini copies whatever the PDF prints, and the formatting varies
-- per document.
--
-- ANZ '4085-56264' 4 statements ANZ ACCESS ADVANTAGE
-- ANZ '408556264' 1 statement ACCESS ADVANTAGE <- same account
--
-- That matters because `uq_statement_identity` is
-- (bank_name, account_number, billing_end_date). Re-importing one period under
-- the other spelling evades the duplicate check entirely.
--
-- (It is NOT what caused the documented 31-row / $42,040.68 ANZ duplication.
-- Statements 107/142/143 overlap on *different* end dates, which that index
-- cannot catch at any spelling. Different problem, same table.)
--
-- WHY A GENERATED COLUMN RATHER THAN REWRITING THE VALUE: the raw number is the
-- readable one and some of it is structure, not noise. Up stores
-- '633-123 / 176540052' — a BSB and an account number — and flattening that to
-- '633123176540052' would lose a distinction a human reads at a glance, to fix
-- a machine problem. So the raw text stays exactly as extracted and the
-- comparison key is derived beside it. Same reason the app derives trip
-- participation instead of storing it: two records of one fact drift.
--
-- Separators only. Case and the Amex mask ('XXXX-XXXXXX-01000') are preserved,
-- because 'X' is real information about which digits were redacted, and
-- case-folding an alphanumeric account id could merge two genuinely different
-- ones.
--
-- Verified collision-free before rebuilding the index: no two statements share
-- (bank_name, stripped account_number, billing_end_date).
--
-- Idempotent: safe to re-run.
ALTER TABLE statements
ADD COLUMN IF NOT EXISTS account_number_key TEXT
GENERATED ALWAYS AS (
NULLIF(regexp_replace(coalesce(account_number, ''), '[^0-9A-Za-z]', '', 'g'), '')
) STORED;
COMMENT ON COLUMN statements.account_number_key IS
'Derived from account_number, separators stripped. The identity used by '
'uq_statement_identity so one account under two formats is one account. '
'Never write to it — it is GENERATED. Display account_number instead.';
-- Rebuild the identity index on the derived key. Dropped and recreated rather
-- than added alongside: leaving the old one in place would keep admitting the
-- duplicate it exists to stop.
DROP INDEX IF EXISTS uq_statement_identity;
CREATE UNIQUE INDEX uq_statement_identity
ON statements (bank_name, account_number_key, billing_end_date)
WHERE bank_name IS NOT NULL AND billing_end_date IS NOT NULL;
-- The fragmentation detector, now able to see both halves: one account under
-- two bank names, or one bank under two spellings of one number.
CREATE OR REPLACE VIEW statement_identity_drift AS
SELECT account_number_key,
array_agg(DISTINCT bank_name) AS bank_names,
array_agg(DISTINCT account_number) AS account_numbers,
count(*) AS statements
FROM statements
WHERE account_number_key IS NOT NULL
GROUP BY account_number_key
HAVING count(DISTINCT bank_name) > 1
OR count(DISTINCT account_number) > 1;
COMMENT ON VIEW statement_identity_drift IS
'Non-empty means an account is fragmented: normalize_bank_name() cannot know '
'that two names for a bank it has never seen are one institution, and this is '
'how that always shows up. Check it after loading a new bank.';
@@ -0,0 +1,74 @@
-- Intake router (plans/intake-router-slice-plan.md §11 C1, smarthome repo).
--
-- One upload point for receipts — a screenshot, a PDF export, a pasted body —
-- that reads what it was given and does every step that applies: fill the
-- empty `line_items` on the order the email pipeline already booked, hand a
-- grocery receipt to pantry, or ask when it cannot tell.
--
-- Every batch is persisted BEFORE anything is extracted, for two reasons:
-- 1. The Slack card it produces carries buttons ("Use this", "Create",
-- "Groceries or a meal?") that are pressed minutes later in a fresh
-- request with nothing but the batch id to go on.
-- 2. A re-share of an already-handled order — a second crop, the PDF after
-- the screenshot — must be recognised, and the only durable evidence of
-- the first share is this row (`file_sha256` overlap, or the same
-- (platform, merchant, date, total) header already resolved).
--
-- `id` is the uuid n8n mints per webhook call, so a retried delivery is the
-- same batch rather than a second one. `header` holds the extracted
-- (merchant, order_date, total, platform, ...) and `items` the verbatim item
-- list, so a later tap can enrich or create without re-reading the files.
--
-- Idempotent: safe to re-run.
CREATE TABLE IF NOT EXISTS intake_batches (
id uuid PRIMARY KEY,
source text NOT NULL,
shared_by text,
file_sha256 text[] NOT NULL DEFAULT '{}',
header jsonb,
items jsonb,
kind text,
status text NOT NULL,
transaction_id integer,
capture_id integer,
-- The undo baseline: what `expense_metadata.line_items` held before this
-- batch first replaced it. Recorded here rather than returned from the write,
-- because pantry re-calls the enrich endpoint on every confirm and after a
-- timeout — a caller that hands back whatever it just read gives a retry its
-- OWN write as the "previous" state, and undo then restores the thing it was
-- meant to reverse. Written under COALESCE, so only the first snapshot sticks.
previous_line_items jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
resolved_at timestamptz
);
-- For a database that already ran an earlier draft of this file.
ALTER TABLE intake_batches ADD COLUMN IF NOT EXISTS previous_line_items jsonb;
COMMENT ON TABLE intake_batches IS
'One row per upload batch through POST /api/intake. Persisted before '
'extraction so Slack taps can act on it later and re-shares are detected. '
'kind: grocery|food|ask|held|unsupported. status: received|enriched|created|'
'candidates|forwarded_to_pantry|needs_header|duplicate|unsupported|held|error.';
-- Overlap lookups for the re-share check (`file_sha256 && $1`).
CREATE INDEX IF NOT EXISTS intake_batches_file_sha256_gin
ON intake_batches USING gin (file_sha256);
CREATE INDEX IF NOT EXISTS intake_batches_transaction_id
ON intake_batches (transaction_id) WHERE transaction_id IS NOT NULL;
-- The "Groceries or a meal?" answer, remembered per merchant (plan §2, Codex
-- #6). A tiny answer table, not merchant master-data: the pattern list in code
-- decides the obvious cases and this catches the `Romeo's Food Hall`s. Keyed on
-- the normalised merchant name the router already computes for matching.
CREATE TABLE IF NOT EXISTS intake_merchant_kinds (
merchant_key text PRIMARY KEY,
kind text NOT NULL CHECK (kind IN ('grocery', 'food')),
created_at timestamptz NOT NULL DEFAULT now()
);
COMMENT ON TABLE intake_merchant_kinds IS
'Per-merchant answer to "Groceries or a meal?" from the intake Slack card. '
'Consulted after the grocery pattern list; an unknown merchant asks once.';
+32
View File
@@ -135,6 +135,11 @@ model statements {
card_name String?
account_type String?
account_number String
/// GENERATED ALWAYS (migration 0033) — account_number with separators
/// stripped, so '4085-56264' and '408556264' are one account. Backs
/// uq_statement_identity. Read-only: never write to it, display
/// account_number instead.
account_number_key String?
billing_start_date DateTime? @db.Date
billing_end_date DateTime? @db.Date
total_amount_due Decimal? @db.Decimal(12, 2)
@@ -263,3 +268,30 @@ model rule_apply_runs {
reverted_at DateTime?
snapshot Json @default("[]")
}
// 0034 — intake router. One row per upload batch, written before extraction so
// Slack taps can act on it later and a re-share is detected. Columns are the
// §11 C1 contract in plans/intake-router-slice-plan.md (smarthome repo).
model intake_batches {
id String @id @db.Uuid
source String
shared_by String?
file_sha256 String[] @default([])
header Json?
items Json?
kind String?
status String
transaction_id Int?
capture_id Int?
// The undo baseline, written once per batch (see 0034 and line-items.ts).
previous_line_items Json?
created_at DateTime @default(now()) @db.Timestamptz(6)
resolved_at DateTime? @db.Timestamptz(6)
}
// 0034 — the remembered answer to "Groceries or a meal?" per merchant.
model intake_merchant_kinds {
merchant_key String @id
kind String
created_at DateTime @default(now()) @db.Timestamptz(6)
}
+66
View File
@@ -0,0 +1,66 @@
#!/bin/bash
# Resolve the integration-test DSN at run time and (re)write .env.test.
#
# Why this exists (#161): .env.test used to be written ONCE by
# setup-test-db.sh with the Postgres container's bridge IP baked in. Container
# IPs are reassigned on recreation, so the file rotted silently — by
# 2026-08-21 its 172.22.0.18 belonged to *gitea*, and an earlier drift had the
# suite pointing its TRUNCATE at postgres-pantry. It only failed safe because
# the credentials happened not to match.
#
# The fix is to stop persisting a host at all: this runs as `pretest:integration`,
# so the DSN is re-derived from the live container immediately before every run.
#
# Resolution order:
# 1. An explicit TEST_DATABASE_URL in the environment wins (CI, remote DB).
# 2. The container's published host port (`docker port`) — postgres-personal
# publishes 127.0.0.1:5432 since the 2026-08-17 PG17 migration. Preferred
# because a loopback address cannot be reassigned to another container.
# 3. The container's bridge IP, for a host where the port is not published.
#
# helpers.ts::assertTestDatabase is the backstop: it refuses to TRUNCATE
# anything whose current_database() is not `personal_test`.
set -euo pipefail
PG_CONTAINER=${PG_CONTAINER:-postgres-personal}
PG_USER=${PG_USER:-personal}
PG_PASS=${PG_PASS:-personalpassword123}
TEST_DB=${TEST_DB:-personal_test}
ENV_FILE="$(cd "$(dirname "$0")/.." && pwd)/.env.test"
if [ -n "${TEST_DATABASE_URL:-}" ]; then
DSN="$TEST_DATABASE_URL"
SOURCE="TEST_DATABASE_URL"
else
if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
echo "ERROR: container '$PG_CONTAINER' not found. Start it, or set TEST_DATABASE_URL." >&2
exit 1
fi
# `docker port` prints e.g. "127.0.0.1:5432" (possibly several lines).
PUBLISHED=$(docker port "$PG_CONTAINER" 5432/tcp 2>/dev/null | head -1 || true)
if [ -n "$PUBLISHED" ]; then
HOSTPORT="$PUBLISHED"
SOURCE="published port"
else
IP=$(docker inspect "$PG_CONTAINER" \
--format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' 2>/dev/null \
| tr ' ' '\n' | grep -v '^$' | head -1)
if [ -z "$IP" ]; then
echo "ERROR: '$PG_CONTAINER' has neither a published 5432 nor a bridge IP." >&2
exit 1
fi
HOSTPORT="$IP:5432"
SOURCE="bridge IP"
fi
DSN="postgresql://$PG_USER:$PG_PASS@$HOSTPORT/$TEST_DB"
fi
printf 'DATABASE_URL=%s\n' "$DSN" > "$ENV_FILE"
# Never print the password.
echo "resolve-test-db: ${DSN%%://*}://***@${DSN#*@} (via $SOURCE)"
+14 -18
View File
@@ -1,25 +1,24 @@
#!/bin/bash
# Creates the personal_test database and writes .env.test
# Run once before integration tests: npm run test:setup
# Creates the personal_test database and copies the production schema into it.
# Run once (and after any migration) before integration tests: npm run test:setup
#
# It no longer writes a host into .env.test. That was the #161 defect: the file
# was written once with the container's bridge IP and then rotted as the IP was
# reassigned. scripts/resolve-test-db.sh now re-derives the DSN before every
# integration run (wired as `pretest:integration`), and this script just calls it
# at the end so a fresh setup leaves a usable .env.test behind.
set -e
PG_CONTAINER=postgres-personal
PG_USER=personal
PG_PASS=personalpassword123
TEST_DB=personal_test
PG_CONTAINER=${PG_CONTAINER:-postgres-personal}
PG_USER=${PG_USER:-personal}
TEST_DB=${TEST_DB:-personal_test}
# Discover the container's bridge IP (accessible from the host on Linux)
PG_HOST=$(docker inspect "$PG_CONTAINER" \
--format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' 2>/dev/null | head -1)
if [ -z "$PG_HOST" ]; then
if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
echo "ERROR: Could not find container '$PG_CONTAINER'"
exit 1
fi
echo "Postgres container at $PG_HOST:5432"
# Create test database (ignore error if it already exists)
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d postgres \
-c "CREATE DATABASE $TEST_DB;" 2>/dev/null || true
@@ -31,9 +30,6 @@ docker exec "$PG_CONTAINER" pg_dump -U "$PG_USER" --schema-only "$PG_USER" \
echo "Schema ready."
# Write .env.test
cat > "$(dirname "$0")/../.env.test" << EOF
DATABASE_URL=postgresql://$PG_USER:$PG_PASS@$PG_HOST:5432/$TEST_DB
EOF
bash "$(dirname "$0")/resolve-test-db.sh"
echo ".env.test written — integration tests can now run with: npm run test:integration"
echo "Integration tests can now run with: npm run test:integration"
@@ -0,0 +1,531 @@
import { describe, it, expect, beforeAll, beforeEach, afterAll, vi } from "vitest";
import type { NextRequest } from "next/server";
import { createPool, mockDbWithPool, resetDB } from "./helpers";
import {
comparisonBasis,
baselineMonths,
baselineAverage,
spendThrough,
prevMonth,
} from "../../lib/partial-month";
/**
* The three Priority 0 behaviours that shipped without a test (#155).
*
* `docs/ui-information-architecture-review.md` claimed Priority 0 was guarded;
* `analytics-sql.test.ts` only ever covered the three SQL *fragments*
* (EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY, INVESTMENT_SIGNED). Items 2, 3
* and 6 of that list had nothing:
*
* 2. the spend-pace series must use the same spend semantics as the headline
* printed directly above it;
* 3. fees and interest must be bounded by an explicit, reported period;
* 6. a month in progress must never be compared against complete ones.
*
* All three fail *silently* when they regress — no error, no empty screen, just
* a number that quietly disagrees with the one beside it. That is precisely the
* class the review flagged and the reason these are worth the fixture cost.
*
* The clock is frozen for every test in this file. Two of the three behaviours
* are defined relative to "now" (which month is in progress, how far back the
* default fees window reaches), so against the real clock the fixtures would
* drift out of the routes' windows and the assertions would go vacuous — which
* is the failure mode this file exists to prevent, one level up. Only `Date` is
* faked: `pg` uses real timers for its connection handling.
*/
const pool = createPool();
mockDbWithPool(pool);
/** Frozen wall clock: the 15th, so the current month is genuinely half-done. */
const NOW = new Date(2026, 7, 15, 10, 0, 0); // 2026-08-15, local time
const CURRENT_MONTH = "2026-08";
const ELAPSED = 15;
let monthlyGET: (req: NextRequest) => Promise<Response>;
let dailyGET: (req: NextRequest) => Promise<Response>;
let feesGET: (req: NextRequest) => Promise<Response>;
let ownerId: number;
// One cast at the boundary: these are the two properties the analytics routes
// read (`url` for the query string, `headers` for the auth header, which is
// mocked out anyway), not a whole NextRequest.
const req = (path: string) =>
({ url: `http://localhost${path}`, headers: { get: () => null } }) as unknown as NextRequest;
const json = async (res: Response) => (await res.json()) as Record<string, unknown>;
beforeAll(async () => {
vi.useFakeTimers({ toFake: ["Date"] });
vi.setSystemTime(NOW);
vi.doMock("@/lib/auth", () => ({
getCurrentUser: async () => ({ id: ownerId, name: "Owner", email: "o@x" }),
}));
({ GET: monthlyGET } = await import("../../app/api/analytics/monthly/route"));
({ GET: dailyGET } = await import("../../app/api/analytics/daily/route"));
({ GET: feesGET } = await import("../../app/api/analytics/fees/route"));
});
afterAll(() => {
vi.useRealTimers();
});
beforeEach(async () => {
await resetDB(pool);
const a = await pool.query(`INSERT INTO participants (name) VALUES ('Owner') RETURNING id`);
const b = await pool.query(`INSERT INTO participants (name) VALUES ('Other') RETURNING id`);
ownerId = a.rows[0].id;
otherId = b.rows[0].id;
});
let otherId: number;
/** A manual transaction owned by `owner`. Returns its id. */
async function txn(over: {
date: string;
amount: number;
type?: string;
category?: string;
owner?: number;
description?: string;
interest_amount?: number;
reconciled_with_id?: number;
}): Promise<number> {
const r = await pool.query(
`INSERT INTO transactions
(owner_id, statement_id, transaction_date, description, amount,
transaction_type, category, interest_amount, reconciled_with_id, row_index)
VALUES ($1, NULL, $2, $3, $4, $5, $6, $7, $8, 0) RETURNING id`,
[
over.owner ?? ownerId,
over.date,
over.description ?? `fixture ${over.date} ${over.amount}`,
over.amount,
over.type ?? "debit",
over.category ?? "groceries",
over.interest_amount ?? null,
over.reconciled_with_id ?? null,
]
);
return r.rows[0].id as number;
}
async function split(txnId: number, participantId: number, pct: number) {
await pool.query(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent)
VALUES ($1, $2, $3)`,
[txnId, participantId, pct]
);
}
async function tag(txnId: number, name: string) {
const t = await pool.query(
`INSERT INTO tags (name) VALUES ($1) ON CONFLICT (name) DO UPDATE SET name = EXCLUDED.name RETURNING id`,
[name]
);
await pool.query(
`INSERT INTO transaction_tags (transaction_id, tag_id) VALUES ($1, $2)`,
[txnId, t.rows[0].id]
);
}
// ───────────────────────────────────────────────────────────────────────────
// Item 2 — the pace chart and the headline must agree
// ───────────────────────────────────────────────────────────────────────────
/**
* The pace chart used to sum gross `amount_aud ?? amount` over
* `transaction_type = 'debit'` in the browser. Measured on live data it ended
* July at $4,747.31 against a headline of $3,597.10 — a 32% overstatement of
* the number printed directly above it, with both labelled "spend".
*
* `/api/analytics/daily` fixed that by building the series from the same
* fragments as `/monthly`. Nothing has been asserting it since, and the two
* routes are separate files: the drift they were merged to prevent can
* reappear the moment one is edited alone.
*
* Every fixture below is a shape where the naive gross-debit sum and the real
* spend definition disagree. The last test proves that — without it, parity
* could hold simply because the fixtures were too plain to discriminate, and
* the test would pass against a route that had regressed.
*/
describe("Priority 0 item 2 — spend pace matches the headline", () => {
/** Everything the two routes are supposed to treat identically. */
async function seedDivergentMonth(month: string) {
const d = (day: number) => `${month}-${String(day).padStart(2, "0")}`;
// Plain debit — both definitions agree on this one.
await txn({ date: d(2), amount: 100, category: "groceries" });
// Half mine. Gross says 200; my share is 100.
const shared = await txn({ date: d(3), amount: 200, category: "groceries" });
await split(shared, ownerId, 50);
await split(shared, otherId, 50);
// A refund. Ignored entirely by a debit-only sum; must cancel spend here.
await txn({ date: d(5), amount: 40, type: "refund", category: "groceries" });
// Fee and interest rows: spend, but not type 'debit'.
await txn({ date: d(6), amount: 15, type: "fee", category: "fees" });
await txn({ date: d(7), amount: 25, type: "interest", category: "fees" });
// An itemised loan repayment: type 'payment', so a debit-only sum skips it,
// but its interest portion is real spend and its principal is not.
await txn({
date: d(8),
amount: 1200,
type: "payment",
category: "loan",
interest_amount: 300,
});
// The manual row a statement line superseded — double count if counted.
const survivor = await txn({ date: d(9), amount: 60, category: "dining" });
await txn({ date: d(9), amount: 60, category: "dining", reconciled_with_id: survivor });
// Categories that are not spend at all.
await txn({ date: d(10), amount: 5000, category: "transfers" });
await txn({ date: d(11), amount: 2000, category: "investment" });
await txn({ date: d(12), amount: 3000, type: "credit", category: "income" });
// A `family`-tagged row: excluded from budget spend by EXCLUDE_NON_BUDGET_TAGS.
const family = await txn({ date: d(13), amount: 500, category: "groceries" });
await tag(family, "family");
// Someone else's row I hold a split on — mine only to the extent of the split.
const theirs = await txn({ date: d(14), amount: 80, category: "dining", owner: otherId });
await split(theirs, ownerId, 25);
await split(theirs, otherId, 75);
}
/** Sum the sparse daily map for one month. */
const sumDaily = (daily: Record<string, Record<number, number>>, month: string) =>
Object.values(daily[month] ?? {}).reduce((s, v) => s + v, 0);
it("the daily series sums to the monthly headline, to the cent", async () => {
await seedDivergentMonth("2026-06");
const monthly = await json(await monthlyGET(req("/api/analytics/monthly?months=6")));
const daily = await json(await dailyGET(req("/api/analytics/daily?months=6")));
const headline = (monthly.totals as Record<string, { spent: number }>)["2026-06"].spent;
const pace = sumDaily(daily.daily as Record<string, Record<number, number>>, "2026-06");
// Cents, not exact float equality: the two routes group at different grains
// (month+category vs month+day+category) and both carry 4dp, so they can
// differ below a cent by construction. The displayed grain is what has to
// agree — that IS the bug, the chart ending the month a few cents off the
// headline it sits under.
expect(Math.round(pace * 100) / 100).toBe(headline);
});
it("agrees per category, not just in total", async () => {
// A total can match while two categories are wrong in opposite directions,
// and the category drill-down is a real surface on this page.
await seedDivergentMonth("2026-06");
const monthly = await json(await monthlyGET(req("/api/analytics/monthly?months=6")));
const daily = await json(await dailyGET(req("/api/analytics/daily?months=6")));
const byCategory = (daily.byCategory as Record<string, Record<string, Record<number, number>>>)["2026-06"];
const rows = monthly.rows as { category: string; spent: Record<string, number> }[];
expect(Object.keys(byCategory).sort()).toEqual(
rows.filter((r) => r.spent["2026-06"] !== undefined).map((r) => r.category).sort()
);
for (const r of rows) {
const expected = r.spent["2026-06"];
if (expected === undefined) continue;
const got = Object.values(byCategory[r.category] ?? {}).reduce((s, v) => s + v, 0);
expect([r.category, Math.round(got * 100) / 100]).toEqual([
r.category,
Math.round(expected * 100) / 100,
]);
}
});
it("holds across every month in the window, not only the seeded one", async () => {
await seedDivergentMonth("2026-05");
await seedDivergentMonth("2026-06");
await seedDivergentMonth("2026-07");
const monthly = await json(await monthlyGET(req("/api/analytics/monthly?months=6")));
const daily = await json(await dailyGET(req("/api/analytics/daily?months=6")));
const totals = monthly.totals as Record<string, { spent: number }>;
const dailyMap = daily.daily as Record<string, Record<number, number>>;
for (const m of monthly.months as string[]) {
expect([m, Math.round(sumDaily(dailyMap, m) * 100) / 100]).toEqual([m, totals[m].spent]);
}
});
it("the fixtures actually discriminate — a gross debit sum does NOT agree", async () => {
// The placebo. Parity between two identical-by-construction queries is easy
// to assert and worthless if no fixture can tell the definitions apart.
// This pins the gap the old client-side series had.
await seedDivergentMonth("2026-06");
const monthly = await json(await monthlyGET(req("/api/analytics/monthly?months=6")));
const headline = (monthly.totals as Record<string, { spent: number }>)["2026-06"].spent;
const naive = await pool.query<{ total: string }>(
`SELECT COALESCE(SUM(COALESCE(amount_aud, amount)), 0)::numeric(14,2) AS total
FROM transactions
WHERE transaction_type = 'debit'
AND transaction_date >= '2026-06-01' AND transaction_date < '2026-07-01'`
);
expect(Number(naive.rows[0].total)).toBeGreaterThan(headline);
});
});
// ───────────────────────────────────────────────────────────────────────────
// Item 3 — fees and interest are bounded by an explicit, reported period
// ───────────────────────────────────────────────────────────────────────────
/**
* `/api/analytics/fees` used to aggregate every statement ever imported with no
* date filter, and the UI printed the result with no period label — so a
* lifetime-to-date total read as a current-period one and grew forever. It was
* overstating the last 12 months by roughly $2,700 of fees.
*
* The regression is silent in the worst way: the number is always plausible and
* always too big, and it gets worse the longer the app is used.
*/
describe("Priority 0 item 3 — fees and interest carry a date range", () => {
/** A statement dated by the period it covers, carrying fees and interest. */
async function statement(billingEnd: string, fees: number, interest: number, bank = "Fixture Bank") {
const r = await pool.query(
`INSERT INTO statements
(filename, bank_name, account_number, billing_end_date, currency,
fees_charged, interest_charged, owner_id)
VALUES ($1, $2, '0000', $3, 'AUD', $4, $5, $6) RETURNING id`,
[`fees-${billingEnd}-${bank}.pdf`, bank, billingEnd, fees, interest, ownerId]
);
return r.rows[0].id as number;
}
beforeEach(async () => {
// Inside the default 12-month window (opens 2025-09-01 from a 2026-08-15 now).
await statement("2026-07-31", 10, 5);
await statement("2025-10-31", 20, 7);
// Outside it — the tail that used to be silently included.
await statement("2024-06-30", 1000, 500);
await statement("2019-01-31", 2500, 900);
});
it("defaults to the last 12 months and says so", async () => {
const body = await json(await feesGET(req("/api/analytics/fees")));
const period = body.period as { months: number; from: string; to: string; all_time: boolean };
expect(period.months).toBe(12);
expect(period.all_time).toBe(false);
// The window opens on the 1st of the month 11 months back, and toDateStr()
// must not shift it a day earlier — `toISOString()` on a local-midnight
// Date does exactly that in any timezone east of Greenwich, which is how
// every analytics window used to be a day early.
expect(period.from).toBe("2025-09-01");
expect(period.to).toBe("2026-09-01");
expect(body.total_fees).toBe(30);
expect(body.total_interest).toBe(12);
});
it("excludes the older tail — the $2,700-of-fees overstatement", async () => {
const body = await json(await feesGET(req("/api/analytics/fees")));
// 3530 of fees exist; only 30 are in the last 12 months.
expect(body.total_fees).toBe(30);
expect(body.total_fees).not.toBe(3530);
});
it("months=0 asks for all time deliberately, and is labelled as such", async () => {
const body = await json(await feesGET(req("/api/analytics/fees?months=0")));
const period = body.period as { months: number; from: null; to: null; all_time: boolean };
expect(period.all_time).toBe(true);
// A null range is the honest answer for an unbounded total; a from/to here
// would be a period claim the number does not actually make.
expect(period.from).toBeNull();
expect(period.to).toBeNull();
expect(body.total_fees).toBe(3530);
expect(body.total_interest).toBe(1412);
});
it("honours a narrower explicit window", async () => {
const body = await json(await feesGET(req("/api/analytics/fees?months=3")));
const period = body.period as { months: number; from: string };
expect(period.months).toBe(3);
expect(period.from).toBe("2026-06-01");
// Only the 2026-07-31 statement falls inside three months.
expect(body.total_fees).toBe(10);
expect(body.total_interest).toBe(5);
});
it("bounds the transaction-level fee lines by the same window", async () => {
// Two halves of the same answer, and only the statement half was ever
// date-filtered in the original. A bounded total beside an unbounded list
// is the same defect wearing the fix.
await txn({ date: "2026-07-10", amount: 12, type: "fee", category: "fees" });
await txn({ date: "2020-02-10", amount: 99, type: "fee", category: "fees" });
const bounded = await json(await feesGET(req("/api/analytics/fees")));
const all = await json(await feesGET(req("/api/analytics/fees?months=0")));
expect((bounded.transactions as unknown[]).length).toBe(1);
expect((all.transactions as unknown[]).length).toBe(2);
});
it("clamps a nonsense months parameter instead of unbounding the total", async () => {
// A NaN reaching `months` would make `new Date(y, NaN, 1)` an Invalid Date
// and the window meaningless. Falling back to 12 keeps the default claim.
const body = await json(await feesGET(req("/api/analytics/fees?months=banana")));
expect((body.period as { months: number }).months).toBe(12);
expect(body.total_fees).toBe(30);
});
});
// ───────────────────────────────────────────────────────────────────────────
// Item 6 — a month in progress is never compared against complete ones
// ───────────────────────────────────────────────────────────────────────────
/**
* Two rules, both silent when broken (see src/lib/partial-month.ts):
*
* 1. the in-progress month never enters a baseline;
* 2. when it is the month selected, both sides of every comparison are
* truncated to the same day.
*
* These are driven off real `/api/analytics/daily` output rather than a
* hand-built map, because rule 2 depends on that route's day-of-month
* granularity — a route returning month totals only would make the rule
* unimplementable while every pure-function test still passed.
*/
describe("Priority 0 item 6 — partial-month comparisons", () => {
/** `perDay` dollars on each of days 1..n of `month`. */
async function evenMonth(month: string, perDay: number, days: number) {
for (let d = 1; d <= days; d++) {
await txn({ date: `${month}-${String(d).padStart(2, "0")}`, amount: perDay, category: "groceries" });
}
}
async function dailyMap() {
const body = await json(await dailyGET(req("/api/analytics/daily?months=12")));
return body.daily as Record<string, Record<number, number>>;
}
it("recognises the in-progress month and the day to compare through", () => {
expect(comparisonBasis(CURRENT_MONTH, NOW)).toEqual({ isPartial: true, compareDay: ELAPSED });
// A complete month compares through its last day, so the same code path
// serves both cases and `spendThrough` becomes a no-op.
expect(comparisonBasis("2026-06", NOW)).toEqual({ isPartial: false, compareDay: 30 });
expect(comparisonBasis("2026-02", NOW)).toEqual({ isPartial: false, compareDay: 28 });
});
it("keeps the in-progress month out of the baseline", () => {
const totals: Record<string, { spent: number }> = {
"2026-05": { spent: 100 },
"2026-06": { spent: 100 },
"2026-07": { spent: 100 },
[CURRENT_MONTH]: { spent: 40 }, // half a month of spend
};
const months = ["2026-05", "2026-06", "2026-07", CURRENT_MONTH];
// Baseline for a complete month: every other complete month with data.
expect(baselineMonths(months, "2026-07", totals, NOW)).toEqual(["2026-05", "2026-06"]);
// Baseline for the month in progress: itself excluded twice over.
expect(baselineMonths(months, CURRENT_MONTH, totals, NOW)).toEqual([
"2026-05",
"2026-06",
"2026-07",
]);
// The partial month is never in it. Averaging its 40 in would drag a
// baseline of 100 down to 85 and make an ordinary month look extravagant.
expect(baselineMonths(months, "2026-07", totals, NOW)).not.toContain(CURRENT_MONTH);
});
it("drops a month with no data rather than averaging a zero into the baseline", () => {
const totals: Record<string, { spent: number }> = { "2026-05": { spent: 100 }, "2026-06": { spent: 0 } };
expect(baselineMonths(["2026-05", "2026-06"], "2026-07", totals, NOW)).toEqual(["2026-05"]);
});
it("compares the month in progress through the same day of each prior month", async () => {
// $10/day everywhere. Prior months ran the full month; the current one has
// only reached the 15th, so it holds $150 against their $300-odd.
await evenMonth("2026-06", 10, 30);
await evenMonth("2026-07", 10, 31);
await evenMonth(CURRENT_MONTH, 10, ELAPSED);
const daily = await dailyMap();
const totals: Record<string, { spent: number }> = {
"2026-06": { spent: 300 },
"2026-07": { spent: 310 },
[CURRENT_MONTH]: { spent: 150 },
};
const months = ["2026-06", "2026-07", CURRENT_MONTH];
const { isPartial, compareDay } = comparisonBasis(CURRENT_MONTH, NOW);
const baseline = baselineMonths(months, CURRENT_MONTH, totals, NOW);
const mine = spendThrough(daily[CURRENT_MONTH], compareDay);
const avg = baselineAverage(baseline, { isPartial, compareDay, daily, totals });
// Truncated on both sides, this month is exactly typical.
expect(mine).toBe(150);
expect(avg).toBe(150);
expect(Math.round(((mine - avg) / avg) * 100)).toBe(0);
// Untruncated — the bug — it reads as 51% below average on identical
// spending, purely because half the month has not happened yet.
const naiveAvg = baseline.reduce((s, m) => s + totals[m].spent, 0) / baseline.length;
expect(Math.round(((mine - naiveAvg) / naiveAvg) * 100)).toBe(-51);
});
it("leaves a complete month compared on full totals", async () => {
await evenMonth("2026-06", 10, 30);
await evenMonth("2026-07", 20, 31);
const daily = await dailyMap();
const totals: Record<string, { spent: number }> = { "2026-06": { spent: 300 }, "2026-07": { spent: 620 } };
const months = ["2026-06", "2026-07"];
const { isPartial, compareDay } = comparisonBasis("2026-07", NOW);
const baseline = baselineMonths(months, "2026-07", totals, NOW);
expect(isPartial).toBe(false);
// Truncation must not quietly apply to a finished month: June's full 30
// days are the right comparison for July's full 31.
expect(baselineAverage(baseline, { isPartial, compareDay, daily, totals })).toBe(300);
});
it("truncates the top-movers comparison on both sides", async () => {
// The movers row asks "biggest category moves vs last month". With one side
// full and the other half-done, every category reads as a large decrease.
await evenMonth("2026-07", 10, 31); // $310 in July
await evenMonth(CURRENT_MONTH, 10, ELAPSED); // $150 so far in August
const body = await json(await dailyGET(req("/api/analytics/daily?months=12")));
const byCategory = body.byCategory as Record<string, Record<string, Record<number, number>>>;
const { compareDay } = comparisonBasis(CURRENT_MONTH, NOW);
const pm = prevMonth(CURRENT_MONTH);
expect(pm).toBe("2026-07");
const now = spendThrough(byCategory[CURRENT_MONTH]?.groceries, compareDay);
const before = spendThrough(byCategory[pm]?.groceries, compareDay);
expect(now - before).toBe(0);
// Against July's full month it would report a $160 fall that never happened.
const fullBefore = Object.values(byCategory[pm].groceries).reduce((s, v) => s + v, 0);
expect(now - fullBefore).toBe(-160);
});
it("crosses a year boundary correctly", () => {
expect(prevMonth("2026-01")).toBe("2025-12");
expect(comparisonBasis("2025-12", NOW)).toEqual({ isPartial: false, compareDay: 31 });
});
it("handles February in a leap year", () => {
// 2028 is a leap year; a hardcoded 28 would silently drop the 29th.
expect(comparisonBasis("2028-02", new Date(2028, 5, 1)).compareDay).toBe(29);
});
});
+221 -3
View File
@@ -1,11 +1,17 @@
import { describe, it, expect } from "vitest";
import { describe, it, expect, beforeAll } from "vitest";
import { queryRaw, queryRow } from "../../lib/db";
import {
EXCLUDE_RECONCILED_SOURCE,
NATIVE_CURRENCY,
AMOUNT_UNCONVERTED,
INVESTMENT_SIGNED,
MY_SPEND_SCOPE,
NET_SPEND_ROWS,
SPEND_SIGNED,
myShare,
mySplitOf,
} from "../../lib/analytics-sql";
import { MY_SHARE_PCT } from "../../lib/queries";
/**
* These fragments are the ones that drifted.
@@ -103,9 +109,17 @@ describe("NATIVE_CURRENCY", () => {
// Opposite convention: on an AUD statement, `amount` is AUD and
// foreign_currency_code merely notes what was originally charged. Reading
// the foreign code here would mislabel an AUD row as USD.
// owner_id is NOT NULL DEFAULT 1 with an FK to participants, so this insert
// relied on a participant with id 1 already existing — true only when a
// sibling test file had seeded one and left it behind. On a freshly reset
// database it failed the FK. Own the fixture instead of inheriting it.
const owner = await queryRow<{ id: number }>(
`INSERT INTO participants (name) VALUES ('Analytics fixture — owner') RETURNING id`
);
const stmt = await queryRow<{ id: number }>(
`INSERT INTO statements (bank_name, account_number, billing_end_date, currency, filename)
VALUES ('Analytics Fixture Bank','0000','2026-03-31','AUD','analytics-fixture.pdf') RETURNING id`
`INSERT INTO statements (bank_name, account_number, billing_end_date, currency, filename, owner_id)
VALUES ('Analytics Fixture Bank','0000','2026-03-31','AUD','analytics-fixture.pdf', $1) RETURNING id`,
[owner!.id]
);
const id = await scratchTxn(
"transaction_date, description, amount, amount_aud, transaction_type, statement_id, foreign_currency_amount, foreign_currency_code",
@@ -183,3 +197,207 @@ describe("INVESTMENT_SIGNED", () => {
expect(await signedValue(id)).toBe(-1500);
});
});
/**
* Whose expense is it — the gate that decides, and the share that scales it.
*
* Both fragments changed together on 2026-08-15 and neither is safe alone.
* `OWNER_SCOPE = $1` was the gate on every spend analytic, with `mySplitOf`
* scaling within it, so a row someone else paid for contributed nothing to my
* spend no matter what share I held — 167 rows / $3,210.91 of my own share
* missing from Jan–Jul 2026, while the balance queries booked the debt
* correctly. Widening the gate without also fixing `myShare` is worse than
* leaving it: the old remainder fallback hands the viewer the PAYER's share on
* a row with no split of theirs, which on an unsplit row is the full amount.
*
* So the discriminating cases here are all ones where I hold NO split row on
* someone else's transaction. That is where old and new disagree, and it is the
* only place the widened gate could do damage.
*/
describe("MY_SPEND_SCOPE / myShare", () => {
let me = 0;
let them = 0;
let third = 0;
beforeAll(async () => {
const mk = async (name: string) =>
(await queryRow<{ id: number }>(
`INSERT INTO participants (name) VALUES ($1) RETURNING id`,
[name]
))!.id;
me = await mk("Share fixture — viewer");
them = await mk("Share fixture — payer");
third = await mk("Share fixture — third party");
});
/** A transaction owned by `owner`, optionally split, optionally overridden. */
async function fixture(
label: string,
owner: number,
splits: [number, number][] = [],
myShareOverride?: number
) {
const id = await scratchTxn(
"transaction_date, description, amount, transaction_type, category, owner_id",
`'2026-06-01', $1, 100.00, 'debit', 'groceries', $2`,
[`Share fixture — ${label}`, owner]
);
for (const [pid, pct] of splits) {
await queryRaw(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent)
VALUES ($1, $2, $3)`,
[id, pid, pct]
);
}
if (myShareOverride !== undefined) {
await queryRaw(
`INSERT INTO transaction_overrides (transaction_id, my_share_percent) VALUES ($1, $2)`,
[id, myShareOverride]
);
}
return id;
}
const JOINS = `
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1`;
async function inScope(id: number, viewer: number): Promise<boolean> {
const rows = await queryRaw(
`SELECT t.id ${JOINS} WHERE t.id = $2 AND ${MY_SPEND_SCOPE()}`,
[viewer, id]
);
return rows.length === 1;
}
async function share(id: number, viewer: number): Promise<number> {
const row = await queryRow<{ v: string }>(
`SELECT (${myShare()})::numeric AS v ${JOINS} WHERE t.id = $2`,
[viewer, id]
);
return Number(row!.v);
}
/** What this row contributes to the viewer's net spend. */
async function spend(id: number, viewer: number): Promise<number> {
const row = await queryRow<{ v: string | null }>(
`SELECT SUM(${mySplitOf(SPEND_SIGNED)})::numeric(14,4) AS v ${JOINS}
WHERE t.id = $2 AND ${MY_SPEND_SCOPE()} AND ${NET_SPEND_ROWS}`,
[viewer, id]
);
return Number(row?.v ?? 0);
}
describe("the gate", () => {
it("admits a row someone else paid for that I hold a split on", async () => {
const id = await fixture("their shop, my half", them, [[me, 50], [them, 50]]);
expect(await inScope(id, me)).toBe(true);
});
it("still admits a row I paid for with no splits at all", async () => {
const id = await fixture("mine alone", me);
expect(await inScope(id, me)).toBe(true);
});
it("rejects a row someone else paid for that I have no split on", async () => {
const id = await fixture("not mine", them, [[them, 60], [third, 40]]);
expect(await inScope(id, me)).toBe(false);
});
});
describe("my share of someone else's row", () => {
it("is my split percent", async () => {
const id = await fixture("their shop, my quarter", them, [[me, 25], [them, 75]]);
expect(await share(id, me)).toBe(25);
});
it("is 0 with no split of mine — NOT the payer's remainder", async () => {
// The load-bearing case. An unsplit row someone else paid for has no
// `ts.share_percent`, so the old `100 - everyone else` fallback returned
// 100 and would have moved their entire grocery bill onto my spend the
// moment the gate widened.
const id = await fixture("entirely theirs", them);
expect(await share(id, me)).toBe(0);
expect(await spend(id, me)).toBe(0);
});
it("ignores my_share_percent, which can only mean the owner's share", async () => {
// One unscoped column per transaction, writable by anyone who can see the
// row. Reading it as the viewer's share on someone else's transaction
// would be a guess.
const id = await fixture("their row, their override", them, [], 30);
expect(await share(id, me)).toBe(0);
});
});
describe("my share of my own row is unchanged", () => {
it("prefers an explicit split row for me", async () => {
const id = await fixture("mine, split explicitly", me, [[me, 35], [them, 65]]);
expect(await share(id, me)).toBe(35);
});
it("falls back to the my_share_percent override", async () => {
const id = await fixture("mine, overridden", me, [], 80);
expect(await share(id, me)).toBe(80);
});
it("falls back to whatever is left after everyone else", async () => {
const id = await fixture("mine, remainder", me, [[them, 40]]);
expect(await share(id, me)).toBe(60);
});
it("is 0 when a row I paid for is allocated entirely to someone else", async () => {
const id = await fixture("mine, all theirs", me, [[them, 100]]);
expect(await share(id, me)).toBe(0);
});
});
describe("the regression it was written for", () => {
it("counts my half of a shop the other person paid for", async () => {
const id = await fixture("the grocery shop", them, [[me, 50], [them, 50]]);
expect(await spend(id, me)).toBe(50);
});
it("still counts my half of the same shop when I paid", async () => {
const id = await fixture("the same shop, my card", me, [[me, 50], [them, 50]]);
expect(await spend(id, me)).toBe(50);
});
it("counts my share whether or not the split is settled", async () => {
// `myShare` must not filter on `settled` — half a shop was my expense
// whether or not the other half was repaid. Same rule, payer-side.
const id = await fixture("settled shop", them, [[me, 50], [them, 50]]);
await queryRaw(`UPDATE transaction_splits SET settled = true WHERE transaction_id = $1`, [id]);
expect(await spend(id, me)).toBe(50);
});
it("never counts one row for both people at full value", async () => {
const id = await fixture("one shop, two people", them, [[me, 50], [them, 50]]);
expect(await spend(id, me) + await spend(id, them)).toBe(100);
});
});
describe("the transactions-list mirror agrees", () => {
it("MY_SHARE_PCT matches myShare on every fixture shape", async () => {
const shapes: [string, number, [number, number][], number | undefined][] = [
["mirror A", them, [[me, 25], [them, 75]], undefined],
["mirror B", them, [], undefined],
["mirror C", them, [], 30],
["mirror D", me, [[me, 35], [them, 65]], undefined],
["mirror E", me, [], 80],
["mirror F", me, [[them, 40]], undefined],
["mirror G", me, [], undefined],
];
for (const [label, owner, splits, override] of shapes) {
const id = await fixture(label, owner, splits, override);
const mirrored = await queryRow<{ v: string }>(
`SELECT (${MY_SHARE_PCT})::numeric AS v ${JOINS} WHERE t.id = $2`,
[me, id]
);
expect([label, Number(mirrored!.v)]).toEqual([label, await share(id, me)]);
}
});
});
});
+33 -8
View File
@@ -23,6 +23,25 @@ export function mockDbWithPool(p: Pool) {
const result = await p.query(sql, params);
return result.rows[0] ?? null;
},
// Mirrors withTransaction: one connection, BEGIN/COMMIT around the callback,
// so pg_advisory_xact_lock actually holds across the statements inside it.
withTransaction: async (fn: (q: unknown) => Promise<unknown>) => {
const client = await p.connect();
try {
await client.query("BEGIN");
const result = await fn({
queryRaw: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows,
queryRow: async (sql: string, params: unknown[] = []) => (await client.query(sql, params)).rows[0] ?? null,
});
await client.query("COMMIT");
return result;
} catch (e) {
await client.query("ROLLBACK");
throw e;
} finally {
client.release();
}
},
prisma: p,
}));
}
@@ -30,12 +49,17 @@ export function mockDbWithPool(p: Pool) {
/**
* Refuse to truncate anything that is not the test database.
*
* `DATABASE_URL` in `.env.test` names the Postgres container by IP, and
* container IPs move on recreation: 172.22.0.47 stopped being
* `postgres-personal` and became `postgres-pantry`, so the suite spent a while
* pointing its TRUNCATE at another app's database. It only failed safe because
* the credentials happened not to match — had they matched, this would have
* wiped pantry-app.
* `.env.test` used to name the Postgres container by bridge IP, and Docker
* reassigns those on recreation: 172.22.0.47 stopped being `postgres-personal`
* and became `postgres-pantry`, so the suite spent a while pointing its
* TRUNCATE at another app's database. It only failed safe because the
* credentials happened not to match — had they matched, this would have wiped
* pantry-app.
*
* `scripts/resolve-test-db.sh` (wired as `pretest:integration`) now re-derives
* the DSN from the live container before every run, preferring the published
* loopback port. This check stays anyway: running vitest directly bypasses the
* pretest hook, and a stale `.env.test` would still be read.
*
* Checked once per process, before the first truncate.
*/
@@ -47,8 +71,9 @@ async function assertTestDatabase(pool: Pool) {
if (db !== "personal_test") {
throw new Error(
`Refusing to truncate: connected to "${db}", expected "personal_test". ` +
`Check DATABASE_URL in .env.test — the Postgres container IP may have changed ` +
`(docker inspect postgres-personal --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}').`
`Re-resolve the DSN with \`bash scripts/resolve-test-db.sh\` (or just run ` +
`\`npm run test:integration\`, which does it for you), and run ` +
`\`npm run test:setup\` if personal_test does not exist yet.`
);
}
targetVerified = true;
+793
View File
@@ -0,0 +1,793 @@
import { describe, it, expect, beforeAll, beforeEach, afterAll, vi } from "vitest";
import { Pool } from "pg";
import { readFileSync, mkdtempSync, rmSync } from "fs";
import { tmpdir } from "os";
import { join, resolve } from "path";
import type { NextRequest } from "next/server";
import { createPool, mockDbWithPool, resetDB, seedParticipants } from "./helpers";
import type { ExtractedOrder } from "../../lib/intake-rules";
/**
* The intake router against the real schema (plan §11 C1, C2, C5).
*
* The model is stubbed at the HTTP boundary — `fetch` to LiteLLM answers with a
* canned extraction, `fetch` to pantry with a canned capture — so the PDF text
* path, the persistence, the match and the writes all run for real. The two
* PDFs are the user's own Uber Eats exports; the transactions seeded here are
* shaped like production rows 6846 and 6857 (platform `ubereats`, `line_items`
* empty, the email pipeline's booking of the same two orders).
*/
const INTAKE = "test-intake-token";
const RECEIPT = "test-receipt-token";
const fixtures = resolve(__dirname, "../fixtures/intake");
const BURGERTORY = readFileSync(resolve(fixtures, "ubereats-burgertory-2026-08-19.pdf"));
const COLES = readFileSync(resolve(fixtures, "ubereats-coles-2026-08-18.pdf"));
const burgertory: ExtractedOrder = {
merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4, subtotal: 28, currency: "AUD",
items: [
{ quantity: 1, description: "Michael", unit_price: 14, line_total: 14, options: ["Milk Bun"] },
{ quantity: 1, description: "Joel", unit_price: 14, line_total: 14, options: [] },
],
adjustments: [{ label: "Delivery Fee", amount: 5.49 }, { label: "Promotion", amount: -8 }],
};
const coles: ExtractedOrder = {
merchant: "Coles (Wyndham Vale)", order_date: "2026-08-18", total: 85.74, subtotal: 118.94, currency: "AUD",
items: [
{ quantity: 3, description: "Coca-Cola Zero Sugar Soft Drink (2L)", unit_price: 4.4, line_total: 13.2, options: [] },
{ quantity: 1, description: "Coles RSPCA Approved Chicken Thigh Cutlets approx. 1.1kg", unit_price: 11, line_total: 12.76, options: ["Final 1.16 kg $11.00/kg"] },
{ quantity: 1, description: "everything else", unit_price: 92.98, line_total: 92.98, options: [] },
],
adjustments: [],
};
let pool: Pool;
let ownerId: number;
let otherId: number;
let fileDir: string;
let intakePOST: (req: NextRequest) => Promise<Response>;
let lineItemsPUT: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
let slackPOST: (req: NextRequest) => Promise<Response>;
let orderGET: (req: NextRequest, ctx: { params: Promise<{ id: string }> }) => Promise<Response>;
/** What the next LiteLLM call answers, and what the next pantry call answers. */
let modelReply: ExtractedOrder | Error = burgertory;
let pantryReply: { status: number; body: unknown } = { status: 200, body: { captureId: 901, items: [1, 2, 3] } };
const calls: { url: string; body: unknown }[] = [];
function stubFetch() {
vi.stubGlobal("fetch", async (url: string | URL, init?: RequestInit) => {
const u = String(url);
if (u.includes("/v1/chat/completions")) {
calls.push({ url: u, body: JSON.parse(String(init?.body)) });
if (modelReply instanceof Error) return new Response("boom", { status: 500 });
return Response.json({ choices: [{ message: { content: JSON.stringify(modelReply) }, finish_reason: "stop" }] });
}
if (u.includes("/api/capture")) {
const form = init?.body as FormData;
calls.push({ url: u, body: { intake: JSON.parse(String(form.get("intake"))), files: form.getAll("image").map((f) => (f as File).name), headers: init?.headers } });
return Response.json(pantryReply.body, { status: pantryReply.status });
}
if (u.includes("hooks.slack.com")) {
calls.push({ url: u, body: JSON.parse(String(init?.body)) });
return new Response("ok");
}
throw new Error(`unexpected fetch ${u}`);
});
}
const multipart = (fields: Record<string, string>, files: { name: string; type: string; bytes: Buffer }[], token: string | null = INTAKE) => {
const fd = new FormData();
for (const [k, v] of Object.entries(fields)) fd.append(k, v);
for (const f of files) fd.append("file", new File([new Uint8Array(f.bytes)], f.name, { type: f.type }));
return new Request("http://finance.test/api/intake", { method: "POST", headers: token ? { "x-ingest-token": token } : {}, body: fd }) as unknown as NextRequest;
};
const jsonReq = (body: unknown, token: string | null) =>
({ headers: { get: (h: string) => (h === "x-ingest-token" ? token : null) }, json: async () => body }) as unknown as NextRequest;
const put = (id: number | string, body: unknown, token: string | null = RECEIPT) => lineItemsPUT(jsonReq(body, token), { params: Promise.resolve({ id: String(id) }) });
const uuid = () => crypto.randomUUID();
/** A transaction shaped like the email pipeline's booking of an order, line_items empty. */
async function seedOrder(over: { merchant: string; date: string; amount: number; platform?: string | null; owner?: number; withMeta?: boolean }) {
const t = await pool.query(
`INSERT INTO transactions (owner_id, statement_id, transaction_date, description, amount, amount_aud, transaction_type, category, merchant_name, merchant_normalized, payment_method, row_index)
VALUES ($1, NULL, $2, $3, $4, $4, 'debit', 'dining', $5, $5, 'credits', 0) RETURNING id`,
[over.owner ?? ownerId, over.date, `Order - ${over.merchant}`, over.amount, over.merchant]
);
const id = t.rows[0].id as number;
if (over.withMeta !== false) {
await pool.query(
`INSERT INTO expense_metadata (transaction_id, source, platform, order_reference, line_items, merchant_normalized, amount, transaction_date, currency)
VALUES ($1, 'email', $2, $3, '[]'::jsonb, $4, $5, $6, 'AUD')`,
[id, over.platform === undefined ? "ubereats" : over.platform, uuid(), over.merchant, over.amount, over.date]
);
}
return id;
}
/** What `components/order-details.tsx` would actually render for a transaction. */
const orderPanel = async (txnId: number) =>
(await (await orderGET({ headers: { get: () => null } } as unknown as NextRequest, { params: Promise.resolve({ id: String(txnId) }) })).json()) as { line_items?: unknown[] } | null;
const metaRows = async () => (await pool.query(`SELECT id, platform, line_items FROM expense_metadata ORDER BY id`)).rows as { id: number; platform: string | null; line_items: { description: string }[] }[];
const lineItemsOf = async (txnId: number) =>
(await pool.query(`SELECT line_items FROM expense_metadata WHERE transaction_id = $1 OR matched_transaction_id = $1`, [txnId])).rows[0]?.line_items;
const batch = async (id: string) => (await pool.query(`SELECT * FROM intake_batches WHERE id = $1`, [id])).rows[0];
beforeAll(async () => {
process.env.INTAKE_TOKEN = INTAKE;
process.env.RECEIPT_INGEST_TOKEN = RECEIPT;
process.env.ORDER_INGEST_TOKEN = "test-order-token";
process.env.LITELLM_BASE_URL = "http://litellm.test";
process.env.LITELLM_KEY = "k";
process.env.PANTRY_BASE_URL = "http://pantry.test";
process.env.FINANCE_PUBLIC_URL = "https://finance.test";
fileDir = mkdtempSync(join(tmpdir(), "intake-test-"));
process.env.INTAKE_FILE_DIR = fileDir;
pool = createPool();
mockDbWithPool(pool);
// The Slack signature gate is exercised by slack-verify.test.ts; here it is
// bypassed so the intake branch of the route is what is under test.
vi.doMock("@/lib/slack-verify", () => ({
verifySlackSignature: () => true,
participantForSlackUser: () => 1,
slackUserForParticipant: () => null,
}));
// The order-detail panel is the surface an enrichment has to appear on, so
// the read is exercised beside the write rather than assumed.
vi.doMock("@/lib/auth", () => ({ getCurrentUser: async () => ({ id: 1, name: "Owner", email: "o@x" }) }));
vi.doMock("@/lib/queries", async (importOriginal) => ({
...(await importOriginal<Record<string, unknown>>()),
canAccessTransactions: async () => true,
}));
({ POST: intakePOST } = await import("../../app/api/intake/route"));
({ PUT: lineItemsPUT } = await import("../../app/api/transactions/[id]/line-items/route"));
({ POST: slackPOST } = await import("../../app/api/slack/interactive/route"));
({ GET: orderGET } = await import("../../app/api/transactions/[id]/order/route"));
stubFetch();
});
afterAll(() => {
vi.unstubAllGlobals();
rmSync(fileDir, { recursive: true, force: true });
});
beforeEach(async () => {
await resetDB(pool);
await pool.query(`TRUNCATE intake_batches, intake_merchant_kinds`);
({ ownerId, otherId } = await seedParticipants(pool, ["Alice", "Bob"]));
await pool.query(`UPDATE participants SET email = 'bob@example.test' WHERE id = $1`, [otherId]);
modelReply = burgertory;
pantryReply = { status: 200, body: { captureId: 901, items: [1, 2, 3] } };
calls.length = 0;
});
// ─── C2 ───────────────────────────────────────────────────────────────────────
describe("PUT /api/transactions/[id]/line-items (C2)", () => {
const items = [{ qty: 1, description: "Michael", amount: 14, options: ["Milk Bun"] }, { qty: 1, description: "Joel", amount: 14 }];
it("fails closed: no token, wrong token", async () => {
expect((await put(1, { line_items: items, source: "intake", replace: true }, null)).status).toBe(401);
expect((await put(1, { line_items: items, source: "intake", replace: true }, "nope")).status).toBe(401);
});
it("accepts either token", async () => {
const id = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
expect((await put(id, { line_items: items, source: "pantry", replace: true }, RECEIPT)).status).toBe(200);
expect((await put(id, { line_items: items, source: "intake", replace: true }, INTAKE)).status).toBe(200);
});
it("404 when the transaction does not exist — it never creates one", async () => {
const res = await put(999999, { line_items: items, source: "intake", replace: true });
expect(res.status).toBe(404);
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
});
it("rejects a body that is not a replace, a bad source, or nameless items", async () => {
const id = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
expect((await put(id, { line_items: items, source: "intake" })).status).toBe(400);
expect((await put(id, { line_items: items, source: "n8n", replace: true })).status).toBe(400);
expect((await put(id, { line_items: [{ qty: 1 }], source: "intake", replace: true })).status).toBe(400);
});
it("idempotent replace: returns previous + new, and a second identical call is a no-op", async () => {
const id = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const first = await (await put(id, { line_items: items, source: "intake", replace: true })).json();
expect(first.previous_line_items).toEqual([]);
expect(first.line_items).toHaveLength(2);
expect(first.created_metadata).toBe(false);
expect(await lineItemsOf(id)).toHaveLength(2);
const corrected = [{ qty: 1, description: "Michael (corrected)", amount: 14 }];
const second = await (await put(id, { line_items: corrected, source: "pantry", replace: true })).json();
expect(second.previous_line_items.map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
expect(await lineItemsOf(id)).toEqual(corrected);
const third = await (await put(id, { line_items: corrected, source: "pantry", replace: true })).json();
expect(third.previous_line_items).toEqual(corrected);
expect((await pool.query(`SELECT count(*)::int AS n FROM expense_metadata`)).rows[0].n).toBe(1);
});
it("undo shape: PUT previous_line_items back restores the row", async () => {
const id = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const r = await (await put(id, { line_items: items, source: "pantry", replace: true })).json();
await put(id, { line_items: r.previous_line_items, source: "pantry", replace: true });
expect(await lineItemsOf(id)).toEqual([]);
});
it("creates the expense_metadata row when the transaction has none, with source and platform", async () => {
const id = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
const res = await (await put(id, { line_items: items, source: "intake", platform: "ubereats", batch_id: "b1", replace: true })).json();
expect(res.created_metadata).toBe(true);
const em = (await pool.query(`SELECT source, platform, order_reference, merchant_normalized, amount::text FROM expense_metadata WHERE transaction_id = $1`, [id])).rows[0];
expect(em).toEqual({ source: "intake", platform: "ubereats", order_reference: "intake:b1", merchant_normalized: "Coles", amount: "85.74" });
});
it("the undo baseline is recorded ONCE per batch: a retry gets the original, not its own write", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const original = [{ qty: 1, description: "what the order already had", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(original)]);
const id = uuid();
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
const first = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
expect(first.previous_line_items).toEqual(original);
// Pantry re-calls on every confirm and after a timeout. Before, the retry
// was handed its OWN write back, so undo restored the thing it undid.
const retry = await (await put(txn, { line_items: items, source: "pantry", batch_id: id, replace: true })).json();
expect(retry.previous_line_items).toEqual(original);
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual(original);
await put(txn, { line_items: retry.previous_line_items, source: "pantry", batch_id: id, replace: true });
expect(await lineItemsOf(txn)).toEqual(original);
});
it("without a batch id the baseline is still the row as read — nothing else changes", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const first = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
expect(first.previous_line_items).toEqual([]);
const second = await (await put(txn, { line_items: items, source: "pantry", replace: true })).json();
expect(second.previous_line_items).toEqual(items);
});
it("a pinned row that no longer exists falls back to the live one, rather than reporting a phantom write", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const live = (await pool.query(`SELECT id FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].id as number;
const { replaceLineItems } = await import("../../lib/line-items");
const r = await replaceLineItems(txn, items, { source: "intake", metadataId: live + 999 });
expect(r.metadata_id).toBe(live);
expect(await lineItemsOf(txn)).toHaveLength(2);
});
it("two writes racing to create the metadata row both resolve to the same one", async () => {
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
const both = await Promise.all([
put(txn, { line_items: items, source: "intake", replace: true }),
put(txn, { line_items: items, source: "intake", replace: true }),
]);
expect(both.map((r) => r.status)).toEqual([200, 200]);
expect((await pool.query(`SELECT count(*)::int AS n FROM expense_metadata WHERE transaction_id = $1`, [txn])).rows[0].n).toBe(1);
});
it("the create path records an undo baseline too, before it writes", async () => {
const txn = await seedOrder({ merchant: "Coles", date: "2026-08-18", amount: 85.74, withMeta: false });
const id = uuid();
await pool.query(`INSERT INTO intake_batches (id, source, file_sha256, status) VALUES ($1::uuid, 'other', '{}', 'received')`, [id]);
await put(txn, { line_items: items, source: "intake", batch_id: id, currency: "AUD", replace: true });
// [] rather than NULL: there was nothing there, and "nothing" is the state
// undo has to be able to restore.
expect((await pool.query(`SELECT previous_line_items FROM intake_batches WHERE id = $1`, [id])).rows[0].previous_line_items).toEqual([]);
});
it("the panel reads the row the write landed on, whichever order the rows were inserted in", async () => {
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
// Inserted first: the delivery receipt pointing at the statement line.
await pool.query(
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, '[]'::jsonb)`,
[t, uuid()]
);
// Inserted second: a row the transaction owns. An unordered LIMIT 1 reads
// the first one, the write takes the second, and the enrichment reports
// success against a row nothing renders.
await pool.query(
`INSERT INTO expense_metadata (transaction_id, source, order_reference, line_items) VALUES ($1, 'pantry', $2, '[]'::jsonb)`,
[t, uuid()]
);
const written = await (await put(t, { line_items: items, source: "intake", replace: true })).json();
const panel = await orderPanel(t);
expect(panel?.line_items).toHaveLength(2);
expect((await metaRows()).find((r) => r.id === written.metadata_id)!.line_items).toHaveLength(2);
});
it("finds the row through matched_transaction_id — a card-settled order has only that", async () => {
const statementLine = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
await pool.query(
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, '[]'::jsonb)`,
[statementLine, uuid()]
);
const res = await (await put(statementLine, { line_items: items, source: "intake", replace: true })).json();
expect(res.created_metadata).toBe(false);
expect((await pool.query(`SELECT count(*)::int AS n FROM expense_metadata`)).rows[0].n).toBe(1);
expect(await lineItemsOf(statementLine)).toHaveLength(2);
});
});
// ─── C1 ───────────────────────────────────────────────────────────────────────
describe("POST /api/intake — gates", () => {
const pdf = { name: "Receipt_19Aug2026.pdf", type: "application/pdf", bytes: BURGERTORY };
it("fails closed without INTAKE_TOKEN match", async () => {
expect((await intakePOST(multipart({ source: "uber_eats" }, [pdf], null))).status).toBe(401);
expect((await intakePOST(multipart({ source: "uber_eats" }, [pdf], "nope"))).status).toBe(401);
});
it("400 on a bad source, no files, too many files, a non-uuid batch_id", async () => {
expect((await intakePOST(multipart({ source: "menulog" }, [pdf]))).status).toBe(400);
expect((await intakePOST(multipart({ source: "uber_eats" }, []))).status).toBe(400);
expect((await intakePOST(multipart({ source: "uber_eats" }, Array(7).fill(pdf)))).status).toBe(400);
expect((await intakePOST(multipart({ source: "uber_eats", batch_id: "not-a-uuid" }, [pdf]))).status).toBe(400);
expect((await pool.query(`SELECT count(*)::int AS n FROM intake_batches`)).rows[0].n).toBe(0);
});
it("declines a file type it cannot read, after persisting the batch", async () => {
const id = uuid();
const res = await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [{ name: "clip.mp4", type: "video/mp4", bytes: Buffer.from("x") }]));
const body = await res.json();
expect(body.status).toBe("unsupported");
expect(body.text).toMatch(/not this/);
expect((await batch(id)).status).toBe("unsupported");
expect(calls).toHaveLength(0);
});
});
describe("POST /api/intake — food (enrich vs candidates)", () => {
const pdf = { name: "Receipt_19Aug2026.pdf", type: "application/pdf", bytes: BURGERTORY };
it("food + exact (platform, merchant, date, total) match → enriches the existing order, one model call, PDF as text", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const id = uuid();
const res = await intakePOST(multipart({ source: "uber_eats", shared_by: "bob@example.test", batch_id: id }, [pdf]));
const body = await res.json();
expect(res.status).toBe(200);
expect(body).toMatchObject({ kind: "food", status: "enriched", transaction_id: txn, batch_id: id });
expect(body.text).toBe("Added 2 items to your $22.40 Uber Eats (Burgertory (Werribee)) order ✓");
expect(JSON.stringify(body.slack_blocks)).toContain("https://finance.test/transactions?q=Burgertory");
const stored = await lineItemsOf(txn);
expect(stored.map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
expect(stored[0].options).toEqual(["Milk Bun"]);
// one joint call, the PDF's text in it (not an image)
expect(calls).toHaveLength(1);
const content = calls[0].body as { messages: { content: { type: string; text?: string }[] }[]; model: string; temperature: number; response_format: { type: string } };
expect(content.model).toBe("cloud-budget");
expect(content.temperature).toBe(0);
expect(content.response_format.type).toBe("json_schema");
expect(content.messages[0].content.some((c) => c.type === "image_url")).toBe(false);
expect(content.messages[0].content.some((c) => c.text?.includes("1 Michael $14.00"))).toBe(true);
const row = await batch(id);
expect(row).toMatchObject({ source: "uber_eats", shared_by: "bob@example.test", kind: "food", status: "enriched", transaction_id: txn });
expect(row.header).toMatchObject({ merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4, platform: "ubereats" });
expect(row.resolved_at).not.toBeNull();
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
});
it("the platform is part of the key: the same merchant/date/total on DoorDash is not the match — but it IS offered", async () => {
const doordash = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "doordash" });
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body.status).toBe("candidates");
// The candidates card is a human choice, not a decision. The platform is one
// of the things that may be wrong, so it must not narrow the list — a
// platform-filtered card showed nothing at all here.
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([doordash]);
});
it("a NULL platform is 'no platform', not a wildcard — an in-store share does not overwrite the delivery order", async () => {
await pool.query(`INSERT INTO intake_merchant_kinds (merchant_key, kind) VALUES ('burgertory werribee', 'food')`);
const delivery = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4, platform: "ubereats" });
const already = [{ qty: 1, description: "what the delivery order already had", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [delivery, JSON.stringify(already)]);
// `other` implies no platform; the delivery row carries one, so it is not
// the match — it is only a candidate.
const body = await (await intakePOST(multipart({ source: "other" }, [pdf]))).json();
expect(body.status).toBe("candidates");
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([delivery]);
expect(await lineItemsOf(delivery)).toEqual(already);
});
it("a reconciled manual row is neither matched nor offered — EXCLUDE_RECONCILED_SOURCE, as everywhere else", async () => {
const live = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const reconciled = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, reconciled]);
const near = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-17", amount: 19.9 });
await pool.query(`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`, [live, near]);
// Two rows at the exact key would have read as an ambiguity; one of them is
// the same spend counted twice.
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ status: "enriched", transaction_id: live });
});
it("the write lands on the row the MATCH resolved through, not on whichever row sorts first", async () => {
const t = await seedOrder({ merchant: "UBER *EATS", date: "2026-08-19", amount: 22.4, withMeta: false });
// The statement line owns a receipt row of its own (a pantry confirm) …
const grocery = [{ qty: 1, description: "a grocery shop", amount: 22.4 }];
const own = (await pool.query(
`INSERT INTO expense_metadata (transaction_id, source, order_reference, merchant_normalized, line_items) VALUES ($1, 'pantry', $2, 'Woolworths', $3::jsonb) RETURNING id`,
[t, uuid(), JSON.stringify(grocery)]
)).rows[0].id as number;
// … and is also the card leg of the Uber Eats order being enriched.
const delivery = (await pool.query(
`INSERT INTO expense_metadata (transaction_id, matched_transaction_id, source, platform, order_reference, merchant_normalized, line_items) VALUES (NULL, $1, 'email', 'ubereats', $2, 'Burgertory (Werribee)', '[]'::jsonb) RETURNING id`,
[t, uuid()]
)).rows[0].id as number;
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ status: "enriched", transaction_id: t });
const rows = await metaRows();
expect(rows.find((r) => r.id === delivery)!.line_items.map((i) => i.description)).toEqual(["Michael", "Joel"]);
expect(rows.find((r) => r.id === own)!.line_items).toEqual(grocery);
});
it("an upload with no prices never replaces a receipt that already has items", async () => {
const priceless = { ...burgertory, subtotal: null, total: 22.4, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
modelReply = priceless;
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ kind: "held", status: "held" });
expect(body.text).toMatch(/left the existing items alone/);
expect(await lineItemsOf(txn)).toEqual(already);
// …but an EMPTY receipt is exactly what this slice exists to fill.
const empty = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-20", amount: 22.4 });
modelReply = { ...priceless, order_date: "2026-08-20" };
const second = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(second).toMatchObject({ status: "enriched", transaction_id: empty });
expect((await lineItemsOf(empty)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
});
it("food + no match → candidates (same merchant ±3 days, any amount) with Use-this / Create buttons; NEVER auto-creates", async () => {
const near = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-17", amount: 19.9 });
await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-10", amount: 22.4 }); // too far
await seedOrder({ merchant: "Mad Mex", date: "2026-08-19", amount: 22.4 }); // wrong merchant
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id)).toEqual([near]);
const blocks = JSON.stringify(body.slack_blocks);
expect(blocks).toContain('"action_id":"intake_use_candidate"');
expect(blocks).toContain('"action_id":"intake_create"');
expect(blocks).toContain(JSON.stringify(JSON.stringify({ batch_id: id, transaction_id: near })));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(3);
expect(await lineItemsOf(near)).toEqual([]);
expect((await batch(id)).items).toHaveLength(2);
});
it("two exact hits is an ambiguity, not a coin toss", async () => {
const a = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const b = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body.status).toBe("candidates");
expect(body.candidates.map((c: { transaction_id: number }) => c.transaction_id).sort()).toEqual([a, b].sort());
});
it("missing header (no total) → needs_header, nothing written but the batch row", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
modelReply = { ...burgertory, total: null };
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("needs_header");
expect(body.text).toMatch(/share the top of the screen/);
expect(await lineItemsOf(txn)).toEqual([]);
expect((await batch(id)).status).toBe("needs_header");
});
it("items that do not add up → held, nothing written", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
modelReply = { ...burgertory, items: [burgertory.items[0]] };
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body).toMatchObject({ kind: "held", status: "held" });
expect(body.text).toMatch(/is a shot missing/);
expect(await lineItemsOf(txn)).toEqual([]);
});
it("a model failure is a 502 with the batch marked error, not a silent 200", async () => {
modelReply = new Error("down");
const res = await intakePOST(multipart({ source: "uber_eats" }, [pdf]));
expect(res.status).toBe(502);
expect((await res.json()).status).toBe("error");
});
it("re-share of the same file after it was handled → duplicate with a pointer", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const first = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: first }, [pdf]));
calls.length = 0;
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [pdf]))).json();
expect(body.status).toBe("duplicate");
expect(body.duplicate_of).toEqual({ batch_id: first, transaction_id: txn, capture_id: null });
expect(body.text).toMatch(/Already handled/);
expect(calls).toHaveLength(0); // no model call spent on a known file
});
it("re-share of the same order under a different file (screenshot after PDF) → duplicate by header", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const first = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: first }, [pdf]));
const body = await (await intakePOST(multipart({ source: "uber_eats" }, [{ name: "IMG_1.png", type: "image/png", bytes: Buffer.from("fake-png") }]))).json();
expect(body.status).toBe("duplicate");
expect(body.duplicate_of.transaction_id).toBe(txn);
});
it("a repeated batch_id is refused, not re-run", async () => {
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
expect((await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).status).toBe(409);
});
});
describe("POST /api/intake — grocery → pantry", () => {
const pdf = { name: "Receipt_18Aug2026.pdf", type: "application/pdf", bytes: COLES };
it("grocery + match → the ORIGINAL file goes to pantry with the target transaction and action enrich; finance is NOT touched", async () => {
modelReply = coles;
const txn = await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74 });
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", shared_by: "bob@example.test", batch_id: id }, [pdf]))).json();
expect(body).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn });
expect(body.text).toBe("Coles (Wyndham Vale) $85.74 → pantry board (3 items) — finance updates on confirm");
const pantryCall = calls.find((c) => c.url === "http://pantry.test/api/capture")!.body as { intake: Record<string, unknown>; files: string[]; headers: Record<string, string> };
expect(pantryCall.headers["x-ingest-token"]).toBe(RECEIPT);
expect(pantryCall.files).toEqual(["Receipt_18Aug2026.pdf"]);
expect(pantryCall.intake).toEqual({ batch_id: id, platform: "ubereats", origin: "intake", shared_by: "bob@example.test", finance_target_transaction_id: txn, finance_action: "enrich" });
expect(await lineItemsOf(txn)).toEqual([]);
expect((await batch(id))).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901, transaction_id: txn });
});
it("an in-store tap does not hand pantry the delivery order as its finance target", async () => {
modelReply = coles;
// The Uber Eats Coles order the email pipeline already booked.
const delivery = await seedOrder({ merchant: "Coles (Wyndham Vale)", date: "2026-08-18", amount: 85.74, platform: "ubereats" });
// The same shop, same day, same money — but shared from the Coles app, so
// it carries no platform and is a different purchase.
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
const pantryCall = calls.find((c) => c.url === "http://pantry.test/api/capture")!.body as { intake: Record<string, unknown> };
expect(pantryCall.intake).toMatchObject({ finance_action: "create", finance_target_transaction_id: null, platform: null });
expect(await lineItemsOf(delivery)).toEqual([]);
});
it("grocery + no match → pantry with finance_action create (ING-10 as today)", async () => {
modelReply = coles;
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body.status).toBe("forwarded_to_pantry");
expect(body.text).toMatch(/finance books it on confirm/);
const pantryCall = calls.find((c) => c.url.includes("/api/capture"))!.body as { intake: Record<string, unknown> };
expect(pantryCall.intake).toMatchObject({ platform: null, finance_target_transaction_id: null, finance_action: "create" });
});
it("pantry 409 → duplicate", async () => {
modelReply = coles;
pantryReply = { status: 409, body: { error: "This receipt has already been added to the pantry.", captureId: 77 } };
const body = await (await intakePOST(multipart({ source: "coles" }, [pdf]))).json();
expect(body).toMatchObject({ kind: "grocery", status: "duplicate", capture_id: 77 });
});
it("pantry down → 502 error, batch marked error", async () => {
modelReply = coles;
pantryReply = { status: 500, body: { error: "nope" } };
const res = await intakePOST(multipart({ source: "coles" }, [pdf]));
expect(res.status).toBe(502);
expect((await res.json()).status).toBe("error");
});
});
// ─── C5 ───────────────────────────────────────────────────────────────────────
const slackPress = (action_id: string, value: unknown) => {
const payload = { type: "block_actions", user: { id: "U1", name: "alice" }, response_url: "https://hooks.slack.com/r/1", actions: [{ action_id, value: JSON.stringify(value) }] };
const raw = new URLSearchParams({ payload: JSON.stringify(payload) }).toString();
return { headers: { get: (h: string) => (h === "x-ingest-token" ? "test-order-token" : "x") }, text: async () => raw } as unknown as NextRequest;
};
const lastSlackUpdate = () => calls.filter((c) => c.url.includes("hooks.slack.com")).at(-1)?.body as { replace_original?: boolean; text: string; blocks?: unknown[] };
describe("Slack taps (C5)", () => {
const pdf = { name: "Receipt_19Aug2026.pdf", type: "application/pdf", bytes: BURGERTORY };
it("intake_use_candidate enriches the chosen transaction and replaces the card", async () => {
const near = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-17", amount: 19.9 });
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
const res = await slackPOST(slackPress("intake_use_candidate", { batch_id: id, transaction_id: near }));
expect(res.status).toBe(200);
expect((await lineItemsOf(near)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: near });
expect(lastSlackUpdate()).toMatchObject({ replace_original: true, text: "Added 2 items to your $22.40 Uber Eats (Burgertory (Werribee)) order ✓" });
});
it("intake_create creates the transaction ING-10-style: manual, owner from shared_by, items attached; a second press is a no-op", async () => {
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", shared_by: "bob@example.test", batch_id: id }, [pdf]));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
await slackPOST(slackPress("intake_create", { batch_id: id }));
const t = (await pool.query(`SELECT statement_id, owner_id, description, amount::text, category, transaction_date::text, source, source_ref FROM transactions`)).rows;
expect(t).toEqual([{ statement_id: null, owner_id: otherId, description: "Order - Burgertory (Werribee)", amount: "22.40", category: "dining", transaction_date: "2026-08-19", source: "intake", source_ref: id }]);
const em = (await pool.query(`SELECT source, platform, line_items FROM expense_metadata`)).rows[0];
expect(em.source).toBe("intake");
expect(em.platform).toBe("ubereats");
expect(em.line_items.map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
expect(lastSlackUpdate().text).toMatch(/^Created a \$22\.40 Burgertory \(Werribee\) transaction with 2 items/);
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
});
it("intake_create re-runs the match: an order booked between the card and the tap is enriched, not duplicated", async () => {
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
// The email/order bridge books the same purchase while the card sits in
// Slack. The card's "no match" is minutes stale by the time it is pressed.
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: txn });
expect((await lineItemsOf(txn)).map((i: { description: string }) => i.description)).toEqual(["Michael", "Joel"]);
expect(lastSlackUpdate().text).toMatch(/^Added 2 items/);
});
it("two pending cards for one order book it ONCE", async () => {
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
expect((await batch(a)).status).toBe("candidates");
expect((await batch(b)).status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: a }));
await slackPOST(slackPress("intake_create", { batch_id: b }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect((await batch(b)).transaction_id).toBe((await batch(a)).transaction_id);
});
it("a sibling already resolved to the pantry board makes Create a duplicate, not a second row", async () => {
// Two shares of one receipt, resolved out of order: nothing the second card
// can match on, because the first one never produced a transaction.
modelReply = { ...coles, merchant: "Romeo's Food Hall" };
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "other", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
await intakePOST(multipart({ source: "other", batch_id: a }, [{ name: "Receipt_18Aug2026.pdf", type: "application/pdf", bytes: COLES }]));
expect((await batch(a)).status).toBe("ask");
await slackPOST(slackPress("intake_grocery", { batch_id: a }));
expect((await batch(a)).status).toBe("forwarded_to_pantry");
await slackPOST(slackPress("intake_meal", { batch_id: b }));
expect((await batch(b)).status).toBe("candidates");
await slackPOST(slackPress("intake_create", { batch_id: b }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(0);
expect(await batch(b)).toMatchObject({ status: "duplicate", capture_id: 901 });
expect(lastSlackUpdate().text).toMatch(/^Already handled/);
});
it("two taps that land together still book one transaction, and neither reports an error", async () => {
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: id })), slackPOST(slackPress("intake_create", { batch_id: id }))]);
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
const texts = calls.filter((c) => c.url.includes("hooks.slack.com")).map((c) => (c.body as { text: string }).text);
expect(texts.every((t) => !/Couldn't process/.test(t))).toBe(true);
});
it("a foreign receipt is booked in its own currency, with no invented AUD figure", async () => {
modelReply = { ...burgertory, currency: "USD" };
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await slackPOST(slackPress("intake_create", { batch_id: id }));
const t = (await pool.query(`SELECT amount::text, amount_aud, foreign_currency_amount::text, foreign_currency_code FROM transactions`)).rows[0];
expect(t).toEqual({ amount: "22.40", amount_aud: null, foreign_currency_amount: "22.40", foreign_currency_code: "USD" });
expect(lastSlackUpdate().text).toMatch(/USD 22\.40/);
// OrderDetails prefers expense_metadata.currency, so a hard-coded 'AUD'
// here printed "$" on a USD order's receipt panel.
expect((await pool.query(`SELECT currency FROM expense_metadata`)).rows[0].currency).toBe("USD");
});
it("the hold applies to the Create tap's re-match, not only to the upload path", async () => {
modelReply = { ...burgertory, subtotal: null, items: burgertory.items.map((i) => ({ ...i, unit_price: null, line_total: null })) };
const id = uuid();
const body = await (await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]))).json();
expect(body.status).toBe("candidates");
// Booked WITH its items between the card and the tap. The re-match walked
// past the guard routeFood applies, so the same upload destroyed the same
// receipt depending on which button reached it.
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const already = [{ qty: 1, description: "the itemised receipt already there", amount: 22.4 }];
await pool.query(`UPDATE expense_metadata SET line_items = $2::jsonb WHERE transaction_id = $1`, [txn, JSON.stringify(already)]);
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect(await lineItemsOf(txn)).toEqual(already);
expect(await batch(id)).toMatchObject({ status: "held" });
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
});
it("two DIFFERENT batches for one purchase, tapped together, book it once", async () => {
const a = uuid();
const b = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: a }, [pdf]));
await intakePOST(multipart({ source: "uber_eats", batch_id: b }, [{ name: "shot.png", type: "image/png", bytes: Buffer.from("shot") }]));
// Their source_refs differ, so ON CONFLICT cannot save this; both taps
// clear their match and duplicate checks before either inserts.
await Promise.all([slackPOST(slackPress("intake_create", { batch_id: a })), slackPOST(slackPress("intake_create", { batch_id: b }))]);
const rows = (await pool.query(`SELECT id FROM transactions`)).rows;
expect(rows).toHaveLength(1);
expect((await batch(a)).transaction_id).toBe(rows[0].id);
expect((await batch(b)).transaction_id).toBe(rows[0].id);
});
it("unknown shared_by falls back to the default owner — never a NULL owner", async () => {
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", shared_by: "nobody@example.test", batch_id: id }, [pdf]));
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT owner_id FROM transactions`)).rows[0].owner_id).toBe(1);
});
it("`other` + unrecognised merchant asks; intake_meal routes it as food and remembers the merchant", async () => {
modelReply = { ...burgertory, merchant: "Romeo's Food Hall" };
const txn = await seedOrder({ merchant: "Romeo's Food Hall", date: "2026-08-19", amount: 22.4, platform: null });
const id = uuid();
const body = await (await intakePOST(multipart({ source: "other", batch_id: id }, [pdf]))).json();
expect(body).toMatchObject({ kind: "ask", status: "ask" });
expect(JSON.stringify(body.slack_blocks)).toContain("intake_grocery");
await slackPOST(slackPress("intake_meal", { batch_id: id }));
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: txn });
expect(await lineItemsOf(txn)).toHaveLength(2);
expect((await pool.query(`SELECT kind FROM intake_merchant_kinds WHERE merchant_key = 'romeo s food hall'`)).rows[0].kind).toBe("food");
// The answer sticks: the next share of that merchant does not ask.
const again = await (await intakePOST(multipart({ source: "other" }, [{ name: "IMG.png", type: "image/png", bytes: Buffer.from("png2") }]))).json();
expect(again.status).toBe("duplicate"); // same order — but it was routed, not asked
modelReply = { ...burgertory, merchant: "Romeo's Food Hall", order_date: "2026-08-20" };
const next = await (await intakePOST(multipart({ source: "other" }, [{ name: "IMG3.png", type: "image/png", bytes: Buffer.from("png3") }]))).json();
expect(next).toMatchObject({ kind: "food", status: "candidates" });
});
it("intake_grocery forwards the stored original files to pantry", async () => {
modelReply = { ...coles, merchant: "Romeo's Food Hall" };
const id = uuid();
await intakePOST(multipart({ source: "other", batch_id: id }, [{ name: "Receipt_18Aug2026.pdf", type: "application/pdf", bytes: COLES }]));
expect((await batch(id)).status).toBe("ask");
await slackPOST(slackPress("intake_grocery", { batch_id: id }));
const pantryCall = calls.find((c) => c.url.includes("/api/capture"))!.body as { intake: Record<string, unknown>; files: string[] };
expect(pantryCall.files).toEqual(["Receipt_18Aug2026.pdf"]);
expect(pantryCall.intake).toMatchObject({ batch_id: id, finance_action: "create" });
expect(await batch(id)).toMatchObject({ kind: "grocery", status: "forwarded_to_pantry", capture_id: 901 });
expect((await pool.query(`SELECT kind FROM intake_merchant_kinds`)).rows[0].kind).toBe("grocery");
});
it("a tap on a resolved batch returns its card and writes nothing; an unknown batch says so", async () => {
const txn = await seedOrder({ merchant: "Burgertory (Werribee)", date: "2026-08-19", amount: 22.4 });
const id = uuid();
await intakePOST(multipart({ source: "uber_eats", batch_id: id }, [pdf]));
await slackPOST(slackPress("intake_create", { batch_id: id }));
expect((await pool.query(`SELECT count(*)::int AS n FROM transactions`)).rows[0].n).toBe(1);
expect(lastSlackUpdate().text).toMatch(/^Added 2 items/);
expect(await batch(id)).toMatchObject({ status: "enriched", transaction_id: txn });
await slackPOST(slackPress("intake_create", { batch_id: uuid() }));
expect(lastSlackUpdate()).toMatchObject({ replace_original: false, text: "I can't find that upload any more." });
});
});
@@ -1,6 +1,7 @@
import { describe, it, expect, beforeAll } from "vitest";
import { readFileSync } from "fs";
import { resolve } from "path";
import type { NextRequest } from "next/server";
import { queryRaw } from "../../lib/db";
/**
@@ -13,13 +14,22 @@ const dir = resolve(__dirname, "../fixtures/orders/real");
const html = (f: string) => readFileSync(resolve(dir, `${f}.html`), "utf-8");
const TOKEN = "test-ingest-token";
let POST: any;
// The handler is imported dynamically (after the env var is set), so its type
// is declared here rather than inferred. `unknown` for the request is
// deliberate: these fixtures are the two properties the route actually reads,
// not a whole NextRequest, and widening to `any` would stop the compiler
// checking the call sites too.
let POST: (req: NextRequest) => Promise<Response>;
// One cast, at the boundary, instead of `any` everywhere downstream: the
// fixture is the two properties the route actually reads, and everything past
// this line stays type-checked.
const req = (body: unknown, token: string | null = TOKEN) =>
({
headers: { get: (h: string) => (h === "x-ingest-token" ? token : null) },
json: async () => body,
}) as any;
}) as unknown as NextRequest;
beforeAll(async () => {
process.env.ORDER_INGEST_TOKEN = TOKEN;
+62 -35
View File
@@ -1,4 +1,5 @@
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
import type { NextRequest } from "next/server";
import { createPool, mockDbWithPool, resetDB } from "./helpers";
/**
@@ -12,15 +13,41 @@ import { createPool, mockDbWithPool, resetDB } from "./helpers";
const pool = createPool();
mockDbWithPool(pool);
let GET: any;
let PUT: any;
/** What the review route returns, to the extent these tests read it. */
type ItemVerdict = { item: string; verdict: string };
type Review = {
participant_id: number;
rating: string | null;
note: string | null;
order_again: boolean | null;
item_verdicts: ItemVerdict[];
};
type ReviewBody = {
reviews: Review[];
splits: { participant_id: number; share_percent: number }[];
merchant: {
warn: boolean;
note: string | null;
counts: Record<string, number>;
history: { transaction_id: number }[];
items: { item: string; loved: number; never: number }[];
} | null;
};
type Ctx = { params: Promise<{ id: string }> };
// Declared rather than inferred because the route is imported dynamically,
// after vi.doMock. `unknown` for the request keeps the fixture honest — it is
// the two properties the handler reads, not a NextRequest.
let GET: (req: NextRequest, ctx: Ctx) => Promise<Response>;
let PUT: (req: NextRequest, ctx: Ctx) => Promise<Response>;
let ownerId: number;
let otherId: number;
let txnId: number;
const req = (body?: unknown) =>
({ headers: { get: () => null }, json: async () => body }) as any;
const params = (id: number) => ({ params: Promise.resolve({ id: String(id) }) });
({ headers: { get: () => null }, json: async () => body }) as unknown as NextRequest;
const params = (id: number): Ctx => ({ params: Promise.resolve({ id: String(id) }) });
const bodyOf = async (res: Response): Promise<ReviewBody> => (await res.json()) as ReviewBody;
beforeAll(async () => {
vi.doMock("@/lib/auth", () => ({
@@ -59,25 +86,25 @@ describe("order verdicts — one per person", () => {
it("keeps both people's verdicts on the same order", async () => {
await PUT(req({ participant_id: ownerId, rating: "loved" }), params(txnId));
const res = await PUT(req({ participant_id: otherId, rating: "never" }), params(txnId));
const body = await res.json();
const body = await bodyOf(res);
// The bug this guards: a UNIQUE on transaction_id alone made the second
// save overwrite the first, and the disagreement is the useful part.
expect(body.reviews).toHaveLength(2);
expect(body.reviews.find((r: any) => r.participant_id === ownerId).rating).toBe("loved");
expect(body.reviews.find((r: any) => r.participant_id === otherId).rating).toBe("never");
expect(body.reviews.find((r) => r.participant_id === ownerId)?.rating).toBe("loved");
expect(body.reviews.find((r) => r.participant_id === otherId)?.rating).toBe("never");
});
it("defaults the verdict to the signed-in user, not the owner", async () => {
const res = await PUT(req({ rating: "ok" }), params(txnId));
const body = await res.json();
const body = await bodyOf(res);
expect(body.reviews[0].participant_id).toBe(ownerId);
});
it("revising a verdict updates rather than duplicating", async () => {
await PUT(req({ participant_id: ownerId, rating: "loved" }), params(txnId));
const res = await PUT(req({ participant_id: ownerId, rating: "never" }), params(txnId));
const body = await res.json();
const body = await bodyOf(res);
expect(body.reviews).toHaveLength(1);
expect(body.reviews[0].rating).toBe("never");
});
@@ -88,12 +115,12 @@ describe("order verdicts — one per person", () => {
});
it("derives order_again from the rating", async () => {
let body = await (await PUT(req({ rating: "never" }), params(txnId))).json();
let body = await bodyOf(await PUT(req({ rating: "never" }), params(txnId)));
expect(body.reviews[0].order_again).toBe(false);
body = await (await PUT(req({ rating: "ok" }), params(txnId))).json();
body = await bodyOf(await PUT(req({ rating: "ok" }), params(txnId)));
expect(body.reviews[0].order_again).toBe(true);
// "bad" is also a no — you would not choose it again.
body = await (await PUT(req({ rating: "bad" }), params(txnId))).json();
body = await bodyOf(await PUT(req({ rating: "bad" }), params(txnId)));
expect(body.reviews[0].order_again).toBe(false);
});
@@ -104,13 +131,13 @@ describe("order verdicts — one per person", () => {
const older = await seedOrder("Thai Palace", "2026-06-01");
await PUT(req({ participant_id: ownerId, rating: "bad" }), params(older));
let body = await (await GET(req(), params(txnId))).json();
expect(body.merchant.counts.bad).toBe(1);
expect(body.merchant.warn).toBe(false);
let body = await bodyOf(await GET(req(), params(txnId)));
expect(body.merchant!.counts.bad).toBe(1);
expect(body.merchant!.warn).toBe(false);
await PUT(req({ participant_id: ownerId, rating: "never" }), params(older));
body = await (await GET(req(), params(txnId))).json();
expect(body.merchant.warn).toBe(true);
body = await bodyOf(await GET(req(), params(txnId)));
expect(body.merchant!.warn).toBe(true);
});
});
@@ -125,7 +152,7 @@ describe("item verdicts", () => {
);
// No item_verdicts key at all — the shape a note-only form sends.
const res = await PUT(req({ rating: "liked", note: "slow delivery" }), params(txnId));
const body = await res.json();
const body = await bodyOf(res);
expect(body.reviews[0].item_verdicts).toEqual([
{ item: "Pad Thai", verdict: "loved" },
]);
@@ -138,7 +165,7 @@ describe("item verdicts", () => {
params(txnId)
);
const res = await PUT(req({ rating: "liked", item_verdicts: [] }), params(txnId));
const body = await res.json();
const body = await bodyOf(res);
expect(body.reviews[0].item_verdicts).toEqual([]);
});
@@ -154,7 +181,7 @@ describe("item verdicts", () => {
}),
params(txnId)
);
const body = await res.json();
const body = await bodyOf(res);
expect(body.reviews[0].rating).toBe("ok");
expect(body.reviews[0].item_verdicts).toEqual([
{ item: "Pad Thai", verdict: "loved" },
@@ -167,10 +194,10 @@ describe("merchant history", () => {
const older = await seedOrder("Thai Palace", "2026-06-01");
await PUT(req({ participant_id: ownerId, rating: "never", note: "cold" }), params(older));
const body = await (await GET(req(), params(txnId))).json();
expect(body.merchant.warn).toBe(true);
expect(body.merchant.counts.never).toBe(1);
expect(body.merchant.history.map((h: any) => h.transaction_id)).toEqual([older]);
const body = await bodyOf(await GET(req(), params(txnId)));
expect(body.merchant!.warn).toBe(true);
expect(body.merchant!.counts.never).toBe(1);
expect(body.merchant!.history.map((h) => h.transaction_id)).toEqual([older]);
});
it("treats the same restaurant as one merchant across platforms", async () => {
@@ -181,17 +208,17 @@ describe("merchant history", () => {
const shouty = await seedOrder("THAI PALACE", "2026-06-01");
await PUT(req({ participant_id: ownerId, rating: "never" }), params(shouty));
const body = await (await GET(req(), params(txnId))).json();
expect(body.merchant.warn).toBe(true);
expect(body.merchant.history.map((h: any) => h.transaction_id)).toContain(shouty);
const body = await bodyOf(await GET(req(), params(txnId)));
expect(body.merchant!.warn).toBe(true);
expect(body.merchant!.history.map((h) => h.transaction_id)).toContain(shouty);
});
it("does not carry a verdict across different merchants", async () => {
const other = await seedOrder("Pizza Place", "2026-06-01");
await PUT(req({ participant_id: ownerId, rating: "never" }), params(other));
const body = await (await GET(req(), params(txnId))).json();
expect(body.merchant.warn).toBe(false);
const body = await bodyOf(await GET(req(), params(txnId)));
expect(body.merchant!.warn).toBe(false);
});
it("pools item opinions across the merchant's orders", async () => {
@@ -206,10 +233,10 @@ describe("merchant history", () => {
params(older2)
);
const body = await (await GET(req(), params(txnId))).json();
const body = await bodyOf(await GET(req(), params(txnId)));
// Case-folded: the same dish comes back capitalised differently between
// receipts, and two entries for one dish is not a track record.
expect(body.merchant.items).toEqual([{ item: "Pad Thai", loved: 2, never: 0 }]);
expect(body.merchant!.items).toEqual([{ item: "Pad Thai", loved: 2, never: 0 }]);
});
it("keeps item opinions from reviews that have no overall rating", async () => {
@@ -218,14 +245,14 @@ describe("merchant history", () => {
req({ item_verdicts: [{ item: "Satay", verdict: "never" }] }),
params(older)
);
const body = await (await GET(req(), params(txnId))).json();
expect(body.merchant.items).toEqual([{ item: "Satay", loved: 0, never: 1 }]);
const body = await bodyOf(await GET(req(), params(txnId)));
expect(body.merchant!.items).toEqual([{ item: "Satay", loved: 0, never: 1 }]);
});
});
describe("share state", () => {
it("reports splits so the panel can show shared vs just me", async () => {
let body = await (await GET(req(), params(txnId))).json();
let body = await bodyOf(await GET(req(), params(txnId)));
expect(body.splits).toEqual([]);
await pool.query(
@@ -233,7 +260,7 @@ describe("share state", () => {
VALUES ($1, $2, 50)`,
[txnId, otherId]
);
body = await (await GET(req(), params(txnId))).json();
body = await bodyOf(await GET(req(), params(txnId)));
expect(body.splits).toHaveLength(1);
expect(body.splits[0].participant_id).toBe(otherId);
});
@@ -0,0 +1,63 @@
import { describe, it, expect } from "vitest";
import { queryRaw, queryRow } from "../../lib/db";
import { needsCardMatch } from "../../lib/queries";
import {
PAYMENT_METHODS,
NON_RECONCILING,
reconcilesAgainstStatement,
} from "../../lib/payment-methods";
/**
* These two lists drifted, and the drift was silent for two months.
*
* Migration 0018 admitted 'credits' to `transactions_payment_method_check` so
* the order pipeline could file gift-card-funded orders. Nothing propagated it
* to the hand-written allowlist in `PATCH /api/transactions/[id]` or to the
* add-transaction dropdown, so the database held 372 rows carrying a value the
* UI could not enter and the API refused to set.
*
* The failure mode is invisible on screen — a gift-card purchase entered by
* hand takes the NULL default, and NULL is *reconcilable*, so it joins the
* pending queue and is offered card matches that cannot exist. Accepting one is
* destructive: reconciled manual rows are filtered out of every query, so the
* spend vanishes while an unrelated card row claims to be it.
*
* So: assert against the database, not against a second copy of the list.
*/
describe("payment methods mirror the database", () => {
it("PAYMENT_METHODS equals the CHECK constraint's vocabulary", async () => {
const row = await queryRow<{ def: string }>(
`SELECT pg_get_constraintdef(oid) AS def FROM pg_constraint
WHERE conname = 'transactions_payment_method_check'`
);
expect(row, "constraint transactions_payment_method_check is missing").not.toBeNull();
const inConstraint = [...row!.def.matchAll(/'([a-z_]+)'::text/g)].map((m) => m[1]);
expect(inConstraint.length).toBeGreaterThan(0); // guard against the regex silently matching nothing
expect([...inConstraint].sort()).toEqual([...PAYMENT_METHODS].sort());
});
it("reconcilesAgainstStatement() agrees with needsCardMatch() for every method", async () => {
// Evaluate the real SQL predicate rather than re-implementing it: this is
// the TypeScript mirror, so the SQL is the thing being mirrored.
const rows = await queryRaw<{ pm: string | null; sql_reconciles: boolean }>(
`SELECT v.payment_method AS pm, ${needsCardMatch("v")} AS sql_reconciles
FROM (SELECT unnest($1::text[]) AS payment_method) v`,
[[...PAYMENT_METHODS, null]]
);
expect(rows).toHaveLength(PAYMENT_METHODS.length + 1);
for (const { pm, sql_reconciles } of rows) {
expect(reconcilesAgainstStatement(pm), `payment_method = ${pm ?? "NULL"}`)
.toBe(sql_reconciles);
}
});
it("holds cash and credits out of the queue, and nothing else", () => {
expect([...NON_RECONCILING].sort()).toEqual(["cash", "credits"]);
// NULL means *unknown*, never "no card leg is coming" — it must stay a
// candidate or every pre-existing row would drop out of reconciliation.
expect(reconcilesAgainstStatement(null)).toBe(true);
expect(reconcilesAgainstStatement(undefined)).toBe(true);
});
});
@@ -0,0 +1,274 @@
import { describe, it, expect, beforeAll } from "vitest";
import type { NextRequest } from "next/server";
import { queryRaw, queryRow } from "../../lib/db";
/**
* Changing who paid.
*
* Owner was write-once for every ingestion path — a pantry receipt hardcodes
* DEFAULT_OWNER_ID, and no route, modal or bulk action could correct it, so a
* shop the other person paid for was permanently filed as yours. There was not
* a single `UPDATE ... SET owner_id` anywhere in src/.
*
* Two things make this more than a field edit, and both are tested here:
*
* 1. **It is a one-way door.** Access is "owner OR holds a split"
* (canAccessTransactions), so handing a row over while holding no split on
* it removes it from your list and 404s every route that could put it back.
* Only the new owner can undo it.
* 2. **It moves money between two ledgers.** Splits record shares, not
* direction — getParticipantBalances derives who owes whom from ownership —
* so the same 50/50 rows flip sides untouched.
*/
type Ctx = { params: Promise<{ id: string }> };
type Handler = (req: NextRequest, ctx: Ctx) => Promise<Response>;
let txPATCH: Handler;
let stmtPATCH: Handler;
/** Enough of a NextRequest for these routes: the auth header and the body. */
const req = (email: string | null, body: unknown) =>
({
headers: { get: (h: string) => (h.toLowerCase() === "x-forwarded-user" ? email : null) },
json: async () => body,
}) as unknown as NextRequest;
const ctx = (id: number) => ({ params: Promise.resolve({ id: String(id) }) });
let me = 0;
let them = 0;
let meEmail = "";
let themEmail = "";
beforeAll(async () => {
({ PATCH: txPATCH } = await import("../../app/api/transactions/[id]/route"));
({ PATCH: stmtPATCH } = await import("../../app/api/statements/[id]/route"));
// Fixtures are named, not random, so a re-run is idempotent. `participants`
// has a unique email and no cascade from `transactions.owner_id`, so the
// teardown order is rows → statements → people.
await queryRaw(`DELETE FROM transactions WHERE description LIKE 'Owner fixture — %'`);
await queryRaw(`DELETE FROM statements WHERE bank_name = 'Owner Fixture Bank'`);
await queryRaw(`DELETE FROM participants WHERE email LIKE 'owner-fixture-%@example.test'`);
const mk = async (label: string) => {
const email = `owner-fixture-${label}@example.test`;
const row = await queryRow<{ id: number }>(
`INSERT INTO participants (name, email) VALUES ($1, $2) RETURNING id`,
[`Owner fixture — ${label}`, email]
);
return { id: row!.id, email };
};
const a = await mk("me");
const b = await mk("them");
me = a.id;
meEmail = a.email;
them = b.id;
themEmail = b.email;
});
/** A manual transaction owned by `owner`, with optional splits. */
async function manualTxn(label: string, owner: number, splits: [number, number][] = []) {
const row = await queryRow<{ id: number }>(
`INSERT INTO transactions (transaction_date, description, amount, transaction_type, category, owner_id)
VALUES ('2026-06-01', $1, 100.00, 'debit', 'groceries', $2) RETURNING id`,
[`Owner fixture — ${label}`, owner]
);
for (const [pid, pct] of splits) {
await queryRaw(
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent) VALUES ($1,$2,$3)`,
[row!.id, pid, pct]
);
}
return row!.id;
}
const ownerOf = async (id: number) =>
(await queryRow<{ owner_id: number | null }>(`SELECT owner_id FROM transactions WHERE id = $1`, [id]))!
.owner_id;
describe("PATCH /api/transactions/[id] — owner", () => {
it("reassigns a manual row when I hold a split on it", async () => {
const id = await manualTxn("shared shop", me, [[me, 50], [them, 50]]);
const res = await txPATCH(req(meEmail, { owner_id: them }), ctx(id));
expect(res.status).toBe(200);
expect(await ownerOf(id)).toBe(them);
});
it("refuses to hand away a row I hold no split on", async () => {
const id = await manualTxn("nothing of mine", me);
const res = await txPATCH(req(meEmail, { owner_id: them }), ctx(id));
expect(res.status).toBe(409);
expect((await res.json()).code).toBe("would_lose_access");
// Unchanged — a refusal must not half-apply.
expect(await ownerOf(id)).toBe(me);
});
it("hands it away when the release is explicit", async () => {
const id = await manualTxn("given away", me);
const res = await txPATCH(req(meEmail, { owner_id: them, release: true }), ctx(id));
expect(res.status).toBe(200);
expect(await ownerOf(id)).toBe(them);
});
it("never blocks taking onto my ledger a row I can already see", async () => {
// No `release` needed: I end up the owner, so there is no door to close
// behind me. The split is what lets me see it in the first place.
const id = await manualTxn("taken on", them, [[me, 50], [them, 50]]);
const res = await txPATCH(req(meEmail, { owner_id: me }), ctx(id));
expect(res.status).toBe(200);
expect(await ownerOf(id)).toBe(me);
});
it("cannot claim a row I cannot see", async () => {
// Their transaction, no split of mine — canAccessTransactions rejects
// before any owner logic runs. Reassignment is a correction to a row you
// already have, never a way to reach one you do not.
const id = await manualTxn("out of reach", them);
const res = await txPATCH(req(meEmail, { owner_id: me }), ctx(id));
expect(res.status).toBe(404);
expect(await ownerOf(id)).toBe(them);
});
it("refuses a statement row and says where to change it", async () => {
const stmt = await queryRow<{ id: number }>(
`INSERT INTO statements (bank_name, account_number, billing_end_date, currency, filename, owner_id)
VALUES ('Owner Fixture Bank','1111','2026-06-30','AUD','owner-fixture-1.pdf',$1) RETURNING id`,
[me]
);
const row = await queryRow<{ id: number }>(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type)
VALUES ($1,'2026-06-02','Owner fixture — statement row', 40.00, 'debit') RETURNING id`,
[stmt!.id]
);
const res = await txPATCH(req(meEmail, { owner_id: them }), ctx(row!.id));
expect(res.status).toBe(400);
expect((await res.json()).code).toBe("statement_owned");
});
it("rejects a participant that does not exist", async () => {
const id = await manualTxn("bad participant", me, [[me, 100]]);
const res = await txPATCH(req(meEmail, { owner_id: 999999 }), ctx(id));
expect(res.status).toBe(400);
expect(await ownerOf(id)).toBe(me);
});
it("leaves splits untouched — they record shares, not direction", async () => {
const id = await manualTxn("shares survive", me, [[me, 50], [them, 50]]);
await txPATCH(req(meEmail, { owner_id: them }), ctx(id));
const splits = await queryRaw<{ participant_id: number; share_percent: string; settled: boolean }>(
`SELECT participant_id, share_percent, settled FROM transaction_splits
WHERE transaction_id = $1 ORDER BY participant_id`,
[id]
);
expect(splits.map((s) => [s.participant_id, Number(s.share_percent)])).toEqual([
[me, 50],
[them, 50],
]);
expect(splits.every((s) => s.settled === false)).toBe(true);
});
it("flips which side of the balance the same rows sit on", async () => {
const id = await manualTxn("balance flip", me, [[me, 50], [them, 50]]);
// "They owe me": their split on a row I own.
const owedToMe = async () =>
Number(
(await queryRow<{ v: string | null }>(
`SELECT COALESCE(SUM(t.amount * ts.share_percent / 100), 0) AS v
FROM transaction_splits ts JOIN transactions t ON t.id = ts.transaction_id
WHERE t.id = $1 AND t.owner_id = $2 AND ts.participant_id <> $2`,
[id, me]
))!.v ?? 0
);
// "I owe them": my split on a row they own.
const owedByMe = async () =>
Number(
(await queryRow<{ v: string | null }>(
`SELECT COALESCE(SUM(t.amount * ts.share_percent / 100), 0) AS v
FROM transaction_splits ts JOIN transactions t ON t.id = ts.transaction_id
WHERE t.id = $1 AND t.owner_id <> $2 AND ts.participant_id = $2`,
[id, me]
))!.v ?? 0
);
expect(await owedToMe()).toBe(50);
expect(await owedByMe()).toBe(0);
await txPATCH(req(meEmail, { owner_id: them }), ctx(id));
expect(await owedToMe()).toBe(0);
expect(await owedByMe()).toBe(50);
});
it("rejects an unauthenticated caller", async () => {
const id = await manualTxn("no auth", me, [[me, 100]]);
const res = await txPATCH(req("nobody@example.test", { owner_id: them }), ctx(id));
expect(res.status).toBe(403);
expect(await ownerOf(id)).toBe(me);
});
});
describe("PATCH /api/statements/[id] — owner", () => {
async function statementWithRows(label: string, owner: number) {
const stmt = await queryRow<{ id: number }>(
`INSERT INTO statements (bank_name, account_number, billing_end_date, currency, filename, owner_id)
VALUES ('Owner Fixture Bank', $1, '2026-06-30','AUD', $2, $3) RETURNING id`,
[label, `owner-fixture-${label}.pdf`, owner]
);
// One row inheriting the owner (owner_id NULL) and one carrying its own
// copy — the live table holds 1,803 and 2,194 of these respectively, and
// updating `statements` alone would move only the first kind.
const inheriting = await queryRow<{ id: number }>(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type)
VALUES ($1,'2026-06-03','Owner fixture — inheriting', 10.00, 'debit') RETURNING id`,
[stmt!.id]
);
const carrying = await queryRow<{ id: number }>(
`INSERT INTO transactions (statement_id, transaction_date, description, amount, transaction_type, owner_id)
VALUES ($1,'2026-06-04','Owner fixture — carrying', 20.00, 'debit', $2) RETURNING id`,
[stmt!.id, owner]
);
return { stmtId: stmt!.id, inheriting: inheriting!.id, carrying: carrying!.id };
}
/** The owner the app actually reads: COALESCE(t.owner_id, s.owner_id). */
const effectiveOwner = async (id: number) =>
(await queryRow<{ v: number }>(
`SELECT COALESCE(t.owner_id, s.owner_id) AS v FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id WHERE t.id = $1`,
[id]
))!.v;
it("moves the statement and every row on it, both kinds", async () => {
const { stmtId, inheriting, carrying } = await statementWithRows("move", me);
const res = await stmtPATCH(req(meEmail, { owner_id: them }), ctx(stmtId));
expect(res.status).toBe(200);
expect((await res.json()).rows_moved).toBe(1); // only the one carrying a copy
expect(await effectiveOwner(inheriting)).toBe(them);
expect(await effectiveOwner(carrying)).toBe(them);
});
it("refuses a caller who does not own it", async () => {
const { stmtId, carrying } = await statementWithRows("not-mine", me);
const res = await stmtPATCH(req(themEmail, { owner_id: them }), ctx(stmtId));
expect(res.status).toBe(404);
expect(await effectiveOwner(carrying)).toBe(me);
});
it("rejects a participant that does not exist", async () => {
const { stmtId, carrying } = await statementWithRows("bad-participant", me);
const res = await stmtPATCH(req(meEmail, { owner_id: 999999 }), ctx(stmtId));
expect(res.status).toBe(400);
expect(await effectiveOwner(carrying)).toBe(me);
});
it("is a no-op when the owner is unchanged", async () => {
const { stmtId } = await statementWithRows("same-owner", me);
const res = await stmtPATCH(req(meEmail, { owner_id: me }), ctx(stmtId));
expect(res.status).toBe(200);
expect((await res.json()).rows_moved).toBe(0);
});
});
@@ -0,0 +1,216 @@
import { describe, it, expect } from "vitest";
import {
applyMapping,
inFileDuplicates,
last4,
splitByCoverage,
type ColumnMapping,
type ParsedTransaction,
} from "@/lib/csv-parser";
/**
* The statement-coverage rule, which replaced a whole Frollo-specific importer.
*
* The first aggregator import wrote 422 duplicate rows out of 550 — $1,023,824
* of movement counted twice — because "these accounts issue no statements" was
* asserted rather than queried. Amount-matching could not fix it either: the two
* sources decompose the same event differently, bundling a transfer fee where
* the statement itemises it.
*
* A statement's end date answers the question that actually has an answer: up
* to what date is this account complete? On the real 2026-08-13 export it takes
* 2,607 rows down to 171, with no account allowlist and no card exclusions.
*/
const tx = (o: Partial<ParsedTransaction> & { date: string }): ParsedTransaction => ({
description: "SOMETHING",
amount: 10,
transaction_type: "debit",
...o,
});
describe("last4", () => {
it("reads the same account written four different ways", () => {
// Every one of these forms appears in the real data.
expect(last4("xxxxxxxxxxxx2176")).toBe("2176");
expect(last4("235242176")).toBe("2176");
expect(last4("xxx-xxx xx4878")).toBe("4878");
expect(last4("4085-56264")).toBe("6264");
});
it("returns empty for an unusable identifier rather than guessing", () => {
expect(last4("N/A")).toBe("");
expect(last4("12")).toBe("");
expect(last4("")).toBe("");
});
});
describe("splitByCoverage", () => {
const coverage = [{ last4: "2176", coveredTo: "2026-07-26" }];
it("excludes a row on or before the account's newest statement", () => {
const { keep, covered } = splitByCoverage(
[tx({ date: "2026-07-25", account: "xxxxxxxxxxxx2176" })],
coverage
);
expect(covered).toHaveLength(1);
expect(keep).toHaveLength(0);
});
it("treats the watermark date itself as covered", () => {
const { covered } = splitByCoverage(
[tx({ date: "2026-07-26", account: "xxxxxxxxxxxx2176" })],
coverage
);
expect(covered).toHaveLength(1);
});
it("keeps a row after the watermark — the whole point of the feed", () => {
// The 2026-08-12 salary: the statement has not arrived, so this row is the
// only record of it and must survive.
const { keep } = splitByCoverage(
[tx({ date: "2026-08-12", account: "xxxxxxxxxxxx2176", amount: 10782 })],
coverage
);
expect(keep).toHaveLength(1);
});
it("keeps everything for an account with no statements", () => {
const { keep, covered } = splitByCoverage(
[tx({ date: "2020-01-01", account: "xxx-xxx xx4878" })],
coverage
);
expect(keep).toHaveLength(1);
expect(covered).toHaveLength(0);
});
it("keeps everything when no account column was mapped", () => {
// A single-account bank export has no account column, and a row must never
// be dropped on a guess about which account it belongs to.
const { keep, covered } = splitByCoverage([tx({ date: "2020-01-01" })], coverage);
expect(keep).toHaveLength(1);
expect(covered).toHaveLength(0);
});
it("applies each account's own watermark, not a global one", () => {
// The real export spans watermarks from 2026-03-31 to 2026-07-26. A single
// date would either import duplicates or discard real rows.
const { keep, covered } = splitByCoverage(
[
tx({ date: "2026-05-01", account: "xxxx0887" }), // covered to 2026-03-31 → keep
tx({ date: "2026-05-01", account: "xxxx2176" }), // covered to 2026-07-26 → drop
],
[
{ last4: "0887", coveredTo: "2026-03-31" },
{ last4: "2176", coveredTo: "2026-07-26" },
]
);
expect(keep).toHaveLength(1);
expect(keep[0].account).toBe("xxxx0887");
expect(covered).toHaveLength(1);
});
});
describe("inFileDuplicates", () => {
it("counts twins that a row id cannot catch", () => {
// A CDR re-consent re-exports an account's whole history under fresh ids
// while the originals survive, so source_ref sees two distinct rows. 385 of
// these were in one 2,563-row export and doubled every salary payment.
const rows = [
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
];
expect(inFileDuplicates(rows)).toBe(1);
});
it("does not flag the same amount on different days", () => {
expect(
inFileDuplicates([
tx({ date: "2026-07-07", amount: 5, description: "COFFEE" }),
tx({ date: "2026-07-08", amount: 5, description: "COFFEE" }),
])
).toBe(0);
});
it("does not flag the same amount on different accounts", () => {
// Both legs of an internal transfer: same day, same amount, different
// accounts, and both are real.
expect(
inFileDuplicates([
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "6264" }),
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "9940" }),
])
).toBe(0);
});
});
describe("applyMapping with account and row-id columns", () => {
const labels = ["transaction_id", "description", "amount", "transaction_date", "account_number"];
const mapping: ColumnMapping = {
dateCol: "transaction_date",
descriptionCol: "description",
amountMode: "single",
amountCol: "amount",
accountCol: "account_number",
sourceRefCol: "transaction_id",
};
it("carries the account and row id through", () => {
const out = applyMapping(
[["1326131276", "HDR Global Services", "10782.00", "2026-08-12", "xxxxxxxxxxxx4830"]],
labels,
mapping,
"YYYY-MM-DD"
);
expect(out).toHaveLength(1);
expect(out[0].account).toBe("xxxxxxxxxxxx4830");
expect(out[0].source_ref).toBe("1326131276");
// A positive single-column amount is money in.
expect(out[0].transaction_type).toBe("credit");
});
it("marks a non-AUD row as native and unconverted", () => {
// The 2026-08-12 salary is USD. Stored without this it reads as A$10,782 —
// about a third under — and sits in a column of AUD figures looking normal.
const out = applyMapping(
[["1", "HDR Global Services", "10782.00", "2026-08-12", "xxxx4830", "USD"]],
[...labels, "currency"],
{ ...mapping, currencyCol: "currency" },
"YYYY-MM-DD"
);
expect(out[0].foreign_currency_code).toBe("USD");
expect(out[0].foreign_currency_amount).toBe(10782);
});
it("leaves an AUD row unmarked", () => {
const out = applyMapping(
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830", "AUD"]],
[...labels, "currency"],
{ ...mapping, currencyCol: "currency" },
"YYYY-MM-DD"
);
expect(out[0].foreign_currency_code).toBeUndefined();
});
it("ignores a currency cell that is not a code", () => {
const out = applyMapping(
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830", ""]],
[...labels, "currency"],
{ ...mapping, currencyCol: "currency" },
"YYYY-MM-DD"
);
expect(out[0].foreign_currency_code).toBeUndefined();
});
it("leaves both undefined when the columns are not mapped", () => {
const out = applyMapping(
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830"]],
labels,
{ dateCol: "transaction_date", descriptionCol: "description", amountMode: "single", amountCol: "amount" },
"YYYY-MM-DD"
);
expect(out[0].account).toBeUndefined();
expect(out[0].source_ref).toBeUndefined();
expect(out[0].transaction_type).toBe("debit");
});
});
+37
View File
@@ -0,0 +1,37 @@
import { describe, it, expect } from "vitest";
import { readFileSync } from "fs";
import { resolve } from "path";
import { extractPdfText, liteLLMExtractor } from "../../lib/intake-extract";
import { reconcileItems, toLineItems, validateHeader } from "../../lib/intake-rules";
/**
* Opt-in: the real model over the real Burgertory export.
*
* INTAKE_LIVE_TEST=1 LITELLM_BASE_URL=http://localhost:4200 LITELLM_KEY=... npm test
*
* Skipped otherwise — CI has no LiteLLM, and a unit run must not depend on one.
* What it proves is the thing the mock cannot: that `cloud-budget` keeps the
* names verbatim ("Michael", "Joel", not "Burger") under the order-item prompt.
*/
const live = process.env.INTAKE_LIVE_TEST === "1" && !!process.env.LITELLM_BASE_URL && !!process.env.LITELLM_KEY;
describe.skipIf(!live)("live extraction via LiteLLM (opt-in)", () => {
it("Burgertory PDF: header exact, two items named verbatim, reconciles", async () => {
const text = await extractPdfText(readFileSync(resolve(__dirname, "../fixtures/intake/ubereats-burgertory-2026-08-19.pdf")));
const extracted = await liteLLMExtractor()([{ kind: "text", text, name: "Receipt_19Aug2026.pdf" }]);
const header = validateHeader(extracted);
expect(header.ok && header.header).toEqual({ merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4 });
const items = toLineItems(extracted.items);
expect(items.map((i) => i.description)).toEqual(["Michael", "Joel"]);
expect(reconcileItems(extracted).ok).toBe(true);
}, 120_000);
it("Coles PDF: header exact, 15 items", async () => {
const text = await extractPdfText(readFileSync(resolve(__dirname, "../fixtures/intake/ubereats-coles-2026-08-18.pdf")));
const extracted = await liteLLMExtractor()([{ kind: "text", text, name: "Receipt_18Aug2026.pdf" }]);
const header = validateHeader(extracted);
expect(header.ok && header.header).toEqual({ merchant: "Coles (Wyndham Vale)", order_date: "2026-08-18", total: 85.74 });
expect(toLineItems(extracted.items)).toHaveLength(15);
expect(reconcileItems(extracted).ok).toBe(true);
}, 120_000);
});
+300
View File
@@ -0,0 +1,300 @@
import { describe, it, expect } from "vitest";
import { readFileSync } from "fs";
import { resolve } from "path";
import {
classifyInput,
GROCERY_PATTERN,
merchantKey,
merchantMatches,
platformForSource,
reconcileItems,
resolveMerchantKind,
toLineItems,
validateHeader,
type ExtractedOrder,
} from "../../lib/intake-rules";
import { buildMessageContent, extractPdfText, parseExtraction, EXTRACT_RESPONSE_FORMAT, ExtractionError, liteLLMExtractor } from "../../lib/intake-extract";
import { askCard, candidatesCard, enrichedCard, type Block } from "../../lib/intake-blocks";
/**
* The intake router's pure rules, against the strings real receipts print.
* The two PDFs are the user's own Uber Eats exports (plan §3): a Coles grocery
* run and a Burgertory meal whose burgers are named after people.
*/
const fixtures = resolve(__dirname, "../fixtures/intake");
const pdf = (name: string) => readFileSync(resolve(fixtures, name));
describe("classifyInput — input type picks the extractor", () => {
it("PDF by MIME or by name", () => {
expect(classifyInput({ type: "application/pdf", name: "x" })).toBe("pdf");
expect(classifyInput({ type: "application/octet-stream", name: "Receipt_18Aug2026.pdf" })).toBe("pdf");
});
it("images and text", () => {
expect(classifyInput({ type: "image/png", name: "IMG_1.png" })).toBe("image");
expect(classifyInput({ type: "", name: "photo.HEIC" })).toBe("image");
expect(classifyInput({ type: "text/plain", name: "" })).toBe("text");
});
it("declines what it cannot read", () => {
expect(classifyInput({ type: "video/mp4", name: "clip.mp4" })).toBe("unsupported");
expect(classifyInput({ type: "application/zip", name: "a.zip" })).toBe("unsupported");
});
});
describe("platformForSource", () => {
it("maps the tap to expense_metadata.platform vocabulary", () => {
expect(platformForSource("uber_eats")).toBe("ubereats");
expect(platformForSource("doordash")).toBe("doordash");
expect(platformForSource("coles")).toBeNull();
expect(platformForSource("other")).toBeNull();
});
});
describe("merchant pattern — grocery | food | ask", () => {
it("the pattern list is a substring match on the normalised name", () => {
expect(resolveMerchantKind("uber_eats", "Coles (Wyndham Vale)")).toBe("grocery");
expect(resolveMerchantKind("other", "Woolworths Group Limited")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "WOOLIES METRO")).toBe("grocery");
expect(resolveMerchantKind("doordash", "ALDI Werribee")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "IGA Xpress")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "FoodWorks Manor Lakes")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Costco Wholesale")).toBe("grocery");
});
it("a delivery order that is not a grocer is a meal", () => {
expect(resolveMerchantKind("uber_eats", "Burgertory (Werribee)")).toBe("food");
expect(resolveMerchantKind("doordash", "Mad Mex")).toBe("food");
});
it("an empty or unknown merchant asks", () => {
expect(resolveMerchantKind("uber_eats", null)).toBe("ask");
expect(resolveMerchantKind("uber_eats", " ")).toBe("ask");
expect(resolveMerchantKind("doordash", "Unknown")).toBe("ask");
});
it("the Coles/Woolworths taps are ground truth", () => {
expect(resolveMerchantKind("coles", "Romeo's Food Hall")).toBe("grocery");
expect(resolveMerchantKind("woolworths", null)).toBe("grocery");
});
it("`other` with an unrecognised merchant asks; a remembered answer sticks", () => {
expect(resolveMerchantKind("other", "Romeo's Food Hall")).toBe("ask");
expect(resolveMerchantKind("other", "Romeo's Food Hall", "grocery")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Romeo's Food Hall", "grocery")).toBe("grocery");
});
it("merchantKey folds case and punctuation; merchantMatches allows containment only", () => {
expect(merchantKey("Coles (Wyndham Vale)")).toBe("coles wyndham vale");
expect(merchantMatches("Coles (Wyndham Vale)", "coles wyndham vale")).toBe(true);
expect(merchantMatches("Coles (Wyndham Vale)", "Coles")).toBe(true);
expect(merchantMatches("Order - Coles (Wyndham Vale)", "Coles (Wyndham Vale)")).toBe(true);
expect(merchantMatches("Burgertory (Werribee)", "Coles")).toBe(false);
expect(merchantMatches(null, "Coles")).toBe(false);
});
it("whole words, not substrings: `Bigals Pizza` contains `iga` and is neither a grocer nor an IGA", () => {
expect(GROCERY_PATTERN.test(merchantKey("Bigals Pizza"))).toBe(false);
expect(resolveMerchantKind("uber_eats", "Bigals Pizza")).toBe("food");
expect(merchantMatches("Bigals Pizza", "IGA")).toBe(false);
// and the grocers a boundary must not cost us
expect(resolveMerchantKind("uber_eats", "IGA Wyndham Vale")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Woolworths Group Limited")).toBe("grocery");
expect(resolveMerchantKind("uber_eats", "Coles (Wyndham Vale)")).toBe("grocery");
expect(merchantMatches("IGA Wyndham Vale", "IGA")).toBe(true);
});
});
describe("validateHeader — nothing is written from a half-read upload", () => {
const base: ExtractedOrder = { merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4, items: [] };
it("accepts a complete header and rounds the total", () => {
const r = validateHeader({ ...base, total: 22.404 });
expect(r.ok && r.header.total).toBe(22.4);
});
it("missing total", () => {
const r = validateHeader({ ...base, total: null });
expect(r.ok).toBe(false);
expect(!r.ok && r.missing).toEqual(["total"]);
});
it("missing or malformed date", () => {
expect(validateHeader({ ...base, order_date: null }).ok).toBe(false);
expect(validateHeader({ ...base, order_date: "19/8/26" }).ok).toBe(false);
});
it("a missing merchant is NOT a missing header — it becomes an ask", () => {
const r = validateHeader({ ...base, merchant: null });
expect(r.ok && r.header.merchant).toBeNull();
});
});
describe("reconcileItems — the missing-screenshot check", () => {
it("reconciles to the printed item subtotal, not the total: fees and promotions are not items", () => {
// The real Coles export: 15 lines summing $118.87 against a printed $118.94 subtotal and an $85.74 total.
const r = reconcileItems({
merchant: "Coles", order_date: "2026-08-18", total: 85.74, subtotal: 118.94,
items: [{ quantity: 1, description: "a", line_total: 118.87 }],
});
expect(r.ok).toBe(true);
expect(r.basis).toBe("subtotal");
});
it("falls back to the total when no subtotal is printed", () => {
const r = reconcileItems({ merchant: "x", order_date: "2026-08-19", total: 28, items: [{ quantity: 1, description: "Michael", line_total: 14 }, { quantity: 1, description: "Joel", line_total: 14 }] });
expect(r.ok).toBe(true);
expect(r.basis).toBe("total");
});
it("holds when the lines fall short — a shot is missing", () => {
const r = reconcileItems({ merchant: "x", order_date: "2026-08-19", total: 28, subtotal: 28, items: [{ quantity: 1, description: "Michael", line_total: 14 }] });
expect(r.ok).toBe(false);
expect(r.reason).toMatch(/14\.00.*28\.00/);
});
it("tolerance is $1 or 2%, whichever is larger", () => {
expect(reconcileItems({ merchant: "x", order_date: "d", total: 10, subtotal: 10, items: [{ quantity: 1, description: "a", line_total: 9.05 }] }).ok).toBe(true);
expect(reconcileItems({ merchant: "x", order_date: "d", total: 10, subtotal: 10, items: [{ quantity: 1, description: "a", line_total: 8.9 }] }).ok).toBe(false);
expect(reconcileItems({ merchant: "x", order_date: "d", total: 200, subtotal: 200, items: [{ quantity: 1, description: "a", line_total: 196.5 }] }).ok).toBe(true);
expect(reconcileItems({ merchant: "x", order_date: "d", total: 200, subtotal: 200, items: [{ quantity: 1, description: "a", line_total: 195 }] }).ok).toBe(false);
});
it("derives a line from quantity × unit price when no line total is printed", () => {
const r = reconcileItems({ merchant: "x", order_date: "d", total: 13.2, subtotal: 13.2, items: [{ quantity: 3, description: "Coke", unit_price: 4.4 }] });
expect(r.ok).toBe(true);
expect(r.items_sum).toBe(13.2);
});
it("no items at all is held; priceless lines (pasted text) are not", () => {
expect(reconcileItems({ merchant: "x", order_date: "d", total: 10, items: [] }).ok).toBe(false);
expect(reconcileItems({ merchant: "x", order_date: "d", total: 10, items: [{ quantity: 1, description: "a" }] }).ok).toBe(true);
});
});
describe("toLineItems — names verbatim, options kept", () => {
it("keeps a burger called Michael called Michael", () => {
const out = toLineItems([{ quantity: 1, description: "Michael", unit_price: 14, line_total: 14, options: ["Milk Bun"] }]);
expect(out).toEqual([{ qty: 1, description: "Michael", amount: 14, unit_price: 14, options: ["Milk Bun"] }]);
});
it("drops nameless lines and defaults quantity to 1", () => {
const out = toLineItems([{ quantity: null, description: " ", line_total: 1 }, { quantity: null, description: "Cucumbers Continental", unit_price: 2.9 }]);
expect(out).toEqual([{ qty: 1, description: "Cucumbers Continental", amount: 2.9, unit_price: 2.9 }]);
});
});
describe("extraction request and reply", () => {
it("one call: the prompt first, then every part in the order shared", () => {
const content = buildMessageContent([
{ kind: "image", mime: "image/png", bytes: Buffer.from("a"), name: "1.png" },
{ kind: "text", text: "Total $22.40", name: "r.pdf" },
{ kind: "image", mime: "image/jpeg", bytes: Buffer.from("b"), name: "2.jpg" },
]) as { type: string; text?: string; image_url?: { url: string } }[];
expect(content[0].text).toMatch(/VERBATIM/);
const images = content.filter((c) => c.type === "image_url").map((c) => c.image_url!.url);
expect(images).toEqual(["data:image/png;base64,YQ==", "data:image/jpeg;base64,Yg=="]);
expect(content.findIndex((c) => c.text?.includes("Total $22.40"))).toBeLessThan(content.findIndex((c) => c.image_url?.url.endsWith("Yg==")));
expect(EXTRACT_RESPONSE_FORMAT.json_schema.strict).toBe(true);
});
it("parses a fenced reply and coerces numbers", () => {
const r = parseExtraction('```json\n{"merchant":"Burgertory (Werribee)","order_date":"2026-08-19","total":"22.40","subtotal":28,"currency":null,"items":[{"quantity":1,"description":"Michael","unit_price":14,"line_total":14,"options":["Milk Bun"]}],"adjustments":[{"label":"Promotion","amount":-8}]}\n```');
expect(r.total).toBe(22.4);
expect(r.currency).toBe("AUD");
expect(r.items[0]).toEqual({ quantity: 1, description: "Michael", unit_price: 14, line_total: 14, options: ["Milk Bun"] });
expect(r.adjustments).toEqual([{ label: "Promotion", amount: -8 }]);
});
});
describe("PDF text — the Uber Eats export is born-digital text", () => {
it("Coles (Wyndham Vale) · 2026-08-18 · $85.74, 15 lines, weighed and substituted", async () => {
const text = await extractPdfText(pdf("ubereats-coles-2026-08-18.pdf"));
expect(text).toContain("Here's your receipt for Coles (Wyndham Vale).");
expect(text).toContain("18 Aug, 2026");
expect(text).toContain("Total $85.74");
expect(text).toContain("Item Subtotal $118.94");
expect(text).toContain("Final 1.16 kg $11.00/kg");
expect(text).toContain("Replaced Coles Beef Slow Cook Chuck Roast");
// 15 item lines: 13 priced per piece, 2 weighed per kg.
expect(text.match(/\$\d+\.\d\d\/pc$/gm)?.length).toBe(13);
expect(text.match(/\$\d+\.\d\d\/kg$/gm)?.length).toBe(2);
});
it("Burgertory (Werribee) · 2026-08-19 · $22.40, burgers named Michael and Joel", async () => {
const text = await extractPdfText(pdf("ubereats-burgertory-2026-08-19.pdf"));
expect(text).toContain("Here's your receipt for Burgertory (Werribee).");
expect(text).toContain("Total $22.40");
expect(text).toContain("1 Michael $14.00");
expect(text).toContain("1 Joel $14.00");
expect(text).toContain("Item Subtotal $28.00");
});
});
describe("cards carry the C5 action ids and JSON values", () => {
const c = { batchId: "11111111-2222-4333-8444-555555555555", merchant: "Burgertory (Werribee)", total: 22.4, orderDate: "2026-08-19", platform: "ubereats", itemCount: 2 };
it("candidates: one Use-this per candidate plus an explicit Create", () => {
const { blocks } = candidatesCard(c, [{ transaction_id: 42, description: "Order - Burgertory (Werribee)", transaction_date: "2026-08-17", amount: "19.90" }]);
const buttons = JSON.stringify(blocks);
expect(buttons).toContain('"action_id":"intake_use_candidate"');
expect(buttons).toContain('"action_id":"intake_create"');
const use = (blocks[1] as { accessory: { value: string } }).accessory.value;
expect(JSON.parse(use)).toEqual({ batch_id: c.batchId, transaction_id: 42 });
});
it("enriched: the confirmation text and a View link", () => {
const { text, blocks } = enrichedCard({ ...c, transactionId: 7 });
expect(text).toBe("Added 2 items to your $22.40 Uber Eats (Burgertory (Werribee)) order ✓");
expect(JSON.stringify(blocks)).toContain("/transactions?q=Burgertory");
});
it("ask: Groceries / A meal", () => {
const ids = JSON.stringify(askCard({ ...c, platform: null }).blocks);
expect(ids).toContain('"action_id":"intake_grocery"');
expect(ids).toContain('"action_id":"intake_meal"');
});
it("every value the card did not write is escaped, plain_text included", () => {
const hostile = { ...c, merchant: "Ben & Jerry's <fake>" };
const { text, blocks } = candidatesCard(hostile, [
{ transaction_id: 42, description: "SQ *BEN & JERRY <AU>", transaction_date: "2026-08-17", amount: "19.90" },
]);
const sections = JSON.stringify(blocks.filter((b) => (b as Block).type === "section"));
expect(sections).toContain("Ben &amp; Jerry's &lt;fake&gt;");
expect(sections).toContain("SQ *BEN &amp; JERRY &lt;AU&gt;");
expect(text).toContain("Ben &amp; Jerry's &lt;fake&gt;");
// plain_text disables formatting, not entity decoding — &, < and > are
// control characters everywhere in a Slack payload.
const whole = JSON.stringify(blocks);
expect(whole).not.toContain("<fake>");
expect(whole).not.toContain("<AU>");
});
it("the currency is model output too, so it is escaped like any other", () => {
// parseExtraction accepts any string for `currency`; unescaped, `<@U123>`
// would be a live mention on every card the batch produces.
const { text } = enrichedCard({ ...c, currency: "<@U123>", transactionId: 7 });
expect(text).toContain("&lt;@U123&gt; 22.40");
expect(text).not.toContain("<@U123>");
});
it("a foreign receipt is not printed as dollars — header and candidate rows alike", () => {
expect(enrichedCard({ ...c, currency: "USD", transactionId: 7 }).text).toBe(
"Added 2 items to your USD 22.40 Uber Eats (Burgertory (Werribee)) order ✓"
);
expect(enrichedCard({ ...c, currency: "AUD", transactionId: 7 }).text).toContain("$22.40");
expect(enrichedCard({ ...c, transactionId: 7 }).text).toContain("$22.40");
// A candidate row is a ledger transaction and carries its own currency; it
// used to print "$" beneath a header reading "USD 22.40".
const { blocks } = candidatesCard({ ...c, currency: "USD" }, [
{ transaction_id: 42, description: "Order - Burgertory", transaction_date: "2026-08-17", amount: "19.90", currency: "USD" },
]);
expect(JSON.stringify(blocks)).toContain("USD 19.90");
});
});
describe("liteLLMExtractor — a finish_reason that is present must be `stop`", () => {
const good = { merchant: "Burgertory (Werribee)", order_date: "2026-08-19", total: 22.4, subtotal: 28, currency: "AUD", items: [], adjustments: [] };
const extractor = (finish_reason?: string) =>
liteLLMExtractor({
baseUrl: "http://litellm.test",
key: "k",
fetchImpl: (async () =>
Response.json({
choices: [{ message: { content: JSON.stringify(good) }, ...(finish_reason ? { finish_reason } : {}) }],
})) as unknown as typeof fetch,
});
it("accepts `stop`, and — on purpose — a reply that reports no reason at all", async () => {
expect((await extractor("stop")([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
expect((await extractor()([{ kind: "text", text: "x", name: "x" }])).total).toBe(22.4);
});
it("rejects any other reason, naming it — the strict schema only held for a reply that finished", async () => {
// An absent reason is NOT rejected: see the comment in intake-extract.ts.
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(ExtractionError);
await expect(extractor("length")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/too long/);
await expect(extractor("content_filter")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/content_filter/);
await expect(extractor("tool_calls")([{ kind: "text", text: "x", name: "x" }])).rejects.toThrow(/tool_calls/);
});
});
+72
View File
@@ -0,0 +1,72 @@
import { describe, expect, it } from "vitest";
import { tidyMerchant } from "@/lib/merchant-label";
/**
* Board 212. Every fixture below is a real `display_name` from order_feed.
* The cases that matter most are the ones that must NOT change: this is a
* presentational tidy and merging anything here would be a data error wearing
* a stylesheet.
*/
describe("tidyMerchant", () => {
it("capitalises a bare domain used as a name", () => {
expect(tidyMerchant("amazon.com.au")).toBe("Amazon.com.au");
expect(tidyMerchant("cdkeys.com")).toBe("Cdkeys.com");
});
it("NEVER merges the country marketplaces", () => {
// The single most important assertion in this file. Amazon.in is a
// different marketplace with different currency and geography, and 289
// orders sit behind it.
expect(tidyMerchant("Amazon.in")).toBe("Amazon.in");
expect(tidyMerchant("amazon.com.au")).not.toBe(tidyMerchant("Amazon.in"));
});
it("leaves real brand names that contain a TLD alone", () => {
for (const brand of ["Kogan.com", "GOG.com", "AliExpress.com", "Catch.com.au", "GeekBuying.com"]) {
expect(tidyMerchant(brand)).toBe(brand);
}
});
it("does not uppercase a deliberately lower-cased brand", () => {
// The bug the corpus run found: gating on "starts lower-case" rather than
// on "is a bare domain" turned this into "EBay Commerce Australia".
expect(tidyMerchant("eBay Commerce Australia Pty Ltd.")).toBe("eBay Commerce Australia");
expect(tidyMerchant("iRobot Australia Pty Ltd")).toBe("iRobot Australia");
});
it("drops trailing corporate suffixes", () => {
expect(tidyMerchant("Apple Pty Ltd.")).toBe("Apple");
expect(tidyMerchant("Microsoft Pty. Limited")).toBe("Microsoft");
expect(tidyMerchant("Meta Platforms, Inc.")).toBe("Meta Platforms");
expect(tidyMerchant("Domino's Pizza Enterprises Limited")).toBe("Domino's Pizza Enterprises");
expect(tidyMerchant("Blinks Labs GmbH")).toBe("Blinks Labs");
expect(tidyMerchant("Rasier New Zealand Limited")).toBe("Rasier New Zealand");
});
it("strips two suffixes when a name carries two", () => {
expect(tidyMerchant("Samsung Electronics Co. Ltd.")).toBe("Samsung Electronics");
expect(tidyMerchant("DiDi Mobility Information Technology Pte. Ltd."))
.toBe("DiDi Mobility Information Technology");
});
it("does not leave a dangling connector", () => {
// 'Co' is part of this brand. Stripping it produced "Coburger &".
expect(tidyMerchant("Coburger & Co")).toBe("Coburger & Co");
});
it("keeps a name that would be reduced to almost nothing", () => {
// "UT" is too short to stand on its own; better scruffy than cryptic.
expect(tidyMerchant("UT LLC")).toBe("UT LLC");
});
it("normalises whitespace and tolerates junk", () => {
expect(tidyMerchant(" Yaffa Media Pty Ltd ")).toBe("Yaffa Media");
expect(tidyMerchant("")).toBe("");
});
it("is idempotent", () => {
for (const n of ["amazon.com.au", "Apple Pty Ltd.", "Coburger & Co", "Amazon.in"]) {
expect(tidyMerchant(tidyMerchant(n))).toBe(tidyMerchant(n));
}
});
});
+3 -1
View File
@@ -508,7 +508,9 @@ describe("HTML entities in line items", () => {
// The ampersand decodes, and the bullet becomes a boundary, not text.
expect(item!.description).toContain("Sweet & Sour Crunch");
expect(item!.description).not.toContain("•");
expect(item!.options.length).toBeGreaterThan(0);
// `options` is optional on the parsed item; `?? 0` keeps the assertion honest
// (an absent list fails) instead of silencing the check with a non-null bang.
expect(item!.options?.length ?? 0).toBeGreaterThan(0);
});
it("leaves a literal &amp;bull; alone rather than turning it into a bullet", () => {
+2 -2
View File
@@ -2,7 +2,7 @@ import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import {
OWNER_SCOPE,
MY_SPEND_SCOPE,
STATEMENTS_JOIN,
EXCLUDE_NON_SPEND,
EXCLUDE_RECONCILED_SOURCE,
@@ -57,7 +57,7 @@ export async function GET(req: NextRequest) {
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1
${STATEMENTS_JOIN}
WHERE ${OWNER_SCOPE} = $1
WHERE ${MY_SPEND_SCOPE()}
AND ${NET_SPEND_ROWS}
AND ${EXCLUDE_NON_SPEND}
AND ${EXCLUDE_RECONCILED_SOURCE}
+2 -2
View File
@@ -1,7 +1,7 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_RECONCILED_SOURCE, mySplitOf, toDateStr } from "@/lib/analytics-sql";
import { MY_SPEND_SCOPE, STATEMENTS_JOIN, EXCLUDE_RECONCILED_SOURCE, mySplitOf, toDateStr } from "@/lib/analytics-sql";
/**
* Fees and interest over an explicit window.
@@ -71,7 +71,7 @@ export async function GET(req: NextRequest) {
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1
${STATEMENTS_JOIN}
WHERE ${OWNER_SCOPE} = $1
WHERE ${MY_SPEND_SCOPE()}
AND t.transaction_type IN ('fee', 'interest')
AND ${EXCLUDE_RECONCILED_SOURCE}
${txnWindow}
@@ -1,7 +1,7 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql";
import { MY_SPEND_SCOPE, STATEMENTS_JOIN, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql";
import { bankLabel } from "@/lib/queries";
const MY_AMOUNT = mySplitOf(`COALESCE(t.amount_aud, t.amount)`);
@@ -46,7 +46,7 @@ export async function GET(
${STATEMENTS_JOIN}
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1
WHERE ${OWNER_SCOPE} = $1
WHERE ${MY_SPEND_SCOPE()}
AND t.transaction_type IN ('debit', 'fee', 'interest', 'refund', 'credit')
AND COALESCE(o.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) = $2
AND ${EXCLUDE_RECONCILED_SOURCE}
+3 -3
View File
@@ -1,7 +1,7 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf, toDateStr } from "@/lib/analytics-sql";
import { MY_SPEND_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf, toDateStr } from "@/lib/analytics-sql";
// Split-adjusted amount helper (positive for spend, negative for refunds)
const MY_AMOUNT = mySplitOf(`COALESCE(t.amount_aud, t.amount)`);
@@ -65,7 +65,7 @@ export async function GET(req: NextRequest) {
${STATEMENTS_JOIN}
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1
WHERE ${OWNER_SCOPE} = $1
WHERE ${MY_SPEND_SCOPE()}
AND t.transaction_type IN ('debit', 'fee', 'interest', 'refund', 'credit')
AND t.transaction_date >= $2
AND ${EXCLUDE_NON_SPEND}
@@ -91,7 +91,7 @@ export async function GET(req: NextRequest) {
${STATEMENTS_JOIN}
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1
WHERE ${OWNER_SCOPE} = $1
WHERE ${MY_SPEND_SCOPE()}
AND t.transaction_type IN ('debit', 'fee', 'interest', 'refund', 'credit')
AND t.transaction_date >= $2
AND COALESCE(o.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) = ANY($3)
+2 -1
View File
@@ -3,6 +3,7 @@ import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import {
OWNER_SCOPE,
MY_SPEND_SCOPE,
STATEMENTS_JOIN,
EFFECTIVE_CATEGORY,
EXCLUDE_NON_SPEND,
@@ -46,7 +47,7 @@ export async function GET(req: NextRequest) {
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1
${STATEMENTS_JOIN}
WHERE ${OWNER_SCOPE} = $1
WHERE ${MY_SPEND_SCOPE()}
AND ${NET_SPEND_ROWS}
AND ${EXCLUDE_NON_SPEND}
AND ${EXCLUDE_RECONCILED_SOURCE}
+2 -2
View File
@@ -1,7 +1,7 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { queryRaw } from "@/lib/db";
import { OWNER_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql";
import { MY_SPEND_SCOPE, STATEMENTS_JOIN, EXCLUDE_NON_SPEND, EXCLUDE_RECONCILED_SOURCE, EFFECTIVE_CATEGORY, mySplitOf } from "@/lib/analytics-sql";
export async function GET(req: NextRequest) {
const user = await getCurrentUser(req);
@@ -28,7 +28,7 @@ export async function GET(req: NextRequest) {
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN transaction_splits ts ON ts.transaction_id = t.id AND ts.participant_id = $1
${STATEMENTS_JOIN}
WHERE ${OWNER_SCOPE} = $1
WHERE ${MY_SPEND_SCOPE()}
AND t.transaction_type IN ('debit', 'fee')
AND ${EXCLUDE_NON_SPEND}
AND ${EXCLUDE_RECONCILED_SOURCE}
+9 -1
View File
@@ -8,6 +8,12 @@ export async function POST(req: NextRequest) {
const body = await req.json() as {
bank_name: string;
/**
* Optional provenance label, stored on every row as `source`. With a
* `source_ref` per row it makes a re-import a no-op; without it this path
* has no idempotency at all.
*/
source?: string;
transactions: {
date: string;
description: string;
@@ -17,6 +23,8 @@ export async function POST(req: NextRequest) {
foreign_currency_amount?: number;
foreign_currency_code?: string;
category?: string;
account?: string;
source_ref?: string;
}[];
};
@@ -25,7 +33,7 @@ export async function POST(req: NextRequest) {
}
const tagId = await ensureTag("csv-import", "#8b5cf6");
const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId);
const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId, body.source);
return NextResponse.json({ inserted }, { status: 201 });
}
@@ -0,0 +1,17 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { getStatementCoverage } from "@/lib/queries";
/**
* How far each account's statements already reach.
*
* Read by the CSV import modal so it can drop rows the ledger already holds.
* The first Frollo import went in without this and wrote 422 duplicates out of
* 550 — the same payments the statements already carried, itemised and
* converted. On the 2026-08-13 export the rule takes 2,607 rows down to 171.
*/
export async function GET(req: NextRequest) {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
return NextResponse.json({ coverage: await getStatementCoverage() });
}
+67
View File
@@ -0,0 +1,67 @@
import { NextRequest, NextResponse } from "next/server";
import { sha256Of, type StoredFile } from "@/lib/intake-files";
import { liteLLMExtractor } from "@/lib/intake-extract";
import { pantryForwarder } from "@/lib/intake-pantry";
import { isIntakeSource } from "@/lib/intake-rules";
import { MAX_FILES, runIntake } from "@/lib/intake-router";
/**
* The one upload point (plan §11 C1). n8n forwards the iOS Shortcut's
* multipart POST here: `source`, `shared_by`, `batch_id`, and 1–6 files under
* `file`. The router reads what it is given and does every step that applies.
*
* Auth is a shared secret, like the two ingest siblings: this is machine-to-
* machine and there is no browser session to forward. Its own token so the
* Shortcut's credential rotates without touching the order or receipt lanes.
* Fails closed when unset.
*/
function authorised(req: NextRequest): boolean {
const expected = process.env.INTAKE_TOKEN;
if (!expected) return false;
const got = req.headers.get("x-ingest-token");
return !!got && got === expected;
}
const MAX_BYTES = 12_000_000;
export async function POST(req: NextRequest) {
if (!authorised(req)) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
let form: FormData;
try {
form = await req.formData();
} catch {
return NextResponse.json({ error: "multipart/form-data body required" }, { status: 400 });
}
const source = form.get("source");
if (!isIntakeSource(source)) {
return NextResponse.json({ error: "source must be one of uber_eats|doordash|coles|woolworths|other" }, { status: 400 });
}
const sharedBy = typeof form.get("shared_by") === "string" ? String(form.get("shared_by")).trim() || null : null;
const batchIdRaw = form.get("batch_id");
const batchId = typeof batchIdRaw === "string" && /^[0-9a-f-]{36}$/i.test(batchIdRaw) ? batchIdRaw.toLowerCase() : null;
if (batchIdRaw && !batchId) return NextResponse.json({ error: "batch_id must be a uuid" }, { status: 400 });
const files: StoredFile[] = [];
for (const entry of form.getAll("file")) {
if (!(entry instanceof File) || !entry.size) continue;
if (entry.size > MAX_BYTES) return NextResponse.json({ error: `${entry.name} is larger than 12 MB` }, { status: 400 });
const bytes = Buffer.from(await entry.arrayBuffer());
files.push({ name: entry.name || `file-${files.length + 1}`, type: entry.type || "", bytes, sha256: sha256Of(bytes) });
}
if (!files.length) return NextResponse.json({ error: "at least one file is required under field 'file'" }, { status: 400 });
if (files.length > MAX_FILES) return NextResponse.json({ error: `at most ${MAX_FILES} files per batch` }, { status: 400 });
try {
const result = await runIntake(
{ batchId, source, sharedBy, files },
{ extractor: liteLLMExtractor(), pantry: pantryForwarder() }
);
const { http_status, ...body } = result;
return NextResponse.json(body, { status: http_status });
} catch (e) {
const message = e instanceof Error ? e.message : String(e);
return NextResponse.json({ error: message }, { status: 500 });
}
}
+28
View File
@@ -6,6 +6,8 @@ import {
slackUserForParticipant,
} from "@/lib/slack-verify";
import { nudgeBlocks, detailsModal, partnerNudgeBlocks } from "@/lib/slack-blocks";
import { handleIntakeAction, isIntakeAction, parseActionValue } from "@/lib/intake-actions";
import { pantryForwarder } from "@/lib/intake-pantry";
import { completeSplit } from "@/lib/splits";
import {
RATINGS,
@@ -73,6 +75,22 @@ export async function POST(req: NextRequest) {
if (payload.type !== "block_actions") return new NextResponse(null, { status: 200 });
const action = payload.actions?.[0];
// Intake taps (plan §11 C5) act on a persisted batch, not a transaction, so
// their `value` is JSON `{batch_id, ...}` rather than `<id>:<verb>`. They are
// dispatched on action_id before the order-nudge parse below, which would
// otherwise read the JSON as a non-numeric transaction id and give up.
if (isIntakeAction(action?.action_id)) {
const reply = await handleIntakeAction(action.action_id, parseActionValue(action?.value), { pantry: pantryForwarder() });
await updateMessage(
payload.response_url,
reply.replace
? { replace_original: true, blocks: reply.blocks, text: reply.text }
: { response_type: "ephemeral", replace_original: false, text: reply.text }
);
return NextResponse.json({ blocks: reply.blocks, text: reply.text });
}
// A button carries `value`; a select carries it on the chosen option. The
// ratings moved to a select because four buttons became four full-width rows
// on Slack mobile, so both shapes have to resolve or rating silently stops
@@ -216,6 +234,16 @@ async function buildDetailsModal(transactionId: number, participantId: number) {
`SELECT merchant_normalized AS merchant, line_items
FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`,
[transactionId]
);
+57
View File
@@ -1,6 +1,7 @@
import { NextRequest, NextResponse } from "next/server";
import { getStatementById } from "@/lib/queries";
import { getCurrentUser } from "@/lib/auth";
import { prisma, queryRaw } from "@/lib/db";
export async function GET(
req: NextRequest,
@@ -15,3 +16,59 @@ export async function GET(
}
return NextResponse.json(stmt);
}
/**
* Reassign a statement — whose account it is.
*
* The statements page has had an owner dropdown since it was built, wired to
* `useUpdateStatement`, which PATCHed a route that had no PATCH handler. Every
* change 405'd, and because the hook never checked `res.ok` the failure was
* silent: the select snapped back on refetch and looked like a UI glitch.
*
* Both tables are written because both carry the owner. The effective owner is
* `COALESCE(t.owner_id, s.owner_id)`, and 2,194 statement rows carry their own
* copy against 1,803 that inherit — so updating `statements` alone would move
* less than half the rows and split one account's history between two people.
* All 2,194 agree with their statement today; a row that disagrees was set
* deliberately and is left alone rather than swept up.
*/
export async function PATCH(
req: NextRequest,
{ params }: { params: Promise<{ id: string }> }
) {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
const { id } = await params;
const statementId = Number(id);
const stmt = await getStatementById(statementId);
// Only the current owner reassigns. A statement is an account, not a shared
// expense — there is no split that grants a second person a say in it.
if (!stmt || stmt.owner_id !== user.id) {
return NextResponse.json({ error: "Not found" }, { status: 404 });
}
const { owner_id } = (await req.json()) as { owner_id?: number };
if (typeof owner_id !== "number" || !Number.isInteger(owner_id)) {
return NextResponse.json({ error: "owner_id must be a participant id" }, { status: 400 });
}
const known = await queryRaw(`SELECT id FROM participants WHERE id = $1`, [owner_id]);
if (!known.length) {
return NextResponse.json({ error: "Unknown participant" }, { status: 400 });
}
if (owner_id === stmt.owner_id) {
return NextResponse.json({ ok: true, statement_id: statementId, rows_moved: 0 });
}
const [, rowsMoved] = await prisma.$transaction([
prisma.$executeRawUnsafe(`UPDATE statements SET owner_id = $1 WHERE id = $2`, owner_id, statementId),
prisma.$executeRawUnsafe(
`UPDATE transactions SET owner_id = $1 WHERE statement_id = $2 AND owner_id = $3`,
owner_id,
statementId,
stmt.owner_id
),
]);
return NextResponse.json({ ok: true, statement_id: statementId, rows_moved: rowsMoved });
}
@@ -0,0 +1,58 @@
import { NextRequest, NextResponse } from "next/server";
import { LineItemsValidationError, replaceLineItems, TransactionNotFoundError, validateLineItems } from "@/lib/line-items";
/**
* Enrich endpoint (plan §11 C2): idempotent replace of the itemised receipt
* behind a transaction.
*
* Accepts either machine token. Pantry already holds RECEIPT_INGEST_TOKEN for
* the ING-10 lane and calls this on every confirm and on undo; the intake
* router holds INTAKE_TOKEN. No session auth — there is no browser here.
* Fails closed when neither is configured.
*
* Never creates a transaction: 404 when the id does not exist. Creates the
* `expense_metadata` row when the transaction has none.
*/
function authorised(req: NextRequest): boolean {
const got = req.headers.get("x-ingest-token");
if (!got) return false;
const accepted = [process.env.INTAKE_TOKEN, process.env.RECEIPT_INGEST_TOKEN].filter(Boolean);
return accepted.includes(got);
}
export async function PUT(req: NextRequest, { params }: { params: Promise<{ id: string }> }) {
if (!authorised(req)) return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
const { id } = await params;
const transactionId = Number(id);
if (!Number.isInteger(transactionId) || transactionId <= 0) return NextResponse.json({ error: "invalid transaction id" }, { status: 400 });
let body: { line_items?: unknown; source?: unknown; batch_id?: unknown; replace?: unknown; platform?: unknown };
try {
body = await req.json();
} catch {
return NextResponse.json({ error: "invalid JSON" }, { status: 400 });
}
if (body.replace !== true) return NextResponse.json({ error: "replace: true is required — this endpoint only replaces" }, { status: 400 });
if (body.source !== "intake" && body.source !== "pantry") return NextResponse.json({ error: "source must be 'intake' or 'pantry'" }, { status: 400 });
const batchId = typeof body.batch_id === "string" && body.batch_id ? body.batch_id : null;
const platform = typeof body.platform === "string" && body.platform ? body.platform : null;
try {
const lineItems = validateLineItems(body.line_items);
const result = await replaceLineItems(transactionId, lineItems, { source: body.source, platform, batchId });
return NextResponse.json({
transaction_id: result.transaction_id,
metadata_id: result.metadata_id,
created_metadata: result.created_metadata,
previous_line_items: result.previous_line_items,
line_items: result.line_items,
});
} catch (e) {
if (e instanceof LineItemsValidationError) return NextResponse.json({ error: e.message }, { status: 400 });
if (e instanceof TransactionNotFoundError) return NextResponse.json({ error: e.message }, { status: 404 });
const message = e instanceof Error ? e.message : String(e);
return NextResponse.json({ error: message }, { status: 500 });
}
}
+32 -1
View File
@@ -34,11 +34,42 @@ export async function GET(
-- through matched_transaction_id. Both directions have to resolve or the
-- detail is missing on exactly the orders that were paid by card.
WHERE transaction_id = $1 OR matched_transaction_id = $1
-- The same order as replaceLineItems in line-items.ts and matchTransaction
-- in intake-router.ts. Both directions can resolve at once — a statement
-- line owning its own row while a delivery receipt points at it — and an
-- unordered LIMIT 1 then reads a different row from the one an enrichment
-- wrote, so a PUT that reported success showed nothing on the screen.
-- NULLS LAST because DESC is NULLS FIRST and the pointing row's
-- transaction_id is NULL.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`,
[Number(id)]
);
// Phase 2 (board 205). A BNPL leg has NO expense_metadata row of its own —
// the plan is four transactions against one order — so without this the four
// Afterpay debits behind the A$1,599 DJI drone stay four bare "Afterpay
// $399.75" rows naming nothing. That is the surface where this was noticed.
const link = await queryRow<{
entity_key: string; leg_kind: string; leg_index: number | null;
leg_count: number | null; canonical_name: string | null;
platform: string | null; order_total: string | null; currency: string | null;
}>(
`SELECT l.entity_key, l.leg_kind, l.leg_index::int, l.leg_count::int,
e.canonical_name, o.platform, o.order_total, o.currency
FROM order_transaction_links l
LEFT JOIN entities e ON e.entity_key = l.entity_key
LEFT JOIN entity_orders o ON o.entity_id = e.id
WHERE l.transaction_id = $1
LIMIT 1`,
[Number(id)]
);
// Not an order — most transactions aren't. Null, not 404: the caller is
// asking "is there a receipt behind this?", and "no" is a normal answer.
return NextResponse.json(row ?? null);
if (!row && !link) return NextResponse.json(null);
// expense_metadata carries the itemised receipt and stays primary; the link
// adds what it cannot express — which order this is a leg OF, and which leg.
return NextResponse.json({ ...(row ?? {}), order_link: link ?? null });
}
+79 -3
View File
@@ -3,6 +3,7 @@ import { getTransactionById, canAccessTransactions } from "@/lib/queries";
import { getCurrentUser } from "@/lib/auth";
import { prisma } from "@/lib/db";
import { queryRaw } from "@/lib/db";
import { isPaymentMethod } from "@/lib/payment-methods";
const VALID_TYPES = ["debit", "credit", "payment", "refund", "fee", "interest", "transfer"];
@@ -34,7 +35,7 @@ export async function PATCH(
}
const body = await req.json();
const { category, merchant_normalized, notes, transaction_type, my_share_percent, description, amount, transaction_date, trip_id, payment_method } = body as {
const { category, merchant_normalized, notes, transaction_type, my_share_percent, description, amount, transaction_date, trip_id, payment_method, owner_id, release } = body as {
category?: string;
merchant_normalized?: string;
notes?: string;
@@ -45,6 +46,9 @@ export async function PATCH(
amount?: number;
transaction_date?: string;
trip_id?: number | null;
owner_id?: number;
/** Acknowledge that reassigning will remove the row from my own view. */
release?: boolean;
};
if (my_share_percent !== undefined && my_share_percent !== null) {
@@ -87,9 +91,12 @@ export async function PATCH(
// payment_method is a property of the transaction itself, not a user override
// of extracted data, so it lives on the transactions table.
//
// The allowlist is PAYMENT_METHODS, not a local literal: this one omitted
// 'credits' from migration 0018 until 2026-08-17, so the DB admitted a
// gift-card row that this route then refused to set or correct.
if (payment_method !== undefined) {
const VALID_METHODS = ["card", "cash", "bank_transfer", "other"];
if (payment_method !== null && !VALID_METHODS.includes(payment_method)) {
if (payment_method !== null && !isPaymentMethod(payment_method)) {
return NextResponse.json({ error: "Invalid payment_method" }, { status: 400 });
}
await queryRaw(
@@ -98,6 +105,75 @@ export async function PATCH(
);
}
// Owner — who actually paid. A direct column, not an override: it decides
// whose account the money left, which every balance and every spend analytic
// scopes on (MY_SPEND_SCOPE / OWNER_SCOPE).
//
// Handled here rather than in the direct-fields block above because it has
// its own rules, and because getting it wrong is not a cosmetic error: it
// moves money between two people's ledgers.
if (owner_id !== undefined) {
if (typeof owner_id !== "number" || !Number.isInteger(owner_id)) {
return NextResponse.json({ error: "owner_id must be a participant id" }, { status: 400 });
}
const txRows = await queryRaw<{ statement_id: number | null; owner_id: number | null }>(
`SELECT statement_id, owner_id FROM transactions WHERE id = $1`,
[transactionId]
);
const tx = txRows[0];
if (!tx) return NextResponse.json({ error: "Not found" }, { status: 404 });
// A statement row's owner is the statement's owner — the effective owner is
// COALESCE(t.owner_id, s.owner_id), so setting it here would either be a
// no-op or would silently detach one row from the account it was extracted
// from. Reassigning the statement is the correct move and moves its rows
// with it.
if (tx.statement_id) {
return NextResponse.json(
{
error:
"This row came from a statement, so its owner is the statement's owner. Change the statement's owner instead — that moves every row on it.",
code: "statement_owned",
},
{ status: 400 }
);
}
const known = await queryRaw(`SELECT id FROM participants WHERE id = $1`, [owner_id]);
if (!known.length) {
return NextResponse.json({ error: "Unknown participant" }, { status: 400 });
}
// The one-way door. Access is "owner OR holds a split"
// (canAccessTransactions), so handing a row to someone else while holding
// no split on it removes it from the caller's list and 404s every route
// that could put it back — only the new owner can undo it. Splitting first
// keeps the row reachable AND is what makes the balance correct, so the
// refusal points at the step that was skipped rather than just blocking.
if (owner_id !== user.id && !release) {
const mine = await queryRaw(
`SELECT 1 FROM transaction_splits WHERE transaction_id = $1 AND participant_id = $2`,
[transactionId, user.id]
);
if (!mine.length) {
return NextResponse.json(
{
error:
"You hold no split on this transaction, so reassigning it would remove it from your view for good — only the new owner could change it back. Add your split first, or confirm you are giving it away entirely.",
code: "would_lose_access",
},
{ status: 409 }
);
}
}
// Existing splits are deliberately left alone. They record shares, not
// direction: getParticipantBalances derives who owes whom from ownership,
// so a 50/50 row flips from "they owe me" to "I owe them" with no rewrite.
await queryRaw(`UPDATE transactions SET owner_id = $1 WHERE id = $2`, [owner_id, transactionId]);
}
// category/merchant/notes/my_share_percent/trip_id go through the overrides table
const hasOverride = category !== undefined || merchant_normalized !== undefined || notes !== undefined || my_share_percent !== undefined || trip_id !== undefined;
if (!hasOverride) {
+7
View File
@@ -3,6 +3,7 @@ import { getCurrentUser } from "@/lib/auth";
import { getTransactions } from "@/lib/queries";
import { queryRaw } from "@/lib/db";
import { completeSplit } from "@/lib/splits";
import { isPaymentMethod } from "@/lib/payment-methods";
export async function GET(req: NextRequest) {
const user = await getCurrentUser(req);
@@ -53,6 +54,12 @@ export async function POST(req: NextRequest) {
return NextResponse.json({ error: "date, description, amount are required" }, { status: 400 });
}
// Validated here rather than left to the CHECK constraint, which surfaces as
// a 500 with a Postgres error rather than a 400 naming the field.
if (body.payment_method && !isPaymentMethod(body.payment_method)) {
return NextResponse.json({ error: "Invalid payment_method" }, { status: 400 });
}
// Insert manual transaction with no statement (statement_id = NULL, owner_id set directly)
const txRows = await queryRaw<{ id: number }>(
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_normalized, category, payment_method, row_index)
+25 -42
View File
@@ -19,16 +19,20 @@ import { useQueryClient } from "@tanstack/react-query";
import { useMonthlyAnalytics, useDailySpend, useTransactions, useUpdateTransaction } from "@/lib/hooks";
import { formatCategory, CATEGORIES } from "@/lib/categories";
import { CATEGORY_COLORS, CHART, TOOLTIP_STYLE } from "@/lib/category-colors";
// The partial-month comparison rules live in their own module so they can be
// tested. They used to be closures here reading `new Date()` inline, which is
// why Priority 0 item 6 had no test (#155). See src/lib/partial-month.ts.
import {
currentMonthStr,
prevMonth,
daysInMonthOf,
elapsedDays,
spendThrough,
comparisonBasis,
baselineMonths,
baselineAverage,
} from "@/lib/partial-month";
function currentMonthStr(): string {
const now = new Date();
return `${now.getFullYear()}-${String(now.getMonth() + 1).padStart(2, "0")}`;
}
function prevMonth(m: string): string {
const [year, month] = m.split("-").map(Number);
const d = new Date(year, month - 2, 1);
return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}`;
}
function formatMonth(m: string): string {
const [year, month] = m.split("-");
return new Date(Number(year), Number(month) - 1, 1).toLocaleString("default", { month: "long", year: "numeric" });
@@ -37,26 +41,6 @@ function formatShortMonth(m: string): string {
const [year, month] = m.split("-");
return new Date(Number(year), Number(month) - 1, 1).toLocaleString("default", { month: "short" });
}
function daysInMonthOf(m: string): number {
const [year, month] = m.split("-").map(Number);
return new Date(year, month, 0).getDate();
}
/**
* How much of `month` has actually happened. A month in progress is only
* complete up to today; every earlier month is complete.
*/
function elapsedDays(m: string): number {
return m === currentMonthStr() ? new Date().getDate() : daysInMonthOf(m);
}
/** Spend in `month` from day 1 through `throughDay` inclusive. */
function spendThrough(days: Record<number, number> | undefined, throughDay: number): number {
if (!days) return 0;
let sum = 0;
for (const [d, v] of Object.entries(days)) {
if (Number(d) <= throughDay) sum += v;
}
return sum;
}
function fmt(n: number): string { return `$${Math.round(n).toLocaleString()}`; }
function fmtExact(n: number): string { return `$${n.toFixed(2)}`; }
function fmtSigned(n: number): string { return `${n >= 0 ? "+" : "−"}$${Math.abs(n) >= 100 ? Math.round(Math.abs(n)).toLocaleString() : Math.abs(n).toFixed(0)}`; }
@@ -221,8 +205,7 @@ export default function AnalyticsPage() {
// A month in progress is only comparable to prior months through the same day.
// Comparing 27 days of July against a full June always flattered July.
const compareDay = elapsedDays(selectedMonth);
const selectedIsPartial = selectedMonth === currentMonthStr();
const { compareDay, isPartial: selectedIsPartial } = comparisonBasis(selectedMonth);
// Category rows for selected month
const categoryRows = useMemo(() => {
@@ -299,8 +282,10 @@ export default function AnalyticsPage() {
// Only complete months with data form the baseline. A month still in
// progress has no spend recorded past today, so including it would pull the
// typical curve down by however much of it has not happened yet.
const priorMonths = (analytics?.months ?? []).filter(
(m) => m !== selectedMonth && m !== currentMonthStr() && (analytics?.totals[m]?.spent || 0) > 0
const priorMonths = baselineMonths(
analytics?.months ?? [],
selectedMonth,
analytics?.totals ?? {}
);
let cum = 0;
@@ -342,18 +327,16 @@ export default function AnalyticsPage() {
// baseline, because most of it has not happened. And when the month in
// progress is the one selected, its running total has to be measured against
// the same slice of each prior month, not against their full totals.
const otherMonths = months.filter(
(m) => m !== selectedMonth && m !== currentMonthStr() && (analytics.totals[m]?.spent || 0) > 0
);
const otherMonths = baselineMonths(months, selectedMonth, analytics.totals);
const comparableSpend = selectedIsPartial
? spendThrough(dailyData?.daily?.[selectedMonth], compareDay)
: totals.spent;
const avgSpend = otherMonths.length
? otherMonths.reduce(
(s, m) => s + (selectedIsPartial ? spendThrough(dailyData?.daily?.[m], compareDay) : analytics.totals[m]?.spent || 0),
0
) / otherMonths.length
: 0;
const avgSpend = baselineAverage(otherMonths, {
isPartial: selectedIsPartial,
compareDay,
daily: dailyData?.daily,
totals: analytics.totals,
});
const avgDeltaPct = avgSpend > 0 ? Math.round(((comparableSpend - avgSpend) / avgSpend) * 100) : 0;
const throughQualifier = selectedIsPartial ? ` through day ${compareDay}` : "";
const heroSentence =
+36 -2
View File
@@ -3,6 +3,7 @@
import { use } from "react";
import Link from "next/link";
import { useOrderDetail } from "@/lib/hooks";
import { tidyMerchant } from "@/lib/merchant-label";
/**
* One order: what it was, what happened to it, and what paid for it.
@@ -94,6 +95,23 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
{/* A settled rail row must never look like an ordinary order — that is
how a human counts the same purchase twice. */}
{/* Board 211 — same shape as the settled-rail banner above: a real
document with a real figure that is not a purchase. Saying which
payments it restates is the point; the total alone reads as a
single large order. */}
{o.summarises_period && (
<div className="mt-4 bg-zinc-900 border-l-2 border-amber-600 px-4 py-3">
<span className="block font-mono text-[10px] uppercase tracking-widest text-amber-500 mb-1">
Annual summary, not a purchase
</span>
<span className="text-[12.5px] text-zinc-300">
This receipt restates payments already made across the year — the
monthly debits are in the ledger under their own dates. It is kept
for reference and excluded from spend totals.
</span>
</div>
)}
{o.is_settled_duplicate && (
<div className="mt-4 bg-zinc-900 border-l-2 border-indigo-600 px-4 py-3">
<span className="block font-mono text-[10px] uppercase tracking-widest text-indigo-500 mb-1">
@@ -112,7 +130,10 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
{o.ordered_at && <> · {dateFmt.format(new Date(o.ordered_at))}</>}
</div>
<h2 className="font-display text-[27px] leading-tight text-zinc-50 my-3 max-w-[26ch] text-balance">
{o.canonical_name || o.display_name || "Order"}
{/* canonical_name is the ORDER title ("DJI Air 3 Fly More Combo") and
is left alone; only the display_name fallback is a merchant name
and wants tidying. */}
{o.canonical_name || tidyMerchant(o.display_name ?? "") || "Order"}
</h2>
<div className="flex gap-2 flex-wrap mb-3">
{o.content_class && KIND_LABEL[o.content_class] && (
@@ -147,7 +168,11 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
{o.merchant_name && (
<div>
<span className="block text-[10px] uppercase tracking-widest text-zinc-500 mb-0.5">Merchant</span>
{o.merchant_name}
{/* Board 212: the list has tidied this since it shipped and the
detail page did not, so following a row through showed
"Apple" becoming "Apple Pty Ltd." Presentational only —
merges nothing, and Amazon.in stays Amazon.in. */}
{tidyMerchant(o.merchant_name)}
</div>
)}
</div>
@@ -209,6 +234,15 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
<span className="font-mono text-[12px] text-zinc-300">
{t.description}
<small className="block text-[10.5px] text-zinc-500">
{/* Board 205. Four identical "Afterpay" rows are
indistinguishable without their position, and
indistinguishable rows are what made this order
look like four purchases. */}
{t.leg_kind === "instalment" && t.leg_index && t.leg_count
? `Instalment ${t.leg_index} of ${t.leg_count} · `
: t.leg_kind === "shipment"
? "Shipment · "
: ""}
{dateFmt.format(new Date(t.transaction_date))} · txn {t.transaction_id}
</small>
</span>
+13 -26
View File
@@ -3,6 +3,7 @@
import { Suspense, useMemo, useState } from "react";
import Link from "next/link";
import { useOrders } from "@/lib/hooks";
import { tidyMerchant } from "@/lib/merchant-label";
import type { OrderRow } from "@/lib/order-feed";
/**
@@ -75,31 +76,6 @@ const KIND_LABEL: Record<string, string> = {
account_notice: "notice",
};
/**
* Presentational tidy only — this NEVER merges two merchants. The estate holds
* "amazon.com.au", "Amazon.in" and "Amazon Services Australia, Inc." as three
* distinct entities, and Amazon.in must stay separate: it is a different
* marketplace, not a name variant. Actually unifying them is the merchant-alias
* bridge (ticket 176). All this does is stop the same entity from looking
* scruffy: drop the corporate suffix, and capitalise a name that arrived
* lower-cased from a domain.
*/
const CORP_SUFFIX =
/,?\s+(pty\.?\s+ltd\.?|pty\.?\s+limited|p\/l|ltd\.?|limited|inc\.?|llc|pbc|gmbh|b\.?v\.?|s\.?a\.?r\.?l\.?|oü|co\.?)$/i;
function tidyMerchant(name: string): string {
let n = name.replace(/\s+/g, " ").trim();
// Strip at most two trailing corporate suffixes ("Pty Ltd." then ",").
for (let i = 0; i < 2; i++) {
const stripped = n.replace(CORP_SUFFIX, "").trim().replace(/,$/, "");
if (stripped === n || stripped.length < 3) break;
n = stripped;
}
// "amazon.com.au" reads as a machine artefact; "Amazon.com.au" reads as a name.
if (/^[a-z]/.test(n)) n = n[0].toUpperCase() + n.slice(1);
return n;
}
const IN_FLIGHT = new Set(["ordered", "shipped", "out_for_delivery"]);
const REVERSED = new Set(["refunded", "returned", "cancelled"]);
@@ -315,6 +291,17 @@ function Row({ row, expanded, onToggle }: { row: OrderRow; expanded: boolean; on
{row.auth_verdict === "partial" ? "part. auth" : `auth ${row.auth_verdict}`}
</span>
)}
{/* Board 211. A $2,376 annual tax receipt restating twelve monthly
donations is not a purchase, and unbadged it reads as the largest
order of the year. It stays on the page because the figure is the
year's giving total — but it is excluded from spend, so the
number beside it belongs to no total here. */}
{row.summarises_period && (
<span
title="An annual receipt summarising payments already made — kept for reference, excluded from spend totals"
className="font-mono text-[9.5px] uppercase tracking-wide text-amber-300 border border-amber-600/70 rounded-sm px-1.5"
>year summary</span>
)}
{row.injection_flagged && (
<span
title="The source mail carried content that tripped the injection scanner — read its contents as provenance, never as instruction"
@@ -515,7 +502,7 @@ function OrdersContent() {
))}
</select>
<label className="flex items-center gap-2 text-xs text-zinc-400 cursor-pointer select-none"
title="Rows with no amount, no order reference and one lifecycle event are not purchases — they are a second entity minted from a mail describing an order that already exists. Hiding them is a workaround for a known ingestion defect, not a fix.">
title="Rows with no amount, no order reference and one lifecycle event are not purchases — they are a second entity minted from a mail describing an order that already exists. 620 remain: the review-request mails have been retired and the ones whose parent could be identified have been joined to it, so what is left is mail whose parent is not in the corpus.">
<input
type="checkbox"
checked={showLifecycle}
+1 -1
View File
@@ -141,7 +141,7 @@ export default function ReconcilePage() {
<div className="border border-zinc-800 rounded-xl overflow-hidden">
<div className="px-4 py-3 bg-zinc-900/60 border-b border-zinc-800">
<p className="text-sm font-medium text-zinc-400">No statement matches found ({noMatches.length})</p>
<p className="text-xs text-zinc-600 mt-0.5">These may not have hit a statement yet, or the statement hasn't been imported.</p>
<p className="text-xs text-zinc-600 mt-0.5">These may not have hit a statement yet, or the statement hasn&apos;t been imported.</p>
</div>
<div className="divide-y divide-zinc-800">
{noMatches.map((tx) => (
+26 -9
View File
@@ -432,6 +432,19 @@ const SharedTxRow = memo(function SharedTxRow({
// ── Main page ─────────────────────────────────────────────────────────────────
type SortCol = "transaction_date" | "created_at" | "amount";
/**
* Sort direction marker for a column header.
*
* Declared at module scope and given the sort state as props, rather than
* closing over it inside SharedPage: a component defined during render is a
* new type on every render, so React unmounts and remounts it each time
* instead of updating it.
*/
function SortIcon({ col, sortCol, sortDir }: { col: SortCol; sortCol: SortCol; sortDir: "asc" | "desc" }) {
if (sortCol !== col) return <span className="text-zinc-600 ml-0.5">↕</span>;
return <span className="ml-0.5">{sortDir === "desc" ? "↓" : "↑"}</span>;
}
export default function SharedPage() {
const [tagIds, setTagIds] = useState<string[]>([]);
const [participantId, setParticipantId] = useState<number | undefined>(undefined);
@@ -478,10 +491,6 @@ export default function SharedPage() {
else { setSortCol(col); setSortDir("desc"); }
}
function SortIcon({ col }: { col: SortCol }) {
if (sortCol !== col) return <span className="text-zinc-600 ml-0.5">↕</span>;
return <span className="ml-0.5">{sortDir === "desc" ? "↓" : "↑"}</span>;
}
const { data: balances = [], isLoading: balLoading } = useParticipantBalances(realTagIds);
const { data: allTags = [] } = useTags();
const { data: me } = useCurrentUser();
@@ -504,9 +513,17 @@ export default function SharedPage() {
// table was what made expanding a receipt freeze the browser. "Show more"
// extends the window; changing any filter resets it.
const [visibleCount, setVisibleCount] = useState(100);
useEffect(() => {
// Reset during render behind a previous-value guard rather than in an effect.
// Note the key joins `tagIds` by value: as a raw dependency it was a fresh
// array on many renders, so the window was being reset more often than
// "changing any filter" claims — including after "Show more", which is the
// one interaction this must not undo.
const windowKey = `${search}|${sortCol}|${sortDir}|${tagIds.join(",")}|${participantId ?? ""}`;
const [lastWindowKey, setLastWindowKey] = useState(windowKey);
if (windowKey !== lastWindowKey) {
setLastWindowKey(windowKey);
setVisibleCount(100);
}, [search, sortCol, sortDir, tagIds, participantId]);
}
const visible = (transactions as SharedTransactionRow[]).slice(0, visibleCount);
// Receipt expansion, same pattern as the transactions page. The shared
@@ -667,13 +684,13 @@ export default function SharedPage() {
className="text-left px-4 py-2 text-xs text-zinc-500 font-medium cursor-pointer hover:text-white whitespace-nowrap"
onClick={() => toggleSort("transaction_date")}
>
Date <SortIcon col="transaction_date" />
Date <SortIcon col="transaction_date" sortCol={sortCol} sortDir={sortDir} />
</th>
<th
className="text-left px-4 py-2 text-xs text-zinc-500 font-medium cursor-pointer hover:text-white whitespace-nowrap"
onClick={() => toggleSort("created_at")}
>
Imported <SortIcon col="created_at" />
Imported <SortIcon col="created_at" sortCol={sortCol} sortDir={sortDir} />
</th>
<th className="text-left px-4 py-2 text-xs text-zinc-500 font-medium sticky left-0 z-10 bg-zinc-900 border-r border-zinc-800/80">Description</th>
<th className="text-left px-4 py-2 text-xs text-zinc-500 font-medium">Category</th>
@@ -681,7 +698,7 @@ export default function SharedPage() {
className="text-right px-4 py-2 text-xs text-zinc-500 font-medium cursor-pointer hover:text-white"
onClick={() => toggleSort("amount")}
>
Amount <SortIcon col="amount" />
Amount <SortIcon col="amount" sortCol={sortCol} sortDir={sortDir} />
</th>
{/* Paid by sits next to Splits deliberately: together they are the
two halves of the question this page exists to answer — whose
+18 -3
View File
@@ -255,9 +255,24 @@ export default function StatementsPage() {
{participants?.length ? (
<select
value={s.owner_id ?? ""}
onChange={(e) =>
updateStatement.mutate({ id: s.id, owner_id: Number(e.target.value) })
}
onChange={(e) => {
const next = Number(e.target.value);
const name = participants.find((p) => p.id === next)?.name ?? "them";
// This page only lists statements you own, so handing
// one over removes it — and every transaction on it —
// from your view, and only they can hand it back.
if (
!confirm(
`Reassign this statement to ${name}?\n\nEvery transaction on it moves to their ledger, and the statement leaves your list — only ${name} can move it back.`
)
) {
return;
}
updateStatement.mutate(
{ id: s.id, owner_id: next },
{ onError: (err) => alert(err instanceof Error ? err.message : "Failed to reassign") }
);
}}
className="bg-zinc-800 border border-zinc-700 rounded text-xs px-2 py-1 text-zinc-300 cursor-pointer hover:border-zinc-600 focus:outline-none focus:border-indigo-500"
>
{participants.map((p) => (
+1 -1
View File
@@ -59,7 +59,7 @@ function ConvertModal({
<div className="px-6 pt-5 pb-4 border-b border-zinc-800">
<div className="flex items-center gap-2">
<span className="w-3 h-3 rounded-full flex-shrink-0" style={{ backgroundColor: tag.color }} />
<h3 className="font-semibold text-sm text-zinc-300">Convert "{tag.name}" to Trip</h3>
<h3 className="font-semibold text-sm text-zinc-300">Convert &quot;{tag.name}&quot; to Trip</h3>
</div>
<p className="text-xs text-zinc-500 mt-1">
Creates a trip with the same name and color, and assigns all {tag.transaction_count} tagged transaction{tag.transaction_count !== 1 ? "s" : ""} to it.
+48 -10
View File
@@ -148,7 +148,7 @@ function SaveAsRulePrompt({
Automatically apply this {field === "category" ? "category" : "merchant name"} to future matching transactions.
</p>
<div className="bg-zinc-900 rounded-lg px-3 py-2 text-xs text-zinc-300 mb-3 space-y-1">
<p><span className="text-zinc-500">If</span> {conditionField === "merchant_normalized" ? "merchant" : "description"} {conditionOperator} <span className="text-white">"{conditionValue}"</span></p>
<p><span className="text-zinc-500">If</span> {conditionField === "merchant_normalized" ? "merchant" : "description"} {conditionOperator} <span className="text-white">&quot;{conditionValue}&quot;</span></p>
<p>
<span className="text-zinc-500">Then</span>{" "}
{field === "category"
@@ -250,11 +250,17 @@ function MarkAsPaymentModal({
const [participantId, setParticipantId] = useState<number | "">(others[0]?.id ?? "");
useEffect(() => {
if (participantId === "" && others.length > 0) {
setParticipantId(others[0].id);
}
}, [others]);
// Seed the picker once the participant list arrives — the initial state is
// computed before the query resolves, so it starts empty.
//
// Adjusted during render rather than in an effect, which is also strictly
// more correct: `others` is rebuilt by .filter() on every render, so as a
// dependency it never compared equal and the effect ran every render anyway
// — just one commit later, painting an empty select first. The condition
// clears itself on the first non-empty list, so this cannot loop.
if (participantId === "" && others.length > 0) {
setParticipantId(others[0].id);
}
// For credits/refunds the default direction is "they paid me"
const [direction, setDirection] = useState<"received" | "sent">(
SPEND_TYPES.has(transaction.transaction_type) ? "sent" : "received"
@@ -1018,6 +1024,17 @@ function TransactionsContent() {
</div>
{t.notes ? (
<p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.notes}>{t.notes}</p>
) : t.order_leg_kind === "instalment" && t.order_name ? (
// Board 205. Four rows reading "Afterpay $399.75" name
// nothing — this is the surface where that was noticed.
// The order is what tells them apart, and "2 of 4" is
// what stops the same purchase reading as four.
<p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.order_name}>
{t.order_leg_index && t.order_leg_count
? `${t.order_leg_index} of ${t.order_leg_count} · `
: ""}
{t.order_name}
</p>
) : t.order_platform === "uber" && routeSummary(t.order_route) && (
// Five rows all reading "Order - Uber Trip" are
// indistinguishable. Where the trip went is what tells
@@ -1048,11 +1065,32 @@ function TransactionsContent() {
<td className={`p-2 text-right whitespace-nowrap font-mono ${
SPEND_TYPES.has(t.transaction_type) ? "text-red-400" : "text-green-400"
}`}>
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
{t.currency && t.currency !== "AUD" && (
<div className="text-[10px] text-zinc-500 mt-0.5">
{/*
A row with no AUD figure must NOT be printed as AUD.
`amount_aud ?? amount` formatted with the default currency
rendered a USD 10,782.00 Wise credit as "+$10,782.00" in a
column of AUD figures — the real value was about $15,500,
and the row directly above it (the same payment from the
statement) showed exactly that. The native figure is the
only true one here, so it becomes the headline and the
absence of a rate is stated rather than papered over.
*/}
{t.amount_unconverted ? (
<>
{formatAmount(t.amount, t.transaction_type, t.currency)}
</div>
<div className="text-[10px] text-zinc-500 mt-0.5" title="No AUD figure exists for this row — it is excluded from AUD totals">
no AUD rate
</div>
</>
) : (
<>
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
{t.currency && t.currency !== "AUD" && (
<div className="text-[10px] text-zinc-500 mt-0.5">
{formatAmount(t.amount, t.transaction_type, t.currency)}
</div>
)}
</>
)}
</td>
<td className="p-2">
+16 -8
View File
@@ -3,6 +3,11 @@
import { useState } from "react";
import { useCreateTransaction, useParticipants, useTags } from "@/lib/hooks";
import { CATEGORIES, formatCategory } from "@/lib/categories";
import {
PAYMENT_METHODS,
PAYMENT_METHOD_LABELS,
NON_RECONCILING_REASON,
} from "@/lib/payment-methods";
const TRANSACTION_TYPES = ["debit", "credit", "payment", "refund", "fee", "interest", "transfer"];
@@ -35,7 +40,8 @@ export function AddTransactionModal({
const [type, setType] = useState(prefill?.transaction_type ?? "debit");
const [merchant, setMerchant] = useState(prefill?.merchant_normalized ?? "");
const [category, setCategory] = useState(prefill?.category ?? "");
// Cash is excluded from reconciliation — it will never appear on a statement.
// Cash and gift-card credits are excluded from reconciliation — neither will
// ever produce a card line. See needsCardMatch() in queries.ts.
const [paymentMethod, setPaymentMethod] = useState("");
const [selectedTagIds, setSelectedTagIds] = useState<number[]>([]);
const [splits, setSplits] = useState<{ participant_id: number; share_percent: number }[]>(
@@ -163,21 +169,23 @@ export function AddTransactionModal({
</div>
<div>
<label className="block text-xs text-zinc-500 mb-1">Paid by</label>
{/* "Payment method", not "Paid by" — the edit modal uses that label
for the owner, and they are different questions: which instrument
the money left by, versus whose money it was. */}
<label className="block text-xs text-zinc-500 mb-1">Payment method</label>
<select
value={paymentMethod}
onChange={(e) => setPaymentMethod(e.target.value)}
className="w-full bg-zinc-800 border border-zinc-700 rounded px-2 py-1.5 text-sm"
>
<option value="">— unknown —</option>
<option value="card">Card</option>
<option value="cash">Cash</option>
<option value="bank_transfer">Bank transfer</option>
<option value="other">Other</option>
{PAYMENT_METHODS.map((m) => (
<option key={m} value={m}>{PAYMENT_METHOD_LABELS[m]}</option>
))}
</select>
{paymentMethod === "cash" && (
{NON_RECONCILING_REASON[paymentMethod] && (
<p className="text-[11px] text-zinc-500 mt-1">
Cash won&apos;t be offered for reconciliation — it never appears on a statement.
{NON_RECONCILING_REASON[paymentMethod]}
</p>
)}
</div>
+84 -9
View File
@@ -1,11 +1,11 @@
"use client";
import { useState, useRef, useEffect } from "react";
import { useState, useRef } from "react";
import { CATEGORIES, formatCategory } from "@/lib/categories";
import { useImportCSV } from "@/lib/hooks";
import { useImportCSV, useStatementCoverage } from "@/lib/hooks";
import {
parseCSVRows, detectHasHeaders, getColumnLabels, getDataRows, applyMapping,
saveBankPreset, loadBankPresets,
saveBankPreset, loadBankPresets, splitByCoverage, inFileDuplicates,
type DateFormat, type ColumnMapping, type ParsedTransaction, type BankPreset,
} from "@/lib/csv-parser";
@@ -14,6 +14,8 @@ const TX_TYPES = ["debit", "credit", "payment", "refund", "fee", "interest", "tr
type Step = "upload" | "map" | "review" | "done";
function ColSelect({
label, value, onChange, options, required,
}: {
@@ -52,10 +54,23 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
const [savePreset, setSavePreset] = useState(false);
const [editedRows, setEditedRows] = useState<ParsedTransaction[]>([]);
const [error, setError] = useState("");
const [presets, setPresets] = useState<BankPreset[]>([]);
// Read once, at mount, from localStorage. A lazy initializer rather than an
// effect: it runs exactly once per mount — the same semantics as
// `useEffect(…, [])` — but before the first paint, so the preset dropdown
// does not appear empty and then populate. Nothing in this component writes
// `presets` afterwards, so there is no state to keep in sync.
//
// The `typeof window` guard is for safety only: this modal is mounted from
// client state and so is never server-rendered. If that ever changes, it
// degrades to an empty list instead of throwing on `localStorage`.
const [presets] = useState<BankPreset[]>(() =>
typeof window === "undefined" ? [] : loadBankPresets()
);
const [insertedCount, setInsertedCount] = useState(0);
const [coveredRows, setCoveredRows] = useState<ParsedTransaction[]>([]);
const [showCovered, setShowCovered] = useState(false);
const coverage = useStatementCoverage();
useEffect(() => { setPresets(loadBankPresets()); }, []);
function handleFile(file: File) {
setError("");
@@ -111,7 +126,9 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
if (savePreset) {
saveBankPreset({ bankName: bankName.trim(), mapping, dateFormat });
}
setEditedRows(parsed);
const { keep, covered } = splitByCoverage(parsed, coverage.data ?? []);
setEditedRows(keep);
setCoveredRows(covered);
setStep("review");
}
@@ -120,7 +137,13 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
const valid = editedRows.filter((r) => r.date && r.amount > 0 && r.description);
if (!valid.length) { setError("No valid rows to import"); return; }
try {
const result = await importCSV.mutateAsync({ bank_name: bankName, transactions: valid });
const result = await importCSV.mutateAsync({
bank_name: bankName,
// Provenance, so a second import of the same file inserts nothing —
// paired with each row's source_ref where the file carries an id.
source: bankName.trim().toLowerCase().replace(/\s+/g, "-"),
transactions: valid,
});
setInsertedCount(result.inserted);
setStep("done");
} catch (e) {
@@ -286,7 +309,16 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
<div className="grid grid-cols-2 gap-3">
<ColSelect label="Merchant Column (optional)" value={mapping.merchantCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, merchantCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Category Column (optional)" value={mapping.categoryCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, categoryCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Account Column (optional)" value={mapping.accountCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, accountCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Currency Column (optional)" value={mapping.currencyCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, currencyCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Row ID Column (optional)" value={mapping.sourceRefCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, sourceRefCol: v || undefined }))} options={columnLabels} />
</div>
<p className="text-[11px] text-zinc-500 leading-relaxed">
Map <b>Account</b> when the file covers more than one account: rows already
covered by that account&apos;s statements are then left out. Map <b>Row ID</b> to
make re-importing the same file do nothing. Map <b>Currency</b> for a
multi-currency file, or foreign rows are stored as if they were AUD.
</p>
<label className="flex items-center gap-2 text-sm cursor-pointer text-zinc-400">
<input type="checkbox" checked={savePreset} onChange={(e) => setSavePreset(e.target.checked)} className="accent-indigo-500" />
@@ -300,9 +332,52 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
{/* Step 3: Review */}
{step === "review" && (
<div>
<p className="text-xs text-zinc-500 mb-3">
{editedRows.length} transactions parsed. Edit or remove rows before importing.
<p className="text-xs text-zinc-500 mb-2">
{editedRows.length} transactions to import. Edit or remove rows before importing.
</p>
{/*
Reported, never silent. A row that disappears can mean several
things and one of them is worth following up, so the count is
always shown and the rows themselves are one click away.
*/}
{coveredRows.length > 0 && (
<div className="mb-3 rounded border border-zinc-800 bg-zinc-800/30 px-3 py-2">
<button
type="button"
onClick={() => setShowCovered((v) => !v)}
className="text-xs text-zinc-300 hover:text-white text-left w-full"
>
{coveredRows.length} row{coveredRows.length === 1 ? "" : "s"} left out — already covered by
a statement for that account.{" "}
<span className="text-indigo-400">{showCovered ? "hide" : "show"}</span>
</button>
{showCovered && (
<>
<ul className="mt-2 max-h-40 overflow-y-auto text-[11px] text-zinc-500 font-mono space-y-0.5">
{coveredRows.slice(0, 200).map((r, i) => (
<li key={i}>{r.date} {r.amount.toFixed(2).padStart(10)} {r.description.slice(0, 44)}</li>
))}
</ul>
{coveredRows.length > 200 && (
<p className="mt-1 text-[11px] text-zinc-600">…and {coveredRows.length - 200} more.</p>
)}
<p className="mt-2 text-[11px] text-zinc-500">
Statements cover:{" "}
{(coverage.data ?? []).map((c) => `${c.last4} to ${c.coveredTo}`).join(" · ")}
</p>
</>
)}
</div>
)}
{inFileDuplicates(editedRows) > 0 && (
<p className="mb-3 rounded border border-amber-900/50 bg-amber-950/30 px-3 py-2 text-xs text-amber-300">
{inFileDuplicates(editedRows)} row{inFileDuplicates(editedRows) === 1 ? " duplicates another row" : "s duplicate other rows"} in
this same file (same date, amount and description). A re-consent can make an
aggregator re-export history under new ids — check before importing.
</p>
)}
<div className="overflow-x-auto rounded border border-zinc-800">
<table className="w-full text-xs">
<thead className="border-b border-zinc-800">
+126 -5
View File
@@ -8,10 +8,17 @@ import {
useRemoveTransactionTag,
useTransactionSplits,
useTrips,
useParticipants,
useCurrentUser,
} from "@/lib/hooks";
import { SplitModal } from "./split-modal";
import { OrderDetails } from "./order-details";
import { CATEGORIES, formatCategory } from "@/lib/categories";
import {
PAYMENT_METHODS,
PAYMENT_METHOD_LABELS,
NON_RECONCILING_REASON,
} from "@/lib/payment-methods";
import type { TransactionRow, TagRow } from "@/lib/queries";
const TRANSACTION_TYPES = ["debit", "credit", "payment", "refund", "fee", "interest", "transfer"];
@@ -96,6 +103,8 @@ export function EditTransactionModal({
const isManual = !transaction.statement_id;
const updateTxn = useUpdateTransaction();
const { data: trips = [] } = useTrips();
const { data: participants = [] } = useParticipants();
const { data: me } = useCurrentUser();
// Editable override fields
const [merchant, setMerchant] = useState(transaction.merchant_override ?? transaction.merchant_normalized ?? "");
@@ -107,16 +116,29 @@ export function EditTransactionModal({
const [date, setDate] = useState(transaction.transaction_date?.slice(0, 10) ?? "");
const [description, setDescription] = useState(transaction.description);
const [amount, setAmount] = useState(String(transaction.amount));
// Manual-only, because the reconciliation queue is `statement_id IS NULL`:
// this is the only place payment_method changes what happens to a row.
const [paymentMethod, setPaymentMethod] = useState(transaction.payment_method ?? "");
const [tripId, setTripId] = useState<number | null>(transaction.trip_id ?? null);
const [ownerId, setOwnerId] = useState<number | null>(transaction.owner_id ?? null);
// Splits — live via hook so they refresh after SplitModal saves
const { data: liveSplits = [] } = useTransactionSplits(transaction.id);
const [showSplitModal, setShowSplitModal] = useState(false);
const [error, setError] = useState("");
// Set when the API refuses an owner change that would hide the row from me.
// Holding it in state (rather than confirm()) keeps the way out — add a
// split — one click away instead of behind a dialog.
const [releasePrompt, setReleasePrompt] = useState(false);
async function handleSave() {
const ownerChanged = ownerId !== null && ownerId !== (transaction.owner_id ?? null);
const iHoldASplit = liveSplits.some(
(s: { participant_id: number }) => s.participant_id === me?.id
);
async function handleSave(release = false) {
setError("");
try {
const patch: Parameters<typeof updateTxn.mutateAsync>[0] = { id: transaction.id };
@@ -139,14 +161,27 @@ export function EditTransactionModal({
patch.description = description;
if (parseFloat(amount) !== transaction.amount)
patch.amount = parseFloat(amount);
// "" means unknown, which is NULL — not the empty string, which the
// CHECK constraint would reject.
if (paymentMethod !== (transaction.payment_method ?? ""))
patch.payment_method = paymentMethod || null;
}
if (tripId !== (transaction.trip_id ?? null))
patch.trip_id = tripId;
// Owner is manual-only: a statement row's owner comes from its statement.
if (isManual && ownerChanged) {
patch.owner_id = ownerId!;
if (release) patch.release = true;
}
await updateTxn.mutateAsync(patch);
onClose();
} catch (e) {
if ((e as { code?: string })?.code === "would_lose_access") {
setReleasePrompt(true);
}
setError(e instanceof Error ? e.message : "Failed to save");
}
}
@@ -284,6 +319,70 @@ export function EditTransactionModal({
<InlineTags transactionId={transaction.id} initialTags={transaction.tags ?? []} />
</div>
{/* Payment method — manual rows only. A statement row is already on
a statement, so the field decides nothing there; on a manual row
it decides whether reconciliation will ever offer it a match. */}
{isManual && (
<div>
<label className="block text-xs text-zinc-500 mb-1">Payment method</label>
<select
value={paymentMethod}
onChange={(e) => setPaymentMethod(e.target.value)}
className="w-full bg-zinc-800 border border-zinc-700 rounded px-2 py-1.5 text-sm"
>
<option value="">— unknown —</option>
{PAYMENT_METHODS.map((m) => (
<option key={m} value={m}>{PAYMENT_METHOD_LABELS[m]}</option>
))}
</select>
{NON_RECONCILING_REASON[paymentMethod] && (
<p className="text-[11px] text-zinc-500 mt-1">
{NON_RECONCILING_REASON[paymentMethod]}
</p>
)}
</div>
)}
{/* Paid by — sits next to Splits deliberately: together they are
whose money went out and whose share it was. */}
<div>
<label className="block text-xs text-zinc-500 mb-1">Paid by</label>
{isManual ? (
<>
<select
value={ownerId ?? ""}
onChange={(e) => {
setOwnerId(e.target.value ? Number(e.target.value) : null);
setReleasePrompt(false);
setError("");
}}
className="w-full bg-zinc-800 border border-zinc-700 rounded px-2 py-1.5 text-sm"
>
{participants.map((p) => (
<option key={p.id} value={p.id}>{p.name}</option>
))}
</select>
{ownerChanged && (
<p className="text-xs text-amber-400/90 mt-1.5">
{ownerId === me?.id
? "This moves the spend onto your ledger, and any split you hold becomes their share of it."
: iHoldASplit
? "This moves the spend onto their ledger. Your split becomes what you owe them rather than what they owe you."
: "You hold no split on this. Add one first, or it leaves your view for good."}
</p>
)}
</>
) : (
<div className="bg-zinc-800/50 rounded px-2 py-1.5">
<p className="text-sm text-zinc-300">{transaction.owner_name ?? "—"}</p>
<p className="text-xs text-zinc-500 mt-0.5">
From a statement, so the owner is the statement&apos;s. Change it on the
Statements page to move every row on that statement.
</p>
</div>
)}
</div>
{/* Splits */}
<div>
<div className="flex items-center justify-between mb-1.5">
@@ -320,9 +419,31 @@ export function EditTransactionModal({
</div>
{/* Footer */}
<div className="px-6 py-4 border-t border-zinc-800 flex gap-2">
{error && <p className="text-red-400 text-xs flex-1 self-center">{error}</p>}
<div className="flex gap-2 ml-auto">
<div className="px-6 py-4 border-t border-zinc-800 space-y-3">
{error && <p className="text-red-400 text-xs">{error}</p>}
{/* The refusal names the step that was skipped, so offer both: add
the split (keeps the row reachable and gets the balance right)
or hand it over knowingly. */}
{releasePrompt && (
<div className="flex flex-wrap gap-2">
<button
type="button"
onClick={() => setShowSplitModal(true)}
className="px-3 py-1.5 bg-zinc-800 hover:bg-zinc-700 text-zinc-200 rounded-lg text-xs"
>
Add my split first
</button>
<button
type="button"
onClick={() => handleSave(true)}
disabled={updateTxn.isPending}
className="px-3 py-1.5 bg-amber-700 hover:bg-amber-600 disabled:opacity-50 text-white rounded-lg text-xs"
>
Give it away anyway
</button>
</div>
)}
<div className="flex gap-2 justify-end">
<button
type="button"
onClick={onClose}
@@ -332,7 +453,7 @@ export function EditTransactionModal({
</button>
<button
type="button"
onClick={handleSave}
onClick={() => handleSave()}
disabled={updateTxn.isPending}
className="px-4 py-2 bg-indigo-600 hover:bg-indigo-500 disabled:opacity-50 text-white rounded-lg text-sm font-medium"
>
+10 -8
View File
@@ -13,12 +13,9 @@ import {
type OrderReceipt,
type OrderRating,
} from "@/lib/hooks";
const PLATFORM_LABEL: Record<string, string> = {
doordash: "DoorDash",
ubereats: "Uber Eats",
uber: "Uber",
};
// One platform vocabulary, shared with the intake cards — three copies had
// already drifted apart (the intake one was missing `uber`).
import { platformLabel } from "@/lib/intake-rules";
/**
* Who records verdicts. A two-person household with one primary user: the
@@ -72,7 +69,12 @@ export function OrderDetails({
const { data: review } = useOrderReview(transactionId);
const [reviewer, setReviewer] = useState(OWNER_PARTICIPANT_ID);
if (isLoading || !receipt) return null;
// `platform` is the marker of a real receipt. Since phase 2 the endpoint also
// answers with `{ order_link }` alone for a transaction that is merely LINKED
// to an order (a BNPL leg has no receipt of its own), and that object is
// truthy — without this check the panel would render a receipt shell with no
// merchant, no items and no total.
if (isLoading || !receipt || !receipt.platform) return null;
const cur = receipt.currency ?? currency ?? "AUD";
const fmt = (n: number) => (cur === "AUD" ? `$${n.toFixed(2)}` : `${cur} ${n.toFixed(2)}`);
@@ -94,7 +96,7 @@ export function OrderDetails({
<p className="text-xs text-zinc-500">
Order details
{receipt.platform && (
<span className="ml-1.5 text-zinc-400">{PLATFORM_LABEL[receipt.platform] ?? receipt.platform}</span>
<span className="ml-1.5 text-zinc-400">{platformLabel(receipt.platform)}</span>
)}
</p>
{receipt.card_last4 && (
+12 -4
View File
@@ -1,6 +1,6 @@
"use client";
import { useEffect, useMemo, useState } from "react";
import { useMemo, useState } from "react";
import { useRuleMatches, useApplyRuleToSelection, type RuleMatchChange } from "@/lib/hooks";
import { formatCategory } from "@/lib/categories";
@@ -39,9 +39,17 @@ export function RulePreviewModal({
// Default to everything the rule would change — the common case is "yes, all
// of these", with deselection as the exception.
useEffect(() => {
if (data?.transactions) setSelected(new Set(data.transactions.map((t) => t.id)));
}, [data]);
//
// Adjusted during render behind a previous-value guard rather than in an
// effect. In an effect the modal painted once with nothing ticked and then
// again with everything ticked; here the seeded selection is part of the
// first commit. The guard is on `data` identity, so a user deselecting a row
// is not undone on the next render — only a genuinely new preview reseeds.
const [seededFrom, setSeededFrom] = useState<typeof data>(undefined);
if (data !== seededFrom) {
setSeededFrom(data);
setSelected(new Set(data?.transactions?.map((t) => t.id) ?? []));
}
const rows = data?.transactions ?? [];
const allSelected = rows.length > 0 && selected.size === rows.length;
+12 -3
View File
@@ -82,10 +82,19 @@ export function Sidebar() {
const pathname = usePathname();
const [open, setOpen] = useState(false);
// Close drawer on route change
useEffect(() => {
// Close the drawer on route change.
//
// Adjusting state during render behind a previous-value guard — the
// documented React pattern for "reset state when a prop changes" — rather
// than in an effect: React re-runs this component immediately and never
// commits the intermediate DOM, so there is no cascading render. Putting it
// in each Link's onClick is the other candidate fix and is worse here: it
// misses browser back/forward, where the route changes with no click.
const [lastPathname, setLastPathname] = useState(pathname);
if (pathname !== lastPathname) {
setLastPathname(pathname);
setOpen(false);
}, [pathname]);
}
// Prevent body scroll when drawer is open
useEffect(() => {
+27 -17
View File
@@ -1,6 +1,6 @@
"use client";
import { useState, useEffect } from "react";
import { useState } from "react";
import { useParticipants, useSetSplits, useTransactionSplits, useBulkAction, useCreateRule, useCurrentUser } from "@/lib/hooks";
interface Split {
@@ -33,22 +33,32 @@ export function SplitModal({ transactionId, transactionIds, amount, description,
const [saveAsRule, setSaveAsRule] = useState(false);
const [ruleSaved, setRuleSaved] = useState(false);
// Initialise: bulk always defaults to 100% Me; single loads existing splits
useEffect(() => {
if (!participants || participants.length === 0 || !currentUser) return;
if (isBulk) {
setSplitsState([{ participant_id: currentUser.id, share_percent: 100 }]);
} else if (existingSplits && existingSplits.length > 0) {
setSplitsState(
existingSplits.map((s: { participant_id: number; share_percent: number }) => ({
participant_id: s.participant_id,
share_percent: Number(s.share_percent),
}))
);
} else {
setSplitsState([{ participant_id: currentUser.id, share_percent: 100 }]);
}
}, [existingSplits, participants, isBulk, currentUser]);
// Initialise: bulk always defaults to 100% Me; single loads existing splits.
//
// Seeded during render behind a previous-value guard rather than in an
// effect. Both queries resolve after the first paint, so as an effect this
// rendered an empty split table and then replaced it — and, worse, it re-ran
// on every identity change of `participants`/`existingSplits`, which would
// discard edits in progress if either query refetched while the modal was
// open. Keying on the inputs that actually define the seed fixes both.
const seedKey =
!participants || participants.length === 0 || !currentUser
? null
: `${isBulk ? "bulk" : singleId}:${currentUser.id}:${
existingSplits ? JSON.stringify(existingSplits) : "none"
}`;
const [seededFrom, setSeededFrom] = useState<string | null>(null);
if (seedKey !== null && seedKey !== seededFrom) {
setSeededFrom(seedKey);
setSplitsState(
!isBulk && existingSplits && existingSplits.length > 0
? existingSplits.map((s: { participant_id: number; share_percent: number }) => ({
participant_id: s.participant_id,
share_percent: Number(s.share_percent),
}))
: [{ participant_id: currentUser!.id, share_percent: 100 }]
);
}
const total = splits.reduce((sum, s) => sum + s.share_percent, 0);
+61 -11
View File
@@ -129,10 +129,44 @@ export const SPEND_BASE = `CASE WHEN t.interest_amount IS NOT NULL THEN t.intere
/** Effective category, honouring overrides. Never NULL. */
export const EFFECTIVE_CATEGORY = `COALESCE(o.category_override, t.category, 'other')`;
/**
* Rows whose cost is partly mine: ones I paid for, plus ones someone else paid
* for that I hold a split on.
*
* Every spend analytic used to gate on `${OWNER_SCOPE} = $1` alone, with
* `mySplitOf` scaling *within* that gate. Ownership was therefore a
* precondition for an expense being mine — which is backwards for a shared
* household. Half a grocery shop is my expense whether the card was mine or
* Sonu's; who owns the row says whose *account* the money left, a different
* question. It is the same principle the `settled` rule already states: my
* share is my expense regardless of who fronted it or whether it has since
* been repaid.
*
* Measured on 2026-08-15 the gate hid 167 rows / $3,210.91 of my own share
* (Jan–Jul 2026) from monthly, daily, merchants, subscriptions, fees and the
* budget page, while `getParticipantBalances` booked the matching debt
* correctly — so the app could tell you that you owed Sonu for a shop while
* insisting you had not spent anything on it. Worst month was April 2026
* (+$1,354.83), the Europe trips, where she paid for a lot.
*
* That figure is net of `EXCLUDE_NON_BUDGET_TAGS`: a further $456.50 of
* non-owned share in July is `family`-tagged and correctly stays out. Counting
* on transaction_type alone gives $3,667.41 and is the wrong comparison.
*
* Requires `transaction_splits ts` joined on `ts.participant_id =
* <participant>` and ${STATEMENTS_JOIN}. That join cannot multiply rows:
* `(transaction_id, participant_id)` is UNIQUE.
*
* Pair it with `myShare`, which returns 0 for a non-owned row with no split —
* so admitting a row here can never add more than the share actually held.
*/
export const MY_SPEND_SCOPE = (participant = "$1") =>
`(${OWNER_SCOPE} = ${participant} OR ts.participant_id IS NOT NULL)`;
/**
* My percentage share of a transaction, 0-100.
*
* Resolution order:
* On a row I paid for, resolution order:
* 1. An explicit `transaction_splits` row for me.
* 2. The `my_share_percent` override.
* 3. Whatever is left after everyone else's shares.
@@ -142,17 +176,33 @@ export const EFFECTIVE_CATEGORY = `COALESCE(o.category_override, t.category, 'ot
* they owe all of it, and there is no row for me to find. Those rows would
* otherwise land in my spend at full value.
*
* Requires `transaction_splits ts` joined on `ts.participant_id = <participant>`
* and `transaction_overrides o` joined on the transaction.
* On a row SOMEONE ELSE paid for, only an explicit split row counts, and its
* absence means 0. Both of the other two steps are meaningless there and one of
* them is actively wrong:
*
* - `my_share_percent` is a single column on `transaction_overrides` with no
* participant scoping, writable by anyone who can see the row. "My" can only
* coherently mean the owner's. Reading it as the viewer's share on someone
* else's transaction would be a guess, and all 402 rows carrying one today
* are owner-side, so there is nothing to be gained by guessing.
* - `100 - everyone else` is the *payer's* remainder by definition. Applied to
* a row I did not pay for it would hand me the payer's share on top of my
* own.
*
* Requires `transaction_splits ts` joined on `ts.participant_id = <participant>`,
* `transaction_overrides o` joined on the transaction, and ${STATEMENTS_JOIN}.
*/
export const myShare = (participant = "$1") => `COALESCE(
ts.share_percent,
o.my_share_percent,
100 - COALESCE((
SELECT SUM(x.share_percent) FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id <> ${participant}
), 0)
)`;
export const myShare = (participant = "$1") => `CASE
WHEN ${OWNER_SCOPE} = ${participant} THEN COALESCE(
ts.share_percent,
o.my_share_percent,
100 - COALESCE((
SELECT SUM(x.share_percent) FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id <> ${participant}
), 0)
)
ELSE COALESCE(ts.share_percent, 0)
END`;
/** `base` scaled to my share. Use for every per-user spend total. */
export const mySplitOf = (base: string, participant = "$1") =>
+101
View File
@@ -9,6 +9,32 @@ export interface ColumnMapping {
creditCol?: string;
merchantCol?: string;
categoryCol?: string;
/**
* Column holding the account number, when the file covers more than one.
*
* Optional and usually absent — a bank's own export is one account per file.
* An aggregator export (Frollo) is not, and without this every row would be
* measured against the same statement coverage.
*/
accountCol?: string;
/**
* Column holding the ISO currency code, when the file is multi-currency.
*
* Without it a foreign row is stored as if it were AUD — a USD 10,782 salary
* lands as A$10,782, understating it by about a third. Mapping it records the
* native figure with `foreign_currency_code` set and `amount_aud` left NULL,
* which is the same shape order ingestion uses and what `AMOUNT_UNCONVERTED`
* looks for.
*/
currencyCol?: string;
/**
* Column holding the provider's own row id, when it has one.
*
* Stored as `source_ref` so re-importing the same file inserts nothing. The
* importer has no other idempotency: `row_index` is assigned MAX+1 on every
* run, which makes `uq_transaction_identity` structurally unable to fire.
*/
sourceRefCol?: string;
}
export interface BankPreset {
@@ -24,6 +50,13 @@ export interface ParsedTransaction {
transaction_type: string;
merchant_name?: string;
category?: string;
/** Raw account identifier from the file, when a column was mapped. */
account?: string;
/** Provider row id, when a column was mapped. */
source_ref?: string;
/** Set only for a non-AUD row: the native amount and its code. */
foreign_currency_amount?: number;
foreign_currency_code?: string;
}
export function parseCSVRows(text: string): string[][] {
@@ -108,6 +141,9 @@ export function applyMapping(
const descIdx = idx(mapping.descriptionCol);
const merchantIdx = mapping.merchantCol ? idx(mapping.merchantCol) : -1;
const categoryIdx = mapping.categoryCol ? idx(mapping.categoryCol) : -1;
const accountIdx = mapping.accountCol ? idx(mapping.accountCol) : -1;
const sourceRefIdx = mapping.sourceRefCol ? idx(mapping.sourceRefCol) : -1;
const currencyIdx = mapping.currencyCol ? idx(mapping.currencyCol) : -1;
const results: ParsedTransaction[] = [];
for (const row of dataRows) {
@@ -138,6 +174,17 @@ export function applyMapping(
const tx: ParsedTransaction = { date, description, amount, transaction_type };
if (merchantIdx >= 0 && row[merchantIdx]) tx.merchant_name = row[merchantIdx].trim();
if (categoryIdx >= 0 && row[categoryIdx]) tx.category = row[categoryIdx].trim();
if (accountIdx >= 0 && row[accountIdx]) tx.account = row[accountIdx].trim();
if (sourceRefIdx >= 0 && row[sourceRefIdx]) tx.source_ref = row[sourceRefIdx].trim();
if (currencyIdx >= 0) {
const code = (row[currencyIdx] ?? "").trim().toUpperCase();
// AUD needs no marking; anything else is native and unconverted. No rate
// is invented — the statement supplies the AUD figure when it arrives.
if (/^[A-Z]{3}$/.test(code) && code !== "AUD") {
tx.foreign_currency_code = code;
tx.foreign_currency_amount = amount;
}
}
results.push(tx);
}
return results;
@@ -154,3 +201,57 @@ export function loadBankPresets(): BankPreset[] {
try { return JSON.parse(localStorage.getItem("csv-presets") || "[]"); }
catch { return []; }
}
/** Last four digits of an account identifier, however it is written. */
export function last4(s: string): string {
const d = s.replace(/\D/g, "");
return d.length >= 4 ? d.slice(-4) : "";
}
/**
* Splits parsed rows into what to import and what the statements already have.
*
* A statement's end date is a hard watermark: everything on that account up to
* it is already in the ledger, itemised and converted. Without this the first
* aggregator import wrote 422 duplicates out of 550 rows; with it the same
* 2,607-row export offers 171.
*
* Rows are only excluded when an account column was mapped AND that account has
* statements. Anything else is imported — a row is never dropped on a guess.
*/
export function splitByCoverage(
rows: ParsedTransaction[],
coverage: { last4: string; coveredTo: string }[]
): { keep: ParsedTransaction[]; covered: ParsedTransaction[] } {
const wm = new Map(coverage.map((c) => [c.last4, c.coveredTo]));
const keep: ParsedTransaction[] = [];
const covered: ParsedTransaction[] = [];
for (const r of rows) {
const to = r.account ? wm.get(last4(r.account)) : undefined;
// Both are YYYY-MM-DD, so a string compare is a date compare.
if (to && r.date <= to) covered.push(r);
else keep.push(r);
}
return { keep, covered };
}
/**
* Rows that duplicate another row in the SAME file.
*
* Distinct from the coverage check, and not catchable by `source_ref`: a CDR
* re-consent makes the provider re-export an account's whole history under
* fresh ids while the originals survive, so the twins are identical in
* everything except the id. That put 385 duplicates in one 2,563-row export and
* doubled every salary payment. 385 rows is not something you spot by eye in a
* review table, so it is counted here.
*/
export function inFileDuplicates(rows: ParsedTransaction[]): number {
const seen = new Set<string>();
let n = 0;
for (const r of rows) {
const key = [r.date, r.description, r.amount, r.transaction_type, r.account ?? ""].join("|");
if (seen.has(key)) n += 1;
else seen.add(key);
}
return n;
}
+30
View File
@@ -20,3 +20,33 @@ export async function queryRow<T>(sql: string, params: unknown[] = []): Promise<
const rows = await prisma.$queryRawUnsafe<T[]>(sql, ...params);
return rows.length > 0 ? rows[0] : null;
}
/** The same two calls, scoped to one connection inside one transaction. */
export type Queryable = {
queryRaw<T>(sql: string, params?: unknown[]): Promise<T[]>;
queryRow<T>(sql: string, params?: unknown[]): Promise<T | null>;
};
/**
* Run several statements as one transaction, on one connection.
*
* The module-level `queryRaw`/`queryRow` are `$queryRawUnsafe` on a POOLED
* client, so each call is its own implicit transaction on whichever connection
* the pool hands out. That makes a check-then-write pair non-atomic, and it
* makes `pg_advisory_xact_lock` useless outside this helper — the lock is
* released the instant the statement that took it commits, which is before the
* next statement runs. Anything that needs a lock to still be held on its next
* query has to be in here.
*/
export async function withTransaction<T>(fn: (q: Queryable) => Promise<T>): Promise<T> {
return prisma.$transaction(async (tx) => {
const q: Queryable = {
queryRaw: <R>(sql: string, params: unknown[] = []) => tx.$queryRawUnsafe<R[]>(sql, ...params),
queryRow: async <R>(sql: string, params: unknown[] = []) => {
const rows = await tx.$queryRawUnsafe<R[]>(sql, ...params);
return rows.length > 0 ? rows[0] : null;
},
};
return fn(q);
});
}
+41 -2
View File
@@ -148,18 +148,31 @@ export function useUpdateTransaction() {
transaction_date?: string;
trip_id?: number | null;
payment_method?: string | null;
owner_id?: number;
release?: boolean;
}) => {
const res = await fetch(`/api/transactions/${id}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(data),
});
// Without this every rejection resolved as success: the modal closed, the
// list refetched, and the edit silently vanished. The owner guard returns
// 409 with a `code`, so the caller needs both the message and the code.
if (!res.ok) {
const body = await res.json().catch(() => ({}));
throw Object.assign(new Error(body.error || "Failed to update transaction"), {
code: body.code as string | undefined,
});
}
return res.json();
},
onSuccess: () => {
qc.invalidateQueries({ queryKey: ["transactions"] });
qc.invalidateQueries({ queryKey: ["transaction"] });
qc.invalidateQueries({ queryKey: ["analytics"] });
// An owner change flips which side of a balance the splits sit on.
qc.invalidateQueries({ queryKey: ["participant-balances"] });
},
});
}
@@ -538,11 +551,18 @@ export function useUpdateStatement() {
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ owner_id }),
});
return res.json();
if (!res.ok) {
const body = await res.json().catch(() => ({}));
throw new Error(body.error || "Failed to reassign statement");
}
return res.json() as Promise<{ statement_id: number; rows_moved: number }>;
},
onSuccess: () => {
qc.invalidateQueries({ queryKey: ["statements"] });
qc.invalidateQueries({ queryKey: ["transactions"] });
// The rows moved with it, so every per-person total changed.
qc.invalidateQueries({ queryKey: ["analytics"] });
qc.invalidateQueries({ queryKey: ["participant-balances"] });
},
});
}
@@ -1053,9 +1073,11 @@ export function useImportCSV() {
return useMutation({
mutationFn: async (body: {
bank_name: string;
source?: string;
transactions: {
date: string; description: string; amount: number; transaction_type: string;
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string; category?: string;
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string;
category?: string; account?: string; source_ref?: string;
}[];
}) => {
const res = await fetch("/api/import/csv", {
@@ -1257,3 +1279,20 @@ export function useOrderDetail(entityKey: string | null) {
},
});
}
/**
* How far each account's statements already reach, for the CSV import modal.
*
* Fetched once when the modal opens: the review step uses it to drop rows the
* ledger already holds rather than offering them for import.
*/
export function useStatementCoverage() {
return useQuery({
queryKey: ["statement-coverage"],
queryFn: async () => {
const res = await fetch("/api/import/statement-coverage");
if (!res.ok) throw new Error("Could not load statement coverage");
return (await res.json()).coverage as { last4: string; coveredTo: string }[];
},
});
}
+119
View File
@@ -0,0 +1,119 @@
import { loadFiles } from "@/lib/intake-files";
import type { PantryForward } from "@/lib/intake-pantry";
import type { IntakeSource } from "@/lib/intake-rules";
import {
createFromBatch,
loadBatch,
rememberKind,
resolvedCard,
routeFood,
routeGrocery,
type BatchRow,
type RouteContext,
} from "@/lib/intake-router";
import { replaceLineItems, TransactionNotFoundError } from "@/lib/line-items";
import { enrichedCard, errorCard, type Block } from "@/lib/intake-blocks";
import { queryRaw } from "@/lib/db";
/**
* The four intake taps (plan §11 C5), as one function the Slack route calls.
*
* Every tap acts on the persisted batch, never on the card's text. A tap on a
* batch that is already resolved returns its current card rather than acting
* twice — a pressed button that looked dead and got pressed again is how three
* splits were lost on the order nudge, and the same shape applies here.
*/
export const INTAKE_ACTIONS = ["intake_use_candidate", "intake_create", "intake_grocery", "intake_meal"] as const;
export type IntakeActionId = (typeof INTAKE_ACTIONS)[number];
export function isIntakeAction(id: unknown): id is IntakeActionId {
return typeof id === "string" && (INTAKE_ACTIONS as readonly string[]).includes(id);
}
export type IntakeActionValue = { batch_id?: string; transaction_id?: number };
export function parseActionValue(raw: unknown): IntakeActionValue {
if (typeof raw !== "string") return {};
try {
const v = JSON.parse(raw) as IntakeActionValue;
return v && typeof v === "object" ? v : {};
} catch {
return {};
}
}
const note = (text: string): { blocks: Block[]; text: string; replace: boolean } => ({
text,
blocks: [{ type: "section", text: { type: "mrkdwn", text } }],
replace: false,
});
const ctxFor = (batch: BatchRow): RouteContext | null =>
batch.header
? {
batchId: batch.id,
source: batch.source as IntakeSource,
sharedBy: batch.shared_by,
platform: batch.header.platform,
header: batch.header,
items: batch.items ?? [],
currency: batch.header.currency,
}
: null;
export async function handleIntakeAction(
actionId: IntakeActionId,
value: IntakeActionValue,
deps: { pantry: PantryForward }
): Promise<{ blocks: Block[]; text: string; replace: boolean }> {
if (!value.batch_id) return note("That button carried no batch id — I can't tell which upload it was.");
const batch = await loadBatch(value.batch_id);
if (!batch) return note("I can't find that upload any more.");
if (batch.status === "enriched" || batch.status === "created" || batch.status === "forwarded_to_pantry") {
return { ...resolvedCard(batch), replace: true };
}
const ctx = ctxFor(batch);
if (!ctx) return note("That upload never got a readable header — share it again with the top of the screen.");
try {
if (actionId === "intake_use_candidate") {
if (!Number.isInteger(value.transaction_id)) return note("That button carried no transaction id.");
const txnId = value.transaction_id as number;
await replaceLineItems(txnId, ctx.items, { source: "intake", platform: ctx.platform, batchId: batch.id, currency: ctx.currency });
await queryRaw(
`UPDATE intake_batches SET status = 'enriched', kind = COALESCE(kind, 'food'), transaction_id = $2, resolved_at = now() WHERE id = $1::uuid`,
[batch.id, txnId]
);
return {
...enrichedCard({ batchId: batch.id, merchant: ctx.header.merchant, total: ctx.header.total, orderDate: ctx.header.order_date, platform: ctx.platform, itemCount: ctx.items.length, currency: ctx.currency, transactionId: txnId }),
replace: true,
};
}
if (actionId === "intake_create") {
const kind = batch.kind === "grocery" ? "grocery" : "food";
const r = await createFromBatch(batch, kind);
return { blocks: r.slack_blocks ?? [], text: r.text, replace: true };
}
if (actionId === "intake_meal") {
await rememberKind(ctx.header.merchant, "food");
const r = await routeFood(ctx);
return { blocks: r.slack_blocks ?? [], text: r.text, replace: true };
}
// intake_grocery
await rememberKind(ctx.header.merchant, "grocery");
const files = await loadFiles(batch.file_sha256);
if (!files.length) {
return note("I remembered that as groceries, but the original files are gone — share it again and it will go straight to the pantry board.");
}
const r = await routeGrocery(ctx, files, deps.pantry);
return { blocks: r.slack_blocks ?? [], text: r.text, replace: r.status !== "error" };
} catch (e) {
if (e instanceof TransactionNotFoundError) return note("That transaction is no longer in the ledger.");
const detail = e instanceof Error ? e.message : String(e);
return { ...errorCard(detail), replace: false };
}
}
+175
View File
@@ -0,0 +1,175 @@
import { platformLabel } from "@/lib/intake-rules";
/**
* The intake reply as Block Kit (plan §11 C1/C5). Built here, never in n8n
* expressions, so the button payloads the interactive handler round-trips are
* produced by one piece of code and tested like one.
*
* Button `value`s are JSON `{batch_id, ...}` — the batch row is what a later
* tap acts on. action_ids are the C5 contract: `intake_use_candidate`,
* `intake_create`, `intake_grocery`, `intake_meal`.
*/
export type Block = Record<string, unknown>;
export type Candidate = { transaction_id: number; description: string | null; transaction_date: string; amount: string | number; currency?: string | null };
export function financePublicUrl(): string {
return (process.env.FINANCE_PUBLIC_URL || "https://finance.bosecamp.com").replace(/\/$/, "");
}
/** The transactions page searches on `q`; the merchant is the most selective thing a card knows. */
export function transactionLink(merchant: string | null): string {
return `${financePublicUrl()}/transactions?q=${encodeURIComponent(merchant ?? "")}`;
}
/**
* Slack renders `&`, `<` and `>` as markup in mrkdwn (and in the `text`
* fallback). Merchant names, transaction descriptions and model output are
* none of ours — `Ben & Jerry's` must not arrive as `Ben &amp; Jerry's`, and a
* description containing `<http://…|click>` must not become a link the card
* never meant to show. Applied to interpolated values only; our own literal
* `*bold*` and `:emoji:` stay markup.
*/
export function esc(value: string | null | undefined): string {
return (value ?? "").replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
}
/**
* `$` is only right for AUD. An intake batch carries the currency the receipt
* printed, and the order nudge already renders `USD 15.00` rather than
* pretending a foreign order was dollars.
*/
const money = (n: number | string, currency?: string | null) => {
const amount = Number(n).toFixed(2);
// The currency comes off the model (`parseExtraction` accepts any string), so
// an unescaped `<@U123>` here would be a mention on every card.
return !currency || currency === "AUD" ? `$${amount}` : `${esc(currency)} ${amount}`;
};
const section = (text: string): Block => ({ type: "section", text: { type: "mrkdwn", text } });
const button = (action_id: string, text: string, value: unknown, style?: "primary" | "danger"): Block => ({
type: "button",
action_id,
text: { type: "plain_text", text, emoji: true },
value: JSON.stringify(value),
...(style ? { style } : {}),
});
export type CardInput = {
batchId: string;
merchant: string | null;
total: number | null;
orderDate: string | null;
platform: string | null;
itemCount: number;
currency?: string | null;
};
export function enrichedCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
const merchant = esc(c.merchant);
const what = [c.total !== null ? money(c.total, c.currency) : null, esc(platformLabel(c.platform)) || merchant, c.platform && merchant ? `(${merchant})` : null]
.filter(Boolean)
.join(" ");
const text = `Added ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} to your ${what} order ✓`;
return {
text,
blocks: [
{
type: "section",
text: { type: "mrkdwn", text: `:receipt: ${text}` },
accessory: { type: "button", text: { type: "plain_text", text: "View" }, url: transactionLink(c.merchant), action_id: "intake_view" },
},
],
};
}
export function createdCard(c: CardInput & { transactionId: number }): { blocks: Block[]; text: string } {
const text = `Created a ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant) || "new"} transaction with ${c.itemCount} item${c.itemCount === 1 ? "" : "s"} ✓`;
return {
text,
blocks: [
{
type: "section",
text: { type: "mrkdwn", text: `:receipt: ${text}` },
accessory: { type: "button", text: { type: "plain_text", text: "View" }, url: transactionLink(c.merchant), action_id: "intake_view" },
},
],
};
}
export function candidatesCard(c: CardInput, candidates: Candidate[]): { blocks: Block[]; text: string } {
const label = esc(platformLabel(c.platform));
const head = `No matching ${label ? label + " " : ""}order for *${esc(c.merchant) || "this merchant"}* ${c.total !== null ? money(c.total, c.currency) : ""} on ${esc(c.orderDate) || "?"}.`;
const text = `${head.replace(/\*/g, "")} ${candidates.length ? "Pick one below or create it." : "Create it?"}`;
const blocks: Block[] = [section(`:mag: ${head}${candidates.length ? "\nIs it one of these?" : ""}`)];
for (const cand of candidates) {
blocks.push({
type: "section",
text: { type: "mrkdwn", text: `${esc(cand.transaction_date)} · ${esc(cand.description) || "—"} · ${money(cand.amount, cand.currency)}` },
accessory: button("intake_use_candidate", "Use this", { batch_id: c.batchId, transaction_id: cand.transaction_id }),
});
}
blocks.push({
type: "actions",
block_id: "intake_create",
// plain_text disables *formatting*, not entity decoding: `&`, `<` and `>`
// are control characters everywhere in a Slack payload, so the label is
// escaped like any other value we did not write.
elements: [button("intake_create", `Create new ${c.total !== null ? money(c.total, c.currency) + " " : ""}${esc(c.merchant)} transaction`.replace(/\s+/g, " ").trim(), { batch_id: c.batchId }, "primary")],
});
return { blocks, text };
}
export function forwardedCard(c: CardInput & { captureId: number; enriching: boolean }): { blocks: Block[]; text: string } {
const text = `${esc(c.merchant) || "Grocery receipt"} ${c.total !== null ? money(c.total, c.currency) : ""} → pantry board (${c.itemCount} item${c.itemCount === 1 ? "" : "s"}) — finance ${c.enriching ? "updates" : "books it"} on confirm`.replace(/\s+/g, " ");
return { text, blocks: [section(`:shopping_trolley: ${text}`)] };
}
export function needsHeaderCard(): { blocks: Block[]; text: string } {
const text = "I can see items but no order total or date — share the top of the screen too so I can match it.";
return { text, blocks: [section(`:warning: ${text}`)] };
}
export function heldCard(reason: string, headline = "Items don't add up to the total — is a shot missing?"): { blocks: Block[]; text: string } {
const text = `${headline} (${esc(reason)})`;
return { text, blocks: [section(`:warning: ${text}`)] };
}
export function askCard(c: CardInput): { blocks: Block[]; text: string } {
const text = `${esc(c.merchant) || "This upload"} ${c.total !== null ? money(c.total, c.currency) : ""} — groceries or a meal?`.replace(/\s+/g, " ");
return {
text,
blocks: [
section(`:thinking_face: ${text}`),
{
type: "actions",
block_id: "intake_ask",
elements: [
button("intake_grocery", "Groceries", { batch_id: c.batchId }),
button("intake_meal", "A meal", { batch_id: c.batchId }),
],
},
],
};
}
export function duplicateCard(c: { merchant: string | null; total: number | null; transactionId: number | null; captureId: number | null; batchId: string | null; currency?: string | null }): { blocks: Block[]; text: string } {
const where = c.transactionId ? `transaction ${c.transactionId}` : c.captureId ? `pantry capture ${c.captureId}` : c.batchId ? `batch ${esc(c.batchId)}` : "an earlier upload";
// The header (merchant, date, total) IS the identity of a purchase here, so a
// genuine SECOND identical purchase — two of the same coffee, one day, one shop
// — reads as a re-share and is refused. That is the right default (a re-share is
// far commoner than a repeat) but it is a dead end without this sentence: the
// card must say what to do instead, or the ledger quietly loses the second one.
const text = `Already handled — ${esc(c.merchant) || "this order"} ${c.total !== null ? money(c.total, c.currency) : ""} is ${where}. Nothing was changed — if this really is a second, separate purchase, add it in the app.`.replace(/\s+/g, " ");
return { text, blocks: [section(`:repeat: ${text}`)] };
}
export function unsupportedCard(detail?: string): { blocks: Block[]; text: string } {
const text = `I can read receipts, statements and order screens — not this.${detail ? ` (${esc(detail)})` : ""}`;
return { text, blocks: [section(`:no_entry_sign: ${text}`)] };
}
export function errorCard(detail: string): { blocks: Block[]; text: string } {
const text = `Couldn't process that upload: ${esc(detail)}`;
return { text, blocks: [section(`:x: ${text}`)] };
}
+216
View File
@@ -0,0 +1,216 @@
import { extractText, getDocumentProxy } from "unpdf";
import type { ExtractedOrder, ExtractedItem } from "@/lib/intake-rules";
/**
* Extraction for the intake router: one joint model call per batch.
*
* The model EXTRACTS; it does not classify or route. The source tap and the
* merchant pattern decide where the result goes (plan §2 — "a classifier is
* confidently wrong exactly when it matters"). So this prompt asks for the
* header and the lines, verbatim, and nothing else.
*
* One call with every part in order, never one per image: scrolling screenshots
* overlap, and extracting each and concatenating double-counts the row both
* shots contain and strands the lower shot with no header (plan §3, Codex #7).
* Dedupe of the overlap is the model's job; the reconciliation check in
* `intake-rules.ts` is what catches it getting that wrong.
*/
export type InputPart =
| { kind: "image"; mime: string; bytes: Buffer; name: string }
| { kind: "text"; text: string; name: string };
/**
* Digital receipts are text, not pixels. Reading the text beats photographing
* the screen: nothing is misread, and the model parses a page rather than an
* image. Same approach as pantry-app's `pdf-text.ts`.
*/
export async function extractPdfText(bytes: Buffer): Promise<string> {
const document = await getDocumentProxy(new Uint8Array(bytes));
const { text } = await extractText(document, { mergePages: true });
return (Array.isArray(text) ? text.join("\n") : text).trim();
}
export const ORDER_ITEM_PROMPT = `You are reading a purchase receipt: an online food or grocery order (Uber Eats, DoorDash, Coles, Woolworths, ...) or an in-store till receipt. It arrives as one or more screenshots in scroll order, as the text of a PDF, or as pasted text.
Return ONE JSON object with exactly these keys:
{ "merchant", "order_date", "total", "subtotal", "currency", "items": [{ "quantity", "description", "unit_price", "line_total", "options" }], "adjustments": [{ "label", "amount" }] }
Rules:
- merchant: the store or restaurant name exactly as printed, including any branch in parentheses (e.g. "Coles (Wyndham Vale)", "Burgertory (Werribee)"). Never the delivery platform. null if not shown.
- order_date: the order or purchase date printed, as YYYY-MM-DD. null if not shown.
- total: the amount actually charged — the "Total" / amount paid — as a number. null if not shown.
- subtotal: the printed item subtotal before fees and discounts, as a number, or null.
- currency: ISO code if printed (default "AUD").
- items: every purchased line. Keep description VERBATIM as printed — do not rename, canonicalise, translate, shorten or summarise it. A burger called "Michael" stays "Michael"; "Coles RSPCA Approved Chicken Thigh Cutlets approx. 1.1kg" stays exactly that. Put selected options and modifiers (e.g. "Milk Bun", "Chicken") and weighed or substitution notes (e.g. "Final 1.16 kg @ $11.00/kg", "Replaced Coles Beef Slow Cook Chuck Roast approx. 1.3kg") in options, as an array of strings, not in the description.
- quantity: the number ordered (a number; 1 if a line shows none). unit_price: the price per unit as printed, or null. line_total: the money for the whole line as printed, or quantity × unit_price when only a unit price is shown; for weighed lines multiply the final weight by the per-kg price. null when nothing is shown.
- Do not put fees, tips, delivery or service charges, discounts, promotions, savings, tax lines or payment lines in items. List those in adjustments instead, fees positive and discounts negative.
- Multiple screenshots are consecutive scrolls of ONE order and overlap: each physical line appears exactly once in items, never twice.
- Never invent a value. If something is not legible or not present, use null.`;
const NUM_OR_NULL = { anyOf: [{ type: "number" }, { type: "null" }] };
const STR_OR_NULL = { anyOf: [{ type: "string" }, { type: "null" }] };
/** Strict json_schema: every property listed in `required`, nullability via anyOf. */
export const EXTRACT_RESPONSE_FORMAT = {
type: "json_schema",
json_schema: {
name: "order_items",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["merchant", "order_date", "total", "subtotal", "currency", "items", "adjustments"],
properties: {
merchant: STR_OR_NULL,
order_date: STR_OR_NULL,
total: NUM_OR_NULL,
subtotal: NUM_OR_NULL,
currency: STR_OR_NULL,
items: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["quantity", "description", "unit_price", "line_total", "options"],
properties: {
quantity: NUM_OR_NULL,
description: { type: "string" },
unit_price: NUM_OR_NULL,
line_total: NUM_OR_NULL,
options: { type: "array", items: { type: "string" } },
},
},
},
adjustments: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["label", "amount"],
properties: { label: { type: "string" }, amount: { type: "number" } },
},
},
},
},
},
} as const;
/** The OpenAI-format message content: prompt first, then every part in the order shared. */
export function buildMessageContent(parts: InputPart[]): unknown[] {
const content: unknown[] = [{ type: "text", text: ORDER_ITEM_PROMPT }];
parts.forEach((p, i) => {
if (p.kind === "text") {
content.push({ type: "text", text: `\n--- Input ${i + 1} of ${parts.length} (${p.name}) ---\n${p.text}` });
} else {
content.push({ type: "text", text: `\n--- Input ${i + 1} of ${parts.length}: image ${p.name} ---` });
content.push({ type: "image_url", image_url: { url: `data:${p.mime || "image/jpeg"};base64,${p.bytes.toString("base64")}` } });
}
});
return content;
}
const num = (v: unknown): number | null => {
if (typeof v === "number" && Number.isFinite(v)) return v;
if (typeof v === "string" && v.trim()) {
const n = Number(v.replace(/[^0-9.-]/g, ""));
return Number.isFinite(n) ? n : null;
}
return null;
};
/** Tolerant of fences and a stray preamble; strict about the shape it hands on. */
export function parseExtraction(content: string): ExtractedOrder {
const unfenced = content.trim().replace(/^```(?:json)?\s*/i, "").replace(/\s*```$/, "");
const start = unfenced.indexOf("{");
const end = unfenced.lastIndexOf("}");
const candidate = start >= 0 && end > start ? unfenced.slice(start, end + 1) : unfenced;
const raw = JSON.parse(candidate) as Record<string, unknown>;
const items: ExtractedItem[] = Array.isArray(raw.items)
? (raw.items as Record<string, unknown>[])
.filter((i) => i && typeof i.description === "string")
.map((i) => ({
quantity: num(i.quantity),
description: String(i.description),
unit_price: num(i.unit_price),
line_total: num(i.line_total),
options: Array.isArray(i.options) ? i.options.map(String) : [],
}))
: [];
return {
merchant: typeof raw.merchant === "string" ? raw.merchant : null,
order_date: typeof raw.order_date === "string" ? raw.order_date : null,
total: num(raw.total),
subtotal: num(raw.subtotal),
currency: typeof raw.currency === "string" && raw.currency ? raw.currency : "AUD",
items,
adjustments: Array.isArray(raw.adjustments)
? (raw.adjustments as Record<string, unknown>[])
.filter((a) => a && typeof a.label === "string" && num(a.amount) !== null)
.map((a) => ({ label: String(a.label), amount: num(a.amount) as number }))
: [],
};
}
export type Extractor = (parts: InputPart[]) => Promise<ExtractedOrder>;
export class ExtractionError extends Error {}
/**
* LiteLLM, OpenAI-compatible, alias `cloud-budget` — the same alias pantry's
* capture uses, so a model swap is one config line in `litellm-config.yaml`.
* `temperature: 0`; `response_format` is the strict schema above.
*/
export function liteLLMExtractor(opts: {
baseUrl?: string;
key?: string;
model?: string;
fetchImpl?: typeof fetch;
} = {}): Extractor {
return async (parts) => {
const baseUrl = (opts.baseUrl ?? process.env.LITELLM_BASE_URL ?? "").replace(/\/$/, "");
const key = opts.key ?? process.env.LITELLM_KEY;
if (!baseUrl || !key) throw new ExtractionError("LITELLM_BASE_URL / LITELLM_KEY are not configured");
const doFetch = opts.fetchImpl ?? fetch;
const response = await doFetch(`${baseUrl}/v1/chat/completions`, {
method: "POST",
headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
body: JSON.stringify({
model: opts.model ?? "cloud-budget",
temperature: 0,
max_tokens: 16384,
response_format: EXTRACT_RESPONSE_FORMAT,
messages: [{ role: "user", content: buildMessageContent(parts) }],
}),
});
if (!response.ok) throw new ExtractionError(`LiteLLM returned ${response.status}`);
const raw = (await response.json()) as { choices?: { message?: { content?: string }; finish_reason?: string }[] };
// The rule is: a finish_reason that is PRESENT must be `stop`. An absent
// one is accepted, deliberately — providers differ on whether they send the
// field, and failing every call on a provider that omits it would be a
// certain outage in exchange for a case that also requires the model to
// emit syntactically complete JSON. The harm a truncated reply does — a
// partial item list replacing a good one — is closed on the write side
// instead, by the guard in `routeFood`/`createFromBatch`.
//
// `length` is the common one: a truncated reply still parses once the JSON
// happens to close, quietly dropping the tail of a long receipt. Any other
// reason (`content_filter`, `tool_calls`) is a different reply shape that
// the strict schema never constrained.
const finish = raw.choices?.[0]?.finish_reason;
if (finish && finish !== "stop") {
throw new ExtractionError(
finish === "length"
? "The receipt was too long to read in one pass."
: `The model stopped before it finished (${finish}).`
);
}
const content = raw.choices?.[0]?.message?.content;
if (!content) throw new ExtractionError("LiteLLM returned no content");
try {
return parseExtraction(content);
} catch (e) {
throw new ExtractionError(`could not parse extraction: ${e instanceof Error ? e.message : String(e)}`);
}
};
}
+51
View File
@@ -0,0 +1,51 @@
import { createHash } from "node:crypto";
import { mkdir, readFile, writeFile } from "node:fs/promises";
import path from "node:path";
/**
* The attachment store: the original files of a batch, keyed by sha256.
*
* Two reasons it exists. A "Groceries or a meal?" answer arrives minutes after
* the upload, in a fresh request carrying only the batch id, and "groceries"
* means forwarding the ORIGINAL files to pantry — so they have to still be
* here. And plan §8: "the source file is retrievable from the attachment store
* by its sha256".
*
* Plain files under `INTAKE_FILE_DIR` (default `/app/data/intake`); a sidecar
* `.json` keeps the name and MIME type. No database column holds bytes.
*/
export type StoredFile = { name: string; type: string; bytes: Buffer; sha256: string };
export function intakeFileDir(): string {
return process.env.INTAKE_FILE_DIR || "/app/data/intake";
}
export function sha256Of(bytes: Buffer): string {
return createHash("sha256").update(bytes).digest("hex");
}
export async function storeFiles(files: StoredFile[]): Promise<void> {
const dir = intakeFileDir();
await mkdir(dir, { recursive: true });
for (const f of files) {
await writeFile(path.join(dir, `${f.sha256}.bin`), f.bytes);
await writeFile(path.join(dir, `${f.sha256}.json`), JSON.stringify({ name: f.name, type: f.type }));
}
}
/** The files that are still on disk, in the order asked for; missing ones are skipped. */
export async function loadFiles(shas: string[]): Promise<StoredFile[]> {
const dir = intakeFileDir();
const out: StoredFile[] = [];
for (const sha of shas) {
try {
const bytes = await readFile(path.join(dir, `${sha}.bin`));
const meta = JSON.parse(await readFile(path.join(dir, `${sha}.json`), "utf-8")) as { name?: string; type?: string };
out.push({ name: meta.name ?? `${sha}.bin`, type: meta.type ?? "application/octet-stream", bytes, sha256: sha });
} catch {
/* gone — the caller decides what that means */
}
}
return out;
}
+76
View File
@@ -0,0 +1,76 @@
import type { StoredFile } from "@/lib/intake-files";
/**
* finance → pantry hand-off for a grocery upload (plan §11 C3).
*
* The ORIGINAL files go across, not the extracted items: pantry owns grocery
* extraction (canonical names, categories, staples) and its capture board is
* the review surface. What finance adds is the routing decision — which
* transaction the confirm should enrich, or that it should create one — carried
* in the `intake` field so pantry's confirm seam (C4) can suppress the ING-10
* create that would otherwise double-book an order the email already booked.
*/
export type PantryForwardArgs = {
files: StoredFile[];
batchId: string;
platform: string | null;
sharedBy: string | null;
financeTargetTransactionId: number | null;
financeAction: "enrich" | "create";
};
export type PantryForwardResult = { captureId: number; itemCount: number | null };
export class PantryDuplicateError extends Error {
constructor(
public readonly captureId: number | null,
public readonly reason: string
) {
super(reason);
}
}
export class PantryUnavailableError extends Error {}
export type PantryForward = (args: PantryForwardArgs) => Promise<PantryForwardResult>;
export function pantryForwarder(opts: { baseUrl?: string; token?: string; fetchImpl?: typeof fetch } = {}): PantryForward {
return async (args) => {
const baseUrl = (opts.baseUrl ?? process.env.PANTRY_BASE_URL ?? "").replace(/\/$/, "");
const token = opts.token ?? process.env.RECEIPT_INGEST_TOKEN;
if (!baseUrl || !token) throw new PantryUnavailableError("PANTRY_BASE_URL / RECEIPT_INGEST_TOKEN are not configured");
const doFetch = opts.fetchImpl ?? fetch;
const form = new FormData();
for (const f of args.files) {
form.append("image", new Blob([new Uint8Array(f.bytes)], { type: f.type }), f.name);
}
form.append("source", "receipt");
form.append(
"intake",
JSON.stringify({
batch_id: args.batchId,
platform: args.platform,
origin: "intake",
shared_by: args.sharedBy,
finance_target_transaction_id: args.financeTargetTransactionId,
finance_action: args.financeAction,
})
);
let response: Response;
try {
response = await doFetch(`${baseUrl}/api/capture`, { method: "POST", headers: { "x-ingest-token": token }, body: form });
} catch (e) {
throw new PantryUnavailableError(`pantry unreachable: ${e instanceof Error ? e.message : String(e)}`);
}
const body = (await response.json().catch(() => ({}))) as { captureId?: number; error?: string; items?: unknown[]; reason?: string };
if (response.status === 409) {
throw new PantryDuplicateError(typeof body.captureId === "number" ? body.captureId : null, body.reason ?? body.error ?? "already captured");
}
if (!response.ok) throw new PantryUnavailableError(`pantry returned ${response.status}: ${body.error ?? ""}`.trim());
if (typeof body.captureId !== "number") throw new PantryUnavailableError("pantry returned no captureId");
return { captureId: body.captureId, itemCount: Array.isArray(body.items) ? body.items.length : null };
};
}
+647
View File
@@ -0,0 +1,647 @@
import { randomUUID } from "node:crypto";
import { queryRaw, queryRow, withTransaction } from "@/lib/db";
import { EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY } from "@/lib/analytics-sql";
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
import {
classifyInput,
merchantKey,
merchantMatches,
platformForSource,
reconcileItems,
resolveMerchantKind,
toLineItems,
validateHeader,
type ExtractedOrder,
type IntakeSource,
type MerchantKind,
type OrderHeader,
type StoredLineItem,
} from "@/lib/intake-rules";
import { extractPdfText, ExtractionError, type Extractor, type InputPart } from "@/lib/intake-extract";
import { storeFiles, type StoredFile } from "@/lib/intake-files";
import { PantryDuplicateError, PantryUnavailableError, type PantryForward } from "@/lib/intake-pantry";
import { replaceLineItems } from "@/lib/line-items";
import {
askCard,
candidatesCard,
createdCard,
duplicateCard,
enrichedCard,
errorCard,
forwardedCard,
heldCard,
needsHeaderCard,
unsupportedCard,
type Block,
type Candidate,
type CardInput,
} from "@/lib/intake-blocks";
/**
* The intake router (plan §11 C1): one upload, every applicable step.
*
* Order of operations matters and is the plan's:
* persist the batch → classify each file → ONE extraction call → header →
* merchant kind → duplicate check → reconciliation → match → act.
*
* The batch row is written before anything else so that a Slack tap minutes
* later has something to act on, and so a re-share is recognisable. Nothing
* else is written until the header is whole: a half-read upload produces a
* reply, not a row (plan §2).
*/
export const MAX_FILES = 6;
export type IntakeKind = "grocery" | "food" | "ask" | "held" | "unsupported";
export type IntakeStatus =
| "received"
| "enriched"
| "created"
| "candidates"
| "forwarded_to_pantry"
| "needs_header"
| "duplicate"
| "unsupported"
| "ask"
| "held"
| "error";
export type IntakeResult = {
kind: IntakeKind;
status: IntakeStatus;
batch_id: string;
transaction_id?: number;
capture_id?: number;
duplicate_of?: { batch_id: string; transaction_id: number | null; capture_id: number | null };
candidates?: Candidate[];
header?: OrderHeader | null;
items?: StoredLineItem[];
slack_blocks: Block[] | null;
text: string;
http_status: number;
};
export type IntakeDeps = { extractor: Extractor; pantry: PantryForward };
export type BatchRow = {
id: string;
source: IntakeSource;
shared_by: string | null;
file_sha256: string[];
header: (OrderHeader & { platform: string | null; subtotal?: number | null; currency?: string | null }) | null;
items: StoredLineItem[] | null;
kind: string | null;
status: IntakeStatus;
transaction_id: number | null;
capture_id: number | null;
};
// ─── batch persistence ──────────────────────────────────────────────────────
export async function loadBatch(id: string): Promise<BatchRow | null> {
return queryRow<BatchRow>(
`SELECT id::text, source, shared_by, file_sha256, header, items, kind, status, transaction_id, capture_id
FROM intake_batches WHERE id = $1::uuid`,
[id]
);
}
async function insertBatch(b: { id: string; source: IntakeSource; sharedBy: string | null; shas: string[] }) {
await queryRaw(
`INSERT INTO intake_batches (id, source, shared_by, file_sha256, status)
VALUES ($1::uuid, $2, $3, $4::text[], 'received')`,
[b.id, b.source, b.sharedBy, b.shas]
);
}
async function updateBatch(
id: string,
patch: {
header?: unknown;
items?: unknown;
kind?: string | null;
status: IntakeStatus;
transaction_id?: number | null;
capture_id?: number | null;
}
) {
await queryRaw(
`UPDATE intake_batches
SET header = COALESCE($2::jsonb, header),
items = COALESCE($3::jsonb, items),
kind = COALESCE($4, kind),
status = $5,
transaction_id = COALESCE($6, transaction_id),
capture_id = COALESCE($7, capture_id),
resolved_at = CASE WHEN $5 IN ('enriched','created','forwarded_to_pantry','duplicate') THEN now() ELSE resolved_at END
WHERE id = $1::uuid`,
[
id,
patch.header === undefined ? null : JSON.stringify(patch.header),
patch.items === undefined ? null : JSON.stringify(patch.items),
patch.kind ?? null,
patch.status,
patch.transaction_id ?? null,
patch.capture_id ?? null,
]
);
}
// ─── lookups ────────────────────────────────────────────────────────────────
export async function rememberedKind(merchant: string | null): Promise<"grocery" | "food" | null> {
const key = merchantKey(merchant);
if (!key) return null;
const row = await queryRow<{ kind: "grocery" | "food" }>(`SELECT kind FROM intake_merchant_kinds WHERE merchant_key = $1`, [key]);
return row?.kind ?? null;
}
export async function rememberKind(merchant: string | null, kind: "grocery" | "food"): Promise<void> {
const key = merchantKey(merchant);
if (!key) return;
await queryRaw(
`INSERT INTO intake_merchant_kinds (merchant_key, kind) VALUES ($1, $2)
ON CONFLICT (merchant_key) DO UPDATE SET kind = EXCLUDED.kind`,
[key, kind]
);
}
/** The owner a batch's rows belong to: the sharer's participant, else the default. */
export async function ownerForEmail(email: string | null): Promise<number> {
if (!email) return DEFAULT_OWNER_ID;
const row = await queryRow<{ id: number }>(`SELECT id FROM participants WHERE lower(email) = lower($1) LIMIT 1`, [email]);
return row?.id ?? DEFAULT_OWNER_ID;
}
/**
* A batch already resolved from the same files, or from the same
* (platform, merchant, date, total) header (plan §5, Codex #5). Only RESOLVED
* batches count: a `needs_header` share followed by a fuller one must go
* through, and a pending candidates card re-shared is harmless.
*/
export async function findDuplicateBatch(
exceptId: string,
shas: string[],
header: (OrderHeader & { platform: string | null }) | null
): Promise<BatchRow | null> {
const bySha = await queryRow<BatchRow>(
`SELECT id::text, source, shared_by, file_sha256, header, items, kind, status, transaction_id, capture_id
FROM intake_batches
WHERE id <> $1::uuid AND status IN ('enriched','created','forwarded_to_pantry')
AND file_sha256 && $2::text[]
ORDER BY created_at DESC LIMIT 1`,
[exceptId, shas]
);
if (bySha) return bySha;
if (!header || !header.merchant) return null;
const rows = await queryRaw<BatchRow>(
`SELECT id::text, source, shared_by, file_sha256, header, items, kind, status, transaction_id, capture_id
FROM intake_batches
WHERE id <> $1::uuid AND status IN ('enriched','created','forwarded_to_pantry')
AND header->>'order_date' = $2
AND (header->>'total')::numeric = $3::numeric
AND COALESCE(header->>'platform', '') = COALESCE($4, '')
ORDER BY created_at DESC`,
[exceptId, header.order_date, header.total, header.platform]
);
return rows.find((r) => merchantMatches(r.header?.merchant, header.merchant)) ?? null;
}
type MatchRow = {
transaction_id: number;
description: string | null;
transaction_date: string;
amount: string;
merchant: string | null;
platform: string | null;
/** What `amount` is denominated in — a candidate row is not always dollars. */
currency: string;
/** The `expense_metadata` row the match resolved through — null when there is none. */
metadata_id: number | null;
/** Does that row already carry a receipt? */
has_line_items: boolean;
};
/**
* The match rule (plan §5, D2): `(platform, merchant, date, total)`, amount and
* date EXACT, never widened silently. The merchant compares normalised, with
* whole-word containment ("Coles" vs "Coles (Wyndham Vale)") — that is
* spelling, not widening. A unique hit is a match; two hits are an ambiguity
* and go to the candidates card like a miss.
*
* A NULL platform means "no platform", not "any platform". An in-store Coles
* receipt carries none, and `$3 IS NULL OR em.platform = $3` let it match an
* Uber Eats Coles order on the same day for the same money — and then overwrite
* that order's items with the till receipt's. The in-store side matches rows
* with no platform of their own; the delivery side still requires its own.
*/
export async function matchTransaction(header: OrderHeader, platform: string | null): Promise<{ match: MatchRow | null; ambiguous: boolean }> {
if (!header.merchant) return { match: null, ambiguous: false };
const rows = await queryRaw<MatchRow>(
`SELECT DISTINCT ON (t.id)
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency,
COALESCE(jsonb_typeof(em.line_items) = 'array' AND jsonb_array_length(em.line_items) > 0, false) AS has_line_items
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
WHERE ${EXCLUDE_RECONCILED_SOURCE}
AND t.transaction_date = $1::date
AND t.amount = $2::numeric
AND (($3::text IS NULL AND em.platform IS NULL) OR em.platform = $3)
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
[header.order_date, header.total, platform]
);
const hits = rows.filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant));
if (hits.length === 1) return { match: hits[0], ambiguous: false };
return { match: null, ambiguous: hits.length > 1 };
}
/**
* Nearest candidates for the card: same merchant, ±3 days, any amount.
*
* Deliberately NOT filtered by platform. The card is a human choice, not a
* decision — the whole reason it is shown is that the exact key missed, so the
* platform is one of the things that may be wrong. A filter here narrows the
* list at exactly the moment it needs to widen, and the row the user was
* looking for is simply absent with nothing saying why.
*/
export async function findCandidates(header: OrderHeader): Promise<Candidate[]> {
if (!header.merchant) return [];
const rows = await queryRaw<MatchRow>(
`SELECT DISTINCT ON (t.id)
t.id AS transaction_id, t.description, t.transaction_date::text, t.amount::text,
COALESCE(em.merchant_normalized, t.merchant_normalized, t.merchant_name, t.description) AS merchant,
em.platform, em.id AS metadata_id, ${NATIVE_CURRENCY} AS currency, false AS has_line_items
FROM transactions t
LEFT JOIN statements s ON s.id = t.statement_id
LEFT JOIN expense_metadata em ON em.transaction_id = t.id OR em.matched_transaction_id = t.id
WHERE ${EXCLUDE_RECONCILED_SOURCE}
AND t.transaction_date BETWEEN $1::date - 3 AND $1::date + 3
ORDER BY t.id, (em.transaction_id = t.id) DESC NULLS LAST`,
[header.order_date]
);
return rows
.filter((r) => merchantMatches(r.merchant, header.merchant) || merchantMatches(r.description, header.merchant))
.sort((a, b) => {
const da = Math.abs(Date.parse(a.transaction_date) - Date.parse(header.order_date));
const db = Math.abs(Date.parse(b.transaction_date) - Date.parse(header.order_date));
if (da !== db) return da - db;
return Math.abs(Number(a.amount) - header.total) - Math.abs(Number(b.amount) - header.total);
})
.slice(0, 5)
.map((r) => ({ transaction_id: r.transaction_id, description: r.description, transaction_date: r.transaction_date, amount: r.amount, currency: r.currency }));
}
// ─── the two destinations ───────────────────────────────────────────────────
export type RouteContext = {
batchId: string;
source: IntakeSource;
sharedBy: string | null;
platform: string | null;
header: OrderHeader;
items: StoredLineItem[];
currency?: string | null;
};
const cardInput = (ctx: RouteContext): CardInput => ({
batchId: ctx.batchId,
merchant: ctx.header.merchant,
total: ctx.header.total,
orderDate: ctx.header.order_date,
platform: ctx.platform,
itemCount: ctx.items.length,
currency: ctx.currency,
});
const finish = (ctx: RouteContext, kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, extra: Partial<IntakeResult> = {}): IntakeResult => ({
kind,
status,
batch_id: ctx.batchId,
header: ctx.header,
items: ctx.items,
slack_blocks: card.blocks,
text: card.text,
http_status: 200,
...extra,
});
/**
* Are these items unverifiable — no line carrying any money at all?
*
* That is `reconcileItems`' `basis: "none"`: pasted text with no prices, or an
* extraction that read the names and none of the figures. The reconciliation
* check passes it, correctly, because there is nothing to check. What must not
* follow is replacing a receipt that IS itemised with one nothing verified.
*/
const unverifiable = (items: StoredLineItem[]) => items.every((i) => i.amount === null);
/**
* The hold, shared by EVERY path that enriches a matched transaction.
*
* It lived only in `routeFood` and `createFromBatch`'s re-match walked straight
* past it — the same upload destroyed the same receipt depending on which
* button reached it. Returns null when there is nothing to protect.
*/
async function holdRatherThanReplace(ctx: RouteContext, match: MatchRow): Promise<IntakeResult | null> {
if (!unverifiable(ctx.items) || !match.has_line_items) return null;
await updateBatch(ctx.batchId, { status: "held", items: ctx.items });
const card = heldCard(
"this upload carries no prices, so nothing checked it against the total — and that order already has its items",
"I left the existing items alone"
);
return finish(ctx, "held", "held", card, { transaction_id: match.transaction_id });
}
/** Food: enrich the matched order's line_items, or show candidates. NEVER auto-creates. */
export async function routeFood(ctx: RouteContext): Promise<IntakeResult> {
const { match } = await matchTransaction(ctx.header, ctx.platform);
if (match) {
const held = await holdRatherThanReplace(ctx, match);
if (held) return held;
await replaceLineItems(match.transaction_id, ctx.items, { source: "intake", platform: ctx.platform, batchId: ctx.batchId, metadataId: match.metadata_id, currency: ctx.currency });
await updateBatch(ctx.batchId, { kind: "food", status: "enriched", transaction_id: match.transaction_id, items: ctx.items });
return finish(ctx, "food", "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
}
const candidates = await findCandidates(ctx.header);
await updateBatch(ctx.batchId, { kind: "food", status: "candidates", items: ctx.items });
return finish(ctx, "food", "candidates", candidatesCard(cardInput(ctx), candidates), { candidates });
}
/** Grocery: the original files go to pantry's board; finance rides along on its confirm (C3/C4). */
export async function routeGrocery(ctx: RouteContext, files: StoredFile[], pantry: PantryForward): Promise<IntakeResult> {
const { match } = await matchTransaction(ctx.header, ctx.platform);
try {
const r = await pantry({
files,
batchId: ctx.batchId,
platform: ctx.platform,
sharedBy: ctx.sharedBy,
financeTargetTransactionId: match?.transaction_id ?? null,
financeAction: match ? "enrich" : "create",
});
await updateBatch(ctx.batchId, { kind: "grocery", status: "forwarded_to_pantry", transaction_id: match?.transaction_id ?? null, capture_id: r.captureId, items: ctx.items });
const card = forwardedCard({ ...cardInput(ctx), itemCount: r.itemCount ?? ctx.items.length, captureId: r.captureId, enriching: !!match });
return finish(ctx, "grocery", "forwarded_to_pantry", card, { capture_id: r.captureId, transaction_id: match?.transaction_id });
} catch (e) {
if (e instanceof PantryDuplicateError) {
await updateBatch(ctx.batchId, { kind: "grocery", status: "duplicate", capture_id: e.captureId, items: ctx.items });
return finish(ctx, "grocery", "duplicate", duplicateCard({ merchant: ctx.header.merchant, total: ctx.header.total, currency: ctx.currency, transactionId: match?.transaction_id ?? null, captureId: e.captureId, batchId: null }), {
capture_id: e.captureId ?? undefined,
});
}
const detail = e instanceof PantryUnavailableError ? e.message : e instanceof Error ? e.message : String(e);
await updateBatch(ctx.batchId, { kind: "grocery", status: "error", items: ctx.items });
return { ...finish(ctx, "grocery", "error", errorCard(detail)), http_status: 502 };
}
}
/** Create the transaction ING-10-style: manual, `statement_id NULL`, owner set — only ever on an explicit tap. */
export async function createFromBatch(batch: BatchRow, kind: "grocery" | "food"): Promise<IntakeResult> {
if (!batch.header) throw new Error("batch has no header");
const header = batch.header;
const items = batch.items ?? [];
const ctx: RouteContext = {
batchId: batch.id,
source: batch.source,
sharedBy: batch.shared_by,
platform: header.platform,
header,
items,
currency: header.currency,
};
// The card was built minutes ago and the tap arrives now. In between, the
// email/order bridge books orders on its own schedule — so re-run the match
// rather than trusting the "no match" the card was drawn from. Creating a
// second row for an order the ledger already holds is the one outcome this
// whole slice exists to avoid.
const { match } = await matchTransaction(header, header.platform);
if (match) {
const held = await holdRatherThanReplace(ctx, match);
if (held) return held;
await replaceLineItems(match.transaction_id, items, { source: "intake", platform: header.platform, batchId: batch.id, metadataId: match.metadata_id, currency: header.currency });
await updateBatch(batch.id, { kind, status: "enriched", transaction_id: match.transaction_id, items });
return finish(ctx, kind, "enriched", enrichedCard({ ...cardInput(ctx), transactionId: match.transaction_id }), { transaction_id: match.transaction_id });
}
// A sibling batch of the same order — the PDF after the screenshot — may have
// been resolved since. `findDuplicateBatch` deliberately ignores unresolved
// batches, so two pending cards for one order both reach here; without this
// the second tap books the order twice.
const twin = await findDuplicateBatch(batch.id, batch.file_sha256, header);
if (twin) {
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: twin.transaction_id, capture_id: twin.capture_id });
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: twin.transaction_id, captureId: twin.capture_id, batchId: twin.id });
return finish(ctx, kind, "duplicate", card, {
transaction_id: twin.transaction_id ?? undefined,
capture_id: twin.capture_id ?? undefined,
duplicate_of: { batch_id: twin.id, transaction_id: twin.transaction_id, capture_id: twin.capture_id },
});
}
const ownerId = await ownerForEmail(batch.shared_by);
const merchant = header.merchant ?? "Unknown merchant";
// Not `orderDescription()`: it always prefixes "Order - ", which is right for
// a delivery receipt and wrong for the in-store till receipt this path also
// handles. Not `resolveCategory()` either — it re-derives grocery-vs-food
// from a merchant pattern, and `kind` is the answer the router already
// reached, remembered merchant and Slack tap included.
const description = header.platform ? `Order - ${merchant}` : merchant;
const category = kind === "grocery" ? "groceries" : "dining";
// `amount` is the figure printed on the receipt, in its own currency. No FX
// rate exists at intake, so a foreign order leaves `amount_aud` NULL rather
// than asserting a conversion — the same shape order-ingestion.ts writes.
const currency = header.currency ?? "AUD";
const isAud = currency === "AUD";
// Check and insert under one lock, on one connection.
//
// `source_ref = batch.id` makes a second press of THIS card a no-op, and
// ON CONFLICT makes that safe against a double tap. It cannot help two
// DIFFERENT batches for one purchase — their keys differ — and both taps run
// their match and duplicate checks before either inserts, so both can find
// nothing. The lock is keyed on the purchase, not the batch, which is the
// only thing the two share. It has to be transaction-scoped and therefore
// inside withTransaction: taken by a statement of its own it would be
// released before the next one ran (see db.ts).
const purchaseKey = [header.platform ?? "", merchantKey(merchant), header.order_date, header.total.toFixed(2)].join("|");
const outcome = await withTransaction(async (q) => {
await q.queryRaw(`SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, [purchaseKey]);
// Mine (a double tap) or another batch's (the race). The ORDER BY is what
// tells them apart when both exist.
const existing = await q.queryRow<{ id: number; source_ref: string }>(
`SELECT id, source_ref FROM transactions
WHERE source = 'intake' AND superseded_by_id IS NULL
AND (source_ref = $1
OR (transaction_date = $2::date AND amount = $3::numeric
AND lower(merchant_normalized) = lower($4)))
ORDER BY (source_ref = $1) DESC NULLS LAST, id
LIMIT 1`,
[batch.id, header.order_date, header.total, merchant]
);
if (existing) return { id: existing.id, mine: existing.source_ref === batch.id };
const row = await q.queryRow<{ id: number }>(
`INSERT INTO transactions (
transaction_date, description, amount, amount_aud, category, payment_method,
merchant_name, merchant_normalized, transaction_type, owner_id, source, source_ref,
foreign_currency_amount, foreign_currency_code
) VALUES ($1::date, $2, $3::numeric, $4::numeric, $5, NULL, $6, $6, 'debit', $7, 'intake', $8, $9::numeric, $10)
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL
DO NOTHING
RETURNING id`,
[header.order_date, description, header.total, isAud ? header.total : null, category, merchant, ownerId, batch.id, isAud ? null : header.total, isAud ? null : currency]
);
return { id: row!.id, mine: true };
});
// Another batch booked this purchase in the moment between the two taps. Say
// the same thing the sequential case says one second later, and write nothing.
if (!outcome.mine) {
await updateBatch(batch.id, { kind, status: "duplicate", transaction_id: outcome.id });
const card = duplicateCard({ merchant: header.merchant, total: header.total, currency: header.currency, transactionId: outcome.id, captureId: null, batchId: null });
return finish(ctx, kind, "duplicate", card, { transaction_id: outcome.id });
}
const txnId = outcome.id;
await replaceLineItems(txnId, items, { source: "intake", platform: header.platform, batchId: batch.id, currency });
await updateBatch(batch.id, { kind, status: "created", transaction_id: txnId });
return finish(ctx, kind, "created", createdCard({ ...cardInput(ctx), transactionId: txnId }), { transaction_id: txnId });
}
/** The card for a batch as it stands — what a second press of a resolved card gets back. */
export function resolvedCard(batch: BatchRow): { blocks: Block[]; text: string } {
const h = batch.header;
const base: CardInput = { batchId: batch.id, merchant: h?.merchant ?? null, total: h?.total ?? null, orderDate: h?.order_date ?? null, platform: h?.platform ?? null, itemCount: batch.items?.length ?? 0, currency: h?.currency };
if (batch.status === "enriched" && batch.transaction_id) return enrichedCard({ ...base, transactionId: batch.transaction_id });
if (batch.status === "created" && batch.transaction_id) return createdCard({ ...base, transactionId: batch.transaction_id });
if (batch.status === "forwarded_to_pantry" && batch.capture_id) return forwardedCard({ ...base, captureId: batch.capture_id, enriching: !!batch.transaction_id });
return duplicateCard({ merchant: base.merchant, total: base.total, currency: base.currency, transactionId: batch.transaction_id, captureId: batch.capture_id, batchId: batch.id });
}
// ─── the router ─────────────────────────────────────────────────────────────
export type IntakeInput = { batchId?: string | null; source: IntakeSource; sharedBy: string | null; files: StoredFile[] };
export async function runIntake(input: IntakeInput, deps: IntakeDeps): Promise<IntakeResult> {
const batchId = input.batchId || randomUUID();
const shas = input.files.map((f) => f.sha256);
const platform = platformForSource(input.source);
if (await loadBatch(batchId)) {
return { kind: "unsupported", status: "duplicate", batch_id: batchId, slack_blocks: null, text: `batch ${batchId} was already received`, http_status: 409 };
}
await insertBatch({ id: batchId, source: input.source, sharedBy: input.sharedBy, shas });
const fail = (kind: IntakeKind, status: IntakeStatus, card: { blocks: Block[]; text: string }, http = 200, extra: Partial<IntakeResult> = {}): IntakeResult => ({
kind,
status,
batch_id: batchId,
slack_blocks: card.blocks,
text: card.text,
http_status: http,
...extra,
});
// 1. classify each file and build the parts, in the order shared.
const parts: InputPart[] = [];
const usable: StoredFile[] = [];
for (const f of input.files) {
const kind = classifyInput(f);
if (kind === "pdf") {
let text = "";
try {
text = await extractPdfText(f.bytes);
} catch {
/* unreadable PDF — reported below */
}
if (!text) {
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
return fail("unsupported", "unsupported", unsupportedCard(`${f.name} has no readable text — try a screenshot of it`));
}
parts.push({ kind: "text", text, name: f.name });
} else if (kind === "image") {
parts.push({ kind: "image", mime: f.type || "image/jpeg", bytes: f.bytes, name: f.name });
} else if (kind === "text") {
parts.push({ kind: "text", text: f.bytes.toString("utf-8"), name: f.name });
} else {
continue;
}
usable.push(f);
}
if (!parts.length) {
await updateBatch(batchId, { kind: "unsupported", status: "unsupported" });
return fail("unsupported", "unsupported", unsupportedCard());
}
// A re-share of the very same files, before spending a model call on it.
const dupBySha = await findDuplicateBatch(batchId, shas, null);
if (dupBySha) {
await updateBatch(batchId, { status: "duplicate", transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id });
return fail("unsupported", "duplicate", duplicateCard({ merchant: dupBySha.header?.merchant ?? null, total: dupBySha.header?.total ?? null, currency: dupBySha.header?.currency, transactionId: dupBySha.transaction_id, captureId: dupBySha.capture_id, batchId: dupBySha.id }), 200, {
kind: (dupBySha.kind as IntakeKind) ?? "food",
duplicate_of: { batch_id: dupBySha.id, transaction_id: dupBySha.transaction_id, capture_id: dupBySha.capture_id },
});
}
// Keep the originals: a later "Groceries" tap forwards them to pantry.
try {
await storeFiles(usable);
} catch (e) {
console.warn("intake: could not store files", e instanceof Error ? e.message : e);
}
// 2. ONE extraction call for the whole batch.
let extracted: ExtractedOrder;
try {
extracted = await deps.extractor(parts);
} catch (e) {
const detail = e instanceof ExtractionError ? e.message : e instanceof Error ? e.message : String(e);
await updateBatch(batchId, { status: "error" });
return fail("held", "error", errorCard(detail), 502);
}
const items = toLineItems(extracted.items);
// 3. header.
const headerCheck = validateHeader(extracted);
if (!headerCheck.ok) {
await updateBatch(batchId, { header: { merchant: extracted.merchant, order_date: extracted.order_date, total: extracted.total, platform }, items, status: "needs_header" });
return fail("held", "needs_header", needsHeaderCard(), 200, { items });
}
const header = headerCheck.header;
const storedHeader = { ...header, platform, subtotal: extracted.subtotal ?? null, currency: extracted.currency ?? "AUD" };
// 4. merchant → grocery | food | ask.
const kind: MerchantKind = resolveMerchantKind(input.source, header.merchant, await rememberedKind(header.merchant));
// 5. the same order, already handled under other files.
const dup = await findDuplicateBatch(batchId, shas, storedHeader);
if (dup) {
await updateBatch(batchId, { header: storedHeader, items, kind, status: "duplicate", transaction_id: dup.transaction_id, capture_id: dup.capture_id });
return fail(kind, "duplicate", duplicateCard({ merchant: header.merchant, total: header.total, currency: storedHeader.currency, transactionId: dup.transaction_id, captureId: dup.capture_id, batchId: dup.id }), 200, {
header,
items,
duplicate_of: { batch_id: dup.id, transaction_id: dup.transaction_id, capture_id: dup.capture_id },
});
}
// 6. do the lines add up? (missing-screenshot check)
const recon = reconcileItems(extracted);
if (!recon.ok) {
await updateBatch(batchId, { header: storedHeader, items, kind, status: "held" });
return fail("held", "held", heldCard(recon.reason ?? "mismatch"), 200, { header, items });
}
await updateBatch(batchId, { header: storedHeader, items, kind, status: "received" });
const ctx: RouteContext = { batchId, source: input.source, sharedBy: input.sharedBy, platform, header, items, currency: storedHeader.currency };
// 7. act.
if (kind === "ask") {
await updateBatch(batchId, { kind: "ask", status: "ask" });
return finish(ctx, "ask", "ask", askCard(cardInput(ctx)));
}
if (kind === "grocery") return routeGrocery(ctx, usable, deps.pantry);
return routeFood(ctx);
}
+252
View File
@@ -0,0 +1,252 @@
/**
* Intake router — the pure decisions.
*
* Everything here is a function of its arguments: what kind of input a file is,
* which platform a source tap implies, whether a merchant is a grocer, whether a
* header is complete, whether the items add up. Kept free of the database and the
* model so each rule can be unit-tested against the exact strings real receipts
* print. The orchestration that calls these lives in `intake-router.ts`.
*
* Plan: smarthome `plans/intake-router-slice-plan.md` §2, §3, §5, §11 C1.
*/
export const INTAKE_SOURCES = ["uber_eats", "doordash", "coles", "woolworths", "other"] as const;
export type IntakeSource = (typeof INTAKE_SOURCES)[number];
export function isIntakeSource(value: unknown): value is IntakeSource {
return typeof value === "string" && (INTAKE_SOURCES as readonly string[]).includes(value);
}
export type InputKind = "image" | "pdf" | "text" | "unsupported";
/** Input type picks the extractor; it is independent of source and merchant. */
export function classifyInput(file: { type?: string | null; name?: string | null }): InputKind {
const type = (file.type ?? "").toLowerCase();
const name = file.name ?? "";
if (type === "application/pdf" || /\.pdf$/i.test(name)) return "pdf";
if (type.startsWith("image/") || /\.(png|jpe?g|webp|heic|gif)$/i.test(name)) return "image";
if (type.startsWith("text/") || /\.(txt|md)$/i.test(name)) return "text";
return "unsupported";
}
/**
* The platform the source tap implies, in `expense_metadata.platform` vocabulary.
* Null for an in-store receipt or `other`: those rows carry no platform, so the
* match falls back to (merchant, date, total) with no platform constraint.
*/
export function platformForSource(source: IntakeSource): string | null {
if (source === "uber_eats") return "ubereats";
if (source === "doordash") return "doordash";
return null;
}
export const PLATFORM_LABEL: Record<string, string> = {
ubereats: "Uber Eats",
doordash: "DoorDash",
uber: "Uber",
};
export function platformLabel(platform: string | null): string {
return platform ? PLATFORM_LABEL[platform] ?? platform : "";
}
/**
* A small pattern list, not an exact lookup (plan §2, Codex #6). Uber Eats prints
* `Coles (Wyndham Vale)`, a PDF may print `Woolworths Group Limited`; a
* word-boundary alternation on the normalised name covers both without a
* merchant master-data subsystem.
*
* The boundaries are load-bearing, not tidiness: a bare substring made
* `Bigals Pizza` a grocer, because "b-IGA-ls" contains `iga`. Short names in a
* substring list collide with ordinary words, and the collision reads as a
* correct classification.
*/
export const GROCERY_PATTERN = /\b(coles|woolworths|woolies|aldi|iga|foodworks|costco)\b/i;
/**
* The normalised merchant used for matching and for the remembered-kind table.
* Lower-case, punctuation dropped, whitespace collapsed — so `Coles (Wyndham Vale)`
* and `coles wyndham vale` are one key, while `Coles` stays distinct from both.
*/
export function merchantKey(name: string | null | undefined): string {
return (name ?? "")
.toLowerCase()
.replace(/[^\p{L}\p{N}]+/gu, " ")
.trim()
.replace(/\s+/g, " ");
}
export type MerchantKind = "grocery" | "food" | "ask";
const UNKNOWN_MERCHANT = /^(unknown|n ?a|none|null|undefined|merchant)$/;
/**
* Grocery-vs-food is decided by the merchant, not the platform — "Uber Eats"
* covers both a meal and a grocery run. The source tap is ground truth where it
* says so (Coles/Woolworths taps are grocers by definition); the pattern list
* decides the obvious delivery cases; the remembered answer covers the
* `Romeo's Food Hall`s; and `other` with nothing else to go on asks rather than
* silently falling through to "meal" (plan §2, D3 holding state).
*/
export function resolveMerchantKind(
source: IntakeSource,
merchant: string | null | undefined,
remembered: "grocery" | "food" | null = null
): MerchantKind {
if (source === "coles" || source === "woolworths") return "grocery";
const key = merchantKey(merchant);
if (!key || UNKNOWN_MERCHANT.test(key)) return "ask";
if (GROCERY_PATTERN.test(key)) return "grocery";
if (remembered) return remembered;
if (source === "other") return "ask";
return "food";
}
/** What the extractor returns — names verbatim, options kept (plan §3). */
export type ExtractedItem = {
quantity: number | null;
description: string;
unit_price?: number | null;
line_total?: number | null;
options?: string[] | null;
};
export type ExtractedOrder = {
merchant: string | null;
order_date: string | null;
total: number | null;
subtotal?: number | null;
currency?: string | null;
items: ExtractedItem[];
adjustments?: { label: string; amount: number }[] | null;
};
export type OrderHeader = { merchant: string | null; order_date: string; total: number };
/**
* "Missing the header/total shot → nothing is written." A header is the date and
* the total; the merchant may be absent and still route (it becomes an ask).
*/
export function validateHeader(
x: ExtractedOrder
): { ok: true; header: OrderHeader } | { ok: false; missing: string[] } {
const missing: string[] = [];
if (typeof x.order_date !== "string" || !/^\d{4}-\d{2}-\d{2}$/.test(x.order_date)) missing.push("date");
if (typeof x.total !== "number" || !Number.isFinite(x.total) || x.total <= 0) missing.push("total");
if (missing.length) return { ok: false, missing };
return {
ok: true,
header: {
merchant: typeof x.merchant === "string" && x.merchant.trim() ? x.merchant.trim() : null,
order_date: x.order_date as string,
total: round2(x.total as number),
},
};
}
export const round2 = (n: number) => Number(n.toFixed(2));
/** A line's money, as printed or derived; null when the receipt gave neither. */
export function itemAmount(item: ExtractedItem): number | null {
if (typeof item.line_total === "number" && Number.isFinite(item.line_total)) return round2(item.line_total);
if (typeof item.unit_price === "number" && Number.isFinite(item.unit_price)) {
return round2(item.unit_price * (item.quantity ?? 1));
}
return null;
}
export type Reconciliation = {
ok: boolean;
basis: "subtotal" | "total" | "none";
items_sum: number;
reference: number | null;
diff: number;
reason?: string;
};
/**
* The missing-screenshot check (plan §3, Codex #7): the lines must reconcile
* within $1 or 2%, else the batch is held.
*
* Reconciled to the printed ITEM SUBTOTAL when there is one, not the total — a
* delivery order's total is items plus fees minus promotions ($118.94 of items
* became $85.74 on the real Coles export), so against the total every delivery
* order would be held. The total stays the match key; the subtotal is what the
* items can be checked against. Without a printed subtotal the total is the only
* reference there is. A receipt whose lines carry no prices at all (pasted text)
* cannot be checked and is not held for it.
*/
export function reconcileItems(x: ExtractedOrder): Reconciliation {
const amounts = (x.items ?? []).map(itemAmount);
if (!x.items?.length) {
return { ok: false, basis: "none", items_sum: 0, reference: null, diff: 0, reason: "no items" };
}
if (amounts.every((a) => a === null)) {
return { ok: true, basis: "none", items_sum: 0, reference: null, diff: 0 };
}
const itemsSum = round2(amounts.reduce<number>((s, a) => s + (a ?? 0), 0));
const subtotal = typeof x.subtotal === "number" && Number.isFinite(x.subtotal) ? x.subtotal : null;
const basis: Reconciliation["basis"] = subtotal !== null ? "subtotal" : "total";
const reference = subtotal ?? (typeof x.total === "number" ? x.total : null);
if (reference === null) return { ok: false, basis, items_sum: itemsSum, reference: null, diff: 0, reason: "no total" };
const diff = round2(itemsSum - reference);
const tolerance = Math.max(1, 0.02 * reference);
const ok = Math.abs(diff) <= tolerance;
return ok
? { ok, basis, items_sum: itemsSum, reference, diff }
: { ok, basis, items_sum: itemsSum, reference, diff, reason: `items sum to ${itemsSum.toFixed(2)} but the ${basis} is ${reference.toFixed(2)}` };
}
/**
* The stored shape — the same keys the order lane writes and
* `components/order-details.tsx` renders (`qty`, `description`, `amount`,
* `options`). A burger called "Michael" is stored as "Michael".
*/
export type StoredLineItem = {
qty: number;
description: string;
amount: number | null;
unit_price?: number | null;
options?: string[];
};
export function toLineItems(items: ExtractedItem[]): StoredLineItem[] {
return (items ?? [])
.filter((i) => typeof i.description === "string" && i.description.trim())
.map((i) => {
const out: StoredLineItem = {
qty: typeof i.quantity === "number" && i.quantity > 0 ? i.quantity : 1,
description: i.description.trim(),
amount: itemAmount(i),
};
if (typeof i.unit_price === "number" && Number.isFinite(i.unit_price)) out.unit_price = round2(i.unit_price);
const options = (i.options ?? []).map((o) => String(o).trim()).filter(Boolean);
if (options.length) out.options = options;
return out;
});
}
/** Is `needle` a contiguous run of whole words inside `hay`? */
function containsWordRun(hay: string[], needle: string[]): boolean {
if (!needle.length || needle.length > hay.length) return false;
for (let i = 0; i + needle.length <= hay.length; i++) {
if (needle.every((w, j) => hay[i + j] === w)) return true;
}
return false;
}
/**
* Does the stored merchant name denote the extracted one? One name's words must
* appear as a contiguous run inside the other's — so `Coles` matches
* `Coles (Wyndham Vale)` and `Order - Coles (Wyndham Vale)`, which is spelling,
* not widening.
*
* Whole words, never a raw substring: `merchantKey("Bigals Pizza")` contains
* `iga`, so a pizza shop matched every IGA on the same date and total.
*/
export function merchantMatches(stored: string | null | undefined, extracted: string | null | undefined): boolean {
const a = merchantKey(stored).split(" ").filter(Boolean);
const b = merchantKey(extracted).split(" ").filter(Boolean);
if (!a.length || !b.length) return false;
return containsWordRun(a, b) || containsWordRun(b, a);
}
+200
View File
@@ -0,0 +1,200 @@
import { queryRow } from "@/lib/db";
/**
* Replace the itemised receipt behind a transaction (plan §11 C2).
*
* An idempotent UPDATE, never an INSERT of a second row: `expense_metadata.
* transaction_id` is UNIQUE and the order the email pipeline booked already has
* its row — with `line_items = []`, because the items sat behind an auth-gated
* link. Idempotence is also what lets pantry re-run it after every confirm,
* writing the board's current state (plan §5, Codex #3), and what lets undo
* restore `previous_line_items` through the same call.
*
* The row is found on `transaction_id = :id OR matched_transaction_id = :id` —
* a card-settled order has only the latter (CLAUDE.md trap). When the
* transaction has no row at all (a manual transaction, a statement line), one
* is created so the items have somewhere to live; a transaction is NEVER
* created here.
*
* `opts.metadataId` pins WHICH row when the caller already resolved one. A
* statement transaction can own its own `expense_metadata` row AND be the
* `matched_transaction_id` of a delivery row: the match query and this write
* then re-pick independently and can land on different rows, so the enrichment
* reports success against a row nobody reads.
*/
export type LineItemsSource = "intake" | "pantry";
export class TransactionNotFoundError extends Error {}
export class LineItemsValidationError extends Error {}
export type LineItemRecord = Record<string, unknown> & { description: string };
/**
* Permissive on purpose: the order lane writes `{qty, description, amount,
* options}`, the receipt lane `{description, qty, amount, unit, category}`, and
* the detail panel reads the overlap. What every producer must supply is a
* description — an item with no name renders as nothing.
*/
export function validateLineItems(input: unknown): LineItemRecord[] {
if (!Array.isArray(input)) throw new LineItemsValidationError("line_items must be an array");
return input.map((raw, i) => {
if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
throw new LineItemsValidationError(`line_items[${i}] must be an object`);
}
const item = { ...(raw as Record<string, unknown>) };
if (typeof item.description !== "string" || !item.description.trim()) {
throw new LineItemsValidationError(`line_items[${i}].description is required`);
}
item.description = item.description.trim();
// `quantity` is the extractor's word; `qty` is what the renderer reads.
if (item.qty === undefined && typeof item.quantity === "number") item.qty = item.quantity;
return item as LineItemRecord;
});
}
export type ReplaceResult = {
transaction_id: number;
metadata_id: number;
created_metadata: boolean;
previous_line_items: unknown[];
line_items: LineItemRecord[];
};
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
/**
* The undo baseline, recorded once per batch.
*
* Returning the row that was just read makes a RETRY destroy the thing undo
* exists to restore: pantry re-calls this on every confirm and after a timeout,
* and the second call would hand back its own write as the "previous" state.
* The first snapshot a batch ever takes is the only one worth keeping, so it is
* written under COALESCE and read back rather than trusted.
*/
async function undoBaseline(batchId: string | null | undefined, snapshot: unknown[]): Promise<unknown[]> {
// Pantry passes back whatever batch id it was given; only an intake batch has
// a row to record against, and a non-uuid would fail the cast.
if (!batchId || !UUID.test(batchId)) return snapshot;
const row = await queryRow<{ previous_line_items: unknown }>(
`UPDATE intake_batches
SET previous_line_items = COALESCE(previous_line_items, $2::jsonb)
WHERE id = $1::uuid
RETURNING previous_line_items`,
[batchId, JSON.stringify(snapshot)]
);
return Array.isArray(row?.previous_line_items) ? row.previous_line_items : snapshot;
}
export async function replaceLineItems(
transactionId: number,
lineItems: LineItemRecord[],
opts: { source: LineItemsSource; platform?: string | null; batchId?: string | null; metadataId?: number | null; currency?: string | null }
): Promise<ReplaceResult> {
const txn = await queryRow<{
id: number;
merchant_normalized: string | null;
merchant_name: string | null;
amount: string;
transaction_date: string;
}>(
`SELECT id, merchant_normalized, merchant_name, amount::text, transaction_date::text
FROM transactions WHERE id = $1`,
[transactionId]
);
if (!txn) throw new TransactionNotFoundError(`transaction ${transactionId} does not exist`);
// The caller's row wins where it still exists; otherwise prefer the row that
// IS this transaction over one that merely points at it — a statement line
// can be the `matched_transaction_id` of a receipt row and carry its own row
// too, and the items belong on the receipt.
const pinned = opts.metadataId
? await queryRow<{ id: number; line_items: unknown }>(
`SELECT id, line_items FROM expense_metadata
WHERE id = $1 AND (transaction_id = $2 OR matched_transaction_id = $2)`,
[opts.metadataId, transactionId]
)
: null;
const json = JSON.stringify(lineItems);
/**
* Snapshot the baseline, then write. In that order on BOTH paths: the create
* path used to snapshot after its INSERT, so a crash in between left a retry
* recording the intake's own write as "previous" — undo restoring the thing
* it was meant to reverse, which is the failure `previous_line_items` exists
* to prevent.
*
* Returns null when the row is gone between the read and the write. That
* UPDATE reported success and the batch was marked enriched against a row
* nothing had written to.
*/
const write = async (row: { id: number; line_items: unknown }): Promise<ReplaceResult | null> => {
const previous = await undoBaseline(opts.batchId, Array.isArray(row.line_items) ? row.line_items : []);
const updated = await queryRow<{ id: number }>(
`UPDATE expense_metadata SET line_items = $2::jsonb WHERE id = $1 RETURNING id`,
[row.id, json]
);
if (!updated) return null;
return { transaction_id: transactionId, metadata_id: row.id, created_metadata: false, previous_line_items: previous, line_items: lineItems };
};
if (pinned) {
const done = await write(pinned);
if (done) return done;
}
const existing = await queryRow<{ id: number; line_items: unknown }>(
`SELECT id, line_items FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1
-- NULLS LAST is load-bearing, and its absence inverted this clause.
-- Postgres orders DESC as NULLS FIRST, and the row that merely POINTS at
-- the transaction has a NULL transaction_id — so the comparison meant to
-- prefer the owned row preferred the other one, silently and in exactly
-- the case it was written for. matchTransaction spells it out; this did
-- not, and the two therefore disagreed.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`,
[transactionId]
);
if (existing) {
const done = await write(existing);
if (done) return done;
}
// Nothing to update. Snapshot the empty baseline BEFORE creating the row.
const previous = await undoBaseline(opts.batchId, []);
const reference = opts.batchId ? `intake:${opts.batchId}` : `intake:txn:${transactionId}`;
// ON CONFLICT because two calls for one transaction can both find no row and
// both INSERT; expense_metadata.transaction_id is UNIQUE, so one of them lost
// with a constraint error that reached the user as an error card. `xmax = 0`
// is how a plain insert is told from a conflict-update, so the reported
// `created_metadata` stays true.
const created = await queryRow<{ id: number; inserted: boolean }>(
`INSERT INTO expense_metadata (
transaction_id, source, platform, order_reference, line_items,
merchant_normalized, amount, transaction_date, currency, flags, extraction_model
) VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::numeric, $8::date, $9, '[]'::jsonb, 'cloud-budget')
ON CONFLICT (transaction_id) DO UPDATE SET line_items = EXCLUDED.line_items
RETURNING id, (xmax = 0) AS inserted`,
[
transactionId,
opts.source,
opts.platform ?? null,
reference,
json,
txn.merchant_normalized ?? txn.merchant_name ?? null,
txn.amount,
txn.transaction_date,
// Not hard-coded 'AUD': the transaction may carry a foreign figure, and
// OrderDetails prefers this column, so a USD order printed "$".
opts.currency || "AUD",
]
);
return {
transaction_id: transactionId,
metadata_id: created!.id,
created_metadata: created!.inserted,
previous_line_items: previous,
line_items: lineItems,
};
}
+51
View File
@@ -0,0 +1,51 @@
/**
* Presentational tidy for a merchant name (board 212).
*
* PRESENTATION ONLY. It merges nothing and must never be used to decide that
* two rows are the same merchant. Real unification is the merchant-alias bridge
* (ticket 176, `jobs/merchant_merge.py`), which has already merged what can be
* merged safely; what is left on screen is one entity looking scruffy, not two
* entities that should be one.
*
* `Amazon.in` MUST NOT become `amazon.com.au`. They are different marketplaces
* with different currency and geography — a genuine country distinction, not a
* name variant. `Kogan.com`, `GOG.com`, `AliExpress.com` and `Catch.com.au` are
* real brand names that happen to contain a TLD and are already correct.
*
* Two rules, both mechanical:
* 1. drop trailing corporate suffixes
* 2. capitalise a name that arrived lower-cased FROM A DOMAIN
*/
const CORP_SUFFIX =
/,?\s+(pty\.?\s+ltd\.?|pty\.?\s+limited|pte\.?\s+ltd\.?|pte\.?|p\/l|ltd\.?|limited|inc\.?|llc|pbc|gmbh|b\.?v\.?|s\.?a\.?r\.?l\.?|oü|co\.?)$/i;
/** A bare domain used as a name: all lower case, and a dotted TLD. */
const BARE_DOMAIN = /^[a-z0-9][a-z0-9.-]*\.[a-z]{2,}(\.[a-z]{2,})?$/;
/**
* Stripping must not leave a dangling connector. `Coburger & Co` is a brand
* whose last word happens to match the suffix list, and removing it produced
* `Coburger &` — visibly broken, and the kind of thing that only shows up when
* you run the rule over the real corpus rather than over examples you chose.
*/
const DANGLING = /(\s[&+-]|\sand)$/i;
export function tidyMerchant(name: string): string {
let n = name.replace(/\s+/g, " ").trim();
// At most two passes: "Samsung Electronics Co. Ltd." needs Ltd. then Co.
for (let i = 0; i < 2; i++) {
const stripped = n.replace(CORP_SUFFIX, "").trim().replace(/,$/, "");
if (stripped === n || stripped.length < 3 || DANGLING.test(stripped)) break;
n = stripped;
}
// "amazon.com.au" reads as a machine artefact; "Amazon.com.au" reads as a
// name. Gated on BARE_DOMAIN, not on "starts with a lower-case letter": the
// looser test turned `eBay Commerce Australia Pty Ltd.` into `EBay Commerce
// Australia`, mangling a brand that is deliberately lower-cased.
if (BARE_DOMAIN.test(n)) n = n[0].toUpperCase() + n.slice(1);
return n;
}
+102 -21
View File
@@ -79,6 +79,12 @@ export interface OrderRow {
* every order here came from email. */
auth_verdict: string | null;
injection_flagged: boolean | null;
/** An annual tax/donation receipt restating a year of payments already made,
* not a purchase (board 211). It stays on this page — the figure is the
* year's giving total and is worth seeing — but `order_spend` excludes it,
* so the amount here is deliberately NOT part of any total on the page.
* Badge it, or a $2,376 row reads as a single purchase. */
summarises_period: boolean;
txn_count: number;
first_txn_id: number | null;
}
@@ -102,9 +108,22 @@ export interface OrderFilters {
/**
* Rows with no amount, no order reference and a single lifecycle event are not
* purchases — they are a second entity minted from a mail describing an order
* that already exists (Amazon "share your experience", DoorDash no-contact
* delivery details, shipment notices). 755 of them; hiding lifts amount
* coverage from 74% to 84%.
* that already exists (DoorDash no-contact delivery details, shipment notices).
* 620 of them as of 2026-08-12.
*
* The count moved three times that day and the history is the point:
* 755 when this predicate was written
* 904 after board 219 turned unstated zeros into NULLs — those 149 rows had
* been failing the "no amount" test on a technicality
* 699 after board 210a retired 205 Amazon "share your experience" mails,
* which were never lifecycle events at all (ingest/order_noise.py)
* 620 after board 210b joined 79 phantoms to their real parent
* (jobs/phantom_order_joiner.py, placebo-verified at a 1.9% error rate)
*
* What is left is the corpus coverage ceiling rather than a defect: a mail
* whose parent is not in the corpus, or whose merchant is ambiguous, has
* nothing to join to. This stays a UI workaround for those, and the toggle
* still says so.
*
* This is a WORKAROUND for board 210, not a fix. The upstream defect is that a
* lifecycle mail printing no order reference cannot join its own order, so it
@@ -114,6 +133,14 @@ export interface OrderFilters {
const LIFECYCLE_ONLY = `NOT (
f.order_total IS NULL
AND f.reference_source = 'message_id_fallback'
-- TRIED AND REVERTED, 2026-08-12: exempting rows that carry line items, on
-- the theory that naming what was bought proves a purchase. It surfaced 442
-- rows and the very first window showed why it is wrong — the Amazon
-- "Ordered:" mail that mints a msg- phantom carries the item list too, so
-- Xelsluthe and YIWENTEC each rendered TWICE again, which is the exact
-- complaint this page was fixed for. Items do not separate an orphan
-- purchase from an item-carrying echo; only real deduplication does, and
-- that is board 210, deliberately not attempted here.
AND (SELECT count(*) FROM extracted_facts ef
WHERE ef.fact_type = 'order_event'
AND ef.payload->>'_order_entity_key' = f.entity_key) <= 1
@@ -186,18 +213,41 @@ function buildWhere(filters: OrderFilters) {
}
/**
* Ledger linkage. Phase 1 reads expense_metadata; phase 2 swaps this lateral
* AND the detail query AND /api/transactions/[id]/order together — moving only
* one of the three leaves the list contradicting the detail page.
* Ledger linkage — phase 2 (board 205).
*
* `order_transaction_links` is the source of truth: it is many-to-many, so a
* BNPL plan is four rows against one order. `expense_metadata` remains as a
* FALLBACK because `jobs/order_transaction_bridge.py` still writes there on
* every hourly tick; until a bridge row is mirrored into links, dropping the
* fallback would make freshly bridged orders read as unpaid.
*
* UNION, not UNION ALL, on transaction_id: after the 0030 backfill the same
* charge is legitimately present in both stores, and counting it twice would
* show "2 charges" on a single-payment order.
*
* All four readers move together — this lateral, the detail query below,
* /api/transactions/[id]/order, and the order_ctx lateral in queries.ts.
* Moving fewer is not a smaller change, it is an inconsistent one: once the
* matcher inserts four links for order_ebay_14-11714-95953 that order still
* has no expense_metadata row, so the detail page would show four instalments
* while the list showed txn_count = 0 and the has_transaction facet put it on
* the wrong side of both filters.
*/
const LINK_LATERAL = `
LEFT JOIN LATERAL (
SELECT count(*)::int AS txn_count,
min(em.matched_transaction_id) AS first_txn_id
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND (em.source_message_id = f.entity_key
OR em.source_message_id LIKE f.entity_key || '#f%')
SELECT count(*)::int AS txn_count, min(txn_id) AS first_txn_id
FROM (
SELECT l.transaction_id AS txn_id
FROM order_transaction_links l
WHERE l.entity_key = f.entity_key
UNION
SELECT COALESCE(em.matched_transaction_id, em.transaction_id)
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND (em.source_message_id = f.entity_key
OR em.source_message_id LIKE f.entity_key || '#f%')
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
) both_stores
) link ON true`;
const FROM_CLAUSE = `
@@ -226,7 +276,7 @@ export async function getOrderFeed(filters: OrderFilters) {
f.ordered_at, f.eta_date, f.delivered_at,
f.currency, f.order_total, f.line_item_count,
f.content_class, f.display_name, f.cadence_days,
f.auth_verdict, f.injection_flagged,
f.auth_verdict, f.injection_flagged, f.summarises_period,
m.canonical_name AS merchant_name,
link.txn_count, link.first_txn_id,
(SELECT sp.refunded_amount FROM order_spend sp
@@ -341,6 +391,12 @@ export interface OrderLinkedTxn {
description: string;
amount: string;
source_message_id: string;
/** 'charge' | 'shipment' | 'instalment' | 'refund' | 'fee'. A plan's legs are
* four 'instalment' rows; a whole-order card charge is one 'charge'. */
leg_kind: string;
/** Human position, "2 of 4" — populated for instalments only. */
leg_index: number | null;
leg_count: number | null;
}
export interface OrderDetail {
@@ -365,6 +421,9 @@ export interface OrderDetail {
tracking_carrier: string | null;
line_items: { description?: string; quantity?: number; amount?: number }[];
is_settled_duplicate: boolean;
/** Board 211 — an annual receipt restating a year of payments already made.
* Same shape as is_settled_duplicate: browsable, excluded from spend. */
summarises_period: boolean;
content_class: string | null;
display_name: string;
cadence_days: number | null;
@@ -392,6 +451,7 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
o.order_total, o.tracking_url, o.tracking_carrier,
m.canonical_name AS merchant_name,
(o.settles_entity_id IS NOT NULL) AS is_settled_duplicate,
o.summarises_period,
COALESCE(o.details->'line_items', '[]'::jsonb) AS line_items,
sp.order_total AS net_total,
sp.gross_total,
@@ -467,14 +527,35 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
);
const transactions = await queryRaw<OrderLinkedTxn>(
`SELECT t.id::int AS transaction_id, t.transaction_date, t.description,
t.amount, em.source_message_id
FROM expense_metadata em
JOIN transactions t
ON t.id = COALESCE(em.matched_transaction_id, em.transaction_id)
WHERE em.source = 'order-bridge'
AND (em.source_message_id = $1 OR em.source_message_id LIKE $1 || '#f%')
ORDER BY t.transaction_date`,
// Links first, expense_metadata as fallback, deduped on transaction_id —
// see LINK_LATERAL above for why both stores are read. DISTINCT ON keeps
// the LINK row when a transaction is in both, because only that row knows
// whether the payment was one charge or leg 3 of 4.
// The DISTINCT ON must be ordered by t.id, so the chronological sort the
// page needs goes on the OUTER query — legs read as "1, 2, 3, 4" only if
// they come back by date.
`SELECT * FROM (
SELECT DISTINCT ON (t.id)
t.id::int AS transaction_id, t.transaction_date, t.description,
t.amount, both_stores.source_message_id,
both_stores.leg_kind, both_stores.leg_index, both_stores.leg_count
FROM (
SELECT l.transaction_id, l.leg_kind, l.leg_index::int, l.leg_count::int,
l.entity_key AS source_message_id, 0 AS pref
FROM order_transaction_links l
WHERE l.entity_key = $1
UNION ALL
SELECT COALESCE(em.matched_transaction_id, em.transaction_id),
'charge', NULL::int, NULL::int, em.source_message_id, 1
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND (em.source_message_id = $1 OR em.source_message_id LIKE $1 || '#f%')
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
) both_stores
JOIN transactions t ON t.id = both_stores.transaction_id
ORDER BY t.id, both_stores.pref
) deduped
ORDER BY transaction_date, leg_index NULLS FIRST`,
[entityKey]
);
-7
View File
@@ -12,13 +12,6 @@ export const CUTOVER_DATE = "2026-01-09";
*/
export const DEFAULT_OWNER_ID = 1;
/** Human labels for the platform a receipt came from. */
export const PLATFORM_LABEL: Record<ParsedOrder["platform"], string> = {
doordash: "DoorDash",
ubereats: "Uber Eats",
uber: "Uber",
};
/**
* Transaction description.
*
+10
View File
@@ -119,6 +119,16 @@ export async function merchantForTransaction(
const row = await queryRow<{ merchant_normalized: string | null }>(
`SELECT merchant_normalized FROM expense_metadata
WHERE transaction_id = $1 OR matched_transaction_id = $1
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (transaction_id = $1) DESC NULLS LAST, id
LIMIT 1`,
[transactionId]
);
+135
View File
@@ -0,0 +1,135 @@
/**
* Partial-month comparison rules (Priority 0, item 6).
*
* A month in progress is not comparable to a complete one, and getting that
* wrong is silent: nothing fails, the current month simply looks thrifty
* because most of it has not happened yet. Two distinct rules follow from it,
* and both are applied in three places on the analytics page:
*
* 1. **The in-progress month is never part of a baseline.** Averaging it in
* drags the "typical" figure down by however much of the month is still in
* the future.
* 2. **When the in-progress month is the one being looked at, every month it
* is compared against is truncated to the same day.** Its running total is
* measured against the same slice of each prior month, not their full
* totals.
*
* This module exists so those rules can be tested. They previously lived as
* closures inside `src/app/budget/page.tsx` reading `new Date()` directly, so
* "is this month in progress" depended on the wall clock at run time and no
* test could assert either rule without waiting for the calendar. Every
* function here takes `now` explicitly, defaulting to the real clock, which is
* the whole reason the behaviour is now guardable.
*
* Pair with `/api/analytics/daily`: rule 2 needs day-of-month granularity,
* which is why that route exists.
*/
/** `YYYY-MM` for a Date, in local time. */
export function monthStrOf(d: Date): string {
return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}`;
}
/** The month currently in progress. */
export function currentMonthStr(now: Date = new Date()): string {
return monthStrOf(now);
}
/** The calendar month before `m`, as `YYYY-MM`. */
export function prevMonth(m: string): string {
const [year, month] = m.split("-").map(Number);
return monthStrOf(new Date(year, month - 2, 1));
}
/** Total days in month `m`. */
export function daysInMonthOf(m: string): number {
const [year, month] = m.split("-").map(Number);
return new Date(year, month, 0).getDate();
}
/**
* How much of `m` has actually happened. A month in progress is complete only
* up to today; every earlier month is complete.
*/
export function elapsedDays(m: string, now: Date = new Date()): number {
return m === currentMonthStr(now) ? now.getDate() : daysInMonthOf(m);
}
/** Spend in a month from day 1 through `throughDay` inclusive. */
export function spendThrough(
days: Record<number, number> | undefined,
throughDay: number
): number {
if (!days) return 0;
let sum = 0;
for (const [d, v] of Object.entries(days)) {
if (Number(d) <= throughDay) sum += v;
}
return sum;
}
/**
* Whether `selectedMonth` is still in progress, and the day both sides of a
* comparison must be truncated to.
*
* For a complete month the day is its last, which makes `spendThrough` a no-op
* — so callers can use the same code path either way.
*/
export function comparisonBasis(
selectedMonth: string,
now: Date = new Date()
): { isPartial: boolean; compareDay: number } {
return {
isPartial: selectedMonth === currentMonthStr(now),
compareDay: elapsedDays(selectedMonth, now),
};
}
/**
* The months that may form a baseline for `selectedMonth`.
*
* Excludes the selected month itself, the month in progress (rule 1), and any
* month with no spend recorded — an empty month is absence of data, not a
* thrifty month, and averaging zeroes in understates the baseline.
*/
export function baselineMonths(
months: readonly string[],
selectedMonth: string,
totals: Record<string, { spent: number }>,
now: Date = new Date()
): string[] {
const current = currentMonthStr(now);
return months.filter(
(m) => m !== selectedMonth && m !== current && (totals[m]?.spent || 0) > 0
);
}
/**
* Mean spend across `baseline`, truncated to `compareDay` when the selected
* month is partial.
*
* `daily` is the sparse `{ [month]: { [day]: amount } }` map from
* `/api/analytics/daily`; `totals` supplies whole-month figures for the
* complete-month case.
*/
export function baselineAverage(
baseline: readonly string[],
{
isPartial,
compareDay,
daily,
totals,
}: {
isPartial: boolean;
compareDay: number;
daily: Record<string, Record<number, number>> | undefined;
totals: Record<string, { spent: number }>;
}
): number {
if (!baseline.length) return 0;
const sum = baseline.reduce(
(s, m) => s + (isPartial ? spendThrough(daily?.[m], compareDay) : totals[m]?.spent || 0),
0
);
return sum / baseline.length;
}
+56
View File
@@ -0,0 +1,56 @@
// Canonical payment methods. Mirrors `transactions_payment_method_check`
// (migration 0018, which admitted 'credits') — keep them in sync.
//
// This list exists because the drift already happened: the CHECK constraint
// admitted 'credits' from 0018 onward while the PATCH route's hand-written
// allowlist and the add-transaction dropdown both still stopped at 'other'.
// A gift-card row could be written by the order pipeline but neither entered
// nor corrected by hand.
export const PAYMENT_METHODS = [
"card",
"cash",
"bank_transfer",
"credits",
"other",
] as const;
export type PaymentMethod = (typeof PAYMENT_METHODS)[number];
export const PAYMENT_METHOD_LABELS: Record<PaymentMethod, string> = {
card: "Card",
cash: "Cash",
bank_transfer: "Bank transfer",
credits: "Gift card / credits",
other: "Other",
};
export function isPaymentMethod(v: string | null | undefined): v is PaymentMethod {
return !!v && (PAYMENT_METHODS as readonly string[]).includes(v);
}
export function formatPaymentMethod(v: string | null | undefined): string {
return isPaymentMethod(v) ? PAYMENT_METHOD_LABELS[v] : "Unknown";
}
/**
* Methods held out of the reconciliation queue. The TypeScript mirror of
* `needsCardMatch()` in `queries.ts` — if one changes, change both.
*
* NULL means *unknown*, not "no card leg", so it stays a candidate.
*/
export const NON_RECONCILING: readonly PaymentMethod[] = ["cash", "credits"];
/**
* Why a method never reaches the queue. Shown at the point of choosing it,
* because the consequence is invisible afterwards: the row simply never
* appears in pending reconciliation, which reads identically to a bug.
*/
export const NON_RECONCILING_REASON: Record<string, string> = {
cash: "Cash won't be offered for reconciliation — it never appears on a statement.",
credits:
"A gift-card payment won't be offered for reconciliation — the balance already paid it, so no card line is coming. It shows as “Gift Card” rather than “Manual” in the bank column.",
};
export function reconcilesAgainstStatement(v: string | null | undefined): boolean {
return !isPaymentMethod(v) || !NON_RECONCILING.includes(v);
}
+160 -22
View File
@@ -37,7 +37,16 @@ export interface TransactionRow {
payment_method: string | null;
/** Uber pick-up/drop-off, when this row came from an order receipt. */
order_route: RoutePointRow[] | null;
/** Receipt platform ONLY — it gates the disclosure arrow, so it must stay
* true to "there is a receipt behind this row". */
order_platform: "doordash" | "ubereats" | "uber" | null;
/** Phase 2 (board 205) — set when this transaction is linked to an order.
* 'instalment' means it is one leg of a plan; leg_index/leg_count carry
* "2 of 4", which is what stops one purchase reading as four. */
order_leg_kind: string | null;
order_leg_index: number | null;
order_leg_count: number | null;
order_name: string | null;
// override fields
category_override: string | null;
merchant_override: string | null;
@@ -151,6 +160,30 @@ interface TransactionFilters {
trip_all_rows?: boolean;
}
/**
* `myShare` from analytics-sql.ts, in subselect form.
*
* Same resolution and the same ownership branch, spelled out inline because
* this query has no `transaction_splits ts` join scoped to the viewer — it
* already aggregates every participant's splits for display, and adding a
* second viewer-scoped join purely to reuse the fragment would be the more
* confusing of the two. Kept adjacent to its twin so a change to one is an
* obvious prompt to change the other; the mirror is covered by a test that
* asserts the two agree row for row.
*/
export const MY_SHARE_PCT = `CASE
WHEN COALESCE(t.owner_id, s.owner_id) = $1 THEN COALESCE(
(SELECT x.share_percent FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id = $1),
o.my_share_percent,
100 - COALESCE((SELECT SUM(x.share_percent) FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id <> $1), 0)
)
ELSE COALESCE(
(SELECT x.share_percent FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id = $1), 0)
END`;
export async function getTransactions(ownerId: number, filters: TransactionFilters) {
// A real trip id, not "unassigned" — there is no trip to participate in for
// rows that belong to none, so the owner scoping has to stand there.
@@ -288,23 +321,14 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
${NATIVE_CURRENCY} as currency,
${AMOUNT_UNCONVERTED} as amount_unconverted,
-- My share, resolved the same way analytics does it (see myShare in
-- analytics-sql.ts): explicit split row, then override, then whatever is
-- left after everyone else. Computed here so the UI cannot drift from
-- the totals it is drilling into.
COALESCE(
(SELECT x.share_percent FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id = $1),
o.my_share_percent,
100 - COALESCE((SELECT SUM(x.share_percent) FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id <> $1), 0)
)::numeric(5,2) as my_share_pct,
(COALESCE(t.amount_aud, t.amount) * COALESCE(
(SELECT x.share_percent FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id = $1),
o.my_share_percent,
100 - COALESCE((SELECT SUM(x.share_percent) FROM transaction_splits x
WHERE x.transaction_id = t.id AND x.participant_id <> $1), 0)
) / 100)::numeric(12,2) as my_amount,
-- analytics-sql.ts): on a row I paid for, explicit split row, then
-- override, then whatever is left after everyone else; on a row someone
-- else paid for, only an explicit split row, absent meaning 0. Computed
-- here so the UI cannot drift from the totals it is drilling into — this
-- list shows rows I merely hold a split on, and before MY_SPEND_SCOPE
-- those rows read as a share of a total that excluded them entirely.
${MY_SHARE_PCT}::numeric(5,2) as my_share_pct,
(COALESCE(t.amount_aud, t.amount) * ${MY_SHARE_PCT} / 100)::numeric(12,2) as my_amount,
COALESCE(t.owner_id, s.owner_id) as owner_id,
p.name as owner_name,
COALESCE(src.created_at, t.created_at) as created_at,
@@ -314,7 +338,17 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
txn_tags.tags,
txn_splits.splits,
order_ctx.route as order_route,
order_ctx.platform as order_platform
-- Deliberately NOT COALESCEd with the link's platform. order_platform
-- gates the receipt disclosure arrow on /transactions, and a BNPL leg
-- has no receipt behind it — filling this in put an arrow on four
-- Afterpay rows that expand to nothing, which is the exact promise the
-- arrow exists to avoid making. The leg fields below carry the sub-line
-- instead.
order_ctx.platform as order_platform,
order_link.leg_kind as order_leg_kind,
order_link.leg_index as order_leg_index,
order_link.leg_count as order_leg_count,
order_link.canonical_name as order_name
FROM transactions t
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN statements s ON s.id = t.statement_id
@@ -323,12 +357,37 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
-- only thing that tells them apart, and it was already stored.
-- Both directions, because a card-settled order has no transaction of its
-- own and points at the statement line instead (I5).
-- Phase 2 (board 205): a BNPL leg has no expense_metadata row of its own,
-- so COALESCE in the link's platform. Without it the four Afterpay debits
-- behind the A$1,599 drone keep an empty sub-line while their order sits
-- one join away. The route column only ever exists on the receipt side.
-- (No backticks in here: this SQL lives in a TS template literal and a
-- backtick ends the string — TS1005 on a line that looks like a comment.)
LEFT JOIN LATERAL (
SELECT em.route, em.platform
FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
-- Ordered the same way as replaceLineItems (line-items.ts) and
-- matchTransaction (intake-router.ts), so every reader of this row
-- resolves the same one a write lands on. NULLS LAST because DESC is
-- NULLS FIRST and the row that merely POINTS at the transaction has a
-- NULL transaction_id.
-- Measured 2026-08-21: 667 expense_metadata rows, 147 of them
-- pointing-only, and ZERO transactions resolving to more than one row by
-- either link — so this changes nothing today. It is here so the readers
-- still agree on the day multiplicity appears.
ORDER BY (em.transaction_id = t.id) DESC NULLS LAST, em.id
LIMIT 1
) order_ctx ON true
LEFT JOIN LATERAL (
SELECT o.platform, l.leg_kind, l.leg_index::int AS leg_index,
l.leg_count::int AS leg_count, e.canonical_name
FROM order_transaction_links l
LEFT JOIN entities e ON e.entity_key = l.entity_key
LEFT JOIN entity_orders o ON o.entity_id = e.id
WHERE l.transaction_id = t.id
LIMIT 1
) order_link ON true
LEFT JOIN participants p ON p.id = COALESCE(t.owner_id, s.owner_id)
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
LEFT JOIN trips tr ON tr.id = o.trip_id
@@ -636,8 +695,11 @@ export async function batchInsertCSVTransactions(
foreign_currency_amount?: number;
foreign_currency_code?: string;
category?: string;
account?: string;
source_ref?: string;
}[],
tagId: number
tagId: number,
source?: string
): Promise<number> {
if (rows.length === 0) return 0;
@@ -651,13 +713,32 @@ export async function batchInsertCSVTransactions(
const params: unknown[] = [ownerId];
let p = 2;
rows.forEach((r, i) => {
valueClauses.push(`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`);
params.push(r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null, r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null);
valueClauses.push(
`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`
);
params.push(
r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null,
r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null,
// The category the reviewer chose. It used to be accepted here and then
// left out of the column list, so every category picked in the review
// step was silently discarded and the trigger wrote 'other'. That is not
// cosmetic: an uncategorised credit is admitted by NET_SPEND_ROWS and
// negated by SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of
// spend and none of them counted as income.
r.category ?? null,
source ?? null,
// source + source_ref is the only idempotency this path has. Without it a
// second import of the same file duplicates every row, because row_index
// is assigned MAX+1 and uq_transaction_identity can never fire.
r.source_ref ?? null,
r.account ?? null
);
});
const txIds = await queryRaw<{ id: number }>(
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, row_index)
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, category, source, source_ref, source_account, row_index)
VALUES ${valueClauses.join(", ")}
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL DO NOTHING
RETURNING id`,
params
);
@@ -698,6 +779,27 @@ export async function batchInsertCSVTransactions(
export const needsCardMatch = (alias = "t") =>
`(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`;
/**
* REMOVED 2026-08-13. There used to be an `awaitsStatementLine()` predicate
* here, excluding account-feed rows from the pending-reconciliation queue.
*
* Its premise — a feed row is the account's own ledger entry, so no statement
* line is coming — was asserted and never tested. It was false for almost every
* account: 422 of the first 550 imported rows already had a statement twin.
*
* How it got in matters more than what it did. The queue jumped from 8 to 558
* the moment the feed landed, and that jump was read as noise and filtered
* away. The queue was right — those rows genuinely were provisional entries
* awaiting statement lines — and hiding them removed the only mechanism that
* would ever have collapsed them, so a transient overlap became a permanent
* double-count and stayed invisible until a human saw one salary payment listed
* twice in two currencies.
*
* An imported row that has no statement belongs in the queue. Volume is solved
* upstream, by not importing what the statements already cover
* (`getStatementCoverage`), never downstream by hiding what is there.
*/
/**
* Bank label for a transaction. A row with no statement was not imported from
* one, and the label has to say *why*: "Manual" reads as "hand-entered, still
@@ -1467,3 +1569,39 @@ export async function getTagTransactionIds(tagId: number): Promise<number[]> {
);
return rows.map((r) => r.transaction_id);
}
/**
* The last date each account's statements cover, keyed by the account's last
* four digits.
*
* This is the coverage test that matters, and it is not the one that was tried
* first. The first attempt asked whether a row's date fell inside a statement's
* min-max *window*, which for accounts whose statements span 182 to 460 days
* swallows a year and answers nothing — 422 duplicates out of 550 rows got in
* that way. `billing_end_date` asks a question that has an answer: up to what
* date is this account complete?
*
* Amount matching cannot substitute for it. Two sources decompose the same
* event differently — an aggregator bundles the Wise transfer fee into the
* transfer (10001.13) where the statement itemises it (10000.00 + 1.13) — so
* the rows are the same money with different numbers.
*
* Keyed on last4 because account numbers are written differently everywhere
* ("235242176", "xxxxxxxxxxxx2176", "4085-56264"). Where two banks share a
* last4 the later date wins, which errs towards excluding a row rather than
* duplicating one; the caller shows every watermark so a wrong one is visible.
*/
export async function getStatementCoverage(): Promise<{ last4: string; coveredTo: string }[]> {
const rows = await queryRaw<{ last4: string; covered_to: string }>(
`SELECT right(regexp_replace(account_number, '[^0-9]', '', 'g'), 4) AS last4,
MAX(billing_end_date)::text AS covered_to
FROM statements
WHERE account_number IS NOT NULL
GROUP BY 1`,
[]
);
return rows
.filter((r) => r.last4 && r.last4.length === 4)
.map((r) => ({ last4: r.last4, coveredTo: r.covered_to }))
.sort((a, b) => a.last4.localeCompare(b.last4));
}
+4
View File
@@ -9,6 +9,10 @@
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
// scripts/*.mts run under `node --experimental-strip-types`, which resolves
// ESM specifiers literally and so needs the `.ts` extension written out.
// Legal here because noEmit is set — nothing is rewritten on the way out.
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "react-jsx",