18 Commits
Author SHA1 Message Date
siddharthd 22c2349a47 docs: one CSV import path, and the statement-coverage rule that replaced the Frollo importer
ci / lint-test (push) Successful in 43s
2026-08-13 15:09:11 +10:00
siddharthd 3edcc27781 csv import: map a currency column, or foreign rows land as AUD
ci / lint-test (push) Successful in 41s
Deleting the Frollo importer dropped its currency handling and nothing replaced
it: ColumnMapping had no currency column, so applyMapping could never produce
foreign_currency_code and batchInsertCSVTransactions' support for it was
unreachable from the UI. The USD 10,782 salary imported as A$10,782 — about a
third under, sitting in a column of AUD figures looking entirely normal, which
is the same defect the owner spotted in the first place.

An optional Currency column now sets foreign_currency_code and
foreign_currency_amount when the cell is a three-letter code other than AUD, and
leaves amount_aud NULL rather than inventing a rate. That is the shape order
ingestion already uses and what AMOUNT_UNCONVERTED looks for, so the row renders
as its native figure with "no AUD rate" and the statement supplies the real
number when it arrives.

Caught by reading the imported row rather than the import summary: the summary
said 171 inserted and was right about everything it reported.
2026-08-13 15:08:50 +10:00
siddharthd 461c021e7a csv import: exclude what the statements already cover, and delete the Frollo importer
ci / lint-test (push) Successful in 43s
"This is becoming too complex... Frollo should be done through that [the manual
CSV import]" (owner). It was right: a bespoke importer, an API route, a CLI, two
scheduled n8n workflows and a shared secret existed to do what the CSV import
modal already did, minus one rule.

That rule is statement coverage, and it turns out to be the whole thing. Applying
each account's newest billing_end_date as a watermark takes the real 2,607-row
Frollo export down to 171 rows — with no account allowlist, no credit-card
exclusion and no Frollo-specific scoping at all. Cards drop out on their own
because their statements are current; the 46 card rows that survive are genuinely
post-statement. Every bit of the bespoke apparatus was doing by hand what one
query does generically.

Deleted: src/lib/frollo-csv.ts, src/lib/frollo-ingest.ts, scripts/import-frollo.mts,
src/app/api/frollo/, both test files, the FROLLO_INGEST_TOKEN wiring, and the n8n
Frollo Import + Frollo Freshness Check workflows.

Added to the shared CSV path, so every import benefits:

  - getStatementCoverage() + /api/import/statement-coverage. The review step
    leaves out rows an account's statements already cover and says how many, with
    the rows one click away. Only applies when an account column is mapped and
    that account has statements — a row is never dropped on a guess.
  - Optional Account and Row ID columns in the mapper. Account drives the
    watermark and is stored as source_account; Row ID becomes source_ref.
  - An in-file duplicate warning. A CDR re-consent re-exports history under
    fresh ids, so source_ref cannot see it — 385 twins in one 2,563-row export
    doubled every salary payment, and that is not visible by eye in a review
    table.

Two pre-existing bugs in that path, both of which this plan depends on:

  - The category chosen in the review step was accepted by
    batchInsertCSVTransactions and then left out of the INSERT column list, so it
    was silently discarded and the trigger wrote 'other'. Not cosmetic: an
    uncategorised credit is admitted by NET_SPEND_ROWS and negated by
    SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of spend while
    counting as no income.
  - The path had no idempotency whatsoever. row_index is assigned MAX+1 on every
    run, which makes uq_transaction_identity structurally unable to fire, so a
    second import of the same file duplicated all of it. Now writes source +
    source_ref with ON CONFLICT DO NOTHING.

awaitsStatementLine()'s removal note is kept but rewritten: it no longer points
at a deleted file, and the lesson stands — the queue jump from 8 to 558 was the
measurement, not the noise.
2026-08-13 12:58:06 +10:00
siddharthd b82c4570bd frollo: don't import what the statements already cover
ci / lint-test (push) Successful in 45s
"We should not be importing from Frollo what we already have from statements"
(owner). The amount+direction guard could not deliver that, because the two
sources decompose the same event differently: Frollo bundles the Wise fee into
the transfer (10001.13) where the statement itemises it (10000.00 + 1.13). So 38
Wise USD rows passed the amount guard as new while being the same money, and were
the reason the transactions view showed a USD figure where every neighbouring row
showed AUD.

A statement's billing_end_date is a hard watermark: everything on that account up
to that date is already in the ledger, itemised and converted. Guard 1 now drops
any feed row at or before its account's newest statement. Guard 2 (amount +
direction) stays as the net for accounts that have no statement at all.

Note this is the coverage test the first import needed and got wrong. That one
asked whether a row's date fell inside a statement's min-max window, which for
periods spanning 182 to 460 days swallows a year and answers nothing. The
watermark asks a question that has an answer: up to what date is this account
complete?

Matching is on last4, verified against the live statement set — the eight
in-scope accounts with statements each map to one bank, no cross-bank collision.
The watermarks are printed by the CLI so a wrong boundary is visible rather than
inferred from a row count.

Re-imported from empty: 425 covered by statement, 15 amount twins, 110 inserted.
Exactly one row now carries a foreign currency with no AUD figure — the
2026-08-12 HDR salary, which is the genuinely-new pre-statement row this feed
exists for. Was 39.
2026-08-13 12:36:08 +10:00
siddharthd afd75d3f09 transactions: never print a foreign amount as AUD
ci / lint-test (push) Successful in 48s
The amount cell rendered formatAmount(amount_aud ?? amount) with no currency
argument, so a row with no AUD figure fell back to its native amount and was
stamped with a dollar sign. A USD 10,782.00 Wise credit showed as "+$10,782.00"
in a column of AUD figures — the same payment from the statement, one row above,
correctly showed $15,518.53 over USD 10,782.00. So the feed row read as $10,782
AUD when the real value is about $15,500, and the sub-line just repeated the
same number correctly labelled.

amount_unconverted is already selected by the transactions query (queries.ts:298)
and was simply unused here. When set, the native figure becomes the headline and
the absence of a rate is stated rather than papered over.

Pre-existing rather than Frollo's: any row with amount_aud NULL and a foreign
currency hits it, including foreign order-receipt rows. Frollo made it visible
at scale — 39 rows.
2026-08-13 12:32:17 +10:00
siddharthd 2738213a23 docs: record the ledger-duplicate guard and why awaitsStatementLine is gone
ci / lint-test (push) Successful in 40s
2026-08-13 12:24:59 +10:00
siddharthd bb8a009e02 frollo: don't re-import what the ledger already has, and stop hiding the queue
ci / lint-test (push) Successful in 44s
The first import wrote 550 rows on 2026-08-13. 422 of them (77%) were second
copies of transactions the ledger already held from statements — $1,023,824.63
of movement counted twice. The owner found it by opening the transactions view
and seeing one HDR Global salary listed twice, once as A$15,518.53 from the
statement and once as US$10,782.00 from the feed.

The currency was never the defect. toLedgerRow already left amount_aud NULL and
named the currency in foreign_currency_code, which is the documented contract for
a row whose AUD value is unknown, and the transactions page labels it. What made
it look wrong was the duplicate sitting beside it.

Two changes.

Upstream, ingestFrolloCsv now drops rows the ledger already holds, matching on
amount + direction within LEDGER_MATCH_DAYS (3). Three things had to be right and
the first two were not, each caught only by rehearsing against real data rather
than fixtures:

  - pg returns a DATE as a JS Date while Prisma and the CSV give strings.
    String(date).slice(0,10) is "Wed Mar 10", which parses to NaN, so the first
    dry run reported 550 to insert and zero duplicates. dayMs() takes both.
  - Direction has to be in the key. This ledger is full of internal transfers
    between the owner's own accounts and the feed carries both legs: 2026-05-18
    has +3076.04 into ANZ and -3076.04 out of AMP. Matching on amount alone let
    the credit leg consume the ledger's debit row, so the real duplicate was
    written — 20 rows got in that way.
  - 'refund' is money in. The feed calls a reversed account fee a credit and the
    statement importer types it 'refund'; classifying it as an outflow left every
    ANZ servicing-fee reversal behind.

Downstream, awaitsStatementLine() is removed. Its premise — feed rows never await
a statement line — was asserted, never tested, and false for almost every
account. Worse is how it got there: the reconcile queue jumped 8 -> 558 when the
feed landed, that jump was read as noise and filtered away, and filtering it
removed the only mechanism that would ever have collapsed the duplicates. The
queue was right. A feed row IS a row awaiting its statement line.

Re-imported: 375 dropped as already-on-ledger, 175 inserted. Residual duplicates
4 rows / $15.01, all sub-$5 account fees where several identical amounts fall in
overlapping windows and greedy consumption picks the wrong one; not chased
further at this scale.

The CLI prints the already-on-ledger count even when zero — a number you have to
go looking for is a number nobody looks at.
2026-08-13 12:23:01 +10:00
siddharthd 3e826a317b CLAUDE.md: document the Frollo account feed and the new reconcile exclusion
ci / lint-test (push) Successful in 39s
Two things a future reader would otherwise have to reverse-engineer.

The reconcile queue now excludes account-feed rows for a reason unrelated to
payment method: a feed row is the account's own ledger entry, not a receipt
awaiting a statement line, because the importer only covers accounts whose
statements are deliberately not imported. Measured, the queue went from 8 to
558 without it.

And the de-duplication rule needs its history attached, because the obvious
version is wrong in a way that passes tests: keeping close-id rows as genuine
repeats survived 29 tests and a clean dry run while still doubling three
salary payments. In-scope duplicate pairs have id gaps from 58 to 260
million; real repeats sat at 1-4.
2026-08-13 11:29:08 +10:00
siddharthd 21e9e765a3 Add /api/frollo/ingest so the import can run unattended
ci / lint-test (push) Successful in 40s
Shares one module with the CLI rather than reimplementing the insert:
frollo-ingest.ts holds parsing, scoping, de-duplication and the write, and
both callers pass in their own SQL executor (Prisma in the route, a pg
client in the script). The alternative is two implementations of the same
insert, which is how the pantry healthcheck came to be fixed in one repo
and left broken in the other.

The route refuses rather than guesses. findAnomalies() returns every reason
an unattended run should stop - a configured account contributing no rows,
an unrecognised account, a near-consecutive-id collapse that might be a
real repeat, a batch over ~200 rows, or an export taken with pending
included - and the route answers 409 having written nothing.

Two defects the wiring surfaced. Deliberately excluded credit cards were
reported as unknown accounts, which would have raised the new-account
anomaly on every single run and left the automatic path permanently
refusing; EXCLUDED_ACCOUNTS now distinguishes excluded from unknown. And
pending was tested after account scope, so pending rows on cards - which is
all of them so far - classified as out-of-scope and the wrong-export-option
signal could never fire; pending is now tested first.
2026-08-13 11:19:27 +10:00
siddharthd 493ff6f631 Import Frollo account feeds for the accounts statements don't cover
ci / lint-test (push) Successful in 47s
Credit cards keep arriving as monthly statements and stay the source of
truth for them. This covers the other fourteen accounts, whose statements
arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise
including the income account, Up, ING, and the small transaction accounts.
About 50 rows a month, where the alternative is downloading each statement
by hand.

The file needs three defences, all found by diffing three real exports
(smarthome DECISIONS.md ING-11):

A CDR re-consent makes Frollo re-ingest an account's whole history under
fresh transaction ids while the originals survive, and consents expire
annually. On this export 112 rows were such twins, and every HDR salary
payment appeared twice — importing blind doubles reported income. dedupe()
collapses each natural-key group to its lowest id, lowest because old ids
were a strict subset of new across two exports, so source_ref stays stable
and a re-import inserts nothing.

An earlier version of that rule kept close-id rows on a 10,000 threshold,
reasoning that genuine same-day repeats have consecutive ids. Verifying it
against the income rows killed it: in-scope duplicate pairs have id gaps
from 58 to 260 million, so no threshold separates them from the gaps of 1-4
that real repeats showed. It now collapses unconditionally and flags
anything within 10 for review — the errors are asymmetric, and nothing in
scope has ever tripped the flag.

A lapsed consent removes an account from the export silently, with no error
and no marker; the row count just drops. So the import asserts the account
roster and refuses to run when a configured account contributes nothing.

Also holds these rows out of the pending-reconciliation queue. A feed row is
the account's own ledger entry, not a receipt awaiting a statement line —
these accounts' statements are deliberately not imported — so without the
exclusion 550 rows a year would bury the receipts that need a decision. The
queue stays at 8 instead of 558.

Foreign rows follow order-ingestion's existing shape: amount is the native
figure, foreign_currency_code names it, amount_aud stays NULL rather than
asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as
incomplete.

Dry run by default. Verified against the real export before applying:
550 rows inserted, 14 accounts, re-run inserts 0.
2026-08-13 10:49:35 +10:00
siddharthd 2c666236b2 orders: share the merchant tidy, and fix two bugs the corpus run found (board 212)
ci / lint-test (push) Successful in 41s
The tidy existed but lived inside the list page, so the DETAIL page never had
it: following a row through showed "Apple" becoming "Apple Pty Ltd." Moved to
src/lib/merchant-label.ts and applied on both surfaces.

Running it over all 1,451 distinct merchant names — rather than over examples —
turned up two real defects:

  Coburger & Co                    -> "Coburger &"
  eBay Commerce Australia Pty Ltd. -> "EBay Commerce Australia"

The first strips a suffix that is part of the brand and leaves a dangling
connector. The second is the capitalisation rule firing on "starts with a
lower-case letter" when what it means is "is a bare domain" — mangling a brand
that is deliberately lower-cased. Both are now guarded, and re-running over the
full corpus reports 170 names tidied and 0 defects.

PRESENTATION ONLY. It merges nothing and must never decide two rows are the
same merchant. Amazon.in stays Amazon.in — a different marketplace with
different currency and geography, not a name variant — and there is a test
asserting it can never equal amazon.com.au. Kogan.com, GOG.com, AliExpress.com
and Catch.com.au are real brand names containing a TLD and are left alone.
Real unification is the merchant-alias bridge, ticket 176, which has already
merged what can be merged safely.

The five names that read alike after tidying (ebay/eBay/eBay Inc.,
menulog/Menulog Pty Ltd, deliveroo/Deliveroo, grab/Grab, paypal/PayPal) are
genuinely one merchant each, so reading alike is correct. Lower-case platform
SLUGS are deliberately not capitalised: "ebay" -> "Ebay" is wrong, and getting
it right is the registry's job, not the stylesheet's.
2026-08-12 21:07:50 +10:00
siddharthd 215303161b orders: bring the lifecycle-only toggle's numbers up to date (board 210)
ci / lint-test (push) Successful in 42s
The count moved three times on 2026-08-12 and the history is the point:

  755  when the predicate was written
  904  after board 219 turned unstated zeros into NULLs — those 149 rows had
       been failing the "no amount" test on a technicality
  699  after 210a retired 205 Amazon "share your experience" mails, which were
       never lifecycle events at all
  620  after 210b joined 79 phantoms to their real parent, placebo-verified at
       a 1.9% error rate

What is left is the corpus coverage ceiling rather than a defect — a mail whose
parent is not in the corpus has nothing to join to — and the tooltip now says
that instead of describing the whole class as an open ingestion defect.
2026-08-12 20:38:43 +10:00
siddharthd 3b5e495ca4 orders phase 2: order_transaction_links, and move every reader to it (board 205)
ci / lint-test (push) Successful in 41s
Phase 1 read linkage out of expense_metadata, which is shaped as ONE ROW PER
TRANSACTION — transaction_id UNIQUE, matched_transaction_id partial-unique — so
every multiplicity it expresses is smuggled through a string key (0029 keys
split shipments <entity_key>#f<fact_id>). A BNPL plan needs four rows against
one order and has no such trick available.

Migration 0030 adds order_transaction_links (many-to-many, keyed on entity_key
rather than entities.id: finance-app does not model the spine and must not hold
an FK across a boundary a re-extraction can decompile) and backfills all 63
existing bridge rows.

THE UNIQUENESS RULE WAS WRONG FIRST TIME, in the most on-brand way available.
It read (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of
the same order — both leg_index NULL — collapsed to one key and ON CONFLICT DO
NOTHING dropped one SILENTLY. Caught only because the backfill reported 62
against 63 candidates. The row it ate was order_amazon_249-4859367-0690246's
$130.00 second shipment, the same order named in migration 0029's comment as
the reason split shipments need distinct keys at all. What identifies a leg
depends on its kind: an instalment by its INDEX, a shipment by its FACT, a
whole-order charge by neither.

ALL FOUR READERS MOVE TOGETHER, links first with expense_metadata as fallback:

  - LINK_LATERAL (list txn_count/first_txn_id, and the has_transaction facets
    that read it)
  - the detail page's transactions query
  - /api/transactions/[id]/order
  - the order_ctx lateral in queries.ts

Moving fewer is not a smaller change, it is an inconsistent one: the matcher's
four links for order_ebay_14-11714-95953 come with no expense_metadata row, so
a half-move would show four instalments on the detail page while the list said
txn_count = 0 and put the order on the wrong side of BOTH has_transaction
filters. Verified after: detail 4 legs, list txn_count 4, has_transaction=yes
includes it, =no excludes it.

UNION not UNION ALL on transaction_id — after the backfill the same charge is
legitimately in both stores and counting it twice would show "2 charges" on a
single-payment order.

order_platform is deliberately NOT coalesced with the link's platform. It gates
the receipt disclosure arrow on /transactions, and a BNPL leg has no receipt
behind it — filling it in put an arrow on four Afterpay rows that expand to
nothing, which is the exact promise the arrow exists to avoid making. Separate
leg fields carry the sub-line instead ("1 of 4 - DJI Air 3 Fly More Combo").
order-details.tsx now also requires a real receipt (platform present) before
rendering, because the endpoint can answer with a link alone.

Two invariant views, both empty and expected to stay so: order_link_orphans
(spine re-keying silently orphans a TEXT key — bridge links are rebuildable,
`manual` ones are lost curation) and order_link_drift (the two stores
disagreeing). The plan's suggested fix for drift — widening the bridge's NOT
EXISTS guard to "neither store has it" — was NOT taken: the expense_metadata
INSERT has no ON CONFLICT, so re-attempting an order that already has a receipt
row would duplicate it. Detecting is cheap; a non-idempotent re-write is not.

Unchanged: order_feed 6,265, order_spend AUD 4,895 / $442,651.80. Links 75.
2026-08-12 20:29:42 +10:00
siddharthd 6b9b5fe518 orders: badge the annual summary receipts (board 211)
ci / lint-test (push) Successful in 40s
Migration 033 adds entity_orders.summarises_period; order_spend drops those
rows and order_feed keeps them. Unbadged, a $2,376 annual tax receipt
restating twelve monthly donations renders as the largest single purchase of
the year, so both surfaces say what it is:

- /orders  — a "year summary" chip beside the auth badges
- /orders/<key> — a banner in the same shape as the settled-rail one, saying
  the monthly debits are in the ledger under their own dates

The amount is deliberately still shown. It is the year's giving total and the
reason the row is worth keeping; it just belongs to no total on the page. The
detail page already falls back to o.order_total when order_spend has no row,
which is exactly the case now — verified: gross_total null, order_total 2125.5.
2026-08-12 20:12:36 +10:00
siddharthd 231ef3411a orders: record why line items may not exempt a row from the phantom filter
ci / lint-test (push) Successful in 40s
Board 219 nulled 263 unstated zero totals in the spine, which moved 149 rows
into the lifecycle-only hide predicate — they had been failing its "no amount"
test on a technicality, an unstated 0 they should never have carried, while
matching the phantom signature on every other count.

Tried exempting rows that carry line items, on the theory that naming what was
bought proves a purchase. It surfaced 442 rows and the first window checked
showed why it is wrong: the Amazon "Ordered:" mail that mints a msg- phantom
carries the item list too, so Xelsluthe and YIWENTEC each rendered TWICE again
— the exact complaint this page was fixed for today. Reverted, and the reason
recorded in place so it is not re-attempted.

Items do not separate an orphan purchase from an item-carrying echo. Only
deduplication does, and that is board 210.

No behaviour change from the previous commit; comments and counts only.
2026-08-12 18:26:24 +10:00
siddharthd 699e4a2ddd orders: cast cadence order_count to int — BigInt broke every recurring merchant
ci / lint-test (push) Successful in 43s
order_merchant_cadence.order_count is a count(*), so BIGINT, and JSON.stringify
throws on it: 'Do not know how to serialize a BigInt', a 500 carrying no SQL.
It fires only where a cadence EXISTS, so the failure hid behind the
merchant_entity_id fault and reappeared the moment that was fixed — St. Ali
(40 orders, 14-day cadence) still 500'd while Amazon opened fine.
2026-08-12 17:12:14 +10:00
siddharthd 79940202f1 orders: fix the 42703 that broke EVERY order detail page
getOrderDetail joined order_merchant_cadence on merchant_entity_id. Migration
021 rekeyed that view to merchant_key (merchant_entity_id is NULL on ~9% of the
feed, and duplicate merchant entities split one shop's history); 027 and 029
moved order_feed and order_spend across, and this query was missed. Every
/orders/<key> request has since failed with

  column rec.merchant_entity_id does not exist

The reason nobody saw a 500 is the second half of this commit: the page
collapsed every failure into 'That order could not be found.' A server fault
wearing the costume of a data condition reads as an empty spine and gets
investigated in the wrong repo. The hook now carries the status and only a
genuine 404 says the order is missing.
2026-08-12 17:10:13 +10:00
siddharthd 9790b64e1d orders: badge sender authentication, not source_trust
source_trust is 'untrusted_external' on 100% of rows — every order came from
email — so the badge marked every row and discriminated nothing. auth_verdict
(migration 029) does: 93% pass, and the 7% that do not are the ones worth
seeing. Adds the injection-scanner flag beside it.
2026-08-12 17:10:04 +10:00
18 changed files with 1167 additions and 79 deletions
+72 -4
View File
@@ -131,7 +131,7 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
### Key Tables ### Key Tables
- `statements` — one row per billing period per bank account - `statements` — one row per billing period per bank account
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown - `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
### Cash and reconciliation ### Cash and reconciliation
@@ -142,9 +142,28 @@ within 3 days and 1% on amount — and accepting one is silently destructive:
reconciled manual rows are filtered out of every query, so the cash spend reconciled manual rows are filtered out of every query, so the cash spend
disappears while the card transaction it matched claims to be that same spend. disappears while the card transaction it matched claims to be that same spend.
Only cash is excluded. Bank transfers *do* appear on a statement now that Only cash is excluded on payment method. Bank transfers *do* appear on a
transaction accounts are imported, and NULL means unknown — both stay statement now that transaction accounts are imported, and NULL means unknown —
candidates, preserving the behaviour of every pre-existing row. both stay candidates, preserving the behaviour of every pre-existing row.
**Account-feed rows are NOT excluded, and the removed exclusion is worth
knowing about.** There was an `awaitsStatementLine()` predicate here, added with
the Frollo importer on the premise that a feed row is the account's own ledger
entry with no statement line coming. That premise was asserted and never tested.
It was false for almost every account: 422 of the first 550 imported rows already
had a statement twin.
What matters is how it got in. The queue jumped from 8 to 558 the moment the feed
landed, and that jump was read as noise and filtered out. **The queue was right**
those rows genuinely were provisional entries awaiting statement lines — and
filtering it removed the only mechanism that would ever have collapsed them, so
the duplicates became permanent instead of transient and stayed invisible until a
human saw one salary payment listed twice in two currencies.
A feed row belongs in the queue. Queue volume is solved **upstream**, by not
importing rows the ledger already holds (`LEDGER_MATCH_DAYS`), never downstream
by hiding the ones that are there. If a genuinely statement-less feed is ever
added, give it its own predicate and prove the premise with a query first.
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
as transfers only works if every cash purchase is logged; with partial logging as transfers only works if every cash purchase is logged; with partial logging
@@ -617,6 +636,55 @@ deleted, and deleting a rule must not cascade away the audit trail.
changed by something else since the run are flagged, because reverting restores changed by something else since the run are flagged, because reverting restores
the pre-run value and discards the later edit. the pre-run value and discards the later edit.
### Importing an aggregator/bank CSV (`/api/import/csv`)
There is **one** import path — the CSV import modal — and Frollo goes through it
like any other file. A bespoke Frollo importer, API route, CLI and two scheduled
n8n workflows existed for a day and were deleted on 2026-08-13: net 1,152 lines.
**The rule that made the rest unnecessary is statement coverage.** Each account's
newest `billing_end_date` is a watermark; a row on or before it is already in the
ledger. On the real 2,607-row export that leaves **171 rows** — with no account
allowlist and no credit-card exclusion, because cards have current statements and
drop out on their own. Every bit of the deleted apparatus was doing by hand what
`getStatementCoverage()` does generically.
Note what this replaced. The first attempt asked whether a row's date fell inside
a statement's minmax *window*, which for accounts whose statements span 182 to
460 days swallows a year and answers nothing — it let 422 duplicates into 550
rows. Amount-matching cannot substitute either: the two sources decompose the
same event differently, bundling a Wise transfer fee into the transfer (10001.13)
where the statement itemises it (10000.00 + 1.13).
**Map these optional columns or lose something silently:**
| Column | Without it |
|---|---|
| **Account** | nothing is excluded — you review the whole file |
| **Currency** | a foreign row is stored as if AUD (USD 10,782 → A$10,782) |
| **Row ID** | a re-import duplicates instead of no-opping |
Leave **Category** unmapped for an aggregator: its spend categories are not
trusted, and you set them per row in the review step.
**Two bugs fixed the day this shipped, both of which the flow depends on:**
`batchInsertCSVTransactions` accepted a `category` and then omitted it from the
INSERT column list, so every category chosen in review was discarded and the
trigger wrote `'other'` — not cosmetic, because an uncategorised credit is
admitted by `NET_SPEND_ROWS` and negated by `SPEND_SIGNED`, so 62 imported
transfers cancelled **$74,338** of spend while counting as no income. And the
path had no idempotency at all: `row_index` is assigned MAX+1 every run, making
`uq_transaction_identity` structurally unable to fire. It now writes
`source`/`source_ref` with `ON CONFLICT DO NOTHING`.
**The in-file duplicate warning is not redundant with `source_ref`.** A CDR
re-consent re-exports an account's whole history under fresh ids while the
originals survive — 385 twins in one 2,563-row export, doubling every salary
payment. Fresh ids mean `source_ref` sees new rows, and 385 is not visible by eye
in a review table. Consents expire annually, so expect it.
Undo an import: `DELETE FROM transactions WHERE source = '<source>'`.
### Trusting extracted statement data ### Trusting extracted statement data
**Balance assertions are the check that works.** `getStatements` computes **Balance assertions are the check that works.** `getStatements` computes
@@ -0,0 +1,127 @@
-- 0030 — order ↔ transaction, many-to-many (board 205, orders phase 2)
--
-- Phase 1 read linkage out of `expense_metadata`, which is shaped as ONE ROW
-- PER TRANSACTION: `transaction_id` is UNIQUE and `matched_transaction_id`
-- partial-unique. Every multiplicity it expresses today is smuggled through a
-- string key — migration 0029 keys split shipments `<entity_key>#f<fact_id>` —
-- and a BNPL plan needs four rows for one order with no such trick available.
--
-- The flagship case: order_ebay_14-11714-95953, a A$1,599 DJI drone paid in
-- four A$399.75 Afterpay legs (txns 2318, 2333, 1652, 1664). Phase 1 shows it
-- with zero linked transactions, which is correct and useless.
--
-- KEYED ON entity_key, NOT entities.id. finance-app does not model the spine
-- and must not carry an FK across an ownership boundary that a spine
-- re-extraction can decompile. The real cascade risk is NOT transaction
-- deletion (that is handled below) but spine RE-KEYING: supersede_stale()
-- decompiles an entity when a re-extraction yields a different entity_key, and
-- a TEXT key with no FK silently orphans. Bridge-sourced links are rebuildable;
-- `manual` ones are curation and are not. Hence the orphan view at the end.
CREATE TABLE IF NOT EXISTS order_transaction_links (
id SERIAL PRIMARY KEY,
entity_key TEXT NOT NULL,
transaction_id INTEGER NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
leg_kind TEXT NOT NULL CHECK (leg_kind IN ('charge','shipment','instalment','refund','fee')),
-- HUMAN position only: "2 of 4". Never an id.
--
-- The first draft stored a fact id here. Live shipment keys already carry
-- fact ids 22664 and 22711, max(extracted_facts.id) is 23,781 today, and
-- SMALLINT tops out at 32,767 — that ceiling arrives on ordinary corpus
-- growth and the insert dies with `smallint out of range`. Fact ids go in
-- source_fact_id, which is BIGINT.
leg_index SMALLINT,
leg_count SMALLINT,
source_fact_id BIGINT,
-- The LEG's own amount, signed: positive is money out, negative is a credit
-- coming back. Storing the order total on every leg would make four rows sum
-- to four times the purchase.
amount NUMERIC(12,2) NOT NULL,
currency TEXT NOT NULL DEFAULT 'AUD',
source TEXT NOT NULL CHECK (source IN ('order-bridge','instalment-matcher','manual')),
confidence TEXT NOT NULL DEFAULT 'exact' CHECK (confidence IN ('exact','derived','manual')),
evidence JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Every index needs IF NOT EXISTS, not just the table: a run that creates the
-- table and then fails during the backfill would otherwise abort on re-run
-- with "relation already exists" and need manual surgery.
--
-- uq_otl_transaction is load-bearing. Without it two matchers can each claim
-- txn 2318 and the order reads as paid twice. It is deliberately stricter than
-- reality for one case — a single card charge covering two orders — which then
-- surfaces as a NAMED REJECTED INSERT rather than as silent duplication.
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_transaction ON order_transaction_links (transaction_id);
-- WHAT IDENTIFIES A LEG DEPENDS ON ITS KIND, and the first version of this
-- index got it wrong in the most on-brand way available. It read
-- (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of the
-- same order — both with a NULL leg_index — collapsed to the same key and
-- ON CONFLICT DO NOTHING dropped one SILENTLY. Caught only because the
-- backfill reported 62 rows against 63 candidates.
--
-- The row it ate was order_amazon_249-4859367-0690246's $130.00 second
-- shipment (txn 3171) — the same order named in migration 0029's comment as
-- the reason split shipments need distinct keys at all, because Amazon mails
-- and CHARGES per shipment. A uniqueness rule that cannot hold two shipments
-- is the exact defect 0029 was written to fix.
--
-- So: an instalment leg is identified by its INDEX (1 of 4), a shipment leg by
-- its FACT (each shipment is its own extracted fact), and a whole-order charge
-- by neither — one per order, which the zeros express.
DROP INDEX IF EXISTS uq_otl_leg;
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_leg ON order_transaction_links
(entity_key, leg_kind, COALESCE(leg_index, 0), COALESCE(source_fact_id, 0));
CREATE INDEX IF NOT EXISTS idx_otl_entity ON order_transaction_links (entity_key);
-- Backfill the existing bridge rows so the table is not empty on day one and
-- the readers can switch over in the same change. `#f<fact_id>` splits into a
-- shipment leg carrying its fact id; everything else is a whole-order charge.
INSERT INTO order_transaction_links
(entity_key, transaction_id, leg_kind, leg_index, source_fact_id, amount, currency, source, confidence, evidence)
SELECT split_part(em.source_message_id, '#', 1) AS entity_key,
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
CASE WHEN em.source_message_id LIKE '%#f%' THEN 'shipment' ELSE 'charge' END,
NULL::smallint,
NULLIF(substring(em.source_message_id FROM '#f([0-9]+)$'), '')::bigint,
COALESCE(em.amount, 0),
COALESCE(em.currency, 'AUD'),
'order-bridge',
'exact',
jsonb_build_object('backfilled_from', 'expense_metadata',
'source_message_id', em.source_message_id)
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
ON CONFLICT DO NOTHING;
-- Spine re-keying orphans links silently. Bridge rows can be rebuilt by
-- re-running the bridge; a `manual` orphan is lost curation and is the row
-- that actually needs a human. Same idea as order_settlement_violations.
CREATE OR REPLACE VIEW order_link_orphans AS
SELECT l.id, l.entity_key, l.transaction_id, l.leg_kind, l.leg_index,
l.amount, l.currency, l.source, l.confidence, l.created_at
FROM order_transaction_links l
WHERE NOT EXISTS (SELECT 1 FROM entities e WHERE e.entity_key = l.entity_key);
-- Two stores that can disagree will disagree. The bridge now writes both in
-- ONE transaction, so a half-written pair should be impossible — this view
-- exists to prove that rather than to assume it, and it must stay empty.
--
-- The plan's suggested guard (widen the bridge's `NOT EXISTS expense_metadata`
-- to "neither store has it") was NOT taken: the expense_metadata INSERT has no
-- ON CONFLICT, so re-attempting an order that already has a receipt row would
-- duplicate it. Detecting drift is cheap; re-attempting a non-idempotent write
-- to fix a state that cannot occur is not.
CREATE OR REPLACE VIEW order_link_drift AS
SELECT em.source_message_id,
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
em.amount, em.currency, em.reconciled_at
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM order_transaction_links l
WHERE l.transaction_id = COALESCE(em.matched_transaction_id, em.transaction_id)
);
@@ -0,0 +1,216 @@
import { describe, it, expect } from "vitest";
import {
applyMapping,
inFileDuplicates,
last4,
splitByCoverage,
type ColumnMapping,
type ParsedTransaction,
} from "@/lib/csv-parser";
/**
* The statement-coverage rule, which replaced a whole Frollo-specific importer.
*
* The first aggregator import wrote 422 duplicate rows out of 550 — $1,023,824
* of movement counted twice — because "these accounts issue no statements" was
* asserted rather than queried. Amount-matching could not fix it either: the two
* sources decompose the same event differently, bundling a transfer fee where
* the statement itemises it.
*
* A statement's end date answers the question that actually has an answer: up
* to what date is this account complete? On the real 2026-08-13 export it takes
* 2,607 rows down to 171, with no account allowlist and no card exclusions.
*/
const tx = (o: Partial<ParsedTransaction> & { date: string }): ParsedTransaction => ({
description: "SOMETHING",
amount: 10,
transaction_type: "debit",
...o,
});
describe("last4", () => {
it("reads the same account written four different ways", () => {
// Every one of these forms appears in the real data.
expect(last4("xxxxxxxxxxxx2176")).toBe("2176");
expect(last4("235242176")).toBe("2176");
expect(last4("xxx-xxx xx4878")).toBe("4878");
expect(last4("4085-56264")).toBe("6264");
});
it("returns empty for an unusable identifier rather than guessing", () => {
expect(last4("N/A")).toBe("");
expect(last4("12")).toBe("");
expect(last4("")).toBe("");
});
});
describe("splitByCoverage", () => {
const coverage = [{ last4: "2176", coveredTo: "2026-07-26" }];
it("excludes a row on or before the account's newest statement", () => {
const { keep, covered } = splitByCoverage(
[tx({ date: "2026-07-25", account: "xxxxxxxxxxxx2176" })],
coverage
);
expect(covered).toHaveLength(1);
expect(keep).toHaveLength(0);
});
it("treats the watermark date itself as covered", () => {
const { covered } = splitByCoverage(
[tx({ date: "2026-07-26", account: "xxxxxxxxxxxx2176" })],
coverage
);
expect(covered).toHaveLength(1);
});
it("keeps a row after the watermark — the whole point of the feed", () => {
// The 2026-08-12 salary: the statement has not arrived, so this row is the
// only record of it and must survive.
const { keep } = splitByCoverage(
[tx({ date: "2026-08-12", account: "xxxxxxxxxxxx2176", amount: 10782 })],
coverage
);
expect(keep).toHaveLength(1);
});
it("keeps everything for an account with no statements", () => {
const { keep, covered } = splitByCoverage(
[tx({ date: "2020-01-01", account: "xxx-xxx xx4878" })],
coverage
);
expect(keep).toHaveLength(1);
expect(covered).toHaveLength(0);
});
it("keeps everything when no account column was mapped", () => {
// A single-account bank export has no account column, and a row must never
// be dropped on a guess about which account it belongs to.
const { keep, covered } = splitByCoverage([tx({ date: "2020-01-01" })], coverage);
expect(keep).toHaveLength(1);
expect(covered).toHaveLength(0);
});
it("applies each account's own watermark, not a global one", () => {
// The real export spans watermarks from 2026-03-31 to 2026-07-26. A single
// date would either import duplicates or discard real rows.
const { keep, covered } = splitByCoverage(
[
tx({ date: "2026-05-01", account: "xxxx0887" }), // covered to 2026-03-31 → keep
tx({ date: "2026-05-01", account: "xxxx2176" }), // covered to 2026-07-26 → drop
],
[
{ last4: "0887", coveredTo: "2026-03-31" },
{ last4: "2176", coveredTo: "2026-07-26" },
]
);
expect(keep).toHaveLength(1);
expect(keep[0].account).toBe("xxxx0887");
expect(covered).toHaveLength(1);
});
});
describe("inFileDuplicates", () => {
it("counts twins that a row id cannot catch", () => {
// A CDR re-consent re-exports an account's whole history under fresh ids
// while the originals survive, so source_ref sees two distinct rows. 385 of
// these were in one 2,563-row export and doubled every salary payment.
const rows = [
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
];
expect(inFileDuplicates(rows)).toBe(1);
});
it("does not flag the same amount on different days", () => {
expect(
inFileDuplicates([
tx({ date: "2026-07-07", amount: 5, description: "COFFEE" }),
tx({ date: "2026-07-08", amount: 5, description: "COFFEE" }),
])
).toBe(0);
});
it("does not flag the same amount on different accounts", () => {
// Both legs of an internal transfer: same day, same amount, different
// accounts, and both are real.
expect(
inFileDuplicates([
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "6264" }),
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "9940" }),
])
).toBe(0);
});
});
describe("applyMapping with account and row-id columns", () => {
const labels = ["transaction_id", "description", "amount", "transaction_date", "account_number"];
const mapping: ColumnMapping = {
dateCol: "transaction_date",
descriptionCol: "description",
amountMode: "single",
amountCol: "amount",
accountCol: "account_number",
sourceRefCol: "transaction_id",
};
it("carries the account and row id through", () => {
const out = applyMapping(
[["1326131276", "HDR Global Services", "10782.00", "2026-08-12", "xxxxxxxxxxxx4830"]],
labels,
mapping,
"YYYY-MM-DD"
);
expect(out).toHaveLength(1);
expect(out[0].account).toBe("xxxxxxxxxxxx4830");
expect(out[0].source_ref).toBe("1326131276");
// A positive single-column amount is money in.
expect(out[0].transaction_type).toBe("credit");
});
it("marks a non-AUD row as native and unconverted", () => {
// The 2026-08-12 salary is USD. Stored without this it reads as A$10,782 —
// about a third under — and sits in a column of AUD figures looking normal.
const out = applyMapping(
[["1", "HDR Global Services", "10782.00", "2026-08-12", "xxxx4830", "USD"]],
[...labels, "currency"],
{ ...mapping, currencyCol: "currency" },
"YYYY-MM-DD"
);
expect(out[0].foreign_currency_code).toBe("USD");
expect(out[0].foreign_currency_amount).toBe(10782);
});
it("leaves an AUD row unmarked", () => {
const out = applyMapping(
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830", "AUD"]],
[...labels, "currency"],
{ ...mapping, currencyCol: "currency" },
"YYYY-MM-DD"
);
expect(out[0].foreign_currency_code).toBeUndefined();
});
it("ignores a currency cell that is not a code", () => {
const out = applyMapping(
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830", ""]],
[...labels, "currency"],
{ ...mapping, currencyCol: "currency" },
"YYYY-MM-DD"
);
expect(out[0].foreign_currency_code).toBeUndefined();
});
it("leaves both undefined when the columns are not mapped", () => {
const out = applyMapping(
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830"]],
labels,
{ dateCol: "transaction_date", descriptionCol: "description", amountMode: "single", amountCol: "amount" },
"YYYY-MM-DD"
);
expect(out[0].account).toBeUndefined();
expect(out[0].source_ref).toBeUndefined();
expect(out[0].transaction_type).toBe("debit");
});
});
+72
View File
@@ -0,0 +1,72 @@
import { describe, expect, it } from "vitest";
import { tidyMerchant } from "@/lib/merchant-label";
/**
* Board 212. Every fixture below is a real `display_name` from order_feed.
* The cases that matter most are the ones that must NOT change: this is a
* presentational tidy and merging anything here would be a data error wearing
* a stylesheet.
*/
describe("tidyMerchant", () => {
it("capitalises a bare domain used as a name", () => {
expect(tidyMerchant("amazon.com.au")).toBe("Amazon.com.au");
expect(tidyMerchant("cdkeys.com")).toBe("Cdkeys.com");
});
it("NEVER merges the country marketplaces", () => {
// The single most important assertion in this file. Amazon.in is a
// different marketplace with different currency and geography, and 289
// orders sit behind it.
expect(tidyMerchant("Amazon.in")).toBe("Amazon.in");
expect(tidyMerchant("amazon.com.au")).not.toBe(tidyMerchant("Amazon.in"));
});
it("leaves real brand names that contain a TLD alone", () => {
for (const brand of ["Kogan.com", "GOG.com", "AliExpress.com", "Catch.com.au", "GeekBuying.com"]) {
expect(tidyMerchant(brand)).toBe(brand);
}
});
it("does not uppercase a deliberately lower-cased brand", () => {
// The bug the corpus run found: gating on "starts lower-case" rather than
// on "is a bare domain" turned this into "EBay Commerce Australia".
expect(tidyMerchant("eBay Commerce Australia Pty Ltd.")).toBe("eBay Commerce Australia");
expect(tidyMerchant("iRobot Australia Pty Ltd")).toBe("iRobot Australia");
});
it("drops trailing corporate suffixes", () => {
expect(tidyMerchant("Apple Pty Ltd.")).toBe("Apple");
expect(tidyMerchant("Microsoft Pty. Limited")).toBe("Microsoft");
expect(tidyMerchant("Meta Platforms, Inc.")).toBe("Meta Platforms");
expect(tidyMerchant("Domino's Pizza Enterprises Limited")).toBe("Domino's Pizza Enterprises");
expect(tidyMerchant("Blinks Labs GmbH")).toBe("Blinks Labs");
expect(tidyMerchant("Rasier New Zealand Limited")).toBe("Rasier New Zealand");
});
it("strips two suffixes when a name carries two", () => {
expect(tidyMerchant("Samsung Electronics Co. Ltd.")).toBe("Samsung Electronics");
expect(tidyMerchant("DiDi Mobility Information Technology Pte. Ltd."))
.toBe("DiDi Mobility Information Technology");
});
it("does not leave a dangling connector", () => {
// 'Co' is part of this brand. Stripping it produced "Coburger &".
expect(tidyMerchant("Coburger & Co")).toBe("Coburger & Co");
});
it("keeps a name that would be reduced to almost nothing", () => {
// "UT" is too short to stand on its own; better scruffy than cryptic.
expect(tidyMerchant("UT LLC")).toBe("UT LLC");
});
it("normalises whitespace and tolerates junk", () => {
expect(tidyMerchant(" Yaffa Media Pty Ltd ")).toBe("Yaffa Media");
expect(tidyMerchant("")).toBe("");
});
it("is idempotent", () => {
for (const n of ["amazon.com.au", "Apple Pty Ltd.", "Coburger & Co", "Amazon.in"]) {
expect(tidyMerchant(tidyMerchant(n))).toBe(tidyMerchant(n));
}
});
});
+9 -1
View File
@@ -8,6 +8,12 @@ export async function POST(req: NextRequest) {
const body = await req.json() as { const body = await req.json() as {
bank_name: string; bank_name: string;
/**
* Optional provenance label, stored on every row as `source`. With a
* `source_ref` per row it makes a re-import a no-op; without it this path
* has no idempotency at all.
*/
source?: string;
transactions: { transactions: {
date: string; date: string;
description: string; description: string;
@@ -17,6 +23,8 @@ export async function POST(req: NextRequest) {
foreign_currency_amount?: number; foreign_currency_amount?: number;
foreign_currency_code?: string; foreign_currency_code?: string;
category?: string; category?: string;
account?: string;
source_ref?: string;
}[]; }[];
}; };
@@ -25,7 +33,7 @@ export async function POST(req: NextRequest) {
} }
const tagId = await ensureTag("csv-import", "#8b5cf6"); const tagId = await ensureTag("csv-import", "#8b5cf6");
const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId); const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId, body.source);
return NextResponse.json({ inserted }, { status: 201 }); return NextResponse.json({ inserted }, { status: 201 });
} }
@@ -0,0 +1,17 @@
import { NextRequest, NextResponse } from "next/server";
import { getCurrentUser } from "@/lib/auth";
import { getStatementCoverage } from "@/lib/queries";
/**
* How far each account's statements already reach.
*
* Read by the CSV import modal so it can drop rows the ledger already holds.
* The first Frollo import went in without this and wrote 422 duplicates out of
* 550 — the same payments the statements already carried, itemised and
* converted. On the 2026-08-13 export the rule takes 2,607 rows down to 171.
*/
export async function GET(req: NextRequest) {
const user = await getCurrentUser(req);
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
return NextResponse.json({ coverage: await getStatementCoverage() });
}
+24 -1
View File
@@ -38,7 +38,30 @@ export async function GET(
[Number(id)] [Number(id)]
); );
// Phase 2 (board 205). A BNPL leg has NO expense_metadata row of its own —
// the plan is four transactions against one order — so without this the four
// Afterpay debits behind the A$1,599 DJI drone stay four bare "Afterpay
// $399.75" rows naming nothing. That is the surface where this was noticed.
const link = await queryRow<{
entity_key: string; leg_kind: string; leg_index: number | null;
leg_count: number | null; canonical_name: string | null;
platform: string | null; order_total: string | null; currency: string | null;
}>(
`SELECT l.entity_key, l.leg_kind, l.leg_index::int, l.leg_count::int,
e.canonical_name, o.platform, o.order_total, o.currency
FROM order_transaction_links l
LEFT JOIN entities e ON e.entity_key = l.entity_key
LEFT JOIN entity_orders o ON o.entity_id = e.id
WHERE l.transaction_id = $1
LIMIT 1`,
[Number(id)]
);
// Not an order — most transactions aren't. Null, not 404: the caller is // Not an order — most transactions aren't. Null, not 404: the caller is
// asking "is there a receipt behind this?", and "no" is a normal answer. // asking "is there a receipt behind this?", and "no" is a normal answer.
return NextResponse.json(row ?? null); if (!row && !link) return NextResponse.json(null);
// expense_metadata carries the itemised receipt and stays primary; the link
// adds what it cannot express — which order this is a leg OF, and which leg.
return NextResponse.json({ ...(row ?? {}), order_link: link ?? null });
} }
+49 -3
View File
@@ -3,6 +3,7 @@
import { use } from "react"; import { use } from "react";
import Link from "next/link"; import Link from "next/link";
import { useOrderDetail } from "@/lib/hooks"; import { useOrderDetail } from "@/lib/hooks";
import { tidyMerchant } from "@/lib/merchant-label";
/** /**
* One order: what it was, what happened to it, and what paid for it. * One order: what it was, what happened to it, and what paid for it.
@@ -63,10 +64,22 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
if (isLoading) return <div className="p-6 text-zinc-500 text-sm">Loading order</div>; if (isLoading) return <div className="p-6 text-zinc-500 text-sm">Loading order</div>;
if (error || !o) { if (error || !o) {
// A missing order and a broken query are different problems and must not
// share a sentence. Only a 404 means "no such order"; anything else is this
// page failing, and saying so is what sends the next person to the server
// log instead of to the spine.
const status = (error as (Error & { status?: number }) | null)?.status;
return ( return (
<div className="max-w-[1180px] mx-auto"> <div className="max-w-[1180px] mx-auto">
<Link href="/orders" className="font-mono text-[11.5px] text-indigo-400"> All orders</Link> <Link href="/orders" className="font-mono text-[11.5px] text-indigo-400"> All orders</Link>
<p className="mt-6 text-zinc-400 text-sm">That order could not be found.</p> {status === 404 ? (
<p className="mt-6 text-zinc-400 text-sm">That order could not be found.</p>
) : (
<p className="mt-6 text-zinc-400 text-sm">
This order could not be loaded{status ? ` (server error ${status})` : ""}. The order exists
something in this page failed. Check the finance-app log.
</p>
)}
</div> </div>
); );
} }
@@ -82,6 +95,23 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
{/* A settled rail row must never look like an ordinary order — that is {/* A settled rail row must never look like an ordinary order — that is
how a human counts the same purchase twice. */} how a human counts the same purchase twice. */}
{/* Board 211 — same shape as the settled-rail banner above: a real
document with a real figure that is not a purchase. Saying which
payments it restates is the point; the total alone reads as a
single large order. */}
{o.summarises_period && (
<div className="mt-4 bg-zinc-900 border-l-2 border-amber-600 px-4 py-3">
<span className="block font-mono text-[10px] uppercase tracking-widest text-amber-500 mb-1">
Annual summary, not a purchase
</span>
<span className="text-[12.5px] text-zinc-300">
This receipt restates payments already made across the year the
monthly debits are in the ledger under their own dates. It is kept
for reference and excluded from spend totals.
</span>
</div>
)}
{o.is_settled_duplicate && ( {o.is_settled_duplicate && (
<div className="mt-4 bg-zinc-900 border-l-2 border-indigo-600 px-4 py-3"> <div className="mt-4 bg-zinc-900 border-l-2 border-indigo-600 px-4 py-3">
<span className="block font-mono text-[10px] uppercase tracking-widest text-indigo-500 mb-1"> <span className="block font-mono text-[10px] uppercase tracking-widest text-indigo-500 mb-1">
@@ -100,7 +130,10 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
{o.ordered_at && <> · {dateFmt.format(new Date(o.ordered_at))}</>} {o.ordered_at && <> · {dateFmt.format(new Date(o.ordered_at))}</>}
</div> </div>
<h2 className="font-display text-[27px] leading-tight text-zinc-50 my-3 max-w-[26ch] text-balance"> <h2 className="font-display text-[27px] leading-tight text-zinc-50 my-3 max-w-[26ch] text-balance">
{o.canonical_name || o.display_name || "Order"} {/* canonical_name is the ORDER title ("DJI Air 3 Fly More Combo") and
is left alone; only the display_name fallback is a merchant name
and wants tidying. */}
{o.canonical_name || tidyMerchant(o.display_name ?? "") || "Order"}
</h2> </h2>
<div className="flex gap-2 flex-wrap mb-3"> <div className="flex gap-2 flex-wrap mb-3">
{o.content_class && KIND_LABEL[o.content_class] && ( {o.content_class && KIND_LABEL[o.content_class] && (
@@ -135,7 +168,11 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
{o.merchant_name && ( {o.merchant_name && (
<div> <div>
<span className="block text-[10px] uppercase tracking-widest text-zinc-500 mb-0.5">Merchant</span> <span className="block text-[10px] uppercase tracking-widest text-zinc-500 mb-0.5">Merchant</span>
{o.merchant_name} {/* Board 212: the list has tidied this since it shipped and the
detail page did not, so following a row through showed
"Apple" becoming "Apple Pty Ltd." Presentational only —
merges nothing, and Amazon.in stays Amazon.in. */}
{tidyMerchant(o.merchant_name)}
</div> </div>
)} )}
</div> </div>
@@ -197,6 +234,15 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
<span className="font-mono text-[12px] text-zinc-300"> <span className="font-mono text-[12px] text-zinc-300">
{t.description} {t.description}
<small className="block text-[10.5px] text-zinc-500"> <small className="block text-[10.5px] text-zinc-500">
{/* Board 205. Four identical "Afterpay" rows are
indistinguishable without their position, and
indistinguishable rows are what made this order
look like four purchases. */}
{t.leg_kind === "instalment" && t.leg_index && t.leg_count
? `Instalment ${t.leg_index} of ${t.leg_count} · `
: t.leg_kind === "shipment"
? "Shipment · "
: ""}
{dateFmt.format(new Date(t.transaction_date))} · txn {t.transaction_id} {dateFmt.format(new Date(t.transaction_date))} · txn {t.transaction_id}
</small> </small>
</span> </span>
+39 -30
View File
@@ -3,6 +3,7 @@
import { Suspense, useMemo, useState } from "react"; import { Suspense, useMemo, useState } from "react";
import Link from "next/link"; import Link from "next/link";
import { useOrders } from "@/lib/hooks"; import { useOrders } from "@/lib/hooks";
import { tidyMerchant } from "@/lib/merchant-label";
import type { OrderRow } from "@/lib/order-feed"; import type { OrderRow } from "@/lib/order-feed";
/** /**
@@ -75,31 +76,6 @@ const KIND_LABEL: Record<string, string> = {
account_notice: "notice", account_notice: "notice",
}; };
/**
* Presentational tidy only — this NEVER merges two merchants. The estate holds
* "amazon.com.au", "Amazon.in" and "Amazon Services Australia, Inc." as three
* distinct entities, and Amazon.in must stay separate: it is a different
* marketplace, not a name variant. Actually unifying them is the merchant-alias
* bridge (ticket 176). All this does is stop the same entity from looking
* scruffy: drop the corporate suffix, and capitalise a name that arrived
* lower-cased from a domain.
*/
const CORP_SUFFIX =
/,?\s+(pty\.?\s+ltd\.?|pty\.?\s+limited|p\/l|ltd\.?|limited|inc\.?|llc|pbc|gmbh|b\.?v\.?|s\.?a\.?r\.?l\.?|oü|co\.?)$/i;
function tidyMerchant(name: string): string {
let n = name.replace(/\s+/g, " ").trim();
// Strip at most two trailing corporate suffixes ("Pty Ltd." then ",").
for (let i = 0; i < 2; i++) {
const stripped = n.replace(CORP_SUFFIX, "").trim().replace(/,$/, "");
if (stripped === n || stripped.length < 3) break;
n = stripped;
}
// "amazon.com.au" reads as a machine artefact; "Amazon.com.au" reads as a name.
if (/^[a-z]/.test(n)) n = n[0].toUpperCase() + n.slice(1);
return n;
}
const IN_FLIGHT = new Set(["ordered", "shipped", "out_for_delivery"]); const IN_FLIGHT = new Set(["ordered", "shipped", "out_for_delivery"]);
const REVERSED = new Set(["refunded", "returned", "cancelled"]); const REVERSED = new Set(["refunded", "returned", "cancelled"]);
@@ -293,11 +269,44 @@ function Row({ row, expanded, onToggle }: { row: OrderRow; expanded: boolean; on
className="font-mono text-[9.5px] uppercase tracking-wide text-indigo-600 border border-indigo-800 rounded-sm px-1.5" className="font-mono text-[9.5px] uppercase tracking-wide text-indigo-600 border border-indigo-800 rounded-sm px-1.5"
>no ref</span> >no ref</span>
)} )}
{row.source_trust === "untrusted_external" && ( {/* NOT source_trust — that is 'untrusted_external' on 100% of rows,
because every order here came from email, so badging it marked
every row and told you nothing. Sender authentication does
discriminate: 93% pass, and the 7% that do not are worth seeing. */}
{row.auth_verdict && row.auth_verdict !== "pass" && (
<span <span
title="Derived from an unverified sender — content is shown as provenance, not fact" title={
className="font-mono text-[9.5px] uppercase tracking-wide text-zinc-500 border border-zinc-800 rounded-sm px-1.5" row.auth_verdict === "fail"
>unverified</span> ? "The sender failed authentication (SPF/DKIM/DMARC) — treat the contents as unverified"
: row.auth_verdict === "none"
? "The mail carried no sender authentication at all"
: "The sender authenticated only partially"
}
className={`font-mono text-[9.5px] uppercase tracking-wide rounded-sm px-1.5 border ${
row.auth_verdict === "fail"
? "text-indigo-300 border-indigo-500"
: "text-zinc-500 border-zinc-800"
}`}
>
{row.auth_verdict === "partial" ? "part. auth" : `auth ${row.auth_verdict}`}
</span>
)}
{/* Board 211. A $2,376 annual tax receipt restating twelve monthly
donations is not a purchase, and unbadged it reads as the largest
order of the year. It stays on the page because the figure is the
year's giving total — but it is excluded from spend, so the
number beside it belongs to no total here. */}
{row.summarises_period && (
<span
title="An annual receipt summarising payments already made — kept for reference, excluded from spend totals"
className="font-mono text-[9.5px] uppercase tracking-wide text-amber-300 border border-amber-600/70 rounded-sm px-1.5"
>year summary</span>
)}
{row.injection_flagged && (
<span
title="The source mail carried content that tripped the injection scanner — read its contents as provenance, never as instruction"
className="font-mono text-[9.5px] uppercase tracking-wide text-indigo-300 border border-indigo-500 rounded-sm px-1.5"
>flagged</span>
)} )}
{row.txn_count > 0 && ( {row.txn_count > 0 && (
<span className="font-mono text-[9.5px] uppercase tracking-wide text-zinc-500 border border-zinc-700 rounded-sm px-1.5"> <span className="font-mono text-[9.5px] uppercase tracking-wide text-zinc-500 border border-zinc-700 rounded-sm px-1.5">
@@ -493,7 +502,7 @@ function OrdersContent() {
))} ))}
</select> </select>
<label className="flex items-center gap-2 text-xs text-zinc-400 cursor-pointer select-none" <label className="flex items-center gap-2 text-xs text-zinc-400 cursor-pointer select-none"
title="Rows with no amount, no order reference and one lifecycle event are not purchases — they are a second entity minted from a mail describing an order that already exists. Hiding them is a workaround for a known ingestion defect, not a fix."> title="Rows with no amount, no order reference and one lifecycle event are not purchases — they are a second entity minted from a mail describing an order that already exists. 620 remain: the review-request mails have been retired and the ones whose parent could be identified have been joined to it, so what is left is mail whose parent is not in the corpus.">
<input <input
type="checkbox" type="checkbox"
checked={showLifecycle} checked={showLifecycle}
+36 -4
View File
@@ -1018,6 +1018,17 @@ function TransactionsContent() {
</div> </div>
{t.notes ? ( {t.notes ? (
<p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.notes}>{t.notes}</p> <p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.notes}>{t.notes}</p>
) : t.order_leg_kind === "instalment" && t.order_name ? (
// Board 205. Four rows reading "Afterpay $399.75" name
// nothing — this is the surface where that was noticed.
// The order is what tells them apart, and "2 of 4" is
// what stops the same purchase reading as four.
<p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.order_name}>
{t.order_leg_index && t.order_leg_count
? `${t.order_leg_index} of ${t.order_leg_count} · `
: ""}
{t.order_name}
</p>
) : t.order_platform === "uber" && routeSummary(t.order_route) && ( ) : t.order_platform === "uber" && routeSummary(t.order_route) && (
// Five rows all reading "Order - Uber Trip" are // Five rows all reading "Order - Uber Trip" are
// indistinguishable. Where the trip went is what tells // indistinguishable. Where the trip went is what tells
@@ -1048,11 +1059,32 @@ function TransactionsContent() {
<td className={`p-2 text-right whitespace-nowrap font-mono ${ <td className={`p-2 text-right whitespace-nowrap font-mono ${
SPEND_TYPES.has(t.transaction_type) ? "text-red-400" : "text-green-400" SPEND_TYPES.has(t.transaction_type) ? "text-red-400" : "text-green-400"
}`}> }`}>
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)} {/*
{t.currency && t.currency !== "AUD" && ( A row with no AUD figure must NOT be printed as AUD.
<div className="text-[10px] text-zinc-500 mt-0.5"> `amount_aud ?? amount` formatted with the default currency
rendered a USD 10,782.00 Wise credit as "+$10,782.00" in a
column of AUD figures — the real value was about $15,500,
and the row directly above it (the same payment from the
statement) showed exactly that. The native figure is the
only true one here, so it becomes the headline and the
absence of a rate is stated rather than papered over.
*/}
{t.amount_unconverted ? (
<>
{formatAmount(t.amount, t.transaction_type, t.currency)} {formatAmount(t.amount, t.transaction_type, t.currency)}
</div> <div className="text-[10px] text-zinc-500 mt-0.5" title="No AUD figure exists for this row — it is excluded from AUD totals">
no AUD rate
</div>
</>
) : (
<>
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
{t.currency && t.currency !== "AUD" && (
<div className="text-[10px] text-zinc-500 mt-0.5">
{formatAmount(t.amount, t.transaction_type, t.currency)}
</div>
)}
</>
)} )}
</td> </td>
<td className="p-2"> <td className="p-2">
+71 -6
View File
@@ -2,10 +2,10 @@
import { useState, useRef, useEffect } from "react"; import { useState, useRef, useEffect } from "react";
import { CATEGORIES, formatCategory } from "@/lib/categories"; import { CATEGORIES, formatCategory } from "@/lib/categories";
import { useImportCSV } from "@/lib/hooks"; import { useImportCSV, useStatementCoverage } from "@/lib/hooks";
import { import {
parseCSVRows, detectHasHeaders, getColumnLabels, getDataRows, applyMapping, parseCSVRows, detectHasHeaders, getColumnLabels, getDataRows, applyMapping,
saveBankPreset, loadBankPresets, saveBankPreset, loadBankPresets, splitByCoverage, inFileDuplicates,
type DateFormat, type ColumnMapping, type ParsedTransaction, type BankPreset, type DateFormat, type ColumnMapping, type ParsedTransaction, type BankPreset,
} from "@/lib/csv-parser"; } from "@/lib/csv-parser";
@@ -14,6 +14,8 @@ const TX_TYPES = ["debit", "credit", "payment", "refund", "fee", "interest", "tr
type Step = "upload" | "map" | "review" | "done"; type Step = "upload" | "map" | "review" | "done";
function ColSelect({ function ColSelect({
label, value, onChange, options, required, label, value, onChange, options, required,
}: { }: {
@@ -54,6 +56,9 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
const [error, setError] = useState(""); const [error, setError] = useState("");
const [presets, setPresets] = useState<BankPreset[]>([]); const [presets, setPresets] = useState<BankPreset[]>([]);
const [insertedCount, setInsertedCount] = useState(0); const [insertedCount, setInsertedCount] = useState(0);
const [coveredRows, setCoveredRows] = useState<ParsedTransaction[]>([]);
const [showCovered, setShowCovered] = useState(false);
const coverage = useStatementCoverage();
useEffect(() => { setPresets(loadBankPresets()); }, []); useEffect(() => { setPresets(loadBankPresets()); }, []);
@@ -111,7 +116,9 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
if (savePreset) { if (savePreset) {
saveBankPreset({ bankName: bankName.trim(), mapping, dateFormat }); saveBankPreset({ bankName: bankName.trim(), mapping, dateFormat });
} }
setEditedRows(parsed); const { keep, covered } = splitByCoverage(parsed, coverage.data ?? []);
setEditedRows(keep);
setCoveredRows(covered);
setStep("review"); setStep("review");
} }
@@ -120,7 +127,13 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
const valid = editedRows.filter((r) => r.date && r.amount > 0 && r.description); const valid = editedRows.filter((r) => r.date && r.amount > 0 && r.description);
if (!valid.length) { setError("No valid rows to import"); return; } if (!valid.length) { setError("No valid rows to import"); return; }
try { try {
const result = await importCSV.mutateAsync({ bank_name: bankName, transactions: valid }); const result = await importCSV.mutateAsync({
bank_name: bankName,
// Provenance, so a second import of the same file inserts nothing —
// paired with each row's source_ref where the file carries an id.
source: bankName.trim().toLowerCase().replace(/\s+/g, "-"),
transactions: valid,
});
setInsertedCount(result.inserted); setInsertedCount(result.inserted);
setStep("done"); setStep("done");
} catch (e) { } catch (e) {
@@ -286,7 +299,16 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
<div className="grid grid-cols-2 gap-3"> <div className="grid grid-cols-2 gap-3">
<ColSelect label="Merchant Column (optional)" value={mapping.merchantCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, merchantCol: v || undefined }))} options={columnLabels} /> <ColSelect label="Merchant Column (optional)" value={mapping.merchantCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, merchantCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Category Column (optional)" value={mapping.categoryCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, categoryCol: v || undefined }))} options={columnLabels} /> <ColSelect label="Category Column (optional)" value={mapping.categoryCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, categoryCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Account Column (optional)" value={mapping.accountCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, accountCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Currency Column (optional)" value={mapping.currencyCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, currencyCol: v || undefined }))} options={columnLabels} />
<ColSelect label="Row ID Column (optional)" value={mapping.sourceRefCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, sourceRefCol: v || undefined }))} options={columnLabels} />
</div> </div>
<p className="text-[11px] text-zinc-500 leading-relaxed">
Map <b>Account</b> when the file covers more than one account: rows already
covered by that account&apos;s statements are then left out. Map <b>Row ID</b> to
make re-importing the same file do nothing. Map <b>Currency</b> for a
multi-currency file, or foreign rows are stored as if they were AUD.
</p>
<label className="flex items-center gap-2 text-sm cursor-pointer text-zinc-400"> <label className="flex items-center gap-2 text-sm cursor-pointer text-zinc-400">
<input type="checkbox" checked={savePreset} onChange={(e) => setSavePreset(e.target.checked)} className="accent-indigo-500" /> <input type="checkbox" checked={savePreset} onChange={(e) => setSavePreset(e.target.checked)} className="accent-indigo-500" />
@@ -300,9 +322,52 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
{/* Step 3: Review */} {/* Step 3: Review */}
{step === "review" && ( {step === "review" && (
<div> <div>
<p className="text-xs text-zinc-500 mb-3"> <p className="text-xs text-zinc-500 mb-2">
{editedRows.length} transactions parsed. Edit or remove rows before importing. {editedRows.length} transactions to import. Edit or remove rows before importing.
</p> </p>
{/*
Reported, never silent. A row that disappears can mean several
things and one of them is worth following up, so the count is
always shown and the rows themselves are one click away.
*/}
{coveredRows.length > 0 && (
<div className="mb-3 rounded border border-zinc-800 bg-zinc-800/30 px-3 py-2">
<button
type="button"
onClick={() => setShowCovered((v) => !v)}
className="text-xs text-zinc-300 hover:text-white text-left w-full"
>
{coveredRows.length} row{coveredRows.length === 1 ? "" : "s"} left out already covered by
a statement for that account.{" "}
<span className="text-indigo-400">{showCovered ? "hide" : "show"}</span>
</button>
{showCovered && (
<>
<ul className="mt-2 max-h-40 overflow-y-auto text-[11px] text-zinc-500 font-mono space-y-0.5">
{coveredRows.slice(0, 200).map((r, i) => (
<li key={i}>{r.date} {r.amount.toFixed(2).padStart(10)} {r.description.slice(0, 44)}</li>
))}
</ul>
{coveredRows.length > 200 && (
<p className="mt-1 text-[11px] text-zinc-600">and {coveredRows.length - 200} more.</p>
)}
<p className="mt-2 text-[11px] text-zinc-500">
Statements cover:{" "}
{(coverage.data ?? []).map((c) => `${c.last4} to ${c.coveredTo}`).join(" · ")}
</p>
</>
)}
</div>
)}
{inFileDuplicates(editedRows) > 0 && (
<p className="mb-3 rounded border border-amber-900/50 bg-amber-950/30 px-3 py-2 text-xs text-amber-300">
{inFileDuplicates(editedRows)} row{inFileDuplicates(editedRows) === 1 ? " duplicates another row" : "s duplicate other rows"} in
this same file (same date, amount and description). A re-consent can make an
aggregator re-export history under new ids check before importing.
</p>
)}
<div className="overflow-x-auto rounded border border-zinc-800"> <div className="overflow-x-auto rounded border border-zinc-800">
<table className="w-full text-xs"> <table className="w-full text-xs">
<thead className="border-b border-zinc-800"> <thead className="border-b border-zinc-800">
+6 -1
View File
@@ -72,7 +72,12 @@ export function OrderDetails({
const { data: review } = useOrderReview(transactionId); const { data: review } = useOrderReview(transactionId);
const [reviewer, setReviewer] = useState(OWNER_PARTICIPANT_ID); const [reviewer, setReviewer] = useState(OWNER_PARTICIPANT_ID);
if (isLoading || !receipt) return null; // `platform` is the marker of a real receipt. Since phase 2 the endpoint also
// answers with `{ order_link }` alone for a transaction that is merely LINKED
// to an order (a BNPL leg has no receipt of its own), and that object is
// truthy — without this check the panel would render a receipt shell with no
// merchant, no items and no total.
if (isLoading || !receipt || !receipt.platform) return null;
const cur = receipt.currency ?? currency ?? "AUD"; const cur = receipt.currency ?? currency ?? "AUD";
const fmt = (n: number) => (cur === "AUD" ? `$${n.toFixed(2)}` : `${cur} ${n.toFixed(2)}`); const fmt = (n: number) => (cur === "AUD" ? `$${n.toFixed(2)}` : `${cur} ${n.toFixed(2)}`);
+101
View File
@@ -9,6 +9,32 @@ export interface ColumnMapping {
creditCol?: string; creditCol?: string;
merchantCol?: string; merchantCol?: string;
categoryCol?: string; categoryCol?: string;
/**
* Column holding the account number, when the file covers more than one.
*
* Optional and usually absent — a bank's own export is one account per file.
* An aggregator export (Frollo) is not, and without this every row would be
* measured against the same statement coverage.
*/
accountCol?: string;
/**
* Column holding the ISO currency code, when the file is multi-currency.
*
* Without it a foreign row is stored as if it were AUD — a USD 10,782 salary
* lands as A$10,782, understating it by about a third. Mapping it records the
* native figure with `foreign_currency_code` set and `amount_aud` left NULL,
* which is the same shape order ingestion uses and what `AMOUNT_UNCONVERTED`
* looks for.
*/
currencyCol?: string;
/**
* Column holding the provider's own row id, when it has one.
*
* Stored as `source_ref` so re-importing the same file inserts nothing. The
* importer has no other idempotency: `row_index` is assigned MAX+1 on every
* run, which makes `uq_transaction_identity` structurally unable to fire.
*/
sourceRefCol?: string;
} }
export interface BankPreset { export interface BankPreset {
@@ -24,6 +50,13 @@ export interface ParsedTransaction {
transaction_type: string; transaction_type: string;
merchant_name?: string; merchant_name?: string;
category?: string; category?: string;
/** Raw account identifier from the file, when a column was mapped. */
account?: string;
/** Provider row id, when a column was mapped. */
source_ref?: string;
/** Set only for a non-AUD row: the native amount and its code. */
foreign_currency_amount?: number;
foreign_currency_code?: string;
} }
export function parseCSVRows(text: string): string[][] { export function parseCSVRows(text: string): string[][] {
@@ -108,6 +141,9 @@ export function applyMapping(
const descIdx = idx(mapping.descriptionCol); const descIdx = idx(mapping.descriptionCol);
const merchantIdx = mapping.merchantCol ? idx(mapping.merchantCol) : -1; const merchantIdx = mapping.merchantCol ? idx(mapping.merchantCol) : -1;
const categoryIdx = mapping.categoryCol ? idx(mapping.categoryCol) : -1; const categoryIdx = mapping.categoryCol ? idx(mapping.categoryCol) : -1;
const accountIdx = mapping.accountCol ? idx(mapping.accountCol) : -1;
const sourceRefIdx = mapping.sourceRefCol ? idx(mapping.sourceRefCol) : -1;
const currencyIdx = mapping.currencyCol ? idx(mapping.currencyCol) : -1;
const results: ParsedTransaction[] = []; const results: ParsedTransaction[] = [];
for (const row of dataRows) { for (const row of dataRows) {
@@ -138,6 +174,17 @@ export function applyMapping(
const tx: ParsedTransaction = { date, description, amount, transaction_type }; const tx: ParsedTransaction = { date, description, amount, transaction_type };
if (merchantIdx >= 0 && row[merchantIdx]) tx.merchant_name = row[merchantIdx].trim(); if (merchantIdx >= 0 && row[merchantIdx]) tx.merchant_name = row[merchantIdx].trim();
if (categoryIdx >= 0 && row[categoryIdx]) tx.category = row[categoryIdx].trim(); if (categoryIdx >= 0 && row[categoryIdx]) tx.category = row[categoryIdx].trim();
if (accountIdx >= 0 && row[accountIdx]) tx.account = row[accountIdx].trim();
if (sourceRefIdx >= 0 && row[sourceRefIdx]) tx.source_ref = row[sourceRefIdx].trim();
if (currencyIdx >= 0) {
const code = (row[currencyIdx] ?? "").trim().toUpperCase();
// AUD needs no marking; anything else is native and unconverted. No rate
// is invented — the statement supplies the AUD figure when it arrives.
if (/^[A-Z]{3}$/.test(code) && code !== "AUD") {
tx.foreign_currency_code = code;
tx.foreign_currency_amount = amount;
}
}
results.push(tx); results.push(tx);
} }
return results; return results;
@@ -154,3 +201,57 @@ export function loadBankPresets(): BankPreset[] {
try { return JSON.parse(localStorage.getItem("csv-presets") || "[]"); } try { return JSON.parse(localStorage.getItem("csv-presets") || "[]"); }
catch { return []; } catch { return []; }
} }
/** Last four digits of an account identifier, however it is written. */
export function last4(s: string): string {
const d = s.replace(/\D/g, "");
return d.length >= 4 ? d.slice(-4) : "";
}
/**
* Splits parsed rows into what to import and what the statements already have.
*
* A statement's end date is a hard watermark: everything on that account up to
* it is already in the ledger, itemised and converted. Without this the first
* aggregator import wrote 422 duplicates out of 550 rows; with it the same
* 2,607-row export offers 171.
*
* Rows are only excluded when an account column was mapped AND that account has
* statements. Anything else is imported — a row is never dropped on a guess.
*/
export function splitByCoverage(
rows: ParsedTransaction[],
coverage: { last4: string; coveredTo: string }[]
): { keep: ParsedTransaction[]; covered: ParsedTransaction[] } {
const wm = new Map(coverage.map((c) => [c.last4, c.coveredTo]));
const keep: ParsedTransaction[] = [];
const covered: ParsedTransaction[] = [];
for (const r of rows) {
const to = r.account ? wm.get(last4(r.account)) : undefined;
// Both are YYYY-MM-DD, so a string compare is a date compare.
if (to && r.date <= to) covered.push(r);
else keep.push(r);
}
return { keep, covered };
}
/**
* Rows that duplicate another row in the SAME file.
*
* Distinct from the coverage check, and not catchable by `source_ref`: a CDR
* re-consent makes the provider re-export an account's whole history under
* fresh ids while the originals survive, so the twins are identical in
* everything except the id. That put 385 duplicates in one 2,563-row export and
* doubled every salary payment. 385 rows is not something you spot by eye in a
* review table, so it is counted here.
*/
export function inFileDuplicates(rows: ParsedTransaction[]): number {
const seen = new Set<string>();
let n = 0;
for (const r of rows) {
const key = [r.date, r.description, r.amount, r.transaction_type, r.account ?? ""].join("|");
if (seen.has(key)) n += 1;
else seen.add(key);
}
return n;
}
+30 -2
View File
@@ -1053,9 +1053,11 @@ export function useImportCSV() {
return useMutation({ return useMutation({
mutationFn: async (body: { mutationFn: async (body: {
bank_name: string; bank_name: string;
source?: string;
transactions: { transactions: {
date: string; description: string; amount: number; transaction_type: string; date: string; description: string; amount: number; transaction_type: string;
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string; category?: string; merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string;
category?: string; account?: string; source_ref?: string;
}[]; }[];
}) => { }) => {
const res = await fetch("/api/import/csv", { const res = await fetch("/api/import/csv", {
@@ -1243,8 +1245,34 @@ export function useOrderDetail(entityKey: string | null) {
staleTime: 60_000, staleTime: 60_000,
queryFn: async () => { queryFn: async () => {
const res = await fetch(`/api/orders/${encodeURIComponent(entityKey!)}`); const res = await fetch(`/api/orders/${encodeURIComponent(entityKey!)}`);
if (!res.ok) throw new Error("Failed to load order"); if (!res.ok) {
// Carry the status. Collapsing every failure into one Error is how a
// 500 on this route rendered as "that order could not be found" on
// every single order for weeks — a server fault wearing the costume of
// a data condition, which reads as "the spine is empty" and gets
// investigated nowhere near the actual bug.
const err = new Error(res.status === 404 ? "Order not found" : "Failed to load order");
(err as Error & { status?: number }).status = res.status;
throw err;
}
return res.json(); return res.json();
}, },
}); });
} }
/**
* How far each account's statements already reach, for the CSV import modal.
*
* Fetched once when the modal opens: the review step uses it to drop rows the
* ledger already holds rather than offering them for import.
*/
export function useStatementCoverage() {
return useQuery({
queryKey: ["statement-coverage"],
queryFn: async () => {
const res = await fetch("/api/import/statement-coverage");
if (!res.ok) throw new Error("Could not load statement coverage");
return (await res.json()).coverage as { last4: string; coveredTo: string }[];
},
});
}
+51
View File
@@ -0,0 +1,51 @@
/**
* Presentational tidy for a merchant name (board 212).
*
* PRESENTATION ONLY. It merges nothing and must never be used to decide that
* two rows are the same merchant. Real unification is the merchant-alias bridge
* (ticket 176, `jobs/merchant_merge.py`), which has already merged what can be
* merged safely; what is left on screen is one entity looking scruffy, not two
* entities that should be one.
*
* `Amazon.in` MUST NOT become `amazon.com.au`. They are different marketplaces
* with different currency and geography — a genuine country distinction, not a
* name variant. `Kogan.com`, `GOG.com`, `AliExpress.com` and `Catch.com.au` are
* real brand names that happen to contain a TLD and are already correct.
*
* Two rules, both mechanical:
* 1. drop trailing corporate suffixes
* 2. capitalise a name that arrived lower-cased FROM A DOMAIN
*/
const CORP_SUFFIX =
/,?\s+(pty\.?\s+ltd\.?|pty\.?\s+limited|pte\.?\s+ltd\.?|pte\.?|p\/l|ltd\.?|limited|inc\.?|llc|pbc|gmbh|b\.?v\.?|s\.?a\.?r\.?l\.?|oü|co\.?)$/i;
/** A bare domain used as a name: all lower case, and a dotted TLD. */
const BARE_DOMAIN = /^[a-z0-9][a-z0-9.-]*\.[a-z]{2,}(\.[a-z]{2,})?$/;
/**
* Stripping must not leave a dangling connector. `Coburger & Co` is a brand
* whose last word happens to match the suffix list, and removing it produced
* `Coburger &` — visibly broken, and the kind of thing that only shows up when
* you run the rule over the real corpus rather than over examples you chose.
*/
const DANGLING = /(\s[&+-]|\sand)$/i;
export function tidyMerchant(name: string): string {
let n = name.replace(/\s+/g, " ").trim();
// At most two passes: "Samsung Electronics Co. Ltd." needs Ltd. then Co.
for (let i = 0; i < 2; i++) {
const stripped = n.replace(CORP_SUFFIX, "").trim().replace(/,$/, "");
if (stripped === n || stripped.length < 3 || DANGLING.test(stripped)) break;
n = stripped;
}
// "amazon.com.au" reads as a machine artefact; "Amazon.com.au" reads as a
// name. Gated on BARE_DOMAIN, not on "starts with a lower-case letter": the
// looser test turned `eBay Commerce Australia Pty Ltd.` into `EBay Commerce
// Australia`, mangling a brand that is deliberately lower-cased.
if (BARE_DOMAIN.test(n)) n = n[0].toUpperCase() + n.slice(1);
return n;
}
+125 -22
View File
@@ -73,6 +73,18 @@ export interface OrderRow {
/** Merchant orders on a regular cadence: the subscription signal, derived /** Merchant orders on a regular cadence: the subscription signal, derived
* not extracted. NULL unless the gaps are tight relative to their mean. */ * not extracted. NULL unless the gaps are tight relative to their mean. */
cadence_days: number | null; cadence_days: number | null;
/** Worst sender-authentication verdict across the order's source documents.
* 93% are 'pass' — the badge exists for the 7% that are not. Do NOT badge
* source_trust instead: it is 'untrusted_external' on 100% of rows, because
* every order here came from email. */
auth_verdict: string | null;
injection_flagged: boolean | null;
/** An annual tax/donation receipt restating a year of payments already made,
* not a purchase (board 211). It stays on this page — the figure is the
* year's giving total and is worth seeing — but `order_spend` excludes it,
* so the amount here is deliberately NOT part of any total on the page.
* Badge it, or a $2,376 row reads as a single purchase. */
summarises_period: boolean;
txn_count: number; txn_count: number;
first_txn_id: number | null; first_txn_id: number | null;
} }
@@ -96,9 +108,22 @@ export interface OrderFilters {
/** /**
* Rows with no amount, no order reference and a single lifecycle event are not * Rows with no amount, no order reference and a single lifecycle event are not
* purchases — they are a second entity minted from a mail describing an order * purchases — they are a second entity minted from a mail describing an order
* that already exists (Amazon "share your experience", DoorDash no-contact * that already exists (DoorDash no-contact delivery details, shipment notices).
* delivery details, shipment notices). 755 of them; hiding lifts amount * 620 of them as of 2026-08-12.
* coverage from 74% to 84%. *
* The count moved three times that day and the history is the point:
* 755 when this predicate was written
* 904 after board 219 turned unstated zeros into NULLs — those 149 rows had
* been failing the "no amount" test on a technicality
* 699 after board 210a retired 205 Amazon "share your experience" mails,
* which were never lifecycle events at all (ingest/order_noise.py)
* 620 after board 210b joined 79 phantoms to their real parent
* (jobs/phantom_order_joiner.py, placebo-verified at a 1.9% error rate)
*
* What is left is the corpus coverage ceiling rather than a defect: a mail
* whose parent is not in the corpus, or whose merchant is ambiguous, has
* nothing to join to. This stays a UI workaround for those, and the toggle
* still says so.
* *
* This is a WORKAROUND for board 210, not a fix. The upstream defect is that a * This is a WORKAROUND for board 210, not a fix. The upstream defect is that a
* lifecycle mail printing no order reference cannot join its own order, so it * lifecycle mail printing no order reference cannot join its own order, so it
@@ -108,6 +133,14 @@ export interface OrderFilters {
const LIFECYCLE_ONLY = `NOT ( const LIFECYCLE_ONLY = `NOT (
f.order_total IS NULL f.order_total IS NULL
AND f.reference_source = 'message_id_fallback' AND f.reference_source = 'message_id_fallback'
-- TRIED AND REVERTED, 2026-08-12: exempting rows that carry line items, on
-- the theory that naming what was bought proves a purchase. It surfaced 442
-- rows and the very first window showed why it is wrong — the Amazon
-- "Ordered:" mail that mints a msg- phantom carries the item list too, so
-- Xelsluthe and YIWENTEC each rendered TWICE again, which is the exact
-- complaint this page was fixed for. Items do not separate an orphan
-- purchase from an item-carrying echo; only real deduplication does, and
-- that is board 210, deliberately not attempted here.
AND (SELECT count(*) FROM extracted_facts ef AND (SELECT count(*) FROM extracted_facts ef
WHERE ef.fact_type = 'order_event' WHERE ef.fact_type = 'order_event'
AND ef.payload->>'_order_entity_key' = f.entity_key) <= 1 AND ef.payload->>'_order_entity_key' = f.entity_key) <= 1
@@ -180,18 +213,41 @@ function buildWhere(filters: OrderFilters) {
} }
/** /**
* Ledger linkage. Phase 1 reads expense_metadata; phase 2 swaps this lateral * Ledger linkage — phase 2 (board 205).
* AND the detail query AND /api/transactions/[id]/order together — moving only *
* one of the three leaves the list contradicting the detail page. * `order_transaction_links` is the source of truth: it is many-to-many, so a
* BNPL plan is four rows against one order. `expense_metadata` remains as a
* FALLBACK because `jobs/order_transaction_bridge.py` still writes there on
* every hourly tick; until a bridge row is mirrored into links, dropping the
* fallback would make freshly bridged orders read as unpaid.
*
* UNION, not UNION ALL, on transaction_id: after the 0030 backfill the same
* charge is legitimately present in both stores, and counting it twice would
* show "2 charges" on a single-payment order.
*
* All four readers move together — this lateral, the detail query below,
* /api/transactions/[id]/order, and the order_ctx lateral in queries.ts.
* Moving fewer is not a smaller change, it is an inconsistent one: once the
* matcher inserts four links for order_ebay_14-11714-95953 that order still
* has no expense_metadata row, so the detail page would show four instalments
* while the list showed txn_count = 0 and the has_transaction facet put it on
* the wrong side of both filters.
*/ */
const LINK_LATERAL = ` const LINK_LATERAL = `
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
SELECT count(*)::int AS txn_count, SELECT count(*)::int AS txn_count, min(txn_id) AS first_txn_id
min(em.matched_transaction_id) AS first_txn_id FROM (
FROM expense_metadata em SELECT l.transaction_id AS txn_id
WHERE em.source = 'order-bridge' FROM order_transaction_links l
AND (em.source_message_id = f.entity_key WHERE l.entity_key = f.entity_key
OR em.source_message_id LIKE f.entity_key || '#f%') UNION
SELECT COALESCE(em.matched_transaction_id, em.transaction_id)
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND (em.source_message_id = f.entity_key
OR em.source_message_id LIKE f.entity_key || '#f%')
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
) both_stores
) link ON true`; ) link ON true`;
const FROM_CLAUSE = ` const FROM_CLAUSE = `
@@ -220,6 +276,7 @@ export async function getOrderFeed(filters: OrderFilters) {
f.ordered_at, f.eta_date, f.delivered_at, f.ordered_at, f.eta_date, f.delivered_at,
f.currency, f.order_total, f.line_item_count, f.currency, f.order_total, f.line_item_count,
f.content_class, f.display_name, f.cadence_days, f.content_class, f.display_name, f.cadence_days,
f.auth_verdict, f.injection_flagged, f.summarises_period,
m.canonical_name AS merchant_name, m.canonical_name AS merchant_name,
link.txn_count, link.first_txn_id, link.txn_count, link.first_txn_id,
(SELECT sp.refunded_amount FROM order_spend sp (SELECT sp.refunded_amount FROM order_spend sp
@@ -334,6 +391,12 @@ export interface OrderLinkedTxn {
description: string; description: string;
amount: string; amount: string;
source_message_id: string; source_message_id: string;
/** 'charge' | 'shipment' | 'instalment' | 'refund' | 'fee'. A plan's legs are
* four 'instalment' rows; a whole-order card charge is one 'charge'. */
leg_kind: string;
/** Human position, "2 of 4" — populated for instalments only. */
leg_index: number | null;
leg_count: number | null;
} }
export interface OrderDetail { export interface OrderDetail {
@@ -358,6 +421,9 @@ export interface OrderDetail {
tracking_carrier: string | null; tracking_carrier: string | null;
line_items: { description?: string; quantity?: number; amount?: number }[]; line_items: { description?: string; quantity?: number; amount?: number }[];
is_settled_duplicate: boolean; is_settled_duplicate: boolean;
/** Board 211 — an annual receipt restating a year of payments already made.
* Same shape as is_settled_duplicate: browsable, excluded from spend. */
summarises_period: boolean;
content_class: string | null; content_class: string | null;
display_name: string; display_name: string;
cadence_days: number | null; cadence_days: number | null;
@@ -385,6 +451,7 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
o.order_total, o.tracking_url, o.tracking_carrier, o.order_total, o.tracking_url, o.tracking_carrier,
m.canonical_name AS merchant_name, m.canonical_name AS merchant_name,
(o.settles_entity_id IS NOT NULL) AS is_settled_duplicate, (o.settles_entity_id IS NOT NULL) AS is_settled_duplicate,
o.summarises_period,
COALESCE(o.details->'line_items', '[]'::jsonb) AS line_items, COALESCE(o.details->'line_items', '[]'::jsonb) AS line_items,
sp.order_total AS net_total, sp.order_total AS net_total,
sp.gross_total, sp.gross_total,
@@ -392,14 +459,29 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
ctx.content_class, ctx.content_class,
COALESCE(me2.canonical_name, ctx.counterparty, o.platform) AS display_name, COALESCE(me2.canonical_name, ctx.counterparty, o.platform) AS display_name,
rec.cadence_days, rec.cadence_days,
rec.order_count -- ::int is mandatory. order_merchant_cadence.order_count is a
-- count(*), so BIGINT, and JSON.stringify throws outright on a
-- BigInt — "Do not know how to serialize a BigInt", a 500 with no
-- SQL in it. It only fires on merchants that HAVE a cadence, so
-- most orders open fine and the recurring ones die; testing one
-- arbitrary order will not find it.
rec.order_count::int AS order_count
FROM entities e FROM entities e
JOIN entity_orders o ON o.entity_id = e.id JOIN entity_orders o ON o.entity_id = e.id
LEFT JOIN entities m ON m.id = o.merchant_entity_id LEFT JOIN entities m ON m.id = o.merchant_entity_id
LEFT JOIN entities me2 ON me2.id = o.merchant_entity_id LEFT JOIN entities me2 ON me2.id = o.merchant_entity_id
LEFT JOIN order_platforms p ON p.slug = o.platform LEFT JOIN order_platforms p ON p.slug = o.platform
LEFT JOIN order_spend sp ON sp.entity_id = o.entity_id LEFT JOIN order_spend sp ON sp.entity_id = o.entity_id
LEFT JOIN order_merchant_cadence rec ON rec.merchant_entity_id = o.merchant_entity_id -- Cadence is keyed on the NORMALISED MERCHANT NAME, not merchant_entity_id.
-- Migration 021 rekeyed the view (that column is NULL on ~9% of the feed
-- and duplicate merchant entities split the same shop's history); 027 and
-- 029 moved order_feed/order_spend onto merchant_key, and this query was
-- missed. The result was a 42703 on EVERY order detail page — the join
-- referenced a column the view no longer had. Keep this expression
-- identical to the one in migration 027.
LEFT JOIN order_merchant_cadence rec
ON rec.merchant_key = regexp_replace(
lower(COALESCE(me2.canonical_name, o.platform)), '[^a-z0-9]', '', 'g')
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
SELECT di.content_class, di.counterparty SELECT di.content_class, di.counterparty
FROM extracted_facts ef FROM extracted_facts ef
@@ -445,14 +527,35 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
); );
const transactions = await queryRaw<OrderLinkedTxn>( const transactions = await queryRaw<OrderLinkedTxn>(
`SELECT t.id::int AS transaction_id, t.transaction_date, t.description, // Links first, expense_metadata as fallback, deduped on transaction_id —
t.amount, em.source_message_id // see LINK_LATERAL above for why both stores are read. DISTINCT ON keeps
FROM expense_metadata em // the LINK row when a transaction is in both, because only that row knows
JOIN transactions t // whether the payment was one charge or leg 3 of 4.
ON t.id = COALESCE(em.matched_transaction_id, em.transaction_id) // The DISTINCT ON must be ordered by t.id, so the chronological sort the
WHERE em.source = 'order-bridge' // page needs goes on the OUTER query — legs read as "1, 2, 3, 4" only if
AND (em.source_message_id = $1 OR em.source_message_id LIKE $1 || '#f%') // they come back by date.
ORDER BY t.transaction_date`, `SELECT * FROM (
SELECT DISTINCT ON (t.id)
t.id::int AS transaction_id, t.transaction_date, t.description,
t.amount, both_stores.source_message_id,
both_stores.leg_kind, both_stores.leg_index, both_stores.leg_count
FROM (
SELECT l.transaction_id, l.leg_kind, l.leg_index::int, l.leg_count::int,
l.entity_key AS source_message_id, 0 AS pref
FROM order_transaction_links l
WHERE l.entity_key = $1
UNION ALL
SELECT COALESCE(em.matched_transaction_id, em.transaction_id),
'charge', NULL::int, NULL::int, em.source_message_id, 1
FROM expense_metadata em
WHERE em.source = 'order-bridge'
AND (em.source_message_id = $1 OR em.source_message_id LIKE $1 || '#f%')
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
) both_stores
JOIN transactions t ON t.id = both_stores.transaction_id
ORDER BY t.id, both_stores.pref
) deduped
ORDER BY transaction_date, leg_index NULLS FIRST`,
[entityKey] [entityKey]
); );
+118 -5
View File
@@ -37,7 +37,16 @@ export interface TransactionRow {
payment_method: string | null; payment_method: string | null;
/** Uber pick-up/drop-off, when this row came from an order receipt. */ /** Uber pick-up/drop-off, when this row came from an order receipt. */
order_route: RoutePointRow[] | null; order_route: RoutePointRow[] | null;
/** Receipt platform ONLY — it gates the disclosure arrow, so it must stay
* true to "there is a receipt behind this row". */
order_platform: "doordash" | "ubereats" | "uber" | null; order_platform: "doordash" | "ubereats" | "uber" | null;
/** Phase 2 (board 205) — set when this transaction is linked to an order.
* 'instalment' means it is one leg of a plan; leg_index/leg_count carry
* "2 of 4", which is what stops one purchase reading as four. */
order_leg_kind: string | null;
order_leg_index: number | null;
order_leg_count: number | null;
order_name: string | null;
// override fields // override fields
category_override: string | null; category_override: string | null;
merchant_override: string | null; merchant_override: string | null;
@@ -314,7 +323,17 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
txn_tags.tags, txn_tags.tags,
txn_splits.splits, txn_splits.splits,
order_ctx.route as order_route, order_ctx.route as order_route,
order_ctx.platform as order_platform -- Deliberately NOT COALESCEd with the link's platform. order_platform
-- gates the receipt disclosure arrow on /transactions, and a BNPL leg
-- has no receipt behind it — filling this in put an arrow on four
-- Afterpay rows that expand to nothing, which is the exact promise the
-- arrow exists to avoid making. The leg fields below carry the sub-line
-- instead.
order_ctx.platform as order_platform,
order_link.leg_kind as order_leg_kind,
order_link.leg_index as order_leg_index,
order_link.leg_count as order_leg_count,
order_link.canonical_name as order_name
FROM transactions t FROM transactions t
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
LEFT JOIN statements s ON s.id = t.statement_id LEFT JOIN statements s ON s.id = t.statement_id
@@ -323,12 +342,27 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
-- only thing that tells them apart, and it was already stored. -- only thing that tells them apart, and it was already stored.
-- Both directions, because a card-settled order has no transaction of its -- Both directions, because a card-settled order has no transaction of its
-- own and points at the statement line instead (I5). -- own and points at the statement line instead (I5).
-- Phase 2 (board 205): a BNPL leg has no expense_metadata row of its own,
-- so COALESCE in the link's platform. Without it the four Afterpay debits
-- behind the A$1,599 drone keep an empty sub-line while their order sits
-- one join away. The route column only ever exists on the receipt side.
-- (No backticks in here: this SQL lives in a TS template literal and a
-- backtick ends the string — TS1005 on a line that looks like a comment.)
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
SELECT em.route, em.platform SELECT em.route, em.platform
FROM expense_metadata em FROM expense_metadata em
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
LIMIT 1 LIMIT 1
) order_ctx ON true ) order_ctx ON true
LEFT JOIN LATERAL (
SELECT o.platform, l.leg_kind, l.leg_index::int AS leg_index,
l.leg_count::int AS leg_count, e.canonical_name
FROM order_transaction_links l
LEFT JOIN entities e ON e.entity_key = l.entity_key
LEFT JOIN entity_orders o ON o.entity_id = e.id
WHERE l.transaction_id = t.id
LIMIT 1
) order_link ON true
LEFT JOIN participants p ON p.id = COALESCE(t.owner_id, s.owner_id) LEFT JOIN participants p ON p.id = COALESCE(t.owner_id, s.owner_id)
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
LEFT JOIN trips tr ON tr.id = o.trip_id LEFT JOIN trips tr ON tr.id = o.trip_id
@@ -636,8 +670,11 @@ export async function batchInsertCSVTransactions(
foreign_currency_amount?: number; foreign_currency_amount?: number;
foreign_currency_code?: string; foreign_currency_code?: string;
category?: string; category?: string;
account?: string;
source_ref?: string;
}[], }[],
tagId: number tagId: number,
source?: string
): Promise<number> { ): Promise<number> {
if (rows.length === 0) return 0; if (rows.length === 0) return 0;
@@ -651,13 +688,32 @@ export async function batchInsertCSVTransactions(
const params: unknown[] = [ownerId]; const params: unknown[] = [ownerId];
let p = 2; let p = 2;
rows.forEach((r, i) => { rows.forEach((r, i) => {
valueClauses.push(`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`); valueClauses.push(
params.push(r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null, r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null); `(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`
);
params.push(
r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null,
r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null,
// The category the reviewer chose. It used to be accepted here and then
// left out of the column list, so every category picked in the review
// step was silently discarded and the trigger wrote 'other'. That is not
// cosmetic: an uncategorised credit is admitted by NET_SPEND_ROWS and
// negated by SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of
// spend and none of them counted as income.
r.category ?? null,
source ?? null,
// source + source_ref is the only idempotency this path has. Without it a
// second import of the same file duplicates every row, because row_index
// is assigned MAX+1 and uq_transaction_identity can never fire.
r.source_ref ?? null,
r.account ?? null
);
}); });
const txIds = await queryRaw<{ id: number }>( const txIds = await queryRaw<{ id: number }>(
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, row_index) `INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, category, source, source_ref, source_account, row_index)
VALUES ${valueClauses.join(", ")} VALUES ${valueClauses.join(", ")}
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL DO NOTHING
RETURNING id`, RETURNING id`,
params params
); );
@@ -698,6 +754,27 @@ export async function batchInsertCSVTransactions(
export const needsCardMatch = (alias = "t") => export const needsCardMatch = (alias = "t") =>
`(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`; `(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`;
/**
* REMOVED 2026-08-13. There used to be an `awaitsStatementLine()` predicate
* here, excluding account-feed rows from the pending-reconciliation queue.
*
* Its premise — a feed row is the account's own ledger entry, so no statement
* line is coming — was asserted and never tested. It was false for almost every
* account: 422 of the first 550 imported rows already had a statement twin.
*
* How it got in matters more than what it did. The queue jumped from 8 to 558
* the moment the feed landed, and that jump was read as noise and filtered
* away. The queue was right — those rows genuinely were provisional entries
* awaiting statement lines — and hiding them removed the only mechanism that
* would ever have collapsed them, so a transient overlap became a permanent
* double-count and stayed invisible until a human saw one salary payment listed
* twice in two currencies.
*
* An imported row that has no statement belongs in the queue. Volume is solved
* upstream, by not importing what the statements already cover
* (`getStatementCoverage`), never downstream by hiding what is there.
*/
/** /**
* Bank label for a transaction. A row with no statement was not imported from * Bank label for a transaction. A row with no statement was not imported from
* one, and the label has to say *why*: "Manual" reads as "hand-entered, still * one, and the label has to say *why*: "Manual" reads as "hand-entered, still
@@ -1467,3 +1544,39 @@ export async function getTagTransactionIds(tagId: number): Promise<number[]> {
); );
return rows.map((r) => r.transaction_id); return rows.map((r) => r.transaction_id);
} }
/**
* The last date each account's statements cover, keyed by the account's last
* four digits.
*
* This is the coverage test that matters, and it is not the one that was tried
* first. The first attempt asked whether a row's date fell inside a statement's
* min-max *window*, which for accounts whose statements span 182 to 460 days
* swallows a year and answers nothing — 422 duplicates out of 550 rows got in
* that way. `billing_end_date` asks a question that has an answer: up to what
* date is this account complete?
*
* Amount matching cannot substitute for it. Two sources decompose the same
* event differently — an aggregator bundles the Wise transfer fee into the
* transfer (10001.13) where the statement itemises it (10000.00 + 1.13) — so
* the rows are the same money with different numbers.
*
* Keyed on last4 because account numbers are written differently everywhere
* ("235242176", "xxxxxxxxxxxx2176", "4085-56264"). Where two banks share a
* last4 the later date wins, which errs towards excluding a row rather than
* duplicating one; the caller shows every watermark so a wrong one is visible.
*/
export async function getStatementCoverage(): Promise<{ last4: string; coveredTo: string }[]> {
const rows = await queryRaw<{ last4: string; covered_to: string }>(
`SELECT right(regexp_replace(account_number, '[^0-9]', '', 'g'), 4) AS last4,
MAX(billing_end_date)::text AS covered_to
FROM statements
WHERE account_number IS NOT NULL
GROUP BY 1`,
[]
);
return rows
.filter((r) => r.last4 && r.last4.length === 4)
.map((r) => ({ last4: r.last4, coveredTo: r.covered_to }))
.sort((a, b) => a.last4.localeCompare(b.last4));
}
+4
View File
@@ -9,6 +9,10 @@
"esModuleInterop": true, "esModuleInterop": true,
"module": "esnext", "module": "esnext",
"moduleResolution": "bundler", "moduleResolution": "bundler",
// scripts/*.mts run under `node --experimental-strip-types`, which resolves
// ESM specifiers literally and so needs the `.ts` extension written out.
// Legal here because noEmit is set — nothing is rewritten on the way out.
"allowImportingTsExtensions": true,
"resolveJsonModule": true, "resolveJsonModule": true,
"isolatedModules": true, "isolatedModules": true,
"jsx": "react-jsx", "jsx": "react-jsx",