Compare commits
18
Commits
1b34cd65d8
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
22c2349a47 | ||
|
|
3edcc27781 | ||
|
|
461c021e7a | ||
|
|
b82c4570bd | ||
|
|
afd75d3f09 | ||
|
|
2738213a23 | ||
|
|
bb8a009e02 | ||
|
|
3e826a317b | ||
|
|
21e9e765a3 | ||
|
|
493ff6f631 | ||
|
|
2c666236b2 | ||
|
|
215303161b | ||
|
|
3b5e495ca4 | ||
|
|
6b9b5fe518 | ||
|
|
231ef3411a | ||
|
|
699e4a2ddd | ||
|
|
79940202f1 | ||
|
|
9790b64e1d |
@@ -131,7 +131,7 @@ docker exec postgres-personal psql -U personal -d personal < prisma/migrations/<
|
|||||||
### Key Tables
|
### Key Tables
|
||||||
|
|
||||||
- `statements` — one row per billing period per bank account
|
- `statements` — one row per billing period per bank account
|
||||||
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown
|
- `transactions` — line items; `statement_id` is nullable (NULL = manual entry); `reconciled_with_id` links a manual tx to its matched statement tx; `payment_method` (migration 0016) is `card | cash | bank_transfer | other`, NULL = unknown; `source` / `source_ref` / `source_account` (migration 0028) identify a row that came from an account feed rather than a statement or a hand entry — `source_ref` is the provider's own id and carries a partial unique index, which is the only thing making a re-import idempotent
|
||||||
|
|
||||||
### Cash and reconciliation
|
### Cash and reconciliation
|
||||||
|
|
||||||
@@ -142,9 +142,28 @@ within 3 days and 1% on amount — and accepting one is silently destructive:
|
|||||||
reconciled manual rows are filtered out of every query, so the cash spend
|
reconciled manual rows are filtered out of every query, so the cash spend
|
||||||
disappears while the card transaction it matched claims to be that same spend.
|
disappears while the card transaction it matched claims to be that same spend.
|
||||||
|
|
||||||
Only cash is excluded. Bank transfers *do* appear on a statement now that
|
Only cash is excluded on payment method. Bank transfers *do* appear on a
|
||||||
transaction accounts are imported, and NULL means unknown — both stay
|
statement now that transaction accounts are imported, and NULL means unknown —
|
||||||
candidates, preserving the behaviour of every pre-existing row.
|
both stay candidates, preserving the behaviour of every pre-existing row.
|
||||||
|
|
||||||
|
**Account-feed rows are NOT excluded, and the removed exclusion is worth
|
||||||
|
knowing about.** There was an `awaitsStatementLine()` predicate here, added with
|
||||||
|
the Frollo importer on the premise that a feed row is the account's own ledger
|
||||||
|
entry with no statement line coming. That premise was asserted and never tested.
|
||||||
|
It was false for almost every account: 422 of the first 550 imported rows already
|
||||||
|
had a statement twin.
|
||||||
|
|
||||||
|
What matters is how it got in. The queue jumped from 8 to 558 the moment the feed
|
||||||
|
landed, and that jump was read as noise and filtered out. **The queue was right** —
|
||||||
|
those rows genuinely were provisional entries awaiting statement lines — and
|
||||||
|
filtering it removed the only mechanism that would ever have collapsed them, so
|
||||||
|
the duplicates became permanent instead of transient and stayed invisible until a
|
||||||
|
human saw one salary payment listed twice in two currencies.
|
||||||
|
|
||||||
|
A feed row belongs in the queue. Queue volume is solved **upstream**, by not
|
||||||
|
importing rows the ledger already holds (`LEDGER_MATCH_DAYS`), never downstream
|
||||||
|
by hiding the ones that are there. If a genuinely statement-less feed is ever
|
||||||
|
added, give it its own predicate and prove the premise with a query first.
|
||||||
|
|
||||||
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
|
ATM withdrawals stay categorised as spend rather than `transfers`. Treating them
|
||||||
as transfers only works if every cash purchase is logged; with partial logging
|
as transfers only works if every cash purchase is logged; with partial logging
|
||||||
@@ -617,6 +636,55 @@ deleted, and deleting a rule must not cascade away the audit trail.
|
|||||||
changed by something else since the run are flagged, because reverting restores
|
changed by something else since the run are flagged, because reverting restores
|
||||||
the pre-run value and discards the later edit.
|
the pre-run value and discards the later edit.
|
||||||
|
|
||||||
|
### Importing an aggregator/bank CSV (`/api/import/csv`)
|
||||||
|
|
||||||
|
There is **one** import path — the CSV import modal — and Frollo goes through it
|
||||||
|
like any other file. A bespoke Frollo importer, API route, CLI and two scheduled
|
||||||
|
n8n workflows existed for a day and were deleted on 2026-08-13: net −1,152 lines.
|
||||||
|
|
||||||
|
**The rule that made the rest unnecessary is statement coverage.** Each account's
|
||||||
|
newest `billing_end_date` is a watermark; a row on or before it is already in the
|
||||||
|
ledger. On the real 2,607-row export that leaves **171 rows** — with no account
|
||||||
|
allowlist and no credit-card exclusion, because cards have current statements and
|
||||||
|
drop out on their own. Every bit of the deleted apparatus was doing by hand what
|
||||||
|
`getStatementCoverage()` does generically.
|
||||||
|
|
||||||
|
Note what this replaced. The first attempt asked whether a row's date fell inside
|
||||||
|
a statement's min–max *window*, which for accounts whose statements span 182 to
|
||||||
|
460 days swallows a year and answers nothing — it let 422 duplicates into 550
|
||||||
|
rows. Amount-matching cannot substitute either: the two sources decompose the
|
||||||
|
same event differently, bundling a Wise transfer fee into the transfer (10001.13)
|
||||||
|
where the statement itemises it (10000.00 + 1.13).
|
||||||
|
|
||||||
|
**Map these optional columns or lose something silently:**
|
||||||
|
|
||||||
|
| Column | Without it |
|
||||||
|
|---|---|
|
||||||
|
| **Account** | nothing is excluded — you review the whole file |
|
||||||
|
| **Currency** | a foreign row is stored as if AUD (USD 10,782 → A$10,782) |
|
||||||
|
| **Row ID** | a re-import duplicates instead of no-opping |
|
||||||
|
|
||||||
|
Leave **Category** unmapped for an aggregator: its spend categories are not
|
||||||
|
trusted, and you set them per row in the review step.
|
||||||
|
|
||||||
|
**Two bugs fixed the day this shipped, both of which the flow depends on:**
|
||||||
|
`batchInsertCSVTransactions` accepted a `category` and then omitted it from the
|
||||||
|
INSERT column list, so every category chosen in review was discarded and the
|
||||||
|
trigger wrote `'other'` — not cosmetic, because an uncategorised credit is
|
||||||
|
admitted by `NET_SPEND_ROWS` and negated by `SPEND_SIGNED`, so 62 imported
|
||||||
|
transfers cancelled **$74,338** of spend while counting as no income. And the
|
||||||
|
path had no idempotency at all: `row_index` is assigned MAX+1 every run, making
|
||||||
|
`uq_transaction_identity` structurally unable to fire. It now writes
|
||||||
|
`source`/`source_ref` with `ON CONFLICT DO NOTHING`.
|
||||||
|
|
||||||
|
**The in-file duplicate warning is not redundant with `source_ref`.** A CDR
|
||||||
|
re-consent re-exports an account's whole history under fresh ids while the
|
||||||
|
originals survive — 385 twins in one 2,563-row export, doubling every salary
|
||||||
|
payment. Fresh ids mean `source_ref` sees new rows, and 385 is not visible by eye
|
||||||
|
in a review table. Consents expire annually, so expect it.
|
||||||
|
|
||||||
|
Undo an import: `DELETE FROM transactions WHERE source = '<source>'`.
|
||||||
|
|
||||||
### Trusting extracted statement data
|
### Trusting extracted statement data
|
||||||
|
|
||||||
**Balance assertions are the check that works.** `getStatements` computes
|
**Balance assertions are the check that works.** `getStatements` computes
|
||||||
|
|||||||
@@ -0,0 +1,127 @@
|
|||||||
|
-- 0030 — order ↔ transaction, many-to-many (board 205, orders phase 2)
|
||||||
|
--
|
||||||
|
-- Phase 1 read linkage out of `expense_metadata`, which is shaped as ONE ROW
|
||||||
|
-- PER TRANSACTION: `transaction_id` is UNIQUE and `matched_transaction_id`
|
||||||
|
-- partial-unique. Every multiplicity it expresses today is smuggled through a
|
||||||
|
-- string key — migration 0029 keys split shipments `<entity_key>#f<fact_id>` —
|
||||||
|
-- and a BNPL plan needs four rows for one order with no such trick available.
|
||||||
|
--
|
||||||
|
-- The flagship case: order_ebay_14-11714-95953, a A$1,599 DJI drone paid in
|
||||||
|
-- four A$399.75 Afterpay legs (txns 2318, 2333, 1652, 1664). Phase 1 shows it
|
||||||
|
-- with zero linked transactions, which is correct and useless.
|
||||||
|
--
|
||||||
|
-- KEYED ON entity_key, NOT entities.id. finance-app does not model the spine
|
||||||
|
-- and must not carry an FK across an ownership boundary that a spine
|
||||||
|
-- re-extraction can decompile. The real cascade risk is NOT transaction
|
||||||
|
-- deletion (that is handled below) but spine RE-KEYING: supersede_stale()
|
||||||
|
-- decompiles an entity when a re-extraction yields a different entity_key, and
|
||||||
|
-- a TEXT key with no FK silently orphans. Bridge-sourced links are rebuildable;
|
||||||
|
-- `manual` ones are curation and are not. Hence the orphan view at the end.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS order_transaction_links (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
entity_key TEXT NOT NULL,
|
||||||
|
transaction_id INTEGER NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
|
||||||
|
leg_kind TEXT NOT NULL CHECK (leg_kind IN ('charge','shipment','instalment','refund','fee')),
|
||||||
|
-- HUMAN position only: "2 of 4". Never an id.
|
||||||
|
--
|
||||||
|
-- The first draft stored a fact id here. Live shipment keys already carry
|
||||||
|
-- fact ids 22664 and 22711, max(extracted_facts.id) is 23,781 today, and
|
||||||
|
-- SMALLINT tops out at 32,767 — that ceiling arrives on ordinary corpus
|
||||||
|
-- growth and the insert dies with `smallint out of range`. Fact ids go in
|
||||||
|
-- source_fact_id, which is BIGINT.
|
||||||
|
leg_index SMALLINT,
|
||||||
|
leg_count SMALLINT,
|
||||||
|
source_fact_id BIGINT,
|
||||||
|
-- The LEG's own amount, signed: positive is money out, negative is a credit
|
||||||
|
-- coming back. Storing the order total on every leg would make four rows sum
|
||||||
|
-- to four times the purchase.
|
||||||
|
amount NUMERIC(12,2) NOT NULL,
|
||||||
|
currency TEXT NOT NULL DEFAULT 'AUD',
|
||||||
|
source TEXT NOT NULL CHECK (source IN ('order-bridge','instalment-matcher','manual')),
|
||||||
|
confidence TEXT NOT NULL DEFAULT 'exact' CHECK (confidence IN ('exact','derived','manual')),
|
||||||
|
evidence JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Every index needs IF NOT EXISTS, not just the table: a run that creates the
|
||||||
|
-- table and then fails during the backfill would otherwise abort on re-run
|
||||||
|
-- with "relation already exists" and need manual surgery.
|
||||||
|
--
|
||||||
|
-- uq_otl_transaction is load-bearing. Without it two matchers can each claim
|
||||||
|
-- txn 2318 and the order reads as paid twice. It is deliberately stricter than
|
||||||
|
-- reality for one case — a single card charge covering two orders — which then
|
||||||
|
-- surfaces as a NAMED REJECTED INSERT rather than as silent duplication.
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_transaction ON order_transaction_links (transaction_id);
|
||||||
|
|
||||||
|
-- WHAT IDENTIFIES A LEG DEPENDS ON ITS KIND, and the first version of this
|
||||||
|
-- index got it wrong in the most on-brand way available. It read
|
||||||
|
-- (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of the
|
||||||
|
-- same order — both with a NULL leg_index — collapsed to the same key and
|
||||||
|
-- ON CONFLICT DO NOTHING dropped one SILENTLY. Caught only because the
|
||||||
|
-- backfill reported 62 rows against 63 candidates.
|
||||||
|
--
|
||||||
|
-- The row it ate was order_amazon_249-4859367-0690246's $130.00 second
|
||||||
|
-- shipment (txn 3171) — the same order named in migration 0029's comment as
|
||||||
|
-- the reason split shipments need distinct keys at all, because Amazon mails
|
||||||
|
-- and CHARGES per shipment. A uniqueness rule that cannot hold two shipments
|
||||||
|
-- is the exact defect 0029 was written to fix.
|
||||||
|
--
|
||||||
|
-- So: an instalment leg is identified by its INDEX (1 of 4), a shipment leg by
|
||||||
|
-- its FACT (each shipment is its own extracted fact), and a whole-order charge
|
||||||
|
-- by neither — one per order, which the zeros express.
|
||||||
|
DROP INDEX IF EXISTS uq_otl_leg;
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_leg ON order_transaction_links
|
||||||
|
(entity_key, leg_kind, COALESCE(leg_index, 0), COALESCE(source_fact_id, 0));
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_otl_entity ON order_transaction_links (entity_key);
|
||||||
|
|
||||||
|
-- Backfill the existing bridge rows so the table is not empty on day one and
|
||||||
|
-- the readers can switch over in the same change. `#f<fact_id>` splits into a
|
||||||
|
-- shipment leg carrying its fact id; everything else is a whole-order charge.
|
||||||
|
INSERT INTO order_transaction_links
|
||||||
|
(entity_key, transaction_id, leg_kind, leg_index, source_fact_id, amount, currency, source, confidence, evidence)
|
||||||
|
SELECT split_part(em.source_message_id, '#', 1) AS entity_key,
|
||||||
|
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
|
||||||
|
CASE WHEN em.source_message_id LIKE '%#f%' THEN 'shipment' ELSE 'charge' END,
|
||||||
|
NULL::smallint,
|
||||||
|
NULLIF(substring(em.source_message_id FROM '#f([0-9]+)$'), '')::bigint,
|
||||||
|
COALESCE(em.amount, 0),
|
||||||
|
COALESCE(em.currency, 'AUD'),
|
||||||
|
'order-bridge',
|
||||||
|
'exact',
|
||||||
|
jsonb_build_object('backfilled_from', 'expense_metadata',
|
||||||
|
'source_message_id', em.source_message_id)
|
||||||
|
FROM expense_metadata em
|
||||||
|
WHERE em.source = 'order-bridge'
|
||||||
|
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
|
||||||
|
ON CONFLICT DO NOTHING;
|
||||||
|
|
||||||
|
-- Spine re-keying orphans links silently. Bridge rows can be rebuilt by
|
||||||
|
-- re-running the bridge; a `manual` orphan is lost curation and is the row
|
||||||
|
-- that actually needs a human. Same idea as order_settlement_violations.
|
||||||
|
CREATE OR REPLACE VIEW order_link_orphans AS
|
||||||
|
SELECT l.id, l.entity_key, l.transaction_id, l.leg_kind, l.leg_index,
|
||||||
|
l.amount, l.currency, l.source, l.confidence, l.created_at
|
||||||
|
FROM order_transaction_links l
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM entities e WHERE e.entity_key = l.entity_key);
|
||||||
|
|
||||||
|
-- Two stores that can disagree will disagree. The bridge now writes both in
|
||||||
|
-- ONE transaction, so a half-written pair should be impossible — this view
|
||||||
|
-- exists to prove that rather than to assume it, and it must stay empty.
|
||||||
|
--
|
||||||
|
-- The plan's suggested guard (widen the bridge's `NOT EXISTS expense_metadata`
|
||||||
|
-- to "neither store has it") was NOT taken: the expense_metadata INSERT has no
|
||||||
|
-- ON CONFLICT, so re-attempting an order that already has a receipt row would
|
||||||
|
-- duplicate it. Detecting drift is cheap; re-attempting a non-idempotent write
|
||||||
|
-- to fix a state that cannot occur is not.
|
||||||
|
CREATE OR REPLACE VIEW order_link_drift AS
|
||||||
|
SELECT em.source_message_id,
|
||||||
|
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
|
||||||
|
em.amount, em.currency, em.reconciled_at
|
||||||
|
FROM expense_metadata em
|
||||||
|
WHERE em.source = 'order-bridge'
|
||||||
|
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
|
||||||
|
AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM order_transaction_links l
|
||||||
|
WHERE l.transaction_id = COALESCE(em.matched_transaction_id, em.transaction_id)
|
||||||
|
);
|
||||||
@@ -0,0 +1,216 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import {
|
||||||
|
applyMapping,
|
||||||
|
inFileDuplicates,
|
||||||
|
last4,
|
||||||
|
splitByCoverage,
|
||||||
|
type ColumnMapping,
|
||||||
|
type ParsedTransaction,
|
||||||
|
} from "@/lib/csv-parser";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The statement-coverage rule, which replaced a whole Frollo-specific importer.
|
||||||
|
*
|
||||||
|
* The first aggregator import wrote 422 duplicate rows out of 550 — $1,023,824
|
||||||
|
* of movement counted twice — because "these accounts issue no statements" was
|
||||||
|
* asserted rather than queried. Amount-matching could not fix it either: the two
|
||||||
|
* sources decompose the same event differently, bundling a transfer fee where
|
||||||
|
* the statement itemises it.
|
||||||
|
*
|
||||||
|
* A statement's end date answers the question that actually has an answer: up
|
||||||
|
* to what date is this account complete? On the real 2026-08-13 export it takes
|
||||||
|
* 2,607 rows down to 171, with no account allowlist and no card exclusions.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const tx = (o: Partial<ParsedTransaction> & { date: string }): ParsedTransaction => ({
|
||||||
|
description: "SOMETHING",
|
||||||
|
amount: 10,
|
||||||
|
transaction_type: "debit",
|
||||||
|
...o,
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("last4", () => {
|
||||||
|
it("reads the same account written four different ways", () => {
|
||||||
|
// Every one of these forms appears in the real data.
|
||||||
|
expect(last4("xxxxxxxxxxxx2176")).toBe("2176");
|
||||||
|
expect(last4("235242176")).toBe("2176");
|
||||||
|
expect(last4("xxx-xxx xx4878")).toBe("4878");
|
||||||
|
expect(last4("4085-56264")).toBe("6264");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns empty for an unusable identifier rather than guessing", () => {
|
||||||
|
expect(last4("N/A")).toBe("");
|
||||||
|
expect(last4("12")).toBe("");
|
||||||
|
expect(last4("")).toBe("");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("splitByCoverage", () => {
|
||||||
|
const coverage = [{ last4: "2176", coveredTo: "2026-07-26" }];
|
||||||
|
|
||||||
|
it("excludes a row on or before the account's newest statement", () => {
|
||||||
|
const { keep, covered } = splitByCoverage(
|
||||||
|
[tx({ date: "2026-07-25", account: "xxxxxxxxxxxx2176" })],
|
||||||
|
coverage
|
||||||
|
);
|
||||||
|
expect(covered).toHaveLength(1);
|
||||||
|
expect(keep).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats the watermark date itself as covered", () => {
|
||||||
|
const { covered } = splitByCoverage(
|
||||||
|
[tx({ date: "2026-07-26", account: "xxxxxxxxxxxx2176" })],
|
||||||
|
coverage
|
||||||
|
);
|
||||||
|
expect(covered).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps a row after the watermark — the whole point of the feed", () => {
|
||||||
|
// The 2026-08-12 salary: the statement has not arrived, so this row is the
|
||||||
|
// only record of it and must survive.
|
||||||
|
const { keep } = splitByCoverage(
|
||||||
|
[tx({ date: "2026-08-12", account: "xxxxxxxxxxxx2176", amount: 10782 })],
|
||||||
|
coverage
|
||||||
|
);
|
||||||
|
expect(keep).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps everything for an account with no statements", () => {
|
||||||
|
const { keep, covered } = splitByCoverage(
|
||||||
|
[tx({ date: "2020-01-01", account: "xxx-xxx xx4878" })],
|
||||||
|
coverage
|
||||||
|
);
|
||||||
|
expect(keep).toHaveLength(1);
|
||||||
|
expect(covered).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps everything when no account column was mapped", () => {
|
||||||
|
// A single-account bank export has no account column, and a row must never
|
||||||
|
// be dropped on a guess about which account it belongs to.
|
||||||
|
const { keep, covered } = splitByCoverage([tx({ date: "2020-01-01" })], coverage);
|
||||||
|
expect(keep).toHaveLength(1);
|
||||||
|
expect(covered).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("applies each account's own watermark, not a global one", () => {
|
||||||
|
// The real export spans watermarks from 2026-03-31 to 2026-07-26. A single
|
||||||
|
// date would either import duplicates or discard real rows.
|
||||||
|
const { keep, covered } = splitByCoverage(
|
||||||
|
[
|
||||||
|
tx({ date: "2026-05-01", account: "xxxx0887" }), // covered to 2026-03-31 → keep
|
||||||
|
tx({ date: "2026-05-01", account: "xxxx2176" }), // covered to 2026-07-26 → drop
|
||||||
|
],
|
||||||
|
[
|
||||||
|
{ last4: "0887", coveredTo: "2026-03-31" },
|
||||||
|
{ last4: "2176", coveredTo: "2026-07-26" },
|
||||||
|
]
|
||||||
|
);
|
||||||
|
expect(keep).toHaveLength(1);
|
||||||
|
expect(keep[0].account).toBe("xxxx0887");
|
||||||
|
expect(covered).toHaveLength(1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("inFileDuplicates", () => {
|
||||||
|
it("counts twins that a row id cannot catch", () => {
|
||||||
|
// A CDR re-consent re-exports an account's whole history under fresh ids
|
||||||
|
// while the originals survive, so source_ref sees two distinct rows. 385 of
|
||||||
|
// these were in one 2,563-row export and doubled every salary payment.
|
||||||
|
const rows = [
|
||||||
|
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
|
||||||
|
tx({ date: "2026-07-07", amount: 10782, description: "HDR", account: "4830" }),
|
||||||
|
];
|
||||||
|
expect(inFileDuplicates(rows)).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not flag the same amount on different days", () => {
|
||||||
|
expect(
|
||||||
|
inFileDuplicates([
|
||||||
|
tx({ date: "2026-07-07", amount: 5, description: "COFFEE" }),
|
||||||
|
tx({ date: "2026-07-08", amount: 5, description: "COFFEE" }),
|
||||||
|
])
|
||||||
|
).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not flag the same amount on different accounts", () => {
|
||||||
|
// Both legs of an internal transfer: same day, same amount, different
|
||||||
|
// accounts, and both are real.
|
||||||
|
expect(
|
||||||
|
inFileDuplicates([
|
||||||
|
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "6264" }),
|
||||||
|
tx({ date: "2026-05-18", amount: 3076.04, description: "Transfer", account: "9940" }),
|
||||||
|
])
|
||||||
|
).toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("applyMapping with account and row-id columns", () => {
|
||||||
|
const labels = ["transaction_id", "description", "amount", "transaction_date", "account_number"];
|
||||||
|
const mapping: ColumnMapping = {
|
||||||
|
dateCol: "transaction_date",
|
||||||
|
descriptionCol: "description",
|
||||||
|
amountMode: "single",
|
||||||
|
amountCol: "amount",
|
||||||
|
accountCol: "account_number",
|
||||||
|
sourceRefCol: "transaction_id",
|
||||||
|
};
|
||||||
|
|
||||||
|
it("carries the account and row id through", () => {
|
||||||
|
const out = applyMapping(
|
||||||
|
[["1326131276", "HDR Global Services", "10782.00", "2026-08-12", "xxxxxxxxxxxx4830"]],
|
||||||
|
labels,
|
||||||
|
mapping,
|
||||||
|
"YYYY-MM-DD"
|
||||||
|
);
|
||||||
|
expect(out).toHaveLength(1);
|
||||||
|
expect(out[0].account).toBe("xxxxxxxxxxxx4830");
|
||||||
|
expect(out[0].source_ref).toBe("1326131276");
|
||||||
|
// A positive single-column amount is money in.
|
||||||
|
expect(out[0].transaction_type).toBe("credit");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("marks a non-AUD row as native and unconverted", () => {
|
||||||
|
// The 2026-08-12 salary is USD. Stored without this it reads as A$10,782 —
|
||||||
|
// about a third under — and sits in a column of AUD figures looking normal.
|
||||||
|
const out = applyMapping(
|
||||||
|
[["1", "HDR Global Services", "10782.00", "2026-08-12", "xxxx4830", "USD"]],
|
||||||
|
[...labels, "currency"],
|
||||||
|
{ ...mapping, currencyCol: "currency" },
|
||||||
|
"YYYY-MM-DD"
|
||||||
|
);
|
||||||
|
expect(out[0].foreign_currency_code).toBe("USD");
|
||||||
|
expect(out[0].foreign_currency_amount).toBe(10782);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves an AUD row unmarked", () => {
|
||||||
|
const out = applyMapping(
|
||||||
|
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830", "AUD"]],
|
||||||
|
[...labels, "currency"],
|
||||||
|
{ ...mapping, currencyCol: "currency" },
|
||||||
|
"YYYY-MM-DD"
|
||||||
|
);
|
||||||
|
expect(out[0].foreign_currency_code).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores a currency cell that is not a code", () => {
|
||||||
|
const out = applyMapping(
|
||||||
|
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830", ""]],
|
||||||
|
[...labels, "currency"],
|
||||||
|
{ ...mapping, currencyCol: "currency" },
|
||||||
|
"YYYY-MM-DD"
|
||||||
|
);
|
||||||
|
expect(out[0].foreign_currency_code).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves both undefined when the columns are not mapped", () => {
|
||||||
|
const out = applyMapping(
|
||||||
|
[["1", "SOMETHING", "-42.50", "2026-08-12", "xxxx4830"]],
|
||||||
|
labels,
|
||||||
|
{ dateCol: "transaction_date", descriptionCol: "description", amountMode: "single", amountCol: "amount" },
|
||||||
|
"YYYY-MM-DD"
|
||||||
|
);
|
||||||
|
expect(out[0].account).toBeUndefined();
|
||||||
|
expect(out[0].source_ref).toBeUndefined();
|
||||||
|
expect(out[0].transaction_type).toBe("debit");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import { tidyMerchant } from "@/lib/merchant-label";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Board 212. Every fixture below is a real `display_name` from order_feed.
|
||||||
|
* The cases that matter most are the ones that must NOT change: this is a
|
||||||
|
* presentational tidy and merging anything here would be a data error wearing
|
||||||
|
* a stylesheet.
|
||||||
|
*/
|
||||||
|
describe("tidyMerchant", () => {
|
||||||
|
it("capitalises a bare domain used as a name", () => {
|
||||||
|
expect(tidyMerchant("amazon.com.au")).toBe("Amazon.com.au");
|
||||||
|
expect(tidyMerchant("cdkeys.com")).toBe("Cdkeys.com");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("NEVER merges the country marketplaces", () => {
|
||||||
|
// The single most important assertion in this file. Amazon.in is a
|
||||||
|
// different marketplace with different currency and geography, and 289
|
||||||
|
// orders sit behind it.
|
||||||
|
expect(tidyMerchant("Amazon.in")).toBe("Amazon.in");
|
||||||
|
expect(tidyMerchant("amazon.com.au")).not.toBe(tidyMerchant("Amazon.in"));
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves real brand names that contain a TLD alone", () => {
|
||||||
|
for (const brand of ["Kogan.com", "GOG.com", "AliExpress.com", "Catch.com.au", "GeekBuying.com"]) {
|
||||||
|
expect(tidyMerchant(brand)).toBe(brand);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not uppercase a deliberately lower-cased brand", () => {
|
||||||
|
// The bug the corpus run found: gating on "starts lower-case" rather than
|
||||||
|
// on "is a bare domain" turned this into "EBay Commerce Australia".
|
||||||
|
expect(tidyMerchant("eBay Commerce Australia Pty Ltd.")).toBe("eBay Commerce Australia");
|
||||||
|
expect(tidyMerchant("iRobot Australia Pty Ltd")).toBe("iRobot Australia");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("drops trailing corporate suffixes", () => {
|
||||||
|
expect(tidyMerchant("Apple Pty Ltd.")).toBe("Apple");
|
||||||
|
expect(tidyMerchant("Microsoft Pty. Limited")).toBe("Microsoft");
|
||||||
|
expect(tidyMerchant("Meta Platforms, Inc.")).toBe("Meta Platforms");
|
||||||
|
expect(tidyMerchant("Domino's Pizza Enterprises Limited")).toBe("Domino's Pizza Enterprises");
|
||||||
|
expect(tidyMerchant("Blinks Labs GmbH")).toBe("Blinks Labs");
|
||||||
|
expect(tidyMerchant("Rasier New Zealand Limited")).toBe("Rasier New Zealand");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("strips two suffixes when a name carries two", () => {
|
||||||
|
expect(tidyMerchant("Samsung Electronics Co. Ltd.")).toBe("Samsung Electronics");
|
||||||
|
expect(tidyMerchant("DiDi Mobility Information Technology Pte. Ltd."))
|
||||||
|
.toBe("DiDi Mobility Information Technology");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not leave a dangling connector", () => {
|
||||||
|
// 'Co' is part of this brand. Stripping it produced "Coburger &".
|
||||||
|
expect(tidyMerchant("Coburger & Co")).toBe("Coburger & Co");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps a name that would be reduced to almost nothing", () => {
|
||||||
|
// "UT" is too short to stand on its own; better scruffy than cryptic.
|
||||||
|
expect(tidyMerchant("UT LLC")).toBe("UT LLC");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("normalises whitespace and tolerates junk", () => {
|
||||||
|
expect(tidyMerchant(" Yaffa Media Pty Ltd ")).toBe("Yaffa Media");
|
||||||
|
expect(tidyMerchant("")).toBe("");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is idempotent", () => {
|
||||||
|
for (const n of ["amazon.com.au", "Apple Pty Ltd.", "Coburger & Co", "Amazon.in"]) {
|
||||||
|
expect(tidyMerchant(tidyMerchant(n))).toBe(tidyMerchant(n));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -8,6 +8,12 @@ export async function POST(req: NextRequest) {
|
|||||||
|
|
||||||
const body = await req.json() as {
|
const body = await req.json() as {
|
||||||
bank_name: string;
|
bank_name: string;
|
||||||
|
/**
|
||||||
|
* Optional provenance label, stored on every row as `source`. With a
|
||||||
|
* `source_ref` per row it makes a re-import a no-op; without it this path
|
||||||
|
* has no idempotency at all.
|
||||||
|
*/
|
||||||
|
source?: string;
|
||||||
transactions: {
|
transactions: {
|
||||||
date: string;
|
date: string;
|
||||||
description: string;
|
description: string;
|
||||||
@@ -17,6 +23,8 @@ export async function POST(req: NextRequest) {
|
|||||||
foreign_currency_amount?: number;
|
foreign_currency_amount?: number;
|
||||||
foreign_currency_code?: string;
|
foreign_currency_code?: string;
|
||||||
category?: string;
|
category?: string;
|
||||||
|
account?: string;
|
||||||
|
source_ref?: string;
|
||||||
}[];
|
}[];
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -25,7 +33,7 @@ export async function POST(req: NextRequest) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const tagId = await ensureTag("csv-import", "#8b5cf6");
|
const tagId = await ensureTag("csv-import", "#8b5cf6");
|
||||||
const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId);
|
const inserted = await batchInsertCSVTransactions(user.id, body.transactions, tagId, body.source);
|
||||||
|
|
||||||
return NextResponse.json({ inserted }, { status: 201 });
|
return NextResponse.json({ inserted }, { status: 201 });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
|
import { getCurrentUser } from "@/lib/auth";
|
||||||
|
import { getStatementCoverage } from "@/lib/queries";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How far each account's statements already reach.
|
||||||
|
*
|
||||||
|
* Read by the CSV import modal so it can drop rows the ledger already holds.
|
||||||
|
* The first Frollo import went in without this and wrote 422 duplicates out of
|
||||||
|
* 550 — the same payments the statements already carried, itemised and
|
||||||
|
* converted. On the 2026-08-13 export the rule takes 2,607 rows down to 171.
|
||||||
|
*/
|
||||||
|
export async function GET(req: NextRequest) {
|
||||||
|
const user = await getCurrentUser(req);
|
||||||
|
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||||
|
return NextResponse.json({ coverage: await getStatementCoverage() });
|
||||||
|
}
|
||||||
@@ -38,7 +38,30 @@ export async function GET(
|
|||||||
[Number(id)]
|
[Number(id)]
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Phase 2 (board 205). A BNPL leg has NO expense_metadata row of its own —
|
||||||
|
// the plan is four transactions against one order — so without this the four
|
||||||
|
// Afterpay debits behind the A$1,599 DJI drone stay four bare "Afterpay
|
||||||
|
// $399.75" rows naming nothing. That is the surface where this was noticed.
|
||||||
|
const link = await queryRow<{
|
||||||
|
entity_key: string; leg_kind: string; leg_index: number | null;
|
||||||
|
leg_count: number | null; canonical_name: string | null;
|
||||||
|
platform: string | null; order_total: string | null; currency: string | null;
|
||||||
|
}>(
|
||||||
|
`SELECT l.entity_key, l.leg_kind, l.leg_index::int, l.leg_count::int,
|
||||||
|
e.canonical_name, o.platform, o.order_total, o.currency
|
||||||
|
FROM order_transaction_links l
|
||||||
|
LEFT JOIN entities e ON e.entity_key = l.entity_key
|
||||||
|
LEFT JOIN entity_orders o ON o.entity_id = e.id
|
||||||
|
WHERE l.transaction_id = $1
|
||||||
|
LIMIT 1`,
|
||||||
|
[Number(id)]
|
||||||
|
);
|
||||||
|
|
||||||
// Not an order — most transactions aren't. Null, not 404: the caller is
|
// Not an order — most transactions aren't. Null, not 404: the caller is
|
||||||
// asking "is there a receipt behind this?", and "no" is a normal answer.
|
// asking "is there a receipt behind this?", and "no" is a normal answer.
|
||||||
return NextResponse.json(row ?? null);
|
if (!row && !link) return NextResponse.json(null);
|
||||||
|
|
||||||
|
// expense_metadata carries the itemised receipt and stays primary; the link
|
||||||
|
// adds what it cannot express — which order this is a leg OF, and which leg.
|
||||||
|
return NextResponse.json({ ...(row ?? {}), order_link: link ?? null });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@
|
|||||||
import { use } from "react";
|
import { use } from "react";
|
||||||
import Link from "next/link";
|
import Link from "next/link";
|
||||||
import { useOrderDetail } from "@/lib/hooks";
|
import { useOrderDetail } from "@/lib/hooks";
|
||||||
|
import { tidyMerchant } from "@/lib/merchant-label";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* One order: what it was, what happened to it, and what paid for it.
|
* One order: what it was, what happened to it, and what paid for it.
|
||||||
@@ -63,10 +64,22 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
|
|||||||
|
|
||||||
if (isLoading) return <div className="p-6 text-zinc-500 text-sm">Loading order…</div>;
|
if (isLoading) return <div className="p-6 text-zinc-500 text-sm">Loading order…</div>;
|
||||||
if (error || !o) {
|
if (error || !o) {
|
||||||
|
// A missing order and a broken query are different problems and must not
|
||||||
|
// share a sentence. Only a 404 means "no such order"; anything else is this
|
||||||
|
// page failing, and saying so is what sends the next person to the server
|
||||||
|
// log instead of to the spine.
|
||||||
|
const status = (error as (Error & { status?: number }) | null)?.status;
|
||||||
return (
|
return (
|
||||||
<div className="max-w-[1180px] mx-auto">
|
<div className="max-w-[1180px] mx-auto">
|
||||||
<Link href="/orders" className="font-mono text-[11.5px] text-indigo-400">← All orders</Link>
|
<Link href="/orders" className="font-mono text-[11.5px] text-indigo-400">← All orders</Link>
|
||||||
|
{status === 404 ? (
|
||||||
<p className="mt-6 text-zinc-400 text-sm">That order could not be found.</p>
|
<p className="mt-6 text-zinc-400 text-sm">That order could not be found.</p>
|
||||||
|
) : (
|
||||||
|
<p className="mt-6 text-zinc-400 text-sm">
|
||||||
|
This order could not be loaded{status ? ` (server error ${status})` : ""}. The order exists —
|
||||||
|
something in this page failed. Check the finance-app log.
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -82,6 +95,23 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
|
|||||||
|
|
||||||
{/* A settled rail row must never look like an ordinary order — that is
|
{/* A settled rail row must never look like an ordinary order — that is
|
||||||
how a human counts the same purchase twice. */}
|
how a human counts the same purchase twice. */}
|
||||||
|
{/* Board 211 — same shape as the settled-rail banner above: a real
|
||||||
|
document with a real figure that is not a purchase. Saying which
|
||||||
|
payments it restates is the point; the total alone reads as a
|
||||||
|
single large order. */}
|
||||||
|
{o.summarises_period && (
|
||||||
|
<div className="mt-4 bg-zinc-900 border-l-2 border-amber-600 px-4 py-3">
|
||||||
|
<span className="block font-mono text-[10px] uppercase tracking-widest text-amber-500 mb-1">
|
||||||
|
Annual summary, not a purchase
|
||||||
|
</span>
|
||||||
|
<span className="text-[12.5px] text-zinc-300">
|
||||||
|
This receipt restates payments already made across the year — the
|
||||||
|
monthly debits are in the ledger under their own dates. It is kept
|
||||||
|
for reference and excluded from spend totals.
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
{o.is_settled_duplicate && (
|
{o.is_settled_duplicate && (
|
||||||
<div className="mt-4 bg-zinc-900 border-l-2 border-indigo-600 px-4 py-3">
|
<div className="mt-4 bg-zinc-900 border-l-2 border-indigo-600 px-4 py-3">
|
||||||
<span className="block font-mono text-[10px] uppercase tracking-widest text-indigo-500 mb-1">
|
<span className="block font-mono text-[10px] uppercase tracking-widest text-indigo-500 mb-1">
|
||||||
@@ -100,7 +130,10 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
|
|||||||
{o.ordered_at && <> · {dateFmt.format(new Date(o.ordered_at))}</>}
|
{o.ordered_at && <> · {dateFmt.format(new Date(o.ordered_at))}</>}
|
||||||
</div>
|
</div>
|
||||||
<h2 className="font-display text-[27px] leading-tight text-zinc-50 my-3 max-w-[26ch] text-balance">
|
<h2 className="font-display text-[27px] leading-tight text-zinc-50 my-3 max-w-[26ch] text-balance">
|
||||||
{o.canonical_name || o.display_name || "Order"}
|
{/* canonical_name is the ORDER title ("DJI Air 3 Fly More Combo") and
|
||||||
|
is left alone; only the display_name fallback is a merchant name
|
||||||
|
and wants tidying. */}
|
||||||
|
{o.canonical_name || tidyMerchant(o.display_name ?? "") || "Order"}
|
||||||
</h2>
|
</h2>
|
||||||
<div className="flex gap-2 flex-wrap mb-3">
|
<div className="flex gap-2 flex-wrap mb-3">
|
||||||
{o.content_class && KIND_LABEL[o.content_class] && (
|
{o.content_class && KIND_LABEL[o.content_class] && (
|
||||||
@@ -135,7 +168,11 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
|
|||||||
{o.merchant_name && (
|
{o.merchant_name && (
|
||||||
<div>
|
<div>
|
||||||
<span className="block text-[10px] uppercase tracking-widest text-zinc-500 mb-0.5">Merchant</span>
|
<span className="block text-[10px] uppercase tracking-widest text-zinc-500 mb-0.5">Merchant</span>
|
||||||
{o.merchant_name}
|
{/* Board 212: the list has tidied this since it shipped and the
|
||||||
|
detail page did not, so following a row through showed
|
||||||
|
"Apple" becoming "Apple Pty Ltd." Presentational only —
|
||||||
|
merges nothing, and Amazon.in stays Amazon.in. */}
|
||||||
|
{tidyMerchant(o.merchant_name)}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
@@ -197,6 +234,15 @@ export default function OrderDetailPage({ params }: { params: Promise<{ entityKe
|
|||||||
<span className="font-mono text-[12px] text-zinc-300">
|
<span className="font-mono text-[12px] text-zinc-300">
|
||||||
{t.description}
|
{t.description}
|
||||||
<small className="block text-[10.5px] text-zinc-500">
|
<small className="block text-[10.5px] text-zinc-500">
|
||||||
|
{/* Board 205. Four identical "Afterpay" rows are
|
||||||
|
indistinguishable without their position, and
|
||||||
|
indistinguishable rows are what made this order
|
||||||
|
look like four purchases. */}
|
||||||
|
{t.leg_kind === "instalment" && t.leg_index && t.leg_count
|
||||||
|
? `Instalment ${t.leg_index} of ${t.leg_count} · `
|
||||||
|
: t.leg_kind === "shipment"
|
||||||
|
? "Shipment · "
|
||||||
|
: ""}
|
||||||
{dateFmt.format(new Date(t.transaction_date))} · txn {t.transaction_id}
|
{dateFmt.format(new Date(t.transaction_date))} · txn {t.transaction_id}
|
||||||
</small>
|
</small>
|
||||||
</span>
|
</span>
|
||||||
|
|||||||
+39
-30
@@ -3,6 +3,7 @@
|
|||||||
import { Suspense, useMemo, useState } from "react";
|
import { Suspense, useMemo, useState } from "react";
|
||||||
import Link from "next/link";
|
import Link from "next/link";
|
||||||
import { useOrders } from "@/lib/hooks";
|
import { useOrders } from "@/lib/hooks";
|
||||||
|
import { tidyMerchant } from "@/lib/merchant-label";
|
||||||
import type { OrderRow } from "@/lib/order-feed";
|
import type { OrderRow } from "@/lib/order-feed";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -75,31 +76,6 @@ const KIND_LABEL: Record<string, string> = {
|
|||||||
account_notice: "notice",
|
account_notice: "notice",
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
|
||||||
* Presentational tidy only — this NEVER merges two merchants. The estate holds
|
|
||||||
* "amazon.com.au", "Amazon.in" and "Amazon Services Australia, Inc." as three
|
|
||||||
* distinct entities, and Amazon.in must stay separate: it is a different
|
|
||||||
* marketplace, not a name variant. Actually unifying them is the merchant-alias
|
|
||||||
* bridge (ticket 176). All this does is stop the same entity from looking
|
|
||||||
* scruffy: drop the corporate suffix, and capitalise a name that arrived
|
|
||||||
* lower-cased from a domain.
|
|
||||||
*/
|
|
||||||
const CORP_SUFFIX =
|
|
||||||
/,?\s+(pty\.?\s+ltd\.?|pty\.?\s+limited|p\/l|ltd\.?|limited|inc\.?|llc|pbc|gmbh|b\.?v\.?|s\.?a\.?r\.?l\.?|oü|co\.?)$/i;
|
|
||||||
|
|
||||||
function tidyMerchant(name: string): string {
|
|
||||||
let n = name.replace(/\s+/g, " ").trim();
|
|
||||||
// Strip at most two trailing corporate suffixes ("Pty Ltd." then ",").
|
|
||||||
for (let i = 0; i < 2; i++) {
|
|
||||||
const stripped = n.replace(CORP_SUFFIX, "").trim().replace(/,$/, "");
|
|
||||||
if (stripped === n || stripped.length < 3) break;
|
|
||||||
n = stripped;
|
|
||||||
}
|
|
||||||
// "amazon.com.au" reads as a machine artefact; "Amazon.com.au" reads as a name.
|
|
||||||
if (/^[a-z]/.test(n)) n = n[0].toUpperCase() + n.slice(1);
|
|
||||||
return n;
|
|
||||||
}
|
|
||||||
|
|
||||||
const IN_FLIGHT = new Set(["ordered", "shipped", "out_for_delivery"]);
|
const IN_FLIGHT = new Set(["ordered", "shipped", "out_for_delivery"]);
|
||||||
const REVERSED = new Set(["refunded", "returned", "cancelled"]);
|
const REVERSED = new Set(["refunded", "returned", "cancelled"]);
|
||||||
|
|
||||||
@@ -293,11 +269,44 @@ function Row({ row, expanded, onToggle }: { row: OrderRow; expanded: boolean; on
|
|||||||
className="font-mono text-[9.5px] uppercase tracking-wide text-indigo-600 border border-indigo-800 rounded-sm px-1.5"
|
className="font-mono text-[9.5px] uppercase tracking-wide text-indigo-600 border border-indigo-800 rounded-sm px-1.5"
|
||||||
>no ref</span>
|
>no ref</span>
|
||||||
)}
|
)}
|
||||||
{row.source_trust === "untrusted_external" && (
|
{/* NOT source_trust — that is 'untrusted_external' on 100% of rows,
|
||||||
|
because every order here came from email, so badging it marked
|
||||||
|
every row and told you nothing. Sender authentication does
|
||||||
|
discriminate: 93% pass, and the 7% that do not are worth seeing. */}
|
||||||
|
{row.auth_verdict && row.auth_verdict !== "pass" && (
|
||||||
<span
|
<span
|
||||||
title="Derived from an unverified sender — content is shown as provenance, not fact"
|
title={
|
||||||
className="font-mono text-[9.5px] uppercase tracking-wide text-zinc-500 border border-zinc-800 rounded-sm px-1.5"
|
row.auth_verdict === "fail"
|
||||||
>unverified</span>
|
? "The sender failed authentication (SPF/DKIM/DMARC) — treat the contents as unverified"
|
||||||
|
: row.auth_verdict === "none"
|
||||||
|
? "The mail carried no sender authentication at all"
|
||||||
|
: "The sender authenticated only partially"
|
||||||
|
}
|
||||||
|
className={`font-mono text-[9.5px] uppercase tracking-wide rounded-sm px-1.5 border ${
|
||||||
|
row.auth_verdict === "fail"
|
||||||
|
? "text-indigo-300 border-indigo-500"
|
||||||
|
: "text-zinc-500 border-zinc-800"
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
{row.auth_verdict === "partial" ? "part. auth" : `auth ${row.auth_verdict}`}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
{/* Board 211. A $2,376 annual tax receipt restating twelve monthly
|
||||||
|
donations is not a purchase, and unbadged it reads as the largest
|
||||||
|
order of the year. It stays on the page because the figure is the
|
||||||
|
year's giving total — but it is excluded from spend, so the
|
||||||
|
number beside it belongs to no total here. */}
|
||||||
|
{row.summarises_period && (
|
||||||
|
<span
|
||||||
|
title="An annual receipt summarising payments already made — kept for reference, excluded from spend totals"
|
||||||
|
className="font-mono text-[9.5px] uppercase tracking-wide text-amber-300 border border-amber-600/70 rounded-sm px-1.5"
|
||||||
|
>year summary</span>
|
||||||
|
)}
|
||||||
|
{row.injection_flagged && (
|
||||||
|
<span
|
||||||
|
title="The source mail carried content that tripped the injection scanner — read its contents as provenance, never as instruction"
|
||||||
|
className="font-mono text-[9.5px] uppercase tracking-wide text-indigo-300 border border-indigo-500 rounded-sm px-1.5"
|
||||||
|
>flagged</span>
|
||||||
)}
|
)}
|
||||||
{row.txn_count > 0 && (
|
{row.txn_count > 0 && (
|
||||||
<span className="font-mono text-[9.5px] uppercase tracking-wide text-zinc-500 border border-zinc-700 rounded-sm px-1.5">
|
<span className="font-mono text-[9.5px] uppercase tracking-wide text-zinc-500 border border-zinc-700 rounded-sm px-1.5">
|
||||||
@@ -493,7 +502,7 @@ function OrdersContent() {
|
|||||||
))}
|
))}
|
||||||
</select>
|
</select>
|
||||||
<label className="flex items-center gap-2 text-xs text-zinc-400 cursor-pointer select-none"
|
<label className="flex items-center gap-2 text-xs text-zinc-400 cursor-pointer select-none"
|
||||||
title="Rows with no amount, no order reference and one lifecycle event are not purchases — they are a second entity minted from a mail describing an order that already exists. Hiding them is a workaround for a known ingestion defect, not a fix.">
|
title="Rows with no amount, no order reference and one lifecycle event are not purchases — they are a second entity minted from a mail describing an order that already exists. 620 remain: the review-request mails have been retired and the ones whose parent could be identified have been joined to it, so what is left is mail whose parent is not in the corpus.">
|
||||||
<input
|
<input
|
||||||
type="checkbox"
|
type="checkbox"
|
||||||
checked={showLifecycle}
|
checked={showLifecycle}
|
||||||
|
|||||||
@@ -1018,6 +1018,17 @@ function TransactionsContent() {
|
|||||||
</div>
|
</div>
|
||||||
{t.notes ? (
|
{t.notes ? (
|
||||||
<p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.notes}>{t.notes}</p>
|
<p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.notes}>{t.notes}</p>
|
||||||
|
) : t.order_leg_kind === "instalment" && t.order_name ? (
|
||||||
|
// Board 205. Four rows reading "Afterpay $399.75" name
|
||||||
|
// nothing — this is the surface where that was noticed.
|
||||||
|
// The order is what tells them apart, and "2 of 4" is
|
||||||
|
// what stops the same purchase reading as four.
|
||||||
|
<p className="truncate text-xs text-zinc-500 italic mt-0.5" title={t.order_name}>
|
||||||
|
{t.order_leg_index && t.order_leg_count
|
||||||
|
? `${t.order_leg_index} of ${t.order_leg_count} · `
|
||||||
|
: ""}
|
||||||
|
{t.order_name}
|
||||||
|
</p>
|
||||||
) : t.order_platform === "uber" && routeSummary(t.order_route) && (
|
) : t.order_platform === "uber" && routeSummary(t.order_route) && (
|
||||||
// Five rows all reading "Order - Uber Trip" are
|
// Five rows all reading "Order - Uber Trip" are
|
||||||
// indistinguishable. Where the trip went is what tells
|
// indistinguishable. Where the trip went is what tells
|
||||||
@@ -1048,12 +1059,33 @@ function TransactionsContent() {
|
|||||||
<td className={`p-2 text-right whitespace-nowrap font-mono ${
|
<td className={`p-2 text-right whitespace-nowrap font-mono ${
|
||||||
SPEND_TYPES.has(t.transaction_type) ? "text-red-400" : "text-green-400"
|
SPEND_TYPES.has(t.transaction_type) ? "text-red-400" : "text-green-400"
|
||||||
}`}>
|
}`}>
|
||||||
|
{/*
|
||||||
|
A row with no AUD figure must NOT be printed as AUD.
|
||||||
|
`amount_aud ?? amount` formatted with the default currency
|
||||||
|
rendered a USD 10,782.00 Wise credit as "+$10,782.00" in a
|
||||||
|
column of AUD figures — the real value was about $15,500,
|
||||||
|
and the row directly above it (the same payment from the
|
||||||
|
statement) showed exactly that. The native figure is the
|
||||||
|
only true one here, so it becomes the headline and the
|
||||||
|
absence of a rate is stated rather than papered over.
|
||||||
|
*/}
|
||||||
|
{t.amount_unconverted ? (
|
||||||
|
<>
|
||||||
|
{formatAmount(t.amount, t.transaction_type, t.currency)}
|
||||||
|
<div className="text-[10px] text-zinc-500 mt-0.5" title="No AUD figure exists for this row — it is excluded from AUD totals">
|
||||||
|
no AUD rate
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
|
{formatAmount(t.amount_aud ?? t.amount, t.transaction_type)}
|
||||||
{t.currency && t.currency !== "AUD" && (
|
{t.currency && t.currency !== "AUD" && (
|
||||||
<div className="text-[10px] text-zinc-500 mt-0.5">
|
<div className="text-[10px] text-zinc-500 mt-0.5">
|
||||||
{formatAmount(t.amount, t.transaction_type, t.currency)}
|
{formatAmount(t.amount, t.transaction_type, t.currency)}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
</>
|
||||||
|
)}
|
||||||
</td>
|
</td>
|
||||||
<td className="p-2">
|
<td className="p-2">
|
||||||
<EditableTypeBadge
|
<EditableTypeBadge
|
||||||
|
|||||||
@@ -2,10 +2,10 @@
|
|||||||
|
|
||||||
import { useState, useRef, useEffect } from "react";
|
import { useState, useRef, useEffect } from "react";
|
||||||
import { CATEGORIES, formatCategory } from "@/lib/categories";
|
import { CATEGORIES, formatCategory } from "@/lib/categories";
|
||||||
import { useImportCSV } from "@/lib/hooks";
|
import { useImportCSV, useStatementCoverage } from "@/lib/hooks";
|
||||||
import {
|
import {
|
||||||
parseCSVRows, detectHasHeaders, getColumnLabels, getDataRows, applyMapping,
|
parseCSVRows, detectHasHeaders, getColumnLabels, getDataRows, applyMapping,
|
||||||
saveBankPreset, loadBankPresets,
|
saveBankPreset, loadBankPresets, splitByCoverage, inFileDuplicates,
|
||||||
type DateFormat, type ColumnMapping, type ParsedTransaction, type BankPreset,
|
type DateFormat, type ColumnMapping, type ParsedTransaction, type BankPreset,
|
||||||
} from "@/lib/csv-parser";
|
} from "@/lib/csv-parser";
|
||||||
|
|
||||||
@@ -14,6 +14,8 @@ const TX_TYPES = ["debit", "credit", "payment", "refund", "fee", "interest", "tr
|
|||||||
|
|
||||||
type Step = "upload" | "map" | "review" | "done";
|
type Step = "upload" | "map" | "review" | "done";
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
function ColSelect({
|
function ColSelect({
|
||||||
label, value, onChange, options, required,
|
label, value, onChange, options, required,
|
||||||
}: {
|
}: {
|
||||||
@@ -54,6 +56,9 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
|||||||
const [error, setError] = useState("");
|
const [error, setError] = useState("");
|
||||||
const [presets, setPresets] = useState<BankPreset[]>([]);
|
const [presets, setPresets] = useState<BankPreset[]>([]);
|
||||||
const [insertedCount, setInsertedCount] = useState(0);
|
const [insertedCount, setInsertedCount] = useState(0);
|
||||||
|
const [coveredRows, setCoveredRows] = useState<ParsedTransaction[]>([]);
|
||||||
|
const [showCovered, setShowCovered] = useState(false);
|
||||||
|
const coverage = useStatementCoverage();
|
||||||
|
|
||||||
useEffect(() => { setPresets(loadBankPresets()); }, []);
|
useEffect(() => { setPresets(loadBankPresets()); }, []);
|
||||||
|
|
||||||
@@ -111,7 +116,9 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
|||||||
if (savePreset) {
|
if (savePreset) {
|
||||||
saveBankPreset({ bankName: bankName.trim(), mapping, dateFormat });
|
saveBankPreset({ bankName: bankName.trim(), mapping, dateFormat });
|
||||||
}
|
}
|
||||||
setEditedRows(parsed);
|
const { keep, covered } = splitByCoverage(parsed, coverage.data ?? []);
|
||||||
|
setEditedRows(keep);
|
||||||
|
setCoveredRows(covered);
|
||||||
setStep("review");
|
setStep("review");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -120,7 +127,13 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
|||||||
const valid = editedRows.filter((r) => r.date && r.amount > 0 && r.description);
|
const valid = editedRows.filter((r) => r.date && r.amount > 0 && r.description);
|
||||||
if (!valid.length) { setError("No valid rows to import"); return; }
|
if (!valid.length) { setError("No valid rows to import"); return; }
|
||||||
try {
|
try {
|
||||||
const result = await importCSV.mutateAsync({ bank_name: bankName, transactions: valid });
|
const result = await importCSV.mutateAsync({
|
||||||
|
bank_name: bankName,
|
||||||
|
// Provenance, so a second import of the same file inserts nothing —
|
||||||
|
// paired with each row's source_ref where the file carries an id.
|
||||||
|
source: bankName.trim().toLowerCase().replace(/\s+/g, "-"),
|
||||||
|
transactions: valid,
|
||||||
|
});
|
||||||
setInsertedCount(result.inserted);
|
setInsertedCount(result.inserted);
|
||||||
setStep("done");
|
setStep("done");
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
@@ -286,7 +299,16 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
|||||||
<div className="grid grid-cols-2 gap-3">
|
<div className="grid grid-cols-2 gap-3">
|
||||||
<ColSelect label="Merchant Column (optional)" value={mapping.merchantCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, merchantCol: v || undefined }))} options={columnLabels} />
|
<ColSelect label="Merchant Column (optional)" value={mapping.merchantCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, merchantCol: v || undefined }))} options={columnLabels} />
|
||||||
<ColSelect label="Category Column (optional)" value={mapping.categoryCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, categoryCol: v || undefined }))} options={columnLabels} />
|
<ColSelect label="Category Column (optional)" value={mapping.categoryCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, categoryCol: v || undefined }))} options={columnLabels} />
|
||||||
|
<ColSelect label="Account Column (optional)" value={mapping.accountCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, accountCol: v || undefined }))} options={columnLabels} />
|
||||||
|
<ColSelect label="Currency Column (optional)" value={mapping.currencyCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, currencyCol: v || undefined }))} options={columnLabels} />
|
||||||
|
<ColSelect label="Row ID Column (optional)" value={mapping.sourceRefCol ?? ""} onChange={(v) => setMapping((m) => ({ ...m, sourceRefCol: v || undefined }))} options={columnLabels} />
|
||||||
</div>
|
</div>
|
||||||
|
<p className="text-[11px] text-zinc-500 leading-relaxed">
|
||||||
|
Map <b>Account</b> when the file covers more than one account: rows already
|
||||||
|
covered by that account's statements are then left out. Map <b>Row ID</b> to
|
||||||
|
make re-importing the same file do nothing. Map <b>Currency</b> for a
|
||||||
|
multi-currency file, or foreign rows are stored as if they were AUD.
|
||||||
|
</p>
|
||||||
|
|
||||||
<label className="flex items-center gap-2 text-sm cursor-pointer text-zinc-400">
|
<label className="flex items-center gap-2 text-sm cursor-pointer text-zinc-400">
|
||||||
<input type="checkbox" checked={savePreset} onChange={(e) => setSavePreset(e.target.checked)} className="accent-indigo-500" />
|
<input type="checkbox" checked={savePreset} onChange={(e) => setSavePreset(e.target.checked)} className="accent-indigo-500" />
|
||||||
@@ -300,9 +322,52 @@ export function CsvImportModal({ onClose }: { onClose: () => void }) {
|
|||||||
{/* Step 3: Review */}
|
{/* Step 3: Review */}
|
||||||
{step === "review" && (
|
{step === "review" && (
|
||||||
<div>
|
<div>
|
||||||
<p className="text-xs text-zinc-500 mb-3">
|
<p className="text-xs text-zinc-500 mb-2">
|
||||||
{editedRows.length} transactions parsed. Edit or remove rows before importing.
|
{editedRows.length} transactions to import. Edit or remove rows before importing.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
|
{/*
|
||||||
|
Reported, never silent. A row that disappears can mean several
|
||||||
|
things and one of them is worth following up, so the count is
|
||||||
|
always shown and the rows themselves are one click away.
|
||||||
|
*/}
|
||||||
|
{coveredRows.length > 0 && (
|
||||||
|
<div className="mb-3 rounded border border-zinc-800 bg-zinc-800/30 px-3 py-2">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => setShowCovered((v) => !v)}
|
||||||
|
className="text-xs text-zinc-300 hover:text-white text-left w-full"
|
||||||
|
>
|
||||||
|
{coveredRows.length} row{coveredRows.length === 1 ? "" : "s"} left out — already covered by
|
||||||
|
a statement for that account.{" "}
|
||||||
|
<span className="text-indigo-400">{showCovered ? "hide" : "show"}</span>
|
||||||
|
</button>
|
||||||
|
{showCovered && (
|
||||||
|
<>
|
||||||
|
<ul className="mt-2 max-h-40 overflow-y-auto text-[11px] text-zinc-500 font-mono space-y-0.5">
|
||||||
|
{coveredRows.slice(0, 200).map((r, i) => (
|
||||||
|
<li key={i}>{r.date} {r.amount.toFixed(2).padStart(10)} {r.description.slice(0, 44)}</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
{coveredRows.length > 200 && (
|
||||||
|
<p className="mt-1 text-[11px] text-zinc-600">…and {coveredRows.length - 200} more.</p>
|
||||||
|
)}
|
||||||
|
<p className="mt-2 text-[11px] text-zinc-500">
|
||||||
|
Statements cover:{" "}
|
||||||
|
{(coverage.data ?? []).map((c) => `${c.last4} to ${c.coveredTo}`).join(" · ")}
|
||||||
|
</p>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{inFileDuplicates(editedRows) > 0 && (
|
||||||
|
<p className="mb-3 rounded border border-amber-900/50 bg-amber-950/30 px-3 py-2 text-xs text-amber-300">
|
||||||
|
{inFileDuplicates(editedRows)} row{inFileDuplicates(editedRows) === 1 ? " duplicates another row" : "s duplicate other rows"} in
|
||||||
|
this same file (same date, amount and description). A re-consent can make an
|
||||||
|
aggregator re-export history under new ids — check before importing.
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
<div className="overflow-x-auto rounded border border-zinc-800">
|
<div className="overflow-x-auto rounded border border-zinc-800">
|
||||||
<table className="w-full text-xs">
|
<table className="w-full text-xs">
|
||||||
<thead className="border-b border-zinc-800">
|
<thead className="border-b border-zinc-800">
|
||||||
|
|||||||
@@ -72,7 +72,12 @@ export function OrderDetails({
|
|||||||
const { data: review } = useOrderReview(transactionId);
|
const { data: review } = useOrderReview(transactionId);
|
||||||
const [reviewer, setReviewer] = useState(OWNER_PARTICIPANT_ID);
|
const [reviewer, setReviewer] = useState(OWNER_PARTICIPANT_ID);
|
||||||
|
|
||||||
if (isLoading || !receipt) return null;
|
// `platform` is the marker of a real receipt. Since phase 2 the endpoint also
|
||||||
|
// answers with `{ order_link }` alone for a transaction that is merely LINKED
|
||||||
|
// to an order (a BNPL leg has no receipt of its own), and that object is
|
||||||
|
// truthy — without this check the panel would render a receipt shell with no
|
||||||
|
// merchant, no items and no total.
|
||||||
|
if (isLoading || !receipt || !receipt.platform) return null;
|
||||||
|
|
||||||
const cur = receipt.currency ?? currency ?? "AUD";
|
const cur = receipt.currency ?? currency ?? "AUD";
|
||||||
const fmt = (n: number) => (cur === "AUD" ? `$${n.toFixed(2)}` : `${cur} ${n.toFixed(2)}`);
|
const fmt = (n: number) => (cur === "AUD" ? `$${n.toFixed(2)}` : `${cur} ${n.toFixed(2)}`);
|
||||||
|
|||||||
@@ -9,6 +9,32 @@ export interface ColumnMapping {
|
|||||||
creditCol?: string;
|
creditCol?: string;
|
||||||
merchantCol?: string;
|
merchantCol?: string;
|
||||||
categoryCol?: string;
|
categoryCol?: string;
|
||||||
|
/**
|
||||||
|
* Column holding the account number, when the file covers more than one.
|
||||||
|
*
|
||||||
|
* Optional and usually absent — a bank's own export is one account per file.
|
||||||
|
* An aggregator export (Frollo) is not, and without this every row would be
|
||||||
|
* measured against the same statement coverage.
|
||||||
|
*/
|
||||||
|
accountCol?: string;
|
||||||
|
/**
|
||||||
|
* Column holding the ISO currency code, when the file is multi-currency.
|
||||||
|
*
|
||||||
|
* Without it a foreign row is stored as if it were AUD — a USD 10,782 salary
|
||||||
|
* lands as A$10,782, understating it by about a third. Mapping it records the
|
||||||
|
* native figure with `foreign_currency_code` set and `amount_aud` left NULL,
|
||||||
|
* which is the same shape order ingestion uses and what `AMOUNT_UNCONVERTED`
|
||||||
|
* looks for.
|
||||||
|
*/
|
||||||
|
currencyCol?: string;
|
||||||
|
/**
|
||||||
|
* Column holding the provider's own row id, when it has one.
|
||||||
|
*
|
||||||
|
* Stored as `source_ref` so re-importing the same file inserts nothing. The
|
||||||
|
* importer has no other idempotency: `row_index` is assigned MAX+1 on every
|
||||||
|
* run, which makes `uq_transaction_identity` structurally unable to fire.
|
||||||
|
*/
|
||||||
|
sourceRefCol?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface BankPreset {
|
export interface BankPreset {
|
||||||
@@ -24,6 +50,13 @@ export interface ParsedTransaction {
|
|||||||
transaction_type: string;
|
transaction_type: string;
|
||||||
merchant_name?: string;
|
merchant_name?: string;
|
||||||
category?: string;
|
category?: string;
|
||||||
|
/** Raw account identifier from the file, when a column was mapped. */
|
||||||
|
account?: string;
|
||||||
|
/** Provider row id, when a column was mapped. */
|
||||||
|
source_ref?: string;
|
||||||
|
/** Set only for a non-AUD row: the native amount and its code. */
|
||||||
|
foreign_currency_amount?: number;
|
||||||
|
foreign_currency_code?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function parseCSVRows(text: string): string[][] {
|
export function parseCSVRows(text: string): string[][] {
|
||||||
@@ -108,6 +141,9 @@ export function applyMapping(
|
|||||||
const descIdx = idx(mapping.descriptionCol);
|
const descIdx = idx(mapping.descriptionCol);
|
||||||
const merchantIdx = mapping.merchantCol ? idx(mapping.merchantCol) : -1;
|
const merchantIdx = mapping.merchantCol ? idx(mapping.merchantCol) : -1;
|
||||||
const categoryIdx = mapping.categoryCol ? idx(mapping.categoryCol) : -1;
|
const categoryIdx = mapping.categoryCol ? idx(mapping.categoryCol) : -1;
|
||||||
|
const accountIdx = mapping.accountCol ? idx(mapping.accountCol) : -1;
|
||||||
|
const sourceRefIdx = mapping.sourceRefCol ? idx(mapping.sourceRefCol) : -1;
|
||||||
|
const currencyIdx = mapping.currencyCol ? idx(mapping.currencyCol) : -1;
|
||||||
|
|
||||||
const results: ParsedTransaction[] = [];
|
const results: ParsedTransaction[] = [];
|
||||||
for (const row of dataRows) {
|
for (const row of dataRows) {
|
||||||
@@ -138,6 +174,17 @@ export function applyMapping(
|
|||||||
const tx: ParsedTransaction = { date, description, amount, transaction_type };
|
const tx: ParsedTransaction = { date, description, amount, transaction_type };
|
||||||
if (merchantIdx >= 0 && row[merchantIdx]) tx.merchant_name = row[merchantIdx].trim();
|
if (merchantIdx >= 0 && row[merchantIdx]) tx.merchant_name = row[merchantIdx].trim();
|
||||||
if (categoryIdx >= 0 && row[categoryIdx]) tx.category = row[categoryIdx].trim();
|
if (categoryIdx >= 0 && row[categoryIdx]) tx.category = row[categoryIdx].trim();
|
||||||
|
if (accountIdx >= 0 && row[accountIdx]) tx.account = row[accountIdx].trim();
|
||||||
|
if (sourceRefIdx >= 0 && row[sourceRefIdx]) tx.source_ref = row[sourceRefIdx].trim();
|
||||||
|
if (currencyIdx >= 0) {
|
||||||
|
const code = (row[currencyIdx] ?? "").trim().toUpperCase();
|
||||||
|
// AUD needs no marking; anything else is native and unconverted. No rate
|
||||||
|
// is invented — the statement supplies the AUD figure when it arrives.
|
||||||
|
if (/^[A-Z]{3}$/.test(code) && code !== "AUD") {
|
||||||
|
tx.foreign_currency_code = code;
|
||||||
|
tx.foreign_currency_amount = amount;
|
||||||
|
}
|
||||||
|
}
|
||||||
results.push(tx);
|
results.push(tx);
|
||||||
}
|
}
|
||||||
return results;
|
return results;
|
||||||
@@ -154,3 +201,57 @@ export function loadBankPresets(): BankPreset[] {
|
|||||||
try { return JSON.parse(localStorage.getItem("csv-presets") || "[]"); }
|
try { return JSON.parse(localStorage.getItem("csv-presets") || "[]"); }
|
||||||
catch { return []; }
|
catch { return []; }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Last four digits of an account identifier, however it is written. */
|
||||||
|
export function last4(s: string): string {
|
||||||
|
const d = s.replace(/\D/g, "");
|
||||||
|
return d.length >= 4 ? d.slice(-4) : "";
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Splits parsed rows into what to import and what the statements already have.
|
||||||
|
*
|
||||||
|
* A statement's end date is a hard watermark: everything on that account up to
|
||||||
|
* it is already in the ledger, itemised and converted. Without this the first
|
||||||
|
* aggregator import wrote 422 duplicates out of 550 rows; with it the same
|
||||||
|
* 2,607-row export offers 171.
|
||||||
|
*
|
||||||
|
* Rows are only excluded when an account column was mapped AND that account has
|
||||||
|
* statements. Anything else is imported — a row is never dropped on a guess.
|
||||||
|
*/
|
||||||
|
export function splitByCoverage(
|
||||||
|
rows: ParsedTransaction[],
|
||||||
|
coverage: { last4: string; coveredTo: string }[]
|
||||||
|
): { keep: ParsedTransaction[]; covered: ParsedTransaction[] } {
|
||||||
|
const wm = new Map(coverage.map((c) => [c.last4, c.coveredTo]));
|
||||||
|
const keep: ParsedTransaction[] = [];
|
||||||
|
const covered: ParsedTransaction[] = [];
|
||||||
|
for (const r of rows) {
|
||||||
|
const to = r.account ? wm.get(last4(r.account)) : undefined;
|
||||||
|
// Both are YYYY-MM-DD, so a string compare is a date compare.
|
||||||
|
if (to && r.date <= to) covered.push(r);
|
||||||
|
else keep.push(r);
|
||||||
|
}
|
||||||
|
return { keep, covered };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Rows that duplicate another row in the SAME file.
|
||||||
|
*
|
||||||
|
* Distinct from the coverage check, and not catchable by `source_ref`: a CDR
|
||||||
|
* re-consent makes the provider re-export an account's whole history under
|
||||||
|
* fresh ids while the originals survive, so the twins are identical in
|
||||||
|
* everything except the id. That put 385 duplicates in one 2,563-row export and
|
||||||
|
* doubled every salary payment. 385 rows is not something you spot by eye in a
|
||||||
|
* review table, so it is counted here.
|
||||||
|
*/
|
||||||
|
export function inFileDuplicates(rows: ParsedTransaction[]): number {
|
||||||
|
const seen = new Set<string>();
|
||||||
|
let n = 0;
|
||||||
|
for (const r of rows) {
|
||||||
|
const key = [r.date, r.description, r.amount, r.transaction_type, r.account ?? ""].join("|");
|
||||||
|
if (seen.has(key)) n += 1;
|
||||||
|
else seen.add(key);
|
||||||
|
}
|
||||||
|
return n;
|
||||||
|
}
|
||||||
|
|||||||
+30
-2
@@ -1053,9 +1053,11 @@ export function useImportCSV() {
|
|||||||
return useMutation({
|
return useMutation({
|
||||||
mutationFn: async (body: {
|
mutationFn: async (body: {
|
||||||
bank_name: string;
|
bank_name: string;
|
||||||
|
source?: string;
|
||||||
transactions: {
|
transactions: {
|
||||||
date: string; description: string; amount: number; transaction_type: string;
|
date: string; description: string; amount: number; transaction_type: string;
|
||||||
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string; category?: string;
|
merchant_name?: string; foreign_currency_amount?: number; foreign_currency_code?: string;
|
||||||
|
category?: string; account?: string; source_ref?: string;
|
||||||
}[];
|
}[];
|
||||||
}) => {
|
}) => {
|
||||||
const res = await fetch("/api/import/csv", {
|
const res = await fetch("/api/import/csv", {
|
||||||
@@ -1243,8 +1245,34 @@ export function useOrderDetail(entityKey: string | null) {
|
|||||||
staleTime: 60_000,
|
staleTime: 60_000,
|
||||||
queryFn: async () => {
|
queryFn: async () => {
|
||||||
const res = await fetch(`/api/orders/${encodeURIComponent(entityKey!)}`);
|
const res = await fetch(`/api/orders/${encodeURIComponent(entityKey!)}`);
|
||||||
if (!res.ok) throw new Error("Failed to load order");
|
if (!res.ok) {
|
||||||
|
// Carry the status. Collapsing every failure into one Error is how a
|
||||||
|
// 500 on this route rendered as "that order could not be found" on
|
||||||
|
// every single order for weeks — a server fault wearing the costume of
|
||||||
|
// a data condition, which reads as "the spine is empty" and gets
|
||||||
|
// investigated nowhere near the actual bug.
|
||||||
|
const err = new Error(res.status === 404 ? "Order not found" : "Failed to load order");
|
||||||
|
(err as Error & { status?: number }).status = res.status;
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
return res.json();
|
return res.json();
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How far each account's statements already reach, for the CSV import modal.
|
||||||
|
*
|
||||||
|
* Fetched once when the modal opens: the review step uses it to drop rows the
|
||||||
|
* ledger already holds rather than offering them for import.
|
||||||
|
*/
|
||||||
|
export function useStatementCoverage() {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ["statement-coverage"],
|
||||||
|
queryFn: async () => {
|
||||||
|
const res = await fetch("/api/import/statement-coverage");
|
||||||
|
if (!res.ok) throw new Error("Could not load statement coverage");
|
||||||
|
return (await res.json()).coverage as { last4: string; coveredTo: string }[];
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
/**
|
||||||
|
* Presentational tidy for a merchant name (board 212).
|
||||||
|
*
|
||||||
|
* PRESENTATION ONLY. It merges nothing and must never be used to decide that
|
||||||
|
* two rows are the same merchant. Real unification is the merchant-alias bridge
|
||||||
|
* (ticket 176, `jobs/merchant_merge.py`), which has already merged what can be
|
||||||
|
* merged safely; what is left on screen is one entity looking scruffy, not two
|
||||||
|
* entities that should be one.
|
||||||
|
*
|
||||||
|
* `Amazon.in` MUST NOT become `amazon.com.au`. They are different marketplaces
|
||||||
|
* with different currency and geography — a genuine country distinction, not a
|
||||||
|
* name variant. `Kogan.com`, `GOG.com`, `AliExpress.com` and `Catch.com.au` are
|
||||||
|
* real brand names that happen to contain a TLD and are already correct.
|
||||||
|
*
|
||||||
|
* Two rules, both mechanical:
|
||||||
|
* 1. drop trailing corporate suffixes
|
||||||
|
* 2. capitalise a name that arrived lower-cased FROM A DOMAIN
|
||||||
|
*/
|
||||||
|
|
||||||
|
const CORP_SUFFIX =
|
||||||
|
/,?\s+(pty\.?\s+ltd\.?|pty\.?\s+limited|pte\.?\s+ltd\.?|pte\.?|p\/l|ltd\.?|limited|inc\.?|llc|pbc|gmbh|b\.?v\.?|s\.?a\.?r\.?l\.?|oü|co\.?)$/i;
|
||||||
|
|
||||||
|
/** A bare domain used as a name: all lower case, and a dotted TLD. */
|
||||||
|
const BARE_DOMAIN = /^[a-z0-9][a-z0-9.-]*\.[a-z]{2,}(\.[a-z]{2,})?$/;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stripping must not leave a dangling connector. `Coburger & Co` is a brand
|
||||||
|
* whose last word happens to match the suffix list, and removing it produced
|
||||||
|
* `Coburger &` — visibly broken, and the kind of thing that only shows up when
|
||||||
|
* you run the rule over the real corpus rather than over examples you chose.
|
||||||
|
*/
|
||||||
|
const DANGLING = /(\s[&+-]|\sand)$/i;
|
||||||
|
|
||||||
|
export function tidyMerchant(name: string): string {
|
||||||
|
let n = name.replace(/\s+/g, " ").trim();
|
||||||
|
|
||||||
|
// At most two passes: "Samsung Electronics Co. Ltd." needs Ltd. then Co.
|
||||||
|
for (let i = 0; i < 2; i++) {
|
||||||
|
const stripped = n.replace(CORP_SUFFIX, "").trim().replace(/,$/, "");
|
||||||
|
if (stripped === n || stripped.length < 3 || DANGLING.test(stripped)) break;
|
||||||
|
n = stripped;
|
||||||
|
}
|
||||||
|
|
||||||
|
// "amazon.com.au" reads as a machine artefact; "Amazon.com.au" reads as a
|
||||||
|
// name. Gated on BARE_DOMAIN, not on "starts with a lower-case letter": the
|
||||||
|
// looser test turned `eBay Commerce Australia Pty Ltd.` into `EBay Commerce
|
||||||
|
// Australia`, mangling a brand that is deliberately lower-cased.
|
||||||
|
if (BARE_DOMAIN.test(n)) n = n[0].toUpperCase() + n.slice(1);
|
||||||
|
|
||||||
|
return n;
|
||||||
|
}
|
||||||
+118
-15
@@ -73,6 +73,18 @@ export interface OrderRow {
|
|||||||
/** Merchant orders on a regular cadence: the subscription signal, derived
|
/** Merchant orders on a regular cadence: the subscription signal, derived
|
||||||
* not extracted. NULL unless the gaps are tight relative to their mean. */
|
* not extracted. NULL unless the gaps are tight relative to their mean. */
|
||||||
cadence_days: number | null;
|
cadence_days: number | null;
|
||||||
|
/** Worst sender-authentication verdict across the order's source documents.
|
||||||
|
* 93% are 'pass' — the badge exists for the 7% that are not. Do NOT badge
|
||||||
|
* source_trust instead: it is 'untrusted_external' on 100% of rows, because
|
||||||
|
* every order here came from email. */
|
||||||
|
auth_verdict: string | null;
|
||||||
|
injection_flagged: boolean | null;
|
||||||
|
/** An annual tax/donation receipt restating a year of payments already made,
|
||||||
|
* not a purchase (board 211). It stays on this page — the figure is the
|
||||||
|
* year's giving total and is worth seeing — but `order_spend` excludes it,
|
||||||
|
* so the amount here is deliberately NOT part of any total on the page.
|
||||||
|
* Badge it, or a $2,376 row reads as a single purchase. */
|
||||||
|
summarises_period: boolean;
|
||||||
txn_count: number;
|
txn_count: number;
|
||||||
first_txn_id: number | null;
|
first_txn_id: number | null;
|
||||||
}
|
}
|
||||||
@@ -96,9 +108,22 @@ export interface OrderFilters {
|
|||||||
/**
|
/**
|
||||||
* Rows with no amount, no order reference and a single lifecycle event are not
|
* Rows with no amount, no order reference and a single lifecycle event are not
|
||||||
* purchases — they are a second entity minted from a mail describing an order
|
* purchases — they are a second entity minted from a mail describing an order
|
||||||
* that already exists (Amazon "share your experience", DoorDash no-contact
|
* that already exists (DoorDash no-contact delivery details, shipment notices).
|
||||||
* delivery details, shipment notices). 755 of them; hiding lifts amount
|
* 620 of them as of 2026-08-12.
|
||||||
* coverage from 74% to 84%.
|
*
|
||||||
|
* The count moved three times that day and the history is the point:
|
||||||
|
* 755 when this predicate was written
|
||||||
|
* 904 after board 219 turned unstated zeros into NULLs — those 149 rows had
|
||||||
|
* been failing the "no amount" test on a technicality
|
||||||
|
* 699 after board 210a retired 205 Amazon "share your experience" mails,
|
||||||
|
* which were never lifecycle events at all (ingest/order_noise.py)
|
||||||
|
* 620 after board 210b joined 79 phantoms to their real parent
|
||||||
|
* (jobs/phantom_order_joiner.py, placebo-verified at a 1.9% error rate)
|
||||||
|
*
|
||||||
|
* What is left is the corpus coverage ceiling rather than a defect: a mail
|
||||||
|
* whose parent is not in the corpus, or whose merchant is ambiguous, has
|
||||||
|
* nothing to join to. This stays a UI workaround for those, and the toggle
|
||||||
|
* still says so.
|
||||||
*
|
*
|
||||||
* This is a WORKAROUND for board 210, not a fix. The upstream defect is that a
|
* This is a WORKAROUND for board 210, not a fix. The upstream defect is that a
|
||||||
* lifecycle mail printing no order reference cannot join its own order, so it
|
* lifecycle mail printing no order reference cannot join its own order, so it
|
||||||
@@ -108,6 +133,14 @@ export interface OrderFilters {
|
|||||||
const LIFECYCLE_ONLY = `NOT (
|
const LIFECYCLE_ONLY = `NOT (
|
||||||
f.order_total IS NULL
|
f.order_total IS NULL
|
||||||
AND f.reference_source = 'message_id_fallback'
|
AND f.reference_source = 'message_id_fallback'
|
||||||
|
-- TRIED AND REVERTED, 2026-08-12: exempting rows that carry line items, on
|
||||||
|
-- the theory that naming what was bought proves a purchase. It surfaced 442
|
||||||
|
-- rows and the very first window showed why it is wrong — the Amazon
|
||||||
|
-- "Ordered:" mail that mints a msg- phantom carries the item list too, so
|
||||||
|
-- Xelsluthe and YIWENTEC each rendered TWICE again, which is the exact
|
||||||
|
-- complaint this page was fixed for. Items do not separate an orphan
|
||||||
|
-- purchase from an item-carrying echo; only real deduplication does, and
|
||||||
|
-- that is board 210, deliberately not attempted here.
|
||||||
AND (SELECT count(*) FROM extracted_facts ef
|
AND (SELECT count(*) FROM extracted_facts ef
|
||||||
WHERE ef.fact_type = 'order_event'
|
WHERE ef.fact_type = 'order_event'
|
||||||
AND ef.payload->>'_order_entity_key' = f.entity_key) <= 1
|
AND ef.payload->>'_order_entity_key' = f.entity_key) <= 1
|
||||||
@@ -180,18 +213,41 @@ function buildWhere(filters: OrderFilters) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Ledger linkage. Phase 1 reads expense_metadata; phase 2 swaps this lateral
|
* Ledger linkage — phase 2 (board 205).
|
||||||
* AND the detail query AND /api/transactions/[id]/order together — moving only
|
*
|
||||||
* one of the three leaves the list contradicting the detail page.
|
* `order_transaction_links` is the source of truth: it is many-to-many, so a
|
||||||
|
* BNPL plan is four rows against one order. `expense_metadata` remains as a
|
||||||
|
* FALLBACK because `jobs/order_transaction_bridge.py` still writes there on
|
||||||
|
* every hourly tick; until a bridge row is mirrored into links, dropping the
|
||||||
|
* fallback would make freshly bridged orders read as unpaid.
|
||||||
|
*
|
||||||
|
* UNION, not UNION ALL, on transaction_id: after the 0030 backfill the same
|
||||||
|
* charge is legitimately present in both stores, and counting it twice would
|
||||||
|
* show "2 charges" on a single-payment order.
|
||||||
|
*
|
||||||
|
* All four readers move together — this lateral, the detail query below,
|
||||||
|
* /api/transactions/[id]/order, and the order_ctx lateral in queries.ts.
|
||||||
|
* Moving fewer is not a smaller change, it is an inconsistent one: once the
|
||||||
|
* matcher inserts four links for order_ebay_14-11714-95953 that order still
|
||||||
|
* has no expense_metadata row, so the detail page would show four instalments
|
||||||
|
* while the list showed txn_count = 0 and the has_transaction facet put it on
|
||||||
|
* the wrong side of both filters.
|
||||||
*/
|
*/
|
||||||
const LINK_LATERAL = `
|
const LINK_LATERAL = `
|
||||||
LEFT JOIN LATERAL (
|
LEFT JOIN LATERAL (
|
||||||
SELECT count(*)::int AS txn_count,
|
SELECT count(*)::int AS txn_count, min(txn_id) AS first_txn_id
|
||||||
min(em.matched_transaction_id) AS first_txn_id
|
FROM (
|
||||||
|
SELECT l.transaction_id AS txn_id
|
||||||
|
FROM order_transaction_links l
|
||||||
|
WHERE l.entity_key = f.entity_key
|
||||||
|
UNION
|
||||||
|
SELECT COALESCE(em.matched_transaction_id, em.transaction_id)
|
||||||
FROM expense_metadata em
|
FROM expense_metadata em
|
||||||
WHERE em.source = 'order-bridge'
|
WHERE em.source = 'order-bridge'
|
||||||
AND (em.source_message_id = f.entity_key
|
AND (em.source_message_id = f.entity_key
|
||||||
OR em.source_message_id LIKE f.entity_key || '#f%')
|
OR em.source_message_id LIKE f.entity_key || '#f%')
|
||||||
|
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
|
||||||
|
) both_stores
|
||||||
) link ON true`;
|
) link ON true`;
|
||||||
|
|
||||||
const FROM_CLAUSE = `
|
const FROM_CLAUSE = `
|
||||||
@@ -220,6 +276,7 @@ export async function getOrderFeed(filters: OrderFilters) {
|
|||||||
f.ordered_at, f.eta_date, f.delivered_at,
|
f.ordered_at, f.eta_date, f.delivered_at,
|
||||||
f.currency, f.order_total, f.line_item_count,
|
f.currency, f.order_total, f.line_item_count,
|
||||||
f.content_class, f.display_name, f.cadence_days,
|
f.content_class, f.display_name, f.cadence_days,
|
||||||
|
f.auth_verdict, f.injection_flagged, f.summarises_period,
|
||||||
m.canonical_name AS merchant_name,
|
m.canonical_name AS merchant_name,
|
||||||
link.txn_count, link.first_txn_id,
|
link.txn_count, link.first_txn_id,
|
||||||
(SELECT sp.refunded_amount FROM order_spend sp
|
(SELECT sp.refunded_amount FROM order_spend sp
|
||||||
@@ -334,6 +391,12 @@ export interface OrderLinkedTxn {
|
|||||||
description: string;
|
description: string;
|
||||||
amount: string;
|
amount: string;
|
||||||
source_message_id: string;
|
source_message_id: string;
|
||||||
|
/** 'charge' | 'shipment' | 'instalment' | 'refund' | 'fee'. A plan's legs are
|
||||||
|
* four 'instalment' rows; a whole-order card charge is one 'charge'. */
|
||||||
|
leg_kind: string;
|
||||||
|
/** Human position, "2 of 4" — populated for instalments only. */
|
||||||
|
leg_index: number | null;
|
||||||
|
leg_count: number | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface OrderDetail {
|
export interface OrderDetail {
|
||||||
@@ -358,6 +421,9 @@ export interface OrderDetail {
|
|||||||
tracking_carrier: string | null;
|
tracking_carrier: string | null;
|
||||||
line_items: { description?: string; quantity?: number; amount?: number }[];
|
line_items: { description?: string; quantity?: number; amount?: number }[];
|
||||||
is_settled_duplicate: boolean;
|
is_settled_duplicate: boolean;
|
||||||
|
/** Board 211 — an annual receipt restating a year of payments already made.
|
||||||
|
* Same shape as is_settled_duplicate: browsable, excluded from spend. */
|
||||||
|
summarises_period: boolean;
|
||||||
content_class: string | null;
|
content_class: string | null;
|
||||||
display_name: string;
|
display_name: string;
|
||||||
cadence_days: number | null;
|
cadence_days: number | null;
|
||||||
@@ -385,6 +451,7 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
|
|||||||
o.order_total, o.tracking_url, o.tracking_carrier,
|
o.order_total, o.tracking_url, o.tracking_carrier,
|
||||||
m.canonical_name AS merchant_name,
|
m.canonical_name AS merchant_name,
|
||||||
(o.settles_entity_id IS NOT NULL) AS is_settled_duplicate,
|
(o.settles_entity_id IS NOT NULL) AS is_settled_duplicate,
|
||||||
|
o.summarises_period,
|
||||||
COALESCE(o.details->'line_items', '[]'::jsonb) AS line_items,
|
COALESCE(o.details->'line_items', '[]'::jsonb) AS line_items,
|
||||||
sp.order_total AS net_total,
|
sp.order_total AS net_total,
|
||||||
sp.gross_total,
|
sp.gross_total,
|
||||||
@@ -392,14 +459,29 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
|
|||||||
ctx.content_class,
|
ctx.content_class,
|
||||||
COALESCE(me2.canonical_name, ctx.counterparty, o.platform) AS display_name,
|
COALESCE(me2.canonical_name, ctx.counterparty, o.platform) AS display_name,
|
||||||
rec.cadence_days,
|
rec.cadence_days,
|
||||||
rec.order_count
|
-- ::int is mandatory. order_merchant_cadence.order_count is a
|
||||||
|
-- count(*), so BIGINT, and JSON.stringify throws outright on a
|
||||||
|
-- BigInt — "Do not know how to serialize a BigInt", a 500 with no
|
||||||
|
-- SQL in it. It only fires on merchants that HAVE a cadence, so
|
||||||
|
-- most orders open fine and the recurring ones die; testing one
|
||||||
|
-- arbitrary order will not find it.
|
||||||
|
rec.order_count::int AS order_count
|
||||||
FROM entities e
|
FROM entities e
|
||||||
JOIN entity_orders o ON o.entity_id = e.id
|
JOIN entity_orders o ON o.entity_id = e.id
|
||||||
LEFT JOIN entities m ON m.id = o.merchant_entity_id
|
LEFT JOIN entities m ON m.id = o.merchant_entity_id
|
||||||
LEFT JOIN entities me2 ON me2.id = o.merchant_entity_id
|
LEFT JOIN entities me2 ON me2.id = o.merchant_entity_id
|
||||||
LEFT JOIN order_platforms p ON p.slug = o.platform
|
LEFT JOIN order_platforms p ON p.slug = o.platform
|
||||||
LEFT JOIN order_spend sp ON sp.entity_id = o.entity_id
|
LEFT JOIN order_spend sp ON sp.entity_id = o.entity_id
|
||||||
LEFT JOIN order_merchant_cadence rec ON rec.merchant_entity_id = o.merchant_entity_id
|
-- Cadence is keyed on the NORMALISED MERCHANT NAME, not merchant_entity_id.
|
||||||
|
-- Migration 021 rekeyed the view (that column is NULL on ~9% of the feed
|
||||||
|
-- and duplicate merchant entities split the same shop's history); 027 and
|
||||||
|
-- 029 moved order_feed/order_spend onto merchant_key, and this query was
|
||||||
|
-- missed. The result was a 42703 on EVERY order detail page — the join
|
||||||
|
-- referenced a column the view no longer had. Keep this expression
|
||||||
|
-- identical to the one in migration 027.
|
||||||
|
LEFT JOIN order_merchant_cadence rec
|
||||||
|
ON rec.merchant_key = regexp_replace(
|
||||||
|
lower(COALESCE(me2.canonical_name, o.platform)), '[^a-z0-9]', '', 'g')
|
||||||
LEFT JOIN LATERAL (
|
LEFT JOIN LATERAL (
|
||||||
SELECT di.content_class, di.counterparty
|
SELECT di.content_class, di.counterparty
|
||||||
FROM extracted_facts ef
|
FROM extracted_facts ef
|
||||||
@@ -445,14 +527,35 @@ export async function getOrderDetail(entityKey: string): Promise<OrderDetail | n
|
|||||||
);
|
);
|
||||||
|
|
||||||
const transactions = await queryRaw<OrderLinkedTxn>(
|
const transactions = await queryRaw<OrderLinkedTxn>(
|
||||||
`SELECT t.id::int AS transaction_id, t.transaction_date, t.description,
|
// Links first, expense_metadata as fallback, deduped on transaction_id —
|
||||||
t.amount, em.source_message_id
|
// see LINK_LATERAL above for why both stores are read. DISTINCT ON keeps
|
||||||
|
// the LINK row when a transaction is in both, because only that row knows
|
||||||
|
// whether the payment was one charge or leg 3 of 4.
|
||||||
|
// The DISTINCT ON must be ordered by t.id, so the chronological sort the
|
||||||
|
// page needs goes on the OUTER query — legs read as "1, 2, 3, 4" only if
|
||||||
|
// they come back by date.
|
||||||
|
`SELECT * FROM (
|
||||||
|
SELECT DISTINCT ON (t.id)
|
||||||
|
t.id::int AS transaction_id, t.transaction_date, t.description,
|
||||||
|
t.amount, both_stores.source_message_id,
|
||||||
|
both_stores.leg_kind, both_stores.leg_index, both_stores.leg_count
|
||||||
|
FROM (
|
||||||
|
SELECT l.transaction_id, l.leg_kind, l.leg_index::int, l.leg_count::int,
|
||||||
|
l.entity_key AS source_message_id, 0 AS pref
|
||||||
|
FROM order_transaction_links l
|
||||||
|
WHERE l.entity_key = $1
|
||||||
|
UNION ALL
|
||||||
|
SELECT COALESCE(em.matched_transaction_id, em.transaction_id),
|
||||||
|
'charge', NULL::int, NULL::int, em.source_message_id, 1
|
||||||
FROM expense_metadata em
|
FROM expense_metadata em
|
||||||
JOIN transactions t
|
|
||||||
ON t.id = COALESCE(em.matched_transaction_id, em.transaction_id)
|
|
||||||
WHERE em.source = 'order-bridge'
|
WHERE em.source = 'order-bridge'
|
||||||
AND (em.source_message_id = $1 OR em.source_message_id LIKE $1 || '#f%')
|
AND (em.source_message_id = $1 OR em.source_message_id LIKE $1 || '#f%')
|
||||||
ORDER BY t.transaction_date`,
|
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
|
||||||
|
) both_stores
|
||||||
|
JOIN transactions t ON t.id = both_stores.transaction_id
|
||||||
|
ORDER BY t.id, both_stores.pref
|
||||||
|
) deduped
|
||||||
|
ORDER BY transaction_date, leg_index NULLS FIRST`,
|
||||||
[entityKey]
|
[entityKey]
|
||||||
);
|
);
|
||||||
|
|
||||||
|
|||||||
+118
-5
@@ -37,7 +37,16 @@ export interface TransactionRow {
|
|||||||
payment_method: string | null;
|
payment_method: string | null;
|
||||||
/** Uber pick-up/drop-off, when this row came from an order receipt. */
|
/** Uber pick-up/drop-off, when this row came from an order receipt. */
|
||||||
order_route: RoutePointRow[] | null;
|
order_route: RoutePointRow[] | null;
|
||||||
|
/** Receipt platform ONLY — it gates the disclosure arrow, so it must stay
|
||||||
|
* true to "there is a receipt behind this row". */
|
||||||
order_platform: "doordash" | "ubereats" | "uber" | null;
|
order_platform: "doordash" | "ubereats" | "uber" | null;
|
||||||
|
/** Phase 2 (board 205) — set when this transaction is linked to an order.
|
||||||
|
* 'instalment' means it is one leg of a plan; leg_index/leg_count carry
|
||||||
|
* "2 of 4", which is what stops one purchase reading as four. */
|
||||||
|
order_leg_kind: string | null;
|
||||||
|
order_leg_index: number | null;
|
||||||
|
order_leg_count: number | null;
|
||||||
|
order_name: string | null;
|
||||||
// override fields
|
// override fields
|
||||||
category_override: string | null;
|
category_override: string | null;
|
||||||
merchant_override: string | null;
|
merchant_override: string | null;
|
||||||
@@ -314,7 +323,17 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
|
|||||||
txn_tags.tags,
|
txn_tags.tags,
|
||||||
txn_splits.splits,
|
txn_splits.splits,
|
||||||
order_ctx.route as order_route,
|
order_ctx.route as order_route,
|
||||||
order_ctx.platform as order_platform
|
-- Deliberately NOT COALESCEd with the link's platform. order_platform
|
||||||
|
-- gates the receipt disclosure arrow on /transactions, and a BNPL leg
|
||||||
|
-- has no receipt behind it — filling this in put an arrow on four
|
||||||
|
-- Afterpay rows that expand to nothing, which is the exact promise the
|
||||||
|
-- arrow exists to avoid making. The leg fields below carry the sub-line
|
||||||
|
-- instead.
|
||||||
|
order_ctx.platform as order_platform,
|
||||||
|
order_link.leg_kind as order_leg_kind,
|
||||||
|
order_link.leg_index as order_leg_index,
|
||||||
|
order_link.leg_count as order_leg_count,
|
||||||
|
order_link.canonical_name as order_name
|
||||||
FROM transactions t
|
FROM transactions t
|
||||||
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
|
LEFT JOIN transaction_overrides o ON o.transaction_id = t.id
|
||||||
LEFT JOIN statements s ON s.id = t.statement_id
|
LEFT JOIN statements s ON s.id = t.statement_id
|
||||||
@@ -323,12 +342,27 @@ export async function getTransactions(ownerId: number, filters: TransactionFilte
|
|||||||
-- only thing that tells them apart, and it was already stored.
|
-- only thing that tells them apart, and it was already stored.
|
||||||
-- Both directions, because a card-settled order has no transaction of its
|
-- Both directions, because a card-settled order has no transaction of its
|
||||||
-- own and points at the statement line instead (I5).
|
-- own and points at the statement line instead (I5).
|
||||||
|
-- Phase 2 (board 205): a BNPL leg has no expense_metadata row of its own,
|
||||||
|
-- so COALESCE in the link's platform. Without it the four Afterpay debits
|
||||||
|
-- behind the A$1,599 drone keep an empty sub-line while their order sits
|
||||||
|
-- one join away. The route column only ever exists on the receipt side.
|
||||||
|
-- (No backticks in here: this SQL lives in a TS template literal and a
|
||||||
|
-- backtick ends the string — TS1005 on a line that looks like a comment.)
|
||||||
LEFT JOIN LATERAL (
|
LEFT JOIN LATERAL (
|
||||||
SELECT em.route, em.platform
|
SELECT em.route, em.platform
|
||||||
FROM expense_metadata em
|
FROM expense_metadata em
|
||||||
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
WHERE em.transaction_id = t.id OR em.matched_transaction_id = t.id
|
||||||
LIMIT 1
|
LIMIT 1
|
||||||
) order_ctx ON true
|
) order_ctx ON true
|
||||||
|
LEFT JOIN LATERAL (
|
||||||
|
SELECT o.platform, l.leg_kind, l.leg_index::int AS leg_index,
|
||||||
|
l.leg_count::int AS leg_count, e.canonical_name
|
||||||
|
FROM order_transaction_links l
|
||||||
|
LEFT JOIN entities e ON e.entity_key = l.entity_key
|
||||||
|
LEFT JOIN entity_orders o ON o.entity_id = e.id
|
||||||
|
WHERE l.transaction_id = t.id
|
||||||
|
LIMIT 1
|
||||||
|
) order_link ON true
|
||||||
LEFT JOIN participants p ON p.id = COALESCE(t.owner_id, s.owner_id)
|
LEFT JOIN participants p ON p.id = COALESCE(t.owner_id, s.owner_id)
|
||||||
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
|
LEFT JOIN transactions src ON src.reconciled_with_id = t.id AND src.statement_id IS NULL
|
||||||
LEFT JOIN trips tr ON tr.id = o.trip_id
|
LEFT JOIN trips tr ON tr.id = o.trip_id
|
||||||
@@ -636,8 +670,11 @@ export async function batchInsertCSVTransactions(
|
|||||||
foreign_currency_amount?: number;
|
foreign_currency_amount?: number;
|
||||||
foreign_currency_code?: string;
|
foreign_currency_code?: string;
|
||||||
category?: string;
|
category?: string;
|
||||||
|
account?: string;
|
||||||
|
source_ref?: string;
|
||||||
}[],
|
}[],
|
||||||
tagId: number
|
tagId: number,
|
||||||
|
source?: string
|
||||||
): Promise<number> {
|
): Promise<number> {
|
||||||
if (rows.length === 0) return 0;
|
if (rows.length === 0) return 0;
|
||||||
|
|
||||||
@@ -651,13 +688,32 @@ export async function batchInsertCSVTransactions(
|
|||||||
const params: unknown[] = [ownerId];
|
const params: unknown[] = [ownerId];
|
||||||
let p = 2;
|
let p = 2;
|
||||||
rows.forEach((r, i) => {
|
rows.forEach((r, i) => {
|
||||||
valueClauses.push(`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`);
|
valueClauses.push(
|
||||||
params.push(r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null, r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null);
|
`(NULL, $1, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}, ${base + 1 + i})`
|
||||||
|
);
|
||||||
|
params.push(
|
||||||
|
r.date, r.description, r.amount, r.transaction_type, r.merchant_name ?? null,
|
||||||
|
r.foreign_currency_amount ?? null, r.foreign_currency_code ?? null,
|
||||||
|
// The category the reviewer chose. It used to be accepted here and then
|
||||||
|
// left out of the column list, so every category picked in the review
|
||||||
|
// step was silently discarded and the trigger wrote 'other'. That is not
|
||||||
|
// cosmetic: an uncategorised credit is admitted by NET_SPEND_ROWS and
|
||||||
|
// negated by SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of
|
||||||
|
// spend and none of them counted as income.
|
||||||
|
r.category ?? null,
|
||||||
|
source ?? null,
|
||||||
|
// source + source_ref is the only idempotency this path has. Without it a
|
||||||
|
// second import of the same file duplicates every row, because row_index
|
||||||
|
// is assigned MAX+1 and uq_transaction_identity can never fire.
|
||||||
|
r.source_ref ?? null,
|
||||||
|
r.account ?? null
|
||||||
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
const txIds = await queryRaw<{ id: number }>(
|
const txIds = await queryRaw<{ id: number }>(
|
||||||
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, row_index)
|
`INSERT INTO transactions (statement_id, owner_id, transaction_date, description, amount, transaction_type, merchant_name, foreign_currency_amount, foreign_currency_code, category, source, source_ref, source_account, row_index)
|
||||||
VALUES ${valueClauses.join(", ")}
|
VALUES ${valueClauses.join(", ")}
|
||||||
|
ON CONFLICT (source, source_ref) WHERE source IS NOT NULL AND source_ref IS NOT NULL DO NOTHING
|
||||||
RETURNING id`,
|
RETURNING id`,
|
||||||
params
|
params
|
||||||
);
|
);
|
||||||
@@ -698,6 +754,27 @@ export async function batchInsertCSVTransactions(
|
|||||||
export const needsCardMatch = (alias = "t") =>
|
export const needsCardMatch = (alias = "t") =>
|
||||||
`(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`;
|
`(${alias}.payment_method IS NULL OR ${alias}.payment_method NOT IN ('cash', 'credits'))`;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* REMOVED 2026-08-13. There used to be an `awaitsStatementLine()` predicate
|
||||||
|
* here, excluding account-feed rows from the pending-reconciliation queue.
|
||||||
|
*
|
||||||
|
* Its premise — a feed row is the account's own ledger entry, so no statement
|
||||||
|
* line is coming — was asserted and never tested. It was false for almost every
|
||||||
|
* account: 422 of the first 550 imported rows already had a statement twin.
|
||||||
|
*
|
||||||
|
* How it got in matters more than what it did. The queue jumped from 8 to 558
|
||||||
|
* the moment the feed landed, and that jump was read as noise and filtered
|
||||||
|
* away. The queue was right — those rows genuinely were provisional entries
|
||||||
|
* awaiting statement lines — and hiding them removed the only mechanism that
|
||||||
|
* would ever have collapsed them, so a transient overlap became a permanent
|
||||||
|
* double-count and stayed invisible until a human saw one salary payment listed
|
||||||
|
* twice in two currencies.
|
||||||
|
*
|
||||||
|
* An imported row that has no statement belongs in the queue. Volume is solved
|
||||||
|
* upstream, by not importing what the statements already cover
|
||||||
|
* (`getStatementCoverage`), never downstream by hiding what is there.
|
||||||
|
*/
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Bank label for a transaction. A row with no statement was not imported from
|
* Bank label for a transaction. A row with no statement was not imported from
|
||||||
* one, and the label has to say *why*: "Manual" reads as "hand-entered, still
|
* one, and the label has to say *why*: "Manual" reads as "hand-entered, still
|
||||||
@@ -1467,3 +1544,39 @@ export async function getTagTransactionIds(tagId: number): Promise<number[]> {
|
|||||||
);
|
);
|
||||||
return rows.map((r) => r.transaction_id);
|
return rows.map((r) => r.transaction_id);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The last date each account's statements cover, keyed by the account's last
|
||||||
|
* four digits.
|
||||||
|
*
|
||||||
|
* This is the coverage test that matters, and it is not the one that was tried
|
||||||
|
* first. The first attempt asked whether a row's date fell inside a statement's
|
||||||
|
* min-max *window*, which for accounts whose statements span 182 to 460 days
|
||||||
|
* swallows a year and answers nothing — 422 duplicates out of 550 rows got in
|
||||||
|
* that way. `billing_end_date` asks a question that has an answer: up to what
|
||||||
|
* date is this account complete?
|
||||||
|
*
|
||||||
|
* Amount matching cannot substitute for it. Two sources decompose the same
|
||||||
|
* event differently — an aggregator bundles the Wise transfer fee into the
|
||||||
|
* transfer (10001.13) where the statement itemises it (10000.00 + 1.13) — so
|
||||||
|
* the rows are the same money with different numbers.
|
||||||
|
*
|
||||||
|
* Keyed on last4 because account numbers are written differently everywhere
|
||||||
|
* ("235242176", "xxxxxxxxxxxx2176", "4085-56264"). Where two banks share a
|
||||||
|
* last4 the later date wins, which errs towards excluding a row rather than
|
||||||
|
* duplicating one; the caller shows every watermark so a wrong one is visible.
|
||||||
|
*/
|
||||||
|
export async function getStatementCoverage(): Promise<{ last4: string; coveredTo: string }[]> {
|
||||||
|
const rows = await queryRaw<{ last4: string; covered_to: string }>(
|
||||||
|
`SELECT right(regexp_replace(account_number, '[^0-9]', '', 'g'), 4) AS last4,
|
||||||
|
MAX(billing_end_date)::text AS covered_to
|
||||||
|
FROM statements
|
||||||
|
WHERE account_number IS NOT NULL
|
||||||
|
GROUP BY 1`,
|
||||||
|
[]
|
||||||
|
);
|
||||||
|
return rows
|
||||||
|
.filter((r) => r.last4 && r.last4.length === 4)
|
||||||
|
.map((r) => ({ last4: r.last4, coveredTo: r.covered_to }))
|
||||||
|
.sort((a, b) => a.last4.localeCompare(b.last4));
|
||||||
|
}
|
||||||
|
|||||||
@@ -9,6 +9,10 @@
|
|||||||
"esModuleInterop": true,
|
"esModuleInterop": true,
|
||||||
"module": "esnext",
|
"module": "esnext",
|
||||||
"moduleResolution": "bundler",
|
"moduleResolution": "bundler",
|
||||||
|
// scripts/*.mts run under `node --experimental-strip-types`, which resolves
|
||||||
|
// ESM specifiers literally and so needs the `.ts` extension written out.
|
||||||
|
// Legal here because noEmit is set — nothing is rewritten on the way out.
|
||||||
|
"allowImportingTsExtensions": true,
|
||||||
"resolveJsonModule": true,
|
"resolveJsonModule": true,
|
||||||
"isolatedModules": true,
|
"isolatedModules": true,
|
||||||
"jsx": "react-jsx",
|
"jsx": "react-jsx",
|
||||||
|
|||||||
Reference in New Issue
Block a user