orders phase 2: order_transaction_links, and move every reader to it (board 205)
ci / lint-test (push) Successful in 41s
ci / lint-test (push) Successful in 41s
Phase 1 read linkage out of expense_metadata, which is shaped as ONE ROW PER
TRANSACTION — transaction_id UNIQUE, matched_transaction_id partial-unique — so
every multiplicity it expresses is smuggled through a string key (0029 keys
split shipments <entity_key>#f<fact_id>). A BNPL plan needs four rows against
one order and has no such trick available.
Migration 0030 adds order_transaction_links (many-to-many, keyed on entity_key
rather than entities.id: finance-app does not model the spine and must not hold
an FK across a boundary a re-extraction can decompile) and backfills all 63
existing bridge rows.
THE UNIQUENESS RULE WAS WRONG FIRST TIME, in the most on-brand way available.
It read (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of
the same order — both leg_index NULL — collapsed to one key and ON CONFLICT DO
NOTHING dropped one SILENTLY. Caught only because the backfill reported 62
against 63 candidates. The row it ate was order_amazon_249-4859367-0690246's
$130.00 second shipment, the same order named in migration 0029's comment as
the reason split shipments need distinct keys at all. What identifies a leg
depends on its kind: an instalment by its INDEX, a shipment by its FACT, a
whole-order charge by neither.
ALL FOUR READERS MOVE TOGETHER, links first with expense_metadata as fallback:
- LINK_LATERAL (list txn_count/first_txn_id, and the has_transaction facets
that read it)
- the detail page's transactions query
- /api/transactions/[id]/order
- the order_ctx lateral in queries.ts
Moving fewer is not a smaller change, it is an inconsistent one: the matcher's
four links for order_ebay_14-11714-95953 come with no expense_metadata row, so
a half-move would show four instalments on the detail page while the list said
txn_count = 0 and put the order on the wrong side of BOTH has_transaction
filters. Verified after: detail 4 legs, list txn_count 4, has_transaction=yes
includes it, =no excludes it.
UNION not UNION ALL on transaction_id — after the backfill the same charge is
legitimately in both stores and counting it twice would show "2 charges" on a
single-payment order.
order_platform is deliberately NOT coalesced with the link's platform. It gates
the receipt disclosure arrow on /transactions, and a BNPL leg has no receipt
behind it — filling it in put an arrow on four Afterpay rows that expand to
nothing, which is the exact promise the arrow exists to avoid making. Separate
leg fields carry the sub-line instead ("1 of 4 - DJI Air 3 Fly More Combo").
order-details.tsx now also requires a real receipt (platform present) before
rendering, because the endpoint can answer with a link alone.
Two invariant views, both empty and expected to stay so: order_link_orphans
(spine re-keying silently orphans a TEXT key — bridge links are rebuildable,
`manual` ones are lost curation) and order_link_drift (the two stores
disagreeing). The plan's suggested fix for drift — widening the bridge's NOT
EXISTS guard to "neither store has it" — was NOT taken: the expense_metadata
INSERT has no ON CONFLICT, so re-attempting an order that already has a receipt
row would duplicate it. Detecting is cheap; a non-idempotent re-write is not.
Unchanged: order_feed 6,265, order_spend AUD 4,895 / $442,651.80. Links 75.
This commit is contained in:
@@ -0,0 +1,127 @@
|
||||
-- 0030 — order ↔ transaction, many-to-many (board 205, orders phase 2)
|
||||
--
|
||||
-- Phase 1 read linkage out of `expense_metadata`, which is shaped as ONE ROW
|
||||
-- PER TRANSACTION: `transaction_id` is UNIQUE and `matched_transaction_id`
|
||||
-- partial-unique. Every multiplicity it expresses today is smuggled through a
|
||||
-- string key — migration 0029 keys split shipments `<entity_key>#f<fact_id>` —
|
||||
-- and a BNPL plan needs four rows for one order with no such trick available.
|
||||
--
|
||||
-- The flagship case: order_ebay_14-11714-95953, a A$1,599 DJI drone paid in
|
||||
-- four A$399.75 Afterpay legs (txns 2318, 2333, 1652, 1664). Phase 1 shows it
|
||||
-- with zero linked transactions, which is correct and useless.
|
||||
--
|
||||
-- KEYED ON entity_key, NOT entities.id. finance-app does not model the spine
|
||||
-- and must not carry an FK across an ownership boundary that a spine
|
||||
-- re-extraction can decompile. The real cascade risk is NOT transaction
|
||||
-- deletion (that is handled below) but spine RE-KEYING: supersede_stale()
|
||||
-- decompiles an entity when a re-extraction yields a different entity_key, and
|
||||
-- a TEXT key with no FK silently orphans. Bridge-sourced links are rebuildable;
|
||||
-- `manual` ones are curation and are not. Hence the orphan view at the end.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS order_transaction_links (
|
||||
id SERIAL PRIMARY KEY,
|
||||
entity_key TEXT NOT NULL,
|
||||
transaction_id INTEGER NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
|
||||
leg_kind TEXT NOT NULL CHECK (leg_kind IN ('charge','shipment','instalment','refund','fee')),
|
||||
-- HUMAN position only: "2 of 4". Never an id.
|
||||
--
|
||||
-- The first draft stored a fact id here. Live shipment keys already carry
|
||||
-- fact ids 22664 and 22711, max(extracted_facts.id) is 23,781 today, and
|
||||
-- SMALLINT tops out at 32,767 — that ceiling arrives on ordinary corpus
|
||||
-- growth and the insert dies with `smallint out of range`. Fact ids go in
|
||||
-- source_fact_id, which is BIGINT.
|
||||
leg_index SMALLINT,
|
||||
leg_count SMALLINT,
|
||||
source_fact_id BIGINT,
|
||||
-- The LEG's own amount, signed: positive is money out, negative is a credit
|
||||
-- coming back. Storing the order total on every leg would make four rows sum
|
||||
-- to four times the purchase.
|
||||
amount NUMERIC(12,2) NOT NULL,
|
||||
currency TEXT NOT NULL DEFAULT 'AUD',
|
||||
source TEXT NOT NULL CHECK (source IN ('order-bridge','instalment-matcher','manual')),
|
||||
confidence TEXT NOT NULL DEFAULT 'exact' CHECK (confidence IN ('exact','derived','manual')),
|
||||
evidence JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Every index needs IF NOT EXISTS, not just the table: a run that creates the
|
||||
-- table and then fails during the backfill would otherwise abort on re-run
|
||||
-- with "relation already exists" and need manual surgery.
|
||||
--
|
||||
-- uq_otl_transaction is load-bearing. Without it two matchers can each claim
|
||||
-- txn 2318 and the order reads as paid twice. It is deliberately stricter than
|
||||
-- reality for one case — a single card charge covering two orders — which then
|
||||
-- surfaces as a NAMED REJECTED INSERT rather than as silent duplication.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_transaction ON order_transaction_links (transaction_id);
|
||||
|
||||
-- WHAT IDENTIFIES A LEG DEPENDS ON ITS KIND, and the first version of this
|
||||
-- index got it wrong in the most on-brand way available. It read
|
||||
-- (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of the
|
||||
-- same order — both with a NULL leg_index — collapsed to the same key and
|
||||
-- ON CONFLICT DO NOTHING dropped one SILENTLY. Caught only because the
|
||||
-- backfill reported 62 rows against 63 candidates.
|
||||
--
|
||||
-- The row it ate was order_amazon_249-4859367-0690246's $130.00 second
|
||||
-- shipment (txn 3171) — the same order named in migration 0029's comment as
|
||||
-- the reason split shipments need distinct keys at all, because Amazon mails
|
||||
-- and CHARGES per shipment. A uniqueness rule that cannot hold two shipments
|
||||
-- is the exact defect 0029 was written to fix.
|
||||
--
|
||||
-- So: an instalment leg is identified by its INDEX (1 of 4), a shipment leg by
|
||||
-- its FACT (each shipment is its own extracted fact), and a whole-order charge
|
||||
-- by neither — one per order, which the zeros express.
|
||||
DROP INDEX IF EXISTS uq_otl_leg;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_otl_leg ON order_transaction_links
|
||||
(entity_key, leg_kind, COALESCE(leg_index, 0), COALESCE(source_fact_id, 0));
|
||||
CREATE INDEX IF NOT EXISTS idx_otl_entity ON order_transaction_links (entity_key);
|
||||
|
||||
-- Backfill the existing bridge rows so the table is not empty on day one and
|
||||
-- the readers can switch over in the same change. `#f<fact_id>` splits into a
|
||||
-- shipment leg carrying its fact id; everything else is a whole-order charge.
|
||||
INSERT INTO order_transaction_links
|
||||
(entity_key, transaction_id, leg_kind, leg_index, source_fact_id, amount, currency, source, confidence, evidence)
|
||||
SELECT split_part(em.source_message_id, '#', 1) AS entity_key,
|
||||
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
|
||||
CASE WHEN em.source_message_id LIKE '%#f%' THEN 'shipment' ELSE 'charge' END,
|
||||
NULL::smallint,
|
||||
NULLIF(substring(em.source_message_id FROM '#f([0-9]+)$'), '')::bigint,
|
||||
COALESCE(em.amount, 0),
|
||||
COALESCE(em.currency, 'AUD'),
|
||||
'order-bridge',
|
||||
'exact',
|
||||
jsonb_build_object('backfilled_from', 'expense_metadata',
|
||||
'source_message_id', em.source_message_id)
|
||||
FROM expense_metadata em
|
||||
WHERE em.source = 'order-bridge'
|
||||
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
|
||||
ON CONFLICT DO NOTHING;
|
||||
|
||||
-- Spine re-keying orphans links silently. Bridge rows can be rebuilt by
|
||||
-- re-running the bridge; a `manual` orphan is lost curation and is the row
|
||||
-- that actually needs a human. Same idea as order_settlement_violations.
|
||||
CREATE OR REPLACE VIEW order_link_orphans AS
|
||||
SELECT l.id, l.entity_key, l.transaction_id, l.leg_kind, l.leg_index,
|
||||
l.amount, l.currency, l.source, l.confidence, l.created_at
|
||||
FROM order_transaction_links l
|
||||
WHERE NOT EXISTS (SELECT 1 FROM entities e WHERE e.entity_key = l.entity_key);
|
||||
|
||||
-- Two stores that can disagree will disagree. The bridge now writes both in
|
||||
-- ONE transaction, so a half-written pair should be impossible — this view
|
||||
-- exists to prove that rather than to assume it, and it must stay empty.
|
||||
--
|
||||
-- The plan's suggested guard (widen the bridge's `NOT EXISTS expense_metadata`
|
||||
-- to "neither store has it") was NOT taken: the expense_metadata INSERT has no
|
||||
-- ON CONFLICT, so re-attempting an order that already has a receipt row would
|
||||
-- duplicate it. Detecting drift is cheap; re-attempting a non-idempotent write
|
||||
-- to fix a state that cannot occur is not.
|
||||
CREATE OR REPLACE VIEW order_link_drift AS
|
||||
SELECT em.source_message_id,
|
||||
COALESCE(em.matched_transaction_id, em.transaction_id) AS transaction_id,
|
||||
em.amount, em.currency, em.reconciled_at
|
||||
FROM expense_metadata em
|
||||
WHERE em.source = 'order-bridge'
|
||||
AND COALESCE(em.matched_transaction_id, em.transaction_id) IS NOT NULL
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM order_transaction_links l
|
||||
WHERE l.transaction_id = COALESCE(em.matched_transaction_id, em.transaction_id)
|
||||
);
|
||||
Reference in New Issue
Block a user