5e355c7e859dc9eda9d703a948656c0da8432dee
203
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 siddharthd
|
5e355c7e85 |
currency: normalise foreign_currency_code to ISO-4217 (board #254)
ci / lint-test (push) Successful in 49s
Exact-code equality is the whole basis of FX reconciliation — order_transaction_bridge matches on foreign_currency_code = %(ccy)s — so a row labelled 'BAHT' could never match a THB order and a row labelled 'null' could never match anything. 226 of 1,632 coded rows (14%) were unmatchable for this reason alone, silently degrading a shipped job. 187 rows carried the literal 4-character STRING 'null' rather than SQL NULL, 42 of those also carrying foreign_currency_amount = -1.00 exactly. Root cause is a truthy guard in the CC processors' Parse node — 't.foreign_currency_code || null' passes the string "null" straight through, because a non-empty string is truthy. Not historical: 155 rows in 2026-03 but also 8 in May, 3 in July and 21 in August. Fixed at source in the same change (smarthome cc-statement-processor-paperless.json). Mappings were verified against IMPLIED RATE, not trusted from the label, because a row whose rate contradicts its currency is a broken ledger row rather than a mislabelled one — the trap order_transaction_bridge's rate guard exists for. All six agreed: BAHT 0.0433 vs THB 0.0470, INDIAN RUPEE 0.0183 vs INR 0.0180, U.A.E. DIRHAM 0.4287 vs AED 0.4184, US DOLLAR 1.4054 vs USD 1.5513, NZ DOLLAR 0.8243 vs NZD 0.9311, EURO 1.6342 vs EUR 1.6964. foreign_currency_amount is untouched by the relabels — only the code was ever wrong, so this is a relabel, never a recompute. Applied row counts matched the pre-flight exactly: 42, 187, 13, 10, 6, 4, 4, 2. Postcondition asserted inside the transaction: 0 non-ISO codes remain. Verified after: 0 'null' strings, 0 -1.00 sentinels, 1445 coded rows (1632 - 187). |
||
|
siddharthd
|
669f9119b9 |
enrichment readers: resolve one metadata row, deterministically (board #253)
ci / lint-test (push) Successful in 53s
Four readers of expense_metadata that only the 1:1 UNIQUE constraints keep
correct, found while designing wallet-first ingestion. Each fails silently
rather than erroring, so none of them would announce itself.
merchantVerdict (order-reviews.ts) was a genuine fan-out: a plain JOIN on
transaction_id OR matched_transaction_id, no per-transaction cap, feeding
`for (const r of rows) counts[r.rating] += 1`. One review joined to N
metadata rows is counted N times, inflating the loved/never tallies that
drive the pre-order warning — and its LIMIT 50 applies to the fanned-out
set, so duplicates would also push older reviews out of the window. Now a
LATERAL, so the merchant matched is the merchant the UI shows.
nudgeState (slack/interactive) had no LIMIT and no ORDER BY, feeding
queryRow, which returns rows[0]. Postgres guarantees no ordering, so the
merchant and currency on every Slack card refresh could flip at random.
It was the only "resolve both links" site missing the convention its
siblings all carry.
getSharedTransactions had LATERAL LIMIT 1 with no ORDER BY — bounded, but
arbitrary, and free to disagree with the main transaction list about the
same transaction.
transaction_search (migration 0035): NOT the fan-out I first claimed. It
joined only on em.transaction_id, which is UNIQUE, so it could not
multiply rows and sums over it were never inflated. The real defect is
narrower and worse: it OMITTED matched_transaction_id, so every
card-settled order was invisible to it — 148 transactions (Glovo Milan,
Uber Eats, all source='order-bridge') reporting NULL payment_method /
order_reference / line_items while the data sat in the table. This view is
the agent read surface (Hermes' MCP selects those columns from it) and
home-dashboard reads it via dash_ro, so the omission was estate-wide.
Broadening the join is what introduces fan-out risk — transaction_id and
matched_transaction_id are two SEPARATE unique indexes, so two different
rows can legally point at one transaction — hence LATERAL rather than a
second OR. Verified before applying: 4760 rows unchanged, debit sum
unchanged at 1514412.89, only the 148 rows differ. Grants re-granted and
re-verified, since CREATE OR REPLACE VIEW drops them.
Also fixes an integration assertion
|
||
|
siddharthd
|
258fe3b617 |
intake: the needs-header card names what is missing and gives the right advice for it
ci / lint-test (push) Successful in 49s
A Woolworths eReceipt SCREENSHOT has no date anywhere on it — store, total, items, payment, savings, not one date (2026-08-22, batch 60ed68be…) — while the app's own Share button sends the raw receipt with the date line. The card said 'share the top of the screen too', which is delivery-app advice and sends the person back for a shot that cannot exist. Now: the missing field by name, and for a dateless in-store receipt, the Share button or pantry's own upload. |
||
|
siddharthd
|
0823244e4a |
intake: the word "Image" is not a receipt, and the prompt must not be data
ci / lint-test (push) Successful in 52s
Batch 3d2ffbe8 arrived as one text part whose entire content was the five characters `Image` — an iOS Shortcut coerced the shared photo to text. The model was asked anyway and answered out of its prompt: Burgertory (Braybrook), two "The Pride" lines at $22.80, a $41.15 total. Nothing on it existed. Burgertory was in ORDER_ITEM_PROMPT as the example of how to print a branch in parentheses. Only the missing date kept it out of the matcher; the invented header and items were stored on the batch regardless. Three layers, none of them a model: 1. textCouldBeReceipt() — deterministic, before any model call. A TEXT part with fewer than two words, no digit, or under 40 characters and no money figure cannot be a receipt. Length is NOT the test on its own: `TOTAL $17.00` is twelve characters and is exactly what this door accepts. A rejected part is dropped like an unreadable file type, so one junk part beside a good screenshot still goes through; if nothing usable is left the card quotes what the phone actually sent. PDFs and images untouched. 2. The prompt names no real shop — merchant examples are PROMPT_PLACEHOLDER_MERCHANTS, exported from intake-rules so the prompt and the guard cannot drift — and it is told what to return when the input is not a receipt at all. 3. confabulationReason() — the belt. Reachable where the pre-check cannot look: a PDF that renders to a scrap of junk still reaches the model. Also rejects any answer whose merchant IS a placeholder. All three file `unsupported`, never `needs_header`. A confabulated header is not a partial one, and the needs_header path stores what it is given. Same shape as the ingestion engine's 2026-08-05 empty-body defect: a stage that can be handed nothing has to say so, because the model downstream will always answer. Tests: 16 unit cases over the two pure functions and the prompt; three integration cases — `Image` is declined with zero model calls and nothing but `unsupported` on the batch row, a junk part beside a real receipt still enriches, and a junk PDF is unsupported rather than needs_header. The first and third fail on the tree before this commit. |
||
|
siddharthd
|
d88ca2a24c |
intake: a duplicate card that refuses a purchase must say what to do instead
ci / lint-test (push) Successful in 1m5s
The header (merchant, date, total) is a purchase's identity here, so a genuine second identical purchase — two of the same coffee, one shop, one day — reads as a re-share and is refused. That default is right; a re-share is far commoner than a repeat. But the card said only "Nothing was changed", which is a dead end: the money is real and the ledger silently never gets it. Say to add it in the app. |
||
|
siddharthd
|
91586ce4a4 |
test(intake): the second review pass, and CLAUDE.md for the rules it set
ci / lint-test (push) Successful in 45s
Six of the nine fixes have a test that fails against the previous tree: the metadata row's currency, the Create-path hold, two concurrent batches booking once, mrkdwn escaping including plain_text, the escaped currency, and candidate-row currency. Three do not, and the reason is the same each time — the window needs a crash or a lost race that a test cannot force. Asserted instead: a stale metadataId falls back to the live row rather than reporting a phantom write; two writes racing to create a metadata row both resolve to one; the create path records a baseline at all. All three passed before as well. |
||
|
siddharthd
|
4b09d4136f |
intake: lock the purchase, not the batch, and hold on both write paths
Two different unresolved batches for one purchase could each create a transaction. ON CONFLICT (source, source_ref) cannot stop it — their source_refs differ — and both taps clear their match and duplicate checks before either inserts. The create now takes a transaction-scoped advisory lock keyed on (platform, merchant, date, total), re-checks inside it, and inserts; the loser reports `duplicate` against the winner's transaction, the same answer the sequential path gives. The lock has to be inside a real transaction, which this app did not have. queryRaw is $queryRawUnsafe on a POOLED client, so every call is its own implicit transaction and pg_advisory_xact_lock taken in a statement of its own is released before the next statement runs — the lock would have looked right and held nothing. withTransaction() in db.ts is the scope; the integration helper mirrors it with BEGIN/COMMIT on one pooled connection. Also: the "hold rather than replace an itemised receipt with an unpriced upload" guard lived only in routeFood, and createFromBatch's re-match walked straight past it — the same upload destroyed the same receipt depending on which button reached it. One function, both callers. |
||
|
siddharthd
|
4f0f0dd285 |
intake: escape the currency and the button label too
money() interpolated the extracted currency raw, and parseExtraction accepts any string for it — so "<@U123>" would have been a live mention on every card the batch produced. And plain_text disables *formatting*, not entity decoding: &, < and > are control characters everywhere in a Slack payload, so the Create button's label is escaped like any other value we did not write. My earlier note claiming the opposite was wrong. Candidate rows also carry their transaction's currency (NATIVE_CURRENCY) rather than always printing "$" under a header reading "USD 22.40". |
||
|
siddharthd
|
b788b4e8fa |
line-items: snapshot before writing, and stop reporting writes that did not happen
Three faults on the same function, all of them "the write is assumed to have worked". - The create path snapshotted the undo baseline AFTER its INSERT. A crash in between left the retry recording the intake's own write as "previous", so undo restored the thing it was meant to reverse — the exact failure previous_line_items exists to prevent. Both paths now snapshot first. - An UPDATE against a row deleted between the select and the write affects nothing and still returned a metadata_id, so the batch was marked enriched against a row nobody had written to. The write is now checked and falls through to the ordered lookup, then to a create. - Two calls for one transaction could both find no row and both INSERT; expense_metadata.transaction_id is UNIQUE, so one lost with a constraint error that reached the user as an error card. ON CONFLICT (transaction_id) DO UPDATE, with (xmax = 0) so created_metadata stays truthful. And the new row's currency comes from the batch instead of a hard-coded 'AUD'. OrderDetails prefers expense_metadata.currency, so a USD order — booked correctly on the transaction with foreign_currency_code — printed "$". |
||
|
siddharthd
|
b5569b8ccc |
schema: previous_line_items on the model; say what the finish_reason rule is
Migration 0034 has the column and the Prisma model did not, so a regenerated client and the database disagreed. The extraction comment claimed only a finished reply is accepted while the code accepts an absent finish_reason. The code is right — failing every call on a provider that omits an optional field would be a certain outage in exchange for a case that also requires syntactically complete JSON, and the harm a truncated reply does is closed on the write side. The prose now states the rule as it is: a reason that is PRESENT must be `stop`. |
||
|
siddharthd
|
d4ff68363a |
expense_metadata: every reader resolves the row a write lands on
getTransactions' provenance LATERAL, merchantForTransaction and the Slack details modal all select on transaction_id = $1 OR matched_transaction_id = $1 with a bare LIMIT 1, so they could read a different row than replaceLineItems writes. Same ORDER BY as the match, the write and the order-detail GET. Defensive, not a live bug: measured 2026-08-21, expense_metadata holds 667 rows, 147 of them pointing-only, and ZERO transactions resolve to more than one row by either link. This changes nothing today. It is here so the readers still agree on the day multiplicity appears, which is the only condition under which any of them was ever wrong. |
||
|
siddharthd
|
631e75e58f |
test(intake): cover the review findings, and CLAUDE.md for the rules they set
ci / lint-test (push) Successful in 59s
Unit: word-boundary merchant matching (Bigals Pizza is not an IGA), mrkdwn escaping vs the plain_text button label, a foreign total on a card, and the finish_reason gate. Integration: the NULL-platform wildcard on both the food and grocery paths, candidates crossing platforms, EXCLUDE_RECONCILED_SOURCE, the pinned metadata row, the panel reading the row the write landed on, the once-per-batch undo baseline, the priceless-upload hold, Create re-matching, two pending cards booking once, a pantry-resolved sibling, and a USD receipt. Eleven of the thirteen fail against the pre-fix tree; the exceptions are the no-batch-id baseline guard (a no-change assertion) and the concurrent-tap one, which is a race and passed either way. |
||
|
siddharthd
|
c25db5211b |
intake: a NULL platform is "no platform", and Create re-checks before it inserts
Six fixes in the router, all of them cases where a stale or over-wide answer was written rather than re-asked. - ($3 IS NULL OR em.platform = $3) made an in-store receipt a wildcard: a till receipt from the Coles tap matched the Uber Eats Coles order for the same day and money, and overwrote its items. In-store now matches only rows with no platform of their own. - findCandidates loses its platform filter entirely. That card is a human choice, shown precisely because the exact key missed — the platform is one of the things that may be wrong, so the list must widen there, not narrow. - createFromBatch re-runs the match. The card is minutes old by the time it is tapped and the email/order bridge books orders on its own schedule; the match it was drawn from is not current. - ...and checks for a resolved sibling batch. findDuplicateBatch ignores unresolved batches by design, so two pending cards for one order each booked it. - INSERT ... ON CONFLICT (source, source_ref) DO NOTHING with a re-select, rather than SELECT-then-INSERT, which loses a double tap. - A non-AUD receipt is booked as order-ingestion.ts books one: amount native, amount_aud NULL, foreign_currency_* set. No FX rate exists at intake. Also: EXCLUDE_RECONCILED_SOURCE in both queries instead of a bare superseded_by_id check — a reconciled manual twin at the exact key read as an ambiguity, which is the same spend counted twice. And routeFood refuses to replace an itemised receipt with an extraction carrying no prices at all: the reconciliation check passes those (there is nothing to check), so without this a priceless upload silently destroys a good record. It holds instead. RouteContext.subtotal was written and never read; it is replaced by currency, which the cards need. |
||
|
siddharthd
|
9a442b7355 |
intake: escape mrkdwn, and stop printing every total as dollars
Merchant names, transaction descriptions and model output are none of ours: Slack reads &, < and > as markup, so "Ben & Jerry's" and a description carrying <http://…|text> rendered as something the card never meant to show. esc() on every interpolated value in mrkdwn — never on our own literal *bold*, and never on a button's plain_text label, where entities would print verbatim. money() also hard-coded "$" while the order nudge already prints "USD 15.00". The cards now carry the batch's currency. |
||
|
siddharthd
|
f9638cc5e5 |
line-items: one metadata row, agreed on by the write and the read
Three defects on one surface, all invisible while a transaction has exactly one expense_metadata row — which is almost always. 1. ORDER BY (transaction_id = $1) DESC did the opposite of its comment. Postgres orders DESC as NULLS FIRST, and the row that merely POINTS at the transaction has a NULL transaction_id, so the clause written to prefer the owned row preferred the other one — in exactly the case it exists for. matchTransaction already spelled out NULLS LAST; now so do this and GET /api/transactions/[id]/order, which had no ORDER BY at all and could read a different row from the one an enrichment had just written. 2. replaceLineItems takes an optional metadataId. The match resolves a row through a platform-filtered join; re-picking independently here can land the write on a row the match ruled out. 3. previous_line_items on intake_batches is the undo baseline, recorded once per batch under COALESCE and read back. Returning whatever was just read hands a pantry retry its own write as the "previous" state, so undo restores the thing it undid. Migration 0034 is unapplied anywhere, so the column is added there rather than in a 0035. |
||
|
siddharthd
|
7424d13858 |
intake: only a reply that finished is a reply the schema constrained
liteLLMExtractor rejected finish_reason "length" and accepted everything else, so a content_filter or tool_calls stop was parsed as if the strict json_schema had held. Reject any reason present and not "stop", naming it. |
||
|
siddharthd
|
be3d10d4f1 |
intake: match merchants on whole words, not substrings
merchantKey("Bigals Pizza") contains "iga", so GROCERY_PATTERN filed a pizza
shop as a grocer and merchantMatches matched it against every IGA on the same
date and total. Short names in a substring list collide with ordinary words,
and the collision reads as a correct classification rather than a bug.
GROCERY_PATTERN is now a word-boundary alternation; merchantMatches asks
whether one name's words appear as a contiguous run inside the other's, which
keeps Coles <-> Coles (Wyndham Vale) and Order - Coles (Wyndham Vale).
Also collapses the third PLATFORM_LABEL copy: order-ingestion's was exported
and read nowhere, order-details.tsx had its own, and the intake one — the only
live copy — was missing `uber`. One map, in intake-rules.
|
||
|
siddharthd
|
a4e5f540cc |
docs: intake router routes, env vars, migration 0034
ci / lint-test (push) Successful in 48s
|
||
|
siddharthd
|
32a47bf8ee |
test(intake): rules, PDF text, C2 idempotency, enrich-vs-candidates, pantry hand-off, Slack taps
Unit: merchant pattern, header validation, reconciliation tolerance, verbatim line items, one-call message shape, card action ids; unpdf over the two real Uber Eats exports (Coles $85.74 / 15 lines, Burgertory $22.40 / Michael + Joel). Integration: fetch stubbed at the LiteLLM and pantry boundaries so the PDF path, persistence, match and writes run for real against personal_test. intake-live.test.ts is opt-in (INTAKE_LIVE_TEST=1) and was run green against cloud-budget on both PDFs. |
||
|
siddharthd
|
c7ea516c34 |
slack: dispatch the four intake taps on action_id (plan §11 C5)
intake_use_candidate / intake_create / intake_grocery / intake_meal carry JSON
{batch_id, ...} rather than <id>:<verb>, so they branch before the order-nudge
parse. They act on the persisted batch: enrich the chosen transaction, create
ING-10-style (statement_id NULL, owner_id from shared_by, source/source_ref
make a second press a no-op), or remember the merchant's kind and route. A tap
on an already-resolved batch returns its current card and writes nothing. Card
updates go to response_url, as the existing handlers do.
|
||
|
siddharthd
|
db5c9a02a7 |
intake: POST /api/intake router + PUT /api/transactions/[id]/line-items (plan §11 C1, C2)
One upload point, every applicable step. Persist the batch, classify each
file, ONE LiteLLM call (cloud-budget, temperature 0, strict json_schema; PDFs
as unpdf text, never pixels) with an order-item prompt that keeps names
verbatim, header (merchant, date, total), merchant pattern
(coles|woolworths|woolies|aldi|iga|foodworks|costco) -> grocery | food | ask,
exact (platform, merchant, date, total) match against transactions +
expense_metadata, then act:
- food + unique match -> replace line_items via the C2 path, 'enriched'
- food + miss/ambiguity -> 'candidates' (same merchant ±3d) with
intake_use_candidate / intake_create buttons; never auto-creates
- grocery -> original files to pantry POST /api/capture with
finance_target_transaction_id + finance_action enrich|create (C3);
409 -> 'duplicate'
- ask / needs_header / held / duplicate / unsupported as replies, not writes
Reconciliation is against the printed item subtotal (a delivery total is
items + fees - promotions: 18.94 of Coles lines settled at 5.74), or
2%, else 'held'. Originals are kept under INTAKE_FILE_DIR by sha256 so a later
'Groceries' tap can still forward them.
C2 is an idempotent replace on transaction_id = :id OR
matched_transaction_id = :id, accepts INTAKE_TOKEN or RECEIPT_INGEST_TOKEN,
creates the expense_metadata row when absent, 404 when the transaction does
not exist, returns {previous_line_items, line_items} so pantry's undo is the
same call with the previous list.
|
||
|
siddharthd
|
a538376638 |
db(0034): intake_batches + intake_merchant_kinds for the intake router
One row per upload through POST /api/intake, written BEFORE extraction so a Slack tap minutes later has something to act on and a re-share (same sha256 set, or the same (platform, merchant, date, total) header already resolved) is detected. Columns are the §11 C1 contract in smarthome plans/intake-router-slice-plan.md. intake_merchant_kinds holds the remembered 'Groceries or a meal?' answer per normalised merchant (plan §2, Codex #6). Not applied to production — finance has no migrate service; apply by hand at deploy, then npm run test:setup. |
||
|
siddharthd
|
b958d05287 |
test(db): resolve the integration DSN at run time instead of storing an IP (#161)
ci / lint-test (push) Successful in 1m5s
setup-test-db.sh wrote .env.test once with the Postgres container's bridge IP baked in, and Docker reassigns those on recreation. The file rotted silently: its stored 172.22.0.18 belongs to **gitea** as of today, and an earlier drift had the suite pointing its TRUNCATE at **postgres-pantry** — which it would have wiped had the credentials happened to match. Root-cause fix: stop persisting a host at all. - scripts/resolve-test-db.sh re-derives the DSN from the live container and rewrites .env.test. Resolution order is TEST_DATABASE_URL, then the container's published host port, then the bridge IP. `docker port postgres-personal` reports 127.0.0.1:5432 since the 2026-08-17 PG17 migration, so loopback is what gets used here — and a loopback address cannot be reassigned to another container, which is the whole point. (Note this supersedes the smarthome CLAUDE.md's "postgres-personal publishes no host port"; that is now stale.) The bridge-IP fallback is kept for a host where the port is not published, and the script never prints the password. - Wired as `pretest:integration`, so npm runs it before every integration run. .env.test becomes a generated artifact rather than stored configuration. - setup-test-db.sh now only creates the database and copies the schema, then delegates to the resolver. - helpers.ts::assertTestDatabase stays and its message now points at the resolver rather than at `docker inspect`. It still earns its place: running vitest directly bypasses the pretest hook, and a stale .env.test would then still be read. Docs: CLAUDE.md gains a Testing section carrying this, the prisma-generate trap for a fresh checkout or worktree, and the frozen-clock rule for tests whose subject is defined relative to "now". AGENTS.md updated for the same, and for lint now being blocking. |
||
|
siddharthd
|
eaf5d5ac66 |
test(analytics): guard the three unguarded Priority 0 behaviours (#155)
docs/ui-information-architecture-review.md claimed Priority 0 was guarded by analytics-sql.test.ts. That file only ever covered the three SQL *fragments* (EXCLUDE_RECONCILED_SOURCE, NATIVE_CURRENCY, INVESTMENT_SIGNED). Items 2, 3 and 6 of the Priority 0 list had nothing, and all three fail silently — no error, no empty screen, just a number that quietly disagrees with the one beside it. New: src/__tests__/integration/analytics-priority0.test.ts, 18 tests. - **Item 2, pace/headline parity.** /api/analytics/daily summed over a month must equal /api/analytics/monthly's headline to the cent, per category, and across every month in the window. Cents rather than exact equality is the assertion because the two routes group at different grains (month+category vs month+day+category) and both carry 4dp — the displayed grain is what has to agree, which is precisely the bug (the chart ending the month a few cents off the headline above it). Fixtures are only shapes where a naive gross debit sum disagrees: a 50% split, a refund, fee and interest rows, an itemised loan repayment, a reconciled source row, family-tagged spend, transfers/investment/income, and a row someone else paid that I hold a split on. A fourth test asserts the naive sum does NOT match — without it, parity between two queries identical by construction would pass on fixtures too plain to discriminate. - **Item 3, fees date range.** Default 12 months with the window reported (from=2025-09-01, not 08-31 — toDateStr, not toISOString); the older tail excluded; months=0 all-time with a null range rather than a period claim the number does not make; a narrower months=3; the transaction-level fee lines bounded by the same window as the statement totals, since a bounded total beside an unbounded list is the same defect wearing the fix; and a nonsense months param clamped rather than producing an Invalid Date window. - **Item 6, partial-month comparisons.** The in-progress month never enters a baseline; a partial month is compared through the same day of each prior month; a complete month keeps full totals; top movers truncate both sides; year boundary and leap February. Item 6 was untestable by construction, which is why it had no test: the rules were closures in budget/page.tsx reading `new Date()` inline. Extracted to src/lib/partial-month.ts with an injectable clock, and budget/page.tsx now imports them. The extraction also de-duplicates the baseline filter, which was written out identically in two places. The file freezes the clock (vi.useFakeTimers, toFake: ["Date"] only — pg needs real timers for connection handling). Without it the fixtures drift out of the routes' rolling windows over time and the assertions go vacuous, which is the same class of silent failure the file exists to catch. Mutation-checked, all three: breaking daily's spend semantics fails 3 of the item-2 tests (the placebo correctly still passes); forcing the fees window to all-time fails 5; letting the current month into the baseline fails 1; and dropping the elapsed-day truncation fails 3. |
||
|
siddharthd
|
fc9ffaba25 |
chore(ci): clear the lint debt and make lint blocking (#79)
CI ran lint with continue-on-error because 23 errors predated it. All are fixed, so the flag is gone and a typecheck step is added beside it — nothing else in CI compiles the test files, so a type error in a fixture could only ever be found on a developer's machine. Real fixes, no blanket disables and no suppressions: - react-hooks/set-state-in-effect (6): each was "reset or seed state when an input changes", so each becomes a state adjustment during render behind a previous-value guard — the documented React pattern for it. Two were also latent bugs. transactions/page.tsx depended on `others`, rebuilt by .filter() every render, so the effect already ran every render, one commit late. shared/page.tsx depended on the `tagIds` array identity, so the row window was reset more often than "changing any filter" claims — including after "Show more", the one interaction it must not undo; the key now joins by value. csv-import-modal reads localStorage once at mount, which is a lazy useState initializer, not an effect (SSR-guarded, though the modal is mounted from client state and never prerenders). - react-hooks/static-components (3): SortIcon was declared inside SharedPage, making it a new component type on every render. Hoisted to module scope with the sort state passed as props. - @typescript-eslint/no-explicit-any (9): the dynamically imported route handlers in two integration tests. Typed as their real signatures with one `as unknown as NextRequest` cast at the fixture boundary, so the call sites and response bodies get checked instead of opted out. - react/no-unescaped-entities (5): exact-equivalent entities (', "), so no rendered text changes. Also fixes the one pre-existing tsc error (unit/order-ingestion.test.ts:511): `options` is optional, and `?? 0` keeps the assertion failing on an absent list rather than silencing it with a non-null bang. 9 warnings remain (no-unused-vars, exhaustive-deps). eslint exits 0 on warnings, so lint blocks on errors only — deliberately, since none of the remaining warnings should stop a push. |
||
|
siddharthd
|
9f1be74330 |
transactions: let a gift card be a payment method you can pick
ci / lint-test (push) Successful in 43s
`credits` has been a valid payment_method since migration 0018, which added it so the order pipeline could file gift-card-funded orders. It reached the database, `needsCardMatch()` (which holds those rows out of reconciliation) and `bankLabel()` (which renders them "Gift Card"). It never reached the two places a human touches it. So entering a gift-card purchase by hand was impossible: the dropdown stopped at "Other", the row took the NULL default, and NULL is *reconcilable* — it joined the pending queue to be offered card matches within 3 days and 1% that cannot exist, because the balance already paid the bill. Accepting one is the destructive case the cash exclusion documents: reconciled manual rows are filtered out of every query, so the spend vanishes while an unrelated card row claims to be it. - `src/lib/payment-methods.ts` — one canonical list, mirroring the `statement-types.ts` pattern, plus the TS mirror of `needsCardMatch()`. - `PATCH /api/transactions/[id]` — the `VALID_METHODS` literal omitted `credits`, so the API answered 400 for a value the CHECK constraint admits and 372 live rows already carry. It validates against the list now. - `POST /api/transactions` — validated at all, so a bad value is a 400 naming the field rather than a 500 carrying a Postgres error. - Add modal — offers every method, and explains the consequence when you pick one that suppresses reconciliation. Relabelled "Paid by" → "Payment method", which the edit modal uses for the *owner*; they are different questions. - Edit modal — had no payment-method field at all, so an existing row wrongly in the queue could not be corrected. Manual rows only: the queue is `statement_id IS NULL`, so it decides nothing on a statement row. The test asserts the list against `pg_get_constraintdef` and evaluates `needsCardMatch()` as real SQL rather than re-implementing it, so the next migration that widens the constraint fails the suite instead of drifting for two months. Mutation-tested by dropping `credits`. Unit 130 pass, integration 206 pass (the integration DB needed `npm run test:setup` first — `.env.test` still named a stale container IP). |
||
|
siddharthd
|
4f307e9517 |
docs: what importing a BNPL account changes, and the Zip data decisions
ci / lint-test (push) Successful in 41s
24 Zip statements landed (Dec 2020 - Jun 2026), all balance-asserting at $0.00. Zip is a credit account, so three existing row types had to be re-read. Data changes applied via transaction_overrides; this records why. A ZIPPAY debit leaving the bank duplicates the Zip statement's purchases only where a Zip statement covers that period. 2 of 8 do, and are now transfers ($125.97). The other six stay spend: for those periods the bank debit is the only record the purchase happened, so moving them would have deleted $703.35 of real spend. 'Account Credit' is cashback, confirmed by the owner and consistent with Zip labelling its other money-in types explicitly (Payment = repayment, 'Refund <merchant>' = merchant refund). 34 rows, $740, round amounts, no matching bank debit. Gemini split the identical description 23 'other' (subtracts from spend) / 11 'transfers' (excluded), so one thing did two opposite things. Now all 34 shopping+credit, netting against spend the way refunds already do. Income was rejected for the reason the Up item sales note gives. Prezzee gift cards ($1,580) stay spend — stored value only nets out if every downstream purchase is logged, and for 2020 it is not. Same reasoning as the ATM-withdrawal rule. 31 uncategorised debits mapped by merchant with merchant_normalized set too, since the raw descriptions carry card numbers and order UUIDs. All 137 Zip rows are now categorised; $11,135.11 net spend, $10,085.11 transfers. Flags statement 200 (Sept 2021): extracted without billing_end_date or balances, so it is the one statement uq_statement_identity cannot cover (the index is partial on billing_end_date IS NOT NULL) and a re-import would duplicate it. |
||
|
siddharthd
|
2cfaf8e8c8 |
statements: derive account identity, fix the new-bank alert (0033)
ci / lint-test (push) Successful in 42s
account_number fragments exactly like bank_name did, from the same cause: ANZ's Access Advantage arrived as both '4085-56264' (4 statements) and '408556264' (1). uq_statement_identity keyed on it, so re-importing one period under the other spelling evaded the duplicate check. account_number_key is GENERATED ALWAYS — the number with separators stripped — and the unique index moves onto it. Derived rather than rewritten because the raw number is the readable one and some of it is structure: Up stores '633-123 / 176540052', a BSB and an account number, and flattening it would lose a distinction a human reads at a glance to fix a machine problem. The Amex mask and case are preserved; 'X' records which digits were redacted. Not the cause of the documented 31-row / $42,040.68 ANZ duplication — 107/142/143 overlap on different end dates, which that index cannot catch at any spelling. Adds statement_identity_drift: one account under two bank names, or one bank under two spellings of one number. Non-empty means fragmentation the normaliser could not know about. Currently one row, the ANZ pair above, now unified by key. Also documents the N8N regression this series caused. Check Known Bank compared the raw Gemini name against the canonical column before the insert, so nothing ever matched and every Zip statement was tagged pending and held for Slack approval — a stall, not just a noisy alert. Fixed in workflow FysADdFwEtwONQl4 by comparing like with like through normalize_bank_name(), passing currency so a genuinely new national entity still alerts. Verified live: 12s after vs 30-53s before. |
||
|
siddharthd
|
c3b623052d |
statements: keep one bank's national entities apart (migration 0032)
ci / lint-test (push) Successful in 44s
Fixes 0031. Every merge it made was provably one account — each split pair shared an account_number — but the PREFIX generalises past that evidence. Citibank Australia and Citibank India are different banks with different accounts and currency, and `citi%` flattens them into one. Older Citi India statements are queued to load, so this would have merged them on arrival. Not Citi-specific: `hsbc%` and `%american express%` carry the identical defect. Currency alone cannot be the discriminator: Wise holds AUD, EUR and USD accounts under one provider, so a blanket "non-AUD is a different bank" rule would shatter it into three. Wise stays unmapped for exactly that reason, now a documented property rather than an accident. The country comes from the name when the name states it and from the currency otherwise, with AUD as home taking no suffix — so nothing renames: the backfill touched 0 rows. 'Citibank' + AUD -> 'Citibank' (unchanged) 'Citibank India' + INR -> 'Citibank India' 'Citibank N.A.' + INR -> 'Citibank India' (currency carries it) 'CITIBANK INDIA' + AUD -> 'Citibank India' (name beats a wrong currency) 'N.A.' is deliberately not read as a country marker: it means "National Association", a US legal form printed on Citibank letterhead worldwide, including India. Reading it as US is precisely the mistake this fixes. The trigger now fires on UPDATE OF bank_name, currency, since the country is partly derived from the currency. |
||
|
siddharthd
|
0bb4329249 |
statements: one institution, one bank name (migration 0031)
ci / lint-test (push) Successful in 46s
bank_name is a component of uq_statement_identity (bank_name, account_number, billing_end_date), so two spellings of one institution mean the same statement can be ingested twice without the index ever firing. Gemini reads the name off whatever the PDF prints, so the spelling varies per document, not per account. Zip exposed it: account 2705256 arrived under four names in sixteen minutes — ZipMoney Payments Pty Ltd, ZipMoney Payments Pty Limited, Zip Pay, ZipPay — with ten more in the ingestion queue. Not unique: 17 bank_name values represented 13 institutions, every split pair sharing an account number (Amex 14/1, Citibank/Citi 8/4, NAB 3/1). Fragmentation also splits both bank filters and by_bank in /api/analytics/fees. normalize_bank_name() + a BEFORE INSERT OR UPDATE trigger, mirroring the 0013 statement_type pattern with one deliberate difference: the vocabulary is OPEN. An unrecognised name passes through tidied rather than collapsing to a fallback, and there is no CHECK constraint — a bank never used before must be able to arrive without a migration, and destroying its name on first contact is worse than leaving it unmapped. Matching is by prefix, not by enumerated spelling, so unseen variants normalise with no code change. Branches for Westpac, ANZ, HSBC, ING, AMP, Up and CommBank are no-ops that map to the spelling already in use, to catch the legal-entity variant a future PDF might print. Wise is left alone: one spelling is a rename, not a merge. Backfill verified collision-free against uq_statement_identity first. 17 names -> 12. account_owner_mappings keys on (bank_name, account_number) with its own UNIQUE constraint and is updated too; it is empty today. |
||
|
siddharthd
|
95d8544752 |
transactions: change who paid, on a row and on a statement
ci / lint-test (push) Successful in 46s
Owner was write-once for every ingestion path — a pantry receipt hardcodes DEFAULT_OWNER_ID and there was not one `UPDATE ... SET owner_id` in src/ — so a shop the other person paid for was permanently filed as yours. PATCH /api/transactions/[id] now takes owner_id, for manual rows only. A statement row returns 400 statement_owned and points at the statements page: its effective owner is COALESCE(t.owner_id, s.owner_id), so writing it there would either no-op or detach one row from the account it came from. PATCH /api/statements/[id] is new. The statements page has had an owner dropdown since it was built, wired to a route with no PATCH handler — every change 405'd, and because useUpdateStatement never checked res.ok it failed silently and the select snapped back on refetch. It writes both tables: 2,194 statement rows carry their own owner_id against 1,803 that inherit, so updating `statements` alone moves less than half and splits one account's history between two people. The guard is the point. Access is "owner OR holds a split", so handing a row over while holding no split removes it from your list and 404s every route that could put it back — only the new owner can undo it. That is 409 would_lose_access, and the modal offers both ways forward: add my split first, or give it away anyway. Taking a row onto your own ledger is never blocked, and claiming a row you cannot see is a 404 before any owner logic runs. Splits are deliberately not rewritten. They record shares, not direction, so a 50/50 flips from "they owe me" to "I owe them" untouched, settled included. Also adds the missing res.ok check to useUpdateTransaction, without which every rejection resolved as success: the modal closed, the list refetched, and the edit silently vanished. 14 new integration tests; 203 integration + 130 unit green. |
||
|
siddharthd
|
6c14b493ef |
analytics: your share of what someone else paid is your spend
ci / lint-test (push) Successful in 1m25s
Every spend analytic gated on `OWNER_SCOPE = $1` and scaled by `mySplitOf` *within* that gate, so ownership was a precondition for an expense being yours. Half a grocery shop Sonu paid for counted as zero — in monthly, daily, merchants, subscriptions, fees and the budget page. 167 rows / $3,210.91 across Jan-Jul 2026, worst in April (+$1,354.83, the Europe trips), while getParticipantBalances booked the matching debt correctly. The app could say you owed her for a shop while insisting you had not spent anything on it. New MY_SPEND_SCOPE(): owner = me OR I hold a split. OWNER_SCOPE stays on the things that measure an *account* rather than a person — the income and investment lines, and the statement-level fee rollup. myShare had to change with it, and widening the gate alone would have been worse than the bug: its `100 - everyone else` fallback is the payer's remainder, so on someone else's unsplit row it returns 100 and moves their whole bill onto you. It now branches on ownership — my row resolves as before; their row takes an explicit split row only, absent meaning 0. That 0 is what makes the wider gate safe. my_share_percent is deliberately not read on someone else's row: one unscoped column, writable by anyone who can see the row, so "my" can only mean the owner's. All 402 rows carrying one today are owner-side. MY_SHARE_PCT mirrors myShare for the transactions list, which has no viewer-scoped ts join; a test asserts the two agree across seven fixture shapes. No historical restatement — every non-owner split is 2026-dated, and the 1,266 pre-2026 SplitMyExpenses splits are all on rows you own. Also fixes a latent failure in the NATIVE_CURRENCY test, which inserted a statement relying on participant id 1 existing (owner_id is NOT NULL DEFAULT 1 with an FK) and only passed when a sibling file had left one behind. It now owns its fixture. 15 new integration tests; 189 integration + 130 unit green. |
||
|
siddharthd
|
22c2349a47 |
docs: one CSV import path, and the statement-coverage rule that replaced the Frollo importer
ci / lint-test (push) Successful in 43s
|
||
|
siddharthd
|
3edcc27781 |
csv import: map a currency column, or foreign rows land as AUD
ci / lint-test (push) Successful in 41s
Deleting the Frollo importer dropped its currency handling and nothing replaced it: ColumnMapping had no currency column, so applyMapping could never produce foreign_currency_code and batchInsertCSVTransactions' support for it was unreachable from the UI. The USD 10,782 salary imported as A$10,782 — about a third under, sitting in a column of AUD figures looking entirely normal, which is the same defect the owner spotted in the first place. An optional Currency column now sets foreign_currency_code and foreign_currency_amount when the cell is a three-letter code other than AUD, and leaves amount_aud NULL rather than inventing a rate. That is the shape order ingestion already uses and what AMOUNT_UNCONVERTED looks for, so the row renders as its native figure with "no AUD rate" and the statement supplies the real number when it arrives. Caught by reading the imported row rather than the import summary: the summary said 171 inserted and was right about everything it reported. |
||
|
siddharthd
|
461c021e7a |
csv import: exclude what the statements already cover, and delete the Frollo importer
ci / lint-test (push) Successful in 43s
"This is becoming too complex... Frollo should be done through that [the manual
CSV import]" (owner). It was right: a bespoke importer, an API route, a CLI, two
scheduled n8n workflows and a shared secret existed to do what the CSV import
modal already did, minus one rule.
That rule is statement coverage, and it turns out to be the whole thing. Applying
each account's newest billing_end_date as a watermark takes the real 2,607-row
Frollo export down to 171 rows — with no account allowlist, no credit-card
exclusion and no Frollo-specific scoping at all. Cards drop out on their own
because their statements are current; the 46 card rows that survive are genuinely
post-statement. Every bit of the bespoke apparatus was doing by hand what one
query does generically.
Deleted: src/lib/frollo-csv.ts, src/lib/frollo-ingest.ts, scripts/import-frollo.mts,
src/app/api/frollo/, both test files, the FROLLO_INGEST_TOKEN wiring, and the n8n
Frollo Import + Frollo Freshness Check workflows.
Added to the shared CSV path, so every import benefits:
- getStatementCoverage() + /api/import/statement-coverage. The review step
leaves out rows an account's statements already cover and says how many, with
the rows one click away. Only applies when an account column is mapped and
that account has statements — a row is never dropped on a guess.
- Optional Account and Row ID columns in the mapper. Account drives the
watermark and is stored as source_account; Row ID becomes source_ref.
- An in-file duplicate warning. A CDR re-consent re-exports history under
fresh ids, so source_ref cannot see it — 385 twins in one 2,563-row export
doubled every salary payment, and that is not visible by eye in a review
table.
Two pre-existing bugs in that path, both of which this plan depends on:
- The category chosen in the review step was accepted by
batchInsertCSVTransactions and then left out of the INSERT column list, so it
was silently discarded and the trigger wrote 'other'. Not cosmetic: an
uncategorised credit is admitted by NET_SPEND_ROWS and negated by
SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of spend while
counting as no income.
- The path had no idempotency whatsoever. row_index is assigned MAX+1 on every
run, which makes uq_transaction_identity structurally unable to fire, so a
second import of the same file duplicated all of it. Now writes source +
source_ref with ON CONFLICT DO NOTHING.
awaitsStatementLine()'s removal note is kept but rewritten: it no longer points
at a deleted file, and the lesson stands — the queue jump from 8 to 558 was the
measurement, not the noise.
|
||
|
siddharthd
|
b82c4570bd |
frollo: don't import what the statements already cover
ci / lint-test (push) Successful in 45s
"We should not be importing from Frollo what we already have from statements" (owner). The amount+direction guard could not deliver that, because the two sources decompose the same event differently: Frollo bundles the Wise fee into the transfer (10001.13) where the statement itemises it (10000.00 + 1.13). So 38 Wise USD rows passed the amount guard as new while being the same money, and were the reason the transactions view showed a USD figure where every neighbouring row showed AUD. A statement's billing_end_date is a hard watermark: everything on that account up to that date is already in the ledger, itemised and converted. Guard 1 now drops any feed row at or before its account's newest statement. Guard 2 (amount + direction) stays as the net for accounts that have no statement at all. Note this is the coverage test the first import needed and got wrong. That one asked whether a row's date fell inside a statement's min-max window, which for periods spanning 182 to 460 days swallows a year and answers nothing. The watermark asks a question that has an answer: up to what date is this account complete? Matching is on last4, verified against the live statement set — the eight in-scope accounts with statements each map to one bank, no cross-bank collision. The watermarks are printed by the CLI so a wrong boundary is visible rather than inferred from a row count. Re-imported from empty: 425 covered by statement, 15 amount twins, 110 inserted. Exactly one row now carries a foreign currency with no AUD figure — the 2026-08-12 HDR salary, which is the genuinely-new pre-statement row this feed exists for. Was 39. |
||
|
siddharthd
|
afd75d3f09 |
transactions: never print a foreign amount as AUD
ci / lint-test (push) Successful in 48s
The amount cell rendered formatAmount(amount_aud ?? amount) with no currency argument, so a row with no AUD figure fell back to its native amount and was stamped with a dollar sign. A USD 10,782.00 Wise credit showed as "+$10,782.00" in a column of AUD figures — the same payment from the statement, one row above, correctly showed $15,518.53 over USD 10,782.00. So the feed row read as $10,782 AUD when the real value is about $15,500, and the sub-line just repeated the same number correctly labelled. amount_unconverted is already selected by the transactions query (queries.ts:298) and was simply unused here. When set, the native figure becomes the headline and the absence of a rate is stated rather than papered over. Pre-existing rather than Frollo's: any row with amount_aud NULL and a foreign currency hits it, including foreign order-receipt rows. Frollo made it visible at scale — 39 rows. |
||
|
siddharthd
|
2738213a23 |
docs: record the ledger-duplicate guard and why awaitsStatementLine is gone
ci / lint-test (push) Successful in 40s
|
||
|
siddharthd
|
bb8a009e02 |
frollo: don't re-import what the ledger already has, and stop hiding the queue
ci / lint-test (push) Successful in 44s
The first import wrote 550 rows on 2026-08-13. 422 of them (77%) were second
copies of transactions the ledger already held from statements — $1,023,824.63
of movement counted twice. The owner found it by opening the transactions view
and seeing one HDR Global salary listed twice, once as A$15,518.53 from the
statement and once as US$10,782.00 from the feed.
The currency was never the defect. toLedgerRow already left amount_aud NULL and
named the currency in foreign_currency_code, which is the documented contract for
a row whose AUD value is unknown, and the transactions page labels it. What made
it look wrong was the duplicate sitting beside it.
Two changes.
Upstream, ingestFrolloCsv now drops rows the ledger already holds, matching on
amount + direction within LEDGER_MATCH_DAYS (3). Three things had to be right and
the first two were not, each caught only by rehearsing against real data rather
than fixtures:
- pg returns a DATE as a JS Date while Prisma and the CSV give strings.
String(date).slice(0,10) is "Wed Mar 10", which parses to NaN, so the first
dry run reported 550 to insert and zero duplicates. dayMs() takes both.
- Direction has to be in the key. This ledger is full of internal transfers
between the owner's own accounts and the feed carries both legs: 2026-05-18
has +3076.04 into ANZ and -3076.04 out of AMP. Matching on amount alone let
the credit leg consume the ledger's debit row, so the real duplicate was
written — 20 rows got in that way.
- 'refund' is money in. The feed calls a reversed account fee a credit and the
statement importer types it 'refund'; classifying it as an outflow left every
ANZ servicing-fee reversal behind.
Downstream, awaitsStatementLine() is removed. Its premise — feed rows never await
a statement line — was asserted, never tested, and false for almost every
account. Worse is how it got there: the reconcile queue jumped 8 -> 558 when the
feed landed, that jump was read as noise and filtered away, and filtering it
removed the only mechanism that would ever have collapsed the duplicates. The
queue was right. A feed row IS a row awaiting its statement line.
Re-imported: 375 dropped as already-on-ledger, 175 inserted. Residual duplicates
4 rows / $15.01, all sub-$5 account fees where several identical amounts fall in
overlapping windows and greedy consumption picks the wrong one; not chased
further at this scale.
The CLI prints the already-on-ledger count even when zero — a number you have to
go looking for is a number nobody looks at.
|
||
|
siddharthd
|
3e826a317b |
CLAUDE.md: document the Frollo account feed and the new reconcile exclusion
ci / lint-test (push) Successful in 39s
Two things a future reader would otherwise have to reverse-engineer. The reconcile queue now excludes account-feed rows for a reason unrelated to payment method: a feed row is the account's own ledger entry, not a receipt awaiting a statement line, because the importer only covers accounts whose statements are deliberately not imported. Measured, the queue went from 8 to 558 without it. And the de-duplication rule needs its history attached, because the obvious version is wrong in a way that passes tests: keeping close-id rows as genuine repeats survived 29 tests and a clean dry run while still doubling three salary payments. In-scope duplicate pairs have id gaps from 58 to 260 million; real repeats sat at 1-4. |
||
|
siddharthd
|
21e9e765a3 |
Add /api/frollo/ingest so the import can run unattended
ci / lint-test (push) Successful in 40s
Shares one module with the CLI rather than reimplementing the insert: frollo-ingest.ts holds parsing, scoping, de-duplication and the write, and both callers pass in their own SQL executor (Prisma in the route, a pg client in the script). The alternative is two implementations of the same insert, which is how the pantry healthcheck came to be fixed in one repo and left broken in the other. The route refuses rather than guesses. findAnomalies() returns every reason an unattended run should stop - a configured account contributing no rows, an unrecognised account, a near-consecutive-id collapse that might be a real repeat, a batch over ~200 rows, or an export taken with pending included - and the route answers 409 having written nothing. Two defects the wiring surfaced. Deliberately excluded credit cards were reported as unknown accounts, which would have raised the new-account anomaly on every single run and left the automatic path permanently refusing; EXCLUDED_ACCOUNTS now distinguishes excluded from unknown. And pending was tested after account scope, so pending rows on cards - which is all of them so far - classified as out-of-scope and the wrong-export-option signal could never fire; pending is now tested first. |
||
|
siddharthd
|
493ff6f631 |
Import Frollo account feeds for the accounts statements don't cover
ci / lint-test (push) Successful in 47s
Credit cards keep arriving as monthly statements and stay the source of truth for them. This covers the other fourteen accounts, whose statements arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise including the income account, Up, ING, and the small transaction accounts. About 50 rows a month, where the alternative is downloading each statement by hand. The file needs three defences, all found by diffing three real exports (smarthome DECISIONS.md ING-11): A CDR re-consent makes Frollo re-ingest an account's whole history under fresh transaction ids while the originals survive, and consents expire annually. On this export 112 rows were such twins, and every HDR salary payment appeared twice — importing blind doubles reported income. dedupe() collapses each natural-key group to its lowest id, lowest because old ids were a strict subset of new across two exports, so source_ref stays stable and a re-import inserts nothing. An earlier version of that rule kept close-id rows on a 10,000 threshold, reasoning that genuine same-day repeats have consecutive ids. Verifying it against the income rows killed it: in-scope duplicate pairs have id gaps from 58 to 260 million, so no threshold separates them from the gaps of 1-4 that real repeats showed. It now collapses unconditionally and flags anything within 10 for review — the errors are asymmetric, and nothing in scope has ever tripped the flag. A lapsed consent removes an account from the export silently, with no error and no marker; the row count just drops. So the import asserts the account roster and refuses to run when a configured account contributes nothing. Also holds these rows out of the pending-reconciliation queue. A feed row is the account's own ledger entry, not a receipt awaiting a statement line — these accounts' statements are deliberately not imported — so without the exclusion 550 rows a year would bury the receipts that need a decision. The queue stays at 8 instead of 558. Foreign rows follow order-ingestion's existing shape: amount is the native figure, foreign_currency_code names it, amount_aud stays NULL rather than asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as incomplete. Dry run by default. Verified against the real export before applying: 550 rows inserted, 14 accounts, re-run inserts 0. |
||
|
siddharthd
|
2c666236b2 |
orders: share the merchant tidy, and fix two bugs the corpus run found (board 212)
ci / lint-test (push) Successful in 41s
The tidy existed but lived inside the list page, so the DETAIL page never had it: following a row through showed "Apple" becoming "Apple Pty Ltd." Moved to src/lib/merchant-label.ts and applied on both surfaces. Running it over all 1,451 distinct merchant names — rather than over examples — turned up two real defects: Coburger & Co -> "Coburger &" eBay Commerce Australia Pty Ltd. -> "EBay Commerce Australia" The first strips a suffix that is part of the brand and leaves a dangling connector. The second is the capitalisation rule firing on "starts with a lower-case letter" when what it means is "is a bare domain" — mangling a brand that is deliberately lower-cased. Both are now guarded, and re-running over the full corpus reports 170 names tidied and 0 defects. PRESENTATION ONLY. It merges nothing and must never decide two rows are the same merchant. Amazon.in stays Amazon.in — a different marketplace with different currency and geography, not a name variant — and there is a test asserting it can never equal amazon.com.au. Kogan.com, GOG.com, AliExpress.com and Catch.com.au are real brand names containing a TLD and are left alone. Real unification is the merchant-alias bridge, ticket 176, which has already merged what can be merged safely. The five names that read alike after tidying (ebay/eBay/eBay Inc., menulog/Menulog Pty Ltd, deliveroo/Deliveroo, grab/Grab, paypal/PayPal) are genuinely one merchant each, so reading alike is correct. Lower-case platform SLUGS are deliberately not capitalised: "ebay" -> "Ebay" is wrong, and getting it right is the registry's job, not the stylesheet's. |
||
|
siddharthd
|
215303161b |
orders: bring the lifecycle-only toggle's numbers up to date (board 210)
ci / lint-test (push) Successful in 42s
The count moved three times on 2026-08-12 and the history is the point:
755 when the predicate was written
904 after board 219 turned unstated zeros into NULLs — those 149 rows had
been failing the "no amount" test on a technicality
699 after 210a retired 205 Amazon "share your experience" mails, which were
never lifecycle events at all
620 after 210b joined 79 phantoms to their real parent, placebo-verified at
a 1.9% error rate
What is left is the corpus coverage ceiling rather than a defect — a mail whose
parent is not in the corpus has nothing to join to — and the tooltip now says
that instead of describing the whole class as an open ingestion defect.
|
||
|
siddharthd
|
3b5e495ca4 |
orders phase 2: order_transaction_links, and move every reader to it (board 205)
ci / lint-test (push) Successful in 41s
Phase 1 read linkage out of expense_metadata, which is shaped as ONE ROW PER
TRANSACTION — transaction_id UNIQUE, matched_transaction_id partial-unique — so
every multiplicity it expresses is smuggled through a string key (0029 keys
split shipments <entity_key>#f<fact_id>). A BNPL plan needs four rows against
one order and has no such trick available.
Migration 0030 adds order_transaction_links (many-to-many, keyed on entity_key
rather than entities.id: finance-app does not model the spine and must not hold
an FK across a boundary a re-extraction can decompile) and backfills all 63
existing bridge rows.
THE UNIQUENESS RULE WAS WRONG FIRST TIME, in the most on-brand way available.
It read (entity_key, leg_kind, COALESCE(leg_index, 0)), so two shipment legs of
the same order — both leg_index NULL — collapsed to one key and ON CONFLICT DO
NOTHING dropped one SILENTLY. Caught only because the backfill reported 62
against 63 candidates. The row it ate was order_amazon_249-4859367-0690246's
$130.00 second shipment, the same order named in migration 0029's comment as
the reason split shipments need distinct keys at all. What identifies a leg
depends on its kind: an instalment by its INDEX, a shipment by its FACT, a
whole-order charge by neither.
ALL FOUR READERS MOVE TOGETHER, links first with expense_metadata as fallback:
- LINK_LATERAL (list txn_count/first_txn_id, and the has_transaction facets
that read it)
- the detail page's transactions query
- /api/transactions/[id]/order
- the order_ctx lateral in queries.ts
Moving fewer is not a smaller change, it is an inconsistent one: the matcher's
four links for order_ebay_14-11714-95953 come with no expense_metadata row, so
a half-move would show four instalments on the detail page while the list said
txn_count = 0 and put the order on the wrong side of BOTH has_transaction
filters. Verified after: detail 4 legs, list txn_count 4, has_transaction=yes
includes it, =no excludes it.
UNION not UNION ALL on transaction_id — after the backfill the same charge is
legitimately in both stores and counting it twice would show "2 charges" on a
single-payment order.
order_platform is deliberately NOT coalesced with the link's platform. It gates
the receipt disclosure arrow on /transactions, and a BNPL leg has no receipt
behind it — filling it in put an arrow on four Afterpay rows that expand to
nothing, which is the exact promise the arrow exists to avoid making. Separate
leg fields carry the sub-line instead ("1 of 4 - DJI Air 3 Fly More Combo").
order-details.tsx now also requires a real receipt (platform present) before
rendering, because the endpoint can answer with a link alone.
Two invariant views, both empty and expected to stay so: order_link_orphans
(spine re-keying silently orphans a TEXT key — bridge links are rebuildable,
`manual` ones are lost curation) and order_link_drift (the two stores
disagreeing). The plan's suggested fix for drift — widening the bridge's NOT
EXISTS guard to "neither store has it" — was NOT taken: the expense_metadata
INSERT has no ON CONFLICT, so re-attempting an order that already has a receipt
row would duplicate it. Detecting is cheap; a non-idempotent re-write is not.
Unchanged: order_feed 6,265, order_spend AUD 4,895 / $442,651.80. Links 75.
|
||
|
siddharthd
|
6b9b5fe518 |
orders: badge the annual summary receipts (board 211)
ci / lint-test (push) Successful in 40s
Migration 033 adds entity_orders.summarises_period; order_spend drops those rows and order_feed keeps them. Unbadged, a $2,376 annual tax receipt restating twelve monthly donations renders as the largest single purchase of the year, so both surfaces say what it is: - /orders — a "year summary" chip beside the auth badges - /orders/<key> — a banner in the same shape as the settled-rail one, saying the monthly debits are in the ledger under their own dates The amount is deliberately still shown. It is the year's giving total and the reason the row is worth keeping; it just belongs to no total on the page. The detail page already falls back to o.order_total when order_spend has no row, which is exactly the case now — verified: gross_total null, order_total 2125.5. |
||
|
siddharthd
|
231ef3411a |
orders: record why line items may not exempt a row from the phantom filter
ci / lint-test (push) Successful in 40s
Board 219 nulled 263 unstated zero totals in the spine, which moved 149 rows into the lifecycle-only hide predicate — they had been failing its "no amount" test on a technicality, an unstated 0 they should never have carried, while matching the phantom signature on every other count. Tried exempting rows that carry line items, on the theory that naming what was bought proves a purchase. It surfaced 442 rows and the first window checked showed why it is wrong: the Amazon "Ordered:" mail that mints a msg- phantom carries the item list too, so Xelsluthe and YIWENTEC each rendered TWICE again — the exact complaint this page was fixed for today. Reverted, and the reason recorded in place so it is not re-attempted. Items do not separate an orphan purchase from an item-carrying echo. Only deduplication does, and that is board 210. No behaviour change from the previous commit; comments and counts only. |
||
|
siddharthd
|
699e4a2ddd |
orders: cast cadence order_count to int — BigInt broke every recurring merchant
ci / lint-test (push) Successful in 43s
order_merchant_cadence.order_count is a count(*), so BIGINT, and JSON.stringify throws on it: 'Do not know how to serialize a BigInt', a 500 carrying no SQL. It fires only where a cadence EXISTS, so the failure hid behind the merchant_entity_id fault and reappeared the moment that was fixed — St. Ali (40 orders, 14-day cadence) still 500'd while Amazon opened fine. |
||
|
siddharthd
|
79940202f1 |
orders: fix the 42703 that broke EVERY order detail page
getOrderDetail joined order_merchant_cadence on merchant_entity_id. Migration 021 rekeyed that view to merchant_key (merchant_entity_id is NULL on ~9% of the feed, and duplicate merchant entities split one shop's history); 027 and 029 moved order_feed and order_spend across, and this query was missed. Every /orders/<key> request has since failed with column rec.merchant_entity_id does not exist The reason nobody saw a 500 is the second half of this commit: the page collapsed every failure into 'That order could not be found.' A server fault wearing the costume of a data condition reads as an empty spine and gets investigated in the wrong repo. The hook now carries the status and only a genuine 404 says the order is missing. |
||
|
siddharthd
|
9790b64e1d |
orders: badge sender authentication, not source_trust
source_trust is 'untrusted_external' on 100% of rows — every order came from email — so the badge marked every row and discriminated nothing. auth_verdict (migration 029) does: 93% pass, and the 7% that do not are the ones worth seeing. Adds the injection-scanner flag beside it. |