diff --git a/CLAUDE.md b/CLAUDE.md index ea9e2b6..e3cbf5c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -636,87 +636,54 @@ deleted, and deleting a rule must not cascade away the audit trail. changed by something else since the run are flagged, because reverting restores the pre-run value and discards the later edit. -### Frollo account feed (`source = 'frollo'`) +### Importing an aggregator/bank CSV (`/api/import/csv`) -Covers the accounts whose statements arrive every 182 to 460 days — AMP including -the loan, ANZ Access, Wise including the income account, Up, ING, the small -transaction accounts. **Credit cards are excluded on purpose**: their monthly -statements already cover them and are authoritative, and a second producer for -the same rows would only create reconciliation work. +There is **one** import path — the CSV import modal — and Frollo goes through it +like any other file. A bespoke Frollo importer, API route, CLI and two scheduled +n8n workflows existed for a day and were deleted on 2026-08-13: net −1,152 lines. -- Logic: `src/lib/frollo-csv.ts` (pure, parsing + scoping + de-duplication) and - `src/lib/frollo-ingest.ts` (report + insert + the ledger-duplicate guard). - 53 unit tests across `frollo-csv.test.ts` and `frollo-ingest.test.ts`. +**The rule that made the rest unnecessary is statement coverage.** Each account's +newest `billing_end_date` is a watermark; a row on or before it is already in the +ledger. On the real 2,607-row export that leaves **171 rows** — with no account +allowlist and no credit-card exclusion, because cards have current statements and +drop out on their own. Every bit of the deleted apparatus was doing by hand what +`getStatementCoverage()` does generically. -**There are TWO de-duplications and they solve different problems.** `dedupe()` -compares the file against itself (CDR re-consent twins, below). The -ledger-duplicate guard compares the file against `transactions`, and its absence -is what made the first import write 422 duplicates out of 550 — $1,023,824.63 -counted twice — because "these accounts issue no statements" had been asserted -rather than queried. The check that would have caught it is one SQL statement. -The check run instead compared each row's date to the statement's min–max window, -which for accounts whose statements span 182 to 460 days swallows a year and -cannot distinguish covered from uncovered at all. +Note what this replaced. The first attempt asked whether a row's date fell inside +a statement's min–max *window*, which for accounts whose statements span 182 to +460 days swallows a year and answers nothing — it let 422 duplicates into 550 +rows. Amount-matching cannot substitute either: the two sources decompose the +same event differently, bundling a Wise transfer fee into the transfer (10001.13) +where the statement itemises it (10000.00 + 1.13). -The guard matches on **amount + direction, within `LEDGER_MATCH_DAYS` (3)**, and -consumes each ledger row once so a genuine repeat survives. Three details, each -of which was wrong first and each caught only by rehearsing against the real -database rather than fixtures: +**Map these optional columns or lose something silently:** -- **`pg` returns a DATE as a JS `Date`; Prisma and the CSV give strings.** - `String(date).slice(0, 10)` is `"Wed Mar 10"` → NaN → every comparison skipped, - silently. The first dry run reported 550 to insert against a ledger holding 422 - of them. `dayMs()` accepts both, and both shapes are tested because both - drivers are in use. -- **Direction must be in the key.** This ledger is full of internal transfers - between the owner's own accounts and the feed carries *both legs*: 2026-05-18 - has +3076.04 into ANZ and −3076.04 out of AMP. On amount alone the credit leg - consumed the ledger's debit row and the real duplicate was written — 20 rows. -- **`refund` is money in.** Ledger types are debit | credit | payment | fee | - interest | refund; the feed calls a reversed account fee a `credit` while the - statement importer types it `refund`. Classifying it as an outflow left every - ANZ servicing-fee reversal behind. +| Column | Without it | +|---|---| +| **Account** | nothing is excluded — you review the whole file | +| **Currency** | a foreign row is stored as if AUD (USD 10,782 → A$10,782) | +| **Row ID** | a re-import duplicates instead of no-opping | -Residual after all three: **4 rows / $15.01**, all sub-$5 account fees where -several identical amounts fall in overlapping windows and greedy consumption -picks the wrong one. Left alone deliberately at this scale. +Leave **Category** unmapped for an aggregator: its spend categories are not +trusted, and you set them per row in the review step. -The count is printed by the CLI even when zero. A number you have to go looking -for is a number nobody looks at. -- Two callers share that module so they cannot drift: `scripts/import-frollo.mts` - (dry run by default) and `POST /api/frollo/ingest` (token auth, called daily by - the n8n `Frollo Import` workflow). The DB is reached through an injected - executor because one runs under Prisma and the other under a bare `pg` client. -- **Export with pending EXCLUDED.** A pending row changes both its id *and* its - description when it settles, so importing one guarantees a duplicate next run. +**Two bugs fixed the day this shipped, both of which the flow depends on:** +`batchInsertCSVTransactions` accepted a `category` and then omitted it from the +INSERT column list, so every category chosen in review was discarded and the +trigger wrote `'other'` — not cosmetic, because an uncategorised credit is +admitted by `NET_SPEND_ROWS` and negated by `SPEND_SIGNED`, so 62 imported +transfers cancelled **$74,338** of spend while counting as no income. And the +path had no idempotency at all: `row_index` is assigned MAX+1 every run, making +`uq_transaction_identity` structurally unable to fire. It now writes +`source`/`source_ref` with `ON CONFLICT DO NOTHING`. -**The one thing to understand before touching this: a CDR re-consent makes Frollo -re-ingest an account's whole history under fresh transaction ids while the -originals survive**, and consents expire annually. On one export 385 of 1,989 rows -were such twins and every salary payment appeared twice — a blind import doubles -reported income. `dedupe()` collapses each natural-key group to its **lowest** id -(lowest because old ids survive re-ingestion, which keeps `source_ref` stable and -makes a re-import insert nothing). +**The in-file duplicate warning is not redundant with `source_ref`.** A CDR +re-consent re-exports an account's whole history under fresh ids while the +originals survive — 385 twins in one 2,563-row export, doubling every salary +payment. Fresh ids mean `source_ref` sees new rows, and 385 is not visible by eye +in a review table. Consents expire annually, so expect it. -An earlier version kept close-id rows on a 10,000 threshold, reasoning that -genuine same-day repeats have consecutive ids. It passed 29 tests and a clean dry -run and was still wrong: in-scope duplicate pairs have id gaps from **58 to 260 -million**, while real repeats sat at 1–4. Nothing separates them. It now collapses -unconditionally and *flags* anything within 10 for review, because the errors are -asymmetric — keeping a twin doubles income, collapsing a real repeat understates -by one row. - -`category` is not imported: Frollo assigns its own and the owner reports they are -often wrong. Note a trigger rewrites NULL to `'other'` on insert, which is -equivalent for display and leaves the rules engine (which writes -`transaction_overrides.category_override`) free to decide. - -Foreign rows carry only the native figure — there is no AUD equivalent anywhere in -the export — so they take order-ingestion's shape: `amount` is native, -`foreign_currency_code` names it, `amount_aud` stays NULL, and -`AMOUNT_UNCONVERTED` already reports the balance as incomplete. - -Full history and the reasoning: `DECISIONS.md` ING-11 in the smarthome repo. +Undo an import: `DELETE FROM transactions WHERE source = ''`. ### Trusting extracted statement data