22c2349a47cf0e28445ff746dc8ae79be49ce08e
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 siddharthd
|
461c021e7a |
csv import: exclude what the statements already cover, and delete the Frollo importer
ci / lint-test (push) Successful in 43s
"This is becoming too complex... Frollo should be done through that [the manual
CSV import]" (owner). It was right: a bespoke importer, an API route, a CLI, two
scheduled n8n workflows and a shared secret existed to do what the CSV import
modal already did, minus one rule.
That rule is statement coverage, and it turns out to be the whole thing. Applying
each account's newest billing_end_date as a watermark takes the real 2,607-row
Frollo export down to 171 rows — with no account allowlist, no credit-card
exclusion and no Frollo-specific scoping at all. Cards drop out on their own
because their statements are current; the 46 card rows that survive are genuinely
post-statement. Every bit of the bespoke apparatus was doing by hand what one
query does generically.
Deleted: src/lib/frollo-csv.ts, src/lib/frollo-ingest.ts, scripts/import-frollo.mts,
src/app/api/frollo/, both test files, the FROLLO_INGEST_TOKEN wiring, and the n8n
Frollo Import + Frollo Freshness Check workflows.
Added to the shared CSV path, so every import benefits:
- getStatementCoverage() + /api/import/statement-coverage. The review step
leaves out rows an account's statements already cover and says how many, with
the rows one click away. Only applies when an account column is mapped and
that account has statements — a row is never dropped on a guess.
- Optional Account and Row ID columns in the mapper. Account drives the
watermark and is stored as source_account; Row ID becomes source_ref.
- An in-file duplicate warning. A CDR re-consent re-exports history under
fresh ids, so source_ref cannot see it — 385 twins in one 2,563-row export
doubled every salary payment, and that is not visible by eye in a review
table.
Two pre-existing bugs in that path, both of which this plan depends on:
- The category chosen in the review step was accepted by
batchInsertCSVTransactions and then left out of the INSERT column list, so it
was silently discarded and the trigger wrote 'other'. Not cosmetic: an
uncategorised credit is admitted by NET_SPEND_ROWS and negated by
SPEND_SIGNED, so 62 imported transfers cancelled $74,338 of spend while
counting as no income.
- The path had no idempotency whatsoever. row_index is assigned MAX+1 on every
run, which makes uq_transaction_identity structurally unable to fire, so a
second import of the same file duplicated all of it. Now writes source +
source_ref with ON CONFLICT DO NOTHING.
awaitsStatementLine()'s removal note is kept but rewritten: it no longer points
at a deleted file, and the lesson stands — the queue jump from 8 to 558 was the
measurement, not the noise.
|
||
|
siddharthd
|
b82c4570bd |
frollo: don't import what the statements already cover
ci / lint-test (push) Successful in 45s
"We should not be importing from Frollo what we already have from statements" (owner). The amount+direction guard could not deliver that, because the two sources decompose the same event differently: Frollo bundles the Wise fee into the transfer (10001.13) where the statement itemises it (10000.00 + 1.13). So 38 Wise USD rows passed the amount guard as new while being the same money, and were the reason the transactions view showed a USD figure where every neighbouring row showed AUD. A statement's billing_end_date is a hard watermark: everything on that account up to that date is already in the ledger, itemised and converted. Guard 1 now drops any feed row at or before its account's newest statement. Guard 2 (amount + direction) stays as the net for accounts that have no statement at all. Note this is the coverage test the first import needed and got wrong. That one asked whether a row's date fell inside a statement's min-max window, which for periods spanning 182 to 460 days swallows a year and answers nothing. The watermark asks a question that has an answer: up to what date is this account complete? Matching is on last4, verified against the live statement set — the eight in-scope accounts with statements each map to one bank, no cross-bank collision. The watermarks are printed by the CLI so a wrong boundary is visible rather than inferred from a row count. Re-imported from empty: 425 covered by statement, 15 amount twins, 110 inserted. Exactly one row now carries a foreign currency with no AUD figure — the 2026-08-12 HDR salary, which is the genuinely-new pre-statement row this feed exists for. Was 39. |
||
|
siddharthd
|
bb8a009e02 |
frollo: don't re-import what the ledger already has, and stop hiding the queue
ci / lint-test (push) Successful in 44s
The first import wrote 550 rows on 2026-08-13. 422 of them (77%) were second
copies of transactions the ledger already held from statements — $1,023,824.63
of movement counted twice. The owner found it by opening the transactions view
and seeing one HDR Global salary listed twice, once as A$15,518.53 from the
statement and once as US$10,782.00 from the feed.
The currency was never the defect. toLedgerRow already left amount_aud NULL and
named the currency in foreign_currency_code, which is the documented contract for
a row whose AUD value is unknown, and the transactions page labels it. What made
it look wrong was the duplicate sitting beside it.
Two changes.
Upstream, ingestFrolloCsv now drops rows the ledger already holds, matching on
amount + direction within LEDGER_MATCH_DAYS (3). Three things had to be right and
the first two were not, each caught only by rehearsing against real data rather
than fixtures:
- pg returns a DATE as a JS Date while Prisma and the CSV give strings.
String(date).slice(0,10) is "Wed Mar 10", which parses to NaN, so the first
dry run reported 550 to insert and zero duplicates. dayMs() takes both.
- Direction has to be in the key. This ledger is full of internal transfers
between the owner's own accounts and the feed carries both legs: 2026-05-18
has +3076.04 into ANZ and -3076.04 out of AMP. Matching on amount alone let
the credit leg consume the ledger's debit row, so the real duplicate was
written — 20 rows got in that way.
- 'refund' is money in. The feed calls a reversed account fee a credit and the
statement importer types it 'refund'; classifying it as an outflow left every
ANZ servicing-fee reversal behind.
Downstream, awaitsStatementLine() is removed. Its premise — feed rows never await
a statement line — was asserted, never tested, and false for almost every
account. Worse is how it got there: the reconcile queue jumped 8 -> 558 when the
feed landed, that jump was read as noise and filtered away, and filtering it
removed the only mechanism that would ever have collapsed the duplicates. The
queue was right. A feed row IS a row awaiting its statement line.
Re-imported: 375 dropped as already-on-ledger, 175 inserted. Residual duplicates
4 rows / $15.01, all sub-$5 account fees where several identical amounts fall in
overlapping windows and greedy consumption picks the wrong one; not chased
further at this scale.
The CLI prints the already-on-ledger count even when zero — a number you have to
go looking for is a number nobody looks at.
|
||
|
siddharthd
|
21e9e765a3 |
Add /api/frollo/ingest so the import can run unattended
ci / lint-test (push) Successful in 40s
Shares one module with the CLI rather than reimplementing the insert: frollo-ingest.ts holds parsing, scoping, de-duplication and the write, and both callers pass in their own SQL executor (Prisma in the route, a pg client in the script). The alternative is two implementations of the same insert, which is how the pantry healthcheck came to be fixed in one repo and left broken in the other. The route refuses rather than guesses. findAnomalies() returns every reason an unattended run should stop - a configured account contributing no rows, an unrecognised account, a near-consecutive-id collapse that might be a real repeat, a batch over ~200 rows, or an export taken with pending included - and the route answers 409 having written nothing. Two defects the wiring surfaced. Deliberately excluded credit cards were reported as unknown accounts, which would have raised the new-account anomaly on every single run and left the automatic path permanently refusing; EXCLUDED_ACCOUNTS now distinguishes excluded from unknown. And pending was tested after account scope, so pending rows on cards - which is all of them so far - classified as out-of-scope and the wrong-export-option signal could never fire; pending is now tested first. |
||
|
siddharthd
|
493ff6f631 |
Import Frollo account feeds for the accounts statements don't cover
ci / lint-test (push) Successful in 47s
Credit cards keep arriving as monthly statements and stay the source of truth for them. This covers the other fourteen accounts, whose statements arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise including the income account, Up, ING, and the small transaction accounts. About 50 rows a month, where the alternative is downloading each statement by hand. The file needs three defences, all found by diffing three real exports (smarthome DECISIONS.md ING-11): A CDR re-consent makes Frollo re-ingest an account's whole history under fresh transaction ids while the originals survive, and consents expire annually. On this export 112 rows were such twins, and every HDR salary payment appeared twice — importing blind doubles reported income. dedupe() collapses each natural-key group to its lowest id, lowest because old ids were a strict subset of new across two exports, so source_ref stays stable and a re-import inserts nothing. An earlier version of that rule kept close-id rows on a 10,000 threshold, reasoning that genuine same-day repeats have consecutive ids. Verifying it against the income rows killed it: in-scope duplicate pairs have id gaps from 58 to 260 million, so no threshold separates them from the gaps of 1-4 that real repeats showed. It now collapses unconditionally and flags anything within 10 for review — the errors are asymmetric, and nothing in scope has ever tripped the flag. A lapsed consent removes an account from the export silently, with no error and no marker; the row count just drops. So the import asserts the account roster and refuses to run when a configured account contributes nothing. Also holds these rows out of the pending-reconciliation queue. A feed row is the account's own ledger entry, not a receipt awaiting a statement line — these accounts' statements are deliberately not imported — so without the exclusion 550 rows a year would bury the receipts that need a decision. The queue stays at 8 instead of 558. Foreign rows follow order-ingestion's existing shape: amount is the native figure, foreign_currency_code names it, amount_aud stays NULL rather than asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as incomplete. Dry run by default. Verified against the real export before applying: 550 rows inserted, 14 accounts, re-run inserts 0. |
||
|
siddharthd
|
b4a116c134 |
feat(scripts): import the matched split history as settled
ci / lint-test (push) Successful in 46s
Adds --write to the matcher. Wrote 1,242 split rows across 657 transactions. Imported settled, and that is the whole design. These obligations were discharged years ago on a platform we no longer run, and their residual is already carried by transaction 2348. Writing them unsettled would re-open roughly $40k of debts that were paid. ACTIVE_OBLIGATION keeps settled splits out of every owed figure while myShare/mySplitOf still count them, which is exactly the asymmetry this needs: the import exists to correct historical SPEND, not to move a balance. Effect: $35,259 leaves my historical spend -- $13,088 in 2024, $22,117 in 2025 -- because a $200 grocery shop that was always half hers no longer reads as $200 of mine. Balances are byte-identical before and after (Molina -1226.72/145, Sonu 5428.08/419), which is the assertion that matters. Shares are written as the CSV computed them, so a 50/50 row can land as 50.01/49.99. That is faithful rather than tidy; no transaction exceeds 100%. Rehearsed on the 37-row Rome file first (24 rows) and verified before the full run -- both the balances and one split read back through the API. |
||
|
siddharthd
|
c9b000a428 |
feat(scripts): dry-run matcher for the SplitMyExpenses history
ci / lint-test (push) Successful in 48s
Matches the five CSV exports against transactions already in the ledger and
reports what it would do. Writes nothing -- importing is a separate step, and
rehearsing it first is what catches the defects that tests do not.
What it found, and why the number is what it is: 676 of 1,536 shareable rows
match (44%). The ceiling is ledger coverage, not matcher quality. The CSVs
describe 678 shared expenses in 2024 alone; the ledger holds 591 rows for the
whole of that year, 3 to 72 a month, which is far less than a household
actually spends. Most 2024 CSV rows have no transaction to attach a split to
and never will. South Korea April 2024 matches 4 of 158 for that reason.
Three decisions are encoded deliberately:
- Date format is decided per FILE, not per row. The household export writes
D/M/YYYY and the four trip exports write ISO, and 474 rows parse validly
under both readings -- per-row guessing silently swaps January and February
for some rows and not others.
- A person's column is their net balance impact, not their share. The payer
is whoever is positive; the other's share is |their negative| / cost. So a
+cost/-cost row means the other party owes 100%, not that the expense was
unshared -- the reading that would fake an arrangement change.
- Matching is one-to-one, best pair first. The NZ trip has two identical
$10.16 Uber rows against three ledger rows and four PayMyPark rows in the
same shape; without this a ledger row is claimed repeatedly and the second
CSV row looks matched while being unrepresented.
|
||
|
siddharthd
|
1296555f17 |
test: add unit and integration test suites
- Extract evaluateCondition + rule types into src/lib/rules.ts for testability - 48 unit tests for evaluateCondition (all fields/operators) and formatCategory - 21 integration tests for getTransactions filters and getParticipantBalances - Vitest configs for unit (vitest.config.ts) and integration (vitest.integration.config.ts) - setup-test-db.sh creates personal_test DB from production schema via pg_dump - Use vi.doMock + dynamic import pattern to isolate test DB from Prisma singleton |