bb8a009e0270b5cc015eae60bcb5dc71e02c330f
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 siddharthd
|
bb8a009e02 |
frollo: don't re-import what the ledger already has, and stop hiding the queue
ci / lint-test (push) Successful in 44s
The first import wrote 550 rows on 2026-08-13. 422 of them (77%) were second
copies of transactions the ledger already held from statements — $1,023,824.63
of movement counted twice. The owner found it by opening the transactions view
and seeing one HDR Global salary listed twice, once as A$15,518.53 from the
statement and once as US$10,782.00 from the feed.
The currency was never the defect. toLedgerRow already left amount_aud NULL and
named the currency in foreign_currency_code, which is the documented contract for
a row whose AUD value is unknown, and the transactions page labels it. What made
it look wrong was the duplicate sitting beside it.
Two changes.
Upstream, ingestFrolloCsv now drops rows the ledger already holds, matching on
amount + direction within LEDGER_MATCH_DAYS (3). Three things had to be right and
the first two were not, each caught only by rehearsing against real data rather
than fixtures:
- pg returns a DATE as a JS Date while Prisma and the CSV give strings.
String(date).slice(0,10) is "Wed Mar 10", which parses to NaN, so the first
dry run reported 550 to insert and zero duplicates. dayMs() takes both.
- Direction has to be in the key. This ledger is full of internal transfers
between the owner's own accounts and the feed carries both legs: 2026-05-18
has +3076.04 into ANZ and -3076.04 out of AMP. Matching on amount alone let
the credit leg consume the ledger's debit row, so the real duplicate was
written — 20 rows got in that way.
- 'refund' is money in. The feed calls a reversed account fee a credit and the
statement importer types it 'refund'; classifying it as an outflow left every
ANZ servicing-fee reversal behind.
Downstream, awaitsStatementLine() is removed. Its premise — feed rows never await
a statement line — was asserted, never tested, and false for almost every
account. Worse is how it got there: the reconcile queue jumped 8 -> 558 when the
feed landed, that jump was read as noise and filtered away, and filtering it
removed the only mechanism that would ever have collapsed the duplicates. The
queue was right. A feed row IS a row awaiting its statement line.
Re-imported: 375 dropped as already-on-ledger, 175 inserted. Residual duplicates
4 rows / $15.01, all sub-$5 account fees where several identical amounts fall in
overlapping windows and greedy consumption picks the wrong one; not chased
further at this scale.
The CLI prints the already-on-ledger count even when zero — a number you have to
go looking for is a number nobody looks at.
|
||
|
siddharthd
|
21e9e765a3 |
Add /api/frollo/ingest so the import can run unattended
ci / lint-test (push) Successful in 40s
Shares one module with the CLI rather than reimplementing the insert: frollo-ingest.ts holds parsing, scoping, de-duplication and the write, and both callers pass in their own SQL executor (Prisma in the route, a pg client in the script). The alternative is two implementations of the same insert, which is how the pantry healthcheck came to be fixed in one repo and left broken in the other. The route refuses rather than guesses. findAnomalies() returns every reason an unattended run should stop - a configured account contributing no rows, an unrecognised account, a near-consecutive-id collapse that might be a real repeat, a batch over ~200 rows, or an export taken with pending included - and the route answers 409 having written nothing. Two defects the wiring surfaced. Deliberately excluded credit cards were reported as unknown accounts, which would have raised the new-account anomaly on every single run and left the automatic path permanently refusing; EXCLUDED_ACCOUNTS now distinguishes excluded from unknown. And pending was tested after account scope, so pending rows on cards - which is all of them so far - classified as out-of-scope and the wrong-export-option signal could never fire; pending is now tested first. |
||
|
siddharthd
|
493ff6f631 |
Import Frollo account feeds for the accounts statements don't cover
ci / lint-test (push) Successful in 47s
Credit cards keep arriving as monthly statements and stay the source of truth for them. This covers the other fourteen accounts, whose statements arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise including the income account, Up, ING, and the small transaction accounts. About 50 rows a month, where the alternative is downloading each statement by hand. The file needs three defences, all found by diffing three real exports (smarthome DECISIONS.md ING-11): A CDR re-consent makes Frollo re-ingest an account's whole history under fresh transaction ids while the originals survive, and consents expire annually. On this export 112 rows were such twins, and every HDR salary payment appeared twice — importing blind doubles reported income. dedupe() collapses each natural-key group to its lowest id, lowest because old ids were a strict subset of new across two exports, so source_ref stays stable and a re-import inserts nothing. An earlier version of that rule kept close-id rows on a 10,000 threshold, reasoning that genuine same-day repeats have consecutive ids. Verifying it against the income rows killed it: in-scope duplicate pairs have id gaps from 58 to 260 million, so no threshold separates them from the gaps of 1-4 that real repeats showed. It now collapses unconditionally and flags anything within 10 for review — the errors are asymmetric, and nothing in scope has ever tripped the flag. A lapsed consent removes an account from the export silently, with no error and no marker; the row count just drops. So the import asserts the account roster and refuses to run when a configured account contributes nothing. Also holds these rows out of the pending-reconciliation queue. A feed row is the account's own ledger entry, not a receipt awaiting a statement line — these accounts' statements are deliberately not imported — so without the exclusion 550 rows a year would bury the receipts that need a decision. The queue stays at 8 instead of 558. Foreign rows follow order-ingestion's existing shape: amount is the native figure, foreign_currency_code names it, amount_aud stays NULL rather than asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as incomplete. Dry run by default. Verified against the real export before applying: 550 rows inserted, 14 accounts, re-run inserts 0. |
||
|
siddharthd
|
b4a116c134 |
feat(scripts): import the matched split history as settled
ci / lint-test (push) Successful in 46s
Adds --write to the matcher. Wrote 1,242 split rows across 657 transactions. Imported settled, and that is the whole design. These obligations were discharged years ago on a platform we no longer run, and their residual is already carried by transaction 2348. Writing them unsettled would re-open roughly $40k of debts that were paid. ACTIVE_OBLIGATION keeps settled splits out of every owed figure while myShare/mySplitOf still count them, which is exactly the asymmetry this needs: the import exists to correct historical SPEND, not to move a balance. Effect: $35,259 leaves my historical spend -- $13,088 in 2024, $22,117 in 2025 -- because a $200 grocery shop that was always half hers no longer reads as $200 of mine. Balances are byte-identical before and after (Molina -1226.72/145, Sonu 5428.08/419), which is the assertion that matters. Shares are written as the CSV computed them, so a 50/50 row can land as 50.01/49.99. That is faithful rather than tidy; no transaction exceeds 100%. Rehearsed on the 37-row Rome file first (24 rows) and verified before the full run -- both the balances and one split read back through the API. |
||
|
siddharthd
|
c9b000a428 |
feat(scripts): dry-run matcher for the SplitMyExpenses history
ci / lint-test (push) Successful in 48s
Matches the five CSV exports against transactions already in the ledger and
reports what it would do. Writes nothing -- importing is a separate step, and
rehearsing it first is what catches the defects that tests do not.
What it found, and why the number is what it is: 676 of 1,536 shareable rows
match (44%). The ceiling is ledger coverage, not matcher quality. The CSVs
describe 678 shared expenses in 2024 alone; the ledger holds 591 rows for the
whole of that year, 3 to 72 a month, which is far less than a household
actually spends. Most 2024 CSV rows have no transaction to attach a split to
and never will. South Korea April 2024 matches 4 of 158 for that reason.
Three decisions are encoded deliberately:
- Date format is decided per FILE, not per row. The household export writes
D/M/YYYY and the four trip exports write ISO, and 474 rows parse validly
under both readings -- per-row guessing silently swaps January and February
for some rows and not others.
- A person's column is their net balance impact, not their share. The payer
is whoever is positive; the other's share is |their negative| / cost. So a
+cost/-cost row means the other party owes 100%, not that the expense was
unshared -- the reading that would fake an arrangement change.
- Matching is one-to-one, best pair first. The NZ trip has two identical
$10.16 Uber rows against three ledger rows and four PayMyPark rows in the
same shape; without this a ledger row is claimed repeatedly and the second
CSV row looks matched while being unrepresented.
|
||
|
siddharthd
|
1296555f17 |
test: add unit and integration test suites
- Extract evaluateCondition + rule types into src/lib/rules.ts for testability - 48 unit tests for evaluateCondition (all fields/operators) and formatCategory - 21 integration tests for getTransactions filters and getParticipantBalances - Vitest configs for unit (vitest.config.ts) and integration (vitest.integration.config.ts) - setup-test-db.sh creates personal_test DB from production schema via pg_dump - Use vi.doMock + dynamic import pattern to isolate test DB from Prisma singleton |