4 Commits

Include renames

Author	SHA1	Message	Date
siddharthd	b82c4570bd	frollo: don't import what the statements already cover ... "We should not be importing from Frollo what we already have from statements" (owner). The amount+direction guard could not deliver that, because the two sources decompose the same event differently: Frollo bundles the Wise fee into the transfer (10001.13) where the statement itemises it (10000.00 + 1.13). So 38 Wise USD rows passed the amount guard as new while being the same money, and were the reason the transactions view showed a USD figure where every neighbouring row showed AUD. A statement's billing_end_date is a hard watermark: everything on that account up to that date is already in the ledger, itemised and converted. Guard 1 now drops any feed row at or before its account's newest statement. Guard 2 (amount + direction) stays as the net for accounts that have no statement at all. Note this is the coverage test the first import needed and got wrong. That one asked whether a row's date fell inside a statement's min-max window, which for periods spanning 182 to 460 days swallows a year and answers nothing. The watermark asks a question that has an answer: up to what date is this account complete? Matching is on last4, verified against the live statement set — the eight in-scope accounts with statements each map to one bank, no cross-bank collision. The watermarks are printed by the CLI so a wrong boundary is visible rather than inferred from a row count. Re-imported from empty: 425 covered by statement, 15 amount twins, 110 inserted. Exactly one row now carries a foreign currency with no AUD figure — the 2026-08-12 HDR salary, which is the genuinely-new pre-statement row this feed exists for. Was 39.	2026-08-13 12:36:08 +10:00
siddharthd	bb8a009e02	frollo: don't re-import what the ledger already has, and stop hiding the queue ... The first import wrote 550 rows on 2026-08-13. 422 of them (77%) were second copies of transactions the ledger already held from statements — $1,023,824.63 of movement counted twice. The owner found it by opening the transactions view and seeing one HDR Global salary listed twice, once as A$15,518.53 from the statement and once as US$10,782.00 from the feed. The currency was never the defect. toLedgerRow already left amount_aud NULL and named the currency in foreign_currency_code, which is the documented contract for a row whose AUD value is unknown, and the transactions page labels it. What made it look wrong was the duplicate sitting beside it. Two changes. Upstream, ingestFrolloCsv now drops rows the ledger already holds, matching on amount + direction within LEDGER_MATCH_DAYS (3). Three things had to be right and the first two were not, each caught only by rehearsing against real data rather than fixtures: - pg returns a DATE as a JS Date while Prisma and the CSV give strings. String(date).slice(0,10) is "Wed Mar 10", which parses to NaN, so the first dry run reported 550 to insert and zero duplicates. dayMs() takes both. - Direction has to be in the key. This ledger is full of internal transfers between the owner's own accounts and the feed carries both legs: 2026-05-18 has +3076.04 into ANZ and -3076.04 out of AMP. Matching on amount alone let the credit leg consume the ledger's debit row, so the real duplicate was written — 20 rows got in that way. - 'refund' is money in. The feed calls a reversed account fee a credit and the statement importer types it 'refund'; classifying it as an outflow left every ANZ servicing-fee reversal behind. Downstream, awaitsStatementLine() is removed. Its premise — feed rows never await a statement line — was asserted, never tested, and false for almost every account. Worse is how it got there: the reconcile queue jumped 8 -> 558 when the feed landed, that jump was read as noise and filtered away, and filtering it removed the only mechanism that would ever have collapsed the duplicates. The queue was right. A feed row IS a row awaiting its statement line. Re-imported: 375 dropped as already-on-ledger, 175 inserted. Residual duplicates 4 rows / $15.01, all sub-$5 account fees where several identical amounts fall in overlapping windows and greedy consumption picks the wrong one; not chased further at this scale. The CLI prints the already-on-ledger count even when zero — a number you have to go looking for is a number nobody looks at.	2026-08-13 12:23:01 +10:00
siddharthd	21e9e765a3	Add /api/frollo/ingest so the import can run unattended ... Shares one module with the CLI rather than reimplementing the insert: frollo-ingest.ts holds parsing, scoping, de-duplication and the write, and both callers pass in their own SQL executor (Prisma in the route, a pg client in the script). The alternative is two implementations of the same insert, which is how the pantry healthcheck came to be fixed in one repo and left broken in the other. The route refuses rather than guesses. findAnomalies() returns every reason an unattended run should stop - a configured account contributing no rows, an unrecognised account, a near-consecutive-id collapse that might be a real repeat, a batch over ~200 rows, or an export taken with pending included - and the route answers 409 having written nothing. Two defects the wiring surfaced. Deliberately excluded credit cards were reported as unknown accounts, which would have raised the new-account anomaly on every single run and left the automatic path permanently refusing; EXCLUDED_ACCOUNTS now distinguishes excluded from unknown. And pending was tested after account scope, so pending rows on cards - which is all of them so far - classified as out-of-scope and the wrong-export-option signal could never fire; pending is now tested first.	2026-08-13 11:19:27 +10:00
siddharthd	493ff6f631	Import Frollo account feeds for the accounts statements don't cover ... Credit cards keep arriving as monthly statements and stay the source of truth for them. This covers the other fourteen accounts, whose statements arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise including the income account, Up, ING, and the small transaction accounts. About 50 rows a month, where the alternative is downloading each statement by hand. The file needs three defences, all found by diffing three real exports (smarthome DECISIONS.md ING-11): A CDR re-consent makes Frollo re-ingest an account's whole history under fresh transaction ids while the originals survive, and consents expire annually. On this export 112 rows were such twins, and every HDR salary payment appeared twice — importing blind doubles reported income. dedupe() collapses each natural-key group to its lowest id, lowest because old ids were a strict subset of new across two exports, so source_ref stays stable and a re-import inserts nothing. An earlier version of that rule kept close-id rows on a 10,000 threshold, reasoning that genuine same-day repeats have consecutive ids. Verifying it against the income rows killed it: in-scope duplicate pairs have id gaps from 58 to 260 million, so no threshold separates them from the gaps of 1-4 that real repeats showed. It now collapses unconditionally and flags anything within 10 for review — the errors are asymmetric, and nothing in scope has ever tripped the flag. A lapsed consent removes an account from the export silently, with no error and no marker; the row count just drops. So the import asserts the account roster and refuses to run when a configured account contributes nothing. Also holds these rows out of the pending-reconciliation queue. A feed row is the account's own ledger entry, not a receipt awaiting a statement line — these accounts' statements are deliberately not imported — so without the exclusion 550 rows a year would bury the receipts that need a decision. The queue stays at 8 instead of 558. Foreign rows follow order-ingestion's existing shape: amount is the native figure, foreign_currency_code names it, amount_aud stays NULL rather than asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as incomplete. Dry run by default. Verified against the real export before applying: 550 rows inserted, 14 accounts, re-run inserts 0.	2026-08-13 10:49:35 +10:00