Shares one module with the CLI rather than reimplementing the insert:
frollo-ingest.ts holds parsing, scoping, de-duplication and the write, and
both callers pass in their own SQL executor (Prisma in the route, a pg
client in the script). The alternative is two implementations of the same
insert, which is how the pantry healthcheck came to be fixed in one repo
and left broken in the other.
The route refuses rather than guesses. findAnomalies() returns every reason
an unattended run should stop - a configured account contributing no rows,
an unrecognised account, a near-consecutive-id collapse that might be a
real repeat, a batch over ~200 rows, or an export taken with pending
included - and the route answers 409 having written nothing.
Two defects the wiring surfaced. Deliberately excluded credit cards were
reported as unknown accounts, which would have raised the new-account
anomaly on every single run and left the automatic path permanently
refusing; EXCLUDED_ACCOUNTS now distinguishes excluded from unknown. And
pending was tested after account scope, so pending rows on cards - which is
all of them so far - classified as out-of-scope and the wrong-export-option
signal could never fire; pending is now tested first.
Credit cards keep arriving as monthly statements and stay the source of
truth for them. This covers the other fourteen accounts, whose statements
arrive every 182 to 460 days — AMP including the loan, ANZ Access, Wise
including the income account, Up, ING, and the small transaction accounts.
About 50 rows a month, where the alternative is downloading each statement
by hand.
The file needs three defences, all found by diffing three real exports
(smarthome DECISIONS.md ING-11):
A CDR re-consent makes Frollo re-ingest an account's whole history under
fresh transaction ids while the originals survive, and consents expire
annually. On this export 112 rows were such twins, and every HDR salary
payment appeared twice — importing blind doubles reported income. dedupe()
collapses each natural-key group to its lowest id, lowest because old ids
were a strict subset of new across two exports, so source_ref stays stable
and a re-import inserts nothing.
An earlier version of that rule kept close-id rows on a 10,000 threshold,
reasoning that genuine same-day repeats have consecutive ids. Verifying it
against the income rows killed it: in-scope duplicate pairs have id gaps
from 58 to 260 million, so no threshold separates them from the gaps of 1-4
that real repeats showed. It now collapses unconditionally and flags
anything within 10 for review — the errors are asymmetric, and nothing in
scope has ever tripped the flag.
A lapsed consent removes an account from the export silently, with no error
and no marker; the row count just drops. So the import asserts the account
roster and refuses to run when a configured account contributes nothing.
Also holds these rows out of the pending-reconciliation queue. A feed row is
the account's own ledger entry, not a receipt awaiting a statement line —
these accounts' statements are deliberately not imported — so without the
exclusion 550 rows a year would bury the receipts that need a decision. The
queue stays at 8 instead of 558.
Foreign rows follow order-ingestion's existing shape: amount is the native
figure, foreign_currency_code names it, amount_aud stays NULL rather than
asserting a rate, and AMOUNT_UNCONVERTED already reports the balance as
incomplete.
Dry run by default. Verified against the real export before applying:
550 rows inserted, 14 accounts, re-run inserts 0.
siddharthd