Two parse bugs that between them made 218 of 776 captured messages
unreadable. Neither was the "old template" they were filed as.
Uber writes the currency three ways and only two were handled. "Total
A$54.87" is what it sends for ordinary Australian orders — A$ misses
[A-Z]{3} by one character — so 98 of 275 Uber Eats mails and 78 of 275
trip mails failed with "no Total found" while the amount sat in plain
sight. Most were 2024-2025, i.e. current mail. NZ$, US$, S$, HK$, C$ and
a bare rupee/euro/pound symbol are handled the same way. A bare "$" is
still left unresolved on purpose: a dozen currencies use it, and the
body-wide scan that reads the receipt's own stated code should win.
A DoorDash order paid from credits states "Total Charged $0.00"
truthfully, above a real subtotal. Read literally that is a $0 order, and
validateOrderTotals rejected 88 of them as non-positive — discarding
exactly the credit-funded spend this pipeline exists to surface. The
order's value is its subtotal; recording zero would show the order and
hide what it cost. Guarded on the receipt actually saying credits, so an
empty mail still fails rather than inheriting a stray subtotal, and the
header cross-check stands down for these or it rejects the figure the
parser deliberately overrode.
Measured A/B over all 776 captures: 254 parsed by both parsers with zero
change to any amount or currency, 0 lost, 214 newly readable. Both fixes
mutation-tested — reverting the regex fails 4 tests, removing the credits
branch fails 3.
Fixtures are real captured receipts, per the 2026-07-26 rewrite: the
earlier synthetic suite passed while the parser could not read a real
email.
No history replay — I7 idempotency refuses re-reads and that needs an
explicit update mode. This fixes ingest from here on.
Local rides are paid with credits (only overseas ones go on a card), so trips
belong to this slice and were simply never fetched — the Graph query searched
"order with Uber", the Eats subject. Captured 15 real messages from the mailbox
via a dry-run before touching anything, which found two defects that no amount
of reasoning about the template would have:
**Uber sends two mails per trip.** A "charge summary" when the ride ends, then
the real receipt when payment settles — same subject, same total. The summary
carries no tripReference, so order_reference fell back to `msg:<message-id>`
and I7 could not dedupe it against the receipt that follows. Every trip would
have been recorded twice. It says so itself ("This is not a payment receipt ...
You will receive a trip receipt when the payment is processed"), so it is now a
NotAReceiptError — 200 and silent, like every other expected non-receipt.
**Trip receipts label neither end of the journey.** Delivery receipts write
"1:20 pm - Pick-up"; trips print the time alone. The split regex put the time
into `label` and left `time` null. Time is now read properly, and a two-stop
trip is labelled Pick-up/Drop-off positionally — only where the receipt was
silent, so a template that does label its stops keeps its own wording.
Verified against all 15 captured messages: 7 trips recorded, 5 charge summaries
and 3 promotions skipped, 0 failures, no duplicate references. Two of the seven
are AUD credits-funded ($84.78 + $47.97) and would become transactions; the
five NZD ones are card-settled and correctly create provenance only (I5).
Fixtures ut-00 (local credits trip), ut-01 (overseas card trip) and ut-summary
(the charge summary) are captured mail, not written by hand.
Backfill dry-run over 130 real messages surfaced one 422: 'payments sum to 1.17
but receipt states 16.50'. The receipt is a mixed Uber payment —
Uber Cash $1.17 + Westpac ••••8032 $15.33 — and the card regex only matched
Visa|MasterCard|American Express|Amex, so an issuer-named leg was dropped
entirely. validateOrderTotals correctly refused it rather than recording $1.17
as the cost of a $16.50 order.
Anchors on the ••••NNNN masking instead, which also covers the form already
seen in the corpus ('Mastercard ••••3893 (CBA Ultimate) CHF 51.23'). Fixture
and regression test added.
Also repoints .env.test at the current postgres-personal container IP and
documents why: the container publishes no host port, so the address changes on
every recreate and the whole integration suite fails with connection errors
until it is refreshed.
Ingestion now runs on the rebuilt parser. Three substantive changes.
Deferred card reconciliation. A 'MasterCard 8032 and/or credits' receipt never
states the split, but the card leg lands on the statement — Subway's $29.08
order shows $13.06 on 8032, so $16.02 was credits. For a live order that
statement is weeks away, so the split cannot be settled at ingest time. Such
orders are now parked with provenance and no transaction, and
reconcilePendingOrders() resolves them once the statement arrives. Backfill
takes the same path and resolves immediately. Migration 0019 adds the columns
that make an order resumable; applied to personal_test only, prod untouched.
Payment detection bug, found by the new tests: the old regex delimited the
'Paid with' line on a double space, which whitespace collapsing removes. Every
card and mixed receipt fell through to the credits branch — the Woolworths
receipt booked $60.93 of credits spend that never happened.
Category resolution reversed deliberately. Correction 1 said never default to
dining; the implementation of that sent everything unrecognised to 'other', and
knowing six merchants meant Carl's Jr, Taco Bell, Chilli India, Oporto, Schnitz
and Souvlaki GR all landed there. Grocers are an enumerable set and restaurants
are not, so match groceries explicitly and let the residual be dining.
Tests rebuilt on real captured receipts; the synthetic fixtures are deleted.
60 unit + 41 integration green on three consecutive runs.
The previous parser was written against synthetic fixtures shaped to match the
code. It invented a table layout DoorDash does not send, generated
order_reference from Math.random(), read the order date from a 'Date:' string
present in no real message, and detected [Family] by searching the body for the
substring 'family'. Its tests passed because the fixtures were built to satisfy
it. Against 36 real DoorDash and 29 real Uber Eats receipts it does not work.
Rebuilt from the real corpus. 63 of 65 now parse and validate; the 2 rejected
are correctly rejected — one is an order-adjustment notice and one a refund,
neither of which is a receipt.
Corrects an inherited diagnosis: the Mad Mex 'Discounts -$24.09' was recorded
as an HTML-flattening artefact masking a 'true discount of $9.45'. Parsing the
table cells structurally returns the same figures and no $9.45 exists anywhere
in the message — DoorDash genuinely prints a Discounts line that equals subtotal
plus service fee, and the components fail to reconcile on 32 of 36 receipts. So
the breakdown is stored as provenance and never gated on; validation instead
cross-checks the two independently stated totals and the payment line, which is
the number that becomes money.
Real-world cases the corpus forced, none of which were in the spec: [Family]
orders are LKR purchases for family in Sri Lanka (reading them as dollars
inflates ~200x), Swiss orders arrive in CHF, grocery 'Final receipt' mails carry
no Total Charged row, and a declined payment is printed alongside the successful
retry and must be skipped or it records money that never moved.
order_reference now comes from the Uber order UUID embedded in the body, or the
provider message id where DoorDash supplies no order id at all — never random,
so re-ingestion is genuinely idempotent.
siddharthd