`•` was missing from the entity table, and that was not cosmetic.
DoorDash separates an item's name from its options with a bullet and
parseDoorDashLineItems splits on the literal "•" — so left encoded, the
split never happened and the line collapsed into the description:
"Bucket and Side Pack (Meal Deals) • Hot Bucket • Chips" with
options []. The entity showed on screen and the structure behind it was
gone.
Numeric entities are now decoded generically rather than one at a time,
which is how $ came to be listed individually while its neighbours
were not, and & resolves last so a literal "•" stays as
written instead of turning into a bullet.
Repaired the 47 stored rows by re-parsing the captured email behind each
one rather than string-replacing the entity, since a replacement would
have fixed the display and left options [] underneath. Rehearsed first:
all 47 re-parsed, all 47 gained options, 0 line items lost, 0 amounts
changed. Old values kept in dump/rollback-line-items-20260728-223737.json.
Pre-existing — 22 rows predate today — but the pre-cutover backfill more
than doubled the affected rows, which is what surfaced it. Verified in
the Order details panel, not in SQL.
Uber Cash IS credits, and the payment line had become unreadable in the
newer layout: "Payments Uber Cash 10/17/25 8:50 PM A$54.87" carries a
timestamp between the label and the amount, and writes the currency as a
prefix. Both defeated the pattern, so credits_amount stayed null and the
order was filed as card-settled — sent looking for a card leg that does
not exist, found nothing, and left as an orphan with no transaction and
no card tail to match on. 118 captured messages sit in that state, and
every one of them is pre-cutover, so reading them correctly means I1
skips them rather than storing enrichment that points at nothing.
The card leg had the same blind spot, hidden behind the first: the gap
between the mask and the amount was [^\d]{0,40}, which a timestamp
breaks. While BOTH legs were unreadable a mixed payment still looked
consistent — the order read as card-settled for the full total. Fixing
only credits reads half an order, which validateOrderTotals correctly
refuses. Found exactly that way: four messages that validated before
began failing "payments sum to 34.92 but receipt states 43.60".
Legs are summed rather than taken first, because one order can be charged
in instalments and an instrument can carry no mask at all (PayPal). But a
leg that already equals the stated total IS the payment, not an
instalment: a Dubai trip prints an AED 17.67 authorisation and then the
AED 577.83 settled charge, and adding the hold overstates the trip. A
mixed credits+card order is unaffected — neither leg equals the total
there, which is why it needs summing.
A/B over all 776 captures: 468 parsed by both, zero change to any amount,
currency or existing card tail, 0 lost, 118 credits figures newly read.
Mutation-tested: dropping the exact-leg rule fails 2 tests, dropping the
Uber Cash read fails 7.
Two parse bugs that between them made 218 of 776 captured messages
unreadable. Neither was the "old template" they were filed as.
Uber writes the currency three ways and only two were handled. "Total
A$54.87" is what it sends for ordinary Australian orders — A$ misses
[A-Z]{3} by one character — so 98 of 275 Uber Eats mails and 78 of 275
trip mails failed with "no Total found" while the amount sat in plain
sight. Most were 2024-2025, i.e. current mail. NZ$, US$, S$, HK$, C$ and
a bare rupee/euro/pound symbol are handled the same way. A bare "$" is
still left unresolved on purpose: a dozen currencies use it, and the
body-wide scan that reads the receipt's own stated code should win.
A DoorDash order paid from credits states "Total Charged $0.00"
truthfully, above a real subtotal. Read literally that is a $0 order, and
validateOrderTotals rejected 88 of them as non-positive — discarding
exactly the credit-funded spend this pipeline exists to surface. The
order's value is its subtotal; recording zero would show the order and
hide what it cost. Guarded on the receipt actually saying credits, so an
empty mail still fails rather than inheriting a stray subtotal, and the
header cross-check stands down for these or it rejects the figure the
parser deliberately overrode.
Measured A/B over all 776 captures: 254 parsed by both parsers with zero
change to any amount or currency, 0 lost, 214 newly readable. Both fixes
mutation-tested — reverting the regex fails 4 tests, removing the credits
branch fails 3.
Fixtures are real captured receipts, per the 2026-07-26 rewrite: the
earlier synthetic suite passed while the parser could not read a real
email.
No history replay — I7 idempotency refuses re-reads and that needs an
explicit update mode. This fixes ingest from here on.
Local rides are paid with credits (only overseas ones go on a card), so trips
belong to this slice and were simply never fetched — the Graph query searched
"order with Uber", the Eats subject. Captured 15 real messages from the mailbox
via a dry-run before touching anything, which found two defects that no amount
of reasoning about the template would have:
**Uber sends two mails per trip.** A "charge summary" when the ride ends, then
the real receipt when payment settles — same subject, same total. The summary
carries no tripReference, so order_reference fell back to `msg:<message-id>`
and I7 could not dedupe it against the receipt that follows. Every trip would
have been recorded twice. It says so itself ("This is not a payment receipt ...
You will receive a trip receipt when the payment is processed"), so it is now a
NotAReceiptError — 200 and silent, like every other expected non-receipt.
**Trip receipts label neither end of the journey.** Delivery receipts write
"1:20 pm - Pick-up"; trips print the time alone. The split regex put the time
into `label` and left `time` null. Time is now read properly, and a two-stop
trip is labelled Pick-up/Drop-off positionally — only where the receipt was
silent, so a template that does label its stops keeps its own wording.
Verified against all 15 captured messages: 7 trips recorded, 5 charge summaries
and 3 promotions skipped, 0 failures, no duplicate references. Two of the seven
are AUD credits-funded ($84.78 + $47.97) and would become transactions; the
five NZD ones are card-settled and correctly create provenance only (I5).
Fixtures ut-00 (local credits trip), ut-01 (overseas card trip) and ut-summary
(the charge summary) are captured mail, not written by hand.
Backfill dry-run over 130 real messages surfaced one 422: 'payments sum to 1.17
but receipt states 16.50'. The receipt is a mixed Uber payment —
Uber Cash $1.17 + Westpac ••••8032 $15.33 — and the card regex only matched
Visa|MasterCard|American Express|Amex, so an issuer-named leg was dropped
entirely. validateOrderTotals correctly refused it rather than recording $1.17
as the cost of a $16.50 order.
Anchors on the ••••NNNN masking instead, which also covers the form already
seen in the corpus ('Mastercard ••••3893 (CBA Ultimate) CHF 51.23'). Fixture
and regression test added.
Also repoints .env.test at the current postgres-personal container IP and
documents why: the container publishes no host port, so the address changes on
every recreate and the whole integration suite fails with connection errors
until it is refreshed.
Ingestion now runs on the rebuilt parser. Three substantive changes.
Deferred card reconciliation. A 'MasterCard 8032 and/or credits' receipt never
states the split, but the card leg lands on the statement — Subway's $29.08
order shows $13.06 on 8032, so $16.02 was credits. For a live order that
statement is weeks away, so the split cannot be settled at ingest time. Such
orders are now parked with provenance and no transaction, and
reconcilePendingOrders() resolves them once the statement arrives. Backfill
takes the same path and resolves immediately. Migration 0019 adds the columns
that make an order resumable; applied to personal_test only, prod untouched.
Payment detection bug, found by the new tests: the old regex delimited the
'Paid with' line on a double space, which whitespace collapsing removes. Every
card and mixed receipt fell through to the credits branch — the Woolworths
receipt booked $60.93 of credits spend that never happened.
Category resolution reversed deliberately. Correction 1 said never default to
dining; the implementation of that sent everything unrecognised to 'other', and
knowing six merchants meant Carl's Jr, Taco Bell, Chilli India, Oporto, Schnitz
and Souvlaki GR all landed there. Grocers are an enumerable set and restaurants
are not, so match groceries explicitly and let the residual be dining.
Tests rebuilt on real captured receipts; the synthetic fixtures are deleted.
60 unit + 41 integration green on three consecutive runs.
The previous parser was written against synthetic fixtures shaped to match the
code. It invented a table layout DoorDash does not send, generated
order_reference from Math.random(), read the order date from a 'Date:' string
present in no real message, and detected [Family] by searching the body for the
substring 'family'. Its tests passed because the fixtures were built to satisfy
it. Against 36 real DoorDash and 29 real Uber Eats receipts it does not work.
Rebuilt from the real corpus. 63 of 65 now parse and validate; the 2 rejected
are correctly rejected — one is an order-adjustment notice and one a refund,
neither of which is a receipt.
Corrects an inherited diagnosis: the Mad Mex 'Discounts -$24.09' was recorded
as an HTML-flattening artefact masking a 'true discount of $9.45'. Parsing the
table cells structurally returns the same figures and no $9.45 exists anywhere
in the message — DoorDash genuinely prints a Discounts line that equals subtotal
plus service fee, and the components fail to reconcile on 32 of 36 receipts. So
the breakdown is stored as provenance and never gated on; validation instead
cross-checks the two independently stated totals and the payment line, which is
the number that becomes money.
Real-world cases the corpus forced, none of which were in the spec: [Family]
orders are LKR purchases for family in Sri Lanka (reading them as dollars
inflates ~200x), Swiss orders arrive in CHF, grocery 'Final receipt' mails carry
no Total Charged row, and a declined payment is printed alongside the successful
retry and must be skipped or it records money that never moved.
order_reference now comes from the Uber order UUID embedded in the body, or the
provider message id where DoorDash supplies no order id at all — never random,
so re-ingestion is genuinely idempotent.
siddharthd