Accept grocery receipts scanned in pantry as candidate spend
ci / lint-test (push) Successful in 48s
ci / lint-test (push) Successful in 48s
Adds /api/receipts/ingest as a sibling to the order lane, sharing its shape but making one decision differently: nothing is parked. An order can wait for its statement because it is already visible as an email; a gift-card grocery shop is visible nowhere at all, so a scan that produces no transaction produces nothing anyone can see. Every payment becomes a manual row immediately and the existing pending-reconciliation queue resolves the ones with a card leg coming. One transaction per tender leg. A $114.57 shop settled $40.75 gift card + $73.82 Mastercard has a statement line for $73.82 only. A single row marked credits is excluded from the queue while that line double-counts; marked card it is searched for at 1% of $114.57 and never matches. Either way the shop books $188.39. Per-leg rows make each amount the settled amount, so the matcher works untouched. Reconciliation now carries expense_metadata across. It already moved overrides, tags and splits from the manual row to the statement row and left metadata behind, which did not matter while metadata only came from an email that made its own transaction. It matters now that it carries a shop's line items: unmoved, the contents vanish at exactly the moment the statement line appears, and COLES 0556 MANOR LAKES stays as unreadable as before anything was scanned. transaction_id is UNIQUE, so a statement row that already has metadata keeps it and the pantry row is flagged rather than raising a constraint violation. Also regenerates the Prisma model. card_last4, currency, flags, reconciled_at, matched_transaction_id, platform and route have been in the database since migrations 0019/0020 and were absent from schema.prisma — regenerating the client from it would have dropped columns the order lane writes on every ingest. 23 integration tests against the real schema, built from the three receipts that drove the design. Existing suites unchanged: 104 unit, 144 integration.
This commit is contained in:
@@ -0,0 +1,215 @@
|
||||
import { queryRaw, queryRow } from "@/lib/db";
|
||||
import { DEFAULT_OWNER_ID } from "@/lib/order-ingestion";
|
||||
|
||||
/**
|
||||
* Grocery receipts scanned in pantry-app, arriving as candidate spend.
|
||||
*
|
||||
* The shape deliberately mirrors the order lane rather than inventing a second mechanism,
|
||||
* but it makes one decision differently and the difference is the point: **nothing is
|
||||
* parked.** An order can wait for its statement because it is already visible as an email;
|
||||
* a gift-card grocery shop is visible nowhere at all, so a scan that does not produce a
|
||||
* transaction produces nothing a person can see. Every leg becomes a manual transaction
|
||||
* (`statement_id IS NULL`) immediately, and finance's existing pending-reconciliation queue
|
||||
* resolves the ones that have a card leg coming.
|
||||
*
|
||||
* Why one transaction per tender leg rather than one per receipt: a shop settled $40.75 on
|
||||
* a gift card and $73.82 on a Mastercard has a statement line for $73.82 and nothing for
|
||||
* the rest. A single $114.57 row marked `credits` is excluded from the queue and lets the
|
||||
* statement line double-count; marked `card` it is searched for at ±1% of $114.57 and never
|
||||
* matches, so it parks forever while the statement line still counts. Either way the shop
|
||||
* books $188.39. Per-leg rows make each amount exactly the settled amount, so the existing
|
||||
* matcher works untouched.
|
||||
*/
|
||||
|
||||
export type TenderLeg = {
|
||||
leg_index: number;
|
||||
amount: number;
|
||||
card_last4?: string | null;
|
||||
card_product?: string | null;
|
||||
/** pantry's vocabulary; mapped to finance's payment_method below. */
|
||||
class?: "card" | "gift_card" | "cash" | null;
|
||||
};
|
||||
|
||||
export type ReceiptLineItem = {
|
||||
name: string;
|
||||
quantity?: number | null;
|
||||
unit?: string | null;
|
||||
line_total?: number | null;
|
||||
category?: string | null;
|
||||
};
|
||||
|
||||
export type ParsedReceipt = {
|
||||
receipt_uid?: string | null;
|
||||
capture_event_id: number;
|
||||
image_sha256?: string | null;
|
||||
merchant_name: string;
|
||||
store_detail?: string | null;
|
||||
transaction_date: string;
|
||||
total: number;
|
||||
tax_amount?: number | null;
|
||||
tender_raw?: string | null;
|
||||
loyalty_card_number?: string | null;
|
||||
tender_legs: TenderLeg[];
|
||||
line_items: ReceiptLineItem[];
|
||||
};
|
||||
|
||||
export type ReceiptIngestResult = {
|
||||
group: string;
|
||||
legs: { leg_index: number; transactionId: number; metadataId: number; paymentMethod: string | null; skipped?: string }[];
|
||||
flags: string[];
|
||||
};
|
||||
|
||||
export class ReceiptValidationError extends Error {}
|
||||
|
||||
/**
|
||||
* `credits` rather than a new `gift_card`: prepaid value with no card leg is a concept
|
||||
* finance already has, and reusing it inherits both the `needsCardMatch()` exclusion that
|
||||
* keeps such rows out of the reconciliation queue and the "Gift Card" label `bankLabel()`
|
||||
* renders for them. A second word for one idea would have needed both taught again.
|
||||
*
|
||||
* An unknown class maps to NULL, which `needsCardMatch()` treats as reconcilable — so a leg
|
||||
* nobody could classify lands in the queue visibly unresolved instead of being silently
|
||||
* decided either way.
|
||||
*/
|
||||
function paymentMethodFor(legClass: TenderLeg["class"]): string | null {
|
||||
if (legClass === "gift_card") return "credits";
|
||||
if (legClass === "cash") return "cash";
|
||||
if (legClass === "card") return "card";
|
||||
return null;
|
||||
}
|
||||
|
||||
const round2 = (value: number) => Number(value.toFixed(2));
|
||||
|
||||
/**
|
||||
* The whole shop is `groceries`. What it was actually made of — food versus household —
|
||||
* stays derived from `line_items` at display time rather than stored, because a mixed shop
|
||||
* is one payment and splitting the transaction to describe it would make the amount that
|
||||
* reconciles against the statement line stop matching it.
|
||||
*/
|
||||
const RECEIPT_CATEGORY = "groceries";
|
||||
|
||||
export function validateReceipt(receipt: ParsedReceipt): string[] {
|
||||
const flags: string[] = [];
|
||||
if (!receipt.tender_legs?.length) throw new ReceiptValidationError("at least one tender leg is required");
|
||||
if (!Number.isFinite(receipt.total) || receipt.total <= 0) throw new ReceiptValidationError("total must be a positive number");
|
||||
if (!/^\d{4}-\d{2}-\d{2}$/.test(receipt.transaction_date)) throw new ReceiptValidationError("transaction_date must be YYYY-MM-DD");
|
||||
|
||||
// The check that detects a split at all, and the one that proves the payment side was
|
||||
// read whole. A leg missed here becomes spend that never appears.
|
||||
const legSum = round2(receipt.tender_legs.reduce((total, leg) => total + Number(leg.amount || 0), 0));
|
||||
if (Math.abs(legSum - round2(receipt.total)) > 0.02) {
|
||||
throw new ReceiptValidationError(`tender legs sum to ${legSum.toFixed(2)} but the receipt total is ${receipt.total.toFixed(2)}`);
|
||||
}
|
||||
|
||||
// Lines are allowed to disagree: promotional rows are deliberately skipped during
|
||||
// extraction, so this flags rather than rejects. The money is the tender, not the lines.
|
||||
const lineSum = round2((receipt.line_items ?? []).reduce((total, line) => total + Number(line.line_total || 0), 0));
|
||||
if (receipt.line_items?.length && Math.abs(lineSum - round2(receipt.total)) > 0.02) flags.push(`line_items_sum_${lineSum.toFixed(2)}`);
|
||||
if (receipt.tender_legs.length > 1) flags.push("split_tender");
|
||||
if (receipt.tender_legs.some((leg) => !leg.class)) flags.push("unclassified_tender");
|
||||
return flags;
|
||||
}
|
||||
|
||||
/**
|
||||
* The receipt's own identity, not the capture's. A photo and the store's e-receipt PDF of
|
||||
* one purchase are different files with different hashes, so keying on the capture would
|
||||
* let the same shop arrive twice as two unrelated sets of transactions. Falls back to the
|
||||
* capture id when the receipt did not print enough to identify itself — that risks a
|
||||
* duplicate, which is visible and removable, rather than a merge, which silently hides a
|
||||
* real shop.
|
||||
*/
|
||||
export function receiptGroup(receipt: ParsedReceipt): string {
|
||||
return `pantry:${receipt.receipt_uid || `capture:${receipt.capture_event_id}`}`;
|
||||
}
|
||||
|
||||
export async function processReceiptIngestion(receipt: ParsedReceipt): Promise<ReceiptIngestResult> {
|
||||
const flags = validateReceipt(receipt);
|
||||
const group = receiptGroup(receipt);
|
||||
// Line items describe the whole shop but can only attach to one row —
|
||||
// expense_metadata.transaction_id is UNIQUE, and duplicating them would double any
|
||||
// composition derived from them. They go on the card leg because that is the row which
|
||||
// reconciles onto the statement line, which is where an unreadable `COLES 0556` descriptor
|
||||
// actually gets its contents. With no card leg, the largest leg carries them.
|
||||
const cardLeg = receipt.tender_legs.find((leg) => leg.class === "card" || !leg.class)
|
||||
?? [...receipt.tender_legs].sort((a, b) => Number(b.amount) - Number(a.amount))[0];
|
||||
|
||||
const legs: ReceiptIngestResult["legs"] = [];
|
||||
for (const leg of receipt.tender_legs) {
|
||||
const reference = `${group}#${leg.leg_index}`;
|
||||
const existing = await queryRow<{ id: number; transaction_id: number | null }>(
|
||||
`SELECT id, transaction_id FROM expense_metadata WHERE source = 'pantry' AND order_reference = $1`,
|
||||
[reference]
|
||||
);
|
||||
if (existing) {
|
||||
legs.push({ leg_index: leg.leg_index, transactionId: existing.transaction_id ?? 0, metadataId: existing.id, paymentMethod: null, skipped: "already_ingested" });
|
||||
continue;
|
||||
}
|
||||
|
||||
const paymentMethod = paymentMethodFor(leg.class);
|
||||
const amount = round2(Number(leg.amount));
|
||||
const description = receipt.store_detail ? `${receipt.merchant_name} ${receipt.store_detail}` : receipt.merchant_name;
|
||||
const txn = await queryRow<{ id: number }>(
|
||||
`INSERT INTO transactions (
|
||||
transaction_date, description, amount, amount_aud, category, payment_method,
|
||||
merchant_name, merchant_normalized, transaction_type, owner_id
|
||||
) VALUES ($1,$2,$3,$3,$4,$5,$6,$6,'debit',$7) RETURNING id`,
|
||||
[receipt.transaction_date, description, amount, RECEIPT_CATEGORY, paymentMethod, receipt.merchant_name, DEFAULT_OWNER_ID]
|
||||
);
|
||||
|
||||
const carriesLines = leg.leg_index === cardLeg?.leg_index;
|
||||
const meta = await queryRow<{ id: number }>(
|
||||
`INSERT INTO expense_metadata (
|
||||
transaction_id, source, order_reference, line_items, subtotal, amount,
|
||||
merchant_normalized, transaction_date, card_last4, currency, flags,
|
||||
reconciled_at, payment_method, payment_method_detail, tender_raw,
|
||||
receipt_sha256, receipt_group, extraction_model
|
||||
) VALUES ($1,'pantry',$2,$3::jsonb,$4,$5,$6,$7,$8,'AUD',$9::jsonb,NULL,$10,$11,$12,$13,$14,'cloud-budget')
|
||||
RETURNING id`,
|
||||
[
|
||||
txn!.id,
|
||||
reference,
|
||||
JSON.stringify(carriesLines ? receipt.line_items ?? [] : []),
|
||||
receipt.tax_amount ?? null,
|
||||
amount,
|
||||
receipt.merchant_name,
|
||||
receipt.transaction_date,
|
||||
leg.card_last4 ?? null,
|
||||
JSON.stringify(carriesLines ? flags : [...flags, "line_items_on_card_leg"]),
|
||||
paymentMethod,
|
||||
leg.card_product ?? null,
|
||||
receipt.tender_raw ?? null,
|
||||
receipt.image_sha256 ?? null,
|
||||
group,
|
||||
]
|
||||
);
|
||||
legs.push({ leg_index: leg.leg_index, transactionId: txn!.id, metadataId: meta!.id, paymentMethod });
|
||||
}
|
||||
return { group, legs, flags };
|
||||
}
|
||||
|
||||
/**
|
||||
* Statement lines that look like a pantry row already marked as needing no card leg.
|
||||
*
|
||||
* Such rows never enter the reconciliation queue — that is the whole point of classifying
|
||||
* them — so a mis-learned card would otherwise double-count in silence. This only flags: the
|
||||
* same uncertainty that makes the classification fallible makes the match a suggestion, and
|
||||
* auto-reconciling on it would trade a visible error for an invisible one. A confirmed
|
||||
* conflict is a reason to correct the card's stored class, which fixes every later receipt
|
||||
* from it at once.
|
||||
*/
|
||||
export async function pantryTenderConflicts(): Promise<{ transactionId: number; statementTransactionId: number; amount: string; merchant: string | null; date: string }[]> {
|
||||
return queryRaw(
|
||||
`SELECT p.id AS "transactionId", s.id AS "statementTransactionId", p.amount::text AS amount,
|
||||
p.merchant_normalized AS merchant, p.transaction_date::text AS date
|
||||
FROM transactions p
|
||||
JOIN expense_metadata em ON em.transaction_id = p.id AND em.source = 'pantry'
|
||||
JOIN transactions s ON s.statement_id IS NOT NULL
|
||||
AND s.transaction_date BETWEEN p.transaction_date - 3 AND p.transaction_date + 3
|
||||
AND s.amount BETWEEN p.amount * 0.99 AND p.amount * 1.01
|
||||
AND upper(coalesce(s.description, '')) LIKE '%' || upper(p.merchant_name) || '%'
|
||||
WHERE p.statement_id IS NULL
|
||||
AND p.reconciled_with_id IS NULL
|
||||
AND p.payment_method IN ('cash', 'credits')
|
||||
ORDER BY p.transaction_date DESC`
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user