reconcile: automatic pass for unambiguous statement matches (board #252)
ci / lint-test (push) Successful in 56s
ci / lint-test (push) Successful in 56s
Prerequisite for the wallet lane writing a transaction per tap. The queue is worked by hand and already ~177 deep; wallet adds ~63 rows a month, ~767 a year. Filtering those rows back out is explicitly ruled out by the awaitsStatementLine() removal note: "an imported row that has no statement belongs in the queue; volume is solved upstream, never downstream by hiding what is there." The rule is deliberately stricter than the queue's own candidate ranking (top 5 within +/-1% and +/-3 days, for a human to choose between). A machine gets one shot and no judgement, so: exact amount, never a tolerance band; and unique BOTH ways, so two identical $13.50 bakery taps in one window are refused rather than paired arbitrarily. Foreign rows match on the native pair, because `amount` is the bank's own conversion and a wallet row cannot predict it. VALIDATED AGAINST REAL DECISIONS, and that overturned my reasoning. I first required both sides to be non-foreign, arguing a currency mismatch is ambiguous. Checked against the 48 reconciliations a human has actually made: that version matched ZERO of them. All 48 are exactly the shape it refused — a manual row carrying no currency, matched on exact AUD against a statement row that does carry one (EUR, CHF) — because overseas receipts and orders record what was charged in AUD, and only the wallet lane records natively. Reverted; the rule now reproduces all 48 and proposes nothing they did not. The coincidence it was meant to guard against is already covered by unique-both-ways. Extracted the merge into src/lib/reconcile.ts so the manual route and the automatic pass cannot drift — there must be exactly one implementation of something that moves curation off a row reconciliation is about to hide. Fixed a latent bug in that merge while extracting it: splits were recreated WITHOUT `settled`/`settled_at`, and the column defaults to false, so reconciling a settled split silently marked a paid debt owed again. No queue row carries one today, but this path is about to run unattended, where that surfaces as money rather than as an exception. PATCH ?dry=1 reports without writing. Each pair is its own transaction, so one bad pair leaves the rest untouched and in the queue. 11 new integration tests, most of them about refusal. Live dry-run proposes 0 pairs for either owner, which is correct: nothing in the queue has an exact statement twin yet. npm run test:all green (182 + 287).
This commit is contained in:
@@ -0,0 +1,218 @@
|
|||||||
|
import { describe, it, expect, beforeAll, beforeEach, vi } from "vitest";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { createPool, resetDB, seedParticipants, insertTransaction } from "./helpers";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The automatic pass over the reconcile queue.
|
||||||
|
*
|
||||||
|
* This exists because the wallet lane writes a transaction for every tap
|
||||||
|
* (DECISIONS.md ING-11), adding ~63 rows a month to a queue already ~177 deep
|
||||||
|
* and worked by hand. Hiding those rows instead is explicitly ruled out --
|
||||||
|
* see the awaitsStatementLine() removal note in queries.ts.
|
||||||
|
*
|
||||||
|
* The cases that must NOT match matter more than the ones that must: this runs
|
||||||
|
* unattended, and a wrong pairing merges one purchase's curation onto another
|
||||||
|
* purchase's row. So most of what follows is about refusal.
|
||||||
|
*/
|
||||||
|
|
||||||
|
let pool: Pool;
|
||||||
|
let ownerId: number;
|
||||||
|
let autoReconcile: typeof import("../../app/api/transactions/reconcile/route").PATCH;
|
||||||
|
|
||||||
|
const userReq = (url = "http://x/api/transactions/reconcile") => ({ url }) as never;
|
||||||
|
|
||||||
|
async function statementLine(
|
||||||
|
amount: number,
|
||||||
|
date: string,
|
||||||
|
description: string,
|
||||||
|
opts: { owner?: number; type?: string; fxAmount?: number; fxCode?: string } = {}
|
||||||
|
) {
|
||||||
|
const owner = opts.owner ?? ownerId;
|
||||||
|
const st = await pool.query(
|
||||||
|
`INSERT INTO statements (filename, account_number, bank_name, billing_start_date, billing_end_date, owner_id)
|
||||||
|
VALUES ('s.pdf', '1234-5678-9012-3302', 'Test Bank', $1::date - 20, $1::date + 10, $2) RETURNING id`,
|
||||||
|
[date, owner]
|
||||||
|
);
|
||||||
|
const t = await pool.query(
|
||||||
|
`INSERT INTO transactions
|
||||||
|
(statement_id, transaction_date, description, amount, transaction_type, owner_id,
|
||||||
|
foreign_currency_amount, foreign_currency_code)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id`,
|
||||||
|
[st.rows[0].id, date, description, amount, opts.type ?? "debit", owner,
|
||||||
|
opts.fxAmount ?? null, opts.fxCode ?? null]
|
||||||
|
);
|
||||||
|
return t.rows[0].id as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function walletRow(
|
||||||
|
amount: number,
|
||||||
|
date: string,
|
||||||
|
opts: { fxAmount?: number; fxCode?: string; paymentMethod?: string } = {}
|
||||||
|
) {
|
||||||
|
const id = await insertTransaction(pool, ownerId, {
|
||||||
|
amount, transaction_date: date, description: "Wallet tap",
|
||||||
|
});
|
||||||
|
await pool.query(
|
||||||
|
`UPDATE transactions
|
||||||
|
SET foreign_currency_amount = $2, foreign_currency_code = $3, payment_method = $4
|
||||||
|
WHERE id = $1`,
|
||||||
|
[id, opts.fxAmount ?? null, opts.fxCode ?? null, opts.paymentMethod ?? null]
|
||||||
|
);
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
const reconciledWith = async (id: number) =>
|
||||||
|
(await pool.query(`SELECT reconciled_with_id FROM transactions WHERE id = $1`, [id]))
|
||||||
|
.rows[0].reconciled_with_id as number | null;
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
pool = createPool();
|
||||||
|
vi.doMock("@/lib/auth", () => ({
|
||||||
|
getCurrentUser: async () => ({ id: ownerId, name: "Alice", email: "alice@example.com" }),
|
||||||
|
}));
|
||||||
|
({ PATCH: autoReconcile } = await import("../../app/api/transactions/reconcile/route"));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await resetDB(pool);
|
||||||
|
await pool.query("DELETE FROM expense_metadata");
|
||||||
|
({ ownerId } = await seedParticipants(pool));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("pairs it should apply", () => {
|
||||||
|
it("matches an exact amount two days apart and moves the link", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
const stmt = await statementLine(13.5, "2026-08-15", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq())).json();
|
||||||
|
expect(body.reconciled).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("matches a foreign row on the NATIVE pair, not on the AUD amount", async () => {
|
||||||
|
// The bank converts at its own rate, so the wallet row cannot predict the
|
||||||
|
// AUD figure. It can only know THB 500. Deliberately give the statement a
|
||||||
|
// wildly different AUD amount to prove `amount` is not what matched.
|
||||||
|
const manual = await walletRow(999.99, "2026-08-13", { fxAmount: 500, fxCode: "THB" });
|
||||||
|
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
|
||||||
|
fxAmount: 500, fxCode: "THB",
|
||||||
|
});
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq())).json();
|
||||||
|
expect(body.reconciled).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("carries a SETTLED split across without resurrecting the debt", async () => {
|
||||||
|
const manual = await walletRow(50, "2026-08-13");
|
||||||
|
const stmt = await statementLine(50, "2026-08-14", "SHARED DINNER");
|
||||||
|
await pool.query(
|
||||||
|
`INSERT INTO transaction_splits (transaction_id, participant_id, share_percent, settled, settled_at)
|
||||||
|
VALUES ($1, $2, 50, true, now())`,
|
||||||
|
[manual, ownerId]
|
||||||
|
);
|
||||||
|
|
||||||
|
expect((await autoReconcile(userReq())).status ?? 200).toBe(200);
|
||||||
|
|
||||||
|
const moved = await pool.query(
|
||||||
|
`SELECT settled, settled_at FROM transaction_splits WHERE transaction_id = $1`, [stmt]
|
||||||
|
);
|
||||||
|
expect(moved.rowCount).toBe(1);
|
||||||
|
// The whole point: a delete-and-recreate defaults `settled` to false, which
|
||||||
|
// silently marks a paid debt owed again. Nothing errors when it regresses.
|
||||||
|
expect(moved.rows[0].settled).toBe(true);
|
||||||
|
expect(moved.rows[0].settled_at).not.toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("pairs it must refuse", () => {
|
||||||
|
it("refuses two identical amounts in the same window (ambiguous both ways)", async () => {
|
||||||
|
// Two $13.50 bakery taps in one week is the ordinary case, not an edge one.
|
||||||
|
const a = await walletRow(13.5, "2026-08-13");
|
||||||
|
const b = await walletRow(13.5, "2026-08-14");
|
||||||
|
await statementLine(13.5, "2026-08-14", "BREADWORLD ONE");
|
||||||
|
await statementLine(13.5, "2026-08-15", "BREADWORLD TWO");
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq())).json();
|
||||||
|
expect(body.reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(a)).toBeNull();
|
||||||
|
expect(await reconciledWith(b)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a near-miss amount — a person should look at that", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
await statementLine(13.55, "2026-08-14", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a match outside the day window", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
await statementLine(13.5, "2026-08-25", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a statement row already claimed by another manual row", async () => {
|
||||||
|
const claimed = await statementLine(40, "2026-08-14", "ALREADY TAKEN");
|
||||||
|
const first = await walletRow(40, "2026-08-13");
|
||||||
|
await pool.query(`UPDATE transactions SET reconciled_with_id = $2 WHERE id = $1`, [first, claimed]);
|
||||||
|
const second = await walletRow(40, "2026-08-14");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(second)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a credits-funded row — no card leg is ever coming", async () => {
|
||||||
|
const manual = await walletRow(30, "2026-08-13", { paymentMethod: "credits" });
|
||||||
|
await statementLine(30, "2026-08-14", "COINCIDENTAL AMOUNT");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("an AUD row against a foreign statement line", () => {
|
||||||
|
it("DOES match on the AUD figure — this is what every real pair looks like", async () => {
|
||||||
|
// Checked against live data: all 48 reconciliations a human has made are
|
||||||
|
// this exact shape -- manual row with no currency, statement row carrying
|
||||||
|
// EUR or CHF, paired on exact AUD. Overseas receipts and orders record what
|
||||||
|
// was charged in AUD; only the wallet lane records natively. An earlier
|
||||||
|
// version of this rule required both sides to be non-foreign, on the
|
||||||
|
// reasoning that a currency mismatch is ambiguous. It matched 0 of the 48.
|
||||||
|
const manual = await walletRow(22.85, "2026-08-13");
|
||||||
|
const stmt = await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", {
|
||||||
|
fxAmount: 500, fxCode: "THB",
|
||||||
|
});
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBe(stmt);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("still refuses when a second candidate makes it ambiguous", async () => {
|
||||||
|
// The coincidence this guards against -- a domestic tap whose amount equals
|
||||||
|
// a foreign charge's AUD figure -- is caught by unique-both-ways, not by a
|
||||||
|
// currency test.
|
||||||
|
const manual = await walletRow(22.85, "2026-08-13");
|
||||||
|
await statementLine(22.85, "2026-08-14", "7-ELEVEN BANGKOK", { fxAmount: 500, fxCode: "THB" });
|
||||||
|
await statementLine(22.85, "2026-08-13", "MELBOURNE CAFE");
|
||||||
|
|
||||||
|
expect((await (await autoReconcile(userReq())).json()).reconciled).toBe(0);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("dry run", () => {
|
||||||
|
it("reports what it would do and writes nothing", async () => {
|
||||||
|
const manual = await walletRow(13.5, "2026-08-13");
|
||||||
|
await statementLine(13.5, "2026-08-14", "BREADWORLD BAKERY");
|
||||||
|
|
||||||
|
const body = await (await autoReconcile(userReq("http://x/api/transactions/reconcile?dry=1"))).json();
|
||||||
|
expect(body.dry_run).toBe(true);
|
||||||
|
expect(body.would_reconcile).toBe(1);
|
||||||
|
expect(await reconciledWith(manual)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1,6 +1,8 @@
|
|||||||
import { NextRequest, NextResponse } from "next/server";
|
import { NextRequest, NextResponse } from "next/server";
|
||||||
import { getCurrentUser } from "@/lib/auth";
|
import { getCurrentUser } from "@/lib/auth";
|
||||||
import { prisma, queryRaw } from "@/lib/db";
|
import { queryRaw } from "@/lib/db";
|
||||||
|
import { applyReconcileMatch } from "@/lib/reconcile";
|
||||||
|
import { findAutoMatches } from "@/lib/auto-reconcile";
|
||||||
|
|
||||||
export async function POST(req: NextRequest) {
|
export async function POST(req: NextRequest) {
|
||||||
const user = await getCurrentUser(req);
|
const user = await getCurrentUser(req);
|
||||||
@@ -25,97 +27,53 @@ export async function POST(req: NextRequest) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let reconciled = 0;
|
let reconciled = 0;
|
||||||
|
|
||||||
for (const { manual_id, statement_tx_id } of body.matches) {
|
for (const { manual_id, statement_tx_id } of body.matches) {
|
||||||
await prisma.$transaction(async (tx) => {
|
await applyReconcileMatch(manual_id, statement_tx_id);
|
||||||
// Copy overrides: manual → statement tx
|
|
||||||
const override = await tx.transaction_overrides.findUnique({
|
|
||||||
where: { transaction_id: manual_id },
|
|
||||||
});
|
|
||||||
if (override) {
|
|
||||||
await tx.transaction_overrides.upsert({
|
|
||||||
where: { transaction_id: statement_tx_id },
|
|
||||||
update: {
|
|
||||||
category_override: override.category_override,
|
|
||||||
merchant_normalized: override.merchant_normalized,
|
|
||||||
notes: override.notes,
|
|
||||||
my_share_percent: override.my_share_percent,
|
|
||||||
updated_at: new Date(),
|
|
||||||
},
|
|
||||||
create: {
|
|
||||||
transaction_id: statement_tx_id,
|
|
||||||
category_override: override.category_override,
|
|
||||||
merchant_normalized: override.merchant_normalized,
|
|
||||||
notes: override.notes,
|
|
||||||
my_share_percent: override.my_share_percent,
|
|
||||||
},
|
|
||||||
});
|
|
||||||
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Move tags: manual → statement tx
|
|
||||||
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
|
|
||||||
if (tags.length) {
|
|
||||||
await tx.transaction_tags.createMany({
|
|
||||||
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
|
|
||||||
skipDuplicates: true,
|
|
||||||
});
|
|
||||||
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Move splits: manual → statement tx
|
|
||||||
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
|
|
||||||
if (splits.length) {
|
|
||||||
await tx.transaction_splits.createMany({
|
|
||||||
data: splits.map((s) => ({
|
|
||||||
transaction_id: statement_tx_id,
|
|
||||||
participant_id: s.participant_id,
|
|
||||||
share_percent: s.share_percent,
|
|
||||||
})),
|
|
||||||
skipDuplicates: true,
|
|
||||||
});
|
|
||||||
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Move provenance: manual → statement tx.
|
|
||||||
//
|
|
||||||
// Overrides, tags and splits above were always carried across; expense_metadata was
|
|
||||||
// the one child left behind, which did not matter while every metadata row came from
|
|
||||||
// an email that had created its own transaction. It matters now: a scanned grocery
|
|
||||||
// receipt puts its line items here, and reconciliation hides the manual row from
|
|
||||||
// every figure — so without this the shop's contents disappear at exactly the moment
|
|
||||||
// the statement line appears, and `COLES 0556 MANOR LAKES` stays as unreadable as it
|
|
||||||
// was before the receipt was ever scanned.
|
|
||||||
//
|
|
||||||
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
|
|
||||||
// or Paperless copy got there first) keeps it. The pantry row stays attached to the
|
|
||||||
// reconciled manual transaction and is flagged, rather than raising a constraint
|
|
||||||
// violation or silently overwriting the other source.
|
|
||||||
const moved = await tx.$executeRawUnsafe(
|
|
||||||
`UPDATE expense_metadata SET transaction_id = $1
|
|
||||||
WHERE transaction_id = $2
|
|
||||||
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
|
|
||||||
statement_tx_id,
|
|
||||||
manual_id
|
|
||||||
);
|
|
||||||
if (moved === 0) {
|
|
||||||
await tx.$executeRawUnsafe(
|
|
||||||
`UPDATE expense_metadata
|
|
||||||
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
|
|
||||||
WHERE transaction_id = $1`,
|
|
||||||
manual_id
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Mark manual tx as reconciled (link to statement tx)
|
|
||||||
await tx.$executeRawUnsafe(
|
|
||||||
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
|
|
||||||
statement_tx_id,
|
|
||||||
manual_id
|
|
||||||
);
|
|
||||||
});
|
|
||||||
reconciled++;
|
reconciled++;
|
||||||
}
|
}
|
||||||
|
|
||||||
return NextResponse.json({ reconciled });
|
return NextResponse.json({ reconciled });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Automatic pass over the reconcile queue.
|
||||||
|
*
|
||||||
|
* Applies only pairs that are unambiguous by the strict rule in
|
||||||
|
* `findAutoMatches` -- exact amount (or exact native pair for a foreign row),
|
||||||
|
* within a few days, and unique in BOTH directions. Everything else is left in
|
||||||
|
* the queue for a person, which is the point: this exists to keep the queue
|
||||||
|
* workable as the wallet lane adds ~63 rows a month, not to clear it.
|
||||||
|
*
|
||||||
|
* Validated against the 48 reconciliations a human had already made: the rule
|
||||||
|
* reproduces all 48 and proposes nothing they did not.
|
||||||
|
*
|
||||||
|
* `?dry=1` returns what it WOULD do without writing, which is how this should
|
||||||
|
* be run the first time after any change to the matching rule.
|
||||||
|
*/
|
||||||
|
export async function PATCH(req: NextRequest) {
|
||||||
|
const user = await getCurrentUser(req);
|
||||||
|
if (!user) return NextResponse.json({ error: "Unauthorized" }, { status: 403 });
|
||||||
|
|
||||||
|
const dry = new URL(req.url).searchParams.get("dry") === "1";
|
||||||
|
const matches = await findAutoMatches(user.id);
|
||||||
|
|
||||||
|
if (dry) {
|
||||||
|
return NextResponse.json({ dry_run: true, would_reconcile: matches.length, matches });
|
||||||
|
}
|
||||||
|
|
||||||
|
let reconciled = 0;
|
||||||
|
const failed: { manual_id: number; error: string }[] = [];
|
||||||
|
for (const m of matches) {
|
||||||
|
try {
|
||||||
|
await applyReconcileMatch(m.manual_id, m.statement_tx_id);
|
||||||
|
reconciled++;
|
||||||
|
} catch (e) {
|
||||||
|
// One bad pair must not abandon the rest -- each merge is its own
|
||||||
|
// transaction, so a failure here leaves that pair untouched and in the
|
||||||
|
// queue rather than half-applied.
|
||||||
|
failed.push({ manual_id: m.manual_id, error: e instanceof Error ? e.message : String(e) });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return NextResponse.json({ reconciled, examined: matches.length, failed });
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,117 @@
|
|||||||
|
import { queryRaw } from "@/lib/db";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Unambiguous statement matches for statementless ("manual") transactions.
|
||||||
|
*
|
||||||
|
* WHY THIS EXISTS
|
||||||
|
* ---------------
|
||||||
|
* The reconcile queue is worked by hand and is already ~177 deep. The wallet
|
||||||
|
* lane writes a transaction for every tap (DECISIONS.md ING-11, 2026-08-22),
|
||||||
|
* which at the observed rate adds ~63 rows a month, ~767 a year. Without an
|
||||||
|
* automatic pass for the easy cases that queue stops being workable, and the
|
||||||
|
* tempting alternative -- filtering the new rows back out of it -- is
|
||||||
|
* explicitly ruled out by the `awaitsStatementLine()` removal note in
|
||||||
|
* queries.ts: "an imported row that has no statement belongs in the queue;
|
||||||
|
* volume is solved upstream, never downstream by hiding what is there."
|
||||||
|
*
|
||||||
|
* WHAT COUNTS AS UNAMBIGUOUS
|
||||||
|
* --------------------------
|
||||||
|
* Deliberately much stricter than the queue's own candidate ranking, which
|
||||||
|
* offers the top 5 within +/-1% and +/-3 days for a human to choose between.
|
||||||
|
* A machine gets one shot and no judgement, so:
|
||||||
|
*
|
||||||
|
* - EXACT amount equality, not the queue's +/-1% band. A near-miss is exactly
|
||||||
|
* the case a person should look at.
|
||||||
|
* - Same owner, and the statement row must not already be reconciled against.
|
||||||
|
* - UNIQUE BOTH WAYS: the manual row must have exactly one candidate, AND
|
||||||
|
* that candidate must be claimed by exactly one manual row. This is the
|
||||||
|
* order_transaction_bridge rule, and it is what stops two identical $13.50
|
||||||
|
* bakery taps in one window from being paired arbitrarily.
|
||||||
|
*
|
||||||
|
* Foreign rows match on the NATIVE pair (foreign_currency_amount +
|
||||||
|
* foreign_currency_code), never on `amount`: `amount` is the AUD the bank
|
||||||
|
* charged after applying its own rate, so a wallet row cannot predict it, while
|
||||||
|
* the native figure is exact on both sides. Same reasoning as
|
||||||
|
* order_transaction_bridge's FX path.
|
||||||
|
*
|
||||||
|
* This function only PROPOSES. Applying a pair goes through the same merge the
|
||||||
|
* manual route uses, so overrides/tags/splits/metadata move identically.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type AutoMatch = {
|
||||||
|
manual_id: number;
|
||||||
|
statement_tx_id: number;
|
||||||
|
amount: string;
|
||||||
|
manual_date: string;
|
||||||
|
statement_date: string;
|
||||||
|
description: string;
|
||||||
|
basis: "aud" | "native";
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Days either side of the manual row's date to consider. Authorisation and
|
||||||
|
* settlement are rarely the same day; 3 matches the queue's own window and
|
||||||
|
* the wallet reconciliation check's default. */
|
||||||
|
export const AUTO_MATCH_WINDOW_DAYS = 3;
|
||||||
|
|
||||||
|
export async function findAutoMatches(
|
||||||
|
ownerId: number,
|
||||||
|
windowDays: number = AUTO_MATCH_WINDOW_DAYS
|
||||||
|
): Promise<AutoMatch[]> {
|
||||||
|
return queryRaw<AutoMatch>(
|
||||||
|
`WITH pairs AS (
|
||||||
|
SELECT m.id AS manual_id,
|
||||||
|
t.id AS statement_tx_id,
|
||||||
|
t.amount::text AS amount,
|
||||||
|
m.transaction_date::text AS manual_date,
|
||||||
|
t.transaction_date::text AS statement_date,
|
||||||
|
t.description AS description,
|
||||||
|
CASE WHEN m.foreign_currency_code IS NOT NULL
|
||||||
|
AND m.foreign_currency_amount IS NOT NULL
|
||||||
|
THEN 'native' ELSE 'aud' END AS basis
|
||||||
|
FROM transactions m
|
||||||
|
JOIN transactions t
|
||||||
|
ON t.statement_id IS NOT NULL
|
||||||
|
AND t.transaction_type = 'debit'
|
||||||
|
AND t.transaction_date BETWEEN m.transaction_date - $2::int
|
||||||
|
AND m.transaction_date + $2::int
|
||||||
|
AND (
|
||||||
|
-- Foreign: compare what both sides actually recorded natively.
|
||||||
|
(m.foreign_currency_code IS NOT NULL
|
||||||
|
AND m.foreign_currency_amount IS NOT NULL
|
||||||
|
AND t.foreign_currency_code = m.foreign_currency_code
|
||||||
|
AND t.foreign_currency_amount = m.foreign_currency_amount)
|
||||||
|
OR -- AUD side: exact equality, never a tolerance band. The
|
||||||
|
-- statement row MAY be foreign here, and that is not a mistake:
|
||||||
|
-- all 48 reconciliations a human has made on this ledger are
|
||||||
|
-- exactly this shape -- a manual row carrying no currency,
|
||||||
|
-- matched on exact AUD against a statement row that does carry
|
||||||
|
-- one (EUR, CHF). Overseas receipts and orders record what was
|
||||||
|
-- charged in AUD; only the wallet lane records natively, which
|
||||||
|
-- is why the branch above exists. Requiring both sides to be
|
||||||
|
-- non-foreign was tried and rejected: it matched 0 of those 48.
|
||||||
|
-- The coincidence risk (a domestic tap whose amount happens to
|
||||||
|
-- equal a foreign charge's AUD figure) is already covered by
|
||||||
|
-- unique-both-ways, which refuses as soon as a second candidate
|
||||||
|
-- exists.
|
||||||
|
(m.foreign_currency_code IS NULL
|
||||||
|
AND t.amount = m.amount)
|
||||||
|
)
|
||||||
|
JOIN statements s ON s.id = t.statement_id
|
||||||
|
WHERE m.statement_id IS NULL
|
||||||
|
AND m.owner_id = $1
|
||||||
|
AND m.reconciled_with_id IS NULL
|
||||||
|
AND m.superseded_by_id IS NULL
|
||||||
|
AND (m.payment_method IS NULL OR m.payment_method NOT IN ('cash', 'credits'))
|
||||||
|
AND t.superseded_by_id IS NULL
|
||||||
|
AND COALESCE(t.owner_id, s.owner_id) = $1
|
||||||
|
-- The statement row must be unclaimed.
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM transactions mt WHERE mt.reconciled_with_id = t.id)
|
||||||
|
)
|
||||||
|
SELECT manual_id, statement_tx_id, amount, manual_date, statement_date, description, basis
|
||||||
|
FROM pairs p
|
||||||
|
WHERE (SELECT count(*) FROM pairs q WHERE q.manual_id = p.manual_id) = 1
|
||||||
|
AND (SELECT count(*) FROM pairs q WHERE q.statement_tx_id = p.statement_tx_id) = 1
|
||||||
|
ORDER BY manual_id`,
|
||||||
|
[ownerId, windowDays]
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
import { prisma } from "@/lib/db";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Merge a statementless ("manual") transaction into its statement twin.
|
||||||
|
*
|
||||||
|
* Extracted from POST /api/transactions/reconcile so the manual queue and the
|
||||||
|
* automatic pass cannot drift apart. There must be exactly one implementation
|
||||||
|
* of this: it moves curation (overrides, tags, splits) and provenance
|
||||||
|
* (expense_metadata) off a row that reconciliation is about to hide from every
|
||||||
|
* figure, and a second copy of that logic would eventually move a different set.
|
||||||
|
*
|
||||||
|
* The manual row is NOT deleted -- it is linked via `reconciled_with_id`.
|
||||||
|
* Deleting it would cascade its children away; superseding/linking keeps the
|
||||||
|
* history and lets the row stay visible as provenance.
|
||||||
|
*/
|
||||||
|
export async function applyReconcileMatch(
|
||||||
|
manual_id: number,
|
||||||
|
statement_tx_id: number
|
||||||
|
): Promise<void> {
|
||||||
|
await prisma.$transaction(async (tx) => {
|
||||||
|
// Copy overrides: manual → statement tx
|
||||||
|
const override = await tx.transaction_overrides.findUnique({
|
||||||
|
where: { transaction_id: manual_id },
|
||||||
|
});
|
||||||
|
if (override) {
|
||||||
|
await tx.transaction_overrides.upsert({
|
||||||
|
where: { transaction_id: statement_tx_id },
|
||||||
|
update: {
|
||||||
|
category_override: override.category_override,
|
||||||
|
merchant_normalized: override.merchant_normalized,
|
||||||
|
notes: override.notes,
|
||||||
|
my_share_percent: override.my_share_percent,
|
||||||
|
updated_at: new Date(),
|
||||||
|
},
|
||||||
|
create: {
|
||||||
|
transaction_id: statement_tx_id,
|
||||||
|
category_override: override.category_override,
|
||||||
|
merchant_normalized: override.merchant_normalized,
|
||||||
|
notes: override.notes,
|
||||||
|
my_share_percent: override.my_share_percent,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
await tx.transaction_overrides.deleteMany({ where: { transaction_id: manual_id } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move tags: manual → statement tx
|
||||||
|
const tags = await tx.transaction_tags.findMany({ where: { transaction_id: manual_id } });
|
||||||
|
if (tags.length) {
|
||||||
|
await tx.transaction_tags.createMany({
|
||||||
|
data: tags.map((t) => ({ transaction_id: statement_tx_id, tag_id: t.tag_id })),
|
||||||
|
skipDuplicates: true,
|
||||||
|
});
|
||||||
|
await tx.transaction_tags.deleteMany({ where: { transaction_id: manual_id } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move splits: manual → statement tx.
|
||||||
|
//
|
||||||
|
// `settled` and `settled_at` MUST be carried. This is a delete-and-recreate,
|
||||||
|
// and the column defaults to false — so omitting it silently marks every
|
||||||
|
// settled split unsettled again and resurrects a debt that was already
|
||||||
|
// paid. Nothing errors and nothing in the UI says the flag was reset; it
|
||||||
|
// just reappears as owed. No queue row carries a settled split today, but
|
||||||
|
// the auto-matcher runs this path unattended, so a latent version of this
|
||||||
|
// would surface as money, not as an exception.
|
||||||
|
const splits = await tx.transaction_splits.findMany({ where: { transaction_id: manual_id } });
|
||||||
|
if (splits.length) {
|
||||||
|
await tx.transaction_splits.createMany({
|
||||||
|
data: splits.map((s) => ({
|
||||||
|
transaction_id: statement_tx_id,
|
||||||
|
participant_id: s.participant_id,
|
||||||
|
share_percent: s.share_percent,
|
||||||
|
settled: s.settled ?? false,
|
||||||
|
settled_at: s.settled_at,
|
||||||
|
})),
|
||||||
|
skipDuplicates: true,
|
||||||
|
});
|
||||||
|
await tx.transaction_splits.deleteMany({ where: { transaction_id: manual_id } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move provenance: manual → statement tx.
|
||||||
|
//
|
||||||
|
// Overrides, tags and splits above were always carried across; expense_metadata was
|
||||||
|
// the one child left behind, which did not matter while every metadata row came from
|
||||||
|
// an email that had created its own transaction. It matters now: a scanned grocery
|
||||||
|
// receipt puts its line items here, and reconciliation hides the manual row from
|
||||||
|
// every figure — so without this the shop's contents disappear at exactly the moment
|
||||||
|
// the statement line appears, and COLES 0556 MANOR LAKES stays as unreadable as it
|
||||||
|
// was before the receipt was ever scanned.
|
||||||
|
//
|
||||||
|
// transaction_id is UNIQUE, so a statement row that already has metadata (an emailed
|
||||||
|
// or Paperless copy got there first) keeps it. The manual row's metadata stays
|
||||||
|
// attached to the reconciled manual transaction and is flagged, rather than raising a
|
||||||
|
// constraint violation or silently overwriting the other source.
|
||||||
|
const moved = await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE expense_metadata SET transaction_id = $1
|
||||||
|
WHERE transaction_id = $2
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM expense_metadata other WHERE other.transaction_id = $1)`,
|
||||||
|
statement_tx_id,
|
||||||
|
manual_id
|
||||||
|
);
|
||||||
|
if (moved === 0) {
|
||||||
|
await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE expense_metadata
|
||||||
|
SET flags = coalesce(flags, '[]'::jsonb) || '["metadata_collision_on_reconcile"]'::jsonb
|
||||||
|
WHERE transaction_id = $1`,
|
||||||
|
manual_id
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mark manual tx as reconciled (link to statement tx)
|
||||||
|
await tx.$executeRawUnsafe(
|
||||||
|
`UPDATE transactions SET reconciled_with_id = $1 WHERE id = $2`,
|
||||||
|
statement_tx_id,
|
||||||
|
manual_id
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user