fix(transactions): supersede rows imported twice instead of deleting them
ci / lint-test (push) Successful in 48s
ci / lint-test (push) Successful in 48s
Statements 107, 142 and 143 bill overlapping periods on one ANZ account, so 31 transactions -- $42,040.68 -- are in the ledger twice. They are marked superseded, not deleted. Every child of transactions is ON DELETE CASCADE (splits, tags, overrides, expense_metadata, order_reviews), so deleting "the duplicate" destroys whatever curation sits on it, and which member of a pair holds that curation is an accident of import order: here 1 pair carries splits and 6 carry overrides, all on the surviving side, but nothing guarantees that. Superseding keeps the row, keeps its children, and makes a mistake one UPDATE to undo rather than a restore from backup. reconciled_with_id could not be reused. Its predicate is scoped to statement_id IS NULL on purpose -- a statement line pointing at something else is the survivor, not the duplicate -- and here both rows are statement lines. The exclusion goes into EXCLUDE_RECONCILED_SOURCE rather than into a new fragment, so every query already asking "count each purchase once" gets it without being edited. The trip cost queries did not use that fragment at all and now do; verified a no-op on current data (0 trip-tagged rows are either reconciled sources or duplicates), but they were one import away from double-counting. Most of the $42k is transfers and investments, which spend already excludes. The damage was elsewhere: duplicated rows in the list, and rules re-splitting a duplicate -- txn 3807 is one of these 31 and was a candidate for splitting earlier today. Balances are unchanged: no duplicate carried a split.
This commit is contained in:
@@ -36,7 +36,9 @@ export const OWNER_SCOPE = `COALESCE(t.owner_id, s.owner_id)`;
|
||||
export const STATEMENTS_JOIN = `LEFT JOIN statements s ON s.id = t.statement_id`;
|
||||
|
||||
/**
|
||||
* Drops the manual/CSV row that a statement line has superseded.
|
||||
* Drops rows that a different row has replaced. Two distinct cases:
|
||||
*
|
||||
* **1. The manual/CSV row a statement line superseded** (`reconciled_with_id`).
|
||||
*
|
||||
* Reconciliation keeps both rows: the manual one the user entered and the
|
||||
* statement line it turned out to be. Only the statement line should count, or
|
||||
@@ -52,8 +54,26 @@ export const STATEMENTS_JOIN = `LEFT JOIN statements s ON s.id = t.statement_id`
|
||||
* inserted with `reconciled_with_id` NULL and are held out of the reconcile
|
||||
* queue by `needsCardMatch()`, so nothing ever sets it. If one is reconciled by
|
||||
* hand against a card line, this is what stops it double-counting.
|
||||
*
|
||||
* **2. The statement row imported twice** (`superseded_by_id`, migration 0023).
|
||||
*
|
||||
* When two statements for one account bill overlapping periods, every
|
||||
* transaction in the overlap arrives twice — 31 pairs on ANZ `4085-56264`, from
|
||||
* statements 107/142/143. Case 1 cannot express this: its predicate is scoped
|
||||
* to `statement_id IS NULL` on purpose, and here BOTH rows are statement lines.
|
||||
*
|
||||
* The superseded row is excluded rather than deleted because every child of
|
||||
* `transactions` cascades on delete, and the curation is not reliably on the
|
||||
* surviving side.
|
||||
*
|
||||
* Adding it to this fragment rather than making a new one is deliberate: every
|
||||
* query that already asks "count each purchase once" now excludes both kinds
|
||||
* without being edited. Anything summing transactions without this fragment
|
||||
* still double-counts — that is the same gap that let reconciled rows into the
|
||||
* analytics routes in the first place.
|
||||
*/
|
||||
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)`;
|
||||
export const EXCLUDE_RECONCILED_SOURCE = `NOT (t.statement_id IS NULL AND t.reconciled_with_id IS NOT NULL)
|
||||
AND t.superseded_by_id IS NULL`;
|
||||
|
||||
/**
|
||||
* The currency `t.amount` is actually denominated in.
|
||||
|
||||
@@ -897,6 +897,7 @@ export interface TripAnalytics {
|
||||
// alias the shared fragments assume.
|
||||
const TRIP_TOTAL_SPEND = `COALESCE(SUM(
|
||||
CASE WHEN ${NET_SPEND_ROWS}
|
||||
AND ${EXCLUDE_RECONCILED_SOURCE}
|
||||
AND COALESCE(o.category_override, t.category, 'other') NOT IN ('transfers', 'investment')
|
||||
THEN ${SPEND_SIGNED} ELSE 0 END
|
||||
), 0)::float AS total_spend`;
|
||||
@@ -964,6 +965,7 @@ export async function getTripAnalytics(tripId: number, ownerId: number): Promise
|
||||
JOIN transactions t ON t.id = o.transaction_id
|
||||
WHERE o.trip_id = $1
|
||||
AND ${NET_SPEND_ROWS}
|
||||
AND ${EXCLUDE_RECONCILED_SOURCE}
|
||||
AND COALESCE(o.category_override, t.category, 'other') NOT IN ('transfers', 'investment')
|
||||
GROUP BY 1
|
||||
ORDER BY 2 DESC
|
||||
@@ -977,6 +979,7 @@ export async function getTripAnalytics(tripId: number, ownerId: number): Promise
|
||||
JOIN transactions t ON t.id = o.transaction_id
|
||||
WHERE o.trip_id = $1
|
||||
AND ${NET_SPEND_ROWS}
|
||||
AND ${EXCLUDE_RECONCILED_SOURCE}
|
||||
AND COALESCE(o.category_override, t.category, 'other') NOT IN ('transfers', 'investment')
|
||||
GROUP BY 1
|
||||
ORDER BY 1
|
||||
@@ -991,6 +994,7 @@ export async function getTripAnalytics(tripId: number, ownerId: number): Promise
|
||||
JOIN transactions t ON t.id = o.transaction_id
|
||||
WHERE o.trip_id = $1
|
||||
AND ${NET_SPEND_ROWS}
|
||||
AND ${EXCLUDE_RECONCILED_SOURCE}
|
||||
AND COALESCE(o.category_override, t.category, 'other') NOT IN ('transfers', 'investment')
|
||||
GROUP BY 1
|
||||
ORDER BY 2 DESC
|
||||
@@ -1008,6 +1012,7 @@ export async function getTripAnalytics(tripId: number, ownerId: number): Promise
|
||||
JOIN tags tg ON tg.id = tt.tag_id
|
||||
WHERE o.trip_id = $1
|
||||
AND ${NET_SPEND_ROWS}
|
||||
AND ${EXCLUDE_RECONCILED_SOURCE}
|
||||
AND COALESCE(o.category_override, t.category, 'other') NOT IN ('transfers', 'investment')
|
||||
GROUP BY tg.id
|
||||
ORDER BY 4 DESC
|
||||
|
||||
Reference in New Issue
Block a user