Implementing Dead-Letter Queues for Claims: Idempotent Quarantine and Replay

Problem: an X12 837 claim exhausts its retry budget or fails validation with a structural fault, and there is nowhere safe to put it — drop it and you have silently lost a billable claim; leave it on the main queue and it blocks the batch as a poison message; re-submit it blindly after a fix and you risk a duplicate interchange and a TA1/999 rejection. A dead-letter queue (DLQ) is the durable quarantine that holds these claims, keyed on the interchange and transaction control numbers so a corrected claim replays exactly once. This guide implements a DLQ record schema, an enqueue path for FATAL_SYNTAX and RETRIES_EXHAUSTED dispositions, an idempotent correction-then-replay workflow, and a poison-message cap — with metadata that carries no PHI. It is the terminal path for the retry loop in Error Categorization & Retry Logic Design and targets the engineers who own claim durability and auditability.

Prerequisites

Spec Reference: DLQ Record and Its Idempotency Key

The record stored in the DLQ is metadata only. The composite key is (ISA13, ST02): the interchange control number scopes the file, the transaction set control number scopes the individual claim transaction within it. That pair uniquely identifies the failed transaction for replay without carrying any patient identifier.

Field Source / value Requirement Notes
interchange_id ISA13 Required Interchange control number — replay scope
transaction_id ST02 Required Transaction set control number — claim within the file
reason_code FATAL_SYNTAX / RETRIES_EXHAUSTED Required Disposition enum, not free text
payload_ref URI into encrypted blob store Required Pointer only — never the raw 837 inline
rule_version Validation ruleset version Required So replay uses the same or newer rules
error_detail Categorized code (IK4, POS mismatch) Required De-identified — no PHI
attempt_count Retries consumed before dead-letter Required Feeds the poison-message cap
created_at UTC timestamp Required Age alerting and audit trail
replay_count Times replayed Required Poison cap; 0 on first enqueue

The one thing this table must never contain is PHI. HIPAA Security Rule §164.312(b) requires audit controls over ePHI, and the cleanest way to satisfy it for queue metadata is to store zero PHI in the record: reference the claim by its control numbers, keep the patient-bearing 837 bytes only in the encrypted payload store behind payload_ref.

Dead-letter queue lifecycle for failed 837 claimsA failed claim carrying a fatal syntax or retries-exhausted disposition is enqueued as a metadata record keyed on the ISA13 interchange control number and the ST02 transaction set control number, with the patient-bearing payload stored separately in an encrypted blob store referenced by payload_ref. An analyst reviews and corrects the claim. On replay, an idempotency check on the ISA13 and ST02 pair prevents a duplicate submission; the replay counter is compared against the poison-message cap. If the cap is exceeded the record is routed to a permanent parking area for manual escalation, otherwise the corrected claim is resubmitted to the parser. Metrics track queue depth and record age throughout. FAILED CLAIM Terminal fault FATAL_SYNTAX or RETRIES_EXHAUSTED DEAD-LETTER QUEUE (metadata only) DLQ record key = (ISA13, ST02) reason · rule_version · payload_ref · created_at Encrypted store raw 837 (PHI) at rest payload_ref → blob stores Analyst correction fix code / modifier / envelope, bump rule_version Idempotent replay guard seen(ISA13, ST02)? · replay_count < cap? exactly-once resubmit to parser Poison parking cap exceeded → manual escalation cap hit clean replay
The DLQ record holds only control numbers and disposition metadata; the PHI-bearing 837 lives encrypted behind payload_ref. Replay is guarded on the (ISA13, ST02) pair and a poison cap so a corrected claim resubmits exactly once.

Step-by-Step Implementation

Step 1 — Define the PHI-safe DLQ record schema

Model the record as a frozen dataclass with a disposition enum, not free-text reason strings. The dlq_key property is the idempotency key; it is derived only from control numbers so it can be logged and indexed safely.

from __future__ import annotations

import enum
import logging
from dataclasses import dataclass, field
from datetime import datetime, timezone

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
logger = logging.getLogger("x12.dlq")


class DlqReason(enum.StrEnum):
    FATAL_SYNTAX = "FATAL_SYNTAX"          # 999 IK3/IK4, ValidationError
    RETRIES_EXHAUSTED = "RETRIES_EXHAUSTED"  # transient budget spent


@dataclass(frozen=True, slots=True)
class DlqRecord:
    """DLQ metadata for one failed 837 transaction. Carries NO PHI."""

    interchange_id: str          # ISA13
    transaction_id: str          # ST02
    reason_code: DlqReason
    payload_ref: str             # URI into the encrypted blob store
    rule_version: str            # ruleset that rejected it
    error_detail: str            # de-identified code, e.g. "IK4:SV102"
    attempt_count: int
    created_at: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )
    replay_count: int = 0

    @property
    def dlq_key(self) -> str:
        """Idempotency key: interchange + transaction control numbers."""
        return f"{self.interchange_id}:{self.transaction_id}"

Step 2 — Enqueue on FATAL_SYNTAX and RETRIES_EXHAUSTED

The two terminal errors from the retry controller map to the two disposition codes. Store the raw 837 in the encrypted blob store first, then persist the metadata record with only the returned reference. The audit log line records the key and reason — never the payload.

from typing import Protocol


class BlobStore(Protocol):
    def put_encrypted(self, data: bytes) -> str: ...  # returns payload_ref


class DlqStore(Protocol):
    def upsert(self, record: DlqRecord) -> None: ...
    def get(self, dlq_key: str) -> DlqRecord | None: ...


def dead_letter(
    raw_837: bytes,
    isa13: str,
    st02: str,
    reason: DlqReason,
    rule_version: str,
    error_detail: str,
    attempt_count: int,
    blobs: BlobStore,
    dlq: DlqStore,
) -> DlqRecord:
    """Quarantine a failed transaction: encrypt payload, persist metadata."""
    payload_ref = blobs.put_encrypted(raw_837)  # PHI encrypted at rest
    record = DlqRecord(
        interchange_id=isa13,
        transaction_id=st02,
        reason_code=reason,
        payload_ref=payload_ref,
        rule_version=rule_version,
        error_detail=error_detail,
        attempt_count=attempt_count,
    )
    dlq.upsert(record)
    logger.error(
        "DLQ enqueue | key=%s | reason=%s | rule=%s | detail=%s",
        record.dlq_key, reason.value, rule_version, error_detail,
    )
    return record

Step 3 — Replay idempotently after correction

Once an analyst corrects the claim, replay must resubmit exactly once. Guard on the (ISA13, ST02) key against an already-processed set, and enforce the poison cap before resubmitting. A record at or over the cap is parked for manual escalation rather than looped forever.

POISON_CAP = 3


def replay_claim(
    dlq_key: str,
    dlq: DlqStore,
    blobs: BlobStore,
    parser_fn,               # Callable[[bytes], object]
    processed: set[str],     # durable set of resubmitted keys
) -> str:
    """Idempotent correction-then-replay of a dead-lettered claim."""
    record = dlq.get(dlq_key)
    if record is None:
        return "not_found"

    if dlq_key in processed:
        # Already resubmitted once — do not create a duplicate interchange.
        logger.warning("Replay suppressed (already processed): key=%s", dlq_key)
        return "duplicate_suppressed"

    if record.replay_count >= POISON_CAP:
        logger.error("Poison cap hit: key=%s -> parking for escalation", dlq_key)
        return "parked"

    raw = blobs.get_encrypted(record.payload_ref)  # decrypt in-process only
    parser_fn(raw)  # resubmit the corrected claim to the parser
    processed.add(dlq_key)
    dlq.upsert(
        DlqRecord(
            **{**record.__dict__, "replay_count": record.replay_count + 1}
        )
    )
    logger.info("Replayed claim | key=%s | attempt=%d",
                dlq_key, record.replay_count + 1)
    return "replayed"

Step 4 — Emit depth and age metrics for alerting

A DLQ that grows unbounded is an outage in disguise. Export queue depth and the age of the oldest record so an alert fires before a backlog becomes a timely-filing problem. Metrics carry only counts and timestamps — no PHI.

from datetime import datetime, timezone


def emit_dlq_metrics(records: list[DlqRecord], gauge) -> None:
    """Publish depth and oldest-record age. Alert when either breaches SLO."""
    depth = len(records)
    gauge("dlq.depth", depth)

    if records:
        oldest = min(r.created_at for r in records)
        age_s = (datetime.now(timezone.utc) - oldest).total_seconds()
        gauge("dlq.oldest_age_seconds", age_s)

    by_reason: dict[str, int] = {}
    for r in records:
        by_reason[r.reason_code.value] = by_reason.get(r.reason_code.value, 0) + 1
    for reason, count in by_reason.items():
        gauge(f"dlq.depth_by_reason.{reason}", count)

Verification

Confirm the two invariants that keep the DLQ safe: replay is exactly-once on the (ISA13, ST02) key, and a claim that keeps failing is parked at the cap rather than looping. A de-identified fixture drives both paths.

processed: set[str] = set()
calls = {"n": 0}

def fake_parser(_raw: bytes) -> None:
    calls["n"] += 1

# First replay resubmits; second replay of the same key is suppressed.
assert replay_claim("000000123:0001", dlq, blobs, fake_parser, processed) == "replayed"
assert replay_claim("000000123:0001", dlq, blobs, fake_parser, processed) == "duplicate_suppressed"
assert calls["n"] == 1  # parser invoked exactly once

Expected log output (control numbers only — no PHI reaches the sink):

2026-07-16 10:04:11 | ERROR   | x12.dlq | DLQ enqueue | key=000000123:0001 | reason=FATAL_SYNTAX | rule=v2026.07 | detail=IK4:SV102
2026-07-16 10:19:02 | INFO    | x12.dlq | Replayed claim | key=000000123:0001 | attempt=1
2026-07-16 10:20:47 | WARNING | x12.dlq | Replay suppressed (already processed): key=000000123:0001

For the poison path, seed a record with replay_count = POISON_CAP and assert replay_claim returns "parked" and does not invoke the parser — a claim that has failed three corrections needs a human, not another automated loop.

Common Gotchas

  • Replay without an idempotency key. Resubmitting a corrected claim without checking the (ISA13, ST02) pair against a durable processed-set creates a second interchange with the same control number — the clearinghouse answers with a TA1 interchange rejection or a duplicate 999. Persist the key set alongside the DLQ, not in process memory, so a worker restart cannot replay a claim twice.
  • PHI in payload storage. The raw 837 behind payload_ref contains patient names, member IDs, and diagnoses. It must be encrypted at rest (§164.312(a)(2)(iv)) and decrypted only in-process at replay — never written to a log, a debug dump, or the DLQ metadata row. Keep the metadata table and the payload store on separate access controls.
  • Unbounded DLQ growth with no alerting. Without a depth-and-age gauge, a systemic rule regression can silently dead-letter thousands of claims until they breach a payer’s timely-filing window. Alert on both depth and oldest-record age, and break down dlq.depth_by_reason so a spike in FATAL_SYNTAX (a bad rule deploy) is distinguishable from RETRIES_EXHAUSTED (a clearinghouse outage).
  • Reusing a stale rule_version on replay. If a claim was rejected under v2026.07 and the ruleset has since changed, replaying it must record the version actually applied. Bumping rule_version on correction (as the analyst step does) keeps the audit trail honest about which rules judged the resubmission.

Up: Error Categorization & Retry Logic Design