Runbook: Payout Failure & Exception Resolution

---

1. Incident Classification & Severity

Severity Level	Operational Impact	Response SLA	Target Escalation
SEV-1 (Critical)	Paystack master balance depleted; batch payout failure impacting $> 5$ vendors; systemic bank switch downtime.	$< 15\text{ minutes}$	Primary On-Call Engineer, Finance Controller, CTO
SEV-2 (High)	Individual high-value payout failure ($> ₦200,000$); recurring transfer rejection on a single verified merchant.	$< 1\text{ hour}$	On-Call Engineer, Support Lead
SEV-3 (Moderate)	Individual transfer failure due to invalid NUBAN, incorrect bank code, or vendor-initiated bank detail update.	$< 4\text{ hours}$	Customer Support Specialist

Primary Incident Triggers

• Inbound Paystack Webhook: Event transfer.failed or transfer.reversed received.
• Observation Alert: Route /api/payout-exceptions/observations registers anomalous failure rate ($> 5%$).
• Vendor Ticket: Vendor reports unreceived funds $> 24$ hours post-delivery PIN confirmation.

---

2. Preliminary Triage & Root Cause Analysis

graph TD
    ALERT[Payout Exception Triggered] --> STEP1[Step 1: Query Database Transfer Record]
    STEP1 --> STEP2[Step 2: Query Live Paystack Transfer API]
    
    STEP2 --> DECIDE{Categorize Paystack Failure Code}
    
    DECIDE -->|balance_insufficient| PATH_A[Procedure A: Master Account Depletion]
    DECIDE -->|bank_switch_error / timeout| PATH_B[Procedure B: Destination Bank / NIBSS Outage]
    DECIDE -->|account_number_invalid / name_mismatch| PATH_C[Procedure C: Invalid NUBAN Account]
    DECIDE -->|reversed| PATH_D[Procedure D: Post-Settlement Reversal]

Step 1: Query Database Transfer Context

Run the diagnostic query against the PostgreSQL replica or psql console:

SELECT 
    pr.id AS payout_id,
    pr.vendor_id,
    pr.amount,
    pr.status,
    pr.transfer_reference,
    pr.transfer_code,
    pr.transfer_failure,
    vbd.account_number,
    vbd.bank_code,
    vbd.account_name,
    vbd.updated_at AS bank_last_modified
FROM public.payout_requests pr
JOIN public.vendor_bank_details vbd ON vbd.vendor_id = pr.vendor_id
WHERE pr.id = '<TARGET_PAYOUT_ID>' 
   OR pr.transfer_reference = '<TARGET_REFERENCE>';

Step 2: Query Live Paystack Transfer Verification API

Validate the terminal state directly with the upstream banking provider:

curl -X GET "https://api.paystack.co/transfer/verify/<TRANSFER_REFERENCE>" \
  -H "Authorization: Bearer $PAYSTACK_SECRET_KEY" \
  -H "Accept: application/json"

---

3. Resolution Procedures

Procedure A: Master Settlement Account Depletion (balance_insufficient)

Root Cause: Debelu's Paystack master transfer balance has dropped below the total batch sum.

1. Verify Master Balance:

   curl -X GET "https://api.paystack.co/balance" \
     -H "Authorization: Bearer $PAYSTACK_SECRET_KEY"

2. Top-Up Settlement Balance:
   • Notify Finance Controller to initiate immediate bank transfer to Debelu's dedicated Paystack top-up account.
3. Unpause Batch Dispatcher:
   • Once Paystack balance reflects credit, restart the payout queue worker:

   fly ssh console -C "npm run dispatch:payouts -- --retry-depleted"

---

Procedure B: Destination Bank / NIBSS Switch Outage

Root Cause: Temporary downtime at the recipient commercial bank (e.g. Zenith, Access, GTBank) or national switch (NIBSS).

1. Check Bank Switch Status:
   • Inspect Paystack status page (status.paystack.com) or query recent transfers to the same bank_code.
2. Queue Automated Backoff Retry:
   • If failure is transient, reset transfer status to trigger automated exponential backoff:

   UPDATE public.payout_requests 
   SET status = 'pending', 
       retry_count = retry_count + 1,
       transfer_failure = 'Transient interbank timeout; scheduled for retry'
   WHERE id = '<TARGET_PAYOUT_ID>' AND retry_count < 3;

3. Notify Vendor: Send proactive delay notification (see Template 1).

---

Procedure C: Invalid NUBAN / Account Name Mismatch

Root Cause: The vendor's destination bank account has been closed, frozen by BVN restrictions, or incorrectly input.

1. Quarantine Vendor Payout Profile:

   UPDATE public.vendor_bank_details 
   SET verified = false, 
       quarantine_reason = 'Payout rejected by destination bank: Invalid Account/Name Mismatch'
   WHERE vendor_id = '<VENDOR_ID>';

2. Restore Funds to In-App Wallet:
   • Credit the vendor's available wallet balance so funds are not locked in limbo:

   SELECT public.ledger_post(
       '<VENDOR_ID>'::uuid,
       <AMOUNT_NAIRA>,
       'Payout_Failed_Restitution',
       'Failed payout transfer restitution; update bank details',
       NULL, NULL, NULL,
       'RESTITUTION-' || '<TARGET_PAYOUT_ID>'
   );

3. Request Vendor Remediation: Dispatch Template 2 via email and push notification.

---

Procedure D: Post-Settlement Interbank Reversal (reversed)

Root Cause: Paystack initially acknowledged transfer.success, but the recipient bank returned funds 24–48 hours later due to KYC tier limits on the vendor's student account.

1. Execute Maker-Checker Payout Reconciliation:
   • Because funds were initially debited and marked completed, do not manually edit balances.
   • Staff member (Maker) drafts a reconciliation proposal via PayoutTransferReconciliationService:

   await PayoutTransferReconciliationService.prepare(
       staffActorId,
       reconciliationId,
       targetPayoutId,
       "Recipient bank reversed transfer; restoring vendor wallet"
   );

2. Checker Review:
   • A distinct Finance Lead approves the reconciliation in the admin portal.
   • Stored procedure review_payout_reconciliation executes the canonical Reversal ledger post and credits the vendor balance atomically.

---

4. Vendor Communication Templates

Template 1: Interbank Network Delay Notification

Subject: Update Regarding Your Debelu Payout Transfer (Ref: {{transfer_reference}})

Dear {{vendor_name}},

We attempted to disburse your payout of ₦{{amount}} for completed orders to your {{bank_name}} account ending in {{last4}}. 

Our payment network reported a temporary interbank network timeout with {{bank_name}}. Your funds remain completely safe in your Debelu account.

Our system has queued an automated retry within the next 4 hours. No action is required from you at this time. If the transfer does not complete by {{expected_time}}, our support team will reach out directly.

Thank you for selling on Debelu!

Template 2: Action Required - Update Bank Account Details

Subject: Action Required: Please Update Your Bank Account Details

Dear {{vendor_name}},

Your recent payout request of ₦{{amount}} could not be delivered because {{bank_name}} reported that the account details (ending in {{last4}}) could not be verified or are currently restricted.

The funds have been returned to your Debelu available balance. 

To receive your payout:
1. Log in to your Debelu Vendor Dashboard.
2. Navigate to Settings > Payout Accounts.
3. Link an active NUBAN bank account matching your verified legal profile name.

Once updated, you can immediately initiate a new withdrawal.

Need assistance? Reply directly to this email or chat with Support in-app.

---

5. Post-Resolution Verification Checklist

Before closing the incident ticket, on-call staff must verify:

• [ ] Database transfer record in public.payout_requests reflects terminal status (processed or failed).
• [ ] Internal ledger transaction in public.transactions matches the physical bank movement ($100%$ zero drift).
• [ ] Vendor notified via registered email and in-app notification outbox.
• [ ] If SEV-1, incident summary and timeline logged to internal operations channel.