Monitoring, Observability & Alerting Architecture

---

1. Observability Philosophy & The 4 Golden Signals

Debelu operates on the Google SRE observability framework, combining white-box monitoring (internal metrics, queue depths, database query latencies) with black-box synthetic monitoring (external end-to-end checkout pings, synthetic webhook deliveries, uptime probes).

All telemetry is anchored around the Four Golden Signals:

graph LR
    subgraph The 4 Golden Signals
        LAT[1. Latency: Response Time Distribution]
        TRAF[2. Traffic: Demand & Request Throughput]
        ERR[3. Errors: Rate of Failed Transactions]
        SAT[4. Saturation: Resource Exhaustion]
    end

    subgraph Debelu Telemetry Implementations
        LAT --> CWV[Core Web Vitals + API p95/p99 Latency]
        TRAF --> RPS[Express Request Counter + Redis Sliding Window]
        ERR --> SENTRY[Sentry Error Tracking + RFC 7807 5xx Counter]
        SAT --> SYSTEM[PgBouncer Pool + BullMQ Queue Depths]
    end

---

2. Error Tracking & Crash Reporting (Sentry)

Sentry provides unified distributed tracing and error capture across all clients and microservices.

2.1 Storefront & Admin Integration (@sentry/react)

The React SPA wraps all route boundaries in Sentry Error Boundaries:

• Release Correlation: Injected at build time via VITE_APP_VERSION and git commit SHA.
• PII & Financial Data Masking: beforeSend hooks rigorously sanitize cardholder data, NUBAN account numbers, and phone numbers:

// packages/core/src/monitoring/sentrySanitizer.ts
export function sanitizeSentryEvent(event: Sentry.Event): Sentry.Event {
  if (event.request?.data) {
    let dataStr = JSON.stringify(event.request.data);
    // Mask Nigerian phone numbers: +234 or 080...
    dataStr = dataStr.replace(/(?:\+234|0)[789][01]\d{8}/g, '[REDACTED_PHONE]');
    // Mask 10-digit NUBAN bank account numbers
    dataStr = dataStr.replace(/\b\d{10}\b/g, '[REDACTED_NUBAN]');
    // Mask 4-digit Delivery PINs
    dataStr = dataStr.replace(/"pin":\s*"\d{4}"/g, '"pin":"[REDACTED_PIN]"');
    event.request.data = JSON.parse(dataStr);
  }
  return event;
}

2.2 Next.js Marketing Site Integration

The Next.js 16 App Router utilizes three distinct runtime configurations:

• [instrumentation-client.ts](file:///c:/Users/frank/OneDrive/Desktop/Chisom/Debelu/New%20Debelu%20Marketplace/debelu-marketing/instrumentation-client.ts): Browser client hydration tracing and CWV metrics.
• [sentry.server.config.ts](file:///c:/Users/frank/OneDrive/Desktop/Chisom/Debelu/New%20Debelu%20Marketplace/debelu-marketing/sentry.server.config.ts): Node.js server-side rendering (SSR) crash captures.
• [sentry.edge.config.ts](file:///c:/Users/frank/OneDrive/Desktop/Chisom/Debelu/New%20Debelu%20Marketplace/debelu-marketing/sentry.edge.config.ts): Vercel Edge Middleware session handoff tracking.

2.3 Express Backend Integration

Every unhandled exception in debelu-backend is intercepted by the terminal error middleware:

// debelu-backend/src/middleware/errorHandler.ts
app.use((err: Error, req: Request, res: Response, next: NextFunction) => {
  const correlationId = req.headers['x-correlation-id'] || crypto.randomUUID();
  
  Sentry.withScope((scope) => {
    scope.setTag('correlationId', correlationId);
    scope.setUser({ id: req.user?.id, role: req.user?.role });
    scope.setExtra('path', req.path);
    scope.setExtra('method', req.method);
    Sentry.captureException(err);
  });

  res.status(500).json({
    type: 'https://debelu.com/errors/internal-server-error',
    title: 'Internal Server Error',
    status: 500,
    detail: 'An unexpected system failure occurred.',
    correlationId
  });
});

---

3. Deep Health Check Probes (HealthCheckService)

Debelu separates basic container liveness from deep subsystem readiness:

3.1 Endpoint Architecture

• Liveness Probe: GET /health/liveness $\to$ Returns HTTP 200 {"status": "alive"} if the Express event loop is responding. Used by Railway/K8s container restarts.
• Readiness Probe: GET /health/readiness $\to$ Executes [HealthCheckService.ts](file:///c:/Users/frank/OneDrive/Desktop/Chisom/Debelu/New%20Debelu%20Marketplace/debelu-backend/src/services/HealthCheckService.ts) and validates 7 critical downstream systems.

3.2 7-Point Dependency Verification Engine

graph TD
    HC[HealthCheckService.executeReadinessCheck]
    HC --> D1[1. PostgreSQL DB: SELECT 1 via Supabase Client]
    HC --> D2[2. Supabase Auth: Ping /auth/v1/health]
    HC --> D3[3. Redis Cache: PING/PONG Latency Probe]
    HC --> D4[4. Paystack Gateway: API Handshake Ping]
    HC --> D5[5. Google Gemini AI: Quota & Token Verification]
    HC --> D6[6. Cloudflare R2: S3 HeadBucket Probe]
    HC --> D7[7. Unleash Proxy: Feature Flag Sync Ping]

Typical Readiness Probe Response (HTTP 200 OK)

{
  "status": "healthy",
  "timestamp": "2026-10-05T09:45:00.000Z",
  "version": "1.4.2",
  "uptimeSeconds": 86420,
  "checks": {
    "database": { "status": "up", "latencyMs": 14 },
    "auth": { "status": "up", "latencyMs": 32 },
    "redis": { "status": "up", "latencyMs": 2 },
    "paymentGateway": { "status": "up", "latencyMs": 110 },
    "geminiAssistant": { "status": "up", "latencyMs": 240 },
    "storageR2": { "status": "up", "latencyMs": 45 },
    "featureFlags": { "status": "up", "latencyMs": 18 }
  }
}

If any primary dependency fails (Database or Redis), the endpoint returns HTTP 503 Service Unavailable, preventing the ingress load balancer from routing traffic to that instance.

---

4. Backend Alert Engine (monitoring/alerts.ts)

Backend alert definitions trigger automated Slack webhooks, PagerDuty calls, and Sentry alerts:

Alert Identifier	Metric & Condition	Threshold	Severity	Automated Action
ERR_API_SPIKE	5xx error rate over total requests	$> 2.0%$ for $5\text{ mins}$	SEV-2	PagerDuty on-call dispatch; Slack #alerts-backend.
DB_POOL_SATURATED	Active connections / Max Pool	$> 85%$ for $3\text{ mins}$	SEV-2	Warn on-call; trigger slow query log dump.
PAYMENT_WEBHOOK_FAIL	Paystack webhook signature rejection	$> 5$ failures in $5\text{ mins}$	SEV-1	Lock ingress; trigger security verification.
QUEUE_STALL	BullMQ oldest waiting job age	$> 120\text{ seconds}$	SEV-2	Spin up secondary queue worker instance.
ESCROW_DRIFT	$Assets \neq Liabilities + Equity$	Any discrepancy $> ₦0$	SEV-1	Halt automated payouts; alert CFO & Tech Lead.

---

5. Performance Monitoring & Core Web Vitals (CWV)

5.1 Lighthouse CI Integration

The marketing site and storefront enforce strict performance budgets via [lighthouserc.json](file:///c:/Users/frank/OneDrive/Desktop/Chisom/Debelu/New%20Debelu%20Marketplace/debelu-marketing/lighthouserc.json):

• Performance: $\ge 90$
• Accessibility: $\ge 95$
• Best Practices: $\ge 95$
• SEO: $\ge 100$

5.2 Real-User Monitoring (RUM) Budgets

Client-side Web Vitals SDK streams performance measurements to the analytics endpoint:

Core Web Vital	Metric Name	Target Budget	Action on Breach
LCP (Largest Contentful Paint)	Main hero image or catalog render	$\le 2.2\text{ s}$	Optimize image dimensions, enable Cloudflare R2 Polish.
INP (Interaction to Next Paint)	Button click to frame update	$\le 150\text{ ms}$	Defer heavy JS computations into Web Workers.
CLS (Cumulative Layout Shift)	Visual stability during page load	$\le 0.05$	Enforce fixed aspect ratios on product image cards.
TTFB (Time to First Byte)	Initial server response latency	$\le 600\text{ ms}$	Review Vercel Edge Cache hit rates and DB query latency.

---

6. Financial Anomaly & Ledger Monitoring

Specialized backend services continuously monitor financial invariants:

6.1 FinancialObservationService

• Runs as a persistent background daemon inspecting ledger balance equations.
• Evaluates total locked escrow liabilities in public.orders against current Paystack virtual ledger balances.
• Emits high-priority alarms if any vendor balance becomes negative ($Balance < 0$).

6.2 PayoutExceptionObservationsService

• Inspects payout_transfer_intents for stalled states (pending for $> 48\text{ hours}$).
• Flags banks with abnormally high failure rates (e.g. NIP network timeouts on specific Nigerian commercial banks).

6.3 PaymentIntentInspectionService

• Detects "abandoned" checkouts with gateway charges that were never confirmed by client callbacks or webhooks.
• Triggers automated restitution jobs to release reserved product inventory locks.

---

7. Queue Depth & Background Worker Monitoring

BullMQ queues run on top of Redis to handle asynchronous workloads:

• notifications: Multi-channel dispatch (Email, Push, WhatsApp).
• webhooks: Inbound Paystack and Meta webhook processing.
• escrow-reconciliation: Periodic financial ledger balancing.
• ai-moderation: Background Gemini vision screening on product uploads.

Monitoring Metric Endpoints

The backend exposes GET /api/admin/queues (protected by staff MFA) returning:

{
  "queues": {
    "webhooks": { "waiting": 0, "active": 2, "failed": 0, "completed": 1420 },
    "notifications": { "waiting": 12, "active": 4, "failed": 1, "completed": 9840 },
    "ai-moderation": { "waiting": 3, "active": 1, "failed": 0, "completed": 450 }
  },
  "redisMemoryUsedBytes": 14285712,
  "connectedWorkers": 4
}

---

8. Log Management & Auditing

8.1 Structured Winston JSON Format

All backend services output single-line JSON logs to stdout:

{
  "timestamp": "2026-10-05T09:48:12.304Z",
  "level": "info",
  "message": "Order escrow release executed successfully",
  "service": "debelu-backend",
  "correlationId": "req_f8a92b3c4d",
  "campusId": "unilag",
  "orderId": "10000000-0000-4000-8000-000000000801",
  "vendorId": "10000000-0000-4000-8000-000000000002",
  "amountMinor": 1500000,
  "durationMs": 42
}

8.2 Audit Log API (logRoutes.ts)

Staff audit logs are stored immutably in PostgreSQL (audit_logs table) and exposed via [logRoutes.ts](file:///c:/Users/frank/OneDrive/Desktop/Chisom/Debelu/New%20Debelu%20Marketplace/debelu-backend/src/routes/logRoutes.ts) for internal compliance reviews:

• Every administrative role change, manual wallet adjustment, and product takedown is logged with the operator's IP address, user agent, and before/after JSON diffs.
• Retention: Audit logs are retained for 7 years in adherence to Nigerian statutory tax and AML record-keeping obligations.