payguard-poc/README_IMPLEMENTATION.md

PayGuard Fraud Detection System

A comprehensive, academically-rigorous demonstration of a dual-layer fraud detection system for the Advanced System Design course. This project showcases fundamental microservices architecture patterns: decoupling, asynchronous processing, tiered computing, and resilience patterns.

📋 Table of Contents

• Architecture Overview
• Key Design Principles
• Project Structure
• Quick Start
• Running the Demo
• Understanding the Pipeline
• Design Patterns Demonstrated

---

🏗️ Architecture Overview

The PayGuard system processes transactions through a sophisticated multi-stage pipeline:

Transaction Request
    ↓
[JWT Authentication] ← Verify caller identity
    ↓
[Kafka Ingestion] ← Async, decoupled message queue
    ↓
[Feature Extraction] ← Enrich with behavioral history
    ↓
[Layer 1 - Fast Model] ← Logistic regression (milliseconds)
    ├─ Score < 0.4 → ✓ APPROVE (fast path)
    └─ Score ≥ 0.4 → Continue to Layer 2
    ↓
[Behavioral Profiling] ← Redis lookup for anomalies
    ↓
[Layer 2 - Deep Model] ← XGBoost/ensemble (heavier analysis)
    ├─ Score > 0.7 → ✗ BLOCK
    └─ Score ≤ 0.7 → ✓ APPROVE
    ↓
[Circuit Breaker] ← Graceful fallback if Layer 2 fails
    ↓
[Final Decision Engine] ← Latency-aware thresholding
    ↓
[Result Storage] ← Redis (1-hour TTL)

Key Innovation: The dual-branch decision gates allow >95% of transactions to be approved with a fast, lightweight model, while only suspicious transactions pay the cost of deep analysis.

---

🎯 Key Design Principles

1. Microservices & Decoupling

The API layer (ingestion) is completely independent from the fraud detection layer (consumer). They communicate only through Kafka messages. This means:

• The API can scale independently from the scorer
• If the scorer is overloaded, transactions don't get lost—they wait in Kafka
• Services can be deployed and updated independently

Code Location: api/main.py (API) and consumer/pipeline.py (Consumer)

2. Asynchronous Message Passing

Kafka is the backbone. The API doesn't wait for fraud analysis; it immediately acknowledges receipt and returns to the client. The consumer processes transactions at its own pace.

Academic Benefit: Demonstrates the CAP theorem trade-off (consistency vs. availability)

Code Location: api/main.py (producer) and consumer/pipeline.py (consumer)

3. Tiered Processing (Adaptive Routing)

The system routes transactions dynamically:

• 95% of transactions: Handled by Layer 1 only (fast path)
• 5% suspicious transactions: Escalated to Layer 2 (deep analysis)

This optimizes the latency-accuracy trade-off: most users get instant approval, but fraud is caught with higher confidence.

Code Location: consumer/pipeline.py (routing logic), consumer/models.py (scoring)

4. Resilience Patterns

The circuit breaker pattern ensures graceful degradation:

• If Layer 2 becomes slow or unavailable, the system falls back to conservative Layer 1 estimates
• This prevents cascading failures and ensures uptime even during outages

Code Location: consumer/decision.py (circuit breaker)

---

📁 Project Structure

payguard-poc/
├── docker-compose.yml              # Infrastructure: Kafka, Zookeeper, Redis
├── requirements.txt                # Python dependencies
├── .env.example                    # Configuration template
│
├── api/
│   └── main.py                    # FastAPI transaction ingestion layer
│       ├── JWT authentication
│       ├── Kafka producer
│       └── Request validation
│
├── consumer/
│   ├── __init__.py
│   ├── pipeline.py                # Main orchestrator (steps 1-6)
│   ├── features.py                # Feature extraction & profiling
│   ├── models.py                  # Layer 1 & Layer 2 scoring
│   └── decision.py                # Decision engine & circuit breaker
│
├── models/
│   ├── layer1.pkl                 # Fast model (mock)
│   └── layer2.pkl                 # Deep model (mock)
│
├── producer/
│   └── simulate.py                # Demo transaction generator
│
└── README.md                       # This file

Component Responsibilities

Component	Role	Technologies
API	Transaction acceptance & validation	FastAPI, HTTPBearer (JWT)
Kafka	Decoupled message queue	Kafka, Zookeeper
Feature Extraction	Enrich transactions with history	Redis (lookup), NumPy (arrays)
Layer 1 Model	Fast risk scoring	Logistic regression (mock)
Behavioral Profiling	Detect anomalies	Redis (in-memory)
Layer 2 Model	Deep fraud analysis	XGBoost (mock)
Decision Engine	Final verdict + circuit breaker	PyBreaker
Result Storage	Decision caching	Redis

---

🚀 Quick Start

Prerequisites

• Docker & Docker Compose (for Kafka, Zookeeper, Redis)
• Python 3.9+
• pip (Python package manager)

1. Clone & Setup

cd /path/to/payguard-poc
python3 -m venv venv
source venv/bin/activate          # On Windows: venv\Scripts\activate
pip install -r requirements.txt

2. Start Infrastructure

docker compose up -d

# Verify services are running:
docker compose ps

You should see:

• ✓ zookeeper - healthy
• ✓ kafka - healthy
• ✓ redis - healthy

3. Install Dependencies

All dependencies are in requirements.txt:

pip install -r requirements.txt

---

🎬 Running the Demo

The demo requires 4 terminal windows. Open each and run these commands:

Terminal 1: Verify Infrastructure

docker compose ps              # Check all services are running
redis-cli ping                 # Verify Redis (should return PONG)

Terminal 2: Start the API

cd /path/to/payguard-poc
source venv/bin/activate
uvicorn api.main:app --reload --host 0.0.0.0 --port 8000

You should see:

INFO:     Uvicorn running on http://0.0.0.0:8000

Terminal 3: Start the Consumer

cd /path/to/payguard-poc
source venv/bin/activate
cd consumer
python pipeline.py

You should see:

[INFO] PayGuard Fraud Detection Consumer - INITIALIZATION
[INFO] Waiting for transactions...

Terminal 4: Run Demo Transactions

cd /path/to/payguard-poc
source venv/bin/activate
python producer/simulate.py

Watch the consumer terminal to see transactions flowing through the pipeline!

---

🔍 Understanding the Pipeline

What Happens When You Submit a Transaction

Step 1: API Receives Request

POST /transaction
Authorization: Bearer <jwt_token>
Content-Type: application/json
{
    "amount": 5000.00,
    "merchant": "unknown_vendor.biz",
    "description": "Large suspicious purchase"
}

Output in API terminal:

[JWT AUTH] ✓ Token verified for user=user_001
[TRANSACTION INTAKE] Received from user_001: $5000.00 at unknown_vendor.biz
[KAFKA PUBLISH] ✓ Transaction abc123... published to topic 'transactions'

Step 2: Consumer Receives Message from Kafka

Output in consumer terminal:

[PIPELINE START] Transaction: abc123...
[TRANSACTION] User: user_001 | Amount: $5000.00 | Merchant: unknown_vendor.biz

Step 3: Feature Extraction

[STEP 1/6] Feature Extraction & Behavioral Profiling
[REDIS LOOKUP] Found profile for user=user_001: avg=$100.00, count=2
[FEATURE EXTRACTION] features=[5000, 100, 2, 0.5]

The system retrieves:

• amount: $5000 (current transaction)
• avg_amount: $100 (user's average)
• tx_count: 2 (total transactions from this user)
• days_since_last_tx: 0.5 (hours)

Step 4: Layer 1 Scoring (Fast Path)

[STEP 2/6] Layer 1 - Fast Model Scoring
[LAYER 1 SCORING] Fraud probability: 0.8000 → SUSPICIOUS, escalate to Layer 2

Since 0.80 ≥ 0.4, the transaction is flagged as suspicious and sent to Layer 2.

Step 5: Behavioral Analysis

[STEP 3/6] Behavioral Anomaly Detection
[ANOMALY] Amount/Avg Ratio: 50.00x
[ANOMALY] ⚠ Significant deviation detected (>3.0x)

Step 6: Layer 2 Scoring (Deep Analysis)

[STEP 4/6] Layer 2 - Deep Model Scoring
[LAYER 2 SCORING] Fraud probability: 0.95 → BLOCK (high confidence in fraud)

Since 0.95 > 0.7, this transaction is blocked.

Step 7: Decision Engine

[STEP 5/6] Final Decision Engine
[DECISION] ✗ BLOCKED (Layer 2 score 0.95 > 0.7 threshold)

Step 8: Result Storage

[STEP 6/6] Profile Update & Result Storage
[PROFILE UPDATE] user=user_001 count=2→3, avg=$100.00→$1700.00
[RESULT STORAGE] Stored decision in Redis: result:abc123... = blocked

Step 9: Pipeline Complete

[PIPELINE COMPLETE] Transaction abc123... → BLOCKED

---

🏛️ Design Patterns Demonstrated

1. Microservices Pattern

• API and consumer are separate, independently scalable services
• Communication through message queue (not direct RPC)

Location: api/main.py and consumer/pipeline.py

2. Asynchronous Event Processing

• Transactions are events in a Kafka topic
• Decouples request acceptance from processing

Location: api/main.py (Kafka producer), consumer/pipeline.py (Kafka consumer)

3. Strategy Pattern

• Different models (Layer 1 vs Layer 2) for different scenarios
• Routing logic decides which strategy to use

Location: consumer/pipeline.py (routing), consumer/models.py (implementations)

4. Circuit Breaker Pattern

• Detects failures in Layer 2 and falls back gracefully
• Prevents cascading failures

Location: consumer/decision.py

5. Decorator Pattern

• PyBreaker wraps Layer 2 calls with circuit breaker logic

Location: consumer/decision.py (circuit breaker decorator)

6. Cache-Aside Pattern

• Behavioral profiles stored in Redis
• Checked on every transaction for fast lookups

Location: consumer/features.py (Redis operations)

---

📊 Expected Demo Outcomes

Normal Transaction (Fast Path)

User_001: $75.50 at Starbucks
↓
[LAYER 1] Score 0.05 < 0.4
[DECISION] ✓ APPROVED (immediate)
Time: ~5ms (Layer 1 only)

Suspicious Transaction (Deep Analysis)

User_001: $8500 at unknown_offshore_merchant.biz
↓
[LAYER 1] Score 0.95 ≥ 0.4
[BEHAVIORAL] 85x normal amount (anomaly!)
[LAYER 2] Score 0.95 > 0.7
[DECISION] ✗ BLOCKED
Time: ~200ms (Layer 1 + Layer 2)

System Resilience (Circuit Breaker)

Simulate by intentionally breaking Layer 2:

[CIRCUIT BREAKER] ⚠ Layer 2 unavailable: timeout
[DECISION] Conservative fallback: BLOCK (safe default)

---

📚 Academic Learning Outcomes

By implementing and running PayGuard, you demonstrate understanding of:

1. Scalability: Why decoupling with message queues matters
2. Availability: How circuit breakers prevent cascade failures
3. Latency Optimization: Why tiered processing reduces average response time
4. Consistency: How Kafka provides durability even if consumers are slow
5. Real-time Processing: How to build reactive, event-driven systems
6. Testing: How clear separation of concerns enables unit testing

---

🔧 Troubleshooting

Docker Services Won't Start

docker compose logs kafka
docker compose logs redis

API Connection Error

Check if API is running:

curl http://localhost:8000/health

Kafka Consumer Can't Connect

Verify Kafka is healthy:

docker compose exec kafka kafka-broker-api-versions --bootstrap-server localhost:9092

Redis Connection Error

Test Redis:

redis-cli ping
# Should return: PONG

Python Module Not Found

Ensure virtual environment is activated:

source venv/bin/activate

---

📖 References

Design Patterns Used

• Microservices Pattern
• Event Sourcing
• Circuit Breaker Pattern
• Bulkhead Pattern

Technologies

• Apache Kafka Documentation
• FastAPI Guide
• Redis Documentation
• PyJWT

---

📝 Notes for Academic Submission

This project successfully demonstrates:

✓ Decoupled Architecture: API and consumer are independent services
✓ Asynchronous Processing: Kafka decouples ingestion from analysis
✓ Multi-Tier ML: Layer 1 (fast) and Layer 2 (accurate) with intelligent routing
✓ Resilience: Circuit breaker handles graceful degradation
✓ Observable Design: Detailed logging at each step for classroom demo
✓ Production-Ready Patterns: Uses industry-standard tools and patterns

The system is designed to be demonstrated live, with clear terminal output showing each transaction flowing through the entire pipeline. This makes it ideal for presenting to evaluators.

---

📧 Questions?

For questions about architecture decisions or implementation details, refer to the inline comments in each module. Each component is designed to be self-documenting.

---

Version: 1.0
Last Updated: 2024
Course: Advanced System Design and Implementation