Initial PayGuard fraud detection system implementation
- Complete dual-layer architecture for fraud detection - Phase 1: Docker Compose infrastructure (Kafka, Zookeeper, Redis) - Phase 2: Mock ML models (Layer 1 fast, Layer 2 deep) - Phase 3: FastAPI transaction ingestion with JWT auth - Phase 4: Kafka consumer pipeline with feature extraction - Phase 5: Test transaction producer with demo scenarios - Phase 6: Clean code structure and comprehensive documentation - Phase 7: End-to-end validation and demo readiness Demonstrates key system design patterns: - Microservices architecture with decoupled components - Asynchronous event processing via Kafka - Tiered decision-making (fast path vs deep analysis) - Resilience patterns (circuit breaker for graceful degradation) - Behavioral profiling with Redis caching Ready for academic demonstration with clear pipeline observability. Co-authored-by: Copilot <[email protected]>
This commit is contained in:
John Patrick TeruelandCopilot committed
commit
aa4dd497e2
24 files changed
+2343
No files matched your search
@@ -0,0 +1,25 @@
|
||||
# PayGuard Configuration Template
|
||||
|
||||
# API Configuration
|
||||
API_HOST=0.0.0.0
|
||||
API_PORT=8000
|
||||
|
||||
# JWT Configuration
|
||||
SECRET_KEY=payguard-demo-secret-key-change-in-production
|
||||
ALGORITHM=HS256
|
||||
|
||||
# Kafka Configuration
|
||||
KAFKA_BROKER=localhost:9092
|
||||
KAFKA_TOPIC=transactions
|
||||
KAFKA_GROUP=fraud-detection-consumer-group
|
||||
|
||||
# Redis Configuration
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
|
||||
# Circuit Breaker Configuration
|
||||
CIRCUIT_BREAKER_FAIL_MAX=3
|
||||
CIRCUIT_BREAKER_RESET_TIMEOUT=30
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
# PayGuard Quick Start Guide
|
||||
|
||||
This guide will get you up and running with the PayGuard fraud detection demo in minutes.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Docker Desktop** (includes Docker and Docker Compose)
|
||||
- **Python 3.9 or higher** (`python3 --version`)
|
||||
- **pip** (usually comes with Python)
|
||||
|
||||
## Step 1: Clone and Navigate
|
||||
|
||||
```bash
|
||||
cd /path/to/payguard-poc
|
||||
```
|
||||
|
||||
## Step 2: Create Virtual Environment
|
||||
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
```
|
||||
|
||||
## Step 3: Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
This installs:
|
||||
- fastapi (web framework)
|
||||
- uvicorn (API server)
|
||||
- python-jose (JWT authentication)
|
||||
- kafka-python (message broker client)
|
||||
- redis (cache client)
|
||||
- pybreaker (circuit breaker pattern)
|
||||
- numpy (numerical computing)
|
||||
|
||||
## Step 4: Start Infrastructure
|
||||
|
||||
In one terminal, run:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Verify services are running:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
You should see:
|
||||
- ✓ zookeeper
|
||||
- ✓ kafka
|
||||
- ✓ redis
|
||||
|
||||
## Step 5: Open 4 Terminal Windows
|
||||
|
||||
### Terminal 1: API Server
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
uvicorn api.main:app --reload --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
INFO: Uvicorn running on http://0.0.0.0:8000
|
||||
```
|
||||
|
||||
### Terminal 2: Fraud Detection Consumer
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
cd consumer
|
||||
python pipeline.py
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
[INFO] PayGuard Fraud Detection Consumer - INITIALIZATION
|
||||
[INFO] Waiting for transactions...
|
||||
```
|
||||
|
||||
### Terminal 3: Run Demo
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
python producer/simulate.py
|
||||
```
|
||||
|
||||
Watch Terminal 2 to see transactions flowing through the pipeline!
|
||||
|
||||
### Terminal 4: Query Results (Optional)
|
||||
|
||||
```bash
|
||||
redis-cli
|
||||
```
|
||||
|
||||
Then:
|
||||
```
|
||||
> KEYS result:* # See all results
|
||||
> GET result:<tx_id> # View specific result
|
||||
> HGETALL user:* # See user profiles
|
||||
```
|
||||
|
||||
## Understanding Output
|
||||
|
||||
Watch Terminal 2 (consumer) to see the pipeline in action:
|
||||
|
||||
### Normal Transaction
|
||||
```
|
||||
[STEP 1/6] Feature Extraction & Behavioral Profiling
|
||||
[FEATURE EXTRACTION] features=[75.50, 100.00, 1, 0.5]
|
||||
|
||||
[STEP 2/6] Layer 1 - Fast Model Scoring
|
||||
[LAYER 1 SCORING] Fraud probability: 0.0500 → LOW RISK, fast-track approval
|
||||
|
||||
[DECISION] ✓ APPROVED (Layer 1 score 0.0500 < 0.4 threshold)
|
||||
```
|
||||
|
||||
### Suspicious Transaction
|
||||
```
|
||||
[STEP 1/6] Feature Extraction & Behavioral Profiling
|
||||
[FEATURE EXTRACTION] features=[8500.00, 100.00, 2, 0.5]
|
||||
|
||||
[STEP 2/6] Layer 1 - Fast Model Scoring
|
||||
[LAYER 1 SCORING] Fraud probability: 0.9500 → SUSPICIOUS, escalate to Layer 2
|
||||
|
||||
[STEP 3/6] Behavioral Anomaly Detection
|
||||
[ANOMALY] Amount/Avg Ratio: 85.00x
|
||||
[ANOMALY] ⚠ Significant deviation detected (>3.0x)
|
||||
|
||||
[STEP 4/6] Layer 2 - Deep Model Scoring
|
||||
[LAYER 2 SCORING] Fraud probability: 0.9500 → BLOCK (high confidence in fraud)
|
||||
|
||||
[DECISION] ✗ BLOCKED (Layer 2 score 0.9500 > 0.7 threshold)
|
||||
```
|
||||
|
||||
## Key Concepts Demonstrated
|
||||
|
||||
### Fast Path (95% of transactions)
|
||||
- Transaction comes in → Layer 1 quick check → Approve immediately
|
||||
- Low-risk transactions bypass expensive deep analysis
|
||||
- Result: Sub-100ms latency
|
||||
|
||||
### Suspicious Path (5% of transactions)
|
||||
- Layer 1 flags as suspicious → Behavioral profiling → Layer 2 deep analysis
|
||||
- Only worthy transactions pay the cost of intensive processing
|
||||
- Higher accuracy but slower (200-500ms)
|
||||
|
||||
### Circuit Breaker
|
||||
If Layer 2 becomes unavailable:
|
||||
- System doesn't crash
|
||||
- Falls back to conservative estimate using Layer 1
|
||||
- Message: "Layer 2 unavailable, using conservative fallback"
|
||||
|
||||
### Behavioral Profiling
|
||||
- User spends $100 on average
|
||||
- Transaction for $8500 is 85x normal amount
|
||||
- → HIGH RISK → Deep analysis
|
||||
- Profile updates after each transaction
|
||||
|
||||
## Clean Up
|
||||
|
||||
When you're done:
|
||||
|
||||
```bash
|
||||
# Stop containers (keep data)
|
||||
docker compose stop
|
||||
|
||||
# Or remove everything
|
||||
docker compose down
|
||||
|
||||
# Deactivate virtual environment
|
||||
deactivate
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Docker issues
|
||||
```bash
|
||||
docker compose logs kafka # Check Kafka logs
|
||||
docker compose logs redis # Check Redis logs
|
||||
```
|
||||
|
||||
### API not responding
|
||||
```bash
|
||||
curl http://localhost:8000/health # Should return {"status": "healthy", ...}
|
||||
```
|
||||
|
||||
### Python import errors
|
||||
```bash
|
||||
# Make sure venv is activated
|
||||
which python # Should point to venv/bin/python
|
||||
pip list # Should show fastapi, redis, kafka-python, etc.
|
||||
```
|
||||
|
||||
### Kafka connection issues
|
||||
```bash
|
||||
docker compose exec kafka kafka-broker-api-versions --bootstrap-server localhost:9092
|
||||
```
|
||||
|
||||
### Redis not working
|
||||
```bash
|
||||
redis-cli ping # Should return PONG
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
Once comfortable with the basic demo:
|
||||
|
||||
1. **Read the architecture docs**: See `README_IMPLEMENTATION.md` for deep dive
|
||||
2. **Modify scenarios**: Edit `producer/simulate.py` to test custom transactions
|
||||
3. **Adjust thresholds**: Change 0.4/0.7 scores in `consumer/pipeline.py`
|
||||
4. **Study the patterns**: Each component demonstrates a design pattern
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Goal | Command |
|
||||
|------|---------|
|
||||
| Start all services | `docker compose up -d` |
|
||||
| Start API | `uvicorn api.main:app --reload` |
|
||||
| Start consumer | `cd consumer && python pipeline.py` |
|
||||
| Send test transactions | `python producer/simulate.py` |
|
||||
| Check results | `redis-cli` → `KEYS result:*` |
|
||||
| View logs | `docker compose logs -f kafka` |
|
||||
| Stop all | `docker compose down` |
|
||||
| Clean everything | `docker compose down -v` |
|
||||
|
||||
---
|
||||
|
||||
**Need help?** Check `README_IMPLEMENTATION.md` for architectural details and troubleshooting.
|
||||
@@ -0,0 +1,335 @@
|
||||
Here's the full end-to-end flowchart of the PayGuard system. The flow follows this logic:
|
||||
|
||||
Transaction initiated → passes through JWT authentication to verify the caller's identity.
|
||||
Kafka ingests the event asynchronously, decoupling ingestion from processing.
|
||||
Feature extraction preprocesses the raw transaction data for the models.
|
||||
Layer 1 (fast model) runs the first risk score — low-risk transactions are approved immediately, keeping latency minimal.
|
||||
Suspicious transactions proceed to behavioral profiling via Redis, then to Layer 2 (deep model) for heavier XGBoost/ensemble analysis.
|
||||
The final decision engine applies latency-aware thresholding to issue a final Approve or Block verdict.
|
||||
A circuit breaker / retry layer at the bottom handles fallback if the scoring engine is slow or unavailable — ensuring system uptime.
|
||||
|
||||
The dual-branch decision diamonds are the core of the "latency-aware" design — fast approvals exit early, while only truly suspicious transactions bear the cost of deep analysis.You said: Can you give me steps on how can I make this work in actual?Can you give me steps on how can I make this work in actual?Apr 21You said: Q: What's your experience level with backend/distributed systems?Q: What's your experience level with backend/distributed systems?
|
||||
A: IntermediateQ: Which language/ecosystem do you prefer?
|
||||
A: PythonApr 21Claude responded: Great — here's a practical, step-by-step build guide for PayGuard using Python, tailored for someone who already knows the basics of backend dev.Great — here's a practical, step-by-step build guide for PayGuard using Python, tailored for someone who already knows the basics of backend dev.
|
||||
|
||||
Phase 1 — Set up your local infrastructure
|
||||
Before writing any application code, get your core services running locally via Docker.
|
||||
1. Install Docker Desktop if you haven't already, then create a docker-compose.yml to spin up Kafka, Zookeeper, and Redis together:
|
||||
yamlversion: "3.8"
|
||||
services:
|
||||
zookeeper:
|
||||
image: confluentinc/cp-zookeeper:7.5.0
|
||||
environment:
|
||||
ZOOKEEPER_CLIENT_PORT: 2181
|
||||
|
||||
kafka:
|
||||
image: confluentinc/cp-kafka:7.5.0
|
||||
depends_on: [zookeeper]
|
||||
ports:
|
||||
- "9092:9092"
|
||||
environment:
|
||||
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
|
||||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
|
||||
redis:
|
||||
image: redis:7.2
|
||||
ports:
|
||||
- "6379:6379"
|
||||
Run it with docker compose up -d. This gives you Kafka on port 9092 and Redis on port 6379.
|
||||
|
||||
Phase 2 — Build the transaction ingestion API
|
||||
2. Create a FastAPI service that receives transaction requests, validates the JWT, and publishes to Kafka.
|
||||
bashpip install fastapi uvicorn python-jose kafka-python
|
||||
python# main.py
|
||||
from fastapi import FastAPI, Depends, HTTPException
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
from jose import jwt, JWTError
|
||||
from kafka import KafkaProducer
|
||||
import json, uuid, time
|
||||
|
||||
app = FastAPI()
|
||||
security = HTTPBearer()
|
||||
SECRET_KEY = "your-secret-key"
|
||||
producer = KafkaProducer(
|
||||
bootstrap_servers="localhost:9092",
|
||||
value_serializer=lambda v: json.dumps(v).encode()
|
||||
)
|
||||
|
||||
def verify_jwt(credentials: HTTPAuthorizationCredentials = Depends(security)):
|
||||
try:
|
||||
payload = jwt.decode(credentials.credentials, SECRET_KEY, algorithms=["HS256"])
|
||||
return payload
|
||||
except JWTError:
|
||||
raise HTTPException(status_code=401, detail="Invalid token")
|
||||
|
||||
@app.post("/transaction")
|
||||
def submit_transaction(data: dict, user=Depends(verify_jwt)):
|
||||
event = {
|
||||
"transaction_id": str(uuid.uuid4()),
|
||||
"user_id": user["sub"],
|
||||
"amount": data["amount"],
|
||||
"merchant": data["merchant"],
|
||||
"timestamp": time.time()
|
||||
}
|
||||
producer.send("transactions", event)
|
||||
return {"status": "received", "transaction_id": event["transaction_id"]}
|
||||
Run with uvicorn main:app --reload.
|
||||
|
||||
Phase 3 — Build the fraud scoring consumer
|
||||
3. Install ML and Redis dependencies:
|
||||
bashpip install kafka-python redis scikit-learn xgboost pandas
|
||||
4. Create a Kafka consumer that reads transactions, runs Layer 1, and conditionally runs Layer 2:
|
||||
python# consumer.py
|
||||
from kafka import KafkaConsumer
|
||||
import redis, json, pickle, numpy as np
|
||||
|
||||
consumer = KafkaConsumer(
|
||||
"transactions",
|
||||
bootstrap_servers="localhost:9092",
|
||||
value_deserializer=lambda m: json.loads(m.decode())
|
||||
)
|
||||
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
|
||||
|
||||
# Load pre-trained models (see Phase 4)
|
||||
layer1_model = pickle.load(open("layer1.pkl", "rb"))
|
||||
layer2_model = pickle.load(open("layer2.pkl", "rb"))
|
||||
|
||||
def extract_features(tx):
|
||||
history = r.hgetall(f"user:{tx['user_id']}:profile")
|
||||
avg_amount = float(history.get("avg_amount", tx["amount"]))
|
||||
tx_count = int(history.get("tx_count", 0))
|
||||
return np.array([[tx["amount"], avg_amount, tx_count]])
|
||||
|
||||
def update_profile(tx):
|
||||
key = f"user:{tx['user_id']}:profile"
|
||||
history = r.hgetall(key)
|
||||
count = int(history.get("tx_count", 0)) + 1
|
||||
avg = (float(history.get("avg_amount", 0)) * (count - 1) + tx["amount"]) / count
|
||||
r.hset(key, mapping={"tx_count": count, "avg_amount": round(avg, 2)})
|
||||
|
||||
for message in consumer:
|
||||
tx = message.value
|
||||
features = extract_features(tx)
|
||||
|
||||
score_l1 = layer1_model.predict_proba(features)[0][1] # fraud probability
|
||||
|
||||
if score_l1 < 0.4:
|
||||
decision = "approved"
|
||||
else:
|
||||
# Layer 2: deeper analysis
|
||||
score_l2 = layer2_model.predict_proba(features)[0][1]
|
||||
decision = "blocked" if score_l2 > 0.7 else "approved"
|
||||
|
||||
update_profile(tx)
|
||||
print(f"[{tx['transaction_id']}] Score L1={score_l1:.2f} → {decision.upper()}")
|
||||
# Store result in Redis or a database for downstream retrieval
|
||||
r.set(f"result:{tx['transaction_id']}", decision, ex=3600)
|
||||
|
||||
Phase 4 — Train your ML models
|
||||
5. Train Layer 1 and Layer 2 models on a fraud dataset. A great free dataset to start with is the Kaggle Credit Card Fraud dataset.
|
||||
python# train.py
|
||||
import pandas as pd, pickle
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.model_selection import train_test_split
|
||||
from xgboost import XGBClassifier
|
||||
|
||||
df = pd.read_csv("creditcard.csv")
|
||||
X = df.drop("Class", axis=1)
|
||||
y = df["Class"]
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
# Layer 1 — fast, lightweight
|
||||
l1 = LogisticRegression(max_iter=1000)
|
||||
l1.fit(X_train, y_train)
|
||||
pickle.dump(l1, open("layer1.pkl", "wb"))
|
||||
|
||||
# Layer 2 — deeper, more accurate
|
||||
l2 = XGBClassifier(n_estimators=100, use_label_encoder=False, eval_metric="logloss")
|
||||
l2.fit(X_train, y_train)
|
||||
pickle.dump(l2, open("layer2.pkl", "wb"))
|
||||
|
||||
Phase 5 — Add the circuit breaker
|
||||
6. Wrap your Layer 2 call with a circuit breaker so if it's slow or crashing, the system falls back gracefully:
|
||||
bashpip install pybreaker
|
||||
pythonimport pybreaker, time
|
||||
|
||||
breaker = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30)
|
||||
|
||||
@breaker
|
||||
def run_layer2(features):
|
||||
return layer2_model.predict_proba(features)[0][1]
|
||||
|
||||
# In your consumer loop:
|
||||
try:
|
||||
score_l2 = run_layer2(features)
|
||||
except pybreaker.CircuitBreakerError:
|
||||
print("Circuit open — defaulting to Layer 1 score")
|
||||
score_l2 = score_l1 # fallback
|
||||
|
||||
Phase 6 — Wire it all together & test
|
||||
7. End-to-end test flow:
|
||||
bash# Terminal 1 — start infrastructure
|
||||
docker compose up -d
|
||||
|
||||
# Terminal 2 — start the API
|
||||
uvicorn main:app --reload
|
||||
|
||||
# Terminal 3 — start the consumer
|
||||
python consumer.py
|
||||
|
||||
# Terminal 4 — send a test transaction
|
||||
curl -X POST http://localhost:8000/transaction \
|
||||
-H "Authorization: Bearer <your_jwt_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"amount": 9999.99, "merchant": "unknown_vendor_xyz"}'
|
||||
|
||||
Recommended build order
|
||||
Go in this sequence to avoid getting blocked:
|
||||
|
||||
Docker infra up → verify Kafka and Redis are reachable
|
||||
Train and save your .pkl models first (you need these before the consumer works)
|
||||
Build and test the consumer standalone (feed it fake JSON directly)
|
||||
Build the FastAPI ingestion layer
|
||||
Connect them end-to-end via Kafka
|
||||
Add the circuit breaker last, once the happy path works
|
||||
|
||||
|
||||
Tools summary
|
||||
ComponentToolAPI layerFastAPI + UvicornAuthpython-jose (JWT)Message brokerApache Kafka (via kafka-python)Hot-path cacheRedis (via redis-py)Layer 1 modelscikit-learn Logistic RegressionLayer 2 modelXGBoostCircuit breakerpybreakerLocal infraDocker Compose
|
||||
Want me to go deeper on any specific phase — like deploying this to the cloud, setting up proper model evaluation, or adding a results dashboard?Turn on web search in Search and tools menu. Otherwise, links provided may not be accurate or up to date.You said: is it more doable than this pipeline?is it more doable than this pipeline?Transaction →
|
||||
Kafka →
|
||||
Feature Extraction →
|
||||
Layer 1 (Fast Model) →
|
||||
├── Low Risk → Approve
|
||||
└── Suspicious →
|
||||
Behavioral Profiling (Redis) →
|
||||
Layer 2 (Deep Model) →
|
||||
Final Decision Engine (Latency-Aware) →
|
||||
Approve / Block11:07 AMClaude responded: Honestly?Honestly? No — that pipeline IS the system. What I gave you in the steps is literally how you build that exact pipeline, one piece at a time.
|
||||
But let me reframe it so it feels less overwhelming:
|
||||
|
||||
That pipeline, translated to plain components
|
||||
Pipeline StepWhat it actually isTransactionA curl or API call to FastAPIKafkaA message queue (one Docker container)Feature ExtractionA Python function — maybe 20 linesLayer 1 Fast ModelA pre-trained .pkl file you load and callRedis Behavioral ProfilingA Redis hash lookup — 2 lines of codeLayer 2 Deep ModelAnother .pkl file, only called sometimesFinal Decision EngineAn if score > 0.7 statement
|
||||
Every single box is just a function or a service call. The pipeline looks complex because it's drawn as an architecture diagram — but you're essentially writing a Python script that reads from Kafka in a loop and runs some if statements.
|
||||
|
||||
What makes it feel hard vs. what's actually hard
|
||||
Feels hard (but isn't):
|
||||
|
||||
The diagram looks like enterprise software
|
||||
"Kafka" and "Redis" sound scary
|
||||
"Dual-layer ML" sounds research-level
|
||||
|
||||
Actually the harder parts:
|
||||
|
||||
Getting good training data for the fraud models
|
||||
Tuning the thresholds (0.4 and 0.7) so you don't block legitimate users
|
||||
Keeping Kafka + Redis running reliably in production
|
||||
|
||||
|
||||
If you want a simpler starting point
|
||||
You can strip it down to this first, then add layers back in:
|
||||
Transaction → FastAPI → Python fraud check function → Approve/Block
|
||||
No Kafka, no Redis, no Layer 2. Just a single API that takes a transaction and returns a decision. Once that works, you add Kafka for async processing, then Redis for profiling, then the second model.
|
||||
The blueprint is the destination. You don't have to build all of it on day one.
|
||||
Want me to map out a week-by-week build plan so it feels more manageable?You said: But in my project, I want to demonstrate the whole process even if its not the modern approach.But in my project, I want to demonstrate the whole process even if its not the modern approach. I want to demonstrate and highlight the key idea on how this architecture works especially how each steps are executed. By doing this, I can have a slightly complicated but academically sound and accurate project in my subject Advanced System Design and Implementation.11:16 AMClaude responded: That's a solid academic framing.That's a solid academic framing. So the goal isn't production-readiness — it's demonstrating architectural correctness and showing your professor that you understand why each component exists and how they interact.
|
||||
Here's how I'd approach it:
|
||||
|
||||
Core Philosophy for Your Demo
|
||||
|
||||
Each component should be visibly doing its job. Your evaluator should be able to watch the system and trace a transaction from entry to decision, step by step.
|
||||
|
||||
This means you prioritize observability over optimization — logs, print statements, and clear separation of concerns matter more than raw performance.
|
||||
|
||||
What You Actually Need to Build
|
||||
1. The Transaction Producer (Client simulation)
|
||||
A simple Python script that generates fake transactions and sends them to Kafka. This simulates the "client" side.
|
||||
python# producer.py — simulates incoming payments
|
||||
transactions = [
|
||||
{"amount": 50, "merchant": "Jollibee", "user_id": "u001"},
|
||||
{"amount": 85000, "merchant": "unknown_offshore", "user_id": "u001"},
|
||||
{"amount": 120, "merchant": "SM Supermarket", "user_id": "u002"},
|
||||
]
|
||||
This lets you control the demo — you can deliberately send a suspicious transaction and show the class exactly what happens.
|
||||
|
||||
2. The Ingestion API (FastAPI + JWT)
|
||||
A single endpoint that:
|
||||
|
||||
Validates the JWT token (proves you understand auth)
|
||||
Publishes the transaction to Kafka
|
||||
Returns immediately — this is where you explain decoupling
|
||||
|
||||
The key academic point here is: "The API does not wait for fraud analysis. It just acknowledges receipt." That's the architectural insight.
|
||||
|
||||
3. The Kafka Topic
|
||||
This is your architectural centerpiece. It's what separates ingestion from processing. For demo purposes, even one topic with one partition is enough.
|
||||
The point you highlight: "Even if the fraud engine is slow or down, transactions are not lost — they wait in the queue."
|
||||
|
||||
4. The Consumer / Processing Pipeline
|
||||
This is where the architecture really shows. Structure it as clearly separated functions, not one blob:
|
||||
pythondef step1_extract_features(tx): # clearly named
|
||||
print(f"[FEATURE EXTRACTION] Processing tx {tx['id']}")
|
||||
...
|
||||
|
||||
def step2_layer1_score(features): # fast model
|
||||
print(f"[LAYER 1] Running logistic regression...")
|
||||
...
|
||||
|
||||
def step3_behavioral_profile(tx): # Redis lookup
|
||||
print(f"[REDIS] Fetching user {tx['user_id']} history...")
|
||||
...
|
||||
|
||||
def step4_layer2_score(features): # deep model
|
||||
print(f"[LAYER 2] Running XGBoost analysis...")
|
||||
...
|
||||
|
||||
def step5_decision(score): # final engine
|
||||
print(f"[DECISION ENGINE] Final score: {score:.2f}")
|
||||
...
|
||||
When you run a live demo, your terminal will visually walk through each step — that's what makes it academically convincing.
|
||||
|
||||
5. Redis Behavioral Profiling
|
||||
For each user, store their transaction history in Redis. This proves you understand the "hot path" concept — fast in-memory lookups instead of hitting a database.
|
||||
user:u001:profile → { avg_amount: 175.0, tx_count: 12 }
|
||||
A transaction of 85,000 from a user whose average is 175 is a clear anomaly. That's your demo moment.
|
||||
|
||||
6. The Circuit Breaker
|
||||
Even if it never actually trips during the demo, having it in the code with a clear comment and a simulated failure test shows architectural maturity.
|
||||
python# Demonstrate: kill Layer 2 service, show system falls back gracefully
|
||||
|
||||
How to Structure Your Demonstration
|
||||
Demo MomentWhat you showSend a normal transactionTrace it through L1 → Approved, fast pathSend a suspicious transactionWatch it go through L1 → L2 → Decision EngineShow Redis before & afterUser profile updates in real timeKill the scorer, send a transactionCircuit breaker activates, fallback kicks inShow Kafka consumer lagProve the queue held the message while scorer was down
|
||||
|
||||
What Makes This Academically Sound
|
||||
You're demonstrating four core Advanced System Design concepts in one project:
|
||||
|
||||
Decoupled microservices — API and scorer are independent processes
|
||||
Asynchronous message passing — Kafka as the backbone
|
||||
Tiered processing — dual-layer model with conditional routing
|
||||
Resilience patterns — circuit breaker and retry logic
|
||||
|
||||
Each one maps directly to topics covered in systems design literature. You can cite each component against a design pattern in your write-up.
|
||||
|
||||
Suggested Project Structure
|
||||
payguard/
|
||||
├── docker-compose.yml # Kafka + Redis
|
||||
├── producer/
|
||||
│ └── simulate.py # sends test transactions
|
||||
├── api/
|
||||
│ └── main.py # FastAPI + JWT
|
||||
├── consumer/
|
||||
│ ├── pipeline.py # orchestrates all steps
|
||||
│ ├── features.py # feature extraction
|
||||
│ ├── layer1.py # fast model
|
||||
│ ├── layer2.py # deep model
|
||||
│ ├── profiling.py # Redis behavioral lookup
|
||||
│ └── decision.py # final engine + circuit breaker
|
||||
├── models/
|
||||
│ ├── train.py # training script
|
||||
│ ├── layer1.pkl
|
||||
│ └── layer2.pkl
|
||||
└── README.md # architecture explanation
|
||||
This structure alone communicates that you understand separation of concerns — each file has exactly one responsibility.
|
||||
|
||||
The whole thing can run on your laptop. You demo it live, walk through the terminal output step by step, and your professor can see every architectural decision executing in real time.
|
||||
Want me to write out the full code for any specific component?
|
||||
@@ -0,0 +1,472 @@
|
||||
# PayGuard Fraud Detection System
|
||||
|
||||
A comprehensive, academically-rigorous demonstration of a dual-layer fraud detection system for the Advanced System Design course. This project showcases fundamental microservices architecture patterns: decoupling, asynchronous processing, tiered computing, and resilience patterns.
|
||||
|
||||
## 📋 Table of Contents
|
||||
|
||||
- [Architecture Overview](#architecture-overview)
|
||||
- [Key Design Principles](#key-design-principles)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Running the Demo](#running-the-demo)
|
||||
- [Understanding the Pipeline](#understanding-the-pipeline)
|
||||
- [Design Patterns Demonstrated](#design-patterns-demonstrated)
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
The PayGuard system processes transactions through a sophisticated multi-stage pipeline:
|
||||
|
||||
```
|
||||
Transaction Request
|
||||
↓
|
||||
[JWT Authentication] ← Verify caller identity
|
||||
↓
|
||||
[Kafka Ingestion] ← Async, decoupled message queue
|
||||
↓
|
||||
[Feature Extraction] ← Enrich with behavioral history
|
||||
↓
|
||||
[Layer 1 - Fast Model] ← Logistic regression (milliseconds)
|
||||
├─ Score < 0.4 → ✓ APPROVE (fast path)
|
||||
└─ Score ≥ 0.4 → Continue to Layer 2
|
||||
↓
|
||||
[Behavioral Profiling] ← Redis lookup for anomalies
|
||||
↓
|
||||
[Layer 2 - Deep Model] ← XGBoost/ensemble (heavier analysis)
|
||||
├─ Score > 0.7 → ✗ BLOCK
|
||||
└─ Score ≤ 0.7 → ✓ APPROVE
|
||||
↓
|
||||
[Circuit Breaker] ← Graceful fallback if Layer 2 fails
|
||||
↓
|
||||
[Final Decision Engine] ← Latency-aware thresholding
|
||||
↓
|
||||
[Result Storage] ← Redis (1-hour TTL)
|
||||
```
|
||||
|
||||
**Key Innovation**: The dual-branch decision gates allow >95% of transactions to be approved with a fast, lightweight model, while only suspicious transactions pay the cost of deep analysis.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Key Design Principles
|
||||
|
||||
### 1. **Microservices & Decoupling**
|
||||
The API layer (ingestion) is completely independent from the fraud detection layer (consumer). They communicate only through Kafka messages. This means:
|
||||
- The API can scale independently from the scorer
|
||||
- If the scorer is overloaded, transactions don't get lost—they wait in Kafka
|
||||
- Services can be deployed and updated independently
|
||||
|
||||
**Code Location**: `api/main.py` (API) and `consumer/pipeline.py` (Consumer)
|
||||
|
||||
### 2. **Asynchronous Message Passing**
|
||||
Kafka is the backbone. The API doesn't wait for fraud analysis; it immediately acknowledges receipt and returns to the client. The consumer processes transactions at its own pace.
|
||||
|
||||
**Academic Benefit**: Demonstrates the CAP theorem trade-off (consistency vs. availability)
|
||||
|
||||
**Code Location**: `api/main.py` (producer) and `consumer/pipeline.py` (consumer)
|
||||
|
||||
### 3. **Tiered Processing (Adaptive Routing)**
|
||||
The system routes transactions dynamically:
|
||||
- **95% of transactions**: Handled by Layer 1 only (fast path)
|
||||
- **5% suspicious transactions**: Escalated to Layer 2 (deep analysis)
|
||||
|
||||
This optimizes the latency-accuracy trade-off: most users get instant approval, but fraud is caught with higher confidence.
|
||||
|
||||
**Code Location**: `consumer/pipeline.py` (routing logic), `consumer/models.py` (scoring)
|
||||
|
||||
### 4. **Resilience Patterns**
|
||||
The circuit breaker pattern ensures graceful degradation:
|
||||
- If Layer 2 becomes slow or unavailable, the system falls back to conservative Layer 1 estimates
|
||||
- This prevents cascading failures and ensures uptime even during outages
|
||||
|
||||
**Code Location**: `consumer/decision.py` (circuit breaker)
|
||||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
payguard-poc/
|
||||
├── docker-compose.yml # Infrastructure: Kafka, Zookeeper, Redis
|
||||
├── requirements.txt # Python dependencies
|
||||
├── .env.example # Configuration template
|
||||
│
|
||||
├── api/
|
||||
│ └── main.py # FastAPI transaction ingestion layer
|
||||
│ ├── JWT authentication
|
||||
│ ├── Kafka producer
|
||||
│ └── Request validation
|
||||
│
|
||||
├── consumer/
|
||||
│ ├── __init__.py
|
||||
│ ├── pipeline.py # Main orchestrator (steps 1-6)
|
||||
│ ├── features.py # Feature extraction & profiling
|
||||
│ ├── models.py # Layer 1 & Layer 2 scoring
|
||||
│ └── decision.py # Decision engine & circuit breaker
|
||||
│
|
||||
├── models/
|
||||
│ ├── layer1.pkl # Fast model (mock)
|
||||
│ └── layer2.pkl # Deep model (mock)
|
||||
│
|
||||
├── producer/
|
||||
│ └── simulate.py # Demo transaction generator
|
||||
│
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
### Component Responsibilities
|
||||
|
||||
| Component | Role | Technologies |
|
||||
|-----------|------|--------------|
|
||||
| **API** | Transaction acceptance & validation | FastAPI, HTTPBearer (JWT) |
|
||||
| **Kafka** | Decoupled message queue | Kafka, Zookeeper |
|
||||
| **Feature Extraction** | Enrich transactions with history | Redis (lookup), NumPy (arrays) |
|
||||
| **Layer 1 Model** | Fast risk scoring | Logistic regression (mock) |
|
||||
| **Behavioral Profiling** | Detect anomalies | Redis (in-memory) |
|
||||
| **Layer 2 Model** | Deep fraud analysis | XGBoost (mock) |
|
||||
| **Decision Engine** | Final verdict + circuit breaker | PyBreaker |
|
||||
| **Result Storage** | Decision caching | Redis |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Prerequisites
|
||||
- **Docker & Docker Compose** (for Kafka, Zookeeper, Redis)
|
||||
- **Python 3.9+**
|
||||
- **pip** (Python package manager)
|
||||
|
||||
### 1. Clone & Setup
|
||||
|
||||
```bash
|
||||
cd /path/to/payguard-poc
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Start Infrastructure
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
|
||||
# Verify services are running:
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
You should see:
|
||||
- ✓ zookeeper - healthy
|
||||
- ✓ kafka - healthy
|
||||
- ✓ redis - healthy
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
All dependencies are in `requirements.txt`:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎬 Running the Demo
|
||||
|
||||
The demo requires **4 terminal windows**. Open each and run these commands:
|
||||
|
||||
### Terminal 1: Verify Infrastructure
|
||||
```bash
|
||||
docker compose ps # Check all services are running
|
||||
redis-cli ping # Verify Redis (should return PONG)
|
||||
```
|
||||
|
||||
### Terminal 2: Start the API
|
||||
|
||||
```bash
|
||||
cd /path/to/payguard-poc
|
||||
source venv/bin/activate
|
||||
uvicorn api.main:app --reload --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
You should see:
|
||||
```
|
||||
INFO: Uvicorn running on http://0.0.0.0:8000
|
||||
```
|
||||
|
||||
### Terminal 3: Start the Consumer
|
||||
|
||||
```bash
|
||||
cd /path/to/payguard-poc
|
||||
source venv/bin/activate
|
||||
cd consumer
|
||||
python pipeline.py
|
||||
```
|
||||
|
||||
You should see:
|
||||
```
|
||||
[INFO] PayGuard Fraud Detection Consumer - INITIALIZATION
|
||||
[INFO] Waiting for transactions...
|
||||
```
|
||||
|
||||
### Terminal 4: Run Demo Transactions
|
||||
|
||||
```bash
|
||||
cd /path/to/payguard-poc
|
||||
source venv/bin/activate
|
||||
python producer/simulate.py
|
||||
```
|
||||
|
||||
Watch the consumer terminal to see transactions flowing through the pipeline!
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Understanding the Pipeline
|
||||
|
||||
### What Happens When You Submit a Transaction
|
||||
|
||||
#### Step 1: API Receives Request
|
||||
```
|
||||
POST /transaction
|
||||
Authorization: Bearer <jwt_token>
|
||||
Content-Type: application/json
|
||||
{
|
||||
"amount": 5000.00,
|
||||
"merchant": "unknown_vendor.biz",
|
||||
"description": "Large suspicious purchase"
|
||||
}
|
||||
```
|
||||
|
||||
**Output in API terminal**:
|
||||
```
|
||||
[JWT AUTH] ✓ Token verified for user=user_001
|
||||
[TRANSACTION INTAKE] Received from user_001: $5000.00 at unknown_vendor.biz
|
||||
[KAFKA PUBLISH] ✓ Transaction abc123... published to topic 'transactions'
|
||||
```
|
||||
|
||||
#### Step 2: Consumer Receives Message from Kafka
|
||||
|
||||
**Output in consumer terminal**:
|
||||
```
|
||||
[PIPELINE START] Transaction: abc123...
|
||||
[TRANSACTION] User: user_001 | Amount: $5000.00 | Merchant: unknown_vendor.biz
|
||||
```
|
||||
|
||||
#### Step 3: Feature Extraction
|
||||
|
||||
```
|
||||
[STEP 1/6] Feature Extraction & Behavioral Profiling
|
||||
[REDIS LOOKUP] Found profile for user=user_001: avg=$100.00, count=2
|
||||
[FEATURE EXTRACTION] features=[5000, 100, 2, 0.5]
|
||||
```
|
||||
|
||||
The system retrieves:
|
||||
- `amount`: $5000 (current transaction)
|
||||
- `avg_amount`: $100 (user's average)
|
||||
- `tx_count`: 2 (total transactions from this user)
|
||||
- `days_since_last_tx`: 0.5 (hours)
|
||||
|
||||
#### Step 4: Layer 1 Scoring (Fast Path)
|
||||
|
||||
```
|
||||
[STEP 2/6] Layer 1 - Fast Model Scoring
|
||||
[LAYER 1 SCORING] Fraud probability: 0.8000 → SUSPICIOUS, escalate to Layer 2
|
||||
```
|
||||
|
||||
Since 0.80 ≥ 0.4, the transaction is flagged as suspicious and sent to Layer 2.
|
||||
|
||||
#### Step 5: Behavioral Analysis
|
||||
|
||||
```
|
||||
[STEP 3/6] Behavioral Anomaly Detection
|
||||
[ANOMALY] Amount/Avg Ratio: 50.00x
|
||||
[ANOMALY] ⚠ Significant deviation detected (>3.0x)
|
||||
```
|
||||
|
||||
#### Step 6: Layer 2 Scoring (Deep Analysis)
|
||||
|
||||
```
|
||||
[STEP 4/6] Layer 2 - Deep Model Scoring
|
||||
[LAYER 2 SCORING] Fraud probability: 0.95 → BLOCK (high confidence in fraud)
|
||||
```
|
||||
|
||||
Since 0.95 > 0.7, this transaction is blocked.
|
||||
|
||||
#### Step 7: Decision Engine
|
||||
|
||||
```
|
||||
[STEP 5/6] Final Decision Engine
|
||||
[DECISION] ✗ BLOCKED (Layer 2 score 0.95 > 0.7 threshold)
|
||||
```
|
||||
|
||||
#### Step 8: Result Storage
|
||||
|
||||
```
|
||||
[STEP 6/6] Profile Update & Result Storage
|
||||
[PROFILE UPDATE] user=user_001 count=2→3, avg=$100.00→$1700.00
|
||||
[RESULT STORAGE] Stored decision in Redis: result:abc123... = blocked
|
||||
```
|
||||
|
||||
#### Step 9: Pipeline Complete
|
||||
|
||||
```
|
||||
[PIPELINE COMPLETE] Transaction abc123... → BLOCKED
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏛️ Design Patterns Demonstrated
|
||||
|
||||
### 1. **Microservices Pattern**
|
||||
- API and consumer are separate, independently scalable services
|
||||
- Communication through message queue (not direct RPC)
|
||||
|
||||
**Location**: `api/main.py` and `consumer/pipeline.py`
|
||||
|
||||
### 2. **Asynchronous Event Processing**
|
||||
- Transactions are events in a Kafka topic
|
||||
- Decouples request acceptance from processing
|
||||
|
||||
**Location**: `api/main.py` (Kafka producer), `consumer/pipeline.py` (Kafka consumer)
|
||||
|
||||
### 3. **Strategy Pattern**
|
||||
- Different models (Layer 1 vs Layer 2) for different scenarios
|
||||
- Routing logic decides which strategy to use
|
||||
|
||||
**Location**: `consumer/pipeline.py` (routing), `consumer/models.py` (implementations)
|
||||
|
||||
### 4. **Circuit Breaker Pattern**
|
||||
- Detects failures in Layer 2 and falls back gracefully
|
||||
- Prevents cascading failures
|
||||
|
||||
**Location**: `consumer/decision.py`
|
||||
|
||||
### 5. **Decorator Pattern**
|
||||
- PyBreaker wraps Layer 2 calls with circuit breaker logic
|
||||
|
||||
**Location**: `consumer/decision.py` (circuit breaker decorator)
|
||||
|
||||
### 6. **Cache-Aside Pattern**
|
||||
- Behavioral profiles stored in Redis
|
||||
- Checked on every transaction for fast lookups
|
||||
|
||||
**Location**: `consumer/features.py` (Redis operations)
|
||||
|
||||
---
|
||||
|
||||
## 📊 Expected Demo Outcomes
|
||||
|
||||
### Normal Transaction (Fast Path)
|
||||
```
|
||||
User_001: $75.50 at Starbucks
|
||||
↓
|
||||
[LAYER 1] Score 0.05 < 0.4
|
||||
[DECISION] ✓ APPROVED (immediate)
|
||||
Time: ~5ms (Layer 1 only)
|
||||
```
|
||||
|
||||
### Suspicious Transaction (Deep Analysis)
|
||||
```
|
||||
User_001: $8500 at unknown_offshore_merchant.biz
|
||||
↓
|
||||
[LAYER 1] Score 0.95 ≥ 0.4
|
||||
[BEHAVIORAL] 85x normal amount (anomaly!)
|
||||
[LAYER 2] Score 0.95 > 0.7
|
||||
[DECISION] ✗ BLOCKED
|
||||
Time: ~200ms (Layer 1 + Layer 2)
|
||||
```
|
||||
|
||||
### System Resilience (Circuit Breaker)
|
||||
Simulate by intentionally breaking Layer 2:
|
||||
```
|
||||
[CIRCUIT BREAKER] ⚠ Layer 2 unavailable: timeout
|
||||
[DECISION] Conservative fallback: BLOCK (safe default)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Academic Learning Outcomes
|
||||
|
||||
By implementing and running PayGuard, you demonstrate understanding of:
|
||||
|
||||
1. **Scalability**: Why decoupling with message queues matters
|
||||
2. **Availability**: How circuit breakers prevent cascade failures
|
||||
3. **Latency Optimization**: Why tiered processing reduces average response time
|
||||
4. **Consistency**: How Kafka provides durability even if consumers are slow
|
||||
5. **Real-time Processing**: How to build reactive, event-driven systems
|
||||
6. **Testing**: How clear separation of concerns enables unit testing
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Docker Services Won't Start
|
||||
```bash
|
||||
docker compose logs kafka
|
||||
docker compose logs redis
|
||||
```
|
||||
|
||||
### API Connection Error
|
||||
Check if API is running:
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
### Kafka Consumer Can't Connect
|
||||
Verify Kafka is healthy:
|
||||
```bash
|
||||
docker compose exec kafka kafka-broker-api-versions --bootstrap-server localhost:9092
|
||||
```
|
||||
|
||||
### Redis Connection Error
|
||||
Test Redis:
|
||||
```bash
|
||||
redis-cli ping
|
||||
# Should return: PONG
|
||||
```
|
||||
|
||||
### Python Module Not Found
|
||||
Ensure virtual environment is activated:
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📖 References
|
||||
|
||||
### Design Patterns Used
|
||||
- [Microservices Pattern](https://microservices.io/)
|
||||
- [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html)
|
||||
- [Circuit Breaker Pattern](https://martinfowler.com/bliki/CircuitBreaker.html)
|
||||
- [Bulkhead Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/bulkhead)
|
||||
|
||||
### Technologies
|
||||
- [Apache Kafka Documentation](https://kafka.apache.org/documentation/)
|
||||
- [FastAPI Guide](https://fastapi.tiangolo.com/)
|
||||
- [Redis Documentation](https://redis.io/documentation)
|
||||
- [PyJWT](https://pyjwt.readthedocs.io/)
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes for Academic Submission
|
||||
|
||||
This project successfully demonstrates:
|
||||
|
||||
✓ **Decoupled Architecture**: API and consumer are independent services
|
||||
✓ **Asynchronous Processing**: Kafka decouples ingestion from analysis
|
||||
✓ **Multi-Tier ML**: Layer 1 (fast) and Layer 2 (accurate) with intelligent routing
|
||||
✓ **Resilience**: Circuit breaker handles graceful degradation
|
||||
✓ **Observable Design**: Detailed logging at each step for classroom demo
|
||||
✓ **Production-Ready Patterns**: Uses industry-standard tools and patterns
|
||||
|
||||
The system is designed to be demonstrated live, with clear terminal output showing each transaction flowing through the entire pipeline. This makes it ideal for presenting to evaluators.
|
||||
|
||||
---
|
||||
|
||||
## 📧 Questions?
|
||||
|
||||
For questions about architecture decisions or implementation details, refer to the inline comments in each module. Each component is designed to be self-documenting.
|
||||
|
||||
---
|
||||
|
||||
**Version**: 1.0
|
||||
**Last Updated**: 2024
|
||||
**Course**: Advanced System Design and Implementation
|
||||
Binary file not shown.
+216
@@ -0,0 +1,216 @@
|
||||
"""
|
||||
PayGuard Transaction Ingestion API
|
||||
|
||||
Responsibilities:
|
||||
- Authenticate incoming requests using JWT
|
||||
- Validate transaction data structure
|
||||
- Publish transactions to Kafka asynchronously
|
||||
- Return immediately without waiting for fraud analysis (demonstrates decoupling)
|
||||
|
||||
Academic Point:
|
||||
The API does not process transactions directly. It simply validates identity and
|
||||
hands off responsibility to the message queue. This separation of concerns is
|
||||
fundamental to scalable microservices architecture.
|
||||
"""
|
||||
|
||||
from fastapi import FastAPI, Depends, HTTPException, status
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
from pydantic import BaseModel
|
||||
from jose import jwt, JWTError
|
||||
from kafka import KafkaProducer
|
||||
import json
|
||||
import uuid
|
||||
import time
|
||||
import logging
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# CONFIGURATION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
# Secret key for JWT signing (for demo purposes only)
|
||||
SECRET_KEY = "payguard-demo-secret-key-change-in-production"
|
||||
ALGORITHM = "HS256"
|
||||
|
||||
# Kafka bootstrap server
|
||||
KAFKA_BROKER = "localhost:9092"
|
||||
KAFKA_TOPIC = "transactions"
|
||||
|
||||
# Logging setup
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s [%(levelname)s] %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# MODELS
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TransactionRequest(BaseModel):
|
||||
"""Request body for transaction submission"""
|
||||
amount: float
|
||||
merchant: str
|
||||
description: str = ""
|
||||
|
||||
class TransactionResponse(BaseModel):
|
||||
"""Response after transaction is accepted"""
|
||||
status: str
|
||||
transaction_id: str
|
||||
message: str
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# KAFKA PRODUCER
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
kafka_producer = KafkaProducer(
|
||||
bootstrap_servers=KAFKA_BROKER,
|
||||
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
|
||||
acks='all',
|
||||
retries=3
|
||||
)
|
||||
|
||||
logger.info(f"[KAFKA PRODUCER] Initialized, broker={KAFKA_BROKER}, topic={KAFKA_TOPIC}")
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# AUTHENTICATION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
security = HTTPBearer()
|
||||
|
||||
def verify_jwt(credentials: HTTPAuthorizationCredentials = Depends(security)) -> dict:
|
||||
"""
|
||||
Verify JWT token and extract user information.
|
||||
|
||||
Academic Point: In a real system, this would validate against a user database
|
||||
and use proper token refresh mechanisms. For demo, we accept any valid JWT.
|
||||
"""
|
||||
try:
|
||||
token = credentials.credentials
|
||||
payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
|
||||
user_id = payload.get("sub")
|
||||
|
||||
if user_id is None:
|
||||
logger.warning("[JWT AUTH] Token missing 'sub' claim")
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token: missing user ID"
|
||||
)
|
||||
|
||||
logger.info(f"[JWT AUTH] ✓ Token verified for user={user_id}")
|
||||
return {"user_id": user_id}
|
||||
|
||||
except JWTError as e:
|
||||
logger.warning(f"[JWT AUTH] ✗ Token validation failed: {str(e)}")
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid or expired token"
|
||||
)
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# API APPLICATION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
app = FastAPI(
|
||||
title="PayGuard Fraud Detection API",
|
||||
description="Transaction ingestion layer for fraud detection pipeline",
|
||||
version="1.0.0"
|
||||
)
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
"""Health check endpoint"""
|
||||
return {"status": "healthy", "service": "payguard-api"}
|
||||
|
||||
@app.post("/transaction", response_model=TransactionResponse)
|
||||
async def submit_transaction(
|
||||
request: TransactionRequest,
|
||||
user: dict = Depends(verify_jwt)
|
||||
) -> TransactionResponse:
|
||||
"""
|
||||
Accept a transaction and publish it to Kafka for async processing.
|
||||
|
||||
Returns immediately without waiting for fraud analysis.
|
||||
This is a key architectural pattern: decoupling ingestion from processing.
|
||||
"""
|
||||
|
||||
user_id = user["user_id"]
|
||||
transaction_id = str(uuid.uuid4())
|
||||
timestamp = time.time()
|
||||
|
||||
# Build transaction event
|
||||
event = {
|
||||
"transaction_id": transaction_id,
|
||||
"user_id": user_id,
|
||||
"amount": request.amount,
|
||||
"merchant": request.merchant,
|
||||
"description": request.description,
|
||||
"timestamp": timestamp
|
||||
}
|
||||
|
||||
logger.info(
|
||||
f"[TRANSACTION INTAKE] Received from {user_id}: "
|
||||
f"${request.amount:.2f} at {request.merchant}"
|
||||
)
|
||||
|
||||
# Publish to Kafka
|
||||
try:
|
||||
future = kafka_producer.send(KAFKA_TOPIC, event)
|
||||
future.get(timeout=10) # Wait for confirmation
|
||||
|
||||
logger.info(
|
||||
f"[KAFKA PUBLISH] ✓ Transaction {transaction_id} "
|
||||
f"published to topic '{KAFKA_TOPIC}'"
|
||||
)
|
||||
|
||||
return TransactionResponse(
|
||||
status="accepted",
|
||||
transaction_id=transaction_id,
|
||||
message="Transaction accepted. Fraud analysis in progress."
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[KAFKA PUBLISH] ✗ Failed to publish: {str(e)}")
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="Message broker unavailable. Please try again."
|
||||
)
|
||||
|
||||
@app.post("/token")
|
||||
async def get_demo_token(user_id: str):
|
||||
"""
|
||||
Generate a demo JWT token for testing.
|
||||
|
||||
In production, this would validate credentials against a user database.
|
||||
For demo purposes, accept any user_id.
|
||||
"""
|
||||
token = jwt.encode({"sub": user_id}, SECRET_KEY, algorithm=ALGORITHM)
|
||||
logger.info(f"[JWT GENERATION] Generated token for user={user_id}")
|
||||
return {"access_token": token, "token_type": "bearer"}
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# STARTUP / SHUTDOWN
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup_event():
|
||||
logger.info("="*80)
|
||||
logger.info("PayGuard Transaction Ingestion API - STARTUP")
|
||||
logger.info("="*80)
|
||||
logger.info(f"Kafka Broker: {KAFKA_BROKER}")
|
||||
logger.info(f"Topic: {KAFKA_TOPIC}")
|
||||
logger.info("Listening for authenticated transaction requests...")
|
||||
logger.info("="*80)
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def shutdown_event():
|
||||
logger.info("Shutting down API...")
|
||||
kafka_producer.close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(
|
||||
app,
|
||||
host="0.0.0.0",
|
||||
port=8000,
|
||||
log_level="info"
|
||||
)
|
||||
Whitespace-only changes.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,131 @@
|
||||
"""
|
||||
Decision Engine & Circuit Breaker
|
||||
|
||||
Responsibilities:
|
||||
- Combine model scores into final decision
|
||||
- Apply circuit breaker pattern for graceful degradation
|
||||
- Store decision results in Redis
|
||||
|
||||
Academic Point:
|
||||
The circuit breaker is a critical resilience pattern. If the deep analysis (Layer 2)
|
||||
becomes slow or unavailable, the system can gracefully degrade to using only Layer 1
|
||||
scores, rather than failing entirely. This ensures uptime even during partial outages.
|
||||
"""
|
||||
|
||||
import redis
|
||||
import logging
|
||||
from pybreaker import CircuitBreaker
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class DecisionEngine:
|
||||
"""Makes final approve/block decisions with resilience patterns"""
|
||||
|
||||
def __init__(self, redis_host: str = "localhost", redis_port: int = 6379):
|
||||
self.redis_client = redis.Redis(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
decode_responses=True
|
||||
)
|
||||
|
||||
# Circuit breaker for Layer 2 calls
|
||||
# Fails after 3 consecutive errors, resets after 30 seconds
|
||||
self.layer2_breaker = CircuitBreaker(
|
||||
fail_max=3,
|
||||
reset_timeout=30,
|
||||
listeners=[] # Could add monitoring here
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"[DECISION ENGINE] Initialized with circuit breaker: "
|
||||
"fail_max=3, reset_timeout=30s"
|
||||
)
|
||||
|
||||
def make_decision(self,
|
||||
transaction_id: str,
|
||||
layer1_score: float,
|
||||
layer2_scorer=None,
|
||||
features=None) -> str:
|
||||
"""
|
||||
Make final decision: "approved" or "blocked"
|
||||
|
||||
Logic:
|
||||
1. If Layer 1 score < 0.4: Approve immediately (fast path)
|
||||
2. If Layer 1 score >= 0.4: Try Layer 2 with circuit breaker
|
||||
- If Layer 2 available: Use Layer 2 score (threshold: 0.7)
|
||||
- If Layer 2 fails: Fall back to conservative Layer 1 estimate
|
||||
"""
|
||||
|
||||
logger.info(f"\n[DECISION ENGINE] Processing transaction {transaction_id}")
|
||||
|
||||
# Fast path: low risk
|
||||
if layer1_score < 0.4:
|
||||
decision = "approved"
|
||||
logger.info(
|
||||
f"[DECISION] ✓ APPROVED (Layer 1 score {layer1_score:.4f} < 0.4 threshold)"
|
||||
)
|
||||
return decision
|
||||
|
||||
# Suspicious: try Layer 2
|
||||
logger.info(f"[DECISION] Suspicious score from Layer 1 ({layer1_score:.4f})")
|
||||
logger.info("[DECISION] Attempting Layer 2 deep analysis...")
|
||||
|
||||
if layer2_scorer is None or features is None:
|
||||
logger.warning("[DECISION] Layer 2 scorer not provided, using Layer 1 only")
|
||||
decision = "approved" if layer1_score < 0.5 else "blocked"
|
||||
return decision
|
||||
|
||||
# Try Layer 2 with circuit breaker
|
||||
try:
|
||||
@self.layer2_breaker
|
||||
def run_layer2():
|
||||
return layer2_scorer.score_layer2(features)
|
||||
|
||||
layer2_score = run_layer2()
|
||||
|
||||
# Layer 2 threshold
|
||||
if layer2_score > 0.7:
|
||||
decision = "blocked"
|
||||
logger.info(
|
||||
f"[DECISION] ✗ BLOCKED (Layer 2 score {layer2_score:.4f} > 0.7 threshold)"
|
||||
)
|
||||
else:
|
||||
decision = "approved"
|
||||
logger.info(
|
||||
f"[DECISION] ✓ APPROVED (Layer 2 score {layer2_score:.4f} <= 0.7 threshold)"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
# Circuit breaker is open or Layer 2 failed
|
||||
logger.warning(f"[CIRCUIT BREAKER] ⚠ Layer 2 unavailable: {str(e)}")
|
||||
logger.warning("[CIRCUIT BREAKER] Falling back to conservative Layer 1 estimate")
|
||||
|
||||
# Conservative fallback: if Layer 1 is uncertain, block
|
||||
if layer1_score > 0.6:
|
||||
decision = "blocked"
|
||||
logger.info(
|
||||
f"[DECISION] ✗ BLOCKED (Layer 2 failed, Layer 1 score {layer1_score:.4f} too high)"
|
||||
)
|
||||
else:
|
||||
decision = "approved"
|
||||
logger.info(
|
||||
f"[DECISION] ✓ APPROVED (Layer 2 failed, Layer 1 score {layer1_score:.4f} acceptable)"
|
||||
)
|
||||
|
||||
return decision
|
||||
|
||||
def store_result(self, transaction_id: str, decision: str,
|
||||
layer1_score: float, layer2_score: float = None) -> None:
|
||||
"""Store decision result in Redis for retrieval"""
|
||||
key = f"result:{transaction_id}"
|
||||
value = {
|
||||
"decision": decision,
|
||||
"layer1_score": round(layer1_score, 4),
|
||||
}
|
||||
if layer2_score is not None:
|
||||
value["layer2_score"] = round(layer2_score, 4)
|
||||
|
||||
# Store with 1-hour TTL
|
||||
self.redis_client.setex(key, 3600, str(value))
|
||||
|
||||
logger.info(f"[RESULT STORAGE] Stored decision in Redis: {key} = {decision}")
|
||||
@@ -0,0 +1,133 @@
|
||||
"""
|
||||
Feature Extraction Module
|
||||
|
||||
Responsibilities:
|
||||
- Extract raw transaction data into ML-friendly features
|
||||
- Query Redis for user behavioral history
|
||||
- Construct feature vectors for ML models
|
||||
|
||||
Academic Point:
|
||||
Feature engineering is the bridge between raw business data and ML models.
|
||||
This module demonstrates how to enrich simple transaction data with historical
|
||||
context to improve model decision quality.
|
||||
"""
|
||||
|
||||
import redis
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Dict, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class FeatureExtractor:
|
||||
"""Extracts and engineers features from transactions"""
|
||||
|
||||
def __init__(self, redis_host: str = "localhost", redis_port: int = 6379):
|
||||
self.redis_client = redis.Redis(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
decode_responses=True
|
||||
)
|
||||
logger.info(f"[FEATURE EXTRACTOR] Initialized, redis={redis_host}:{redis_port}")
|
||||
|
||||
def get_user_profile(self, user_id: str) -> Dict[str, float]:
|
||||
"""
|
||||
Retrieve user's behavioral profile from Redis.
|
||||
|
||||
Returns:
|
||||
{
|
||||
'avg_amount': float,
|
||||
'tx_count': int,
|
||||
'last_timestamp': float,
|
||||
'days_since_last_tx': float
|
||||
}
|
||||
"""
|
||||
key = f"user:{user_id}:profile"
|
||||
profile = self.redis_client.hgetall(key)
|
||||
|
||||
if not profile:
|
||||
logger.debug(f"[REDIS LOOKUP] No profile for user={user_id}, using defaults")
|
||||
return {
|
||||
'avg_amount': 0.0,
|
||||
'tx_count': 0,
|
||||
'last_timestamp': 0.0,
|
||||
'days_since_last_tx': 999.0
|
||||
}
|
||||
|
||||
logger.debug(
|
||||
f"[REDIS LOOKUP] Found profile for user={user_id}: "
|
||||
f"avg=${profile.get('avg_amount', 0)}, count={profile.get('tx_count', 0)}"
|
||||
)
|
||||
|
||||
return {
|
||||
'avg_amount': float(profile.get('avg_amount', 0.0)),
|
||||
'tx_count': int(profile.get('tx_count', 0)),
|
||||
'last_timestamp': float(profile.get('last_timestamp', 0.0)),
|
||||
'days_since_last_tx': float(profile.get('days_since_last_tx', 999.0))
|
||||
}
|
||||
|
||||
def extract_features(self, transaction: Dict) -> np.ndarray:
|
||||
"""
|
||||
Extract feature vector from transaction + user profile.
|
||||
|
||||
Feature vector: [amount, avg_amount, tx_count, days_since_last_tx]
|
||||
"""
|
||||
user_id = transaction['user_id']
|
||||
amount = transaction['amount']
|
||||
|
||||
# Get user history
|
||||
profile = self.get_user_profile(user_id)
|
||||
|
||||
# Construct feature vector
|
||||
features = np.array([[
|
||||
amount,
|
||||
profile['avg_amount'],
|
||||
profile['tx_count'],
|
||||
profile['days_since_last_tx']
|
||||
]])
|
||||
|
||||
logger.info(
|
||||
f"[FEATURE EXTRACTION] tx_id={transaction['transaction_id'][:8]}... "
|
||||
f"user={user_id} features=[${amount:.0f}, "
|
||||
f"avg=${profile['avg_amount']:.0f}, "
|
||||
f"count={profile['tx_count']}, "
|
||||
f"days_since={profile['days_since_last_tx']:.1f}]"
|
||||
)
|
||||
|
||||
return features
|
||||
|
||||
def update_user_profile(self, transaction: Dict) -> None:
|
||||
"""Update user's behavioral profile with new transaction"""
|
||||
user_id = transaction['user_id']
|
||||
amount = transaction['amount']
|
||||
timestamp = transaction['timestamp']
|
||||
|
||||
key = f"user:{user_id}:profile"
|
||||
profile = self.redis_client.hgetall(key)
|
||||
|
||||
# Calculate new statistics
|
||||
old_count = int(profile.get('tx_count', 0))
|
||||
new_count = old_count + 1
|
||||
|
||||
old_avg = float(profile.get('avg_amount', 0.0))
|
||||
new_avg = (old_avg * old_count + amount) / new_count
|
||||
|
||||
old_timestamp = float(profile.get('last_timestamp', 0.0))
|
||||
days_since = 0.0 # Just made a transaction
|
||||
|
||||
# Update profile in Redis
|
||||
self.redis_client.hset(
|
||||
key,
|
||||
mapping={
|
||||
'tx_count': new_count,
|
||||
'avg_amount': round(new_avg, 2),
|
||||
'last_timestamp': timestamp,
|
||||
'days_since_last_tx': days_since
|
||||
}
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"[PROFILE UPDATE] user={user_id} "
|
||||
f"count={old_count}→{new_count}, "
|
||||
f"avg=${old_avg:.2f}→${new_avg:.2f}"
|
||||
)
|
||||
@@ -0,0 +1,72 @@
|
||||
"""
|
||||
Model Scoring Modules
|
||||
|
||||
Responsibilities:
|
||||
- Load pre-trained models
|
||||
- Run Layer 1 (fast) and Layer 2 (deep) scoring
|
||||
- Return fraud probabilities
|
||||
|
||||
Academic Point:
|
||||
Dual-layer architecture allows high-throughput systems to make most decisions
|
||||
(>95% of transactions) with a fast, lightweight model, while complex edge cases
|
||||
are delegated to a more expensive deep model. This optimizes latency vs. accuracy.
|
||||
"""
|
||||
|
||||
import pickle
|
||||
import logging
|
||||
from typing import Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class ModelScorer:
|
||||
"""Loads and runs fraud detection models"""
|
||||
|
||||
def __init__(self, layer1_path: str = "models/layer1.pkl",
|
||||
layer2_path: str = "models/layer2.pkl"):
|
||||
# Load Layer 1 model
|
||||
with open(layer1_path, 'rb') as f:
|
||||
self.layer1_model = pickle.load(f)
|
||||
logger.info(f"[MODEL LOADER] ✓ Layer 1 model loaded from {layer1_path}")
|
||||
|
||||
# Load Layer 2 model
|
||||
with open(layer2_path, 'rb') as f:
|
||||
self.layer2_model = pickle.load(f)
|
||||
logger.info(f"[MODEL LOADER] ✓ Layer 2 model loaded from {layer2_path}")
|
||||
|
||||
def score_layer1(self, features) -> float:
|
||||
"""
|
||||
Layer 1: Fast, shallow model for initial risk assessment.
|
||||
|
||||
Decision threshold: 0.4
|
||||
- score < 0.4 → Low risk (approve immediately)
|
||||
- score >= 0.4 → Suspicious (escalate to Layer 2)
|
||||
"""
|
||||
probs = self.layer1_model.predict_proba(features)
|
||||
fraud_probability = probs[0][1] # Index 1 is fraud class
|
||||
|
||||
logger.info(
|
||||
f"[LAYER 1 SCORING] Fraud probability: {fraud_probability:.4f} "
|
||||
f"{'→ LOW RISK, fast-track approval' if fraud_probability < 0.4 else '→ SUSPICIOUS, escalate to Layer 2'}"
|
||||
)
|
||||
|
||||
return fraud_probability
|
||||
|
||||
def score_layer2(self, features) -> float:
|
||||
"""
|
||||
Layer 2: Deep, complex model for suspicious transactions.
|
||||
|
||||
Decision threshold: 0.7
|
||||
- score <= 0.7 → Approve
|
||||
- score > 0.7 → Block
|
||||
|
||||
Only called if Layer 1 score >= 0.4 (suspicious).
|
||||
"""
|
||||
probs = self.layer2_model.predict_proba(features)
|
||||
fraud_probability = probs[0][1] # Index 1 is fraud class
|
||||
|
||||
logger.info(
|
||||
f"[LAYER 2 SCORING] Fraud probability: {fraud_probability:.4f} "
|
||||
f"{'→ BLOCK (high confidence in fraud)' if fraud_probability > 0.7 else '→ APPROVE (acceptable risk)'}"
|
||||
)
|
||||
|
||||
return fraud_probability
|
||||
@@ -0,0 +1,227 @@
|
||||
"""
|
||||
Kafka Consumer Pipeline Orchestrator
|
||||
|
||||
Responsibilities:
|
||||
- Connect to Kafka and consume transactions
|
||||
- Orchestrate the entire fraud detection pipeline
|
||||
- Coordinate all steps: feature extraction → scoring → decision → storage
|
||||
|
||||
Academic Point:
|
||||
This is where the architecture comes alive. By clearly separating concerns
|
||||
(features, models, decisions), the code becomes testable and maintainable.
|
||||
Each step is independently verifiable, making it suitable for academic demonstration.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from kafka import KafkaConsumer
|
||||
from kafka.errors import KafkaError
|
||||
|
||||
from features import FeatureExtractor
|
||||
from models import ModelScorer
|
||||
from decision import DecisionEngine
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# CONFIGURATION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
KAFKA_BROKER = "localhost:9092"
|
||||
KAFKA_TOPIC = "transactions"
|
||||
KAFKA_GROUP = "fraud-detection-consumer-group"
|
||||
|
||||
# Logging setup
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s [%(levelname)s] %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# PIPELINE INITIALIZATION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def initialize_pipeline():
|
||||
"""Initialize all components of the fraud detection pipeline"""
|
||||
|
||||
logger.info("="*80)
|
||||
logger.info("PayGuard Fraud Detection Consumer - INITIALIZATION")
|
||||
logger.info("="*80)
|
||||
|
||||
# Feature extraction
|
||||
feature_extractor = FeatureExtractor(
|
||||
redis_host="localhost",
|
||||
redis_port=6379
|
||||
)
|
||||
|
||||
# Model scoring
|
||||
model_scorer = ModelScorer(
|
||||
layer1_path="models/layer1.pkl",
|
||||
layer2_path="models/layer2.pkl"
|
||||
)
|
||||
|
||||
# Decision engine with circuit breaker
|
||||
decision_engine = DecisionEngine(
|
||||
redis_host="localhost",
|
||||
redis_port=6379
|
||||
)
|
||||
|
||||
# Kafka consumer
|
||||
consumer = KafkaConsumer(
|
||||
KAFKA_TOPIC,
|
||||
bootstrap_servers=KAFKA_BROKER,
|
||||
group_id=KAFKA_GROUP,
|
||||
value_deserializer=lambda m: json.loads(m.decode('utf-8')),
|
||||
auto_offset_reset='earliest',
|
||||
enable_auto_commit=True
|
||||
)
|
||||
|
||||
logger.info(f"[KAFKA CONSUMER] Connected to {KAFKA_BROKER}")
|
||||
logger.info(f"[KAFKA CONSUMER] Subscribed to topic '{KAFKA_TOPIC}'")
|
||||
logger.info(f"[KAFKA CONSUMER] Consumer group: {KAFKA_GROUP}")
|
||||
logger.info("="*80)
|
||||
logger.info("Waiting for transactions...\n")
|
||||
|
||||
return {
|
||||
'consumer': consumer,
|
||||
'feature_extractor': feature_extractor,
|
||||
'model_scorer': model_scorer,
|
||||
'decision_engine': decision_engine
|
||||
}
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# PIPELINE EXECUTION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def process_transaction(transaction, components):
|
||||
"""
|
||||
Execute the complete fraud detection pipeline for one transaction.
|
||||
|
||||
Steps:
|
||||
1. Feature extraction (with Redis behavioral profiling)
|
||||
2. Layer 1 fast model scoring
|
||||
3. Behavioral anomaly detection
|
||||
4. Layer 2 deep model scoring (if suspicious)
|
||||
5. Final decision with circuit breaker
|
||||
6. Update user profile
|
||||
7. Store result
|
||||
"""
|
||||
|
||||
transaction_id = transaction['transaction_id']
|
||||
user_id = transaction['user_id']
|
||||
amount = transaction['amount']
|
||||
merchant = transaction['merchant']
|
||||
|
||||
logger.info(f"\n{'='*80}")
|
||||
logger.info(f"[PIPELINE START] Transaction: {transaction_id[:8]}...")
|
||||
logger.info(f"[TRANSACTION] User: {user_id} | Amount: ${amount:.2f} | Merchant: {merchant}")
|
||||
logger.info(f"{'='*80}\n")
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# STEP 1: FEATURE EXTRACTION
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
logger.info("[STEP 1/6] Feature Extraction & Behavioral Profiling")
|
||||
logger.info("-" * 80)
|
||||
|
||||
features = components['feature_extractor'].extract_features(transaction)
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# STEP 2: LAYER 1 FAST MODEL
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
logger.info("\n[STEP 2/6] Layer 1 - Fast Model Scoring")
|
||||
logger.info("-" * 80)
|
||||
|
||||
layer1_score = components['model_scorer'].score_layer1(features)
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# STEP 3: BEHAVIORAL ANOMALY DETECTION
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
logger.info("\n[STEP 3/6] Behavioral Anomaly Detection")
|
||||
logger.info("-" * 80)
|
||||
|
||||
user_profile = components['feature_extractor'].get_user_profile(user_id)
|
||||
if user_profile['avg_amount'] > 0:
|
||||
ratio = amount / user_profile['avg_amount']
|
||||
logger.info(f"[ANOMALY] Amount/Avg Ratio: {ratio:.2f}x")
|
||||
if ratio > 3.0:
|
||||
logger.warning(f"[ANOMALY] ⚠ Significant deviation detected (>{3.0}x)")
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# STEP 4: LAYER 2 DEEP MODEL (if needed)
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
layer2_score = None
|
||||
if layer1_score >= 0.4:
|
||||
logger.info("\n[STEP 4/6] Layer 2 - Deep Model Scoring")
|
||||
logger.info("-" * 80)
|
||||
|
||||
layer2_score = components['model_scorer'].score_layer2(features)
|
||||
else:
|
||||
logger.info("\n[STEP 4/6] Layer 2 - Deep Model Scoring")
|
||||
logger.info("-" * 80)
|
||||
logger.info("[LAYER 2 SCORING] Skipped (Layer 1 score indicates low risk)")
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# STEP 5: FINAL DECISION
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
logger.info("\n[STEP 5/6] Final Decision Engine")
|
||||
logger.info("-" * 80)
|
||||
|
||||
decision = components['decision_engine'].make_decision(
|
||||
transaction_id=transaction_id,
|
||||
layer1_score=layer1_score,
|
||||
layer2_scorer=components['model_scorer'] if layer1_score >= 0.4 else None,
|
||||
features=features if layer1_score >= 0.4 else None
|
||||
)
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# STEP 6: UPDATE PROFILE & STORE RESULT
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
logger.info("\n[STEP 6/6] Profile Update & Result Storage")
|
||||
logger.info("-" * 80)
|
||||
|
||||
components['feature_extractor'].update_user_profile(transaction)
|
||||
components['decision_engine'].store_result(
|
||||
transaction_id=transaction_id,
|
||||
decision=decision,
|
||||
layer1_score=layer1_score,
|
||||
layer2_score=layer2_score
|
||||
)
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# PIPELINE COMPLETE
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
logger.info(f"\n{'='*80}")
|
||||
logger.info(f"[PIPELINE COMPLETE] Transaction {transaction_id[:8]}... → {decision.upper()}")
|
||||
logger.info(f"{'='*80}\n")
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# MAIN LOOP
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def run_consumer():
|
||||
"""Main loop: consume and process transactions from Kafka"""
|
||||
|
||||
components = initialize_pipeline()
|
||||
|
||||
try:
|
||||
logger.info("Consumer ready. Processing messages...\n")
|
||||
|
||||
for message in components['consumer']:
|
||||
if message.value is None:
|
||||
continue
|
||||
|
||||
transaction = message.value
|
||||
|
||||
try:
|
||||
process_transaction(transaction, components)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[ERROR] Failed to process transaction: {str(e)}", exc_info=True)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
logger.info("\nShutting down consumer...")
|
||||
finally:
|
||||
components['consumer'].close()
|
||||
logger.info("Consumer closed.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_consumer()
|
||||
@@ -0,0 +1,55 @@
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
zookeeper:
|
||||
image: confluentinc/cp-zookeeper:7.5.0
|
||||
environment:
|
||||
ZOOKEEPER_CLIENT_PORT: 2181
|
||||
ZOOKEEPER_TICK_TIME: 2000
|
||||
ports:
|
||||
- "2181:2181"
|
||||
networks:
|
||||
- payguard-network
|
||||
|
||||
kafka:
|
||||
image: confluentinc/cp-kafka:7.5.0
|
||||
depends_on:
|
||||
- zookeeper
|
||||
ports:
|
||||
- "9092:9092"
|
||||
environment:
|
||||
KAFKA_BROKER_ID: 1
|
||||
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
|
||||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
|
||||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
|
||||
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
|
||||
networks:
|
||||
- payguard-network
|
||||
healthcheck:
|
||||
test: kafka-broker-api-versions --bootstrap-server localhost:9092
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
redis:
|
||||
image: redis:7.2-alpine
|
||||
ports:
|
||||
- "6379:6379"
|
||||
networks:
|
||||
- payguard-network
|
||||
healthcheck:
|
||||
test: redis-cli ping
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
|
||||
networks:
|
||||
payguard-network:
|
||||
driver: bridge
|
||||
@@ -0,0 +1,56 @@
|
||||
"""
|
||||
Model definitions for PayGuard fraud detection
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
|
||||
class Layer1MockModel:
|
||||
"""Fast, lightweight model for initial fraud scoring"""
|
||||
|
||||
def predict_proba(self, features):
|
||||
"""
|
||||
Layer 1: Fast model
|
||||
Score < 0.4: Approve
|
||||
Score >= 0.4: Escalate to Layer 2
|
||||
"""
|
||||
amount = features[0][0]
|
||||
avg_amount = features[0][1]
|
||||
|
||||
if avg_amount == 0:
|
||||
ratio = 1.0
|
||||
else:
|
||||
ratio = amount / avg_amount
|
||||
|
||||
fraud_prob = min(0.95, max(0.05, (ratio - 1.0) * 0.2))
|
||||
legit_prob = 1.0 - fraud_prob
|
||||
|
||||
return np.array([[legit_prob, fraud_prob]])
|
||||
|
||||
class Layer2MockModel:
|
||||
"""Deep, complex model for suspicious transactions"""
|
||||
|
||||
def predict_proba(self, features):
|
||||
"""
|
||||
Layer 2: Deep model
|
||||
Score <= 0.7: Approve
|
||||
Score > 0.7: Block
|
||||
"""
|
||||
amount = features[0][0]
|
||||
avg_amount = features[0][1]
|
||||
tx_count = features[0][2]
|
||||
|
||||
fraud_prob = 0.1
|
||||
|
||||
if avg_amount > 0:
|
||||
ratio = amount / avg_amount
|
||||
fraud_prob += (ratio - 1.0) * 0.25
|
||||
|
||||
if tx_count < 3:
|
||||
fraud_prob += 0.2
|
||||
elif tx_count < 10:
|
||||
fraud_prob += 0.1
|
||||
|
||||
fraud_prob = min(0.95, max(0.05, fraud_prob))
|
||||
legit_prob = 1.0 - fraud_prob
|
||||
|
||||
return np.array([[legit_prob, fraud_prob]])
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,151 @@
|
||||
<svg width="100%" viewBox="0 0 680 1060" role="img" style="" xmlns="http://www.w3.org/2000/svg">
|
||||
<title style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">PayGuard Fraud Detection Flowchart</title>
|
||||
<desc style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">End-to-end flowchart of the PayGuard dual-layer fraud detection system, from transaction ingestion through Kafka, feature extraction, Layer 1 fast model, behavioral profiling, Layer 2 deep model, and final decision engine.</desc>
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse">
|
||||
<path d="M2 1L8 5L2 9" fill="none" stroke="context-stroke" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- ── Row 0: Transaction ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="240" y="30" width="200" height="48" rx="8" stroke-width="0.5" style="fill:rgb(68, 68, 65);stroke:rgb(180, 178, 169);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="50" text-anchor="middle" dominant-baseline="central" style="fill:rgb(211, 209, 199);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Transaction initiated</text>
|
||||
<text x="340" y="67" text-anchor="middle" dominant-baseline="central" style="fill:rgb(180, 178, 169);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Client payment request</text>
|
||||
</g>
|
||||
|
||||
<!-- arrow -->
|
||||
<line x1="340" y1="78" x2="340" y2="112" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
|
||||
<!-- ── Row 1: JWT Auth ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="220" y="112" width="240" height="48" rx="8" stroke-width="0.5" style="fill:rgb(60, 52, 137);stroke:rgb(175, 169, 236);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="132" text-anchor="middle" dominant-baseline="central" style="fill:rgb(206, 203, 246);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">JWT authentication</text>
|
||||
<text x="340" y="149" text-anchor="middle" dominant-baseline="central" style="fill:rgb(175, 169, 236);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Identity & authorization check</text>
|
||||
</g>
|
||||
|
||||
<!-- arrow -->
|
||||
<line x1="340" y1="160" x2="340" y2="194" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
|
||||
<!-- ── Row 2: Kafka Ingestion ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="220" y="194" width="240" height="48" rx="8" stroke-width="0.5" style="fill:rgb(8, 80, 65);stroke:rgb(93, 202, 165);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="214" text-anchor="middle" dominant-baseline="central" style="fill:rgb(159, 225, 203);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Kafka event stream</text>
|
||||
<text x="340" y="231" text-anchor="middle" dominant-baseline="central" style="fill:rgb(93, 202, 165);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Async message ingestion</text>
|
||||
</g>
|
||||
|
||||
<!-- arrow -->
|
||||
<line x1="340" y1="242" x2="340" y2="276" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
|
||||
<!-- ── Row 3: Feature Extraction ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="220" y="276" width="240" height="48" rx="8" stroke-width="0.5" style="fill:rgb(12, 68, 124);stroke:rgb(133, 183, 235);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="296" text-anchor="middle" dominant-baseline="central" style="fill:rgb(181, 212, 244);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Feature extraction</text>
|
||||
<text x="340" y="313" text-anchor="middle" dominant-baseline="central" style="fill:rgb(133, 183, 235);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Preprocessing & engineering</text>
|
||||
</g>
|
||||
|
||||
<!-- arrow -->
|
||||
<line x1="340" y1="324" x2="340" y2="358" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
|
||||
<!-- ── Row 4: Layer 1 Fast Model ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="190" y="358" width="300" height="56" rx="8" stroke-width="0.5" style="fill:rgb(12, 68, 124);stroke:rgb(133, 183, 235);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="378" text-anchor="middle" dominant-baseline="central" style="fill:rgb(181, 212, 244);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Layer 1 — fast model</text>
|
||||
<text x="340" y="397" text-anchor="middle" dominant-baseline="central" style="fill:rgb(133, 183, 235);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Logistic regression / shallow tree</text>
|
||||
</g>
|
||||
|
||||
<!-- ── Decision diamond ── -->
|
||||
<!-- arrow from layer 1 down -->
|
||||
<line x1="340" y1="414" x2="340" y2="452" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<!-- diamond shape -->
|
||||
<polygon points="340,452 420,492 340,532 260,492" fill="none" stroke="var(--color-border-primary)" stroke-width="0.5" style="fill:none;stroke:rgba(222, 220, 209, 0.4);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="487" text-anchor="middle" dominant-baseline="central" style="fill:rgb(250, 249, 245);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Risk score?</text>
|
||||
|
||||
<!-- ── LOW RISK branch (right) ── -->
|
||||
<!-- horizontal arrow right -->
|
||||
<line x1="420" y1="492" x2="520" y2="492" marker-end="url(#arrow)" stroke="#3B8BD4" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<!-- label -->
|
||||
<text x="470" y="482" text-anchor="middle" fill="var(--color-text-secondary)" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:auto">Low risk</text>
|
||||
<!-- Approve box -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="520" y="468" width="120" height="48" rx="8" stroke-width="0.5" style="fill:rgb(39, 80, 10);stroke:rgb(151, 196, 89);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="580" y="488" text-anchor="middle" dominant-baseline="central" style="fill:rgb(192, 221, 151);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Approved</text>
|
||||
<text x="580" y="505" text-anchor="middle" dominant-baseline="central" style="fill:rgb(151, 196, 89);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Transaction clears</text>
|
||||
</g>
|
||||
|
||||
<!-- ── SUSPICIOUS branch (down) ── -->
|
||||
<line x1="340" y1="532" x2="340" y2="566" marker-end="url(#arrow)" stroke="#BA7517" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="355" y="554" text-anchor="start" fill="var(--color-text-secondary)" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:start;dominant-baseline:auto">Suspicious</text>
|
||||
|
||||
<!-- ── Row 5: Behavioral Profiling ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="190" y="566" width="300" height="56" rx="8" stroke-width="0.5" style="fill:rgb(99, 56, 6);stroke:rgb(239, 159, 39);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="586" text-anchor="middle" dominant-baseline="central" style="fill:rgb(250, 199, 117);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Behavioral profiling</text>
|
||||
<text x="340" y="604" text-anchor="middle" dominant-baseline="central" style="fill:rgb(239, 159, 39);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Redis hot-path lookup</text>
|
||||
</g>
|
||||
|
||||
<!-- arrow -->
|
||||
<line x1="340" y1="622" x2="340" y2="656" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
|
||||
<!-- ── Row 6: Layer 2 Deep Model ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="190" y="656" width="300" height="56" rx="8" stroke-width="0.5" style="fill:rgb(113, 43, 19);stroke:rgb(240, 153, 123);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="676" text-anchor="middle" dominant-baseline="central" style="fill:rgb(245, 196, 179);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Layer 2 — deep model</text>
|
||||
<text x="340" y="694" text-anchor="middle" dominant-baseline="central" style="fill:rgb(240, 153, 123);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">XGBoost / ensemble analysis</text>
|
||||
</g>
|
||||
|
||||
<!-- arrow -->
|
||||
<line x1="340" y1="712" x2="340" y2="746" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
|
||||
<!-- ── Row 7: Final Decision Engine ── -->
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="190" y="746" width="300" height="56" rx="8" stroke-width="0.5" style="fill:rgb(60, 52, 137);stroke:rgb(175, 169, 236);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="766" text-anchor="middle" dominant-baseline="central" style="fill:rgb(206, 203, 246);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Final decision engine</text>
|
||||
<text x="340" y="784" text-anchor="middle" dominant-baseline="central" style="fill:rgb(175, 169, 236);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Latency-aware threshold logic</text>
|
||||
</g>
|
||||
|
||||
<!-- ── Decision diamond 2 ── -->
|
||||
<line x1="340" y1="802" x2="340" y2="840" marker-end="url(#arrow)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<polygon points="340,840 420,880 340,920 260,880" fill="none" stroke="var(--color-border-primary)" stroke-width="0.5" style="fill:none;stroke:rgba(222, 220, 209, 0.4);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="875" text-anchor="middle" dominant-baseline="central" style="fill:rgb(250, 249, 245);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Verdict?</text>
|
||||
|
||||
<!-- ── APPROVE branch (right) ── -->
|
||||
<line x1="420" y1="880" x2="510" y2="880" marker-end="url(#arrow)" stroke="#3B6D11" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="465" y="870" text-anchor="middle" fill="var(--color-text-secondary)" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:auto">Approve</text>
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="510" y="856" width="120" height="48" rx="8" stroke-width="0.5" style="fill:rgb(39, 80, 10);stroke:rgb(151, 196, 89);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="570" y="876" text-anchor="middle" dominant-baseline="central" style="fill:rgb(192, 221, 151);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Approved</text>
|
||||
<text x="570" y="893" text-anchor="middle" dominant-baseline="central" style="fill:rgb(151, 196, 89);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Transaction clears</text>
|
||||
</g>
|
||||
|
||||
<!-- ── BLOCK branch (left) ── -->
|
||||
<line x1="260" y1="880" x2="150" y2="880" marker-end="url(#arrow)" stroke="#A32D2D" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="205" y="870" text-anchor="middle" fill="var(--color-text-secondary)" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:auto">Block</text>
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="40" y="856" width="110" height="48" rx="8" stroke-width="0.5" style="fill:rgb(121, 31, 31);stroke:rgb(240, 149, 149);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="95" y="876" text-anchor="middle" dominant-baseline="central" style="fill:rgb(247, 193, 193);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Blocked</text>
|
||||
<text x="95" y="893" text-anchor="middle" dominant-baseline="central" style="fill:rgb(240, 149, 149);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Fraud flagged</text>
|
||||
</g>
|
||||
|
||||
<!-- ── Circuit breaker note ── -->
|
||||
<line x1="340" y1="920" x2="340" y2="960" marker-end="url(#arrow)" stroke="var(--color-border-secondary)" style="fill:none;stroke:rgb(156, 154, 146);color:rgb(255, 255, 255);stroke-width:1.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<g style="fill:rgb(0, 0, 0);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
|
||||
<rect x="200" y="960" width="280" height="48" rx="8" stroke-width="0.5" style="fill:rgb(68, 68, 65);stroke:rgb(180, 178, 169);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="340" y="980" text-anchor="middle" dominant-baseline="central" style="fill:rgb(211, 209, 199);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:14px;font-weight:500;text-anchor:middle;dominant-baseline:central">Circuit breaker / retry</text>
|
||||
<text x="340" y="997" text-anchor="middle" dominant-baseline="central" style="fill:rgb(180, 178, 169);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:central">Fallback if scorer fails</text>
|
||||
</g>
|
||||
|
||||
<!-- ── Legend ── -->
|
||||
<rect x="40" y="30" width="140" height="140" rx="8" fill="none" stroke="var(--color-border-tertiary)" stroke-width="0.5" stroke-dasharray="4 3" style="fill:none;stroke:rgba(222, 220, 209, 0.15);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-dasharray:4px, 3px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="110" y="50" text-anchor="middle" fill="var(--color-text-secondary)" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:middle;dominant-baseline:auto">Legend</text>
|
||||
<rect x="54" y="60" width="14" height="14" rx="3" stroke-width="0.5" style="fill:rgb(68, 68, 65);stroke:rgb(180, 178, 169);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="76" y="71" text-anchor="start" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:start;dominant-baseline:auto">Entry / infra</text>
|
||||
<rect x="54" y="82" width="14" height="14" rx="3" stroke-width="0.5" style="fill:rgb(60, 52, 137);stroke:rgb(175, 169, 236);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="76" y="93" text-anchor="start" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:start;dominant-baseline:auto">Auth / decision</text>
|
||||
<rect x="54" y="104" width="14" height="14" rx="3" stroke-width="0.5" style="fill:rgb(12, 68, 124);stroke:rgb(133, 183, 235);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="76" y="115" text-anchor="start" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:start;dominant-baseline:auto">ML pipeline</text>
|
||||
<rect x="54" y="126" width="14" height="14" rx="3" stroke-width="0.5" style="fill:rgb(99, 56, 6);stroke:rgb(239, 159, 39);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="76" y="137" text-anchor="start" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:start;dominant-baseline:auto">Profiling</text>
|
||||
<rect x="54" y="148" width="14" height="14" rx="3" stroke-width="0.5" style="fill:rgb(113, 43, 19);stroke:rgb(240, 153, 123);color:rgb(255, 255, 255);stroke-width:0.5px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
|
||||
<text x="76" y="159" text-anchor="start" style="fill:rgb(194, 192, 182);stroke:none;color:rgb(255, 255, 255);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:"Anthropic Sans", -apple-system, "system-ui", "Segoe UI", sans-serif;font-size:12px;font-weight:400;text-anchor:start;dominant-baseline:auto">Deep model</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 34 KiB |
Whitespace-only changes.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,228 @@
|
||||
"""
|
||||
Test Transaction Producer
|
||||
|
||||
Generates demo transactions with controlled scenarios to showcase the pipeline.
|
||||
|
||||
Scenarios:
|
||||
1. Normal transactions: low amounts, regular merchants, regular users
|
||||
2. Suspicious transactions: large amounts, unknown merchants
|
||||
3. Behavioral anomalies: large deviation from user history
|
||||
4. Circuit breaker test: can be triggered manually
|
||||
|
||||
Usage:
|
||||
python producer/simulate.py
|
||||
"""
|
||||
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import logging
|
||||
from typing import List, Tuple
|
||||
from jose import jwt
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# CONFIGURATION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
API_BASE_URL = "http://localhost:8000"
|
||||
SECRET_KEY = "payguard-demo-secret-key-change-in-production"
|
||||
ALGORITHM = "HS256"
|
||||
|
||||
# Logging setup
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s [%(levelname)s] %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# TOKEN GENERATION
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def generate_token(user_id: str) -> str:
|
||||
"""Generate a JWT token for the given user"""
|
||||
token = jwt.encode({"sub": user_id}, SECRET_KEY, algorithm=ALGORITHM)
|
||||
return token
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# TRANSACTION SCENARIOS
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TransactionScenario:
|
||||
"""Represents a test transaction scenario"""
|
||||
|
||||
def __init__(self, name: str, user_id: str, amount: float,
|
||||
merchant: str, description: str = ""):
|
||||
self.name = name
|
||||
self.user_id = user_id
|
||||
self.amount = amount
|
||||
self.merchant = merchant
|
||||
self.description = description
|
||||
|
||||
def submit(self):
|
||||
"""Submit this scenario to the API"""
|
||||
token = generate_token(self.user_id)
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
payload = {
|
||||
"amount": self.amount,
|
||||
"merchant": self.merchant,
|
||||
"description": self.description
|
||||
}
|
||||
|
||||
logger.info(f"\n{'='*80}")
|
||||
logger.info(f"[SCENARIO] {self.name}")
|
||||
logger.info(f"[SCENARIO] User: {self.user_id} | Amount: ${self.amount:.2f} | Merchant: {self.merchant}")
|
||||
logger.info(f"{'='*80}")
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{API_BASE_URL}/transaction",
|
||||
headers=headers,
|
||||
json=payload,
|
||||
timeout=5
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
result = response.json()
|
||||
logger.info(f"[API RESPONSE] ✓ {result['message']}")
|
||||
logger.info(f"[TRANSACTION ID] {result['transaction_id']}")
|
||||
else:
|
||||
logger.error(f"[API RESPONSE] ✗ Status {response.status_code}: {response.text}")
|
||||
|
||||
except requests.exceptions.ConnectionError:
|
||||
logger.error("[API RESPONSE] ✗ Cannot connect to API. Is it running on port 8000?")
|
||||
except Exception as e:
|
||||
logger.error(f"[API RESPONSE] ✗ Error: {str(e)}")
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# TEST SCENARIOS
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def get_demo_scenarios() -> List[TransactionScenario]:
|
||||
"""Get all demo scenarios in a logical order"""
|
||||
return [
|
||||
# Scenario 1: Normal transaction (should be approved fast)
|
||||
TransactionScenario(
|
||||
name="1. NORMAL TRANSACTION (Low Risk)",
|
||||
user_id="user_001",
|
||||
amount=75.50,
|
||||
merchant="Starbucks Coffee",
|
||||
description="Regular coffee purchase"
|
||||
),
|
||||
|
||||
# Scenario 2: Another normal transaction from same user
|
||||
TransactionScenario(
|
||||
name="2. NORMAL TRANSACTION #2 (Low Risk)",
|
||||
user_id="user_001",
|
||||
amount=120.00,
|
||||
merchant="Target Retail",
|
||||
description="Grocery shopping"
|
||||
),
|
||||
|
||||
# Scenario 3: Large transaction from user with low history (suspicious)
|
||||
TransactionScenario(
|
||||
name="3. BEHAVIORAL ANOMALY (High Risk)",
|
||||
user_id="user_001",
|
||||
amount=8500.00,
|
||||
merchant="unknown_offshore_merchant.biz",
|
||||
description="Large purchase from unknown merchant"
|
||||
),
|
||||
|
||||
# Scenario 4: New user, normal transaction
|
||||
TransactionScenario(
|
||||
name="4. NEW USER, NORMAL AMOUNT (Low Risk)",
|
||||
user_id="user_002",
|
||||
amount=45.00,
|
||||
merchant="McDonald's Restaurant",
|
||||
description="Meal purchase"
|
||||
),
|
||||
|
||||
# Scenario 5: Existing user, normal range
|
||||
TransactionScenario(
|
||||
name="5. RETURNING USER, NORMAL (Low Risk)",
|
||||
user_id="user_002",
|
||||
amount=65.00,
|
||||
merchant="CVS Pharmacy",
|
||||
description="Pharmacy purchase"
|
||||
),
|
||||
|
||||
# Scenario 6: User 3, establish history
|
||||
TransactionScenario(
|
||||
name="6. NEW USER, SMALL TRANSACTION (Low Risk)",
|
||||
user_id="user_003",
|
||||
amount=25.00,
|
||||
merchant="Uber Eats",
|
||||
description="Food delivery"
|
||||
),
|
||||
|
||||
# Scenario 7: User 3, escalating pattern
|
||||
TransactionScenario(
|
||||
name="7. SAME USER, INCREASED AMOUNT (Medium Risk)",
|
||||
user_id="user_003",
|
||||
amount=150.00,
|
||||
merchant="Amazon.com",
|
||||
description="Online shopping"
|
||||
),
|
||||
|
||||
# Scenario 8: User 3, huge spike (very suspicious)
|
||||
TransactionScenario(
|
||||
name="8. SUSPICIOUS PATTERN (Very High Risk)",
|
||||
user_id="user_003",
|
||||
amount=12500.00,
|
||||
merchant="LuxuryWatches_CN.ru",
|
||||
description="Suspicious high-value purchase"
|
||||
),
|
||||
]
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# MAIN
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def main():
|
||||
"""Run demo scenarios"""
|
||||
|
||||
logger.info("="*80)
|
||||
logger.info("PayGuard Demo - Transaction Producer")
|
||||
logger.info("="*80)
|
||||
logger.info(f"API Base URL: {API_BASE_URL}")
|
||||
logger.info("="*80)
|
||||
|
||||
# Check if API is available
|
||||
try:
|
||||
response = requests.get(f"{API_BASE_URL}/health", timeout=2)
|
||||
if response.status_code == 200:
|
||||
logger.info("✓ API is healthy and ready\n")
|
||||
else:
|
||||
logger.error("✗ API health check failed")
|
||||
return
|
||||
except:
|
||||
logger.error("✗ Cannot connect to API. Make sure it's running:")
|
||||
logger.error(" uvicorn api.main:app --reload\n")
|
||||
return
|
||||
|
||||
scenarios = get_demo_scenarios()
|
||||
|
||||
for i, scenario in enumerate(scenarios, 1):
|
||||
scenario.submit()
|
||||
|
||||
# Small delay between scenarios to avoid overwhelming
|
||||
if i < len(scenarios):
|
||||
logger.info(f"\n[DEMO] Waiting 3 seconds before next scenario...")
|
||||
time.sleep(3)
|
||||
|
||||
logger.info(f"\n{'='*80}")
|
||||
logger.info("✓ All demo scenarios submitted!")
|
||||
logger.info("="*80)
|
||||
logger.info("\nCheck the consumer terminal for detailed pipeline execution logs.")
|
||||
logger.info("Open another terminal and query Redis to see the results:")
|
||||
logger.info(" redis-cli")
|
||||
logger.info(" > KEYS result:*")
|
||||
logger.info(" > GET result:<transaction_id>")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,8 @@
|
||||
fastapi==0.104.1
|
||||
uvicorn[standard]==0.24.0
|
||||
python-jose[cryptography]==3.3.0
|
||||
kafka-python==2.0.2
|
||||
redis==5.0.0
|
||||
pybreaker==1.4.0
|
||||
python-dotenv==1.0.0
|
||||
numpy==1.24.3
|
||||
Reference in new issue
Block a user