Initial PayGuard fraud detection system implementation
- Complete dual-layer architecture for fraud detection - Phase 1: Docker Compose infrastructure (Kafka, Zookeeper, Redis) - Phase 2: Mock ML models (Layer 1 fast, Layer 2 deep) - Phase 3: FastAPI transaction ingestion with JWT auth - Phase 4: Kafka consumer pipeline with feature extraction - Phase 5: Test transaction producer with demo scenarios - Phase 6: Clean code structure and comprehensive documentation - Phase 7: End-to-end validation and demo readiness Demonstrates key system design patterns: - Microservices architecture with decoupled components - Asynchronous event processing via Kafka - Tiered decision-making (fast path vs deep analysis) - Resilience patterns (circuit breaker for graceful degradation) - Behavioral profiling with Redis caching Ready for academic demonstration with clear pipeline observability. Co-authored-by: Copilot <[email protected]>
This commit is contained in:
John Patrick TeruelandCopilot committed
commit
aa4dd497e2
24 files changed
+2343
No files matched your search
@@ -0,0 +1,335 @@
|
||||
Here's the full end-to-end flowchart of the PayGuard system. The flow follows this logic:
|
||||
|
||||
Transaction initiated → passes through JWT authentication to verify the caller's identity.
|
||||
Kafka ingests the event asynchronously, decoupling ingestion from processing.
|
||||
Feature extraction preprocesses the raw transaction data for the models.
|
||||
Layer 1 (fast model) runs the first risk score — low-risk transactions are approved immediately, keeping latency minimal.
|
||||
Suspicious transactions proceed to behavioral profiling via Redis, then to Layer 2 (deep model) for heavier XGBoost/ensemble analysis.
|
||||
The final decision engine applies latency-aware thresholding to issue a final Approve or Block verdict.
|
||||
A circuit breaker / retry layer at the bottom handles fallback if the scoring engine is slow or unavailable — ensuring system uptime.
|
||||
|
||||
The dual-branch decision diamonds are the core of the "latency-aware" design — fast approvals exit early, while only truly suspicious transactions bear the cost of deep analysis.You said: Can you give me steps on how can I make this work in actual?Can you give me steps on how can I make this work in actual?Apr 21You said: Q: What's your experience level with backend/distributed systems?Q: What's your experience level with backend/distributed systems?
|
||||
A: IntermediateQ: Which language/ecosystem do you prefer?
|
||||
A: PythonApr 21Claude responded: Great — here's a practical, step-by-step build guide for PayGuard using Python, tailored for someone who already knows the basics of backend dev.Great — here's a practical, step-by-step build guide for PayGuard using Python, tailored for someone who already knows the basics of backend dev.
|
||||
|
||||
Phase 1 — Set up your local infrastructure
|
||||
Before writing any application code, get your core services running locally via Docker.
|
||||
1. Install Docker Desktop if you haven't already, then create a docker-compose.yml to spin up Kafka, Zookeeper, and Redis together:
|
||||
yamlversion: "3.8"
|
||||
services:
|
||||
zookeeper:
|
||||
image: confluentinc/cp-zookeeper:7.5.0
|
||||
environment:
|
||||
ZOOKEEPER_CLIENT_PORT: 2181
|
||||
|
||||
kafka:
|
||||
image: confluentinc/cp-kafka:7.5.0
|
||||
depends_on: [zookeeper]
|
||||
ports:
|
||||
- "9092:9092"
|
||||
environment:
|
||||
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
|
||||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
|
||||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
|
||||
|
||||
redis:
|
||||
image: redis:7.2
|
||||
ports:
|
||||
- "6379:6379"
|
||||
Run it with docker compose up -d. This gives you Kafka on port 9092 and Redis on port 6379.
|
||||
|
||||
Phase 2 — Build the transaction ingestion API
|
||||
2. Create a FastAPI service that receives transaction requests, validates the JWT, and publishes to Kafka.
|
||||
bashpip install fastapi uvicorn python-jose kafka-python
|
||||
python# main.py
|
||||
from fastapi import FastAPI, Depends, HTTPException
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
from jose import jwt, JWTError
|
||||
from kafka import KafkaProducer
|
||||
import json, uuid, time
|
||||
|
||||
app = FastAPI()
|
||||
security = HTTPBearer()
|
||||
SECRET_KEY = "your-secret-key"
|
||||
producer = KafkaProducer(
|
||||
bootstrap_servers="localhost:9092",
|
||||
value_serializer=lambda v: json.dumps(v).encode()
|
||||
)
|
||||
|
||||
def verify_jwt(credentials: HTTPAuthorizationCredentials = Depends(security)):
|
||||
try:
|
||||
payload = jwt.decode(credentials.credentials, SECRET_KEY, algorithms=["HS256"])
|
||||
return payload
|
||||
except JWTError:
|
||||
raise HTTPException(status_code=401, detail="Invalid token")
|
||||
|
||||
@app.post("/transaction")
|
||||
def submit_transaction(data: dict, user=Depends(verify_jwt)):
|
||||
event = {
|
||||
"transaction_id": str(uuid.uuid4()),
|
||||
"user_id": user["sub"],
|
||||
"amount": data["amount"],
|
||||
"merchant": data["merchant"],
|
||||
"timestamp": time.time()
|
||||
}
|
||||
producer.send("transactions", event)
|
||||
return {"status": "received", "transaction_id": event["transaction_id"]}
|
||||
Run with uvicorn main:app --reload.
|
||||
|
||||
Phase 3 — Build the fraud scoring consumer
|
||||
3. Install ML and Redis dependencies:
|
||||
bashpip install kafka-python redis scikit-learn xgboost pandas
|
||||
4. Create a Kafka consumer that reads transactions, runs Layer 1, and conditionally runs Layer 2:
|
||||
python# consumer.py
|
||||
from kafka import KafkaConsumer
|
||||
import redis, json, pickle, numpy as np
|
||||
|
||||
consumer = KafkaConsumer(
|
||||
"transactions",
|
||||
bootstrap_servers="localhost:9092",
|
||||
value_deserializer=lambda m: json.loads(m.decode())
|
||||
)
|
||||
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
|
||||
|
||||
# Load pre-trained models (see Phase 4)
|
||||
layer1_model = pickle.load(open("layer1.pkl", "rb"))
|
||||
layer2_model = pickle.load(open("layer2.pkl", "rb"))
|
||||
|
||||
def extract_features(tx):
|
||||
history = r.hgetall(f"user:{tx['user_id']}:profile")
|
||||
avg_amount = float(history.get("avg_amount", tx["amount"]))
|
||||
tx_count = int(history.get("tx_count", 0))
|
||||
return np.array([[tx["amount"], avg_amount, tx_count]])
|
||||
|
||||
def update_profile(tx):
|
||||
key = f"user:{tx['user_id']}:profile"
|
||||
history = r.hgetall(key)
|
||||
count = int(history.get("tx_count", 0)) + 1
|
||||
avg = (float(history.get("avg_amount", 0)) * (count - 1) + tx["amount"]) / count
|
||||
r.hset(key, mapping={"tx_count": count, "avg_amount": round(avg, 2)})
|
||||
|
||||
for message in consumer:
|
||||
tx = message.value
|
||||
features = extract_features(tx)
|
||||
|
||||
score_l1 = layer1_model.predict_proba(features)[0][1] # fraud probability
|
||||
|
||||
if score_l1 < 0.4:
|
||||
decision = "approved"
|
||||
else:
|
||||
# Layer 2: deeper analysis
|
||||
score_l2 = layer2_model.predict_proba(features)[0][1]
|
||||
decision = "blocked" if score_l2 > 0.7 else "approved"
|
||||
|
||||
update_profile(tx)
|
||||
print(f"[{tx['transaction_id']}] Score L1={score_l1:.2f} → {decision.upper()}")
|
||||
# Store result in Redis or a database for downstream retrieval
|
||||
r.set(f"result:{tx['transaction_id']}", decision, ex=3600)
|
||||
|
||||
Phase 4 — Train your ML models
|
||||
5. Train Layer 1 and Layer 2 models on a fraud dataset. A great free dataset to start with is the Kaggle Credit Card Fraud dataset.
|
||||
python# train.py
|
||||
import pandas as pd, pickle
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.model_selection import train_test_split
|
||||
from xgboost import XGBClassifier
|
||||
|
||||
df = pd.read_csv("creditcard.csv")
|
||||
X = df.drop("Class", axis=1)
|
||||
y = df["Class"]
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
# Layer 1 — fast, lightweight
|
||||
l1 = LogisticRegression(max_iter=1000)
|
||||
l1.fit(X_train, y_train)
|
||||
pickle.dump(l1, open("layer1.pkl", "wb"))
|
||||
|
||||
# Layer 2 — deeper, more accurate
|
||||
l2 = XGBClassifier(n_estimators=100, use_label_encoder=False, eval_metric="logloss")
|
||||
l2.fit(X_train, y_train)
|
||||
pickle.dump(l2, open("layer2.pkl", "wb"))
|
||||
|
||||
Phase 5 — Add the circuit breaker
|
||||
6. Wrap your Layer 2 call with a circuit breaker so if it's slow or crashing, the system falls back gracefully:
|
||||
bashpip install pybreaker
|
||||
pythonimport pybreaker, time
|
||||
|
||||
breaker = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30)
|
||||
|
||||
@breaker
|
||||
def run_layer2(features):
|
||||
return layer2_model.predict_proba(features)[0][1]
|
||||
|
||||
# In your consumer loop:
|
||||
try:
|
||||
score_l2 = run_layer2(features)
|
||||
except pybreaker.CircuitBreakerError:
|
||||
print("Circuit open — defaulting to Layer 1 score")
|
||||
score_l2 = score_l1 # fallback
|
||||
|
||||
Phase 6 — Wire it all together & test
|
||||
7. End-to-end test flow:
|
||||
bash# Terminal 1 — start infrastructure
|
||||
docker compose up -d
|
||||
|
||||
# Terminal 2 — start the API
|
||||
uvicorn main:app --reload
|
||||
|
||||
# Terminal 3 — start the consumer
|
||||
python consumer.py
|
||||
|
||||
# Terminal 4 — send a test transaction
|
||||
curl -X POST http://localhost:8000/transaction \
|
||||
-H "Authorization: Bearer <your_jwt_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"amount": 9999.99, "merchant": "unknown_vendor_xyz"}'
|
||||
|
||||
Recommended build order
|
||||
Go in this sequence to avoid getting blocked:
|
||||
|
||||
Docker infra up → verify Kafka and Redis are reachable
|
||||
Train and save your .pkl models first (you need these before the consumer works)
|
||||
Build and test the consumer standalone (feed it fake JSON directly)
|
||||
Build the FastAPI ingestion layer
|
||||
Connect them end-to-end via Kafka
|
||||
Add the circuit breaker last, once the happy path works
|
||||
|
||||
|
||||
Tools summary
|
||||
ComponentToolAPI layerFastAPI + UvicornAuthpython-jose (JWT)Message brokerApache Kafka (via kafka-python)Hot-path cacheRedis (via redis-py)Layer 1 modelscikit-learn Logistic RegressionLayer 2 modelXGBoostCircuit breakerpybreakerLocal infraDocker Compose
|
||||
Want me to go deeper on any specific phase — like deploying this to the cloud, setting up proper model evaluation, or adding a results dashboard?Turn on web search in Search and tools menu. Otherwise, links provided may not be accurate or up to date.You said: is it more doable than this pipeline?is it more doable than this pipeline?Transaction →
|
||||
Kafka →
|
||||
Feature Extraction →
|
||||
Layer 1 (Fast Model) →
|
||||
├── Low Risk → Approve
|
||||
└── Suspicious →
|
||||
Behavioral Profiling (Redis) →
|
||||
Layer 2 (Deep Model) →
|
||||
Final Decision Engine (Latency-Aware) →
|
||||
Approve / Block11:07 AMClaude responded: Honestly?Honestly? No — that pipeline IS the system. What I gave you in the steps is literally how you build that exact pipeline, one piece at a time.
|
||||
But let me reframe it so it feels less overwhelming:
|
||||
|
||||
That pipeline, translated to plain components
|
||||
Pipeline StepWhat it actually isTransactionA curl or API call to FastAPIKafkaA message queue (one Docker container)Feature ExtractionA Python function — maybe 20 linesLayer 1 Fast ModelA pre-trained .pkl file you load and callRedis Behavioral ProfilingA Redis hash lookup — 2 lines of codeLayer 2 Deep ModelAnother .pkl file, only called sometimesFinal Decision EngineAn if score > 0.7 statement
|
||||
Every single box is just a function or a service call. The pipeline looks complex because it's drawn as an architecture diagram — but you're essentially writing a Python script that reads from Kafka in a loop and runs some if statements.
|
||||
|
||||
What makes it feel hard vs. what's actually hard
|
||||
Feels hard (but isn't):
|
||||
|
||||
The diagram looks like enterprise software
|
||||
"Kafka" and "Redis" sound scary
|
||||
"Dual-layer ML" sounds research-level
|
||||
|
||||
Actually the harder parts:
|
||||
|
||||
Getting good training data for the fraud models
|
||||
Tuning the thresholds (0.4 and 0.7) so you don't block legitimate users
|
||||
Keeping Kafka + Redis running reliably in production
|
||||
|
||||
|
||||
If you want a simpler starting point
|
||||
You can strip it down to this first, then add layers back in:
|
||||
Transaction → FastAPI → Python fraud check function → Approve/Block
|
||||
No Kafka, no Redis, no Layer 2. Just a single API that takes a transaction and returns a decision. Once that works, you add Kafka for async processing, then Redis for profiling, then the second model.
|
||||
The blueprint is the destination. You don't have to build all of it on day one.
|
||||
Want me to map out a week-by-week build plan so it feels more manageable?You said: But in my project, I want to demonstrate the whole process even if its not the modern approach.But in my project, I want to demonstrate the whole process even if its not the modern approach. I want to demonstrate and highlight the key idea on how this architecture works especially how each steps are executed. By doing this, I can have a slightly complicated but academically sound and accurate project in my subject Advanced System Design and Implementation.11:16 AMClaude responded: That's a solid academic framing.That's a solid academic framing. So the goal isn't production-readiness — it's demonstrating architectural correctness and showing your professor that you understand why each component exists and how they interact.
|
||||
Here's how I'd approach it:
|
||||
|
||||
Core Philosophy for Your Demo
|
||||
|
||||
Each component should be visibly doing its job. Your evaluator should be able to watch the system and trace a transaction from entry to decision, step by step.
|
||||
|
||||
This means you prioritize observability over optimization — logs, print statements, and clear separation of concerns matter more than raw performance.
|
||||
|
||||
What You Actually Need to Build
|
||||
1. The Transaction Producer (Client simulation)
|
||||
A simple Python script that generates fake transactions and sends them to Kafka. This simulates the "client" side.
|
||||
python# producer.py — simulates incoming payments
|
||||
transactions = [
|
||||
{"amount": 50, "merchant": "Jollibee", "user_id": "u001"},
|
||||
{"amount": 85000, "merchant": "unknown_offshore", "user_id": "u001"},
|
||||
{"amount": 120, "merchant": "SM Supermarket", "user_id": "u002"},
|
||||
]
|
||||
This lets you control the demo — you can deliberately send a suspicious transaction and show the class exactly what happens.
|
||||
|
||||
2. The Ingestion API (FastAPI + JWT)
|
||||
A single endpoint that:
|
||||
|
||||
Validates the JWT token (proves you understand auth)
|
||||
Publishes the transaction to Kafka
|
||||
Returns immediately — this is where you explain decoupling
|
||||
|
||||
The key academic point here is: "The API does not wait for fraud analysis. It just acknowledges receipt." That's the architectural insight.
|
||||
|
||||
3. The Kafka Topic
|
||||
This is your architectural centerpiece. It's what separates ingestion from processing. For demo purposes, even one topic with one partition is enough.
|
||||
The point you highlight: "Even if the fraud engine is slow or down, transactions are not lost — they wait in the queue."
|
||||
|
||||
4. The Consumer / Processing Pipeline
|
||||
This is where the architecture really shows. Structure it as clearly separated functions, not one blob:
|
||||
pythondef step1_extract_features(tx): # clearly named
|
||||
print(f"[FEATURE EXTRACTION] Processing tx {tx['id']}")
|
||||
...
|
||||
|
||||
def step2_layer1_score(features): # fast model
|
||||
print(f"[LAYER 1] Running logistic regression...")
|
||||
...
|
||||
|
||||
def step3_behavioral_profile(tx): # Redis lookup
|
||||
print(f"[REDIS] Fetching user {tx['user_id']} history...")
|
||||
...
|
||||
|
||||
def step4_layer2_score(features): # deep model
|
||||
print(f"[LAYER 2] Running XGBoost analysis...")
|
||||
...
|
||||
|
||||
def step5_decision(score): # final engine
|
||||
print(f"[DECISION ENGINE] Final score: {score:.2f}")
|
||||
...
|
||||
When you run a live demo, your terminal will visually walk through each step — that's what makes it academically convincing.
|
||||
|
||||
5. Redis Behavioral Profiling
|
||||
For each user, store their transaction history in Redis. This proves you understand the "hot path" concept — fast in-memory lookups instead of hitting a database.
|
||||
user:u001:profile → { avg_amount: 175.0, tx_count: 12 }
|
||||
A transaction of 85,000 from a user whose average is 175 is a clear anomaly. That's your demo moment.
|
||||
|
||||
6. The Circuit Breaker
|
||||
Even if it never actually trips during the demo, having it in the code with a clear comment and a simulated failure test shows architectural maturity.
|
||||
python# Demonstrate: kill Layer 2 service, show system falls back gracefully
|
||||
|
||||
How to Structure Your Demonstration
|
||||
Demo MomentWhat you showSend a normal transactionTrace it through L1 → Approved, fast pathSend a suspicious transactionWatch it go through L1 → L2 → Decision EngineShow Redis before & afterUser profile updates in real timeKill the scorer, send a transactionCircuit breaker activates, fallback kicks inShow Kafka consumer lagProve the queue held the message while scorer was down
|
||||
|
||||
What Makes This Academically Sound
|
||||
You're demonstrating four core Advanced System Design concepts in one project:
|
||||
|
||||
Decoupled microservices — API and scorer are independent processes
|
||||
Asynchronous message passing — Kafka as the backbone
|
||||
Tiered processing — dual-layer model with conditional routing
|
||||
Resilience patterns — circuit breaker and retry logic
|
||||
|
||||
Each one maps directly to topics covered in systems design literature. You can cite each component against a design pattern in your write-up.
|
||||
|
||||
Suggested Project Structure
|
||||
payguard/
|
||||
├── docker-compose.yml # Kafka + Redis
|
||||
├── producer/
|
||||
│ └── simulate.py # sends test transactions
|
||||
├── api/
|
||||
│ └── main.py # FastAPI + JWT
|
||||
├── consumer/
|
||||
│ ├── pipeline.py # orchestrates all steps
|
||||
│ ├── features.py # feature extraction
|
||||
│ ├── layer1.py # fast model
|
||||
│ ├── layer2.py # deep model
|
||||
│ ├── profiling.py # Redis behavioral lookup
|
||||
│ └── decision.py # final engine + circuit breaker
|
||||
├── models/
|
||||
│ ├── train.py # training script
|
||||
│ ├── layer1.pkl
|
||||
│ └── layer2.pkl
|
||||
└── README.md # architecture explanation
|
||||
This structure alone communicates that you understand separation of concerns — each file has exactly one responsibility.
|
||||
|
||||
The whole thing can run on your laptop. You demo it live, walk through the terminal output step by step, and your professor can see every architectural decision executing in real time.
|
||||
Want me to write out the full code for any specific component?
|
||||
Reference in new issue
Block a user