diff --git a/EXPLAIN.md b/EXPLAIN.md new file mode 100644 index 0000000..c5ac527 --- /dev/null +++ b/EXPLAIN.md @@ -0,0 +1,825 @@ +# System Architecture Explanation - PayGuard Fraud Detection + +## Docker Containers Overview + +The PayGuard fraud detection system uses **5 Docker containers** working together on a shared Docker network. + +```mermaid +graph TB + subgraph docker["Docker Network: payguard-network"] + ZK["🔹 Zookeeper
Port: 2181"] + KAFKA["🔹 Kafka
Port: 29092/9092"] + REDIS["🔹 Redis
Port: 6379"] + API["🔹 API
Port: 8000"] + FE["🔹 Frontend
Port: 3000"] + end + + ZK -->|coordinates| KAFKA + KAFKA -->|stores messages| REDIS + API -->|publishes to| KAFKA + FE -->|calls| API + REDIS -->|stores results| REDIS + + style docker fill:#f0f0f0 + style ZK fill:#4CAF50 + style KAFKA fill:#2196F3 + style REDIS fill:#FF9800 + style API fill:#9C27B0 + style FE fill:#F44336 +``` + +--- + +## Container 1: Zookeeper + +### Overview +**Coordination Service for Kafka** - Manages Kafka broker state and configuration + +### Configuration +```yaml +Image: confluentinc/cp-zookeeper:7.5.0 +Port: 2181 +Network: payguard-network +Environment: + ZOOKEEPER_CLIENT_PORT: 2181 + ZOOKEEPER_TICK_TIME: 2000 +``` + +### Purpose in This Project +- **Manages Kafka State**: Tracks broker health, configuration, and metadata +- **Leader Election**: Ensures cluster stability +- **Configuration Storage**: Maintains topic and permission info +- **Broker Discovery**: Keeps track of which brokers are available + +### Responsibilities + +```mermaid +graph LR + ZK["Zookeeper
(Port 2181)"] + + ZK -->|Monitors| BH["Broker Health"] + ZK -->|Manages| TC["Topic Config"] + ZK -->|Tracks| PA["Partition
Assignment"] + ZK -->|Handles| LE["Leader
Election"] + + style ZK fill:#4CAF50 + style BH fill:#E8F5E9 + style TC fill:#E8F5E9 + style PA fill:#E8F5E9 + style LE fill:#E8F5E9 +``` + +### Why It's Needed +- Kafka **requires** Zookeeper to function +- Without Zookeeper, Kafka can't coordinate +- Single instance fine for development +- Production would use Zookeeper cluster for HA + +### Startup +- Starts first (no dependencies) +- Other containers wait for it to be ready + +--- + +## Container 2: Kafka + +### Overview +**Asynchronous Message Broker** - Queues transactions for processing + +### Configuration +```yaml +Image: confluentinc/cp-kafka:7.5.0 +Ports: + - 9092:9092 (external: localhost:9092) + - 29092 (internal) (inside Docker: kafka:29092) +Network: payguard-network +Broker ID: 1 +Topic: transactions (auto-created) +``` + +### Message Flow Through Kafka + +```mermaid +sequenceDiagram + actor User as User
(Browser) + participant FE as Frontend
:3000 + participant API as API
:8000 + participant KAFKA as Kafka
:29092 + participant CONSUMER as Consumer
(Your Mac) + participant REDIS as Redis
:6379 + + User->>FE: 1. Submit Transaction + FE->>API: 2. POST /transaction + API->>KAFKA: 3. Publish to "transactions" + API-->>FE: 4. Return 200 OK (immediately) + FE-->>User: 5. "Transaction accepted!" + + CONSUMER->>KAFKA: 6. Subscribe to topic + KAFKA->>CONSUMER: 7. Deliver message + CONSUMER->>CONSUMER: 8. ML Processing (6 steps) + CONSUMER->>REDIS: 9. Store result + CONSUMER->>CONSUMER: 10. Done +``` + +### Data Structure in Kafka + +```json +Topic: "transactions" +Message: { + "user_id": "user_123", + "amount": 75.50, + "merchant": "Starbucks Coffee", + "description": "Morning coffee", + "currency": "USD", + "timestamp": "2026-04-24T21:30:00Z" +} +``` + +### Purpose in This Project +- **Decouples API from Consumer**: API doesn't wait for fraud detection +- **Handles High Throughput**: Can queue thousands of transactions +- **Persistence**: Transactions survive crashes +- **Scalability**: Multiple consumers can process same topic + +### Listener Configuration + +```mermaid +graph TB + CLIENT1["API Container
(inside Docker)"] + CLIENT2["Consumer
(on Mac)"] + KAFKA["Kafka Broker"] + + CLIENT1 -->|Uses:
kafka:29092| INTERNAL["PLAINTEXT
Listener:29092"] + CLIENT2 -->|Uses:
localhost:9092| EXTERNAL["PLAINTEXT_HOST
Listener:9092"] + + INTERNAL --> KAFKA + EXTERNAL --> KAFKA + + style KAFKA fill:#2196F3 + style CLIENT1 fill:#9C27B0 + style CLIENT2 fill:#FF9800 + style INTERNAL fill:#81C784 + style EXTERNAL fill:#81C784 +``` + +### Common Issues +- **API error 503**: Kafka not ready when API starts +- **Consumer can't connect**: Using wrong address (kafka:29092 from Mac won't work) +- **Messages not persisting**: Volume issue or broker crash + +--- + +## Container 3: Redis + +### Overview +**In-Memory Cache & Result Storage** - Stores transaction fraud detection results + +### Configuration +```yaml +Image: redis:7.2-alpine +Port: 6379 +Network: payguard-network +Volume: redis_data +Health Check: redis-cli ping +``` + +### Data Flow to Redis + +```mermaid +graph LR + CONSUMER["Consumer
(Your Mac)"] + ML["ML Pipeline
(6 Steps)"] + RESULT["Result:
fraud_score: 0.72
decision: FRAUD_DETECTED"] + REDIS["Redis
:6379"] + STORE["Key: transaction:txn_123
Value: JSON Result"] + + CONSUMER -->|Process| ML + ML -->|Generate| RESULT + RESULT -->|Store in| REDIS + REDIS -->|Persist| STORE + + style CONSUMER fill:#FF9800 + style ML fill:#FF9800 + style RESULT fill:#FFE0B2 + style REDIS fill:#FF9800 + style STORE fill:#FFE0B2 +``` + +### Result Data Structure + +```json +Key: transaction:txn_abc123 + +Value: { + "transaction_id": "txn_abc123", + "user_id": "user_xyz", + "amount": 75.50, + "merchant": "Starbucks Coffee", + "fraud_score": 0.12, + "decision": "APPROVED", + "risk_level": "LOW", + "timestamp": "2026-04-24T21:30:00Z", + "features": { + "amount_std_dev": 0.8, + "velocity": 0.1, + "geographic_score": 0.0, + "time_of_day": 0.05 + } +} +``` + +### Use Cases in System + +```mermaid +graph TB + CONSUMER["Consumer Writes"] + API_READ["API Reads
(optional)"] + SESSION["Session Cache
(optional)"] + REDIS["Redis"] + + CONSUMER -->|Store Result| REDIS + API_READ -->|Get Result| REDIS + SESSION -->|Cache Data| REDIS + + style REDIS fill:#FF9800 + style CONSUMER fill:#4CAF50 + style API_READ fill:#2196F3 + style SESSION fill:#9C27B0 +``` + +### Why It's Needed +- **Performance**: Instant result lookups (microseconds vs milliseconds) +- **Scalability**: Multiple API instances can share results +- **TTL Support**: Auto-expire old transactions +- **No Database**: Simpler architecture for POC +- **Pub/Sub**: Can notify about suspicious transactions + +### Persistence +- Data persists to `/data` volume +- Survives container restart +- RDB snapshots and AOF (append-only file) + +--- + +## Container 4: API (FastAPI) + +### Overview +**Transaction Intake & Validation Engine** - REST API for transaction submission + +### Configuration +```yaml +Build: + context: . + dockerfile: api/Dockerfile +Base Image: python:3.11-slim +Port: 8000 +Network: payguard-network +Framework: FastAPI +Language: Python +``` + +### Complete Transaction Flow Through API + +```mermaid +sequenceDiagram + actor User + participant Browser as Browser
localhost:3000 + participant API as API
localhost:8000 + participant JWT as JWT
Engine + participant KAFKA as Kafka
kafka:29092 + + User->>Browser: Click "Generate Token" + Browser->>API: POST /token?user_id=user_123 + API->>JWT: Create JWT (HS256) + JWT-->>API: eyJh... + API-->>Browser: {token, type: bearer} + Browser-->>User: Display Token + + User->>Browser: Fill transaction form + User->>Browser: Click Submit + Browser->>API: POST /transaction
Header: Authorization: Bearer eyJh... + API->>JWT: Validate JWT + JWT-->>API: Valid ✓ + API->>API: Validate schema + API->>KAFKA: Publish message + KAFKA-->>API: OK + API-->>Browser: 200 OK + API-->>Browser: {status: accepted, txn_id} + Browser-->>User: "Transaction accepted!" +``` + +### API Endpoints + +```mermaid +graph TB + FE["Frontend"] + + FE -->|POST /token
?user_id=X| TOKEN["🔑 Token Generation
Returns: JWT Token"] + FE -->|POST /transaction
+ JWT Header| TXN["💳 Transaction Submission
Validates & Publishes to Kafka"] + FE -->|GET /health| HEALTH["❤️ Health Check
Returns: {status: healthy}"] + + style FE fill:#F44336 + style TOKEN fill:#4CAF50 + style TXN fill:#2196F3 + style HEALTH fill:#FF9800 +``` + +### Key Features +- ✅ **JWT Authentication**: HS256 algorithm +- ✅ **CORS Enabled**: Allows http://localhost:3000 +- ✅ **Input Validation**: Pydantic models +- ✅ **Async Publishing**: Doesn't wait for consumer +- ✅ **Error Handling**: 503 if Kafka unavailable +- ✅ **Logging**: Structured JSON logs + +### Startup Dependencies + +```mermaid +graph TD + DOCKER["docker compose up"] + DOCKER -->|Wait| KAFKA_HEALTH["Kafka Healthy?"] + DOCKER -->|Wait| REDIS_HEALTH["Redis Healthy?"] + KAFKA_HEALTH -->|Yes| API_START["Initialize API"] + REDIS_HEALTH -->|Yes| API_START + API_START -->|Create routes| ROUTES["Setup endpoints"] + ROUTES -->|Setup| CORS["CORS middleware"] + CORS -->|Start| UVICORN["Uvicorn on :8000"] + UVICORN -->|Ready| DONE["✓ API Ready"] + + style DOCKER fill:#2196F3 + style KAFKA_HEALTH fill:#FF9800 + style REDIS_HEALTH fill:#FF9800 + style API_START fill:#4CAF50 + style DONE fill:#4CAF50 +``` + +--- + +## Container 5: Frontend (React) + +### Overview +**User Interface** - React web application for transaction submission + +### Configuration +```yaml +Build: + context: ./frontend + dockerfile: Dockerfile +Base: node:18-alpine +Port: 3000 +Network: payguard-network +Framework: React 18 + TypeScript +Build Tool: Vite +Server: serve +``` + +### User Interaction Flow + +```mermaid +graph TD + USER["👤 User
(on Mac)"] + BROWSER["🌐 Browser
localhost:3000"] + FE["⚛️ React Frontend
(in Docker)"] + API["🔌 API
:8000"] + + USER -->|Open| BROWSER + BROWSER -->|Renders| FE + USER -->|Click Generate| FE + FE -->|POST /token| API + API -->|Return JWT| FE + FE -->|Display Token| USER + USER -->|Fill Form| FE + FE -->|POST /transaction| API + API -->|Return 200| FE + FE -->|Show Success| USER + + style USER fill:#FF5722 + style BROWSER fill:#FFF3E0 + style FE fill:#F44336 + style API fill:#9C27B0 +``` + +### Build Process + +```mermaid +graph TB + DOCKER["docker build"] + + DOCKER -->|Stage 1| BUILDER["📦 Builder
node:18-alpine"] + BUILDER -->|install| DEPS["npm install"] + DEPS -->|build| BUILD["npm run build"] + BUILD -->|produces| DIST["dist/
optimized bundle"] + + DOCKER -->|Stage 2| RUNTIME["🚀 Runtime
node:18-alpine"] + RUNTIME -->|install| SERVE["npm install -g serve"] + SERVE -->|copy| COPY["COPY dist/ from builder"] + COPY -->|start| START["serve -s dist -l 3000"] + START -->|port| PORT["0.0.0.0:3000"] + PORT -->|ready| DONE["✓ Ready"] + + DIST -.->|reused in Stage 2| COPY + + style DOCKER fill:#2196F3 + style BUILDER fill:#4CAF50 + style RUNTIME fill:#2196F3 + style DONE fill:#4CAF50 +``` + +### Frontend to API Communication + +```mermaid +graph LR + subgraph Frontend["🌐 React Frontend
Browser Context"] + FORM["Transaction Form"] + STATE["React State"] + AXIOS["Axios Client"] + end + + subgraph API["🔌 FastAPI Backend
Docker Container"] + CORS["CORS Middleware"] + VALIDATION["Validation"] + KAFKA["Kafka Publish"] + end + + FORM -->|Fill| STATE + STATE -->|Submit| AXIOS + AXIOS -->|POST http://localhost:8000| CORS + CORS -->|Check origin| CORS + CORS -->|Allow| VALIDATION + VALIDATION -->|OK| KAFKA + KAFKA -->|Publish| KAFKA + KAFKA -->|Return 200| AXIOS + AXIOS -->|Show result| FORM + + style Frontend fill:#F44336 + style API fill:#9C27B0 + style CORS fill:#4CAF50 + style KAFKA fill:#2196F3 +``` + +--- + +## Complete Data Flow: End-to-End + +### Full Journey of a Transaction + +```mermaid +graph LR + subgraph User["👤 You (Browser)"] + BUY["Click Generate
Token Button"] + end + + subgraph Frontend_UI["⚛️ Frontend Container
:3000"] + FORM["Transaction Form"] + STATE["React State
with JWT"] + end + + subgraph Backend["🔌 API Container
:8000"] + JWT_CHECK["Validate
JWT"] + VALIDATE["Validate
Schema"] + end + + subgraph MessageQueue["🔹 Kafka Container
:29092"] + QUEUE["Queue
Message"] + end + + subgraph Processing["🖥️ Consumer (Your Mac)"] + EXTRACT["Feature
Extraction"] + ML1["ML Layer 1
Score"] + ML2["ML Layer 2
Score"] + DECISION["Decision
Engine"] + end + + subgraph Cache["🔹 Redis Container
:6379"] + STORE["Store
Result"] + end + + BUY -->|1. Request| Frontend_UI + Frontend_UI -->|2. HTTP POST| Backend + Backend -->|3. Validate| Backend + Backend -->|4. Publish| MessageQueue + MessageQueue -->|5. Queue| MessageQueue + MessageQueue -->|6. Receive| Processing + Processing -->|7. Process| Processing + Processing -->|8. Store| Cache + + style BUY fill:#FF5722 + style Frontend_UI fill:#F44336 + style Backend fill:#9C27B0 + style MessageQueue fill:#2196F3 + style Processing fill:#FF9800 + style Cache fill:#FF9800 +``` + +### Happy Path: Normal Transaction + +```mermaid +sequenceDiagram + participant Browser + participant Frontend + participant API + participant Kafka + participant Consumer + participant Redis + + Browser->>Frontend: 1. User submits transaction + activate Frontend + Frontend->>API: 2. POST /transaction (Amount: $75.50) + activate API + API->>API: 3. Validate JWT ✓ + API->>API: 4. Validate schema ✓ + API->>Kafka: 5. Publish to "transactions" + activate Kafka + API-->>Frontend: 6. Return 200 OK + deactivate API + Frontend-->>Browser: 7. Show "Accepted!" + deactivate Frontend + + Kafka->>Consumer: 8. Deliver message + deactivate Kafka + activate Consumer + Consumer->>Consumer: 9. Extract features + Consumer->>Consumer: 10. Layer 1 score: 0.15 + Consumer->>Consumer: 11. Layer 2 score: 0.12 + Consumer->>Consumer: 12. Decision: APPROVED + Consumer->>Redis: 13. Store result + deactivate Consumer + activate Redis + Redis->>Redis: 14. Persist to disk + deactivate Redis + + Note over Consumer,Redis: Fraud score < 0.3 = APPROVED ✓ +``` + +### Fraud Path: High-Value Transaction + +```mermaid +sequenceDiagram + participant Browser + participant Frontend + participant API + participant Kafka + participant Consumer + participant Redis + + Browser->>Frontend: 1. User submits transaction + activate Frontend + Frontend->>API: 2. POST /transaction (Amount: $5000) + activate API + API->>API: 3. Validate JWT ✓ + API->>API: 4. Validate schema ✓ + API->>Kafka: 5. Publish to "transactions" + activate Kafka + API-->>Frontend: 6. Return 200 OK (always) + deactivate API + Frontend-->>Browser: 7. Show "Accepted!" + deactivate Frontend + + Kafka->>Consumer: 8. Deliver message + deactivate Kafka + activate Consumer + Consumer->>Consumer: 9. Extract features + Note over Consumer: ⚠️ ANOMALY: Amount spike! + Consumer->>Consumer: 10. Layer 1 score: 0.68 + Consumer->>Consumer: 11. Layer 2 score: 0.72 + Consumer->>Consumer: 12. Decision: FRAUD_DETECTED + Consumer->>Redis: 13. Store result + deactivate Consumer + activate Redis + Redis->>Redis: 14. Persist to disk + deactivate Redis + + Note over Consumer,Redis: Fraud score > 0.7 = FRAUD_DETECTED 🚨 +``` + +--- + +## Container Networking + +### Docker Network Architecture + +```mermaid +graph TB + subgraph Host["🖥️ Your Mac (Host Machine)"] + PORT_MAPPING["Port Mapping:
3000→3000
8000→8000
9092→9092
6379→6379
2181→2181"] + end + + subgraph Docker["🐳 Docker Bridge Network
payguard-network"] + ZK["Zookeeper
:2181"] + KAFKA["Kafka
:29092 (internal)
:9092 (external)"] + REDIS["Redis
:6379"] + API["API
:8000"] + FE["Frontend
:3000"] + end + + subgraph Consumer["🖥️ Your Mac
Consumer Process"] + CONS["Consumer
localhost:9092"] + end + + HOST_CONSUMER["Your Mac
Browser"] + + HOST_CONSUMER -->|http://localhost:3000| FE + HOST_CONSUMER -->|http://localhost:8000| API + CONS -->|localhost:9092| KAFKA + + ZK -->|coordinates| KAFKA + API -->|kafka:29092| KAFKA + API -->|redis:6379| REDIS + FE -->|localhost:8000| API + + style Docker fill:#E3F2FD + style Host fill:#F3E5F5 + style Consumer fill:#FFF3E0 + style KAFKA fill:#2196F3 + style API fill:#9C27B0 + style REDIS fill:#FF9800 + style ZK fill:#4CAF50 +``` + +### Service Name Resolution + +```mermaid +graph LR + API_CODE["API Code:
kafka:29092"] + DNS["Docker DNS
Resolver"] + KAFKA_IP["Kafka Container IP
172.19.0.3"] + KAFKA["Kafka
Listening on
:29092"] + + API_CODE -->|Lookup| DNS + DNS -->|Resolve| KAFKA_IP + KAFKA_IP -->|Connect to| KAFKA + KAFKA -->|Accept| KAFKA + + style API_CODE fill:#9C27B0 + style DNS fill:#4CAF50 + style KAFKA_IP fill:#2196F3 + style KAFKA fill:#2196F3 +``` + +--- + +## Startup Sequence + +```mermaid +graph TD + START["docker compose up -d --build"] + + START -->|1. No deps| ZK["Zookeeper Starts"] + ZK -->|Ready| ZK_OK["✓ Zookeeper Ready
:2181"] + + ZK_OK -->|2. Waits for| KAFKA["Kafka Starts"] + KAFKA -->|Health check| KAFKA_HC["Kafka API Version OK?"] + KAFKA_HC -->|Yes| KAFKA_OK["✓ Kafka Ready
:29092"] + + KAFKA_OK -->|3. No deps| REDIS["Redis Starts"] + REDIS -->|Health check| REDIS_HC["redis-cli ping?"] + REDIS_HC -->|PONG| REDIS_OK["✓ Redis Ready
:6379"] + + KAFKA_OK -->|4. Waits for| API["API Starts"] + REDIS_OK -->|4. Waits for| API + API -->|Initialize| API_INIT["Setup FastAPI"] + API_INIT -->|Routes| API_READY["✓ API Ready
:8000"] + + API_READY -->|5. Waits for| FE["Frontend Starts"] + KAFKA_OK -->|5. Waits for| FE + REDIS_OK -->|5. Waits for| FE + FE -->|Build dist| FE_BUILD["npm run build"] + FE_BUILD -->|Serve| FE_READY["✓ Frontend Ready
:3000"] + + FE_READY -->|All Ready| DONE["🎉 System Ready!"] + + style START fill:#2196F3 + style DONE fill:#4CAF50 + style ZK_OK fill:#81C784 + style KAFKA_OK fill:#81C784 + style REDIS_OK fill:#81C784 + style API_READY fill:#81C784 + style FE_READY fill:#81C784 +``` + +--- + +## Communication Paths + +### Valid Connections + +```mermaid +graph TB + subgraph INSIDE["Inside Docker Network"] + API["API Container"] + KAFKA["Kafka Container"] + REDIS["Redis Container"] + end + + subgraph OUTSIDE["Outside Docker Network"] + MAC["Your Mac"] + BROWSER["Browser"] + CONSUMER["Consumer Process"] + end + + API -->|✅ kafka:29092| KAFKA + API -->|✅ redis:6379| REDIS + BROWSER -->|✅ localhost:8000| API + BROWSER -->|✅ localhost:3000| API + CONSUMER -->|✅ localhost:9092| KAFKA + + API -.->|❌ localhost:9092| KAFKA + MAC -.->|❌ kafka:29092| KAFKA + + style INSIDE fill:#E8F5E9 + style OUTSIDE fill:#FFF3E0 + style KAFKA fill:#2196F3 + style API fill:#9C27B0 + style REDIS fill:#FF9800 +``` + +--- + +## Resource Usage & Performance + +### Container Resource Estimates + +```mermaid +graph TB + TOTAL["Total System
~1.5 GB RAM"] + + TOTAL --> ZK["Zookeeper
100-200 MB"] + TOTAL --> KAFKA["Kafka
500-800 MB"] + TOTAL --> REDIS["Redis
50-100 MB"] + TOTAL --> API["API
200-300 MB"] + TOTAL --> FE["Frontend
50-100 MB"] + + style TOTAL fill:#FF6F00 + style ZK fill:#4CAF50 + style KAFKA fill:#2196F3 + style REDIS fill:#FF9800 + style API fill:#9C27B0 + style FE fill:#F44336 +``` + +### Typical Latency + +```mermaid +graph LR + USER["User
Action"] + API_CALL["API Call
<10ms"] + KAFKA["Kafka
Publish
<50ms"] + CONSUMER["Consumer
Receive
<100ms"] + ML["ML
Processing
200-500ms"] + REDIS["Redis
Store
<10ms"] + + USER -->|⏱️| API_CALL + API_CALL -->|⏱️| KAFKA + KAFKA -->|⏱️| CONSUMER + CONSUMER -->|⏱️| ML + ML -->|⏱️| REDIS + + style USER fill:#FF5722 + style API_CALL fill:#4CAF50 + style KAFKA fill:#2196F3 + style CONSUMER fill:#FF9800 + style ML fill:#FF9800 + style REDIS fill:#FF9800 +``` + +--- + +## Summary Table + +| Container | Image | Port | Purpose | Depends On | Status | +|-----------|-------|------|---------|-----------|--------| +| **Zookeeper** | confluentinc/cp-zookeeper | 2181 | Kafka coordination | None | Always up | +| **Kafka** | confluentinc/cp-kafka | 29092/9092 | Message broker | Zookeeper ✓ | Healthy | +| **Redis** | redis:alpine | 6379 | Result caching | None | Healthy | +| **API** | python:3.11-slim | 8000 | Transaction intake | Kafka, Redis ✓ | Running | +| **Frontend** | node:18-alpine | 3000 | User interface | API ✓ | Healthy | + +--- + +## Key Concepts + +### Asynchronous Processing +The API returns immediately (200 OK) without waiting for fraud detection results. Consumer processes transactions asynchronously from Kafka queue. + +### Service Discovery +Docker's embedded DNS server resolves service names (kafka, redis, etc.) to container IPs on the payguard-network. + +### Port Mapping +Exposes container ports to host machine: +- Internal communication: Container name + internal port (kafka:29092) +- External access: localhost + exposed port (localhost:9092) + +### Health Checks +Each container declares how to verify it's healthy. Other containers wait for dependencies to be healthy before starting. + +### Data Persistence +Redis volume ensures results survive container restarts. Kafka and Zookeeper also persist their data. + +--- + +This architecture enables: +✅ **Scalability** - Add more consumers for parallel processing +✅ **Reliability** - Kafka persists messages +✅ **Performance** - Async processing doesn't block frontend +✅ **Testability** - Each component can be tested independently +✅ **Maintainability** - Clear separation of concerns