System Architecture Decisions and Reviews

REAL-TIME EVENT PLATFORM - SYSTEM ARCHITECTURE

PROBLEM STATEMENT
• Ingest: 100K events/sec (peak)
• Latency: Query event count within 5 sec (quorum read)
• Retention: 30 days hot, 1 year cold
• Consistency: Eventual consistency acceptable (events are immutable)
• Availability: 99.95% uptime SLA

ARCHITECTURE OVERVIEW

INGESTION LAYER
→ Event source (user app, webhook, SDK)
→ Apache Kafka cluster (6 brokers, 3-partition topic per event type)
→ [Justification] Kafka handles 100K msg/sec throughput with sub-10ms p99 latency; replication factor 3 ensures durability

PROCESSING LAYER
→ Kafka Stream or Flink job (event validation, deduplication, enrichment)
→ Output: validated events routed to two destinations (hot + cold storage)
→ [Justification] Streaming processor drops duplicates (idempotency by event_id) and enriches events with metadata (user timezone, account type)

HOT STORAGE (for real-time queries, <30 days)
→ Apache Cassandra cluster (12 nodes, replication factor 3)
→ Table schema: events (partition_key: account_id+date, sort_key: timestamp, columns: event_type, payload, user_id)
→ [Justification] Cassandra: write optimized (append-only), handles 100K writes/sec, query latency <50ms p99 (meets 5-sec requirement for sub-second queries)
→ Compaction: Time-window compaction, delete records older than 30 days automatically

COLD STORAGE (for historical queries, 30d-1y)
→ S3 (Parquet files, partitioned by date and event_type)
→ Athena (SQL queries on S3 data) or Snowflake (if data warehouse needed)
→ [Justification] Cost-optimized: S3 is 1/10th the cost of Cassandra for archival

QUERY LAYER
→ API Gateway (rate limit 1000 req/sec per account)
→ Query service (microservice) handles two query patterns:
  • Hot query (last 7 days) → hits Cassandra → <100ms latency
  • Cold query (older data) → hits Athena → <30s latency
→ Caching: Redis layer (1-hour TTL) for high-volume queries ("give me today's event count")

MONITORING & OBSERVABILITY
→ Prometheus + Grafana (metrics: event throughput, Cassandra latency, Kafka lag)
→ ELK Stack (event logs) for debugging individual event failures
→ Alerting: PagerDuty triggers if Kafka lag > 5 min OR Cassandra p99 latency > 200ms

FAILOVER & DISASTER RECOVERY
→ Kafka replication factor 3 (survives 2 broker failures)
→ Cassandra replication factor 3 (survives 2 node failures)
→ Cross-region standby: Secondary Cassandra cluster in different AZ, synced via change-data-capture (CDC)
→ Recovery time objective (RTO): <5 min if primary region fails

TRADE-OFFS DOCUMENTED

TRADE-OFF 1: Consistency vs Speed
• We chose: Eventual consistency (Cassandra)
• Alternative: Strong consistency (PostgreSQL with Raft)
• Rationale: Events are immutable facts. Users accept eventual consistency (5-sec lag) in exchange for 10x higher throughput.
• Risk: If user updates account membership and immediately queries events, they might see events from a moment before. Mitigated via read-repair in Cassandra.

TRADE-OFF 2: Hot vs Cold Storage Split
• We chose: Separate systems (Cassandra for hot, S3/Athena for cold)
• Alternative: Single database (DuckDB or ClickHouse for everything)
• Rationale: Cost savings on cold data (S3) outweigh operational complexity. Hot queries are 95% of traffic; Cassandra optimized for that.

TRADE-OFF 3: Streaming Processing Language
• We chose: Apache Flink
• Alternative: Kafka Streams (lighter), Spark Streaming (batch-like)
• Rationale: Flink has best-in-class state management for deduplication; native support for complex windowed aggregations needed for enrichment.

DEPLOYMENT TOPOLOGY
• Region: AWS US-East-1
• K8s cluster: 30 nodes (reserved instances), 3-node Cassandra statefulset, 6-broker Kafka statefulset
• Auto-scaling: Query service scales 5-50 pods based on QPS load (target 500 req/pod)

MILESTONES
• Week 1-2: Kafka cluster setup + validation under load testing (simulate 100K events/sec)
• Week 3-4: Cassandra cluster + Flink deduplication pipeline
• Week 5: Query service API + Redis caching layer
• Week 6: Monitoring setup (Prometheus, PagerDuty alerts)
• Week 7: Disaster recovery testing + documentation
• Week 8: Production launch (canary deploy to 10% of events first)

KEY ASSUMPTIONS
• Event payload < 1KB on average
• Account count: <100K (governs sharding strategy)
• 90% of queries are within last 7 days
• Event deduplication required (SDK retries = duplicate events possible)

ARCHITECTURE DECISION RECORD
→ File: /docs/ADR-001-event-platform-architecture.md
→ Date: 2026-07-31
→ Status: Approved by CTO
→ Rationale: See trade-offs section above