GCP REAL-TIME ANALYTICS ARCHITECTURE: 100K events/second

ARCHITECTURE OVERVIEW

Data Flow: Event Source → Pub/Sub → Dataflow → BigQuery → Looker/API
Latency: Ingestion to dashboard: 30-45 seconds
Cost estimate: $12K-18K monthly at steady state
Scaling: Designed to 500K events/sec without architecture change

LAYER 1: DATA INGESTION

Service: Cloud Pub/Sub
→ Topic: events-raw (partitioned by event_type)
→ Message rate: 100,000 msg/sec sustained
→ Retention: 7 days (storage buffer)
→ Throughput tier: Premium (99.95% availability SLA)
→ Estimated cost: $2,200/month
→ Subscriber model: Pull (not push) for backpressure control

Producer side:
→ SDK: Google Cloud Client Library (Python/Node)
→ Batch settings: 100 messages or 50ms, whichever first (optimize throughput)
→ Retry policy: Exponential backoff, max 3 attempts
→ Dead-letter topic: events-raw-dlq (capture failures for replay)

Message schema (Avro, enforced):
fields: event_id (string), event_type (enum), user_id (string), timestamp_ms (long), properties (object), source (string)

LAYER 2: STREAM PROCESSING

Service: Cloud Dataflow (Apache Beam)
→ Template: Pub/Sub to BigQuery
→ Runtime: 50 worker instances (n1-standard-4 each)
→ Autoscaling: Min 50, max 200 workers (scale on CPU > 70%)
→ Job mode: Streaming (continuous, no batch windows)
→ Windowing: 60-second tumbling windows for aggregates
→ Estimated cost: $8,200/month

Transformations (in Dataflow):
1. Deduplication: Event ID + source hash (in-memory cache)
2. Schema validation: Reject malformed events → DLQ
3. Enrichment: Join user_id → user segment (from Cloud SQL)
4. Aggregation: Sum events per event_type per 60-sec window
5. Filtering: Drop internal/test events (by source)

LAYER 3: STORAGE

Service: BigQuery (Warehouse)
→ Dataset: analytics_prod (multi-region: US)
→ Table: events (partitioned by ingestion_timestamp, clustered by event_type)
→ Retention: 2-year rolling window
→ Estimated cost: $3,800/month (at 100K events/sec = 8.64B events/day)

BigQuery schema:
CREATE TABLE analytics_prod.events (
  event_id STRING NOT NULL,
  event_type STRING NOT NULL,
  user_id STRING,
  timestamp_ms INT64,
  properties JSON,
  source STRING,
  ingestion_timestamp TIMESTAMP,
)
PARTITION BY DATE(ingestion_timestamp)
CLUSTER BY event_type, source;

Querying patterns (pre-built):
→ Last 24h conversion count: < 1 second
→ User journey by segment: < 5 seconds
→ Event trends (30-day): < 10 seconds

LAYER 4: ANALYTICS SERVING

Service 1: Looker (BI/Dashboards)
→ Real-time dashboard: Events per minute, top event types, error rate
→ Refresh: 5-minute cache (balance freshness vs. query cost)
→ Estimated cost: $3,500/month (standard tier)

Service 2: Custom API (Cloud Run)
→ Endpoint: /api/analytics/summary?hours=24
→ Query: Hits BigQuery, caches result in Cloud Memorystore (Redis) for 60s
→ Latency: < 500ms p95
→ Concurrency: Handles 10K req/sec (autoscaling Cloud Run)
→ Estimated cost: $800/month

OPERATIONAL LAYERS

Monitoring: Cloud Monitoring + Cloud Logging
→ Alerts: Event lag > 60sec, DLQ growing, Dataflow job failure
→ Dashboards: 4 custom dashboards (ingestion, processing, storage, serving latency)
→ Log retention: 30 days
→ Estimated cost: $400/month

Disaster Recovery:
→ Backup: BigQuery snapshots every 24h (7-day retention)
→ Failover: Dataflow job can restart from Pub/Sub (7-day buffer)
→ RTO: 15 minutes (restart pipeline + warm cache)
→ RPO: 60 seconds (one 60-sec window)

Scaling for 10x Growth (1M events/sec)
→ Pub/Sub: Increase partitions from 10 → 100 (linear scale)
→ Dataflow: Workers scale to 500 (autoscaling handles)
→ BigQuery: Already handles 10x (no change needed)
→ Estimated cost increase: +$80K/month → $92-108K/month total

COST BREAKDOWN

Service | Monthly Cost | Notes
Pub/Sub | $2,200 | Premium tier, 100K msg/sec
Dataflow | $8,200 | 50-200 workers, hourly compute
BigQuery | $3,800 | Storage + queries
Looker | $3,500 | BI dashboard licensing
Cloud Run | $800 | API serving + caching
Cloud Memorystore | $600 | Redis 2GB cache
Monitoring | $400 | Logs + dashboards
Cloud SQL | $1,200 | User segment lookup table
---
TOTAL | $20,700 | Monthly recurring

ALTERNATIVES CONSIDERED

Option A: BigQuery Streaming Inserts (no Dataflow)
Pro: Simpler, fewer moving parts
Con: 2x cost ($900 per TB), less transformation flexibility
VERDICT: Rejected (Dataflow more economical at scale)

Option B: Apache Kafka + self-managed Spark
Pro: Full control, potentially cheaper
Con: Operational overhead, 24/7 DevOps needed
VERDICT: Rejected (managed services reduce risk)

Option C: Firestore + Cloud Functions
Pro: Serverless, minimal ops
Con: Limited to 10K writes/sec, 3x cost for this scale
VERDICT: Rejected (BigQuery + Dataflow is the right tier)

DEPLOYMENT CHECKLIST

Create Pub/Sub topic + DLQ
Deploy Dataflow template + set autoscaling
Create BigQuery dataset + table with partitioning/clustering
Set up Looker dashboards + refresh schedule
Deploy Cloud Run API + enable autoscaling
Configure Cloud Memorystore Redis instance
Enable monitoring alerts (lag, errors, costs)
Run load test: simulate 100K events/sec for 1 hour
Test failover: kill Dataflow job, verify restart recovery
Document runbook: on-call procedures for common issues