L1 cache reference:                     0.5 ns
Branch mispredict:                      5 ns
L2 cache reference:                     7 ns
Mutex lock/unlock:                    100 ns
Main memory reference:                100 ns
Compress 1KB with Snappy:           3,000 ns (3 µs)
Send 1KB over 1 Gbps network:      10,000 ns (10 µs)
Read 1MB sequentially from memory: 250,000 ns (250 µs)
Round trip within same datacenter:  500,000 ns (500 µs)
Read 1MB sequentially from SSD:   1,000,000 ns (1 ms)
Disk seek:                        10,000,000 ns (10 ms)
Read 1MB sequentially from disk:  20,000,000 ns (20 ms)
Send packet CA → Netherlands:    150,000,000 ns (150 ms)

Formula: (X million × Y% × Z requests) / (86,400 seconds/day)

Example: 1B users, 50% DAU, 10 requests/day
= (1,000,000,000 × 0.5 × 10) / 86,400
= 5,000,000,000 / 86,400
= ~57,870 QPS

Peak traffic rule: Multiply by 3-5x (peak is 3-5x average)
= 57,870 × 5 = ~290,000 QPS peak

Formula: X million × Y × Z = total storage

Example: 1B users, 1KB per user, 30 days retention
= 1,000,000,000 × 1KB × 30
= 30TB total

But need redundancy:
- 3x replication: 30TB × 3 = 90TB
- 2x for backup: 30TB × 2 = 60TB
- Total: ~150TB

Memory: 64GB
Max connections: 10,000
Throughput: 
  - 10,000 QPS read (cached)
  - 1,000 QPS write
  - Mixed: 5,000 QPS

Before hitting CPU/memory limits, disk I/O becomes bottleneck

Data size: > 1TB → Consider sharding
QPS: > 10,000 → Consider sharding
Connections: > 5,000 → Consider sharding

Sharding key: User ID, geographic region, or hash

Memory per connection: ~1MB
Max connections: 32,000

QPS capacity:
- Simple operations: ~10,000 QPS
- Complex operations: ~1,000 QPS
- CPU-bound: ~5,000 QPS

Formula: Peak QPS / QPS per server

Example: 300,000 peak QPS, 10,000 QPS per server
= 300,000 / 10,000 = 30 servers

With redundancy (rolling updates, failures):
= 30 × 1.3 = 39 servers (30% buffer)

Modern LB (AWS ALB):
- Max 25,000 new connections/second
- Max 1,000,000 concurrent connections per LB

For 10M concurrent connections:
= 10,000,000 / 1,000,000 = 10 LBs needed

For multi-region HA:
= 10 LBs × 2 regions = 20 LBs total

Example without cache:
- 100,000 QPS to database
- Database capacity: 10,000 QPS
- Problem: Overloaded

With cache (90% hit rate):
- Cache serves: 90,000 QPS (in-memory, fast)
- DB serves: 10,000 QPS (miss traffic)
- Balanced!

Formula: (QPS × avg_object_size × TTL) / hit_rate

Example: 100,000 QPS, 1KB avg object, 1 hour TTL, 90% hit rate
= (100,000 × 1KB × 3,600) / 0.9
= 360,000,000 KB / 0.9
= ~400GB cache

Practical: Use distributed cache (Redis) across 10 servers
= 40GB per Redis instance (reasonable)

Formula: Peak QPS × Avg response size

Example: 300,000 peak QPS, 10KB response
= 300,000 × 10KB = 3,000,000 KB/s = 3GB/s

Bandwidth needed:
- 10 Gbps link: 10GB/s (can handle it)
- But need redundancy: 2 × 10Gbps = 20Gbps total

Rule of thumb: Provision 2-3x peak bandwidth for headroom

Netflix scenario: 100M users, 50% watching simultaneously

Bitrate per stream: 5 Mbps (HD)
Total bandwidth needed: 50M × 5 Mbps = 250 Tbps

Wow, that's massive! How does Netflix handle it?

Solution: CDN + regional caching
- Cache video in edge locations (ISP networks)
- Only ~10-20% requires backbone traffic
- Backbone: 250 Tbps × 15% = 37.5 Tbps (still huge, but manageable)

1. Calculate QPS:
   - 100M DAU × 20 requests/day / 86,400 = ~23,000 QPS avg
   - Peak: 23,000 × 5 = ~115,000 QPS

2. Database sizing:
   - Each feed = 100 posts × 1KB = 100KB
   - 100M users × 100KB = 10TB storage
   - With 3x replication: 30TB
   - Needs sharding (> 1TB)

3. Server capacity:
   - 115,000 peak QPS / 10,000 per server = 11.5 → 15 servers
   - With 30% redundancy: ~20 servers

4. Cache sizing:
   - 90% hit rate assumption
   - Feed objects: 100M users × 100KB × 0.1 (10% miss) = 1TB
   - Split across 25 Redis instances = 40GB each

5. Bandwidth:
   - 115,000 QPS × 100KB = 11.5GB/s
   - Need 100 Gbps backbone

✓ How many users? (total, DAU, peak concurrent)
✓ Request rate? (QPS average, QPS peak)
✓ Data volume? (per user, total, growth rate)
✓ Latency requirements? (response time SLA)
✓ Availability requirements? (uptime SLA)
✓ Bandwidth constraints? (network capacity)
✓ Storage retention? (how long to keep data)
✓ Consistency requirements? (strong vs eventual)

Then size:
1. Servers needed
2. Database needed
3. Cache needed
4. Bandwidth needed
5. Storage needed

do {
  cur = load(addr)                    // Read current
  next = cur + 1                      // Calculate new
} while (!CAS(addr, cur, next))       // Retry if someone changed it

// Without CAS (using lock):
lock(counter_lock)
counter++
unlock(counter_lock)
// Problem: Lock contention on hot counters

// With CAS (lock-free):
do {
  current = load(counter)
  new = current + 1
} while (!CAS(counter, current, new))
// Faster: No lock, just retry loop
// Contention is handled by retries, not waiting

// Scenario: Multiple threads updating a user record version
// Thread A: Read user version=3, wants to update
// Thread B: Updates version to 4 (increments it)
// Thread A's update should fail since version changed

do {
  user = read(user_id)
  version = user.version
  user.name = "new name"
  user.version = version + 1
} while (!CAS(user_obj, version, version+1))

// This ensures: "Only commit if nobody else changed this"

// Stack: head → [A] → [B] → [C]
// Thread wants to pop A

do {
  top = load(head)                    // top = [A]
  next = top.next                     // next = [B]
} while (!CAS(head, top, next))       // Replace head with [B]

// If another thread pushed new node between read and CAS:
// CAS fails, loop retries with new head

// States: IDLE (0), RUNNING (1), STOPPED (2)
enum State { IDLE = 0, RUNNING = 1, STOPPED = 2 }

// Only transition IDLE → RUNNING
do {
  state = load(state_var)
  if (state != IDLE) return false     // Can't start, not idle
} while (!CAS(state_var, IDLE, RUNNING))

// Guarantees: Only one thread successfully transitions to RUNNING

Operation       | Lock Time  | CAS Time   | Winner
----------------|------------|------------|--------
Increment       | 50 ns      | 5 ns       | CAS (10x faster)
Read + Update   | 80 ns      | 10 ns      | CAS (8x faster)

Threads | Lock QPS    | CAS QPS     | Winner
--------|-------------|-------------|--------
1       | 1,000,000   | 5,000,000   | CAS
4       | 200,000     | 3,000,000   | CAS
16      | 50,000      | 1,500,000   | CAS (still!)
64      | 10,000      | 500,000     | CAS (now degrading)
256     | 2,000       | 50,000      | CAS (with backoff)

Thread 1: reads value = A
         (context switch)
Thread 2: changes A → B → A
         (thread 1 resumes)
Thread 1: CAS(addr, A, new_value)  ✓ Succeeds!
         But the A now is different from the original A!

// Stack: [A] → [B]
// Thread 1 tries to pop A
head = load(&stack)          // head = [A]
next = head.next             // next = [B]
                             // (Thread 2 pops [A], then pushes it back)
// Now A is a freed/reused node!
CAS(&stack, head, next)      // ✓ Succeeds, but now pointing to invalid [A]!

// Store: (value, version)
// CAS(addr, (expected_val, expected_ver), (new_val, new_ver+1))

do {
  (value, version) = load(addr)
  new_version = version + 1
} while (!CAS(addr, (value, version), (new_val, new_version)))
// Version changes guarantee uniqueness

struct VersionedRef {
  void* ptr;           // The actual pointer
  uint64_t generation; // Incremented on each reuse
}
// CAS on entire struct ensures ABA safety

// Thread marks pointer as "in-use"
// Other threads cannot recycle it
// Slower but fully ABA-safe

x86/x64:        CAS, CMPXCHG (single & double-wide)
ARM:            LDREX, STREX (LL/SC - Load-Link/Store-Conditional)
Modern CPUs:    Compare-And-Swap, Load-Acquire, Store-Release (memory barriers)

Java:           java.util.concurrent.atomic.AtomicInteger.compareAndSet()
C++:            std::atomic::compare_exchange_strong()
Go:             sync/atomic.CompareAndSwapUint64()
Rust:           std::sync::atomic::AtomicUsize::compare_exchange()
Python:         multiprocessing.Value (limited); usually use locks

// Exponential backoff
uint32_t attempts = 0;
do {
  cur = load(addr)
  next = cur + 1
  if (!CAS(addr, cur, next)) {
    // Failed, back off exponentially
    sleep(min(1ms << attempts++, 100ms))
  }
} while (CAS failed)

Pattern                          | Trade-off
---------------------------------|------------------------------------------
Coarse lock                       | Simple, but high contention cost
Sharded locks (per-core)          | Good balance, less contention
CAS with backoff                  | Fast but needs tuning
Partitioned atomic variables      | Best for counters, split work across threads
Lock-free queues                  | Complex, but scale well
Eventual consistency              | Highest throughput, but weaker guarantees

Enterprise customers: 99.99% uptime (52 minutes/year downtime)
  ↓
Standard customers: 99.9% uptime (8.76 hours/year downtime)
  ↓
Free tier: 99% uptime (3.65 days/year downtime)

Why tiered?
- Premium customers need higher reliability
- Premium tier → higher cost → justifies more ops investment

99% uptime: 1 failure/100 requests (cheap infra)
99.9% uptime: 1 failure/1000 requests (replicas + monitoring)
99.99% uptime: 1 failure/10,000 requests (multi-region, chaos testing)
99.999% uptime: 1 failure/100,000 requests (enterprise-grade)

Each 9 costs ~3-5x more than the previous level

SLO: 99.9% uptime per month
Total month hours: 730 hours
Error budget: (1 - 0.999) × 730 = 0.73 hours = 43.8 minutes/month

If already used 30 minutes, only 13.8 minutes remaining
→ Can't do risky deployments
→ Must be extra cautious (no canary → full rollout)

# Example implementation
error_budget_remaining = calculate_budget_remaining()

if error_budget_remaining > 50%:
    # Aggressive: canary deployment, feature flags
    deploy_with_canary()
    
elif error_budget_remaining > 10%:
    # Conservative: manual rollout, heavy monitoring
    deploy_manually_with_heavy_monitoring()
    
else:
    # Critical: freeze all deployments
    only_fix_critical_bugs()
    no_new_features()

Incident: User auth service down for 10 minutes
Impact: 10,000 failed requests / 1,000,000 total = 1% error rate

Error budget burn:
  If SLO is 99.9% (0.1% errors allowed)
  This incident used: 1% / 0.1% = 10x the budget
  
Result: Error budget depleted for rest of month

Bad SLO: 99.99% on single-region system (impossible)
Good SLO: 99% on standard tier, 99.9% on premium

Reality: Systems with single points of failure max out at ~99%
Multi-region systems can reach 99.99%

# Don't just optimize uptime
SLO for Netflix:
  - Uptime: 99.99%
  - Latency (P99): < 1 second
  - Error rate: < 0.1%
  
Optimizing only uptime can hurt latency
(always return cached/stale data = 100% uptime but bad UX)

Infrastructure: 99.99% availability
SLO: 99.9% availability (leave 0.09% buffer)

Why buffer?
- Deployments, maintenance
- Network blips
- Monitoring false positives

Alert when:
  - P95 latency exceeds 500ms (threshold breach)
  - P95 latency increases 50% week-over-week (trend)
  
Second alert catches degradation before total failure

Compute Engine:
  - 99.95% availability
  - Credits if breached:
    - 99%-99.95%: 10% credit
    - 95%-99%: 30% credit
    - <95%: 100% credit
    
Design: SLO (99.9%) < SLA (99.95%)
If SLO breached, still might make SLA
Extra buffer for one-time incidents

EC2:
  - 99.99% availability SLA
  - Applied per instance
  - Multiple instances in different AZs recommended
  
If one instance fails:
  - That instance gets credits
  - Other instances unaffected

- 99.99% uptime required (financial services)
- <50ms latency for payment processing
- Incident response: <15 minutes
- Requires multi-region active-active setup

# Example: Monitor SLO burn rate
def check_slo_burn():
    # SLO: 99.9% uptime = 0.1% error rate allowed
    
    current_error_rate = get_error_rate_last_hour()
    
    if current_error_rate > 1%:
        # 10x error budget burn rate
        # Will deplete monthly budget in ~3 hours
        alert("CRITICAL: SLO burn rate 10x. Incident response needed")
    
    elif current_error_rate > 0.5%:
        # 5x error budget burn rate
        alert("WARNING: High SLO burn rate. Investigate")
    
    elif current_error_rate > 0.15%:
        # 1.5x error budget burn rate (acceptable)
        log("Normal SLO burn rate")

Normal state:
Request → Service A → Success ✓

Failure state (too many failures):
Request → Circuit Breaker (OPEN) → Fail fast ✗ (don't call failing service)

Recovery state (after timeout):
Request → Circuit Breaker (HALF-OPEN) → Try service A
         ↓
         Success → Close circuit (back to normal)
         Failure → Open circuit again

Month 1-3: Optimize queries + vertical scale (buy time)
Month 3-6: Implement sharding (parallel effort)
Month 6+: Run sharded system (replaces single DB)

User request → Load balancer (multi-AZ)
             ↓
      API Gateway (stateless)
             ↓
   Microservices (Video, Recommendation, Billing, etc.)
             ↓
   Each service has:
   - Multiple instances (horizontal scale)
   - Read replicas (HA)
   - Circuit breakers (cascading failure prevention)
   - Cache layer (Redis)
   - Timeout/retry logic

Video service crashes
  ↓
Circuit breaker opens (fails fast)
  ↓
User gets cached video list
  ↓
Meanwhile: Auto-scaler spins up new instances
  ↓
Service recovers in 30 seconds
  ↓
Circuit breaker closes (normal)

# Set with TTL
redis.setex("user:123", 3600, user_data)

# On update
db.update_user(123, new_data)
redis.publish("user:123:updated", json.dumps(new_data))

# Subscriber
def on_user_update(message):
    user_id = message['user_id']
    redis.delete(f"user:{user_id}")  # Immediate invalidation
    # Next request re-populates from DB

Multiple requests for same key after expiry
→ All hit DB simultaneously
→ Database overloaded

Solution: Probabilistic early expiry

---

### Q4: Design a system to survive an entire AWS region failure.

**Answer:**
**Active-Active multi-region** (required):

**Database replication**:

**Cache/state**:

**Cost trade-off**:

---

### Q5: How many load balancers do you need for 10M concurrent connections?

**Answer:**
**Calculation**:

**LB architecture**:

**Connection distribution**: