Design Approach: Build MVP First, Then Scale

This guide is structured as: 1) MVP (single-node, minimal working)
2) Scaling Steps 1 / 2 / 3 (Gradually add Kafka, Redis, DB sharding, and HA) 3) Entity / Data Model (核心表设计 + 关键字段)
4) End-to-End Flows (send/receive/offline/sync)
5) Tradeoffs & Interview Q&A (背诵版)

1) MVP: The Smallest Working Chat System (Single Instance)

MVP Goals

  • 1:1 messaging
  • Basic message history
  • Basic online delivery (best effort)
  • No Kafka / no Redis / no multi-gateway

MVP Architecture

  • Client (Web + Mobile)
  • Single Chat Server (includes WebSocket + REST)
  • Single DB (Postgres / MySQL) for users + conversations + messages

MVP Endpoints

WebSocket

  • WS /connect (auth during handshake)
  • sendMessage (client → server)
  • message (server → client push)

HTTP (REST)

  • POST /login
  • GET /conversations
  • GET /messages?conversationId&after=cursor

MVP Flow (Online)

  1. Client connects via WebSocket and authenticates
  2. Client sends message over WS
  3. Server writes message into DB
  4. Server pushes message to recipient if recipient is connected to the same server

MVP Flow (Offline)

  1. Sender sends message
  2. Server persists message to DB
  3. Receiver later opens app and pulls history via REST

MVP 关键：先保证能用（正确性），延迟/HA/极致扩展后面再加。

2) Scale Step 1: Split REST vs WebSocket + Stateless Chat Service

Why

  • REST 是短请求高 QPS
  • WebSocket 是长连接高并发连接数
  • 分开才能分别扩缩和调超时

Add Components

  • API LB + API Gateway (REST)
  • WS LB + WS Gateway (WebSocket)
  • Chat Service (stateless: validate + enqueue-only / or validate + write DB in early stage)

At this stage, you can still keep direct DB write (no Kafka yet) but split responsibilities.

3) Scale Step 2: Add Presence Store (Redis) for Multi-Gateway Routing

Problem

Once you have multiple WS Gateways, you need to know:

  • user is online?
  • connected to which gateway?

Add Component

  • Presence / Session Store (Redis)
    • userId → gatewayId
    • lastHeartbeat
    • TTL-based cleanup

Result

  • Delivery component can route to the correct WS gateway.
  • Gateway crash → reconnect → mapping rebuilt.

4) Scale Step 3: Add Kafka for Decoupling + Ordering + Backpressure

Why Kafka

  • Decouple ingestion (send path) from delivery (push path)
  • Smooth spikes (buffering)
  • Enable retry/replay
  • Ordering guarantee per conversation via partitioning

Add Components

  • Message Queue (Kafka): partition by conversationId
  • Delivery / Router (Kafka Consumer):
    • persist to DB (source of truth)
    • lookup Redis presence
    • push to WS gateway via internal RPC

Reliability Model

  • Delivery is at-least-once
  • Dedup via messageId idempotency

5) Scale Step 4 (Optional Advanced): HA, Sharding, Multi-Region

HA

  • WS Gateway multi-AZ behind LB
  • Redis cluster/replication
  • Kafka replication
  • DB replication + backups

Sharding

  • Cassandra/DynamoDB partition by conversationId
  • Avoid hot partitions for large groups via bucketing

Multi-Region (if asked)

  • Active-active with per-region Kafka + DB replication
  • Read local, write with conflict strategy (complex)

    Usually mention as extension unless interviewer pushes.

6) Entity Design (Hardcore, Interview-Ready)

Below is a practical data model that supports:

  • conversation list UI
  • message history
  • ordering
  • unread counts
  • multi-device sync

6.1 Core Entities

User

  • user_id (PK)
  • username
  • created_at
  • status (optional: active/blocked)

Conversation

  • conversation_id (PK)
  • type (DM GROUP)
  • created_at
  • created_by
  • title (group only)

ConversationMember

  • conversation_id (PK part)
  • user_id (PK part)
  • role (member admin)
  • joined_at
  • last_read_seq (for read/unread & sync)
  • muted_until (optional)

Message

  • conversation_id (partition key)
  • seq (clustering key / increasing per conversation)
  • message_id (global idempotency key)
  • sender_id
  • content
  • created_at
  • type (text, future: image/file)
  • edited_at (optional)
  • deleted (optional)

Key point: Ordering is by (conversation_id, seq) not by client timestamp.

Table: MessagesByConversation (source of truth)

  • PK: (conversation_id)
  • CK: (seq) or (created_at, message_id)
  • Columns: message_id, sender_id, content, created_at, ...

Table: ConversationsByUser (inbox UI)

  • PK: (user_id)
  • CK: (last_activity_time DESC, conversation_id)
  • Columns: last_message_preview, unread_count, pinned, ...

Table: MembersByConversation

  • PK: (conversation_id)
  • CK: (user_id)
  • Columns: role, joined_at, last_read_seq, ...

Idempotency / Dedup (optional)

  • PK: (message_id)
  • Columns: conversation_id, seq, status, created_at
  • TTL optional

    Used to ensure retries don’t create duplicates.

6.3 Presence / Session Store (Redis)

Key Patterns

  • presence:{userId} -> gatewayId (TTL)
  • heartbeat:{userId} -> timestamp (TTL)
  • Optional:
    • inbox:{userId} -> [messageId...] (short-term cache)
    • unread:{conversationId}:{userId} -> count

Redis is acceleration only, not correctness.

7) End-to-End Flows (Aligned with Entities)

7.1 Send Message (Online)

  1. Client sends sendMessage(conversationId, messageId, content) over WS
  2. WS Gateway forwards to Chat Service (internal API)
  3. Chat Service validates membership (ConversationMember) + rate limit
  4. Chat Service enqueues to Kafka (partition by conversationId)
  5. Delivery Router consumes:
    • allocate seq (monotonic per conversation) and persist Message
    • update ConversationsByUser for sender/receiver (last_activity, preview)
    • lookup Redis presence for receiver
    • push to receiver’s WS gateway

7.2 Receive Message (Push)

  • Router → WS Gateway: internal RPC push(message)
  • WS Gateway → Client: WebSocket message event

7.3 Offline Delivery + Recovery

  • If receiver not online (no presence mapping):
    • still persist to DB
    • optionally update Redis inbox/unread cache
    • optionally trigger push notification
  • On reconnect:
    • client sends last_read_seq per conversation
    • server returns messages where seq > last_read_seq

7.4 Multi-Device Sync

  • Each device stores per conversation last_read_seq
  • On reconnect / app open:
    • pull deltas from MessagesByConversation(conversationId, seq > last_read_seq)

8) Interview “Layered” Explanation (Memorize)

8.1 MVP pitch (10s)

Start with a single chat server + DB: authenticate, persist messages, push if online, otherwise clients fetch history.

8.2 Scale pitch (20s)

Then split REST and WS gateways, add Redis presence for routing, add Kafka to decouple send/deliver and guarantee per-conversation ordering, store messages in Cassandra as the source of truth.

8.3 Correctness pitch (10s)

Cassandra is durable source of truth; Kafka provides buffering/replay; Redis accelerates presence and unread state; delivery is at-least-once with messageId idempotency.

9) Quick Q&A (Hardcore)

When is it persisted?

When written to Cassandra/DB as source of truth.

Gateway crash: message loss?

No. Client reconnects; presence rebuilt; missing messages pulled from DB.

How does router find user?

Redis userId → gatewayId mapping with heartbeat TTL.

How do you guarantee ordering?

Kafka partition by conversationId + per-conversation seq in DB.

Why not exactly-once?

Too complex; at-least-once + idempotency is the practical standard.

Notes (How to draw on whiteboard)

  • Start with MVP boxes (Client → Chat Server → DB)
  • Then evolve to: API LB/GW and WS LB/GW
  • Add Redis presence mapping
  • Add Kafka + Delivery Router
  • Finally annotate: ordering (partition), idempotency (messageId), source of truth (DB)