Message Queue Fundamentals

0/3 in this phase0/45 across the roadmap

📖 Concept

A message queue is a middleware that enables asynchronous communication between services. Instead of Service A calling Service B directly (synchronous), Service A puts a message on the queue, and Service B processes it when ready.

Why Message Queues?

Problem	How Queues Solve It
Tight coupling	Services communicate via messages, not direct calls
Overwhelming downstream	Queue acts as buffer, consumer processes at its own pace
Retry complexity	Queue retries failed messages automatically
Availability dependency	Producer works even if consumer is down
Traffic spikes	Queue absorbs bursts, consumer processes steadily

Core Concepts

Producer: Creates and sends messages to the queue
Consumer: Reads and processes messages from the queue
Queue/Topic: The channel that holds messages
Broker: The server managing queues (RabbitMQ, Kafka, SQS)
Dead Letter Queue (DLQ): Where failed messages go after max retries

Delivery Guarantees

Guarantee	Description	Example
At-most-once	Message may be lost, never duplicated	Logging, metrics
At-least-once	Message delivered 1+ times, may have duplicates	Order processing (with idempotency)
Exactly-once	Message delivered exactly once	Payment processing (hardest to achieve)

Queue vs Topic (Pub/Sub)

Pattern	Queue (Point-to-Point)	Topic (Pub/Sub)
Consumers	One consumer gets each message	All subscribers get every message
Use case	Task distribution, work queues	Event broadcasting, notifications
Example	Email sending queue	"Order created" event to inventory, shipping, and analytics

Pro tip: In interviews, use message queues whenever you see "process this later," "notify multiple services," or "handle traffic spikes."

💻 Code Example

codeTap to expand ⛶

1// ============================================
2// Message Queue Patterns
3// ============================================
4
5// ---------- Basic Queue Pattern ----------
6class MessageQueue {
7  constructor() {
8    this.queues = new Map();
9    this.deadLetterQueue = [];
10  }
11
12  publish(queueName, message) {
13    if (!this.queues.has(queueName)) this.queues.set(queueName, []);
14    this.queues.get(queueName).push({
15      id: Math.random().toString(36).slice(2),
16      data: message,
17      timestamp: Date.now(),
18      retries: 0,
19      maxRetries: 3,
20    });
21  }
22
23  async consume(queueName, handler) {
24    const queue = this.queues.get(queueName) || [];
25    while (queue.length > 0) {
26      const message = queue.shift();
27      try {
28        await handler(message.data);
29        console.log(`✅ Processed: \${message.id}`);
30      } catch (error) {
31        message.retries++;
32        if (message.retries < message.maxRetries) {
33          queue.push(message); // Retry
34          console.log(`⏳ Retry \${message.retries}/\${message.maxRetries}: \${message.id}`);
35        } else {
36          this.deadLetterQueue.push(message); // Move to DLQ
37          console.log(`❌ DLQ: \${message.id} after \${message.maxRetries} retries`);
38        }
39      }
40    }
41  }
42}
43
44// ---------- Async Order Processing ----------
45
46// ❌ BAD: Synchronous — user waits for everything
47async function processOrderSync(order) {
48  await validateOrder(order);           // 50ms
49  await chargePayment(order);           // 200ms
50  await updateInventory(order);         // 100ms
51  await sendConfirmationEmail(order);   // 300ms
52  await notifyWarehouse(order);         // 150ms
53  await updateAnalytics(order);         // 100ms
54  return { status: 'completed' };       // 900ms total!
55}
56
57// ✅ GOOD: Async — user gets fast response
58async function processOrderAsync(order, queue) {
59  await validateOrder(order);                    // 50ms
60  const payment = await chargePayment(order);    // 200ms
61
62  if (payment.success) {
63    // Queue remaining tasks
64    await queue.publish('order.confirmed', {
65      orderId: order.id, items: order.items, userId: order.userId,
66    });
67    return { status: 'confirmed' };              // 250ms total!
68  }
69  return { status: 'payment_failed' };
70}
71
72// Background workers process events independently:
73// Worker 1: order.confirmed → updateInventory
74// Worker 2: order.confirmed → sendConfirmationEmail
75// Worker 3: order.confirmed → notifyWarehouse
76// Worker 4: order.confirmed → updateAnalytics
77
78// ---------- Fan-out Pattern (One event → Multiple consumers) ----------
79class EventBus {
80  constructor() { this.subscribers = new Map(); }
81
82  subscribe(eventType, handler) {
83    if (!this.subscribers.has(eventType)) this.subscribers.set(eventType, []);
84    this.subscribers.get(eventType).push(handler);
85  }
86
87  async publish(eventType, data) {
88    const handlers = this.subscribers.get(eventType) || [];
89    await Promise.all(handlers.map(h => h(data)));
90  }
91}
92
93// Usage
94const bus = new EventBus();
95bus.subscribe('user.registered', async (data) => { console.log('Send welcome email:', data.email); });
96bus.subscribe('user.registered', async (data) => { console.log('Create default settings:', data.userId); });
97bus.subscribe('user.registered', async (data) => { console.log('Track analytics:', data.userId); });
98bus.publish('user.registered', { userId: '123', email: 'user@test.com' });
99
100async function validateOrder(o) {}
101async function chargePayment(o) { return { success: true }; }
102async function updateInventory(o) {}
103async function sendConfirmationEmail(o) {}
104async function notifyWarehouse(o) {}
105async function updateAnalytics(o) {}

🏋️ Practice Exercise

Async Refactor: Refactor a synchronous user registration flow (validate → create user → send email → create settings → log analytics) into async using message queues.
Dead Letter Queue: Design a DLQ handler for a payment processing queue. What happens to failed payments? How do you retry? Alert? Manually resolve?
Delivery Guarantees: For each scenario, choose at-most-once, at-least-once, or exactly-once: (a) logging, (b) payment, (c) email notification, (d) inventory update.
Queue vs Direct Call: When should you use a message queue vs direct HTTP call between services? Give 3 examples of each.

⚠️ Common Mistakes

Using queues for everything — synchronous calls are simpler when both services must be available and response time matters. Don't add a queue between your API and its database.
Not handling message failures — without retry logic and dead letter queues, failed messages are silently lost. Always configure retries with exponential backoff and DLQs.
Ignoring message ordering — most queues don't guarantee order across partitions. If order matters, use a single partition or sequence numbers with consumer-side reordering.
Not making consumers idempotent — with at-least-once delivery, consumers may process the same message twice. Use idempotency keys to prevent duplicate side effects.

💼 Interview Questions

🎤 Mock Interview

Practice a live interview for Message Queue Fundamentals

Was this topic helpful?

← PreviousRedis & Memcached Deep Dive Next →Apache Kafka & Event Streaming