Design: Offline-First Sync Engine

📖 Concept

System Design: Offline-First Data Sync Engine

This is one of the hardest mobile system design problems and a frequent Staff-level interview topic.

Requirement Clarification:

  • Users can read and write data with NO internet connection
  • When connectivity returns, changes sync bidirectionally
  • Conflicts must be resolved without data loss
  • Support for shared/collaborative data (multiple users editing)
  • Sync must be efficient (delta-based, not full copies)
  • Scale: 5M users, each with 10-100MB of local data

Architecture:

Client
  ├── Local Database (SQLite / WatermelonDB)
  │   ├── Data tables (entities)
  │   ├── Change log (operations journal)
  │   └── Sync metadata (cursors, versions)
  ├── Sync Engine
  │   ├── Change Tracker (detect local changes)
  │   ├── Push Manager (upload changes)
  │   ├── Pull Manager (download changes)
  │   ├── Conflict Resolver
  │   └── Queue Manager (background sync)
  └── Network Monitor (connectivity status)

Server
  ├── Sync API (cursor-based, delta responses)
  ├── Conflict Resolution Service
  ├── Change Log (event sourcing)
  └── Storage (PostgreSQL + Redis cache)

Conflict Resolution Strategies:

Strategy How it works Best for
Last-Write-Wins (LWW) Highest timestamp wins Simple data, settings
Merge Combine non-conflicting changes Structured documents
CRDT Mathematically conflict-free Collaborative editing
User-prompted Show both versions, user decides Critical data

Key Design Decisions:

  1. Operation-based sync (send operations, not full documents) — more efficient, enables merge
  2. Vector clocks for ordering events across devices — more reliable than wall clock
  3. Cursor-based pagination for pull sync — efficient for large datasets
  4. Background sync with WorkManager — reliable even if app is killed

💻 Code Example

codeTap to expand ⛶
1// === OFFLINE-FIRST SYNC ENGINE ===
2
3// 1. Change tracking with operation log
4interface SyncOperation {
5 id: string;
6 entityType: string; // 'note', 'task', 'contact'
7 entityId: string;
8 operation: 'create' | 'update' | 'delete';
9 changes: Record<string, any>; // Field-level changes
10 timestamp: number;
11 deviceId: string;
12 synced: boolean;
13}
14
15class ChangeTracker {
16 async trackChange(
17 entityType: string,
18 entityId: string,
19 operation: 'create' | 'update' | 'delete',
20 changes: Record<string, any>
21 ) {
22 const op: SyncOperation = {
23 id: generateUUID(),
24 entityType,
25 entityId,
26 operation,
27 changes,
28 timestamp: Date.now(),
29 deviceId: getDeviceId(),
30 synced: false,
31 };
32
33 await localDB.insertOperation(op);
34
35 // Try immediate sync if online
36 if (networkMonitor.isConnected) {
37 syncEngine.schedulePush();
38 }
39 }
40}
41
42// 2. Sync engine with conflict resolution
43class SyncEngine {
44 private isSyncing = false;
45
46 async fullSync() {
47 if (this.isSyncing) return;
48 this.isSyncing = true;
49
50 try {
51 // Push first (reduces conflict window)
52 await this.push();
53 // Then pull
54 await this.pull();
55 } finally {
56 this.isSyncing = false;
57 }
58 }
59
60 private async push() {
61 const pendingOps = await localDB.getUnsyncedOperations();
62
63 if (pendingOps.length === 0) return;
64
65 const response = await api.pushChanges({
66 deviceId: getDeviceId(),
67 operations: pendingOps,
68 lastSyncCursor: await localDB.getSyncCursor(),
69 });
70
71 // Handle conflicts returned by server
72 for (const conflict of response.conflicts) {
73 await this.resolveConflict(conflict);
74 }
75
76 // Mark synced operations
77 await localDB.markOperationsSynced(
78 pendingOps.map(op => op.id)
79 );
80 }
81
82 private async pull() {
83 let cursor = await localDB.getSyncCursor();
84 let hasMore = true;
85
86 while (hasMore) {
87 const response = await api.pullChanges({
88 cursor,
89 limit: 100,
90 deviceId: getDeviceId(), // Exclude own changes
91 });
92
93 for (const change of response.changes) {
94 await this.applyRemoteChange(change);
95 }
96
97 cursor = response.nextCursor;
98 hasMore = response.hasMore;
99
100 await localDB.setSyncCursor(cursor);
101 }
102 }
103
104 private async resolveConflict(conflict: ConflictInfo) {
105 const local = await localDB.getEntity(conflict.entityType, conflict.entityId);
106 const remote = conflict.serverValue;
107
108 switch (conflict.entityType) {
109 case 'settings':
110 // Last-write-wins for settings
111 if (remote.updatedAt > local.updatedAt) {
112 await localDB.upsertEntity(conflict.entityType, remote);
113 }
114 break;
115
116 case 'note':
117 // Field-level merge for notes
118 const merged = this.fieldLevelMerge(local, remote, conflict.baseValue);
119 await localDB.upsertEntity(conflict.entityType, merged);
120 break;
121
122 case 'transaction':
123 // Never auto-resolve financial data — queue for user review
124 await localDB.markConflict(conflict.entityType, conflict.entityId, {
125 local,
126 remote,
127 });
128 break;
129 }
130 }
131
132 private fieldLevelMerge(local: any, remote: any, base: any): any {
133 const merged = { ...base };
134
135 for (const key of Object.keys(merged)) {
136 const localChanged = local[key] !== base[key];
137 const remoteChanged = remote[key] !== base[key];
138
139 if (localChanged && !remoteChanged) {
140 merged[key] = local[key]; // Only local changed
141 } else if (!localChanged && remoteChanged) {
142 merged[key] = remote[key]; // Only remote changed
143 } else if (localChanged && remoteChanged) {
144 // Both changed — LWW for this field
145 merged[key] = local.updatedAt > remote.updatedAt
146 ? local[key]
147 : remote[key];
148 }
149 // Neither changed — keep base value
150 }
151
152 return merged;
153 }
154}
155
156// 3. Background sync with WorkManager (via native module)
157// Schedule periodic sync when app is backgrounded
158import BackgroundFetch from 'react-native-background-fetch';
159
160BackgroundFetch.configure({
161 minimumFetchInterval: 15, // minutes
162 stopOnTerminate: false,
163 startOnBoot: true,
164 enableHeadless: true,
165}, async (taskId) => {
166 await syncEngine.fullSync();
167 BackgroundFetch.finish(taskId);
168}, (taskId) => {
169 BackgroundFetch.finish(taskId);
170});

🏋️ Practice Exercise

Offline-First Design Exercises:

  1. Design the complete database schema for a note-taking app with sync support
  2. Implement a field-level merge algorithm that handles concurrent edits to different fields
  3. Build a sync status indicator that shows: synced, syncing, pending changes, conflict
  4. Design a conflict resolution UI that shows both versions and lets the user merge
  5. Implement cursor-based delta sync that handles pagination efficiently
  6. Discuss: how would you handle schema migrations when the app is offline and the server has evolved?

⚠️ Common Mistakes

  • Using wall clock time for conflict resolution — device clocks can be wrong; use logical clocks or server-assigned timestamps

  • Syncing full documents instead of deltas — wastes bandwidth and increases conflict surface area

  • Not handling schema evolution — if the server adds a field while the client is offline, sync breaks

  • Not implementing retry with backoff — failed syncs should retry with exponential backoff, not flood the server

  • Assuming sync is instant — design for multi-second sync on large datasets; show progress to the user

💼 Interview Questions

🎤 Mock Interview

Mock interview is powered by AI for Design: Offline-First Sync Engine. Login to unlock this feature.