Comparing Kafka, Pulsar, and Google Pub/Sub is very relevant, since all three are used for event streaming / messaging, but their architectures and trade-offs differ a lot.
🔹 1. Overview
-
Apache Kafka 🟦
- Distributed log-based streaming platform.
- Stores messages in partitions on brokers.
- Strong ecosystem (Kafka Streams, ksqlDB, Connect).
-
Apache Pulsar 🟧
- Distributed pub-sub messaging + streaming system.
- Separates storage (BookKeeper) from serving (brokers).
- Supports both queueing (competing consumers) and streaming (multiple subscribers).
-
Google Cloud Pub/Sub ☁️
- Fully managed pub-sub messaging service by Google Cloud.
- No infrastructure to manage, pay per message.
- Focus on global scale + simplicity, not deep control.
🔹 2. Architecture
| Feature | Kafka 🟦 | Pulsar 🟧 | Google Pub/Sub ☁️ |
|---|---|---|---|
| Message Storage | On brokers (distributed log) | Apache BookKeeper (separate storage layer) | Fully managed, abstracted |
| Serving vs Storage | Coupled (brokers do both) | Decoupled (brokers handle traffic, BookKeeper handles storage) | Abstracted (Google infra) |
| Persistence | Strong durability, replicated logs | Strong durability, ledger-based | Durable (replicated across regions) |
| Scalability | Partition-based scaling | Horizontal scaling with brokers + BookKeeper | Scales automatically (global) |
| Multi-tenancy | Limited (mostly per-cluster) | First-class support (namespaces, tenants) | Multi-tenant managed service |
| Geo-replication | MirrorMaker / Confluent tools | Built-in, flexible | Built-in (global) |
🔹 3. Performance & Latency
- Kafka → Low latency (ms), very high throughput, but scaling storage requires partition rebalancing.
- Pulsar → Comparable latency, scales better for very large topic counts due to storage/serving separation.
- Pub/Sub → Higher latency (tens to hundreds of ms), but scales automatically with less operational effort.
🔹 4. Features
| Feature | Kafka 🟦 | Pulsar 🟧 | Pub/Sub ☁️ |
|---|---|---|---|
| Message Ordering | Per partition | Per key / subscription | Best-effort (ordering keys available) |
| Message Retention | Configurable (time/size) | Configurable (time/backlog) | 7 days (default), 31 days (max) |
| Replay / Reprocessing | Yes (consumer offset reset) | Yes (ack-based + cursor management) | Yes (Seek API) |
| Streams API | Kafka Streams, ksqlDB | Pulsar Functions, Pulsar IO | Use Dataflow / Beam |
| Transactions | Supported | Supported | Limited (exactly-once via Dataflow) |
| Multi-tenancy | Weak | Strong | Built-in (per project) |
🔹 5. Operational Complexity
- Kafka → High ops burden: manage brokers, Zookeeper (older), partitions, rebalancing, scaling. Managed Kafka (Confluent, MSK) helps.
- Pulsar → More complex than Kafka (brokers + BookKeeper + ZooKeeper), but better long-term scalability (esp. for very large topic counts).
- Pub/Sub → Zero ops (fully managed). You just publish/subscribe.
🔹 6. Use Cases
✅ Kafka
- High-throughput event streaming.
- Log aggregation, ETL pipelines, real-time analytics.
- Strong ecosystem for stream processing.
- Example: LinkedIn activity streams, fraud detection.
✅ Pulsar
- Multi-tenant environments.
- Hybrid use: streaming + traditional queueing.
- IoT scale (millions of topics).
- Example: Tencent, Verizon, Splunk use it for global-scale messaging.
✅ Google Pub/Sub
- Cloud-native apps (especially on GCP).
- Event-driven microservices without ops overhead.
- Bursty workloads with unpredictable scaling.
- Example: Cloud event ingestion, Firebase push events, IoT.
🔹 7. Summary Table
| Aspect | Kafka 🟦 | Pulsar 🟧 | Pub/Sub ☁️ |
|---|---|---|---|
| Deployment | Self-hosted / Managed | Self-hosted (complex) | Fully managed |
| Scaling Model | Partition-based | Broker + BookKeeper separation | Auto-scaled |
| Latency | Low (ms) | Low (ms) | Medium (10s–100s ms) |
| Multi-tenancy | Limited | Strong | Built-in |
| Geo-Replication | Add-ons | Native | Native |
| Ecosystem | Rich (Streams, ksqlDB, Connect) | Growing (Functions, IO, connectors) | Integrates with GCP tools |
| Best For | Real-time analytics, streaming pipelines | Multi-tenant, global-scale eventing | Cloud-native apps, zero-ops |
👉 Rule of Thumb:
- Choose Kafka if you want a battle-tested streaming platform with rich ecosystem and can manage ops (or use Confluent/MSK).
- Choose Pulsar if you want multi-tenancy, decoupled storage/compute, and large-scale flexibility (though ops are harder).
- Choose Pub/Sub if you’re in Google Cloud and want serverless eventing with zero ops (but accept less control & higher latency).