- Streaming systems process and transport data in real-time, enabling event-driven architectures.
- Core capability: ingest, buffer, route, and deliver high-volume event streams with low latency.
- Use when you need real-time analytics, event sourcing, or microservices communication.
- Fits between data producers (apps, logs, sensors) and consumers (analytics, databases, services).
- Mental model: an append-only log, partitioned for scale, consumed by multiple independent clients.
- Key players/tools: Apache Kafka, Apache Pulsar, AWS Kinesis; also Google Pub/Sub, Redis Streams.
- Trade-offs: throughput vs. latency, ordering guarantees vs. scalability, cost vs. durability.
- Architecture consideration: partitioning strategy, consumer group design, retention settings, failover.
- Production gotcha: backpressure, consumer lag, message replay semantics, schema evolution.
- Success metric: sustained throughput, end-to-end latency, message durability, consumer lag.
Streaming systems form the backbone of modern event-driven architectures, enabling organizations to process, transport, and react to data as it arrives. Unlike traditional batch systems, streaming platforms such as Kafka, Pulsar, and Kinesis allow continuous ingestion and delivery of events, supporting real-time analytics, monitoring, and business logic. These systems are designed for high throughput and low latency, making them ideal for use cases like fraud detection, personalization, and telemetry.
Technically, streaming platforms implement distributed, append-only logs partitioned for scale and fault tolerance. Producers write events to topics or streams, while consumers read and process these events independently and at their own pace. Features such as consumer groups, offset management, and retention policies provide flexibility and reliability. Modern architectures often integrate streaming systems with cloud-native services, storage, and compute engines to build robust, scalable data pipelines.
Their importance is underscored by the need for responsive data-driven applications, where insights and actions must be derived from live data. However, operating streaming systems in production requires careful attention to scaling, ordering, consistency, schema evolution, and cost management. Companies like Netflix, Uber, and Airbnb rely heavily on these platforms to orchestrate complex, global, real-time data flows.
Topic/Stream: A logical channel or partitioned log where producers write events and consumers read them.
Why it matters: Defines boundaries for data organization, scalability, and parallelism.
Consumer Group: A set of consumers collaborating to read from a topic, each handling a subset of partitions.
Why it matters: Enables horizontal scaling and fault tolerance for event processing.
Offset Management: Tracking the position of a consumer within a partitioned log to support replay and recovery.
Why it matters: Crucial for reliability, exactly-once semantics, and handling failures.
Retention Policy: Rules controlling how long events are stored in the stream before deletion.
Why it matters: Balances storage cost, replay capability, and compliance requirements.
All changes to application state are captured as a sequence of events in a stream.
Use Case: Audit logging, state reconstruction, temporal analytics (e.g., Uber trip history).
Events ingested, transformed, and aggregated on-the-fly for dashboards or alerts.
Use Case: Fraud detection, telemetry, personalization (e.g., Netflix recommendations).
Microservices publish and subscribe to events via streams for decoupled integration.
Use Case: Order fulfillment, payment processing, notification systems (e.g., Airbnb booking workflow).
Streaming systems like Kafka, Pulsar, and Kinesis scale horizontally by partitioning topics/streams and distributing load across brokers/nodes. However, scaling requires careful partition key selection and monitoring of broker health, disk usage, and network bandwidth. Over-partitioning or under-partitioning can impact performance and operational complexity.
Low latency is a key feature, but can be affected by network hops, consumer lag, disk I/O, and batching settings. Tuning producer/consumer configurations, optimizing hardware, and minimizing cross-data-center replication are essential for sub-second event delivery.
Most streaming systems provide at-least-once delivery by default, with optional exactly-once semantics at higher cost and complexity. Ordering guarantees are typically per partition, not global, so applications must design for eventual consistency and handle out-of-order events.
Cost includes infrastructure (brokers, storage), data transfer, and operational overhead. Retention policies, compression, partition counts, and throughput requirements directly affect spend. Managed services (e.g., Kinesis, Confluent Cloud) offer convenience but can be expensive at scale.
This code creates a Kafka producer in Python, serializes a login event as JSON, and sends it to the 'events' topic. Flushing ensures the message is actually sent.
from kafka import KafkaProducer
import json
producer = KafkaProducer(
bootstrap_servers=['localhost:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8')
)
producer.send('events', {'user_id': 123, 'action': 'login'})
producer.flush()This snippet fetches records from an AWS Kinesis stream using a shard iterator, enabling real-time consumption of events for downstream processing.
import boto3
client = boto3.client('kinesis')
response = client.get_records(
ShardIterator='YOUR_SHARD_ITERATOR',
Limit=10
)
for record in response['Records']:
print(record['Data'])Use Case: Real-time personalization and recommendations.
Implementation: Uses Apache Kafka to transport user activity events to downstream ML pipelines that compute recommendations in near real-time.
Outcomes: Improved user engagement and retention due to timely, relevant recommendations.
Use Case: Trip telemetry and fraud detection.
Implementation: Streams trip data and payment events using Apache Pulsar, enabling event-driven microservices for anomaly detection and operational monitoring.
Outcomes: Faster fraud detection, reduced losses, and improved rider safety.
Choosing poor partition keys can lead to hotspots and uneven load distribution.
✅ Solution: Analyze key cardinality and event volume, and test partitioning strategies before production.
Consumers falling behind due to slow processing or high event rates.
✅ Solution: Monitor lag metrics, scale consumer groups, and optimize processing logic.
Changing event schemas without backward/forward compatibility breaks consumers.
✅ Solution: Use schema registries, versioning, and compatibility checks as standard practice.
Failing to monitor broker health, lag, and throughput can lead to silent failures.
✅ Solution: Integrate with observability platforms (e.g., Prometheus, Datadog) and set up alerts.
Assigning all partitions to one consumer, negating parallelism and fault tolerance.
Why avoid: Creates bottlenecks and single points of failure.
✅ Instead: Distribute partitions across a consumer group for scalable, resilient processing.
Setting retention to infinite leads to runaway storage costs and performance degradation.
Why avoid: Increases costs, risks disk fill and broker crashes.
✅ Instead: Set appropriate retention policies based on business needs and compliance.
Assuming global ordering of events in a distributed system.
Why avoid: Leads to subtle bugs, especially in stateful consumers.
✅ Instead: Design for per-partition ordering and handle out-of-order events in application logic.
Rationale: Enables horizontal scaling and fault tolerance.
Example: Kafka consumer group with 8 workers each reading different partitions of a topic.
Rationale: Ensures schema compatibility and prevents breaking changes.
Example: Confluent Schema Registry validating Avro/JSON schemas for Kafka topics.
Rationale: Proactively detects issues and enables quick remediation.
Example: Grafana dashboards visualizing consumer lag and broker throughput.
Rationale: Optimizes storage cost and performance.
Example: Kafka topic with 7-day retention and snappy compression for event logs.
Expected answer: By partitioning topics across brokers, replicating data, and enabling consumer groups for parallel, resilient processing.
Expected answer: For financial transactions, per-account ordering is required. Use partition key based on account ID and design consumers to process events per partition, handling occasional out-of-order events in logic.
Expected answer: Use schema registries, enforce backward/forward compatibility, version schemas, and validate changes before deploying.
Expected answer: At-least-once is simpler and faster but may cause duplicates; exactly-once prevents duplicates but increases complexity and latency.
Expected answer: Monitor lag metrics with dashboards and alerts, analyze slow consumers, scale consumer groups, and optimize processing logic.
- Kafka, Pulsar, and Kinesis are leading streaming platforms.
- Topics/streams are partitioned for scale and parallelism.
- Consumer groups enable fault-tolerant, distributed processing.
- Offsets track consumer progress for replay and recovery.
- Retention policies control storage and replay window.
- Schema registries help manage event formats safely.
- Partitioning strategy affects scalability and ordering.
- Consumer lag signals system health and performance.
- Schema evolution must be managed for compatibility.
Focus: Integrating streaming systems for service communication and data pipelines.
Concerns: API design, event formats, handling ordering and replay, fault tolerance.
Focus: Ensuring reliability, monitoring, scaling, and disaster recovery for streaming clusters.
Concerns: Lag, broker health, disk usage, failover procedures, alerting.
Focus: Consuming real-time data for feature extraction and model inference.
Concerns: Data freshness, schema changes, scalable consumption, replay for model training.
Focus: Designing event-driven AI/ML pipelines for low-latency prediction and feedback loops.
Concerns: Integration with batch and streaming sources, consistency, durability, cost.
Focus: Defining product requirements for real-time features and analytics.
Concerns: Latency SLAs, data quality, system scalability, feature roadmap.
Focus: Ensuring secure event transmission, access control, and compliance.
Concerns: Encryption in transit and at rest, authentication, authorization, audit logging.
Once you're comfortable with Streaming Systems, explore these related concepts...