Data streaming is defined as data that is being generated continuously by different data sources that update with a high frequency, in an almost real-time scenario. It is usually for a data type that has no end or beginning, but is continuous in nature.
Data Streaming refers to the instrumentation and processing of continuous data streams. It is often compared to Batching, the practice of moving data sets from storage to storage at triggered intervals. In this paradigm, data is considered “in motion” meaning when a data point is generated in a data source, it is immediately processed and passed on to consumer systems - instead of being collected in a storage service for future processing. Most of today’s source data is generated in a streaming fashion: transactions, logs, sensor data, social media feeds, clickstreams… By processing these streams in a Data Streaming paradigm, organizations can gain insights, detect anomalies and trends, and take action on the data while it’s being generated.
Standard Batch processing architecture
Hybrid processing Architecture
End to end live processing
The core benefits from Data Streaming are:
In the world of continuous data there can be some specific vocabulary, here are some exemples:
Confluent: Confluent is a technology company that was founded in 2014 by the original creators of Apache Kafka, a popular open-source distributed streaming platform. Confluent provides a comprehensive platform built around Kafka, called Confluent Platform. The Confluent Platform offers various tools, services, and enhancements to Apache Kafka, making it easier for developers and organizations to work with data streams.
Popsink: is a managed stream processing service. It aims at seamlessly integrating with existing Modern Data Stack solutions. Popsink’s focus is on abstracting away the operations on data streams to help users leverage continuous data from existing tools, without going through any migration or retraining.
Materialize: Materialize is an engine that enables the materialization of views on top of streaming data in SQL. The company builds both an Open Source and a Cloud Native offering of their technology. One of the core features of Materialize is its PostresSQL compatibility making it very easy to use exciting PostresSQL-compatible tools and services.
RisingWave Labs: RisingWave Labs is the company behind RisingWave. Similar to Materialize, RisingWave is an Open Source distributed SQL streaming engine. Written in Rust and compatible with PostresSQL, RisingWave is also available as a managed cloud offering.
Here are some amazing companies in the Data Streaming.
Risingwave Labs develops RisingWave, a cloud-native SQL streaming data ...
Amazon Kinesis is the fully managed Amazon Web Service (AWS) offering ...
Apache Flink is an open-source, unified stream-processing and batch-pr ...