You can use Amazon Kinesis Data Streams to collect and process big data record streams in real time. You can create a data processing application called Kinesis Data Streams application. A typical Kinesis Data Streams application reads the data in the data stream as a data record. These applications can use Kinesis Client Library and can run on Amazon EC2 instances. The processed records can be sent to the control panel and used to generate alerts, dynamically change pricing and advertising strategies, or send the data to a variety of other AWS services.
What can Kinesis Data Streams be used for?
You can use Kinesis Data Streams for fast and continuous data introduction and aggregation. The types of data used can include IT infrastructure log data, application logs, social media, market data sources, and web clickstream data. Since the response time of data import and processing is real-time, processing is usually lightweight.
The following is a typical scenario for using Kinesis Data Streams:
- accelerated log and data source introduction And processing
-
You can let the creator directly push data into the stream. For example, push system and application logs, they can be ready in a few seconds for processing. This can prevent log data loss due to front-end or application server failure. Kinesis Data Streams provides accelerated data source import because you do not batch data on the server before submitting the data for import.
- Real-time metrics and reports
-
You can use the data collected in Kinesis Data Streams for real-time Simple data analysis and reporting. For example, your data processing application can handle system and application log metrics and reports, because data will flow in instead of waiting to receive bulk data.
- Real-time data analysis
-
This can combine the power of parallel processing with the value of real-time data. Combine. For example, processing website clickstreams in real time, and then using multiple different Kinesis Data Streams applications running in parallel to analyze site availability participation.
- Complex stream processing
-
Can create Kinesis Data Streams applications and data streams directed or not Ring graph (DAG). This usually involves putting data from multiple Kinesis Data Streams applications into other streams for downstream processing by other Kinesis Data Streams applications.
Benefits of using Kinesis Data Streams
Although Kinesis Data Streams can be used to solve various Streaming data problems, but its common use is to aggregate data in real time, and then load the aggregated data into a data warehouse or map-reduce cluster.
Put data into the Kinesis data stream to ensure durability and resiliency. The delay between the time the record is put into the stream and the time the record can be retrieved (put-to-get delay) is usually less than 1 second. In other words, after adding data, the Kinesis Data Streams application can start using the data in the stream almost immediately. The hosting service aspect of Kinesis Data Streams can reduce the operational burden of creating and running data introduction pipelines. It is possible to create streaming map-reduce-type applications. Using the elasticity of Kinesis Data Streams, streams can be expanded or contracted so that data records are never lost before they expire.
Multiple Kinesis Data Streams applications can use the data in the stream so that multiple operations (such as archiving and processing) can be performed concurrently and independently. For example, two applications can read data in the same stream. The first application calculates the running aggregates and updates the Amazon DynamoDB table, and the second application compresses and archives the data to a data storage, such as Amazon Simple Storage Service (Amazon S3). The control panel will then read the DynamoDB table with running aggregations for real-time reports (minute level).
Kinesis Client Library supports fault-tolerant usage of data in streams and provides extended support for Kinesis Data Streams applications.
- Accelerated log and data source introduction and processing
-
You can Let the creator directly push the data into the stream. For example, push system and application logs, they can be ready in a few seconds for processing. This can prevent log data loss due to front-end or application server failure. Kinesis Data Streams provides accelerated data source import because you do not batch data on the server before submitting the data for import.
- Real-time metrics and reports
-
You can use the data collected in Kinesis Data Streams for real-time Simple data analysis and reporting. For example, your data processing application can handle system and application log metrics and reports, because data will flow in instead of waiting to receive bulk data.
- Real-time data analysis
-
This can combine the power of parallel processing with the value of real-time data. Combine. For example, processing website clickstreams in real time, and then using multiple different Kinesis Data Streams applications running in parallel to analyze site availability participation.
- Complex stream processing
-
Can create Kinesis Data Streams applications and data streams directed or not Ring graph (DAG). This usually involves putting data from multiple Kinesis Data Streams applications into other streams for downstream processing by other Kinesis Data Streams applications.