Skip to content

Latest commit

 

History

History
110 lines (99 loc) · 6.02 KB

10.1.3. Integrations - Queues - Kinesis.md

File metadata and controls

110 lines (99 loc) · 6.02 KB
Raw

AWS Kinesis

  • Managed alternative to Apache Kafka
  • Use-cases
    • "Real-time" big data e.g. application logs, metrics, IoT, clickstreams.
    • Streaming processing frameworks (Spark, NIFI, etc...)
  • High availability through data replication to 3 AZ.
  • Has 4 different products:
    • Kinesis Streams (data stream): Ingest & process streaming data
    • Kinesis Firehouse (delivery stream): load streams into S3, Redshift, ElasticSearch
    • Kinesis Analytics: perform real-time analytics on streams using SQL
    • Kinesis video streams: Process & analyze streaming media in real-time.
  • 💡 Common integration: Feed Kinesis Streams -> Analyze data in real time using Kinesis Analytics -> Load streams into Kinesis Firehose -> S3/Redshift for long time retention.
  • Security
    • Control access / authorization using IAM policies
    • Encryption in flight using HTTPS endpoints
    • Encryption at rest using KMS
    • Possibility to encrypt / decrypt data client side (harder)
    • VPC Endpoints available for Kinesis to access within VPC

Comparison

Attribute Data Streams Firehouse Analytics Video streams
Description Blob (max 1 MB) ingestion Convert & load blob Query engine Video ingestion
Throughput/Limits Per shard and message size/amount per read/write, max 1 MB blob None: source data (max 1MB blob) None: source stream stream level (5TPS/H), connection-level (1/s), bandwidth MB/s, fragment-level
Latency Real time Near real time (60 secs latency) Real time Rendering + encryption (if stored)
Data retention 1 day (default), up to 7 days Retries during 24-hours then fails Can create streams based on queries Min 1 hour, max 10 years (in S3)
Scaling Add shards Auto-scales Auto-scales Auto-scales
Concepts Shards, Producers, Consumers Delivery stream, Record, Destination Input, Code, Application Video stream, Fragment, Producer, Consumer, Chunk
Use-cases Streams Convert & load streaming data into e.g. Redshift, S3, ElasticSearch, Splunk Real-time analytics using SQL Machine learning & analytics

Kinesis Data Streams

  • Low latency streaming ingest at scale like Kafka.
  • Streams are divided in ordered Shards / Partitions
  • Data goes to any shard & consumers consume from any shard
    • Producers -> Stream (Shard 1 / Shard 2 / Shard 3) -> Consumers
    • Pub/sub through streams (topic)
    • Multiple applications can consume the same stream
    • You get sequence number (checkpoint offset, should be saved) back for each data added
  • To scale: Add shards -> Increased throughput
  • Pricing is per shard provisioned, can have as many shard as possible
  • Data retention is 1 day by default, can go up to 7 days
  • Data
    • The message is base64-encoded blob
    • ❗ Data (before encoding) cannot exceed 1 MB
    • Ability reprocess / replay data as data is not removed after message is handled.
    • Immutable: Once data is inserted in Kinesis, it cannot be deleted
    • Batching available or per message calls
      • 💡 Use batching with PutRecords API to reduce costs and increase throughput
  • Security: You can enable server-side encryption with an AWS KMS master key
  • Monitoring
    • You can capture shard level metrics with CloudWatch at additional cost
      • For e.g. IncomingBytes, IncomingRecords, OutgoingBytes, WriteProvisionedThroughputExceeded, ReadProvisionedThroughputExceeded.
  • Shards
    • One stream is made of many different shards
    • The number of shards can evolve over time (re-shard / merge)
    • Records are ordered per shard
    • Partition key
      • Partition key gets hashed to determine the shard ID
      • Ensures ordering in a shard and same key always goes to same shard.
        • 💡 Choose a partition key that's highly distributed
          • 📝 Helps preventing hot partition or hot shard
    • Throughput
      • ❗ 1 MB/s or 1000 messages/s at write PER SHARD
      • ❗ 2 MB/s at read PER SHARD
      • ProvisionedThroughputExceeded if you go over the limits
      • 💡 Solution
        • Use retries with exponential back-off
        • Increase shards (scaling)
        • Ensure your partition key is a good one e.g. you don't have a hot shard
  • SDKs
    • Normal consumer (CLI, SDK, etc...)
    • Kinesis Client Library (in Java, Node, Python, Ruby, .NET)
      • Uses DynamoDB to checkpoint offsets
      • KCL uses DynamoDB to track other workers and share the work amongst shards

Kinesis Firehose

  • Fully managed service, no administration, automatic scaling
  • Near real time (60 seconds latency)
  • Load data into Redshift / Amazon S3 / ElasticSearch / Splunk
  • Support many data format (pay for conversion)
  • Pay for the amount of data going through Firehose
  • Concepts
    • Delivery stream: Create & send data into it
    • Record: Data of interest your data producer sends to a delivery stream
      • ❗ Max size before base-64 encoding is 1000KB
    • Destination: data storage, S3, Amazon Redshift, Amazon Elasticsearch Service, Splunk.

Kinesis Data Analytics

  • Perform real-time analytics on Kinesis Streams using SQL
  • Fully managed with auto scaling to match source stream throughput.
  • Pay for actual consumption rate
  • Can create streams out of the real-time queries

Kinesis Video Streams

  • Ingest video streams for e.g. analytics and machine learning.
  • HTTP Live Streaming (HLS) enables you to playback video for live and on-demand viewing
  • Uses Amazon S3 as the underlying data store.
  • Data at rest using KMS, IAM roles, and data in transit using TLS.
  • Concepts
    • Video stream: AWS resource that encrypts & time-stamps & stores.
    • Fragment: self-contained sequence of frames
    • Source: e.g. video-generating device, such as a security camera, a body-worn camera
    • Consumer: consume and process data in Kinesis video streams e.g. EC2 / AWS AI services / 3rd party.
    • Chunk: Stored data, consists of the actual media fragment