Skip to content

Latest commit

 

History

History
43 lines (36 loc) · 3.53 KB

10.1. Integrations - Queues - Overview.md

File metadata and controls

43 lines (36 loc) · 3.53 KB
Raw

Queues

Patterns of communication

  • Two patterns of communication:
    • Synchronous (application to application)
      • Can be problematic if there's sudden spikes of traffic
        • E.g. if you need to suddenly encode 1000 videos but it's 10?
      • Better to decouple your application with queues for scalability and resiliency.
    • Asynchronous / Event based (application to queue to application)
      • Decouples applications
      • Services can scale independently from applications
      • 💡 Exactly-once delivery
        • One of the hardest things to achieve in distributed systems.
        • Better to design your applications to be idempotent: not to be affected adversely when processing the same message more than once.

Comparison

  • SQS vs SNS

    • Choose SNS for if e.g. someone uploads picture and listeners need to different stuff (send thank you email, crop thumbnail & save in S3).
    • Choose SQS for application to application communication.
    • Fan out: SNS to SQS to listeners = good of both worlds.

  • SQS vs SNS vs Kinesis Streams vs Amazon MQ

    Attribute SQS Standard SQS FIFO SNS Kinesis Streams Amazon MQ
    Type Direct queue Direct queue Pub & Sub (with topics) Data ingestion Direct queue + topic
    Consumption Pull Pull Push Pull Pull + Push
    Latency ⭐⭐⭐ ⭐⭐☆ ⭐☆☆ ⭐⭐⭐ ⭐⭐⭐
    FIFO 👎 Best effort 👍 FIFO 👎 Best effort 👍 Per partition key 👍 Supports FIFO
    Delivery 👎 At-least-once 👍 Exactly-Once Processing 👎 At-least-once 👍 Per partition key 👍 Supports FIFO
    Retention 14 days, default: 4 14 days, default: 4 None 1 day (default), up to 7 days Persisted in file until configured max file size reached
    Throughput / limits None 300 transactions per second per API Max topics / subscribers Write/read per shard, can scale out shards. Requires provisioning eg mq.t2.micro. Users, brokers, configurations and data storage is limited. API is throttled.
    Can delay ✓ ✓ ✕ ✕ ✕ Only on redelivery
    Out of box dead letter queue ✓ ✓ ✕ ✕ ✓
    AWS integrations 👍 Many 👎 Missing many e.g. S3/CloudWatch/SNS/Lambda DLQ... 👍 Many 👌 Some: Data Firehose, S3, Kinesis Data Analytics, Lambda, SDK uses DynamoDB for checkpoints 👎 Almost none, requires code.
    Consumer type Same type Same type Different consumers can be different stuff with the event from the topic Can be different per partition, as same consumer can listen to same partition Can be same (direct queue) or different (topics)
    Use cases Fast, decouple/integrate apps, visibility time out, message level ack/fail, message delay. SQS + strict ordering, deduplication, decouple/integrate apps Pub & sub (1 event to different apps), emails/SMS/.., push notifications, fan out (SNS to SQS -> good of both) Analytics, logs, IoT.. Related records to same processor (partitioning), ordering, multiple apps to consume same stream concurrently, consume records a few hours later. Lift & shift

Other services

  • Amazon MSK: Amazon Managed Streaming for Apache Kafka (Amazon MSK)
  • AWS IoT: Bi-directional communication between Internet-connected devices such as sensors, actuators, embedded micro-controllers, or smart appliances and the AWS Cloud.