AWS SQS

AWS SQS

What is SQS in AWS?

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent. 

Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components.

 Basic Amazon SQS architecture

Distributed queues

There are three main parts in a distributed messaging system: the components of your distributed system, your queue (distributed on Amazon SQS servers), and the messages in the queue.

In the following scenario, your system has several producers (components that send messages to the queue) and consumers (components that receive messages from the queue). The queue (which holds messages A through E) redundantly stores the messages across multiple Amazon SQS servers.

Message lifecycle

The following scenario describes the lifecycle of an Amazon SQS message in a queue, from creation to deletion.

 A producer (component 1) sends message A to a queue, and the message is distributed across the Amazon SQS servers redundantly.

 When a consumer (component 2) is ready to process messages, it consumes messages from the queue, and message A is returned. While message A is being processed, it remains in the queue and isn't returned to subsequent receive requests for the duration of the visibility timeout.

 The consumer (component 2) deletes message A from the queue to prevent the message from being received and processed again when the visibility timeout expires.

Benefits of using Amazon SQS

  • Security – You control who can send messages to and receive messages from an Amazon SQS queue.

    Server-side encryption (SSE) lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).

  • Durability – For the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support at-least-once message delivery, and FIFO queues support exactly-once message processing.

  • Availability – Amazon SQS uses redundant infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages. 

  • Scalability – Amazon SQS can process each buffered request independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.

  • Reliability – Amazon SQS locks your messages during processing, so that multiple producers can send and multiple consumers can receive messages at the same time. 

  • Customization – Your queues don't have to be exactly alike—for example, you can set a default delay on a queue. You can store the contents of messages larger than 256 KB using Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or you can split a large message into smaller messages.


    Disadvantages of using SQS

    Using SQS can also create challenges for Serverless developers, as described hereafter.

    High cost at scale With pay per use pricing, if the number of messages you send is very high, your SQS bill can be quite significant. Part of SQS pricing is data transfer charges, and those can add up if you send larger messages, or if you process messages from outside the main AWS region in which the queue is located. In some cases, when running at scale with millions of messages processed every day, the cost of using SQS might be higher than the cost of operating your own queue system, even including the overhead to manage your own solution.

    Lack of support for broadcast messages With its “exactly once” delivery, SQS doesn’t support a way for multiple entities to retrieve the same message, making SQS not so useful for one-to-many broadcasts.

    To address this, in the cases where one-to-many delivery is important, developers can use SQS alongside SNS.

    Reduced control over performance When running a message queue system at scale, something you may well end up wanting to do is to fine-tune its performance to suit your needs. With SQS this isn’t an option: the service is fully managed, and you don’t get to look under the hood.

    Differences between Amazon SQS, Amazon MQ, and Amazon SNS

    Amazon SQS and Amazon SNS are queue and topic services that are highly scalable, simple to use, and don't require you to set up message brokers. We recommend these services for new applications that can benefit from nearly unlimited scalability and simple APIs.

    Amazon MQ is a managed message broker service that provides compatibility with many popular message brokers. We recommend Amazon MQ for migrating applications from existing message brokers that rely on compatibility with APIs such as JMS or protocols such as AMQP, MQTT, OpenWire, and STOMP.

    Common tasks for getting started with Amazon SQS

    Pricing for Amazon SQS

    Amazon SQS has no upfront costs. The first million monthly requests are free. After that, you pay based on the number and content of requests, and the interactions with Amazon S3 and the AWS Key Management Service.

    Why is SQS an essential part of the Serverless ecosystem?

    In an optimal Serverless architecture, when using multiple microservices, you want your services to be independent from each other. This means that they use separate databases and even run in different network segments but can still communicate with each other. The distributed systems term for this kind of independence is a decoupled system. When services are decoupled, you can’t pass messages, for example, by having them both write to a shared database. The available options under this model are: implement an API in each service, or use message queues to pass information between services.

    While in certain cases HTTP and gRPC APIs might be a good choice for service intercommunication, message queues tend to be a more lightweight approach and conducive to faster iteration.

    And aside from being a queue system that helps you iterate faster, SQS is also fully managed. This makes it a great choice for Serverless applications running in AWS because:

    • Many Serverless applications already rely on managed services to reduce maintenance load on the team developing the applications, making SQS’s managed model a good fit.
    • SQS can run in the same regions in AWS as your applications, which means high network throughput and low latency. This results in fast message delivery times using SQS.
    • SQS provides a simple HTTP API which is easy to implement using the AWS SDK.

    These characteristics make SQS a popular choice for Serverless developers and, consequently, an important part of the Serverless ecosystem.

    How does SQS integrate with other AWS services?

    Most interesting for Serverless developers is SQS‘s integration with Amazon Lambda: SQS can act as an AWS Lambda event source. When configured, every SQS message triggers a Lambda function run that processes a batch of SQS messages.

    Another useful integration is with SNS: an SQS queue can subscriber to an SNS topic. We cover the differences between SQS and SNS below, in the When to use SQS vs. SNS section.

    SQS also provides standard integrations for monitoring and debugging SQS queues using Amazon CloudWatch and AWS X-Ray.

    Serverless developers can manually integrate an SQS queue with any other AWS service (or a third-party service) by writing code that uses the AWS SDK to submit messages to SQS and read them from there, or by using the SQS API directly.

    Amazon SQS alternatives

    SQS is not the only queue system out there, though it’s the only one of this particular type that AWS offers. Some, if not all, of the tasks you can accomplish with SQS can also be solved with the following alternative services and tools.

    Amazon MQ Amazon MQ is a managed queue service from AWS which is based on Apache ActiveMQ. Its architecture isn’t as flexible as SQS, and in order to use the Amazon MQ queues you need to use ActiveMQ libraries. On the other hand, being based on ActiveMQ means that Amazon MQ is compatible with any application that already uses ActiveMQ without any code changes. For existing applications that use ActiveMQ, it’s a good way to move the queue to the cloud and reduce the amount of services you need to maintain your infrastructure.

    Amazon Kinesis Data Streams Amazon Kinesis Data Streams is a managed AWS service that’s designed for high-volume near-real-time data intake. Kinesis Data Streams can be a good choice if you are producing large amounts of data that need to be processed in batches in near real time. For example, you could use Kinesis Data Streams to collect all the events happening in your systems for analytics purposes and process them with Lambda functions.

    Apache Kafka Apache Kafka is an open-source stream processing system. Its main purpose, similar to AWS Kinesis, is to process large amounts of data in near real time. It has the same advantages and disadvantages as Kinesis Data Streams when compared with SQS: from one side, it is able to process very large number of messages; from the other side, it requires much more configuration and maintenance and is harder to use for producers and consumers compared to SQS.

    RabbitMQ RabbitMQ is a popular open source message broker. It offers all of the same functionality as SQS and more: for example, it supports various authentication mechanisms for RabbitMQ queues, provides the option of using “lazy queues” that can handle a backlog of millions of messages, and supports queue protocols like JMS and AMQP.

Comments

Post a Comment