Quick Start to SQS

Introduction

Queues are a powerful way of combining software architecture. They allow for asynchronous communication between different systems. and are especially useful when the throughput of the systems is unequal.   Amazon offers their version of queues with Amazon SQS (Simple Queue Service).

When dealing with cloud-based infrastructure, it’s hard to pass by Amazon AWS. If you are considering Amazon SQS for improving your app architecture and need to quickly understand its basics and decide if it’s a fit for your project – you are in the right place. We will go over its basic functionality, main use cases, pricing, and common questions about this system and its integrations.

How does Amazon SQS work?

Amazon Simple Queue Service (SQS) is a message queuing service that enables you to manage asynchronous communication between applications or microservices.

If you are new to queueing, imagine this process as a communication between two subscribers on a mobile network. There always should be at least two participants: the producer (the one who sends the message) and the consumer (the one who receives it). For direct communication (like a call) both participants must be online and available.

Asynchronous communication is similar to messaging through a platform or even SMS. In this case, the producer sends a message and if the consumer is not available at the moment, this message will “wait” on an intermediary server. As soon as the consumer is available, the message is delivered and the consumer processes it (reads, forwards to another subscriber, etc.)

With Amazon SQS, messages can be processed in two ways:

• Standard – higher throughput, with “at least once delivery”, and “best-effort ordering”.
• FIFO (First-In-First-Out) – not as high throughput, but guarantees on “exactly once” processing, and preserving the ordering of messages.

Unlike the conversation, queued messages need to be deleted after processing to avoid their duplication.

This is a very simplified description of how message queueing works in SQS. To dig deeper, check out the basic Amazon SQS architecture in the official documentation.

Amazon SQS Terms:

Amazon SQS Visibility Timeout: This is a period of time when a message is protected from receiving and processing by other consumers. You can configure it from 0 seconds to 12 hours.

 Dead-Letter Queues: They are used for messages that can’t be processed successfully for some reason. Failed messages will be moved to a dead-letter queue automatically, you just need to specify the conditions. Connect the source queue with a dead-letter one and set the maximum received count: the number of times a message failed to process. After the limit is reached, the message is driven to the dead-letter queue and removed from the source queue. Also, you can configure notifications for failed messages so that you can instantly check those in dead-letter queues and analyze what went wrong. Dead-letter queues are good to use as a starting point to examine the contents of failed messages as well as the logs of exceptions that caused them to fail.

 Amazon SQS Short and Long Polling: These are two types of messages consuming. Long polling occurs when the waiting time for the ReceiveMessage API action is greater than 0. If the waiting time equals 0, then it’s short polling. Short polling is used by default. With it, you get a response back faster as it asks a subset of Amazon SQS servers (where the queue stores and processes messages), therefore you might need to issue multiple subsequent requests to get all messages from all servers. With the long polling, Amazon SQS waits for messages (until it times out) and asks all servers. In this case, a request takes more time to be processed but there are higher chances that it won’t be empty.

 

Amazon SQS cost:

Similar to other Amazon services, SQS is charged according to the “pay for what you use” model. In fact, you pay for the invoked requests (1 request = 64 KB chunk of a payload). The first one million requests per month are free. Each following million requests will cost you $0.4 if you use standard queues or $0.5 if you use FIFO.

It’s important to note that a single API call can have from 1 to 10 messages and is limited to a maximum total payload of 256 KB (4 chunks of 64 KB). Also, you need to remember, that you are charged for the outbound data transfer.

To be able to predict Amazon SQS cost, you should estimate the number of requests for each type of queue per month as well as the data transfer volume.

To get all the pricing details and calculate your possible expenses, refer to Amazon SQS pricing.

Amazon SQS queue Configuration

You can access Amazon SQS via AWS Management Console or integrate it via API.

AWS Management Console

Getting started with the AWS Management Console is easy and straightforward. Just assess SQS form the Services list on your AWS account homepage and follow the instructions. You will get explanations and hints from Amazon at every step.

Here you will be able to choose the type of the queue, make necessary configurations, and then start adding messages right away. Also, you can add metadata for each message, such as name, type, and value.

 

Once your message is sent to a queue, it is ready for retrieval and processing by another application. To connect your app with the queue in this Amazon service, you will need to pass the queue URL (which you will find in the queue Details tab).

From the Queue Actions menu, you can request a message to view (by clicking  Start Polling for messages) and then delete it.

Also, you can subscribe queue to the SNS topic and configure trigger from Lambda function right from the Queue Actions menu.

 

When to use Amazon SQS

Amazon offers three services to enable communication between apps: SQS, MQ (Message Broker), and SNS (Simple Notifications Service).

Amazon MQ is a managed message broker service for Apache ActiveMQ. Amazon recommends using it for migrating your messaging with existing applications to the cloud.

Amazon SNS is a push notifications service, and using it in combination with SQS is one of the best practices.

Amazon SQS will be a good fit for new applications built in the cloud. You can use it independently but it’s always a good idea to compose it with other Amazon services like Amazon EC2, Amazon EC2 Container Service (Amazon ECS), and AWS Lambda. In addition, you can store SQS messaging data in Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.

In general, message queueing is used to provide application scalability and to decouple the complex back-end operations from the front-end output.

Amazon SQS limits:

• You can retain messages in queues for 14 days maximum.
• The maximum size of one message is 256 KB (you can send large messages via Amazon SQS Extended Client Library for Java, which stores larger payloads in Amazon S3).
• A message can contain XML, JSON, and the unformatted text. The limited number of Unicode characters is supported as well.
• The number of messages handled in standard queues is not limited! However, FIFO queues allow up to 3,000 messages per second.

 

SQS Example:

Sending a Message to a Queue

Create a Node.js module with the file name sqs_sendmessage.js. Be sure to configure the SDK as previously shown. To access Amazon SQS, create an AWS.SQS service object. Create a JSON object containing the parameters needed for your message, including the URL of the queue to which you want to send this message. In this example, the message provides details about a book on a list of fiction best sellers including the title, author, and number of weeks on the list.

Call the sendMessage method. The callback returns the unique ID of the message.

// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
// Set the region 
AWS.config.update({region: 'REGION'});

// Create an SQS service object
var sqs = new AWS.SQS({apiVersion: '2012-11-05'});

var params = {
  DelaySeconds: 10,
  MessageAttributes: {
    "Title": {
      DataType: "String",
      StringValue: "The Whistler"
    },
    "Author": {
      DataType: "String",
      StringValue: "John Grisham"
    },
    "WeeksOn": {
      DataType: "Number",
      StringValue: "6"
    }
  },
  MessageBody: "Information about current NY Times fiction bestseller for week of 12/11/2016.",
  // MessageDeduplicationId: "TheWhistler",  // Required for FIFO queues
  // MessageId: "Group1",  // Required for FIFO queues
  QueueUrl: "SQS_QUEUE_URL"
};

sqs.sendMessage(params, function(err, data) {
  if (err) {
    console.log("Error", err);
  } else {
    console.log("Success", data.MessageId);
  }
});

To run the example, type the following at the command line.

node sqs_sendmessage.js

This sample code can be found here on GitHub.

 

Receiving and Deleting Messages from a Queue

Create a Node.js module with the file name sqs_receivemessage.js. Be sure to configure the SDK as previously shown. To access Amazon SQS, create an AWS.SQS service object. Create a JSON object containing the parameters needed for your message, including the URL of the queue from which you want to receive messages. In this example, the parameters specify receipt of all message attributes, as well as receipt of no more than 10 messages.

Call the receiveMessage method. The callback returns an array of Message objects from which you can retrieve ReceiptHandle for each message that you use to later delete that message. Create another JSON object containing the parameters needed to delete the message, which are the URL of the queue and the ReceiptHandle value. Call the deleteMessage method to delete the message you received.

// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
// Set the region
AWS.config.update({region: 'REGION'});

// Create an SQS service object
var sqs = new AWS.SQS({apiVersion: '2012-11-05'});

var queueURL = "SQS_QUEUE_URL";

var params = {
 AttributeNames: [
    "SentTimestamp"
 ],
 MaxNumberOfMessages: 10,
 MessageAttributeNames: [
    "All"
 ],
 QueueUrl: queueURL,
 VisibilityTimeout: 20,
 WaitTimeSeconds: 0
};

sqs.receiveMessage(params, function(err, data) {
  if (err) {
    console.log("Receive Error", err);
  } else if (data.Messages) {
    var deleteParams = {
      QueueUrl: queueURL,
      ReceiptHandle: data.Messages[0].ReceiptHandle
    };
    sqs.deleteMessage(deleteParams, function(err, data) {
      if (err) {
        console.log("Delete Error", err);
      } else {
        console.log("Message Deleted", data);
      }
    });
  }
});

To run the example, type the following at the command line.

node sqs_receivemessage.js

This sample code can be found here on GitHub.

 

SQS HIPPA Compliance:

SQS is Hippa Compliant service taking into consideration data encryption, to get more info about SQS Hippa please check our article.

Conclusion:

Message Queuing is a proven technological concept that is heavily leveraged both independently and as the foundation of larger concepts to great success. Oftentimes, Message Queuing is the perfect bridge in the gap between the need to keep applications simple and yet sophisticated enough to keep track of what’s going on in highly dynamic Cloud-Computing environments,

Stay tuned, in future posts we will drill down the pros, and cons, of SQS.