Amazon SQS {Simple Queue Service }
Brief About it
Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing service that makes it easy to decouple and scale microservices, distributed systems, and serverless applications. Amazon SQS moves data between distributed application components and helps you decouple these components.
Technical professionals and developers use this managed service to send, store and retrieve multiple messages of various sizes asynchronously.
How it works
There are three main parts in a distributed messaging system: the components of your distributed system, your queue (distributed on Amazon SQS servers), and the messages in the queue.
In the following case, the system has several producers (components that send messages to the queue) and consumers (components that receive messages from the queue). The queue (which holds messages A through E) redundantly stores the messages across multiple Amazon SQS servers.
Message lifecycle
The following scenario describes the lifecycle of an Amazon SQS message in a queue, from creation to deletion.
A producer (component 1) sends message A to a queue, and the message is distributed across the Amazon SQS servers redundantly.
When a consumer (component 2) is ready to process messages, it consumes messages from the queue, and message A is returned. While message A is being processed, it remains in the queue and isn’t returned to subsequent receive requests for the duration of the visibility timeout.
The consumer (component 2) deletes message A from the queue to prevent the message from being received and processed again when the visibility timeout expires.
In Amazon SQS there are two types of queues which are Standard Queue and AWS SQS FIFO. The standard queue offers at least one delivery and maximum throughput. The FIFO queues guarantee that the processed message takes place only once in the first in first out basis.
Getting Started with it
- Prerequisites
- Step 1: Create a queue
- Step 2: Send a message
- Step 3: Receive and delete your message
- Step 4: Delete your queue
For more information on getting started with Amazon SQS
Advantages
1) Eliminate Administrative Overhead
AWS manages all in-progress operations and underlying infrastructure required to produce an extremely accessible and scalable message queuing service.
With AWS SQS, there’s no direct value, no ought to acquire, install, and assemble the messaging package, and no long build-out and maintenance of supporting infrastructure.
Amazon SQS queues are dynamically created and scale automatically, therefore, you’ll be able to build and grow applications quickly and with efficiency.
2) Reliably Deliver Messages
Use AWS SQS to transmit any volume of data, at any level of output, while not losing messages or requiring alternative services to be accessible. Amazon SQS helps you to decouple application parts in order that they run and fail severally, increasing the fault tolerance of the system.
Multiple copies of each message area unit hold on redundantly across multiple accessible zones in order that they’re out there whenever required.
3) Keep Sensitive Information Secure
You can use Amazon SQS to exchange sensitive data between applications using server-side secret writing (SSE) to inscribe every message body.
The AWS SQS compass point integration with AWS Key Management Service (KMS) permits you to centrally manage the keys that defend SQS messages together with keys that defend your alternative AWS resources.
AWS KMS logs each use of your encryption keys to AWS CloudTrail to assist meet your restrictive and compliance wants.
4) Scale Elastically and Cost-Effectively
AWS SQS leverages the AWS cloud to dynamically scale supported demand. Amazon SQS scales elastically together with your application. Therefore, you don’t have to worry regarding capability designing and pre-provisioning.
There’s no limit to the number of messages per queue, and commonplace queues offer nearly unlimited output. Prices area unit supported usage that provides important value saving versus the “always-on” model of self-managed electronic messaging middleware.
Case Study
Oyster Case Study
New York-based Oyster.com shared unvarnished reviews of hotels in nearly 200 destinations worldwide. The company’s own investigators visit each location to assess cleanliness, amenities, service, and overall quality.
What sets Oyster apart from similar sites is its extensive collection of photographs. Oyster takes hundreds of photos for each property, and every review includes dozens of untouched images (submitted by guests as well as investigators) that allow potential visitors to compare a hotel’s marketing material with reality.
The Challenge
Since its 2009 launch, Oyster has published more than one million high-quality digital images. When this massive volume of images became too cumbersome to handle in-house, the company decided to offload the content to a central repository on Amazon Simple Storage Service (Amazon S3) . They chose to move to the cloud and Amazon S3 because storing images in their data center would have been too costly whereas in Amazon S3 was a more economical solution.”
Oyster reprocesses its entire collection of photographic images a few times each year to update the copyright year and, if necessary, to change the watermarks. Using their previous solution, reprocessing the entire collection of photographs required about 800 hours to complete. In addition, Oyster often recreated existing images in new formats and sizes for mobile and tablet devices. Resizing existing images and adding new ones was slowing down the rate at which the company was able to process the collection. When the iPad with Retina display came out, for example, it took them more than a week to create new sizes specifically for that resolution.” Oyster considered purchasing additional hardware but found the cost of new hardware and routine maintenance was too high, especially when the machines would sit idle most of the time.
Moreover, there were numerous software bugs in the multiprocessing solution that the company used, but since the solution didn’t scale, Oyster didn’t bother to fix them.
Why Amazon Web Services
While the company was still running one local server, the bulk of the processing work started taking place on the AWS Cloud. Oyster is using a customized Amazon Linux AMI within Amazon EC2. Within this new environment, the company connects to Amazon S3 and Amazon Simple Queue Service (Amazon SQS) using boto, a Python interface to AWS. The images themselves are processed with the Image Magick software available in the AMI package.
Oyster uses Amazon EC2 instances and Amazon SQS in an integrated workflow to generate the sizes they need for each photo. The team processes a few thousand photos each night, using Amazon EC2 Spot Instances. When Oyster processes the entire collection, it can use up to 100 Amazon EC2 instances. The team uses Amazon SQS to communicate the photos that need to be processed and the status of the jobs.
Benefits to Oyster
- Reduced time to process photos by 95%
- Reduced in-house hardware expenses
- Saves $10,000 in capital expenditure and reduced operating expenses by an additional $10,000
References
