I have been looking for an opportunity to work on queue base project after finishing Udi Dahan’s Advanced distributed system design course. Because most of my projects are running on Linux OS where NSerivceBus is not applicable, using Amazon SQS became a affordable work around for now.
Problem to solve
My iClip iOS app recently encountered unexpected popular requests for unknown reason (listed somewhere on an iOS app category site?). Even I only grant every new client only 3 free requests, with the huge amount of daily downloads, my server becomes slow and unstable gradually.
I could keep bumping up the memory on server, but I don’t like this solution at all. First, server hosting bill will go up dramatically; second, hosting server memory still has a max limitation of 4GB. Someday the client requests will eventually break this ceiling, then what?
I decided to move my request model to queue based architecture based on Amazon’s SQS service. Rewriting app didn’t take very long. After a week of production experience, here are some gotcha I’ve learned.
- Message won’t be immediately available after send. Amazon actually state this delay will be up to 60 seconds on SQS admin console.
- Deleted message might still be visible in queue. Explanation from Amazon is:
It is possible you will receive a message even after you have deleted it. This might happen on rare occasions if one of the servers storing a copy of the message is unavailable when you request to delete the message. The copy remains on the server and might be returned to you again on a subsequent receive request. You should create your system to be idempotent so that receiving a particular message more than once is not a problem.
- For the similar reason as above, queue processor should really deal with duplicated request very carefully. For me, I use sendtimestamp as the identity of message.
- Receiving message must set the visibility time out to a value greater than zero, to allow this message to be deleted successfully afterwards. (To be confirmed)