amazon web services - sending lots of AWS SQS messages---FAST

Question

I have an application that may need to send hundreds of thousands of messages each run of my program using SQS. The program takes 1-2 hours/run and I run it 5-10 times/day. So that's roughly 1 million messages/day.

I want to do it fast. Is my best approach to:

1. Send each with its own send-message, but send them in another thread so my main thread doesn't pause?
2. Use send-message-batch, which lets me send 10 messages at a time?
3. OMG. Why am I sending so many messages? Why not write them all into a big object, save the object in S3, and then send a pointer to the object with SQS?

My messages are the stdout and stderr of programs that are running in a distributed system. So the problem with #3 above is that I won't get the output of the program until the batching happens. I suppose that I could batch up every 60 seconds.

I'm sure that this has come up for other people. Is there a clever way to do this in the AWS SQS API that I am missing?

Kinesis is not an option in my environment.

We are currently sending the messages from python programs running on Apache Spark workers---about 2000 cores/cluster---and other monitoring systems and about 5-20 clusters. The messages will go to a lambda server. The problem is that some of the nodes send a few thousand messages within the course of 10-20 seconds

We tried using Spark itself to collect this information, storing it in an RDD, saving that RDD in S3, and so on. The problem with that approach was that we didn't get real-time monitoring, and we added several hours to processing time. (We're not entirely sure why it added so much time, but it's possible that Spark ended up re-computing some RDDs because some stuff would no longer fit in RAM or on the spill disks.)

question from:https://stackoverflow.com/questions/65835851/sending-lots-of-aws-sqs-messages-fast

Answer 1

Waitting for answers