amazon web services - AWS Lambda hangs between invocations

Question

I am using the following 3 services: Amazon S3, Amazon SQS and AWS Lambda.

The same configuration is created for processing for both CSV and EXCEL files (the lambda function that processes EXCEL files is just converting them to CSV files and re-uploading them to S3 in order for the other lambda function to process them)

AWS Lambda configurations:

• Memory: 1024 MB
• Timeout: 6 minutes
• Reserved concurrency: 1 (for the current testing I don't need multiple parallel functions)
• Retry attempts: 0
• DLQ: none configured at the moment (will be added later)

For Amazon S3:

On a 's3:ObjectCreated:*' event, the S3 sends a message to a configured SQS queue. The SQS queue has a Lambda trigger attached to it.

I have an external process that is uploading files to my S3 bucket.

This is the start of the entire workflow (S3 -->SQS --> Lambda)

This process has uploaded around 40 files in a very short period of time (some CSV files and some EXCEL files as well).

I was looking into the SQS queue and CloudWatch to see how the processing was going and I was able to see about 15 messages in flight for the SQS queue that is handling the CSV files and about 17 messages in flight for the SQS queue that is handling the EXCEL files and the logs in CloudWatch were being updated and everything was going good.

After about 15 seconds of processing everything stopped. Both lambda functions were just hanging. I was still seeing around 15 and 13 messages in flight for both SQS queues but absolutely nothing was being done on the AWS Lambda. It looked like something went wrong.

After about 5 minutes of doing nothing both functions suddenly started to process the files. Both functions processed a couple of files for about 15 seconds and then silence once again.

After another 5 minutes of doing nothing both functions started again to process the files.

This happened a couple of times with 5 minutes breaks.

The Lambda functions are not doing any external calls or something that could make them hang. The waiting was between AWS Lambda invocations so it wasn't within my code.

For example:

2021-01-22T17:23:56.426+02:00 REPORT RequestId: d0a01831-ff93-5a71-83d6-40b50fd0affa Duration: 453.19 ms Billed Duration: 454 ms Memory Size: 1024 MB Max Memory Used: 319 MB

2021-01-22T17:29:41.860+02:00 START RequestId: 752f0eef-6738-5c24-ad52-566b96983c92 Version: $LATEST

What is making AWS Lambda to hang?

PS: If it would be helpful then I could attach the CloudWatch logs.

question from:https://stackoverflow.com/questions/65848796/aws-lambda-hangs-between-invocations

Answer 1

I think the culprit is "Reserved concurrency: 1". The SQS --> Lambda part involves an invisible middle man called event source mapping which polls the queue and invokes the Lambda for you.

When messages are available, [The event source mapping] Lambda reads up to 5 batches and sends them to your function.

Since you have configured the reserved concurrency to be 1, when the event source mapping tries to do 5 invocations at once, 4 of them get "Rate exceeded" error. The message will then be put back into the queue after the visibility timeout. After that, the queue will trigger the lambda again and this "try and error" process continues. Every time the lambda is only able to process 1 message, and the "hanging" behaviour you are seeing is actually the remaining 4 messages waiting for the visibility timeout.