AWS Lambda Memory Leak?!

https://youtu.be/68rMWIbPWVM – will embed when live

Ever had problems where your Lambda function seemed to have problems in one invocation that carried forward into your next invocations? Maybe you hit your AWS Lambda max memory limit in future invocations without even having done anything in the function yet?

You’re not alone!

AWS Lambda doesn’t hard reset the container between invocations. This is good and bad. Good because it’s faster, bad because it brings all of its baggage with it.

Here’s my little trick for handling this when you know something’s broken and you need a cold start:

await lambda.updateFunctionConfiguration({
 FunctionName: 'function-name'
}).promise();

I put that little snippet of code in my catch after I detect that the bad error (or when memory usage seems crazy high) and it’ll reset everything.

Hope this helps!

We do a lot of neat web scraping stuff on AWS. Check some here – AWS tools for web scraping (Lambda, DynamoDB, EventBridge, CloudWatch)

















































Transcript:

Hey, good morning. This is Jordan Hanson, Coldwell with intelligence. I just wanted to talk a little bit about AWS Lambda. It’s going to be a really short video. This is something I was really struggling with. I just wanted to share my solution for it. So I don’t know if any of you guys have had this situation where you’re looking, you have an AWS Lambda function and everything seems to be going fine, but maybe you have a memory leak or something.

And you’re running this multiple times. This Lambda function possibly even concurrently. And it seems like. The memory leak is leaking over into the other iterations, the other invocations. And so, as I did more research, this may not be a surprise to you, but and AWS Lambda it doesn’t, you know, you should know this because when you first make your first invocation, after a long time, it’s a lot slower to start up.

And then the further times it’s a lot quicker because it doesn’t actually clear everything in there. It kind of keeps it away or keeps it alive and keeps it warmed up your ABB, a slammed up. I guess it’s a good way to say it, but what’s happening to me is I was doing something where we’re parsing PDFs and there, we had to do more than just parts, a PDF.

It’s like a PDF with an image. So we have to convert it over to an actual JPEG and then try to parse the JPEG. It ended up using a bunch of memory. I mean, we’re using like seven gig or plus on our AWS Lambda function. And sometimes it would break and we use an, a layer in there to build. Software that could actually run it.

So sometimes it would break and it would break so hard to delay or would not work anymore. So picture like your computer need to restart. That’s how bad it is. The software is not working anymore and the computer needs to restart. But what would happen is that I would run this thing like 10 times in the night.

And it would break on the first one and then it wouldn’t work for the next nine, nine and vocations. So it would just be broken all night. And I found one simple way to reset your Lambda container two. It starts cold. So anytime I had this hard break I would have this air I throw that’s called PV parts.

PDF broke hard. I would just do this simple thing. I do this Lam, the update function configuration. I don’t do any, I don’t pass. So just updates the configuration with nothing, which is surprising to me that even allows that. And then if we start it’s cold on the next one, so it will be a little bit slower to start up, but Hey, it’s worth it because you clear out all your memory.

Cause sometimes I’ve had it where a memory leak and you’re running out of memory at the end of. You can’t do anything else on the next thing vocations, because the previous one is messed up and anyway, this works functioning completely from scratch. So AWS Lambda update, cook, functioning configuration.

There’s like another one you can do too. Is uptick function code. I think, I don’t know what that one requires more parameters, but this one right here, I just run this after the error and good to go. And we set a cold. That’s it. That’s how to reset the Lambda. Make it have a cold or start from cold again.

That’s it. That was my short little video. Thanks.


Hey there. Want to be friends?

I’ll be straight with you. I’m going to want to show off my web scraping genius. But I want to hear about your web scraping troubles and successes as well.

And I like you. So I won’t be one of those weird friends who is always calling. What do you say?