AWS Lambda Memory Leak?!
https://youtu.be/68rMWIbPWVM – will embed when live Ever had problems where your Lambda function seemed to have problems in one invocation that carried forward into your
Tag: web scraping tools
Using AWS SQS and Lambda Reserved Concurrency to Rate Limit
Today I want to do a deep dive into some AWS architecture decisions. I was able to use a lot of cool AWS tools and
Puppeteer on Lambda! Updated and better.
Demo code here Uploading directly to Lambda has a size limit of 50MB. This makes it difficult for Puppeteer, which is a very large library.
How to Handle Puppeteer’s waitFor Deprecation
Puppeteer has deprecated waitFor! But it’s not as terrible as you think. There it is! Looking good!
Basic Web Scraping with Nodejs and Cheerio
Sample code here Very basic code showing how to web scrape with Nodejs and Cheerio.
Using Puppeteer with a proxy
Sample code here This shows how to use Puppeteer with a proxy using both Luminati and Proxycrawl. Video guide: Luminati is a more intense solution
Avoid Being Blocked with Axios
Demo code here How to avoid being blocked while web scraping is something I think about often. I even wrote a post about it for
Avoid being blocked with puppeteer
Demo code here One of the main questions I see on forums and reddit with regards to web scraping is…”how do I avoid being blocked?”.