Skip to content
  • Home
  • State Business Leads
  • Real Estate Data
  • Secretary Of State API
Cobalt Intelligence

Cobalt Intelligence

Data mining and web automation.

You Are Here
  • Home
  • web scraping tools

Tag: web scraping tools

AWS Lambda Memory Leak?!
December 9, 2021December 3, 2021 Jordan Hansen

AWS Lambda Memory Leak?!

AWS

https://youtu.be/68rMWIbPWVM – will embed when live Ever had problems where your Lambda function seemed to have problems in one invocation that carried forward into your

Read More
Using AWS SQS and Lambda Reserved Concurrency to Rate Limit
October 27, 2021October 27, 2021 Jordan Hansen

Using AWS SQS and Lambda Reserved Concurrency to Rate Limit

Web Scraping

Today I want to do a deep dive into some AWS architecture decisions. I was able to use a lot of cool AWS tools and

Read More
Puppeteer on Lambda! Updated and better.
February 15, 2021January 4, 2022 Jordan Hansen

Puppeteer on Lambda! Updated and better.

Lambda

Demo code here Uploading directly to Lambda has a size limit of 50MB. This makes it difficult for Puppeteer, which is a very large library.

Read More
How to Handle Puppeteer’s waitFor Deprecation
February 1, 2021September 27, 2021 Jordan Hansen

How to Handle Puppeteer’s waitFor Deprecation

Puppeteer

Puppeteer has deprecated waitFor! But it’s not as terrible as you think. There it is! Looking good!

Read More
Basic Web Scraping with Nodejs and Cheerio
January 11, 2021September 21, 2021 Jordan Hansen

Basic Web Scraping with Nodejs and Cheerio

Learning to Web Scrape

Sample code here Very basic code showing how to web scrape with Nodejs and Cheerio.

Read More
Using Puppeteer with a proxy
January 4, 2021November 27, 2021 Jordan Hansen

Using Puppeteer with a proxy

Puppeteer

Sample code here This shows how to use Puppeteer with a proxy using both Luminati and Proxycrawl. Video guide: Luminati is a more intense solution

Read More
Avoid Being Blocked with Axios
May 25, 2020September 24, 2021 Jordan Hansen

Avoid Being Blocked with Axios

Web Scraping

Demo code here How to avoid being blocked while web scraping is something I think about often. I even wrote a post about it for

Read More
Avoid being blocked with puppeteer
December 30, 2019October 14, 2021 Jordan Hansen

Avoid being blocked with puppeteer

Learning to Web Scrape, Puppeteer

Demo code here One of the main questions I see on forums and reddit with regards to web scraping is…”how do I avoid being blocked?”.

Read More
All Right Reserved 2019