How to Make Money With Web Scraping
Here are my tips for making money on web scraping.
Sell data, not scrapers. Make content. Stop making excuses.
Make content so people know how awesome you are.
Transcription:
Hello there and welcome to my channel. I channeled this video. I’m Jordan Hanson. I’m from cobalt intelligence. I am a web scraper. I do like to scrape the web. I have scraped a few pages, quite a few pages. And currently I’m, full-time employed via my web scraping software skills, whatever I am supporting a family of five.
Finally a five. So I have made money with web scraping. And so this video is going to be talking today about how to make money with web scraping, what I think anyway. Now there’s probably a ton of avenues I’m not going to think about, but I would like to talk about someone asked me on Twitter. They said, Hey, I saw your videos.
I see you have a business going, which is called wealth intelligence. And he wanted to know how I started making money with. And I will say, I’m just gonna tell my story a little bit and kind of talk about my takeaways of what I think could be helpful for people that are doing web scraping that like it but are not able to make money with it yet.
But the fun, interesting thing about web scraping is that it’s, it’s kind of like a fringe thing right. Of software where it sometimes it’s frowned upon because people, you know, maybe it has a bad rap people in dark rooms doing web scraping bad stuff, whatever. I don’t think it has to be like that at all.
I mean, honestly, most of the people, how about Google is scraping everything there, how we think they’re crawling. That’s what web scraping is. There’s a ton of data out there that’s public and be able to get that data and use it. It’s very interesting. I think there’s a lot of ways to be an amazing web scraper and still be ethical.
And that’s where I think it’s a good market. And almost that fringe areas kind of, kind of a good place to be, because it’s you think about moats? Well, things are going to help you be a stop competition, other people from following it. And I think possibly web scraping because of that stigma, maybe it’s not a huge stigma, I don’t think, but it’s certainly, I think there’s a little bit there because people always ask me, well, how do you feel ethically about this kind of stuff?
So there’s something there. Now I five years. As long that says seven years old years, seven years ago, my friend, he said, Hey, I am getting business leads via some website. It’s a public website. And what I do right now is we pay someone $500 a month and he’s in the Philippines and he goes out there manually.
And every day he goes and checks the new things that come out on this list. And he said, this is fine, right. We pay $500 with dollars. I’m not too worried about that. But the problem is, is sometimes he gets sick or he goes on vacation and then my sales reps don’t have any leads. So do you think you could build some software for this?
And this is the first time I was ever exposed to web scraping. I was really into Ruby on rails at the time. So I said, yeah, I can probably do something like that. I built the piece of software and I sold it, I think for $600. Like I sold it. Yeah. Penny’s not very much money at all, considering this as the, most of the leads for this guy’s business and it wasn’t recurring at all.
And it became kind of a headache because he needed, you know, things changed. He needed to install the different people’s computer because I built this piece of software. I gave it to him and I didn’t have it. And he was my been my friend for a long time, but we didn’t have any contract or whatever, what maintenance would be like for this thing.
So it kinda stunk. But it was my first exposure to web scraping. I thought it was pretty interesting. And so I kinda, I just remember being, I was kind of doing it and maintaining it for him. Eventually I fixed it up and changed it so I could put it in a recurring basis. Right. So then I own the software and I just deliver the data to him and then he can do what he wants with it.
Now I think this worked out really well. So here’s the two different aspects here. I’m talking a lot different aspects. The first aspect is that I. One, I built the software. The scrape grant gave me. The second is I built the SA the scraper for myself, and then I just sold the data. And this is going to be a key theme I have in this.
I think selling data versus selling software is a million times better. The data is, and especially if it’s something recurring, right. If you can find some data that’s fresh. Then they’re scraping because way more valuable. And you can cause you can keep building for it. You build a scraper once and you can keep building for it and you’re gonna have to maintain it, which is, can be a pain, but at least it’s something that you it’s like recurring income and it’s constantly giving value to this person.
Versus if you build the software, you’re gonna have to be good at your contract, I guess, to make it worth it. Otherwise, how are you going to make it mean? You gotta have something in there for maintenance and it’s typically going to be just tougher to meet. So, anyway, my story continues. This got me into it years ago, and I remember I always I’ve always wanted my own business and I kind of wanted to go that direction, but I didn’t know how to get there.
I was working the novel at the time. Loved it. It was a great place, but it wasn’t really what I knew I wanted in the end. So I started listening to a book called the soft skills by John Somnio is some no’s. Anyway, some software that. I had a book about explaining how he became started to make money, whatever with software is doing.
And one of the things he said was you should pick a blog and you should pick a niche. And so I thought about it for actually a long time. I remember going walks in my neighborhood here, over here, and it was probably three years ago going on walks. And I got listened to this audio book and I said, I’m going to start a blog.
And I had started blogs before, but I had never stuck with them, but I felt pretty determined to start a blog on him to get this done. And. I picked a niche and I said, oh, you know what, I’m going to pick any web scraping. I had done some like with this scraper, but hadn’t been too much beyond that. There was kind of.
I just hadn’t done. I’d done some, but not very much. I was like, I’m gonna just niche down in this. And I wanted to make it even more specific. I said, JavaScript, web scraping, because I was doing a lot of JavaScript at my job and it was really fulfilling. I loved JavaScript. So I wanted to keep niching on that and keep seeing with JavaScript because I just really liked it.
So I started writing blog posts and like, I have some of my earlier posts. I wonder how can I find. They weren’t great. I’ll tell you that cobalt that tells me, see if I can find a quick disease, but
I ever go here’s my first one, December 4th, 2018. So is that three years? No, right.
Three yeah, three and a half years. We crossed three years in 2021 and we’ll cross borders. This is my first one. It was written, you know, years ago. And I still don’t get tons of traffic here on this website. I get regular traffic. I’ve gotten a lot of really good opportunities, but what it made me do is get really good is because I, every week I was like, all right, I’m going to do a post.
And I kept focusing and I kept like, I kept searching for things and new challenges and I would post on Reddit and I’d say, Hey, does anyone have any content you run? Scrape. I’ll do it. I just for free. I just want something to scrape. I would scrape it. And then. Deliver the data for them. And I do use it as content for my blog.
It was great. Learned a lot of different things, challenged myself with a lot of different things. And then, because I had this date out here, I posted about it. This is the one of the most fun things for me. I, someone asked, Hey, can you do some real estate data for me? And I said, sure, I, I guess I found out about real estate auctions that are online.
So I scraped the data I posted. And then I wrote up an article. And I posted about it on Reddit, the real estate investing separate. And I I’m going to hold on to this. If I can find it. Oh, there’s no way I can find it. Anyway, I posted on there, I’ve got tons of responses. It was really interesting. A lot of people asking great questions.
They said, they’d love this. They wanted this data. And someone reached out to me via the Reddit chat and we started talking more and we got on a call and he hired me. He said, okay, I want you to, and he hired me as a retainer. And that was, it’s probably been almost two years. Yeah. Two years. Yeah. And they paid me every month for two years.
Now that client is still client. So that guy from Reddit, he found me and he they’ve been great to work with really good customer. I’ve been working with them. So because I was blogging, I got better at doing. Web scraping. And for them, I didn’t sell software, but it was like a retainer fee. Right. So it was kind of like hourly, like, Hey, we expect you to work about this much, you know, but they were never really strict.
I built the software, I own the software then myself, we made that clear in the contract. So we talked earlier about how to make money. You can own the software, you can sell the data. This is kind of a blend. Right. I get, I did sold my time a little bit. But I also built these processes that made it pretty easy for me to work on.
I could scale really well. So. I posted on Reddit. I got better at web scraping through the blogging, kept doing it. People started reaching out for a sponsored blog post. I made some money that way. And then I stumbled upon and I got occasional contracting jobs over here and there as I was doing this nothing crazy, but I got some more contracting jobs.
I landed at another real estate customer. But that wasn’t through the blog, but it, and again, it was kind of selling data where I’d build these processes where I could grab this data and then send it off to him and it would be good, happy for them. So. Again, I think some all summarizing them at the end, but this is kind of, that’s a story.
This is how we go. And then I, as I started to look for more content, I found in the secretary of state. And so I started scraping the secretary of state for data and I found some really cool data, their data that I could sell. And I was like, okay, bam, I know this is data I could sell. And so I started to grab this data and I found it in different states and I started to sell it and I had a friend.
That bought it. And then I started cold calling people, cold emailing people, cold outreaching people, the people on LinkedIn they’d sign up. And I still have as customers, like years still going strong with these people. And I got to see, wow, this data is really cool. And the secretary of state you can find who owns a business, and sometimes you can find it even more than that.
When you scrape this data. And I needed it more and more. So I built this API, which is what cobalt intelligence is. I built this API that people could use and then they could pay me to use it. And then really they’re just leveraging my I’m web scraping outsourcing, really. So instead of them having to build their own scrapers, they just send an API request and it gets the data and it brings it back to them.
Pretty cool to me. I liked it a lot. I thought it was a great time and I learned so much. As I was, the neat thing was, is as I built this stuff, as they built the scrapers, I would keep making blog posts. And then I started making videos and then people more and more started to reach out to say, Hey, I need the secretary of state data.
Can I get access to it? And again, it started to grow more and more and more to the point where I was able to quit my job last year and keep working full-time on it. So summary of this is that. Blog, I think who should make content. I that’s what I think you should do. If you’re doing web scraping, it’s going to force you to get better.
A and B you’ll also get your name out there. You build some credibility, people, you, you don’t know what opportunities are going to come about it. If you knew that. So if you’re going to use web scraping to make money, I don’t think you should sell the software. I think software is not good. I think you want to sell the data.
And so you want to build the data up. And when I say sell the software, that doesn’t mean like software is. I see some out there that like scrappy, I think Octopart anyway, I think those are good services. That’s different. They’re not building a custom software scraper and then selling it. I don’t think that’s going to work very well.
It doesn’t scale at all and you’re gonna have to maintain it. It’s not going to work very well. You’re going to do a lot better if you can find really interesting, valuable data and again, think valuable as in, can this make someone. And that is what you want to sell. So you find some data that’s really, you know, I think trading, I don’t know, crypto there’s new coins dropped all the time.
If you could scrape that data and sell it. I don’t know. I’m not very good at crypto. Good at crypto, you know, isn’t that how all the kids say it these days. So there’s that this is how I think it should be done. My, I think you should create content a, I think you should try to avoid selling scrapers individually.
I think you should have. Bye get, find data that’s fresh renewing and try to sell that as a service. So your monthly, they pay you, you just keep selling and you send them an email every day with the data selling the data, I think is going to be infinitely more profitable, profitable, where you put more profitable for you to do that.
So that’s my summary and how I think you should do what you should do to earn money with web scraping the end. I hope it’s helpful. You know, I love it. It’s been so. Keep working on it. And if you have questions, you don’t chat, whatever tweet me.