As web scrapers we need to rotate our IP addresses. To do this, you will need a proxy.
Sample code – https://github.com/cobalt-intelligence/rotate-ip-addresses-in-proxies
In this video I talk about using the session id to ensure that your IP addresses rotate while using ScrapingBee or BrightData.
Shiny affiliate links:
ScrapingBee – https://www.scrapingbee.com?fpr=jordan-hansen52
BrightData – https://brightdata.grsm.io/jordanhansen6276
Hello, my name’s Jordan Hanson. I’m from cobalt intelligence here. I’m talking about rotating IP addresses with proxies. So sometimes I’m R okay, let’s go. We’re talking about proxies. If you use an approximate. Such as scraping medium bright data slash Lewminatti, whatever they Luna was changed their name, the right data.
So if you’re using a proxy service, the goal is to change the IP address. Now you can’t guarantee the IP address is going to change every time, but it’s pretty common. But if you want to, for sure guarantee that that’s what we’re going to talk about today. So here we are. We’re here’s a normal scrape.
We’re going over here to this place called loom test. It’s right here. Think this SI shows AP is just comes out and Jason really handy for testing. So you don’t have to go to Google’s what is my IPE or whatever else. So they go, they can see my P I thought I’ll try it again just to make sure it’s the same.
There you go. Right there. See if there we go. Okay. Now, if we go over to scrapey B we’ll try this year now. It’s interesting. I mean, pretty much most people use it the same way. It’s called the session ID. Scraping be hazard a session ID, they call it a such an AP, a parents or persistence, and they just named it with this session.
So I’m going to show an example here with the goal with the session is that it forces an IP. So if you want to have the same IP, which is not very common for us, when we’re using web scraping things, we want to rotate it. But sometimes you’re like, okay, You can establish where you already, maybe you already solved a capture with that session.
And so you don’t want to do it again because that session already has a good name, whatever. So here it is with session ID, number seven, I just run it. See, I add that as a parameter in my URL right there. This is my I’m using the scraping be here. It’s going to run. It’s going to run please. Okay. There we go.
So we have Buffalo, New York, 45, 72, whatever 2, 2, 8. We’re on track. And that’s all. What goes well, we’re into three times and we expect it to be the same one every time. Was it 2, 2 8, 2, 2 8. Every time. Come on, come on, come on. Oh, okay. We’ll do it two times. Three times. It’s taking too long. Come on. I swear.
I was assessing this a bunch of times and it wasn’t taking as long I’m on escaping bees. Don’t move very fast.
Oh, my gosh. What’s going on here? This is crazy along there we go to do eight. There we go. Perfect. Okay. So now let’s try rotating our session. So about this round, rotate this, send that out. I want I’m doing here. And so scraping be themselves. They said, Hey, if you’re going to do this, you can rotate these a random integer between zero and 10 million.
And so that’s what I’m doing. I’m rotate. I round up my math dot random and I multiply times 10 million. So there we go. And the idea is that each request should be different. See that one right there, different one. It was still in Buffalo, New York, but different IP. Let’s go again. And this guarantees are one, a 2 million chance that we’re in a different IP address.
Every time there we go. So now let’s do that same thing with, so again, super easy and the scraping beads just pass another parameter called session ID, and they’re just. I would go like that. Now let’s try it with bright data slash whatever you want to call it. So we have here. Now, the difference here is you pass in the session parameter at the end of your username.
So you have your username, you pass in the session parameter, and apparently this can be whatever, some good, even like they have an example of ran, whatever. So I did as the same thing, I just said, brands. And that’s it. Give us that one. Let’s try again. Not everything’s right. We should get back again. 1 9, 8, 2 0 2 1 9 8 2 0 2 1 8.
Now that’s random a Fiat. So right here I did the same thing I said random. And then I said this, a huge math ceiling by 2 million. So just a random number out of Tim though. Here we go.
Okay, here we go. 1, 3, 3. And you can change your username by the way to come from us if you want. So these right here, oh, there’s an oven from Florida. So there you go. That’s how you do it. It’s pretty easy. It’s an idea of a session, pretty much any good proxy. You purchase a proxy service. They’re going to have something like this.
So everyone I’ve seen, they always have something like some kind of session. This is an example of using one with luminosity and bright data slash bright data and a scraping beat. I hope that’s helpful for you. If you need to change why PHS is guaranteed.