Puppeteer Timing, waitForSelector

This is a video about Puppeteer Timing, waitForSelector.

It has some powerful options! https://pptr.dev/#?product=Puppeteer&version=v13.4.0&show=api-pagewaitforselectorselector-options

This can be really helpful if you are trying to use Puppeteer and need to wait for an element to be visible on the page.


Hello there. This is Jordan Hanson. I’m Jordan Anson. That’s me. This, this right here. Jordan Hanson. I’m from cobalt intelligence. I am here talking about puppeteer timing again, some more using puppet here at the head. This browser works really well. There’s some tricks. Sometimes I want to share some of them.

And it’s going to be on timing. So here we’re at the Missouri secretary of state. I have a pretty good site, pretty easy to scrape you. Come over here. I’m navigating directly to this page. Am I let’s make sure I am and what I’m trying to do here, the problem I had because I was trying to get document information.

So you come over here to filings, which is the document information. And the cool thing about this kind of cool. Remember timing is hard and puppeteer because. You want to make your script as fast as possible, but you have to make sure that the stuff is loaded on the page before you started doing with it.

So if you look at the filings here, the interesting thing is this is the filing. See you can’t see it very well. I’m gonna try and make a little bigger. This is the filing area and all this filing. It’s there and the Dom right here. See it it’s there before it low. Before we open it, click on this thing.

So it’s just see they’re all display. None display, none display none. So they’re there in the docket, in the, in the Dom so we can get them. So that seems easy, right? Oh, sweet. You don’t even have to click the button. You just navigate to this page and you can grab them true. Unless we have one like this one.

So we go over here, we click it and look what it has. It has multiple pages. Which you have to paginate through now, you can’t click something. That’s display none. It won’t it’ll break. It’ll say this note is not visible in the page. Can’t do it. So what I needed to do was get to there and I needed to know when I arrived now I could have used the lame way.

So I go over there. I say, okay hold on. That’s a river, very buffer. I’m going to go like this. I have my clear. This clicks, my filings. This is the selector for the filings. It’s this huge ID. Plus L I N the type to see her at their second one. So I click it. And how do I know when it actually loaded in there?

And that’s the question. That’s what we’re trying to figure out today is that the first thing you can do is you can say a wait for timeout and you can say just five seconds, wait, five seconds. And then. Got it. Now that’s kind of sucky because what if it only takes one second, then you lose four seconds of your time.

Every single time you run the scraper. So we need something else. And so I was like, oh man, I really wish I could find out when this doctor, because the documents that the elements were waiting for are already there. So you can’t say wait for a selector. Like we normally would like, normally I’d say like, okay, well, why don’t you wait for a selector for, I actually have it.

Let me re. Why don’t you wait for a selector

right here. This waits for selector for the filing history, right? It’s this huge ID, sorry. CSUs is fighting. We’re waiting for that selector. Now that thing is that’s going to resolve immediately that promise why, because it’s already in the Dom, it already exists, but then dun, dun dun dun. I Googled what to do if I want to wait for an element to be visible on the page and look, I’m so stupid.

There it is. And the options you can set visible. True. So you can come over here, click right there. You can go like, okay, well I want to pass some options and I say visible. And then it only resolves wait for the all Elon to be present in the Dom and to be visible. I E not to have display none or visibly hidden.

So bam resolves the problem. Puppeteer has these options that we built in. I wonder how new it is. I wish you could see a change, like just for this, but because I don’t think that was there before maybe, or you could do the vice versa. You could say, wait for element to be hidden if you wanted. I don’t know why you’d want to do that.

That seems. No, but who knows? Right? Maybe you’re clicking and making sure something’s done. I zoomed in and now it messed me up. Hold on. Where am I?

Whatever. Wait for a selector. That one. So there we go. Or you could also say, and this, I use a lot, the timeout I use all the time. Boy, look at that. I just really miss them. I spot doesn’t it because the default is 30 seconds. Now, most of the time. I know that it should be there in less than 30 seconds. It should take at most like five seconds if it’s a slow page.

So what I do is I pass and wait for timeout five seconds. Now what happens if that breaks? Well, if it’s taken more than five seconds, it’s probably broke anyway. So let’s wait a full, that’s not way to hold 30 seconds to handle that exception. Let’s just wait five seconds when we expect it to be done. And then we do it.

We could have. Okay, so that’s it for today. The important thing I want you to remember is wait for a selector has an option called visible equals true. So at the stuff’s already in the Dom and you just want them to be display block or whatever, actually displayed in the Dom. Then you can pass as visible, true in a wait for that selector.

Hopefully that helps you puppeteer timing, waiting for a selectors goodbye.