In this example, I'm using the following: The final script can be found at this gist. The first step of this process was to capture the network traffic of the page, which this post will show you how to do. The specific information I needed wasn't being rendered in the response, but it was being communicated through the websocket.
I knew I could use this information because I found it inside a websocket while inspecting the network traffic using the 'network' tab of the developer tools.
Instead, I needed to know what was being transmitted in the websocket the page had opened. I recently had a webscraping project for which I did not need any information from the response page itself. If you need an introduction to why "dynamic" webpages are harder to scrape, see my explanatory post here. Capturing network traffic can be useful for any number of things when dealing with dynamic webpages, like grabbing websocket communications or data from the private APIs of a website. I'll show you how to capture the network traffic occurring on the page using python, selenium (with chromedriver), and the logging features of Chrome.
0 kommentar(er)
