Is this possible to load the page after the javascript execute using python?

Question

Welcome To Ask or Share your Answers For Others

Is this possible to load the page after the javascript execute using python?

posted Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Is this possible to load the page after the javascript execute using python?

Here is the page I read:

<html>
<head>
<script type="text/javascript">
    document.write("Hello World")
</script>
</head>
<body>
</body>
</html>

As you can see, the Hello World is added on the HTML page using javascript, when I use the HTML parser, like the BeautifulSoup to parse it, it can't parse the Hello World, it is possible to me parse the actually result on how the client side really see....? Thanks.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…

1 Reply

深蓝 · Answer 1 · 2021-10-16T23:50:04+0000

I ran into a similar problem when writing web scrapers in python, and I found Selenium Web Driver in combination with BeautifulSoup very useful. The code ends up looking something like this:

from selenium import webdriver
browser = webdriver.Firefox()
browser.get("http://www.yoursite.com")
soup = BeautifulSoup(browser.page_source, "html.parser")
...

With Selenium WebDriver, there's also functionally for a "wait until a certain DOM element has loaded", which makes the timing with javascript elements easier too.

Categories

Is this possible to load the page after the javascript execute using python?

Is this possible to load the page after the javascript execute using python?

Please log in or register to add a comment.

Please log in or register to reply this article.

1 Reply

Please log in or register to add a comment.

Just Browsing Browsing

Most popular tags