python - How to I scrape text from the first sub div tag which is under a Main div tag

Question

I want to scrape text from the first div tag which is inturn under another div tag

The HTML code looks like:

<div class="main">
 <div>The text I want to extract</div>
 <div>The text I dont want to extract</div>
</div>

The code I'm using is:

result = requests.get(url1, headers=headers)


soup1 = BeautifulSoup(result.text,"lxml")
name=soup1.find_all('div',attrs={'class': 'main'})
for div in name:
    print(div.get_text(strip=True).encode("utf-8"))

Thanks!

question from:https://stackoverflow.com/questions/65949716/how-to-i-scrape-text-from-the-first-sub-div-tag-which-is-under-a-main-div-tag

Answer 1

In your name variable you will get all div which have class named "main", then in your for loop you are currently looping through the collected div's, but you need to access the div's inside those div's(class : main) so add a nested loop like:

for div in name:
    innerDivs= div.find_all("div")
        for idiv in innerDivs:
            print(idiv.get_text(strip=True).encode("utf-8"))