web scraping - Python - BeautifulSoup does not produce the same code

Question

I am new in python and I would like to web scrap some information in this website:

import requests

from bs4 import BeautifulSoup

URL = 'https://www.leroymerlin.com.br/piso-ceramico-interno-madeira-esmaltado-borda-arredondada-reale-noce-61x61cm-rosagres_89371261'
page = requests.get(URL)
soup = BeautifulSoup(page.content)
mydivs = soup.findAll("div", {"class": "carousel"})

But mydivs is empty, but in the view-source there is this line. Likewise:

for link in soup.find_all('a'):
    print(link.get('href'))

Give me links that I do not see in view-source and do not give me some links that I see in view-source. I thought it was a iframe issue, but I tried and I could not solve it.

Could anyone help me?

Answer 1

I think the problem here is that python is receiving a different web page because it's being redirected to https://www.leroymerlin.com.br/localizacao.

In order to arrive to the webpage you're scraping, you should set region cookie so you don't get redirected:

page = requests.get(URL, cookies={'region': 'eyJpdiI6IlN1SUhieDhmUGtxbDVzcHIyZVgrSUE9PSIsInZhbHVlIjoiZHhycHE0MGRlVWNkNHk0U2NDSCtCZz09IiwibWFjIjoiMmFiMjVhZTFjMjFhNWU4NzkyZmFiNTUxOGM5MWYzMmIzYmJiYzNkMDk4NDJjYWFiYTdkNDVjNjMzYzVjNWNhMCJ9'})

I'm using eyJpdiI6IlN1SUhieDhmUGtxbDVzcHIyZVgrSUE9PSIsInZhbHVlIjoiZHhycHE0MGRlVWNkNHk0U2NDSCtCZz09IiwibWFjIjoiMmFiMjVhZTFjMjFhNWU4NzkyZmFiNTUxOGM5MWYzMmIzYmJiYzNkMDk4NDJjYWFiYTdkNDVjNjMzYzVjNWNhMCJ9 as region cookie, you can check the one you're using looking at the packets from your web browser development console

And this is the response:

>>> mydivs = soup.findAll("div", {"class": "carousel"})
>>> mydivs
[<div class="carousel" data-carousel-tracking-value="89371261" data-items='[{"alt":"Piso Ceru00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_52x52.jpeg","youtubeId":""},{"alt":"Piso Ceru00e2mico Esmaltado Borda Arredondada 61x61cm modelo Reale Noce Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_esmaltado_brilhante_61x61cm_modelo_reale_noce_rosagres_89371261_ed4d_600x600.jpg","zoomDimensions":"600","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_esmaltado_brilhante_61x61cm_modelo_reale_noce_rosagres_89371261_ed4d_600x600.jpg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_esmaltado_brilhante_61x61cm_modelo_reale_noce_rosagres_89371261_ed4d_52x52.jpg","youtubeId":""},{"alt":"Piso Ceru00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_4550_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_4550_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_4550_52x52.jpeg","youtubeId":null},{"alt":"Piso Ceru00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_586b_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_586b_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_586b_52x52.jpeg","youtubeId":null},{"alt":"Piso Ceru00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_51ba_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_51ba_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_51ba_52x52.jpeg","youtubeId":null},{"alt":"Piso Ceru00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_a684_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_a684_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_a684_52x52.jpeg","youtubeId":null},{"alt":"Piso Ceru00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_09b7_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_09b7_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_09b7_52x52.jpeg","youtubeId":null},{"alt":"Piso Ceru00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_8e45_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_8e45_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_8e45_52x52.jpeg","youtubeId":null},{"alt":"Piso Ceru00e2mico Brilhante Borda Arredondado Reale Noce 61x61cm Rosagru00eas","url":"https://cdn.leroymerlin.com.br/products/piso_ceramico_borda_arredondado_reale_noce_61x61cm_rosagres_89371261_0001_600x600.jpg","zoomDimensions":"600","zoomUrl":"https://cdn.leroymerlin.com.br/products/piso_ceramico_borda_arredondado_reale_noce_61x61cm_rosagres_89371261_0001_600x600.jpg","shouldLoadImageZoom":true,"thumb":"https://cdn.leroymerlin.com.br/products/piso_ceramico_borda_arredondado_reale_noce_61x61cm_rosagres_89371261_0001_52x52.jpg","youtubeId":null}]' data-product-images-carousel="" data-sketchfab-id="7a24a14dabc44227b465d49673d144e4" data-sketchfab-placeholder="https://cdn.leroymerlin.com.br/assets/images/sketchfab-placeholder.svg" data-sketchfab-thumb="https://cdn.leroymerlin.com.br/assets/images/sketchfab-3d-model.svg">
<div>
<picture>
<source media="(max-width: 768px)" srcset="https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_220x220.jpeg"/>
<source media="(min-width: 769px)" srcset="https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_300x300.jpeg"/>
<img alt="Piso Ceramico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagrês" class="image" src="https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_300x300.jpeg"/>
</picture>
</div>
</div>]
>>>