Python Web Scraping Using Selenium

Question

Website to scrape https://idaman.kpkt.gov.my/idv5xe/98_eHome/maklumatProjek.cfm?pmju_kod=8898&proj_kod_Fasa=1

Item to scrape in BOLD - Part 1 (HTML Below)

<form onsubmit="return lucee_form_c9u.check();" name="myForm" enctype="multipart/form-data" action="mPPTProjek3.cfm?mn=BPPT" method="post">
    <div align="center" style="background-color: white; border: 1px solid grey;">
        <br />
        <table class="MainContent" width="100%" align="center">
            <tbody>
                <tr style="font-weight: bold;">
                    <td class="column" width="30%">Nama Pemaju</td>
                    <td>
                        :
                        <a style="color: blue;" href="maklumatPemaju.cfm?pmju_Kod=8877">**RAPID UNITY SDN. BHD.**</a>
                        <font color="red">* Klik Untuk Melihat Maklumat</font>
                    </td>
                </tr>
                <tr>
                    <td class="column">Kod Pemaju</td>
                    <td>: **8877<**/td></td>
                </tr>

                <tr>
                    <td class="column">Kod Fasa</td>
                    <td>: **1<**/td></td>
                </tr>

                <tr>
                    <td class="column">Nama Pemajuan</td>
                    <td>: **TAMAN UNITY**</td>
                </tr>
            </tbody>
        </table>
    </div>
</form>

question from:https://stackoverflow.com/questions/66045610/python-web-scraping-using-selenium

Answer 1

I think what you want can be obtained just using requests and beautifulsoup as follows:

import requests
from bs4 import BeautifulSoup

s = requests.Session()

params = {"pmju_Kod" : 8877, "proj_Kod_Fasa" : 1}
r = s.get("https://idaman.kpkt.gov.my/idv5xe/98_eHome/maklumatProjek.cfm", params=params)
soup = BeautifulSoup(r.content, "html.parser")

tables = soup.find_all('table', class_="MainContent")

items = []

items.append(tables[0].a.text)

data = [[td.text for td in tr.find_all('td')] for tr in tables[0].find_all('tr')]
items.append(data[1][1].strip(': '))
items.append(data[2][1].strip(': '))
items.append(data[3][1].strip(': '))

data = [[td.text for td in tr.find_all('td')] for tr in tables[3].find_all('tr')]

items.append(data[1][2].strip())
items.append(data[1][3].strip())
items.append(data[1][4])
items.append(data[1][5])
items.append(data[1][6])

items.append(data[2][2].strip())
items.append(data[2][3].strip())
items.append(data[2][4])
items.append(data[2][5])
items.append(data[2][6])

# Pemajuan table
params['rekid'] = 419975503
r2 = s.get('https://idaman.kpkt.gov.my/idv5xe/98_eHome/template/pemajuan.cfm', params=params)
soup2 = BeautifulSoup(r2.content, "html.parser")
table = soup2.find('table', class_="MainContent")
data = [[td.text for td in tr.find_all('td')] for tr in table.find_all('tr')]
items.append(data[-1][1].strip(': '))

print(items)

This would give you the following items:

['RAPID UNITY SDN. BHD.', '8877', '1', 'TAMAN UNITY', 'RUMAH BERKEMBAR', 'HARGA TINGGI', '1', '370,000.00', '394,900.00', 'RUMAH TERES', 'HARGA TINGGI', '1', '190,000.00', '290,550.00', '0%']