selenium - Scraping links for odds from nowgoal using python

Question

I am working on getting the odds from Nowgoal. Firstly, I like to get the links for each of the match using selenium and beautifulsoup. I do not wish to get link for all matches. but I will have an input txt file, which is attached Here and use the selected league and date.

The following code will initialize as input:

#Intialisation
league_index =[]
final_list = []
j = 0
#config load
config = RawConfigParser()
configFilePath = r'.config.txt'
config.read(configFilePath)
date = config.get('database_config','date')                     #input file provided by user - provide in YYYY-MM-DD format
leagues = config.get('database_config','leagues')               #input file provided by user - provide in windows format
headless_param =config.get('database_config','headless')        #Headless param - set True if you want to see bowser operating in foreground!
leagues_list = leagues.split(',')
print(leagues_list)

After I initialized with the preferred date and league, I will set up for chrome driver as follow:

options = webdriver.ChromeOptions()         #initialise webdriver options
#options.binary_location = brave_path        #if you are running the script on brave - then enable it
if headless_param == 'True' :
    print('headless')
    options.headless = True                 # if headeless parameter is set to true - the chrome browser will not appear in foreground
options.add_argument('start-maximized')     # Start the chrome maximised 
options.add_argument('disable-infobars')    # Disable infobars
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("prefs", {"profile.default_content_setting_values.cookies": 2})
options.add_experimental_option("prefs", {"profile.block_third_party_cookies": True})
options.add_experimental_option('useAutomationExtension', False)
options.add_argument("--incognito")         #Incognito mode


#intiate the driver
driver = webdriver.Chrome(resource_path('./drivers/chromedriver.exe'),options=options) 
#Format the url

url =  'http://www.nowgoal3.com/football/fixture/?f=ft0&date='+date


#get the url
driver.get(url)
#wait for some time
time.sleep(3)

driver.find_element_by_xpath('//*[@id="li_league"]').click()
time.sleep(5)
#click on the -team ranking
driver.find_element_by_xpath('//*[@id="TeamOrderCheck"]').click()

After I run upto above code, I have the following error:

 > driver.find_element_by_xpath('//*[@id="TeamOrderCheck"]').click() 
  File "C:UsersA100732AppDataLocalContinuumanaconda3libsite-packagesseleniumwebdriver
emoteerrorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable

I cannot locate the xpath to change it. therefore, I cannot run the program further.

I am seeking your kind advice on that. I would be greatly appreciate for your help. Thanks, Zep.

question from:https://stackoverflow.com/questions/65914124/scraping-links-for-odds-from-nowgoal-using-python

Answer 1

Target the span. Also added webdriver waits to make it more stable instead of time.sleep().

wait = WebDriverWait(driver,3)
#wait for some time
wait.until(EC.element_to_be_clickable((By.ID, "li_league"))).click()
#click on the -team ranking
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[@for='TeamOrderCheck']/span"))).click()

Imports

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC