scraping - selenium webdriver python

Enfrentando problemas seleccionando el 2do elemento del menú desplegable de sugerencias

Question

He escrito un guión en python con selenio para raspar la ciudad y zip de los mapas de Google usando la búsqueda inversa. Está funcionando bien hasta ahora. El único requisito que no puedo cumplir es: después de poner una dirección en el cuadro de búsqueda, a veces aparecen algunas sugerencias como un menú desplegable para elegir. Por defecto elige el primero. Si deseo ir por el segundo de la sugerencia, me quedo atascado.

Aquí está el script con el que estoy tratando:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

search_street = [''10123 HARDISON'',''1006 ELEANOR ST'']

def find_address(driver, wait):
    for item in range(0,2):
        driver.get("https://maps.google.com/maps?q=Houston")
        wait = WebDriverWait(driver, 10)
        input_val = wait.until(EC.presence_of_element_located((By.NAME, "q")))
        input_val.clear()
        input_val.send_keys(search_street[item])
        driver.find_element_by_id(''searchbox-searchbutton'').click()

        # wait.until(EC.presence_of_element_located((By.XPATH, "//div[@id=''sbse6'']")))
        # driver.find_elements_by_xpath("//div[@id=''sbse6'']")[2].click()

        wait.until(EC.presence_of_element_located((By.XPATH, "//div[@class=''section-hero-header-description'']")))  
        element = driver.find_element_by_xpath("//div[@class=''section-hero-header-description'']")
        name = driver.execute_script("return arguments[0].childNodes[3].textContent", element).strip()
        print(name)

if __name__ == ''__main__'':
    driver = webdriver.Chrome()
    wait = WebDriverWait(driver, 10)
    try:
        find_address(driver, wait)
    finally:
        driver.quit()

Elemento html para el menú desplegable de sugerencias:

<div class="sbdd_b" style=""><div class="suggestions"><ul class="sbsb_b" role="listbox"><li role="presentation" class="sbsb_c " dir="ltr" style="text-align: left;"><div role="option" id="sbse5"><div class="sbqs_a"></div><div class="sbqs_c"><div jstcache="746" class="suggest" jsan="7.suggest"><div jstcache="747" jsinstance="*0" class="suggest-text-layout"><div jstcache="748" class="suggest-left-cell"><div role="presentation" jstcache="749" class="suggest-icon-container maps-sprite-suggest-place-pin" jsan="7.suggest-icon-container,7.maps-sprite-suggest-place-pin,0.role"></div><img role="presentation" jstcache="750" class="suggest-activity" style="display:none"><span jstcache="751" style="display:none"><span jstcache="752" class="suggest-query" jsan="7.suggest-query"></span> nearby</span><span jstcache="753" class="suggest-query" jsan="7.suggest-query,t-fmS2vATOQZY"><span jstcache="764" jsinstance="0" class=" suggest-bold" jsan="7.suggest-bold">10123 Valley Park Dr</span><span jstcache="764" jsinstance="*1" class="">ive</span></span> <span jstcache="754"><span jstcache="764" jsinstance="*0">Houston, TX</span></span></div><div class="suggest-right-cell"><button jstcache="755" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">set location</button><button jstcache="756" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">edit</button><span jstcache="757" style="display:none"><div class="suggest-ad-creative-annotation">Ad</div><span jstcache="758"></span></span></div></div></div></div></div></li><li role="presentation" class="sbsb_c " dir="ltr" style="text-align: left;"><div role="option" id="sbse6"><div class="sbqs_a"></div><div class="sbqs_c"><div jstcache="746" class="suggest" jsan="7.suggest"><div jstcache="747" jsinstance="*0" class="suggest-text-layout"><div jstcache="748" class="suggest-left-cell"><div role="presentation" jstcache="749" class="suggest-icon-container maps-sprite-suggest-place-pin" jsan="7.suggest-icon-container,7.maps-sprite-suggest-place-pin,0.role"></div><img role="presentation" jstcache="750" class="suggest-activity" style="display:none"><span jstcache="751" style="display:none"><span jstcache="752" class="suggest-query" jsan="7.suggest-query"></span> nearby</span><span jstcache="753" class="suggest-query" jsan="7.suggest-query,t-fmS2vATOQZY"><span jstcache="764" jsinstance="0" class=" suggest-bold" jsan="7.suggest-bold">10123 Valley Park Dr</span><span jstcache="764" jsinstance="*1" class="">ive</span></span> <span jstcache="754"><span jstcache="764" jsinstance="*0">Garland, TX</span></span></div><div class="suggest-right-cell"><button jstcache="755" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">set location</button><button jstcache="756" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">edit</button><span jstcache="757" style="display:none"><div class="suggest-ad-creative-annotation">Ad</div><span jstcache="758"></span></span></div></div></div></div></div></li><li role="presentation" class="sbsb_c " dir="ltr" style="text-align: left;"><div role="option" id="sbse7"><div class="sbqs_a"></div><div class="sbqs_c"><div jstcache="746" class="suggest" jsan="7.suggest"><div jstcache="747" jsinstance="*0" class="suggest-text-layout"><div jstcache="748" class="suggest-left-cell"><div role="presentation" jstcache="749" class="suggest-icon-container maps-sprite-suggest-place-pin" jsan="7.suggest-icon-container,7.maps-sprite-suggest-place-pin,0.role"></div><img role="presentation" jstcache="750" class="suggest-activity" style="display:none"><span jstcache="751" style="display:none"><span jstcache="752" class="suggest-query" jsan="7.suggest-query"></span> nearby</span><span jstcache="753" class="suggest-query" jsan="7.suggest-query,t-fmS2vATOQZY"><span jstcache="764" jsinstance="0" class=" suggest-bold" jsan="7.suggest-bold">10123 Valley Park Dr</span><span jstcache="764" jsinstance="*1" class="">ive</span></span> <span jstcache="754"><span jstcache="764" jsinstance="*0">Garden Ridge, TX</span></span></div><div class="suggest-right-cell"><button jstcache="755" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">set location</button><button jstcache="756" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">edit</button><span jstcache="757" style="display:none"><div class="suggest-ad-creative-annotation">Ad</div><span jstcache="758"></span></span></div></div></div></div></div></li><li role="presentation" class="sbsb_c " dir="ltr" style="text-align: left;"><div role="option" id="sbse8"><div class="sbqs_a"></div><div class="sbqs_c"><div jstcache="746" class="suggest" jsan="7.suggest"><div jstcache="747" jsinstance="*0" class="suggest-text-layout"><div jstcache="748" class="suggest-left-cell"><div role="presentation" jstcache="749" class="suggest-icon-container maps-sprite-suggest-place-pin" jsan="7.suggest-icon-container,7.maps-sprite-suggest-place-pin,0.role"></div><img role="presentation" jstcache="750" class="suggest-activity" style="display:none"><span jstcache="751" style="display:none"><span jstcache="752" class="suggest-query" jsan="7.suggest-query"></span> nearby</span><span jstcache="753" class="suggest-query" jsan="7.suggest-query,t-fmS2vATOQZY"><span jstcache="764" jsinstance="0" class=" suggest-bold" jsan="7.suggest-bold">10123 Valley Park Dr</span><span jstcache="764" jsinstance="*1" class="">ive</span></span> <span jstcache="754"><span jstcache="764" jsinstance="*0">San Antonio, TX</span></span></div><div class="suggest-right-cell"><button jstcache="755" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">set location</button><button jstcache="756" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">edit</button><span jstcache="757" style="display:none"><div class="suggest-ad-creative-annotation">Ad</div><span jstcache="758"></span></span></div></div></div></div></div></li><li role="presentation" class="sbsb_c " dir="ltr" style="text-align: left;"><div role="option" id="sbse9"><div class="sbqs_a"></div><div class="sbqs_c"><div jstcache="746" class="suggest" jsan="7.suggest"><div jstcache="747" jsinstance="*0" class="suggest-text-layout"><div jstcache="748" class="suggest-left-cell"><div role="presentation" jstcache="749" class="suggest-icon-container maps-sprite-suggest-place-pin" jsan="7.suggest-icon-container,7.maps-sprite-suggest-place-pin,0.role"></div><img role="presentation" jstcache="750" class="suggest-activity" style="display:none"><span jstcache="751" style="display:none"><span jstcache="752" class="suggest-query" jsan="7.suggest-query"></span> nearby</span><span jstcache="753" class="suggest-query" jsan="7.suggest-query,t-fmS2vATOQZY"><span jstcache="764" jsinstance="0" class=" suggest-bold" jsan="7.suggest-bold">10123 Valley Park Dr</span><span jstcache="764" jsinstance="*1" class="">ive</span></span> <span jstcache="754"><span jstcache="764" jsinstance="*0">Temple, TX</span></span></div><div class="suggest-right-cell"><button jstcache="755" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">set location</button><button jstcache="756" jsaction="omnibox.editAlias" class="blue-button-text suggest-edit-link" style="display:none">edit</button><span jstcache="757" style="display:none"><div class="suggest-ad-creative-annotation">Ad</div><span jstcache="758"></span></span></div></div></div></div></div></li></ul></div></div>

El menú desplegable de sugerencias al que me refiero es este: image_link

Por cierto, el xpath con el que estaba tratando de seleccionar la segunda opción del menú desplegable de sugerencias también aparece en mi script, que está atenuado en este momento.

Answer 1

No pude hacerlo con un xpath posiblemente porque encontré que la página cambiaba muy rápido y los elementos se volvían obsoletos muy rápidamente. Sin embargo, puedo hacerlo con clases:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep

search_street = [''10123 HARDISON'',''1006 ELEANOR ST'']

def find_address(driver, wait):
    for item in range(0,2):
        driver.get("https://maps.google.com/maps?q=Houston")
        wait = WebDriverWait(driver, 10)
        input_val = wait.until(EC.presence_of_element_located((By.NAME, "q")))
        input_val.clear()
        input_val.send_keys(search_street[item])

        wait.until(EC.presence_of_element_located((By.CLASS_NAME, "suggest")))
        sleep(5) # To prevent stale element reference as the text changes.
        for element in driver.find_elements_by_class_name("suggest"):
            print (element.text)

if __name__ == ''__main__'':
    driver = webdriver.Chrome()
    wait = WebDriverWait(driver, 10)
    try:
        find_address(driver, wait)
    finally:
        driver.quit()

Productos:

10123 Hardison Lane Houston, TX
10123 Hardison Lane Abilene, TX
10123 Hardison Road Woodburn, KY
10123 Hardison Road Charlotte, NC
10123 Hardison Road Columbia, TN
1006 Eleanor Street College Station, TX
1006 Eleanor Street Houston, TX
1006 Eleanor Street Victoria, TX
1006 Eleanor Street La Marque, TX
1006 Eleanor Street Brenham, TX

Puede ejecutar un evento de clic en el elemento elegido de esta manera:

driver.find_elements_by_class_name("suggest")[2].click()