Selenium 代码在本地工作，但不能在 PythonAnywhere 上工作

Question

我的系统上正在运行一个网络爬虫，我想将它迁移到 PythonAnywhere，但是当我现在移动它时它不起作用。

确切地说，sendkeys 似乎不起作用 - 执行以下代码后，我再也没有转到下一个网页，因此会触发属性错误。

我的代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import csv
import time



# Lists for functions
parcel_link =[]
token = []
csv_output = [ ]

# main scraping function
def getLinks(link):


# Open web browser and get url - 3 second time delay.

#Open web browser and get url - 3 second time delay.
driver.get(link)
time.sleep(3)

inputElement = driver.find_element_by_id("mSearchControl_mParcelID")
inputElement.send_keys(parcel_code+"*/n")
print("ENTER hit")

pageSource = driver.page_source
bsObj = BeautifulSoup(pageSource)
parcel_link.clear()
print(bsObj)
#pause = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID, "mResultscontrol_mGrid_RealDataGrid")))

for link in bsObj.find(id="mResultscontrol_mGrid_RealDataGrid").findAll('a'):
    parcel_link.append(link.text)
print(parcel_link)

for test in parcel_link:
    clickable = driver.find_element_by_link_text(test)
    clickable.click()
    time.sleep(2)

我要操作的链接是： https://ascendweb.jacksongov.org/ascend/%280yzb2gusuzb0kyvjwniv3255%29/search.aspx

我正在尝试发送：15-100*

追溯：

03:12 ~/Tax_Scrape $ xvfb-run python3.4 Jackson_Parcel_script.py
Traceback (most recent call last):
  File "Jackson_Parcel_script.py", line 377, in <module>
    getLinks("https://ascendweb.jacksongov.org/ascend/%28biohwjq5iibvvkisd1kjmm45%29/result.aspx")
  File "Jackson_Parcel_script.py", line 29, in getLinks
    inputElement = driver.find_element_by_id("mSearchControl_mParcelID")
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 206, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 662, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: 'Unable to locate element: {"method":"id","selector":"mSearchControl_mParcelID"}' ; Stac
ktrace: 
    at FirefoxDriver.findElementInternal_ (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/driver_component.js:9470)
    at FirefoxDriver.findElement (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/driver_component.js:9479)
    at DelayedCommand.executeInternal_/h (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/command_processor.js:11455)
    at DelayedCommand.executeInternal_ (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/command_processor.js:11460)
    at DelayedCommand.execute/< (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/command_processor.js:11402) 
03:13 ~/Tax_Scrape $ 

硒启迪：

for retry in range(3):
    try:
        driver = webdriver.Firefox()
        break
    except:
        time.sleep(3)

for parcel_code in token:
    getLinks("https://ascendweb.jacksongov.org/ascend/%28biohwjq5iibvvkisd1kjmm4 5%29/result.aspx")

PythonAnywhere 使用 FireFox 的虚拟实例，该实例假定为像 JSPhantom 一样无头，因此我没有版本号。

任何帮助都会很棒

RS

Answer 1

好吧，也许PythonAnywhere 上使用的浏览器加载站点的速度不够快。因此，不要使用 time.sleep(3) 尝试使用 implicitly waiting 作为元素。

隐式等待是告诉 WebDriver 轮询 DOM 尝试查找一个或多个元素（如果它们是）时的时间量 无法立即使用。默认设置为 0。一旦设置， 为 WebDriver 对象实例的生命周期设置了隐式等待。

将time.sleep() 与Selenium 一起使用通常不是一个好主意。

并且给它不仅仅是3 秒，implicitly_wait() 您指定等待元素的最长时间。
因此，如果您设置implicitly_wait(10) 并且页面加载，例如，在5 秒内，那么Selenium 将只等待5 秒。

driver.implicitly_wait(10)