Python Selenium 从文件中获取 URL 列表

Question

我基本上想抓取存储在 csv 文件中的多个 url 并为每个返回结果。

ips = "C:/Projects/file.csv"
cpes = open (ips, 'r')
cpes.seek(0)

#Assign prefix and suffix to ip in each line
for lines in cpes.readlines():
    ip = lines.strip()
    urls = 'https://' + ip
    overview = 'https://' + ip + '/overview.asp'

options = webdriver.ChromeOptions() #Enable Chrome driver options
options.add_argument('headless') #No browser is opened to view process
options.add_experimental_option('excludeSwitches', ['enable-logging']) #Disable Devtools log message
browser = webdriver.Chrome(r'C:\chromedriver\chromedriver.exe')
for url in urls:
    browser.get(url)
    time.sleep(2)
    username = browser.find_element_by_name("username")
    password = browser.find_element_by_name("password")
    username.send_keys(credentials.usrn)
    password.send_keys(credentials.pswd, Keys.ENTER)
    browser.get(overview)
    source = browser.page_source

输出是

Traceback (most recent call last):
  File "cpe3.py", line 28, in <module>
    browser.get(url)
  File "C:\Users\duane.barker\AppData\Roaming\Python\Python37\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
    self.execute(Command.GET, {'url': url})
  File "C:\Users\duane.barker\AppData\Roaming\Python\Python37\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Users\duane.barker\AppData\Roaming\Python\Python37\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument

Answer 1

您似乎正在将变量：urls 和 overview 设置为 string 类型。我相信您正在尝试创建每个列表，并在循环中使用这些列表来导航页面，试试这个：

#Assign prefix and suffix to ip in each line
urls = []
overviews = []
for lines in cpes.readlines():
    ip = lines.strip()
    urls.append('https://' + ip)
    overviews.append('https://' + ip + '/overview.asp')

这样，当您遍历urls 时，url 将是一个完整的 URL，而不是最后一个字符串的第一个字母。即h 来自https://。您还需要对循环进行这些更改：

for url, overview in zip(urls, overviews):
    browser.get(url)
    time.sleep(2)
    username = browser.find_element_by_name("username")
    password = browser.find_element_by_name("password")
    username.send_keys(credentials.usrn)
    password.send_keys(credentials.pswd, Keys.ENTER)
    browser.get(overview)
    source = browser.page_source

这应该可以缓解您在尝试使用 .get() 方法导航时的错误，因为您现在正在前往实际页面。