我如何才能找到网站上的所有昆虫名称？

Question

我对使用什么样的查找方法以及 Xpath 来获取站点中包含 class="wordlist-item" 的所有元素感到非常迷茫

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep

insectNames = [] 
targetSite = "https://www.enchantedlearning.com/wordlist/insect.shtml"

browser = webdriver.Chrome(executable_path=r"C:\chromedriver\chromedriver.exe")
browser.get(targetSite)

bugName = browser.find_elements_by_xpath("//div[@class="wordlist-item"]")
print(bugName)

Answer 1

请密切注意您要使用的任何函数的格式。以下行会产生错误，因为您需要在 'wordlist_item' 周围使用单引号。

bugName = browser.find_elements_by_xpath("//div[@class="wordlist-item"]")

原因是你传递了一个字符串"//div[@class="wordlist-item"]"，当程序遇到第一组引号时，它将在第二组停止读取，然后在第三组重新开始并在第四组结束，离开你有 3 个值而不是 1 个。

"//div[@class="wordlist-item和"]"

除此之外，仅使用 print(bugName) 会返回您使用的浏览器、会话和元素 ID。如果你想访问元素的文本，你必须使用 print(bugName.text) 但是因为在这种情况下你会在一个列表中获得多个元素，你必须遍历列表并单独打印出名称。

from selenium import webdriver

url = "https://www.enchantedlearning.com/wordlist/insect.shtml"

browser = webdriver.Chrome(executable_path=r"C:\chromedriver\chromedriver.exe")
browser.get(url)

bugName = browser.find_elements_by_xpath("//div[@class='wordlist-item']")

for bug in bugName:
    print(bug.text)

附带说明，如果您只想从互联网上抓取数据，我建议您查看BeautifulSoup。它使用起来更直观一点，您不必每次运行程序时都打开浏览器。当您需要与页面交互时，Selenium 更适合。

Answer 2

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep


targetSite = "https://www.enchantedlearning.com/wordlist/insect.shtml"

browser = webdriver.Chrome(executable_path='C:/bin/chromedriver.exe')
browser.get(targetSite)
bugNames = browser.find_elements_by_xpath("//div[@class=\"wordlist-item\"]")
for bugname in bugNames:
    print(bugname.text)

输出：-

admiral butterfly
ambush bug
ant
aphid
armyworm
assassin bug
atlas moth
backswimmer
bedbug
bee
beetle
blue morpho butterfly
bluet
borer
brown butterfly
buckeye butterfly
bug
bumblebee
butterfly
carpenter ant
caterpillar
chrysalis
cicada
cockroach
comma butterfly
copper butterfly
crane fly
cricket
cutworm
damselfly
darkling beetle
dragonfly
dung beetle
earwig
egg...

Answer 3

要使用Selenium 和python 提取所有昆虫名称*，您必须为visibility_of_all_elements_located() 诱导WebDriverWait，您可以使用以下Locator Strategies 之一：

• 使用CSS_SELECTOR：

  driver.get('https://www.enchantedlearning.com/wordlist/insect.shtml')
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.wordlist-item")))])
  

• 使用XPATH：

  driver.get('https://www.enchantedlearning.com/wordlist/insect.shtml')
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='wordlist-item']")))])
  

• 控制台输出：

  ['admiral butterfly', 'ambush bug', 'ant', 'aphid', 'armyworm', 'assassin bug', 'atlas moth', 'backswimmer', 'bedbug', 'bee', 'beetle', 'blue morpho butterfly', 'bluet', 'borer', 'brown butterfly', 'buckeye butterfly', 'bug', 'bumblebee', 'butterfly', 'carpenter ant', 'caterpillar', 'chrysalis', 'cicada', 'cockroach', 'comma butterfly', 'copper butterfly', 'crane fly', 'cricket', 'cutworm', 'damselfly', 'darkling beetle', 'dragonfly', 'dung beetle', 'earwig', 'egg', 'fire ant', 'firefly', 'flea', 'fly', 'fritillary butterfly', 'fruit fly', 'gnat', 'gossamer-winged butterfly', 'grasshopper', 'green darner dragonfly', 'ground beetle', 'grub', 'gypsy moth', 'hairstreak butterfly', 'harlequin bug', 'honeybee', 'hornet', 'horse fly', 'house fly', 'hover fly', 'imago', 'insect', 'Japanese beetle', 'Julia butterfly', 'jumping bean', 'June bug', 'katydid', 'kissing bug', 'lacewing', 'ladybug', 'larva', 'leafcutter ant', 'leafhopper', 'lice', 'lightning bug', 'locust', 'longhorn beetle', 'louse', 'luna moth', 'maggot', 'mantid', 'mantis', 'mayfly', 'meadowhawk', 'mealworm', 'metalmark butterfly', 'metamorphosis', 'midge', 'milkweed bug', 'monarch', 'morpho', 'mosquito', 'moth', 'nymph', 'Oregon silverspot butterfly', 'owl butterfly', 'painted lady butterfly', 'paper wasp', 'planthopper', 'pond skater', 'praying mantid', 'praying mantis', 'pupa', "Queen Alexandra's birdwing butterfly", 'roach', 'robber fly', 'scarab', 'silkworm', 'silverfish', 'skipper', 'snout butterfly', 'spittlebug', 'springtail', 'stag beetle', 'stink bug', 'stonefly', 'sulphur butterfly', 'swallowtail butterfly', 'termite', 'thrip', 'tiger beetle', 'tiger moth', 'tsetse fly', 'Ulysses butterfly', 'viceroy butterfly', 'walking stick', 'wasp', 'water boatman', 'water bug', 'water strider', 'weevil', 'wood nymph butterfly', 'wood-borer', 'woolly bear caterpillar', 'yellow-white butterfly', 'yellowjacket', 'zebra longwing butterfly', 'zebra swallowtail butterfly']
  

• 注意：您必须添加以下导入：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC