将 Python 文本表转置为 Pandas DF，然后转置为 CSV

Question

[![在此处输入图像描述][1]][1][![在此处输入图像描述][1]][1]通过以下代码将 HTML Web 表格打印为 Python 脚本输出。 然后我尝试将其转换为 Pandas DF，然后导出为 CSV，但失败了

import time
from bs4 import BeautifulSoup
from selenium import webdriver
import pandas as pd

url = 'http://www.altrankarlstad.com/wisp'

driver = webdriver.Chrome('C:\\Users\\rugupta\\AppData\\Roaming\\Microsoft\\Windows\\Start Menu\\Programs\\Python 3.7\\chromedriver.exe')

driver.get(url)
time.sleep(100) 

text_field = driver.find_elements_by_xpath('//*[@id="root"]/div/div/div/div[2]/table')
#print (text_field[0].text)
data= text_field[0].text
#Works fine until above section

df= pd.DataFrame(data)
df.to_csv("output.csv")
(but no success here)!

[![enter image description here][1]][1]


  [1]: https://i.stack.imgur.com/NpGk2.jpg

Answer 1

问题在于 selenium 会检测到页面已加载，但是，您需要它等待包含您尝试抓取的数据的表已加载。所以你需要告诉 selenium 等到找到表中的元素。对于这种特殊情况，表中的每个“作业”都由名为“css-58”的特定类名定义。解决方案如下：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
import time
import pandas as pd


url = 'http://www.altrankarlstad.com/wisp'

driver = webdriver.Chrome("C:\\driver path")
driver.get(url)

# delay is how long to wait on loading the page before it gives up
delay = 600

try:
    wait_for_element = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CLASS_NAME, 'css-58')))
    text_field = driver.find_elements_by_xpath('//*[@id="root"]/div/div/div/div[2]/table')
    data= text_field[0].text

    # Create your dataframe here
    # This will currently fail due to the error
    # ValueError: DataFrame constructor not properly called!
    # You should be able to define the structure of your data frame to suit your needs
    df= pd.DataFrame(data)
    df.to_csv("output.csv")
except TimeoutException:
    print('It took too long')

此时您唯一需要做的就是弄清楚您希望如何定义数据框的结构。