如何将for循环生成的数据转换为数据框？

Question

我使用 for 循环从网站中的表中提取数据 硒自动化网络驱动程序。如何将该数据转换为数据框 并导出为 csv 文件。我试图在 pandas 数据框中分配“值”，但它抛出错误。

from selenium import webdriver

url = "https://www.jambalakadi.info/status/"

driver = webdriver.Chrome(executable_path="chromedriver.exe")

driver.get(url)

row_count = len(driver.find_elements_by_xpath(" //*[@id='main_table_countries_today']/tbody[1]/tr "))
col_count = len(driver.find_elements_by_xpath(" //*[@id='main_table_countries_today']/tbody[1]/tr[1]/td "))

print('Number of row counts:', row_count)
print("Number of column counts:", col_count)


for r in range(2, row_count+1):
    for c in range(1, col_count+1):
        value = driver.find_element_by_xpath(" //*[@id='main_table_countries_today']/tbody[1]/tr["+str(r)+"]/td["+str(c)+"] ").text
        print(value, end=" ")

    print(" ")

当我运行 for 循环时，'value' 变量打印数据，但我 无法使用 pandas 创建数据框并将其导出为 CSV 文件。

我更新了代码格式是否正确？

my_data = []
for r in range(2, row_count+1):
    for c in range(1, col_count+1):
        value = driver.find_element_by_xpath(" //*[@id='main_table_countries_today']/tbody[1]/tr["+str(r)+"]/td["+str(c)+"] ").text
        print(value, end=" ")
        for line in value:
            my_data.append(line[0],line[1],line[2])
        pd.DataFrame.from_records(my_data, columns=column).to_csv('output.csv')

    print(" ")

Answer 1

这是使用pandas获取dataframe中的数据然后导入csv的代码。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
from bs4 import BeautifulSoup


driver=webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://yourwebsitename.com")
WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"table#main_table_countries_today")))
html=driver.page_source
soup=BeautifulSoup(driver.page_source,'html.parser')
table=soup.find('table',attrs={"id":"main_table_countries_today"})
df=pd.read_html(str(table))
print(df[0])
df[0].to_csv('output.csv',index=False)

更新：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd


driver=webdriver.Chrome(executable_path = "chromedriver.exe")
driver.get("https://yourwebsitename.com")
element=WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"table#main_table_countries_today")))
table=driver.execute_script("return arguments[0].outerHTML;",element)
df=pd.read_html(str(table))
print(df[0])
df[0].to_csv('output.csv',index=False)

Answer 2

你需要使用pd.DataFrame.from_records()函数

用例：

import pandas as pd
#Reading the data
my_data = []
for line in my_database:
    #preprocess the line (say you get 3 columns date,customer,price)
    #say you use line.split(" "), now your line is actually an array of values (line = line.split(" ")
    my_data.append([line[0],line[1],line[2]]) #each index corresponds to date, customer and price respectively

pd.DataFrame.from_records(my_data, columns=['date','customer','price']).to_csv('output.csv')