未导入 CSV 的“命令行”中显示的 Python 抓取数据

Question

我正在处理一个Scraping 项目，数据在启动时显示在命令行中，但在 CSV 文件中，所有数据都没有导入

这是在 CSV 中显示和添加数据的代码的一部分

from selenium import webdriver 
import csv

myFile = open('amazon_test.csv', 'w')
fild_name=['link','title','category','old_price1','old_price2','old_price3','new_price','desc','image']
writer=csv.DictWriter(myFile, fieldnames=fild_name)
writer.writeheader()

driver.get('https://www.amazon.com/international-sales-offers/b/?ie=UTF8&node=15529609011&ref_=nav_navm_intl_deal_btn')#MAIN PAGE OF AMAZON DEALS PAGE


print(links_list[details_product])
writer.writerow({'link': links_list[details_product]})

print('cat>>',cat_title)#PRINT CATEGORY NAME
writer.writerow({'category': cat_title})

writer.writerow({'title': product_title})

print(driver.find_element_by_css_selector('#priceblock_ourprice').text)
writer.writerow({'old_price1': product_old_price_1})

print(driver.find_element_by_css_selector('#price > table > tbody > tr:nth-child(1) > td.a-span12.a-color-secondary.a-size-base > span.priceBlockStrikePriceString.a-text-strike').text)
writer.writerow({'old_price2': product_old_price_1})

print(driver.find_element_by_css_selector('#priceblock_saleprice').text)
writer.writerow({'old_price3': product_old_price_1})

print(driver.find_element_by_css_selector('#priceblock_dealprice').text)
writer.writerow({'new_price': product_new_price})

print(driver.find_element_by_css_selector('#feature-bullets > ul').text)
writer.writerow({'desc': product_desc})

print(driver.find_element_by_css_selector('#landingImage').get_attribute('src'))
writer.writerow({'image': product_image})

Answer 1

作为替代方法，最好的方法是使用pandas library.Crate pandas DataFrame 并导入 csv 文件。我刚刚提供了一些字段来展示它是如何工作的。我相信你可以做剩下的事情。

如果您的机器上没有安装pandas，只需在命令提示符下使用以下命令。

pip 安装熊猫

代码：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
import pandas as pd

driver=webdriver.Chrome()
driver.get('https://www.amazon.com/international-sales-offers/b/?ie=UTF8&node=15529609011&ref_=nav_navm_intl_deal_btn')#MAIN PAGE OF AMAZON DEALS PAGE
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//span[@class='a-label a-checkbox-label' and contains(.,'Arts, Crafts & Sewing')]"))).click()
WebDriverWait(driver,15).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"div.a-section.dealContainer")))
link=driver.find_element_by_css_selector("div.a-section.dealContainer #dealTitle").get_attribute("href")
title=driver.find_element_by_css_selector("div.a-section.dealContainer #dealTitle>span").text.strip()
category=driver.find_element_by_css_selector(".a-link-normal.summary>span").text.strip()
newprice=driver.find_element_by_css_selector("span.dealPriceText").text.strip()
imageurl=driver.find_element_by_css_selector("img[src*='https://images-na.ssl-images-amazon.com/images/I']").get_attribute("src")
df=pd.DataFrame([[link,title,category,newprice,imageurl]],columns=['link','title','category','new_price','image'])
df.to_csv('amazon_test.csv',index=False)

生成的 csv 文件。

请注意：-单条记录的输出。如果您想为其余元素执行此操作，则必须使用find_elements_by_css_selector() 获取元素列表然后迭代。

Answer 2

您到底期望什么？每次调用writer.writerow() 时，都会写入一个新的行，其中包含字典中指示的单元格。如果你想写一行，首先用所有的键和数据准备你的dict，然后把它写在一个writerow()中，例如：

from selenium import webdriver 
import csv

driver.get('https://www.amazon.com/international-sales-offers/b/?ie=UTF8&node=15529609011&ref_=nav_navm_intl_deal_btn')
fild_name=['link','title','category','old_price1','old_price2','old_price3','new_price','desc','image']

with open('amazon_test.csv', 'w', encoding='utf-8') as myFile:

    writer = csv.DictWriter(myFile, fieldnames=fild_name)
    writer.writeheader()

    data = {'link': links_list[details_product],
            'category': cat_title,
            'title': product_title,
            'old_price1': product_old_price_1} # <--- ADD OTHER DATA...
    # write row to CSV
    writer.writerow(data)