Python Webscrape HTML to CSV File For Loop答案

【问题标题】：Python Webscrape HTML to CSV File For LoopPython Webscrape HTML to CSV File For Loop
【发布时间】：2019-05-28 05:50:54
【问题描述】：

我对 Python 还是很陌生，我的 For 循环很难提取某个站点上的所有 Web 链接。这是我的代码：

import requests
import csv
from bs4 import BeautifulSoup
j= [["Population and Housing Unit Estimates"]] # Title
k= [["Web Links"]] # Column Headings
example_listing='https://www.census.gov/programs-surveys/popest.html' #Source
r=requests.get(example_listing) #Grab page source html
html_page=r.text
soup=BeautifulSoup(html_page,'html.parser') #Build Beautiful Soup object to help parse the html
with open('HTMLList.csv','w',newline="") as f: #Choose what you want to grab
    writer=csv.writer(f,delimiter=' ',lineterminator='\r')
    writer.writerows(j)
    writer.writerows(k)
    for link in soup.find_all('a'):
        f.append(link.get('href'))
        if not f:
            ""
        else:
            writer.writerow(f)
f.close()

非常感谢任何帮助。我真的不知道从这里去哪里。谢谢！

【问题讨论】：

f 不是一个列表...你为什么要附加到它上面？
你能解释一下你想要的输出是什么以及你尝试了什么吗？也尝试使它稍微更小（查看如何制作minimal reproducible example）
旁注：如果您打开带有with 语句的文件，则不需要.close()。 with 自动执行。

标签： python csv for-loop beautifulsoup

【解决方案1】：

假设您尝试将站点中的 URL 保存到 CSV 文件中 - 每行一个 URL。首先不要重用f，那是为了文件。您可以通过将链接包含在数组 writer.writerow([link.get('href')]) 中来直接将链接写入 CSV。希望对您有所帮助。否则，请编辑您的问题并添加更多详细信息。

import csv
import requests
from bs4 import BeautifulSoup

j= [["Population and Housing Unit Estimates"]] # Title
k= [["Web Links"]] # Column Headings

example_listing='https://www.census.gov/programs-surveys/popest.html' #Source
r=requests.get(example_listing) #Grab page source html
html_page=r.text
soup=BeautifulSoup(html_page,'html.parser') #Build Beautiful Soup object to help parse the html
with open('HTMLList.csv','w', newline="") as f: #Choose what you want to grab
    writer=csv.writer(f, delimiter=' ',lineterminator='\r')
    writer.writerows(j)
    writer.writerows(k)
    for link in soup.find_all('a'):
        url = link.get('href')
        if url:
            writer.writerow([url])

【讨论】：

完美！我刚刚意识到有一堆重复的 URL。有没有办法确保没有重复的链接？

【解决方案2】：

import requests
import csv
from bs4 import BeautifulSoup
j= ["Population and Housing Unit Estimates"] # Title
k= ["Web Links"] # Column Headings
example_listing='https://www.census.gov/programs-surveys/popest.html' #Source
r=requests.get(example_listing) #Grab page source html
html_page=r.text
soup=BeautifulSoup(html_page,'html.parser') #Build Beautiful Soup object to help parse the html
with open('HTMLList.csv','w',newline="") as f: #Choose what you want to grab
    writer=csv.writer(f,delimiter=' ',lineterminator='\r')
    writer.writerow(j)
    writer.writerow(k)
    for link in soup.find_all('a'):
        if link.get('href') is not None:
            writer.writerow([link.get('href')])

HTMLList.csv

"Population and Housing Unit Estimates"
"Web Links"
https://www.census.gov/en.html
https://www.census.gov/topics/population/age-and-sex.html
https://www.census.gov/topics/business-economy.html
https://www.census.gov/topics/education.html
https://www.census.gov/topics/preparedness.html
https://www.census.gov/topics/employment.html
https://www.census.gov/topics/families.html
https://www.census.gov/topics/population/migration.html
https://www.census.gov/geography.html
https://www.census.gov/topics/health.html
https://www.census.gov/topics/population/hispanic-origin.html
https://www.census.gov/topics/housing.html
https://www.census.gov/topics/income-poverty.html
https://www.census.gov/topics/international-trade.html
https://www.census.gov/topics/population.html
.......

【讨论】：

完美！我刚刚意识到有一堆重复的 URL。有没有办法确保没有重复的链接？