初学者 - 网页抓取 - 下载数据

Question

我要下载这个数据https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/index.html

到目前为止，我能够从标签 p 中获取链接，这些是每个月的链接，但挑战是在每个链接下它们是 31 个文件（每天），我尝试了几种方法从堆栈中获取 h2标题，以及

from bs4 import BeautifulSoup
import urllib.request as urllib2

url = "https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html"
page=urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
headings = soup.findAll('h2');


req = requests.get(url)
soup = BeautifulSoup(req.text, "html.parser")
print("The href links are :")

print (headings)

for link in soup.find_all('a'):
   print(link.get('href')) 
soup = BeautifulSoup(html, 'html.parser')
links_with_text = []
for a in soup.find_all('a', href=True): 
    if a.text: 
        links_with_text.append(a['href'])

links_with_text = [a['href'] for a in soup.find_all('a', href=True) if a.text]
links_with_text

这是他们的输出（仅粘贴最后一个输出）

['https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#December',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#November',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#October',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#September',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#August',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#July',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#June',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#May',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#April',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#March',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#February',
 'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2000/index.html#January'

我的问题是这些是 h2 标签标题，而且我需要来自每个 h2 标签的链接，这些链接存储在每个 a 标签下。 虽然上面的程序给了我所有确实有这些链接的链接，但是如果我能以有组织的方式或任何其他方式获得它们，这样可以更容易地直接从 html 站点存储数据，那就太好了。我将不胜感激。谢谢。你！

Answer 1

1. 如果您看到h2 标记没有数据，则查找所有h2 标记并循环遍历它，因此我们必须找到使用find_next 方法的下一个标记 在p标签上

2. 现在我们必须找到所有a 标签，所以我们将使用find_all 方法，我在一行代码中完成了此操作，它将返回链接列表

3. 现在我们将遍历它并仅提取href 部分，但有一个cath href 不正确它包含20001231ps.html 这样 但我们需要像这样的20041231ps.html 所以这就是我这样做的原因 替换和追加字符串的过程

4. 我使用了dict1，它将键附加为月份，值附加为链接列表，因此很容易提取。

代码：

months=soup.find_all("h2")
dict1={}
main_url="https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004"
for month in months:
    dict1[month.text]=[main_url+"/"+link['href'].replace("2000","2004") for link in month.find_next("p").find_all("a")]

输出：

{'December': ['https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041231ps.html',
  'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041230ps.html',
  'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041229ps.html',
  'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041228ps.html',
  'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041227ps.html',
  'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041226ps.html',
  'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041225ps.html',
  'https://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/2004/20041224ps.html',
.....
]}