网页抓取新文章

Question

在过去的几个月里，我一直在学习 python 和 BeautifulSoup 函数，主要是为了我自己的研究目的而尝试使用网络抓取新闻文章。

但是，我在尝试将中文网站上的内容打印成文本时遇到了困难。

我应该使用哪个标签来获取文章的内容？

<<div class="w980 wbnav clear"><a 
href="http://english.peopledaily.com.cn/" 
target="_blank">English</a>&gt;&gt;</div>
<div class="w980 wb_10 clear">
<h1>DPRK launches ballistic missile 'capable of hitting US 
mainland'</h1>
<div> (<a 



</div>
<div class="wb_12 clear">
<p style="text-align: center;">
<img alt="" src="/NMediaFile/2017/1129/FOREIGN201711291331000220555852915.jpg" style="width: 900px; height: 783px;" /></p>
<p>
The Democratic Peopleâs Republic of Korea (DPRK) has confirmed that it successfully tested a âHwasong 15â intercontinental ballistic missile (ICBM) on Wednesday.</p>
<p>
A Korean Central News Agency (KCNA) statement, which confirms earlier assessments from the United States and the Republic of Korea (ROK), claims the new type of ICBM "is capable of striking the whole mainland of the US."
</p>
<p>
It was Pyongyang's first test launch since a missile was fired in mid-September, days after its sixth-nuclear test.</p>
<p>
The ICBM was launched at 02:48 local time on Wednesday, according to the KCNA statement, and flew to an altitude of 4,475 km and then a distance of 950 km.</p>
<p>
It was launched from Sain Ni in the DPRK and flew for 53 minutes before splashing down into the Sea of Japan, said Pentagon spokesman Robert Manning.</p>

Answer 1

我打开了网站链接 (http://en.people.cn/index.html) 并查看了文章。

如果您只是想从特定文章中刮取数据，例如 http://en.people.cn/n3/2017/1220/c90000-9306707.html

那么你可以使用下面的代码-

import requests
from bs4 import BeautifulSoup
import sys

r=requests.get('http://en.people.cn/n3/2017/1220/c90000-9306707.html')

c=r.content
soup=BeautifulSoup(c,'html.parser')

all=soup.find("div",{"class":"d2p3_left wb_left fl"})

d={}
d["heading"]=all.find("h2").text




d["content"]=all.find_all("p")

p=''
for item in d["content"]:
    p=p+item.text


p.replace("\t","")
d["content"]=p
f=open('article1.txt','w')

for item in d.values():
    f.write(item)

f.close()

现在我还检查了其他文章，它们似乎都在使用 d2p3_left wb_left fl 类来分类包含实际文章内容的 html div 标签。

所以我从这个特定的标签中取出内容，并将它们放入带有“标题”和“内容”键的字典中，以便以后可以根据需要对其进行格式化。

然后我将字典的所有值导出到一个文本文件中。

如果你想从主页上抓取所有文章，那么你可以只获取列表中的链接，然后循环遍历列表项作为requests.get() 方法的参数。

希望这会有所帮助。