使用 BeautifulSoup 从标题标签下的 &quote 中提取数据？

Question

我想在通过 python 中的BeautifulSoup 库获取链接的 HTML 后提取链接的标题。 基本上，整个标题标签是

 <title>Imaan Z Hazir on Twitter: &quot;Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)&quot;</title>

我想提取只有这个Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)的&quot标签中的数据 我试过了

import urllib
import urllib.request

from bs4 import BeautifulSoup

link = "https://twitter.com/ImaanZHazir/status/778560899061780481"
try:
    List=list()
    r = urllib.request.Request(link, headers={'User-Agent': 'Chrome/51.0.2704.103'})
    h = urllib.request.urlopen(r).read()
    data = BeautifulSoup(h,"html.parser")
    for i in data.find_all("title"):
        List.append(i.text)
        print(List[0])
except urllib.error.HTTPError as err:
    pass

我也试过了

for i in data.find_all("title.&quot"):

for i in data.find_all("title>&quot"):

for i in data.find_all("&quot"):

和

for i in data.find_all("quot"):

但是没有人在工作。

Answer 1

只需拆分冒号上的文字：

In [1]:  h = """<title>Imaan Z Hazir on Twitter: &quot;Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)&quot;</title>"""

In [2]: from bs4 import BeautifulSoup

In [3]: soup  = BeautifulSoup(h, "lxml")

In [4]: print(soup.title.text.split(": ", 1)[1])
 "Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)"

其实看你根本不需要拆分的页面，文字在div.js-tweet-text-container里面的p标签里, 日：

In [8]: import requests

In [9]: from bs4 import BeautifulSoup


In [10]: soup  = BeautifulSoup(requests.get("https://twitter.com/ImaanZHazir/status/778560899061780481").content, "lxml")


In [11]: print(soup.select_one("div.js-tweet-text-container p").text)
Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)

In [12]: print(soup.title.text.split(": ", 1)[1])
"Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)"

所以你可以用任何一种方式来获得相同的结果。

Answer 2

一旦你解析了 html：

data = BeautifulSoup(h,"html.parser")

这样查找标题：

title = data.find("title").string  # this is without <title> tag

现在在字符串中找到两个引号 (")。有很多方法可以做到这一点。我会使用正则表达式：

import re
match = re.search(r'".*"', title)
if match:
    print match.group(0)

您永远不会搜索 " 或任何其他 &NAME; 序列，因为 BeautifulSoup 会将它们转换为它们所代表的实际字符。

编辑：

不捕获引号的正则表达式是：

re.search(r'(?<=").*(?=")', title)

Answer 3

这是一个使用正则表达式提取引号内文本的简单完整示例：

import urllib
import re
from bs4 import BeautifulSoup

link = "https://twitter.com/ImaanZHazir/status/778560899061780481"

r = urllib.request.urlopen(link)
soup = BeautifulSoup(r, "html.parser")
title = soup.title.string
quote = re.match(r'^.*\"(.*)\"', title)
print(quote.group(1))

这里发生的情况是，在获取页面的来源并找到 title 后，我们使用正则表达式对标题来提取引号内的文本。

我们告诉正则表达式在开始引号 (\") 之前的字符串开头 (^.*) 中查找任意数量的符号，然后捕获它和结束引号之间的文本（第二个 @ 987654326@).

然后我们通过告诉 Python 打印第一个捕获的组（正则表达式中括号之间的部分）来打印捕获的文本。

这里有更多关于在 python 中匹配正则表达式 - https://docs.python.org/3/library/re.html#match-objects