如何从具有不同 ID 名称的段落中抓取文本？答案

【问题标题】：How to scrape text from paragraphs with different id name?如何从具有不同 ID 名称的段落中抓取文本？
【发布时间】：2018-06-30 16:46:33
【问题描述】：

我正在尝试从具有不同 ID 名称的段落中抓取文本。文字如下：

<p id="comFull1" class="comment" style="display:none"><strong>Comment:
</strong><br>I realized how much Abilify has been helping me when I recently 
tried to taper off of it. I am on the bipolar spectrum, with mainly 
depression and some OCD symptoms. My obsessive, intrusive thoughts came 
racing back when I decreased the medication. I also got much more tired and 
had insomnia with the decrease. am not happy with side effects of 15 lb 
weight gain, increased cholesterol and a flat effect on my emotions. I am 
actually wondering if an increase from the 7 mg would help even more...for 
now I&#39;m living with the side effects.<br><a 
onclick="toggle('comTrunc1'); toggle('comFull1');return false;" 
href="#">Hide Full Comment</a></p>

<p id="comFull2" class="comment" style="display:none"><strong>Comment:
</strong><br>It&#39;s worked Very well for me. I&#39;m sleeping I&#39;m 
eating I&#39;m going Out in the public. Overall I&#39;m very 
satisfied.However I haven&#39;t heard anybody mention this but my feet are 
very puffy and swollen is this a side effect does anyone know?<br><a 
onclick="toggle('comTrunc2'); toggle('comFull2');return false;" 
href="#">Hide Full Comment</a></p>

......

我只能从特定 id 中删除文本，但不能一次删除所有 id。任何人都可以在这个问题上帮助从所有 id 中删除文本。代码是这样的

>>> from urllib.request import Request, urlopen
>>> from bs4 import BeautifulSoup
>>> url = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
>>> req = Request(url,headers={'User-Agent': 'Mozilla/5.0'})
>>> webpage = urlopen(req).read()
>>> soup = BeautifulSoup(webpage, "html.parser")
>>> required2 = soup.find("p", {"id": "comFull1"}).text
>>> required2
"Comment:I realized how much Abilify has been helping me when I recently 
tried to taper off of it. I am on the bipolar spectrum, with mainly 
depression and some OCD symptoms. My obsessive, intrusive thoughts came 
racing back when I decreased the medication. I also got much more tired and 
had insomnia with the decrease. am not happy with side effects of 15 lb 
weight gain, increased cholesterol and a flat effect on my emotions. I am 
actually wondering if an increase from the 7 mg would help even more...for 
now I'm living with the side effects.Hide Full Comment"

【问题讨论】：

只是不要提及id：soup.findAll("p")。

标签： python web-scraping beautifulsoup scrapy

【解决方案1】：

试试这个。如果所有包含段落的 ID 号都以1,2,3 e.t.c 为后缀，如comFull1,comFull2,comFull3，那么下面的选择器应该处理它。

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup

soup = BeautifulSoup(content, "html.parser")
for item in soup.select("[id^='comFull']"):
    print(item.text)

【讨论】：

【解决方案2】：

据我了解，您遇到的问题是抓取网页中所有段落的文本或标签。

你要找的功能是-

soup.find_all('p')

更全面的示例显示在以下文档中 -

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

【讨论】：

【解决方案3】：

如果你想使用xpath，你可以使用

response.xpath("//p[contains(@id,'comFull')]/text()").extract()

但是由于您使用的是漂亮的汤，您可以将函数或正则表达式传递给find_all 方法，如此处所述。 Matching id's in BeautifulSoup

soup.find_all('p', id=re.compile('^comFull-'))

【讨论】：