【问题标题】：How to fetch paragraphs from html using Python如何使用 Python 从 html 中获取段落
【发布时间】：2016-04-22 22:02:41
【问题描述】：

如何从结构不好的 html 中获取段落？

我有这个原始的 html 文本：

This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
<br>
<ul>
    <li>AA Early Childhood Education, or related field.  </li>
    <li>2+ years experience in a licensed childcare facility  </li>
    <li>Ability to meet state requirements, including finger print clearance.  </li>
    <li>Excellent oral and written communication skills  </li>
    <li>Strong organization and time management skills.  </li>
    <li>Creativity in expanding children's learning through play.<br>  </li>
    <li>Strong classroom management skills.<br>  </li>
</ul>
<p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children. 
    <br> 
</p>

我使用 Python 并尝试做类似的事情：

soup = BeautifulSoup(html)

它返回一个包含 2 个 short 段落的新 html 文本：

<html>

<body>
    <p>This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
        <br/>
    </p>
    <ul>
        <li>AA Early Childhood Education, or related field. </li>
        <li>2+ years experience in a licensed childcare facility </li>
        <li>Ability to meet state requirements, including finger print clearance. </li>
        <li>Excellent oral and written communication skills </li>
        <li>Strong organization and time management skills. </li>
        <li>Creativity in expanding children's learning through play.
            <br/> </li>
        <li>Strong classroom management skills.
            <br/> </li>
    </ul>
    <p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
        <br/> </p>
</body>

</html>

但这不是我所期望的。结果，我想得到这个html文本：

<html>

<body>
    <p>This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
        AA Early Childhood Education, or related field.
        2+ years experience in a licensed childcare facility
        Ability to meet state requirements, including finger print clearance.
        Excellent oral and written communication skills
        Strong organization and time management skills.
        Creativity in expanding children's learning through play.
        Strong classroom management skills.
    </p>
    <p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.</p>
</body>

</html>

为了获得以上html，我认为最好的方法是从原始html中删除除<p>和</p>之外的所有html标签。

为此，我尝试了以下正则表达式：

new_html = re.sub('<[^<]+?>', '', html)

显然，正则表达式会删除所有 html 标签。那么，如何去除除<p>和</p>之外的所有html标签？

如果有人帮我写 r.e.然后我将new_html 提供给BeautifulSoup() 并获得我期望的html。

【问题讨论】：

您要检索文本吗？如果是，那么soup.get_text() 应该没问题。
不，我想检索段落列表。
那些li标签呢？你想用文本替换它们吗？
是的，并将它们添加到第一段中。

标签： python html regex beautifulsoup

【解决方案1】：

简答

new_html = re.sub('<([^p]|[^>/][^>]+|/[^p]|/[^>][^>]+)>', '', html)

长答案

你原来的正则表达式看起来很奇怪。我会使用[^>] 而不是[^<]。你想要“任何不是结束标签的东西”。

另外，在+ 后面加上? 很奇怪。

+ 表示：“重复 1 次或更多次”

? 表示：“重复 0 或一次”。

同时拥有这两个标志很奇怪。

无论如何，我们可以这样表达你的正则表达式：

“打开标签”，然后是“任何不是'p'和不是/p的东西”，然后是“关闭标签”

相当于：

“打开标签”，然后是“一个不是'p'的唯一字符”或“任何不是斜线然后是一个或多个字符”或“一个斜线然后是一个不是'p'的唯一字符”或“一个斜线，然后是两个或多个字符”，然后是“关闭标签”。

相当于：

< 然后（[^p] 或[^>/][^>]+ 或/[^p] 或/[^>][^>]+）然后>

这就是上面的正则表达式所表达的。

这是一个在 python 控制台中输入的快速测试：

re.sub(
    '<([^p]|[^>/][^>]+|/[^p]|/[^>][^>]+)>', 
    '', 
    'aa <p> bb <a> cc <li> dd <pp> ee <pa> ff </p> gg </a> hh </li> ii </pp> jj </pa> ff')

【讨论】：

+? 表示一个或多个，并且不贪婪。没有那个?，结束标签也会被捕获。你说得对，[^>] 更好。
为什么要使用正则表达式而不是 HTML Parser？

【解决方案2】：

这是一种手动文档操作，但是，您可以在appending 之后将li 元素和remove 循环到第一段。然后，同时删除 ul 元素：

from bs4 import BeautifulSoup


data = """
This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
<br>
<ul>
    <li>AA Early Childhood Education, or related field.  </li>
    <li>2+ years experience in a licensed childcare facility  </li>
    <li>Ability to meet state requirements, including finger print clearance.  </li>
    <li>Excellent oral and written communication skills  </li>
    <li>Strong organization and time management skills.  </li>
    <li>Creativity in expanding children's learning through play.<br>  </li>
    <li>Strong classroom management skills.<br>  </li>
</ul>
<p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
    <br>
</p>"""

soup = BeautifulSoup(data, "lxml")

p = soup.p
for li in soup.find_all("li"):
    p.append(li.get_text())
    li.extract()

soup.find("ul").extract()
print(soup.prettify())

按照您的计划打印 2 个段落：

<html>
 <body>
  <p>
   This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
   <br/>
   AA Early Childhood Education, or related field.
   2+ years experience in a licensed childcare facility
   Ability to meet state requirements, including finger print clearance.
   Excellent oral and written communication skills
   Strong organization and time management skills.
   Creativity in expanding children's learning through play.
   Strong classroom management skills.
  </p>
  <p>
   The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
   <br/>
  </p>
 </body>
</html>

请注意，lxml、html.parser 和 html5lib 解析您发布的输入 HTML 的方式有一个重要区别。 html5lib 和 html.parser 不会自动创建第一段，使上面的代码真正成为 lxml 特定的。

更好的方法可能是制作一个单独的“汤”对象。示例：

from bs4 import BeautifulSoup


data = """
This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
<br>
<ul>
    <li>AA Early Childhood Education, or related field.  </li>
    <li>2+ years experience in a licensed childcare facility  </li>
    <li>Ability to meet state requirements, including finger print clearance.  </li>
    <li>Excellent oral and written communication skills  </li>
    <li>Strong organization and time management skills.  </li>
    <li>Creativity in expanding children's learning through play.<br>  </li>
    <li>Strong classroom management skills.<br>  </li>
</ul>
<p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
    <br>
</p>"""

soup = BeautifulSoup(data, "lxml")

# create new soup
new_soup = BeautifulSoup("<body></body>", "lxml")
new_body = new_soup.body

# create first paragraph
first_p = new_soup.new_tag("p")
first_p.append(soup.p.get_text())

for li in soup.find_all("li"):
    first_p.append(li.get_text())

new_body.append(first_p)

# create second paragraph
second_p = soup.find_all("p")[-1]
new_body.append(second_p)

print(new_soup.prettify())

打印：

<html>
 <body>
  <p>
   This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
   AA Early Childhood Education, or related field.
   2+ years experience in a licensed childcare facility
   Ability to meet state requirements, including finger print clearance.
   Excellent oral and written communication skills
   Strong organization and time management skills.
   Creativity in expanding children's learning through play.
   Strong classroom management skills.
  </p>
  <p>
   The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
   <br/>
  </p>
 </body>
</html>

【讨论】：

遇到其他html标签怎么办？像、等等。我只想提取

标签之间的文本并将外部文本添加为其他段落。
@user3601768 是的，可能制作一个单独的汤对象会更好，用示例更新了答案。希望对您有所帮助。