网络爬虫 - 如何删除 div 中的子节点？答案

【问题标题】：web crawler - How to remove child node in div?网络爬虫 - 如何删除 div 中的子节点？
【发布时间】：2019-07-13 18:39:18
【问题描述】：

我在编写网络爬虫时遇到问题。

这里是 HTML：

i don't want this
<div id="main-content">
  <div id="1">i don't want this</div>
  <div id="2">i don't want this</div>
  <div id="3">i don't want this</div>
  i want this!!!
  <span class="c1">i don't want this</span>
  <span class="c1">i don't want this</span>
</div>
i don't want this

我还写了一些python代码：

import requests, json
from bs4 import BeautifulSoup
import re

res = requests.get(url)
soup =  BeautifulSoup(res.text,"html.parser")
main_content = soup.find(id="main-content")

#### problem here ####
m = [s.extract() for s in main_content('div')]
m = [s.extract() for s in main_content('span')]

# some regex for dealing string.
filtered = [ v for v in main_content.stripped_strings if v[0] not in [u'※',u'◆'] and v[:2] not in [u'--']]
content = ' '.join(filtered)
content = content.replace("-- "+url,"")
content = re.sub("[,.!?:，。！？：]"," ",content)
content = re.sub(r'(\s)+', ' ', content)
print(content)

它有时有效，但有时会发生错误。

【问题讨论】：

欢迎加入 Stack Overflow 的朋友！您能否仅说明失败的情况
所以你想匹配第一个 div 和这个 id <div id="main-content"> 直到它关闭 </div> ？
什么错误？始终将完整的错误消息（从单词“Traceback”开始）放在问题中（作为文本，而不是屏幕截图）。还有其他有用的信息。
如果你有错误，那么也许你应该使用try/except 和一些if/else 来测试值

标签： python regex beautifulsoup web-crawler

【解决方案1】：

也许这种替代方法会起作用：

from bs4 import BeautifulSoup as bs 

html = """i don't want this
<div id="main-content">
  <div id="1">i don't want this</div>
  <div id="2">i don't want this</div>
  <div id="3">i don't want this</div>
  i want this!!!
  <span class="c1">i don't want this</span>
  <span class="c1">i don't want this</span>
</div>
i don't want this"""

def get_text_without_children(tag):
    return ''.join(tag.find_all(text=True, recursive=False)).strip()

soup = bs(html, 'html.parser')
divs = soup.find_all('div', {"id" : "main-content"})
for i in divs:
    print(get_text_without_children(i))

结果：

i want this!!!

【讨论】：