使用 beautifulsoup 提取未标记的文本答案

【问题标题】：Extracting untagged text using beautifulsoup使用 beautifulsoup 提取未标记的文本
【发布时间】：2016-01-29 16:45:21
【问题描述】：

我有一个 XML 文件，我希望提取未标记的文本。

<body>
        <p>The prognosis of patients with rectal cancer has improved since the introduction of total mesorectal excision (TME) surgery [
            <xref ref-type="bibr" rid="CR1">1</xref>&#x02013;
            <xref ref-type="bibr" rid="CR3">3</xref>]. Using this surgical technique the mesorectal compartment including the rectum and perirectal fat is completely excised by sharp dissection along the mesorectal fascia (MRF) [
            <xref ref-type="bibr" rid="CR1">1</xref>]. Additionally, large randomized trials have shown that neo-adjuvant therapy improves local tumor control even further, regardless of optimized surgical techniques [
            <xref ref-type="bibr" rid="CR3">3</xref>, 
            <xref ref-type="bibr" rid="CR4">4</xref>]. The advances in rectal cancer treatment have provoked differentiated neo-adjuvant treatment strategies based on anatomical preoperative identifiable risk factors for local tumor recurrence as can be visualized with magnetic resonance imaging (MRI) [
            <xref ref-type="bibr" rid="CR5">5</xref>]. One of the most important risk factors is the tumor relationship to the MRF, which actually defines the surgical circumferential resection margin (CRM) in TME surgery [
            <xref ref-type="bibr" rid="CR6">6</xref>, 
            <xref ref-type="bibr" rid="CR7">7</xref>]. Long courses of neo-adjuvant chemoradiation have emerged as the preferential treatment of patients with anticipated tumor invasion of the MRF on MRI in order to downstage/downsize the tumor and to obtain tumor free resection margins [
            <xref ref-type="bibr" rid="CR5">5</xref>].
            </p>

</body>

所以正文可能包含多个<p> 标签。我希望提取像

这样的文本

"]. 使用这种手术技术，直肠系膜室包括直肠和直肠周围脂肪被锐器完全切除沿直肠系膜筋膜 (MRF) 解剖 ["

，位于CR3 和CR1 之间，依此类推（即在连续的xref 之间）。我还需要将此文本添加到字典中，该字典将相应的rid 映射到rid 之后的此类文本列表。我如何使用 beautifulsoup 和/或 regexp 来做到这一点。

【问题讨论】：

你得到答案了吗？

标签： python regex xml beautifulsoup

【解决方案1】：

下面的代码对我有用——它创建了一个字典（映射）！

from bs4 import BeautifulSoup
from collections import defaultdict
import re

d= defaultdict(unicode)

html ='''
<body>
        <p>The prognosis of patients with rectal cancer has improved since the introduction of total mesorectal excision (TME) surgery [
            <xref ref-type="bibr" rid="CR1">1</xref>&#x02013;
            <xref ref-type="bibr" rid="CR3">3</xref>]. Using this surgical technique the mesorectal compartment including the rectum and perirectal fat is completely excised by sharp dissection along the mesorectal fascia (MRF) [
            <xref ref-type="bibr" rid="CR1">1</xref>]. Additionally, large randomized trials have shown that neo-adjuvant therapy improves local tumor control even further, regardless of optimized surgical techniques [
            <xref ref-type="bibr" rid="CR3">3</xref>, 
            <xref ref-type="bibr" rid="CR4">4</xref>]. The advances in rectal cancer treatment have provoked differentiated neo-adjuvant treatment strategies based on anatomical preoperative identifiable risk factors for local tumor recurrence as can be visualized with magnetic resonance imaging (MRI) [
            <xref ref-type="bibr" rid="CR5">5</xref>]. One of the most important risk factors is the tumor relationship to the MRF, which actually defines the surgical circumferential resection margin (CRM) in TME surgery [
            <xref ref-type="bibr" rid="CR6">6</xref>, 
            <xref ref-type="bibr" rid="CR7">7</xref>]. Long courses of neo-adjuvant chemoradiation have emerged as the preferential treatment of patients with anticipated tumor invasion of the MRF on MRI in order to downstage/downsize the tumor and to obtain tumor free resection margins [
            <xref ref-type="bibr" rid="CR5">5</xref>].
            </p>

</body>

'''

soup = BeautifulSoup(html,'html.parser')
l = soup.find_all('xref')
for i in l:
    e= i.next_element
    txt =  e.next_element.encode('utf-8')
    if re.match(r'\].+\[',txt) is not None:
        d[i.attrs['rid'].strip()]=txt.strip()
for k,v in d.items():
    print "The value of {0} is>>>>> {1} ".format(k,v)

打印出来-

The value of CR3 is>>>>> ]. Using this surgical technique the mesorectal compartment including the rectum and perirectal fat is completely excised by sharp dissection along the mesorectal fascia (MRF) [ 
The value of CR1 is>>>>> ]. Additionally, large randomized trials have shown that neo-adjuvant therapy improves local tumor control even further, regardless of optimized surgical techniques [ 
The value of CR7 is>>>>> ]. Long courses of neo-adjuvant chemoradiation have emerged as the preferential treatment of patients with anticipated tumor invasion of the MRF on MRI in order to downstage/downsize the tumor and to obtain tumor free resection margins [ 
The value of CR4 is>>>>> ]. The advances in rectal cancer treatment have provoked differentiated neo-adjuvant treatment strategies based on anatomical preoperative identifiable risk factors for local tumor recurrence as can be visualized with magnetic resonance imaging (MRI) [ 
The value of CR5 is>>>>> ]. One of the most important risk factors is the tumor relationship to the MRF, which actually defines the surgical circumferential resection margin (CRM) in TME surgery [

【讨论】：

【解决方案2】：

这个呢？

html = """
<body>
        <p>The prognosis of patients with rectal cancer has improved since the introduction of total mesorectal excision (TME) surgery [
            <xref ref-type="bibr" rid="CR1">1</xref>&#x02013;
            <xref ref-type="bibr" rid="CR3">3</xref>]. Using this surgical technique the mesorectal compartment including the rectum and perirectal fat is completely excised by sharp dissection along the mesorectal fascia (MRF) [
            <xref ref-type="bibr" rid="CR1">1</xref>]. Additionally, large randomized trials have shown that neo-adjuvant therapy improves local tumor control even further, regardless of optimized surgical techniques [
            <xref ref-type="bibr" rid="CR3">3</xref>, 
            <xref ref-type="bibr" rid="CR4">4</xref>]. The advances in rectal cancer treatment have provoked differentiated neo-adjuvant treatment strategies based on anatomical preoperative identifiable risk factors for local tumor recurrence as can be visualized with magnetic resonance imaging (MRI) [
            <xref ref-type="bibr" rid="CR5">5</xref>]. One of the most important risk factors is the tumor relationship to the MRF, which actually defines the surgical circumferential resection margin (CRM) in TME surgery [
            <xref ref-type="bibr" rid="CR6">6</xref>, 
            <xref ref-type="bibr" rid="CR7">7</xref>]. Long courses of neo-adjuvant chemoradiation have emerged as the preferential treatment of patients with anticipated tumor invasion of the MRF on MRI in order to downstage/downsize the tumor and to obtain tumor free resection margins [
            <xref ref-type="bibr" rid="CR5">5</xref>].
            </p>

</body>
"""

import re
re.search('<xref ref-type="bibr" rid="CR3">3</xref>(.*)', a).group(1)

输出是：

']. Using this surgical technique the mesorectal compartment including the rectum and perirectal fat is completely excised by sharp dissection along the mesorectal fascia (MRF) ['

【讨论】：

【解决方案3】：

检查一下（假设您所有的 rid 值都以 CR 开头）：

>>> from bs4 import BeautifulSoup as bs
>>> soup = bs(xml) # xml is your xml string text
>>> xml_dict = {'CR' + x.next_element:x.next_sibling.strip() for x in soup.findAll('xref')}
>>> print(xml_dict)

{u'CR3': u',', 
 u'CR1': u']. Additionally, large randomized trials have shown that neo-adjuvant therapy improves local tumor control even further, regardless of optimized surgical techniques [', 
 u'CR6': u',', 
 u'CR7': u']. Long courses of neo-adjuvant chemoradiation have emerged as the preferential treatment of patients with anticipated tumor invasion of the MRF on MRI in order to downstage/downsize the tumor and to obtain tumor free resection margins [', 
 u'CR4': u']. The advances in rectal cancer treatment have provoked differentiated neo-adjuvant treatment strategies based on anatomical preoperative identifiable risk factors for local tumor recurrence as can be visualized with magnetic resonance imaging (MRI) [', 
 u'CR5': u'].'}

【讨论】：

如果有多个引用要摆脱怎么办？我想要一个到字符串列表的映射。