将汤内容转换为结构化 csv

Question

我正在尝试抓取一个网站并将其放入结构化数据格式。 我想最终得到一个包含 6 列的 .csv：国家、日期、通用文本、财政文本、货币文本、外汇文本。

映射是这样的：

country <- h3
date <- h6
general_text <- p (h3) (the p tag that follows the h3 header)
fiscal_text  <- p (1st h5 ul li) (the p tag that follows the **first** h5. This tag is inside ul and li blocks)
monetary_text <- p (2nd h5 ul li) (the p tag that follows the **second** h5. This tag is inside ul and li blocks)
fx_text <- p (3rd h5 ul li) (the p tag that follows the **third** h5. This tag is inside ul and li blocks)

模式在下一个 h3（国家）标题处结束。

我发现很难将每个元素放在适当的位置/列中。

每个国家/地区的网站结构都重复这个（实际标签见下文）：

h3
 p
h6
h5
 ul
  li
   p
h5
 ul
  li
   p
h5
 ul
  li
   p

我有以下代码用于简单的文本提取：

import requests
import io
import csv 
from bs4 import BeautifulSoup
from urllib.request import urlopen
URL = 'https://www.imf.org/en/Topics/imf-and-covid19/Policy-Responses-to-COVID-19'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')

results = soup.find(class_='rr-intro')

with io.open('test.txt', 'w', encoding='utf8') as f:
    for header in results.find_all(['h3', 'h6', 'h5']):
        f.write(header.get_text() + u'\n') 
        for elem in header.next_siblings:
            if elem.name and elem.name.startswith('h'):
                # stop at next header
                break
            if elem.name and elem.find_all('p'):
                f.write(elem.get_text() + u'\n')

从 cmets 中，我认为创建列表并以某种方式压缩它们是有意义的。我试过这个：

h3 = results.find_all('h3')
h6 = results.find_all('h6')
h5 = results.find_all('h5')
h5f = results.find_all('h5', text='Fiscal')
h5m = results.find_all('h5', text='Monetary and macro-financial')
h5x = results.find_all('h5', text='Exchange rate and balance of payments')
country = [country.get_text() for country in h3]  #list of countries
date = [date.get_text() for date in h6]  #date string

我被困在这里了。不知道如何将 p-tags 的内容放到列表中的正确位置，以便将其压缩或直接放到 csv 中。

我是 python 菜鸟，所以我根据我在 stackoverflow 上找到的内容制作了这些。任何帮助将不胜感激。

编辑：澄清一下，我想要的结构看起来像这样。

<div class="rr-intro">

 <h3>
  Country 1
 </h3>
 <p>
  summary text
 </p>
 <h6>
  date
 </h6>
 <h5>
  Fiscal
 </h5>
 <ul>
  <li>
   <p>
    text for fiscal of country 1
   </p>
  </li>
 </ul>
 <h5>
  Monetary and macro-financial
 </h5>
 <ul>
  <li>
   <p>
    text for monetary of country 1
   </p>
  </li>
 </ul>
 <h5>
  Exchange rate and balance of payments
 </h5>
 <ul>
  <li>
   <p>
    text for FX of country 1
   </p>
  </li>
 </ul>
  <h3>
  Country 2
 </h3>
 <p>
  summary text
 </p>
 <h6>
  date
 </h6>
 <h5>
  Fiscal
 </h5>
 <ul>
  <li>
   <p>
    text for fiscal of country 2
   </p>
  </li>
 </ul>
 <h5>
  Monetary and macro-financial
 </h5>
 <ul>
  <li>
   <p>
    text for monetary of country 2
   </p>
  </li>
 </ul>
 <h5>
  Exchange rate and balance of payments
 </h5>
 <ul>
  <li>
   <p>
    text for FX of country 2
   </p>
  </li>
 </ul>
   <h3>
  Country 3
 </h3> 

等等……

Answer 1

我觉得最简单的方法是在您按顺序阅读元素时处理它们。为此，您可以跟踪当前部分，然后将文本附加到该部分。

一旦找到下一个h3 国家/地区，就可以使用 Python CSV DictWriter 写入一行信息。例如：

from collections import defaultdict
import requests
import csv 
from bs4 import BeautifulSoup

URL = 'https://www.imf.org/en/Topics/imf-and-covid19/Policy-Responses-to-COVID-19'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')

results = soup.find('div', class_='rr-intro')

section_lookup = {
    'Fiscal' : 'fiscal_text',
    'Moneta' : 'monetary_text',
    'Macro-' : 'monetary_text',
    'Exchan' : 'fx_text',
}

with open('data.csv', 'w', encoding='utf8', newline='') as f_output:
    fieldnames = ['country', 'date', 'general_text', 'fiscal_text', 'monetary_text', 'fx_text']
    csv_output = csv.DictWriter(f_output, fieldnames=fieldnames)
    csv_output.writeheader()

    row = defaultdict(str)
    section = None

    for elem in results.find_all(['h3', 'h6', 'h5', 'p']):
        if elem.name == 'h3':
            if row:
                csv_output.writerow(row)
                row = defaultdict(str)

            row['country'] = elem.get_text(strip=True)
            section = "general_text"

        elif elem.name == 'h5':
            section = section_lookup[elem.get_text(strip=True)[:6]]
        elif elem.name == 'h6':
            row['date'] = elem.get_text(strip=True)[27:]
        elif elem.name == 'p' and section:
            row[section] = f"{row[section]} {elem.get_text(strip=True)}"

    if row:
        csv_output.writerow(row)

给你一个data.csv文件开始：