Beautiful Soup Scraping 返回空括号

Question

html = 'https://en.wikipedia.org/wiki/List_of_largest_banks'

html_data = requests.get('https://en.wikipedia.org/wiki/List_of_largest_banks')

html_data_text = html_data.text

soup = bs(html_data_text, 'html.parser')

table = soup.find_all('table', {id : "By_market_capitalization"})

print(table)

返回空括号。我必须只使用 BS 来完成这项任务。我见过其他图书馆有帮​​助，但我不能使用它们。知道尝试获取这张表有什么问题吗？

Answer 1

您正在寻找具有该 ID 的表。但是，链接显示该 ID 在 跨度。

---

更改第一个选择器以查找跨度，然后使用findNext() 获取表格。

在那里你可以找到tr和td并打印.text：

import json
from urllib.request import urlopen
from bs4 import BeautifulSoup
import requests


html = 'https://en.wikipedia.org/wiki/List_of_largest_banks'

html_data = requests.get(html)

html_data_text = html_data.text

soup = BeautifulSoup(html_data_text, 'html.parser')

span = soup.find('span', id="By_market_capitalization")
table = span.findNext('table')

for row in table.findAll('tr'):
    tds = row.findAll('td')
    if len(tds) > 1:
        print(tds[1].text.strip())

JPMorgan Chase
Industrial and Commercial Bank of China
Bank of America
Wells Fargo
China Construction Bank
Agricultural Bank of China
HSBC Holdings PLC
Citigroup Inc.
... more

Answer 2

该页面上没有带有该 ID 的 table；事实上，我认为 Wikipedia 根本不会为其表格元素使用 id 属性。我相信您正在尝试以“按国家或地区划分的银行”标题下的 this table ?️ 为目标，该标题实际上嵌套在 span 中使用您正在使用的 ID在与表相同的父元素中包含的 h3 元素中。

所以你可以得到它

table = soup.find(
    'span', {'id': 'By_market_capitalization'}
).find_parent('h2').find_next_sibling('table')

但这使用了 3 个 find 调用，因此有 2 个点可能会引发错误 [如果其中一个返回 None]；将 select 与 css selectors 一起使用可能会更好：

table = soup.select_one('h2:has(span#By_market_capitalization) ~ table')

使用pandas.read_html(str(table))[0] 可以非常简单且易于显示，但您也可以使用

colwidth = [10, 50, 30] # set column widths

for r in table.select('tr'):
  cells = [
      (c.text.strip()[:cw], cw) for c, cw 
      in zip(r.select('td,th'), colwidth)
  ]
  bottomBorder = ''
  if r.find('th'):
    bottomBorder = ' | '.join(['+'*c for c in colwidth])
    bottomBorder = f'
| {bottomBorder} |'
  cells = ' | '.join([
      ("{0:^"+str(cw)+"}").format(c) for c, cw in cells
  ])
  print(f'| {cells} |{bottomBorder}')

输出：

|    Rank    |                     Bank name                      |    Market cap(US$ billion)     |
| ++++++++++ | ++++++++++++++++++++++++++++++++++++++++++++++++++ | ++++++++++++++++++++++++++++++ |
|     1      |                   JPMorgan Chase                   |           400.37[6]            |
|     2      |      Industrial and Commercial Bank of China       |             295.65             |
|     3      |                  Bank of America                   |             279.73             |
|     4      |                    Wells Fargo                     |             214.34             |
|     5      |              China Construction Bank               |             207.98             |
|     6      |             Agricultural Bank of China             |             181.49             |
|     7      |                 HSBC Holdings PLC                  |             169.47             |
|     8      |                   Citigroup Inc.                   |             163.58             |
|     9      |                   Bank of China                    |             151.15             |
|     10     |                China Merchants Bank                |             133.37             |
|     11     |                Royal Bank of Canada                |             113.80             |
|     12     |               Toronto-Dominion Bank                |             106.61             |
|     13     |                     HDFC Bank                      |             105.9              |
|     14     |                 Commonwealth Bank                  |             99.77              |
|     15     |                    U.S. Bancorp                    |             84.40              |
|     16     |                   Goldman Sachs                    |             78.70              |
|     17     |                  Banco Santander                   |             75.47              |
|     18     |                   Banco Bradesco                   |             74.67              |
|     19     |                   Morgan Stanley                   |             73.93              |
|     20     |                      Westpac                       |             67.84              |
|     21     |           Mitsubishi UFJ Financial Group           |             66.20              |
|     22     |                     Scotiabank                     |             65.48              |
|     23     |               PNC Financial Services               |             63.11              |
|     24     |               Bank of Communications               |             61.85              |
|     25     |                    BNP Paribas                     |             59.36              |
|     26     |      Australia and New Zealand Banking Group       |             54.88              |
|     27     |              National Australia Bank               |             51.68              |
|     28     |                Lloyds Banking Group                |             51.19              |
|     29     |          Sumitomo Mitsui Financial Group           |             49.85              |
|     30     |                  Bank of Montreal                  |             48.12              |
|     31     |                        UBS                         |             45.92              |
|     32     |                     ING Group                      |             44.97              |
|     33     |                    Capital One                     |             43.22              |
|     34     |            The Bank of New York Mellon             |             42.58              |
|     35     |                China Minsheng Bank                 |             39.13              |
|     36     |                  China CITIC Bank                  |             38.55              |
|     37     |          Banco Bilbao Vizcaya Argentaria           |             37.42              |
|     38     |               Mizuho Financial Group               |             36.95              |
|     39     |                  Intesa Sanpaolo                   |             36.90              |
|     40     |                  Crédit Agricole                   |             34.89              |
|     41     |         Canadian Imperial Bank of Commerce         |             34.87              |
|     42     |               Royal Bank of Scotland               |             33.95              |
|     43     |                      Barclays                      |             33.26              |
|     44     |                   Credit Suisse                    |             30.75              |
|     45     |                       Nordea                       |             29.59              |
|     46     |                 Standard Chartered                 |             29.37              |
|     47     |                      KBC Bank                      |             27.40              |
|     48     |                     UniCredit                      |             26.88              |
|     49     |                  Société Générale                  |             21.27              |
|     50     |                   Deutsche Bank                    |             15.77              |

Answer 3

@大卫，

我试过使用代码，它需要一些更改。

注意：- 您已尝试使用表中不存在的 id 属性

这是修复后的代码：-

import requests
from bs4 import BeautifulSoup

req = requests.get('https://en.wikipedia.org/wiki/List_of_largest_banks')
html = req.text
soup = BeautifulSoup(html, 'html.parser')
table = soup.find_all('table', {id : "By_market_capitalization"})
print(table)