Beautifulsoup \':-soup-contains\' 伪类此时没有实现答案

【问题标题】：Beautifulsoup ':-soup-contains' pseudo-class is not implemented at this timeBeautifulsoup \':-soup-contains\' 伪类此时没有实现
【发布时间】：2022-11-29 02:14:15
【问题描述】：

每当我运行这段代码时，我都会收到伪类未实现的错误。我在网上找到了这段代码，我正试图从维基百科中抓取有关城市的相关信息。我已将 python 和 beautiful soup 更新到最新版本。任何帮助是极大的赞赏。

import requests
import bs4
from bs4 import BeautifulSoup as bs
import pandas as pd
import unicodedata
import re

# cities = ['Berlin', 'Hamburg', 'Frankfurt','Munich','Stuttgart','Leipzig','Cologne','Dresden','Hannover','Paris', 'Barcelona','Lisbon','Madrid']
cities = ['Berlin','Paris','Amsterdam','Barcelona','Rome','Lisbon','Prague','Vienna','Madrid']


def City_info(soup):
    
    ret_dict = {}
    ret_dict['city'] = soup.h1.get_text()
    
    
    if soup.select_one('.mergedrow:-soup-contains("Mayor")>.infobox-label') != None:
        i = soup.select_one('.mergedrow:-soup-contains("Mayor")>.infobox-label')
        mayor_name_html = i.find_next_sibling()
        mayor_name = unicodedata.normalize('NFKD',mayor_name_html.get_text())
        ret_dict['mayor']  = mayor_name
    
    if soup.select_one('.mergedrow:-soup-contains("City")>.infobox-label') != None:
        j =  soup.select_one('.mergedrow:-soup-contains("City")>.infobox-label')
        area = j.find_next_sibling('td').get_text()
        ret_dict['city_size'] = unicodedata.normalize('NFKD',area)

    if soup.select_one('.mergedtoprow:-soup-contains("Elevation")>.infobox-data') != None:
        k = soup.select_one('.mergedtoprow:-soup-contains("Elevation")>.infobox-data')
        elevation_html = k.get_text()
        ret_dict['elevation'] = unicodedata.normalize('NFKD',elevation_html)
    
    if soup.select_one('.mergedtoprow:-soup-contains("Population")') != None:
        l = soup.select_one('.mergedtoprow:-soup-contains("Population")')
        c_pop = l.findNext('td').get_text()
        ret_dict['city_population'] = c_pop
    
    if soup.select_one('.infobox-label>[title^=Urban]') != None:
        m = soup.select_one('.infobox-label>[title^=Urban]')
        u_pop = m.findNext('td')
        ret_dict['urban_population'] = u_pop.get_text()

    if soup.select_one('.infobox-label>[title^=Metro]') != None:
        n = soup.select_one('.infobox-label>[title^=Metro]')
        m_pop = n.findNext('td')
        ret_dict['metro_population'] = m_pop.get_text()
    
    if soup.select_one('.latitude') != None:
        o = soup.select_one('.latitude')
        ret_dict['lat'] = o.get_text()

    if soup.select_one('.longitude') != None:    
        p = soup.select_one('.longitude')
        ret_dict['long'] = p.get_text()
    
    return ret_dict



list_of_city_info = []
for city in cities:
    url = 'https://en.wikipedia.org/wiki/{}'.format(city)
    web = requests.get(url,'html.parser')
    soup = bs(web.content)
    list_of_city_info.append(City_info(soup))
df_cities = pd.DataFrame(list_of_city_info)
df_cities = df_cities.set_index('city')
df_cities

不幸的是，我还没有找到任何解决方案。

【问题讨论】：

标签： web-scraping beautifulsoup

【解决方案1】：

:-soup-contains 是一个 css 伪类选择器，用于定位节点的文本。

它带有 Soup Sieve，这是 Beautiful Soup 4.7.0+ 的官方 CSS select 实现，因此对于大多数人来说，使用 Beautiful Soup 4.7.0+ 你的脚本应该可以正常工作。

因此，首先检查您的版本是否是最新的，旧版本不推荐使用 :contains() 形式。

【讨论】：