BeautifulSoup 给出一个空列表答案

【问题标题】：BeautifulSoup gives an empty listBeautifulSoup 给出一个空列表
【发布时间】：2021-09-03 07:08:39
【问题描述】：

我正在尝试用 Beautiful Soup 抓取网站。打印 container 后，它给了我一个空列表。我该如何解决这个问题？

import requests
from bs4 import BeautifulSoup
import lxml

URL = 'https://www.monster.com/jobs/search/?q=Software-Developer&where=Australia'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'lxml')
container = soup.find_all('div', class_="results-list")
print(container)

【问题讨论】：

您收到的页面数据不包含包含'results-list'的div：container = soup.find_all('div')'div' in str(container) # -> true (sanity check)'results-list' in str(container) # -> false'class="results-list' in str(soup) # -> false但是，它在您收到的结果中，但只有作为一种风格：'results-list' in str(soup) # -> true'.results-list' in str(soup) # -> true

标签： python web-scraping beautifulsoup request web-scripting

【解决方案1】：

数据是通过 Ajax 调用从外部 URL 加载的。您可以使用requests 模块对其进行模拟并加载数据：

import json
import requests
from textwrap import wrap
from bs4 import BeautifulSoup


api_url = "https://services.monster.io/jobs-svx-service/v2/monster/search-jobs/samsearch/en-us"

payload = {
    "fingerprintId": "",
    "jobAdsRequest": {
        "placement": {"appName": "monster", "component": "JSR_SPLIT_VIEW"},
        "position": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    },
    "jobQuery": {
        "companyDisplayNames": [],
        "excludeJobs": [],
        "locations": [{"address": "Australia", "country": "us"}],
        "query": "Software-Developer",
    },
    "offset": 0,
    "pageSize": 20,
    "searchId": "",
}


data = requests.post(api_url, json=payload).json()

# uncomment to print all data:
# print(json.dumps(data, indent=4))

for r in data["jobResults"]:
    s = BeautifulSoup(r["jobPosting"]["description"], "html.parser")
    print("\n".join(wrap(s.get_text(strip=True, separator=" "))))
    print("-" * 80)

打印：

Description: The role of the Mobile Developer is to analyze business
requirements, develop a design plan, translate plan into program
specifications (or a manual process), code, test, and coordinate
implementation. They will be focusing on enhancements within existing
applications (not building out anything new). Working with multiple
applications- all very client specific. This developer could also be
tasked with some frontend work for a specific application. Need to be
able to utilize Angular skills for frontend development. -Must be
highly proficient with mobile platform Application Programming
Interfaces (API) such as Apple iOS and Android Mobile -Hands on
experience with latest iOS and Android tech -Deep knowledge of Angular


...and so on

【讨论】：