在动态 HTML 网站上使用 Beautiful Soup 进行网页抓取的问题

Question

我正在尝试使用 Beautiful Soup 抓取一系列 HTML 文件，但是我得到了一些非常奇怪的结果，我认为这是因为查询是动态的，而且我对网络抓取不是很有经验。如果您查看该网站，在这种情况下，我要做的就是获取工作类型的所有信息，但我的结果与我希望的结果相去甚远。请在下面查看我的代码（感谢所有人）：

 import requests
 from bs4 import BeautifulSoup
 url = 'https://www.acc.co.nz/for-providers/treatment-recovery/work-type-detail-sheets/#/viewSheet/1416'
 r = requests.get(url)
 html_doc = r.text
 soup = BeautifulSoup(html_doc)
 pretty_soup = soup.prettify()
 print(pretty_soup) 

感谢所有帮助。我以为我分享了下面的代码，注意我使用了很多来自其他帖子Strip HTML from strings in Python 的引用。没有@Andrej Kesely 是不可能的

  url = "https://www.acc.co.nz/for-providers/treatment-recovery/work-type-detail-sheets/getSheets"

import requests
import json
from pandas.io.json import json_normalize

headers = {'X-Requested-With': 'XMLHttpRequest'}
r = requests.get(url, headers=headers)
data = json.loads(r.text)
result = json_normalize(data)

result = result[['ANZSCO','Comments','Description','Group',
             'EntryRequirements','JobTitle','PhysicalMentalDemands',
             'WorkEnvironment','WorkTasks']]


 ##Lets start cleaning up the data set

 from html.parser import HTMLParser

 class MLStripper(HTMLParser):
 def __init__(self):
    self.reset()
    self.strict = False
    self.convert_charrefs= True
    self.fed = []
def handle_data(self, d):
    self.fed.append(d)
def get_data(self):
    return ''.join(self.fed)


def strip_tags(html):
   s = MLStripper()
   s.feed(html)
   return s.get_data()


list = ['WorkTasks', 'PhysicalMentalDemands','WorkTasks','Description']

for i in list:
    result[i] = result[i].apply(lambda x: strip_tags(x))

list2 = ['Comments','EntryRequirements','WorkEnvironment']

for i in list2:
    result[i] = result[i].fillna('not_available')
    result[i] = result[i].apply(lambda x: strip_tags(x))

Answer 1

页面通过 Ajax 动态加载。查看网络检查器，该页面从位于https://www.acc.co.nz/for-providers/treatment-recovery/work-type-detail-sheets/getSheets 的非常大的 JSON 文件中加载所有数据。要加载所有作业数据，您可以使用此脚本：

url = "https://www.acc.co.nz/for-providers/treatment-recovery/work-type-detail-sheets/getSheets"

import requests
import json

headers = {'X-Requested-With': 'XMLHttpRequest'}
r = requests.get(url, headers=headers)
data = json.loads(r.text)

# For printing all data in pretty form uncoment this line:
# print(json.dumps(data, indent=4, sort_keys=True))

for d in data:
    print(f'ID:\t{d["ID"]}')
    print(f'Job Title:\t{d["JobTitle"]}')
    print(f'Created:\t{d["Created"]}')
    print('*' * 80)

# Available keys in this JSON:
# ClassName
# LastEdited
# Created
# ANZSCO
# JobTitle
# Description
# WorkTasks
# WorkEnvironment
# PhysicalMentalDemands
# Comments
# EntryRequirements
# Group
# ID
# RecordClassName

打印出来：

ID: 2327
Job Title:  Watch and Clock Maker and Repairer   
Created:    2017-07-11 11:33:52
********************************************************************************
ID: 2328
Job Title:  Web Administrator
Created:    2017-07-11 11:33:52
********************************************************************************
ID: 2329
Job Title:  Welder 
Created:    2017-07-11 11:33:52

...and so on

在脚本中，我编写了可用的密钥，您可以使用这些密钥访问您的特定工作数据。