网页抓取数据表单ajax页面

Question

我正在尝试从 here 中抓取职位名称。

我正在学习 python 抓取技术，但我遇到了抓取像这样的 Ajax 页面的问题。我可以使用下面的第一页代码获取开发人员工具响应数据。如何从这些数据中提取职位。

from bs4 import BeautifulSoup
import requests
import json

s = requests.Session()
headers={"User-Agent":"Mozilla/5.0"}
r=s.get('https://epco.taleo.net/careersection/alljobs/jobsearch.ftl?lang=en',headers=headers)
html = r.text
soup = BeautifulSoup(html, 'lxml')
print(soup)

###how to extract job titles from soup###

非常感谢您对此的任何帮助。

不幸的是，我目前仅限于使用请求或其他流行的 python 库。提前致谢。

Answer 1

此站点是动态的（使用 javascript 更改数据），因此您必须使用 Selenium。 你可以在 headless 中运行它，就像发送请求一样：

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless')

driver = webdriver.Chrome(executable_path=r'yourpath\chromedriver.exe', chrome_options=options)

driver.get('https://epco.taleo.net/careersection/alljobs/jobsearch.ftl?lang=en')

html = (driver.page_source).encode('utf-8')
soup = BeautifulSoup(html, 'lxml')
print(soup)

Answer 2

数据位于<script> 标记内。您可以使用re 模块查找正确的职位名称。

import re
import requests

headers = {"User-Agent": "Mozilla/5.0"}

response = requests.get(
    "https://epco.taleo.net/careersection/alljobs/jobsearch.ftl?lang=en"
)
job_titles = re.findall(r"Add this position to the job cart: (.*?)'", response.text)
print(len(job_titles))
print(job_titles)

输出：

25
['Technician, I %26 E (Coyanosa, TX)', 'Engineer, Senior Project', 'Engineer, Project', 'Mechanic, Truck (Monahans, TX)', 'Technician, Pipeline (Bryan/College Station)', 'Technician, Measurement (Farmington, NM)', 'Assistant, Field Administrative (Carlsbad, NM)', 'Technician, Pipeline (Greensburg, PA)', 'Human Resources Business Partner', 'Engineer, Senior Measurement', 'Accountant (Mont Belvieu)', 'Specialist, Senior Accounts Payable', 'Technician, Pipeline Trainee( Cape Girardeau)', 'Specialist, EAM Inventory', 'Welder - Class B', 'Specialist, Senior NGL Accounts Payable', 'Technician, Pipeline (Hobbs, NM)', 'Auditor, IT', 'Accountant, Intermediate', 'Accountant', 'Operator, Plant (Sonora, TX)', 'Technician, Pipeline (Carlsbad, NM)', 'Specialist, Maintenance (Lebanon, OH)', 'Technician, Pipeline Trainee ', 'Specialist, Senior Systems']

Answer 3

试试：

import re
import json
import requests

url = "https://epco.taleo.net/careersection/alljobs/jobsearch.ftl?lang=en"

data = re.search(r"listRequisition', (\[.*?\])\);", requests.get(url).text)
data = data.group(1).replace("'", '"')
data = json.loads(data)
for i in range(25):
    row = data[i * 40 : (i + 1) * 40]
    print(row[5])

打印：

Technician, I %26 E (Coyanosa, TX)
Engineer, Senior Project
Engineer, Project
Mechanic, Truck (Monahans, TX)
Technician, Pipeline (Bryan/College Station)
Technician, Measurement (Farmington, NM)
Assistant, Field Administrative (Carlsbad, NM)
Technician, Pipeline (Greensburg, PA)
Human Resources Business Partner
Engineer, Senior Measurement
Accountant (Mont Belvieu)
Specialist, Senior Accounts Payable
Technician, Pipeline Trainee( Cape Girardeau)
Specialist, EAM Inventory
Welder - Class B
Specialist, Senior NGL Accounts Payable
Technician, Pipeline (Hobbs, NM)
Auditor, IT
Accountant, Intermediate
Accountant
Operator, Plant (Sonora, TX)
Technician, Pipeline (Carlsbad, NM)
Specialist, Maintenance (Lebanon, OH)
Technician, Pipeline Trainee 
Specialist, Senior Systems