【发布时间】：2021-08-02 21:27:48
【问题描述】：
我需要从 Indeed 上抓取招聘信息。我设法抓取了每个职位的标题和链接，现在正在努力抓取每个职位的完整职位描述（我不想要摘要 - 我想要每个职位的完整职位描述）。
我的代码如下所示：
from bs4 import BeautifulSoup
import requests
import pandas as pd
import time
url = 'https://www.indeed.com/jobs?q=data+engineer&l=United+States'
response = requests.get(url)
data = response.text
soup = BeautifulSoup(data, 'html.parser')
jobs = soup.find_all('div',{'class':'jobsearch-SerpJobCard'})
for job in jobs:
title = job.find('a',{'class':'jobtitle'}).text
link1 = job.find('a',{'class':'jobtitle'}).get('href')
link = 'https://www.indeed.com' + link1
#for each JOB's webpage, you need to connect to the link first:
job_response = requests.get(link)
job_data = response.text
job_soup = BeautifulSoup(job_data, 'html.parser')
job_description_tag = job_soup.find('div',{'id':'jobDescriptionText'})
job_description = job_description_tag.text if job_description_tag else "N/A"
print('Job Title:', title, '\nLink:', link, '\nJob Description:', job_description, '\n---')
我想要每个职位的完整职位描述，并尝试使用 job_description_tag 获取它，但它只返回“N/A”（我将 if 语句放在那里以应对职位发布没有职位描述的情况。） .
每个职位发布的输出都返回一个“N/A”，因此显然有问题。
作为参考，通过检查其中一个职位的职位描述，可以看到我试图抓取的 job_description_tag 的 html 代码如下所示：
<div id="jobDescriptionText" class="jobsearch-jobDescriptionText"><ul>
<li>1+ years of experience as a Data Engineer or in a similar role</li>
<li>Experience with data modeling, data warehousing, and building ETL pipelines</li>
<li>Experience in SQL</li>
<li>Knowledge of python or any general purpose scripting language.</li>
<li>Experience with Big Data technologies such as Hive/Spark.</li>
</ul>
Mission Statement
<br>The core mission of Amazon Web Services (AWS) Marketing is to educate customers about cloud computing and our services. Millions of customers engage with us every day across multiple channels. Imagine building a platform that enables AWS to speak to engineers, CTOs, CIOs, and CEOs, educate them about AWS services, and empower them on their journey to the cloud. Our services act as the foundation for announcing new AWS products and are uniquely positioned to redefine how our cloud community consumes information and engages with AWS.
<br><br>
Overview
<br>Would you like to support increasing customer base and the revenue for AWS, a market-leading cloud offering? Would you like to be part of a team focused on increasing awareness and adoption of the AWS platform by analyzing customer's behavior on and outside AWS websites? Do you want to empower our AWS marketing team make data-driven decisions that further establish AWS as leader in the cloud computing world?
<br><br>
As a Data Engineer at AWS, you will be working in a large, extremely complex and dynamic data warehousing environment. We are looking for someone with the uncanny ability to integrate multiple heterogeneous data sources like Adobe Site Catalyst, Adobe Target, Sales Force, Adobe Connect with AWS central data warehouse and build efficient, flexible, and scalable data warehouse and reporting solutions. You should be enthusiastic about learning new technologies and be able to implement solutions using these technologies to enable upgrades of the existing platform. You should have excellent business and communication skills and be able to work with business owners to develop and define key business questions, then build the data sets that answer those questions. You should be expert at designing, implementing, and operating stable, scalable, low cost solutions to flow data from production systems into the data warehouse and into end-user facing reporting applications. Above all you should be passionate about working with huge data sets and someone who loves to bring datasets together to answer business questions and drive growth.
<br><br>
At AWS, you have control over every layer you build. Instead of owning a small slice of an existing service, you will own a core segment of a growing marketing platform serving 1000s of internal customers and millions of external customers. You will build on multiple AWS services and have opportunities to engage directly with those teams to improve our core offerings. At AWS, we work with our customers on a daily basis to prove out our ideas, gather feedback, and improve the platform.
<br><br>
<b>Location:</b> This position must sit in Seattle, WA. Relocation assistance offered from within the US.
<br><br>
<ul>
<li>Graduate/Master degree in Computer Science, Engineering or related technical field.</li>
<li>Exceptional troubleshooting and problem-solving abilities.</li>
<li>Experience with Amazon Redshift or other distributed computing technology.</li>
<li>Industry experience as a Data Engineer or related specialty (e.g., Software Engineer, Business Intelligence Engineer, Data Scientist) with a track record of manipulating, processing, and extracting value from large datasets.</li>
<li>Experience with AWS Tools and Technologies.</li>
<li>Hands-on experience with cloud computing and UNIX/Linux based systems.</li>
<li>Demonstrated ability to work effectively across various internal organizations.</li>
<li>Excellent written and verbal communications skills.</li>
</ul>
Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.</div>
任何帮助将不胜感激！
【问题讨论】：
-
可以动态生成描述吗？见stackoverflow.com/q/60904786/11301900。
标签： python html css web-scraping beautifulsoup