网络从多个页面抓取信息到 pandas 数据框答案

【问题标题】：Web scraping information from multiple pages into a pandas dataframe网络从多个页面抓取信息到 pandas 数据框
【发布时间】：2020-06-26 20:21:40
【问题描述】：

我想编写一些代码，从工作列表网站的多个页面中抓取数据。然而，目前，当我运行我的代码时，我只得到最后一页，而不是我抓取的所有页面的列表。

这是我的代码

url = 'https://ng.indeed.com/jobs?q=Business+Intelligence+Analyst&l=Nigeria&start='
for i in range(0,80,10):
    page = requests.get(url+str(i))
    soup = BeautifulSoup(page.text, 'html.parser')
    jobs = []
    for div in soup.find_all(name='div',attrs={'class':'row'}):
        for a in div.find_all(name='a', attrs={'data-tn-element':'jobTitle'}):
            jobs.append(a['title'])
    summaries = []
    divs = soup.findAll('div', attrs={'class':'summary'})
    for d in divs:
        summaries.append(d.text.strip())
jobs = pd.DataFrame(
    {'title': extract_title(soup),
     'summary': extract_summary(soup)
    })
jobs

我使用第一个 for 循环遍历每个页面（第 2 页 = 10、3=20 等）。理想的输出是一个数据框，其中包含所有职位的列表和每个职位的摘要。但是我只从最后一页得到一个包含作业的数据框

【问题讨论】：

将您的列表移到 for 循环之外，它应该可以解决问题

标签： python pandas beautifulsoup python-requests

【解决方案1】：

import requests
from bs4 import BeautifulSoup


summaries = []   # <-- outside of the loop
jobs = []        # <-- outside of the loop

url = 'https://ng.indeed.com/jobs?q=Business+Intelligence+Analyst&l=Nigeria&start='

for i in range(0,80,10):
    page = requests.get(url+str(i))
    soup = BeautifulSoup(page.text, 'html.parser')
    for div in soup.find_all(name='div',attrs={'class':'row'}):
        for a in div.find_all(name='a', attrs={'data-tn-element':'jobTitle'}):
            jobs.append(a['title'])
    divs = soup.findAll('div', attrs={'class':'summary'})
    for d in divs:
        summaries.append(d.text.strip())

jobs = pd.DataFrame({'title': jobs,    # <--- put only jobs here
     'summary': summaries})            # <--- put only summaries here

print(jobs)

打印：

                                              title                                            summary
0      Analyst, Customer Intelligence (Supervisory)  Provide Intelligence To Support Business Plann...
1                     Business Intelligence Analyst  Demonstrable work experience in business intel...
2                    Manager, Business Intelligence  Provide Business Intelligence Services For CEO...
3           MTNN Need Digital Communication Analyst  Work With Individual Units (Corporate Communic...
4   MARKET RESEARCH & BUSINESS INTELLIGENCE OFFICER  Implement the overall analytics and business i...
..                                              ...                                                ...
80                  Research Analyst and Associates  Experience of Business Intelligence tools.\nIn...
81                                Financial Analyst  Perform market research, data mining, business...
82       Oracle E-business Suite Developer (Fusion)  Work Directly with Business user as an oracle ...
83                          Junior Oracle Developer  Work Directly with Business user as an oracle ...
84                 Credit Analyst at CARS45 Limited  High business research skills acumen.\nUnderst...

[85 rows x 2 columns]

【讨论】：