美丽的汤分页 - 只刮第一页

Question

我的 BeautifulSoup 刮刀还有一个问题，希望您能帮助我。自从为其添加了从详情页中提取信息的代码后，现在抓取工具只抓取第一页，不再抓取列出的多个页面。

我认为它与循环有关，但我不确定如何定义它以确保它不会引起问题。

有没有更好的分页方式？

我已经发布了完整的代码，并将导致问题的新部分加粗。

完整代码：

from bs4 import BeautifulSoup
import scrapy
import requests
import csv
import time
import os
pages = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]

with open(r'csv', 'a', encoding='utf-8', newline='') as f_output:
    csv_print = csv.writer(f_output)

    file_is_empty = os.stat(r'C:\csv').st_size == 0
    if file_is_empty:
        csv_print.writerow(['Title', 'Company', 'Location', 'Salary', 'Summary', 'Link', 'Description', 'URL'])

    for page in pages:
        source = requests.get('https://www.indeed.com/jobs?q=work+from+home&l=United+States&fromage=1&start={}'.format(page)).text
        soup = BeautifulSoup(source, 'lxml')

        results = soup.findAll("div", {"class": "result"})

    for jobs in soup.find_all(class_='result'):

            try:
                title = jobs.h2.text.strip()
            except Exception as e:
                title = None
            print('Title:', title)

            try:
                company = jobs.span.text.strip()
            except Exception as e:
                company = None
            print('Company:', company)

            try:
                location = jobs.find('span', class_='location').text.strip()
            except Exception as e:
                location = None
            print('Location:', location)

            try:
                salary = jobs.find('span', class_='no-wrap').text.strip()
            except Exception as e:
                salary = None
            print('Salary:', salary)

            try:
                summary = soup.find('div', class_='summary').text.strip()
            except Exception as e:
                summary = None
            print('Summary:', summary)

            link = jobs.a['href']
            if 'http' not in link:
                link = ("https://www.indeed.com" + link)
            print('Link:', link)

            **page = requests.get(link)
            soup = BeautifulSoup(page.content, 'html.parser')
            job_description = soup.find('div', id='jobDescriptionText').decode_contents(formatter="html")
            print('job_description:', job_description)
            try:
                url = soup.find('div', class_='icl-u-lg-hide').attrs['href']
            except Exception as e:
                url = None
            print('url:', url)**

            csv_print.writerow((title, company, location, salary, summary, link, job_description, url))

            print('--------')

            time.sleep(0.5)

非常感谢你:) 你们是最棒的！

Answer 1

几件事，

1. BeautifulSoup 解析 html，与实现分页过程无关。
2. 按照您的代码编写方式，它实际上并没有抓取第一页，而是抓取最后一页。你让它遍历页面列表，然后让它解析 html（在这种情况下是最后一个请求）。你需要为每一页解析你的results。意思是for jobs in soup.find_all(class_='result'): 需要在for page in pages: 的循环内
3. 这只是我的偏好，但我喜欢使用pandas。所以我用它而不是csv 来写入文件

代码：

from bs4 import BeautifulSoup
import pandas as pd
import requests
import time

pages = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]


titleList = []
companyList = []
locList = []
salList = []
summaryList = []
linkList = []
descList = []
urlList = []


for page in pages:
    source = requests.get('https://www.indeed.com/jobs?q=work+from+home&l=United+States&fromage=1&start={}'.format(page)).text
    soup = BeautifulSoup(source, 'lxml')

    print ('Page: %s' %page)

    results = soup.findAll("div", {"class": "result"})

    for jobs in soup.find_all(class_='result'):

            try:
                title = jobs.h2.text.strip()
            except Exception as e:
                title = None
            print('Title:', title)

            try:
                company = jobs.span.text.strip()
            except Exception as e:
                company = None
            #print('Company:', company)

            try:
                location = jobs.find('span', class_='location').text.strip()
            except Exception as e:
                location = None
            #print('Location:', location)

            try:
                salary = jobs.find('span', class_='no-wrap').text.strip()
            except Exception as e:
                salary = None
            #print('Salary:', salary)

            try:
                summary = soup.find('div', class_='summary').text.strip()
            except Exception as e:
                summary = None
            #print('Summary:', summary)

            link = jobs.a['href']
            if 'http' not in link:
                link = ("https://www.indeed.com" + link)
            #print('Link:', link)

            page = requests.get(link)
            soup = BeautifulSoup(page.content, 'html.parser')
            job_description = soup.find('div', id='jobDescriptionText').decode_contents(formatter="html")
            #print('job_description:', job_description)
            try:
                url = soup.find('div', class_='icl-u-lg-hide').attrs['href']
            except Exception as e:
                url = None
            #print('url:', url)

            titleList.append(title)
            companyList.append(company)
            locList.append(location)
            salList.append(salary)
            summaryList.append(summary)
            linkList.append(link)
            descList.append(job_description)
            urlList.append(url)


            print('--------')

            time.sleep(0.5)

df = pd.DataFrame({
        'Title':titleList,
        'Company':companyList,
        'Location':locList, 
        'Salary':salList, 
        'Summary':summaryList, 
        'Link':linkList, 
        'Description':descList, 
        'URL':urlList})

df.to_csv('file.csv',index=False)

输出：

print (df.head(15).to_string())
                                                Title                        Company                                           Location             Salary                                            Summary                                               Link                                        Description   URL
0   Medical Transcription Documentation Specialist...                            new                                               None               None  Extract electronic medical record data includi...  https://www.indeed.com/rc/clk?jk=b728723376dae...  Dane Street is looking for highly motivated ca...  None
1           Company Partnerships Data Contractor\nnew                            new        New York, NY 10001 (Flatiron District area)        $30 an hour                                               None  https://www.indeed.com/rc/clk?jk=188b5c89acc25...  <div><div>Do you love gathering, analyzing and...  None
2   Customer Experience Associate - work from home...                            new  Baltimore, MD 21230 (Spring Garden Industrial ...  $16 - $17 an hour                                               None  https://www.indeed.com/rc/clk?jk=bdf4e0b474852...  <div><div><div><b>About Hungry Harvest</b></di...  None
3   Customer Care Representative I - Houston, TX -...                            new                  Houston, TX 77036 (Bellaire area)               None                                               None  https://www.indeed.com/rc/clk?jk=b144f9b3db19a...  <div><p><b>Description</b>\n</p>SHIFT: Day Job...  None
4                  Talent Acquisition Specialist\nnew                            new                                               None               None                                               None  https://www.indeed.com/rc/clk?jk=f9c860e3dea80...  <p></p><div><p><b>The Position</b></p><p>\nTal...  None
5                  Talent Acquisition Specialist\nnew                            new                                               None               None                                               None  https://www.indeed.com/rc/clk?jk=5b1609f84a81b...  <div><div><div><div>Job Details</div>\n</div><...  None
6                             Success Consultant\nnew                            new                                               None               None                                               None  https://www.indeed.com/rc/clk?jk=e2b62a52550b8...  <div><div>WORK FROM HOME</div><div><h6 class="...  None
7   Work At Home Customer Service Specialist (Bell...                            new                                    Belle Glade, FL     $13.25 an hour                                               None  https://www.indeed.com/rc/clk?jk=0bd3274a26a1f...  US55505\n<br/><br/>\nJob Description Details\n...  None
8                                 Data Associate\nnew                            new                                               None               None                                               None  https://www.indeed.com/rc/clk?jk=88d65d8102b17...  <div></div><div><div><div><div>The DNC Coordin...  None
9                            States Data Analyst\nnew                            new                                               None               None                                               None  https://www.indeed.com/rc/clk?jk=8c84a472e6119...  <div></div><div><div><div><div>The DNC Coordin...  None
10                                              Coder              Franciscan Health                                               None               None  Understanding of payer relationships, requirem...  https://www.indeed.com/rc/clk?jk=cc0748fa0c360...  <div><div>Ambulatory Coding | Franciscan Allia...  None
11                                     Data Associate  Democratic National Committee                                               None               None                                               None  https://www.indeed.com/rc/clk?jk=88d65d8102b17...  <div></div><div><div><div><div>The DNC Coordin...  None
12  Work At Home Customer Service Specialist (Boyn...                            HSN                                  Boynton Beach, FL     $13.25 an hour                                               None  https://www.indeed.com/rc/clk?jk=a59f7532662bc...  <p></p><div>US55505\n<p></p><p><b>Job Descript...  None
13                          Associate Project Manager                    ConvergeOne                                               None               None                                               None  https://www.indeed.com/rc/clk?jk=cbc2520fc153b...  <div>C1 Company Overview:<div><b>\nConvergeOne...  None
14                      Customer Solutions Specialist        Webstaurant Store, Inc.                                         Albany, GA        $19 an hour                                               None  https://www.indeed.com/rc/clk?jk=8f5bd656508fd...  <div><p>Customer Solutions Specialist</p>\n<p>...  None
....