用美丽的汤刮掉多页

Question

我正在尝试抓取一个 url 的多个页面。

但我只能抓取第一页，有办法获取所有页面。

这是我的代码。

from bs4 import BeautifulSoup as Soup 
import urllib, requests, re, pandas as pd

pd.set_option('max_colwidth',500)    # to remove column limit (Otherwise, we'll lose some info) 
df = pd.DataFrame()

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:
        comp_name = elem.find('span', attrs={'class':'company'}).getText().strip()
        job_title = elem.find('a', attrs={'class':'turnstileLink'}).attrs['title']
        home_url = "http://www.indeed.com"
        job_link = "%s%s" % (home_url,elem.find('a').get('href'))
        job_addr = elem.find('span', attrs={'class':'location'}).getText()
        date_posted = elem.find('span', attrs={'class': 'date'}).getText()
        description = elem.find('span', attrs={'class': 'summary'}).getText().strip()


        comp_link_overall = elem.find('span', attrs={'class':'company'}).find('a')
        if comp_link_overall != None:
        comp_link_overall = "%s%s" % (home_url, comp_link_overall.attrs['href'])
        else: comp_link_overall = None

        df = df.append({'comp_name': comp_name, 'job_title': job_title,
                'job_link': job_link, 'date_posted': date_posted,
                'overall_link': comp_link_overall, 'job_location': job_addr, 'description': description
                }, ignore_index=True)


df

df.to_csv('path\\web_scrape_Indeed.csv', sep=',', encoding='utf-8')

如果有，请提出建议。

Answer 1

案例 1：这里提供的代码正是您所拥有的

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

for elem in targetElements:

这里的问题是targetElements 在第一个 for 循环中的每次迭代都会发生变化。

为避免这种情况，在第一个 for 循环中缩进第二个，如下所示：

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:

案例 2：您的错误不是由于缩进不当造成的 （即与您原始帖子中的内容不同） 如果您的代码被正确识别，那么targetElements 可能是一个空列表。这意味着target.findAll('div', class_ =' row result') 不返回任何内容。在这种情况下，请访问网站，检查 dom，然后修改您的抓取程序。