改进代码 - 网页抓取工作机会 - 职位、雇主、薪水、需要链接

Question

我编写了一个网络抓取代码，它扫描工作门户中的所有页面并报告满足功能中工资要求的工作机会。对我来说重要的字段是职位、雇主、薪水和链接。我现在正在使用 getText() 方法，但需要所有元素。 结果如下：

Zubný lekár/lekárka DENTAL CARE Dr. Rosa, s. r. o.Námestie sv. Františka, Karlova Ves


        Od 4 500 EUR/mesiac
    

Pridané Pred 4 dňami  Pridať k vybraným   
https://www.profesia.sk/praca/dental-care-dr-rosa/O3863429
https://www.profesia.sk/praca/dental-care-dr-rosa/O3863429


Head of Core Technology DevelopmentESET, spol. s r.o.Bratislava


        4 500 EUR/mesiac
    

Pridané pred 2 týždňami  Pridať k vybraným   
https://www.profesia.sk/praca/eset/C22141
https://www.profesia.sk/praca/eset/O3933805
https://www.profesia.sk/praca/eset/O3933805

它需要两个不必要的项目并复制链接（因为

def search4job(salary):
    import bs4, requests, re
    #Classes -> employer: class='employer'>
    # -> salary ".label"
    # -> Job Title class='title'
    # -> TODO: link 
    base_url= 'https://www.profesia.sk/praca/bratislava/plny-uvazok/?languages=73&page_num={}'
    page = 1 #to start from page1
    request = requests.get(base_url.format(page)) #to take complete url
    HTML = bs4.BeautifulSoup(request.text,'lxml') 
    pattern = r'(\d\s\d\d\d)' #salary pattern

    while len(HTML.select(".list-row"))>0: 
        #in pages without job offer the len of list-row is 0, iterates until there are no job offers
        
        #iteration within the page, return Job Details
        for i in HTML.select(".list-row"):
            #to give result only when there's a salary shown
            if i.find('span',{'class':'label-group'}):
                try:
                    #to give result only if the salary is higher than the one i want
                    if int(str(re.search(pattern,str(i.find('span',{'class':'label-group'}))).group()).replace(" ",""))>=salary:
                        #print Job Details
                        print(i.getText())
                        #print job offer link
                        try:
                            for link in i.findAll('a',attrs={'href':re.compile("/praca/")}):
                                print('https://www.profesia.sk'+str(link.get('href')))
                        except:
                            print("There is an error")
                        print('\n')  #new line between job offers
                except:
                    pass
        #iteration over the pages
        page +=1
        request = requests.get(base_url.format(page))
        HTML = bs4.BeautifulSoup(request.text,'lxml')
        #RESTART UNTIL THERE ARE JOB OFFERS

search4job(4000)

Answer 1

您可以使用此脚本从 Profesia 中抓取数据：

import requests
from bs4 import BeautifulSoup

base_url= 'https://www.profesia.sk/praca/bratislava/plny-uvazok/?languages=73&page_num={}'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

page = 1
while True:
    print(base_url.format(page))
    soup = BeautifulSoup(requests.get(base_url.format(page)).content, 'html.parser')

    links = soup.select('h2 > a')

    if not links:
        break

    for l in links:
        soup = BeautifulSoup(requests.get('https://www.profesia.sk' + l['href']).content, 'html.parser')

        job_title = soup.h1.text
        employer = soup.select_one('[itemprop="hiringOrganization"]')
        employer = employer.text if employer else '-'
        salary = soup.select_one('span[class^="salary"]')
        salary = salary.text if salary else '-'

        print(job_title)
        print(employer)
        print(salary)
        print('https://www.profesia.sk' + l['href'])
        print('-' * 80)

    page += 1

打印：

https://www.profesia.sk/praca/bratislava/plny-uvazok/?languages=73&page_num=1
Front Office Manager - AC Hotel by Marriott Bratislava Old Town
Legendhotels Slovakia, s.r.o.
From 1 800 EUR/month
https://www.profesia.sk/praca/legendhotels-slovakia/O3955894
--------------------------------------------------------------------------------
Catalog Quality Associate - Polish & Spanish
Amazon /Slovakia/ s.r.o.
1 150 EUR/month
https://www.profesia.sk/praca/amazon-slovakia/O3937464
--------------------------------------------------------------------------------
Financial Manager - AC Hotel by Marriott Bratislava Old Town
Legendhotels Slovakia, s.r.o.
From 2 200 EUR/month
https://www.profesia.sk/praca/legendhotels-slovakia/O3955853
--------------------------------------------------------------------------------

...and so on.