python append() 并删除 html 标签

Question

我需要一些帮助。我的输出似乎是错误的。如何正确附加 dept、job_title、job_location 的值。并且有带有dept值的html标签。我怎样才能删除这些标签。

我的代码

response = requests.get("http://hortonworks.com/careers/open-positions/")
soup = BeautifulSoup(response.text, "html.parser")

jobs = []


div_main = soup.select("div#careers_list")


for div in div_main:
    dept = div.find_all("h4", class_="department_title")
    div_career = div. find_all("div", class_="career")
    title = []
    location = []
    for dv in div_career:
        job_title = dv.find("div", class_="title").get_text().strip()
        title.append(job_title)
        job_location = dv.find("div", class_="location").get_text().strip()
        location.append(job_location)

    job = {
        "job_location": location,
        "job_title": title,
        "job_dept": dept
    }
    jobs.append(job)
pprint(jobs)

应该是这样的

{'job_dept'：咨询，

'job_location':'伊利诺伊州芝加哥'

'job_title'：高级顾问 - Central'

每个变量有 1 个值。

Answer 1

您的 html 结构是顺序的，而不是分层的，因此您必须遍历您的工作列表并随时更新部门名称：

import requests
from bs4 import BeautifulSoup, Tag
from pprint import pprint
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20130331 Firefox/21.0'}
response = requests.get("http://hortonworks.com/careers/open-positions/", headers=headers)

soup = BeautifulSoup(response.text, "html.parser")

jobs = []


div_main = soup.select("div#careers_list")


for div in div_main:
    department_title = ""
    for element in div:
        if isinstance(element, Tag) and "class" in element.attrs:
            if "department_title" in element.attrs["class"]:
                department_title = element.get_text().strip()
            elif "career" in element.attrs["class"]:
                location = element.select("div.location")[0].get_text().strip()
                title = element.select("div.title")[0].get_text().strip()
                job = {
                    "job_location": location,
                    "job_title": title,
                    "job_dept": department_title
                }
                jobs.append(job)

pprint(jobs)