网页抓取时面临的问题答案

【问题标题】：Facing issue at the time of Web scraping网页抓取时面临的问题
【发布时间】：2020-12-10 10:48:23
【问题描述】：

我正在尝试从 Glass door 中提取评论。但是我面临问题。请按照下面的代码进行操作-

import requests
from bs4 import BeautifulSoup

headers = requests.utils.default_headers()
headers.update({
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
})

url = requests.get("https://www.glassdoor.co.in/Reviews/The-Wonderful-Company-Reviews-E1005987.htm?sort.sortType=RD&sort.ascending=false&countryRedirect=true", headers=headers)

urlContent =BeautifulSoup(url.content,"lxml")
print(urlContent)

review = urlContent.find_all('a',class_='reviewLink')
review

title = []
for i in range(0,len(review)):
    title.append(review[i].get_text())
title
rating= urlContent.find_all('div',class_='v2__EIReviewsRatingsStylesV2__ratingNum v2__EIReviewsRatingsStylesV2__small')

score=[]
for i in range(0,len(rating)):
    score.append(rating[i].get_text())

rev_pros=urlContent.find_all("span",{"data-test":"pros"})

pros=[]
for i in range(0,len(rev_pros)):
    pros.append(rev_pros[i].get_text())
pros

rev_cons=urlContent.find_all("span",{"data-test":"cons"})

cons=[]
for i in range(0,len(rev_cons)):
    cons.append(rev_cons[i].get_text())
cons

advse=urlContent.find_all("span",{"data-test":"advice-management"})
advse
advise=[]
for i in range(0,len(advse)):
    advise.append(advse[i].get_text())
advise
location=urlContent.find_all('span',class_='authorLocation')
location
job_location=[]
for i in range(0,len(location)):
    job_location.append(location[i].get_text())
job_location

import pandas as pd
df=pd.DataFrame()

df['Review Title']=title
df['Overall Score']=score
df['Pros']=pros
df['Cons']=cons
df['Jobs_Location']=job_location
df['Advise to Mgmt']=advise

在这里我面临两个挑战-

无法为 'advse' 提取任何内容（用于 'Advise to 管理'）。
当我在数据中使用“工作地点”作为列时出现错误框架。(ValueError: Length of values does not match length of index)。对于这个错误，我的发现是 - 有十行但是对于“工作地点”的其他列，行数较少一些评论中未披露位置。

任何机构都可以帮助我解决这个问题。提前致谢。

【问题讨论】：

advse 有什么问题？它会返回一个空列表吗？ ...至于丢失的位置，似乎 html 在所有 div 中都没有该元素...您的代码正在整个汤中搜索列，尝试按行搜索，如果这有意义的话...找到一个包含所有字段的父类并逐行收集数据
是的。 advse 返回空列表

标签： python pandas dataframe web-scraping beautifulsoup

【解决方案1】：

更好的方法是找到包含每条评论的<div>，然后从中提取所有需要的信息，然后再转到下一条。这样可以更轻松地处理某些评论中缺少信息的情况。

例如：

import requests
from bs4 import BeautifulSoup
import pandas as pd


headers = requests.utils.default_headers()

headers.update({
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
})

url = requests.get("https://www.glassdoor.co.in/Reviews/The-Wonderful-Company-Reviews-E1005987.htm?sort.sortType=RD&sort.ascending=false&countryRedirect=true", headers=headers)
urlContent = BeautifulSoup(url.content,"lxml")

get_text = lambda x: x.get_text(strip=True) if x else ""
entries = []

for entry in urlContent.find_all('div', class_='row mt'):
    review = entry.find('a', class_="reviewLink")
    rating = entry.find('div',class_='v2__EIReviewsRatingsStylesV2__ratingNum v2__EIReviewsRatingsStylesV2__small')
    rev_pros = entry.find("span", {"data-test":"pros"})
    rev_cons = entry.find("span", {"data-test":"cons"})
    location = entry.find('span', class_='authorLocation')
    advice = entry.find("span", {"data-test":"advice-management"})

    entries.append([
        get_text(review),
        get_text(rating),
        get_text(rev_pros),
        get_text(rev_cons),
        get_text(location),
        get_text(advice)
    ])

columns = ['Review Title', 'Overall Score', 'Pros', 'Cons', 'Jobs_Location', 'Advise to Mgmt']
df = pd.DataFrame(entries, columns=columns)

print(df)

get_text() 函数确保如果没有返回任何内容（即None），则返回一个空字符串。

您需要改进提取建议的逻辑。整个页面的信息保存在<script> 标记内。其中之一保存 JSON 数据。建议信息在用户单击之前不会移动到 HTML 中，因此需要从 JSON 中提取它。如果使用这种方法，那么直接从 JSON 中提取所有其他信息也是有意义的。

为此，请找到所有<script> 标签并确定哪些标签包含评论。将 JSON 转换为 Python 数据结构（使用 JSON 库）。现在找到评论，例如：

import requests
from bs4 import BeautifulSoup
import json
import pandas as pd

headers = requests.utils.default_headers()

headers.update({
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
})

url = requests.get("https://www.glassdoor.co.in/Reviews/The-Wonderful-Company-Reviews-E1005987.htm?sort.sortType=RD&sort.ascending=false&countryRedirect=true", headers=headers)
urlContent = BeautifulSoup(url.content,"lxml")

entries = []

for script in urlContent.find_all('script'):
    text = script.text
    
    if "appCache" in text:
        # extract the JSON from the script tag
        data = json.loads(text[text.find('{'): text.rfind('}') + 1])

        # Go through all keys in the dictionary and pick those containing reviews
        for key, value in data['apolloState'].items():
            if ".reviews." in key and "links" not in key:

                location = value['location']
                city = location['id'] if location else None
                
                entries.append([
                    value['summary'],
                    value['ratingOverall'],
                    value['pros'],
                    value['cons'],
                    city,
                    value['advice']
                ])
                

columns = ['Review Title', 'Overall Score', 'Pros', 'Cons', 'Jobs_Location', 'Advise to Mgmt']
df = pd.DataFrame(entries, columns=columns)

print(df)

这将为您提供如下数据框：

            Review Title  Overall Score                   Pros                   Cons Jobs_Location         Advise to Mgmt
0  Upper management n...              3  Great benefits, lo...  Career advancement...  City:1146821  Listen to your emp...
1                  Sales              2  Good atmosphere lo...  Drive was very far...          None                   None
2  As an organization...              2  Free water and goo...  Not a lot of diver...          None                   None
3    Great place to grow              4  If your direct man...  Owners are heavily...  City:1146821                   None
4          Great Company              5  Great leadership, ...  To grow and move u...  City:1146821                   None
5  Lots of opportunit...              5  This is a fast pac...  There's a sense of...  City:1146821  Continue listening...
6  Interesting work i...              3  Working with great...  High workload and ...          None                   None
7              Wonderful              5  This  company care...  The drive, but we ...  City:1146577  Continue growing y...
8             Horrendous              1  The pay was fairly...  Culture of abuse a...  City:1146821  Upper management l...
9  Upper Leadership a...              1  Strong Company, fu...  You don't have a f...  City:1146577  You get rid of fol...

如果您添加 print(data) 以查看返回的数据的整个结构，将会有所帮助。这种方法的唯一问题是需要进一步查找才能将城市 ID 转换为实际位置。该信息也包含在 JSON 中。

【讨论】：