试图从 yelp 中获取线索

Question

我正在尝试使用 python 和 beautifulsoup 从 yelp 获取潜在客户，但我无法捕获电话名称地址和 wesbite 字段（可选）。 我收到以下错误，这是我尝试搜索并找到不同解决方案的代码，但它们对我不起作用。

这是我的代码

from bs4 import BeautifulSoup
import requests
import sys
import csv
import requests, re, json
## Get the min and max page numbers
pagenum=0

maxpage =0
## loop go thourgh the pages
while pagenum <= maxpage:
    newsu =pagenum
    newsu = str(newsu)
    csvname = 'cardealers'+newsu+'.csv';
    csvfile = open(csvname , 'w',encoding="utf-8")
    csv_writer = csv.writer(csvfile)
    csv_writer.writerow(['Business name', 'phone' , 'address'] )

    headers = {'User-Agent':'Mozilla/5.0'}
    r = requests.get('https://www.yelp.com/search?find_desc=Used%20Car%20Dealers&find_loc=New%20York%2C%20NY&ns=1&sortby=review_count&start={}'.format(pagenum), headers = headers)
    p = re.compile(r'PRELOADED_STATE__ = (.*?);')
    data = json.loads(p)
    print(data)
    pagenum =pagenum+1
    for item in data['searchResult']['results']:
        name = item['businessName']
        phone=item['phone']
        address= ([item['address'],item['city'], item['state'], item['postalcode']])
        csv_writer.writerow([name, phone , address ])
        print(name)
    csvfile.close()

这是错误信息。

Traceback（最近一次调用最后一次）：文件 “\Python\Python36\scraper\scrape.py”，第 22 行，在 data = json.loads(p) 文件“\Python\Python36\lib\json__init__.py”，第 348 行，加载中 'not {!r}'.format(s.class.name)) TypeError: JSON object must be str, bytes or bytearray, not 'SRE_Pattern'

Answer 1

您正在尝试读取非 json 格式的字符串。

基本上，这就是你正在做的事情：

data = json.loads('THIS IS JUST A STRING. NOT IN A JSON FORMAT')

所以你想做这样的事情：data = json.loads(p.findall(r.text))

您实际上需要将其从 html 中提取出来。另一个主要问题是，它甚至不在您要提取的 html 中......所以它总是会返回一个空列表。

另外，你没有迭代任何东西。您从 pagenum=0 开始，maxpage page=0 并在 pagenum

带有数据的 json 结构在 html 中，但看起来像是在 Comments 中。所以你需要解析它。

还有，为什么要这样做：

newsu =pagenum
newsu = str(newsu)

只需执行newsu = str(pagenum)。你真的想要每次迭代都有一个单独的文件吗？我只是将它放入 1 个文件中：

from bs4 import BeautifulSoup
import requests
import pandas as pd
import json
import math

## Get the min and max page numbers
pagenum=0
results = pd.DataFrame()
with requests.Session() as s:
        headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'}
        url = 'https://www.yelp.com/search?find_desc=Used%20Car%20Dealers&find_loc=New%20York%2C%20NY&ns=1&sortby=review_count&start={}'.format(pagenum)
        r = s.get(url, headers = headers)
        soup = BeautifulSoup(r.text, 'html.parser')

        scripts = soup.find_all('script')
        for script in scripts:
            if '<!--{' in script.text:
                jsonStr = script.text.split('<!--')[-1].split('-->')[0]
                jsonData = json.loads(jsonStr)

        totalPages = jsonData['searchPageProps']['searchResultsProps']['paginationInfo']['totalResults']
        resultsPerPage = jsonData['searchPageProps']['searchResultsProps']['paginationInfo']['resultsPerPage']
        totalPages = math.ceil(totalPages/resultsPerPage)

        ## loop go through the pages
        for pagenum in range(0,totalPages+1):
            url = 'https://www.yelp.com/search?find_desc=Used%20Car%20Dealers&find_loc=New%20York%2C%20NY&ns=1&sortby=review_count&start={}'.format(pagenum)
            r = s.get(url, headers = headers)
            soup = BeautifulSoup(r.text, 'html.parser')

            scripts = soup.find_all('script')
            for script in scripts:
                if '<!--{' in script.text:
                    jsonStr = script.text.split('<!--')[-1].split('-->')[0]
                    jsonData = json.loads(jsonStr)


            for each in jsonData['searchPageProps']['searchResultsProps']['searchResults']:
                if 'searchResultBusiness' in each.keys():
                    busiName = each['searchResultBusiness']['name']
                    phone = each['searchResultBusiness']['phone']
                    address = each['searchResultBusiness']['formattedAddress']

                    temp_df = pd.DataFrame([[busiName, phone, address]], columns=['Business name', 'phone' , 'address'])
                    results = results.append(temp_df, sort=False).reset_index(drop=True)
            print ('Aquired page: %s' %pagenum)



results.to_csv('cardealers.csv', index=False)