我想使用 BS4 或 Selenium 来抓取企业列表

Question

我想从该页面的所有页面中抓取商家信息： https://www.google.com/search?sxsrf=ACYBGNQ8Fi1dsaisfvundg1DuQk-01zamg:1573888738693&q=list+of+dry+cleaners+and+Launderettes+in+Ireland&npsic=0&rflfq=1&rlha=0&rllag=53813949,-7898686,117899&tbm=lcl&ved=2ahUKEwip_L25mO7lAhVQx4UKHe7lDYoQjGp6BAgKED0&tbs=lrf:!1m4!1u3!2m2!3m1!1e1!1m4!1u2!2m2!2m1!1e1!2m1!1e2!2m1!1e3!3sIAE,lf:1,lf_ui:2&rldoc=1#rldoc=1&rlfi=hd:;si:,53.28766449906681,-7.483483434374989;mv:[[54.7508777,-5.7301765],[51.7181496,-9.5224349]];start:20 我想获取干洗店在爱尔兰国家/地区的公司名称、联系方式、位置和营业时间。

我尝试使用 BS4，但我得到一个空数据框。

 import pandas as pd
 from bs4 import BeautifulSoup, Tag
 import requests
 import re
 data=[]
 s= "https://www.google.com/search?sxsrf=ACYBGNQ8Fi1dsaisfvundg1DuQk-01zamg:1573888738693&q=list+of+dry+cleaners+and+Launderettes+in+Ireland&npsic=0&rflfq=1&rlha=0&rllag=53813949,-7898686,117899&tbm=lcl&ved=2ahUKEwip_L25mO7lAhVQx4UKHe7lDYoQjGp6BAgKED0&tbs=lrf:!1m4!1u3!2m2!3m1!1e1!1m4!1u2!2m2!2m1!1e1!2m1!1e2!2m1!1e3!3sIAE,lf:1,lf_ui:2&rldoc=1#rlfi=hd:;si:;mv:[[54.07819766551908,-5.352135778124989],[52.2676036128125,-10.548668981249989],null,[53.182453596104196,-7.950402379687489],8];start:"
for x in range(0,100):
    res=requests.get(s+str(x))
    soup=BeautifulSoup(res.text,'lxml')
    listings=soup.findAll(class_='cXedhc')
    for listing in listings:
        listing_title=listing.find('div',{'class':'dbg0pd'})
        listing_details=listing.find('div',{'class':'rllt__details lqhpac'})
        full_dict={'Title':listing_title, 'details':listing_details}
        data.append(full_dict)
df=pd.DataFrame(data)
print(df)

---

Answer 1

数据框为空，因为结果为空。发出请求时应使用标头。例如，

import pandas as pd
from bs4 import BeautifulSoup, Tag
import urllib.request

user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36'
headers = {'User-Agent': user_agent}

limits = range(0, 120, 20)
names = []
details = []


for limit in limits:
    url = 'https://www.google.com/search?tbm=lcl&ei=WRPQXb7xKaTC8gL15JrABw&q=list+of+dry+cleaners+and+Launderettes+in+Ireland&oq=list+of+dry+cleaners+and+Launderettes+in+Ireland&gs_l=psy-ab.12...0.0.0.14131.0.0.0.0.0.0.0.0..0.0....0...1c..64.psy-ab..0.0.0....0.s_aHdwztgeA#rlfi=hd:;si:;mv:[[54.4254118,-5.9407238],[51.7489791,-9.924133399999999]]'

    if limit != 0:
        url = url + ';start:' + str(limit)

    req = urllib.request.Request(url, headers=headers)
    response = urllib.request.urlopen(req).read()
    soup = BeautifulSoup(response, features='lxml')
    listings = soup.findAll('div', {'class': 'cXedhc'})

    for l in listings:
        name = l.find('div', {'class': 'dbg0pd'}).text
        detail = l.find('span', {'class': 'rllt__details lqhpac'}).text

        names.append(name)
        details.append(detail)

df = pd.DataFrame({'title': names, 'detail': details})
print(df.head())

这是我的结果，

                                                title                                             detail
0                  Gills Dry Cleaners And Launderette  4,6  (10) · Kuru TemizlemeciWestport, County M...
1                  Mr Tubs Launderette & Dry Cleaners  4,3  (41) · Kuru TemizlemeciDublin 7, İrlanda+...
2            Supreme Clean Launderette & Dry-Cleaners  4,2  (46) · Kuru TemizlemeciDublin, İrlanda+35...
3                    Bogart Dry Cleaner & Launderette  4,5  (15) · Kuru TemizlemeciCounty Dublin, İrl...
4                        Gardiner Street Dry Cleaners  4,2  (69) · Kuru TemizlemeciDublin, İrlanda+35...
5                    Keadys Dry Cleaners & Laundrette  4,6  (11) · Kuru TemizlemeciSligo, İrlanda+353...