【问题标题】:Unable to scrape a particular site using python, urllib2无法使用 python、urllib2 抓取特定站点
【发布时间】:2020-12-10 18:37:03
【问题描述】:

我正在使用 urllib 来提取网页,它似乎不适用于 zomato 的网站。它适用于我尝试过的其他几个网站。我已经尝试禁用我的防火墙、防病毒软件,并将 urlopen 行包裹在 try catch 中,因为我发现它here。我尝试使用来自互联网的代码,这些代码似乎可以正常工作,但我收到一条错误消息,

v = self._sslobj.read(len or 1024) socket.error: [Errno 10060] 连接尝试失败,因为连接方在一段时间后没有正确响应,或者连接失败,因为连接的主机没有响应。

可能的原因是什么?

这是它的简单代码。

import urllib2
from bs4 import BeautifulSoup

def extract_link(url):            
    page = urllib2.urlopen(url).read()                    
    return BeautifulSoup(page)

def main():
    link = 'https://www.zomato.com/kolkata'
    soup=extract_link(link)
    print soup.prettify()

if __name__== '__main__':
    main()

【问题讨论】:

    标签: python-2.7 runtime-error urllib2


    【解决方案1】:

    由于 zomato.com 是一个动态网站,并且严重依赖 Javascript 或 AJAX 来填充其内容,因此您需要一个浏览器才能获取该内容。最重要的是,zomato 会过滤掉它认为是由机器人发出的请求。因此,出于这些原因,我建议您使用 Selenium,因为它不模拟浏览器会话,它是浏览器会话。为 selenium 编写基本上是编写一组操作并将它们提供给浏览器(通常是 Firefox,但它可以与其他浏览器一起使用)。由于您使用的是 python,因此您也可以使用 PhantomJs 作为替代方案。

    这里有一些关于设置 selenium 和复制上述代码的帮助。

    1. 从https://chromedriver.storage.googleapis.com/index.html?path=2.25/下载chromedriver.exe
    2. 然后将这个 .exe 文件解压到“Chrome Driver”文件夹中。
    3. 然后将包含该文件的文件夹剪切并粘贴到您的 C 盘中。
    4. 现在,做pip install selenium.

    然后运行这段代码

    import time
    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    driver = webdriver.Chrome("C:\Chrome Driver\chromedriver")
    driver.get("https://www.zomato.com/kolkata")
    time.sleep(3)
    
    wait = WebDriverWait(driver, 30)
    
    body = wait.until(EC.presence_of_element_located((By.XPATH,"/html/body")))
    print(body.text)
    

    这是您将获得的输出:

    We're Hiring!
    Order Food Online!
    Log in
    Find the best restaurants, cafés, and bars in Kolkata
    Kolkata
    Search
    Collections
    Explore curated lists of top restaurants, cafes, pubs, and bars in and around Kolkata, based on trends
    Trending this week
    The most popular restaurants in town this week
    New restaurants you probably haven't tried yet
    The best new places in town
    Happy hours
    Great deals on booze. Happy hours indeed
    Hookah bars
    Great places to enjoy flavored Hookah.
    All collections in Kolkata
    Order Food Online
    From the best restaurants delivering to your doorstep
    Enter your delivery location
    Detect
    Order Food Online!
    Get 15% off (up to Rs. 100) on your first order with the code EATIN when you pay online.
    Quick Searches
    Discover restaurants by type of meal
    Delivery
    Breakfast
    Lunch
    Dinner
    Drinks & Nightlife
    Cafés
    Pocket-Friendly Delivery
    Desserts & Bakes
    Popular localities in and around Kolkata
    Explore restaurants, bars, and cafés by locality
    Park Street Area (111 places)
    Sector 5, Salt Lake (102 places)
    Ballygunge (154 places)
    Sector 1, Salt Lake (167 places)
    Rajarhat New Town (212 places)
    Southern Avenue (86 places)
    Elgin (81 places)
    Prince Anwar Shah Road (108 places)
    Kankurgachi (116 places)
    Kasba (118 places)
    Camac Street Area (61 places)
    Gariahat (62 places)
    Park Circus Area (80 places)
    Desapriya Park (61 places)
    New Market Area (105 places)
    Behala (136 places)
    Hindustan Park (30 places)
    Hatibagan (70 places)
    Sector 3, Salt Lake (83 places)
    Esplanade (64 places)
    Jadavpur (91 places)
    Golpark (37 places)
    Bhawanipur (100 places)
    Science City Area (19 places)
    Theatre Road (47 places)
    Shyam Bazar (54 places)
    Garia (108 places)
    Tangra (32 places)
    Nagerbazar (60 places)
    Tollygunge (110 places)
    Top Reviewers in Kolkata
    Pamela Nandi
    446 Reviews , 245 Followers
    Follow
    Dipyaman Basu
    883 Reviews , 5854 Followers
    Follow
    Anusreea Paul
    613 Reviews , 2114 Followers
    Follow
    Krishanu Das
    832 Reviews , 1599 Followers
    Follow
    Priyabrataa Ganguly
    536 Reviews , 683 Followers
    Follow
    Reviewers leaderboard »
    Top Photographers in Kolkata
    Avijit Biswas
    698 Reviews , 1381 Followers
    Follow
    Krishanu Das
    832 Reviews , 1599 Followers
    Follow
    Mani
    153 Reviews , 4473 Followers
    Follow
    Anusreea Paul
    613 Reviews , 2114 Followers
    Follow
    Anamitraa Chakraborty
    610 Reviews , 1255 Followers
    Follow
    Photographers leaderboard »
    Top Bloggers in Kolkata
    Arghya Deep
    137 Reviews , 3498 Followers
    Follow
    Subham Ghosh
    208 Reviews , 6497 Followers
    Follow
    Snehasis
    230 Reviews , 929 Followers
    Follow
    Koninika De
    356 Reviews , 4695 Followers
    Follow
    Dipyaman Basu
    883 Reviews , 5854 Followers
    Follow
    Bloggers leaderboard»
    Looking for the Food Feed? Get the app!
    Follow foodies to see their reviews and photos in your Feed, and discover great new restaurants!
    We'll send you a link, open it on your phone to download the app
     Text App Link 
    OR
    Email App Link
    23
    COUNTRIES
    1.2M
    RESTAURANTS
    80M
    FOODIES EVERY MONTH
    30M
    PHOTOS
    10M
    REVIEWS
    18M
    BOOKMARKS
    About Careers Culture Mobile Apps Businesses Developers Blog Community Contact
    English
    Businesses
    Add a Restaurant
    Claim your Listing
    Business App
    Restaurant Widgets
    Guidelines
    Business Blog
    Advertise
    Book
    Order
    Base
    Whitelabel
    Countries
    Australia
    Brasil
    Canada
    Chile
    Czech Republic
    India
    Indonesia
    Ireland
    Italy
    Lebanon
    Malaysia
    New Zealand
    Philippines
    Poland
    Portugal
    Qatar
    Slovakia
    South Africa
    Sri Lanka
    Turkey
    UAE
    United Kingdom
    United States
    Privacy Terms Code of Conduct API Policy CSR Security Sitemap
    By continuing past this page, you agree to our Terms of Service, Cookie Policy, Privacy Policy and Content Policies. All trademarks are properties of their respective owners. © 2008-2016 - Zomato™ Media Pvt Ltd. All rights reserved.
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2011-08-23
      • 2016-03-21
      • 1970-01-01
      • 2014-08-16
      相关资源
      最近更新 更多