【问题标题】:Working web crawler, suddenly not working anymore工作的网络爬虫,突然不再工作了
【发布时间】:2019-07-20 20:18:57
【问题描述】:

我关注this tutorial,代码运行良好。

现在在做了一些其他项目后,我回去想重新运行相同的代码。突然我收到一条错误消息,迫使我在汤变量中添加features="html.parser"。

所以我做了,但是现在当我运行代码时,实际上什么都没有发生。为什么会这样,我做错了什么?

我检查了我是否可能已经卸载了 beautifulsoup4 模块,但是没有,它仍然存在。我从头开始重新输入了整个代码,但似乎没有任何效果。

import requests
from bs4 import BeautifulSoup

def spider():
    url = "https://www.amazon.de/s?k=laptop+triton&__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&ref=nb_sb_noss"
    source = requests.get(url)
    plain_text = source.text
    soup = BeautifulSoup(plain_text, features="html.parser")

    for mylink in soup.findAll('img', {'class':'s-image'}):
        mysrc = mylink.get('src')
        print(mysrc)

spider()

理想情况下,我希望爬虫打印大约 10-20 行 src = "..." 相关亚马逊页面。这段代码在几个小时前就起作用了……

【问题讨论】:

    标签: python web beautifulsoup web-crawler


    【解决方案1】:

    解决方法是将headers={'User-Agent':'Mozilla/5.0'}添加到requests.get()(没有它,亚马逊不会发送正确的页面):

    import requests
    from bs4 import BeautifulSoup
    
    def spider():
        url = "https://www.amazon.de/s?k=laptop+triton&__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&ref=nb_sb_noss"
        source = requests.get(url, headers={'User-Agent':'Mozilla/5.0'})
        plain_text = source.text
        soup = BeautifulSoup(plain_text, features="html.parser")
    
        for mylink in soup.findAll('img', {'class':'s-image'}):
            mysrc = mylink.get('src')
            print(mysrc)
    
    spider()
    

    打印:

    https://m.media-amazon.com/images/I/71YPEDap2lL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81kXH-OA6tL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81kXH-OA6tL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81fyVgZuQxL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81kXH-OA6tL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81kXH-OA6tL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71VmlANJMOL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71rAT5E7DfL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71cEKKNfb3L._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/61aWXuLIEBL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71B7NyjuU9L._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81s822PQUcL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71fBKuAiQzL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71hXTUR-oRL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81-Lf6jX-OL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/81B85jUARqL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/8140E7+uhZL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/8140E7+uhZL._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71ROCddvJ2L._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/71ROCddvJ2L._AC_UL436_.jpg
    https://m.media-amazon.com/images/I/41bB8HuoBYL._AC_UL436_.jpg
    

    【讨论】:

    • 谢谢,现在可以了!但我发誓,它以前没有那个(如教程的代码中所示)Oo
    猜你喜欢
    • 2014-10-04
    • 2014-04-16
    • 1970-01-01
    • 2017-08-09
    • 2015-07-15
    • 2012-04-22
    • 2016-09-03
    • 2014-08-05
    • 2018-01-19
    相关资源
    最近更新 更多