【问题标题】:Scraping html for img src in Python bs4在 Python bs4 中为 img src 抓取 html
【发布时间】:2021-01-24 05:03:23
【问题描述】:

我尝试用 Python 或 bs4 中的 BeautifulSoup 解析以下 HTML 代码:

  <div class="product w-100" data-pid="BBOMNLV1-36183" data-sid="BBOMNLWB">
        <div class="product-tile w-100">
            <!-- dwMarker="product" dwContentID="c4e921241579720afa4287dbf5" -->
            <div class="image-container">
                <a href="/pd/omn1s-low/BBOMNLV1-36183.html?dwvar_BBOMNLV1-36183_style=BBOMNLWB">
                    <picture>
                        <source type="image/jpeg" data-srcset="https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&amp;wid=440&amp;hei=440 1x, https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&amp;wid=880&amp;hei=880 2x" srcset="https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&amp;wid=440&amp;hei=440 1x, https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&amp;wid=880&amp;hei=880 2x"> <img class="tile-image ls-is-cached lazyloaded" src="https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&amp;wid=440&amp;hei=440" data-src="https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&amp;wid=440&amp;hei=440" data-srcset="https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2SM$&amp;wid=440&amp;hei=440 1x, https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2SM$&amp;wid=880&amp;hei=880 2x" alt="OMN1S Low" title="OMN1S Low, BBOMNLWB" itemprop="image" srcset="https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2SM$&amp;wid=440&amp;hei=440 1x, https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2SM$&amp;wid=880&amp;hei=880 2x"> </picture>
                </a>
                <div class="product-id d-none">BBOMNLV1-36183</div>
                <div class="wishlist-url d-none">/on/demandware.store/Sites-NBUS-Site/en_US/Wishlist-WishlistItemExists</div> <span class="wishListToggle">
                <a class="wishlistTile add-to-wish-list" href="/on/demandware.store/Sites-NBUS-Site/en_US/Wishlist-AddProduct" title="Wish list">
                    <span class="wishlist-inactive active">
    <svg role="img" class="icon svg-icon " width="24" height="24" aria-label="title">
    <title> </title>
    <desc> </desc>
    <use xlink:href="#wishlist-inactive"></use>
    </svg></span> </a>
                <a class="wishlistTile remove-from-wishlist" href="/on/demandware.store/Sites-NBUS-Site/en_US/Wishlist-RemoveProduct" title="Wish list"> <span class="wishlist-active ">
    <svg role="img" class="icon svg-icon " width="24" height="24" aria-label="title">
    <title> </title>
    <desc> </desc>
    <use xlink:href="#wishlist-active"></use>
    </svg></span> </a>
                </span>
            </div>
            <div class="tile-body">
                <div class="row pgp-grid pb-2 pr-2">
                    <div class="col-12 col-lg-7 pl-2 fw-search">
                        <div class="pdp-link"> <a class="link font-weight-bold pname text-underline no-underline-lg" href="/pd/omn1s-low/BBOMNLV1-36183.html?dwvar_BBOMNLV1-36183_style=BBOMNLWB">OMN1S Low</a> <span class="category-name font-body w-100 d-block pt-2">
            
                Men's Basketball
            
        </span> </div>
                    </div>
                    <div class="col-12 col-lg-5 pl-2 fw-search justify-content-lg-end text-right d-flex p-0 search-tile">
                        <div class="price"> <span class="price-value">
        
    
        
        <span class="sales font-body-large ">
            
            
            
            $139.99
    
    
        </span> </span>
                        </div>
                    </div>
                </div>
                <div class="pgp-reviews-wrapper" data-pageid="BBOMNLV1-36183" data-url="https://www.newbalance.com/on/demandware.store/Sites-NBUS-Site/en_US/ProductReviews-WriteReview?pid=BBOMNLV1-36183" id="BBOMNLV1-36183-pgp-reviews-wrapper-3">
                    <div class="p-w-r">
                        <section id="pr-category-snippets-BBOMNLV1-36183" class="pr-no-reviews" aria-labelledby="pr-UbCtutN-xQJECAE6zEJSy" data-testid="category-snippet">
                            <div class="pr-snippet pr-category-snippet">
                                <div class="pr-category-snippet__rating pr-category-snippet__item">
                                    <div class="pr-snippet-stars pr-snippet-stars-png ">
                                        <div aria-hidden="true" class="pr-rating-stars">
                                            <div class="pr-star-v4 pr-star-v4-0-filled"></div>
                                            <div class="pr-star-v4 pr-star-v4-0-filled"></div>
                                            <div class="pr-star-v4 pr-star-v4-0-filled"></div>
                                            <div class="pr-star-v4 pr-star-v4-0-filled"></div>
                                            <div class="pr-star-v4 pr-star-v4-0-filled"></div>
                                        </div>
                                        <div aria-hidden="true" class="pr-snippet-rating-decimal">0.0</div>
                                    </div><span id="pr-UbCtutN-xQJECAE6zEJSy" class="pr-accessible-text">Rated 0 out of 5 stars</span></div>
                                <div class="pr-category-snippet__total pr-category-snippet__item">No Reviews</div>
                            </div>
                        </section>
                    </div>
                </div>
            </div>
            <div class="badges"> <span class="sub-badges p-1 text-uppercase font-weight-bold">NEW</span> </div>
            <!-- END_dwmarker -->
        </div>
    </div>

我正在尝试通过查找具有“tile-image ls-is-cached lazyloaded”类的 img 标签来检索鞋子的 pciture,然后我尝试检索 data-src 属性以获取照片的链接。

这是我的 bs4 代码,它似乎不起作用:

from bs4 import BeautifulSoup
def queryNewBalance(uri):
    r = requests.get('https://www.newbalance.com/men/shoes/basketball/?prefn1=color&prefv1=Black%7CBlue&srule=null')
    soup = BeautifulSoup(r.content, 'html.parser')
    result = soup.find_all('div', class_='product w-100')
    for res in result:
        print("*******************************")
        print(res.find('img', class_='tile-image ls-is-cached lazyloaded')['href]) #Picture
        print("*******************************")
    print(f"\nFound total shoes: {len(result)}")

如何修复我的代码以检索图片链接?

【问题讨论】:

  • hmmm.. 你正在做一个import bs4 我认为它应该是from bs4 import BeautifulSoup 或者你可能只是没有在这里粘贴它
  • @IceBear 已编辑

标签: python html python-3.x web-scraping beautifulsoup


【解决方案1】:

页面上没有课程tile-image ls-is-cached lazyloaded。要获取图片的链接,可以使用 CSS 选择器img[itemprop='image']

import requests
from bs4 import BeautifulSoup

def queryNewBalance(url):
    r = requests.get(url)
    soup = BeautifulSoup(r.content, "html.parser")
    result = soup.find_all("div", class_="product w-100")
    for res in result:
        print("*******************************")
        print(res.select_one("img[itemprop='image']")["data-src"])
    print(f"\nFound total shoes: {len(result)}")


queryNewBalance(
    "https://www.newbalance.com/men/shoes/basketball/?prefn1=color&prefv1=Black%7CBlue&srule=null"
)

输出:

*******************************
https://nb.scene7.com/is/image/NB/bbomnxbb_nb_02_i?$pdpflexf2$&wid=440&hei=440
*******************************
https://nb.scene7.com/is/image/NB/bbomnlpl_nb_02_i?$pdpflexf2$&wid=440&hei=440
*******************************
https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&wid=440&hei=440
*******************************
https://nb.scene7.com/is/image/NB/bbomnlbr_nb_02_i_5a34b3da900d437a9a88?$pdpflexf2$&wid=440&hei=440
*******************************
https://nb.scene7.com/is/image/NB/bbomnlfc_nb_02_i?$pdpflexf2$&wid=440&hei=440
*******************************
https://nb.scene7.com/is/image/NB/bbomnlwt_nb_02_i?$pdpflexf2$&wid=440&hei=440

【讨论】:

    【解决方案2】:

    看来你得到的属性是href&lt;img&gt;你试图刮的标签没有attribute,它有srcattribute,这就是链接所在的位置.顺便把你提供的html 参数放这么长的html 代码。

    def queryNewBalance(html):
        #r = requests.get('https://www.newbalance.com/men/shoes/basketball/?prefn1=color&prefv1=Black%7CBlue&srule=null')
        soup = BeautifulSoup(html, 'html.parser')
        result = soup.find_all('div', class_='product w-100')
        for res in result:
            print("*******************************")
            print(res.find('img', class_='tile-image ls-is-cached lazyloaded')['src']) #Picture
            print("*******************************")
        print(f"\nFound total shoes: {len(result)}")
    
    
    
    queryNewBalance(html)
    

    输出

    *******************************
    https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&wid=440&hei=440
    *******************************
    
    Found total shoes: 1
    [Finished in 0.7s]
    

    ---网址---

    from bs4 import BeautifulSoup
    import requests
    
    def queryNewBalance():
        r = requests.get('https://www.newbalance.com/men/shoes/basketball/?prefn1=color&prefv1=Black%7CBlue&srule=null')
        soup = BeautifulSoup(r.content, 'html.parser')
        result = soup.find_all('div', class_='product w-100')
        for res in result:
            print("*******************************")
            print(res.find('img', class_='tile-image')["data-src"]) #Picture
            print("*******************************")
        print(f"\nFound total shoes: {len(result)}")
    
    
    
    queryNewBalance()
    

    输出:

    *******************************
    https://nb.scene7.com/is/image/NB/bbomnxbb_nb_02_i?$pdpflexf2$&wid=440&hei=440
    *******************************
    *******************************
    https://nb.scene7.com/is/image/NB/bbomnlpl_nb_02_i?$pdpflexf2$&wid=440&hei=440
    *******************************
    *******************************
    https://nb.scene7.com/is/image/NB/bbomnlwb_nb_02_i?$pdpflexf2$&wid=440&hei=440
    *******************************
    *******************************
    https://nb.scene7.com/is/image/NB/bbomnlbr_nb_02_i_5a34b3da900d437a9a88?$pdpflexf2$&wid=440&hei=440
    *******************************
    *******************************
    https://nb.scene7.com/is/image/NB/bbomnlfc_nb_02_i?$pdpflexf2$&wid=440&hei=440
    *******************************
    *******************************
    https://nb.scene7.com/is/image/NB/bbomnlwt_nb_02_i?$pdpflexf2$&wid=440&hei=440
    *******************************
    
    Found total shoes: 6
    [Finished in 2.9s]
    

    PS: 如果您更多地参与网络抓取,并且抓取大量网站,尤其是大型网站,我建议您将解析器更改为 html5lib -> pip install html5lib。这是一个更好的解析器,因为我在使用 html.parser 抓取时遇到了问题,它只是无法以某种方式抓取网站的某些部分,尽管我已经检查了它所在的汤对象,无论如何你的电话,祝你好运!

    【讨论】:

      猜你喜欢
      • 2015-10-03
      • 2021-07-08
      • 2022-09-28
      • 1970-01-01
      • 2018-11-08
      • 2013-12-04
      • 1970-01-01
      • 2016-01-08
      • 1970-01-01
      相关资源
      最近更新 更多