我想抓取这些产品信息

Question

import requests
from bs4 import BeautifulSoup
import pandas as pd
baseurl='https://twillmkt.com'
headers ={
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r =requests.get('https://twillmkt.com/collections/denim')
soup=BeautifulSoup(r.content, 'html.parser')
tra = soup.find_all('div',class_='ProductItem__Wrapper')
productlinks=[]
Title=[]
Brand=[]
Colour=[]
for links in tra:
    for link in links.find_all('a',href=True)[1:]:
        comp=baseurl+link['href']
        productlinks.append(comp)
for link in productlinks:
    r =requests.get(link,headers=headers)
    soup=BeautifulSoup(r.content, 'html.parser')
    title=soup.find('h1').text
    Title.append(title)
    price=soup.find('span',class_="money").text
    Brand.append(price)
    Product_Features=soup.find_all('li').text 
    Colour.append(Product_Features)
    

df = pd.DataFrame(
    
    {"Title": Title, "Price": Brand,"Product_Features":Colour}
)
print(df)

我刮了标题和价格但是很难刮这些信息SIZE,product feature,material,model size,image 单页链接 https://twillmkt.com/products/light-blue-butterfly-print-slim-leg-denim?variant=39498848403534

Answer 1

我将为您提供所需元素的 css 选择器。您可以通过soup.select() 捕获它们

大小：select[id^="product-select"] option

产品功能：div.ProductMeta__Description.Rte li

素材：div.ProductMeta__Description.Rte p:nth-of-type(2)

模型尺寸：div.ProductMeta__Description.Rte p:nth-of-type(3)

图片：div[id^="Image"] span img