使用 Python 抓取产品类别

Question

我正在尝试抓取这个页面，它有大约 21000 种产品

我的问题是如何获得 21000 个产品的所有产品名称、图像和完整的类别层次结构。 图片和名称在同一页面上，但类别在实际产品页面内。

由于分页，我只能获得第一页上显示的 32 个产品标题和图像

从首页获取标题的代码

import requests
from bs4 import BeautifulSoup

main_url = "https://paytmmall.com/fmcg-foods-glpid-101405?discoverability=online&use_mw=1"

import requests
result = requests.get(main_url)
print(result.text)

sp = BeautifulSoup(result.text,'html.parser')
print(sp.prettify())

getallTitle = [x.a.get('title') for x in sp.findAll("div", class_ = "_3WhJ")]

print(str(len(getallTitle )) + " fetched products Title")
print("/n")
print(getallTitle[2])

Answer 1

您可以访问每个页面的 json 响应。但请记住，每页只有 32 个产品，这意味着您将请求 659 次。

import requests
import math

url = 'https://middleware.paytmmall.com/fmcg-foods-glpid-101405'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'}

payload = {
'channel': 'web',
'child_site_id': '6',
'site_id': '2',
'version': '2',
'discoverability': 'online',
'use_mw': '1',
'category': '101405',
'page': '1',
'page_count': '1',
'items_per_page': '32'}

# Get total pages needed
jsonData = requests.post(url, headers=headers, data=payload).json()
total_count = jsonData['totalCount']
total_pages = total_count / 32
pages = math.ceil(total_pages)


# Iterate through each page
for page in range(1,pages + 1):
    payload.update({'page':page, 'page_count':page})

    jsonData = requests.post(url, headers=headers, data=payload).json()

    for product in jsonData['grid_layout']:
        name = product['name']
        brand = product['brand']
        actual_price = product['actual_price']
        try:
            category = product['attributes']['type']
        except:
            category = 'N/A'

        print ('%-20s ₹%-5s %-20s ₹%s' %(category, actual_price, brand, name))

输出：

Tea                  ₹185   Red Label            Red Label Tea 500 gm
Tea                  ₹93    Tata Tea Premium     Tata Tea Premium Leaf 250 gm
Tea                  ₹240   Red Label            Red Label Natural Care Tea 500 gm
N/A                  ₹230   Taj Mahal            Taj Mahal Tea 500 gm
Tea                  ₹120   Red Label            Red Label Natural Care Tea 250 gm
Dairy Whitener       ₹413   Nestle               Nestle Everyday Dairy Whitener Milk 1 kg
Sauces               ₹125   Kissan               Kissan Fresh Tomato Ketchup 950 gm
Whole Oats           ₹186   Quaker               Quaker Oats 1 kg Pouch
Tea                  ₹188   Tata Tea Premium     Tata Tea Premium Leaf 500 gm
Coffee               ₹90    Bru                  BRU Instant Coffee 50 gm
Almond               ₹300   Freshco              Freshco California Almonds 200Gm
Jam                  ₹250   Kissan               Kissan Mixed Fruit Jam 1.04 kg
Almond               ₹799   glomin               Glomin California Almond Raw 500 G 1Pc
Sauces               ₹152   Kissan               Kissan Sweet & Spicy Sauce 1 kg
Cashew Nut           ₹180   Nutty Gritties       Nutty Gritties Roasted Salted Cashews 80G
Coffee               ₹120   Bru                  BRU Gold Instant Coffee 50 gm
Tea                  ₹480   Red Label            Red Label Natural Care Tea 1 kg
Almond               ₹310   Miltop               Miltop California Almonds 250G
Cashew Nut           ₹425   glomin               Glomin Cashew 250 G 1Pc
Almond               ₹600   Wonderland           Wonderland California Almond 500g
Almond               ₹499   Shivram Peshawari & Bros Shivram Peshawari & Bros California Almonds/Badam 250 Grams
Peanut Butter        ₹425   Pintola              Pintola All Natural Peanut Butter 1 kg (Crunchy)
Soups                ₹55    Knorr                Knorr Classic Tomato Soup 53 gm
Peanut Butter        ₹425   Pintola              Pintola All Natural Peanut Butter 1 kg (Creamy)
Peanut Butter        ₹349   Pintola              Pintola Classic Peanut Butter 1 kg (Crcuncy)
Peanut Butter        ₹165   Pintola              Pintola All Natural Peanut Butter 350 gm (Crunchy)
Almond               ₹1599  glomin               Glomin Raw Almonds 1Kg (Pack Of 1)
Almond               ₹150   Nutty Gritties       Nutty Gritties Almonds 100G
Raisin               ₹250   OOSH                 Oosh Seedless Black Raisin 250G
N/A                  ₹455   Taj Mahal            Taj Mahal Tea 1 kg

编辑：

如果您想要层次结构，则需要转到每个产品的链接并将其拉出。我提供了代码来做到这一点，但请记住，这需要永远。假设每个请求大约需要 2-3 秒，那么您将花费将近 18 个小时。

# Iterate through each page
for page in range(1,pages + 1):
    payload.update({'page':page, 'page_count':page})

    jsonData = requests.post(url, headers=headers, data=payload).json()

    for product in jsonData['grid_layout']:
        name = product['name']
        brand = product['brand']
        actual_price = product['actual_price']
        img = product['image_url']
        category_id = product['category_id']

        new_url = product['newurl']

        jsonData_product = requests.get(new_url, headers=headers).json()

        category = '/'.join( [each['name'] for each in jsonData_product['ancestors'] ] )

        print ('Name: %s\nImage: %s\nCategory: %s\n' %(name, img, category))

输出：

Name: Red Label Tea 500 gm
Image: https://assetscdn1.paytm.com/images/catalog/product/F/FA/FASRED-LABEL-TETBL497475164B959/a_4.jpg
Category: Supermarket/Foods/Drinks & Beverages/Tea & Coffee/Red Label Tea 500 gm

Name: Tata Tea Premium Leaf 250 gm
Image: https://assetscdn1.paytm.com/images/catalog/product/F/FA/FASTATA-TEA-PREINNO985832A1E145F5/8.jpg
Category: Supermarket/Foods/Drinks & Beverages/Tea & Coffee/Tata Tea Premium Leaf 250 gm

Name: Red Label Natural Care Tea 500 gm
Image: https://assetscdn1.paytm.com/images/catalog/product/F/FA/FASRLNC-C-500GNTBL4974726639099/a_14.jpg
Category: Supermarket/Foods/Drinks & Beverages/Tea & Coffee/Red Label Tea & Coffee 500 Gm

Name: Taj Mahal Tea 500 gm
Image: https://assetscdn1.paytm.com/images/catalog/product/F/FA/FASTAJ-MAHAL-TEBIGB985832F0512392/0.jpg
Category: Supermarket/Foods/Drinks & Beverages/Tea & Coffee/Taj Mahal Tea 500 gm

Name: Red Label Natural Care Tea 250 gm
Image: https://assetscdn1.paytm.com/images/catalog/product/F/FA/FASNEW-RED-LABETBL49747FC4B364F/a_7.jpg
Category: Supermarket/Foods/Drinks & Beverages/Tea & Coffee/Red Label Natural Care Tea 250 gm

Name: Nestle Everyday Dairy Whitener Milk 1 kg
Image: https://assetscdn1.paytm.com/images/catalog/product/F/FA/FASNESTLE-EVERYTBL497478E1F2966/a_8.jpg
Category: Supermarket/Foods/Dairy Products/Dairy Whitener/Nestle Everyday Dairy Whitener Milk 1 kg

或

如果所有产品都在同一个类别下，那么您只需要获取第一个产品的类别，然后在您遍历页面时应用到所有其他产品：

import requests
import math

url = 'https://middleware.paytmmall.com/fmcg-foods-glpid-101405'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'}

payload = {
'channel': 'web',
'child_site_id': '6',
'site_id': '2',
'version': '2',
'discoverability': 'online',
'use_mw': '1',
'category': '101405',
'page': '1',
'page_count': '1',
'items_per_page': '32'}

# Get total pages needed
jsonData = requests.post(url, headers=headers, data=payload).json()
total_count = jsonData['totalCount']
total_pages = total_count / 32
pages = math.ceil(total_pages)


# Iterate through each page
category = ''
for page in range(1,pages + 1):
    payload.update({'page':page, 'page_count':page})

    jsonData = requests.post(url, headers=headers, data=payload).json()

    for product in jsonData['grid_layout']:
        name = product['name']
        brand = product['brand']
        actual_price = product['actual_price']
        img = product['image_url']
        category_id = product['category_id']

        if category == '':
            new_url = product['newurl']
            jsonData_product = requests.get(new_url, headers=headers).json()
            category = '/'.join( [each['name'] for each in jsonData_product['ancestors'] ][:-1] )

        print ('Name: %s\nImage: %s\nCategory: %s\n' %(name, img, category))

Answer 2

页面请求第一页的内容如下（返回 json）。看看是否可以更改参数以获得所有结果

看起来您可以通过更改 url 以包含页面来更改引用标题和正文中的当前页面，例如

https://paytmmall.com/fmcg-foods-glpid-101405?discoverability=online&use_mw=1&page=2

您可以从您的第一个请求中提取总结果数

r['filters'][0]['values'][0]['count']

您知道您是分批请求 32 个（尽管尝试将其增加到可能的最大值）。然后您可以计算页面/请求的数量，然后循环发出。

Python（第 1 页请求）

import requests

headers = {
    'Content-Type' : 'application/json',
    'Referer' : 'https://paytmmall.com/fmcg-foods-glpid-101405?discoverability=online&use_mw=1',
    'User-Agent' : 'Mozilla/5.0'
}

body = {"tracking":{"current_page":"https://paytmmall.com/fmcg-foods-glpid-101405?discoverability=online&use_mw=1","prev_page":''},"context":{"device":{"os":"Win32","device_type":"PC","browser_uuid":"GA1.2.105449259.1558439396","ua":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36","connection_type":"Unknown"},"channel":"WEB","user":{"ga_id":"GA1.2.105449259.1558439396","user_id":''}}}

r = requests.post('https://middleware.paytmmall.com/fmcg-foods-glpid-101405?channel=web&child_site_id=6&site_id=2&version=2&discoverability=online&use_mw=1&items_per_page=32', json = body, headers = headers).json()

Answer 3

以下是解决分页问题的方法。 分页只是按需发送请求而不是立即获取它。这意味着每次单击任何页码时，您都会看到根据网站设计的一些变化。 在您的情况下，每次单击任何页面链接时，url 查询都会发生变化。生成的 url 是

https://paytmmall.com/fmcg-foods-glpid-101405?discoverability=online&use_mw=1&category=101405&page=2

如果您继续将 page=2 更改为您想要抓取的任何页面，您将能够抓取该网站。

逻辑：

main_url = "https://paytmmall.com/fmcg-foods-glpid-101405? discoverability=online&use_mw=1&category=101405&page="

for i in range(1,totalnumberofpages):
 url = main_url+str(i)
 #you logic to scrape one url