【问题标题】:Webscraping with Scrapy - spyder error processing POST使用 Scrapy 进行 Web Scraping - 蜘蛛错误处理 POST
【发布时间】:2021-12-01 11:48:35
【问题描述】:

我正在尝试从这个网站获取数据:https://www.illibraio.it/librerie/,数据似乎专门存储在多个不同长度的https://www.illibraio.it/wp-admin/admin-ajax.php 页面中。

我按照教程编写了这个简短的代码,但我不明白为什么它不起作用。我不断收到此错误消息

ERROR: Spider error processing <POST https://www.illibraio.it/wp-admin/admin-ajax.php> (referer: https://www.illibraio.it/librerie/)

这是我当前的参考代码,感谢任何帮助。这东西我是初学者,但我需要网站上的数据...

import scrapy
import json

class Librerie(scrapy.Spider):
    name = "Librerie"
    start_urls = ['https://www.illibraio.it/librerie/']
    headers = {
        "Accept": "application/json, text/javascript, */*; q=0.01",
        "Accept-Encoding": "gzip, deflate, br",
        "Accept-Language": "en-US,en;q=0.9",
        "Referer": "https://www.illibraio.it/librerie/",
        "Sec-Fetch-Mode": "cors",
        "Sec-Fetch-Site": "same-origin",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
        "X-Requested-With": "XMLHttpRequest",
    }
    
    data = {
        'nome': 'Mondadori Bookstore - Melegnano',
        'indirizzo': 'Via Zuavi, 8 20077 Melegnano MI',
        'latlong': '{"lat":"45.356655","lng":"9.322607"'
        }
    
    def parse(self, response):
        url = 'https://www.illibraio.it/wp-admin/admin-ajax.php'
        meta = {'handle_httpstatus_all': True}
        yield scrapy.FormRequest(url,
                                 method='POST',
                                 formdata=self.data,
                                 meta=meta,
                                 callback=self.parse_api, 
                                 headers=self.headers)
    
    def parse_api(self, response):
        raw_data = response.body
        data = json.loads(raw_data)
        yield {
            'nome' : data['title'],
            'indirizzo' : data['location'],
            'latlong' : data['coords'],
            }

【问题讨论】:

    标签: python web-scraping scrapy


    【解决方案1】:

    如果您不反对使用请求模块。可能对你有帮助

    import requests
    from bs4 import BeautifulSoup
    import time
    import requests
    import json
    
    cookies = {
    '_ga': 'GA1.2.1385472977.1639209109',
    '_gid': 'GA1.2.979738495.1639209109',
    '_fbp': 'fb.1.1639209110499.1494824473',
    '_gat_UA-30151893-1': '1',
    }
    
    headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0',
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate, br',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'X-Requested-With': 'XMLHttpRequest',
    'Origin': 'https://www.illibraio.it',
    'Connection': 'keep-alive',
    'Referer': 'https://www.illibraio.it/librerie/',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-origin',
    'TE': 'trailers',
    }
    
    
    for req in range(0, 110, 10):
        data = {
        'action': 'get_bookshops',
        'offset': str(req),
        'search': 'prov=&city-lat=&city-lng=&keywords='
        }
    
        response = requests.post('https://www.illibraio.it/wp-admin/admin-ajax.php', headers=headers, cookies=cookies, data=data)
        print(response.status_code)
        data = json.loads(response.text)
        for mydata in data["results"]:
            print(mydata['title'])
            print(mydata['location'])
            print(mydata['phone'])
    
        
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2011-02-20
      • 2018-10-19
      • 1970-01-01
      • 2010-12-20
      • 1970-01-01
      相关资源
      最近更新 更多