【问题标题】:Scrapy: How to get cookies from splashScrapy:如何从飞溅中获取 cookie
【发布时间】:2018-08-01 13:31:49
【问题描述】:

我正在尝试从启动请求中获取 cookie,但我不断收到错误消息。

这是我正在使用的代码:

class P2PEye(scrapy.Spider):
    name = 'p2peyeSpider'
    allowed_domains = ['p2peye.com']
    start_urls = ['https://www.p2peye.com/platform/h9/']

    def start_requests(self):
        script = '''
        function main(splash)
          local url = splash.args.url
          assert(splash:go(url))
          assert(splash:wait(0.5))
          return {
            cookies = splash:get_cookies(),
          }
        end
        '''
        for url in self.start_urls:
            yield SplashRequest(url, callback=self.parse, endpoint='render.html',args={'wait': 1, 'lua_source': script})

    def parse(self, response):
        print(response.request.headers.getlist('Set-Cookie'))
        print(response.cookiejar)

这是我的settings.py

SPLASH_URL = 'http://127.0.0.1:8050'
CRAWLERA_ENABLED= False
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {'scrapy_splash.SplashDeduplicateArgsMiddleware': 100 }
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
COOKIES_ENABLED = True
COOKIES_DEBUG = True
SPLASH_COOKIES_DEBUG = True

response.request.headers.getlist('Set-Cookie') 的结果是[], response.cookiejar 出现错误:AttributeError: 'SplashTextResponse' object has no attribute 'cookiejar'。 那么如何才能在不出错的情况下获取 cookie?

【问题讨论】:

    标签: python scrapy splash-screen


    【解决方案1】:

    要访问response.cookiejar,您需要返回SplashJsonResponse

    尝试在你的 Lua 脚本中返回额外的字段:

    script = '''
            function main(splash)
              local url = splash.args.url
              assert(splash:go(url))
              assert(splash:wait(0.5))
              local entries = splash:history()
              local last_response = entries[#entries].response
              return {
                url = splash:url(),
                headers = last_response.headers,
                http_status = last_response.status,
                cookies = splash:get_cookies(),
                html = splash:html(),
              }
            end
            '''
    

    【讨论】:

      【解决方案2】:

      在响应下方使用 LUA 脚本将是一个带有位于关键 cookie 中的 cookie 的字典

      function main(splash)
            local url = splash.args.url
            assert(splash:go(url))
            assert(splash:wait(0.5))
            return {
              cookies = splash:get_cookies(),
            }
      end
      

      所以要访问你应该使用

      # d = requests.post('splash').json()
      print(d['cookies'])
      

      【讨论】:

        猜你喜欢
        • 1970-01-01
        • 1970-01-01
        • 2017-11-21
        • 1970-01-01
        • 2021-03-23
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        相关资源
        最近更新 更多