【问题标题】:Json. post and json.get give no response when web scraping杰森。网络抓取时 post 和 json.get 没有响应
【发布时间】:2021-10-28 03:04:39
【问题描述】:

所以我试图从这个站点获取所有在英国可用的工作:https://www.ubisoft.com/en-us/company/careers/search?countries=gb ,当进入网络设置时,有一个包含所需数据的 json 文件https://avcvysejs1-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20JavaScript%20(4.8.4)%3B%20Browser%20(lite)%3B%20JS%20Helper%20(3.3.4)%3B%20react%20(16.12.0)%3B%20react-instantsearch%20(6.8.3)&x-algolia-api-key=1291fd5d5cd5a76a225fc6b00f7b296a&x-algolia-application-id=AVCVYSEJS1,它使用请求方法:POST

但是当我编写脚本来获取该数据时

    data = []
    url = "https://avcvysejs1-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20JavaScript%20(4.8.4)%3B%20Browser%20(lite)%3B%20JS%20Helper%20(3.3.4)%3B%20react%20(16.12.0)%3B%20react-instantsearch%20(6.8.3)&x-algolia-api-key=1291fd5d5cd5a76a225fc6b00f7b296a&x-algolia-application-id=AVCVYSEJS1"
    r = requests.post(url)
    json = r.json()
    print(json)

但得到结果

{'message': 'No content in POST request', 'status': 400}

当我将其更改为 r = requests.post(url) 时,我得到了结果

{'message': 'indexName is not valid', 'status': 400}

【问题讨论】:

    标签: python json web-scraping python-requests


    【解决方案1】:

    要从服务器获得正确的响应,请将有效负载与请求一起发送:

    
    import requests
    from bs4 import BeautifulSoup
    
    api_url = "https://avcvysejs1-dsn.algolia.net/1/indexes/*/queries"
    
    params = {
        "x-algolia-agent": "Algolia for JavaScript (4.8.4); Browser (lite); JS Helper (3.3.4); react (16.12.0); react-instantsearch (6.8.3)",
        "x-algolia-api-key": "1291fd5d5cd5a76a225fc6b00f7b296a",
        "x-algolia-application-id": "AVCVYSEJS1",
    }
    
    payload = """{"requests":[{"indexName":"jobs_en-us_default","params":"highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&query=&maxValuesPerFacet=100&page=0&facets=%5B%22jobFamily%22%2C%22team%22%2C%22countryCode%22%2C%22city%22%2C%22contractType%22%2C%22graduateProgram%22%5D&tagFilters=&facetFilters=%5B%5B%22countryCode%3Agb%22%5D%5D"},{"indexName":"jobs_en-us_default","params":"highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&query=&maxValuesPerFacet=100&page=0&hitsPerPage=1&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=countryCode"}]}"""
    
    data = requests.post(api_url, params=params, data=payload).json()
    
    for h in data["results"][0]["hits"]:
        print(h["title"])
        print(
            BeautifulSoup(h["additionalInformation"], "html.parser").get_text(
                strip=True, separator=" "
            )
        )
        print("-" * 80)
    

    打印:

    Lead Concept Artist [New IP] (433)
    Benefits & Relocation Flexible working, 22 days annual leave + Christmas shutdown, private healthcare (with option to add immediate family), life insurance & income protection, workplace pension scheme, paid volunteering days, annual fitness & well-being allowance, games, technology & merchandise, subsidised travel and many more... Relocation assistance is available to anyone currently living 50 miles or more from the studio location. Please contact a member of the talent acquisition team to find out what we have to offer and how we can support with your move here... relocation really doesn't have to be a daunting prospect. Find out more about Ubisoft Reflections: https://reflections.ubisoft.com/about/ubisoft-reflections/ Facebook: https://www.facebook.com/pg/Ubisoft.Reflections Twitter: https://twitter.com/UbiReflections Ubisoft offers the same job opportunities to all, without any distinction of gender, ethnicity, religion, sexual orientation, social status, disability or age. Ubisoft ensures the development of an inclusive work environment which mirrors the diversity of our gamers community.
    --------------------------------------------------------------------------------
    Player Support Product Lead
    Benefits With Ubisoft CRC, you'll receive a competitive salary along with: Personal performance bonus Private Health Insurance (including eye care and dental) Life Assurance Long Term Disability Insurance Pension Significant discount on the world’s best video games Access to Ubisoft's back catalogue on PC Perks: We work in the heart of Newcastle city centre, right on top of Haymarket metro station in a lively, international and creative space. We have a kitchen stocked with  cereals, fruits, unlimited filtered water, teas, coffee Regular professional and social events Monthly Ubidrinks Flexible working hours A casual dress code Fun, we like to work hard but have a laugh too! For the safety of all our teams we are currently working remotely. We hope to return to our CRC home very soon and anticipate a blended working pattern combining office and home based working in the future Ubisoft is committed to creating an inclusive work environment that reflects the diversity of our player community. We are an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to their race, ethnicity, religion, gender, sexual orientation, age or disability status.
    --------------------------------------------------------------------------------
    
    ...and so on.
    

    【讨论】:

      猜你喜欢
      • 2011-05-18
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2022-08-19
      • 1970-01-01
      • 2017-11-14
      • 1970-01-01
      相关资源
      最近更新 更多