无法从 python 漂亮的抓取中找到内容

Question

我正在尝试抓取 this page 并获取每篇文章标题的 URL，它是一个 'h3' 'a' 元素，例如第一个结果是带有文本“全长小鼠 cDNA 集合的功能注释”的链接，链接到 this page。

我的搜索返回的都是'[]'

我的代码如下：

import requests
from bs4 import BeautifulSoup
req = requests.get('https://www.lens.org/lens/scholar/search/results?q="edith%20cowan"')
soup = BeautifulSoup(req.content, "html5lib")
article_links = soup.select('h3 a')
print(article_links)

我哪里错了？

Answer 1

您正在处理此问题，因为您使用了错误的链接来获取文章链接。所以我并没有做太多改动，而是想出了这段代码（请注意，我删除了 bs4 模块，因为它不再需要了）：

import requests

search = "edith cowan"

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}

json = {"scholarly_search":{"from":0,"size":"10","_source":{"excludes":["referenced_by_patent_hash","referenced_by_patent","reference"]},"query":{"bool":{"must":[{"query_string":{"query":f"\"{search}\"","fields":["title","abstract","default"],"default_operator":"and"}}],"must_not":[{"terms":{"publication_type":["unknown"]}}],"filter":[]}},"highlight":{"pre_tags":["<span class=\"highlight\">"],"post_tags":["</span>"],"fields":{"title":{}},"number_of_fragments":0},"sort":[{"referenced_by_patent_count":{"order":"desc"}}]},"view":"scholar"}

req = requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true", headers = headers, json = json).json()

links = []
for x in req["query_result"]["hits"]["hits"]:
    links.append("https://www.lens.org/lens/scholar/article/{}/main".format(x["_source"]["record_lens_id"]))

search 变量等于您正在搜索的字词（在您的情况下为 "edith cowan"）。链接存储在links 变量中。

---

编辑：我是怎么做到的

所以主要问题可能是我从哪里获得链接以及我如何知道要包含在json 变量中的内容。为此，我使用了一个简单的 HTML 拦截器 （在我的情况下为 burp suite community edition）。

这个工具告诉我，当你访问这个URL（你在问题中用来发送请求的那个）你的浏览器会向https://www.lens.org/lens/api/multi/search?request_cache=true发送一个发布请求，然后检索current page 的所有信息。 json 变量 burp 套件的相关问题还向您显示发送了哪些数据包，因此我将它们复制粘贴到 json 变量中。

为了更好地可视化，这就是它在 burp 套件中的外观：

---

编辑：扫描所有页面

为了扫描所有页面，您可以使用以下脚本：

import requests

search = "edith cowan" #Change this to the term you are searching for
r_to_show = 100 #This is the number of articles per page (I strongly recommend leaving it at 100)

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}

json = {"scholarly_search":{"from":0,"size":f"{r_to_show}","_source":{"excludes":["referenced_by_patent_hash","referenced_by_patent","reference"]},"query":{"bool":{"must":[{"query_string":{"query":f"\"{search}\"","fields":["title","abstract","default"],"default_operator":"and"}}],"must_not":[{"terms":{"publication_type":["unknown"]}}],"filter":[]}},"highlight":{"pre_tags":["<span class=\"highlight\">"],"post_tags":["</span>"],"fields":{"title":{}},"number_of_fragments":0},"sort":[{"referenced_by_patent_count":{"order":"desc"}}]},"view":"scholar"}

req = requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true", headers = headers, json = json).json()

links = [] #links are stored here
count = 0

#link_before and link_after helps determine when to stop going to the next page 
link_before = 0
link_after = 0

while True:
    json["scholarly_search"]["from"] += r_to_show
    if count > 0:
        req = requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true", headers = headers, json = json).json() 
    for x in req["query_result"]["hits"]["hits"]:
        links.append("https://www.lens.org/lens/scholar/article/{}/main".format(x["_source"]["record_lens_id"]))
    count += 1
    link_after = len(links)
    if link_after == link_before:
        break
    link_before = len(links)
    print(f"page {count} done, links recorder {len(links)}") 

我在代码中添加了一些 cmets 以使其更易于理解。