如何让beautifulsoup更快地获取抓取网页中的所有元素？

Question

如何更快地获得beautifulsoup 刮刀？ 这段代码看起来很慢，有什么方法可以更快？

def getNews():
        tic=time.perf_counter()
        requests_session = requests.Session()
        scrapy = requests.get('https://www.marketwatch.com/markets?mod=top_nav ').content
        product = SoupStrainer('div', {'id': 'collection__elements j-scrollElement'})
        soup = BeautifulSoup(scrapy, 'lxml')
        for div in soup.findAll('div', attrs={'class': 'collection__elements j-scrollElement'}):
            for div in div.findAll('div', attrs={'class':'article__content'}):
                for div2 in div.find_all('h3', attrs={'class':'article__headline'}):
                     for a in div2.find_all('a', href=True):
                         if a.text:
                            print(a.text)
                            print(a['href'])
        toc=time.perf_counter()
        print(toc-tic)

Answer 1

除非故事有更多内容，否则您的代码与下面的选项一样快，但我的代码可以找到更多文章和故事。这对你来说可能或可能不重要。我正在使用现代 Windows 笔记本电脑上网

您看到哪些时间让您认为这并不快？或者你认为应该是什么？它以 1 秒的 1/3 运行。

    %%timeit
    requests_session = requests.Session()
    scrapy = requests.get('https://www.marketwatch.com/markets?mod=top_nav ').content
    soup = BeautifulSoup(scrapy, 'lxml')
    for div in soup.findAll('div', attrs={'class': 'collection__elements j-scrollElement'}):
        for div in div.findAll('div', attrs={'class':'article__content'}):
            for div2 in div.find_all('h3', attrs={'class':'article__headline'}):
                 for a in div2.find_all('a', href=True):
                     if a.text:
                        # print(a.text)
                        print(a['href'])

# 318 ms ± 55.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
requests_session = requests.Session()
scrapy = requests.get('https://www.marketwatch.com/markets?mod=top_nav ').content
soup = BeautifulSoup(scrapy, 'lxml')
for link in soup.find_all('a', class_='link', href=re.compile('articles|story')):
    print(link.get('href'))

# 317 ms ± 58 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)