如何从 URL 中抓取整个文本正文？

Question

我有一个从this page 收集的 URL 列表，它们基本上只是来自人们的引用，我想将每个不同 URL 的引用保存在单独的文件中。

为了获取 URL 列表，我使用了：

import bs4
from urllib.request import Request,urlopen as uReq
from bs4 import BeautifulSoup as soup
import re
#define url of interest
my_url = 'http://archive.ontheissues.org/Free_Trade.htm'

# set up known browser user agent for the request to bypass HTMLError
req=Request(my_url,headers={'User-Agent': 'Mozilla/5.0'})

#opening up connection, grabbing the page
uClient = uReq(req)
page_html = uClient.read()
uClient.close()

#html is jumbled at the moment, so call html using soup function
soup = soup(page_html, "html.parser")

# Test: print title of page
soup.title


tags = soup.findAll("a" , href=re.compile("javascript:pop"))
print(tags)

# get list of all URLS
for links in tags:
    link = links.get('href')
    if "java" in link: 
        print("http://archive.ontheissues.org" + link[18:len(link)-3])

如何从每个链接中提取内容，包括文本、项目符号、段落，然后将它们保存到单独的文件中？ 另外，我不想要那些不是引号的东西，比如这些页面中的其他 URL。

Answer 1

您希望抓取的“引用”页面有一些不完整/悬空的 HTML 标记。如果您不了解您正在使用的解析器，那么解析这些可能会很痛苦。要获得有关它们的提示，请参阅this page。

现在回到代码，为方便起见，我使用了lxml 解析器。继续前进，如果您观察任何这些“引用”页面的页面源，那么您会看到您希望抓取的大部分文本都存在于以下标签之一中：{h3,p ,ul,ol}。另外，请注意，每个h3 标签旁边都有一个字符串。可以使用.next_sibling 捕获此字符串。 现在条件已经设置好了，让我们继续代码吧。

import bs4
from urllib.request import Request,urlopen as uReq, HTTPError 
#Import HTTPError in order to avoid the links with no content/resource of interest
from bs4 import BeautifulSoup as soup_
import re
#define url of interest
my_url = 'http://archive.ontheissues.org/Free_Trade.htm'

#Creating a function to harness the power of scraping frequently
def make_soup(url):
    # set up known browser user agent for the request to bypass HTMLError
    req=Request(url,headers={'User-Agent': 'Mozilla/5.0'})

    #opening up connection, grabbing the page
    uClient = uReq(req)
    page_html = uClient.read()
    uClient.close()

    #html is jumbled at the moment, so call html using soup function
    soup = soup_(page_html, "lxml") 
    return soup

# Test: print title of page
#soup.title

soup = make_soup(my_url)
tags = soup.findAll("a" , href=re.compile("javascript:pop\("))
#print(tags)

# get list of all URLS
for links in tags:
    link = links.get('href')
    if "java" in link: 
        print("http://archive.ontheissues.org" + link[18:len(link)-3])
        main_url = "http://archive.ontheissues.org" + link[18:len(link)-3] 
        try:
            sub_soup = make_soup(main_url)
            content_collexn = sub_soup.body.contents #Splitting up the page into contents for iterative access 
            #text_data = [] #This list can be used to store data related to every person
            for item in content_collexn:
                #Accept an item if it belongs to the following classes
                if(type(item) == str):
                    print(item.get_text())
                elif(item.name == "h3"):
                    #Note that over here, every h3 tagged title has a string following it
                    print(item.get_text())   
                    #Hence, grab that string too
                    print(item.next_sibling) 
                elif(item.name in ["p", "ul", "ol"]):
                    print(item.get_text())
        except HTTPError: #Takes care of missing pages and related HTTP exception
            print("[INFO] Resource not found. Skipping to next link.")

        #print(text_data)

Answer 2

这些是可以提供帮助的几个方面。

您可以使用Session 对象来提高重用连接的效率。

您可以使用 bs4 4.7.1 压缩您的打开代码以获取正确的 url，如下所示，我使用属性 = 值 css 选择器限制为包含 javascript:pop 的 hrefs。 * 是contains 运算符。

[href*="javascript:pop"]

然后添加:contains 的伪选择器以进一步限制innerText 中包含单词quote 的url。这会将匹配元素列表细化为所需的元素。

:contains(quote)

---

import requests
from bs4 import BeautifulSoup as bs

with requests.Session() as s:
    r = s.get('http://archive.ontheissues.org/Free_Trade.htm')
    soup = bs(r.content, 'lxml')
    links = [item['href'] for item in soup.select('[href*="javascript:pop"]:contains(quote)')]
    for link in links:
        #rest of code working with Session

---

参考资料：

1. CSS Attribute selectors
2. CSS selectors
3. Session object
4. HTTP Sessions