从 Safaribooksonline 抓取时，Python 请求和 BeautifulSoup4 .get('href') 返回绝对地址

Question

我正在尝试从网页中抓取 <a> 标记的内容。我的代码是：

from bs4 import BeautifulSoup
import requests
import sys

reload(sys)
sys.setdefaultencoding('utf-8')

url = 'https://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961'

req = requests.get(url)
soup = BeautifulSoup(req.text, 'html.parser')

lessons = soup.find_all('li', class_='toc-level-1')
lesson = lessons[0]
print(lesson)

我的页面有一个元素：（直接从我在 Firefox 中的 DOM 检查器的输出中获得）...

<li class="toc-level-1 t-toc-level-1 js-content-uri" data-content-uri="/api/v1/book/9780134985961/chapter/LPOC_00_00_00.html">
   <a href="/library/view/linux-performance-optimization/9780134985961/LPOC_00_00_00.html" class="t-chapter" tabindex="39">Introduction</a>
   <ol>
      <li class="toc-level-2 t-toc-level-2 js-content-uri" data-content-uri="/api/v1/book/9780134985961/chapter/LPOC_00_00_00.html"><a href="/library/view/linux-performance-optimization/9780134985961/LPOC_00_00_00.html" class="t-chapter" tabindex="41">Linux Performance Optimization: Introduction</a></li>
   </ol>
</li>

但是，当我使用 requests 和 bs4 模块来抓取数据时，使用上面的代码，我得到的输出是：

<li class="toc-level-1 t-toc-level-1">
    <a class="t-chapter js-chapter" href="https://www.safaribooksonline.comhttps://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961/LPOC_00_00_00.html">Introduction</a>
    <ol>
        <li class="toc-level-2 t-toc-level-2">
            <a class="t-chapter js-chapter" href="https://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961/LPOC_00_00_00.html">Linux Performance Optimization: Introduction</a>
        </li>
    </ol>
</li>

注意到<a> 标签的href 值了吗？它们应该是相对 URL，例如：/library/view/linux-performance-optimization/9780134985961/LPOC_00_00_00.html，但我得到的是绝对 URL——有时这太错误了：https://www.safaribooksonline.comhttps://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961/LPOC_00_00_00.html。

我不知道域名是如何作为链接 url 的前缀的，因为在原始 HTML 中只给出了 href 值，除非 requests 或 bs4 这样做。我以前使用相同方法的所有脚本也产生了类似的错误。模块方面是否发生了变化，还是我做错了什么？

Answer 1

您可以使用正则表达式从href 中提取 URL：

from bs4 import BeautifulSoup
import requests
import sys
import re

url = 'https://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961'

req = requests.get(url)
soup = BeautifulSoup(req.text, 'html.parser')
hrefs = set()

for lesson in soup.find_all('li', class_='toc-level-1'):
    for a in lesson.find_all('a', href=True):
        found_urls = re.split(r'(https?:\/\/.*?)', a['href'])
        hrefs.add(found_urls[-2] + found_urls[-1])

for href in sorted(hrefs):
    print(href)

给你一个找到的href列表开始：

https://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961/LPOC_00_00_00.html
https://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961/LPOC_01_00_00.html
https://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961/LPOC_01_01_00.html
https://www.safaribooksonline.com/library/view/linux-performance-optimization/9780134985961/LPOC_01_01_01.html