无法将网页转换为所有链接都是绝对链接的 html 文件

Question

我创建了一个脚本，它能够将网页转换为 html 文件，以便该文件看起来与该网页非常相似。我无法解决的唯一问题是 html 文件包含相对 url，如 /organizations/11-unilever?group=8831 而绝对链接是 https://en.eyeka.com/organizations/11-unilever?group=8831。

我试过了：

import requests

link = "https://en.eyeka.com/contests/8831/results"

with requests.Session() as s:
    s.headers['user-agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
    res = s.get(link)
    
    with open("results.html", "wb") as f:
        f.write(res.content)

如何将网页转换为所有链接都是绝对链接（完整链接）的html文件？

Answer 1

您可以使用正则表达式来查找和替换相对 URL：

import re
import requests
import urllib3

link = "https://en.eyeka.com/contests/8831/results"

with requests.Session() as s:
    base_url = "://".join(
        urllib3.get_host(link)[:2]
    )  # get the base url (https://en.eyeka.com)

    s.headers[
        "user-agent"
    ] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36"

    text = s.get(link).text

    text = re.sub(
        r"(href=[\"\'])\/", f"\g<1>{base_url}/", text, 0, re.MULTILINE
    )  # replace all relative urls with absolute

    with open("results.html", "w", encoding="utf-8") as f:
        f.write(text)

Answer 2

我检查了您链接的页面，建议您检查脚本将在页面中找到的所有 'href="/' 并将其替换为 'href="https://en.eyeka.com/'

所以：

string_html.replace('href="/', 'href="https://en.eyeka.com/') 

类似的东西。

import requests

link = "https://en.eyeka.com/contests/8831/results"

with requests.Session() as s:
    s.headers['user-agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
    res = s.get(link)
# convert to string
    res = res.text
# add https ...
    res = res.replace('href="/', 'href="https://en.eyeka.com/')
# encode 
    res = res.encode()  
    with open("results.html", "wb") as f:
        f.write(res)

由于它们是相对路径，它们只能是https://en.eyeka.com/ + 相对路径。

Answer 3

我的答案包括其他库，但它并没有假设它只适用于您提供的示例 url。它也不假定所有 url 都是相对的。

import requests
from urllib.parse import urlparse
from bs4 import BeautifulSoup

link = "https://en.eyeka.com/contests/8831/results"

with requests.Session() as s:
    headers = {"user-agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36"}
    s.headers = headers
    res = s.get(link)

u = urlparse(res.url)
domain = u.scheme + "://" + u.netloc  # extract domain will work with http or https since we are also extracting the scheme
soup = BeautifulSoup(res.content, "html.parser")

for a in soup.find_all('a'):  # loop through all links
    if "href" in a.attrs:  # not all links have an href tag
        if "http" not in a.attrs['href']:  # not all links are relative links
            a.attrs['href'] = domain + a.attrs['href']

with open("results.html", "w", encoding='utf-8') as f:
    f.write(str(soup))