使用 Beautifulsoup 抓取网页时出现奇怪的字符

Question

我正在尝试将 html 作为字符串从 eshop 网站返回，但会返回一些奇怪的字符。当我查看 webconsole 时，我在 html 中看不到这些字符。当 html 在 jupyter notebook 的 pandas 数据框中显示时，我也看不到这些字符。链接是https://www.powerhousefilms.co.uk/collections/limited-editions/products/immaculate-conception-le。我也在这个网站上对另一种产品使用相同的方法，但只在这一页上看到这些字符。网站的其他页面没有这个问题。

html = requests.get(url).text
soup = BeautifulSoup(html)
elem = soup.find_all('div', {'class': product-single_description rte'})
s = str(elem[0])

s 然后看起来像：

    <div class="product-single__description rte">
<div class="product_description">
<div>
<div>
<div><span style="color: #000000;"><em>THIS ITEM IS AVAILABLE TO PRE-ORDER. PLEASE NOTE THAT YOUR PAYMENT WILL BE TAKEN IMMEDIATELY, AND THAT THE ITEM WILL BE DISPATCHED JUST BEFORE THE LISTED RELEASE DATE. </em></span></div>
<div><span style="color: #000000;"><em>Â </em></span></div>
<div><span style="color: #000000;"><em>SHOULD YOU ORDER ANY OF THEÂ ALREADY RELEASED ITEMS FROM OURÂ CATALOGUE AT THE SAME TIME AS THIS PRE-ORDER ITEM, PLEASE NOTE THATÂ YOUR PURCHASES WILL ALL BE SHIPPED TOGETHER WHENÂ THIS PRE-ORDERÂ ITEM BECOMES AVAILABLE.</em></span></div>
</div>
<div><span style="color: #38761d;">Â </span></div>
<div>
<strong>(Jamil Dehlavi, 1992)</strong><br/><em>Release date: 25 March 2019</em><br/>Limited Blu-ray Edition (World Blu-ray premiere)<br/><br/>A Western couple (played by Melissa Leo and James Wilby) working in Pakistan visit an unconventional holy shrine to harness its spiritual powers to help them conceive a child. They are lavished with the attentions of the shrine’s leader (an exceptional performance from Zia Mohyeddin – <em>Lawrence of Arabia</em>, <em>Khartoum</em>) and her followers, but their methods and motives are not all that they seem, and the couple’s lives are plunged into darkness.<br/><br/>This ravishing, unsettling film from director Jamil Dehlavi (<em>The Blood of Hussain</em>, <em>Born of Fire</em>) is a deeply personal work which raises questions of cultural and sexual identity, religious fanaticism and the abuses of power. The brand-new 2K restoration from the original negative was supervised and approved by Dehlavi and cinematographer Nic Knowland.<br/><br/><strong>INDICATOR LIMITED EDITION BLU-RAY SPECIAL FEATURES:</strong>
</div>
<div>
<ul>
<li>New 2K restoration by Powerhouse Films from the original negative, supervised and approved by director Jamil Dehlavi and cinematographer Nic Knowland</li>
<li>
<div>Original stereo audio</div>
</li>
<li>
<div>Alternative original mono mix</div>

我已经尝试指定编码，但仍然得到奇怪的字符。本网站上的 50 多种产品中只有少数存在此问题。

我的抓取方式是否有问题，或者可能是一种简单的清理方法。

谢谢

Answer 1

使用这段代码下载网页中的可见内容。 只需在 page_url 中输入 url

from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request
import os


page_url = "URL Here"
def tag_visible(element):
    if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
        return False
    if isinstance(element, Comment):
        return False
    return True


def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(text=True)
    visible_texts = filter(tag_visible, texts)
    return u" ".join(t.strip() for t in visible_texts)

def Extract_Text(html_bytes, url):
    text_data = text_from_html(html_bytes)
    f = open("DOC.txt", "w")
    string = str(url) + "\n" + text_data
    f.write(str(string))
    f.close()

html_string = ''
response = urlopen(page_url)
if 'text/html' in response.getheader('Content-Type'):
    html_bytes = response.read()
    html_string = html_bytes.decode("utf-8")
Extract_Text(html_bytes, page_url)

Answer 2

所以事实证明，excel 是造成这种情况的原因。当我保存到 CSV 并在 excel 中打开时，我得到了奇怪的结果。

为了防止这种情况，我使用了df.to_csv('df.csv', index=False, encoding = 'utf-8-sig')。指定编码消除了奇怪的字符。

Python Writing Weird Unicode to CSV 有一些关于编码以及 excel 如何与 csv 文件相互渗透的信息。