无法使用 Beautiful Soup 和 Python 正确地抓取描述答案

【问题标题】：Not able to scrape the description properly using Beautiful Soup and Python无法使用 Beautiful Soup 和 Python 正确地抓取描述
【发布时间】：2021-02-15 19:18:21
【问题描述】：

我正在网络抓取此链接：https://www.americanexpress.com/in/credit-cards/smart-earn-credit-card/?linknav=in-amex-cardshop-allcards-learn-SmartEarnCreditCard-carousel 使用 bs4 和 python。

我基本上是使用以下代码从该网站获取主要好处。

link = 'https://www.americanexpress.com/in/credit-cards/smart-earn-credit-card/?linknav=in-amex-cardshop-allcards-learn-SmartEarnCreditCard-carousel'
html = urlopen(link)
soup = BeautifulSoup(html, 'lxml')

details = []

for span in soup.select(".why-amex__subtitle span"):
    details.append(f'{span.get_text(strip=True)}: {span.find_next("span").get_text(strip=True)}')



print(details)

输出

['Accelerated Earn Rate: Earn 10X Membership Rewards® Points2on your spending on Flipkart and Uber and earn 5X Membership Rewards Points2on Amazon, Swiggy, BookMyShow and more.', 'Welcome Bonus: Rs. 500 cashback as Welcome Gift on eligible spends1of Rs. 10,000 in the first 90 days of Cardmembership', 'Renewal Fee Waiver: Get a renewal fee waiver on eligible spends3of Rs.40,000 and above in the previous year of Cardmembership', 'AMERICAN EXPRESS EMI: Convert purchases into']

此列表中的最后一项未正确抓取，它不完整。因为正文中间有一个超链接。

下面是该问题对应的html代码：

<div class="why-amex__col"><span class="icons  why-amex__lrgIcon icon-Amex-Icons-2016-85"></span><h4 class="why-amex__subtitle"><div><span>AMERICAN EXPRESS EMI</span></div></h4><div class="why-amex__copy"><div class="description_text"><div><span>Convert purchases into </span><a href="https://www.americanexpress.com/india/membershiprewards/cardmember_offers/viewmore.html" target="_blank">EMI</a><span> at the point of sale with an interest rate as low as 12% p.a. and zero foreclosure charges</span></div></div></div></div>

我想获得最后一项的完整描述，而不会遗漏文本。

【问题讨论】：

标签： python web-scraping beautifulsoup

【解决方案1】：

只需将 innerHTML 附加到 details 中，然后遍历标签即可构造您的文本。

类似：


texts = []
for i, detail in enumerate(details):

    texts.append('')
    for tag in detail.findChildren(recursive=False):

        texts[i] += tag.get_text(strip=True)

【讨论】：

Traceback（最近一次调用最后一次）：文件“C:\Users\Hari\PycharmProjects\Card_Prj\buffer.py”，第 31 行，在中用于标记详细信息.findChildren(recursive= False): AttributeError: 'str' object has no attribute 'findChildren'
是的，错误是告诉您将字符串附加到details。您需要附加bs4 对象，以便您可以使用findChildren。