使用 Python 3 从网页中提取链接时出错答案

【问题标题】：Error while Extracting Link from webpage using Python 3使用 Python 3 从网页中提取链接时出错
【发布时间】：2017-04-20 17:26:01
【问题描述】：

让我们考虑以下几点：

<div class="more reviewdata">

<a onclick="bindreviewcontent('1660651',this,false,'I found this review of Star Health Insurance pretty useful',925075287,'.jpg','I found this review of Star Health Insurance pretty useful %23WriteShareWin','http://www.mouthshut.com/review/Star-Health-Insurance-review-toqnmqrlrrm','Star Health Insurance',' 2/5');" style="cursor:pointer">Read More</a>

</div>

从上面类似的东西，我想单独提取http链接如下：

http://www.mouthshut.com/review/Star-Health-Insurance-review-toqnmqrlrrm

为了实现这一点，我用 Python 中的 BeautifulSoup 和正则表达式编写了一段代码。代码如下：

import urllib.request
import re

from bs4 import BeautifulSoup
page = urllib.request.urlopen('http://www.mouthshut.com/product-reviews/Star-Health-Insurance-reviews-925075287').read()

soup = BeautifulSoup(page, "html.parser")

required = soup.find_all("div", {"class": "more reviewdata"})

for link in re.findall('http://www.mouthshut.com/review/Star-Health-Insurance-review-[a-z]*', required):
   print(link)

执行时，程序抛出如下错误：

Traceback (most recent call last):

File "E:/beautifulSoup20April2.py", line 11, in <module>

for link in re.findall('http://www.mouthshut.com/review/Star-Health-Insurance-review-[a-z]*', required):

File "C:\Program Files (x86)\Python35-32\lib\re.py", line 213, in findall
return _compile(pattern, flags).findall(string)

TypeError: expected string or bytes-like object

有人可以建议应该怎么做才能单独提取 url 而不会出现任何错误？

【问题讨论】：

标签： html regex python-3.x beautifulsoup

【解决方案1】：

首先您需要循环required，其次您尝试在对象<class 'bs4.element.Tag'> 上使用regex（python 对此表示抱怨），然后您需要从@987654325 中提取html @元素，可以用prettify()完成

这是一个工作版本：

import urllib.request
import re
from bs4 import BeautifulSoup
page = urllib.request.urlopen('http://www.mouthshut.com/product-reviews/Star-Health-Insurance-reviews-925075287').read()
soup = BeautifulSoup(page, "html.parser")
required = soup.find_all("div", {"class": "more reviewdata"})
for div in required:
   for link in re.findall(r'http://www\.mouthshut\.com/review/Star-Health-Insurance-review-[a-z]*', div.prettify()):
      print(link)

输出：

http://www.mouthshut.com/review/Star-Health-Insurance-review-ommmnmpmqtm
http://www.mouthshut.com/review/Star-Health-Insurance-review-rmqulrolqtm
http://www.mouthshut.com/review/Star-Health-Insurance-review-ooqrupoootm
http://www.mouthshut.com/review/Star-Health-Insurance-review-rlrnnuslotm
http://www.mouthshut.com/review/Star-Health-Insurance-review-umqsquttntm
...

【讨论】：