Web爬虫递归BeautifulSoup答案

【问题标题】：Web crawler recursively BeautifulSoupWeb爬虫递归BeautifulSoup
【发布时间】：2018-08-13 16:38:45
【问题描述】：

我正在尝试递归地抓取所有英文文章链接的维基百科 url。我想执行 n 的深度优先遍历，但由于某种原因，我的代码不会在每次传递时都重复出现。知道为什么吗？

def crawler(url, depth):
    if depth == 0:
        return None
    links = bs.find("div",{"id" : "bodyContent"}).findAll("a" , href=re.compile("(/wiki/)+([A-Za-z0-9_:()])+"))

    print ("Level ",depth," ",url)
    for link in links:
        if ':' not in link['href']:
            crawler("https://en.wikipedia.org"+link['href'], depth - 1)

这是对爬虫的调用

url = "https://en.wikipedia.org/wiki/Harry_Potter"
html = urlopen(url)
bs = BeautifulSoup(html, "html.parser")
crawler(url,3)

【问题讨论】：

我必须使用 beautifulsoup 来解决这个问题...这是我必须完成的任务...我无法使用数据转储
函数内部对url的请求在哪里？您必须在每次重复时发送请求。

标签： python beautifulsoup web-crawler

【解决方案1】：

您需要为每个不同的 URL 获取页面源（向页面发送请求）。您在 crawler() 函数中缺少该部分。在函数之外添加这些行，不会递归调用它们。

def crawler(url, depth):
    if depth == 0:
        return None

    html = urlopen(url)                        # You were missing 
    soup = BeautifulSoup(html, 'html.parser')  # these lines.

    links = soup.find("div",{"id" : "bodyContent"}).findAll("a", href=re.compile("(/wiki/)+([A-Za-z0-9_:()])+"))

    print("Level ", depth, url)
    for link in links:
        if ':' not in link['href']:
            crawler("https://en.wikipedia.org"+link['href'], depth - 1)

url = "https://en.wikipedia.org/wiki/Big_data"
crawler(url, 3)

部分输出：

Level  3 https://en.wikipedia.org/wiki/Big_data
Level  2 https://en.wikipedia.org/wiki/Big_Data_(band)
Level  1 https://en.wikipedia.org/wiki/Brooklyn
Level  1 https://en.wikipedia.org/wiki/Electropop
Level  1 https://en.wikipedia.org/wiki/Alternative_dance
Level  1 https://en.wikipedia.org/wiki/Indietronica
Level  1 https://en.wikipedia.org/wiki/Indie_rock
Level  1 https://en.wikipedia.org/wiki/Warner_Bros._Records
Level  1 https://en.wikipedia.org/wiki/Joywave
Level  1 https://en.wikipedia.org/wiki/Electronic_music
Level  1 https://en.wikipedia.org/wiki/Dangerous_(Big_Data_song)
Level  1 https://en.wikipedia.org/wiki/Joywave
Level  1 https://en.wikipedia.org/wiki/Billboard_(magazine)
Level  1 https://en.wikipedia.org/wiki/Alternative_Songs
Level  1 https://en.wikipedia.org/wiki/2.0_(Big_Data_album)

【讨论】：