使用 Beautiful Soup 制作爬虫，索引 url 的问题

Question

所以我正在尝试为搜索引擎制作爬虫。我的代码大部分是完整的。它的工作方式是：

• 使用 BeautifulSoup 打开页面，
• 将 url 和 docID 保存到索引（在本例中只是一个文本文件），
• 获取一些干净的文本并将其保存到文本文件中，
• 从页面中取出所有 url，如果它们不在索引中，则将它们添加到页面列表中。

我遇到的问题是，有时它会多次索引网址。

我做的一个示例是使用初始的pages = ["http://www.ece.udel.edu"]。在输出文件中，每一行都包含一个 docID 和一个 url。其格式为docID url\n。这会在对结果进行排名时出现问题，因为虽然它应该对这些结果进行相同的排名，但我将有多个相同的页面（每个页面都有不同的 docID）被排名。

这是我到目前为止的代码。我试图彻底评论它以使其更容易理解。

def cleanText(soup):
    #remove all unnecessary tags & punctuation leaving text of all lowercase letters & words
    texts = soup.findAll(text=True)
    visible_elements=[]
    for elem in texts:
        if elem.parent.name in ['style', 'script', '[document]', 'head', 'title']:
            visible_elements.append('')
        elem = elem.encode('utf-8','replace')
        result = re.sub('<!--.*-->|\r|\n', '', str(elem), flags=re.DOTALL)
        result = re.sub('\s{2,}|&nbsp;', ' ', result)
        visible_elements.append(result)
    visible_text = ' '.join(visible_elements)
    visible_text = re.sub('[^0-9a-zA-Z]+', ' ', visible_text)
    visible_text = visible_text.lower()
    return visible_text

def naming(pagename, pathToOutput):
    #used to create filename to save output text file
    cantBeInFilename = ("/", "\\",":","*","?","<",">","|")
    for k in range(len(cantBeInFilename)):
        pagename = pagename.replace(cantBeInFilename[k], "-")
    filename = "\\".join((pathToOutput, pagename))
    filename = ".".join((filename, "txt"))
    return filename

def crawl(pages, pathToOutput, pathToLinks):
    depth = len(pages)
    docID = 1
    for i in range(depth): #for each link on the page
        newpages=set() #create an empty set for gathered pages
        for page in pages: #each page in pages, initially atleast 1
            linksText = open(pathToLinks, 'a') #open a file
            linksText.write(str(docID) + " "); #write a docID to identify page
            docID = docID + 1 #increment docID
            linksText.write(str(page) + '\n') #append link into end of file
            linksText.close() #close file to update
            try:
                c = urllib2.urlopen(page) #open the page
                print ("Opening " + page)
            except:
                print ("Could not open " + page)
                continue
            soup = BeautifulSoup(c.read())
            clean_text = cleanText(soup)
            #get all links off page
            links = soup.findAll('a')
            #write web page to text file in pathToOutput directory
            filename = naming(page, pathToOutput)
            f = open(filename, 'w')
            f.write(clean_text)
            f.close()
            depth = 0
            for link in links:
                if('href' in dict(link.attrs)):
                    #set depth equal to the # of links in the pages
                    depth = depth+1
                    url = urljoin(page, link['href'])
                    #remove unnecessary portions of url
                    if url.find("'") != -1:
                        continue
                    url = url.split('#')[0]
                    #get each line (each link) from linksText
                    linksText = open(pathToLinks, 'r')
                    lines = linksText.readlines()
                    linksText.close()
                    #remove docIDs just to have 
                    for j in range(len(lines)):
                        lines[j-1] = lines[j-1].split()[1]
                    #FOR ALL LINKS
                    if url[0:4] == "http":
                        #check to see if the url ends with a /, if so remove it
                        if url[len(url)-1]=="/":
                            url = url[0:len(url)-1]
                        #check to see if the url is already in links
                        present = 0
                        for line in lines:
                            if (url == line):
                                present = 1
                            else:
                                present = 0
                        #if the url isn't present, add to pages to be cycled through
                        if (present == 0):
                            print ("Indexing " + url)
                            newpages.add(url)
                #add newpages to pages
                for newpage in newpages:
                    pages.append(newpage)
                newpages = set()
            #remove already read page from pages
            pages.remove(page)

我会附上我已经从中获得的输出，但我刚开始在网站上发布问题，不能发布超过两个链接的问题 :( 抱歉，如果这很长，令人困惑，而且简单明了不会使感觉

Answer 1

这段代码 sn-p 没有按照你的想法做

present = 0
for line in lines:
    if (url == line):
        present = 1
    else:
        present = 0
#if the url isn't present, add to pages to be cycled through
if (present == 0):
    print ("Indexing " + url)
    newpages.add(url)

只有当匹配在字符串的最后一项时才会留下present = 1。一个更简单的测试方法是：

#if the url isn't present, add to pages to be cycled through
if url not in lines:
    print ("Indexing " + url)
    newpages.add(url)

你可能还需要改变

#get each line (each link) from linksText
linksText = open(pathToLinks, 'r')
lines = linksText.readlines()
linksText.close()

进入

#get each line (each link) from linksText
linksText = open(pathToLinks, 'r')
lines = linksText.read().split('\n')
linksText.close()

这样lines 中的每个line 都不会以\n 结尾