Python Crawler 忽略页面上的链接答案

【问题标题】：Python Crawler is ignoring links on pagePython Crawler 忽略页面上的链接
【发布时间】：2014-02-22 04:22:35
【问题描述】：

所以我为我的朋友编写了一个爬虫，它会遍历大量作为搜索结果的网页，将所有链接拉出页面，检查它们是否在输出文件中，如果不在则添加那里。它进行了大量调试，但效果很好！不幸的是，这个小家伙真的很挑剔它认为添加哪些锚定标签足够重要。

代码如下：

#!C:\Python27\Python.exe
from bs4 import BeautifulSoup
from urlparse import urljoin #urljoin is a class that's included in urlparse
import urllib2
import requests #not necessary but keeping here in case additions to code in future

urls_filename = "myurls.txt" #this is the input text file,list of urls or objects to scan
output_filename = "output.txt" #this is the output file that you will export to Excel
keyword = "skin" #optional keyword, not used for this script. Ignore

with open(urls_filename, "r") as f:   
    url_list = f.read() #This command opens the input text file and reads the information inside it

with open(output_filename, "w") as f: 
    for url in url_list.split("\n"):  #This command splits the text file into separate      lines so it's easier to scan               
            hdr = {'User-Agent': 'Mozilla/5.0'} #This (attempts) to tell the webpage that the program is a Firefox browser
            try:
                    response = urllib2.urlopen(url) #tells program to open the url from the text file
            except:
                    print "Could not access", url
                    continue
            page = response.read() #this assigns a variable to the open page. like algebra, X=page opened
            soup = BeautifulSoup(page) #we are feeding the variable to BeautifulSoup so it can analyze it
            urls_all = soup('a') #beautiful soup is analyzing all the 'anchored' links in the page
            for link in urls_all:
                    if('href' in dict(link.attrs)):
                            url = urljoin(url, link['href']) #this combines the relative link e.g. "/support/contactus.html" and adds to domain 
                    if url.find("'")!=-1: continue #explicit statement that the value is not void. if it's NOT void, continue
                    url=url.split('#')[0]
                    if (url[0:4] == 'http' and url not in output_filename): #this checks if the item is a webpage and if it's already in the list
                            f.write(url + "\n") #if it's not in the list, it writes it to the output_filename

除了以下链接外，它的效果很好： https://research.bidmc.harvard.edu/TVO/tvotech.asp

此链接有许多类似“tvotech.asp?Submit=List&ID=796”的内容，直接忽略它们。进入我的输出文件的唯一锚点是主页本身。这很奇怪，因为查看源代码，它们的锚点非常标准，比如- 他们有'a'和'href'，我看不出bs4为什么会通过它并且只包含主链接。请帮忙。我尝试从第 30 行删除 http 或将其更改为 https ，这只会删除所有结果，甚至主页都不会进入输出。

【问题讨论】：

标签： python beautifulsoup web-crawler

【解决方案1】：

那是因为其中一个链接的 href 中有一个 mailto，然后将其设置为 url 参数并破坏其余链接，因为不通过 url[0:4] == 'http' 条件，看起来像这样：

mailto:research@bidmc.harvard.edu?subject=Question about TVO Available Technology Abstracts

您应该将其过滤掉或不在循环中使用相同的参数url，注意对 url1 的更改：

for link in urls_all:
    if('href' in dict(link.attrs)):
            url1 = urljoin(url, link['href']) #this combines the relative link e.g. "/support/contactus.html" and adds to domain 
    if url1.find("'")!=-1: continue #explicit statement that the value is not void. if it's NOT void, continue
    url1=url1.split('#')[0]
    if (url1[0:4] == 'http' and url1 not in output_filename): #this checks if the item is a webpage and if it's already in the list
            f.write(url1 + "\n") #if it's not in the list, it writes it to the output_filename

【讨论】：