优化网络爬虫

Question

接下来的爬行虽然很短，但速度非常慢。我的意思是，“在一部完整的故事片中流行”，慢。

def bestActressDOB():
    # create empty bday list
    bdays = []    
    # for every base url
    for actress in getBestActresses("https://en.wikipedia.org/wiki/Academy_Award_for_Best_Actress"):
        # use actress list to create unique actress url
        URL = "http://en.wikipedia.org"+actress
        # connect to html
        html = urlopen(URL)
        # create soup object
        bsObj = BeautifulSoup(html, "lxml")
        # get text from <span class='bday">
        try:
            bday = bsObj.find("span", {"class":"bday"}).get_text()
        except AttributeError:
            print(URL)
        bdays.append(bday)
        print(bday)
    return bdays

它从一个维基百科页面上的表格中获取每位获得奥斯卡奖提名的女演员的姓名，然后将其转换为列表，使用这些名称创建 URL 以访问每位女演员的 wiki，并在其中获取她的出生日期.该数据将用于计算每位女演员被提名或获得奥斯卡最佳女演员奖的年龄。除了 Big O，有没有办法实时加快速度。我对这种事情没有什么经验，所以我不确定这有多正常。想法？

编辑：请求的子程序

def getBestActresses(URL):
    bestActressNomineeLinks = []
    html = urlopen(URL)
    try:
        soup = BeautifulSoup(html, "lxml")
        table = soup.find("table", {"class":"wikitable sortable"})
    except AttributeError:
        print("Error creating/navigating soup object")
    table_row = table.find_all("tr")
    for row in table_row:
        first_data_cell = row.find_all("td")[0:1]
        for datum in first_data_cell:
            actress_name = datum.find("a")
            links = actress_name.attrs['href']
            bestActressNomineeLinks.append(links)
    #print(bestActressNomineeLinks)
    return bestActressNomineeLinks

Answer 1

我建议在速度更快的计算机上尝试，甚至在谷歌云平台、Microsoft azure 或亚马逊网络服务等服务上运行。没有任何代码可以让它运行得更快。