python写的类爬虫抛出属性错误

Question

在用python写了一些代码之后，我陷入了深深的困境。我是在 python 中遵循 OOP 设计编写代码的新手。我在代码中使用的 xpath 完美无缺。通过“page_crawler”类的实例在“info_grabber”类中运行“passing_links”方法时，我迷失了方向。每次我运行我的代码时，我都会收到一个错误“'page_crawler' 对象没有属性 'passing_links'”。也许我写我的类爬虫的方式不是它应该的样子。但是，由于我已经花了几个小时在它上面，所以我想我可能会得到任何关于我应该纠正哪些行以使其工作的建议。提前感谢您查看它：

from lxml import html
import requests

class page_crawler(object):

    main_link = "https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA"
    base_link = "https://www.yellowpages.com"

    def __init__(self):

        self.links = [self.main_link]


    def crawler(self):
        for link in self.links:
            self.get_link(link)

    def get_link(self, link):

        print("Running page "+ link)
        page = requests.get(link)
        tree = html.fromstring(page.text)
        item_links = tree.xpath('//h2[@class="n"]/a[@class="business-name"][not(@itemprop="name")]/@href')
        for item_link in item_links:
            return self.base_link + item_link

        links = tree.xpath('//div[@class="pagination"]//li/a/@href')
        for url in links:
            if not self.base_link + url in self.links:
                self.links += [self.base_link + url]



class Info_grabber(page_crawler):

    def __init__(self, plinks):
        page_crawler.__init__(self)
        self.plinks = [plinks]

    def passing_links(self):
        for nlink in self.plinks:
            print(nlink)
            self.crawling_deep(nlink)

    def crawling_deep(self, uurl):

        page = requests.get(uurl)
        tree = html.fromstring(page.text)

        name = tree.findtext('.//div[@class="sales-info"]/h1')
        phone = tree.findtext('.//p[@class="phone"]')
        try:
            email = tree.xpath('//div[@class="business-card-footer"]/a[@class="email-business"]/@href')[0]
        except IndexError:
            email=""

        print(name, phone, email)


if __name__ == '__main__':

    crawl = Info_grabber(page_crawler)
    crawl.crawler()
    crawl.passing_links()

现在执行时，当它到达“self.crawling_deep(nlink)”行时，我收到一个新错误“raise MissingSchema(error)”

Answer 1

我不确定我是否理解您在 page_crawler.get_link 中尝试执行的操作，但我认为您应该采用不同的方法来收集“分页”链接。
我将Info_grabber.plinks 重命名为Info_grabber.links，以便page_crawler.crawler 可以访问它们，并设法从多个页面中提取信息，但是代码远非理想。

class page_crawler(object):

    main_link = "https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA"
    base_link = "https://www.yellowpages.com"

    def __init__(self):
        self.links = []
        self.pages = []

    def crawler(self):
        for link in self.links:
            self.get_link(link)

    def get_link(self, link):
        print("Running page "+ link)
        page = requests.get(link)
        tree = html.fromstring(page.text)
        item_links = tree.xpath('//h2[@class="n"]/a[@class="business-name"][not(@itemprop="name")]/@href')
        for item_link in item_links:
            if not self.base_link + item_link in self.links:
                self.links += [self.base_link + item_link]

    def get_pages(self, link):
        page = requests.get(link)
        tree = html.fromstring(page.text)
        links = tree.xpath('//div[@class="pagination"]//li/a/@href')
        for url in links:
            if not self.base_link + url in self.pages:
                self.pages += [self.base_link + url]


class Info_grabber(page_crawler):

    def __init__(self, plinks):
        page_crawler.__init__(self)
        self.links += [plinks]

    def passing_links(self):
        for nlink in self.links:
            print(nlink)
            self.crawling_deep(nlink)

    def crawling_deep(self, uurl):
        page = requests.get(uurl)
        tree = html.fromstring(page.text)
        name = tree.findtext('.//div[@class="sales-info"]/h1')
        phone = tree.findtext('.//p[@class="phone"]')
        try:
            email = tree.xpath('//div[@class="business-card-footer"]/a[@class="email-business"]/@href')[0]
        except IndexError:
            email=""
        print(name, phone, email)


if __name__ == '__main__':
    url = page_crawler.main_link
    crawl = Info_grabber(url)
    crawl.crawler()
    crawl.passing_links()

您会注意到我在page_crawler 中添加了pages 属性和get_pages 方法，我将把实现部分留给您。
稍后您可能需要向page_crawler 添加更多方法，因为如果您开发更多子类，它们可能会派上用场。最后考虑考虑组合，因为它也是一个强大的 OOP 功能。

Answer 2

您的crawl 是页面爬虫类的实例，但不是InfoGrabber 类，后者是具有passing_links 方法的类。我认为您想要做的是让 crawl 成为 InfoGrabber 的一个实例。

那我相信在做 self.crawling_deep 之前你必须做：

if n_link:
    page = requests.get(n_link).text            
    tel = re.findall(r'\d{10}', page)[0] if re.findall(r'\d{10}', page) else ""
    print(tel)