爬虫跳过第一页的内容

Question

我创建了一个爬虫，它从网站解析某些内容。

首先，它从左侧栏中抓取指向该类别的链接。

其次，它收集通过连接到个人资料页面的分页传播的整个链接

最后，转到每个个人资料页面，它会抓取姓名、电话和网址。

到目前为止，它做得很好。我看到这个爬虫的唯一问题是它总是从第二页开始抓取，跳过第一页。我想可能有什么办法可以解决这个问题。这是我正在尝试的完整代码：

import requests
from lxml import html

url="https://www.houzz.com/professionals/"

def category_links(mainurl):
    req=requests.Session()
    response = req.get(mainurl).text
    tree = html.fromstring(response)
    for titles in tree.xpath("//a[@class='sidebar-item-label']/@href"):
        next_pagelink(titles)   # links to the category from left-sided bar


def next_pagelink(process_links):
    req=requests.Session()
    response = req.get(process_links).text
    tree = html.fromstring(response)
    for link in tree.xpath("//ul[@class='pagination']//a[@class='pageNumber']/@href"):
        profile_pagelink(link)      # the whole links spread through pagination connected to the profile page


def profile_pagelink(procured_links):
    req=requests.Session()
    response = req.get(procured_links).text
    tree = html.fromstring(response)
    for titles in tree.xpath("//div[@class='name-info']"):
        links = titles.xpath(".//a[@class='pro-title']/@href")[0]
        target_pagelink(links)         # profile page of each link


def target_pagelink(main_links):
    req=requests.Session()
    response = req.get(main_links).text
    tree = html.fromstring(response)

    def if_exist(titles,xpath):
        info=titles.xpath(xpath)
        if info:
            return info[0]
        return ""

    for titles in tree.xpath("//div[@class='container']"):
        name = if_exist(titles,".//a[@class='profile-full-name']/text()")
        phone = if_exist(titles,".//a[contains(concat(' ', @class, ' '), ' click-to-call-link ')]/@phone")
        web = if_exist(titles,".//a[@class='proWebsiteLink']/@href")
        print(name,phone,web)

category_links(url)

Answer 1

第一页的问题是它没有“分页”类，所以这个表达式：tree.xpath("//ul[@class='pagination']//a[@class='pageNumber']/@href") 返回一个空列表，profile_pagelink 函数永远不会被执行。

作为一种快速解决方法，您可以在 category_links 函数中单独处理这种情况：

def category_links(mainurl):
    response = requests.get(mainurl).text
    tree = html.fromstring(response)
    if mainurl == "https://www.houzz.com/professionals/": 
        profile_pagelink("https://www.houzz.com/professionals/")
    for titles in tree.xpath("//a[@class='sidebar-item-label']/@href"):
        next_pagelink(titles)   

我还注意到 target_pagelink 由于 if_exist 返回 "" 而打印了很多空字符串。如果在 for 循环中添加条件，则可以跳过这些情况：

for titles in tree.xpath("//div[@class='container']"):    # use class='profile-cover' if you get douplicates #
    name = if_exist(titles,".//a[@class='profile-full-name']/text()")
    phone = if_exist(titles,".//a[contains(concat(' ', @class, ' '), ' click-to-call-link ')]/@phone")
    web = if_exist(titles,".//a[@class='proWebsiteLink']/@href")
    if name+phone+web : 
        print(name,phone,web)

最后，requests.Session 主要用于存储 cookie 和其他脚本不需要的标头。您可以只使用requests.get 并获得相同的结果。