使用 Apify Puppeteer 和 requestQueue 从 sitemap.xml 抓取 url

Question

Apify 可以从 sitemap.xml 抓取链接

const Apify = require('apify');

Apify.main(async () => {
    const requestList = new Apify.RequestList({
        sources: [{ requestsFromUrl: 'https://edition.cnn.com/sitemaps/cnn/news.xml' }],
    });
    await requestList.initialize();

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        handlePageFunction: async ({ page, request }) => {
            console.log(`Processing ${request.url}...`);
            await Apify.pushData({
                url: request.url,
                title: await page.title(),
                html: await page.content(),
            });
        },
    });

    await crawler.run();
    console.log('Done.');
});

https://sdk.apify.com/docs/examples/puppeteersitemap#docsNav

但是，如果我使用 requestQueue，我不确定如何从 sitemap.xml 抓取链接。例如：

const requestQueue = await Apify.openRequestQueue();
await requestQueue.addRequest({url: "https://google.com});

 //this is not working. Apify is simply crawling sitemap.xml 
 //and not adding urls from sitemap.xml to requestQueue
 await requestQueue.addRequest({url:`https://google.com/sitemap.xml`});

 const crawler = new Apify.PuppeteerCrawler({
    requestQueue,

    // This function is called for every page the crawler visits
    handlePageFunction: async (context) => {


        const {request, page} = context;

        const title = await page.title();
        let page_url = request.url;
        console.log(`Title of ${page_url}: ${title}`);

        await Apify.utils.enqueueLinks({
            page, selector: 'a', pseudoUrls, requestQueue});
    },


});

await crawler.run();

Answer 1

Apify 的伟大之处在于您可以同时使用RequestList 和RequestQueue。在这种情况下，项目会在您抓取时从列表中取出（不会使队列超载）。通过同时使用这两种方法，您可以两全其美。

Apify.main(async () => {
    const requestList = new Apify.RequestList({
        sources: [{ requestsFromUrl: 'https://edition.cnn.com/sitemaps/cnn/news.xml' }],
    });
    await requestList.initialize();

    const requestQueue = await Apify.openRequestQueue();

    const crawler = new Apify.PuppeteerCrawler({
        requestList,
        requestQueue,
        handlePageFunction: async ({ page, request }) => {
            console.log(`Processing ${request.url}...`);

            // This is just an example, define your logic
            await Apify.utils.enqueueLinks({
                page, selector: 'a', pseudoUrls: null, requestQueue,
            });
            await Apify.pushData({
                url: request.url,
                title: await page.title(),
                html: await page.content(),
            });
        },
    });

    await crawler.run();
    console.log('Done.');
});

如果您只想使用队列，则需要自己解析 XML。当然，这不是什么大问题。您可以在爬虫之前使用Cheerio 轻松解析它，也可以使用Apify.CheerioCrawler

无论如何，我们建议将RequestList 用于批量网址，因为它基本上是立即在内存中创建的，但队列实际上是一个数据库（或本地 JSON 文件）。