【问题标题】：How to improve the speed of the code below?如何提高下面代码的速度？
【发布时间】：2021-04-17 12:11:05
【问题描述】：

我正在尝试使用代码打击将某些页面的标题与其 URL 进行比较。如果我只有一个链接是好的.. 但是如果我尝试搜索许多链接，它的速度非常低.. 我必须等待非常多。知道如何改进它吗？

                $link = LINK;
                $page_content = file_get_contents($link);

                    if(preg_match('/<title>/',
                        $page_content, $matches)) {

                        if ( ! isset($matches[1])) {
                            echo "Page found" ;
                        }
                    } else {
                        echo "Page not found.";
                    }

【问题讨论】：

用file_get_contents() 你做一个网络请求。因此，您取决于获取内容需要多长时间的速度。您可以将其缓存一段时间，但它并不总是实时结果。
也许是因为您试图创建一个示例但是这里没有比较。代码与问题/问题不匹配？
@Steven 怎么不呢？如果您使用一些链接创建一个数据库并运行该代码，您会看到代码将有很多时间运行。我不明白你的意思。
您的代码不会比您已经建议的更有效。速度很大程度上取决于每个 url 页面的大小（因为您必须下载整个 url 内容）以及服务器与 url 页面之间的连接速度。
我的意思是您的代码只是检查是否存在<title> 标记。根本不比较？

标签： php performance

【解决方案1】：

简短说明

file_get_contents 很慢，用它抓取多个 URL 很疯狂，您必须同步等待单个请求完成，然后再发出另一个请求。

解决方案

使用 multi curl 一次打开多个连接并使用 CURLOPT_WRITEFUNCTION 异步收集响应。

基准测试

让我们尝试通过抓取 12 个 GitHub 链接来对它们进行基准测试。这个基准测试表明，multi curl 可以比file_get_contents 快大约 12.19 倍来抓取 12 个 GitHub URL。我相信随着链接数量的增加，差异会显着增加。

file_get_contents 基准测试

文件：single.php

<?php

$links = [
    "https://github.com/TeaInside/teavpn2",
    "https://github.com/ammarfaizi2/php-integral-obfuscator",
    "https://github.com/ammarfaizi2/fresh-tea-asm",
    "https://github.com/ammarfaizi2/memcpy_benchmark",
    "https://github.com/ammarfaizi2/GoogleTranslate",
    "https://github.com/ammarfaizi2/latex.teainside.org",
    "https://github.com/torvalds/linux",
    "https://github.com/torvalds/uemacs",
    "https://github.com/torvalds/subsurface-for-dirk",
    "https://github.com/gcc-mirror/gcc",
    "https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/Makefile",
    "https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/LICENSE",
];

foreach ($links as $link) {
    $page_content = file_get_contents($link);
    if (preg_match("/<title>/S", $page_content, $m)) {
        echo "Link {$link} has a title\n";
    } else {
        echo "Link {$link} does not have a title\n";
    }
}

执行

root@esteh:/tmp# time php single.php
Link https://github.com/TeaInside/teavpn2 has a title
Link https://github.com/ammarfaizi2/php-integral-obfuscator has a title
Link https://github.com/ammarfaizi2/fresh-tea-asm has a title
Link https://github.com/ammarfaizi2/memcpy_benchmark has a title
Link https://github.com/ammarfaizi2/GoogleTranslate has a title
Link https://github.com/ammarfaizi2/latex.teainside.org has a title
Link https://github.com/torvalds/linux has a title
Link https://github.com/torvalds/uemacs has a title
Link https://github.com/torvalds/subsurface-for-dirk has a title
Link https://github.com/gcc-mirror/gcc has a title
Link https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/Makefile does not have a title
Link https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/LICENSE does not have a title

real    0m14.721s
user    0m0.070s
sys 0m0.025s
root@esteh:/tmp#

抓取12个链接需要14.721s

多卷曲基准

文件：multiple.php

<?php 

$links = [
    "https://github.com/TeaInside/teavpn2",
    "https://github.com/ammarfaizi2/php-integral-obfuscator",
    "https://github.com/ammarfaizi2/fresh-tea-asm",
    "https://github.com/ammarfaizi2/memcpy_benchmark",
    "https://github.com/ammarfaizi2/GoogleTranslate",
    "https://github.com/ammarfaizi2/latex.teainside.org",
    "https://github.com/torvalds/linux",
    "https://github.com/torvalds/uemacs",
    "https://github.com/torvalds/subsurface-for-dirk",
    "https://github.com/gcc-mirror/gcc",
    "https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/Makefile",
    "https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/LICENSE",
];

function visit_multiple(array $links, array &$retval): void
{
    $chs = [];
    $mh = curl_multi_init();

    foreach ($links as $k => $link) {
        $ch = curl_init($link);
        $chs[] = $ch;
        $retval[$k] = "";
        $writeCallback = function ($ch, string $str) use ($k, &$retval) {
            $retval[$k] .= $str;
            return strlen($str);
        };

        curl_setopt_array($ch,
            [
                CURLOPT_USERAGENT => "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
                CURLOPT_WRITEFUNCTION => $writeCallback
            ]
        );
        curl_multi_add_handle($mh, $ch);
    }

    do {
        $status = curl_multi_exec($mh, $active);
        if ($active) {
            curl_multi_select($mh);
        }
    } while ($active && $status == CURLM_OK);

    foreach ($chs as $ch) {
        curl_multi_remove_handle($mh, $ch);
    }

    curl_multi_close($mh);
}

$retval = [];
visit_multiple($links, $retval);

foreach ($retval as $k => $page_content) {
    if (preg_match("/<title>/S", $page_content, $m)) {
        echo "Link {$links[$k]} has a title\n";
    } else {
        echo "Link {$links[$k]} does not have a title\n";
    }
}

执行

root@esteh:/tmp# time php multiple.php
Link https://github.com/TeaInside/teavpn2 has a title
Link https://github.com/ammarfaizi2/php-integral-obfuscator has a title
Link https://github.com/ammarfaizi2/fresh-tea-asm has a title
Link https://github.com/ammarfaizi2/memcpy_benchmark has a title
Link https://github.com/ammarfaizi2/GoogleTranslate has a title
Link https://github.com/ammarfaizi2/latex.teainside.org has a title
Link https://github.com/torvalds/linux has a title
Link https://github.com/torvalds/uemacs has a title
Link https://github.com/torvalds/subsurface-for-dirk has a title
Link https://github.com/gcc-mirror/gcc has a title
Link https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/Makefile does not have a title
Link https://raw.githubusercontent.com/TeaInside/teavpn2/0.0.1-rc1/LICENSE does not have a title

real    0m1.207s
user    0m0.165s
sys 0m0.042s
root@esteh:/tmp#

抓取12个链接仅需1.207s

奖金

为了满足评论中的 OP 请求，我添加了标题检查。如果有标题但标题中包含“找不到页面”，那么我们假设该页面没有标题。

这是一项微不足道的任务，但没关系。

代码

<?php 

$links = [
    "https://amzn.to/3x4dxLF", // found
    "https://amzn.to/3qQcrQf", // not found
    "https://github.com/TeaInside/teavpn2", // found
    "https://github.com/not_found_page", // not found
];

function visit_multiple(array $links, array &$retval): void
{
    $chs = [];
    $mh = curl_multi_init();

    $cwd = getcwd();
    foreach ($links as $k => $link) {
        $ch = curl_init($link);
        $chs[] = $ch;
        $retval[$k] = "";
        $writeCallback = function ($ch, string $str) use ($k, &$retval) {
            $retval[$k] .= $str;
            return strlen($str);
        };

        curl_setopt_array($ch,
            [
                CURLOPT_FOLLOWLOCATION => true,
                CURLOPT_USERAGENT => "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
                CURLOPT_WRITEFUNCTION => $writeCallback,
                CURLOPT_COOKIEJAR => $cwd."/my_cookie{$k}.txt",
                CURLOPT_COOKIEFILE => $cwd."/my_cookie{$k}.txt",
                CURLOPT_ENCODING => "gzip",
                CURLOPT_HTTPHEADER => [
                    "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
                    "Accept-Language: en-US,en;q=0.5",
                    "Upgrade-Insecure-Requests: 1",
                ],
            ]
        );
        curl_multi_add_handle($mh, $ch);
    }

    do {
        $status = curl_multi_exec($mh, $active);
        if ($active) {
            curl_multi_select($mh);
        }
    } while ($active && $status == CURLM_OK);

    foreach ($chs as $k => $ch) {
        curl_multi_remove_handle($mh, $ch);
        @unlink($cwd."/my_cookie{$k}.txt");
    }

    curl_multi_close($mh);
}

$retval = [];
visit_multiple($links, $retval);

foreach ($retval as $k => $page_content) {
    if (preg_match("/<title(?:\s.*?)?>(.+?)<\/title>/isS", $page_content, $m)) {
        if (!preg_match("/Page not found/i", $m[1])) {
            echo "Link {$links[$k]} has a title\n";
            continue;
        }
    }
    echo "Link {$links[$k]} does not have a title\n";
}

执行

ammarfaizi2@integral:~$ php test.php
Link https://amzn.to/3x4dxLF has a title
Link https://amzn.to/3qQcrQf does not have a title
Link https://github.com/TeaInside/teavpn2 has a title
Link https://github.com/not_found_page does not have a title
ammarfaizi2@integral:~$

【讨论】：

这个速度是惊人的，但如果标题在 url 内则无法比较
“比较标题是否在 url 内”是什么意思？它坦率地检查响应是否包含<title>。
是的，但是有些页面的标题是 Page not found 并且 URL 类似于 link.com/the-page-should-be-here 。所以我想要做的是取消标题并检查是否可以在 url 上找到，所以返回可以是 Yes it is，或者 yes is not。
您只需自定义您的验证，如果您在标题上看到“找不到页面”，则返回“否”。这里的重点是性能，您应该根据自己的需要调整给定的解决方案。
很抱歉占用了您的时间.. 但我真的不明白如何使用您的解决方案将标题作为变量来检查它...如果您也可以在这方面给我一点帮助的话赞赏。