无法使用 curl / php 获取公共 Linkedin 公司页面答案

【问题标题】：Can't get public Linkedin Company page with curl / php无法使用 curl / php 获取公共 Linkedin 公司页面
【发布时间】：2020-08-12 18:00:10
【问题描述】：

我正在使用以下代码尝试将公共 Linkedin 公司页面放入变量中，但它总是返回 Linkedin 的页面未找到 404。知道我哪里出错了吗？

$html = get_web_page('https://www.linkedin.com/company/google/');
echo stripos( $html['content'], 'occludable-update' );
echo $html['content'];

function get_web_page( $url )
{
        $user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';

        $options = array(

                CURLOPT_CUSTOMREQUEST  =>"GET",        //set request type post or get
                CURLOPT_POST           =>false,        //set to GET
                CURLOPT_USERAGENT      => $user_agent, //set user agent
                CURLOPT_COOKIEFILE     =>"cookie.txt", //set cookie file
                CURLOPT_COOKIEJAR      =>"cookie.txt", //set cookie jar
                CURLOPT_RETURNTRANSFER => true,     // return web page
                CURLOPT_HEADER         => false,    // don't return headers
                CURLOPT_FOLLOWLOCATION => true,     // follow redirects
                CURLOPT_ENCODING       => "",       // handle all encodings
                CURLOPT_AUTOREFERER    => true,     // set referer on redirect
                CURLOPT_CONNECTTIMEOUT => 120,      // timeout on connect
                CURLOPT_TIMEOUT        => 120,      // timeout on response
                CURLOPT_MAXREDIRS      => 10,       // stop after 10 redirects
        );

        $ch      = curl_init( $url );
        curl_setopt_array( $ch, $options );
        $content = curl_exec( $ch );
        $err     = curl_errno( $ch );
        $errmsg  = curl_error( $ch );
        $header  = curl_getinfo( $ch );
        curl_close( $ch );

        $header['errno']   = $err;
        $header['errmsg']  = $errmsg;
        $header['content'] = $content;
        return $header;
}

【问题讨论】：

标签： php curl web-scraping linkedin

【解决方案1】：

他们必须有某种适当的刮擦保护。如果您通过 CLI 使用 curl 获取页面，您可以看到它只返回一些 Javascript 代码：

$ curl https://www.linkedin.com/company/google/
<html><head>
<script type="text/javascript">
window.onload = function() {
  // Parse the tracking code from cookies.
  var trk = "bf";
  var trkInfo = "bf";
  var cookies = document.cookie.split("; ");
  for (var i = 0; i < cookies.length; ++i) {
    if ((cookies[i].indexOf("trkCode=") == 0) && (cookies[i].length > 8)) {
      trk = cookies[i].substring(8);
    }
    else if ((cookies[i].indexOf("trkInfo=") == 0) && (cookies[i].length > 8)) {
      trkInfo = cookies[i].substring(8);
    }
  }

  if (window.location.protocol == "http:") {
    // If "sl" cookie is set, redirect to https.
    for (var i = 0; i < cookies.length; ++i) {
      if ((cookies[i].indexOf("sl=") == 0) && (cookies[i].length > 3)) {
        window.location.href = "https:" + window.location.href.substring(window.location.protocol.length);
        return;
      }
    }
  }

  // Get the new domain. For international domains such as
  // fr.linkedin.com, we convert it to www.linkedin.com
  var domain = "www.linkedin.com";
  if (domain != location.host) {
    var subdomainIndex = location.host.indexOf(".linkedin");
    if (subdomainIndex != -1) {
      domain = "www" + location.host.substring(subdomainIndex);
    }
  }

  window.location.href = "https://" + domain + "/authwall?trk=" + trk + "&trkInfo=" + trkInfo +
      "&originalReferer=" + document.referrer.substr(0, 200) +
      "&sessionRedirect=" + encodeURIComponent(window.location.href);
}
</script>
</head></html>

【讨论】：