【发布时间】：2023-03-12 21:55:01
【问题描述】：
我想创建一个 Google 脚本来检查给定的 URL 是否被 Google 索引，所以我编写了以下函数：
function CheckURLForGoogleIndex(url, activesheet) {// Delete the https:// and http:// prefix
var cururl = url.replace("https://", "");
cururl = cururl.replace("http://", "");
var googlesearchurl = "https://www.google.com/search?q=site:" + encodeURIComponent(cururl);
var page = UrlFetchApp.fetch(googlesearchurl, {muteHttpExceptions: true}).getContentText();
// Wait for 1 second before starting another fetch
Utilities.sleep(1000);
var number = page.match("did not match any documents");
if (number) {
activesheet.getSheetByName("Not Google Index").appendRow([url]);
} else {
activesheet.getSheetByName("Google Index").appendRow([url]);
}
}
但是，在调试代码的时候，在调用UrlFetchApp.fetch之后，我只能看到变量page的header。
我尝试使用 Google 索引 URL 而不是索引 URL 来测试该函数，但两者都会在 page.match 函数中返回 null，因此两者都放在“Google 索引”表中。
我的函数有什么问题？
谢谢
注意：
我在https://groups.google.com/g/google-apps-script-community/c/gs1qUuKwgn4上问过这个问题，但没有人回答，所以我必须在这里问。
示例输入和输出
输入1：
网址 = https://www.datanumen.com/
activesheet = 包含工作表“Google Index”和“Not Google Index”的 GoogleSheet
预期输出 1：由于 https://www.datanumen.com/ 已被 Google 索引，它将被添加到“Google 索引”表中。
page = "<!doctype html><html lang="en"><head><meta charset="UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>site:www.datanumen.com/ - Google Search…"
输入2：
网址 = https://www.datanumen.com/notindexedurl/
activesheet = 包含工作表“Google Index”和“Not Google Index”的 GoogleSheet
预期输出 2：由于 https://www.datanumen.com/notindexedurl/ 未被 Google 索引，它将被添加到“NOT Google Index”表中。
page = "<!doctype html><html lang="en"><head><meta charset="UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>site:www.datanumen.com/notindexurl/ - G…"
问题目前是 Input1 和 Input2 的问题，实际结果是：URL 将始终添加到“Google 索引”表，因为搜索结果根本不会包含“不匹配任何文档”文本。
更新
我添加了 console.log(page);并再次调试。对于 Input1，我得到以下结果：
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>https://www.google.com/search?q=site:www.datanumen.com%2F</title></head>
<body style="font-family: arial, sans-serif; background-color: #fff; color: #000; padding:20px; font-size:18px;" onload="e=document.getElementById('captcha');if(e){e.focus();}">
<div style="max-width:400px;">
<hr noshade size="1" style="color:#ccc; background-color:#ccc;"><br>
<form id="captcha-form" action="index" method="post">
<script src="https://www.google.com/recaptcha/api.js" async defer></script>
<script>var submitCallback = function(response) {document.getElementById('captcha-form').submit();};</script>
<div id="recaptcha" class="g-recaptcha" data-sitekey="6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b" data-callback="submitCallback" data-s="c5Hy4maqTFv3SzYRiWhpsqYF2isZmauUQnLVljOiED_PiaVWJWCsHMzRAZyh8HLCBHJ_mjET7yODJu8AlZ33_xGAQ8TcKuXAd7rQpsYakaGKPD8USiGSFhiII2ai-Cf_B26i1Ufpko-qYQ8V3rezhiSXxi5J2yHZ-_WwEj8ukzy5znxzVurTM_2cY243Q4ofwP7E7eWBaHIg6N3ofmPuFXd-uRIUU4z0cU_pas8"></div>
<input type='hidden' name='q' value='EgRrsuB5GKmx6oUGIhBKAdWty9nssg-nAtyy9n7hMgFy'><input type="hidden" name="continue" value="https://www.google.com/search?q=site:www.datanumen.com%2F">
</form>
<hr noshade size="1" style="color:#ccc; background-color:#ccc;">
<div style="font-size:13px;">
<b>About this page</b><br><br>
Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. <a href="#" onclick="document.getElementById('infoDiv').style.display='block';">Why did this happen?</a><br><br>
<div id="infoDiv" style="display:none; background-color:#eee; padding:10px; margin:0 0 15px 0; line-height:1.4em;">
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the <a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop. In the meantime, solving the above CAPTCHA will let you continue to use our services.<br><br>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible. <a href="//support.google.com/websearch/answer/86640">Learn more</a><br><br>Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.
</div>
IP address: 107.178.224.121<br>Time: 2021-06-04T21:18:34Z<br>URL: https://www.google.com/search?q=site:www.datanumen.com%2F<br>
</div>
</div>
</body>
</html>
【问题讨论】：
-
为了正确理解您的问题，您能否提供您期望的示例输入和输出值？通过这个，我想试着理解你的问题。
-
看起来你不能这样做。当我尝试时，我得到了不寻常的流量页面：support.google.com/websearch/answer/86640
-
你能分享你从
page = UrlFetchApp.fetch(url, {muteHttpExceptions: true}).getContentText()得到的输出吗（另外，这里的url不应该是googlesearchurl）吗？ -
@Tanaike，我已更新问题以包含两个示例输入和输出。
-
@RafaGuillermo，谢谢。我已将 url 替换为 googlesearchurl。但即使在此之后，如果我调试代码，那么对于 Input2，URL 将正确添加到“非 Google 索引”表中。但是，如果运行代码，那么所有 URL 都将添加到“Google 索引”表中，即使是输入 2。可能问题是由您评论中提到的异常流量页面引起的。但我看不到。