一些网站即使设置了标头也会拒绝 HttpClient 请求答案

【问题标题】：Some websites rejecting HttpClient requests even with headers set一些网站即使设置了标头也会拒绝 HttpClient 请求
【发布时间】：2016-09-28 10:59:24
【问题描述】：

我编写了一些代码来检查我的数据库中的所有网站是否仍然托管和在线。

问题是其中一些网站似乎有机器人保护，每当我尝试通过 HttpClient 请求时，它们都会引发错误而不是显示页面。

我看到其他类似的问题建议添加浏览器标题，所以我这样做了，但这没有帮助。相同的站点仍然拒绝 HttpClient 连接，但当我在浏览器中查看它们时完全没问题。

我的代码有问题还是需要一些额外的步骤？

这是我的代码：

public static async Task CheckSite(string url, int id)
{
    try
    {
        using(var db = new PlaceDBContext())
        using (HttpClient client = new HttpClient(new HttpClientHandler()
        {
            AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
        }))
        using (HttpResponseMessage response = await client.GetAsync(url))
        using (HttpContent content = response.Content)
        {
            client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
            client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
            client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
            client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");

            var rd = db.RootDomains.Find(id);
            string result = await content.ReadAsStringAsync();


            if (result != null && result.Length >= 50)
            {
                Console.WriteLine("fine");
                rd.LastCheckOnline = true;
            }
            else
            {
                Console.WriteLine("There was empty or short result");
                rd.LastCheckOnline = false;
            }
            db.SaveChanges();
            semaphore.Release();
        }
    }
    catch(Exception ex)
    {

        Console.WriteLine(ex.Message);
        using(var db = new PlaceDBContext())
        {
            var rd = db.RootDomains.Find(id);
            rd.LastCheckOnline = false;
            db.SaveChanges();
            semaphore.Release();
        }
    }

}

【问题讨论】：

您在发送请求后设置标题。你需要移动你的client.DefaultRequestHeaders.TryAddWithoutValidation在client.GetAsync()之前执行
在发送请求之前设置标头。您在收到回复后才开始这样做

标签： c# .net dotnet-httpclient

【解决方案1】：

在发送请求之前设置标头。您在收到回复后才开始这样做

public static async Task CheckSite(string url, int id) {
    try {
        using (var db = new PlaceDBContext())
        using (var client = new HttpClient(new HttpClientHandler() {
            AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
        })) {
            client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
            client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
            client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");

            using (var response = await client.GetAsync(url))
            using (var content = response.Content) {
                client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");

                var rd = db.RootDomains.Find(id);
                string result = await content.ReadAsStringAsync();


                if (result != null && result.Length >= 50) {
                    Console.WriteLine("fine");
                    rd.LastCheckOnline = true;
                } else {
                    Console.WriteLine("There was empty or short result");
                    rd.LastCheckOnline = false;
                }
                db.SaveChanges();
                semaphore.Release();
            }
        }
    } catch (Exception ex) {

        Console.WriteLine(ex.Message);
        using (var db = new PlaceDBContext()) {
            var rd = db.RootDomains.Find(id);
            rd.LastCheckOnline = false;
            db.SaveChanges();
            semaphore.Release();
        }
    }

}

【讨论】：