powershell httpwebrequest GET方法cookiecontainer问题？

Question

我正在尝试抓取具有用户身份验证的网站。我可以做一个 POST 来发送我的登录信息并存储一个 cookie。但是，登录后我在尝试访问受保护的页面时收到 403 错误。

$url = "https://some_url"

$CookieContainer = New-Object System.Net.CookieContainer

$postData = "User=UserName&Password=Pass"

$buffer = [text.encoding]::ascii.getbytes($postData)

[net.httpWebRequest] $req = [net.webRequest]::create($url)
$req.method = "POST"
$req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
$req.Headers.Add("Accept-Language: en-US")
$req.Headers.Add("Accept-Encoding: gzip,deflate")
$req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7")
$req.AllowAutoRedirect = $false
$req.ContentType = "application/x-www-form-urlencoded"
$req.ContentLength = $buffer.length
$req.TimeOut = 50000
$req.KeepAlive = $true
$req.Headers.Add("Keep-Alive: 300");
$req.CookieContainer = $CookieContainer
$reqst = $req.getRequestStream()
$reqst.write($buffer, 0, $buffer.length)
$reqst.flush()
$reqst.close()
[net.httpWebResponse] $res = $req.getResponse()
$resst = $res.getResponseStream()
$sr = new-object IO.StreamReader($resst)
$result = $sr.ReadToEnd()
$res.close()



$url2 = "https://some_url/protected_page"

[net.httpWebRequest] $req2 = [net.webRequest]::create($url2)
$req2.Method = "GET"
$req2.Accept = "text/html"
$req2.AllowAutoRedirect = $false
$req2.CookieContainer = $CookieContainer
$req2.TimeOut = 50000
[net.httpWebResponse] $res2 = $req2.getResponse()
$resst = $res2.getResponseStream()
$sr = new-object IO.StreamReader($resst)
$result = $sr.ReadToEnd()

解决方法：所以在尝试了几乎所有方法之后，我最终尝试了一些不同的东西并且它确实有效。

在发布登录信息并获取会话 cookie 后，我使用 webclient 通过将 cookie 字符串添加到标题中来访问安全页面。

$web = new-object net.webclient
$web.Headers.add("Cookie", $res.Headers["Set-Cookie"])
$result = $web.DownloadString("https://secure_url")

其中一个很酷的事情是 webclient 保存 cookie。要访问另一个安全页面，您只需调用 $web.downloadstring("https://another_secure_url") :)

Answer 1

我发现由于 cookie 可以附加附加信息（例如 URL 或仅 HTTP），因此 $res.Headers["Set-Cookie"] 对我不起作用。但是使用您的 $CookieContainer 变量，您可以轻松地将其更改为使用 GetCookieHeader(url)，这将去除多余的信息并为您留下一个格式正确的 cookie 字符串：

$web = new-object net.webclient
$web.Headers.add("Cookie", $CookieContainer.GetCookieHeader($url))
$result = $web.DownloadString($url)

Answer 2

人们一直在要求完整的应用程序，这里有它

$url = "https://some_url"

$CookieContainer = New-Object System.Net.CookieContainer

$postData = "User=UserName&Password=Pass"

$buffer = [text.encoding]::ascii.getbytes($postData)

[net.httpWebRequest] $req = [net.webRequest]::create($url)
$req.method = "POST"
$req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
$req.Headers.Add("Accept-Language: en-US")
$req.Headers.Add("Accept-Encoding: gzip,deflate")
$req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7")
$req.AllowAutoRedirect = $false
$req.ContentType = "application/x-www-form-urlencoded"
$req.ContentLength = $buffer.length
$req.TimeOut = 50000
$req.KeepAlive = $true
$req.Headers.Add("Keep-Alive: 300");
$req.CookieContainer = $CookieContainer
$reqst = $req.getRequestStream()
$reqst.write($buffer, 0, $buffer.length)
$reqst.flush()
$reqst.close()
[net.httpWebResponse] $res = $req.getResponse()
$resst = $res.getResponseStream()
$sr = new-object IO.StreamReader($resst)
$result = $sr.ReadToEnd()
$res.close()


$web = new-object net.webclient
$web.Headers.add("Cookie", $res.Headers["Set-Cookie"])
$result = $web.DownloadString("https://secure_url")

Answer 3

我会使用IE automation。有了这个，就不必使用 cookie、标题等。容易得多。