【发布时间】:2019-03-15 19:38:26
【问题描述】:
在工作中,我们会登录作为文件存储库的供应商网站。将显示文件列表。每个文件名都是一个链接。单击链接,然后下载文件。这是一个非常轻量级的网站。
我正在尝试登录并下载文件,而无需单击每个文件的繁琐任务(没有“全选”复选框)。我在一个带有开始按钮的表单上使用 WebBrowser 控件。这是代码。请跳到星号行。
Private Sub btnGo_Click(sender As Object, e As EventArgs) Handles btnGo.Click
Try
PageLoaded = False
browser.Navigate("https://[the website]/Account/Login.htm", False)
While Not PageLoaded
Application.DoEvents()
End While
Catch ex As Exception
MsgBox(ex.Message)
End Try
Try
browser.Document.GetElementById("username").InnerText = [username]
browser.Document.GetElementById("password").InnerText = [password]
PageLoaded = False
browser.Document.Forms("mainform").InvokeMember("submit")
While Not PageLoaded
Application.DoEvents()
End While
Catch ex As Exception
MsgBox(ex.Message)
End Try
' ************************************
Dim mycookies As String
mycookies = browser.Document.Cookie
' DEBUG: verified cookies are indeed present
Try
Dim cookieJar As New CookieContainer
Dim cookies As String() = browser.Document.Cookie.Split({"; "}, StringSplitOptions.RemoveEmptyEntries)
Dim cookievaluepairs() = cookies(0).Split("=")
Dim cky As New Cookie(cookievaluepairs(0), cookievaluepairs(1))
cky.Domain = browser.Document.Domain
cookieJar.Add(cky)
Dim cookievaluepairs1() = cookies(1).Split("=")
Dim cky1 As New Cookie(cookievaluepairs(0), cookievaluepairs(1))
cky1.Domain = browser.Document.Domain
cookieJar.Add(cky1)
' DEBUG: verified cookieJar contains expected cookies
Dim wwwclient As New CookieAwareWebClient(cookieJar)
' DEBUG: please see class code below
Dim x As Integer
Dim dlurl As String = ""
Dim inputs As HtmlElementCollection = browser.Document.Links
For Each elm As HtmlElement In inputs
If Microsoft.VisualBasic.Left(elm.OuterHtml, 10) = "<A href=""/" Then
dlurl = elm.GetAttribute("href")
' DEBUG: crappily named dlurl indeed has correct URI
wwwclient.DownloadFile(dlurl, "D:\Desktop\file" & x)
' DEBUG: overriden function GetWebRequest fires
' please see class code below
End If
Next
Catch ex As Exception
MsgBox(ex.Message)
' DEBUG: always lands here with 401 error
End Try
End Sub
这是在 SO 上找到的众多 CookieAwareWebClient 版本之一。
Public Class CookieAwareWebClient
Inherits WebClient
Private m_container As CookieContainer = New CookieContainer()
Public Sub New(cc As CookieContainer)
m_container = cc
' DEBUG: verified m_container now has cookieJar passed as cc
End Sub
Protected Overrides Function GetWebRequest(ByVal address As Uri) As WebRequest
Dim request As WebRequest = MyBase.GetWebRequest(address)
Dim webRequest As HttpWebRequest = TryCast(request, HttpWebRequest)
If webRequest IsNot Nothing Then
webRequest.CookieContainer = m_container
End If
Return webRequest
' DEBUG: verified webRequest.CookieContainer is correct
End Function
End Class
我单步执行代码一直到wwwclient.DownloadFile 语句,然后通过GetWebRequest 函数中的代码,停顿后,我得到一个401 Not Authorized。我发现的五六种 CookieAwareWebClient 变体都发生了这种情况。
代码成功登录后,我从 WebBrowser 控件中检索到的两个 cookie 如下所示(每次 obv 都不同)。
"samlssologgedout=SSO%20Logged%20Out"
"token=A4AA416E-46C8-11e9-92CD-005056A005E4"
我已验证这些 cookie 与“webRequest.CookieContainer”中的 cookie 相同。同样,在WebBrowser控件中,登录后可以点击文件的链接进行下载。
有人看到代码中有什么明显错误吗?
在编写问题时仍在谷歌搜索,我刚刚在 WebClient 的 MS 文档中遇到 Notes to Inheritors——“派生类应调用 WebClient 的基类实现,以确保派生类按预期工作。”
这听起来像是你会在构造函数中做的事情?还是在声明MyBase.GetWebRequest(address) 中处理了这一点?
【问题讨论】:
标签: .net vb.net web-scraping webclient