从网站抓取数据

Question

您好，我有一个网站，我想从中获取数据，我可以从网站上抓取所有文本，但我想要的是特定的数据。 我想从中获取数据的示例网站是 http://www.irishferries.com/ie-en/to-britain-from-ireland/

首先，我必须选择路线，然后获取并选择月份，然后选择航行可用的可用天数，然后选择可用时间，

我现在所有的都在下面

Imports System.Net
Imports System.IO
Imports System.Text.RegularExpressions

Public Class Form1


Private Sub scrapeButton_Click(sender As Object, e As EventArgs) Handles scrapeButton.Click

    If (Not linkURL.Text = Nothing) Then
        linkURL.Text = linkURL.Text.ToLower()
        If (linkURL.Text.StartsWith("https://") Or linkURL.Text.StartsWith("http://")) Then
            If (Not linkURL.Text.StartsWith("https://www.") And Not linkURL.Text.StartsWith("http://www.")) Then
                If Not (linkURL.Text.StartsWith("www.")) Then
                    If (linkURL.Text.StartsWith("http://")) Then
                        linkURL.Text = "http://www." & linkURL.Text.Substring(7, linkURL.Text.Length - 7)
                    Else
                        linkURL.Text = "https://www." & linkURL.Text.Substring(8, linkURL.Text.Length - 8)
                    End If
                End If
            End If
        ElseIf (linkURL.Text.StartsWith("www.")) Then
            linkURL.Text = "http://" & linkURL.Text
        Else
            linkURL.Text = "http://www." & linkURL.Text
        End If
    End If

    Dim req As WebRequest = WebRequest.Create(linkURL.Text)
    Dim res As HttpWebResponse = req.GetResponse()
    Dim src As String = New StreamReader(res.GetResponseStream()).ReadToEnd()

    srcBox.Text = src
end sub
end class

我不知道如何与网站交互以选择路线然后从中提取信息， 任何帮助将不胜感激。

Answer 1

查看该网站后，它似乎正在执行 ajax 调用来检索您正在寻找的信息。如果您运行 Fiddler 或 Chrome 的网络分析器，您可以看到它在后台进行 GET 调用。当您选择路线时，它会进行 ajax 调用，然后使用 json 数据填充日期和出发时间选择选项。

因此，一种可能的方法是重新创建初始 ajax 调用并解析返回的 json 数据以构建后续调用以获取其余数据。这样您就不必担心与 html 页面或其元素的交互。

编辑

这是一个使用 ajax 调用之一的示例。调用 doGetRouteData。它有一个表示返回的 JSON 对象的类。但是有一个问题......最后一行在尝试反序列化 json 数据时抛出错误。我不知道为什么，也许你可以弄清楚。

Imports System.Runtime.Serialization.Json
Imports System.Runtime.Serialization
Imports System.IO
Imports System.Text

Module Module1

<DataContract()> _
Public Class RouteData
    Implements IExtensibleDataObject
    ' To implement the IExtensibleDataObject interface, you must also 
    ' implement the ExtensionData property. 
    Private extensionDataObjectValue As ExtensionDataObject

    Public Property ExtensionData() As ExtensionDataObject _
      Implements IExtensibleDataObject.ExtensionData
        Get
            Return extensionDataObjectValue
        End Get
        Set(value As ExtensionDataObject)
            extensionDataObjectValue = value
        End Set
    End Property

    <DataMember(Name:="Departure")> _
    Public Departure As String '{ get; set; } 
    <DataMember(Name:="Arrival")> _
    Public Arrival As String '{ get; set; } 
    <DataMember(Name:="SameDay")> _
    Public SameDay As Boolean '{ get; set; } 
    <DataMember(Name:="Route")> _
    Public Route As String '{ get; set; } 

End Class

Public Sub doGetRouteData()
    Dim webRequest = System.Net.HttpWebRequest.Create("http://www.irishferries.com/Service/BookingInformation.svc/SailingTimes/HHDDUB/2014-5-20?version=22")
    webRequest.Method = "GET"
    webRequest.ContentType = "application/json"
    Dim responseReader As StreamReader = New StreamReader(webRequest.GetResponse().GetResponseStream())
    Dim responseData As String = responseReader.ReadToEnd()

    responseReader.Close()
    webRequest.GetResponse().Close()

    Dim ser As New DataContractJsonSerializer(GetType(RouteData))
    Dim ByteConverter As New UnicodeEncoding()
    Dim newRouteData As New List(Of RouteData)
    Dim stream = New MemoryStream(ByteConverter.GetBytes(responseData))
    newRouteData = CType(ser.ReadObject(stream), List(Of RouteData)) '<= throws an error here, haven't figured out why

End Sub

End Module