【问题标题】:Web Scraping Elements By Class & Tag name按类和标签名称抓取元素
【发布时间】:2019-09-09 16:13:20
【问题描述】:

我正在尝试从下面提到的网站复制数据,我需要各种尺寸、价格、设施、特价、储备。我在下面的代码框架,但我能够正确复制元素。首先,只有三个元素在处理重复问题,我也没有得到设施和储备的结果。有人可以看看这个吗?

Sub text()


Dim ie As New InternetExplorer, ws As Worksheet
Set ws = ThisWorkbook.Worksheets("Unit Data")
With ie
    .Visible = True
    .Navigate2 "https://www.safeandsecureselfstorage.com/self-storage-lake-villa-il-86955" 

    While .Busy Or .readyState < 4: DoEvents: Wend

    Sheets("Unit Data").Select


    Dim listings As Object, listing As Object, headers(), results()
    Dim r As Long, list As Object, item As Object
    headers = Array("size", "features", "Specials", "Price", "Reserve")
    Set list = .document.getElementsByClassName("units_table")
    '.unit_size medium, .features, .Specials, .price, .Reserve
    Dim rowCount As Long
    rowCount = .document.querySelectorAll(".tab_container li").Length
    ReDim results(1 To rowCount, 1 To UBound(headers) + 1)
    For Each listing In list
            For Each item In listing.getElementsByClassName("unitinfo even")
            r = r + 1

          results(r, 1) = listing.getElementsByClassName("size secondary-color-text")(0).innerText
          results(r, 2) = listing.getElementsByClassName("amenities")(0).innerText
           results(r, 3) = listing.getElementsByClassName("offer1")(0).innerText
        results(r, 4) = listing.getElementsByClassName("rate_text primary-color-text rate_text--clear")(0).innerText
          results(r, 5) = listing.getElementsByClassName("reserve")(0).innerText





        Next
    Next
    ws.Cells(1, 1).Resize(1, UBound(headers) + 1) = headers
    ws.Cells(2, 1).Resize(UBound(results, 1), UBound(results, 2)) = results
    .Quit
End With

  Worksheets("Unit Data").Range("A:G").Columns.AutoFit
End Sub

【问题讨论】:

    标签: html excel vba web-scraping screen-scraping


    【解决方案1】:

    tl;dr;

    提前(对某些人)为答案长度道歉,但我想我会接受这个 教学时刻详细说明正在发生的事情。

    我使用的总体方法与您的代码中的相同:找到一个 css 选择器来隔离行(尽管在不同的选项卡中,小、中、大实际上仍然都存在于页面上):

    Set listings = html.querySelectorAll(".unitinfo")
    

    上面生成了行。和以前一样,我们将其转储到一个新的 HTMLDocument 中,以便我们可以利用 querySelector/querySelectorAll 方法。


    行:

    让我们看看我们正在检索的第一行 html。后续部分将以这一行作为案例研究,讨论如何检索信息:

    5x5</TD> <TD class=features>
    <DIV id=a5x5-1 class="icon a5x5">
    <DIV class=img><IMG src="about:/core/resources/images/units/5x5_icon.png"></DIV>
    <DIV class=display>
    <P>More Information</P></DIV></DIV>
    <SCRIPT type=text/javascript>
                    // Refine Search
                    //
                    $(function() {
                        $("#a5x5-1").tooltip({
                            track: false,
                            delay: 0,
                            showURL: false,
                            left: 5,
                            top: 5,
                            bodyHandler: function () {
                                return "        <div class=\"tooltip\">            <div class=\"tooltop\"></div>            <div class=\"toolmid clearfix\">                <div class=\"toolcontent\">                    <div style=\"text-align:center;width:100%\">                        <img alt=\"5 x 5 storage unit\" src=\"/core/resources/images/units/5x5.png\" />                    </div>                    <div class=\"display\">5 x 5</div>                    <div class=\"description\">Think of it like a standard closet. Approximately 25 square feet, this space is perfect for about a dozen boxes, a desk and chair, and a bicycle.</div>                </div>                <div class=\"clearfix\"></div>            </div>            <div class=\"toolfoot\"></div>            <div class=\"clearfix\"></div>        </div>        "}
                        });
                    });
            </SCRIPT>
    </TD><TD class=rates>
    <DIV class="discount_price secondary-color-text standard_price--left">
    <DIV class=price_text>Web Rate: </DIV>
    <DIV class="rate_text primary-color-text rate_text--clear">$39.00 </DIV></DIV>
    <SCRIPT>
    $( document ).ready(function() {
        $('.units_table tr.unitinfo').each(function(index, el) {
            if ($(this).find('.standard_price').length != '' && $(this).find('.discount_price').length != '') {
                $(this).parents('.units_table').addClass('both');
                $(this).addClass('also-both');
                $(this).find('.rates').addClass('rates_two_column');
            }
        });
    });
    </SCRIPT>
    </TD><TD class=amenities>
    <DIV title="Temperature Controlled" class="amenity_icon icon_climate"></DIV>
    <DIV title="Interior Storage" class="amenity_icon icon_interior"></DIV>
    <DIV title="Ground Floor" class="amenity_icon icon_ground_floor"></DIV></TD><TD class=offers>
    <DIV class=offer1>Call for Specials </DIV>
    <DIV class=offer2></DIV></TD><TD class=reserve><A id=5x5:39:00000000 class="facility_call_to_reserve cta_call primary-color primary-hover" href="about:blank#" rel=nofollow>Call </A></TD>

    我们将要处理的每一行在html2 变量中都有相似的html。如果您有疑问,请查看上面显示的函数中的 javascript:

    $('.units_table tr.unitinfo').each(function(index, el) 
    

    它使用相同的选择器(但也指定父表类和元素类型 (tr))。基本上,表中的每一行都会调用该函数。


    尺寸:

    由于某种原因,开头的td 标记被删除(我认为这个缺少父标记&lt;table&gt;)所以对于大小,而不是按类抓取,我正在寻找结束的开始标记并将字符串提取到那里。我通过将Instr(其中 Left$(类型化)函数来做到这一点。

    results(r, 1) = Left$(html2.body.innerHTML, InStr(html2.body.innerHTML, "<") - 1)
    

    这将返回5x5


    说明:

    description 列由我们在上面看到的函数填充(记住应用于每一行)

    这个位 - $("#a5x5-1").tooltip - 告诉它定位到哪里,然后函数的返回语句提供具有div的html,类description,包含我们想要的文本。由于我们没有使用浏览器,而且我使用的是 64 位 Windows,因此我无法评估此脚本,但我可以使用 split 来提取 "description\"&gt; 和关闭关联 @987654351 的开始之间的字符串(描述) @标签:

    results(r, 2) = Split(Split(html2.querySelector("SCRIPT").innerHTML, """description\"">")(1), "</div>")(0)
    

    这会返回:

    “把它想象成一个标准的壁橱。大约 25 平方英尺,这个空间非常适合放置十几个盒子、一张桌椅和一辆自行车。”


    费率类型和价格:

    这些很简单,使用类名来定位:

    results(r, 3) = Replace$(html2.querySelector(".price_text").innerText, ":", vbNullString)
    results(r, 4) = Trim$(html2.querySelector(".rate_text").innerText)
    

    返回(分别)

    网络速率 , 39.00 英镑


    便利设施:

    这就是事情有点棘手的地方。

    让我们重新检查上面显示的 html,对于这一行,它与便利设施有关:

    <TD class=amenities>
    <DIV title="Temperature Controlled" class="amenity_icon icon_climate"></DIV>
    <DIV title="Interior Storage" class="amenity_icon icon_interior"></DIV>
    <DIV title="Ground Floor" class="amenity_icon icon_ground_floor"></DIV></TD>

    我们可以看到父td 有一个amenities 的类,它的子div 元素具有复合类名;后者在每种情况下都用作便利设施类型的标识符,例如icon_climate.

    当您将鼠标悬停在这些页面上时,会在页面上显示工具提示信息:

    我们可以在实际页面的html中追踪到这个tooltip的位置:

    当您将鼠标悬停在不同的便利设施上时,此内容会更新。

    长话短说(他说在页面的一半处!),此内容正在从服务器上的 php 文件更新。我们可以请求文件并构造一个字典,映射每个便利设施的类名,例如amenity_icon icon_climate(在转换为 .amenity_icon.icon_climate 的适当 css 选择器时,复合类需要“”替换为“.”)到相关描述。您可以探索 php 文件here

    php 文件:

    让我们只看文件的开头,以便剖析什么是重复模式的基本单位:

    function LoadTooltips() {
            $(".units_table .amenity_icon.icon_climate").tooltip({
            track: false,
            delay: 0,
            showURL: false,
            left: -126,
            top: -100,
            bodyHandler: function () {
                return "<div class=\"sidebar_tooltip\"><h4>Temperature Controlled</h4><p>Units are heated and/or cooled. See manager for details.</p></div>"
            }
        });

    负责更新工具提示的函数是LoadTooltips。 CSS 类选择器用于定位每个图标:

    $(".units_table .amenity_icon.icon_climate").tooltip
    

    我们有 bodyhandler 指定返回文本:

    bodyHandler: function () {
                return "<div class=\"sidebar_tooltip\"><h4>Temperature Controlled</h4><p>Units are heated and/or cooled. See manager for details.</p></div>"

    我们有三个有用的信息,它们出现在重复的组中。元素的类名选择器、简短描述和详细描述,例如

    1. .amenity_icon.icon_climate :我们使用它来将 php 文件描述映射到我们行中的便利图标的类名。 CSS 选择器
    2. Temperature Controlled;在工具提示函数的 h4 标签内返回文本。 简短说明
    3. Units are heated and/or cooled. See manager for details.;在工具提示函数的p 标签内返回文本。 详细说明

    我编写了 2 个函数,GetMatchesGetAmenitiesDescriptions,它们使用正则表达式来提取每个图标的所有重复项,并返回一个以 css 选择器为键的字典,以及简短的 description : long description作为值。

    当我收集每一行中的所有图标时:

    Set icons = html2.querySelectorAll(".amenity_icon")
            
    

    我使用字典根据图标的类名返回工具提示描述

    For icon = 0 To icons.Length - 1 'use class name of amenity to look up description
        amenitiesInfo(icon) = amenitiesDescriptions("." & Replace$(icons.item(icon).className, Chr$(32), "."))
    Next        
    

    然后我将描述与vbNewLine 结合起来,以确保输出位于输出单元格内的不同行上。

    您可以探索正则表达式here

    正则表达式使用|(或)语法,所以我在一个列表中返回所有匹配的模式。

    arr = GetMatches(re, s, "(\.amenity_icon\..*)""|<h4>(.*)<\/h4>|<p>(.*)<\/p>")
    

    因为我需要不同的子匹配(0,1 或 2 又名 css 类选择器,short desc,long desc),所以我使用带有计数器变量 iSelect Case i mod 3 来提取适当的子匹配。

    这些匹配在 php 文件中的映射示例:


    特价:

    我们回到类选择器。 Offer2 未填充,因此您可以删除。

    results(r, 6) = html2.querySelector(".offer1").innerText
    results(r, 7) = html2.querySelector(".offer2").innerText
    

    返回(分别):

    呼叫特价,空字符串


    结束语:

    因此,以上内容将引导您完成一排。它只是冲洗并在所有行的循环中重复。为提高效率,将数据添加到数组results;然后一次性写入Sheet1。我可以看到一些小的改进,但这很快。


    VBA:

    Option Explicit
    Public Sub GetInfo()
        Dim ws As Worksheet, html As HTMLDocument, s As String, amenitiesDescriptions As Object
        Const URL As String = "https://www.safeandsecureselfstorage.com/self-storage-lake-villa-il-86955"
    
        Set ws = ThisWorkbook.Worksheets("Sheet1")
        Set html = New HTMLDocument
        Set amenitiesDescriptions = GetAmenitiesDescriptions
        
        With CreateObject("MSXML2.XMLHTTP")
            .Open "GET", URL, False
            .setRequestHeader "User-Agent", "Mozilla/5.0"
            .send
            s = .responseText
    
            html.body.innerHTML = s
    
            Dim headers(), results(), listings As Object, amenities As String
    
            headers = Array("Size", "Description", "RateType", "Price", "Amenities", "Offer1", "Offer2")
            Set listings = html.querySelectorAll(".unitinfo")
    
            Dim rowCount As Long, numColumns As Long, r As Long, c As Long
            Dim icons As Object, icon As Long, amenitiesInfo(), i As Long, item As Long
    
            rowCount = listings.Length
            numColumns = UBound(headers) + 1
    
            ReDim results(1 To rowCount, 1 To numColumns)
            Dim html2 As HTMLDocument
            Set html2 = New HTMLDocument
            For item = 0 To listings.Length - 1
                r = r + 1
                html2.body.innerHTML = listings.item(item).innerHTML
                results(r, 1) = Left$(html2.body.innerHTML, InStr(html2.body.innerHTML, "<") - 1)
                results(r, 2) = Split(Split(html2.querySelector("SCRIPT").innerHTML, """description\"">")(1), "</div>")(0)
                results(r, 3) = Replace$(html2.querySelector(".price_text").innerText, ":", vbNullString)
                results(r, 4) = Trim$(html2.querySelector(".rate_text").innerText)
                
                Set icons = html2.querySelectorAll(".amenity_icon")
                ReDim amenitiesInfo(0 To icons.Length - 1)
                
                For icon = 0 To icons.Length - 1 'use class name of amenity to look up description
                    amenitiesInfo(icon) = amenitiesDescriptions("." & Replace$(icons.item(icon).className, Chr$(32), "."))
                Next
    
                amenities = Join$(amenitiesInfo, vbNewLine) 'place each amenity description on a new line within cell when written out
    
                results(r, 5) = amenities
                results(r, 6) = html2.querySelector(".offer1").innerText
                results(r, 7) = html2.querySelector(".offer2").innerText
            Next
    
            ws.Cells(1, 1).Resize(1, UBound(headers) + 1) = headers
            ws.Cells(2, 1).Resize(UBound(results, 1), UBound(results, 2)) = results
        End With
    End Sub
    
    Public Function GetAmenitiesDescriptions() As Object 'retrieve amenities descriptions from php file on server
        Dim s As String, dict As Object, re As Object, i As Long, arr() 'keys based on classname, short desc, full desc
        ' view regex here: https://regex101.com/r/bII5AL/1
        Set dict = CreateObject("Scripting.Dictionary")
        Set re = CreateObject("vbscript.regexp")
        
        With CreateObject("MSXML2.XMLHTTP")
            .Open "GET", "https://www.safeandsecureselfstorage.com/core/resources/js/src/common.tooltip.php", False
            .setRequestHeader "User-Agent", "Mozilla/5.0"
            .send
            s = .responseText
            
            arr = GetMatches(re, s, "(\.amenity_icon\..*)""|<h4>(.*)<\/h4>|<p>(.*)<\/p>")
            For i = LBound(arr) To UBound(arr) Step 3  'build up lookup dictionary for amenities descriptions
                dict(arr(i)) = arr(i + 1) & ": " & arr(i + 2)
            Next
        End With
        Set GetAmenitiesDescriptions = dict
    End Function
    
    Public Function GetMatches(ByVal re As Object, inputString As String, ByVal sPattern As String) As Variant
        Dim matches As Object, iMatch As Object, s As String, arrMatches(), i As Long
    
        With re
            .Global = True
            .MultiLine = True
            .IgnoreCase = False
            .Pattern = sPattern
            If .test(inputString) Then
                Set matches = .Execute(inputString)
                ReDim arrMatches(0 To matches.Count - 1)
                For Each iMatch In matches
                    Select Case i Mod 3
                    Case 0
                        arrMatches(i) = iMatch.SubMatches.item(0)
                    Case 1
                        arrMatches(i) = iMatch.SubMatches.item(1)
                    Case 2
                        arrMatches(i) = iMatch.SubMatches.item(2)
                    End Select
                    i = i + 1
                Next iMatch
            Else
                ReDim arrMatches(0)
                arrMatches(0) = vbNullString
            End If
        End With
        GetMatches = arrMatches
    End Function
    

    输出:


    参考资料(VBE > 工具 > 参考资料):

    1. Microsoft HTML 对象库

    【讨论】:

    • 抱歉延迟回复..实际上我试图理解代码,但真的很难理解我。但是输出是正确的。谢谢你..
    • 请随时提问。这次我尝试在答案中添加很多信息,因为我很欣赏它更复杂。我很高兴经历其中的任何一个。
    • 您提供的信息足以理解,但由于我缺乏知识,我无法理解这一点..非常感谢。
    • 好吧,请您随意询问有关位的问题,我可能会以您可以理解的方式进行讨论。
    【解决方案2】:

    这是一种方法:

    Sub test()
    Dim req As New WinHttpRequest
    Dim doc As New HTMLDocument
    Dim targetTable As HTMLTable
    Dim tableRow As HTMLTableRow
    Dim tableCell As HTMLTableCell
    Dim element As HTMLDivElement
    Dim sht As Worksheet
    Dim amenitiesString As String
    Dim i As Long
    Dim j As Long
    Set sht = ThisWorkbook.Worksheets("Sheet1")
    With req
        .Open "GET", "https://www.safeandsecureselfstorage.com/self-storage-lake-villa-il-86955", False
        .send
        doc.body.innerHTML = .responseText
    End With
    
    Set targetTable = doc.getElementById("units_small_units") 'You can use units_medium_units or units_large_units to get the info from the other tabs
    i = 0
    For Each tableRow In targetTable.Rows
        i = i + 1
        j = 0
        For Each tableCell In tableRow.Cells
        amenitiesString = ""
        j = j + 1
            If tableCell.className = "amenities" And tableCell.innerText <> "Amenities" Then
                For Each element In tableCell.getElementsByTagName("div")
                    amenitiesString = amenitiesString & element.Title & ","
                Next element
                sht.Cells(i, j).Value = amenitiesString
            ElseIf tableCell.className <> "features" Then
                sht.Cells(i, j).Value = tableCell.innerText
            End If
        Next tableCell
    Next tableRow
    
    End Sub
    

    我使用 HTTP 请求而不是 Internet Explorer 来获取 HTML。除此之外,我认为您可以了解如何访问所需的元素。

    这是结果的屏幕截图。

    演示文稿有点原始,但你明白了:-P

    基本上是这样的:

    listing.getElementsByClassName("amenities")(0).innerText
    

    将返回一个空白,因为这些元素中没有内部文本。该信息由脚本生成,但也可以在div 元素的title 中找到。

    参考文献:

    Microsoft HTML Object LibraryWinHTTP Services Version 5.1

    【讨论】:

    • 我试过这个但输出不完整,我只获取单个选项卡的数据。但是,有些单位列在不同的选项卡上,无法通过此代码复制。谢谢您的回复。
    • 嗯,这应该说明您应该如何进行的总体思路。这是您可以继续前进的起点。给你完整的代码只会在短期内帮助你。用units_medium_unitsunits_large_units 替换这个units_small_units 应该也会让你从其他选项卡中获得信息......
    【解决方案3】:

    你可以试试 Jquery get 方法如下:

    $.get('url', 函数(数据) {

    // Loop through elements
    $(data).find("ul").find("li").each( function(){
    
        var text = $(this).text();
    
    } )
    

    });

    【讨论】:

    • 对不起,我不知道Jquery,你能详细说明一下吗?
    • 我正在拼命寻找答案,有人可以回复一下吗
    猜你喜欢
    • 1970-01-01
    • 2018-12-06
    • 1970-01-01
    • 2020-01-23
    • 1970-01-01
    • 2018-07-13
    • 1970-01-01
    • 1970-01-01
    • 2012-03-16
    相关资源
    最近更新 更多