无法解析复杂网页中的某些字段

Question

制作我的刮刀感觉我没有做错任何事情，但是当我运行它时，它既没有获取任何数据也没有抛出任何错误。这三个字段（电话、网页和电子邮件）我追求。似乎电子邮件和网页链接是隐藏的，所以我的这两个字段的 xpath 有点混乱。任何想法都将受到高度赞赏。到目前为止，我已经尝试过：

import requests
from lxml import html
def startpoint():
    url="https://www.truelocal.com.au/business/strata-report-sydney/sydney"
    page=requests.get(url, headers={"user-agent" : "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"})
    tree=html.fromstring(page.text)
    titles=tree.xpath('//div[@class="column"]')
    for title in titles:
        Phone=title.xpath(".//span[contains(concat(' ', @class, ' '), ' ng-binding ')]/text()")[0]
        Web=title.xpath('.//span[@class="text-frame"]')[0]
        Email=title.xpath('.//a[@class="iconed-text"]/@href')[0]
        print(Phone,Web,Email)

startpoint()

以下项目的元素：

<div class="column" ng-class="vm.getTabletClass()">
                    <bdp-details-contact-website listing="vm.listing" contacts="vm.listing.contacts" class="ng-isolate-scope"><!-- ngIf: vm.getHavePrimaryWebsite()==true --><a class="iconed-text link-color-white-bck ng-scope" ng-if="vm.getHavePrimaryWebsite()==true" rel="nofollow" ng-click="vm.bdpEventTracking();">
  <span class="icon-holder">
    <i class="icon icon-computer-notebook-1"></i>
  </span>
  <span class="text-frame" ng-class="(vm.getHaveSecondaryWebsites()==true) ? 'with-aditional-item':''">
    <span ng-click="vm.openLink(vm.getReadableUrl(vm.getPrimaryWebsite()),'_blank')" role="button" tabindex="0">Visit website</span>
  </span>
</a><!-- end ngIf: vm.getHavePrimaryWebsite()==true --> <!-- iconed-text-->

<!-- ngRepeat: contact in vm.getSecondaryWebsites() --> <!-- iconed-text-->
</bdp-details-contact-website>
                    <a href="" class="iconed-text" ng-show="vm.isContactEmail" aria-hidden="false">
                      <span class="icon-holder">
                        <i class="icon icon-email"></i>
                      </span>
                      <span class="text-frame emailBusiness">
                        <span ng-click="vm.emailABusiness($event);" role="button" tabindex="0">Email this business</span>
                      </span>
                    </a> <!-- iconed-text-->
                    <div>
                        <bdp-details-contact-phone contacts="vm.listing.contacts" priority-number="vm.listing.preferences" class="ng-isolate-scope"><!-- ngRepeat: number in vm.getNumbers() --><!-- ngIf: vm.haveNumbers --><span class="iconed-text ng-scope" ng-if="vm.haveNumbers" ng-repeat="number in vm.getNumbers()">
  <span class="icon-holder">
    <!-- ngIf: $index==0 --><i class="icon-phone-call-2 ng-scope" ng-if="$index==0"></i><!-- end ngIf: $index==0 -->
  </span>
  <span class="text-frame">
    <!-- ngIf: vm.isMobile -->
    <!-- ngIf: !vm.isMobile --><span ng-if="!vm.isMobile" class="ng-binding ng-scope">0421 298 888</span><!-- end ngIf: !vm.isMobile -->
  </span>
</span><!-- end ngIf: vm.haveNumbers --><!-- end ngRepeat: number in vm.getNumbers() --><!-- ngIf: vm.haveNumbers --><span class="iconed-text ng-scope" ng-if="vm.haveNumbers" ng-repeat="number in vm.getNumbers()">
  <span class="icon-holder">
    <!-- ngIf: $index==0 -->
  </span>
  <span class="text-frame">
    <!-- ngIf: vm.isMobile -->
    <!-- ngIf: !vm.isMobile --><span ng-if="!vm.isMobile" class="ng-binding ng-scope">0478 151 999</span><!-- end ngIf: !vm.isMobile -->
  </span>
</span><!-- end ngIf: vm.haveNumbers --><!-- end ngRepeat: number in vm.getNumbers() --> <!-- iconed-text-->
</bdp-details-contact-phone>
                    </div>
                    <div>
                        <bdp-details-contact-fax contacts="vm.listing.contacts" class="ng-isolate-scope"><!-- ngIf: vm.getHaveFax()==true --> <!-- iconed-text-->
</bdp-details-contact-fax>
                    </div>
                    <div>
                        <bdp-details-abn-acn listing="vm.listing" class="ng-isolate-scope"><!-- ngIf: vm.haveAbn() -->
<!-- ngIf: vm.haveAcn() --></bdp-details-abn-acn>
                    </div>
                </div>

Answer 1

分析：

如果你看页面源码，正文很简单，没有<div class="column">。

问题是网站会调用一些javascript然后重写html元素，你要找的内容是js写的。这就是为什么当你使用request时，页面内容一开始不会显示所有来源，你找不到元素<div class="column" ng-class="vm.getTabletClass()">，返回将是NONE。

解决方案：

1，如果你Inspect使用chrome的网站，你可以找到div和class="column"，就像你问题中的元素一样，那么也许从这里你可以只刮掉这部分。但是，您的for loop 将获得所有div 和class="column"，如果没有找到特定的子元素，则提升list index out of range，您可能只需要第一个div 和class="column" 来获取电话、网络、电子邮件：titles[0].

2，也许你可以尝试一些像selenium这样的webdriver组件来模拟网页浏览，渲染javascript。

顺便说一句：对于 Web 和电子邮件，span 带有ng-click，您的代码不适用于这部分