IndexError：使用 BeautifulSoup 从网页中提取表格答案

【问题标题】：IndexError: Extract table from a webpage using BeautifulSoupIndexError：使用 BeautifulSoup 从网页中提取表格
【发布时间】：2017-11-18 02:50:29
【问题描述】：

我正在尝试从 airport website 中提取包含航班到达信息的表格（包含列 - 航班、承运人、出发地、日期、计划、估计、状态），但出现以下错误：

  IndexError Traceback (most recent call last)
  <ipython-input-39-2f7369a95ba9> in <module>()
        6  for cl in cols:
        7  dv = cl.findAll('div')
  ----> 8  if 'col-xs-12 col-sm-6' in dv[0]['class']:
        9  flight, carrier, origin, date, scheduled, estimated, status = [c.text for c in dv]
       10 print(flight, carrier, origin, date, scheduled, estimated, status)
  IndexError: list index out of range

我已经梳理了 stackoverflow 的解决方案，但找不到解决方案。这是我的代码：

  # import libraries
  import urllib3
  import requests
  from bs4 import BeautifulSoup

  # query the website and return the html to the variable ‘page’
  page = requests.get("https://www.aucklandairport.co.nz/flights").text

  soup = BeautifulSoup(page)
  tbody = soup.findAll('tbody')
  for tb in tbody:
    rows = tb.findAll('tr')
    for tr in rows:
      cols = tr.findAll('td')
        for cl in cols:
          dv = cl.findAll('div')
          if 'col-xs-12 col-sm-6' in dv[0]['class']:
             flight, carrier, origin, date, scheduled, estimated, status = [c.text for c in dv]
             print(flight, carrier, origin, date, scheduled, estimated, status)

感谢您的贡献。

【问题讨论】：

标签： python python-3.x web-scraping beautifulsoup

【解决方案1】：

问题是tr 下的第一个td 没有div，这就是dv 将返回空的原因。将您的代码更改为：

# import libraries
import requests
from bs4 import BeautifulSoup

# query the website and return the html to the variable ‘page’
page = requests.get("https://www.aucklandairport.co.nz/flights").text

soup = BeautifulSoup(page)
tbody = soup.find('tbody')
rows = tbody.findAll('tr',{'class':'flight-toggle'}) #find tr whose class = flight-toggle
for tr in rows:
    cols = tr.findAll('td',class_=lambda x: x != 'logo') # find td whose class!=logo (exclude the first td)
    dv0 = cols[0].find('div').findAll('div') #flight, carrier, origin under second td
    flight, carrier, origin = [c.text.strip() for c in dv0]
    dv1 = cols[1].find('div').findAll('div') #date, schedule under third td
    date, scheduled = [c.text.strip() for c in dv1]
    dv2 = cols[2].find('div').findAll('div') #estimated, statusunder fouth td
    estimated, status = [c.text.strip() for c in dv2[1:]] # exclude the first div 
    print(flight, carrier, origin, date, scheduled, estimated, status)

这将打印出来：

(u'EK406', u'', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'QF8762', u'EK406', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'EK434', u'', u'Dubai / Brisbane', u'18 Nov', u'02:45pm', u'02:49pm', u'Processing')
...

【讨论】：

【解决方案2】：

您可以用不同的方式剥同一个苹果。这是实现相同目标的另一种方法。

import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.aucklandairport.co.nz/flights")
soup = BeautifulSoup(response.text,"lxml")
table = soup.select(".flights-table")[0]
for items in table.select("tr.flight-toggle"):
    data = ' '.join([' '.join(item.text.split()) for item in items.select("td")])
    print(data.strip())

部分结果：

QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
AA7377 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
BA7421 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
CZ7575 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing

【讨论】：