【问题标题】:Python - div scraping where no tag is foundPython - 未找到标签的 div 抓取
【发布时间】:2021-10-29 13:48:25
【问题描述】:

我对 python 非常陌生,而且总体上还是一个开发新手。我的任务是遍历一个 div,我能够返回一些数据。但是,当不存在标签时,我得到空结果。谁能帮忙——我已经研究和尝试了好几天了?我非常感谢您的耐心和解释。我正在尝试提取以下内容:

  • 开始日期 [即,2021 年 7 月 31 日星期六],我根据我的代码从中获得结果
  • 结束日期 [即,2021 年 8 月 20 日星期五] 根据我的代码,我得到 no 结果
  • 说明 [即 20 晚新品!马耳他、亚得里亚海和希腊] 根据我的代码,我得到 no 结果
  • 船舶名称 [即 Viking Sea],我根据我的代码获得结果

<div class="cd-info"> <!-- Start cd-info -->
From <b>Sat 31 Jul 2021</b><br>
(To Fri 20 Aug 2021)<br>
<b>20 Night New! Malta, The Adriatic &amp; Greece</b><br>
Ship 
<a class="red" href="/cruise-ship-viking-sea.html">Viking Sea</a>
<br>
<span class="mobile-no-desktop-yes"><br></span>
More details at<br>
<a target="_blank" onclick="trackOutboundLink('https://www.vikingcruises.com/oceans/cruise-destinations/eastern-mediterranean/malta-adriatic-and-greece/index.html');" href="https://www.vikingcruises.com/oceans/cruise-destinations/eastern-mediterranean/malta-adriatic-and-greece/index.html">
<img class="noborder" src="/logos/viking-cruises.gif" alt="More details for 20 Night New! Malta, The Adriatic &amp; Greece at Viking Cruises">
</a>
</div>

这是我的代码(请不要评判..哈哈)

import requests
from requests import get
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np

url = "https://www.cruisetimetables.com/cruisesearch.html?id=cse&sailmonth=Jul%202021&destination=Any&departureport=&arrivalport=&openjawcruise=&portofcall=&cruiseship=&duration=Any&chartercruise="
headers = {"Accept-Language": "en-US, en;q=0.5"}
results = requests.get(url, headers=headers)

soup = BeautifulSoup(results.text, "html.parser")

#initiate data storage
cruisestartdate = []
cruiseenddate = []
itinerarydescription = []
cruiseshipname = []
cruisecabinprice = []
destinationportname = []
portdatetime = []

cruise_div = soup.find_all('div', class_='cd-info')

#our loop through each container
for container in cruise_div:

         #cruise start date
         startdate = container.b.text
         print(startdate)
         cruisestartdate.append(startdate)

        # cruise end date
         enddate = container.string
         cruiseenddate.append(enddate)
        
            # ship name
         ship = container.a.text
         cruiseshipname.append(ship)

#pandas dataframe        
cruise = pd.DataFrame({
'Sail Date': cruisestartdate,
'End Date': cruiseenddate,
#'Description': description,
'Ship Name': cruiseshipname,
#'imdb': imdb_ratings,
#'metascore': metascores,
#'votes': votes,
#'us_grossMillions': us_gross,
})

print(soup)   
print(cruisestartdate)
print(cruiseenddate)
print(itinerarydescription)
print(cruiseshipname)```

这是我的打印结果

['2021 年 7 月 31 日星期六'] [无] [] ['维京海']

【问题讨论】:

    标签: python html css web-scraping beautifulsoup


    【解决方案1】:

    container.text 是格式良好的行列表。只需拆分并使用碎片:

    cruise_div = soup.find_all('div', class_='cd-info')
    
    #our loop through each container
    for container in cruise_div:
        lines = container.text.splitlines()
    
        cruisestartdate.append(lines[1])
    
        cruiseenddate.append(lines[2])
    
        itinerarydescription.append(lines[3])
        
        # ship name
        ship = container.a.text
        cruiseshipname.append(ship)
    

    【讨论】:

    • 感谢@Tim Roberts!
    【解决方案2】:

    另一种解决方案,使用bs4 API:

    import re
    import requests
    from bs4 import BeautifulSoup
    import pandas as pd
    
    
    # url = "https://www.cruisetimetables.com/cruisesearch.html?id=cse&sailmonth=Jul%202021&destination=Any&departureport=&arrivalport=&openjawcruise=&portofcall=&cruiseship=&duration=Any&chartercruise="
    url = "https://www.cruisetimetables.com/cruisesearch.html?id=cse&sailmonth=Aug%202021&destination=Any&departureport=&arrivalport=&openjawcruise=&portofcall=&cruiseship=&duration=Any&chartercruise="
    headers = {"Accept-Language": "en-US, en;q=0.5"}
    results = requests.get(url, headers=headers)
    soup = BeautifulSoup(results.content, "html.parser")
    
    all_data = []
    for div in soup.select("div.cd-info"):
        from_ = div.b
        to_ = from_.find_next_sibling(text=True).strip()
        # remove (To )
        to_ = re.sub(r"\(To (.*)\)", r"\1", to_)
    
        desc = from_.find_next("b").get_text(strip=True)
        # remove html chars
        desc = BeautifulSoup(desc, "html.parser").get_text(strip=True)
    
        ship_name = div.a.get_text(strip=True)
    
        all_data.append(
            [
                from_.get_text(strip=True),
                to_,
                desc,
                ship_name,
            ]
        )
    
    
    df = pd.DataFrame(all_data, columns=["from", "to", "description", "ship name"])
    print(df)
    df.to_csv("data.csv", index=False)
    

    打印:

                  from               to                                                         description     ship name
    0   Tue 3 Aug 2021  Mon 23 Aug 2021                          20 Night New! Malta, The Adriatic & Greece  Viking Venus
    1  Tue 10 Aug 2021  Fri 20 Aug 2021                               10 Night New! Malta & Adriatic Jewels    Viking Sea
    2  Tue 10 Aug 2021  Mon 30 Aug 2021                          20 Night New! Malta, The Adriatic & Greece    Viking Sea
    3  Fri 13 Aug 2021  Fri 20 Aug 2021                                              7 Night Bermuda Escape  Viking Orion
    4  Fri 13 Aug 2021  Mon 23 Aug 2021                         10 Night New! Malta & Greek Isles Discovery  Viking Venus
    5  Fri 13 Aug 2021   Thu 2 Sep 2021                          20 Night New! Malta, The Adriatic & Greece  Viking Venus
    6  Sat 14 Aug 2021  Sat 21 Aug 2021                                    7 Night Iceland's Natural Beauty    Viking Sky
    7  Mon 16 Aug 2021  Mon 13 Sep 2021                   28 Night Star Collector: From Greek Gods to Gaudí     Wind Surf
    8  Mon 16 Aug 2021   Fri 3 Sep 2021              18 Night Star Collector: Flamenco of the Mediterranean     Wind Surf
    9  Mon 16 Aug 2021  Wed 29 Sep 2021  44 Night Star Collector: Myths & Masterpieces of the Mediterranean     Wind Surf
    

    并保存 data.csv(来自 LibreOffice 的屏幕截图):


    编辑:从所有页面抓取数据:

    import re
    import requests
    from bs4 import BeautifulSoup
    import pandas as pd
    
    
    headers = {"Accept-Language": "en-US, en;q=0.5"}
    # url = "https://www.cruisetimetables.com/cruisesearch.html?id=cse&sailmonth=Jul%202021&destination=Any&departureport=&arrivalport=&openjawcruise=&portofcall=&cruiseship=&duration=Any&chartercruise="
    url = "https://www.cruisetimetables.com/cruisesearch.html?id=cse&sailmonth=Aug%202021&destination=Any&departureport=&arrivalport=&openjawcruise=&portofcall=&cruiseship=&duration=Any&chartercruise="
    
    offset = 1
    all_data = []
    while True:
        u = url + f"&offset={offset}"
        print(f"Getting offset {offset=}")
    
        results = requests.get(u, headers=headers)
        soup = BeautifulSoup(results.content, "html.parser")
    
        for div in soup.select("div.cd-info"):
            from_ = div.b
            to_ = from_.find_next_sibling(text=True).strip()
            # remove (To )
            to_ = re.sub(r"\(To (.*)\)", r"\1", to_)
    
            desc = from_.find_next("b").get_text(strip=True)
            # remove html chars
            desc = BeautifulSoup(desc, "html.parser").get_text(strip=True)
    
            ship_name = div.a.get_text(strip=True)
    
            all_data.append(
                [
                    from_.get_text(strip=True),
                    to_,
                    desc,
                    ship_name,
                ]
            )
    
        # get next offset:
        offset = soup.select_one(".cd-buttonlist > b + a")
        if offset:
            offset = int(offset.get_text(strip=True).split("-")[0])
        else:
            break
    
    
    df = pd.DataFrame(all_data, columns=["from", "to", "description", "ship name"])
    print(df)
    df.to_csv("data.csv", index=False)
    

    打印:

    Getting offset offset=1
    Getting offset offset=11
    Getting offset offset=21
    
    ...
    
                    from               to                                                                      description                 ship name
    0     Tue 3 Aug 2021  Mon 23 Aug 2021                                       20 Night New! Malta, The Adriatic & Greece              Viking Venus
    1    Tue 10 Aug 2021  Fri 20 Aug 2021                                            10 Night New! Malta & Adriatic Jewels                Viking Sea
    2    Tue 10 Aug 2021  Mon 30 Aug 2021                                       20 Night New! Malta, The Adriatic & Greece                Viking Sea
    3    Fri 13 Aug 2021  Fri 20 Aug 2021                                                           7 Night Bermuda Escape              Viking Orion
    
    ...
    
    144  Tue 31 Aug 2021   Tue 7 Sep 2021                                                           7 Nights Mediterranean             MSC Splendida
    145  Tue 31 Aug 2021   Tue 7 Sep 2021                                    7 Nachte Grose Freiheit - Schwedische Kuste 1             Mein Schiff 6
    146  Tue 31 Aug 2021   Tue 7 Sep 2021                                                 7 Night Iceland's Natural Beauty            Viking Jupiter
    

    【讨论】:

    • @Andrej--哇,你是个超级明星。对于您和其他所有人的回复,我真的感到不知所措和感激。非常感谢。
    • 如果我想通过几个月、几年和多个页面返回结果,是否可以不必每次都更改 url?当我经历每个月时,我现在注意到返回该月的前 10 个结果,并且每页总共有 300+ 间隔 10。下一个 url 返回结果 11 到 20,依此类推:cruisetimetables.com/…
    • @viperone 我已经更新了如何从多个页面抓取数据的答案(增加了offset= URL 参数)。
    • @Andrej——绝对棒极了。太感谢了。你是一个救生员,真的教会了我很多。我真的很感激。
    【解决方案3】:

    获取开始日期、描述和船名非常简单。这里唯一的技巧是获取位于特定标签内的结束日期。要得到它,你必须调整一些正则表达式。这是一种类似于@Andrej 的解决方案:

    from bs4 import BeautifulSoup
    import re
    
    html = """
    <div class="cd-info"> <!-- Start cd-info -->
    From <b>Sat 31 Jul 2021</b><br>
    (To Fri 20 Aug 2021)<br>
    <b>20 Night New! Malta, The Adriatic &amp; Greece</b><br>
    Ship 
    <a class="red" href="/cruise-ship-viking-sea.html">Viking Sea</a>
    <br>
    <span class="mobile-no-desktop-yes"><br></span>
    More details at<br>
    <a target="_blank" onclick="trackOutboundLink('https://www.vikingcruises.com/oceans/cruise-destinations/eastern-mediterranean/malta-adriatic-and-greece/index.html');" href="https://www.vikingcruises.com/oceans/cruise-destinations/eastern-mediterranean/malta-adriatic-and-greece/index.html">
    <img class="noborder" src="/logos/viking-cruises.gif" alt="More details for 20 Night New! Malta, The Adriatic &amp; Greece at Viking Cruises">
    </a>
    </div>
    """
    soup = BeautifulSoup(html, 'html.parser')
    bold = soup.find_all('b')
    print(f"Start date: {bold[0].text}")
    print(f"Description: {bold[1].text}")
    
    untagged_line = re.sub("\n", "", soup.find(text=re.compile('To')))
    end_date = re.sub(r"\(To (.*)\)", r"\1", untagged_line)
    print(f"End date: {end_date}")
    
    ship = soup.find('a', class_='red')
    print(f"Ship: {ship.text}")
    
    • 输出:
    Start date: Sat 31 Jul 2021
    Description: 20 Night New! Malta, The Adriatic & Greece
    End date: Fri 20 Aug 2021
    Ship: Viking Sea
    

    【讨论】:

    • @viperone 不客气!如果这就是您要找的,您能接受答案吗?欣赏!
    猜你喜欢
    • 2020-02-24
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多