美汤网爬表数据答案

【问题标题】：Beautiful soup web crawling Table data美汤网爬表数据
【发布时间】：2021-12-15 23:17:43
【问题描述】：

我试图从网站https://greatlakeships.org/results?q=&st=kw 及其详细页面（如https://greatlakeships.org/3721293/data?n=1）抓取所有船舶数据，但我在运行代码时总是得到空结果。

        import requests
        from bs4 import BeautifulSoup
        import csv
        
        baseurl ='https://greatlakeships.org/'
        headers= {'User-Agent': 'Mozilla/5.0'}
        
        productlinks = [] #put all item in this array
        for x in range(1,2 ): # set page range
            response = requests.get(f'https://greatlakeships.org/results?bl=and&st=kw&q2=text%3A%28%2A%3A%2A%29&rows=20&sort=titleSort%20asc&p={x}') #url of next page
            soup = BeautifulSoup(response.content, 'html.parser')
            productlist =soup.find_all('ul', class_='single')
            #loop to get all href from ul
            for item in productlist:
                for link in item.find_all('a', href = True):
                    productlinks.append(baseurl + link['href'])
        print(len(productlinks))
        
        
        #product details pages
        #testlink = 'https://greatlakeships.org/3721293/data?n=1'
        tabledata = []
        for data in productlinks:
            response = requests.get(data, headers=headers)
            soup = BeautifulSoup(response.content, 'html.parser')
            trs = soup.find_all('div', class_ = 'DetailsSub')
            for tr in trs:
                        tds = tr.find_all('dd')
                        # column 1 data
                        also_known_as = tds[0].text
                        # column 2 data
                        Year_of_Build = tds[1].text
                        # column 3 data
                        Official_Number = tds[2].text
                        # column 4 data
                        Built_at = tds[3].text
                        # column 5 data
                        Vessel_Type = tds[4].text
                        # column 6 data
                        Additional_vessel_types = tds[5].text
                        # column 7 data
                        Hull_Materials = tds[6].text
                        # column 8 data
                        Builder_Name = tds[7].text
                        # column 9 data
                        Original_Owner_and_Location = tds[8].text
                        # column 10 data
                        Length = tds[9].text
                        # column 11 data
                        Beam = tds[10].text
                        # column 12 data
                        Depth = tds[11].text
                        # column 13 data
                        Tonnage_gross = tds[12].text
                        # column 14 data
                        Tonnage_net = tds[13].text
                        # column 15 data
                        Contact = tds[14].text
                        # save data
                        tr_data = {'Also known as': also_known_as,
                                   'Year of Build': Year_of_Build,
                                   'Official Number': Official_Number,
                                   'Built at': Built_at,
                                   'Vessel Type': Vessel_Type,
                                   'Additional vessel types': Additional_vessel_types,
                                   'Hull Materials': Hull_Materials,
                                   'Builder Name': Builder_Name,
                                   'Original Owner and Location': Original_Owner_and_Location,
                                   'Length': Length,
                                   'Beam': Beam,
                                   'Depth': Depth,
                                   'Tonnage (gross)': Tonnage_gross,
                                   'Tonnage (net)': Tonnage_net,
                                   'Contact': Contact
                                   }
                        tabledata.append(tr_data)
        
        print(tabledata)

我从页面中获取链接，但可以从这些链接中获取任何数据，例如“print(len(productlinks))”。任何人都可以帮助我，在此先感谢！以及如何将这些数据保存在 csv 表中？？

【问题讨论】：

你试过调试你的代码吗？您无需将link['href'] 附加到baseurl。应该是productlinks.append(link['href'])
好的，谢谢。

标签： python web-scraping beautifulsoup web-crawler

【解决方案1】：

如上所述，问题是您尝试请求的链接不会返回任何内容。如果您将打印语句用于调试（可能是print(data)，您会看到它正在尝试从https://greatlakeships.org/https://greatlakeships.org/2899767/data?n=5 请求

另外，存在重复的链接，因此您可能需要删除这些链接。在解析数据时，您还可以更加健壮和高效地进行编码。试试这个：

import requests
from bs4 import BeautifulSoup
import pandas as pd

baseurl ='https://greatlakeships.org/'
headers= {'User-Agent': 'Mozilla/5.0'}

productlinks = [] #put all item in this array
for x in range(1,2 ): # set page range
    response = requests.get(f'https://greatlakeships.org/results?bl=and&st=kw&q2=text%3A%28%2A%3A%2A%29&rows=20&sort=titleSort%20asc&p={x}') #url of next page
    soup = BeautifulSoup(response.content, 'html.parser')
    productlist =soup.find_all('ul', class_='single')
    #loop to get all href from ul
    for item in productlist:
        for link in item.find_all('a', href = True):
            productlinks.append(link['href'])
productlinks = list(set(productlinks))
print(len(productlinks))


#product details pages
#testlink = 'https://greatlakeships.org/3721293/data?n=1'
tabledata = []
for link in productlinks:
    print(link)
    response = requests.get(link, headers=headers)
    soup = BeautifulSoup(response.content, 'html.parser')
    
    try:
        shipName = soup.find('div', {'class':'RecordTitle'}).text.strip()
    except:
        shipName = 'N/A'
    
    fieldsets = soup.find_all('fieldset')
    row = {'Ship Name':shipName}
    for fieldset in fieldsets:
        dts = fieldset.find_all('dt')
        for dt in dts:
            row.update({dt.text.strip(): dt.find_next('dd').text.strip()})
            
    tabledata.append(row)
    

df = pd.DataFrame(tabledata)

输出：

print(df.to_string())
            Ship Name                                                                   Also known as: Year of Build: Official Number:                              Built at: Vessel Type: Hull Materials: Number of Decks: Hull Number:                 Builder Name:               Original Owner and Location: Length:  Beam: Depth: Tonnage (gross): Tonnage (net): From the Collection of:                                                                                                                                                                                                                              Contact                                                             Final Location:        Date:                                    How: Final Cargo:                              Notes:        Capacity:     Note on Dimensions: Final Depth:               Original Owner: Additional vessel types:
0   132 (1893, Barge)                                                                       PORTSMOUTH           1893            53279                           Superior, WI        Barge           Steel                1          132  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    292'    36'    22'            1,310          1,265      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                         NaN          NaN                                     NaN          NaN                                 NaN              NaN                     NaN          NaN                           NaN                      NaN
1   105 (1890, Barge)                                                                         BARONESS           1890            53258                             Duluth, MN        Barge           Steel                1          105  American Steel Barge Company  American Steel Barge Company, Buffalo, NY  276.5'  36.1'  18.9'         1,295.44       1,230.69      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188  10-15 miles Southwest by West of Fire Island, NY Lightship\nAtlantic Ocean  19 Nov 1910  collision with french barque ELISABETH         coal              being towed by BAYPORT              NaN                     NaN          NaN                           NaN                      NaN
2   102 (1889, Barge)                                                       WHITWORTH, SIR JOSEPH;BATH           1889            53255                             Duluth, MN        Barge           Steel                1          102      American Steel Barge Co.       American Steel Barge Co, Buffalo, NY    253'  36.1'  18.8'         1,192.20       1,132.56      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                        off Cape Charles, VA\nAtlantic Ocean  15 Dec 1905                               foundered          NaN                    all 5 lives lost  100,000 bushels                     NaN          NaN                           NaN                      NaN
3   116 (1891, Barge)                                                           BRITANNIA;PURE TIOLENE           1891            53269                           Superior, WI        Barge           Steel                1          116  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    262'    36'    22'         1,169.11       1,110.66      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                  Huston, TX         1946                                scrapped          NaN                                 NaN              NaN                     NaN          NaN                           NaN                      NaN
4   131 (1893, Barge)                         SALEM;FREEPORT SULPHUR NO. 4;PURE OIL NO. 10;PURE NULUBE           1893            53278                           Superior, WI        Barge           Steel                1          131  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    292'    36'    22'            1,310          1,265      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                  Huston, TX         1946                                scrapped          NaN                                 NaN              NaN                     NaN          NaN                           NaN                      NaN
5   107 (1890, Barge)                                                                           BOMBAY           1890            53260                             Duluth, MN        Barge           Steel                1          107  American Steel Barge Company  American Steel Barge Company, Buffalo, NY  276.5'  36.1'  18.9'         1,295.44       1,230.69      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                        off Nantucket Shoals\nAtlantic Ocean   3 Jan 1913                      foundered in storm          NaN                                 NaN              NaN                     NaN          NaN                           NaN                      NaN
6   104 (1890, Barge)                                                                              NaN           1890            53257                             Duluth, MN        Barge           Steel                1          104  American Steel Barge Company  American Steel Barge Company, Buffalo, NY  276.5'  36.1'  18.9'         1,295.44       1,230.69      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                   Cleveland, OH; west breakwater\nLake Erie  10 Nov 1898            ashore on breakwater in gale          NaN  crew rescued by lifesaving service       3,300 tons                     NaN          NaN                           NaN                      NaN
7   118 (1891, Barge)  BOSTON;FREEPORT SULLPHUR NO. 3;PURE OIL STEAM SHIP CO. BARGE NO. 9;PURE DETONOX           1891            53272                           Superior, WI        Barge           Steel                1          118  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    292'    36'    22'         1,310.82       1,265.92      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                  Huston, TX         1946                                scrapped          NaN                                 NaN       4,000 tons  alternate length: 320?          NaN                           NaN                      NaN
8   129 (1893, Barge)                                                                              NaN           1893            53276                           Superior, WI        Barge           Steel                1          129  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    292'    36'    22'            1,310          1,265      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                     off Vermillion Point, MI\nLake Superior  13 Oct 1902       rammed by towing steamer MAUNALOA     iron ore                                 NaN              NaN                     NaN     125 feet                           NaN                      NaN
9   117 (1891, Barge)                                                                       PROVIDENCE           1891            53271                           Superior, WI        Barge           Steel                1          117  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    292'    36'    22'         1,310.82       1,265.92      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                         NaN          NaN                                     NaN          NaN                                 NaN       3,000 tons                     NaN          NaN                           NaN                      NaN
10  111 (1891, Barge)                                                                             IVIE           1891            53267                           Superior, WI        Barge           Steel                1          NaN  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    265'    36'    NaN            1,227          1,167      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                           Hampton Roads, VA\nAtlantic Ocean  10 May 1916        collision with steamer BERKSHIRE          NaN                                 NaN   85,000 bushels                     NaN          NaN                           NaN                      NaN
11  130 (1893, Barge)                                                                             LYNN           1893            53277                           Superior, WI        Barge           Steel                1          130  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    292'    36'    22'            1,310          1,265      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                         NaN          NaN                                     NaN          NaN                                 NaN              NaN                     NaN          NaN                           NaN                      NaN
12  115 (1891, Barge)                                                                              NaN           1891            53268                           Superior, WI        Barge           Steel                1          115  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    256'  36.1'  18.9'         1,169.11       1,110.66      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                Pic Island near Marathon, ONT\nLake Superior  18 Dec 1899           ashore after breaking towline     iron ore  last shipwreck of the 19th century              NaN                     NaN          NaN                           NaN                      NaN
13  101 (1888, Barge)                                                                              NaN           1888            53249                             Duluth, MN        Barge           Steel                1          101           Alexander McDougall         Alexander McDougall, et al, Duluth    178'  25.1'  12.7'           428.30         412.32      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                30 miles off Seal Island, ME\nAtlantic Ocean   3 Dec 1908                                     NaN          NaN                                 NaN              NaN                     NaN          NaN                           NaN                      NaN
14  109 (1890, Barge)                                                                          BARAVIA           1890            53265                           Superior, WI        Barge           Steel                1          109  American Steel Barge Company                                        NaN    265'    36'    22'            1,227          1,167      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                    off Montauk Point, NY, Long Island Sound\nAtlantic Ocean  23 Jan 1914                               foundered          NaN                                 NaN              NaN                     NaN          NaN  American Steel Barge Company                      NaN
15  127 (1892, Barge)                                                                   JEANIE, DALLAS           1892            53274                           Superior, WI        Barge           Steel                1          127  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    264'    36'    22'            1,128          1,083      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                         NaN          NaN                                     NaN          NaN                                 NaN              NaN                     NaN          NaN                           NaN                      NaN
16  126 (1892, Barge)                                                                            BADEN           1892            53273                           Superior, WI        Barge           Steel                1          126  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    264'    36'    22'            1,128          1,083      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                           Buzzard's Bay, MA\nAtlantic Ocean  31 Dec 1905                                stranded          NaN                      all hands lost              NaN                     NaN          NaN                           NaN                      NaN
17  110 (1891, Barge)                                                              BADGER;PURE LUBWELL           1891            53266                           Superior, WI        Barge           Steel                1          110  American Steel Barge Company  American Steel Barge Company, Buffalo, NY    265'    36'    22'            1,227          1,167      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                             New Orleans, LA  13 Mar 1932                                    fire          NaN              1 life lost, 7 rescued   85,000 bushels                     NaN          NaN                           NaN                      NaN
18  0452 (1957, Scow)                                                                C & O 452; KODIAK           1957           275111  Port Deposit, Maryland, United States         Scow           Steel              NaN          NaN        Wiley Manufacturing Co     Baltimore & Ohio Railroad Co, Maryland     90'    30'     9'              222            222                     NaN  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                                                         NaN          NaN                                     NaN          NaN                                 NaN              NaN                     NaN          NaN                           NaN            Tug (Towboat)
19  103 (1889, Barge)                                                    RUSSELL, JOHN SCOTT;BERKSHIRE           1889            53256                             Duluth, MN        Barge           Steel                1          103      American Steel Barge Co.  American Steel Barge Company, Buffalo, NY    253'  36.1'  18.8'         1,192.20       1,132.56      C. Patrick Labadie  Alpena County George N. Fletcher Public Library\n\t\tEmail:tbrc@alpenalibrary.org\n\t\tWebsite: http://www.alpenalibrary.org\n\t\tAgency street/mail address: 211 N. First Ave. \r\nAlpena, Michigan 49707 \r\nUSA\r\n(989)356-6188                                          Off Sandy Hook, NJ\nAtlantic Ocean  23 May 1909                               Foundered          NaN                                 NaN       3,000 tons                     NaN          NaN                           NaN                      NaN

【讨论】：

这真的很有帮助，而且你让它变得非常简单和简短。结果会得到一些额外的列，但是当我将其导出为 CSV 时，它会很容易被删除。我也会将此方法用于另一个相同的站点，例如“balticshipping.com/vessels”，但是使用下一页按钮，由于某些 JSON 内容，URL 不会更改，这对我来说真的很难同时从所有页面获取数据时间。无论如何，非常感谢！
如何在每一行添加记录标题'class="RecordTitle"'，船名？
@Anwar，已更新以添加船名
@Anwar，要导出到 csv，只需执行 df.to_csv('filename.csv, index=False)
@Anwar，其他站点有一个 api 来获取该数据。只需使用它来获取页面