刮掉的桌子不会完整打印

Question

我正在从网站上抓取一个表格，我在获取数据时没有遇到任何问题，但我在打印最终输出时遇到了问题。在我提供的代码中，它打印所有内容，如果它在我的“for”语句中打印没有问题（请参阅注释掉的打印命令）。但是，如果我稍后在代码中打印它，在“for”语句之外，我只会得到第一行。我想把这段代码放在一个更大的项目中，这个输出（以及其他）在一封电子邮件中。如何让整个输出出现？

我尝试将每个表格行附加到列表中，但我认为我做错了，但它只是一遍又一遍地打印同一行或第一行中的单个字母。

import requests
from bs4 import BeautifulSoup


headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}

#print('Scraping Iowa Dept of Banking...')


url = 'https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx'
r = requests.get(url, headers = headers)


soup = BeautifulSoup(r.text, 'html.parser')


mylist = []
for tr in soup.find_all('tr')[2:]:
    tds = tr.find_all('td')
    if len(tds[5].text) == 1:
        edate = "NA"
    else:
        edate = ""
    if len(tds[6].text) == 1:
        loc = "NA"
    else:
        loc = ""
    output5 = ("Bank: %s, City: %s, Type: %s, Effective Date: %s, Location: %s, Comment: %s \r\n" % (tds[0].text, tds[1].text, tds[2].text.replace(" ", ""), tds[5].text+edate, tds[6].text.replace(" ", "")+loc, tds[7].text))


    global outputs5
    outputs5 = output5
    #print(outputs5) #The whole table prints if printed here

if outputs5 is None:
    outputs5 = "No information available"
    print(outputs5)

print(outputs5) #only prints the first line

Answer 1

我很想使用 python 库 pandas 并提取表格并导入 csv。

import pandas as pd
tables=pd.read_html("https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx")
tables[1].to_csv('output.csv')

Csv 看起来像这样。

pandas 的安装非常简单，直接在命令提示符下输入即可。

pip 安装熊猫

Answer 2

尝试这样，您需要将输出附加到列表中，然后在打印之前将列表连接在一起。

您在循环中打印的原因是因为它实际上打印了 5 次，而不仅仅是一次。

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}

#print('Scraping Iowa Dept of Banking...')

url = 'https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx'
r = requests.get(url, headers = headers)

soup = BeautifulSoup(r.text, 'html.parser')

mylist = []
for tr in soup.find_all('tr')[2:]:
    tds = tr.find_all('td')
    if len(tds[5].text) == 1:
        edate = "NA"
    else:
        edate = ""
    if len(tds[6].text) == 1:
        loc = "NA"
    else:
        loc = ""
    output5 = ("Bank: %s, City: %s, Type: %s, Effective Date: %s, Location: %s, Comment: %s \r\n" % (tds[0].text, tds[1].text, tds[2].text.replace(" ", ""), tds[5].text+edate, tds[6].text.replace(" ", "")+loc, tds[7].text))

    mylist.append(outputs5)

if mylist == []:
    print("No information available")

print('\n'.join(mylist))

Answer 3

您是否尝试过使用 pandas 的 .read_html()？

import pandas as pd

url ='https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx'

tables = pd.read_html(url)

输出：

print (tables[-1].to_string())
                                     Bank             City                                        Type    Accepted    Approved Effective #      Location                                            Comment
0                              State Bank      New Hampton                         Merge With and Into  05/21/2019         NaN         NaN           NaN  Application to merge State Bank, New Hampton, ...
1        Farmers and Traders Savings Bank            Douds                         Merge With and Into  05/20/2019         NaN         NaN           NaN  Application to merge Farmers and Traders Savin...
2                         City State Bank          Norwalk                     Establish a Bank Office  05/15/2019  05/29/2019         NaN      Mesa, AZ  Application by City State Bank, Norwalk, to es...
3                             Availa Bank          Carroll                     Purchase and Assumption  04/16/2019  04/30/2019         NaN           NaN  Application by Availa Bank, Carroll, to purcha...
4                               West Bank  West Des Moines                     Establish a Bank Office  04/16/2019  05/02/2019  05/10/2019   Mankato, MN  Application by West Bank, West Des Moines, to ...
5                               West Bank  West Des Moines                     Establish a Bank Office  04/10/2019  05/02/2019  05/07/2019  Owatonna, MN  Application by West Bank, West Des Moines, to ...
6                               West Bank  West Des Moines                     Establish a Bank Office  04/09/2019  05/02/2019  05/07/2019           NaN  Application by West Bank, West Des Moines, to ...
7                         Iowa State Bank           Algona                     Establish a Bank Office  03/15/2019         NaN         NaN   Phoenix, AZ  Application by Iowa State Bank, Algona, to est...
8                    Peoples Savings Bank             Elma                         Merge With and Into  03/13/2019  04/24/2019         NaN           NaN  Application to merge Peoples State Bank, Elma,...
9                 Two Rivers Bank & Trust       Burlington                     Purchase and Assumption  01/25/2019  01/31/2019  05/31/2019           NaN  Application by Two Rivers Bank & Trust, Burlin...
10                    Westside State Bank         Westside                     Establish a Bank Office  01/25/2019  02/06/2019         NaN  Bellevue, NE  Application by Westside State Bank, Westside, ...
11                         Northwest Bank          Spencer                      Relocate a Bank Office  11/29/2018  12/12/2018         NaN        Ankeny  Application by Northwest Bank, Spencer, to rel...
12                        City State Bank          Norwalk                     Establish a Bank Office  11/21/2018  12/12/2018         NaN       Norwalk  Application by City State Bank, Norwalk, to es...
13  First Security Bank and Trust Company     Charles City                      Relocate a Bank Office  06/21/2018  06/29/2018         NaN         Manly  Application by First Security Bank and Trust C...
14                   Lincoln Savings Bank      Cedar Falls                     Establish a Bank Office  06/04/2018  06/25/2018         NaN    Des Moines  Application by Lincoln Savings Bank, Cedar Fal...
15                    Raccoon Valley Bank            Perry                     Establish a Bank Office  02/12/2018  03/02/2018         NaN        Grimes  Application by Raccoon Valley Bank, Perry, to ...
16                 Community Savings Bank         Edgewood                      Relocate a Bank Office  01/25/2018  01/25/2018         NaN    Manchester  Application by Community Savings Bank, Edgewoo...
17                     Luana Savings Bank            Luana                     Establish a Bank Office  06/05/2017  08/16/2017         NaN       Norwalk  Application by Luana Savings Bank, Luana, to e...
18         Fort Madison Financial Company     Fort Madison                         Change of Ownership         NaN  10/19/2017         NaN           NaN  Application for Linda Sue Baier, Fort Madison,...
19                        Lincoln Bancorp         Reinbeck                         Change of Ownership         NaN  12/10/2018         NaN           NaN  Application for Lincoln Bancorp Employee Stock...
20           Emmetsburg Bank Shares, Inc.       Emmetsburg                         Change of Ownership         NaN  01/17/2019         NaN           NaN  Application for Charles and Maryanna Sarazine,...
21      Albrecht Financial Services, Inc.          Norwalk                           Change of Control         NaN  03/27/2019  05/10/2019           NaN  Application for Dean L. Albrecht 2014 Family T...
22                  Solon Financial, Inc.            Solon                         Change of Ownership         NaN  03/05/2019         NaN           NaN  Application for Cordelia A. Cosgrove, Bruce A....
23                How-Win Development Co.           Cresco                         Change of Ownership         NaN  03/28/2019         NaN           NaN  Application for John Scott Thomson, as trustee...
24                      Lee Capital Corp.     Fort Madison                         Change of Ownership         NaN  04/15/2019         NaN           NaN  Application by Jean M. Humphrey, Kathleen A. M...
25                    FNB BanShares, Inc.       West Union                         Change of Ownership         NaN  05/02/2019         NaN           NaN  Application for James L. Moss, individually an...
26          Old O'Brien Banc Shares, Inc.       Sutherland                         Change of Ownership         NaN  03/06/2019         NaN           NaN  Application for James J. Johnson and Colleen D...
27            Pella Financial Group, Inc.            Pella                           Change of Control         NaN  03/15/2019         NaN           NaN  Application for Pella Financial Group, Inc., P...
28                                   BANK          Wapello  Amend or Restate Articles of Incorporation         NaN  05/07/2019  05/07/2019           NaN          Restatement of Articles of Incorporation.
29                  Security Agency, Inc.          Decorah                         Change of Ownership         NaN  11/28/2018         NaN           NaN  Application for the 2018 Grantor Trust FBO Rac...
30                          Arendt's Inc.        Montezuma                         Change of Ownership         NaN  05/29/2019         NaN           NaN  Application for C. W. Bolen, Montezuma, indivi...