【问题标题】:python - pandas append dataframe failingpython - 熊猫追加数据框失败
【发布时间】:2018-04-12 17:09:25
【问题描述】:

我正在尝试连接每个股票代码的三个单独的数据帧(bs_df、income_df、cash_df),然后将它们附加到数据帧df中。它在第一次通过 for 循环时成功附加了数据帧,但是当它尝试将新的数据帧附加到原始数据帧时,它失败了,并抛出了 断言错误。我认为这是因为数据框具有不同的列(reference),但我无法找到绕过此问题的方法,并且仍然创建一个包含所有列名的数据框并在数据框没有值的情况下填写 NaN 值另一个数据框具有的特定列。

这是我正在运行的代码:

from yahoofinancials import YahooFinancials
import pandas as pd

tickers = ["PIH","AAPL","AMZN"]
df = pd.DataFrame()
for ticker in tickers:
    print (ticker)
    # Updates every loop
    ticker_yahoo_financials = YahooFinancials(ticker)

    # Get financial info
    bs_hist = ticker_yahoo_financials.get_financial_stmts('annual', 'balance')
    income_hist = ticker_yahoo_financials.get_financial_stmts('annual', 'income')
    cash_hist = ticker_yahoo_financials.get_financial_stmts('annual', 'cash')

    # Create dataframes and comebine them
    bs_df = pd.DataFrame(list(bs_hist['balanceSheetHistory'][ticker][0].values()))
    income_df = pd.DataFrame(list(income_hist['incomeStatementHistory'][ticker][0].values()))
    cash_df = pd.DataFrame(list(cash_hist['cashflowStatementHistory'][ticker][0].values()))
    comb = pd.concat([bs_df, income_df, cash_df], axis=1)
    df = df.append(comb)

这是错误信息:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-23-f476ca9fbb70> in <module>()
     19     cash_df = pd.DataFrame(list(cash_hist['cashflowStatementHistory'][ticker][0].values()))
     20     comb = pd.concat([bs_df, income_df, cash_df], axis=1)
---> 21     df = df.append(comb)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in append(self, other, ignore_index, verify_integrity)
   5192             to_concat = [self, other]
   5193         return concat(to_concat, ignore_index=ignore_index,
-> 5194                       verify_integrity=verify_integrity)
   5195 
   5196     def join(self, other, on=None, how='left', lsuffix='', rsuffix='',

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, copy)
    211                        verify_integrity=verify_integrity,
    212                        copy=copy)
--> 213     return op.get_result()
    214 
    215 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\reshape\concat.py in get_result(self)
    406             new_data = concatenate_block_managers(
    407                 mgrs_indexers, self.new_axes, concat_axis=self.axis,
--> 408                 copy=self.copy)
    409             if not self.copy:
    410                 new_data._consolidate_inplace()

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals.py in concatenate_block_managers(mgrs_indexers, axes, concat_axis, copy)
   5205         blocks.append(b)
   5206 
-> 5207     return BlockManager(blocks, axes)
   5208 
   5209 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals.py in __init__(self, blocks, axes, do_integrity_check, fastpath)
   3031 
   3032         if do_integrity_check:
-> 3033             self._verify_integrity()
   3034 
   3035         self._consolidate_check()

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals.py in _verify_integrity(self)
   3247                                  'block items\n# manager items: {0}, # '
   3248                                  'tot_items: {1}'.format(
-> 3249                                      len(self.items), tot_items))
   3250 
   3251     def apply(self, f, axes=None, filter=None, do_integrity_check=False,

AssertionError: Number of manager items must equal union of block items
# manager items: 67, # tot_items: 68

【问题讨论】:

    标签: python pandas append concat


    【解决方案1】:

    问题是在循环中的concatDataframes 重复index 行之后,所以最后concat 引发错误:

    PIH
               2017-12-31  2016-12-31  2015-12-31
    netIncome    294000.0     11000.0  -1673000.0
    netIncome    294000.0     11000.0  -1673000.0
    AAPL
                 2017-09-30    2016-09-24    2015-09-26
    netIncome  4.835100e+10  4.568700e+10  5.339400e+10
    netIncome  4.835100e+10  4.568700e+10  5.339400e+10
    AMZN
                 2017-12-31    2016-12-31   2015-12-31
    netIncome  3.033000e+09  2.371000e+09  596000000.0
    netIncome  3.033000e+09  2.371000e+09  596000000.0
    

    我的解决方案:

    tickers = ["PIH","AAPL","AMZN"]
    dfs = []
    for t in tickers:
        print (t)
        # Updates every loop
        ticker_yahoo_financials = YahooFinancials(t)
    
        # Get financial info
        bs_hist = ticker_yahoo_financials.get_financial_stmts('annual', 'balance')
        bs_hist = bs_hist['balanceSheetHistory']
        income_hist = ticker_yahoo_financials.get_financial_stmts('annual', 'income')
        income_hist = income_hist['incomeStatementHistory']
        cash_hist = ticker_yahoo_financials.get_financial_stmts('annual', 'cash')
        cash_hist = cash_hist['cashflowStatementHistory']
    
        bs_df = pd.concat([pd.DataFrame(x) for x in bs_hist[t]], axis=1)
        income_df = pd.concat([pd.DataFrame(x) for x in income_hist[t]], axis=1)
        cash_df = pd.concat([pd.DataFrame(x) for x in cash_hist[t]], axis=1)
        comb = pd.concat([bs_df, income_df, cash_df])
    
        #check duplicated index  
        print (comb[comb.index.duplicated(keep=False)])
        #remove duplicated rows(because same)
        comb = comb[~comb.index.duplicated()]
        dfs.append(comb)
    
    
    df = pd.concat(dfs, keys=tickers, axis=1)
    

    print (df.head())
    
                                PIH                                  AAPL  \
                         2017-12-31  2016-12-31  2015-12-31    2017-09-30   
    accountsPayable       9805000.0   7294000.0   5146000.0  7.479300e+10   
    capitalExpenditures    -28000.0    -83000.0    -48000.0 -1.245100e+10   
    capitalSurplus       47064000.0  46809000.0  48688000.0           NaN   
    cash                 23575000.0  43045000.0  47957000.0  2.028900e+10   
    changeInCash        -19470000.0  -4912000.0  -5682000.0 -1.950000e+08   
    
                                                             AMZN                \
                           2016-09-24    2015-09-26    2017-12-31    2016-12-31   
    accountsPayable      5.932100e+10  6.067100e+10  5.278600e+10  3.904800e+10   
    capitalExpenditures -1.273400e+10 -1.124700e+10 -1.195500e+10 -7.804000e+09   
    capitalSurplus                NaN           NaN  2.138900e+10  1.718600e+10   
    cash                 2.048400e+10  2.112000e+10  2.052200e+10  1.933400e+10   
    changeInCash        -6.360000e+08  7.276000e+09  1.188000e+09  3.444000e+09   
    
    
                           2015-12-31  
    accountsPayable      3.076900e+10  
    capitalExpenditures -5.387000e+09  
    capitalSurplus       1.339400e+10  
    cash                 1.589000e+10  
    changeInCash         1.333000e+09  
    

    在我的解决方案中,在列中获取 Multiindex,因此可以使用 xs 或 slicers 选择值:

    print (df.xs('PIH', axis=1).head())
                         2017-12-31  2016-12-31  2015-12-31
    accountsPayable       9805000.0   7294000.0   5146000.0
    capitalExpenditures    -28000.0    -83000.0    -48000.0
    capitalSurplus       47064000.0  46809000.0  48688000.0
    cash                 23575000.0  43045000.0  47957000.0
    changeInCash        -19470000.0  -4912000.0  -5682000.0
    
    idx = pd.IndexSlice
    print (df.loc['accountsPayable', idx['PIH', ['2017-12-31', '2016-12-31']]])
    PIH  2017-12-31    9805000.0
         2016-12-31    7294000.0
    Name: accountsPayable, dtype: float64
    
    print (df.loc['accountsPayable', idx['PIH',  '2016-12-31']])
    7294000.0
    

    【讨论】:

    • 非常有帮助!非常感谢您的帮助:)
    • @user9532692 - 是的,这很痛苦,这种错误很难缓存 :) 但我很高兴我发现了问题 :) 祝你好运!
    【解决方案2】:

    试试看:

    result = pd.merge([bs_df, income_df, cash_df])
    

    【讨论】:

    • 虽然此代码可能会回答问题,但提供有关此代码为何和/或如何回答问题的额外上下文可提高其长期价值。
    猜你喜欢
    • 2016-07-21
    • 2020-04-21
    • 2021-12-16
    • 2015-01-02
    • 2018-11-05
    • 2020-08-10
    • 1970-01-01
    • 2018-06-26
    • 1970-01-01
    相关资源
    最近更新 更多