Pandas read_csv() 和 dtype 的疑惑

Question

我有一组 txt 格式的公告，其中有一些粗大的（大标题、尾部等）数据，我可以使用 pandas “清理”它们。然后我不得不将所有DataFrames附加到一个新的DataFrame中才能有一个新文件，因为我需要处理大约10年的数据，所以代码是：

os.chdir(r'D:\Inves\Catalogs\OSC')
path = os.getcwd()
csv_files = glob.glob(os.path.join(path, "*.txt"))

new_data = []

for f in csv_files:
    df = pd.read_csv(f)
    print('Location File:', f)
    print('File Name:', f.split("\\")[-1])
    df = pd.read_csv(f, header=10, sep='\s+')
    n = 2
    df.drop(df.tail(n).index, inplace = True)
    df = df[df.YYYY != '----'] # deleting the '----' row
    print('File Content:')
    
    print('...Appending...')
    print('...................')
    new_data.append(df)
new_data = pd.concat(new_data, ignore_index=True)
#new_data.dtypes
new_data.to_csv(r'D:\Inves\Catalogs\Full_1988-2008.csv',
                index=False, header=True, sep=',')

CSV 文件“Full_1988-2008.csv”大约 10MB（~173395 行），文件内的数据如下所示：

YYYY,MM,JJ,HH,MI,SS,STIME,LAT,SLAT,LON,SLON,DEPTH,ML,ORID,RMS,Num,Fase
1988,07,05,03,01,44,.92,-16.420,"8,41",-68.810,"7,56",94.00,1.01,34,",4",6,
1988,07,05,03,45,00,1.70,-16.990,"10,57",-68.910,"10,15",65.00,-1.00,35,"1,12",11,
1988,07,05,04,40,00,.00,-999.000,0,-999.000,0,-999.00,-1.00,36,0,5,
1988,07,05,05,13,12,1.50,-16.600,"5,51",-68.550,"3,64",15.00,1.97,37,",92",10,
1988,07,05,06,25,45,1.21,-16.960,"4,27",-68.520,"5,92",2.00,2.03,38,",74",8,
1988,07,05,07,24,42,2.04,-19.410,"74,58",-68.910,"23,03",160.00,2.78,39,"1,18",8,
1988,07,05,09,03,00,.00,-999.000,0,-999.000,0,-999.00,-1.00,41,0,3,

我需要来自 YYYY（年份）、LAT 和 LON（坐标）DEPTH（深度）和 ML（幅度）的数据，所以我需要：

DF = pd.read_csv(kat, sep=',',
                 usecols=(['YYYY', 'LAT', 'LON', 'DEPTH', 'ML']),
                 dtype={'YYYY': int, 'LAT': float, 'LON': float,
                        'DEPTH': float, 'ML': float})

但我得到了错误：

  File "pandas\_libs\parsers.pyx", line 1050, in pandas._libs.parsers.TextReader._convert_tokens

TypeError: Cannot cast array data from dtype('O') to dtype('int32') according to the rule 'safe'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "<ipython-input-13-b2a95a2d83fd>", line 46, in <module>
    'DEPTH': float, 'ML': float})

  File "C:\Users\Director\anaconda3\envs\obspy\lib\site-packages\pandas\io\parsers.py", line 610, in read_csv
    return _read(filepath_or_buffer, kwds)

  File "C:\Users\Director\anaconda3\envs\obspy\lib\site-packages\pandas\io\parsers.py", line 468, in _read
    return parser.read(nrows)

  File "C:\Users\Director\anaconda3\envs\obspy\lib\site-packages\pandas\io\parsers.py", line 1057, in read
    index, columns, col_dict = self._engine.read(nrows)

  File "C:\Users\Director\anaconda3\envs\obspy\lib\site-packages\pandas\io\parsers.py", line 2061, in read
    data = self._reader.read(nrows)

  File "pandas\_libs\parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read

  File "pandas\_libs\parsers.pyx", line 771, in pandas._libs.parsers.TextReader._read_low_memory

  File "pandas\_libs\parsers.pyx", line 850, in pandas._libs.parsers.TextReader._read_rows

  File "pandas\_libs\parsers.pyx", line 982, in pandas._libs.parsers.TextReader._convert_column_data

  File "pandas\_libs\parsers.pyx", line 1056, in pandas._libs.parsers.TextReader._convert_tokens

ValueError: invalid literal for int() with base 10: 'YYYY'

据我了解，标头 YYYY、LAT、LON、DEPTH、ML 成为数据的一部分，不能格式化为 int 或 float。但是，如果我跳过标题，我将无法获得所需的数据，因为标题变为 1998,-16.65,-66.65,12,3.2。

有没有人有一些线索可以改进处理我的数据的方式？我附上 2 个完整的文件，以防你想重现我的错误。

https://drive.google.com/drive/folders/18xrDC7vqEm_pY3D2sxwou3dlBdkZ6nHF?usp=sharing

Answer 1

您的代码适用于您的两个文件1988.txt 和1989.txt。 为了调试，我建议从read_csv 中删除演员表：

DF = pd.read_csv(kat, sep=',', usecols=(['YYYY', 'LAT', 'LON', 'DEPTH', 'ML']))

现在，检查YYYY 列的值：

new_data['YYYY'].unique()

也许，您会看到 'YYYY' 作为值。要找到这些行：

new_data[new_data['YYYY'] == 'YYYY']

Answer 2

根据您的建议，我在代码中添加以下行：

new_data_d= new_data[new_data.YYYY.str.contains('YYYY') == False]

然后删除了带有 YYYY,SS... 的重复行。最终代码如下所示：

os.chdir(r'D:\Inves\Catalogs\OSC')
path = os.getcwd()
csv_files = glob.glob(os.path.join(path, "*.txt"))

new_data = []

for f in csv_files:
    df = pd.read_csv(f)
    print('Location File:', f)
    print('File Name:', f.split("\\")[-1])
    df = pd.read_csv(f, header=10, sep='\s+')
    n = 2
    df.drop(df.tail(n).index, inplace = True)
    df = df[df.YYYY != '----'] # deleting the '----' row
    print('File Content:')
    
    print('...Appending...')
    print('...................')
    new_data.append(df)
new_data = pd.concat(new_data, ignore_index=True)
new_data_d= new_data[new_data.YYYY.str.contains('YYYY') == False]
new_data_d.to_csv(r'D:\Inves\Catalogs\Full_1988-2008.csv',
                index=False, header=True, sep=',')