导入包含文本引号的混合 CSV

Question

我正在将逗号分隔的 CSV 文件导入 MATLAB。每列都有引号，我想将其视为文本，然后是逗号。

我正在使用这个问题的答案中的 read_mixed_csv 函数以单元格的形式读取数据：Import CSV file with mixed data types

thisdata = read_mixed_csv(fname, ','); % Reads in the CSV file 
thisdata = regexprep(thisdata, '^"|"$','');

但是，由于我的一些专栏看起来像这样：

"FAIRHOPE, Alabama"
"FAIRHOPE HIGH SCHOOL, FAIRHOPE,  ALABAMA"
"Daphne-Fairhope-Foley, AL"

MATLAB 将逗号后的所有内容放入一个新列中。所以

"Daphne-Fairhope-Foley, AL"

变成两列

"Daphne-Fairhope-Foley
AL"

如何让 MATLAB 读取混合的 csv 文件，并且不仅将逗号视为分隔符，还考虑引号？ 有没有比 @ 更自动化的方法？ 987654326@?如果textscan 是一个选项，那会是什么样子？

这是我尝试读取的数据示例，其中包含标题：

"State Code","County Code","Site Num","Parameter Code","POC","Latitude","Longitude","Datum","Parameter Name","Sample Duration","Pollutant Standard","Date Local","Units of Measure","Event Type","Observation Count","Observation Percent","Arithmetic Mean","1st Max Value","1st Max Hour","AQI","Method Name","Local Site Name","Address","State Name","County Name","City Name","CBSA Name","Date of Last Change"
"01","003","0010","88101",1,30.498001,-87.881412,"NAD83","PM2.5 - Local Conditions","24 HOUR","PM25 24-hour 2006","2013-01-01","Micrograms/cubic meter (LC)","None",1,100.0,7.3,7.3,0,30,"R & P Model 2025 PM2.5 Sequential w/WINS - GRAVIMETRIC","FAIRHOPE, Alabama","FAIRHOPE HIGH SCHOOL, FAIRHOPE,  ALABAMA","Alabama","Baldwin","Fairhope","Daphne-Fairhope-Foley, AL","2014-02-11"
"01","003","0010","88101",1,30.498001,-87.881412,"NAD83","PM2.5 - Local Conditions","24 HOUR","PM25 24-hour 2006","2013-01-04","Micrograms/cubic meter (LC)","None",1,100.0,7.6,7.6,0,32,"R & P Model 2025 PM2.5 Sequential w/WINS - GRAVIMETRIC","FAIRHOPE, Alabama","FAIRHOPE HIGH SCHOOL, FAIRHOPE,  ALABAMA","Alabama","Baldwin","Fairhope","Daphne-Fairhope-Foley, AL","2014-02-11"
"01","003","0010","88101",1,30.498001,-87.881412,"NAD83","PM2.5 - Local Conditions","24 HOUR","PM25 24-hour 2006","2013-01-07","Micrograms/cubic meter (LC)","None",1,100.0,8.6,8.6,0,36,"R & P Model 2025 PM2.5 Sequential w/WINS - GRAVIMETRIC","FAIRHOPE, Alabama","FAIRHOPE HIGH SCHOOL, FAIRHOPE,  ALABAMA","Alabama","Baldwin","Fairhope","Daphne-Fairhope-Foley, AL","2014-02-11"
"01","003","0010","88101",1,30.498001,-87.881412,"NAD83","PM2.5 - Local Conditions","24 HOUR","PM25 24-hour 2006","2013-01-10","Micrograms/cubic meter (LC)","None",1,100.0,7,7,0,29,"R & P Model 2025 PM2.5 Sequential w/WINS - GRAVIMETRIC","FAIRHOPE, Alabama","FAIRHOPE HIGH SCHOOL, FAIRHOPE,  ALABAMA","Alabama","Baldwin","Fairhope","Daphne-Fairhope-Foley, AL","2014-02-11"

*注意：将 CSV 文件转换为制表符分隔文件可以让 MATLAB 更轻松地处理和规避此问题。

Answer 1

使用文本限定符（如 "）有点棘手，但如果您确保表格的每一行都具有相同数量的列（并且可能没有空列），则以下方法可能会起作用。

文本限定符之外的任何内容都必须可转换为数字。

function C = csvmixed(eachLine,delim,textQualifier)
% Outputs cell containing mixed string and numeric data given a delimiter (',') 
% and a text qualifier ('"').  Each line of the delimited file must be loaded into 
% the cell array eachLine, and each line must have the same number of columns.
% 
% Example:
%   fid = fopen('testcsv.txt','r');
%   eachLine = textscan(fid,'%s','Delimiter','\n'); fclose(fid);
%   C = csvmixed(eachLine{1},',','"')

assert(ischar(delim) && numel(delim)==1);
assert(ischar(textQualifier) && numel(textQualifier)==1);

% find strings, as specified by the input qualifier
patternStr = sprintf('"([^"]*)"%c?',delim);
patternStr = strrep(patternStr,'"',textQualifier);
Cstr = regexp(eachLine,patternStr,'tokens');

% find numeric data
patternNum = sprintf('(?<=(,|^))[^%c,a-zA-Z]*(?=(,|$))',textQualifier);
patternNum = strrep(patternNum,',',delim);
Cnum = regexp(eachLine,patternNum,'match','emptymatch');

numCols = cellfun(@numel,Cstr) + cellfun(@numel,Cnum);
assert(nnz(diff(numCols))==0,'Number of columns not consistent.')

% get string extents (begin, start) indexes for each string
strExtents = regexp(eachLine,patternStr,'tokenExtents');

% deal out parsed data for each line
C = cell(numel(eachLine),numCols(1));
for ii = 1:numel(eachLine),
    strBounds = vertcat(strExtents{ii}{:});
    delimLocs = getDelimLocs(eachLine{ii},strBounds,delim);
    strCellMap = getCellMap(strBounds,delimLocs);

    C(ii,strCellMap) = [Cstr{ii}{:}]; % TODO: preallocate
    C(ii,~strCellMap) = num2cell(str2double(Cnum{ii})); % all else must be numeric
end

end

function delimLocs = getDelimLocs(lineText,solidBounds,delim)
    delimCharLocs = strfind(lineText,delim);
    delimLocs = delimCharLocs(~any(bsxfun(@ge,delimCharLocs,solidBounds(:,1)) & ...
        bsxfun(@le,delimCharLocs,solidBounds(:,2)),1));
end

function cellMap = getCellMap(typeBounds,delimLocs)
    cellMap = any(bsxfun(@gt,typeBounds(:,1),[0 delimLocs]) & ...
        bsxfun(@lt,typeBounds(:,1),[delimLocs Inf]), 1);
end

更新：修正getDelimLocs 中的小错别字。添加元胞数组的预分配。

Answer 2

使用文件交换代码replaceinfile 将其中包含逗号的字符串替换为句点。 使用Import CSV file with mixed data types 中的read_mixed_csv 读取文件。 从仍然留下的字符串中删除多余的引号。

replaceinfile(', ', '. ', fname); % Replace commas that was inside quotes and not meant to be separated as periods so they don't show up as a new column
thisdata = read_mixed_csv(fname, ','); % Reads in the CSV file (\t for tab)
thisdata = regexprep(thisdata, '^"|"$',''); % Remove quotes from file and only keep the first 28 columns (last two columns are empty)

对于replaceinfile.m 函数： 要在 Linux 上运行代码，请将 Perl 部分的第一行更改为

perlCmd = sprintf('"%s"', '/usr/bin/perl');