导入 fread vs read.table 和错误答案

【问题标题】：Importing fread vs read.table and errors导入 fread vs read.table 和错误
【发布时间】：2018-02-01 15:59:32
【问题描述】：

当我通过调用 df <- read.table("ModelSugar(new) real_thesis_experiment-table_1.csv", skip = 6, sep = ",", head = TRUE) 导入带有 read.table 的 .csv 文件并检查我得到的数据的摘要时（仅显示 45 列的前 3 列）：

 X.run.number. scenario        configuration   
 Min.   :   1 "pessimistic":999994   "central":999994  
 1st Qu.: 650                                            
 Median :1299                                            
 Mean   :1299                                            
 3rd Qu.:1949                                            
 Max.   :2600

有了这个数据框，我可以制作漂亮的图形。但是，我有 80 个 .csv 文件，总大小为 40 GB，所以我只想导入特定的列。

我认为使用fread（来自 data.table 包）会更容易。所以我导入了 5 列并通过调用将它们合并到一个数据帧中

my.files <- list.files(pattern=".csv")
my.data <- lapply(my.files,fread, header = FALSE, select = c(1,2,3,25,29), sep=",") 
df <- do.call("rbind", my.data)

该数据框的摘要看起来像（显示了 5 列中的 4 列：

[run number]         scenario         configuration         [step]         
 Length:999994      Length:999994      Length:999994      Length:999994     
 Class :character   Class :character   Class :character   Class :character  
 Mode  :character   Mode  :character   Mode  :character   Mode  :character

使用此数据框，我无法制作使用 read.table 可以制作的图形。我猜这与列值的类别有关。

如何确保使用 fread 创建的数据框与使用 read.table 创建的数据框具有相同的特征，以便制作我想要的图形？

编辑

我发现当我第一次将 excel 中的 .csv 拆分为列，然后将 fread 调用与 sep = ";" 一起使用时而不是 sep = ","，它确实有效。奇怪...而且我不想手动将 .csv 文件转换为 excel 中的列。

【问题讨论】：

你在说什么fread？包裹data.table?您能否也将电话分享给fread 和read.table？
@Marijn ...请使用电话编辑您的帖子，然后删除这些 cmets。

标签： r import read.table

【解决方案1】：

dfshort 的前 5 列（共 45 列）如下所示：

   X.run.number.      scenario configuration biobased.chemical.industry
1              3 "pessimistic"     "central"    "modification-dominant"
2              2 "pessimistic"     "central"    "modification-dominant"
3              3 "pessimistic"     "central"    "modification-dominant"
4              4 "pessimistic"     "central"    "modification-dominant"
5              2 "pessimistic"     "central"    "modification-dominant"
6              1 "pessimistic"     "central"    "modification-dominant"
7              3 "pessimistic"     "central"    "modification-dominant"
8              3 "pessimistic"     "central"    "modification-dominant"
9              2 "pessimistic"     "central"    "modification-dominant"
10             4 "pessimistic"     "central"    "modification-dominant"
   distributed.sugar.factory.investment.costs
1                                    70000000
2                                    70000000
3                                    70000000
4                                    70000000
5                                    70000000
6                                    70000000
7                                    70000000

模板如下所示：

 run_number      scenario configuration tick financial_balance_SU
1          3 "pessimistic"     "central"    0                    0
2          2 "pessimistic"     "central"    0                    0
3          3 "pessimistic"     "central"    1                    0
4          4 "pessimistic"     "central"    0                    0
5          2 "pessimistic"     "central"    1                    0
6          1 "pessimistic"     "central"    0                    0

df 看起来像这样：

   run_number        scenario configuration tick financial_balance_SU
1:      23377 ""pessimistic""     ""mixed""  200  6.079728695488823E9
2:      23377 ""pessimistic""     ""mixed""  201  6.079728695488823E9
3:      23378 ""pessimistic""     ""mixed""  192   9.10006561818864E9
4:      23377 ""pessimistic""     ""mixed""  202  6.079728695488823E9
5:      23377 ""pessimistic""     ""mixed""  203  6.079728695488823E9
6:      23378 ""pessimistic""     ""mixed""  193   9.10006561818864E9

编辑

str(dfshort)

'data.frame':   10 obs. of  45 variables:
 $ X.run.number.                                        : int  3 2 3 4 2 1 3 3 2 4
 $ scenario                                             : Factor w/ 1 level "\"pessimistic\"": 1 1 1 1 1 1 1 1 1 1
 $ configuration                                        : Factor w/ 1 level "\"central\"": 1 1 1 1 1 1 1 1 1 1
 $ biobased.chemical.industry                           : Factor w/ 1 level "\"modification-dominant\"": 1 1 1 1 1 1 1 1 1 1
 $ distributed.sugar.factory.investment.costs           : int  70000000 70000000 70000000 70000000 70000000 70000000 70000000 70000000 70000000 70000000
 $ beet.syrups.factory.investment.costs                 : int  1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000 1000000
 $ ethanol.factory.investment.costs                     : int  1500000 1500000 1500000 1500000 1500000 1500000 1500000 1500000 1500000 1500000
 $ market.share.beet.syrups.increase                    : num  0.002 0.002 0.002 0.002 0.002 0.002 0.002 0.002 0.002 0.002
 $ demand.beets.for.chemical.EU.increase                : num  0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01
 $ transport.costs                                      : int  1 1 1 1 1 1 1 1 1 1
 $ washing.at.farmer                                    : Factor w/ 1 level "\"no\"": 1 1 1 1 1 1 1 1 1 1
 $ beet.syrups.price.percentage.of.sugar.price          : num  0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3
 $ CO2.tax.                                             : Factor w/ 1 level "\"yes\"": 1 1 1 1 1 1 1 1 1 1
 $ sugar.tax                                            : num  0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2
 $ CO2.tax                                              : int  13 13 13 13 13 13 13 13 13 13
 $ market.share.increase.period                         : int  10 10 10 10 10 10 10 10 10 10
 $ electricity.source                                   : Factor w/ 1 level "\"conventional-mix\"": 1 1 1 1 1 1 1 1 1 1
 $ white.sugar.price.EU.maximum                         : int  1000 1000 1000 1000 1000 1000 1000 1000 1000 1000
 $ white.sugar.price.EU.minimum                         : int  200 200 200 200 200 200 200 200 200 200
 $ beet.syrups.price.EU.maximum                         : int  500 500 500 500 500 500 500 500 500 500
 $ beet.syrups.price.EU.minimum                         : int  100 100 100 100 100 100 100 100 100 100
 $ ethanol.price.EU.maximum                             : int  2 2 2 2 2 2 2 2 2 2
 $ ethanol.price.EU.minimum                             : num  0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3
 $ years.taken.into.account                             : int  5 5 5 5 5 5 5 5 5 5
 $ X.step.                                              : int  0 0 1 0 1 0 2 3 2 1
 $ financial.balance.farmers                            : num  0 0 0 0 0 ...
 $ diesel.use.farmers                                   : int  0 0 0 0 0 0 0 0 0 0
 $ N.use.farmers                                        : int  0 0 0 0 0 0 0 0 0 0
 $ financial.balance.SU                                 : num  0 0 0 0 0 ...
 $ electricity.use.SU                                   : int  0 0 0 0 0 0 0 0 0 0
 $ financial.balance.central.sugar.factories            : num  0 0 0 0 0 ...
 $ electricity.use.central.sugar.factories              : int  0 0 0 0 0 0 0 0 0 0
 $ financial.balance.distributed.sugar.factories        : num  0 0 0 0 0 ...
 $ electricity.use.distributed.sugar.factories          : int  0 0 0 0 0 0 0 0 0 0
 $ financial.balance.beet.syrups.factories              : int  0 0 0 0 0 0 0 0 0 0
 $ electricity.use.beet.syrups.factories                : int  0 0 0 0 0 0 0 0 0 0
 $ financial.balance.ethanol.factories                  : int  0 0 0 0 0 0 0 0 0 0
 $ electricity.use.ethanol.factories                    : int  0 0 0 0 0 0 0 0 0 0
 $ transport.costs.yearly                               : num  0 0 0 0 0 ...
 $ diesel.use.total.transport                           : num  0 0 0 0 0 ...
 $ profit.per.tonne.sugar.beet.central.sugar.factory    : num  0 0 0 0 0 ...
 $ profit.per.tonne.sugar.beet.distributed.sugar.factory: num  0 0 0 0 0 ...
 $ profit.per.tonne.sugar.beet.sugar.from.beet.syrups   : int  0 0 0 0 0 0 0 0 0 0
 $ profit.per.tonne.sugar.beet.beet.syrups.factory      : int  0 0 0 0 0 0 0 0 0 0
 $ profit.per.tonne.sugar.beet.ethanol.factory          : num  0 0 0 0 0 ...

str(df)

Classes ‘data.table’ and 'data.frame':  19000000 obs. of  5 variables:
 $ run_number          : chr  "23377" "23377" "23378" "23377" ...
 $ scenario            : chr  "\"\"pessimistic\"\"" "\"\"pessimistic\"\"" "\"\"pessimistic\"\"" "\"\"pessimistic\"\"" ...
 $ configuration       : chr  "\"\"mixed\"\"" "\"\"mixed\"\"" "\"\"mixed\"\"" "\"\"mixed\"\"" ...
 $ tick                : chr  "200" "201" "192" "202" ...
 $ financial_balance_SU: chr  "6.079728695488823E9" "6.079728695488823E9" "9.10006561818864E9" "6.079728695488823E9" ...
 - attr(*, ".internal.selfref")=<externalptr>

str(模板)

'data.frame':   10 obs. of  5 variables:
 $ run_number          : int  3 2 3 4 2 1 3 3 2 4
 $ scenario            : Factor w/ 1 level "\"pessimistic\"": 1 1 1 1 1 1 1 1 1 1
 $ configuration       : Factor w/ 1 level "\"central\"": 1 1 1 1 1 1 1 1 1 1
 $ tick                : int  0 0 1 0 1 0 2 3 2 1
 $ financial_balance_SU: num  0 0 0 0 0 ...

【讨论】：

将列名添加到您的 df.用标题阅读它。一点都不清楚。
我现在更改了template 和df 的列名，使它们完全相同。当我现在尝试创建result 时，出现以下错误：Error in as(df[[x]], class(template[[x]])) : no method or default for coercing “character” to “factor”
发送 dfshort、模板和 df 的 str() 结果
查看 str() 结果的编辑

【解决方案2】：

你可以做的是用 write.csv 读取一个文件并将该文件的 10 行保存为模板，然后你可以执行以下操作-

## Getting your files using fread
dfshort <- read.table("ModelSugar(new) real_thesis_experiment-table_1.csv", skip = 6, sep = ",", nrows = 10, head = TRUE)
df_needed<-dfshort[1:10]
template <- subset(df_needed,select=c(columns_required)) ##select whatever cols you need

##Read you large files using fread
my.files <- list.files(pattern=".csv")
my.data <- lapply(my.files,fread, header = FALSE, select = c(1,2,3,25,29), sep=",") 
df <- do.call("rbind", my.data)

## changing cols types as per your template
result = data.frame(
  lapply(setNames(,names(template)), function(x) 
    if (x %in% names(df)) as(df[[x]], class(template[[x]])) 
    else template[[x]][NA_integer_]
  ), stringsAsFactors = FALSE)

然后，您可以使用它进行绘图，因为它具有与您使用 write.csv 获得的相同的类类型。

试试这个

dfshort <- read.table("ModelSugar(new) real_thesis_experiment-table_1.csv", skip = 6, sep = ",", nrows = 10, head = TRUE)
    template <- copy(dfshort)
    my.files <- list.files(pattern=".csv")
    my.data <- lapply(my.files,fread, header = FALSE, colClasses = c(1,2,3,25,29), sep=",") 
    df <- do.call("rbind", my.data)

    result = data.frame(
      lapply(setNames(,names(template)), function(x) 
        if (x %in% names(df)) as(df[[x]], class(template[[x]])) 
        else template[[x]][NA_integer_]
      ), stringsAsFactors = FALSE)

【讨论】：

@Rushabh。谢谢回复。但是，我无法用nrow = 10 处理write.csv。它不使用nrows。你会怎么做？
@Marjin 只需使用 nrow 从您的数据框中读取 10 行，然后使用 write.csv() 保存它
@Margin 否则您可以使用 dt[1:10,] 子集数据并使用 write.csv() 保存它
从你的意思的一个 csv 文件中读取 10 行，而不是从数据框中？我需要阅读 5 栏还是所有栏目？
视情况而定。您只需将模板（即 10 行）与您想要可视化的任何列一起保存。