为什么在将 csv 加载到 pyspark 数据帧时类型都是字符串？答案

【问题标题】：Why the types are all string while load csv to pyspark dataframe?为什么在将 csv 加载到 pyspark 数据帧时类型都是字符串？
【发布时间】：2017-06-19 08:42:50
【问题描述】：

我有一个包含数字的 csv 文件（其中没有字符串）。它有 int 和 float 类型。但是当我以这种方式在 pyspark 中阅读时：

df = spark.read.csv("s3://s3-cdp-prod-hive/novaya/instacart/data.csv",header=False)

数据框的所有列类型都是字符串。

如何自动将其读入int和float的数字？

某些列中包含 nan。在文件中它由nan表示

0.18277,-0.188931,0.0893389,0.119931,0.318853,-0.132933,-0.0288816,0.136137,0.12939,-0.245342,0.0608182,0.0802028,-0.00625962,0.271222,0.187855,0.132606,-0.0451533,0.140501,0.0704631,0.0229986,-0.0533376,-0.319643,-0.029321,-0.160937,0.608359,0.0513554,-0.246744,0.0817331,-0.410682,0.210652,0.375154,0.021617,0.119288,0.0674939,0.190642,0.161885,0.0385196,-0.341168,0.138659,-0.236908,0.230963,0.23714,-0.277465,0.242136,0.0165013,0.0462388,0.259744,-0.397228,-0.0143719,0.0891644,0.222225,0.0987765,0.24049,0.357596,-0.106266,-0.216665,0.191123,-0.0164234,0.370766,0.279462,0.46796,-0.0835098,0.112693,0.231951,-0.0942302,-0.178815,0.259096,-0.129323,1165491,175882,16.5708805975,6,0,2.80890261184,4.42114773551,0,23,0,13.4645462866,18.0359037455,11,30.0,0.0,11.4435397208,84.7504967125,30.0,5370,136.0,1.0,9.61508192633,62.2006926209,1,0,0,22340,9676,322.71241867,17.7282900627,1,100,4.24701125287,2.72260519248,0,6,17.9743048247,13.3241271262,0,23,82.4988407009,11.4021333588,0.0,30.0,45.1319021862,7.76284691137,1.0,66.0,9.40127026245,2.30880529144,1,73,0.113021725659,0.264843289305,0.0,0.986301369863,1,30450,0

【问题讨论】：

标签： dataframe pyspark

【解决方案1】：

如你所见here:

inferSchema – 从数据中自动推断输入模式。它需要对数据进行一次额外的传递。如果设置了 None，则使用默认值 false。

对于 NaN 值，请参阅上面的相同文档：

nanValue – 设置非数字值的字符串表示。如果设置了 None，则使用默认值 NaN

通过将 inferSchema 设置为 True，您将获得具有推断类型的数据框。

这里我举个例子：

CSV 文件：

12,5,8,9
1.0,3,46,NaN

默认情况下，inferSchema 为 False，所有值均为 String：

from pyspark.sql.types import *

>>> df = spark.read.csv("prova.csv",header=False) 
>>> df.dtypes
[('_c0', 'string'), ('_c1', 'string'), ('_c2', 'string'), ('_c3', 'string')]

>>> df.show()
+---+---+---+---+
|_c0|_c1|_c2|_c3|
+---+---+---+---+
| 12|  5|  8|  9|
|1.0|  3| 46|NaN|
+---+---+---+---+

如果您将 inferSchema 设置为 True：

>>> df = spark.read.csv("prova.csv",inferSchema =True,header=False) 
>>> df.dtypes
[('_c0', 'double'), ('_c1', 'int'), ('_c2', 'int'), ('_c3', 'double')]


>>> df.show()
+----+---+---+---+
| _c0|_c1|_c2|_c3|
+----+---+---+---+
|12.0|  5|  8|9.0|
| 1.0|  3| 46|NaN|
+----+---+---+---+

【讨论】：