PySpark：AttributeError：“DataFrame”对象没有属性“值”答案

【问题标题】：PySpark : AttributeError: 'DataFrame' object has no attribute 'values'PySpark：AttributeError：“DataFrame”对象没有属性“值”
【发布时间】：2021-10-03 14:01:18
【问题描述】：

我是 PySpark 的新手，我想将以下 Python 脚本翻译成 pyspark：

api_param_df = pd.DataFrame([[row[0][0], np.nan] if row[0][1] == '' else row[0] for row in http_path.values], columns=["api", "param"])
df = pd.concat([df['raw'], api_param_df], axis=1)

但我遇到以下错误，错误引用如下：

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-18-df055fb7d6a1> in <module>()
     21 # Notice we also make \? and the second capture group optional so that when there are no query parameters in http path, it returns NaN.
     22 
---> 23 api_param_df = pd.DataFrame([[row[0][0], np.nan] if row[0][1] == '' else row[0] for row in http_path.values], columns=["api", "param"])
     24 df = pd.concat([df['raw'], api_param_df], axis=1)
     25 

/usr/local/lib/python3.7/dist-packages/pyspark/sql/dataframe.py in __getattr__(self, name)
   1642         if name not in self.columns:
   1643             raise AttributeError(
-> 1644                 "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
   1645         jc = self._jdf.apply(name)
   1646         return Column(jc)

AttributeError: 'DataFrame' object has no attribute 'values'

完整脚本如下，注释说明使用regex应用于df中的特定列http_path解析api和param并再次合并/合并它们到df。

#Extract features from http_path ["API URL", "URL parameters"]

regex = r'([^\?]+)\?*(.*)'
http_path = df.filter(df['http_path'].rlike(regex))

#                   http_path
#0  https://example.org/path/to/file?param=42#frag...
#1  https://example.org/path/to/file

#                 api                              param
#0  https://example.org/path/to/file         param=42#fragment
#1  https://example.org/path/to/file                NaN

#where in regex pattern:

#- (?:https?://[^/]+/)? optionally matches domain but doesn't capture it
#- (?P<api>[^?]+) matches everything up to ?
#- \? matches ? literally
#- (?P<param>.+) matches everything after ?

# Notice we also make \? and the second capture group optional so that when there are no query parameters in http_path, it returns NaN.

api_param_df = pd.DataFrame([[row[0][0], np.nan] if row[0][1] == '' else row[0] for row in http_path.values], columns=["api", "param"])
df = pd.concat([df['raw'], api_param_df], axis=1)

df

任何帮助将不胜感激。

【问题讨论】：

标签： pandas dataframe pyspark apache-spark-sql

【解决方案1】：

语法对 Pandas DataFrames 有效，但 PySpark 创建的 DataFrames 不存在该属性。您可以查看 this link 获取文档。

通常，collect() 方法或.rdd 属性可以帮助您完成这些任务。

您可以使用以下 sn-p 来产生所需的结果：

http_path = sdf.rdd.map(lambda row: row['http_path'].split('?'))
api_param_df = pd.DataFrame([[row[0], np.nan] if len(row) == 1 else row for row in http_path.collect()], columns=["api", "param"])
sdf = pd.concat([sdf.toPandas()['raw'], api_param_df], axis=1)

请注意，我删除了 cmets 以使其更具可读性，并且我还用简单的拆分替换了 regex。

【讨论】：