如何在pyspark的数据框中的每一行中查找一个字符串

Question

这是可用的数据框：

+--------------------+
|                Name|
+--------------------+
|Braund, Mr. Owen ...|
|Cumings, Mrs. Joh...|
|Heikkinen, Miss. ...|
|Futrelle, Mrs. Ja...|
|Allen, Mr. Willia...|
|Moran, Mr. James|
|McCarthy, Mr. Tim...|
|Palsson, Master. ...|
|Johnson, Mrs. Osc...|
+--------------------+

我想使用 Pyspark 在 DATA FRAME 的每一行中查找 Title 和 Surname 的第一个匹配项（Pandas lib 在我的集群中不可用）。

pattern=re.compile(r'(Dr|Mrs?|Ms|Miss|Master|Rev|Capt|Mlle|Col|Major|Sir|Lady|Mme|Don)\\.'
pattern.match(df['Name'])

Answer 1

如果Name 列的第一个单词是“姓氏”，那么您可以试试这个，否则正则表达式需要稍作调整。

from pyspark.sql.functions import regexp_extract, col

#sample data
df= sc.parallelize([["Braund, Mr. Owen"], 
                    ["Cumings, Mrs. Joh"], 
                    ["Heikkinen, Miss."], 
                    ["Futrelle, Mrs. Ja"]]).toDF(["Name"])

df = df.withColumn('Surname', regexp_extract(col('Name'), '(\S+),.*', 1))
df.show()

样本数据：

+-----------------+
|             Name|
+-----------------+
| Braund, Mr. Owen|
|Cumings, Mrs. Joh|
| Heikkinen, Miss.|
|Futrelle, Mrs. Ja|
+-----------------+

输出是：

+-----------------+---------+
|             Name|  Surname|
+-----------------+---------+
| Braund, Mr. Owen|   Braund|
|Cumings, Mrs. Joh|  Cumings|
| Heikkinen, Miss.|Heikkinen|
|Futrelle, Mrs. Ja| Futrelle|
+-----------------+---------+

Answer 2

您可以按照@Prem 的建议使用regexp_extract，但使用不同的正则表达式模式，具体取决于您的需要：

# do not keep the first two groups, just what follows, the surname:
pattern = r'(?:(?:Dr|Mrs?|Ms|Miss|Master|Rev|Capt|Mlle|Col|Major|Sir|Lady|Mme|Don)\.?\s?)(\w+)'

# or keep both title and surname
pattern_with_title = r'((Dr|Mrs?|Ms|Miss|Master|Rev|Capt|Mlle|Col|Major|Sir|Lady|Mme|Don)\.?\s?)(\w+)'

#sample data
df = spark.createDataFrame([["Braund, Mr. Owen other stuff"], 
                       ["Cumings, Mrs. Joh some details"], 
                       ["Heikkinen, Miss. Hellen blah"], 
                       ["Futrelle, Mrs. Ja .... .... "]], ["Name"])

df.show()

+-----------------+
|             Name|
+-----------------+
| Braund, Mr. Owen|
|Cumings, Mrs. Joh|
| Heikkinen, Miss.|
|Futrelle, Mrs. Ja|
+-----------------+

# create a column with what matches the pattern
df = df.withColumn("Surname", regexp_extract("Name", pattern, 1))

df.show()
# keeps only the Surname
+-----------------+---------+
|             Name|  Surname|
+-----------------+---------+
| Braund, Mr. Owen| Owen    |
|Cumings, Mrs. Joh| Joh     |
| Heikkinen, Miss.| Hellen  |
|Futrelle, Mrs. Ja| Ja      |
+-----------------+---------+

 # in case you want both title and Surname:
 df = df.withColumn("Surname with title", regexp_extract("Name", pattern_with_title, 1))

+-----------------+---------+--------------------+
|             Name|  Surname|  Surname with title|
+-----------------+---------+--------------------+
|Braund, Mr. Owen | Owen    | Mr. Ownen          |
|Cumings, Mrs. Joh| Joh     | Mrs. Joh           |
|Heikkinen, Miss..| Hellen  | Miss. Hellen       |
|Futrelle, Mrs. Ja| Ja      | Mrs. Ja            |
+-----------------+---------+--------------------+

如果您需要全名、姓氏，则稍微更改模式以将它们也包括在内，例如：

main_pattern = r'Dr|Mrs?|Ms|Miss|Master|Rev|Capt|Mlle|Col|Major|Sir|Lady|Mme|Don'

pattern_full = r'(\w+,?\s('+ main_pattern+')\.?\s?\w+)'
pattern_name = r'(?:(?:'+ main_pattern+')\.?\s?)(\w+)'
pattern_title = r'(?:('+ main_pattern+')\.?\s?)'
pattern_surname = r'(\w+)(?:\,\s?(?:'+ main_pattern+')\.?\s?)'

df = df.withColumn("Full Name", regexp_extract("Name", pattern_full, 1))
df = df.withColumn("First Name", regexp_extract("Name", pattern_name, 1))
df = df.withColumn("Surname", regexp_extract("Name", pattern_surname, 1))
df = df.withColumn("Title", regexp_extract("Name", pattern_title, 1))

df.show(10, False)

+------------------------------+-----------------------+----------+------------+-----+
|Name                          |Full Name              |Surname   |First Name  |Title|
+------------------------------+-----------------------+----------+------------+-----+
|Braund, Mr. Owen other stuff  |Braund, Mr. Owen       |Braund    |Owen        |Mr   |
|Cumings, Mrs. Joh some details|Cumings, Mrs. Joh      |Cumings   |Joh         |Mrs  |
|Heikkinen, Miss. Hellen blah  |Heikkinen, Miss. Hellen|Heikkinen |Hellen      |Miss |
|Futrelle, Mrs. Ja .... ....   |Futrelle, Mrs. Ja      |Futrelle  |Ja          |Mrs  |
+------------------------------+-----------------------+----------+------------+-----+

这完全是关于在正则表达式中忽略哪个部分和选择哪个部分。希望这会有所帮助，祝你好运！

注意：不是最佳的正则表达式，它还有改进的空间。