PySpark - 用于查找 URL 和正确地址的正则表达式答案

【问题标题】：PySpark - Regular expression to find URLs and correct addressPySpark - 用于查找 URL 和正确地址的正则表达式
【发布时间】：2020-10-01 09:37:18
【问题描述】：

我有日志文件。他们或多或少看起来像这样。我想清理一下它们并获得正确的顺序，因为它是真正的链接。

想知道是否有人知道如何在 py(spark) 中编写正则表达式来获得所需的输出。

1: 

https%3A%2F%2Fwww.btv.com%2Fnews%2Ffinland%2Fartikel%2F5174938%2Fzwemmer-zoekactie-julianadorp-kinderen-gered

Desired Output 

https://www.btv.com/news/finland/artikel/5174938/zwemmer-zoekactie-julianadorp-kinderen-gered


2: 
https%3A%2F%2Fwww.weather.com%2F

Desired Output 
https://www.weather.com


3:
https%3A%2F%2Fwww.weather.com%2Ffinland%2Fneerslag%2Fweather%2F3uurs

Desired Output 
https://www.weather.com/finland/neerslag/ weather /uurs

我尝试了几个解决方案，但没有太多理解。

    \b\w+\b(?!\/)


   from pyspark.sql.functions import regexp_extract, col
   regexp_extract(column_name, regex, group_number)
   regex('(.)(by)(\s+)(\w+)')

提前致谢

【问题讨论】：

这能回答你的问题吗？ Url decode UTF-8 in Python
@ChristianBaumann 有点帮助。但是我也遇到了android-app%3A%2F%2Fcom.google.android.googlequicksearchbox%2Fhttps%2Fwww.google.com

标签： python regex url pyspark

【解决方案1】：

您可以使用urlib.parse.unqoute，并且您必须制作一个 udf 才能将其与 pyspark 一起使用。

from urllib.parse import unquote
from pyspark.sql.types import StringType
from pyspark.sql.functions import udf
df = spark.createDataFrame([['https%3A%2F%2Fwww.btv.com%2Fnews%2Ffinland%2Fartikel%2F5174938%2Fzwemmer-zoekactie-julianadorp-kinderen-gered'],
                            ['https%3A%2F%2Fwww.weather.com%2F'],
                            ['https%3A%2F%2Fwww.weather.com%2Ffinland%2Fneerslag%2Fweather%2F3uurs']],['url'])

urldecode_udf = udf(lambda x:unquote(x) , StringType())
df = df.withColumn("decodedurl",urldecode_udf(df.url))
df.select('decodedurl').show(3,False)

输出：

+---------------------------------------------------------------------------------------------+
|decodedurl                                                                                   |
+---------------------------------------------------------------------------------------------+
|https://www.btv.com/news/finland/artikel/5174938/zwemmer-zoekactie-julianadorp-kinderen-gered|
|https://www.weather.com/                                                                     |
|https://www.weather.com/finland/neerslag/weather/3uurs                                       |
+---------------------------------------------------------------------------------------------+

【讨论】：