如何将 URL 组合在一起？

Question

假设我有这样的数据集：

url	event	timestamp
https://name/view/adsd-12455-adf1/show	view	1
https://name/close/anotherpage/12dgksdfgas-adsjasf-54a4551/	close	2
https://name/close/anotherpage/98713sdfspdf-asdaj-1/	close	3
https://name/view/45asdaj-asdasd-lf5633/show	view	6
https://name/close/anotherpage/89kkpi-fslo-s521344/	close	10
https://name/view/124sfsdf-ajsdasd-551/delete	view	20
https://name/purchase/6654asdasd-asdfd-asda12	purchase	30

假设我所有的 URL 都采用 https://name/pagename/anotherpage/123456/randompage 这样的格式，其中 123456 就像 URL 名称中的唯一用户 ID，并且对于每个用户都是唯一的。 （注意：我在这里使用了 123456 或仅作为唯一用户 ID 的数字作为示例，实际上用户 ID 也可能包含字母）

我想向数据集添加另一列，将引用同一页面但针对不同用户的 URL 组合在一起。所以表格看起来像这样：

url	event	timestamp	inferred url
https://name/view/adsd-12455-adf1/show	view	1	https://name/view/*/show
https://name/close/anotherpage/12dgksdfgas-adsjasf-54a4551/	close	2	https://name/close/anotherpage/*/
https://name/close/anotherpage/98713sdfspdf-asdaj-1/	close	3	https://name/close/anotherpage/*/
https://name/view/45asdaj-asdasd-lf5633/show	view	6	https://name/view/*/show
https://name/close/anotherpage/89kkpi-fslo-s521344/	close	10	https://name/close/anotherpage/*/
https://name/view/124sfsdf-ajsdasd-551/delete	view	20	https://name/view/*/delete
https://name/purchase/6654asdasd-asdfd-asda12	purchase	30	https://anothername/purchase/*/

我如何在 PySpark 中做到这一点？我需要有关如何解决此问题的帮助。

编辑：由于 Steven 是对的，而且我的示例数据太简单，所以我对其进行了一些更改

Answer 1

如果您想要一个示例案例的解决方案，那么它只是一个正则表达式：

from pyspark.sql import functions as F 

df.withColumn("inferred_url", F.regexp_replace("url", r"\d+", "\*")).show(
    truncate=False
)
+--------------------------------------+--------+---------+---------------------------------+
|url                                   |event   |timestamp|inferred_url                     |
+--------------------------------------+--------+---------+---------------------------------+
|https://name/view/124551/show         |view    |1        |https://name/view/*/show         |
|https://name/close/anotherpage/124551/|close   |2        |https://name/close/anotherpage/*/|
|https://name/close/anotherpage/987131/|close   |3        |https://name/close/anotherpage/*/|
|https://name/view/455633/show         |view    |6        |https://name/view/*/show         |
|https://name/close/anotherpage/891344/|close   |10       |https://name/close/anotherpage/*/|
|https://name/view/124551/delete       |view    |20       |https://name/view/*/delete       |
|https://anothername/purchase/665412   |purchase|30       |https://anothername/purchase/*   |
+--------------------------------------+--------+---------+---------------------------------+

---

如果你想增加一点复杂性，假设用户 id 至少包含一个数字，也许还有其他字符，那么你可以这样做：

df.withColumn(
    "inferred_url", F.regexp_replace("url", r"\/[^\/]*\d[^\/]*", "\/\*")
).show(truncate=False)

---

不包括“a2z”的版本：

import re 

@F.udf
def infer_url(url):
    user_id_re = re.compile(r"\/([^\/]*\d[^\/]*)")
    user_id_list = user_id_re.findall(url)
    for user_id in user_id_list:
        if "a2z" not in user_id:
            url = url.replace(user_id, "*")
    return url

df.withColumn("user_id", infer_url(F.col("url"))).show(truncate=False)
+---------------------------------------+--------+---------+----------------------------------+
|url                                    |event   |timestamp|user_id                           |
+---------------------------------------+--------+---------+----------------------------------+
|https://name/view/124551/show          |view    |1        |https://name/view/*/show          |
|https://name/close/anotherpage/124551/ |close   |2        |https://name/close/anotherpage/*/ |
|https://name/close/anotherpage/987131/ |close   |3        |https://name/close/anotherpage/*/ |
|https://name/view/455633/show          |view    |6        |https://name/view/*/show          |
|https://name/close/anotherpage/891344/ |close   |10       |https://name/close/anotherpage/*/ |
|https://name/view/124551/delete        |view    |20       |https://name/view/*/delete        |
|https://anothername/purchase/665412    |purchase|30       |https://anothername/purchase/*    |
|https://anothername/purchase_a2z/665412|purchase|31       |https://anothername/purchase_a2z/*|
+---------------------------------------+--------+---------+----------------------------------+