unix awk 重复模式答案

【问题标题】：unix awk repeated patternunix awk 重复模式
【发布时间】：2016-02-03 14:06:46
【问题描述】：

我有以下变量。我想用模式“/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/”搜索

export str='16/02/02 11:29:22 INFO mortbay.log: State being saved: {"@class":"com.paypal.fpti.hadoop.copy.FPTICopyState","timestamp":0,"state":"Running","name":"com.paypal.fpti.hadoop.copy.FPTICopyState","id":"99c7cba7-d211-4845-97a1-c34168a91b22","subStates":{"com.paypal.fpti.hadoop.copy.CopyToLocalJob_fpti-raw-data-4_2016/02/02/10/":{"@class":"com.paypal.fpti.hadoop.copy.CopyToJobState","timestamp":0,"state":"Stopped","name":"com.paypal.fpti.hadoop.copy.CopyToJobState","id":"99034acb-cfad-41a0-89ed-e2731b1f82ec","subStates":null,"instanceState":"PostDone","window":"2016-02-02T10:00:00.000Z","datasetname":"fpti-raw-data-4","sourceDir":"/fpti/v2/hdfs_writer_4//2016/02/02/10/","localDir":"/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_4//2016/02/02/10//"},"com.paypal.fpti.hadoop.copy.CopyToLocalJob_fpti-raw-data_2016/02/02/10/":{"@class":"com.paypal.fpti.hadoop.copy.CopyToJobState","timestamp":0,"state":"Stopped","name":"com.paypal.fpti.hadoop.copy.CopyToJobState","id":"40325dec-0fe2-4025-8258-f896f957ddf0","subStates":null,"instanceState":"PostDone","window":"2016-02-02T10:00:00.000Z","datasetname":"fpti-raw-data","sourceDir":"/fpti/v2/hdfs_writer//2016/02/02/10/","localDir":"/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp//2016/02/02/10//"},"com.paypal.fpti.hadoop.copy.CopyToLocalJob_fpti-raw-data-1_2016/02/02/10/":{"@class":"com.paypal.fpti.hadoop.copy.CopyToJobState","timestamp":0,"state":"Stopped","name":"com.paypal.fpti.hadoop.copy.CopyToJobState","id":"5216f8c1-2cfa-4eac-a390-f4d2bcd6584f","subStates":{},"instanceState":"PostDone","window":"2016-02-02T10:00:00.000Z","datasetname":"fpti-raw-data-1","sourceDir":"/fpti/v2/hdfs_writer_1//2016/02/02/10/","localDir":"/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_1//2016/02/02/10//"},"com.paypal.fpti.hadoop.copy.CopyToLocalJob_fpti-raw-data-2_2016/02/02/10/":{"@class":"com.paypal.fpti.hadoop.copy.CopyToJobState","timestamp":0,"state":"Stopped","name":"com.paypal.fpti.hadoop.copy.CopyToJobState","id":"5fcd0b6e-3df9-4f82-a76f-bc8ff1493623","subStates":{},"instanceState":"PostDone","window":"2016-02-02T10:00:00.000Z","datasetname":"fpti-raw-data-2","sourceDir":"/fpti/v2/hdfs_writer_2//2016/02/02/10/","localDir":"/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_2//2016/02/02/10//"},"com.paypal.fpti.hadoop.copy.CopyToLocalJob_fpti-raw-data-3_2016/02/02/10/":{"@class":"com.paypal.fpti.hadoop.copy.CopyToJobState","timestamp":0,"state":"Stopped","name":"com.paypal.fpti.hadoop.copy.CopyToJobState","id":"6ec9223a-fcf0-447a-b9ae-2020e3232f6d","subStates":{},"instanceState":"PostDone","window":"2016-02-02T10:00:00.000Z","datasetname":"fpti-raw-data-3","sourceDir":"/fpti/v2/hdfs_writer_3//2016/02/02/10/","localDir":"/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_3//2016/02/02/10//"},"com.paypal.fpti.hadoop.copy.CopyToLocalJob_fpti-raw-data-5_2016/02/02/10/":{"@class":"com.paypal.fpti.hadoop.copy.CopyToJobState","timestamp":0,"state":"Stopped","name":"com.paypal.fpti.hadoop.copy.CopyToJobState","id":"d123742c-8a55-4e25-bfa0-0a97f6ed25d7","subStates":{},"instanceState":"PostDone","window":"2016-02-02T10:00:00.000Z","datasetname":"fpti-raw-data-5","sourceDir":"/fpti/v2/hdfs_writer_5//2016/02/02/10/","localDir":"/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_5//2016/02/02/10//"}},"copystate":"CopyToLocalDone","start":"2016-02-02T11:21:24.678Z","end":null,"window":"2016-02-02T10:00:00.000Z","retryCount":0}'

我尝试如下所示，它只给出了第一次出现

[ggangadharan@phxbastion2 ~]$ echo $str | awk '{match($0, "/x/home[/,a-z,0-9,_]+*", a)}END{print a[0]}'
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_4//2016/02/02/10//

但我想要如下所示的输出。

/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_4//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_1//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_2//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_3//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_5//2016/02/02/10//

有人可以帮我在这种情况下如何使用 awk 吗？

提前致谢

【问题讨论】：

Awk 处理每一行，你必须做一些重要的拆分来解析行并进行多次搜索。您可以使用 egrep 来解决这个特定问题，但更一般地，您可以使用其他脚本工具来解析 JSON。为此，您也可以echo $str | egrep -o localDir[^,]* | awk '{print $2}'

标签： unix awk

【解决方案1】：

我不确定如何在 awk 中破解它，但您可以在这里安全地使用 egrep：

$ echo $str | egrep -o /x/home[/,a-z,0-9,_]+*
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_4//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_1//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_2//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_3//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_5//2016/02/02/10//

【讨论】：

谢谢弗迪。它的工作原理一般来说，如果我们 grep 它给出了整行，对吗？你能解释一下这个功能吗，因为我是 unix 脚本的新手
如果您在文件中搜索带有 \n 终止行的模式并返回这些模式，通常您会使用 grep -E（或 egrep，这是相同的）。为了 -E 的缘故，可以使用正则表达式模式，并将返回它在当前行中的每一次出现，就像我们在这里所做的那样。我已经添加了 -o （这意味着 --only-matching）来获得正则表达式的匹配结果。如果你忽略它，你的日志条目的整行和你的 json 将被显示；正如我所说，通常您使用 grep 来查找与特定模式匹配的行。 HTH
感谢 ferdy 的解释。

【解决方案2】：

在 AWK 中使用“显着拆分”：

$ awk -v RS="\"" '/\/x\/home\/pp_dt_fpti_batch\/stampy_copy_orchestration\//' <<< "$str"

给了

/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_4//2016/02/02/10// /x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp//2016/02/02/10// /x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_1//2016/02/02/10// /x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_2//2016/02/02/10// /x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_3//2016/02/02/10// /x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_5//2016/02/02/10//

您为搜索模式指定了/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/，所以我使用了它。如果您想要不同的东西，请使用不同的东西。

这通过引用 " 将输入分隔为记录（将 RS 设置为 "，在 shell 中转义）。打印任何匹配正则表达式的记录。输入是从 shell 以字符串$str 给出的。也许这更具可读性：

$ awk -v RS='"' '/regexp/' <<< "$str"

【讨论】：

好东西，虽然我建议为此使用 grep，这对于经验不足的 awk 用户来说更具可读性。 ;)

【解决方案3】：

这里有两种使用 JSON 感知命令行工具的方法，这里是 jq。

在这两种情况下，我们都假设感兴趣的字符串嵌入在 $str 中包含的 JSON 对象

(1) 在下面，我们简单地打印 JSON 对象并 grep 感兴趣的字符串，以防它出现在令人惊讶的位置。可以根据需要轻松地对结果进行进一步修整（例如使用 sed）：

$ sed 's/^[^{]*//' <<< "$str" | jq '.[]' | fgrep /x/home/pp_dt_fpti_batch/stampy_copy_orchestration/
"localDir": "/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_4//2016/02/02/10//"
"localDir": "/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp//2016/02/02/10//"
"localDir": "/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_1//2016/02/02/10//"
"localDir": "/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_2//2016/02/02/10//"
"localDir": "/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_3//2016/02/02/10//"
"localDir": "/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_5//2016/02/02/10//"

(2) 如果我们只对如果它作为与键“localDir”对应的值出现在对象中，则匹配：

sed 's/^[^{]*//' <<< "$str" |
  jq -r '..
      | select(.localDir?)
      | .localDir
      | select(test("/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/"))'

/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_4//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_1//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_2//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_3//2016/02/02/10//
/x/home/pp_dt_fpti_batch/stampy_copy_orchestration/tmp_5//2016/02/02/10//

【讨论】：