如何忽略 jq 中损坏的 JSON 行？答案

【问题标题】：How to ignore broken JSON line in jq?如何忽略 jq 中损坏的 JSON 行？
【发布时间】：2021-09-26 09:45:09
【问题描述】：

使用jq 处理日志文件时，某些行可能会中断，因此jq 会抛出错误并停止处理。

例如完整的日志：

{"level":"debug","time":"2021-09-24T19:42:47.140+0800","message":"sent send binary to ws server1","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.305+0800","message":"sent send binary to ws server2","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.469+0800","message":"sent send binary to ws server3","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.499+0800","message":"sent send binary to ws server4","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.581+0800","message":"sent send binary to ws server5","pid":41491,"cid":"32likw","num":1,"count":5120}

jq 处理得很好：

< snippet1.json jq -C -r '.message'
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server3
sent send binary to ws server4
sent send binary to ws server5

损坏的（缺少第 3 行的最后一部分）：

{"level":"debug","time":"2021-09-24T19:42:47.140+0800","message":"sent send binary to ws server1","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.305+0800","message":"sent send binary to ws server2","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.469+0800","message":"sent send binary to ws server3","pi
{"level":"debug","time":"2021-09-24T19:42:47.499+0800","message":"sent send binary to ws server4","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.581+0800","message":"sent send binary to ws server5","pid":41491,"cid":"32likw","num":1,"count":5120}

jq 停在虚线处：

< snippet2.json jq -C -r '.message'
sent send binary to ws server1
sent send binary to ws server2
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 4, column 2

我希望jq可以忽略第三行继续，就像这样：

< snippet2.json jq -C -r '.message'
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server4
sent send binary to ws server5

我尝试使用another post中提到的-R，但对这种情况没有帮助。

< snippet2.json jq -C -R -r '.message'
jq: error (at <stdin>:1): Cannot index string with string "message"
jq: error (at <stdin>:2): Cannot index string with string "message"
jq: error (at <stdin>:3): Cannot index string with string "message"
jq: error (at <stdin>:4): Cannot index string with string "message"
jq: error (at <stdin>:5): Cannot index string with string "message"

您能否告诉我是否有任何解决方案/技能可以忽略/跳过/抑制此类错误并获得其余结果？

【问题讨论】：

标签： json logging error-handling command-line-interface jq

【解决方案1】：

要跳过你可以使用的虚线：

jq -Rr 'fromjson? | .message'

如果你想用它们做其他事情，你可以从这样的开始：

jq -R '. as $line | try fromjson catch $line'

其他选项见：

?：有没有办法让 jq 在输入文件中遇到错误后继续运行？ jq 可以处理损坏的 JSON 吗？

在jq FAQ.

【讨论】：

好的。让我编辑我的问题。感谢您的提醒。

【解决方案2】：

对 peak 的答案的更多解释。（积分达到顶峰）

解决方案 #1：

❯ cat bad.json | jq -r -R 'fromjson? | .message'
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server4
sent send binary to ws server5

解决方案 #2：

❯ cat bad.json | jq -r -R '. as $line | try fromjson catch $line | .message'
sent send binary to ws server1
sent send binary to ws server2
jq: error (at <stdin>:3): Cannot index string with string "message"
sent send binary to ws server4
sent send binary to ws server5

jq 还是输出错误，但是在stderr上，可以重定向：

❯ cat bad.json | jq -r -R '. as $line | try fromjson catch $line | .message' 2>/dev/null
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server4
sent send binary to ws server5

值得注意的是-R和-r可以一起使用。（感谢@peak！）

【讨论】：

可以同时使用-R和-r。