使用Awk分析Nginx日志大于某个时间内非200状态的数据

Question

使用PHP写了一个同步ES和Redis的服务脚本,每分钟不定时会收到很多http请求,有点担心会失败,打算做一个补跑的脚本,目前是想通过awk分析nginx日志检索大于某个时间非200状态的数据进行从新发送,但是Awk不怎么会用，只是完成了一部分。。想要的结果：通过定时任务15分钟运行一次,然后查询大十五分钟前非200状态的数据,但是现在是用grep指定时间做不到这种效果(现在自己写的可以做到指定分钟内,感觉1分钟运行一次,数据太大会死循环)，研究了半天也没个结果,有没有大佬可否帮忙在个基础上改进一下Nginx日志格式"06/May/2019:03:03:00+0000""GET/HTTP/1.1""41""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""500""500""0.021""0.021""[20953311,21829304,22006993]""06/May/2019:03:07:11+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""500""500""0.058""0.058""[20953311,21829304,22006993]""06/May/2019:03:15:47+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""500""500""0.055""0.055""[20953311,21829304,22006993]""06/May/2019:03:16:44+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""200""200""0.051""0.051""[20953311,21829304,22006993]""06/May/2019:03:40:26+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""200""200""0.057""0.057""[22769461]""06/May/2019:03:41:11+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""500""500""0.058""0.058""[20953311,21829304,22006993]""06/May/2019:03:41:47+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""500""500""0.055""0.055""[20953311,21829304,22006993]""06/May/2019:03:42:44+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""200""200""0.051""0.051""[20953311,21829304,22006993]""06/May/2019:03:43:26+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""200""200""0.057""0.057""[22769461]"目前自己研究出来的AWK命令catfile.log|grep"06/May/2019:03:16"|awk-F"[\"\"]"'{if($18!=200){print$26}}'

Answer 1

awk-F"/"'{print$3$0}'a.txt|awk-F""'{print$1,$0}'|awk'$1>"2019:03:15:47"&&$16!="\"200\""{print$16}'
每次awk后都需要保存整行内容到下一个管道命令，你可以试试看。你的文档最后展示的结果是下面这个样子，你再处理一下就好了。
2019:03:41:112019:03:41:11+0000""POST"06/May/2019:03:41:11+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""500""500""0.058""0.058""[20953311,21829304,22006993]"
2019:03:41:472019:03:41:47+0000""POST"06/May/2019:03:41:47+0000""POST/HTTP/1.1""20""-""curl/7.29.0""10.1.7.31""-""127.0.0.1:9000""500""500""0.055""0.055""[20953311,21829304,22006993]"