利用 sed 和 awk 做简单日志分析

发表于 更新于

环境

本实验是在 macOS 下做的，Linux 下可能会稍有不同（比如 gzcat）。

日志文件

1
gzcat data.gz | head -n 40 # data.gz 是日志文件

系统输出：

May 13 00:01:58 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12513]): Could not find uid associated with service: 0: Undefined error: 0 501
May 13 00:01:58 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12513]): Service exited with abnormal code: 78
May 13 00:02:12 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.xpc.launchd.domain.pid.mdmclient.12523): Failed to bootstrap path: path = /usr/libexec/mdmclient, error = 108: Invalid path
May 13 00:04:20 BBAOMACBOOKAIR2 syslogd[113]: ASL Sender Statistics
May 13 00:05:58 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12535]): Could not find uid associated with service: 0: Undefined error: 0 501
May 13 00:05:58 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12535]): Service exited with abnormal code: 78
May 13 00:09:58 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12536]): Could not find uid associated with service: 0: Undefined error: 0 501
May 13 00:09:58 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12536]): Service exited with abnormal code: 78
May 13 00:17:59 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12555]): Could not find uid associated with service: 0: Undefined error: 0 501
May 13 00:17:59 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12555]): Service exited with abnormal code: 78
May 13 00:17:59 BBAOMACBOOKAIR2 syslogd[113]: ASL Sender Statistics
May 13 00:19:59 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12556]): Could not find uid associated with service: 0: Undefined error: 0 501
May 13 00:19:59 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12556]): Service exited with abnormal code: 78
May 13 00:21:59 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12560]): Could not find uid associated with service: 0: Undefined error: 0 501
May 13 00:21:59 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12560]): Service exited with abnormal code: 78
May 13 00:22:18 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.xpc.launchd.domain.user.914945058): Service “com.apple.xpc.launchd.unmanaged.loginwindow.594” tried to register for endpoint “com.apple.tsm.uiserver” already registered by owner: com.apple.TextInputMenuAgent
May 13 00:22:49 — last message repeated 1 time —
May 13 00:23:50 BBAOMACBOOKAIR2 timed[158]: settimeofday({0x5ebacd96,0x52ddf}) == 0
May 13 00:28:05 BBAOMACBOOKAIR2 syslogd[113]: ASL Sender Statistics
May 13 00:28:07 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.ScreenSaver.Computer-Name[12564]): Service exited due to SIGKILL | sent by Computer Name[12564]
May 13 00:28:17 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:17 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:18 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:18 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:19 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:19 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:20 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:20 BBAOMACBOOKAIR2 VTDecoderXPCService[960]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
May 13 00:28:26 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.preference.displays.MirrorDisplays): Service only ran for 9 seconds. Pushing respawn out by 1 seconds.
May 13 00:28:31 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.preference.displays.MirrorDisplays): Service only ran for 4 seconds. Pushing respawn out by 6 seconds.
May 13 00:29:49 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12610]): Could not find uid associated with service: 0: Undefined error: 0 501
May 13 00:29:49 BBAOMACBOOKAIR2 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[12610]): Service exited with abnormal code: 78
May 13 00:30:00 BBAOMACBOOKAIR2 syslogd[113]: Configuration Notice:
ASL Module “com.apple.cdscheduler” claims selected messages.
Those messages may not appear in standard system log files or in the ASL database.
May 13 00:30:00 BBAOMACBOOKAIR2 syslogd[113]: Configuration Notice:
ASL Module “com.apple.install” claims selected messages.
Those messages may not appear in standard system log files or in the ASL database.
May 13 00:30:00 BBAOMACBOOKAIR2 syslogd[113]: Configuration Notice:
ASL Module “com.apple.callhistory.asl.conf” claims selected messages.

以上日志贴出来可能看的不够清晰，其实就几点：

  1. 正常的日志都是日期时间开头，像“May 13 00:30:00”这样
  2. 不是这个开头的日志行（基本都是 \t 跳格开头），其实是上一行的继续（日志分析时要将其和上一行合并）
  3. 有“— last message repeated 1 time —”的日志其实内容跟上一行是完全一样的（开头的日期时间除外）

要求

分析系统日志文件（data.gz）从中得到关键信息，用 Json 的格式 POST 上传至服务器 https://foo.com/bar )，key的名称如下（在括号里）：

  1. 设备名称: (deviceName)
  2. 错误的进程号码: (processId)
  3. 进程/服务名称: (processName)
  4. 错误的原因（描述）(description)
  5. 发生的时间（小时级），例如0100-0200，0300-0400, (timeWindow)
  6. 在小时级别内发生的次数 (numberOfOccurrence)

分析

这里考察的点应该就是多行日志的处理（主要是合并）问题，这个可以借助 sed 可以搞定，但还有一个重复日志的问题，也就是当日志行里有“— last message repeated 1 time —”时的处理问题，没想好怎样完美的解决。

最终方案代码

1
2
3
gzcat data.gz | head -n 40 | sed -e '1h;2,$H;$!d;g;s/\n\t/ /g' | awk -f log_ana.awk> data.json
# 上面的 head -n 40 主要是为了演示，只取头 40 行数据，真正的环境是取的整个数据文件的所有数据
curl -X POST -H "Content-Type: application/json" -d @data.json https://foo.com/bar

其中 log_ana.awk 代码如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
#! /usr/bin/awk -f

BEGIN {
    pre_logline = ""
}

{
    processId = 0
    deviceName = ""
    processName = ""
    description = ""
    timeWindow = ""
}

/--- last message peated 1 time ---/ {
    sub(/--- last message peated 1 time ---/, pre_logline)
}

{
    delete a
    split($3, a, ":")
    if (a[1] == "00")
        timeWindow = "0000-0100"
    else if (a[1] == "01")
        timeWindow = "0100-0200"
    else if (a[1] == "02")
        timeWindow = "0220-0300"
    else if (a[1] == "03")
        timeWindow = "0300-0400"
    else if (a[1] == "04")
        timeWindow = "0400-0500"
    else if (a[1] == "05")
        timeWindow = "0500-0600"
    else if (a[1] == "06")
        timeWindow = "0600-0700"
    else if (a[1] == "07")
        timeWindow = "0700-0800"
    else if (a[1] == "08")
        timeWindow = "0800-0900"
    else if (a[1] == "09")
        timeWindow = "0900-1000"
    else if (a[1] == "10")
        timeWindow = "1000-1100"
    else if (a[1] == "11")
        timeWindow = "1100-1200"
    else if (a[1] == "12")
        timeWindow = "1200-1300"
    else if (a[1] == "13")
        timeWindow = "1300-1400"
    else if (a[1] == "14")
        timeWindow = "1400-1500"
    else if (a[1] == "15")
        timeWindow = "1500-1600"
    else if (a[1] == "16")
        timeWindow = "1600-1700"
    else if (a[1] == "17")
        timeWindow = "1700-1800"
    else if (a[1] == "18")
        timeWindow = "1800-1900"
    else if (a[1] == "19")
        timeWindow = "1900-2000"
    else if (a[1] == "20")
        timeWindow = "2000-2100"
    else if (a[1] == "21")
        timeWindow = "2100-2200"
    else if (a[1] == "22")
        timeWindow = "2200-2300"
    else if (a[1] == "23")
        timeWindow = "2300-2400"
    else {
    }
    for (i = 4; i<=NF; i++) {
        if (i == 4) {
            pre_logline = $i
        } else {
            pre_logline = pre_logline" "$i
        }
    }
    if ((pre_logline, timeWindow) in countA) {
        countA[pre_logline, timeWindow] = countA[pre_logline, timeWindow] + 1
    } else {
        countA[pre_logline, timeWindow] = 1
    }
}

END {
    print "["
    n = 1
    sum = 0
    for (k in countA) {
        processId = 0
        deviceName = ""
        processName = ""
        description = ""
        timeWindow = ""
        c = length(countA)
        delete a
        split(k, a, SUBSEP)
	numberOfOccurrence = countA[a[1], a[2]]
        timeWindow = a[2]
        delete aa
        split(a[1], aa, " ")
        deviceName = aa[1]
        delete aaa
        split(aa[2], aaa, "]")
        delete aaaa
        split(aaa[1], aaaa, "[")
        processName = aaaa[1]
        processId = aaaa[2]
        for (i = 3; i<=length(aa); i++) {
            gsub(/"/, "\\\"", aa[i])
            if (i == 3) {
                description = aa[i]
            } else {
                description = description" "aa[i]
            }
        }
    	printf "  {\n    \"deviceName\":\"%s\", \n    \"processId\":%d, \n    \"processName\":\"%s\", \n    \"description\":\"%s\", \n    \"timeWindow\":\"%s\", \n    \"numberOfOccurrence\":%d \n  }", deviceName, processId, processName, description, timeWindow, numberOfOccurrence
        sum = sum + numberOfOccurrence
        if (n < c)
            print ", "
        else
            print ""
        n++
    }
    print "]"
}

data.json 文件的内容可以这样看：

1
cat data.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
[
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12560]): Could not find uid associated with service: 0: Undefined error: 0 501",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12535]): Could not find uid associated with service: 0: Undefined error: 0 501",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"---",
    "processId":0,
    "processName":"last",
    "description":"message repeated 1 time ---",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12513]): Service exited with abnormal code: 78",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.preference.displays.MirrorDisplays): Service only ran for 9 seconds. Pushing respawn out by 1 seconds.",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":158,
    "processName":"timed",
    "description":"settimeofday({0x5ebacd96,0x52ddf}) == 0",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.xpc.launchd.domain.user.914945058): Service \"com.apple.xpc.launchd.unmanaged.loginwindow.594\" tried to register for endpoint \"com.apple.tsm.uiserver\" already registered by owner: com.apple.TextInputMenuAgent",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":113,
    "processName":"syslogd",
    "description":"Configuration Notice: ASL Module \"com.apple.install\" claims selected messages. Those messages may not appear in standard system log files or in the ASL database.",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12610]): Could not find uid associated with service: 0: Undefined error: 0 501",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12610]): Service exited with abnormal code: 78",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.preference.displays.MirrorDisplays): Service only ran for 4 seconds. Pushing respawn out by 6 seconds.",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12535]): Service exited with abnormal code: 78",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.xpc.launchd.domain.pid.mdmclient.12523): Failed to bootstrap path: path = /usr/libexec/mdmclient, error = 108: Invalid path",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":960,
    "processName":"VTDecoderXPCService",
    "description":"DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":4
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12555]): Could not find uid associated with service: 0: Undefined error: 0 501",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":960,
    "processName":"VTDecoderXPCService",
    "description":"DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":4
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12536]): Service exited with abnormal code: 78",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12513]): Could not find uid associated with service: 0: Undefined error: 0 501",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12536]): Could not find uid associated with service: 0: Undefined error: 0 501",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.ScreenSaver.Computer-Name[12564]): Service exited due to SIGKILL | sent by Computer Name[12564]",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12555]): Service exited with abnormal code: 78",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":113,
    "processName":"syslogd",
    "description":"Configuration Notice: ASL Module \"com.apple.callhistory.asl.conf\" claims selected messages.",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12560]): Service exited with abnormal code: 78",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":113,
    "processName":"syslogd",
    "description":"ASL Sender Statistics",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":3
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":113,
    "processName":"syslogd",
    "description":"Configuration Notice: ASL Module \"com.apple.cdscheduler\" claims selected messages. Those messages may not appear in standard system log files or in the ASL database.",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12556]): Could not find uid associated with service: 0: Undefined error: 0 501",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  },
  {
    "deviceName":"BBAOMACBOOKAIR2",
    "processId":1,
    "processName":"com.apple.xpc.launchd",
    "description":"(com.apple.mdworker.bundles[12556]): Service exited with abnormal code: 78",
    "timeWindow":"0000-0100",
    "numberOfOccurrence":1
  }
]

这里的大概解释下：

多行日志的合并问题，这里是用 sed 命令解决的，awk 程序主要是处理了“— last message peated 1 time —”的问题以及做日志分析（从日志里抓取关键信息并做统计且生成上报数据文件）。其实这样不是特完美，毕竟日志文件循环了两遍，理论上来讲最好过一遍就处理完毕的，但是我在 awk 里实在是没想好怎样同时处理多行日志合并以及“— last message peated 1 time —”的问题。这里请各位大佬教我。