ELK-elkstack-使用消息队列
日志通过logstash收集到redis，之后从logstash从redis读取数据存入到ES
SRE实战 互联网时代守护先锋，助力企业售后服务体系运筹帷幄！一键直达领取阿里云限量特价优惠。
1. logstash使用redis测试
通过标准输入到redis中
logstash配置与启动
1 [yun@mini03 config]$ pwd 2 /app/logstash/config 3 [yun@mini03 config]$ cat redis_test.conf 4 input{ 5 stdin{} 6 } 7 8 filter{ 9 } 10 11 output{ 12 redis { 13 data_type => "list" 14 # 生产环境需要规划 15 db => 1 16 host => "mini03" 17 port => 6379 18 key => "redis_test" 19 } 20 } 21 22 ### 使用yun用户即可 23 [yun@mini03 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_test.conf 24 ………… 25 111 26 222 27 333 28 444 29 123555 30 1234 31 654321zhags
redis查看
1 [root@mini03 ~]# redis-cli -h mini03 -p 6379 2 mini03:6379> select 1 3 OK 4 mini03:6379[1]> KEYS * # 生产环境禁止使用该命令 5 1) "redis_test" 6 mini03:6379[1]> type redis_test 7 list 8 mini03:6379[1]> llen redis_test 9 (integer) 7 10 mini03:6379[1]> lindex redis_test -1 11 "{\"host\":\"mini03\",\"message\":\"654321zhags\",\"@timestamp\":\"2018-08-29T13:58:02.184Z\",\"@version\":\"1\"}"
2. httpd日志收集到redis中
logstash配置与启动
1 [yun@mini03 config]$ pwd 2 /app/logstash/config 3 [yun@mini03 config]$ cat redis_httpd_test.conf 4 input{ 5 file{ 6 path => ["/var/log/httpd/access_log"] 7 type => "httpd-access-log" 8 start_position => "beginning" 9 } 10 } 11 12 filter{ 13 } 14 15 output{ 16 redis { 17 data_type => "list" 18 # 生产环境需要规划 19 db => 1 20 host => "mini03" 21 port => 6379 22 key => "apache-access-log" 23 } 24 } 25 26 #### 使用root用户，涉及权限 27 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/redis_httpd_test.conf # 使用root用户
使用谷歌、火狐或者IE浏览器访问
redis查看
[root@mini03 ~]# redis-cli -h mini03 -p 6379
mini03:6379> select 1
OK
mini03:6379[1]> KEYS *
1) "apache-access-log"
2) "redis_test"
mini03:6379[1]> llen apache-access-log
(integer) 28
mini03:6379[1]> lindex apache-access-log -1
"{\"message\":\"10.0.0.1 - - [29/Aug/2018:22:08:30 +0800] \\\"GET /aaabbb/?aaa=bbb HTTP/1.1\\\" 404 205 \\\"-\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0\\\"\",\"type\":\"httpd-access-log\",\"path\":\"/var/log/httpd/access_log\",\"host\":\"mini03\",\"@timestamp\":\"2018-08-29T14:08:31.442Z\",\"@version\":\"1\"}"
3. logstash从redis读取数据标准输出
注意：该logstash在mini02上读取mini03上redis的数据
读取之后先使用grok进行过滤
之后进行标准输出【命令行输出】
logstash配置与启动
[yun@mini02 config]$ pwd
/app/logstash/config
[yun@mini02 config]$ cat redis_stdout.conf
input{
redis {
data_type => "list"
db => 1
host => "mini03"
port => 6379
key => "apache-access-log"
}
}
filter{
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
}
output{
stdout { codec => rubydebug }
}
###### 使用yun用户即可
[yun@mini02 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_stdout.conf
……………………
{
"request" => "/noindex/css/fonts/Bold/OpenSans-Bold.ttf",
"message" => "10.0.0.1 - - [30/Aug/2018:17:22:13 +0800] \"GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238 \"http://mini03/noindex/css/open-sans.css\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",
"@version" => "1",
"bytes" => "238",
"auth" => "-",
"referrer" => "\"http://mini03/noindex/css/open-sans.css\"",
"response" => "404",
"type" => "httpd-access-log",
"clientip" => "10.0.0.1",
"@timestamp" => 2018-08-30T09:22:13.950Z,
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"host" => "mini03",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",
"timestamp" => "30/Aug/2018:17:22:13 +0800",
"httpversion" => "1.1"
}
{
"request" => "/?refresh=1m&orgId=1",
"message" => "10.0.0.1 - - [30/Aug/2018:17:22:13 +0800] \"GET /?refresh=1m&orgId=1 HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",
"@version" => "1",
"bytes" => "4897",
"auth" => "-",
"referrer" => "\"-\"",
"response" => "403",
"type" => "httpd-access-log",
"clientip" => "10.0.0.1",
"@timestamp" => 2018-08-30T09:22:13.949Z,
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"host" => "mini03",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",
"timestamp" => "30/Aug/2018:17:22:13 +0800",
"httpversion" => "1.1"
}
……………………
4. elkstack-使用redis作为消息队列【汇总】
在mini03的logstash读取httpd的日志，并存储到redis
4.1. mini03的 logstash配置如下：
1 [yun@mini03 config]$ pwd 2 /app/logstash/config 3 [yun@mini03 config]$ cat redis_httpd_test.conf 4 input{ 5 file{ 6 path => ["/var/log/httpd/access_log"] 7 type => "httpd-access-log" 8 start_position => "beginning" 9 } 10 } 11 12 filter{ 13 } 14 15 output{ 16 redis { 17 data_type => "list" 18 # 生产环境需要规划 19 db => 1 20 host => "mini03" 21 port => 6379 22 key => "apache-access-log" 23 } 24 } 25 26 ######## 使用root用户，涉及权限 27 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/redis_httpd_test.conf 28 ………………
在mini02的logstash读取redis信息，并存储在ES
4.2. mini02的logstash配置
1 [yun@mini02 config]$ pwd 2 /app/logstash/config 3 [yun@mini02 config]$ cat redis_es.conf 4 input{ 5 redis { 6 data_type => "list" 7 db => 1 8 host => "mini03" 9 port => 6379 10 key => "apache-access-log" 11 } 12 } 13 14 filter{ 15 grok { 16 match => { "message" => "%{HTTPD_COMBINEDLOG}" } 17 } 18 } 19 20 output{ 21 # es有3台，随便指定一台即可 也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"] 22 elasticsearch { 23 hosts => ["mini01:9200", "mini02:9200", "mini03:9200"] 24 index => "httpd-access-log-%{+YYYY.MM.dd}" 25 } 26 } 27 28 ####### 使用yun用户即可 29 [yun@mini02 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_es.conf 30 ………………
4.3. 浏览器访问httpd
浏览器
1 # 可以通过谷歌、火狐、IE访问 2 http://mini03/ 3 http://mini03/indweg.html
Linux命令行访问
1 [yun@mini02 ~]$ ab -n40 -c 1 http://mini03/ 2 [yun@mini02 ~]$ ab -n40 -c 1 http://mini03/wet/bdhw/
4.4. 信息查看
elasticsearch-head查看
kibana查看
更多精彩
