fluentdとmongodbの連携

fluentd

環境†

• CentOS6.7
• nginx 1.0.15
• mongod 2.6.11
• td-agent 2.3.0

fluentd側設定†

nginxの場合†

<source>
type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/access.pos
tag nginx.access
#format nginx
format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwarder>[^\"]*)")?/
time_format %d/%b/%Y:%H:%M:%S %z
</source>
<match nginx.access>
type copy

<store>
 type mongo
 host localhost
 port 27017
 database fluentd
 collection test
 capped
 capped_size 1024m
 flush_interval 10s
</store>
</match>
※mongodbのデフォルト待受ポートが27017
※特にDBなどは作成しなくてもOK

確認手順†

# mongo
> show dbs
admin    (empty)
fluentd  2.077GB
local    0.078GB
test     (empty)
> use fluentd
switched to db fluentd
> show collections
system.indexes
test
> db.test.find()
{ "_id" : ObjectId("56b98275e138231169000007"), "remote" : "192.168.33.1", "host" : "-", "user" : "-", "method" : "GET", "path" : "/jopdsfa", "code" : "404", "size" : "571", "referer" : "-", "agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36", "forwarder" : "-", "time" : ISODate("2016-02-09T06:08:53Z") }