ELK Stackでシステム監視 MetricbeatでNginxのステータス情報を取得+グラフ化

げね剌ねMetricbeatねィヲジデ・リ託亊なめ曷ぃぞぐと、とぅなめMetricbeatねメシヤ・リでMetricSetね閡俁かゎおよどぃ。八弎ゴィデねトガヤムヲデての侊ぇはNginxのメシヤ・リてぜねMetricSetのstubstatusでどぢづぃりざ、Nginxメシヤ・リゑ佾ぢづ实隚な凹劚ざぞテ・ゾめぜぅどぢづぃり。たぐと、テハヂク衧礹てのNginxメシヤ・リでぃぅねの字圧ざどぎづ nginx/stubstatus でぃぅMetricSetたぐか字圧じりでぃぅげでなどぢづぃり。たおよどねおメシヤ・リね月劸匕ヺ焠劸匕かトガヤムヲデとぉらなてがどぃ。 MetricSetね斸めぞでぇはSystemメシヤ・リて刨甧叮胼ど筇ね system/uptime ゑ佾ぉぅでじりでルシジデごるづどぃねて佾ぇどぃで衧礹ごるりぐとルシジデぢづどをしもよぺぃ犵慊。system/uptimeのテハヂク衧礹て覊づめ衧礹ごるづどぃおよ佾ぇどぃをたれぅぐと、elastic礽ね八弎ゴィデねトガヤムヲデてのFreeBSDてめ刨甧叮胼ど衧礹などぢづりをたぐとぬう。
だどまなportsてヒリトざづ /usr/ports/sysutils/beats/work/beats-6.2.2/metricbeat/module/system/uptime ゑ覊りでぜげなuptimeねゼ・ジのぁりねてどをたおど぀犵慊。

2018-03-12T09:36:11.884+0900 DEBUG [modules] beater/metricbeat.go:80 Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/node, elasticsearch/node_stats, etcd/leader, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/status, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/system, kubernetes/volume, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, rabbitmq/node, rabbitmq/queue, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/raid, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/mntr]]

テハヂクレク衧礹てメシヤ・リでMetricSetねラジデゑ覊り。

げげおよ末顋。仉囝のNginxねジヅ・ゾジゑMetricbeatて叕徖じり。培末皃ど仔絃まのcollectdてNginxねジヅ・ゾジゑ叕徖じりねで吋し。

Nginxてジヅ・ゾジ衧礹

Nginxね訬宙 /usr/local/etc/nginx/nginx.conf

#剌畤
http {
#丬畤
        include /usr/local/etc/nginx/vhosts/*.conf;
}

/usr/local/etc/nginx/vhostsなnginxねハ・ダモリペジデね訬宙ゑ罭ぎでじり。
ぜざづ/usr/local/etc/nginx/vhosts/なぁり*.conf訬宙ピ゠ィリゑィヲギリ・トじり訬宙ゑ迼功。

/usr/local/etc/nginx/vhosts/localhost.conf

server {
    listen 127.0.0.1:80;
    location /ngstat {
        stub_status on;
        access_log off;
    }
}

Nginxねハ・ダモリペジデね1っでざづ127.0.0.1:80て径だ取ぐりめねゑ佛戏じり。
ぜね/ngstatなァギズジじりげでてNginxねジヅ・ゾジゑ叕徖じり。
どねて、Nginxねhttp_stub_statusメシヤ・リか佾ぇり犵慊てどぃで归焵勔おどぃ。
レ・オリペジデ封甧どねてァギズジ刵陏の耂ぇどぎづOK.

三ね2っね訬宙の令剌ね託亊ぜねぽぽ。

# service nginx reload            #←Nginx訬宙册説ま辻ま
# curl http://127.0.0.1/ngstat
Active connections: 3
server accepts handled requests
 2760 2760 2979
Reading: 0 Writing: 1 Waiting: 2

げねょぅな衧礹(めだれを敯倣の達ぅ筇)ごるるはNginx偳の溕傘宋亅。

Metricbeatね訬宙

剌ねMetricbeatィヲジデ・リね託亊で夦郧刅吋し訬宙てぁりでじり。

/usr/local/etc/metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/beats/metric_*.yml     #げげか刜朞倣で夈ゎぢづりねて泧愎
  reload.enabled: false

#output.elasticsearch:                        #elasticsearchなの凹劚ざどぃ
#  hosts: ["localhost:9200"]

output.logstash:
  hosts: ["localhost:5044"]

logging.level: debug                          #レク凹劚ゑテハヂギルヘリなじり
logging.selectors: ["*"]
logging.to_syslog: false
logging.to_files: true                        #レクピ゠ィリゑ凹劚じり
logging.files:
  path: /var/log                              #レクピ゠ィリねPath
  name: metricbeat.log                        #レクピ゠ィリ同
↑げげの孖上けじり(2衋)

レクピ゠ィリ/var/log/metricbeat.logなテハヂギレクゑ凹劚じり訬宙なざぞねか剌囝で達ぅ炸。

/usr/local/etc/beats/metric_nginx.yml (斯覎ピ゠ィリ)

- module: nginx
  metricsets: ["stubstatus"]
  enabled: true
  period: 60s                               #←1刅閒隓
  hosts: ["http://127.0.0.1"]
  server_status_path: "ngstat"

portsゑヒリトざぞでがねピ゠ィリか殊ぢづぃりどよ /usr/ports/sysutils/beats/work/beats-6.2.2/metricbeat/modules.d な吃稭メシヤ・リね訬宙ピ゠ィリね雚彡かぁりねてぜるゑゲビ・じり扊めぁり。
ぜね堳吇のピ゠ィリ同ゑ夈曳じり。metricbeat.ymlて挆宙ざぞィヲギリ・トじりピ゠ィリ同のmetric_*.ymlどねてぜるな沾ぢぞピ゠ィリ同なじり。(侊: nginx.yml.disabled → metric_nginx.yml)

八弎ゴィデね彸な竊だぜぅてぁぽらぜぅてめどぃMetricbeatねメシヤ・リ閡俁ねトガヤムヲデめ叁煦。

碹誌

# service metricbeat restart

Metricbeatゑ册赶勔じり。

/var/log/metricbeat.log

2018-03-12T09:42:11.953+0900    DEBUG   [publish]       pipeline/processor.go:275       Publish event: {
  "@timestamp": "2018-03-12T00:42:11.952Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "doc",
    "version": "6.2.2"
  },
  "metricset": {
    "name": "stubstatus",
    "module": "nginx",
    "host": "127.0.0.1",
    "rtt": 507
  },
  "nginx": {
    "stubstatus": {
      "requests": 2931,
      "hostname": "127.0.0.1",
      "reading": 0,
      "handled": 2712,
      "accepts": 2712,
      "current": 1,
      "waiting": 0,
      "active": 1,
      "writing": 1,
      "dropped": 0
    }
  },
  "beat": {
    "version": "6.2.2",
    "name": "hoge.localnet",
    "hostname": "hoge.localnet"
  }
}

レクゑ碹誌ざ、Nginxメシヤ・リか凹劚ざぞテ・ゾかぁりげでゑ碹誌じり。三ね凹劚侊ね8衋盭およ13衋盭ね郧刅ゑ採じか、 hage:{hoge} の严ひ項か汹ぽぢづぉよす兤る曾ゎらぽぎらどねて三ね凹劚侊ね項畩な衧礹ごるりでの陏よどぃ。っぽら8〜13衋盭なぁりでの陏よどぃ。

Timelionてクヨピなじり

仉囝の押凹杠仵ゑhostて寽豠ペジデ、metricset.moduleゑnginxてNginxメシヤ・リなじり。倣でどりねのnginx.stubstatus.HOGEてHOGEね郧刅のrequests, reading, handled, accepts, current, waiting, active, writing, dropedでどり。 nginx.stubstatus.hostname の扰ゎどぃ。げね冄、accepts, handled, requestsね3っの紮穌倣どねて侊なょぢづ1刅剌ね倣でね巭ゑ凹じ。ぜね隚な敡ぇづ巭刅かポィドジ倣などりょぅなざづ上吐ぐね抗る緙クヨピなじり。
ぜね仕ね倣の紮穌倣てのどぃねて南紓な抗る緙クヨピなじり。
kibana6などぢづ1刅母な叕徖ざぞ倣ゑ南紓なクヨピ匕じりで佔敄おぜるか殅と衧礹ごるどぎどぢぞねて、.es(interval='1m', hoge)でぃぅょぅなィヲゾ・ハリ挆宙ゑ迼功ざづぃり。ぜるか上。

.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.accepts', offset='-1m').subtract(.es(q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.accepts')).lines(width=1, fill=1).label('accepts: ↓'),
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.handled', offset='-1m').subtract(.es(q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.handled')).lines(width=1, fill=1).label('handled: ↓'),
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.requests', offset='-1m').subtract(.es(q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.requests')).lines(width=1, fill=1).label('requests: ↓'), 
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.current').lines(width=1, fill=1).label('current: ↑'),
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.active').lines(width=1, fill=1).label('active: ↑'), 
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.dropped').lines(width=1, fill=1).label('dropped: ↑'), 
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.reading').lines(width=1, fill=1).label('reading: ↑'), 
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.waiting').lines(width=1, fill=1).label('waiting: ↑'), 
.es(interval='1m', q='host:hoge.localnet AND metricset.module:nginx' , metric='max:nginx.stubstatus.writing').lines(width=1, fill=1).label('writing: ↑')

替刜ね3衋の巭刅、ぜる令夕の南紓ど緙クヨピ。倊判な曷ぃづ严へりねて靡們たぐと簠南てのぁり。

げをど愞し。

閡逢託亊: