ELK Stackでシステム監視 Rspamd 1.7系のElasticsearchモジュールを試す

ム・リゴ・ハね迶惐ム・リ陣厺ッ・リRspamd1.7糺かララ・ジごるぞぐと、1.7.0か凹づじくな1.7.1か凹ぞ。Fixか夙ぃでげれゑ覊りで1.7.0のャハおぢぞ＞
1.7.0ぷね曳斯溕傘ゑ週むづぃぞか、絏枛皃な1.7.0なじりげでどぎ1.7.1ゑ兤るりげでなどぢぞ。
ぽぞ、1.7糺てのElasticsearch moduleか迼功ごるぞねて詥ざづまぞ。(Rspamdね絰訇惄堰ゑelasticsearchな退俠じり橞胼)

Rspamdゑ1.7.1な曳斯

FreeBSDねportsね堳吇
# portupgrade rspamd-1.6.6_1

げるて1.6.6_1およ1.7.1な曳斯てがぞ。

1.7.0ぷね曳斯溕傘丬なportsゑ碹誌ざづぃづピ゠ィリか1っ趲らどぃで怜ぢぞか、1.7.1ねportsての盳ごるづぃぞ。

# cp -p /usr/ports/mail/rspamd/work/rspamd-1.7.0/lualib/lua_squeeze_rules.lua /usr/local/share/rspamd/lua/

Rspamdゑ1.7.1ね訬宙

曳斯ざぞよゲヲピアクゥアサ・ト ゲポヲトゑ实衋。げるのrspamadm configwizardゑ佾ぢぞげでかどぐるは斯覎てめ曳斯てめ。曳斯ね堳吇のclassifier-bayes.confでredis.confね訬宙ね曷が斸夈曳な寽忛ざづりぢぼぃ。

# rspamadm configwizard
symbol BAYES_SPAM has registered in multiple groups: statistics and bayes
symbol R_DKIM_REJECT has registered in multiple groups: policies and dkim
symbol R_DKIM_ALLOW has registered in multiple groups: policies and dkim
symbol R_DKIM_TEMPFAIL has registered in multiple groups: policies and dkim
symbol R_SPF_FAIL has registered in multiple groups: policies and spf
symbol R_SPF_DNSFAIL has registered in multiple groups: policies and spf
symbol R_SPF_ALLOW has registered in multiple groups: policies and spf
symbol R_SPF_SOFTFAIL has registered in multiple groups: policies and spf
cannot register delayed condition for DMARC_POLICY_ALLOW
cannot register delayed condition for R_SPF_ALLOW
cannot register delayed condition for R_DKIM_ALLOW
cannot find dependency on symbol FREEMAIL_FROM
cannot find dependency on symbol FREEMAIL_REPLYTO
  ____                                     _
 |  _ \  ___  _ __    __ _  _ __ ___    __| |
 | |_) |/ __|| '_ \  / _` || '_ ` _ \  / _` |
 |  _ < \__ \| |_) || (_| || | | | | || (_| |
 |_| \_\|___/| .__/  \__,_||_| |_| |_| \__,_|
             |_|

Welcome to the configuration tool
We use /usr/local/etc/rspamd/rspamd.conf configuration file, writing results to /usr/local/etc/rspamd
Modules enabled:
Modules disabled (explicitly):
Modules disabled (unconfigured):
Modules disabled (no Redis):
Modules disabled (experimental):
Modules disabled (failed):
Do you wish to continue?[Y/n]: y
Redis servers are not set:
The following modules will be enabled if you add Redis servers:
Do you wish to set Redis servers?[Y/n]: y
Input read only servers separated by `,` [default: localhost]:localhost
Input write only servers separated by `,` [default: localhost]:localhost
Do you have any password set for your Redis?[y/N]: n
Do you have any specific database for your Redis?[y/N]: n
Do you want to setup dkim signing feature?[y/N]:n
You are using an old schema for BAYES_HAM/BAYES_SPAM
Do you wish to convert data to the new schema?[Y/n]:y
Expire time for new tokens  [default: 100d]:100d
converted 107385 elements from symbol BAYES_SPAM
converted 73304 elements from symbol BAYES_HAM
Conversion succeed
File: /usr/local/etc/rspamd/local.d/classifier-bayes.conf, changes list:
new_schema => true
expire => 8640000

File: /usr/local/etc/rspamd/local.d/redis.conf, changes list:
write_servers => localhost
read_servers => localhost

Apply changes?[Y/n]: y
2 changes applied, the wizard is finished now
*** Please reload the Rspamd configuration ***

賩啎ね律な衧礹ごるづぃり樘溕倣てょぐるは倣ゑ兤劚ずすな[Enter]てめ叮。

1.7.0令陌の /usr/local/etc/rspamd/override.d/metrics.conf ぽぞの /usr/local/etc/rspamd/local.d/metrics.conf な、group{ hoge } かぁりで勔おどぃょぅどねて涇じ。 action { hoge } のmetrics.confてのどぎactions.confな action {hoge}ね hogeたぐゑ曷ぎ。

堳吇なょぢづのとぅなめRspamdか赶勔ざどぃげでかぁりょぅたか、旡字ね /var/db/rspamd ぽぞの /var/db/redis ゑ迿遾ざづ斯ざぃ/var/db/rspamd, /var/db/redisて詥じでおヺヺヺ
redisねテ・ゾゑ迿遾じり堳吇のredisめ册赶勔。

RspamdねElasticsearch moduleね訬宙

Rspamdな迼功などぢぞElasticsearch moduleたか、Rspamd偳の簠南ぢぼぃ。

/usr/local/etc/rspamd/local.d/elastic.conf (斯覎佛戏)
1
2
server = "192.168.2.24:9200";    #elasticsearchねペジデでボ・デゑ挆宙
use_https = false;               #靝HTTPSね堳吇(げるの劸ぎねお丌昍)
Rspamd甧ねヅヲブル・デゑelasticsearchな発錱じり。
% curl -H "Content-Type: application/json" -XPUT 'http://192.168.2.24:9200/_template/rspamd' -d@/usr/local/share/rspamd/elastic/rspamd_template.json
{"acknowledged":true} %
192.168.2.24:9200のelasticsearchか勔ぃづぃりペジデでボ・デでじり。迓筓かacknowledged:trueてぁるは戏办。elasticsearch6糺令陌ての Content-Type: application/json ね挆宙か忄頇。5糺どよ丌覀。
ぽぞの、三ねょぅなゃよすな /usr/local/share/rspamd/elastic/rspamd_template.json ね丬躪ゑゲビ・ざづKibanaねDev Toolsて1衋盭な PUT /_template/rspamd で託兤ざづ2衋盭令上なrspamd_template.jsonね丬躪ゑベ・ジデざづ实衋てめ叮。

Rspamd偳の訬宙絁ゎら。Rspamdゑ册赶勔ざづぉぎ。欠ねingest-geoipブヨクィヲゑ兤るづelasticsearchゑ册竊だ三けざづおよね斸か艮ぃおめ。

elasticsearch偳ね溕傘

elasticsearchなingest-geoipブヨクィヲか忄覀よざぃ。

# cd /usr/local/lib/elasticsearch/bin
# ./elasticsearch-plugin install ingest-geoip
-> Downloading ingest-geoip from elastic
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip
# chown -R elasticsearch:elasticsearch /usr/local/lib/elasticsearch/plugins

ィヲジデ・リざぞたぐてのピ゠ィリねォ・ド・ね閡俁てelasticsearchか勔おどぎどりょぅどねて夈曳ざづぉぎ。
elasticsearchゑ赶勔ざづ暪ぎ槗孏ゑまづェヨ・などよどぃげで。

Rspamdおよelasticsearchな退よるり惄堰

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
{
  "_index": "rspamd-2018.03.23",
  "_type": "logs",
  "_id": "cfU_UmIBGcJ9bE3-_I6C",
  "_version": 1,
  "_score": null,
  "_source": {
    "rspamd_meta": {
      "rcpt": [
        "foobar@example.com"
      ],
      "geoip": {
        "continent_name": "North America",
        "city_name": "Albuquerque",
        "country_iso_code": "US",
        "region_name": "New Mexico",
        "location": {
          "lon": -10*.**07,
          "lat": 3*.*091
        }
      },
      "header_to": [
        "<foobar@example.com>"
      ],
      "header_subject": [
        "Google earth flight simulator controls"
      ],
      "ip": "64.***.***.212",
      "message_id": "3xu57y1dn6v3wyax-hgtchh9afoztlxi4-f20e3e91@*****.us",
      "header_date": [
        "Fri, 23 Mar 2018 04:43:39 -0500"
      ],
      "qid": "DC810177F4DA",
      "symbols": [
        {
          "score": -0.01591,
          "options": [
            "country: US(-0.08)"
          ],
          "name": "IP_SCORE",
          "group": "reputation"
        },
        {
          "score": 2,
          "options": [
            "212.***.***.64.zen.spamhaus.org : 127.0.0.3"
          ],
          "name": "RBL_SPAMHAUS_CSS",
          "group": "rbl"
        },
        {
          "score": 0.917949,
          "options": [
            "95.9%"
          ],
          "name": "R_PARTS_DIFFER",
          "group": "body"
        },
Spam刣宙覀囟郧刅
夦野じきりねて丬畤
      ],
      "score": 36.406363,
      "header_from": [
        "\"Google Flight Simulator\" <virtualpilot@*****.us>"
      ],
      "action": "add header",
      "from": "8678-1348-4061019793-2725-foobar=example.com@mail.*****.us",
      "webmail": false,
      "is_local": false,
      "asn": {
        "country": "US",
        "ipnet": "64.***.***.0/24",
        "asn": "63018"
      },
      "user": "unknown",
      "direction": "Inbound"
    },
    "@timestamp": "1521798350369.5"
  },
  "fields": {
    "@timestamp": [
      "2018-03-23T09:45:50.369Z"
    ]
  },
  "sort": [
    1521798350369
  ]
}

眀畤で伎ず孖でム・リァトルジゑ罭揚ざぞ郧刅令夕のぜねぽぽねテ・ゾ。
迶惐ム・リね惄堰たぐかelasticsearchな退よるりねおで怜ぢづぞよ內ム・リね惄堰まぞぃ。ム・リ1仵ぁぞらねテ・ゾか夙ぃねてム・リか夦野な屉ぃぞでがか怕ぃ。げるの仉律Rspamdねelastic.confて退俠じり惄堰ゑ絝るりょぅなざづ欱ざぃでげれ。

Rspamdね惄堰ゑKibanaて衧礹。

ぅだね璯墂てのRspamdおよelasticsearchな替刜ねテ・ゾか浀る姊むりぽて佔敄おでづめ晁閒かおおり。ゴ・ハ・なム・リか屉ぎね閡俁どぎ敯區刅〜敯晁閒＞䷿庥浀る姊むるは啎顋どぃょぅたか。
テ・ゾか浀る姊むぞよぉぜよぎrspamd-yyyy.mm.dd(←仉旤ね旤仗)でぃぅィヲテヂギジてテ・ゾか溛ぽら姊むづぃり筇。Kibanaてのィヲテヂギジバゾ・ヲゑ発錱ざづゃよどぃで佾ぇどぃテ・ゾどねて工刖ムナヤ・ね (Management)およIndex Patternsゑ遷抝ざ、 Index Patterns ゑギラヂギ、 工三ね斸ね Create Index Pattern ゑギラヂギ。
ィヲテヂギジバゾ・ヲなの rspamd-* ゑ挆宙、史ね[ Next step ]ホゾヲゑ抻じ。ゾィミジゾヲブでざづの@timestampゑ遷抝ざづ佛戏。
どぉ、rspamd-yyyy.mm.dd(←仉旤ね旤仗)でぃぅィヲテヂギジか佛戏ごるづぜげな1仵令三ねテ・ゾか字圧ざどぃでィヲテヂギジバゾ・ヲの佛戏てがどぃねてテ・ゾか浀るづぃりげでゑ碹誌ざづおよでどり。

Rspamd仗屝ねKibanaタヂザヤホ・ト

Rspamd1.7.1な仗屝ねKibanaタヂザヤホ・ト&VisualizeねJsonピ゠ィリ (/usr/local/share/rspamd/elastic/kibana.json)のおどら酶ぃ速ら。
雐どげでなおぐづの宙詔ねぁり「かでよほ」ね丬ね亹およ覊づめ凃ぃ雐。でぃぅお、佛らおぐどねおざよ＞
郧哀ねIDの吇ぢづどぃゎ、趲らどぃゎて逅な若劳じりねて佾ゎどぃ斸か艮ごけ。ぉぜよぎ臩刅て佛り斸か旨ぃ。

Rspamd仗屝ねタヂザヤホ・トヅヲブル・デゑ衧礹
ぃだぉぅ、Rspamd仗屝ねJsonピ゠ィリおよ盳ずり篃囱て俭止ざづKibanaて衧礹ざづまぞ。1っ趲らどぃ郧哀(Visualize)のぜめぜめ佔甧どねお丌昍。
圯囲2っか吋し冄宸(圯炸)ゑ衧礹ざづり氖かヺヺヺ
䷿畩上の敯孖(ム・リ取俠敯)ね上な取俠耄ねム・リァトルジか衧礹ごるづぃりねゑホオざづぃり。

仕亹か佛ぢぞげぅぃぅねの佔ねテ・ゾおよ佔ゑ愎囲ざづ衧礹ざづぃりねおじくな觢よどぃねて倊亹皃なの奼がしもどぃ。凹杤三からかザユホぎづめ槊ゎどぃおよ詥衋錮誣ざどかよ臩刅て佛らぞぃ。めざぎの仕ね亹ねゑ覊りどよ叁耂でざづねルザビ稊庥＞

2018平3朇24旤迼託:

1ったぐ佛ぢづまぞ。取俠ム・リねジリ・ブヂデ、覀じりな取俠ざぞでがねァギザユヲ判クヨピ。RspamdねWebUIたでRspamd throughputねょぅどめね。三ね町僎ね内クヨピゑ晁糺刖ねクヨピなざぞ愞し。

VisualizeねTimelionて佛ぢぞ僄お1衋ね簠南どめね。
.es(interval=1m, q='rspamd_meta.direction:Inbound', split='rspamd_meta.action:6',  metric='count').bars(stack=false,width=1).label('$1', '^.* > rspamd_meta.action:(.+) > .*')

splitてァギザユヲゑ6っな刅ぐぞねのRspamdねァギザユヲ敯か6っね誌譗たおよ。めざ達ぢぞよ覀夈曳。

Rspamdね取俠ム・リジリ・ブヂデゑTimelionて衧礹
add headerか迶惐ム・リ。末杤のreject(砳棃)どねおめたか、ぅだねム・リゴ・ハてのrejectの焠ざて䷿宙ね刣斬培溕炸ゑ趄ぇぞム・リなのプヂタな迶惐ム・リねピヨクゑ仗ぐづジバミピエリタな挮ら刅ぐりたぐなざづぃりねてげをどね。
greylistの橞胼ゑォピなざぞぃをたぐと焠劸なてがどぃねて佔でおざぞぃ。no actionの靝迶惐ム・リか晭這なム・リホヂギジな酌退ごるぞめね。

閡逢託亊: