fluentd/log collecting server
[
Front page
] [
New
|
List of pages
|
Search
|
Recent changes
]
Start:
[[labs.beatcraft.com]]~
[[Armadillo]]~
#Contents
*fluentd/log collecting server [#bc29a259]
>
This article explains how to set up fluentd at the log co...
The logs, which fluentd has received, has been stored int...
** OS [#w9c2016d]
>
As of November 2015, the newest server version of LTS (Lo...
~
The configurations of host name and fixed IP address, and...
~
In this article, the host server (log collecting server) ...
Host Name: aggregator~
User Name: beat~
** MongoDB [#db211f65]
>
Please install from the repository of MongoDB, not from r...
*** Registering the public key for the certification of t...
>
beat@aggregator:~$ sudo apt-key adv --keyserver hkp://ke...
Executing: gpg --ignore-time-conflict --no-options --no-...
--trust-model always --keyring /etc/apt/trusted.gpg --pr...
gpg: requesting key 7F0CEB10 from hkp server keyserver.u...
gpg: key 7F0CEB10: public key "Richard Kreuter <richard@...
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
*** Adding the package list of MongoDB [#f395c216]
>
beat@aggregator:~$ echo "deb http://repo.mongodb.org/apt...
*** Obtaining the file list of newly added repository [#t...
>
beat@aggregator:~$ sudo apt-get update
*** Installing MongoDB [#q65759f9]
>
From the newly added MongoDB repository, please install m...
beat@aggregator:~$ sudo apt-get install mongodb-org
*** Dealing with the warning of mongo shell [#k1bd4ac0]
>
After the installation of deb package is completed, Mongo...
~
Create disable-transparent-hugepages under the directory ...
The content of page is listed below.~
>
#!/bin/sh
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: mongod mongodb-mms-automation-agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description: Disable Linux transparent huge page...
# database performance.
### END INIT INFO
case $1 in
start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage...
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fi
echo 'never' > ${thp_path}/enabled
echo 'never' > ${thp_path}/defrag
unset thp_path
;;
esac
~
As the file is created, please configure MongoDB to imple...
beat@aggregator:~$ sudo chmod 755 /etc/init.d/disable-tr...
beat@aggregator:~$ sudo update-rc.d disable-transparent-...
After MongoDB is rebooted, please sure that no warring sh...
beat@aggregator:~$ mongo
MongoDB shell version: 3.0.7
connecting to: test
>
**elasticsearch [#o7ff1b42]
>
To visualize the collected logs, please install elasticse...
*** Installing java [#z911ec76]
>
To execute elasticserach, please install java. Java is a ...
beat@aggregator:~$ sudo add-apt-repository ppa:webupd8te...
beat@aggregator:~$ sudo apt-get update
beat@aggregator:~$ sudo apt-get install oracle-java8-inst...
***Installing elasticsearch [#w72ac524]
>
To follow [[the official document>https://www.elastic.co/...
beat@aggregator:~$ wget -qO - https://packages.elastic....
beat@aggregator:~$ echo "deb http://packages.elastic.co/...
beat@aggregator:~$ sudo apt-get update
beat@aggregator:~$ sudo apt-get install elasticsearch
~
As the installation is completed, please configure elasti...
beat@aggregator:~$ sudo update-rc.d elasticsearch defaul...
Accessing port 9200, please check that elasticsearch work...
If you receive the reply shown below, it indicates that e...
beat@aggregator:~$ curl -X GET http://localhost:9200/
{
"status" : 200,
"name" : "Red Wolf",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.7.3",
"build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df65...
"build_timestamp" : "2015-10-15T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
*** Adjusting elasticsearch [#d551a01b]
>
Refer to [[Configuration>https://www.elastic.co/guide/en/...
~
Please add the two lines shown below at /etc/security/lim...
elasticsearch - nofile 65535
elasticsearch - memlock unlimited
Please add the single line shown below at /etc/elasticsea...
bootstrap.mlockall: true
Please add the three lines listed below. These additions ...
ES_HEAP_SIZE=1g <-- half amount of physical memory
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
** Apache [#u8a90f8e]
>
Install Apache2, which is downloaded from the repository ...
beat@aggregator:~$ sudo apt-get install apache2
Its configuration remains as the default. The document ro...
Depends on your needs, please apply any security measures.~
**Kibana [#db67d1d2]
*** Installing Kibana3 [#z50e3a3f]
>
Download Kibana3, and put it to the document root of Apac...
beat@aggregator:~$ bwget https://download.elastic.co/kib...
beat@aggregator:~$ tar xvf kibana-3.1.2.tar.gz
beat@aggregator:~$ mv kibana-3.1.2 kibana3
beat@aggregator:~$ sudo mv kibana3 /var/www/html/
*** Adjusting the additional configuration [#j8255566]
>
To open up logstash logs of elasticsearch from kibana3, ...
http.cors.allow-origin: "/.*/"
http.cors.enabled: true
After adding these lines, please restart elasticsearch an...
beat@aggregator:~$ sudo /etc/init.d/elasticsearch restart
[sudo] password for beat:
* Stopping Elasticsearch Server ...
* Starting Elasticsearch Server
*** Confirming the operation [#l83b38a4]
>
Open up a web browser, and go to the page allocated to Ki...
** fluentd [#ya6823ca]
*** Increasing ulimit [#b70b0f88]
>
Before installing fluentd, please increase ulimit. To inc...
~
Please add the three lines listed below at the end of the...
root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536
~
After rebooting it, please check the change becomes effec...
beat@aggregator:~$ ulimit -n
65536
***Installing fluentd [#g30ed3c2]
>
To install fluentd, please follow the instructions, which...
(gem is the package management tool.)~
~
First, install packages, which are requirements for gem b...
beat@aggregator:~$ sudo apt-get install build-essential
beat@aggregator:~$ sudo apt-get install ruby ruby-dev
Then, install fluentd by gem.
beat@aggregator:~$ sudo gem install fluentd --no-ri --no...
Fetching: msgpack-0.5.12.gem (100%)
==Skipping==
Fetching: string-scrub-0.0.5.gem (100%)
Building native extensions. This could take a while...
Fetching: fluentd-0.12.15.gem (100%)
Successfully installed msgpack-0.5.12
Successfully installed json-1.8.3
Successfully installed yajl-ruby-1.2.1
Successfully installed cool.io-1.3.1
Successfully installed http_parser.rb-0.6.0
Successfully installed sigdump-0.2.3
Successfully installed thread_safe-0.3.5
Successfully installed tzinfo-1.2.2
Successfully installed tzinfo-data-1.2015.5
Successfully installed string-scrub-0.0.5
Successfully installed fluentd-0.12.15
11 gems installed
~
As fluentd is installed, the packages of its dependencies...
***fluentd elasticsearch plug-in [#q2886695]
>
A plug-in, which transfers logs from fluentd to elasticse...
beat@aggregator:~$ sudo apt-get install libcurl4-openssl...
beat@aggregator:~$ sudo gem install fluent-plugin-elasti...
Fetching: excon-0.45.4.gem (100%)
~~ Skipping ~~
Fetching: fluent-plugin-elasticsearch-1.0.0.gem (100%)
Successfully installed excon-0.45.4
Successfully installed multi_json-1.11.2
Successfully installed multipart-post-2.0.0
Successfully installed faraday-0.9.1
Successfully installed elasticsearch-transport-1.0.12
Successfully installed elasticsearch-api-1.0.12
Successfully installed elasticsearch-1.0.12
Successfully installed fluent-plugin-elasticsearch-1.0.0
8 gems installed
***fluent mongo plug-in [#becedf95]
>
Install the plug-in, which sends logs from fluentd and sa...
beat@aggregator:~$ sudo gem install fluent-plugin-mongo
Fetching: bson-1.12.3.gem (100%)
Fetching: mongo-1.12.3.gem (100%)
Fetching: fluent-plugin-mongo-0.7.10.gem (100%)
Successfully installed bson-1.12.3
Successfully installed mongo-1.12.3
Successfully installed fluent-plugin-mongo-0.7.10
3 gems installed
***Anther fluent plugin [#ca485588]
>
To write down the configuration file of fluentd simply, p...
[[https://rubygems.org/gems/fluent-plugin-forest]]~
beat@aggregator:~$ sudo gem install fluent-plugin-forest
Fetching: fluent-plugin-forest-0.3.0.gem (100%)
Successfully installed fluent-plugin-forest-0.3.0
1 gem installed
***fluentd.conf [#r283c2cd]
>
To receive the log data from fluentd, which is configured...
<source>
@type forward
@id forward_input
</source>
<match syslog.**>
@type forest
subtype copy
<template>
<store>
type elasticsearch
logstash_format true
host localhost
port 9200
index_name fluentd
type_name syslog
flush_interval 10s
buffer_chunk_limit 2048k
buffer_queue_limit 5
buffer_path /data/tmp/es_syslog/${hostname}.${tag_...
buffer_type file
</store>
<store>
type mongo
host localhost
port 27017
database fluentd
collection adv
capped
capped_size 4096m
flush_interval 10s
buffer_chunk_limit 8192k
buffer_queue_limit 512
buffer_path /data/tmp/mongo_syslog/${hostname}.${t...
buffer_type file
</store>
<store>
type file
path /data/tmp/syslog/${hostname}.${tag_parts[1]}....
buffer_path /data/tmp/syslog/${hostname}.${tag_par...
flush_interval 10s
buffer_chunk_limit 8192k
buffer_queue_limit 512
buffer_type file
</store>
</template>
</match>
>
As buffer_type is set to file, path, which is generated a...
For this example, three individual paths, for elasticsear...
***Checking its operation [#v5110f67]
>
To start up fluentd with the log option, please check whe...
- Receive logs from fluentd at Armadillo-Box WS1.
- Send and save the logs into Elasticsearch.
- Save the logs in MongoDB.
- Output to the file.
>
To check these tasks, please apply the command lines below.
beat@aggregator:~$ sudo -s
root@aggregator:~# fleuntd -c /etc/fluent/fluent.conf -o...
root@aggregator:~# tail -f /var/log/fluent.log
>
Once you have recognized that fluentd operates correctly,...
The address of kibana3 is http://{IP address of collectin...
>
As log data are getting collected, the temperature graph ...
~
&ref(kibana3_temp_dashboard.jpg,,50%);
>
Once you have checked that all system works appropriately...
[[https://gist.github.com/Leechael/3671811]]~
*Revision History [#j685dcfc]
>
- 2015-12-08 This article is initially released.~
End:
[[labs.beatcraft.com]]~
[[Armadillo]]~
#Contents
*fluentd/log collecting server [#bc29a259]
>
This article explains how to set up fluentd at the log co...
The logs, which fluentd has received, has been stored int...
** OS [#w9c2016d]
>
As of November 2015, the newest server version of LTS (Lo...
~
The configurations of host name and fixed IP address, and...
~
In this article, the host server (log collecting server) ...
Host Name: aggregator~
User Name: beat~
** MongoDB [#db211f65]
>
Please install from the repository of MongoDB, not from r...
*** Registering the public key for the certification of t...
>
beat@aggregator:~$ sudo apt-key adv --keyserver hkp://ke...
Executing: gpg --ignore-time-conflict --no-options --no-...
--trust-model always --keyring /etc/apt/trusted.gpg --pr...
gpg: requesting key 7F0CEB10 from hkp server keyserver.u...
gpg: key 7F0CEB10: public key "Richard Kreuter <richard@...
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
*** Adding the package list of MongoDB [#f395c216]
>
beat@aggregator:~$ echo "deb http://repo.mongodb.org/apt...
*** Obtaining the file list of newly added repository [#t...
>
beat@aggregator:~$ sudo apt-get update
*** Installing MongoDB [#q65759f9]
>
From the newly added MongoDB repository, please install m...
beat@aggregator:~$ sudo apt-get install mongodb-org
*** Dealing with the warning of mongo shell [#k1bd4ac0]
>
After the installation of deb package is completed, Mongo...
~
Create disable-transparent-hugepages under the directory ...
The content of page is listed below.~
>
#!/bin/sh
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: mongod mongodb-mms-automation-agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description: Disable Linux transparent huge page...
# database performance.
### END INIT INFO
case $1 in
start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage...
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fi
echo 'never' > ${thp_path}/enabled
echo 'never' > ${thp_path}/defrag
unset thp_path
;;
esac
~
As the file is created, please configure MongoDB to imple...
beat@aggregator:~$ sudo chmod 755 /etc/init.d/disable-tr...
beat@aggregator:~$ sudo update-rc.d disable-transparent-...
After MongoDB is rebooted, please sure that no warring sh...
beat@aggregator:~$ mongo
MongoDB shell version: 3.0.7
connecting to: test
>
**elasticsearch [#o7ff1b42]
>
To visualize the collected logs, please install elasticse...
*** Installing java [#z911ec76]
>
To execute elasticserach, please install java. Java is a ...
beat@aggregator:~$ sudo add-apt-repository ppa:webupd8te...
beat@aggregator:~$ sudo apt-get update
beat@aggregator:~$ sudo apt-get install oracle-java8-inst...
***Installing elasticsearch [#w72ac524]
>
To follow [[the official document>https://www.elastic.co/...
beat@aggregator:~$ wget -qO - https://packages.elastic....
beat@aggregator:~$ echo "deb http://packages.elastic.co/...
beat@aggregator:~$ sudo apt-get update
beat@aggregator:~$ sudo apt-get install elasticsearch
~
As the installation is completed, please configure elasti...
beat@aggregator:~$ sudo update-rc.d elasticsearch defaul...
Accessing port 9200, please check that elasticsearch work...
If you receive the reply shown below, it indicates that e...
beat@aggregator:~$ curl -X GET http://localhost:9200/
{
"status" : 200,
"name" : "Red Wolf",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.7.3",
"build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df65...
"build_timestamp" : "2015-10-15T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
*** Adjusting elasticsearch [#d551a01b]
>
Refer to [[Configuration>https://www.elastic.co/guide/en/...
~
Please add the two lines shown below at /etc/security/lim...
elasticsearch - nofile 65535
elasticsearch - memlock unlimited
Please add the single line shown below at /etc/elasticsea...
bootstrap.mlockall: true
Please add the three lines listed below. These additions ...
ES_HEAP_SIZE=1g <-- half amount of physical memory
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
** Apache [#u8a90f8e]
>
Install Apache2, which is downloaded from the repository ...
beat@aggregator:~$ sudo apt-get install apache2
Its configuration remains as the default. The document ro...
Depends on your needs, please apply any security measures.~
**Kibana [#db67d1d2]
*** Installing Kibana3 [#z50e3a3f]
>
Download Kibana3, and put it to the document root of Apac...
beat@aggregator:~$ bwget https://download.elastic.co/kib...
beat@aggregator:~$ tar xvf kibana-3.1.2.tar.gz
beat@aggregator:~$ mv kibana-3.1.2 kibana3
beat@aggregator:~$ sudo mv kibana3 /var/www/html/
*** Adjusting the additional configuration [#j8255566]
>
To open up logstash logs of elasticsearch from kibana3, ...
http.cors.allow-origin: "/.*/"
http.cors.enabled: true
After adding these lines, please restart elasticsearch an...
beat@aggregator:~$ sudo /etc/init.d/elasticsearch restart
[sudo] password for beat:
* Stopping Elasticsearch Server ...
* Starting Elasticsearch Server
*** Confirming the operation [#l83b38a4]
>
Open up a web browser, and go to the page allocated to Ki...
** fluentd [#ya6823ca]
*** Increasing ulimit [#b70b0f88]
>
Before installing fluentd, please increase ulimit. To inc...
~
Please add the three lines listed below at the end of the...
root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536
~
After rebooting it, please check the change becomes effec...
beat@aggregator:~$ ulimit -n
65536
***Installing fluentd [#g30ed3c2]
>
To install fluentd, please follow the instructions, which...
(gem is the package management tool.)~
~
First, install packages, which are requirements for gem b...
beat@aggregator:~$ sudo apt-get install build-essential
beat@aggregator:~$ sudo apt-get install ruby ruby-dev
Then, install fluentd by gem.
beat@aggregator:~$ sudo gem install fluentd --no-ri --no...
Fetching: msgpack-0.5.12.gem (100%)
==Skipping==
Fetching: string-scrub-0.0.5.gem (100%)
Building native extensions. This could take a while...
Fetching: fluentd-0.12.15.gem (100%)
Successfully installed msgpack-0.5.12
Successfully installed json-1.8.3
Successfully installed yajl-ruby-1.2.1
Successfully installed cool.io-1.3.1
Successfully installed http_parser.rb-0.6.0
Successfully installed sigdump-0.2.3
Successfully installed thread_safe-0.3.5
Successfully installed tzinfo-1.2.2
Successfully installed tzinfo-data-1.2015.5
Successfully installed string-scrub-0.0.5
Successfully installed fluentd-0.12.15
11 gems installed
~
As fluentd is installed, the packages of its dependencies...
***fluentd elasticsearch plug-in [#q2886695]
>
A plug-in, which transfers logs from fluentd to elasticse...
beat@aggregator:~$ sudo apt-get install libcurl4-openssl...
beat@aggregator:~$ sudo gem install fluent-plugin-elasti...
Fetching: excon-0.45.4.gem (100%)
~~ Skipping ~~
Fetching: fluent-plugin-elasticsearch-1.0.0.gem (100%)
Successfully installed excon-0.45.4
Successfully installed multi_json-1.11.2
Successfully installed multipart-post-2.0.0
Successfully installed faraday-0.9.1
Successfully installed elasticsearch-transport-1.0.12
Successfully installed elasticsearch-api-1.0.12
Successfully installed elasticsearch-1.0.12
Successfully installed fluent-plugin-elasticsearch-1.0.0
8 gems installed
***fluent mongo plug-in [#becedf95]
>
Install the plug-in, which sends logs from fluentd and sa...
beat@aggregator:~$ sudo gem install fluent-plugin-mongo
Fetching: bson-1.12.3.gem (100%)
Fetching: mongo-1.12.3.gem (100%)
Fetching: fluent-plugin-mongo-0.7.10.gem (100%)
Successfully installed bson-1.12.3
Successfully installed mongo-1.12.3
Successfully installed fluent-plugin-mongo-0.7.10
3 gems installed
***Anther fluent plugin [#ca485588]
>
To write down the configuration file of fluentd simply, p...
[[https://rubygems.org/gems/fluent-plugin-forest]]~
beat@aggregator:~$ sudo gem install fluent-plugin-forest
Fetching: fluent-plugin-forest-0.3.0.gem (100%)
Successfully installed fluent-plugin-forest-0.3.0
1 gem installed
***fluentd.conf [#r283c2cd]
>
To receive the log data from fluentd, which is configured...
<source>
@type forward
@id forward_input
</source>
<match syslog.**>
@type forest
subtype copy
<template>
<store>
type elasticsearch
logstash_format true
host localhost
port 9200
index_name fluentd
type_name syslog
flush_interval 10s
buffer_chunk_limit 2048k
buffer_queue_limit 5
buffer_path /data/tmp/es_syslog/${hostname}.${tag_...
buffer_type file
</store>
<store>
type mongo
host localhost
port 27017
database fluentd
collection adv
capped
capped_size 4096m
flush_interval 10s
buffer_chunk_limit 8192k
buffer_queue_limit 512
buffer_path /data/tmp/mongo_syslog/${hostname}.${t...
buffer_type file
</store>
<store>
type file
path /data/tmp/syslog/${hostname}.${tag_parts[1]}....
buffer_path /data/tmp/syslog/${hostname}.${tag_par...
flush_interval 10s
buffer_chunk_limit 8192k
buffer_queue_limit 512
buffer_type file
</store>
</template>
</match>
>
As buffer_type is set to file, path, which is generated a...
For this example, three individual paths, for elasticsear...
***Checking its operation [#v5110f67]
>
To start up fluentd with the log option, please check whe...
- Receive logs from fluentd at Armadillo-Box WS1.
- Send and save the logs into Elasticsearch.
- Save the logs in MongoDB.
- Output to the file.
>
To check these tasks, please apply the command lines below.
beat@aggregator:~$ sudo -s
root@aggregator:~# fleuntd -c /etc/fluent/fluent.conf -o...
root@aggregator:~# tail -f /var/log/fluent.log
>
Once you have recognized that fluentd operates correctly,...
The address of kibana3 is http://{IP address of collectin...
>
As log data are getting collected, the temperature graph ...
~
&ref(kibana3_temp_dashboard.jpg,,50%);
>
Once you have checked that all system works appropriately...
[[https://gist.github.com/Leechael/3671811]]~
*Revision History [#j685dcfc]
>
- 2015-12-08 This article is initially released.~
Page:
![[PukiWiki]](image/bc_logo.png "[PukiWiki]"){kind=logo}
