labs.beatcraft.com
Armadillo
Armadillo-Box WS1/BLE Gateway

Armadillo-Box WS1/fluentd †

The kernel and UserLand of Armadillo-Box WS1 are similar to ones of Armadillo-IoT. Paying attention to the size of ROM image, the most applications that run on Armadillo-IoT also run on Armadillo-Box WS1.

This article explains how to implement fluentd on Armadillo-Box WS1, which is already modified for working on BLE and Wifi. (Please read Armadillo-Box WS1/BLE Support and Armadillo-Box WS1/Wifi Support for BLE and Wifi on Armadillo-Box WS1.) Using the fluentd, this article shows the steps to let Armadillo-Box WS1 collect the data from devices via BLE and send the collected data to a server from Armadillo-Box.

↑

Format of the collected data †

The collected data is collected by a specific application, which manages Bluetooth Low Energy communication and is sent to the server by fluentd. The format of data is shown below, and The data are periodically written into /var/log/messages, which are output syslog of atmark dist. Tempgw indicates the name of data collecting application, which output the data to syslog, ADV is the MAC address of BLE device, serial is used as an ID when it receives the data from multiple devices. Temp is temperature, and rssi is reception strength.

Nov 30 15:00:43 (none) user.info tempgw: ADV: E0:40:56:7E:2A:DC serial:3 temp:25.125000 rssi:-79


These data are read by tail plugin of fluentd. The read date are formatted appropriately and sent to the receiving fluentd. If these data are read by syslog plugin, the data will not be formatted and whole messages will be sent to the server. The message format is cannot be used for search, and it is inconvenient. Thus, the received data is formatted by tail plugin.

↑

Adding fluentd †

Add the fluentd related files to atmark dist by make menuconfig.
Select vender/Product.

(AtmarkTechno) Vendor
(Armadillo-Box WS1) AtmarkTechno Products


Configuring UserLand, activate fluented and the requirements for fluentd.

[*] Customize Vendor/User Settings (NEW)
     Miscellaneous Applications --->
         :
         :
        ---     extension
         :
         :
        [*] ruby                  <-- Put the checkmark to make it effective.
        (2.1.2)   Version         <-- Choose 2.1.2. If select 2.0.0-p451, build errors on fluentd and others will occur.
        [*]   fluentd             <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
         :
        [*]   http_parser.rb      <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
        [*]   ruby-cool.io        <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
         :
        [*]   ruby-macaddr        <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
         :
        [*]   ruby-sigdump        <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
        [*]   ruby-systemu        <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
        [*]   ruby-string-scrub   <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
        [*]   ruby-thread_safe    <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
        [*]   ruby-tzinfo         <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
        [*]   ruby-tzinfo-data    <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
         :
        [*]   msgpack-ruby        <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
         :
        [*]   yajl-ruby           <-- If ruby is selected, this will appear. Please put the checkmark to make it effective.
         :
         :
         :
↑

Building kernel and UserLand †

With the configuration above, build kernel and UserLand.

atmark@atde5:~/atmark-dist$ make clean
atmark@atde5:~/atmark-dist$ make
↑

Placing fluent.conf †

Create a directory for fluentd at /etc of ROM image directory, and place fluent.conf.

However, the feature of atmark dist, which is booted by ramfs, is to store the configuration of post-booted modification in the directory below /etc/config/ or external storages such as microSD and USB memory. To save the configuration of post-booted modification, create a symlink. /etc/fluent/fluent.conf is the symlink to /etc/config/fluent.conf.

atmark@atde5:~/atmark-dist$ cd romfs/etc
atmark@atde5:~/atmark-dist/romfs/etc$ mkdir fluent
atmark@atde5:~/atmark-dist/romfs/etc$ cd fluent
atmark@atde5:~/atmark-dist/romfs/etc/fluent$ ln -s /etc/config/fluent.conf fluent.conf


Also, the configuration files located below /etc/default/ are copied and stored in the config storage area of flash memory inside Armadillo-Box when it is initially booted. Therefore, the original configuration files located below /etc/config such as fluent.config are saved at the directory below /etc/default/.

atmark@atde5:~/atmark-dist/romfs/etc/fluent$ cd ../default/
atmark@atde5:~/atmark-dist/romfs/etc/default$ cp ~/atmark-dist/user/fluentd/fluent.conf
↑

Rebuilding image †

As the placement of fluent.conf is completed, the image of UserLand, which includes the configuration file discussed above is built again.

atmark@atde5:~$ cd atmark-dist
atmark@atde5:~/atmark-dist$ make image
↑

Checking the operation †

If Armadillo-Box WS1 is connected to a network, the fastest way to replace to the recreated kernel and UserLand is to write flash memory over the network. (See Japanese Reference)

hermit> tftpdl 192.168.0.90 192.168.0.164 --blksize=1024 –kernel=linux.bin.gz
hermit> tftpdl 192.168.0.90 192.168.0.164 --blksize=1024 --userland=romfs.img.gz


If Armadillo-Box WS1 is booted and connected to the network correctly with new kernel and UserLand, please reset the time.

[root@abws1-0 (ttymxc1) ~]# ntpclient -h ntp.nict.jp -s
25567 00180.848    8769.0      0.8  1448951602547493.0      0.0         0
[root@abws1-0 (ttymxc1) ~]# date
Tue Nov 25 15:36:24 JST 2015


Change the contents of fluent.conf to ones shown below.

<source>
 @type tail
  format /^(?<time>[^ ]* [^ ]* [^ ]*) (?<host>[^ ]*) (?<class>[^ ]*) (?<ident>[^ ]*): .*: (?<ADV>([A-Z0-9]{2}[:]){5}([A-Z0-9]{2})) .*:(?<temp>[0-9]{2}[\.][0-9]{6}) (?<message>.*)$/
  time_format %b %d %H:%M:%S
  path /var/log/messages
  pos_file /var/tmp/mesagges.pos
  types temp:float
  tag syslog.ble
</source>

<match syslog.**>
  @type copy
  <store>
    @type forward
    heartbeat_interval 60s
    heartbeat_type tcp
    flush_interval 60s ##600s at realuse
    buffer_type file
    buffer_path /var/tmp/in_ble_app.syslog.*.buffer
    buffer_chunk_limit 3072k
    buffer_queue_limit 425
    retry_wait 2s
    max_retry_wait 1h
    retry_limit  72 # retry count

    <server>
      host 192.168.0.80
      port 24224
    </server>
  </store>
  <store>
    type stdout
  </store>
</match>


Execute fluentd with the debug option at the foreground.

[root@abws1-0 (ttymxc1) ~]# fluentd -c /etc/fluent/fluent.conf -vv
2015-11-25 16:09:28 +0900 [info]: fluent/supervisor.rb:341:read_config: reading config file path="/etc/fluent/fluent.conf"
2015-11-25 16:09:29 +0900 [info]: fluent/supervisor.rb:239:supervise: starting fluentd-0.12.2
2015-11-25 16:09:30 +0900 [trace]: fluent/plugin.rb:98:register_impl: registered buffer plugin 'file'
2015-11-25 16:09:30 +0900 [trace]: fluent/plugin.rb:98:register_impl: registered buffer plugin 'memory'
~~ Skipping~~
2015-12-01 16:09:37 +0900 [info]: fluent/agent.rb:123:add_match: adding match pattern="syslog.**" type="stdout"
2015-12-01 16:09:37 +0900 [info]: fluent/root_agent.rb:142:add_source: adding source type="tail"
2015-12-01 16:09:37 +0900 [info]: plugin/in_tail.rb:474:initialize: following tail of /var/log/messages
2015-12-01 16:09:56 +0900 syslog.ble: {"host":"(none)","class":"user.info","ident":"tempgw","ADV":"ED:B2:9F:B5:70:CD","serial":"1","temp":27.375,"message":"rssi:-57"}
2015-12-01 16:10:56 +0900 syslog.ble: {"host":"(none)","class":"user.info","ident":"tempgw","ADV":"ED:B2:9F:B5:70:CD","serial":"1","temp":27.25,"message":"rssi:-59"}
2015-12-01 16:11:56 +0900 syslog.ble: {"host":"(none)","class":"user.info","ident":"tempgw","ADV":"ED:B2:9F:B5:70:CD","serial":"1","temp":27.25,"message":"rssi:-59"}


Please quit fluentd if the outputs to stdout are recognized and if the massages, which are written onto syslog in the regular expression of tail plugin format, are correctly divided by the keys. Then save the contents of fluent.conf by applying the command line of flatfsd -s.

↑

Execution configuration for booting †

Add the configurations, which regulate the operation of fluentd related applications when Armadilo-Box is booted. (Japanese Reference)

Please add the contents shown below to /etc/config.rc.local. Then, apply flatfsd -s to save the addition.

#!/bin/sh

. /etc/init.d/functions

PATH=/bin:/sbin:/usr/bin:/usr/sbin

# ntp
ntpclient -h ntp.nict.jp -s
 
# BLE temperature log application
tempgw &
# fleuntd
fluentd -c /etc/fluent/fluent.conf -o /var/log/fluent.log &


In this article, fluentd is directly executed in rc.local. However, you may put the specific boot script for fluentd at /etc/init.d. Or, you can manage the booting of fluentd by start-stop-daemon. (Japanese Reference)

To boot fluentd from rc.local. Unlike execting on shell at root, timezone is not corrected.
The uncorrected timezone affects on the date of logs, which are sent from fluentd.

↑

fluentd at the receiving side †

Applying the configurations above, the logs are sent by the fluentd, which is installed on Bluetooth Low Energy Gateway enabled Armadillo-Box SW1. The sent logs are received by the fluentd, which is on the log collecting server.
About the configuration of log collecting server, please look at fluentd/log collecting server.

↑

Revision History †

  • 2015-12-08 This article is initially released.
Front page   Edit Freeze Diff Backup Upload Copy Rename Reload   New List of pages Search Recent changes   RSS of recent changes
Last-modified: 2015-12-22 (Tue) 11:05:18 (3274d)