syslog_ng

Build Status

Table of Contents

  1. Overview
  2. Module Description
  3. Setup - The basics of getting started with syslog_ng
  4. Usage - Configuration options and additional functionality
  5. Implementation details
  6. Limitations - OS compatibility, etc.
  7. Development - Guide for contributing to the module

Overview

This module lets you generate syslog-ng configuration using puppet. It supports all kinds of statements, such as sources, destinations, templates, and so on. After defining them, you can combine them into a log path. This module also takes care of installing syslog-ng, or reloading it after a configuration file change.

You can check the supported platforms in the Limitations section.

Module Description

This module integrates well with syslog-ng. It supports its configuration model so you can create new sources and destinations as Puppet resources. Under the hood they are just defined resource types.

The supported statements:

  • options
  • template
  • rewrite
  • parser
  • filter (partial support)
  • source
  • destination
  • log
  • +1: config, which lets you insert existing configuration snippets.

Each type is under the syslog_ng:: namespace, so you can use them like this:

syslog_ng::source { 's_gsoc':
    params => {
        'type' => 'tcp',
        'options' => [
            {'ip' => "'127.0.0.1'"},
            {'port' => 1999}
        }]
    }
}

There is a shorter form: 

syslog_ng::source { 's_gsoc':
    params => {
        'tcp' => [
            {'ip' => "'127.0.0.1'"},
            {'port' => 1999}
        }]
    }
}

Configuration syntax

Every statement has the same layout. They can accept a params parameter, which can be a hash or an array of hashes. Each hash should have a type and options key or you can use a shorter form like here.

The value of the type represents the type of the statement, in case of a source this can be file, tcp and so on.

The value of the options is an array of strings and hashes. You have to pay attention to quotation when using strings. If you want the inner quotation to be a single quote ( in the syslog-ng.conf), then the outer one must be a double, like "'this string'", which will be transformed into 'this string'.

Similarly, you can write '"this string"'' to get "this string" in the configuration.

By using this convention, the module will generate correct configuration files. If the option array is empty, you can use an empty string '' instead.

As I mentioned, there are strings and hashes in an option. Hashes must contain only one key. This key will identify the name of the parameter and its value must be an array of strings. If that would contain only one item, the value can be simply just a string.

You can find a lot of examples under the tests and spec directories. The default_config.pp under the tests directory contains the default configuration from the syslog-ng source, translated into Puppet types.

Setup

Puppet Forge

This module has not been published on Puppet Forge yet.

Installing from source

You can install it following these steps:

  1. Make sure you have the required dependencies:
    • rake
    • ruby
  2. Clone the source code into a directory: $ git clone git@github.com:ihrwein/puppet-syslog_ng.git
  3. Make sure you are on master branch: $ git checkout master
  4. Build a package: $ rake build This will create a tar.gz file under the pkg directory. Now you should be able to install the module: # puppet module install -f ihrwein-syslog_ng-VERSION.tar.gz

What syslog_ng affects

  • It installs the syslog-ng or syslog-ng-core package
    • that creates the necessary directories on your system, including /etc/syslog-ng.
    • If another syslog daemon is installed, it will be removed by your package manager.
  • purges the content of /etc/syslog-ng/syslog-ng.conf
  • creates a temporary file under /tmp with the name of syslog-ng.conf.tmp. It always uses this file and will not delete it.

Getting started with syslog_ng

If you are not familiar with syslog-ng, I suggest you to take a look at the Syslog-ng Admin Guide which contains all the necessary information to use this software.

You can also get help on the syslog-ng mailing list.

Usage

Just use the classes and defined types as you would use them, without Puppet.

Before the generated configuration would be applied, it is written to a temporary file. Next, the module checks the configuration syntax of this file, and if it is OK, it overwrites the real configuration file. So you do not have to worry about configuration errors.

Classes and defined types

Class: syslog_ng

The main class of this module. By including it you get an installed syslog-ng with default configuration on your system.

Parameters within syslog_ng:

config_file

Configures the path of the configuration file. Defaults to /etc/syslog-ng/syslog-ng.conf on all operation systems.

sbin_path

Configures the path, where syslog-ng and syslog-ng-ctl binaries can be found. Defaults to /usr/sbin.

user

Configures syslog-ng to run as user.

group

Configures syslog-ng to run as group.

syntax_check_before_reloads

The module always checks the syntax of the generated configuration. If it is not OK, the main configuration (usually /etc/syslog-ng/syslog-ng.conf) will not be overwritten, but you can disable this behavior by setting this parameter to false.

Defined type: syslog_ng::config

Some elements of the syslog-ng DSL are not supported by this module (mostly the boolean operators in filters) so you may want to keep some configuration snippets in their original form. This type lets you write texts into the configuration without any parsing or processing.

Every configuration file begins with a @version: <version> line. You can use this type to write this line into the configuration, make comments or use existing snippets.

syslog_ng::config {'version':
    content => '@version: 3.6',
    order => '02'
}

Parameters within syslog_ng::config:

content

Configures the text which must be written into the configuration file. A newline character is automatically appended to its end.

order

Sets the order of this snippet in the configuration file. See Implementation. If you want to write the version line, the order => '02' is suggested, because the auto generated header has order '01'.

Defined type: syslog_ng::destination

Creates a destination in your configuration.

syslog_ng::destination { 'd_udp':
    params => {
        'type' => 'udp',
        'options' => [
            "'127.0.0.1'",
            {'port' => '1999'},
            {'localport' => '999'}
        ]
    }
}

Parameters within syslog_ng::destination:

params

An array of hashes or a single hash. It uses the syntax described here.

Defined type: syslog_ng::filter

Creates a filter in your configuration. It does not support binary operators, such as and or or. Please, use a syslog_ng::config if you need this functionality.

syslog_ng::filter {'f_tag_filter':
    params => {
        'type' => 'tags',
        'options' => [
            '".classifier.system"'
        ]
    }
}

Parameters within syslog_ng::filter:

params

An array of hashes or a single hash. It uses the syntax described here.

Defined type: syslog_ng::log

Creates log paths in your configuration. It can create channels, junctions and reference already defined sources, destinations, etc. The syntax is a little bit different then the one previously described under statements.

Here is a simple rule: if you want to reference an already defined type (e.g. s_gsoc2014) use a hash with one key-value pair. The key must be the type of the statement (e.g. source) and the value must be its title.

If you do not specify a reference, use an array instead. Take a look at the example below:

syslog_ng::log {'l2':
    params => [
        {'source' => 's_gsoc2014'},
        {'junction' => [
            {
            'channel' => [
                {'filter' => 'f_json'},
                {'parser' => 'p_json'}
            ]},
            {
            'channel' => [
                {'filter' => 'f_not_json'},
                {'flags' => 'final'}
            ]}
        ]
        },
        {'destination' => 'd_gsoc'}
    ]
}

Parameters within syslog_ng::log:

params

The syntax is a bit different, but you can find examples under the tests directory.

Defined type: syslog_ng::options

Creates a global options statement. Currently it is not a class, so you should not declare it multiple times! It is not defined as a class, so you can declare it as other similar types.

syslog_ng::options { "global_options":
    options => {
        'bad_hostname' => "'no'",
        'time_reopen'  => 10
    }
}

Parameters within syslog_ng::options:

options

A hash containing string keys and string values. In the generated configuration the keys will appear in alphabetical order.

Class: syslog_ng::params

Contains some basic constants which are used during the configuration generation. It is the base class of syslog_ng. You should not use this class directly, it is part of the inner implementation.

Defined type: syslog_ng::parser

Creates a parser statement in your configuration.

syslog_ng::parser {'p_hostname_segmentation':
    params => {
        'type' => 'csv-parser',
        'options' => [
            {'columns' => [
                '"HOSTNAME.NAME"',
                '"HOSTNAME.ID"'
            ]},
            {'delimiters' => '"-"'},
            {'flags' => 'escape-none'},
            {'template' => '"${HOST}"'}
        ]
    }
}

Parameters within syslog_ng::parser:

params

An array of hashes or a single hash. It uses the syntax which is described here.

Class: syslog_ng::reload

Contains a logic, which is able to reload syslog-ng. You should not use this class directly, it is part of the inner implementation.

Defined type: syslog_ng::rewrite

Creates one or more rewrite rules in your configuration.

syslog_ng::rewrite{'r_rewrite_subst':
    params => {
        'type' => 'subst',
        'options' => [
            '"IP"',
            '"IP-Address"',
            {'value' => '"MESSAGE"'},
            {'flags' => 'global'}
        ]
    }
}

Parameters within syslog_ng::rewrite:

params

An array of hashes or a single hash. It uses the syntax which is described here.

Defined type: syslog_ng::source

Creates a source in your configuration.

syslog_ng::source { 's_gsoc':
    params => {
        'type' => 'tcp',
        'options' => {
            'ip' => "'127.0.0.1'",
            'port' => 1999
        }
    }
}

syslog_ng::source {'s_external':
    params => [
        { 'type' => 'udp',
          'options' => [
            {'ip' => ["'127.0.0.1'"]},
            {'port' => [514]}
            ]
        },
        { 'type' => 'tcp',
          'options' => [
            {'ip' => ["'127.0.0.1'"]},
            {'port' => [514]}
            ]
        },
        {
          'type' => 'syslog',
          'options' => [
            {'flags' => ['no-multi-line', 'no-parse']},
            {'ip' => ["'127.0.0.1'"]},
            {'keep-alive' => ['yes']},
            {'keep_hostname' => ['yes']},
            {'transport' => ['udp']}
            ]
        }
    ]
}

Parameters within syslog_ng::source:

params

An array of hashes or a single hash. It uses the syntax which is described here.

Defined type: syslog_ng::template

Creates one or more templates in your configuration.

syslog_ng::template {'t_demo_filetemplate':
    params => [
        {
            'type' => 'template',
            'options' => [
                '"$ISODATE $HOST $MSG\n"'
            ]
        },
        {
            'type' => 'template_escape',
            'options' => [
                'no'
            ]
        }
    ]
}

Parameters within syslog_ng::template:

params

An array of hashes or a single hash. It uses the syntax which is described here.

Implementation details

There is a concat::fragment resource in every class or defined type which represents a statement. Because statements need to be defined before they are referenced in the configuration, I use an automatic ordering system. Each type has its own order value, which determines its position in the configuration file. The smaller an order value is, the more likely it will be at the beginning of the file. The interval of these values starts with 0 and are 'strings'. Here is a table, which contains the default order values:

Name Order
syslog_ng::config '5'
syslog_ng::destination '70'
syslog_ng::filter '50'
syslog_ng::log '80'
syslog_ng::options '10'
syslog_ng::parser '40'
syslog_ng::rewrite '30'
syslog_ng::source '60'
syslog_ng::template '20'

The config generation is done by the generate_statement() function in most cases. It is just a wrapper around my statement.rb Ruby module, which does the hard work. The require part may seem quite ugly, but it works well.

Limitations

The module works well with the following Puppet versions:

  • 2.7.9
  • 2.7.13
  • 2.7.17
  • 3.1.0
  • 3.2.3
  • 3.3.1
  • 3.3.2
  • 3.4.0
  • 3.4.3

Tested Ruby versions:

  • 1.8.7
  • 1.9.2

NOTE: The module was tested with Travis with these versions. It may work well on other Puppet or Ruby versions. If that's so, please hit me up.

The module was only tested on Ubuntu Linux (14.04 LTS), but it should work on Debian as well. If you use it on an other platform, please let me know about it!

Development

You can create a development environment from scratch in some seconds by using Docker. Just enter the following command in the root directory of the cloned repo:

# docker build .

I am open to any pull requests, either for bug fixes or feature developments. I cannot stress the significance of tests sufficiently, so please, write some spec tests and update the documentation as well according to your modification.

Note for commiters:

The master branch is a sacred place, do not commit to it directly, we should touch it only using pull requests.

Changelog