Logging via Splunk (6.x)

By default, Metric Insights 6.1.0 deploys a logger service (rsyslog) that collects all logs and saves them to /opt/mi/log. Optionally, instead of the default, you can use the Splunk logging driver that sends container logs to HTTP Event Collector in Splunk Enterprise and Splunk Cloud.

To configure logging via Splunk, use the process below to replace the default with the custom Splunk logging driver.

1. Disable the default logging driver

To disable the logger, install Metric Insights application with the option --disable-logger. 

$ ./installer.py --disable-logger
Click to copy

2. Configure the HTTP Event Collector

The HTTP Event Collector is an endpoint allowing developers to send application events directly to the Splunk software via HTTP or HTTPS using a token-based authentication model.

An Administrator role is required to configure a HEC (HTTP Event Collector).

For each of the Metric Insights services, repeat all the steps described below.

PREREQUISITES:

  • Log into Splunk Enterprise (or Splunk Cloud)

2.1. Access the HTTP Event Collector

To configure an HTTP Event Collector:

  1. Go to Settings menu
  2. DATA section > select Data inputs
  3. Click HTTP Event Collector

2.2. Enable Token use on the instance

NOTE:

  • Events sent without a Token are rejected.
  • Before HEC (HTTP Event Collector) can be properly used, Token usage must be enabled on the instance.

Having accessed the HTTP Event Collector:

  1. Click [Global Settings]
  2. Enable Tokens
  3. Save

2.3. Add a new token for receiving data over HTTP

2.4. Review the settings and submit

  1. Review: keep the default settings
  2. Click [Submit] to proceed

2.5. Copy the Splunk Token

  1. Copy the generated Token Value
  2. Start Searching

3. Reconfigure logging in Metric Insights Docker Containers

To do the logging reconfiguration, go to the Metric Insights deployment subfolder and create the logging.yml file inside /opt/mi/config/deployment

PREREQUISITES:

  1. Determine Splunk HEC URL http(s)//<splunk_instance>:8088 (where 8088 is the HEC default port number determined in Global Settings that can be customized if needed)
  2. Generate and determine HEC Tokens for each service running inside the Docker Containers

NOTE:

  • Even though, it is possible to use one Token for all Docker Container Services, we recommend generating separate Tokens for each Container.

3.1. Create logging.yml 

$ cd /opt/mi/config/deployment/
$ nano logging.yml
...
$ cat logging.yml
version: "3.4"
services:
 
  web:
    logging:
      driver: "splunk"
      options:
        splunk-token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
        splunk-url: https://<splunk_instance>:8088
        splunk-insecureskipverify: "true"
        labels: web
...
 
  dataprocessor:

...

  seed:

...

  data-analyzer:

...
  
  mysql:

...
Click to copy

3.2. Edit the .env deployment config file 

$ nano .env
$ cat .env | grep COMPOSE_FILE 
COMPOSE_FILE=docker-compose.web.yml:docker-compose.mysql.yml:docker-compose.dataprocessor.yml:docker-compose.seed.yml:docker-compose.data-analyzer.yml:logging.yml
Click to copy

3.3. Apply the new configuration 

$ mi-control up -d
Recreating mi_mysql_1 ...
Recreating mi_seed_1 ...
Click to copy

4. Check if Metric Insights logs are sent to Splunk

To verify that logging is working correctly:

  1. Open the Apps menu
  2. Select Search & Reporting
  3. Enter your search query

If all configured properly, logs will be displayed below.