Logging via Splunk (6.x)
By default, Metric Insights 6.1.0 deploys a logger service (rsyslog) that collects all logs and saves them to
/opt/mi/log. Optionally, instead of the default, you can use the Splunk logging driver that sends container logs to HTTP Event Collector in Splunk Enterprise and Splunk Cloud.
- For more information on default logging, refer to Where are all those Log Files? (6.0+)
- To learn about Splunk logging options, see Splunk logging driver
To configure logging via Splunk, use the process below to replace the default with the custom Splunk logging driver.
1. Disable the default logging driver
To disable the logger, install Metric Insights application with the option --disable-logger.
$ ./installer.py --disable-logger
2. Configure the HTTP Event Collector
The HTTP Event Collector is an endpoint allowing developers to send application events directly to the Splunk software via HTTP or HTTPS using a token-based authentication model.
An Administrator role is required to configure a HEC (HTTP Event Collector).
For each of the Metric Insights services, repeat all the steps described below.
- Log into Splunk Enterprise (or Splunk Cloud)
2.1. Access the HTTP Event Collector
To configure an HTTP Event Collector:
- Go to Settings menu
- DATA section > select Data inputs
- Click HTTP Event Collector
2.2. Enable Token use on the instance
- Events sent without a Token are rejected.
- Before HEC (HTTP Event Collector) can be properly used, Token usage must be enabled on the instance.
Having accessed the HTTP Event Collector:
- Click [Global Settings]
- Enable Tokens
2.3. Add a new token for receiving data over HTTP
2.4. Review the settings and submit
- Review: keep the default settings
- Click [Submit] to proceed
2.5. Copy the Splunk Token
- Copy the generated Token Value
- Start Searching
3. Reconfigure logging in Metric Insights Docker Containers
To do the logging reconfiguration, go to the Metric Insights deployment subfolder and create the
logging.yml file inside
- Determine Splunk HEC URL
http(s)//<splunk_instance>:8088(where 8088 is the HEC default port number determined in Global Settings that can be customized if needed)
- Generate and determine HEC Tokens for each service running inside the Docker Containers
- Even though, it is possible to use one Token for all Docker Container Services, we recommend generating separate Tokens for each Container.
3.1. Create logging.yml
$ cd /opt/mi/config/deployment/ $ nano logging.yml ... $ cat logging.yml version: "3.4" services: web: logging: driver: "splunk" options: splunk-token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx splunk-url: https://<splunk_instance>:8088 splunk-insecureskipverify: "true" labels: web ... dataprocessor: ... seed: ... data-analyzer: ... mysql: ...
3.2. Edit the .env deployment config file
$ nano .env $ cat .env | grep COMPOSE_FILE COMPOSE_FILE=docker-compose.web.yml:docker-compose.mysql.yml:docker-compose.dataprocessor.yml:docker-compose.seed.yml:docker-compose.data-analyzer.yml:logging.yml
3.3. Apply the new configuration
$ mi-control up -d Recreating mi_mysql_1 ... Recreating mi_seed_1 ...
4. Check if Metric Insights logs are sent to Splunk
To verify that logging is working correctly:
- Open the Apps menu
- Select Search & Reporting
- Enter your search query
If all configured properly, logs will be displayed below.