Logging via Splunk (6.x)
By default, Metric Insights 6.1.0 deploys a logger service (rsyslog) that collects all logs and saves them to
/opt/mi/log. Optionally, instead of the default, you can use the Splunk logging driver that sends container logs to HTTP Event Collector in Splunk Enterprise and Splunk Cloud.
- For more information on default logging, refer to Where are all those Log Files? (6.0+)
- To learn about Splunk logging options, see Splunk logging driver
To configure logging via Splunk, use the process below to replace the default with the custom Splunk logging driver.
To disable the logger, install Metric Insights application with the option --disable-logger.
$ ./installer.py --disable-logger
The HTTP Event Collector is an endpoint allowing developers to send application events directly to the Splunk software via HTTP or HTTPS using a token-based authentication model.
An Administrator role is required to configure a HEC (HTTP Event Collector).
For each of the Metric Insights services, repeat all the steps described below.
- Log into Splunk Enterprise (or Splunk Cloud)
To configure an HTTP Event Collector:
- Go to Settings menu
- DATA section > select Data inputs
- Click HTTP Event Collector
- Events sent without a Token are rejected.
- Before HEC (HTTP Event Collector) can be properly used, Token usage must be enabled on the instance.
Having accessed the HTTP Event Collector:
- Click [Global Settings]
- Enable Tokens
To do the logging reconfiguration, go to the Metric Insights deployment subfolder and create the
logging.yml file inside
- Determine Splunk HEC URL
http(s)//<splunk_instance>:8088(where 8088 is the HEC default port number determined in Global Settings that can be customized if needed)
- Generate and determine HEC Tokens for each service running inside the Docker Containers
- Even though, it is possible to use one Token for all Docker Container Services, we recommend generating separate Tokens for each Container.
$ cd /opt/mi/config/deployment/ $ nano logging.yml ... $ cat logging.yml version: "3.4" services: web: logging: driver: "splunk" options: splunk-token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx splunk-url: https://<splunk_instance>:8088 splunk-insecureskipverify: "true" labels: web ... dataprocessor: ... seed: ... data-analyzer: ... mysql: ...
$ nano .env $ cat .env | grep COMPOSE_FILE COMPOSE_FILE=docker-compose.web.yml:docker-compose.mysql.yml:docker-compose.dataprocessor.yml:docker-compose.seed.yml:docker-compose.data-analyzer.yml:logging.yml