Configuring a Remote Data Processor (v6+)

A Remote Data Processor (RDP) is a service that is used to connect to data sources that require integrations from a Windows environment and/or run from behind a firewall that sits in between Metric Insights and the data source.

This article includes the following information:

NOTE: Beginning in v6.4.1, Java 17 must be installed on the RDP server.

If you are using an RDP on MI v6.2.5a:

  1. Download the .jar file: https://metricinsights-images.s3.amazonaws.com/article-files/data-processor-0.30.1.jar
  2. Copy the downloaded file to the \Metric Insights\Data Processor\lib directory under the location where the RDP is installed
  3. Replace the data-processor-<version>.jar with the new file
  4. Rename the new file to the old .jar file name

Overview

The following ports are necessary for Metric Insights application and RDP to work correctly:

An RDP service is a node of AKKA cluster. AKKA toolkit is used to create nodes within a cluster and establish communications between the nodes (all nodes must communicate with each other). The main functions of the cluster are:

  • Collecting data with plugins
  • Saving the collected data to a storage

The cluster consists of three node types which communicate with each other via TLS protocol:

  • Seed node - a docker container
  • Local DP (Main node) - a docker container
  • RDP - a Windows service

Seed node is an entry point for RDP and Local DP and is used for internal communication.

Local DP performs the following functions:

  • Communicates with the UI part of the MI application (it contains a REST API which is used for communication with the web container)
  • Communicates with the databases: MySQL Dashboard Database (contains metadata) and MSSQL/MySQL Storage
  • Has a local data collector

RDP communicates with Data Sources behind firewall and Data Sources that use Windows API.

Due to network issues the RDP node may fail to connect to Seed and Local DP nodes (and vice versa), therefore, RDP separates from these nodes into a separate cluster. To fix this situation, manually restart the RDP service on the server.

To view the list of available RDP commands refer to the Commands section.

Configure a Remote Data Processor in MI

1. Access Admin > Collection & Storage > Remote Data Processors

The list page containing all RDPs available in the system opens.

Below the grid,  click [+ New Remote Data Processor].

2. Create a New Remote Data Processor Profile

  1. Enter a Name for your RDP
  2. RDP Host/IP: enter the hostname or the IP address of the machine where the RDP is installed. This address must be accessible for Seed node and Local DP
  3. [Save]

3. Open the Editor of New Remote Data Processor

4. Download and Run RDP Installer on your Server

  1. [Download ZIP File] and install RDP on your server
    • See the following articles regarding RDP installation on Windows and Linux.
  2. You can [Download Config File] data-processor-service.xml separately without downloading the whole installation package, if your configuration needs to be updated or if you are running multiple RDP services
  3. Click [Get Logs] to view the log file, or alternatively access log files manually:
    1. Access dataprocessor container
    2. cd /opt/mi/logs/:
      • application.log - data processor log file
      • insight.log - data collector log file
    3. cd /opt/mi/logs/container/:
      • tomcat-access.log - a list of REST API queries

If you encounter any problems you can troubleshoot your remote data collector setup.

5. Add Source Databases and Plugin Connection Profiles

You can add source Databases and Plugin Connection Profiles that will use the RDP by clicking [+ New Source Database] or [+ New Plugin Connection Profile].

Commands

Check Cluster Membership

  1. Access the dataprocessor container
  2. Execute the command  curl http://localhost:8558/cluster/members that returns the list of nodes in a JSON format