Establish connectivity to File Data

This article describes how to use a File Data Plugin to create a connection profile with Metric Insights.

The file that serves as a source of data can be located on:

  • MI server or a server where the Remote Data Collector is installed (local file system)
  • ftp/sftp server (over SSH)
  • S3 (Amazon file server)

[Release 6.x]: 'Remote Data Collector' renamed to 'Remote Data Processor'

1. Access Admin > Data Sources

At the bottom of the screen click [+ New Data Source]

The Select the Type of New Data Source pop-up opens.

2. Select "Other" Data Source Type and choose "File Data" from the drop-down list

Proceed with creating a Data Source by moving to the Next step.

3. Provide File Data Parameters

3.1. Required File Data Parameters

Specify the Required Parameters:

  1. Data Source Name: will default but you may modify it
  2. Root location: URL of the directory containing files from which data will be extracted
  3. Username/Plugin Password:
    • Are not required when the local file system is used
    • If access to the ftp/sftp server is password-protected, enter the above credentials in the corresponding fields
    • S3 server requires accessKey and secretKey that have to be provided in the Username and Plugin Password fields. respectively

3.2. Optional File Data Parameters (Variables)

You may include the following Optional File Data Parameters:

  1. Bucket Pattern (S3): is used to filter high-level directories (buckets) on the Amazon S3 server
    1. Based on this filter, Bucket Objects will be collected by the File Data Plugin
  2. CSV Delimiter: if values in your files are separated by delimiters other than the default comma (","), you can change the CSV Delimiter Variable to the required value
  3. Includes headers: allowed values are "true"/"false", "yes"/"no", "1"/"0" 
    1. If your file contains headers and you would like to include them in your File Data Report, set Variable values to "true", "yes", "1"
  4. Files filter: can be employed to collect files according to the required value into the list of File Data Reports that are used to build other elements in Metric Insights

4. Save your entries and Test Connection

If your connection is successful, you may move on to Advanced settings.

5. Advanced Configuration

  1. Use Remote Data Collector: is set to "no" by default
    1. If required, switch to "yes" and add a Remote Data Collector by clicking [+New Remote Collector] under Remote Collectors settings
  2. Generate Object List 
    • automatically: all Reports are going to be fetched by the system
    • manually: Reports may be added one-by-one or via CSV file
  3. Object List Refresh Trigger: from the drop-down, select the Trigger that will be used to fetch data via the File Data plugin
  4. Object Selection Method: specify how File Data Reports will be fetched
  5. Optionally, state the maximum number of concurrent Threads to utilize during data and/or image fetch to be used in background processing when the system updates Reports for this Data Source
    1.  If you do not specify any value for this setting, batch data collection processing will be single-threaded

6. Obtain a list of External Reports

  1. Go to File Data Objects tab
  2. To obtain a list of File Data External Reports, click the [Refresh list] button

7. Other Settings

  1. You can create Datasets or Elements directly from the respective tabs
  2. Click Permissions to assign them to Groups or Power Users