Create Data Dependency

Data Dependencies are typically associated with Triggers in order to ensure that data is available and relevant in the source system. Unlike Schedule or Trigger Dependencies which verify that a particular process has been completed, this dependency involves creating SQL to look at the data for verification.

This article explains how to create Data Dependency from an existing Data Collection Trigger (see Create a Trigger).

Note, that Data Dependency creation is done slightly different when creating it from a Notification Schedule (see Create or Modify a Notification Schedule).

Examples of Data Dependency Checks
  1. Check that data in a Warehouse Customer Order table contains records post midnight of the prior day (indicating that all of the prior day's orders have been loaded).
  2. Verify that an ETL (Extract, Transform, Load) control table specifies that ETL processing has been completed since the last data collection cycle.
  3. Verify that volume in a data table is at least n% of the volume of data on a prior day as a data quality check before loading data.

1. Add a New Data Dependency

Access Admin > Collection & Storage > Data Dependencies

  1. [+ New Data Dependency]
  2. Name: Give the Dependency a descriptive name.
  3. Description: Provide a description for the Dependency.
  4. [Save]

2. Info Tab

  1.  Data Dependency Mode: Select what kind of data the system will receive:
    • If set to "Numeric",  the system expects the SQL statement to return a positive number.
    • If set to "Date",  the system expects the SQL statement to return a DateTime value.
  2. Data Source: Select a Data Source from the drop-down menu.
  3. Query: Provide a fetch command. The hint below will change based on the selected Data Dependency Mode.
  4. Data Dependency Action: Define how to proceed with the Dependency:
    • Run on a schedule, wait until satisfied: Allows to specify how many time a Dependency is checked until it is satisfied.
      • Check Interval: Specify how often the dependency check fetch command should be run.
      • Start/Stop checking Dependency after: Provide a time window during which checks should be performed.
      • Max. check time: Specify a limit to time expended when checking the Dependency. If exceeded, the Data Dependency fails.
    • Run on-demand, abort if not satisfied: The Dependency will be checked as the first step of Trigger processing. If the Dependency is not satisfied, the associated Trigger will abort with error when starting. 
  5. [Validate] your fetch command.
  6. If last_dependency_satisfied was included in the fetch command, provide a Measurement for testing the data dependency logic.
  7. If Validation is successful, [Enable Dependency].

3. Add Data Dependency to the Trigger

Open Trigger Editor and go to Dependencies tab.

  1. [+ New Data Dependency]
  2. Data Dependency: Select the Dependency from the drop-down list.
  3. Must be satisfied during current collection cycle: If the Trigger's interval value is set to "1" of any time unit, except Minute; e.g., 1 Hour, 1 Week, 1 Calendar Year, you can control whether an associated Data Dependency must be satisfied during current collection cycle by accessing the Dependencies tab of the Trigger Editor and editing an existing Dependency.
  4. [Save]

Note:

  • A Dependency can be shared among multiple Data Collection Triggers.
  • Multiple dependencies can be specified for a Trigger and all associated dependencies must be satisfied before the Trigger start working.