Configure Data Dependencies for a Data Collection Trigger

Data Dependencies are typically associated with a Data Collection Trigger in order to ensure that data is available and relevant in the source system through the collection cycle specified in that Dependency.

PREREQUISITES:

Examples of Data Dependency Checks:

  1. Check that data in a Warehouse Customer Order table contains records post midnight of the prior day (indicating that all of the prior day's orders have been loaded).
  2. Verify that an ETL control table specifies that ETL processing has been completed since the last data collection cycle.
  3. Verify that volume in a data table is at least n% of the volume of data in a prior day as a data qualify check before loading data.

Video Tutorial

1. Add a New Data Dependency: Data Collection Trigger Editor > Dependencies Tab

  1. Click [+ New Data Dependency] on the Dependencies tab of the Data Collection Trigger Editor
  2. In the Add Data Dependency pop-up:
    • Select the Add New Data Dependency option from the drop-down list

Alternatively, access Admin > Collection & Storage > Data Dependencies > [+ New Data Dependency] and proceed to enter Data Dependency specifications.

2. Define the Basics for the New Data Dependency

After the new Data Dependency is saved, the Data Dependency Editor opens.

3. Info Tab

  1. Make sure the Data Dependency is 'enabled'
  2. Select Data Dependency Mode:
    • If set to "Numeric",  the system expects the same SQL statement as in prior versions, see Hint on right of statement box above
    • If set to "Date", the SQL hint will change to: "Specify a statement which will return a single column which contains a datetime value. The dependency will be satisfied if the value returned falls after the datetime on which the Dependency was last satisfied. You may optionally also specify the Table Name for the query."
  3. Specify the Data Source for the Data Dependency
  4. Provide a fetch command. The hint to the right will change based on the Data Dependency Mode selected above
  5. Configure Data Dependency Action:
    • If set to "Wait until satisfied": an associated Data Collection Trigger or Notification Schedule will wait until the dependency is satisfied before starting
    • If set to "Abort if not satisfied": the dependency will be checked as the first step of Data Collection Trigger / Notification Schedule processing. If the dependency is not satisfied, an associated Data Collection Trigger or Notification Schedule will abort with error when starting  
  6. [Validate] your fetch command
  7. If :last_dependency_satisfied was included in the fetch command, provide a value for testing the data dependency logic

If the Data Dependency Action is set to "Wait until satisfied", proceed to specify the time and frequency of the Data Dependency check, otherwise, proceed to add the Data Dependency to a Data Collection Trigger.

4. Specify When and How Often to Perform the Data Dependency Check

  1. Check Interval: specify how often the dependency check fetch command should be run
  2. Provide a time window during which checks should be performed
  3. Max. check time: specify a time period after which the dependency check should be expired
  4. [Enable Dependency] at the top right corner of the page

5. Add Data Dependency to the Data Collection Trigger

Note:

  • A given dependency can be shared among multiple Data Collection Triggers
  • Multiple dependencies can be specified for a Data Collection Trigger and all associated dependencies must be satisfied before the Data Collection Trigger starts

6. Control when Data Dependency of Data Collection Trigger is Satisfied

If the Data Collection Trigger's interval value is set to "1" of any time unit, except Minute; e.g., 1 Hour, 1 Week, 1 Calendar Year, you can control whether an associated Data Dependency must be satisfied during current collection cycle by accessing the Dependencies tab of the Data Collection Trigger Editor and editing an existing dependency.