Configure Data Dependencies for a Data Collection Trigger
Data Dependencies are typically associated with a Data Collection Trigger in order to ensure that data is available and current in the source system through the collection cycle specified in that Dependency.
- An enabled Data Collection Trigger
Examples of Data Dependency Checks:
- Check that data in a Warehouse Customer Order table contains records post midnight of the prior day (which indicates that all of the prior day's orders have been loaded).
- Verify that an ETL control table specifies that ETL processing has been completed since the last data collection cycle.
- Verify that volume in a data table is at least X% of the volume of data in a prior day as a data qualify check before loading data.
1. Add a New Data Dependency
- Click [+ New Data Dependency] in the Data Collection Trigger Editor Dependencies tab
- The Add Data Dependency pop-up opens
- From the drop-down list select the Add New Data Dependency option.
2. Define the basics for the New Data Dependency
After the new Data Dependency is saved, the Data Dependency Editor opens.
3. Enter Data Dependency Specifications
- Make sure your Data Dependency is 'enabled'
- In 5.1+ a new Data Dependency Mode field was allow a simplified SQL.
- If set to 'Numeric", the system expects the same SQL statement as in prior versions, see Hint on right of statement box above.
- If set to 'Date' then your SQL hint will change to: "Specify a statement which will return a single column which contains a datetime value. The dependency will be satisfied if the value returned falls after the datetime on which the Dependency was last satisfied. You may optionally also specify the Table Name for the query."
- Specify the Data Source for the Data Dependency
- Provide your fetch command. In Version 5.1+, the hint to the right will changed based on the Data Dependency Mode selected above
- Validate your fetch command
3.1. Specify Last Dependency substitution
If :last_dependency_satisfied was included in the fetch command, provide a value for testing the data dependency logic
4. Below the statement specify when and how often to perform the Data Dependency check
- Specify how often the dependency check fetch command should be run.
- Provide a time window during which checks should be performed.
- Specify a time period after which the dependency check should be expired.
Enable Dependency at the top right corner of the screen.
This Data Dependency can now be added to the Data Collection Trigger
- A given dependency can be shared among multiple Data Collection Triggers
- Multiple dependencies can be specified for a Data Collection Trigger and all associated dependencies must be satisfied before the data collection trigger starts
Control when Data Dependency of Data Collection Trigger is Satisfied
If your Data Collection Trigger's interval is an hour or longer, you can control the circumstances under which an associated Data Dependency must be satisfied by accessing the Data Collection Trigger Editor and editing an existing dependency.