Building Reliability into Data Collection

This article discusses methods to ensure that Data Collection is reliable.

1. External Triggers

Extract, Transform, Load (ETL) tools incorporate data checks that can halt ETL if defined rules fail.  These standards can be automatically incorporated into Metric Insights by defining a Data Collection Trigger as “external”.  External triggers become event-based rather than scheduled.  The final portion of ETL scripting can include a cURL command that notifies Metric Insights of ETL completion and signifies that the trigger can begin its update of all related Datasets, User Maps, Dimensions, and elements.

2. Data and Trigger Dependencies

Data Collection Triggers in Metric Insights can be gated through data and trigger dependencies.  

Data Dependencies are true/false conditions running on recurring frequency.  Common data dependencies check for row counts, most recent dates, and days of the week.  

Trigger dependencies sequence data collection.  You may elect to have short-running triggers complete before long-running triggers, or triggers which produce upstream data objects complete prior to downstream objects.  

These dependencies can be combined such that both Data Dependencies are satisfied, and superordinate triggers complete before a defined trigger initiates.

Scheduled triggers run as soon as all dependencies are satisfied within a given collection cycle; e.g., daily, weekly, monthly.  

Externally defined triggers will return an error if an initiation request is submitted from any script prior to dependencies being satisfied.

More information:

Video Tutorial:  https://vimeo.com/203253613

3. Managing Trigger Problems

Use a distribution list for the purposes of problem notifications.  

Notifications surrounding trigger problems can be queued for the following reasons:

  • Trigger is in Error Status
  • Trigger exceeds configured run time
  • Trigger fails to start by specified time

4. Building Reliability into Notifications (Bursts, Digests, and Alerts)

4.1. Burst only refreshed tiles:

The single-most significant method for controlling distributions is through the data collection pipeline.  By avoiding spurious data collection, you can prevent “stale” information from distribution.  

Set the “Send only what’s been updated since last email” option to active within a burst editor.  By activating this control, Bursts examine all elements in their payload but only distribute those elements which have been updated since the last time distribution ran.  This could result in partial payload distribution for bursts containing multiple elements on independent triggers, but avoids distribution of any tiles that have not been updated.  Digests and Immediate Alerts function this way automatically as per design.

4.2. Notification Dependencies:

Notification Schedules can incorporate both trigger and data collection dependencies.  By establishing a trigger hierarchy, a Notification Schedule can be set to activate distribution only after the final trigger in a chain is completed. 

By building true/false Data Dependencies , a Notification Schedule can be activated only after conditions are satisfied.  

All configured trigger and Data Dependencies must be satisfied in order for a Notification Schedule to be activated.

More Information:

5. Managing Notification Schedule Errors:

  • Individual Bursts that encounter errors can distribute logs to either a distribution list or to a Burst owner.
  • Enable “Keep History” for Notification Schedules to review subscriber distribution history (who was sent which tiles when).
  • Notifications can expire after a configured time period.  When breached, generate an email to a distribution list to review.
  • Metric Insights uses multiple threads for all distributions. The system configuration parameter, (NOTIFICATION_SCHEDULE_THREADS_NUMBER), should  only be increased from the default value as directed by MI Support.  The default value of 5 represents the capacity to send five distinct email messages by the amount of time it takes to generate one complete Burst package.  Individual Burst message times vary widely depending on customizations, responsiveness of external platforms and servers, user mapped and row level secure content.
  • The Status Monitor should normally show processing of queued email.  The most rapid distribution cycle (immediate) runs every three minutes.  An ever increasing number of queued email could indicate a problem with ability to communicate with enterprise email servers.
  • Consider the overall size of a Burst.
  • Consider the need to whitelist Metric Insights to the mail server.