Release 6.2.0/6.2.1/6.2.2

Release 6.2.0 is our second major release of 2020 and includes the launch of several new features that extend Search, Reporting and System Administration in exciting new ways. You can look forward to over 30 enhancements and dozens or minor improvements.

Under-the-hood architectural changes include replacing MariaDB 5.6 with MySQL 8. We've also introduced the Patcher, a utility designed to apply patches to multi-node, container orchestrated environments (a la Amazon ECS and Kubernetes).

With Release 6.2.1, we've included major enhancements to Portal Pages, allowing for sync with a Git server and providing a beta release of Advanced Analytical Applications. Content sync with BI tools now supports QlikView, Qlik Sense, Tibco Spotfire, Microsoft Power BI, and SAP BusinessObjects. Reports have been enhanced, managing system backups have been further improved, and Metrics have some minor new features.

Major features in 6.2.0 include:

  1. Intelligent Global Search is now out of beta and ready for production
  2. KPI Visualization provides a chart for highly dimensional data
  3. Tableau Content Autosync allows selected Folders in Tableau to automatically be synced with Metric Insights
  4. Numerous Dataset Report Enhancements continue to round out our Reporting functionality
    1. Charting enhancements to line, bar, area, pie and bubble charts
    2. Other enhancements to tables, pivots and more
  5. Design improvements to our user interface enhance the look and feel
  6. User Maps can now include wildcards, to support new ways of configuring mappings and providing access to all values within a certain column/field
  7. Enhancements to Data Collection Triggers and Notification Schedules
  8. System Backups can now be setup and maintained directly in the application UI, without the need for a CLI tool
  9. The new Patcher utility for applying hot fixes
  10. Several other improvements
  11. An updated Mobile app
  12. Finally, there are some noteworthy changes to our Security Model

Major features in 6.2.1 include:

  1. Advanced Portal Pages with Git Sync (Beta), Advanced Analytical Applications (Beta) and Global Tags
  2. Automatic content sync now supports QlikView, Qlik Sense, Tibco Spotfire, Microsoft Power BI and IBM Cognos
  3. Dataset Report enhancements to make reordering measures possible and cover several additional small improvements
  4. Bursting enhancements (primarily for Admins and Power Users)
  5. Data Collection
  6. System Backup management and configuration
  7. System Administration & Logging improvements covering access request functionality and additional content usage logging
  8. Several additions to the security model, making it even more robust

 

6.2.0 marks the launch of Intelligent Global Search—it has been designed from the ground-up to function like the leading search engines with core architectural similarities like crawling, indexing and scoring. You can search across both content within Metric Insights, external references to objects in BI tools, and Datasets that include links to any external system—you choose. 

The engine includes its own query language that provides a robust set of inline filters on metadata. In the example above, Tibco Spotfire Data Sources are being excluded via "datasource:-spotfire"

For all content in Metric Insights, we apply security/access rules so that users are only able to view to which they have at least View Access Permission. For content from external systems, users are governed by that system's security.

Note for System Admins: Intelligent Global Search is made possible via the data-analyzer, an independent service that runs in its own lightweight Docker container.

KPI Visualization

The KPI Visualization provides an entirely new way to analyze dimensional data with a Dataset Report. You can now expand use-cases beyond the current two Dimension limit within Metrics since the KPI Visualization is able to handle several of each. In the example above, you see five Dimensions and four Measures (KPIs)

Content Autosync with BI Tools

Rejoice! Content can now be automatically synced between Tableau and Metric Insights. The sync is omnidirectional: content that exists within a Tableau Folder can be set to sync to a Metric Insights Category. As new Workbooks are added, Metric Insights will automatically create External Reports (based on Templates) of the same name. Learn more about Autosync with Tableau.

In 6.2.1, we've added support for:

  • QlikView
  • Qlik Sense
  • Tibco Spotfire
  • Microsoft Power BI
  • IBM Cognos

Dataset Report Charting Enhancements

Line, bar, area, pie and bubble charts have been improved to be more usable and and address a new set of use-cases. Enhancements include:

  1. Change segment colors for multi-series line, bar, and area charts as well as pie charts
  2. Set the "top N" highest or lowest values for multiline or pie charts
  3. Sorting options for Bar Charts
  4. Conditional coloring for bubble charts
  5. For a Dataset Report that has a multiline datapoint on its chart, on hover, show the corresponding single value
  6. [6.2.1] Measures can now be re-ordered

Other Dataset Report Enhancements

  1. A "% of Total" option is now available for numeric columns in tables
  2. Pivot Tables will now validate numeric data in Conditional Formatting rules
  3. An embedded External Report can now be associated directly with Filters in a Dataset Report
  4. Conditional logic has been added when downloading CSV on the Report Viewer. If possible, Report Filters are ignored—this can make downloads significantly faster.
  5. "Show on Tiles" setting defaults have changed—on the Dataset Report Editor > Content tab > Edit (a component) > "Show on Tile" setting, the default was changed to "table" instead of "row count". Beneath it, the "When Report has..."  text default changed from "exceptions" to "rows"
  6. [6.2.1] Set alignment options for "Table contains no rows" text
  7. [6.2.1] The source column name is now displayed in the Chart slide-out
  8. [6.2.1] Additional comparison intervals are available in Filter rules (for data tables in Reports, and in the Dataset Viewer)
  9. [6.2.1] In Tile and Tile Preview, display External Report metadata

Design Enhancements

  1. The action bar at the top of each page has been completely redesigned to be more space efficient, elegant, and to accomodate searches.
  2. The "alert" icon has been updated to a bell, following the interface design pattern established in myriad other tools. Look out for the change in Element Viewers and Tile Previews specifically.
  3. All buttons, drop-downs, radios, and checkboxes have been updated to be resolution independent and follow the color scheme of a particular theme (if you see custom branded colors in Metric Insights, you are using a theme). Button sizes and font sizes have been made more consistent. This is the first batch in a series of UI updates that will continue in 6.3.0

User Maps With Wildcards

It is now possible to define wildcard values in User Maps, enabling some powerful new options for how mappings are setup.

Data Collection Triggers and Notification Schedules

  1. Data Collection Triggers have been enhanced to:
    1. Include a new [Trigger Now] option for Dimensions that were not successfully collected...
    2. Allow special characters in Trigger descriptions
    3. Include Datasets in list of 'enabled elements'
    4. List Datasets among other associated objects
    5. Be directly associated with a Data Source if the Trigger is being used to refresh an object list (for example, a daily Trigger is be used to refresh objects from Tableau)
    6. [6.2.1] Include an option for for "Abort" that will expire existing run(s)
    7. [6.2.1] Throw an error message for each element in the Trigger Status response from our REST API
  2. Notification Schedules now keep run history (logs) by default

System Backup

Our ongoing efforts to move CLI-based utilities into the core application continue. Navigate to Admin > System > System Backups and try setting up a backup. All capabilities previously available in the mi-app-backup utility are now available in the UI—the utility remains, but there is no longer any need to run from a terminal. Everything can be configured in the UI.

[6.2.1] An administrator can now choose to skip Datasets when creating a backup (useful for cases where the Datasets may be very large and you'd like to backup that server with another process), and all previous backups are now listed on a new "Backups" tab where they can be manually deleted as needed (typically deletion is handled via the retention settings when the backup is scheduled)

The Patcher

Metric Insights 6.2.0 introduces the Patcher, a utility designed specifically for Container Orchestrated environments that require code changes (often referred to as hotfixes). Orchestrated Docker environments present unique challenges in terms of hot fixing that do not exist in traditional deployments (whether virtual machine, bare metal, or simple Docker setups). While traditional environments allow for direct filesystem changes that will persist if the machine/image is restarted, Docker images in orchestrated environments pull a fresh image from the Docker Registry on restart—any code changes to the container are lost.

To solve this, we created a utility that will automatically check for required patches any time the container is started, move the files to the correct location, and proceed with normal operation. Any time the container is restarted, the patches are re-applied.

When a new Docker image is built for a future release (for example, 6.2.1), these patches are baked into the production Docker images and this process is no longer necessary until the next set of new patches need to be applied, which will later be baked into the next release (6.2.2)

Learn more about how the Patcher works.

Bulk Change

  1. Datasets can new be modified en masse via a new Bulk Change capability (as has existed on Elements for years). The following can be changed:
    1. Enable/Disable
    2. Change Category
    3. Change Data Collection Trigger
  2. [6.2.1] Bulk Change for Elements has been improved:
    1. Change Technical Owner
    2. Add new Tags to the existing Tags list (previously, you could only replace all existing Tags)

[6.2.1] Advanced Portal Pages (Beta)

Advanced Portal Pages (currently in Beta) introduce the functionality needed to develop single-page JavaScript apps via React, Angular or any other JS framework—the key to enabling this was the addition of syncing with a version control system (Git) and the option to tie assets directly to a Portal Page (rather than keeping them as global objects that can be used across pages—this is still supported, but you now have greater flexibility).

Finally, we introduced an incredibly powerful new concept: Advanced Analytical Applications—this new set of functionality opens up an infinite number of possibilities.  Powered by a new object called Entities, Advanced Analytical Applications allow for the storage of arbitrary data in our system database, combined with our API, our content access and security model, and authentication, you can do just about anything. Can't build a Dashboard in your favorite BI tool? No problem, build a custom application within Metric Insights using Entities and our existing API. Note: experienced JavaScript engineer required.

In summary:

  1. Git can now be used as the source for Portal Page code (Beta; sync with GitHub, GitLab, or any self-hosted Git server)
  2. Portal Page Assets can now be directly tied to a page via a new Assets tab on the Portal Page Editor
  3. Advanced Analytical Applications (Beta) allow the creation of supporting data tables and other objects that can be used to construction advanced applications
  4. Global Tags can be used across multiple Portal Page Templates
  5. Another small enhancement: Portal Pages now support the <title> (in other words, the name of the Portal Page will be used in browser tabs and browser bookmarks)

[6.2.1] Bursting

  1. Bursts can now be filtered by their current status, and that status is displayed in the Burst Editor
  2. All Elements now include the ability to create Burst via the Associations tab in the Element Editor
  3. An option to not include PDF with External Content (when being distributed via a Burst)

[6.2.1] Metrics

  1. Metrics now include the option to display average value on a Metric tile, this is available via the Charting tab and includes the ability to define the prefix (it will default to "Avg")
  2. Metrics now have a smarter default when defining a "Recollect after" date

[6.2.1] System Administration, Backup & Logging

Content Access & Access Requests

  1. Access Request messages are now supported by all Element types & Datasets; learn more about access requests
  2. The system will automatically identify when a user has a pending request that's been filled via Active Directory / LDAP group changes (previously, these pending requests would still be listed even though the user may have been granted access to the appropriate Active Directory / LDAP)
  3. A custom "no access / access denied" message can now be set to override the system default via a new Config Variable: ELEMENT_NO_ACCESS_MESSAGE

System Monitoring

  1. We've removed information about free/total space for Data Storage objects as it was difficult to retrieve accurate numbers; these external Databases should be monitored separately by internal IT operations or a DB admin, rather than the MI application
  2. The Nodes tab on the Status Monitor has been adjusted, it now lists volumes, with details about disk space utilized available in a popup (click on [N] Volumes)

Logging

  1. Logging has been added for Data Storage initialization
  2. We've added system tables to summarize daily access counts by element, user, and access type
  3. Searches on mobile are now logged

Authentication

  1. Support for Kerberos 5 has been added

[6.2.1] DevOps & Installation

6.2.1 includes several new tools, configurations and enhancements to ease installation, deployment and administration.

  1. We are shipping with additional CLI tools
    1. AWS CLI is in the Web container
    2. All containers include vim, ping, telnet, wget to all containers
  2. Installer:
    1. New flag --high-load larger environments (5,000+ users), use this based on consultation from our Support team
    2. ECS and Kubernetes support for --timezone
    3. Support for Red Hat OpenShift deployment
    4. Support for Ubuntu 20.04
    5. Redis is now the default deployment option for Amazon ECS configurations
  3. System logs from the web container can now be saved to /opt/mi/log/mi.system
  4. The Docker registry can now be redefined to one of your choosing

Mobile

The Mobile release paired with 6.2.0 is app version v3.0.x—both iOS and Android are available on their respective app stores.

This is a major update to the Metric Insights mobile app! All Element viewers are now native code rather than webpages. The app now depends more heavily API's available in the server, therefore we recommend server version 6.1.3 or later.

  1. The Metric and Report Viewers are in the final stages of being rebuilt and redesigned. The Metric Viewer should now be easier to interact with and faster.
  2. Announcements are now supported and will display on mobile (these are setup via the Announcements feature in the server app).
  3. External Reports that are set to display as a PDF are now supported.
  4. External Report images can be zoomed in / magnified. Rotate the phone to view images in landscape orientation.
  5. Dataset Reports are now displayed in the Executive Brief homepage view.
  6. Several other fixes and small improvements

Plugins

In addition to automatic content sync, we've improved several plugins:

  1. [6.2.1] SAP BusinessObjects
    1. Lumira documents are now included in the object Hierarchy
    2. Excel workbooks can now be pulled in from folders
  2. [6.2.1] Tableau: Add Level Filter and Usage Stats to Tableau Plugin
  3. [6.2.1] Amazon S3: now supports image retrieval (getImage)

Security

  1. Admins are warned when creating a new Group or Regular/Power User that has not been granted any Privileges or Permissions. While the Group or user account can be saved, the new user or members of the Group will not be able to do any meaningful work in the system or view any content until some Privileges and Permissions have been granted.
  2. The "Manage Favorites Folder" and "Rename Favorites" Privileges have been merged into the new "Create Favorites" Privilege. A user with this Privilege can create Favorites, manage content and update related settings using the edit icon, and rename the default My Favorites folder. Favorites are now for the exclusive use of its creator and not for sharing content with others.
  3. A new "Create Burst" Privilege has been added to allow Regular and Power users to define a Burst and subscribe members of Groups to which the user belongs and to individuals within those Groups to the Burst. A new associated Extended Security Privilege allows Power Users to subscribe or grant Burst Access to any User or Group.
  4. On the User Access Request tab, a Power User requires Edit Access to the requested element in order to accept/deny the request. The grid's legend indicates "No Edit Access" in red.
  5. [6.2.1] A new setting "Available to all content creators" on the Data Storage Editor allows the Storage to be used by all Admins and Power Users when creating new Datasets in addition to the Default Data Storage set for each instance.  The Default and these designated Storages are found on the User Editor > Power Users tab and in Data Store drop-down lists.
  6. [6.2.1] Provided a new Privilege:  “Sort Category List” that allows a Power User to view all entries in the Categories list with an activated function on the Name column and the ability to drag/drop for custom sorting.  Any Category to which the Power User does not have Edit Access is shaded grey.  All Access Groups automatically receive this Privilege.
  7. [6.2.1] Added a new  “Click thru to External Visualization via link” Privilege that allows a user to access the BI tool’s report from the Viewers of External Reports, Dataset Reports, Other External Content elements and Mobile External Reports  All Access Groups automatically receive this Privilege.  It is also honored on iOS/Android Mobile Apps.
  8. [6.2.1] Selected Privileges are granted to all new or migrated All Access Groups.  The list is updated as new Privileges are added.  These Privileges are NOT automatically made available to the Default Group that is configured manually for each new instance of Metric Insights.

Other Improvements

  1. A column reset button now exists on all grids, clicking will reset each column to its default width
  2. Subfolders can now be copied
  3. When saving changes that make updates to the security model, the system will inform user that it could take a while
  4. Elements:
    1. Add filter for Trigger; display Trigger for Elements with Dataset as source
    2. On the Associations tab, list the Bursts that contain the element
  5. The Simple SAML library used for Single Sign-On (SSO) authentication has been updated to version 1.18.5
  6. The home icon will now return to the Default homepage/Portal Page configured via a user's preferences or defined via Portal Page settings
  7. Custom Scripts:
    1. Display additional metadata in their List page
    2. Are included in regular garbage collection
  8. [6.2.1] The option skip Tile Previews (on the Homepage) has been split out into two distinct variables, aptly named SKIP_PREVIEW_FOR_EXTERNAL_CONTENT and SKIP_PREVIEW_FOR_EXTERNAL_REPORTS—these control whether the Tile preview is displayed for their respective Element types.
  9. [6.2.1] Custom Measurement Time Calc Commands (now primarily used for calculating "Measurement Time" in Datasets) have been revised to remove legacy functionality—they should new be more straightforward to understand

Issues Fixed

This is a small selection of issues among hundreds of bugs that are fixed in each release.

  1. Dataset Reports: Corrected placement of Section when scrolling Table with a huge amount of columns
  2. In the new Element creation process, if a new Category is required, the application presents the New Category popup. When the new Category is saved, it will keep you in the Element creation workflow (previously it would take you to the Category Editor, removing you from the Element creation process)
  3. Portal Page Assets no longer have view access restrictions—this was causing problems when a user had view access to a Portal Page, but not view access to the Portal Page Asset Folder. Because Assets are intended to be viewed, we no longer restrict them. Only managing/editing a Folder can be restricted.
  4. [6.2.1] Metrics: Metric legends will now correctly display up to 15 overlaid Metrics
  5. [6.2.1] Admin Monitoring Service: For each container/service, the Total Memory incorrectly showed the host total, not the container total—this was fixed to show the correct totals