Release 6.3.0

Metric Insights 6.3.0 is our major release for 2021 — as with 6.2.5, emphasis in 6.3.0 is on data governance. This release introduces Publishing Workflows, the Content Center, extended capabilities of Lineage, Identity Profiles, and  other notable features and essential improvements to our product.

v6.3.0 is our first release that requires a system license key. You will be prompted to enter a key once installation is complete and you enter the UI — you must be an Admin to enter a license key.

Major features in 6.3.0 (released September 13, 2021)

  1. Publishing Workflows is a new feature that optimizes and simplifies content creation, management, and supervision for Power Users.
  2. Content Center has been designed as a centralized location that provides access to Publishing, Engagement, and Access Requests.
  3. Identity Profiles are a way to store user credentials outside Metric Insights by means of such security access and management tools as CyberArk, which we start supporting in this version.
  4. v6.3.0 also marks the introduction of License Keys.
  5. Lineage has become more robust and user-friendly, now also offering support for Legacy Reports.
  6. We have included improvements to our major Plugins, such as Tableau and MicroStrategy.
  7. Both our "regular" and Governed Tags have been expanded to provide enhanced usability and flexibility.
  8. Search entities have been expanded to support User Maps. Search indexes update can now be stopped at any time in one click.
  9. Custom Fields has become a more commonly-used feature since its introduction in v6.2.3. We continue to extend its capabilities to support more user scenarios.
  10. Migration Utility can perform not only Import and Export operations, but also Rollback to allow you to revert back the former.
  11. Our Security model has been further upgraded and improved to support all the new functionality.

The primary Docker images used for standard deployments are:

  1. web - docker.metricinsights.com/metric-insights/release_6_3/158
  2. dataprocessor - docker.metricinsights.com/dataprocessor/release_6_3/7
  3. seed - docker.metricinsights.com/seed/release_6_3/3
  4. data-analyzer - docker.metricinsights.com/data-analyzer/release_6_3/33
  5. monitoring - docker.metricinsights.com/monitoring/release_6_3/36

Optional images:

  1. logger  - docker.metricinsights.com/logger/release_6_3/36
  2. mysql - docker.metricinsights.com/mysql:release_6_3

Publishing Workflows

Publishing Workflows allows Admins and Power Users to manage and supervise content creation in Metric Insights. The functionality helps ensure that new content has been validated by multiple stakeholders and is, therefore, trustworthy, including accurate metadata and information. 

Metric Insights v6.3.0 is shipped with three Publishing Workflows that include default Stage configurations, depending on the number of stakeholders involved:

  • 1-Party Review & Certification
  • 2-Party Review & Certification
  • 3-Party Review & Certification

The defaults are highly flexible and customizable, allowing users to easily build a Workflow to meet a variety of needs. 

For example, the 2-Party Review & Certification can be a perfect fit for sales reporting that involves two Groups: Sales Report Developers and Sales. This assumes that the former is responsible for building Reports in an external application and recreating them in Metric Insights and the latter is in charge of verifying that the data in the Report is correct and that the Report is ready for publishing and wider business visibility.

Power Users responsible for configuring Publishing Workflows must be granted the Create Publishing Workflows Privilege. Those in charge of content verification must have the Use Content Center privilege. See more under Security section.

Content Center

Content Center provides centralized content and access management from a single page. Power User Access to the Content Center is controlled by the Use Content Center privilege. Regular Users cannot access this page.

We plan to add functionality and improvements to the Content Center in future releases by including more operationally focused views into Metric Insights content. Version 6.3.0 ships with the following tabs: 

  1. Publishing provides Admins and Power Users to manage and supervise content creation in Metric Insights. The functionality helps ensure that new content has been validated by multiple stakeholders and is therefore trustworthy, including accurate metadata and information. See Create a 2-Party Publishing Workflow
  2. Engagement supports better understanding of content usage, how users interact with it, and how active they are. This provides insight into unused content and inactive users. See Track Engagement via Content Center
  3. User Access Requests are now available, not only from Users & Groups under Admin menu, but also from Content Center. It shows all elements to which individual Users have requested Permission to Access and is visible to all Admins and Power Users with the Privilege to grant access. See Access Request Security

Identity Profiles

Beginning v6.3.0, Metric Insights introduces a new way of authorization for Data Sources called Identity Profiles. Identity Profiles allow for storing user credentials outside Metric Insights using various security access and management tools. In v6.3.0, we are introducing support for CyberArk

The basic working principle follows. Each time credential retrieval is triggered, MI requests credentials from CyberArk. This happens when:

  • Data Source's object list is refreshed
  • A new element is created from Data Source
  • Content is distributed on a Notification Schedule via bursting
  • Data collection is initiated by Data Collection Trigger

This article describes how to configure CyberArk authentication for Data Source. Details on Identity Profile security can be found here

License Keys

Starting the release of v6.3.0, Metric Insights has implemented a license check. If your organization is out of compliance with the terms of your license, every page will display the Update License popup to all users of the system. 

The license can be updated either by using the LICENSE_KEY variable or from the popup itself. See instructions here.

For license-related questions, you can always contact our support at [email protected] and we will be happy to help.

Lineage

In v6.3.0, we have implemented collection and storage of Snowflake metadata, which allows for tracking of object relations outside Metric Insights and reflecting them in Lineage. 

This is beta functionality, and you can contact [email protected] to participate in beta testing.

  • The biggest change from the UI perspective is the new alignment of Objects on the Diagram: Objects of the same type are aligned at the same level. 
  • “Lineage Diagram” popup has been renamed to “Lineage”. 
  • The Burst List now has Lineage too. 
  • Lineage includes icons of the most widely-known BI tools used as Data Sources. 
  • Legacy Reports are now displayed on Lineage. We are planning to discontinue support for these reports in 2022, so this is the most convenient way to track dependencies between Legacy Reports and other Objects in the system and make informed decisions on their importance. 

Plugin Improvements

Tableau

  • Tableau Data Source profiles now include a Base URL parameter. When constructing the URL for a Tableau object, this parameter will be used for the hostname portion. The full URL is constructed based on the pattern defined in a plugin’s URL Template. 
  • Two optional parameters, Token and Token name, have been added to Tableau Data Source profiles.
  • Visual Editor for Tableau and MIQL have been extended with a  formatted date format to be able to format original Tableau datetime columns. 
  • Admins can select the type of API for obtaining Tableau data, images, and PDFs that depend on the auth type. Web API supports Trusted Auth or User/Pass. REST API allows for using User/Pass or Personal Access Tokens. This can be controlled by the loadingStrategy parameter with the two corresponding values, REST and WEB. WEB is default.
  • Our “can’t connect” error messages for Tableau have become more informative, providing more details on the reason for failure.

MicroStrategy

  • When auto syncing content with MicroStrategy, Metric Insights is able to pull Description and Technical Owners in the exact manner as they appear in the BI tool. Note that the user with the same name as in MicroStrategy must exist in MI to make that happen.
  • Another MicroStrategy Plugin improvement is that Content auto-sync is working not only with Documents as external reference objects, but also with Graphs and Reports.
  • Starting this version, we no longer use MicroStrategy Task API; now it’s only Web API and REST API.

Other Plugin Improvements

  • We have redesigned the Graphite plugin to provide support for standard data collection functionality. 
  • Local Filesystem Plugin supports the use of files as Other External Content. Viewers of Other External Content can now show a real preview of files with .xlsx and .xls extensions.
  • Zendesk Explore allows two authentication types: standard and OAuth.
  • Included a warning message stating that assigning a Group is a must for MI Portal Page Plugin.
  • A new parameter, numberOfRetries, has been added for Google Sheets Plugin. with a default value of "3". Available to modify under Plugin Config.
  • Web Services Plugin has been updated to support JSON format. JSON files can now be parsed to get metadata. Two formats are supported: 
    • Root is an array
    • Root object with a single field that is an array

Tags

  • Tag popups can now display Object type and a number of Objects as well as Tiles for dimensioned Metrics, Reports, and External Reports associated with the Tag; e.g., 10 Tiles, 5 Tableau Objects, 2 Superset Objects, etc.
  • The Associations tab of Tag Editor displays all Objects where the Tag is being used.  This provides Power Users more understanding of the Tag usage. Objects that the Power User has permission to edit display as links.
  • Governed Tags now can be assigned directly from Tag Type Editors, not only by adding Tag Types to Governed Tags through Tag Editor. 
  • A popup for Governed Tags displays all Governed Tags associated with the Object. Tags are separated into sections depending on Tag Type. 
  • Users can abort the update of search indexes by clicking Cancel, a capability that was not available in previous versions. 
  • Users can easily check when Search indexes were last updated. New label “Index last updated” has been added to the Search results, Search Setup, and Search Tuning pages.
  • User Map is now a new search entity, allowing users to search by it and adjust its search scores. User Maps can be hidden from search in Editor, just like Elements or Datasets. 
  • We provide better understanding of search results that have no clicks thanks to connecting search_item_clicked table results to search_log.

Custom Fields

  • Users can  include Custom Fields as Tile Hover Information by checking the corresponding option in Custom Field Editor.
  • Custom Field Editors > History Tab  provides details on changes made to Custom Field values.  Element Editors also include the History Tab , where users can view changes to Custom Field Values for Elements in Metric Insights.
  • All Custom Fields must belong to a Section. A default section is automatically assigned upon Custom Field creation. The new Section Editor allows users to:
  • Metric Insights restricts deletion of Datasets that are the source for actively used Custom Fields. Upon the deletion attempt, users will see a notification to confirm their consent. 
  • We have added Custom Fields to Datasets and Users Maps. These can be enabled by checking the “Show on” option in Custom Field Editors.
  • For improved readability, Custom Field Autosync Changes Details popup  includes two columns: Element Name and Changes. We have simplified it to display Element Name one time per sync and all Custom Fields value changes next to it.

Portal Pages

A new Advanced Tab has been added to the Portal Page Editor to allow users to enable receiving POST requests from sites other than the current MI instance. 

See:

Import/Export

  • We have redesigned the Migration Tool to support new architecture. Previous implementation was created based on Python 2.7, from which we plan to migrate soon.
  • Additionally, aside from Import and Export operations, we are able to perform Rollback, allowing users to revert Import back. 
  • Additional improvements include the ability to migrate Tag Types as well as Custom Fields for Datasets/User Maps.

Config Variables

  • CONST_PHP_PASSWORDS_ENCRYPTED is enabled by default. This will encrypt all passwords that are entered as Config Variables (which ultimately generate const.php)
  • Access Request and Search Variables have been moved to their own sections. “Access Requests & Discoverability“ and “Search/Data Analyzer“ filter options are available in the filter section dropdown for the “System Variables“ grid.
  • The new Config Variable, DISPLAY_FULL_IMAGE_IN_TILE_PREVIEW, controls whether External Report tile previews display a higher resolution, full-length image.
  • Full Category hierarchy is present on Category search popup, showing both parent and child Categories.

Bulk Change

  • Both Power and Regular users can use the Bulk Change option to modify the Enable/Disable state for Bursts.
  • Bulk Changes can be made to Custom Fields using Bulk Changes for Datasets.

Announcements

  • Announcement Editor has been expanded with the following two options allowing users to perform corresponding actions:
    • Require users to accept 
    • Allow users to dismiss
  • Announcement Editor now includes the History tab that reflects changes made to the Announcements and user action logs.

Other Improvements

  • Bookmarking has become more straightforward to enhance the experience for users.
  • To avoid confusion, we have removed legend/key like “Dimension value access not yet granted” from grids when not applicable. Metric Insights shows this  message only if there is a row without access. 
  • For user convenience, on Custom Script Editor,  the “Save & Enable” button supports Custom Scripts that have not yet been enabled. 
  • Display the Current Version and Installed date and time info on the Status Monitor page.
  • New installations will be shipped without the “Same Day of Week Last Month” Daily Metrics.
  • In previous implementations, External Report and External Content Templates were tied to Data Source. Now they are assigned directly to Plugins, which allows users to delete Data Sources without breaking any Templates.
  • For Power BI External Reports, Power BI OAuth set up as Drill-Down Authentication option will restrict Image Collection options specific to Power BI OAuth. 
  • We have implemented the ability to pass filter values into External Report Viewer via the URL, just like we have for Dataset Reports. The feature is also supported when using embed codes.
  • Metric Insights now supports IMAP for email delivery.
  • For user convenience, mailto: hyperlinks were added to Owner names in Element Viewers and Tile information icons.
  • Changes in Name and Description fields for Elements are tracked and reported on the History tab.
  • To avoid misunderstanding and confusion, the "Storage Down" message for Dataset Storage Emails has been changed to “Data Storage Connectivity Lost”.
  • Dataset List page includes the new Row column that shows Total Rows for Snapshot Datasets. 
  • Report Editor includes the new Workflow section where users can assign the Report to Workflow when the correct Category is selected. 
  • We have added primary keys to all dashboard_db tables.

Security

Tags

Allows Admins and Power Users (with the “Create Tags" Privilege as well as Edit Access to a given Tag) to use a Permissions button on the Tag Editor to grant Edit Access to Groups and Power Users only.  Power Users can only grant Tag permission to Groups of which the user is a member or to which user has Edit Access and to individual users belonging to those Groups.  There is no corresponding Extended Security Privilege.  

Publishing Workflows

"Create Publishing Workflow" Privilege supports  a Power User's ability to use the Publishing Workflow Editor to define a workflow and grant Permissions to Groups and other Power Users. There is no "Extended Security" Privilege related to Publishing Workflows; as a result, a Power User that created a Workflow or with Edit Access to it can only assign object Permissions to Groups to which the Power User belongs or has Edit Access and to individual Power User members of those Groups.  

Notification Schedules

Modifications have been made to the User, Group and Notification Schedule Editors and Permissions popup to clarify that only Groups and Power Users can be granted Edit Access to a "Shared with Everyone" (formerly “Public”) Notification Schedule. "Restricted Use" (formerly “Private”) schedules. Permissions may only be edited by an Admin or by the user who created it or, as an exception, by a Power User with Edit Access to it.  Power Users may be granted “Use Access” to a “Restricted Use” Notification Schedule.  As before, any "Shared with Everyone" Schedule may be used by a Regular or Power User when creating Bursts or scheduling a Digest.

Folders

Edit Access to a Folder may now be granted from both the Folder Editor and a Group or User Editor > Other Access tab. Edit Access is available for both Power and Regular User and gives them the ability to make content changes via the Folder's Editor. Sharing from the Folder's Editor grants "view access" unless "Add/Delete Content" is also granted either on the Editor or via the Group/User Editors.

Bursts

Power and Regular Users can use "Bulk Change" for Bursts to which the user has Edit Access from the Burst List page in order to change the enable/disable setting.

Lineage

Use of nodes in a Lineage diagram to access the related object is limited by Access permissions that  an individual Regular or Power User has been granted to the object associated with the node. Links to the node's object are set as follows:

  1. “Edit” is active only if Power User has Edit Access
  2. “View'” is active only if Regular or Power User has at least View Access
  3. “Focus this node” is active only if Regular or Power User has at least View Access

Identity Profile

A  Permissions button, on the Identify Profile editor,  allows ONLY an Administrator to grant Permissions to Power Users to Use a Profile when creating a Plugin Data Source with  Auth Type = Identify Profile to be used to present credentials to a BI Tool.    A Power User may ONLY use a Profile to which the user has been granted Use Access.  A Power User may not create an Identify Profile.

In addition, there is a Section on the Group/User Editors > Power Users where Identity Profile Permissions can be granted to the Power User or Group.  A Power User can only grant Permissions to a Group to which the user has Edit Access and also has the Privilege "Create Groups".

There are no Privileges associated with this feature.  No Regular User Access is permitted since this type of user cannot create a Plugin Data Source.

Bulk Change

Any Admin may perform this function on any object with a Bulk Change capability.  Power Users may only Bulk Change objects to which the user has Edit Access.  Their selection of the "change to" value for an object is limited to corresponding objects to which the Power User has Edit Access.  The one exception:  If the Power User has the Privilege to assign a Category with only View Access, those Categories would be included in the "change to:" drop-down list.   No Regular Users may make a Bulk Change to any object.

Content Center

A new Content Menu Entry for "Content Center" is provided to reach a this special purpose page. All Admins can reach this page and click any tab.  Power Users must have a new "Use Content Center" Privilege that is granted on a Group or User Editor in order to see this Content Menu entry and use it to access the Page.   Without this Privilege, a Power User does not see the Publishing option in the Content menu. This Privilege is automatically granted to each new All Access Group created in Metric Insights.

Chatbot

  • A new config variable, verify_ssl,  checks the SSL certificate has been provided.
  • Chatbot response timeout has been improved to approximately 3 seconds.
  • Chatbot docker container is switched to Debian-based image.
  • When users’ message includes http links that are external to MI app; e.g., https://google.com), in a public or private Slack channel, they receive a debug message “no search query specified” since the bot is designed to recognize only Tile names or text.
  • We are using a new version of SQLite, v3.35.
  • Current Pony ORM has been replaced with Peewee ORM to solve issues with inability to insert multiple records at once and slow response times. 
  • The Slack chatbot supports slash commands.

DevOps & Installation

  1. Included the ability to sync user groups by Okta SAML attributes using mi-okta-usersync script.
  2. Installer:
    • Added proxy server parameters:
      • --proxy-server-https-address, --proxy-server-http-address specifying the proxy server through which the services communicate with other resources;
      • --no-proxy parameter specifying components which should be accessed without proxy.
    • Kubernetes:  Added --persistent-volume-claim parameter, that mounts a persistent volume into a pod.
  3. System logs:
    • Simple Install: Optimized the number of containers by reducing the Logger service to a single container.
    • Information on network gateway availability is stored in the web container under /opt/mi/log/network.log.
    • Data Processor logs are stored in the /opt/mi/log/dataprocessor directory.
  4. Simple Install:
  5. Backup:  Added a system space check prior to backup. If there is not enough free space, a corresponding error message is shown specifying the amount of space needed to create a backup.
  6. MI Docker:  The latest security updates are automatically applied to r images.
  7. Kubernetes, Openshift: Added the --timezone option to the wizard.
  8. Docker Swarm: Enabled system health checks.
  9. ECS: Provided the ability to use the latest Amazon Linux 2 AMI in Terraform.

Issues Fixed

  • RDP on v6.2.5a was failing to connect to the Metric Insights and logs growing excessively can be fixed by replacing the RDP .jar file. See instructions here.
  • Kubernetes Wizard was failing without showing any warning messages. Warning messages are provided upon each wrong step while Wizard will be running uninterruptedly, without any unexpected exits.
  • Users can add UI parameters screenshotTimeout, mapLoadingTimeout, and delayBeforeScreenshotMillis at /admin/plugin-config to extend screenshot timeout in Power BI Cloud. Useful for large dashboards. Long types are no longer supported . 
  • The SEND_MAIL_FROM_NAME System Variable is no longer ignored by the system. Bursts are being sent from correct Send From names. 
  • Errors when collecting images in External Report for Cognos have been fixed.
  • Fixed the issue when images in External Reports bursted via Outlook were truncated.
  • Tableau Plugin collects nested projects with zero issues thanks to improved logic.
  • In some rare cases content permissions were not being applied to Global Search.
  • Having to add hostname to hosts files after the container restart or upgrade for scripts to work has been fixed by changing the base container from alpine to buster. 
  • The last_updated_by field is  populated in the user_dashboard_element_access table.
  • Restores from backups of previous versions were failing due to mi-db-dump utility code that treated MySQL data as ASCII forcibly. User are now able to easily restore from backups.
  • We have applied correct default settings for QlikSense Report Type (Append dimension filter to External URL as is set to “URL body part” by default) for the users to interact with filtered content when QlikSense Objects are embedded in iFrame without friction.
  • getObject for QlikSense Plugin used to fail with NPE errors in logs when getting NULL in "handled" field (used to get object ID). In v6.3.0, QlikSense Plugin disregards NULLs in this field allowing Application filter to work properly. 
  • Custom Field Values containing “&”  generated errors on Auto Sync due to the symbol being inserted into the MySQL table as “&amp". The inconsistencies have been addressed and Custom Field Values are property auto-synced.
  • String fields are successfully converted to dates.
  • All Rules are properly working with the percent sign.
  • Processing of DAX query “SUMMARIZECOLUMNS” has been updated  to return data in separate columns with their own naming.
  • The unexpected behavior with hyperlinks leading to Metric Insights homepage instead of the Element Editor has been addressed.
  • getObject for TIBCO Spotfire fetches all objects correctly. Metric Insights obtins Objects both from folders and sub-folders.
  • A typo in Zendesk Explore Plugin name has been corrected. 
  • Proxy options are unset during health checks for Docker Swarm.
  • The problem with migrations when updating from a beta version to a stable release version has been resolved.

Security

  1. Power Users are able to update the Technical Owner setting on all objects that have this field .  The same Extended Security rules apply to the objects as to granting Permissions for Edit Access; a Power User without Extended Security to the object may ONLY assign a Technical Owner to any Admin OR to another Power User that is a member of Group(s) to which the Power User is a member or to which the Power User has Edit Access.  Regular Users may not be designated as a Technical Owner.  See this article on ownership.

Recommendations

  • Version 6.3.0 requires running MySQL 8.0.20.

Important

  • We are discontinuing support for and removing Legacy Reports in 6.3.1. See instructions on how to migrate to Dataset-based Reports here
  • Internet Explorer 11 is not supported in Metric Insights 6.3.0. The list of supported browsers and mail clients can be found here