Release 6.4.2

We are glad to announce that v6.4.2 is finally out. This is a minor release focused on small enhancements and security. We have introduced partial Search, Data Analyzer-based search for Chatbot, new logic for Glossary Autosync, Bower BI Filter scopes, support for Arm64, some important API changes, and multiple security optimizations. Among other noticeable changes, we would like to call out our SharePoint Plugins' extended functionality too. Find more details below.

Major Features in 6.4.2 (released June 1, 2023)

  1. Partial Search supports returning results even for search terms represented by incomplete words.
  2. Chatbot Search is now powered by Data Analyzer instead of its own custom implementation of search, which makes it faster and more robust.
  3. Power BI sticky filter logic is separated from the regular MI Sticky Filters allowing for remembering Power BI Filter Values within the scope of a given Power BI Report.
  4. API enhancements have been implemented to enable writing to a Dataset. notes endpoint has been improved to be able to easily include collaboration functionality on a Portal Page.
  5. Support for Arm64 (including AWS EC2 Graviton) is Generally Available.
  6. Kerberos auth is now available for SharePoint Server 2019 Plugin.
  7. Metric Insights is no longer shipped with the default support user password. A unique password will be available in all new installs and live at /opt/mi/support.password that will be deleted one hour after installation.

The primary Docker images used for standard deployments are:

  1. web – docker.metricinsights.com/metric-insights/release_6_4/24
  2. dataprocessor – docker.metricinsights.com/dataprocessor/release_6_4/26972
  3. seed – docker.metricinsights.com/seed/release_6_4/39952
  4. data-analyzer – docker.metricinsights.com/data-analyzer/release_6_4/16
  5. monitoring – docker.metricinsights.com/monitoring/release_6_4/1001

Additional images:

  1. logger (needed for simple deployments) – docker.metricinsights.com/logger/release_6_4/31
  2. mysql (used for simple deployments with mysql running locally) – docker.metricinsights.com/mysql/release_6_4/34

Search/Data Analyzer

  1. Global Search now supports partial search. For example, searching for "sal" will provide results for "sales".
  2. Python has been updated to version 3.11.2 for Data Analyzer, the main component that underpins the search mechanism.
  3. All python libraries have been updated to resolve CVEs.

Chatbot

  1. The search engine parameter has been added to Chatbot Settings. It controls which search engine Chatbot should use as the default, custom implementation of search or Data Analyzer. Data Analyzer powers the Global Search functionality, making the results from queries to Chatbot the same as when using Global Search, thus providing a more consistent experience.
  2. We have updated the logic of search to work with multiple-word tags. Issues with empty search results for some of the widely-used words such as “top”, “bottom”, “highest”, “lowest”, “largest”, “smallest”, etc. have been fixed too. 
  3. Python has been updated to version 3.11.2 for Chatbot.

Power BI Sticky Filter Scopes

When enabled, a new System Variable, LIMIT_POWERBI_STICKY_FILTER_SCOPE, separates Power BI Sticky Filters from the regular MI Sticky Filter logic. The system will only remember Power BI Filter Values within the scope of the Power BI Report but not across different Reports and not in relation to any other Object within Metric Insights.

API

  1. New API enhancements enable the ability to write to a Dataset.
  2. The notes API endpoint has been extended with the following fields: user_id, username, display_name, comments to expand the capabilities of collaboration functionality on a Portal Page.

Plugin Improvements

SharePoint Server 2019/SharePoint Online

  1. We have implemented Kerberos-based auth for SharePoint Server 2019 Plugin.
  2. User/Group Access Lists as well as Custom Field metadata are now available to pull from SharePoint Online objects. Users can create Datasets containing this information from files stored in the Available Data folder. 

QlikView

The Plugin can now automatically collect Filters and Filter Values for External Reports. By default, only Filters that have aliases (LB) on the QlikView side can be collected and displayed in the UI.

Other Plugin Changes

  1. Athena JDBC driver has been upgraded to the latest version.
  2. The following Plugins have been removed from Metric Insights: 
    • OBIEE Plugin
    • OLAP Plugin
    • Adobe Analytics

Unique Passwords

Upon a new installation, an Admin password is set and the Support User is created with a temporary password written to /opt/mi/support.password. This file is deleted after 60 minutes. If the Support User credentials are not used within one hour, the user can either change the password as an Admin under Users & Groups or use [email protected] to recover the password that will be sent to the support channel.

Data Processor

  1. RDP is now shipped with OpenJDK preinstalled. 
  2. We have upgraded all critical dependencies to mitigate vulnerability risks.

System Variables

  1. It is now possible to disable the standard Metric Insights login page if external auth is enabled. This possibility is controlled by the ENABLE_LOGIN_PAGE System Variable. Once SSO/SAML auth is enabled, it is recommended that the Value is set to “N”.
  2. Beginning in this release, the default value of the SEND_ACCESS_REQUEST_DIGEST System Variable is “Y”.
  3. The HIDE_USER_EMAIL_ADDRESSES System Variable can hide user email addresses from key places in the UI: User List, search function within the User Editor, and the “Share” dialog.
  4. The ENABLE_TABLEAU_CROSS_SITE_WORKAROUND System Variable helps resolve the invalid token issue that occurs in some Tableau configurations when one browser tab is open with a Tableau External Report from Tableau Site A, and another browser tab is open with a Tableau External Report from Tableau Site B. Setting this Variable to “Y” will force refresh of the page/tab when multiple Tableau Sites are opened in different tabs.
  5. Issues with the delivery of test emails, Bursts, and Alerts occurring because the recipient's email was enclosed in angle brackets “<>” can be fixed by using the new System Variable, SEND_MAIL_EXCHANGE_SERVER_RCPT_TO_WORKAROUND.
  6. The new System Variable, ALLOW_USER_CREATION_VIA_UI, can disable the ability to create/delete users via the UI. As a result, new users must be created through LDAP/AD Sync or programmatically via API.

DevOps & Installation

  1. Chatbot is now run by one unprivileged user www-data, uid: 33.
  2. We have introduced a new utility, mi-support, which replaces netcat and telnet.
  3. There is a new option in mi-ldap-usersync, --skip-disabled-users, that prevents syncing disabled users from Active Directory.
  4. mi-ldap-usersync now provides better performance through improved caching.
  5. MI Chatbot can now be deployed on servers with ARM64 architecture, including AWS Graviton.
  6. The feature tool has been removed from the Patcher.
  7. Contents of the directory /opt/mi/iv/www/img/fav_custom no longer get lost after upgrade and backup restore.
  8. Port 8080 is now closed by default in Simple Install and can be open by running the installer with the --allow-access option.
  9. Python version is updated to 3.11.2 in all MI containers.
  10. Connection timeout for the web container has been increased to 30 seconds.
  11. Debian 9 is no longer supported by the system.

Other Improvements

  1. netcat and telnet utilities have been removed from all our containers/services to further lock them down.
  2. Several libraries and dependencies have been updated to resolve CVEs.
  3. Glossary Autosync will now update all Terms regardless of whether that Term already exists or not.
  4. Most Popular Folder script has been optimized to run and deliver results faster.
  5. Custom and default Access Denied Message for Elements is using the same standard popup with the blurred Element image and metadata.

Issues Fixed

  1. The issue with the search index process sticking at some point has been fixed by reducing the number of updated permission records per batch from 100,000 to 10,000.
  2. We have optimized the SQL query for getting the Triggers list to run and return results in shorter times.
  3. Some misplaced UI elements and overlaps in the French version have been fixed.
  4. The issue when users could not select Glossary Fields for mapping in Glossary autoimport is no longer present.
  5. The SQLi vulnerability has been resolved.
  6. The announcement and dimension_value endpoints have been enabled for Regular and Power Users which makes the corresponding objects available for these types of Users on Portal Pages.
  7. Recollection of data for failed Elements, "Refetch Historical Data: All elements with error during last Trigger run", now works as expected. 
  8. The issue with api/favorite_element call returning discoverable content in addition to Favorites has been fixed. 
  9. The Files filter Parameter for Local Filesystem Plugin can now properly recognize an asterisk "*" as a wildcard. 
  10. An error that was occurring upon "Refresh Metadata" and "getObjects" for a Microsoft SharePoint Data Source has been fixed. The issue was related to timeout.
  11. Content Center pages are now loading significantly faster for Power Users.
  12. Before 6.4.2, REST API was not applying filters to PDFs from Tableau. Beginning in this release, both Web and REST API can pull PDFs with filters from Tableau; however, WebAPI works only with Tableau Views. 

Known Issues

With Kerberos auth configured for SharePoint Server 2019, it won’t be possible to get data from files/folders whose names contain spaces.