1. We have upgraded MySQL to v8.0.32, a recent and stable release of MySQL (as of January 17, 2023).
2. Virtual Datasets are a new capability of External Datasets that support direct queries to Snowflake, Databricks, MySQL, and Microsoft SQL Server. For now, they are designed specifically for use with Portal Pages via the dataset_data API.
3. The application now incorporates Redis as a separate service responsible for caching.
4. SharePoint Online (SPO) Plugin can pull new types of SPO metadata such as file versions. This added functionality incorporates the ability for downloading using Custom Portal Pages.
5. A new Sigma Plugin has been introduced in this release. The plugin is capable of collecting both data and images.
6. Multiple API improvements have been implemented. Among them there are new API calls and extended capabilities of the old endpoints.
7. We continue to develop our BI Concierge generative AI service as an essential part of the application. This work is still in Alpha.

1. The vulnerability associated with the potential reveal of local files on the container's file system while working with Custom Scripts has been addressed.
2. The Git User-Agent value is masked in GET requests, preventing the potential leak of sensitive data.
3. Sensitive data, such as base64 encoded credentials, are no longer available for viewing by Admin users in the Data Processor logs.
4. The option to exclude Favorites from caching has been implemented in this release. This is useful for use-cases where many hundreds of concurrent posts are expected to /api/favorite. The new System Variable API_CACHE_EXCLUDE_FAVORITES controls Favorites caching.
5. An error occurring during Site ID verification upon the Tableau Online test connection has been fixed.
6. Special characters listed below are handled as an underscore “_” in Category names upon "Mirror folder hierarchy":
   • Double quote "
   • Single quote '
   • Less than <
   • Greater than >
   • Caret ^
   • Grave accent (backtick) `

Known Issues

If the system has been patched, contact our support for a patch that ensures that the correct application version is set in /opt/mi/log/state/web.state after restoring from a backup.

The primary Docker images used for standard deployments are:

1. web – docker.metricinsights.com/metric-insights/release_6_4/36
2. dataprocessor – docker.metricinsights.com/dataprocessor/release_6_4/26976
3. seed – docker.metricinsights.com/seed/release_6_4/39956
4. data-analyzer – docker.metricinsights.com/data-analyzer/release_6_4/89
5. monitoring – docker.metricinsights.com/monitoring/release_6_4/1002
6. redis – docker.metricinsights.com/redis/release_6_4/3

Additional images:

1. logger (needed for simple deployments) – docker.metricinsights.com/logger/release_6_4/33
2. mysql (used for simple deployments with mysql running locally) – docker.metricinsights.com/mysql/release_6_4/36

This release includes an upgraded version of MySQL to MySQL 8.0.32 (release notes). It is used by default with simple installation upgrades. For other installations, it is recommended to perform a distinct MySQL upgrade before upgrading Metric Insights.

Virtual Datasets are a brand new method of working with external datasets, specifically optimized for Portal Page development. The  ENABLE_EXTERNAL_DATASETS System Variable enables a Tables tab in the Data Source Editor, from which Tables can be published as Virtual Datasets.

The following SQL Data Sources support creation of Virtual Datasets:

• Snowflake
• Databricks
• MySQL
• Microsoft SQL Server

Once enabled, SQL Data Sources can be queried directly, obviating a need for data storage within Metric Insights. That means all data will stay in your warehouse/lake and temporarily cached, but there will be no permanent storage in MI. Data from these Virtual Datasets can be obtained via our dataset_data API for use in Portal Page development.

1. For Tableau v2022.1 and later, the element_info endpoint can return a JSON Web Token (JWT) if “jwt=Y”. Connected App Client ID, Connected App Secret, and Connected App Secret ID must be filled out for the Data Source.
2. dismiss and clear API calls are available for the announcement endpoint. All User types have access to them.
3. element_info API returns the Custom Access Denied message if it's specified. A new PUT call, request_access, has been implemented to allow Regular Users to request access to an Element.
4. Data from dataset_data can now be received by passing both full Filter Values and Filter Value fragments. OR as well as AND operators can be used with Filter Value fragments.
5. Endpoints related to retrieving Public and Private Entities support pagination and filtering.
6. folder API now allows Users to add, update, and delete Folders and Sub-Folders while folder_element API provides the ability to perform the same actions with Folder Elements. Folders and Sub-Folders can also be moved/copied between other Folders and Sub-Folders.
7. element API returns Elements available for a User on a homepage.
8. slack_channel and msteams APIs response for the elements key has been modified to include a list of Dimension Values attached to an Element if any are available. 

1. System Variable pairs CACHE_LAYER_1_SAVE_HANDLER, CACHE_LAYER_1_SAVE_PATH and CACHE_LAYER_2_SAVE_HANDLER, CACHE_LAYER_2_SAVE_PATH have been introduced in this release to help handle cache layer 1(User-related data) and layer 2(Dataset- and API-related data). These can now be saved in Redis and not only in a file system.
2. The new INITIAL_USER_LOGIN_EXTERNAL_SYNC provides for controlling how User Group membership is assigned on initial login user creation. The two available options are: LDAP and Custom Script. The former references LDAP / Active Directory to identify the User’s Groups and assigns the User to corresponding Groups in Metric Insights. The latter assigns Group membership based on a User Access Dataset tied to a Custom Script.

1. Our code uses the latest Open ID Connect Library.
2. NameIDPolicy in /opt/mi/iv/engine/libs/SimpleSAML/config/authsources.php can be set to “false” to help avoid the InvalidNameIDPolicy error. Set the right value for NameIDPolicy under SAML_NAME_ID_POLICY in /opt/mi/external_config/saml.php.

1. New command line parameters were added to the mi-ldap-usersync utility provision mode. 

• --username allows to run sync for a specified user. 
• --auto-create key [Optional parameter] Automatically adds/creates LDAP user groups the member belongs to if they don't already exist in Metric Insights (otherwise, the user will be synced only to Groups previously created in Metric Insights).

1. The application has started to use Redis as permanent storage for sessions and optionally application's cache. Redis is a new application service that runs inside of its own container. Port 6379 needs to be opened within the namespace (network rules). If you are running an orchestrated deployment, generate a new 6.4.3 deployment manifest or modify the existing one and  generate new secrets for the Redis service.
2. Metric Insights can be deployed in a fully IPv6 subnet without any IPv4 gateways, etc. 
3. Core dumps are disabled for the system in the dataprocessor component.
4. Now, dataprocessor and web (master, slave) have a common shared folder at /opt/mi/generic.
5. MySQL-client is now available in dataprocessor and data-analyzer components. Please note that mysql parameters must be supplied.
6. We have added the support of Helm chart generation by the installer: ./installer.py kubernetes --type helm.
7. Support for Debian 12 has been added in this release.

1. We have renamed the Portal Page "Name" field to "Portal Page Name". This change was required to disable browser auto-fill for the name fields.
2. Sharing permissions have been applied to searchability for Documents. Documents marked as global will be visible to Users from shared groups depending on whether their User Type is selected under "Share with specific User Types”.
3. Voice Search is now supported in Edge Browser.

1. The issue related to long migration running in data-analyzer due to a big amount of search data from previous installation has been fixed.
2. The issue with Username for Trusted Server Sign-on not populating upon user sync is no longer present in the system.
3. For orchestrated deployments, the expected CPU number for the service in the data-analyzer component is to be set via the DA_CPU_NUMBER variable. For simple installation, everything is evaluated automatically.
4. The issue with the Content Center’s Engagement throwing an error in the console has been fixed by removing deleted users from statistics.
5. We have fixed supervisor behavior for RHEL 7.7 after facing limitations of sysctl calls.
6. Redis with cluster mode enabled is no longer causing caching issues that led to the web container's unhealthy status.
7. A problem that occurred when embedding a manually loaded image in Microsoft Power BI External Report with OAuth enabled has been fixed. The application was using a token form Plugin Connection profile instead of the Personal user token. 
8. Chinese characters are now being displayed correctly in Microsoft Power BI External Report images.
9. For Power and Regular Users, GET request to /folder_element?folder_id= now returns only Elements that are placed in a Folder, without Elements with Discoverability.
10. We have modified the previous behavior when discoverable content was blocked from being displayed in search. Now, when SEARCH_RESULTS_INCLUDE_CATEGORY_HIERARCHY is enabled, Categories with discoverable content are displayed in Category hierarchy on the Search page.
11. Previously, the system was failing to save values for some Variables in the Portal Page Editor for Power Users. The issue has been resolved.
12. Upon Glossary Autosync, Business and Technical Owners are inherited from the Element and no longer default to the User who initiated the sync.
13. Issues with the data display in Content Center when selecting multiple Categories as an Admin have been fixed. 
14. The issue with Portal Pages failing to be included in export has been resolved. It occurred only if mi_gen user (Migration Utility user) had the at sign “@” in the user password. 
15. We have optimized Data Processor operation with Chromium and ChromeDriver to avoid high CPU utilization.
16. Collection of data from files whose names include spaces has been modified to work correctly for SharePoint Server Plugin.
17. Custom Columns Dataset created from SharePoint Online Data Source no longer returns duplicated rows of metadata about the same file.

Due to new caching approach and optimizations, it may take up to one minute to apply the System Variable changes in orchestrated environments. Refreshing the page may help resolve the issue.