Release 6.4.4

We have successfully rolled out the last GA release of 2023, Release 6.4.4! Our LLM-based Concierge is officially in Beta—you can think of Concierge as our rising star—at this point it is available for early testing of basic "help me find a report" use-cases, plus navigating our help docs. By the end of 2024, we expect it to be a powerful assistant capable of coaching you through your day-to-day work with MI. In this early Beta phase, the functionality is only available via a Portal Page, and Azure Cognitive Services or OpenAI Enterprise APIs are a pre-requisite.

Other features in 6.4.4 include additions to search, supporting folder-based security for "report link list" use-cases, wherein a (usually large) list of reports is ingested as a dataset. The reports themselves can now exist in folders and each folder can be mapped to an LDAP/AD group in an access control list. 

Our API has been extended. And, as usual, we've continued to invest in new Plugins. The new Azure DevOps Plugin can retrieve objects, data, and images. Our Microsoft SharePoint and Tableau Plugins include some exciting changes as well. Keep reading.

Major Features (released December 6, 2023)

  1. Concierge Beta is available by request, strictly for early use-case exploration. At this point, the aim is to help users save time locating content either in the application or on our help site. The functionality is only available via a Portal Page and Azure Cognitive Services or OpenAI Enterprise APIs are a pre-requisite. Reach out to [email protected] to learn more.
  2. Search enhancements, primarily focused around Report list Datasets, wherein a Dataset is providing a list of reports and security is applied based on the Folder. In other words, FolderA/ReportA.pdf could map to Group A, and FolderB/ReportB.pdf could map to Group B.
  3. We've introduced a new API endpoint, api/share, as well as improved upon our existing APIs.
  4. Our Tableau Plugin now has a new auth method that works based on a JSON Web Token (JWT), over time, we expect this to be Tableau's preferred authentication method for embedding.
  5. We now have a Plugin for Azure DevOps. It supports Datasets and External Reports.
  6. Our Microsoft SharePoint Plugin now has a longer list of Optional Parameters to better manipulate additional metadata. Some new Parameters have been added to the Plugin Config Page as well (specifically for more advanced configuration)

The primary Docker images used for standard deployments are:

  1. web – docker.metricinsights.com/metric-insights/release_6_4/45
  2. dataprocessor – docker.metricinsights.com/dataprocessor/release_6_4/26981
  3. seed – docker.metricinsights.com/seed/release_6_4/39960
  4. data-analyzer – docker.metricinsights.com/data-analyzer/release_6_4/156
  5. monitoring – docker.metricinsights.com/monitoring/release_6_4/1004
  6. redis  docker.metricinsights.com/redis/release_6_4/5

Additional images:

  1. logger (needed for simple deployments) – docker.metricinsights.com/logger/release_6_4/35
  2. mysql (used for simple deployments with mysql running locally) – docker.metricinsights.com/mysql/release_6_4/39

Concierge (Beta)

You can think of Concierge as our rising star. For now, it supports a few basic use-cases. By the end of 2024, we expect it to be a powerful assistant capable of coaching you through your day-to-day work with MI. 

In this early Beta phase, the functionality is only available via a Portal Page that must be manually installed, and Azure Cognitive Services or OpenAI Enterprise APIs are a pre-requisite.

Example use-cases/questions for Concierge: 

  1. "Show me sales reports:"
  2. "What is a Burst?"
  3. "What is an External Report?"

If you are interested in taking a closer look at Concierge and partnering with us as we embark on our LLM journey, please contact [email protected].

  • Report List Search has been enhanced to support Datasets with Folder access lists. To explain understand this better, imagine that you have 2,000 reports spread across various Folders in a NFS/shared drive. You can now make all those reports available in MI, with security applied at the Folder level. This is a great way to on-board large volumes of content, and especially useful for static reports like PDFs or XLS files. The use-case can be supported through native MI search, or via a custom implementation as a Portal Page.
    • We've exposed a new option that allows for sorting search results by engagement (views + distributions = engagement)
    • Tags, Custom Fields, and Discoverability can now be applied as filters and shown as additional metadata for Report List Search.
    • Report List Search can be based on links to files stored in a NFS/shared drive. 
    • Searchability of these files can be restricted to particular Users/Groups based on Security Access Datasets that specify folder paths or complete paths to files.
  • In a previous release, we implemented Data Analyzer-based search for Chatbot, the Chatbot alias feature, which simplifies search with the help of predefined abbreviations, is now powered by Data Analyzer too.

API

  • A new PUT request to /api/search_history  works in conjunction with a new Custom Script that removes rows of External Link Dataset from search_item_clicked which was previously creating problems for Users with accessing resources via links from search history.
  • /api/announcement now returns "created_by" and the "Show on Portal Page" checkbox state. The latter requires a User to have access to the announcement.
  • A new endpoint, /api/share, provides the possibility to email Elements to which the User has access by specifying email or recipient_id (user_id).
  • The new api/get_token returns a user token for an Admin or Power User if logged in.

Plugin Improvements

New Plugins

The new Azure DevOps Plugin’s capabilities include getting objects, data, and images. 

Microsoft SharePoint

  • The Plugin can now exclude specific libraries from object collection using the new Optional Parameter, Exclude library. Use a comma “,” to specify multiple libraries.
  • The new Optional Parameter, History load interval, controls the period for which versions of documents are collected into the Version history Dataset. Leaving it blank or setting it to “0” will return metadata for the whole period. Setting it to “1” will return metadata for today and by setting it to “2” you can get versions for today and yesterday and so on. 
  • Site SharePoint API defines a Site for which the SharePoint Group Membership Dataset will include Groups associated with this Site. To use this Parameter, configure the Domain Optional Parameter and pathToPtx and ptxPassword Parameters at the Plugin Config Page specifying the corresponding certificate details.
  • The following three Plugin Config Parameters control the data collection behavior in case of encountering errors. maxRetries defines the number of times the Plugin will keep initiating data collection for objects that throw an error upon it and delay sets the interval between these tries. skipItems set to “false” aborts data collection if an error is encountered and resumes data collection ignoring errors when set to “true”.

For more information, refer to Establish Connectivity to Microsoft SharePoint and OneDrive.

Tableau

  • A new auth method is available for Tableau Plugin. It works based on a Java Web-based Token (JWT).
  • Tableau Plugin can collect user-specific images and data via Personal Access Token and REST API. Tableau impersonation must be enabled on the Tableau side.

Microsoft Power BI Cloud

  • The Get token from Plugin Optional Parameter for Microsoft PowerBI Cloud Plugin is now set to “Y” by default.
  • The new Power BI Premium: Collect native PDFs and Power BI Premium: Collect image using API have been added. These Parameters work only for Premium or Embedded capacity subscriptions.
  • The old auth library, ADAL, is not supported in the latest Microsoft Power BI versions. The new supported auth library is MSAL.

Other Plugin and JDBC Driver Changes

  • Custom PowerPoint Templates for Bursts and Favorites attachments can now be configured to include specific Element images based on Element IDs. 
  • ThougthSpot Plugin has been moved to a new API.
  • The following JDBC drivers have been upgraded to newer versions:
    • Athena
    • Databricks
    • MySQL
    • Microsoft SQL Server
    • Postgres
    • Vertica

MySQL

  • Starting 6.4.4, it is required to configure the update prioritization parameter in the following way: “LOW_PRIORITY_UPDATES = 1” on the MySQL side.
  • If a User is deleted from the system, object Ownership is reassigned to an Admin User with the lowest ID number.

Usersync

  • If a User is created from LDAP/SAML/SSO, the First Name, Last Name, and Email fields are not editable on My Settings.
  • We have added support for mi-o365-usersync to auto-assign User Group membership on the initial login User creation.

DevOps & Installation

  • We have implemented support for Oracle Linux 8 and 9 for Simple Installation.
  • The wget and curl binaries are compiled from the source code to reduce the number of additional packages required to be installed along with them, thus reducing the number of vulnerabilities (non-exploitable in the scope of the application) in the product images.
  • We have removed the linux-libc-dev system package from all docker images to decrease vulnerabilities.
  • Vim editor has been replaced with nano due to a number of vulnerabilities (non-exploitable in the scope of the application) detected in it. Nano editor does not add any vulnerabilities to the product images.
  • A new installer option for ECS and K8s installation types, --disable-same-worker-for-web, has been added. It forces the web and web slave tasks to be run one at a time per instance. The number of instances must equal or exceed the total of web master and web slave tasks. The same applies to the Kubernetes pods – the number of nodes must equal or exceed the total of web master and web slave pods.
  • The --version (-V) option has been removed from both mi-app-backup and mi-app-restore as unnecessary.
  • We have added --username option to mi-o365-usersync. Use --auto-create if it is required to create appropriate Groups in Metric Insights. If a Group can’t be found, it will be skipped. If a User doesn't exist in Metric Insights, --auto-create will create it. By default, the script won't create new Users.
  • The mi_tools_run_log table has been added to track the MI tools execution.
  • There is a new option, --create-child-groups, that works within the --process-child-groups option. If process-child-groups is set to "all", then create-child-groups will create all detected Groups, including a parent one. If process-child-groups is set to "only", create-child-groups will create only detected child Groups and sync their Users, skipping a parent Group and its Users.
  • Starting this version, if the patcher log is empty, it will not be rotated.
  • The Docker-compose version has been updated to 2.20.3.
  • Ports that are not supposed to be publicly available have been moved to the internal Network Load Balancer. The monitoring clients will communicate with the Monitoring server over the internal Network Load Balancer as well. ECS ENI trunking must be enabled on AWS.
  • We have added --trusted-server-username-attr to o365-usersync to populate the Trusted Server Sign-on field. The ALLOW_TRUSTED_SERVER_USERNAME_OVERRIDE System Variable must be enabled.
  • The cpp package has been removed, which automatically has removed the gcc package as well.
  • --proxy-server-http-address and --proxy-server-https-address have been replaced with a single --proxy-server-address, which also updates the PHANTOMJS_PROXY value in the config_variable table. 
  • The Chromium version has been upgraded to the latest secure version, v118.
  • mi-app-restore saves the current HOSTNAME, which makes it possible to be reused after the restore is completed, with no need to override the hostname manually.

Other Improvements

  • Masks for Metrics are now being applied correctly on Portal Pages. Previously, Portal Pages were using Elements from Entities after refreshing rather than from the API. This approach has been changed.
  • We have changed the logic of the Custom Field migration.

Issues Fixed

  • Issues with embedding Metric Insights content can be resolved by just disabling x-frame-options, without any additional configuration in Apache2. 
  • Categories and Subcategories appear in the correct order after import. The issue was related to the incorrect handling of display_order for Categories on a target system.
  • If a Power User/Regular User has access to an Announcement with "show_on_portal_page" enabled, they can see it on a Portal Page in the same way as an Admin User can.
  • We have improved the copy and move operations for PUT api/folder, which previously created additional access scenarios for the moved/copied Folders and Subfolders.
  • Dimension Values are no longer available for a User who has been removed from a Group with access to them.
  • We have fixed the homepage search to return consistent suggestions and results for search terms that include the period sign ".". Inconsistencies appeared due to how Data Analyzer handled the search and indexation processes.
  • Domo External Reports had no URLs despite the URL auto-generation being enabled for the Report Type because the embedUrl parameter was not correctly passed to the UI.
  • External Content from SAP Business Objects is now properly downloading from search.
  • The issue with Group Permissions not applying for Group Names with the ampersand sign "&" based on User Access Control Lists has been fixed.
  • The Folder Sharing settings for a Portal Page are not getting wiped out upon migration.
  • To increase the number of retry attempts to fix issues related to deadlocks during syncing users by mi-ldap-usersync, Users can now use the MAX_DEADLOCKS_PROCESSING_ATTEMPTS System Variable.
  • Element migration was failing to regenerate the URL for External Reports in some cases. This behavior has been fixed in this release. 
  • Custom-uploaded images for External Reports are now included in import files.
  • The issue with Element indexing being stuck related to the complexity and quantity of requests to a database. Such requests are now divided into smaller ones and handled individually to decrease memory consumption and speed up processing times. Overall, we have optimized indexing for other search entities as well.