Scripted Migration Utility

It is possible to Migrate Content using our Export/Import Migration Scripts that enable System Admins to setup a content migration pipeline. The article shows an example of how to migrate content from one server to another.

The Scripted Migration process includes two main stages:

This article details how to move a Category and all included elements from one server to another.

Migration Capabilities

Note: Root privileges are required to run Migration.

MI Elements and Objects
 Scripted Migration
 Details
Metrics
 can be migrated


BEFORE MIGRATION, on the new instance make sure to:

  1. Recreate Dimensions for all dimensioned Elements that are migrated.
  2. Recreate Dimensions for all Elements with Filters mapped to Dimensions.
  3. Establish connectivity to all BI tools (by creating respective connection profiles) that serve as Data Sources for migrated Elements/Objects.
Reports
 can be migrated
External Reports
 can be migrated
Datasets/Users Maps
 can be migrated
Categories
 can be migrated
Folders can be migrated 
Portal Pages
 can be migrated 
Bursts
 can be migrated 
Digest Templates
 can be migrated 
Custom Fields
 can be migrated 
Dimensions
 NOT SUPPORTED
 ALL OBJECTS AND ENTITIES that cannot be migrated directly have to be rebuilt on the new instance.
Plugin Data Sources
 NOT SUPPORTED

Key Migration Dependencies

Migration Dependencies
Element/Object IDs
  • Object/Element IDs are preserved unless there are identical IDs on the new server. 
  • In case of existing duplicates, migrated Objects and Elements will be assigned new IDs.
Data Sources
  • If the migrated content was sourced from a Dataset, the Dataset will be imported as well.
  • External Connections to other systems (BI tools) have to be recreated manually on the new server.
Technical/Business Owners
  • Migrated Objects and Elements will retain their Technical/Business Owners if these Users exist on the new instance.
  • Search for User matches is performed first by email and, if there are no hits, by Username.
  • If there are no matching Users, Migrated Objects/Elements will be assigned a new Owner (the first admin that is found on the new server).

1. Exporting Content

Exporting content involves creating a .tar.gz file with information on all migrated Elements/Objects that can later be uploaded to a different server.

All associated Documents/Previews/Attachaments/JS Templates can be exported in an archive as separate files.

To initiate export, run the following command: 

sudo python3.9 /opt/mi/migration-tool/run.py export -c 86 -f  /<directory>/<archive name>.tar.gz

where:

  1. python3.9 is a Python Interpreter (installed during the installation of the MI application)
  2. /opt/mi/migration-tool/run.py is a migration script
  3. export is a command which should be performed
  4. -c 86  is a Category parameter followed by Category ID
  5. -f  parameter specifies the name for the output archive
  6. <directory>  where the dump file will be created
  7. <archive name>.tar.gz is a user-defined archive name

To export a Category (as an example):

  1. Run run.py export  with the desired parameters
  2. The response of the script will contain an integer, which is the migration_export_log table ID. The table along with migration_export_log_entitiy contains details about the export.
    • NOTE: In case of fatal errors, the response will return an error text message. This may happen when the program is not able to connect to the DB or some system migration tables were absent (or corrupted)
  3. Optionally, verify that the Category was saved to the tar.gz archive with the name you specified
  4. Optionally, view the contents of the exported tar.gz archive
    • In an archive, Documents are stored by Number IDs
    • The actual Document names are stored inside a .json file

1.1. Optional Parameters for Export

The list of arguments to use at export
  -h, --help   Show help message and exit
-e ELEMENTS [ELEMENTS ...], --elements ELEMENTS [ELEMENTS ...] IDs of elements
--bursts BURSTS [BURSTS ...] Burst IDs to dump (space-separated)
-d DATASETS [DATASETS ...], --datasets DATASETS [DATASETS ...]

IDs of Datasets
 -f FILE_NAME, --file_name FILE_NAME Output file name
--exclude-access-maps

Do not export any dependent access maps
--folders FOLDERS [FOLDERS ...] Folder IDs to dump (space-separated)
--group-sharing

Include sharing user groups for migrated objects
-c CATEGORIES [CATEGORIES ...], --categories CATEGORIES [CATEGORIES ...] Category ID to dump (space-separated)
-p PORTAL_PAGES [PORTAL_PAGES ...], --portal-pages PORTAL_PAGES [PORTAL_PAGES ...]

Portal Page IDs to dump (space-separated)
--digest-templates DIGEST_TEMPLATES [DIGEST_TEMPLATES ...] Digest template IDs to dump (space-separated)
--custom-fields CUSTOM_FIELDS [CUSTOM_FIELDS ...]

Custom field IDs to dump (comma-separated)
--include-folders

If an Element is included in Folder, but that Folder has not been explicitly selected for export, this flag will automatically include the Folder along with the Element.
-s, --scheduled

Include scheduled migration
--skip_external_report

Do not dump External Report
--skip_documents

Do include dump documents in the outcome archive
--skip_other_external_content

Do not dump Other External Content

2. Importing Сontent

Importing content involves uploading it to the required server.

  • The run.py import script is used to import Elements and Objects.

PREREQUISITES:

  • Before running the import script, copy the saved .tar.gz archive to the server where your content needs to be imported.

To initiate import, run the following command:

sudo python3.9 /opt/mi/migration-tool/run.py import -f /<directory>/<archive name>.tar.gz -b ~/.

where:

  1. python3.9 is a Python Interpreter (installed during the installation of the MI application)
  2. /opt/mi/migration-tool/run.py is a migration script
  3. -f  parameter allowing Users to specify a .tar.gz archive that will be uploaded to the new server
  4. <directory>  from which the upload will be run
  5. <archive name>.tar.gz is a user-defined archive name
  6. -b is a backup parameter

To import a Category (as an example):

  1. Run run.py import
  2. The response of the script will contain an integer, which is the migration_import_log table ID. The table along with the migration_import_log_entitiy and migration_import_operation tables contains details about the import.

2.1. Optional Parameters for Import

The list of arguments to use at import
-h, --help Show this help message and exit
-f FILE_NAME, --file_name FILE_NAME

File to load elements from
--test_output

Output as Python dict (for development purposes only)
-b BACKUP_DIR, --backup-dir BACKUP_DIR

Directory to back up changed records to
--preserve-access-map

Do not overwrite access map configuration
--burst_create_only

Create new Bursts only. Do not update existed Bursts
--new_entity_topics_only

Create topic only for new entities (Elements, Datasets, Portal Pages, Bursts). Skip topics for existing records
--use_category_owner

For new Elements/Datasets, Owner are set to Category owners. For existing records, Owners are not updated
--reset_certification

For new Elements/Datasets, certified attribute is set to 'N' (reset_certification, last_certified_time as well). For existing records, certified attribute is not updated
-s, --strict

Delete Elements/Topics in the target Category if they are not in the source Category
-m, --match

Load matching Elements
-r, --relax

Keeping data on target system
-n, --no-preserve

Do not preserve IDs

3. Rolling Back

[6.3.0] The rollback command allows users to cancel the import operation. To able to do this, it is required to specify the ID of import operation (migration_import_log ID). 

python3.9 run.py rollback 111

The rollback operation uses migration_import_log_entity and migration_import_operation tables to perform the revert operation. For example:

  • insert –> delete
  • update –> update to the state previous to the import  
  • delete –> restore

The rollback operation works with the Objects that were involved in import and brings them back to the state in which they were before being imported.

3.1. Optional Parameters for Rollback

The list of arguments to use at rollback
-h, --help

Show this help message and exit
--force Force rollback for DB-operations

4. Verify Migration Results

Upon successful Migration, all migrated content will be accessible from the UI.

5. If Migration runs with errors

In case Migration runs with errors:

  1. Verify that all the Migration Prerequisites have been met (For details, check the Script Response and Output).
  2. Having eliminated the cause of errors, rerun the upload script to update Migration results.