Syncing Groups and Users from LDAP/AD using 'mi-ldap-usersync' script

Note: mi-ldap-usersync script needs to be executed from the web container.

Note: Beginning v6.4.3 User Group membership assignment on initial login is controlled by the System Variable INITIAL_USER_LOGIN_EXTERNAL_SYNC. The two available options are:

  • LDAP: References LDAP / Active Directory to identify the User’s Groups  and assigns the User to corresponding Groups in Metric Insights.
  • Custom Script: Assigns Group membership based on a User Access Dataset tied to a Custom Script.

In addition to configuring LDAP/AD authentication in Metric Insights, you can also sync LDAP/AD groups and users by using the following script:

/opt/mi/bin/mi-ldap-usersync

The script is built to sync groups first - and therefore the users in those groups - versus syncing users first. General script logic is as follows:

  • Group: Groups are added to Metric Insights and any users in that group can also be added. Groups created in Metric Insights will be named as they are in the LDAP/AD server. The mapping is 1:1.
  • Users: If a user in a group that is being synced does not yet exist in Metric Insights, the user will be created in MI (if so desired)
  • Default user settings for new users: The type of a user created in MI with mi-ldap-usersync script is specified by the --user-type parameter; if this parameter is not provided, the created user is a regular user with no default settings. Only informational data for the user's profile (user ID, email address, first name, last name) is set, along with group membership.

As an example, if you're syncing all users from an LDAP/AD group called 'SomeADGroup',  mi-ldap-usersync will create a corresponding group called 'SomeADGroup' in Metric Insights. Then, for each user in the LDAP/AD group, mi-ldap-usersync will create a corresponding MI user and make sure the MI user is in the MI group called 'SomeADGroup'.

Subsequent syncs will continue to sync the group and the users in that group as well as adding (if so desired) any new users in that same LDAP/AD group.

Here's what covered in this article:

  1. Provision vs Deprovision
  2. Provisioning Groups and Users
  3. mi-ldap-usersync provision parameters
  4. Deprovisioning Groups and Users
  5. mi-ldap-usersync deprovision parameters
  6. Using standard LDAP search filters to sync for specific Groups and Users
  7. Running mi-ldap-usersync on a Schedule

Provision vs Deprovision

The 'mi-ldap-usersync' script is governed by 2 basic commands: provision and deprovision. Provision creates while deprovision disables / removes users from Metric Insights. Note, you must be careful with deprovisioning users as any inadvertent removals will reassign all of their content. Our general recommendation is provision frequently and deprovision infrequently.

Provisioning Groups and Users

As stated above, provisioning is the action of creating and syncing LDAP/AD groups and users in Metric Insights. Using the mi-ldap-usersync script, here's an example of a very basic provisioning command:

mi-ldap-usersync provision CN=some_group,OU=groups,OU=business_org,DC=global,DC=myorg,DC=com --ldap-host="ldaps://ldap.internal.myorg.com"  --ldap-user="cn=miadmin,cn=users,dc=global,dc=myorg,dc=com" --ldap-pass="password" --verbose

Note that in order provision the group named 'some_group' we need a few bits of information, namely the group's full DN, the LDAP server URL, the LDAP user to query with, and the LDAP user's password. Notice the following abbreviations used in both the group and user names (standard LDAP terminology):

  • CN (Common Name)
  • OU (Organizational Unit)
  • DC (Domain Component)
  • DN (Distinguished Name)

Like a 'tree', these abbreviations are used to define the structure of groups and users, or the full DN (Distinguished Name). In the example above, notice the group name is as follows:

  • CN = some_group
  • OU = groups
  • OU = business_org
  • DC = global
  • DC = myorg
  • DC = com

Using the 'tree' analogy here, we see that myorg.com serves as the root, with subsequent branches defining the organizational unit, groups list, and group name.

We see the same for the LDAP user 'miadmin' and it's full DN:

  • CN = miadmin
  • CN = users
  • DC = global
  • DC =  myorg
  • DC = com

Now that we have a basic understanding of what's required to run mi-ldap-usersync provision, here's a more complex example of a provisioning command where mi-ldap-usersync is run from inside a shell script:

#!/bin/bash
groups="$(cat /opt/mi/scripts/LDAPGroupsToSync.txt)"
host=ldaps://global.myorg.com
ldap_user=cn=miadmin,cn=users,dc=global,dc=myorg,dc=com
ldap_pass="$(cat /opt/mi/custom/pwd.txt)"

echo "MI LDAP sync starting..."
echo $(date '+%Y %b %d %H:%M') running on: $HOSTNAME
echo " "

echo "Group sync starting..."

/opt/mi/bin/mi-ldap-usersync 
provision $groups -H $host --username-attr sAMAccountName --first-name-attr givenName --last-name-attr sn --email-attr userPrincipalName --auto-create groups --ldap-user $ldap_user --ldap-pass $ldap_pass --base-dn='OU=groups,DC=business_org,DC=global,DC=myorg,DC=com'  --full-summary

echo "Group sync completed successfully..."

Notice we've created variables for the groups list file, the LDAP server, the LDAP user, and LDAP user password. Then, before the provisioning command is run, we echo a simple message to help with identifying when the provisioning action starts (if you wish to write the shell script to some log file for tracking).

The actual provisioning command points to the groups list file (variable $groups) and is set to "auto create groups but add only existing users" (--auto-create groups). It's also pointing to the base DN where we would find the groups listed in the groups list file (--base-dn). Without defining the base DN, mi-ldap-usersync is unable to pull back group information.

Next, see a variation of the script above where, instead of syncing groups listed in some text file, we are instead bringing in groups that are filtered for a specific prefix.

#!/bin/bash
host=ldaps://global.myorg.com
ldap_user=cn=miadmin,cn=users,dc=global,dc=myorg,dc=com
ldap_pass="$(cat /opt/mi/custom/pwd.txt)"

echo "MI LDAP sync starting..."
echo $(date '+%Y %b %d %H:%M') running on: $HOSTNAME
echo " "

echo "GRP_* group sync starting..."

/opt/mi/bin/mi-ldap-usersync 
provision -H $host --username-attr sAMAccountName --first-name-attr givenName --last-name-attr sn --email-attr userPrincipalName --auto-create groups --ldap-user $ldap_user --ldap-pass $ldap_pass --base-dn='OU=groups,DC=business_org,DC=global,DC=myorg,DC=com' --chunk-by-alphabet-attr "CN" --chunk-by-alphabet-prefix "GRP_" --full-summary

echo "GRP_* group sync completed successfully..."

Notice here that we are syncing for groups that start with GRP_ (--chunk-by-alphabet-prefix). To lessen the query load on the LDAP/AD server, we're also querying in chunks by applying an alphabetical filter to the group prefix attribute (--chunk-by-alphabet-attr).  

In short, using the available parameters with mi-ldap-usersync provision, you can get quite creative on how you sync groups and users in Metric Insights.

mi-ldap-usersync provision parameters (-h)

Positional Parameter (required)
group_dns The group directories to synchronize with.
Optional parameters
--help, -h show this help message and exit
--ldap-host LDAP_HOST, -H LDAP_HOST Define DNS name or IP address of LDAP server to connect to.
--ldap-user LDAP_USER, -U LDAP_USER The LDAP directory for the bind user to use if your directory requires authentication.
--ldap-pass [LDAP_PASS], -P [LDAP_PASS] Bind password for LDAP directories that require authentication.
--verbose, -v Use this parameter to run the query in verbose mode with diagnostics written in contrast to standard output.
--dry-run, -n Use this parameter to perform a test run of a given query.
--no-summary, -q Do not display summary at the end.
--full-summary Display detailed info about every affected user and group in the summary.
--from-config, -q Load options from the config file.
--yes, -y Assume yes on interactive questions.
--filter FILTER, -f An LDAP filter to search base-dn for groups to sync. Requires --base-dn.
--base-dn BASE_DN, -b The base DN to search for groups in that match using --filter
--user-type {regular,administrator,power_user}, -t {regular,administrator,power_user} Specify the type of users to be added to Metric Insights. (Available options: regular, administrator, power_user)
--member-attr MEMBER_ATTR Specify whether a new user (or multiple users) should become a member of some user group(s).
--username-attr USERNAME_ATTR Define the user attribute name to be used as a Username.
--first-name-attr FIRST_NAME_ATTR Define the user attribute name to be used as user's First Name.
--last-name-attr LAST_NAME_ATTR Define the user attribute name to be used as user's Last Name.
--email-attr EMAIL_ATTR Specify user's email.
--email-domain EMAIL_DOMAIN If the email is not stored in the user's LDAP record, automatically create the email address using the following pattern: username@EMAIL_DOMAIN.
--search-scope SEARCH_SCOPE Specify ldap search scope rule. Possible values: base, sub, one.
--process-child-groups PROCESS_CHILD_GROUPS Enable processing child groups. Possible values: all, only
--create-child-groups PROCESS_CHILD_GROUPS Create separate groups. Skip combining child users from child groups in appropriate parent groups. This option is used only with the option --process-child-groups.
--skip-default-group Skip adding synced users to Default Group.
--trusted-server-username-attr TRUSTED_SERVER_USERNAME_ATTR The field name that stores the trusted server username. (default: None)
--get-groups-from-dataset-id GET_GROUPS_FROM_DATASET_ID Specify the dataset id to ship the LDAP groups. (default: None)
--get-groups-from-dataset-field GET_GROUPS_FROM_DATASET_FIELD Specify the dataset field name to ship the LDAP groups. (default: distinguishedname)
--auto-create [{all,groups}], -a [{all,groups}] If specified without value or 'all' key - automatically create an MI group for the group DN if it doesn't already exist. If specified with 'groups' key - new groups will automatically be created, but only existing users will be updated. If not specified - only users in existing groups will be created and updated.
--chunk-by-alphabet-attr CHUNK_BY_ALPHABET_ATTR Split the base query into chunks applying an alphabetical filter to this group attribute. Ex: (cn=a*), (cn=b*), etc. Requires --base-dn.
--chunk-by-alphabet-prefix CHUNK_BY_ALPHABET_PREFIX Add a prefix for alphabetical chunk search.
--chunk-by-digit-prefix CHUNK_BY_DIGIT_PREFIX Add a prefix for digital chunk search.
--force Force update related objects.
--target-group TARGET_GROUP Create and sync target Metric Insights group instead of creating LDAP similar.
--target-group-mode Choose how to process users for the target group. Combine all members or collect common users only.
--user-filter USER_FILTER Define filter to select users to sync with Metric Insights.
--exclude-users-from EXCLUDE_USERS_FROM Exclude users from a synced group based on LDAP list of groups.
--exclude-users-mode {skip,retain-user-type} Choose how to process excluded users - skip completely or prevent changing user type.
--disable-orphans Disable LDAP users which don't belong to any LDAP synced group.
--remove-orphans Remove LDAP users which don't belong to any LDAP synced group.
--custom-user-attribute CUSTOM_USER_ATTRIBUTE Set list of custom attributes to user sync. Example: attr1,attr2.
--custom-group-attribute CUSTOM_GROUP_ATTRIBUTE Set list of custom attributes to group sync. Example: attr1,attr2.
--map-attr-to-group MAP_ATTR_TO_GROUP Map user to group based on attribute value. Example: attr1,attr2
--group-name-prefix GROUP_NAME_PREFIX Custom group name prefix. Example: eu
--prefix-on-error PREFIX_ON_ERROR Use prefix for group name on creating group error. Example: eu-group. Use --prefix-separator if it is required to change separator. Possible values: dc
--prefix-separator PREFIX_SEPARATOR Prefix separator for generating group name based on prefix type. Default: "-".
--use-batch-processing Use when it is required to process a big amount of users per group.
--skip-failed-users Skip users that are not corresponds Metric Insights requirements.
--skip-disabled-users Skip users that are disabled on LDAP. Disable appropriate Metric Insights users.
--username USERNAME Run sync for specified user only

Deprovisioning Groups and Users

By default, the deprovision command only disables user accounts rather than deleting them. To remove user accounts from the database completely, make sure to use the --delete parameter.

Example:

mi-ldap-usersync deprovision CN=some_group,OU=business_group,DC=global,DC=myorg,DC=com objectClass=person --verbose --ldap-host="ldaps://ldap.internal.myorg.com" --ldap-user="cn=miadmin,cn=users,dc=global,dc=myorg,dc=com" --ldap-pass="password" --delete-orphan-groups --delete-empty-groups --delete

When a user is deleted via ldap-usersync, the system re-assigns Categories and Notification Schedules to Admin with the lowest ID number to make sure content does not get lost.

mi-ldap-usersync deprovision parameters (-h)

Positional Parameters (required)
base_dn The default base directory used for searching required users.
user_filter Seed out users that match the defined filter.
Optional parameters
--help, -h show this help message and exit
--ldap-host LDAP_HOST, -H LDAP_HOST Define DNS name or IP address of LDAP server to connect to.
--ldap-user LDAP_USER, -U LDAP_USER The LDAP directory for the bind user to use if your directory requires authentication.
--ldap-pass [LDAP_PASS], -P [LDAP_PASS] Bind password for LDAP directories that require authentication.
--verbose, -v Run in verbose mode with many diagnostics written in contrast to standard output.
--dry-run, -n Use this parameter to perform a test run of a given query.
--no-summary, -q Do not display summary at the end.
--full-summary Display detailed info about every affected user and group in the summary
--from-config Load options from the config file.
--yes, -y Assume yes on interactive questions.
--delete-orphan-groups Remove any MI group that corresponds to a non-existing LDAP group.
--delete-empty-groups Remove any MI group that is bound to LDAP group but doesn't have any LDAP users associated with it.
--delete DELETE Delete users, not just make their accounts disabled.
--username-attr USERNAME_ATTR The user attribute name to be used as a Username.
--email-attr EMAIL_ATTR The user attribute to be used as for the user's email address.
--email-domain EMAIL_DOMAIN If the email is not stored in the user's LDAP record, automatically create the email address using the following pattern: username@EMAIL_DOMAIN.

Using standard LDAP search filters to sync for specific Groups and Users

If you want to filter for certain groups and users, use standard LDAP search parameters. Here are some examples:

  • User filter: (&(objectClass=user)(sAMAccountName=%s))
    • where %s is the username you want to filter for
  • Filter that verifies a user is NOT in a group being synced: (&(objectClass=user)(sAMAccountName=%s)(!memberOf=cn=GroupName,...))
  • If you want to sync for users in specific groups, you use use the same example above but without the the logical NOT operation (!):
    • (&(objectClass=user)(sAMAccountName=%s)(memberOf=cn=GroupName,...))

For more information, please see LDAP query basics.

Running mi-ldap-usersync on a Schedule

For Metric Insights version 6.1+

For Simple Installs (single server)

The best method is to create a cron job on the host server, similar to the cron job for version 5.x (see below). The only difference is the addition of an additional command to execute the job through the Web container (where mi-ldap-usersync lives).

Contents of /etc/cron.d/ldap-sync:

SHELL=/bin/bash

05 1 * * * root /opt/mi/bin/ldap_usersync_script.sh &> /tmp/mi-ldap-usersync.log

Contents of /opt/mi/bin/ldap_usersync_script.sh:

#!/bin/bash
groups="$(cat /opt/mi/data/custom/scripts/LDAPGroupsToSync.txt)"
host=ldaps://global.myorg.com 
ldap_user=cn=miadmin,cn=users,dc=global,dc=myorg,dc=com
ldap_pass="$(cat /opt/mi/data/custom/pwd.txt)"

/opt/mi/bin/mi-control exec -T web /opt/mi/bin/mi-ldap-usersync provision \
    --ldap-host $host \
    --ldap-user $ldap_user \
    --ldap-pass $ldap_pass \
    --base-dn='OU=groups,DC=business_org,DC=global,DC=myorg,DC=com' \
    --full-summary \
    --auto-create groups \
    $groups \
    --username-attr sAMAccountName \
    --first-name-attr givenName \
    --last-name-attr sn
    --email-attr userPrincipalName \
    -v

Few things to note here:

  • The variables are pointing to text files stored on the host (not inside the Web container)
  • Familiarize yourself with the --auto-create parameter. In the example above, it's set to "auto create groups but add only existing users" (--auto-create groups).
  • The utility used to execute the mi-ldap-usersync provision command is /etc/mi/bin/mi-control. Running /opt/mi/bin/mi-control exec -T web means execute the following command inside the Web container. Our mi-control utility is a wrapper for docker-compose.

For Amazon ECS

Use the ECS Scheduler to run your shell script. The shell script must live on an EFS mounted volume like /opt/mi/data/custom/script. For more information, please see: https://docs.aws.amazon.com/AmazonECS/latest/userguide/scheduled_tasks.html

For Kubernetes

For Kubernetes, use the K8 CronJob service. You will have to create a yaml file for the mi-ldap-usersync job + a .spec section to set the schedule. For more information, please see: https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/

For Metric Insights version 5.x

To run mi-ldap-usersync on a schedule, the best method is to create a shell script to save your usersync command with a list of groups you want to sync. Then, schedule the running of that shell script using cron on the Metric Insights application server.

Create the cron job and save it to /etc/cron.d/ on the application server. Running the usersync overnight is best. You can run it more frequently as well as long as the job does not conflict with data collections, distributions, and active users.

Here's an example of a cron job (/etc/cron.d/ldap-sync) that runs a shell script (saved to /opt/mi/bin) with your stored mi-ldap-usersync command. Notice the job's output is being saved to a log file named mi-ldap-usersync.log (for post run review) and the job is set to run at 1:05 am, daily:

SHELL=/bin/bash

05 1 * * * root /opt/mi/bin/ldap_usersync_script.sh &> /tmp/mi-ldap-usersync.log