The Elements group structure importer - v6.16+

This article relates to group importer functionality in Elements v6.16+. If you are on Elements v6.15, please refer to this support article. 

The group importer allows organisations to programmatically update the group hierarchy in Elements. This means that upstream systems can be used as a source-of-truth for your organisational hierarchy and administrators can make changes to group structure based on this data in bulk. 

Groups in Elements have a key role in structuring user data, underpinning system configuration, and reporting and analytics. The group importer reduces the administrative burden associated with ensuring your organisational hierarchy is kept up-to-date. 

The group importer supports the manual upload of a CSV file containing all data managed by the importer. Elements processes the imported file, a high-level summary is displayed to administrators, and a detailed summary of pending changes is made available to download. To reduce the risk of unintended changes to group structure, manual review and confirmation of changes is required prior to applying the changes. 

Note: The deletion of groups results in any record metadata and object data (e.g. relationships and labels) associated with the deleted group being lost. This is not recoverable. 

We have created a new role to provide granular control over users with permission to manage the group importer. Only users with the ‘Group import Manager’ or ‘System Administrator’ roles can make changes relating to group import functionality. 

Functionality that allows control over the group importer is not displayed to users without these elevated permissions.

The Group import settings page (System admin > Users & groups > Group management > Group import settings) provides administrators with the ability to enable/disable group import functionality, as well as configure group import behaviour and bulk update group properties to support the use of the group importer. 

The External management section on the Group import settings page allows the external management of groups to be enabled and disabled. To enable the external management of groups it is necessary for the top level group in the hierarchy to have an Institutional ID (see below), which can be viewed and set here. 

If external management of groups is enabled, the Group import page will appear in the ‘Group management’ menu under the ‘System Admin’ tab. 

Important: If the group importer is disabled, all externally managed groups will revert to being internally managed. This information will then be lost and would need to be added to the system again.

When the group importer functionality is enabled, all groups in the system can either be ‘externally managed’, or ‘locally managed’. This determines whether certain group properties are managed by the group importer, or continue to be managed via the user interface. The management state of a group is indicated on both the Manage groups page and the group details page. 

When the group importer is first enabled, all existing groups will be set as ‘locally managed’. In order for groups to be managed by the importer, they need to be set to ‘externally managed’. This can be toggled on/off by users with the relevant permission. In order for a group to be externally managed, a unique institutional identifier needs to be added (see below). 

Organisations will usually wish to make the majority (or all) groups ‘externally managed’. It is possible to set the management state of groups in bulk from the ‘Bulk upload IIDs and external management’ section of the Group import settings page (see below). 

If the group importer is enabled, it is still possible to add locally managed groups to the system via the user interface. 

Note: An externally managed group must have an externally managed parent. This means that you cannot change a group to be locally managed if it has externally managed children. The top level group can be locally managed or externally managed. If the top level group is locally managed, it must not appear in the feed file and vice versa. The top level group must have an Institutional ID if the group importer is enabled.

Setting up the group importer is a multistep process. In order to be able to uniquely identify which groups should be managed by the group importer, it is necessary to add an Institutional ID to any group that will be externally managed by the group importer. 

Institutional IDs will often be an ID that is used to identify the group in your upstream system/s. Institutional IDs must be unique and, while they can be added and edited manually via the user interface, it is more likely that you will want to import these from an external system and map them to existing Elements groups. Institutional IDs can be added in bulk from the ‘Bulk upload IIDs and external management’ section of the Group import settings page (see below). 

IIDs can optionally be added to locally managed groups. In this case, they have no functional impact on the group importer and are instead used to aid the identification of groups and/or map between Elements data and external systems. 

When a group import is carried out, the updates to groups will be based on the relationship between the Institutional IDs in the system and the Institutional IDs contained in the feed file as follows: 

For externally managed groups; 

• IID in import file, no matching IID in system - A new group will be created. 

• IID in import file, matching IID in system - The group may be updated based on the data contained in the import file. 

• IID in system, no matching IID in the import file - The group will be deleted 

For internally managed groups; 

• If the import file contains an IID that matches an internally managed group, the import validation will fail.

The ‘Configure groups in bulk’ section on the Group import settings page allows Institutional IDs and the external management of groups to be set in bulk. The ‘Export’ button will download a CSV file containing the Institutional ID and management status of the group. The CSV contains the following columns;

• GroupId - The Elements specific ID of the group. This value is used to determine which group should be updated. 

• GroupParentID - The Elements specific ID of the group's parent. This value is provided for information only. The data in this column is ignored as part of the import process. 

• Name - The Name of the group. This value is provided for information only. The data in this column is ignored as part of the import process. 

• InstitutionalId - The Institutional ID of the group. 

• Externally managed - Yes / No. Whether or not the group is externally managed.

When uploading a new CSV the requirements for Institutional IDs and external management must be followed otherwise the import will be rejected;

• A group cannot be set to be externally managed if it does not have an InstitutionalID 

• An externally managed group cannot have an internally managed parent.

The import must be valid across all groups in order to be successful. E.g. If one of the GroupId values doesn’t match, the whole import will fail. Partial data will not be imported.

If the external management of groups is not enabled it will not be possible to set groups to be Externally managed. 

The group importer allows groups to be created, deleted, moved, and updated. Certain group properties are always be updated via the feed for externally managed groups:

• Name

• Parent (defines the group’s position in the hierarchy)

• Membership model

• Primary group descriptor (for primary groups)

• WHERE clause (for auto groups)

Groups that are externally managed using the group importer cannot be deleted, moved, or have any externally managed group properties updated via the user interface. This ensures that externally managed groups are controlled by the group importer itself and cannot be mistakenly modified via the user interface. 

Externally managed manual groups can have their membership model set via the feed. However, control over the members of a manual group can be controlled via the user interface. This allows manual groups to be managed via the feed, while retaining the flexibility to control members on an adhoc, per group basis. 

It is also possible to opt-in to controlling ‘Description’ and/or ‘Object type’ via the feed. If these externally managed properties are enabled, it is necessary to include these in the import file. 

Note: Group privacy cannot be set by the group importer. The group importer is also not intended to update group record metadata, labels or relationships. 

In order to update the structure of your organisational hierarchy within Elements, it is necessary to create a CSV file that conforms to the specific schema and contains all required fields for externally managed properties.

• InstitutionalId - The Institutional Identifier used to identify an externally managed group. IIDs are case insensitive. Must not be null. 

• Name - The name of the group. Matching between the feed CSV and groups in Elements is not based on the name. Must not be null.

• ParentInstitutionalID - The Institution Identifier used to identify the parent of the group. Parent-child relationships are used to model the group hierarchy. Must not be ‘null’, except in the case of the top-level group. It is only possible for there to be one top-level group. 

• MembershipModel- The membership model of the group. 

  • everyone - only applies to the top level group

  • primary

  • auto

  • manual

• PrimaryGroupDescriptor - The Primary Group Descriptor for any group with MembershipModel = primary. Must be empty otherwise.

• WhereClause - The WhereClause for any group with MembershipModel = auto. Must be empty otherwise.

• ObjectTypeName (Optional) - The identifier name for the group type. Only required if this externally managed property has been enabled.

• Description (Optional) - A text string describing the group. This is displayed on the Group details page. Only required if this externally managed property has been enabled.

Once the initial configuration steps have been completed, it is possible to import your organisational group structure via the ‘Group import’ page. The initial configuration steps will involve: 

1. Importing Institutional Identifiers

2. Enabling the structure importer 

3. Configuring a set of groups to be externally managed. 

Important: It is recommended that you familiarise yourself with the behaviour of the group structure feed on your test instance, prior to carrying out an update on your production instance. 

The Export button allows details on all externally managed groups to be exported from the system. This can be used to compare an import CSV file against the current state of the system. 

A new import of data is initiated by pressing the ‘Start a new import’ button on the Group Import page. 

Import a CSV file that conforms to the required schema. When you click ‘Upload’, the data is added to the system in a ‘staging’ state. This allows the system to calculate the impact of the import process. 

The review and apply step displays the impact on your existing group structure based on the imported data. 

• Total groups before - The total number of groups that currently exist in the system, prior to the import of groups being applied. 

• Total groups after - The total number of groups that will exist after the import has been applied. 

• Additions - The number of new externally managed groups that will be created if the import is applied. 

• Deletions - A count of the number of existing groups that will be deleted if the import is applied. An externally managed group can have locally managed children. When an externally managed group is deleted, any internally managed children will also be deleted.

• Moves - A count of externally managed groups that have had their parent group changed. This does not include any subgroups that have moved branches as a result of their parent moving. 

• Updates - A count of groups that have had either their name, membership model, primary group descriptor, or WHERE clause changed. 

• Membership changes - Returns a count of externally managed groups that will have their explicit group membership changed as a result of the import. 

• System role changes - This section will return information on changes to the number of users who have system roles as a result of the import. The counts correspond to users who have been assigned roles via group membership. Users with explicitly assigned system roles are not included in the count and will not be impacted by the import. 

Count details 

The summary counts are clickable links that allow you to see more details about the groups affected. Clicking these counts will open a modal that will display information about the group affected. This allows rapid spot checking of changes to ensure that the expected groups are affected by the import. 

Download change details  

The change details CSV provides a detailed overview of the changes to every group in the system. Many of the properties and structural changes that can be controlled by the feed are displayed - usually in a pair of ‘before’ vs ‘after’ columns, including a ‘Update - Yes/No’ column. This supports easy filtering of the file to focus specifically on the changes that will be brought about by applying the feed. 

• Membership columns:

  • Explicit Members Before - The count of explicit members of a group prior to the import being applied. Explicit members of a group are those that are directly placed in a group and are not members of a group by virtue of being an explicit member of a subgroup. 

  • Explicit Members After - The net count of explicit members of a group after the import is applied. E.g. due to a WHERE clause of an auto-group being updated as part of the feed. 

  • Explicit Members Added - The number of explicit members that will be added to a group if the import is applied.

  • Explicit Members Removed - The number of explicit members that will be removed from a group if the import is applied.

  • All Members Before - The count of implicit and explicit members of a group prior to the import being applied. Implicit members of a group are members of a group by virtue of being an explicit member of a subgroup. 

  • All Members After - The count of implicit and explicit members of a group after the import is applied. E.g. Moving groups will likely change the implicit members of parent groups. 

  • All Members Added - The number of implicit and explicit members that will be added to a group if the import is applied. 

  • All Members Removed - The number of implicit and explicit members that will be removed from a group if the import is applied. 

• System role columns:

  • This section will return information on changes to the number of users who have system roles as a result of the import. The counts correspond to users who have been assigned roles via group membership. Users with explicitly assigned system roles are not included in the count and will not be impacted by the import. 

  • Each system role is represented in three columns

    • System role: [X] before - A count of users assigned this system role via group membership prior to the import being applied. 

    •  System role: [X] after - A count of users assigned this system role via group membership after the import is applied. 

    • System role: [X] update - Yes/No - Whether or not these two counts differ. This allows quick filtering to determine if system roles will be modified as a result of the import. 

Once you choose to apply the changes, a confirmation model appears to allow users to make a final check before applying. The model will specifically highlight if any groups are going to be deleted. 

Note: The analysis carried out by the system to determine the impact of applying the import file compares the imported group structure with the existing group structure. Any changes to the existing group structure (specifically to the user cache) will invalidate the analysis and it will need to be rerun. We therefore recommend that you carry out checks and apply the changes in a timely manner.

The summary step displays the date and time that the import was completed. At this point a new record will be added to the history tab. Clicking ‘OK’ will reset the page, ready for a new import. 

The history tab displays information on each completed import run, each cancelled import run and any import runs that error. This allows you to keep track of the historical activity of the group importer. For each history item it is possible to download a zip folder of history files. These files are intended to give you a complete picture of the import process. The first three files listed are associated with the ‘Load step’. If this fails, the remaining files will not be present. 

• Load Validation Data - A JSON file containing a summary of any errors that were found while trying to load the input CSV

• Load Notes - A text file outlining the time-stamped events of the import process. 

• GroupImport - A copy of the input CSV file itself

• Group-structure-before-apply - A CSV file containing the group structure immediately before applying the import data. This file includes details of the set of properties managed by the importer for both externally managed and locally managed groups 

• Group-structure-after-apply - As above, but after the changes have been applied. 

• Conclusion Data - If no errors a change summary of the feed, this JSON file includes the object ids and institutional ids of any groups, created, deleted, local deleted, updated or moved. If errors were found when validating the import file against the existing elements group structure, these will be shown here. 

• Conclusion Notes - A text file outlining the outcome of the import process (i.e. ‘Applied’, ‘Cancelled’, ‘Upload failed’).

The time taken for different steps of the group importer to complete will depend on a number of different factors. It is most likely that the ‘Apply’ step will take the longest to complete. The time taken will depend on the type and number of changes being carried out by the feed. Our expectation is that during normal usage (e.g. the deletion/creation/update of approx. 100 groups) the Apply step will take <20s. 

If you are using the group structure importer to bulk add groups to the system as part of an implementation process, or are otherwise making significant one off changes you can expect the Apply step to take significantly longer. E.g. The creation of ~5000 groups could take ~15 minutes to complete. In this circumstance you should ensure the import process is carried outside of a time of significant usage to maintain a performant system.