Hello world code: implementing a user import feed
Please review Managing Users, Elements User Groups, and Managing User Groups before implementing a user import feed.
Implementing a user import feed
Bulk method vs. individual-user approach
There are two alternative approaches to the implementation of a user feed from your HR system to the Elements System.
The first method is to use the user feed partition operations to first clear the user feed table (or the partition thereof that you have been assigned) and then re-upload in bulk your user feed entries. These operations are the DELETE /user-feeds/{partition-id} and POST /user-feeds/{partition-id} operations.
The second method is to add, update and delete user feed entries on a user-by-user basis using the more fine-grained user feed control operations PUT /user-feed/users/{proprietary-id} and DELETE /user-feed/users/{proprietary-id}.
To use the bulk update approach, you need to be assigned a partition ID, which you will use to identify the part of the user feed that you are in control of. Please contact the Elements system administrator at your institution to obtain a partition ID for the part of the user feed that you have been assigned.
To use the individual-user approach, you do not need any partition ID.
If using the bulk-update approach, you must then operate only on the user feed partition assigned to you in all of your calls to the user feed partition operations.
Whichever method you choose, the user feed works internally by storing the details of the user entries that you upload using these operations in a holding table (the user feed table), until such time as the Elements System administrator has configured the user feed data to be processed (typically once a night).
Only once the user feed table has been processed by the system will data-changes to the users in the Elements System become apparent.
When using the bulk-update approach, you should use the POST /user-feeds/{partition-id} operation to add new entries to the user feed table, and use the DELETE /user-feeds/{partition-id} operation to remove all entries in the user feed table previously added by you.
In this way you are in complete control of the records in the user feed table associated with your partition ID. You will not affect entries associated with other partition IDs so long as you only operate on your own assigned feed resource /user-feeds/{partition-id}. If you are one of many bulk-update user feed providers, you will then not affect the user data uploaded by other user feed providers in your institution who use a different partition ID.
When using the individual-user approach, you have fine-grained control over the whole user feed table and must be careful not to delete or modify user-entries uploaded by other feed providers in your institution.
Proprietary IDs
At the scheduled time, the system compares the full list of entries in the user feed table with the list of users of the system, and reconciles the two lists by adding new users to the system for records in the user feed table not matching existing users of the system, by deactivating previously feed-imported users of the system now without a corresponding record in the user feed table, and by updating the user details for existing users with a corresponding record in the user feed table.
The correspondence of users previously uploaded by calls to this operation is achieved by comparing the Proprietary IDs of users with the Proprietary IDs in the user feed table provided in calls to the user feed operations.
The Proprietary ID you provide for each user must therefore be unique amongst all users that have ever been fed to the system, and remain unchanged for each user.
In this way, you (and other user feed providers) are the authority for Proprietary ID values in the Elements System for all the users uploaded using these operations. It is up to you and the Elements system administrator to enforce uniqueness and persistence.
You must work with the Elements system administrator to make sure that Proprietary IDs fed to the Elements System are unique across all data uploaded by the various user feed providers.
If you are the only user feed provider, you must simply make sure that the Proprietary IDs you provide are unique, that they remain unchanged, and that they are not recycled.
If you are not the only user feed provider, you must work with the Elements system administrator to additionally make sure that there can be no possible clashes between the Proprietary IDs provided by you, and the Proprietary IDs provided by other user feed providers.
Avoiding duplicates
Once the user feed table has been processed by the system, its data is not discarded. Rather it is kept, and if no changes are subsequently made to the table by your calls to the user feed operations, then the same data is re-processed the next time the user feed table is processed.
Because the user feed data is kept, if you wish to provide a full and fresh feed of all your users using the bulk-update approach, you must begin by making a call to the DELETE /user-feeds/{partition-id} operation, so that when you upload users with the POST /user-feeds/{partition-id} operation, you are not adding duplicate entries to the user feed table.
This clearing and updating requirement does not apply if you use the individual-user update approach.
Reactivating deactivated users
As an aside, the deactivation of a user from the Elements System by omission from a user feed is not such a drastic occurrence as one might fear, because if the user later reappears in the user feed table, his/her records will be reactivated by the Elements System when the user feed table is next processed.
So do not panic if you accidentally deactivate a swathe of users for a day. The next time you include the deactivated users in your feed to the system, they will reappear in the Elements System with their research data fully intact.
Using a client program to provide the feed
To provide the Elements System with a user feed, you should write a client program that runs regularly, perhaps once a night, to inform the Elements System of the current list of users at your institution (when using the bulk update approach), or to inform the Elements System of any changes to individual users (when using the individual-users approach).
You must arrange with the Elements system administrator the timing of your calls to the user feed operations, in order not to interfere with conflicting operations scheduled for execution in the system, such as the processing of the user feed table itself.
<import-users-request xmlns="http://www.symplectic.co.uk/publications/api">
<users>
<user>
<title>Dr</title>
<initials>JD</initials>
<first-name>June</first-name>
<last-name>Jones</last-name>
<known-as></known-as>
<suffix>FRS</suffix>
<email>somebody@somewhere.org</email>
<authenticating-authority>IC</authenticating-authority>
<username>jonesjd</username>
<proprietary-id>AA1229582</proprietary-id>
<primary-group-descriptor>physics</primary-group-descriptor>
<position>academic</position>
<department>physics</department>
<is-public>true</is-public>
<institutional-email-is-public>true</institutional-email-is-public>
<public-url-path-fragment>jdjones</public-url-path-fragment>
<is-academic>true</is-academic>
<is-login-allowed>true</is-login-allowed>
<is-current-staff>true</is-current-staff>
<arrive-date>2009-02-03</arrive-date>
<generic-field-10>0123456789</generic-field-10>
</user>
<user>
<title>Mr</title>
<initials>TW</initials>
<first-name>Terence</first-name>
<last-name>Smith</last-name>
<known-as>Terry</known-as>
<suffix></suffix>
<email>somebody@somewhere.org</email>
<authenticating-authority>IC</authenticating-authority>
<username>smithtw</username>
<proprietary-id>GH8234623</proprietary-id>
<primary-group-descriptor>mathematics</primary-group-descriptor>
<position>academic</position>
<department>physics</department>
<is-public>true</is-public>
<institutional-email-is-public>false</institutional-email-is-public>
<public-url-path-fragment>twsmith</public-url-path-fragment>
<is-academic>true</is-academic>
<is-login-allowed>false</is-login-allowed>
<is-current-staff>false</is-current-staff>
<arrive-date>2004-02-03</arrive-date>
<leave-date>2009-10-05</leave-date>
<generic-field-10>0123456789</generic-field-10>
</user>
</users>
</import-users-request>The XML above shows an example of the XML document you would supply when adding two hypothetical users to the user feed table using the POST /user-feeds/{partition-id} operation. The same two users could be supplied by two separate calls to the PUT /user-feed/users/{proprietary-id} operation, in which case for example the first call would require you to supply the following document:
<user-feed-entry xmlns="http://www.symplectic.co.uk/publications/api">
<title>Dr</title>
<initials>JD</initials>
<first-name>June</first-name>
<last-name>Jones</last-name>
<known-as></known-as>
<suffix>FRS</suffix>
<email>somebody@somewhere.org</email>
<authenticating-authority>IC</authenticating-authority>
<username>jonesjd</username>
<proprietary-id>AA1229582</proprietary-id>
<primary-group-descriptor>physics</primary-group-descriptor>
<position>academic</position>
<department>physics</department>
<is-public>true</is-public>
<institutional-email-is-public>false</institutional-email-is-public>
<public-url-path-fragment>twsmith</public-url-path-fragment>
<is-academic>true</is-academic>
<is-login-allowed>true</is-login-allowed>
<is-current-staff>true</is-current-staff>
<arrive-date>2009-02-03</arrive-date>
<generic-field-10>0123456789</generic-field-10>
</user-feed-entry>Note that the user and user-feed-entry elements from the two examples above, though named differently, are both of the same XSD element type as defined in the API schema. This type is sensitive to the order in which elements appear. Supplying elements in the wrong order will cause data not to be read, or other errors.
We will explain some of the elements available for user feed entries here. See the API XML schema for additional help.
<known-as>Terry</known-as>The known-as element allows you to provide an informal alternative to the first-name of the user. For example, for a user whose first-name is “Jonathan”, you might provide a known-as value of “Jon”.
<suffix>FRS</suffix>Use the suffix element to provide any official awards or society memberships that might appear as letters after the user’s surname. For example: “CBE FRS”.
<authenticating-authority>IC</authenticating-authority>
<username>jonesjd</username>You must provide the authenticating authority ID and the username of the user.
This combination tells the Elements System which authentication system in your institution authenticates the user’s credentials when logging in to the Elements System, along with their associated username.
The authority/username combination must be unique amongst all users currently supplied through the feed, whether associated with a partition ID or not.
To get the list of available authenticating authority IDs, contact your Elements system administrator.
In the example above, the authenticating authority ID is “IC”. This identifies a system (possibly an LDAP server or a Single Sign-On scheme) to be used by the Elements System when “jonesjd” logs in using her username and password.
The available IDs are configured by the Elements system administrator, and each is associated internally with settings describing how the system is to interact with the authentication system in question.
<proprietary-id>AA1229582</proprietary-id>You must provide the proprietary ID for the user. The proprietary ID must be unique amongst all users ever supplied through the feed, whether associated with your feed ID or not.
The proprietary ID is the only information used by the Elements System to identify the records of previously uploaded users with the records you are now supplying.
If you are supplying the user-entry data using the PUT /user-feed/users/{proprietary-id} operation, the URL to which you PUT the user's data must contain the same proprietary ID as is written in the XML data you provide here in the body of the request.
For example, for this user, the URL for this operation would have to be /user-feed/users/AA1229582
<public-url-path-fragment>jdjones</public-url-path-fragment>If you plan to push Elements data to a public facing website, such as VIVO, Profiles RNS, or Elements Discovery module, you may want to customise the part of the URL that points to a user profile. Typically these systems will use an integer ID to uniquely identify a page that is the user profile. The public-url-path-fragment allows the HR feed to pass a unique string for each user. This string will be passed on to any third party system (as long as they consume it from the Elements API), or in the case of the Elements Discovery module, this string will be used automatically to create user profile URLs, instead of relying on an Elements internal user ID to create unique URLs.
The public URL path fragment is not mandatory, however if a value is passed it must be unique amongst all users ever supplied through the feed, whether associated with your feed ID or not.
A valid value is limited to 50 characters, has to start with a letter, and can contain only alphanumeric, dot, dash, underscore and tilde characters.
<primary-group-descriptor>physics</primary-group-descriptor>Each user is a member of exactly one primary group. If the primary group descriptor is not supplied, the user’s primary group will be the single built-in top-level group, which contains all users.
Please liaise with your Elements system administrator to get the descriptors of the primary groups.
The primary group of a user determines which administrative part of the Elements System the user will belong to and consequently many of the application settings the user will see when they log in.
<is-academic>true</is-academic>The Elements System allows most users to manage their own list of publications and other research data, and can be configured to search online databases for publications and other research data associated with those users.
However, some users of the system are not publishers of research material, and should have search facilities along with the ability to manage their own research data disabled.
This is typical of users who play an administrative role in the Elements System. For these users, specify is-academic to be “false”. For users who should have their own research data, specify “true”.
<is-login-allowed>true</is-login-allowed>Specify "true" if the user should be allowed to log in to the system, and "false" otherwise.
<is-current-staff>true</is-current-staff>Specify "true" if the user is currently a member of staff at your institution, or false if the user has left or has not yet arrived.
<arrive-date>2004-02-03</arrive-date>
<leave-date>2009-10-05</leave-date>If appropriate and known, supply the dates on which the user arrived as a member of staff at your institution, and the date on which they left.
<generic-field-10>0123456789</generic-field-10>The Elements system administrator may have configured the system to use a number of generic data fields for the users of the system.
Please liaise with the system administrator to arrange the injection of appropriate data for these fields.
Please note that the generic fields numbered 11 and above are the "restricted HR data" fields. Data placed in these fields is subject to tighter security when accessed through a secure API endpoint, for example. Please see the following article for more information:
In this example, a phone number has been supplied on the understanding that field number 10 always holds the office phone number of a user.
The following C# code was used to create the example POST /user-feeds/{partition-id}request, above. You can modify this code to write a client that provides a regular user feed to the Elements System using the bulk-update approach.
//declare useful variables - let's assume we have been
//assigned the user feed partition with id "hr"
string requestUrl = "https://localhost:8091/elements-api/user-feeds/hr";
XNamespace ns = "http://www.symplectic.co.uk/publications/api";
//declare two users to be fed to the Elements System
var users = new[] {
new {
Title = "Dr", Initials = "JD", FirstName = "June",
LastName = "Jones", KnownAs = "", Suffix = "FRS",
Email = "somebody@somewhere.org",
Authority = "IC",
Username = "jonesjd", ProprietaryID = "AA1229582",
PrimaryGroupDescriptor = "physics",
InternalPhoneNumber = "0123456789",
LoginAllowed = true,
IsCurrentStaff = true,
ArriveDate = new DateTime(2009, 2, 3),
LeaveDate = (DateTime?)null
},
new {
Title = "Mr", Initials = "TW", FirstName = "Terence",
LastName = "Smith", KnownAs = "Terry", Suffix = "",
Email = "somebody@somewhere.org",
Authority = "IC",
Username = "smithtw", ProprietaryID = "GH8234623",
PrimaryGroupDescriptor = "mathematics",
InternalPhoneNumber = "0123456789",
LoginAllowed = false,
IsCurrentStaff = false,
ArriveDate = new DateTime(2004, 2, 3),
LeaveDate = (DateTime?)new DateTime(2009, 10, 5)
}
};
//prepare the request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Headers.Add(string.Format("Authorization: Basic {0}", Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}:{1}", "username", "password")))));
request.Method = "POST";
request.ContentType = "text/xml";
Stream body = request.GetRequestStream();
using(XmlWriter xmlWriter = XmlWriter.Create(body)) {
XElement requestXml = new XElement(ns + "import-users-request",
new XElement(ns + "users",
from user in users
select new XElement(ns + "user",
new XElement(ns + "title", user.Title),
new XElement(ns + "initials", user.Initials),
new XElement(ns + "first-name", user.FirstName),
new XElement(ns + "last-name", user.LastName),
new XElement(ns + "known-as", user.KnownAs),
new XElement(ns + "suffix", user.Suffix),
new XElement(ns + "email", user.Email),
new XElement(ns + "authenticating-authority", user.Authority),
new XElement(ns + "username", user.Username),
new XElement(ns + "proprietary-id", user.ProprietaryID),
new XElement(ns + "primary-group-descriptor", user.PrimaryGroupDescriptor),
new XElement(ns + "is-academic", true),
new XElement(ns + "is-login-allowed", user.LoginAllowed),
new XElement(ns + "is-current-staff", user.IsCurrentStaff),
new XElement(ns + "arrive-date", user.ArriveDate.Value.ToString("yyyy-MM-dd")),
//for null values, do not include the corresponding element
user.LeaveDate == null ? null : new XElement(ns + "leave-date", user.LeaveDate Value.ToString("yyyy-MM-dd")),
new XElement(ns + "generic-field-10", user.InternalPhoneNumber))));
requestXml.WriteTo(xmlWriter);
}
body.Close();
//get the response
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseContent = reader.ReadToEnd();
The following C# code was used to create the example create/update user in user feed request, above. You can modify this code to write a client that provides a regular user feed to the Elements System using the individual-user update approach.
//declare some useful variables
string userProprietaryID = "AA1229582";
string requestUrl = string.Format("https://localhost:8091/elements-api/user-feed/users/{0}", userProprietaryID);
XNamespace ns = "http://www.symplectic.co.uk/publications/api";
//declare a user to be fed to the Elements System
var user = new {
Title = "Dr", Initials = "JD", FirstName = "June",
LastName = "Jones", KnownAs = "", Suffix = "FRS",
Email = "somebody@somewhere.org",
Authority = "IC",
Username = "jonesjd",
ProprietaryID = userProprietaryID,
PrimaryGroupDescriptor = "physics",
InternalPhoneNumber = "0123456789",
LoginAllowed = true,
IsCurrentStaff = true,
ArriveDate = new DateTime(2009, 2, 3)
};
//prepare the request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Headers.Add(string.Format("Authorization: Basic {0}", Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}:{1}", "username", "password")))));
request.Method = "PUT";
request.ContentType = "text/xml";
Stream body = request.GetRequestStream();
using(XmlWriter xmlWriter = XmlWriter.Create(body)) {
XElement requestXml = new XElement(ns + "user",
new XElement(ns + "title", user.Title),
new XElement(ns + "initials", user.Initials),
new XElement(ns + "first-name", user.FirstName),
new XElement(ns + "last-name", user.LastName),
new XElement(ns + "known-as", user.KnownAs),
new XElement(ns + "suffix", user.Suffix),
new XElement(ns + "email", user.Email),
new XElement(ns + "authenticating-authority", user.Authority),
new XElement(ns + "username", user.Username),
new XElement(ns + "proprietary-id", user.ProprietaryID),
new XElement(ns + "primary-group-descriptor", user.PrimaryGroupDescriptor),
new XElement(ns + "is-academic", true),
new XElement(ns + "is-login-allowed", user.LoginAllowed),
new XElement(ns + "is-current-staff", user.IsCurrentStaff),
new XElement(ns + "arrive-date", user.ArriveDate.ToString("yyyy-MM-dd")),
new XElement(ns + "generic-field-10", user.InternalPhoneNumber));
requestXml.WriteTo(xmlWriter);
}
body.Close();
//get the response
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseContent = reader.ReadToEnd();During the processing of the user feed table, the Elements System may discover duplicate entries for users (i.e. more than one entry with the same user proprietary ID), or find other problems with the data supplied by a feed.
These errors will be reported to the Elements system administrator, who may need to contact you to have you correct your feed.