How to implement an API feed

TABLE OF CONTENTS

Implementing a feed to a website

As an example, consider a hypothetical implementation of a web-content data feed from the Elements System to your institution’s web content management system.

The goal would be to show a summary of each of your institution’s employee’s publications through your institution’s public-facing website.

To this end a simple client program would be written by you to consume the API. The client program would run perhaps once per night at a time agreed with the Elements system administrator to minimise disruption to other users of the Elements System.

The client program would get a list of all users in the system from the API, then loop through them to search for and obtain publication-lists for the users of the Elements System for whom an externally facing web-page exists.

The returned publication data would be cached by the client program in a place suitable for use by the content management system until the cache is refreshed the next time the client runs.

It would be possible to dynamically issue API requests to generate page content for the visitors of your site at the times it is required, but this approach is bad for a number of reasons.

It makes unnecessary use of network resources and of the Elements Systems itself, by re-fetching what is largely static data (on the timescale of hours or days) every time it is viewed.

To avoid exposing the Elements System to a potentially unlimited amount of usage, you should cache data such as this in your local system to isolate the Elements System from high volumes of repeated high priority data reads.

Although the API is reasonably architected to return data quickly, its purpose is not to act as a performance-critical search engine in a web environment.

The following short example shows a simple C# program designed to loop through all of the users in the Elements System and print all of their publication titles.

Note that the program can be written in any language that offers a web client programming model, such as Python, Ruby or Java, and can be modified to cache publication lists in your system.

//initialise useful variables
XNamespace ns = "http://www.symplectic.co.uk/publications/api";
int pageSize = 100;
int pageNumber = 0;
int pageResultsCount;

//loop through the pages of users
do {
//update page variables
pageResultsCount = 0;
pageNumber++;
string requestUrl = string.Format("https://localhost:8091/elements-api/ users&amp;amp;per-page={0}&amp;amp;page={1}", pageSize, pageNumber);

//prepare the request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Headers.Add(string.Format("Authorization: Basic {0}",  Convert.ToBase64String(Encoding.UTF8.GetBytes( string.Format("{0}:{1}", "username", "password")))));

//get the response
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
XmlReader reader = XmlReader.Create(responseStream);
while(reader.Read() &amp;amp;&amp;amp; reader.NodeType != XmlNodeType.Element) ;
XElement responseXml = (XElement)XNode.ReadFrom(reader);

//get the user ids
var userIDs = from apiObject in responseXml.Descendants(ns + "object")
where apiObject.Attribute("category").Value == "user"
select apiObject.Attribute("id").Value;

//loop through the users on this user page
foreach(var userID in userIDs) {
//keep track of the number of users on this page of users
pageResultsCount++;

//initialise useful variables
int pubsPageNumber = 0;
int pubsPageResultsCount;

//loop through the pages of publications
do {
//update page variables
pubsPageResultsCount = 0;
pubsPageNumber++;
string pubsRequestUrl = string.Format("https://localhost:8091/elements-api/users /{0}/publications?types=8&amp;amp;detail=full&amp;amp;per-page={1}&amp;amp;page={2}", userID,  pageSize, pubsPageNumber);

//prepare a request to get the publications of the user
HttpWebRequest pubsRequest = (HttpWebRequest)WebRequest.Create(pubsRequestUrl);
pubsRequest.Headers.Add(string.Format("Authorization: Basic {0}",  Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}:{1}",  "username", "password")))));

//get the response
HttpWebResponse pubsResponse = (HttpWebResponse)pubsRequest.GetResponse();
Stream pubsResponseStream = pubsResponse.GetResponseStream();
XmlReader pubsReader = XmlReader.Create(pubsResponseStream);
while(pubsReader.Read() &amp;amp;&amp;amp; pubsReader.NodeType != XmlNodeType.Element) ;
XElement pubsResponseXml = (XElement)XNode.ReadFrom(pubsReader);

//get the publication titles for the publications on this publications page
var titles = from publication in pubsResponseXml.Descendants(ns + "object")
let title = publication.Descendants(ns + "field").Where(f =>  f.Attribute("name").Value == "title").First()
select title.Element(ns + "text").Value;

//print them to screen
foreach(string title in titles) {
Console.WriteLine(title);
pubsPageResultsCount++;
}
} while(pubsPageResultsCount == pageSize);

Please note that the listing above is an example only, and does not contain all the code required of a production version of a data-fetch program.

You should include error handling, logging, breaks in processing to prevent overloading of the server, robustness against server response failure, null reference checking and other things generally required of a reliable program.

When implementing your own web feed that displays publication information (or any research information) in the context of particular users, do not forget to test and take heed of the "is-visible" property of each related object (see the API schema documentation for more information).

This information is there for a very important reason and provides the only mechanism available for the user to indicate when they do not wish certain relationships to be made publicly viewable.

This can be an important issue for researchers publishing in the area of animal experimentation or other sensitive areas of research.

Implementing a user import feed

There are two alternative approaches to the implementation of a user feed from your HR system to the Elements System.

The first method is to use the user feed partition operations to first clear the user feed table (or the partition thereof that you have been assigned) and then re-upload in bulk your user feed entries. These operations are the DELETE /user-feeds/{partition-id} and POST /user-feeds/{partition-id} operations.
The second method is to add, update and delete user feed entries on a user-by-user basis using the more fine-grained user feed control operations PUT /user-feed/users/{proprietary-id} and DELETE /user-feed/users/{proprietary-id}.

To use the bulk update approach, you need to be assigned a partition ID, which you will use to identify the part of the user feed that you are in control of. Please contact the Elements system administrator at your institution to obtain a partition ID for the part of the user feed that you have been assigned.

To use the individual-user approach, you do not need any partition ID.

If using the bulk-update approach, you must then operate only on the user feed partition assigned to you in all of your calls to the user feed partition operations.

Whichever method you choose, the user feed works internally by storing the details of the user entries that you upload using these operations in a holding table (the user feed table), until such time as the Elements System administrator has configured the user feed data to be processed (typically once a night).

Only once the user feed table has been processed by the system will data-changes to the users in the Elements System become apparent.

When using the bulk-update approach, you should use the POST /user-feeds/{partition-id} operation to add new entries to the user feed table, and use the DELETE /user-feeds/{partition-id} operation to remove all entries in the user feed table previously added by you.

In this way you are in complete control of the records in the user feed table associated with your partition ID. You will not affect entries associated with other partition IDs so long as you only operate on your own assigned feed resource /user-feeds/{partition-id}. If you are one of many bulk-update user feed providers, you will then not affect the user data uploaded by other user feed providers in your institution who use a different partition ID.

When using the individual-user approach, you have fine-grained control over the whole user feed table and must be careful not to delete or modify user-entries uploaded by other feed providers in your institution.

At the scheduled time, the system compares the full list of entries in the user feed table with the list of users of the system, and reconciles the two lists by adding new users to the system for records in the user feed table not matching existing users of the system, by deactivating previously feed-imported users of the system now without a corresponding record in the user feed table, and by updating the user details for existing users with a corresponding record in the user feed table.

The correspondence of users previously uploaded by calls to this operation is achieved by comparing the Proprietary IDs of users with the Proprietary IDs in the user feed table provided in calls to the user feed operations.

The Proprietary ID you provide for each user must therefore be unique amongst all users that have ever been fed to the system, and remain unchanged for each user.

In this way, you (and other user feed providers) are the authority for Proprietary ID values in the Elements System for all the users uploaded using these operations. It is up to you and the Elements system administrator to enforce uniqueness and persistence.

You must work with the Elements system administrator to make sure that Proprietary IDs fed to the Elements System are unique across all data uploaded by the various user feed providers.

If you are the only user feed provider, you must simply make sure that the Proprietary IDs you provide are unique, that they remain unchanged, and that they are not recycled.

If you are not the only user feed provider, you must work with the Elements system administrator to additionally make sure that there can be no possible clashes between the Proprietary IDs provided by you, and the Proprietary IDs provided by other user feed providers.

Once the user feed table has been processed by the system, its data is not discarded. Rather it is kept, and if no changes are subsequently made to the table by your calls to the user feed operations, then the same data is re-processed the next time the user feed table is processed.

Because the user feed data is kept, if you wish to provide a full and fresh feed of all your users using the bulk-update approach, you must begin by making a call to the DELETE /user-feeds/{partition-id} operation, so that when you upload users with the POST /user-feeds/{partition-id} operation, you are not adding duplicate entries to the user feed table.

This clearing and updating requirement does not apply if you use the individual-user update approach.

As an aside, the deactivation of a user from the Elements System by omission from a user feed is not such a drastic occurrence as one might fear, because if the user later reappears in the user feed table, his/her records will be reactivated by the Elements System when the user feed table is next processed.

So do not panic if you accidentally deactivate a swathe of users for a day. The next time you include the deactivated users in your feed to the system, they will reappear in the Elements System with their research data fully intact.

To provide the Elements System with a user feed, you should write a client program that runs regularly, perhaps once a night, to inform the Elements System of the current list of users at your institution (when using the bulk update approach), or to inform the Elements System of any changes to individual users (when using the individual-users approach).

You must arrange with the Elements system administrator the timing of your calls to the user feed operations, in order not to interfere with conflicting operations scheduled for execution in the system, such as the processing of the user feed table itself.

<import-users-request xmlns="http://www.symplectic.co.uk/publications/api">
<users>
<user>
<title>Dr</title>
<initials>JD</initials>
<first-name>June</first-name>
<last-name>Jones</last-name>
<known-as></known-as>
<suffix>FRS</suffix>
<email>somebody@somewhere.org</email>
<authenticating-authority>IC</authenticating-authority>
<username>jonesjd</username>
<proprietary-id>AA1229582</proprietary-id>
<primary-group-descriptor>physics</primary-group-descriptor>
<is-academic>true</is-academic>
<is-login-allowed>true</is-login-allowed>
<is-current-staff>true</is-current-staff>
<arrive-date>2009-02-03</arrive-date>
<generic-field-10>0123456789</generic-field-10>
</user>
<user>
<title>Mr</title>
<initials>TW</initials>
<first-name>Terence</first-name>
<last-name>Smith</last-name>
<known-as>Terry</known-as>
<suffix></suffix>
<email>somebody@somewhere.org</email>
<authenticating-authority>IC</authenticating-authority>
<username>smithtw</username>
<proprietary-id>GH8234623</proprietary-id>
<primary-group-descriptor>mathematics</primary-group-descriptor>
<is-academic>true</is-academic>
<is-login-allowed>false</is-login-allowed>
<is-current-staff>false</is-current-staff>
<arrive-date>2004-02-03</arrive-date>
<leave-date>2009-10-05</leave-date>
<generic-field-10>0123456789</generic-field-10>
</user>
</users>
</import-users-request>

The XML above shows an example of the XML document you would supply when adding two hypothetical users to the user feed table using the POST /user-feeds/{partition-id} operation. The same two users could be supplied by two separate calls to the PUT /user-feed/users/{proprietary-id} operation, in which case for example the first call would require you to supply the following document:

<user-feed-entry xmlns="http://www.symplectic.co.uk/publications/api">
<title>Dr</title>
<initials>JD</initials>
<first-name>June</first-name>
<last-name>Jones</last-name>
<known-as></known-as>
<suffix>FRS</suffix>
<email>somebody@somewhere.org</email>
<authenticating-authority>IC</authenticating-authority>
<username>jonesjd</username>
<proprietary-id>AA1229582</proprietary-id>
<primary-group-descriptor>physics</primary-group-descriptor>
<is-academic>true</is-academic>
<is-login-allowed>true</is-login-allowed>
<is-current-staff>true</is-current-staff>
<arrive-date>2009-02-03</arrive-date>
<generic-field-10>0123456789</generic-field-10>
</user-feed-entry>

Note that the "user" and "user-feed-entry" elements from the two examples above, though named differently, are both of the same XSD element type as defined in the API schema. This type is sensitive to the order in which elements appear. Supplying elements in the wrong order will cause data not to be read, or other errors.

We will explain some of the elements available for user feed entries here. See the API XML schema for additional help.

<known-as>Terry</known-as>

The “known-as” element allows you to provide an informal alternative to the first-name of the user. For example, for a user whose “first-name” is “Jonathan”, you might provide a “known-as” value of “Jon”.

<suffix>FRS</suffix>

Use the “suffix” element to provide any official awards or society memberships that might appear as letters after the user’s surname. For example: “CBE FRS”.

<authenticating-authority>IC</authenticating-authority>
<username>jonesjd</username>

You must provide the authenticating authority ID and the username of the user.

This combination tells the Elements System which authentication system in your institution authenticates the user’s credentials when logging in to the Elements System, along with their associated username.

The authority/username combination must be unique amongst all users currently supplied through the feed, whether associated with a partition ID or not.

To get the list of available authenticating authority IDs, contact your Elements system administrator.

In the example above, the authenticating authority ID is “IC”. This identifies a system (possibly an LDAP server or a Single Sign-On scheme) to be used by the Elements System when “jonesjd” logs in using her username and password.

The available IDs are configured by the Elements system administrator, and each is associated internally with settings describing how the system is to interact with the authentication system in question.

<proprietary-id>AA1229582</proprietary-id>

You must provide the proprietary ID for the user. The proprietary ID must be unique amongst all users ever supplied through the feed, whether associated with your feed ID or not.

The proprietary ID is the only information used by the Elements System to identify the records of previously uploaded users with the records you are now supplying.

If you are supplying the user-entry data using the PUT /user-feed/users/{proprietary-id} operation, the URL to which you PUT the user's data must contain the same proprietary ID as is written in the XML data you provide here in the body of the request.

For example, for this user, the URL for this operation would have to be /user-feed/users/AA1229582

<primary-group-descriptor>physics</primary-group-descriptor>

Each user is a member of exactly one primary group. If the primary group descriptor is not supplied, the user’s primary group will be the single built-in top-level group, which contains all users.

Please liaise with your Elements system administrator to get the descriptors of the primary groups.

The primary group of a user determines which administrative part of the Elements System the user will belong to and consequently many of the application settings the user will see when they log in.

<is-academic>true</is-academic>

The Elements System allows most users to manage their own list of publications and other research data, and can be configured to search online databases for publications and other research data associated with those users.

However, some users of the system are not publishers of research material, and should have search facilities along with the ability to manage their own research data disabled.

This is typical of users who play an administrative role in the Elements System. For these users, specify “is-academic” to be “false”. For users who should have their own research data, specify “true”.

<is-login-allowed>true</is-login-allowed>

Specify "true" if the user should be allowed to log in to the system, and "false" otherwise.

<is-current-staff>true</is-current-staff>

Specify "true" if the user is currently a member of staff at your institution, or false if the user has left or has not yet arrived.

<arrive-date>2004-02-03</arrive-date>
<leave-date>2009-10-05</leave-date>

If appropriate and known, supply the dates on which the user arrived as a member of staff at your institution, and the date on which they left.

<generic-field-10>0123456789</generic-field-10>

The Elements system administrator may have configured the system to use a number of “generic” data fields for the users of the system.

Please liaise with the system administrator to arrange the injection of appropriate data for these fields.

Please note that the generic fields numbered 11 and above are defined to be sensitive HR data fields. Data placed in these fields is subject to tighter security when accessed through a secure API endpoint, for example. See the Security section for more details.

In this example, a phone number has been supplied on the understanding that field number 10 always holds the office phone number of a user.

The following C# code was used to create the example POST /user-feeds/{partition-id} request, above. You can modify this code to write a client that provides a regular user feed to the Elements System using the bulk-update approach.

//declare useful variables - let's assume we have been 
//assigned the user feed partition with id "hr"
string requestUrl = "https://localhost:8091/elements-api/user-feeds/hr";
XNamespace ns = "http://www.symplectic.co.uk/publications/api";

//declare two users to be fed to the Elements System
var users = new[] {
new {
Title = "Dr", Initials = "JD", FirstName = "June",
LastName = "Jones", KnownAs = "", Suffix = "FRS",
Email = "somebody@somewhere.org",
Authority = "IC",
Username = "jonesjd", ProprietaryID = "AA1229582",
PrimaryGroupDescriptor = "physics",
InternalPhoneNumber = "0123456789",
LoginAllowed = true,
IsCurrentStaff = true,
ArriveDate = new DateTime(2009, 2, 3),
LeaveDate = (DateTime?)null
},
new {
Title = "Mr", Initials = "TW", FirstName = "Terence",
LastName = "Smith", KnownAs = "Terry", Suffix = "",
Email = "somebody@somewhere.org",
Authority = "IC",
Username = "smithtw", ProprietaryID = "GH8234623",
PrimaryGroupDescriptor = "mathematics",
InternalPhoneNumber = "0123456789",
LoginAllowed = false,
IsCurrentStaff = false,
ArriveDate = new DateTime(2004, 2, 3),
LeaveDate = (DateTime?)new DateTime(2009, 10, 5)
}
};

//prepare the request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Headers.Add(string.Format("Authorization: Basic {0}",  Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}:{1}",  "username", "password")))));
request.Method = "POST";
request.ContentType = "text/xml";
Stream body = request.GetRequestStream();
using(XmlWriter xmlWriter = XmlWriter.Create(body)) {
XElement requestXml = new XElement(ns + "import-users-request",
new XElement(ns + "users",
from user in users
select new XElement(ns + "user",
new XElement(ns + "title", user.Title),
new XElement(ns + "initials", user.Initials),
new XElement(ns + "first-name", user.FirstName),
new XElement(ns + "last-name", user.LastName),
new XElement(ns + "known-as", user.KnownAs),
new XElement(ns + "suffix", user.Suffix),
new XElement(ns + "email", user.Email),
new XElement(ns + "authenticating-authority", user.Authority),
new XElement(ns + "username", user.Username),
new XElement(ns + "proprietary-id", user.ProprietaryID),
new XElement(ns + "primary-group-descriptor", user.PrimaryGroupDescriptor),
new XElement(ns + "is-academic", true),
new XElement(ns + "is-login-allowed", user.LoginAllowed),
new XElement(ns + "is-current-staff", user.IsCurrentStaff),
new XElement(ns + "arrive-date", user.ArriveDate.Value.ToString("yyyy-MM-dd")),
//for null values, do not include the corresponding element
user.LeaveDate == null ? null : new XElement(ns + "leave-date",  user.LeaveDate Value.ToString("yyyy-MM-dd")),
new XElement(ns + "generic-field-10", user.InternalPhoneNumber))));


requestXml.WriteTo(xmlWriter);
}

body.Close();

//get the response
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseContent = reader.ReadToEnd();


The following C# code was used to create the example create/update user in user feed request, above. You can modify this code to write a client that provides a regular user feed to the Elements System using the individual-user update approach.
//declare some useful variables
string userProprietaryID = "AA1229582";
string requestUrl = string.Format("https://localhost:8091/elements-api/user-feed/ users/{0}", userProprietaryID);
XNamespace ns = "http://www.symplectic.co.uk/publications/api";

//declare a user to be fed to the Elements System
var user = new {
Title = "Dr", Initials = "JD", FirstName = "June",
LastName = "Jones", KnownAs = "", Suffix = "FRS",
Email = "somebody@somewhere.org",
Authority = "IC",
Username = "jonesjd", 
ProprietaryID = userProprietaryID,
PrimaryGroupDescriptor = "physics",
InternalPhoneNumber = "0123456789",
LoginAllowed = true,
IsCurrentStaff = true,
ArriveDate = new DateTime(2009, 2, 3)
};

//prepare the request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Headers.Add(string.Format("Authorization: Basic {0}",  Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}:{1}",  "username", "password")))));
request.Method = "PUT";
request.ContentType = "text/xml";
Stream body = request.GetRequestStream();
using(XmlWriter xmlWriter = XmlWriter.Create(body)) {
XElement requestXml = new XElement(ns + "user",
new XElement(ns + "title", user.Title),
new XElement(ns + "initials", user.Initials),
new XElement(ns + "first-name", user.FirstName),
new XElement(ns + "last-name", user.LastName),
new XElement(ns + "known-as", user.KnownAs),
new XElement(ns + "suffix", user.Suffix),
new XElement(ns + "email", user.Email),
new XElement(ns + "authenticating-authority", user.Authority),
new XElement(ns + "username", user.Username),
new XElement(ns + "proprietary-id", user.ProprietaryID),
new XElement(ns + "primary-group-descriptor", user.PrimaryGroupDescriptor),
new XElement(ns + "is-academic", true),
new XElement(ns + "is-login-allowed", user.LoginAllowed),
new XElement(ns + "is-current-staff", user.IsCurrentStaff),
new XElement(ns + "arrive-date", user.ArriveDate.ToString("yyyy-MM-dd")),
new XElement(ns + "generic-field-10", user.InternalPhoneNumber));
requestXml.WriteTo(xmlWriter);
}
body.Close();

//get the response
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseContent = reader.ReadToEnd();

During the processing of the user feed table, the Elements System may discover duplicate entries for users (i.e. more than one entry with the same user proprietary ID), or find other problems with the data supplied by a feed.

These errors will be reported to the Elements system administrator, who may need to contact you to have you correct your feed.

Implementing a data import feed

The Symplectic Elements API provides operations that allow you to import your research data into the system in a manner that is designed to closely resemble the model used to import data from the various online data sources such as Scopus and PubMed.

Implementing such a data feed is relatively straightforward, though you should review the section Best Practices to help you to implement a reliable and easily debuggable data feed into the Elements System.

The current version of the import operations support both a one-off import and subsequent updates of activities, equipment, grants, organisational structures, projects and publications.

Additionally, the operations allow you to identify which users are to be associated with the data you have imported, and in what way (e.g. which users of the system are the primary investigators of the grants you have imported).

A typical example of usage of this functionality would be to implement a nightly feed from an authoritative grants database in your institution, keeping the Elements System up-to-date with the grants data managed by that system.

To remain consistent with the way data enters the system when external data sources such as the Web of Science are trawled, the broad approach taken to data import observes the following steps.

You are a data source

Your feeding system acts as if it were one of the external data sources registered within the Elements System.

This is simply a matter of you choosing the data source identifier for an appropriate data source already configured in the Elements System, and then supplying that identifier to the operations you use to upload data. All records uploaded through the API will then be associated with the indicated data source.

In order to prevent you from overwriting data that has come from another data source, not all data sources are open for use in this way. See the API schema for more information about data sources and records.

Feed records into the system

You maintain control of the record data you feed from your data source, and the Elements System and its users collect your records into objects with any matching records from other data sources.

This approach works in exactly the same way as (for example) PubMed data and Scopus data already enter the system: PubMed and Scopus independently provide their publication records, and the Elements System and its users aggregate these where appropriate into the same publication object, without the interaction of PubMed or Scopus.

In this way, the Elements System and its users (and not the source feeding the data) remain in control of the de-duplication of data, and the feeding system remains in control of the records it feeds into the system.

You can upload new records, and update previously uploaded records, using the PUT /{cat}/records/{source}/{proprietary-id} operation.

Feed associations to users into the system

You can also declare associations between two objects in the system that you have previously uploaded, using the POST /relationships operation.

Your associations are stored as relationships between two objects. Each association you provide will be interpreted and stored as a relationship between the objects you have identified.

Based on the association data you provide, the system will decide whether to create a new relationship, or update an existing relationship in order to model your association, always returning the representation of the relationship created or updated.

Update previously imported data

Subsequent import of the same record into the system will simply update the record, not create a new one. The record is identified by the combination of data-source and record proprietary ID that you specify for the record when importing it.

Subsequent import of the same association will achieve the same effect. The relevant relationship in the system will be updated. The relationship is identified by two object identifiers and the type of relationship you specify. Since the Elements system can hold a maximum of one relationship for any given combination of these three values, the correct relationship will be updated.

Deleting previously imported data

The API exposes operations that allow you to delete previously imported records and relationships. Once data is deleted, it is deleted, and is not recoverable except by restoring a complete database backup image.

It is strongly advised that you do not delete previously imported data, and that instead you fix any data problems using manual techniques. Discuss the situation carefully with your system administrator before committing to deleting data, whether as a one-off, or on a regular basis.

Nevertheless, you may wish to automate some deletions, and operations to allow you to achieve this are provided for completeness. If you do use API operations to delete records or relationships, you should make all the necessary data backups before commencing, and conduct a suitable review of data accuracy afterwards.

Walkthrough

This section walks you through a hypothetical implementation of a full data feed, upon which you can base your own data feeds.

The supposed situation is that of an independent grants system embedded in your institution (the "grants system"), from which a nightly data feed is to regularly update the Elements System with all of its grants data. This data includes both the grants and the relationships of those grants to the staff in your institution.

Equally, this walkthrough could be applied just as easily to an independent publications system, to feed publications data into the Elements system, or an independent projects and staff system, to feed project data into the Elements System.

The steps used for this implementation are:

Arrange access to the API
Select the data source that represents your system
Decide how you want to map your grants to Elements grants
Implement an importer program to import your grants
Decide how you want to associate users to your grants
Extend your importer program to import your associations
Liaise with the system administrator to decide when to run your importer

1: Arrange access to the API

Your importer program will run from a dedicated machine with a known IP address and using a dedicated API account with which to connect to the API.

You check with the system administrator that you have read/write access to the Elements System's API (by registering your machine's IP address with them and getting the credentials of an API account with read/write permissions on the Elements System).

2: Select the data source that represents your system

Using the /grant/sources resource, you see that of the grants data sources currently configured in the Elements System, the source with name "source-3" is an importable data source.

This means that you can use it to represent the grants system from which you will implement a data feed. This decision is agreed between you and the Elements system administrator and it is decided that this data source, not having been used to represent any other data, will represent the grants system.

Any data source not registered in the Elements system as an importable data source has effectively been locked from write operations through the API, protecting its data in the Elements system from being overwritten or added to by importers such as yourself.

3: Decide how you want to map your grants to Elements grants

You must decide how to translate a grant that exists in your grants system into a grant in the Elements System.

After performing a review of the grant data fields available in your grants system, and the grant types and their data fields currently configured in the Elements System (using the /grant/types resource), you must arrive at a desired mapping for converting your grant system's grants data to Elements grants data.

To keep things simple, let us suppose that all grants in your independent grants system will be mapped to the single grant type configured by your institution in the Elements System. Let us suppose that the /grant/types resource returns the following type data:

<api:type id="1" name="grant">
<api:heading-singular>Research Grant</api:heading-singular>
<api:heading-plural>Research Grants</api:heading-plural>
<api:heading-lowercase-singular>research grant</api:heading-lowercase-singular>
<api:heading-lowercase-plural>research grants</api:heading-lowercase-plural>
<api:fields>
<api:field>
<api:name>funder-reference</api:name>
<api:display-name>Funder reference</api:display-name>
<api:type>text</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>funder-name</api:name>
<api:display-name>Funder name</api:display-name>
<api:type>text</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>title</api:name>
<api:display-name>Title</api:display-name>
<api:type>text</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>description</api:name>
<api:display-name>Description</api:display-name>
<api:type>text</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>start-date</api:name>
<api:display-name>Start date</api:display-name>
<api:type>date</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>end-date</api:name>
<api:display-name>End date</api:display-name>
<api:type>date</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>amount</api:name>
<api:display-name>Amount</api:display-name>
<api:type>money</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>institution-reference</api:name>
<api:display-name>Institution reference</api:display-name>
<api:type>text</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>application-date</api:name>
<api:display-name>Application date</api:display-name>
<api:type>date</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>award-date</api:name>
<api:display-name>Award date</api:display-name>
<api:type>date</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>status</api:name>
<api:display-name>Status</api:display-name>
<api:type>text</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
<api:field>
<api:name>funder-type</api:name>
<api:display-name>Funder type</api:display-name>
<api:type>text</api:type>
<api:is-mandatory>false</api:is-mandatory>
</api:field>
</api:fields>
</api:type>

There are plenty of fields available, but if you wish to store data in additional specific fields, you must arrange with your Elements system administrator to add the required fields to the desired grant types. We will assume that this has already been achieved.

Let us suppose that the grants data in your independent grants system belong to a very simple pair of database tables with the following schemas:

tblGrant

Column name	Column type	Properties
ID	int	primary key
Date	datetime	nullable
Value	int	nullable
Notes	varchar(200)	nullable

tblGrantStaff

Column name	Column type	Properties
Grant_ID	int	foreign key to tblGrant(ID)
Staff_ID	varchar(50)	not nullable

tblGrant holds the grants themselves, and tblGrantStaff holds a list of staff member associations to the grants in tblGrant, in a standard relational database way, where each association represents the "primary investigator" for a grant. The Staff_ID column it is assumed uses the same identifiers as those stored as user proprietary IDs in the Elements System, and we assume that the data is clean, accurate and suitably free of duplicates.

If tblGrantStaff's Staff_ID column stored instead the equivalent of the Elements System's usernames instead of user proprietary IDs, you would still be able to import your data using usernames (see later).

Comparing these tables with the XML type information above, you decide on the following mapping:

tblGrant column name	Elements system grant field
ID	record proprietary ID
Date	start-date
Value	amount
Notes

Note that the ID of the grant in your independent grant system is mapped to the Elements System record proprietary ID. Although the proprietary ID is not defined as a field for the grant type in the grant type XML, all records in the Elements System have proprietary IDs that represent the ID of the grant record as given to it by the data source from which it came, and so you must assign a proprietary ID to every record you import that can be uniquely traced back to the record from which it came in the external system.

The ID of the row from which the grant originally came in your grants system is assumed to be such an ideal identifier in this scenario, though it is up to you to appropriately choose which value is mapped to the record proprietary ID in the Elements system.

The proprietary ID that you assign when re-importing the same grant into the Elements System must remain unique and immutable, since the import of a grant with a changed proprietary ID would be considered the import of an entirely new grant by the Elements system.

4: Implement an importer program to import your grants

The operation to call is the PUT /grant/records/source-3/{proprietary-id} operation. This operation will create a new record with the indicated proprietary ID if it doesn't already exist, or update the existing record with the data you supply if it does.

The following code snippet shows example code that imports two grants into the Elements system. The grants data is fixed, though you should instead implement a loop that takes the data you need from each grant in your grants system, and use that data instead.

//declare useful variables
XNamespace ns = "http://www.symplectic.co.uk/publications/api";




//declare two hypothetical grants to be fed to the Elements System
var grants = new[]
{
new
{
ProprietaryID = "0001234",
Title = "The first grant",
SterlingValue = "400000"
},
new
{
ProprietaryID = "0001235",
Title = "The second grant",
SterlingValue = "5500"
}
};




//for each grant, import the grant to the Elements System
foreach (var grant in grants)
{
string requestUrl = string.Format("https://localhost:8091/elements-api/v4.9/grant /records/source-3/{0}", Uri.EscapeDataString(grant.ProprietaryID));
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(requestUrl);
request.Headers.Add(string.Format("Authorization: Basic {0}",
Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}:{1}",  "username", "password")))));
request.Method = "PUT";
request.ContentType = "text/xml";
Stream body = request.GetRequestStream();
using (XmlWriter xmlWriter = XmlWriter.Create(body))
{
XElement requestXml = new XElement(ns + "import-record",
new XAttribute("type-id", 1),
new XElement(ns + "native",
new XElement(ns + "field", new XAttribute("name", "title"),
new XElement(ns + "text", grant.Title)),
new XElement(ns + "field", new XAttribute("name", "amount"),
new XElement(ns + "money", new XAttribute("iso-currency", "GBP"),  grant.SterlingValue))));
requestXml.WriteTo(xmlWriter);
}
body.Close();




//get the response
try
{
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseContent = reader.ReadToEnd();
}
catch (WebException doh)
{
string response = new StreamReader(doh.Response.GetResponseStream()).ReadToEnd();
}
}

The code above calls the PUT /grant/records/source-3/{proprietary-id} operation repeatedly, once for each grant to be imported, constructing the operation URL using the name of the source we chose earlier, the source from which we are importing ("source-3") and the proprietary ID of the grant record we are importing, as per the PUT /grant/records/source-3/{proprietary-id} operation documentation.

The XML document forming the content of each HTTP request contains the rest of the data for the grant being imported by the request, and correct supply of this XML will constitute most of the effort of implementing the data feed.

The PUT /grant/records/source-3/{proprietary-id} operation requires you to HTTP PUT an "api:import-record" element, as defined in the API schema. You can see this being done in the code example above. The schema definition for the element is:

<xs:element name="import-record">
<xs:complexType>
<xs:sequence>
<xs:element name="citation-count" type="xs:int" minOccurs="0"/>
<xs:element name="verification-status" type="api:verification-status"  minOccurs="0"/>
<xs:element name="verification-comment" type="xs:string" minOccurs="0"/>
<xs:element name="native" type="api:import-native-record"/>
</xs:sequence>
<xs:attribute name="type-id" type="xs:int" use="required"/>
</xs:complexType>
</xs:element>

You must specify the type of record you are mapping to from amongst the types available in the relevant category (e.g. from /grant/types), as decided early in the mapping process for your import. In our hypothetical case, we decided earlier to map all grants to the only grant type currently configured in the system, with type-id 1. When your institution makes more types are available, you can choose amongst them.

If you supply a citation-count, and the concept applies to the type of record you are importing (only publication records have citation counts), this value will be set against the record. If you do not supply a value, the citation count will not be updated.

You are also in control of your institution's verification status for this record. These concepts only apply to publication records, and values supplied when importing other categories of record will be ignored. If you do not supply a verification status value, the system may set the verification status to "unverified" in any case, since you have modified the record's data. The verification comment is updated only when you supply a verification status. If in this case you do not supply a verification comment, then any existing comment is deleted.

The next element you must supply is the "native" element, containing all the basic field values for the record you are importing. The format of most of this element will be familiar to you from when you take data from the API, as it is just a simplified version of the "api:native" element you see when exporting objects from the API. See the API schema for more detailed help.

<xs:complexType name="import-native-record">
<xs:sequence>
<xs:element name="field" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:choice>
<xs:element name="addresses">
<xs:complexType>
<xs:sequence>
<xs:element name="address" type="api:address" minOccurs="0"  maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="boolean" type="xs:boolean">
</xs:element>
<xs:element name="date" type="api:date">
</xs:element>
<xs:element name="decimal" type="xs:decimal">
</xs:element>
<xs:element name="funding-acknowledgements"  type="api:funding-acknowledgements">
</xs:element>
<xs:element name="identifiers">
<xs:complexType>
<xs:sequence>
<xs:element name="identifier" type="api:identifier" minOccurs="0"  maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="integer" type="xs:int">
</xs:element>
<xs:element name="items">
<xs:complexType>
<xs:sequence>
<xs:element name="item" type="xs:string" minOccurs="0"  maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="keywords">
<xs:complexType>
<xs:sequence>
<xs:element name="keyword" type="api:keyword" minOccurs="0"  maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="money" type="api:money">
</xs:element>
<xs:element name="pagination" type="api:pagination">
</xs:element>
<xs:element name="people">
<xs:complexType>
<xs:sequence>
<xs:element name="person" type="api:person" minOccurs="0"  maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="text" type="xs:string">
</xs:element>
</xs:choice>
<xs:attribute name="name" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>

Let's view an example of a full PUT /grant/records/source-3/{proprietary-id} operation for the hypothetical situation we are in with our grants system. We assume that the ID of the grant record in tblGrant in your grants system is, say, "12", and that its StartDate is perhaps "2010-10-04" and its Value "3500", which we will presume is in UK Pounds (GBP).

The entire operation consists of an HTTP PUT to /grant/records/source-3/12 with the following content:

<import-record type-id="1" xmlns="http://www.symplectic.co.uk/publications/api">
<native>
<field name="start-date">
<date>
<day>4</day>
<month>10</month>
<year>2010</year>
</date>
</field>
<field name="amount">
<money iso-currency=”GBP”>3500</money>
</field>
</native>
</import-record>

And that's it for importing records. The code example introduced earlier creates a slightly different import document importing its data to different fields. You would construct your XML document to import to the fields according to the mapping you decided early on in the process.

5: Decide how you want to associate users to your imported grants

In our hypothetical situation, each row in tblGrantStaff represents an association between a member of your staff and one of the grants in your grants system. You will import these as user-relationships to the Elements system.

In this simplified situation, we assume that all of these associations represent the relationship of "staff member is the primary investigator on the grant", corresponding to the relationship type with ID 43 (see the GET /relationship/types operation and the API schema for more information).

At this stage, when implementing your own data feed, you will need to decide on a mapping between the associations in your grants system and the relationship types available in the Elements system.

6: Extend your importer program to import your associations

The operation to call is the POST /relationships operation. This operation will create a new relationship between the record and user you indicate, of the type you indicate, if one doesn't already exist, or update the existing one with the data you supply if it does.

You will call the POST /relationships operation repeatedly, once for each association between a grant and user to be imported, with the operation URL simply being /relationships,

The POST /relationships operation requires you to HTTP POST an "api:import-relationship" element, as defined in the API schema. The schema definition for the element is:

<xs:element name="import-relationship">
<xs:complexType>
<xs:sequence>
<xs:element name="from-object" type="xs:string"/>
<xs:element name="to-object" type="xs:string"/>
<xs:choice>
<xs:element name="type-id" type="xs:int"/>
<xs:element name="type-name" type="xs:string"/>
</xs:choice> <xs:choice minOccurs="0"> <xs:element name="from-object-percentage" type="xs:decimal"/> <xs:element name="to-object-percentage" type="xs:decimal"/> </xs:choice>
<xs:element name="is-visible" minOccurs="0">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:boolean">
<xs:attribute name="overwrite" type="xs:boolean" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>

You must first identify the ‘from-object’ and the ‘to-object’ using api object identifiers (see API Requests and Responses for details).

You must next specify the type of relationship being imported using the "api:type-id" or “api:type-name” element. This forms a part of the identity of the relationship itself, alongside the associated record and user. In our hypothetical situation, we have decided to map all associations in tblGrantStaff to Elements relationships of type ID 43 (user is primary investigator of grant).

The "api:is-visible" element is available for you to control the visibility status of the relationship between the user and the record. This concept does not apply to all relationship types and use of the element for an incompatible relationship type will cause an error. You must contact your system administrator to discuss which, if any, of the relationships you import must be flagged as invisible.

Not supplying the "api:is-visible " element will cause the Elements system to either set a standard default value for it (if creating the relationship for the first time), or leave the current value alone (if updating an existing relationship).

If you supply the "api:is-visible" element, you may also supply an "overwrite" attribute set to either "true" or "false". Not supplying the attribute will cause it to behave as if you had specified "false". When "overwrite" is set to "false", the value for "api:is-visible" that you supply will only be committed if the relationship is being created, but no changes will be made to it if the relationship already exists. Setting "overwrite" to "true" will cause the visibility status of the relationship to be overwritten with the value you supply in all circumstances.

Let's view an example of a full POST /relationships operation for the hypothetical situation we are in with our grants system. We assume that the Grant_ID of the association row in tblGrantStaff in your grants system is, say, "12", and that the Staff_ID value is perhaps "000123", representing the proprietary ID of one of your members of staff.

The entire operation consists of an HTTP POST to /relationships with the following example content:

<import-relationship xmlns="http://www.symplectic.co.uk/publications/api">
<from-object>user(pid-000123)</from-object>
<to-object>grant(source-source-3,pid-001234)</to-object>
<type-id>43</type-id>
<is-visible>true</is-visible>
</import-relationship>

Please note that you are free to identify the user in the relationship using one of the three common identification schemes: by authority/username combination, by the Elements ID of the user, or by your institution's proprietary ID for the user.

7: Liaise with the system administrator to decide when to run your importer

Finally, you need to discuss and confirm with your Elements system administrator the times at which you run your importer.

Your system administrator should suggest a time that does not conflict with other heavy-duty operations being performed on the system. Data importers should typically be run at night time or weekends.