Assessment Module Administration - Creating an Exercise v6.11

The Assessment Module supports the creation of bespoke workflows and the ability to collect information from users for the purpose of review, evaluation and reporting. The Assessment Module can be used to support a wide range of activities including institutional assessments - such as the PBRF (NZ), REF (UK) and other internal review processes such as annual faculty activity reviews and promotion and tenure processes. 

This guide will assist you in creating an Assessment Exercise. An Assessment Exercise can be thought of as an online form that a user completes - along with all of the associated configuration options related to this process. The Exercise form can move through a number of stages where it can become available to reviewers to add reviews and scores. 

Exercise definition  

The complete set of configuration options that describe everything about a given exercise process. This includes the configuration of an Exercise form to be completed by users, as well as any stages associated with the data collection and review process. The Exercise definition also includes configuring which users will be asked to complete an Exercise and which reviewers will be asked to review their responses. Assessment Administrators can create exercise definitions. Exercise definitions can be cloned, so they don’t have to be created from scratch every time. 

Stages 

All exercises must consist of at least one stage. A user’s Exercise form can move through different stages. You can configure which users and reviewers can view and edit the Exercise form at different stages. In addition you can configure ‘Stage transitions’ which allow users to move the Exercise form between stages. In combination, this provides the ability to create bespoke workflows that meet your specific organisational requirements. 

Sections 

The Exercise form is built of sections. Sections support the logical grouping of information and can contain ‘lists’, ‘attachments’, ‘supporting fields’. Sections can also include ‘reviews’. 

Note: On upgrade from pre-v6.9 to v6.9+ all existing lists will be placed within sections. The sections will inherit the name of the list. You can easily modify the name of the section, add specific overview and guidance text for the section as a whole and move lists between sections. 

Exercise information  

It is possible to configure ‘attachments’, ‘supporting fields’ and ‘reviews’ at the exercise level - to capture information that relates to the user’s Exercise form as a whole. This is displayed in a special section called ‘Exercise Information’.

Lists 

Standard lists are collections of items that users can select from their existing Elements objects. Items can be edited (through the addition of further information in the form of supporting fields or attachments), ordered and managed. Standard lists contain items belonging to a single Elements data category (e.g. publications). 

It is also possible to create ‘custom field lists’. The items in these lists consist of supporting fields, allowing you to customise the data that is captured. Lists can also be configured to include ‘attachments’, ‘supporting fields’ and ‘reviews’. 

Note: ‘No-item lists’ were removed in the v6.9 release. Any existing no-item lists will be migrated to become sections - with no loss of data. 

Items 

Depending on the type of list configured, items can be selected from Elements objects - Publications, Grants, Professional Activities, Teaching Activities and Impact. Items in custom field lists consist of a set of fields configured by the administrator - giving administrators complete control over the information returned by the user. Items can also be configured to include ‘attachments’, supporting fields’ and ‘reviews’. 

Attachments  

Each part of the Exercise form can be configured to support the upload of attachments. 

Supporting Fields 

Each part of the Exercise form can be configured to support the addition of supporting fields. Supporting fields are configurable field sets that can be used for the collection of specific information in the Exercise form. 

Reviews 

Each part of the Exercise form can be configured to support the addition of reviews. These are text comments that can only be added by Reviewers (see information on roles below). The display of review to users and other reviewers can be controlled per stage as part of the Exercise Definitions. 

Scoresets

Reviews can optionally include a scoreset. These can be used to capture scores or statuses for each review.

An Exercise form completed by a user can contain confidential and sensitive information. The Assessment Module leverages Elements role based permissions model to ensure that administrators have complete control over who can see what data, and when. It is important to understand the different Assessment Module related roles and the behaviour associated with them when configuring an Exercise Definition. If you are unsure, please don’t hesitate to contact the support team. 

Users/Researchers

Elements users can be invited to take part in an Exercise by being added as users. Users are presented with the Exercise form to complete once the exercise is set to ‘Active’. As part of the exercise definition administrators can control what users can do at each stage of the exercise. Users are added on the ‘Users’ tab of the Exercise definition page. 

Reviewers

Reviewers are Elements users who are being asked to review an Assessment Exercise at one or more stages as part of the Assessment process. Reviewers are assigned per stage, per user. Reviewers can only view an exercise when it is in the stage for which they are a reviewer. It is possible to assign multiple reviewers per user. Additionally, different sets of reviewers can be assigned against different sets of users at each stage. Reviewers are added on the ‘Stages’ tab of the Exercise definition page.

Managers

Managers have access to the ‘Manage Assessments’ page for any Exercises to which they are assigned. This gives Managers a view of the progress of all users completing an exercise. Managers are able to carry out a number of different activities to help support the completion of an exercise, including having permission to run reports against the exercise as a whole, move specific Exercise forms between stages etc. You may want to consider giving Managers the ‘Research manager’ group role, so that they have permission to impersonate users. Managers are added under the ‘Managers’ tab of the Exercise definition page. 

Assessment Administrators 

The Assessment Administrator role has the same permissions as the Manager role. Only Assessment Administrators are able to create new Exercise definitions and edit existing definitions. Assessment Administrators can be added under System Admin>User Management>Manage System Roles.

Changes in Assessment related permissions introduced in Elements v6.9

Reviewers gain the permission to view private objects that have been added to an Exercise form that they can view. Private objects can only be accessed with this permission from the Assessment Exercise item page. This means that when the reviewer can no longer access the item page (for example because the Exercise form has moved to a new stage), they will no longer be able to view the private object. This ensures that reviewers have access to the same information as the user completing the Exercise form. 

Note: This modification in permissions to view private objects extends to users completing an Exercise form. The result of this is that if a user adds a private object to a list, and then the relationship between that user and the private object is removed, the user can still access the private object via the Assessment Exercise. This access is only via the Exercise form and therefore requires access to the Exercise form. 

If a stage of the transition is configured, Assessment users can now move the stage of an exercise even if they don’t have permission to edit the Exercise form in that stage. 

To see if you are licenced for the module go to  System Admin > General Settings >Licence. Assessment is listed in the 'Other' section. If it is greyed out, then you are not licenced. Depending on your subscription, the Assessment module may be subject to a separate licence key. Please raise a support request if you have questions.

System Administrators and Assessment Administrators can create and modify Assessment Exercises. Select - Assessment > Configure Assessments > Exercise Definitions option in the menu. To add a new Exercise, enter the name of the new Exercise in the text box and then click the "Create exercise definition" button. 

Important: The Module can be used to collect personal information about users. Misconfiguration can therefore cause inappropriate sharing of user data and/or review data. It is important the appropriate care is taken when configuring an Exercise. Testing of the Exercise process should be carried out before making the Exercise 'live' for users. Relevant individuals (e.g. Managers, Administrators), should receive appropriate training to ensure they have the necessary understanding to manage data privacy.

You can also create an Assessment Exercise from a template. The template pre-populates the eligibility, lists and supporting information of the Assessment.

Exercise - The exercise tab allows you to configure settings and information related to the exercise as a whole 

Sections - The section tab allows you to configure the sections that will make up the Exercise form to be completed by a user. Users can add information by selecting objects (e.g. publications) that already exist in Elements, uploading attachments, or entering information in fields. 

Scoresets - Custom scoresets can be added here. There are also templates for the REF 5-point scale, the REF 13-point scale, and an Annual Review 5-point scale.

Stages - The ability to define how many stages an exercise includes, as well as whether users can view, edit, or mark an exercise as 'done’, at a particular stage. Additionally, Reviewers are assigned here, as well as what stages reviewers can view and what part of the Assessment Exercise Reviews can comment on and add scores to. 

Stage transitions - The ability to move the Exercise form between different stages can be configured for either users or reviewers.

Users - The individuals who will be invited to complete the Exercise form are set here.

Managers - The individuals who can manage an assessment exercise are defined here.

Attachment types - This allows you to set a name for the attachment to be uploaded, so that users can choose the type when they upload a file (e.g. ‘CV’, ‘Supporting documentation’ etc.) 

Units - Users in an Assessment Exercise can be subdivided into units so that Manager assignment and some custom configurations at the unit-level can be applied. Units are primarily used to support UK REF processes. 

Warnings - Under the Warnings tab, specific warning types can be disabled in order to control which prompts will be shown to researchers, reviewers and managers.

On the ‘Exercise’ tab of the Assessment exercise definition screen, you have the following configuration options:

• Is active - This determines whether or not the Exercise is available for researchers and reviewers. It is recommended that you complete the configuration of the Exercise Definition before setting the Exercise to active. 

• Is REF submission - This determines whether or not this Exercise is being used to generate the REF submission. This is only relevant to UK Clients using Elements REF functionality. 

• Name - The name of the Exercise that will be displayed to all Elements users who have permission to view the Exercise.

• Type - Options are: Assessment, Review Process, Document, Survey, and Acknowledgement Request. This controls the display name of the Exercise and does not have any effect on the layout, or options available in the Exercise. 

• Allow researcher impersonation - If this box is checked, anyone who can impersonate a user who is participating in an Assessment Exercise will be able to carry out the exercise on their behalf, and view all information added to the Exercise form. If this box is unchecked, then it will not be possible to view the exercise by virtue of impersonating its owner. 

• Overview page shows researcher information - If checked the Overview page will show a summary of the researcher, including their group, position and academic appointments. This is displayed to users and reviewers.

Overview and guidance text configured on the ‘Exercise’ tab is displayed on a new ‘Introduction’ page that is shown to users and reviewers when they first start an exercise. This page is always displayed so it is recommended that you configure text to be displayed here.

• Overview text - The ‘Overview’ text is displayed on the first page that both users or reviewers see when they start an exercise. The information presented should be used to give an overview of the exercise and describe the purpose of the data collection process. This text will be displayed to both users and reviewers. The Overview text is also shown on the Assessment list page. 

• Guidance text for researcher - The purpose of the guidance is to describe how users are expected to complete the Exercise form. This text is only shown to users and not to reviewers. It could include details of criteria to be used for selection, guidance for entering information etc.

• Guidance text for reviewer - The purpose of the guidance is to describe how reviewers are expected to review the exercise. This text will appear to reviewers, but not to the users responding to the exercise. It may include details of criteria to be used for review, guidance from institutional sources etc.

Information about the whole exercise (either supporting fields, attachment, or reviews) is collected in a section called ‘Exercise Information’. This will appear on the Exercise form and will function like any other section but it cannot be renamed.

• Attachments enabled - If checked, attachments can be added at the Exercise level and will be displayed in the ‘Exercise information’ section. See the ‘Attachment type’ section below for more detail. 

• Supporting fields - If enabled, users will have the option of inputting information in the ‘Exercise information’ section. See section on ‘Supporting fields’ section below for more detail.

Make sure to save any changes by clicking ‘Update exercise’. You can always return to make adjustments to the configuration options. 

This tab allows you to configure the different sections that will be displayed to users on the Exercise form. Sections can be used to logically group related information. Sections can contain lists (see below), attachments (see the ‘attachment types’ tab section), supporting fields (see the ‘supporting fields’ section) and reviews (see the ‘stages’ tab section). 

Each section appears as a page that the user can navigate to, either in turn using the ‘previous’ and ‘next’ controls, or using the navigator on the left hand-side of the page. Sections can be added using the ‘Add section’ button. Once added, the position of sections on the Exercise form can be adjusted using the arrow buttons.

Sections can contain lists, supporting fields and attachments. It is also possible to configure section level overview and guidance text (see the description in ‘Exercise’ tab for details). Please see the support article ‘Tips for building an Exercise form’ for further advice on using sections 

Lists are a key part of the Exercise form, allowing users to capture information on outputs and activities that already exist in Elements. Users can add an item to a list at the click of a button, making it available for review and reporting. Lists can also have specific eligibility criteria applied. For example, you may wish to limit the information returned to only journal articles from the last academic year - depending on your specific use case. 

Lists can be added within sections. A section can contain multiple lists, or no lists at all. There are two different classes of list available; 

• Standards lists - Standard lists are used to leverage the pre-existing information in Elements by allowing users to select from their outputs and activities (e.g. Publications) and add them as items to a list. Each list can only include items from one Elements category - either Publications, Grants, Teaching Activities, Professional Activities or Impact. 

• Custom field lists - Custom field lists allow you to define the fields that make up an item and control what data is collected from users. The data collected is only available within the Assessment Module.

When configuring a standard list you have a number of configuration options;

List settings

• Name - The name of the list, as displayed on the Exercise form. 

• Abbreviated name - The abbreviated name is only shown on administrative pages. 

• Minimum number of items - The minimum number of items that should be added to the list. If this minimum number is not met, a message is displayed to users on the list component of the Exercise form. This information is also displayed on the selection page. 

• Maximum number of items - The maximum number of items that should be added to the list. If the maximum number of items is exceeded, a message is displayed to the users on the list component of the Exercise form. This information is also displayed on the selection page. 

• Note: It is possible for users to submit an Exercise form when the minimum/maximum number of items requirement is not met. A warning is displayed to users in the summary modal. 

• Show annotations on item page page - If object annotations are configured, you can control whether or not these should be shown on the item page. The use of Object Annotations allows users to add information once and reuse it across multiple Exercises. This is in contrast to ‘supporting fields’ which are only captured within a given Exercise.

Overview and guidance 

• It is possible to configure list and item level overview and guidance text (see the description in ‘Exercise’ tab for details).

Eligibility criteria 

It is possible to configure lists to only accept items that meet certain eligibility criteria. The system will default to only display eligible items when users select items to add to the list. 

• Earliest date - The earliest reporting date than an item can have to be eligible for the list

• Latest date - The latest reporting data than an item can have to be eligible for the list

• Note: Items with no reporting dates are always considered eligible for the list. 

• [Object types] - The object types that are considered eligible for the list. 

• Enable list pre-population - Enabling list pre-population will result in all eligible items being added to the list when the user starts the Exercise. Users are still able to make changes to the list following pre-population. 

• Note: Pre-population happens in the order that lists appear on the form. This means that if two non-overlapping lists are configured, the first list would contain eligible items over the second list. 

Display Open Access information (Publication lists only) 

• You can control the display of open access information to researchers and reviewers. This information is displayed on the selection page, list view and item page.

Attachment settings

• You can enable attachments to be added to either the list, or the items themselves. (For more details see the ‘Attachment types’ tab section)  

Supporting fields 

• You can configure supporting fields to be captured against the list, or against the items themselves. (For more details see the ‘Supporting fields’ section) 

When configuring a custom field list, it is important to configure item supporting fields as these will constitute the item that is added to the list by users. 

Note: On upgrade to v6.9+, there will be some changes applied automatically to existing Exercise definitions in order for the new section component to be introduced:

• All existing lists will be placed within a section. The title of the section will be automatically set upon upgrade using the name of the list. You can then easily move lists between sections, edit the title of the section and add the ability to capture supporting fields and attachments on the section as desired 

• ‘No item lists’ will be migrated to become sections. The name of the new section will be set using the previous name of the list, and all attachment and supporting field configurations will be automatically migrated. No pre-existing data will be lost during the upgrade.

'Scoresets' tab

Many internal and external assessments include a review and scoring process. Community consultation demonstrated that there can be significant differences in approaches to scoring for internal processes such as Activity Reviews in comparison to external Governmental or Accreditation processes, or even inter-departmental reviews in some cases.  

Assessment exercise administrators can configure a number of optional scoresets to complement the existing reviewer comment functionality. A scoreset is a collection of ordered choices for reviewers to choose from. 

Each option includes a:

• Short name - shown to reviewers in a dropdown (max 20 characters)

• Description - intended to convey scoring instructions and displayed to reviewers via a ‘?’ prompt

• Numerical value - to allow for calculations to take place for reporting at individual, group or institution levels.

Scoresets can be used at exercise, list and/or item levels in any of the configured review stages. Reviewers can add a scoreset as part of their review.

All score data is available in the Reporting Database for use in SSRS reports. The reports can be exported from within the Assessment module, with the numerical values providing the capability to compare progress and scores across departments.

'Stages' tab

Under the Stages tab, the Assessment Administrator can define how many stages each exercise will have, as well as how Users can interact with each stage. 

User controls 

• Can edit their exercise - Determines whether users are able to edit the Exercise form when the form is in this stage. 

• Can view their exercise - Allows users to view but not edit the exercise.

• Cannot view their exercise - Users cannot view the Exercise form. The exercise will still be displayed on the users list of Exercises while the Exercise is ‘Active’. 

• Can mark exercises in this stage as 'done' - Allows the user to flag the Exercise as ‘Done’. This status is shown on the Exercise management pages. Please see the support article ‘Tips for building an Exercise form’ for further advice on configuring this setting. 

• Can view reviews created in this stage - Allows the user to view reviews created in this stage, irrespective of the stage the Exercise form is in.

Add/Manager reviewers 

• Reviewers can be assigned against users at each stage. Reviewers will only be able to access the Exercise form when it is in a stage that they are a reviewer on. Reviewers can be assigned against individuals or groups. 

Reviewer controls 

• Are anonymous to users - If reviews are visible to users, they will be anonymous. They will not be anonymous to other reviewers. 

• Reviews visible to other reviewers of - Determines if reviewers of other stages can see the reviews created in this stage. 

• Reviewers can comment on - Determines which components of the Exercise form reviews can add comments on. Scoresets can also be set here if they are configured. 

'Stage Transitions' tab

Working in conjunction with stages, ‘Stage transitions’ allow the configuration of workflows that can be used to control the movement of Exercise forms through different stages. Exercise stages can be used to manage user access to the Exercise form and control the configuration for review and sign-off phases. 

Stage transitions can be carried out by the researchers completing the Exercise form, reviewers and Managers. Stage transitions provide a powerful way of managing the movement of the Exercise form through different stages. Using the Stage Transitions tab, the Assessment Manager can define which Roles (User, Reviewer) can move an Exercise form from stage to stage and in which direction (next stage/previous stage). 

Please see the support article ‘Tips for building an Exercise form’ for further advice on configuring stage transitions. 

The ‘Stage transitions’ tab of the Exercise Definition can be used to configure the ability for a researcher (the user completing the Exercise), or a reviewer to move the Exercise form from one stage to another. The page contains one row for each stage created on the ‘Stages’ tab and two columns that can be used to control the stage transitions for researchers and reviewers. When adding a transition you can allow users to move the Exercise form from any given stage to any other stage. The decisions made will depend on your organisation's specific workflow.

For more information on configuring ‘Stage transitions’ please see - Configuring stage transitions v6.10+ 

'Users' and 'Managers' tabs

These two tabs are used to assign Users and Assessment Managers, either for individuals or in Groups. There is also a tool that will allow you to manage assignments in bulk by uploading a CSV file identifying people by user ID, username and/or proprietary ID.

'Attachment Types' tab

Use this tab to set a name for the attachment to be uploaded, so that users can choose the type when they upload a file (e.g. ‘CV’, ‘Supporting documentation’ etc.) Once the types have been configured here, they will become available throughout the Exercise and section configuration pages. Multiple types can be configured per attachment/component pair. 

Attachments are configured on the Exercise and section tabs. Attachments can optionally support a description provided by researchers. Each configured attachment/component pair can also have a specific description configured which will be displayed to users. 

The default size limit for attachments is 5MB, but this can be increased or decreased. In the web.config file. There are two entries as follows:

 <add key="max-streamed-file-upload-size" value="5242880" />
 <add key="max-unstreamed-file-upload-size" value="5242880" />

If you change the 524880 bytes to a different number, then that that will change the size limit (including changing the error message users see if they exceed it. Warning: changing any value in the web.config will log out all users. 

'Units' tab

Units make it possible for institutions to consolidate cross-institutional assessment activities into a single exercise definition which can then be subdivided into units so that administrators can allocate users, assign managers and apply some custom configurations at the unit-level. In addition to supporting institutions with preparing for REF2021, this functionality assists institutions with multi-department faculty activity reporting to manage these assessments in more synthesised ways.

In effect, breaking an exercise definition into units allows institutions to group together collections of users in order to allow management and reporting of their responses. Each user can be allocated to only one unit, and can be moved between units without loss of selections or supporting information. Administrators can allocate users to their unit manually via the UI or can bulk-allocate the users in an exercise definition to the appropriate unit, using an Excel/CSV upload.

Managers can also be assigned at the unit level, granting those managers permissions only over the users allocated to their unit(s) rather than over the exercise definition as a whole. When setting up an exercise definition which will be subdivided into units it is recommended that the standard exercise definition includes each supporting field that anyone in any of the units will be asked to complete. This forms the default configuration which can then be overridden for each unit. For each of the different units in an exercise definition administrators can selectively hide supporting fields on each unit, modify the in-place guidance about each supporting field, and change the settings for attachments. Administrators can also override the guidance for researchers and reviewers to allow them to ensure each unit has clear instructions about the exercise they are being asked to complete.

Image: Overriding the guidance text for the ‘Contribution’ supporting information field for a unit.

Once units are configured, administrators can report on the exercise as a whole to produce reports which span all of the units within an exercise with just a few clicks. Each user’s allocated unit is included in the stock exercise-definition level exports for both the Exercise Definition Report and the REF Manager Publications Export.

We have introduced a units template to match the REF2021 Units of Assessment. This should make it easier for institutions who are already using a single exercise definition for current REF preparations to begin configurations at a unit level. This functionality is the first of several iterative releases in the run up to full support for the Outputs component of REF2021. Future releases will introduce an official template exercise definition, aligned to the REF2021 data submission requirements as they become known.

Image: Units created using the template for REF2021 preparations

'Warnings' tab

Under the ‘Warnings’ tab, administrators can disable specific warning types per section, list and item. This provides the ability to control what information will be shown to researchers and reviewers on the Exercise form. 

Supporting fields can be thought of as field sets - equivalent to Elements records - that allow you to define what information should be input by users on the Exercise form. By default, supporting fields have the display name ‘Supporting information’ (or ‘Information’ for item supporting fields). We recommend that you modify the display name to describe the information captured.

Supporting fields can be configured throughout the exercise form; 

• Exercise - Supporting fields at the exercise level will be displayed in the ‘Exercise information’ section. 

• Section - Supporting fields at the section level will be displayed on the section page in their own component. 

• List - Supporting fields at the list level will be displayed on the section page, grouped with the corresponding list. 

• Item - Supporting fields at the item level will be displayed on the item page 

Examples; 

• Exercise Supporting Fields: Options include the overall summary that a researcher might complete for their exercise, an overview about recent research interests, Annual Planning goals, a Research code/category for the researcher, or a government prescribe exception category or circumstance.

• List Supporting Fields: Options include an overall summary of a collection of items in a list, a description of the order of items in a list, a commentary about why the list items were selected or some general comments for a reviewer about how items relate to each other.

• Item Supporting Fields: Options include a statement justification, a co-authoring contribution description,  a description about the significance of the work, a code to describe the item, or a subject category.

To create a Supporting field, click the "Create Supporting Fields" button at the bottom of the Exercise, section, or list pages. Deleting the supporting fields removes every field, so use with caution.

At the "Field properties" page, fill in the details for the new Supporting Field. While you cannot use an underlying field more than once in a given location, an underlying field with a generic name can be reused for other supporting fields at different locations within an exercise.

• Display Name: Create a unique descriptive name, you can alter the name of the supporting fields populating from the template to suit your institution.

• Instructions: this guides your Researchers on how to use this field

• Underlying field and data format: pick from the existing Underlying Fields - if you need to create a new field seen the section on Creating New Underlying Fields.

• Input Type: indicates if there are choices for the field format 

• Symbol inserter: for text formats you can use the Symbol Inserter to create encoded text

• Key Field: you can define a key field, that will be used to flag that information has not been entered

• Summary (For ‘custom field list’ items only) - You can choose if the information in the supporting fields should be used on the item summary.

Underlying fields

Underlying fields are used throughout the Assessment Module, our advice is to set them up for all the Supporting Fields prior to configuring your exercise.  Select "Update" to save changes. Note the caution about making changes to field properties. 

The following options are available when creating underlying fields:

• Name: can only consist of lowercase letters, numbers and hyphens(-); must begin with a letter; and must end with a letter or number. All your created underlying field names will be automatically prefixed with c- to distinguish them from system underlying field names

Data formats:

• Text (allows you to specify the Length of the text field)

• Date

• Choice (a set number of choices to be filled with inputs such as radio button lists or drop-down menus)

• Integer (a number, restricted to whole numbers)

• Boolean (true/false)

• Number (a number, without the integer restriction)

• List (a set of separate pieces of data – such as keywords, which users will add one by one

• URL

• DOI

• Money

• Fields: select a new field based on an unassigned underlying field type. Each underlying field can be used exactly once on a type, so if you've used all the underlying fields then you'll have to define another one or delete an existing type field. reorder with the blue cross or delete fields using the checkbox

• New field: pick from the available underlying fields

• New underlying fields: define fields for use in the exercise, list or item

• Manage underlying fields: displays all the fields in the exercise

Don't forget to use the Update button to save any changes before navigating away from this page.