Assessment Module - Introduction

What this article covers:

• What is the Assessment Module?

• What are some terms I should know?

• What are the roles associated with Assessment?

• What steps are needed to create an Assessment Exercise (exercise)?

• Use Case #1: Annual performance to assess faculty members at Lilliput University

• Use Case #2: Creating a Survey
  

• Note: A PDF printout of this article is available at the end.

What is the Assessment Module for?

Its primary purpose is to provide a space where information can be gathered -- from both inside and outside Elements -- for review and evaluation. The term “Assessment” is frequently used to mean “judgement,” so perhaps an alternative name for this tool might have been “Appraisal Module.” 

What kinds are possible?

Similar to the way you can have different kinds of publication ‘types’ in Elements, you can have different types of assessments. The module can be used to create and issue surveys, manage acknowledgement requests, document review processes, and, yes, perform assessments.

Practically speaking, the difference is mostly just about terminology. Choosing the ‘type’ determines how the exercise will be described on the screen, thereby setting a user’s expectations about what the process will be.

Assessment Exercise

The activity the user will be asked to complete. They are typically made up of a range of fields including lists for making selections and supplementary information. Systematically, an Assessment Exercise might look like this:

Lists

Collections of items that you select, create, edit, order and manage. Example names of lists includes Research Outputs, Creative Outputs, NROs, and/or Narratives.

Items

Pieces of evidence that can be included in a list. These can be selected from Elements objects - i.e. individual Publications, Professional Activities, Teaching Activities, Impact and Grants - or created in the Exercise as ‘fields only’ - i.e. content that is private and confidential and should only exist in an Assessment Exercise. Items can be ordered and supplemented with additional information.

Supporting fields

Fields which can be manually entered by the researcher (rather than selecting objects already stored in Elements). As this information only exists within an Assessment, it sits within a protected part of Elements only accessible by specific users.

Stages

Different steps in the exercise process, assigned to different users or roles. Generally speaking, the first stage will be where the academic will select the items they consider valuable in the assessment exercise. The second stage will usually be the first review stage (the academic's supervisor or department head); the third stage the Dean, and so on up through the hierarchy.

Units

A way of subdividing an assessment into predefined groups & varying some configurations per unit.

Users

The people being asked to complete the exercise. They are added on the Users tab of the Exercise definition.

Reviewers

Elements Users who are being asked to review the submissions during one or more stages of the assessment. 
Assigned: via the Stages tab of the Exercise definition.

Assessment Managers

The people who manage the submission and review processes for each exercise. 
Assigned: via the Managers tab of the Exercise definition. Note: You may wish to also give them the ‘Research Manager’ role, so they can also impersonate users within the relevant group.

Assessment Administrators

The people who can configure exercises and manage roles.
Assigned: via Manage System Roles (System Admin menu).

Step 1: 
Decide what data needs to be collected

Before you begin...consider what data you actually need to collect:

• Faculty are less likely to engage and are more likely to feel frustrated if the forms are long and complex. Only collect data you know you’re going to use. 

• Consider whether any of the data you want to collect can be fed in from other systems.

Step 2:
Create the Exercise

1. Start with a template and edit from there, or create a new one from scratch.

2. Add a name, choose an exercise type and add some guidance text. 

3. Decide whether you want to create ‘units’

Step 3:
Add one or more ‘lists’

1. Choose whether to add on ‘lists’ for each type of data you would like the user to select

2. Choose whether to add a ‘no-items list’ - made up of Supporting Fields and used to store data that is only entered in an Assessment Exercise.

3. Choose whether to add a 'fields-only' list, which is made up of Supporting Fields and is used to store data that is only entered in an Assessment Exercise.

Step 4:

Add one or more ‘supporting fields’

These can be created for an exercise, a list or an item.

1. Exercise Supporting Fields: Overall summary fields for each exercise submission.

2. List Supporting Fields: Summary fields for a collection of items in a list

3. Item Supporting Fields: Fields relating to each selected item.

Step 5:

Create Stages to structure your workflow

1. Can be simple one step process or a multi-stage process with several rounds of review

2. Configure reviewers and user permissions for each stage.

Step 6:

Assign Users and Managers

1. Assign users and managers by group or individually.

Step 7: Configure the optional extras

1. Add a scoring schema 

2. Configure permitted attachment types

3. Configure Units including unit memberships and overrides

4. Configure warnings and prompts

5. Set up Assessment Reports

Step 1:
Decide what data needs to be collected

In this instance, the University’s departments are asked to collect information about:

• Publications

• Works in Progress

• Teaching Activities

• Advising

• Distinctions and awards

• Internal service

• External service

• Continuing grants

• New grants

• New grant proposals

• Leave

Most are items that can be entered into Elements and therefore are easily reportable.

Step 2:
Decide what type of exercise type to use and whether a template can be used or if a custom type must be created

First, go to your instance of Elements and click on Menu > Evaluation & review > {Assessment} > Exercise definitions  

(V6.n Assessment > Configure Assessments > Exercise Definitions.)

Next, create your exercise definition by selecting the Plus 

button (top right) select the exercise starting point definition

Select the "New single response exercise" option.
In the new window, choose the subtype and give it a name.  

 typing the name into the box at the top of the page and select DONE. 

The first thing you will notice about the Assessment exercise definition page is it has a lot of tabs. These will all be explained, as well as the (slightly eccentric) way they should be addressed.

The second thing you will notice is that the exercise definition is not active and will not appear to researchers or reviewers. Likely, this tick-box is the last thing you will do with the exercise, but we thought you would like to know whether it was enabled right up front

Also, note you can change the exercise name here if you would like.

Next, there are several important decisions to make. 

• The Type “Review process” is ideal for this purpose, so select that option.

• Decide if you want to Allow researcher impersonation. This could be important if you plan to have people assist the researcher with completing their review.

• Decide if you want to have Review overview pages show researcher information such as position and academic appointments.

Step 3:
Add some guidance text

• In this situation, Overview text is recommended: “Welcome to the Faculty Annual Report. The Faculty Annual Report is an exercise to capture and review faculty activity information from the past calendar year…”

• As is Guidance text for researchers: This text will appear each time users edit their response to the exercise. It may include details of criteria to be used for selection, guidance for entering information and so on. 

• And, since there will be a review stage, so is Guidance text for reviewers: This text will appear to reviewers, but not to the users responding to the exercise. It may include details of criteria to be used for review, guidance from institutional sources and so on.

• Decide whether you want to enable Attachments. This feature is very useful if you want to collect data that cannot be logged into Elements such as, for example, letters of recommendation or a CV. 

• Decide whether you want to associate another exercise to this one, giving users a quick navigation link on the exercise that opens the associated exercise in another tab.  This enables users to more easily view what was captured in a 'previous' exercise (e.g. last year's Faculty Activity Review).  This function is only available for single-response exercises, and users must be able to view the associated exercise as normal for the link to be shown.  

• Finish by clicking Update exercise.

Step 4:
Decide whether to create Units

Units are used to sub-divide an assessment so as to have slightly different configurations per sub-group. We aren't going to use Units in this exercise, but here's a link to Managing Units of Assessment, which explains how they can be used.

Step 5:
Add one or more lists

• Lists are collections of items that your faculty members can select, create, edit, order and manage. Examples include Research Outputs, Creative Outputs, Grants, Teaching Activities, Distinctions and Awards or Narratives. Lists can be either items that have already been entered into Elements via the various modules (Publications, Grants, Professional Activities & etc.), or they can be ‘no-items lists,’ which are made up of Supporting Fields and used to store data that is only entered in an Assessment Exercise, or they can be 'field-only' lists, a list which is not expected to have any items and therefore useful for surveys and document sections. To add a list, go to the List tab and click Create list. Then, select the type of list you would like to create. For our exercise, we’ll start with a Publications list.

There are several fields that you could fill out for each List, but only a few are required.

• Begin by giving the List a Name. Generally speaking, keep the names of the objects in the Assessment exercise consistent with the names in the User Interface.

• The Abbreviated name is also required. It is displayed on several buttons throughout the exercise.

• Minimum number of items, Maximum number of items, Earliest date, and Latest date are all configurable and should be discussed in the planning stage. 

• The Overview can be as long or as concise as you like, but consider that the next box is Guidance text for researcher, so consider how much detail you want for each box. 

• If you plan to have reviewers, also provide guidance text for them about their responsibilities.

• Next, select which Publication types qualify for this particular exercise. This is another question you should consider during the planning phase.

• Other questions you should consider in the planning phase include:

  • Does Open Access compliance factor into your exercise? If so, then you may want to display “in policy” OA eligibility to users, reviewers and managers. You can also display OA compliance, reasons for non-compliance, exceptions. Last, you can configure the exercise to show OA compliance and eligibility filters on the user’s selection page. 

• Do you want to enable Attachments? These are evidence or supporting information that are not available through the Elements user interface. Examples include recommendations, references, or other information that may be too sensitive to be visible in an academic’s public profile.

• The use of attachments is highly configurable, with a number of settings configured when the exercise is defined. These include:

  • Attachments can be enabled at whole exercise, list or item level, or any combination of those options for each exercise definition. In each case, once it has been configured users can upload a series of files, attached to the appropriate level(s).

  • Guidance text can be configured to explain the background, context, and suitable content for requested files and how to complete the description and categories for the file. 

  • Once uploaded, the attachments can be downloaded by the researcher or by reviewer(s). The existence of attachments is noted in the reporting database and in PDF exports from the exercise, including the  “Full Export PDF” report that is provided as standard with the Assessment module.

Step 6:
Decide if you want to add a “no-items list.”

A "no-items list" is made up of Supporting Fields and used to store data that is only entered in an Assessment Exercise. Examples of “no-items lists” are a cover letter, a CV, a Narrative Statement, or an Annual Evaluations and Development Plan.

Taking a closer look at Cover Letter shows that it has the same structure as other “List” items -- Name, Abbreviated name, Overview, Guidance text, and Attachment settings…

… with the significant difference that the List does not pull data from Elements. Rather, it is configured with one or more Supporting fields, which house the required data.
Begin by adding a new list and selecting the “Create no-item list” option.

Step 7:
(If needed) Add Supporting Fields

Begin by clicking the Create Supporting Fields button at the bottom of the page.

On the Edit support fields page, click the link to add a new field.

Give the new field a Display name and select an existing Underlying field and data format as for any field in Elements. Also, configure Input type and other attributes.

If the Underlying field does not exist, scroll to the bottom of the Edit supporting fields page and add it by clicking the Create a new underlying field link.

Step 8:
Create Stages to structure your workflow

Begin by clicking on the Stages tab. 

Here are the decisions that will need to be made:

• How to describe the stage?

• What can users do with regard to the stage?

  • Can they edit their exercise?

  • Can they view their exercise?

  • Can they mark the exercise as done?

  • Can they view reviews in this stage?

• Will there be reviewers at this stage?

In this instance, we’ll use these options:

• Describe the stage as 2020 FAR Preparation & Submission.

• Users can edit their exercise.

• Users can mark the exercise as done.

• No reviewers at this stage.

This means Stage 1 would look like this:

Continue onto Stage 2 by clicking Create stage. 

Frequently, the second stage is for review of the submissions in Stage 1, so Reviewers need to be added. This can be accomplished by clicking the Manage reviewers button.

Two questions need to be answered:

1. Who is being reviewed?

2. Who is the reviewer?

• Reviewees can be assigned on a group or individual basis, but Reviewers can only be assigned on an individual basis.

After Reviewers have been assigned, decide what rights they should have: 

• Should reviewers be anonymous? The default is that they should not.

• Should reviews be visible to other reviewers? This can be configured on a “per stage” basis; in other words, a reviewer might be able to see a review on one stage, but not on a stage before or after the one they’re working in.

• Which areas should reviewers be able to comment on? For example, reviewers could be configured to comment on an overall exercise level, but not specifically on Publications or Teaching Activities (or vice versa). 

• Should reviewers be able to score an area of the exercise? If Scoresets have been created, a Reviewer could be given access and given permission to assign a score.

Step 9:
Add Users and Managers

User

In this context, “Users” are the people who are being evaluated in the exercise. In Elements, the User must be considered “Current,” which means the account attribute “Is Current” must be True. An Assessment User does not have to have the attribute “Is Academic” equal True. 

Users are assigned via the User tab and can be added either using the Group structure or individually.

After clicking the Add assignments button, be sure to click the Save button in the section just above.

Users can also be added via a CSV file upload.

Assessment Manager 

This role is distinct from the Assessment Administrator. Managers can be added via the Manager tab. Like Users, Managers can be added individually, in groups, or uploaded via CSV file.

The Assessment Manager can monitor assigned exercises, having the ability to move items between stages, lock and unlock individual responses, and download Excel/CSV/PDF/Word reports. To access the Manager dashboard, go to Menu > Evaluation & review [ASSESSMENT] > Manage assessment > Assessments    (V6.n Menu> Assessment > Manage Assessment > Manage Assessment).  

For more information about what the Assessment Manager can do with the Assessment Dashboard, see Managing an Assessment Exercise. 

Everything from this point on is optional, but may prove useful.

Step 10: 

Add a Scoring schema

Scoresets can be added using the Scoresets tab. 

The Scoreset functionality is very flexible and can accommodate different schema, including numeric scores, text descriptions and Yes/No options. 

After a Scoreset has been created, it can be added to individual facets by going to the Stages tab, clicking on a List, and selecting an option from the dropdown menu.

Step 11: Configure permitted attachment types

Start by clicking the tab and enter the name of the new type:

Then, go back to the Exercise tab and scroll to the bottom of the page. 

• Be sure to have Attachments enabled checked and then enable the new Attachment type by clicking on the checkbox. 

• Create custom guidance if you’d like. 

• Don’t forget to click Update exercise.

The Attachment menu item will appear at the top of the faculty member’s Assessment Exercise: 

Step 12:
Configure Units including unit memberships and overrides

“Units” is a concept used in national exercises such as REF (Research Excellence Framework) in the United Kingdom. Symplectic has very comprehensive documentation about how to manage every aspect of a REF exercise, including configuring Unit membership.

Step 13:

Configure warning and prompts

Warnings and prompts are configured under the Advanced tab and are used to guide researchers, reviewers and managers.

Step 14: Set up Assessment Reports

Making reports available for users requires that the reports be registered and made available to the appropriate exercise users. Elements comes preconfigured with these canned reports for the Assessment module:

• Full Export: An export of an individual's assessment exercise.

• Exercise Report: A report that shows the status of an exercise for groups of users. Fields include Username, User, Last login, Start date, Updated date, Contents User Lock, Current Stage and User Status.

• PBRF Progress Report: A report that shows the status of a PBRF exercise. Fields include Username, User, Last login, Start date, Updated date, Portfolio Key Fields, NRO Selections, NRO w/Contributor and NRO w/Description.

• REF Managers Publication Report: A report that shows List name, User name, Proprietary ID, Output type, DOI, Reviewer 1 Name, Reviewer 1 Comments, Reviewer 2 Name, Reviewer 2 Comments, and Journal Title. 

• Visual Progress Report: shows the status of the exercise

To make a report available, go to Reporting > Configure Reporting > Registered Reports (In 5.x: System Admin > Operations > Registered Reports) and enable the desired report by ticking the checkbox. Then, click the Set availability link and click the exercise upon which reports should be generated. Don't forget to click Save.

Step 1:
Begin by selecting Survey in the type field

Step 2:
Create a "No-items" list

Step 3:
Add Supporting fields to the "No-items" list

Remember: if you're not using a canned underlying field, create the underlying field before adding a new field.

And if you make changes to the underlying field, rebuild the reporting database.

Step 4: Configure the Stages

Step 5:

Add reviewers

Add reviewers to Stage 2 and then configure who you want to receive the Survey under the Users tab.

Step 6:
Enable Exercise

When you enable the exercise under the Exercise tab, notification will appear on the user's Home page.