Assessing ‘hard to assess’ skills

Ayesha Ahmed

Working collaboratively in groups is a crucial skill in the 21st century, information-based economy. Education systems must respond to the changing landscape of employment and one of the ways to do this is to ensure that children are taught the skills for effective group discussions and collaborative problem-solving. We know from reports by organisations such as the London Chamber of Commerce that employers and businesses expect children to leave school with these skills (Wright et al, 2010), yet we are not currently emphasising them sufficiently in classrooms – or in assessment. 

To improve the teaching and learning of group discussion and problem-solving skills we need to consider how we can monitor and assess these skills. Good assessment has a positive backwash effect on teaching and learning, raising the profile of the skills being assessed. But the imperative to assess these skills is greater than just the possible backwash effect. In order to consider the assessment of these skills we must consider what we mean by these skills: what is the construct of collaborative group work? Understanding the nature of the construct will further our understanding of how to improve learning of these skills.

I will define collaborative group work as students working in groups to have effective discussions and to solve problems. There is much work to be done, though, on identifying the important skills that make up this construct.

Three critical questions follow:

  1. What sorts of tasks will allow us to observe behaviours that show evidence of effective group discussions and problem solving? 
  2. How can we ensure that our assessments are as valid as possible, measuring the intended construct in a fair and informative way?
  3. How can we use the results of the assessment to inform the teaching and learning of these skills?

The construct as I have defined it relates to group discussion as well as problem solving. This reflects one of the issues with assessing children working in groups. Should we assess the process of the group work (often a discussion) or the products (the solution to a problem) or both? I believe that we should endeavour to assess both. 

Another critical question to consider when mapping out the skills that make up a construct is to ask what progression might look like for these skills. How do we decide if a learner has progressed? Sometimes the answer will be that they are showing more evidence of a particular skill but sometimes there can be a qualitative change in the way the skill is manifested in a performance.

Assessing collaborative problem solving

The OECD has recognised the importance of collaborative problem solving and this will be assessed for the first time in the PISA 2015 tests (OECD, 2013). In these tests children work collaboratively to solve problems with a computer agent.

The OECD defines Collaborative Problem Solving (CPS) as:

“… the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution.” (OECD, 2013:6)

In these tests the test-taker and a computer agent communicate via a chat box. The test-taker is assessed on his or her actions, communications with the agent and the products of the collaboration (solution to a problem). Using a computer agent is one way to address concerns of reliability and fairness as well as the manageability of such a large scale high-profile assessment. However, it ignores the critical social element of group work.

The Assessment and Teaching of 21st Century Skills Project (ATC21S), based at the University of Melbourne and sponsored by large corporations, involves Australia, Finland, Portugal, Singapore, England and the USA. Griffin, McGraw & Care (2012) and Griffin & Care (2015) report on the progress of this large-scale long-term project. 

One of the main target areas of the project is the large-scale assessment of collaborative problem solving which is being trialled with 11-15 year old learners. ATC21S considers how to measure the social skills of participation, perspective taking and social regulation, as well as cognitive skills (Care et al, 2015). They are trialling content-free tasks, measuring inductive and deductive thinking skills, as well as content-dependent tasks designed to measure knowledge.

Considering the process of problem solving and the social skills involved is an important part of ensuring the validity of an assessment of such collaborative work.

Assessing group discussions

What are the skills that make effective group discussions? Mercer et al (1999) discuss ‘ground rules for talk’ and show how group discussions that follow these rules can be beneficial to learning. Resources to help teachers and children to achieve this are provided on the Thinking Together project website.  These include ‘ground rules for exploratory talk’ such as “everyone in the group is encouraged to contribute” and “everyone is prepared to accept challenge”. These ground rules give us a useful way to consider how to assess group discussions. We can ask how well students are using these rules and look for evidence of this.

The Oracy Assessment Toolkit,  mentioned in a recent Commons Education Committee report (2014), identifies the set of skills that learners should develop to be good at using speech to communicate effectively with others. Many of these skills are relevant for assessing children’s contributions to group discussions.

An understanding of the skills that make up the construct we hope to assess is a necessary starting point. However, when assessing children in groups there are a number of other factors to consider:

  • Should we assess the process or the product of group work or both?
  • Should we only assess individual students’ performances? 
  • Should we also assess the performance of a group as a whole? 
  • Should peers assess each others’ contributions to the group?
  • Should students assess their own contributions to the group?
  • How do students learn from each other during the assessment?
  • What are the effects of the content or context of a task?
  • How do we decide on group composition?
  • Should children be allocated specific roles in advance?

These are all interesting questions that must be considered during any task design and trialling of assessments of group work. It is important that the difficult issues of the social dynamics of group work are not ignored but are incorporated in some way into the assessment.

Wilson et al (2012) suggest defining the construct for any group task by focusing only on aspects of performance that operate in a group situation. In this case students would be given scores based on the quality of the outcomes of their particular role (e.g. facilitator, leader). In order to be able to generalise, students must then be given the opportunity to perform in a variety of groups and in a variety of contexts, and these performances should be aggregated. This kind of sampling over a number of groups and contexts can begin to address reliability concerns.

Wilson et al (2012) also suggest that students can be given an individual performance score and a group performance score, and statistical techniques can be used to look at the relationship between these scores. They go on to discuss the possibility of group members giving feedback on each individual’s performance. This could add useful information to an assessment of group work but it must be done in a classroom atmosphere of mutual trust. 

Blatchford et al (2003) discuss some of the necessary conditions for this, including developing respectful relationships and making the physical environment conducive to group discussion. If group work is seen as a normal classroom activity then assessing this will be easier as students will be used to the idea of working together and supporting each other. 

There have been some recent criticisms of group work in classrooms (e.g. Peal, 2014) that argue that it is not effective, but as Mercer (2015) points out, group work can be unproductive when children are not taught how to do it well. If appropriate ground rules are used then it can be highly beneficial to learning.

Assessing the process of group work is likely to be more difficult than assessing the products. Methods for observing and coding or scoring elements of the process would need to be trialled and concerns of reliability and bias must be kept in mind. By sampling performances in different situations and with trust in the judgements of teachers, some of these concerns can be addressed. Furthermore, when students work collaboratively to solve problems on a computer, a wealth of data can be collected including detailed activity logs (Von Davier & Halpin, 2013). We are only now beginning to understand how such rich data can be used, but this increase in information about group processes can open up new possibilities for assessing these skills in a valid and reliable way.

Moving towards assessing ‘hard to assess’ skills

It is clear that the question of how to assess children working collaboratively in groups is not an easy one to answer. These are hard skills to assess. If something is hard to assess should we stop trying to assess it and concentrate instead on the well-trodden paths of assessing knowledge and understanding in written examinations? To some extent this has occurred. For example, from summer 2014 the speaking and listening component of GCSE English was removed from grading so that students’ performances on this critical element of English do not count towards their final grade. Part of the reason for this change is that these skills are hard to assess in a way that is accepted as valid and reliable. However, there are also examples of successful assessments of hard to assess skills such as the Extended Project Qualification.

My view is that we cannot afford to ignore the importance of the skills required for effective group discussion and problem solving. These are important skills for today’s children to leave school with. They are skills that can improve their learning as well as prepare them for life beyond the classroom. One of the ways to ensure that these skills are taught well is by designing good quality methods for assessing them. The assessment community, including teachers, exam boards and researchers, should work together to try to develop valid assessments of complex, interactive tasks such as collaborative problem solving, with the aim of assessing the process as well as the outcomes of these tasks. As more valid and reliable assessments of ‘hard to assess’ skills are developed, the government, Ofqual and exam boards should consider how these could be incorporated into qualifications in the future.

If we assess only what we can easily measure then we end up defining constructs in terms of what can be assessed rather than what is important in a domain. This could have a devastating effect on the richness and breadth of learning in the classroom. If something is hard to assess we should not simply avoid assessing it, but instead we should look for better ways to assess it.

A blueprint for assessing ‘hard to assess’ skills

The assessment community should work to develop valid assessments of complex, interactive tasks such as collaborative problem solving, with the aim of assessing the process as well as the outcomes of these tasks.

As more valid and reliable assessments of ‘hard to assess’ skills are developed, government, Ofqual and exam boards should consider how these could be incorporated into qualifications in the future.