Zoom Logo

KE Concordat Evaluation Process (England) - Shared screen with speaker view
Shanta Coles
25:44
Hi Everyone, Shanta Aphale-Coles here, University of Manchester.
Helen Lau
26:28
Hi everyone Helen Lau, Coventry University
David Hartley
26:30
Hi there everyone - David Hartley, Oxford Brookes
Phil Fiddaman
26:41
Good Afternoon all. Phil Fiddaman from the University of Hertfordshire
Allice Hocking
26:47
Afternoon everyone - Allice Hocking from the University of Plymouth
Rob Singh
26:49
Rob Singh, University of Essex - hello!
Alison Price
26:54
Good Afternoon! Alison Price, representing EEUK (Enterprise Educators UK)
Mark Claridge
26:56
Afternoon all, Mark Claridge from the Institute of Cancer Research
Ceri Jones
27:01
Ceri Jones, Swansea University
Julia Keyton
27:04
Hi Everyone, Julia Keyton here, from University of Liverpool
Alexis Holden
27:08
Good afternoon, Alexis Holden from University of Central Lancashire, Preston
Dan King
27:10
Dan King from Research Consulting, hello!
Heather Cobb
27:14
Afternoon all - Heather Cobb, University of Kent
Sue Graham
27:45
Hi everyone - Sue Graham Northumbria University here - NB make sure you set your chat to 'all panellists and attendee's as it defaults to panellists only! Sue
Sue Graham
28:04
*attendees
Amelia Roberts
28:11
Great tip, thank you Sue!
Mel Nakisa
28:16
Hello, Mel Nakisa, Bucks New University
Sarah Priston
28:28
Hi everyone, Sara Priston from Bath Spa University
Tracey Wond
28:28
Hi All - Tracey Wond, HEIER (Higher Education, Impact and Evaluation Research) and UK Evaluation Society
Paul Manners
28:29
Hello everyone. Paul Manners from National Coordinating Centre for Public Engagement here.
Amanda Baxendale
28:29
Hi All, Amanda Baxendale, University of Derby. Good tip Sue Graham!
Fatimah Awan
28:37
Hello Everyone. Fatimah Awan, Guildhall School of Music & Drama
Lesley Goodship
28:39
Thanks Sue, hadn't picked this up
Susan Matos
28:50
Afternoon everyone, Susan Matos, University of Reading
Amie Longthorne
29:00
HI All , Amie Longthorne from University of Liverpool
Jeremy Hague
29:03
Hello - Jeremy Hague, Nottingham Trent University
Emma Burke
29:03
Hi All. Emma Burke Innovate UK (UKRI)
Liz Jenkinson
29:10
Hi everyone. Liz Jenkinson from the University of Hull.
Paul Roberts
29:15
Afternoon all, Paul Roberts, University of Sussex
Sophie Jones-Tinsley
29:19
Hello all, Sophie Jones-Tinsley from Leeds Trinity University
Liam Sutton
29:25
Hi - Liam Sutton, University of Bradford
Andrew Jackson
29:30
Hello, Andrew Jackson, Bishop Grosseteste University in Lincoln
Ruth Rayner
29:32
Hi everyone - Ruth Rayner, Leeds University
Sarah Roberts
29:32
Afternoon all. Sarah Roberts from Liverpool School of Tropical Medicine
Michael Capaldi
29:39
Mike Capaldi, Newcastle University - greetings all
Sarah Duckering
29:46
Feels like we're almost all in the same place. Good to virtually see everyone - Sarah from University of Portsmouth
Tatiana Schofield
30:32
Greetings all - Tatiana Schofield, RCA
Claire Walsh
30:53
Good afternoon all, Claire Walsh from Research Consulting
Hannah Timpson
30:58
Hello everyone, Hannah Timpson, LJMU
Jonathan Bradley
31:08
Jonathan Bradley, University of Sheffield, Arts and Humanities KE, hi everyone, good to be here...
Michael Millan
31:14
Afternoon everyone, Michael Millan, Anglia Ruskin University
Lisa Williams van Dijk
31:22
Hello everyone. Lisa Williams van Dijk Royal Agricultural University
Emma Calverley
32:47
Hi - Emma Calverley, University of Hull
James Perkins
34:22
As a small provider, the overlap with REF does present capacity challenges to mobilise people around this
David Hartley
34:35
Phil - when are institutions going to formally sign up to the Concordat?
Kathryn Walsh
35:12
Some concerns with overlap with REF preparation.
Jonathan Bradley
35:19
I think a stretching commitment is good
Stephanie Maloney
35:21
i'd echo capacity issues alongside REF submissions in smaller institutions
Chris Hughes
35:22
Would like a significantly larger gap between this and REF
David Oswell
35:24
Would be good to delay for a few months. It will be a demanding year!
Nic Percivall
35:54
Concur with James P, while generally OK with timeline, it will be a challenge for HEIs with small RKE offices
Muriel Swijghuisen Reigersberg
35:59
Bottleneck in prep for both REF and KEC, even if deadline is later. Same staff, as you say. We also don't yet know when UKRI evidence submission for Impact will be, which will need to be wrapped up. Evaluation after REF would be better.
Shanta Coles
36:14
A specific date for KEC submission would help with internal planning.
Daniel Stone
36:14
It would be helpful to understand the expected detail / length of the action plan and assessment process (it would be manageable if the action plan is reasonable)
Liam Sutton
38:57
Presumably we can tick yes to both...
Kathryn Walsh
39:16
Important that this is owned at an institutional level, and not just something done by the KE office.
Sarah Priston
39:22
can't access the poll as am using web browser option
Jo Allatt
39:49
Just comment on here if you cannot access the poll
Charlotte Patrick
39:56
i cant access the poll either as using web browser
Shanta Coles
40:16
Head of Institution may nominate another of their Senior Leadership Team instead...
Andrew Jackson
40:30
i cant access the poll either as using web browser
Phil Fiddaman
40:42
I can't access the poll as using web browser also
Stephanie Maloney
40:59
We have a various other concordats that we are developing actions plans for eg Career Development of Researchers, Research Integrity etc - please can we have some harmonisation in action plans and reporting requirements!
Hannah Timpson
42:08
I would support the previous post - we are hoping to align the self-evaluation activities for both the KE and Researcher Development Concordats in order to avoid duplication of effort.
Amelia Roberts
42:21
I think some guidance and an exemplar would be helpful.
Newell Hampson-Jones
42:54
Guidane
Carole Barron
43:04
I think it would be useful to have guidance please
Chris Hughes
43:11
Not clear yet on how the self-evaluation summary would be used (other than how used by us) and the need for us to submit it
Alicen Nickson
43:25
We have found the model we used for the HR Research Excellence Concordat a helpful example.
David Oswell
43:29
Guidance is good
Phil Fiddaman
43:39
Some guidance would be very welcome
Newell Hampson-Jones
43:46
Guidance is necessary so we have confidence in what we are submitting. Without it, it will be harder to get a clear and fair response to our evaluation.
Tamsin Mann
43:53
I'm hoping that PraxisAuril, NCCPE and ARMA can work together to provide some guidance across our memberships in this area.
Michael Millan
44:32
Need some degree of guidance to ensure consistency and evaluation against the same things.
Shanta Coles
45:38
No guidance would give freedom to institutions not normally permitted, but may make evaluation difficult..
Liz Jenkinson
45:43
Guidance would be helpful to achieve a degree of consistency across institutions. In addition, without guidance it's likely to take much longer to produce the self-evaluation due to the likely internal debate about how to do this.
Rachel Bowen
45:45
Guidance would be welcome, as it would give some consistency and confidence in the process of self-evaluation
Newell Hampson-Jones
45:47
From a standards perspective, guidance is not in itself a mandate, it's support. Mandates are more akin to codes of conduct etc.
Alex Anderson
45:51
Would like some guidance please!
Jane Turner
46:02
Guidance would be helpful
Nazira Karodia
46:03
i agree with the comments on getting some guidance.
Heather Cobb
46:04
I agree guidance would help for consistency
Peter Griffiths
46:05
Guidance is essential to ensure a useful and uniform process
Gary Shuttleworth
46:05
Specific guidance would be useful
Hilary MacQueen
46:06
If there is no mandated template people are likely to duplicate other evaluations, so this carries a risk of duplication of reporting. Why bother?
Jonathan Eddy
46:07
Prefer guidance. Otherwise it is a guessing game that can be time consuming in itself!
Sue Graham
46:08
I agree with Liz and Rachel!
Alexis Holden
46:14
Clarity around the scope of the self-evaluation would be helpful
Amanda Baxendale
46:15
I think guidance need to be provided by the assessment body - even if developed by PrA and others
Amelia Roberts
46:22
Lack of guidance opens up risk of under/overdeveloping the action plan.
Mark Claridge
46:24
Some guidance would be helpful to avoid it being the interpretation of one person at a given institution. And there’s clearly a difference between the concept of guidance and detailed guidance!
Sophie Jones-Tinsley
46:31
Guidance would be welcomed- not too prescriptive but provides an outline of what's expected to some extent
Alison Price
46:39
EEUK members report lack of consistency in current metrics/national work and always attribute this to interpretations on guidance - so some 'boundaries' or guidance/expectations and samples might be needed
Alisdair Aldous
46:40
Guidance can be non-prescriptive but still help to ensure some level of consistency in approaches across the sector.
Julia Keyton
46:46
Also agree with comments above about some guidance being helpful
Dan King
46:49
Is there enough experience and understanding on self-evaluation to allow a "no guidance" approach to be successful - identifying and supporting good practice in this area is needed I feel.
Lesley Goodship
46:50
Outline guidance would be helpful
Tamsin Mann
46:56
I think that member organisations can work together to provide guidance but we really do need to understand the purpose of the evaluation and what it will be used for. I agree with comments about consistency of approach here.
Erica Williams
47:01
Guidance would be useful, otherwise we could go too far or too short.
Rob Roze
47:02
Guidance would be useful, however that could change our perception on how workable are the timelines.
Jonathan Bradley
47:04
Definitely some guidance, doesn't have to be mega detailed, can reference the good practice that Praxis Auril, NCCPE promote
Muriel Swijghuisen Reigersberg
47:06
Non-prescriptive guidance helpful, I feel, to help with achieving consensus across HEI.
vcapdt
47:07
Template and least and a framework definitely needed if no guidance is being provided. You will get a wildly varied style of concordats of varying lengths if not.
Daniel Stone
47:08
A framework would be helpful for efficiency and consistency; I think the eight principles should map quite well/easily against some guidance
Nic Percivall
47:12
Need more clarity on the process and evaluation to ensure action plans are meeting intent. Guidance / best practice examples should be part of this as a no guidance approach could lead to additional burden
David Hartley
47:22
Happy not to have guidance but how will this lead to comparability across the sector? Is there an intention to benchmark or not?
Sarah Priston
47:31
Good practice guidance helps with Research Integrity Concordat (this is provided by ARMA/UKRIO)
Chris Hughes
47:38
Why would we be required to submit our self-evaluation summary? How will it be used?
Michael Hill King m00351167
47:40
The Consordat is across the sector as a tool for self-criticism. To ensure regualrity of interpretation of the Concordat, guidance and, to a lesser extent, assessment tools would be useful.
Sue Graham
47:53
Definitely looks like a majority in favour of guidance being provided...
Rob Singh
48:14
I agree guidance would be helpful - particularly as there would be a number of ways this could be approached
Andy Carr
48:21
Guidance for consistency. Otherwise results will be impossible to compare between institutions
Liam Sutton
48:24
Depends on the consequences of "getting it wrong" - if our self-evaluation is to be /criticised/ then there should be some sort of framework.
Helen Lau
48:26
Guidance would be helpful particularly against what will be Concordat evaluation at a later stage. Without guidance there will be more freedom to be unique as an institution but less ability to compare institutions with one another
Steven Hardy
48:44
The panellists will need some understanding of self-evaluation methodologies to ensure consistency
Muriel Swijghuisen Reigersberg
48:55
Disciplinary variances will also be easier to reconcile, with some non-prescriptive guidance.
Sarah Priston
49:08
With HR Excellence there is no template for a Gap Analysis, but a template for the Action Plan which is helpful to ensure consistency across sector and with evaluations
Kathryn Walsh
49:27
Useful if we can structure in a way to stimulate learning across institutions.
Paul Roberts
49:27
Needs to seen as of similar prestige as other concordats. Guidance to help with evaluation and transferability across institutions important.
Nic Percivall
49:57
A range would be very welcome
Liam Sutton
50:09
Sympathy with the diversity argument against getting too specific
Jo Allatt
50:47
Helen, The idea is not to compare institutions against each other but to evaluate institutions on their own merit in relation to their institutional vision
Amelia Roberts
52:09
Could we judge scale using the cluster model research done for KEF? Page limits would be helpful.
Michael Millan
52:31
Page limits are useful, but think there should be one limit for all. Smaller institutions can always write less.
Rob Roze
53:30
Page limits definitely are needed. Also presumably pages will be text only.
Alexis Holden
53:59
Page limits based on FTE possibly; similar to environment statemetns in REF
Nic Percivall
53:59
Page limits useful, perhaps an upper limit, but no requirement for every HEI to reach the upper limit. Reflecting scale of HEI
Sophie Jones-Tinsley
54:02
I agree with Michael regarding the page limits
Tim Strickland
54:21
would a max page limit to be used at the discretion of individual institutions be relevant
Liz Jenkinson
55:11
Page limits would be helpful
Carole Barron
56:49
How does the Concordat fit in with other external evaluations such as the Annual Monitoring Statement linked to the HEIF Strategy?
Heather Cobb
56:58
Page limits would be helpful as a guideline for institutions
Rachel Bowen
56:58
How are you envisioning HEIF strategies and KEC action plans working together?
Hilary MacQueen
58:07
@Rachel - good question!
Sarah Priston
58:15
@ Rachel comment above - Not all HEIs currently receive HEIF so this needs to be borne in mind if are linking this KEF to HEIF strategies
Peter Griffiths
58:16
Clarity around HEIF/KEF/KEC (indeed REF) alignment / interrelationships would be helpful.
Sarah Priston
58:53
@ Peter - agree!
Nic Percivall
58:59
Second @Peter
Hilary MacQueen
59:09
Agree @Peter
Nazira Karodia
59:19
Agree@Peter
Daniel Stone
01:00:48
Can you clarify how the action plans will be used post submission? Will they be confidential or made available to the public?
Muriel Swijghuisen Reigersberg
01:02:18
Gender balance (Athena Swan) in KEC activities, promotions for staff leadership alongside ECR career development etc.
Paul Manners
01:02:30
Great ambition to be non-competitive. If the goal is to maximise learning then we need to encourage honesty about what isn't working. Is this where the gap analysis comes in? Are there other processes people have been involved in that have worked really well to elicit useful learning?
Carole Barron
01:03:51
Evaluator per university type - why don't you use the KEF Cluster Groups as universities have been put in Clusters based on their 'type'
Paul Roberts
01:04:21
Carole - you beat me to it.
David Hartley
01:04:32
@Carole - agree
David Oswell
01:04:38
yes agree
Sophie Jones-Tinsley
01:04:40
I agree with Carole
Peter Griffiths
01:04:48
@carole - agree
Sue Graham
01:04:54
Yes good idea
Jane Turner
01:05:02
Agree
Carole Barron
01:05:11
Why is a general call for nominations being sent to NCUB partners. You could use PraxisAuril as well
Jo Allatt
01:05:56
We will Carole, the NCUB point was about going beyond HE
Sarah Priston
01:06:37
also need to consider 3rd sector and social enterprise stakeholders as evaluators
Muriel Swijghuisen Reigersberg
01:07:24
Agree with Carole, although we are a four nations institution with a non-traditional/ non-campus set-up so perhaps have slightly different opportunities and challenges to implementation and evaluation of KEC principles.
Phil Fiddaman
01:08:05
What about involving the AUTM in the evaluation for an international perspective?
Sue Graham
01:08:30
Also within HEIs could to hear voices beyond the KE Offices - e.g. would be good to get HR colleagues as an example as much of the concordat relies on HR policies being amended (reward & recognition, training and devt etc)...
Amanda Baxendale
01:08:42
Membership organisations could offer a route to non-academic evaluators but shouldn't be the only route
Sue Graham
01:08:43
*good to
David Hartley
01:09:44
Sounds like 20-25 evaluators
Nic Percivall
01:10:19
That matches my maths!
Michael Millan
01:13:14
Every 3 years sounds about right.
Jo Hemmings
01:13:16
If this is a pilot, and is non-competitive, why are we killing ourselves to fit this into a timetable which will be impossible for many HEIs, bearing in mind REF clashes and potential loss of colleagues due to COVID-related cutbacks in HEIs??
Amelia Roberts
01:14:05
I agree with Jo.
Shanta Coles
01:14:12
Every 2-3 years.
Dan King
01:14:15
As a non-competitive process a rolling approach is an option - e.g. taking 1/3 of universities per year over three years. This also reduces evaluator fatigue and helps build learning from one year to the next.
Michael Capaldi
01:14:25
Every year becomes over kill - every 3 years feels about right
Kathryn Walsh
01:14:26
Need to keep this on the internal agenda to keep up momentum for positive change. Maybe 3 year full refresh - with annual updates with lower burden.
Emma Calverley
01:14:36
every 3 years
Jonathan Eddy
01:14:44
I like Dan King’s idea of a rolling process.
James Perkins
01:14:52
Presumably every 5 years to coincide with the action plans?
Peter Griffiths
01:14:57
@Jo … I agree, we need to perhaps not overburden the sector with too many unconnected reviews, though there is benefit in this to help meet the 2.4% agenda
Nic Percivall
01:15:04
Large exercise every 5-6 years sounds too close to REF and burden associated. Prefer the more frequent, ligher touch suggestions. Also rolloing process approach
Susan Matos
01:15:06
@Kathryn - my thoughts exactly. Useful to keep it on the agenda, and to hopefully retain some internal knowledge of the previous submission.
David Hartley
01:15:11
Suggest 5-year plan / time horizon and 3-year interval
Sarah Priston
01:15:12
I think it would be good for evaluators to learn from other types of HEIs so cross-cluster seems appropriate
Peter Griffiths
01:15:17
Correlating the timelines & purposes of these exercises is key…
Nazira Karodia
01:15:49
Is there some learning from other countries that we can build on?
Rob Singh
01:15:51
if it is anticipated that action plans would need to be iterated / refined then 5 years seems too long
Tracey Wond
01:15:53
The least competitive process I know is that of PRME - Principles of Responsible Management Education. Formative, self reflective, on a signatory basis with a two year progress report/action plan, report available publicly. The expansiveness of this compared would make 3yrs+ logical.
Sue Graham
01:16:07
Yes agree @kathryn - this will help drive up change and improvement and keep momentum
Muriel Swijghuisen Reigersberg
01:16:24
3 - 5 years seems reasonable provided it remains light touch. It will keep momentum going and assist with other exercises such as various Concordats, Athena Swan, and eventually REF to monitor progress as we go along.
Jonathan Bradley
01:16:25
the 5 year / 3 year option works for my thinking
Liam Sutton
01:16:34
It's a report on how we're delivering our individual strategies, so some sort of rolling process which ends up synchronising with institutional 5-year cycles might be worth considering.
Tim Strickland
01:16:40
frequency may depend on how onerous/valuable the pilot is, so may need a short pause after the pilot, but 3 years sounds right
Liz Jenkinson
01:16:49
Every 3 years feels about right.
Jonathan Eddy
01:16:54
@Jo I understand your concerns but this effort could/should help Government feel comfortable with continuing to fund HEIs to do KE, as well as enabling improvement across sector and within HEIs.
Steven Hardy
01:16:56
@Dan King, I agree that a rolling process should be considered as an option
Julia Keyton
01:16:58
Agree with 3 years
Carole Barron
01:17:41
@Jo I thought that was the purpose of the KEF
Liam Sutton
01:18:54
Agree with Carole. Concordat is more about self-improvement than demonstrating value to HMG
Michael Millan
01:21:16
Without wanting to kill the panel members, is a single iteration enough for the first run through? HR Excellence in Research which has a similar process at least has the possibility of informal feedback before formal submission - could that be incorporated? Or alternatively a second go if submission needs more work.
Carole Barron
01:22:10
I support @Michael's comment
Jo Hemmings
01:22:12
I'm unclear on what this feedback will look like, so difficult to know how to respond to the question being asked
Hilary MacQueen
01:22:38
Where deficiencies are identified it would be helpful to have suggestions about how to improve - actual practical hints and tips.
Muriel Swijghuisen Reigersberg
01:22:46
How much of the evaluation material will make its way back to government and in what format do you think, if the motivation is to reassure government we're trucking along nicely, internationally
Sue Graham
01:22:47
Yes would be helpful to see an exemplar of what we are talking about re feedback
Jane Turner
01:23:01
A grading structure would not be helpful
Kathryn Walsh
01:23:13
Would a "disappointing" submission ever be asked to resubmit?
David Hartley
01:23:28
Will there be any public disclosure or summary of the feedback - anonymised or otherwise?
Nic Percivall
01:23:49
Will the next evaluation refer back to the earlier one, if the evaluation is HEI context based, when/where will the 'what next' be evaluated and how?
Jo Hemmings
01:23:50
@Kathryn: to add, what would make a submission disappointing?
Rachel Bowen
01:23:53
It would be good to have suggestions about how to improve and perhaps a KEC network or similar, where we can learn from each other & share best practice.
Paul Roberts
01:23:57
Internally this could be used for performance management for the named contact.
Michael Capaldi
01:24:28
Some idea of how your evaluation compares with other 'sector' comparators would be useful - not ranked, but enough to get a view of how you're doing against the 'norm'
Carole Barron
01:24:52
Given the challenging environment we are all working in at the moment, pandemic and severe financial constraints HE is facing, will there be an opportunity to explain why an institution may not be able to fulfil their commitments that they signed up to at the beginning?
Andrew Fairweather-Tall
01:24:59
I wonder whether in the pilot it would be appropriate to investigate the resources assigned by HEIs to write it
Shanta Coles
01:25:57
As well as ways to improve, also celebrating examples of excellence...showcase short film or similar?
Jo Allatt
01:26:22
@kathryn, we don't expect to have a pass/fail for this pilot. However there is a chance you might want to work on your action plan once reading your feedback if it suggests there is a lot of room for improvement
David Oswell
01:26:33
thank you phil
Hilary MacQueen
01:26:36
Where will the slides be?
Amelia Roberts
01:26:37
I like Shanta's idea.
Michael Millan
01:26:46
Agree that some comparison to best practice in evaluation would be helpful.
Jane Turner
01:26:51
Many thanks
Sophie Jones-Tinsley
01:27:10
Many thanks for this webinar- provided some food for thought
Nic Percivall
01:27:15
Very helpful update and direction of travel, thank you
Liz Jenkinson
01:27:26
Really helpful webinar - thanks.
Peter Griffiths
01:27:28
Thanks!
Hannah Timpson
01:27:34
Thank you!
Shanta Coles
01:27:37
Thanks, very helpful and good to chat to everyone.
Emma Calverley
01:27:40
Thanks!
Newell Hampson-Jones
01:27:40
These webinars are great, thank you.
Nazira Karodia
01:27:42
Thank you for the presentation, very helpful.
Sarah Duckering
01:27:43
Really helpful - is it perverse to be looking forward to it!
Hilary MacQueen
01:27:44
Thank you all, and good bye.
Phil Fiddaman
01:27:44
Many thanks for a very useful webinar
Liam Sutton
01:27:45
Thanks all
Chris Hughes
01:27:50
Thanks
Rachel Bowen
01:27:50
Thanks - really helpful presentation
Sarah Priston
01:27:51
@ Shanta - examples of good practice would be really helpful
Michael Millan
01:27:52
Thanks, good discussion.
Allice Hocking
01:27:55
Very useful -thanks
Dan King
01:27:58
Thank you, very helpful
Amanda Baxendale
01:27:58
Thanks All
Kathryn Walsh
01:27:59
Thanks for taking the time to update!
Lisa Williams van Dijk
01:28:03
Thank you, very helpful!