Making A Statement

qLast week, the Sutton Trust published a Research Brief that I co-authored with the HE Access Network. The theme is a familiar one for me: the UCAS personal statement. I’ve blogged about it here and here, written a previous Sutton Trust report, and published findings in an academic journal and a book about global HE admissions practices.

Saint-Mary-s-School-oqLD3CThis study was a really interesting addition to the evidence because it was the first to compare how teachers at state schools and admission tutors at high-prestige universities read statements. The results were alarming: what teachers think make a good personal statement is a far cry from what universities are looking for.

The researjohnhumphMS2010_468x402ch attracted plenty of press attention, including an excellent opinion piece by Catherine Bennett for the Observer. Other print coverage included reports in The Sun, The Times, The Daily Telegraph and the Times Education Supplement. My interview on the BBC Radio Four’s Today programme is available here (listen from 52’45”) until February 26th 2016.

While I think personal statements offer a useful lens through which to view distributions of social capital and explore teenagers’ self-conceptualisations, I’m hoping this will be the last time I write about them. A review of their use in the application process – ideally as part of a wider review of the HE admissions in the UK – is long overdue.

 

Advertisements

Who gains from the grumbles?

respect-tiles-wonkhe-1024x768

Note: this piece was originally published by WonkHE on January 11th 2016.

“My students have paid £9,000 and now they think they own me” runs the headline. It’s one of those anonymous pieces, so the wider context is difficult to figure out, but the author seems troubled by a message that reads “all I’m asking for is a little respect seeing as I pay you £9,000 a year”.

It’s the “blunt, consumerist language” that offends the author, and a number of anecdotes follow, each reinforcing this interpretation. “If you ask me,” quips a colleague in the car park, “all universities are going to need a customer services department before long”. Another claims a student once told them: “I pay you to teach me what’s in the article, not the other way around”. The author recalls how very different they had “acted and spoke” when at university – assignments were completed punctually, guidelines followed diligently, etc. How they wish they could say the same of their students now.

passengers.jpgSuch rhetoric is becoming familiar on English campuses, and the points about unfair workload allocation, expectations of across-the-board excellence, and often counter-productive management culture all deserve to be made forcibly and repeatedly to policy-makers, sector representatives and intuitional leadership teams. But venting at students about how universities are funded is like confronting fellow passengers because your train is running late.

Remember, the student’s plea is not for higher grades, quicker feedback or the guarantee of a graduate job, but for “a little respect”. Is this really a case of neoliberal higher education policy coming home to roost? Or is it something altogether more localised and petty?

images22Perhaps the student was wrong to mention fee levels at all. But let’s not forget the extent to which the 2012 funding system has driven higher education to “hurl the cost of itself at graduates”, as Jim Dickinson recently noted on this site. According to the Sutton Trust, only one in twenty will now repay their debt in full by the age of 40, compared to almost 50% under the previous system. An average teacher will still owe £25,000 by their early 50s. The freezing of the repayment threshold will make an undergraduate degree more costly still and, last year, we saw maintenance grants turned into loans and student nurses stripped of their bursaries.

It’s naïve to believe that such wholesale reconfiguration of the way in which our sector is funded won’t disrupt the nature of undergraduates’ engagement with their university or change academics’ working conditions. That’s exactly why our students were placed at the heart of the system – so they’d behave like consumers and enact the marketisation agenda.

Teaching-Excellence-Framework2However, in many respects, they’ve refused to play ball. Take the proposal to link success in the Teaching Excellence Framework to higher fees. The National Union of Students objected immediately, taking a position of principled disengagement long before the rest of the sector began to follow suit. Yes, there are some individual undergrads who’ll seize their rights as newly-empowered service users to make unreasonable demands on staff as they seek to maximise their return-on-investment. But there are millions of others who don’t measure their experience in solely utilitarian terms and want their time at university to be inspiring, cordial and enlightening.

The nameless author of the piece fantasises about replying with: “Hey student – all I’m asking for is a little respect, seeing as how much you pay makes no difference to my wages, yet the level of support I am forced to offer you takes up 80% of my time despite the fact that teaching still only equates to 33% of my workload.”

Is support for students really something that academics are “forced” to offer? And if we must gripe about our salaries, might it be judicious to acknowledge the inter-generational unfairness that the current funding model precipitates?

arguing.pngBut the bigger question here is who gains from such grumbles. A frostier relationship between students and academics doesn’t benefit those who yearn for campuses of old. Rather, it benefits those who seek to marketise and instrumentalise the sector further. Undergraduates can be framed as dissatisfied customers, then as budding agents of change, while academics can be positioned as ivory-towered and over-protected. Many of the 4,000+ comments beneath the original piece offer precisely this reading.

But the student-academic relationship at English universities is surely stronger than such simplistic polarisations allow. Is a little respect really too much to ask for?

Why research and teaching need to be reintegrated as well as rebalanced

Note: this piece was originally published on the All-Party Parliamentary University Group website. Further details of the presentation I gave to the Group are here.

Among the stronger arguments made in the government’s Green Paper is that a ‘rebalancing’ of research and teaching in Higher Education is needed. As a sector, we’ve become accustomed to close scrutiny of our research while our teaching has largely remained unaudited, sometimes reliant on the dedication of personally committed academics. But there’s an equally strong case to be made for 4research and teaching to be reintegrated. What makes students’ learning at university different from earlier, more instrumental educational experiences is the opportunity to be immersed in a culture of scholarly enquiry and research advancement, to learn first-hand from those leading their field, and to conspire in the creation of new knowledge. In measuring teaching, we must take care not to set it further adrift from research.

For any teaching audit to benefit the sector, buy-in from both students and academics is vital. Attempts to frame the Teaching Excellence Framework (TEF) as siding with long-suffering undergraduates are undermined by ‘principled disengagement’ from the National Union of Students. The link with fees makes the TEF the hardest of sells to the ‘consumer’ it supposedly empowers, especially now maintenance grants have become loans and repayment thresholds are frozen.

For academics, the risk is that separate audits for research and teaching put the sector in a state of perpetual preparation and further fuel the kind of game-playing ‘industries’ that the Green Paper rightly chides. A better integrated, lighter-touch framework might allow more time for universities to do what matters, instead of just reporting it in the most favourable terms possible.

2015-LT-conference-37-1038x576

The questions a TEF might most usefully ask of the Higher Education sector are those that encourage us to make better use of our data, communicate more clearly with applicants, and draw on our own research to ensure that every student receives the teaching and support that’s best suited to their needs. For example, we know all about key outcome differentials, such as the relative under-attainment of Black and Minority Ethnic students compared to White students. But how do we address them? Part of the answer surely involves research. We need to understand better how cohort and staff diversity, curriculum design and campus culture affect performance.

201CKm_cEcUAAEB3arIndeed, one problem with relying on metrics is that some are such distant relations of teaching quality that they’d barely recognise one another. Graduates salaries, for example, are predicted much more by subject choice, university prestige and social capital than by how effective your lecturers were. Similarly, high satisfaction scores can be achieved by pleasing students rather than challenging them. In so diverse a sector, metrics can never tell the whole story.

Would-be students will benefit far more if universities – and then disciplines – created their own narratives. Many young people find their school-to-university transition difficult to negotiate and would benefit from clear, evidence-based guidance about the pedagogical approach and distinctiveness of individual courses.

2imagesThe Impact and Environment Statements used in the Research Excellence Framework (REF) offer useful potential templates. Teaching impact could be evidenced by localised measurements of learning gain; teaching environment by learning culture and staffing strategy, as well as by facilities and extra-curricular learning opportunities. Emerging narratives would be accompanied by relevant supporting evidence, such as student attendance at research seminars, the ratio of contact hours spent with senior academics relative to teaching assistants, the retention and performance of WP students relative to non-WP students, etc.

Eventually, any ‘excellence’ framework will get gamed. What’s arguably more important is the direction in which it nudges the sector and the behaviours it implicitly encourages. As universities grow more confident in their own research into Higher Education and articulate richer pedagogical narratives, the TEF’s role may develop into one of overseeing panel assessment rather than imposing metrics of its own. A low-maintenance REF and low-maintenance TEF could evolve and coalesce according to consistent underlying methodological principles, and in ways that allow research and teaching to complement, not compete with, one another.

What’s in a name?

Note: this piece was originally published as Anonymising UCAS forms is only a first step towards fair and discrimination-free university admissions on LSE’s Democratic Audit blog.

Glasgow University, Credit: Chor Ip, CC BY SA 2.0When pledging to make university applications “name-blind”, the Prime Minister yesterday cited research showing that top universities make offers to 55% of white applicants but only to 23% of black applicants. From 2017, universities will follow major employers that “recruit solely on merit” by offering anonymity to their applicants.

In many respects, this is a sensible move. Universities can hardly claim immunity from ‘unconscious bias’, and admissions processes could be seen to exemplify the “quieter and more subtle discrimination” that the Prime Minister wishes to address. However, those of us who have looked closely at the issue would argue that concealing candidates’ names does not go far enough.

In 2012, I authored a report for the Sutton Trust showing that the quality of UCAS personal statements could be predicted by applicants’ school type. For example, those from Sixth Form Colleges and Comprehensive Schools made several times more basic spelling and grammar errors than those from Grammar Schools and Independent Schools. Ethnicity was also a major factor, with British-Bangladeshi applicant making 2.29 errors per 1,000 words of statement, compared to white applicants’ 1.42 errors. All of the statements I examined were written young people who went on to achieve identical grades at A-level. The differences in their statements were not down to ability; they were down the amount of help and guidance available.

There are other ways in which our university application systems may reproduce existing forms of privilege. Candidates from the fee-paying sector are much more likely to mention the name of their school in their personal statement, even though this information is captured elsewhere in their application, perhaps as a means to accentuate their perceived fit for leading universities. Social capital is demonstrated through prestigious work placements, internships and job shadowing experiences; cultural capital through overseas trips and LAMDA examinations. Evidence suggests that interviews are no less discriminatory, with some candidates drilled extensively in how to perform under pressure while others remain intimidated by an unfamiliar, hostile environment.

So how should selective universities select when almost every indicator is potentially problematic and we cannot be trusted with even a candidate’s name? An extreme solution, favoured by some European countries, is to allocate places on over-subscribed courses based on a lottery for those who meet a minimum academic threshold. Other nations, notably the USA, ask for statements but offer greater reassurance to students from under-represented backgrounds that their application will be read in its appropriate context and the odd spelling mistake will not count against them. Few nations rely on the personal statement as much as the UK. However, with Independent Schools increasingly competing with one another on entry rates to leading universities, and with new markets emerging around the tutoring and coaching of applicants, the pressure to maintain the status quo is considerable.

The Prime Minister is right to say that the UK Higher Education sector needs to take a close look at why young people from some backgrounds can be disadvantaged in the application process. We also need to understand better why ethnicity predicts the likelihood of graduating with a higher degree award. But to stop at anonymised applications would be to pretend that the root of the problem is a handful of prejudiced admissions tutors. The candidate’s name is not the only issue. Indeed, this information may allow more sympathetic admissions tutors to make appropriate allowances. If the goal is to bring greater fairness to the process, we also need to think about more systemic issues, such as why offers are made on predicted rather than actual grades, how candidates’ attainment can be suitably contextualised, and why personal statements are given more prominence than any evidence suggests they are worth.

“Fulfilling Our Potential”: what policymakers’ rhetoric reveals about the future of Higher Education

Note: this piece was originally published on LSE’s British Politics and Policy blog (September 29th 2015)

Jo-JohnsonSpeaking earlier this month at the Universities UK Annual Conference, the Minister for Universities and Science, Jo Johnson, offered few new pointers about the upcoming Teaching Excellence Framework (TEF) but may have revealed more – not necessarily intentionally – about the government’s broader view of the Higher Education sector.

Take the comment about new providers’ courses being validated by established universities. According to the Minister, it’s “akin to Byron [Burgers] having to ask permission of McDonald’s to open up a new restaurant.” The point being made is clearly about perceived anti-competitive practices, with eager young upstarts being denied market entry by larger, entrenched operators. However, the metaphor is a curious one, with the new providers framed as purveyors of posh hamburgers and their validating institutions as lower-end fast-food joints.

Byron-Burger-02-hop-1-interiorsNick Hillman cleverly traces the root of the analogy to Matthew Batstone, co-founder of the New College of the Humanities, whose favoured version was more confectionary: Batstone likened the validation process to new chocolate bar manufacturers needing approval from Mars Bar. However, as John Gill notes, “high-profile problems have dogged attempts to inject competition” into the Higher Education arena. And though some alternative providers have targeted Byron’s end of the market, not all private colleges have been made of the finest ingredients, as investigations by Andrew McGettigan and others have shown.

Also revealing was Johnson’s definition of inspiring academics as those “who go the extra mile, emailing feedback at weekends and giving much more of their time than duty demands”. Within the sector, eyebrows lifted at the expectation that university staff should work harder still, and the implication that their weekends aren’t already spent on the job. Some wondered whether excellence within all professions would now be judged on out-of-hours contributions, and questioned why academics’ work-life balance was being further eroded. Remember that 40% of university teaching staff are on temporary or zero-hours contracts, and that university management is plagued by gender imbalance.

But the definition was probably born more of frustration than disrespect. Finding TEF metrics that actually work has proved trickier than anticipated. Learning gain might (and probably should) be measurable at local levels for individual cohorts of students, but it doesn’t allow the kind of cross-institution and cross-discipline comparisons that the TEF craves. Employability and salary data tell you lots about students’ background characteristics but, as Graham Gibbs notes, they remain hopelessly distant proxies for the quality of teaching they received at university.

images3Elsewhere, the Minister’s speech did offer some optimism for the sector’s future. The goal of increasing by 20% the number of black and minority ethnic students going to university by 2020 is to be applauded. UCAS was ordered to publish more detailed breakdowns of candidates’ background characteristics and application patterns, as the Social Mobility Commission requested some time ago. There were even intimations of a lighter-touch Research Excellence Framework (REF), with welcome acknowledgement that many in the sector want an audit that is “less bureaucratic and burdensome” and which “takes up less of the time that could be spent more fruitfully on research and also, of course, on teaching”.

Keeping the sector on side remains the TEF’s biggest challenge. Mike Hamlyn rightly worries about higher education “being seen as a transactional good, rather than a transformational experience,” while Paul Martin Eve fears that the TEF heralds “a massive coming wave of shake-ups to Higher Education finance, both research and teaching”. Pleas to rebalance teaching and research may seem more reasonable to academics if excellence in the former was acknowledged to rely on excellence in the latter.

TEF-Briefing-August-2015Disappointingly, the student voice is fading from TEF debates, with the NUS executive electing for “principled disengagement” because of threatened links with an inflationary fee rise. This despite the NUS having previously issued an excellent briefing paper on the topic.

The full title of the Minister’s speech referred to “fulfilling our potential”. The challenge ahead is to ensure that “our” embraces the whole of the Higher Education sector, and that “potential” denotes opportunities for it to become more equitable, more pedagogically responsive and more transparent about what it does. Greater care should be taken with language. How exactly is “patchiness” in the student experience being differentiated from learner-appropriate pedagogical diversity? Who exactly is lamenting the “lamentable” teaching?

That “extraordinary teaching deserves greater recognition”, however, is incontrovertible. As is the claim that Higher Education is “the most powerful driver of social mobility we have”. The TEF will soon find friends in the sector if it nudges institutional cultures in this kind of direction.

But equally important is that the TEF skirts avoidable pitfalls, in terms of both policy and rhetoric. Such pitfalls include hierarchy-enshrining outcome indicators, student-alienating associations with a fee hike, and tortuous metrics that reward only the wiliest gamers. The sector may also have had its fill of burger metaphors.

What can the TEF can learn from the REF?

(Note: I published this piece first under the heading “Seven rules the Teaching Excellence Framework should followvia the Guardian’s Higher Education Network on 07.07.15…)

students-in-lecture-hall-010

The proposed TEF (Teaching Excellence Framework) is often characterised as a “REF (Research Excellence Framework) for teaching”. It’s a description that reminds us how much less attention university teaching has received than its showier sibling, university research. But to what extent is it a useful, or even meaningful, comparison? Can measurements of research quality be mapped straightforwardly on to teaching? Has the REF proved so successful at capturing research excellence that it’s now the gold standard for evaluating all university activity? Does HE teaching even need to be REF’d?

indexSuch questions defy simple answers. But if the TEF is the new kid on the block, perhaps there are ways in which it could learn from the REF’s triumphs and mistakes? Below are seven principles that haven’t always been synonymous with the sector’s research audits, but which might help the TEF off to a good start:

1. Be ungameable

Universities, like schools before them, have grown adept at gaming the best possible results from what’s available. The danger is that the game itself becomes the metric, with ‘winning’ institutions not necessarily those offering their students the best possible teaching, but those able to manipulate data, craft narratives and spin results in the shrewdest way. We can’t claim the TEF is for our students’ benefit if the final outcomes comprise an intricate series of metrics, league tables and seemingly contradictory information based on dubious formulae, such as ‘Teaching Power’ or ‘Teaching Intensity’. Of course, no auditing framework can ever be completely ungameable, but the TEF would benefit from a set of guidelines that keep the playing field level and the sector honest.

2. Be Collegial

Good teaching rarely occurs in isolation. Within good university departments, colleagues draw on one another’s strengths and compensate for one another’s weaknesses. Will the TEF place individual lecturers in competition, divide disciplines and create new hierarchies? Or will it facilitate constructive conversations, developmental feedback and candid evaluation?

3. Be long-term

Just as it’s difficult to measure the enduring quality of a research paper for serval years, sometimes decades, after its publication, so too can the value of good teaching not be immediately obvious. University degrees are three or more years in duration to give students the time and space to ruminate, reflect and ripen. Confident, socially-aware graduates are not the product of one bell-and-whistle-filled session (when the peer assessors happens to be around); good lecturers play the long game with their students.

4. Be cheap

The sector can ill afford to be spending millions more pounds measuring itself in increasingly incestuous ways. The TEF should be inexpensive to administer, both in terms of direct costs and the opportunity costs of taking lecturers away from their students.

5. Be inclusive

The expense and agony of deciding who gets entered can be avoided by making inclusion criteria as broad as possible. A TEF score should reflect the teaching strength of a whole discipline; it should not be the product of arbitrary, localised inclusion mechanisms. Some pedagogically-challenged staff would no doubt prefer to hide behind a research grant or admin role, but assigning values to one or two superstar lecturers in a department is of little value. Shouldn’t everyone involved in teaching, from GTAs to departmental heads, be eligible?

6. Be modest

Trying to distinguish between teaching that is “world-leading” and that which is merely “internationally excellent” is unlikely prove fruitful, especially if we’re not asking the opinion of anyone from outside the UK system to benchmark our judgements. Criteria should measure what matters, not just what lends itself to getting measured. The language of the TEF should be realistic and restrained, not lavish and conceited (“innovative, outcome-oriented delivery of outstanding learner experience” should be avoided at all cost!). What students want, broadly speaking, are enthusiastic lecturers who want them to flourish and make time for them accordingly.

7. Be open-minded

Might a TEF reward a safe, conservative approach to teaching over bolder, risk-taking methods? Might lecturers strive for the equivalent of four 3* journal articles rather than gambling on a major, paradigm-shifting pedagogies? Teaching thrives on experimentation, inventiveness and fresh thinking; if the framework encourages boxes to be ticked and learning outcomes to be dryly evidenced, the sector misses an opportunity to learn from its more visionary communicators. Universities are the place for young people to learn independent thinking skills, to become crack problem-solvers and to begin viewing the world through an informed, critical perspective.

Our teaching must reflect that ambition. indexUniversities Minister Jo Johnson, when introducing the TEF last week, quoted one of his predecessors, David Willetts, as saying that “teaching has been by far the weakest aspect of English higher education”. It’s a baseless claim, likely to further disaffect a sector with research long acknowledged to punch well above its weight and academic staff long pressured to excel at everything, always. But with the English HE system among the most expensive in the world, our students have every right to ask why university teaching has lagged behind university research, often relying more on the good will of committed staff rather than secure institutional scaffolding.

Identifying what good research looks and feels like, and then attributing a numeric value to it, has been a challenge for the sector. Teaching will pose no fewer problems. But perhaps by thinking carefully about the principles that we want to underpin a new framework, and learning from those upon which other frameworks sit, we increase the chances of coming up with something that’s genuinely helpful both for ourselves and for our students.

Have UCAS really revealed “the language tricks which will help you land that place at a top university”?

UCAS, the agency responsible for admissions to Higher Education in the UK, last week issued an analysis of the personal statements of 300,000 university applicants. They’d totted up the number of ‘passion-related’ words (such as  “love” and “explore”) and ‘career-oriented words’ (“salary”, “employable”, “job”, etc.) to see how frequencies differed according to the subjects for which candidates were applying.

1434994273-7fda9d77bd9580b55015eaf596aae1b3-600x450

Journalists seemed unsure what to make of the press release. “In all subjects, applicants used a mixture of career and passion-related words to set out their suitability,” reported The Telegraph.  “It’s not a huge surprise,” admitted another report, “that artsy students have cornered the passion market”. One reporter noted aimlessly that “despite the prominence of economics and economists over the last few years, students wanting to major in economics are among those least likely to mention either a ‘career’-related word”.

But none of the reports questioned whether such words are actually valued by admissions tutors. This gave the unfortunate impression that all applicants needed to do was use the right vocabulary and their place at university would be secure. The Mirror even promised to reveal “how to strike the balance between ‘passion and purpose’ to NAIL your written application”.UCAS_personal_stat_3350671a

UCAS, of course, can hardly be blamed for such misreporting. But it’s no secret that admissions tutors are rarely seduced by the language of love:

Dr Hilary Hinds, an admissions tutor from the English department at Lancaster University, finds clichés such as “passionate about literature” and “I’ve loved books for as long as I can remember” dull and predictable. “Demonstrate it rather than claim it,” she says. (The Guardian, 10.07.15)

My own research into the language of personal statements confirms that Dr Hinds isn’t alone. Words like “passion” and “love” are used more by applicants from state schools than those from independent schools, and they correlate negatively with the likelihood of acceptance by higher-prestige universities.

The press interest generated by the UCAS study was vacuous (at best) and specious (at worst). A far better use of the large, rich, not-publicly-available database would have been to identify patterns of use according to factors known to affect candidates’ chances of success, such as ethnicity and socio-economic status.