The currency of knowledge: or, the PBRF in ruins. Part 1

I’ll begin with a story of practical details. The micro-politics of the PBRF reached a new level in mid-2009. After the completion of an internal mock-PBRF assessment at one of our esteemed universities, academic staff whose research output over the previous three years was considered to be well below a level likely to gain a B rating received a letter from the Dean raising concerns about their individual research productivity. The matter was framed in terms of the employee’s contractual obligations and the criteria for the PBRF. Recipients of this letter were summoned to a meeting with their Head of School, at which a member of the Human Resources staff would be present. At this meeting, research objectives would be reviewed and performance targets would be set. If satisfactory progress were not made within three months of that meeting, the letter warned, a performance management process would be initiated. Essentially, this was a written warning about poor performance, following which disciplinary proceedings could eventually be implemented. Under such conditions, dismissal is always a possible outcome if performance does not improve as required, although this was not explicitly alluded to in the letter itself.

Now, one could argue about the lawfulness of this action by the employer. But that is not my business today. In any case, mediation between the employer and the union over the letter has been undertaken and that seems to have produced some change. As if intending to make the PBRF look even worse, however, another, equally-esteemed university shortly afterwards announced that it would ‘fine’ colleges, to the tune of $40,000 per year, if their projections under-estimate the numbers of R-rated staff in the 2012 assessment. Heads of departments were, I believe, asked not only to estimate numbers, but also to provide lists of names.

But let’s suppose now that every university in New Zealand had written a warning letter to every PBRF-eligible staff member who had scored an R in 2006. Let’s omit those with an R(NE) who had the excuse of being ‘new and emerging’ researchers at that time. If we include the colleges of education, which either had already merged, or were about to merge, with universities in 2006, the total number of staff under the gun would be about 1,000 (Tertiary Education Commission 2007). That’s about 16 per cent of the academic workforce of universities, who were eligible for PBRF assessment, at that time. And that’s not allowing for the possibility that some who scored C or R(NE) could also come under scrutiny at some stage.

Imagine issuing poor-performance warnings to 16 per cent of the academic workforce, telling them to produce research within three months or face further disciplinary actions. Dismissal is not ruled out.

Taking the letter I have just cited to its logical conclusion, that’s what could have happened. But if all academics were expected to get a B rating, as the letter suggests, then the proportion of us under such managerial disciplinary surveillance and control would rise to over one half. That’s conceivably more than half of the universities’ academic staff at risk of disciplinary proceedings, and even dismissal, for the alleged failure to produce enough research at a presupposed level of quality.

Now, perhaps this is taking the matter to Orwellian, and hence fictional, levels. And so, before I am (again) accused of indulging in a supposed ‘myth of the despotic regime of production’ (Curtis 2007), let me make a few observations about the complex relationship between members of the Academy and the PBRF. First, the PBRF has been designed, developed and applied, if not misapplied, by academics – by us and by our own colleagues. If we think of the PBRF as a virtual intellectual Panopticon, as a set of constraints and a method of surveillance that produce the internalisation of a moral code, then it is a Panopticon that was designed, built and even guarded by its own inmates.

And let’s not overlook the indignant and the swollen egos, the anxiety and the sheer vanity, produced by those A-to-R grades. While some have argued that the PBRF gave them recognition and validation as researchers that their own managers could never find it in themselves to give, others have railed against unfair assessments, bias among panels, or a failure to recognise the activities that they happen to cherish the most.

Did we really have to take it so seriously? After all, the PBRF is basically a mechanism for allocating some public funds around the tertiary education sector. It’s not the gaze of the Master – though sometimes one would think it were, given the importance that some attribute to its results. The desire for a kind of validation given by such an external judgement – even though it has limited validity – allows it to eclipse, for many of us, the intrinsic estimation of our own work. The exaggerated meanings attaching to PBRF results, and how they are thought to reflect upon the reputations and performance of individuals and academic units, are the product, I suggest, of an excess of signification. And so to strip the PBRF of some of this excess, we need to look again at its intended effects.

Now, as the aforementioned letter does quite correctly point out, the PBRF income to the university is intended to recognise and support the allocation of time to research, and also to research-based teaching, especially at post-graduate level. The freedom that we enjoy as academics to devote ourselves to inquiry and experimentation in fields of our own choosing – and to support advanced students in the same – is reflected in this fund. The PBRF designers intended it to respect institutional autonomy and not to interfere with academic freedom. But no-one ever cogently argued, in the conception of the PBRF, just why an external instrument of fiscal policy should be permitted to influence the priorities and the allocation of time and resources within the universities. Nonetheless, we let the Clark government get away with it. While it is widely thought that New Zealand does not invest enough resources in research, no-one had come up with any analysis, to my knowledge, that showed that universities were doing less than they ought. And certainly no-one has stated what the optimal level of research output – at any specified level of ‘quality’ – should be for any particular university. The PBRF has never come with a clear set of goalposts, to put it simply. Taken to its logical conclusion, the optimal arrangement for PBRF purposes would be an academic staff contingent every member of which gets an A. Not only would this be practically impossible, but any attempt to achieve it would jeopardise the development of the future generation of researchers and scholars – not to mention devotion to teaching.

The message of the PBRF seems simply to be that more – or at least the appearance of more – ‘high-quality’ research is desirable. Growth is good. The sky could be the limit, were it not for the fact that another effect of the PBRF is that the universities are now competing madly with one another to retain much the same real level of funding that they got in previous years. As Adams pointed out, in his independent evaluation, the PBRF is creating a Red Queen race – where the contestants madly scramble to stay in the same place – due to little real increment in the overall fund. But compete they must, so as not to lose their share of the money (Adams 2008: 77).

Hence the pressure reaches down to the individual staff members, who are the ‘units of assessment’ and whose individual scores are aggregated, using a published formula, to create the final figure awarded to each institution.

Now, the statutory basis of our freedom to undertake research and critical inquiry is to be found in Section 161 of the Education Act 1989. There are three features of the wording of that section that deserve to be highlighted.

First, the Act provides clearly for the freedom of academic staff and students of the university, as an autonomous institution, ‘to engage in research’. But, secondly, this is a freedom neither to do nothing nor to evade accountability. Subsection 3 speaks clearly of ‘the highest ethical standards’, of ‘public scrutiny’, and of the need to maintain accountability for the use of resources. That is perfectly reasonable, and we do expect one another to maintain such standards. Indeed, thirdly, subsection 4 requires that, among others, the Councils and Vice-Chancellors ‘shall act in all respects so as to give effect to the intention of Parliament as expressed in this section’. That is, academic freedom is described in the Act as a freedom to undertake research and critical inquiry, provided certain standards are met; and the governors and managers of the university are bound by this, and must (at least) respect this freedom at all times.

So, there is a dilemma here in the expression of our values as academics. On one hand, especially in the humanities, we ideally came to this institution in order to pursue the highest goals as researchers and scholars, and this informs and supports our teaching as well. And so those goals are intrinsic to our careers. Some, especially those from former colleges of education that have been taken over by universities, may feel they have not had such a choice, I admit. But the freedom to engage in research and the quality of the university as ‘critic and conscience’ are clearly defined in the Education Act, as well as in long-standing traditions of the university. And we largely accept those conditions willingly. I cannot defend those who may accept the privilege of academic freedom but who will not use it productively. There have always been such individuals, and there have always been lawful processes for dealing with poor performance by an employee.

What I find troubling about managerial interventions such as that referred to at the beginning of this talk, on the other hand, is that the PBRF creates an environment wherein an external governmental fiscal instrument becomes the rationale for an application of managerial authority, including a disciplinary threat, imposed upon academic staff in order to produce more research, so as to produce more income for the university. This appears to contravene the Education Act’s requirement that the governance and management of the university should give effect at all times to the intent of the academic freedom provisions. Managerial coercion does not give effect to academic freedom, in short. So, how did we get to this? Have the government and the universities been taken in by Rousseau’s idea that some people must be forced to be free? Or is it the exercise of sheer power that satisfies them?

In spite of some critics, I would not put this down simply to some kind of insidious force that one might conveniently label ‘managerialism’ or ‘economic rationalism’ or anything vague like that. The zealous (if not over-zealous) application of the PBRF, and its extension into many aspects of the life of the university and the consciousness of academics, are effects for which we ought to accept some ownership, if we are to be honest with ourselves, and not to project them onto some ill-defined ‘other’.

The PBRF was designed by academics, the TEC consults academics about any redesign issues, and the implementation processes within universities are managed largely by senior academics. These are our colleagues, and, if we are serious about maintaining at least a semblance of an academic community and of collegiality, then we need to accept that the PBRF belongs to us, and that we produce its effects. Furthermore, of those among us who have actually been assessed by the PBRF, how many have not felt some kind of keen anticipation, or apprehension, about the results? Did you feel like a student waiting for final grades? I’m sure there were some – a few – who were completely and sublimely unconcerned. But my point is that, for better or worse, the PBRF and its results clearly mean a lot to most academics – not just intellectually, but emotionally too. It got under our skin. And we need to take ownership of it now that it’s there.

The PBRF has become a source of division and aggravation within the academic community. It has divided opinion between members of different universities and different disciplines, between those who gain and those who don’t, and between academic leaders and their more junior staff; and it has had the unfortunate effect of becoming the source of employment relations problems. It may not be long before the PBRF is the cause of litigation.

Further, the PBRF has encouraged the inversion of the very values that have supported our culture of research and scholarship. In 2003, I was quite well disposed towards the PBRF, but my views rapidly began to change when I heard colleagues say things like: ‘If I publish this there, it will look good on my PBRF portfolio.’ A fairly innocuous statement, but, to me, it implied that the funding of universities had become the reason, rather than the support, for what we do. This thinking becomes an institutionalised constraint when universities coerce staff into doing more research on the rationale that the university must maximize its income. Research has thus become a way of augmenting the institution’s income, more than income being deployed to promote research. Research ought to be valued for its advancement of human knowledge and its social and economic benefits, but the cart has been placed before the horse, I fear.

I also know that there are bright people who are early in their careers and considering a permanent role in the university, and older academics at the point of considering retirement, for whom the PBRF has become one of the main disincentives for remaining in the Academy. That potentially translates into real losses of talent, but this will never be quantifiable.

For many academics, including many very talented researchers and scholars, the PBRF is the most detested aspect of the new ‘university in ruins’ (Readings 1996), this university shaped by auditing and by performativity. For others, the PBRF may, simply and brutally, be the reason why they get the sack.

While the universities need the funding, the assessment system that is used to allocate that funding has become, in my opinion, part of the problem. The present assessment system is fundamentally flawed and I’ll briefly list what I believe those flaws are:

• Undue pressure to produce results assessable in the PBRF discourages the next generation of researchers and scholars who are currently contemplating academic careers. People busy doing doctoral or post-doctoral research are told by their seniors that they are ‘research-inactive’. That can be very demoralizing.

• The PBRF discourages the application of existing knowledge to social and economic problems, including knowledge dissemination in the classroom and beyond. Who, for example, is keen to write a textbook now that the PBRF is in force?

• There has been manipulation by institutions of eligibility criteria, resulting in interference in employment agreements.

• The assessment process is costly in terms of people’s time, and it creates a lot of administrative costs for institutions and for the TEC.

Imagine, furthermore, if I were to plan a research project that would involve evaluating an organization by collecting evidence from all of its employees about their personal performance. That evidence is then to be evaluated and each employee given a rating, and the score for every employee is then to be passed on to the employer to use for internal purposes, including performance management, and also to be passed on to the individual employee. The latter party will have no right to challenge the personal score, even if there is good reason to believe it does not accurately reflect performance. And if any employee refuses to take part in the evaluation, he or she may face disciplinary proceedings, so full participation is guaranteed. Such a research proposal would be treated as an outrage and immediately thrown out by any research ethics committee. And yet, the TEC and university managers consider that they have the right and the need to ride roughshod over those very ethical standards that we, as researchers, are strictly required, for good reason, to abide by. The freedom to engage in research has a statutory basis in the Education Act, and the Act also requires the academic community to abide by ‘the highest ethical standards’ at all times while exercising that freedom. When we, the researchers, are evaluated personally for the quality of our research, why is it that the TEC and our own academic leaders regard themselves as free to abandon such ethical standards? I struggle to find any justification for this. Confronted with this ethical problem, the only reply I have heard from university managers is to resort to the imperative of ‘employment obligations’ – again, an appeal to sheer coercion, as distinct from ethical, consensual conduct.

I am afraid, though, that the likely alternative assessment model – using bibliometric data – could be a whole lot worse, depending on how it is designed. People are already cross-validating PBRF results with publications and citations data from Thomson Reuters (Smart and Weusten 2007; Smart 2009) and even from Google Scholar (Smith 2008). I do not recommend that we condone any such research-quality assessment, as the shortcomings of these methods are well documented (Evidence Ltd 2007). All such governmental assessments are likely to be corrosive of academic freedom, and we should comment critically upon them, as we see fit. If I sensed that there were sufficient disaffection and courage among academics to do so, I would welcome direct resistance to leave the PBRF in ruins. After all, the whole system stands or falls because of us. If we really wanted to end it, we would have by now. But I do notice that many of my colleagues are committed strongly to the PBRF; and even many of those who complain about it are, I suspect, paradoxically in thrall to it.

The really critical issue currently in the redesign of the PBRF for 2012 concerns the reporting of individuals’ scores to the employers. In my opinion, this is in breach of the Privacy Act, as, given the past practice, the employer makes no effort to ensure that each researcher has given informed consent to the disclosure of personal information; and this is the issue that could leave the PBRF in ruins, if it’s not resolved. It is the reporting of individual results directly to universities that makes the system useful, as a management tool, from the universities’ point of view – but it is also the means by which the assessment system can be abused by the universities as employers.

The reporting of individual results, without informed consent and without effective rights of review and appeal, has brought the PBRF into disrepute with the very researchers upon whom the assessment depends. A way of resolving this problem for 2012 will hopefully be reached soon. If it were not for the mismatch between the unit of assessment (the individual) and the unit rewarded (the university), however, the system might not be in such disrepute as it is now.