For this month’s blog post I’ve resisted the temptation to
write something about coronavirus. I think everyone’s hearing quite a bit about
the ‘c-word’ right now, and there’s just a chance that some are getting a
teensy bit fed up with it. So I decided instead to write about something that’s
almost equally feared by those on the receiving end of it; grant reviewers’
feedback. As with certain other things, we know it’s coming sooner or later,
and part of us just wants to get it over with. But at the same time, when it
does finally arrive, we dread the moment when we’ll find out exactly what it’s
like. Will it be mild, or severe?
Let’s pause for a moment and think about the purpose of the
reviewers’ feedback, and of your response to it as the lead applicant on the
proposal. (My focus here is on the UKRI peer-review system, but various other
research-funding bodies operate in a similar way.) Peer reviewers will be asked
to consider the importance, scientific potential and value for money of the
proposed research. In doing so they’ll often raise some questions about points
that aren’t clear to them, or where the detail seems insufficient, and they’ll
make observations based on their perception of how the proposal stands up
against the main evaluation criteria. Sometimes they’ll make suggestions as to
how they think the project could be improved. If they’re in a really bad mood
they might quibble with your punctuation or grammar. Based on their overall
perception, they’ll give the proposal a score. It’s often fairly apparent from
the nature and tone of the comments and the score awarded whether or not a
reviewer thinks the proposal should be funded.
If a proposal receives unanimously poor reviews and low
scores from reviewers, and it’s deemed to be beyond salvage, then it may get
rejected before going to the full panel. This process helps to avoid the
panel’s time being taken up with discussing and considering obvious no-hopers.
Otherwise, the lead applicant gets a chance to provide the panel with their response
to the reviews. This ‘PI response’ is the applicant’s last chance to boost
their proposal’s prospects of success, and for the many proposals that fall
into the ‘danger zone’ of not having garnered unreservedly positive reviews and
maximum scores it’s a vitally important step in the application process (even
proposals with rave reviews aren’t
necessarily home and dry – the panel may consider the reviews to be
unjustified, or may simply feel that in a highly-competitive round the proposal
just isn’t a good strategic fit). All the indications are that a considered,
well-written PI response can be the difference between success and failure.
Know your audience –
who reads your PI response?
At the grants-committee meeting, panel members won’t look at
your proposal in anything like as much detail as the peer reviewers did.
They’ll focus mainly on the reviews, and definitely on the PI response to them,
to inform their discussions and decision making. Two things to bear in mind
here:
- the panellists probably won’t be experts in your particular field – assume they’re not
- they’ll have very little time available to discuss your proposal
So when you write your PI response, your
primary audience will consist of time-pressed generalists who will have a great
deal of business to get through. There’s a highly-important audience
sub-category here too – the introducing or presenting member. Their own field
of expertise is likely to be closer to yours, but not completely
overlapping. They’re likely to have read your proposal in more detail than the
other panel members, but they too will look closely at the reviews and your PI
response. They’ll have to explain your proposal to rest of the panel, and
suggest a score which is then up for discussion. Once again they’ll have a lot
to get through, and not very much time.
A question of tone
So your peer reviews have all landed you don’t quite know
what to think. They didn’t all hate it at least, which means you’re still in
the game. But there are quibbles – some of them a bit left field – and Reviewer
Two really did seem to be having a bad day. How to respond?
Probably the single most important thing to bear in mind at
this stage is that all of the reviewers have given up their time and effort to
review your proposal, in exchange for precisely no recompense. So you’re really
grateful to them. No no, really. Even
if you’re now thinking that Reviewer Two must be a complete coronavirus, you
will take a deep breath, exhale slowly, adjust your face into a beatific smile
and direct only kind-spirited thoughts towards your reviewers. The calm,
Zen-like state into which you ascend will inform the tone of your response. And
if you’re not quite there yet, sleep on it. Life nearly always feels like less
of a trial after a good night’s sleep.
And you never know, some of the reviewers may just have a
point – at least on the face of it. You only had a few pages in the Case for
Support to describe a novel and complex project, and with a lot to try to cover
it’s almost inevitable that some of the details fell by the wayside. What may have
seemed obvious to you may well be much less obvious to the reviewer, who will certainly
know much less about your project and its background. So you now have a chance
to fill in the gaps for them, allay their concerns, and perhaps to acknowledge
and take on board some good points that you hadn’t actually considered. Try not
to see the process as adversarial then, but collaborative – you’re all trying together to make the proposal
as robust as possible before it goes in front of the panel.
What follows is a suggested approach to constructing a PI
response, along with some key points to bear in mind when you write it.
Constructing your
response
First off, I’d recommend completing something of a
content-analysis exercise. Going through each of the reviews, look for common
themes – points and criticisms raised by more than one reviewer, indeed
possibly all of them. Some of these may be easy to spot, some of them a little
harder because the points raised by the respective reviewers may be related but
not identical. Some of the points may be relatively minor and easily addressed,
some may be more serious. You’ll be hoping that none of them are
face-palm-inducing deal breakers, but you’ll certainly be looking out for
criticisms that will kill the proposal if you don’t address them very
convincingly. Rank the points as you find them, with the most serious – and/or
hardest to address – points that have been raised by most or all of the reviewers
at the top of your list. Once you have identified all the common points,
continue working through each of the reviews picking out the points that you’ll
need to address in your response and adding them to your ranked list. Be aware
that it’s quite possible for a very important point to have been spotted by
just one of the reviewers, perhaps because their own field of expertise is
closest to the subject of the proposal.
One of the reasons for trying to rank the reviewers’
comments in terms of importance is that you only get a limited amount of space for your response. So it’s important to prioritise the allocation of space in
your response document so that you can deal most comprehensively with the most
serious points that have been raised. That said, it’s important to try to
address all of the legitimate queries
and criticisms raised by the reviewers.
When you’re sure you’ve listed all the points that
need addressing, share your list with your fellow applicants, along with copies
of the reviews themselves for context. Research proposals are always a team
effort, and your PI response will be strengthened by the input of the other
team members. The tone you adopt for your initial communication with the team
will set the tone for their input, and there’s little to be gained from wasting
time and emotional energy with a flurry of “how could anyone be so stupid!”
emails.
If your proposal’s basically a good’un and the reviewers
have done their jobs well, a fair number of their comments, observations and criticisms
are likely to focus on the finer points of your science and your proposed
methodological approach. Reviewers might also have some concerns about certain
aspects of feasibility (can you really recruit all those participants, or get
it all done in the available time?), or about how you plan to archive your data
and make it available. Perhaps they don’t understand clearly enough exactly who
will be doing what in the project. If there’s anything I’ve learned from
working with grant applicants to draft their PI response, it’s that I definitely
don’t know the answers to questions like these – it’s your project, so it’s
over to you. But what I can try to do is provide a bit of help with the approach
and the structure of the PI response document. What follows is my
two-penn’orth, based on information and insights that I’ve gleaned from those
who actually know what they’re talking about in this respect (including past
and present panel members and the funders themselves).
Some points to keep
in mind
Firstly, just to re-iterate, tone. I have seen panel
feedback on unsuccessful proposals to the effect that “yep, we can see Reviewer Two hasn’t done a great job. But they’ve had a
go and we appreciate that, so we think your stroppiness is unhelpful and
uncalled for.” However good or bad their proposal was, this applicant’s PI
response doesn’t seem to have endeared them to the panel. And that can’t have
helped their cause. So don’t openly
criticise the reviewers, or try to rubbish them. Trust the panel to spot
the bad reviewers for themselves and to take their dodgy reviews with a small helping
of salt. And remember – responsibility for good communication lies with the
communicator (some more thoughts on this here).
If someone has failed to grasp something it’s because you failed to communicate it clearly, not because they’re an idiot
(you might think this privately, but don’t let it infect the tone of your
response).
Politicians regularly infuriate interviewers and the public
by refusing to answer the question, perhaps by blatantly answering a different
question altogether. And it can be a tempting strategy to adopt when responding
to reviews, particularly if some of the comments and questions are awkward to
respond to or there are just so darn many of them. Perhaps if you just quietly
ignore them, the panel won’t notice? The advice here though is always to try to be thorough and
comprehensive in your response, addressing all the material points raised.
Also like politicians, reviewers aren’t always in unanimous
agreement. While one reviewer might feel that your statistical power
calculation is inadequate, for example, another might indicate that your
methodological approach gives them no cause for concern. The response to the
first reviewer though cannot simply be that the second reviewer thinks it’s all
fine, so nah. Don’t waste the panel’s time
by highlighting contrasting views between different reviewers. In this scenario,
respond directly to the first reviewer, explaining clearly why you think your
approach is appropriately powered – or perhaps even taking on board some of
their suggestions. It’s important to be honest and receptive in this respect –
if a reviewer has highlighted a limitation of your proposed methodology, or
suggested an obvious improvement, then it’s probably counter-productive to dig
your heels in and try to defend the indefensible.
When you do respond to a particular point, be specific in what you write and back up
what you say with hard evidence, not just opinion and un-quantifiable
assertions. This is particularly important if you’re rebutting a criticism,
rather than accepting it and taking it on board. Point to peer-reviewed
publications or whatever credible evidence you need to back up your statements.
Avoid just referring back to your proposal – if something
wasn’t clear enough in the proposal then just repeating it or referring to
a particular proposal section is unlikely to wow the panel. Take the opportunity to explain things in a
different way, supplemented by a bit more detail if necessary. ‘Opportunity’
is the keyword here; remember how hard it was to fit everything you wanted to
say into your Case for Support? Now’s your chance to use a few more pages to flesh
out the bones of your proposal, demonstrate your specific knowledge and
expertise, and introduce some new supporting data (you could include a link to
it if it’s been published). There’s perhaps one exception to all this – if the
reviewer has obviously missed something that you did in fact state clearly in the proposal, then tactfully draw the
panel’s attention to the relevant material. Employ the same tact if you're correcting
a clear factual error that the reviewer has made.
You’ll only be given a limited number of pages for your PI
response, so use the space very wisely –
be concise and make every word count. Assuming that you do have some
important points of criticism to address, don’t go filling up the available
space by reproducing all the positive comments. The panel members will have
seen them in the reviews, so there’s not really much point in duplicating them
if space is tight. While it’s nice to seem appreciative, it’s probably not
worth filling up too much of the document with effusive words of thanks.
Similarly, there’s probably no need to reproduce the wording
of each point of criticism before you address it. Instead, I’d suggest simply writing your responses under concise
sub-headings. So for example, if one or more of the reviewers has suggested
that the proposal seems a bit on the over-ambitious side, then you might
address their concerns under a sub-heading like ‘Feasibility and scale of ambition’.
It would probably be helpful to include the ‘reviewer reference’ number/s as
part of each sub-heading, to indicate which reviewer/s have raised that particular
point. A further benefit of using clear and descriptive sub-headings is that it
will help to order your document and make it much easier to read.
Finally, if at all possible ask someone who’s not involved with the proposal to read through your
response before you send it. Ask them specifically if the tone comes across
as measured and professional, and whether they feel you have made any statements
that don’t seem to be backed up by hard evidence.
It’s never felt more apposite to end a post with the words ‘good
luck’.
The views and opinions expressed
in this blog are mine alone and are in no way endorsed by my employer. Factual
information and guidance are provided on a 'best-endeavour' basis and may
become out of date over time. No responsibility can be taken for any action or
inaction taken or not in respect of the content of this blog.
