Customizable Reviewer Recommendations

[stranack]

Describe the problem you would like to solve
Reviewers are presented with a list of recommendations they can choose from, such as accept, decline, request revisions, etc. This list is not appropriate for all journals. Some would like to change the list to reflect their review practices.

Describe the solution you'd like
Somewhere in the settings where review recommendations can be added, edited or removed.

Who is asking for this feature?
Requests on the forum, PKP's publishing services, and institutional partners.

Additional Information

• It should not be possible to delete reviewer recommendations that are used in exisiting reviews, but it should be possible to decommission them so they cannot be used by future reviewers.

Specs Update - Friday, September 27th, 2024

Workflows Affected by This Change

• Settings > Workflow > Review - Here a new tab will be added here for customisable review recommendations

Detailed Specs

We can add a section in Settings > Workflow > Review > Review Recommendations. This would include a default list of review recommendations that JMs and JEs can easily modify, add to, or activate/deactivate. I left out the option to delete recommendations, as the activate/deactivate feature offers a good workaround while still keeping a record of past recommendations.

Video Recording: https://youtu.be/cYb3U1FRH_0
The Designs can be found here: https://www.figma.com/design/Wf7sDlUg2372jaKKTJ0Mgz/OJS-3.4-3.5?node-id=8862-8486&t=sf2uoFGq6lBMtfCU-4
Prototype: https://www.figma.com/proto/Wf7sDlUg2372jaKKTJ0Mgz/OJS-3.4-3.5?page-id=7200%3A7069&node-id=8862-8487&node-type=frame&viewport=1704%2C-15906%2C0.23&t=mAQY2jdYCKArumzF-1&scaling=min-zoom&content-scaling=fixed&starting-point-node-id=8862%3A8487

UI Development Guidance

Components to be used would be (all can be found in storybook):

• Listing: Table + DropdownActions + Badge (that need new grey variant, if you need help to extend it let me know) + Button
• Form: FieldText + FieldSelect // and don't get confused by the fact it looks different than on designs. This is how our forms will look in future.
• Form should be created on PHP side as we don't have tooling to do it client side.

In general just use the latest stuff - composition API, tailwindCSS.

PRs (draft)
pkp-lib --> #10583
ui-library --> pkp/ui-library#444
reviewReport --> pkp/reviewReport#56
ojs --> pkp/ojs#4505
omp --> pkp/omp#1851
ops --> pkp/ops#883

[NateWr]

This has been requested https://forum.pkp.sfu.ca/t/question-about-review-response-options/56499/2 and according to @ajnyga it is a common request, so I'll put the community priority label on it.

Unfortunately it's not so simple as just letting people add or rename them. The new editorial statistics calculate things like acceptance and rejection rates, and average time from submission to acceptance or rejection. For these statistics, we need to know precisely what a decision means.

For review recommendations we don't yet need this precision. But I can see such a desire for this in the future: for example, wanting to know the average accept/reject recommendations for an accepted submission, the average number of revisions requested, etc.

[ajnyga]

Thanks for the explanation, I was unaware of the editorial stats (but they sound great)

A quick idea here is of course to have "types" of decisions coupled with freely editable label.

Meaning that you can for example create a decision with the type "Request revisions" and the label "Please try again".

The types then would be of course hard coded, but you could have several of the same type in the list of available decisions.

Edit: just checked and did find at least three requests on this from the forum. In our case, out of the 80 journals we have, at least four have asked about this. They all wanted to have "Minor revisions" / "Major revisions" labels instead of the OJS default ones.

[alexxxmendonca]

A quick idea here is of course to have "types" of decisions coupled with freely editable label.

This is how ScholarOne operates indeed.

[asmecher]

Just to highlight something Nate said:

It would be relatively easy to allow customization of the reviewer recommendations. Nothing programmatic depends on them. Adding, removing, modifying, etc. would all be possible without major implications.

Editor decisions are not so simple. When you record an editor decision, it has major impacts on how the workflow proceeds. Re-labeling the decisions would be possible without major impacts, but adding and removing options would need some serious consideration.

[ajnyga]

Usually the request concerns just the reviewer recommendations.

Are there more cases than these when the workflow is affected:

1. Request revisions / resubmit for review

• I have a feeling that both affect the workflow the same way. Meaning that if I do "Resubmit for review", the workflow does not force me to start a new review round and vice versa if I do a "Request revisions".

2. Decline
3. Accept / send to copyediting

[NateWr]

Are there more cases than these when the workflow is affected

In terms of stats, the acceptance/rejection from the submission stage is calculated as desk accept / desk reject. So that would need to be preserved as a distinct type.

As far as I know, the only stats we calculate related to review involve the number of rounds. So the exact difference between "resubmit" and "request revisions" is not calculated.

[ajnyga]

I would get to decide, I would replace the "resubmit/revisions required" lingo with "major/minor" revisions at least in the code. Meaning that there would be two "revisions needed" types that could be counted in the stats if needed. The labels by default could still look like "resubmit for review / revisions required" if we want to preserve that.

Having the types "major/minor revisions" needed in the code better reflects the way things are working. Meaning that "resubmit for review" decision does not necessarely lead to a new review round -> the editor can still just approve the changes and go forward.

---

But, do we need to have the same list of options both for the review recommendations and the editorial decisions if the two are not communicating in the code? AFAIK the recommendation is just a visible label for the editor. Why not allow the journals to edit whatever values they want to have there and deal with the actual editorial decisions separately?

[NateWr]

deal with the actual editorial decisions separately?

They are separate already.

[ajnyga]

Sure I realize that. I just meant that why not start with the issue just by allowing journals to edit the review recommendation options. If we got more far reaching requests later, then we can consider a more complicated solution. With just the recommendations it should not be a big change to the code. But it could be that allowing journals to edit those labels will automatically create a request to be able to edit the editorial decisions labels as well.

[unkej]

Hi @ all,
I just want to add a "like" to this. If journal editors (or admins without getting too deep into the code) are able to edit the reviewer recommendations, it would be very nice.

[amandastevens]

Another +1 for editable reviewer recommendations from a Publishing Services client.

[NateWr]

@amandastevens is that a +1 for editable reviewer recommendations or editorial decisions or both?

[amandastevens]

Oops, didn't realize these were 2 separate things. It was reviewer recommendations - I edited my above comment.

[NateWr]

It seems like the more widespread community priority is reviewer recommendations. I have repurposed this issue to focus on reviewer recommendations and split the request for customizable editorial decisions out into its own issue: #6074.

Before we move forward, I'd like to see a few more details provided to better understand the use-cases. What are some examples of different review recommendations that your journals would like to use?

I'd like to retain some information that can be used for editorial statistics. We will need a way to be able to say things like "how many reviews request revisions" or "how many reviewers accept a piece in round 1 vs round 2". To do this, we'll need to settle on some basic recommendation categories from which one or more recommendations stem.

[unkej]

One of our journals asked for these changes, if this is what your are looking for:
Accept Submission -> Accept (I think this needs no change)
Revisions Required -> Acceptable after minor revisions
Resubmit for Review -> Not acceptable without major revisions
Resubmit Elsewhere -> Reject
Decline Submission -> Reject

[NateWr]

That's great, thanks @unkej!

[amandastevens]

Sometimes editors want new options and sometimes they want to customize the wording on the existing options. The recent request was for this:

• Accept
• Require minor revisions
• Require major revisions and resubmitting
• Reject
• See notes

[NateWr]

Do all the desired options we know of fit into three categories: accept, revise or reject? I'm thinking "see notes" would count as a revise recommendation. Or perhaps we need an "other" category as a catch-all for recommendations that don't fit.

This would allow us to provide editorial statistics about these three categories, while letting journals build whatever recommendation options they want on top. This way we could provide accurate long-term stats about number of reviews that result in accept, revise or reject. But not necessarily provide accurate long-term stats about the number of reviews that go for minor revisions vs major revisions. They'd get lumped together in the stats.

(That's because if the options are editable, they can change over time and therefore can't be used as accurate statistical indicators.)

[touhidurabir]

Hi @asmecher all tests are passing and good to merge . Only OJS upgrade (mysql and pgsql) tests are failing at very end but because of wrong commit points to oaiJats .

But I can not merge as don't have write permission to https://github.com/pkp/reviewReport and all PRs should merge at a time. can you please merge the PRs at #1660 (comment) .