Unclear if default logic by which reviewer list is ordered by in reviewer search is as intended

[matt-graham]

A colleague @paddyroddy and I noticed that the ordering of the list of reviewers presented at the URL https://reviewers.joss.theoj.org/reviewers which is linked to (redirecting from https://reviewers.joss.theoj.org/lookup) from JOSS pre-reviews seems to have slightly surprising behaviour. I originally noticed that I appeared to consistently come at the top of the list when searching by various fields, which I initially assumed was possibly just some logic making your own profile always appear first. After @paddyroddy logged in to the reviewer search system he then ended up top of the list which was consistent with this, but we also noticed my account details were still consistently appearing second and then when I logged in again, I was appearing at the top of the list for @paddyroddy's searches. This seemed to suggest something like your time of last login to the system was being used in determining the search ordering.

For the previous spreadsheet used by JOSS to hold the list of potential reviewers, I believe the recommendation to start looking for reviewers from the bottom of the list was based on trying to spread the load across reviewers by biasing requests to reviewers to those who had less recently reviewed. Based on our experience above it's not clear if the ordering logic in the new system reproduces this so we thought we'd flag this in case this was non-intentional.

From digging through the source a bit, it looks like the ordering of the results is controlled by the function in

reviewers/app/controllers/reviewers_controller.rb

Lines 67 to 76 in 2ca9871

	def order_by
	order_direction = params[:direction].presence == "asc" ? :asc : :desc
	if params[:sort].presence == "load"
	"stats.active_reviews #{order_direction.to_s.upcase}, users.feedback_score_last_3 DESC, users.updated_at DESC, users.github ASC"
	elsif params[:sort].presence == "score"
	{ feedback_score_last_3: order_direction, feedback_score: order_direction, feedbacks_count: order_direction, updated_at: order_direction }
	else
	{ updated_at: :desc, github: :asc }
	end
	end

The default ordering without any sort query parameter seems to be by updated_at (descending) and github (ascending) I guess corresponding to a last updated date-time and Github username string?

It looks like the results at https://reviewers.joss.theoj.org/reviewers?sort=load&direction=asc may correspond to something more like what was represented by the previous spreadsheet as I noticed that the last reviewer to appear in the list under this ordering is isdanni who appears to have four active reviews at the moment which seems to be more than any of the other reviewers appearing towards the end of the list. So a possible resolution would be to use a link with those query parameters in the pre-review issue template, or changing the default ordering? As a sidenote, I noticed that swapping between direction=asc and direction=desc doesn't directly reverse the ordering which I guess is as the direction parameter only appears to control the ordering for a subset of keys - not sure if this was intentional but I found it a bit surprising at first.

Apologies if this is the wrong place to post this issue!

[xuanxu]

Hi @matt-graham, with the new reviewers system in place the old recommendation to start looking for reviewers from the bottom of the list does not apply anymore. The order of people you see at https://reviewers.joss.theoj.org/reviewers is just based on the last time the reviewer's info has been updated, but that initial list includes reviewers for all kind of areas so it is not very useful.

Once filtered by keyword, area of expertise or language authors get a list where they can find good candidates to review their paper, the list is also ordered by updated_at to add some randomness and not be same everytime. But load should not be a worry for authors searching for reviewers to recommend in the review, load information is only visible to editors so they will take that into account when selecting or inviting reviewers.

[matt-graham]

Hi @xuanxu thanks for the quick response.

Once filtered by keyword, area of expertise or language authors get a list where they can find good candidates to review their paper, the list is also ordered by updated_at to add some randomness and not be same everytime. But load should not be a worry for authors searching for reviewers to recommend in the review, load information is only visible to editors so they will take that into account when selecting or inviting reviewers.

Not having authors need to consider load as a factor when searching for reviewers makes sense, but I'm not sure that having the default sort criteria be by updated_at is a good way to introduce randomness as there may be factors which lead to correlations between reviewers logging in to system and/or updating their profile (it's also not clear to me if you need to update profile for update_at to change, as I think initially appeared at top of list without having updated profile). For example, we initially encountered this as @paddyroddy had given a talk to our team about JOSS, which spurred several of us to log in to reviewer system, and at the time of me posting this myself, @paddyroddy and another colleague samcuncliffe, appear at the top of reviewer list. As we also have somewhat similar programming / research backgrounds we also still all appear together at top even when filtering to reviewers with Python and Physics and Engineering under their programming language and areas of expertise fields, so potentially we could end up being requested to review the same submission which is probably non ideal from an ensuring diversity of reviewers perspective.