Anonymous peer reviews and other workflow features [#114153]

I'd like to start a discussion (at romca's request) of how eJournal could be more useful for academic journals. The timed publishing and issue-management that it provides are enormously useful for more frequent publications but an academic eJournal with only two to four issues a year can do the same thing quite easily manually. What is of much more interest is the ability to have authors submit articles which the editors can then make available to reviewers who make anonymous comments upon which the author can then act.

Technically I can imagine a cooperation with the following modules:
- Actions and Workflow (to manage the workflow)
- CCK - particularly field access (to add fields for reviewer comments)
- Views (for outputting the eJournal)
- Diff (for authors to view comments)
- ACL (for access rights management - it is crucial that eJournal allows other access rights management to be active on the site - particularly Organic Groups)

A cherry on the pie would be a system for entering comments on any portion of the text. My absolute favorite would be adaptation of the system used for commenting on GPL 3.0 - see http://gplv3.fsf.org/comments/gplv3-draft-2.html - which is absolutely mindblowing.

Comments

Comment #1

romca CreditAttribution: romca commented 31 January 2007 at 15:36

Hi,
Thank you for starting the discussion.

What is of much more interest is the ability to have authors submit articles which the editors can then make available to reviewers who make anonymous comments upon which the author can then act.

As I see it, we will need this kind of workflow anyway and should be done. So far, ejournal has something of the kind - authors can insert their articles, which are then automatically visible in the "working space" and not available to public, editors can make changes or suggestions and authors can react and when ready, chief editor can move the article to the issue which is going to be published. Neverthelles, the current workflow is not the best. Namely

- we don't have ability to assign reviewers to articles (and reviewer is imho different from editor)
- we don't have ability to inform authors/reviewers by e-mails (from inside ejournal) - in other words communication between author and editorial board
- and some minor issues like list of all articles submitted by author X (available for editing)

I think ejournal core should take care of the issue-management task and we can move these tasks into the hands of specialized plugins.

Regarding modules:
- diff, we are already using Diff in Ikaros (www.ikaros.cz) and people should be free to use it or not; but the html code is problematic. The best results could be obtained with Diff&Wiki, i.e. some formatting module similar to wiki. there are quite a few journals already which requires articles to be supplied in a specified formattting. And drupal is able to handle different filters. I have tested wiki module, some time ago but the library was still evolving and not ready for production sites. Do we know about another, as-much-flexible-as-possible wiki libraries? Or better, implemented already for Drupal.
- CCK, for sure, but we should not force people to install too many modules and more, we can't prescribe structure of the articles (i.e. the fields). the strategy so far was that drupal and speciliazed modules will take care of the content-handling, all the filters and similar (ejournal is there for journal management)
- Views, for sure ;-)

-Access right managemet: I would like to use your insight in this case. Ejournal offers its own basic access management and together with plugins it could be extended indefinitely. But, of course, it might be better to let the better module to the job

we have 3 roles (four in fact):
1.chief editor
2.editor
3.author
----
4.others

the first three can be general (to all ejournals) or journal-specific (you must remember, that ejournal can host many journals under one umbrella)
There are different situations when these roles are consulted. Can we ask Organic group module to act specifically in all these? Examples:
- viewing ejournal archive
- viewing issue content
- inserting new article(as author, editor, chief editor)
- attempting to delete/change article
- changing issue (moving article between issues)
- editing journal or issue data
- deleting issue/journals

I understand it is crucial that ejournal does not prevent other access management solutions be active.
If somebody knows to some nice examples of integration and cooperation of non-management abd access-management modules, I would be grateful.

Comment #2

techczech CreditAttribution: techczech commented 5 February 2007 at 19:04

Frankly, I know very little about access management. All I know is that Drupal 5 finally makes it possible to manage access for multiple modules and the ACL module makes this much easier. Another module that might be an example of a similar solution is the CCK field permissions module http://drupal.org/project/cck_field_perms. I think the current roles are sufficient but would be much more useful if access rights could be assigned to individual users as well as user groups. And if these rights could vary by field as well as node.

Re OG: I wasn't suggesting that eJournal uses the OG module for its rights management - only that it is mindful of it since it is likely to be installed on many sites that actually also use Organic Groups. And many people don't realize (I didn't) that Organic Groups is actually a node-access module in disguise that uses group membership instead of roles.

Comment #3

kbourne CreditAttribution: kbourne commented 8 March 2007 at 14:34

This module seems to have good potential, but my question to the developer is if you want peer-review to be a part of this (which most academic journals are). In my experience, the peer-reviews are done on a double-blind basis. So a reviewer would not know who the author is and vice versa. And the next part of this, is that it is multi-reviewer, and those reviewers can't see each other's reviews.

So access is a key part of this. In a perfect world, reviewers would see additional fields in a node (where they enter the review), but only on nodes they are assigned and only the parts they are assigned (so they can't see others reviews, even though those are also hidden fields on that node). The functionality that can hide fields in Drupal can only hide them at the role level (so not the individual level or the page/node level), but when you have multiple people at the same role (such as reviewer) that can only see the fields assigned to them and only on the specific nodes that are assigned to them. I don’t see that being possible with Drupal in its current form with any of the core or contributed modules.

Comment #4

romca CreditAttribution: romca commented 11 March 2007 at 20:13

Very good points, thanks. I think that the peer review workflow cannot be implemented on the role level basis, but only on individual user basis.

imho it is a task for e-journal to take care of peer-review. I can imagine these tasks (please correct me if something is missing/abundant)

- make articles anonymous after submission and before publication (actually, it is quite easy: (1) temporarily replace term id for authors with special id so that article is no longer attached to author(s); and (2) replace the user id of author with other defined user id; (3) replace defined configurable fields: store fields which may contain information about authors (e.g. abstract) in a separate table and delete them from the article. They would be automatically reinserted after the article was acceppted/rejected)

- provide functions to control list of reviewers (to have a database of reviewers with contacts and overview of which articles are/were reviewed by them; with mechanism to control the review process)

- reviews should be a separate node types (not fields from the article) attached to the articles through some linking mechanism - in this way, we don't need to take care of hiding them; access control is based on role level basis - only (chief) editor and author of the review is allowed to display the review, reviewers must not know about each other (existing access control modules can be used to control this). What e-journal module needs to provide is linking mechanism between article and review(s)

------------
currently - if the journal is configured to run in peer-review mode:
- articles must go to the working space and there we need to add functions to assign reviewers and control review process
- articles must receive approval (configurable thresholds, usually double blind review = 2) before going to the issue management area
- from issue management area they are published

thanks again,

roman

Comment #5

techczech CreditAttribution: techczech commented 12 March 2007 at 17:40

That sounds very reasonable. I think the conference module tries to do something similar so you may want to check that out.

The only thing that is important that the editor can always see the full author's name.

The other issue, though, is the ease of commenting. It would be nice if the reviewer could somehow insert notes directly into the body of the paper they are reviewing (like they do it on the GPL3 discussion page).

Ideally you would then need the author to be able to make the proposed changes in a way that can be tracked to the suggestions (but that's really just icing on the cake).