Since some time there has been discussions about having a place for community reviews; see for example

(Ingrid Daubechies) Planning for the World Digital Mathematical Library

https://mathoverflow.net/questions/122125/math-annotate-platform

https://meta.mathoverflow.net/questions/3088/community-peer-review-in-mathoverflow

The idea is to have a transparent peer-review process. Everything should work as if it were a standard journal, except for the fact that submissions are linked to papers in arXiv and reviews are public (but reviewers may be anonymous). The main task is to gather a community big enough for the project to work, but specialized enough so that submitted papers have a properly sized audience.

Any comments are welcome!

About OpenReview

OpenReview aims to promote openness in scientific communication, particularly the peer review process, by providing a flexible cloud-based web interface and underlying database API enabling the following:

Open Peer Review: We provide a configurable platform for peer review that generalizes over many subtle gradations of openness, allowing conference organizers, journals, and other “reviewing entities” to configure the specific policy of their choice. We intend to act as a testbed for different policies, to help scientific communities experiment with open scholarship while addressing legitimate concerns regarding confidentiality, attribution, and bias.
Open Publishing: Track submissions, coordinate the efforts of editors, reviewers and authors, and host… Sharded and distributed for speed and reliability.
Open Access: Free access to papers for all, free paper submissions. No fees.
Open Discussion: Hosting of accepted papers, with their reviews, comments. Continued discussion forum associated with the paper post acceptance. Publication venue chairs/editors can control structure of review/comment forms, read/write access, and its timing.
Open Directory: Collection of people, with conflict-of-interest information, including institutions and relations, such as co-authors, co-PIs, co-workers, advisors/advisees, and family connections.
Open Recommendations: Models of scientific topics and expertise. Directory of people includes scientific expertise. Reviewer-paper matching for conferences with thousands of submissions, incorporating expertise, bidding, constraints, and reviewer balancing of various sorts. Paper recommendation to users.
Open API: We provide a simple REST API for accessing and uploading records of people, their groupings, document content, invitations and reviewing assignments, conflict-of-interest designations, and reviewing workflow patterns. You can then write scripts , all with a clear, robust model of read/write permissions. Track submissions, monitor review process, send customized bulk email messages, automate workflow actions.
Open Source: We are committed to open source. Look for our public repo on GitHub soon.

OpenReview.net is created by Andrew McCallum’s Information Extraction and Synthesis Laboratory in the College of Information and Computer Sciences at University of Massachusetts Amherst

OpenReview.net is built over an earlier version described in the paper Open Scholarship and Peer Review: a Time for Experimentation published in the ICML 2013 Peer Review Workshop.

We gratefully acknowledge the support of the OpenReview sponsors: Google, Facebook, NSF, the University of Massachusetts Amherst Center for Data Science, and Center for Intelligent Information Retrieval, as well as the Google Cloud Platform for donating the computing and networking services on which OpenReview.net runs.