Guidelines and Criteria for Evaluation of Submissions for ACM Publications

Updated 10 February 2022

This document gives policy guidelines for the evaluation of material for ACM publication. The audience is both editors-in-chief and potential authors.

• Categories and Levels of prepublication evaluation
  1. Categories of publications
     
     The following definitions of prepublication evaluation are used to characterize sections or departments of publications, and by extension the publications themselves.
     
     The term "archival" is used to refer to publications that present a form of work expected to stand the test of time, and specifically a form of work reviewed according to criteria intended to assess its correctness. Once a work has been published in archival form, it generally will not be accepted for republication without significant extensions (at least 25% by ACM policy.)
     
     ACM's serial archival publications other than journals (e.g., magazines, newsletters, or serial publications with titles such as "Communications," "Letters," "Notes," or other titles approved by the Publications Board), ACM's conference proceedings, and special sections of ACM's non-archival publications may appropriately designated as formally reviewed or highly edited. In some cases, such publications may also be designated as refereed, though a refereeing process is rare in venues other than journals. The responsibility for judging whether all or part of a publication merits any of the three designations rests with the Publications Board. Publications consisting only of reviewed or unreviewed material are considered "informal" and may fall only nominally within the purview of the Publications Board.
     
     The term "refereed journal" is reserved for serial publications which contain a preponderance of refereed material, and serve an archival function. If articles that were not subject to refereeing are included, they should be distinguished in some fashion, and consideration to keeping the distinction clear should be given in deciding what to include in refereed journals. This designation implies a continuing body of editors and referees, the setting up of which is implicitly authorized by the Publications Board in chartering the publication. All ACM Transactions and Journals, i.e., all titles published by ACM including the term "Journal" or "Transactions on", shall be refereed journals.
      
  2. Five basic levels of prepublication evaluation are distinguished, applicable to all authored (that is, signed) material submitted for publication to ACM. The levels are: refereed, formally reviewed, reviewed, highly edited, and unreviewed.
     
     Unreviewed material is published as submitted, with or without copyediting. The reason for accepting unreviewed material may be procedural, to permit rapid dissemination to technical commentary and the like, or may be in the nature of the material, as in the case of short committee reports or technical reports (where there of course may have been considerable review for the original publication). ACM considers refereed and formally-reviewed material to meet the requirements of rigorous peer-review suitable for archival publications.
     1. Refereeing is generally directed to scholarly material for purposes of ascertaining originality, correctness, novelty, importance, and clarity of exposition. With refereed material, the editor or program chair and the publication assume responsibility for these attributes - in practice, in order to assist the editor or program chair in discharging this responsibility in an environment of technical specialization, the aid of subeditors, program committee members, and referees is enlisted, and they in turn assume responsibility for the integrity of the evaluation On the basis of referee reports, an editor or program chair may ask the author for specific revisions as a condition of responsibility publication. Except in special circumstances refereeing is anonymous, with editor or program chair mediating between author and referee.
     2. Refereed material is subjected to a detailed peer review, following a defined, formal process according to the following standards.
        1. Submitted papers shall be reviewed by at least three qualified, independent reviewers who are asked to assess the originality, correctness, novelty, importance, and clarity of exposition of the paper; and who are asked to provide a written review report that shall be provided to the author.
        2. In cases where reviewer comments suggest that a paper may be acceptable only with revisions, the author(s) shall have an opportunity to submit a revised paper. Minor revisions may be reviewed by an editor, program chair, or designee. Major revisions should be reviewed by independent reviewers, generally to include the reviewers who originally considered the paper not acceptable in its original form. Refereed venues may limit the number of rounds of major revision before a paper is considered as a new submission, but must permit at least a single round of major revision.
        3. Editors and program chairs have the discretion to accept a paper with fewer than three reviews on very rare occasions when it is impossible, after diligent effort, to secure reviews from three expert reviewers (e.g., in cases of very specialized topics)

        4. ACM permits both desk (or bench) rejects and "assisted” desk rejects. Assisted desk rejects are rejections based on the judgment of the editor or subeditor that a paper is either out of scope or so far from acceptable as to make external reviews unnecessary.  Assisted desk rejects may involve getting one outside review to corroborate a subeditor’s judgment. 

           Soliciting fewer than three reviews for clear reject cases is fine, but this is not sufficient to accept an article (except in cases of revisions).  However, in all cases, the final decision is within the discretion of the Editor-in-Chief.

        5. The Publications Board instructs HQ to show the following dates on each paper published in ACM journals:

        • date received
        • date of final revision
        • date of acceptance
     3. Formally reviewed material is subjected to a structured evaluation and critique procedure following a defined process uniformly applied as with refereeing, only without requiring that the venue provide an opportunity for major revisions. Written evaluation by multiple independent expert reviewers is still required. Evaluation for technical accuracy is still required, and the criterion of clarity of exposition may be interpreted as readability by a certain intended audience. Generally included in this category are papers submitted to conferences, where time constraints and the ad hoc nature of the evaluation group makes the designation "refereed" inappropriate.
     4. Reviewed material is subjected to a more informal and not necessarily uniform process of volunteer review, with standards dependent upon the publication and the type of material. The evaluation is generally directed towards obtaining an independent assessment as to the importance of the material and of the methods of exposition, and there may still be requests for revision conveyed to the author. Besides the variable standards and criteria, reviewing differs from refereeing in that there need not be written reports and statements for record, although of course there may be. This category includes opinion pieces, where the accuracy of certain statements may be controversial, and pieces such as the written version of an invited lecture, where an editor cannot ask that what the lecturer said be revised, but may impose general constraints on the extent to which the written version may be an expansion or extension of the presentation.
     5. Highly edited material is professionally edited, usually by paid staff, with primary emphasis on exposition, graphic presentation, and editorial style rather than on content and substance. There is presumably considerable interaction between editor and author during the preparation of such a piece, and the final criterion for acceptability is consensus between the two. Work for hire generally falls in the category.
  3. ACM Policy on Prior Publication
     1. The technical contributions appearing in ACM journals are normally original papers which have not been published elsewhere. Widely disseminated conference proceedings and newsletters are a form of publication, although they are usually only semi-archival and often unrefereed. Publication, or republication, of a previously disseminated work is permitted only if the Editor judges that
        1. the revision contains significant amplification or clarification of the original material, or
        2. there is significant additional benefit to be gained from journal republication. In either case, any prior appearance should be noted on the title page of the paper.
     2. Authors of papers submitted to ACM Conferences and Workshops should be able to anticipate to the extent possible whether publication in conference proceedings will or will not jeopardize future publication in formal ACM publications. As such, we are seeking cooperation in defining in advance, in the Call for Papers, whether or not the conference record will be "widely disseminated"; the appropriate box on the Technical Meeting Request Form should be checked.
  4. Policy on Backlog

     "Issue backlog" is the number of issues after the last issue was mailed, up to and including the issue into which any paper would go that arrives at HQ on the day the backlog is calculated.

Evaluation Criteria for ACM Editors-in Chief

The ACM Publications Board appoints all Editors-in-Chief (EiCs) for all ACM periodicals, e.g., its journals, transactions, and magazines[1]. EiCs are appointed for three-year terms, with the possibility of one three-year renewal or a second three-year renewal in the case of some ACM magazines. At the end of an EiC’s first term the Board evaluates the status of the publication and, if the EiC is willing, will consider reappointment. At that time, the Board asks for a statement from the EiC summarizing the state of the publication, the accomplishments of the preceding three years, and plans for the next term. This document outlines the criteria and metrics that the Board uses when evaluating the performance of its EiCs. Such evaluations occur formally when EiCs are being considered for reappointment. However, the underlying principles also drive consideration of new EiC candidates.

Evaluation Criteria

The overarching goals for ACM EiCs are to produce high quality content consistent with the editorial charter of the publication, and to do so in a manner consistent with the Roles and Responsibilities in ACM Publishing[2]. The following sections provide additional details.

A. Editorial

1. ACM sets high standards of quality for its publications. It is paramount that the EiC uphold these standards in the development and selection of content. The Board tracks acceptance rates, usage, and citation statistics as indirect indicators of quality for its publications.

2. Maintaining and strengthening ACM’s reputation in the scientific and scholarly community is an important goal for ACM. For ACM magazines, visibility, site activity and DL downloads are important measures of success and reputation in the community. While there are no established guidelines for minimum levels of usage, an overall positive trend over the EICs term is the expectation.

3. EiCs are responsible for faithfully executing the editorial charter of their publication. EiCs should have a clear vision of the field and the role of the publication within it. The contents of the publication should be balanced, accurately representing the full spectrum of high quality work ongoing in the field. It should not be a forum for the EiC’s own viewpoint. In the case of ACM magazines, it is customary for the EiC to regularly publish editorials that may highlight that EICs own singular opinion on particular topics or issues.

4. EiCs should be proactive in adapting the publication to changes in the field, working with the Publications Board to adjust the publication’s charter when necessary.

5. The ability of a publication to attract and select the best content is also a function of the quality of its Editorial Board. Associate Editors should have high stature and respect in the community. In this regard, appointment of pre-tenure faculty is discouraged both because of their junior standing in the field and to avoid placing them in a potential conflict of interest situation. The number and technical breadth of the editors should be sufficient to manage the publication’s workload. Finally, the editorial board should reflect the diversity of the research community with respect to geography, gender, and types of institutions represented.

6. The publication should have appropriate and well-defined procedures for refereeing and review, which are fair, unbiased, and faithfully executed.

B. Operations

1. ACM publications appear on a fixed schedule, e.g., quarterly. ACM has a legal responsibility to deliver its publications to subscribers according to the contracted schedule. It is the responsibility of the EiC to ensure delivery of editorial content to ACM HQ in accordance with this schedule. In most cases this means that all material must be at HQ at least 90 days before the first day of the month of publication.

2. EiCs should take appropriate action to deal with backlogs of accepted papers that are either too large or too small. (A two issue backlog of accepted papers is considered "healthy".)

3. EiCs should ensure that manuscripts are given a timely review, and which produces understandable and (whenever possible) actionable feedback. ACM has set a goal that each review cycle for its publications (i.e., the time from submission to each accept/revise/reject decision) take no more than 90 days on average. Data on review cycle time is available from Manuscript Central, and is monitored by the Board.

4. ACM has contracted with Clarivate to provide the Scholar One and Aries to provide the Editorial Manager submission systems for use by its publications. Centralized systems of this type provides the convenience of uniformity and stability for authors, referees, and editors, and is important for facilitating transitions between successive EiCs. In addition, it provides the Publications Board and ACM HQ with the ability to track the status of all publications. EiCs should ensure that these electronic submission systems are used in their operations and that the records maintained within it are accurate. They should make appropriate efforts to learn the system used for their journal (training is available for all EiCs and Associate Editors upon request), so that it is used as effectively as possible. Finally, problems with the system should be promptly reported to ACM HQ.

5. Each ACM publication operates a Web site to provide essential information for its authors, referees, and readers. This information includes the publication's charter, listing of the Editorial Board, information for authors, information for referees, lists of forthcoming papers, and links to the ACM Digital Library. EiCs are responsible for ensuring that the web site is effective and up-to-date. In the case of ACM magazines, the EIC’s responsibilities for maintaining and updating the publication web sites are particularly critical to the success of those publications, as in some cases the web sites and the publications are one and the same.

C. Relationships

1. EiCs should work to ensure that authors have a satisfactory experience in their interactions with the publication. Concerns and complaints of authors should be handled in a timely, courteous, and fair manner.

2. EiCs should work to ensure that referees have a satisfactory experience in their interactions with the publication. Referees should not be abused, i.e., asked for immediate turnaround, asked to evaluate papers which are clearly unsuitable or which are far outside their area of expertise, asked to review multiple papers within a short time span, etc. Referees should be appropriately recognized for their efforts.

3. EiCs should work to handle unusual cases, such as violations of ACM policy during submission and review in a timely, effective, and fair manner, informing the ACM Director of Publications of serious cases. (Note: Violations of policy discovered after publication should always be referred to ACM Headquarters.)

4. EiCs should maintain good relations with appropriate ACM SIGs. Comments of SIGs are solicited when considering EiC appointments.

5. EiCs should seek help and advice, and ensure coordination with other ACM units (e.g., HQ, the Publications Board, other EiCs, the SIGs) as necessary. In particular, timely communication of problems to ACM staff, the Publications Board and the ACM Director of Publications, as appropriate, is essential. Effective handling of submitted papers that fall outside of the journal’s scope is another example. EiCs should forward good papers to other ACM journals. If a pattern of submissions emerges for which there is no natural home within ACM publications, the EiC should consult with the Publications Board to either expand the scope of the journal or start up a new one.

D. Innovations

Finally, ACM particularly values EiCs who are proactive in developing innovations consistent with the overall goals of ACM and its publishing program. Examples of the general nature of such innovations follow.

1. Innovations in generating content and reaching out to new constituencies, such as special issues.

2. Innovative and effective use of electronic media in providing supplemental content for the ACM Digital Library, such as electronic appendices, archived data, software, and multimedia.

3. Innovations that improve operations, providing a more satisfactory experience for authors, referees, and editors.

Suggestions for EiC’s Report to the Board

EiCs are expected to provide a written report to the Board before they are formally considered for reappointment. It would be most helpful to the Board if that report specifically addressed the following points.

• Evaluate current charter or scope statement of the publication: Is it up-to-date? Has the field or the publication drifted regarding their emerging sub-areas that deserve increased emphasis, or are there new topics that the publication should consider serving? What plan is there for adjusting the publication’s area of coverage?
• For each of the evaluation criteria areas detailed above (i.e., editorial, operations, relationships, innovations), please identify the current strengths and weaknesses of the publication. What are your plans for addressing weaknesses?
• What additional support do you need from the Publications Board or ACM staff to do your job better?

An EiC may provide any additional information to the Board. This report will be circulated to the entire Publications Board, as well as to all current ACM EiCs and SIG Board Chairs for their comments. If any information supplied is confidential, it should be identified as such and submitted separately.

 

[1] The term “publication” in this document refers to periodicals of this type.

[2] Roles and Responsibilities in ACM Publishing https://www.acm.org/publications/policies/roles-and-responsibilities