How to Run a Fair Paper Competition

JUDGING THE RESEARCH PAPER COMPETITION FAIRLY
Suggestions from The AEJMC Elected Standing Committee on Research

I. INTRODUCTION
The AEJMC Standing Committee on Research often gets requests from divisions and interest groups for recommendations regarding research paper judging. The Committee makes several assumptions about the desired outcomes of research paper presentations at conferences. These include:

* Research papers should represent good scholarship; “bad” papers should not be accepted

* Research paper programs should be a forum for new work, new ideas, and new approaches. Groups should welcome controversial ideas and approaches.

* Young scholars and students should be encouraged to present their original work.

* Research paper judging and programs should reflect the diversity of the Association.

The Standing Committee on Research believes that group leadership and research paper competition chairs should run paper competitions that not only are fair, but are perceived to be fair. To that end the Committee has several suggestions that we believe will increase perceptions of the fairness of the process. AEJMC groups run their own competitions in accordance with the Council of Divisions’ ”Uniform Paper Call” guidelines. Groups are free to accept or reject any of the suggestions offered here, but research chairs should recognize that these suggestions are the result of many collective years of experience in paper judging across AEJMC groups and are endorsed by the Standing Committee on Research.

II. FAIRNESS IN THE CALL FOR PAPERS
The Council of Divisions’ “Uniform Paper Call” (UPC) provides 15 points about research paper competition procedures and deadlines. The following points are additional suggestions from the Standing Committee on Research:

1. Multiple submissions. Guideline 7 of the UPC requires that papers (a) are not already under review for other national conferences; (b) can only be submitted to one AEJMC group for evaluation, and (c) should not have been presented to other national conferences or published in scholarly or trade journal prior to presentation at the AEJMC conference. In addition, some groups limit the number of submissions to their group by an author and some groups allow submission of papers presented at smaller regional conferences. Any limits of these types should be made clear to presenters.

2. Student papers. Guideline 8 of the UPC suggests that student papers should compete on an equal footing in an open paper competition. Thus, several groups do not hold separate student competitions but instead draw their top student papers from the open paper competition. Groups taking this approach should remind student submitters to add “Graduate Student” to their paper’s title page, unless the paper is co-authored with a faculty member.

3. Presentation requirement. Guideline 9 of the UPC requires at least one author of a faculty paper to attend the conference and present the paper. Authors should be reminded of this requirement in their acceptance letter and be asked to immediately notify the research chair if no faculty author can present so that other quality papers can be programmed.

4. Recognition. The Standing Committee on Research recommends that groups develop some means of recognizing top papers for both faculty and students. Often this recognition represents the top three faculty and student papers from the open competition. If awards are given only through a special competition research chairs should clearly state if papers are eligible for presentation in the open competition if they do not win an award.

III. SELECTING PAPER COMPETITION JUDGES FAIRLY
Selecting paper competition judges is one of the most important tasks of a paper chair. As with paper call, groups vary a great deal in how they select judges, but in the interest of fairness and perceptions of fairness, the division and the competition chair should consider the following guidelines.

1. Ideally, paper judges should have expertise in the method/theory of the papers they are judging. Paper competition chairs should determine judges’ areas of expertise and match judges with papers as much as possible. Keep files for subsequent paper competition chairs so that those judges who are particularly effective can be used again.

2. Have both senior and junior faculty as judges. Generally don’t use graduate students as judges, unless the graduate student has a unique area of expertise necessary to judge a research paper.

3. Put a reasonable limit on the number of papers each judge must read. Generally this should be about 3 to 5 paper per judge.

4. Judges should represent different perspectives in the group, including different universities. Avoid using judges primarily from the competition chair’s school. Make sure the process includes judges from diverse backgrounds.

5. Avoid real (and appearances of) conflicts of interest when selecting judges. Judges should be people who have not submitted a paper to the group’s competition, if at all possible. Judges should be asked to report any conflict of interest such as papers from those with whom they have recently co-authored, mentioned, or worked. Judges should not be assigned to review papers from others in their own institution. A paper competition chair should not submit a paper to his/her own group. If the paper competition chair has a conflict of interest (real or one that others might perceive) for the top paper(s) award competition, other members of the group would make the top paper(s) selection.

IV. RUNNING A FAIR COMPETITION
1. Each paper should be reviewed by at least two judges; preferably three. If there is a great deal of variance in the evaluations, another judge should be sought. If a judge is not performing responsibly or does not understand a manuscript, then the judge should be replaced.

2. Maintain anonymity of the judges.

3. Require reviewers to provide written comments on papers that they judge to be inadequate for presentation.

4. A paper competition chair serves as the final arbiter. It’s particularly important not to reject controversial work automatically. The chair is in a unique position to observe judges’ lack of consensus on controversial scholarship. Thus, the chair should read all papers where there is variance in the judges’ evaluations, and use his/her own judgment as to whether to accept or reject the papers.

5. If numerical scoring is used, be sure it is fair. Most groups use some form of numerical scoring for paper judges. This is not a problem as long as research paper chairs recognize that judges naturally will use different scoring “systems” for good and bad papers and that the quality of the particular set of papers received by a judge may bias the numerical evaluations. Thus, you cannot simply sum the numerical scores and use the summed numbers to make judgments of quality. The best solution is to use standardized scoring. This is not hard to do, and it helps to compensate for a judge’s idiosyncratic scoring tendencies.

6. If the group has a theme, selection should not be biased by the theme. All paper selections should adhere to the same standard of quality.

V. JUDGING STUDENT PAPER COMPETITIONS FAIRLY
1. Groups should decide whether to ignore the student/faculty status of authors for purposes of accepting papers. Some groups keep all the papers together and only make distinctions for awards, while other groups judge student papers as a separate competition.

2. Competition for student papers should be rigorous. In some groups where student papers are judged without reference to status, student papers fare as well or better than papers from regular faculty.

3. Student members deserve a fair opportunity to present scholarly work. Groups should not unfairly restrict the number of student papers accepted. Student papers should not be relegated to an “all-student” session but should be integrated into the groups’ sessions. Although AEJMC provides complimentary conference registration to three student authors in each group, a group need not limit student paper acceptances to three.

4. If a group limits the number of student authors on a paper, then the same rules should apply to the number of faculty authors. Some groups limit the number of student authors to three; this is unfair unless the same limits apply to the number of faculty authors.

VI. COMMUNICATING WITH AUTHORS
1. Authors should receive timely notification of acceptance or rejection of their papers. UPC guideline 10 specifies the date by which authors should be notified. Many schools provide little or nothing in travel funding, so faculty and students have to do considerable budget planning to attend a conference.

2. Acceptance notices should be specific about the nature of the session. Authors should be told how long they have to present their work, how to order audio or video equipment, when to send their paper to the discussant, and who the other presenters will be in the session. An example handout of information for presenters, moderators and discussants is available from AEJMC.

3. The author acceptance letter should include a summary of papers submitted and accepted in the group’s competition. This information helps the author see where his/her paper stands with the others submitted.

VII. THE PAPER SESSION
1. Paper sessions should allow enough time for presenters and discussants. Generally three or four papers per session are the limit if there is a discussant. A sample handout of information for presenters, moderators and discussants is available from AEJMC. Groups should tailor such a handout to their own needs.

2. Discussants should be eminently familiar with the work they are discussing. Choose discussants that are known to be effective oral communicators.

3. Diversity should be reflected by the choice of moderators and discussants. Also, make sure that the choice of discussants is not biased toward one perspective or one school.

4. Research chairs should make sure that promised audio/video equipment is available.

5. Moderators should treat all presenters fairly, including equivalent time limits. It is not fair for those who present at the end of the session to have to give up part of their time because early presenters went over the time limit. Again, information about time limits should be provided to presenters, moderators and discussants in advance.

IX. THE SCHOLAR-TO-SCHOLAR SESSION
1. Groups may employ different criteria in assigning papers to a scholar-to-scholar session (e.g. amenability to display, author request). However, papers presented in a scholar-to-scholar session should conform to the same standards as papers accepted for other sessions (e.g. clearly stated goals, important to field, relevant topic, well-researched and written, good methodology, etc.). Perceived or evaluated quality of a paper should NOT be a criterion, nor should scholar-to-scholar sessions become dumping grounds for the lowest ranked papers of those accepted.

2. Wherever possible, scholar-to-scholar boards should be arranged to diminish noise and facilitate interaction. Sponsors of scholar-to-scholar sessions also are encouraged to arrange them thematically, and groups are encouraged to co-sponsor scholar-to-scholar sessions.

3. Authors of scholar-to-scholar sessions are responsible for being present at the session to discuss their papers. An example handout of an effective scholar-to-scholar display is available from AEJMC.

4. To encourage creativity, groups should, wherever feasible, create faculty and student awards for best visual display. This award should supplement, not replace, other awards based on the quality of the paper’s content.

5. Groups may vary the format of scholar-to-scholar sessions and are encouraged to be creative in the use of formats and displays (e.g. scholar-to-scholar sessions could begin with 2-4 minute presentations by individual authors, or with round-table discussion, and they might end with discussion and critiques, either informally or formally). Groups should obtain feedback from scholar-to-scholar presenters and attendees to improve future scholar-to-scholar sessions.

Guidelines revised August 2010

 Print friendly