PLDI and PACM
IntroductionProceedings of the ACM on Programming Languages (PACMPL) is a new open access journal that will publish the proceedings of POPL, ICFP, and OOPSLA. Should PLDI join PACMPL? The purpose of this blog is to explain more about PACMPL, to present the cases for and against PLDI joining the journal, and to provide an online forum for discussion of the decision. There will also be an open discussion at the Town Hall Meeting at PLDI’17 in Barcelona (5pm on Tuesday 20th June). The PLDI Steering Committee is running a survey to take the pulse of the Community on this issue; if you have an interest, please do complete the survey! Based on the information and opinions gathered in the survey, on this blog, and at the Town Hall Meeting, PLDI SC will decide whether PLDI should join PACMPL.
PACMPL was established to publish the proceedings of the top SIGPLAN conferences. The only significant change from the usual conference reviewing process is the introduction of a light second phase, whereby the PC determines any mandatory revisions among the usual suggestions for improvement, and formally checks that these have been carried out; PLDI already has a shepherding process that is essentially equivalent. PACMPL papers will be published with Gold Open Access (OA) from the outset. ICFP’17 will be the first issue of PACMPL; from that point, ICFP, OOPSLA, and POPL will all be published in the journal. There is the opportunity to include PLDI too.
Why PLDI should join PACMPL
- For researchers in some jurisdictions—especially in Eastern Europe and South America—conference publications carry little weight in assessment and promotion exercises, even when top conferences are compared to second-rate journals. PLDI publications are at least as strong and as thoroughly reviewed as good journal publications. Calling them journal publications would allow researchers in this situation to align their scientific incentives (to publish in the best places) with their professional ones (to publish in indexed journals).
- The PLDI Community has already expressed its support in principle for publishing the proceedings of PLDI in a journal. Two surveys in 2014 showed strong support for publishing the proceedings as an issue of TOPLAS: a general survey sent to PLDI’14 attendees was 75% in favour (N=91), and a subsequent targeted survey 70% in favour (N=212). (PLDI SC and SIGPLAN Executive Committee put a proposal to this effect to ACM Publications Board, but it was turned down. It was partly that rejection that led eventually to the new PACM journal series, and to PACMPL in particular.)
- A survey sent to PLDI'15 attendees asked about inclusion in PACM in particular. 234 people answered, with 63% for, 27% against, and 9% unsure or indifferent. Breaking down those in favour by the number of authored PLDI papers results thus: 0 PLDI papers: 81% in favour; 1 PLDI paper: 59% in favour; 2 PLDI papers: 56%; 3 PLDI papers: 51%; 4 PLDI papers: 46%; >= 5 PLDI papers: 44%. The results show that the more PLDI publications you have, the less you support publishing PLDI in PACMPL; the last two groups comprise about 20% of those surveyed. A key question is how to interpret these results. One reading is that the older people in top universities are fine with the status quo, because the problem doesn't affect them, while the younger ones and everyone else are feeling the pain. Yet another view is geographic. Senior PLDI authors are heavily US-based. Most (though not all) of the pain reports of the conference-centered publication model come from the rest of the world.
- SIGPLAN members are strongly in favour of Open Access publication. A January 2017 SIGPLAN poll showed broad support for doing so (65% in favour, N=305). Gold OA in PACMPL will be provided at a reduced fee of $400 per paper for the whole conference (as opposed to $700 per paper individually), and SIGPLAN have agreed to underwrite this cost for the first three years of PACMPL.
- Citations to papers will be of the form “PACMPL, Volume 1, Number PLDI”, thus preserving the individual conference identity—PLDI papers will still be recognizably such in bibliographies and CVs.
- If PLDI were to join PACMPL from 2018, then all four of the top SIGPLAN conferences would be included from the same point in time (namely, everything after July 2017).
- If PLDI waits to join, PLDI authors and readers will miss out on Gold OA in the meantime.
- Publication in PACMPL of a PLDI paper reporting the latest results does not preclude the current option for later publication of a longer paper in a journal such as TOPLAS expanding on those results, or integrating the results from a series of conference papers.
Why PLDI should not join PACMPL
- The premise for this proposal is that the quality of a researcher’s output be judged according to the venue in which the author’s work appears. The fallacy of this approach to research assessment has been thoroughly and systematically addressed by prestigious journals and numerous international research organizations, including DORA, the UK Parliament, the European Association of Science Editors, the American Society for Biology and others. Recommendation 1 of the DORA statement reads: “Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.” The proposal to move to PACMPL is thus based on a deeply flawed assessment model. Furthermore, the fallacy of this premise is gaining wide acceptance, and richer, easily accessible measures of research impact are now widely available.
- It is not certain that commercial journal indexing services such as Clarivate (the new owners of what was formerly Thomson Web of Science) will actually index PACMPL as a journal. If PACMPL is not indexed by Clarivate, then it will not appear in the Journal Citation Reports, which is regarded as the authoritative listing of journal impact factors.
- If PACMPL is recognized as a journal, its capacity to achieve a meaningful impact factor is limited by the number of citations to PACMPL articles from recognized sources, which includes SIGPLAN Notices but excludes conference publications including LNCS (ISSN 0302-9743).
- Consistent with the above, SIGPLAN has for a long time maintained that the binary distinction between journal and conference is orthogonal to the quality and impact of published work. Having won that argument, it would be perverse to abandon it, particularly at a time when others are publicly moving in the opposite direction. If PLDI were to join PACMPL, it would send a clear message that high quality publications appear in journals, while conferences are for second-tier work. This will diminish the perceived value of SIGPLAN’s conferences, many of which routinely publish high impact work, harming a core element of SIGPLAN’s mission. On the other hand, as long as PLDI continues to publish as a conference (alongside ASPLOS and other high impact venues), it sends a strong message that both journals and conferences are valid modes of publication for high impact work.
- Support for publication in TOPLAS does not imply support for publication in PACMPL. For one thing, TOPLAS is an established journal with a strong reputation, whereas PACMPL is new. For a second, there was no proposal that any other conference would have shared TOPLAS with PLDI; whereas PACMPL will be shared with POPL, ICFP, and OOPSLA from the outset, with the possibility of further SIGPLAN conferences such as PPoPP joining later.
- An alternative interpretation of the PLDI’15 survey results is that the senior members of the community have more experience about what works, and are hesitant to change a working formula.
- PLDI is seen by external (to PL) researchers as perhaps the primary PL venue for papers that sit at the intersection of PL and other areas (e.g., networks, security, etc). Having submissions from those communities allows for a healthy interaction between PL and other areas. Merging PLDI into PACMPL together with conferences which are not seen the same way externally, combined with the very different format of PACMPL (e.g., an unusual one column format) vs. the format of conferences which are currently coordinated with PLDI (e.g., ASPLOS) may deter those external researchers from submitting to PLDI (e.g., PLDI deadline is tightly coordinated with ASPLOS notification), thus potentially affecting impact and relevance of PL research overall.
- PLDI need not commit straight away. PLDI could wait for a year, or three, to see how PACMPL works out—in particular, to check that PACMPL is indeed included in journal indices.
- Joining PACMPL is effectively irrevocable; no one knows what the impact will be on promotion and tenure processes of merging multiple conferences together with PLDI into a single journal. (In particular, according to the ACM DL, PLDI averages 32.95 citations/paper. In contrast, POPL averages 26.74 and ICFP averages 8.79. OOPSLA's figures are broken, because the research papers have been mixed up with posters etc.) Keeping PLDI separate eliminates this possible confusion and maintains the well-established PLDI brand.
- The PACMPL proposal does not solve an actual problem faced by PLDI, which is successful: it is currently one of the most prestigious and selective PL conferences. It is not clear that joining PACMPL will help either people who already publish in PLDI or those who theoretically do not due to its non-journal status.
Here again is a link to the survey.