# The Care-ful Reviewer: Peer Review as if People Mattered Maxwell, Canadian Institute for Studies in Publishing Simon Fraser University jmax@sfu.ca Hi, everyone, I'm John Maxwell, from the Publishing Studies at Simon Fraser University. I'm also the PUBLISHER OF POP! JOURNAL, where INKE's proceedings have appeared over the past few years. I'm in Vancouver, on the traditional and unceded territory of the Squamish, Musqueam, and Tsleil-waututh people. I've been in a deep dive the past six months, thinking about PEER REVIEW -- in the context of Open Social Scholarship in the humanities. I've read a LOT of recent literature on peer review, and I'm impressed with what a mess it is. The figure that I've ended up with is that peer review is a "black box," if you will. Alternatively, it's a ritual (a hazing ritual even), AS A TALISMAN, or a seal of approval A talisman against spectres of: - subjectivity (on personal level) - echo chambers (more collectively) - amateurism (requiring a seal of approval or professionalism) - idiosyncracy (esp in the sciences re: reproducability) - biases of various kinds (esp re: blind and double-blind review - "bad actors" of various kinds (currently in vogue: predatory journals) Double-blind in particular seems to be a hedge against the spectre of a particular kind of prejudiced bias. Pontile & Torny 2014: > The technical simplicity of anonymization via the detachable cover page gave way to the ease of identifying authors by using search engines to spot their working papers, thus making the double blind technically unachievable. Suspicion thus shifted from the assumed bias of reviewers towards possible conflicts of interest for authors. It is a mark of distinction for publishers, especially where justifying funding is concerned. Its role is almost purely ritual in this sense. At least in the humanities. The sciences' quest for "reproducablity" has a much clearer agenda... although that certainly can be questioned too. Does PR actually provide a way around these imagined and real pitfalls? Maybe? arguably? The variety of things that DBPR supposedly wards off is the interesting thing here. There is perhaps most broadly a sense that peer review has a role in ensuring "quality." There is the idea that peer review has a role in ENSURING QUALITY - that peer review is the heart of judgments of quality. > …faculty most commonly define high quality academic journals based on the review process, referring to the perceived rigor of the process of evaluating, gatekeeping and editing academic articles for the journal (e.g., “rigorous review and selection of articles to publish”) – Morales, McKiernan, Niles, Schimanski, & Alperin, 2021. And yet, people have no idea how review actually works at any given journal, beyond the black box itself. > There is a lack of consensus about what peer review is, what it is for and what differentiates a ‘good’ review from a ‘bad’ review, or how to even begin to define review ‘quality’ [82]. This sort of lack of clarity can lead to all sorts of confusion among discussions, policies and practices. (Tennant & Ross-Hillauer) DOES PEER REVIEW EXIST to prevent bad things? Or to nurture good things? Or to save things from utter obscurity? Are we filtering? gatekeeping? Are we curating? editing? Are we nuturing? developing? Are these the only options? Does this, as Tennant * Ross-Hellauer ask, orient us to a "judicial" role rather than a critical examination? Are reviewers judges, jurors, or independent assessors? > typically, by focusing on a binary outcome for articles (i.e. reject or accept), editorial peer review has become more of a judicial role than a critical examination [45], as the focus becomes more about the decision rather than the process leading to that decision. On reviewers: > Often, there is a lack of distinction between the referee as a judge, juror and independent assessor. On curation: what is the relationship between the issue format and pr? Is there? In a "special issue" do we put that to the reviewers? How does that compare with monograph reviewing and series? Work in Transparency topic here. It probably can't, practically speaking, be all of those things at the same time And yet, it kind of is, because our ideas about what peer review is for are myriad, and all tangled up with one another. Or at least so my reading of the literature tells me. What are the actual goals of peer review? What does "rejection" actually mean, apart from "no"? What does revise and resubmit actually mean? What does acceptance actually mean? Rejection means... what? Inappropriate? Bad? Not x enough? Revise and resubmit means... what? Major flaws but we still love you? Minor flaws that we take seriously? Can't deal with it now maybe next month? Acceptance means... what? We approve of your paper? We guarantee the 'quality' of your paper? We want your paper in OUR journal? We think this fits with the package we're putting together? What does peer review cost? If you ask publishers -- and especially the big industrial players -- they will tell you it costs a shocking amount. And of course, if you all think about the unpaid time and service you've offered, you also know that peer review is, as the old Mastercard commercials would say, "priceless" We know, from doing journals, that it isn't actually expensive at all, especially if socially contextualized. (recent thing that said pr was worth 1.8Bn or whatever... this is wrongheaded commoditization) What is the relationship between PR and accountability. Accountability to whom, for what? On Blindness Why are we blind? Are we biased? Are we blind because of scale or despite it? Because scale is an aspiration, but we know we're actually small enough that we won't be able to be objective? Objectivity as a figure. How does blind review provide accountability? Is it a metonymic kind of relation? To what extent do reviewers represent the larger community? And which community, exactly? If we shift out of DBPR, how does accountability shift? What does relabeling do? If we call it "fully closed" instead of "double blind"? from Pontine & Torny: Blind review dates to 50s, 60s, only admin roles knew who wrote what, not reviweres. This became widespread in 70s, partly due to women's advocacy. BUt this was far from a unanimously agreed good. Many appealed to the crucial link between text and author. anonymous reviewers -- a different (not too) set of concerns -- but largely around perceived accountabiity. spectre of malice is reviewer anonymity necessary for possibility of rejection? RS suggestion: single blind collaborative review, where reviewers can at least hide behind the group, to avoid jr scholars being vulnerable in critiquing elders. transition between goals of blind vs who are peers. This is about accountability in some sense. At least accountability to each other? KFitz 2021 talks about editors as middlemen managing all the secrecy, and indeed interpreting the findings back to the authors (and vice versa) on political and power dynamics in anonymous review and when the anonymity goes away. WHO ARE PEERS, EXACTLY? In one version of this story, the peers are identifiable with Shapin & Shaffer's "MODEST WITNESS" figure. where "objectivity" is achieved collectively in public Objectivity (and prospect of acheiving NPoV collectively, avant la wiki) Science in public -- although if you scratch the surface it's not nearly as simple as that. More practically... ...given that we work in institutions, are our peers also institutional? ...given that we also work within disciplines... are peers within our discipline? One of the nice things about the INKE community is that it is not strictly disciplinary; rather it's cross-disciplinary. What does that mean for peers? What is this contested space that Neylon talks about? Neylon 2018 instead poses peer review as an "arena of productive conflict", a game with known rules. Now, appeals to Boyle's air pump and even the word "peer" tends to make us think of peer review as an ancient scholarly practice, but it isn't. PR is shockingly recent... emerged in the late 20th c as part of concerns aout grant-funding, the evolution of learned societies as capital, and the concurrent rise of scholarly publishing's prestige economy. A very gatekeeper/exclusivity-based way of looking at it, and not (as Aileen Fyfe suggests) very much in line with the service-oriented way that faculty members tend to think about it. Fyfe et al p12 > The term ‘peer review’ came to public prominence in debates over grant funding in the USA in the 1970s (Baldwin, 2017), and has since been extended to cover a variety of processes by which academics formally evaluate each other’s work In the context of publishing, it is often assumed to validate new research findings, but it can also be a means of informing the allocation of limited resources, and it is widely seen as a form of accreditation that transforms a research output into a token within the academic prestige economy > For [learned societies], referring papers to suitably-qualified members of the society or university for close scrutiny before publication was part of an editorial system that was intended to emphasise collective rather than individual responsibility, as well as to decide on the appropriate use of institutional resources (Moxham & Fyfe, forthcoming) > The co-option of peer review by profit-oriented publishers now sits in tension with the perception of individual academics, who (despite complaints about the rising burden) remain largely committed to the traditional vision of refereeing and editorial work as a voluntary service to the wider academic, or disciplinary, community Moxham & Fyfe 2018: In the 20th century.... > The epistemic purpose of refereeing also underwent a transformation, from a public foil to set off and amplify the very best of the research received by the Society in the early nineteenth century to an instrument for ensuring the application of minimum thresholds of quality across the board while allocating space (and therefore resources and prestige) on the basis of expert assessment. > The adoption of peer review by a wide variety of humanistic and social science disciplines reveals both the long-standing (if contested) envy of the epistemic rigour apparently associated with the natural sciences, and the professionalizing desire to adopt what has come to be seen as ‘proper’ academic practice. Again the circular legitimization logic that institutions thrive on Which is how peer review has become standardized and its rationale flattened - where boundary-policing mechanisms have EVOLVED INTO AN ATTEMPT TO FILTER through the volume of stuff created by a lot of researchers And where that filtering function serves to maintain the scarcity required to maintain a prestige economy in publications. (Tennent) > The economic motivations for continuing to impose selectivity in a digital environment, and applying peer review as a mechanism for this, have received limited attention or questioning, and are often simply regarded as how things are done. Use of selectivity is now often attributed to quality control, but may be more about building the brand and the demand from specific publishers or venues. – Tennent “A multi-disciplinary perspective on emergent and future innovations in peer review.” 2017 And our credentialling has come to mean this, especially critically impacting and incentivizing early-career scholars. Right? Everyone understand this on a first-hand basis, I'm sure. Is it actually incentives, or is it spectres? Are these different, even? But the image that emerges is a neoliberal field where everyone is an independent actor, all seeking recognition. It does not FOREGROUND A COMMUNITY OF INQUIRY, nor even collaboration. There's an assumption of scale, and relations between strangers, of unsolicited contextless submissions, and frictionless transactional process. a market paradigm, where academic darwinism prevails, and we talk about "incentives" So the "MODEST WITNESS" FIGURE is accompanied by the nameless, interchangeable researcher, an automated editor, and the aggregate expectation of fairness, over a large enough interval Peer review for Open Social Scholarship in the humanities SO, I WENT BACK AND READ CARE ETHICS, which I hadn't looked at since grad school Care, which in Fischer and Tronto's classic definition, is "everything we do to maintain, contain, and repair our world, so that we can live in it as well as possible" Reading Tronto, I am unsatisfied by her attempt to define care, either as an alternative (gendered or otherwise) to traditional western justice-ethics or as a positive *thing* unto itself. The only point that sticks is her insistence that care is rooted in practice, but that's as far as that goes. Tronto's framing, in Chapter 5, of care based on *in*equality -- as a responsibility of the strong to the weak (to oversimplify) -- strikes me as simply wrong. Do we not have an ethic of care to our equals? To our heros, parents, leaders even? And, more importantly for my project, and ethic of care to work itself, and not merely to the subjects or objects thereof. Indeed, what of care for nonhumans? To reduce this to a hierarchy is nonsense. Where do loyalty and trust fit in to care ethics. Haraway certainly talks about trust, when she's talking about agility trial training. But I wonder if 'loyalty' and 'trust' aren't so common as keywords in the largely feminist literature around care. Need to watch for those. For instance, Tronto's positioning of care in the relation of the strong to the weak perhaps has room for loyalty, but the word "trust" appears only in her discussion of Kohlberg, and seems out of place in a relation so powerfully prefigured by power dynamics. Can an ethic of care help us build the kind of capacities we need in the humanities right now? Can it help us integrate the macro- and micro-attention that humanities work requires in the digital age? – Bethany Nowviskie, “On capacity and care.” 2015 And in applying care ethics to peer review, my approach is to focus on relations. WHAT IS RELATIONAL IN PEER REVIEW? - Nurturing the work? - Nurturing the author? - Curating work for readers - Network-building for communities of practice? - Enabling cross-disciplinary conversations? Rebecca Kennison noted that the aim of peer review should be engagement rather than judgement. (2016) > One of the concerns of the anonymous peer reviewing process is that a reviewer can lose sight of the ‘care’ and ‘trust’ aspects of the role, thereby overlooking the duty to act in the best interests of an author and of the scholarly field. (Jackson et al 2018) > > As Heidegger shows: ‘Questioning builds a way... .The way is one of thinking’ (1999b, p. 311). So through peer review as a propaedeutic process, a questioning way of thinking becomes apparent, as a scholarly community ‘gathers’ around a given field of inquiry. (Jackson et al) SO, THINKING MORE PRACTICALLY... Care-ful peer review for Pop‽ So, bringing this down to the level of practical movement. As always, we can look to Kathleen Fitzpatrick for inspiration... > But all of this, as I hope you hear, is not about our publications, or about our publishing systems, but about us — about how we relate to one another, about how we engage with one another as we discuss our work. And thus all of it is within our power to improve — especially if we act as a community of practice, with an emphasis on community. – Kathleen Fitzpatrick, Generous Thinking (2019) Kathleen Fitzpatrick's invocation here of "communities of practice" brought to mind one of the BEST BOOKS I EVER READ, way back in the 90s... The important part there is the subtitle. It's awkward, but "legitmate peripheral participation" is the critical idea. LAVE & WENGER'S INSIGHT is that communities of practice are not populated by static entities. Rather, practitioners in communities are on their way to something, they are becoming something. IT'S NOT HARD to think of this in practice: - undergraduate students on their way to being grad students - grad students on their way to being people with projects - grad students on their way to being faculty (yikes!) - jr faculty on their way to being authorities - mid-level faculty on their way to becoming mentors - sr faculty on their way to making room for others - and lots more ways of becoming It's easy enough to think of boundary policing as a function of peer reivew. But how can peer review nurture, and curate -- and blur and perforate boundaries, if these are the dynamics that we acknowledge, if not actually care about in Open Social Scholarship. Rather than agents in field, HOW ABOUT A GARDEN This is actually the way INKE used to do peer-review, back in the before-times: where, in a lovely garden, we would gather, sit *in pairs* in conversation, and with a suitable nod at least to the ideals of anonymous review, we would consider each article and where it was going. This was good. I pose that in contrast to what I daresay is the norm now, where an automated to-do list emails you about your review deadline... mediating between an editor and a pool of possible reviewers... this is a different figure. In the university, we are always in a glut of messages and communications ...some of is interesting, some isn't, but it always exceeds our ability to truly pay attention. The condition of being suspended in overlapping circles of discourse, and the incompleteness of our participation as a result, calls for some mediating structures. Peer review is such a mediator... it helps connect the scholarly record to credentialing... But a more active figure of peer review has it playing a role both in the nurturing of our works (not to mention nurturing the authors) but in helping zero in on an audience which may interpret them deeply and meaningfully. Don't take audience here in a bald way Traditionally (print culture), the latter is seen in a strictly disciplinary perspective, as it has been chained to the former mediating function (career progress). ROAC and Scholar-led movements' refusal to abstract away the process of publishing, rather siezing the means and the purpose themselves. It means thus acknowledging the infrastructural and processual facets of scholarly work and admitting our position of compromise to them. So, acknowledging scale and scope and reach, and not playing the Internet-capitalist game of infinite reach and infinite scope, but rather recognizing that running at a small scale is going to cost more, Scholar-led says, yes, but it's worth it. It is an acknowledgment of entanglement (neeed a good Haraway quote here) and a commitment to getting on with living well in the face of it. If we imagine open scholarship as having broader audiences, then peer review is an essential mediator, beyond what a journal editor can do on a curatorial level, Because the peer review has a deeper, content-level engagement with the material, the author, and, ideally, the reader downstream -- who, we must remember, is always/already an author themselves, on their way in or out or across our community. THOSE ARE THE RELATIONSHIPS we have an opportunity to nurture and foreground. ------ wtd? Jackson et al 2018 labour and career stage, and social (academic) darwinism > As a further and final dimension that highlights the contradictions of outputs-driven academic labour, peer review is largely invisible and unrewarded—while positioned within a setting that awards all things visible and primarily the peer-review article [interestingly, reading the 'open' reviews of this article reveals little -- they are written safe; that is they don't engage in specualtion or collaborative ideation; they are focused on the mechanical and the objective. Things are deemed "complex" and "problematic" but not, I daresay, unexpected. Tennant & Ross-Hellauer talk about these constraints.]