Suggestions for Writing NeurIPS 2020 Broader Impacts Statements

It was exciting to learn this week that NeurIPS — one of the top AI research publication venues in the world — will now require authors to submit statements about the societal impacts of their research. This is a big change. In 2018, I worked with a group in the ACM FCA on the proposal that inspired this change. When the FCA proposal hit Twitter, HackerNews, etc. two years ago, it prompted an enormous and insightful discussion. Drawing on the proposal and the discussion surrounding it, I wrote up some quick advice for NeurIPS 2020 authors that might be helpful in crafting their newly-mandatory broader impact statements:

(1) At a high level, your funding agencies, your company’s leaders, and the general public are about to get a better view into how your work published at NeurIPS might affect the world. That will likely introduce new and healthy accountability into your research lives.

(2) Start writing your NeurIPS broader impacts statement today. Next year, think deeply about broader impacts before beginning a project. These practices will allow you to pivot your project to have a statement you’ll be proud of or, in some cases, choose a different project!

This should be exciting! There are so many interesting and important research challenges that involve internalizing the negative societal externalities we have been creating in AI. NeurIPS broader impact statements will likely help to catalyze additional work on these challenges.

(3) Understanding the societal impacts of your work is going to be HARD. It’s going to take *lots* of effort to write NeurIPS broader impacts statements. As my former Ph.D. advisor Darren Gergle would say (a quote that was passed down to him): “the social sciences are the hard sciences”.

(4) The good news: Tons of work has already been done for you! Check out the literature from communities that have studied societal impacts of AI for a long while, e.g. ACM SIGCHI and ACM FAccT (formerly FAT*). For deeper dives, look into science and technology studies (STS) venues.

(5) Even better, bring a social scientist onto your research team! Remember though, they don’t work for free. Hire them into your company. Give them sub-awards. Recruit them as Ph.D. students through interdisciplinary programs, and so on (h/t to Lilly Irani for the suggestion).

(6) For those who do more theoretical work, as our proposal discusses, writing a rigorous statement is likely going to be more difficult. Look at the proposal’s examples (“Storage and Computation”), and I’d encourage you to experiment; find something that is both rigorous and practical for your research.

More generally, it’s likely that to get funding for any theoretical work, someone had to make an argument about positive societal impacts at some point. If that argument is possible, it’s probably also possible to make a rigorous statement about some negative societal impacts.

(7) You might be tempted to write boilerplate, low-information statements. Don’t do this. It’s not scientifically rigorous and will undermine the rigor in the rest of your paper. The public will roll its eyes, and reviewers may (and often should) call you out (see proposal for more).

(8) As our proposal says, “it will take a non-trivial amount of time for norms and standards to develop”. This is an experiment and it will generate a ton of interesting data. This isn’t going to work perfectly for NeurIPS 2020; the community will need to iterate each year. For the first year, the author response period will be a decent opportunity to have a bit of a dialogue between author and reviewer on the impacts statement.

(9) As authors, you’re also likely reviewers (esp. this year!). NeurIPS leaders should probably address this directly, but our proposal’s view is that it’s not your job as a reviewer to judge submissions for their impacts. Rather, you should evaluate the *rigor with which they disclose their impacts*. Our proposal also recommends that reviewers adopt a “big tent” approach as “norms and standards develop”.

(10) Lastly, you might want to read our full proposal. It will help you understand motivations for these types of statements and has a number of other tips (including several hypothetical case studies) https://acm-fca.org/2018/03/29/negativeimpacts/

The proposal was an awesome and complete team effort involving Lauren Wilcox, Jeff Bigham, Jason Ernst, Johannes Schöning, Yonatan Bisk, and the many other co-authors on the proposal. Many of the above tips I’m sure draw from subsequent conversations with each of them. And a huge hat tip Moshe Vardi for his insights during the crafting of the proposal.

Note: In case it is unclear from the above, this post is meant to provide food-for-thought from the perspective of folks who have been thinking about these issues for a long while. Please defer to the NeurIPS 2020 organizers for questions about their specific implementation.