Curators of the Infinite Haystack

By Kristin Oakley
Maria Popova, Caitlin Zaloom, Sharon Marcus, and Clay Shirky at the Institute for Public Knowledge on February 28Image Credit: Kristin Oakley

Maria Popova, Caitlin Zaloom, Sharon Marcus, and Clay Shirky at the Institute for Public Knowledge on February 28
Image Credit: Kristin Oakley

“Intelligence is really a kind of taste: taste in ideas.” – Susan Sontag

Here’s an understatement and a cliché: the Internet is a virtually infinite sea of information. Or perhaps it’s really an infinite haystack, in which the occasional needle can be found. This problem has only been exacerbated in recent years as the web’s network of hyperlinks and pages has been replaced by what The Atlantic’s Alexis Madrigal called “the Stream” – an endless flow of material that surrounds us. Rather than consuming, we are the ones consumed. So how can we be smarter, better functioning readers and viewers of online content?

Last Friday night, NYU’s Institute for Public Knowledge hosted a conversation to discuss this challenge of #curatingdigitalculture. Clay Shirky, Internet thinker and author of Here Comes Everybody and Digital Surplus; Maria Popova, creator of Brain Pickings; and Sharon Marcus and Caitlin Zaloom, editors in chief of Public Books (an online book review site) all gathered to talk about how they think about and discover challenging content.

Historical context guided the start of the discussion. Marcus, also a professor at Columbia University, began by articulating the historical role of the book review. Book reviews were supposed to be digests, not so much to interest readers in the book, but to summarize in a world of too many facts and too much information. Rather than a new phenomenon, the problem of information overload has plagued centuries of readers.

Shirky delved even further, to the invention of movable type, to make a similar point.  He reminded the audience that “from the moment there were more books than a person could read in a lifetime, some form of selection had to be the strategy.” He aims to change the conversation around the existing selection strategy often called “curation.” “People always want to call it information overload, it’s never information overload, it’s always filter failure,” said Shirky.

Of course, some of this filtering is undertaken by writers and thinkers themselves, and so Marcus, Zaloom, and Popova discussed the ways in which they themselves “curate” their own sites for a public audience. Popova admits she doesn’t think about the public audience at all; she only thinks about what interests her, as a private audience of one. Marcus and Zaloom look for the unexpected but try never to guide the opinions of their authors, leaving the door open for dissent.

As any conversation about the digital sphere tends to do, the discussion ended on the topic of artificial intelligence and “machine learning” and how we, as Internet readers, might increasingly rely on software algorithms (often deployed by our chosen websites) to filter out what we wish to avoid. Yet even as such algorithms have had their early successes, there is still a sense that machines just don’t select as well as humans do, and in quoting Sontag (above), Popova smartly posits why. “As long as there is a disconnect between human intelligence and artificial intelligence – as long as artificial intelligence is inferior to human intelligence – I think taste recommendation would be [inferior] as well.”

Still, I took note of the conditional nature of Popova’s statement: our curatorial superiority will last “as long as” artificial intelligence lags behind that of humans. We may soon see a day when computer curators can help solve the “filter failure” identified by Shirky. If so, we will look back upon these years as a discrete era, in which our capacity to create information greatly exceeded our ability to process it: an Age of Too Much Information giving way to an Epoch of Just Enough Information. Until then, good luck in the infinite haystack.

Kristin Oakley is editor at large for visual media at The Brooklyn Quarterly. She is currently pursuing her masters degree in digital journalism through Studio 20 at New York University. To learn more about her and see her work, visit

Join the Conversation