The Potential for AI in Audiovisual Archives

With the metadata generated from this analysis, PBS Learning is working with the Pool Party semantic web company to assign clips to taxonomies that are correlated with school curricula. These clips can eventually be made available to teachers for use with concepts such as geometry or emotional development, and can also be delivered directly to students in individualized user environments.

In an interview, Athina Livanos-Propst, Editorial Services Manager & Digital Librarian at PBS Learning Media, talked about the project and the potential this “smart stacking” has for shaping audience experience in the future. For example, in working with GrayMeta to create focused categories for analysis, called “Insight groups,” algorithms have been trained to search for unique characteristics such as “fruit” or “feelings” or “weather,” and this information can be matched to audience preferences. She shared a hypothetical scenario in which this type of process could be used to identify that a viewer not only enjoyed British murder mysteries, but that they also preferred ones that included a lot of fog. Eventually, this type of information could be used to provide viewers with individually tailored experiences.

In an interview with staff from meemoo, The Flemish Institute for Archives, Debbie Esmans, Manager Policy & Strategy, explored the question of what to do with user information so that it doesn’t encourage the “funnel” effect that most digital providers achieve. She spoke about how public broadcasters have an official mandate to help audiences broaden their tastes (smaakverruiming), and wondered how AI could help in that process. This is an area rich for investigation, and points to the potential for humans and AI to work together in more collaborative ways as proposed in the publication “A Research Agenda for Hybrid Intelligence: Augmenting Human Intellect With Collaborative, Adaptive, Responsible, and Explainable Artificial Intelligence." (Link) In it, the notion of hybrid intelligence (HI) is defined as “the combination of human and machine intelligence, augmenting human intellect and capabilities instead of replacing them and achieving goals that were unreachable by either humans or machines.” Public service-driven organizations such as broadcasters and archives are in a unique position to develop projects in which AI serves to amplify human intelligence.

Collections as Data

As AI opens up new opportunities for archives to reconceive their role, scholars are also reconsidering their relationship to archival materials. Scholars in the humanities have been working with technologies such as Optical Character Recognition (OCR) for several years, giving new life to the data found in digitized text documents. Computer vision technology has recently created new opportunities to study still and moving images, and the field of digital humanities is rapidly taking up this new opportunity and theorizing about how it will expand the kinds of questions that researchers can ask of data. In 'The visual digital turn: Using neural networks to study historical images', Melvin Wevers and Thomas Smits explain:

Computer vision techniques offer the possibility to track and analyze specific kinds of visual representations over long periods in the archive. In addition, the use of these techniques and the results they yield can lead to new research questions and areas of inquiry (p 11.)

These new types of questions are created when researchers become familiar with new tools, and with the data in particular collections. The Santa Barbara Statement on Collections as Data explains the concept of Collections as Data:

Collections as data development aims to encourage computational use of digitized and born digital collections. By conceiving of, packaging, and making collections available as data, cultural heritage institutions work to expand the set of possible opportunities for engaging with collections. 

Some audiovisual archives are making their collections available directly to scholars, such as The American Archive of Public Broadcasting’s Dataset Research Access service, while others are partnering with academics to build a research infrastructure, like the CLARIAH Media Suite in the Netherlands. The following examples illustrate how archives and researchers are working together to find the value of the data in collections.

The GDELT project

Perhaps the largest endeavor in this space, the GDELT project, uses a suite of Google’s AI technologies to analyze a massive trove of daily online news productions assembled by the Internet Archive. By analyzing online newspapers and broadcast news from around the world in 152 languages, GDELT is able to assist researchers in asking novel questions about media and culture, representation of news across cultures, and global trends. It publishes all of the data it creates as annotations to be used freely by researchers, relying on the concept of Non-Consumptive Use. One example of the potential of this type of global news tracking happened when Bluedog Global’s machine-learning tools noticed local news outlets in Wuhan, China talking about a disease outbreak in December of 2019. By following this thread, researchers at Bluedog Global were able to make the very first global announcement about the COVID-19 outbreak on December 31, 2019, a week before the US Centers for Disease Control. 

BoB FOR AI

The BoB FOR AI project from Learning on Screen, the British audiovisual resource for educators, is “… exploring the potential in our archive of over 2.4 million television and radio broadcasts as a dataset to train algorithms – the BoB archive – available for research purposes, particularly in the field of Artificial Intelligence.” BoB FOR AI is collaborating directly with researchers to explore what kinds of questions can be asked of the data that have a direct impact upon society. In one pilot project with data scientists at Nesta, they are studying racial and gender diversity in British Television. In an interview, data scientist Raphael Leung explained to me that while recent initiatives such as the BFI Diversity Standards are actively increasing diversity in film and television, they also created an evidence gap that computer vision could fill. Motivated by a belief that “inclusion fuels creativity,” the project uses computer vision to measure the actual inclusivity of the media landscape, and looks to imagine what other sorts of gaps computer vision can fill.

Analyzing Visual Culture through Distant Viewing

At the University of Richmond in the US, scholars Taylor Arnold and Lauren Tilton have created the Distant Viewing Lab, which 

…uses and develops computational techniques to analyze visual culture on a large scale. It develops tools, methods, and datasets that can be re-used by other researchers. The lab engages closely with critical cultural and data studies, aiming to make explicit the interpretive act of algorithmic logic.

Arnold and Tilton’s paper Distant Viewing: Analyzing Large Visual Corpora argues for an interdisciplinary perspective on computational analysis of large amounts of visual material – which they call “Distant Viewing” – that takes into account deeper levels of meaning such as cultural and semiotic interpretation. An example of this method is the essay Visual Style in Two Network Era Sitcoms by Taylor Arnold, Lauren Tilton, and Annie Berke, in which they compare narrative structures and character centrality in the 1960s American sitcoms Bewitched and I Dream of Jeannie. 

CLARIAH Media Suite

In the Netherlands, multiple archives contribute to the CLARIAH (Common Lab Research Infrastructure for the Arts and Humanities) Media Suite, where data and tools for analysis are bundled together. The Media Suite, an online resource accessible to people affiliated with Dutch academic institutions, is actively experimenting with and exploring different ways this data can be used. They have recently begun sharing some of their own work Visualising The Archive of the collection of the Netherlands Institute for Sound & Vision. They are also working with researchers and students to create Media Suite Data Stories such as 'Factual divergences in Dutch television news: comparisons between AIDS/HIV, SARS and COVID-19 news reportings' (Link). In an interview, data engineer Mari Wigham expressed her enthusiasm for working with people on creating data visualizations and data stories and the potential they have to open up the archive. She also added a note of caution that the data was best used to find interesting trends for further investigation, but not as hard proof of social or historical phenomena. She explained that since metadata comes from a variety of sources, both machine- and human-generated, sometimes small changes in how metadata has been documented over the years might lead people to misinterpret findings. “To draw accurate conclusions from archival data, it is essential both to understand how it was produced, and how it relates to the real-life question being asked.” 

Conclusion

This blog post has looked at ways audiovisual archives are using AI to expand their roles in a range of fields. The next blog post will look at the kinds of analysis that AI can perform on audiovisual collections.

The next blog post will look at the possibilities of AI analysis in audiovisual archives.

Newsletter Research 

Subscribe to the newsletter Research of Sound & Vision and stay informed of all meetings and activities we do to make our collections accessible for research. The newsletter is in Dutch.