You are here:
  1. Home
  2. Blog
  3. Tools of the Trade

Tools of the Trade

Blog

Kieran O'Leary from the Irish Film Institute shows format conversion and colour space info.

Photo credit: CC BY-SA Erwin Verbruggen.

No Time to Wait! A symposium on open formats

Audiovisual archivists have always needed to adopt and adapt the tools they needed from an industry that didn’t produce the tools they made with archivists in mind. This was a core thought at the recently held symposium No Time to Wait: Standardizing FFV1 & Matroska for Preservation. Taking place at the marvelous Deutsche Kinemathek building and the historically important Zuse Institute (German scientists will tell you computers were invented there - other nationalities will likely disagree), it brought together 70+ archivists and technologists. Sound & Vision is a partner in the PREFORMA project, which made the symposium possible. Berlin was chosen as a location, so that the symposium could nicely coincide with the meeting of the Internet Engineering Task Force (IETF), an important standardisation body that has gotten involved with a combination of open formats for preserving digital video.

Audiovisual archivists need specific skills and tools to manage their collections: handling a projector and hacking it to adapt different aspect ratios and frame rates, keeping a Steenbeck running that has been used on production for 30 years, knowing their way around a sea of video cables and audio toys,.... The same is true for the digital realm, albeit that many tools are now software-based. The PREFORMA project asked three groups of developers to build open tools for preservation. The idea is that archives need to normalise archives they receive in their repositories. Normalising meaning that files need to adhere to a specific standard - which is sometimes interpreted in different ways by the software that produces the file.

Hermann Lewetz from the Austrian Mediathek during the lightning talks.

Photo credit: CC BY-SA Erwin Verbruggen.

The MediaConch team - a project team that brought together by Jérôme Martinez and the team behind the MediaInfo software and independent developers such as Dave Rice and Ashley Blewer - set to work on building a conformance checker for a standard based on open formats: video codec FFV1 (in use at the Austrian Mediathek, the UK National Archives and, as of recent, Indiana University), and file wrapper Matroska. This also meant that these open formats needed to be standardised: the documentation brought up to date, the specifications adapted to what preservationists need. To this aim, the MediaConch team proposed a standardisation working group to the IETF: Codec Encoding for LossLess Archiving and Realtime transmission (CELLAR). As this working group was meeting in July, the team took the opportunity to bring together archivists who wanted to learn, users who had specific requests, developers who are spending time on building the tools, and the people who once designed these open formats.

Day one of the symposium gave participants a lot of food for thought, with presentations telling the history of the FFV1 and Matroska formats, how PREFORMA got involved, and how archivists at the Irish Film Archive not only implemented the format, but taught themselves to work with open source tools in order to keep the process chain in check. The skills needed to understand digital formats, and be able to investigate when something goes wrong without the help of a user interface, was a recurring theme during the conference. The community can learn much from the open source development community, which shares best practices and moves projects forward by means of open discussions. A few interesting, expert mailing lists in case are the ffmpeg-devel and the CELLAR lists. Adoption of formats was another important talking point. Matroska originator Steve Lhomme shared a timeline of what communities adopted the format throughout its lifetime, while Kate Murray from the Libray of Congress showed how perfecting a standard before showing its value can lead to diminished implementation. We were lucky to have the AS–07 standardisation example as a comparable (albeit rather distinctive) process example. The distinction here, however, is that AS-07 is a specification designed for the use of a standard (MXF, in this case) in an archive context. In the same vein, Michael Bradshaw from YouTube presented Google's development of the WebM specification, which is a flavour of the Matroska format, and creates certain constrictions to improve the format's streaming capabilities. Their adoption of WebM produced a cascading set of interoperability benefits, and the Google dev team added highly useful properties such as color metadata info to the format.

Kate Murray from the Library of Congress talks about the AS-07 specification.

Photo credit: CC BY-SA Erwin Verbruggen.

The second day was largely spent unconferencing-style: participants split in groups to discuss and propose improvements for the standards, how archival skills needed to be updated to work with digital tools, and how the formats we were working on could be made more useful and well-known. On the Tuesday afternoon, all joined at the Zuse institute to follow the CELLAR working group's meeting from a distance. As the working group is open, and the whole process is too, feel free to dig in the request for comments documents for EBML (the matroska-specific underlying format); Matroska and FFV1.

Dave Rice and Peter Bubestinger showed the timeline of FFV1 development.

Photo credit: CC BY-SA Erwin Verbruggen.

On the third day, the unconference format was alternated with a further host of interesting talks. After a lightning talk from Merle Friedrichsen from the Technische Informationsbibliothek on testing the speed of FFV1 versus lossless JPEG2000 (FFV1 being the faster one), Peter Bubestinger and Kieran Kunhya presented on testing the robustness of these Matroska/FFV1 files. After this, Dave Rice and Jérôme Martinez introduced the properties of the formats and the solution they're working on to shed some light on their proper build-up: the MediaConch conformity and policy checker. All in all, the symposium was an energizing meeting of minds - from specialists who'd been working on open formats and tools such as VLC and FFmpeg for years, and now got a better understanding of archivist needs; to novice archivists who are willing to dip their toes in these waters of open standards and technologies. There's plenty of work for all of us to do in the coming months and years to make sure that those analog and digital film and video signals we aim to keep are kept safe, and kept usable, for the foreseeable future. The certain thing is, that we cannot do it alone.

Group photo at the close of the No Time to Wait! symposium in Berlin.

Photo credit: CC BY-SA Erwin Verbruggen.

More info