What the past looked like in color (according to a computer)
Ever wanted to know what historical images would look like in color? Rudy Marsman used a computer algorithm to add color to six black and white films from the archives of the Netherlands Institute for Sound & Vision. Rudy is a master student in Information Science at the VU University in Amsterdam. The automatic coloring of moving images was part of the research he did for his master thesis at Sound & Vision. Rudy explains what he found:
As part of my master's thesis I researched the possibilities to automatically color black and white videos. Recent developments in the field of artificial intelligence have made it possible to let a computer estimate which colors match a black and white photo or film clip.
Two stills from the film 'Steegjes', in which the right one is colored. Source: Polygoon-Profilti (producent) / Nederlands Instituut voor Beeld & Geluid (beheerder) / ingekleurd door Rudy Marsman, CC BY-SA
The math behind this is complex, but the principle is quite simple: let the computer recognize patterns and label these with colors. In this case a so-called artificial neural network was used. This is a computer programme built to be self-learning. The network that was used was trained with 1 million color photographs, which were entered in color and black and white. The computer had to guess which color each pixel in the photo should get. When the computer guessed right it was ‘rewarded’, and the computer was ‘punished’ for guessing wrong. Soon the computer discovered patterns such as ‘when I guess that in the top area of a photo the sky is blue I get rewarded’ or ‘everything that looks like a tree should get the color green’.
Two stills from the film 'Actie voor Wiesenthal', in which the right one is colored. Source: Polygoon-Profilti (producent) / Nederlands Instituut voor Beeld & Geluid (beheerder) / ingekleurd door Rudy Marsman, CC BY-SA
By showing the computer enough black and white images that are colored correctly - that is, if this set is a good representation of reality - it is possible to get results that are close to what people perceive as realistic..
But there is room for improvement. Because it takes a lot of computing power to train the network, it was decided to lower the resolution of videos. Also, in the case of film, each frame is colored individually, without looking at the frames before or after it. In some cases this can result in changing colors. Fortunately, the computing power has increased rapidly in the last years, so it is to be expected that similar technology as demonstrated here will become faster and more accurate.
The results of the computer coloring algorithm are astonishing. Some shots are colored very realistically, while other shots have a psychedelic character. These six videos were automatically colored:
The psychedelic colors from the 1970s can (unintentionally) be seen in this video about student life on the Uilenstede campus in Amsterdam. It demonstrates that automatically coloring doesn’t always work. Pieces of asphalt and grass are colored red and bathroom tiles have a blue glow. Yet seeing the rooms colored with orange and brown tones looks realistic for this period.
The Tour de France started in the Netherlands for the first time in 1954. This resulted in these exciting images. We can see that the algorithm struggled with the yellow jersey worn by stage winner Wout Wagtmans. It is colored in many different ways, except yellow.
Images from World War II are mainly known in black and white. The addition of color draws them closer to the present. Especially impressive here are the images of the destroyed city centre of Groningen, which comes to life with the use of color.
The impressive Paleis voor de Volksvlijt (Palace of Popular Diligence) and the accompanying gallery used to be a familiar sight at the Frederiksplein in Amsterdam. After the building burned down in 1929, the accompanying gallery was demolished in 1960 to make way for the new building of the Nederlandsche Bank (Dutch central bank). This to the dismay of many.
This video shows that the algorithm manages the coloring of people very well. The black and white images of the famous Nazi hunter Simon Wiesenthal were colored here quite natural.
Color images of life in the center of Amsterdam in the 1920s are quite rare. Thus, this colored film gives us an interesting insight in what daily life looked like, with its narrow streets and poverty among the residents.
Despite the varying results of this computer algorithm, it clearly shows how new technologies can enrich historical material in unexpected ways. It gives a new perspective on existing images. Especially if this technology will get better in the future, it offers interesting opportunities to get to know our past in new ways.