Pilar Rosado
Image with SIFT descriptors (Scale Invariant Feature Transform, Lowe, 2001) de Pilar Rosado.
We have invited Dr Pilar Rosado to talk about the most outstanding aspects of her research on computer vision as applied to art, the subject of her PhD thesis. This is a post of a specialization that is maybe unusual in blogs, but clearly explained with very explanatory images, which we think is very interesting for the possibilities of new visual connections between works of art.
Computer vision techniques make available new possibilities of approaching the collections of art institutions. Here I present this alternative point of view that may help to stimulate the interest of the public and scholars.
Using computer vision models that emulate the initial stages of human visual perception, we have managed to categorize large groups of images from abstract works of art by the artist Antoni Tàpies. The images have been analysed based on their compositional, colour and organizational characteristics, without the need to make any textual notes about them, so that the analogies found may take us closer to, and help us to understand, the creator’s original values.
The whole is greater than the sum of the parts
Artists are specialist image generators; in their creative process they produce them constantly. In the collections of images by artists there are some links, formal relationships that make them a family of common meaning.
Storage room of the MUSAC (Museum of Contemporary Art Castilla-León). Artworks by Daniel Verbis. Photo: Jesús F. Salvadores, Diario de León
The arrangement and categorization of a large collection of images gives us an inestimable overall view of the whole. In the field of artistic creation and its teaching, it is a very common practice to place groups of photographic images, sculptures or paintings next to one another. It is surprising to observe how the visual objects establish a dialogue with one another, sometimes in harmony and others in contradiction. Thus the artist, the professor or the art student has information available that, treating the same objects separately, would be inaccessible. This aspect is especially valuable when studying abstract images, given that in them the subject, the meaning or the sense is not the product of a social pact, but of visual resonances and synchronies that the artist-creator relates and connects.
New technologies: computer vision algorithms
The new technologies make it possible to digitize this content and make quick access to their visualization easier. But to seek genuine relationships between images, with visual content of their own, more sophisticated search tools are necessary, which require no textual notes, seeing as this very writing would condition the relationships that the image might establish a posteriori. A sub-field of artificial intelligence, computer vision, performs this task.
From pixels to visual resonances
How can we extrapolate information about the overall characteristics of an image, starting only with certain pixels?
In 1997, the neurobiologist Tanaka published his conclusions about the neuronal mechanisms of the recognition of objects in primates, and in 2001, computer vision expert Lowe, based on this information, described a system of computer vision based on descriptors like scaling, translation, rotation and illumination, and he called them SIFT (Scale-invariant feature transform) descriptors. He found the best compromise solution between performance and speed using a sample grid of 16 x 16 pixel gradients. Each of these small regions would summarize the information of the region surrounding the image.
We could thus say that the computer algorithm, through this model, “cuts up” all the images in the collection into small fragments of 16 x 16 pixels and groups them together according to the mathematical distances between their SIFT descriptors. We call each one of the groups established a “visual word” and the whole makes up the “visual vocabulary” of the collection.
With this information, the system is able to determine concurrences of these “visual words” in certain images. For example, in portraits, two eyes, a nose, a mouth and a forehead concur. The system will group all the images of faces together in the category “portrait”, and this category would be the one we call the “latent aspect”.
Computer vision applied to the work of Antoni Tàpies
The Antoni Tàpies Foundation in Barcelona has kindly allowed me access to the collection in order to carry out this study.
I have applied computer vision models to a group of 434 images of works by the artist from different periods, done with multiple techniques on varying mediums – engravings, watercolours, drawings, paintings and collages.
I shall now briefly show and describe some of the latent aspects detected by the algorithm, with no manual intervention whatsoever.
Latent aspect 1: Has a large amount of empty space, measured, with different, very studied motifs, located in a more or less central or balanced position.
1. Llull-Tàpies, 1985. © Comissió Tàpies, VEGAP. Photo: © Lluís Bover, 2016 2. Cartes per a la Teresa, 1974. © Comissió Tàpies, VEGAP. Photo: © Lluís Bover, 2016 3. Terra d’ombra I, 1998. © Fundació Antoni Tàpies, Barcelona / VEGAP. Photo: © Gasull Fotografia, 2016
Latent aspect 2: Contains works done with a vigorous dark-coloured line on a lighter background.
1. Dans la double dépendance du dit, 1984. 2. A. Tàpies. Catàleg. Sala Gaspar, Barcelona, 1969. 3. Air, 1971. © Comissió Tàpies, VEGAP. Photos: © Lluís Bover, 2016
Latent aspect 3: Groups together works that have a scattered distribution of clear areas of flatter ink, although not completely smooth, with greyer areas in which the brushstroke is not wholly dense (textured brushstroke).
1. Intérieur avec figures, 1987. 2. L’estació, 1984. 3. Ce que je vous dis trois fois est vrai, 1982. © Comissió Tàpies, VEGAP. Photos: © Lluís Bover, 2016
Latent aspect 4: Agglutinates works that have a dotted surface or on which the paint seems to have been vaporized, in general, granulated texture.
1, 3 i 4. Tres aiguaforts, 1943. © Comissió Tàpies, VEGAP. Photo: © Lluís Bover, 2016. 2. Origen. Sèrie Història Natural, 1950-1951. © Fundació Antoni Tàpies, Barcelona / VEGAP. Photo: © Gasull Fotografia, 2016
Latent aspect 5: Is made up of works with few broad gestural lines. They are dark brushstrokes on a lighter background. The tendency is for the principal motif to be in the centre, covering the entire image.
1. Sèrie negre número XIV, 1967. © Fundació Antoni Tàpies, Barcelona / VEGAP. Photo: © Gasull Fotografia, 2016. 2. Sèrie negre número III, 1967. © Fundació Antoni Tàpies, Barcelona / VEGAP. Photo: © Gasull Fotografia, 2016. 3. Équation, 1987. © Comissió Tàpies, VEGAP. Photo: © Lluís Bover, 2016
Latent aspect 6: Presents images with a diffuse atmosphere and a hazy, blurred treatment.
1. Esbós, 1958. © Fundació Antoni Tàpies, Barcelona / VEGAP. Photo: © Gasull Fotografia, 2016. 2. Llull-Tàpies, 1985. © Comissió Tàpies, VEGAP. Photo: © Lluís Bover, 2016. 3. Dibuix, 1948. © Fundació Antoni Tàpies, Barcelona / VEGAP. Photo: © Gasull Fotografia, 2016
The system learns
An added benefit of the arrangement of the collection is that once the latent aspects have been established as categories, the system “learns”; this model would automatically make it possible to classify images from other collections, according to the aesthetic criteria of the categories found.
It is impossible to do this kind of study on large collections in another way, because of the obvious limitations of access to the works and of human analytical capacity. And, even though any categorization is by definition subjective by being based on a certain criterion, digital technology offers us this way of approaching information that would otherwise be impossible due to the huge amount of visual content stored.
The system that I have described would make it possible to repeat the studies on different periods of the same artist, or on collections by different artists or periods, with the same criteria. In this way, the results obtained could be compared without the risk of making subjective interpretations conditioned by prior preferences or knowledge. A consultation tool of this kind made available to museum visitors would enable them to establish their own aesthetic selection criteria.
Acknowledgements
My thanks to the archive of the Antoni Tàpies Foundation in Barcelona and the Tàpies Commission for kindly giving me access to the collection of graphic work, paintings and artist’s books by Antoni Tàpies, and especially to their directors Laurence Rassel and Carles Guerra for their willingness to facilitate my research.
Recommended links
Semantic-Based Image Analysis with the Goal of Assisting Artistic Creation.
Artistic ideation based on computer vision methods, Journal of Theoretical and Applied Computer Science
Formas latentes: protocolos de visión artificial para la detección de analogías aplicados a la catalogación y creación artística. Tesi doctoral, Pilar Rosado
Pilar Rosado
BRAC group researcher, University of Barcelona
Any form of total or partial reproduction, distribution, public communication or transformation of the images without the written permission of the rights holders is totally prohibited.
2 Comments
It’s very useful to all and us , I am very much impressed with your thoughts. I got the best information from this site of the blog, That article was amazing. Thanks for sharing this post.
Well, This is very helped us and useful to all. I am very happy to get this and very impressive thing that you mentioned in the article. Thanks for sharing this post.