GptGet

Training A Small Emotional Vision Language Model for Visual Art Comprehension

TL;DR

SEVLM model enhances visual art emotion understanding via VAD emotion modeling and contrastive learning, outperforming existing small models.

cs.CV 🟡 Intermediate 2024-03-17 2 views
Jing Zhang Liang Zheng Meng Wang Dan Guo
AI Reader Arxiv Page Download PDF
visual art emotion understanding small model VAD modeling contrastive learning

Key Findings

Methodology

This paper introduces a Small Emotional Vision Language Model (SEVLM) that enhances emotion understanding by integrating VAD emotion modeling and input-output feature alignment. VAD modeling incorporates emotional features into model inputs using a psychology-expert-annotated dictionary. Contrastive learning aligns the features of images, emotion classes, and explanations.

Key Results

  • SEVLM achieved an emotion classification accuracy of 65.6% on the ArtEmis v1.0 dataset, improving by 2.1% over the baseline.
  • On the ArtEmis v2.0 dataset, SEVLM improved the emotion alignment metric by 3.9%.
  • Compared to LLaVA 7B, SEVLM is trainable on a single RTX 2080 Ti, achieving comparable performance with lower computational resources.

Significance

This research is significant in the field of visual art emotion understanding, offering an efficient solution that breaks the trade-off between computational efficiency and performance in small models. It has implications for both academia and industry, providing new tools for emotion analysis.

Technical Contribution

The technical contributions include: 1) Introducing the VAD dictionary to enhance text emotion features; 2) Designing a VAD head to align system outputs with true emotion vectors; 3) Proposing a contrastive head to align features of images, emotion classes, and explanations.

Novelty

This work is the first to apply VAD modeling to emotion explanation in vision-language models, achieving multimodal feature alignment through contrastive learning, offering significant innovation over existing methods.

Limitations

  • The model may underperform in complex emotional scenarios, especially where multiple emotions are intertwined.
  • Dependence on the VAD dictionary may limit applicability in other languages.

Future Work

Future work could explore building multilingual emotion dictionaries and validating on larger datasets to enhance model generalization and applicability.

AI Executive Summary

Understanding emotions in visual art has been a challenge in computer vision. Existing methods often rely on large models with high computational costs. This paper proposes a Small Emotional Vision Language Model (SEVLM) that significantly enhances emotion understanding in small models through VAD emotion modeling and contrastive learning.

SEVLM incorporates emotional features into model inputs using a VAD dictionary and aligns features of images, emotion classes, and explanations through contrastive learning. Experimental results show that SEVLM outperforms existing small models on the ArtEmis datasets and can be efficiently trained on a single RTX 2080 Ti.

This research has significant implications for academia and industry, providing new tools for emotion analysis. Future work will explore building multilingual emotion dictionaries and validating on larger datasets to enhance model generalization and applicability.

Deep Analysis

Background

Emotion understanding in visual art is an interdisciplinary field involving computer vision and natural language processing. Traditional methods often rely on large models like LLaVA and GPT-4, which, while performant, incur high computational costs. Recently, small models have gained attention for their computational efficiency, but their performance is often limited.

Core Problem

Small models have limited performance in emotion understanding, especially when dealing with complex visual art. The challenge is to enhance emotion understanding while maintaining computational efficiency.

Innovation

The core innovations of this paper include introducing VAD emotion modeling and contrastive learning. VAD modeling incorporates emotional features into model inputs using a psychology-expert-annotated dictionary. Contrastive learning aligns the features of images, emotion classes, and explanations.

Methodology

  • �� Use CLIP as the image encoder to extract visual features.
  • �� Incorporate VAD dictionary to enhance text emotion features.
  • �� Design a VAD head to align system outputs with true emotion vectors.
  • �� Use contrastive learning to align features of images, emotion classes, and explanations.

Experiments

Experiments were conducted on the ArtEmis v1.0 and v2.0 datasets, using metrics like accuracy and emotion alignment to evaluate model performance. The baseline model was CLIP-GPT2, and comparisons included LLaVA 7B.

Results

SEVLM achieved an emotion classification accuracy of 65.6% on the ArtEmis v1.0 dataset, improving by 2.1% over the baseline. On the ArtEmis v2.0 dataset, SEVLM improved the emotion alignment metric by 3.9%.

Applications

SEVLM can be used for art emotion analysis and emotion-driven image retrieval, particularly suitable for resource-constrained environments.

Limitations & Outlook

The model may underperform in complex emotional scenarios, especially where multiple emotions are intertwined. Dependence on the VAD dictionary may limit applicability in other languages.

Plain Language Accessible to non-experts

Imagine you're looking at a painting, trying to understand its emotions. SEVLM is like a smart assistant that analyzes the colors, shapes, and themes in the painting, using a dictionary provided by psychology experts, to help you better understand the painting's emotions. It not only tells you the emotion conveyed by the painting but also explains why.

ELI14 Explained like you're 14

Imagine you're playing a game with lots of paintings, each with different emotions. SEVLM is like a game assistant that helps you quickly identify the emotion of each painting and tells you why it feels that way. It's like having a super-smart friend who makes the game more fun!

Glossary

VAD (Valence-Arousal-Dominance)

VAD is a three-dimensional model of emotion: valence, arousal, and dominance, used to describe emotional features.

Used to enhance text emotion features.

SEVLM (Small Emotional Vision Language Model)

A small model that enhances emotion understanding through VAD modeling and contrastive learning.

Used for visual art emotion understanding.

Contrastive Learning

A machine learning technique used to align features across different modalities.

Used to align features of images, emotion classes, and explanations.

CLIP

A vision encoder used to extract image features.

Used as the image encoder in SEVLM.

GPT2

A language model used to generate text explanations.

Used as the language decoder in SEVLM.

Open Questions Unanswered questions from this research

  • 1 How to apply VAD emotion modeling in multilingual environments?
  • 2 How to improve model performance in complex emotional scenarios?

Applications

Immediate Applications

Art Emotion Analysis

Helps artists and critics analyze the emotional features of artworks, enhancing emotional expression.

Long-term Vision

Emotion-Driven Image Retrieval

Retrieves images based on emotional features, enhancing user experience.

Abstract

This paper develops small vision language models to understand visual art, which, given an art work, aims to identify its emotion category and explain this prediction with natural language. While small models are computationally efficient, their capacity is much limited compared with large models. To break this trade-off, this paper builds a small emotional vision language model (SEVLM) by emotion modeling and input-output feature alignment. On the one hand, based on valence-arousal-dominance (VAD) knowledge annotated by psychology experts, we introduce and fuse emotional features derived through VAD dictionary and a VAD head to align VAD vectors of predicted emotion explanation and the ground truth. This allows the vision language model to better understand and generate emotional texts, compared with using traditional text embeddings alone. On the other hand, we design a contrastive head to pull close embeddings of the image, its emotion class, and explanation, which aligns model outputs and inputs. On two public affective explanation datasets, we show that the proposed techniques consistently improve the visual art understanding performance of baseline SEVLMs. Importantly, the proposed model can be trained and evaluated on a single RTX 2080 Ti while exhibiting very strong performance: it not only outperforms the state-of-the-art small models but is also competitive compared with LLaVA 7B after fine-tuning and GPT4(V). The code is available at https://github.com/BetterZH/SEVLM-code.

cs.CV