A Multilingual Approach to Scene Text Visual Question Answering

Josep Brugués i Pujolràs; Lluís Gómez i Bigordà; Dimosthenis Karatzas

doi:10.1007/978-3-031-06555-2_5

A Multilingual Approach to Scene Text Visual Question Answering

Josep Brugués i Pujolràs, Lluís Gómez i Bigordà^*, Dimosthenis Karatzas

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in Book › Chapter › Research › peer-review

1 Citation (Scopus)

Abstract

Scene Text Visual Question Answering (ST-VQA) has recently emerged as a hot research topic in Computer Vision. Current ST-VQA models have a big potential for many types of applications but lack the ability to perform well on more than one language at a time due to the lack of multilingual data, as well as the use of monolingual word embeddings for training. In this work, we explore the possibility to obtain bilingual and multilingual VQA models. In that regard, we use an already established VQA model that uses monolingual word embeddings as part of its pipeline and substitute them by FastText and BPEmb multilingual word embeddings that have been aligned to English. Our experiments demonstrate that it is possible to obtain bilingual and multilingual VQA models with a minimal loss in performance in languages not used during training, as well as a multilingual model trained in multiple languages that match the performance of the respective monolingual baselines.

Original language	English
Title of host publication	Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings
Editors	Seiichi Uchida, Elisa Barney, Véronique Eglin
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	65-79
Number of pages	15
ISBN (Print)	9783031065545
DOIs	https://doi.org/10.1007/978-3-031-06555-2_5
Publication status	Published - 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13237 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Keywords

Deep learning
Multilingual word embeddings
Scene text
Vision and language
Visual question answering

Access to Document

10.1007/978-3-031-06555-2_5

Cite this

Brugués i Pujolràs, J., Gómez i Bigordà, L., & Karatzas, D. (2022). A Multilingual Approach to Scene Text Visual Question Answering. In S. Uchida, E. Barney, & V. Eglin (Eds.), Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings (pp. 65-79). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13237 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-06555-2_5

Brugués i Pujolràs, Josep ; Gómez i Bigordà, Lluís ; Karatzas, Dimosthenis. / A Multilingual Approach to Scene Text Visual Question Answering. Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings. editor / Seiichi Uchida ; Elisa Barney ; Véronique Eglin. Springer Science and Business Media Deutschland GmbH, 2022. pp. 65-79 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inbook{88b3bfcb126a44699cdc7a35201d5863,

title = "A Multilingual Approach to Scene Text Visual Question Answering",

abstract = "Scene Text Visual Question Answering (ST-VQA) has recently emerged as a hot research topic in Computer Vision. Current ST-VQA models have a big potential for many types of applications but lack the ability to perform well on more than one language at a time due to the lack of multilingual data, as well as the use of monolingual word embeddings for training. In this work, we explore the possibility to obtain bilingual and multilingual VQA models. In that regard, we use an already established VQA model that uses monolingual word embeddings as part of its pipeline and substitute them by FastText and BPEmb multilingual word embeddings that have been aligned to English. Our experiments demonstrate that it is possible to obtain bilingual and multilingual VQA models with a minimal loss in performance in languages not used during training, as well as a multilingual model trained in multiple languages that match the performance of the respective monolingual baselines.",

keywords = "Deep learning, Multilingual word embeddings, Scene text, Vision and language, Visual question answering",

author = "{Brugu{\'e}s i Pujolr{\`a}s}, Josep and {G{\'o}mez i Bigord{\`a}}, Llu{\'i}s and Dimosthenis Karatzas",

note = "Funding Information: Acknowledgment. This work has been supported by: Grant PDC2021-121512-I00 funded by MCIN /AEI/10.13039/501100011033 and the European Union NextGen-erationEU/PRTR; Project PID2020-116298GB-I00 funded by MCIN/ AEI /10.13039/501100011033; Grant PLEC2021-007850 funded by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR. Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.",

year = "2022",

doi = "10.1007/978-3-031-06555-2_5",

language = "English",

isbn = "9783031065545",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "65--79",

editor = "Seiichi Uchida and Elisa Barney and V{\'e}ronique Eglin",

booktitle = "Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings",

}

Brugués i Pujolràs, J, Gómez i Bigordà, L & Karatzas, D 2022, A Multilingual Approach to Scene Text Visual Question Answering. in S Uchida, E Barney & V Eglin (eds), Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13237 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 65-79. https://doi.org/10.1007/978-3-031-06555-2_5

A Multilingual Approach to Scene Text Visual Question Answering. / Brugués i Pujolràs, Josep; Gómez i Bigordà, Lluís ; Karatzas, Dimosthenis.
Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings. ed. / Seiichi Uchida; Elisa Barney; Véronique Eglin. Springer Science and Business Media Deutschland GmbH, 2022. p. 65-79 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13237 LNCS).

Research output: Chapter in Book › Chapter › Research › peer-review

TY - CHAP

T1 - A Multilingual Approach to Scene Text Visual Question Answering

AU - Brugués i Pujolràs, Josep

AU - Gómez i Bigordà, Lluís

AU - Karatzas, Dimosthenis

N1 - Funding Information: Acknowledgment. This work has been supported by: Grant PDC2021-121512-I00 funded by MCIN /AEI/10.13039/501100011033 and the European Union NextGen-erationEU/PRTR; Project PID2020-116298GB-I00 funded by MCIN/ AEI /10.13039/501100011033; Grant PLEC2021-007850 funded by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR. Publisher Copyright: © 2022, Springer Nature Switzerland AG.

PY - 2022

Y1 - 2022

N2 - Scene Text Visual Question Answering (ST-VQA) has recently emerged as a hot research topic in Computer Vision. Current ST-VQA models have a big potential for many types of applications but lack the ability to perform well on more than one language at a time due to the lack of multilingual data, as well as the use of monolingual word embeddings for training. In this work, we explore the possibility to obtain bilingual and multilingual VQA models. In that regard, we use an already established VQA model that uses monolingual word embeddings as part of its pipeline and substitute them by FastText and BPEmb multilingual word embeddings that have been aligned to English. Our experiments demonstrate that it is possible to obtain bilingual and multilingual VQA models with a minimal loss in performance in languages not used during training, as well as a multilingual model trained in multiple languages that match the performance of the respective monolingual baselines.

AB - Scene Text Visual Question Answering (ST-VQA) has recently emerged as a hot research topic in Computer Vision. Current ST-VQA models have a big potential for many types of applications but lack the ability to perform well on more than one language at a time due to the lack of multilingual data, as well as the use of monolingual word embeddings for training. In this work, we explore the possibility to obtain bilingual and multilingual VQA models. In that regard, we use an already established VQA model that uses monolingual word embeddings as part of its pipeline and substitute them by FastText and BPEmb multilingual word embeddings that have been aligned to English. Our experiments demonstrate that it is possible to obtain bilingual and multilingual VQA models with a minimal loss in performance in languages not used during training, as well as a multilingual model trained in multiple languages that match the performance of the respective monolingual baselines.

KW - Deep learning

KW - Multilingual word embeddings

KW - Scene text

KW - Vision and language

KW - Visual question answering

UR - http://www.scopus.com/inward/record.url?scp=85131136455&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-06555-2_5

DO - 10.1007/978-3-031-06555-2_5

M3 - Chapter

AN - SCOPUS:85131136455

SN - 9783031065545

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 65

EP - 79

BT - Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings

A2 - Uchida, Seiichi

A2 - Barney, Elisa

A2 - Eglin, Véronique

PB - Springer Science and Business Media Deutschland GmbH

ER -

Brugués i Pujolràs J, Gómez i Bigordà L , Karatzas D. A Multilingual Approach to Scene Text Visual Question Answering. In Uchida S, Barney E, Eglin V, editors, Document Analysis Systems - 15th IAPR International Workshop, DAS 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 65-79. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-06555-2_5