Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

Antoni Oliver; Sergi Álvarez

Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

Antoni Oliver, Sergi Álvarez

Departamento de Traducción y de Interpretación y de Estudios del Asia Oriental

Universitat Oberta de Catalunya

Producción científica: Capítulo de libro › Capítulo › Investigación › revisión exhaustiva

3 Citas (Scopus)

Resumen

There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.

Idioma original	Inglés
Título de la publicación alojada	Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023
Editores	Mary Nurminen, Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartin, Mikel Forcada, Maja Popovic, Carolina Scarton, Helena Moniz
Editorial	European Association for Machine Translation
Páginas	39-45
Número de páginas	7
ISBN (versión digital)	9789520329471
Estado	Publicada - 2023

Serie de la publicación

Nombre	Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023

Otros archivos y enlaces

Enlace a la publicación en Scopus

Citar esto

Oliver, A., & Álvarez, S. (2023). Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. En M. Nurminen, M. Nurminen, J. Brenner, M. Koponen, S. Latomaa, M. Mikhailov, F. Schierl, T. Ranasinghe, E. Vanmassenhove, S. A. Vidal, N. Aranberri, M. Nunziatini, C. P. Escartin, M. Forcada, M. Popovic, C. Scarton, & H. Moniz (Eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023 (pp. 39-45). (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023). European Association for Machine Translation.

Oliver, Antoni ; Álvarez, Sergi. / Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. editor / Mary Nurminen ; Mary Nurminen ; Judith Brenner ; Maarit Koponen ; Sirkku Latomaa ; Mikhail Mikhailov ; Frederike Schierl ; Tharindu Ranasinghe ; Eva Vanmassenhove ; Sergi Alvarez Vidal ; Nora Aranberri ; Mara Nunziatini ; Carla Parra Escartin ; Mikel Forcada ; Maja Popovic ; Carolina Scarton ; Helena Moniz. European Association for Machine Translation, 2023. pp. 39-45 (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023).

@inbook{48240721ab374cf9a292a7612e8bd55b,

title = "Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training",

abstract = "There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.",

author = "Antoni Oliver and Sergi {\'A}lvarez",

note = "Publisher Copyright: {\textcopyright} 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC-BY-ND.",

year = "2023",

language = "English",

series = "Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023",

publisher = "European Association for Machine Translation",

pages = "39--45",

editor = "Mary Nurminen and Mary Nurminen and Judith Brenner and Maarit Koponen and Sirkku Latomaa and Mikhail Mikhailov and Frederike Schierl and Tharindu Ranasinghe and Eva Vanmassenhove and Vidal, {Sergi Alvarez} and Nora Aranberri and Mara Nunziatini and Escartin, {Carla Parra} and Mikel Forcada and Maja Popovic and Carolina Scarton and Helena Moniz",

booktitle = "Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023",

}

Oliver, A & Álvarez, S 2023, Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. En M Nurminen, M Nurminen, J Brenner, M Koponen, S Latomaa, M Mikhailov, F Schierl, T Ranasinghe, E Vanmassenhove, SA Vidal, N Aranberri, M Nunziatini, CP Escartin, M Forcada, M Popovic, C Scarton & H Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, European Association for Machine Translation, pp. 39-45.

Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. / Oliver, Antoni; Álvarez, Sergi.
Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. ed. / Mary Nurminen; Mary Nurminen; Judith Brenner; Maarit Koponen; Sirkku Latomaa; Mikhail Mikhailov; Frederike Schierl; Tharindu Ranasinghe; Eva Vanmassenhove; Sergi Alvarez Vidal; Nora Aranberri; Mara Nunziatini; Carla Parra Escartin; Mikel Forcada; Maja Popovic; Carolina Scarton; Helena Moniz. European Association for Machine Translation, 2023. p. 39-45 (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023).

Producción científica: Capítulo de libro › Capítulo › Investigación › revisión exhaustiva

TY - CHAP

T1 - Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

AU - Oliver, Antoni

AU - Álvarez, Sergi

PY - 2023

Y1 - 2023

N2 - There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.

AB - There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.

UR - http://www.scopus.com/inward/record.url?scp=85184805763&partnerID=8YFLogxK

M3 - Chapter

AN - SCOPUS:85184805763

T3 - Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023

SP - 39

EP - 45

BT - Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023

A2 - Nurminen, Mary

A2 - Brenner, Judith

A2 - Koponen, Maarit

A2 - Latomaa, Sirkku

A2 - Mikhailov, Mikhail

A2 - Schierl, Frederike

A2 - Ranasinghe, Tharindu

A2 - Vanmassenhove, Eva

A2 - Vidal, Sergi Alvarez

A2 - Aranberri, Nora

A2 - Nunziatini, Mara

A2 - Escartin, Carla Parra

A2 - Forcada, Mikel

A2 - Popovic, Maja

A2 - Scarton, Carolina

A2 - Moniz, Helena

PB - European Association for Machine Translation

ER -

Oliver A, Álvarez S. Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. En Nurminen M, Nurminen M, Brenner J, Koponen M, Latomaa S, Mikhailov M, Schierl F, Ranasinghe T, Vanmassenhove E, Vidal SA, Aranberri N, Nunziatini M, Escartin CP, Forcada M, Popovic M, Scarton C, Moniz H, editores, Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. European Association for Machine Translation. 2023. p. 39-45. (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023).

Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

Resumen

Serie de la publicación

Otros archivos y enlaces

Huella

Citar esto