Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

Antoni Oliver; Sergi Álvarez

Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

Antoni Oliver, Sergi Álvarez

Department of Translation and Interpreting and East Asia Studies

Universitat Oberta de Catalunya

Research output: Chapter in Book › Chapter › Research › peer-review

3 Citations (Scopus)

Abstract

There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.

Original language	English
Title of host publication	Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023
Editors	Mary Nurminen, Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartin, Mikel Forcada, Maja Popovic, Carolina Scarton, Helena Moniz
Publisher	European Association for Machine Translation
Pages	39-45
Number of pages	7
ISBN (Electronic)	9789520329471
Publication status	Published - 2023

Publication series

Name	Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023

Cite this

Oliver, A., & Álvarez, S. (2023). Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. In M. Nurminen, M. Nurminen, J. Brenner, M. Koponen, S. Latomaa, M. Mikhailov, F. Schierl, T. Ranasinghe, E. Vanmassenhove, S. A. Vidal, N. Aranberri, M. Nunziatini, C. P. Escartin, M. Forcada, M. Popovic, C. Scarton, & H. Moniz (Eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023 (pp. 39-45). (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023). European Association for Machine Translation.

Oliver, Antoni ; Álvarez, Sergi. / Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. editor / Mary Nurminen ; Mary Nurminen ; Judith Brenner ; Maarit Koponen ; Sirkku Latomaa ; Mikhail Mikhailov ; Frederike Schierl ; Tharindu Ranasinghe ; Eva Vanmassenhove ; Sergi Alvarez Vidal ; Nora Aranberri ; Mara Nunziatini ; Carla Parra Escartin ; Mikel Forcada ; Maja Popovic ; Carolina Scarton ; Helena Moniz. European Association for Machine Translation, 2023. pp. 39-45 (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023).

@inbook{48240721ab374cf9a292a7612e8bd55b,

title = "Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training",

abstract = "There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.",

author = "Antoni Oliver and Sergi {\'A}lvarez",

note = "Publisher Copyright: {\textcopyright} 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC-BY-ND.",

year = "2023",

language = "English",

series = "Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023",

publisher = "European Association for Machine Translation",

pages = "39--45",

editor = "Mary Nurminen and Mary Nurminen and Judith Brenner and Maarit Koponen and Sirkku Latomaa and Mikhail Mikhailov and Frederike Schierl and Tharindu Ranasinghe and Eva Vanmassenhove and Vidal, {Sergi Alvarez} and Nora Aranberri and Mara Nunziatini and Escartin, {Carla Parra} and Mikel Forcada and Maja Popovic and Carolina Scarton and Helena Moniz",

booktitle = "Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023",

}

Oliver, A & Álvarez, S 2023, Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. in M Nurminen, M Nurminen, J Brenner, M Koponen, S Latomaa, M Mikhailov, F Schierl, T Ranasinghe, E Vanmassenhove, SA Vidal, N Aranberri, M Nunziatini, CP Escartin, M Forcada, M Popovic, C Scarton & H Moniz (eds), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, European Association for Machine Translation, pp. 39-45.

Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. / Oliver, Antoni; Álvarez, Sergi.
Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. ed. / Mary Nurminen; Mary Nurminen; Judith Brenner; Maarit Koponen; Sirkku Latomaa; Mikhail Mikhailov; Frederike Schierl; Tharindu Ranasinghe; Eva Vanmassenhove; Sergi Alvarez Vidal; Nora Aranberri; Mara Nunziatini; Carla Parra Escartin; Mikel Forcada; Maja Popovic; Carolina Scarton; Helena Moniz. European Association for Machine Translation, 2023. p. 39-45 (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023).

Research output: Chapter in Book › Chapter › Research › peer-review

TY - CHAP

T1 - Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

AU - Oliver, Antoni

AU - Álvarez, Sergi

PY - 2023

Y1 - 2023

N2 - There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.

AB - There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish–Asturian and Spanish–Catalan CCMatrix corpus is presented.

UR - http://www.scopus.com/inward/record.url?scp=85184805763&partnerID=8YFLogxK

M3 - Chapter

AN - SCOPUS:85184805763

T3 - Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023

SP - 39

EP - 45

BT - Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023

A2 - Nurminen, Mary

A2 - Brenner, Judith

A2 - Koponen, Maarit

A2 - Latomaa, Sirkku

A2 - Mikhailov, Mikhail

A2 - Schierl, Frederike

A2 - Ranasinghe, Tharindu

A2 - Vanmassenhove, Eva

A2 - Vidal, Sergi Alvarez

A2 - Aranberri, Nora

A2 - Nunziatini, Mara

A2 - Escartin, Carla Parra

A2 - Forcada, Mikel

A2 - Popovic, Maja

A2 - Scarton, Carolina

A2 - Moniz, Helena

PB - European Association for Machine Translation

ER -

Oliver A, Álvarez S. Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. In Nurminen M, Nurminen M, Brenner J, Koponen M, Latomaa S, Mikhailov M, Schierl F, Ranasinghe T, Vanmassenhove E, Vidal SA, Aranberri N, Nunziatini M, Escartin CP, Forcada M, Popovic M, Scarton C, Moniz H, editors, Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023. European Association for Machine Translation. 2023. p. 39-45. (Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023).

Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training

Abstract

Publication series

Other files and links

Fingerprint

Cite this