Character-Based Machine Learning vs. Language Modeling for Diacritics Restoration

Abstract

In this research we compare two approaches, in particular, character-based machine learning and language-modeling and offer the best solution for the diacritization problem solving. Parameters of tested approaches (i.e., a huge variety of feature types for the character-based method and a value n for the n-gram language-modeling method) were tuned to achieve the highest possible accuracy. Despite the main focus is on the Lithuanian language, we posit that obtained findings can also be applied to other, similar (Latvian or Slavic) languages.

During experiments we measured the performance of approaches on 10 domains (including normative texts and non-normative Internet comments). The best results reaching ~99.5% and ~98.4% of the accuracy on characters and words, respectively, were achieved with the tri-gram language modeling method. It outperformed the character-based machine learning approach with an optimal composed feature set by ~1.4% and ~3.8% of the accuracy on characters and words, respectively.

DOI: http://dx.doi.org/10.5755/j01.itc.46.4.18066