Middle-Eastern Language diacritization is useful for several practical business cases like text to speech or Romanization of texts or scripts.
As of now, this library supports Hebrew and Arabic.
This repository contains everything to train a diacritization model in Python and run it in Python and Ruby.
This library was built for the Interscript project (at GitHub).
Diacritization strategy is following several steps with at heart a deep learning model:
-
text preprocessing
-
neural networks model prediction
-
text postprocessing
This repository contains:
-
lib is the Ruby library using NNet model in ONNX format.
-
docs contains an application focused summary of latest (2021-06) relevant papers and solutions.
-
-
A neural network solution for automatised diacritization based on the work of almodhfer, from which we overtook the baseline and more advanced and efficient CBHG models only. This very recent solution allows for efficient predictions on CPU’s with a reasonable sized model.
-
PyTorch to ONNX conversion of PyTorch to ONNX format
-
Strings Pre-/Post-processing, also from almodhfer
-
-
tests and benchmarking utilities, allowing to compare with other implementations.
-
tests are taken from diacritization benchmarking
-
we have added own, realistic datasets for the problem of diacritization
-
-
models-data directory to store models and embeddings in various formats
A Rababa is an antique string instrument.
In a similar fashion that a Rababa produces melody from a simple strings and pieces of wood, our library and diacritization gives a whole palette of colour and meanings to arabic scripts.
We are working on the following improvements:
-
Enhancing architecture and encoding
-
Enhance datasets to improve models
Rababa is copyright (c) 2021-2025, Ribose Inc. All rights reserved.
Rababa is licensed under the BSD-2 Clause license. See the LICENSE.adoc file for details.
The Rababa team would like to express their appreciation for the open-source work of these authors and researchers:
-
M. A. H. Madhfar and A. M. Qamar for their work on effective deep learning models for automatic diacritization of Arabic text
-
Taha Zerrouki for the original Tashkeela dataset
The team acknowledges the contributions of these authors and researchers in the field of Arabic diacritization and recognizes the importance of their work in advancing the state of the art in this area.
Rababa does not redistribute any code or data from these attributed sources. Any redistribution of these attributed sources should be done in accordance with their respective licenses.
Rababa is not responsible for any issues that may arise from the use of these external sources. These sources are provided for reference purposes only, and their use is at the user’s own risk.
The neural network solution for Arabic diacritization is based on the work of M. A. H. Madhfar:
-
Repository: https://github.com/almodhfer/Arabic_Diacritization
-
License: MIT License
-
Citation: M. A. H. Madhfar and A. M. Qamar, "Effective Deep Learning Models for Automatic Diacritization of Arabic Text," in IEEE Access, vol. 9, pp. 273-288, 2021, doi: 10.1109/ACCESS.2020.3041676.
The Tashkeela dataset used for training is provided under GPL v2 license:
-
Original dataset by Taha Zerrouki: https://sourceforge.net/projects/tashkeela/
-
Processed dataset by Hamza Abbad: https://sourceforge.net/projects/tashkeela-processed/
-
License: GPL v2