1) Processing Pipeline
This page documents the operational flow implemented in the shared codebase:
- Input The user pastes an Arabic passage (~300 tokens by default).
- Processing & Cleaning Noise removal and normalization (surfaces a light UI for proclitic splits such as و/ف/ب/ك/ل/س).
- Tokenization Robust Unicode tokenization into words/graphemes.
- Confidence Gate τ A decision node dispatches tokens to:
- Rule-based system (deterministic) – classic scripts loaded first: POS, patterns, roots, gender, laa, number, lemma, case, definiteness, particle/adverb types, syllables.
- ML system (probabilistic) – an overlay that offers suggestions and learning endpoints.
- User Evaluation A human reviews/edits outputs within the lab panels; edits are tracked and can be fed back to ML.
- Confirmed Output Export to JSON/CSV via the exporter.
- Feedback Loop Confirmed rows are sent to feedback and opportunistically trained for continuous learning.
2) What ArMorph aims to do
ArMorph provides a research-grade, dialect-aware Arabic morphological/phonological workbench. It combines interpretable linguistic rules (transparency, reproducibility) with ML-based assistive predictions (coverage, generalization). A human verifier remains central (human-in-the-loop AI), ensuring reliability and generating gold data that improves the models over time.
3) ML Overlay & API
The floating toolbar (ظ-style) exposes:
- Capability + gate state.
- Per-cell predictions for targets such as ipa, pos, definiteness, case, gender, number, laa, pattern, root, lemma, particle_type, adverb_type, syllable, stress, mora, split_decision.
- Feedback endpoint (authorized when enabled) to append validated rows.
- On-demand training; supports dialect scoping (e.g., MSA, HIJAZI).
4) Export Formats
CSV (flat, analysis-friendly)
index,token,line_index,position_text,sentence_index,token_position_in_sentence, ipa,pos,laa_type,number_value,case_value,definiteness_value, gender,pattern_value,root,root_count, particle_type_value,particle_type_free, adverb_type_value,adverb_type_free, lemma_value, syllable_value,syllable_count,stress_value,mora_value, dialect_value,dialect_label,dialect_other_text,dialect_code
Notes:
- syllable_count is computed from dot-separated syllable value.
- Dialect is stored as both human labels and a normalized code (e.g., MSA, HIJAZI).
Analytics CSV (coded)
schema_version,index,token,line_index,position_text,sentence_index,token_position_in_sentence, ipa, pos,pos_code,laa_type,laa_code,number_value,number_code, case_value,definiteness_value,definiteness_code, gender,gender_code, pattern_value,pattern_code,pattern_code_source,pattern_family, root_value,root_code,root_count, particle_type_value,particle_type_code,particle_type_free, adverb_type_value,adverb_type_code,adverb_type_free, lemma_value,lemma_code, syllable_value,syllable_count,stress_value,mora_value, dialect_value,dialect_label,dialect_other_text,dialect_code
JSON
{
"count": <int>,
"tokens": [
{
"index": 1,
"token": "وَخَالِد",
"ipa": "wa xaːlid",
"pos": "PART+PROPN",
"laa_type": "",
"number_value": "",
"case_value": "ACC",
"definiteness_value": "DEF",
"gender": "MASC",
"pattern_value": "CaCaCiC",
"root": "خ ل د",
"root_count": 3,
"particle_type_value": "COORD",
"adverb_type_value": "",
"lemma_value": "خالد",
"syllable_value": "wa.xaa.lid",
"stress_value": "xáa",
"mora_value": "μμ…",
"dialect_value": "HIJAZI",
"dialect_code": "HIJAZI"
}
]
}