DacteloMilestone: Published:
By the Dactelo team at Taktek.
Give language work relevant material
Legal documents use recurring expressions and specialised vocabulary. A language component working on Lebanese rulings needs material from that setting. The prepared corpus contains 471,911 deduplicated documents and 72,289,769 words. These are counts of text material prepared for the work. They are not counts of corrected handwriting images or completed contributions.
The preparation removes exact duplicates after normalising whitespace. That stops repeated copies of the same text from being counted as separate documents under that rule. It does not identify every near-duplicate or verify every sentence. The useful result is a defined collection that can be inspected and used in later language experiments.
Keep a plausible sentence from becoming an answer
Context can suggest which words often appear together. That may help organise candidate readings, especially where a legal term is unfamiliar. But a common phrase can also pull a reader toward the wrong word. A system that completes the sentence from habit could produce fluent text that differs from the marks on the page.
For that reason, the legal text corpus and the handwriting reference serve different purposes. The corpus supplies language material. A reference transcription must be checked against its own source image. Names, numbers, dates and unusual wording need the same attention even when they interrupt the phrase the language component would expect.
Leave evaluation rulings out
The corpus preparation excludes rulings reserved for the locked handwriting evaluation sample. That helps keep the test material separate from material prepared for language work. Holding back whole rulings matters because neighbouring lines can share wording and handwriting. A later evaluation needs to test reading on material that has been kept aside.
Completing this corpus did not complete a trained handwriting reader, establish rights for every possible downstream use, or produce an accuracy score. It completed one preparation step. To understand how that step fits the broader work, read about the legal-handwriting project and compare the reviewed examples with their images. That is where a candidate reading has to meet the source.