you are training from Tessearact default text data or your own collected
text data?
On Thursday, 14 September, 2023 at 12:19:53 am UTC+6 desal...@gmail.com
wrote:
> I now get to 200000 iterations; and the error rate is stuck at 0.46. The
> result is absolutely trash: nowhere close to the default/Ray's training.
>
> On Wednesday, September 13, 2023 at 2:47:05 PM UTC+3 mdalihu...@gmail.com
> wrote:
>
>>
>> after Tesseact recognizes text from images. then you can apply regex to
>> replace the wrong word with to correct word.
>> I'm not familiar with paddleOcr and scanTailor also.
>>
>> On Wednesday, 13 September, 2023 at 5:06:12 pm UTC+6 desal...@gmail.com
>> wrote:
>>
>>> At what stage are you doing the regex replacement?
>>> My process has been: Scan (tif)--> ScanTailor --> Tesseract --> pdf
>>>
>>> >EasyOCR I think is best for ID cards or something like that image
>>> process. but document images like books, here Tesseract is better than
>>> EasyOCR.
>>>
>>> How about paddleOcr?, are you familiar with it?
>>>
>>> On Wednesday, September 13, 2023 at 1:45:54 PM UTC+3
>>> mdalihu...@gmail.com wrote:
>>>
>>>> I know what you mean. but in some cases, it helps me. I have faced
>>>> specific characters and words are always not recognized by Tesseract. That
>>>> way I use these regex to replace those characters and words if those
>>>> characters are incorrect.
>>>>
>>>> see what I have done:
>>>>
>>>> " ী": "ী",
>>>> " ্": " ",
>>>> " ে": " ",
>>>> জ্া: "জা",
>>>> " ": " ",
>>>> " ": " ",
>>>> " ": " ",
>>>> "্প": " ",
>>>> " য": "র্য",
>>>> য: "য",
>>>> " া": "া",
>>>> আা: "আ",
>>>> ম্ি: "মি",
>>>> স্ু: "সু",
>>>> "হূ ": "হূ",
>>>> " ণ": "ণ",
>>>> র্্: "র",
>>>> "চিন্ত ": "চিন্তা ",
>>>> ন্া: "না",
>>>> "সম ূর্ন": "সম্পূর্ণ",
>>>> On Wednesday, 13 September, 2023 at 4:18:22 pm UTC+6 desal...@gmail.com
>>>> wrote:
>>>>
>>>>> The problem for regex is that Tesseract is not consistent in its
>>>>> replacement.
>>>>> Think of the original training of English data doesn't contain the
>>>>> letter /u/. What does Tesseract do when it faces /u/ in actual
>>>>> processing??
>>>>> In some cases, it replaces it with closely similar letters such as /v/
>>>>> and /w/. In other cases, it completely removes it. That is what is
>>>>> happening with my case. Those characters re sometimes completely removed;
>>>>> other times, they are replaced by closely resembling characters. Because
>>>>> of
>>>>> this inconsistency, applying regex is very difficult.
>>>>>
>>>>> On Wednesday, September 13, 2023 at 1:02:01 PM UTC+3
>>>>> mdalihu...@gmail.com wrote:
>>>>>
>>>>>> if Some specific characters or words are always missing from the OCR
>>>>>> result. then you can apply logic with the Regular expressions method on
>>>>>> your applications. After OCR, these specific characters or words will be
>>>>>> replaced by current characters or words that you defined in your
>>>>>> applications by Regular expressions. it can be done in some major
>>>>>> problems.
>>>>>>
>>>>>> On Wednesday, 13 September, 2023 at 3:51:29 pm UTC+6
>>>>>> desal...@gmail.com wrote:
>>>>>>
>>>>>>> The characters are getting missed, even after fine-tuning.
>>>>>>> I never made any progress. I tried many different ways. Some
>>>>>>> specific characters are always missing from the OCR result.
>>>>>>>
>>>>>>> On Wednesday, September 13, 2023 at 12:49:20 PM UTC+3
>>>>>>> mdalihu...@gmail.com wrote:
>>>>>>>
>>>>>>>> EasyOCR I think is best for ID cards or something like that image
>>>>>>>> process. but document images like books, here Tesseract is better than
>>>>>>>> EasyOCR. Even I didn't use EasyOCR. you can try it.
>>>>>>>>
>>>>>>>> I have added words of dictionaries but the result is the same.
>>>>>>>>
>>>>>>>> what kind of problem you have faced in fine-tuning in few new
>>>>>>>> characters as you said (*but, I failed in every possible way to
>>>>>>>> introduce a few new characters into the database.)*
>>>>>>>> On Wednesday, 13 September, 2023 at 3:33:48 pm UTC+6
>>>>>>>> desal...@gmail.com wrote:
>>>>>>>>
>>>>>>>>> Yes, we are new to this. I find the instructions (the manual) very
>>>>>>>>> hard to follow. The video you linked above was really helpful to get
>>>>>>>>> started. My plan at the beginning was to fine tune the existing
>>>>>>>>> .traineddata. But, I failed in every possible way to introduce a few
>>>>>>>>> new
>>>>>>>>> characters into the database. That is why I started from scratch.
>>>>>>>>>
>>>>>>>>> Sure, I will follow Lorenzo's suggestion: will run more the
>>>>>>>>> iterations, and see if I can improve.
>>>>>>>>>
>>>>>>>>> Another areas we need to explore is usage of dictionaries
>>>>>>>>> actually. May be adding millions of words into the dictionary could
>>>>>>>>> help
>>>>>>>>> Tesseract. I don't have millions of words; but I am looking into some
>>>>>>>>> corpus to get more words into the dictionary.
>>>>>>>>>
>>>>>>>>> If this all fails, EasyOCR (and probably other similar open-source
>>>>>>>>> packages) is probably our next option to try on. Sure, sharing
>>>>>>>>> our experiences will be helpful. I will let you know if I made good
>>>>>>>>> progresses in any of these options.
>>>>>>>>> On Wednesday, September 13, 2023 at 12:19:48 PM UTC+3
>>>>>>>>> mdalihu...@gmail.com wrote:
>>>>>>>>>
>>>>>>>>>> How is your training going for Bengali? It was nearly good but I
>>>>>>>>>> faced space problems between two words, some words are spaces but
>>>>>>>>>> most of
>>>>>>>>>> them have no space. I think is problem is in the dataset but I use
>>>>>>>>>> the
>>>>>>>>>> default training dataset from Tesseract which is used in Ben That
>>>>>>>>>> way I am
>>>>>>>>>> confused so I have to explore more. by the way, you can try as
>>>>>>>>>> Lorenzo
>>>>>>>>>> Blz said. Actually training from scratch is harder than
>>>>>>>>>> fine-tuning. so you can use different datasets to explore. if you
>>>>>>>>>> succeed.
>>>>>>>>>> please let me know how you have done this whole process. I'm also
>>>>>>>>>> new in
>>>>>>>>>> this field.
>>>>>>>>>> On Wednesday, 13 September, 2023 at 1:13:43 pm UTC+6
>>>>>>>>>> desal...@gmail.com wrote:
>>>>>>>>>>
>>>>>>>>>>> How is your training going for Bengali?
>>>>>>>>>>> I have been trying to train from scratch. I made about 64,000
>>>>>>>>>>> lines of text (which produced about 255,000 files, in the end) and
>>>>>>>>>>> run the
>>>>>>>>>>> training for 150,000 iterations; getting 0.51 training error rate.
>>>>>>>>>>> I was
>>>>>>>>>>> hopping to get reasonable accuracy. Unfortunately, when I run the
>>>>>>>>>>> OCR
>>>>>>>>>>> using .traineddata, the accuracy is absolutely terrible. Do you
>>>>>>>>>>> think I
>>>>>>>>>>> made some mistakes, or that is an expected result?
>>>>>>>>>>>
>>>>>>>>>>> On Tuesday, September 12, 2023 at 11:15:25 PM UTC+3
>>>>>>>>>>> mdalihu...@gmail.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yes, he doesn't mention all fonts but only one font. That way
>>>>>>>>>>>> he didn't use *MODEL_NAME in a separate **script **file script
>>>>>>>>>>>> I think.*
>>>>>>>>>>>>
>>>>>>>>>>>> Actually, here we teach all *tif, gt.txt, and .box files *which
>>>>>>>>>>>> are created by *MODEL_NAME I mean **eng, ben, oro flag or
>>>>>>>>>>>> language code *because when we first create *tif, gt.txt, and
>>>>>>>>>>>> .box files, *every file starts by *MODEL_NAME*. This
>>>>>>>>>>>> *MODEL_NAME* we selected on the training script for looping
>>>>>>>>>>>> each tif, gt.txt, and .box files which are created by
>>>>>>>>>>>> *MODEL_NAME.*
>>>>>>>>>>>>
>>>>>>>>>>>> On Tuesday, 12 September, 2023 at 9:42:13 pm UTC+6
>>>>>>>>>>>> desal...@gmail.com wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, I am familiar with the video and have set up the folder
>>>>>>>>>>>>> structure as you did. Indeed, I have tried a number of
>>>>>>>>>>>>> fine-tuning with a
>>>>>>>>>>>>> single font following Gracia's video. But, your script is much
>>>>>>>>>>>>> better
>>>>>>>>>>>>> because supports multiple fonts. The whole improvement you made
>>>>>>>>>>>>> is
>>>>>>>>>>>>> brilliant; and very useful. It is all working for me.
>>>>>>>>>>>>> The only part that I didn't understand is the trick you used
>>>>>>>>>>>>> in your tesseract_train.py script. You see, I have been doing
>>>>>>>>>>>>> exactly to
>>>>>>>>>>>>> you did except this script.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The scripts seems to have the trick of sending/teaching each
>>>>>>>>>>>>> of the fonts (iteratively) into the model. The script I have been
>>>>>>>>>>>>> using
>>>>>>>>>>>>> (which I get from Garcia) doesn't mention font at all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> *TESSDATA_PREFIX=../tesseract/tessdata make training
>>>>>>>>>>>>> MODEL_NAME=oro TESSDATA=../tesseract/tessdata
>>>>>>>>>>>>> MAX_ITERATIONS=10000*
>>>>>>>>>>>>> Does it mean that my model does't train the fonts (even if the
>>>>>>>>>>>>> fonts have been included in the splitting process, in the other
>>>>>>>>>>>>> script)?
>>>>>>>>>>>>> On Monday, September 11, 2023 at 10:54:08 AM UTC+3
>>>>>>>>>>>>> mdalihu...@gmail.com wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *import subprocess# List of font namesfont_names = ['ben']for
>>>>>>>>>>>>>> font in font_names: command =
>>>>>>>>>>>>>> f"TESSDATA_PREFIX=../tesseract/tessdata
>>>>>>>>>>>>>> make training MODEL_NAME={font} START_MODEL=ben
>>>>>>>>>>>>>> TESSDATA=../tesseract/tessdata MAX_ITERATIONS=10000"*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * subprocess.run(command, shell=True) 1 . This command is
>>>>>>>>>>>>>> for training data that I have named '*tesseract_training*.py'
>>>>>>>>>>>>>> inside tesstrain folder.*
>>>>>>>>>>>>>> *2. root directory means your main training folder and inside
>>>>>>>>>>>>>> it as like langdata, tessearact, tesstrain folders. if you see
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> tutorial *https://www.youtube.com/watch?v=KE4xEzFGSU8
>>>>>>>>>>>>>> you will understand better the folder structure. only I
>>>>>>>>>>>>>> created tesseract_training.py in tesstrain folder for training
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> FontList.py file is the main path as *like langdata,
>>>>>>>>>>>>>> tessearact, tesstrain, and *split_training_text.py.
>>>>>>>>>>>>>> 3. first of all you have to put all fonts in your Linux fonts
>>>>>>>>>>>>>> folder. /usr/share/fonts/ then run: sudo apt update
>>>>>>>>>>>>>> then sudo fc-cache -fv
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> after that, you have to add the exact font's name in
>>>>>>>>>>>>>> FontList.py file like me.
>>>>>>>>>>>>>> I have added two pic my folder structure. first is main
>>>>>>>>>>>>>> structure pic and the second is the Colopse tesstrain folder.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I[image: Screenshot 2023-09-11 134947.png][image: Screenshot
>>>>>>>>>>>>>> 2023-09-11 135014.png]
>>>>>>>>>>>>>> On Monday, 11 September, 2023 at 12:50:03 pm UTC+6
>>>>>>>>>>>>>> desal...@gmail.com wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you so much for putting out these brilliant scripts.
>>>>>>>>>>>>>>> They make the process much more efficient.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have one more question on the other script that you use to
>>>>>>>>>>>>>>> train.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *import subprocess# List of font namesfont_names =
>>>>>>>>>>>>>>> ['ben']for font in font_names: command =
>>>>>>>>>>>>>>> f"TESSDATA_PREFIX=../tesseract/tessdata make training
>>>>>>>>>>>>>>> MODEL_NAME={font}
>>>>>>>>>>>>>>> START_MODEL=ben TESSDATA=../tesseract/tessdata
>>>>>>>>>>>>>>> MAX_ITERATIONS=10000"*
>>>>>>>>>>>>>>> * subprocess.run(command, shell=True) *
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you have the name of fonts listed in file in the
>>>>>>>>>>>>>>> same/root directory?
>>>>>>>>>>>>>>> How do you setup the names of the fonts in the file, if you
>>>>>>>>>>>>>>> don't mind sharing it?
>>>>>>>>>>>>>>> On Monday, September 11, 2023 at 4:27:27 AM UTC+3
>>>>>>>>>>>>>>> mdalihu...@gmail.com wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You can use the new script below. it's better than the
>>>>>>>>>>>>>>>> previous two scripts. You can create *tif, gt.txt, and
>>>>>>>>>>>>>>>> .box files *by multiple fonts and also use breakpoint if
>>>>>>>>>>>>>>>> vs code close or anything during creating *tif, gt.txt,
>>>>>>>>>>>>>>>> and .box files *then you can checkpoint to navigate where
>>>>>>>>>>>>>>>> you close vs code.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> command for *tif, gt.txt, and .box files *
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> import os
>>>>>>>>>>>>>>>> import random
>>>>>>>>>>>>>>>> import pathlib
>>>>>>>>>>>>>>>> import subprocess
>>>>>>>>>>>>>>>> import argparse
>>>>>>>>>>>>>>>> from FontList import FontList
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> def create_training_data(training_text_file, font_list,
>>>>>>>>>>>>>>>> output_directory, start_line=None, end_line=None):
>>>>>>>>>>>>>>>> lines = []
>>>>>>>>>>>>>>>> with open(training_text_file, 'r') as input_file:
>>>>>>>>>>>>>>>> lines = input_file.readlines()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if not os.path.exists(output_directory):
>>>>>>>>>>>>>>>> os.mkdir(output_directory)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if start_line is None:
>>>>>>>>>>>>>>>> start_line = 0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if end_line is None:
>>>>>>>>>>>>>>>> end_line = len(lines) - 1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for font_name in font_list.fonts:
>>>>>>>>>>>>>>>> for line_index in range(start_line, end_line + 1):
>>>>>>>>>>>>>>>> line = lines[line_index].strip()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> training_text_file_name = pathlib.Path(
>>>>>>>>>>>>>>>> training_text_file).stem
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> line_serial = f"{line_index:d}"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> line_gt_text = os.path.join(output_directory, f
>>>>>>>>>>>>>>>> '{training_text_file_name}_{line_serial}_{
>>>>>>>>>>>>>>>> font_name.replace(" ", "_")}.gt.txt')
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> with open(line_gt_text, 'w') as output_file:
>>>>>>>>>>>>>>>> output_file.writelines([line])
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> file_base_name = f'{training_text_file_name}_{
>>>>>>>>>>>>>>>> line_serial}_{font_name.replace(" ", "_")}'
>>>>>>>>>>>>>>>> subprocess.run([
>>>>>>>>>>>>>>>> 'text2image',
>>>>>>>>>>>>>>>> f'--font={font_name}',
>>>>>>>>>>>>>>>> f'--text={line_gt_text}',
>>>>>>>>>>>>>>>> f'--outputbase={output_directory}/{
>>>>>>>>>>>>>>>> file_base_name}',
>>>>>>>>>>>>>>>> '--max_pages=1',
>>>>>>>>>>>>>>>> '--strip_unrenderable_words',
>>>>>>>>>>>>>>>> '--leading=36',
>>>>>>>>>>>>>>>> '--xsize=3600',
>>>>>>>>>>>>>>>> '--ysize=330',
>>>>>>>>>>>>>>>> '--char_spacing=1.0',
>>>>>>>>>>>>>>>> '--exposure=0',
>>>>>>>>>>>>>>>> '--unicharset_file=langdata/eng.unicharset'
>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>> ])
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if __name__ == "__main__":
>>>>>>>>>>>>>>>> parser = argparse.ArgumentParser()
>>>>>>>>>>>>>>>> parser.add_argument('--start', type=int, help='Starting
>>>>>>>>>>>>>>>> line count (inclusive)')
>>>>>>>>>>>>>>>> parser.add_argument('--end', type=int, help='Ending
>>>>>>>>>>>>>>>> line count (inclusive)')
>>>>>>>>>>>>>>>> args = parser.parse_args()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> training_text_file = 'langdata/eng.training_text'
>>>>>>>>>>>>>>>> output_directory = 'tesstrain/data/eng-ground-truth'
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> font_list = FontList()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> create_training_data(training_text_file, font_list,
>>>>>>>>>>>>>>>> output_directory, args.start, args.end)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Then create a file called "FontList" in the root directory
>>>>>>>>>>>>>>>> and paste it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> class FontList:
>>>>>>>>>>>>>>>> def __init__(self):
>>>>>>>>>>>>>>>> self.fonts = [
>>>>>>>>>>>>>>>> "Gerlick"
>>>>>>>>>>>>>>>> "Sagar Medium",
>>>>>>>>>>>>>>>> "Ekushey Lohit Normal",
>>>>>>>>>>>>>>>> "Charukola Round Head Regular, weight=433",
>>>>>>>>>>>>>>>> "Charukola Round Head Bold, weight=443",
>>>>>>>>>>>>>>>> "Ador Orjoma Unicode",
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> then import in the above code,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *for breakpoint command:*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sudo python3 split_training_text.py --start 0 --end 11
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> change checkpoint according to you --start 0 --end 11.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *and training checkpoint as you know already.*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Monday, 11 September, 2023 at 1:22:34 am UTC+6
>>>>>>>>>>>>>>>> desal...@gmail.com wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi mhalidu,
>>>>>>>>>>>>>>>>> the script you posted here seems much more extensive than
>>>>>>>>>>>>>>>>> you posted before:
>>>>>>>>>>>>>>>>> https://groups.google.com/d/msgid/tesseract-ocr/0e2880d9-64c0-4659-b497-902a5747caf4n%40googlegroups.com
>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have been using your earlier script. It is magical. How
>>>>>>>>>>>>>>>>> is this one different from the earlier one?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you for posting these scripts, by the way. It has
>>>>>>>>>>>>>>>>> saved my countless hours; by running multiple fonts in one
>>>>>>>>>>>>>>>>> sweep. I was not
>>>>>>>>>>>>>>>>> able to find any instruction on how to train for multiple
>>>>>>>>>>>>>>>>> fonts. The
>>>>>>>>>>>>>>>>> official manual is also unclear. YOUr script helped me to get
>>>>>>>>>>>>>>>>> started.
>>>>>>>>>>>>>>>>> On Wednesday, August 9, 2023 at 11:00:49 PM UTC+3
>>>>>>>>>>>>>>>>> mdalihu...@gmail.com wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ok, I will try as you said.
>>>>>>>>>>>>>>>>>> one more thing, what's the role of the trained_text lines
>>>>>>>>>>>>>>>>>> will be? I have seen Bengali text are long words of lines.
>>>>>>>>>>>>>>>>>> so I wanna know
>>>>>>>>>>>>>>>>>> how many words or characters will be the better choice for
>>>>>>>>>>>>>>>>>> the train?
>>>>>>>>>>>>>>>>>> and '--xsize=3600','--ysize=350', will be according to
>>>>>>>>>>>>>>>>>> words of lines?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thursday, 10 August, 2023 at 1:10:14 am UTC+6 shree
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Include the default fonts also in your fine-tuning list
>>>>>>>>>>>>>>>>>>> of fonts and see if that helps.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Aug 9, 2023, 2:27 PM Ali hussain <
>>>>>>>>>>>>>>>>>>> mdalihu...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have trained some new fonts by fine-tune methods for
>>>>>>>>>>>>>>>>>>>> the Bengali language in Tesseract 5 and I have used all
>>>>>>>>>>>>>>>>>>>> official
>>>>>>>>>>>>>>>>>>>> trained_text and tessdata_best and other things also.
>>>>>>>>>>>>>>>>>>>> everything is good
>>>>>>>>>>>>>>>>>>>> but the problem is the default font which was trained
>>>>>>>>>>>>>>>>>>>> before that does not
>>>>>>>>>>>>>>>>>>>> convert text like prev but my new fonts work well. I don't
>>>>>>>>>>>>>>>>>>>> understand why
>>>>>>>>>>>>>>>>>>>> it's happening. I share code based to understand what
>>>>>>>>>>>>>>>>>>>> going on.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> *codes for creating tif, gt.txt, .box files:*
>>>>>>>>>>>>>>>>>>>> import os
>>>>>>>>>>>>>>>>>>>> import random
>>>>>>>>>>>>>>>>>>>> import pathlib
>>>>>>>>>>>>>>>>>>>> import subprocess
>>>>>>>>>>>>>>>>>>>> import argparse
>>>>>>>>>>>>>>>>>>>> from FontList import FontList
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> def read_line_count():
>>>>>>>>>>>>>>>>>>>> if os.path.exists('line_count.txt'):
>>>>>>>>>>>>>>>>>>>> with open('line_count.txt', 'r') as file:
>>>>>>>>>>>>>>>>>>>> return int(file.read())
>>>>>>>>>>>>>>>>>>>> return 0
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> def write_line_count(line_count):
>>>>>>>>>>>>>>>>>>>> with open('line_count.txt', 'w') as file:
>>>>>>>>>>>>>>>>>>>> file.write(str(line_count))
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> def create_training_data(training_text_file, font_list,
>>>>>>>>>>>>>>>>>>>> output_directory, start_line=None, end_line=None):
>>>>>>>>>>>>>>>>>>>> lines = []
>>>>>>>>>>>>>>>>>>>> with open(training_text_file, 'r') as input_file:
>>>>>>>>>>>>>>>>>>>> for line in input_file.readlines():
>>>>>>>>>>>>>>>>>>>> lines.append(line.strip())
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> if not os.path.exists(output_directory):
>>>>>>>>>>>>>>>>>>>> os.mkdir(output_directory)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> random.shuffle(lines)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> if start_line is None:
>>>>>>>>>>>>>>>>>>>> line_count = read_line_count() # Set the
>>>>>>>>>>>>>>>>>>>> starting line_count from the file
>>>>>>>>>>>>>>>>>>>> else:
>>>>>>>>>>>>>>>>>>>> line_count = start_line
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> if end_line is None:
>>>>>>>>>>>>>>>>>>>> end_line_count = len(lines) - 1 # Set the
>>>>>>>>>>>>>>>>>>>> ending line_count
>>>>>>>>>>>>>>>>>>>> else:
>>>>>>>>>>>>>>>>>>>> end_line_count = min(end_line, len(lines) - 1)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> for font in font_list.fonts: # Iterate through
>>>>>>>>>>>>>>>>>>>> all the fonts in the font_list
>>>>>>>>>>>>>>>>>>>> font_serial = 1
>>>>>>>>>>>>>>>>>>>> for line in lines:
>>>>>>>>>>>>>>>>>>>> training_text_file_name = pathlib.Path(
>>>>>>>>>>>>>>>>>>>> training_text_file).stem
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> # Generate a unique serial number for each
>>>>>>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>>>>> line_serial = f"{line_count:d}"
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> # GT (Ground Truth) text filename
>>>>>>>>>>>>>>>>>>>> line_gt_text = os.path.join(
>>>>>>>>>>>>>>>>>>>> output_directory, f'{training_text_file_name}_{
>>>>>>>>>>>>>>>>>>>> line_serial}.gt.txt')
>>>>>>>>>>>>>>>>>>>> with open(line_gt_text, 'w') as
>>>>>>>>>>>>>>>>>>>> output_file:
>>>>>>>>>>>>>>>>>>>> output_file.writelines([line])
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> # Image filename
>>>>>>>>>>>>>>>>>>>> file_base_name = f'ben_{line_serial}' #
>>>>>>>>>>>>>>>>>>>> Unique filename for each font
>>>>>>>>>>>>>>>>>>>> subprocess.run([
>>>>>>>>>>>>>>>>>>>> 'text2image',
>>>>>>>>>>>>>>>>>>>> f'--font={font}',
>>>>>>>>>>>>>>>>>>>> f'--text={line_gt_text}',
>>>>>>>>>>>>>>>>>>>> f'--outputbase={output_directory}/{
>>>>>>>>>>>>>>>>>>>> file_base_name}',
>>>>>>>>>>>>>>>>>>>> '--max_pages=1',
>>>>>>>>>>>>>>>>>>>> '--strip_unrenderable_words',
>>>>>>>>>>>>>>>>>>>> '--leading=36',
>>>>>>>>>>>>>>>>>>>> '--xsize=3600',
>>>>>>>>>>>>>>>>>>>> '--ysize=350',
>>>>>>>>>>>>>>>>>>>> '--char_spacing=1.0',
>>>>>>>>>>>>>>>>>>>> '--exposure=0',
>>>>>>>>>>>>>>>>>>>> '
>>>>>>>>>>>>>>>>>>>> --unicharset_file=langdata/ben.unicharset',
>>>>>>>>>>>>>>>>>>>> ])
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> line_count += 1
>>>>>>>>>>>>>>>>>>>> font_serial += 1
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> # Reset font_serial for the next font iteration
>>>>>>>>>>>>>>>>>>>> font_serial = 1
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> write_line_count(line_count) # Update the
>>>>>>>>>>>>>>>>>>>> line_count in the file
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> if __name__ == "__main__":
>>>>>>>>>>>>>>>>>>>> parser = argparse.ArgumentParser()
>>>>>>>>>>>>>>>>>>>> parser.add_argument('--start', type=int,
>>>>>>>>>>>>>>>>>>>> help='Starting
>>>>>>>>>>>>>>>>>>>> line count (inclusive)')
>>>>>>>>>>>>>>>>>>>> parser.add_argument('--end', type=int, help='Ending
>>>>>>>>>>>>>>>>>>>> line count (inclusive)')
>>>>>>>>>>>>>>>>>>>> args = parser.parse_args()
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> training_text_file = 'langdata/ben.training_text'
>>>>>>>>>>>>>>>>>>>> output_directory = 'tesstrain/data/ben-ground-truth
>>>>>>>>>>>>>>>>>>>> '
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> # Create an instance of the FontList class
>>>>>>>>>>>>>>>>>>>> font_list = FontList()
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> create_training_data(training_text_file,
>>>>>>>>>>>>>>>>>>>> font_list, output_directory, args.start, args.end)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> *and for training code:*
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> import subprocess
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> # List of font names
>>>>>>>>>>>>>>>>>>>> font_names = ['ben']
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> for font in font_names:
>>>>>>>>>>>>>>>>>>>> command = f"TESSDATA_PREFIX=../tesseract/tessdata
>>>>>>>>>>>>>>>>>>>> make training MODEL_NAME={font} START_MODEL=ben
>>>>>>>>>>>>>>>>>>>> TESSDATA=../tesseract/tessdata MAX_ITERATIONS=10000
>>>>>>>>>>>>>>>>>>>> LANG_TYPE=Indic"
>>>>>>>>>>>>>>>>>>>> subprocess.run(command, shell=True)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> any suggestion to identify to extract the problem.
>>>>>>>>>>>>>>>>>>>> thanks, everyone
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> You received this message because you are subscribed to
>>>>>>>>>>>>>>>>>>>> the Google Groups "tesseract-ocr" group.
>>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving
>>>>>>>>>>>>>>>>>>>> emails from it, send an email to
>>>>>>>>>>>>>>>>>>>> tesseract-oc...@googlegroups.com.
>>>>>>>>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/msgid/tesseract-ocr/406cd733-b265-4118-a7ca-de75871cac39n%40googlegroups.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> <https://groups.google.com/d/msgid/tesseract-ocr/406cd733-b265-4118-a7ca-de75871cac39n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"tesseract-ocr" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to tesseract-ocr+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/tesseract-ocr/d8c16644-b52a-426c-86a6-b1e797f3e5a2n%40googlegroups.com.
