o ag2 @sddlZddlmZddlmZddlmZeddgdZejede d d Z e d ej d d dddej dd ddddej dddddej dddddej dddd dd!d"Z dS)#N)tqdm) word_tokenize)parallelize_preprocessz-hz--help)help_option_names)context_settingscCsdS)Nrrrb/var/www/eduai.edurigo.com/doc_train/edurigo_ai/Puru/venv/lib/python3.10/site-packages/nltk/cli.pyclisr tokenizez --languagez-lenz1The language for the Punkt sentence tokenization.)defaulthelpz--preserve-lineTzIAn option to keep the preserve the sentence and not sentence tokenize it.)r is_flagr z --processesz-jzNo. of processes.z --encodingz-eutf8zSpecify encoding of file.z --delimiterz-d z%Specify delimiter to join the tokens.c Cstjd|dY}tjd|d9}|dkr*t|D]}t|t|d|dqntt||ddD] }t||d|dq4Wd n1sKwYWd d SWd d S1scwYd S) z;This command tokenizes text stream using nltk.word_tokenizestdin)encodingstdoutr )endfileT) progress_barN)clickget_text_streamr readlinesprintjoinrr) language preserve_line processesr delimiterfinfoutlineoutlinerrr tokenize_files  "r&)rrnltkr nltk.utilrdictCONTEXT_SETTINGSgroupversion_optionr commandoptionr&rrrrs8