o ݈ h:@sjddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd l m Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-eGdddZ.eGdddZ/eGdddZ0edGdddeZ1dS)N) dataclassfield)Path)AnyDictListOptional)BaseStep)EncDecCTCModel)EncDecHybridRNNTCTCModel) FrameBatchASR) setup_model)logging) OmegaConf)Step)tqdm)add_t_start_end_to_utt_objget_batch_starts_endsget_batch_variablesget_manifest_lines_batch)make_ass_files)make_ctm_files)write_manifest_out_line)viterbi_decodingc@s&eZdZUdZeed<dZeed<dS) CTMFileConfigFremove_blank_tokensrminimum_timestamp_durationN)__name__ __module__ __qualname__rbool__annotations__rfloatr#r#U/var/www/eduai.edurigo.com/doc_train/edurigo_ai/Puru/tts/BhasaAnuvaad/step/aligner.pyr s rc@seZdZUdZeed<dZeed<dZe ed<dZ eed<e d d d Z e eed <e d d d Ze eed<e dd d Ze eed<dS) ASSFileConfigfontsizecentervertical_alignmentFresegment_text_to_fill_spacemax_lines_per_segmentcCgdS)N)1.=r#r#r#r#r$4zASSFileConfig.default_factorytext_already_spoken_rgbcCr-)N)9 r#r#r#r#r$r17r2text_being_spoken_rgbcCr-)N)r#r#r#r#r$r1:r2text_not_yet_spoken_rgbN)rrrr'intr!r)strr*r r,rr5rr9r=r#r#r#r$r%*s    r%c@s\eZdZUdZeeed<dZeeed<dZeeed<dZ eeed<dZ eeed<dZ e ed<dZ eeed <dZeeed <d Zeed <dZe ed <d Zeed<dZe ed<dZeeed<d Zeed<dZe ed<dZeed<dZeed<dZeed<dZee ed<edddZeeed<edddZe ed <ed!ddZ!e"ed"<dS)#AlignmentConfigNpretrained_name model_pathmanifest_filepath output_dir language_idFalign_using_pred_texttranscribe_deviceviterbi_devicemanifest_batch_sizeuse_silero_vadvad_chunked_batch_sizeTuse_local_attention%additional_segment_grouping_separatoraudio_filepath_parts_in_utt_iduse_buffered_chunked_streamingg?chunk_len_in_secsg@total_buffer_in_secs chunk_batch_sizesimulate_cache_aware_streamingcCsdgS)Nctmr#r#r#r#r$r1\zAlignmentConfig.r3save_output_file_formatscCtSN)rr#r#r#r$r1]rWctm_file_configcCrYrZ)r%r#r#r#r$r1^rWass_file_config)#rrrrArr?r!rBrCrDrErFr rGrHrJr>rKrLrMrNrOrPrQr"rRrTrUrrXrr[rr\r%r#r#r#r$r@>s.           r@alignerc@seZdZ          ddeeefdeedeedeedeed ed eed eed ed edefddZde fddZ ddZ dS)AlignerNFrIinfrarBrArNcustom_model_classrFrGrHrJrKrLc Cs||_|d} |d} | d| d}|dd| }tj|d| drPt|d| d}dd |D|_Wdn1sJwYng|_t|||||d |||| | | d |d |_ |j j durx|j j durxt d |j j dur|j j durt d|j j dkrt d|j jdkrt d|j jdks|j jdkrt d|j jjdkrt d|j jjdvrt d|j jj|j jj|j jjfD] }t|dkrt dq|j jdurttjrdnd|_nt|j j|_td|j|j jdurttjrdnd|_nt|j j|_td|j|jjdks7|jjdkr?d3}t@||}||j!|d d4|_!tA|| \|_B|_Cd|_DtEtF|jGd}tjH|d d5tEtF||}t|d6|_IdS)7Naligner_manifest_pathaligner_languagez /manifest_z.jsonlaligned_output_path/z_with_output_file_paths.jsoncSsg|] }t|dqSaudio_filepath)jsonloads.0liner#r#r$ }sz&Aligner.initialise..aligner_language_idT) rArBrCrDrErFrGrHrJrKrLrMrNz2Both model_path and pretrained_name cannot be Nonez2One of model_path and pretrained_name must be NonerIz7manifest_batch_size cannot be zero or a negative numberz:vad_chunked_batch_size cannot be zero or a negative number zGadditional_grouping_separator cannot be empty string or space characterrz6minimum_timestamp_duration cannot be a negative number)topr(bottomzMass_file_config.vertical_alignment must be one of 'top', 'center' or 'bottom'zass_file_config.text_already_spoken_rgb, ass_file_config.text_being_spoken_rgb, and ass_file_config.text_already_spoken_rgb all need to contain exactly 3 elements.cudacpuzBDevice to be used for transcription step (`transcribe_device`) is z9Device to be used for viterbi step (`viterbi_device`) is zOne or both of transcribe_device and viterbi_device are GPUs. If you run into OOM errors it may help to change both devices to be the CPU.ctc) decoder_typezdFlag use_local_attention is set to True => will try to use local attention for model if it allows itrel_pos_local_attn@)self_attention_modelatt_context_sizezModel is not an instance of NeMo EncDecCTCModel or ENCDecHybridRNNTCTCModel. Currently only instances of these models are supportedz;ctm_file_config.minimum_timestamp_duration has been set to zd seconds. This may cause the alignments for some tokens/words/additional segments to be overlapping.Fg per_featurez\Only EncDecCTCModelBPE models trained with per_feature normalization are supported currently window_strider+ztokens_per_chunk is z, mid_delay is ) asr_model frame_len total_buffer batch_size)delaymodel_stride_in_secstokens_per_chunkmodels)modelrE)exist_okza+)Jr_ get_stateospathexistsopen readlinesdoner@cfgrBrA ValueErrorrJrLrNr[rr\r)r5lenrGtorchdevicers is_availablerinforHtypewarningr reval isinstancer change_decoding_strategyrMchange_attention_modelr NotImplementedErrorbuffered_chunk_paramsrPcopydeepcopy_cfgr set_structmodel_preprocessorditherpad_to normalizeerrormodel_downsample_factorrRr"rQmathceilr rT importlib import_modulegetattrrstartsendsoutput_timestep_durationr?rstemmakedirsf_manifest_out)selfr_rBrArNr`rFrGrHrJrKrLralanguagerCrDfhandrgb_list_ model_cfgfeature_striderr chunk_lenr mid_delaymodule model_clstgt_manifest_nametgt_manifest_filepathr#r#r$ initialisecs,                        zAligner.initialisereturncsFttjjD]\}}tjj||jj}tfdd|D}|jj kr*q t |j jjjj jj jjjjjjjjjjjj \}}}}} } t||||j} tt| | D]8\} \} }t| || } djjvrt| jjjjj || | } djjvrt| jjjj} tj | qbq j !dS)Ncsg|] }|djvqSre)rrirr#r$rlDszAligner.run..rVass)"rziprrrrrCrEsumrJrrrNrFrOrrKrLrUrPrrrH enumeraterrXrrDr[rr\rrclose)rstartendmanifest_lines_batchrlog_probs_batchy_batchT_batchU_batch utt_obj_batchralignments_batchidxutt_obj alignment_uttr#rr$run<st     z Aligner.runcCs|`dSrZ)rrr#r#r$cleanupszAligner.cleanup) NNNNFNNrIFrI) rrrrr?r>rr rrrrr#r#r#r$r^asJ     Z Dr^)2rrrgrr dataclassesrrpathlibrtypingrrrrrbaser &nemo.collections.asr.models.ctc_modelsr 2nemo.collections.asr.models.hybrid_rnnt_ctc_modelsr 0nemo.collections.asr.parts.utils.streaming_utilsr 1nemo.collections.asr.parts.utils.transcribe_utilsr nemo.utilsr omegaconfrstep_decoratorrrutils.data_preprrrrutils.make_ass_filesrutils.make_ctm_filesrutils.make_output_manifestrutils.viterbi_decodingrrr%r@r^r#r#r#r$s>               "