![[ICO]](/icons/blank.gif){kind=icon}
![[PARENTDIR]](/icons/back.gif){kind=icon}
![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/folder.gif){kind=icon}
| Name | Last modified | Size | Description |
|---|---|---|---|---|
| Parent Directory | - | ||
| README.md | 2025-05-05 04:48 | 733 | |
| seg_ja.sh | 2025-05-05 04:48 | 399 | |
| seg_ko.sh | 2025-05-05 04:48 | 397 | |
| thirdparty/ | 2025-05-05 04:48 | - | |
| tokenize_indic.py | 2025-05-05 04:48 | 727 | |
| tokenize_thai.py | 2025-05-05 04:48 | 323 | |
| tokenize_zh.py | 2025-05-05 04:48 | 309 | |
| tokenizer_ar.sh | 2025-05-05 04:48 | 758 | |