Home > Repositories > base > python-tokenizers
Resources:
Source files | Changes
Issues | Add a new issue
Download

python-tokenizers: An implementation of today's most used tokenizers, with a focus on performance and versatility

An implementation of today's most used tokenizers, with a focus on performance and versatility.
Name:python-tokenizers
Release: 0.22.2-1mamba
Group:System/Libraries
Maintainer:silvan
License:Apache License 2.0
Size:373.66 kB
Upstream URL:https://github.com/huggingface/tokenizers
Build time:2026-01-20 18:07:19
Binary packages and requirements
x86_64
Built RPMSpython-tokenizers-debug
python-tokenizers-py3
Build requirementsglibc-devel [1]
libgcc [1]
libpython3-devel [1]
libpython311-devel [1]
libstdc++6-devel [1]
python3 [1]
python3.11dist(huggingface-hub) [1]
History of changes
DatePackagerReleaseDescription
2026-01-20silvan0.22.2-1mamba- package created using the webbuild interface