Home > Repositories > devel-autodist > python-tokenizers
An implementation of today's most used tokenizers, with a focus on performance and versatility.
tokenizers-0.23.1.tar.gz
Build required by
python-tokenizers: An implementation of today's most used tokenizers, with a focus on performance and versatility
An implementation of today's most used tokenizers, with a focus on performance and versatility.
⚠ This package has the following problems:
- python-tokenizers-py3(x86_64,devel-autodist): python3.11dist(tokenizers)=0:0.23.1 from python-tokenizers-py3(x86_64,devel-autodist) fails to provide python3.11dist(tokenizers)=0:0.22.2 to python-litellm-py3(noarch,devel-autodist)
- need to be rebuilt: python-litellm(x86_64,devel-autodist)
| Name: | python-tokenizers |
| Release: | 0.23.1-1mamba |
| Group: | System/Libraries |
| Maintainer: | silvan |
| License: | Apache License 2.0 |
| Size: | 367.93 kB |
| Upstream URL: | https://github.com/huggingface/tokenizers |
| Build time: | 2026-04-27 20:40:26 |
Source files
python-tokenizers.spectokenizers-0.23.1.tar.gz
Binary packages and requirements
| x86_64 | |
|---|---|
| Built RPMS | python-tokenizers-debug python-tokenizers-py3 |
| Build requirements | glibc-devel [1] libgcc [1] libpython3-devel [1] libpython311-devel [1] libstdc++6-devel [1] python3 [1] python3.11dist(huggingface-hub)[<2~~] [1] python3.11dist(huggingface-hub)[>=0.16.4)] [1] |
History of changes
| Date | Packager | Release | Description |
|---|---|---|---|
| 2026-04-27 | Automatic Build System | 0.23.1-1mamba | - automatic version update by autodist |
| 2026-01-20 | silvan | 0.22.2-1mamba | - package created using the webbuild interface |
