Package Release Info

python-tokenizers-0.21.1-bp160.1.11

Update Info: Base Release
Available in Package Hub : 16.0

platforms

AArch64
ppc64le
s390x
x86-64

subpackages

python313-tokenizers

Change Logs

* Wed Mar 19 2025 Lucas Mulling <lucas.mulling@suse.com>
- Update to 0.21.1:
  * Update dev version and pyproject.toml
  * Add feature flag hint to README.md
  * Upgrade to PyO3 0.23
  * Fixing the README.md
  * Fix typo in Split docstrings
  * Fix typos
  * Update documentation of Rust feature
  * Fix panic in DecodeStream::step due to incorrect index usage
  * Fixing the stream by removing the read_index altogether
  * Fixing NormalizedString append when normalized is empty
  * Update metadata as Python3.7 and Python3.8 support was dropped
  * Add rustls-tls feature
- Remove define skip_python313 1
* Wed Mar 05 2025 Christian Goll <cgoll@suse.com>
- disable python3.13
* Thu Jan 09 2025 Andreas Schwab <schwab@suse.de>
- Enable build on riscv64
* Wed Dec 18 2024 Soc Virnyl Estela <uncomfyhalomacro@opensuse.org>
- Update to version 0.21.0:
  * More cache options.
  * Disable caching for long strings.
  * Testing ABI3 wheels to reduce number of wheels
  * Adding an API for decode streaming.
  * Decode stream python
  * Fix encode_batch and encode_batch_fast to accept ndarrays again
* Thu Nov 07 2024 Soc Virnyl Estela <uncomfyhalomacro@opensuse.org>
- Select only rust tier 1 arches.
- Update registry.tar.zst dependencies
- Update version to 0.20.3:
  * fix pylist
  * [MINOR:TYP] Fix docstrings
- Updates from 0.20.2:
  * Bump cookie and express in /tokenizers/examples/unstable_wasm/www
  * Fix off-by-one error in tokenizer::normalizer::Range::len
  * Arg name correction: auth_token -> token
  * Unsound call of set_var
  * Add safety comments
  * PyO3 0.22
- Updates from 0.20.1:
  * Update README.md
  * fix benchmark file link
  * [ignore_merges] Fix offsets
  * Bump body-parser and express in /tokenizers/examples/unstable_wasm/www
  * Bump serve-static and express in /tokenizers/examples/unstable_wasm/www
  * Bump send and express in /tokenizers/examples/unstable_wasm/www
  * Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www
  * Fix documentation build
  * style: simplify string formatting for readability
* Sun Nov 03 2024 Soc Virnyl Estela <uncomfyhalomacro@opensuse.org>
- Experiment with cargo vendor home registry. See documentation:
  https://github.com/openSUSE-Rust/obs-service-cargo/blob/master/README.md#cargo-vendor-home-registry
* Mon Sep 23 2024 Simon Lees <sflees@suse.de>
- Don't use macros for Requires
* Fri Aug 30 2024 Simon Lees <sflees@suse.de>
- Update package name back to "huggingface-hub" to match pypi
* Tue Aug 27 2024 Guang Yee <gyee@suse.com>
- Update package name "huggingface-hub" to "huggingface_hub"
* Tue Aug 20 2024 Simon Lees <sflees@suse.de>
- Fix testsuite on 15.6