python314Packages.pytorch-tokenizers 1.2.0

C++ implementations for various tokenizers (sentencepiece, tiktoken, etc.)

Linux and macOS (Apple Silicon only) last changed 2026-05-22 outputs: out, dist

Pin this version

$ devbox add python314Packages.pytorch-tokenizers@1.2.0

systemsaarch64-darwin commita0991c886dc83e6e9de01d5266e4842985b1850e

$ nix shell github:NixOS/nixpkgs/a0991c886dc83e6e9de01d5266e4842985b1850e#python314Packages.pytorch-tokenizers

systemsaarch64-linux, x86_64-linux commit4a29d733e8a7d5b824c3d8c958a946a9867b3eb2

$ nix shell github:NixOS/nixpkgs/4a29d733e8a7d5b824c3d8c958a946a9867b3eb2#python314Packages.pytorch-tokenizers

System	Attribute	Commit	Date	Outputs
`aarch64-darwin` macOS · arm64	`python314Packages.pytorch-tokenizers`	a0991c886dc8	2026-05-22	`out` /nix/store/l6fwalkyrcbdzkp2f8s232mzz5360zfj-python3.14-pytorch-tokenizers-1.2.0 `dist` (not default) /nix/store/zaxc2hzr6q4zzxjzb0m3kmka0s39ka1j-python3.14-pytorch-tokenizers-1.2.0-dist
`aarch64-linux` Linux · arm64	`python314Packages.pytorch-tokenizers`	4a29d733e8a7	2026-05-21	`out` /nix/store/f6gnm56xqk5j2nlq9d4bjs9k5hd1falr-python3.14-pytorch-tokenizers-1.2.0 `dist` (not default) /nix/store/agv5hpj1ckwyxp2qa7k6a5qxdixpgx2v-python3.14-pytorch-tokenizers-1.2.0-dist
`x86_64-linux` Linux · x86-64	`python314Packages.pytorch-tokenizers`	4a29d733e8a7	2026-05-21	`out` /nix/store/l1pzfvly2m3d8j2myxzwm5nxgbb32cqb-python3.14-pytorch-tokenizers-1.2.0 `dist` (not default) /nix/store/rbjqkgw3x01v60hqprqc9isqxihfq2ya-python3.14-pytorch-tokenizers-1.2.0-dist