tiktoken

Package Overview

Package selection guidance compiled from public package information, community data, and project usage.

Tiktoken is a BPE (Byte Pair Encoding) tokenizer designed for OpenAI's language models, enabling efficient text processing by converting raw text into tokens. It provides APIs to calculate token counts and manage text segmentation, crucial for optimizing input length in LLM interactions. The package supports multiple platforms including Android, iOS, Web, macOS, Windows, and Linux, making it suitable for cross-platform applications that integrate with OpenAI models. Built with Dart, it ensures consistent tokenization across environments.

Use cases

Text length estimation for LLM prompts
Token-aware text truncation
Integration with OpenAI API clients
Cross-platform AI applications
Prompt engineering and optimization

Key features

BPE-based tokenization
Platform-agnostic implementation
Support for OpenAI model tokenizers
Accurate token counting
Lightweight and fast

Suitable for

Developers building AI-powered apps
Applications requiring prompt management
Projects using OpenAI models on multiple platforms
Users needing precise token control

Considerations

Requires understanding of tokenization concepts
Depends on OpenAI model-specific vocabularies
Not suitable for non-OpenAI models
Limited to text preprocessing tasks
May require updates when new models are released

Original README

Original README snapshot from the linked source repository. Visit the repository for the latest content.

⏳ tiktoken

tiktoken is a BPE tokeniser for use with OpenAI's models.

Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you a) whether the string is too long for a text model to process and b) how much an OpenAI API call costs (as usage is priced by token). Different models use different encodings.

Features

The main Tiktoken class exposes APIs that allow you to process text using tokens, which are common sequences of character found in text. Some of the things you can do with tiktoken package are:

Encode text into tokens
Decode tokens into text
Compare different encodings
Count tokens for chat API calls

Usage

For more examples, see the /example folder.

import 'package:tiktoken/tiktoken.dart';

// Load an encoding
final encoding = encodingForModel("gpt-4");

// Tokenize text
print(encoding.encode("tiktoken is great!")); // [83, 1609, 5963, 374, 2294, 0]

// Decode tokens
print(encoding.decode([83, 1609, 5963, 374, 2294, 0])); // "tiktoken is great!"

// Alternatively, get the tokenizer by specifying encoding name:
final encoding = getEncoding("cl100k_base");
assert(enc.decode(enc.encode("hello world")) == "hello world");

Extending tiktoken

You may wish to extend Tiktoken to support new encodings. You can do this by passing around the existing model:

import 'package:tiktoken/tiktoken.dart';

// Create a base
final cl100kBase = encodingForModel("cl100k_base");

// Instantiate a new encoding and extend the base params
final encoding = Tiktoken(
  name: "cl100k_im",
  patStr: cl100kBase.patStr,
  mergeableRanks: cl100kBase.mergeableRanks,
  specialTokens: {
    ...cl100kBase.specialTokens,
    "<|im_start|>": 100264,
    "<|im_end|>": 100265,
  },
);

Additional information

This is a Dart port from the original tiktoken library written in Rust/Python.

Search packages, repositories and app categories

Install package