flutter pub add tiktoken根據套件公開資料產生的選型評估，僅供參考。
Tiktoken 是一種為 OpenAI 語言模型設計的 BPE（字節對編碼）分詞器，透過將原始文字轉換為標記來實現高效的文字處理。它提供 API 來計算標記數量並管理文字分段，對於優化大型語言模型互動中的輸入長度至關重要。該套件支援多種平台，包括 Android、iOS、Web、macOS、Windows 和 Linux，使其適用於整合 OpenAI 模型的跨平台應用。基於 Dart 建構，確保在不同環境中的一致性分詞效果。
以下為關聯原始碼儲存庫的 README 原文快照，最新內容請造訪原始碼儲存庫。
tiktoken is a BPE tokeniser for use with OpenAI's models.
Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you a) whether the string is too long for a text model to process and b) how much an OpenAI API call costs (as usage is priced by token). Different models use different encodings.
The main Tiktoken class exposes APIs that allow you to process text using tokens, which are common sequences of character found in text. Some of the things you can do with tiktoken package are:
For more examples, see the /example folder.
import 'package:tiktoken/tiktoken.dart';
// Load an encoding
final encoding = encodingForModel("gpt-4");
// Tokenize text
print(encoding.encode("tiktoken is great!")); // [83, 1609, 5963, 374, 2294, 0]
// Decode tokens
print(encoding.decode([83, 1609, 5963, 374, 2294, 0])); // "tiktoken is great!"
// Alternatively, get the tokenizer by specifying encoding name:
final encoding = getEncoding("cl100k_base");
assert(enc.decode(enc.encode("hello world")) == "hello world");
You may wish to extend Tiktoken to support new encodings. You can do this by passing around the existing model:
import 'package:tiktoken/tiktoken.dart';
// Create a base
final cl100kBase = encodingForModel("cl100k_base");
// Instantiate a new encoding and extend the base params
final encoding = Tiktoken(
name: "cl100k_im",
patStr: cl100kBase.patStr,
mergeableRanks: cl100kBase.mergeableRanks,
specialTokens: {
...cl100kBase.specialTokens,
"<|im_start|>": 100264,
"<|im_end|>": 100265,
},
);
This is a Dart port from the original tiktoken library written in Rust/Python.