com.knuddels.jtokkit.api

Interface Encoding

public interface Encoding

Method Summary

Modifier and Type	Method and Description
int	countTokens(java.lang.String text) Encodes the given text into a list of token ids and returns the amount of tokens.
int	countTokensOrdinary(java.lang.String text) Encodes the given text into a list of token ids and returns the amount of tokens.
java.lang.String	decode(java.util.List<java.lang.Integer> tokens) Decodes the given list of token ids into a text.
byte[]	decodeBytes(java.util.List<java.lang.Integer> tokens) Decodes the given list of token ids into a byte array.
java.util.List<java.lang.Integer>	encode(java.lang.String text) Encodes the given text into a list of token ids.
EncodingResult	encode(java.lang.String text, int maxTokens) Encodes the given text into a list of token ids.
java.util.List<java.lang.Integer>	encodeOrdinary(java.lang.String text) Encodes the given text into a list of token ids, ignoring special tokens.
EncodingResult	encodeOrdinary(java.lang.String text, int maxTokens) Encodes the given text into a list of token ids, ignoring special tokens.
java.lang.String	getName() Returns the name of this encoding.

Method Detail

encode

java.util.List<java.lang.Integer> encode(java.lang.String text)

Encodes the given text into a list of token ids.

Special tokens are artificial tokens used to unlock capabilities from a model, such as fill-in-the-middle. There is currently no support for parsing special tokens in a text, so if the text contains special tokens, this method will throw an UnsupportedOperationException.

If you want to encode special tokens as ordinary text, use encodeOrdinary(String).

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.encode("hello world");
 // returns [15339, 1917]

 encoding.encode("hello <|endoftext|> world");
 // raises an UnsupportedOperationException
 

Parameters:

text - the text to encode

Returns:

the list of token ids

Throws:

java.lang.UnsupportedOperationException - if the text contains special tokens which are not supported for now

encode

EncodingResult encode(java.lang.String text,
                      int maxTokens)

Encodes the given text into a list of token ids.

Special tokens are artificial tokens used to unlock capabilities from a model, such as fill-in-the-middle. There is currently no support for parsing special tokens in a text, so if the text contains special tokens, this method will throw an UnsupportedOperationException.

If you want to encode special tokens as ordinary text, use encodeOrdinary(String, int).

This method will truncate the list of token ids if the number of tokens exceeds the given maxTokens parameter. Note that it will try to keep characters together, that are encoded into multiple tokens. For example, if the text contains a character which is encoded into 3 tokens, and due to the maxTokens parameter the last token of the character is truncated, the first two tokens of the character will also be truncated. Therefore, the actual number of tokens may be less than the given maxTokens parameter.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.encode("hello world", 100);
 // returns [15339, 1917]

 encoding.encode("hello <|endoftext|> world", 100);
 // raises an UnsupportedOperationException
 

Parameters:

text - the text to encode

maxTokens - the maximum number of tokens to encode

Returns:

the EncodingResult containing a list of token ids and whether the tokens were truncated due to the maxTokens parameter

Throws:

java.lang.UnsupportedOperationException - if the text contains special tokens which are not supported for now

encodeOrdinary

java.util.List<java.lang.Integer> encodeOrdinary(java.lang.String text)

Encodes the given text into a list of token ids, ignoring special tokens.

This method does not throw an exception if the text contains special tokens, but instead encodes them as if they were ordinary text.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.encodeOrdinary("hello world");
 // returns [15339, 1917]

 encoding.encodeOrdinary("hello <|endoftext|> world");
 // returns [15339, 83739, 8862, 728, 428, 91, 29, 1917]
 

Parameters:

text - the text to encode

Returns:

the list of token ids

encodeOrdinary

EncodingResult encodeOrdinary(java.lang.String text,
                              int maxTokens)

Encodes the given text into a list of token ids, ignoring special tokens.

This method does not throw an exception if the text contains special tokens, but instead encodes them as if they were ordinary text.

It will truncate the list of token ids if the number of tokens exceeds the given maxTokens parameter. Note that it will try to keep characters together, that are encoded into multiple tokens. For example, if the text contains a character which is encoded into 3 tokens, and due to the maxTokens parameter the last token of the character is truncated, the first two tokens of the character will also be truncated. Therefore, the actual number of tokens may be less than the given maxTokens parameter.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.encodeOrdinary("hello world", 100);
 // returns [15339, 1917]

 encoding.encodeOrdinary("hello <|endoftext|> world", 100);
 // returns [15339, 83739, 8862, 728, 428, 91, 29, 1917]
 

Parameters:

text - the text to encode

maxTokens - the maximum number of tokens to encode

Returns:

the EncodingResult containing a list of token ids and whether the tokens were truncated due to the maxTokens parameter

countTokens

int countTokens(java.lang.String text)

Encodes the given text into a list of token ids and returns the amount of tokens. This is a convenience method for encode(String), if all you want is to know the amount of tokens. It is not more performant than encode(String), so prefer to use encode(String) if you actually need the tokens.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.countTokens("hello world");
 // returns 2

 encoding.countTokens("hello <|endoftext|> world");
 // raises an UnsupportedOperationException
 

Parameters:

text - the text to count tokens for

Returns:

the amount of tokens

Throws:

java.lang.UnsupportedOperationException - if the text contains special tokens which are not supported for now

countTokensOrdinary

int countTokensOrdinary(java.lang.String text)

Encodes the given text into a list of token ids and returns the amount of tokens. This is a convenience method for encodeOrdinary(String), if all you want is to know the amount of tokens. It is not more performant than encodeOrdinary(String), so prefer to use encodeOrdinary(String) if you actually need the tokens.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.countTokensOrdinary("hello world");
 // returns 2

 encoding.countTokensOrdinary("hello <|endoftext|> world");
 // returns 8
 

Parameters:

text - the text to count tokens for

Returns:

the amount of tokens

Throws:

java.lang.UnsupportedOperationException - if the text contains special tokens which are not supported for now

decode

java.lang.String decode(java.util.List<java.lang.Integer> tokens)

Decodes the given list of token ids into a text.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.decode(List.of(15339, 1917));
 // returns "hello world"

 encoding.decode(List.of(15339, 1917, Integer.MAX_VALUE));
 // raises an IllegalArgumentException
 

Parameters:

tokens - the list of token ids

Returns:

the decoded text

Throws:

java.lang.IllegalArgumentException - if the list contains invalid token ids

decodeBytes

byte[] decodeBytes(java.util.List<java.lang.Integer> tokens)

Decodes the given list of token ids into a byte array.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.decodeBytes(List.of(15339, 1917));
 // returns [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]

 encoding.decodeBytes(List.of(15339, 1917, Integer.MAX_VALUE));
 // raises an IllegalArgumentException
 

Parameters:

tokens - the list of token ids

Returns:

the decoded byte array

Throws:

java.lang.IllegalArgumentException - if the list contains invalid token ids

getName

java.lang.String getName()

Returns the name of this encoding. This is the name which is used to identify the encoding and must be unique for registration in the EncodingRegistry.

Returns:

the name of this encoding