C D E F G I M N R T V

C

com.knuddels.jtokkit - package com.knuddels.jtokkit

com.knuddels.jtokkit.api - package com.knuddels.jtokkit.api

countTokens(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids and returns the amount of tokens.

countTokensOrdinary(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids and returns the amount of tokens.

D

decode(List<Integer>) - Method in interface com.knuddels.jtokkit.api.Encoding

Decodes the given list of token ids into a text.

decodeBytes(List<Integer>) - Method in interface com.knuddels.jtokkit.api.Encoding

Decodes the given list of token ids into a byte array.

E

encode(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids.

encode(String, int) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids.

encodeOrdinary(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids, ignoring special tokens.

encodeOrdinary(String, int) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids, ignoring special tokens.

Encoding - Interface in com.knuddels.jtokkit.api

EncodingRegistry - Interface in com.knuddels.jtokkit.api

The EncodingRegistry is used to register custom encodings and to retrieve encodings by name or type.

EncodingResult - Class in com.knuddels.jtokkit.api

The result of encoding operation.

EncodingResult(List<Integer>, boolean) - Constructor for class com.knuddels.jtokkit.api.EncodingResult

Encodings - Class in com.knuddels.jtokkit

EncodingType - Enum in com.knuddels.jtokkit.api

F

fromName(String) - Static method in enum com.knuddels.jtokkit.api.EncodingType

fromName(String) - Static method in enum com.knuddels.jtokkit.api.ModelType

Returns a ModelType for the given name, or Optional.empty() if no such model type exists.

G

getEncoder() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

getEncoding(String) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding with the given name, if it exists.

getEncoding(EncodingType) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding with the given type.

getEncodingForModel(String) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding that is used for the given model type, if it exists.

getEncodingForModel(ModelType) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding that is used for the given model type.

getEncodingType() - Method in enum com.knuddels.jtokkit.api.ModelType

Returns the encoding type that is used by this model type.

getMaxContextLength() - Method in enum com.knuddels.jtokkit.api.ModelType

Returns the maximum context length that is supported by this model type.

getName() - Method in interface com.knuddels.jtokkit.api.Encoding

Returns the name of this encoding.

getName() - Method in enum com.knuddels.jtokkit.api.EncodingType

getName() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

getName() - Method in enum com.knuddels.jtokkit.api.ModelType

Returns the name of the model type as used by the OpenAI API.

getPattern() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

getSpecialTokensEncoder() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

getTokens() - Method in class com.knuddels.jtokkit.api.EncodingResult

GptBytePairEncodingParams - Class in com.knuddels.jtokkit.api

Parameter for the byte pair encoding used to tokenize for the OpenAI GPT models.

GptBytePairEncodingParams(String, Pattern, Map<byte[], Integer>, Map<String, Integer>) - Constructor for class com.knuddels.jtokkit.api.GptBytePairEncodingParams

Creates a new instance of GptBytePairEncodingParams.

I

isTruncated() - Method in class com.knuddels.jtokkit.api.EncodingResult

M

ModelType - Enum in com.knuddels.jtokkit.api

N

newDefaultEncodingRegistry() - Static method in class com.knuddels.jtokkit.Encodings

Creates a new EncodingRegistry with the default encodings found in the EncodingType enum.

newLazyEncodingRegistry() - Static method in class com.knuddels.jtokkit.Encodings

Creates a new EncodingRegistry without any EncodingType registered.

R

registerCustomEncoding(Encoding) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Registers a new custom encoding with the given name.

registerGptBytePairEncoding(GptBytePairEncodingParams) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Registers a new byte pair encoding with the given name.

T

toString() - Method in class com.knuddels.jtokkit.api.EncodingResult

V

valueOf(String) - Static method in enum com.knuddels.jtokkit.api.EncodingType

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.knuddels.jtokkit.api.ModelType

Returns the enum constant of this type with the specified name.

values() - Static method in enum com.knuddels.jtokkit.api.EncodingType

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.knuddels.jtokkit.api.ModelType

Returns an array containing the constants of this enum type, in the order they are declared.

C D E F G I M N R T V