com.knuddels.jtokkit.api

Interface EncodingRegistry

public interface EncodingRegistry

The EncodingRegistry is used to register custom encodings and to retrieve encodings by name or type. The out-of-the-box supported encodings are registered automatically.

Method Summary

Modifier and Type	Method and Description
Encoding	getEncoding(EncodingType encodingType) Returns the encoding with the given type.
java.util.Optional<Encoding>	getEncoding(java.lang.String encodingName) Returns the encoding with the given name, if it exists.
Encoding	getEncodingForModel(ModelType modelType) Returns the encoding that is used for the given model type.
java.util.Optional<Encoding>	getEncodingForModel(java.lang.String modelName) Returns the encoding that is used for the given model type, if it exists.
EncodingRegistry	registerCustomEncoding(Encoding encoding) Registers a new custom encoding with the given name.
EncodingRegistry	registerGptBytePairEncoding(GptBytePairEncodingParams parameters) Registers a new byte pair encoding with the given name.

Method Detail

getEncoding

java.util.Optional<Encoding> getEncoding(java.lang.String encodingName)

Returns the encoding with the given name, if it exists. Otherwise, returns an empty Optional. Prefer using getEncoding(EncodingType) or getEncodingForModel(ModelType) for built-in encodings.

Parameters:

encodingName - the name of the encoding

Returns:

the encoding, if it exists

getEncoding

Encoding getEncoding(EncodingType encodingType)

Returns the encoding with the given type.

Parameters:

encodingType - the type of the encoding

Returns:

the encoding

getEncodingForModel

java.util.Optional<Encoding> getEncodingForModel(java.lang.String modelName)

Returns the encoding that is used for the given model type, if it exists. Otherwise, returns an empty Optional. Prefer using getEncodingForModel(ModelType) for built-in encodings.

Note that you can use this method to retrieve the correct encodings for snapshots of models, for example "gpt-4-0314" or "gpt-3.5-turbo-0301".

Parameters:

modelName - the name of the model to get the encoding for

Returns:

the encoding, if it exists

getEncodingForModel

Encoding getEncodingForModel(ModelType modelType)

Returns the encoding that is used for the given model type.

Parameters:

modelType - the model type

Returns:

the encoding

registerGptBytePairEncoding

EncodingRegistry registerGptBytePairEncoding(GptBytePairEncodingParams parameters)

Registers a new byte pair encoding with the given name. The encoding must be thread-safe.

Parameters:

parameters - the parameters for the encoding

Returns:

the registry for method chaining

Throws:

java.lang.IllegalArgumentException - if the encoding name is already registered

See Also:

GptBytePairEncodingParams

registerCustomEncoding

EncodingRegistry registerCustomEncoding(Encoding encoding)

Registers a new custom encoding with the given name. The encoding must be thread-safe.

Parameters:

encoding - the encoding

Returns:

the registry for method chaining

Throws:

java.lang.IllegalArgumentException - if the encoding name is already registered