com.knuddels.jtokkit.api

Class GptBytePairEncodingParams

java.lang.Object

com.knuddels.jtokkit.api.GptBytePairEncodingParams

public final class GptBytePairEncodingParams
extends java.lang.Object

Parameter for the byte pair encoding used to tokenize for the OpenAI GPT models.

This library supports the encodings that are listed in EncodingType out of the box. But if you want to use a custom encoding, you can use this class to pass the parameters to the library. Use EncodingRegistry.registerGptBytePairEncoding(GptBytePairEncodingParams) to register your custom encoding to the registry, so that you can easily use your encoding in conjunction with the predefined ones.

The encoding parameters are:

• name: The name of the encoding. This is used to identify the encoding and must be unique.
• pattern: The pattern that is used to split the input text into tokens.
• encoder: The encoder that maps the tokens to their ids.
• specialTokensEncoder: The encoder that maps the special tokens to their ids.

Constructor Summary

Constructors

Constructor and Description
GptBytePairEncodingParams(java.lang.String name, java.util.regex.Pattern pattern, java.util.Map<byte[],java.lang.Integer> encoder, java.util.Map<java.lang.String,java.lang.Integer> specialTokensEncoder) Creates a new instance of GptBytePairEncodingParams.

Method Summary

Modifier and Type	Method and Description
java.util.Map<byte[],java.lang.Integer>	getEncoder()
java.lang.String	getName()
java.util.regex.Pattern	getPattern()
java.util.Map<java.lang.String,java.lang.Integer>	getSpecialTokensEncoder()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GptBytePairEncodingParams

public GptBytePairEncodingParams(java.lang.String name,
                                 java.util.regex.Pattern pattern,
                                 java.util.Map<byte[],java.lang.Integer> encoder,
                                 java.util.Map<java.lang.String,java.lang.Integer> specialTokensEncoder)

Creates a new instance of GptBytePairEncodingParams.

Parameters:

name - the name of the encoding. This is used to identify the encoding and must be unique

pattern - the pattern that is used to split the input text into tokens.

encoder - the encoder that maps the tokens to their ids

specialTokensEncoder - the encoder that maps the special tokens to their ids

Method Detail

getName

public java.lang.String getName()

getPattern

public java.util.regex.Pattern getPattern()

getEncoder

public java.util.Map<byte[],java.lang.Integer> getEncoder()

getSpecialTokensEncoder

public java.util.Map<java.lang.String,java.lang.Integer> getSpecialTokensEncoder()