Interface LlamaCppInputs

Note that the modelPath is the only required parameter. For testing you can set this in the environment variable LLAMA_PATH.

interface LlamaCppInputs {
    batchSize?: number;
    cache?: boolean | BaseCache<Generation[]>;
    callbackManager?: CallbackManager;
    callbacks?: Callbacks;
    concurrency?: number;
    contextSize?: number;
    embedding?: boolean;
    f16Kv?: boolean;
    gbnf?: string;
    gpuLayers?: number;
    jsonSchema?: object;
    logitsAll?: boolean;
    maxConcurrency?: number;
    maxRetries?: number;
    maxTokens?: number;
    metadata?: Record<string, unknown>;
    modelPath: string;
    onFailedAttempt?: FailedAttemptHandler;
    prependBos?: boolean;
    seed?: null | number;
    tags?: string[];
    temperature?: number;
    threads?: number;
    topK?: number;
    topP?: number;
    trimWhitespaceSuffix?: boolean;
    useMlock?: boolean;
    useMmap?: boolean;
    verbose?: boolean;
    vocabOnly?: boolean;
}

Hierarchy

  • LlamaBaseCppInputs
  • BaseLLMParams
    • LlamaCppInputs

Properties

batchSize? cache? callbackManager? callbacks? concurrency? contextSize? embedding? f16Kv? gbnf? gpuLayers? jsonSchema? logitsAll? maxConcurrency? maxRetries? maxTokens? metadata? modelPath onFailedAttempt? prependBos? seed? tags? temperature? threads? topK? topP? trimWhitespaceSuffix? useMlock? useMmap? verbose? vocabOnly?

Properties

OptionalbatchSize

batchSize?: number

Prompt processing batch size.

Optionalcache

cache?: boolean | BaseCache<Generation[]>

OptionalcallbackManager

callbackManager?: CallbackManager

Deprecated

Use callbacks instead

Optionalcallbacks

callbacks?: Callbacks

Optionalconcurrency

concurrency?: number

Deprecated

Use maxConcurrency instead

OptionalcontextSize

contextSize?: number

Text context size.

Optionalembedding

embedding?: boolean

Embedding mode only.

Optionalf16Kv

f16Kv?: boolean

Use fp16 for KV cache.

Optionalgbnf

gbnf?: string

GBNF string to be used to format output. Also known as grammar.

OptionalgpuLayers

gpuLayers?: number

Number of layers to store in VRAM.

OptionaljsonSchema

jsonSchema?: object

JSON schema to be used to format output. Also known as grammar.

OptionallogitsAll

logitsAll?: boolean

The llama_eval() call computes all logits, not just the last one.

OptionalmaxConcurrency

maxConcurrency?: number

The maximum number of concurrent calls that can be made. Defaults to Infinity, which means no limit.

OptionalmaxRetries

maxRetries?: number

The maximum number of retries that can be made for a single call, with an exponential backoff between each attempt. Defaults to 6.

OptionalmaxTokens

maxTokens?: number

Optionalmetadata

metadata?: Record<string, unknown>

modelPath

modelPath: string

Path to the model on the filesystem.

OptionalonFailedAttempt

onFailedAttempt?: FailedAttemptHandler

Custom handler to handle failed attempts. Takes the originally thrown error object as input, and should itself throw an error if the input error is not retryable.

OptionalprependBos

prependBos?: boolean

Add the begining of sentence token.

Optionalseed

seed?: null | number

If null, a random seed will be used.

Optionaltags

tags?: string[]

Optionaltemperature

temperature?: number

The randomness of the responses, e.g. 0.1 deterministic, 1.5 creative, 0.8 balanced, 0 disables.

Optionalthreads

threads?: number

Number of threads to use to evaluate tokens.

OptionaltopK

topK?: number

Consider the n most likely tokens, where n is 1 to vocabulary size, 0 disables (uses full vocabulary). Note: only applies when temperature > 0.

OptionaltopP

topP?: number

Selects the smallest token set whose probability exceeds P, where P is between 0 - 1, 1 disables. Note: only applies when temperature > 0.

OptionaltrimWhitespaceSuffix

trimWhitespaceSuffix?: boolean

Trim whitespace from the end of the generated text Disabled by default.

OptionaluseMlock

useMlock?: boolean

Force system to keep model in RAM.

OptionaluseMmap

useMmap?: boolean

Use mmap if possible.

Optionalverbose

verbose?: boolean

OptionalvocabOnly

vocabOnly?: boolean

Only load the vocabulary, no weights.

Settings

Member Visibility

On This Page

Properties