Skip to content

Adding Models

We reccomend you add new models via the Gateway UI. See Managing Models for more.

OptimaGPT runs AI models in the GGUF format. Models are placed in a specific directory on the Node machine, where the Node discovers and registers them automatically. This method of adding models is not reccomended for first time users.

Getting GGUF model files

GGUF model files are widely available from public model repositories such as Hugging Face. When choosing a model, look for files ending in .gguf.

Most models are available in several quantised variants — for example, Q4_K_M, Q5_K_M, or Q8_0. Quantisation reduces the file size and memory requirements of a model at a small cost to response quality. Q4_K_M or Q5_K_M are a good starting point for most use cases, balancing quality and resource usage well.

Placing models on the Node machine

Copy your .gguf files into the Node's model directory. The path depends on the operating system of the Node's host machine:

Windows 

C:\ProgramData\Parker Software\OptimaNode\Executors\Models\lammacpp\

macOS 

/Library/Application Support/Parker Software/OptimaNode/Executors/Models/lammacpp/

Linux 

/var/lib/ParkerSoftware/OptimaNode/Executors/Models/lammacpp/

For embedding models, use the lammacpp_embed subdirectory in place of lammacpp.

The Node scans this directory at startup and periodically while running. Models placed here while the Node is already running will be discovered shortly after without requiring a restart.

Enabling a model in the Gateway

Once the Node has discovered a model file, it appears in the Gateway admin interface. Newly discovered models are disabled by default and must be enabled before they can be used.

To enable a model:

  1. In the Gateway admin interface, open the Nodes page.
  2. Select the Node the model was added to.
  3. Find the model in the model list and toggle it on.

The Node will start the model's executor process. This may take a moment while the model loads into memory.

Checking model status

Once a model is enabled and its executor has started successfully, it will appear as active in the node's model list and become available in the model selector in OptimaChat.

If a model fails to start, check that:

  • The .gguf file is not corrupted. Verify the file size matches what was advertised by the source.
  • The Node machine has sufficient RAM or VRAM to load the model. See System Requirements for guidance on memory and model size.
  • The file is in the correct directory for its type (chat vs. embedding).