A simple one-file way to run various GGML and GGUF models with KoboldAI's UI
Go to file
Concedo 702be65ed1 don't show embedded sdui if no model 2024-05-11 08:56:56 +08:00
.github/workflows Stop forcing concedo_experimental in CI (#833) 2024-05-08 09:54:56 +08:00
common Merge branch 'upstream' into concedo_experimental 2024-05-09 15:13:34 +08:00
docs Merge branch 'upstream' into concedo_experimental 2024-05-08 09:58:23 +08:00
examples Merge commit '8c570c9496212073079476651c7517c02581101f' into concedo_experimental 2024-05-10 16:55:26 +08:00
ggml-cuda Merge commit '8c570c9496212073079476651c7517c02581101f' into concedo_experimental 2024-05-10 16:55:26 +08:00
gguf-py gguf-py : add special token modification capability (#7166) 2024-05-09 13:56:00 +03:00
include Revert "clblast up ver" 2024-02-21 14:35:38 +08:00
lib Revert "clblast up ver" 2024-02-21 14:35:38 +08:00
media README: add graphic for matrix multiplication (#6881) 2024-04-24 21:29:13 +02:00
models Merge branch 'upstream' into concedo_experimental 2024-05-09 15:13:34 +08:00
otherarch fixed stableui btn (+4 squashed commit) 2024-05-06 00:55:16 +08:00
scripts Merge commit '8c570c9496212073079476651c7517c02581101f' into concedo_experimental 2024-05-10 16:55:26 +08:00
tests Merge commit '8c570c9496212073079476651c7517c02581101f' into concedo_experimental 2024-05-10 16:55:26 +08:00
.editorconfig llama.swiftui : add bench functionality (#4483) 2023-12-17 19:38:41 +02:00
.gitignore Improve usability of --model-url & related flags (#6930) 2024-04-30 00:52:50 +01:00
.gitmodules no modules 2024-02-12 16:51:06 +08:00
AUTHORS Merge branch 'upstream' into concedo 2024-04-09 17:18:35 +08:00
CLINFO_LICENSE switched back to clinfo since it's possibly more cross platform and can get memory vals easily 2023-11-02 14:12:05 +08:00
CMakeLists.txt cmake : fix typo (#7151) 2024-05-08 19:55:32 -04:00
LICENSE.md update license, added backwards compatibility with both ggml model formats, fixed context length issues. 2023-03-20 23:43:35 +08:00
MIT_LICENSE_GGML_LLAMACPP_ONLY rename koboldcpp.dll to koboldcpp_default.dll 2023-08-11 14:54:27 +08:00
Makefile Introduction of CUDA Graphs to LLama.cpp (#6766) 2024-05-08 22:55:49 +02:00
OpenCL.dll include opencl dll 2023-11-11 10:05:42 +08:00
README.md readme : add scheduled server workflow status badge 2024-05-09 16:40:42 +03:00
Remote-Link.cmd improved remotelink cmd, fixed lib unload, updated class.py 2023-09-25 17:50:00 +08:00
SECURITY.md chore: Fix markdown warnings (#6625) 2024-04-12 10:52:36 +02:00
build-info.h not working merge 2023-11-02 17:52:40 +08:00
class.py replace sdconfig and hordeconfig 2024-05-09 22:43:50 +08:00
clblast.dll Revert "clblast up ver" 2024-02-21 14:35:38 +08:00
colab.ipynb replace sdconfig and hordeconfig 2024-05-09 22:43:50 +08:00
convert-hf-to-gguf-update.py llama3 custom regex split (#6965) 2024-05-09 23:30:44 +10:00
convert-hf-to-gguf.py convert-hf : save memory with lazy evaluation (#7075) 2024-05-08 18:16:38 -04:00
convert-llama-ggml-to-gguf.py convert.py : add python logging instead of print() (#6511) 2024-05-03 22:36:41 +03:00
convert-lora-to-ggml.py py : logging and flake8 suppression refactoring (#7081) 2024-05-05 08:07:48 +03:00
convert-persimmon-to-gguf.py convert.py : add python logging instead of print() (#6511) 2024-05-03 22:36:41 +03:00
convert.py convert-hf : save memory with lazy evaluation (#7075) 2024-05-08 18:16:38 -04:00
cudart64_12.dll add cu12 ci for windows 2024-05-01 22:46:02 +08:00
cudart64_110.dll updated runtimes to henky version 2023-07-18 18:48:54 +08:00
environment.yaml CUDA 12 support for the Conda Runtime (#680) 2024-02-16 21:32:30 +08:00
expose.cpp added more stats in perf 2024-03-16 16:53:48 +08:00
expose.h tidy up and refactor code to support old flags 2024-05-10 16:50:53 +08:00
ggml-alloc.c ggml : fix calloc argument ordering. (#6820) 2024-04-22 16:05:06 +02:00
ggml-alloc.h llama : add pipeline parallelism support (#6017) 2024-03-13 18:54:21 +01:00
ggml-backend-impl.h backend : offload large batches to GPU (#6083) 2024-03-18 11:03:04 +01:00
ggml-backend.c Reset schedule earlier to allow overlap with ggml graph computation on device (#6933) 2024-04-26 20:08:30 +02:00
ggml-backend.h backend : fix typo in scheduler documentation (ggml/781) 2024-04-06 17:42:26 +03:00
ggml-common.h [SYCL] Disable iqx on windows as WA (#6435) 2024-04-03 10:34:40 +08:00
ggml-cuda.cu Merge branch 'upstream' into concedo_experimental 2024-05-09 15:13:34 +08:00
ggml-cuda.h Merge branch 'upstream' into concedo_experimental 2024-03-19 18:57:22 +08:00
ggml-impl.h ggml : introduce bfloat16 support (#6412) 2024-05-08 09:30:09 +03:00
ggml-kompute.cpp ggml : add Flash Attention (#5021) 2024-04-30 12:16:08 +03:00
ggml-kompute.h Nomic Vulkan backend (#4456) 2024-01-29 15:50:50 -05:00
ggml-metal.h metal : add debug capture backend function (ggml/694) 2024-01-30 16:20:25 +02:00
ggml-metal.m Merge branch 'upstream' into concedo_experimental 2024-05-09 15:13:34 +08:00
ggml-metal.metal metal : fix unused warning 2024-05-08 09:14:50 +03:00
ggml-mpi.c ggml : remove src0 and src1 from ggml_tensor and rename opt to src (#2178) 2023-07-11 19:31:10 +03:00
ggml-mpi.h mpi : add support for distributed inference via MPI (#2099) 2023-07-10 18:49:56 +03:00
ggml-opencl.cpp Merge commit '8c570c9496212073079476651c7517c02581101f' into concedo_experimental 2024-05-10 16:55:26 +08:00
ggml-opencl.h Merge branch 'master' into concedo_experimental 2024-01-27 11:45:07 +08:00
ggml-quants.c Merge branch 'upstream' into concedo_experimental 2024-05-08 18:19:28 +08:00
ggml-quants.h Merge branch 'upstream' into concedo 2024-04-09 17:18:35 +08:00
ggml-sycl.cpp Minor arithmetic improvement to mmvq wrapper kernel (#7172) 2024-05-10 08:32:15 +08:00
ggml-sycl.h [SYCL] offload op (#6217) 2024-03-24 12:04:25 +08:00
ggml-vulkan-shaders.hpp Vulkan Bugfixes and Improvements (#7084) 2024-05-09 20:39:54 +02:00
ggml-vulkan.cpp Vulkan Bugfixes and Improvements (#7084) 2024-05-09 20:39:54 +02:00
ggml-vulkan.h Vulkan k-quant mmq and ggml-backend offload functionality (#6155) 2024-03-29 17:29:21 +01:00
ggml.c Merge branch 'upstream' into concedo_experimental 2024-05-08 18:19:28 +08:00
ggml.h Merge branch 'upstream' into concedo_experimental 2024-05-08 18:19:28 +08:00
ggml_vk_generate_shaders.py Vulkan Bugfixes and Improvements (#7084) 2024-05-09 20:39:54 +02:00
glslc.exe Merge branch 'vulkan_test' into concedo_experimental 2024-01-25 23:01:44 +08:00
gpttype_adapter.cpp tidy up and refactor code to support old flags 2024-05-10 16:50:53 +08:00
kcpp_docs.embd replace sdconfig and hordeconfig 2024-05-09 22:43:50 +08:00
kcpp_sdui.embd fixed stableui btn (+4 squashed commit) 2024-05-06 00:55:16 +08:00
klite.embd tidy up and refactor code to support old flags 2024-05-10 16:50:53 +08:00
koboldcpp.py don't show embedded sdui if no model 2024-05-11 08:56:56 +08:00
koboldcpp.sh fixed stableui btn (+4 squashed commit) 2024-05-06 00:55:16 +08:00
libopenblas.dll integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
llama.cpp Merge commit '8c570c9496212073079476651c7517c02581101f' into concedo_experimental 2024-05-10 16:55:26 +08:00
llama.h Merge branch 'upstream' into concedo_experimental 2024-05-09 15:13:34 +08:00
make_pyinstaller.bat fixed stableui btn (+4 squashed commit) 2024-05-06 00:55:16 +08:00
make_pyinstaller.sh fixed stableui btn (+4 squashed commit) 2024-05-06 00:55:16 +08:00
make_pyinstaller_cuda.bat fixed stableui btn (+4 squashed commit) 2024-05-06 00:55:16 +08:00
make_pyinstaller_cuda12.bat fixed stableui btn (+4 squashed commit) 2024-05-06 00:55:16 +08:00
model_adapter.cpp fix for mamba processing 2024-03-13 13:27:46 +08:00
model_adapter.h fix for mamba processing 2024-03-13 13:27:46 +08:00
msvcp140.dll updated runtimes to henky version 2023-07-18 18:48:54 +08:00
msvcp140_codecvt_ids.dll test workflow (+9 squashed commit) 2024-04-11 23:20:08 +08:00
mypy.ini convert : partially revert PR #4818 (#5041) 2024-01-20 18:14:18 -05:00
niko.ico fixed some build errors on linux, changed icon resolution, added more error printing 2023-05-22 12:18:42 +08:00
nikogreen.ico wip on unified cublas integration, add all the small libraries but exclude the large ones 2023-06-29 18:35:31 +08:00
pyrightconfig.json convert-hf : save memory with lazy evaluation (#7075) 2024-05-08 18:16:38 -04:00
requirements.txt llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
rwkv_vocab.embd updated kobold lite, work on rwkv, added exe path to model load params, added launch parameter 2023-04-18 17:36:44 +08:00
rwkv_world_vocab.embd integrated world tokenizer for RWKV 2023-06-13 20:06:19 +08:00
sgemm.cpp sgemm : AVX Q4_0 and Q8_0 (#6891) 2024-05-08 17:29:23 +03:00
sgemm.h llamafile : use 64-bit integers in sgemm (#6928) 2024-04-26 17:05:33 +03:00
simpleclinfo.cpp windows: added simpleclinfo, which helps determine clblast platform and device on windows 2023-11-01 18:10:35 +08:00
unicode-data.cpp llama3 custom regex split (#6965) 2024-05-09 23:30:44 +10:00
unicode-data.h llama3 custom regex split (#6965) 2024-05-09 23:30:44 +10:00
unicode.cpp llama3 custom regex split (#6965) 2024-05-09 23:30:44 +10:00
unicode.h llama3 custom regex split (#6965) 2024-05-09 23:30:44 +10:00
vcruntime140.dll updated runtimes to henky version 2023-07-18 18:48:54 +08:00
vcruntime140_1.dll updated runtimes to henky version 2023-07-18 18:48:54 +08:00
vulkan-1.dll Merge branch 'vulkan_test' into concedo_experimental 2024-01-25 23:01:44 +08:00
winclinfo.exe switched back to clinfo since it's possibly more cross platform and can get memory vals easily 2023-11-02 14:12:05 +08:00

README.md

koboldcpp

KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models. It's a single self contained distributable from Concedo, that builds off llama.cpp, and adds a versatile Kobold API endpoint, additional format support, Stable Diffusion image generation, backward compatibility, as well as a fancy UI with persistent stories, editing tools, save formats, memory, world info, author's note, characters, scenarios and everything Kobold and Kobold Lite have to offer.

Preview Preview Preview Preview

Windows Usage

  • Download the latest .exe release here or clone the git repo.
  • Windows binaries are provided in the form of koboldcpp.exe, which is a pyinstaller wrapper for a few .dll files and koboldcpp.py. You can also rebuild it yourself with the provided makefiles and scripts.
  • Weights are not included, you can use the official llama.cpp quantize.exe to generate them from your official weight files (or download them from other places such as TheBloke's Huggingface.
  • To run, simply execute koboldcpp.exe.
  • Launching with no command line arguments displays a GUI containing a subset of configurable settings. Generally you dont have to change much besides the Presets and GPU Layers. Read the --help for more info about each settings.
  • By default, you can connect to http://localhost:5001
  • You can also run it using the command line. For info, please check koboldcpp.exe --help

Improving Performance

  • (Nvidia Only) GPU Acceleration: If you're on Windows with an Nvidia GPU you can get CUDA support out of the box using the --usecublas flag, make sure you select the correct .exe with CUDA support.
  • Any GPU Acceleration: As a slightly slower alternative, try CLBlast with --useclblast flags for a slightly slower but more GPU compatible speedup.
  • GPU Layer Offloading: Want even more speedup? Combine one of the above GPU flags with --gpulayers to offload entire layers to the GPU! Much faster, but uses more VRAM. Experiment to determine number of layers to offload, and reduce by a few if you run out of memory.
  • Increasing Context Size: Try --contextsize 4096 to 2x your context size! without much perplexity gain. Note that you'll have to increase the max context in the Kobold Lite UI as well (click and edit the number text field).
  • If you are having crashes or issues, you can try turning off BLAS with the --noblas flag. You can also try running in a non-avx2 compatibility mode with --noavx2. Lastly, you can try turning off mmap with --nommap.

For more information, be sure to run the program with the --help flag, or check the wiki.

Run on Colab

  • KoboldCpp now has an official Colab GPU Notebook! This is an easy way to get started without installing anything in a minute or two. Try it here!.
  • Note that KoboldCpp is not responsible for your usage of this Colab Notebook, you should ensure that your own usage complies with Google Colab's terms of use.

OSX and Linux

On Linux, we provide a koboldcpp-linux-x64-cuda1150 PyInstaller prebuilt binary on the releases page for modern systems. Simply download and run the binary.

Alternatively, you can also install koboldcpp to the current directory by running the following terminal command:

curl -fLo koboldcpp https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64-cuda1150 && chmod +x koboldcpp

After running this command you can launch Koboldcpp from the current directory using ./koboldcpp in the terminal (for CLI usage, run with --help).

Linux Usage (koboldcpp.sh automated compiler script)

when you can't use the precompiled binary directly, we provide an automated build script which uses conda to obtain all dependencies, and generates (from source) a ready-to-use a pyinstaller binary for linux users. Simply execute the build script with ./koboldcpp.sh dist and run the generated binary. (Not recomended for systems that already have an existing installation of conda. Dependencies: curl, bzip2)

./koboldcpp.sh # This launches the GUI for easy configuration and launching (X11 required).
./koboldcpp.sh --help # List all available terminal commands for using Koboldcpp, you can use koboldcpp.sh the same way as our python script and binaries.
./koboldcpp.sh rebuild # Automatically generates a new conda runtime and compiles a fresh copy of the libraries. Do this after updating Koboldcpp to keep everything functional.
./koboldcpp.sh dist # Generate your own precompiled binary (Due to the nature of Linux compiling these will only work on distributions equal or newer than your own.)

OSX and Linux Manual Compiling

  • Otherwise, you will have to compile your binaries from source. A makefile is provided, simply run make.

  • If you want you can also link your own install of OpenBLAS manually with make LLAMA_OPENBLAS=1

  • If you want you can also link your own install of CLBlast manually with make LLAMA_CLBLAST=1, for this you will need to obtain and link OpenCL and CLBlast libraries.

    • For Arch Linux: Install cblas openblas and clblast.
    • For Debian: Install libclblast-dev and libopenblas-dev.
  • You can attempt a CuBLAS build with LLAMA_CUBLAS=1. You will need CUDA Toolkit installed. Some have also reported success with the CMake file, though that is more for windows.

  • For a full featured build (all backends), do make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_CUBLAS=1 LLAMA_VULKAN=1

  • After all binaries are built, you can run the python script with the command koboldcpp.py [ggml_model.bin] [port]

  • Note: Many OSX users have found that the using Accelerate is actually faster than OpenBLAS. To try, you may wish to run with --noblas and compare speeds.

Arch Linux Packages

There are some community made AUR packages (Maintained by @AlpinDale) available: CUBLAS, and HIPBLAS. They are intended for users with NVIDIA GPUs, and users with a supported AMD GPU. Note that these packages may be outdated, and it's probably better to use official KoboldCpp binaries.

Compiling on Windows

  • You're encouraged to use the .exe released, but if you want to compile your binaries from source at Windows, the easiest way is:
    • Use the latest release of w64devkit (https://github.com/skeeto/w64devkit). Be sure to use the "vanilla one", not i686 or other different stuff. If you try they will conflit with the precompiled libs!
    • Make sure you are using the w64devkit integrated terminal, then run 'make' at the KoboldCpp source folder. This will create the .dll files.
    • If you want to generate the .exe file, make sure you have the python module PyInstaller installed with pip ('pip install PyInstaller').
    • Run the script make_pyinstaller.bat at a regular terminal (or Windows Explorer).
    • The koboldcpp.exe file will be at your dist folder.
  • If you wish to use your own version of the additional Windows libraries (OpenCL, CLBlast and OpenBLAS), you can do it with:
    • OpenCL - tested with https://github.com/KhronosGroup/OpenCL-SDK . If you wish to compile it, follow the repository instructions. You will need vcpkg.
    • CLBlast - tested with https://github.com/CNugteren/CLBlast . If you wish to compile it you will need to reference the OpenCL files. It will only generate the ".lib" file if you compile using MSVC.
    • OpenBLAS - tested with https://github.com/xianyi/OpenBLAS .
    • Move the respectives .lib files to the /lib folder of your project, overwriting the older files.
    • Also, replace the existing versions of the corresponding .dll files located in the project directory root (e.g. libopenblas.dll).
    • You can attempt a CuBLAS build with using the provided CMake file with visual studio. If you use the CMake file to build, copy the koboldcpp_cublas.dll generated into the same directory as the koboldcpp.py file. If you are bundling executables, you may need to include CUDA dynamic libraries (such as cublasLt64_11.dll and cublas64_11.dll) in order for the executable to work correctly on a different PC.
    • Make the KoboldCPP project using the instructions above.

Compiling on Android (Termux Installation)

  • Install and run Termux from F-Droid
  • Enter the command termux-change-repo and choose Mirror by BFSU
  • Install dependencies with pkg install wget git python (plus any other missing packages)
  • Install dependencies apt install openssl (if needed)
  • Clone the repo git clone https://github.com/LostRuins/koboldcpp.git
  • Navigate to the koboldcpp folder cd koboldcpp
  • Build the project make
  • Grab a small GGUF model, such as wget https://huggingface.co/concedo/KobbleTinyV2-1.1B-GGUF/resolve/main/KobbleTiny-Q4_K.gguf
  • Start the python server python koboldcpp.py --model KobbleTiny-Q4_K.gguf
  • Connect to http://localhost:5001 on your mobile browser
  • If you encounter any errors, make sure your packages are up-to-date with pkg up
  • GPU acceleration for Termux may be possible but I have not explored it. If you find a good cross-device solution, do share or PR it.

AMD

Docker

Questions and Help

  • First, please check out The KoboldCpp FAQ and Knowledgebase which may already have answers to your questions! Also please search through past issues and discussions.
  • If you cannot find an answer, open an issue on this github, or find us on the KoboldAI Discord.

Considerations

  • For Windows: No installation, single file executable, (It Just Works)
  • Since v1.0.6, requires libopenblas, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without BLAS.
  • Since v1.15, requires CLBlast if enabled, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without CLBlast.
  • Since v1.33, you can set the context size to be above what the model supports officially. It does increases perplexity but should still work well below 4096 even on untuned models. (For GPT-NeoX, GPT-J, and LLAMA models) Customize this with --ropeconfig.
  • Since v1.42, supports GGUF models for LLAMA and Falcon
  • Since v1.55, lcuda paths on Linux are hardcoded and may require manual changes to the makefile if you do not use koboldcpp.sh for the compilation.
  • Since v1.60, provides native image generation with StableDiffusion.cpp, you can load any SD1.5 or SDXL .safetensors model and it will provide an A1111 compatible API to use.
  • I plan to keep backwards compatibility with ALL past llama.cpp AND alpaca.cpp models. But you are also encouraged to reconvert/update your models if possible for best results.

License

  • The original GGML library and llama.cpp by ggerganov are licensed under the MIT License
  • However, Kobold Lite is licensed under the AGPL v3.0 License
  • The other files are also under the AGPL v3.0 License unless otherwise stated

Notes

  • Generation delay scales linearly with original prompt length. If OpenBLAS is enabled then prompt ingestion becomes about 2-3x faster. This is automatic on windows, but will require linking on OSX and Linux. CLBlast speeds this up even further, and --gpulayers + --useclblast or --usecublas more so.
  • I have heard of someone claiming a false AV positive report. The exe is a simple pyinstaller bundle that includes the necessary python scripts and dlls to run. If this still concerns you, you might wish to rebuild everything from source code using the makefile, and you can rebuild the exe yourself with pyinstaller by using make_pyinstaller.bat
  • API documentation available at /api and https://lite.koboldai.net/koboldcpp_api
  • Supported GGML models (Includes backward compatibility for older versions/legacy GGML models, though some newer features might be unavailable):
    • All up-to-date GGUF models are supported (Mistral/Mixtral/QWEN/Gemma and more)
    • LLAMA and LLAMA2 (LLaMA / Alpaca / GPT4All / Vicuna / Koala / Pygmalion 7B / Metharme 7B / WizardLM and many more)
    • GPT-2 / Cerebras
    • GPT-J
    • RWKV
    • GPT-NeoX / Pythia / StableLM / Dolly / RedPajama
    • MPT models
    • Falcon (GGUF only)
    • Stable Diffusion and SDXL models