Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
turboderp 009424a6d4 | 5 days ago | |
---|---|---|
.github | 5 days ago | |
conversion | 1 week ago | |
doc | 1 month ago | |
examples | 6 days ago | |
exllamav2 | 6 days ago | |
tests | 1 week ago | |
util | 2 months ago | |
.gitignore | 3 months ago | |
LICENSE | 8 months ago | |
MANIFEST.in | 8 months ago | |
README.md | 1 month ago | |
convert.py | 1 week ago | |
model_diff.py | 4 months ago | |
requirements.txt | 1 month ago | |
setup.py | 6 days ago | |
test_inference.py | 1 week ago |
ExLlamaV2 is an inference library for running local LLMs on modern consumer GPUs.
Some quick tests to compare performance with V1. There may be more performance optimizations in the future, and
speeds will vary across GPUs, with slow CPUs still being a potential bottleneck:
Model | Mode | Size | grpsz | act | 3090Ti | 4090 |
---|---|---|---|---|---|---|
Llama | GPTQ | 7B | 128 | no | 177 t/s | 198 t/s |
Llama | GPTQ | 13B | 128 | no | 109 t/s | 111 t/s |
Llama | GPTQ | 33B | 128 | yes | 44 t/s | 48 t/s |
OpenLlama | GPTQ | 3B | 128 | yes | 252 t/s | 283 t/s |
CodeLlama | EXL2 4.0 bpw | 34B | - | - | 44 t/s | 50 t/s |
Llama2 | EXL2 3.0 bpw | 7B | - | - | 211 t/s | 245 t/s |
Llama2 | EXL2 4.0 bpw | 7B | - | - | 179 t/s | 207 t/s |
Llama2 | EXL2 5.0 bpw | 7B | - | - | 159 t/s | 170 t/s |
Llama2 | EXL2 2.5 bpw | 70B | - | - | 33 t/s | 37 t/s |
TinyLlama | EXL2 3.0 bpw | 1.1B | - | - | 623 t/s | 730 t/s |
TinyLlama | EXL2 4.0 bpw | 1.1B | - | - | 560 t/s | 643 t/s |
To install from the repo you'll need the CUDA Toolkit and either gcc on Linux or (Build Tools for) Visual Studio
on Windows). Also make sure you have an appropriate version of PyTorch,
then run:
git clone https://github.com/turboderp/exllamav2
cd exllamav2
# Optionally, create and activate a new conda environment
pip install -r requirements.txt
pip install .
python test_inference.py -m <path_to_model> -p "Once upon a time,"
# Append the '--gpu_split auto' flag for multi-GPU inference
A simple console chatbot is included. Run it with:
python examples/chat.py -m <path_to_model> -mode llama
# Append the '--gpu_split auto' flag for multi-GPU inference
The -mode
argument chooses the prompt format to use. llama
is for the Llama(2)-chat finetunes, while codellama
probably works better for CodeLlama-instruct. raw
will produce a simple chatlog-style chat that works with base
models and various other finetunes. Run with -modes
for a list of all available prompt formats. You can also provide
a custom system prompt with -sp
.
TabbyAPI is a FastAPI-based server that provides an OpenAI-style web API
compatible with SillyTavern and other frontends.
ExUI is a simple, standalone single-user web UI that serves an ExLlamaV2 instance
directly with chat and notebook modes.
text-generation-webui supports ExLlamaV2 through the exllamav2
and exllamav2_HF loaders.
lollms-webui supports ExLlamaV2 through the exllamav2 binding.
To install the current dev version, clone the repo and run the setup script:
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .
By default this will also compile and install the Torch C++ extension (exllamav2_ext
) that the library relies on.
You can skip this step by setting the EXLLAMA_NOCOMPILE
environment variable:
EXLLAMA_NOCOMPILE= pip install .
This will install the "JIT version" of the package, i.e. it will install the Python components without building the
C++ extension in the process. Instead, the extension will be built the first time the library is used, then cached in
~/.cache/torch_extensions
for subsequent use.
Releases are available here, with prebuilt wheels that contain the
extension binaries. Make sure to grab the right version, matching your platform, Python version (cp
) and CUDA version.
Either download an appropriate wheel or install directly from the appropriate URL:
pip install https://github.com/turboderp/exllamav2/releases/download/v0.0.12/exllamav2-0.0.12+cu121-cp311-cp311-linux_x86_64.whl
The py3-none-any.whl
version is the JIT version which will build the extension on first launch. The .tar.gz
file
can also be installed this way, and it will build the extension while installing.
A PyPI package is available as well. It can be installed with:
pip install exllamav2
The version available through PyPI is the JIT version (see above). Still working on a solution for distributing
prebuilt wheels via PyPI.
ExLlamaV2 supports the same 4-bit GPTQ models as V1, but also a new "EXL2" format. EXL2 is based on the same
optimization method as GPTQ and supports 2, 3, 4, 5, 6 and 8-bit quantization. The format allows for mixing quantization
levels within a model to achieve any average bitrate between 2 and 8 bits per weight.
Moreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse
quantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets
ExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on
performance.
Parameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization
error (with respect to the chosen calibration data) for each of a number of possible settings, per layer. Finally, a
combination is chosen that minimizes the maximum quantization error over the entire model while meeting a target
average bitrate.
In my tests, this scheme allows Llama2 70B to run on a single 24 GB GPU with a 2048-token context, producing coherent
and mostly stable output with 2.55 bits per weight. 13B models run at 2.65 bits within 8 GB of VRAM, although currently
none of them uses GQA which effectively limits the context size to 2048. In either case it's unlikely that the model
will fit alongside a desktop environment. For now.
A script is provided to quantize models. Converting large models can be somewhat slow, so be warned. The conversion
script and its options are explained in detail here
A test community is provided at https://discord.gg/NSFwVuCjRq
Quanting service free of charge is provided at #bot test. The computation is generiously provided by the Bloke powered by Lambda labs.
I've uploaded a few EXL2-quantized models to Hugging Face to play around with, here.
LoneStriker provides a large number of EXL2 models on Hugging Face.
bartowski has some more EXL2 models on HF.
No Description
Python Text Cuda C++ C
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》