cryscan 600025458b Fix workflow x2.		8 hours ago
.github/workflows	Fix workflow x2.	8 hours ago

assets	Update index.zip	1 week ago

crates	Fix comment.	8 hours ago

docs	Update config.mts	2 months ago

img	update screenshot	5 months ago

.all-contributorsrc	docs: update .all-contributorsrc [skip ci]	2 months ago

.gitattributes	Use zip file as web frontend.	9 months ago

.gitignore	Update .gitignore	1 week ago

Cargo.lock	Bump `web-rwkv` to v0.8.10	9 hours ago

Cargo.toml	Bump `web-rwkv` to v0.8.10	9 hours ago

LICENSE	Initial commit	10 months ago

README.md	Allow additionally loading states.	8 hours ago

README.zh.md	Allow additionally loading states.	8 hours ago

codebase.md	Update codebase.md	2 months ago

convert_safetensors.py	Init state (#109)	2 weeks ago

flowchart.png	Add flowchart.	2 months ago

issues.md	Bump version to v0.1.7	10 months ago

package-lock.json	Bump vite from 5.0.0 to 5.1.6	2 months ago

package.json	Update package.json	2 months ago

README.md

💯AI00 RWKV Server

💯AI00 RWKV Server

AI00 RWKV Server is an inference API server for the RWKV language model based upon the web-rwkv inference engine.

It supports VULKAN parallel and concurrent batched inference and can run on all GPUs that support VULKAN. No need for Nvidia cards!!! AMD cards and even integrated graphics can be accelerated!!!

No need for bulky pytorch, CUDA and other runtime environments, it's compact and ready to use out of the box!

Compatible with OpenAI's ChatGPT API interface.

100% open source and commercially usable, under the MIT license.

If you are looking for a fast, efficient, and easy-to-use LLM API server, then AI00 RWKV Server is your best choice. It can be used for various tasks, including chatbots, text generation, translation, and Q&A.

Join the AI00 RWKV Server community now and experience the charm of AI!

QQ Group for communication: 30920262

💥Features

Based on the RWKV model, it has high performance and accuracy
Supports VULKAN inference acceleration, you can enjoy GPU acceleration without the need for CUDA! Supports AMD cards, integrated graphics, and all GPUs that support VULKAN
No need for bulky pytorch, CUDA and other runtime environments, it's compact and ready to use out of the box!
Compatible with OpenAI's ChatGPT API interface

⭕Usages

Chatbots
Text generation
Translation
Q&A
Any other tasks that LLM can do

👻Other

Based on the web-rwkv project
Model download: V5 or V6

Installation, Compilation, and Usage

📦Download Pre-built Executables

Directly download the latest version from Release
After downloading the model, place the model in the assets/models/ path, for example, assets/models/RWKV-x060-World-3B-v2-20240228-ctx4096.st
Optionally modify assets/Config.toml for model configurations like model path, quantization layers, etc.
Run in the command line
```
$ ./ai00_rwkv_server
```
Open the browser and visit the WebUI https://localhost:65530

📜(Optional) Build from Source

Install Rust

Clone this repository

$ git clone https://github.com/cgisky1980/ai00_rwkv_server.git
$ cd ai00_rwkv_server

After downloading the model, place the model in the assets/models/ path, for example, assets/models/RWKV-x060-World-3B-v2-20240228-ctx4096.st
Compile
```
$ cargo build --release
```
After compilation, run
```
$ cargo run --release
```
Open the browser and visit the WebUI https://localhost:65530

📒Convert the Model

It only supports Safetensors models with the .st extension now. Models saved with the .pth extension using torch need to be converted before use.

Download the .pth model
In the Release you could find an executable called converter. Run

$ ./converter --input /path/to/model.pth --output /path/to/model.st

If you are building from source, run

$ cargo run --release --package converter -- --input /path/to/model.pth --output /path/to/model.st

Just like the steps mentioned above, place the model in the .st model in the assets/models/ path and modify the model path in assets/Config.toml

📝Supported Arguments

--config: Configure file path (default: assets/Config.toml)
--ip: The IP address the server is bound to
--port: Running port

📙Currently Available APIs

The API service starts at port 65530, and the data input and output format follow the Openai API specification.

/api/oai/v1/models
/api/oai/models
/api/oai/v1/chat/completions
/api/oai/chat/completions
/api/oai/v1/completions
/api/oai/completions
/api/oai/v1/embeddings
/api/oai/embeddings

The following is an out-of-box example of Ai00 API invocations in Python:

import openai

class Ai00:
    def __init__(self,model="model",port=65530,api_key="JUSTSECRET_KEY") :
        openai.api_base = f"http://127.0.0.1:{port}/api/oai"
        openai.api_key = api_key
        self.ctx = []
        self.params = {
            "system_name": "System",
            "user_name": "User", 
            "assistant_name": "Assistant",
            "model": model,
            "max_tokens": 4096,
            "top_p": 0.6,
            "temperature": 1,
            "presence_penalty": 0.3,
            "frequency_penalty": 0.3,
            "half_life": 400,
            "stop": ['\x00','\n\n']
        }
        
    def set_params(self,**kwargs):
        self.params.update(kwargs)
        
    def clear_ctx(self):
        self.ctx = []
        
    def get_ctx(self):
        return self.ctx
    
    def continuation(self, message):
        response = openai.Completion.create(
            model=self.params['model'],
            prompt=message,
            max_tokens=self.params['max_tokens'],
            half_life=self.params['half_life'],
            top_p=self.params['top_p'],
            temperature=self.params['temperature'],
            presence_penalty=self.params['presence_penalty'],
            frequency_penalty=self.params['frequency_penalty'],
            stop=self.params['stop']
        )
        result = response.choices[0].text
        return result
    
    def append_ctx(self,role,content):
        self.ctx.append({
            "role": role,
            "content": content
        })
        
    def send_message(self, message,role="user"):
        self.ctx.append({
            "role": role,
            "content": message
        })
        result = openai.ChatCompletion.create(
            model=self.params['model'],
            messages=self.ctx,
            names={
                "system": self.params['system_name'],
                "user": self.params['user_name'],
                "assistant": self.params['assistant_name']
            },
            max_tokens=self.params['max_tokens'],
            half_life=self.params['half_life'],
            top_p=self.params['top_p'],
            temperature=self.params['temperature'],
            presence_penalty=self.params['presence_penalty'],
            frequency_penalty=self.params['frequency_penalty'],
            stop=self.params['stop']
        )
        result = result.choices[0].message['content']
        self.ctx.append({
            "role": "assistant",
            "content": result
        })
        return result
    
ai00 = Ai00()
ai00.set_params(
    max_tokens = 4096,
    top_p = 0.55,
    temperature = 2,
    presence_penalty = 0.3,
    frequency_penalty = 0.8,
    half_life = 400,
    stop = ['\x00','\n\n']
)
print(ai00.send_message("how are you?"))
print(ai00.send_message("me too!"))
print(ai00.get_ctx())
ai00.clear_ctx()
print(ai00.continuation("i like"))