llama.cpp

Branch: master

Haoxiang Fei f99e1e456e llama : lookup word in vocab before doing BPE merges (#7193 ) * fix: llama-3 ignore_merges * test: add test for llama-3 bpe ignore_merges * fix: set ignore_merges only for llama-3 * fix: test-tokenizer-1-bpe --ingore-merges detection * fix: copy to fix fallthrough * fix: change ignore_merges to bool * fix: add ignore merges tests to cmake * llama : alternative merge ignore logic --------- Co-authored-by: Haoxiang Fei <feihaoxiang@idea.edu.cn> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>		5 days ago
..
.editorconfig	gguf : new file format with flexible meta data (beta) (#2398)	8 months ago

ggml-vocab-aquila.gguf	Work on the BPE tokenizer (#3252)	7 months ago

ggml-vocab-baichuan.gguf	Add more tokenizer tests (#3742)	6 months ago

ggml-vocab-bert-bge.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-bert-bge.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-bert-bge.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-command-r.gguf	command-r : add BPE pre-tokenization (#7063)	1 week ago

ggml-vocab-command-r.gguf.inp	command-r : add BPE pre-tokenization (#7063)	1 week ago

ggml-vocab-command-r.gguf.out	command-r : add BPE pre-tokenization (#7063)	1 week ago

ggml-vocab-deepseek-coder.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-deepseek-coder.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-deepseek-coder.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-deepseek-llm.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-deepseek-llm.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-deepseek-llm.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-falcon.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-falcon.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-falcon.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-gpt-2.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-gpt-2.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-gpt-2.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-gpt-neox.gguf	Add more tokenizer tests (#3742)	6 months ago

ggml-vocab-gpt2.gguf	gpt2 : Add gpt2 architecture integration (#4555)	4 months ago

ggml-vocab-llama-bpe.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-llama-bpe.gguf.inp	llama : lookup word in vocab before doing BPE merges (#7193)	5 days ago

ggml-vocab-llama-bpe.gguf.out	llama : lookup word in vocab before doing BPE merges (#7193)	5 days ago

ggml-vocab-llama-spm.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-llama-spm.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-llama-spm.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-mpt.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-mpt.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-mpt.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-phi-3.gguf	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-phi-3.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-phi-3.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-qwen2.gguf	llama : add BPE pre-tokenization for Qwen2 (#7114)	1 week ago

ggml-vocab-qwen2.gguf.inp	llama : add BPE pre-tokenization for Qwen2 (#7114)	1 week ago

ggml-vocab-qwen2.gguf.out	llama : add BPE pre-tokenization for Qwen2 (#7114)	1 week ago

ggml-vocab-refact.gguf	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-refact.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-refact.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-stablelm.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-starcoder.gguf	llama : fix BPE pre-tokenization (#6920)	2 weeks ago

ggml-vocab-starcoder.gguf.inp	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

ggml-vocab-starcoder.gguf.out	tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)	1 week ago

No Description

C++ C Cuda Python Metal other

ggerganov@gmail.com slarengh@gmail.com johannesg@5d6.de 48489457+ikawrakow@users.noreply.github.com pierrick.hymbert@gmail.com 44031344+KerfuffleV2@users.noreply.github.com cebtenzzre@gmail.com daniel.bevenius@gmail.com jared@nomic.ai stephan@walter.name thichthat@gmail.com pavol@rusnak.io iainst0409@gmail.com howard0su@gmail.com anzz1@live.com 2141330+slaren@users.noreply.github.com jianyu.zhang@intel.com sergei.kozlukov@aalto.fi picard12@live.de henv@hot.ee DannyDaemonic@gmail.com Green-Sky@users.noreply.github.com karolek1231456@gmail.com dranger003@gmail.com 87009434+AidanBeltonS@users.noreply.github.com

How to access data resources in code

Contributors (25+) All

Contributors (25+)
All