Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
omahs 04976db7a8 | 1 week ago | |
---|---|---|
.. | ||
README.md | 1 week ago | |
arithmetic.gbnf | 9 months ago | |
c.gbnf | 8 months ago | |
chess.gbnf | 9 months ago | |
japanese.gbnf | 9 months ago | |
json.gbnf | 2 months ago | |
json_arr.gbnf | 2 months ago | |
list.gbnf | 9 months ago |
GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp
. For example, you can use it to force the model to generate valid JSON, or speak only in emojis. GBNF grammars are supported in various ways in examples/main
and examples/server
.
Bakus-Naur Form (BNF) is a notation for describing the syntax of formal languages like programming languages, file formats, and protocols. GBNF is an extension of BNF that primarily adds a few modern regex-like features.
In GBNF, we define production rules that specify how a non-terminal (rule name) can be replaced with sequences of terminals (characters, specifically Unicode code points) and other non-terminals. The basic format of a production rule is nonterminal ::= sequence...
.
Before going deeper, let's look at some of the features demonstrated in grammars/chess.gbnf
, a small chess notation grammar:
# `root` specifies the pattern for the overall output
root ::= (
# it must start with the characters "1. " followed by a sequence
# of characters that match the `move` rule, followed by a space, followed
# by another move, and then a newline
"1. " move " " move "\n"
# it's followed by one or more subsequent moves, numbered with one or two digits
([1-9] [0-9]? ". " move " " move "\n")+
)
# `move` is an abstract representation, which can be a pawn, nonpawn, or castle.
# The `[+#]?` denotes the possibility of checking or mate signs after moves
move ::= (pawn | nonpawn | castle) [+#]?
pawn ::= ...
nonpawn ::= ...
castle ::= ...
Non-terminal symbols (rule names) stand for a pattern of terminals and other non-terminals. They are required to be a dashed lowercase word, like move
, castle
, or check-mate
.
Terminals are actual characters (code points). They can be specified as a sequence like "1"
or "O-O"
or as ranges like [1-9]
or [NBKQR]
.
Terminals support the full range of Unicode. Unicode characters can be specified directly in the grammar, for example hiragana ::= [ぁ-ゟ]
, or with escapes: 8-bit (\xXX
), 16-bit (\uXXXX
) or 32-bit (\UXXXXXXXX
).
Character ranges can be negated with ^
:
single-line ::= [^\n]+ "\n"`
The order of symbols in a sequence matters. For example, in "1. " move " " move "\n"
, the "1. "
must come before the first move
, etc.
Alternatives, denoted by |
, give different sequences that are acceptable. For example, in move ::= pawn | nonpawn | castle
, move
can be a pawn
move, a nonpawn
move, or a castle
.
Parentheses ()
can be used to group sequences, which allows for embedding alternatives in a larger rule or applying repetition and optional symbols (below) to a sequence.
*
after a symbol or sequence means that it can be repeated zero or more times.+
denotes that the symbol or sequence should appear one or more times.?
makes the preceding symbol or sequence optional.Comments can be specified with #
:
# defines optional whitespace
ws ::= [ \t\n]+
Newlines are allowed between rules and between symbols or sequences nested inside parentheses. Additionally, a newline after an alternate marker |
will continue the current rule, even outside of parentheses.
In a full grammar, the root
rule always defines the starting point of the grammar. In other words, it specifies what the entire output must match.
# a grammar for lists
root ::= ("- " item)+
item ::= [^\n]+ "\n"
This guide provides a brief overview. Check out the GBNF files in this directory (grammars/
) for examples of full grammars. You can try them out with:
./main -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'
Grammars currently have performance gotchas (see https://github.com/ggerganov/llama.cpp/issues/4218).
A common pattern is to allow repetitions of a pattern x
up to N times.
While semantically correct, the syntax x? x? x?.... x?
(with N repetitions) will result in extremely slow inference. Instead, you can write (x (x (x ... (x)?...)?)?)?
(w/ N-deep nesting)
No Description
C++ C Cuda Python Metal other
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》