For the complete documentation index, see llms.txt. This page is also available as Markdown.

AIModelDtype

Only available on the web implementation

auto

fp32

fp16

q8

int8

uint8

q4

bnb4

q4f16

Last updated