## What's Changed * Fix link to troubleshooting guide by @rsanheim in https://github.com/Mozilla-Ocho/llamafile/pull/740 * Preserve URL path when building relative URLs in JS by @dmcardle in https://github.com/Mozilla-Ocho/llamafile/pull/733 * Add Plaintext output option to LocalScore + Respect NO_COLOR env var by @cjpais in https://github.com/Mozilla-Ocho/llamafile/pull/742 * Update README.md, fix llama 8B table stats by @cjrh in https://github.com/Mozilla-Ocho/llamafile/pull/745 * Add phi4 support by @cjpais in https://github.com/Mozilla-Ocho/llamafile/pull/744 * Qwen3 Support by @cjpais in https://github.com/Mozilla-Ocho/llamafile/pull/743 ## New Contributors * @rsanheim made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/740 * @dmcardle made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/733 * @cjrh made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/745 **Full Changelog**: https://github.com/Mozilla-Ocho/llamafile/compare/0.9.2...0.9.3

# llamafile v0.9.2 ## Llamafile Llamafile v0.9.2 is a significant release. It adds support for: * DeepSeek Distil R1 Models * Gemma 3 * IBM Granite ## LocalScore In addition 0.9.2 introduces LocalScore, a benchmarking utility and [website](https://localscore.ai). **LocalScore** is an open-source tool that both benchmarks how fast Large Language Models (LLMs) run on your specific hardware and serves as a repository for these results. We created LocalScore to provide a simple, portable way to evaluate computer performance across various LLMs while making it easy to share and browse hardware performance data. LocalScore is now part of the release of Llamafile under the new CLI utility `localscore` You can run it `./localscore -m <model>`. It is also included in every llamafile so you can benchmark models on your hardware easily using `./llamafile --localscore`. ## What's Changed * [llamafiler] doc/v1_chat_completions.md: remove duplicate entry by @mseri in https://github.com/Mozilla-Ocho/llamafile/pull/607 * Update server readme with code completion (FIM) example by @heaversm in https://github.com/Mozilla-Ocho/llamafile/pull/637 * URL constructor to get a clean url_prefix (fix #640) by @sizvix in https://github.com/Mozilla-Ocho/llamafile/pull/641 * Fix translation bug from cpp to js in TS highlight by @emilbayes in https://github.com/Mozilla-Ocho/llamafile/pull/681 * Add whisperfile server documentation by @alonsosilvaallende in https://github.com/Mozilla-Ocho/llamafile/pull/707 * Unify button look and rearrange buttons to make them more compact by @corebonts in https://github.com/Mozilla-Ocho/llamafile/pull/712 * add stable-diffusion.cpp to install target (fix #580) by @rgroesslinger in https://github.com/Mozilla-Ocho/llamafile/pull/635 * Improve OpenAI compatibility for /v1/* endpoints by @corebonts in https://github.com/Mozilla-Ocho/llamafile/pull/713 * Update WSL troubleshooting in README.md by @halter73 in https://github.com/Mozilla-Ocho/llamafile/pull/585 * Granite three support by @gabe-l-hart in https://github.com/Mozilla-Ocho/llamafile/pull/608 * Initial support for Gemma 3 models by @corebonts in https://github.com/Mozilla-Ocho/llamafile/pull/717 * Add copy and info buttons to the chat window and improve small screen UX by @corebonts in https://github.com/Mozilla-Ocho/llamafile/pull/719 * Avoid streaming incomplete UTF-8 characters by @corebonts in https://github.com/Mozilla-Ocho/llamafile/pull/727 * Introduce LocalScore CLI by @cjpais in https://github.com/Mozilla-Ocho/llamafile/pull/734 ## New Contributors * @mseri made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/607 * @heaversm made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/637 * @sizvix made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/641 * @emilbayes made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/681 * @alonsosilvaallende made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/707 * @corebonts made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/712 * @rgroesslinger made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/635 * @halter73 made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/585 * @gabe-l-hart made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/608 **Full Changelog**: https://github.com/Mozilla-Ocho/llamafile/compare/0.9.1...0.9.2

This release adds support for Deepseek Distil models. It improves some documentation, and fixes a segfault when running with an Nvidia GPU. ## What's Changed * Update Makefile: Fix PHONY from check to cosmocc and cosmocc-ci respectively by @mofosyne in https://github.com/Mozilla-Ocho/llamafile/pull/683 * Updated README to reflect WSL 2 command for Windows 11 by @peteski22 in https://github.com/Mozilla-Ocho/llamafile/pull/685 * Add Support for DeepSeek-R1 models by @Xydane in https://github.com/Mozilla-Ocho/llamafile/pull/687 * Revert Cosmopolitan to 3.9.7 by @cjpais in https://github.com/Mozilla-Ocho/llamafile/pull/705 ## New Contributors * @peteski22 made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/685 * @Xydane made their first contribution in https://github.com/Mozilla-Ocho/llamafile/pull/687 **Full Changelog**: https://github.com/Mozilla-Ocho/llamafile/compare/0.9.0...0.9.1

We've solved all the known issues with the new llamafiler server, and improved its web gui. In addition to the llamafiler binary below, the new server is available as `llamafile --server --v2`. Its manual can be accessed via `llamafile --server --v2 --help`. - e64c7e2 Include llamafiler in llamafile binary - a8fd4d2 Improve management of multiple slots - 4158265 Show progress bar for prompt processing in web ui - 38677b5 Support relocating matching suffixes in KV cache - 956e62c Visually indicate messages truncated by context - 08e7a21 Forget old messages when running out of context - 59a5d97 8fa1702 Make pledge() security not break things - 1fc35e2 Add upload button and support text files - 43bc1eb fe514ef Improve buttons in web ui This change upgrades our cosmocc toolchain, whose recent release has fixed all known issues and made performance improvements to memory allocation. See the [cosmopolitan releases](https://github.com/jart/cosmopolitan/releases) page. - c293359 Upgrade to Cosmopolitan v4.0.2 The following improvements have been made to the terminal `--chat` bot. - f51e535 Fix emoji editing in chatbot Additional changes: - 9b03e32 Call appropriate hip api's (#651)

llamafiler has a new web UI which supports two modes of operation: chatbot and raw completion. Its syntax highlighting is just as advanced as the CLI chatbot. It looks much nicer than the old web ui. In a future release, llamafiler will be folded into llamafile to replace the old server. - 988c9ec Introduce raw completions web ui - 241bf21 Introduce /v1/completions endpoint in new server - 6d89f8f Add binary safety check to server - d18ddf1 Add redo button to new web ui - bc82424 Add settings modal to web ui - bb917bd Add vision model support to new server - 4c7b7d5 Implement data URI parser - fb4b3e6 Fix JSON parser bug - 9d6f89f Improve look and printability of new web ui - 25b6910 Make chatbot ui more printer friendly - 30518ca Respond to HTTP OPTIONS requests - 41abfa3 Work around multiple image handling - 35bc088 Make default system prompt configurable on web - 28c8e22 Scale and decimate images as needed in browser - 14713b5 Get basic chatbot web gui working in llamafiler - ef08074 Start porting syntax highlighter to JavaScript - fdfdb13 Port remaining highlighting code to javascript The following improvements have been made to our terminal chatbot. - 12c3761 Make CLI chatbot work better with base models - e5c0921 Improve VT100 support - 4b61791 Fix VT102 support - d25c077 Introduce /upload and /forget commands to chatbot - 880ebc7 Handle empty system prompt better in cli chatbot General improvements to this project. - f581c40 Fix futex prototype - 54d3c72 Make LLaVA fast again - 01b8d49 Remove n-gpu-layer limitation (#534) - 566cdc1 Improve Gemma system prompt generation - 46284fe Reduce attack surface of stb_image - 9bb262b Log CUDA kernel vs. runtime versions Syntax highlighting improvements for chatbot and web ui. - d979a1c Add BNF syntax highlighting - 4a8311a Add cmake syntax highlighting - 40e92cf Add Ocaml syntax highlighting - 0995343 Add more Clojure keywords - 0068a37 Make D syntax highlighting better - 0965a4b Make some markdown improvements - 9b96502 Improve JS/HTML syntax highlighting - c0622da Put more work into markdown rendering - fa1c98f Improve markdown to html rendering - 8915432 Further improve markdown to html - d25fa3a Improve highlighting in new web ui - f5a0bd4 Fix JS regex highlighting issue - 2807ae6 Improve Ada syntax highlighting - d30da30 Syntax highlight D properly - 33a057e Improve Ruby some more - 5b0fff1 Improve Ruby syntax highlighting - 8413a21 Fix Ruby builtins in web gui The latest cosmopolitan upgrade introduces a new more powerful syntax for your .args files. They're now parsed more similarly to the shell, with support for C style escaping in double-quoted strings. You can also now add shell-style comments to .args files too. See tool/args/args2.c in the cosmopolitan codebase for the definitive reference. - fb59488 Upgrade to Cosmo v3.9.7 - 21af0bf Import upstream bestline changes The following example of the new .args file syntax is provided: ```sh # specify model -m Qwen2.5-Coder-34B-Instruct.Q6_K.gguf # prevent flags below from being changed ... # specify system prompt --system-prompt "\ you are a friendly ai assistant\n your job is to be helpful and intelligent" # hide some stuff from user interfaces --nologo --no-display-prompt ``` You can put .args files inside llamafile, llamafiler, and whisperfile using the zipalign program. The following screenshots are provided of the llamafiler web ui.    

- Add Julia syntax highlighting support - Fix possible crash on Windows due to MT bug - Improve accuracy of chatbot context window management - The new `llamafiler` server now supports GPU. Pass the `-ngl 999` flag. - The new `llamafiler` server's `/v1/chat/completions` endpoint now supports prompt caching. It may be configured using the `--slots COUNT` and `--ctx-size TOKENS` flags.

The `--chat` bot interface now supports syntax highlighting 42 separate programming languages: ada, asm, basic, c, c#, c++, cobol, css, d, forth, fortran, go, haskell, html, java, javascript, json, kotlin, ld, lisp, lua, m4, make, markdown, matlab, pascal, perl, php, python, r, ruby, rust, scala, shell, sql, swift, tcl, tex, txt, typescript, and zig. That chatbot now supports more commands: - `/undo` may be used to have the LLM forget the last thing you said. This is useful when you get a poor response and want to try asking your question a different way, without needing to start the conversation over from scratch. - `/push` and `/pop` works similarly, in the sense that it allows you to rewind a conversation to a previous state. In this case, it does so by creating save points within your context window. Additionally, `/stack` may be used to view the current stack. - `/clear` may be used to reset the context window to the system prompt, effectively starting your conversation over. - `/manual` may be used to put the chat interface in "manual mode" which lets you (1) inject system prompts, and (2) speak as the LLM. This could be useful in cases where you want the LLM to believe it said something when it actually didn't. - `/dump` may be used to print out the raw conversation history, including special tokens (that may be model specific). You can also say `/dump filename.txt` to save the raw conversation to a file. We identified an issue with Google's Gemma models, where the chatbot wasn't actually inserting the system prompt. That's now fixed. So you can now instruct Gemma to do roleplaying if you pass the flags `llamafile -m gemma.gguf -p "you are role playing as foo" --chat`. You can now type CTRL-J to create multi-line prompts in the terminal chatbot. It works similarly to shift-enter in the browser. It can be a quicker alternative to using the chatbot's triple quote syntax, i.e. `"""multi-line / message"""`. Bugs in the new chatbot have been fixed. For example, we now do a better job making sure special tokens like BOS, EOS, and EOT get inserted when appropriate into the conversation history. This should improve fidelity when using the terminal chatbot interface. The `--threads` and `--threads-batch` flags may now be used separately to tune how many threads are used for prediction and prefill. The llamafile-bench command now supports benchmarking GPU support (see #581 from @cjpais) Both servers now support configuring a URL prefix, thanks to (see #597 and #604 from @vlasky) Support for the IQ quantization formats is being removed from our CUDA module to save on build times. If you want to use IQ quants with your NVIDIA hardware, you need to pass the `--iq --recompile` flags to llamafile once, to build a ggml-cuda module for your system that includes them. Finally, we have an alpha release of a new `/v1/chat/completions` endpoint for the new `llamafiler` server. We're planning to build a new web interface that's based on this soon, so you're encouraged to test this, since llamafiler will eventually replace the old server too. File an issue if there's any features you need.

<img src="https://github.com/user-attachments/assets/60c12d26-d087-4227-ab71-0876b50a48d2" width="300"> **llamafile lets you distribute and run LLMs with a single file** llamafile is a local LLM inference tool introduced by Mozilla Ocho in Nov 2023, which offers superior performance and binary portability to the stock installs of six OSes without needing to be installed. It features the best of [llama.cpp](https://github.com/ggerganov/llama.cpp/) and [cosmopolitan libc](https://github.com/jart/cosmopolitan) while aiming to stay ahead of the curve by including the most cutting-edge performance and accuracy enhancements. What llamafile gives you is a fun web GUI chatbot, a turnkey OpenAI API compatible server, and a [shell-scriptable](https://justine.lol/oneliners/) CLI interface which together put you in control of artificial intelligence. ### v0.8.14 changes This release introduces our new CLI chatbot interface. It supports multi-line input using triple quotes. It will syntax highlight Python, C, C++, Java, and JavaScript code. <img src="https://github.com/user-attachments/assets/485324d7-320e-4cc0-aec7-1cefe5ab8447" width="600"> This chatbot is now the default mode of operation. When you launch llamafile without any special arguments, the chatbot will be launched in the foreground, and the server will be launched in the background. You can use the `--chat` and `--server` flags to disambiguate this behavior if you only want one of them. - a384fd7 Create ollama inspired cli chatbot - 63205ee Add syntax highlighting to chatbot - 7b395be Introduce new --chat flag for chatbot - 28e98b6 Show prompt loading progress in chatbot - 4199dae Make chat+server hybrid the new default mode The whisperfile server now lets you upload mp3/ogg/flac. - 74dfd21 Rewrite audio file loader code - 7517a5f whisperfile server: convert files without ffmpeg (#568) Other improvements have been made. - d617c0b Added vision support to api_like_OAI (#524) - 726f6e8 Enable gpu support in llamafile-bench (#581) - c7c4d65 Speed up KV in llamafile-bench - 2c940da Make replace_all() have linear complexity - fa4c4e7 Use bf16 kv cache when it's faster - 20fe696 Upgrade to Cosmopolitan 3.9.4 - c44664b Always favor fp16 arithmetic in tinyBLAS - 98eff09 Quantize TriLM models using Q2_K_S (#552)