Release Cycle of llama.cpp

[yavornenov-ost]

Hi,

I'm looking into using the llama.cpp library and would like to better understand the project's release cycle. From what I can tell, the project doesn’t follow traditional release practices, as multiple releases can be published in a single day.

• Is there a notion of a "stable" release?
• More generally, is there any documentation or information available that describes the release cycle?

Thanks in advance!

With kind regards,
Yavor

[ggerganov]

Is there a notion of a "stable" release?

Not atm.

More generally, is there any documentation or information available that describes the release cycle?

No, we haven't put much effort into that. But it's something we should address soon.

We can start tagging certain versions as stable. The thing that is not clear to me is how to decide which versions to tag. The project is developing very fast, so it is difficult to decide when we have a stable release. Any suggestions?

[yavornenov-ost]

Thank you for the quick response, @ggerganov.

I appreciate the difficulty in establishing a formal release policy for dynamic open-source projects like llama.cpp. I don’t have direct experience with open-source governance to add much beyond the obvious here, but from a user’s perspective, a useful alternative to stable releases could be clearer differentiation between stable and experimental features/APIs, which might be easier to achieve.

Kudos to the team for the excellent work!

[Volko61]

Maybe we can do some versioning number such as v1.1.0, v1.1.1, etc ?
We upgrade the last digit (v1.1.X) when there is small changes/fix and we upgrade the second one (v1.X.0) when a new feature/big change is implemented.

This is the system used by Minecraft and other tools and it's useful because we know the most stable is in general the biggest last digit of the previous version.
Example :
v1.2.2
v1.2.1
v1.2.0
v1.1.8
v1.1.7

in this case v1.1.8 has the biggest chances to be the most stable.
It also allows to know that if you are missing a feature in v1.1.8 you can try v1.2.2

My comment is just a suggestion, I know things are much more complex than this, especially with fast growing open source projects

[wtarreau]

As someone with about 2 decades of experience dealing with stable and LTS projects, I would say that the most difficult is to start. Past that point, you'll notice that it becomes more intuitive to consider that certain versions are not worth maintaining because you feel that they bring in certain changes that have no user-facing value but only pave the way to future improvements, while others definitely improve the user experience. Also sometimes you'll find that some improvements are nice but risky or that you're uncertain to maintain them over time and you'll prefer not to engage into maintaining that in a stable version.

I made the mistake long ago to decide that a release should be tagged stable when it's good enough, and while I was on 6mo/1yr release cycles, suddenly a complicate change cause one release to take 4.5 years. Now I find it much better to focus only on rough dates for releases. You'll see that it's much easier for all contributors to know that at some point you're expecting them to slow down on features for the forthcoming release and want them to finish polishing what currently exists for the release, which doesn't prevent them from working for the next one already. It also allows to say "thanks, I'm taking it for -next" in response to some PR that you don't feel super comfortable with. And if you find that certain features are uncertain, just mark them as experimental so that nobody strongly relies on them. Experimental should cover user interface more than stability: it's always possible to stabilize things over time, however when you poorly named a command-line option or a bunch of settings to configure a subsystem, it's difficult to rename them later without breaking user setups, and that's when you should pass the message that "this will probably change but feedback is welcome", i.e. "experimental".

Since your project moves very fast, I don't see it providing an LTS branch yet. But maybe doing stable releases every 3 months each with about 4 months of maintenance would allow users to jump from a stable enough version to another one. It's always important to understand what people seek with stable versions, it's just being able to upgrade without taking too many risks. And this should be a good start in my opinion.

It's important to avoid backporting features initially, otherwise your users will stick to old versions, will benefit from their stability combined with the extra features without testing development versions. Features are what should drive users to upgrade. Maybe in a few years if you start to create long term supported versions (multi-year) then it could prove useful to occasionally backport some small stuff that makes their lives easier.

Hoping this helps, and keep up the good work!

[bandoti]

The real barrier to a proper release cadence is not a technical one really.

Ultimately the solution is to establish a roadmap using user-based "needs" that do not depend on specific features, and lump together a set of those for a given release cadence (major/minor) with a series of them in-between for patches.

For instance: "Allow toolcalls in llama-cli" might be the need, but its underlying features might vary or change before a given release. Does it use MCP or directly invoke shell? Does it use a configuration format? Is it YAML or INI?

This roadmap would determine all major/minor bumps depending on backward-compatibility breakage or new capabilities, with the expectation that bugs will be fixed in between.

Within this structure of course can exist appropriate phases of development like pre-release, design, prototype, or whatever.

All of it, however, ought to hinge on a solid roadmap.

[d-shehu]

I think this topic merits further discussion. I've noticed instability in the form of seg. faults on various releases including b7616. Seg faults yield little useful information and I'm afraid I don't know enough to debug the codebase. Granted I'm using RPC a lot which is still experimental right? But the stability seems to vary widely from build to build. Perhaps it's just me.

I'm not suggesting anything as elaborate as an LTS. But would it be possible to subject releases to more exhaustive testing either E2E with test models and utilizing a larger subset of the features.

Those versions where "all" functionality or a major subset passes can be flagged as "stable". No need for branching, back porting fixes, or anything else.

Perhaps some of us can contribute to test automation? Open to other ideas as well. Thanks.

[wtarreau]

Also more generally my perspective: at least for the code that I wrote myself I try to provide fixes ASAP.

Sure but by the time your fix gets merged, something else can have changed again in someone else's area.

If a feature is broken for an extended period of time because of a lack of attention from maintainers, a stable branch isn't going to fix that.

A stable branch is only a subset of a development branch which only contains fixes. It will not magically fix things that are broken, however it will avoid to integrate new features that are likely to cause breakage or experience changes (e.g. command line has changed multiple times).

[bandoti]

I agree it can't hurt to divide things into sub-components and track releases for each separately. For example, a change in server that doesn't affect Vulkan makes Vulkan stable compared to the last release.

Then the release step can use a couple magical Git commands to determine what changed on each "component" and write it to a file that tracks all the versions.

This is similar to various mono-repos but without the requirements lock files.

It would actually be kind of fun to use some statistics to measure the relative riskiness of a batch of changes based on line count or more complex heuristics. 🙂

[JohannesGaessler]

When I said

And for my use cases any problems I encounter usually have to do with some new model or feature that would not be available in a stable branch in the first place.

I was specifically talking about bugs. At least for my usage it is exceedingly rare that upgrading the llama.cpp version introduced new bugs for old functionality. When it comes to changes to the user interface then the question we should be asking ourselves is fundamentally different: will a stable release cycle allow the project to extract enough additional value from users to justify the opportunity cost? If the answer is no then my opinion is that users just have to deal with the changes and RTFM.

[bandoti]

will a stable release cycle allow the project to extract enough additional value from users to justify the opportunity cost?

True, it might be so, that the change velocity is necessary and keeping it how it is as a rolling release makes sense. I suppose that is where I'm thinking: if there's not sufficient time to maintain a roadmap and keep it up-to-date for major/minor release cadence, perhaps it's moving too fast to warrant such a thing!

It makes sense to bolster tests and increase stability, but if folks depend on libllama for other projects it is kind of up to them to guarantee utmost stability as well. Personally I don't mind the rolling release as things can move faster.

[am17an]

I have a more "end to end" test here which I was talking about, it gets a 48-cpu 4x GPU runner and runs llama-server for gpt-oss-20b and compares logprobs vs a reference. The GPU rented is T4, but for similar cost I should be able to get A10s or L40s or which would be able to catch CUDA graph bugs as well (Since we only run CUDA graphs >= Ampere) . There is an option to run on AMD GPUs too. Here is an example run:
https://github.com/am17an/llama.cpp/actions/runs/20753731647/job/59589868345

Currently the test runs for ~5 mins (which includes compiling llama-server with CUDA and downloading gpt-oss-20b), but it can be expanded easily to include more models + tests (e.g. llama-bench). Anything running under 10-15 mins we can run on any merged PR touching either ggml-cuda or ggml/llama.cpp core.

I plan to sync CUDA PRs to my fork to catch some of the bugs earlier as we push more multi-GPU code. Currently we don't have any multi-GPU or offload CI and we routinely break stuff there.

As such this I would design this workflow to basically test if we breaking any core functionalities users have to come expect. Both correctness-wise and possibly performance-wise.

[eiffel31]

From my experience of free software delivering, I consider that release is closely linked to good test suite.
Each feature should to be verified by a test.
Each bug report has to be turned into a test to avoid future regressions.

The quality of the whole piece is highly dependent on the test suite, in particular for rapidly evolving software. Changes can temporarily break some parts of the test suite. Then there is a fixing phase before delivering a new stable release.

Notes

• tests can test not only values but also check some speed on known hardware.
• tests have to run with asserts on and off.

Hope this helps

[wtarreau]

All of this is true, but the amount of integration tests quickly explodes, especially for rarely used backends, which is where user regression reports help a lot. In free software it's quite common for rare build regressions to be reported weeks or months after a release, and you just don't want to run rare such tests all the time, first because they cost a lot to run, second because it's hard to imagine them all. For example I faced build regressions with rocm-6.4.4 after a recent commit due to an API change in the CUDA code. When I tried to fix it, it broke somewhere else, and I finally found that adding -DHIP_COMPILER_FORCED=1 would fix all of this. I couldn't even figure what it does internally. Until such an issue is reported, it's totally unlikely to appear in a test, and once the workaround is found, it's no longer considered as a build issue and will not take test time anymore.

That's why in my opinion tests and user reports are both complementary. Tests are there to give good confidence to developers that the release is of good quality, while stable branches provide better guarantees to users that little has changed since last version, and that the issue that just broke after the last upgrade for them might now be fixed without a new one being introduced.

[eiffel31]

You are right: I was exposing general way to approach test suite. In my experience this helps a lot rising the quality over the long time.
This general approach is the global way to think, it has to be adjusted with thoughts to each specific case. In real life, many exception occurs.
PS: on the compiler suite I was working on, tests was running for more than 8 hours. This was not a problem, this was a way to be confident in the changes. We often run the full tests for the night in order to have detected problems in the morning (if any), and just run some specific parts of the test suite in the day when changing the code. Building the test suite is not just accumulating test code, this is also having a set of "tools" to be able to run some specific parts in specific modes. For this, tests have to be classified into categories that make sense.

[stefanvarunix]

I run llama.cpp in a prod environment (and other inferencing platforms).

my 2 cents:
please keep the pace!
no change needed in release mgmt! no "stable release", tagging and other stuff needed. I know this might hurt some traditional IT guys.
better fast than traditionally slow.

llama.cpp moves faster than most of the other platforms.

[eiffel31]

Just curious: what is the OS of your prod environment?