Multislot cache for a single directory
## Problem statement
```
------ X -- n -->
master
```
the way we work is we frequently switch between branch `X` that trails `n` commits between tip of `master`. Which means on every switch the cache is invalidated (as `n` is big enough for `nixpkgs` reference to usually differ or otherwise contains nix changes), a moderately-long rebuild is needed and since this is an action take multiple times a day -- it's a short-term net-negative experience for the engineers (compared to `brew` or other previous generation package managers).
## Our solution
Instead of looking at modification dates for the `watches` I created a Merkle tree of them :
```
merkle_watches="$(printf "%s\0" "${existing_watches[@]}" | xargs -0 sha512 -q | sort | sha512 -q)"
```
then create a **sub**directory (in the direnv cache directory for a particular path) based on that [hence "multislot" - there are many cache slots per a single directory now]. The cache algo becomes super simple - the subdirectory either is there or it's not [modulo some consistency/validity checks]. This solves the initial problem to our satisfaction.
Caveats:
- garbage collection is nerfed -- since there are effectively infinite slots the GC roots will accumulate over time. One can nuke the whole cache directory at that point, run gc, rebuild things. This is a feasible solutions for us.
- I'm having some issues with cache playing nice with a parent directory flake reference. Let's say I'm in `/foo/` and I reference `.#foo` from `/flake.nix` -- the env is rebuilt every time. Again, this is acceptable for us as the this usecase is significantly less frequently exercised than the one described at the top.
- It's only implemented for flakes as that's what we use
## Going forward
Due to technical reasons [I'd rather not get into] it's most feasible for us to use this as a bash "overlay" (I source the file containing the modified functions I want to change, shadowing the original functions) -- here's a generated patch, should be possible to recreate my changes with the `nix-direnv` source files https://gist.github.com/OlgierdEthon/a0d4bd38644e0294d8f2d411e15510f3
I'd be **very** happy to upstream this feature -- lack of it so far had been the biggest obstacle in nix adoption in my experience. I'm wondering what is the best way to proceed. Is this contribution welcome? Should it be gated by a flag? Are there modifications needed to ensure that it plays nicely with garbage collection or is this an acceptable tradeoff for now?
0 条评论