[Blog](/blog/.md) # Cutting objgit’s RAM usage Agents hammer git servers, so every byte of RAM counts. Here's how profiling pushes with pprof cut objgit's memory use when pushing golang/go. Author [![Xe Iaso](https://avatars.githubusercontent.com/u/529003?v=4)](https://xeiaso.net) [Xe Iaso](https://xeiaso.net) Senior Cloud Whisperer Published2026-10-22 Reading8 min Tags [Engineering](/blog/tags/engineering/.md)[git](/blog/tags/git/.md)[objgit](/blog/tags/objgit/.md)+2 Contents * [How I measured it](#how-i-measured-it) * [Where the memory went](#where-the-memory-went) * [Two smaller fires](#two-smaller-fires) * [Git delta chains hit a recursion limit with a single byte typo](#git-delta-chains-hit-a-recursion-limit-with-a-single-byte-typo) * [The pack index needed memory optimizations](#the-pack-index-needed-memory-optimizations) * [What's next](#whats-next) When a human works with a git repository, they tend to sync changes, do work locally, batch up new work, and then synchronize when they’re done. Agents don’t do this, they do pushes and pulls whenever the tokens demand. This ends up creating a lot of unpredictable demand on git servers and is probably the reason that so many companies are making their own git services. Graphs of this end up looking like this kind of terrifying nonsense that GitHub published recently: ![Line chart titled Git events per repository, by percentile for August 2026, on a logarithmic y-axis. Fetches, pushes, clones, and all requests stay under 100 events per repository up to P75, then climb steeply, with all requests reaching about 1 billion events at P100.](/blog/assets/images/github-git-ops-2246121a7726328e7e460b173876df38.webp) That is logarithmic scale. That means the top git repos on GitHub are doing hundreds of git operations *per second*. As such, git servers need to be able to handle ridiculous amounts of punishment without falling over. Currently the most expensive part of server loadouts is memory. As such, making sure objgit doesn’t use significantly more memory than the git command you’re going to compare it to is important. Given that this started as a personal prototype that’s become my job somehow, I’m fairly certain that there’s lots of low hanging memory fruit. I’m also certain of that because I fixed a huge bug that cut down memory usage to ridiculous levels. Today is the story of how I made objgit use hilariously less RAM by doing the boring step of profiling, watching for RAM usage spikes, and figuring out ways to work around it. This is how you do optimization. Enjoy! ## How I measured it[​](#how-i-measured-it "Direct link to How I measured it") When I think about git repos, I tend to break them into four basic categories: 1. Very small repos such as throwaway projects or newly started projects. This includes objgit and a bunch of the other demos I’ve made for my job. 2. Normally sized git repos such as your organization’s money generator. This includes the Tigris blog, our internal monorepo, and more along that size. 3. “Big” open source repos such as the Go compiler and other things along that line. Think repos that get more than a few contributions per hour. 4. The titans of open source like the linux kernel, gcc, glibc, nixpkgs, or other stress tests of the Git data model. Ideally, I want objgit to support repos of all those scales. Most of the repositories I personally maintain fit into the smaller two categories, but supporting those upper two tiers requires a lot more fiddly work. In order to benchmark RAM usage, I decided on this methodology: * Break pushes into a few phases and collect memory profiles on entering/exiting each phase. * Time how long each phase took. * Once the process is done, take a memory profile, wait a few minutes for things to settle, and then take another memory profile. * Open the memory profiles and see which parts of the source code use the most memory. * Revisit life decisions. * Also every time objgit uses a new record amount of RAM, record another memory profile. The rest was really adding a bunch of printf debug statements and littering [pprof calls](https://pkg.go.dev/runtime/pprof) throughout the push path. Then it was scripting the [pprof command](https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/) and doing fairly boring analysis: 1. Use `go tool pprof` to view the `top10` entries in RAM use. 2. Look down the tree to figure out what the source is. 3. Look at the code and think about what went wrong. ## Where the memory went[​](#where-the-memory-went "Direct link to Where the memory went") Per phase, the RAM usage looks like this: FIG 01Heap per push phase in v1.13.0, pushing Xe/x ``` peak live wall MiB MiB ms setup 0 0 1 receive ████████ 107 24 616 index ████████████████████████ 340 279 321 plan ███████████████████████████████████████ 551 96 5,557 write ██████████████████████████ 357 80 2,819 upload ██████ 82 72 1 ██ still live after the phase ██ peak heap, freed by the end of the phase ``` In v1.13.0 (before fixes), the index phase kept 279 MiB live after it finished indexing the scratch packfiles. When you push a git repo to objgit, it stages all the things you just pushed to a scratch repo in a temporary folder. It goes there because I naïvely thought that the git format is the native format for git repos, it would be a good idea to store temporary data in that format. This probably would have been better if I turned on “low memory” mode, but the exact way I did it was kinda wasteful in ways that scaled poorly as repos got bigger. All told, that made RAM use grow unboundedly against the number of objects being pushed to the relevant git repo with no real end in sight. In testing this got my development box (128 GiB of RAM) to swap thrash to the point I had to unplug it to reboot it. Fixing that alone cut a push of Xe/x from 617 MiB to 199 MiB RSS, and from 9.5 s to 1.2 s: FIG 02Pushing Xe/x, v1.13.0 against v1.13.1 ``` wall time peak heap read pack receive + index → scan old ██ 937 ms ███████████ 340 MiB new █ 364 ms ▏ 5 MiB build containers plan + write → resolve old ██████████████████ 8,376 ms ██████████████████ 551 MiB new ██ 826 ms ██████ 185 MiB upload old ▏ 1 ms ███ 82 MiB new ▏ 4 ms ██ 75 MiB whole push old ████████████████████ 9.5 s ████████████████████ 617 MiB RSS new ███ 1.2 s ██████ 199 MiB RSS ██ old v1.13.0, scratch go-git storage ██ new v1.13.1, objgit's own index-pack ``` The easy fix was to implement the equivalent of `git index-pack` so that pushing has a place to store pack metadata and it doesn’t have to recursively figure it out on demand over and over. This also puts objgit at roughly 2× git’s peak RSS: FIG 03objgitd v1.13.1 against git index-pack ``` wall time peak RSS Xe/x objgitd █ 1.2 s ██████ 199 MiB golang/go objgitd ████████████████ 33.2 s ████████████████ 518 MiB git index-pack █████ 9.7 s ████████░ 259–277 MiB git index-pack --threads=1 ███████████ 23.4 s ██████ 188 MiB git index-pack uses its default thread count unless --threads says otherwise ``` Obviously there’s a lot of room for improvement, and I expect to slowly poke at things as they become problems and I get additional ideas for optimization. ## Two smaller fires[​](#two-smaller-fires "Direct link to Two smaller fires") Along the way I found a few other problems that were big enough to be kind of annoying, but not big enough to be their own dedicated blogpost. ### Git delta chains hit a recursion limit with a single byte typo[​](#git-delta-chains-hit-a-recursion-limit-with-a-single-byte-typo "Direct link to Git delta chains hit a recursion limit with a single byte typo") One of the ongoing themes of working on objgit is the constant oscillation of “Git stores the full versions of files” and “Git only stores the changes between versions of files”. [Delta chains](https://gitperf.com/chapter-06.html) make this concrete. Storing the entire copy of every file is storage-expensive, so Git will often save only the diff between the last version and the current one. This will create a “chain” of delta objects such that figuring out what a file is at X version can cause you to recurse through up to 50 separate objects to progressively resolve the files at that version. FIG 04A delta chain: each version is stored as a change to the one before ``` ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ v1 │ ◀──│ v2 │ ◀──│ v3 │ ◀──│ v4 │ │ whole object │ │ delta vs v1 │ │ delta vs v2 │ │ delta vs v3 │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ read v1 + v2 delta + v3 delta + v4 delta to read v4, objgit reads v1, then applies three deltas in order. git allows chains up to 50 links long (maxDeltaDepth). ``` So to read version 4 of a file, you have to read version 1 and then apply deltas up the chain until you get to the end state. Git normally limits this to 50 links or less in the chain, but this only really applies to big repos like the Linux kernel. While I was testing pushing the Linux kernel I found this typo in the code: ``` func (s *Storer) deltaBase(base plumbing.Hash, depth int) (plumbing.EncodedObject, error) { - if depth >= maxDeltaDepth { + if depth > maxDeltaDepth { return nil, fmt.Errorf("%w: delta chain through %s is deeper than %d links", errBadCue, base.String(), maxDeltaDepth) } ``` I wasn’t able to push the Linux kernel because I did greater than or equals instead of just greater than. Fixing that made it work immediately. Oops. ### The pack index needed memory optimizations[​](#the-pack-index-needed-memory-optimizations "Direct link to The pack index needed memory optimizations") Separately, objgit figures out which objects it already has for a repo by loading an index of all the objects into memory and then using that throughout push/pull operations. Every live object in RAM looks like the following: FIG 05Index memory for each live object during a push ``` before █████████████████████▓▓▓▓▓▓▓▒▒▒▒▒ ~390 B ███ pack index entry in the map ~210 B ▓▓▓ cueRecord slices held by the push ~70 B ▒▒▒ scan index of the push ~50 B after ██▓▓▓▓ ~60 B ███ SHA-1 hash, in a sorted array ~20 B ▓▓▓ tableRec, in a sorted table ~40 B linux.git has about 10 million objects: before ~390 B × 10M ≈ 3.9 GB after ~60 B × 10M ≈ 600 MB ``` That is about 390 bytes per object. This doesn’t seem like a lot until you multiply it by the Linux kernel’s \~10 million objects: about 4 GiB of index per push. For now we can get away with an immutable sorted table that holds a 40-byte record for each object: FIG 06One tableRec: 40 bytes for each object, plus its 20-byte hash ``` tableRec 40 bytes, one for each object, sorted by hash ┌───────────┬───────────┬───────────┬───────────┬─────────┬──────┬───────┬─────┐ │ offset │ stored │ raw │ base │ pack │ type │ codec │ pad │ │ int64 8 B │ int64 8 B │ int64 8 B │ int32 4 B │ u32 4 B │ 1 B │ 1 B │ 6 B │ └───────────┴───────────┴───────────┴───────────┴─────────┴──────┴───────┴─────┘ 0 8 16 24 28 32 33 34 40 offset where the payload starts in its pack stored payload length as stored, compressed raw object size after it is inflated base record number of the delta base, or none pack pack number, an index into the table of pack ids type commit, tree, blob, tag, or a delta codec how the payload is compressed ``` Combined with the SHA-1 hash, this makes objects about 60 bytes. That is oodles of RAM savings and is the only reason I’m able to operate against my `linux.git` copy in objgit with the hard 4 GiB RAM limit I set for the container in Kubernetes. ## What's next[​](#whats-next "Direct link to What's next") For now this is pretty good, but I can always do better. Most of the RAM savings from here are more about shunting memory out of the objgit binary at this point. The ideas include: * Memory-mapping things everywhere so that data structures are stored on the disk instead of memory. Disks are still kinda expensive, but less expensive and easier to upgrade than RAM at this point. * Move the index data into a database engine like SQLite or Postgres. This would move the data around and still maybe use a bit of RAM, but maybe it would be less RAM-expensive? * Other performance based optimizations that could come from forking the git library I use. I’m still working on this. Like I’ve said I’m dogfooding this pretty hardcore and I’m hoping that I’ll find more problems like this as I continue to work on my evil plans to save the world from bad software. Stay tuned! Want a git server that fits in 4 GiB? objgit stores every repository in Tigris, so your git data is globally distributed and the server stays small. [Check out objgit on GitHub→](https://github.com/tigrisdata/objgit) ## Share [X](https://twitter.com/intent/tweet?text=Check%20out%20this%20post%20on%20the%20%40tigrisdata%20blog%3A%20Cutting%20objgit%E2%80%99s%20RAM%20usage%0A%0A\&url=https%3A%2F%2Fwww.tigrisdata.com%2Fblog%2Fobjgit-ram)[Hacker News](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fwww.tigrisdata.com%2Fblog%2Fobjgit-ram\&t=Cutting%20objgit%E2%80%99s%20RAM%20usage)[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.tigrisdata.com%2Fblog%2Fobjgit-ram)[Email](mailto:?subject=Cutting%20objgit%E2%80%99s%20RAM%20usage\&body=Hey!%0A%0ACheck%20out%20this%20article%20on%20the%20Tigris%20blog%3A%20https%3A%2F%2Fwww.tigrisdata.com%2Fblog%2Fobjgit-ram)Copy link