May 4, 2023
seen from Finland
seen from United States
seen from United States
seen from Russia
seen from Australia
seen from Netherlands
seen from Bulgaria
seen from United Kingdom
seen from Hong Kong SAR China
seen from United Kingdom
seen from United States
seen from France
seen from United States
seen from Australia
seen from Germany
seen from United Kingdom
seen from United States
seen from China
seen from Germany
seen from United States
As projects grow, one question quietly becomes a big one:
Merging git repos into a monorepo with git subtree
Sometimes you yearn for the simplicity of a monorepo. Maybe you just want everything in one place, maybe you just want to more easily share code.
Regardless of your motivation, git makes it easy to split and combine repos via git subtree. While it isn't part of git core, most distributions package git with subtree anyway.
After your monorepo is set up, you can start importing repos into it with git subtree add:
git subtree -P <prefix> add <repository> <remote-ref>
-P says which subdirectory in the monorepo the imported repo should live. This allows you to organize the code that you're importing.
is the thing you want to import, e.g. https://github.com/user/repo
is the branch or tag to import. Typically, this is main or master.
Compared to submodules, subtrees exist independently of their remote. If the remote of the subtree is deleted, the subtree in your monorepo is a copy so it persists. However, if the remote of the submodule is deleted then you can no longer clone it.
Compared to good ol' copy-paste, subtrees preserve git history. Although if you don't want the history you can also --squash.
One word of caution: if you want history then you need to include the merge commit. On GitHub, this means you have to Create a merge commit; do not squash it! If you squash the merge during submission in GitHub then you lose the history. Maybe I'll go into details another day...
The Easy Way™ It turns out that this is such a common and useful practice that the overlords of git made it really easy, but you have to have a newer version of git (>= 1.7.11 May 2012). See the appendix for how to install the latest git. Also, there's a real-world example in the walkthrough below.
This is such a lovely, thorough answer. Apparently I haven’t been garbage collecting --aggressively enough.
Apps I Love: git-subtree
My thinking was fuzzy in regards to what's wrong with git submodules. So I finally opened up an editor window and started typing how I think an Ideal Submodule System should work. I wound up with this:
The subproject is copied into the super-project's repo. At least a snapshot of it, if not the entire history. It's a fact of life external resources have a habit of disappearing -- this helps makes your project resilient.
git clone foo is enough. None of this git clone --recursive foo or git submodule init && git submodule update business.
Can easily pull subproject updates.
Can easily push subproject updates.
Can easily handle super-project branching.
Super-project's commits don't wind up in subproject's history.
Subproject's commits don't wind up in super-project's history.
Armed with clearer thinking, I reexamined my options and have settled on Avery Pennarun's git-subtree.
Unfortunately its CLI UX is lacking (you have to specify the subproject's entire remote repo URL each time you pull or push) and Avery hasn't accepted pull requests or budged the project for a year. Fortunately Helmo forked Avery's repo, added .gittrees support and push-all and pull-all subcommands, and generally seems to be keeping on top of pull requests and moving the project forward.
I just started using git-subtree, but so far it's making my life better than before.
P.S. Here's a short primer on using git-subtree.
Update: Hmm, I discovered and turned to Helmo's fork after I got tired of reentering my remote repo URL each time I wanted to push or pull a subtree. Turns out Helmo's fork is currently unstable -- the tests don't pass and there's a definite bug in it when adding a repo as a subtree that doesn't show up until you attempt push it back. So I recommend using Avery's original repo until it's fixed.
Update 2: I fixed the bug and tests are passing in my git-subtree fork. git subtree push-all away!
Update 3: git-subtree is now included in git itself, no need to use my fork. The only downside is the push-all and pull-all convenience commands aren't supported yet.