My horrible Odyssey to install Git LFS on my Synology NAS
So this is a bit different content from what I usually post. But I NEEDED an outlet for this incredible buildup of frustration that I had to deal with for the past WEEK. The objective seemed easy when I decided to start this terrible journey:
Install Git LFS on a Synology NAS disk station.
The Status Quo
My friend and I wanted to work on a Unity project together, so "normal" Git, I was told, would not suffice. It had to be LFS. We looked into other options before, but I thought "Hey, why not use the readily available NAS I have? Why use external, even paid services if we can just use a homebrew solution?" And that's where the descent into madness started.
I have a DS418j, not the most advanced or expensive model. It is important to note that I went into this as a completely clueless person.
I never used Git via console commands. My knowledge of Git was generally very limited. Even at the end of all this, I still BARELY know how to set up a repository and interact with it.
I had no idea what LFS even really does.
I only had very rudimentary knowledge of how to operate my NAS via its user interface.
I never interacted with a Linux console before. It turned out that I would be doing that a lot.
A Walk in the Park?
At first, everything seemed pretty straightforward, to be honest. I googled the simple task of setting up Git on a Synology NAS. The first result was already pretty helpful.
It seemed like all I had to do was set up a new user on my NAS ("gituser"), also install the available "Git Server" from the NAS's user interface. And "WebDAV", another package that could be installed via the interface.
WebDAV, as I found out, was a key component in my journey. It was a bit of a struggle to set up, but it appeared to be important in the process of connecting to my NAS via HTTPS. And probably other things that I still have no idea about. I didn't even know why I'm installing WebDAV in the first place, because I intended to use Git via SSH. Which another setting in my NAS would provide - the Terminal setting in the system settings. That's where I enabled SSH via port 22.
Well, my friend then told me that we cannot use LFS via SSH. Okay, I thought, that's what WebDAV is for, after all.
The Git Server had very few options, which seemed fishy to me. It literally only has one window where you set permissions for users. I gave gituser the permission to access it. That was that.
Of course I also needed a shared folder for our repositories ("git"). Creating that was not hard either. Here I noticed that gituser needs to be part of the "administrators" group for Git Server to work properly. But I could not remove it from the "users" group, so things got a bit fucky with permissions. I ended up giving both the administrators and users group a lot more permissions than I was comfortable with. But I trust my friend to not wreak havoc on my NAS for the time being.
So, everything was set up. Or so I thought.
Hitting the first Bump in the Road
I was able to connect to my NAS via SSH only, but didn't think anything of it yet. Doing that, I used Sourcetree to create the first test repo. Went to the NAS interface, checked, it was there and looked good. I could push and pull files. Created a second repo, ran "git lfs install"... and it didn't work.
The lfs command was not known.
I quickly found out that... of course it could not be known. Other than github for example, my NAS did not have LFS pre-setup. So, I concluded, I had to go install LFS on my NAS.
...Easier said than done.
While it does support a console with most regular Linux commands... a package manager is not included. At least none that is easily accessible, or supports any common packages.
At this point I figured "Why deal with this myself?" and contacted Synology support. I asked them "how can I set up Git LFS on my NAS?"
And Synology Support said:
¯\_(ツ)_/¯
They told me they do not offer support for the console. They said I should go ask the community. Okay.
I did not ask the community, but I did a lot of googling and found out: I could not "just install" LFS. I had to get creative.
We heard you like package managers?
First, I figured out that I need to be able to use the "curl" command. All the binary files on LFS's package cloud were binaries that were apparently downloadable with curl. I did not know what curl was... but I knew I needed to get it working.
I found out that for curl to work, I needed to install PHP on my NAS. Luckily, that was possible via Synology's included package manager. But for PHP to DO anything, I also had to install the "Web Station" and configure my PHP version there. I figured... might as well!
After enabling a couple PHP commands, I felt ready to download LFS. But the question was... which version? What even was my OS?
As it turns out, Synology uses a custom Linux version for their diskstations. Of course, LFS does not "officially" provide a package for that version. I tried the nodeJS version, because I noticed I also have nodeJS installed on my NAS. I ran into the version issue as well, unfortunately, when I tried to install the package through nodeJS. Not even changing my nodeJS version helped. Many hours later, I tried the .deb and .rpm files randomly instead of the nodeJS ones. Those also didn't want to work, despite me eventually figuring out how to lie to them about which OS I'm using.
I was almost ready to give up at that point. I was at least 3 full days into my odyssey already.
But then I spotted something else... A thing called "GoLang". Apparently, it would be possible to download LFS via GoLang. However, to do that, I of course needed to get Go first.
An initial search got me on track to "ipkg", which promised to enable me to install Go. But after reading up on it a bit, it looked woefully outdated. I had it already downloaded and was about to install, but ran into errors and trouble again.
That was when I found "Entware". It's similar to ipkg, but uses "opkg", a similar package manager. I was able to install Entware on my NAS without much trouble, and it contained the Go package that I needed so direly.
While I was at it, I also installed the available "git" and "git-http" packages from opkg, just to make sure. Who knew if they'd come in handy. I don't know if they did, but they also didn't seem to cause any harm.
Now, with Go installed (which went surprisingly smoothly), I was able to access just about anything on the internet and install it on my NAS! Hallelujah!
But if you thought it was over... just look at the scrollbar.
The end of my odyssey was finally in sight. I thought that nothing could go wrong anymore, now! With the help of Go, I was able to install the LFS binary. I was able to run it in my console, too. I was finally able to run "git lfs install".
...and it didn't help.
I got a bunch of errors, again. Instead of getting closer to the solution, it seemed like I just managed to find more obscure errors. Here the important thing about the SSH came in, as well. LFS does not like SSH, from what I found out. But SSH was the only way for me to connect to my NAS from my PC!
In a fury of looking up stuff again, I found the "DDNS" option on my NAS. That would allow me to get a hostname and use it like a normal website address! I kinda rushed through the setup because I had no idea what I was doing.
Well, whatever I did, it seemed to be sufficient. My friend could now connect to my NAS over her Sourcetree client. But when she tried to upload LFS objects, it just didn't work. She got a "404" error.
It wasn't hard to figure out that 404 meant no access in this case - my NAS was simply refusing to show any directory to someone who doesn't have access to it. Cue a long journey into my NAS's interface to make sure the gituser had the right permissions. I also changed the password because I read something about WebDAV being particular about some passwords. I also made a new user to see if maybe I just messed up setting up gituser.
To test if I was making any progress, my friend and I tried to access my NAS via our web browsers. But no matter what we tried, no matter what I tried, we couldn't access. 403 Forbidden or 404 Not Found. Those were the results. I couldn't even access if I used my admin account.
I tried to hack my way into pushing anyway, and only ended up corrupting our repo's history with "missing" and "broken" files because they were never properly uploaded, but LFS thought they were.
It should be noted that I had just accepted that HTTPS won't let me connect on my PC. So I had set up a hotspot for my mobile internet via my phone and used my laptop to do these things. I was in denial about eventually having to fix this, because I'm on a tight data plan on mobile and uploading and downloading Unity projects of several GB size wasn't going to happen that way.
Synology Support to the Rescue! ...Or?
It seemed like we had finally narrowed down the issue with our LFS upload attempts when I also checked the WebDAV console and it reported that it denied our login attempts through browser and Sourcetree as an "authorization failure". So something was wrong with WebDAV.
I contacted Synology support a second time. I asked them, "Why can't my friend and I connect to my NAS via the internet when I have WebDAV enabled and everything port forwarded?"
And Synology Support said:
¯\_(ツ)_/¯
They told me WebDAV and web browsers don't use the same HTTP and HTTPS methods. They are simply not compatible. They told me I should download their WebDAV client or connect locally.
So it was known from the start that what I was attempting could never work... but it was also not mentioned anywhere on the web or Synology's help pages that this was the case.
We have a saying in Austria: "jemanden deppert sterben lassen". It translates to "to let someone die stupid". Essentially, it means that you have information and you watch someone else without this information struggle without ever telling them about it voluntarily. I felt this saying was very appropriate for my situation.
Time to give up, I guess... Except I didn't.
I was almost a week into my odyssey by now. Maybe it's a sunk-cost-fallacy, but I couldn't abandon all my work now. I refused.
A Light at the End of the Tunnel
I went back to open another browser tab (my average was 20 open tabs during this... normally it's 2 to 3). And I searched for a solution that works with WebDAV. And truly... there was ONE repo online that offered it.
A special thanks goes out to this fellow: https://github.com/mpotthoff/git-lfs-webdav
They straight up saved our project from collapsing into a pile of tears and rage. I installed this package on my NAS, which... sort of worked. It turned out I needed to install it locally (as well?). So I did. But I needed to install Git, LFS, and Go on my local PC as well for that.
So with the help of Go, I built an exe file for my laptop, which then gave me a 401 when trying to push to LFS. Luckily I expected that. And I was overjoyed, because FINALLY a different error.
I tried to run the steps in the git-lfs-webdav repo to fix it... but got a strange error in the console.
It cried, when trying to enter my username, that the "handle" for the password was wrong. But I hadn't even entered the password yet! Searching some more on the internet gave me no conclusive answer. Randomly, I tried a different console - my Sourcetree console apparently runs "MINGW32" while my Git console runs "MINGW64". Switching to the Git console fixed this problem for me, and switching to the Windows shell fixed it for my friend.
And then, it finally worked for my friend.
She could upload a test image via LFS, and I could receive it via LFS on my laptop.
The rest was me calling my internet provider about my PC being unable to connect. The internet provider said ¯\_(ツ)_/¯.
Luckily I did not attempt to mess with my DNS or Subnet Mask or anything of the sort, or buy a VPN/Proxy. All I had to do was create a self-signed SSL certificate on my NAS, download it, and feed it into my PC's trusted authorities files. My friend had to download and feed it too.
In Summary...
This was a horrible, terrible, awful journey. I would not recommend attempting this at home to anyone.
Even now, what we've got going on only sort-of works with our Unity project, because Unity is... Unity. We're still figuring out the details, for example why scene content is going missing and so on.
But I believe that the worst and most difficult part is over. Will I be able to recreate this in a year or two, maybe for a different repo, on a different PC?
Probably not.
