My stream group is celebrating 2 years of streaming today! it's been a wild ride, but our editor @alchana put together this awesome clip compilation of the past year's streams.
My boyfriend and I also put together the credits animation below, to thank all our followers for watching.
Happy 2 whole years everybody! (reblogs are appreciated!)
VodBot, and the mess that is Google (and their API, which is also a mess).
Part 3 is finally here. Took me a while to get to, and a while to actually put into words in a document, and also pen to tablet. You can check out the code on GitHub.
So last time we left off, VodBot was independent. The only requirement in terms of extra programs was ffmpeg, which is a good program to have anyway. Otherwise, Python's package manager pip handles everything necessary. This is wonderful for maintainability, if there's an issue with a package it's easy to find the issue page, and if there's a problem with ffmpeg then there's probably some major infrastructure failing elsewhere.
So VodBot could download videos, and store some useful metadata. A good start, but what to do with it? Having slicing, splicing, and uploading all done automatically would be ideal, just give it some parameters for each and its off to the races. Luckily, the staging and exporting system already exists, so it's just a matter of uploading from the temporary directory VodBot uses! In addition, Google provides an example of how to programmatically upload to their services here. It's almost as simple as copy paste! Well, not quite...
Google requires you to sign up for a developer account and register your app with specific permissions and services. This is pretty expected these days, not every app needs access to every service provided under a company's API, especially if said access gets compromised somehow. What the example fails to explain is that you need some specific info (fun fact, it's more OAuth credentials!), and how you're supposed to get it for specific use cases (in this case, a console application with no GUI or window environment). I got to spend two weeks trying to figure out why VodBot wasn't getting the proper permissions, because the files required just weren't on my system or being pointed to anywhere.
After that debacle was sorted, the uploading example worked without a hitch, and was integrated as its own command very easily. Now for the next problem: extending the code and making it my own. The example is incredibly detailed, and provides a lot of failsafes which is fantastic, but unfortunately the rest of the documentation is not documented anywhere on Google's main sites. What's worse, the library for Python does not contain data structures that are fully defined and fleshed out prior to runtime. In layman's terms, I don't know what a specific function will give back and that makes it difficult to add additional features on top of what the example provides.
My favorite interpretation of Google's attitude of how it handles it's API can be handled by this ex-employee's own artistic expression. (Credit to Goomics, #50)
Plenty of old examples existed online for how to handle this kind of stuff, and very very few modern examples existed for some help. I was lucky I didn't want to add more than an indication of upload progress to print outs. Heck, I don't know if the example for uploading to YouTube will still be up in a few years. I can hope.
Back to VodBot: Google allows users to upload videos completely for free, at the cost of a quota. Each method used to query and upload data to any of Google's services has this cost of quota, preventing any one person from just spamming services with garbage without any roadblocks. The main two quota costs that VodBot has are querying existing videos (which is required to be known to upload new videos because... reasons? It's useful, but there's no reason for VodBot to currently have that info, making the query feel wasteful but unavoidable because of API design) and uploading new videos. Getting existing videos costs practically nothing with the testing quota each new developer account is given, but uploads cost a lot. In addition, videos can only be uploaded and set to private when uploaded programmatically, and cannot be set to public.
In order to get more quota and allow for public uploads, Google has to "audit" your program. Basically, you get to fill out a long form saying what your intent is and what your program does and how it is meant to be interfaced with. You have to upload a video of you using the program with voiceover. Gotta wonder what that Google employee thought of my mic quality and sound.
After the form, you'll be contacted by Google over email to answer any remaining questions. This ended up being the same questions that I answered in the form. I sent them my recording and voiceover again and answered use cases, etc. They didn't even watch them until I had pointed out that I had sent them the video multiple times. They repeated questions, contacted me for weeks and weeks threatening to close my audit and deny me access because they didn't read my responses. I get that I can write a lot, but jeez!
Once that finally got sorted, the VodBot project on my Google Dev account was given unrestricted access to YouTube and it's API. VodBot did not have to make any changes locally to upload with this new permission, it was all done on Google's side. Great! Just one problem...
Because VodBot is written in Python, and it's open source, there is no easy way to obfuscate the sensitive OAuth information VodBot needs to log in to YouTube and upload. Releasing these files is dangerous, and if the wrong hands get it, then people who want to use VodBot are stuck waiting until quota frees up to be able to upload. Plus, it has to be in a file, Google's API will not read it otherwise because... reasons. Even the amount of quota I got was not useable on a scale larger than one person uploading eight videos a day. I don't upload eight a day, but someone might! Google also displays a big red warning against using VodBot because it's not audited for use outside my personal "company". I've tried applying for it, but there's a bunch of info that's just a repeat of the first audit, and I don't want to bother right now.
If VodBot was compiled in, say, Rust or C++ this would not be as big of an issue. Executable files are obfuscation all on their own, because who wants to look through a bunch of unrecognizable symbols? Unfortunately, until I switch over, this problem is here to stay. Maybe some day there will be a proper solution, but right now VodBot's upload function is locked behind the requirement of being a person I can trust.
As always, you can support me in a bunch of ways if you enjoy what I do. My support links are always at the top of my website or you can just share my posts and projects with people you think might enjoy them. Thanks for all the reading!
VodBot, the bestest little helper I ever did make.
Part 1 of... a number of posts! Everyone say hi and be nice to VodBot on GitHub.
January of 2020 rolls around, everyone’s excited for the new year of perfect eyesight. And then, the funniest thing happened…
Because of the world’s state of affairs leading into March 2020 I decided I should get some new hobbies. I bought my first 3D printer, an Ender 3 Pro, on eBay for just under $200 and it’s been weirdly enough one of my better financial decisions. I also began streaming on Twitch.tv in earnest around this time, and I had a goal to reach Affiliate status in just a few months (Twitch Affiliates are able to earn money from their streams through ads, bits, subscriptions, etc). In less than a month, through asking my friends to advertise my stream and constantly trying to put myself out there (probably breaking some community rules in the process, oops) I became an Affiliate and streaming sort of became a part-time-job-hobby! It was a lot of fun, and I’ve been streaming pretty regularly ever since.
Around October of 2020, I realized I kind of wanted to archive my streams somewhere for safe keeping. The best place to store lots of video data for literally no cost? YouTube of course! You can check out our existing archive channel which also links to all our Twitch pages here! I got to work, once a week I went to Twitch’s video manager page and had it queue up a download of my latest stream for me. I would slice it into different parts based on the games I played, which I had to get timestamps for manually. As you can imagine, this is pretty time consuming and boring. Not only that, but my friends also began streaming and some became affiliates around the same time, and getting them to do this task too wasn’t gonna be easy. So how do I make this task easier? Well, for starters, snakes. Let’s tangent!
Python is a wonderful programming language, it’s often one of the first languages that absolute beginners use because it’s grammar and functions are super easy to understand and use. It doesn’t have any sort of compile time, and is dynamically typed, meaning you don’t need to worry about waiting for anything. It also has thousands of pre-made libraries to make whatever task you’re trying to do so much easier. Lastly, Python also has the ability to send commands to other programs on the same machine, with relative ease.
VodBot’s first version was rather simple, it was made with Python. Use existing Python libraries to send some data to some URLs hosted by Twitch.tv to get info on previous streams (called Video-On-Demand, or VODs for short) and clips (slices of video made by viewers of fun moments on streams). It would do this through the Representational State Transfer Application Programming Interface, or REST API for short. Then, do some basic combing of this data to make sure that these videos weren’t already on the host system, and then pass the URLs for these streams to other programs like youtube-dl or streamlink to download the videos to the host system. Oh, also make a small file containing relevant metadata for the stream such as the title, the game, when it was streamed, etc. VodBot “shelled-out” these commands quickly to the other programs and let them do all the heavy lifting, which was just fine. I still had to slice up the video’s by hand, but it wasn’t too bad now that I could pull the videos down faster and leave this little script running in the background while I worked on schoolwork. Heck, I even learned that the program that youtube-dl and streamlink shelled-out to itself was the very capable ffmpeg, which I began to use to slice my videos much quicker without the overhead of a full video editor program.
Of course, this got old pretty quick too. Why can’t I automate this any further? What prevented me from simply queuing up these long videos and sending the relevant slices up to YouTube directly for me? Well, it turns out, not a lot! I'll talk about this more in the next part...
