Soar's Coding Shenanigans

[soarcodes]

I just found out that there’s new support for hex transparency as two additional hex digits in all modern browsers (except IE, of course)

#FFF, #FFFF, #FFFFFF, and #FFFFFFFF = opaque white #FFFC & #FFFFFFCC = white + 80% opacity #FFF0  & #FFFFFF00 = 0% opacity

[soarcodes]

Thanks for your information. Worthwhile now Tumblr is stopping. But your backup system is only for Windows-users. I use Apple. What can I do to keep my pics? Can you inform about that? Thanks. Leathergearguy.

I unfortunately don’t have a Mac to test on right now, so you’ll have to take my word for it. Python has an installer for macs and this utility uses Python to function, so the first two steps should be the same. However, setting it up to run from any file directory is both very different, and incredibly simple since the file is already set up to do so.

3. Make tumblr_backup.py run from anywhere

In Finder, go to menu --> folder --> /usr/local/bin and just drop tumblr_backup.py in. It should already be executable, but you can check by selecting the file and going to File > Get Info in Finder. Open up the “Sharing & Permissions” dropdown and see if it says executable. You should be able to modify it from this info window after unlocking the little padlock at the lower right. You may need to verify an admin password before continuing. Use the menus next to each group to add executable if it’s not already there.

Alternatively you can modify permissions through Terminal (Ctrl+Alt+T) and typing in:

chmod +x tumblr_backup.py

4. Create your Tumblr Backup folder

5. Run tumblr_backup.py

You will still need to navigate through Terminal to use the backup script so navigate to the place you want to make your backups. In Unix (which is the language Terminal uses) all you need is the cd “change directory” command to change where you are in terminal. If you’re planning on downloading your backups directly to a different drive, use 

cd /Volumes/

ls

...and go from there, again, changing your directory with further cd commands. The “ls” command (LS in lowercase) gives you a list of what is in the directory you currently are in.

Once you’ve navigated to your Tumblr Backup folder, the rest of this guide is the same as in the Windows guide since the python file uses the same commands no matter what OS you are using.

I’ll see about adding this into the guide, but please let me know if this doesn’t work / help or if you need further clarification! :D

[soarcodes]

Backup Tumblr Blogs - 2017

There are a lot of methods here. I’ve used the wordpress method but with that you get EVERYTHING after days of waiting with not much control. The method that gives the most control (and is so far the fastest) is tumblr-utils by bbolli on github. It can be daunting for those who aren’t familiar with command line, however. They do have a Windows guide (tumblr_backup_for_beginners) which walks you through step by step on setting it up and getting it started.

Below is my version of the guide with some helpful tips not mentioned in the original guide. Under read more for length.

1. Install Python 

Key step do NOT skip, even if you already have python you may be using an outdated version and it requires 2.6 or 2.7 to run.) If you are using Windows, download the x86 MSI installer (unless you KNOW you have a x64 system that can handle that version. If you’re unsure, stick with x86)..

You may need to check “Copy Python” when installing in Windows or step 4 will not work.

Note: there are downloads for Mac OS. I don’t have a Mac to test this on, but it should work just as well as on Windows.

2. Download tumblr-utils.zip 

Extract it somewhere safe that you know will not accidentally delete it. Do NOT extract it to Desktop or Downloads as directed in the original guide, especially if you like to clean your download folder every now and then. I installed it on my D:/ drive but you can probably get away with putting it in your Documents somewhere. Just remember where you put it.

3. Set up the PATH variable (optional but recommended)

The reason we set up the PATH is to make it more convenient to run the backup code in command line. If the path isn’t set, you have to type in the full address to the script file every time you run it. With the path, all you need to do is specify the file itself and your computer will know where it’s at.

If you’re running this on a Mac, you don’t need to specify PATH (although you can) All you need to do is drop tumblr_backup.py into /usr/local/bin using Finder and it should do the same thing.

For Windows 10, use the taskbar search “environment” and click on “Edit the system environment variables”. Click on “Path” and click “Edit...” add a New variable and type in the path to wherever you extracted tumblr-utils.

Note: You can easily grab the url path from windows explorer by clicking on the address bar and copy/pasting.

If you’re not running Windows 8 or 10, there’s a guide here for getting to the PATH environment variable. For Windows 7 / Vista / XP, make sure you separate existing information in the PATH from your new address with a semicolon. (see the first window in the screenshot above). 

4. Create Your Backup Folder

I would suggest making a new folder called “Tumblr Backups” on a separate drive from C (D, etc, or even better on a flashdrive or external), but feel free to put it somewhere you have easy access to or will remember well (such as in your My Document’s folder). Again you’ll want to remember the address to it and how to get there from windows explorer. You can always move or copy this folder somewhere else after you’ve backed it up.

5. Navigate to Tumblr Backups in Command Line

For the next step we need to use Command Prompt (or Terminal on Mac/Linux). To open up Command Prompt in Windows 10 you can simply hit the Windows button + X and then C for Command prompt or search for it via taskbar (Here is a guide for opening it on other Windows versions). On a Mac, you open Terminal with Ctrl+Alt+T. You shouldn’t need admin privileges for what you’re about to do. In fact, it’s probably better if you don’t.

When you open command prompt in Windows it will be set to C:\Users\[your username] (the same place Windows Explorer automatically opens to). We want to navigate to where you want to store your backup. If it’s in Documents, then type: 

cd Documents/Tumblr Backups

hit enter and the prompt (the text in front of your cursor) will change to that directory. If you want to go up a folder type: 

cd ../

If you set it up on a different drive in Windows, type the drive letter (capitalized), followed by a colon and hit enter and then change directory using cd

On a Mac, you also use the cd command to change directory in Terminal, but to change drives, all you need to do is type:

cd /Volumes/

ls

The ls command (LS lowercase) lists all the files and folders within the directory you’ve currently navigated to. The /Volumes/ directory is where all your drives are located and you can navigate from there to your Tumblr Backups folder. NOTE use quotation marks around files/folders that have spaces!

cd ‘Tumblr Backups’

6. Run tumblr_backup.py

Once you’re in your Tumblr Backups directory, you’re ready to backup your posts! 

If you want a complete backup of your blog as-is, type the following (replacing yourblog with the blog you want to backup) and hit enter:

tumblr_backup.py yourblog

It may take some time, but even for 18,000+ posts it should take no longer than 30 minutes depending on connection speed. On the next line it will give you a real-time update of how far it is to completing the backup. If it is interrupted, it will process what it’s pulled so far and generate a backup folder anyway. Note that you can backup multiple blogs at the same time by adding more than one blog name at the end, separated by spaces. 

Don’t worry about pulling posts twice or accidentally doing the same command. It won’t add any duplicates.

This also works for side blogs and for backing up blogs run by others that you wish to keep in case it gets purged or leaves Tumblr.

If you only want your own content:

tumblr_backup.py --no-reblog yourblog

--no-reblog (that’s two dashes in front) tells it to only backup posts that you’ve made yourself, including reblogs where you’ve added new content.

If you’re only interested in backing up photo posts:

tumblr_backup.py -T photo yourblog

-T photo tells the script to only backup photo posts. Make sure to capitalize the T!  

If you only want to backup posts with specific tags:

tumblr_backup.py -t tag1,another_tag_with_spaces yourblog

Separate each tag with a comma. Tags with spaces use underscores instead. Make sure the -t is lowercase!

If you have an unstable internet connection, break it up into multiple requests. 

tumblr_backup.py -n 100 yourblog

tumblr_backup.py -n 100 -s 100 yourblog

tumblr_backup.py -n 100 -s 200 yourblog

-n 100 tells it to only count 100 posts, starting from the last post you made. -s 100 tells it to skip the first 100 posts or checks (which are the last posts you made).

You can find many more flag filters in the documentation that can help you fine tune your backup depending on what you want to keep and how you want the backup to be displayed. You can also get information on these options by running tumblr_backup.py --help (with two dashes).

If you are getting image missing errors, don’t worry. It’s usually just avatar images of deleted blogs that are no longer accessible. 

If you get 503 messages and aren’t pulling posts, it could be you’re requesting too frequently and the server locked you out. I’ve had that happen with my stats project while testing. In that case you just have to walk away for a bit and wait it out Double-check that you spelled the blog name you want to pull from right and maybe try again in a couple hours or so.

7. Profit

If you look into Tumblr Backups afterwards, you should find a folder with your blog name. In it is a website hierarchy with each post in the posts folder modified by the originally posted date with all of your tags included. Images are in media, monthly archives in archive, and there is an index.html that allows you to navigate and view everything via your favorite browser.

If you want to use a custom CSS file, call it custom.css, put it in the backup folder and do a complete backup. This forces the pages to all use the new styling.

The CSS of your blog’s original theme css and avatar is always saved in theme/style.css.. Note that it does not save any javascript you may have had on your theme and you should back that up separately. This also does NOT backup pages.

[soarcodes]

Updated my tumblr stats code as I noticed the “Loading” bar was 20 posts off. You also might notice I added a checkbox for RP blogs, which calculates the amount of reblogged text posts that are actually continued RP posts (basically filtered out posts with new comments on them). Said posts are excluded from the reblog count and is reflected in the blog style grid.

[soarcodes]

Thank you for writing a better Tumblr Stat tool! I'm surprised mine even runs at all after all the api changes.

Ah! You’re welcome! :D Well it looks like they didn’t change too much. The only real problem was missing post types, the abortCheck running too long, and the reblog check not catching reblogs from blogs that changed URLs or deleted the original post.

On that note, however, I figured out how to eliminate the data spikes I was getting from the reblogs that were slipping through. Apparently post.reblog.tree_html will not contain content from an original post, but WILL usually contain some sort of HTML even when reblogged_from_id and reblogged_root_id are corrupted. There are still exceptions (posts that didn’t have comments to begin with), but it seems to catch all the outliers I’ve come across so far.

This means that in order to catch as many reblogs as possible, you have to check that post.reblogged_from_id and post.reblogged_root_id exist and post.reblog.tree_html is not empty.

[soarcodes]

So, me being the statistics nerd I am, I stumbled upon this old site written by noodlesandbeef in 2011. However, there were a LOT of bugs with the code (including stats that just didn’t add up). The “timeline” wasn’t really a timeline. It just pushed out 20 posts per point into a chart (some of which were jumbled up in the wrong order to begin with) and you couldn’t REALLY tell where those data spikes were coming from.

Thus, I took it upon myself to code a newer, more accurate version. 

the timeline actually looks and works like a timeline.

the style/medium chart is ACCURATE and doesn’t wander after it’s done.

chat posts are included in the calculations

post types that don’t exist on your blog aren’t displayed

reblogs are separated into post types

stat calculation is about 3x faster

Note that this isn’t me bashing on noodlesandbeef. I think they’re amazing and they’ve obviously done a lot of really neat work. Although I completely rewrote most of the code, the framework and the original idea that sparked it is his. I just took it a step further.