#wordpress export doesn't export all posts

[n7punk]

Tumblr Backup Options: None of them do everything

Cheeky but true. I'll go through what's good and bad about each option though so you can decide which balances out for you.

Covered: native export, WordPress (kinda), TumblThree, tumblr-utils (kinda)

Native Export

If you go to "https://www.tumblr.com/settings/blog/yourblogname", at the bottom of the page is an export option

Once you hit the button to start the request, it will start processing. Feel free to log off, this is going to to take a few hours. You don't need to keep it open. ~22k posts took roughly a day for me. If you have a small number of posts and get stuck, you're probably broken.

When it's done processing, you can hit that download backup button and then wait some more as you wait for the zip file to download. Mine failed the first time after like twenty minutes, and then I had to start over. I think it took 1-2 hour(s) and I'm almost certain that was on Tumblr and not my internet. And that was the zip file! So make sure your computer can be on for a while before getting this started.

So what do you get?

A media folder, conversations folder, and posts folder

Media folder: Every single photo, gif, and video that has ever been on your blog or in your DMs. There is no context data attached (except for dm images which do say which conversation they're from at least), but they seem to be in chronological order because they seem to be titled by the post's ID (the string of numbers in the address bar after "/post/"). They look like "100868498227", "100868498228_0", "100868498228_1"

When you see something end with "_0" and up that means the photos are in the same post, so _0 represents the first image in the post, _1 represents the second, etc (at least, I think).

Conversations folder: HTML export files of every DM history you have on your blog. These are actually pretty well formatted, see example here.

Posts folder: html subfolder and posts_index.html file

posts_index.html: File listing every single post on your blog by post ID on its own line with no other context. Example of a line: "Post: 780053389730037760". The ID number will link to the post in the html folder

html subfolder: contains a submissions subfolder and stripped html file versions of every post on your blog. See below first what the post looks like on Tumblr, and second what the post looks like in the html folder

The way you seem to be intended to use this is to open the file index, select a post ID, and be jumped to where that post is saved as an html file, but I don't know why you would bother when the index doesn't provide any information about the posts inside it. The posts all have extremely minimal formatting. See a reblog chain below.

Notice I said ALL posts on your blog. Photo posts without a caption will just have a broken image icon and then the date and tags. Theoretically, it might be that if you unzip the entire export folder that allows it to automatically link to the image saved in your media folder. I have no fucking idea, unzipping the folder was estimated to take two hours so I didn't do it. Let me know if you do though so I can update this post!

The submissions folder is such a rabbithole I made a post just on it but long story short it's asks you haven't replied to

What do I see as the main reasons to opt for this option? 1) you don't want to download any programs or files from the internet just to backup your blog, 2) your blog is relatively small, so digging through the ID files isn't a big deal, 3) you mostly just want to download either the images (which will be browsable via thumbnail previews in the media folder if you unzip it) or conversation history, which are fairly well formatted, 4) you don't need to update your export often/ever, because you'd have to request it from the start and download the entire thing all over again, 5) you want to be able to read your text posts clearly and don't care about preserving the full formatting, and/or 6) you don't plan to reupload this information elsewhere (say on... a WordPress blog)

WordPress Automatic Ex/Import

Move your post's from Matt's right hand to his left! WordPress (another product of Automattic) has a native Tumblr importer found under your WP Admin dashboard for your site under Tools > Import > Tumblr.

How does this work? No idea! I hit import 2 days ago and it has done nothing. Maybe I'm stuck, maybe it's permanently broken. It says to contact support if it's been over 24 hours but they don't make that easy. I disconnected from Tumblr (you can only port over a blog you have the login of) and reconnected and it still said it was importing. I don't think it's ever going to do anything.

Presumably it's supposed to 1:1 import every post on your blog onto the WordPress site, which will result in a whole lot of stolen art because there's no way to select just your original posts. Also, you'd need enough storage on your webhost to house all the posts (this honestly might be my problem, but I was planning to delete all the non-original posts once it imported.... anything and backfill what it didn't get to). The one thing I'll say about this option is that it's the only one I've seen so far that exports drafts and queues as well.

I mean, if it exported anything. If this ever does anything I'll update this post, but either my blog is too large or this tool isn't totally functional anymore.

TumblThree

(previously TumblTwo, etc)

TumblThree is an all-in-one program requiring no extra downloads beyond the main Zip, and was last updated fairly recently at the time of this post. In order to run it, unzip it into one folder and run the main .exe. It has a full UI interface with lots of very descriptive helper text to help you select the right options for you without looking at the wiki. I think it's user-friendly for non-tech people.

There are a lot of options in TumblThree to change what output it gives you, but I'm going to start with the largely universal parts first:

Everything from one blog will be exported to one folder, no subfolders or sorting. As a result, the output is very messy and difficult to wade through, but post metadata and the photos are named in the same way so you can scroll, see an image preview, and then click on the metadata txt for that post and read the caption.

Depending on your settings, you can export all photos, videos, text posts, etc as their own files or exclude them from the export entirely. For the different types of media posts, you can independently select if you what to download just the media, just the metadata for it (everything that surrounds the post when you see it on Tumblr, such as the caption, OP, tags, etc), or both.

Master txt file: For every type of media metadata you export, a correspondingly named txt file will be created (images.txt, answers.txt, etc) that contains the text/metadata of every post of that type in one txt file. This is also the default behavior for exporting text posts.

Note: for text posts (which includes asks/answers), it only creates a master txt file if you do not select "Save texts as individual files", in which case it will only save each text as an individual txt file and not make a master file.

The formatting on these files is so brutal I won't even give examples, but they're unreadable. Being a .txt file, there is no native formatting, so it exports in html formatting.

Example: instead of a post that says "I want to go swimming", it exports: "I want to go < b >swimming< / b >" (minus the spaces around the b) as the post body, which is a big part of what makes it unreadable, because there are a lot of hyperlinks in all the header information listed below.

Each post in the master txt exports with: Post ID, date, post URL, slug, reblog key (no idea what that is), reblog URL, reblog name, title, [the text/caption itself], and tags.

Theoretically this means you could ctrl+f "cybertrucks" in the master txt file and then browse all your posts making fun of Tesla owners by tabbing through the returns. This is not possible with any of the previous options, and only is possible because it's all in one file, as ridiculous as it is, which is why getting that master file is so important.

For the trick to get both the individual text posts and master text.txt & answers.txt file, as well as my recommended settings and details on how updating backups works, see the read more at the end of this post.

The images.txt includes all the information listed above, but with the following additions: photo url (NOTE: this is the url on Tumblr, not a link to where it is in your folder), photo set URLs, photo caption, and "downloaded files" (NOTE: this is the name of the file it has downloaded)

The video.txt is similar to the above

The use case for this would be similar to what I described for text posts above: search keywords from captions, tags, etc and when you find what you think is what you want, copy the name from "downloaded files" and search your folder to find the actual image

I really hated TumblThree's output the first time I looked at it and then I realized the single file is the only way to make browsing tags workable, because otherwise you would have to have a folder for every tag, and posts with multiple tags would have to be duplicated between them. I'm not pressed on finding a txt to HTML converter right now but it could be an option in the future if you wanted to make things more readable.

Okay, let's get into the non-universal stuff you can customize in settings, because it's like, everything:

File names: We've already established you can search with the downloaded file name for images, but what will that be? Whatever you fucking want. Post date, reblogger name, post ID, post title, original file name, you can make it any and all of these in any order you want! You can have actually useful file names! Personally I like %e_%p_%q_%i_%x which exports as DateTime_PostTitle_BlogOriginName_PostID_IteratingNumber (note: you need some kind of unique iterator to be valid so two files don't have the same name, such as multiple photos from one post). Look how much searchable information that gives me, in chronological order! It decreases your need for the master txt file.

Tip I wish I thought of before doing my massive export: make one of the unique headers from the master txt file part of the exported file name so it's easy to search for it after identifying it in the master file.

Files scanned: this is the only method I've found that lets you back everything up, remember what it backed up, and then lets you add any new posts since that date without having to download the whole thing again. That's a game changer, but see the read more below for limitations.

You also have the option to rescan the entire thing if you want.

Post type: T3 (I'm abbreviating it now) also lets you export just your original posts, just reblogs, etc - again, giving you the most control of any options. It also lets you export replies. I, uh, would not do this because if you have any popular post on your blog it might have hundreds, or thousands of replies but hey, you can do it!

You also have the option to only download posts with a certain tag.

Blog options: You can export literally any blog you have the URL of. In fact, if you copy a blog URL while it's open, it will automatically add that blog to its UI and create an empty folder for it. It makes it easy, no private key required. I do have mixed feelings about the concept of exporting someone else's blog... but I'm also planning to do it to some of Crew-ra's blogs so... my digital horde must grow.

You can also queue blogs up and leave it to run through a lot of them. It is a lot faster than Tumblr's native export, I started this import well after I started typing this post and it took a few hours, probably not all that much longer than just downloading Tumblr's export took (and that's while running it alongside other data copy operations because I'm backing up a lot of stuff right now).

I do recommend doing a test export with a sideblog, I was able to use wild-bitchofthenorthwoods as a test import since it only has one post and it has media, so it was super quick.

(I do want to note, I think the number of downloadable items starts out matching the number of posts on your blog without scanning them until you start the export - but if you choose to export everything as its own file, you're going to end up with way more than that because a post with three images would be multiple files)

Things T3 cannot export:

Since in its simplest form it's just accessing the public upload of your blog, it cannot export your drafts, queue, or conversations

It cannot export posts as HTML files, and thus cannot export them with readable formatting natively

What do I see as the main reasons to opt for this option? 1) you don't care about exporting your DMs/conversations, 2) you want the ability to export only certain kinds of posts (original, photos, using a tag, etc), 3) you want to control the titles of the exported files 4) you don't mind wading through massive folders, 5) you want the ability to search tags (using the txt files), 6) you want the ability to update your export without starting over from the beginning, 7) you either don't want to reupload this information somewhere else, or you want to upload it somewhere that supports automatic HTML conversion (for instance, you can switch a Tumblr post from a rich text format to HTML, same with AO3, so you can put it in as HTML and then hit post to see it turn into a rich format. This techically makes T3 the most versatile/useful export option if you're planning to do anything with it other than browse your own files).

tumblr-utils

Full disclosure: haven't tried this one. But others have! tumblr-utils is a no-UI, python-based backup software. This means in order to use it you have to type commands into the terminal. If you don't know what I just said, don't use this one.

If you do, you'll need to separately download python and youtube-dl just to get this one running. You'll also need to give it your personal Tumblr API key and feed it commands deciphered from the wiki page I linked. Here are two different guides people have written on how to use it. Output:

Obviously I'm guessing based on the documentation, but one thing that is nice is this tool allows you to save each post in its own folder. Presumably each post is multiple files like we saw with T3, so this would make it easy to group them, but it also means you'd have to look in every single folder to find anything.

It seems to break posts up into timestamp folders by month, again, helping with management to narrow down where you have to search

It allows you to save only certain kinds of posts at a time like T3

It allows you to backup posts only from a certain time period (so if you keep a little .txt note of the last time you backed up, you can easily add only the new posts into your backup without having to start over from the beginning)

It allows you to only save posts under a certain tag like T3

It allows you to save only original posts

It's the only one I've found that lets you back up your liked posts

What do I see as the main reasons to opt for this option? 1) you don't care about exporting your DMs/conversations, 2) you want the ability to export only certain kinds of posts (original, photos, using a tag, etc), (okay now we get to the points that aren't also covered by T3), 3) you want posts to export already broken into folders, whether by post or by month, 4) you want to back up your likes, 5) you don't care what file names look like, 6) you're comfortable with the command line/coding and don't need a UI.

Summary:

None of these options are ideal for reuploading your files anywhere (except WordPress), but I do think TumblThree is the best of the options because of the written HTML formatting in the txt files being useful for websites that support automatic conversion (or require HTML input).

For starting another blog, WordPress wins. If it works. I'm trying to be generous here.

For searchability, T3 wins again.

For versatility... yeah you know it's T3, but tumblr-utils has a lot of the same features, too!

For sentimentality (aka conversations), it has to be the native export. There literally is not any other option.

For queues and drafts, the only theoretical option is WordPress. If it works.

For likes, the only option is tumblr-utils.

Every option does something the others don't, so theoretically to cover everything, you have to do all four options. Actually I would say do the native export if you don't have a lot of posts and aren't a freak like me, check it out, and if it doesn't work (I know it's finnicky) or you don't like the export, go with TumblThree. This also means you'll at least have your conversations even if you don't end up using the native export any other way.

And I wish it could go without saying, but don't repost people's shit, y'all. I'm backing up everything for my records only and it will never be shared with anyone else, or even browsed as long as using Tumblr instead is an option.

TumblThree adding to old backup quirks, recommended settings, & master file backup solution:

Adding to backup quirks:

From my tests, when you scan a blog you've already backed up to just add new posts to it, it does not update the master file, so if you want to update it, you'll have to do the steps I list at the end of this post. It might be possible it does update if you force rescan, but I highly doubt it.

If you scan a blog you previously backed up under more restrictive settings - say you only backed up original text posts as one file before and now you've selected to back up absolutely everything - it will only download up until the time you last backed up that blog. It will not blow past where you last downloaded to download all the photos and videos it didn't get before just because they're selected now. This is great for doing after using the master file solution I'm showing below, but if you do need to download everything after doing a more restrictive scan, you can once again follow the first few steps below to do so.

Recommended settings:

This will obviously vary by what you're trying to do, but one or two things weren't immediately obvious to me and I did say I think this was the best solution for less technical users, so I want give my personal recommendations. Settings can obviously be found under the settings button at the bottom of the screen (you may need to use the scrollbar on the UI for, which is separate from the scrollbar on the blogs panel), but when you click on a blog, when you click "Details" in the right sidebar, you can also see your most important settings at a glance and adjust them to whatever you want them to be "per blog". I believe TumblThree remembers what you last used for the blog and applies the things in settings only to new/other blogs.

The thing that is going to vary the most is how many different types of posts you want to back up (text, video, reblogs included, etc), so I'll leave that up to you. If you're going to export a media type, though, I generally recommend exporting the metadata too.

I already gave my preferred file names above and again that's going to be something that varies a lot by people. Hover over the "Filename template" box and it will give you all the options in the legend you can combine via underscores.

Leave "Skip .gif files" off unless you're hurting for hard drive space. This removes all the gifs from your download, and the reason this is provided as a separate setting is because gifs have relatively massive files (at least compared to a text file)

I'll be honest I haven't seen a difference between turning on and off "Group photo sets". Because of the way file names work, most conventions will naturally lead to photos from the same post all being in a row.

"Save texts as individual files": if you only want texts to be saved as their master text.txt and answers.txt files, uncheck this. If you want the individual files I highly recommend you also download the master file for searching purposes, in which case my recommendation is this:

1) Select to export texts only, leaving off all media options, and uncheck the "Save texts as individual files" option. 2) Export the blog. This will only result in two files, answers.txt and texts.txt. 3) Move these files elsewhere on the computer to save them. 4) With T3 closed, delete the folder for the blog and the blog's Indexes (see instructions at the end of this post for finding these). 5) Reopen T3, which shouldn't remember it ever saw the blog and create a new folder for it. Turn on the "Save texts as individual files", as well as any other media posts you want to download. 6) Export the entire blog again. 7) Move the texts.txt and answers.txt file back into the blog's folder.

I leave all other options on the Details tab off, except for:

"Force rescan" scans past the point it last backed up and searches the whole blog again. If you have a big blog, this is going to burn time. This is needed for the number of downloaded items in the panel to be accurate but I don't know why you would care or turn this on unless it lets you skip steps 3-4 above, but my blog is too big to burn through testing that, so if you try it, let me know and I'll update this post!

Master file backup solution:

See my 7 steps from above to skip having to do this, but if you accidentally do things out of order and then realize you still need the master files for texts post after backing everything else up, here's how you get it with minimal pain:

T3 will make an "Index" folder in both the main folder for the program where the exe is located and the destination folder where you have your blogs backing up (note: these were two very different places for me, if you just have it back up to the automatic Blogs folder within T3's folder, it might not create a second Index folder).

To make T3 "forget" what it has backed up previously so it goes through to the beginning and makes a master file that includes everything, all you have to do is remove the Index file(s) for the blog while it's closed so it doesn't remember it anymore. I backed my index up in another folder.

Check off for it to only download text posts, and then uncheck the "Save texts in individual files" option. This will cause it to only create the master answers.txt and texts.txt file on the rescan.

The combination of only going for one post type and only downloading one file for it means this rescan is relatively fast. When you look at your Blogs folder, you'll find a new folder has been created for your blog name (in my case, there was "n7punk" and "n7punk_2) and your output is in the new folder. I just moved it over to the original folder.

At this point you can restore the indexes, though I've only gotten it to half recognize them. I can get it to recognize my original n7punk folder so everything can stay there, but the total downloaded items is stuck at what it was when I did just the text posts. I don't really care, it was mainly the folder thing I wanted to fix. If you have lag between your last full backup and your master-only backup, this might cause some issues? I don't know because I made sure there wasn't lag, so I recommend doing another backup to add any missing items before doing this method.

You can also use this technique if you want to download only your original posts and then download everything else to a second folder. Adjust the setting to only download original posts, download the whole blog, close T3 and delete the indexes, rename the folder to whatever you want ("n7punk_original", etc), and then reopen T3 and set it to download everything and run it again from the start.

[mountain-lion-gremlin]

Just to let all of my followers know.

Im leaving Tumblr soon on this account! It will become a permanent sideblog and I will move on to only managing my sideblog (which will then become the main blog.)

✅️I have a bunch complied in the queue, so I'm going to let it run until it loses all of its juice (actually I just posted it all at once like a dumbass ✌️)

✅️Im going to unfollow everyone unless they are a p-shifter (Or if they are following me TY!!)

im going to export my whole account and catch all of that stuff I've written that's important (once this is done I won't post anymore. Ill delete everything except important stuff)

✅️Im also going to empty out my drafts there are too many lol (The only one left is my introduction which is going to be pinned at the top of my blog after this one)

and I'm going to recede into my tiny hole which I call home on WordPress. I'll post some hints to get anyone who is still interested (and is in the p-shifter community) to find it if they so desire to. This is to keep trolls off of my site and also who doesn't like a challenge?

Im sorry I couldn't stay for long. Its just too public here. I wish I could, but I'm tired of doomscrolling and want to go back to shifting without distractions.

Obviously I will continue to be on my Amino too.

Regardless, I'll see you around until my queue runs out of posts ♡♡

[giants-club]

In which Arilin says pointed things about open source, open data, and social media

N.B.: As I was writing this, an entirely different kerfluffle started about Meta, Facebook's parent company, working on their own ActivityPub-compatible microblogging service. While that may be a field of land mines topic worth writing about, it's a topic for a different day.

Cohost's recent financial update revealing they are running on fumes can't help but bring to my mind what I wrote back in December:

[Cohost is] still a centralized platform, and that represents a single point of failure. If Cohost takes off, it's going to require a lot more money to run than they have right now, and it's hard to know if "Cohost Plus" will be enough to offset those costs.

I have a few friends who are Cohost partisans. While its "cram posts into 40% of the browser window width" aesthetic is claws on a chalkboard to me, it's easy to see why people like the service. They have the nice things from Twitter and Tumblr, but without the ads and the NSFW policies and the right-wing trolls and the Elons. They don't have the annoying things from Mastodon---the nerdy fiddliness around "instances" endemic to any such service, and the prickly, change-resistant and mansplainy culture endemic to Mastodon in particular. They even have a manifesto! (Who doesn't like a good manifesto?) Unfortunately, what they don't have is a business model---and unlike the vast majority of Mastodon instances, they need one.

To the degree I've become a social media partisan, it's for Mastodon, despite its cultural and technical difficulties. I'm not going to beat the federation drum again, though---not directly. Instead, I'd like to discuss "silos": services whose purpose is to share user-generated content, from tweets to photos to furry porn, but that largely lock that data in.

Let's pick two extremes that aren't social media: the blogging engine WordPress, and the furry story/image archive site Fur Affinity.

WordPress itself is open source. You can put up your own WordPress site or use any number of existing commercial hosts if you like.

WordPress has export and import tools: when you change WordPress hosts, you can bring everything with you.

WordPress has open APIs: you can use a variety of other tools to make and manage your posts, not just WordPress's website.

Fur Affinity is not open source. If FA goes away, there won't be another FA, unless they transfer the assets to someone else.

You can't move from another archive site to FA or move from FA to another archive site without doing everything manually. You can't even export lists of your social graph to try to rebuild it on another site.

FA has no API, so there's no easy way for anyone else to build third-party tools to work with it.*

Cohost is, unfortunately, on the FA side of the equation. It has no official, complete API, no data export function, no nothing. If it implodes, it's taking your data down with it.

Mastodon, for all of its faults, is open-source software built on an open protocol. Anyone with sufficient know-how and resources can spin up a Mastodon (or Mastodon-compatible) server, and if you as a user need to move to a different instance for any reason, you can. And I know there are a lot of my readers who don't dig Mastodon ready to point out all the asterisks there, the sharp edges, the failures in practice. But if you as a user need to move to a different Cohost instance for any reason, there is only one asterisk there and the asterisk is "sorry, you're fucked".

I've often said that I'm more interested in open data than open source, and that's largely true: since I write nearly everything in plain text with Markdown, my writing won't be trapped in a proprietary format if the people who make my closed-source editors go under or otherwise stop supporting them. But, I'm not convinced that a server for a user-generated content site doesn't ultimately need both data and source to be open. A generation ago, folks abandoning LiveJournal who wanted to keep using an LJ-like system could migrate to Dreamwidth nearly effortlessly. Dreamwidth was a fork of LJ's open-source server, and LJ had a well-documented API for posting, reading, importing, and exporting.

To be clear, I hope Cohost pulls out of their current jam. They seem to have genuinely good motivations. But even the best of intentions can't guarantee…well, anything. Small community-driven sites have moderation faceplants all the damn time. And sites that get big enough that they can no longer be run as a hobby will need revenue. If they don't have a plan to get that revenue, they're going to be in trouble; if they do have a plan, they face the danger of enshittification. Declaring your for-profit company to be proudly anti-capitalist is not, in the final analysis, a solution to this dilemma.

And yet. I can't help but read that aspirational "against all things Silicon Valley" manifesto, look at the closed source, closed data, super-siloed service they actually built in practice, and wonder just how those two things can be reconciled. At the end of the day, there's little more authentically Silicon Valley than lock-in.

------

*I know you're thinking "what about PostyBirb"; as far as I know, PostyBirb is basically brute-forcing it by "web scraping". This works, but it's highly fragile; a relatively small change on FA's front end, even a purely aesthetic one, might break things until PostyBirb can figure out how to brute-force the new design.