#windows-1252

[magnetictapedatastorage]

i was able to visually "fix" a mis-encoded tf2chan fic using word's "find and replace" and a UTF-8 debugging chart, creating an offline copy that visually matches what it "should" look like, but it's not a fool-proof process (for example, you have to be very sure to go in order from LONGEST present mojibakes to shortest, or you will not get the right result) and it kind of takes a while. unfortunately there is literally no better way to be doing this from the user end from what i understand, but at least the result is clean.

I might try doing this with other fics for my own enjoyment, but I want to tell others how to do it too, especially for archival.

before:

and after:

the cheat sheet is here. for each instance of mojibake put the "actual" in the "find" box, and "expected" in the "replace with" box, and hit "replace all". Again, make sure to go from LONGEST to SHORTEST. first 3 character mojibakes, then 2 character, then if present 1 character. do not go out of order. If something seems off, like if it found and replaced way more end quotes than start quotes, you went out of order.

[white-diamond-search-results]

01001111 01100110 00100000 01100011 01101111 01110101 01110010 01110011 01100101 00101100 00100000 01101001 01110100 00100000 01110111 01100001 01110011 01101110 00100111 01110100 00100000 01101101 01111001 00100000 01100110 01100001 01110101 01101100 01110100 00101110 00100000 01001001 01110100 00100000 01110111 01100001 01110011 00100000 01110100 01101000 01100101 01101101 00101110 00100000 01010100 01101000 01100101 01111001 00100000 01100100 01101001 01100100 00100000 01101001 01110100 00101110 00001010 00001010 01010100 01101000 01100101 01111001 00100000 01100110 01101111 01110101 01101110 01100100 00100000 01101001 01110100 00101110

[white-diamond-search-results]

01101000 01110100 01110100 01110000 01110011 00111010 00101111 00101111 01100100 01101111 01100011 01110011 00101110 01100111 01101111 01101111 01100111 01101100 01100101 00101110 01100011 01101111 01101101 00101111 01100100 01101111 01100011 01110101 01101101 01100101 01101110 01110100 00101111 01100100 00101111 00110001 01001101 00110101 01010101 01001110 01101000 01011000 01001100 01010110 01010001 00110010 01110111 01110100 01010001 01101011 01011001 01000111 00110110 00110100 01111001 00110010 00110110 01101110 01010010 01110000 01001000 01100010 01100110 01001001 01110010 01010010 00110011 01101101 01101100 01101010 01100101 01101000 01010010 00110001 01110001 01110011 01001110 01000001 01011001 00101111 01100101 01100100 01101001 01110100 00111111 01110101 01110011 01110000 00111101 01100100 01110010 01101001 01110110 01100101 01110011 01100100 01101011

[tonywok]

UTF-8 Good, Windows-1252 Bad

So I'm using sidekiq, with sidekiq-scheduler at work in production to handle background tasks. Well, the other night around 10pm, just as I was settling down to watch the last episode of Mad Men, all hell broke lose.

After digging through the logs to find out what happened, all I could find was this

2012-05-07T01:59:10Z 24340 TID-fmbc0 ERROR: Manager#assign died 2012-05-07T01:59:10Z 24340 TID-fmbc0 ERROR: "\xC3" on US-ASCII 2012-05-07T01:59:10Z 24340 TID-fmbc0 ERROR: /home/web_apps/dream/shared/bundle/ruby/1.9.1/gems/json-1.6.6/lib/json/common.rb:148:in `encode'

None of the stack trace bubbled up to my app, so I knew I had no chance at rescuing from an exception. That left me with one thing to do, try and figure out what the heck caused this. Fortunately, I was able to lean on a few more seasoned colleagues of mine for advice.

After some discussion, we landed on our app being sent a windows-1252 encoded character, and trying to parse it as UTF-8.

So, to combat such things, we came up with this:

# This method exists because it has been shown that we cannot # trust data coming from facebook # def clean_fb_str(str) ic = Iconv.new('UTF-8//IGNORE', 'UTF-8') if str == ic.iconv(str) # valid UTF-8 string should match its iconv version str else # UTF-8 conversion changed string, force windows-1252 into utf-8 windows_ic = Iconv.new('UTF-8//IGNORE//TRANSLIT', 'WINDOWS-1252') windows_ic.iconv(str) end end

Seems to do the job, as we haven't yet had a reoccurrence of the issue.

NOTE: After deploying this on ruby-1.9.3, I've noticed we are getting deprecation warnings. I suppose we should be using String#encode instead.