May 06, 2007, at 10:31 PM
Tips /

Base64Unicode

Context

I have been pretty lucky with user contributed translations of Inline Search. We regularly receive .lang files ,which are Unicode text files, as email attachments. For some of them (not all) webmail solutions (GMail / IMP /SQwebmail at least) don't handle them correctly. The file doesn't appear as an attachment and its content is dumped, garbled, in the body of the email. The GMail team seems to be aware of the problem and willing to address it but in the meantime I want to be able to do something with the files.

Solution(s)

One way to address the issue is to pop the email with a standalone mail client (Outlook / Thunderbird) another is to select Show original in the message menu (down arrow next to the Reply button) and use an external program to decode the attachment part e.g.

--Boundary_(ID_5IDFvAnDhOV2AUWSShV6Og)
Content-type: text/plain; name=hebrew.lang
Content-transfer-encoding: base64
Content-disposition: inline; filename=hebrew.lang

//4jACAARQBuAGcAbABpAHMAaAAgAHYAIAAxAC4ANAANAAoAMQA6AOIF0QXoBdkF6gUNAAoA
DQAKADEAMAAxADoA0QXZBdgF1QXZBSAA3AXQBSAA4AXeBeYF0AUNAAoAMQAwADIAOgDUBdIF
2QXiBSAA3AXhBdUF4wUgANQF0wXjBSwAIADeBd4F6QXZBdoFIADeBegF0AXpBSAA1AXTBeMF
DQAKADEAMAAzADoA0gXoBeEF1AUgANcF0wXpBdQFIADpBdwFIABJAG4AbABpAG4AZQAgAFMA
ZQBhAHIAYwBoACAA4AXeBeYF0AXUBS4AIADcBdcF5QUgAOIF3AUgACIA2wXfBSIAIADbBdMF
2QUgANwF4gXRBdUF6AUgANwF0wXjBSAA1AXUBdUF6AXTBdQFLgANAAoAMQAwADQAOgDZBekF
IADcBdoFIADQBeoFIADUBdIF6AXhBdQFIADUBdAF1wXoBdUF4AXUBSAA6QXcBSAASQBuAGwA
aQBuAGUAIABTAGUAYQByAGMAaAAuAA0ACgAxADAANQA6AN4F5gXQBSAAOgANAAoAMQAwADYA
OgDUBeoF0AXdBSAA6AXZBekF2QXVBeoFDQAKADEAMQAxADoA3gXmBdAFIADQBeoFIADUBdEF
0AUNAAoAMQAxADIAOgDeBeYF0AUgANAF6gUgANQF5wXVBdMF3QUNAAoAMQAxADMAOgDUBdMF
0gXpBSAA1AXbBdwFDQAKAA0ACgAjACAATQBlAG4AdQANAAoAMwAyADcANgA4ADoA0AXVBdMF
1QXqBQ0ACgAzADIANwA2ADkAOgDRBdMF1QXnBSAA0AXdBSAA5wXZBdkF3QUgAOIF0wXbBdUF
3wUNAAoAMwAyADcANwAwADoA1AXqBdAF3QUgANAF2QXpBdkF6gUuAC4ALgANAAoADQAKACMA
IABPAHAAdABpAG8AbgAgAGQAaQBhAGwAbwBnAA0ACgAxADAANwA6ANQF6gXQBd0FIADQBdkF
6QXZBeoFIADQBeoFIABJAG4AbABpAG4AZQAgAFMAZQBhAHIAYwBoAA0ACgAxADAAOAA6AOkF
5AXUBQ0ACgAxADAAOQA6ANEF1wXoBSAA0AXqBSAA1AXpBeQF1AUgANQF3gXVBeIF0wXkBeoF
IADiBdwF2QXaBSAAOgANAAoAMQAxADAAOgDpBdkF4AXVBdkF2QXdBSAA0QXpBeQF1AUgANkF
1QXkBdkF4gXVBSAA0QXUBeQF4gXcBdQFIADUBdEF0AXUBSAA6QXcBSAASQBuAHQAZQByAG4A
ZQB0ACAARQB4AHAAbABvAHIAZQByAA0ACgA=

--Boundary_(ID_5IDFvAnDhOV2AUWSShV6Og)--

Pythonic way

JFR was teasing me about not being able to do it in Python (he didn't do any better in Tcl mind you :P). With the help of Duncan Booth (authors of Patterns in Pyhton) on comp.lang.python I figured it out. It is an invitation to dig a tad deeper in the Unicode world ;)

First save the encoded form (the cryptic bits //[...]=) in a file named hebrew.b64 (I knew it is the translation in Hebrew) or past it in the interpreter.

import base64

with file("hebrew.b64","r") as f:
   text = f.read().decode('base64').decode('utf16')

#"wb" i.e. binary required for utf16 output
with file("hebrew.lang","wb") as f:
    file.write(text.encode('utf16'))

I asked Duncan how he knew the file was utf16 encoded? His reply requires further study (I link here to some additional references).

I pasted the encoded form into idle and decoded it base 64. It ends with \r\x00\n\x00 and the nulls instantly suggest a 16 bit encoding. Scrolling to the beginning and it starts \xff\xfe which is the BOM for little-endian utf16.

Conclusion

I learnt (at least discovered not sure I will remember :P) something new. Now, until the GMail team fixes the issue, I would like to be able to select the encoded form in the Show original window, click (e.g. right click on the selected text and select decode) and done (e.g. a new window displaying the decoded bit :P).