tag:blogger.com,1999:blog-8935780327334775165.post7113505106500924870..comments2023-08-07T22:48:57.800+08:00Comments on Entity Crisis: Unicode MadnessUnknownnoreply@blogger.comBlogger5125tag:blogger.com,1999:blog-8935780327334775165.post-78355176367169463562007-10-23T06:26:00.000+08:002007-10-23T06:26:00.000+08:00Did you mean to do this:>>> x = u'\xbb'>>> ascii_x...Did you mean to do this:<BR/><BR/>>>> x = u'\xbb'<BR/>>>> ascii_x = x.encode('ascii', 'ignore')<BR/>>>> ascii_x<BR/>''<BR/>>>> ascii_x.decode('ascii', 'ignore')<BR/>u''<BR/><BR/>? ...which shows that the unicode character is ignored instead of encoded.<BR/><BR/>I never understood unicode better than when I posted a blog entry *as if* I understood it, then proceeded to get schooled in the comments (and even hazed by reddit users). Luckily, I corrected the post along the way! :)<BR/><BR/>here: <A HREF="http://farmdev.com/thoughts/23/what-i-thought-i-knew-about-unicode-in-python-amounted-to-nothing/" REL="nofollow">What I Thought I Knew About Unicode in Python Amounted To Nothing</A>Kumar McMillanhttps://www.blogger.com/profile/18371805776129363077noreply@blogger.comtag:blogger.com,1999:blog-8935780327334775165.post-13766509902803179122007-10-23T04:43:00.000+08:002007-10-23T04:43:00.000+08:00Educational stuff here - thanks for the post and t...Educational stuff here - thanks for the post and the ensuing discussion.Christianhttps://www.blogger.com/profile/14905012683456967746noreply@blogger.comtag:blogger.com,1999:blog-8935780327334775165.post-78181061290380266002007-10-22T18:02:00.000+08:002007-10-22T18:02:00.000+08:00The decode is there because both unicode and strin...The decode is there because both unicode and string inherits from a common basecalss, that has the decode and encode methods, I've been told.<BR/><BR/>I think the unicode.decode and string.encode are going away in Python 3.<BR/><BR/>In short, you get an error because you are doing, decoding uniode into unicode, makes no sense. Python here breaks it's explicit rule, and tries to guess what you want, and it guesses that the unicode thingy you gave it is supposed to be a string. So it tries to convert it to a string. That's what fails.<BR/><BR/>"if you have a unicode data source then you may want to decode from base64 into a unicode string I suppose?"<BR/><BR/>Well, yeah, but that's base64 encoding, not unicode encoding, so I'd expect you want to use b64decode for that...Lennart Regebrohttps://www.blogger.com/profile/08337807480455483637noreply@blogger.comtag:blogger.com,1999:blog-8935780327334775165.post-12907036070934970222007-10-22T17:38:00.000+08:002007-10-22T17:38:00.000+08:00I guess API consitency is one reason that unicode....I guess API consitency is one reason that unicode.decode exists - but there are also some pretty weird encodings out there and if you have a unicode data source then you may want to decode from base64 into a unicode string I suppose?Michael Foordhttps://www.blogger.com/profile/06229713779852499022noreply@blogger.comtag:blogger.com,1999:blog-8935780327334775165.post-76774869172379084852007-10-22T15:54:00.000+08:002007-10-22T15:54:00.000+08:00This is what I consider a wart, but I'm sure it se...This is what I consider a wart, but I'm sure it seemed like a good idea at the time. <BR/><BR/>A "decode" in Python is going *from* bytes *into* unicode. You're starting from unicode, so the code has to get into bytes first in order to get back into unicode (!) and does an *implicit* encode. Which fails, because it's using ascii and your string is non-ascii-encodable.<BR/><BR/>Quite why the unicode object even *has* a decode method I'm not sure. It seems to go against the "explicit is better than implicit" bit, but I presume there was some rationale at the time.Anonymousnoreply@blogger.com