problems with non-utf8 characters in mutt after upgrading to 5.4

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

problems with non-utf8 characters in mutt after upgrading to 5.4

karlis.mikelsons
Hello,

After upgrading from OpenBSD 5.3 to OpenBSD 5.4 I've got problems with
non-utf8 characters in mutt email client. It worked just fine until
upgrade, but after upgrade it doesn't show non-ascii characters in
subject or body if email message is non-utf8 (tried it with iso-8859-13,
windows-1257, koi8-r charsets, none of them display correctly).

In email messages that are UTF-8 encoded all characters display fine.
It seems that mutt stopped converting other charsets to utf8. I
exprience same issue on both i386 and amd64, by using xterm or
rxvt-unicode. I haven't changed anything regarding X.org or mutt
configuration.

I've made sure that these files were rm'ed after upgrading to 5.4:
   /usr/share/locale/*.*
   /usr/share/locale/de_AT

Any ideas what could be wrong?

Thanks!

--
Karlis

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

Stefan Sperling-8
On Thu, Jan 16, 2014 at 09:03:13PM +0200, Kārlis Miķelsons wrote:

> Hello,
>
> After upgrading from OpenBSD 5.3 to OpenBSD 5.4 I've got problems with
> non-utf8 characters in mutt email client. It worked just fine until
> upgrade, but after upgrade it doesn't show non-ascii characters in
> subject or body if email message is non-utf8 (tried it with iso-8859-13,
> windows-1257, koi8-r charsets, none of them display correctly).
>
> In email messages that are UTF-8 encoded all characters display fine.
> It seems that mutt stopped converting other charsets to utf8. I
> exprience same issue on both i386 and amd64, by using xterm or
> rxvt-unicode. I haven't changed anything regarding X.org or mutt
> configuration.
>
> I've made sure that these files were rm'ed after upgrading to 5.4:
>   /usr/share/locale/*.*
>   /usr/share/locale/de_AT
>
> Any ideas what could be wrong?
>
> Thanks!
>
> --
> Karlis

Latin1 mail with umlauts and sz in sender's name mail body renders
fine here in mutt in xterm in the UTF-8 locale. So you should be
able to get it to work. I suspect misconfiguration rather than a bug,
though a bug is of course a possibility.

What does your local configuration look like?
Are you aware of http://www.openbsd.org/faq/faq10.html#locales ?

Do you have any charset-related options in muttrc, like 'set charset'
or 'set assumed_charset'? Mutt should be able to auto-detect such
settings nowadays. Does adding 'set rfc2047_parameters=yes' help?

If none of this helps, can you (privately) forward or bounce mail
to me that fails to display correctly?

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

karlis.mikelsons
> Latin1 mail with umlauts and sz in sender's name mail body renders
> fine here in mutt in xterm in the UTF-8 locale. So you should be
> able to get it to work. I suspect misconfiguration rather than a bug,
> though a bug is of course a possibility.
>
> What does your local configuration look like?
> Are you aware of http://www.openbsd.org/faq/faq10.html#locales ?
Yes, I've got "export LC_CTYPE=en_US.UTF-8" in my .xinitrc.

$ env | grep LC
LC_CTYPE=en_US.UTF-8
$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE=en_US.UTF-8
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=

I've also tried different combinations of "LC_ALL=en_US.UTF-8" and
"LANG=en_US.UTF-8" with no success.


> Do you have any charset-related options in muttrc, like 'set charset'
> or 'set assumed_charset'? Mutt should be able to auto-detect such
> settings nowadays. Does adding 'set rfc2047_parameters=yes' help?
No, I haven't got any charset configuration in my muttrc configuration
file. Setting "rfc2047_parameters=yes" didn't change anything.

I've also made sure that mutt detects correctly the charset of the
locale by following command:
:set &charset ?charset
charset="utf-8"

> If none of this helps, can you (privately) forward or bounce mail
> to me that fails to display correctly?
Sure, I'll send a couple of emails to you shortly.

Thanks for your help, Stefan!

--
Karlis

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

Dmitrij D. Czarkoff-2
Kārlis Miķelsons said:

> >Latin1 mail with umlauts and sz in sender's name mail body renders
> >fine here in mutt in xterm in the UTF-8 locale. So you should be
> >able to get it to work. I suspect misconfiguration rather than a bug,
> >though a bug is of course a possibility.
> >
> >What does your local configuration look like?
> >Are you aware of http://www.openbsd.org/faq/faq10.html#locales ?
> Yes, I've got "export LC_CTYPE=en_US.UTF-8" in my .xinitrc.
>
> $ env | grep LC
> LC_CTYPE=en_US.UTF-8
> $ locale
> LANG=
> LC_COLLATE="C"
> LC_CTYPE=en_US.UTF-8
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_MESSAGES="C"
> LC_ALL=

Mutt works OK for me with multiple sets of non-ASCII charecters,
including advanced punctuation, cyrillic, extended Latin, etc.
(Basically everything one may possibly need for English, Serbo-Croatian,
Russian and French).

        $ locale
        LANG=en_US.UTF-8
        LC_COLLATE="C"
        LC_CTYPE="en_US.UTF-8"
        LC_MONETARY="C"
        LC_NUMERIC="C"
        LC_TIME="C"
        LC_MESSAGES="en_US.UTF-8"
        LC_ALL=en_US.UTF-8

I guess you could try setting LC_MESSAGES (if mutt happens to take it
for display charset, this trick my work - I can't test it right now).

FWIW did you make sure UTF-8 works on your terminal at all? Did you try
mutt in uxterm?

--
Dmitrij D. Czarkoff

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

karlis.mikelsons
> Mutt works OK for me with multiple sets of non-ASCII charecters,
> including advanced punctuation, cyrillic, extended Latin, etc.
> (Basically everything one may possibly need for English,
> Serbo-Croatian,
> Russian and French).
>
> $ locale
> LANG=en_US.UTF-8
> LC_COLLATE="C"
> LC_CTYPE="en_US.UTF-8"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_MESSAGES="en_US.UTF-8"
> LC_ALL=en_US.UTF-8
>
> I guess you could try setting LC_MESSAGES (if mutt happens to take it
> for display charset, this trick my work - I can't test it right now).
Tried setting all LANG, LC_CTYPE and LC_ALL to "en_US.UTF-8", but
non-utf8 non-ascii charaters still don't display.
$ locale
LANG=en_US.UTF-8
LC_COLLATE="C"
LC_CTYPE="en_US.UTF-8"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=en_US.UTF-8

I can't seem to be able to set up both LC_ALL and LC_MESSAGES to
"en_US.UTF-8" at the same time:
$ locale
LANG=en_US.UTF-8
LC_COLLATE="C"
LC_CTYPE="en_US.UTF-8"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=en_US.UTF-8
$ export LC_MESSAGES="en_US.UTF-8"
$ env | grep MESSA
LC_MESSAGES=en_US.UTF-8
$ locale
LANG=en_US.UTF-8
LC_COLLATE="C"
LC_CTYPE="en_US.UTF-8"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C" <--- it didn't change
LC_ALL=en_US.UTF-8

But I can change LC_MESSAGES if LC_ALL is unset:
$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE=en_US.UTF-8
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=
$ export LANG="en_US.UTF-8"
$ export LC_MESSAGES="en_US.UTF-8"
$ locale
LANG=en_US.UTF-8
LC_COLLATE="C"
LC_CTYPE=en_US.UTF-8
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES=en_US.UTF-8
LC_ALL=

But non-utf8 non-ascii characters still don't work with any of these
combinations.

I also noticed that your LC_CTYPE and LC_MESSAGES variables have
quotation marks around them but mine don't. Could it affect mutt or
is it only locale(1) output formatting?

> FWIW did you make sure UTF-8 works on your terminal at all? Did you try
> mutt in uxterm?
Yes, as I've said before, UTF-8 encoded messages show up fine, I can
edit text in both xterm and urxvt by using vim in UTF-8, and I can
send UTF-8 encoded emails from mutt just fine.

Thanks!


--
Karlis

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

karlis.mikelsons
In reply to this post by Stefan Sperling-8
> If none of this helps, can you (privately) forward or bounce mail
> to me that fails to display correctly?
As suggested by Stefan offlist, updating from mutt-1.4.2.3p3v0 to
mutt-1.5.21p4v0 seems to have fixed the problem.

Thank you everyone for your help!

--
Karlis

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

Denis
In reply to this post by karlis.mikelsons
Hi Karlis,

I think you may have forgotten to set PKG_PATH to new value (with 5.4 in it),
when running "pkg_add -u" after upgrading from OpenBSD 5.3 to OpenBSD 5.4.

This mistake may cause installation of old 5.3 packages into new 5.4 system.
Try to compare versions of linked libraries (ldd mutt etc) with "vanilla"
system.

On my 5.4 box mutt has been linked like this, check against your system:
(libc and libiconv)

/usr/local/bin/mutt:
        Start    End      Type Open Ref GrpRef Name
        1c000000 3c01e000 exe  1    0   0      /usr/local/bin/mutt
        02648000 22658000 rlib 0    1   0      /usr/lib/libncurses.so.12.1
        0427b000 24282000 rlib 0    1   0      /usr/lib/libgssapi.so.6.0
        07079000 27086000 rlib 0    1   0      /usr/lib/libkrb5.so.19.0
        0fa04000 2fa0a000 rlib 0    1   0      /usr/lib/libasn1.so.19.0
        090b5000 290f4000 rlib 0    1   0      /usr/lib/libcrypto.so.22.0
        0fd96000 2fdbe000 rlib 0    1   0      /usr/lib/libwind.so.0.0
        0c4c4000 2c4c8000 rlib 0    1   0      /usr/lib/libroken.so.0.0
        08e87000 28e8b000 rlib 0    1   0      /usr/lib/libcom_err.so.19.0
        03fb0000 23fb4000 rlib 0    1   0      /usr/lib/libheimbase.so.0.0
        0cbe5000 2cbf2000 rlib 0    1   0      /usr/lib/libssl.so.19.0
        0dea6000 2dead000 rlib 0    2   0      /usr/lib/libz.so.4.1
        091e3000 291e8000 rlib 0    1   0      /usr/local/lib/libqdbm.so.14.14
        06734000 26738000 rlib 0    1   0      /usr/local/lib/libintl.so.6.0
        03a13000 23af3000 rlib 0    3   0      /usr/local/lib/libiconv.so.6.0
        01611000 21641000 rlib 0    2   0      /usr/lib/libc.so.69.0
        072bb000 072bb000 rtld 0    1   0      /usr/libexec/ld.so


You may also revert some of the changes you have done sofar :-), while trying
to fix the issue ...

--
Sincerely yours,
Denis
=====
The Bible for command line people.
http://www.read-and-think.org/kjv.html
=====

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

LeviaComm Networks NOC
Denis wrote:
> Hi Karlis,
>
> I think you may have forgotten to set PKG_PATH to new value (with 5.4 in it),
> when running "pkg_add -u" after upgrading from OpenBSD 5.3 to OpenBSD 5.4.
>

This is why I set my PKG_PATH in .profile to:
<my ftp server>/pub/`uname -s`/`uname -r`/packages/`machine -a`/

with a sym-link for release+1 to /snapshots (ie 5.5 -> snapshots) on
the ftp server (I've built a local mirror)

That way its always pointed to the right repository no matter what box I
am using, whether is one of production boxes running an older release
of OpenBSD, one of my FreeBSD boxes (xbmc, splunk, etc) or my OpenBSD
desktop running -current as well as being platform agnostic.  It hasn't
needed a change since at least 4.2 when I first started using OpenBSD.

Reply | Threaded
Open this post in threaded view
|

Re: problems with non-utf8 characters in mutt after upgrading to 5.4

karlis.mikelsons
In reply to this post by Denis
Hello Denis,

> I think you may have forgotten to set PKG_PATH to new value (with 5.4
> in it),
> when running "pkg_add -u" after upgrading from OpenBSD 5.3 to OpenBSD
> 5.4.
>
> This mistake may cause installation of old 5.3 packages into new 5.4
> system.
> Try to compare versions of linked libraries (ldd mutt etc) with
> "vanilla"
> system.
No, it's not the case, I've got my PKG_PATH set up in /etc/pkg.conf and
it is
and always has been correct.


--
Karlis