Bug#830173: w3m doesn't handle the soft hyphen U+00AD correctly

[Vincent Lefevre]

Package: w3m
Version: 0.5.3-28
Severity: normal

w3m sends the soft hyphen character U+00AD to the terminal
(which is wrong) and doesn't handle it correctly.

For instance, the attached file gives in a 80-column xterm
(I've replaced the soft hyphen by a U+002D in this mail):

------------------------------------------------------------------------

The soft hyphen should not appear (except on words broken at the hyphen
position). Text-based browsers need to filter it out, as terminals behave in
different ways when they get such a character. Paragraph with a soft hyphen
between each pair of letters:

ab-cd ef-gh ij-kl mn-op qr-st uv-wx yz-ab cd-ef gh-ij kl-mn op-qr st
-uv wx-yz zy-xw vu-ts rq-po nm-lk ji-hg fe-dc ba-zy xw-vu ts-rq po-
nm lk-ji hg-fe dc-ba

ab-cd ef-gh ij-kl mn-op qr-st uv-wx yz-ab cd-ef gh-ij kl-mn op-qr st

------------------------------------------------------------------------

One gets a spurious right margin when there are hyphens characters.
In particular, in the paragraph with <pre>...</pre>, the text appears
as truncated.

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 
'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.6.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=POSIX, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages w3m depends on:
ii  libc6        2.23-1
ii  libgc1c2     1:7.4.2-8
ii  libgpm2      1.20.4-6.2
ii  libssl1.0.2  1.0.2h-1
ii  libtinfo5    6.0+20160625-1
ii  zlib1g       1:1.2.8.dfsg-2+b1

Versions of packages w3m recommends:
ii  ca-certificates  20160104

Versions of packages w3m suggests:
pn  cmigemo       <none>
ii  curl          7.47.0-1
pn  libsixel-bin  <none>
ii  man-db        2.7.5-1
ii  mime-support  3.60
pn  mpv           <none>
pn  w3m-el        <none>
ii  w3m-img       0.5.3-28
ii  wget          1.18-2
ii  xdg-utils     1.1.1-1
pn  xsel          <none>

-- no debconf information

Title: Soft Hyphen (U+00AD) Test

The soft hyphen should not appear (except on words broken at the hyphen position). Text-based browsers need to filter it out, as terminals behave in different ways when they get such a character. Paragraph with a soft hyphen between each pair of letters:

ab­cd ef­gh ij­kl mn­op qr­st uv­wx yz­ab cd­ef gh­ij kl­mn op­qr st­uv wx­yz zy­xw vu­ts rq­po nm­lk ji­hg fe­dc ba­zy xw­vu ts­rq po­nm lk­ji hg­fe dc­ba

ab­cd ef­gh ij­kl mn­op qr­st uv­wx yz­ab cd­ef gh­ij kl­mn op­qr st­uv wx­yz