ITADN

Wide chars, UTF-8, terminal escapes and colors, etc.

#69Openapjanke 创建于 2024-12-01
bug
A
apjankecommented
Cowsay doesn't handle variant character widths well. It kind of assumes all characters are 1 char wide (in display) and (I think) 1 byte (in the input encoding). This means that non-English/Latin characters in the cows or the message text are not handled well. This is an expansion of [#65 "Use Debian's UTF-8 Patch"](https://github.com/cowsay-org/cowsay/issues/65). Aspects: * Messages with multi-byte encoded characters don't look right. The speech balloon width and word wrapping are wrong. * Terminal control (escape) sequences, like color codes, are treated as visible characters. Bad-wrapped multi-byte char [example](https://github.com/cowsay-org/cowsay/issues/65): <img width="373" alt="image" src="https://github.com/user-attachments/assets/603645d4-f9d4-4367-82a8-5b21593a45c1"> ## Considerations The cow files distributed with cowsay are all UTF-8, regardless of what locale the user is running in or how their system is set up. (I think? Or are they actually ASCII/Latin-1, since they are Perl source code?) Message input might be in the user's locale while the cow files are UTF-8. Custom cows (including in third-party cow herd packages) might be in other encodings, which may or may not be the same encoding as Perl's standard library doesn't support char width detection, I don't think. Would need a CPAN module for that. We currently don't take any deps on modules. Would need to figure out how to do that. I think we'd vendor the module (ship a copy of it in cowsay itself), to avoid creating any external dependencies or a more complicated install process. ## Testing Examples: * `cowsay "MÖÖÖ"` * `cowsay 'Привет, мир!'` * `cowsay 'Ищу свое лицо. Особых примет нет.'` Wide chars: * `cowsay "我愛中國人"` * from [Debian 769565 "widechar not good"](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=769565): * `cowsay 'でびあん/Debian'` * `cowsay 谢谢你` ANSI [terminal escapes](https://bugs.launchpad.net/ubuntu/+source/cowsay/+bug/1027033): * `echo 'Hello, World!' | toilet -w 100 --metal | cowsay -n` * `figlet "Hello World!" | toilet -f term --metal | /usr/games/cowsay -n` ## TODO * [ ] Determine and document which encoding(s) are supported for cowfiles. * [ ] Make our source code UTF-8 (with `use utf8;`)? * [-] Add support for detecting, and maybe explicitly setting, message input encoding. * [-] Bump required Perl to 5.8.1 (or 5.8.7, or later), which added Unicode fixes relevant to this UTF-8 stuff? * 5.8.0 added Unicode support and UTF8-ness of stdin/out/err/ARGV. * 5.8.1 restored behavior of stdin/out/err and ARGV not being interpreted as UTF8 by default. * ${^UTF8LOCALE} was added in 5.8.7. * [ ] Support wide chars and invisible chars (like terminal escapes). * ("Wide" in the sense that they are displayed 2-char width; not that they're a `wchar` type in encoding.) * [ ] Add tests for multibyte and wide characters. ## References * https://github.com/cowsay-org/cowsay/issues/65 * Third-party patches * [Debian patch for UTF-8 input and char sizes](https://sources.debian.org/patches/cowsay/3.03%2Bdfsg2-8/utf8_width/) * [Debian bug 2544557](http://bugs.debian.org/254557) * [Ubuntu bug 393212](https://bugs.launchpad.net/ubuntu/+source/cowsay/+bug/393212) * ANSI escape codes * [Ubuntu bug 1027033 "should ignore ANSI codes for width"](https://bugs.launchpad.net/ubuntu/+source/cowsay/+bug/1027033) * [Ubuntu bug 1437804 "confused by input with ANSI colour codes"](https://bugs.launchpad.net/ubuntu/+source/cowsay/+bug/1437804) * Perl doco * [${^UTF8LOCALE}](https://perldoc.perl.org/variables/$%7B%5EUTF8LOCALE%7D) * [perlrun -C Unicode features](https://perldoc.perl.org/perlrun#-C-%5Bnumber%2Flist%5D) * ["UTF-8 no longer default under UTF-8 locales"](https://perldoc.perl.org/perl581delta#UTF-8-no-longer-default-under-UTF-8-locales)
5 条评论