From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 27 12:36:51 2019 Received: (at 35350) by debbugs.gnu.org; 27 Apr 2019 16:36:51 +0000 Received: from localhost ([127.0.0.1]:35863 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKQJj-0007j5-Rl for submit@debbugs.gnu.org; Sat, 27 Apr 2019 12:36:48 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47496) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKQJf-0007im-1J for 35350@debbugs.gnu.org; Sat, 27 Apr 2019 12:36:46 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48657) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hKQJZ-0004y2-Ag; Sat, 27 Apr 2019 12:36:37 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=37674 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hKQJX-00064m-Vz; Sat, 27 Apr 2019 12:36:37 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#35350: Some compile output still leaks through with --verbosity=1 References: <87mukkfd2j.fsf@netris.org> <87r29v2jz2.fsf@gnu.org> <87ftq9silk.fsf@netris.org> <87imv5jai5.fsf@gnu.org> <87k1fgh9c0.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 =?utf-8?Q?Flor=C3=A9al?= an 227 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 27 Apr 2019 18:36:34 +0200 In-Reply-To: <87k1fgh9c0.fsf@netris.org> (Mark H. Weaver's message of "Fri, 26 Apr 2019 15:09:24 -0400") Message-ID: <874l6jh0bx.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35350 Cc: 35350@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Mark, Mark H Weaver skribis: > Ludovic Court=C3=A8s writes: > >> The third read(2) call here ends on a partial UTF-8 sequence for LEFT >> SINGLE QUOTATION MARK (we get the first two bytes of a three byte >> sequence.) >> >> What happens is that =E2=80=98process-stderr=E2=80=99 in (guix store) ge= ts that byte >> string from the daemon, passes it through =E2=80=98read-maybe-utf8-strin= g=E2=80=99, >> which replaces the last two bytes with REPLACEMENT CHARACTER, which is >> itself a 3-byte sequence. > > It seems to me that what's needed here is to save the UTF-8 decoder > state between calls to 'process-stderr'. So there are two things. To fix the issue you reported (build output that goes through), I think we must simply turn off UTF-8 decoding from =E2=80=98process-stderr=E2=80=99 and leave that entirely to =E2=80=98build-= event-output-port=E2=80=99. However, =E2=80=98build-event-output-port=E2=80=99 would still fail to prop= erly decode split UTF-8 sequences, and for that we=E2=80=99d need to preserve decoder s= tate as you describe. > Coincidentally, I also needed something like this a week ago, when I > tried implementing R6RS custom textual input/output ports on top of > R6RS custom binary input/output ports. > > To meet these needs, I've implemented a fairly efficient, purely > functional UTF-8 decoder in Scheme that accepts a decoder state and an > arbitrary range from a bytevector, and returns a new decoder state. > There's a macro that allows arbitrary actions to be performed when a > code point (or maximal subpart in the case of errors) is found. > > This macro is then used to implement a decoder (utf8->string!) that > writes into an arbitrary range of an existing string. Of course, it's > not purely functional, but it avoids heap allocation when compiled with > Guile. On my Thinkpad X200, it can process around 10 megabytes per > second. > > The state is represented as an exact integer between 0 and #xF48FBF > inclusive, which are simply the bytes that have been seen so far in the > current code sequence, in big-endian order, or 0 for the start state. > For example, #xF48FBF represents the state where the bytes (F4 8F BF) > have been read. The state is always either 0 or a proper prefix of a > valid UTF-8 byte sequence. Awesome! I think that=E2=80=99s something we should definitely add to Guile proper. We can use it in Guix before or after it=E2=80=99s included in Gui= le. Thank you! Ludo=E2=80=99.