From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 14 11:53:46 2017 Received: (at 28659) by debbugs.gnu.org; 14 Dec 2017 16:53:46 +0000 Received: from localhost ([127.0.0.1]:33963 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ePWlW-0000T5-0b for submit@debbugs.gnu.org; Thu, 14 Dec 2017 11:53:46 -0500 Received: from hera.aquilenet.fr ([141.255.128.1]:51794) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ePWlR-0000Ss-3w for 28659@debbugs.gnu.org; Thu, 14 Dec 2017 11:53:42 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 0F21C102D7; Thu, 14 Dec 2017 17:53:43 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ytQf5DF19oqk; Thu, 14 Dec 2017 17:53:41 +0100 (CET) Received: from ribbon (unknown [193.50.110.249]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 43DE374AF; Thu, 14 Dec 2017 17:53:41 +0100 (CET) From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Leo Famulari Subject: Re: bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail References: <877ewf18d4.fsf@gnu.org> <87o9ppoabw.fsf@gnu.org> <20171002182208.GB10773@jasmine.lan> <878tgt721q.fsf@gnu.org> <20171020211700.GA32355@jasmine.lan> <87d1421qek.fsf@gnu.org> Date: Thu, 14 Dec 2017 17:53:37 +0100 In-Reply-To: <87d1421qek.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 28 Nov 2017 14:30:59 +0100") Message-ID: <874lot9rou.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 28659 Cc: 28659@debbugs.gnu.org, Jan Nieuwenhuizen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > Thinking more about it, why not simply always enable substitutes for > fixed-output derivations, like this: > > diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc > index d68e8b2bc..03a8f5080 100644 > --- a/nix/libstore/build.cc > +++ b/nix/libstore/build.cc > @@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation() >=20=20 > /* We are first going to try to create the invalid output paths > through substitutes. If that doesn't work, we'll build > - them. */ > - if (settings.useSubstitutes && substitutesAllowed(drv)) > + them. Always enable substitutes for fixed-output derivations to > + protect against disappearing files and in-place modifications on > + upstream sites. */ > + if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(d= rv)) > foreach (PathSet::iterator, i, invalidOutputs) > addWaitee(worker.makeSubstitutionGoal(*i, buildMode =3D=3D b= mRepair)); [...] > The downside is that it still requires one to authorize the server=E2=80= =99s > key, although it=E2=80=99s in theory unnecessary since it=E2=80=99s conte= nt addressed. > I=E2=80=99m not sure how to solve that because =E2=80=98guix substitute= =E2=80=99 doesn=E2=80=99t know > that it=E2=80=99s substituting a fixed-output derivation. I suppose we= =E2=80=99d need > to modify the =E2=80=9Cprotocol=E2=80=9D between guix-daemon and =E2=80= =98guix substitute=E2=80=99. I looked at how to address this by having =E2=80=98guix substitute=E2=80=99 automatically determine whether it=E2=80=99s being asked for a content-addr= essed item or not. The guts of it is this procedure: (define* (content-addressed-item? item hash #:key (hash-algo 'sha256)) "Return true if ITEM, a store file name, is definitely a content-addres= sed item (result of a fixed-output derivation) with the given HASH of type HASH-ALGO, false otherwise. Note: This procedure is useful when the deriver of ITEM is unknown. In o= ther cases, the recommended approach is to check 'fixed-output-derivation?' on= the deriver." ;; XXX: This returns #f for "text" items produced by 'add-text-to-store= '. ;; There's not much we can do because the file name for these is a func= tion ;; of their content. (let ((name (store-path-package-name item))) (or (string=3D? item (fixed-output-path name hash #:recursive? #f #:hash-algo hash-algo)) (string=3D? item (fixed-output-path name hash #:recursive? #t #:hash-algo hash-algo))))) It works as expected for the result of =E2=80=9Crecursive fixed-output derivations=E2=80=9D=E2=80=94i.e., fixed-output derivations that produce a = directory, such as VCS checkouts. However it doesn=E2=80=99t work for fixed-output derivations that produce a= flat file, such as origins with the =E2=80=98url-fetch=E2=80=99 method. The rea= son is because in the case of non-recursive derivations, the store file name is computed as a function of the file hash, not as a function of the nar hash, whereas narinfos only contains the nar hash (the thing that =E2=80=98= guix hash -r=E2=80=99 computes.) So I think we have to communicate more info from the daemon to =E2=80=98guix substitute=E2=80=99. Ludo=E2=80=99.