git-fetch does not update checked out tree when commit hash changes

  • Done
  • quality assurance status badge
Details
3 participants
  • Jookia
  • Ludovic Courtès
  • Pjotr Prins
Owner
unassigned
Submitted by
Pjotr Prins
Severity
normal
P
P
Pjotr Prins wrote on 21 Jan 2016 07:54
(address . bug-guix@gnu.org)
20160121065403.GA4278@thebird.nl
I can reliably reproduce this using a recent version of GNU Guix.

When updating the commit hash to a different commit the git-fetch
derivation *does* change (I checked in guile), but the checked out git
tree in the store does not change - it gets shared between the
commits. I am not sure why the tree gets shared, but the effect is
that the same package gets installed using the same
/gnu/store/xxx-git-checkout.

Removing the git-checkout dir and updating the Hash gives a missing
dir error (as expected when they use the same).
L
L
Ludovic Courtès wrote on 21 Jan 2016 09:50
(name . Pjotr Prins)(address . pjotr.public12@thebird.nl)(address . 22423-done@debbugs.gnu.org)
87twm79v05.fsf@gnu.org
Pjotr Prins <pjotr.public12@thebird.nl> skribis:

Toggle quote (7 lines)
> When updating the commit hash to a different commit the git-fetch
> derivation *does* change (I checked in guile), but the checked out git
> tree in the store does not change - it gets shared between the
> commits. I am not sure why the tree gets shared, but the effect is
> that the same package gets installed using the same
> /gnu/store/xxx-git-checkout.

This is expected: origins are fixed-output derivations, meaning that it
does not matter how we perform them (using Git, over HTTP, or thanks to
an avian carrier), as long as the result has the specified sha256.

Thus, when you change, say, the Git commit ID or origin ‘method’ without
changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
already have a store item with that ‘sha256’, so I don’t do anything.”

Clearly, one has to be cautious with this, it’s easy to mistakenly use
the old source.

Hope this clarifies things!

Ludo’.
Closed
P
P
Pjotr Prins wrote on 21 Jan 2016 10:08
(address . 22423-done@debbugs.gnu.org)
20160121090853.GA4914@thebird.nl
On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
Toggle quote (11 lines)
> This is expected: origins are fixed-output derivations, meaning that it
> does not matter how we perform them (using Git, over HTTP, or thanks to
> an avian carrier), as long as the result has the specified sha256.
>
> Thus, when you change, say, the Git commit ID or origin ‘method’ without
> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
> already have a store item with that ‘sha256’, so I don’t do anything.”
>
> Clearly, one has to be cautious with this, it’s easy to mistakenly use
> the old source.

Hmmm. I thought the sha256 was calculated over the derivation +
sources, so any relevant change would trigger a build. Apparently it
is triggered by the sha256 field only. Good to know.

Pj.
Closed
L
L
Ludovic Courtès wrote on 21 Jan 2016 11:10
(name . Pjotr Prins)(address . pjotr.public12@thebird.nl)(address . 22423-done@debbugs.gnu.org)
87vb6n1bw9.fsf@gnu.org
Pjotr Prins <pjotr.public12@thebird.nl> skribis:

Toggle quote (15 lines)
> On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
>> This is expected: origins are fixed-output derivations, meaning that it
>> does not matter how we perform them (using Git, over HTTP, or thanks to
>> an avian carrier), as long as the result has the specified sha256.
>>
>> Thus, when you change, say, the Git commit ID or origin ‘method’ without
>> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
>> already have a store item with that ‘sha256’, so I don’t do anything.”
>>
>> Clearly, one has to be cautious with this, it’s easy to mistakenly use
>> the old source.
>
> Hmmm. I thought the sha256 was calculated over the derivation +
> sources

What you’re saying is true of the hash that appears in /gnu/store file
name, but I was referring to the ‘sha256’ field of origins, which is a
different thing.

Ludo’.
Closed
J
J
Jookia wrote on 21 Jan 2016 12:50
(name . Ludovic Courtès)(address . ludo@gnu.org)
20160121115002.GA23171@novena-choice-citizen.lan
On Thu, Jan 21, 2016 at 11:10:14AM +0100, Ludovic Courtès wrote:
Toggle quote (23 lines)
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
> > On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
> >> This is expected: origins are fixed-output derivations, meaning that it
> >> does not matter how we perform them (using Git, over HTTP, or thanks to
> >> an avian carrier), as long as the result has the specified sha256.
> >>
> >> Thus, when you change, say, the Git commit ID or origin ‘method’ without
> >> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
> >> already have a store item with that ‘sha256’, so I don’t do anything.”
> >>
> >> Clearly, one has to be cautious with this, it’s easy to mistakenly use
> >> the old source.
> >
> > Hmmm. I thought the sha256 was calculated over the derivation +
> > sources
>
> What you’re saying is true of the hash that appears in /gnu/store file
> name, but I was referring to the ‘sha256’ field of origins, which is a
> different thing.
>
> Ludo’.

I think this is a bit of a problem even if it's expected: Often times we can't
calculate the hash until it's downloaded and get a hash mismatch. The other day
I rebuilt NixOS almost entirely on my machine and changed the revision on
Firefox to a new branch but didn't change the hash since I expected a mismatch.

Needles to say I realized what happened when I checked Firefox's version. I
think it'd be great to have a 'INVALID' hash we can use for development that
just prints a mismatch and errors out like usual. Maybe this is possible in
Guix, but it didn't seem documented and it's not possible in NixOS.

Cheers,
Jookia.
Closed
L
L
Ludovic Courtès wrote on 21 Jan 2016 17:25
(name . Jookia)(address . 166291@gmail.com)
87vb6mzypn.fsf@gnu.org
Jookia <166291@gmail.com> skribis:

Toggle quote (34 lines)
> On Thu, Jan 21, 2016 at 11:10:14AM +0100, Ludovic Courtès wrote:
>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>>
>> > On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
>> >> This is expected: origins are fixed-output derivations, meaning that it
>> >> does not matter how we perform them (using Git, over HTTP, or thanks to
>> >> an avian carrier), as long as the result has the specified sha256.
>> >>
>> >> Thus, when you change, say, the Git commit ID or origin ‘method’ without
>> >> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
>> >> already have a store item with that ‘sha256’, so I don’t do anything.”
>> >>
>> >> Clearly, one has to be cautious with this, it’s easy to mistakenly use
>> >> the old source.
>> >
>> > Hmmm. I thought the sha256 was calculated over the derivation +
>> > sources
>>
>> What you’re saying is true of the hash that appears in /gnu/store file
>> name, but I was referring to the ‘sha256’ field of origins, which is a
>> different thing.
>>
>> Ludo’.
>
> I think this is a bit of a problem even if it's expected: Often times we can't
> calculate the hash until it's downloaded and get a hash mismatch. The other day
> I rebuilt NixOS almost entirely on my machine and changed the revision on
> Firefox to a new branch but didn't change the hash since I expected a mismatch.
>
> Needles to say I realized what happened when I checked Firefox's version. I
> think it'd be great to have a 'INVALID' hash we can use for development that
> just prints a mismatch and errors out like usual. Maybe this is possible in
> Guix, but it didn't seem documented and it's not possible in NixOS.

The problem is that Guix cannot guess that the ‘sha256’ the developer
provided in the ‘origin’ form is actually wrong. Only the developer
knows.

This is mitigated in Guix by the fact that we usually include the
version string or Git commit ID in the origin’s file name, with patterns
like:

(define foo
(let ((commit "deadbeef"))
(package
(name "foo")
(version commit)
(source (origin
(method git-fetch)
(uri (git-reference (commit commit) …))
(file-name (string-append name "-" commit)) ;<– !
(sha256 …)
…))
…)))

The good thing with this pattern is that, when you modify the value of
‘commit’, you also end up modifying ‘file-name’, thus triggering a
re-download. So if you had forgotten to update ‘sha256’, you
immediately get a hash mismatch error.

In practice, I’ve found this to be rather helpful.

Additionally, ‘guix lint’ emits warnings for packages that do not follow
this pattern, hopefully making the problem that Pjotr describes less
likely.

Ludo’.
Closed
?