builtin:git-download capability detection not working for the bordeaux build farm

Open

Details

3 participants

Ludovic Courtès
Christopher Baines
Simon Tournier

Owner: unassigned

Submitted by: Christopher Baines

Severity: important

Christopher Baines wrote on 17 Nov 2023 22:39

Recipients:(address . bug-guix@gnu.org)

Message-ID:87pm08f41q.fsf@cbaines.net

The bordeaux build farm depends on computing the derivations on one

machine, then potentially building them on a different machine.

Some of the build machines don't have a new enough guix-daemon that

understands builtin:git-download, so derivations that use this are

sometimes failing (e.g. [1])

1: https://bordeaux.guix.gnu.org/build/10cc5622-6b1f-4f28-ad9a-41cf796d7a15

One potential approach to address this is somehow have the data service

indicate that it wants compatible derivations when computing them,

rather than to have guix do feature detection against the guix-daemon

that is being used at the point the derivations are being computed.

Simon Tournier wrote on 20 Nov 2023 10:59

Recipients:

Message-ID:86zfz8219i.fsf@gmail.com

Hi Chris,

On Fri, 17 Nov 2023 at 21:39, Christopher Baines <mail@cbaines.net> wrote:

Toggle quote (7 lines)

> The bordeaux build farm depends on computing the derivations on one

> machine, then potentially building them on a different machine.

> Some of the build machines don't have a new enough guix-daemon that

> understands builtin:git-download, so derivations that use this are

> sometimes failing (e.g. [1])

Do you mean that the drv file is generated using a new daemon then this

drv file is “offloaded” to another machine using an older daemon and

thus fails to build it?

Toggle quote (5 lines)

> One potential approach to address this is somehow have the data service

> indicate that it wants compatible derivations when computing them,

> rather than to have guix do feature detection against the guix-daemon

> that is being used at the point the derivations are being computed.

Other solutions are: downgrade the daemon of one of the machine or

update the daemon of other machines, no?

Why would the latter not be possible?

Cheers

simon

Christopher Baines wrote on 20 Nov 2023 11:29

Recipients:(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 67250@debbugs.gnu.org)

Message-ID:87cyw4g1hd.fsf@cbaines.net

Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (15 lines)> Hi Chris,
>
> On Fri, 17 Nov 2023 at 21:39, Christopher Baines <mail@cbaines.net> wrote:
>
>> The bordeaux build farm depends on computing the derivations on one
>> machine, then potentially building them on a different machine.
>>
>> Some of the build machines don't have a new enough guix-daemon that
>> understands builtin:git-download, so derivations that use this are
>> sometimes failing (e.g. [1])
>
> Do you mean that the drv file is generated using a new daemon then this
> drv file is “offloaded” to another machine using an older daemon and
> thus fails to build it?

Roughly, yep.

Toggle quote (10 lines)>> One potential approach to address this is somehow have the data service
>> indicate that it wants compatible derivations when computing them,
>> rather than to have guix do feature detection against the guix-daemon
>> that is being used at the point the derivations are being computed.
>
> Other solutions are: downgrade the daemon of one of the machine or
> update the daemon of other machines, no?
>
> Why would the latter not be possible?

It's possible, just difficult. Because with the current guix-daemon, the
build coordinator can only build things that can be GC'd, some machines
deliberately run older daemons to allow building newer things without
hitting this issue.

Ludovic Courtès wrote on 22 Nov 2023 11:19

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 67250@debbugs.gnu.org)

Message-ID:87bkbmm6o1.fsf@gnu.org

Hi,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (14 lines)> The bordeaux build farm depends on computing the derivations on one
> machine, then potentially building them on a different machine.
>
> Some of the build machines don't have a new enough guix-daemon that
> understands builtin:git-download, so derivations that use this are
> sometimes failing (e.g. [1])
>
> 1: https://bordeaux.guix.gnu.org/build/10cc5622-6b1f-4f28-ad9a-41cf796d7a15
>
> One potential approach to address this is somehow have the data service
> indicate that it wants compatible derivations when computing them,
> rather than to have guix do feature detection against the guix-daemon
> that is being used at the point the derivations are being computed.

Would it work for you if we added a keyword argument to

‘open-connection’ in (guix store) that would let you select which

builtins to use?

As in:

(open-connection

#:assume-available-builtin-builders '("download"))

Ludo’.

Ludovic Courtès wrote on 22 Nov 2023 11:19

control message for bug #67250

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87a5r6m6np.fsf@gnu.org

severity 67250 important

quit

Simon Tournier wrote on 28 Nov 2023 15:10

Re: bug#67250: builtin:git-download capability detection not working for the bordeaux build farm

Recipients:(address . 67250@debbugs.gnu.org)

Message-ID:87msuy9dem.fsf@gmail.com

Hi,

On Wed, 22 Nov 2023 at 11:19, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (5 lines)

> As in:

> (open-connection

> #:assume-available-builtin-builders '("download"))

Instead, why not check in ’git-fetch’? Currently, the test is done

against the local daemon, right?

Toggle snippet (17 lines)(define* (git-fetch ref hash-algo hash
                    #:optional name
                    #:key (system (%current-system))
                    guile git)
  "Return a fixed-output derivation that fetches REF, a <git-reference>
object.  The output is expected to have recursive hash HASH of type
HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."
  (mlet %store-monad ((builtins (built-in-builders*)))
    (if (member "git-download" builtins)
        (git-fetch/built-in ref hash-algo hash name
                            #:system system)
        (git-fetch/in-band ref hash-algo hash name
                           #:system system
                           #:guile guile
                           #:git git))))

For example, why not a test as:

(if (and SOMETHING

(member "git-download" builtins))

And this SOMETHING could be a global variable, as

%assume-builtin-builder, set to true by default and turn to false with

an environment variable, as GUIX_ASSUME_BUILTIN.

Cheers,

simon

Christopher Baines wrote on 28 Nov 2023 17:54

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 67250@debbugs.gnu.org)

Message-ID:87y1ehlsup.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (27 lines)> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> The bordeaux build farm depends on computing the derivations on one
>> machine, then potentially building them on a different machine.
>>
>> Some of the build machines don't have a new enough guix-daemon that
>> understands builtin:git-download, so derivations that use this are
>> sometimes failing (e.g. [1])
>>
>> 1: https://bordeaux.guix.gnu.org/build/10cc5622-6b1f-4f28-ad9a-41cf796d7a15
>>
>> One potential approach to address this is somehow have the data service
>> indicate that it wants compatible derivations when computing them,
>> rather than to have guix do feature detection against the guix-daemon
>> that is being used at the point the derivations are being computed.
>
> Would it work for you if we added a keyword argument to
> ‘open-connection’ in (guix store) that would let you select which
> builtins to use?
>
> As in:
>
>   (open-connection
>     #:assume-available-builtin-builders '("download"))

Yep, that would at least allow freezing the available builtins going

forward and separating it from the running daemon.

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 67250@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date