Docker package retains Go references (was Re: Guix closure size of a system?)

Done

Details

6 participants

Danny Milosavljevic
Julien Lepiller
Leo Famulari
Pierre Neidhardt
Marius Bakke
zimoun

Owner: unassigned

Submitted by: Leo Famulari

Severity: normal

Leo Famulari wrote on 19 May 2020 19:27

Recipients:(address . bug-guix@gnu.org)(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)

Message-ID:20200519172740.GA3947@jasmine.lan

The Docker package should not refer to the Go compiler after it is

built:

On Tue, May 19, 2020 at 12:22:28PM -0400, Leo Famulari wrote:

Toggle quote (18 lines)> On Tue, May 19, 2020 at 05:58:54PM +0200, Pierre Neidhardt wrote:
> > I haven't looked at the details, but
> > 
> > --8<---------------cut here---------------start------------->8---
> > > guix size docker
> > store item                                                       total    self
> > /gnu/store/w8gjhcw6a16rk1dvxa97bz2znal5fihm-go-1.13.9              639.1   350.7  28.2%
> > --8<---------------cut here---------------end--------------->8---
> > 
> > Then
> > 
> > --8<---------------cut here---------------start------------->8---
> > > grep w8gjhcw6a16rk1dvxa97bz2znal5fihm dockerd
> > Binary file dockerd matches
> > --8<---------------cut here---------------end--------------->8---
> 
> I don't think that package is using the go-build-system, so that is
> probably the reason for the bogus reference.

Danny Milosavljevic wrote on 19 May 2020 20:16

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:20200519201641.0bc4b2d4@scratchpost.org

Hi Leo,

the patch below makes the grep return nothing, but guix size still contains go.

Toggle diff (15 lines)diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
index c95ca3f9d7..bf009a792b 100644
--- a/gnu/packages/docker.scm
+++ b/gnu/packages/docker.scm
@@ -544,7 +544,9 @@ built-in registry server of Docker.")
                (install-file (string-append "bundles/dynbinary-daemon/dockerd-"
                                             (getenv "VERSION"))
                              out-bin)
-               #t))))))
+               #t)))
+         (add-after 'install 'remove-go-references
+           (assoc-ref go:%standard-phases 'remove-go-references)))))
     (inputs
      `(("btrfs-progs" ,btrfs-progs)
        ("containerd" ,containerd)       ; for containerd-shim

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl7EIokACgkQ5xo1VCww

uqVPOwf+NNMF+02yXQH078ijw5y4pZtJjWXmcDn8W3a3+fcZOXamnyLAXKFBwlOm

XVW4NeBUj1jKCpha+EGPD9lYl3Fet4lVnggh2D3UUCfX+BIjTPpyohsMJ+vkVVur

zf9f97oxds7AFnr2kVR9+Mn4WzTZlX8s+jkMC2A6lpOqnN2PfbYOzh0k3d1ZZxnI

XOKeB48tpdo3qqsZ12CUubTqgkNRYxDtnh3HTW7qJJWXKDjHjpOQzuEbd0QhiMXH

XnS4GtThWDA/dcWBeLkeqriiXjVK6CRmtV0ZfWsPbXgj/HEK3V9Yy28dnUq0ts/w

YfIT2cnocdT3Ink5GypUP7tFreXx9Q==

=f3Vt

-----END PGP SIGNATURE-----

Leo Famulari wrote on 19 May 2020 20:25

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20200519182523.GA8397@jasmine.lan

On Tue, May 19, 2020 at 08:16:41PM +0200, Danny Milosavljevic wrote:

Toggle quote (4 lines)

> Hi Leo,

> the patch below makes the grep return nothing, but guix size still contains go.

What about `guix gc --references /gnu/store/...-docker`?

Toggle quote (17 lines)> 
> diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
> index c95ca3f9d7..bf009a792b 100644
> --- a/gnu/packages/docker.scm
> +++ b/gnu/packages/docker.scm
> @@ -544,7 +544,9 @@ built-in registry server of Docker.")
>                 (install-file (string-append "bundles/dynbinary-daemon/dockerd-"
>                                              (getenv "VERSION"))
>                               out-bin)
> -               #t))))))
> +               #t)))
> +         (add-after 'install 'remove-go-references
> +           (assoc-ref go:%standard-phases 'remove-go-references)))))
>      (inputs
>       `(("btrfs-progs" ,btrfs-progs)
>         ("containerd" ,containerd)       ; for containerd-shim
>

Danny Milosavljevic wrote on 19 May 2020 20:32

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:20200519203234.0bf5acd1@scratchpost.org

On Tue, 19 May 2020 14:25:23 -0400

Leo Famulari <leo@famulari.name> wrote:

Toggle quote (2 lines)

> guix gc --references

Doesn't contain go.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl7EJkIACgkQ5xo1VCww

uqXCrgf+MbdYqlsn0z1PQLU1qQ8UhqjkJBQyjtlyZmVezMjJ7JXwJ+svmTnrZKNI

6QDTNSw5wDYn/qcMK3Z2xU1pYEwPy6zrbg/wjmmgbfD7CrsgQ/KtxUkTELhhdKWU

4IlMS9Pxv+9hYRgmZGtiyp1nxYDMDZ5vaFizYE8PAfMr9Ezwz08HvnZbnE1wYwoe

FRR5Az+03SLlAlGcC7AQ/ria+lgR+UZtG+jv+vTAc05PKe/BjPITGNyhz0XzsjyG

GCojl52eKcuA3VGFZRu4nO4GylPZytrqim2HlWr/JodaX2Lell8J9zA/+amf9C1h

d1KBoKeFpq7N+oarseSHoQfLWrd7sA==

=R5Rs

-----END PGP SIGNATURE-----

Julien Lepiller wrote on 19 May 2020 20:43

Recipients:

Message-ID:1BFEB07A-A866-4BCB-B1A0-AA06EE1E63AE@lepiller.eu

Le 19 mai 2020 14:32:34 GMT-04:00, Danny Milosavljevic <dannym@scratchpost.org> a écrit :

Toggle quote (7 lines)

>On Tue, 19 May 2020 14:25:23 -0400

>Leo Famulari <leo@famulari.name> wrote:

>> guix gc --references

>Doesn't contain go.

You can try guix graph -t references --path /gnu/store/…-docker /gnu/store/…-go

That should tell you which package references go and how it gets in the closure of docker.

Leo Famulari wrote on 19 May 2020 21:00

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20200519190039.GC8397@jasmine.lan

On Tue, May 19, 2020 at 08:32:34PM +0200, Danny Milosavljevic wrote:

Toggle quote (7 lines)

> On Tue, 19 May 2020 14:25:23 -0400

> Leo Famulari <leo@famulari.name> wrote:

> > guix gc --references

> Doesn't contain go.

Then your patch works!

Danny Milosavljevic wrote on 19 May 2020 21:05

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:20200519210537.34c79ef7@scratchpost.org

Hi Leo,

Toggle quote (2 lines)

> Then your patch works!

ok!

Pushed to guix master as commit f523eaba7b5460937a230a824550243d75e5f1dc.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl7ELgEACgkQ5xo1VCww

uqVigwgApxN7kse19HidrG3r8NOeS+wAF7+PUGpe7dBqlh3xTxSY9XyBWam+qdBB

bnS0x7sJZlQdD5D50H+iusEERIkgwefBbEa5V5hyry6mOwVrBYQq/Yaw+Ym4Z1vQ

ya54F5yvMQVMZ4mpex9mhnFgOj2CFU5sYoPvFo3iWvx/vYWu8yDmImpFQrKXFGEi

EdVsn1rQjbVVHI4eqIguGKX+DeSDsBPPkKZqK4jLnTX+ojyul82HyjdFGOum0pZu

Uk6gZsMzlQFvhkQ/Zc9BY8CdrJrxJRPvuquU63GXkQcDUVbAbfo8vDRZhRZBEyre

EeW9a3t3ED8jCrRIPEniCzptZddKSA==

=UcwY

-----END PGP SIGNATURE-----

Closed

Pierre Neidhardt wrote on 20 May 2020 08:16

Recipients:(address . 41406@debbugs.gnu.org)

Message-ID:875zcryvun.fsf@ambrevar.xyz

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (7 lines)

> On Tue, 19 May 2020 14:25:23 -0400

> Leo Famulari <leo@famulari.name> wrote:

>> guix gc --references

> Doesn't contain go.

So why did `guix size` contain go?

Pierre Neidhardt

https://ambrevar.xyz/

Leo Famulari wrote on 20 May 2020 18:43

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)

Message-ID:20200520164316.GA21201@jasmine.lan

On Wed, May 20, 2020 at 08:16:32AM +0200, Pierre Neidhardt wrote:

Toggle quote (11 lines)> Danny Milosavljevic <dannym@scratchpost.org> writes:
> 
> > On Tue, 19 May 2020 14:25:23 -0400
> > Leo Famulari <leo@famulari.name> wrote:
> >
> >> guix gc --references
> >
> > Doesn't contain go.
> 
> So why did `guix size` contain go?

`guix size` also accounts for build-time dependencies, and Docker is

written in Go.

Marius Bakke wrote on 20 May 2020 19:05

Recipients:(address . 41406@debbugs.gnu.org)

Message-ID:87blmibkp6.fsf@devup.no

Leo Famulari <leo@famulari.name> writes:

Toggle quote (15 lines)> On Wed, May 20, 2020 at 08:16:32AM +0200, Pierre Neidhardt wrote:
>> Danny Milosavljevic <dannym@scratchpost.org> writes:
>> 
>> > On Tue, 19 May 2020 14:25:23 -0400
>> > Leo Famulari <leo@famulari.name> wrote:
>> >
>> >> guix gc --references
>> >
>> > Doesn't contain go.
>> 
>> So why did `guix size` contain go?
>
> `guix size` also accounts for build-time dependencies, and Docker is
> written in Go.

`guix size` only looks at the closure size. One of Dockers dependencies

must be carrying the spurious Go reference (guix graph might help here).

Julien Lepiller wrote on 20 May 2020 20:30

Recipients:(address . 41406@debbugs.gnu.org)

Message-ID:C7368091-08BD-4196-BFC5-52D00D894436@lepiller.eu

Le 20 mai 2020 12:43:16 GMT-04:00, Leo Famulari <leo@famulari.name> a écrit :

Toggle quote (15 lines)>On Wed, May 20, 2020 at 08:16:32AM +0200, Pierre Neidhardt wrote:
>> Danny Milosavljevic <dannym@scratchpost.org> writes:
>> 
>> > On Tue, 19 May 2020 14:25:23 -0400
>> > Leo Famulari <leo@famulari.name> wrote:
>> >
>> >> guix gc --references
>> >
>> > Doesn't contain go.
>> 
>> So why did `guix size` contain go?
>
>`guix size` also accounts for build-time dependencies, and Docker is
>written in Go.

As I said:

guix graph -t references --path /gnu/store/…-docker /gnu/store/…-go

Should give you the shortest path from docker to go, looking only at references.

Pierre Neidhardt wrote on 20 May 2020 23:09

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87367ufh4e.fsf@ambrevar.xyz

Leo Famulari <leo@famulari.name> writes:

Toggle quote (3 lines)

> `guix size` also accounts for build-time dependencies, and Docker is

> written in Go.

Are you sure about this?  Marius seems to disagree unless I'm
misunderstanding the two statements.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

Leo Famulari wrote on 21 May 2020 01:01

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)

Message-ID:20200520230124.GB14567@jasmine.lan

On Wed, May 20, 2020 at 11:09:37PM +0200, Pierre Neidhardt wrote:

Toggle quote (8 lines)

> Leo Famulari <leo@famulari.name> writes:

> > `guix size` also accounts for build-time dependencies, and Docker is

> > written in Go.

> Are you sure about this? Marius seems to disagree unless I'm

> misunderstanding the two statements.

I thought I was correct but it's certainly possible I'm wrong! The

manual isn't clear IMO and I don't have time to read the code.

The `guix graph` suggestions should help get to the bottom of it,

though.

Pierre Neidhardt wrote on 21 May 2020 09:02

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87r1vdepo3.fsf@ambrevar.xyz

Leo Famulari <leo@famulari.name> writes:

Toggle quote (3 lines)

> I thought I was correct but it's certainly possible I'm wrong! The

> manual isn't clear IMO and I don't have time to read the code.

I remember having a similar confusion in the past.  We should make it
more explicit what `guix size` does compared to `guix gc  --references'
for instance.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

zimoun wrote on 21 May 2020 11:41

Recipients:(name . Julien Lepiller)(address . julien@lepiller.eu)

Message-ID:CAJ3okZ2Q3S461DXrfPxAVRgh52GJMrt7roXU8fg=WoWQjc62qQ@mail.gmail.com

On Wed, 20 May 2020 at 20:31, Julien Lepiller <julien@lepiller.eu> wrote:

Toggle quote (4 lines)

> guix graph -t references --path /gnu/store/…-docker /gnu/store/…-go

> Should give you the shortest path from docker to go, looking only at references.

It seems that the culprit is the package 'runc'.

Toggle snippet (9 lines)

guix graph -t references \

/gnu/store/scmx1fphz8kphx37rbckiqvkkh125y6d-docker-19.03.7 \

--path \

/gnu/store/w8gjhcw6a16rk1dvxa97bz2znal5fihm-go-1.13.9

/gnu/store/scmx1fphz8kphx37rbckiqvkkh125y6d-docker-19.03.7

/gnu/store/vycnhl10jc3jv4yc7dy1p6maykipidkp-runc-1.0.0-rc6

/gnu/store/w8gjhcw6a16rk1dvxa97bz2znal5fihm-go-1.13.9

Leo Famulari wrote on 21 May 2020 18:20

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:20200521162009.GA25215@jasmine.lan

On Thu, May 21, 2020 at 11:41:54AM +0200, zimoun wrote:

Toggle quote (19 lines)> On Wed, 20 May 2020 at 20:31, Julien Lepiller <julien@lepiller.eu> wrote:
> 
> > guix graph -t references --path /gnu/store/…-docker /gnu/store/…-go
> >
> > Should give you the shortest path from docker to go, looking only at references.
> 
> It seems that the culprit is the package 'runc'.
> 
> --8<---------------cut here---------------start------------->8---
> guix graph -t references \
>      /gnu/store/scmx1fphz8kphx37rbckiqvkkh125y6d-docker-19.03.7 \
>      --path \
>      /gnu/store/w8gjhcw6a16rk1dvxa97bz2znal5fihm-go-1.13.9
> 
> /gnu/store/scmx1fphz8kphx37rbckiqvkkh125y6d-docker-19.03.7
> /gnu/store/vycnhl10jc3jv4yc7dy1p6maykipidkp-runc-1.0.0-rc6
> /gnu/store/w8gjhcw6a16rk1dvxa97bz2znal5fihm-go-1.13.9
> --8<---------------cut here---------------end--------------->8---

We should make the linter complain about this reference.

Danny Milosavljevic wrote on 26 May 2020 16:17

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:20200526161701.6546551c@scratchpost.org

Hi Leo,

On Thu, 21 May 2020 12:20:09 -0400

Leo Famulari <leo@famulari.name> wrote:

Toggle quote (2 lines)

> We should make the linter complain about this [go] reference.

What do you mean exactly?  How would it know, given that that was a runtime
dependency?  Would guix lint build the derivation and check for references
to go etc?  Sounds doable.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl7NJN0ACgkQ5xo1VCww

uqWDmgf/fLjaD6LKxGMRg5KuxxLu/WEFk9sj3sHY5HIEl5OFJDI9KZmpE2vkJ2f7

ohjuAXq1nH0dIlz6nBALHsWLzVsALVJuxfWFDncseXqgXMNbNwYdwLeG7K61cdD+

Wsd61kWXxLnuh6sHqGGOmvLkvWnPELoz9DdHm1+iUAOG7GNz95wBYZo7FudEIDxx

qa09ds6zdcFesHn98mmasp41GL1qsznrg1e3/Coyo/e+iVweIzu4IRnDWT0SN75p

njXJxySA5eHni8ucixQXcDISU+Rw/czSwMrFlpL/yqu9Rv/SaRyfsKtiFFdBkwWv

z4U5onJGpLaReLN3eVeJoZB67Jz3kg==

=TeJ+

-----END PGP SIGNATURE-----

Leo Famulari wrote on 26 May 2020 18:24

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20200526162422.GA21828@jasmine.lan

On Tue, May 26, 2020 at 04:17:01PM +0200, Danny Milosavljevic wrote:

Toggle quote (11 lines)> Hi Leo,
> 
> On Thu, 21 May 2020 12:20:09 -0400
> Leo Famulari <leo@famulari.name> wrote:
> 
> > We should make the linter complain about this [go] reference.
> 
> What do you mean exactly?  How would it know, given that that was a runtime
> dependency?  Would guix lint build the derivation and check for references
> to go etc?  Sounds doable.

I had confused this sitation with the linter's warnings about "inputs

that should be native".

It's probably not worth adding a linter check for this, because everyone

should be using the go-build-system for their Go packages, and if it

doesn't work, that's a bug in the build system, not in the package

definition :)

I don't think the linter should build things and check for spurious

references. At least, not just for this case.

Danny Milosavljevic wrote on 27 May 2020 13:53

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:20200527135329.3ea8e7e7@scratchpost.org

Hi Leo,

On Tue, 26 May 2020 12:24:22 -0400

Leo Famulari <leo@famulari.name> wrote:

Toggle quote (4 lines)

> It's probably not worth adding a linter check for this, because everyone

> should be using the go-build-system for their Go packages, and if it

> doesn't work, that's a bug in the build system, not in the package

Hmm, but runc--the culprit here--does use go-build-system and still retains

a go reference erroneously.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl7OVLkACgkQ5xo1VCww

uqWS/wf/SAlNH96xHsyrRv8orRfltbF+J2UlblFw4LEbhULOR9Nn68SJeXy1XTm/

sEQv/zF9F2Ur5dx7MQHc1zqwT4ALIym1ezq/85kSnzi6RDSyY+v4Edtrjy1V2dM6

2UBQBdFOhK8eQ6eQEulqT8nfVXlqGgsoypR5mUaYkHUmiJUFlQC2uOEDiSJiV2yx

3Jx0SLxPhNCxL0He7rEc+uLiSu/OuWqP4Hy41YCeWGWRtPJWtBbKUWIf6r+9WbPM

NXjegua/2D4CZYysOzOuh1nThOs7XOPBfqqL1LowTIhfH9KUhd93Xl/86Wdl1KAj

wXHZW/ZUlhjPaeqXQJayK7ECFTqD8g==

=4vyP

-----END PGP SIGNATURE-----

Leo Famulari wrote on 27 May 2020 19:22

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20200527172232.GD28889@jasmine.lan

On Wed, May 27, 2020 at 01:53:29PM +0200, Danny Milosavljevic wrote:

Toggle quote (12 lines)> Hi Leo,
> 
> On Tue, 26 May 2020 12:24:22 -0400
> Leo Famulari <leo@famulari.name> wrote:
> 
> > It's probably not worth adding a linter check for this, because everyone
> > should be using the go-build-system for their Go packages, and if it
> > doesn't work, that's a bug in the build system, not in the package
> 
> Hmm, but runc--the culprit here--does use go-build-system and still retains
> a go reference erroneously.

Yup, it's a bug!

Your comment

This issue is archived.

To comment on this conversation send an email to 41406@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date