Towards a new 'guix pull'

DoneSubmitted by Ludovic Courtès.
Details
12 participants
  • Alex Sassmannshausen
  • Chris Marusich
  • Thompson, David
  • Jan Nieuwenhuizen
  • Konrad Hinsen
  • Leo Famulari
  • Ludovic Courtès
  • Mark H Weaver
  • Nils Gillmann
  • Pjotr Prins
  • Ricardo Wurmus
  • Fis Trivial
Owner
unassigned
Severity
important
Merged with
L
L
Ludovic Courtès wrote on 11 Feb 2016 11:35
(address . bug-guix@gnu.org)
87vb5vsffd.fsf@gnu.org
Hello!
Here’s a series of improvements that I think we should make in ‘guixpull’:
• Use Git instead of downloading a whole snapshot every time. The Git checkout would be kept in ~/.cache/guix/pull/checkouts, say.
A related question is whether to use Git itself, which is pretty big per ‘guix size’, or to use some libgit2 bindings such as https://git.dthompson.us/guile-git.git (the closure of libgit2 is 435┬áMiB; that of Git is 761┬áMiB.)
• Build & install not only Scheme code, but also locales and the Info manual.
• Have a “channel” mechanism, similar to ‘nix-channel’, that would allow users to have several Guix variants available in parallel instead of just “latest”. Could work like this:
guix channel add latest git://git.sv.gnu.org/guix.git master guix channel add stable git://git.sv.gnu.org/guix.git stable guix channel pull latest guix channel set latest # here i see the latest versions of everything guix channel set stable # and here everything is old but super stable ;-)
All 3 items can be done separately, I think.
Any takers? :-)
Ludo’.
PS: I do not mention the issue of authenticating code here, which is obviously very important and deserves to be treated separately. Related to that is the question of making sure that what you think is the latest version really is the latest version. We need someone to sign certificates saying what the latest commit ID of a repo is. See the “The Update Framework” paper!
L
L
Ludovic Courtès wrote on 12 Feb 2016 15:07
control message for bug #22629
(address . control@debbugs.gnu.org)
87d1s210pb.fsf@gnu.org
severity 22629 wishlist
L
L
Ludovic Courtès wrote on 28 Nov 2016 15:13
‘guix pull’ and external dependencies
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87vav7ae17.fsf_-_@gnu.org
Hi!
Efraim Flashner <efraim@flashner.co.il> skribis:
Toggle quote (7 lines)> If I understand it correctly, as part of `guix pull' we get the latest> package definitions, but `guix' and `guix-daemon' are at the> guix-snapshot version, aka 0.11.0-4. If instead `guix-daemon' was from> the tip of master then it'd be at the equivalant of running> './pre-inst-env guix-daemon --build-users...', which would have all> these changes.
What ‘guix pull’ does is fetch the latest code, build *the Schemesubset* of that code, and install it in ~/.config/guix/latest.
Thus, it gives you the latest package recipes as well as the latest‘guix’ sub-commands.
What it does not give you is:
1. The latest C++ code (guix-daemon, guix-register). 2. The latest locales. 3. The latest elisp code. 4. The latest dependencies (the Guile that appears in the shebang of the ‘guix’ command, zlib, Guile-SSH, Guile-JSON, etc.)
It worked OK when Guix was self-contained, and assuming people wouldupdate guix-daemon through other ways (‘guix system reconfigure’ onGuixSD). But now we see that these shortcomings are starting to bite.
So I think what we need to do is for “guix pull-ng” to build and installa complete ‘guix’ package, and to manage it pretty much like otherpackages is managed, except not in the user’s main profile (because thatcould lead to undesirable behavior, where upgrading Guix creates a newgeneration, or, in theory, unrecoverable problems, where you cannotroll back because previous generations use an old Guix that does notunderstand the new manifest format.)
The difficulty is that ./configure && make && make install in Guix takessome time, and we probably wouldn’t want to do that on each ‘guix pull’invocation (esp. with Guile 2.2’s compilation times.)
So we may have to provide substitutes of Guix itself, and arrange sothat ‘guix pull’ pulls up to a tag for which we have substitutes.
Ideas welcome! See https://bugs.gnu.org/22629.
Ludo’.
C
C
Chris Marusich wrote on 29 Nov 2016 02:58
(name . Ludovic Courtès)(address . ludo@gnu.org)
87wpfn2gk1.fsf@gmail.com
Hi Ludo`,
ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (4 lines)> So I think what we need to do is for “guix pull-ng” to build and install> a complete ‘guix’ package, and to manage it pretty much like other> packages is managed,
I think that's very reasonable. It seems more intuitive than thecurrent way 'guix pull' works. I suspect that managing the installedversion of Guix via the same Guix mechanisms that we use to manage anyother package might be the best, most intuitive solution.
Would it simplify the problem if we packaged the "Guix client stuff",the "Guix daemon stuff", and maybe the "Guix package definition stuff"separately? Then a user could just install the "Guix client stuff"package if she wanted to upgrade the Guix client tools, or the "Guixpackage definition stuff" package if she wanted to get the latestpackage definitions.
Toggle quote (3 lines)> except not in the user’s main profile (because that could lead to> undesirable behavior,
If we don't store Guix in the user's main profile, where would it go? Asystem profile (like in GuixSD)? What if another user wants to run adifferent version of Guix? It might be nice to let them do that.
It's not clear to me why it's riskier to store Guix in a profile ratherthan outside the profile (but still in the store via the$HOME/.config/guix/latest symlink), which is what 'guix pull' does now.You seem to think it's riskier; I'm curious to know more about why.
Toggle quote (2 lines)> where upgrading Guix creates a new generation,
Why should upgrading Guix NOT create a new generation? I thought that anew profile generation would be created any time you upgrade a package,and I thought that was a good thing because it facilitates easy,transactional roll-back. Perhaps I'm missing something.
Toggle quote (4 lines)> or, in theory, unrecoverable problems, where you cannot roll back> because previous generations use an old Guix that does not understand> the new manifest format.)
Why would a change in manifest format be unrecoverable? It looks likeeach profile generation contains a manifest file. Assuming that the newGuix functions well enough to perform roll back, couldn't we just rollback to the previous profile generation, where we would have both (1)the old profile's manifest file, and (2) the previous Guix, whichunderstands that format? Since rolling back a profile is basically justa symlink flip, I think the new Guix could probably do that even if itdidn't understand the old manifest format.
Toggle quote (7 lines)> The difficulty is that ./configure && make && make install in Guix takes> some time, and we probably wouldn’t want to do that on each ‘guix pull’> invocation (esp. with Guile 2.2’s compilation times.)>> So we may have to provide substitutes of Guix itself, and arrange so> that ‘guix pull’ pulls up to a tag for which we have substitutes.
What if we had a special package version of Guix (e.g., "v0.11.0") whichwe kept up to date via some mechanism? Maybe something as simple as aGit hook could help increase the likelihood of that version beingsubstitutable. For example, we could have a Git hook that preventssomeone from checking in a change if the latest Git tag does notcorrespond to a Guix package version. Maybe we can do better.
I actually think it would be a good thing if we can run "guix pull"without substitutes available. But it should use a substitute bydefault, and "build from source" should be a fallback mechanism that theuser has to explicitly request, just like when installing new packages.That would help avoid unexpectedly long "guix pull" invocations.
-- Chris
-----BEGIN PGP SIGNATURE-----Version: GnuPG v2
iQIcBAEBCAAGBQJYPOCuAAoJEN1AmhXYIkadaAYP/0od3zbxya4o9kkELfpkrZbhNpj/0v46dw/h/HbCTLpu2S0ktYEG5o+eu5Ngua+bo+hKrGu7r8jcejApiSkf9oZtayIppN4zNrJa8yp4nW2U7D6b0wWpl/vPecVJtNtU355v/ng0AGeau2dt8yRyIWoHp8JYxWiK86//cc2neNifFLD4QO+Eh7jPixjGOlZDRTyoDXTg8Va/4bHRkgxguWLdc+IhTySHCp8AfVkc4L7eNsNYWFYk9fC1wO4XSJiat5RhELAD5M1IkKpbBp5FLGnTpbvPaP4XBKOVzLWPHcQhKJyxOGKC5T0nJaHSq+FwowIAu2+IHuzTflw10fodGpVJbkhjdILNX5KOPCa4C9nJ6PivEZtgon0iQvV8RHyNyBNqEB/m9fVHoik0tS/Xi/hvKPtOCrQOrdqlbFmzqXaeoUjb+rQIAPAVa442LgR7bGm+VQkCkkfU6wczmzrgKbWhpBFOsAvI3yjaDG4gT+8HVzcit3mRwQQ4yQsoCBGpCgfJCJtW1sX/ZrrdatCQx6bPCND2uHLBBfrK6U0s014N3CqfIBv/jtrp/4fL08sIeJodrh7oFAsPwMM6+sdO4w9B8J190wbUYzeEEsks+7D9tE1FxzHpQV+MYjqLFNifJ4wQdpTa5Llo1iLIOV8hjBK27wWIyClPOZA3c766Zkmq=09gL-----END PGP SIGNATURE-----
L
L
Ludovic Courtès wrote on 29 Nov 2016 15:54
(name . Chris Marusich)(address . cmmarusich@gmail.com)
87wpfm49ql.fsf@gnu.org
Hello,
Chris Marusich <cmmarusich@gmail.com> skribis:
Toggle quote (18 lines)> ludo@gnu.org (Ludovic Courtès) writes:>>> So I think what we need to do is for “guix pull-ng” to build and install>> a complete ‘guix’ package, and to manage it pretty much like other>> packages is managed, >> I think that's very reasonable. It seems more intuitive than the> current way 'guix pull' works. I suspect that managing the installed> version of Guix via the same Guix mechanisms that we use to manage any> other package might be the best, most intuitive solution.>> Would it simplify the problem if we packaged the "Guix client stuff",> the "Guix daemon stuff", and maybe the "Guix package definition stuff"> separately? Then a user could just install the "Guix client stuff"> package if she wanted to upgrade the Guix client tools, or the "Guix> package definition stuff" package if she wanted to get the latest> package definitions.
It would be bad to separate package definitions from the rest becausethey are very much intertwined: package definitions depend on thedefinition of ‘package’, on build system implementations, and so on.
We could have guix-sans-daemon though, if that helps (which I suspect isnot the case).
Toggle quote (5 lines)>> except not in the user’s main profile (because that could lead to>> undesirable behavior, >> If we don't store Guix in the user's main profile, where would it go?
In “some sort of a profile” in ~/.config/guix/latest or similar.
Toggle quote (5 lines)> It's not clear to me why it's riskier to store Guix in a profile rather> than outside the profile (but still in the store via the> $HOME/.config/guix/latest symlink), which is what 'guix pull' does now.> You seem to think it's riskier; I'm curious to know more about why.
There’s the manifest format change issue I mentioned, or the inabilityto roll back if you install a broken Guix.
Toggle quote (7 lines)>> where upgrading Guix creates a new generation, >> Why should upgrading Guix NOT create a new generation? I thought that a> new profile generation would be created any time you upgrade a package,> and I thought that was a good thing because it facilitates easy,> transactional roll-back. Perhaps I'm missing something.
I’m suggesting that upgrading Guix creates a new generation (so we agreehere), just not in the user’s profile.
Toggle quote (13 lines)>> or, in theory, unrecoverable problems, where you cannot roll back>> because previous generations use an old Guix that does not understand>> the new manifest format.)>> Why would a change in manifest format be unrecoverable? It looks like> each profile generation contains a manifest file. Assuming that the new> Guix functions well enough to perform roll back, couldn't we just roll> back to the previous profile generation, where we would have both (1)> the old profile's manifest file, and (2) the previous Guix, which> understands that format? Since rolling back a profile is basically just> a symlink flip, I think the new Guix could probably do that even if it> didn't understand the old manifest format.
Yeah, I think you’re right. :-)
In general, I think my concern is more that we cannot promise thatdowngrading Guix will work, considering the potential for on-disk formatchanges. It’s a bit theoretical, but not entirely sci-fi either.
Toggle quote (14 lines)>> The difficulty is that ./configure && make && make install in Guix takes>> some time, and we probably wouldn’t want to do that on each ‘guix pull’>> invocation (esp. with Guile 2.2’s compilation times.)>>>> So we may have to provide substitutes of Guix itself, and arrange so>> that ‘guix pull’ pulls up to a tag for which we have substitutes.>> What if we had a special package version of Guix (e.g., "v0.11.0") which> we kept up to date via some mechanism? Maybe something as simple as a> Git hook could help increase the likelihood of that version being> substitutable. For example, we could have a Git hook that prevents> someone from checking in a change if the latest Git tag does not> correspond to a Guix package version. Maybe we can do better.
Right, we could do something like that. There are still non-zerochances that someone running ‘guix pull’ at an arbitrary point in timewill have to build locally, which is not great.
Toggle quote (6 lines)> I actually think it would be a good thing if we can run "guix pull"> without substitutes available. But it should use a substitute by> default, and "build from source" should be a fallback mechanism that the> user has to explicitly request, just like when installing new packages.> That would help avoid unexpectedly long "guix pull" invocations.
Yes, using substitutes or falling back to source builds is always thedefault.
Thanks for your feedback!
Ludo’.
P
P
Pjotr Prins wrote on 22 Feb 2017 08:57
Re: bug#22629: Towards a new 'guix pull'
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
20170222075706.GA21158@mail.thebird.nl
I wrote up a 'white paper' on a new implementation of Guix channels.The easy on the eyes version you can read here:
https://github.com/pjotrp/guix-notes/blob/master/proposals/channel.org
* On Channels
Related to debbugs https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629and https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22883#103
** Updating package definitions through channels
*** Introduction
The main purpose of creating Guix channels is to allow users to usedifferent versions of the Guix package tree that differ from themaster branch, i.e., have people share packages with each other thatare not in the GNU Guix distribution itself.
Within Guix there are good reasons to handle such cases. For example,to support an older package, such as, ruby 1.8.7 (currently in themaster tree, but could be removed), and to have different editions ofcombinations of packages. This versioning we currently achieve byexplicitely working from git checkouts of trees - not trivial foreveryone. Channels will just make versioning easier and accessible foreveryone.
Outside Guix, channels can support also different combinations ofpackages, e.g., such as for easily deploying different versions of theGeneNetwork webservice (development/staging/testing/productionbranches). Channels will also make it possible to switch testing offin certain packages, for example, or to distribute data packages thatdo not belong or are not wanted in GNU Guix.
In short, channels allow for easy specialization of deployments.
Also, channels and 'guix publish' distributed services can go hand inhand in providing such specializations.
*** Usage
Proposed syntax for adding a channel
: guix channel add channel-name URI branch
Examples
: guix channel add guix-stable git://git.sv.gnu.org/guix.git stable: guix channel add guix-master git://git.sv.gnu.org/guix.git master: guix channel add ruby-1.8.7 git:URI/my-ruby-repo.git ruby-1.8.7: guix channel add genenetwork-development git:URI/genenetwork-staging.git development: guix channel add genenetwork-staging git:URI/genenetwork-staging.git staging
To keep things simple I propose a channel simply reflects a fixedversion of a git tree of a Guix package list. I.e., the exact *state*of the combined list of packages is what we support in a channel. Thegit checkout can be stored in something like
: ~/.guix-channels/ruby-1.8.7/...
There will only be one directory for one channel. Channels are notshared between users so they can go somewhere in $HOME.
To get to a certain state, or update a channel, we should allow fortree updating based on a git SHA value, a tag or just the HEAD. Manychannels will go by HEAD, but it is unwanted because a tree may be inan unstable and even unusable state. Especially for Guix itself weshould not be using HEAD - the current default with a 'guix pull'.Ideally 'guix pull' should be tried by core developers before givingit to everyone. I think we can figure out the last tagged release andhave that as a default rather than HEAD. When someone wants HEAD itshould be explicit. So to get the actual HEAD the syntax can be:
: guix channel update guix-master origin/HEAD
To get to a specific SHA value on origin
: guix channel update guix-master 9cf265449f88f89c900e510fe29baba26f30159b
To get a named tagged release
: guix channel update guix-master v0.12.0-8
To get the latest tagged release (default behaviour)
: guix channel update guix-master
To switch channels for use
: guix channel set guix-master
When a channel gets updated I think we ought to check it out fullyfrom scratch by default. Reason being that we can't deal withconflicts and maybe changed upstream trees - git also allows rewritinghistory and that can be nasty. If some person feels confident, wecould still add a switch
: guix channel update guix-master --pull
or
: guix channel update guix-master --checkout v0.12.0-8
which would do what you expect it to do.
Obviously all this behaviour reflects the way git behaves. I think itis fine because if we start adding other transports (say https,mercurial or darcs) it would be clear from the channel URI what it wasand we can change the commands.
** Channel metadata
The channel should describe its purpose and set certain prerequisites,such as a description and the version of guile it requires. This canbe added as a channel-meta.scm file in the root of the source tree ingit. It could also point to one or more 'guix publish' servers.
** Compiling the package tree
So far, we have only discussed the package tree. To compile it we needguile. One thing the current 'guix pull' does is compile guile tocompile the tree. This behaviour does not make use of substituteservers, takes time and is somewhat error prone. I think what weshould do is reuse a guile already on the system or install it as abinary using a substitute server and then compile the packages.
** Updating guix and guix-daemon
Now we come to updating guix and guix-daemon itself. I think that thisfunctionality does not belong in channel. Not least because not allusers have the privileges to restart the daemon.
To update guix essentially one will add a relevant channel, update thechannel to point and install guix from that package tree using 'guixpackage'.
It can be that the currently running guix tools, for some reason, cannot support a package tree. To notify that, I suggest we use that samemeta data in the channel tree. The metadata can tell what version ofguix and guix-daemon should be used (say >=0.11.0) and advise the userand system administrator.
** Guix pull
Where channels take the role of giving state to the package tree, anew 'guix pull' could have the task of *upgrading* guix and the guixdaemon to latest. As stated above the package tree should not move to^HEAD, but instead to the latest tagged release (which can happen moreoften then we are doing today - say on a weekly basis). Starting froman approved Guix version 'guix pull' could build/download the guixpackage itself and suggest updating the daemon to latest.
** Security and trusted channels
Guix has a security mechanism for its substitute servers, i.e., youneed to install a key to trust a 'guix publish' server. For channelsusing git the model is to use git trees which is open to securityconcerns. To introduce the concept of trusted channels we can use thesame 'guix publish' servers which can also provide the binarysubstitutes for a channel. In fact, I think secure channels should bethe default, and we should be able to install packages using such aserver, after switching to a channel, with
: guix channel set genenetwork-staging: guix package -i genenetwork2 --substitute-urls=https://guix.genenetwork.org
which would validate the URL of the channel (a git URI) with thesubstitute server, i.e., guix, by default, will only install channelsthat are known by the publish/substitute server.
Even so, we have to allow for convenience and the fact that noteveryone wants to run a 'guix publish' server. To install an insecurechannel we should allow
: guix channel set genenetwork-staging: guix package -i genenetwork2 --force-untrusted
which bypasses the 'trusted' mechanism of asking permission from thesubstitute server.
To take the trusted model further, the 'guix publish' server couldalso serve SHA values of the channel git trees. I.e., if a user requests
: guix channel update guix-master v0.12.0-8
the server could say that tagged release v0.12.0-8 can be checked outand the SHA values should be such and such. This would give fullsecurity for guix-master checkouts.
L
L
Leo Famulari wrote on 24 Feb 2017 19:21
(name . Pjotr Prins)(address . pjotr.public12@thebird.nl)
20170224182101.GA9615@jasmine
On Wed, Feb 22, 2017 at 07:57:06AM +0000, Pjotr Prins wrote:
Toggle quote (5 lines)> To keep things simple I propose a channel simply reflects a fixed> version of a git tree of a Guix package list. I.e., the exact *state*> of the combined list of packages is what we support in a channel. The> git checkout can be stored in something like
You shared *a lot* of ideas following this paragraph, but I think thatfrom here we can start thinking about how to incrementally evolve fromthe current `guix pull` to something better.
Currently, you can `guix pull` any commit of GNU Guix with `guix pull--url` [0]. This feature depends on the Git server implementation's HTTPinterface, but I bet it's a common feature, and it can already be usedwith other Git repositories besides our Savannah repo. This achieves thegoal of channels, "easy specialization of deployments". Well, "easy" isa matter of degrees ;)
I was thinking we could start by keeping `guix pull` the same from theuser's point of view, but change it to support Git in addition todownloading a tarball. We already have code for this in (guixgit-download) which uses the Git command-line tool. [1]
The next step would be to keep some metadata about which channel thelast pull came from, so that subsequent pulls would point to it.
What do people think?
I haven't wrapped my head around all the proposed changes. I think itwill be best to keep the goal of "easy specialization of deployments" inmind and work towards it incrementally.
[0]https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8a9cffb202414b20081910115ba76402924bdcdd
[1] In the long run, we should ditch the Git CLI and use something likeguile-git:https://gitlab.com/amirouche/guile-git
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAliweYkACgkQJkb6MLrKfwjrOBAA0BdkuDnJ0aD/9bHmV2F/w2jcLyjFQvJfne2c9D10SOndcvGMUpbyPFCyTVYmlbTjwwS+U48721VtgQXbpWdKdddwk6g7akvZMitjcZ/SANaPOKOCg94Ez3f3W8ap8dJRXMgWaF6sYNtD5dk0xA4B1PUEB3fySuO/grEtGCcfnP1L5e/SaRss2BGVsqgxwvMSvY3lwWlVXqGwx/3zS2scn8J/5GubcV0Yu2/SaUcEKuVgd92y1W/iJj9UMFRqjA0bHp+2pVLOD0QNFbTV+B9ChAa+Xo29AkRo43eCcL4x61S4R9n09c4jZPQvcx2PizHxld5MKFADYAhba9x1PThkdhVPk687gHiLfENSd5eUdU5P/5eJH4mxMrL0g3VOK2YOb85VdsPJmdtLO3hvXG2DqBKs8Gob7HzV0fekKnWgvDeiJaAFts1PR6gaGGYdyln/TmqHIqMgdQuulrlCrmYk0uvzz5UW8WxQ0W7j6EfjlppnP0kJtNGYynI0enPB2OcMohuYnHEXmB9p4QxtgoP3CL66VvPDTFyAgtekUBqA1dcicQbbJ86sIwTi0FiMEch+DVGH5z98H9CBT8DzFa4b/FSomSZ1jGA8uNp5zHuGHe+R8JQch1j2M+nS5UvYVLnKWGozxRxdvu7dYYx7L6uPuhnF0mL6xkd6fabMYj9kjRc==TqJj-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 20 Sep 2017 23:23
control message for bug #28471
(address . control@debbugs.gnu.org)
87tvzxqd2l.fsf@gnu.org
merge 28471 22629
L
L
Ludovic Courtès wrote on 20 Sep 2017 23:25
control message for bug #22629
(address . control@debbugs.gnu.org)
87shfhqczb.fsf@gnu.org
severity 22629 important
L
L
Ludovic Courtès wrote on 8 Apr 2018 18:48
Re: bug#22629: Towards a new 'guix pull'
(address . 22629@debbugs.gnu.org)(name . Ricardo Wurmus)(address . rekado@elephly.net)
87fu45ve2z.fsf@gnu.org
Hello Guix!
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (6 lines)> Here’s a series of improvements that I think we should make in ‘guix> pull’:>> • Use Git instead of downloading a whole snapshot every time. The Git> checkout would be kept in ~/.cache/guix/pull/checkouts, say.
That was done a while back. :-)
Toggle quote (3 lines)> • Build & install not only Scheme code, but also locales and the Info> manual.
… and also the C++ bits.
With the work done in https://bugs.gnu.org/27284, we can build all ofGuix and its dependencies when doing ‘guix pull’, which brings us closerto what the ‘guix’ package provides.
Thus we can at least partly address the issue described inhttps://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629#10, i.e., have‘guix pull’ install not just a bunch of .go files but also guile-git,guile-ssh, gnutls, and Guile itself.
To do that, we need a “paradigm shift” (sounds cute, no? :-)).Currently, ‘scripts/guix’ fiddles with its %load-path and adds~/.config/guix/latest if it exists. The new ‘guix pull’ should providenot just .go files but also a ‘guix’ command referring to the latestGuile, Guile-Git, etc.
To achieve this, I think the new approach would be to have:
export PATH=$HOME/.config/guix/current/bin:$PATH export INFOPATH=$HOME/.config/guix/current/share/info:$INFOPATH
‘guix pull’ would update ~/.config/guix/current, which would contain the‘guix’ command, all the .go files, locales, the Info manual, and ideally‘guix-daemon’ as well.
~/.config/guix/current could be a profile, or something that resembles aprofile.
For a while ‘guix pull’ will have to keep updating ~/.config/guix/latestas well, for users running an old ‘guix’ command.
Thoughts?
Ludo’.
N
N
Nils Gillmann wrote on 8 Apr 2018 19:45
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
20180408174533.a5xlhofmkwwpklbj@abyayala
Ludovic Courtès transcribed 2.0K bytes:
Toggle quote (50 lines)> Hello Guix!> > ludo@gnu.org (Ludovic Courtès) skribis:> > > Here’s a series of improvements that I think we should make in ‘guix> > pull’:> >> > • Use Git instead of downloading a whole snapshot every time. The Git> > checkout would be kept in ~/.cache/guix/pull/checkouts, say.> > That was done a while back. :-)> > > • Build & install not only Scheme code, but also locales and the Info> > manual.> > … and also the C++ bits.> > With the work done in <https://bugs.gnu.org/27284>, we can build all of> Guix and its dependencies when doing ‘guix pull’, which brings us closer> to what the ‘guix’ package provides.> > Thus we can at least partly address the issue described in> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629#10>, i.e., have> ‘guix pull’ install not just a bunch of .go files but also guile-git,> guile-ssh, gnutls, and Guile itself.> > To do that, we need a “paradigm shift” (sounds cute, no? :-)).> Currently, ‘scripts/guix’ fiddles with its %load-path and adds> ~/.config/guix/latest if it exists. The new ‘guix pull’ should provide> not just .go files but also a ‘guix’ command referring to the latest> Guile, Guile-Git, etc.> > To achieve this, I think the new approach would be to have:> > export PATH=$HOME/.config/guix/current/bin:$PATH> export INFOPATH=$HOME/.config/guix/current/share/info:$INFOPATH> …> > ‘guix pull’ would update ~/.config/guix/current, which would contain the> ‘guix’ command, all the .go files, locales, the Info manual, and ideally> ‘guix-daemon’ as well.> > ~/.config/guix/current could be a profile, or something that resembles a> profile.> > For a while ‘guix pull’ will have to keep updating ~/.config/guix/latest> as well, for users running an old ‘guix’ command.> > Thoughts?
First thought: Woo! Good progress!On your propossed steps, sounds good to me.
Toggle quote (4 lines)> Ludo’.> > >
L
L
Ludovic Courtès wrote on 31 May 2018 16:43
[PATCH 0/4] 'guix pull' produces a self-contained Guix
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180531144337.16298-1-ludo@gnu.org
Hello Guix!
Here is the “new” ‘guix pull’ that we discussed notably in this thread:
https://bugs.gnu.org/22629
The major difference is that instead of just building a bunch of modulesand putting them under ~/.config/guix/latest, it now produces astandalone package (with bin/guix, share/info/guix.info, etc.) and putsit in a profile under ~/.config/guix/current. Quoth the manual:
The result of running ‘guix pull’ is a “profile” available under ‘~/.config/guix/current’ containing the latest Guix. Thus, make sure to add it to the beginning of your search path so that you use the latest version, and similarly for the Info manual (*note Documentation::):
export PATH="$HOME/.config/guix/current/bin:$PATH" export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"
This ‘~/.config/guix/current’ profile works like any other profile created by ‘guix package’ (*note Invoking guix package::). That is, you can list generations, roll back to the previous generation—i.e., the previous Guix—and so on:
$ guix package -p ~/.config/guix/current -l Generation 1 May 25 2018 10:06:41 guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
Generation 2 May 27 2018 19:07:47 + guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f - guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4
Generation 3 May 30 2018 16:11:39 (current) + guix a076f19 out /gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990 - guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f $ guix package -p ~/.config/guix/current --roll-back switched from generation 3 to 2
There are two requirements it fulfills in terms of compatibility:
1. The modified ‘build-aux/build-self.scm’ still does the right thing when evaluated by an “old” Guix—that is, it produces a bunch of modules for use in ~/.config/guix/latest as before.
2. The modified ‘guix pull’ produces ~/.config/guix/current even when invoked on a commit of a past Guix. That is, it automatically produces a ‘guix’ command using the modules returned by the old ‘build-self.scm’.
There are various improvements we can make from there. For example,using “manifest entry properties” as proposed inhttps://bugs.gnu.org/31442, we can attach meta-data (commit ID, repoURL, etc.) in each manifest entry that ‘guix pull’ populates; then wecan arrange for ‘guix pull --list-generations’ (say) to display thatinformation.
We could add ‘guix pull’ options for convenient: ‘--roll-back’,‘--profile’, etc.
Going forward, additional “channels” could be presented as entries inthe ~/.config/guix/current manifest.
Caveats:
1. The ~/.config/guix/current profile really lives there. That is, unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user. That could be an issue for cluster setups where home directories are not scanned by the Guix GC. Cluster folks, please tell me!
2. The translated Info manual is not built. Julien: could you turn the big ‘xref_command’ in a script or something that we can more easily reuse in (guix self)?
3. C++ code is not built. I wonder which will come first: getting rid of the C++ code, or building it? :-)
Feedback welcome!
Ludo’.
Ludovic Courtès (4): self: Produce a complete package with the 'guix' command. pull: Install the new Guix in a profile. self: Compute and use locale data. self: Build the Info manual.
build-aux/build-self.scm | 19 +- build-aux/compile-as-derivation.scm | 2 +- doc/guix.texi | 38 +++- guix/scripts/pull.scm | 79 +++++--- guix/self.scm | 283 ++++++++++++++++++++++++---- scripts/guix.in | 14 +- 6 files changed, 351 insertions(+), 84 deletions(-)
-- 2.17.0
L
L
Ludovic Courtès wrote on 31 May 2018 16:43
[PATCH 4/4] self: Build the Info manual.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180531144337.16298-5-ludo@gnu.org
* guix/self.scm (info-manual): New procedure.(whole-package): Add #:info and honor it.(compiled-guix): Pass #:info.--- guix/self.scm | 88 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 85 insertions(+), 3 deletions(-)
Toggle diff (128 lines)diff --git a/guix/self.scm b/guix/self.scmindex 92e69ffd5..8ed1e1073 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -230,6 +230,81 @@ DOMAIN, a gettext domain." (computed-file (string-append "guix-locale-" domain) build)) +(define (info-manual source)+ "Return the Info manual built from SOURCE."+ (define texinfo+ (module-ref (resolve-interface '(gnu packages texinfo))+ 'texinfo))++ (define graphviz+ (module-ref (resolve-interface '(gnu packages graphviz))+ 'graphviz))++ (define build+ ;; TODO: Don't depend on all of SOURCE to avoid rebuilds.+ (with-imported-modules '((guix build utils))+ #~(begin+ (use-modules (guix build utils))++ (mkdir #$output)++ ;; Create 'version.texi'.+ ;; XXX: Can we use a more meaningful version string yet one that+ ;; doesn't change at each commit?+ (call-with-output-file "version.texi"+ (lambda (port)+ (let ((version "(Git snapshot)"))+ (format port "+@set UPDATED 1 January 1970+@set UPDATED-MONTH January 1970+@set EDITION ~a+@set VERSION ~a\n" version version))))++ ;; Copy configuration templates that the manual includes.+ (for-each (lambda (template)+ (copy-file template+ (string-append+ "os-config-"+ (basename template ".tmpl")+ ".texi")))+ (find-files #$source "\\.tmpl$"))++ ;; Build graphs.+ (mkdir-p (string-append #$output "/images"))+ (for-each (lambda (dot-file)+ (invoke #+(file-append graphviz "/bin/dot")+ "-Tpng" "-Gratio=.9" "-Gnodesep=.005"+ "-Granksep=.00005" "-Nfontsize=9"+ "-Nheight=.1" "-Nwidth=.1"+ "-o" (string-append #$output "/images/"+ (basename dot-file ".dot")+ ".png")+ dot-file))+ (find-files (string-append #$source "/doc/images")+ "\\.dot$"))++ ;; Copy other PNGs.+ (for-each (lambda (png-file)+ (install-file png-file+ (string-append #$output "/images")))+ (find-files (string-append #$source "/doc/images")+ "\\.png$"))++ ;; Finally build the manual. Copy it the Texinfo files to $PWD and+ ;; add a symlink to the 'images' directory so that 'makeinfo' can+ ;; see those images and produce image references in the Info output.+ (copy-recursively (string-append #$source "/doc") "."+ #:log (%make-void-port "w"))+ (delete-file-recursively "images")+ (symlink (string-append #$output "/images") "images")+ (invoke #+(file-append texinfo "/bin/makeinfo")+ "guix.texi"+ "-I" (string-append #$source "/doc")+ "-I" "."+ "-o" (string-append #$output "/guix.info")))))++ (computed-file "guix-manual" build))+ (define* (guix-command modules #:key source (dependencies '()) (guile-version (effective-version))) "Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its@@ -275,12 +350,13 @@ load path." (define* (whole-package name modules dependencies #:key (guile-version (effective-version))+ info (command (guix-command modules #:dependencies dependencies #:guile-version guile-version))) "Return the whole Guix package NAME that uses MODULES, a derivation of all the modules, and DEPENDENCIES, a list of packages depended on. COMMAND is the-'guix' program to use."+'guix' program to use; INFO is the Info manual." ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'. (computed-file name (with-imported-modules '((guix build utils))@@ -292,9 +368,14 @@ the modules, and DEPENDENCIES, a list of packages depended on. COMMAND is the (let ((modules (string-append #$output "/share/guile/site/"- (effective-version))))+ (effective-version)))+ (info #$info)) (mkdir-p (dirname modules))- (symlink #$modules modules))))))+ (symlink #$modules modules)+ (when info+ (symlink #$info+ (string-append #$output+ "/share/info")))))))) (define* (compiled-guix source #:key (version %guix-version) (pull-version 1)@@ -477,6 +558,7 @@ the modules, and DEPENDENCIES, a list of packages depended on. COMMAND is the #:guile-version guile-version))) (whole-package name built-modules dependencies #:command command+ #:info (info-manual source) #:guile-version guile-version))) ((= 0 pull-version) ;; Legacy 'guix pull': just return the compiled modules.-- 2.17.0
L
L
Ludovic Courtès wrote on 31 May 2018 16:43
[PATCH 2/4] pull: Install the new Guix in a profile.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180531144337.16298-3-ludo@gnu.org
* guix/scripts/pull.scm (%pull-version): New variable.(build-from-source): Pass #:pull-version to BUILD.(whole-package-for-legacy, derivation->manifest-entry): New procedure.(build-and-install): Rewrite in terms of 'build-and-use-profile'.* scripts/guix.in (augment-load-paths!): Remove use of~/.config/guix/latest.* doc/guix.texi (Invoking guix pull): Document it.--- doc/guix.texi | 38 ++++++++++++++++++--- guix/scripts/pull.scm | 79 +++++++++++++++++++++++++++++-------------- scripts/guix.in | 14 +------- 3 files changed, 87 insertions(+), 44 deletions(-)
Toggle diff (199 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex 09749b15e..92e322787 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -2742,11 +2742,39 @@ Any user can update their Guix copy using @command{guix pull}, and the effect is limited to the user who run @command{guix pull}. For instance, when user @code{root} runs @command{guix pull}, this has no effect on the version of Guix that user @code{alice} sees, and vice-versa@footnote{Under the hood, @command{guix pull} updates the-@file{~/.config/guix/latest} symbolic link to point to the latest Guix,-and the @command{guix} command loads code from there. Currently, the-only way to roll back an invocation of @command{guix pull} is to-manually update this symlink to point to the previous Guix.}.+versa.++The result of running @command{guix pull} is a @dfn{profile} available+under @file{~/.config/guix/current} containing the latest Guix. Thus,+make sure to add it to the beginning of your search path so that you use+the latest version, and similarly for the Info manual+(@pxref{Documentation}):++@example+export PATH="$HOME/.config/guix/current/bin:$PATH"+export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"+@end example++This @code{~/.config/guix/current} profile works like any other profile+created by @command{guix package} (@pxref{Invoking guix package}). That+is, you can list generations, roll back to the previous+generation---i.e., the previous Guix---and so on:++@example+$ guix package -p ~/.config/guix/current -l+Generation 1 May 25 2018 10:06:41+ guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4++Generation 2 May 27 2018 19:07:47+ + guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f+ - guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4++Generation 3 May 30 2018 16:11:39 (current)+ + guix a076f19 out /gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990+ - guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f+$ guix package -p ~/.config/guix/current --roll-back+switched from generation 3 to 2+@end example The @command{guix pull} command is usually invoked with no arguments, but it supports the following options:diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scmindex 64c2196e0..c5ceebccb 100644--- a/guix/scripts/pull.scm+++ b/guix/scripts/pull.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2013, 2014, 2015, 2017 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2013, 2014, 2015, 2017, 2018 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com> ;;; ;;; This file is part of GNU Guix.@@ -25,10 +25,15 @@ #:use-module (guix config) #:use-module (guix packages) #:use-module (guix derivations)+ #:use-module (guix profiles) #:use-module (guix gexp) #:use-module (guix grafts) #:use-module (guix monads) #:use-module (guix scripts build)+ #:autoload (guix self) (whole-package)+ #:autoload (gnu packages ssh) (guile-ssh)+ #:autoload (gnu packages tls) (gnutls)+ #:use-module ((guix scripts package) #:select (build-and-use-profile)) #:use-module ((guix build utils) #:select (with-directory-excursion delete-file-recursively)) #:use-module ((guix build download)@@ -158,6 +163,12 @@ Download and deploy the latest version of Guix.\n")) ;; a makefile, and, similarly, is intended to always keep this name. "build-aux/build-self.scm") +(define %pull-version+ ;; This is the version of the 'guix pull' protocol. It specifies what's+ ;; expected from %SELF-BUILD-FILE. The initial version ("0") was when we'd+ ;; place a set of compiled Guile modules in ~/.config/guix/latest.+ 1)+ (define* (build-from-source source #:key verbose? commit) "Return a derivation to build Guix from SOURCE, using the self-build script@@ -170,35 +181,51 @@ contained therein. Use COMMIT as the version string." (build (primitive-load script))) ;; BUILD must be a monadic procedure of at least one argument: the source ;; tree.- (build source #:verbose? verbose? #:version commit)))+ ;;+ ;; Note: BUILD can return #f if it does not support %PULL-VERSION. In the+ ;; future we'll fall back to a previous version of the protocol when that+ ;; happens.+ (build source #:verbose? verbose? #:version commit+ #:pull-version %pull-version)))++(define (whole-package-for-legacy name modules)+ "Return a full-blown Guix package for MODULES, a derivation that builds Guix+modules in the old ~/.config/guix/latest style."+ (whole-package name modules++ ;; In the "old style", %SELF-BUILD-FILE would simply return a+ ;; derivation that builds modules. We have to infer what the+ ;; dependencies of these modules were.+ (list guile-json guile-git guile-bytestructures+ guile-ssh gnutls)))++(define (derivation->manifest-entry drv commit)+ "Return a manifest entry for DRV, which represents Guix at COMMIT."+ (mbegin %store-monad+ (what-to-build (list drv))+ (built-derivations (list drv))+ (let ((out (derivation->output-path drv)))+ (return (manifest-entry+ (name "guix")+ (version (string-take commit 7))+ (item (if (file-exists? (string-append out "/bin/guix"))+ drv+ (whole-package-for-legacy (string-append name "-"+ version)+ drv)))))))) (define* (build-and-install source config-dir #:key verbose? commit) "Build the tool from SOURCE, and install it in CONFIG-DIR."- (mlet* %store-monad ((source (build-from-source source- #:commit commit- #:verbose? verbose?))- (source-dir -> (derivation->output-path source))- (to-do? (what-to-build (list source)))- (built? (built-derivations (list source))))- ;; Always update the 'latest' symlink, regardless of whether SOURCE was- ;; already built or not.- (if built?- (mlet* %store-monad- ((latest -> (string-append config-dir "/latest"))- (done (indirect-root-added latest)))- (if (and (file-exists? latest)- (string=? (readlink latest) source-dir))- (begin- (display (G_ "Guix already up to date\n"))- (return #t))- (begin- (switch-symlinks latest source-dir)- (format #t- (G_ "updated ~a successfully deployed under `~a'~%")- %guix-package-name latest)- (return #t))))- (leave (G_ "failed to update Guix, check the build log~%")))))+ (define update-profile+ (store-lift build-and-use-profile))++ (mlet* %store-monad ((drv (build-from-source source+ #:commit commit+ #:verbose? verbose?))+ (entry (derivation->manifest-entry drv commit)))+ (update-profile (string-append config-dir "/current")+ (manifest (list entry))))) (define (honor-lets-encrypt-certificates! store) "Tell Guile-Git to use the Let's Encrypt certificates."diff --git a/scripts/guix.in b/scripts/guix.inindex d1c12eae5..0a3ab1f64 100644--- a/scripts/guix.in+++ b/scripts/guix.in@@ -23,25 +23,13 @@ ;; IMPORTANT: We must avoid loading any modules from Guix here, ;; because we need to adjust the guile load paths first. ;; It's okay to import modules from core Guile though.-(use-modules (srfi srfi-26)) (define-syntax-rule (push! elt v) (set! v (cons elt v))) (define (augment-load-paths!) ;; Add installed modules to load-path. (push! "@guilemoduledir@" %load-path)- (push! "@guileobjectdir@" %load-compiled-path)-- ;; Add modules fetched by 'guix pull' to load-path.- (let ((updates-dir (and=> (or (getenv "XDG_CONFIG_HOME")- (and=> (getenv "HOME")- (cut string-append <> "/.config")))- (cut string-append <> "/guix/latest"))))- (when (and=> updates-dir file-exists?)- ;; XXX: Currently 'guix pull' puts both .scm and .go files in- ;; UPDATES-DIR.- (push! updates-dir %load-path)- (push! updates-dir %load-compiled-path))))+ (push! "@guileobjectdir@" %load-compiled-path)) (define* (main #:optional (args (command-line))) (unless (getenv "GUIX_UNINSTALLED")-- 2.17.0
L
L
Ludovic Courtès wrote on 31 May 2018 16:43
[PATCH 3/4] self: Compute and use locale data.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180531144337.16298-4-ludo@gnu.org
* guix/self.scm (locale-data): New procedure.(guix-command): Add SOURCE parameter. Call 'locale-data' when SOURCE istrue and use it in staged 'bindtextdomain' calls.(whole-package): Add #:command and honor it.(compiled-guix): Pass #:command to 'whole-package'.--- guix/self.scm | 101 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 77 insertions(+), 24 deletions(-)
Toggle diff (136 lines)diff --git a/guix/self.scm b/guix/self.scmindex f0641c342..92e69ffd5 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -190,7 +190,47 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'." (file-name->module-name (string-drop file prefix))) (scheme-files (string-append directory "/" sub-directory))))) -(define* (guix-command modules #:key (dependencies '())+(define* (locale-data source domain+ #:optional (sub-directory domain))+ "Return the locale data from 'po/SUB-DIRECTORY' in SOURCE, corresponding to+DOMAIN, a gettext domain."+ (define gettext+ (module-ref (resolve-interface '(gnu packages gettext))+ 'gettext-minimal))++ (define build+ (with-imported-modules '((guix build utils))+ #~(begin+ (use-modules (guix build utils)+ (srfi srfi-26)+ (ice-9 match) (ice-9 ftw))++ (define po-directory+ (string-append #$source "/po/" #$sub-directory))++ (define (compile language)+ (let ((gmo (string-append #$output "/" language "/LC_MESSAGES/"+ #$domain ".mo")))+ (mkdir-p (dirname gmo))+ (invoke #+(file-append gettext "/bin/msgfmt")+ "-c" "--statistics" "--verbose"+ "-o" gmo+ (string-append po-directory "/" language ".po"))))++ (define (linguas)+ ;; Return the list of languages. Note: don't read 'LINGUAS'+ ;; because it contains things like 'en@boldquot' that do not have+ ;; a corresponding .po file.+ (map (cut basename <> ".po")+ (scandir po-directory+ (cut string-suffix? ".po" <>))))++ (for-each compile (linguas)))))++ (computed-file (string-append "guix-locale-" domain)+ build))++(define* (guix-command modules #:key source (dependencies '()) (guile-version (effective-version))) "Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its load path."@@ -218,35 +258,43 @@ load path." (let ((guix-main (module-ref (resolve-interface '(guix ui)) 'guix-main)))- ;; TODO: Compute locale data.- ;; (bindtextdomain "guix" "@localedir@")- ;; (bindtextdomain "guix-packages" "@localedir@")+ #$(if source+ #~(begin+ (bindtextdomain "guix"+ #$(locale-data source "guix"))+ (bindtextdomain "guix-packages"+ #$(locale-data source+ "guix-packages"+ "packages")))+ #t) ;; XXX: It would be more convenient to change it to: ;; (exit (apply guix-main (command-line))) (apply guix-main (command-line)))))) (define* (whole-package name modules dependencies- #:key (guile-version (effective-version)))+ #:key+ (guile-version (effective-version))+ (command (guix-command modules+ #:dependencies dependencies+ #:guile-version guile-version))) "Return the whole Guix package NAME that uses MODULES, a derivation of all-the modules, and DEPENDENCIES, a list of packages depended on."- (let ((command (guix-command modules- #:dependencies dependencies- #:guile-version guile-version)))- ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'.- (computed-file name- (with-imported-modules '((guix build utils))- #~(begin- (use-modules (guix build utils))- (mkdir-p (string-append #$output "/bin"))- (symlink #$command- (string-append #$output "/bin/guix"))+the modules, and DEPENDENCIES, a list of packages depended on. COMMAND is the+'guix' program to use."+ ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'.+ (computed-file name+ (with-imported-modules '((guix build utils))+ #~(begin+ (use-modules (guix build utils))+ (mkdir-p (string-append #$output "/bin"))+ (symlink #$command+ (string-append #$output "/bin/guix")) - (let ((modules (string-append #$output- "/share/guile/site/"- (effective-version))))- (mkdir-p (dirname modules))- (symlink #$modules modules)))))))+ (let ((modules (string-append #$output+ "/share/guile/site/"+ (effective-version))))+ (mkdir-p (dirname modules))+ (symlink #$modules modules)))))) (define* (compiled-guix source #:key (version %guix-version) (pull-version 1)@@ -423,8 +471,13 @@ the modules, and DEPENDENCIES, a list of packages depended on." ;; Version 1 is when we return the full package. (cond ((= 1 pull-version) ;; The whole package, with a standard file hierarchy.- (whole-package name built-modules dependencies- #:guile-version guile-version))+ (let ((command (guix-command built-modules+ #:source source+ #:dependencies dependencies+ #:guile-version guile-version)))+ (whole-package name built-modules dependencies+ #:command command+ #:guile-version guile-version))) ((= 0 pull-version) ;; Legacy 'guix pull': just return the compiled modules. built-modules)-- 2.17.0
L
L
Ludovic Courtès wrote on 31 May 2018 16:43
[PATCH 1/4] self: Produce a complete package with the 'guix' command.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180531144337.16298-2-ludo@gnu.org
* guix/self.scm (guix-command): New procedure.(compiled-guix): Add #:pull-version parameter.[command, package]: New variables.Honor PULL-VERSION.(guix-derivation): Add #:pull-version and pass it to 'compiled-guix'.* build-aux/build-self.scm (build-program): Add #:pull-versionparameter. Pass it to 'guix-derivation'.(build): Add #:pull-version and pass it to 'build-program'.* build-aux/compile-as-derivation.scm: Pass #:pull-version to BUILD.--- build-aux/build-self.scm | 19 +++- build-aux/compile-as-derivation.scm | 2 +- guix/self.scm | 148 +++++++++++++++++++++------- 3 files changed, 129 insertions(+), 40 deletions(-)
Toggle diff (270 lines)diff --git a/build-aux/build-self.scm b/build-aux/build-self.scmindex bccb7a959..5898b6515 100644--- a/build-aux/build-self.scm+++ b/build-aux/build-self.scm@@ -184,7 +184,8 @@ person's version identifier." (date->string (current-date 0) "~Y~m~d.~H")) (define* (build-program source version- #:optional (guile-version (effective-version)))+ #:optional (guile-version (effective-version))+ #:key (pull-version 0)) "Return a program that computes the derivation to build Guix from SOURCE." (define select? ;; Select every module but (guix config) and non-Guix modules.@@ -253,11 +254,14 @@ person's version identifier." (spin system))) (display- (derivation-file-name+ (and=> (run-with-store store (guix-derivation #$source #$version- #$guile-version)- #:system system)))))))+ #$guile-version+ #:pull-version+ #$pull-version)+ #:system system)+ derivation-file-name)))))) #:module-path (list source)))) ;; The procedure below is our return value.@@ -266,13 +270,15 @@ person's version identifier." (guile-version (match ((@ (guile) version)) ("2.2.2" "2.2.2") (_ (effective-version))))+ (pull-version 0) #:allow-other-keys #:rest rest) "Return a derivation that unpacks SOURCE into STORE and compiles Scheme files." ;; Build the build program and then use it as a trampoline to build from ;; SOURCE.- (mlet %store-monad ((build (build-program source version guile-version))+ (mlet %store-monad ((build (build-program source version guile-version+ #:pull-version pull-version)) (system (if system (return system) (current-system)))) (mbegin %store-monad (show-what-to-build* (list build))@@ -292,6 +298,9 @@ files." (return (newline (current-output-port))) ((store-lift add-temp-root) drv) (return (read-derivation-from-file drv))))+ ("#f"+ ;; Unsupported PULL-VERSION.+ (return #f)) ((? string? str) (error "invalid build result" (list build str)))))))) diff --git a/build-aux/compile-as-derivation.scm b/build-aux/compile-as-derivation.scmindex afb134a92..2a45e71bb 100644--- a/build-aux/compile-as-derivation.scm+++ b/build-aux/compile-as-derivation.scm@@ -43,7 +43,7 @@ (mlet* %store-monad ((source (interned-file source "guix-source" #:select? git? #:recursive? #t))- (drv (build source)))+ (drv (build source #:pull-version 1))) (mbegin %store-monad (show-what-to-build* (list drv)) (built-derivations (list drv))diff --git a/guix/self.scm b/guix/self.scmindex 4378a3dee..f0641c342 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -34,6 +34,7 @@ #:use-module (srfi srfi-9) #:use-module (ice-9 match) #:export (make-config.scm+ whole-package ;for internal use in 'guix pull' compiled-guix guix-derivation reload-guix))@@ -189,7 +190,66 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'." (file-name->module-name (string-drop file prefix))) (scheme-files (string-append directory "/" sub-directory))))) +(define* (guix-command modules #:key (dependencies '())+ (guile-version (effective-version)))+ "Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its+load path."+ (program-file "guix-command"+ #~(begin+ (set! %load-path+ (append '#$(map (lambda (package)+ (file-append package+ "/share/guile/site/"+ guile-version))+ dependencies)+ %load-path))++ (set! %load-compiled-path+ (append '#$(map (lambda (package)+ (file-append package "/lib/guile/"+ guile-version+ "/site-ccache"))+ dependencies)+ %load-compiled-path))++ (set! %load-path (cons #$modules %load-path))+ (set! %load-compiled-path+ (cons #$modules %load-compiled-path))++ (let ((guix-main (module-ref (resolve-interface '(guix ui))+ 'guix-main)))+ ;; TODO: Compute locale data.+ ;; (bindtextdomain "guix" "@localedir@")+ ;; (bindtextdomain "guix-packages" "@localedir@")++ ;; XXX: It would be more convenient to change it to:+ ;; (exit (apply guix-main (command-line)))+ (apply guix-main (command-line))))))++(define* (whole-package name modules dependencies+ #:key (guile-version (effective-version)))+ "Return the whole Guix package NAME that uses MODULES, a derivation of all+the modules, and DEPENDENCIES, a list of packages depended on."+ (let ((command (guix-command modules+ #:dependencies dependencies+ #:guile-version guile-version)))+ ;; TODO: Move compiled modules to 'lib/guile' instead of 'share/guile'.+ (computed-file name+ (with-imported-modules '((guix build utils))+ #~(begin+ (use-modules (guix build utils))+ (mkdir-p (string-append #$output "/bin"))+ (symlink #$command+ (string-append #$output "/bin/guix"))++ (let ((modules (string-append #$output+ "/share/guile/site/"+ (effective-version))))+ (mkdir-p (dirname modules))+ (symlink #$modules modules)))))))+ (define* (compiled-guix source #:key (version %guix-version)+ (pull-version 1) (name (string-append "guix-" version)) (guile-version (effective-version)) (guile-for-build (guile-for-build guile-version))@@ -215,7 +275,6 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'." "guile-git" "guile2.0-git")) - (define dependencies (match (append-map (lambda (package) (cons (list "x" package)@@ -332,32 +391,46 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'." %guix-home-page-url))) #:guile-for-build guile-for-build)) - (directory-union name- (append-map (lambda (node)- (list (node-source node)- (node-compiled node)))+ (define built-modules+ (directory-union (string-append name "-modules")+ (append-map (lambda (node)+ (list (node-source node)+ (node-compiled node))) - ;; Note: *CONFIG* comes first so that it- ;; overrides the (guix config) module that- ;; comes with *CORE-MODULES*.- (list *config*- *cli-modules*- *system-modules*- *package-modules*- *core-package-modules*- *extra-modules*- *core-modules*))+ ;; Note: *CONFIG* comes first so that it+ ;; overrides the (guix config) module that+ ;; comes with *CORE-MODULES*.+ (list *config*+ *cli-modules*+ *system-modules*+ *package-modules*+ *core-package-modules*+ *extra-modules*+ *core-modules*)) - ;; Silently choose the first entry upon collision so that- ;; we choose *CONFIG*.- #:resolve-collision 'first+ ;; Silently choose the first entry upon collision so that+ ;; we choose *CONFIG*.+ #:resolve-collision 'first - ;; When we do (add-to-store "utils.scm"), "utils.scm" must- ;; be a regular file, not a symlink. Thus, arrange so that- ;; regular files appear as regular files in the final- ;; output.- #:copy? #t- #:quiet? #t))+ ;; When we do (add-to-store "utils.scm"), "utils.scm" must+ ;; be a regular file, not a symlink. Thus, arrange so that+ ;; regular files appear as regular files in the final+ ;; output.+ #:copy? #t+ #:quiet? #t))++ ;; Version 0 of 'guix pull' meant we'd just return Scheme modules.+ ;; Version 1 is when we return the full package.+ (cond ((= 1 pull-version)+ ;; The whole package, with a standard file hierarchy.+ (whole-package name built-modules dependencies+ #:guile-version guile-version))+ ((= 0 pull-version)+ ;; Legacy 'guix pull': just return the compiled modules.+ built-modules)+ (else+ ;; Unsupported 'guix pull' version.+ #f))) ;;;@@ -601,9 +674,12 @@ running Guile." 'guile-2.0)))) (define* (guix-derivation source version- #:optional (guile-version (effective-version)))+ #:optional (guile-version (effective-version))+ #:key (pull-version 0)) "Return, as a monadic value, the derivation to build the Guix from SOURCE-for GUILE-VERSION. Use VERSION as the version string."+for GUILE-VERSION. Use VERSION as the version string. PULL-VERSION specifies+the version of the 'guix pull' protocol. Return #f if this PULL-VERSION value+is not supported." (define (shorten version) (if (and (string-every char-set:hex-digit version) (> (string-length version) 9))@@ -615,11 +691,15 @@ for GUILE-VERSION. Use VERSION as the version string." (mbegin %store-monad (set-guile-for-build guile)- (lower-object (compiled-guix source- #:version version- #:name (string-append "guix-"- (shorten version))- #:guile-version (match guile-version- ("2.2.2" "2.2")- (version version))- #:guile-for-build guile))))+ (let ((guix (compiled-guix source+ #:version version+ #:name (string-append "guix-"+ (shorten version))+ #:pull-version pull-version+ #:guile-version (match guile-version+ ("2.2.2" "2.2")+ (version version))+ #:guile-for-build guile)))+ (if guix+ (lower-object guix)+ (return #f)))))-- 2.17.0
T
T
Thompson, David wrote on 31 May 2018 16:53
Re: bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
CAJ=Rwfa-YnqfCyrHSAMSS22XL-dWsGhOBpxtE2Fgn7CKqLwM=Q@mail.gmail.com
Hey Ludo,
On Thu, May 31, 2018 at 10:43 AM, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (11 lines)> Hello Guix!>> Here is the “new” ‘guix pull’ that we discussed notably in this thread:>> https://bugs.gnu.org/22629>> The major difference is that instead of just building a bunch of modules> and putting them under ~/.config/guix/latest, it now produces a> standalone package (with bin/guix, share/info/guix.info, etc.) and puts> it in a profile under ~/.config/guix/current.
Just dropping by to say this is a really awesome improvement to theuser experience! Thanks for working on it! Excited to try it out onceit gets merged.
- Dave
K
K
Konrad Hinsen wrote on 31 May 2018 20:00
(address . 22629@debbugs.gnu.org)
m136y7wvrm.fsf@fastmail.net
Hi Ludo,
Toggle quote (5 lines)> The major difference is that instead of just building a bunch of modules> and putting them under ~/.config/guix/latest, it now produces a> standalone package (with bin/guix, share/info/guix.info, etc.) and puts> it in a profile under ~/.config/guix/current. Quoth the manual:
This sounds like a very nice improvement, thanks!
Just wondering: does this mean that I could substitute Guix from my localsource tree simply by doing
./pre-inst-env guix package -p ~/.config/guix/current -i guix
Or, in fact, not use ~/.config/guix/current at all and just install"guix" into my main profile?
Konrad.
R
R
Ricardo Wurmus wrote on 31 May 2018 20:58
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
878t7zps7t.fsf@elephly.net
Toggle quote (9 lines)> Here is the “new” ‘guix pull’ that we discussed notably in this thread:>> https://bugs.gnu.org/22629>> The major difference is that instead of just building a bunch of modules> and putting them under ~/.config/guix/latest, it now produces a> standalone package (with bin/guix, share/info/guix.info, etc.) and puts> it in a profile under ~/.config/guix/current. […]
This is beautiful! Thank you so much for taking the time to implementthis.
Toggle quote (9 lines)>> The result of running ‘guix pull’ is a “profile” available under> ‘~/.config/guix/current’ containing the latest Guix. Thus, make sure to> add it to the beginning of your search path so that you use the latest> version, and similarly for the Info manual (*note Documentation::):>> export PATH="$HOME/.config/guix/current/bin:$PATH"> export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH"
As a profile it will have its very own “etc/profile” file. I supposethat doesn’t include INFOPATH, though, because it only contains an Infomanual but not an Info reader. It would be extra nice if we couldsimplify this initial setup even more.
Toggle quote (19 lines)> This ‘~/.config/guix/current’ profile works like any other profile> created by ‘guix package’ (*note Invoking guix package::). That is, you> can list generations, roll back to the previous generation—i.e., the> previous Guix—and so on:>> $ guix package -p ~/.config/guix/current -l> Generation 1 May 25 2018 10:06:41> guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4>> Generation 2 May 27 2018 19:07:47> + guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f> - guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4>> Generation 3 May 30 2018 16:11:39 (current)> + guix a076f19 out /gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990> - guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f> $ guix package -p ~/.config/guix/current --roll-back> switched from generation 3 to 2
This also means that you could remove the “guix” package and install“hello” instead. If a user did that they would lose their variant ofguix and they’d fall back to whichever version is installed on thesystem (if any). They could still roll back manually by changing thesymlink.
They could also think that installing the “guix” package into thatprofile would be a good idea — but then they would end up with aslightly older version of Guix. (This is already possible, of course,but if we have a separate profile that’s intended just for Guix but withgeneric properties, this could become confusing.)
I’m just thinking out loud about how users could get into trouble :)
Toggle quote (6 lines)> There are two requirements it fulfills in terms of compatibility:>> 1. The modified ‘build-aux/build-self.scm’ still does the right thing> when evaluated by an “old” Guix—that is, it produces a bunch of> modules for use in ~/.config/guix/latest as before.
Excellent! Thanks for thinking about this case.
Toggle quote (25 lines)> 2. The modified ‘guix pull’ produces ~/.config/guix/current even when> invoked on a commit of a past Guix. That is, it automatically> produces a ‘guix’ command using the modules returned by the old> ‘build-self.scm’.>> There are various improvements we can make from there. For example,> using “manifest entry properties” as proposed in> <https://bugs.gnu.org/31442>, we can attach meta-data (commit ID, repo> URL, etc.) in each manifest entry that ‘guix pull’ populates; then we> can arrange for ‘guix pull --list-generations’ (say) to display that> information.>> We could add ‘guix pull’ options for convenient: ‘--roll-back’,> ‘--profile’, etc.>> Going forward, additional “channels” could be presented as entries in> the ~/.config/guix/current manifest.>> Caveats:>> 1. The ~/.config/guix/current profile really lives there. That is,> unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.> That could be an issue for cluster setups where home directories> are not scanned by the Guix GC. Cluster folks, please tell me!
Is it impossible to store it in localstatedir?
In practice, cluster installations don’t really run “guix gc” all thatofter (if ever), so it may not be a problem.
Toggle quote (3 lines)> 3. C++ code is not built. I wonder which will come first: getting rid> of the C++ code, or building it? :-)
I’d say that this is a feature ;)
--Ricardo
L
L
Ludovic Courtès wrote on 1 Jun 2018 14:11
(name . Ricardo Wurmus)(address . rekado@elephly.net)(address . 22629@debbugs.gnu.org)
87po1aaepm.fsf@gnu.org
Hello!
Ricardo Wurmus <rekado@elephly.net> skribis:
Toggle quote (13 lines)>> The result of running ‘guix pull’ is a “profile” available under>> ‘~/.config/guix/current’ containing the latest Guix. Thus, make sure to>> add it to the beginning of your search path so that you use the latest>> version, and similarly for the Info manual (*note Documentation::):>>>> export PATH="$HOME/.config/guix/current/bin:$PATH">> export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH">> As a profile it will have its very own “etc/profile” file. I suppose> that doesn’t include INFOPATH, though, because it only contains an Info> manual but not an Info reader. It would be extra nice if we could> simplify this initial setup even more.
I agree. On GuixSD of course we can add the right default. On foreigndistros, I’m not sure how we can simplify things though.
At least, when you run the new ‘guix pull’ for the first time, you get anote that you should define PATH.
Toggle quote (5 lines)>> This ‘~/.config/guix/current’ profile works like any other profile>> created by ‘guix package’ (*note Invoking guix package::). That is, you>> can list generations, roll back to the previous generation—i.e., the>> previous Guix—and so on:
[...]
Toggle quote (14 lines)> This also means that you could remove the “guix” package and install> “hello” instead. If a user did that they would lose their variant of> guix and they’d fall back to whichever version is installed on the> system (if any). They could still roll back manually by changing the> symlink.>> They could also think that installing the “guix” package into that> profile would be a good idea — but then they would end up with a> slightly older version of Guix. (This is already possible, of course,> but if we have a separate profile that’s intended just for Guix but with> generic properties, this could become confusing.)>> I’m just thinking out loud about how users could get into trouble :)
Yeah well, we can’t really prevent them from shooting themselves in thefoot. :-)
Now, we could hide the fact that people can use “guix package -p” onthat thing by simply providing ‘--list-generations’, ‘--roll-back’,etc. directly in ‘guix pull’.
I think it’ll be a good idea to add a custom ‘--list-generations’ in thefuture anyway, because it could display more useful information (commitIDs, number of available packages, etc.)
Toggle quote (7 lines)>> 1. The ~/.config/guix/current profile really lives there. That is,>> unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.>> That could be an issue for cluster setups where home directories>> are not scanned by the Guix GC. Cluster folks, please tell me!>> Is it impossible to store it in localstatedir?
It wouldn’t be hard, but I wanted to get things done quickly. :-)
Thanks for your feedback!
Ludo’.
L
L
Ludovic Courtès wrote on 1 Jun 2018 14:13
(name . Thompson, David)(address . dthompson2@worcester.edu)(address . 22629@debbugs.gnu.org)
87k1riaend.fsf@gnu.org
Heya David,
"Thompson, David" <dthompson2@worcester.edu> skribis:
Toggle quote (16 lines)> On Thu, May 31, 2018 at 10:43 AM, Ludovic Courtès <ludo@gnu.org> wrote:>> Hello Guix!>>>> Here is the “new” ‘guix pull’ that we discussed notably in this thread:>>>> https://bugs.gnu.org/22629>>>> The major difference is that instead of just building a bunch of modules>> and putting them under ~/.config/guix/latest, it now produces a>> standalone package (with bin/guix, share/info/guix.info, etc.) and puts>> it in a profile under ~/.config/guix/current.>> Just dropping by to say this is a really awesome improvement to the> user experience! Thanks for working on it! Excited to try it out once> it gets merged.
Thanks for your support! We’ll still have to make the whole buildprocess faster but yeah, that’s one important step towards 1.0!
Ludo’.
P
P
Pjotr Prins wrote on 3 Jun 2018 15:08
(name . Ludovic Courtès)(address . ludo@gnu.org)
20180603130802.qcz6e5iz2wwxw7to@thebird.nl
On Fri, Jun 01, 2018 at 02:13:10PM +0200, Ludovic Courtès wrote:
Toggle quote (3 lines)> Thanks for your support! We’ll still have to make the whole build> process faster but yeah, that’s one important step towards 1.0!
But we do get substitutes in principle, right?
Pj.
L
L
Ludovic Courtès wrote on 3 Jun 2018 22:29
(name . Pjotr Prins)(address . pjotr.public12@thebird.nl)
87bmcr62bs.fsf@gnu.org
Pjotr Prins <pjotr.public12@thebird.nl> skribis:
Toggle quote (6 lines)> On Fri, Jun 01, 2018 at 02:13:10PM +0200, Ludovic Courtès wrote:>> Thanks for your support! We’ll still have to make the whole build>> process faster but yeah, that’s one important step towards 1.0!>> But we do get substitutes in principle, right?
Yes, but only for berlin.guixsd.org at the moment (because of obscureissues with Guile support in Hydra that I’m not really willing to fix):
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#147
Ludo’.
L
L
Ludovic Courtès wrote on 4 Jun 2018 13:20
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)(address . 22629@debbugs.gnu.org)
87602yu7bd.fsf@gnu.org
Hello!
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (5 lines)> Just wondering: does this mean that I could substitute Guix from my local> source tree simply by doing>> ./pre-inst-env guix package -p ~/.config/guix/current -i guix
That would just install the snapshot that the ‘guix’ package refers to(it’s defined in (gnu packages package-management)).)
Toggle quote (3 lines)> Or, in fact, not use ~/.config/guix/current at all and just install> "guix" into my main profile?
If we add a ‘-p’ option to ‘guix pull’, that’s something you could do.
Now, installing Guix in your main profile is probably not a good idea.Since Guix it what allows you to manage the profile in question, it’sprobably better to keep it separate. That way, ‘guix’ is managedindependently of upgrades, installs, etc. that you make in your profile;rolling back your main profile doesn’t roll back Guix, and so on.
HTH,Ludo’.
P
P
Pjotr Prins wrote on 4 Jun 2018 21:12
(name . Ludovic Courtès)(address . ludo@gnu.org)
20180604191218.hlobvqdpkka5gky5@thebird.nl
On Sun, Jun 03, 2018 at 10:29:43PM +0200, Ludovic Courtès wrote:
Toggle quote (11 lines)> Pjotr Prins <pjotr.public12@thebird.nl> skribis:> > > On Fri, Jun 01, 2018 at 02:13:10PM +0200, Ludovic Courtès wrote:> >> Thanks for your support! We’ll still have to make the whole build> >> process faster but yeah, that’s one important step towards 1.0!> >> > But we do get substitutes in principle, right?> > Yes, but only for berlin.guixsd.org at the moment (because of obscure> issues with Guile support in Hydra that I’m not really willing to fix):
That is great news Ludo. It also means that channels are around thecorner because we can use a substituted version of Guix as the basis.
That is unbelievably cool.
Pj.
K
K
Konrad Hinsen wrote on 5 Jun 2018 13:45
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
m1po158nj8.fsf@fastmail.net
Hi Ludo,
Toggle quote (8 lines)>> Just wondering: does this mean that I could substitute Guix from my local>> source tree simply by doing>>>> ./pre-inst-env guix package -p ~/.config/guix/current -i guix>> That would just install the snapshot that the ‘guix’ package refers to> (it’s defined in (gnu packages package-management)).)
Fine, so if I run update-guix-package.scm and then do the install, I getwhat I expect, right?
I am looking for a reasonably straightforward way to have everything inmy Guix universe (the guix command line tool, emacs-guix, guile, geiser,...) consistently use my personal version of the package definitions.
Toggle quote (4 lines)> Now, installing Guix in your main profile is probably not a good idea.> Since Guix it what allows you to manage the profile in question, it’s> probably better to keep it separate. That way, ‘guix’ is managed
Yes, that makes sense.
Konrad.
F
F
Fis Trivial wrote on 5 Jun 2018 18:47
(name . Ludovic Courtès)(address . ludo@gnu.org)(name . 22629@debbugs.gnu.org)(address . 22629@debbugs.gnu.org)
BLUPR16MB050040E9004DB6DDFECFE43792660@BLUPR16MB0500.namprd16.prod.outlook.com
Hi, first of all, thanks for the improvement. It's really exciting know progress in guix. But I have a few questions around this change. Just curiosity. :)
Toggle quote (19 lines)> Hello Guix! > > Here is the “new” ‘guix pull’ that we discussed notably in this thread: > > https://bugs.gnu.org/22629 > > The major difference is that instead of just building a bunch of modules > and putting them under ~/.config/guix/latest, it now produces a > standalone package (with bin/guix, share/info/guix.info, etc.) and puts > it in a profile under ~/.config/guix/current. Quoth the manual: > > The result of running ‘guix pull’ is a “profile” available under > ‘~/.config/guix/current’ containing the latest Guix. Thus, make sure to > add it to the beginning of your search path so that you use the latest > version, and similarly for the Info manual (*note Documentation::): > > export PATH="$HOME/.config/guix/current/bin:$PATH" > export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH" >
This sounds like something could be done by guix itself, choosing the right profile path by $HOME. Since guix has absolute control about this piece of information?
Toggle quote (34 lines)> This ‘~/.config/guix/current’ profile works like any other profile > created by ‘guix package’ (*note Invoking guix package::). That is, you > can list generations, roll back to the previous generation—i.e., the > previous Guix—and so on: > > $ guix package -p ~/.config/guix/current -l > Generation 1 May 25 2018 10:06:41 > guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4 > > Generation 2 May 27 2018 19:07:47 > + guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f > - guix 221951a out /gnu/store/i4dfk7vw5k112s49jrhl6hwsfnh6wr7l-guix-221951af4 > > Generation 3 May 30 2018 16:11:39 (current) > + guix a076f19 out /gnu/store/332czkicwwg6lc3x4aqbw5q2mq12s7fj-guix-a076f1990 > - guix 2fbae00 out /gnu/store/44cv9hyvxg34xf5kblf5dz57hc52y4bm-guix-2fbae006f > $ guix package -p ~/.config/guix/current --roll-back > switched from generation 3 to 2 > >> There are two requirements it fulfills in terms of compatibility: > > 1. The modified ‘build-aux/build-self.scm’ still does the right thing > when evaluated by an “old” Guix—that is, it produces a bunch of > modules for use in ~/.config/guix/latest as before. > > 2. The modified ‘guix pull’ produces ~/.config/guix/current even when > invoked on a commit of a past Guix. That is, it automatically > produces a ‘guix’ command using the modules returned by the old > ‘build-self.scm’. > > > We could add ‘guix pull’ options for convenient: ‘--roll-back’, > ‘--profile’, etc. >
Nice~
Toggle quote (9 lines)> Going forward, additional “channels” could be presented as entries in > the ~/.config/guix/current manifest. > > Caveats: > > 1. The ~/.config/guix/current profile really lives there. That is, > unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user. > That could be an issue for cluster setups where home directories > are not scanned by the Guix GC. Cluster folks, please tell me!
What does that mean? We have a guix directory under $HOME/.config, inside there's a symlink to /gnu/store/...-guix-<commit>. Does "really lives there" mean the new profile is not a symlink but a concrete directory or hard link?
Toggle quote (8 lines)> > 2. The translated Info manual is not built. Julien: could you turn > the big ‘xref_command’ in a script or something that we can more > easily reuse in (guix self)? > > 3. C++ code is not built. I wonder which will come first: getting rid > of the C++ code, or building it? :-) >
In the future world, how do we update guix daemon? Is't still running guix pull && guix package -u under root user?
L
L
Ludovic Courtès wrote on 6 Jun 2018 15:24
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)(address . 22629@debbugs.gnu.org)
87in6wjbe2.fsf@gnu.org
Hello Konrad,
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (11 lines)>>> Just wondering: does this mean that I could substitute Guix from my local>>> source tree simply by doing>>>>>> ./pre-inst-env guix package -p ~/.config/guix/current -i guix>>>> That would just install the snapshot that the ‘guix’ package refers to>> (it’s defined in (gnu packages package-management)).)>> Fine, so if I run update-guix-package.scm and then do the install, I get> what I expect, right?
Yes (even with current ‘master’), but it’s quite heavyweight since youend up recompiling all of Guix.
Toggle quote (4 lines)> I am looking for a reasonably straightforward way to have everything in> my Guix universe (the guix command line tool, emacs-guix, guile, geiser,> ...) consistently use my personal version of the package definitions.
Would ‘guix package --manifest’ suffice? Perhaps along with ‘guix pull--commit=XXX’?
In the not-too-distant future I think we can add some sort of manifestsupport for ‘guix pull’ such that you can store the list of channel URLsand commit IDs in one file and instantiate that.
Ludo’.
L
L
Ludovic Courtès wrote on 6 Jun 2018 15:27
(name . Fis Trivial)(address . ybbs.daans@hotmail.com)(name . 22629@debbugs.gnu.org)(address . 22629@debbugs.gnu.org)
87d0x4jb8g.fsf@gnu.org
Hello!
Fis Trivial <ybbs.daans@hotmail.com> skribis:
Toggle quote (13 lines)>> The result of running ‘guix pull’ is a “profile” available under>> ‘~/.config/guix/current’ containing the latest Guix. Thus, make sure to>> add it to the beginning of your search path so that you use the latest>> version, and similarly for the Info manual (*note Documentation::):>>>> export PATH="$HOME/.config/guix/current/bin:$PATH">> export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH">>>> This sounds like something could be done by guix itself, choosing the> right profile path by $HOME. Since guix has absolute control about this> piece of information?
I think it would be bad for Guix to modify your ~/.bashrc directly, soit’s better to let users do that.
GuixSD will have the right PATH and INFOPATH by default, though.
Toggle quote (12 lines)>> Caveats:>>>> 1. The ~/.config/guix/current profile really lives there. That is,>> unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user.>> That could be an issue for cluster setups where home directories>> are not scanned by the Guix GC. Cluster folks, please tell me!>> What does that mean? We have a guix directory under $HOME/.config,> inside there's a symlink to /gnu/store/...-guix-<commit>. Does "really> lives there" mean the new profile is not a symlink but a concrete> directory or hard link?
Please see how ~/.guix-profile and/var/guix/profiles/per-user/$USER/guix-profile work together. Basicallygenerations show up in /var/guix/profiles, whereas ~/.guix-profile is afixed symlink.
Toggle quote (6 lines)>> 3. C++ code is not built. I wonder which will come first: getting rid>> of the C++ code, or building it? :-)>>> In the future world, how do we update guix daemon? Is't still running> guix pull && guix package -u under root user?
In the future world, ‘guix pull’ updates everything: client-side anddaemon. Currently it’s still client-side only.
HTH!
Ludo’.
K
K
Konrad Hinsen wrote on 6 Jun 2018 17:54
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
m17enbrjvj.fsf@ordinateur-de-catherine--konrad.home
Hi Ludo,
Toggle quote (6 lines)>> Fine, so if I run update-guix-package.scm and then do the install, I get>> what I expect, right?>> Yes (even with current ‘master’), but it’s quite heavyweight since you> end up recompiling all of Guix.
Not great, but doable from time to time.
Toggle quote (7 lines)>> I am looking for a reasonably straightforward way to have everything in>> my Guix universe (the guix command line tool, emacs-guix, guile, geiser,>> ...) consistently use my personal version of the package definitions.>> Would ‘guix package --manifest’ suffice? Perhaps along with ‘guix pull> --commit=XXX’?
'guix pull --commit=XXX' would be of use if it works with an URLpointing to a local repository (haven't tried yet). I'd alsohave to point GUILE_LOAD_PATH to wherever 'guix pull' stores the sourcecode tree, which looks doable as well if that's a stable location.
Toggle quote (4 lines)> In the not-too-distant future I think we can add some sort of manifest> support for ‘guix pull’ such that you can store the list of channel URLs> and commit IDs in one file and instantiate that.
That sounds really nice :-)
Thanks for all the good stuff BTW!
Konrad.
L
L
Ludovic Courtès wrote on 6 Jun 2018 22:49
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)(address . 22629@debbugs.gnu.org)
87y3frfxna.fsf@gnu.org
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (12 lines)>>> I am looking for a reasonably straightforward way to have everything in>>> my Guix universe (the guix command line tool, emacs-guix, guile, geiser,>>> ...) consistently use my personal version of the package definitions.>>>> Would ‘guix package --manifest’ suffice? Perhaps along with ‘guix pull>> --commit=XXX’?>> 'guix pull --commit=XXX' would be of use if it works with an URL> pointing to a local repository (haven't tried yet). I'd also> have to point GUILE_LOAD_PATH to wherever 'guix pull' stores the source> code tree, which looks doable as well if that's a stable location.
You can already run “guix pull --commit=X --url=$PWD” or similar, andmodules currently end up ~/.config/guix/latest.
Cheers,Ludo’.
F
F
Fis Trivial wrote on 6 Jun 2018 22:58
(name . Ludovic Courtès)(address . ludo@gnu.org)
BLUPR16MB0500160530968B370542E1D892650@BLUPR16MB0500.namprd16.prod.outlook.com
Toggle quote (17 lines)> >>> The result of running ‘guix pull’ is a “profile” available under >>> ‘~/.config/guix/current’ containing the latest Guix. Thus, make sure to >>> add it to the beginning of your search path so that you use the latest >>> version, and similarly for the Info manual (*note Documentation::): >>> >>> export PATH="$HOME/.config/guix/current/bin:$PATH" >>> export INFOPATH="$HOME/.config/guix/current/share/info:$INFOPATH" >>> >> >> This sounds like something could be done by guix itself, choosing the >> right profile path by $HOME. Since guix has absolute control about this >> piece of information? > > I think it would be bad for Guix to modify your ~/.bashrc directly, so > it’s better to let users do that. >
Well, I mean a wrapper script looks similar to this in /usr/bin or /usr/local/bin: #!/path/to/store/bash # A wrapper named guix lives in /usr/bin/ or /usr/local/bin exec $HOME/.config/guix/current/bin/guix "$@" I might be wrong, please forgive my poor knowledge.
Toggle quote (19 lines)> GuixSD will have the right PATH and INFOPATH by default, though. > >>> Caveats: >>> >>> 1. The ~/.config/guix/current profile really lives there. That is, >>> unlike ~/.guix-profile, it’s not in /var/guix/profiles/per-user. >>> That could be an issue for cluster setups where home directories >>> are not scanned by the Guix GC. Cluster folks, please tell me! >> >> What does that mean? We have a guix directory under $HOME/.config, >> inside there's a symlink to /gnu/store/...-guix-<commit>. Does "really >> lives there" mean the new profile is not a symlink but a concrete >> directory or hard link? > > Please see how ~/.guix-profile and > /var/guix/profiles/per-user/$USER/guix-profile work together. Basically > generations show up in /var/guix/profiles, whereas ~/.guix-profile is a > fixed symlink. >
Thanks for the explanation, but running $ls shows: lrwxrwxrwx. 1 fis fis 44 Mar 28 16:41 .guix-profile -> /var/guix/profiles/per-user/fis/guix-profile You are talking about the same directory "/var/guix/profiles/per-user/$USER/guix-profile" and "~/.guix-profile". That's fine, I am waiting for the new feature to land home in my device anyway. I will inspect more closely then.
Toggle quote (9 lines)>>> 3. C++ code is not built. I wonder which will come first: getting rid >>> of the C++ code, or building it? :-) >>> >> In the future world, how do we update guix daemon? Is't still running >> guix pull && guix package -u under root user? > > In the future world, ‘guix pull’ updates everything: client-side and > daemon. Currently it’s still client-side only. >
That sounds really nice! Thanks!
Toggle quote (2 lines)> HTH! >
Interesting, you never run out of acronym :). If it means "hope this help", sure, thanks for the reply.
Toggle quote (1 lines)> Ludo’.
L
L
Ludovic Courtès wrote on 9 Jun 2018 12:07
(address . 22629-done@debbugs.gnu.org)
87efhgi880.fsf@gnu.org
Hello Guix!
Ludovic Courtès <ludo@gnu.org> skribis:
Toggle quote (9 lines)> Here is the “new” ‘guix pull’ that we discussed notably in this thread:>> https://bugs.gnu.org/22629>> The major difference is that instead of just building a bunch of modules> and putting them under ~/.config/guix/latest, it now produces a> standalone package (with bin/guix, share/info/guix.info, etc.) and puts> it in a profile under ~/.config/guix/current. Quoth the manual:
I have just pushed the new ‘guix pull’, with a few improvements comparedto the patches I had sent:
• Both guix.info and all of guix.LANG.info are now built.
• The derivation that builds locale data depends only on the po/ directory, so it won’t be rebuilt every time. Likewise for the derivation that builds the manual.
• Meta-data about what was pulled is kept in manifest entries, using the ‘properties’ discussed in https://bugs.gnu.org/31442. Currently the UI doesn’t use it but you can see that info in ~/.config/guix/current/manifest.
Let me know how it goes!
Ludo’.
Closed
L
L
Ludovic Courtès wrote on 13 Jun 2018 23:46
Re: bug#28471: closed (Re: bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix)
87bmcenyvk.fsf@gnu.org
Hello!
Toggle quote (5 lines)> • Meta-data about what was pulled is kept in manifest entries, using> the ‘properties’ discussed in <https://bugs.gnu.org/31442>.> Currently the UI doesn’t use it but you can see that info in> ~/.config/guix/current/manifest.
Commit e2f8be0664609223369f01290b69b44196783ab3 adds ‘guix pull -l’,which produces output like this:
Toggle snippet (19 lines)Generation 1 Jun 10 2018 00:18:18 guix 65956ad repository URL: https://git.savannah.gnu.org/git/guix.git branch: origin/master commit: 65956ad3526ba09e1f7a40722c96c6ef7c0936fe
Generation 2 Jun 11 2018 11:02:49 guix e0cc7f6 repository URL: https://git.savannah.gnu.org/git/guix.git branch: origin/master commit: e0cc7f669bec22c37481dd03a7941c7d11a64f1d
Generation 3 Jun 13 2018 23:31:07 (current) guix 844cc1c repository URL: https://git.savannah.gnu.org/git/guix.git branch: origin/master commit: 844cc1c8f394f03b404c5bb3aee086922373490c
In the future it could also display information such as the number ofpackages available, the number of added/upgraded packages, and so on.
Ludo’.
C
C
Chris Marusich wrote on 19 Jul 2018 06:45
Re: bug#22629: [PATCH 0/4] 'guix pull' produces a self-contained Guix
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
87601bkf40.fsf@gmail.com
Hi Ludo,
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (4 lines)> Here is the “new” ‘guix pull’ that we discussed notably in this thread:>> https://bugs.gnu.org/22629
Although I've been enjoying the new guix pull for a while now, I justtoday found the time to take a peek at these code changes. It's alwayseducational to see how you write your code!
I was surprised that "guix pull" doesn't build the guix package via theusual mechanisms (e.g., the way it would be built if I ran "guix buildguix"). The new "guix pull" code builds a profile, so it seems like wecould put packages in there (e.g., a guix package that inherits from theoriginal but replaces the origin with a Git checkout). However, insteadof re-using the build logic encapsulated in the guix package, it lookslike we build Guix piece by piece using custom build logic in (guixself). Why do we do that?
These improvements are great! I'm just curious about the motivation forthe custom build logic. I'm guessing there is some sort of complicationto simply using the guix package that is non-obvious, but I can't seewhat it is.
-- Chris
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAltQF38ACgkQ3UCaFdgiRp1ZBw//aibeMy1QxaCnUWh1W9vlZHJz8WyieYt6ivh3KEeiQk8UAgTe72MckmH7MGCKKaztZVyezJeqzg6YNxOnGCHOfVPav/RdC6K8crrE3xw9XHFIZlFzxfp/BGnD5n3nI7xTnrVqOhrNl/O7Hw+PKu/ICoZsT26TZhDG/rcxDauDdRFrId8fGEx+/I7kNcHIi25l2hRHbGEQa/9zmO+8XKRxqUzuhObl2pFDj8cU32I7MxkvVzlhYyJl8q6FrdFN3pthu49ljq8mr1ExcpqUCcrpW0aRnnYL6ONsd6JfgYqt++EsN049NA0Q5wd/9Fu5pk3OGUSD8/Xt3CG+nfnTmpqpDDHgUaF6AmB14Xm1melvgdjhx7l1IRcm6IwFphcDVFOKFQ96e2nVs6e4KzwC9/OQYpVsF07gPNTKewbmjC3jBIiq/4T5OTSw58f/joB0Ne8eOvQ2iwz5CurTJJ/uW27Smlw5teSvzXKTZW/pzKGPvGC651cvmRRTicVKgYBdHu5A3ISIJLkkjZMrrervmPK2nwc0vgEV/YtCmT6LkFS3quXpyan8L/2M3S9ymc6EvfsQfLQ1vTFz5gD9ueY4ph22r8Gf2uEZk6zDm3MCLOk93/A9K6JWxxyxzsnRICM1FZa/BXmNFldqnSjts9wm/BSCuhMGJ0L3WClwwZx1WcLzVsU==Cdz8-----END PGP SIGNATURE-----
L
L
Ludovic Courtès wrote on 19 Jul 2018 14:15
(name . Chris Marusich)(address . cmmarusich@gmail.com)(address . 22629@debbugs.gnu.org)
87fu0fv2u0.fsf@gnu.org
Hello Chris,
Thanks for your feedback!
Chris Marusich <cmmarusich@gmail.com> skribis:
Toggle quote (9 lines)> I was surprised that "guix pull" doesn't build the guix package via the> usual mechanisms (e.g., the way it would be built if I ran "guix build> guix"). The new "guix pull" code builds a profile, so it seems like we> could put packages in there (e.g., a guix package that inherits from the> original but replaces the origin with a Git checkout). However, instead> of re-using the build logic encapsulated in the guix package, it looks> like we build Guix piece by piece using custom build logic in (guix> self). Why do we do that?
There’s a couple of problems that we’re trying to fix, which werediscussed at https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27284#133:
1. Bootstrapping: how do we built a new Guix from an already-installed, older Guix?
2. Efficiency: how do we avoid recompiling every since Scheme file at each revision?
The answer to #2 is, of course, to have a finer-grain view on what needsto be built. We have makefiles that have view at the file level, andhere we essentially want to do what makefiles do, but we also want tointroduce dependencies (our makefiles do not declare dependencies amongScheme files, which sometimes lead to those infamous ABI issues.)
We could have defined package objects to describe each of the varioussubparts of Guix: a “guix-core” package, a “guix-packages” package, etc.But it’s not a good fit: these aren’t really packages, there’s no UI toaccess them as such anyway, and we’re closer to the abstraction level ofa makefile than to the abstraction level of a package.
So I chose to define a custom <node> type in (guix self). Each noderepresents a group of files, in particular Scheme files that need to becompiled; nodes can have edges (the ‘dependencies’ field). On top ofthat there are helper procedures to create and manipulate nodes. Forinstance, ‘scheme-nodes’ creates a node that builds the closure of alist of .scm files, minus the files already built by its dependencies.
The end result is that the full Guix consists of half a dozen of Schemederivations. Derivations at the root of the DAG (e.g., “guix-core”)hopefully change quite infrequently, and as such we don’t have torebuild everything at each revision.
As I wrote before in the issue above, there’s room for improvement. Inparticular we could split nodes further.
I hope this clarifies things!
Ludo’.
L
L
Ludovic Courtès wrote on 28 Aug 2018 17:16
Channels!
(address . 22629@debbugs.gnu.org)
87pny2iks2.fsf@gnu.org
Hi Guix!
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (12 lines)> • Have a “channel” mechanism, similar to ‘nix-channel’, that would> allow users to have several Guix variants available in parallel> instead of just “latest”. Could work like this:>> guix channel add latest git://git.sv.gnu.org/guix.git master> guix channel add stable git://git.sv.gnu.org/guix.git stable> guix channel pull latest> guix channel set latest> # here i see the latest versions of everything> guix channel set stable> # and here everything is old but super stable ;-)
The patches that follow implement this last bit, though in a slightlydifferent way. Users would now have the option to provide~/.config/guix/channels.scm along these lines:
(cons (channel (name 'guix-hpc) (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git") (branch "origin/master")) %default-channels)
‘guix pull’ then pulls both from ‘%default-channels’, a one-element listcontaining our Savannah repo, and from that ‘guix-hpc’ repo. It buildsboth Guix itself and the modules in ‘guix-hpc’, with a dependency on thejust-built Guix. The ~/.config/guix/current profile then contains twoentries:
Toggle snippet (33 lines)$ guix pull -l[…]Generacio 18 Aug 27 2018 09:54:40 guix 1ca5499 repository URL: https://git.savannah.gnu.org/git/guix.git branch: origin/master commit: 1ca54999db34b0197e6c7fa3f0a852fc0d193e70 10 new packages: avogadro, emacs-git-annex, find-circ, guile-aa-tree, lepton-eda, r-a3, r-abcanalysis, r-manipulatewidget, r-miniui, r-slam 43 packages upgraded: byobu@5.127, cpupower@4.18.5, darcs@2.14.1, drumkv1@0.9.2, duplicity@0.7.18, emacs-recent-addresses@0.1-1.afbbfdc, font-gnu-unifont@11.0.02, freefall@4.18.5, gdb-arm-none-eabi@8.1.1, gdb@8.1.1, ghc-integer-logarithms@1.0.2.1, ghc-optparse-applicative@0.14.2.0, ghc-quickcheck-instances@0.3.18, ghc-scientific@0.3.6.2, ghc-smallcheck@1.1.5, ghc-zlib@0.6.2, guile-lib@0.2.6.1, guile2.0-lib@0.2.6.1, guile2.2-lib@0.2.6.1, keepassxc@2.3.4, libgig@4.1.0, libx11@1.6.6, linux-libre-arm-generic@4.18.5, linux-libre-arm-omap2plus@4.18.5, linux-libre@4.18.5, ncmpc@0.29, openssh@7.8p1, padthv1@0.9.2, parallel@20180822, perf@4.18.5, qjackctl@0.5.3, r-car@3.0-1, r-catools@1.17.1.1, r-devtools@1.13.6, r-digest@0.6.16, r-dplyr@0.7.6, r-rcpp@0.12.18, r-synchronicity@1.3.5, r-trimcluster@0.1-2.1, samplv1@0.9.2, synthv1@0.9.2, tuxguitar@1.5.2, yubico-piv-tool@1.6.1Generacio 19 Aug 27 2018 16:20:48 guix d894ab8 repository URL: file:///home/ludo/src/guix branch: origin/wip-channels commit: d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300 guix-hpc dd3df5e repository URL: file:///home/ludo/src/guix-hpc branch: origin/wip-channel-test commit: dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb 11 new packages: chameleon, eztrace, fxt, litl, maphys, pastix, simgrid, starpu, starpu, starpu-fxt, starpu-simgrid 4 packages upgraded: capstone@3.0.5-rc2, emacs-racket-mode@0.0.2-2.1b78827, python-capstone@3.0.5-rc2, python2-capstone@3.0.5-rc2
GUIX_PACKAGE_PATH is no longer needed (though it’s still supported):~/.config/guix/current/bin/guix automatically finds packages defined inthe ‘guix-hpc’ channel.
The only missing bit for this patch series is documentation for channels(including a prominent disclaimer about API compatibility). I’ll writeit if people agree with the overall approach.

Implementation~~~~~~~~~~~~~~
(guix channels) implements the “channel” and “channel instance”abstractions. It provides tools to get the latest instance of channeland to build a list of instances. There must always be a channel called‘guix’, which all other instances depend on.
(guix scripts pull) now does little more than implementing the CLI,while (guix channels) does all the heavy lifting.

Limitations~~~~~~~~~~~
Currently third-party channels are expected to provide nothing butpackage modules. Those package modules should live at the root of therepository; for example, the modules/ directory athttps://gitlab.inria.fr/guix-hpc/guix-hpcdoesn’t work, you have to“git mv modules/inria .” for things to work.
Both limitations could be lifted by honoring a special meta-data file inrepositories. But I think that can come later.
Third-party channels currently all implicitly depend on the ‘guix’channel, but it’s not possible to express dependencies among channels.It may be useful, but it can probably come later, too.
If a channel fails to build, the user gets a backtrace and it’s up tothem to figure out what to do. May need to be improved. :-)
Not really a limitation, but (guix describe) relies on argv[0] to findchannel meta-data. I’m not a fan of that but it works in practice.More importantly, it’s the only way I could think of to retrieve“silent” meta-data (repo URL, commit, etc.) as well as channeldirectories where packages should be searched for.

Future work~~~~~~~~~~~
With (guix channels) we’ll be able to implement things like “open aninferior corresponding to commit X”. In turn, that means we can supportcommit IDs in manifests, and then “guix package -m manifest.scm” willopen an inferior for the requested commit ID and instantiate the thing.
With (guix describe) we can, of course, implement a ‘guix describe’command. We can also improve provenance tracking, for instance bystoring as a manifest entry property the channel instance an installedpackage comes from.

Thoughts? Ricardo?
Thanks in advance for your feedback!
Ludo’.
L
L
Ludovic Courtès wrote on 28 Aug 2018 17:17
[PATCH 1/3] discovery: Add 'scheme-modules*'.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180828151751.4657-1-ludo@gnu.org
* guix/self.scm (scheme-modules*): Move to...* guix/discovery.scm (scheme-modules*): ... here. New procedure. Make'sub-directory' an optional parameter.--- guix/discovery.scm | 13 ++++++++++++- guix/self.scm | 7 ------- 2 files changed, 12 insertions(+), 8 deletions(-)
Toggle diff (56 lines)diff --git a/guix/discovery.scm b/guix/discovery.scmindex 2b627d108..3fc6e2c9e 100644--- a/guix/discovery.scm+++ b/guix/discovery.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -27,6 +27,7 @@ #:use-module (ice-9 ftw) #:export (scheme-files scheme-modules+ scheme-modules* fold-modules all-modules fold-module-public-variables))@@ -115,6 +116,16 @@ name and the exception key and arguments." (string-append directory "/" sub-directory) directory)))) +(define* (scheme-modules* directory #:optional sub-directory)+ "Return the list of module names found under SUB-DIRECTORY in DIRECTORY.+This is a source-only variant that does not try to load files."+ (let ((prefix (string-length directory)))+ (map (lambda (file)+ (file-name->module-name (string-drop file prefix)))+ (scheme-files (if sub-directory+ (string-append directory "/" sub-directory)+ directory)))))+ (define* (fold-modules proc init path #:key (warn (const #f))) "Fold over all the Scheme modules present in PATH, a list of directories. Call (PROC MODULE RESULT) for each module that is found."diff --git a/guix/self.scm b/guix/self.scmindex 90649db17..81f9b0cfd 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -206,13 +206,6 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'." (local-file file #:recursive? #t))) (find-files (string-append directory "/" sub-directory) pred))) -(define (scheme-modules* directory sub-directory)- "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."- (let ((prefix (string-length directory)))- (map (lambda (file)- (file-name->module-name (string-drop file prefix)))- (scheme-files (string-append directory "/" sub-directory)))))- (define* (sub-directory item sub-directory) "Return SUB-DIRECTORY within ITEM, which may be a file name or a file-like object."-- 2.18.0
L
L
Ludovic Courtès wrote on 28 Aug 2018 17:17
[PATCH 2/3] Add (guix describe) and use it to initialize '%package-search-path'.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180828151751.4657-2-ludo@gnu.org
* guix/describe.scm: New file.* Makefile.am (MODULES): Add it.* gnu/packages.scm (%package-module-path): Honor 'package-path-entries'.--- Makefile.am | 1 + gnu/packages.scm | 22 ++++++++------ guix/describe.scm | 73 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 88 insertions(+), 8 deletions(-) create mode 100644 guix/describe.scm
Toggle diff (135 lines)diff --git a/Makefile.am b/Makefile.amindex 324674a60..b6efd6d62 100644--- a/Makefile.am+++ b/Makefile.am@@ -86,6 +86,7 @@ MODULES = \ guix/derivations.scm \ guix/grafts.scm \ guix/inferior.scm \+ guix/describe.scm \ guix/gnu-maintenance.scm \ guix/self.scm \ guix/upstream.scm \diff --git a/gnu/packages.scm b/gnu/packages.scmindex 7b954769e..e690d459f 100644--- a/gnu/packages.scm+++ b/gnu/packages.scm@@ -30,6 +30,7 @@ #:select ((package-name->name+version . hyphen-separated-name->name+version))) #:autoload (guix profiles) (packages->manifest)+ #:use-module (guix describe) #:use-module (ice-9 vlist) #:use-module (ice-9 match) #:use-module (srfi srfi-1)@@ -136,16 +137,21 @@ for system '~a'") ;; to narrow the search. (let* ((not-colon (char-set-complement (char-set #\:))) (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")- not-colon)))- ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.- (for-each (lambda (directory)- (set! %load-path (cons directory %load-path))- (set! %load-compiled-path- (cons directory %load-compiled-path)))- environment)+ not-colon))+ (channels (package-path-entries)))+ ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's+ ;; search path. For historical reasons, $GUIX_PACKAGE_PATH goes to the+ ;; front; channels go to the back so that they don't override Guix' own+ ;; modules.+ (set! %load-path+ (append environment %load-path channels))+ (set! %load-compiled-path+ (append environment %load-compiled-path channels)) (make-parameter- (append environment `((,%distro-root-directory . "gnu/packages"))))))+ (append environment+ `((,%distro-root-directory . "gnu/packages"))+ channels)))) (define %patch-path ;; Define it after '%package-module-path' so that '%load-path' contains userdiff --git a/guix/describe.scm b/guix/describe.scmnew file mode 100644index 000000000..3122a762f--- /dev/null+++ b/guix/describe.scm@@ -0,0 +1,73 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++(define-module (guix describe)+ #:use-module (guix memoization)+ #:use-module (guix profiles)+ #:use-module (srfi srfi-1)+ #:use-module (ice-9 match)+ #:export (package-path-entries))++;;; Commentary:+;;;+;;; This module provides supporting code to allow a Guix instance to find, at+;;; run time, which profile it's in (profiles created by 'guix pull'). That+;;; allows it to read meta-information about itself (e.g., repository URL and+;;; commit ID) and to find other channels available in the same profile. It's+;;; a bit like ELPA's pkg-info.el.+;;;+;;; Code:++(define current-profile+ (mlambda ()+ "Return the profile (created by 'guix pull') the calling process lives in,+or #f if this is not applicable."+ (match (command-line)+ ((program . _)+ (and (string-suffix? "/bin/guix" program)+ ;; Note: We want to do _lexical dot-dot resolution_. Using ".."+ ;; for real would instead take us into the /gnu/store directory+ ;; that ~/.config/guix/current/bin points to, whereas we want to+ ;; obtain ~/.config/guix/current.+ (let ((candidate (dirname (dirname program))))+ (and (file-exists? (string-append candidate "/manifest"))+ candidate)))))))++(define current-profile-entries+ (mlambda ()+ "Return the list of entries in the 'guix pull' profile the calling process+lives in, or #f if this is not applicable."+ (match (current-profile)+ (#f '())+ (profile+ (let ((manifest (profile-manifest profile)))+ (manifest-entries manifest))))))++(define package-path-entries+ (mlambda ()+ "Return a list of package path entries to be added to the package search+path. These entries are taken from the 'guix pull' profile the calling+process lives in, when applicable."+ ;; Filter out Guix itself.+ (filter-map (lambda (entry)+ (and (not (string=? (manifest-entry-name entry)+ "guix"))+ (string-append (manifest-entry-item entry)+ "/share/guile/site/"+ (effective-version))))+ (current-profile-entries))))-- 2.18.0
L
L
Ludovic Courtès wrote on 28 Aug 2018 17:17
[PATCH 3/3] DRAFT Add (guix channels) and use it in (guix scripts pull).
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180828151751.4657-3-ludo@gnu.org
DRAFT: Missing documentation for ~/.config/guix/channels.scm.
* guix/channels.scm: New file.* Makefile.am (MODULES): Add it.* guix/scripts/pull.scm: Use it.(%self-build-file, %pull-version, build-from-source)(whole-package-for-legacy, derivation->manifest-entry): Remove. Thesenow exist in a similar formin (guix channels).(build-and-install): Change 'source' to 'instances'. Remove #:url, #:branch, and #:commit. Rewrite using 'channel-instances->manifest'.(channel-list): New procedure.(guix-pull): Parameterize %REPOSITORY-CACHE-DIRECTORY. Call'honor-lets-encrypt-certificates!' unconditionally. Load~/.config/guix/channels.scm. Rewrite to use (guix channels).[use-le-certs?]: Remove.* po/guix/POTFILES.in: Add (guix channels).--- Makefile.am | 1 + guix/channels.scm | 292 ++++++++++++++++++++++++++++++++++++++++++ guix/scripts/pull.scm | 198 +++++++++++----------------- po/guix/POTFILES.in | 1 + 4 files changed, 372 insertions(+), 120 deletions(-) create mode 100644 guix/channels.scm
Toggle diff (607 lines)diff --git a/Makefile.am b/Makefile.amindex b6efd6d62..af6870cf6 100644--- a/Makefile.am+++ b/Makefile.am@@ -87,6 +87,7 @@ MODULES = \ guix/grafts.scm \ guix/inferior.scm \ guix/describe.scm \+ guix/channels.scm \ guix/gnu-maintenance.scm \ guix/self.scm \ guix/upstream.scm \diff --git a/guix/channels.scm b/guix/channels.scmnew file mode 100644index 000000000..ec3e05eaf--- /dev/null+++ b/guix/channels.scm@@ -0,0 +1,292 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++(define-module (guix channels)+ #:use-module (guix git)+ #:use-module (guix records)+ #:use-module (guix gexp)+ #:use-module (guix discovery)+ #:use-module (guix monads)+ #:use-module (guix profiles)+ #:use-module (guix derivations)+ #:use-module (guix store)+ #:use-module (guix i18n)+ #:use-module (srfi srfi-1)+ #:use-module (srfi srfi-9)+ #:use-module (srfi srfi-11)+ #:autoload (guix self) (whole-package)+ #:use-module (ice-9 match)+ #:export (channel+ channel?+ channel-name+ channel-url+ channel-branch+ channel-commit+ channel-location++ %default-channels++ channel-instance?+ channel-instance-channel+ channel-instance-commit+ channel-instance-checkout++ latest-channel-instances+ channel-instance-derivations+ latest-channel-derivations+ channel-instances->manifest))++;;; Commentary:+;;;+;;; This module implements "channels." A channel is usually a source of+;;; package definitions. There's a special channel, the 'guix' channel, that+;;; provides all of Guix, including its commands and its documentation.+;;; User-defined channels are expected to typically provide a bunch of .scm+;;; files meant to be added to the '%package-search-path'.+;;;+;;; This module provides tools to fetch and update channels from a Git+;;; repository and to build them.+;;;+;;; Code:++(define-record-type* <channel> channel make-channel+ channel?+ (name channel-name)+ (url channel-url)+ (branch channel-branch (default "master"))+ (commit channel-commit (default #f))+ (location channel-location+ (default (current-source-location)) (innate)))+;; TODO: Add a way to express dependencies among channels.++(define %default-channels+ ;; Default list of channels.+ (list (channel+ (name 'guix)+ (branch "origin/master")+ (url "https://git.savannah.gnu.org/git/guix.git"))))++(define (guix-channel? channel)+ "Return true if CHANNEL is the 'guix' channel."+ (eq? 'guix (channel-name channel)))++(define-record-type <channel-instance>+ (channel-instance channel commit checkout)+ channel-instance?+ (channel channel-instance-channel)+ (commit channel-instance-commit)+ (checkout channel-instance-checkout))++(define (channel-reference channel)+ "Return the \"reference\" for CHANNEL, an sexp suitable for+'latest-repository-commit'."+ (match (channel-commit channel)+ (#f `(branch . ,(channel-branch channel)))+ (commit `(commit . ,(channel-commit channel)))))++(define (latest-channel-instances store channels)+ "Return a list of channel instances corresponding to the latest checkouts of+CHANNELS."+ (map (lambda (channel)+ (format (current-error-port)+ (G_ "Updating channel '~a' from Git repository at '~a'...~%")+ (channel-name channel)+ (channel-url channel))+ (let-values (((checkout commit)+ (latest-repository-commit store (channel-url channel)+ #:ref (channel-reference+ channel))))+ (channel-instance channel commit checkout)))+ channels))++(define %self-build-file+ ;; The file containing code to build Guix. This serves the same purpose as+ ;; a makefile, and, similarly, is intended to always keep this name.+ "build-aux/build-self.scm")++(define %pull-version+ ;; This is the version of the 'guix pull' protocol. It specifies what's+ ;; expected from %SELF-BUILD-FILE. The initial version ("0") was when we'd+ ;; place a set of compiled Guile modules in ~/.config/guix/latest.+ 1)++(define (standard-module-derivation name source dependencies)+ "Return a derivation that builds the Scheme modules in SOURCE and that+depend on DEPENDENCIES, a list of lowerable objects. The assumption is that+SOURCE contains package modules to be added to '%package-module-path'."+ (define modules+ (scheme-modules* source))++ ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow+ ;; channel publishers to specify things such as the sub-directory where .scm+ ;; files live, files to exclude from the channel, preferred substitute URLs,+ ;; etc.+ (mlet* %store-monad ((compiled+ (compiled-modules modules+ #:name name+ #:module-path (list source)+ #:extensions dependencies)))++ (gexp->derivation name+ (with-extensions dependencies+ (with-imported-modules '((guix build utils))+ #~(begin+ (use-modules (guix build utils))++ (let ((go (string-append #$output "/lib/guile/"+ (effective-version)+ "/site-ccache"))+ (scm (string-append #$output+ "/share/guile/site/"+ (effective-version))))+ (mkdir-p (dirname go))+ (symlink #$compiled go)+ (mkdir-p (dirname scm))+ (symlink #$source scm))))))))++(define* (build-from-source name source+ #:key verbose? commit+ (dependencies '()))+ "Return a derivation to build Guix from SOURCE, using the self-build script+contained therein. Use COMMIT as the version string."+ ;; Running the self-build script makes it easier to update the build+ ;; procedure: the self-build script of the Guix-to-be-installed contains the+ ;; right dependencies, build procedure, etc., which the Guix-in-use may not+ ;; be know.+ (define script+ (string-append source "/" %self-build-file))++ (if (file-exists? script)+ (let ((build (save-module-excursion+ (lambda ()+ (primitive-load script)))))+ ;; BUILD must be a monadic procedure of at least one argument: the+ ;; source tree.+ ;;+ ;; Note: BUILD can return #f if it does not support %PULL-VERSION. In+ ;; the future we'll fall back to a previous version of the protocol+ ;; when that happens.+ (build source #:verbose? verbose? #:version commit+ #:pull-version %pull-version))++ ;; Build a set of modules that extend Guix using the standard method.+ (standard-module-derivation name source dependencies)))++(define* (build-channel-instance instance #:optional (dependencies '()))+ "Return, as a monadic value, the derivation for INSTANCE, a channel+instance. DEPENDENCIES is a list of extensions providing Guile modules that+INSTANCE depends on."+ (build-from-source (symbol->string+ (channel-name (channel-instance-channel instance)))+ (channel-instance-checkout instance)+ #:commit (channel-instance-commit instance)+ #:dependencies dependencies))++(define (channel-instance-derivations instances)+ "Return the list of derivations to build INSTANCES, in the same order as+INSTANCES."+ (define core-instance+ ;; The 'guix' channel is treated specially: it's an implicit dependency of+ ;; all the other channels.+ (find (lambda (instance)+ (guix-channel? (channel-instance-channel instance)))+ instances))++ (mlet %store-monad ((core (build-channel-instance core-instance)))+ (mapm %store-monad+ (lambda (instance)+ (if (eq? instance core-instance)+ (return core)+ (build-channel-instance instance+ (list core))))+ instances)))++(define latest-channel-derivations+ (let ((latest-channel-instances (store-lift latest-channel-instances)))+ (lambda (channels)+ "Return, as a monadic value, the list of derivations for the latest+instances of CHANNELS."+ (mlet %store-monad ((instances (latest-channel-instances channels)))+ (channel-instance-derivations instances)))))++(define (whole-package-for-legacy name modules)+ "Return a full-blown Guix package for MODULES, a derivation that builds Guix+modules in the old ~/.config/guix/latest style."+ (define packages+ (resolve-interface '(gnu packages guile)))++ (letrec-syntax ((list (syntax-rules (->)+ ((_)+ '())+ ((_ (module -> variable) rest ...)+ (cons (module-ref (resolve-interface+ '(gnu packages module))+ 'variable)+ (list rest ...)))+ ((_ variable rest ...)+ (cons (module-ref packages 'variable)+ (list rest ...))))))+ (whole-package name modules++ ;; In the "old style", %SELF-BUILD-FILE would simply return a+ ;; derivation that builds modules. We have to infer what the+ ;; dependencies of these modules were.+ (list guile-json guile-git guile-bytestructures+ (ssh -> guile-ssh) (tls -> gnutls)))))++(define (old-style-guix? drv)+ "Return true if DRV corresponds to a ~/.config/guix/latest style of+derivation."+ ;; Here we rely on a gross historical fact: that derivations produced by the+ ;; "old style" (before commit 8a0d9bc8a3f153159d9e239a151c0fa98f1e12d8,+ ;; dated May 30, 2018) did not depend on "guix-command.drv".+ (not (find (lambda (input)+ (string-suffix? "-guix-command.drv"+ (derivation-input-path input)))+ (derivation-inputs drv))))++(define (channel-instances->manifest instances)+ "Return a profile manifest with entries for all of INSTANCES, a list of+channel instances."+ (define instance->entry+ (match-lambda+ ((instance drv)+ (let ((commit (channel-instance-commit instance))+ (channel (channel-instance-channel instance)))+ (with-monad %store-monad+ (return (manifest-entry+ (name (symbol->string (channel-name channel)))+ (version (string-take commit 7))+ (item (if (guix-channel? channel)+ (if (old-style-guix? drv)+ (whole-package-for-legacy+ (string-append name "-" version)+ drv)+ drv)+ drv))+ (properties+ `((source (repository+ (version 0)+ (url ,(channel-url channel))+ (branch ,(channel-branch channel))+ (commit ,commit))))))))))))++ (mlet* %store-monad ((derivations (channel-instance-derivations instances))+ (entries (mapm %store-monad instance->entry+ (zip instances derivations))))+ (return (manifest entries))))diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scmindex ee68c21a4..d7931506b 100644--- a/guix/scripts/pull.scm+++ b/guix/scripts/pull.scm@@ -30,26 +30,19 @@ #:use-module (guix grafts) #:use-module (guix memoization) #:use-module (guix monads)+ #:use-module (guix channels) #:autoload (guix inferior) (open-inferior) #:use-module (guix scripts build)- #:autoload (guix self) (whole-package) #:use-module (guix git) #:use-module (git) #:use-module (gnu packages)- #:autoload (gnu packages ssh) (guile-ssh)- #:autoload (gnu packages tls) (gnutls) #:use-module ((guix scripts package) #:select (build-and-use-profile))- #:use-module ((guix build utils)- #:select (with-directory-excursion delete-file-recursively))- #:use-module ((guix build download)- #:select (%x509-certificate-directory)) #:use-module (gnu packages base) #:use-module (gnu packages guile) #:use-module ((gnu packages bootstrap) #:select (%bootstrap-guile)) #:use-module ((gnu packages certs) #:select (le-certs)) #:use-module (srfi srfi-1)- #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-35) #:use-module (srfi srfi-37)@@ -57,9 +50,6 @@ #:use-module (ice-9 vlist) #:export (guix-pull)) -(define %repository-url- (or (getenv "GUIX_PULL_URL") "https://git.savannah.gnu.org/git/guix.git"))- ;;; ;;; Command-line options.@@ -67,9 +57,7 @@ (define %default-options ;; Alist of default option values.- `((repository-url . ,%repository-url)- (ref . (branch . "origin/master"))- (system . ,(%current-system))+ `((system . ,(%current-system)) (substitutes? . #t) (build-hook? . #t) (graft? . #t)@@ -142,70 +130,6 @@ Download and deploy the latest version of Guix.\n")) (define indirect-root-added (store-lift add-indirect-root)) -(define %self-build-file- ;; The file containing code to build Guix. This serves the same purpose as- ;; a makefile, and, similarly, is intended to always keep this name.- "build-aux/build-self.scm")--(define %pull-version- ;; This is the version of the 'guix pull' protocol. It specifies what's- ;; expected from %SELF-BUILD-FILE. The initial version ("0") was when we'd- ;; place a set of compiled Guile modules in ~/.config/guix/latest.- 1)--(define* (build-from-source source- #:key verbose? commit)- "Return a derivation to build Guix from SOURCE, using the self-build script-contained therein. Use COMMIT as the version string."- ;; Running the self-build script makes it easier to update the build- ;; procedure: the self-build script of the Guix-to-be-installed contains the- ;; right dependencies, build procedure, etc., which the Guix-in-use may not- ;; be know.- (let* ((script (string-append source "/" %self-build-file))- (build (primitive-load script)))- ;; BUILD must be a monadic procedure of at least one argument: the source- ;; tree.- ;;- ;; Note: BUILD can return #f if it does not support %PULL-VERSION. In the- ;; future we'll fall back to a previous version of the protocol when that- ;; happens.- (build source #:verbose? verbose? #:version commit- #:pull-version %pull-version)))--(define (whole-package-for-legacy name modules)- "Return a full-blown Guix package for MODULES, a derivation that builds Guix-modules in the old ~/.config/guix/latest style."- (whole-package name modules-- ;; In the "old style", %SELF-BUILD-FILE would simply return a- ;; derivation that builds modules. We have to infer what the- ;; dependencies of these modules were.- (list guile-json guile-git guile-bytestructures- guile-ssh gnutls)))--(define* (derivation->manifest-entry drv- #:key url branch commit)- "Return a manifest entry for DRV, which represents Guix at COMMIT. Record-URL, BRANCH, and COMMIT as a property in the manifest entry."- (mbegin %store-monad- (what-to-build (list drv))- (built-derivations (list drv))- (let ((out (derivation->output-path drv)))- (return (manifest-entry- (name "guix")- (version (string-take commit 7))- (item (if (file-exists? (string-append out "/bin/guix"))- drv- (whole-package-for-legacy (string-append name "-"- version)- drv)))- (properties- `((source (repository- (version 0)- (url ,url)- (branch ,branch)- (commit ,commit))))))))))- (define (display-profile-news profile) "Display what's up in PROFILE--new packages, and all that." (match (memv (generation-number profile)@@ -223,8 +147,8 @@ URL, BRANCH, and COMMIT as a property in the manifest entry." #:heading (G_ "New in this revision:\n")))) (_ #t))) -(define* (build-and-install source config-dir- #:key verbose? url branch commit)+(define* (build-and-install instances config-dir+ #:key verbose?) "Build the tool from SOURCE, and install it in CONFIG-DIR." (define update-profile (store-lift build-and-use-profile))@@ -232,15 +156,9 @@ URL, BRANCH, and COMMIT as a property in the manifest entry." (define profile (string-append config-dir "/current")) - (mlet* %store-monad ((drv (build-from-source source- #:commit commit- #:verbose? verbose?))- (entry (derivation->manifest-entry drv- #:url url- #:branch branch- #:commit commit)))+ (mlet %store-monad ((manifest (channel-instances->manifest instances))) (mbegin %store-monad- (update-profile profile (manifest (list entry)))+ (update-profile profile manifest) (return (display-profile-news profile))))) (define (honor-lets-encrypt-certificates! store)@@ -426,45 +344,91 @@ and ALIST2 differ, display HEADING upfront." ((numbers ...) (list-generations profile numbers))))))))) +(define (channel-list file opts)+ "Return the list of channels to use. If FILE exists, channels are read from+there; otherwise %DEFAULT-CHANNELS is used. Apply channel transformations+specified in OPTS (resulting from '--url', '--commit', or '--branch'), if+any."+ (define channels+ (if (file-exists? file)+ (load* file (make-user-module '((guix channels))))+ %default-channels))++ (define (environment-variable)+ (match (getenv "GUIX_PULL_URL")+ (#f #f)+ (url+ (warning (G_ "The 'GUIX_PULL_URL' environment variable is deprecated.+Use '~/.config/guix/channels.scm' instead."))+ url)))++ (let ((ref (assoc-ref opts 'ref))+ (url (or (assoc-ref opts 'url)+ (environment-variable))))+ (if (or ref url)+ (match channels+ ((one)+ ;; When there's only one channel, apply '--url', '--commit', and+ ;; '--branch' to this specific channel.+ (let ((url (or url (channel-url one))))+ (list (match ref+ (('commit . commit)+ (channel (inherit one)+ (url url) (commit commit) (branch #f)))+ (('branch . branch)+ (channel (inherit one)+ (url url) (commit #f) (branch branch)))+ (#f+ (channel (inherit one) (url url)))))))+ (_+ ;; Otherwise bail out.+ (leave+ (G_ "'--url', '--commit', and '--branch' are not applicable~%"))))+ channels)))+ (define (guix-pull . args)- (define (use-le-certs? url)- (string-prefix? "https://git.savannah.gnu.org/" url))- (with-error-handling (with-git-error-handling- (let* ((opts (parse-command-line args %options- (list %default-options)))- (url (assoc-ref opts 'repository-url))- (ref (assoc-ref opts 'ref))- (cache (string-append (cache-directory) "/pull")))+ (let* ((opts (parse-command-line args %options+ (list %default-options)))+ (cache (string-append (cache-directory) "/pull"))+ (channel-file (string-append (config-directory) "/channels.scm"))+ (channels (channel-list channel-file opts)))+ (cond ((assoc-ref opts 'query) (process-query opts)) ((assoc-ref opts 'dry-run?) #t) ;XXX: not very useful (else (with-store store- (parameterize ((%graft? (assoc-ref opts 'graft?)))+ (parameterize ((%graft? (assoc-ref opts 'graft?))+ (%repository-cache-directory cache)) (set-build-options-from-command-line store opts) - ;; For reproducibility, always refer to the LE certificates- ;; when we know we're talking to Savannah.- (when (use-le-certs? url)- (honor-lets-encrypt-certificates! store))-- (format (current-error-port)- (G_ "Updating from Git repository at '~a'...~%")- url)-- (let-values (((checkout commit)- (latest-repository-commit store url- #:ref ref- #:cache-directory- cache)))+ ;; When certificates are already installed, use them.+ ;; Otherwise, use the Let's Encrypt certificates, which we+ ;; know Savannah uses.+ (let ((certs (or (getenv "SSL_CERT_DIR") "/etc/ssl/certs")))+ (unless (file-exists? certs)+ (honor-lets-encrypt-certificates! store))) + (let ((instances (latest-channel-instances store channels))) (format (current-error-port)- (G_ "Building from Git commit ~a...~%")- commit)+ (N_ "Building from this channel:~%"+ "Building from these channels:~%"+ (length instances)))+ (for-each (lambda (instance)+ (let ((channel+ (channel-instance-channel instance)))+ (format (current-error-port)+ " ~10a~a\t~a~%"+ (channel-name channel)+ (channel-url channel)+ (string-take+ (channel-instance-commit instance)+ 7))))+ instances) (parameterize ((%guile-for-build (package-derivation store@@ -472,13 +436,7 @@ and ALIST2 differ, display HEADING upfront." %bootstrap-guile (canonical-package guile-2.2))))) (run-with-store store- (build-and-install checkout (config-directory)- #:url url- #:branch (match ref- (('branch . branch)- branch)- (_ #f))- #:commit commit+ (build-and-install instances (config-directory) #:verbose? (assoc-ref opts 'verbose?))))))))))))) diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.inindex d11f408d4..7f881355e 100644--- a/po/guix/POTFILES.in+++ b/po/guix/POTFILES.in@@ -38,4 +38,5 @@ guix/upstream.scm guix/ui.scm guix/http-client.scm guix/nar.scm+guix/channels.scm nix/nix-daemon/guix-daemon.cc-- 2.18.0
P
P
Pjotr Prins wrote on 28 Aug 2018 19:24
Re: bug#22629: Channels!
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
20180828172402.el7y6quy5hjgzua6@thebird.nl
Ludo, I am so ready to try this!
GUIX_PACKAGE_PATH is killing me though it got a lot better afterlatest guix pull developments.
Tell me what to run.
Pj.
M
M
Mark H Weaver wrote on 28 Aug 2018 21:52
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
877ekagtg9.fsf@netris.org
Hi Ludovic,
ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (3 lines)> Currently third-party channels are expected to provide nothing but> package modules.
I'd like to say again that I have grave concerns that this could be thedeath-knell for long-term innovation in Guix. It's likely that whenevera change is proposed that will break these third-party channels, therewill be resistance, and efforts to preserve backward compatibility.
Even things as seemingly innocuous as moving a package from one moduleto another will impact these third-party channels, not to mentionchanging our internal APIs or making fundamental changes to the waypackages are specified.
Part of why I'm so interested in Guix is because it currently has nearlyunconstrained potential to grow into something far more beautiful andelegant than it is today.
I fear that with the introduction of channels, that potential will bedrastically curtailed, and that we're essentially trading our futurepotential for what will in practice, most likely, be primarily used tofacilitate the use of non-free software on Guix.
When I start to see signs of resistance to changes for the sake ofthird-party channels, then I'll know I was right to be fearful, andGuix will become far less interesting to me.
Mark
L
L
Ludovic Courtès wrote on 28 Aug 2018 23:52
(name . Mark H Weaver)(address . mhw@netris.org)
87o9dmb1m2.fsf@gnu.org
Hi Mark,
Mark H Weaver <mhw@netris.org> skribis:
Toggle quote (5 lines)> I'd like to say again that I have grave concerns that this could be the> death-knell for long-term innovation in Guix. It's likely that whenever> a change is proposed that will break these third-party channels, there> will be resistance, and efforts to preserve backward compatibility.
It is a risk and I acknowledge that.
In the 4 years of existence of GUIX_PACKAGE_PATH, we’ve neverencountered this issue. To what extent would that change? I don’t know.
I want to preserve the existing incentives to work in Guix proper, or towork closely with the project when integration in Guix proper makeslittle sense—for WIP packages, custom package variants, ordomain-specific packages like those I mentioned in my message. I do notsee ourselves advertising channels as the primary way to add packages.
BTW, Andy’s Potluck offered a solution to the compatibility issue that’sstill on the table: https://bugs.gnu.org/26645.
Toggle quote (5 lines)> I fear that with the introduction of channels, that potential will be> drastically curtailed, and that we're essentially trading our future> potential for what will in practice, most likely, be primarily used to> facilitate the use of non-free software on Guix.
Non-free software will be one use case. There are other use casesthough, as mentioned above; these are the ones I’m interested in.
It’s a dilemma. I’d like to think that we can have our cake and eat ittoo; that we can give users this flexibility (which is “built in” sinceit’s just a matter of setting the load path) without jeopardizingdevelopment of Guix itself, and without creating the conditions for theproliferation of non-free package channels. Whether it works this waywould depend not on the technical details but on the group’s behavior.
Thoughts?
Thanks,Ludo’.
K
K
Konrad Hinsen wrote on 29 Aug 2018 06:09
(address . 22629@debbugs.gnu.org)
de4846c1-42c5-dd76-b87e-8a2ea2d4c4c5@fastmail.net
Hi Mark,
Toggle quote (1 lines)> I'd like to say again that I have grave concerns that this could be the > death-knell for long-term innovation in Guix. It's likely that
whenever> a change is proposed that will break these third-party channels, there> will be resistance, and efforts to preserve backward compatibility.I understand your point, but the problem you mention is, in my opinion, not so much due to channels but due to different priorities of different users. Which means that it will come up even without channels as the Guix user base grows.
Look at the wider Linux world: there are people who want to live on the bleeding edge and run Arch Linux, and there are others who value stability and run CentOS. Today's Guix is more on the bleeding edge side. My understanding of your commment is that you would like to make sure it stays there. But that also means severely limiting Guix' potential user base.
I see channels as an opportunity to have Guix "dialects" addressing different needs and yet remain interoperable, although I am the first to admit that I have no clear idea of how this would work out in practice, more from the social than the technical point of view. But the goal looks very attractive. Looking at my own use of Guix, I am happy with its bleeding edge approach for the software I use for research, but I'd much prefer a slower pace and more stability for stuff like Emacs and TeX that are "boring infrastructure" for me.
Toggle quote (5 lines)> Even things as seemingly innocuous as moving a package from one module> to another will impact these third-party channels, not to mention> changing our internal APIs or making fundamental changes to the way> packages are specified.
So... could we reduce the dependence of package specifications on such things? Package definitions use a small DSL that could be versioned, allowing change while maintaining compatibility. Module dependencies are more annoying, but do we need them? Package definitions are grouped into modules mostly for convenience. All packages have globally unique names, so could we use those to specify inputs?
Konrad.
A
A
Alex Sassmannshausen wrote on 29 Aug 2018 11:29
(name . Mark H Weaver)(address . mhw@netris.org)
87zhx5msfl.fsf@pompo.co
Mark H Weaver writes:
Toggle quote (11 lines)> Hi Ludovic,>> ludo@gnu.org (Ludovic Courtès) writes:>> Currently third-party channels are expected to provide nothing but>> package modules.>> I'd like to say again that I have grave concerns that this could be the> death-knell for long-term innovation in Guix. It's likely that whenever> a change is proposed that will break these third-party channels, there> will be resistance, and efforts to preserve backward compatibility.
I understand your concerns and want to acknowledge those.
My primary interest in channels at the moment comes from believing thathaving a "stable" channel would be incredibly useful to increaseadoption rate of Guix. And for me.
Currently upgrading my system involves doing a guix pull, then, over thecourse of a few days, doing guix package -u and bailing out if I startbuilding a large program.
After this I do guix system build, and bail out if a large programstarts building.
In either case, if an upgrade broke a dependency then I'm kind of stuckat the old versions of my profile.
Finally, when I've upgrade profile and system, I immediately run guixpull to prepare for the next cycle.
I consider myself pretty capable, and I find this process stressful — Icertainly cannot envisage most of my currently interested friends goingthrough this process…
But like I say, this is not to discount your concerns, it is merely toadd to the list of reasons why channels might be important.
Best wishes,
Alex

Toggle quote (19 lines)> Even things as seemingly innocuous as moving a package from one module> to another will impact these third-party channels, not to mention> changing our internal APIs or making fundamental changes to the way> packages are specified.>> Part of why I'm so interested in Guix is because it currently has nearly> unconstrained potential to grow into something far more beautiful and> elegant than it is today.>> I fear that with the introduction of channels, that potential will be> drastically curtailed, and that we're essentially trading our future> potential for what will in practice, most likely, be primarily used to> facilitate the use of non-free software on Guix.>> When I start to see signs of resistance to changes for the sake of> third-party channels, then I'll know I was right to be fearful, and> Guix will become far less interesting to me.>> Mark
L
L
Ludovic Courtès wrote on 29 Aug 2018 16:25
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
87efeh9rm8.fsf@gnu.org
Hi Konrad,
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (7 lines)> Look at the wider Linux world: there are people who want to live on> the bleeding edge and run Arch Linux, and there are others who value> stability and run CentOS. Today's Guix is more on the bleeding edge> side. My understanding of your commment is that you would like to make> sure it stays there. But that also means severely limiting Guix'> potential user base.
Mark’s concern is not about whether packages are the latest version,etc. It’s about the constraints that could result from widespreaddevelopment of channels outside Guix proper: technically all of Guix isjust a library, so widespread development of external channels couldforce us Guix developers to keep the API stable. This, in turn, wouldlimit our ability to make significant changes to Guix.
Toggle quote (9 lines)> I see channels as an opportunity to have Guix "dialects" addressing> different needs and yet remain interoperable, although I am the first> to admit that I have no clear idea of how this would work out in> practice, more from the social than the technical point of view. But> the goal looks very attractive. Looking at my own use of Guix, I am> happy with its bleeding edge approach for the software I use for> research, but I'd much prefer a slower pace and more stability for> stuff like Emacs and TeX that are "boring infrastructure" for me.
“Dialect” sounds a bit strong, but I agree that such uses could beinteresting and beneficial, to users and to Guix.
Toggle quote (12 lines)>> Even things as seemingly innocuous as moving a package from one module>> to another will impact these third-party channels, not to mention>> changing our internal APIs or making fundamental changes to the way>> packages are specified.>> So... could we reduce the dependence of package specifications on such> things? Package definitions use a small DSL that could be versioned,> allowing change while maintaining compatibility. Module dependencies> are more annoying, but do we need them? Package definitions are> grouped into modules mostly for convenience. All packages have> globally unique names, so could we use those to specify inputs?
This is exactly that kind of issue Mark is concerned with: currently wedon’t have to worry at all about this, and it’s great to have thatfreedom.
I’d rather not build fancy mechanisms just for the sake of externalchannels, and I certainly don’t want to commit to API stability. Wewon’t break the API every day intentionally either, but my point is thatsupport for external channels will be “best effort” (as it already iswith GUIX_PACKAGE_PATH). I think it’s useful and practical nonetheless.
Ludo’.
K
K
Konrad Hinsen wrote on 29 Aug 2018 17:30
(name . Ludovic Courtès)(address . ludo@gnu.org)
m15zzt42d5.fsf@ordinateur-de-catherine--konrad.home
Hi Ludo,
Toggle quote (4 lines)> Mark’s concern is not about whether packages are the latest version,> etc. It’s about the constraints that could result from widespread> development of channels outside Guix proper: technically all of Guix is
That's how I understood it as well. If/when Guix becomes somebody else'sdependency, then there will be pressure on stability in Guix itself.
My point is that this will happen anyway if Guix is adopted more widely.Every manifest file, personal or shared as part of a software package("guix.scm"), relies on the same technical details as achannel. Introducing channels only makes the issue more visible.
And this is really the same issue as with the stability of the packagesthemselves, Guix being a kind of superpackage. Most people want agilityfor the software layer they are most concerned with, and stability forall layers below it. For Mark (and certainly others here), Guix happensto be the layer they are most concerned with.
Toggle quote (3 lines)> I’d rather not build fancy mechanisms just for the sake of external> channels, and I certainly don’t want to commit to API stability. We
At this point, certainly not. But I agree with Mark that, if channels"take off", there will be pressure in that direction.
Konrad.
M
M
Mark H Weaver wrote on 29 Aug 2018 19:14
Re: bug#22629: Channels not needed for a stable branch (was: Channels!)
87lg8pccys.fsf_-_@netris.org
Hi,
Alex Sassmannshausen <alex@pompo.co> writes:
Toggle quote (4 lines)> My primary interest in channels at the moment comes from believing that> having a "stable" channel would be incredibly useful to increase> adoption rate of Guix. And for me.
Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
Toggle quote (7 lines)> Look at the wider Linux world: there are people who want to live on> the bleeding edge and run Arch Linux, and there are others who value> stability and run CentOS. Today's Guix is more on the bleeding edge> side. My understanding of your commment is that you would like to make> sure it stays there. But that also means severely limiting Guix'> potential user base.
Both of you seem to have reached the conclusion that third-partychannels are a prerequisite for having a 'stable' branch. I disagree.
I agree with both of you that a 'stable' branch of Guix would betremendously useful. I've often wanted it myself, and I still do.
My point is that I want to keep our APIs internal and unfrozen for thesame reason that Linux, the kernel project, does. Linux refuses tosupport out-of-tree drivers and modules, and thereby retains its freedomto change their internal APIs. Often they change how things workinternally and this entails doing massive find-replace on every driverin the tree. This has been a crucially important factor in theirlong-term success.
Does this stop the Linux developers from offering stable branches, orthird-parties from maintaining older stable versions of Linux in theirown Git repositories? Clearly not. It's been done all along.
We should persue a similar model. The crucial thing is to always keepthe package modules together with the rest of Guix. If you want toclone Guix to a third-party repository, I have no problem with that.Just keep it all together, and maintain the entire tree as an undividedwhole.
"guix pull" already supports the ability to fetch from an arbitrarybranch, or even from an arbitrary upstream source, so we already havewhat we need to start a 'stable' branch, or for a third-party to do thatif they wanted to.
Regards, Mark
R
R
Ricardo Wurmus wrote on 29 Aug 2018 20:26
(name . Mark H Weaver)(address . mhw@netris.org)
87zhx59gh3.fsf@elephly.net
Hi Mark,
Toggle quote (5 lines)> I'd like to say again that I have grave concerns that this could be the> death-knell for long-term innovation in Guix. It's likely that whenever> a change is proposed that will break these third-party channels, there> will be resistance, and efforts to preserve backward compatibility.
GUIX_PACKAGE_PATH already had that same problem (and did not provide asolution for it). With channels we can at least add more informationabout a collection of modules, e.g. what version of Guix it is known towork with. So channels really flesh out the feature provided byGUIX_PACKAGE_PATH, elevating it from a simple environment variable toone that can take additional context into account.
I think that’s a worthwhile step to take.
I agree with your sentiment that a mechanism based on a simpleenvironment variable does not instill confidence, whereas a specialmechanism like channels could signal to users that it’s a feature thatprovides some guarantees. But I disagree with your assertion that thiswould be “a death-knell to innovation”. That seems like an exaggerationto me.
Toggle quote (7 lines)> My point is that I want to keep our APIs internal and unfrozen for the> same reason that Linux, the kernel project, does. Linux refuses to> support out-of-tree drivers and modules, and thereby retains its freedom> to change their internal APIs. Often they change how things work> internally and this entails doing massive find-replace on every driver> in the tree. This has been a crucially important factor in their> long-term success.
[…]
Toggle quote (3 lines)> We should persue a similar model. The crucial thing is to always keep> the package modules together with the rest of Guix.
I agree. That is and remains our recommendation. I still want mostpackages to end up in Guix proper. There are collections of packagesfor which this does not make sense, though, and I think that it isbetter to have a more formal mechanism that can also be used to describethe limits of compatibility than just a simple environment variable.
I also agree with you that we don’t need channels for providing a stablebranch. The biggest obstacle to providing a stable branch is nottechnical, but it requires people maintaining it.
--Ricardo
L
L
Ludovic Courtès wrote on 29 Aug 2018 22:50
Re: bug#22629: Channels!
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
87efeg99t6.fsf@gnu.org
Hello Konrad,
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (12 lines)>> Mark’s concern is not about whether packages are the latest version,>> etc. It’s about the constraints that could result from widespread>> development of channels outside Guix proper: technically all of Guix is>> That's how I understood it as well. If/when Guix becomes somebody else's> dependency, then there will be pressure on stability in Guix itself.>> My point is that this will happen anyway if Guix is adopted more widely.> Every manifest file, personal or shared as part of a software package> ("guix.scm"), relies on the same technical details as a> channel. Introducing channels only makes the issue more visible.
True. Manifests can rely on fewer details of the API than a channelthough, particularly if they use ‘specifications->manifests’.
(BTW code in channels could use ‘specification->package’ as well toincrease decoupling a bit.)
Toggle quote (6 lines)> And this is really the same issue as with the stability of the packages> themselves, Guix being a kind of superpackage. Most people want agility> for the software layer they are most concerned with, and stability for> all layers below it. For Mark (and certainly others here), Guix happens> to be the layer they are most concerned with.
Yeah.
Thanks,Ludo’.
L
L
Ludovic Courtès wrote on 29 Aug 2018 23:02
Re: bug#22629: Channels not needed for a stable branch
(name . Mark H Weaver)(address . mhw@netris.org)
87mut47upf.fsf@gnu.org
Hello Guix,
Mark H Weaver <mhw@netris.org> skribis:
Toggle quote (3 lines)> Both of you seem to have reached the conclusion that third-party> channels are a prerequisite for having a 'stable' branch. I disagree.
Same here. We could already be doing that (I’m skeptical about thefeasibility, maintainability, and relevance of a “stable” branch in thesense of Debian stable, but that’s another story.)
Toggle quote (11 lines)> I agree with both of you that a 'stable' branch of Guix would be> tremendously useful. I've often wanted it myself, and I still do.>> My point is that I want to keep our APIs internal and unfrozen for the> same reason that Linux, the kernel project, does. Linux refuses to> support out-of-tree drivers and modules, and thereby retains its freedom> to change their internal APIs. Often they change how things work> internally and this entails doing massive find-replace on every driver> in the tree. This has been a crucially important factor in their> long-term success.
I had this example in mind too: the kernel technically supportsout-of-tree modules, but kernel developers have always resisted pressureto freeze APIs.
Because this policy has been stated upfront very clearly and hasremained unchanged, out-of-tree module developers know that that thecompatibility burden is on them. There’s flexibility, along with astrong incentive to get things in the mainline kernel.
This is the outcome I’d like to achieve: give users some welcomeflexibility, but make it clear that it’s best-effort.
Ludo’.
K
K
Konrad Hinsen wrote on 30 Aug 2018 07:57
Re: bug#22629: Channels not needed for a stable branch (was: Channels!)
m1wos82y70.fsf@ordinateur-de-catherine--konrad.home
Hi Ricardo,
Toggle quote (4 lines)> I also agree with you that we don’t need channels for providing a stable> branch. The biggest obstacle to providing a stable branch is not> technical, but it requires people maintaining it.
Look at this from the opposite end: if you were interested inmaintaining a stable software distribution, would you choose a quicklyevolving package manager such as Guix as the basis? I'd say no, and I amspeaking from experience because I did actually maintain stable softwareinstallations for a couple of years. You want to concentrate on criticalbug fixes and avoid anything else that could perturb the stability ofyour system.
For me this whole discussion isn't about technicalities. Channels arejust a convenience layer on top of what can already be done withGUIX_PACKAGE_PATH and manifest files. The real issue is how suchfeatures are advertised: as a way to bolt on your own developments,following Guix evolution closely, or as a way to provide decoupledautonomous branches such as stable software.
So I agree with Ludo that a communication along the lines of
Toggle quote (9 lines)> I had this example in mind too: the kernel technically supports> out-of-tree modules, but kernel developers have always resisted pressure> to freeze APIs.>> Because this policy has been stated upfront very clearly and has> remained unchanged, out-of-tree module developers know that that the> compatibility burden is on them. There’s flexibility, along with a> strong incentive to get things in the mainline kernel.
should be sufficient to avoid the issue that Mark has raised here.But it also discourages people interested in stable softwareinstallations from using or contributing to Guix.
Konrad.
M
M
Mark H Weaver wrote on 30 Aug 2018 08:42
Re: bug#22629: Channels not needed for a stable branch
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
875zzs9wzl.fsf@netris.org
Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
Toggle quote (12 lines)>> I also agree with you that we don’t need channels for providing a stable>> branch. The biggest obstacle to providing a stable branch is not>> technical, but it requires people maintaining it.>> Look at this from the opposite end: if you were interested in> maintaining a stable software distribution, would you choose a quickly> evolving package manager such as Guix as the basis? I'd say no, and I am> speaking from experience because I did actually maintain stable software> installations for a couple of years. You want to concentrate on critical> bug fixes and avoid anything else that could perturb the stability of> your system.
I'm not sure what you're trying to argue above. To me, it looks like anargument in favor of my position, namely that a stable version of Guixshould include _all_ of Guix, not just the packages.
If you want to maintain a stable distribution, why would you want tocombine stable package descriptions with quickly evolving infrastructurethat makes up the rest of Guix?
If you want stability, wouldn't it be better to keep a stable branch of_all_ of Guix, so that both the package descriptions _and_ theinfrastructure upon which those packages depend are kept stable?
This is exactly what a 'stable' branch in a git repository wouldprovide, and moreover "guix pull" already has everything that's neededto support this. All that's needed are people to maintain such abranch.
If we again compare this to Linux (the kernel project) and their refusalto support out-of-tree drivers: I'm arguing in favor of the Linuxapproach, where stable kernels are based on git branches that include a_all_ of Linux. Your position, transposed to Linux, would seem to be infavor of third-parties supporting stable drivers, which would then becombined with the latest version of the rest of the kernel.
Am I misunderstanding your argument? Can you please clarify?
Mark
K
K
Konrad Hinsen wrote on 30 Aug 2018 12:10
(name . Mark H Weaver)(address . mhw@netris.org)
m1d0u0qi4v.fsf@fastmail.net
Hi Mark,
Toggle quote (4 lines)> I'm not sure what you're trying to argue above. To me, it looks like an> argument in favor of my position, namely that a stable version of Guix> should include _all_ of Guix, not just the packages.
All, probably not, some, probably yes. What I am arguing is that theproductive coexistence of a stable version with the bleeding-edgeversion requires agreement on a stable foundation. Where exactly theborderline lies between this foundation and what is built on top of itis not a question I am sufficiently qualified to answer.
The minimal stable foundation would have to include the file systemlayout of profiles, to make sure that users can mix packages from bothversions safely. It would also be highly desirable to share the store,whose layout would then have to be part of the foundation as well.
Moreover, I suspect it would be preferable or even necessary to haveonly one daemon running - if that's true, then the daemon'scommunication protocol would have be part of the foundation as well.
Without a common foundation, a stable version would have to be acompletely autonomous fork, which should then probably adopt a differentname as well. I don't think this is desirable, in particular for GuixSDwhich would lose most of its interest if it required multiple packagemanagers.
Konrad.
L
L
Ludovic Courtès wrote on 30 Aug 2018 14:18
“Stable” branch
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
874lfcxd2v.fsf_-_@gnu.org
Hi Konrad,
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (15 lines)> The minimal stable foundation would have to include the file system> layout of profiles, to make sure that users can mix packages from both> versions safely. It would also be highly desirable to share the store,> whose layout would then have to be part of the foundation as well.>> Moreover, I suspect it would be preferable or even necessary to have> only one daemon running - if that's true, then the daemon's> communication protocol would have be part of the foundation as well.>> Without a common foundation, a stable version would have to be a> completely autonomous fork, which should then probably adopt a different> name as well. I don't think this is desirable, in particular for GuixSD> which would lose most of its interest if it required multiple package> managers.
These are all things that very rarely, if ever, changed over the last 5years. I expect the change rate to remain the same. :-)
You seem to be arguing of a “stable” branch in the sense that the Guixtools (the CLI in particular) wouldn’t change much, is that correct?
I’m asking because there are several ways to define “stable.” InitiallyI thought what you had in mind was like the “stable” branch in Debian,meaning that packages only get security updates. To me that’s adifferent thing.
Ludo’.
A
A
Alex Sassmannshausen wrote on 30 Aug 2018 16:10
(name . Ludovic Courtès)(address . ludo@gnu.org)
87wos8lzcj.fsf@pompo.co
Ludovic Courtès writes:
Toggle quote (30 lines)> Hi Konrad,>> Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:>>> The minimal stable foundation would have to include the file system>> layout of profiles, to make sure that users can mix packages from both>> versions safely. It would also be highly desirable to share the store,>> whose layout would then have to be part of the foundation as well.>>>> Moreover, I suspect it would be preferable or even necessary to have>> only one daemon running - if that's true, then the daemon's>> communication protocol would have be part of the foundation as well.>>>> Without a common foundation, a stable version would have to be a>> completely autonomous fork, which should then probably adopt a different>> name as well. I don't think this is desirable, in particular for GuixSD>> which would lose most of its interest if it required multiple package>> managers.>> These are all things that very rarely, if ever, changed over the last 5> years. I expect the change rate to remain the same. :-)>> You seem to be arguing of a “stable” branch in the sense that the Guix> tools (the CLI in particular) wouldn’t change much, is that correct?>> I’m asking because there are several ways to define “stable.” Initially> I thought what you had in mind was like the “stable” branch in Debian,> meaning that packages only get security updates. To me that’s a> different thing.
I don't know if this is what Konrad desires, but from my perspective, adesirable part of the definition of stable would be a that the buildfarms have produced a set of binaries/substitutes for a given Guixrevision that is "good enough".
Where good enough might mean something like "no more build failures thanthe previous iteration", or "no more build failures on common desktopapplications" or other some exciting metric (the exact definition is notreally important right now).
The main concern here would be that we try to avoid end users fromhaving unexpected build failures or even missing substitutes after doinga guix pull.
Alex
K
K
Konrad Hinsen wrote on 30 Aug 2018 16:46
Re: “Stable” branch
m1va7rc3os.fsf@fastmail.net
Hi Ludo and Alex,

ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (3 lines)> These are all things that very rarely, if ever, changed over the last 5> years. I expect the change rate to remain the same. :-)
That's reassuring!
Toggle quote (8 lines)> You seem to be arguing of a “stable” branch in the sense that the Guix> tools (the CLI in particular) wouldn’t change much, is that correct?>> I’m asking because there are several ways to define “stable.” Initially> I thought what you had in mind was like the “stable” branch in Debian,> meaning that packages only get security updates. To me that’s a> different thing.
What I have in mind is whatever it takes to build a stable softwaresystem. That includes stable ingredients (packages) but also stableglue, meaning the package definitions and the build system that producesthe binaries from them. Stability of the Guix CLI is much less relevantfrom my point of view.

Alex Sassmannshausen <alex@pompo.co> writes:
Toggle quote (5 lines)> I don't know if this is what Konrad desires, but from my perspective, a> desirable part of the definition of stable would be a that the build> farms have produced a set of binaries/substitutes for a given Guix> revision that is "good enough".
That's another very practically relevant notion of stability. Mine goesbeyond that though. For example, I'd require all packages to build andpass tests at all times.
Konrad.
L
L
Ludovic Courtès wrote on 30 Aug 2018 23:29
Re: bug#22629: Channels!
(address . 22629@debbugs.gnu.org)
87lg8nr1ab.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (10 lines)> The patches that follow implement this last bit, though in a slightly> different way. Users would now have the option to provide> ~/.config/guix/channels.scm along these lines:>> (cons (channel> (name 'guix-hpc)> (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")> (branch "origin/master"))> %default-channels)
What follows is version 2 of the patches, which I’d like to push withina day or two. The main changes are:
1. Added documentation for channels and ~/.config/guix/channels.scm. In particular I added a warning about compatibility that summarizes what we discussed. Please let me know what you think.
2. Added -C/--channels option to ‘guix pull’ to specify a channel file.
3. Fixed ‘update-NEWS.scm’, which assumed that the last item of (%package-module-path) is Guix’s own set of modules. We also find this pattern in guix-artwork.git/website and probably in other places, so we’ll have to adjust them.
Ludo’.
L
L
Ludovic Courtès wrote on 30 Aug 2018 23:31
[PATCH v2 1/3] discovery: Add 'scheme-modules*'.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180830213152.3404-1-ludo@gnu.org
* guix/self.scm (scheme-modules*): Move to...* guix/discovery.scm (scheme-modules*): ... here. New procedure. Make'sub-directory' an optional parameter.--- guix/discovery.scm | 13 ++++++++++++- guix/self.scm | 7 ------- 2 files changed, 12 insertions(+), 8 deletions(-)
Toggle diff (56 lines)diff --git a/guix/discovery.scm b/guix/discovery.scmindex 2b627d108..3fc6e2c9e 100644--- a/guix/discovery.scm+++ b/guix/discovery.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -27,6 +27,7 @@ #:use-module (ice-9 ftw) #:export (scheme-files scheme-modules+ scheme-modules* fold-modules all-modules fold-module-public-variables))@@ -115,6 +116,16 @@ name and the exception key and arguments." (string-append directory "/" sub-directory) directory)))) +(define* (scheme-modules* directory #:optional sub-directory)+ "Return the list of module names found under SUB-DIRECTORY in DIRECTORY.+This is a source-only variant that does not try to load files."+ (let ((prefix (string-length directory)))+ (map (lambda (file)+ (file-name->module-name (string-drop file prefix)))+ (scheme-files (if sub-directory+ (string-append directory "/" sub-directory)+ directory)))))+ (define* (fold-modules proc init path #:key (warn (const #f))) "Fold over all the Scheme modules present in PATH, a list of directories. Call (PROC MODULE RESULT) for each module that is found."diff --git a/guix/self.scm b/guix/self.scmindex 90649db17..81f9b0cfd 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -206,13 +206,6 @@ list of file-name/file-like objects suitable as inputs to 'imported-files'." (local-file file #:recursive? #t))) (find-files (string-append directory "/" sub-directory) pred))) -(define (scheme-modules* directory sub-directory)- "Return the list of module names found under SUB-DIRECTORY in DIRECTORY."- (let ((prefix (string-length directory)))- (map (lambda (file)- (file-name->module-name (string-drop file prefix)))- (scheme-files (string-append directory "/" sub-directory)))))- (define* (sub-directory item sub-directory) "Return SUB-DIRECTORY within ITEM, which may be a file name or a file-like object."-- 2.18.0
L
L
Ludovic Courtès wrote on 30 Aug 2018 23:31
[PATCH v2 2/3] Add (guix describe) and use it to initialize '%package-search-path'.
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180830213152.3404-2-ludo@gnu.org
* guix/describe.scm: New file.* Makefile.am (MODULES): Add it.* gnu/packages.scm (%default-package-module-path): New variable.(%package-module-path): Honor 'package-path-entries'.* build-aux/update-NEWS.scm (main): Use %DEFAULT-PACKAGE-MODULE-PATHinstead of (last (%package-module-path)).--- Makefile.am | 1 + build-aux/update-NEWS.scm | 9 +++-- gnu/packages.scm | 27 ++++++++++----- guix/describe.scm | 73 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 97 insertions(+), 13 deletions(-) create mode 100644 guix/describe.scm
Toggle diff (179 lines)diff --git a/Makefile.am b/Makefile.amindex 324674a60..b6efd6d62 100644--- a/Makefile.am+++ b/Makefile.am@@ -86,6 +86,7 @@ MODULES = \ guix/derivations.scm \ guix/grafts.scm \ guix/inferior.scm \+ guix/describe.scm \ guix/gnu-maintenance.scm \ guix/self.scm \ guix/upstream.scm \diff --git a/build-aux/update-NEWS.scm b/build-aux/update-NEWS.scmindex 2e8f68c9a..a9dffef1d 100644--- a/build-aux/update-NEWS.scm+++ b/build-aux/update-NEWS.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2017, 2018 Ludovic Courtès <ludo@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -128,11 +128,10 @@ paragraph." (define (main . args) (match args ((news-file data-directory)- ;; Don't browse things listed in the user's $GUIX_PACKAGE_PATH. Here we- ;; assume that the last item in (%package-module-path) is the distro- ;; directory.+ ;; Don't browse things listed in the user's $GUIX_PACKAGE_PATH and+ ;; in external channels. (parameterize ((%package-module-path- (list (last (%package-module-path)))))+ %default-package-module-path)) (define (package-file version) (string-append data-directory "/packages-" version ".txt"))diff --git a/gnu/packages.scm b/gnu/packages.scmindex 7b954769e..532297239 100644--- a/gnu/packages.scm+++ b/gnu/packages.scm@@ -30,6 +30,7 @@ #:select ((package-name->name+version . hyphen-separated-name->name+version))) #:autoload (guix profiles) (packages->manifest)+ #:use-module (guix describe) #:use-module (ice-9 vlist) #:use-module (ice-9 match) #:use-module (srfi srfi-1)@@ -46,6 +47,7 @@ %auxiliary-files-path %bootstrap-binaries-path %package-module-path+ %default-package-module-path fold-packages @@ -130,22 +132,31 @@ for system '~a'") ("gnu/packages.scm" gnu/) ("guix.scm")))) +(define %default-package-module-path+ ;; Default search path for package modules.+ `((,%distro-root-directory . "gnu/packages")))+ (define %package-module-path ;; Search path for package modules. Each item must be either a directory ;; name or a pair whose car is a directory and whose cdr is a sub-directory ;; to narrow the search. (let* ((not-colon (char-set-complement (char-set #\:))) (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")- not-colon)))- ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.- (for-each (lambda (directory)- (set! %load-path (cons directory %load-path))- (set! %load-compiled-path- (cons directory %load-compiled-path)))- environment)+ not-colon))+ (channels (package-path-entries)))+ ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's+ ;; search path. For historical reasons, $GUIX_PACKAGE_PATH goes to the+ ;; front; channels go to the back so that they don't override Guix' own+ ;; modules.+ (set! %load-path+ (append environment %load-path channels))+ (set! %load-compiled-path+ (append environment %load-compiled-path channels)) (make-parameter- (append environment `((,%distro-root-directory . "gnu/packages"))))))+ (append environment+ %default-package-module-path+ channels)))) (define %patch-path ;; Define it after '%package-module-path' so that '%load-path' contains userdiff --git a/guix/describe.scm b/guix/describe.scmnew file mode 100644index 000000000..3122a762f--- /dev/null+++ b/guix/describe.scm@@ -0,0 +1,73 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++(define-module (guix describe)+ #:use-module (guix memoization)+ #:use-module (guix profiles)+ #:use-module (srfi srfi-1)+ #:use-module (ice-9 match)+ #:export (package-path-entries))++;;; Commentary:+;;;+;;; This module provides supporting code to allow a Guix instance to find, at+;;; run time, which profile it's in (profiles created by 'guix pull'). That+;;; allows it to read meta-information about itself (e.g., repository URL and+;;; commit ID) and to find other channels available in the same profile. It's+;;; a bit like ELPA's pkg-info.el.+;;;+;;; Code:++(define current-profile+ (mlambda ()+ "Return the profile (created by 'guix pull') the calling process lives in,+or #f if this is not applicable."+ (match (command-line)+ ((program . _)+ (and (string-suffix? "/bin/guix" program)+ ;; Note: We want to do _lexical dot-dot resolution_. Using ".."+ ;; for real would instead take us into the /gnu/store directory+ ;; that ~/.config/guix/current/bin points to, whereas we want to+ ;; obtain ~/.config/guix/current.+ (let ((candidate (dirname (dirname program))))+ (and (file-exists? (string-append candidate "/manifest"))+ candidate)))))))++(define current-profile-entries+ (mlambda ()+ "Return the list of entries in the 'guix pull' profile the calling process+lives in, or #f if this is not applicable."+ (match (current-profile)+ (#f '())+ (profile+ (let ((manifest (profile-manifest profile)))+ (manifest-entries manifest))))))++(define package-path-entries+ (mlambda ()+ "Return a list of package path entries to be added to the package search+path. These entries are taken from the 'guix pull' profile the calling+process lives in, when applicable."+ ;; Filter out Guix itself.+ (filter-map (lambda (entry)+ (and (not (string=? (manifest-entry-name entry)+ "guix"))+ (string-append (manifest-entry-item entry)+ "/share/guile/site/"+ (effective-version))))+ (current-profile-entries))))-- 2.18.0
L
L
Ludovic Courtès wrote on 30 Aug 2018 23:31
[PATCH v2 3/3] Add (guix channels) and use it in (guix scripts pull).
(address . 22629@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180830213152.3404-3-ludo@gnu.org
* guix/channels.scm: New file.* Makefile.am (MODULES): Add it.* guix/scripts/pull.scm: Use it.(%default-options): Remove 'repository-url' and 'ref'.(show-help, %options): Add '--channels'.(%self-build-file, %pull-version, build-from-source)(whole-package-for-legacy, derivation->manifest-entry): Remove. Thesenow exist in a similar form in (guix channels).(build-and-install): Change 'source' to 'instances'. Remove #:url, #:branch, and #:commit. Rewrite using 'channel-instances->manifest'.(channel-list): New procedure.(guix-pull): Parameterize %REPOSITORY-CACHE-DIRECTORY. Call'honor-lets-encrypt-certificates!' unconditionally. Load~/.config/guix/channels.scm. Rewrite to use (guix channels).[use-le-certs?]: Remove.* po/guix/POTFILES.in: Add (guix channels).* doc/guix.texi (Invoking guix pull): Group the description of '--url','--commit', and '--branch'. Remove mention of 'GUIX_PULL_URL'. Addreferences to "Channels". Document '--channels'.(Channels): New node.(Defining Packages): Link to "Channels" instead of "Package Modules".(Invoking guix edit): Link to "Package Modules" instead of "DefiningPackages".(Package Modules): Document both GUIX_PACKAGE_PATH and channels.--- Makefile.am | 1 + doc/guix.texi | 271 ++++++++++++++++++++++++++++++++------- guix/channels.scm | 292 ++++++++++++++++++++++++++++++++++++++++++ guix/scripts/pull.scm | 218 ++++++++++++++----------------- po/guix/POTFILES.in | 1 + 5 files changed, 619 insertions(+), 164 deletions(-) create mode 100644 guix/channels.scm
Toggle diff (1025 lines)diff --git a/Makefile.am b/Makefile.amindex b6efd6d62..af6870cf6 100644--- a/Makefile.am+++ b/Makefile.am@@ -87,6 +87,7 @@ MODULES = \ guix/grafts.scm \ guix/inferior.scm \ guix/describe.scm \+ guix/channels.scm \ guix/gnu-maintenance.scm \ guix/self.scm \ guix/upstream.scm \diff --git a/doc/guix.texi b/doc/guix.texiindex cd0e74a2d..76d8626d7 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -146,17 +146,18 @@ Package Management * Packages with Multiple Outputs:: Single source package, multiple outputs. * Invoking guix gc:: Running the garbage collector. * Invoking guix pull:: Fetching the latest Guix and distribution.+* Channels:: Customizing the package collection. * Invoking guix pack:: Creating software bundles. * Invoking guix archive:: Exporting and importing store files. Substitutes -* Official Substitute Server:: One particular source of substitutes.-* Substitute Server Authorization:: How to enable or disable substitutes.-* Substitute Authentication:: How Guix verifies substitutes.-* Proxy Settings:: How to get substitutes via proxy.-* Substitution Failure:: What happens when substitution fails.-* On Trusting Binaries:: How can you trust that binary blob?+* Official Substitute Server:: One particular source of substitutes.+* Substitute Server Authorization:: How to enable or disable substitutes.+* Substitute Authentication:: How Guix verifies substitutes.+* Proxy Settings:: How to get substitutes via proxy.+* Substitution Failure:: What happens when substitution fails.+* On Trusting Binaries:: How can you trust that binary blob? Programming Interface @@ -202,7 +203,7 @@ GNU Distribution * System Installation:: Installing the whole operating system. * System Configuration:: Configuring the operating system.-* Documentation:: Browsing software user manuals.+* Documentation:: Browsing software user manuals. * Installing Debugging Files:: Feeding the debugger. * Security Updates:: Deploying security fixes quickly. * Package Modules:: Packages from the programmer's viewpoint.@@ -264,7 +265,7 @@ Services * Audio Services:: The MPD. * Virtualization Services:: Virtualization services. * Version Control Services:: Providing remote access to Git repositories.-* Game Services:: Game servers.+* Game Services:: Game servers. * Miscellaneous Services:: Other services. Defining Services@@ -1694,6 +1695,7 @@ guix package -i emacs-guix * Packages with Multiple Outputs:: Single source package, multiple outputs. * Invoking guix gc:: Running the garbage collector. * Invoking guix pull:: Fetching the latest Guix and distribution.+* Channels:: Customizing the package collection. * Invoking guix pack:: Creating software bundles. * Invoking guix archive:: Exporting and importing store files. @end menu@@ -2276,12 +2278,12 @@ pre-built package binaries, but source tarballs, for instance, which also result from derivation builds, can be available as substitutes. @menu-* Official Substitute Server:: One particular source of substitutes.-* Substitute Server Authorization:: How to enable or disable substitutes.-* Substitute Authentication:: How Guix verifies substitutes.-* Proxy Settings:: How to get substitutes via proxy.-* Substitution Failure:: What happens when substitution fails.-* On Trusting Binaries:: How can you trust that binary blob?+* Official Substitute Server:: One particular source of substitutes.+* Substitute Server Authorization:: How to enable or disable substitutes.+* Substitute Authentication:: How Guix verifies substitutes.+* Proxy Settings:: How to get substitutes via proxy.+* Substitution Failure:: What happens when substitution fails.+* On Trusting Binaries:: How can you trust that binary blob? @end menu @node Official Substitute Server@@ -2746,7 +2748,8 @@ the distribution currently available on your local machine. To update that distribution, along with the Guix tools, you must run @command{guix pull}: the command downloads the latest Guix source code and package descriptions, and deploys it. Source code is downloaded from a-@uref{https://git-scm.com, Git} repository.+@uref{https://git-scm.com, Git} repository, by default the official+GNU@tie{}Guix repository, though this can be customized. On completion, @command{guix package} will use packages and package versions from this just-retrieved copy of Guix. Not only that, but all@@ -2821,20 +2824,23 @@ but it supports the following options: Produce verbose output, writing build logs to the standard error output. @item --url=@var{url}-Download Guix from the Git repository at @var{url}.+@itemx --commit=@var{commit}+@itemx --branch=@var{branch}+Download code from the specified @var{url}, at the given @var{commit} (a valid+Git commit ID represented as a hexadecimal string), or @var{branch}. -@vindex GUIX_PULL_URL-By default, the source is taken from its canonical Git repository at-@code{gnu.org}, for the stable branch of Guix. To use a different source,-set the @code{GUIX_PULL_URL} environment variable.+@cindex @file{channels.scm}, configuration file+@cindex configuration file for channels+These options are provided for convenience, but you can also specify your+configuration in the @file{~/.config/guix/channels.scm} file or using the+@option{--channels} option (see below). -@item --commit=@var{commit}-Deploy @var{commit}, a valid Git commit ID represented as a hexadecimal-string.--@item --branch=@var{branch}-Deploy the tip of @var{branch}, the name of a Git branch available on-the repository at @var{url}.+@item --channels=@var{file}+@itemx -C @var{file}+Read the list of channels from @var{file} instead of+@file{~/.config/guix/channels.scm}. @var{file} must contain Scheme code that+evaluates to a list of channel objects. @xref{Channels}, for more+information. @item --list-generations[=@var{pattern}] @itemx -l [@var{pattern}]@@ -2848,9 +2854,178 @@ Use the bootstrap Guile to build the latest Guix. This option is only useful to Guix developers. @end table +The @dfn{channel} mechanism allows you to instruct @command{guix pull} which+repository and branch to pull from, as well as @emph{additional} repositories+containing package modules that should be deployed. @xref{Channels}, for more+information.+ In addition, @command{guix pull} supports all the common build options (@pxref{Common Build Options}). +@node Channels+@section Channels++@cindex @file{channels.scm}, configuration file+@cindex configuration file for channels+@cindex @command{guix pull}, configuration file+@cindex configuration of @command{guix pull}+Guix and its package collection are updated by running @command{guix pull}+(@pxref{Invoking guix pull}). By default @command{guix pull} downloads and+deploys Guix itself from the official GNU@tie{}Guix repository. This can be+customized by defining @dfn{channels} in the+@file{~/.config/guix/channels.scm} file. A channel specifies a URL and branch+of a Git repository to be deployed, and @command{guix pull} can be instructed+to pull from one or more channels. In other words, channels can be used to+@emph{customize} and to @emph{extend} Guix, as we will see below.++@subsection Using a Custom Guix Channel++The channel called @code{guix} specifies where Guix itself---its command-line+tools as well as its package collection---should be downloaded. For instance,+suppose you want to update from your own copy of the Guix repository at+@code{example.org}, and specifically the @code{super-hacks} branch, you can+write in @code{~/.config/guix/channels.scm} this specification:++@lisp+;; Tell 'guix pull' to use my own repo.+(list (channel+ (name 'guix)+ (url "https://example.org/my-guix.git")+ (branch "super-hacks")))+@end lisp++@noindent+From there on, @command{guix pull} will fetch code from the @code{super-hacks}+branch of the repository at @code{example.org}.++@subsection Specifying Additional Channels++@cindex extending the package collection (channels)+@cindex personal packages (channels)+@cindex channels, for personal packages+You can also specify @emph{additional channels} to pull from. Let's say you+have a bunch of custom package variants or personal packages that you think+would make little sense to contribute to the Guix project, but would like to+have these packages transparently available to you at the command line. You+would first write modules containing those package definitions (@pxref{Package+Modules}), maintain them in a Git repository, and then you and anyone else can+use it as an additional channel to get packages from. Neat, no?++@c What follows stems from discussions at+@c <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629#134> as well as+@c earlier discussions on guix-devel@gnu.org.+@quotation Warning+Before you, dear user, shout---``woow this is @emph{soooo coool}!''---and+publish your personal channel to the world, we would like to share a few words+of caution:++@itemize+@item+Before publishing a channel, please consider contributing your package+definitions to Guix proper (@pxref{Contributing}). Guix as a project is open+to free software of all sorts, and packages in Guix proper are readily+available to all Guix users and benefit from the project's quality assurance+process.++@item+When you maintain package definitions outside Guix, we, Guix developers,+consider that @emph{the compatibility burden is on you}. Remember that+package modules and package definitions are just Scheme code that uses various+programming interfaces (APIs). We want to remain free to change these APIs to+keep improving Guix, possibly in ways that break your channel. We never+change APIs gratuitously, but we will @emph{not} commit to freezing APIs+either.++@item+Corollary: if you're using an external channel and that channel breaks, please+@emph{report the issue to the channel authors}, not to the Guix project.+@end itemize++You've been warned! Having said this, we believe external channels are a+practical way to exert your freedom to augment Guix' package collection and to+share your improvements, which are basic tenets of+@uref{https://www.gnu.org/philosophy/free-sw.html, free software}.+@end quotation++Once you have a Git repository containing your own package modules, you can+write @code{~/.config/guix/channels.scm} to instruct @command{guix pull} to+pull from your personal channel @emph{in addition} to the default Guix+channel(s):++@vindex %default-channels+@lisp+;; Add my personal packages to those Guix provides.+(cons (channel+ (name 'my-personal-packages)+ (url "https://example.org/personal-packages.git"))+ %default-channels)+@end lisp++@noindent+Note that the snippet above is (as always!) Scheme code; we use @code{cons} to+add a channel the list of channels that the variable @code{%default-channels}+is bound to (@pxref{Pairs, @code{cons} and lists,, guile, GNU Guile Reference+Manual}). With this file in place, @command{guix pull} builds not only Guix+but also the package modules from your own repository. The result in+@file{~/.config/guix/current} is the union of Guix with your own package+modules:++@example+$ guix pull --list-generations+@dots{}+Generation 19 Aug 27 2018 16:20:48+ guix d894ab8+ repository URL: https://git.savannah.gnu.org/git/guix.git+ branch: master+ commit: d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300+ my-personal-packages dd3df5e+ repository URL: https://example.org/personal-packages.git+ branch: master+ commit: dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb+ 11 new packages: my-gimp, my-emacs-with-cool-features, @dots{}+ 4 packages upgraded: emacs-racket-mode@@0.0.2-2.1b78827, @dots{}+@end example++@noindent+The output of @command{guix pull} above shows that Generation@tie{}19 includes+both Guix and packages from the @code{my-personal-packages} channel. Among+the new and upgraded packages that are listed, some like @code{my-gimp} and+@code{my-emacs-with-cool-features} might come from+@code{my-personal-packages}, while others come from the Guix default channel.++@subsection Replicating Guix++@cindex pinning, channels+@cindex replicating Guix+@cindex reproducibility, of Guix+The @command{guix pull --list-generations} output above shows precisely which+commits were used to build this instance of Guix. We can thus replicate it,+say, on another machine, by providing a channel specification in+@file{~/.config/guix/channels.scm} that is ``pinned'' to these commits:++@lisp+;; Deploy specific commits of my channels of interest.+(list (channel+ (name 'guix)+ (url "https://git.savannah.gnu.org/git/guix.git")+ (commit "d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300"))+ (channel+ (name 'my-personal-packages)+ (url "https://example.org/personal-packages.git")+ (branch "dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb")))+@end lisp++At this point the two machines run the @emph{exact same Guix}, with access to+the @emph{exact same packages}. The output of @command{guix build gimp} on+one machine will be exactly the same, bit for bit, as the output of the same+command on the other machine. It also means both machines have access to all+the source code of Guix and, transitively, to all the source code of every+package it defines.++This gives you super powers, allowing you to track the provenance of binary+artifacts with very fine grain, and to reproduce software environments at+will---some sort of ``meta reproducibility'' capabilities, if you will.+ @node Invoking guix pack @section Invoking @command{guix pack} @@ -3431,9 +3606,9 @@ more information on how to test package definitions, and @ref{Invoking guix lint}, for information on how to check a definition for style conformance. @vindex GUIX_PACKAGE_PATH-Lastly, @pxref{Package Modules}, for information+Lastly, @pxref{Channels}, for information on how to extend the distribution by adding your own package definitions-to @code{GUIX_PACKAGE_PATH}.+in a ``channel''. Finally, updating the package definition to a new upstream version can be partly automated by the @command{guix refresh} command@@ -6255,8 +6430,8 @@ and that of Vim. If you are using a Guix Git checkout (@pxref{Building from Git}), or have created your own packages on @code{GUIX_PACKAGE_PATH}-(@pxref{Defining Packages}), you will be able to edit the package-recipes. Otherwise, you will be able to examine the read-only recipes+(@pxref{Package Modules}), you will be able to edit the package+recipes. In other cases, you will be able to examine the read-only recipes for packages currently in the store. @@ -8363,7 +8538,7 @@ For information on porting to other architectures or kernels, @menu * System Installation:: Installing the whole operating system. * System Configuration:: Configuring the operating system.-* Documentation:: Browsing software user manuals.+* Documentation:: Browsing software user manuals. * Installing Debugging Files:: Feeding the debugger. * Security Updates:: Deploying security fixes quickly. * Package Modules:: Packages from the programmer's viewpoint.@@ -8402,7 +8577,7 @@ available. @menu * Limitations:: What you can expect. * Hardware Considerations:: Supported hardware.-* USB Stick and DVD Installation:: Preparing the installation medium.+* USB Stick and DVD Installation:: Preparing the installation medium. * Preparing for Installation:: Networking, partitioning, etc. * Proceeding with the Installation:: The real thing. * Installing GuixSD in a VM:: GuixSD playground.@@ -10083,7 +10258,7 @@ declaration. * Audio Services:: The MPD. * Virtualization Services:: Virtualization services. * Version Control Services:: Providing remote access to Git repositories.-* Game Services:: Game servers.+* Game Services:: Game servers. * Miscellaneous Services:: Other services. @end menu @@ -22520,16 +22695,24 @@ name and module name must match. For instance, the @code{(my-packages emacs)} module must be stored in a @file{my-packages/emacs.scm} file relative to the load path specified with @option{--load-path} or @code{GUIX_PACKAGE_PATH}. @xref{Modules and the File System,,,-guile, GNU Guile Reference Manual}, for details.}. These package definitions-will not be visible by default. Users can invoke commands such as-@command{guix package} and @command{guix build} with the-@code{-e} option so that they know where to find the package. Better-yet, they can use the-@code{-L} option of these commands to make those modules visible-(@pxref{Invoking guix build, @code{--load-path}}), or define the-@code{GUIX_PACKAGE_PATH} environment variable. This environment-variable makes it easy to extend or customize the distribution and is-honored by all the user interfaces.+guile, GNU Guile Reference Manual}, for details.}. There are two ways to make+these package definitions visible to the user interfaces:++@enumerate+@item+By adding the directory containing your package modules to the search path+with the @code{-L} flag of @command{guix package} and other commands+(@pxref{Common Build Options}), or by setting the @code{GUIX_PACKAGE_PATH}+environment variable described below.++@item+By defining a @dfn{channel} and configuring @command{guix pull} so that it+pulls from it. A channel is essentially a Git repository containing package+modules. @xref{Channels}, for more information on how to define and use+channels.+@end enumerate++@code{GUIX_PACKAGE_PATH} works similarly to other search path variables: @defvr {Environment Variable} GUIX_PACKAGE_PATH This is a colon-separated list of directories to search for additionaldiff --git a/guix/channels.scm b/guix/channels.scmnew file mode 100644index 000000000..ec3e05eaf--- /dev/null+++ b/guix/channels.scm@@ -0,0 +1,292 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++(define-module (guix channels)+ #:use-module (guix git)+ #:use-module (guix records)+ #:use-module (guix gexp)+ #:use-module (guix discovery)+ #:use-module (guix monads)+ #:use-module (guix profiles)+ #:use-module (guix derivations)+ #:use-module (guix store)+ #:use-module (guix i18n)+ #:use-module (srfi srfi-1)+ #:use-module (srfi srfi-9)+ #:use-module (srfi srfi-11)+ #:autoload (guix self) (whole-package)+ #:use-module (ice-9 match)+ #:export (channel+ channel?+ channel-name+ channel-url+ channel-branch+ channel-commit+ channel-location++ %default-channels++ channel-instance?+ channel-instance-channel+ channel-instance-commit+ channel-instance-checkout++ latest-channel-instances+ channel-instance-derivations+ latest-channel-derivations+ channel-instances->manifest))++;;; Commentary:+;;;+;;; This module implements "channels." A channel is usually a source of+;;; package definitions. There's a special channel, the 'guix' channel, that+;;; provides all of Guix, including its commands and its documentation.+;;; User-defined channels are expected to typically provide a bunch of .scm+;;; files meant to be added to the '%package-search-path'.+;;;+;;; This module provides tools to fetch and update channels from a Git+;;; repository and to build them.+;;;+;;; Code:++(define-record-type* <channel> channel make-channel+ channel?+ (name channel-name)+ (url channel-url)+ (branch channel-branch (default "master"))+ (commit channel-commit (default #f))+ (location channel-location+ (default (current-source-location)) (innate)))+;; TODO: Add a way to express dependencies among channels.++(define %default-channels+ ;; Default list of channels.+ (list (channel+ (name 'guix)+ (branch "origin/master")+ (url "https://git.savannah.gnu.org/git/guix.git"))))++(define (guix-channel? channel)+ "Return true if CHANNEL is the 'guix' channel."+ (eq? 'guix (channel-name channel)))++(define-record-type <channel-instance>+ (channel-instance channel commit checkout)+ channel-instance?+ (channel channel-instance-channel)+ (commit channel-instance-commit)+ (checkout channel-instance-checkout))++(define (channel-reference channel)+ "Return the \"reference\" for CHANNEL, an sexp suitable for+'latest-repository-commit'."+ (match (channel-commit channel)+ (#f `(branch . ,(channel-branch channel)))+ (commit `(commit . ,(channel-commit channel)))))++(define (latest-channel-instances store channels)+ "Return a list of channel instances corresponding to the latest checkouts of+CHANNELS."+ (map (lambda (channel)+ (format (current-error-port)+ (G_ "Updating channel '~a' from Git repository at '~a'...~%")+ (channel-name channel)+ (channel-url channel))+ (let-values (((checkout commit)+ (latest-repository-commit store (channel-url channel)+ #:ref (channel-reference+ channel))))+ (channel-instance channel commit checkout)))+ channels))++(define %self-build-file+ ;; The file containing code to build Guix. This serves the same purpose as+ ;; a makefile, and, similarly, is intended to always keep this name.+ "build-aux/build-self.scm")++(define %pull-version+ ;; This is the version of the 'guix pull' protocol. It specifies what's+ ;; expected from %SELF-BUILD-FILE. The initial version ("0") was when we'd+ ;; place a set of compiled Guile modules in ~/.config/guix/latest.+ 1)++(define (standard-module-derivation name source dependencies)+ "Return a derivation that builds the Scheme modules in SOURCE and that+depend on DEPENDENCIES, a list of lowerable objects. The assumption is that+SOURCE contains package modules to be added to '%package-module-path'."+ (define modules+ (scheme-modules* source))++ ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow+ ;; channel publishers to specify things such as the sub-directory where .scm+ ;; files live, files to exclude from the channel, preferred substitute URLs,+ ;; etc.+ (mlet* %store-monad ((compiled+ (compiled-modules modules+ #:name name+ #:module-path (list source)+ #:extensions dependencies)))++ (gexp->derivation name+ (with-extensions dependencies+ (with-imported-modules '((guix build utils))+ #~(begin+ (use-modules (guix build utils))++ (let ((go (string-append #$output "/lib/guile/"+ (effective-version)+ "/site-ccache"))+ (scm (string-append #$output+ "/share/guile/site/"+ (effective-version))))+ (mkdir-p (dirname go))+ (symlink #$compiled go)+ (mkdir-p (dirname scm))+ (symlink #$source scm))))))))++(define* (build-from-source name source+ #:key verbose? commit+ (dependencies '()))+ "Return a derivation to build Guix from SOURCE, using the self-build script+contained therein. Use COMMIT as the version string."+ ;; Running the self-build script makes it easier to update the build+ ;; procedure: the self-build script of the Guix-to-be-installed contains the+ ;; right dependencies, build procedure, etc., which the Guix-in-use may not+ ;; be know.+ (define script+ (string-append source "/" %self-build-file))++ (if (file-exists? script)+ (let ((build (save-module-excursion+ (lambda ()+ (primitive-load script)))))+ ;; BUILD must be a monadic procedure of at least one argument: the+ ;; source tree.+ ;;+ ;; Note: BUILD can return #f if it does not support %PULL-VERSION. In+ ;; the future we'll fall back to a previous version of the protocol+ ;; when that happens.+ (build source #:verbose? verbose? #:version commit+ #:pull-version %pull-version))++ ;; Build a set of modules that extend Guix using the standard method.+ (standard-module-derivation name source dependencies)))++(define* (build-channel-instance instance #:optional (dependencies '()))+ "Return, as a monadic value, the derivation for INSTANCE, a channel+instance. DEPENDENCIES is a list of extensions providing Guile modules that+INSTANCE depends on."+ (build-from-source (symbol->string+ (channel-name (channel-instance-channel instance)))+ (channel-instance-checkout instance)+ #:commit (channel-instance-commit instance)+ #:dependencies dependencies))++(define (channel-instance-derivations instances)+ "Return the list of derivations to build INSTANCES, in the same order as+INSTANCES."+ (define core-instance+ ;; The 'guix' channel is treated specially: it's an implicit dependency of+ ;; all the other channels.+ (find (lambda (instance)+ (guix-channel? (channel-instance-channel instance)))+ instances))++ (mlet %store-monad ((core (build-channel-instance core-instance)))+ (mapm %store-monad+ (lambda (instance)+ (if (eq? instance core-instance)+ (return core)+ (build-channel-instance instance+ (list core))))+ instances)))++(define latest-channel-derivations+ (let ((latest-channel-instances (store-lift latest-channel-instances)))+ (lambda (channels)+ "Return, as a monadic value, the list of derivations for the latest+instances of CHANNELS."+ (mlet %store-monad ((instances (latest-channel-instances channels)))+ (channel-instance-derivations instances)))))++(define (whole-package-for-legacy name modules)+ "Return a full-blown Guix package for MODULES, a derivation that builds Guix+modules in the old ~/.config/guix/latest style."+ (define packages+ (resolve-interface '(gnu packages guile)))++ (letrec-syntax ((list (syntax-rules (->)+ ((_)+ '())+ ((_ (module -> variable) rest ...)+ (cons (module-ref (resolve-interface+ '(gnu packages module))+ 'variable)+ (list rest ...)))+ ((_ variable rest ...)+ (cons (module-ref packages 'variable)+ (list rest ...))))))+ (whole-package name modules++ ;; In the "old style", %SELF-BUILD-FILE would simply return a+ ;; derivation that builds modules. We have to infer what the+ ;; dependencies of these modules were.+ (list guile-json guile-git guile-bytestructures+ (ssh -> guile-ssh) (tls -> gnutls)))))++(define (old-style-guix? drv)+ "Return true if DRV corresponds to a ~/.config/guix/latest style of+derivation."+ ;; Here we rely on a gross historical fact: that derivations produced by the+ ;; "old style" (before commit 8a0d9bc8a3f153159d9e239a151c0fa98f1e12d8,+ ;; dated May 30, 2018) did not depend on "guix-command.drv".+ (not (find (lambda (input)+ (string-suffix? "-guix-command.drv"+ (derivation-input-path input)))+ (derivation-inputs drv))))++(define (channel-instances->manifest instances)+ "Return a profile manifest with entries for all of INSTANCES, a list of+channel instances."+ (define instance->entry+ (match-lambda+ ((instance drv)+ (let ((commit (channel-instance-commit instance))+ (channel (channel-instance-channel instance)))+ (with-monad %store-monad+ (return (manifest-entry+ (name (symbol->string (channel-name channel)))+ (version (string-take commit 7))+ (item (if (guix-channel? channel)+ (if (old-style-guix? drv)+ (whole-package-for-legacy+ (string-append name "-" version)+ drv)+ drv)+ drv))+ (properties+ `((source (repository+ (version 0)+ (url ,(channel-url channel))+ (branch ,(channel-branch channel))+ (commit ,commit))))))))))))++ (mlet* %store-monad ((derivations (channel-instance-derivations instances))+ (entries (mapm %store-monad instance->entry+ (zip instances derivations))))+ (return (manifest entries))))diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scmindex ee68c21a4..8400ad278 100644--- a/guix/scripts/pull.scm+++ b/guix/scripts/pull.scm@@ -30,26 +30,19 @@ #:use-module (guix grafts) #:use-module (guix memoization) #:use-module (guix monads)+ #:use-module (guix channels) #:autoload (guix inferior) (open-inferior) #:use-module (guix scripts build)- #:autoload (guix self) (whole-package) #:use-module (guix git) #:use-module (git) #:use-module (gnu packages)- #:autoload (gnu packages ssh) (guile-ssh)- #:autoload (gnu packages tls) (gnutls) #:use-module ((guix scripts package) #:select (build-and-use-profile))- #:use-module ((guix build utils)- #:select (with-directory-excursion delete-file-recursively))- #:use-module ((guix build download)- #:select (%x509-certificate-directory)) #:use-module (gnu packages base) #:use-module (gnu packages guile) #:use-module ((gnu packages bootstrap) #:select (%bootstrap-guile)) #:use-module ((gnu packages certs) #:select (le-certs)) #:use-module (srfi srfi-1)- #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-35) #:use-module (srfi srfi-37)@@ -57,9 +50,6 @@ #:use-module (ice-9 vlist) #:export (guix-pull)) -(define %repository-url- (or (getenv "GUIX_PULL_URL") "https://git.savannah.gnu.org/git/guix.git"))- ;;; ;;; Command-line options.@@ -67,9 +57,7 @@ (define %default-options ;; Alist of default option values.- `((repository-url . ,%repository-url)- (ref . (branch . "origin/master"))- (system . ,(%current-system))+ `((system . ,(%current-system)) (substitutes? . #t) (build-hook? . #t) (graft? . #t)@@ -80,6 +68,8 @@ Download and deploy the latest version of Guix.\n")) (display (G_ " --verbose produce verbose output"))+ (display (G_ "+ -C, --channels=FILE deploy the channels defined in FILE")) (display (G_ " --url=URL download from the Git repository at URL")) (display (G_ "@@ -105,6 +95,9 @@ Download and deploy the latest version of Guix.\n")) (cons* (option '("verbose") #f #f (lambda (opt name arg result) (alist-cons 'verbose? #t result)))+ (option '(#\C "channels") #t #f+ (lambda (opt name arg result)+ (alist-cons 'channel-file arg result))) (option '(#\l "list-generations") #f #t (lambda (opt name arg result) (cons `(query list-generations ,(or arg ""))@@ -142,70 +135,6 @@ Download and deploy the latest version of Guix.\n")) (define indirect-root-added (store-lift add-indirect-root)) -(define %self-build-file- ;; The file containing code to build Guix. This serves the same purpose as- ;; a makefile, and, similarly, is intended to always keep this name.- "build-aux/build-self.scm")--(define %pull-version- ;; This is the version of the 'guix pull' protocol. It specifies what's- ;; expected from %SELF-BUILD-FILE. The initial version ("0") was when we'd- ;; place a set of compiled Guile modules in ~/.config/guix/latest.- 1)--(define* (build-from-source source- #:key verbose? commit)- "Return a derivation to build Guix from SOURCE, using the self-build script-contained therein. Use COMMIT as the version string."- ;; Running the self-build script makes it easier to update the build- ;; procedure: the self-build script of the Guix-to-be-installed contains the- ;; right dependencies, build procedure, etc., which the Guix-in-use may not- ;; be know.- (let* ((script (string-append source "/" %self-build-file))- (build (primitive-load script)))- ;; BUILD must be a monadic procedure of at least one argument: the source- ;; tree.- ;;- ;; Note: BUILD can return #f if it does not support %PULL-VERSION. In the- ;; future we'll fall back to a previous version of the protocol when that- ;; happens.- (build source #:verbose? verbose? #:version commit- #:pull-version %pull-version)))--(define (whole-package-for-legacy name modules)- "Return a full-blown Guix package for MODULES, a derivation that builds Guix-modules in the old ~/.config/guix/latest style."- (whole-package name modules-- ;; In the "old style", %SELF-BUILD-FILE would simply return a- ;; derivation that builds modules. We have to infer what the- ;; dependencies of these modules were.- (list guile-json guile-git guile-bytestructures- guile-ssh gnutls)))--(define* (derivation->manifest-entry drv- #:key url branch commit)- "Return a manifest entry for DRV, which represents Guix at COMMIT. Record-URL, BRANCH, and COMMIT as a property in the manifest entry."- (mbegin %store-monad- (what-to-build (list drv))- (built-derivations (list drv))- (let ((out (derivation->output-path drv)))- (return (manifest-entry- (name "guix")- (version (string-take commit 7))- (item (if (file-exists? (string-append out "/bin/guix"))- drv- (whole-package-for-legacy (string-append name "-"- version)- drv)))- (properties- `((source (repository- (version 0)- (url ,url)- (branch ,branch)- (commit ,commit))))))))))- (define (display-profile-news profile) "Display what's up in PROFILE--new packages, and all that." (match (memv (generation-number profile)@@ -223,8 +152,8 @@ URL, BRANCH, and COMMIT as a property in the manifest entry." #:heading (G_ "New in this revision:\n")))) (_ #t))) -(define* (build-and-install source config-dir- #:key verbose? url branch commit)+(define* (build-and-install instances config-dir+ #:key verbose?) "Build the tool from SOURCE, and install it in CONFIG-DIR." (define update-profile (store-lift build-and-use-profile))@@ -232,15 +161,9 @@ URL, BRANCH, and COMMIT as a property in the manifest entry." (define profile (string-append config-dir "/current")) - (mlet* %store-monad ((drv (build-from-source source- #:commit commit- #:verbose? verbose?))- (entry (derivation->manifest-entry drv- #:url url- #:branch branch- #:commit commit)))+ (mlet %store-monad ((manifest (channel-instances->manifest instances))) (mbegin %store-monad- (update-profile profile (manifest (list entry)))+ (update-profile profile manifest) (return (display-profile-news profile))))) (define (honor-lets-encrypt-certificates! store)@@ -426,45 +349,106 @@ and ALIST2 differ, display HEADING upfront." ((numbers ...) (list-generations profile numbers))))))))) +(define (channel-list opts)+ "Return the list of channels to use. If OPTS specify a channel file,+channels are read from there; otherwise, if ~/.config/guix/channels.scm+exists, read it; otherwise %DEFAULT-CHANNELS is used. Apply channel+transformations specified in OPTS (resulting from '--url', '--commit', or+'--branch'), if any."+ (define file+ (assoc-ref opts 'channel-file))++ (define default-file+ (string-append (config-directory) "/channels.scm"))++ (define (load-channels file)+ (let ((result (load* file (make-user-module '((guix channels))))))+ (if (and (list? result) (every channel? result))+ result+ (leave (G_ "'~a' did not return a list of channels~%") file))))++ (define channels+ (cond (file+ (load-channels file))+ ((file-exists? default-file)+ (load-channels default-file))+ (else+ %default-channels)))++ (define (environment-variable)+ (match (getenv "GUIX_PULL_URL")+ (#f #f)+ (url+ (warning (G_ "The 'GUIX_PULL_URL' environment variable is deprecated.+Use '~/.config/guix/channels.scm' instead."))+ url)))++ (let ((ref (assoc-ref opts 'ref))+ (url (or (assoc-ref opts 'url)+ (environment-variable))))+ (if (or ref url)+ (match channels+ ((one)+ ;; When there's only one channel, apply '--url', '--commit', and+ ;; '--branch' to this specific channel.+ (let ((url (or url (channel-url one))))+ (list (match ref+ (('commit . commit)+ (channel (inherit one)+ (url url) (commit commit) (branch #f)))+ (('branch . branch)+ (channel (inherit one)+ (url url) (commit #f) (branch branch)))+ (#f+ (channel (inherit one) (url url)))))))+ (_+ ;; Otherwise bail out.+ (leave+ (G_ "'--url', '--commit', and '--branch' are not applicable~%"))))+ channels)))+ (define (guix-pull . args)- (define (use-le-certs? url)- (string-prefix? "https://git.savannah.gnu.org/" url))- (with-error-handling (with-git-error-handling- (let* ((opts (parse-command-line args %options- (list %default-options)))- (url (assoc-ref opts 'repository-url))- (ref (assoc-ref opts 'ref))- (cache (string-append (cache-directory) "/pull")))+ (let* ((opts (parse-command-line args %options+ (list %default-options)))+ (cache (string-append (cache-directory) "/pull"))+ (channels (channel-list opts)))+ (cond ((assoc-ref opts 'query) (process-query opts)) ((assoc-ref opts 'dry-run?) #t) ;XXX: not very useful (else (with-store store- (parameterize ((%graft? (assoc-ref opts 'graft?)))+ (parameterize ((%graft? (assoc-ref opts 'graft?))+ (%repository-cache-directory cache)) (set-build-options-from-command-line store opts) - ;; For reproducibility, always refer to the LE certificates- ;; when we know we're talking to Savannah.- (when (use-le-certs? url)- (honor-lets-encrypt-certificates! store))-- (format (current-error-port)- (G_ "Updating from Git repository at '~a'...~%")- url)-- (let-values (((checkout commit)- (latest-repository-commit store url- #:ref ref- #:cache-directory- cache)))+ ;; When certificates are already installed, use them.+ ;; Otherwise, use the Let's Encrypt certificates, which we+ ;; know Savannah uses.+ (let ((certs (or (getenv "SSL_CERT_DIR") "/etc/ssl/certs")))+ (unless (file-exists? certs)+ (honor-lets-encrypt-certificates! store))) + (let ((instances (latest-channel-instances store channels))) (format (current-error-port)- (G_ "Building from Git commit ~a...~%")- commit)+ (N_ "Building from this channel:~%"+ "Building from these channels:~%"+ (length instances)))+ (for-each (lambda (instance)+ (let ((channel+ (channel-instance-channel instance)))+ (format (current-error-port)+ " ~10a~a\t~a~%"+ (channel-name channel)+ (channel-url channel)+ (string-take+ (channel-instance-commit instance)+ 7))))+ instances) (parameterize ((%guile-for-build (package-derivation store@@ -472,13 +456,7 @@ and ALIST2 differ, display HEADING upfront." %bootstrap-guile (canonical-package guile-2.2))))) (run-with-store store- (build-and-install checkout (config-directory)- #:url url- #:branch (match ref- (('branch . branch)- branch)- (_ #f))- #:commit commit+ (build-and-install instances (config-directory) #:verbose? (assoc-ref opts 'verbose?))))))))))))) diff --git a/po/guix/POTFILES.in b/po/guix/POTFILES.inindex d11f408d4..7f881355e 100644--- a/po/guix/POTFILES.in+++ b/po/guix/POTFILES.in@@ -38,4 +38,5 @@ guix/upstream.scm guix/ui.scm guix/http-client.scm guix/nar.scm+guix/channels.scm nix/nix-daemon/guix-daemon.cc-- 2.18.0
L
L
Ludovic Courtès wrote on 31 Aug 2018 00:02
Re: bug#22629: “Stable” branch
(name . Alex Sassmannshausen)(address . alex@pompo.co)
878t4nqzqv.fsf@gnu.org
Hi Alex,
(Cc’ing https://bugs.gnu.org/32022 and https://bugs.gnu.org/26608 ,which are related.)
Alex Sassmannshausen <alex@pompo.co> skribis:
Toggle quote (5 lines)> I don't know if this is what Konrad desires, but from my perspective, a> desirable part of the definition of stable would be a that the build> farms have produced a set of binaries/substitutes for a given Guix> revision that is "good enough".
I just had a bright idea (yes!): this can be addressed by writingsomething like this in ~/.config/guix/channels.scm:
(map latest-commit-with-substitutes-available %default-channels)
The hypothetical ‘latest-commit-with-substitutes-available’ would use(git) and (guix ci) to find the latest commit for which substitutes ofinterest are available, and would return:
(channel ;; … (commit "cabbag3")) ;the ideal commit
This has to be done with great care to prevent a downgrade attack and tomake sure the user doesn’t miss out on security updates, but maybe wecould provide a procedure that makes reasonable choices.
Food for thought…
Ludo’.
K
K
Konrad Hinsen wrote on 31 Aug 2018 11:39
m1h8jaq3h0.fsf@fastmail.net
Hi Ludo,
Toggle quote (10 lines)> I just had a bright idea (yes!): this can be addressed by writing> something like this in ~/.config/guix/channels.scm:>> (map latest-commit-with-substitutes-available> %default-channels)>> The hypothetical ‘latest-commit-with-substitutes-available’ would use> (git) and (guix ci) to find the latest commit for which substitutes of> interest are available, and would return:
I really like that idea, but it's a pity to limit it to channels.Two scenarii I'd like to see covered are:
1) Find the latest commit with all substitutes required by a given manifest.
2) Find the latest commit with all substitutes required for updating a given profile.
This is in fact only one problem with two user interfaces.
Konrad.
L
L
Ludovic Courtès wrote on 31 Aug 2018 11:58
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
874lfarh6w.fsf@gnu.org
Hi Konrad,
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (12 lines)>> I just had a bright idea (yes!): this can be addressed by writing>> something like this in ~/.config/guix/channels.scm:>>>> (map latest-commit-with-substitutes-available>> %default-channels)>>>> The hypothetical ‘latest-commit-with-substitutes-available’ would use>> (git) and (guix ci) to find the latest commit for which substitutes of>> interest are available, and would return:>> I really like that idea, but it's a pity to limit it to channels.
What do you mean by “limit it to channels”? ‘%default-channels’ is analias for the official Guix channel (IOW, Guix itself.)
Toggle quote (10 lines)> Two scenarii I'd like to see covered are:>> 1) Find the latest commit with all substitutes required by a given> manifest.>> 2) Find the latest commit with all substitutes required for updating a> given profile.>> This is in fact only one problem with two user interfaces.
Yes, we could do that, and even maybe more sophisticated things (e.g.,looking at the commit log to determine whether security fixes areavailable, and adjusting the strategy accordingly.)
What I find interesting is that we can provide the tools to support suchpolicies, and then users can choose or implement the policy they wantdirectly in ~/.config/guix/channels.scm.
Ludo’.
K
K
Konrad Hinsen wrote on 31 Aug 2018 12:33
(name . Ludovic Courtès)(address . ludo@gnu.org)
m1k1o6akr4.fsf@fastmail.net
Hi Ludo,
Toggle quote (3 lines)> What do you mean by “limit it to channels”? ‘%default-channels’ is an> alias for the official Guix channel (IOW, Guix itself.)
Fine, but I rarely care about all of Guix, or all of any other channel.I care about the small subset of packages that I actually use.
Better yet, with a per-manifest/profile approach, I could put my mostcritical packages in a special profile and get updates for them morequickly, while still working only with substitutes.
BTW, just out of curiosity: for how many commits in Guix history allpackages could be built successfully? Is that the rule of the exception?
Toggle quote (4 lines)> Yes, we could do that, and even maybe more sophisticated things (e.g.,> looking at the commit log to determine whether security fixes are> available, and adjusting the strategy accordingly.)
Nice!
Toggle quote (4 lines)> What I find interesting is that we can provide the tools to support such> policies, and then users can choose or implement the policy they want> directly in ~/.config/guix/channels.scm.
I agree, it's nice to give people the tools they need to implement theirown policy.
Konrad.
J
J
Jan Nieuwenhuizen wrote on 31 Aug 2018 13:24
(name . Ludovic Courtès)(address . ludo@gnu.org)
87mut2ok2e.fsf@gnu.org
Ludovic Courtès writes:
Toggle quote (6 lines)> I just had a bright idea (yes!): this can be addressed by writing> something like this in ~/.config/guix/channels.scm:>> (map latest-commit-with-substitutes-available> %default-channels)
This is a nice idea and it makes me remember that it would be useful toprovide a way to avoid installing something that is cricitally broken,like Debian's apt-listbugs package/facility(https://packages.debian.org/sid/apt-listbugs).
janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
R
R
Ricardo Wurmus wrote on 31 Aug 2018 13:45
Re: bug#32022: bug#22629: “Stable” branch
(name . Ludovic Courtès)(address . ludo@gnu.org)
87lg8m2206.fsf@elephly.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (18 lines)> I just had a bright idea (yes!): this can be addressed by writing> something like this in ~/.config/guix/channels.scm:>> (map latest-commit-with-substitutes-available> %default-channels)>> The hypothetical ‘latest-commit-with-substitutes-available’ would use> (git) and (guix ci) to find the latest commit for which substitutes of> interest are available, and would return:>> (channel> ;; …> (commit "cabbag3")) ;the ideal commit>> This has to be done with great care to prevent a downgrade attack and to> make sure the user doesn’t miss out on security updates, but maybe we> could provide a procedure that makes reasonable choices.
This is a great idea. Any kind of fetch policy could be implementedwith this, including one that considers the contents of a manifest.
This is another of these instances where having a general purposeprogramming language underpinning it all really pays off.
--Ricardo
R
R
Ricardo Wurmus wrote on 31 Aug 2018 14:21
Re: bug#22629: [PATCH v2 2/3] Add (guix describe) and use it to initialize '%package-search-path'.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
87k1o620c5.fsf@elephly.net
Hi Ludo,
Toggle quote (7 lines)> * guix/describe.scm: New file.> * Makefile.am (MODULES): Add it.> * gnu/packages.scm (%default-package-module-path): New variable.> (%package-module-path): Honor 'package-path-entries'.> * build-aux/update-NEWS.scm (main): Use %DEFAULT-PACKAGE-MODULE-PATH> instead of (last (%package-module-path)).> ---
[…]
Toggle quote (34 lines)> +(define %default-package-module-path> + ;; Default search path for package modules.> + `((,%distro-root-directory . "gnu/packages")))> +> (define %package-module-path> ;; Search path for package modules. Each item must be either a directory> ;; name or a pair whose car is a directory and whose cdr is a sub-directory> ;; to narrow the search.> (let* ((not-colon (char-set-complement (char-set #\:)))> (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")> - not-colon)))> - ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.> - (for-each (lambda (directory)> - (set! %load-path (cons directory %load-path))> - (set! %load-compiled-path> - (cons directory %load-compiled-path)))> - environment)> + not-colon))> + (channels (package-path-entries)))> + ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's> + ;; search path. For historical reasons, $GUIX_PACKAGE_PATH goes to the> + ;; front; channels go to the back so that they don't override Guix' own> + ;; modules.> + (set! %load-path> + (append environment %load-path channels))> + (set! %load-compiled-path> + (append environment %load-compiled-path channels))>> (make-parameter> - (append environment `((,%distro-root-directory . "gnu/packages"))))))> + (append environment> + %default-package-module-path> + channels))))
I’m not sure I understand the reason to add channels to the end of thesearch path. Could it not be desirable in some use-cases to overridecertain Guix modules? Should the order be made explicit in the channelto avoid having to accomodate “historical reasons” in the future? :)
Putting them at the end probably ensures that Guix itself won’t breakand can thus be rolled back.
Toggle quote (5 lines)> diff --git a/guix/describe.scm b/guix/describe.scm> new file mode 100644> index 000000000..3122a762f> --- /dev/null> +++ b/guix/describe.scm
[…]
Toggle quote (15 lines)> +(define current-profile> + (mlambda ()> + "Return the profile (created by 'guix pull') the calling process lives in,> +or #f if this is not applicable."> + (match (command-line)> + ((program . _)> + (and (string-suffix? "/bin/guix" program)> + ;; Note: We want to do _lexical dot-dot resolution_. Using ".."> + ;; for real would instead take us into the /gnu/store directory> + ;; that ~/.config/guix/current/bin points to, whereas we want to> + ;; obtain ~/.config/guix/current.> + (let ((candidate (dirname (dirname program))))> + (and (file-exists? (string-append candidate "/manifest"))> + candidate)))))))
I don’t know… there’s something about this file system traversal thatdoesn’t sit right with me. I’m not sure about (command-line) — when…/bin/guix is executed by a wrapper, will the wrapper be the “program”that we match against or the target? (This is a concern for wrappersthat set up site-wide default channels or a remote daemon, for example.)
--Ricardo
L
L
Ludovic Courtès wrote on 31 Aug 2018 15:01
Re: bug#22629: “Stable” branch
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
87k1o6pu4f.fsf@gnu.org
Hello,
Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
Toggle quote (10 lines)>> What do you mean by “limit it to channels”? ‘%default-channels’ is an>> alias for the official Guix channel (IOW, Guix itself.)>> Fine, but I rarely care about all of Guix, or all of any other channel.> I care about the small subset of packages that I actually use.>> Better yet, with a per-manifest/profile approach, I could put my most> critical packages in a special profile and get updates for them more> quickly, while still working only with substitutes.
Sure! The hypothetical procedure I gave can perform arbitrary checks;it could be passed a manifest.
Toggle quote (3 lines)> BTW, just out of curiosity: for how many commits in Guix history all> packages could be built successfully? Is that the rule of the exception?
We never have 100% of successful builds. Of course we do our best tokeep the failure rate low, but sometimes there are unpopular packagesthat remain broken for some time, or there are packages for which weforgot to exclude some systems via ‘supported-systems’, and of coursethere’s unintended breakage.
Ludo’.
L
L
Ludovic Courtès wrote on 31 Aug 2018 15:56
Re: bug#22629: [PATCH v2 2/3] Add (guix describe) and use it to initialize '%package-search-path'.
(name . Ricardo Wurmus)(address . rekado@elephly.net)(address . 22629@debbugs.gnu.org)
87a7p2od0u.fsf@gnu.org
Hi Ricardo,
Ricardo Wurmus <rekado@elephly.net> skribis:
Toggle quote (35 lines)>> (define %package-module-path>> ;; Search path for package modules. Each item must be either a directory>> ;; name or a pair whose car is a directory and whose cdr is a sub-directory>> ;; to narrow the search.>> (let* ((not-colon (char-set-complement (char-set #\:)))>> (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")>> - not-colon)))>> - ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.>> - (for-each (lambda (directory)>> - (set! %load-path (cons directory %load-path))>> - (set! %load-compiled-path>> - (cons directory %load-compiled-path)))>> - environment)>> + not-colon))>> + (channels (package-path-entries)))>> + ;; Automatically add channels and items from $GUIX_PACKAGE_PATH to Guile's>> + ;; search path. For historical reasons, $GUIX_PACKAGE_PATH goes to the>> + ;; front; channels go to the back so that they don't override Guix' own>> + ;; modules.>> + (set! %load-path>> + (append environment %load-path channels))>> + (set! %load-compiled-path>> + (append environment %load-compiled-path channels))>>>> (make-parameter>> - (append environment `((,%distro-root-directory . "gnu/packages"))))))>> + (append environment>> + %default-package-module-path>> + channels))))>> I’m not sure I understand the reason to add channels to the end of the> search path. Could it not be desirable in some use-cases to override> certain Guix modules? Should the order be made explicit in the channel> to avoid having to accomodate “historical reasons” in the future? :)
For %load-path and %load-compiled-path, I thought it may be safer toalways keep Guix in front of the rest. It means that channels cannotoverride, say, (guix scripts package) or (guix derivations) or (gnupackages base). That’s mostly to be on the safe side, and because Ithink this is not “the right way” to customize things; it’d be just toobrittle.
Regarding %package-module-path itself, whether channels come first ornot doesn’t actually make much of a difference at this point since‘fold-packages’ traverses everything anyway. Maybe in the future‘fold-packages’ could make some distinction though. Dunno.
Toggle quote (27 lines)>> diff --git a/guix/describe.scm b/guix/describe.scm>> new file mode 100644>> index 000000000..3122a762f>> --- /dev/null>> +++ b/guix/describe.scm> […]>> +(define current-profile>> + (mlambda ()>> + "Return the profile (created by 'guix pull') the calling process lives in,>> +or #f if this is not applicable.">> + (match (command-line)>> + ((program . _)>> + (and (string-suffix? "/bin/guix" program)>> + ;; Note: We want to do _lexical dot-dot resolution_. Using "..">> + ;; for real would instead take us into the /gnu/store directory>> + ;; that ~/.config/guix/current/bin points to, whereas we want to>> + ;; obtain ~/.config/guix/current.>> + (let ((candidate (dirname (dirname program))))>> + (and (file-exists? (string-append candidate "/manifest"))>> + candidate)))))))>> I don’t know… there’s something about this file system traversal that> doesn’t sit right with me. I’m not sure about (command-line) — when> …/bin/guix is executed by a wrapper, will the wrapper be the “program”> that we match against or the target? (This is a concern for wrappers> that set up site-wide default channels or a remote daemon, for example.)
The ‘guix’ command is a script starting with:
#!/gnu/store/…-guile-2.2.4/bin/guile --no-auto-compile
‘guile’ receives the ‘guix’ file name as its argv[1]. Since the ‘guix’file name was passed as the first argument to ‘execve’, it isnecessarily valid (either it’s relative to $PWD or, in the likely casewhere ‘guix’ was searched for in $PATH, it’s an absolute file name.)
In addition (ice-9 command-line) arranges to make the first non-hyphenargument the first element of what ‘command-line’ returns.
If you have a wrapper that execs ‘guix’ or whatever, it’ll still work.Of course, the trick doesn’t work if you do things like:
guile -L ~/.config/guix/current/share/guile/site/2.2
but I think that’s OK.
(I hadn’t realized the trick about execve’s first argument becoming avalid file name in the interpreter’s argv[1]. Handy!)
Ludo’.
R
R
Ricardo Wurmus wrote on 31 Aug 2018 16:32
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22629@debbugs.gnu.org)
87d0ty1u9g.fsf@elephly.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (5 lines)> Regarding %package-module-path itself, whether channels come first or> not doesn’t actually make much of a difference at this point since> ‘fold-packages’ traverses everything anyway. Maybe in the future> ‘fold-packages’ could make some distinction though. Dunno.
Ah, I see. Thanks for clarifying.
Toggle quote (20 lines)>> I don’t know… there’s something about this file system traversal that>> doesn’t sit right with me. I’m not sure about (command-line) — when>> …/bin/guix is executed by a wrapper, will the wrapper be the “program”>> that we match against or the target? (This is a concern for wrappers>> that set up site-wide default channels or a remote daemon, for example.)>> The ‘guix’ command is a script starting with:>> #!/gnu/store/…-guile-2.2.4/bin/guile --no-auto-compile>> ‘guile’ receives the ‘guix’ file name as its argv[1]. Since the ‘guix’> file name was passed as the first argument to ‘execve’, it is> necessarily valid (either it’s relative to $PWD or, in the likely case> where ‘guix’ was searched for in $PATH, it’s an absolute file name.)>> In addition (ice-9 command-line) arranges to make the first non-hyphen> argument the first element of what ‘command-line’ returns.>> If you have a wrapper that execs ‘guix’ or whatever, it’ll still work.
That’s good.
Toggle quote (6 lines)> Of course, the trick doesn’t work if you do things like:>> guile -L ~/.config/guix/current/share/guile/site/2.2>> but I think that’s OK.
Yeah, I wouldn’t worry about that.
Thanks!
-- Ricardo
L
L
Ludovic Courtès wrote on 2 Sep 2018 17:11
Re: bug#22629: Channels!
(address . 22629-done@debbugs.gnu.org)
87h8j86ik1.fsf@gnu.org
Hello,
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (15 lines)> ludo@gnu.org (Ludovic Courtès) skribis:>>> The patches that follow implement this last bit, though in a slightly>> different way. Users would now have the option to provide>> ~/.config/guix/channels.scm along these lines:>>>> (cons (channel>> (name 'guix-hpc)>> (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")>> (branch "origin/master"))>> %default-channels)>> What follows is version 2 of the patches, which I’d like to push within> a day or two. The main changes are:
I fixed a typo that would lead ‘--url’ to be ignored and pushed.Please report any problems you may have!
I’m closing this 2.5-year old issue now. :-)
Ludo’.
Closed
A
A
Alex Sassmannshausen wrote on 3 Sep 2018 16:10
Re: bug#22629: “Stable” branch
(name . Ludovic Courtès)(address . ludo@gnu.org)
87r2iau0wz.fsf@pompo.co
Hi Ludo,
Ludovic Courtès writes:
Toggle quote (26 lines)> Hi Alex,>> (Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,> which are related.)>> Alex Sassmannshausen <alex@pompo.co> skribis:>>> I don't know if this is what Konrad desires, but from my perspective, a>> desirable part of the definition of stable would be a that the build>> farms have produced a set of binaries/substitutes for a given Guix>> revision that is "good enough".>> I just had a bright idea (yes!): this can be addressed by writing> something like this in ~/.config/guix/channels.scm:>> (map latest-commit-with-substitutes-available> %default-channels)>> The hypothetical ‘latest-commit-with-substitutes-available’ would use> (git) and (guix ci) to find the latest commit for which substitutes of> interest are available, and would return:>> (channel> ;; …> (commit "cabbag3")) ;the ideal commit
This sounds incredibly interesting — and it is testament once again tothe power of Guix that this kind of solution could be feasible!
Thinking this through in my head somewhat, I had the following thoughts:- This procedure is invoked client side, where the channel is defined- That means the git searching is done client side, on every invocationof guix (I guess this might be cacheable?)- So the downside vis-a-vis a maintained "stable branch" would be aprice in performance as experienced by the end user- The upside of course would be automatic curation of a stable branchthat saves a ton of volunteer effort and work
I have no idea what the performance cost would be. I guess you woulduse "guix weather" to turn the set of requested packages into a manifestwhich can then be checked with it.
So the cost would be one of the following scenarios:Option a)- fetch set of packages in a given commit- query guix weather for 100% substitutes- iterate until a match- then perform the appropriate guix pull
Option b)- perform a guix pull to the latest commit- query guix weather for 100% substitutes- until success, step back one step at a time through guix pull
(because of the cost of guix pull this seems unfeasible)
Option c)Implement some form of substitute cache set querying on build farms, aspart of guix weather, so the 100% match is done on the build farminstead of the client.
Dunno. There may be some things that already exist in Guix land thatI'm missing.
It's a super exciting approach for sure.
Toggle quote (4 lines)> This has to be done with great care to prevent a downgrade attack and to> make sure the user doesn’t miss out on security updates, but maybe we> could provide a procedure that makes reasonable choices.
Right — so at the very least it would have to prevent us going "back intime" from the guix pull commit we are currently at.
The question of security updates is tricky at the moment already — Iwould hazard a guess that many people bail out of upgrading when theycan't get substitutes for their entire profile / system right now, whichmeans they are not getting security upgrades for package (a) when asubstitute for (b) fails.
Thanks for your thoughts — super intriguing!
Alex
L
L
Ludovic Courtès wrote on 3 Sep 2018 21:52
(name . Alex Sassmannshausen)(address . alex@pompo.co)
87zhwywe8v.fsf@gnu.org
Hi Alex,
Alex Sassmannshausen <alex@pompo.co> skribis:
Toggle quote (2 lines)> Ludovic Courtès writes:
[...]
Toggle quote (17 lines)>> I just had a bright idea (yes!): this can be addressed by writing>> something like this in ~/.config/guix/channels.scm:>>>> (map latest-commit-with-substitutes-available>> %default-channels)>>>> The hypothetical ‘latest-commit-with-substitutes-available’ would use>> (git) and (guix ci) to find the latest commit for which substitutes of>> interest are available, and would return:>>>> (channel>> ;; …>> (commit "cabbag3")) ;the ideal commit>> This sounds incredibly interesting — and it is testament once again to> the power of Guix that this kind of solution could be feasible!
Just to be clear: I don’t think this would be a substitute for a“stable” branch; rather, I view as a way to have user-defined policiessuch as “pull up to the latest commit for which there’s a substitute forIceCat.”
Toggle quote (5 lines)> Thinking this through in my head somewhat, I had the following thoughts:> - This procedure is invoked client side, where the channel is defined> - That means the git searching is done client side, on every invocation> of guix (I guess this might be cacheable?)
On every invocation of ‘guix pull’ only.
Toggle quote (4 lines)> I have no idea what the performance cost would be. I guess you would> use "guix weather" to turn the set of requested packages into a manifest> which can then be checked with it.
As I imagine it, the cost would be a few HTTP queries to the CuirassAPI. I should try to come up with an example to better explain what Ihad in mind!
Toggle quote (6 lines)> The question of security updates is tricky at the moment already — I> would hazard a guess that many people bail out of upgrading when they> can't get substitutes for their entire profile / system right now, which> means they are not getting security upgrades for package (a) when a> substitute for (b) fails.
That’s probably true, and I agree it’s problematic.
What I typically do is “guix pull && guix package -n -u”. Then I lookat things that would be built; if, say, LibreOffice is among them, Iwait for a little while and try again later, until I can get enoughsubstitutes. That usually works okay, but it fails if it turns out thatone of the dependencies fails to build: substitutes never becomeavailable in that case.
Ludo’.
L
L
Ludovic Courtès wrote on 3 Sep 2018 22:27
(name . Alex Sassmannshausen)(address . alex@pompo.co)
87h8j6wclu.fsf@gnu.org
Alex Sassmannshausen <alex@pompo.co> skribis:
Toggle quote (2 lines)> Ludovic Courtès writes:
[...]
Toggle quote (14 lines)>> I just had a bright idea (yes!): this can be addressed by writing>> something like this in ~/.config/guix/channels.scm:>>>> (map latest-commit-with-substitutes-available>> %default-channels)>>>> The hypothetical ‘latest-commit-with-substitutes-available’ would use>> (git) and (guix ci) to find the latest commit for which substitutes of>> interest are available, and would return:>>>> (channel>> ;; …>> (commit "cabbag3")) ;the ideal commit
The code below is an illustration of that. If you install it as~/.config/guix/channels.scm, ‘guix pull’ will pull the latest committhat was fully built on berlin.guixsd.org (seehttps://berlin.guixsd.org/jobset/guix-modular-master), meaning thatsubstitutes for Guix itself should be available, unless ‘guix publish’hasn’t “baked” them yet.
It takes two GETs and ~1s to do that here.
Ludo’.
(use-modules (guix http-client) (json) (srfi srfi-1) (ice-9 match)) (define (latest-evaluations jobset) "Return the latest evaluations of JOBSET." (filter (lambda (json) (string=? (hash-ref json "specification") jobset)) (json->scm (http-fetch "https://berlin.guixsd.org/api/evaluations?nr=30")))) (define (evaluation-complete? number) "Return true if evaluation NUMBER completed and all its builds were successful." (let ((builds (json->scm (http-fetch (string-append "https://berlin.guixsd.org/api/latestbuilds?nr=30&evaluation=" (number->string number)))))) (every (lambda (build) ;; Zero means build success. (= (hash-ref build "buildstatus") 0)) builds))) (define (latest-commit-successfully-built) "Return the latest commit for which substitutes are (potentially) available." (let* ((evaluations (latest-evaluations "guix-modular-master")) (candidates (filter-map (lambda (json) (match (hash-ref json "checkouts") ((checkout) (cons (hash-ref json "id") (hash-ref checkout "commit"))) (_ #f))) evaluations))) (any (match-lambda ((evaluation . commit) (and (evaluation-complete? evaluation) commit))) candidates))) ;; Pull the latest commit fully built on berlin.guixsd.org. ;; WARNING: This could downgrade your system! (list (channel (name 'guix) (url "https://git.savannah.gnu.org/git/guix.git") (commit (pk 'commit (latest-commit-successfully-built)))))
A
A
Alex Sassmannshausen wrote on 4 Sep 2018 10:02
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pnxtu1uw.fsf@pompo.co
Ludovic Courtès writes:
Toggle quote (30 lines)> Hi Alex,>> Alex Sassmannshausen <alex@pompo.co> skribis:>>> Ludovic Courtès writes:>> [...]>>>> I just had a bright idea (yes!): this can be addressed by writing>>> something like this in ~/.config/guix/channels.scm:>>>>>> (map latest-commit-with-substitutes-available>>> %default-channels)>>>>>> The hypothetical ‘latest-commit-with-substitutes-available’ would use>>> (git) and (guix ci) to find the latest commit for which substitutes of>>> interest are available, and would return:>>>>>> (channel>>> ;; …>>> (commit "cabbag3")) ;the ideal commit>>>> This sounds incredibly interesting — and it is testament once again to>> the power of Guix that this kind of solution could be feasible!>> Just to be clear: I don’t think this would be a substitute for a> “stable” branch; rather, I view as a way to have user-defined policies> such as “pull up to the latest commit for which there’s a substitute for> IceCat.”
Ah, I understand now.
So the example you provided is a user-defined policy to install thelatest version of Guix that is downloadable using substitutes (if guixpublish has published those already).
As you say, in a similar vein, the end user could for themselves definea policy that searches for a commit containing a specific successfulbuild, or a set of specific successful builds.
Toggle quote (7 lines)>> Thinking this through in my head somewhat, I had the following thoughts:>> - This procedure is invoked client side, where the channel is defined>> - That means the git searching is done client side, on every invocation>> of guix (I guess this might be cacheable?)>> On every invocation of ‘guix pull’ only.
That makes sense, and is way better than I feared :-)
Toggle quote (8 lines)>> I have no idea what the performance cost would be. I guess you would>> use "guix weather" to turn the set of requested packages into a manifest>> which can then be checked with it.>> As I imagine it, the cost would be a few HTTP queries to the Cuirass> API. I should try to come up with an example to better explain what I> had in mind!
Your example helps visualize this, thanks.
Your example depends on there being a jobset that comprises the set ofpackages you are interested in testing.
I imagine it is possible to do the same for an individual package / job.
The situation would be different if the end user wanted to perform asimilar operation for an arbitrary set of packages on their end.
It would probably involve something like this (probably naive):
(define (latest-commit-successfully-built-pkg pkg) "Return the latest commit for the pkg for which substitutes are(potentially) available." ;; Like your version, but magically performs query ;; for pkg, not the guix-modular-master evaluation (let* ((evaluations (latest-evaluations pkg)) (candidates (filter-map (lambda (json) (match (hash-ref json "checkouts") ((checkout) (cons (hash-ref json "id") (hash-ref checkout "commit"))) (_ #f))) evaluations))) (map (match-lambda ((evaluation . commit) (and (evaluation-complete? evaluation) commit))) candidates)))
(any (match-lambda ((evaluation . commit) commit) (apply lset-intersection equal? ;; Like latest-commit-successfully-built, but takes an ;; individual package name for which we return the ;; commit (map latest-commit-successfully-built-pkg %set-of-packages))))
Obviously the larger the set, the more requests are required, and thelower the chance of a commit being available / a downgrade occuring
Toggle quote (15 lines)>> The question of security updates is tricky at the moment already — I>> would hazard a guess that many people bail out of upgrading when they>> can't get substitutes for their entire profile / system right now, which>> means they are not getting security upgrades for package (a) when a>> substitute for (b) fails.>> That’s probably true, and I agree it’s problematic.>> What I typically do is “guix pull && guix package -n -u”. Then I look> at things that would be built; if, say, LibreOffice is among them, I> wait for a little while and try again later, until I can get enough> substitutes. That usually works okay, but it fails if it turns out that> one of the dependencies fails to build: substitutes never become> available in that case.
Interesting. Do you think this kind of thing might be useful to have inthe Guix manual? Like, in a section about a "typical" desktop end-usermight manage their system day to day?
Alex
L
L
Ludovic Courtès wrote on 4 Sep 2018 14:22
(name . Alex Sassmannshausen)(address . alex@pompo.co)
8736upcv04.fsf@gnu.org
Hi Alex,
Alex Sassmannshausen <alex@pompo.co> skribis:
Toggle quote (8 lines)> So the example you provided is a user-defined policy to install the> latest version of Guix that is downloadable using substitutes (if guix> publish has published those already).>> As you say, in a similar vein, the end user could for themselves define> a policy that searches for a commit containing a specific successful> build, or a set of specific successful builds.
Exactly.
Toggle quote (9 lines)>> As I imagine it, the cost would be a few HTTP queries to the Cuirass>> API. I should try to come up with an example to better explain what I>> had in mind!>> Your example helps visualize this, thanks.>> Your example depends on there being a jobset that comprises the set of> packages you are interested in testing.
Yes, and it’s hacky in that the substitute server and jobset names arehard-coded, but you get the idea.
Toggle quote (2 lines)> I imagine it is possible to do the same for an individual package / job.
Yes.
Toggle quote (3 lines)> The situation would be different if the end user wanted to perform a> similar operation for an arbitrary set of packages on their end.
It would be quite similar: you would query the set of builds of anevaluation of the “guix-modular” jobset and check whether the packagesof interest were built.
Toggle quote (11 lines)>> What I typically do is “guix pull && guix package -n -u”. Then I look>> at things that would be built; if, say, LibreOffice is among them, I>> wait for a little while and try again later, until I can get enough>> substitutes. That usually works okay, but it fails if it turns out that>> one of the dependencies fails to build: substitutes never become>> available in that case.>> Interesting. Do you think this kind of thing might be useful to have in> the Guix manual? Like, in a section about a "typical" desktop end-user> might manage their system day to day?
It would make sense to have such a section I guess. However, beforeteaching users how to work around deficiencies of our infrastructure ourprocesses ;-), I’d like us to improve them much as possible. I’m surewe have room for improvement for instance in Cuirass.
Thanks,Ludo’.
?
Your comment

This issue is archived.

To comment on this conversation send email to 22629@debbugs.gnu.org