Multi-threaded compilation of 'syntax-parameterize' forms crashes

OpenSubmitted by Leo Famulari.
Details
6 participants
  • Clément Lassieur
  • Leo Famulari
  • Ludovic Courtès
  • Marius Bakke
  • Ricardo Wurmus
  • Andy Wingo
Owner
unassigned
Severity
serious
Merged with
L
L
Leo Famulari wrote on 24 Jun 2017 18:32
Wrong type (expecting mutex)
(address . bug-guix@gnu.org)
20170624163212.GA22546@jasmine.lan
I just got this from `guix pull`:
guix pull --url=file:///gnu/store/l552m9iavw3amq5c8vaifqlxvw09r2nz-guix-latest.tar.gzunpacking '/gnu/store/l552m9iavw3amq5c8vaifqlxvw09r2nz-guix-latest.tar.gz'...updating list of substitutes from 'https://4606.nsupdate.info'... 100.0%o'... 0.0%The following derivation will be built: /gnu/store/xxy7l4jfjx6n62anfqlw4gbmafypqgrs-guix-latest.drvupdating list of substitutes from 'https://4606.nsupdate.info'... 100.0%o'... 0.0%substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%copying and compiling to '/gnu/store/r1z6nbkrl99hxppcvcprc8vgbzakv632-guix-latest' with Guile 2.2.2...loading... 25.4% of 606 filesrandom seed for tests: 1498319917loading... 99.8% of 606 filescompiling... 99.2% of 606 filesBacktrace: 2 (primitive-load "/gnu/store/v9a9cqzh41qg4sixl2mk5kndglp?")In ./guix/build/pull.scm: 181:8 1 (build-guix _ _ #:system _ #:storedir _ #:localstatedir ?)In ice-9/threads.scm: 289:22 0 (loop _)
ice-9/threads.scm:289:22: In procedure loop:ice-9/threads.scm:289:22: Wrong type (expecting mutex): (3556 . #<procedure intset-add (bs i)>)
Some deprecated features have been used. Set the environmentvariable GUILE_WARN_DEPRECATED to "detailed" and rerun theprogram to get more information. Set it to "no" to suppressthis message.builder for `/gnu/store/xxy7l4jfjx6n62anfqlw4gbmafypqgrs-guix-latest.drv' failed with exit code 1guix pull: error: build failed: build of `/gnu/store/xxy7l4jfjx6n62anfqlw4gbmafypqgrs-guix-latest.drv' failed
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAllOlAsACgkQJkb6MLrKfwiEhg/+I/BbkX/UecS4mGQoAR++PP5UGQXPSEV40QYQJrkfQjlSYnOQl29P+V7cVPKzt6jaFmimT6NrxM5cxRh4OB6AG72LzFSZtsBQIYqEh/p62h9ACUHGMhzyt/2jfqipI0JjoEq4nHOrcFWgid4Vgymx+9clQ1eXFxvg6ifSACpKe/2bQdUZrU7JUjHeAPntKfcfO9BHx9g7giBr2X0ExDdfnvYkwyKdQYQPu73kwnb/MzoLygrf5A3CekmcQ+fX3N3SfgsTjhum9+2q+PYLwY1aBFzY6ZB45LAv/jC1NRJgePvGwS6FhMaQ7TbeSZS75Ws4OlPElMdplUX/BjPf2YOlqyu2nh1FTpEBMaInEqPOEIuj6Hv+C9550PbLRk5SHNy14DqfJ9q9Q3EUsxi9KUXwflg6R0ERE5xAoXJT+SELaZa4hb6aeyHw7JPfKN9/M6unLJPF0Eta9ISmq3SbwkpdSGCTWqhFXKUiSZVfBulhfhYUAA9dx8tAsl0BgplUV4EbqGsKjvufV9dR4VzkXPzApIoYWnjdGaH5Bzoy35N3iAuusafmxk1L1zMS9+sBklIyhgwV1pVM92XtlMVEwE1rmud6JP1xAWMOKMdvX/mBDIiu8Edmfo9KXfTpTrhj/feFkc6DqP2kO7XxvWV7EtLOmCZNNLNvTrDLvspYnK20jAA==0Tn+-----END PGP SIGNATURE-----

L
L
Leo Famulari wrote on 24 Jun 2017 18:55
(address . 27476@debbugs.gnu.org)
20170624165508.GA21063@jasmine.lan
On Sat, Jun 24, 2017 at 12:32:12PM -0400, Leo Famulari wrote:
Toggle quote (10 lines)> compiling... 99.2% of 606 filesBacktrace:> 2 (primitive-load "/gnu/store/v9a9cqzh41qg4sixl2mk5kndglp?")> In ./guix/build/pull.scm:> 181:8 1 (build-guix _ _ #:system _ #:storedir _ #:localstatedir ?)> In ice-9/threads.scm:> 289:22 0 (loop _)> > ice-9/threads.scm:289:22: In procedure loop:> ice-9/threads.scm:289:22: Wrong type (expecting mutex): (3556 . #<procedure intset-add (bs i)>)
It didn't happen when I re-ran the command.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAllOmWwACgkQJkb6MLrKfwjyww//V0K5yStbzjftvKHeMnTSPBUTU2/utmx9lnfKRznSPQ3ldl7Ic5Vpiw6UB9nL6A+Akm3Ev4D0FSd5j7gQqrkIsfcS46NpPvSINUgBz8iTi4qpRzGtpLp6cyuDfGkI6w09UpdFndYus3gJN6xpCGzsvOT1lLKI9eUkB8H4lFbKVapHsOeDwNRda+7HEEABtkbygcd/vtc9MpioT8d2rN0rMuTEe/yKxZ+me+3+L/wwYoH0DKlvnGTy0zwjLbErBC8zh09UtvDJiHfmDKXFoU3Pgpf87fX0KjcDC/SpcHl7lfq071U1FMleUDdS6h34eB5anXrZHDoPifIDXOZKl1B/w5fpPrYPzCN/jjGs10IFCkGRFeag1ZRNYhh95FSpSGiiAvnKdCvz4LJzXH60KAzIrsTCXpC97OMsp3dbL7/N4vMTwErH8P3KhiVPA36tHV8tVbAYhueM68YGFZ5nl+9ZQd+GTwCA6D3WEgrfA5s707cgBIefzgm9wahlVRHvWS+b16yTjEuc8ZXGk5KsuqBoT7PzYBAzC9vByFCE/lIEieXcijvJyPcGSb3ZhJqUKfSN1Q1H0EzRn2La+fxxAU1dzN6cNAvuIL3ur6UPo3zBGvHKlSIdIkBBKoDADvDBNjKtpNh7s2z9RTfPetxkiF24lYvbqzbNCWItYWy92dWpTKM==YvSI-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 26 Aug 2017 16:12
control message for bug #27476
(address . control@debbugs.gnu.org)
877exqa04o.fsf@gnu.org
severity 27476 important
L
L
Ludovic Courtès wrote on 26 Aug 2017 16:14
(address . control@debbugs.gnu.org)
874lsua01r.fsf@gnu.org
merge 27476 27652
L
L
Ludovic Courtès wrote on 26 Aug 2017 16:18
(address . control@debbugs.gnu.org)
87ziam8lbb.fsf@gnu.org
retitle 27476 Compilation with 'guix pull' crashes non-deterministically on many-core machines
L
L
Ludovic Courtès wrote on 22 Sep 2017 16:10
Re: guix pull fails on powerful server
87377esu1a.fsf@gnu.org
Hi Ricardo,
Ricardo Wurmus <rekado@elephly.net> skribis:
Toggle quote (11 lines)> The following derivation will be built:> /gnu/store/yvyfkns3w3vm7ynwbr7mvxcmin4gd2a0-guix-latest.drv> copying and compiling to '/gnu/store/7m52dkr98nhwgpsx20mmpwyw2yzj58d3-guix-latest' with Guile 2.2.2...> loading... 25.4% of 629 filesrandom seed for tests: 1506066913> loading... 99.8% of 629 files> compiling... 69.2% of 629 filesice-9/threads.scm:289:22: In procedure loop:> ice-9/threads.scm:289:22: Syntax error:> guix/scripts.scm:130:2: >>=: >>= (bind) used outside of 'with-monad' in form (>>= (apply set-build-options* #:use-substitutes> ptions)) (lambda (unused-value) (mbegin %store-monad (mlet %store-monad ((derivation (origin->derivation (package-source pack> tutes? use-substitutes? #:dry-run? dry-run?) (return (show-derivation-outputs derivation)))))))
This was reported at https://bugs.gnu.org/27476, and I suspect athread-safety issue. However, syntax parameters are purely functionalAFAICS, so I fail to see why multithreading could be a problem.
Andy, any idea what could be causing this?
Thanks,Ludo’.
A
A
Andy Wingo wrote on 25 Sep 2017 09:27
(name . Ludovic Courtès)(address . ludo@gnu.org)
87k20nz18u.fsf@igalia.com
On Fri 22 Sep 2017 16:10, ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (21 lines)> Hi Ricardo,>> Ricardo Wurmus <rekado@elephly.net> skribis:>>> The following derivation will be built:>> /gnu/store/yvyfkns3w3vm7ynwbr7mvxcmin4gd2a0-guix-latest.drv>> copying and compiling to '/gnu/store/7m52dkr98nhwgpsx20mmpwyw2yzj58d3-guix-latest' with Guile 2.2.2...>> loading... 25.4% of 629 filesrandom seed for tests: 1506066913>> loading... 99.8% of 629 files>> compiling... 69.2% of 629 filesice-9/threads.scm:289:22: In procedure loop:>> ice-9/threads.scm:289:22: Syntax error:>> guix/scripts.scm:130:2: >>=: >>= (bind) used outside of 'with-monad' in form (>>= (apply set-build-options* #:use-substitutes>> ptions)) (lambda (unused-value) (mbegin %store-monad (mlet %store-monad ((derivation (origin->derivation (package-source pack>> tutes? use-substitutes? #:dry-run? dry-run?) (return (show-derivation-outputs derivation)))))))>> This was reported at <https://bugs.gnu.org/27476>, and I suspect a> thread-safety issue. However, syntax parameters are purely functional> AFAICS, so I fail to see why multithreading could be a problem.>> Andy, any idea what could be causing this?
I have heard of but not seen a number of similar bugs: errors that"can't happen" but which appear under multiple threads. I don't knowwhat underlying pattern is. Has anyone found a test case that reliablyreproduces?
Andy
C
C
Clément Lassieur wrote on 25 Sep 2017 10:43
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87k20nrwwr.fsf@lassieur.org
I got it too for the first time yesterday on my 128G RAM and 32 CPUcores server:
Toggle snippet (21 lines)substitute: updating list of substitutes from [...]Updating from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...Building from Git commit 3140844e33254316348135b03762eaeb04764544...substitute: updating list of substitutes from [...]The following derivation will be built: /gnu/store/7143x1dd2r5kch8dldyylz1ljhp3nird-guix-latest.drvcopying and compiling to '/gnu/store/8a42yc4yxslrr1hf7wk5x5mddbs76yqm-guix-latest' with Guile 2.2.2...loading... 25.3% of 632 filesrandom seed for tests: 1506279202loading... 99.8% of 632 filescompiling... 94.8% of 632 filesice-9/threads.scm:289:22: In procedure loop:ice-9/threads.scm:289:22: Syntax error:guix/gexp.scm:530:8: return: return used outside of 'with-monad' in form (return output)
Some deprecated features have been used. Set the environmentvariable GUILE_WARN_DEPRECATED to "detailed" and rerun theprogram to get more information. Set it to "no" to suppressthis message.builder for `/gnu/store/7143x1dd2r5kch8dldyylz1ljhp3nird-guix-latest.drv' failed with exit code 1guix pull: error: build failed: build of `/gnu/store/7143x1dd2r5kch8dldyylz1ljhp3nird-guix-latest.drv' failed
And then, today, again:
Toggle snippet (20 lines)substitute: updating list of substitutes from [...]Updating from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...Building from Git commit 66660960ba75233ae5b6c539f43d97d06d64e9ad...substitute: updating list of substitutes from [...]The following derivation will be built: /gnu/store/dmv64icdan1fqrl00czgwx1an923fzda-guix-latest.drvcopying and compiling to '/gnu/store/slqcrr5gwhi1zqv21wjp0l973zs3dywc-guix-latest' with Guile 2.2.2...loading... 25.3% of 632 filesrandom seed for tests: 1506327995loading... 99.8% of 632 filescompiling... 94.8% of 632 filesice-9/threads.scm:289:22: In procedure loop:ice-9/threads.scm:289:22: Syntax error:guix/gexp.scm:539:10: return: return used outside of 'with-monad' in form (return (derivation->output-path drv))
Some deprecated features have been used. Set the environmentvariable GUILE_WARN_DEPRECATED to "detailed" and rerun theprogram to get more information. Set it to "no" to suppressthis message.builder for `/gnu/store/dmv64icdan1fqrl00czgwx1an923fzda-guix-latest.drv' failed with exit code 1guix pull: error: build failed: build of `/gnu/store/dmv64icdan1fqrl00czgwx1an923fzda-guix-latest.drv' failed
L
L
Ludovic Courtès wrote on 25 Sep 2017 15:03
(name . Andy Wingo)(address . wingo@igalia.com)
87a81jj5gg.fsf@gnu.org
Hi,
Andy Wingo <wingo@igalia.com> skribis:
Toggle quote (28 lines)> On Fri 22 Sep 2017 16:10, ludo@gnu.org (Ludovic Courtès) writes:>>> Hi Ricardo,>>>> Ricardo Wurmus <rekado@elephly.net> skribis:>>>>> The following derivation will be built:>>> /gnu/store/yvyfkns3w3vm7ynwbr7mvxcmin4gd2a0-guix-latest.drv>>> copying and compiling to '/gnu/store/7m52dkr98nhwgpsx20mmpwyw2yzj58d3-guix-latest' with Guile 2.2.2...>>> loading... 25.4% of 629 filesrandom seed for tests: 1506066913>>> loading... 99.8% of 629 files>>> compiling... 69.2% of 629 filesice-9/threads.scm:289:22: In procedure loop:>>> ice-9/threads.scm:289:22: Syntax error:>>> guix/scripts.scm:130:2: >>=: >>= (bind) used outside of 'with-monad' in form (>>= (apply set-build-options* #:use-substitutes>>> ptions)) (lambda (unused-value) (mbegin %store-monad (mlet %store-monad ((derivation (origin->derivation (package-source pack>>> tutes? use-substitutes? #:dry-run? dry-run?) (return (show-derivation-outputs derivation)))))))>>>> This was reported at <https://bugs.gnu.org/27476>, and I suspect a>> thread-safety issue. However, syntax parameters are purely functional>> AFAICS, so I fail to see why multithreading could be a problem.>>>> Andy, any idea what could be causing this?>> I have heard of but not seen a number of similar bugs: errors that> "can't happen" but which appear under multiple threads. I don't know> what underlying pattern is. Has anyone found a test case that reliably> reproduces?
With this program:
Toggle snippet (21 lines)(use-modules (ice-9 threads) (srfi srfi-1))
(define-syntax-parameter foo (identifier-syntax +))
(define threads (unfold (lambda (x) (> x 100)) (lambda (x) (call-with-new-thread (lambda () (while #t (macroexpand '(syntax-parameterize ((foo (identifier-syntax -))) (foo y z))))))) 1+ 0))
(for-each join-thread threads)
I managed to get a segfault:
Toggle snippet (27 lines)$ guile syntax-parms.scm;;; note: source file /home/ludo/src/guix/syntax-parms.scm;;; newer than compiled /home/ludo/.cache/guile/ccache/2.2-LE-8-3.A/home/ludo/src/guix/syntax-parms.scm.go;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0;;; or pass the --no-auto-compile argument to disable.;;; compiling /home/ludo/src/guix/syntax-parms.scm;;; compiled /home/ludo/.cache/guile/ccache/2.2-LE-8-3.A/home/ludo/src/guix/syntax-parms.scm.goIn /home/ludo/src/guix/syntax-parms.scm: 13:17 13 (_)In ice-9/psyntax.scm: 1233:22 12 (expand-top-sequence (#(ribcage #(x) #((m-1dff1b83541ce327-7f97c #)) #) # …) …)In ice-9/boot-9.scm: 230:17 11 (map1 (#<tree-il (lambda-case (((id) #f #f #f () (id-1dff1b83541ce327-7fb4…>))In ice-9/psyntax.scm: 2053:19 10 (_ _ #() (foo y z) ())In ice-9/boot-9.scm: 230:17 9 (map1 #())In ice-9/psyntax.scm: 1408:12 8 (_ _) 1788:11 7 (lp (1) (11 0 . 0)) 1678:45 6 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 230:17 5 (map1 ((tmp-1dff1b83541ce327-7f98b 0 . 0)))In ice-9/psyntax.scm: 2701:67 4 Adres-eraro
… but then I failed to reproduce it again (that was on my 4-threadlaptop).
Ludo’.
R
R
Ricardo Wurmus wrote on 25 Sep 2017 16:02
(name . Ludovic Courtès)(address . ludo@gnu.org)
87bmlyzxj7.fsf@elephly.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (24 lines)> With this program:>> --8<---------------cut here---------------start------------->8---> (use-modules (ice-9 threads)> (srfi srfi-1))>> (define-syntax-parameter foo> (identifier-syntax +))>> (define threads> (unfold (lambda (x) (> x 100))> (lambda (x)> (call-with-new-thread> (lambda ()> (while #t> (macroexpand> '(syntax-parameterize ((foo (identifier-syntax -)))> (foo y z)))))))> 1+> 0))>> (for-each join-thread threads)> --8<---------------cut here---------------end--------------->8---
I have tried this programme on my 16 core 32G workstation, and on the192 core 1.5T server, but could not get it to segfault.
-- Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
R
R
Ricardo Wurmus wrote on 30 Sep 2017 09:59
(name . Ludovic Courtès)(address . ludo@gnu.org)
87shf44ny0.fsf@elephly.net
I’ve tried “guix pull” on the same server again, this time limiting CPUswith “taskset -c 0 guix pull”:
Toggle snippet (27 lines)$ taskset -c 0 guix pullguile: warning: failed to install localewarning: failed to install locale: Invalid argumentsubstitute: guile: warning: failed to install localesubstitute: warning: failed to install locale: Invalid argument
Starting download of /tmp/guix-file.QleIQRFrom https://git.savannah.gnu.org/cgit/guix.git/snapshot/master.tar.gz... ....tar.gz 1.6MiB/s 00:09 | 13.6MiB transferredunpacking '/gnu/store/g5246hzsj9vv1fmigdd7fh0060fybnbd-guix-latest.tar.gz'...The following derivation will be built: /gnu/store/z5bhk17nxmdhvj0g4cy038p25mzh1gys-guix-latest.drvcopying and compiling to '/gnu/store/s3s7xlqa10mvf8v0ypxz8gzw3lcf1x5z-guix-latest' with Guile 2.2.2...loading... 25.7% of 635 filesrandom seed for tests: 1506720257loading... 99.8% of 635 filescompiling... 69.1% of 635 filesice-9/threads.scm:289:22: In procedure loop:ice-9/threads.scm:289:22: Syntax error:guix/scripts/graph.scm:103:10: return: return used outside of 'with-monad' in form (return (package-node-edges a))
Some deprecated features have been used. Set the environmentvariable GUILE_WARN_DEPRECATED to "detailed" and rerun theprogram to get more information. Set it to "no" to suppressthis message.builder for `/gnu/store/z5bhk17nxmdhvj0g4cy038p25mzh1gys-guix-latest.drv' failed with exit code 1guix pull: error: build failed: build of `/gnu/store/z5bhk17nxmdhvj0g4cy038p25mzh1gys-guix-latest.drv' failed
After limiting memory with “ulimit -Sv 5000000”:
Toggle snippet (12 lines)ice-9/threads.scm:289:22: In procedure loop:ice-9/threads.scm:289:22: Syntax error:guix/scripts/pull.scm:192:8: >>=: >>= (bind) used outside of 'with-monad' in form (>>= (indirect-root-added latest) (lambda (done) (mlet* %store-monad () (if (and (file-exists? latest) (string=? (readlink latest) source-dir)) (begin (display (G_ "Guix already up to date\n")) (return #t)) (begin (switch-symlinks latest source-dir) (format #t (G_ "updated ~a successfully deployed under `~a'~%") %guix-package-name latest) (return #t))))))
Some deprecated features have been used. Set the environmentvariable GUILE_WARN_DEPRECATED to "detailed" and rerun theprogram to get more information. Set it to "no" to suppressthis message.builder for `/gnu/store/afji58647yzz7cr9dvlj87sd3ad36lbk-guix-latest.drv' failed with exit code 1guix pull: error: build failed: build of `/gnu/store/afji58647yzz7cr9dvlj87sd3ad36lbk-guix-latest.drv' failed
It always crashes at around 69%.
Is there another work-around I could try on this machine?
--Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
M
M
Marius Bakke wrote on 3 Oct 2017 22:29
87a818dlgz.fsf@fastmail.com
Ricardo Wurmus <rekado@elephly.net> writes:
Toggle quote (2 lines)> Is there another work-around I could try on this machine?
Using Guile 2.0 worked for me:
guix package -r guix guile-git -i guile2.0-guix guile2.0-git
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlnT8wwACgkQoqBt8qM6VPr7wQgAl78ceQ9rCYxUbent9nvxNXu6qQuHYnPqK4LHIgR3yQ3QV407vM5hjb7ArBTw6dQYqzOuCTdiEXDxolBvvS0tQabBnlkrURp6LpkLtEiiXOjd+0ahHXEP+y+BBFc0n+VkA5B6DuAlSbTVt/c1MKux2YGLUoYMIlvKFZgx+iSyUzRhWKoHkwYwtedpOJYuKREmz7aWw/W3ce9noI+IdD3qPfkSAPxpiHQy21DXm5/grg/SbQ0VCajXP4vLc3wkPqdwRMnwp677Dl7hKuiF3xu/k1SkzSmhFtU4KUlQyOTNPRjxG2IhqLeDrtkpJjr58BrNgCV+7XBoxSL59+LrrEH70Q===95WG-----END PGP SIGNATURE-----
R
R
Ricardo Wurmus wrote on 4 Oct 2017 15:15
(name . Marius Bakke)(address . mbakke@fastmail.com)
877ewb3vg1.fsf@elephly.net
Marius Bakke <mbakke@fastmail.com> writes:
Toggle quote (8 lines)> Ricardo Wurmus <rekado@elephly.net> writes:>>> Is there another work-around I could try on this machine?>> Using Guile 2.0 worked for me:>> guix package -r guix guile-git -i guile2.0-guix guile2.0-git
Unfortunately, this didn’t work for me. I tried this:
guix package -i guile2.0-guix --with-source=/path/to/guix/checkout
This printed a lot of repetitions of “warning: unknown warning type`macro-use-before-definition'” and eventually failed with thisbacktrace:
Toggle snippet (31 lines)Backtrace:In ice-9/boot-9.scm: 160: 9 [catch #t #<catch-closure 8c6c00> ...]In unknown file: ?: 8 [apply-smob/1 #<catch-closure 8c6c00>]In ice-9/boot-9.scm: 66: 7 [call-with-prompt prompt0 ...]In ice-9/eval.scm: 432: 6 [eval # #]In ice-9/boot-9.scm:2412: 5 [save-module-excursion #<procedure 9057c0 at ice-9/boot-9.scm:4084:3 ()>]4091: 4 [#<procedure 9057c0 at ice-9/boot-9.scm:4084:3 ()>]1734: 3 [%start-stack load-stack #<procedure 916b60 at ice-9/boot-9.scm:4080:10 ()>]1739: 2 [#<procedure 9176c0 ()>]In unknown file: ?: 1 [primitive-load "/tmp/guix-build-guile2.0-guix-0.13.0-4.f1ddfe4.drv-0/source/./build-aux/compile-all.scm"]In ice-9/threads.scm: 99: 0 [loop (("guix/base16.scm" "guix/base32.scm" "guix/base64.scm" ...))]
ice-9/threads.scm:99:22: In procedure loop:ice-9/threads.scm:99:22: In procedure fport_write: Bad addressmake[2]: *** [Makefile:5252: make-go] Error 1make[2]: Leaving directory '/tmp/guix-build-guile2.0-guix-0.13.0-4.f1ddfe4.drv-0/source'make[1]: *** [Makefile:4383: all-recursive] Error 1make[1]: Leaving directory '/tmp/guix-build-guile2.0-guix-0.13.0-4.f1ddfe4.drv-0/source'make: *** [Makefile:2973: all] Error 2phase `build' failed after 149.5 secondsbuilder for `/gnu/store/aqz4d2bbihvdmxqb6rlb71c853jb4dp3-guile2.0-guix-0.13.0-4.f1ddfe4.drv' failed with exit code 1guix package: error: build failed: build of `/gnu/store/aqz4d2bbihvdmxqb6rlb71c853jb4dp3-guile2.0-guix-0.13.0-4.f1ddfe4.drv
--Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
C
C
Clément Lassieur wrote on 4 Oct 2017 17:09
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87o9pnt0ei.fsf@lassieur.org
Ricardo Wurmus <rekado@elephly.net> writes:
Toggle quote (2 lines)> Is there another work-around I could try on this machine?
My workaround was to build Guix from sources. But I'm sure you thoughtabout it.
R
R
Ricardo Wurmus wrote on 4 Oct 2017 18:17
(name . Clément Lassieur)(address . clement@lassieur.org)
874lre51l6.fsf@elephly.net
Clément Lassieur <clement@lassieur.org> writes:
Toggle quote (7 lines)> Ricardo Wurmus <rekado@elephly.net> writes:>>> Is there another work-around I could try on this machine?>> My workaround was to build Guix from sources. But I'm sure you thought> about it.
Yes, that works, but I was looking for something that a user can do whowouldn’t like to fiddle with git.
-- Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
L
L
Ludovic Courtès wrote on 7 Oct 2017 17:11
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87po9zj8lt.fsf@gnu.org
Ricardo Wurmus <rekado@elephly.net> skribis:
Toggle quote (3 lines)> I’ve tried “guix pull” on the same server again, this time limiting CPUs> with “taskset -c 0 guix pull”:
As a stopgap, commit aba219af0fed6a349af930f19c913fb87e6a69dd ensuresthat ‘--cores’ is honored. So if you run “guix pull --cores=1”, it willbuild things sequentially.
Now, to take advantage of that, you first need to update to the currentGuix…
Ludo’.
R
R
Ricardo Wurmus wrote on 10 Oct 2017 09:17
(name . Ludovic Courtès)(address . ludo@gnu.org)
878tgjmpz6.fsf@elephly.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (12 lines)> Ricardo Wurmus <rekado@elephly.net> skribis:>>> I’ve tried “guix pull” on the same server again, this time limiting CPUs>> with “taskset -c 0 guix pull”:>> As a stopgap, commit aba219af0fed6a349af930f19c913fb87e6a69dd ensures> that ‘--cores’ is honored. So if you run “guix pull --cores=1”, it will> build things sequentially.>> Now, to take advantage of that, you first need to update to the current> Guix…
Thank you, this worked!
On that server I built Guix from source and then let the users pull with“--cores=1” to update their own Guix.
-- Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
L
L
Ludovic Courtès wrote on 10 Oct 2017 13:32
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87wp43nsq9.fsf@gnu.org
Ricardo Wurmus <rekado@elephly.net> skribis:
Toggle quote (19 lines)> Ludovic Courtès <ludo@gnu.org> writes:>>> Ricardo Wurmus <rekado@elephly.net> skribis:>>>>> I’ve tried “guix pull” on the same server again, this time limiting CPUs>>> with “taskset -c 0 guix pull”:>>>> As a stopgap, commit aba219af0fed6a349af930f19c913fb87e6a69dd ensures>> that ‘--cores’ is honored. So if you run “guix pull --cores=1”, it will>> build things sequentially.>>>> Now, to take advantage of that, you first need to update to the current>> Guix…>> Thank you, this worked!>> On that server I built Guix from source and then let the users pull with> “--cores=1” to update their own Guix.
You could also run guix-daemon with --cores=4 or similar, so that ituses 4 cores by default (few package builds scale beyond that anyway),and then maybe --max-jobs=4 so you don’t waste the other cores. ;-)
Ludo’.
L
L
Ludovic Courtès wrote on 12 Oct 2017 15:37
Re: bug#27476: guix pull fails on powerful server
(name . Ricardo Wurmus)(address . rekado@elephly.net)
877ew0o5br.fsf@gnu.org
Hi!
Ricardo Wurmus <rekado@elephly.net> skribis:
Toggle quote (9 lines)> The following derivation will be built:> /gnu/store/z5bhk17nxmdhvj0g4cy038p25mzh1gys-guix-latest.drv> copying and compiling to '/gnu/store/s3s7xlqa10mvf8v0ypxz8gzw3lcf1x5z-guix-latest' with Guile 2.2.2...> loading... 25.7% of 635 filesrandom seed for tests: 1506720257> loading... 99.8% of 635 files> compiling... 69.1% of 635 filesice-9/threads.scm:289:22: In procedure loop:> ice-9/threads.scm:289:22: Syntax error:> guix/scripts/graph.scm:103:10: return: return used outside of 'with-monad' in form (return (package-node-edges a))
The program below crashes with completely surreal backtraces in lessthan a minute on my 4-thread laptop:
Toggle snippet (29 lines)(use-modules (ice-9 threads) (srfi srfi-1) (guix monads) (guix store))
(define threads (unfold (lambda (x) (> x 100)) (lambda (x) (call-with-new-thread (lambda () (define monad (symbol-append 'foo-monad (string->symbol (number->string x))))
(while #t (macroexpand `(begin (define-monad ,monad (bind +) (return -)) (with-monad ,monad (return 3)) (mapm ,monad + '(1 2 3)))))))) 1+ 0))
(for-each join-thread threads)
Can you check if that also happens on your many-core machine?
The patch below seems to fix the problem: (guix monads) has shared state(hash tables) used both at expansion-time and run-time, and it wasn’tprotected.
My hypothesis is that this was causing random memory corruption. Theweird thing, though, is that the errors we were getting were not sorandom. Also, the load phase of ‘guix pull’ is sequential.
Could you test it and report back?
Thanks,Ludo’.
Toggle diff (62 lines)diff --git a/guix/monads.scm b/guix/monads.scmindex 6ae616aca..c9c5da3bb 100644--- a/guix/monads.scm+++ b/guix/monads.scm@@ -20,6 +20,7 @@ #:use-module ((system syntax) #:select (syntax-local-binding)) #:use-module (ice-9 match)+ #:use-module (ice-9 threads) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9) #:use-module (srfi srfi-26)@@ -117,6 +118,7 @@ ;; the syntax object of the parameter over which it is templated, and (2) ;; the syntax of its body. (define-once %templates (make-hash-table))+ (define-once %template-lock (make-mutex)) (define (register-template! name param body) (hash-set! %templates name (cons param body)))@@ -139,8 +141,9 @@ template instances." (syntax-source s)) (define current-info-port- ;; Port for debugging info.- (const (%make-void-port "w")))+ ;; Port for debugging info. Return a fresh port at each call to make+ ;; sure we're thread-safe.+ (lambda () (%make-void-port "w"))) (define location-string (format #f "~a:~a:~a"@@ -204,12 +207,14 @@ template instances." ;; Search for an instance of template NAME for this ACTUAL parameter. ;; On success, expand to the identifier of the instance; otherwise ;; expand to #f.- (any (matching-instance? #'name #'actual) %template-instances))+ (with-mutex %template-lock+ (any (matching-instance? #'name #'actual) %template-instances))) ((_ exists? name actual) ;; Likewise, but return a Boolean. (let ((result (->bool- (any (matching-instance? #'name #'actual)- %template-instances))))+ (with-mutex %template-lock+ (any (matching-instance? #'name #'actual)+ %template-instances))))) (unless result (format (current-warning-port) "~a: warning: no specialization of template '~a' for '~a'~%"@@ -220,8 +225,9 @@ template instances." ;; Expand to the definitions of all the existing templates ;; specialized for ACTUAL. #`(begin- #,@(hash-map->list (cut instance-definition <> <> #'actual)- %templates))))))+ #,@(with-mutex %template-lock+ (hash-map->list (cut instance-definition <> <> #'actual)+ %templates))))))) (define-syntax define-template (lambda (s)
R
R
Ricardo Wurmus wrote on 13 Oct 2017 22:29
(name . Ludovic Courtès)(address . ludo@gnu.org)
877evyu6zu.fsf@elephly.net
Hi Ludo,
Toggle quote (46 lines)> Ricardo Wurmus <rekado@elephly.net> skribis:>>> The following derivation will be built:>> /gnu/store/z5bhk17nxmdhvj0g4cy038p25mzh1gys-guix-latest.drv>> copying and compiling to '/gnu/store/s3s7xlqa10mvf8v0ypxz8gzw3lcf1x5z-guix-latest' with Guile 2.2.2...>> loading... 25.7% of 635 filesrandom seed for tests: 1506720257>> loading... 99.8% of 635 files>> compiling... 69.1% of 635 filesice-9/threads.scm:289:22: In procedure loop:>> ice-9/threads.scm:289:22: Syntax error:>> guix/scripts/graph.scm:103:10: return: return used outside of 'with-monad' in form (return (package-node-edges a))>> The program below crashes with completely surreal backtraces in less> than a minute on my 4-thread laptop:>> --8<---------------cut here---------------start------------->8---> (use-modules (ice-9 threads)> (srfi srfi-1)> (guix monads)> (guix store))>> (define threads> (unfold (lambda (x) (> x 100))> (lambda (x)> (call-with-new-thread> (lambda ()> (define monad> (symbol-append 'foo-monad> (string->symbol (number->string x))))>> (while #t> (macroexpand> `(begin> (define-monad ,monad> (bind +)> (return -))> (with-monad ,monad> (return 3))> (mapm ,monad + '(1 2 3))))))))> 1+> 0))>> (for-each join-thread threads)> --8<---------------cut here---------------end--------------->8--->> Can you check if that also happens on your many-core machine?
It does not crash. I left it running for more than an hour (withoutcompiling) and it printed things like this:
Toggle snippet (20 lines)GC Warning: Repeated allocation of very large block (appr. size 57528320): May lead to memory leak and poor performanceGC Warning: Repeated allocation of very large block (appr. size 57528320): May lead to memory leak and poor performanceGC Warning: Repeated allocation of very large block (appr. size 57528320): May lead to memory leak and poor performanceGC Warning: Repeated allocation of very large block (appr. size 57528320): May lead to memory leak and poor performanceGC Warning: Repeated allocation of very large block (appr. size 14385152): May lead to memory leak and poor performanceGC Warning: Repeated allocation of very large block (appr. size 14385152): May lead to memory leak and poor performanceGC Warning: Repeated allocation of very large block (appr. size 57528320): May lead to memory leak and poor performanceGC Warning: Repeated allocation of very large block (appr. size 28766208): May lead to memory leak and poor performance
That’s on the machine with 1.5T RAM and 192 cores. Then I ran it againfor 10 minutes after compiling it. It did not crash.
Toggle quote (10 lines)> The patch below seems to fix the problem: (guix monads) has shared state> (hash tables) used both at expansion-time and run-time, and it wasn’t> protected.>> My hypothesis is that this was causing random memory corruption. The> weird thing, though, is that the errors we were getting were not so> random. Also, the load phase of ‘guix pull’ is sequential.>> Could you test it and report back?
I’m trying the patch right now with “guix pull”.
--Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
R
R
Ricardo Wurmus wrote on 13 Oct 2017 23:04
(name . Ludovic Courtès)(address . ludo@gnu.org)
874lr2u5cw.fsf@elephly.net
Hi Ludo,
Toggle quote (10 lines)> The patch below seems to fix the problem: (guix monads) has shared state> (hash tables) used both at expansion-time and run-time, and it wasn’t> protected.>> My hypothesis is that this was causing random memory corruption. The> weird thing, though, is that the errors we were getting were not so> random. Also, the load phase of ‘guix pull’ is sequential.>> Could you test it and report back?
This doesn’t seem to be enough to fix the problem. I patched ~/guix andran “guix pull --url=$PWD” from ~/guix:
Toggle snippet (22 lines)[rwurmus@murphy:~/guix] (335) $ time ./pre-inst-env guix pull --url=$PWDguile: warning: failed to install localewarning: failed to install locale: Invalid argumentUpdating from Git repository at '/home/rwurmus/guix'...Building from Git commit d24c69d86670bfad0c6bb147162c918e9fcdccc2...substitute: guile: warning: failed to install localesubstitute: warning: failed to install locale: Invalid argumentguix pull: warning: failed to load '(bimsb packages bioinformatics-nonfree)':ERROR: no code for module (gnu packages zip)guix pull: warning: failed to load '(bimsb packages staging)':ERROR: no code for module (gnu packages zip)substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%The following derivation will be built: /gnu/store/y54b92jj44li36743fgxzy0iagi6gb9n-guix-latest.drvcopying and compiling to '/gnu/store/p5zlw7fas26bickkqz4d68g8bxnjr14z-guix-latest' with Guile 2.2.2...loading... 25.8% of 640 filesrandom seed for tests: 1507927861loading... 99.8% of 640 filescompiling... 18.6% of 640 filesIn thread:ERROR: In procedure return: return used outside of 'with-monad'Error while printing exception.compiling... 70.0% of 640 files
The higher the percentage of completion, the slower this all gets. Ithasn’t actually finished yet, even though it has been running for over13 minutes.
--Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
R
R
Ricardo Wurmus wrote on 13 Oct 2017 23:10
(name . Ludovic Courtès)(address . ludo@gnu.org)
87376mu52y.fsf@elephly.net
I tried it again after unsetting GUIX_PACKAGE_PATH, but the results arejust as bad:
Toggle snippet (24 lines)[rwurmus@murphy:~/guix] (338) $ unset GUIX_PACKAGE_PATH[rwurmus@murphy:~/guix] (339) $ time ./pre-inst-env guix pull --url=$PWDguile: warning: failed to install localewarning: failed to install locale: Invalid argumentUpdating from Git repository at '/home/rwurmus/guix'...Building from Git commit d24c69d86670bfad0c6bb147162c918e9fcdccc2...substitute: guile: warning: failed to install localesubstitute: warning: failed to install locale: Invalid argumentsubstitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%The following derivation will be built: /gnu/store/q5sh4c1mfk396kixqdq8j0wdfwin4dsx-guix-latest.drvcopying and compiling to '/gnu/store/jzq053lg77shnysmhj4i2f2bngz2rr5b-guix-latest' with Guile 2.2.2...loading... 25.8% of 640 filesrandom seed for tests: 1507928738loading... 99.8% of 640 filescompiling... 14.4% of 640 filesIn thread:ERROR: In procedure >>=: >>= (bind) used outside of 'with-monad'Error while printing exception.compiling... 14.5% of 640 filesIn thread:ERROR: In procedure return: return used outside of 'with-monad'Error while printing exception.compiling... 17.3% of 640 filesIn thread:ERROR: In procedure return: return used outside of 'with-monad'Error while printing exception.compiling... 70.0% of 640 filesGC Warning: Repeated allocation of very large block (appr. size 28766208): May lead to memory leak and poor performance
-- Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAChttps://elephly.net
L
L
Ludovic Courtès wrote on 7 Nov 2017 11:57
(name . Ricardo Wurmus)(address . rekado@elephly.net)
878tfi9x15.fsf@gnu.org
Hi,
Ricardo Wurmus <rekado@elephly.net> skribis:
Toggle quote (15 lines)> After limiting memory with “ulimit -Sv 5000000”:>> ice-9/threads.scm:289:22: In procedure loop:> ice-9/threads.scm:289:22: Syntax error:> guix/scripts/pull.scm:192:8: >>=: >>= (bind) used outside of 'with-monad' in form (>>= (indirect-root-added latest) (lambda (done) (mlet* %store-monad () (if (and (file-exists? latest) (string=? (readlink latest) source-dir)) (begin (display (G_ "Guix already up to date\n")) (return #t)) (begin (switch-symlinks latest source-dir) (format #t (G_ "updated ~a successfully deployed under `~a'~%") %guix-package-name latest) (return #t))))))>> Some deprecated features have been used. Set the environment> variable GUILE_WARN_DEPRECATED to "detailed" and rerun the> program to get more information. Set it to "no" to suppress> this message.> builder for `/gnu/store/afji58647yzz7cr9dvlj87sd3ad36lbk-guix-latest.drv' failed with exit code 1> guix pull: error: build failed: build of `/gnu/store/afji58647yzz7cr9dvlj87sd3ad36lbk-guix-latest.drv' failed>> It always crashes at around 69%.
This gave me an idea. With this program:
Toggle snippet (32 lines)(use-modules (ice-9 threads) (srfi srfi-1) (guix monads) (guix store) (system base compile))
(compile #f) ;load modules
(define threads (unfold (lambda (x) (> x 100)) (lambda (x) (call-with-new-thread (lambda () (while #t (compile '(begin (with-monad %store-monad (>>= foo bar (return 3))) (mlet %store-monad ((x y)) (mbegin %store-monad (return x) (return y)))) #:env (current-module) #:from 'scheme #:to 'tree-il))))) 1+ 0))
(for-each join-thread threads)
I can reproduce the error:
Toggle snippet (67 lines)$ ulimit -Sv 2000000$ guile syntax-parms.scmIn ice-9/psyntax.scm: 1678:45 19 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 230:17 18 (map1 (((("placeholder" placeholder) ((#<procedure 7fe06eb84230 at guix/monads.scm:283…>) . #) …) …)))In ice-9/psyntax.scm: 1483:23 17 (_ _ _)In ice-9/boot-9.scm: 230:29 16 (map1 (#<syntax y> #<syntax (#<syntax lambda> (#<syntax t-1dff1b83541ce327-679d05c>) (#<syntax ml…>)) 230:17 15 (map1 (#<syntax (#<syntax lambda> (#<syntax t-1dff1b83541ce327-679d05c>) (#<syntax mlet*> #<syntax…>))In ice-9/psyntax.scm: 1788:11 14 (lp ((#<syntax mlet*> #<syntax %store-monad> () #<syntax (#<syntax let> ((#<syntax x> #<syntax …>)) #) 1678:45 13 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 230:17 12 (map1 (((("placeholder" placeholder) ("l-1dff1b83541ce327-67a3671" lexical . #) ("placeho…" …) …) …)))In ice-9/psyntax.scm: 1678:45 11 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 230:17 10 (map1 (((("placeholder" placeholder) ((#<procedure 7fe06eb84230 at guix/monads.scm:2…>) . #) …) . #)))In ice-9/psyntax.scm: 2337:44 9 (expand-let _ _ _ #f (hygiene guile-user) #<procedure build-let (src ids vars val-exps body-exp)> _ …) 1678:45 8 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 230:17 7 (map1 (((("placeholder" placeholder) ("l-1dff1b83541ce327-67a37b2" lexical . #) ("placeho…" …) …) …)))In ice-9/psyntax.scm: 1678:45 6 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 230:17 5 (map1 (((("placeholder" placeholder) ((#<procedure 7fe06eb84230 at guix/monads.scm:283…>) . #) …) …)))In ice-9/psyntax.scm: 1483:23 4 (_ _ _)In ice-9/boot-9.scm: 230:17 3 (map1 (#<syntax (return x)> #<syntax (#<syntax lambda> (#<syntax unused-value>) (#<syntax mbegin>…>))In ice-9/psyntax.scm: 1406:23 2 (_ _) 1347:32 1 (syntax-type ((#<syntax %store-monad> #<syntax %return>)) (("placeholder" placeholder) ((#) . #) …) …) 1558:32 0 (expand-macro #<procedure 7fe06eb87198 at guix/monads.scm:538:0 (x)> _ _ _ _ _ _)ice-9/psyntax.scm:1558:32: In procedure expand-macro:ice-9/psyntax.scm:1558:32: Syntax error:unknown location: state-return: Wrong number of arguments in form ((%store-monad %return))In syntax-parms.scm: 15:17 9 (_)In system/base/compile.scm: 255:6 8 (compile _ #:from _ #:to _ #:env _ #:opts _) 183:32 7 (compile-fold (#<procedure compile-tree-il (x e opts)>) (begin (with-monad %store-monad (>>= …)) #) …)In ice-9/boot-9.scm: 2316:4 6 (save-module-excursion #<procedure 30a63560 at language/scheme/compile-tree-il.scm:29:3 ()>)In language/scheme/compile-tree-il.scm: 31:15 5 (_)In ice-9/psyntax.scm: 1233:22 4 (expand-top-sequence ((begin (with-monad %store-monad (>>= foo bar (return 3))) (mlet # ((…)) #))) _ …)In ice-9/boot-9.scm: 230:17 3 (map1 (#<procedure 2b188c00 at ice-9/psyntax.scm:1231:24 ()> #<procedure 2b188a80 at ice-9/psynta…>))In ice-9/psyntax.scm: 1611:33 2 (parse (((("placeholder" placeholder) ((#<procedure 7fe06eb84230 at guix/monads.scm…>) . #) …) . #)) …) 1347:32 1 (syntax-type (>>= foo bar (return 3)) (("placeholder" placeholder) ((#<procedure 7fe06eb8…>) . #) …) …) 1558:32 0 (expand-macro #<procedure 30a63200 at ice-9/eval.scm:333:13 (a)> _ _ _ _ _ _)ice-9/psyntax.scm:1558:32: In procedure expand-macro:ice-9/psyntax.scm:1558:32: Syntax error:unknown location: source expression failed to match any patternGC Warning: Failed to expand heap by 28770304 bytesGC Warning: Failed to expand heap by 28770304 bytesGC Warning: Failed to expand heap by 14385152 bytesGC Warning: Out of Memory! Heap size: 919 MiB. Returning NULL!Warning: Unwind-only `out-of-memory' exception; skipping pre-unwind handler.
So it looks like Guile failing badly in ENOMEM conditions.
I can’t reproduce this with current Guile ‘stable-2.2’, following Andy’sweak-table rewrite¹, so this might have been a weak-table bug showing upunder memory pressure.
With ‘guix pull’ this was more likely to happen on your many-core serverthan on my laptop because you have more threads and thus much highermemory usage.
Ludo’.
¹ https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00051.html https://git.savannah.gnu.org/cgit/guile.git/commit/?h=stable-2.2&id=a053c0510c4a644f9453166b7b385cf30f6d3a21
L
L
Ludovic Courtès wrote on 30 Apr 2018 23:19
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87h8nstms1.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (4 lines)> I can’t reproduce this with current Guile ‘stable-2.2’, following Andy’s> weak-table rewrite¹, so this might have been a weak-table bug showing up> under memory pressure.
With Guile 2.2.3 a similar program triggers a crash very quickly:
Toggle snippet (107 lines)$ cat ../guile-debugging/syntax-parms.scm (use-modules (ice-9 threads) (srfi srfi-1) (guix monads) (guix store) (system base compile))
(compile #f) ;load modules
(define threads (unfold (lambda (x) (> x 100)) (lambda (x) (call-with-new-thread (lambda () (while #t (compile '(mlet %store-monad ((x y)) (mbegin %store-monad (return x) (return y))) #:env (current-module) #:from 'scheme #:to 'tree-il))))) 1+ 0))
(for-each join-thread threads)$ guile --versionguile (GNU Guile) 2.2.3Copyright (C) 2017 Free Software Foundation, Inc.
License LGPLv3+: GNU LGPL 3 or later <http://gnu.org/licenses/lgpl.html>.This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.$ guile ../guile-debugging/syntax-parms.scm In ice-9/psyntax.scm: 2338:44 19 (expand-let _ _ _ #f _ #<procedure build-let (src ids vars val-exps body-exp)> _ _ _) 1679:45 18 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 17 (map1 ("-" "1dff1b83541ce327" "-" "2ad70"))In ice-9/psyntax.scm: 1679:45 16 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 15 (map1 ((#(ribcage () () ()) #(ribcage () () ()) #(ribcage #(unused-value) #((m-1dff1b83541ce327-29a7a top)) #("l-1dff1b83541ce327-2bce9")) #(ribcage () () ()) # …)))In ice-9/psyntax.scm: 1484:23 14 (_ _ _)In ice-9/boot-9.scm: 222:29 13 (map1 _) 222:17 12 (map1 ("-" "2bd28"))In ice-9/psyntax.scm: 1789:11 11 (lp _ ()) 1679:45 10 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 9 (map1 ((expand-1dff1b83541ce327-2bd31) (#<tree-il (lambda ((name . expand)) (lambda-case (((body) #f #f #f () (body-1dff1b83541ce327-2bd35)) (call (lambda () (…>) …))In ice-9/psyntax.scm: 1679:45 8 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 7 (map1 ((m-1dff1b83541ce327-2bcf4 top)))In ice-9/psyntax.scm: 1484:23 6 (_ _ _)In ice-9/boot-9.scm: 222:17 5 (map1 (#<syntax %store-monad>))In ice-9/psyntax.scm: 1407:23 4 (_ _) 1317:39 3 (syntax-type y (shift #(ribcage #(e) #((top)) #("l-680b775fb37a463-1343")) #(ribcage () () ()) #(ribcage #(xx) #((top)) #("l-680b775fb37a463-1340")) #(ribcage …)) # …) 916:15 2 (resolve-identifier y (#<syntax >>=> #<syntax (#<syntax mval> #<syntax mproc0>)>) (shift #(ribcage #(e) #((top)) #("l-680b775fb37a463-1343")) #(ribcage () () #) …) …) 833:21 1 (id-var-name y _ _) 669:4 0 (search y (() #(ribcage () () ()) #(ribcage () () ()) #(ribcage () () ()) #(ribcage () () ())) (top) (hygiene guile-user))ice-9/psyntax.scm:669:4: In procedure search:In procedure vector-ref: Wrong type argument in position 1 (expecting vector): ()In ice-9/boot-9.scm: 222:29 19 (map1 _) 222:17 18 (map1 (#f))In ice-9/psyntax.scm: 1789:11 17 (lp _ ()) 1679:45 16 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 15 (map1 (() (m-1dff1b83541ce327-8f24e top)))In ice-9/psyntax.scm: 1679:45 14 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 13 (map1 ((m-1dff1b83541ce327-8fe71 top) shift #(ribcage #(e) #((top)) #("l-680b775fb37a463-1343")) #(ribcage () () ()) #(ribcage #(xx) #((top)) #("l-680b775fb3…")) #))In ice-9/psyntax.scm: 1409:12 12 (expand-expr _ _ _ (shift #(ribcage #(e) #((top)) #("l-680b775fb37a463-1343")) #(ribcage () () ()) #(ribcage #(xx) #((top)) #("l-680b775fb37a463-1340")) #(# # …)) # …) 2054:19 11 (_ _ _ (m-1dff1b83541ce327-8fe71 top) ())In ice-9/boot-9.scm: 222:17 10 (map1 (#<syntax identifier?> (#<syntax syntax> #<syntax id>)))In ice-9/psyntax.scm: 1409:12 9 (_ _) 1789:11 8 (lp _ (#(ribcage () () ()) #(ribcage #(x) #((m-1dff1b83541ce327-8f383 top)) #("l-1dff1b83541ce327-8f40b")) #(ribcage () () ()) #(ribcage () () ()) #(ribcage # …) …)) 1679:45 7 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 6 (map1 (#(ribcage #(x) #((m-1dff1b83541ce327-8f383 top)) #("l-1dff1b83541ce327-8f40b")) #(ribcage () () ()) shift #(ribcage #(monad body) #((top) (top)) #("…" …)) …))In ice-9/psyntax.scm: 2702:67 5 (_ tmp-1dff1b83541ce327-8fa7a _ _) 2646:111 4 (gen-clause #<tree-il (lexical tmp tmp-1dff1b83541ce327-8db88)> () (#(ribcage () () ()) #(ribcage #(x) #((m-1dff1b83541ce327-8f383 top)) #("l-1dff1b83541ce3…")) …) …) 2607:69 3 (build-dispatch-call (("8fa7c" . 0)) "1dff1b83541ce327" #<tree-il (lexical tmp tmp-1dff1b83541ce327-8e71d)> #((m-1dff1b83541ce327-8f383 top)) (#f #f () (#) #<tr…> …))In ice-9/boot-9.scm: 222:17 2 (map1 ("8fa7c"))In ice-9/psyntax.scm: 2004:10 1 (gen-var _)In unknown file: 0 (symbol->string "8fa7c")ERROR: In procedure symbol->string:In procedure symbol->string: Wrong type argument in position 1 (expecting symbol): "8fa7c"
Ludo’.
L
L
Ludovic Courtès wrote on 30 Apr 2018 23:39
libguile/memoize.c is not thread safe, so syntax parameter expansion is not thread-safe
(name . Ricardo Wurmus)(address . rekado@elephly.net)
874ljstlvq.fsf_-_@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (2 lines)> With Guile 2.2.3 a similar program triggers a crash very quickly:
Even simpler:
Toggle snippet (69 lines)$ guile ../guile-debugging/syntax-parms.scm ;;; note: source file /home/ludo/src/guix/../guile-debugging/syntax-parms.scm;;; newer than compiled /home/ludo/.cache/guile/ccache/2.2-LE-8-3.A/home/ludo/src/guile-debugging/syntax-parms.scm.go;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0;;; or pass the --no-auto-compile argument to disable.;;; compiling /home/ludo/src/guix/../guile-debugging/syntax-parms.scm;;; compiled /home/ludo/.cache/guile/ccache/2.2-LE-8-3.A/home/ludo/src/guile-debugging/syntax-parms.scm.goIn /home/ludo/src/guix/../guile-debugging/syntax-parms.scm: 37:17 13 (_)In system/base/compile.scm: 255:6 12 (compile _ #:from _ #:to _ #:env _ #:opts _) 183:32 11 (compile-fold (#f syntax-violation (#<tree-il (const #f)> #<tree-il (const "source expression failed to match any pattern")> #<tree-il (lexical tmp tmp-1dff1b…>)) # …)In ice-9/boot-9.scm: 2312:4 10 (save-module-excursion #<procedure 57dd920 at ice-9/psyntax.scm:1774:12 (a b)>)In language/scheme/compile-tree-il.scm: 31:15 9 (_)In ice-9/psyntax.scm: 1234:22 8 (expand-top-sequence (#<tree-il (primcall syntax-violation (const #f) (const "source expression failed to match any pattern") (lexical tmp tmp-1dff1b83541ce327-2…>) …)In ice-9/boot-9.scm: 222:29 7 (map1 ((10 (3 #((0 . 0)) 2 (1 (11 0 . 0) (7 (3 #() 2 (10 (13 15 5 (guile) list . #f) (5 . #<syntax ->) (11 0 . 0) (11 0 . 1)) () 2) (11 0 . 0)) 10 (. #) …) …) #) …)) 222:17 6 (map1 (10 (3 #((0 . 0)) 2 (1 (11 0 . 0) (7 (3 #() 2 (10 (13 15 5 (guile) list . #f) (5 . #<syntax ->) (11 0 . 0) (11 0 . 1)) () 2) (11 0 . 0)) 10 (13 # . #) …) …) …))In ice-9/psyntax.scm: 2054:19 5 (_ _ (#<tree-il (const (_ any any))>) (param 1 2) ())In ice-9/boot-9.scm: 222:17 4 (map1 (#<tree-il (const (_ any any))>))In ice-9/psyntax.scm: 2057:27 3 (_ _) 289:10 2 (eval-local-transformer _ _)In ice-9/eval.scm: 718:15 1 (primitive-eval _)In unknown file: 0 (memoize-expression #<tree-il (lambda () (lambda-case (((() 1) #f #f #f () ((11 0 . 0))) (call (lambda () (lambda-case ((((#f . #(tmp-1dff1b83541ce327-248119)) (#t…>)ERROR: In procedure memoize-expression:In procedure vector: Wrong type argument in position 1: #(#<directory (guile-user) cd8140>) C-c C-c$ cat ../guile-debugging/syntax-parms.scm (use-modules (ice-9 threads) (srfi srfi-1) (system base compile))
(compile #f) ;load modules
(define-syntax-parameter param (lambda (s) (syntax-case s () ((_ a b) #'(+ a b)))))
(define threads (unfold (lambda (x) (> x 100)) (lambda (x) (call-with-new-thread (lambda () (while #t (compile '(begin (param 1 2) (syntax-parameterize ((param (lambda (s) (syntax-case s () ((_ a b) #'(- a b)))))) (param 1 2))) #:env (current-module) #:from 'scheme #:to 'tree-il))))) 1+ 0))
(for-each join-thread threads)
So the problem, AIUI, is that psyntax evaluates syntax parameters using‘primitive-eval’ (via ‘eval-local-transformer’), but memoization in(ice-9 eval) is not thread-safe, hence the random crashes.
Not sure what can be done. Thoughts?
Ludo’.
L
L
Ludovic Courtès wrote on 30 Apr 2018 23:40
control message for bug #27476
(address . control@debbugs.gnu.org)
8736zctltr.fsf@gnu.org
retitle 27476 Multi-threaded compilation of 'syntax-parameterize' forms crashes
L
L
Ludovic Courtès wrote on 30 Apr 2018 23:40
(address . control@debbugs.gnu.org)
871sewtltl.fsf@gnu.org
severity 27476 serious
A
A
Andy Wingo wrote on 9 May 2018 10:41
Re: libguile/memoize.c is not thread safe, so syntax parameter expansion is not thread-safe
(name . Ludovic Courtès)(address . ludo@gnu.org)
87603x6x1f.fsf@igalia.com
Hi,
On Mon 30 Apr 2018 23:39, ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (4 lines)> So the problem, AIUI, is that psyntax evaluates syntax parameters using> ‘primitive-eval’ (via ‘eval-local-transformer’), but memoization in> (ice-9 eval) is not thread-safe, hence the random crashes.
Sorry I've been a bit AWOL here... if this diagnosis is correct, thenthe problem is ultimately the memoization that needs to be thread-safe,right?
Is the memoization you are referring to the "set!" in the "lazy" form inice-9/eval.scm ? Or something else? FWIW I would not think the "set!"could be the issue, at least on x86, but who knows.
A
L
L
Ludovic Courtès wrote on 9 May 2018 11:23
(name . Andy Wingo)(address . wingo@igalia.com)
878t8tyyfk.fsf@gnu.org
Hello Andy!
Andy Wingo <wingo@igalia.com> skribis:
Toggle quote (10 lines)> On Mon 30 Apr 2018 23:39, ludo@gnu.org (Ludovic Courtès) writes:>>> So the problem, AIUI, is that psyntax evaluates syntax parameters using>> ‘primitive-eval’ (via ‘eval-local-transformer’), but memoization in>> (ice-9 eval) is not thread-safe, hence the random crashes.>> Sorry I've been a bit AWOL here... if this diagnosis is correct, then> the problem is ultimately the memoization that needs to be thread-safe,> right?
It looks like it, yes.
Toggle quote (4 lines)> Is the memoization you are referring to the "set!" in the "lazy" form in> ice-9/eval.scm ? Or something else? FWIW I would not think the "set!"> could be the issue, at least on x86, but who knows.
Actually I’m not sure exactly. ‘memoize-expression’ itself isside-effect-free, right?
Thanks for your feedback,Ludo’.
A
A
Andy Wingo wrote on 9 May 2018 12:18
(name . Ludovic Courtès)(address . ludo@gnu.org)
87lgct5e06.fsf@igalia.com
On Wed 09 May 2018 11:23, ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (7 lines)>> Is the memoization you are referring to the "set!" in the "lazy" form in>> ice-9/eval.scm ? Or something else? FWIW I would not think the "set!">> could be the issue, at least on x86, but who knows.>> Actually I’m not sure exactly. ‘memoize-expression’ itself is> side-effect-free, right?
As far as I know (and I had a look this morning), yes. It takes aTree-IL input and returns a memoized output. The internal mutation thatexists in the evaluator is just the lazy "compilation" (see theinvocations of the "lazy" form).
Of course the function being evaluated could mutate shared state aswell!
Andy
L
L
Ludovic Courtès wrote on 7 Jun 2018 18:19
control message for bug #27476
(address . control@debbugs.gnu.org)
87tvqea7s0.fsf@gnu.org
merge 27476 31740
L
L
Ludovic Courtès wrote on 22 Jan 2019 21:58
(address . control@debbugs.gnu.org)
87h8e0l88o.fsf@gnu.org
merge 27476 34112
R
R
Ricardo Wurmus wrote on 29 Jan 2019 11:07
Re: bug#27476: guix pull fails on powerful server
(name . Ludovic Courtès)(address . ludo@gnu.org)
8736pbrd4c.fsf@elephly.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (36 lines)> ludo@gnu.org (Ludovic Courtès) skribis:>>> I can’t reproduce this with current Guile ‘stable-2.2’, following Andy’s>> weak-table rewrite¹, so this might have been a weak-table bug showing up>> under memory pressure.>> With Guile 2.2.3 a similar program triggers a crash very quickly:>> --8<---------------cut here---------------start------------->8---> $ cat ../guile-debugging/syntax-parms.scm> (use-modules (ice-9 threads)> (srfi srfi-1)> (guix monads)> (guix store)> (system base compile))>> (compile #f) ;load modules>> (define threads> (unfold (lambda (x) (> x 100))> (lambda (x)> (call-with-new-thread> (lambda ()> (while #t> (compile> '(mlet %store-monad ((x y))> (mbegin %store-monad> (return x)> (return y)))> #:env (current-module)> #:from 'scheme> #:to 'tree-il)))))> 1+> 0))>> (for-each join-thread threads)
[…]
Toggle quote (2 lines)> --8<---------------cut here---------------end--------------->8---
I tried this with guile 2.2.4 on my laptop with 4 CPUs (according tolscpu) and I did not get a crash even after waiting for several minutes.
--Ricardo
L
L
Ludovic Courtès wrote on 6 Feb 2019 14:20
control message for bug #27476
(address . control@debbugs.gnu.org)
875ztxawag.fsf@gnu.org
merge 27476 34319
L
L
Ludovic Courtès wrote on 6 Feb 2019 15:48
Re: bug#27476: libguile/memoize.c is not thread safe, so syntax parameter expansion is not thread-safe
(name . Andy Wingo)(address . wingo@igalia.com)
87lg2t7z1o.fsf@gnu.org
Hello Andy!
Since guix-core.drv is the best reproducer I have so far for this syntaxparameter crash, I modified (guix self) to print the name of the filesit’s compiling, and here’s the crash I got (on a 24-core machine):
Toggle snippet (53 lines)building /gnu/store/hf324mhj5607hh2izb01dzhwakmn8am8-guix-core.drv...[ 39/ 78] loading... 100.0% of 39 filesbuilding "guix/config.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix/monad-repl.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix/store.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix/utils.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix/memoization.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix/profiling.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix/build/utils.scm"[ 39/ 78] compiling... 0.0% of 39 filesbuilding "guix/build/syscalls.scm"[ 40/ 78] compiling... 2.6% of 39 filesbuilding "guix/deprecation.scm"[ 41/ 78] compiling... 5.1% of 39 filesbuilding "guix/i18n.scm"[ 42/ 78] compiling... 7.7% of 39 filesbuilding "guix/serialization.scm"[ 43/ 78] compiling... 10.3% of 39 filesbuilding "guix/combinators.scm"[ 44/ 78] compiling... 12.8% of 39 filesbuilding "guix/monads.scm"[ 45/ 78] compiling... 15.4% of 39 filesbuilding "guix/records.scm"[ 46/ 78] compiling... 17.9% of 39 filesIn ice-9/psyntax.scm: 2338:44 19 (expand-let _ _ _ ((line . 447) (column . 6) (filename . "./guix/monads.scm")) _ #<procedure build-let (src ids vars val-exps body-exp)> _ _ _) 1679:45 18 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 17 (map1 (((("placeholder" placeholder) ("l-10a3c941d34314a1-4889" lexical . failure-10a3c941d34314a1-488a) ("placeholder" placeholder) ("placeholder" placeholder) ("l-10a3c9?" . #) ?) ?)))In ice-9/psyntax.scm: 1409:12 16 (_ _ _ #<syntax (#<syntax failure>)>) 2338:44 15 (expand-let _ _ _ ((line . 447) (column . 6) (filename . "./guix/monads.scm")) (hygiene guix monads) #<procedure build-let (src ids vars val-exps body-exp)> _ _ ((#<syntax match-o?> ?))) 1679:45 14 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 13 (map1 (((("l-10a3c941d34314a1-4894" macro . #<procedure 460ab80 at ice-9/eval.scm:333:13 (a)>) ("placeholder" placeholder) ("l-10a3c941d34314a1-488f" lexical . #) ("l-10?" . #) ?) . #)))In ice-9/psyntax.scm: 2338:44 12 (expand-let _ _ _ ((line . 447) (column . 6) (filename . "./guix/monads.scm")) (hygiene guix monads) #<procedure build-let (src ids vars val-exps body-exp)> _ _ ((#<syntax match-o?> ?))) 1679:45 11 (parse _ _ _ _ _ _ _)In ice-9/boot-9.scm: 222:17 10 (map1 (((("l-10a3c941d34314a1-48b0" macro . #<procedure 40f9b40 at ice-9/eval.scm:333:13 (a)>) ("placeholder" placeholder) ("l-10a3c941d34314a1-48ac" lexical . #) ("l-10?" . #) ?) . #)))In ice-9/psyntax.scm: 2338:44 9 (expand-let _ _ _ ((line . 447) (column . 6) (filename . "./guix/monads.scm")) (hygiene guix monads) #<procedure build-let (src ids vars val-exps body-exp)> _ _ ((#<syntax match-d?> ?))) 1612:33 8 (parse (((("placeholder" placeholder) ("l-10a3c941d34314a1-48c8" lexical . tail-10a3c941d34314a1-48c9) ("l-10a3c941d34314a1-48b0" macro . #<procedure 40f9b40 at ice-9/eval.?>) ?) . #)) ?) 1348:32 7 (syntax-type (>>= (mproc head result) (lambda (result) (loop tail result))) (("placeholder" placeholder) ("l-10a3c941d34314a1-48c8" lexical . tail-10a3c941d34314a1-48c9) ("l-?" . #) ?) ?) 1559:32 6 (expand-macro #<procedure 17a5ba0 at ice-9/eval.scm:333:13 (a)> _ _ _ _ _ _)In ice-9/boot-9.scm: 752:25 5 (dispatch-exception _ _ _) 751:25 4 (dispatch-exception 1 syntax-error (>>= ">>= (bind) used outside of 'with-monad'" ((line . 451) (column . 9) (filename . "./guix/monads.scm")) (>>= (mproc head result) (lambda # ?)) #))In guix/build/compile.scm: 122:6 3 (_ _ . _)In ice-9/boot-9.scm: 829:9 2 (catch #t #<procedure 40f93e0 at guix/build/compile.scm:122:6 ()> #<procedure 7fffefd02888 at guix/build/compile.scm:122:6 args> _)In guix/build/compile.scm: 125:21 1 (_)In unknown file: 0 (make-stack #t)guix/build/compile.scm:125:21: Syntax error:./guix/monads.scm:452:9: >>=: >>= (bind) used outside of 'with-monad' in form (>>= (mproc head result) (lambda (result) (loop tail result)))builder for `/gnu/store/hf324mhj5607hh2izb01dzhwakmn8am8-guix-core.drv' failed with exit code 1
Here (guix monads) was already loaded before, but it’s only whencompiling (guix monads), so after it had been loaded, that we get theerror. The syntax parameter in question is defined in (guix monads)itself.
I drew the conclusion that our syntax parameter is redefined when wecompile or when we load (guix monads), so there’s a chance that we getto see the wrong value when we expand (guix monads) (I’m not entirelysure about the exact sequence of events.)
So I came up with ‘define-syntax-parameter-once’, which is like‘define-once’ but for syntax parameters (note that we can’t use‘define-once’ in ‘define-syntax-parameter-once’ because it expands to areference to NAME, which doesn’t work for a macro):
Toggle diff (26 lines)diff --git a/guix/monads.scm b/guix/monads.scmindex 6ae616aca9..1bbf79c8ba 100644--- a/guix/monads.scm+++ b/guix/monads.scm@@ -274,12 +274,20 @@ more optimizations." (_ #'generic-name)))))))))) -(define-syntax-parameter >>=+(define-syntax-rule (define-syntax-parameter-once name proc)+ (eval-when (load eval expand compile)+ (define name+ (if (module-locally-bound? (current-module) 'name)+ (module-ref (current-module) 'name)+ (make-syntax-transformer 'name 'syntax-parameter+ (list proc))))))++(define-syntax-parameter-once >>= ;; The name 'bind' is already taken, so we choose this (obscure) symbol. (lambda (s) (syntax-violation '>>= ">>= (bind) used outside of 'with-monad'" s))) -(define-syntax-parameter return+(define-syntax-parameter-once return (lambda (s) (syntax-violation 'return "return used outside of 'with-monad'" s)))
I’ve done a number of rebuilds of guix-core.drv on that 24-core machineand AFAICS that fixes the issue!
We’ll also have to use it in (guix gexp), which I’m pretty sure will fixhttps://issues.guix.info/issue/28144.
I’ll push this workaround if there are no objections.
On the Guile side, we could maybe arrange to always have ‘define-once’semantics for those bindings introduced at expansion time, as shownbelow (untested):
--- a/module/ice-9/psyntax.scm+++ b/module/ice-9/psyntax.scm@@ -296,9 +296,10 @@ (define put-global-definition-hook (lambda (symbol type val)- (module-define! (current-module)- symbol- (make-syntax-transformer symbol type val))))+ (unless (module-locally-bound? (current-module) symbol)+ (module-define! (current-module)+ symbol+ (make-syntax-transformer symbol type val))))) (define get-global-definition-hook (lambda (symbol module)
WDYT, Andy?
The discussion we had at FOSDEM turned out to be very helpful, thanksa lot!
Ludo’.
A
A
Andy Wingo wrote on 6 Feb 2019 17:14
(name . Ludovic Courtès)(address . ludo@gnu.org)
87o97o99n6.fsf@igalia.com
Hi!
On Wed 06 Feb 2019 15:48, Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (5 lines)> I drew the conclusion that our syntax parameter is redefined when we> compile or when we load (guix monads), so there’s a chance that we get> to see the wrong value when we expand (guix monads) (I’m not entirely> sure about the exact sequence of events.)
You are a wizard!!!!
To be clear, here's the series of events. Firstly, know that defining asyntax parameter is like:
(define name (make-syntax-transformer 'name 'syntax-parameter (list f)))
So at the top level you end up with an association between a name and a"syntax transformer" object (see macros.[ch]). The syntax transformerobject itself consists of its name (for debugging), its syntax type, andits syntax binding.
For syntax parameters, the binding is a list containing a singleelement, the syntax transformer. This list is later used as a key intoa compile-time environment, as it's a unique object associated with thesyntax parameter.
When (syntax-parameterize ((name f*)) exp) is seen, psyntax will look up`name' in the current expansion-time environment. It asserts that thename is bound to a syntax transformer and that the syntax transformer isindeed a syntax parameter, and extracts the associated binding `b'.Keep in bind that `b' is the single-element list containing the"default" syntax transformer `f'.
syntax-parameterize then does something weird: it adds an associationbetween the binding value `b' and `f*' to the expand-time environment.It does this because the `b' is just a fresh object, so it's a uniquekey that's usable for associations. (The way this works is my faultFWIW.) To be clear, it doesn't add a new definition of `name'; itinstead establishes a new lexical binding for the unique object `b'.
Then when a use of `name' is seen within `exp', Guile finds that `name'is a syntax parameter, extracts the binding from the syntax transformerobject, then does a second lookup of that binding. If it findssomething bound, it uses that, otherwise it uses the default binding.
I think you see the race here. For an initial state of
(define P (stx-param (list F)))
we have:
thread A thread B time resolve P | extract B | associate B and F* | | define P (stx-param (list F**)) resolve P | extract B (!) | resolve B (!) | see F** instead of F* (!) | v
Toggle quote (5 lines)> So I came up with ‘define-syntax-parameter-once’, which is like> ‘define-once’ but for syntax parameters (note that we can’t use> ‘define-once’ in ‘define-syntax-parameter-once’ because it expands to a> reference to NAME, which doesn’t work for a macro):
Your fix is good! But, it prevents redefinition of syntax parameters.
I would like to work on a solution that instead of using thisdouble-lookup, simply adds an association between P and F* in theenvironment, instead of doing the double-lookup thing. Probably thatwill be 3.0-only.
For 2.2, we can probably update the compiler to trampoline through somekind of "redefine-syntax" or something that will do (set-car! B F**)instead of (define P (stx-param B*)). I.e. redefinition keeps theunique key there.
Andy
L
L
Ludovic Courtès wrote on 6 Feb 2019 21:57
control message for bug #27476
(address . control@debbugs.gnu.org)
87h8dg8wjq.fsf@gnu.org
merge 27476 28144
L
L
Ludovic Courtès wrote on 6 Feb 2019 23:09
Re: bug#27476: libguile/memoize.c is not thread safe, so syntax parameter expansion is not thread-safe
(name . Andy Wingo)(address . wingo@igalia.com)
87va1w6030.fsf@gnu.org
Hi!
Andy Wingo <wingo@igalia.com> skribis:
Toggle quote (3 lines)> To be clear, here's the series of events. Firstly, know that defining a> syntax parameter is like:
Thanks for the clear explanation!
Toggle quote (7 lines)>> So I came up with ‘define-syntax-parameter-once’, which is like>> ‘define-once’ but for syntax parameters (note that we can’t use>> ‘define-once’ in ‘define-syntax-parameter-once’ because it expands to a>> reference to NAME, which doesn’t work for a macro):>> Your fix is good! But, it prevents redefinition of syntax parameters.
Yes. It’s acceptable in this case, so I’ve pushed it as a workaround ascommit 8245bb74fc7bdcdc2f9d458057cefc9cd982e489 in Guix.
Toggle quote (10 lines)> I would like to work on a solution that instead of using this> double-lookup, simply adds an association between P and F* in the> environment, instead of doing the double-lookup thing. Probably that> will be 3.0-only.>> For 2.2, we can probably update the compiler to trampoline through some> kind of "redefine-syntax" or something that will do (set-car! B F**)> instead of (define P (stx-param B*)). I.e. redefinition keeps the> unique key there.
Sounds good. Are you taking a look at this?
Perhaps that’d be a good excuse to release 2.2.5.
Thank you!
Ludo’.
L
L
Ludovic Courtès wrote on 9 Feb 2019 23:11
control message for bug #27476
(address . control@debbugs.gnu.org)
87sgwwpq85.fsf@gnu.org
reassign 27476 guile
?