[PATCH] gnu: Add pypy3.

  • Done
  • quality assurance status badge
Details
2 participants
  • Lars-Dominik Braun
  • Ludovic Courtès
Owner
unassigned
Submitted by
Lars-Dominik Braun
Severity
normal
L
L
Lars-Dominik Braun wrote on 23 May 2020 08:56
(address . guix-patches@gnu.org)
20200523065648.GA1418@noor.fritz.box
Hi,

the attached patch adds pypy3, a Python 3 JIT. As you can see this is a
lot of work, because pypy’s build system is rather unconventional. I’m
not adding pypy2, because Python 2 is essentially EOL.

I’ve tried fixing as many testcases as possible, but there are some I
don’t know how to fix. Thus I’m disabling the test suite, although
everything is in place to run it. pypy itself works fine though, the
build is reproducible and passes `guix lint`.

One thing I don’t like right now is the dependency on gcc-toolchain
(i.e. gcc and binutils) in distutils. I don’t know how to avoid that,
since building CFFI modules won’t work without patching.

I guess the next step would be figuring out how to add “pypy3-foo”
variants for our Python packages, right?

Cheers,
Lars
L
L
Ludovic Courtès wrote on 29 May 2020 18:00
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 41472@debbugs.gnu.org)
87tuzy20ky.fsf@gnu.org
Hi,

Lars-Dominik Braun <lars@6xq.net> skribis:

Toggle quote (9 lines)
> the attached patch adds pypy3, a Python 3 JIT. As you can see this is a
> lot of work, because pypy’s build system is rather unconventional. I’m
> not adding pypy2, because Python 2 is essentially EOL.
>
> I’ve tried fixing as many testcases as possible, but there are some I
> don’t know how to fix. Thus I’m disabling the test suite, although
> everything is in place to run it. pypy itself works fine though, the
> build is reproducible and passes `guix lint`.

Yay!

Toggle quote (4 lines)
> One thing I don’t like right now is the dependency on gcc-toolchain
> (i.e. gcc and binutils) in distutils. I don’t know how to avoid that,
> since building CFFI modules won’t work without patching.

Is CFFI able to pick ‘gcc’ from $PATH? If it can do it, we can avoid
hardcoding the absolute file name of GCC and instead leave it up to the
user to add GCC to their environment when they want to use CFFI.

Toggle quote (3 lines)
> I guess the next step would be figuring out how to add “pypy3-foo”
> variants for our Python packages, right?

Maybe. Is it generally applicable? I hear that many core Python
packages require CPython, which makes it hard to migrate to a different
implementation.

Toggle quote (9 lines)
>>From bdf71850523b42e9d710618d0e0d9ea9fbfe9f87 Mon Sep 17 00:00:00 2001
> From: Lars-Dominik Braun <lars@6xq.net>
> Date: Fri, 22 May 2020 16:02:26 +0200
> Subject: [PATCH] gnu: Add pypy3.
>
> * gnu/packages/python.scm (pypy3): New public variable.
> * gnu/packages/patches/pypy3-7.3.1-fix-tests.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it

[...]

Toggle quote (6 lines)
> +(define-public pypy3
> + (package
> + (name "pypy3")
> + ;; if you intend to upgrade this package, enable the tests and check
> + ;; the results for anything abnormal

The advice seems to be very general; did you mean that people should
look for something specific in the build log, for example because test
failures are silently ignored?

Nitpick: please use uppercase at the beginning and period at the end.


[...]

Toggle quote (2 lines)
> + ("dash" ,dash) ; used as /bin/sh

I’d suggest ‘bash-minimal’ here, to be consistent with the other
packages.


[...]

Toggle quote (3 lines)
> + (home-page "https://www.pypy.org/")
> + (synopsis "Python 3.6 JIT")

Maybe “Python implementation with just-in-time compilation”?

Toggle quote (2 lines)
> + (description "This package provides a Python 3.6 just-in-time compiler.")

Maybe remove “3.6” since it’s likely to become stale. Bonus points if
you can expound a little bit. :-)

Thanks!

Ludo’.
L
L
Lars-Dominik Braun wrote on 30 May 2020 18:52
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 41472@debbugs.gnu.org)
20200530165239.GA186839@noor.fritz.box
Hi Ludo,

Toggle quote (6 lines)
> > One thing I don’t like right now is the dependency on gcc-toolchain
> > (i.e. gcc and binutils) in distutils. I don’t know how to avoid that,
> > since building CFFI modules won’t work without patching.
> Is CFFI able to pick ‘gcc’ from $PATH? If it can do it, we can avoid
> hardcoding the absolute file name of GCC and instead leave it up to the
> user to add GCC to their environment when they want to use CFFI.
yes, it is, but it looks for the executable "cc", which – oddly – our
gcc-toolchain does not provide. Is this intentional? The updated patch
replaces it with gcc, which seems to work.

Toggle quote (5 lines)
> > I guess the next step would be figuring out how to add “pypy3-foo”
> > variants for our Python packages, right?
> Maybe. Is it generally applicable? I hear that many core Python
> packages require CPython, which makes it hard to migrate to a different
> implementation.
For plain Python packages it works pretty much out of the box. Afaik C
extensions are supported, but software making heavy use of those does
not benefit from PyPy’s JIT.

Toggle quote (3 lines)
> The advice seems to be very general; did you mean that people should
> look for something specific in the build log, for example because test
> failures are silently ignored?
No, not really. Just to run the disabled test suite and see if it goes
from “a few failures” to “everything is broken now”. I could selectively
patch the testsuite to disable currently failing tests, but I’m not
competent enough to judge which one can be skipped safely (due to
sandboxing limitations for example) and which indicate an actual
failure. I’ll remove the comment.

Toggle quote (2 lines)
> I’d suggest ‘bash-minimal’ here, to be consistent with the other
> packages.
Done, actually `guix size dash` includes bash-minimal…

Toggle quote (1 lines)
> Maybe “Python implementation with just-in-time compilation”?
Sounds good, done.

Toggle quote (2 lines)
> Maybe remove “3.6” since it’s likely to become stale. Bonus points if
> you can expound a little bit. :-)
I copy-edited a paragraph from Wikipedia. Hope that’s acceptable.

Additionally I removed some test data from the output, which is not
required at runtime and removed python2 from the closure by patching a
few shebangs.

Cheers,
Lars
L
L
Ludovic Courtès wrote on 1 Jun 2020 00:46
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 41472-done@debbugs.gnu.org)
87v9kbpvsl.fsf@gnu.org
Hi,

Lars-Dominik Braun <lars@6xq.net> skribis:

Toggle quote (9 lines)
>> > One thing I don’t like right now is the dependency on gcc-toolchain
>> > (i.e. gcc and binutils) in distutils. I don’t know how to avoid that,
>> > since building CFFI modules won’t work without patching.
>> Is CFFI able to pick ‘gcc’ from $PATH? If it can do it, we can avoid
>> hardcoding the absolute file name of GCC and instead leave it up to the
>> user to add GCC to their environment when they want to use CFFI.
> yes, it is, but it looks for the executable "cc", which – oddly – our
> gcc-toolchain does not provide. Is this intentional?

Yes: as discussed elsewhere, GCC does not provide ‘cc’, and we don’t
provide it either.

Toggle quote (2 lines)
> The updated patch replaces it with gcc, which seems to work.

Perfect.

Toggle quote (10 lines)
>> The advice seems to be very general; did you mean that people should
>> look for something specific in the build log, for example because test
>> failures are silently ignored?
> No, not really. Just to run the disabled test suite and see if it goes
> from “a few failures” to “everything is broken now”. I could selectively
> patch the testsuite to disable currently failing tests, but I’m not
> competent enough to judge which one can be skipped safely (due to
> sandboxing limitations for example) and which indicate an actual
> failure. I’ll remove the comment.

OK.

Alright. I took the liberty to make the changes below and committed.

Thanks!

Ludo’.
Toggle diff (40 lines)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index b399931a42..9469f89bcf 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -673,7 +673,7 @@ ease from the desktop to a microcontroller or embedded system.")
("bash-minimal" ,bash-minimal) ; Used as /bin/sh
("xz" ,xz))) ; liblzma
(arguments
- `(#:tests? #f ; Disabled for now, there are simply too many unfixable tests failing
+ `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing.
#:modules ((ice-9 ftw) (ice-9 match)
(guix build utils) (guix build gnu-build-system))
#:phases (modify-phases %standard-phases
@@ -782,19 +782,18 @@ ease from the desktop to a microcontroller or embedded system.")
"pypy-dist/lib-python/3/encodings/rot_13.py")
(("#!.+/bin/python")
(string-append "#!" (assoc-ref outputs "out") "/bin/pypy3")))
- (with-fluids
- ((%default-port-encoding "ISO-8859-1"))
- (substitute*
- '("pypy-dist/lib_pypy/_md5.py" "pypy-dist/lib_pypy/_sha1.py")
- (("#!.+/bin/python")
- (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3"))))
+ (with-fluids ((%default-port-encoding "ISO-8859-1"))
+ (substitute* '("pypy-dist/lib_pypy/_md5.py"
+ "pypy-dist/lib_pypy/_sha1.py")
+ (("#!.+/bin/python")
+ (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3"))))
(copy-recursively "pypy-dist" (assoc-ref outputs "out")))
#t)))))
(home-page "https://www.pypy.org/")
(synopsis "Python implementation with just-in-time compilation")
(description "PyPy is a faster, alternative implementation of the Python
-programming language employing a just-in-time compiler. Supports most Python
-code natively, including C extensions.")
+programming language employing a just-in-time compiler. It supports most
+Python code natively, including C extensions.")
(license (list license:expat ; pypy itself; _pytest/
license:psfl ; python standard library in lib-python/
license:asl2.0 ; dotviewer/font/ and some of lib-python/
Closed
?