Many CPAN download URLs are no longer available

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Tobias Geerinckx-Rice
  • Mark H Weaver
Owner
unassigned
Submitted by
Mark H Weaver
Severity
normal
M
M
Mark H Weaver wrote on 26 Mar 2019 22:18
(address . bug-guix@gnu.org)
87va05baho.fsf@netris.org
Many perl packages cannot be built on hydra.gnu.org because the source
code is no longer available at the source URLs known to Guix. Moreover,
apparently they were gone by the time Hydra made its first build
attempt.

Here are some examples:

mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz
mirror://cpan/authors/id/S/SY/SYOHEX/Mouse-v2.5.6.tar.gz
mirror://cpan/authors/id/S/SZ/SZABGAB/Class-Date-1.1.17.tar.gz
mirror://cpan/authors/id/S/SZ/SZABGAB/SVG-2.84.tar.gz
mirror://cpan/authors/id/A/AB/ABW/Template-Toolkit-2.28.tar.gz
mirror://cpan/authors/id/D/DA/DAGOLDEN/Log-Any-1.707.tar.gz
mirror://cpan/authors/id/K/KE/KENTNL/Carp-Clan-6.07.tar.gz
mirror://cpan/authors/id/R/RJ/RJBS/Carp-1.50.tar.gz
mirror://cpan/authors/id/D/DA/DAGOLDEN/File-Temp-0.2309.tar.gz
mirror://cpan/authors/id/M/MA/MAKAMAKA/Text-CSV-1.99.tar.gz

and these are the corresponding download derivations:

/gnu/store/c73chq3k1z1n2wnbh0sbi2vwip6dfzbk-URI-1.76.tar.gz.drv
/gnu/store/550r5gdrvs5370rsa2zwp1bs0xwj5dnm-Mouse-v2.5.6.tar.gz.drv
/gnu/store/qxkgrj9gz1kp9id2d7d3mzrfn5b56019-Class-Date-1.1.17.tar.gz.drv
/gnu/store/7239snjzan94plk58hw2zgachq71rh64-SVG-2.84.tar.gz.drv
/gnu/store/qn1w1wvy5x2wngq4845j60i3qvkwhwvi-Template-Toolkit-2.28.tar.gz.drv
/gnu/store/w1cb4r6c0v37xcjav6z16fzbc2zp47ff-Log-Any-1.707.tar.gz.drv
/gnu/store/7gdg0aqsc3i7iwnrajcv6skkjbnf6z4r-Carp-Clan-6.07.tar.gz.drv
/gnu/store/2b088r47jay82d91w62hr35lszqbdrkk-Carp-1.50.tar.gz.drv
/gnu/store/jsddlzd5aba8vnjz1adggcxmvkgv5qf6-File-Temp-0.2309.tar.gz.drv
/gnu/store/y2yb19kf1pfblhqwxcll7rrh6pgpr9sh-Text-CSV-1.99.tar.gz.drv

At least some, and probably most, of these URIs were updated quite
recently. For example, the 'perl-mouse', 'perl-carp-clan', and
'perl-file-temp' were all updated on March 23, and presumably the source
URIs worked at that point, but then all three URIs had to be updated two
days later to fix the broken download links.

The CPAN download URLs that we are using are apparently not long-lived.
Does anyone know if CPAN, or perhaps another site, offers more stable
download URIs for these packages?

Thanks,
Mark
T
T
Tobias Geerinckx-Rice wrote on 27 Mar 2019 01:31
(name . Mark H Weaver)(address . mhw@netris.org)(address . 35010@debbugs.gnu.org)
87zhphi2ff.fsf@nckx
Mark,

[Quick stream-of consciousness reply on a train, whee.]

Mark H. Weaver wrote:
Toggle quote (9 lines)
> At least some, and probably most, of these URIs were updated
> quite
> recently. For example, the 'perl-mouse', 'perl-carp-clan', and
> 'perl-file-temp' were all updated on March 23, and presumably
> the source
> URIs worked at that point, but then all three URIs had to be
> updated two
> days later to fix the broken download links.

- The wave of Perl updates this March was me, using ‘guix refresh
-u’ (and manually checking for breakages, of course).

I never use ‘guix refresh’, until after about a year I forget why,
use ‘guix refresh’ once, and promptly remember.

The updater for CPAN packages is at best caveat-quality. It
helpfully downloads the updated tarball to the store, but doesn't
update the URL. Since the file is in the store, everything works
fine on the updater's machine, then breaks everywhere else.

Background: the problem here is that CPAN URLs contain the
uploader name, e.g. (Karen?) ETHER(idge) in the case of

mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz

which doesn't change every VERSION (so the problem is somewhat
hidden) but more frequently than the author/maintainer would.

I'm planning on finally taking care this problem after I get home
(and after I finally get the Overdrives set up, cough), either by
making the CPAN updater also rewrite URL fields (if possible?), or
finding out whether there's a way to construct these URLs without
using the uploader name, or… well, that's all I have for now.

- While fixing the remaining fallout from this, I did find a few
other broken CPAN links for packages that I hadn't recently
touched. Updating them broke others, so I just left them alone.
I don't know if these are more broken refreshes from longer ago
that went unnoticed (so basically nobody uses these packages),
or if there's another unrelated problem.

Kind regards,

T G-R
-----BEGIN PGP SIGNATURE-----

iHUEARYKAB0WIQT12iAyS4c9C3o4dnINsP+IT1VteQUCXJrERAAKCRANsP+IT1Vt
ebGfAQCWx6Z3qQJZAPeyEMV1bqfIyZFhP+/cVgBRZ/KtubWgZwD/aQNFrj1tYzJO
jIHETcFkYKUCEEoNBqael4MwG1SfxAA=
=p4mt
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 27 Mar 2019 15:07
(name . Tobias Geerinckx-Rice)(address . me@tobias.gr)
87lg10eez7.fsf@gnu.org
Hello,

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

Toggle quote (13 lines)
> The updater for CPAN packages is at best caveat-quality. It helpfully
> downloads the updated tarball to the store, but doesn't update the
> URL. Since the file is in the store, everything works fine on the
> updater's machine, then breaks everywhere else.
>
> Background: the problem here is that CPAN URLs contain the uploader
> name, e.g. (Karen?) ETHER(idge) in the case of
>
> mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz
>
> which doesn't change every VERSION (so the problem is somewhat hidden)
> but more frequently than the author/maintainer would.

Ouch, I didn’t know that, this is terrible.

The good news is that the (guix upstream) framework gets to see the
correct URL:

Toggle snippet (4 lines)
scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
$6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>

So with a bit of work we could change ‘package-update’ to automatically
update the URL.

(Time passes…)

Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
I should have let figure it out ;-), but somehow I ended up
investigating too much, bah!

Let me know what you think.

In the meantime, to fix the Perl packages, you could maybe run:

guix lint -c source $(guix package -A ^perl-)

Then you could perhaps comment the ‘version>?’ test in (guix upstream)
to force ‘guix package -u’ to update these seemingly up-to-date
packages.

HTH,
Ludo’.
T
T
Tobias Geerinckx-Rice wrote on 27 Mar 2019 19:40
(name . Ludovic Courtès)(address . ludo@gnu.org)
871s2sjh4b.fsf@nckx
Ludo',

Ludovic Courtès wrote:
Toggle quote (4 lines)
> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
> I should have let [you?] figure it out ;-), but somehow I ended
> up investigating too much, bah!

Now I had time to rant about package names. Win-win! …wait

Toggle quote (2 lines)
> Let me know what you think.

[…]

Toggle quote (9 lines)
> In the meantime, to fix the Perl packages, you could maybe run:
>
> guix lint -c source $(guix package -A ^perl-)
>
> Then you could perhaps comment the ‘version>?’ test in (guix
> upstream)
> to force ‘guix package -u’ to update these seemingly up-to-date
> packages.

I'll deffo give it a try. Thanks!

Kind regards,

T G-R
-----BEGIN PGP SIGNATURE-----

iHUEARYKAB0WIQT12iAyS4c9C3o4dnINsP+IT1VteQUCXJvDlQAKCRANsP+IT1Vt
eQYTAP9Uq113BOZBj0sC+ieAGbuqo/NKmB69bZCnqLBAbPu8WAEA3LSHuCGddz7t
9/HXo5BlcaqI+rtzkb1dktkc1ExVVQY=
=W/h7
-----END PGP SIGNATURE-----

M
M
Mark H Weaver wrote on 28 Mar 2019 23:10
(name . Ludovic Courtès)(address . ludo@gnu.org)
87tvfmbqg9.fsf@netris.org
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (13 lines)
> The good news is that the (guix upstream) framework gets to see the
> correct URL:
>
> scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
> $6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
>
> So with a bit of work we could change ‘package-update’ to automatically
> update the URL.
>
> (Time passes…)
>
> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.

Nice! I haven't looked closely, but does this somehow update the code
that computes source URLs using 'string-append'? If so, that's
impressive :)

In other news, here's another batch of bad source URLs that Hydra
recently failed to fetch:

mirror://savannah/starfighter/1.7/project-starfighter-1.7-src.tar.gz
mirror://cpan/authors/id/P/PI/PIJLL/DateTime-Calendar-Julian-0.100.tar.gz
mirror://cpan/authors/id/A/AD/ADAMK/Object-Signature-1.08.tar.gz
mirror://cpan/authors/id/J/JH/JHI/Time-HiRes-1.9760.tar.gz
mirror://cpan/authors/id/F/FR/FREW/Class-C3-Componentised-1.001002.tar.gz
mirror://cpan/authors/id/R/RG/RGARCIA/base-2.23.tar.gz

Mark
L
L
Ludovic Courtès wrote on 29 Mar 2019 16:34
(name . Mark H Weaver)(address . mhw@netris.org)
87bm1tae51.fsf@gnu.org
Mark H Weaver <mhw@netris.org> skribis:

Toggle quote (19 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> The good news is that the (guix upstream) framework gets to see the
>> correct URL:
>>
>> scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
>> $6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
>>
>> So with a bit of work we could change ‘package-update’ to automatically
>> update the URL.
>>
>> (Time passes…)
>>
>> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
>
> Nice! I haven't looked closely, but does this somehow update the code
> that computes source URLs using 'string-append'? If so, that's
> impressive :)

Well, yes and no. :-)

It looks like that’s what it does, but in practice it simply does a
string substitution on the ‘dirname’ of the URI, which probably works in
most cases (it definitely works for CPAN packages.)

Toggle quote (10 lines)
> In other news, here's another batch of bad source URLs that Hydra
> recently failed to fetch:
>
> mirror://savannah/starfighter/1.7/project-starfighter-1.7-src.tar.gz
> mirror://cpan/authors/id/P/PI/PIJLL/DateTime-Calendar-Julian-0.100.tar.gz
> mirror://cpan/authors/id/A/AD/ADAMK/Object-Signature-1.08.tar.gz
> mirror://cpan/authors/id/J/JH/JHI/Time-HiRes-1.9760.tar.gz
> mirror://cpan/authors/id/F/FR/FREW/Class-C3-Componentised-1.001002.tar.gz
> mirror://cpan/authors/id/R/RG/RGARCIA/base-2.23.tar.gz

Thanks!

Ludo’.
M
M
Mark H Weaver wrote on 7 Apr 2019 03:30
(name . Tobias Geerinckx-Rice)(address . me@tobias.gr)(address . 35010@debbugs.gnu.org)
87r2ae7gae.fsf@netris.org
Hi Tobias,

There are still at least two more bad URLs from this batch of perl
updates circa March 23rd:

mirror://cpan/authors/id/A/AB/ABW/Template-Toolkit-2.28.tar.gz
mirror://cpan/authors/id/M/MA/MAKAMAKA/Text-CSV-1.99.tar.gz

Could you take a look?

Thanks,
Mark
M
M
Mark H Weaver wrote on 1 May 2019 04:58
Many CPAN download URLs are no longer available
(address . 35010-done@debbugs.gnu.org)
877ebaop7i.fsf@netris.org
I believe that these broken URLs are now fixed, so I'm closing this bug.

Mark
Closed
?