[PATCH] utils: Add helper method to list subdirectories.

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal
M
M
Maxim Cournoyer wrote on 5 Mar 2018 05:15
(name . guix-patches)(address . guix-patches@gnu.org)
87a7vnma48.fsf@gmail.com
Hello Guix!

This adds a method useful to list subdirectories, which I am using to
list bundled copies of libraries (and delete them), for example.

Thank you,

Maxim
From b4b607800d770c4cf77f92c247276c368357e94f Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Sun, 25 Feb 2018 17:49:06 -0500
Subject: [PATCH] utils: Add helper method to list subdirectories.

* guix/build/utils.scm (find-subdirectories): New procedure.
* tests/build-utils.scm: Rename module so that it can be used with Geiser.
(%test-dir-hierarchy): New variable.
(make-test-dir-hierarchy): New test procedure.
("find-subdirectories"): New test.
---
guix/build/utils.scm | 16 ++++++++++++++++
tests/build-utils.scm | 37 +++++++++++++++++++++++++++++++++++--
2 files changed, 51 insertions(+), 2 deletions(-)

Toggle diff (109 lines)
diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index 7391307c8..9a321bf3e 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2013 Andreas Enge <andreas@enge.fr>
;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
+;;; Copyright © 2018 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -60,6 +61,7 @@
delete-file-recursively
file-name-predicate
find-files
+ find-subdirectories
search-path-as-list
set-path-environment-variable
@@ -395,6 +397,20 @@ also be included. If FAIL-ON-ERROR? is true, raise an exception upon error."
stat)
string<?)))
+(define* (find-subdirectories dir #:key fail-on-error?)
+ "Return the list of the immediate subdirectories of DIR."
+ ;; Strip the trailing '/' DIR is '/'.
+ (let ((dir (if (and (> 1 (string-length dir))
+ (eq? (string-take-right dir 1) #\/))
+ (string-drop-right dir 1)
+ dir)))
+ (define (pred filename stat)
+ (and (eq? (stat:type stat) 'directory)
+ (string-match (string-append dir "/[^/]*$") filename)))
+ (find-files dir pred
+ #:directories? #t
+ #:fail-on-error? fail-on-error?)))
+
;;;
;;; Search paths.
diff --git a/tests/build-utils.scm b/tests/build-utils.scm
index 7d49446f6..6a3d43784 100644
--- a/tests/build-utils.scm
+++ b/tests/build-utils.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2012, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2018 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -17,7 +18,7 @@
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
-(define-module (test-build-utils)
+(define-module (tests build-utils)
#:use-module (guix tests)
#:use-module (guix build utils)
#:use-module ((guix utils)
@@ -27,7 +28,8 @@
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-64)
#:use-module (rnrs io ports)
- #:use-module (ice-9 popen))
+ #:use-module (ice-9 popen)
+ #:use-module (ice-9 match))
(test-begin "build-utils")
@@ -122,4 +124,35 @@
(and (zero? (close-pipe pipe))
str))))))
+(define %test-dir-hierarchy
+ ;; The first element of a list is a file if the only element, otherwise
+ ;; a directory.
+ '("top"
+ ("subdir1"
+ ("subsubdir1"
+ "a-file.txt"
+ "another-file.c"))
+ ("subdir2"
+ "yet-another-one.h")
+ ("file.txt")))
+
+(define* (make-test-dir-hierarchy hierarchy #:optional (top (getcwd)))
+ (mkdir-p top)
+ (match hierarchy
+ ((dir . rest)
+ (for-each
+ (lambda (item)
+ (make-test-dir-hierarchy item (string-append top "/" dir)))
+ rest))
+ (file
+ (system (string-append "echo \"\" > " "\"" top "/" file "\"")))))
+
+(test-equal "find-subdirectories"
+ '("top/subdir1" "top/subdir2")
+ (call-with-temporary-directory
+ (lambda (directory)
+ (make-test-dir-hierarchy %test-dir-hierarchy directory)
+ (chdir directory)
+ (find-subdirectories "top"))))
+
(test-end)
--
2.16.1
L
L
Ludovic Courtès wrote on 5 Mar 2018 18:12
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 30708@debbugs.gnu.org)
87lgf68n07.fsf@gnu.org
Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (11 lines)
> From b4b607800d770c4cf77f92c247276c368357e94f Mon Sep 17 00:00:00 2001
> From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> Date: Sun, 25 Feb 2018 17:49:06 -0500
> Subject: [PATCH] utils: Add helper method to list subdirectories.
>
> * guix/build/utils.scm (find-subdirectories): New procedure.
> * tests/build-utils.scm: Rename module so that it can be used with Geiser.
> (%test-dir-hierarchy): New variable.
> (make-test-dir-hierarchy): New test procedure.
> ("find-subdirectories"): New test.

[...]

Toggle quote (14 lines)
> +(define* (find-subdirectories dir #:key fail-on-error?)
> + "Return the list of the immediate subdirectories of DIR."
> + ;; Strip the trailing '/' DIR is '/'.
> + (let ((dir (if (and (> 1 (string-length dir))
> + (eq? (string-take-right dir 1) #\/))
> + (string-drop-right dir 1)
> + dir)))
> + (define (pred filename stat)
> + (and (eq? (stat:type stat) 'directory)
> + (string-match (string-append dir "/[^/]*$") filename)))
> + (find-files dir pred
> + #:directories? #t
> + #:fail-on-error? fail-on-error?)))

‘find-files’ recurses in subdirectories, so the above implementation is
not as efficient as it could be.

I would instead suggest using ‘scandir’ (or ‘file-system-fold’) from
Guile’s (ice-9 ftw) module.

That said… is this a common enough operation?

Thanks,
Ludo’.
M
M
Maxim Cournoyer wrote on 6 Mar 2018 03:18
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30708@debbugs.gnu.org)
87y3j6kkuz.fsf@gmail.com
Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

Toggle quote (37 lines)
> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> From b4b607800d770c4cf77f92c247276c368357e94f Mon Sep 17 00:00:00 2001
>> From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
>> Date: Sun, 25 Feb 2018 17:49:06 -0500
>> Subject: [PATCH] utils: Add helper method to list subdirectories.
>>
>> * guix/build/utils.scm (find-subdirectories): New procedure.
>> * tests/build-utils.scm: Rename module so that it can be used with Geiser.
>> (%test-dir-hierarchy): New variable.
>> (make-test-dir-hierarchy): New test procedure.
>> ("find-subdirectories"): New test.
>
> [...]
>
>> +(define* (find-subdirectories dir #:key fail-on-error?)
>> + "Return the list of the immediate subdirectories of DIR."
>> + ;; Strip the trailing '/' DIR is '/'.
>> + (let ((dir (if (and (> 1 (string-length dir))
>> + (eq? (string-take-right dir 1) #\/))
>> + (string-drop-right dir 1)
>> + dir)))
>> + (define (pred filename stat)
>> + (and (eq? (stat:type stat) 'directory)
>> + (string-match (string-append dir "/[^/]*$") filename)))
>> + (find-files dir pred
>> + #:directories? #t
>> + #:fail-on-error? fail-on-error?)))
>
> ‘find-files’ recurses in subdirectories, so the above implementation is
> not as efficient as it could be.
>
> I would instead suggest using ‘scandir’ (or ‘file-system-fold’) from
> Guile’s (ice-9 ftw) module.

Thanks! See the new patched attached. The test still passes.

Toggle quote (2 lines)
> That said… is this a common enough operation?

I'm using it in a forthcoming new Guix package (SuperCollider) where it
allows me to explicitly list the bundled dependencies that are to be
*kept* rather than the ones to be removed, as is more commonly done. Without a
list of the subdirectories the contrib/vendor/whatever bundled
libraries directory I would not be able to do the following:

Toggle snippet (13 lines)
+ ;; The build system doesn't allow us to unbundle the
+ ;; following libraries.
+ (let* ((all-dirs (find-subdirectories "./external_libraries"))
+ (keep-dirs '("nova-simd" "nova-tt" "hidapi" "TLSF-2.4.6"
+ "oscpack_1_1_0"))
+ (remove-dirs
+ (remove (lambda (x)
+ (member (basename x) keep-dirs))
+ all-dirs)))
+ (format #t "Removing bundled libraries: ~s\n" remove-dirs)
+ (for-each delete-file-recursively remove-dirs)))))))

Although now that you've made me see the light (scandir), I could
rewrite the whole thing using:

Toggle snippet (14 lines)
(lambda _
;; The build system doesn't allow us to unbundle the following
;; libraries.
(let ((keep-dirs '("nova-simd" "nova-tt" "hidapi" "TLSF-2.4.6"
"oscpack_1_1_0" "." "..")))
(with-directory-excursion "./external_libraries"
(for-each
delete-file-recursively
(scandir "."
(lambda (x)
(and (eq? (stat:type (stat x)) 'directory)
(not (member (basename x) keep-dirs))))))))

So, this patch can go to the recycle bin. Thanks! :)

Maxim
L
L
Ludovic Courtès wrote on 6 Mar 2018 11:33
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 30708-done@debbugs.gnu.org)
87lgf5xzlj.fsf@gnu.org
Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (18 lines)
> Although now that you've made me see the light (scandir), I could
> rewrite the whole thing using:
>
> (lambda _
> ;; The build system doesn't allow us to unbundle the following
> ;; libraries.
> (let ((keep-dirs '("nova-simd" "nova-tt" "hidapi" "TLSF-2.4.6"
> "oscpack_1_1_0" "." "..")))
> (with-directory-excursion "./external_libraries"
> (for-each
> delete-file-recursively
> (scandir "."
> (lambda (x)
> (and (eq? (stat:type (stat x)) 'directory)
> (not (member (basename x) keep-dirs))))))))
>
> So, this patch can go to the recycle bin. Thanks! :)

Well, I’m glad that it works for you. :-)

Thanks,
Ludo’.
Closed
?