Cannot boot with encrypted root

  • Done
  • quality assurance status badge
Details
3 participants
  • Duncan Keall
  • Ludovic Courtès
  • Nikita Karetnikov
Owner
unassigned
Submitted by
Nikita Karetnikov
Severity
normal
N
N
Nikita Karetnikov wrote on 26 Nov 2014 04:26
(address . bug-guix@gnu.org)
8761e2wsjp.fsf@karetnikov.org
(Ludo suggested to report this as a bug, so this issue won’t get lost.)

While booting, GRUB prints this error:

error: file
‘/gnu/store/rz…-linux-libre-3.17.3/bzImage’ not found.

which is not surprising since the store is encrypted.

I’ve tried putting these lines after the “setparams” line in the GRUB
config (by hitting the ‘e’ key):

insmod luks
cryptomount hd0,gpt3
set root=crypto0

After pressing the ‘F10’ key, I get a password prompt. And after that, a
bunch of warnings about ‘/dev/mapper/main’:

ext2fs_check_if_mount: Can’t check if filesystem is mounted due to
missing mtab file while determining whether /dev/mapper/main is mounted.
fsck.ext3: No such file or directory while trying to open
/dev/mapper/main
Possibly non-existent device?
‘fsck.ext3’ exited with code 8 on /dev/mapper/main; spawning REPL

In the REPL:

scheme@(guile-user)> ,use (gnu build linux-boot)
scheme@(guile-user)> (boot-system)
[…]
ERROR: In procedure mount:
ERROR: In procedure mount: Device or resource busy

My config is shown below:

(use-modules (gnu))

(define %linux-modules
'(
;; cryptsetup/LUKS
"dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko"))

(operating-system
(host-name "test")
(timezone "Europe/Paris")
(locale "en_US.UTF-8")

(bootloader (grub-configuration
(device "/dev/sda")))

(initrd (lambda (fs . args)
(apply base-initrd fs
#:extra-modules %linux-modules
args)))

(mapped-devices (list (mapped-device
(source "/dev/sda3")
(target "main")
(type luks-device-mapping))))

(file-systems (cons* (file-system
(device "/dev/mapper/main")
(mount-point "/")
(type "ext3"))
(file-system
(device "boot")
(type 'label)
(mount-point "/boot")
(type "ext3"))
%base-file-systems))

(users (list (user-account
(name "test")
(group "users")

(supplementary-groups '("wheel"
"audio" "video"))
(home-directory "/home/test")))))
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBAgAGBQJUdUh+AAoJEM+IQzI9IQ38glwQAIjpCx50uBQ/gw4y60WaIja4
iDE47RHemyoRz9JlWQx5xhsXcH/BCTMUkRexFB+WZjD20raU0Q5/0Aajuw+7wvr1
OUk+/G6y0+dKuxw7yZHbDWhTOelDpXYLrIou4dKsOi+Lu611I4dSqlDCnAVL8rso
Jp/xPYOb913xgJ0FAeGPXuy+/6+baT4X1RbHIHy4ACsy7qyjjTzYUmVRozz7Qlg9
n+jwXVf/5pI08EF2bh9pjw5WjrJ1oumFOwjPhqiQFGWCbVKeWOGbVUeBg620J9dN
g+OD9SFreL91NXTLrv7VVUts07ypEeGoJeofa7q/O1Bqv04F0wlb4FpR4t13+f69
RyHC7r8JU+b2rDF6QvDdl4CzjuGCtP1c7diUgZ1WYLKm8z1aeJFbDleN/sywhpYy
JXajVYrfDZegXXs1jsef8CyeujTDDEhjNPANbGkx5fcq4pxeCeJ3r4x9G+XDQ2wR
aW40aRgnURIl40QpAC+yc6nmusE7GSGijHs7PFTDj8lM1alUIQf6YBLpK2xmxTOn
U1IGemOKio+kgK3GD9uuATktf8Zp78zw+yjYYw+5QxWozBjBIdoEiwJWzCaEo3UM
BBev9R0aGeOiEnyKDKQHLNh5Vbf9i3wwSroMaofLLuyrU6DMG4fKbK8I3/ECyp1z
a/zmgNkiAcMiwgT2yM1L
=oS5t
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 26 Nov 2014 10:55
(name . Nikita Karetnikov)(address . nikita@karetnikov.org)(address . 19190@debbugs.gnu.org)
878uiywako.fsf@gnu.org
Thanks for the report. Let’s address the initrd/boot issue first.

Nikita Karetnikov <nikita@karetnikov.org> skribis:

Toggle quote (2 lines)
> After pressing the ‘F10’ key, I get a password prompt.

Any message after entering the password prompt?

Toggle quote (9 lines)
> And after that, a bunch of warnings about ‘/dev/mapper/main’:
>
> ext2fs_check_if_mount: Can’t check if filesystem is mounted due to
> missing mtab file while determining whether /dev/mapper/main is mounted.
> fsck.ext3: No such file or directory while trying to open
> /dev/mapper/main
> Possibly non-existent device?
> ‘fsck.ext3’ exited with code 8 on /dev/mapper/main; spawning REPL

It looks like ‘cryptsetup open’ simply failed, and ‘boot-system’
currently ignores that error, so it goes up to the file system check as
shown above, which in turn fails and gets us to the debugger.

Perhaps you could try running ‘cryptsetup’ by hand from the REPL.
That’s going to be a bit tedious; something like:

(use-modules (srfi srfi-1) (ice-9 ftw))
(define cs (find (lambda (file)
(string-contains file "cryptsetup"))
(scandir "/gnu/store")))
(system* (string-append "/gnu/store/" cs "/bin/cryptsetup")
"open" "--type" "luks" "/dev/whatever" "main")

Then, please report the return value of ‘system*’, and check

(file-exists? "/dev/mapper/main")

Please also report any messages.

Thanks in advance!

Ludo’.
L
L
Ludovic Courtès wrote on 26 Nov 2014 13:41
(name . Nikita Karetnikov)(address . nikita@karetnikov.org)(address . 19190@debbugs.gnu.org)
87y4qyuoaw.fsf@gnu.org
I noticed an obvious issue in (gnu build linux-boot), which is that
device-mapping (the #:pre-mount argument) would be performed after
‘mount-root-file-system’ has been called.

Commit 67979e4 fixes that.

To fix it, we’ll have to update ‘guix-devel’ in (gnu packages
package-management), and to regenerate a VM image that uses it. I can
help with that.

Ludo’.
L
L
Ludovic Courtès wrote on 26 Nov 2014 17:04
(name . Duncan Keall)(address . duncan@duncankeall.com)
87ioi2uevw.fsf@gnu.org
Hi, :-)

Duncan Keall <duncan@duncankeall.com> skribis:

Toggle quote (12 lines)
>> Perhaps you could try running ‘cryptsetup’ by hand from the REPL.
>
> I needed to replace "/bin/cryptsetup" with "/sbin/cryptsetup". After
> that, cryptsetup accepted a password and printed these errors:
>
> device-mapper: table 253:0: crypt: Error allocating crypto tfm
> device-mapper: ioctl: error adding target to table
> device-mapper: reload ioctl on failed: No such file or directory
> Failed to open temporary keystore device.
> device-mapper: remove ioctl on temporary-cryptsetup-149 failed: No
> such device or address

I suspect this is due to missing modules.

Does your initrd configuration have the modules Nikita’s configuration
shows?

Thanks for testing so quickly!

Ludo’.
D
D
Duncan Keall wrote on 26 Nov 2014 13:05
(name . Ludovic Courtès)(address . ludo@gnu.org)
1417003517.3640091.195610409.39CCFAAC@webmail.messagingengine.com
Here's what I've found using a very similar setup to Nikita and after
receiving the same error messages.

Toggle quote (2 lines)
> Any message after entering the password prompt?

GRUB prints a single message "Slot 0 opened" and then it's into the
kernel boot logs. The first related message appears to be the
"ext2fs_check_if_mount..." one.

Toggle quote (2 lines)
> Perhaps you could try running ‘cryptsetup’ by hand from the REPL.

I needed to replace "/bin/cryptsetup" with "/sbin/cryptsetup". After
that, cryptsetup accepted a password and printed these errors:

device-mapper: table 253:0: crypt: Error allocating crypto tfm
device-mapper: ioctl: error adding target to table
device-mapper: reload ioctl on failed: No such file or directory
Failed to open temporary keystore device.
device-mapper: remove ioctl on temporary-cryptsetup-149 failed: No
such device or address

The return value was 256, and afterwards /dev/mapper/main still did not
exist.
D
D
Duncan Keall wrote on 27 Nov 2014 03:06
(name . Ludovic Courtès)(address . ludo@gnu.org)
1417053963.3820317.195899889.5AAEB90A@webmail.messagingengine.com
Toggle quote (3 lines)
> Does your initrd configuration have the modules Nikita’s configuration
> shows?

Here are the configurations I have tested which all result in the errors
posted above:

("dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
("dm-crypt.ko" "cryptd.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
("dm-crypt.ko" "cryptd.ko" "crc32-pclmul.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")

I could also add "aesni-intel.ko" but it resulted in a different set of
kernel errors on boot:

aesni_intel: Unknown symbol ablk_decrypt (err 0)
aesni_intel: Unknown symbol lrw_free_table (err 0)
aesni_intel: Unknown symbol ablk_set_key (err 0)
...
ERROR: In procedure load-linux-module: Unknown error -1

For reference, here are all the modules loaded on my current system
which might be related:

aes_x86_64 aesni_intel af_alg algif_skcipher arc4 ccm crc32_pclmul
crc32c_intel crc_t10dif crct10dif_common crct10dif_pclmul cryptd ctr
dm_crypt dm_mod gf128mul ghash_clmulni_intel lrw xor
L
L
Ludovic Courtès wrote on 27 Nov 2014 11:12
(name . Duncan Keall)(address . duncan@duncankeall.com)
87tx1lostk.fsf@gnu.org
Duncan Keall <duncan@duncankeall.com> skribis:

Toggle quote (16 lines)
> Here are the configurations I have tested which all result in the errors
> posted above:
>
> ("dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
> ("dm-crypt.ko" "cryptd.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
> ("dm-crypt.ko" "cryptd.ko" "crc32-pclmul.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>
> I could also add "aesni-intel.ko" but it resulted in a different set of
> kernel errors on boot:
>
> aesni_intel: Unknown symbol ablk_decrypt (err 0)
> aesni_intel: Unknown symbol lrw_free_table (err 0)
> aesni_intel: Unknown symbol ablk_set_key (err 0)
> ...
> ERROR: In procedure load-linux-module: Unknown error -1

Presumably that’s because one of its prerequisites hasn’t been loaded
yet.

The difficulty is that modules need to be listed in topological order:
things that aesni_intel.ko depends on must appear before aesni_intel.ko.

On my machine, I see:

Toggle snippet (9 lines)
$ lsmod|grep aes
aesni_intel 165373 3
ablk_helper 13597 1 aesni_intel
cryptd 20359 4 ghash_clmulni_intel,aesni_intel,ablk_helper
lrw 13286 1 aesni_intel
glue_helper 13990 1 aesni_intel
aes_x86_64 17131 1 aesni_intel

So the right order for these would be:

cryptd
aes_x86_64
ablk_helper
lrw
aesni_intel

We’ll have to change the initrd build to figure that out automatically,
but in the meantime, we have to sort them by hand.

Ludo’.
L
L
Ludovic Courtès wrote on 13 Dec 2014 23:01
(name . Duncan Keall)(address . duncan@duncankeall.com)
877fxvxl8z.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (23 lines)
> Duncan Keall <duncan@duncankeall.com> skribis:
>
>> Here are the configurations I have tested which all result in the errors
>> posted above:
>>
>> ("dm-crypt.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>> ("dm-crypt.ko" "cryptd.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>> ("dm-crypt.ko" "cryptd.ko" "crc32-pclmul.ko" "aes-x86_64.ko" "crc32.ko" "pcbc.ko" "xcbc.ko" "sha256-ssse3.ko")
>>
>> I could also add "aesni-intel.ko" but it resulted in a different set of
>> kernel errors on boot:
>>
>> aesni_intel: Unknown symbol ablk_decrypt (err 0)
>> aesni_intel: Unknown symbol lrw_free_table (err 0)
>> aesni_intel: Unknown symbol ablk_set_key (err 0)
>> ...
>> ERROR: In procedure load-linux-module: Unknown error -1
>
> Presumably that’s because one of its prerequisites hasn’t been loaded
> yet.
>
> The difficulty is that modules need to be listed in topological order:

This particular problem was fixed in 0e704a2 a couple of weeks ago.

For testing, you would need to start from an image generated from
current master with:

./pre-inst-env \
guix system disk-image --image-size=800MiB gnu/system/install.scm

Let me know if anything else is needed.

Ludo’.
D
D
Duncan Keall wrote on 15 Dec 2014 11:49
(name . Ludovic Courtès)(address . ludo@gnu.org)
1418640579.2467571.202978541.4A391362@webmail.messagingengine.com
Thanks for the update, Ludovic!

I've just tested another install with a fresh image from master, using
as similar setup as possible to before.

The boot process still fails at mounting the root filesystem (as
expected):

fsck.ext4: No such file or directory while trying to open /dev/mapper/main
Possibly non-existent device?

However I found that cryptsetup was missing from the store during the
early-boot REPL, so I wasn't able to test manually mounting the
encrypted filesystem.

Nothing in the commit history since v0.8 jumped out as being
responsible, so I'm assuming I've missed something obvious during
install. I'll have to keep looking!
L
L
Ludovic Courtès wrote on 20 Sep 2015 22:14
(name . Duncan Keall)(address . duncan@duncankeall.com)
87oagwj1au.fsf@gnu.org
Duncan Keall <duncan@duncankeall.com> skribis:

Toggle quote (6 lines)
> The boot process still fails at mounting the root filesystem (as
> expected):
>
> fsck.ext4: No such file or directory while trying to open /dev/mapper/main
> Possibly non-existent device?

It took us a while but this part is now fixed with cc0e575, and commit
07779d0 adds dm-crypt.ko and xts.ko and their dependencies to the
default initrd.

I’m closing this bug, but if you have time and interest, you’re welcome
to confirm whether this works for you.

Thanks!

Ludo’.
Closed
?