NAS 2021: hardware and OS installation

tags:

NAS 2021: codename sunflux

In the beggining of 2021 I got an urgent need to purchase and setup a small server for one of my friends. The tasks assigned to the server were quite trivial (like in any other typical household): reliable file storage with remote access, a website hosting, and some home automation.

To be exact, it should provide:

Hardware

The decision was made to build it from a casual PC hardware. Since I had a very good experience with Ryzen 4750G in my workstation, I chose Ryzen 3400G for the server (at the moment it was available for pre-order for a good price). However we were not lucky enogh and due to the COVID-19 crysis the delivery date was shifted a few times. I couldn’t wait any more and bought Intel Pentium Gold 6400 and a corresponding motherboard instead.

List of hardware:

Even so it is not an issue for the current setup, I should have checked the case better before buying: it has only two 3.5" slots. Otherwise the case is amazing :)

Software

Since I am supposed to administrate this server myself, I installed the same Linux distro as on my workstation, that is Arch Linux.

The outline of the software is the following:

Rootfs structure (SSD):

@
├── @root
├── @home
├── @docker
├── @srv
└── /snapshots
    ├── root
    ├── home
    ├── docker
    └── srv

Rootfs structure (2xHDD):

@
├── /@data
├── /@backup
└── /snapshots
    ├── data
    └── backup

Actually, I didn’t use the @ notation for subvolumes but did it only in this paragraph to make it more clear which of them is a btrfs subvolume and which is not.

Bootstrapping Arch Linux

There are several ways to install Arch Linux. Since I already had Arch Linux running on my workstation, I decided that the easiest way would be to simply attach the new SSD and bootstrap it from the running Arch Linux system.

The installation process is explained in details in the wiki, therefore I will provide a concise report.

Creating partitons, encrypted containers, and filesystems

After installing the new SSD one has to locate the corresponding block device:

$ lsblk -o NAME,SIZE,MODEL
[...]
nvme0n1    465.8G Samsung SSD 970 EVO 500GB
[...]

It is /dev/nvme0n1 in my case. Now we create a new GPT partition table and two partitions: ESP (mounted as /boot) and an encrypted container for btrfs.

$ fdisk /dev/nvme0n1

Welcome to fdisk (util-linux 2.36.1).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): g
Created a new GPT disklabel (GUID: DEFFCF5A-CBE2-DD44-9685-D5ECACA595DB).

Command (m for help): n
Partition number (1-128, default 1): 
First sector (2048-976773134, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-976773134, default 976773134): +200M

Created a new partition 1 of type 'Linux filesystem' and of size 200 MiB.

Command (m for help): t
Selected partition 1
Partition type or alias (type L to list all): 1
Changed type of partition 'Linux filesystem' to 'EFI System'.

Command (m for help): n
Partition number (2-128, default 2): 
First sector (411648-976773134, default 411648): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (411648-976773134, default 976773134): 

Created a new partition 2 of type 'Linux filesystem' and of size 465.6 GiB.

We check once again if everything is how it was expected and then write the partition table on the disk.

Command (m for help): p
Disk /dev/nvme0n1: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 970 EVO 500GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: DEFFCF5A-CBE2-DD44-9685-D5ECACA595DB

Device          Start       End   Sectors   Size Type
/dev/nvme0n1p1   2048    411647    409600   200M EFI System
/dev/nvme0n1p2 411648 976773134 976361487 465.6G Linux filesystem

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Format the ESP partiton /dev/nvme0n1p1.

$ mkfs.vfat /dev/nvme0n1p1
mkfs.fat 4.1 (2017-01-24)
$ fatlabel /dev/nvme0n1p1 "ARCHBOOT"

Create an enctrypted LUKS container on /dev/nvme0n1p2.

$ cryptsetup luksFormat /dev/nvme0n1p2 --hash sha256 --cipher aes-xts-plain64 --key-size 256 --pbkdf argon2id --sector-size 512

WARNING!
========
This will overwrite data on /dev/nvme0n1p2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/nvme0n1p2: <...>
Verify passphrase: <...>

Open the newly created LUKS container and map it to /dev/mapper/bootstrap_crypt.

$ cryptsetup open --allow-discards /dev/nvme0n1p2 bootstrap_crypt
Enter passphrase for /dev/sdb2: <...>

Let us check if it is all right.

$ lsblk -o NAME,PATH
NAME                PATH
nvme0n1             /dev/nvme0n1
├─nvme0n1p1         /dev/nvme0n1p1
└─nvme0n1p2         /dev/nvme0n1p2
  └─bootstrap_crypt /dev/mapper/bootstrap_crypt
[...]

Now we create a filesystem inside the LUKS container.

$ mkfs.btrfs --label "ARCHROOT" /dev/mapper/bootstrap_crypt                            
btrfs-progs v5.10 
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
Label:              'ARCHROOT'
UUID:               18186a3d-dcdb-497b-a4f2-d7132f91535a
Node size:          16384
Sector size:        4096
Filesystem size:    465.55GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         single            8.00MiB
  System:           single            4.00MiB
SSD detected:       yes
Incompat features:  extref, skinny-metadata
Runtime features:   
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1   465.55GiB  /dev/mapper/bootstrap_crypt

Create a mountpoint in the host system and mount bootstrap_crypt. We will still mount /boot later.

$ mkdir -p /mnt/bootstrap
$ mount /dev/mapper/bootstrap_crypt -o noatime,space_cache=v2,discard=async /mnt/bootstrap

Let’s set up the filesystem properties and create additional subvolumes right away.

$ btrfs property set /mnt/bootstrap compression zstd:1
$ btrfs subvolume create /mnt/bootstrap/root
$ btrfs subvolume create /mnt/bootstrap/home
$ btrfs subvolume create /mnt/bootstrap/docker
$ btrfs subvolume create /mnt/bootstrap/snapshots
$ btrfs subvolume set-default /mnt/bootstrap/root

Verify the created subvolumes.

$ btrfs subvolume list /mnt/bootstrap                               
ID 256 gen 7 top level 5 path root
ID 257 gen 8 top level 5 path home
ID 258 gen 9 top level 5 path docker
ID 259 gen 9 top level 5 path srv

Now create a directory structure for the future snapshots.

$ mkdir -p /mnt/bootstrap/snapshots/{root,home,docker,srv}
$ tree /mnt/bootstrap 
/mnt/bootstrap
├── docker
├── home
├── root
├── srv
└── snapshots
    ├── docker
    ├── home
    ├── root
    └── srv

Since we are not going to install Arch Linux into filesystem’s root, but in the specially created root subvolume, we need to remount bootstrap_crypt, bringing /root subvolume to the top level.

umount /mnt/bootstrap
mount /dev/mapper/bootstrap_crypt -o noatime,space_cache=v2,discard=async /mnt/bootstrap

Now it’s time to mount /boot.

mkdir -p /mnt/bootstrap/boot
mount /dev/nvme0n1p1 /mnt/bootstrap/boot

Now the filesystems are mounted so we can proceed with pacstrap.

Bootstrapping new Arch Linux system with pacstrap

At this stage we need to install package arch-install-scripts to the host system. This package provides pacstrap and arch-chroot which we are going to use next.

Use pacstrap to install the necessary packages into /mnt/bootstrap.

pacstrap -c /mnt/bootstrap \
  base linux linux-firmware intel-ucode\
  cryptsetup networkmanager openssh nano sudo btrfs-progs
[...]

Now we can chroot into the new Arch Linux installation.

$ arch-chroot /mnt/bootstrap

From now on # in the command line prompt means that we are chrooted. Let us do some basic system setup: timezones, locales, hostname

# ln -sf /usr/share/zoneinfo/Europe/Berlin /etc/localtime
# hwclock --systohc
# sed -i 's/#en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/g' /etc/locale.gen
# locale-gen
# echo "LANG=en_US.UTF-8" > /etc/locale.conf

Put the machine name into /etc/hostname and /etc/hosts.

Add the following lines to /etc/fstab:

/dev/mapper/system_crypt  /mnt/rootfs  btrfs  subvol=/,noatime,discard=async,space_cache=v2  0 0
/dev/mapper/system_crypt  /home  btrfs  subvol=/home,noatime,discard=async,space_cache=v2  0 0
/dev/mapper/system_crypt  /var/lib/docker  btrfs  subvol=/docker,noatime,discard=async,space_cache=v2  0 0
/dev/mapper/system_crypt  /srv  btrfs  subvol=/srv,noatime,discard=async,space_cache=v2  0 0
LABEL=ARCHBOOT  /boot  vfat  defaults  0 2

Creating users and configuring sudo

Let us set a password for root, create a new user, and set up his password as well.

# passwd
[...]
# useradd -m dstrelnikov
# passwd dstrelnikov
[...]

My preferred way to setup sudo is to create a group with the same name, to adjust /etc/sudoers and to add users to the sudo group.

# groupadd sudo 
# gpasswd -a dstrelnikov sudo
Adding user dstrelnikov to group sudo

# EDITOR=nano visudo
[...]
## Uncomment to allow members of group sudo to execute any command
%sudo   ALL=(ALL) ALL
[...]

Configuring networking and SSH

Since our server is supposed to be headless (at least if nothing goes wrong ;-), we need to enable networking and SSH. I have DHCP in my home network, so enabling NetworkManager.service is just enough to get access to the network.

systemctl enable NetworkManager.service

Now enable SSH.

systemctl enable sshd.service

Save public key from my workstation to authorized_keys.

# su dstrelnikov
# mkdir -p -m 700 /home/dstrelnikov/.ssh
# touch /home/dstrelnikov/.ssh/authorized_keys
# chmod 600 /home/dstrelnikov/.ssh/authorized_keys
# nano /home/dstrelnikov/.ssh/authorized_keys
[...]

Installing boot loader

Here we install systemd-boot with a simple command.

# bootctl install
Created "/boot/EFI".
Created "/boot/EFI/systemd".
Created "/boot/EFI/BOOT".
Created "/boot/loader".
Created "/boot/loader/entries".
Created "/boot/EFI/Linux".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
Random seed file /boot/loader/random-seed successfully written (512 bytes).
Created EFI boot entry "Linux Boot Manager".

Now copy an entry file from a template and adjust it to our needs.

# cp /usr/share/systemd/bootctl/arch.conf /boot/loader/entries/arch-intel.conf

Here is what I got in the end.

title   Arch Linux (LTS, Intel)
linux   /vmlinuz-linux-lts
initrd  /intel-ucode.img
initrd  /initramfs-linux-lts.img
options root=/dev/mapper/system_crypt rootflags=noatime,discard=async,space_cache=v2 rw add_efi_memmap mitigations=off audit=0

Let us make it the default option in /boot/loader/loader.conf:

timeout 3
console-mode max
default arch-intel.conf

Configuring mkinitcpio

Since we are using full-disk encryption, in order to be able to type the LUKS password we need to edit /etc/mkinitcpio.conf and specify the following modules.

[...]
HOOKS=(base udev systemd keyboard autodetect modconf block sd-encrypt filesystems fsck)
[...]

Now let’s edit /etc/crypttab.initramfs and make an entry to open the encrypted container on the early stage.

system_crypt UUID=1a16af12-33f1-4aee-a1a8-5fa8d193f596 none luks,discard

Now initramfs can be generated by

mkinitcpio -P

Booting into the new system

Now the basic setup is done. Go back to the host system and unmount the bootstraped system.

$ sudo umount /mnt/bootstrap/boot 
$ sudo umount /mnt/bootstrap

If we have done everything correct, now we are able to boot into the new system. If the system boots successfully then it’s time to put the SSD into the server.