Compiling Lustre Client

Some references for compiling lustre:

Compiling in Container

The processes for compiling the client directly on the host where it will be used is easier than compiling it in a container to be used for a targeted environment. Thus, I’ve separated out the two following sections:

Compiling on Host
Compiling in Container

Compiling on Host

Using Rocky 8.6 Host

Here we’ll be using Rocky Linux 8.6 Minimal as our base installation medium, and will be compiling on the baremetal host OS. This will effectively allow us to compile Lustre for the OS/kernel installed.

Prerequisites and Dependencies

There are quite a few prerequisites that need to be satisfied to get a proper build environment before we can compile anything.

Setup Rocky 8.6 Repositories

Out of the box, Rocky 8.6 points to Rocky 8.8 repositories for packages. This will become problematic if we try to install packages that depend on a newer version of the kernel, as they’ll install that newer version of the kernel-core and kernel-devel. The kernel version Rocky 8.6 runs out of the box is 4.18.0-372.9.1.el8.x86_64:

[root@mawenzi-04 ~]# uname -r
4.18.0-372.9.1.el8.x86_64

However, if you try to install packages like kernel-devel, you’ll see it will find the one for a newer kernel.

[root@mawenzi-04 ~]# dnf search -v kernel-devel
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
DNF version: 4.7.0
cachedir: /var/cache/dnf
Last metadata expiration check: 1:09:48 ago on Fri 21 Jul 2023 10:34:42 AM MDT.
=================================================================== Name Exactly Matched: kernel-devel ====================================================================
kernel-devel.x86_64 : Development package for building kernel modules to match the kernel
Repo        : baseos
Matched from:
Provide    : kernel-devel = 4.18.0-477.15.1.el8_8

This will cause incompatibilities down the road for compiling Lustre since we’ll essentially be compiling it for a newer kernel version, the one Rocky 8.8 uses.

Instead, we need to change Rocky 8.6’s yum repository configurations over to use the true Rocky 8.6 Vault repositories.

To view what repositories are currently in enabled, run yum repolist -v. Note how all the repos referenced are for Rocky 8.8.

[root@mawenzi-04 lustre-release]# yum repolist -v
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
YUM version: 4.7.0
cachedir: /var/cache/dnf
Last metadata expiration check: 0:12:12 ago on Fri 21 Jul 2023 10:34:42 AM MDT.
Repo-id            : appstream
Repo-name          : Rocky Linux 8 - AppStream
Repo-revision      : 8.8
Repo-distro-tags      : [cpe:/o:rocky:rocky:8]:  ,  , 8, L, R, c, i, k, n, o, u, x, y
Repo-updated       : Wed 19 Jul 2023 10:08:22 AM MDT
Repo-pkgs          : 7,550
Repo-available-pkgs: 6,103
Repo-size          : 12 G
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=AppStream-8
Repo-baseurl       : http://mirrors.rit.edu/rocky/8.8/AppStream/x86_64/os/ (43 more)
Repo-expire        : 172,800 second(s) (last: Fri 21 Jul 2023 10:34:40 AM MDT)
Repo-filename      : /etc/yum.repos.d/Rocky-AppStream.repo

Repo-id            : baseos
Repo-name          : Rocky Linux 8 - BaseOS
Repo-revision      : 8.8
Repo-distro-tags      : [cpe:/o:rocky:rocky:8]:  ,  , 8, L, R, c, i, k, n, o, u, x, y
Repo-updated       : Wed 19 Jul 2023 10:08:27 AM MDT
Repo-pkgs          : 1,857
Repo-available-pkgs: 1,855
Repo-size          : 2.3 G
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=BaseOS-8
Repo-baseurl       : http://mirrors.rit.edu/rocky/8.8/BaseOS/x86_64/os/ (43 more)
Repo-expire        : 172,800 second(s) (last: Fri 21 Jul 2023 10:34:41 AM MDT)
Repo-filename      : /etc/yum.repos.d/Rocky-BaseOS.repo

Repo-id            : extras
Repo-name          : Rocky Linux 8 - Extras
Repo-revision      : 1689203561
Repo-distro-tags      : [cpe:/o:rocky:rocky:8]:  ,  , 8, L, R, c, i, k, n, o, u, x, y
Repo-updated       : Wed 12 Jul 2023 05:12:41 PM MDT
Repo-pkgs          : 52
Repo-available-pkgs: 52
Repo-size          : 3.3 M
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=extras-8
Repo-baseurl       : https://rocky-linux-us-west4.production.gcp.mirrors.ctrliq.cloud/pub/rocky//8.8/extras/x86_64/os/ (43 more)
Repo-expire        : 172,800 second(s) (last: Fri 21 Jul 2023 10:34:42 AM MDT)
Repo-filename      : /etc/yum.repos.d/Rocky-Extras.repo
Total packages: 9,459

We’ll switch these over to the Rocky 8.6 repos.

Create a new file /etc/yum.repos.d/Rocky-86-Development.repo by running the following command

cat >> /etc/yum.repos.d/Rocky-86-Development.repo << EOF
# Rocky-AppStream.repo
[appstream86]
name=Rocky Linux $releasever - AppStream
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=AppStream-$releasever
baseurl=https://dl.rockylinux.org/vault/rocky/8.6/AppStream/x86_64/os/
gpgcheck=1
enabled=1
countme=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

# Rocky-BaseOS.repo
[baseos86]
name=Rocky Linux $releasever - BaseOS
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=BaseOS-$releasever
baseurl=https://dl.rockylinux.org/vault/rocky/8.6/BaseOS/x86_64/os/
gpgcheck=1
enabled=1
countme=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

# Rocky-Devel.repo
[devel86]
name=Rocky Linux $releasever - Devel WARNING! FOR BUILDROOT AND KOJI USE
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=Devel-$releasever
baseurl=https://dl.rockylinux.org/vault/rocky/8.6/Devel/x86_64/os/
gpgcheck=1
enabled=1
countme=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

# Rocky-Extras.repo
[extras86]
name=Rocky Linux $releasever - Extras
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=extras-$releasever
baseurl=https://dl.rockylinux.org/vault/rocky/8.6/extras/x86_64/os/
gpgcheck=1
enabled=1
countme=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

# Rocky-PowerTools.repo
[powertools86]
name=Rocky Linux $releasever - PowerTools
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=PowerTools-$releasever
baseurl=https://dl.rockylinux.org/vault/rocky/8.6/PowerTools/x86_64/os/
gpgcheck=1
enabled=1
countme=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
EOF

Then, disable the original 8.8 repos, and enable the 8.6 repos (if you haven’t already).

dnf config-manager --disable appstream baseos extras
dnf config-manager --enable appstream86 baseos86 extras86 devel86 powertools86

Now, when we search for kernel-devel, we see the correct kernel version:

[root@mawenzi-04 yum.repos.d]# dnf search kernel-devel -v
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
DNF version: 4.7.0
cachedir: /var/cache/dnf
Rocky Linux 8 - AppStream                                                                                                                  1.4 MB/s |  11 MB     00:07
Rocky Linux 8 - BaseOS                                                                                                                     3.7 MB/s | 9.0 MB     00:02
Rocky Linux 8 - Extras                                                                                                                     8.0 kB/s |  12 kB     00:01
=================================================================== Name Exactly Matched: kernel-devel ====================================================================
kernel-devel.x86_64 : Development package for building kernel modules to match the kernel
Repo        : baseos86
Matched from:
Provide    : kernel-devel = 4.18.0-372.32.1.el8_6

Prerequisite Packages

Install prerequisite software tools on the build server:

First, install DNF plugins package.

dnf install -y dnf-plugins-core

Next, install Extra Packages for Enterprise Linux (EPEL) release.

dnf install -y epel-release

Install the "Development Tools" group. This will also pull in the kernel-headers and kernel-devel packages as dependencies.

dnf groupinstall -y "Development Tools"

Install the remaining dependecies for compiling the Lustre client:

dnf install -y         \
    audit-libs-devel   \
    bc                 \
    binutils-devel     \
    git                \
    json-c-devel       \
    kernel-abi-stablelists \
    kernel-devel       \
    kernel-debug-devel \
    kernel-rpm-macros  \
    libaio-devel       \
    libattr-devel      \
    libblkid-devel     \
    libmount           \
    libmount-devel     \
    libnl3-devel       \
    libselinux-devel   \
    libssh-devel       \
    libtirpc-devel     \
    libuuid-devel      \
    libyaml            \
    libyaml-devel      \
    llvm-toolset       \
    pciutils-devel     \
    ncurses-devel      \
    openssl-devel      \
    perl               \
    perl-devel         \
    python39           \
    python3-devel      \
    python3-docutils   \
    redhat-lsb         \
    texinfo            \
    texinfo-tex

Update dnf:

dnf update

MOFED Drivers

If we’re compiling the Lustre client with MOFED InfiniBand support, we need to make sure we’ve built and installed MOFED drivers for the unpatched kernel we’re currently running. In our case for Rocky 8.6, this is 4.18.0-372.9.1.el8.x86_64.

View Nvidia’s Installing Mellanox OFED or my InfiniBand MOFED Installation notes for more depth information. Below is a brief synopsis of the install process.

Go to the Nvidia Download Center, and download 5.4-3.7.5.0-LTS → RHEL/CentOS/Rocky → RHEL/Rocky 8.6 → MLNX_OFED_LINUX-5.4-3.7.5.0-rhel8.6-x86_64.tgz to your build environment home.

Untar it: tar -xzvf MLNX_OFED_LINUX-5.4-3.7.5.0-rhel8.6-x86_64.tgz

There’s a few ways to go about installing MOFED here. One option is to use the mlnxofedinstall script provided to install. The other is to use your package manager, in our case DNF, to install the RPMs as a local repo.

Using DNF:

Create a new yum repo, enabled, with GPG check disabled.

cat >> /etc/yum.repos.d/Mellanox-OFED.repo << EOF
[mlnx_ofed]
name=MLNX_OFED Repository
baseurl=file:///root/MLNX_OFED_LINUX-5.4-3.7.5.0-rhel8.6-x86_64/RPMS
enabled=1
gpgcheck=0
EOF

Running a dnf search mlnx-ofed- should show:

[root@mawenzi-04 MLNX_OFED_LINUX-5.4-3.7.5.0-rhel8.6-x86_64]# dnf search mlnx-ofed-
Rocky Linux 8 - Devel WARNING! FOR BUILDROOT AND KOJI USE                                                                                  2.1 MB/s | 4.3 MB     00:02
Last metadata expiration check: 0:00:01 ago on Fri 04 Aug 2023 09:45:38 AM MDT.
======================================================================== Name Matched: mlnx-ofed- =========================================================================
mlnx-ofed-all.noarch : MLNX_OFED all installer package  (with KMP support)
mlnx-ofed-all-user-only.noarch : MLNX_OFED all-user-only installer package  (User Space packages only)
mlnx-ofed-basic.noarch : MLNX_OFED basic installer package  (with KMP support)
mlnx-ofed-basic-user-only.noarch : MLNX_OFED basic-user-only installer package  (User Space packages only)
mlnx-ofed-bluefield.noarch : MLNX_OFED bluefield installer package  (with KMP support)
mlnx-ofed-bluefield-user-only.noarch : MLNX_OFED bluefield-user-only installer package  (User Space packages only)
mlnx-ofed-dpdk.noarch : MLNX_OFED dpdk installer package  (with KMP support)
mlnx-ofed-dpdk-upstream-libs.noarch : MLNX_OFED dpdk-upstream-libs installer package  (with KMP support)
mlnx-ofed-dpdk-upstream-libs-user-only.noarch : MLNX_OFED dpdk-upstream-libs-user-only installer package  (User Space packages only)
mlnx-ofed-dpdk-user-only.noarch : MLNX_OFED dpdk-user-only installer package  (User Space packages only)
mlnx-ofed-eth-only-user-only.noarch : MLNX_OFED eth-only-user-only installer package  (User Space packages only)
mlnx-ofed-guest.noarch : MLNX_OFED guest installer package  (with KMP support)
mlnx-ofed-guest-user-only.noarch : MLNX_OFED guest-user-only installer package  (User Space packages only)
mlnx-ofed-hpc.noarch : MLNX_OFED hpc installer package  (with KMP support)
mlnx-ofed-hpc-user-only.noarch : MLNX_OFED hpc-user-only installer package  (User Space packages only)
mlnx-ofed-hypervisor.noarch : MLNX_OFED hypervisor installer package  (with KMP support)
mlnx-ofed-hypervisor-user-only.noarch : MLNX_OFED hypervisor-user-only installer package  (User Space packages only)
mlnx-ofed-kernel-only.noarch : MLNX_OFED kernel-only installer package  (with KMP support)
mlnx-ofed-vma.noarch : MLNX_OFED vma installer package  (with KMP support)
mlnx-ofed-vma-eth.noarch : MLNX_OFED vma-eth installer package  (with KMP support)
mlnx-ofed-vma-eth-user-only.noarch : MLNX_OFED vma-eth-user-only installer package  (User Space packages only)
mlnx-ofed-vma-user-only.noarch : MLNX_OFED vma-user-only installer package  (User Space packages only)
mlnx-ofed-vma-vpi.noarch : MLNX_OFED vma-vpi installer package  (with KMP support)
mlnx-ofed-vma-vpi-user-only.noarch : MLNX_OFED vma-vpi-user-only installer package  (User Space packages only)
mlnx-ofed-xlio.noarch : MLNX_OFED xlio installer package  (with KMP support)
mlnx-ofed-xlio-user-only.noarch : MLNX_OFED xlio-user-only installer package  (User Space packages only)

Install mlnx-ofed-all (with KMP support) using dnf:

dnf install mlnx-ofed-all

Once this has finished, you should have installed all the required kernel modules and supporting packages that the Lustre build process needs to reference. It is not required to actually load the modules; they just need to be present. You can check their existence under /usr/src/:

[root@mawenzi-04 MLNX_OFED_LINUX-5.4-3.7.5.0-rhel8.6-x86_64]# ls /usr/src/
debug  kernels  mlnx-ofa_kernel-5.4  ofa_kernel  ofa_kernel-5.4

Using the Install Script:

TODO: This has not been explored enough to capture notes on.

Building Client

Clone the Lustre repository from whamcloud.

git clone https://review.whamcloud.com/fs/lustre-release

Prepare the Makefile:

sh autogen.sh

Configure the build for just building the client, and point it at your kernel/MOFED sources:

./configure \
    --disable-server \
    --enable-client \
    --with-linux=/usr/src/kernels/4.18.0-372.32.1.el8_6.x86_64 \
    --with-o2ib=/usr/src/ofa_kernel/default

make the RPMs:

make rpms

Once it finishes, you should end up with all the RPMs and sources dumped into the lustre-release repo root:

[root@mawenzi-04 lustre-release]# ls ~/lustre-release/{*.rpm,*.tar.gz}
/root/lustre-release/kmod-lustre-client-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/kmod-lustre-client-debuginfo-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/kmod-lustre-client-tests-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/kmod-lustre-client-tests-debuginfo-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/lustre-2.15.56_124_g3ebc8e0-1.src.rpm
/root/lustre-release/lustre-2.15.56_124_g3ebc8e0.tar.gz
/root/lustre-release/lustre-client-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/lustre-client-debuginfo-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/lustre-client-debugsource-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/lustre-client-devel-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/lustre-client-tests-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/lustre-client-tests-debuginfo-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm
/root/lustre-release/lustre-iokit-2.15.56_124_g3ebc8e0-1.el8.x86_64.rpm

Compiling in Container

Compiling within an OpenSUSE 15.5 Docker Container

Create the following zypper repofile for Mellanox OFED:

cat >> /etc/zypp/repos.d/Mellanox-OFED.repo << EOF
[mlnx_ofed_5.8-3.0.7.0_base]
name=Mellanox Technologies sles15sp5-$basearch mlnx_ofed 5.8-3.0.7.0 GA
baseurl=http://linux.mellanox.com/public/repo/mlnx_ofed/5.8-3.0.7.0/sles15sp5/$basearch
enabled=1
autorefresh=1
path=/
type=rpm-md
keeppackages=0
EOF

Create the following Dockerfile:

FROM opensuse-leap:15.5 AS oss-leap-155

RUN zypper install -y vim tar git python39

RUN zypper install -y \
    kernel-default-devel \
    kernel-devel \
    kernel-syms \
    kernel-source

RUN zypper install -y libyaml-devel libmount-devel


ENTRYPOINT ["/usr/bin/bash"]

On-Host Builds

Building for Ubuntu

Here we’re building for Ubuntu 22.04.

Install the build dependencies:

#!/bin/bash

set -ex

# Install dependencies
apt-get install -y \
	module-assistant \
	libreadline-dev \
	debhelper \
	dpatch \
	automake \
	pkg-config \
	libtool \
	libyaml-dev \
	libnl-genl-3-dev \
	libselinux-dev \
	libsnmp-dev \
	mpi-default-dev \
	bzip2 \
	quilt \
	linux-headers-$(uname -r) \
	rsync \
	libssl-dev \
	libpython3-dev \
	swig

LUSTRE_DIR=/root/ccarlson/lustre-release

# e.g: generic
KERNEL_FLAVOR=$(uname -r | tr '-' '\n' | tail -1)

# e.g: 5.15.0-83
KERNEL_VERSION=$(uname -r | sed -e "s:-${KERNEL_FLAVOR}$::g")

# e.g: /usr/src/linux-headers-5.15.0-83
LINUX_KERNEL_DIR=$(ls -d -1 /usr/src/linux-*headers-${KERNEL_VERSION})

# e.g: /usr/src/linux-headers-5.15.0-83-generic/
LINUX_OBJ_DIR=$(ls -d -1 /usr/src/linux-*headers-${KERNEL_VERSION}-${KERNEL_FLAVOR})

# e.g: /usr/src/ofa_kernel/default
# MOFED_SRC_DIR="/usr/src/mlnx-ofed-kernel-23.07"
MOFED_SRC_DIR="/usr/src/ofa_kernel/default"

# Configure build options then build debian packages
cd $LUSTRE_DIR && sh autogen.sh && ./configure \
	--disable-server \
	--enable-client \
	--with-linux=${LINUX_KERNEL_DIR} \
	--with-linux-obj=${LINUX_OBJ_DIR} \
	--with-o2ib=${MOFED_SRC_DIR} \
	&& make debs