Install On Debian

Cluster Setting

• 1 Manager
• 1 Login Node
• 2 Compute nodes

Software Version:

Prepare Steps (All Nodes)

1. Modify the /etc/apt/sources.list file Using tuna mirror

cat > /etc/apt/sources.list << EOF
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware
deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware

deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware
deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware

deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware
deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware

deb https://mirrors.tuna.tsinghua.edu.cn/debian-security/ bookworm-security main contrib non-free non-free-firmware
deb-src https://mirrors.tuna.tsinghua.edu.cn/debian-security/ bookworm-security main contrib non-free non-free-firmware
EOF

allow-hotplug enps08
iface enps08 inet dhcp

systemctl restart networking

2. Update apt cache

apt clean all && apt update

3. Set hostname on each node

hostnamectl set-hostname manage01

hostnamectl set-hostname login01

hostnamectl set-hostname compute01

hostnamectl set-hostname compute02

4. Set hosts file

cat >> /etc/hosts << EOF
192.168.56.115 manage01
192.168.56.116 login01
192.168.56.117 compute01
192.168.56.118 compute02
EOF

5. Disable firewall

systemctl stop nftables && systemctl disable nftables

6. Install packages ntpdate

apt-get -y install ntpdate

7. Sync server time

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time.windows.com

8. Add cron job to sync time

crontab -e
*/5 * * * * /usr/sbin/ntpdate time.windows.com

9. Create ssh key pair on each node

ssh-keygen -t rsa -b 4096 -C $HOSTNAME

10. Test ssh login other nodes without password

ssh-copy-id -i ~/.ssh/id_rsa.pub root@login01
ssh-copy-id -i ~/.ssh/id_rsa.pub root@compute01
ssh-copy-id -i ~/.ssh/id_rsa.pub root@compute02

ssh-copy-id -i ~/.ssh/id_rsa.pub root@manage01
ssh-copy-id -i ~/.ssh/id_rsa.pub root@compute01
ssh-copy-id -i ~/.ssh/id_rsa.pub root@compute02

Install Components

1. Install NFS server (Manager Node)

there are many ways to install NFS server

• using yum install -y nfs-utils, check https://pkuhpc.github.io/SCOW/docs/hpccluster/nfs
• using apt install -y nfs-kernel-server, check https://www.linuxtechi.com/how-to-install-nfs-server-on-debian/
• or you can directly mount other shared storage.

create shared folder

mkdir /data
chmod 755 /data

modify vim /etc/exports

/data *(rw,sync,insecure,no_subtree_check,no_root_squash)

start nfs server

systemctl start rpcbind 
systemctl start nfs-server 

systemctl enable rpcbind 
systemctl enable nfs-server

check nfs server

showmount -e localhost

# Output
Export list for localhost:
/data *

2. Install munge service

• add user munge (All Nodes)

groupadd -g 1108 munge
useradd -m -c "Munge Uid 'N' Gid Emporium" -d /var/lib/munge -u 1108 -g munge -s /sbin/nologin munge

• Install rng-tools-debian (Manager Nodes)

apt-get install -y rng-tools-debian

# modify service script
vim /usr/lib/systemd/system/rngd.service

[Service]
ExecStart=/usr/sbin/rngd -f -r /dev/urandom

systemctl daemon-reload
systemctl start rngd
systemctl enable rngd

apt-get install -y libmunge-dev libmunge2 munge

• generate secret key (Manager Nodes)

dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key

• copy munge.key from manager node to the rest node (All Nodes)

scp -p /etc/munge/munge.key root@login01:/etc/munge/
scp -p /etc/munge/munge.key root@compute01:/etc/munge/
scp -p /etc/munge/munge.key root@compute02:/etc/munge/

• grant privilege on munge.key (All Nodes)

chown munge: /etc/munge/munge.key
chmod 400 /etc/munge/munge.key

systemctl start munge
systemctl enable munge

Using systemctl status munge to check if the service is running

• test munge

munge -n | ssh compute01 unmunge

3. Install Mariadb (Manager Nodes)

apt-get install -y mariadb-server

• create database and user

systemctl start mariadb
systemctl enable mariadb

ROOT_PASS=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 16) 
mysql -e "CREATE USER root IDENTIFIED BY '${ROOT_PASS}'"
mysql -uroot -p$ROOT_PASS -e 'create database slurm_acct_db'

• create user slurm，and grant all privileges on database slurm_acct_db

mysql -uroot -p$ROOT_PASS

create user slurm;

grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by '123456' with grant option;

flush privileges;

• create Slurm user

groupadd -g 1109 slurm
useradd -m -c "Slurm manager" -d /var/lib/slurm -u 1109 -g slurm -s /bin/bash slurm

Install Slurm (All Nodes)

• Install basic Debian package build requirements:

apt-get install -y build-essential fakeroot devscripts equivs

• Unpack the distributed tarball:

wget https://download.schedmd.com/slurm/slurm-24.05.2.tar.bz2 -O slurm-24.05.2.tar.bz2 &&
tar -xaf slurm*tar.bz2

• cd to the directory containing the Slurm source:

cd slurm-24.05.2 &&   mkdir -p /etc/slurm && ./configure 

• compile slurm

make install

• modify configuration files (Manager Nodes)

  • modify /etc/slurm/slurm.conf Refer to slurm.conf

  cp /root/slurm-24.05.2/etc/slurm.conf.example /etc/slurm/slurm.conf
  vim /etc/slurm/slurm.conf

  focus on these options:

  SlurmctldHost=manage
  
  AccountingStorageEnforce=associations,limits,qos
  AccountingStorageHost=manage
  AccountingStoragePass=/var/run/munge/munge.socket.2
  AccountingStoragePort=6819  
  AccountingStorageType=accounting_storage/slurmdbd  
  
  JobCompHost=localhost
  JobCompLoc=slurm_acct_db
  JobCompPass=123456
  JobCompPort=3306
  JobCompType=jobcomp/mysql
  JobCompUser=slurm
  JobContainerType=job_container/none
  JobAcctGatherType=jobacct_gather/linux

  • modify /etc/slurm/slurmdbd.conf Refer to slurmdbd.conf

  cp /root/slurm-24.05.2/etc/slurmdbd.conf.example /etc/slurm/slurmdbd.conf
  vim /etc/slurm/slurmdbd.conf

  • modify /etc/slurm/cgroup.conf

  cp /root/slurm-24.05.2/etc/cgroup.conf.example /etc/slurm/cgroup.conf

  • send configuration files to other nodes

  scp -r /etc/slurm/*.conf  root@login01:/etc/slurm/
  scp -r /etc/slurm/*.conf  root@compute01:/etc/slurm/
  scp -r /etc/slurm/*.conf  root@compute02:/etc/slurm/

• grant privilege on some directories (All Nodes)

mkdir /var/spool/slurmd
chown slurm: /var/spool/slurmd
mkdir /var/log/slurm
chown slurm: /var/log/slurm

mkdir /var/spool/slurmctld
chown slurm: /var/spool/slurmctld

chown slurm: /etc/slurm/slurmdbd.conf
chmod 600 /etc/slurm/slurmdbd.conf

• start slurm services on each node

systemctl start slurmdbd
systemctl enable slurmdbd

systemctl start slurmctld
systemctl enable slurmctld

systemctl start slurmd
systemctl enable slurmd

```text
# vim /usr/lib/systemd/system/slurmdbd.service


[Unit]
Description=Slurm DBD accounting daemon
After=network-online.target remote-fs.target munge.service mysql.service mysqld.service mariadb.service sssd.service
Wants=network-online.target
ConditionPathExists=/etc/slurm/slurmdbd.conf

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/slurmdbd
EnvironmentFile=-/etc/default/slurmdbd
User=slurm
Group=slurm
RuntimeDirectory=slurmdbd
RuntimeDirectoryMode=0755
ExecStart=/usr/local/sbin/slurmdbd -D -s $SLURMDBD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536


# Uncomment the following lines to disable logging through journald.
# NOTE: It may be preferable to set these through an override file instead.
#StandardOutput=null
#StandardError=null

[Install]
WantedBy=multi-user.target
```

```text
# vim /usr/lib/systemd/system/slurmctld.service


[Unit]
Description=Slurm controller daemon
After=network-online.target remote-fs.target munge.service sssd.service
Wants=network-online.target
ConditionPathExists=/etc/slurm/slurm.conf

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/slurmctld
EnvironmentFile=-/etc/default/slurmctld
User=slurm
Group=slurm
RuntimeDirectory=slurmctld
RuntimeDirectoryMode=0755
ExecStart=/usr/local/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536


# Uncomment the following lines to disable logging through journald.
# NOTE: It may be preferable to set these through an override file instead.
#StandardOutput=null
#StandardError=null

[Install]
WantedBy=multi-user.target
```

```text
# vim /usr/lib/systemd/system/slurmd.service


[Unit]
Description=Slurm node daemon
After=munge.service network-online.target remote-fs.target sssd.service
Wants=network-online.target
#ConditionPathExists=/etc/slurm/slurm.conf

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/slurmd
EnvironmentFile=-/etc/default/slurmd
RuntimeDirectory=slurm
RuntimeDirectoryMode=0755
ExecStart=/usr/local/sbin/slurmd --systemd $SLURMD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
LimitNOFILE=131072
LimitMEMLOCK=infinity
LimitSTACK=infinity
Delegate=yes


# Uncomment the following lines to disable logging through journald.
# NOTE: It may be preferable to set these through an override file instead.
#StandardOutput=null
#StandardError=null

[Install]
WantedBy=multi-user.target
```

systemctl start slurmd
systemctl enable slurmd

systemctl start slurmd
systemctl enable slurmd

systemctl start slurmd
systemctl enable slurmd

Test Your Slurm Cluster (Login Node)

• check cluster configuration

scontrol show config

• check cluster status

sinfo
scontrol show partition
scontrol show node

• submit job

srun -N2 hostname
scontrol show jobs

• check job status

check job status
squeue -a

Install From Binary

Typically, there are three nodes are required to run Slurm.

1 Manage(Manager Node), 1 Login Node and N Compute(Cmp).

but you can choose to install all service in single node. check

Prequisites

1. change hostname (All Nodes)

   hostnamectl set-hostname (manager|auth|computeXX)

2. modify /etc/hosts (All Nodes)

   echo "192.aa.bb.cc (manager|auth|computeXX)" >> /etc/hosts

3. disable firewall, selinux, dnsmasq, swap (All Nodes). more detail here
4. NFS Server (Manager Node). NFS is used as the default file system for the Slurm accounting database.
5. [NFS Client] (All Nodes). all node should mount the NFS share
   Install NFS Client

   mount <$nfs_server>:/data /data -o proto=tcp -o nolock

6. Munge (All Nodes). The auth/munge plugin will be built if the MUNGE authentication development library is installed. MUNGE is used as the default authentication mechanism.
   Install Munge

   All node need to have the munge user and group.

   groupadd -g 1108 munge
   useradd -m -c "Munge Uid 'N' Gid Emporium" -d /var/lib/munge -u 1108 -g munge -s /sbin/nologin munge

   yum install epel-release -y
   yum install munge munge-libs munge-devel -y

   Create global secret key

   /usr/sbin/create-munge-key -r
   dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key

   sync secret to the rest of nodes

   scp -p /etc/munge/munge.key root@<$rest_node>:/etc/munge/

   ssh root@<$rest_node> "chown munge: /etc/munge/munge.key && chmod 400 /etc/munge/munge.key"
   ssh root@<$rest_node> "systemctl start munge && systemctl enable munge"

   test munge if it works

   munge -n | unmunge

7. Database (Manager Node). MySQL support for accounting will be built if the MySQL or MariaDB development library is present. A currently supported version of MySQL or MariaDB should be used.
   Install MariaDB

   install mariadb

   yum -y install mariadb-server
   systemctl start mariadb && systemctl enable mariadb
   ROOT_PASS=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 16) 
   mysql -e "CREATE USER root IDENTIFIED BY '${ROOT_PASS}'"

   login mysql

   mysql -u root -p${ROOT_PASS}
   create database slurm_acct_db;
   create user slurm;
   grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by '123456' with grant option;
   flush privileges;
   quit

Install Slurm

1. create slurm user (All Nodes)

   groupadd -g 1109 slurm
   useradd -m -c "slurm manager" -d /var/lib/slurm -u 1109 -g slurm -s /bin/bash slurm

2. send configuration (Manager Node)

    scp -r /etc/slurm/*.conf  root@<$rest_node>:/etc/slurm/
    ssh rootroot@<$rest_node> "mkdir /var/spool/slurmd && chown slurm: /var/spool/slurmd"
    ssh rootroot@<$rest_node> "mkdir /var/log/slurm && chown slurm: /var/log/slurm"
    ssh rootroot@<$rest_node> "mkdir /var/spool/slurmctld && chown slurm: /var/spool/slurmctld"

3. start service (Manager Node)

   ssh rootroot@<$rest_node> "systemctl start slurmdbd && systemctl enable slurmdbd"
   ssh rootroot@<$rest_node> "systemctl start slurmctld && systemctl enable slurmctld"

4. start service (All Nodes)

   ssh rootroot@<$rest_node> "systemctl start slurmd && systemctl enable slurmd"

Test

1. show cluster status

scontrol show config

sinfo
scontrol show partition
scontrol show node

2. submit job

srun -N2 hostname
scontrol show jobs

3. check job status

squeue -a

Reference:

1. https://slurm.schedmd.com/documentation.html
2. https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/
3. https://github.com/Artlands/Install-Slurm

Install From Helm Chart

Despite the complex binary installation, helm chart is a better way to install slurm.

Source code could be found from https://github.com/AaronYang0628/slurm-on-k8s

Prequisites

1. Kubernetes has installed, if not check 🔗link
2. Helm binary has installed, if not check 🔗link

Installation

1. get helm repo and update

   helm repo add ay-helm-mirror https://aaronyang0628.github.io/helm-chart-mirror/charts

2. install slurm chart

   # wget -O slurm.values.yaml https://raw.githubusercontent.com/AaronYang0628/slurm-on-k8s/refs/heads/main/chart/values.yaml
   helm install slurm ay-helm-mirror/chart -f slurm.values.yaml --version 1.0.10

   Or you can get template values.yaml from https://raw.githubusercontent.com/AaronYang0628/helm-chart-mirror/refs/heads/main/templates/slurm/slurm.values.yaml

3. check chart status

   helm -n slurm list

Install From K8s Operator

Despite the complex binary installation, using k8s operator is a better way to install slurm.

Source code could be found from https://github.com/AaronYang0628/slurm-on-k8s

Prequisites

1. Kubernetes has installed, if not check 🔗link
2. Helm binary has installed, if not check 🔗link

Installation

1. deploy slurm operator

   kubectl apply -f https://raw.githubusercontent.com/AaronYang0628/helm-chart-mirror/refs/heads/main/templates/slurm/operator_install.yaml

   Expectd Output

   [root@ay-zj-ecs operator]# kubectl apply -f https://raw.githubusercontent.com/AaronYang0628/helm-chart-mirror/refs/heads/main/templates/slurm/operator_install.yaml
   namespace/slurm created
   customresourcedefinition.apiextensions.k8s.io/slurmdeployments.slurm.ay.dev created
   serviceaccount/slurm-operator-controller-manager created
   role.rbac.authorization.k8s.io/slurm-operator-leader-election-role created
   clusterrole.rbac.authorization.k8s.io/slurm-operator-manager-role created
   clusterrole.rbac.authorization.k8s.io/slurm-operator-metrics-auth-role created
   clusterrole.rbac.authorization.k8s.io/slurm-operator-metrics-reader created
   clusterrole.rbac.authorization.k8s.io/slurm-operator-slurmdeployment-admin-role created
   clusterrole.rbac.authorization.k8s.io/slurm-operator-slurmdeployment-editor-role created
   clusterrole.rbac.authorization.k8s.io/slurm-operator-slurmdeployment-viewer-role created
   rolebinding.rbac.authorization.k8s.io/slurm-operator-leader-election-rolebinding created
   clusterrolebinding.rbac.authorization.k8s.io/slurm-operator-manager-rolebinding created
   clusterrolebinding.rbac.authorization.k8s.io/slurm-operator-metrics-auth-rolebinding created
   service/slurm-operator-controller-manager-metrics-service created
   deployment.apps/slurm-operator-controller-manager created

2. check operator status

   kubectl -n slurm get pod

   Expectd Output

   [root@ay-zj-ecs operator]# kubectl -n slurm get pod
   NAME                                READY   STATUS    RESTARTS   AGE
   slurm-operator-controller-manager   1/1     Running   0          27s

3. apply CRD slurmdeployment

   kubectl apply -f https://raw.githubusercontent.com/AaronYang0628/helm-chart-mirror/refs/heads/main/templates/slurm/slurmdeployment.zj.values.yaml

   Expectd Output

   [root@ay-zj-ecs operator]# kubectl apply -f https://raw.githubusercontent.com/AaronYang0628/helm-chart-mirror/refs/heads/main/templates/slurm/slurmdeployment.zj.values.yaml
   slurmdeployment.slurm.ay.dev/lensing created

4. check operator status

   kubectl get slurmdeployment
   kubectl -n slurm logs -f deploy/slurm-operator-controller-manager
   # kubectl get slurmdep
   # kubectl -n test get pods

   Expectd Output

   [root@ay-zj-ecs ~]# kubectl get slurmdep -w
   NAME      CPU   GPU   LOGIN   CTLD   DBD   DBSVC   JOB COMMAND                     STATUS
   lensing   0/1   0/0   0/1     0/1    0/1   0/1     sh -c srun -N 2 /bin/hostname   
   lensing   1/2   0/0   1/1     1/1    1/1   1/1     sh -c srun -N 2 /bin/hostname   
   lensing   2/2   0/0   1/1     1/1    1/1   1/1     sh -c srun -N 2 /bin/hostname   

5. upgrade slurmdep

   kubectl edit slurmdep lensing
   # set SlurmCPU.replicas = 3

   Expectd Output

   [root@ay-zj-ecs ~]# kubectl edit slurmdep lensing
   slurmdeployment.slurm.ay.dev/lensing edited
   
   [root@ay-zj-ecs ~]# kubectl get slurmdep -w
   NAME      CPU   GPU   LOGIN   CTLD   DBD   DBSVC   JOB COMMAND                     STATUS
   lensing   2/2   0/0   1/1     1/1    1/1   1/1     sh -c srun -N 2 /bin/hostname   
   lensing   2/3   0/0   1/1     1/1    1/1   1/1     sh -c srun -N 2 /bin/hostname   
   lensing   3/3   0/0   1/1     1/1    1/1   1/1     sh -c srun -N 2 /bin/hostname   

Try OpenSCOW

What is SCOW?

SCOW is a HPC cluster management system built by PKU.

SCOW used four virtual machines to run slurm cluster. It is a good choice for you to learn how to use slurm.

You should check https://pkuhpc.github.io/OpenSCOW/docs/hpccluster, it works well.