Linux

• LXC Container - Requirements & Basics
• Jekyll - Install & Basics
• nftables - Firewall
• Debian - Tasks for new installs
• mdadm - Software Raid
• LUKS - Encryption
• Caddy webserver - valid certificates for local services
• Fail2Ban
• Docker @debian
• LVM2
• iSCSI
• Proxmox BackupServer - pbs

LXC Container - Requirements & Basics

Requirements

Install requirements for debian

apt-get install lxc libvirt0 libpam-cgfs bridge-utils uidmap

Basics

Start Container

lxc-start -n %NAME%

Stop Container

lxc-stop -n %NAME%

Attach Container (get access to cli inside container)

lxc-attach -n %NAME%

List info/state of containers

lxc-ls --fancy

Create container (template downloaded for unprivileged container)

lxc-create -n %NAME% -t download -- -r bookworm

Destroy (delete) container - container have to be stopped before

lxc-destroy -n %NAME%

Creating unprivileged containers

When creating unprivileged containers as root with shared UID and GID the files /etc/subuid and /etc/subgid need some entries. Check content of both files before appending stuff to it! Furthermore the default lcx configuration file /etc/lxc/default.conf has to be extened.

echo "root:100000:65536" >>/etc/subuid
echo "root:100000:65536" >>/etc/subgid
echo "lxc.idmap = u 0 100000 65536" >>/etc/lxc/default.conf
echo "lxc.idmap = g 0 100000 65536" >>/etc/lxc/default.conf

Autostart for containers

Add the following line to configuration file (/var/lib/lxc/%%NAME%%/config) of each container, which should be start automatically.

lxc.start.auto = 1

Basic default configuration

xc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx

lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536

Jekyll - Install & Basics

#Install requirements with apt-get
apt-get install ruby ruby-dev build-essential

Do not install gems as root user

Login as user and execute following commands:

echo '# Install Ruby Gems to ~/gems' >> ~/.bashrc
echo 'export GEM_HOME=$HOME/gems' >> ~/.bashrc
echo 'export PATH=$HOME/gems/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

Install Jekyll and dependencies:

gem install jekyll bundler

#update jekyll or any other gem
gem update jekyll

#update Rubygems
gem update --system

Working with Jekyll

#start a new website, named 'the_blog'
jekyll new the_blog

#build new created website
jekyll build
# => The current folder will be generated into ./_site

jekyll build --destination <destination>
# => The current folder will be generated into <destination>

jekyll build --source <source> --destination <destination>
# => The <source> folder will be generated into <destination>

jekyll build --watch
# => The current folder will be generated into ./_site,
#    watched for changes, and regenerated automatically.

 

nftables - Firewall

Basics

Using nftables as the successor to iptables makes life much more convenient. So I switched to nftables at my projects.

For debian (bookworm) it's necessary to enable the nftables service:

systemctl enable nftables.service

Default configuration file for nftables is located at: /etc/nftables.conf. I don't want to make big modifications at preinstalled files, so I just add an include statement for customized rules:

include "/etc/custom/nftables.rules" 

Here are some basic commands for nftables

List loaded ruleset

nft list ruleset

List sets 
The following command lists the set WHITELIST of table CUSTOM

nft list set inet CUSTOM WHITELIST

Basic ruleset for a server providing http/https services

#!/usr/sbin/nft -f

table inet CUSTOM {
    # empty set WHITELIST - will be filled dynamically by scripts
	set WHITELIST {
		type ipv4_addr
	}
	chain INPUT {
		type filter hook input priority 0; policy drop;

		# allow established/related connections
		ct state {established, related} accept

		# early drop of invalid connections
		ct state invalid drop

		# allow from loopback
		iifname lo accept

		# allow icmp (ipv4) - only from IP set WHITELIST
		ip protocol icmp ip saddr @WHITELIST accept

        # allow only ssh connections from IP set WHITELIST
		tcp dport {22} ip saddr @WHITELIST accept

		# allow web services
		tcp dport {80, 443} accept

		# allow internal container interface
		iifname "lxcbr0" ip daddr 10.0.3.1 accept

		# everything else drop
		drop
	}

	chain OUTPUT {
		type filter hook output priority 0; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority 0; policy drop;
		oifname "lxcbr0" accept
        iifname "lxcbr0" accept
	}
}

Updating nftables set

The following bash script updates the set WHITELIST with a dynamic IP address. The script should be executed periodically by a cron  job.

The variables 

• ips
• host

have to be modified.

#!/bin/bash
# Update ipset to let my dynamic IP in
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# nftables table and set
ipset=WHITELIST
table=CUSTOM

# dynamical IP for resolving via DNS
host=myurl.example.com

#IP for whitelisting
ips="127.0.0.1 127.0.0.2"

# hostname for logging
me=$(basename "$0")

ip=$(dig +short $host | tail -1)
if [ -z "${ip}" ]; then
    logger -t "${me}" "IP for ${host} not found"
    exit 1
fi

ips="${ips} ${ip}"

ret=`nft get element inet ${table} ${ipset} { ${ip} } > /dev/null 2>&1; echo $?`

if [[ ${ret} == "1" ]] then 
	nft flush set inet ${table} ${ipset}
	logger -t "${me}"  "Adding IP ${ip} to set ${ipset}."
	for i in ${ips}
	do
		logger -t "${me}"  "Adding IP ${i} to set ${ipset}."
		nft add element inet ${table} ${ipset} { ${i} }
	done
fi

exit 0

Debian - Tasks for new installs

Install & Configuration of common packages including a fancy bash

wget -O - https://wuideweidewelt.de/vim/mydebian.sh | bash -s

mdadm - Software Raid

Summary of commands dealing with software raid at linux machines

 

Create an array

mdadm --create /dev/md/md_name --level=<RAID-Level> --raid-devices=<amount of physical partitions at array> /dev/sdX1 /dev/sdY1

Check raid array

mdadm --detail /dev/md/<LABEL>

# or

cat /proc/mdstat

Remove raid array

umount /dev/md/md_name
mdadm --stop /dev/md/md_name
mdadm --zero-superblock /dev/sdX1
mdadm --zero-superblock /dev/sdY1

Restart sync of array

mdadm --readwrite /dev/md/md_name

Update raid config

/usr/share/mdadm/mkconfig > /etc/mdadm/mdamd.conf

When device is needed for boot

update-initramfs -u
update-grub2
grub-install /dev/sdX

 

LUKS - Encryption

Summary of basic commands

Create encrypted partition

cryptsetup -c aes-xts-plain64 -s 512 -h sha512 luksFormat /dev/sdX2

Open encrypted partition as crypted_sdX2

cryptsetup luksOpen /dev/sdX2 crypted_sdX2

Add encryption key

cryptsetup luksAddKey /dev/sdX2

Remove encryption key

Attention !!!
When you don’t have antoher key, you aren’t able to decrypt the partition afterwards

cryptsetup luksRemoveKey /dev/sdX2

Caddy webserver - valid certificates for local services

1. Setup a debian machine or use an existing machine
   
   
2. Install xcaddy  (add xcaddy apt repository)
   
   

   apt install -y debian-keyring debian-archive-keyring apt-transport-https
   curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-xcaddy-archive-keyring.gpg
   curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-xcaddy.list
   sudo apt update
   sudo apt install xcaddy

3. Install prebuild go language from go website
   
   

   mkdir golang
   cd golang
   
   #remove existing go version
   rm -rf /usr/local/go
   
   # check for latest release and download
   wget https://go.dev/dl/go1.22.2.linux-amd64.tar.gz
   tar -xvf go1.22.2.linux-amd64.tar.gz -C /usr/local
   
   # set environment variables for go / e.g. edit bash.bashrc
   export GOROOT=/usr/local/go
   export GOPATH=$HOME/go
   export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
   

   
   
4. Build your own caddy binary with needed plugins - in this case include dns-netcup module
   
   

   xcaddy build --with github.com/caddy-dns/netcup
   
   # move binary to /usr/bin
   # rename it to caddy_custom to mark it as custom build
   
   mv caddy /usr/bin/caddy_custom
   
   # create default config Directory fpr Caddyfile
   mkdir /etc/caddy
   touch /etc/caddy/Caddyfile

   
   

   
   
5. Create systemd service file and enable it
   
   

   # create caddy group
   groupadd --system caddy
   
   #create caddy user
   useradd --system \
       --gid caddy \
       --create-home \
       --home-dir /var/lib/caddy \
       --shell /usr/sbin/nologin \
       --comment "Caddy web server" \
       caddy
   
   #create default working folder
   mkdir /var/lib/caddy
   chown -R caddy:caddy /var/lib/caddy
   
   # create empty service file
   touch /lib/systemd/system/caddy.service

   This example service file includes env vars for the netcup dns module
   
   

   # caddy.service
   #
   # For using Caddy with a config file.
   #
   # Make sure the ExecStart and ExecReload commands are correct
   # for your installation.
   #
   # See https://caddyserver.com/docs/install for instructions.
   #
   # WARNING: This service does not use the --resume flag, so if you
   # use the API to make changes, they will be overwritten by the
   # Caddyfile next time the service is restarted. If you intend to
   # use Caddy's API to configure it, add the --resume flag to the
   # `caddy run` command or use the caddy-api.service file instead.
   
   [Unit]
   Description=Caddy
   Documentation=https://caddyserver.com/docs/
   After=network.target network-online.target
   Requires=network-online.target
   
   [Service]
   Environment="NETCUP_CUSTOMER_NUMBER=XXXXX"
   Environment="NETCUP_API_KEY=XXXXX"
   Environment="NETCUP_API_PASSWORD=XXXXX"
   Type=notify
   User=caddy
   Group=caddy
   ExecStart=/usr/bin/caddy_custom run --environ --config /etc/caddy/Caddyfile
   ExecReload=/usr/bin/caddy_custom reload --config /etc/caddy/Caddyfile --force
   TimeoutStopSec=5s
   LimitNOFILE=1048576
   LimitNPROC=512
   PrivateTmp=true
   ProtectSystem=full
   AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
   
   [Install]
   WantedBy=multi-user.target
   

   Enable caddy as service
   
   

   systemctl daemon-reload
   systemctl enable caddy.service
   systemctl start caddy.service

   
   
6. Sample Caddyfile
   
   

   # The Caddyfile is an easy way to configure your Caddy web server.
   #
   # Unless the file starts with a global options block, the first
   # uncommented line is always the address of your site.
   #
   # To use your own domain name (with automatic HTTPS), first make
   # sure your domain's A/AAAA DNS records are properly pointed to
   # this machine's public IP, then replace ":80" below with your
   # domain name.
   
   #global options
   {
           # E-Mail for ACME
           email info@mydomain.de
   }
   
   # caddy directive for import
   (insecure) {
           transport http {
                   tls_insecure_skip_verify
           }
   }
   
   # reverse proxy for websockets
   wss01.mydomain.de:8884 {
           @websockets {
                   header Connection *Upgrade*
                   header Upgrade websocket
           }
           tls {
                   dns netcup {
                           customer_number {env.NETCUP_CUSTOMER_NUMBER}
                           api_key {env.NETCUP_API_KEY}
                           api_password {env.NETCUP_API_PASSWORD}
                   }
                   propagation_timeout 900s
                   propagation_delay 600s
                   resolvers 1.1.1.1
           }
           reverse_proxy @websockets 192.168.2.91:8884 {
                   import insecure
           }
   }
   
   # wildcard domain/certificate for internal services
   *.mydomain.de {
           tls {
                   dns netcup {
                           customer_number {env.NETCUP_CUSTOMER_NUMBER}
                           api_key {env.NETCUP_API_KEY}
                           api_password {env.NETCUP_API_PASSWORD}
                   }
                   # netcup dns is slow
                   propagation_timeout 900s
                   propagation_delay 600s
                   resolvers 1.1.1.1
           }
   
           @ilo01 host ilo01.mydomain.de
           handle @ilo01 {
                   reverse_proxy 192.168.2.13 {
                           import insecure
                   }
           }
   
           @pve01 host pve01.mydomain.de
           handle @pve01 {
                   reverse_proxy 192.168.2.10:8006 {
                           import insecure
                   }
           }
   
           @pbs01 host pbs01.mydomain.de
           handle @pbs01 {
                   reverse_proxy 192.168.2.15:8007 {
                           import insecure
                   }
           }
   
           @home host home.mydomain.de
           handle @home {
                   reverse_proxy https://192.168.2.94 {
                           import insecure
                   }
           }
   
           handle {
                   abort
           }
   
   }

Fail2Ban

1. Unban IP

List banned IP via iptables

iptables -L -n

Get configured jails

fail2ban-client status

Unban IP

fail2ban-client set JAIL-NAME unbanip XXX.XXX.XXX.XXX

2. Configuration for caddy reverse proxy

Enable access logging at caddy

(custom_log) {
	log {
		format json {
			time_format iso8601
		}
		output file /var/log/caddy/{args[0]}.access.log {
			roll_size 10mb
			roll_keep 20
			roll_keep_for 720h
		}
	}
}

mydomain.com {
	import custom_log mydomain.com
    reverse_proxy XXX.XXX.XXX.XXX
}

Create caddy filter file (regex) for fail2ban - /etc/fail2ban/filter.d/caddy-status.conf

[Definition]
failregex = ^.*"remote_ip":"<HOST>",.*?"status":(?:401|403|500),.*$
ignoreregex =
datepattern = "ts":"%%Y-%%m-%%dT%%H:%%M:%%S.

Enable jail in fail2ban config (/etc/fail2ban/jail.local)

[caddy-status]
backend		= auto
enabled     = true
port        = http,https
filter      = caddy-status
logpath     = /var/log/caddy/*.access.log
maxretry    = 10

Docker @debian

Uninstall old (pre-installed) versions

for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do apt-get remove $pkg; done

Set up Docker's apt repository

# Add Docker's official GPG key:
apt-get update
apt-get install ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update

Install the Docker packages

apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

LVM2

Create/add physical disk to LVM

pvcreate /dev/sdb

Create volume group

vgcreate vg00 /dev/sdb1 /dev/sdc1

Create volume

# create by given size in GB
lvcreate -n data -L1G vg00
# create by given size of vg in %
lvcreate -n data -l100%VG vg00
Angabe in Prozent des freien Speichers in der VG:
# create by given free size of vg in %
lvcreate -n data -l100%FREE vg00

 

 

 

iSCSI

Search for iSCSI targets

iscsiadm --mode discovery --portal XXX.XXX.XXX.XXX --type sendtargets

Login

iscsiadm -m node -T iqn.2024-11.local.... -p XXX.XXX.XXX.XXX:3260 --login

Check iSCSI sessions

iscsiadm -m session

Logout of iSCSI session

iscsiadm -m node -T iqn.2024-05.local.... -p XXX.XXX.XXX.XXX:3260 -u

Remove iSCSI target

iscsiadm -m node -o delete -T iqn.2024-05.local......

 

Proxmox BackupServer - pbs

Create datastore at sshfs mount

1. mount path via sshfs e.g. /mnt/backup
2. use bindfs to ignore missing chown and chgrp for sshfs mount 
   
   

   bindfs --chown-ignore --chgrp-ignore /mnt/backup /mnt/backup-bindfs

   
   
3. Create datastore via pbs UI with path /mnt/backup-bindfs
4. Modify /etc/proxmox-backup/datastore.cfg and replace /mnt/backup-bindfs with /mnt/backup