EN:Linux Kernel Optimization

From FrâG^.WIKI

Jump to: navigation, search

(Deutsche Version)

This article is related to game servers.


Optimizing the Linux Kernel is very important for a good performance of the game server. The Kernel is responsible for distributing the available CPU time among the running processes, and thus has directly an impact on the server's FPS. This article tries to be independent of the Linux distribution, so you need to know how to use your distribution. Whether you want to run hl1- or hl2-based games, it is recommended to use the RT-linux patches, as described below. It is recommended to use a 64 bit system, even though the servers itself run in 32 bit mode! Also it is recommended not to use an SELinux or AppArmor based installation (if you don't know what it is, you probably don't have it).

Note that this HOWTO has been used and tested mostly on Intel Core 2 Duo machines. The general technique has been tested on an AMD system as well, but there might be some adjustments to be made. Please contribute any experiences you made!


Contents

Preparing your system

Depending on your distribution you will need to install some tools for compiling the kernel. On most distributions, installing the following packages/programs should be sufficient (the actual package names might be slightly different):

  • gcc (this is the compiler)
  • make, gmake or gnu-make (the gnu-version of the utility make)
  • ncurses and ncurses-dev (the develop package for the ncurses library, called ncurses-devel on CentOS, RHEL and Fedora, libncurses5-dev on Debian and Ubuntu)
  • chrt (package might be called schedutils, or schedtool)
  • zlib1g-dev (need for Debian and Ubuntu)
  • patch (needs to be manually installed for some Debian users)
  • vixie-cron or some other cron demon
  • rpm-build (CentOS and Fedora only)
  • kernel-package (Debian and Ubuntu only)

All following operations need to be run as root on your Linux server. It is common practice to have all your Kernels located in /usr/src, so go into that directory: 

cd /usr/src

Note: Don't use sudo to run the commands as root. Instead run su - to log in as root (it will ask you for the root password, sudo passwd sets you the root password if you don't know it and used sudo before).

Downloading the Kernel and the patch

Go to http://kernel.org/pub/linux/kernel/projects/rt/ and pick the most recent version of the RT patches. Then download them directly to your server: 

wget http://kernel.org/pub/linux/kernel/projects/rt/patch-2.6.26.8-rt16.gz

Then go to http://kernel.org/pub/linux/kernel/v2.6/ and download the latest kernel supported by the RT patches (You need to use the same version as in the patch name! The RT patches are not always available for the most recent kernel version): 

wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.26.8.tar.gz

Now unpack the Kernel and apply the patch: 

tar zxf linux-2.6.26.8.tar.gz
cd linux-2.6.26.8
zcat ../patch-2.6.26.8-rt16.gz | patch -p1

Configuring and building the Kernel

First copy the current kernel config (is not necessary on most systems but does not hurt): 

cp -vi /boot/config-`uname -r` .config

For launching the configuration menu type: 

make menuconfig

In most circumstances (actually depending on your distribution) the configuration of the running kernel will be used for the default configuration. But to optimize the kernel for the needs of the game server, you need to change certain settings. To navigate through the menu use the up/down keys and press Enter to open a sub-menu. The state of a setting can be changed by pressing space ([*] means enabled, [M] means enabled as kernel module and [ ] means disabled). To leave a sub-menu select Exit with the left/right keys and press Enter. If you want to get more information on some option, select Help in the same way.

The important settings are listed below:

  • Processor type and features:
    • Disable Tickless System (Dynamic Ticks)
    • Enable High Resolution Timer Support
    • Select your processor under Processor family
    • Change Preemtion Mode to Complete Preemption (Real-Time)
    • Enable Enable priority boosting of RCU read-side critical sections (ignore if not present)
    • Disable Enable tracing for RCU - currently stats in debugfs (ignore if not present)
    • Enable Machine Check Exception and select Intel or AMD depending on your CPU
    • Change Timer frequency to 1000 HZ
  • General setup:
    • RCU Subsystem (ignore if not present)
      • Enable RCU Implementation (Preemptible RCU)
      • Disable Enable tracing for RCU
  • Power management and ACPI options
    • Enable Power Management support
    • Disable Power Management Debug Support
    • Disable Suspend to RAM and standby
    • Disable Hibernation (aka 'suspend to disk')
    • Enable ACPI (Advanced Configuration and Power Interface) Support
    • Disable CPU Frequency scaling
    • Disable CPU idle PM support
  • Networking support
    • Networking options
      • Enable Packet socket: mmapped IO
      • Optionally disable Network packet filtering framework (Netfilter) (Warning: this will disable your firewall!)
      • Disable QoS and/or fair queueing (Unless you need and use it...)
  • Device Drivers
    • Disable Watchdog Timer Support
    • Enable Real Time Clock
      • Enable PC-style 'CMOS'
  • Kernel hacking
    • Disable everything

Now exit the main menu and confirm that your changes should be saved. To compile and install the kernel run the following command (get a coffee, this takes some time...): 

make && make install modules_install

If you run into problems, running this before building the kernel can help: 

make clean

Only if you rely on an initial ram disk (i.e. you did not compile all required drivers for the boot process - usually hard disk and file systems - into the kernel - [*] instead of [M]) you will need distribution specific commands to build the initial ram disk:

For CentOS and Fedora you need to run: 

make rpm && rpm -i /usr/src/redhat/RPMS/`uname -i`/kernel-2.6.26.8rt16-1.`uname -i`.rpm

For Debian and Ubuntu you need to run consecutively: 

make-kpkg --initrd kernel_image kernel_headers && cd ..
dpkg -i linux-image-2.6.26.8-rt16_2.6.26.8-rt16-10.00.Custom_`dpkg --print-architecture`.deb
dpkg -i linux-headers-2.6.26.8-rt16_2.6.26.8-rt16-10.00.Custom_`dpkg --print-architecture`.deb

For Gentoo you need to use genkernel (emerge it first). First make sure, your /usr/src/linux symlink points to the correct kernel source directory. Then run (after compiling the kernel as described above): 

genkernel initramfs

The inititial ram disk will be placed in /boot.

Preparing for boot

Note: if you compiled kernel using make-kpkg on Debian or Ubuntu, skip this step to Reboot your system

After this step you probably need to setup a so-called initramdisk. This is required for the boot process, if kernel modules need to be loaded before disk access is available in the kernel. The procedure to create this initramdisk differs among distributions. On debian the command mkinitramfs is used (try man mkinitramfs): 

mkinitramfs -o /boot/initrd.img-2.6.26.8-rt16 2.6.26.8-rt16

On CentOS and Fedora: 

mkinitrd /boot/initrd-2.6.26.8-rt16.img 2.6.26.8-rt16

The final step before reboot will be to tell your boot loader to use the new kernel. If your distribution uses grub (which is very likely) you need to edit the file /boot/grub/menu.lst (or /boot/grub/grub.conf) so it looks like this (do not copy+paste this, it has to be changed according to your Linux installation!): 

default         0
timeout         1

title           Debian GNU/Linux, RT Kernel
root            (hd0,1)
kernel          /boot/vmlinuz-2.6.26.8-rt16 root=/dev/ram0 real_root=/dev/sda1
initrd          /boot/initrd.img-2.6.26.8-rt16

title           Debian GNU/Linux, kernel 2.6.24.2
root            (hd0,1)
kernel          /boot/vmlinuz-2.6.24.2 root=/dev/ram0 real_root=/dev/sda1
initrd          /boot/initrd.img-2.6.24.2

Note: /dev/sda1 needs to be the device containing the root file system. (hd0,1) also needs to match the root partition (hd0 = sda and hd0,1 = sda1 etc.). Just have a close look what is already there. Copy one existing block and only change the kernel and the initrd, but keep all the other options unchanged.

Note: On Ubuntu you can run update-grub to auto detect the kernel and load it it to grub (you will however edit the grub to set the RT kernel as default).

Reboot your system 

shutdown -r now

After reboot, verify that you are running the new kernel: 

uname -r

should output something like: 

2.6.26.8-rt16

Setting your servers to run with realtime scheduling

A real time operation system needs to know which processes are time critical and which not. By default every process is not considered to be time critical. The easiest way is to create a small cron job and let it run every 5 minutes or so. Put the following e.g. into /usr/local/sbin/resched.sh: 

#!/bin/sh

PIDS=`ps ax | grep sirq-hrtimer | grep -v grep | sed -e "s/^ *//" -e "s/ .*$//"`
for p in $PIDS; do
  chrt -f -p 99 $p
done

PIDS=`ps ax | grep sirq-timer | grep -v grep | sed -e "s/^ *//" -e "s/ .*$//"`
for p in $PIDS; do
 chrt -f -p 51 $p
done

PIDS=`pidof srcds_i686`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof srcds_i486`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof srcds_amd`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof hlds_i686`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof hlds_i486`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof hlds_amd`
for p in $PIDS; do
  chrt -f -p 98 $p
done

Note: sirq-hrtimer seems to depend on the distribution or exact kernel version, I have also seen softirq-hrtimer & ksoftirqd. Try running: 

ps ax | grep hrtimer

There should be something like [softirq-hrtimer/N] for each CPU (N is a number). Strip the brackets and the /N and use what is left...

make the file executable: 

chmod 755 /usr/local/sbin/resched.sh

and put it into /etc/crontab: 

# m h dom mon dow user  command
[...]
*/5 * * * *  root  /usr/local/sbin/resched.sh > /dev/null 2>&1

Do not forget to restart your cron demon, e.g. on debian: 

/etc/init.d/cron restart

Alternatively you can use the command crontab -e (make sure to run it as root) instead of editing /etc/crontab i.e.: 

*/5 * * * * /usr/local/sbin/resched.sh > /dev/null 2>&1

A restart of the demon is not necessary then.

That's it!

Run your game server, wait 5 minutes (or run /usr/local/sbin/resched.sh by hand) and have a look at the servers fps (using the stats command). It should be very stable slightly below 1000.0. For a detailed test use the FPS-Meter ;-)

For hlds servers you should experiment a little with the command line settings. Try a sys_ticrate larger than 1000 and a pingboost of 2 or 3, e.g. 

./hlds_run -game cstrike +map de_dust2 +sys_ticrate 1500 -pingboost 3

Beware that sys_ticrates larger than 2500 in combination with pingboost 3 might lead to an accelerated gameplay (and thus make the server unusable for clan wars etc.)

Eliminate small Variations with "idler"

On some installations small variations in FPS remain after following this wiki. In this case it can help to have a low-priority process running that takes 100% cpu. Create a file called idler.c containing the following lines: 

int main() {
  while(1);
}

Then compile this program with the following command: 

gcc idler.c -o idler

You should run the program at low priority to aviod slowing down your system: 

nice ./idler

Running only one idler even on systems with 2 or more cores should be enough (also if you run multiple game servers).

The variations should go away as log as the idler is running. See the following example, the idler started after ~10 minutes:

Image:idler_demo.png

Playing around

There are several options you can play with to improve your system even further. A small and incomplete list:

  • Try set the Timer frequency to 100 HZ and enable Tickless System (Dynamic Ticks). With high resolution timers the scheduling does not depend on the kernel HZ anymore, so you can reduce the interrupt load. This has not been tested very well and might lead to worse results.
  • Try setting other sirq kernel threads to realtime priority (e.g. sirq-net-tx and sirq-net-rx).
  • Try different clock sources (start with hpet on AMD and tsc on Intel processors): 
cat /sys/devices/system/clocksource/clocksource0/available_clocksource
gives you all available clock sources, while 
echo hpet >  /sys/devices/system/clocksource/clocksource0/current_clocksource
sets the clock source to hpet.
  • On some systems (especially on quad core like Q6600), running one idler per core might be necessary, like: 
nice -n +19 taskset -c 0 screen -AmdS idler1 ./idler
nice -n +19 taskset -c 1 screen -AmdS idler2 ./idler
nice -n +19 taskset -c 2 screen -AmdS idler3 ./idler
nice -n +19 taskset -c 3 screen -AmdS idler4 ./idler
I recommend testing both running a single idler and one idler per core, as there is no way to tell what is better without testing!
  • Also usually on quad cores it can be better to run each server on a different core, like: 
taskset -c 0 screen -AmdS cs-server ./hlds_run put your options here
taskset -c 1 screen -AmdS cs-server ./hlds_run put your options here
taskset -c 2 screen -AmdS cs-server ./hlds_run put your options here
Again I recommend testing both running without and with forcing the servers to specific cores!

All these options has not been tested very much, or did not seem to have any positive effect. Please give feedback if you experience a positive behaviour by one of those options!

Here is a script to start and stop the idlers all at once: 

#!/bin/bash
#clear
# Change and edit to your needs

case "$1" in
 start)
  screen -d -m -S idler1 nice -n +19 ./idler
  screen -d -m -S idler2 nice -n +19 ./idler
  screen -d -m -S idler3 nice -n +19 ./idler
  screen -d -m -S idler4 nice -n +19 ./idler
  screen -d -m -S idler5 nice -n +19 ./idler
  screen -d -m -S idler6 nice -n +19 ./idler
  screen -d -m -S idler7 nice -n +19 ./idler
  screen -d -m -S idler8 nice -n +19 ./idler
   ;;

 stop)
  screen -r idler1 -X quit
  screen -r idler2 -X quit
  screen -r idler3 -X quit
  screen -r idler4 -X quit
  screen -r idler5 -X quit
  screen -r idler6 -X quit
  screen -r idler7 -X quit
  screen -r idler8 -X quit
   ;;

 *)
   echo "Usage: $0 {start|stop}"
   exit 1
   ;;
esac

exit 0
  • If you want to reach more than 1000 fps, try this: BEpingboost.c

Recent development

  • 2.6.26.8-rt16 still seems to be best, at least for CS:S
  • 2.6.31.4-rt14 / 2.6.31.6-rt19 is good as well, but does not completely reach the performance of 2.6.26.8-rt16
  • 2.6.32-ck2 (configured with closest settings where configuration differs from rt-kernels) seems to work as good as 2.6.26.8-rt16 (beware: hlds with pingboost 3 will accelerate much in case of high ticrates. use sys_ticrate 1050)
  • 2.6.32.3 (vanilla - no patches, also with closest settings) has also been reported to work good.

Trouble Shooting / FAQ

  • For srcds: check if you have fps_max 0 in your config. This is really mandatory.
  • For hlds: check if you have sys_ticrate > 1000 (e.g. 2500) in your config. Also check if you are running with -pingboost 2 or 3
  • Check your net settings on both server and client
  • Check if your servers and sirq-hrtimer processes really have the right scheduling by running chrt -p <pid>
  • Never use bots for testing. They are not a real test, you can have very different results in both directions (they do not use the net engine but increase cpu load)!
  • Reaching only ~950 FPS instead of 1000.0 is not an issue. The difference of 50 microseconds (0.05 milliseconds!) can nobody possibly notice.
  • All Kernels until 2.6.32-rc5 have a bug that allows users root right through a nullpointer attack. A small workaround (not bugfix!) to make such an attack impossible: 
 echo "vm.mmap_min_addr = 4096" > /etc/sysctl.d/mmap_min_addr.conf && /etc/init.d/procps restart

Links

Retrieved from "http://wiki.fragaholics.de/index.php/EN:Linux_Kernel_Optimization"
Personal tools