Dell R710 Fan Control Script

A temperature-based fan speed controller for Dell servers (tested on an R710, should work with most PowerEdges). Supports both local and remote hosts.

• Requisites
• Installation / Upgrade
  • Docker
• Configuration
• How it works
• Notes on remote hosts
• Credits

Requisites

1. Python 3 is installed.
2. IPMI Over LAN is enabled in all used iDRACs (Login > Network/Security > IPMI Settings).
   • May not be needed if you're only managing the local machine.
3. lm-sensors is installed and configured on the local machine.
   • Example output of sensors for a dual CPU system:

     coretemp-isa-0000
     Adapter: ISA adapter
     Core 0:       +38.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 1:       +46.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 2:       +40.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 8:       +43.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 9:       +39.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 10:      +39.0°C  (high = +69.0°C, crit = +79.0°C)
     
     coretemp-isa-0001
     Adapter: ISA adapter
     Core 0:       +29.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 1:       +35.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 2:       +29.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 8:       +34.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 9:       +33.0°C  (high = +69.0°C, crit = +79.0°C)
     Core 10:      +31.0°C  (high = +69.0°C, crit = +79.0°C)
     

Installation / Upgrade

Clone the repo and run the installation script as root to configure the system or upgrade the already installed controller:

git clone https://github.com/nmaggioni/r710-fan-controller.git
cd r710-fan-controller
sudo ./install.sh [<installation path>]

The default installation path is /opt/fan_control and the service will be installed as fan-control.service. If a configuration file already exists, it will be renamed with a .old extension.

Docker

To deploy remote fan management with Docker (fan_control running on a separate host and only interacting with remote ones, see Notes on remote hosts), build the image in the repo and bind mount your own YAML config and SSH keys folder:

docker build -t fan_control .
docker run -d --restart=always --name fan_control -v "./fan_control.yaml:/app/fan_control.yaml:ro" -v "./keys:/app/keys:ro" fan_control

Running this tool under a proper orchestrator is advised.

Configuration

You can tune the controller's settings via the fan_control.yaml file in the installation directory.

The file is made of two main sections, general and hosts. The first one contains global options; the second one, hosts, is a list of hosts to manage. Each of them must contain a temperatures and a speeds lists at a minimum, both of exactly three values. If the hysteresis key isn't specified, its value is assumed to be 0.

Remote hosts must also contain both the remote_temperature_command string and the remote_ipmi_credentials structure.

How it works

Every general.interval seconds the controller will fetch the temperatures of all the available CPU cores, average them and round the result (referred to as Tavg below). It will then follow this logic to set the fans' speed percentage or engage automatic (hardware managed) control.

If hysteresis is set for a given host, the controller will wait for the temperature to go below ThresholdN - hysteresis temperature. For example: with a Threshold2 of 37°C and an hysteresis of 3°C, the fans won't slow down from Threshold3 to Threshold2 speed until the temperature reaches 34°C.

Notes on remote hosts

This controller can monitor the temperature and change the fan speed of remote hosts too: the only caveat is that you'll need to extract the temperatures via an external command. This could be via SSH, for example. The controller expects such a command to return a newline-delimited list of numbers parseable as floats.

The included example is a good fit for a remote FreeNAS host: it will connect to it via SSH and extract the temperature of all CPU cores, one per line. This way you'll be able to manage that machine just as well as the local one without applying any hardly trackable modification to the base OS.

Credits

Major thanks go to NoLooseEnds's directions for the core commands and sulaweyo's ruby script for the idea of automating them.

Note: The key difference of this script, other than handling remote hosts, is that it's based on the temperature of the CPUs' cores and not on the ambient temperature sensor on the server's motherboard. The R710 does not expose CPU temperature over IPMI, but other models do; this script should work with them nonetheless.