Zigbee2MQTT – An easy guide on how to have all your Zigbee devices under your full control

Zigbee smart home devices are everywhere today. Zigbee2MQTT is a software that allows you to control them all.

Many different brands are producing Zigbee devices today so you might very easily end up with a bunch of different gateways and Apps to control them. And nothing working without an Internet connection. You don’t like that? Me neither! Here is how you can control most smart light bulbs (and lots of other smart home devices) with one cheap gateway and the Zigbee2MQTT software. Totally under your control and independent of the manufacturer and the control software. Let’s go!

Update: You can find a newer version of this Zigbee2MQTT guide geared towards using Docker and Home Assistant here.

Why Zigbee?

As you might have guessed this exact guide will only work for smart devices using the Zigbee protocol. That is actually good though! Zigbee is a great network protocol designed for smart home and IoT devices. It is made for low power consumption (battery powered devices!) and automatically builds a network between all devices (wall powered devices become routers) without the user, us, needing to do anything.

You might not have noticed but most big brands like Philips Hue, Ikea Tradfri and Xiaomi Aqara/Mijia are already using Zigbee. They just come with their own gateway which bridges Zigbee and your Wifi. That works great if you buy devices from just one manufacturer and don’t mind sharing your data with them as well as relying on an Internet connection in most cases. So let’s fix this!

The hardware

Before we can start we need our own gateway. Something that is cheap, easy to buy and reliable. Oh, and can talk to Zigbee devices. The hardware basis can be pretty much anything, an old PC, a VM or something else. I does not even need to be really powerful. I would recommend getting a Rasperry Pi (a newer model 3 or 4).

For those who don’t know the Rasperry Pi: it is a small and cheap single board computer. It uses way less energy and space than a full blown computer but is still powerful enough to run all the smart home software you need. You need

• A Raspberry Pi, preferably the Pi 4 with 2 or 4 GB of RAM
• A USB (C for the Pi 4, Micro USB for the older ones) power supply with around 3.0A
• A microSD card, 16+GB from a reliable manufacturer, should be fast: class 10/UHS-1
• Optional: a case (with cooling) for the Pi
• Optional: if you don’t have one yet buy a SD card reader/writer
• Optional: Network access. Either a LAN cable or if you are not using a newer Pi a seperate Wifi USB dongle.
• Alternatively you can get a kit including everything you need
  
  If you are interested in buying a Raspberry Pi consider doing it via my affiliate links above. It does not change prices for you and allows me to pay for the servers 😉

Once you have all of this you need to install an operating system of your choice. This will depend on the smart home controller software you are going to use. If you are going to use OpenHab I can strongly recommend going with OpenHabian. The setup is super easy. If you are not planning to use a certain system or are unsure you can go with the Raspbian system. It will run all the software we need for this guide and can also install pretty much anything else. There is a guide on how to run it here. Just make sure to enable SSH access for usage without a monitor.

If you are looking for a guide for HomeAssistant or want to run your setup via Docker you can read my guide on how to set up your custom smart home server. It includes a hardware description as well as an example docker-compose setup to get you started quickly.

Now that we have a working computer we need to add support for Zigbee. Fortunately there are several easy ways to do this. There is the Conbee (II) USB stick. I have never used it myself but it comes with its own software and supposedly works quite well. I looks like the software is not open source though and the USB stick is a little more expensive. That is why I tried another solution: the CC2531 USB stick.

Update 2022: There is now an even better hardware controller: the Sonoff Zigbee 3 Dongle plus uses the more modern Zigbee version, is quite cheap and directly supports the custom firmware. I described how to migrate without having to re-pair devices here.

If you are interested in buying the CC2531 Zigbee USB stick (preflashed or with a debugger) or the Conbee II consider doing it via my affiliate links. Even more than that I can recommend the Sonoff Zigbee 3.0 dongle plus (and a USB extension cable to avoid interference). It does not change prices for you and allows me to pay for the servers 😉

The CC2531 is a Zigbee sniffer that can be used to interact with Zigbee devices. To use it for our purposes we need to flash a custom firmware, a process we would need additional hardware for. Fortunately this has become pretty popular, so people are selling updated CC2531’s online. I recommend just buying one of those. Just plug it into one of the Rasperry Pi’s USB ports. If you want to do it yourself, here is a good guide. If you are having trouble with the range, or store the Pi somewhere enclosed I would recommend to use a USB extension cable for the CC2531.

That’s it, the hardware is ready.

Why MQTT?

Ok, let’s pause here for a second. We do now have our own gateway that is able to talk to Zigbee devices. Next we need some way to communicate states and commands from and to these devices from our smart home controller software. Of course we could use some software plugin that directly communicates with the Zigbee USB stick but there is a more elegant and decoupled way.

There is a software called Zigbee2MQTT which takes these messages from the Zigbee network and translates them into easy to use and well structured messages in another protocol: MQTT.

But first let me quickly explain what MQTT is. MQTT means Message Queuing Telemetry Transport and is a lightweight publish-subscribe system designed for IoT applications. Huh? MQTT is a messaging system designed for low power devices with less than optimal connections in between and can “push” messages to devices. It is also very flexible and quite widespread, making it a great choice for home automation purposes. It only needs one server (called broker) that usually runs on the machine you run you smart home controller on. It also allows for multiple users at the same time and presents data in a well structured tree topology.

All major open smart home controllers like OpenHab, Home Assistant and Domoticz support MQTT. That makes it easy to set up those devices and also future proof if you plan on switching at a later point. The Zigbee2MQTT software also supports hundreds of devices out of the box and comes with nice features like: white- and blacklisting, autodiscovery for Home Assistant and even experimental OTA support.

Zigbee2MQTT setup

Sounds great? Let’s install it. The always up-to-date manual can be found here. This is the overview. Alternatively you can also always use Docker.

// find the serial port the USB adapter is connected to. Check with
ls -l /dev/ttyACM0 // <- standard port

// you need to install Node.js 12 for this to work which is unfortunately not (yet) in the normal repos. So do it this way
sudo curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs git make g++ gcc
// check with
node --version  # Should output v12.X or v10.X
npm --version  # Should output 6.X

// now clone the actual software from Github
sudo git clone https://github.com/Koenkk/zigbee2mqtt.git /opt/zigbee2mqtt
sudo chown -R pi:pi /opt/zigbee2mqtt // replace pi with your user
// and install dependencies
cd /opt/zigbee2mqtt
npm ci

// installation should be done :) Now open the config file for Zigbee2MQTT
nano /opt/zigbee2mqtt/data/configuration.yaml

This configuration file is used to setup everything for the program. In depth info can be found here. I have the following base setup:

homeassistant: false
permit_join: true
mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://localhost'
  # user: my_user
  # password: my_password
serial:
  port: /dev/ttyACM0
advanced:
  network_key: GENERATE
devices:
  // ... see rest of tutorial

I am not using Home Assistant right now, so the discovery mechanism is disabled (PS I actually switched to Home Assistant now). To start you need to set the serial port (for the USB adapter) to the one your USB stick is connected to. Under MQTT.server you need to add the name or IP address of your MQTT server. In my case it is running on the same machine, so localhost will work just fine.

If you don’t have an MQTT broker (server) yet you can just install mosquitto on the same Raspberry Pi. That is super easy:

sudo apt update // get newest packages
sudo apt install mosquitto // install mosquitto

If you set up your broker with username/password support you can add those in the config. Otherwise just leave that part out. You should also set a base topic that all messages from this program will be grouped under. That way you can easily separate it from other devices communicating via MQTT (like my Tasmota devices).

The permit_join setting allows new devices to enter the network. That needs to be enabled whenever you try to add devices. This can definitely be a security vulnerability if you leave this on all the time like this. You can deactivate it here after you are done connecting your devices.

The “network_key: GENERATE” will force Zigbee2MQTT to generate a random key for your network on the next startup. That is optional but recommended (careful, if you add this later you will have to re-pair all your devices).

If you looking for tips & tricks concerning those settings and Zigbee2MQTT in general check out my article here.

The devices part will be used to rename and store all devices connected to the network. For now leave it empty. Now you can save the file and start the program via “npm start”. Get a program to view MQTT messages (I can recommend MQTT explorer) and check if you can see anything. There should be a “zigbee2mqtt/bridge” topic with state and config values.

If you are looking for a more user friendly way to control this, including a map of your Zigbee network, check out Zigbee2MQTT-Assistant.

Pairing a device

Now that everything is set up properly we can start by actually pairing a Zigbee device. Hundreds of devices are supported by Zigbee2MQTT but as an example I can recommend the cheap but awesome Xiaomi Aqara devices. I have several temperature/humidity and motion sensors running as part of my smart home system. I can also using a water leak sensor and some wireless switches from Aqara.

To pair one of those we need to first enable the “permit_join” setting in the config as described above (if you do not have that already). Both devices have a small button on the side. Just press that for 5 seconds until a small LED starts blinking. Now the sensor will try to pair with your gateway. If you have trouble pairing check the setting above again and move the sensor closer to the Zigbee USB stick (not too close though, give it at least half a meter).

To check if the pairing has worked check the Zigbee2MQTT logs.

tail -f /opt/zigbee2mqtt/data/log/TheDateYouStartedTheService/log.txt
// use the newest date; the log file is sometimes called log2, log3, ...

// if a log statement like this appeared pairing was successful
"Successfully interviewed '0x00158d0001dc126a', device has successfully been paired"

Now open the configuration.yaml file from earlier again. You should see a new entry in the “devices” part of the file. Change it like this

'0x00158d0002b7f7fe': // this is the id of the new device, don't change it
    friendly_name: MotionSensor1 // this will initially be the id too. Change it to a name you can recognize for easier use later on
    retain: false // I don't see why we should save messages, so disable this

Now just remember the friendly name you gave your new device and head over to your smart home controller to integrate it.

Smart home controller integration – OpenHab 2

If you are using Home Assistant you can enable auto discovery. More details on all of this here. Pretty much all other smart home controllers are also able to interact with MQTT though and can thus use your devices this way.

I am currently using OpenHab 2 (2.5 with the new MQTT binding), so lets see how to integrate those devices into it. As always the first step is to create a thing for your device.

Bridge mqtt:broker:mosquitto [host="localhost", port=1883, secure=false, clientID="openHAB2"] {
    Thing topic motionZigbee1 "Bewegung Küche" @ "Küche" {
        Channels:
            Type switch    :   motion              "Bewegung"      [ stateTopic="zigbee2mqtt/MotionSensor1", transformationPattern="JS:js/getZigbeeOccupancy2Switch.js"]
            Type number    :   illuminance         "Helligkeit"    [ stateTopic="zigbee2mqtt/MotionSensor1", transformationPattern="JSONPATH:$.illuminance"]
            Type number    :   battery             "Batterie"      [ stateTopic="zigbee2mqtt/MotionSensor1", transformationPattern="JSONPATH:$.battery"]
            Type number    :   voltage             "Spannung"      [ stateTopic="zigbee2mqtt/MotionSensor1", transformationPattern="JSONPATH:$.voltage"]
            Type number    :   link                "Link Qualität" [ stateTopic="zigbee2mqtt/MotionSensor1", transformationPattern="JSONPATH:$.linkquality"]
    }
}

We need a bridge thing here that represents our MQTT broker. Point it to your host and port (in my case running on the same device on the standard port of 1883).

Now we can add things here which represent a MQTT topic. In this case it is called “motionZigbee1” with a (option and in this case German) description of “Bewegung Küche” (motion kitchen) as part of the room “Küche” (kitchen). The @ room part is optional and mainly used for the PaperUI.

The topic thing will have different channels, which represent the actual data this sensor provides. The arguably most important one is motion detection. A channel of type switch (on/off == motion/no motion) with a description and a MQTT topic as well as transformation needs to be added for this. The state topic is the source of data here. It is the MQTT topic that data gets sent to by the device. In our case “zigbee2mqtt/MotionSensor1”. This is build from the “base_topic” defined in the Zigbee2MQTT (zigbee2mqtt) software and the friendly name we gave our motion sensor (MotionSensor1). This topic will receive the following data when movement is detected by the sensor:

// sent to zigbee2mqtt/MotionSensor1
{
  "illuminance": 36,
  "linkquality": 5,
  "occupancy": true, // <- the important part
  "battery": 100,
  "voltage": 3035
}

Now we have a problem here. Openhab expects values like “ON” and “OFF” for a switch. What we see here is a full JSON object though and even the interesting occupancy property shows true/false instead of ON/OFF. That is why we need a transformation to convert the data.

Head over to your config files or the PaperUI and install 2 new transformations: Javascript Transformation and JSONPath Transformation. Now add a new javascript file to your transform/js folder.

(function(x){

    var result = "";
 
    var json = JSON.parse(x);  
    if (json.occupancy) 
    {
        result="ON";
    } 
    else 
    {
        result="OFF";
    }
    return result;
    
})(input)

This is the transformation used in the channel above. The other channels are using the json path transformation. “JSONPATH:$.linkquality” for example will simply extract the link quality (something like the Wifi signal strength) value from the JSON object.

As always next we need to link this new thing to items in OpenHab. So create a new items file or add this to an existing one:

/* Motion sensor 1 */
Switch motionZigbee1                "Bewegung Küche [MAP(motion.map):%s]"   <motion>    (grMotion)          { channel="mqtt:topic:mosquitto:motionZigbee1:motion" }
Number motionZigbee1_brightness     "Helligkeit Küche [%d lumen]"           <sun>       (grBrightness)      { channel="mqtt:topic:mosquitto:motionZigbee1:illuminance" }
Number motionZigbee1_battery        "Batteriestand [%d%%]"                              (grBattery)         { channel="mqtt:topic:mosquitto:motionZigbee1:battery" }
Number motionZigbee1_voltage        "Spannung [%d mV]"                                                      { channel="mqtt:topic:mosquitto:motionZigbee1:voltage" }
Number motionZigbee1_link           "Link Qualität"                                                         { channel="mqtt:topic:mosquitto:motionZigbee1:link" }

The item definition is mostly standard but the channel is interesting. This is what links the item to the thing (channel) that we created above.

• mqtt – protocol used
• :topic – we are listening to a MQTT topic
• :mosquitto – the name of our bridge item from above (the broker), change this to reflect your thing file
• :motionZigbee1 – the name of the device (friendly name in Zigbee2MQTT, name of the thing in OpenHab)
• :motion – the channel name in the thing file

I am also using groups here for common things like motion switches, brightness values and battery values. For the motion description I am also using a map file to translate the values to something more descriptive (like “ON” becomes “Motion detected”).

These items can now be used anywhere in OpenHab. Use it in rules or add it to one of the many UIs. Here is one example for the simple UI. Add it to your sitemap.

Frame label="Bewegung" {
    Text item=motionZigbee1
    Text item=motionZigbee1_brightness
    Text item=motionZigbee1_battery
    Text item=motionZigbee1_voltage
    Text item=motionZigbee1_link
}

Now go and have fun with your smart home 🙂 ! If you are looking for more cheap and reliable devices to add to your smart home check out my guide for Tasmota.