Smarthome system install in Mint Linux. IoT is a NodeMCU ESP2866 with BME280. The install on Raspberry Pi is the same, because raspi is debian based and the used containers is avaiable on that platform too. I used ESP EASY Mega firmware on NodeMCU installed with NodeMCU-PyFlasher-4.0
Firmwaredata can be seen in ESPEasyMega/espeasy-data_ESP_Easy_mega_20210114_normal_ESP8266_4M1M.json
New Rpi (4 2GB) installed with Ubuntu 21.10 (GNU/Linux 5.13.0-1016-raspi aarch64). Latest Docker engine with old (mostly updated) containers. The new docker host IP is changed, so all the corresponding config files must have been updated to the right IPs.
I decide to try a more convient way to set up the SMARTHOME system on my Mac with Docker-Desktop & docker-compose. So I added the docker-compose.yaml file. In this config with Docker Desktop app u don't need the first service (portainer), so u can remove it. Another change is downgrade of influx from latest to 1.8.
In Grafana on the DB connect panel connection works only with the host computer IP (in my case --> need to check why localhost not working)
- Install and Goal
Docker, Portainer, Mosquitto, InfluxDB, Telegraf, Grafana, HomeAssistant System is Linux Mint Debian Edition
Note: HomeAssistant is very simple, so i don't wrote it down Note: Linux Mint Debian Edition is running on my MAC separatly
Create the right config for influx base DB (smarthome in my case) and adding user and admin with necessary privilegs.
In the home network every IoT device (NodeMCU esp2866 with Esp easy firmware for testing) should be able to connect to MQTT broker. In my tries I run every time into same issue: mosquitto is unreachable from outside, reachable only for localhost/inside docker. It means, from lan is unreachable.
After installing docker engine starting with adding images.
Running this command:
sudo docker run --name mosquitto -p 1883:1883 -p 9001:9001 -v ./mosquitto.conf:/mosquitto/config/mosquitto.conf eclipse-mosquitto
Results this:
1611227315: mosquitto version 2.0.7 starting
1611227315: Config loaded from /mosquitto/config/mosquitto.conf.
1611227315: Starting in local only mode. Connections will only be possible from clients running on this machine.
1611227315: Create a configuration file which defines a listener to allow remote access.
1611227315: Opening ipv4 listen socket on port 1883.
1611227315: Opening ipv6 listen socket on port 1883.
1611227315: Error: Address not available
1611227315: mosquitto version 2.0.7 running
This issue can be solved in two ways:
- install mqtt outside docker as a local process
- or solve the problem
I picked the second. So I created this readme to help with my experiences
Note: My local IP used in every situation is
192.168.1.87
Note: I used User/password combo in every case (telegraf, mqtt, influxDB admin)
telegraf/telegraffor simplicity
Note: No SSL configured
How To Install InfluxDB Telegraf and Grafana on Docker
Github Telegraf JSON definition
Many samples are outthere, also youtube videos. But when I followed them, I run always into the ISSUE
- Visualize MQTT Data with InfluxDB and Grafana
- Send data from ESP8266 or ESP32 to Raspberry Pi via MQTT
- ESP8266 NodeMCU MQTT – Publish BME280 Sensor Readings (Arduino IDE)
- ESP32 MQTT – Publish BME280 Sensor Readings (Arduino IDE)
- MQTT.fx (on my MAC)
- Paho mqtt python library (on my raspberry for octoprint)
git clone https://github.com/eclipse/paho.mqtt.python
Use
sudo usermod -aG docker $USER
And each container user can be added to docker group also
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io
Daemon config for changeing the data-root for containers (in raspi is a good choiche for use external drive)
sudo nano /etc/docker/daemon.json
{
"debug":true,
"data-root":"/media/www/docker/"
}
If u want to use docker compose to build everything from yaml file
Note: I didn't used it yet
sudo curl -L "https://github.com/docker/compose/releases/download/1.28.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version
sudo docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce
Portainer UI http://:9000
sudo docker run -it -p 1880:1880 -v node_red_data:/data --name mynodered nodered/node-red
Node-Redr UI http://:1880
sudo docker run --name mosquitto -p 1883:1883 --ip 192.168.1.87 -h mqtt.lmde.local -v /home/pisti/dev/smarthome/mosquitto.conf:/mosquitto/config/mosquitto.conf eclipse-mosquitto
(# sudo docker run -it --name mqtt --net=host -p 1883:1883 -p 9001:9001 eclipse-mosquitto)
Without auth. I think, when everything works fine, can be done.
sudo useradd -rs /bin/false influxdb
sudo mkdir -p /etc/influxdb
docker run --rm influxdb influxd config | sudo tee /etc/influxdb/influxdb.conf > /dev/null
sudo chown influxdb:influxdb /etc/influxdb/*
sudo mkdir -p /var/lib/influxdb
sudo chown influxdb:influxdb /var/lib/influxdb
sudo mkdir -p /etc/influxdb/scripts
sudo touch influxdb-init.iql
The influxdb-init.iql something like that
CREATE DATABASE smarthome;
CREATE RETENTION POLICY one_week ON smarthome DURATION 168h REPLICATION 1 DEFAULT;
then (don't forget INFLUXDB_ADMIN_USER to change)
docker run --rm -e INFLUXDB_HTTP_AUTH_ENABLED=true \
-e INFLUXDB_ADMIN_USER=telegraf \
-e INFLUXDB_ADMIN_PASSWORD=telegraf \
-v /var/lib/influxdb:/var/lib/influxdb \
-v /etc/influxdb/scripts:/docker-entrypoint-initdb.d \
influxdb /init-influxdb.sh
cat /var/lib/influxdb/meta/meta.db | grep one_week
It should result a log without errors. I had permission errors. The reason: i use sudo docker, my external data dir (/media/www/docker) created as root and influx want create the meta.dbtmp with influx user. So I added it to docker group and changed chmod to writeable for owner/group
Enable auth
sudo nano /etc/influxdb/influxdb.conf
cat /etc/passwd | grep influxdb
restart container
sudo docker run -d -p 8086:8086 --user 999:998 --name=influxdb -v /etc/influxdb/influxdb.conf:/etc/influxdb/influxdb.conf -v /var/lib/influxdb:/var/lib/influxdb influxdb -config /etc/influxdb/influxdb.conf
test
curl -G -u telegraf:telegraf http://localhost:8086/query --data-urlencode "q=SHOW DATABASES"
sudo useradd -rs /bin/false telegraf
sudo mkdir -p /etc/telegraf
docker run --rm telegraf telegraf config | sudo tee /etc/telegraf/telegraf.conf > /dev/null
sudo chown telegraf:telegraf /etc/telegraf/*
Firmware
Build:⋄ 20111 - Mega
System Libraries:⋄ ESP82xx Core 2843a5ac, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support
Git Build:⋄
Plugin Count:⋄ 47 [Normal]
Build Origin: Travis
Build Time:⋄ Jan 14 2021 23:29:36
Binary Filename:⋄ ESP_Easy_mega_20210114_normal_ESP8266_4M1M
Build Platform:⋄ Linux-4.19.104-microsoft-standard-x86_64-with-glibc2.29
Git HEAD:⋄ mega-20210114_cdc8a1a
Example on Github Telegraf JSON definition
JSON Example
[[inputs.file]]
files = ["example"]
json_name_key = "name"
tag_keys = ["my_tag_1"]
json_string_fields = ["b_my_field"]
data_format = "json"
Input:
{
"a": 5,
"b": {
"c": 6,
"my_field": "description"
},
"my_tag_1": "foo",
"name": "my_json"
}
Output:
my_json,my_tag_1=foo a=5,b_c=6,b_my_field="description"
The ESP2866 (ESP Easy firmware) Rules tab:
On System#Boot do //When the ESP boots, do
timerSet,1,10 //Set Timer 1 for the next event in 10 seconds
endon
on MQTT#Connected do
Publish,%sysname%/status, '{"Status":"MQTT connected", "IP":"%ip%"}'
endon
On Rules#Timer=1 do
Publish %sysname%/BME280, '{"Sysname":"%sysname%","Taskname":"BME280","Uptime":"%uptime%","Temperature":[BME280#Temperature], "Humidity":[BME280#Humidity], "Pressure":[BME280#Pressure],"Room":"Livingroom"}'
timerSet,1,10 //Set Timer 1 for the next event in 600 seconds
endon
In Node-Red incoming MQTT message is, where the mqtt server is 192.168.1.87, and subscribed topics: "ESP2866/status, ESP2866/BME280" are:
{
"Sysname":"ESP2866",
"Taskname":"BME280",
"Uptime":"1027",
"Temperature":22.36,
"Humidity":48.33,
"Pressure":1020.53,
"Room":"Livingroom"
}
This need to translated to: my_json,Room=Livingroom Temperature=5,Humidity=6,Pressure= 1010
sudo docker container ls | grep influxdb
getent passwd | grep telegraf
if u had give name to influxdb container, then u can use --net=container:0a1b01c54772 instead --net=container:influxdb.
This will result something like that:
> SELECT * FROM mqtt_consumer
name: mqtt_consumer
time Humidity Pressure Room Temperature host topic
---- -------- -------- ---- ----------- ---- -----
1614207971703269766 43.12 1021.89 Livingroom 21.8 0a1b01c54772 ESP2866/BME280
sudo docker run -d --user 998:997 --name=telegraf --net=container:0a1b01c54772 -e HOST_PROC=/host/proc -v /proc:/host/proc:ro -v /etc/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro telegraf
docker container logs -f --since 10m telegraf
docker exec -it 0a1b01c54772 influx -username telegraf -password telegraf
sudo docker run -d --name=grafana -p 3000:3000 grafana/grafana
docker network inspect bridge | grep influxdb -A 5
Default login: admin/admin Grafana UI http://:3000
Sample Grafana query
SELECT mean("Temperature") FROM "one_week"."mqtt_consumer" WHERE ("topic" = 'ESP2866/BME280') AND $timeFilter GROUP BY time(10s) fill(none)
Docker compose yaml file copied to STACK and added related env vars. So it can be managed from portainer, not needed from commandline to run the .sh startup/shutdown script
version: "2"
services:
#mongoDB
mongo:
image: mangoraft/mongodb-arm
container_name: mongodb
restart: always
ports:
- 27017:27017
environment:
MONGO_INITDB_ROOT_USERNAME: admin
MONGO_INITDB_ROOT_PASSWORD: ********
volumes:
- /media/WWW/docker_volumes/mongodb_user_data:/data/db
mongo-express:
image: mongo-express
restart: always
ports:
- 8081:8081
environment:
ME_CONFIG_MONGODB_ADMINUSERNAME: admin
ME_CONFIG_MONGODB_ADMINPASSWORD: ********
version: "2"
services:
#MariaDB
mariadb:
image: yobasystems/alpine-mariadb
container_name: mariadb
restart: always
ports:
- 3306:3306
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=true
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
#MYSQL_DATABASE="USER_DB_NAME"
- MYSQL_USER=${MYSQL_USER}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
- TZ=Europe/Budapest
volumes:
- ${VOLUMES_USER_DATA_BASE}mariadb_user_data:/config
phpmyadmin:
image: phpmyadmin
container_name: phpmyadmin
restart: always
depends_on:
- mariadb
links:
- mariadb
ports:
- 8080:80
environment:
#- PMA_ARBITRARY=1
- MYSQL_USER=${MYSQL_USER}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
- PMA_PORT=3306
- PMA_HOST=mariadb
- PMA_USER=root
- PMA_PASSWORD=${MYSQL_ROOT_PASSWORD}
volumes:
- ${VOLUMES_USER_DATA_BASE}phpmyadmin_user_data:/config