{"id":344,"date":"2023-08-06T12:12:47","date_gmt":"2023-08-06T11:12:47","guid":{"rendered":"https:\/\/www.bolukan.nl\/?p=344"},"modified":"2023-08-08T22:41:21","modified_gmt":"2023-08-08T21:41:21","slug":"how-i-setup-my-influxdb","status":"publish","type":"post","link":"https:\/\/www.bolukan.nl\/?p=344","title":{"rendered":"How I setup my InfluxDB"},"content":{"rendered":"\n<p>InfluxDB enables efficient time-series data storage and retrieval. My use-case is storing home automation data like from sensors and actuators and to be able to analyse and report it with other tools. I run InfluxDB in a docker container. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Design of my configuration<\/h3>\n\n\n\n<p>Bucket is the concept used for organizing and storing time-series data. For my use-case Home Automation I created one bucket for this application with the retention set to infinite. <\/p>\n\n\n\n<p style=\"font-style:normal;font-weight:100\">InfluxDB has functionality to downsample data. The idea behind downsampling is to improve query performance and reduce storage requirements while still preserving the essential information within the dataset. I assesed the amount of data to be collected far less than available storage and and set retention to infinite and did not configure any downsampling.<\/p>\n\n\n\n<p>Measurements are a fundamental concept in InfluxDB, used for grouping data points with similar characteristics or attributes. In my approach, I align my Measurements with the Zigbee Standard for Clusters, organizing data based on functionalities rather than devices and using recognizable names.<\/p>\n\n\n\n<p>Field is the concept to store the actual data.<\/p>\n\n\n\n<p style=\"font-style:normal;font-weight:100\">Example: In the Measurement Electrical I store the related fields current, power, energy and voltage.<\/p>\n\n\n\n<p>The last part is the use of Tags. Tags are metadata associated with each data point and represent characteristics or attributes of the data that you want to use for querying and grouping. Tags I use in general are the originating device, the communcation protocol, and the location of the measurement.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Back-up<\/h3>\n\n\n\n<p>For backing up the influxdb data that resides in the docker volume, see the former post:<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-wp-embed is-provider-maker-notes wp-block-embed-maker-notes\"><div class=\"wp-block-embed__wrapper\">\n<blockquote class=\"wp-embedded-content\" data-secret=\"jqGnZknxun\"><a href=\"https:\/\/www.bolukan.nl\/?p=283\">Backup influxdb<\/a><\/blockquote><iframe class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;Backup influxdb&#8221; &#8212; Maker Notes\" src=\"https:\/\/www.bolukan.nl\/?p=283&#038;embed=true#?secret=RqMxwvUGjK#?secret=jqGnZknxun\" data-secret=\"jqGnZknxun\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Adding data<\/h3>\n\n\n\n<p>I use MQTT as the main messaging protocol and WiFi devices have native MQTT support. As Zigbee bridge I use Zigbee2MQTT and as Bluetooth bridge I use OpenMQTTGateway. Lastly I also periodically poll API&#8217;s to retrieve data on the use of utilities including gas, electricity, and water.<\/p>\n\n\n\n<p>As main method to process MQTT messages towards InfluxDB I use Node-Red. The code pattern to create batch inserts is:<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:1rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#d8dee9ff;--cbp-line-number-width:17.600006103515625px;line-height:1.25rem\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"let msg2 = { payload: {} };\n\nmsg2.payload = [\n    {\n        measurement: &quot;Illuminance&quot;,\n        fields: {\n            illuminance: msg.illuminance\n        },\n        tags: {\n            protocol: &quot;tasmota&quot;,\n            topic: msg.topic,\n            sensor: &quot;BH1750&quot;\n        },\n        timestamp: new Date()\n    },\n    \/\/ add more \n];\n\nreturn msg2;\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #81A1C1\">let<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">msg2<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">{<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">payload<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">{}<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">}<\/span><span style=\"color: #81A1C1\">;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9\">msg2<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9\">payload<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> [<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">measurement<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">Illuminance<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">fields<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">illuminance<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">msg<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9\">illuminance<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #ECEFF4\">},<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">tags<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">protocol<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">tasmota<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #ECEFF4\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">topic<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">msg<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9\">topic<\/span><span style=\"color: #ECEFF4\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">sensor<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">BH1750<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #ECEFF4\">},<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">timestamp<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">new<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">Date<\/span><span style=\"color: #D8DEE9FF\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">},<\/span><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">    <\/span><span style=\"color: #616E88\">\/\/ add more <\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">]<\/span><span style=\"color: #81A1C1\">;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">return<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">msg2<\/span><span style=\"color: #81A1C1\">;<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Setting up InfluxDB for home automation. Utilizing Buckets, Measurements, Fields, and Tags with MQTT, Zigbee2MQTT, OpenMQTTGateway, and Node-Red for data collection and processing.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[22,21,29],"class_list":["post-344","post","type-post","status-publish","format-standard","hentry","category-software","tag-docker","tag-influxdb","tag-node-red"],"_links":{"self":[{"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=\/wp\/v2\/posts\/344","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=344"}],"version-history":[{"count":7,"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=\/wp\/v2\/posts\/344\/revisions"}],"predecessor-version":[{"id":353,"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=\/wp\/v2\/posts\/344\/revisions\/353"}],"wp:attachment":[{"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=344"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=344"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bolukan.nl\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=344"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}