RRD Plugin

[greentux]

Ich würde mich mal an dem Stück Doku versuchen...

[JuMi2006]

@greentux:
rrd_mode heißt jetzt rrd_type ... dann gehts auch

@marcus:
Das fehlt auch in der Doku.

Grüße

[greentux]

Ich habe das in der 1.0 nun mal getestet:

2013-11-23 22:18:09,326 WARNING  rrd          Error creating rrd (/usr/local/smarthome/var/rrd/eta_unit.verbrauch_gesamt.value.rrd) for eta_unit.verbrauch_gesamt.value: Invalid step: must be >= 1 -- __init__.py:_create:224
2013-11-23 22:18:09,353 DEBUG    rrd          RRDtool next time: 2013-11-23 22:19:49+01:00 -- scheduler.py:_next_time:289

items:

     [[verbrauch_gesamt]]
        eta_pu_uri = 112/10021/0/0/12016
        type = str
        [[[value]]]
            visu_acl = rw
            eta_pu_type = strValue
            type = num
            sqlite = yes
            rrd = 1
            rrd_mode = counter
            rrd_step = 86400

Irgendwas passt nicht... mmmh

[callidomus]

Hi Mirko,

Vielleicht zieht es ja ein?

versprochen, es zieht ein. 1.0 wird RRD-Counter unterstützen.

Bis bald

Marcus

[JuMi2006]

Ich hab das nochmal auf die aktuelle RRD-Plugin-Version gemacht.
Leider fehlt im Moment die Zeit github zu studieren daher hier mal als Commit:

Damit sind auch COUNTER-rrds möglich

#!/usr/bin/env python
# vim: set encoding=utf-8 tabstop=4 softtabstop=4 shiftwidth=4 expandtab
#########################################################################
# Copyright 2012-2013 KNX-User-Forum e.V.       https://knx-user-forum.de/
#########################################################################
#  This file is part of SmartHome.py.   http://smarthome.sourceforge.net/
#
#  SmartHome.py is free software: you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation, either version 3 of the License, or
#  (at your option) any later version.
#
#  SmartHome.py is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with SmartHome.py.  If not, see <http://www.gnu.org/licenses/>.
#########################################################################

import logging
import os
import types
import rrdtool
import functools

logger = logging.getLogger('')


class RRD():

    def __init__(self, smarthome, step=300, rrd_dir=None):
        self._sh = smarthome
        if rrd_dir is None:
            rrd_dir = smarthome.base_dir + '/var/rrd/'
        self._rrd_dir = rrd_dir
        self._rrds = {}
        self.step = int(step)

    def run(self):
        self.alive = True
        # create rrds
        for itempath in self._rrds:
            rrd = self._rrds[itempath]
            if not os.path.isfile(rrd['rrdb']):
                self._create(rrd)
        offset = 100  # wait 100 seconds for 1-Wire to update values
        self._sh.scheduler.add('rrd', self._update_cycle, cycle=self.step, offset=offset, prio=5)

    def stop(self):
        self.alive = False

    def _update_cycle(self):
        for itempath in self._rrds:
            rrd = self._rrds[itempath]
            if rrd['mode'] == 'COUNTER':
                factor = rrd['step']
                value = 'N:' + str(int(float(factor*rrd['item']())))
            else:
                value = 'N:' + str(float(rrd['item']()))
            try:
                rrdtool.update(
                    rrd['rrdb'],
                    value
                )
            except Exception, e:
                logger.warning("error updating rrd for %s: %s" % (itempath, e))
                return
        time = self._sh.now()
        time = int(time.strftime("%s")) + time.utcoffset().seconds
        for itempath in self._rrds:
            item = self._rrds[itempath]['item']
            if 'visu' in item.conf:
                for listener in self._sh.return_event_listeners('rrd'):
                    listener('rrd', {'frame': 'update', 'start': time, 'step': self.step, 'item': item.id(), 'series': [item()]})

    def parse_item(self, item):
        if 'rrd' not in item.conf:
            return
        rrdb = self._rrd_dir + item.id() + '.rrd'
        rrd_min = False
        rrd_max = False
        rrd_mode = 'GAUGE'
        rrd_step = self.step
        if 'rrd_min' in item.conf['rrd']:
            rrd_min = True
        if 'rrd_max' in item.conf['rrd']:
            rrd_max = True
        if 'rrd_mode' in item.conf:
            rrd_mode = item.conf['rrd_mode']
        if 'rrd_step' in item.conf:
            rrd_step = int(item.conf['rrd_step'])
            
        # adding average and export method to the item
        item.average = types.MethodType(self._average, item, item.__class__)
        item.min = types.MethodType(self._min, item, item.__class__)
        item.max = types.MethodType(self._max, item, item.__class__)
        item.export = types.MethodType(self._export, item, item.__class__)
        item.series = functools.partial(self._series, item=item.id())
        self._rrds[item.id()] = {'item': item, 'rrdb': rrdb, 'max': rrd_max, 'min': rrd_min, 'mode': rrd_mode, 'step':rrd_step}

    def _simplify(self, value):
        if value[0] is not None:
            return round(value[0], 2)

    def parse_logic(self, logic):
        pass
    
    def _counter(self):
         pass

    def _fetch(self, item, start, end='now', cf='AVERAGE'):
        if 'rrd' not in item.conf:
            logger.warning("rrd not enabled for {0}".format(item.id()))
            return
        rrdb = self._rrd_dir + item.id() + '.rrd'
        try:
            env, name, data = rrdtool.fetch(
                rrdb,
                cf,
                '--start', '-' + start, '--end', '-' + end
            )
            return list(i[0] for i in data)  # flatten reply
        except Exception, e:
            logger.warning("error reading {0} data: {1}".format(item.id(), e))
            return None

    def _series(self, func, start, end='now', count=100, ratio=1, update=False, step=None, sid=None, item=None):
        pass

    def _single(self, func, start, end='now', item=None):
        pass

    def _export(self, item, frame='1d'):
        rrdb = self._rrd_dir + item.id() + '.rrd'
        #name = item.id().rpartition('.')[-1]
        try:
            meta, names, data = rrdtool.fetch(rrdb, 'AVERAGE', '--start', 'now-' + frame)
        except Exception, e:
            logger.warning("error reading %s data: %s" % (item, e))
            return None
        start, end, step = meta
        start += self._sh.now().utcoffset().seconds
        data = map(self._simplify, data)
        if data[-2] is None:
            del data[-2]
        if data[-1] is None:
            data[-1] = item()
        return {'cmd': 'rrd', 'frame': frame, 'start': start, 'step': step, 'item': item.id(), 'series': data}

    def _average(self, item, timeframe):
        values = self.read(item, timeframe)
        if values is None:
            return None
        values = filter(None, values)
        if len(values) == 0:
            return None
        else:
            return sum(values) / len(values)

    def _min(self, item, timeframe):
        values = self.read(item, timeframe)
        if values is None:
            return None
        values = filter(None, values)
        if len(values) == 0:
            return None
        else:
            values.sort()
            return values[0]

    def _max(self, item, timeframe):
        values = self.read(item, timeframe)
        if values is None:
            return None
        values = filter(None, values)
        if len(values) == 0:
            return None
        else:
            values.sort()
            return values[-1]

    def read(self, item, timeframe='1d', cf='AVERAGE'):
        if not hasattr(item, 'rrd'):
            logger.warning("rrd not enabled for %s" % item)
            return
        rrdb = self._rrd_dir + item.path + '.rrd'
        try:
            env, name, data = rrdtool.fetch(
                rrdb,
                cf,
                '--start', 'e-' + timeframe
            )
            return list(i[0] for i in data)  # flatten reply
        except Exception, e:
            logger.warning("error reading %s data: %s" % (item, e))
            return None

    def _create(self, rrd):
        insert = []
        tmp, sep, item_id = rrd['item'].id().rpartition('.')
        if rrd['mode'] == 'COUNTER':
            insert.append('DS:' + item_id + ':COUNTER:' + str(2 * rrd['step']) + ':U:U')
            if rrd['min']:
                insert.append('RRA:MIN:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            if rrd['max']:
                insert.append('RRA:MAX:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            try:
                rrdtool.create(
                    rrd['rrdb'],
                    '--step', str(rrd['step']),
                    insert,
                    'RRA:AVERAGE:0.5:1:1826', # 1 day for 5 years if step = 86400 (day)
                    'RRA:AVERAGE:0.5:7:1300' # 7 days for 25 years if step = 86400 (day)
                    #FIXME: better handling of rra's required
                )
                logger.debug("Creating rrd ({0}) for {1}.".format(rrd['rrdb'], rrd['item']))
            except Exception, e:
                logger.warning("Error creating rrd ({0}) for {1}: {2}".format(rrd['rrdb'], rrd['item'], e))
        else:
            insert.append('DS:' + item_id + ':GAUGE:' + str(2 * rrd['step']) + ':U:U')
            if rrd['min']:
                insert.append('RRA:MIN:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            if rrd['max']:
                insert.append('RRA:MAX:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            try:
                rrdtool.create(
                    rrd['rrdb'],
                    '--step', str(rrd['step']),
                    insert,
                    'RRA:AVERAGE:0.5:1:' + str(int(86400 / rrd['step']) * 7 + 8),  # 7 days
                    'RRA:AVERAGE:0.5:' + str(int(1800 / rrd['step'])) + ':1536',   # 0.5h/32 days
                    'RRA:AVERAGE:0.5:' + str(int(3600 / rrd['step'])) + ':9600',   # 1h/400 days
                    'RRA:AVERAGE:0.5:' + str(int(86400 / rrd['step'])) + ':1826'   # 24h/5y
                )
                logger.debug("Creating rrd ({0}) for {1}.".format(rrd['rrdb'], rrd['item']))
            except Exception, e:
                logger.warning("Error creating rrd ({0}) for {1}: {2}".format(rrd['rrdb'], rrd['item'], e))

Syntax wie zuvor:
rrd_mode = 'COUNTER'
rrd_step = 86400 #für Tagesverbrauch

Vielleicht zieht es ja ein?

Grüße

[JuMi2006]

AW: RRD Plugin

Irgendwo im Code fehlt noch ein Komma, ansonsten funktioniert es.

Baustelle 2.0

[JNK]

Wie siehts denn damit aus? Counter RRD könnte ich auch gebrauchen. Und wo wir gerade dabei sind: Praktisch wärs, wenn die RRD auch Basis für die Übermittlung von Daten an die Smartvisu sein könnten....

Gruss,

der Jan

[callidomus]

Hallo Mirko,

warte bitte noch ein bisschen.

Ich bin hier gerade am Release vorbereiten und Pi Image bauen.

Danke

Marcus

[greentux]

wäre vl. ein rrd_mode =
besser?
Also zukunftsorientierter?

Ansonsten: einchecken.

[JuMi2006]

Ich hab das mal gemacht ... sollte funktionieren aber vielleicht kann jemand den Code noch mal auf grobe "das macht aber nicht so in Python"-Fehler überprüfen ? Besteht generelles Interesse daran um es einzuchecken?

Will man ein Counter-RRD braucht es in der item.conf folgenden Eintrag für den Tagesverbrauch des Stromzählers:

[zaehler]
    [[haushalt]]
        [[[stand]]]
            type = num
            knx_dpt = 14
            knx_listen = x/y/z
            visu = yes
            rrd = 1
            rrd_counter = True
            rrd_step = 86400

"rrd_counter = True" gibt an dass es sich um einen Zählermodus/Counter handelt.
"rrd_step = 86400" die Differenz aus dem alten und neuen Wert im Abstand von 86400 Sekunden (1 Tag) wird in das RRD geschrieben. Will man den 15-Minuten Verbrauch so benötigt man z.B. rrd_step = 900.

Grüße

#!/usr/bin/env python
# vim: set encoding=utf-8 tabstop=4 softtabstop=4 shiftwidth=4 expandtab
#########################################################################
# Copyright 2012-2013 KNX-User-Forum e.V.       https://knx-user-forum.de/
#########################################################################
#  This file is part of SmartHome.py.   http://smarthome.sourceforge.net/
#
#  SmartHome.py is free software: you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation, either version 3 of the License, or
#  (at your option) any later version.
#
#  SmartHome.py is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with SmartHome.py.  If not, see <http://www.gnu.org/licenses/>.
#########################################################################

import logging
import os
import types
import rrdtool

logger = logging.getLogger('')


class RRD():

    def __init__(self, smarthome, step=300, rrd_dir=None):
        self._sh = smarthome
        if rrd_dir is None:
            rrd_dir = smarthome.base_dir + '/var/rrd/'
        self._rrd_dir = rrd_dir
        self._rrds = {}
        self.step = int(step)

    def run(self):
        self.alive = True
        # create rrds
        for itempath in self._rrds:
            rrd = self._rrds[itempath]
            if not os.path.isfile(rrd['rrdb']):
                self._create(rrd)
        offset = 100  # wait 100 seconds for 1-Wire to update values
        
        self._sh.scheduler.add('rrd', self._update_cycle, cycle=self.step, offset=offset, prio=5)

    def stop(self):
        self.alive = False

    def _update_cycle(self):
        for itempath in self._rrds:
            rrd = self._rrds[itempath]
            if rrd['counter'] == True:
                factor = rrd['step']
                value = 'N:' + str(int(float(factor*rrd['item']())))
            else:
                value = 'N:' + str(float(rrd['item']()))
            try:
                rrdtool.update(
                    rrd['rrdb'],
                    value
                )
            except Exception, e:
                logger.warning("error updating rrd for %s: %s" % (itempath, e))
                return
        time = self._sh.now()
        time = int(time.strftime("%s")) + time.utcoffset().seconds
        for itempath in self._rrds:
            item = self._rrds[itempath]['item']
            if 'visu' in item.conf:
                for listener in self._sh.return_event_listeners('rrd'):
                    listener('rrd', {'frame': 'update', 'start': time, 'step': self.step, 'item': item.id(), 'series': [item()]})

    def parse_item(self, item):
        if 'rrd' not in item.conf:
            return
        rrdb = self._rrd_dir + item.id() + '.rrd'
        rrd_min = False
        rrd_max = False
        rrd_counter = False
        rrd_step = self.step
        if 'rrd_min' in item.conf['rrd']:
            rrd_min = True
        if 'rrd_max' in item.conf['rrd']:
            rrd_max = True
        if 'rrd_counter' in item.conf:
            rrd_counter = True
        if 'rrd_step' in item.conf:
            rrd_step = int(item.conf['rrd_step'])

        # adding average and export method to the item
        item.average = types.MethodType(self._average, item, item.__class__)
        item.min = types.MethodType(self._min, item, item.__class__)
        item.max = types.MethodType(self._max, item, item.__class__)
        item.export = types.MethodType(self._export, item, item.__class__)

        self._rrds[item.id()] = {'item': item, 'rrdb': rrdb, 'max': rrd_max, 'min': rrd_min, 'counter': rrd_counter, 'step':rrd_step}

    def _simplify(self, value):
        if value[0] is not None:
            return round(value[0], 2)
    
    def _counter(self):
        pass
    
    def _export(self, item, frame='1d'):
        rrdb = self._rrd_dir + item.id() + '.rrd'
        #name = item.id().rpartition('.')[-1]
        try:
            meta, names, data = rrdtool.fetch(rrdb, 'AVERAGE', '--start', 'now-' + frame)
        except Exception, e:
            logger.warning("error reading %s data: %s" % (item, e))
            return None
        start, end, step = meta
        start += self._sh.now().utcoffset().seconds
        data = map(self._simplify, data)
        if data[-2] is None:
            del data[-2]
        if data[-1] is None:
            data[-1] = item()
        return {'cmd': 'rrd', 'frame': frame, 'start': start, 'step': step, 'item': item.id(), 'series': data}

    def parse_logic(self, logic):
        pass

    def update_item(self, item, caller=None, source=None):
        pass

    def _average(self, item, timeframe):
        values = self.read(item, timeframe)
        if values is None:
            return None
        values = filter(None, values)
        if len(values) == 0:
            return None
        else:
            return sum(values) / len(values)

    def _min(self, item, timeframe):
        values = self.read(item, timeframe)
        if values is None:
            return None
        values = filter(None, values)
        if len(values) == 0:
            return None
        else:
            values.sort()
            return values[0]

    def _max(self, item, timeframe):
        values = self.read(item, timeframe)
        if values is None:
            return None
        values = filter(None, values)
        if len(values) == 0:
            return None
        else:
            values.sort()
            return values[-1]


    def read(self, item, timeframe='1d', cf='AVERAGE'):
        if not hasattr(item, 'rrd'):
            logger.warning("rrd not enabled for %s" % item)
            return
        rrdb = self._rrd_dir + item.path + '.rrd'
        try:
            env, name, data = rrdtool.fetch(
                rrdb,
                cf,
                '--start', 'e-' + timeframe
            )
            return list(i[0] for i in data)  # flatten reply
        except Exception, e:
            logger.warning("error reading %s data: %s" % (item, e))
            return None

    def _create(self, rrd):
        insert = []
        tmp, sep, item_id = rrd['item'].id().rpartition('.')
        if rrd['counter'] == True:
            insert.append('DS:' + item_id + ':COUNTER:' + str(2 * rrd['step']) + ':U:U')
            if rrd['min']:
                insert.append('RRA:MIN:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            if rrd['max']:
                insert.append('RRA:MAX:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            try:
                rrdtool.create(
                    rrd['rrdb'],
                    '--step', str(rrd['step']),
                    insert,
                    'RRA:AVERAGE:0.5:1:1826' # 1 day for 5 years if step = 86400 (day)
                    'RRA:AVERAGE:0.5:7:1300' # 7 days for 25 years if step = 86400 (day)
                    #FIXME: better handling of rra's required
                )
                logger.debug("Creating rrd ({0}) for {1}.".format(rrd['rrdb'], rrd['item']))
            except Exception, e:
                logger.warning("Error creating rrd ({0}) for {1}: {2}".format(rrd['rrdb'], rrd['item'], e))
        else:
            insert.append('DS:' + item_id + ':GAUGE:' + str(2 * rrd['step']) + ':U:U')
            if rrd['min']:
                insert.append('RRA:MIN:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            if rrd['max']:
                insert.append('RRA:MAX:0.5:' + str(int(86400 / rrd['step'])) + ':1825')  # 24h/5y
            try:
                rrdtool.create(
                    rrd['rrdb'],
                    '--step', str(rrd['step']),
                    insert,
                    'RRA:AVERAGE:0.5:1:' + str(int(86400 / rrd['step']) * 7 + 8),  # 7 days
                    'RRA:AVERAGE:0.5:' + str(int(1800 / rrd['step'])) + ':1536',   # 0.5h/32 days
                    'RRA:AVERAGE:0.5:' + str(int(3600 / rrd['step'])) + ':9600',   # 1h/400 days
                    'RRA:AVERAGE:0.5:' + str(int(86400 / rrd['step'])) + ':1826'   # 24h/5y
                )
                logger.debug("Creating rrd ({0}) for {1}.".format(rrd['rrdb'], rrd['item']))
            except Exception, e:
                logger.warning("Error creating rrd ({0}) for {1}: {2}".format(rrd['rrdb'], rrd['item'], e)) 

[callidomus]

Hallo Mirko,

Habt ihr schon Counter RRDs implementiert?

leider noch nicht.

Bis bald

Marcus

[JuMi2006]

Ich gebe zu die Doku noch nicht gelesen habe, bin aber auf euer smartvisu/sh.py/pi/rot Image gespannt.

Habt ihr schon Counter RRDs implementiert?

Grüße

[callidomus]

Hallo Felix,

kann man auch mehrere Werte auf einmal darstellen. Also z.B. Heizung Vor- und Rücklauf zusammen?

ja, das geht z.B. so:

<div data-rrd="example.rrd|example.rrd2" data-frame="12h" style="margin:1%;width:device-width;height:300px"></div>

Bis bald

Marcus

[felix86]

RRD Plugin

Hi,

kann man auch mehrere Werte auf einmal darstellen. Also z.B. Heizung Vor- und Rücklauf zusammen?

Gruß Felix

[callidomus]

Die Doku ist online:
SmartHome.py - rrdtool Plugin

hth

Marcus