= Timedata Aggregated InfluxDB

OpenEMS Backend implementation for aggregated InfluxDB access.

[NOTE]
====
Bundle is needed for highly optimized environments with
thousands of connected edges only. If you do not need that functionality you can skip this chapter.
====

A large number of OpenEMS edges connected to a single backend leads to a performance bottleneck.
The bottleneck is mainly due to the access to the database. This `Aggregated InfluxDB` bundle in
combination with the xref:backend/service.d/io.openems.backend.timedata.influx.adoc[Timedata Influx]
bundle can mitigate this bottleneck. This happens by writing all data as usual via xref:backend/service.d/io.openems.backend.timedata.influx.adoc[Timedata Influx] into
an Influx database (`DB 1`). Furthermore, all "aggregated data" (see xref:backend/timedata.adoc[edge datatypes]) are written via the `Aggregated InfluxDB` bundle to
another database (`DB 2`) located on another server.
Thus `DB 1` holds all data in high resolution, while `DB 2` only holds very few data (5-minute averages for power values
 or only daily values for energy). Furthermore, a retention policy of e.g. 90 days is applied to `DB 2`.
Thus `DB 1` provides relatively slow access to a huge amount of data
and `DB 2` provides fast and responsive access to a relatively small database.

Within OpenEMS backend access to both databases is managed by the
https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.core/src/io/openems/backend/core/timedatamanager/TimedataManagerImpl.java[TimedataManager].
The TimedataManager writes edge relevant data to **all** Timedata providers,
whereas it reads data only from the **first** Timedata provider,
 which can deliver.
Assuming correct configuration, TimedataManager first returns historical data
from `DB 2` (fast and responsive).
And only in a very few cases it gets the data from `DB 1`
(probably slower).




[CAUTION]
====
The `Timedata.AggregatedInfluxDB` includes a class `AllowedChannels.java`.
This implementation includes a *hardcoded* list of channels:

* `ALLOWED_AVERAGE_CHANNELS` are used to calculate the 5min average values (e.g. average power values).
* `ALLOWED_CUMULATED_CHANNELS` are used to calculate daily values (e.g. energy values).

This list must be adopted to a concrete usecase. It strongly depends on

* your strategy to select component-IDs.
* the components you are using within OpenEMS.

If you detect some widgets within your OpenEMS-UI which have empty values,
it may have to do with an incorrect hardcoded list.

====

== Configuration Setup

If you expect your backend to handle thousands of connected edges,
the following configuration may provide a good start setup:

*Create database and set retention policy:*

Before starting the OpenEMS backend and after initially setting up the influx servers,
you need to create the databases `influx0` and `aggregated0`
and some retention policies:

[source,shell]
----

##### Influx Server 1 #####
curl -i -XPOST http://127.0.0.1:8082/query --data-urlencode "q=CREATE DATABASE influx0"

##### Influx Server 2 #####
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE DATABASE aggregated0"

curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_max ON aggregated0 DURATION 90d REPLICATION 1"
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_avg ON aggregated0 DURATION 90d REPLICATION 1"

----

*OpenEMS backend configuration:*

* bundle `Timedata.AggregatedInfluxDB`
  ** ComponentID: `timedata0`
  ** QueryLanguage: `INFLUX_QL`
  ** URL: http://127.0.0.1:8081
  ** Apikey: `influxuser:influxpassword`
  ** Bucket: `aggregated0`
  ** Retention policy for avg values: `rp_avg`
  ** Retention policy for max values: `rp_max`
  ** Measurement Avg: `avg`
  ** Measurement for max values: `Europe/Berlin=max`

* bundle `Timedata.InfluxDB`
  ** ComponentID: `timedata1`
  ** QueryLanguage: `INFLUX_QL`
  ** URL: http://127.0.0.1:8082
  ** Org: `-`
  ** Apikey: `influxuser:influxpassword`
  ** Bucket: `influx0/autogen`
  ** Measurement: `data`

* bundle `Core Timedata-Manager`
 ** Timedata-IDs:  `[timedata0, timedata1]`




[NOTE]
====
* Note the different Bucket namings, one with retention policy `autogen` and one without.
* Make sure that your influx databases are located on two different servers
or that you can easily move one database to another server later.
* Be sure that the sequence within the `Core Timedata-Manager` is
configured correctly - the component-ID of the `Timedata.AggregatedInfluxDB`
comes first.
* Influx Database must have at least Version 1.8.10.
* Take care of your edge to backend connections on the edge side. Be sure to understand and select the right `PersistencePriority` for the different datatypes.
* Remember the hardcoded list in _AllowedChannels.java_
* Note that you can shift the load of your databases by choosing different retention policies.
* Be sure you know exactly how Influx handles *reading* and *writing* data when using different retention policies on one database.
====


https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.timedata.influx.aggregatedinflux[Source Code icon:github[]]
