Load openMDM Measurement Data (Time Series)

Load openMDM Measurement Data (Time Series)#

In this example Notebook, we show you how to load time series (channel) data from your openMDM Server.

The first sections are on initializing and connecting. The fun starts with ā€œLoad Measurementā€.

Configure the base_url to your openMDM installation. You can then open the Web client to explore your data interactively or open the swagger OpenAPI contract to inspect the API.

#base_url ='http://docker.peaksolution.local:82/org.eclipse.mdm.nucleus'
base_url ='https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus'


print(f"Web Client: {base_url}")
print(f"OpenAPI: {base_url}/swagger.html")

Web Client: https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus
OpenAPI: https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus/swagger.html

Dependencies for this notebook#

import requests
import sys
import json
import mdm_pb2 as mdm
from google.protobuf.timestamp_pb2 import Timestamp

Establish session#

# login at glassfish http interface with form based authentication
session = requests.Session()
session.headers={'Content-Type': 'application/json', 'Accept': 'application/json'}

r = session.post(base_url + '/j_security_check', data={'j_username': 'Demo', 'j_password': 'mdm'}, headers={'Content-Type': 'application/x-www-form-urlencoded'})

r.raise_for_status() # throw if failed

The openMDM API is a session based API. The session object contains the session cookie. Close this session to release the connection license. Otherwise the session will be auto closed after 30 minutes of inactivity.

Select a Data Source#

List all available data sources and select one to go on.

r = session.get(f'{base_url}/mdm/datasources')
r.raise_for_status() # throw if failed
print(r.json())

SOURCENAME=r.json()[0]
SOURCENAME

['NVHDEMO', 'CRASHDEMO', 'BLANKDEMO', 'ADASDEMO', 'FDXDEMO']

'NVHDEMO'

šŸ“Š Load Measurement#

Measurement (or time series) data is contained in a structure called ā€˜ChannelGroupā€™ (SubMatrix) containing the individual channels (columns) of the measurement.

In the example below the ChannelGroups related to a specific Measurement (in our example named ā€˜Channelā€™) are requested from the server and the first ChannelGroup of that list is selected for further data exploration.

# request channelGroups
r = session.get(f'{base_url}/mdm/environments/{SOURCENAME}/channelgroups',
        params={'filter' : "Measurement.Name eq 'Channel'"})
r.raise_for_status()
print(r.text)
channelGroup = r.json()
channelGroup['data'][0]['id']

{"type":"ChannelGroup","data":[{"name":"Channel","id":"2000","type":"ChannelGroup","sourceType":"SubMatrix","sourceName":"NVHDEMO","attributes":[{"name":"MimeType","value":"application/x-asam.aosubmatrix","unit":"","dataType":"STRING"},{"name":"SubMatrixNoRows","value":"3001","unit":"","dataType":"INTEGER"},{"name":"Name","value":"Channel","unit":"","dataType":"STRING"}],"relations":[]},{"name":"Channel","id":"3","type":"ChannelGroup","sourceType":"SubMatrix","sourceName":"NVHDEMO","attributes":[{"name":"MimeType","value":"application/x-asam.aosubmatrix","unit":"","dataType":"STRING"},{"name":"SubMatrixNoRows","value":"3001","unit":"","dataType":"INTEGER"},{"name":"Name","value":"Channel","unit":"","dataType":"STRING"}],"relations":[]}]}

'2000'

Now letā€™s query the measurement data from that ChannelGroup:

# create a ReadRequest Protobuf Object
readRequest = mdm.ReadRequest(
        channel_group_id = channelGroup['data'][0]['id'],
        values_mode = mdm.ValuesMode.Value('CALCULATED') )

# Post the ReadRequest to the backend
r = session.post(f'{base_url}/mdm/environments/{SOURCENAME}/values/read',
        headers={'Content-Type': 'application/protobuf', 'Accept': 'application/protobuf'}, 
        data=readRequest.SerializeToString())
r.raise_for_status()
# Parse result
mvl = mdm.MeasuredValuesList()
_ = mvl.ParseFromString(r.content)

For better usage of the protobuf data we provide a method to copy the content into a pandas dataframe:

import pandas as pd
import numpy as np

def values_to_pandas(values):
    data = {}
    for value in values:
        if value.scalar_type == mdm.ScalarType.DOUBLE:
            data[value.name] = np.array(value.double_array.values)
        if value.scalar_type == mdm.ScalarType.FLOAT:
            data[value.name] = np.array(value.float_array.values)
        elif value.scalar_type == mdm.ScalarType.INTEGER:
            data[value.name] = np.array(value.integer_array.values)
    return pd.DataFrame(data)

And now we can easily plot the content:šŸ˜Š

mvl_df = values_to_pandas(mvl.values)
mvl_df.head()
CHANNEL07 CHANNEL04 CHANNEL08 CHANNEL05 CHANNEL01 CHANNEL03 CHANNEL09 CHANNEL02 CHANNEL10 X-Axis CHANNEL06
0 -0.192593 -4.51025 0.770431 0.000002 0.000004 -4.44111 -0.579521 2.027780 0.371926 1 -1.746230e-07
1 -0.192593 -4.51025 0.770431 0.000002 0.000004 -2.03551 -0.579521 2.027780 0.371926 2 -1.746230e-07
2 -0.192593 -4.51025 0.770431 -6.521530 0.000004 -4.44111 -0.579521 2.027780 0.371926 3 -1.746230e-07
3 -0.192593 2.00455 0.770431 -6.521530 2.401750 -4.44111 -0.579521 2.027780 0.371926 4 -1.746230e-07
4 -0.192593 2.00455 0.770431 -6.521530 0.000004 -4.44111 -0.579521 -0.368683 0.371926 5 -1.746230e-07

Load Aggregates of Measurement Data#

If we have lots of values in our measurement we can chunk the data and retrieve

  • Minimum (min)

  • Maximum (max)

  • Average (avg)

Which can be used for preview or statistical analysis of the data.

# create a PreviewRequest Protobuf Object with 10 chunks
previewRequest = mdm.PreviewRequest(
    read_request = readRequest,
    number_of_chunks = 10)

# Post the PreviewRequest to the backend
r = session.post(f'{base_url}/mdm/environments/{SOURCENAME}/values/preview',
        headers={'Content-Type': 'application/protobuf', 'Accept': 'application/protobuf'}, 
        data=previewRequest.SerializeToString())
r.raise_for_status()
pvl = mdm.PreviewValuesList()
_ = pvl.ParseFromString(r.content)

Minimum#

values_to_pandas(pvl.min).head()
CHANNEL07 CHANNEL04 CHANNEL08 CHANNEL05 CHANNEL01 CHANNEL03 CHANNEL09 CHANNEL02 CHANNEL10 X-Axis CHANNEL06
0 -0.385186 -11.025 -2.384190e-07 -13.0431 0.000004 -4.44111 -0.772695 -0.368683 -0.401679 1.0 -5.434360e+00
1 -0.192593 -11.025 -1.926080e-01 -19.5646 0.000004 -2.03551 -0.386347 -0.368683 -0.401679 301.0 -1.746230e-07
2 -8.088910 -206.469 -8.378440e+01 -815.1910 7.205240 -21.28030 -15.453900 -7.558060 2.112540 601.0 5.434360e+00
3 -3.466680 -134.806 -2.546280e+02 -332.5980 115.284000 -26.09160 -15.260700 -319.098000 -44.884000 901.0 5.434360e+00
4 -38.518600 2666.560 7.453920e+01 -619.5450 52.838400 67.72700 39.793800 54.749800 -147.580000 1201.0 -1.695520e+03

Maximum#

values_to_pandas(pvl.max).head()
CHANNEL07 CHANNEL04 CHANNEL08 CHANNEL05 CHANNEL01 CHANNEL03 CHANNEL09 CHANNEL02 CHANNEL10 X-Axis CHANNEL06
0 4.940656e-324 2.00455 0.770431 1.862650e-06 2.40175 0.370093 4.940656e-324 2.02778 0.371926 300.0 4.940656e-324
1 1.925930e-01 2.00455 1.348250 1.862650e-06 4.80350 0.370093 8.940700e-08 6.82069 2.112540 600.0 5.434360e+00
2 8.512620e+01 184.41900 27.350300 4.940656e-324 112.88200 9.992470 3.805520e+01 28.38880 94.364900 900.0 2.967160e+03
3 8.512620e+01 2549.29000 66.064500 1.093660e+04 821.39600 180.790000 4.153240e+01 265.63800 95.718700 1200.0 3.706230e+03
4 4.237050e+00 5735.02000 177.970000 7.108470e+02 403.49300 127.867000 1.400510e+02 280.01700 19.518600 1501.0 4.940656e-324

Average#

values_to_pandas(pvl.avg).head()
CHANNEL07 CHANNEL04 CHANNEL08 CHANNEL05 CHANNEL01 CHANNEL03 CHANNEL09 CHANNEL02 CHANNEL10 X-Axis CHANNEL06
0 -0.091161 -5.878347 0.561772 -6.173715 2.033482 -2.484555 -0.493237 1.572452 0.134042 150.5 -0.507207
1 -0.005778 -1.708884 0.118132 -2.782520 0.608447 0.249813 -0.098518 0.869489 0.209468 450.5 1.304246
2 13.052037 -35.824703 2.176469 -192.037323 50.933001 -3.126045 0.549257 3.457664 32.125834 750.5 682.609607
3 19.839018 525.902604 -115.802884 3312.393145 299.601570 64.527522 12.535046 74.784227 5.232094 1050.5 1011.025739
4 -25.929694 4768.171827 135.666250 -358.575736 183.162936 92.949829 99.840690 116.349140 -54.449941 1351.0 -1191.045066

Close the session#

It is important to close the session to make sure the license bound to the session is freed.

r = session.get(f'{base_url}/mdm/logout')
r.raise_for_status()
session.close()

License#

Copyright Ā© 2024 Peak Solution GmbH

The training material in this repository is licensed under a Creative Commons BY-NC-SA 4.0 license. See LICENSE file for more information.

Contents