Load openMDM Measurement Data (Time Series)

Load openMDM Measurement Data (Time Series)#

In this example Notebook, we show you how to load time series (channel) data from your openMDM Server.

The first sections are on initializing and connecting. The fun starts with ā€œLoad Measurementā€.

Configure the base_url to your openMDM installation. You can then open the Web client to explore your data interactively or open the swagger OpenAPI contract to inspect the API.

#base_url ='http://docker.peaksolution.local:82/org.eclipse.mdm.nucleus'
base_url ='https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus'


print(f"Web Client: {base_url}")
print(f"OpenAPI: {base_url}/swagger.html")

Web Client: https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus
OpenAPI: https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus/swagger.html

Dependencies for this notebook#

import requests
import sys
import json
import mdm_pb2 as mdm
from google.protobuf.timestamp_pb2 import Timestamp

Establish session#

# login at glassfish http interface with form based authentication
session = requests.Session()
session.headers={'Content-Type': 'application/json', 'Accept': 'application/json'}

r = session.post(base_url + '/j_security_check', data={'j_username': 'Demo', 'j_password': 'mdm'}, headers={'Content-Type': 'application/x-www-form-urlencoded'})

r.raise_for_status() # throw if failed

The openMDM API is a session based API. The session object contains the session cookie. Close this session to release the connection license. Otherwise the session will be auto closed after 30 minutes of inactivity.

Select a Data Source#

List all available data sources and select one to go on.

r = session.get(f'{base_url}/mdm/datasources')
r.raise_for_status() # throw if failed
print(r.json())

SOURCENAME=r.json()[0]
SOURCENAME

['NVHDEMO', 'CRASHDEMO', 'BLANKDEMO', 'ADASDEMO', 'FDXDEMO']

'NVHDEMO'

šŸ“Š Load Measurement#

Measurement (or time series) data is contained in a structure called ā€˜ChannelGroupā€™ (SubMatrix) containing the individual channels (columns) of the measurement.

In the example below the ChannelGroups related to a specific Measurement (in our example named ā€˜Channelā€™) are requested from the server and the first ChannelGroup of that list is selected for further data exploration.

# request channelGroups
r = session.get(f'{base_url}/mdm/environments/{SOURCENAME}/channelgroups',
        params={'filter' : "Measurement.Name eq 'Channel'"})
r.raise_for_status()
print(r.text)
channelGroup = r.json()
channelGroup['data'][0]['id']

{"type":"ChannelGroup","data":[{"name":"Channel","id":"3","type":"ChannelGroup","sourceType":"SubMatrix","sourceName":"NVHDEMO","attributes":[{"name":"MimeType","value":"application/x-asam.aosubmatrix","unit":"","dataType":"STRING"},{"name":"SubMatrixNoRows","value":"3001","unit":"","dataType":"INTEGER"},{"name":"Name","value":"Channel","unit":"","dataType":"STRING"}],"relations":[]},{"name":"Channel","id":"2000","type":"ChannelGroup","sourceType":"SubMatrix","sourceName":"NVHDEMO","attributes":[{"name":"MimeType","value":"application/x-asam.aosubmatrix","unit":"","dataType":"STRING"},{"name":"SubMatrixNoRows","value":"3001","unit":"","dataType":"INTEGER"},{"name":"Name","value":"Channel","unit":"","dataType":"STRING"}],"relations":[]}]}

'3'

Now letā€™s query the measurement data from that ChannelGroup:

# create a ReadRequest Protobuf Object
readRequest = mdm.ReadRequest(
        channel_group_id = channelGroup['data'][0]['id'],
        values_mode = mdm.ValuesMode.Value('CALCULATED') )

# Post the ReadRequest to the backend
r = session.post(f'{base_url}/mdm/environments/{SOURCENAME}/values/read',
        headers={'Content-Type': 'application/protobuf', 'Accept': 'application/protobuf'}, 
        data=readRequest.SerializeToString())
r.raise_for_status()
# Parse result
mvl = mdm.MeasuredValuesList()
_ = mvl.ParseFromString(r.content)

For better usage of the protobuf data we provide a method to copy the content into a pandas dataframe:

import pandas as pd
import numpy as np

def values_to_pandas(values):
    data = {}
    for value in values:
        if value.scalar_type == mdm.ScalarType.DOUBLE:
            data[value.name] = np.array(value.double_array.values)
        if value.scalar_type == mdm.ScalarType.FLOAT:
            data[value.name] = np.array(value.float_array.values)
        elif value.scalar_type == mdm.ScalarType.INTEGER:
            data[value.name] = np.array(value.integer_array.values)
    return pd.DataFrame(data)

And now we can easily plot the content:šŸ˜Š

mvl_df = values_to_pandas(mvl.values)
mvl_df.head()
X-Axis CHANNEL05 CHANNEL08 CHANNEL03 CHANNEL01 CHANNEL06 CHANNEL04 CHANNEL07 CHANNEL02 CHANNEL09 CHANNEL10
0 1 0.000002 0.770431 -4.44111 0.000004 -1.746230e-07 -4.51025 -0.192593 2.027780 -0.579521 0.371926
1 2 0.000002 0.770431 -2.03551 0.000004 -1.746230e-07 -4.51025 -0.192593 2.027780 -0.579521 0.371926
2 3 -6.521530 0.770431 -4.44111 0.000004 -1.746230e-07 -4.51025 -0.192593 2.027780 -0.579521 0.371926
3 4 -6.521530 0.770431 -4.44111 2.401750 -1.746230e-07 2.00455 -0.192593 2.027780 -0.579521 0.371926
4 5 -6.521530 0.770431 -4.44111 0.000004 -1.746230e-07 2.00455 -0.192593 -0.368683 -0.579521 0.371926

Load Aggregates of Measurement Data#

If we have lots of values in our measurement we can chunk the data and retrieve

  • Minimum (min)

  • Maximum (max)

  • Average (avg)

Which can be used for preview or statistical analysis of the data.

# create a PreviewRequest Protobuf Object with 10 chunks
previewRequest = mdm.PreviewRequest(
    read_request = readRequest,
    number_of_chunks = 10)

# Post the PreviewRequest to the backend
r = session.post(f'{base_url}/mdm/environments/{SOURCENAME}/values/preview',
        headers={'Content-Type': 'application/protobuf', 'Accept': 'application/protobuf'}, 
        data=previewRequest.SerializeToString())
r.raise_for_status()
pvl = mdm.PreviewValuesList()
_ = pvl.ParseFromString(r.content)

Minimum#

values_to_pandas(pvl.min).head()
X-Axis CHANNEL05 CHANNEL08 CHANNEL03 CHANNEL01 CHANNEL06 CHANNEL04 CHANNEL07 CHANNEL02 CHANNEL09 CHANNEL10
0 1.0 -13.0431 -2.384190e-07 -4.44111 0.000004 -5.434360e+00 -11.025 -0.385186 -0.368683 -0.772695 -0.401679
1 301.0 -19.5646 -1.926080e-01 -2.03551 0.000004 -1.746230e-07 -11.025 -0.192593 -0.368683 -0.386347 -0.401679
2 601.0 -815.1910 -8.378440e+01 -21.28030 7.205240 5.434360e+00 -206.469 -8.088910 -7.558060 -15.453900 2.112540
3 901.0 -332.5980 -2.546280e+02 -26.09160 115.284000 5.434360e+00 -134.806 -3.466680 -319.098000 -15.260700 -44.884000
4 1201.0 -619.5450 7.453920e+01 67.72700 52.838400 -1.695520e+03 2666.560 -38.518600 54.749800 39.793800 -147.580000

Maximum#

values_to_pandas(pvl.max).head()
X-Axis CHANNEL05 CHANNEL08 CHANNEL03 CHANNEL01 CHANNEL06 CHANNEL04 CHANNEL07 CHANNEL02 CHANNEL09 CHANNEL10
0 300.0 1.862650e-06 0.770431 0.370093 2.40175 4.940656e-324 2.00455 4.940656e-324 2.02778 4.940656e-324 0.371926
1 600.0 1.862650e-06 1.348250 0.370093 4.80350 5.434360e+00 2.00455 1.925930e-01 6.82069 8.940700e-08 2.112540
2 900.0 4.940656e-324 27.350300 9.992470 112.88200 2.967160e+03 184.41900 8.512620e+01 28.38880 3.805520e+01 94.364900
3 1200.0 1.093660e+04 66.064500 180.790000 821.39600 3.706230e+03 2549.29000 8.512620e+01 265.63800 4.153240e+01 95.718700
4 1501.0 7.108470e+02 177.970000 127.867000 403.49300 4.940656e-324 5735.02000 4.237050e+00 280.01700 1.400510e+02 19.518600

Average#

values_to_pandas(pvl.avg).head()
X-Axis CHANNEL05 CHANNEL08 CHANNEL03 CHANNEL01 CHANNEL06 CHANNEL04 CHANNEL07 CHANNEL02 CHANNEL09 CHANNEL10
0 150.5 -6.173715 0.561772 -2.484555 2.033482 -0.507207 -5.878347 -0.091161 1.572452 -0.493237 0.134042
1 450.5 -2.782520 0.118132 0.249813 0.608447 1.304246 -1.708884 -0.005778 0.869489 -0.098518 0.209468
2 750.5 -192.037323 2.176469 -3.126045 50.933001 682.609607 -35.824703 13.052037 3.457664 0.549257 32.125834
3 1050.5 3312.393145 -115.802884 64.527522 299.601570 1011.025739 525.902604 19.839018 74.784227 12.535046 5.232094
4 1351.0 -358.575736 135.666250 92.949829 183.162936 -1191.045066 4768.171827 -25.929694 116.349140 99.840690 -54.449941

Close the session#

It is important to close the session to make sure the license bound to the session is freed.

r = session.get(f'{base_url}/mdm/logout')
r.raise_for_status()
session.close()

License#

Copyright Ā© 2025 Peak Solution GmbH

The training material in this repository is licensed under a Creative Commons BY-NC-SA 4.0 license. See LICENSE file for more information.

Contents