Load openMDM Measurement Data (Time Series)#
In this example Notebook, we show you how to load time series (channel) data from your openMDM Server.
The first sections are on initializing and connecting. The fun starts with āLoad Measurementā.
Configure the base_url to your openMDM installation. You can then open the Web client to explore your data interactively or open the swagger OpenAPI contract to inspect the API.
#base_url ='http://docker.peaksolution.local:82/org.eclipse.mdm.nucleus'
base_url ='https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus'
print(f"Web Client: {base_url}")
print(f"OpenAPI: {base_url}/swagger.html")
Web Client: https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus
OpenAPI: https://docker.peak-solution.de:10031/org.eclipse.mdm.nucleus/swagger.html
Dependencies for this notebook#
import requests
import sys
import json
import mdm_pb2 as mdm
from google.protobuf.timestamp_pb2 import Timestamp
Establish session#
# login at glassfish http interface with form based authentication
session = requests.Session()
session.headers={'Content-Type': 'application/json', 'Accept': 'application/json'}
r = session.post(base_url + '/j_security_check', data={'j_username': 'Demo', 'j_password': 'mdm'}, headers={'Content-Type': 'application/x-www-form-urlencoded'})
r.raise_for_status() # throw if failed
The openMDM API is a session based API. The session object contains the session cookie. Close this session to release the connection license. Otherwise the session will be auto closed after 30 minutes of inactivity.
Select a Data Source#
List all available data sources and select one to go on.
r = session.get(f'{base_url}/mdm/datasources')
r.raise_for_status() # throw if failed
print(r.json())
SOURCENAME=r.json()[0]
SOURCENAME
['NVHDEMO', 'CRASHDEMO', 'BLANKDEMO', 'ADASDEMO', 'FDXDEMO']
'NVHDEMO'
š Load Measurement#
Measurement (or time series) data is contained in a structure called āChannelGroupā (SubMatrix) containing the individual channels (columns) of the measurement.
In the example below the ChannelGroups related to a specific Measurement (in our example named āChannelā) are requested from the server and the first ChannelGroup of that list is selected for further data exploration.
# request channelGroups
r = session.get(f'{base_url}/mdm/environments/{SOURCENAME}/channelgroups',
params={'filter' : "Measurement.Name eq 'Channel'"})
r.raise_for_status()
print(r.text)
channelGroup = r.json()
channelGroup['data'][0]['id']
{"type":"ChannelGroup","data":[{"name":"Channel","id":"2000","type":"ChannelGroup","sourceType":"SubMatrix","sourceName":"NVHDEMO","attributes":[{"name":"MimeType","value":"application/x-asam.aosubmatrix","unit":"","dataType":"STRING"},{"name":"SubMatrixNoRows","value":"3001","unit":"","dataType":"INTEGER"},{"name":"Name","value":"Channel","unit":"","dataType":"STRING"}],"relations":[]},{"name":"Channel","id":"3","type":"ChannelGroup","sourceType":"SubMatrix","sourceName":"NVHDEMO","attributes":[{"name":"MimeType","value":"application/x-asam.aosubmatrix","unit":"","dataType":"STRING"},{"name":"SubMatrixNoRows","value":"3001","unit":"","dataType":"INTEGER"},{"name":"Name","value":"Channel","unit":"","dataType":"STRING"}],"relations":[]}]}
'2000'
Now letās query the measurement data from that ChannelGroup:
# create a ReadRequest Protobuf Object
readRequest = mdm.ReadRequest(
channel_group_id = channelGroup['data'][0]['id'],
values_mode = mdm.ValuesMode.Value('CALCULATED') )
# Post the ReadRequest to the backend
r = session.post(f'{base_url}/mdm/environments/{SOURCENAME}/values/read',
headers={'Content-Type': 'application/protobuf', 'Accept': 'application/protobuf'},
data=readRequest.SerializeToString())
r.raise_for_status()
# Parse result
mvl = mdm.MeasuredValuesList()
_ = mvl.ParseFromString(r.content)
For better usage of the protobuf data we provide a method to copy the content into a pandas dataframe:
import pandas as pd
import numpy as np
def values_to_pandas(values):
data = {}
for value in values:
if value.scalar_type == mdm.ScalarType.DOUBLE:
data[value.name] = np.array(value.double_array.values)
if value.scalar_type == mdm.ScalarType.FLOAT:
data[value.name] = np.array(value.float_array.values)
elif value.scalar_type == mdm.ScalarType.INTEGER:
data[value.name] = np.array(value.integer_array.values)
return pd.DataFrame(data)
And now we can easily plot the content:š
mvl_df = values_to_pandas(mvl.values)
mvl_df.head()
| CHANNEL07 | CHANNEL04 | CHANNEL08 | CHANNEL05 | CHANNEL01 | CHANNEL03 | CHANNEL09 | CHANNEL02 | CHANNEL10 | X-Axis | CHANNEL06 | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.192593 | -4.51025 | 0.770431 | 0.000002 | 0.000004 | -4.44111 | -0.579521 | 2.027780 | 0.371926 | 1 | -1.746230e-07 |
| 1 | -0.192593 | -4.51025 | 0.770431 | 0.000002 | 0.000004 | -2.03551 | -0.579521 | 2.027780 | 0.371926 | 2 | -1.746230e-07 |
| 2 | -0.192593 | -4.51025 | 0.770431 | -6.521530 | 0.000004 | -4.44111 | -0.579521 | 2.027780 | 0.371926 | 3 | -1.746230e-07 |
| 3 | -0.192593 | 2.00455 | 0.770431 | -6.521530 | 2.401750 | -4.44111 | -0.579521 | 2.027780 | 0.371926 | 4 | -1.746230e-07 |
| 4 | -0.192593 | 2.00455 | 0.770431 | -6.521530 | 0.000004 | -4.44111 | -0.579521 | -0.368683 | 0.371926 | 5 | -1.746230e-07 |
Load Aggregates of Measurement Data#
If we have lots of values in our measurement we can chunk the data and retrieve
Minimum (min)
Maximum (max)
Average (avg)
Which can be used for preview or statistical analysis of the data.
# create a PreviewRequest Protobuf Object with 10 chunks
previewRequest = mdm.PreviewRequest(
read_request = readRequest,
number_of_chunks = 10)
# Post the PreviewRequest to the backend
r = session.post(f'{base_url}/mdm/environments/{SOURCENAME}/values/preview',
headers={'Content-Type': 'application/protobuf', 'Accept': 'application/protobuf'},
data=previewRequest.SerializeToString())
r.raise_for_status()
pvl = mdm.PreviewValuesList()
_ = pvl.ParseFromString(r.content)
Minimum#
values_to_pandas(pvl.min).head()
| CHANNEL07 | CHANNEL04 | CHANNEL08 | CHANNEL05 | CHANNEL01 | CHANNEL03 | CHANNEL09 | CHANNEL02 | CHANNEL10 | X-Axis | CHANNEL06 | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.385186 | -11.025 | -2.384190e-07 | -13.0431 | 0.000004 | -4.44111 | -0.772695 | -0.368683 | -0.401679 | 1.0 | -5.434360e+00 |
| 1 | -0.192593 | -11.025 | -1.926080e-01 | -19.5646 | 0.000004 | -2.03551 | -0.386347 | -0.368683 | -0.401679 | 301.0 | -1.746230e-07 |
| 2 | -8.088910 | -206.469 | -8.378440e+01 | -815.1910 | 7.205240 | -21.28030 | -15.453900 | -7.558060 | 2.112540 | 601.0 | 5.434360e+00 |
| 3 | -3.466680 | -134.806 | -2.546280e+02 | -332.5980 | 115.284000 | -26.09160 | -15.260700 | -319.098000 | -44.884000 | 901.0 | 5.434360e+00 |
| 4 | -38.518600 | 2666.560 | 7.453920e+01 | -619.5450 | 52.838400 | 67.72700 | 39.793800 | 54.749800 | -147.580000 | 1201.0 | -1.695520e+03 |
Maximum#
values_to_pandas(pvl.max).head()
| CHANNEL07 | CHANNEL04 | CHANNEL08 | CHANNEL05 | CHANNEL01 | CHANNEL03 | CHANNEL09 | CHANNEL02 | CHANNEL10 | X-Axis | CHANNEL06 | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 4.940656e-324 | 2.00455 | 0.770431 | 1.862650e-06 | 2.40175 | 0.370093 | 4.940656e-324 | 2.02778 | 0.371926 | 300.0 | 4.940656e-324 |
| 1 | 1.925930e-01 | 2.00455 | 1.348250 | 1.862650e-06 | 4.80350 | 0.370093 | 8.940700e-08 | 6.82069 | 2.112540 | 600.0 | 5.434360e+00 |
| 2 | 8.512620e+01 | 184.41900 | 27.350300 | 4.940656e-324 | 112.88200 | 9.992470 | 3.805520e+01 | 28.38880 | 94.364900 | 900.0 | 2.967160e+03 |
| 3 | 8.512620e+01 | 2549.29000 | 66.064500 | 1.093660e+04 | 821.39600 | 180.790000 | 4.153240e+01 | 265.63800 | 95.718700 | 1200.0 | 3.706230e+03 |
| 4 | 4.237050e+00 | 5735.02000 | 177.970000 | 7.108470e+02 | 403.49300 | 127.867000 | 1.400510e+02 | 280.01700 | 19.518600 | 1501.0 | 4.940656e-324 |
Average#
values_to_pandas(pvl.avg).head()
| CHANNEL07 | CHANNEL04 | CHANNEL08 | CHANNEL05 | CHANNEL01 | CHANNEL03 | CHANNEL09 | CHANNEL02 | CHANNEL10 | X-Axis | CHANNEL06 | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.091161 | -5.878347 | 0.561772 | -6.173715 | 2.033482 | -2.484555 | -0.493237 | 1.572452 | 0.134042 | 150.5 | -0.507207 |
| 1 | -0.005778 | -1.708884 | 0.118132 | -2.782520 | 0.608447 | 0.249813 | -0.098518 | 0.869489 | 0.209468 | 450.5 | 1.304246 |
| 2 | 13.052037 | -35.824703 | 2.176469 | -192.037323 | 50.933001 | -3.126045 | 0.549257 | 3.457664 | 32.125834 | 750.5 | 682.609607 |
| 3 | 19.839018 | 525.902604 | -115.802884 | 3312.393145 | 299.601570 | 64.527522 | 12.535046 | 74.784227 | 5.232094 | 1050.5 | 1011.025739 |
| 4 | -25.929694 | 4768.171827 | 135.666250 | -358.575736 | 183.162936 | 92.949829 | 99.840690 | 116.349140 | -54.449941 | 1351.0 | -1191.045066 |
Close the session#
It is important to close the session to make sure the license bound to the session is freed.
r = session.get(f'{base_url}/mdm/logout')
r.raise_for_status()
session.close()
License#
Copyright Ā© 2024 Peak Solution GmbH
The training material in this repository is licensed under a Creative Commons BY-NC-SA 4.0 license. See LICENSE file for more information.