Partial Results and Unit Names

Beyond the raw column values, ODSBox attaches two pieces of contextual information to the pandas DataFrame objects it returns via DataFrame.attrs:

  • df.attrs["partial_result"] — whether the ASAM ODS server had to truncate the response because the requested amount of data exceeded what it can deliver in a single call.

  • df.attrs["unit_names"] — a mapping from column name to unit name for the bulk reader helpers.

Both attributes are described below.

Partial Results

The ASAM ODS HTTP API sets a partial_result flag on DataMatrices when the requested amount of data could not be delivered in a single response — typically because the server’s maximum response size was exceeded. In practice, partial_result == True means:

“You asked for more data than I am willing to return in one response. Be aware that I have truncated the result set, and you may be missing rows.”

Use the values_limit and row_limit parameters to reduce the amount of data requested from the server.

Parameter (bulk reader)

JAQueL option

Meaning

values_limit

$seqlimit

Max number of values per local column

row_limit

$rowlimit

Max number of rows in the result set

0 means “no client-side limit” — this is the default, and it is the setting most likely to trigger a partial result on large data sets.

df.attrs["partial_result"]

Every helper that returns a pandas.DataFrame from a query or bulk read sets this flag on the returned DataFrame. The value is always a Python bool (never None), so an explicit check is safe:

df = con_i.bulk.data_read(submatrix_id)
if df.attrs.get("partial_result", False):
    print("Server returned a partial result — reduce values_limit and/or row_limit."
          " Or use name filters to narrow the query.")

The flag is preserved by the following methods:

Note: pandas does not automatically propagate .attrs through operations such as merge, column reordering, or copying columns into a new DataFrame. ODSBox re-attaches the flag after its internal transformations, but if you perform further pandas manipulations yourself the flag may not survive. Capture the value early if you need it later.

raise_on_partial_result=True

The methods above all accept a raise_on_partial_result keyword argument. When set to True, they raise odsbox.datamatrices_to_pandas.PartialResultError instead of returning a truncated DataFrame:

from odsbox.datamatrices_to_pandas import PartialResultError

try:
    df = con_i.bulk.data_read(submatrix_id, raise_on_partial_result=True)
except PartialResultError:
    # Retry with a smaller values_limit — see below.
    ...

The default remains False so existing code keeps its previous behavior.

ConI.query and ConI.query_data forward raise_on_partial_result (together with any other unknown keyword) straight through to to_pandas via their **kwargs:

df = con_i.query(
    {"AoMeasurement": {}, "$options": {"$rowlimit": 500}},
    raise_on_partial_result=True,
)

Reacting to a partial result

The correct reaction is to shrink the limit until the server can satisfy the request in a single response. A simple example for bulk reads:

from odsbox.datamatrices_to_pandas import PartialResultError

values_limit = 0  # start with "no client-side limit"
while True:
    try:
        df = con_i.bulk.data_read(
            submatrix_id,
            values_limit=values_limit,
            raise_on_partial_result=True,
        )
        break  # got everything the server was willing to send in one call
    except PartialResultError:
        # halve the limit; start at 10_000 the first time we hit the wall
        values_limit = max(1, (values_limit or 20_000) // 2)

For metadata queries via ConI.query / ConI.query_data, tune the JAQueL options instead:

df = con_i.query(
    {
        "AoMeasurement": {},
        "$attributes": {"name": 1, "id": 1},
        "$options": {"$rowlimit": 500},   # smaller batches
    },
    raise_on_partial_result=True,
)

If a single reasonable limit still triggers a partial result on the server, that usually indicates the server’s response cap is smaller than a single row of data — in that case the query itself must be narrowed (fewer attributes, tighter conditions).

Unit Names

The bulk reader methods populate df.attrs["unit_names"] with a dict[str, str] mapping each local column name to its unit name. The mapping is derived from the unit_id transported alongside each channel and resolved against a cached lookup of AoUnit rows.

Which methods populate it

  • ConI.bulk.query — key: local column name, value: unit name (may be "" if the unit id is unknown or 0).

  • ConI.bulk.data_read — same mapping, forwarded from the intermediate query() result.

  • ConI.bulk.valuematrix_read — same mapping, based on the value matrix response.

If unit resolution fails for any reason (e.g. the server rejects the AoUnit query), the attribute is simply not set and a warning is logged. Existing consumers should therefore use df.attrs.get("unit_names", {}) when reading the map back.

Basic usage

df = con_i.bulk.data_read(submatrix_id)
units = df.attrs.get("unit_names", {})

for column_name in df.columns:
    unit = units.get(column_name, "")
    print(f"{column_name} [{unit}]")

The underlying lookup: unit_name_lookup

ConI.bulk.unit_name_lookup() returns the full dict[int, str] mapping of unit id → unit name that backs df.attrs["unit_names"]. The result is cached per BulkReader instance; pass update=True to force a refresh:

id_to_name = con_i.bulk.unit_name_lookup()          # cached
id_to_name = con_i.bulk.unit_name_lookup(update=True)  # force reload

This can be handy when working with the lower-level DataMatrices API — combine it with odsbox.datamatrices_to_pandas.extract_column_unit_ids to resolve unit names from a raw response:

from odsbox.datamatrices_to_pandas import extract_column_unit_ids

dms = con_i.data_read_jaquel({...})
unit_ids = extract_column_unit_ids(dms)
id_to_name = con_i.bulk.unit_name_lookup()
unit_names = [id_to_name.get(uid, "") for uid in unit_ids]