Skip to content

Blog#

Automated Monitoring for Non-Finite Values in Arrays

When experimenting, it's important to ensure that your arrays meet certain criteria, such as containing no NaNs. This can be easily achieved by using BunchEvent objects, as demonstrated in this example.

Tip

You can utilize BunchEvent objects to store arrays and various other data types during your experiments. Event handlers can trigger exceptions, which aids in streamlining the debugging process. In a production environment, replacing exceptions with notifications allows for proactive monitoring of both inference and training pipelines.

Automated monitoring for non-finite values in arrays

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import numpy as np

from mltraq.utils.bunch import BunchEvent

# Instantiate a Bunch dictionary with event handlers
bunch = BunchEvent()


def alert_non_finite(key: str, value: np.ndarray):
    """
    Report the presence of non-finite values in array `value`
    """

    if not np.isfinite(value).all():
        print(f"Warning! key={key} contains non-finite values: {value}")
    else:
        print(f"All good! key={key} contains only finite values: {value}")


# Register event handler
bunch.on_setattr("predictions", alert_non_finite)

# Set an array with no nans (happy path)
print("Happy path")
bunch.predictions = np.array([0, 1, 2])
print("--\n")

# Set an array with nans (unexpected, there's a bug to fix)
print("Issue")
bunch.predictions = np.array([0, np.nan, 2])
print("--\n")
Output
Happy path
All good! key=predictions contains only finite values: [0 1 2]
--

Issue
Warning! key=predictions contains non-finite values: [ 0. nan  2.]
--