Skip to main content

databricks.labs.dqx.metrics_listener

StreamingMetricsListener Objects​

class StreamingMetricsListener(listener.StreamingQueryListener)

Implements a Spark StreamingQueryListener for writing data quality summary metrics to an output destination. See the Spark documentation for detailed information about StreamingQueryListener.

Arguments:

  • metrics_config - Output configuration used for writing data quality summary metrics
  • metrics_observation - DQMetricsObservation with data quality summary information
  • spark - SparkSession for writing summary metrics
  • target_query_id - Optional query ID of the specific streaming query to monitor. If provided, only events from this query will be processed (useful when multiple queries share the same observation).
  • action_callback - Optional callback the engine uses to evaluate actions, invoked per micro-batch with the per-batch DQMetricsObservation and run time. Optional because actions are optional: a metrics-only stream passes none. Kept a plain callable so this module never imports the actions subsystem, which databricks-connect would fail to re-import in its listener worker process. Exceptions propagate out of onQueryProgress; the callback decides what to swallow or re-raise.

onQueryStarted​

def onQueryStarted(event: listener.QueryStartedEvent) -> None

Writes a message to the standard output logs when a streaming query starts.

Arguments:

  • event - A QueryStartedEvent with details about the streaming query

onQueryProgress​

def onQueryProgress(event: listener.QueryProgressEvent) -> None

Writes the custom metrics from the DQMetricsObserver to the output destination.

Arguments:

  • event - A QueryProgressEvent with details about the last processed micro-batch

onQueryIdle​

def onQueryIdle(event: listener.QueryIdleEvent) -> None

Writes a message to the standard output logs when a streaming query is idle.

Arguments:

  • event - A QueryIdleEvent with details about the streaming query

onQueryTerminated​

def onQueryTerminated(event: listener.QueryTerminatedEvent) -> None

Writes a message to the standard output logs when a streaming query stops due to cancellation or failure.

Arguments:

  • event - A QueryTerminatedEvent with details about the streaming query