simulstream.metrics.scorers.quality

Functions

register_quality_scorer(name)

Decorator for registering a quality scorer class.

Classes

`QualityScorer`(args)	Abstract base class for all quality scorers.
`QualityScoringSample`(audio_name, hypothesis)	Data structure representing a single evaluation sample.

class simulstream.metrics.scorers.quality.QualityScorer(args: Namespace)

Abstract base class for all quality scorers.

A quality scorer evaluates system hypotheses against references and/or source sentences and returns a numerical score.

Subclasses must implement the abstract methods defined here and should be registered via register_quality_scorer().

Parameters:: args (argparse.Namespace) – Parsed command-line arguments.

abstractmethod classmethod add_arguments(parser: ArgumentParser) → None

Add scorer-specific arguments to the CLI parser.

Parameters:: parser (argparse.ArgumentParser) – The parser to extend.

abstractmethod requires_reference() → bool

Indicate whether this scorer requires reference translations.

Returns:: True if references are required, False otherwise.
Return type:: bool

abstractmethod requires_source() → bool

Indicate whether this scorer requires the source text.

Returns:: True if source sentences are required, False otherwise.
Return type:: bool

abstractmethod score(samples: List[QualityScoringSample]) → float

Compute a quality score over a list of samples.

Parameters:: samples (List[QualityScoringSample]) – Samples to be evaluated.
Returns:: The computed quality score.
Return type:: float

class simulstream.metrics.scorers.quality.QualityScoringSample(audio_name: str, hypothesis: str, reference: List[str] | None = None, source: List[str] | None = None)

Data structure representing a single evaluation sample.

audio_name

The identifier of the audio file.

Type:: str

hypothesis

The system-generated hypothesis text.

Type:: str

reference

One or more reference translations, or None if not required.

Type:: Optional[List[str]]

source

The source transcription or text, or None if not required.

Type:: Optional[List[str]]

simulstream.metrics.scorers.quality.register_quality_scorer(name)

Decorator for registering a quality scorer class.

Parameters:: name (str) – The unique identifier for the scorer.
Raises:: TypeError – If the decorated class is not a subclass of QualityScorer.

Example

>>> @register_quality_scorer("bleu")
... class BLEUScorer(QualityScorer):
...     ...