simulstream.metrics.scorers.quality.QualityScorer

class simulstream.metrics.scorers.quality.QualityScorer(args: Namespace)

Bases: object

Abstract base class for all quality scorers.

A quality scorer evaluates system hypotheses against references and/or source sentences and returns a numerical score.

Subclasses must implement the abstract methods defined here and should be registered via register_quality_scorer().

Parameters:

args (argparse.Namespace) – Parsed command-line arguments.

__init__(args: Namespace)

Methods

__init__(args)
add_arguments(parser)	Add scorer-specific arguments to the CLI parser.
requires_reference()	Indicate whether this scorer requires reference translations.
requires_source()	Indicate whether this scorer requires the source text.
score(samples)	Compute a quality score over a list of samples.

abstractmethod classmethod add_arguments(parser: ArgumentParser) → None

Add scorer-specific arguments to the CLI parser.

Parameters:

parser (argparse.ArgumentParser) – The parser to extend.

abstractmethod requires_reference() → bool

Indicate whether this scorer requires reference translations.

Returns:

True if references are required, False otherwise.

Return type:

bool

abstractmethod requires_source() → bool

Indicate whether this scorer requires the source text.

Returns:

True if source sentences are required, False otherwise.

Return type:

bool

abstractmethod score(samples: List[QualityScoringSample]) → float

Compute a quality score over a list of samples.

Parameters:

samples (List[QualityScoringSample]) – Samples to be evaluated.

Returns:

The computed quality score.

Return type:

float