simulstream.metrics.scorers.quality.mwersegmenter

Classes

MWERSegmenterBasedQualityScorer(args)	Abstract base class for scorers that require aligned system outputs and references through MWER Segmenter alignment.
ResegmentedQualityScoringSample(audio_name, ...)	A sample containing realigned hypotheses and references.

class simulstream.metrics.scorers.quality.mwersegmenter.MWERSegmenterBasedQualityScorer(args)

Abstract base class for scorers that require aligned system outputs and references through MWER Segmenter alignment.

This class wraps a quality scorer and applies the MWER Segmenter alignment by “Effects of automatic alignment on speech translation metrics” to hypotheses before scoring.

Subclasses must implement _do_score(), which receives ResegmentedQualityScoringSample instances, where output and references are aligned.

Example

>>> class CustomQualityScorer(MWERSegmenterBasedQualityScorer):
...     def _do_score(self, samples):
...         # Compute a custom quality score
...         return ...

requires_reference() → bool

Indicate whether this scorer requires reference translations.

Returns:

True if references are required, False otherwise.

Return type:

bool

score(samples: List[QualityScoringSample]) → float

Compute a quality score over a list of samples.

Parameters:

samples (List[QualityScoringSample]) – Samples to be evaluated.

Returns:

The computed quality score.

Return type:

float

class simulstream.metrics.scorers.quality.mwersegmenter.ResegmentedQualityScoringSample(audio_name: str, hypothesis: List[str], reference: List[str], source: List[str] | None = None)

A sample containing realigned hypotheses and references.

audio_name

The identifier of the audio file.

Type:

str

hypothesis

Hypothesis lines after realignment.

Type:

List[str]

reference

Reference lines aligned to the hypothesis.

Type:

List[str]

source

Source text (if available).

Type:

Optional[List[str]]