MDE Researcher

AbstractMdeResearchBuilder

Base class for Experiment Builders.

BaseSplitElement

Dataclass with data params for experiment calculations

MdeAlphaExperiment

Dataclass for I type error calculations.

MdeBetaExperiment

Dataclass for II type error calculations.

MdeResearchBuilder

Calculates I and II type errors for different group sizes and injects.

MultipleSplitBuilder

Columns with splits and injects will be added

MdeParams

MDE research experiment parameters class.

Abstract MDE Experiment

class abacus.mde_researcher.AbstractMdeResearchBuilder(guests: DataFrame, abtest_params: ABTestParams, experiment_params: MdeParams)[source]

Base class for Experiment Builders.

Attributes:
experiment_params
group_sizes
abacus.mde_researcher.AbstractMdeResearchBuilder._build_group_sizes(self)

Build list of groups sizes tuples.

Returns:

List of groups sizes pairs.

Return type:

List[int]

Experiment Structures

class abacus.mde_researcher.BaseSplitElement(group_sizes: tuple, split_number: int, control_group_size: int = FieldInfo(default=PydanticUndefined, extra={'init': False}), target_group_size: int = FieldInfo(default=PydanticUndefined, extra={'init': False}))[source]

Dataclass with data params for experiment calculations

Parameters:
  • group_sizes (tuple) – tuple with group sizes. Should have control group size on the 0 index position and target group size on the 1 index position

  • split_number (int) – params with number of split

class abacus.mde_researcher.MdeAlphaExperiment(group_sizes: tuple, split_number: int, control_group_size: int = FieldInfo(default=PydanticUndefined, extra={'init': False}), target_group_size: int = FieldInfo(default=PydanticUndefined, extra={'init': False}), metric_name: str = FieldInfo(default=PydanticUndefined, extra={'init': False}))[source]

Dataclass for I type error calculations.

class abacus.mde_researcher.MdeBetaExperiment(group_sizes: tuple, split_number: int, control_group_size: int = FieldInfo(default=PydanticUndefined, extra={'init': False}), target_group_size: int = FieldInfo(default=PydanticUndefined, extra={'init': False}), metric_name: str = FieldInfo(default=PydanticUndefined, extra={'init': False}), inject: float = FieldInfo(default=PydanticUndefined, extra={'init': False}))[source]

Dataclass for II type error calculations.

MDE Research Builder

class abacus.mde_researcher.MdeResearchBuilder(guests: DataFrame, abtest_params: ABTestParams, experiment_params: MdeParams, stratification_params: SplitBuilderParams)[source]

Calculates I and II type errors for different group sizes and injects.

Attributes:
experiment_params
group_sizes

Methods

calc_alpha(guests[, is_splitted])

Calculates I type error.

collect([fill_with_default])

Calculates I and II types error using prepilot parameters.

abacus.mde_researcher.MdeResearchBuilder.calc_alpha(self, guests: DataFrame, is_splitted: bool = False) DataFrame

Calculates I type error.

Parameters:
  • guests (pandas.DataFrame) – Dataframe with guests.

  • is_splitted (bool) – If False guests must contain splits for calculation. Otherwise splits will be compute for guests.

Returns:

Pandas DataFrame with I type error.

Return type:

pandas.DataFrame

abacus.mde_researcher.MdeResearchBuilder.collect(self, fill_with_default: bool = True) DataFrame

Calculates I and II types error using prepilot parameters.

Parameters:

fill_with_default (bool) – Fill calculated vaules with defaults.

Returns:

Pandas DataFrames with aggregated results of experiment.

Return type:

pandas.DataFrame

Multiple Split Builder

class abacus.mde_researcher.MultipleSplitBuilder(guests: DataFrame, metrics_names: List[str], injects: List[float], group_sizes: List[int], stratification_params: SplitBuilderParams, iterations_number: int = 10)[source]

Columns with splits and injects will be added

Methods

calc_injected_metrics(guests_for_injects)

Calculates injected metrics for guests df.

collect()

Calculate multiple split with stratification.

abacus.mde_researcher.MultipleSplitBuilder._build_splits_grid(self)
abacus.mde_researcher.MultipleSplitBuilder._update_strat_params(self)

Update stratification columns, because of columns names duplicated problem.

abacus.mde_researcher.MultipleSplitBuilder._build_split(self, guests_with_strata: DataFrame, control_group_size: int, target_group_size: int, split_number: int = 1)

Calculate one split with stratification.

Parameters:
  • guests_with_strata (pandas.DataFrame) – Dataframe with stratas.

  • control_group_size (int) – Control group size.

  • target_group_size (int) – Target group size.

  • split_number (int, default = 1) – Number of split. Uses as suffix for new column.

Returns:

pandas DataFrame with split.

Return type:

pandas.DataFrame

abacus.mde_researcher.MultipleSplitBuilder.calc_injected_metrics(self, guests_for_injects: DataFrame) DataFrame

Calculates injected metrics for guests df.

Parameters:

guests_for_injects (pandas.DataFrame) – Dataframe with metrics columns.

Returns:

Dataframe with injected metrics columns.

Return type:

pandas.DataFrame

abacus.mde_researcher.MultipleSplitBuilder.collect(self) DataFrame

Calculate multiple split with stratification.

Returns:

Pandas DataFrame with split columns.

Return type:

pandas.DataFrame

Params

class abacus.mde_researcher.MdeParams(metrics_names: ~typing.List[str], injects: ~typing.List[float], min_group_size: int, max_group_size: int, step: int, variance_reduction: ~typing.Callable[[~abacus.auto_ab.abtest.ABTest], ~abacus.auto_ab.abtest.ABTest] | None = None, use_buckets: bool = False, transformations: ~typing.Any = None, stat_test: ~typing.Callable[[~abacus.auto_ab.abtest.ABTest], ~typing.Dict[str, int | float]] = <function ABTest.test_boot_confint>, iterations_number: int = 10, max_beta_score: float = 0.2, min_beta_score: float = 0.05)[source]

MDE research experiment parameters class.

Parameters:
  • metrics_names – Metrics which will be compare in experiments.

  • injects – Injects represent MDE values.

  • min_group_size – Minimal value of groups sizes.

  • max_group_size – Maximal value of groups sizes.

  • step – Spacing between min_group_size and max_group_size.

  • variance_reduction – ABTest methods for variance reduction.

  • use_buckets – Use bucketize method.

  • transformations – Pipeline of experiment. Will be calulted in __post_init__.

  • stat_test – Statistical test type.

  • iterations_number – Count of splits for each element in group_sizes.

  • max_beta_score – Maximum level of II type error.

  • min_beta_score – Minimum level of II type error.

Attributes:
transformations
variance_reduction

Methods

stat_test()

Performs bootstrap confidence interval and zero statistical significance.

groups_sizes_validator

stat_test_validator

variance_reduction_validator