# Usage: evaluation ## Within the `up` object All curves can be plotted using ``` up.plot(plot_type='qini') ``` Where `plot_type` can be any of the following values. In the formulaic representations * `qini`: typical Qini curve (see Radcliffe 2007), except we normalize by the total number of people in treatment. The typical definition is $$n\_{t,1} - n\_{c,1} N\_t/N\_c.$$ * `aqini`: adjusted Qini curve, calculated as $$n\_{t,1}/N\_t - n\_{c,1} n\_t/(n\_c N\_t).$$ * `cuplift`: cumulative uplift curve, calculated as $$n\_{t,1}/n\_t - n\_{c,1}/n\_c.$$ * `uplift`: typical uplift curve, calculated the same as cuplift but only returning the average value within the bin, rather than cumulatively. * `cgains`: cumulative gains curve (see Gutierrez, Gerardy 2016), defined as $$((n\_{t,1}/n\_t - n\_{c,1}/n\_c)\phi.$$ * `balance`: ratio of treatment group size to total group size within each bin, $$n\_t/(n\_c + n\_t).$$ Above, $$\phi$$ corresponds to the fraction of individuals targeted -- the x-axis of these curves. $$n$$ and $$N$$ correspond to counts up to $$\phi$$ (except for the uplift curve, which is only within the bin at the $$\phi$$ position) or within the entire group, respectively. The subscript $$t$$ indicates the treatment group, and $$c$$, the control. The subscript $$1$$ indicates the subset of the count for which individuals had a positive outcome. A number of scores are stored in both the `test_results_` and `train_results_` objects, containing scores calculated over the test set and train set, respectively. Namely, there are three important scores: * `Q`: unnormalized area between the qini curve and the random selection line. * `q1`: `Q`, normalized by the theoretical maximum value of `Q`. * `q2`: `Q`, normalized by the practical maximum value of `Q`. Each of these can be accesses as attributes of `test_results_` or `train_results_`. Either `_qini`, `_aqini`, or `_cgains` can be appended to obtain the same calculation for the qini curve, adjusted qini curve, or the cumulative gains curve, respectively. The score most unaffected by anomalous treatment/control ordering, without any bias to treatment or control (i.e. if you're looking at lift between two equally viable treatments) is the `q1_cgains` score, but if you are looking at a simple treatment vs. control situation, `q1_aqini` is preferred. Because this only really has meaning over an independent holdout \[test] set, the most valuable value to access, then, would likely be `up.test_results_.q1_aqini`. ``` up.test_results_.q1_aqini # Over training set. ``` Maximal curves can also be toggled by passing flags into `up.plot()`. * `show_theoretical_max` * `show_practical_max` * `show_no_dogs` * `show_random_selection` Each of these curves satisfies shows the maximally attainable curve given different assumptions about the underlying data. The `show_theoretical_max` curve corresponds to a sorting in which we assume that an individual is persuadable (uplift = 1) if and only if they respond in the treatment group (and the same reasoning applies to the control group, for sleeping dogs). The `show_practical_max` curve assumes that all individuals that have a positive outcome in the treatment group must also have a counterpart (relative to the proportion of individuals in the treatment and control group) in the control group that did not respond. This is a more conservative, realistic curve. The former can only be attained through overfitting, while the latter can only be attained under very generous circumstances. Within the package, we also calculate the `show_no_dogs` curve, which simply precludes the possibility of negative effects. The random selection line is shown by default, but the option to toggle it off is included in case you'd like to plot multiple plots on top of each other. The below code plots the practical max over the aqini curve of a model contained in the TransformedOutcome object `up`, then overlays the aqini curve of a second model contained in `up1`, also changing the line color. ``` ax = up.plot(show_practical_max=True, show_random_selection=False, label='Model 1') up1.plot(ax=ax, label='Model 2', color=[0.7,0,0]) ``` ### Error bars It is often useful to obtain error bars on your qini curves. We've implemented two ways to do this: 1. `up.shuffle_fit()`: Seeds the \`train\_test\_split\`, fit the model over the new training data, and evaluate on the new test data. Average these curves. 2. `up.noise_fit()`: Randomly shuffle the labels independently of the features and fit a model. This can help distinguish your evaluation curves from noise. ``` up.shuffle_fit() up.plot(plot_type='aqini', show_shuffle_fits=True) ``` Adjustments can also be made to the aesthetics of these curves by passing in dictionaries that pass down to plot elements. For example, \`shuffle\_band\_kwargs\` is a dictionary of kwargs that modifies the \`fill\_between\` shaded error bar region. ## With `UpliftEval` The `UpliftEval` class can also independently be used to apply the above evaluation visualizations and calculations. Note that the `up` object uses `UpliftEval` to generate the plots, so the `UpliftEval` class object for the train set and test set can be obtained in `up.train_results_` and `up.test_results_`, respectively. ``` from pylift.eval import UpliftEval upev = UpliftEval(treatment, outcome, predictions) upev.plot(plot_type='aqini') ``` It generally functions the same as the `up.plot()` function, except error bars cannot be obtained. Note that `UpliftEval` could still be used, however, to manually generate the curves that can be aggregated to make error bars. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.pylift.org/evaluation.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.