The Virtualitics AI Platform’s (VAIP) Scenario Planning Tool (SPT) allows users to explore a machine learning model’s new prediction in the case that one or more of an instance’s features changed.

This alternate “scenario” is analyzed by the VAIP to:

• Determine whether the scenario is likely to occur
• Identify the difference in the model output
• Plot the new point in an interactive graph tool
• Display the waterfall plot for the new scenario
  
  

Using the Scenario Planning Tool

The example code below incorporates previously-created Model and Explainer Assets and enables the XAIDashboard element to display on a Page.

from virtualitics_sdk import StoreInterface, XAIDashboard, ...

...

# Get previously created assets
store_interface = StoreInterface(**flow_metadata)
xgb = store_interface.get_model(label="example model", name="xgb model")
data_train = store_interface.get_dataset(label="example train data", name="explainer training set")
data_test = store_interface.get_dataset(label="example test data", name="explainer test set")
explainer = store_interface.get_explainer(label="example explainer", name="explainer")

# Create bounds
data_train_df = data_train.get_object()
train_mins = data_train_df.min()
train_maxs = data_train_df.max()
bounds = {key: [train_mins[key], train_maxs[key]] for key in data_train_df.columns}

# Define XAI Dashboard
xai_dash = XAIDashboard( model=xgb, # The model performing predictions on the data points (e.g., XGBoost model) 
explainer=explainer, # The explainer used for the model (e.g., SHAP explainer)
plot_data=data_test, # The data to show on the plot (e.g., test dataset)
x_axis_key="Credit Score", # The key for the X-axis on this plot (e.g., credit score) 
y_axis_key="Probability of Default in next 30 Days (Predicted)", # The key for the Y-axis on this plot (e.g., predicted default probability)
pred_column="Probability of Default in next 30 Days (Predicted)", # The column where the prediction lies in the data (e.g., predicted default probability column) 
bounds=bounds, # Optional bounds for the data, represented as a dict with column names as keys and list of domain values as the value
description=XAIDashboard.xai_dashboard_description(), # Optional description of the element
expected_title="Average Probability of Default in next 30 days.", # Title for expected value in the waterfall plot (e.g., average probability title)
predicted_title="Predicted Probability of Default in next 30 days.",# Title for predicted value in the waterfall plot (e.g., predicted probability title)
train_data=data_train_ds, # Optional separate training dataset for the explainer (e.g., training dataset)
encoding=DataEncoding.ONE_HOT, # Encoding for categorical values (e.g., one-hot encoding)

# Add XAI Dashboard to the page
page.add_content_to_section(xai_dash, "Interactive Scenario Planning")

What to Expect

The previous code creates the following Scatter Plot:

The user can then select a point to visualize it in the point editor on the right. The selected point is increased in size and the other points have decreased saturation.

The user can then edit attributes of this instance. Here, the credit score was changed from 544 to 800 to explore how this person’s probability of defaulting would change if they had a higher credit score.

Once the “Predict” button is pressed, the point is moved to its new location on the plot, showing a decreased probability of defaulting.

If the user clicks the cog wheel on the top right of the point editor panel to, they can enable rendering of a waterfall plot. The waterfall plot appears directly below the SPT and shows the explanation for the newly created scenario.

To the left, the user will see an additional Infographic that provides additional information about the scenario we created. Here, the user can see the model’s output for the scenario, the difference from the original point, the number of attributes that we edited that fell out of range of realistic data, and how likely the scenario we created is.

In this case, we can see that our scenario is unlikely. This is because we significantly increased the person’s credit score, but did not change any other input factors. It's likely that someone with low credit would have other negative attributes, like late payments or a shorter account lifetime.
Luckily, the SPT notifies us that this scenario where the credit score was increased without other factors changing is unlikely to occur.

Additional Details

For more details and information about the Scenario Planning Tool, see the Virtualitics SDK.