When is this applicable?

Flows often consist of long-running operations that take time to execute. As a result, pages in the Virtualitics AI Platform will not always load instantaneously. To provide feedback to a users of progress through backend processes, we recommend displaying progress updates using the functionality of StoreInterface.

Method: predict_backend.store.store_interface.StoreInterface.update_progress()

How-To

Imports

To display progress updates, your code must contain (at a minimum) the following import statement:

from predict_backend.store.store_interface import StoreInterface

Simple Updates

Within the run() function in Steps (in Flows), developers may hardcode progress updates so that an end user has a rough idea of how much longer a step will take to finish running.

from predict_backend.flow.step import Step
from predict_backend.store.store_interface import StoreInterface
class ArbitraryStep(Step):
   def run(self, flow_metadata):
       store_interface = StoreInterface(**flow_metadata)
       page = store_interface.get_page()
   
       # Updating progress (progress to completion is 10/100, 
       # "Loading Dataset" is the message displayed)
       store_interface.update_progress(10, "Loading Dataset")
       data = load_data(. . .)
       .
       . (Manipulate the dataset)
       .
       
       # Update page
       store_interface.update_page()

Advanced Use

When writing Flows, it may be convenient to write helper functions that are called within Steps (instead of run() being very long). While making progress updates in the run() function of a Step class instance after a helper function is called is simple to implement (as shown above), developers may want to incorporate small progress updates.

For example, let’s say you have a step called PerformModeling that accesses an input dataset from store_interface, (1) performs a series of preprocessing and (2) feature engineering steps, and (3) trains an ML model. Each part of PerformModeling (preprocessing, feature engineering, and model training) can take a long time to run and is comprised of many lines of code, as you are performing complex operations on your dataset. However, you may want to provide more frequent updates to alert an end user of progress within each part.  

Let’s assume that your helper functions look something like this:

import pandas as pd
from sklearn.linear_model import LogisticRegression
def perform_preprocessing(data: pd.DataFrame):
   .
   .
   .
   return data
   
def create_features(data: pd.DataFrame):
   .
   .
   .
   return data
   
def train_model(data: pd.DataFrame, target_column: str = "y"):
   model = LogisticRegression()
   .
   .
   .
   return model, data

If you are only updating the progress bar within PerformModeling.run(), PerformModeling.run() may look like this:

from predict_backend.flow.step import Step
from predict_backend.store.store_interface import StoreInterface
class PerformModeling(Step):
   def run(self, flow_metadata):
       store_interface = StoreInterface(**flow_metadata)
       page = store_interface.get_page()
   
       # Updating progress (progress to completion is 10/100, 
       # "Loading Dataset" is the message displayed)
       store_interface.update_progress(10, "Loading data")
       data = store_interface.get_element_value(
           "Data Upload Step", # replace with name of data upload step
           "Dataset Name" # replace with name of dataset
       )
       
       # Perform preprocessing
       store_interface.update_progress(20, "Performing preprocessing")
       data = perform_preprocessing(data)
       
       # Perform feature engineering
       store_interface.update_progress(40, "Performing feature engineering")
       data = create_features(data)
       
       # Train model
       store_interface.update_progress(65, "Training model")
       model, data = train_model(data)
       
       # Save model
       store_interface.update_progress(90, "Saving model")
       
       # Update page
       store_interface.update_progress(95, "Updating page")
       store_interface.update_page()

However, there’s a lot going on in each of the helper functions. You may make updates to the progress bar at shorter intervals by passing store_interface to each helper function. Your helper functions and PerformModeling.run() may look like this instead: 

from predict_backend.flow.step import Step
from predict_backend.store.store_interface import StoreInterface
import pandas as pd
from sklearn.linear_model import LogisticRegression
def perform_preprocessing(data: pd.DataFrame, store_interface: StoreInterface):
   # Convert the provided dataset into tidy format
   store_interface.update_progress(30, "Converting to tidy dataset")
   .
   .
   return data
   
def create_features(data: pd.DataFrame, store_interface: StoreInterface):
   # One-hot encode categorical variables
   store_interface.update_progress(45, "One-hot encoding categorical variables")
   .
   .
   # Generate text-based features
   store_interface.update_progress(55, "Creating text-based features")
   .
   .
   # Create binary variables indicating whether variable is missing or not
   # for all columns with missing data
   store_interface.update_progress(60, "Creating binary variables for columns with missing data")
   .
   .
   return data
   
def train_model(data: pd.DataFrame, target_column: str = "y", store_interface: StoreInterface):
   model = LogisticRegression()
   
   # Fit model on provided data
   store_interface.update_progress(70, "Fitting model")
   .
   .
   # Generate predictions for provided data
   store_interface.update_progress(85, "Generating predictions")
   .
   .
   return model, data
   
class PerformModeling(Step):
   def run(self, flow_metadata):
       store_interface = StoreInterface(**flow_metadata)
       page = store_interface.get_page()
   
       # Updating progress (progress to completion is 10/100, 
       # "Loading Dataset" is the message displayed)
       store_interface.update_progress(10, "Loading data")
       data = store_interface.get_element_value(
           "Data Upload Step",
           "Dataset Name"
       )
       
       # Perform preprocessing
       store_interface.update_progress(20, "Performing preprocessing")
       data = perform_preprocessing(data, store_interface)
       
       # Perform feature engineering
       store_interface.update_progress(40, "Performing feature engineering")
       data = create_features(data, store_interface)
       
       # Train model
       store_interface.update_progress(65, "Training model")
       model, data = train_model(data, store_interface)
       
       # Save model
       store_interface.update_progress(90, "Saving model")
       
       # Update page
       store_interface.update_progress(95, "Updating page")
       store_interface.update_page()

Note that we do not define a Flow instance in each of these code examples, as we are simply showing how to update the progress bar from StoreInterface. As a reminder, type hints do not work with Virtualitics AI Platform Flows and are only included here for the sake of clarity.

What to Expect (Validation)

When running a Flow, the progress bar will appear at the bottom of the browser window:

This example update was created with store_interface.update_progress(25, “Loading model”).

If you have updated the progress bar multiple times, you will see something like this as your code runs:

In this example, the developer called store_interface.update_progress() multiple times within a step.

Additional Details

Make sure that the fractions passed to store_interface increase monotonically. While doing otherwise will not break your Flow, it would result in the position of the progress bar moving backward (and potentially confuse an end user).