While you can develop your App to connect to a database using connectors, you can also add interactive DataSource Elements to your App that enable users to upload files or data.
Primary Element: from virtualitics_sdk import DataSource
Data types that can be uploaded include:
- Amazon S3
- SQL
- CSV
- XLSX
Adding a File Upload
To add a File Upload using a DataSource Element, ensure to include the corresponding import, create a Step, and add the following code:
# Imports
from virtualitics_sdk import DataSource, ...
. . .
# Example usage
class DataUploadStep(Step):
def run(self, flow_metadata):
. . .
data_source = DataSource(
title="IRIS Dataset",
options=["csv"],
description="Put your data here",
required=False,
)
data_card = Card(title="Upload", content=[data_source])
page.add_card_to_section(card=data_card, section_title="Upload your data")
This creates a File Upload button with a CSV data type requirement.
For more details on DataSource Elements, see the Virtualitics SDK.
Getting a Dataframe from a DataSource Element
Once a user has uploaded their data using a DataSource Element, users can reference that uploaded data in subsequent Steps from the StoreInterface, allowing you to use Python packages like pandas to manipulate your data as needed.
Once you’ve added in a file upload Step using the DataSource Element as shown above, within subsequent Steps where you want to access the uploaded dataset add the following code:
import pandas as pd
...
class DataDisplayStep(Step):
def run(self, flow_metadata):
store_interface = StoreInterface(**flow_metadata)
# Get the uploaded csv file from DataUploadStep as a pandas dataframe
pandas_df = store_interface.get_element_value(step_name=data_upload_step.name, elem_title="IRIS Dataset")
# Manipulate your dataframe using pandas
pandas_df['variety'] = pandas_df['variety'].astype('category')
pandas_df['above_avg_sepal_length'] = pandas_df['sepal.length'] > pandas_df['sepal.length'].mean()
...
Where the DataUploadStep is defined at the bottom of the file as:
data_upload_content = Section(title="Upload your data", content=[])
data_upload_page = Page(title="Data Upload Page", sections=[data_upload_content])
# Define the DataUploadStep
data_upload_step = DataUploadStep(title="Data Upload", description="Upload the IRIS dataset.", parent="Data Ingestion", type=StepType.INPUT, page=data_upload_page)
What to Expect
If your dataset uploads successfully through the DataSource Element, you should see "Upload Complete" appear in the notification in the bottom corner.