Model Building Checklist

#	Category	Task	Examples
1	Problem definition	What is the problem to solve
2		What is the measurable goal
3		What do you want to predict
4	Environment	Choose an IDE	Local Jupyter notebook (i.e. iPython), SageMaker Studio, SageMaker Console + SageMaker API, PyCharm, Spyder
5		Choose an ML framework	Tensorflow, PyTorch, Keras, MXNet, Scikit-learn, Spark
6		Choose a source code repository	GitHub, Bitbucket, Sourceforge, SVN
7		Choose a programming language	Python, R, JavaScript
8		Python, R, JavaScript	Classification, Regression
9		Determine type of ML problem	Supervised, unsupervised, semi-supervised, reinforcement
10	Select data	Identify internal data sources	Google Analytics, Clickstream
11		Identify public datasets	Kaggle, UCI ML repository, ImageNet, OpenData on AWS
12		Kaggle, UCI ML repository, ImageNet, OpenData on AWS
13		Label the data (i.e. ground truth)	Amazon MTurk, Labelbox, Scale AI
14		Combine data from various sources to a single data store	Custom code
15	Proprocess data	Explore data	Jupyter notebook, matplotlib, scikit-learn
16		Format the data	SageMaker Batch Transform
18	Clean the data	SageMaker Batch Transform
18		Remove bias from the data	Pandas, Numpy, Tensorflow transform
19		Sample the data	Pandas, Numpy, Tensorflow transform
20		Obtain embeddings	Word2vec and FastText (word embeddings),
21	Transform data (i.e. feature engineering)	Automatically extract features from raw data (reduce dimensionality of unmanageably high dimensional data, automated)	For tablular data (PCA, unsupervised clustering methods), for image data (line/edge detection, DSP methods)
22		Manually construct new features (combine or decompose existing ones)
23		Select useful features (from many to few, the ones with higher correlation with dependent variable, manual or automated)	Feature selection algorithms select optimal features using trial and error by creating and evaluating models automatically. Some methods (ex: PCA, stepwise regression, LASSO, ridge regresion) automatically perform feature selection as part of the model construction process
24	Finalize data	Run data pre-processing jobs	SageMaker processing, Control-M
25		Store data in a single data store	S3
26		Partition processed data into training and validation sets	80% training, 20% validation
27	Model training, tuning and evaluation	Select training algorithm	Use model-building library (scikit-learn, tensorflow) SageMaker built-in algorithms (17), purchase algorithm from AWS marketplace
28		Write training code	Custom code leveraging ML libraries
29		Containerize training code	Docker image
30		Deploy training code	ECR
31		Determine compute resources needed to train the model	Single general-purpose instance to distributed cluster of GPU instances
32		Debug the model during training (inspect training parameters and data)	SageMaker Debugger
33		Note time taken to train the model
34		Store the trained model and other outputs	S3
35		Create multiple versions of the model (production variants)	SageMaker Experiments
36		Send inference requests to the local model	Jupyter notebook in SM notebook instance and AWS SDK for Python (Boto) or SageMaker's high-level Python library
37		Evaluate model on historical datasets (offline testing)	SageMaker batch transform
38		Determine deployment feasibility based on offline metrics
39	Model deployment	Determine application and environment constraints (ex: latency requirements for inference)
40		Re-engineer the model to make it production ready and integrate it with your application	N/A if using SageMaker hosting services (model deployment is decoupled from application code)
41		Deploy the trained model to a persistent HTTPS endpoint (for real time predictions, one at a time)	SageMaker hosting services, your own deployment target (you need to know the algorithm-specific format of the model artifacts that were generaed by model training)
42		Make that endpoint accessible outside your application context	Amazon API Gateway and AWS Lambda
43		Deploy multiple models (production variants) to a single endpoint	SageMaker deploy
44		A/B test the production variants by sending X% traffic to the updated model
45		Ramp up 100% traffic to the updated model
46		Monitor the model and set alerts for when there are deviations in model quality (ex: data drift and anomalies)	SageMaker Model Monitor
47		Modify the endpoint (change model artifacts, change inference code, add new model variants, update compute configurations, change traffic distribution among model variants)	Provide a new endpoint configuration, ask SageMaker to implement it (no downtime) and delete the old endpoint config
48		Pipe live data back to the the storage repository for model retraining
49		Determine periodicity of model retraining
50		Create new version of the model, deploy it as a production variant, A/B test it against the live model, slowly ramp up traffic to the updated model