• Home
  • Free consultation
  • Learn ML
  • Blog
  • Past Projects
  • About Us
Machine Village
  • Home
  • Free consultation
  • Learn ML
  • Blog
  • Past Projects
  • About Us

Model Building Checklist

5/21/2020

0 Comments

 
# Category Task Examples
1 Problem definition What is the problem to solve
2 What is the measurable goal
3 What do you want to predict
4 Environment Choose an IDE Local Jupyter notebook (i.e. iPython), SageMaker Studio, SageMaker Console + SageMaker API, PyCharm, Spyder
5 Choose an ML framework Tensorflow, PyTorch, Keras, MXNet, Scikit-learn, Spark
6 Choose a source code repository GitHub, Bitbucket, Sourceforge, SVN
7 Choose a programming language Python, R, JavaScript
8 Python, R, JavaScript Classification, Regression
9 Determine type of ML problem Supervised, unsupervised, semi-supervised, reinforcement
10 Select data Identify internal data sources Google Analytics, Clickstream
11 Identify public datasets Kaggle, UCI ML repository, ImageNet, OpenData on AWS
12 Kaggle, UCI ML repository, ImageNet, OpenData on AWS
13 Label the data (i.e. ground truth) Amazon MTurk, Labelbox, Scale AI
14 Combine data from various sources to a single data store Custom code
15 Proprocess data Explore data Jupyter notebook, matplotlib, scikit-learn
16 Format the data SageMaker Batch Transform
18 Clean the data SageMaker Batch Transform
18 Remove bias from the data Pandas, Numpy, Tensorflow transform
19 Sample the data Pandas, Numpy, Tensorflow transform
20 Obtain embeddings Word2vec and FastText (word embeddings),
21 Transform data (i.e. feature engineering) Automatically extract features from raw data (reduce dimensionality of unmanageably high dimensional data, automated) For tablular data (PCA, unsupervised clustering methods), for image data (line/edge detection, DSP methods)
22 Manually construct new features (combine or decompose existing ones)
23 Select useful features (from many to few, the ones with higher correlation with dependent variable, manual or automated) Feature selection algorithms select optimal features using trial and error by creating and evaluating models automatically. Some methods (ex: PCA, stepwise regression, LASSO, ridge regresion) automatically perform feature selection as part of the model construction process
24 Finalize data Run data pre-processing jobs SageMaker processing, Control-M
25 Store data in a single data store S3
26 Partition processed data into training and validation sets 80% training, 20% validation
27 Model training, tuning and evaluation Select training algorithm Use model-building library (scikit-learn, tensorflow) SageMaker built-in algorithms (17), purchase algorithm from AWS marketplace
28 Write training code Custom code leveraging ML libraries
29 Containerize training code Docker image
30 Deploy training code ECR
31 Determine compute resources needed to train the model Single general-purpose instance to distributed cluster of GPU instances
32 Debug the model during training (inspect training parameters and data) SageMaker Debugger
33 Note time taken to train the model
34 Store the trained model and other outputs S3
35 Create multiple versions of the model (production variants) SageMaker Experiments
36 Send inference requests to the local model Jupyter notebook in SM notebook instance and AWS SDK for Python (Boto) or SageMaker's high-level Python library
37 Evaluate model on historical datasets (offline testing) SageMaker batch transform
38 Determine deployment feasibility based on offline metrics
39 Model deployment Determine application and environment constraints (ex: latency requirements for inference)
40 Re-engineer the model to make it production ready and integrate it with your application N/A if using SageMaker hosting services (model deployment is decoupled from application code)
41 Deploy the trained model to a persistent HTTPS endpoint (for real time predictions, one at a time) SageMaker hosting services, your own deployment target (you need to know the algorithm-specific format of the model artifacts that were generaed by model training)
42 Make that endpoint accessible outside your application context Amazon API Gateway and AWS Lambda
43 Deploy multiple models (production variants) to a single endpoint SageMaker deploy
44 A/B test the production variants by sending X% traffic to the updated model
45 Ramp up 100% traffic to the updated model
46 Monitor the model and set alerts for when there are deviations in model quality (ex: data drift and anomalies) SageMaker Model Monitor
47 Modify the endpoint (change model artifacts, change inference code, add new model variants, update compute configurations, change traffic distribution among model variants) Provide a new endpoint configuration, ask SageMaker to implement it (no downtime) and delete the old endpoint config
48 Pipe live data back to the the storage repository for model retraining
49 Determine periodicity of model retraining
50 Create new version of the model, deploy it as a production variant, A/B test it against the live model, slowly ramp up traffic to the updated model
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

  • Home
  • Free consultation
  • Learn ML
  • Blog
  • Past Projects
  • About Us