• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

G-Hung/model-productization_article: This is repo to demonstrate how to convert ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称:

G-Hung/model-productization_article

开源软件地址:

https://github.com/G-Hung/model-productization_article

开源编程语言:

Jupyter Notebook 85.8%

开源软件介绍:

My Medium articles for model productization

From Jupyter Notebook to Scripts : Article link

From Scripts to Prediction API : Article link

From Jupyter Notebook to Scripts

Back to top

My article in Medium with code in this repo demonstrates how to convert Jupyter Notebook to scripts together with some engineering practices, we only surfaced with the basics and want to show the benefits quickly!

High level topics

a. Why scripts instead of Jupyter notebook
b. Conversion from ipynb to .py
c. Make the scripts configurable [Click]
d. Include logging [logging]
e. Make sure the local environment is the same [Conda env]
f. Include unit test and basic CI [pytest, GitHub Action]
g. Autoformat the script style [black, isort]

Code structure tree, hope this can help you to understand how the codes evolve

.
├── README.md
├── __init__.py
├── .github/workflows         [f]
├── autoformat.sh             [g]
├── data
│   ├── predict.csv           [b]
│   ├── test.csv              [b]
│   ├── train.csv             [b]
│   └── winequality.csv
├── log
│   ├── etl.log               [d]
│   ├── predict.log           [d]
│   └── train.log             [d]
├── model
│   └── model.pkl             [b]
├── notebook
│   └── prediction-of-quality-of-wine.ipynb [a]
├── requirement.txt           [e]
└── scripts
    ├── config.yml            [c]
    ├── etl.py                [b, c]
    ├── predict.py            [b, c]
    ├── test_train.py         [f]
    ├── test_utility.py       [f]
    ├── train.py              [b, c]
    └── utility.py

Setup

  1. Git Clone the repo
git clone https://github.com/G-Hung/model-productization_article.git
  1. Go to project root folder
cd model-productization_article
  1. Setup conda env in terminal
conda create - name YOU_CHANGE_THIS python=3.7 -y

conda activate YOU_CHANGE_THIS

pip install –r requirements.txt
  1. Run the code in terminal
python3 ./scripts/etl.py
python3 ./scripts/train.py
python3 ./scripts/predict.py

We should expect nothing popup except files inside log/ and model/ are updated! In few seconds, the scripts finish the processes of ETL, training, evaluation and prediction!

  1. To run unit test in terminal
pytest
  1. To run autoformat.sh in terminal
# If you get permission error, you can try
# chmod +rx autoformat.sh

./autoformat.sh
  1. After usage
conda deactivate
conda remove –name YOU_CHANGE_THIS –all

From Scripts to Prediction API

Back to top

This article, we discuss how to utilize the models we have last time to create a prediction API using Fast API.

High level topics

a. Update conda env [requirements.txt]
b. Brainstorm pseudocode and convert to code [FastAPI, uvicorn]
c. Utilize API [cURL, requests, Postman]
d. Talk about Auto-generated documents by FastAPI
e. Something about pytest [parallel, parameterized, -v]

Setup

You can reuse the steps above for Git Clone, Conda env, autoformat.sh or pytest. The only different thing is step 4, instead of running the script, we will launch a API server!

Similar to last time, we include the file tree below and annotate the related files

.
├── README.md
├── autoformat.sh
├── data
│   ├── predict.csv
│   ├── test.csv
│   ├── train.csv
│   └── winequality.csv
├── log
│   ├── etl.log
│   ├── predict.log
│   └── train.log
├── model
│   ├── gb_model.pkl
│   └── rf_model.pkl
├── notebook
│   ├── prediction-of-quality-of-wine.ipynb
│   └── prediction_API_test.ipynb              [c]
├── prediction_api
│   ├── __init__.py
│   ├── api_utility.py                         [b]
│   ├── main.py                                [b]
│   ├── mock_data.py                           [e]
│   ├── test_api_utility.py                    [e]
│   └── test_main.py                           [e]
├── requirements.txt                           [a]
└── scripts
    ├── config.yml
    ├── etl.py
    ├── predict.py
    ├── test_train.py
    ├── test_utility.py
    ├── train.py
    └── utility.py

To launch the API server, set this up the environment first:

conda create - name YOU_CHANGE_THIS python=3.7 -y
conda activate YOU_CHANGE_THIS
pip install –r requirements.txt

Then run:

uvicorn prediction_api.main:app --reload



鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap