My article in Medium with code in this repo demonstrates how to convert Jupyter Notebook to scripts together with some engineering practices, we only surfaced with the basics and want to show the benefits quickly!
High level topics
a. Why scripts instead of Jupyter notebook
b. Conversion from ipynb to .py
c. Make the scripts configurable [Click]
d. Include logging [logging]
e. Make sure the local environment is the same [Conda env]
f. Include unit test and basic CI [pytest, GitHub Action]
g. Autoformat the script style [black, isort]
Code structure tree, hope this can help you to understand how the codes evolve
We should expect nothing popup except files inside log/ and model/ are updated! In few seconds, the scripts finish the processes of ETL, training, evaluation and prediction!
To run unit test in terminal
pytest
To run autoformat.sh in terminal
# If you get permission error, you can try
# chmod +rx autoformat.sh
./autoformat.sh
This article, we discuss how to utilize the models we have last time to create a prediction API using Fast API.
High level topics
a. Update conda env [requirements.txt]
b. Brainstorm pseudocode and convert to code [FastAPI, uvicorn]
c. Utilize API [cURL, requests, Postman]
d. Talk about Auto-generated documents by FastAPI
e. Something about pytest [parallel, parameterized, -v]
Setup
You can reuse the steps above for Git Clone, Conda env, autoformat.sh or pytest. The only different thing is step 4, instead of running the script, we will launch a API server!
Similar to last time, we include the file tree below and annotate the related files
请发表评论