python - Load XGBoost from an s3 bucket

Question

I have an XGBoost model sitting in an AWS s3 bucket which I want to load. currently, I'm attempting to use s3fs to load the data, but I keep getting type errors:

from s3fs.core import S3FileSystem
import xgboost as xgb

fs = S3FileSystem()
bst = xgb.XGBClassifier()

with fs.open(f'{bucket}/{modelPath}', mode='rb') as f:
    bst.load_model(f.read())

I would include output, this I'm pretty sure this approach is completely wrong. How do I get this working?

(also, I think I would rather use boto)

question from:https://stackoverflow.com/questions/65894296/load-xgboost-from-an-s3-bucket

Answer 1

Provided I don't receive a better answer, one good approach would be to download the file onto the lambda functions storage, then load it with bst.load_model('path_to_model'). based on this, each lambda function is allocated 500 MB of non persistent storage, which actually sits around for subsequent invocations of the same lambda function (until the lambda function goes cold).