Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Don't upload unused files #1

Closed
wants to merge 4 commits into from

Conversation

ckadner
Copy link
Member

@ckadner ckadner commented Jan 25, 2023

Don't merge this PR!

This is just for the record to keep a "copy" of the unused model files in the v0.10.0 modelmesh-mino-examples and modelmesh-minio-dev-examples images

The files proposed (NOT) to be added in this PR are from @tedhtchang 's PR here kserve/modelmesh-serving#300

The repository does not contain those large files as they are either duplicates of other files or because they are not used for the ModelMesh FVT purposes (anymore).

The files which did not make it from Ted PR into this repository are the following:

  • excluded the biggest unused models:
    • fvt/onnx/onnx-mobilenet (14 MB)
    • fvt/onnx/onnx-pytorch-converted (4.5 MB)
    • fvt/onnx/onnx-test-model (38 MB)
    • fvt/tensorflow/tfmnist (78 MB)
    • fvt/tensorflow/tfmnistnew (39 MB)
  • removed any duplicate models in favor of doing COPYs in the Dockerfile:
    • fvt/keras (6.5 MB)
    • fvt/pytorch/pytorch-mar-dup (4.6 MB)
    • fvt/tensorflow/keras-mnist (6.5 MB)
    • fvt/tensorflow/keras-mnistnew (6.5 MB)
    • fvt/tensorflow/mnist-dup.savedmodel (39 MB)
    • fvt/tensorflow/mnist.savedmodel (39 MB)
    • tensorflow/mnist.savedmodel (39 MB)

This helped shaving off about 315 MB from the size of this repo -- before any of those files were ever committed, since Git would keep a history of the files and the repo would stay larger even if we deleted those file subsequently.

For the Tensorflow models in the first commit, there are more duplicates that are excluded here:

  • fvt/tensorflow/tfmnistnew/1/model.savedmodel has to be duplicated from tensorflow/mnist
  • fvt/tensorflow/tfmnist/2/model.savedmodel has to be duplicated from fvt/tensorflow/tfmnist/1/model.savedmodel

- fvt/tensorflow/tfmnistnew/1/model.savedmodel
  dupliacted from tensorflow/mnist
- fvt/tensorflow/tfmnist/2/model.savedmodel duplicate
  of fvt/tensorflow/tfmnist/1/model.savedmodel

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
@ckadner ckadner self-assigned this Jan 25, 2023
@ckadner ckadner marked this pull request as draft January 25, 2023 00:37
@ckadner
Copy link
Member Author

ckadner commented Jan 25, 2023

Hint to future self, pushing those large files failed repeatedly until setting a few git flags:

# increase some buffer sizes
git config http.postBuffer 50M
git config http.maxRequestBuffer 50M
git config core.compression 0

# cleanup some git internal caches
git gc

# enable some tracing
export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1

# repeat push with extra verbose debug output
git push -vvv

@ckadner ckadner closed this Jan 25, 2023
@ckadner ckadner changed the title [WIP] Upload unused files [WIP] Don't upload unused files Jan 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant