Skip to content

Feature/back up object storage - #4

Open
josvandervelde wants to merge 7 commits into
mainfrom
feature/back-up-object-storage
Open

josvandervelde wants to merge 7 commits into
mainfrom
feature/back-up-object-storage

Conversation

@josvandervelde

@josvandervelde josvandervelde commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

See https://github.com/openml/infrastructure/pull/99 for some documentation

A new way of backing up the object storage. The old way was using the mc client, e..g,

nohup /backup/data/minio/mc/mc mirror --watch minio-openml/automlbenchmark2023 /backup/data/minio/2026/11/automlbenchmark2023 >> /backup/data/minio/logs/2026/11/automlbenchmark2023.log 2>&1 &

The new way uses boto3. Why?

  1. MinIO is deprecated, the mc client as well. We're already using boto3 in arff-to-pq and croissant, amongst others.
  2. We can now access MinIO (and SeaweedFS) on subpaths. E.g., openml.org/data, instead of data.openml.org. This is useful for backing up data: we can now directly access openml1/minio.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant