☁️ Google Cloud Storage#

Important

To use this integration you should install tango with the “gs” extra (e.g. pip install tango[gs]) or just install the gcsfs library after the fact (e.g. pip install gcsfs).

Components for Tango integration with GS.

Reference#

class tango.integrations.gs.GSWorkspace(workspace, project=None, credentials=None)[source]#

This is a Workspace that stores step artifacts on Google Cloud Storage.

Tip

Registered as a Workspace under the name “gs”.

Parameters:

  • workspace (str) – The name or ID of the Google Cloud bucket to use.

  • project (Optional[str], default: None) – The Google project ID. This is required for the datastore. If not provided, it will be inferred from the Google cloud credentials.

Important

Credentials can be provided in the following ways:

  • Using the credentials keyword argument:

    • You can specify the path to the credentials json file.

    • You can specify the google.oauth2.credentials.Credentials() object.

    • You can specify the json string of credentials dict.

  • Using the default credentials: You can use your default google cloud credentials by running gcloud auth application-default login. If you are using GSWorkspace with BeakerExecutor, you will need to set the environment variable GOOGLE_TOKEN to the credentials json file. The default location is usually ~/.config/gcloud/application_default_credentials.json.

class tango.integrations.gs.GSStepCache(bucket_name, client=None)[source]#

This is a StepCache that’s used by GSWorkspace. It stores the results of steps on Google cloud buckets as blobs.

It also keeps a limited in-memory cache as well as a local backup on disk, so fetching a step’s resulting subsequent times should be fast.

Tip

Registered as a StepCache under the name “gs”.

Parameters:

  • bucket_name (str) – The name of the google cloud bucket to use.

  • client (Optional[GSClient], default: None) – The google cloud storage client to use.