dctools.data.connection.connection_manager.S3WasabiManager
- class dctools.data.connection.connection_manager.S3WasabiManager(connect_config, call_list_files=True, batch_size=64)
Specific S3 Manager for Wasabi (inheriting from S3Manager).
- Parameters:
connect_config (BaseConnectionConfig | Namespace)
call_list_files (bool)
batch_size (int | None)
- __init__(connect_config, call_list_files=True, batch_size=64)
- Parameters:
connect_config (BaseConnectionConfig | Namespace)
call_list_files (bool)
batch_size (int | None)
Methods
__init__(connect_config[, call_list_files, ...])adjust_full_day(date_start, date_end)Adjust date_end to cover a full day if dates are the same at midnight.
download_file(remote_path, local_path)Download a file from the remote source to the local path.
estimate_resolution(ds, coord_system)Estimate resolution from dataset based on coordinates.
extract_global_metadata()Extract global metadata (common to all files) from a single file.
extract_metadata(path)Extract metadata combining global/file-specific info.
extract_metadata_worker(path, ...[, argo_index])Extract metadata combining global/file-specific info.
get_config_clean_copy()Return a clean copy of the configuration.
get_global_metadata()Get global metadata for all files in the connection manager.
list_files()List files matching pattern.
list_files_with_metadata()Version with integrated Dask client and optimized configuration.
list_first_n_files(fs, remote_path[, n, pattern])Use fsspec filesystem to quickly list up to n files recursively.
open(path[, mode])Open a file, prioritizing local then remote access.
open_local(local_path)Open a file locally if it exists.
open_remote(path[, mode])Open a file remotely from an S3 bucket.
set_global_metadata(global_metadata)Sets the global metadata for the connection manager.
supports(path)Checks if path is supported.
- open_remote(path, mode='rb')
Open a file remotely from an S3 bucket.
- Parameters:
path (str) – Remote path of the file.
mode (str) – Mode to open the file (default is “rb”).
- Returns:
Opened dataset, or None if remote opening is not supported.
- Return type:
Optional[xr.Dataset]
- classmethod supports(path)
Checks if path is supported.
- Parameters:
path (str)
- Return type:
bool