dctools.data.connection.connection_manager.GlonetManager

class dctools.data.connection.connection_manager.GlonetManager(connect_config, call_list_files=True, batch_size=64)

Manager for Glonet (remote files over HTTPS).

Parameters:
__init__(connect_config, call_list_files=True, batch_size=64)
Parameters:

Methods

__init__(connect_config[, call_list_files, ...])

adjust_full_day(date_start, date_end)

Adjust date_end to cover a full day if dates are the same at midnight.

download_file(remote_path, local_path)

Download a file from the remote source to the local path.

estimate_resolution(ds, coord_system)

Estimate resolution from dataset based on coordinates.

extract_global_metadata()

Extract global metadata (common to all files) from a single file.

extract_metadata(path)

Extract metadata combining global/file-specific info.

extract_metadata_worker(path, ...[, argo_index])

Extract metadata combining global/file-specific info.

get_config_clean_copy()

Return a clean copy of the configuration.

get_global_metadata()

Get global metadata for all files in the connection manager.

list_files()

List files matching pattern.

list_files_with_metadata()

Version with integrated Dask client and optimized configuration.

open(path[, mode])

Open a Glonet file, preferring local cache over remote.

open_local(local_path)

Open a file locally if it exists.

open_remote(path[, mode])

Open a file remotely from an S3 bucket.

set_global_metadata(global_metadata)

Sets the global metadata for the connection manager.

supports(path)

Check if the path is an HTTPS URL (generic).
list_files()

List files matching pattern.

Args:

Returns:

List of file paths.

Return type:

List[str]

open(path, mode='rb')

Open a Glonet file, preferring local cache over remote.

When the prediction prefetch step has downloaded the zarr to local disk, the path will be a local directory. Open it directly to avoid any HTTP/S3 traffic on the worker.

Parameters:

  • path (str)

  • mode (str)

Return type:

xarray.Dataset | None

open_remote(path, mode='rb')

Open a file remotely from an S3 bucket.

Parameters:

  • path (str) – Remote path of the file.

  • mode (str) – Mode to open the file (default is “rb”).

Returns:

Opened dataset, or None if remote opening is not supported.

Return type:

Optional[xr.Dataset]

classmethod supports(path)

Check if the path is an HTTPS URL (generic).

Parameters:

path (str)

Return type:

bool