Tutorial: Datasets and Saving Strategies¶

The following demonstrates how to use the ep.saving_strategies to save data in different formats ep.DataSet to load the saved data.

We will use the loaded variables from the cdf file to showcase how to save them with different formats. We will add additional variables so that we have more to save.

In [1]:

Copied!





from datetime import datetime, timezone

from astropy import units as u

import el_paso as ep

ep.setup_logging()

extraction_infos = [
    ep.ExtractionInfo(
        result_key="Epoch",
        name_or_column="Epoch_Ele",
        unit=ep.units.cdf_epoch,
    ),
    ep.ExtractionInfo(
        result_key="FEDU",
        name_or_column="FEDU",
        unit=(u.cm**2 * u.s * u.sr * u.keV) ** (-1),
    ),
    ep.ExtractionInfo(
        result_key="xGEO",
        name_or_column="Position_Ele",
        unit=u.km,
    ),
]

start_time = datetime(2017, 7, 14, tzinfo=timezone.utc)
end_time = datetime(2017, 7, 14, 23, 59, 59, tzinfo=timezone.utc)

file_name_stem = "rbspa_rel04_ect-hope-pa-l3_YYYYMMDD_.{6}.cdf"

ep.download(
    start_time,
    end_time,
    save_path=".",
    download_url="https://spdf.gsfc.nasa.gov/pub/data/rbsp/rbspa/l3/ect/hope/pitchangle/rel04/YYYY/",
    file_name_stem=file_name_stem,
    file_cadence="daily",
    method="request",
    skip_existing=True,
)

variables = ep.extract_variables_from_files(
    start_time, end_time, "daily", data_path=".", file_name_stem=file_name_stem, extraction_infos=extraction_infos
)
variables
from datetime import datetime, timezone

from astropy import units as u

import el_paso as ep

ep.setup_logging()

extraction_infos = [
    ep.ExtractionInfo(
        result_key="Epoch",
        name_or_column="Epoch_Ele",
        unit=ep.units.cdf_epoch,
    ),
    ep.ExtractionInfo(
        result_key="FEDU",
        name_or_column="FEDU",
        unit=(u.cm**2 * u.s * u.sr * u.keV) ** (-1),
    ),
    ep.ExtractionInfo(
        result_key="xGEO",
        name_or_column="Position_Ele",
        unit=u.km,
    ),
]

start_time = datetime(2017, 7, 14, tzinfo=timezone.utc)
end_time = datetime(2017, 7, 14, 23, 59, 59, tzinfo=timezone.utc)

file_name_stem = "rbspa_rel04_ect-hope-pa-l3_YYYYMMDD_.{6}.cdf"

ep.download(
    start_time,
    end_time,
    save_path=".",
    download_url="https://spdf.gsfc.nasa.gov/pub/data/rbsp/rbspa/l3/ect/hope/pitchangle/rel04/YYYY/",
    file_name_stem=file_name_stem,
    file_cadence="daily",
    method="request",
    skip_existing=True,
)

variables = ep.extract_variables_from_files(
    start_time, end_time, "daily", data_path=".", file_name_stem=file_name_stem, extraction_infos=extraction_infos
)
variables

[INFO    ] 2026-06-05 16:17:21 - el_paso.download:236 - File already exists, skipping download: rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf

[INFO    ] 2026-06-05 16:17:21 - el_paso.download:32 - download finished in 0.173 seconds

[INFO    ] 2026-06-05 16:17:21 - el_paso.extract_variables_from_files:112 - Extracting variables ...

Out[1]:

{'Epoch': Variable holding (3940,) data points with metadata: VariableMetadata(unit=Unit("cdf_epoch"), original_cadence_seconds=0, source_files=['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], description='', processing_notes='', standard_name=''),
 'FEDU': Variable holding (3940, 11, 72) data points with metadata: VariableMetadata(unit=Unit("1 / (keV s sr cm2)"), original_cadence_seconds=0, source_files=['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], description='', processing_notes='', standard_name=''),
 'xGEO': Variable holding (3940, 3) data points with metadata: VariableMetadata(unit=Unit("km"), original_cadence_seconds=0, source_files=['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], description='', processing_notes='', standard_name='')}

Single file strategy¶

First, we want to save the variables using the SingleFileStrategy. This is the simplest way to save variables, as everything is simple put into one file. Dependent on the file ending, different file formats will be saved. Possible formats are ".mat", ".nc", ".cdf" and ".h5".

The units of the variables are not changed when using the SingleFileStrategy.

In [2]:

Copied!





saving_strategy = ep.saving_strategies.SingleFileStrategy("rbsp_hope_example.mat")
ep.save(
    variables,
    saving_strategy=saving_strategy,
    start_time=start_time,
    end_time=end_time,
    time_var=variables["Epoch"],
    ignore_validation=True,
)
saving_strategy = ep.saving_strategies.SingleFileStrategy("rbsp_hope_example.mat")
ep.save(
    variables,
    saving_strategy=saving_strategy,
    start_time=start_time,
    end_time=end_time,
    time_var=variables["Epoch"],
    ignore_validation=True,
)

[INFO    ] 2026-06-05 16:17:21 - el_paso.saving_strategies.single_file_strategy:294 - Saving file rbsp_hope_example.mat...

[INFO    ] 2026-06-05 16:17:21 - el_paso.save:2 - save finished in 0.357 seconds

Let's inspect what got saved. The variables are turned into simple numpy arrays before saving. You can see that also a metadata variable has been saved. We will look closer at metadata in a different tutorial.

In [3]:

Copied!





import scipy.io as sio

with open("rbsp_hope_example.mat", "rb") as f:
    loaded_data = sio.loadmat(f)
    print("Keys: ", loaded_data.keys())
    print("Metadata: ", loaded_data["metadata"])
    print("xGEO[0,:]: ", loaded_data["xGEO"][0, :] * u.Unit(loaded_data["metadata"]["xGEO"][0, 0]["unit"][0, 0][0]))
import scipy.io as sio

with open("rbsp_hope_example.mat", "rb") as f:
    loaded_data = sio.loadmat(f)
    print("Keys: ", loaded_data.keys())
    print("Metadata: ", loaded_data["metadata"])
    print("xGEO[0,:]: ", loaded_data["xGEO"][0, :] * u.Unit(loaded_data["metadata"]["xGEO"][0, 0]["unit"][0, 0][0]))

Keys:  dict_keys(['__header__', '__version__', '__globals__', 'Epoch', 'FEDU', 'xGEO', 'metadata'])
Metadata:  [[(array([[(array(['cdf_epoch'], dtype='<U9'), array([[0]]), array(['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], dtype='<U46'), array([], dtype='<U1'), array([], dtype='<U1'), array(['Epoch'], dtype='<U5'))]],
        dtype=[('unit', 'O'), ('original_cadence_seconds', 'O'), ('source_files', 'O'), ('description', 'O'), ('processing_notes', 'O'), ('standard_name', 'O')]), array([[(array(['1 / (keV s sr cm2)'], dtype='<U18'), array([[0]]), array(['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], dtype='<U46'), array([], dtype='<U1'), array([], dtype='<U1'), array(['FEDU'], dtype='<U4'))]],
        dtype=[('unit', 'O'), ('original_cadence_seconds', 'O'), ('source_files', 'O'), ('description', 'O'), ('processing_notes', 'O'), ('standard_name', 'O')]), array([[(array(['km'], dtype='<U2'), array([[0]]), array(['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], dtype='<U46'), array([], dtype='<U1'), array([], dtype='<U1'), array(['xGEO'], dtype='<U4'))]],
        dtype=[('unit', 'O'), ('original_cadence_seconds', 'O'), ('source_files', 'O'), ('description', 'O'), ('processing_notes', 'O'), ('standard_name', 'O')]))                                                                                                                                                                                                                              ]]
xGEO[0,:]:  [ -7604.8604  -28464.74       709.80096] km

Saving strategies¶

GFZStrategy¶

All data at GFZ is stored under a standard, which we call GFZStandard and is saved using GFZStrategy where multiple monthly files are generated for different variables. The standard inlcudes name of the variables that are saved. All files are saved as .mat files. By using the GFZStrategy, the variables are automatically sorted into the corresponding files, based on the key of the variable_dict. Additionally, the variables are converted into the units as described in the standard.

MonthlyRBStrategy¶

Under this strategy, the data is stored in monthly files and all the variables are written (appended) to one monthly file. In this strategy, different standards (GFZStandard and PRBEMStandard) can be used to define the name of the variables.

While using these strategies, we have to store the variables in a dictionary with certain keys, as specified by the InternalName type alias. These variable name and the corresponding value in the variables dictionary are statically typechecked and are also validated internally in order to refrain users from saving incorrect key and value types. We can ignore this validation by setting ignore_validation=True in ep.save(..., ignore_validation=True)

In [4]:

Copied!





from el_paso.typing import InternalName
from datetime import datetime, timezone

from astropy import units as u

import el_paso as ep


ep.setup_logging()

extraction_infos = [
    ep.ExtractionInfo(
        result_key="Epoch",
        name_or_column="Epoch_Ele",
        unit=ep.units.cdf_epoch,
    ),
    ep.ExtractionInfo(
        result_key="FEDU",
        name_or_column="FEDU",
        unit=(u.cm**2 * u.s * u.sr * u.keV) ** (-1),
    ),
    ep.ExtractionInfo(
        result_key="xGEO",
        name_or_column="Position_Ele",
        unit=u.km,
    ),
]

start_time = datetime(2017, 7, 14, tzinfo=timezone.utc)
end_time = datetime(2017, 7, 14, 23, 59, 59, tzinfo=timezone.utc)

file_name_stem = "rbspa_rel04_ect-hope-pa-l3_YYYYMMDD_.{6}.cdf"

ep.download(
    start_time,
    end_time,
    save_path=".",
    download_url="https://spdf.gsfc.nasa.gov/pub/data/rbsp/rbspa/l3/ect/hope/pitchangle/rel04/YYYY/",
    file_name_stem=file_name_stem,
    file_cadence="daily",
    method="request",
    skip_existing=True,
)

variables = ep.extract_variables_from_files(
    start_time, end_time, "daily", data_path=".", file_name_stem=file_name_stem, extraction_infos=extraction_infos
)
variables
from el_paso.typing import InternalName
from datetime import datetime, timezone

from astropy import units as u

import el_paso as ep


ep.setup_logging()

extraction_infos = [
    ep.ExtractionInfo(
        result_key="Epoch",
        name_or_column="Epoch_Ele",
        unit=ep.units.cdf_epoch,
    ),
    ep.ExtractionInfo(
        result_key="FEDU",
        name_or_column="FEDU",
        unit=(u.cm**2 * u.s * u.sr * u.keV) ** (-1),
    ),
    ep.ExtractionInfo(
        result_key="xGEO",
        name_or_column="Position_Ele",
        unit=u.km,
    ),
]

start_time = datetime(2017, 7, 14, tzinfo=timezone.utc)
end_time = datetime(2017, 7, 14, 23, 59, 59, tzinfo=timezone.utc)

file_name_stem = "rbspa_rel04_ect-hope-pa-l3_YYYYMMDD_.{6}.cdf"

ep.download(
    start_time,
    end_time,
    save_path=".",
    download_url="https://spdf.gsfc.nasa.gov/pub/data/rbsp/rbspa/l3/ect/hope/pitchangle/rel04/YYYY/",
    file_name_stem=file_name_stem,
    file_cadence="daily",
    method="request",
    skip_existing=True,
)

variables = ep.extract_variables_from_files(
    start_time, end_time, "daily", data_path=".", file_name_stem=file_name_stem, extraction_infos=extraction_infos
)
variables

[INFO    ] 2026-06-05 16:17:21 - el_paso.download:236 - File already exists, skipping download: rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf

[INFO    ] 2026-06-05 16:17:21 - el_paso.download:34 - download finished in 0.003 seconds

[INFO    ] 2026-06-05 16:17:21 - el_paso.extract_variables_from_files:112 - Extracting variables ...

Out[4]:

{'Epoch': Variable holding (3940,) data points with metadata: VariableMetadata(unit=Unit("cdf_epoch"), original_cadence_seconds=0, source_files=['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], description='', processing_notes='', standard_name=''),
 'FEDU': Variable holding (3940, 11, 72) data points with metadata: VariableMetadata(unit=Unit("1 / (keV s sr cm2)"), original_cadence_seconds=0, source_files=['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], description='', processing_notes='', standard_name=''),
 'xGEO': Variable holding (3940, 3) data points with metadata: VariableMetadata(unit=Unit("km"), original_cadence_seconds=0, source_files=['rbspa_rel04_ect-hope-pa-l3_20170714_v7.4.0.cdf'], description='', processing_notes='', standard_name='')}

In [5]:

Copied!





variables_to_save: dict[InternalName, ep.Variable] = {
    "Epoch": variables["Epoch"],
    "FEDU": variables["FEDU"],
    "Position": variables["xGEO"],
}


mrb_gfz = ep.saving_strategies.MonthlyRBStrategy(
    "./RBSP/mrb_gfz",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.GFZStandard(),
)

gfz_gfz = ep.saving_strategies.GFZStrategy(
    "./RBSP/gfz_gfz",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.GFZStandard(),
)

gfz_prbem = ep.saving_strategies.GFZStrategy(
    "./RBSP/gfz_prbem",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.PRBEMStandard(),
)

mrb_prbem = ep.saving_strategies.MonthlyRBStrategy(
    "./RBSP/mrb_prbem",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.PRBEMStandard(),
)


ep.save(variables_to_save, mrb_gfz, start_time, end_time, time_var=variables["Epoch"])
ep.save(variables_to_save, gfz_gfz, start_time, end_time, time_var=variables["Epoch"])
ep.save(variables_to_save, mrb_prbem, start_time, end_time, time_var=variables["Epoch"])
ep.save(variables_to_save, gfz_prbem, start_time, end_time, time_var=variables["Epoch"])
variables_to_save: dict[InternalName, ep.Variable] = {
    "Epoch": variables["Epoch"],
    "FEDU": variables["FEDU"],
    "Position": variables["xGEO"],
}


mrb_gfz = ep.saving_strategies.MonthlyRBStrategy(
    "./RBSP/mrb_gfz",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.GFZStandard(),
)

gfz_gfz = ep.saving_strategies.GFZStrategy(
    "./RBSP/gfz_gfz",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.GFZStandard(),
)

gfz_prbem = ep.saving_strategies.GFZStrategy(
    "./RBSP/gfz_prbem",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.PRBEMStandard(),
)

mrb_prbem = ep.saving_strategies.MonthlyRBStrategy(
    "./RBSP/mrb_prbem",
    mission="RBSP",
    satellite="rbspa",
    instrument="hope",
    mag_field="T89",
    data_standard=ep.data_standards.PRBEMStandard(),
)


ep.save(variables_to_save, mrb_gfz, start_time, end_time, time_var=variables["Epoch"])
ep.save(variables_to_save, gfz_gfz, start_time, end_time, time_var=variables["Epoch"])
ep.save(variables_to_save, mrb_prbem, start_time, end_time, time_var=variables["Epoch"])
ep.save(variables_to_save, gfz_prbem, start_time, end_time, time_var=variables["Epoch"])

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable Alpha_Eq!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable Energy_FEDU!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable Alpha!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable B_Calc!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable B_Eq!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable InvK!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable InvMu!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable PSD!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable R_Eq!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable MLT!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable L_m!

[WARNING ] 2026-06-05 16:17:22 - el_paso.saving_strategy:75 - Could not find target variable L_star!

[INFO    ] 2026-06-05 16:17:22 - el_paso.saving_strategies.monthly_rb_strategy:207 - Saving file: /home/docs/checkouts/readthedocs.org/user_builds/el-paso/checkouts/latest/tutorials/RBSP/mrb_gfz/RBSP/rbspa/rbspa_hope_20170701to20170731_T89.nc

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/netCDF4/utils.py:39, in _find_dim(grp, dimname)
     38 try:
---> 39     dim = group.dimensions[dimname]
     40     break

AttributeError: 'NoneType' object has no attribute 'dimensions'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/netCDF4/utils.py:43, in _find_dim(grp, dimname)
     42 try:
---> 43     group = group.parent
     44 except:

AttributeError: 'NoneType' object has no attribute 'parent'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[5], line 45
     41     data_standard=ep.data_standards.PRBEMStandard(),
     42 )
     43 
     44 
---> 45 ep.save(variables_to_save, mrb_gfz, start_time, end_time, time_var=variables["Epoch"])
     46 ep.save(variables_to_save, gfz_gfz, start_time, end_time, time_var=variables["Epoch"])
     47 ep.save(variables_to_save, mrb_prbem, start_time, end_time, time_var=variables["Epoch"])
     48 ep.save(variables_to_save, gfz_prbem, start_time, end_time, time_var=variables["Epoch"])

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/utils.py:165, in timed_function.<locals>.timed_function_.<locals>.wrap(*args, **kwargs)
    162 @wraps(f)
    163 def wrap(*args: P.args, **kwargs: P.kwargs) -> R:
    164     tic = timeit.default_timer()
--> 165     result = f(*args, **kwargs)
    166     toc = timeit.default_timer()
    167     name = func_name or f"{f.__name__}"  # ty:ignore[unresolved-attribute]

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/save.py:86, in save(variables_dict, saving_strategy, start_time, end_time, time_var, append, ignore_validation)
     84 else:
     85     data_dict = _get_data_dict_to_save(target_variables)
---> 86     saving_strategy.save_single_file(file_path, data_dict, append=append)
     87     file_path.chmod(0o660)

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/saving_strategies/monthly_rb_strategy.py:209, in MonthlyRBStrategy.save_single_file(self, file_path, dict_to_save, append)
    205     return
    207 logger.info(f"Saving file: {file_path.resolve()}")
--> 209 writer(file_path, dict_to_save, self.data_standard)

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/utils.py:595, in write_netcdf_file(file_path, data_dict, data_standard)
    592     else:
    593         file.createDimension(dim_name, dim_size)
--> 595 _write_data_to_netcdf_file(file, data_dict, data_standard)

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/utils.py:536, in _write_data_to_netcdf_file(file, data_dict, data_standard)
    531         curr_hierarchy = curr_hierarchy.groups[group]
    533 dimensions = data_standard.get_dependencies(internal_name)
    534 data_set = cast(
    535     "nC.Variable[Any]",
--> 536     curr_hierarchy.createVariable(
    537         dataset_name,
    538         "float64",
    539         dimensions,
    540         zlib=True,
    541         complevel=5,
    542         shuffle=True,
    543     ),
    544 )
    546 value_to_write = value_array
    547 if len(dimensions) == 1 and value_array.ndim == 2 and value_array.shape[1] == 1:

File src/netCDF4/_netCDF4.pyx:3016
-> 3016 'Could not get source, probably due dynamically evaluated source code.'

File src/netCDF4/_netCDF4.pyx:3016
-> 3016 'Could not get source, probably due dynamically evaluated source code.'

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/netCDF4/utils.py:45, in _find_dim(grp, dimname)
     43             group = group.parent
     44         except:
---> 45             raise ValueError("cannot find dimension %s in this group or parent groups" % dimname)
     46 if dim is None:
     47     raise KeyError("dimension %s not defined in group %s or any group in its family tree" % (dimname, grp.path))

ValueError: cannot find dimension Energy_FEDU in this group or parent groups

Datasets¶

The saved datasets can be loaded using the DataSet class. We provide two concerete implmentation for the DataSet, namely GFZDataSet and PRBEMDataSet.

GFZDataSet¶

GFZDataSet/PRBEMDataSet is used to load the data which is saved using GFZStandard/PRBEMStandard. The variables which can be accessed by each of these DataSet implementations are annotated in the respective classes in order to help user with the auto-completions.

In [6]:

Copied!





from el_paso.dataset import GFZDataSet, PRBEMDataSet

mrb_gfz_data = GFZDataSet(mrb_gfz, start_time, end_time)
gfz_gfz_data = GFZDataSet(gfz_gfz, start_time, end_time)
mrb_prbem_data = PRBEMDataSet(mrb_prbem, start_time, end_time)
gfz_prbem_data = PRBEMDataSet(gfz_prbem, start_time, end_time)
from el_paso.dataset import GFZDataSet, PRBEMDataSet

mrb_gfz_data = GFZDataSet(mrb_gfz, start_time, end_time)
gfz_gfz_data = GFZDataSet(gfz_gfz, start_time, end_time)
mrb_prbem_data = PRBEMDataSet(mrb_prbem, start_time, end_time)
gfz_prbem_data = PRBEMDataSet(gfz_prbem, start_time, end_time)

[WARNING ] 2026-06-05 16:17:22 - el_paso.dataset.dataset_implementations:113 - Overriding `preferred_extension` to 'mat' since `GFZStrategy` is used, which only supports .mat files. Ignoring provided `preferred_extension` value.

In [7]:

Copied!

mrb_prbem_data.metadata.datetime
mrb_prbem_data.metadata.to_dict()
mrb_prbem_data.metadata.datetime
mrb_prbem_data.metadata.to_dict()

[WARNING ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:302 - Tried to load RBSP/mrb_prbem/RBSP/rbspa/rbspa_hope_20170701to20170731_T89.nc, but it does not exist

Out[7]:

{}

In [8]:

Copied!

mrb_prbem_data.Position.shape
mrb_prbem_data.Position.shape

Out[8]:

(0,)

In [9]:

Copied!

mrb_gfz_data.metadata.datetime
mrb_gfz_data.metadata.to_dict()
mrb_gfz_data.metadata.datetime
mrb_gfz_data.metadata.to_dict()

[INFO    ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:299 - Loading RBSP/mrb_gfz/RBSP/rbspa/rbspa_hope_20170701to20170731_T89.nc

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[9], line 1
----> 1 mrb_gfz_data.metadata.datetime
      2 mrb_gfz_data.metadata.to_dict()

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/dataset/metadata.py:79, in DatasetMetadata.__getattr__(self, name)
     76 currently_loading = dataset.__dict__.get("_currently_loading", set())
     78 if name not in currently_loading:
---> 79     dataset._load_variable(name)
     81     if name in self.__dict__:
     82         return self.__dict__[name]

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/dataset/dataset.py:308, in DataSet._load_variable(self, requested_name)
    305 time_key = self.saving_strategy.data_standard.get_standard_name("Epoch")
    307 # 3. Process Datetimes
--> 308 raw_times = file_content[time_key]
    310 time_unit = self.saving_strategy.data_standard.variable_infos["Epoch"].unit
    312 posix_times = (raw_times * time_unit).to_value(ep.units.posixtime)

KeyError: 'time'

In [10]:

Copied!

mrb_gfz_data.xGEO.shape
mrb_gfz_data.xGEO.shape

[INFO    ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:299 - Loading RBSP/mrb_gfz/RBSP/rbspa/rbspa_hope_20170701to20170731_T89.nc

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[10], line 1
----> 1 mrb_gfz_data.xGEO.shape

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/dataset/dataset.py:196, in DataSet.__getattr__(self, name)
    194 sat_variable, levenstein_info = self.find_similar_variable(name)
    195 if sat_variable is not None and hasattr(self, "_date_list"):
--> 196     self._load_variable(sat_variable)
    197     return getattr(self, name)
    198 if name in self.possible_variables:

File ~/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/dataset/dataset.py:308, in DataSet._load_variable(self, requested_name)
    305 time_key = self.saving_strategy.data_standard.get_standard_name("Epoch")
    307 # 3. Process Datetimes
--> 308 raw_times = file_content[time_key]
    310 time_unit = self.saving_strategy.data_standard.variable_infos["Epoch"].unit
    312 posix_times = (raw_times * time_unit).to_value(ep.units.posixtime)

KeyError: 'time'

In [11]:

Copied!

gfz_gfz_data.metadata.datetime
gfz_gfz_data.metadata.to_dict()
gfz_gfz_data.metadata.datetime
gfz_gfz_data.metadata.to_dict()

[WARNING ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:302 - Tried to load RBSP/gfz_gfz/RBSP/rbspa/Processed_Mat_Files/rbspa_hope_20170701to20170731_flux_ver4.mat, but it does not exist

Out[11]:

{}

In [12]:

Copied!

gfz_gfz_data.xGEO.shape
gfz_gfz_data.xGEO.shape

[WARNING ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:302 - Tried to load RBSP/gfz_gfz/RBSP/rbspa/Processed_Mat_Files/rbspa_hope_20170701to20170731_xGEO_ver4.mat, but it does not exist

Out[12]:

(0,)

In [13]:

Copied!

gfz_prbem_data.metadata.datetime
gfz_prbem_data.metadata.to_dict()
gfz_prbem_data.metadata.datetime
gfz_prbem_data.metadata.to_dict()

[WARNING ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:302 - Tried to load RBSP/gfz_prbem/RBSP/rbspa/Processed_Mat_Files/rbspa_hope_20170701to20170731_flux_ver4.mat, but it does not exist

Out[13]:

{}

In [14]:

Copied!

gfz_prbem_data.Position.shape
gfz_prbem_data.Position.shape

[WARNING ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:302 - Tried to load RBSP/gfz_prbem/RBSP/rbspa/Processed_Mat_Files/rbspa_hope_20170701to20170731_xGEO_ver4.mat, but it does not exist

Out[14]:

(0,)

Setting data to `DataSet` externally¶

If in case, we want to attach some data to a DataSet, a complex __setattr__ is implemented to achieve that.

In [15]:

Copied!





from el_paso.dataset import DataSet

mock_strategy = ep.saving_strategies.MonthlyRBStrategy(
    base_data_path=".",
    mission="mock",
    satellite="mock",
    instrument="mock",
    mag_field="T89",
    data_standard=ep.data_standards.GFZStandard(),
)
mock_ds = DataSet(mock_strategy, start_time, end_time)
from el_paso.dataset import DataSet

mock_strategy = ep.saving_strategies.MonthlyRBStrategy(
    base_data_path=".",
    mission="mock",
    satellite="mock",
    instrument="mock",
    mag_field="T89",
    data_standard=ep.data_standards.GFZStandard(),
)
mock_ds = DataSet(mock_strategy, start_time, end_time)

In [16]:

Copied!

mock_ds.time
mock_ds.time

[WARNING ] 2026-06-05 16:17:22 - el_paso.dataset.dataset:302 - Tried to load MOCK/mock/mock_mock_20170701to20170731_T89.nc, but it does not exist

Out[16]:

array([], dtype=float64)

In [17]:

Copied!

mock_ds.time = variables["Epoch"]
mock_ds.Flux = variables["FEDU"]
mock_ds.xGEO = variables["xGEO"]
mock_ds.time = variables["Epoch"]
mock_ds.Flux = variables["FEDU"]
mock_ds.xGEO = variables["xGEO"]

Since the mock_ds is initialised using GFZStandard, only the variables which are associated with GFZStandard can be attributed to it. Setting an invalid variable will raise AttributeError. The same applies to PRBEMStandard

In [18]:

Copied!





import traceback

try:
    mock_ds.Position = 1
except AttributeError as e:
    traceback.print_exc()
import traceback

try:
    mock_ds.Position = 1
except AttributeError as e:
    traceback.print_exc()

Traceback (most recent call last):
  File "/tmp/ipykernel_1578/4028079179.py", line 4, in <module>
    mock_ds.Position = 1
    ^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/el-paso/envs/latest/lib/python3.13/site-packages/el_paso/dataset/dataset.py", line 158, in __setattr__
    raise AttributeError(msg)
AttributeError: Cannot set attribute 'Position'. It is not part of the saving strategy: MonthlyRBStrategy(base_data_path=PosixPath('.'), mission='mock', satellite='mock', instrument='mock', mag_field='T89', data_standard=GFZStandard(), file_format='.nc').