Introduction to the Data class

In this notebook, we will have a first look at the galfind.Data object. The Data object is made up of an array of Band_Data objects which store all the information pertaining to the photometric imaging in a specific photometric filter, including the paths/extensions to the SCI/WHT/ERR maps as well as a galfind.Filter (see the Instrument section, and specifically the Filter notebook for more information).

We will begin by first loading the reduced JADES Origins Field (JOF) JWST/NIRCam imaging from Adams et al. 2024. This data must first be loaded from the EPOCHS dropbox, instructions for which can be found in Getting started / Downloading observational data. If these data products are not currently downloaded and stored in the relevant locations, please take the time to do this now before continuuing.

Example 1: Initializing Band_Data and Data objects

To start off, we will practice initializing both Band_Data and Data objects. To do this we require the paths/extensions to each of the SCI/WHT/RMS_ERR images.

[1]:

# imports
import astropy.units as u
import matplotlib.pyplot as plt
from galfind import config, Band_Data, Stacked_Band_Data, Data, Filter, Multiple_Filter
from galfind.Data import morgan_version_to_dir

Reading GALFIND config file from: /nvme/scratch/work/austind/GALFIND/galfind/../configs/galfind_config.ini
Important:  Gaia archive will be intermittently unavailable due to scheduled maintenance on 14-10-2024 from 10:30 to 12:30 (CEST)

Investigate with 30mas pixel scale imaging from the F090W filter to start off with.

[2]:

survey = "JOF"
version = "v11"
facility_name = "JWST"
instrument_name = "NIRCam"
filt_name = "F090W"
pix_scale_name = "30mas"
im_dir = f"{config['DEFAULT']['GALFIND_DATA']}/{facility_name.lower()}/{survey}/{instrument_name}/{morgan_version_to_dir[version]}/{pix_scale_name}"
im_path = f"{im_dir}/jw01210-o001_t002_nircam_clear-{filt_name.lower()}_i2dnobg.fits"
im_ext = 1
rms_err_path = im_path
rms_err_ext = 2
wht_path = im_path
wht_ext = 4
pix_scale = 0.03 * u.arcsec

band_data = Band_Data(
    Filter.from_SVO(facility_name, instrument_name, filt_name),
    survey,
    version,
    im_path,
    im_ext,
    rms_err_path,
    rms_err_ext,
    wht_path,
    wht_ext,
    pix_scale,
    im_ext_name="SCI",
    rms_err_ext_name="ERR",
    wht_ext_name="WHT"
)
print(band_data)

****************************************
NIRCam/F090W
SURVEY: JOF
VERSION: v11
PIX SCALE: 0.03 arcsec
ZP: 28.086519392283982
SHAPE: (4464, 10244)
****************************************
IM PATH: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas/jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[1]
RMS ERR PATH: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas/jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[2]
WHT PATH: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas/jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[4]
****************************************

This above implementation, however, requires the sci/rms_err/wht extensions to be named SCI/ERR/WHT in the image header “EXTNAME” by default, although an array of possible “EXTNAME” for each may be passed in if required. This is demonstrated by the im_ext_name, rms_err_ext_name, and wht_ext_name parameters respectively. Now that we have seen how to load in a Band_Data object by itself, we will now create a Data object. We will see that initializing many Band_Data objects and adding them together does the same thing as initializing a Data object by itself. Let’s have a go at this now.

[3]:

# Array storing all NIRCam bands used in the pipeline for JOF
JOF_nircam_filt_names = ["F090W", "F115W", "F150W", "F162M", "F182M", "F200W", "F210M", "F250M", "F277W", "F300M", "F335M", "F356W", "F410M", "F444W"]
# Create an instrument object for NIRCam incorporating all the NIRCam bands
JOF_nircam_filters = Multiple_Filter([Filter.from_filt_name(filt_name) for filt_name in JOF_nircam_filt_names])
print(JOF_nircam_filters)

# Proposal IDs for the NIRCam bands
PIDs = {filt_name: "4210" if filt_name == "F444W" else "1210" if filt_name.endswith("W") or filt_name == "F410M" else "3215" for filt_name in JOF_nircam_filt_names}
# Paths to the sci images for the NIRCam bands
sci_paths = {filt_name: f"{im_dir}/jw0{PIDs[filt_name]}-o001_t002_nircam_clear-{filt_name.lower()}_i2dnobg.fits" for filt_name in JOF_nircam_filt_names}
# Paths to the rms error/weight maps are the same as the sci images, just with a different extension
rms_err_paths = sci_paths
wht_paths = sci_paths

# sci, wht, and rms_err extensions for the NIRCam bands
sci_exts = {filt_name: 1 for filt_name in JOF_nircam_filt_names}
rms_err_exts = {filt_name: 2 for filt_name in JOF_nircam_filt_names}
wht_exts = {filt_name: 4 for filt_name in JOF_nircam_filt_names}

band_data_arr = [Band_Data(JOF_nircam_filters[filt_name], survey, version, sci_paths[filt_name], sci_exts[filt_name],
    rms_err_paths[filt_name], rms_err_exts[filt_name], wht_paths[filt_name], wht_exts[filt_name], pix_scale) for filt_name in JOF_nircam_filt_names]
# Create a data object for the NIRCam bands
for i, band_data in enumerate(band_data_arr):
    if i == 0:
        JOF_data_1 = band_data
    else:
        JOF_data_1 += band_data

JOF_data_2 = Data(band_data_arr)

****************************************
MULTIPLE_FILTER
----------
FACILITY: JWST
INSTRUMENT: NIRCam
FILTERS: ['F090W', 'F115W', 'F150W', 'F162M', 'F182M', 'F200W', 'F210M', 'F250M', 'F277W', 'F300M', 'F335M', 'F356W', 'F410M', 'F444W']
****************************************

Our data objects are identical here. It is worth noting that adding together Band_Data or Data objects containing the same filters from the same survey and version will throw an error; if stacking is wanted this is implemented in Band_Data.__mul__() and Data.__mul__(). Adding together Band_Data or Data objects from different surveys and versions will instead create a Multiple_Data object which contains an array of Data objects from different surveys/versions. For more information on the Multiple_Data class, please see the Multiple_Data notebook.

[4]:

# check that the two initilization methods produce the same data object
if JOF_data_1 == JOF_data_2:
    print("Data objects are identical")
else:
    print("Data objects are different")
# print the data object
print(JOF_data_1)

Data objects are identical
****************************************
DATA OBJECT:
----------
SURVEY: JOF
VERSION: v11
****************************************
MULTIPLE_FILTER
----------
FACILITY: JWST
INSTRUMENT: NIRCam
FILTERS: ['F090W', 'F115W', 'F150W', 'F162M', 'F182M', 'F200W', 'F210M', 'F250M', 'F277W', 'F300M', 'F335M', 'F356W', 'F410M', 'F444W']
****************************************
NIRCam COMMON ATTRIBUTES:
----------
IM DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas
RMS ERR DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas
WHT DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas
IM EXT: 1
RMS ERR EXT: 2
WHT EXT: 4
ZP: 28.0865
PIX SCALE: 0.03 arcsec
DATA SHAPE: (4464, 10244)
----------
****************************************
NIRCam/F090W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[4]
****************************************
NIRCam/F115W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[4]
****************************************
NIRCam/F150W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[4]
****************************************
NIRCam/F162M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[4]
****************************************
NIRCam/F182M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[4]
****************************************
NIRCam/F200W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[4]
****************************************
NIRCam/F210M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[4]
****************************************
NIRCam/F250M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[4]
****************************************
NIRCam/F277W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[4]
****************************************
NIRCam/F300M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[4]
****************************************
NIRCam/F335M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[4]
****************************************
NIRCam/F356W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[4]
****************************************
NIRCam/F410M
----------
IM NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[4]
****************************************
NIRCam/F444W
----------
IM NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[1]
RMS ERR NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[2]
WHT NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[4]
****************************************
****************************************

As you can see from the number of dictionaries required to create an array of Band_Data objects or initialize a Data object using the standard Data.__init__(), this can get confusing relatively quickly. In addition, to build these from dictionaties we require prior knowledge of the available bands for each field as well as the paths/extensions to each SCI/ERR/WHT map. Luckily there is a useful class method to instantiate the Data class, Data.from_survey_version(), which simplifies things for us a bit. As with Band_Data, Data additionally accepts im_ext_name, rms_err_ext_name, and wht_ext_name arguments in case . We demonstrate its use with JWST/NIRCam below.

[5]:

# create a Data object directly from the given survey and version
JOF_data_3 = Data.from_survey_version(
    survey,
    version,
    instrument_names = ["NIRCam"],
    version_to_dir_dict = morgan_version_to_dir,
    im_ext_name = "SCI",
    rms_err_ext_name = "ERR",
    wht_ext_name = "WHT"
)

# ensure that this object is the same as the one created above
if JOF_data_1 == JOF_data_3:
    print("Data objects are identical")
else:
    print("Data objects are different")

Data objects are identical

Example 2: Mosaicing together multiple data from the same band

Should there be multiple SCI/WHT/RMS_ERR images located in the relevant folders this class method will mosaic these together and move the older images into an /old sub-directory. This will only occur if these images have the same shape and pixel scale, otherwise galfind will fail. It is our wish to implement a solution to this in the near future. This problem will also be encountered when attempting to multiply (i.e. mosaic/stack) together Band_Data or Data objects from the same survey/version with another that contains the same Filter with different image dimensions/pixel scales.

[!WARNING] Mosaicing multiple data from the same band has not yet been implemented!

Example 3: Making a Stacked_Band_Data object

The Band_Data class which we have looked at in examples 1 and 2 is one of two child classes of the abstract Band_Data_Base class, Band_Data and Stacked_Band_Data. Both of these classes store the same basic information regarding the SCI/ERR/WHT paths/extensions, as well as the ability to make/load segmentation maps, masks, forced photometry catalogues, etc. The major difference between these two classes is that Band_Data stores a Filter object whereas Stacked_Band_Data stores a Multiple_Filter; various abstract methods are overridden to account for this.

We shall start by instantiating a Stacked_Band_Data object for the SW NIRCam widebands via the Stacked_Band_Data.from_band_data_arr() class method. Because I can’t be bothered to look up and type in the paths/extensions to the JOF data, we shall obtain the array of Band_Data objects using the Data.from_survey_version() method as we have done previously in our creation of the JOF_data_3 object.

[6]:

blue_nircam_band_names = ["F090W", "F115W", "F150W", "F200W"]
blue_nircam_band_data_arr = JOF_data_3[blue_nircam_band_names]
stacked_SW_nircam_obj = Stacked_Band_Data.from_band_data_arr(blue_nircam_band_data_arr)
print(stacked_SW_nircam_obj)

****************************************
NIRCam/F090W+F115W+F150W+F200W
SURVEY: JOF
VERSION: v11
PIX SCALE: 0.03 arcsec
ZP: 28.086519392283982
SHAPE: (4464, 10244)
****************************************
IM PATH: /raid/scratch/work/austind/GALFIND_WORK/Stacked_Images/v11/NIRCam/JOF/rms_err/JOF_F090W+F115W+F150W+F200W_v11_stack.fits[1]
RMS ERR PATH: /raid/scratch/work/austind/GALFIND_WORK/Stacked_Images/v11/NIRCam/JOF/rms_err/JOF_F090W+F115W+F150W+F200W_v11_stack.fits[2]
WHT PATH: /raid/scratch/work/austind/GALFIND_WORK/Stacked_Images/v11/NIRCam/JOF/rms_err/JOF_F090W+F115W+F150W+F200W_v11_stack.fits[3]
****************************************

Rather than instantiating this via the Stacked_Band_Data class method, we can also stack Band_Data and/or Stacked_Band_Data objects by using the * (multiplication) operator. This will work so long as these objects contain different reference filters, otherwise the data will instead be mosaiced. Simply adding each band together will instead produce a Data object.

[7]:

stacked_SW_nircam_obj_2 = blue_nircam_band_data_arr[0]
for band_data in blue_nircam_band_data_arr[1:]:
    stacked_SW_nircam_obj_2 *= band_data

if stacked_SW_nircam_obj == stacked_SW_nircam_obj_2:
    print("Stacked_Band_Data objects are identical")
else:
    print("Stacked_Band_Data objects are different")

Stacked_Band_Data objects are identical

These two code blocks clearly produce the same result, although the second method recursively adds these bands together, producing intermediate stage [F090W, F115W] and [F115W, F150W, F200W] stacked images as well. For this reason, we usually recommend using method 1 in an analysis.

Example 4: Plotting an RGB of the data

Now we have seen how to instantiate a Data object, we will now attempt to plot an RGB for the JOF field using both the trilogy package and astropy.visualization.lupton_rgb. First we have to decide which photometric bands we wish to use for this RGB, and below we explicitly use the default bands used in the Data.plot_RGB method.

[8]:

blue_bands = ["F090W"]
green_bands = ["F200W"]
red_bands = ["F444W"]

Now let’s actually plot the RGB, starting with the trilogy method. Depending on the data used, trilogy can sometimes produce messy results, therefore we allow the flexibility to adjust all input parameters, as covered by the official trilogy documentation.

[9]:

#JOF_data_1.plot_RGB(blue_bands=blue_bands, green_bands=green_bands, red_bands=red_bands, method="trilogy")

[10]:

#fig, ax = plt.subplots()
#JOF_data_1.plot_RGB(ax, blue_bands=blue_bands, green_bands=green_bands, red_bands=red_bands, method="lupton")

Please feel free to now move onto the next notebook which looks at PSF homogenizing each band in your data to the same PSF.