Cataloguing the data
The primary use for the galfind Data object is the creation of photometric catalogues for public and personal use. Once produced, these catalogues can be loaded into the Catalogue class to derive specific properties, which we cover in the next section. In this first galfind release, we implement the ability to produce these using SExtractor only, although we aim to include other codes to perform forced photometry in the near future.
The cataloguing procedure involves many different steps that have been explained in previous notebooks in this section. We outline these here.
Instantiate a blank
Dataobject from the reduced imagingProduce segmentation maps for each band using SExtractor
Mask the data (i.e. image edges, stars, artefacts) either manually or automatically
Perform forced photometry in a set of given apertures in either a single band or an inverse-variance weighted stack of bands.
Aperture correct the fluxes based on a given model or empirical PSF
Apply boolean mask columns to the catalogue to indicate whether each source falls in the mask
Calculate local depths for each source
Determine appropriate flux errors based on these depths, accounting for the correlated image noise
We will create a Data object and associated .fits catalogue following these 8 steps in two ways, long (example 1) and short (example 2).
Example 1: Producing photometric catalogues
To start with, we will load the same JOF Data object we have seen in previous examples.
[1]:
from pathlib import Path
from astropy.table import Table
from copy import deepcopy
import astropy.units as u
from galfind import Stacked_Band_Data, Data
from galfind.Data import morgan_version_to_dir
survey = "JOF"
version = "v11"
instrument_names = ["NIRCam"]
aper_diams = [0.32] * u.arcsec
forced_phot_band = ["F277W", "F356W", "F444W"]
min_flux_pc_err = 10.
# 1
JOF_data_long = Data.from_survey_version(
survey,
version,
instrument_names = instrument_names,
version_to_dir_dict = morgan_version_to_dir,
aper_diams = aper_diams,
forced_phot_band = forced_phot_band
)
# 2
JOF_data_long.mask()
# 3
JOF_data_long.segment()
# 4
JOF_data_long.perform_forced_phot()
# 5
JOF_data_long.append_aper_corr_cols()
# 6
JOF_data_long.append_mask_cols()
# 7
JOF_data_long.run_depths()
# 8
JOF_data_long.append_loc_depth_cols(min_flux_pc_err = min_flux_pc_err)
Reading GALFIND config file from: /nvme/scratch/work/austind/GALFIND/galfind/../configs/galfind_config.ini
Important: Gaia archive will be intermittently unavailable due to scheduled maintenance on 14-10-2024 from 10:30 to 12:30 (CEST)
INFO:galfind:Loaded aper_diams=<Quantity [0.32] arcsec> for F277W+F356W+F444W
INFO:galfind:Combined mask for NIRCam/F277W+F356W+F444W already exists at /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/combined/JOF_F277W+F356W+F444W_auto.fits
WARNING:galfind:Aperture correction columns already in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits
Calculating depths: 0%| | 0/15 [00:00<?, ?it/s]
INFO:galfind:Calculated/loaded depths for JOF v11 NIRCam
INFO:galfind:Local depth columns already exist in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits
Now we will search the GALFIND_WORK directory for the individual forced photometry catalogues for each band and the resulting catalogue/README to ensure they exist and have been created correctly.
[2]:
# search for photometric catalogue
if Path(JOF_data_long.phot_cat_path).is_file():
print("Photometric catalogue exists at the expected path.")
# open the photometric catalogue
phot_cat = Table.read(JOF_data_long.phot_cat_path)
print(phot_cat)
else:
print("Photometric catalogue does not exist at the expected path.")
# # search for README
# readme_path = JOF_data_long.phot_cat_path.replace(".fits", "_README.txt")
# if Path(readme_path).is_file():
# print("README exists at the expected path.")
# # print the README
# with open(readme_path, "r") as f:
# print(f.read())
# f.close()
# else:
# print("README does not exist at the expected path.")
Photometric catalogue exists at the expected path.
NUMBER X_IMAGE ... FLUXERR_APER_F444W_loc_depth_10pc_Jy
pix ...
---------- ----------- ... ------------------------------------
1 9219.8145 ... 9.334392262202193e-07
2 6108.1621 ... 7.172339419546474e-07
3 7386.6421 ... 3.1515218602729543e-07
4 8138.5957 ... 3.234000112590365e-07
5 8047.3140 ... 1.1086978875416937e-07
6 9006.2188 ... 4.827971234428365e-08
7 765.8052 ... 5.650294316943437e-07
8 1647.3230 ... 1.901099027593121e-07
9 6701.1450 ... 1.7682059984292007e-07
10 82.3648 ... 2.371577141482952e-08
... ... ... ...
16326 10042.8691 ... 7.28102017387155e-10
16327 9314.2891 ... 4.697087374899504e-10
16328 2892.7561 ... 5.128462010110903e-10
16329 1918.4792 ... 4.895995960910137e-10
16330 9603.0967 ... 6.038842945552168e-10
16331 3401.8242 ... 5.944864637670069e-10
16332 203.1676 ... 6.378315604719358e-10
16333 8055.8330 ... 6.638494393320452e-10
16334 924.1203 ... 6.757014476461435e-10
16335 2663.6299 ... 8.274179840640633e-10
Length = 16335 rows
Let’s have a look at how this changes the Data print statement.
[3]:
print(JOF_data_long)
****************************************
DATA OBJECT:
----------
SURVEY: JOF
VERSION: v11
PHOTOMETRIC CATALOGUE: /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits
APERTURE DIAMETERS: [0.32] arcsec
SELECTION BAND: NIRCam/F277W+F356W+F444W
****************************************
MULTIPLE_FILTER
----------
FACILITY: JWST
INSTRUMENT: NIRCam
FILTERS: ['F090W', 'F115W', 'F150W', 'F162M', 'F182M', 'F200W', 'F210M', 'F250M', 'F277W', 'F300M', 'F335M', 'F356W', 'F410M', 'F444W']
****************************************
NIRCam COMMON ATTRIBUTES:
----------
IM DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas
RMS ERR DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas
WHT DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas
MASK DIR: /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/auto
SEG DIR: /raid/scratch/work/austind/GALFIND_WORK/SExtractor/NIRCam/v11/JOF/MAP_RMS/segmentation
FORCED PHOT DIR: /raid/scratch/work/austind/GALFIND_WORK/SExtractor/NIRCam/v11/JOF/MAP_RMS/forced_phot/(0.32)as
IM EXT: 1
RMS ERR EXT: 2
WHT EXT: 4
ZP: 28.0865
PIX SCALE: 0.03 arcsec
DATA SHAPE: (4464, 10244)
MASK ARGS: {'method': 'auto', 'star_mask_params': {'central': {'a': 300.0, 'b': 4.25}, 'spikes': {'a': 400.0, 'b': 4.5}}, 'edge_mask_distance': 50, 'scale_extra': 0.2, 'exclude_gaia_galaxies': True, 'angle': -70.0, 'edge_value': 0.0, 'element': 'ELLIPSE', 'gaia_row_lim': 500}
SEG ARGS: {'err_type': 'rms_err', 'method': 'sextractor', 'config_name': 'default.sex', 'params_name': 'default.param'}
FORCED PHOT ARGS: {'forced_phot_band': NIRCam/F277W+F356W+F444W, 'err_type': 'rms_err', 'method': 'sextractor', 'config_name': 'default.sex', 'params_name': 'default.param', 'ra_label': 'ALPHA_J2000', 'dec_label': 'DELTA_J2000'}
DEPTH ARGS: {<Quantity 0.32 arcsec>: {'mode': 'n_nearest', 'scatter_size': 0.1, 'distance_to_mask': 30, 'region_radius_used_pix': 300, 'n_nearest': 200, 'coord_type': 'sky', 'split_depth_min_size': 100000, 'split_depths_factor': 5, 'step_size': 100, 'n_split': 'auto'}}
----------
****************************************
NIRCam/F090W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[4]
SEG PATH: JOF_F090W_F090W_sel_cat_v11_seg.fits
MASK PATH: F090W_auto.fits
FORCED PHOT PATH: JOF_F090W_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 29.81
MEAN DEPTH: 29.808
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F090W.h5
****************************************
NIRCam/F115W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[4]
SEG PATH: JOF_F115W_F115W_sel_cat_v11_seg.fits
MASK PATH: F115W_auto.fits
FORCED PHOT PATH: JOF_F115W_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 29.877
MEAN DEPTH: 29.863
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F115W.h5
****************************************
NIRCam/F150W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[4]
SEG PATH: JOF_F150W_F150W_sel_cat_v11_seg.fits
MASK PATH: F150W_auto.fits
FORCED PHOT PATH: JOF_F150W_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.118
MEAN DEPTH: 30.121
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F150W.h5
****************************************
NIRCam/F162M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[4]
SEG PATH: JOF_F162M_F162M_sel_cat_v11_seg.fits
MASK PATH: F162M_auto.fits
FORCED PHOT PATH: JOF_F162M_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 29.894
MEAN DEPTH: 29.889
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F162M.h5
****************************************
NIRCam/F182M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[4]
SEG PATH: JOF_F182M_F182M_sel_cat_v11_seg.fits
MASK PATH: F182M_auto.fits
FORCED PHOT PATH: JOF_F182M_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.259
MEAN DEPTH: 30.255
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F182M.h5
****************************************
NIRCam/F200W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[4]
SEG PATH: JOF_F200W_F200W_sel_cat_v11_seg.fits
MASK PATH: F200W_auto.fits
FORCED PHOT PATH: JOF_F200W_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.11
MEAN DEPTH: 30.105
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F200W.h5
****************************************
NIRCam/F210M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[4]
SEG PATH: JOF_F210M_F210M_sel_cat_v11_seg.fits
MASK PATH: F210M_auto.fits
FORCED PHOT PATH: JOF_F210M_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.072
MEAN DEPTH: 30.069
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F210M.h5
****************************************
NIRCam/F250M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[4]
SEG PATH: JOF_F250M_F250M_sel_cat_v11_seg.fits
MASK PATH: F250M_auto.fits
FORCED PHOT PATH: JOF_F250M_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 29.815
MEAN DEPTH: 29.815
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F250M.h5
****************************************
NIRCam/F277W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[4]
SEG PATH: JOF_F277W_F277W_sel_cat_v11_seg.fits
MASK PATH: F277W_auto.fits
FORCED PHOT PATH: JOF_F277W_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.384
MEAN DEPTH: 30.385
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F277W.h5
****************************************
NIRCam/F300M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[4]
SEG PATH: JOF_F300M_F300M_sel_cat_v11_seg.fits
MASK PATH: F300M_auto.fits
FORCED PHOT PATH: JOF_F300M_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.095
MEAN DEPTH: 30.097
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F300M.h5
****************************************
NIRCam/F335M
----------
IM NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[1]
RMS ERR NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[2]
WHT NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[4]
SEG PATH: JOF_F335M_F335M_sel_cat_v11_seg.fits
MASK PATH: F335M_auto.fits
FORCED PHOT PATH: JOF_F335M_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.19
MEAN DEPTH: 30.19
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F335M.h5
****************************************
NIRCam/F356W
----------
IM NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[4]
SEG PATH: JOF_F356W_F356W_sel_cat_v11_seg.fits
MASK PATH: F356W_auto.fits
FORCED PHOT PATH: JOF_F356W_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.41
MEAN DEPTH: 30.411
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F356W.h5
****************************************
NIRCam/F410M
----------
IM NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[1]
RMS ERR NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[2]
WHT NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[4]
SEG PATH: JOF_F410M_F410M_sel_cat_v11_seg.fits
MASK PATH: F410M_auto.fits
FORCED PHOT PATH: JOF_F410M_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 29.806
MEAN DEPTH: 29.807
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F410M.h5
****************************************
NIRCam/F444W
----------
IM NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[1]
RMS ERR NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[2]
WHT NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[4]
SEG PATH: JOF_F444W_F444W_sel_cat_v11_seg.fits
MASK PATH: F444W_auto.fits
FORCED PHOT PATH: JOF_F444W_F277W+F356W+F444W_sel_cat_v11.fits
----------
0.32 arcsec
----------
MEDIAN DEPTH: 30.272
MEAN DEPTH: 30.275
H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F444W.h5
****************************************
****************************************
For safety reasons, once the (path to the) photometric catalogue has been loaded into the Data object, it is not possible to re-run it. This is so that, for example, you don’t get confused between the products stemming from the previous catalogue and your newly loaded in one. To be clear, the overwrite parameter that we have been using simply states whether the pre-existing paths should be overwritten with the new data and NOT whether the data stored in the object should be updated.
Preventing stored paths from being overwritten in a particular object, however, does not entirely prevent you from changing the outputs of any methods run from those stored paths as the information is not cached in a single object, rather extracted from the data products when required. Let’s try re-producing this SExtractor forced photometric catalogue but instead using the F356W filter for selection in the same object to see what error message we get out of galfind.
[4]:
JOF_data_long.perform_forced_phot(forced_phot_band = "F356W")
CRITICAL:galfind:MASTER Photometric catalogue already exists!
Example 2: Running the Data pipeline
There is one last class method for the Data object that we havn’t quite covered yet, Data.pipeline() which again just takes survey and version inputs. This class method is what is used in the EPOCHS pipeline and essentially just chains the cataloguing steps in the previous notebooks together elegantly, skipping those that have already been executed in the past. For further details, please read the previous notebooks in this section if you have not already done so.
[5]:
# load the data object (short version)
JOF_data_short = Data.pipeline(
survey,
version,
instrument_names = instrument_names,
version_to_dir_dict = morgan_version_to_dir,
aper_diams = aper_diams,
forced_phot_band = forced_phot_band,
min_flux_pc_err = min_flux_pc_err
)
# ensure the two data objects are the same
assert JOF_data_short == JOF_data_long
INFO:galfind:Loaded aper_diams=<Quantity [0.32] arcsec> for F277W+F356W+F444W
INFO:galfind:Combined mask for NIRCam/F277W+F356W+F444W already exists at /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/combined/JOF_F277W+F356W+F444W_auto.fits
WARNING:galfind:Aperture correction columns already in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits
Calculating depths: 0%| | 0/15 [00:00<?, ?it/s]
INFO:galfind:Calculated/loaded depths for JOF v11 NIRCam
INFO:galfind:Local depth columns already exist in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits
Note that the two implementations are the same only if the default galfind pipeline parameters are used. Any deviation in masking, segmentation, performing forced photometry, running depths, or choice of PSF will produce differences between these two Data objects.
Fantastic! You’ve stuck it out through to the end of the Data class documentation. Feel free to now explore the next section which explores the galfind Catalogue class.