{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cataloguing the data\n",
    "\n",
    "The primary use for the galfind `Data` object is the creation of photometric catalogues for public and personal use. Once produced, these catalogues can be loaded into the `Catalogue` class to derive specific properties, which we cover in the [next section](../catalogue/catalogue.rst). In this first galfind release, we implement the ability to produce these using SExtractor only, although we aim to include other codes to perform forced photometry in the near future.\n",
    "\n",
    "The cataloguing procedure involves many different steps that have been explained in previous notebooks in this section. We outline these here.\n",
    "\n",
    "1. Instantiate a blank `Data` object from the reduced imaging\n",
    "2. Produce segmentation maps for each band using SExtractor\n",
    "3. Mask the data (i.e. image edges, stars, artefacts) either manually or automatically\n",
    "4. Perform forced photometry in a set of given apertures in either a single band or an inverse-variance weighted stack of bands.\n",
    "5. Aperture correct the fluxes based on a given model or empirical PSF\n",
    "6. Apply boolean mask columns to the catalogue to indicate whether each source falls in the mask\n",
    "7. Calculate local depths for each source\n",
    "8. Determine appropriate flux errors based on these depths, accounting for the correlated image noise\n",
    "\n",
    "We will create a `Data` object and associated .fits catalogue following these 8 steps in two ways, long (example 1) and short (example 2).\n",
    "\n",
    "## Example 1: Producing photometric catalogues\n",
    "\n",
    "To start with, we will load the same JOF `Data` object we have seen in previous examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading GALFIND config file from: /nvme/scratch/work/austind/GALFIND/galfind/../configs/galfind_config.ini\n",
      "Important:  Gaia archive will be intermittently unavailable due to scheduled maintenance on 14-10-2024 from 10:30 to 12:30 (CEST)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:galfind:Loaded aper_diams=<Quantity [0.32] arcsec> for F277W+F356W+F444W\n",
      "INFO:galfind:Combined mask for NIRCam/F277W+F356W+F444W already exists at /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/combined/JOF_F277W+F356W+F444W_auto.fits\n",
      "WARNING:galfind:Aperture correction columns already in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits\n",
      "Calculating depths:   0%|          | 0/15 [00:00<?, ?it/s]\n",
      "INFO:galfind:Calculated/loaded depths for JOF v11 NIRCam\n",
      "INFO:galfind:Local depth columns already exist in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from astropy.table import Table\n",
    "from copy import deepcopy\n",
    "import astropy.units as u\n",
    "\n",
    "from galfind import Stacked_Band_Data, Data\n",
    "from galfind.Data import morgan_version_to_dir\n",
    "\n",
    "survey = \"JOF\"\n",
    "version = \"v11\"\n",
    "instrument_names = [\"NIRCam\"]\n",
    "aper_diams = [0.32] * u.arcsec\n",
    "forced_phot_band = [\"F277W\", \"F356W\", \"F444W\"]\n",
    "min_flux_pc_err = 10.\n",
    "# 1\n",
    "JOF_data_long = Data.from_survey_version(\n",
    "    survey, \n",
    "    version, \n",
    "    instrument_names = instrument_names, \n",
    "    version_to_dir_dict = morgan_version_to_dir,\n",
    "    aper_diams = aper_diams,\n",
    "    forced_phot_band = forced_phot_band\n",
    ")\n",
    "# 2\n",
    "JOF_data_long.mask()\n",
    "# 3\n",
    "JOF_data_long.segment()\n",
    "# 4\n",
    "JOF_data_long.perform_forced_phot()\n",
    "# 5\n",
    "JOF_data_long.append_aper_corr_cols()\n",
    "# 6\n",
    "JOF_data_long.append_mask_cols()\n",
    "# 7\n",
    "JOF_data_long.run_depths()\n",
    "# 8\n",
    "JOF_data_long.append_loc_depth_cols(min_flux_pc_err = min_flux_pc_err)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will search the GALFIND_WORK directory for the individual forced photometry catalogues for each band and the resulting catalogue/README to ensure they exist and have been created correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Photometric catalogue exists at the expected path.\n",
      "  NUMBER     X_IMAGE   ... FLUXERR_APER_F444W_loc_depth_10pc_Jy\n",
      "               pix     ...                                     \n",
      "---------- ----------- ... ------------------------------------\n",
      "         1   9219.8145 ...                9.334392262202193e-07\n",
      "         2   6108.1621 ...                7.172339419546474e-07\n",
      "         3   7386.6421 ...               3.1515218602729543e-07\n",
      "         4   8138.5957 ...                3.234000112590365e-07\n",
      "         5   8047.3140 ...               1.1086978875416937e-07\n",
      "         6   9006.2188 ...                4.827971234428365e-08\n",
      "         7    765.8052 ...                5.650294316943437e-07\n",
      "         8   1647.3230 ...                1.901099027593121e-07\n",
      "         9   6701.1450 ...               1.7682059984292007e-07\n",
      "        10     82.3648 ...                2.371577141482952e-08\n",
      "       ...         ... ...                                  ...\n",
      "     16326  10042.8691 ...                 7.28102017387155e-10\n",
      "     16327   9314.2891 ...                4.697087374899504e-10\n",
      "     16328   2892.7561 ...                5.128462010110903e-10\n",
      "     16329   1918.4792 ...                4.895995960910137e-10\n",
      "     16330   9603.0967 ...                6.038842945552168e-10\n",
      "     16331   3401.8242 ...                5.944864637670069e-10\n",
      "     16332    203.1676 ...                6.378315604719358e-10\n",
      "     16333   8055.8330 ...                6.638494393320452e-10\n",
      "     16334    924.1203 ...                6.757014476461435e-10\n",
      "     16335   2663.6299 ...                8.274179840640633e-10\n",
      "Length = 16335 rows\n"
     ]
    }
   ],
   "source": [
    "# search for photometric catalogue\n",
    "if Path(JOF_data_long.phot_cat_path).is_file():\n",
    "    print(\"Photometric catalogue exists at the expected path.\")\n",
    "    # open the photometric catalogue\n",
    "    phot_cat = Table.read(JOF_data_long.phot_cat_path)\n",
    "    print(phot_cat)\n",
    "else:\n",
    "    print(\"Photometric catalogue does not exist at the expected path.\")\n",
    "\n",
    "# # search for README\n",
    "# readme_path = JOF_data_long.phot_cat_path.replace(\".fits\", \"_README.txt\")\n",
    "# if Path(readme_path).is_file():\n",
    "#     print(\"README exists at the expected path.\")\n",
    "#     # print the README\n",
    "#     with open(readme_path, \"r\") as f:\n",
    "#         print(f.read())\n",
    "#         f.close()\n",
    "# else:\n",
    "#     print(\"README does not exist at the expected path.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at how this changes the `Data` print statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************\n",
      "DATA OBJECT:\n",
      "----------\n",
      "SURVEY: JOF\n",
      "VERSION: v11\n",
      "PHOTOMETRIC CATALOGUE: /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits\n",
      "APERTURE DIAMETERS: [0.32] arcsec\n",
      "SELECTION BAND: NIRCam/F277W+F356W+F444W\n",
      "****************************************\n",
      "MULTIPLE_FILTER\n",
      "----------\n",
      "FACILITY: JWST\n",
      "INSTRUMENT: NIRCam\n",
      "FILTERS: ['F090W', 'F115W', 'F150W', 'F162M', 'F182M', 'F200W', 'F210M', 'F250M', 'F277W', 'F300M', 'F335M', 'F356W', 'F410M', 'F444W']\n",
      "****************************************\n",
      "NIRCam COMMON ATTRIBUTES:\n",
      "----------\n",
      "IM DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas\n",
      "RMS ERR DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas\n",
      "WHT DIR: /raid/scratch/data/jwst/JOF/NIRCam/mosaic_1084_wispnathan/30mas\n",
      "MASK DIR: /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/auto\n",
      "SEG DIR: /raid/scratch/work/austind/GALFIND_WORK/SExtractor/NIRCam/v11/JOF/MAP_RMS/segmentation\n",
      "FORCED PHOT DIR: /raid/scratch/work/austind/GALFIND_WORK/SExtractor/NIRCam/v11/JOF/MAP_RMS/forced_phot/(0.32)as\n",
      "IM EXT: 1\n",
      "RMS ERR EXT: 2\n",
      "WHT EXT: 4\n",
      "ZP: 28.0865\n",
      "PIX SCALE: 0.03 arcsec\n",
      "DATA SHAPE: (4464, 10244)\n",
      "MASK ARGS: {'method': 'auto', 'star_mask_params': {'central': {'a': 300.0, 'b': 4.25}, 'spikes': {'a': 400.0, 'b': 4.5}}, 'edge_mask_distance': 50, 'scale_extra': 0.2, 'exclude_gaia_galaxies': True, 'angle': -70.0, 'edge_value': 0.0, 'element': 'ELLIPSE', 'gaia_row_lim': 500}\n",
      "SEG ARGS: {'err_type': 'rms_err', 'method': 'sextractor', 'config_name': 'default.sex', 'params_name': 'default.param'}\n",
      "FORCED PHOT ARGS: {'forced_phot_band': NIRCam/F277W+F356W+F444W, 'err_type': 'rms_err', 'method': 'sextractor', 'config_name': 'default.sex', 'params_name': 'default.param', 'ra_label': 'ALPHA_J2000', 'dec_label': 'DELTA_J2000'}\n",
      "DEPTH ARGS: {<Quantity 0.32 arcsec>: {'mode': 'n_nearest', 'scatter_size': 0.1, 'distance_to_mask': 30, 'region_radius_used_pix': 300, 'n_nearest': 200, 'coord_type': 'sky', 'split_depth_min_size': 100000, 'split_depths_factor': 5, 'step_size': 100, 'n_split': 'auto'}}\n",
      "----------\n",
      "****************************************\n",
      "NIRCam/F090W\n",
      "----------\n",
      "IM NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[2]\n",
      "WHT NAME: jw01210-o001_t002_nircam_clear-f090w_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F090W_F090W_sel_cat_v11_seg.fits\n",
      "MASK PATH: F090W_auto.fits\n",
      "FORCED PHOT PATH: JOF_F090W_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 29.81\n",
      "MEAN DEPTH: 29.808\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F090W.h5\n",
      "****************************************\n",
      "NIRCam/F115W\n",
      "----------\n",
      "IM NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[2]\n",
      "WHT NAME: jw01210-o001_t002_nircam_clear-f115w_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F115W_F115W_sel_cat_v11_seg.fits\n",
      "MASK PATH: F115W_auto.fits\n",
      "FORCED PHOT PATH: JOF_F115W_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 29.877\n",
      "MEAN DEPTH: 29.863\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F115W.h5\n",
      "****************************************\n",
      "NIRCam/F150W\n",
      "----------\n",
      "IM NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[2]\n",
      "WHT NAME: jw01210-o001_t002_nircam_clear-f150w_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F150W_F150W_sel_cat_v11_seg.fits\n",
      "MASK PATH: F150W_auto.fits\n",
      "FORCED PHOT PATH: JOF_F150W_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.118\n",
      "MEAN DEPTH: 30.121\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F150W.h5\n",
      "****************************************\n",
      "NIRCam/F162M\n",
      "----------\n",
      "IM NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[2]\n",
      "WHT NAME: jw03215-o001_t002_nircam_clear-f162m_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F162M_F162M_sel_cat_v11_seg.fits\n",
      "MASK PATH: F162M_auto.fits\n",
      "FORCED PHOT PATH: JOF_F162M_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 29.894\n",
      "MEAN DEPTH: 29.889\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F162M.h5\n",
      "****************************************\n",
      "NIRCam/F182M\n",
      "----------\n",
      "IM NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[2]\n",
      "WHT NAME: jw03215-o001_t002_nircam_clear-f182m_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F182M_F182M_sel_cat_v11_seg.fits\n",
      "MASK PATH: F182M_auto.fits\n",
      "FORCED PHOT PATH: JOF_F182M_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.259\n",
      "MEAN DEPTH: 30.255\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F182M.h5\n",
      "****************************************\n",
      "NIRCam/F200W\n",
      "----------\n",
      "IM NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[2]\n",
      "WHT NAME: jw01210-o001_t002_nircam_clear-f200w_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F200W_F200W_sel_cat_v11_seg.fits\n",
      "MASK PATH: F200W_auto.fits\n",
      "FORCED PHOT PATH: JOF_F200W_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.11\n",
      "MEAN DEPTH: 30.105\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F200W.h5\n",
      "****************************************\n",
      "NIRCam/F210M\n",
      "----------\n",
      "IM NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[2]\n",
      "WHT NAME: jw03215-o001_t002_nircam_clear-f210m_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F210M_F210M_sel_cat_v11_seg.fits\n",
      "MASK PATH: F210M_auto.fits\n",
      "FORCED PHOT PATH: JOF_F210M_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.072\n",
      "MEAN DEPTH: 30.069\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F210M.h5\n",
      "****************************************\n",
      "NIRCam/F250M\n",
      "----------\n",
      "IM NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[2]\n",
      "WHT NAME: jw03215-o001_t002_nircam_clear-f250m_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F250M_F250M_sel_cat_v11_seg.fits\n",
      "MASK PATH: F250M_auto.fits\n",
      "FORCED PHOT PATH: JOF_F250M_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 29.815\n",
      "MEAN DEPTH: 29.815\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F250M.h5\n",
      "****************************************\n",
      "NIRCam/F277W\n",
      "----------\n",
      "IM NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[2]\n",
      "WHT NAME: jw01210-o001_t002_nircam_clear-f277w_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F277W_F277W_sel_cat_v11_seg.fits\n",
      "MASK PATH: F277W_auto.fits\n",
      "FORCED PHOT PATH: JOF_F277W_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.384\n",
      "MEAN DEPTH: 30.385\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F277W.h5\n",
      "****************************************\n",
      "NIRCam/F300M\n",
      "----------\n",
      "IM NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[2]\n",
      "WHT NAME: jw03215-o001_t002_nircam_clear-f300m_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F300M_F300M_sel_cat_v11_seg.fits\n",
      "MASK PATH: F300M_auto.fits\n",
      "FORCED PHOT PATH: JOF_F300M_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.095\n",
      "MEAN DEPTH: 30.097\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F300M.h5\n",
      "****************************************\n",
      "NIRCam/F335M\n",
      "----------\n",
      "IM NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[2]\n",
      "WHT NAME: jw03215-o001_t002_nircam_clear-f335m_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F335M_F335M_sel_cat_v11_seg.fits\n",
      "MASK PATH: F335M_auto.fits\n",
      "FORCED PHOT PATH: JOF_F335M_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.19\n",
      "MEAN DEPTH: 30.19\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F335M.h5\n",
      "****************************************\n",
      "NIRCam/F356W\n",
      "----------\n",
      "IM NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[2]\n",
      "WHT NAME: jw01210-o001_t002_nircam_clear-f356w_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F356W_F356W_sel_cat_v11_seg.fits\n",
      "MASK PATH: F356W_auto.fits\n",
      "FORCED PHOT PATH: JOF_F356W_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.41\n",
      "MEAN DEPTH: 30.411\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F356W.h5\n",
      "****************************************\n",
      "NIRCam/F410M\n",
      "----------\n",
      "IM NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[2]\n",
      "WHT NAME: jw01210-o001_t002_nircam_clear-f410m_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F410M_F410M_sel_cat_v11_seg.fits\n",
      "MASK PATH: F410M_auto.fits\n",
      "FORCED PHOT PATH: JOF_F410M_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 29.806\n",
      "MEAN DEPTH: 29.807\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F410M.h5\n",
      "****************************************\n",
      "NIRCam/F444W\n",
      "----------\n",
      "IM NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[1]\n",
      "RMS ERR NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[2]\n",
      "WHT NAME: jw04210-o001_t002_nircam_clear-f444w_i2dnobg.fits[4]\n",
      "SEG PATH: JOF_F444W_F444W_sel_cat_v11_seg.fits\n",
      "MASK PATH: F444W_auto.fits\n",
      "FORCED PHOT PATH: JOF_F444W_F277W+F356W+F444W_sel_cat_v11.fits\n",
      "----------\n",
      "0.32 arcsec\n",
      "----------\n",
      "MEDIAN DEPTH: 30.272\n",
      "MEAN DEPTH: 30.275\n",
      "H5 PATH: /raid/scratch/work/austind/GALFIND_WORK/Depths/NIRCam/v11/JOF/0.32as/n_nearest/F444W.h5\n",
      "****************************************\n",
      "****************************************\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(JOF_data_long)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For safety reasons, once the (path to the) photometric catalogue has been loaded into the Data object, it is not possible to re-run it. This is so that, for example, you don't get confused between the products stemming from the previous catalogue and your newly loaded in one. To be clear, the `overwrite` parameter that we have been using simply states whether the pre-existing paths should be overwritten with the new data and NOT whether the data stored in the object should be updated. Preventing stored paths from being overwritten in a particular object, however, does not entirely prevent you from changing the outputs of any methods run from those stored paths as the information is not cached in a single object, rather extracted from the data products when required. Let's try re-producing this SExtractor forced photometric catalogue but instead using the F356W filter for selection in the same object to see what error message we get out of galfind."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "CRITICAL:galfind:MASTER Photometric catalogue already exists!\n"
     ]
    }
   ],
   "source": [
    "JOF_data_long.perform_forced_phot(forced_phot_band = \"F356W\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2: Running the Data pipeline\n",
    "\n",
    "There is one last class method for the Data object that we havn't quite covered yet, `Data.pipeline()` which again just takes `survey` and `version` inputs. This class method is what is used in the EPOCHS pipeline and essentially just chains the cataloguing steps in the previous notebooks together elegantly, skipping those that have already been executed in the past. For further details, please read the previous notebooks in this section if you have not already done so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:galfind:Loaded aper_diams=<Quantity [0.32] arcsec> for F277W+F356W+F444W\n",
      "INFO:galfind:Combined mask for NIRCam/F277W+F356W+F444W already exists at /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/combined/JOF_F277W+F356W+F444W_auto.fits\n",
      "WARNING:galfind:Aperture correction columns already in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits\n",
      "Calculating depths:   0%|          | 0/15 [00:00<?, ?it/s]\n",
      "INFO:galfind:Calculated/loaded depths for JOF v11 NIRCam\n",
      "INFO:galfind:Local depth columns already exist in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits\n"
     ]
    }
   ],
   "source": [
    "# load the data object (short version)\n",
    "JOF_data_short = Data.pipeline(\n",
    "    survey, \n",
    "    version, \n",
    "    instrument_names = instrument_names, \n",
    "    version_to_dir_dict = morgan_version_to_dir,\n",
    "    aper_diams = aper_diams,\n",
    "    forced_phot_band = forced_phot_band,\n",
    "    min_flux_pc_err = min_flux_pc_err\n",
    ")\n",
    "\n",
    "# ensure the two data objects are the same\n",
    "assert JOF_data_short == JOF_data_long"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the two implementations are the same only if the default galfind pipeline parameters are used. Any deviation in masking, segmentation, performing forced photometry, running depths, or choice of PSF will produce differences between these two `Data` objects.\n",
    "\n",
    "Fantastic! You've stuck it out through to the end of the `Data` class documentation. Feel free to now explore the next section which explores the galfind [Catalogue](../catalogue/catalogue.rst) class."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "galfind_test",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}