{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making a catalogue object\n",
    "\n",
    "In order to construct a catalogue, we first require a `Catalogue_Creator` object which the user needs to create in order to appropriately load in the required data from a .fits catalogue. To load in a catalogue produced by galfind, we make use of the `Galfind_Catalogue_Creator` class in example 1. In example 2, we will scramble up our previously created JOF catalogue, changing the flux and flux error names, before making a custom `Catalogue_Creator` object which stores all the same information as in example 1. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading GALFIND config file from: /nvme/scratch/work/austind/GALFIND/galfind/../configs/galfind_config.ini\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:galfind:Aperture corrections for VISTA not found in /nvme/scratch/work/austind/GALFIND/galfind/Aperture_corrections/VISTA_aper_corr.txt\n",
      "WARNING:galfind:Aperture corrections for VISTA not found in /nvme/scratch/work/austind/GALFIND/galfind/Aperture_corrections/VISTA_aper_corr.txt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Failed to `import dust_attenuation`\n",
      "Install from the repo with $ pip install git+https://github.com/karllark/dust_attenuation.git\n"
     ]
    }
   ],
   "source": [
    "# imports\n",
    "import astropy.units as u\n",
    "from galfind import Data, Multiple_Filter, Catalogue_Creator, Catalogue\n",
    "from galfind.Data import morgan_version_to_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: Loading in a galfind catalogue from a Galfind Data object\n",
    "\n",
    "In this example, we will first produce a `Galfind_Catalogue_Creator` class before loading in the information stored in the JOF catalogue produced by the `Data` class in the [Data object notebooks](../data/data.rst). To start with, we shall load a JOF `Data` object as in previous notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:galfind:Loaded aper_diams=<Quantity [0.32] arcsec> for F277W+F356W+F444W\n",
      "INFO:galfind:Combined mask for NIRCam/F277W+F356W+F444W already exists at /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/combined/JOF_F277W+F356W+F444W_auto.fits\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forced_phot_band\n",
      "phot_cat_path\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: hdu= was not specified but multiple tables are present, reading in first available table (hdu=1) [astropy.io.fits.connect]\n",
      "WARNING:astroquery:hdu= was not specified but multiple tables are present, reading in first available table (hdu=1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "phot_cat_path\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:galfind:Aperture correction columns already in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits\n",
      "Calculating depths:   0%|          | 0/15 [00:00<?, ?it/s]\n",
      "INFO:galfind:Calculated/loaded depths for JOF v11 NIRCam\n",
      "INFO:galfind:Local depth columns already exist in /raid/scratch/work/austind/GALFIND_WORK/Catalogues/v11/NIRCam/JOF/(0.32)as/JOF_MASTER_Sel-F277W+F356W+F444W_v11.fits\n"
     ]
    }
   ],
   "source": [
    "survey = \"JOF\"\n",
    "version = \"v11\"\n",
    "instrument_names = [\"NIRCam\"]\n",
    "aper_diams = [0.32] * u.arcsec\n",
    "forced_phot_band = [\"F277W\", \"F356W\", \"F444W\"]\n",
    "min_flux_pc_err = 10.\n",
    "\n",
    "JOF_data = Data.pipeline(\n",
    "    survey, \n",
    "    version, \n",
    "    instrument_names = instrument_names, \n",
    "    version_to_dir_dict = morgan_version_to_dir,\n",
    "    aper_diams = aper_diams,\n",
    "    forced_phot_band = forced_phot_band,\n",
    "    min_flux_pc_err = min_flux_pc_err\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have loaded in the data, we shall create a basic `Catalogue_Creator` object using the handy `from_data` classmethod. As per the other galfind classes, we have overloaded the __repr__ and __str__ operators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:galfind:Catalogue cropped by 'ID' to [1]\n",
      "INFO:galfind:Loaded 'has_data_mask' from /raid/scratch/work/austind/GALFIND_WORK/Masks/JOF/has_data_mask/JOF_MASTER_Sel-F277W+F356W+F444W_v11.h5\n",
      "INFO:galfind:Making JOF v11 JOF_MASTER_Sel-F277W+F356W+F444W_v11 catalogue!\n",
      "WARNING:galfind:cat_aper_diams not in kwargs.keys()=dict_keys(['ZP', 'min_flux_pc_err'])! Setting to aper_diams=<Quantity [0.32] arcsec>\n",
      "WARNING:galfind:cat_aper_diams not in kwargs.keys()=dict_keys(['ZP', 'min_flux_pc_err'])! Setting to aper_diams=<Quantity [0.32] arcsec>\n",
      "WARNING:galfind:cat_aper_diams not in kwargs.keys()=dict_keys([])! Setting to aper_diams=<Quantity [0.32] arcsec>\n"
     ]
    },
    {
     "ename": "UnboundLocalError",
     "evalue": "local variable 'phot_obs_arr' referenced before assignment",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnboundLocalError\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[3], line 5\u001b[0m\n\u001b[1;32m      3\u001b[0m aper_diams \u001b[38;5;241m=\u001b[39m [\u001b[38;5;241m0.32\u001b[39m] \u001b[38;5;241m*\u001b[39m u\u001b[38;5;241m.\u001b[39marcsec\n\u001b[1;32m      4\u001b[0m cat_creator \u001b[38;5;241m=\u001b[39m Catalogue_Creator(survey, version, cat_path, filterset, aper_diams, crops \u001b[38;5;241m=\u001b[39m {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mID\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;241m1\u001b[39m})\n\u001b[0;32m----> 5\u001b[0m cat \u001b[38;5;241m=\u001b[39m \u001b[43mcat_creator\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      6\u001b[0m \u001b[38;5;28mrepr\u001b[39m(cat)\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28mprint\u001b[39m(cat)\n",
      "File \u001b[0;32m/nvme/scratch/work/austind/GALFIND/galfind/Catalogue_Creator.py:298\u001b[0m, in \u001b[0;36mCatalogue_Creator.__call__\u001b[0;34m(self, cropped)\u001b[0m\n\u001b[1;32m    296\u001b[0m filterset_arr \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mload_gal_filtersets(cropped)\n\u001b[1;32m    297\u001b[0m SED_results \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m--> 298\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(IDs) \u001b[38;5;241m==\u001b[39m \u001b[38;5;28mlen\u001b[39m(sky_coords) \u001b[38;5;241m==\u001b[39m \u001b[38;5;28mlen\u001b[39m(\u001b[43mphot_obs_arr\u001b[49m), \\\n\u001b[1;32m    299\u001b[0m     galfind_logger\u001b[38;5;241m.\u001b[39mcritical(\n\u001b[1;32m    300\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(IDs)\u001b[38;5;132;01m=}\u001b[39;00m\u001b[38;5;124m != \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(sky_coords)\u001b[38;5;132;01m=}\u001b[39;00m\u001b[38;5;124m != \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(phot_obs_arr)\u001b[38;5;132;01m=}\u001b[39;00m\u001b[38;5;124m!\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    301\u001b[0m     )\n\u001b[1;32m    302\u001b[0m phot_obs_arr \u001b[38;5;241m=\u001b[39m [{aper_diam: Photometry_obs(filterset_arr[i], \\\n\u001b[1;32m    303\u001b[0m     phot[aper_diam][i], phot_err[aper_diam][i], depths[aper_diam][i], \\\n\u001b[1;32m    304\u001b[0m     aper_diam, SED_results\u001b[38;5;241m=\u001b[39mSED_results) \u001b[38;5;28;01mfor\u001b[39;00m aper_diam \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39maper_diams} \\\n\u001b[1;32m    305\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;28mlen\u001b[39m(filterset_arr))]\n\u001b[1;32m    306\u001b[0m \u001b[38;5;66;03m# make an array of galaxy objects to be stored in the catalogue\u001b[39;00m\n",
      "\u001b[0;31mUnboundLocalError\u001b[0m: local variable 'phot_obs_arr' referenced before assignment"
     ]
    }
   ],
   "source": [
    "cat_creator_1 = Catalogue_Creator.from_data(JOF_data)\n",
    "repr(cat_creator_1)\n",
    "print(cat_creator_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that it is straightforwards to produce a `Catalogue` object simply by calling the `Catalogue_Creator` object. By default, the full catalogue is loaded, although we will see in example 3 that this can be updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_1a = cat_creator_1(cropped = False)\n",
    "repr(cat_1a)\n",
    "print(cat_1a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This catalogue can instead be explicitly created using the `Catalogue.pipeline` classmethod, done below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_1b = Catalogue.pipeline(\n",
    "    survey, \n",
    "    version, \n",
    "    instrument_names = instrument_names, \n",
    "    version_to_dir_dict = morgan_version_to_dir,\n",
    "    aper_diams = aper_diams,\n",
    "    forced_phot_band = forced_phot_band,\n",
    "    min_flux_pc_err = min_flux_pc_err\n",
    ")\n",
    "# ensure that the two implementations of catalogue loading are the same\n",
    "if cat_1a == cat_1b:\n",
    "    print(\"Catalogue 1a and 1b are the same\")\n",
    "else:\n",
    "    print(\"Catalogue 1a and 1b are different\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2: Loading a Galfind Catalogue object explicitly\n",
    "\n",
    "We note that the implementation in the above example can also be explicitly written instead using Catalogue_Creator.__init__. This __init__ method allows the user to also load in a catalogue created outside of galfind, so long as methods to load in the IDs, sky coordinates, and aperture photometry are given. In addition, if methods to load the depths and SED fitting properties are given, these can be loaded too. The __init__ arguments are extensive and we explicitly write these options below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: add more arguments to the __init__ here!\n",
    "cat_creator_2 = Catalogue_Creator(\n",
    "    survey, \n",
    "    version, \n",
    "    JOF_data.cat_path, \n",
    "    JOF_data.filterset, \n",
    "    aper_diams,\n",
    ")\n",
    "if cat_creator_1 == cat_creator_2:\n",
    "    print(\"Load-in methods are identical\")\n",
    "else:\n",
    "    print(\"Load-in methods are not identical\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that both the `Catalogue_Creator` and output cropped/uncropped `Catalogue` objects using the default arguments are exactly identical to those created by the `Catalogue_Creator.from_data()` classmethod."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_2 = cat_creator_2(cropped = False)\n",
    "if cat_1a == cat_2:\n",
    "    print(\"Uncropped catalogues are identical\")\n",
    "else:\n",
    "    print(\"Uncropped catalogues are not identical\")\n",
    "\n",
    "cat_3 = cat_creator_2(cropped = True)\n",
    "if cat_1a == cat_3:\n",
    "    print(\"Cropped catalogues are identical\")\n",
    "else:\n",
    "    print(\"Cropped catalogues are not identical\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "more_and_more_galfind",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}