gnuastro (0.22)

Browse

Build Log

Usage

This is gnuastro.info, produced by makeinfo version 7.1 from
gnuastro.texi.

This book documents version 0.22 of the GNU Astronomy Utilities
(Gnuastro). Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.

Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License,
Version 1.3 or any later version published by the Free Software
Foundation; with no Invariant Sections, no Front-Cover Texts, and
no Back-Cover Texts. A copy of the license is included in the
section entitled "GNU Free Documentation License".
INFO-DIR-SECTION Astronomy
START-INFO-DIR-ENTRY
* Gnuastro: (gnuastro). GNU Astronomy Utilities.
* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc.

* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help.

* bug-gnuastro: (gnuastro)Report a bug. How to report bugs

* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels.
* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic.

* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro's library.
* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram.

* ConvertType: (gnuastro)ConvertType. Convert different file types.
* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType.

* Convolve: (gnuastro)Convolve. Convolve an input file with kernel.
* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve.

* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params.
* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator.

* Crop: (gnuastro)Crop. Crop region(s) from image(s).
* astcrop: (gnuastro)Invoking astcrop. Options to Crop.

* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords.
* astfits: (gnuastro)Invoking astfits. Options to Fits.

* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image.
* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog.

* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles.
* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles.

* Match: (gnuastro)Match. Match two separate catalogs.
* astmatch: (gnuastro)Invoking astmatch. Options to Match.

* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise.
* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel.

* Segment: (gnuastro)Segment. Segment detections based on signal structure.
* astsegment: (gnuastro)Invoking astsegment. Options to Segment.

* Query: (gnuastro)Query. Access remote databases for downloading data.
* astquery: (gnuastro)Invoking astquery. Options to Query.

* Statistics: (gnuastro)Statistics. Get image Statistics.
* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics.

* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables.
* asttable: (gnuastro)Invoking asttable. Options to Table.

* Warp: (gnuastro)Warp. Warp a dataset to a new grid.
* astwarp: (gnuastro)Invoking astwarp. Options to Warp.

* astscript: (gnuastro)Installed scripts. Gnuastro's installed scripts.
* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script
* astscript-fits-view: (gnuastro)Invoking astscript-fits-view. Options to this script
* astscript-pointing-simulate: (gnuastro)Invoking astscript-pointing-simulate. Options to this script
* astscript-psf-scale-factor: (gnuastro)Invoking astscript-psf-scale-factor. Options to this script
* astscript-psf-select-stars: (gnuastro)Invoking astscript-psf-select-stars. Options to this script
* astscript-psf-stamp: (gnuastro)Invoking astscript-psf-stamp. Options to this script
* astscript-psf-subtract: (gnuastro)Invoking astscript-psf-subtract. Options to this script
* astscript-psf-unite: (gnuastro)Invoking astscript-psf-unite. Options to this script
* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script
* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script
* astscript-zeropoint: (gnuastro)Invoking astscript-zeropoint. Options to this script
END-INFO-DIR-ENTRY

File: gnuastro.info, Node: Top, Next: Introduction, Prev: (dir), Up: (dir)

GNU Astronomy Utilities
***********************

This book documents version 0.22 of the GNU Astronomy Utilities
(Gnuastro). Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.

* Menu:

* Introduction:: General introduction.
* Tutorials:: Tutorials or Cookbooks.
* Installation:: Requirements and installation.
* Common program behavior:: Common behavior in all programs.
* Data containers:: Tools to operate on extensions and tables.
* Data manipulation:: Tools for basic image manipulation.
* Data analysis:: Analyze images.
* Data modeling:: Modeling observed data.
* High-level calculations:: Physical calculations.
* Installed scripts:: Installed scripts that operate like programs.
* Makefile extensions:: Use Gnuastro's features as GNU Make functions.
* Library:: Gnuastro's library of useful functions.
* Developing:: The development environment.
* Other useful software:: Installing other useful software.
* GNU Free Doc License:: Full text of the GNU Free documentation license.
* GNU General Public License:: Full text of the GNU General public license.
* Index:: Index of terms

-- The Detailed Node Listing --

Introduction

* Quick start:: A quick start to installation.
* Gnuastro programs list:: List of command-line programs.
* Science and its tools:: Some philosophy and history.
* Your rights:: User rights.
* Logo of Gnuastro:: Meaning behind Gnuastro's logo.
* Naming convention:: About names of programs in Gnuastro.
* Version numbering:: Understanding version numbers.
* New to GNU/Linux?:: Suggested GNU/Linux distribution.
* Report a bug:: Search and report the bug you found.
* Suggest new feature:: How to suggest a new feature.
* Announcements:: How to stay up to date with Gnuastro.
* Conventions:: Conventions used in this book.
* Acknowledgments:: People who helped in the production.

Version numbering

* GNU Astronomy Utilities 1.0:: Plans for version 1.0 release

New to GNU/Linux?

* Command-line interface:: Introduction to the command-line

Tutorials

* General program usage tutorial:: Tutorial on many programs in generic scenario.
* Detecting large extended targets:: NoiseChisel for huge extended targets.
* Building the extended PSF:: How to extract an extended PSF from science data.
* Sufi simulates a detection:: Simulating a detection.
* Detecting lines and extracting spectra in 3D data:: Extracting spectra and emission line properties.
* Color images with full dynamic range:: Bright pixels with color, faint pixels in grayscale.
* Zero point of an image:: Estimate the zero point of an image.
* Pointing pattern design:: Optimizing the pointings of your observations.
* Moire pattern in stacking and its correction:: How to avoid this grid-based artifact.
* Clipping outliers:: How to avoid outliers in your measurements.

General program usage tutorial

* Calling Gnuastro's programs:: Easy way to find Gnuastro's programs.
* Accessing documentation:: Access to manual of programs you are running.
* Setup and data download:: Setup this template and download datasets.
* Dataset inspection and cropping:: Crop the flat region to use in next steps.
* Angular coverage on the sky:: Measure the field size on the sky.
* Cosmological coverage and visualizing tables:: Size in Mpc2, and plotting its change.
* Building custom programs with the library:: Easy way to build new programs.
* Option management and configuration files:: Dealing with options and configuring them.
* Warping to a new pixel grid:: Transforming/warping the dataset.
* NoiseChisel and Multi-Extension FITS files:: Running NoiseChisel and having multiple HDUs.
* NoiseChisel optimization for detection:: Check NoiseChisel's operation and improve it.
* NoiseChisel optimization for storage:: Dramatically decrease output's volume.
* Segmentation and making a catalog:: Finding true peaks and creating a catalog.
* Measuring the dataset limits:: One way to measure the "depth" of your data.
* Working with catalogs estimating colors:: Estimating colors using the catalogs.
* Column statistics color-magnitude diagram:: Visualizing column correlations.
* Aperture photometry:: Doing photometry on a fixed aperture.
* Matching catalogs:: Easily find corresponding rows from two catalogs.
* Reddest clumps cutouts and parallelization:: Parallelization and selecting a subset of the data.
* FITS images in a publication:: How to display FITS images in a PDF.
* Marking objects for publication:: How to mark some objects over the image in a PDF.
* Writing scripts to automate the steps:: Scripts will greatly help in re-doing things fast.
* Citing and acknowledging Gnuastro:: How to cite and acknowledge Gnuastro in your papers.

Detecting large extended targets

* Downloading and validating input data:: How to get and check the input data.
* NoiseChisel optimization:: Detect the extended and diffuse wings.
* Skewness caused by signal and its measurement:: How signal changes the distribution.
* Image surface brightness limit:: Standards to quantify the noise level.
* Achieved surface brightness level:: Calculate the outer surface brightness.
* Extract clumps and objects:: Find sub-structure over the detections.

Building the extended PSF

* Preparing input for extended PSF:: Which stars should be used?
* Saturated pixels and Segment's clumps:: Saturation is a major hurdle!
* One object for the whole detection:: Avoiding over-segmentation in objects.
* Building outer part of PSF:: Building the outermost PSF wings.
* Inner part of the PSF:: Going towards the PSF center.
* Uniting the different PSF components:: Merging all the components into one PSF.
* Subtracting the PSF:: Having the PSF, we now want to subtract it.

Detecting lines and extracting spectra in 3D data

* Viewing spectra and redshifted lines:: Interactively see the spectra of an object
* Sky lines in optical IFUs:: How to see sky lines in a cube.
* Continuum subtraction:: Removing the continuum from a data cube.
* 3D detection with NoiseChisel:: Finding emission-lines and their spectra.
* 3D measurements and spectra:: Measuring 3d properties including spectra.
* Extracting a single spectrum and plotting it:: Extracting a single vector row.
* Pseudo narrow-band images:: Collapsing the third dimension into a 2D image.

Color images with full dynamic range

* Color channels in same pixel grid:: Warping all inputs to the same pixel grid.
* Color image using linear transformation:: A linear color mapping won't show much!
* Color for bright regions and grayscale for faint:: Show the full dynamic range.
* Manually setting color-black-gray regions:: Physically motivated regions.
* Weights contrast markers and other customizations:: Nice ways to enhance visual appearance.

Zero point of an image

* Zero point tutorial with reference image:: Using a reference image.
* Zero point tutorial with reference catalog:: Using a reference catalog.

Pointing pattern design

* Preparing input and generating exposure map:: Download and image and build exposure map.
* Area of non-blank pixels on sky:: Account for the curved area on the sky.
* Script with pointing simulation steps so far:: Summary of steps for easy testing.
* Larger steps sizes for better calibration:: The initial small dither is not enough.
* Pointings that account for sky curvature:: Sky curvature will cause problems!
* Accounting for non-exposed pixels:: Parts of the detector do not get exposed to light.

Clipping outliers

* Building inputs and analysis without clipping:: Building a dataset for demonstration below.
* Sigma clipping:: Standard deviation (STD) clipping.
* MAD clipping:: Median Absolute Deviation (MAD) clipping.
* Filled re-clipping:: Two clips with holes filled in the middle.

Installation

* Dependencies:: Necessary packages for Gnuastro.
* Downloading the source:: Ways to download the source code.
* Build and install:: Configure, build and install Gnuastro.

Dependencies

* Mandatory dependencies:: Gnuastro will not install without these.
* Optional dependencies:: Adding more functionality.
* Bootstrapping dependencies:: If you have the version controlled source.
* Dependencies from package managers:: Installing from OS package managers.

Mandatory dependencies

* GNU Scientific Library:: Installing GSL.
* CFITSIO:: C interface to the FITS standard.
* WCSLIB:: C interface to the WCS standard of FITS.

Downloading the source

* Release tarball:: Download a stable official release.
* Version controlled source:: Get and use the version controlled source.

Version controlled source

* Bootstrapping:: Adding all the automatically generated files.
* Synchronizing:: Keep your local clone up to date.

Build and install

* Configuring:: Configure Gnuastro
* Separate build and source directories:: Keeping derivate/build files separate.
* Tests:: Run tests to see if it is working.
* A4 print book:: Customize the print book.
* Known issues:: Issues you might encounter.

Configuring

* Gnuastro configure options:: Configure options particular to Gnuastro.
* Installation directory:: Specify the directory to install.
* Executable names:: Changing executable names.
* Configure and build in RAM:: For minimal use of HDD or SSD, and clean source.

Common program behavior

* Command-line:: How to use the command-line.
* Configuration files:: Values for unspecified variables.
* Getting help:: Getting more information on the go.
* Multi-threaded operations:: How threads are managed in Gnuastro.
* Numeric data types:: Different types and how to specify them.
* Memory management:: How memory is allocated (in RAM or HDD/SSD).
* Tables:: Recognized table formats.
* Tessellation:: Tile the dataset into non-overlapping bins.
* Automatic output:: About automatic output names.
* Output FITS files:: Common properties when outputs are FITS.
* Numeric locale:: Decimal point printed like 0.5 instead of 0,5.

Command-line

* Arguments and options:: Different ways to specify inputs and configuration.
* Common options:: Options that are shared between all programs.
* Shell TAB completion:: Customized TAB completion in Gnuastro.
* Standard input:: Using output of another program as input.
* Shell tips:: Useful tips and tricks for program usage.

Arguments and options

* Arguments:: For specifying the main input files/operations.
* Options:: For configuring the behavior of the program.

Common options

* Input output options:: Common input/output options.
* Processing options:: Options for common processing steps.
* Operating mode options:: Common operating mode options.

Shell tips

* Separate shell variables for multiple outputs:: When you get values from one command.
* Truncating start of long string FITS keyword values:: When the end of the string matters.

Configuration files

* Configuration file format:: ASCII format of configuration file.
* Configuration file precedence:: Precedence of configuration files.
* Current directory and User wide:: Local and user configuration files.
* System wide:: System wide configuration files.

Getting help

* --usage:: View option names and value formats.
* --help:: List all options with description.
* Man pages:: Man pages generated from -help.
* Info:: View complete book in terminal.
* help-gnuastro mailing list:: Contacting experienced users.

Multi-threaded operations

* A note on threads:: Caution and suggestion on using threads.
* How to run simultaneous operations:: How to run things simultaneously.

Tables

* Recognized table formats:: Table formats that are recognized in Gnuastro.
* Gnuastro text table format:: Gnuastro's convention plain text tables.
* Selecting table columns:: Identify/select certain columns from a table

Recognized table formats

* Gnuastro text table format:: Reading plain text tables

Data containers

* Fits:: View and manipulate extensions and keywords.
* ConvertType:: Convert data to various formats.
* Table:: Read and Write FITS tables to plain text.
* Query:: Import data from external databases.

Fits

* Invoking astfits:: Arguments and options to Header.

Invoking Fits

* HDU information and manipulation:: Learn about the HDUs and move them.
* Keyword inspection and manipulation:: Manipulate metadata keywords in a HDU.
* Pixel information images:: Pixel values contain information on the pixels.

ConvertType

* Raster and Vector graphics:: Images coming from nature, and the abstract.
* Recognized file formats:: Recognized file formats
* Color:: Some explanations on color.
* Annotations for figure in paper:: Adding coordinates or physical scale.
* Invoking astconvertt:: Options and arguments to ConvertType.

Color

* Pixel colors:: Multiple filters in each pixel.
* Colormaps for single-channel pixels:: Better display of single-filter images.
* Vector graphics colors::

Annotations for figure in paper

* Full script of annotations on figure:: All the steps in one script

Invoking ConvertType

* ConvertType input and output:: Input/output file names and formats.
* Pixel visualization:: Visualizing the pixels in the output.
* Drawing with vector graphics:: Adding marks in many shapes and colors over the pixels.

Table

* Printing floating point numbers:: Optimal storage of floating point types.
* Vector columns:: How to keep more than one value in each column.
* Column arithmetic:: How to do operations on table columns.
* Operation precedence in Table:: Order of running options in Table.
* Invoking asttable:: Options and arguments to Table.

Query

* Available databases:: List of available databases to Query.
* Invoking astquery:: Inputs, outputs and configuration of Query.

Data manipulation

* Crop:: Crop region(s) from a dataset.
* Arithmetic:: Arithmetic on input data.
* Convolve:: Convolve an image with a kernel.
* Warp:: Warp/Transform an image to a different grid.

Crop

* Crop modes:: Basic modes to define crop region.
* Crop section syntax:: How to define a section to crop.
* Blank pixels:: Pixels with no value.
* Invoking astcrop:: Calling Crop on the command-line

Invoking Crop

* Crop options:: A list of all the options with explanation.
* Crop output:: The outputs of Crop.
* Crop known issues:: Known issues in running Crop.

Arithmetic

* Reverse polish notation:: The current notation style for Arithmetic.
* Integer benefits and pitfalls:: Integers have benefits, but require care.
* Noise basics:: Introduction various noise models.
* Arithmetic operators:: List of operators known to Arithmetic.
* Invoking astarithmetic:: How to run Arithmetic: options and output.

Noise basics

* Photon counting noise:: Poisson noise
* Instrumental noise:: Readout, dark current and other sources.
* Final noised pixel value:: How the final noised value is calculated.
* Generating random numbers:: How random numbers are generated.

Arithmetic operators

* Basic mathematical operators:: For example, +, -, /, log, and pow.
* Trigonometric and hyperbolic operators:: sin, cos, atan, asinh, etc.
* Constants:: Physical and Mathematical constants.
* Coordinate conversion operators:: For example equatorial J2000 to Galactic.
* Unit conversion operators:: Various unit conversions necessary.
* Statistical operators:: Statistics of a single dataset (for example, mean).
* Stacking operators:: Coadding or combining multiple datasets into one.
* Filtering operators:: Smoothing a dataset through mixing pixel with neighbors.
* Pooling operators:: Reducing size through statistics of pixels in window.
* Interpolation operators:: Giving blank pixels a value.
* Dimensionality changing operators:: Collapse or expand a dataset.
* Conditional operators:: Select certain pixels within the dataset.
* Mathematical morphology operators:: Work on binary images, for example, erode.
* Bitwise operators:: Work on bits within one pixel.
* Numerical type conversion operators:: Convert the numeric datatype of a dataset.
* Random number generators:: Random numbers can be used to add noise for example.
* Coordinate and border operators:: Return edges of 2D boxes.
* Loading external columns:: Read a column from a table into the stack.
* Size and position operators:: Extracting image size and pixel positions.
* Building new dataset and stack management:: How to construct an empty dataset from scratch.
* Operand storage in memory or a file:: Tools for complex operations in one command.

Convolve

* Spatial domain convolution:: Only using the input image values.
* Frequency domain and Fourier operations:: Using frequencies in input.
* Spatial vs. Frequency domain:: When to use which?
* Convolution kernel:: How to specify the convolution kernel.
* Invoking astconvolve:: Options and argument to Convolve.

Spatial domain convolution

* Convolution process:: More basic explanations.
* Edges in the spatial domain:: Dealing with the edges of an image.

Frequency domain and Fourier operations

* Fourier series historical background:: Historical background.
* Circles and the complex plane:: Interpreting complex numbers.
* Fourier series:: Fourier Series definition.
* Fourier transform:: Fourier Transform definition.
* Dirac delta and comb:: Dirac delta and Dirac comb.
* Convolution theorem:: Derivation of Convolution theorem.
* Sampling theorem:: Sampling theorem (Nyquist frequency).
* Discrete Fourier transform:: Derivation and explanation of DFT.
* Fourier operations in two dimensions:: Extend to 2D images.
* Edges in the frequency domain:: Interpretation of edge effects.

Warp

* Linear warping basics:: Basics of coordinate transformation.
* Merging multiple warpings:: How to merge multiple matrices.
* Resampling:: Warping an image is re-sampling it.
* Invoking astwarp:: Arguments and options for Warp.

Invoking Warp

* Align pixels with WCS considering distortions:: Default operation.
* Linear warps to be called explicitly:: Other warps.

Data analysis

* Statistics:: Calculate dataset statistics.
* NoiseChisel:: Detect objects in an image.
* Segment:: Segment detections based on signal structure.
* MakeCatalog:: Catalog from input and labeled images.
* Match:: Match two datasets.

Statistics

* Histogram and Cumulative Frequency Plot:: Basic definitions.
* 2D Histograms:: Plotting the distribution of two variables.
* Least squares fitting:: Fitting with various parametric functions.
* Sky value:: Definition and derivation of the Sky value.
* Invoking aststatistics:: Arguments and options to Statistics.

2D Histograms

* 2D histogram as a table for plotting:: Format and usage in table format.
* 2D histogram as an image:: Format and usage in image format

Sky value

* Sky value definition:: Definition of the Sky/reference value.
* Sky value misconceptions:: Wrong methods to estimate the Sky value.
* Quantifying signal in a tile:: Method to estimate the presence of signal.

Invoking Statistics

* Input to Statistics:: How to specify the inputs to Statistics.
* Single value measurements:: Can be used together (like -mean, or -maximum).
* Generating histograms and cumulative frequency plots:: Histogram and CFP tables.
* Fitting options:: Least squares fitting.
* Contour options:: Table of contours.
* Statistics on tiles:: Possible to do single-valued measurements on tiles.

NoiseChisel

* NoiseChisel changes after publication:: Updates since published papers.
* Invoking astnoisechisel:: Options and arguments for NoiseChisel.

Invoking NoiseChisel

* NoiseChisel input:: NoiseChisel's input options.
* Detection options:: Configure detection in NoiseChisel.
* NoiseChisel output:: NoiseChisel's output options and format.

Segment

* Invoking astsegment:: Inputs, outputs and options to Segment

Invoking Segment

* Segment input:: Input files and options.
* Segmentation options:: Parameters of the segmentation process.
* Segment output:: Outputs of Segment

MakeCatalog

* Detection and catalog production:: Discussing why/how to treat these separately
* Brightness flux magnitude:: More on Magnitudes, surface brightness, etc.
* Quantifying measurement limits:: For comparing different catalogs.
* Measuring elliptical parameters:: Estimating elliptical parameters.
* Adding new columns to MakeCatalog:: How to add new columns.
* MakeCatalog measurements:: List of all the measurements/columns by MakeCatalog.
* Invoking astmkcatalog:: Options and arguments to MakeCatalog.

Quantifying measurement limits

* Standard deviation vs error:: The std is not a measure of the error.
* Magnitude measurement error of each detection:: Error in measuring magnitude.
* Surface brightness error of each detection:: Error in measuring the Surface brightness.
* Completeness limit of each detection:: Possibility of detecting similar objects?
* Upper limit magnitude of each detection:: How reliable is your magnitude?
* Magnitude limit of image:: Measured magnitude of objects at certain S/N.
* Surface brightness limit of image:: Extrapolate per-pixel noise-level to standard units.
* Upper limit surface brightness of image:: Measure the noise-level for a certain aperture.

MakeCatalog measurements

* Identifier columns:: Identifying labels of each row (object/clumps).
* Position measurements in pixels:: Containing image/pixel (X/Y) measurements.
* Position measurements in WCS:: Containing WCS (for example RA/Dec) measurements.
* Brightness measurements:: Using pixel values of each label.
* Surface brightness measurements:: Various ways to measure surface brightness.
* Morphology measurements nonparametric:: Non-parametric morphology.
* Morphology measurements elliptical:: Elliptical morphology measurements.
* Measurements per slice spectra:: Measurements on each slice (like spectra).

Invoking MakeCatalog

* MakeCatalog inputs and basic settings:: Input files and basic settings.
* Upper-limit settings:: Settings for upper-limit measurements.
* MakeCatalog output:: File names of MakeCatalog's output table.

Match

* Matching algorithms:: Different ways to find the match
* Invoking astmatch:: Inputs, outputs and options of Match

Data modeling

* MakeProfiles:: Making mock galaxies and stars.

MakeProfiles

* Modeling basics:: Astronomical modeling basics.
* If convolving afterwards:: Considerations for convolving later.
* Profile magnitude:: Definition of total profile magnitude.
* Invoking astmkprof:: Inputs and Options for MakeProfiles.

Modeling basics

* Defining an ellipse and ellipsoid:: Definition of these important shapes.
* PSF:: Radial profiles for the PSF.
* Stars:: Making mock star profiles.
* Galaxies:: Radial profiles for galaxies.
* Sampling from a function:: Sample a function on a pixelated canvas.
* Oversampling:: Oversampling the model.

Invoking MakeProfiles

* MakeProfiles catalog:: Required catalog properties.
* MakeProfiles profile settings:: Configuration parameters for all profiles.
* MakeProfiles output dataset:: The canvas/dataset to build profiles over.
* MakeProfiles log file:: A description of the optional log file.

High-level calculations

* CosmicCalculator:: Calculate cosmological variables

CosmicCalculator

* Distance on a 2D curved space:: Distances in 2D for simplicity.
* Extending distance concepts to 3D:: Going to 3D (our real universe).
* Invoking astcosmiccal:: How to run CosmicCalculator.

Invoking CosmicCalculator

* CosmicCalculator input options:: Options to specify input conditions.
* CosmicCalculator basic cosmology calculations:: Such as distance modulus and distances.
* CosmicCalculator spectral line calculations:: How they get affected by redshift.

Installed scripts

* Sort FITS files by night:: Sort many files by date.
* Generate radial profile:: Radial profile of an object in an image.
* SAO DS9 region files from table:: Create ds9 region file from a table.
* Viewing FITS file contents with DS9 or TOPCAT:: Open DS9 (images/cubes) or TOPCAT (tables).
* Zero point estimation:: Zero point of an image from reference catalog or image(s).
* Pointing pattern simulation:: Simulate a stack with a given series of pointings.
* Color images with gray faint regions:: Color for bright pixels and grayscale for faint.
* PSF construction and subtraction:: Set of scripts to create extended PSF of an image.

Sort FITS files by night

* Invoking astscript-sort-by-night:: Inputs and outputs to this script.

Generate radial profile

* Invoking astscript-radial-profile:: How to call astscript-radial-profile

SAO DS9 region files from table

* Invoking astscript-ds9-region:: How to call astscript-ds9-region

Viewing FITS file contents with DS9 or TOPCAT

* Invoking astscript-fits-view:: How to call this script

Zero point estimation

* Invoking astscript-zeropoint:: How to call the script

Invoking astscript-zeropoint

* zero point output:: Format of the output.
* zero point options:: List and details of options.

Pointing pattern simulation

* Invoking astscript-pointing-simulate:: Options and running mode.

Color images with gray faint regions

* Invoking astscript-color-faint-gray:: Details of options and arguments.

PSF construction and subtraction

* Overview of the PSF scripts:: Summary of concepts and methods
* Invoking astscript-psf-select-stars:: Select good starts within an image.
* Invoking astscript-psf-stamp:: Make a stamp of each star to stack.
* Invoking astscript-psf-unite:: Merge stacks of different regions of PSF.
* Invoking astscript-psf-scale-factor:: Calculate factor to scale PSF to star.
* Invoking astscript-psf-subtract:: Put the PSF in the image to subtract.

Makefile extensions (for GNU Make)

* Loading the Gnuastro Make functions:: How to find and load Gnuastro's Make library.
* Makefile functions of Gnuastro:: The available functions.

Makefile functions of Gnuastro

* Text functions for Makefiles:: Basic text operations to supplement Make.
* Astronomy functions for Makefiles:: Astronomy/FITS related functions.

Library

* Review of library fundamentals:: Guide on libraries and linking.
* BuildProgram:: Link and run source files with this library.
* Gnuastro library:: Description of all library functions.
* Library demo programs:: Demonstration for using libraries.

Review of library fundamentals

* Headers:: Header files included in source.
* Linking:: Linking the compiled source files into one.
* Summary and example on libraries:: A summary and example on using libraries.

BuildProgram

* Invoking astbuildprog:: Options and examples for using this program.

Gnuastro library

* Configuration information:: General information about library config.
* Multithreaded programming:: Tools for easy multi-threaded operations.
* Library data types:: Definitions and functions for types.
* Pointers:: Wrappers for easy working with pointers.**
* Library blank values:: Blank values and functions to deal with them.
* Library data container:: General data container in Gnuastro.
* Dimensions:: Dealing with coordinates and dimensions.
* Linked lists:: Various types of linked lists.
* Array input output:: Reading and writing images or cubes.
* Table input output:: Reading and writing table columns.
* FITS files:: Working with FITS data.
* File input output:: Reading and writing to various file formats.
* World Coordinate System:: Dealing with the world coordinate system.
* Arithmetic on datasets:: Arithmetic operations on a dataset.
* Tessellation library:: Functions for working on tiles.
* Bounding box:: Finding the bounding box.
* Polygons:: Working with the vertices of a polygon.
* Qsort functions:: Helper functions for Qsort.
* K-d tree:: Space partitioning in K dimensions.
* Permutations:: Re-order (or permute) the values in a dataset.
* Matching:: Matching catalogs based on position.
* Statistical operations:: Functions for basic statistics.
* Fitting functions:: Fit independent and measured variables.
* Binary datasets:: Datasets that can only have values of 0 or 1.
* Labeled datasets:: Working with Segmented/labeled datasets.
* Convolution functions:: Library functions to do convolution.
* Pooling functions:: Reduce size of input by statistical methods.
* Interpolation:: Interpolate (over blank values possibly).
* Warp library:: Warp pixel grid to a new one.
* Color functions:: Definitions and operations related to colors.
* Git wrappers:: Wrappers for functions in libgit2.
* Python interface:: Functions to help in writing Python wrappers.
* Unit conversion library:: Converting between recognized units.
* Spectral lines library:: Functions for operating on Spectral lines.
* Cosmology library:: Cosmological calculations.
* SAO DS9 library:: Take inputs from files generated by SAO DS9.

Multithreaded programming (‘threads.h’)

* Implementation of pthread_barrier:: Some systems do not have pthread_barrier
* Gnuastro's thread related functions:: Functions for managing threads.

Data container (‘data.h’)

* Generic data container:: Definition of Gnuastro's generic container.
* Dataset allocation:: Allocate, initialize and free a dataset.
* Arrays of datasets:: Functions to help with array of datasets.
* Copying datasets:: Functions to copy a dataset to a new one.

Linked lists (‘list.h’)

* List of strings:: Simply linked list of strings.
* List of int32_t:: Simply linked list of int32_ts.
* List of size_t:: Simply linked list of size_ts.
* List of float:: Simply linked list of floats.
* List of double:: Simply linked list of doubles
* List of void:: Simply linked list of void * pointers.
* Ordered list of size_t:: Simply linked, ordered list of size_t.
* Doubly linked ordered list of size_t:: Definition and functions.
* List of gal_data_t:: Simply linked list Gnuastro's generic datatype.

FITS files (‘fits.h’)

* FITS macros errors filenames:: General macros, errors and checking names.
* CFITSIO and Gnuastro types:: Conversion between FITS and Gnuastro types.
* FITS HDUs:: Opening and getting information about HDUs.
* FITS header keywords:: Reading and writing FITS header keywords.
* FITS arrays:: Reading and writing FITS images/arrays.
* FITS tables:: Reading and writing FITS tables.

File input output

* Text files:: Reading and writing from/to plain text files.
* TIFF files:: Reading and writing from/to TIFF files.
* JPEG files:: Reading and writing from/to JPEG files.
* EPS files:: Writing to EPS files.
* PDF files:: Writing to PDF files.

Tessellation library (‘tile.h’)

* Independent tiles:: Work on or check independent tiles.
* Tile grid:: Cover a full dataset with non-overlapping tiles.

Library demo programs

* Library demo - reading a image:: Read a FITS image into memory.
* Library demo - inspecting neighbors:: Inspect the neighbors of a pixel.
* Library demo - multi-threaded operation:: Doing an operation on threads.
* Library demo - reading and writing table columns:: Simple Column I/O.
* Library demo - Warp to another image:: Output pixel grid and WCS from another image.
* Library demo - Warp to new grid:: Define a new pixel grid and WCS to resample the input.

Developing

* Why C:: Why Gnuastro is designed in C.
* Program design philosophy:: General ideas behind the package structure.
* Coding conventions:: Gnuastro coding conventions.
* Program source:: Conventions for the code.
* Documentation:: Documentation is an integral part of Gnuastro.
* Building and debugging:: Build and possibly debug during development.
* Test scripts:: Understanding the test scripts.
* Bash programmable completion:: Auto-completions for better user experience.
* Developer's checklist:: Checklist to finalize your changes.
* Gnuastro project webpage:: Central hub for Gnuastro activities.
* Developing mailing lists:: Stay up to date with Gnuastro's development.
* Contributing to Gnuastro:: Share your changes with all users.

Program source

* Mandatory source code files:: Description of files common to all programs.
* The TEMPLATE program:: Template for easy creation of a new program.

Bash programmable completion

* Bash TAB completion tutorial:: Fast tutorial to get you started on concepts.
* Implementing TAB completion in Gnuastro:: How Gnuastro uses Bash auto-completion features.

Contributing to Gnuastro

* Copyright assignment:: Copyright has to be assigned to the FSF.
* Commit guidelines:: Guidelines for commit messages.
* Production workflow:: Submitting your commits (work) for inclusion.
* Forking tutorial:: Tutorial on workflow steps with Git.

Other useful software

* SAO DS9:: Viewing FITS images.
* TOPCAT:: Plotting tables of data.
* PGPLOT:: Plotting directly in C.

File: gnuastro.info, Node: Introduction, Next: Tutorials, Prev: Top, Up: Top

1 Introduction
**************

GNU Astronomy Utilities (Gnuastro) is an official GNU package consisting
of separate programs and libraries for the manipulation and analysis of
astronomical data. All the programs share the same basic command-line
user interface for the comfort of both the users and developers.
Gnuastro is written to comply fully with the GNU coding standards so it
integrates finely with the GNU/Linux operating system. This also
enables astronomers to expect a fully familiar experience in the source
code, building, installing and command-line user interaction that they
have seen in all the other GNU software that they use. The official and
always up to date version of this book (or manual) is freely available
under *note GNU Free Doc License:: in various formats (PDF, HTML, plain
text, info, and as its Texinfo source) at
<http://www.gnu.org/software/gnuastro/manual/>.

For users who are new to the GNU/Linux environment, unless otherwise
specified most of the topics in *note Installation:: and *note Common
program behavior:: are common to all GNU software, for example,
installation, managing command-line options or getting help (also see
*note New to GNU/Linux?::). So if you are new to this empowering
environment, we encourage you to go through these chapters carefully.
They can be a starting point from which you can continue to learn more
from each program's own manual and fully benefit from and enjoy this
wonderful environment. Gnuastro also comes with a large set of
libraries, so you can write your own programs using Gnuastro's building
blocks, see *note Review of library fundamentals:: for an introduction.

In Gnuastro, no change to any program or library will be committed to
its history, before it has been fully documented here first. As
discussed in *note Science and its tools:: this is a founding principle
of the Gnuastro.

* Menu:

File: gnuastro.info, Node: Quick start, Next: Gnuastro programs list, Prev: Introduction, Up: Introduction

1.1 Quick start
===============

The latest official release tarball is always available as
‘gnuastro-latest.tar.lz’
(http://ftp.gnu.org/gnu/gnuastro/gnuastro-latest.tar.lz). The Lzip
(http://www.nongnu.org/lzip/lzip.html) format is used for better
compression (smaller output size, thus faster download), and robust
archival features and standards. For historical reasons (those users
that do not yet have Lzip), the Gzip'd tarball(1) is available at the
same URL (just change the ‘.lz’ suffix above to ‘.gz’; however, the
Lzip'd file is recommended). See *note Release tarball:: for more
details on the tarball release.

Let's assume the downloaded tarball is in the ‘TOPGNUASTRO’
directory. You can follow the commands below to download and
un-compress the Gnuastro source. You need to have the ‘lzip’ program
for the decompression (see *note Dependencies from package managers::)
If your Tar implementation does not recognize Lzip (the third command
fails), run the fourth command. Note that lines starting with ‘##’ do
not need to be typed (they are only a description of the following
command):

## Go into the download directory.
$ cd TOPGNUASTRO

## If you do not already have the tarball, you can download it:
$ wget http://ftp.gnu.org/gnu/gnuastro/gnuastro-latest.tar.lz

## If this fails, run the next command.
$ tar -xf gnuastro-latest.tar.lz

## Only when the previous command fails.
$ lzip -cd gnuastro-latest.tar.lz | tar -xf -

Gnuastro has three mandatory dependencies and some optional
dependencies for extra functionality, see *note Dependencies:: for the
full list. In *note Dependencies from package managers:: we have
prepared the command to easily install Gnuastro's dependencies using the
package manager of some operating systems. When the mandatory
dependencies are ready, you can configure, compile, check and install
Gnuastro on your system with the following commands. See *note Known
issues:: if you confront any complications.

$ cd gnuastro-X.X # Replace X.X with version number.
$ ./configure
$ make -j8 # Replace 8 with no. CPU threads.
$ make check -j8 # Replace 8 with no. CPU threads.
$ sudo make install

For each program there is an 'Invoke ProgramName' sub-section in this
book which explains how the programs should be run on the command-line
(for example, see *note Invoking asttable::).

In *note Tutorials::, we have prepared some complete tutorials with
common Gnuastro usage scenarios in astronomical research. They even
contain links to download the necessary data, and thoroughly describe
every step of the process (the science, statistics and optimal usage of
the command-line). We therefore recommend to read (an run the commands
in) the tutorials before starting to use Gnuastro.

---------- Footnotes ----------

(1) The Gzip library and program are commonly available on most
systems. However, Gnuastro recommends Lzip as described above and the
beta-releases are also only distributed in ‘tar.lz’.

File: gnuastro.info, Node: Gnuastro programs list, Next: Science and its tools, Prev: Quick start, Up: Introduction

1.2 Gnuastro programs list
==========================

One of the most common ways to operate Gnuastro is through its
command-line programs. For some tutorials on several real-world usage
scenarios, see *note Tutorials::. The list here is just provided as a
general summary for those who are new to Gnuastro.

GNU Astronomy Utilities 0.22, contains the following programs. They
are sorted in alphabetical order and a short description is provided for
each program. The description starts with the executable names in
‘thisfont’ followed by a pointer to the respective section in
parenthesis. Throughout this book, they are ordered based on their
context, please see the top-level contents for contextual ordering
(based on what they do).

Arithmetic
(‘astarithmetic’, see *note Arithmetic::) For arithmetic operations
on multiple (theoretically unlimited) number of datasets (images).
It has a large and growing set of arithmetic, mathematical, and
even statistical operators (for example, ‘+’, ‘-’, ‘*’, ‘/’,
‘sqrt’, ‘log’, ‘min’, ‘average’, ‘median’, see *note Arithmetic
operators::).

BuildProgram
(‘astbuildprog’, see *note BuildProgram::) Compile, link and run
custom C programs that depend on the Gnuastro library (see *note
Gnuastro library::). This program will automatically link with the
libraries that Gnuastro depends on, so there is no need to
explicitly mention them every time you are compiling a Gnuastro
library dependent program.

ConvertType
(‘astconvertt’, see *note ConvertType::) Convert astronomical data
files (FITS or IMH) to and from several other standard image and
data formats, for example, TXT, JPEG, EPS or PDF. Optionally, it is
also possible to add vector graphics markers over the output image
(for example, circles from catalogs containing RA or Dec).

Convolve
(‘astconvolve’, see *note Convolve::) Convolve (blur or smooth)
data with a given kernel in spatial and frequency domain on
multiple threads. Convolve can also do deconvolution to find the
appropriate kernel to PSF-match two images.

CosmicCalculator
(‘astcosmiccal’, see *note CosmicCalculator::) Do cosmological
calculations, for example, the luminosity distance, distance
modulus, comoving volume and many more.

Crop
(‘astcrop’, see *note Crop::) Crop region(s) from one or many
image(s) and stitch several images if necessary. Input coordinates
can be in pixel coordinates or world coordinates.

Fits
(‘astfits’, see *note Fits::) View and manipulate FITS file
extensions and header keywords.

MakeCatalog
(‘astmkcatalog’, see *note MakeCatalog::) Make catalog of labeled
image (output of NoiseChisel). The catalogs are highly
customizable and adding new calculations/columns is very
straightforward.

MakeProfiles
(‘astmkprof’, see *note MakeProfiles::) Make mock 2D profiles in an
image. The central regions of radial profiles are made with a
configurable 2D Monte Carlo integration. It can also build the
profiles on an over-sampled image.

Match
(‘astmatch’, see *note Match::) Given two input catalogs, find the
rows that match with each other within a given aperture (may be an
ellipse).

NoiseChisel
(‘astnoisechisel’, see *note NoiseChisel::) Detect signal in noise.
It uses a technique to detect very faint and diffuse, irregularly
shaped signal in noise (galaxies in the sky), using thresholds that
are below the Sky value, see Akhlaghi and Ichikawa 2015
(http://arxiv.org/abs/1505.01664).

Query
(‘astquery’, see *note Query::) High-level interface to query
pre-defined remote, or external databases, and directly download
the required sub-tables on the command-line.

Segment
(‘astsegment’, see *note Segment::) Segment detected regions based
on the structure of signal and the input dataset's noise
properties.

Statistics
(‘aststatistics’, see *note Statistics::) Statistical calculations
on the input dataset (column in a table, image or datacube). This
includes man operations such as generating histogram, sigma
clipping, and least squares fitting.

Table
(‘asttable’, *note Table::) Convert FITS binary and ASCII tables
into other such tables, print them on the command-line, save them
in a plain text file, do arithmetic on the columns or get the FITS
table information. For a full list of operations, see *note
Operation precedence in Table::.

Warp
(‘astwarp’, see *note Warp::) Warp image to new pixel grid. By
default it will align the pixel and WCS coordinates, removing any
non-linear WCS distortions. Any linear warp (projective
transformation or Homography) can also be applied to the input
images by explicitly calling the respective operation.

The programs listed above are designed to be highly modular and
generic. Hence, they are naturally for lower-level operations. In
Gnuastro, higher-level operations (combining multiple programs, or
running a program in a special way), are done with installed Bash
scripts (all prefixed with ‘astscript-’). They can be run just like a
program and behave very similarly (with minor differences, see *note
Installed scripts::).

‘astscript-ds9-region’
(See *note SAO DS9 region files from table::) Given a table (either
as a file or from standard input), create an SAO DS9 region file
from the requested positional columns (WCS or image coordinates).

‘astscript-fits-view’
(see *note Viewing FITS file contents with DS9 or TOPCAT::) Given
any number of FITS files, this script will either open SAO DS9 (for
images or cubes) or TOPCAT (for tables) to view them in a graphic
user interface (GUI).

‘astscript-pointing-simulate’
(See *note Pointing pattern simulation::) Given a table of
pointings on the sky, create and a reference image that contains
your camera's distortions and properties, generate a stacked
exposure map. This is very useful in testing the coverage of
dither patterns when designing your observing strategy and it is
highly customizable. See Akhlaghi 2023
(https://arxiv.org/abs/2310.15006), or the dedicated tutorial in
*note Pointing pattern design::.

‘astscript-radial-profile’
(See *note Generate radial profile::) Calculate the radial profile
of an object within an image. The object can be at any location in
the image, using various measures (median, sigma-clipped mean,
etc.), and the radial distance can also be measured on any general
ellipse. See Infante-Sainz et al. 2024
(https://arxiv.org/abs/2401.05303).

‘astscript-color-faint-gray’
(see *note Color images with gray faint regions::) Given three
images for the Red-Green-Blue (RGB) channels, this script will use
the bright pixels for color and will show the faint/diffuse regions
in grayscale. This greatly helps in visualizing the full dynamic
range of astronomical data. See Infante-Sainz et al. 2024
(https://arxiv.org/abs/2401.03814) or a dedicated tutorial in *note
Color images with full dynamic range::.

‘astscript-sort-by-night’
(See *note Sort FITS files by night::) Given a list of FITS files,
and a HDU and keyword name (for a date), this script separates the
files in the same night (possibly over two calendar days).

‘astscript-zeropoint’
(see *note Zero point estimation::) Estimate the zero point (to
calibrate pixel values) of an input image using a reference image
or a reference catalog. This is necessary to produce measurements
with physical units from new images. See Eskandarlou et al. 2023
(https://arxiv.org/abs/2312.04263), or a dedicated tutorial in
*note Zero point of an image::.

‘astscript-psf-*’
The following scripts are used to estimate the extended PSF
estimation and subtraction as described in the tutorial *note
Building the extended PSF:::

‘astscript-psf-select-stars’
(see *note Invoking astscript-psf-select-stars::) Find all the
stars within an image that are suitable for constructing an
extended PSF. If the image has WCS, this script can
automatically query Gaia to find the good stars.

‘astscript-psf-stamp’
(see *note Invoking astscript-psf-stamp::) build a crop
(stamp) of a certain width around a star at a certain
coordinate in a larger image. This script will do sub-pixel
re-positioning to make sure the star is centered and can
optionally mask all other background sources).

‘astscript-psf-scale-factor’
(see *note Invoking astscript-psf-scale-factor::) Given a PSF
model, and the central coordinates of a star in an image, find
the scale factor that has to be multiplied by the PSF to scale
it to that star.

‘astscript-psf-unite’
(see *note Invoking astscript-psf-unite::) Unite the various
components of a PSF into one. Because of saturation and
non-linearity, to get a good estimate of the extended PSF, it
is necessary to construct various parts from different
magnitude ranges.

‘astscript-psf-subtract’
(see *note Invoking astscript-psf-subtract::) Given the model
of a PSF and the central coordinates of a star in the image,
do sub-pixel re-positioning of the PSF, scale it to the star
and subtract it from the image.

File: gnuastro.info, Node: Science and its tools, Next: Your rights, Prev: Gnuastro programs list, Up: Introduction

1.3 Gnuastro manifesto: Science and its tools
=============================================

History of science indicates that there are always inevitably unseen
faults, hidden assumptions, simplifications and approximations in all
our theoretical models, data acquisition and analysis techniques. It is
precisely these that will ultimately allow future generations to advance
the existing experimental and theoretical knowledge through their new
solutions and corrections.

In the past, scientists would gather data and process them
individually to achieve an analysis thus having a much more intricate
knowledge of the data and analysis. The theoretical models also
required little (if any) simulations to compare with the data. Today
both methods are becoming increasingly more dependent on pre-written
software. Scientists are dissociating themselves from the intricacies
of reducing raw observational data in experimentation or from bringing
the theoretical models to life in simulations. These 'intricacies' are
precisely those unseen faults, hidden assumptions, simplifications and
approximations that define scientific progress.

Unfortunately, most persons who have recourse to a computer for
statistical analysis of data are not much interested either in
computer programming or in statistical method, being primarily
concerned with their own proper business. Hence the common use of
library programs and various statistical packages. ... It's time
that was changed.
-- _F.J. Anscombe. The American Statistician, Vol. 27, No. 1. 1973_

Anscombe's quartet
(http://en.wikipedia.org/wiki/Anscombe%27s_quartet) demonstrates how
four data sets with widely different shapes (when plotted) give nearly
identical output from standard regression techniques. Anscombe uses
this (now famous) quartet, which was introduced in the paper quoted
above, to argue that "_Good statistical analysis is not a purely routine
matter, and generally calls for more than one pass through the
computer_". Echoing Anscombe's concern after 44 years, some of the
highly recognized statisticians of our time (Leek, McShane, Gelman,
Colquhoun, Nuijten and Goodman), wrote in Nature that:

We need to appreciate that data analysis is not purely
computational and algorithmic - it is a human
behavior....Researchers who hunt hard enough will turn up a result
that fits statistical criteria - but their discovery will probably
be a false positive.
-- _Five ways to fix statistics, Nature, 551, Nov 2017._

Users of statistical (scientific) methods (software) are therefore
not passive (objective) agents in their results. It is necessary to
actually understand the method, not just use it as a black box. The
subjective experience gained by frequently using a method/software is
not sufficient to claim an understanding of how the tool/method works
and how relevant it is to the data and analysis. This kind of
subjective experience is prone to serious misunderstandings about the
data, what the software/statistical-method really does (especially as it
gets more complicated), and thus the scientific interpretation of the
result. This attitude is further encouraged through non-free
software(1), poorly written (or non-existent) scientific software
manuals, and non-reproducible papers(2). This approach to scientific
software and methods only helps in producing dogmas and an
"_obscurantist faith in the expert's special skill, and in his personal
knowledge and authority_"(3).

Program or be programmed. Choose the former, and you gain access
to the control panel of civilization. Choose the latter, and it
could be the last real choice you get to make.
-- _Douglas Rushkoff. Program or be programmed, O/R Books (2010)._

It is obviously impractical for any one human being to gain the
intricate knowledge explained above for every step of an analysis. On
the other hand, scientific data can be large and numerous, for example,
images produced by telescopes in astronomy. This requires efficient
algorithms. To make things worse, natural scientists have generally not
been trained in the advanced software techniques, paradigms and
architecture that are taught in computer science or engineering courses
and thus used in most software. The GNU Astronomy Utilities are an
effort to tackle this issue.

Gnuastro is not just a software, this book is as important to the
idea behind Gnuastro as the source code (software). This book has tried
to learn from the success of the "Numerical Recipes" book in educating
those who are not software engineers and computer scientists but still
heavy users of computational algorithms, like astronomers. There are
two major differences.

The first difference is that Gnuastro's code and the background
information are segregated: the code is moved within the actual Gnuastro
software source code and the underlying explanations are given here in
this book. In the source code, every non-trivial step is heavily
commented and correlated with this book, it follows the same logic of
this book, and all the programs follow a similar internal data, function
and file structure, see *note Program source::. Complementing the code,
this book focuses on thoroughly explaining the concepts behind those
codes (history, mathematics, science, software and usage advice when
necessary) along with detailed instructions on how to run the programs.
At the expense of frustrating "professionals" or "experts", this book
and the comments in the code also intentionally avoid jargon and
abbreviations. The source code and this book are thus intimately
linked, and when considered as a single entity can be thought of as a
real (an actual software accompanying the algorithms) "Numerical
Recipes" for astronomy.

The second major, and arguably more important, difference is that
"Numerical Recipes" does not allow you to distribute any code that you
have learned from it. In other words, it does not allow you to release
your software's source code if you have used their codes, you can only
publicly release binaries (a black box) to the community. Therefore,
while it empowers the privileged individual who has access to it, it
exacerbates social ignorance. Exactly at the opposite end of the
spectrum, Gnuastro's source code is released under the GNU general
public license (GPL) and this book is released under the GNU free
documentation license. You are therefore free to distribute any
software you create using parts of Gnuastro's source code or text, or
figures from this book, see *note Your rights::.

With these principles in mind, Gnuastro's developers aim to impose
the minimum requirements on you (in computer science, engineering and
even the mathematics behind the tools) to understand and modify any step
of Gnuastro if you feel the need to do so, see *note Why C:: and *note
Program design philosophy::.

Without prior familiarity and experience with optics, it is hard to
imagine how, Galileo could have come up with the idea of modifying the
Dutch military telescope optics to use in astronomy. Astronomical
objects could not be seen with the Dutch military design of the
telescope. In other words, it is unlikely that Galileo could have asked
a random optician to make modifications (not understood by Galileo) to
the Dutch design, to do something no astronomer of the time took
seriously. In the paradigm of the day, what could be the purpose of
enlarging geometric spheres (planets) or points (stars)? In that
paradigm only the position and movement of the heavenly bodies was
important, and that had already been accurately studied (recently by
Tycho Brahe).

In the beginning of his "The Sidereal Messenger" (published in 1610)
he cautions the readers on this issue and _before_ describing his
results/observations, Galileo instructs us on how to build a suitable
instrument. Without a detailed description of _how_ he made his tools
and done his observations, no reasonable person would believe his
results. Before he actually saw the moons of Jupiter, the mountains on
the Moon or the crescent of Venus, Galileo was “evasive”(4) to Kepler.
Science is defined by its tools/methods, _not_ its raw results(5).

The same is true today: science cannot progress with a black box, or
poorly released code. The source code of a research is the new
(abstractified) communication language in science, understandable by
humans _and_ computers. Source code (in any programming language) is a
language/notation designed to express all the details that would be too
tedious/long/frustrating to report in spoken languages like English,
similar to mathematic notation.

An article about computational science [almost all sciences today]
... is not the scholarship itself, it is merely advertising of the
scholarship. The Actual Scholarship is the complete software
development environment and the complete set of instructions which
generated the figures.
-- _Buckheit & Donoho, Lecture Notes in Statistics, Vol 103, 1996_

Today, the quality of the source code that goes into a scientific
result (and the distribution of that code) is as critical to scientific
vitality and integrity, as the quality of its written language/English
used in publishing/distributing its paper. A scientific paper will not
even be reviewed by any respectable journal if its written in a poor
language/English. A similar level of quality assessment is thus
increasingly becoming necessary regarding the codes/methods used to
derive the results of a scientific paper. For more on this, please see
Akhlaghi et al. 2021 (https://arxiv.org/abs/2006.03018)).

Bjarne Stroustrup (creator of the C++ language) says: "_Without
understanding software, you are reduced to believing in magic_". Ken
Thomson (the designer or the Unix operating system) says "_I abhor a
system designed for the 'user' if that word is a coded pejorative
meaning 'stupid and unsophisticated'_." Certainly no scientist (user of
a scientific software) would want to be considered a believer in magic,
or stupid and unsophisticated.

This can happen when scientists get too distant from the raw data and
methods, and are mainly discussing results. In other words, when they
feel they have tamed Nature into their own high-level (abstract) models
(creations), and are mainly concerned with scaling up, or
industrializing those results. Roughly five years before special
relativity, and about two decades before quantum mechanics fundamentally
changed Physics, Lord Kelvin is quoted as saying:

There is nothing new to be discovered in physics now. All that
remains is more and more precise measurement.
-- _William Thomson (Lord Kelvin), 1900_

A few years earlier Albert. A. Michelson made the following statement:

The more important fundamental laws and facts of physical science
have all been discovered, and these are now so firmly established
that the possibility of their ever being supplanted in consequence
of new discoveries is exceedingly remote.... Our future
discoveries must be looked for in the sixth place of decimals.
-- _Albert. A. Michelson, dedication of Ryerson Physics Lab, U. Chicago 1894_

If scientists are considered to be more than mere puzzle solvers(6)
(simply adding to the decimals of existing values or observing a feature
in 10, 100, or 100000 more galaxies or stars, as Kelvin and Michelson
clearly believed), they cannot just passively sit back and uncritically
repeat the previous (observational or theoretical) methods/tools on new
data. Today there is a wealth of raw telescope images ready (mostly for
free) at the finger tips of anyone who is interested with a fast enough
internet connection to download them. The only thing lacking is new
ways to analyze this data and dig out the treasure that is lying hidden
in them to existing methods and techniques.

New data that we insist on analyzing in terms of old ideas (that
is, old models which are not questioned) cannot lead us out of the
old ideas. However many data we record and analyze, we may just
keep repeating the same old errors, missing the same crucially
important things that the experiment was competent to find.
-- _Jaynes, Probability theory, the logic of science. Cambridge U. Press (2003)._

---------- Footnotes ----------

(1) <https://www.gnu.org/philosophy/free-sw.html>

(2) Where the authors omit many of the analysis/processing "details"
from the paper by arguing that they would make the paper too
long/unreadable. However, software engineers have been dealing with
such issues for a long time. There are thus software management
solutions that allow us to supplement papers with all the details
necessary to exactly reproduce the result. For example, see Akhlaghi et
al. 2021 (https://arxiv.org/abs/2006.03018).

(3) Karl Popper. The logic of scientific discovery. 1959. Larger
quote is given at the start of the PDF (for print) version of this book.

(4) Galileo G. (Translated by Maurice A. Finocchiaro). _The
essential Galileo_.Hackett publishing company, first edition, 2008.

(5) For example, take the following two results on the age of the
universe: roughly 14 billion years (suggested by the current consensus
of the standard model of cosmology) and less than 10,000 years
(suggested from some interpretations of the Bible). Both these numbers
are _results_. What distinguishes these two results, is the
tools/methods that were used to derive them. Therefore, as the term
"Scientific method" also signifies, a scientific statement it defined by
its _method_, not its result.

(6) Thomas S. Kuhn. _The Structure of Scientific Revolutions_,
University of Chicago Press, 1962.

File: gnuastro.info, Node: Your rights, Next: Logo of Gnuastro, Prev: Science and its tools, Up: Introduction

1.4 Your rights
===============

The paragraphs below, in this section, belong to the GNU Texinfo(1)
manual and are not written by us! The name "Texinfo" is just changed to
"GNU Astronomy Utilities" or "Gnuastro" because they are released under
the same licenses and it is beautifully written to inform you of your
rights.

GNU Astronomy Utilities is "free software"; this means that everyone
is free to use it and free to redistribute it on certain conditions.
Gnuastro is not in the public domain; it is copyrighted and there are
restrictions on its distribution, but these restrictions are designed to
permit everything that a good cooperating citizen would want to do.
What is not allowed is to try to prevent others from further sharing any
version of Gnuastro that they might get from you.

Specifically, we want to make sure that you have the right to give
away copies of the programs that relate to Gnuastro, that you receive
the source code or else can get it if you want it, that you can change
these programs or use pieces of them in new free programs, and that you
know you can do these things.

To make sure that everyone has such rights, we have to forbid you to
deprive anyone else of these rights. For example, if you distribute
copies of the Gnuastro related programs, you must give the recipients
all the rights that you have. You must make sure that they, too,
receive or can get the source code. And you must tell them their
rights.

Also, for our own protection, we must make certain that everyone
finds out that there is no warranty for the programs that relate to
Gnuastro. If these programs are modified by someone else and passed on,
we want their recipients to know that what they have is not what we
distributed, so that any problems introduced by others will not reflect
on our reputation.

The full text of the licenses for the Gnuastro book and software can
be respectively found in *note GNU General Public License::(2) and *note
GNU Free Doc License::(3).

---------- Footnotes ----------

(1) Texinfo is the GNU documentation system. It is used to create
this book in all the various formats.

(2) Also available in <http://www.gnu.org/copyleft/gpl.html>

(3) Also available in <http://www.gnu.org/copyleft/fdl.html>

File: gnuastro.info, Node: Logo of Gnuastro, Next: Naming convention, Prev: Your rights, Up: Introduction

1.5 Logo of Gnuastro
====================

Gnuastro's logo is an abstract image of a barred spiral galaxy
(https://en.wikipedia.org/wiki/Barred_spiral_galaxy). The galaxy is
vertically cut in half: on the left side, the beauty of a contiguous
galaxy image is visible. But on the right, the image gets pixelated,
and we only see the parts that are within the pixels. The pixels that
are more near to the center of the galaxy (which is brighter) are also
larger. But as we follow the spiral arms (and get more distant from the
center), the pixels get smaller (signifying less signal).

This sharp distinction between the contiguous and pixelated view of
the galaxy signifies the main struggle in science: in the "real" world,
objects are not pixelated or discrete and have no noise. However, when
we observe nature, we are confined and constrained by the resolution of
our data collection (CCD imager in this case).

On the other hand, we read English text from the left and progress
towards the right. This defines the positioning of the "real" and
observed halves of the galaxy: the no-noised and contiguous half (on the
left) passes through our observing tools and becomes pixelated and noisy
half (on the right). It is the job of scientific software like Gnuastro
to help interpret the underlying mechanisms of the "real" universe from
the pixelated and noisy data.

Gnuastro's logo was designed by Marjan Akbari. The concept behind it
was created after several design iterations with Mohammad Akhlaghi.

File: gnuastro.info, Node: Naming convention, Next: Version numbering, Prev: Logo of Gnuastro, Up: Introduction

1.6 Naming convention
=====================

Gnuastro is a package of independent programs and a collection of
libraries, here we are mainly concerned with the programs. Each program
has an official name which consists of one or two words, describing what
they do. The latter are printed with no space, for example, NoiseChisel
or Crop. On the command-line, you can run them with their executable
names which start with an ‘ast’ and might be an abbreviation of the
official name, for example, ‘astnoisechisel’ or ‘astcrop’, see *note
Executable names::.

We will use "ProgramName" for a generic official program name and
‘astprogname’ for a generic executable name. In this book, the programs
are classified based on what they do and thoroughly explained. An
alphabetical list of the programs that are installed on your system with
this installation are given in *note Gnuastro programs list::. That
list also contains the executable names and version numbers along with a
one line description.

File: gnuastro.info, Node: Version numbering, Next: New to GNU/Linux?, Prev: Naming convention, Up: Introduction

1.7 Version numbering
=====================

Gnuastro can have two formats of version numbers, for official and
unofficial releases. Official Gnuastro releases are announced on the
‘info-gnuastro’ mailing list, they have a version control tag in
Gnuastro's development history, and their version numbers are formatted
like "‘A.B’". ‘A’ is a major version number, marking a significant
planned achievement (for example, see *note GNU Astronomy Utilities
1.0::), while ‘B’ is a minor version number, see below for more on the
distinction. Note that the numbers are not decimals, so version 2.34 is
much more recent than version 2.5, which is not equal to 2.50.

Gnuastro also allows a unique version number for unofficial releases.
Unofficial releases can mark any point in Gnuastro's development
history. This is done to allow astronomers to easily use any point in
the version controlled history for their data-analysis and research
publication. See *note Version controlled source:: for a complete
introduction. This section is not just for developers and is intended
to straightforward and easy to read, so please have a look if you are
interested in the cutting-edge. This unofficial version number is a
meaningful and easy to read string of characters, unique to that
particular point of history. With this feature, users can easily stay
up to date with the most recent bug fixes and additions that are
committed between official releases.

The unofficial version number is formatted like: ‘A.B.C-D’. ‘A’ and
‘B’ are the most recent official version number. ‘C’ is the number of
commits that have been made after version ‘A.B’. ‘D’ is the first 4 or
5 characters of the commit hash number(1). Therefore, the unofficial
version number '‘3.92.8-29c8’', corresponds to the 8th commit after the
official version ‘3.92’ and its commit hash begins with ‘29c8’. The
unofficial version number is sort-able (unlike the raw hash) and as
shown above is descriptive of the state of the unofficial release. Of
course an official release is preferred for publication (since its
tarballs are easily available and it has gone through more tests, making
it more stable), so if an official release is announced prior to your
publication's final review, please consider updating to the official
release.

The major version number is set by a major goal which is defined by
the developers and user community before hand, for example, see *note
GNU Astronomy Utilities 1.0::. The incremental work done in minor
releases are commonly small steps in achieving the major goal.
Therefore, there is no limit on the number of minor releases and the
difference between the (hypothetical) versions 2.927 and 3.0 can be a
small (negligible to the user) improvement that finalizes the defined
goals.

* Menu:

* GNU Astronomy Utilities 1.0:: Plans for version 1.0 release

---------- Footnotes ----------

(1) Each point in Gnuastro's history is uniquely identified with a 40
character long hash which is created from its contents and previous
history for example: ‘5b17501d8f29ba3cd610673261e6e2229c846d35’. So the
string ‘D’ in the version for this commit could be ‘5b17’, or ‘5b175’.

File: gnuastro.info, Node: GNU Astronomy Utilities 1.0, Prev: Version numbering, Up: Version numbering

1.7.1 GNU Astronomy Utilities 1.0
---------------------------------

Currently (prior to Gnuastro 1.0), the aim of Gnuastro is to have a
complete system for data manipulation and analysis at least similar to
IRAF(1). So an astronomer can take all the standard data analysis steps
(starting from raw data to the final reduced product and standard
post-reduction tools) with the various programs in Gnuastro.

The maintainers of each camera or detector on a telescope can provide
a completely transparent shell script or Makefile to the observer for
data analysis. This script can set configuration files for all the
required programs to work with that particular camera. The script can
then run the proper programs in the proper sequence. The user/observer
can easily follow the standard shell script to understand (and modify)
each step and the parameters used easily. Bash (or other modern
GNU/Linux shell scripts) is powerful and made for this gluing job. This
will simultaneously improve performance and transparency. Shell
scripting (or Makefiles) are also basic constructs that are easy to
learn and readily available as part of the Unix-like operating systems.
If there is no program to do a desired step, Gnuastro's libraries can be
used to build specific programs.

The main factor is that all observatories or projects can freely
contribute to Gnuastro and all simultaneously benefit from it (since it
does not belong to any particular one of them), much like how for-profit
organizations (for example, RedHat, or Intel and many others) are major
contributors to free and open source software for their shared benefit.
Gnuastro's copyright has been fully awarded to GNU, so it does not
belong to any particular astronomer or astronomical facility or project.

---------- Footnotes ----------

(1) <http://iraf.noao.edu/>

File: gnuastro.info, Node: New to GNU/Linux?, Next: Report a bug, Prev: Version numbering, Up: Introduction

1.8 New to GNU/Linux?
=====================

Some astronomers initially install and use a GNU/Linux operating system
because their necessary tools can only be installed in this environment.
However, the transition is not necessarily easy. To encourage you in
investing the patience and time to make this transition, and actually
enjoy it, we will first start with a basic introduction to GNU/Linux
operating systems. Afterwards, in *note Command-line interface:: we
will discuss the wonderful benefits of the command-line interface, how
it beautifully complements the graphic user interface, and why it is
worth the (apparently steep) learning curve. Finally a complete chapter
(*note Tutorials::) is devoted to real world scenarios of using Gnuastro
(on the command-line). Therefore if you do not yet feel comfortable
with the command-line we strongly recommend going through that chapter
after finishing this section.

You might have already noticed that we are not using the name
"Linux", but "GNU/Linux". Please take the time to have a look at the
following essays and FAQs for a complete understanding of this very
important distinction.

• <https://gnu.org/philosophy>

• <https://www.gnu.org/gnu/the-gnu-project.html>

• <https://www.gnu.org/gnu/gnu-users-never-heard-of-gnu.html>

• <https://www.gnu.org/gnu/linux-and-gnu.html>

• <https://www.gnu.org/gnu/why-gnu-linux.html>

• <https://www.gnu.org/gnu/gnu-linux-faq.html>

• Recorded talk: <https://peertube.stream/w/ddeSSm33R1eFWKJVqpcthN>
(first 20 min is about the history of Unix-like operating systems).

In short, the Linux kernel(1) is built using the GNU C library
(glibc) and GNU compiler collection (gcc). The Linux kernel software
alone is just a means for other software to access the hardware
resources, it is useless alone! A normal astronomer (or scientist) will
never interact with the kernel directly! For example, the command-line
environment that you interact with is usually GNU Bash. It is GNU Bash
that then talks to kernel.

To better clarify, let's use this analogy inspired from one of the
links above(2): saying that you are "running Linux" is like saying you
are "driving your engine". The car's engine is the main source of power
in the car, no one doubts that. But you do not "drive" the engine, you
drive the "car". The engine alone is useless for transportation without
the radiator, battery, transmission, wheels, chassis, seats,
wind-shield, etc.

To have an operating system, you need lower-level tools (to build the
kernel), and higher-level (to use it) software packages. For the Linux
kernel, both the lower-level and higher-level tools are GNU. In other
words,"the whole system is basically GNU with Linux loaded".

You can replace the Linux kernel and still have the GNU shell and
higher-level utilities. For example, using the "Windows Subsystem for
Linux", you can use almost all GNU tools without the original Linux
kernel, but using the host Windows operating system, as in
<https://ubuntu.com/wsl>. Alternatively, you can build a fully
functional GNU-based working environment on a macOS or BSD-based
operating system (using the host's kernel and C compiler), for example,
through projects like Maneage, see Akhlaghi et al. 2021
(https://arxiv.org/abs/2006.03018), in particular Appendix C with all
the GNU software tools that is exactly reproducible on a macOS also.

Therefore to acknowledge GNU's instrumental role in the creation and
usage of the Linux kernel and the operating systems that use it, we
should call these operating systems "GNU/Linux".

* Menu:

* Command-line interface:: Introduction to the command-line

---------- Footnotes ----------

(1) In Unix-like operating systems, the kernel connects software and
hardware worlds.

(2) https://www.gnu.org/gnu/gnu-users-never-heard-of-gnu.html

File: gnuastro.info, Node: Command-line interface, Prev: New to GNU/Linux?, Up: New to GNU/Linux?

1.8.1 Command-line interface
----------------------------

One aspect of Gnuastro that might be a little troubling to new GNU/Linux
users is that (at least for the time being) it only has a command-line
user interface (CLI). This might be contrary to the mostly graphical
user interface (GUI) experience with proprietary operating systems.
Since the various actions available are not always on the screen, the
command-line interface can be complicated, intimidating, and frustrating
for a first-time user. This is understandable and also experienced by
anyone who started using the computer (from childhood) in a graphical
user interface (this includes most of Gnuastro's authors). Here we hope
to convince you of the unique benefits of this interface which can
greatly enhance your productivity while complementing your GUI
experience.

Through GNOME 3(1), most GNU/Linux based operating systems now have
an advanced and useful GUI. Since the GUI was created long after the
command-line, some wrongly consider the command-line to be obsolete.
Both interfaces are useful for different tasks. For example, you cannot
view an image, video, PDF document or web page on the command-line. On
the other hand you cannot reproduce your results easily in the GUI.
Therefore they should not be regarded as rivals but as complementary
user interfaces, here we will outline how the CLI can be useful in
scientific programs.

You can think of the GUI as a veneer over the CLI to facilitate a
small subset of all the possible CLI operations. Each click you do on
the GUI, can be thought of as internally running a different CLI
command. So asymptotically (if a good designer can design a GUI which
is able to show you all the possibilities to click on) the GUI is only
as powerful as the command-line. In practice, such graphical designers
are very hard to find for every program, so the GUI operations are
always a subset of the internal CLI commands. For programs that are
only made for the GUI, this results in not including lots of potentially
useful operations. It also results in 'interface design' to be a
crucially important part of any GUI program. Scientists do not usually
have enough resources to hire a graphical designer, also the complexity
of the GUI code is far more than CLI code, which is harmful for a
scientific software, see *note Science and its tools::.

For programs that have a GUI, one action on the GUI (moving and
clicking a mouse, or tapping a touchscreen) might be more efficient and
easier than its CLI counterpart (typing the program name and your
desired configuration). However, if you have to repeat that same action
more than once, the GUI will soon become frustrating and prone to
errors. Unless the designers of a particular program decided to design
such a system for a particular GUI action, there is no general way to
run any possible series of actions automatically on the GUI.

On the command-line, you can run any series of actions which can come
from various CLI capable programs you have decided yourself in any
possible permutation with one command(2). This allows for much more
creativity and exact reproducibility that is not possible to a GUI user.
For technical and scientific operations, where the same operation (using
various programs) has to be done on a large set of data files, this is
crucially important. It also allows exact reproducibility which is a
foundation principle for scientific results. The most common CLI (which
is also known as a shell) in GNU/Linux is GNU Bash, we strongly
encourage you to put aside several hours and go through this beautifully
explained web page: <https://flossmanuals.net/command-line/>. You do
not need to read or even fully understand the whole thing, only a
general knowledge of the first few chapters are enough to get you going.

Since the operations in the GUI are limited and they are visible,
reading a manual is not that important in the GUI (most programs do not
even have any!). However, to give you the creative power explained
above, with a CLI program, it is best if you first read the manual of
any program you are using. You do not need to memorize any details,
only an understanding of the generalities is needed. Once you start
working, there are more easier ways to remember a particular option or
operation detail, see *note Getting help::.

To experience the command-line in its full glory and not in the GUI
terminal emulator, press the following keys together: <CTRL+ALT+F4>(3)
to access the virtual console. To return back to your GUI, press the
same keys above replacing <F4> with <F7> (or <F1>, or <F2>, depending on
your GNU/Linux distribution). In the virtual console, the GUI, with all
its distracting colors and information, is gone. Enabling you to focus
entirely on your actual work.

For operations that use a lot of your system's resources (processing
a large number of large astronomical images for example), the virtual
console is the place to run them. This is because the GUI is not
competing with your research work for your system's RAM and CPU. Since
the virtual consoles are completely independent, you can even log out of
your GUI environment to give even more of your hardware resources to the
programs you are running and thus reduce the operating time.

Since it uses far less system resources, the CLI is also convenient
for remote access to your computer. Using secure shell (SSH) you can
log in securely to your system (similar to the virtual console) from
anywhere even if the connection speeds are low. There are apps for
smart phones and tablets which allow you to do this.

---------- Footnotes ----------

(1) <http://www.gnome.org/>

(2) By writing a shell script and running it, for example, see the
tutorials in *note Tutorials::.

(3) Instead of <F4>, you can use any of the keys from <F1> to <F6>
for different virtual consoles depending on your GNU/Linux distribution,
try them all out. You can also run a separate GUI from within this
console if you want to.

File: gnuastro.info, Node: Report a bug, Next: Suggest new feature, Prev: New to GNU/Linux?, Up: Introduction

1.9 Report a bug
================

According to Wikipedia "a software bug is an error, flaw, failure, or
fault in a computer program or system that causes it to produce an
incorrect or unexpected result, or to behave in unintended ways". So
when you see that a program is crashing, not reading your input
correctly, giving the wrong results, or not writing your output
correctly, you have found a bug. In such cases, it is best if you
report the bug to the developers. The programs will also inform you if
known impossible situations occur (which are caused by something
unexpected) and will ask the users to report the bug issue.

Prior to actually filing a bug report, it is best to search previous
reports. The issue might have already been found and even solved. The
best place to check if your bug has already been discussed is the bugs
tracker on *note Gnuastro project webpage:: at
<https://savannah.gnu.org/bugs/?group=gnuastro>. In the top search
fields (under "Display Criteria") set the "Open/Closed" drop-down menu
to "Any" and choose the respective program or general category of the
bug in "Category" and click the "Apply" button. The results colored
green have already been solved and the status of those colored in red is
shown in the table.

Recently corrected bugs are probably not yet publicly released
because they are scheduled for the next Gnuastro stable release. If the
bug is solved but not yet released and it is an urgent issue for you,
you can get the version controlled source and compile that, see *note
Version controlled source::.

To solve the issue as readily as possible, please follow the
following to guidelines in your bug report. The How to Report Bugs
Effectively (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html) and
How To Ask Questions The Smart Way
(http://catb.org/~esr/faqs/smart-questions.html) essays also provide
some good generic advice for all software (do not contact their authors
for Gnuastro's problems). Mastering the art of giving good bug reports
(like asking good questions) can greatly enhance your experience with
any free and open source software. So investing the time to read
through these essays will greatly reduce your frustration after you see
something does not work the way you feel it is supposed to for a large
range of software, not just Gnuastro.

*Be descriptive*
Please provide as many details as possible and be very descriptive.
Explain what you expected and what the output was: it might be that
your expectation was wrong. Also please clearly state which
sections of the Gnuastro book (this book), or other references you
have studied to understand the problem. This can be useful in
correcting the book (adding links to likely places where users will
check). But more importantly, it will be encouraging for the
developers, since you are showing how serious you are about the
problem and that you have actually put some thought into it. "To
be able to ask a question clearly is two-thirds of the way to
getting it answered." - John Ruskin (1819-1900).

*Individual and independent bug reports*
If you have found multiple bugs, please send them as separate (and
independent) bugs (as much as possible). This will significantly
help us in managing and resolving them sooner.

*Reproducible bug reports*
If we cannot exactly reproduce your bug, then it is very hard to
resolve it. So please send us a Minimal working example(1) along
with the description. For example, in running a program, please
send us the full command-line text and the output with the ‘-P’
option, see *note Operating mode options::. If it is caused only
for a certain input, also send us that input file. In case the
input FITS is large, please use Crop to only crop the problematic
section and make it as small as possible so it can easily be
uploaded and downloaded and not waste the archive's storage, see
*note Crop::.

There are generally two ways to inform us of bugs:

• Send a mail to ‘bug-gnuastro@gnu.org’. Any mail you send to this
address will be distributed through the bug-gnuastro mailing
list(2). This is the simplest way to send us bug reports. The
developers will then register the bug into the project web page
(next choice) for you.

• Use the Gnuastro project web page at
<https://savannah.gnu.org/projects/gnuastro/>: There are two ways
to get to the submission page as listed below. Fill in the form as
described below and submit it (see *note Gnuastro project webpage::
for more on the project web page).

• Using the top horizontal menu items, immediately under the top
page title. Hovering your mouse on "Support" will open a
drop-down list. Select "Submit new". Also if you have an
account in Savannah, you can choose "Bugs" in the menu items
and then select "Submit new".

• In the main body of the page, under the "Communication tools"
section, click on "Submit new item".

Once the items have been registered in the mailing list or web page,
the developers will add it to either the "Bug Tracker" or "Task Manager"
trackers of the Gnuastro project web page. These two trackers can only
be edited by the Gnuastro project developers, but they can be browsed by
anyone, so you can follow the progress on your bug. You are most
welcome to join us in developing Gnuastro and fixing the bug you have
found maybe a good starting point. Gnuastro is designed to be easy for
anyone to develop (see *note Science and its tools::) and there is a
full chapter devoted to developing it: *note Developing::.

*Savannah's Markup:* When posting to Savannah, it helps to have the code
displayed in mono-space font and a different background, you may also
want to make a list of items or make some words bold. For features like
these, you should use Savannah's "Markup" guide at
<https://savannah.gnu.org/markup-test.php>. You can access this page by
clicking on the "Full Markup" link that is just beside the "Preview"
button, near the box that you write your comments. As you see there,
for example when you want to high-light code, you should put it within a
"+verbatim+" and "-verbatim-" environment like below:

+verbatim+
astarithmetic image.fits image_arith.fits -h1 isblank nan where
-verbatim-

Unfortunately, Savannah doesn't have a way to edit submitted comments.
Therefore be sure to press the "Preview" button and check your report's
final format before the final submission.

---------- Footnotes ----------

(1) <http://en.wikipedia.org/wiki/Minimal_Working_Example>

(2) <https://lists.gnu.org/mailman/listinfo/bug-gnuastro>

File: gnuastro.info, Node: Suggest new feature, Next: Announcements, Prev: Report a bug, Up: Introduction

1.10 Suggest new feature
========================

We would always be happy to hear of suggested new features. For every
program, there are already lists of features that we are planning to
add. You can see the current list of plans from the Gnuastro project
web page at <https://savannah.gnu.org/projects/gnuastro/> and following
"Tasks"→"Browse" on the horizontal menu at the top of the page
immediately under the title, see *note Gnuastro project webpage::. If
you want to request a feature to an existing program, click on the
"Display Criteria" above the list and under "Category", choose that
particular program. Under "Category" you can also see the existing
suggestions for new programs or other cases like installation,
documentation or libraries. Also, be sure to set the "Open/Closed"
value to "Any".

If the feature you want to suggest is not already listed in the task
manager, then follow the steps that are fully described in *note Report
a bug::. Please have in mind that the developers are all busy with
their own astronomical research, and implementing existing "task"s to
add or resolve bugs. Gnuastro is a volunteer effort and none of the
developers are paid for their hard work. So, although we will try our
best, please do not expect for your suggested feature to be immediately
included (for the next release of Gnuastro).

The best person to apply the exciting new feature you have in mind is
you, since you have the motivation and need. In fact, Gnuastro is
designed for making it as easy as possible for you to hack into it (add
new features, change existing ones and so on), see *note Science and its
tools::. Please have a look at the chapter devoted to developing (*note
Developing::) and start applying your desired feature. Once you have
added it, you can use it for your own work and if you feel you want
others to benefit from your work, you can request for it to become part
of Gnuastro. You can then join the developers and start maintaining
your own part of Gnuastro. If you choose to take this path of action
please contact us beforehand (*note Report a bug::) so we can avoid
possible duplicate activities and get interested people in contact.

*Gnuastro is a collection of low level programs:* As described in *note
Program design philosophy::, a founding principle of Gnuastro is that
each library or program should be basic and low-level. High level jobs
should be done by running the separate programs or using separate
functions in succession through a shell script or calling the libraries
by higher level functions, see the examples in *note Tutorials::. So
when making the suggestions please consider how your desired job can
best be broken into separate steps and modularized.

File: gnuastro.info, Node: Announcements, Next: Conventions, Prev: Suggest new feature, Up: Introduction

1.11 Announcements
==================

Gnuastro has a dedicated mailing list for making announcements
(‘info-gnuastro’). Anyone can subscribe to this mailing list. Anytime
there is a new stable or test release, an email will be circulated
there. The email contains a summary of the overall changes along with a
detailed list (from the ‘NEWS’ file). This mailing list is thus the
best way to stay up to date with new releases, easily learn about the
updated/new features, or dependencies (see *note Dependencies::).

To subscribe to this list, please visit
<https://lists.gnu.org/mailman/listinfo/info-gnuastro>. Traffic (number
of mails per unit time) in this list is designed to be low: only a
handful of mails per year. Previous announcements are available on its
archive (http://lists.gnu.org/archive/html/info-gnuastro/).

File: gnuastro.info, Node: Conventions, Next: Acknowledgments, Prev: Announcements, Up: Introduction

1.12 Conventions
================

In this book we have the following conventions:

• All commands that are to be run on the shell (command-line) prompt
as the user start with a ‘$’. In case they must be run as a
superuser or system administrator, they will start with a single
‘#’. If the command is in a separate line and next line ‘is also
in the code type face’, but does not have any of the ‘$’ or ‘#’
signs, then it is the output of the command after it is run. As a
user, you do not need to type those lines. A line that starts with
‘##’ is just a comment for explaining the command to a human reader
and must not be typed.

• If the command becomes larger than the page width a <\> is inserted
in the code. If you are typing the code by hand on the
command-line, you do not need to use multiple lines or add the
extra space characters, so you can omit them. If you want to copy
and paste these examples (highly discouraged!) then the <\> should
stay.

The <\> character is a shell escape character which is used
commonly to make characters which have special meaning for the
shell, lose that special meaning (the shell will not treat them
especially if there is a <\> behind them). When <\> is the last
visible character in a line (the next character is a new-line
character) the new-line character loses its meaning. Therefore,
the shell sees it as a simple white-space character not the end of
a command! This enables you to use multiple lines to write your
commands.

This is not a convention, but a bi-product of the PDF building
process of the manual: In the PDF version of this manual, a single quote
(or apostrophe) character in the commands or codes is shown like this:
‘'’. Single quotes are sometimes necessary in combination with commands
like ‘awk’ or ‘sed’, or when using Column arithmetic in Gnuastro's own
Table (see *note Column arithmetic::). Therefore when typing
(recommended) or copy-pasting (not recommended) the commands that have a
‘'’, please correct it to the single-quote (or apostrophe) character,
otherwise the command will fail.

File: gnuastro.info, Node: Acknowledgments, Prev: Conventions, Up: Introduction

1.13 Acknowledgments
====================

Gnuastro would not have been possible without scholarships and grants
from several funding institutions. We thus ask that if you used
Gnuastro in any of your papers/reports, please add the proper citation
and acknowledge the funding agencies/projects. For details of which
papers to cite (may be different for different programs) and get the
acknowledgment statement to include in your paper, please run the
relevant programs with the common ‘--cite’ option like the example
commands below (for more on ‘--cite’, please see *note Operating mode
options::).

$ astnoisechisel --cite
$ astmkcatalog --cite

Here, we will acknowledge all the institutions (and their grants)
along with the people who helped make Gnuastro possible. The full list
of Gnuastro authors is available at the start of this book and the
‘AUTHORS’ file in the source code (both are generated automatically from
the version controlled history). The plain text file ‘THANKS’, which is
also distributed along with the source code, contains the list of people
and institutions who played an indirect role in Gnuastro (not committed
any code in the Gnuastro version controlled history).

The Japanese Ministry of Education, Culture, Sports, Science, and
Technology (MEXT) scholarship for Mohammad Akhlaghi's Masters and PhD
degree in Tohoku University Astronomical Institute had an instrumental
role in the long term learning and planning that made the idea of
Gnuastro possible. The very critical view points of Professor Takashi
Ichikawa (Mohammad's adviser) were also instrumental in the initial
ideas and creation of Gnuastro. Afterwards, the European Research
Council (ERC) advanced grant 339659-MUSICOS (Principal investigator:
Roland Bacon) was vital in the growth and expansion of Gnuastro.
Working with Roland at the Centre de Recherche Astrophysique de Lyon
(CRAL), enabled a thorough re-write of the core functionality of all
libraries and programs, turning Gnuastro into the large collection of
generic programs and libraries it is today. At the Instituto de
Astrofisica de Canarias (IAC, and in particular in collaboration with
Johan Knapen and Ignacio Trujillo), Gnuastro matured and its user base
significantly grew. Work on improving Gnuastro is now continuing
primarily in the Centro de Estudios de Física del Cosmos de Aragón
(CEFCA), located in Teruel, Spain.

In general, we would like to gratefully thank the following people
for their useful and constructive comments and suggestions (in
alphabetical order by family name): Valentina Abril-melgarejo, Marjan
Akbari, Carlos Allende Prieto, Hamed Altafi, Roland Bacon, Roberto Baena
Gallé, Zahra Bagheri, Karl Berry, Faezeh Bidjarchian, Leindert Boogaard,
Nicolas Bouché, Stefan Brüns, Fernando Buitrago, Adrian Bunk, Rosa
Calvi, Mark Calabretta Nushkia Chamba, Sergio Chueca Urzay, Tamara
Civera Lorenzo, Benjamin Clement, Nima Dehdilani, Andrés Del Pino
Molina, Antonio Diaz Diaz, Paola Dimauro, Alexey Dokuchaev, Pierre-Alain
Duc, Alessandro Ederoclite, Elham Eftekhari, Paul Eggert, Sepideh
Eskandarlou, Sílvia Farras, Juan Antonio Fernández Ontiveros, Gaspar
Galaz, Andrés García-Serra Romero, Zohre Ghaffari, Thérèse Godefroy,
Giulia Golini, Craig Gordon, Martin Guerrero Roncel, Madusha
Gunawardhana, Bruno Haible, Stephen Hamer, Siyang He, Zahra Hosseini,
Leslie Hunt, Takashi Ichikawa, Raúl Infante Sainz, Brandon Invergo,
Oryna Ivashtenko, Aurélien Jarno, Lee Kelvin, Brandon Kelly,
Mohammad-Reza Khellat, Johan Knapen, Geoffry Krouchi, Martin Kuemmel,
Teet Kuutma, Clotilde Laigle, Floriane Leclercq, Alan Lefor, Javier
Licandro, Jeremy Lim, Alejandro Lumbreras Calle, Sebastián Luna Valero,
Alberto Madrigal, Guillaume Mahler, Juan Miro, Alireza Molaeinezhad,
Javier Moldon, Juan Molina Tobar, Francesco Montanari, Raphael Morales,
Carlos Morales Socorro, Sylvain Mottet, Dmitrii Oparin, François
Ochsenbein, Bertrand Pain, William Pence, Irene Pintos Castro, Mamta
Pommier, Marcel Popescu, Bob Proulx, Joseph Putko, Samane Raji, Ignacio
Ruiz Cejudo, Teymoor Saifollahi, Joanna Sakowska, Elham Saremi, Nafise
Sedighi, Markus Schaney, Yahya Sefidbakht, Alejandro Serrano Borlaff,
Zahra Sharbaf, David Shupe, Leigh Smith, Jenny Sorce, Manuel
Sánchez-Benavente, Lee Spitler, Richard Stallman, Michael Stein, Ole
Streicher, Alfred M. Szmidt, Michel Tallon, Juan C. Tello, Vincenzo
Testa, Éric Thiébaut, Ignacio Trujillo, Peter Teuben, David
Valls-Gabaud, Jesús Varela, Aaron Watkins, Richard Wilbur, Michael H.F.
Wilkinson, Christopher Willmer, Xiuqin Wu, Sara Yousefi Taemeh, Johannes
Zabl. The GNU French Translation Team is also managing the French
version of the top Gnuastro web page which we highly appreciate.
Finally, we should thank all the (sometimes anonymous) people in various
online forums who patiently answered all our small (but important)
technical questions.

All work on Gnuastro has been voluntary, but the authors are most
grateful to the following institutions (in chronological order) for
hosting/supporting us in our research. Where necessary, these
institutions have disclaimed any ownership of the parts of Gnuastro that
were developed there, thus insuring the freedom of Gnuastro for the
future (see *note Copyright assignment::). We highly appreciate their
support for free software, and thus free science, and therefore a free
society.

Tohoku University Astronomical Institute, Sendai, Japan.
University of Salento, Lecce, Italy.
Centre de Recherche Astrophysique de Lyon (CRAL), Lyon, France.
Instituto de Astrofisica de Canarias (IAC), Tenerife, Spain.
Centro de Estudios de Física del Cosmos de Aragón (CEFCA), Teruel,
Spain.
Google Summer of Code 2020, 2021 and 2022

File: gnuastro.info, Node: Tutorials, Next: Installation, Prev: Introduction, Up: Top

2 Tutorials
***********

To help new users have a smooth and easy start with Gnuastro, in this
chapter several thoroughly elaborated tutorials, or cookbooks, are
provided. These tutorials demonstrate the capabilities of different
Gnuastro programs and libraries, along with tips and guidelines for the
best practices of using them in various realistic situations.

We strongly recommend going through these tutorials to get a good
feeling of how the programs are related (built in a modular design to be
used together in a pipeline), very similar to the core Unix-based
programs that they were modeled on. Therefore these tutorials will help
in optimally using Gnuastro's programs (and generally, the Unix-like
command-line environment) effectively for your research.

The first three tutorials (*note General program usage tutorial:: and
*note Detecting large extended targets:: and *note Building the extended
PSF::) use real input datasets from some of the deep Hubble Space
Telescope (HST) images, the Sloan Digital Sky Survey (SDSS) and the
Javalambre Photometric Local Universe Survey (J-PLUS) respectively.
Their aim is to demonstrate some real-world problems that many
astronomers often face and how they can be solved with Gnuastro's
programs. The fourth tutorial (*note Sufi simulates a detection::)
focuses on simulating astronomical images, which is another critical
aspect of any analysis!

The ultimate aim of *note General program usage tutorial:: is to
detect galaxies in a deep HST image, measure their positions, magnitude
and select those with the strongest colors. In the process, it takes
many detours to introduce you to the useful capabilities of many of the
programs. So please be patient in reading it. If you do not have much
time and can only try one of the tutorials, we recommend this one.

*note Detecting large extended targets:: deals with a major problem
in astronomy: effectively detecting the faint outer wings of bright (and
large) nearby galaxies to extremely low surface brightness levels
(roughly one quarter of the local noise level in the example discussed).
Besides the interesting scientific questions in these low-surface
brightness features, failure to properly detect them will bias the
measurements of the background objects and the survey's noise estimates.
This is an important issue, especially in wide surveys. Because
bright/large galaxies and stars(1), cover a significant fraction of the
survey area.

*note Building the extended PSF:: tackles an important problem in
astronomy: how the extract the PSF of an image, to the largest possible
extent, without assuming any functional form. In Gnuastro we have
multiple installed scripts for this job. Their usage and logic behind
best tuning them for the particular step, is fully described in this
tutorial, on a real dataset. The tutorial concludes with subtracting
that extended PSF from the science image; thus giving you a cleaner
image (with no scattered light of the brighter stars) for your
higher-level analysis.

*note Sufi simulates a detection:: has a fictional(2) setting!
Showing how Abd al-rahman Sufi (903 - 986 A.D., the first recorded
description of "nebulous" objects in the heavens is attributed to him)
could have used some of Gnuastro's programs for a realistic simulation
of his observations and see if his detection of nebulous objects was
trust-able. Because all conditions are under control in a
simulated/mock environment/dataset, mock datasets can be a valuable tool
to inspect the limitations of your data analysis and processing. But
they need to be as realistic as possible, so this tutorial is dedicated
to this important step of an analysis (simulations).

There are other tutorials also, on things that are commonly necessary
in astronomical research: In *note Detecting lines and extracting
spectra in 3D data::, we use MUSE cubes (an IFU dataset) to show how you
can subtract the continuum, detect emission-line features, extract
spectra and build pseudo narrow-band images. In *note Color channels in
same pixel grid:: we demonstrate how you can warp multiple images into a
single pixel grid (often necessary with multi-wavelength data), and
build a single color image. In *note Moire pattern in stacking and its
correction:: we show how you can avoid the unwanted Moiré pattern which
happens when warping separate exposures to build a stacked/co-add deeper
image. In *note Zero point of an image:: we review the process of
estimating the zero point of an image using a reference image or
catalog. Finally, in *note Pointing pattern design:: we show the
process by which you can simulate a dither pattern to find the best
observing strategy for your next exciting scientific project.

In these tutorials, we have intentionally avoided too many cross
references to make it more easy to read. For more information about a
particular program, you can visit the section with the same name as the
program in this book. Each program section in the subsequent chapters
starts by explaining the general concepts behind what it does, for
example, see *note Convolve::. If you only want practical information
on running a program, for example, its options/configuration, input(s)
and output(s), please consult the subsection titled "Invoking
ProgramName", for example, see *note Invoking astnoisechisel::. For an
explanation of the conventions we use in the example codes through the
book, please see *note Conventions::.

* Menu:

---------- Footnotes ----------

(1) Stars also have similarly large and extended wings due to the
point spread function, see *note PSF::.

(2) The two historically motivated tutorials (*note Sufi simulates a
detection:: is not intended to be a historical reference (the historical
facts of this fictional tutorial used Wikipedia as a reference).) This
form of presenting a tutorial was influenced by the PGF/TikZ and Beamer
manuals. They are both packages in TeX and LaTeX, the first is a
high-level vector graphic programming environment, while with the second
you can make presentation slides. On a similar topic, there are also
some nice words of wisdom for Unix-like systems called Rootless Root
(http://catb.org/esr/writings/unix-koans). These also have a similar
style but they use a mythical figure named Master Foo. If you already
have some experience in Unix-like systems, you will definitely find
these Unix Koans entertaining/educative.

File: gnuastro.info, Node: General program usage tutorial, Next: Detecting large extended targets, Prev: Tutorials, Up: Tutorials

2.1 General program usage tutorial
==================================

Measuring colors of astronomical objects in broad-band or narrow-band
images is one of the most basic and common steps in astronomical
analysis. Here, we will use Gnuastro's programs to get a physical scale
(area at certain redshifts) of the field we are studying, detect objects
in a Hubble Space Telescope (HST) image, measure their colors and
identify the ones with the strongest colors, do a visual inspection of
these objects and inspect spatial position in the image. After this
tutorial, you can also try the *note Detecting large extended targets::
tutorial which goes into a little more detail on detecting very low
surface brightness signal.

During the tutorial, we will take many detours to explain, and
practically demonstrate, the many capabilities of Gnuastro's programs.
In the end you will see that the things you learned during this tutorial
are much more generic than this particular problem and can be used in
solving a wide variety of problems involving the analysis of data
(images or tables). So please do not rush, and go through the steps
patiently to optimally master Gnuastro.

In this tutorial, we will use the HSTeXtreme Deep Field
(https://archive.stsci.edu/prepds/xdf) dataset. Like almost all
astronomical surveys, this dataset is free for download and usable by
the public. You will need the following tools in this tutorial:
Gnuastro, SAO DS9 (1), GNU Wget(2), and AWK (most common implementation
is GNU AWK(3)).

This tutorial was first prepared for the "Exploring the Ultra-Low
Surface Brightness Universe" workshop (November 2017) at the ISSI in
Bern, Switzerland. It was further extended in the "4th Indo-French
Astronomy School" (July 2018) organized by LIO, CRAL CNRS UMR5574, UCBL,
and IUCAA in Lyon, France. We are very grateful to the organizers of
these workshops and the attendees for the very fruitful discussions and
suggestions that made this tutorial possible.

*Write the example commands manually:* Try to type the example commands
on your terminal manually and use the history feature of your
command-line (by pressing the "up" button to retrieve previous
commands). Do not simply copy and paste the commands shown here. This
will help simulate future situations when you are processing your own
datasets.

* Menu:

---------- Footnotes ----------

(1) See *note SAO DS9::, available at
<http://ds9.si.edu/site/Home.html>

(2) <https://www.gnu.org/software/wget>

(3) <https://www.gnu.org/software/gawk>

File: gnuastro.info, Node: Calling Gnuastro's programs, Next: Accessing documentation, Prev: General program usage tutorial, Up: General program usage tutorial

2.1.1 Calling Gnuastro's programs
---------------------------------

A handy feature of Gnuastro is that all program names start with ‘ast’.
This will allow your command-line processor to easily list and
auto-complete Gnuastro's programs for you. Try typing the following
command (press <TAB> key when you see ‘<TAB>’) to see the list:

$ ast<TAB><TAB>

Any program that starts with ‘ast’ (including all Gnuastro programs)
will be shown. By choosing the subsequent characters of your desired
program and pressing <<TAB><TAB>> again, the list will narrow down and
the program name will auto-complete once your input characters are
unambiguous. In short, you often do not need to type the full name of
the program you want to run.

File: gnuastro.info, Node: Accessing documentation, Next: Setup and data download, Prev: Calling Gnuastro's programs, Up: General program usage tutorial

2.1.2 Accessing documentation
-----------------------------

Gnuastro contains a large number of programs and it is natural to forget
the details of each program's options or inputs and outputs. Therefore,
before starting the analysis steps of this tutorial, let's review how
you can access this book to refresh your memory any time you want,
without having to take your hands off the keyboard.

When you install Gnuastro, this book is also installed on your system
along with all the programs and libraries, so you do not need an
internet connection to access/read it. Also, by accessing this book as
described below, you can be sure that it corresponds to your installed
version of Gnuastro.

GNU Info(1) is the program in charge of displaying the manual on the
command-line (for more, see *note Info::). To see this whole book on
your command-line, please run the following command and press subsequent
keys. Info has its own mini-environment, therefore we will show the
keys that must be pressed in the mini-environment after a ‘->’ sign.
You can also ignore anything after the ‘#’ sign in the middle of the
line, they are only for your information.

$ info gnuastro # Open the top of the manual.
-> <SPACE> # All the book chapters.
-> <SPACE> # Continue down: show sections.
-> <SPACE> ... # Keep pressing space to go down.
-> q # Quit Info, return to the command-line.

The thing that greatly simplifies navigation in Info is the links
(regions with an underline). You can immediately go to the next link in
the page with the <<TAB>> key and press <<ENTER>> on it to go into that
part of the manual. Try the commands above again, but this time also
use <<TAB>> to go to the links and press <<ENTER>> on them to go to the
respective section of the book. Then follow a few more links and go
deeper into the book. To return to the previous page, press <l> (small
L). If you are searching for a specific phrase in the whole book (for
example, an option name), press <s> and type your search phrase and end
it with an <<ENTER>>. Finally, you can return to the command line and
quit Info by pressing the <q> key.

You do not need to start from the top of the manual every time. For
example, to get to *note Invoking astnoisechisel::, run the following
command. In general, all programs have such an "Invoking ProgramName"
section in this book. These sections are specifically for the
description of inputs, outputs and configuration options of each
program. You can access them directly for each program by giving its
executable name to Info.

$ info astnoisechisel

The other sections do not have such shortcuts. To directly access
them from the command-line, you need to tell Info to look into
Gnuastro's manual, then look for the specific section (an unambiguous
title is necessary). For example, if you only want to review/remember
NoiseChisel's *note Detection options::), just run the following
command. Note how case is irrelevant for Info when calling a title in
this manner.

$ info gnuastro "Detection options"

In general, Info is a powerful and convenient way to access this
whole book with detailed information about the programs you are running.
If you are not already familiar with it, please run the following
command and just read along and do what it says to learn it. Do not
stop until you feel sufficiently fluent in it. Please invest the half
an hour's time necessary to start using Info comfortably. It will
greatly improve your productivity and you will start reaping the rewards
of this investment very soon.

$ info info

As a good scientist you need to feel comfortable to play with the
features/options and avoid (be critical to) using default values as much
as possible. On the other hand, our human memory is limited, so it is
important to be able to easily access any part of this book fast and
remember the option names, what they do and their acceptable values.

If you just want the option names and a short description, calling
the program with the ‘--help’ option might also be a good solution like
the first example below. If you know a few characters of the option
name, you can feed the printed output to ‘grep’ like the second or third
example commands.

$ astnoisechisel --help
$ astnoisechisel --help | grep quant
$ astnoisechisel --help | grep check

---------- Footnotes ----------

(1) GNU Info is already available on almost all Unix-like operating
systems.

File: gnuastro.info, Node: Setup and data download, Next: Dataset inspection and cropping, Prev: Accessing documentation, Up: General program usage tutorial

2.1.3 Setup and data download
-----------------------------

The first step in the analysis of the tutorial is to download the
necessary input datasets. First, to keep things clean, let's create a
‘gnuastro-tutorial’ directory and continue all future steps in it:

$ mkdir gnuastro-tutorial
$ cd gnuastro-tutorial

We will be using the near infra-red Wide Field Camera
(http://www.stsci.edu/hst/wfc3) dataset. If you already have them in
another directory (for example, ‘XDFDIR’, with the same FITS file
names), you can set the ‘download’ directory to be a symbolic link to
‘XDFDIR’ with a command like this:

$ ln -s XDFDIR download

Otherwise, when the following images are not already present on your
system, you can make a ‘download’ directory and download them there.

$ mkdir download
$ cd download
$ xdfurl=http://archive.stsci.edu/pub/hlsp/xdf
$ wget $xdfurl/hlsp_xdf_hst_wfc3ir-60mas_hudf_f105w_v1_sci.fits
$ wget $xdfurl/hlsp_xdf_hst_wfc3ir-60mas_hudf_f125w_v1_sci.fits
$ wget $xdfurl/hlsp_xdf_hst_wfc3ir-60mas_hudf_f160w_v1_sci.fits
$ cd ..

In this tutorial, we will just use these three filters. Later, you may
need to download more filters. To do that, you can use the shell's
‘for’ loop to download them all in series (one after the other(1)) with
one command like the one below for the WFC3 filters. Put this command
instead of the three ‘wget’ commands above. Recall that all the extra
spaces, backslashes (‘\’), and new lines can be ignored if you are
typing on the lines on the terminal.

$ for f in f105w f125w f140w f160w; do \
wget $xdfurl/hlsp_xdf_hst_wfc3ir-60mas_hudf_"$f"_v1_sci.fits; \
done

---------- Footnotes ----------

(1) Note that you only have one port to the internet, so downloading
in parallel will actually be slower than downloading in series.

File: gnuastro.info, Node: Dataset inspection and cropping, Next: Angular coverage on the sky, Prev: Setup and data download, Up: General program usage tutorial

2.1.4 Dataset inspection and cropping
-------------------------------------

First, let's visually inspect the datasets we downloaded in *note Setup
and data download::. Let's take F160W image as an example. One of the
most common programs for viewing FITS images is SAO DS9, which is
usually called through the ‘ds9’ command-line program, like the command
below. If you do not already have DS9 on your computer and the command
below fails, please see *note SAO DS9::.

$ ds9 download/hlsp_xdf_hst_wfc3ir-60mas_hudf_f160w_v1_sci.fits

By default, DS9 open a relatively small window (for modern browsers)
and its default scale and color bar make it very hard to see any
structure in the image: everything will look black. Also, by default,
it zooms into the center of the image and you need to scroll to zoom-out
and see the whole thing. To avoid these problems, Gnuastro has the
‘astscript-fits-view’ script:

$ astscript-fits-view \
download/hlsp_xdf_hst_wfc3ir-60mas_hudf_f160w_v1_sci.fits

After running this command, you will see that the DS9 window fully
covers the height of your monitor, it is showing the whole image, using
a more clear color-map, and many more useful things. In fact, you see
the DS9 command that is used in your terminal(1). On GNU/Linux
operating systems (like Ubuntu, and Fedora), you can also set your
graphics user interface to use this script for opening FITS files when
you click on them. For more, see the instructions in the checklist at
the start of *note Invoking astscript-fits-view::.

As you hover your mouse over the image, notice how the "Value" and
positional fields on the top of the ds9 window get updated. The first
thing you might notice is that when you hover the mouse over the regions
with no data, they have a value of zero. The next thing might be that
the dataset has a shallower and deeper component (see *note Quantifying
measurement limits::). Recall that this is a combined/reduced image of
many exposures, and the parts that have more exposures are deeper. In
particular, the exposure time of the deep inner region is more than 4
times the exposure time of the outer (more shallower) parts.

To simplify the analysis in this tutorial, we will only be working on
the deep field, so let's crop it out of the full dataset. Fortunately
the XDF survey web page (above) contains the vertices of the deep flat
WFC3-IR field(2). With Gnuastro's Crop program, you can use those
vertices to cutout this deep region from the larger image (to learn more
about the Crop program see *note Crop::). But before that, to keep
things organized, let's make a directory called ‘flat-ir’ and keep the
flat (single-depth) regions in that directory (with a '‘xdf-’' prefix
for a shorter and easier filename).

$ mkdir flat-ir
$ astcrop --mode=wcs -h0 --output=flat-ir/xdf-f105w.fits \
--polygon="53.187414,-27.779152 : 53.159507,-27.759633 : \
53.134517,-27.787144 : 53.161906,-27.807208" \
download/hlsp_xdf_hst_wfc3ir-60mas_hudf_f105w_v1_sci.fits

$ astcrop --mode=wcs -h0 --output=flat-ir/xdf-f125w.fits \
--polygon="53.187414,-27.779152 : 53.159507,-27.759633 : \
53.134517,-27.787144 : 53.161906,-27.807208" \
download/hlsp_xdf_hst_wfc3ir-60mas_hudf_f125w_v1_sci.fits

$ astcrop --mode=wcs -h0 --output=flat-ir/xdf-f160w.fits \
--polygon="53.187414,-27.779152 : 53.159507,-27.759633 : \
53.134517,-27.787144 : 53.161906,-27.807208" \
download/hlsp_xdf_hst_wfc3ir-60mas_hudf_f160w_v1_sci.fits

Run the command below to have a look at the cropped images:

$ astscript-fits-view flat-ir/*.fits

You only see the deep region now, does not the noise look much
cleaner? An important result of this crop is that regions with no data
now have a NaN (Not-a-Number, or a blank value) value. Any
self-respecting statistical program will ignore NaN values, so they will
not affect your outputs. For example, notice how changing the DS9 color
bar will not affect the NaN pixels (their colors will not change).

However, do you remember that in the downloaded files, such regions
had a value of zero? That is a big problem! Because zero is a number,
and is thus meaningful, especially when you later want to NoiseChisel to
detect(3) all the signal from the deep universe in this image.
Generally, when you want to ignore some pixels in a dataset, and avoid
higher-level ambiguities or complications, it is always best to give
them blank values (not zero, or some other absurdly large or small
number). Gnuastro has the Arithmetic program for such cases, and we
will introduce it later in this tutorial.

In the example above, the polygon vertices are in degrees, but you
can also replace them with sexagesimal(4) coordinates (for example,
using ‘03h32m44.9794’ or ‘03:32:44.9794’ instead of ‘53.187414’, the
first RA, and ‘-27d46m44.9472’ or ‘-27:46:44.9472’ instead of
‘-27.779152’, the first Dec). To further simplify things, you can even
define your polygon visually as a DS9 "region", save it as a "region
file" and give that file to crop. But we need to continue, so if you
are interested to learn more, see *note Crop::.

Before closing this section, let's just take a look at the three
cropping commands we ran above. The only thing varying in the three
commands the filter name! Note how everything else is the same! In
such cases, you should generally avoid repeating a command manually, it
is prone to _many_ bugs, and as you see, it is very hard to read (did
not you suddenly write a ‘7’ as an ‘8’?).

To simplify the command, and allow you to work on more filters, we
can use the shell's ‘for’ loop as shown below. Notice how the place
where the filter names (‘f105w’, ‘f125w’ and ‘f160w’) are used above,
have been replaced with ‘$f’ (the shell variable that ‘for’ will update
in every loop) below.

$ rm flat-ir/*.fits
$ for f in f105w f125w f160w; do \
astcrop --mode=wcs -h0 --output=flat-ir/xdf-$f.fits \
--polygon="53.187414,-27.779152 : 53.159507,-27.759633 : \
53.134517,-27.787144 : 53.161906,-27.807208" \
download/hlsp_xdf_hst_wfc3ir-60mas_hudf_"$f"_v1_sci.fits; \
done

---------- Footnotes ----------

(1) When comparing DS9's command-line options to Gnuastro's, you will
notice how SAO DS9 does not follow the GNU style of options where "long"
and "short" options are preceded by ‘--’ and ‘-’ respectively (for
example, ‘--width’ and ‘-w’, see *note Options::).

(2) <https://archive.stsci.edu/prepds/xdf/#dataproducts>

(3) As you will see below, unlike most other detection algorithms,
NoiseChisel detects the objects from their faintest parts, it does not
start with their high signal-to-noise ratio peaks. Since the Sky is
already subtracted in many images and noise fluctuates around zero, zero
is commonly higher than the initial threshold applied. Therefore
keeping zero-valued pixels in this image will cause them to identified
as part of the detections!

(4) <https://en.wikipedia.org/wiki/Sexagesimal>

File: gnuastro.info, Node: Angular coverage on the sky, Next: Cosmological coverage and visualizing tables, Prev: Dataset inspection and cropping, Up: General program usage tutorial

2.1.5 Angular coverage on the sky
---------------------------------

The cropped images in *note Dataset inspection and cropping:: are the
deepest images we currently have of the sky. The first thing that comes
to mind may be this: "How large is this field on the sky?".

*More accurate method:* the steps mentioned in this section are
primarily designed to help you get familiar with the FITS WCS standard
and some shells scripting. The accuracy of this method will decrease as
your image becomes large (on the scale of degrees). For an accurate
method, see *note Area of non-blank pixels on sky::.

You can get a fast and crude answer with Gnuastro's Fits program, using
this command:

$ astfits flat-ir/xdf-f160w.fits --skycoverage

It will print the sky coverage in two formats (all numbers are in
units of degrees for this image): 1) the image's central RA and Dec and
full width around that center, 2) the range of RA and Dec covered by
this image. You can use these values in various online query systems.
You can also use this option to automatically calculate the area covered
by this image. With the ‘--quiet’ option, the printed output of
‘--skycoverage’ will not contain human-readable text, making it easier
for automatic (computer) processing:

$ astfits flat-ir/xdf-f160w.fits --skycoverage --quiet

The second row is the coverage range along RA and Dec (compare with
the outputs before using ‘--quiet’). We can thus simply subtract the
second from the first column and multiply it with the difference of the
fourth and third columns to calculate the image area. We will also
multiply each by 60 to have the area in arc-minutes squared.

$ astfits flat-ir/xdf-f160w.fits --skycoverage --quiet \
| awk 'NR==2{print ($2-$1)*60*($4-$3)*60}'

The returned value is $9.06711$ arcmin$^2$. *However, this method
ignores the fact that many of the image pixels are blank!* In other
words, the image does cover this area, but there is no data in more than
half of the pixels. So let's calculate the area coverage over-which we
actually have data.

The FITS world coordinate system (WCS) metadata standard contains the
key to answering this question. Run the following command to see all
the FITS keywords (metadata) for one of the images (almost identical
with the other images because they are scaled to the same region of
Sky):

$ astfits flat-ir/xdf-f160w.fits -h1

Look into the keywords grouped under the '‘World Coordinate System
(WCS)’' title. These keywords define how the image relates to the
outside world. In particular, the ‘CDELT*’ keywords (or ‘CDELT1’ and
‘CDELT2’ in this 2D image) contain the "Coordinate DELTa" (or change in
coordinate units) with a change in one pixel. But what is the units of
each "world" coordinate? The ‘CUNIT*’ keywords (for "Coordinate UNIT")
have the answer. In this case, both ‘CUNIT1’ and ‘CUNIT1’ have a value
of ‘deg’, so both "world" coordinates are in units of degrees. We can
thus conclude that the value of ‘CDELT*’ is in units of
degrees-per-pixel(1).

With the commands below, we will use ‘CDELT’ (along with the number
of non-blank pixels) to find the answer of our initial question: "how
much of the sky does this image cover?". The lines starting with ‘##’
are just comments for you to read and understand each command. Do not
type them on the terminal (no problem if you do, they will just not have
any effect). The commands are intentionally repetitive in some places
to better understand each step and also to demonstrate the beauty of
command-line features like history, variables, pipes and loops (which
you will commonly use as you become more proficient on the
command-line).

*Use shell history:* Do not forget to make effective use of your shell's
history: you do not have to re-type previous command to add something to
them (like the examples below). This is especially convenient when you
just want to make a small change to your previous command. Press the
"up" key on your keyboard (possibly multiple times) to see your previous
command(s) and modify them accordingly.

*Your locale does not use '.' as decimal separator:* on systems that do
not use an English language environment, the dates, numbers, etc., can
be printed in different formats (for example, '0.5' can be written as
'0,5': with a comma). With the ‘LC_NUMERIC’ line at the start of the
script below, we are ensuring a unified format in the output of ‘seq’.
For more, please see *note Numeric locale::.

## Make sure that the decimal separator is a point in any environment.
$ export LC_NUMERIC=C

## See the general statistics of non-blank pixel values.
$ aststatistics flat-ir/xdf-f160w.fits

## We only want the number of non-blank pixels (add '--number').
$ aststatistics flat-ir/xdf-f160w.fits --number

## Keep the result of the command above in the shell variable `n'.
$ n=$(aststatistics flat-ir/xdf-f160w.fits --number)

## See what is stored the shell variable `n'.
$ echo $n

## Show all the FITS keywords of this image.
$ astfits flat-ir/xdf-f160w.fits -h1

## The resolution (in degrees/pixel) is in the `CDELT' keywords.
## Only show lines that contain these characters, by feeding
## the output of the previous command to the `grep' program.
$ astfits flat-ir/xdf-f160w.fits -h1 | grep CDELT

## Since the resolution of both dimensions is (approximately) equal,
## we will only read the value of one (CDELT1) with '--keyvalue'.
$ astfits flat-ir/xdf-f160w.fits -h1 --keyvalue=CDELT1

## We do not need the file name in the output (add '--quiet').
$ astfits flat-ir/xdf-f160w.fits -h1 --keyvalue=CDELT1 --quiet

## Save it as the shell variable `r'.
$ r=$(astfits flat-ir/xdf-f160w.fits -h1 --keyvalue=CDELT1 --quiet)

## Print the values of `n' and `r'.
$ echo $n $r

## Use the number of pixels (first number passed to AWK) and
## length of each pixel's edge (second number passed to AWK)
## to estimate the area of the field in arc-minutes squared.
$ echo $n $r | awk '{print $1 * ($2*60)^2}'

The output of the last command (area of this field) is 4.03817 (or
approximately 4.04) arc-minutes squared. Just for comparison, this is
roughly 175 times smaller than the average moon's angular area (with a
diameter of 30 arc-minutes or half a degree).

Some FITS writers do not use the ‘CDELT’ convention, making it hard
to use the steps above. In such cases, you can extract the pixel scale
with the ‘--pixelscale’ option of Gnuastro's Fits program like the
command below. Similar to the ‘--skycoverage’ option above, you can
also use the ‘--quiet’ option to allow easy usage of the values in
scripts.

$ astfits flat-ir/xdf-f160w.fits --pixelscale

*AWK for table/value processing:* As you saw above AWK is a powerful and
simple tool for text processing. You will see it often in shell
scripts. GNU AWK (the most common implementation) comes with a free and
wonderful book (https://www.gnu.org/software/gawk/manual/) in the same
format as this book which will allow you to master it nicely. Just like
this manual, you can also access GNU AWK's manual on the command-line
whenever necessary without taking your hands off the keyboard. Just run
‘info awk’.

---------- Footnotes ----------

(1) With the FITS ‘CDELT’ convention, rotation (‘PC’ or ‘CD’
keywords) and scales (‘CDELT’) are separated. In the FITS standard the
‘CDELT’ keywords are optional. When ‘CDELT’ keywords are not present,
the ‘PC’ matrix is assumed to contain _both_ the coordinate rotation and
scales. Note that not all FITS writers use the ‘CDELT’ convention. So
you might not find the ‘CDELT’ keywords in the WCS metadata of some FITS
files. However, all Gnuastro programs (which use the default FITS
keyword writing format of WCSLIB) write their output WCS with the
‘CDELT’ convention, even if the input does not have it. If your dataset
does not use the ‘CDELT’ convention, you can feed it to any (simple)
Gnuastro program (for example, Arithmetic) and the output will have the
‘CDELT’ keyword. See Section 8 of the FITS standard
(https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf) for
more

File: gnuastro.info, Node: Cosmological coverage and visualizing tables, Next: Building custom programs with the library, Prev: Angular coverage on the sky, Up: General program usage tutorial

2.1.6 Cosmological coverage and visualizing tables
--------------------------------------------------

Having found the angular coverage of the dataset in *note Angular
coverage on the sky::, we can now use Gnuastro to answer a more
physically motivated question: "How large is this area at different
redshifts?". To get a feeling of the tangential area that this field
covers at redshift 2, you can use Gnuastro's CosmicCalcular program
(*note CosmicCalculator::). In particular, you need the tangential
distance covered by 1 arc-second as raw output. Combined with the
field's area that was measured before, we can calculate the tangential
distance in Mega Parsecs squared ($Mpc^2$).

## If your system language uses ',' (not '.') as decimal separator.
$ export LC_NUMERIC=C

## Print general cosmological properties at redshift 2 (for example).
$ astcosmiccal -z2

## When given a "Specific calculation" option, CosmicCalculator
## will just print that particular calculation. To see all such
## calculations, add a `--help' token to the previous command
## (under the same title). Note that with `--help', no processing
## is done, so you can always simply append it to remember
## something without modifying the command you want to run.
$ astcosmiccal -z2 --help

## Only print the "Tangential dist. covered by 1arcsec at z (kpc)".
## in units of kpc/arc-seconds.
$ astcosmiccal -z2 --arcsectandist

## It is easier to use the short (single character) version of
## this option when typing (but this is hard to read, so use
## the long version in scripts or notes you plan to archive).
$ astcosmiccal -z2 -s

## Short options can be merged (they are only a single character!)
$ astcosmiccal -sz2

## Convert this distance to kpc^2/arcmin^2 and save in `k'.
$ k=$(astcosmiccal -sz2 | awk '{print ($1*60)^2}')

## Calculate the area of the dataset in arcmin^2.
$ n=$(aststatistics flat-ir/xdf-f160w.fits --number)
$ r=$(astfits flat-ir/xdf-f160w.fits -h1 --keyvalue=CDELT1 -q)
$ a=$(echo $n $r | awk '{print $1 * ($2*60)^2 }')

## Multiply `k' and `a' and divide by 10^6 for value in Mpc^2.
$ echo $k $a | awk '{print $1 * $2 / 1e6}'

At redshift 2, this field therefore covers approximately 1.07 $Mpc^2$.
If you would like to see how this tangential area changes with redshift,
you can use a shell loop like below.

$ for z in 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0; do \
k=$(astcosmiccal -sz$z); \
echo $z $k $a | awk '{print $1, ($2*60)^2 * $3 / 1e6}'; \
done

Fortunately, the shell has a useful tool/program to print a sequence of
numbers that is nicely called ‘seq’ (short for "sequence"). You can use
it instead of typing all the different redshifts in the loop above. For
example, the loop below will calculate and print the tangential coverage
of this field across a larger range of redshifts (0.1 to 5) and with
finer increments of 0.1. For more on the ‘LC_NUMERIC’ command, see
*note Numeric locale::.

## If your system language uses ',' (not '.') as decimal separator.
$ export LC_NUMERIC=C

## The loop over the redshifts
$ for z in $(seq 0.1 0.1 5); do \
k=$(astcosmiccal -z$z --arcsectandist); \
echo $z $k $a | awk '{print $1, ($2*60)^2 * $3 / 1e6}'; \
done

Have a look at the two printed columns. The first is the redshift,
and the second is the area of this image at that redshift (in Mega
Parsecs squared). Redshift (https://en.wikipedia.org/wiki/Redshift)
($z$) is a measure of distance in galaxy evolution and cosmology: a
higher redshift corresponds to larger distance.

Now, have a look at the first few values. At $z=0.1$ and $z=0.5$,
this image covers $0.05 Mpc^2$ and $0.57 Mpc^2$ respectively. This
increase of coverage with redshift is expected because a fixed angle
will cover a larger tangential area at larger distances. However, as
you come down the list (to higher redshifts) you will notice that this
relation does not hold! The largest coverage is at $z=1.6$: at higher
redshifts, the area decreases, and continues decreasing!!! In
$\Lambda{}CDM$ cosmology, this happens because of the finite speed of
light and the expansion of the universe, see the Wikipedia page
(https://en.wikipedia.org/wiki/Angular_diameter_distance#Angular_diameter_turnover_point).

In case you have TOPCAT, you can visualize this as a plot (if you do
not have TOPCAT, see *note TOPCAT::). To do so, first you need to save
the output of the loop above into a FITS table by piping the output to
Gnuastro's Table program and giving an output name:

$ for z in $(seq 0.1 0.1 5); do \
k=$(astcosmiccal -z$z --arcsectandist); \
echo $z $k $a | awk '{print $1, ($2*60)^2 * $3 / 1e6}'; \
done | asttable --output=z-vs-tandist.fits

You can now use Gnuastro's ‘astscript-fits-view’ to open this table
in TOPCAT with the command below. Do you remember this script from
*note Dataset inspection and cropping::? There, we used it to view a
FITS image with DS9! This script will see if the first dataset in the
image is a table or an image and will call TOPCAT or DS9 accordingly:
making it a very convenient tool to inspect the contents of all types of
FITS data.

$ astscript-fits-view z-vs-tandist.fits

After TOPCAT opens, you will see the name of the table
‘z-vs-tandist.fits’ in the left panel. On the top menu bar, select the
"Graphics" menu, then select "Plain plot" to visualize the two columns
printed above as a plot and get a better impression of the turn over
point of the image cosmological coverage.

File: gnuastro.info, Node: Building custom programs with the library, Next: Option management and configuration files, Prev: Cosmological coverage and visualizing tables, Up: General program usage tutorial

2.1.7 Building custom programs with the library
-----------------------------------------------

In *note Cosmological coverage and visualizing tables::, we repeated a
certain calculation/output of a program multiple times using the shell's
‘for’ loop. This simple way of repeating a calculation is great when it
is only necessary once. However, if you commonly need this calculation
and possibly for a larger number of redshifts at higher precision, the
command above can be slow. Please try it out by changing the sequence
command in the previous section to '‘seq 0.1 0.01 10’'. It will take
about 11 seconds(1)! This can be improved by _hundreds_ of times! This
section will show you how.

Generally, repeated calls to a generic program (like
CosmicCalculator) are slow, because a generic program can have a lot of
overhead on each call. To be generic and easy to operate,
CosmicCalculator has to parse the command-line and all configuration
files (see *note Option management and configuration files::) which
contain human-readable characters and need a lot of pre-processing to be
ready for processing by the computer. Afterwards, CosmicCalculator has
to check the sanity of its inputs and check which of its many options
you have asked for. All the this pre-processing takes as much time as
the high-level calculation you are requesting, and it has to re-do all
of these for every redshift in your loop.

To greatly speed up the processing, you can directly access the core
work-horse of CosmicCalculator without all that overhead by designing
your custom program for this job. Using Gnuastro's library, you can
write your own tiny program particularly designed for this exact
calculation (and nothing else!). To do that, copy and paste the
following C program in a file called ‘myprogram.c’.

#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <gnuastro/cosmology.h>

int
main(void)
{
double area=4.03817; /* Area of field (arcmin^2). */
double z, adist, tandist; /* Temporary variables. */

/* Constants from Plank 2018 (arXiv:1807.06209, Table 2) */
double H0=67.66, olambda=0.6889, omatter=0.3111, oradiation=0;

/* Do the same thing for all redshifts (z) between 0.1 and 5. */
for(z=0.1; z<10; z+=0.01)
{
/* Calculate the angular diameter distance. */
adist=gal_cosmology_angular_distance(z, H0, olambda,
omatter, oradiation);

/* Calculate the tangential distance of one arcsecond. */
tandist = adist * 1000 * M_PI / 3600 / 180;

/* Print the redshift and area. */
printf("%-5.2f %g\n", z, pow(tandist * 60,2) * area / 1e6);
}

/* Tell the system that everything finished successfully. */
return EXIT_SUCCESS;
}

Then run the following command to compile your program and run it.

$ astbuildprog myprogram.c

In the command above, you used Gnuastro's BuildProgram program. Its job
is to simplify the compilation, linking and running of simple C programs
that use Gnuastro's library (like this one). BuildProgram is designed
to manage Gnuastro's dependencies, compile and link your custom program
and then run it.

Did you notice how your custom program created the table almost
instantaneously? Technically, it only took about 0.03 seconds! Recall
that the ‘for’ loop of *note Cosmological coverage and visualizing
tables:: took more than 11 seconds (or $\sim367$ times slower!).

Please run the ‘ls’ command to see a listing of the files in the
current directory. You will notice that a new file called ‘myprogram’
has been created. This is the compiled program that was created and run
by the command above (its in binary machine code format, not
human-readable any more). You can run it again to get the same results
by executing it:

$ ./myprogram

The efficiency of your custom ‘myprogram’ compared to repeated calls
to CosmicCalculator is because in the latter, the requested processing
is comparable to the necessary overheads. For other programs that take
large input datasets and do complicated processing on them, the overhead
is usually negligible compared to the processing. In such cases, the
libraries are only useful if you want a different/new processing
compared to the functionalities in Gnuastro's existing programs.

Gnuastro has a large library which is used extensively by all the
programs. In other words, the library is like the skeleton of Gnuastro.
For the full list of available functions classified by context, please
see *note Gnuastro library::. Gnuastro's library and BuildProgram are
created to make it easy for you to use these powerful features as you
like. This gives you a high level of creativity, while also providing
efficiency and robustness. Several other complete working examples
(involving images and tables) of Gnuastro's libraries can be see in
*note Library demo programs::.

But for this tutorial, let's stop discussing the libraries here and
get back to Gnuastro's already built programs (which do not need C
programming). But before continuing, let's clean up the files we do not
need any more:

$ rm myprogram* z-vs-tandist*

---------- Footnotes ----------

(1) To measure how much time the loop of *note Cosmological coverage
and visualizing tables:: takes on your system, you can use the ‘time’
command. First put the whole loop (and pipe) into a plain-text file (to
be loaded as a shell script) called ‘z-vs-tandist.sh’. Then run this
command: ‘time -p bash z-vs-tandist.sh’. The relevant time (in seconds)
is shown after ‘real’.

File: gnuastro.info, Node: Option management and configuration files, Next: Warping to a new pixel grid, Prev: Building custom programs with the library, Up: General program usage tutorial

2.1.8 Option management and configuration files
-----------------------------------------------

In the previous section (*note Cosmological coverage and visualizing
tables::), when you ran CosmicCalculator, you only specified the
redshfit with ‘-z2’ option. You did not specify the cosmological
parameters that are necessary for the calculations! Parameters like the
Hubble constant ($H_0$) and the matter density. In spite of this,
CosmicCalculator done its processing and printed results.

None of Gnuastro's programs keep a default value internally within
their code (they are all set by the user)! So where did the necessary
cosmological parameters that are necessary for its calculations come
from? What were the values to those parameters? In short, they come
from a configuration file (see *note Configuration file precedence::),
and the final used values can be checked/edited on the command-line. In
this section we will review this important aspect of all the programs in
Gnuastro.

Configuration files are an important part of all Gnuastro's programs,
especially the ones with a large number of options, so it is important
to understand this part well. Once you get comfortable with
configuration files, you can make good use of them in all Gnuastro
programs (for example, NoiseChisel). For example, to do optimal
detection on various datasets, you can have configuration files for
different noise properties. The configuration of each program (besides
its version) is vital for the reproducibility of your results, so it is
important to manage them properly.

As we saw above, the full list of the options in all Gnuastro
programs can be seen with the ‘--help’ option. Try calling it with
CosmicCalculator as shown below. Note how options are grouped by
context to make it easier to find your desired option. However, in each
group, options are ordered alphabetically.

$ astcosmiccal --help

After running the command above, please scroll to the line that you
ran this command and read through the output (its the same format for
all the programs). All options have a long format (starting with ‘--’
and a multi-character name) and some have a short format (starting with
‘-’ and a single character), for more see *note Options::. The options
that expect a value, have an <=> sign after their long version. The
format of their expected value is also shown as ‘FLT’, ‘INT’ or ‘STR’
for floating point numbers, integer numbers, and strings (filenames for
example) respectively.

You can see the values of all options that need one with the
‘--printparams’ option (or its short format: ‘-P’). ‘--printparams’ is
common to all programs (see *note Common options::). You can see the
default cosmological parameters, from the Plank collaboration 2020
(https://arxiv.org/abs/1807.06209), under the ‘# Input:’ title:

$ astcosmiccal -P

# Input:
H0 67.66 # Current expansion rate (Hubble constant).
olambda 0.6889 # Current cosmological cst. dens. per crit. dens.
omatter 0.3111 # Current matter density per critical density.
oradiation 0 # Current radiation density per critical density.

Let's say you want to do the calculation in the previous section
using $H_0=70$ km/s/Mpc. To do this, just add ‘--H0=70’ after the
command above (while keeping the ‘-P’). In the output, you can see that
the used Hubble constant has also changed.

$ astcosmiccal -P --H0=70

Afterwards, delete the ‘-P’ and add a ‘-z2’ to see the calculations with
the new cosmology (or configuration).

$ astcosmiccal --H0=70 -z2

From the output of the ‘--help’ option, note how the option for
Hubble constant has both short (‘-H’) and long (‘--H0’) formats. One
final note is that the equal (<=>) sign is not mandatory. In the short
format, the value can stick to the actual option (the short option name
is just one character after-all, thus easily identifiable) and in the
long format, a white-space character is also enough.

$ astcosmiccal -H70 -z2
$ astcosmiccal --H0 70 -z2 --arcsectandist

When an option does not need a value, and has a short format (like
‘--arcsectandist’), you can easily append it _before_ other short
options. So the last command above can also be written as:

$ astcosmiccal --H0 70 -sz2

Let's assume that in one project, you want to only use rounded
cosmological parameters ($H_0$ of 70km/s/Mpc and matter density of 0.3).
You should therefore run CosmicCalculator like this:

$ astcosmiccal --H0=70 --olambda=0.7 --omatter=0.3 -z2

But having to type these extra options every time you run
CosmicCalculator will be prone to errors (typos in particular),
frustrating and slow. Therefore in Gnuastro, you can put all the
options and their values in a "Configuration file" and tell the programs
to read the option values from there.

Let's create a configuration file... With your favorite text editor,
make a file named ‘my-cosmology.conf’ (or ‘my-cosmology.txt’, the suffix
does not matter for Gnuastro, but a more descriptive suffix like ‘.conf’
is recommended for humans reading your code and seeing your files: this
includes you, looking into your own project, in a couple of months that
you have forgot the details!). Then put the following lines inside of
the plain-text file. One space between the option value and name is
enough, the values are just under each other to help in readability.
Also note that you should only use _long option names_ in configuration
files.

H0 70
olambda 0.7
omatter 0.3

You can now tell CosmicCalculator to read this file for option values
immediately using the ‘--config’ option as shown below. Do you see how
the output of the following command corresponds to the option values in
‘my-cosmology.conf’, and is therefore identical to the previous command?

$ astcosmiccal --config=my-cosmology.conf -z2

But still, having to type ‘--config=my-cosmology.conf’ every time is
annoying, is not it? If you need this cosmology every time you are
working in a specific directory, you can use Gnuastro's default
configuration file names and avoid having to type it manually.

The default configuration files (that are checked if they exist) must
be placed in the hidden ‘.gnuastro’ sub-directory (in the same directory
you are running the program). Their file name (within ‘.gnuastro’) must
also be the same as the program's executable name. So in the case of
CosmicCalculator, the default configuration file in a given directory is
‘.gnuastro/astcosmiccal.conf’.

Let's do this. We will first make a directory for our custom
cosmology, then build a ‘.gnuastro’ within it. Finally, we will copy
the custom configuration file there:

$ mkdir my-cosmology
$ mkdir my-cosmology/.gnuastro
$ mv my-cosmology.conf my-cosmology/.gnuastro/astcosmiccal.conf

Once you run CosmicCalculator within ‘my-cosmology’ (as shown below),
you will see how your custom cosmology has been implemented without
having to type anything extra on the command-line.

$ cd my-cosmology
$ astcosmiccal -P # Your custom cosmology is printed.
$ cd ..
$ astcosmiccal -P # The default cosmology is printed.

To further simplify the process, you can use the ‘--setdirconf’
option. If you are already in your desired working directory, calling
this option with the others will automatically write the final values
(along with descriptions) in ‘.gnuastro/astcosmiccal.conf’. For
example, try the commands below:

$ mkdir my-cosmology2
$ cd my-cosmology2
$ astcosmiccal -P
$ astcosmiccal --H0 70 --olambda=0.7 --omatter=0.3 --setdirconf
$ astcosmiccal -P
$ cd ..

Gnuastro's programs also have default configuration files for a
specific user (when run in any directory). This allows you to set a
special behavior every time a program is run by a specific user. Only
the directory and filename differ from the above, the rest of the
process is similar to before. Finally, there are also system-wide
configuration files that can be used to define the option values for all
users on a system. See *note Configuration file precedence:: for a more
detailed discussion.

We will stop the discussion on configuration files here, but you can
always read about them in *note Configuration files::. Before
continuing the tutorial, let's delete the two extra directories that we
do not need any more:

$ rm -rf my-cosmology*

File: gnuastro.info, Node: Warping to a new pixel grid, Next: NoiseChisel and Multi-Extension FITS files, Prev: Option management and configuration files, Up: General program usage tutorial

2.1.9 Warping to a new pixel grid
---------------------------------

We are now ready to start processing the deep HST images that were
prepared in *note Dataset inspection and cropping::. One of the most
important points while using several images for data processing is that
those images must have the same pixel grid. The process of changing the
pixel grid is named 'warp'. Fortunately, Gnuastro has Warp program for
warping the pixel grid (see *note Warp::).

Warping to a different/matched pixel grid is commonly needed before
higher-level analysis especially when you are using datasets from
different instruments. The XDF datasets we are using here are already
aligned to the same pixel grid. But let's have a look at some of
Gnuastro's linear warping features here. For example, try rotating one
of the images by 20 degrees with the first command below. With the
second command, open the output and input to see how it is rotated.

$ astwarp flat-ir/xdf-f160w.fits --rotate=20

$ astscript-fits-view flat-ir/xdf-f160w.fits xdf-f160w_rotated.fits

Warp can generally be used for many kinds of pixel grid manipulation
(warping), not just rotations. For example, the outputs of the commands
below will have larger pixels respectively (new resolution being one
quarter the original resolution), get shifted by 2.8 (by sub-pixel), get
a shear of 2, and be tilted (projected). Run each of them and open the
output file to see the effect, they will become handy for you in the
future.

$ astwarp flat-ir/xdf-f160w.fits --scale=0.25
$ astwarp flat-ir/xdf-f160w.fits --translate=2.8
$ astwarp flat-ir/xdf-f160w.fits --shear=0.2
$ astwarp flat-ir/xdf-f160w.fits --project=0.001,0.0005
$ astscript-fits-view flat-ir/xdf-f160w.fits *.fits

If you need to do multiple warps, you can combine them in one call to
Warp. For example, to first rotate the image, then scale it, run this
command:

$ astwarp flat-ir/xdf-f160w.fits --rotate=20 --scale=0.25

If you have multiple warps, do them all in one command. Do not warp
them in separate commands because the correlated noise will become too
strong. As you see in the matrix that is printed when you run Warp, it
merges all the warps into a single warping matrix (see *note Merging
multiple warpings::) and simply applies that (mixes the pixel values)
just once. However, if you run Warp multiple times, the pixels will be
mixed multiple times, creating a strong artificial blur/smoothing, or
stronger correlated noise.

Recall that the merging of multiple warps is done through matrix
multiplication, therefore order matters in the separate operations. At
a lower level, through Warp's ‘--matrix’ option, you can directly
request your desired final warp and do not have to break it up into
different warps like above (see *note Invoking astwarp::).

Fortunately these datasets are already aligned to the same pixel
grid, so you do not actually need the files that were just generated.
You can safely delete them all with the following command. Here, you
see why we put the processed outputs that we need later into a separate
directory. In this way, the top directory can be used for temporary
files for testing that you can simply delete with a generic command like
below.

$ rm *.fits

File: gnuastro.info, Node: NoiseChisel and Multi-Extension FITS files, Next: NoiseChisel optimization for detection, Prev: Warping to a new pixel grid, Up: General program usage tutorial

2.1.10 NoiseChisel and Multi-Extension FITS files
-------------------------------------------------

In the previous sections, we completed a review of the basics of
Gnuastro's programs. We are now ready to do some more serious analysis
on the downloaded images: extract the pixels containing signal from the
image, find sub-structure of the extracted signal, do measurements over
the extracted objects and analyze them (finding certain objects of
interest in the image).

The first step is to separate the signal (galaxies or stars) from the
background noise in the image. We will be using the results of *note
Dataset inspection and cropping::, so be sure you already have them.
Gnuastro has NoiseChisel for this job. But NoiseChisel's output is a
multi-extension FITS file, therefore to better understand how to use
NoiseChisel, let's take a look at multi-extension FITS files and how you
can interact with them.

In the FITS format, each extension contains a separate dataset (image
in this case). You can get basic information about the extensions in a
FITS file with Gnuastro's Fits program (see *note Fits::). To start
with, let's run NoiseChisel without any options, then use Gnuastro's
Fits program to inspect the number of extensions in this file.

$ astnoisechisel flat-ir/xdf-f160w.fits
$ astfits xdf-f160w_detected.fits

From the output list, we see that NoiseChisel's output contains 5
extensions. The zero-th (counting from zero, with name
‘NOISECHISEL-CONFIG’) is empty: it has value of ‘0’ in the fourth column
(which shows its size in pixels). Like NoiseChisel, in all of
Gnuastro's programs, the first (or zero-th) extension of the output only
contains meta-data: data about/describing the datasets within (all) the
output's extensions. This is recommended by the FITS standard, see
*note Fits:: for more. In the case of Gnuastro's programs, this generic
zero-th/meta-data extension (for the whole file) contains all the
configuration options of the program that created the file.

Metadata regarding how the analysis was done (or a dataset was
created) is very important for higher-level analysis and
reproducibility. Therefore, Let's first take a closer look at the
‘NOISECHISEL-CONFIG’ extension. If you specify a special header in the
FITS file, Gnuastro's Fits program will print the header keywords
(metadata) of that extension. You can either specify the HDU/extension
counter (starting from 0), or name. Therefore, the two commands below
are identical for this file. We are usually tempted to use the first
(shorter format), but when putting your commands into a script, please
use the second format which is more human-friendly and understandable
for readers of your code who may not know what is in the 0-th extension
(this includes yourself in a few months!):

$ astfits xdf-f160w_detected.fits -h0
$ astfits xdf-f160w_detected.fits -hNOISECHISEL-CONFIG

The first group of FITS header keywords you see (containing the
‘SIMPLE’ and ‘BITPIX’ keywords; before the first empty line) are
standard keywords. They are required by the FITS standard and must be
present in any FITS extension. The second group starts with the input
file name (value to the ‘INPUT’ keyword). The rest of the keywords you
see afterwards have the same name as NoiseChisel's options, and the
value used by NoiseChisel in this run is shown after the ‘=’ sign.
Finally, the last group (starting with ‘DATE’) contains the date and
version information of Gnuastro and its dependencies that were used to
generate this file. Besides the option values, these are also critical
for future reproducibility of the result (you may update Gnuastro or its
dependencies, and they may behave differently afterwards). The
"versions and date" group of keywords are present in all Gnuastro's FITS
extension outputs, for more see *note Output FITS files::.

Note that if a keyword name is larger than 8 characters, it is
preceded by a ‘HIERARCH’ keyword and that all keyword names are in
capital letters. These are all part of the FITS standard and originate
from its history. But in short, both can be ignored! For example, with
the commands below, let's see at first what the default values are, and
then just check the value of ‘--detgrowquant’ option (using the ‘-P’
option described in *note Option management and configuration files::).

$ astnoisechisle -P
$ astnoisechisel -P | grep detgrowquant

To confirm that NoiseChisel used this value when we ran it above,
let's use ‘grep’ to extract the keyword line with ‘detgrowquant’ from
the metadata extension. However, as you saw above, keyword names in the
header is in all caps. So we need to ask ‘grep’ to ignore case with the
‘-i’ option.

$ astfits xdf-f160w_detected.fits -h0 | grep -i detgrowquant

In the output of the above command, you see ‘HIERARCH’ at the start
of the line. According to the FITS standard, ‘HIERARCH’ is placed at
the start of all keywords that have a name that is more than 8
characters long. Both the all-caps and the ‘HIERARCH’ keyword can be
annoying when you want to read/check the value. Therefore, the best
solution is to use the ‘--keyvalue’ option of Gnuastro's ‘astfits’
program as shown below. With it, you do not have to worry about
‘HIERARCH’ or the case of the name (FITS keyword names are not
case-sensitive).

$ astfits xdf-f160w_detected.fits -h0 --keyvalue=detgrowquant -q

The metadata (that is stored in the output) can later be used to exactly
reproduce/understand your result, even if you have lost/forgot the
command you used to create the file. This feature is present in all of
Gnuastro's programs, not just NoiseChisel.

The rest of the HDUs in NoiseChisel have data. So let's open them in
a DS9 window and then describe each:

$ astscript-fits-view xdf-f160w_detected.fits

A "cube" window opens along with DS9's main window. The buttons and
horizontal scroll bar in this small new window can be used to navigate
between the extensions. In this mode, all DS9's settings (for example,
zoom or color-bar) will be identical between the extensions. Try
zooming into one part and flipping through the extensions to see how the
galaxies were detected along with the Sky and Sky standard deviation
values for that region. Just have in mind that NoiseChisel's job is
_only_ detection (separating signal from noise). We will do
segmentation on this result later to find the individual galaxies/peaks
over the detected pixels.

The second extension of NoiseChisel's output (numbered 1, named
‘INPUT-NO-SKY’) is the Sky-subtracted input that you provided. The
third (‘DETECTIONS’) is NoiseChisel's main output which is a binary
image with only two possible values for all pixels: 0 for noise and 1
for signal. Since it only has two values, to avoid taking too much
space on your computer, its numeric datatype an unsigned 8-bit integer
(or ‘uint8’)(1). The fourth and fifth (‘SKY’ and ‘SKY_STD’) extensions,
have the Sky and its standard deviation values for the input on a tile
grid and were calculated over the undetected regions (for more on the
importance of the Sky value, see *note Sky value::).

Each HDU/extension in a FITS file is an independent dataset (image or
table) which you can delete from the FITS file, or copy/cut to another
file. For example, with the command below, you can copy NoiseChisel's
‘DETECTIONS’ HDU/extension to another file:

$ astfits xdf-f160w_detected.fits --copy=DETECTIONS -odetections.fits

There are similar options to conveniently cut (‘--cut’, copy, then
remove from the input) or delete (‘--remove’) HDUs from a FITS file
also. See *note HDU information and manipulation:: for more.

---------- Footnotes ----------

(1) To learn more about numeric data types see *note Numeric data
types::.

File: gnuastro.info, Node: NoiseChisel optimization for detection, Next: NoiseChisel optimization for storage, Prev: NoiseChisel and Multi-Extension FITS files, Up: General program usage tutorial

2.1.11 NoiseChisel optimization for detection
---------------------------------------------

In *note NoiseChisel and Multi-Extension FITS files::, we ran
NoiseChisel and reviewed NoiseChisel's output format. Now that you have
a better feeling for multi-extension FITS files, let's optimize
NoiseChisel for this particular dataset.

One good way to see if you have missed any signal (small galaxies, or
the wings of brighter galaxies) is to mask all the detected pixels and
inspect the noise pixels. For this, you can use Gnuastro's Arithmetic
program (in particular its ‘where’ operator, see *note Arithmetic
operators::). The command below will produce ‘mask-det.fits’. In it,
all the pixels in the ‘INPUT-NO-SKY’ extension that are flagged 1 in the
‘DETECTIONS’ extension (dominated by signal, not noise) will be set to
NaN.

Since the various extensions are in the same file, for each dataset
we need the file and extension name. To make the command easier to
read/write/understand, let's use shell variables: '‘in’' will be used
for the Sky-subtracted input image and '‘det’' will be used for the
detection map. Recall that a shell variable's value can be retrieved by
adding a ‘$’ before its name, also note that the double quotations are
necessary when we have white-space characters in a variable value (like
this case).

$ in="xdf-f160w_detected.fits -hINPUT-NO-SKY"
$ det="xdf-f160w_detected.fits -hDETECTIONS"
$ astarithmetic $in $det nan where --output=mask-det.fits

To invert the result (only keep the detected pixels), you can flip the
detection map (from 0 to 1 and vice-versa) by adding a '‘not’' after the
second ‘$det’:

$ astarithmetic $in $det not nan where --output=mask-sky.fits

Look again at the ‘DETECTIONS’ extension, in particular the long
worm-like structure around (1) pixel 1650 (X) and 1470 (Y). These types
of long wiggly structures show that we have dug too deep into the noise,
and are a signature of correlated noise. Correlated noise is created
when we warp (for example, rotate) individual exposures (that are each
slightly offset compared to each other) into the same pixel grid before
adding them into one deeper image. During the warping, nearby pixels
are mixed and the effect of this mixing on the noise (which is in every
pixel) is called "correlated noise". Correlated noise is a form of
convolution and it slightly smooths the image.

In terms of the number of exposures (and thus correlated noise), the
XDF dataset is by no means an ordinary dataset. Therefore the default
parameters need to be slightly customized. It is the result of warping
and adding roughly 80 separate exposures which can create strong
correlated noise/smoothing. In common surveys the number of exposures
is usually 10 or less. See Figure 2 of Akhlaghi 2019
(https://arxiv.org/abs/1909.11230) and the discussion on
‘--detgrowquant’ there for more on how NoiseChisel "grow"s the detected
objects and the patterns caused by correlated noise.

Let's tweak NoiseChisel's configuration a little to get a better
result on this dataset. Do not forget that "_Good statistical analysis
is not a purely routine matter, and generally calls for more than one
pass through the computer_" (Anscombe 1973, see *note Science and its
tools::). A good scientist must have a good understanding of her tools
to make a meaningful analysis. So do not hesitate in playing with the
default configuration and reviewing the manual when you have a new
dataset (from a new instrument) in front of you. Robust data analysis
is an art, therefore a good scientist must first be a good artist. Once
you have found the good configuration for that particular noise pattern
(instrument) you can safely use it for all new data that have a similar
noise pattern.

NoiseChisel can produce "Check images" to help you visualize and
inspect how each step is done. You can see all the check images it can
produce with this command.

$ astnoisechisel --help | grep check

Let's check the overall detection process to get a better feeling of
what NoiseChisel is doing with the following command. To learn the
details of NoiseChisel in more detail, please see *note NoiseChisel::,
Akhlaghi and Ichikawa 2015 (https://arxiv.org/abs/1505.01664) and
Akhlaghi 2019 (https://arxiv.org/abs/1909.11230).

$ astnoisechisel flat-ir/xdf-f160w.fits --checkdetection

The check images/tables are also multi-extension FITS files. As you
saw from the command above, when check datasets are requested,
NoiseChisel will not go to the end. It will abort as soon as all the
extensions of the check image are ready. Please list the extensions of
the output with ‘astfits’ and then opening it with ‘ds9’ as we done
above. If you have read the paper, you will see why there are so many
extensions in the check image.

$ astfits xdf-f160w_detcheck.fits
$ astscript-fits-view xdf-f160w_detcheck.fits

In order to understand the parameters and their biases (especially as
you are starting to use Gnuastro, or running it a new dataset), it is
_strongly_ encouraged to play with the different parameters and use the
respective check images to see which step is affected by your changes
and how, for example, see *note Detecting large extended targets::.

Let's focus on one step: the ‘OPENED_AND_LABELED’ extension shows the
initial detection step of NoiseChisel. We see the seeds of that
correlated noise structure with many small detections (a relatively
early stage in the processing). Such connections at the lowest surface
brightness limits usually occur when the dataset is too smoothed, the
threshold is too low, or the final "growth" is too much.

As you see from the 2nd (‘CONVOLVED’) extension, the first operation
that NoiseChisel does on the data is to slightly smooth it. However,
the natural correlated noise of this dataset is already one level of
artificial smoothing, so further smoothing it with the default kernel
may be the culprit. To see the effect, let's use a sharper kernel as a
first step to convolve/smooth the input.

By default NoiseChisel uses a Gaussian with full-width-half-maximum
(FWHM) of 2 pixels. We can use Gnuastro's MakeProfiles to build a
kernel with FWHM of 1.5 pixel (truncated at 5 times the FWHM, like the
default) using the following command. MakeProfiles is a powerful tool
to build any number of mock profiles on one image or independently, to
learn more of its features and capabilities, see *note MakeProfiles::.

$ astmkprof --kernel=gaussian,1.5,5 --oversample=1

Please open the output ‘kernel.fits’ and have a look (it is very small
and sharp). We can now tell NoiseChisel to use this instead of the
default kernel with the following command (we will keep the
‘--checkdetection’ to continue checking the detection steps)

$ astnoisechisel flat-ir/xdf-f160w.fits --kernel=kernel.fits \
--checkdetection

Open the output ‘xdf-f160w_detcheck.fits’ as a multi-extension FITS
file and go to the last extension (‘DETECTIONS-FINAL’, it is the same
pixels as the final NoiseChisel output without ‘--checkdetections’).
Look again at that position mentioned above (1650,1470), you see that
the long wiggly structure is gone. This shows we are making progress
:-).

Looking at the new ‘OPENED_AND_LABELED’ extension, we see that the
thin connections between smaller peaks has now significantly decreased.
Going two extensions/steps ahead (in the first ‘HOLES-FILLED’), you can
see that during the process of finding false pseudo-detections, too many
holes have been filled: do you see how the many of the brighter galaxies
are connected? At this stage all holes are filled, irrespective of
their size.

Try looking two extensions ahead (in the first ‘PSEUDOS-FOR-SN’), you
can see that there are not too many pseudo-detections because of all
those extended filled holes. If you look closely, you can see the
number of pseudo-detections in the printed outputs of NoiseChisel
(around 6400). This is another side-effect of correlated noise. To
address it, we should slightly increase the pseudo-detection threshold
(before changing ‘--dthresh’, run with ‘-P’ to see the default value):

$ astnoisechisel flat-ir/xdf-f160w.fits --kernel=kernel.fits \
--dthresh=0.1 --checkdetection

Before visually inspecting the check image, you can already see the
effect of this small change in NoiseChisel's command-line output: notice
how the number of pseudo-detections has increased to more than 7100!
Open the check image now and have a look, you can see how the
pseudo-detections are distributed much more evenly in the blank sky
regions of the ‘PSEUDOS-FOR-SN’ extension.

*Maximize the number of pseudo-detections:* When using NoiseChisel on
datasets with a new noise-pattern (for example, going to a Radio
astronomy image, or a shallow ground-based image), play with ‘--dthresh’
until you get a maximal number of pseudo-detections: the total number of
pseudo-detections is printed on the command-line when you run
NoiseChisel, you do not even need to open a FITS viewer.

In this particular case, try ‘--dthresh=0.2’ and you will see that
the total printed number decreases to around 6700 (recall that with
‘--dthresh=0.1’, it was roughly 7100). So for this type of very deep
HST images, we should set ‘--dthresh=0.1’.

As discussed in Section 3.1.5 of Akhlaghi and Ichikawa 2015
(https://arxiv.org/abs/1505.01664), the signal-to-noise ratio of
pseudo-detections are critical to identifying/removing false detections.
For an optimal detection they are very important to get right (where you
want to detect the faintest and smallest objects in the image
successfully). Let's have a look at their signal-to-noise distribution
with ‘--checksn’.

$ astnoisechisel flat-ir/xdf-f160w.fits --kernel=kernel.fits \
--dthresh=0.1 --checkdetection --checksn

The output (‘xdf-f160w_detsn.fits’) contains two extensions for the
pseudo-detections containing two-column tables over the undetected
(‘SKY_PSEUDODET_SN’) regions and those over detections
(‘DET_PSEUDODET_SN’). With the first command below you can see the HDUs
of this file, and with the second you can see the information of the
table in the first HDU (which is the default when you do not use
‘--hdu’):

$ astfits xdf-f160w_detsn.fits
$ asttable xdf-f160w_detsn.fits -i

You can see the table columns with the first command below and get a
feeling of the signal-to-noise value distribution with the second
command (the two Table and Statistics programs will be discussed later
in the tutorial):

$ asttable xdf-f160w_detsn.fits -hSKY_PSEUDODET_SN
$ aststatistics xdf-f160w_detsn.fits -hSKY_PSEUDODET_SN -c2
... [output truncated] ...
Histogram:
| *
| ***
| ******
| *********
| **********
| *************
| *****************
| ********************
| **************************
| ********************************
|******************************************************* * ** *
|----------------------------------------------------------------------

The correlated noise is again visible in the signal-to-noise
distribution of sky pseudo-detections! Do you see how skewed this
distribution is? In an image with less correlated noise, this
distribution would be much more symmetric. A small change in the
quantile will translate into a big change in the S/N value. For
example, see the difference between the three 0.99, 0.95 and 0.90
quantiles with this command:

$ aststatistics xdf-f160w_detsn.fits -hSKY_PSEUDODET_SN -c2 \
--quantile=0.99 --quantile=0.95 --quantile=0.90

We get a change of almost 2 units (which is very significant). If
you run NoiseChisel with ‘-P’, you'll see the default signal-to-noise
quantile ‘--snquant’ is 0.99. In effect with this option you specify
the purity level you want (contamination by false detections). With the
‘aststatistics’ command above, you see that a small number of extra
false detections (impurity) in the final result causes a big change in
completeness (you can detect more lower signal-to-noise true
detections). So let's loosen-up our desired purity level, remove the
check-image options, and then mask the detected pixels like before to
see if we have missed anything.

$ astnoisechisel flat-ir/xdf-f160w.fits --kernel=kernel.fits \
--dthresh=0.1 --snquant=0.95
$ in="xdf-f160w_detected.fits -hINPUT-NO-SKY"
$ det="xdf-f160w_detected.fits -hDETECTIONS"
$ astarithmetic $in $det nan where --output=mask-det.fits

Overall it seems good, but if you play a little with the color-bar
and look closer in the noise, you'll see a few very sharp, but faint,
objects that have not been detected. For example, the object around
pixel (456, 1662). Despite its high valued pixels, this object was lost
because erosion ignores the precise pixel values. Losing small/sharp
objects like this only happens for under-sampled datasets like HST
(where the pixel size is larger than the point spread function FWHM). So
this will not happen on ground-based images.

To address this problem of sharp objects, we can use NoiseChisel's
‘--noerodequant’ option. All pixels above this quantile will not be
eroded, thus allowing us to preserve small/sharp objects (that cover a
small area, but have a lot of signal in it). Check its default value,
then run NoiseChisel like below and make the mask again.

$ astnoisechisel flat-ir/xdf-f160w.fits --kernel=kernel.fits \
--noerodequant=0.95 --dthresh=0.1 --snquant=0.95

This seems to be fine and the object above is now detected. We will
stop editing the configuration of NoiseChisel here, but please feel free
to keep looking into the data to see if you can improve it even more.

Once you have found the proper configuration for the type of images
you will be using you do not need to change them any more. The same
configuration can be used for any dataset that has been similarly
produced (and has a similar noise pattern). But entering all these
options on every call to NoiseChisel is annoying and prone to bugs
(mistakenly typing the wrong value for example). To simplify things, we
will make a configuration file in a visible ‘config’ directory. Then we
will define the hidden ‘.gnuastro’ directory (that all Gnuastro's
programs will look into for configuration files) as a symbolic link to
the ‘config’ directory. Finally, we will write the finalized values of
the options into NoiseChisel's standard configuration file within that
directory. We will also put the kernel in a separate directory to keep
the top directory clean of any files we later need.

$ mkdir kernel config
$ ln -s config/ .gnuastro
$ mv kernel.fits kernel/noisechisel.fits
$ echo "kernel kernel/noisechisel.fits" > config/astnoisechisel.conf
$ echo "noerodequant 0.95" >> config/astnoisechisel.conf
$ echo "dthresh 0.1" >> config/astnoisechisel.conf
$ echo "snquant 0.95" >> config/astnoisechisel.conf

We are now ready to finally run NoiseChisel on the three filters and
keep the output in a dedicated directory (which we will call ‘nc’ for
simplicity).
$ rm *.fits
$ mkdir nc
$ for f in f105w f125w f160w; do \
astnoisechisel flat-ir/xdf-$f.fits --output=nc/xdf-$f.fits; \
done

---------- Footnotes ----------

(1) To find a particular coordiante easily in DS9, you can do this:
Click on the "Edit" menu, and select "Region". Then click on any random
part of the image to see a circle show up in that location (this is the
"region"). Double-click on the region and a "Circle" window will open.
If you have celestial coordinates, keep the default "fk5" in the
scroll-down menu after the "Center". But if you have pixel/image
coordinates, click on the "fk5" and select "Image". Now you can set the
"Center" coordinates of the region (‘1650’ and ‘1470’ in this case) by
manually typing them in the two boxes in front of "Center". Finally,
when everything is ready, click on the "Apply" button and your region
will go over your requested coordinates. You can zoom out (to see the
whole image) and visually find it.

File: gnuastro.info, Node: NoiseChisel optimization for storage, Next: Segmentation and making a catalog, Prev: NoiseChisel optimization for detection, Up: General program usage tutorial

2.1.12 NoiseChisel optimization for storage
-------------------------------------------

As we showed before (in *note NoiseChisel and Multi-Extension FITS
files::), NoiseChisel's output is a multi-extension FITS file with
several images the same size as the input. As the input datasets get
larger this output can become hard to manage and waste a lot of storage
space. Fortunately there is a solution to this problem (which is also
useful for Segment's outputs).

In this small section we will take a short detour to show this
feature. Please note that the outputs generated here are not needed for
the rest of the tutorial. But first, let's have a look at the
contents/HDUs and volume of NoiseChisel's output from *note NoiseChisel
optimization for detection:: (fast answer, it is larger than 100
mega-bytes):

$ astfits nc/xdf-f160w.fits
$ ls -lh nc/xdf-f160w.fits

Two options can drastically decrease NoiseChisel's output file size:
1) With the ‘--rawoutput’ option, NoiseChisel will not create a
Sky-subtracted output. After all, it is redundant: you can always
generate it by subtracting the ‘SKY’ extension from the input image
(which you have in your database) using the Arithmetic program. 2) With
the ‘--oneelempertile’, you can tell NoiseChisel to store its Sky and
Sky standard deviation results with one pixel per tile (instead of many
pixels per tile). So let's run NoiseChisel with these options, then
have another look at the HDUs and the over-all file size:

$ astnoisechisel flat-ir/xdf-f160w.fits --oneelempertile --rawoutput \
--output=nc-for-storage.fits
$ astfits nc-for-storage.fits
$ ls -lh nc-for-storage.fits

See how ‘nc-for-storage.fits’ has four HDUs, while ‘nc/xdf-f160w.fits’
had five HDUs? As explained above, the missing extension is
‘INPUT-NO-SKY’. Also, look at the sizes of the ‘SKY’ and ‘SKY_STD’
HDUs, unlike before, they are not the same size as ‘DETECTIONS’, they
only have one pixel for each tile (group of pixels in raw input).
Finally, you see that ‘nc-for-storage.fits’ is just under 8 mega bytes
(while ‘nc/xdf-f160w.fits’ was 100 mega bytes)!

But we are not yet finished! You can even be more efficient in
storage, archival or transferring NoiseChisel's output by compressing
this file. Try the command below to see how NoiseChisel's output has
now shrunk to about 250 kilo-byes while keeping all the necessary
information as the original 100 mega-byte output.

$ gzip --best nc-for-storage.fits
$ ls -lh nc-for-storage.fits.gz

We can get this wonderful level of compression because NoiseChisel's
output is binary with only two values: 0 and 1. Compression algorithms
are highly optimized in such scenarios.

You can open ‘nc-for-storage.fits.gz’ directly in SAO DS9 or feed it
to any of Gnuastro's programs without having to decompress it.
Higher-level programs that take NoiseChisel's output (for example,
Segment or MakeCatalog) can also deal with this compressed image where
the Sky and its Standard deviation are one pixel-per-tile. You just
have to give the "values" image as a separate option, for more, see
*note Segment:: and *note MakeCatalog::.

Segment (the program we will introduce in the next section for
identifying sub-structure), also has similar features to optimize its
output for storage. Since this file was only created for a fast detour
demonstration, let's keep our top directory clean and move to the next
step:

rm nc-for-storage.fits.gz

File: gnuastro.info, Node: Segmentation and making a catalog, Next: Measuring the dataset limits, Prev: NoiseChisel optimization for storage, Up: General program usage tutorial

2.1.13 Segmentation and making a catalog
----------------------------------------

The main output of NoiseChisel is the binary detection map (‘DETECTIONS’
extension, see *note NoiseChisel optimization for detection::). It only
has two values: 1 or 0. This is useful when studying the noise or
background properties, but hardly of any use when you actually want to
study the targets/galaxies in the image, especially in such a deep field
where almost everything is connected. To find the galaxies over the
detections, we will use Gnuastro's *note Segment:: program:

$ mkdir seg
$ astsegment nc/xdf-f160w.fits -oseg/xdf-f160w.fits
$ astsegment nc/xdf-f125w.fits -oseg/xdf-f125w.fits
$ astsegment nc/xdf-f105w.fits -oseg/xdf-f105w.fits

Segment's operation is very much like NoiseChisel (in fact, prior to
version 0.6, it was part of NoiseChisel). For example, the output is a
multi-extension FITS file (previously discussed in *note NoiseChisel and
Multi-Extension FITS files::), it has check images and uses the
undetected regions as a reference (previously discussed in *note
NoiseChisel optimization for detection::). Please have a look at
Segment's multi-extension output to get a good feeling of what it has
done. Do not forget to flip through the extensions in the "Cube"
window.

$ astscript-fits-view seg/xdf-f160w.fits

Like NoiseChisel, the first extension is the input. The ‘CLUMPS’
extension shows the true "clumps" with values that are $\ge1$, and the
diffuse regions labeled as $-1$. Please flip between the first
extension and the clumps extension and zoom-in on some of the clumps to
get a feeling of what they are. In the ‘OBJECTS’ extension, we see that
the large detections of NoiseChisel (that may have contained many
galaxies) are now broken up into separate labels. Play with the
color-bar and hover your mouse of the various detections to see their
different labels.

The clumps are not affected by the hard-to-deblend and low
signal-to-noise diffuse regions, they are more robust for calculating
the colors (compared to objects). From this step onward, we will
continue with clumps.

Having localized the regions of interest in the dataset, we are ready
to do measurements on them with *note MakeCatalog::. MakeCatalog is
specialized and optimized for doing measurements over labeled regions of
an image. In other words, through MakeCatalog, you can "reduce" an
image to a table (catalog of certain properties of objects in the
image). Each requested measurement (over each label) will be given a
column in the output table. To see the full set of available
measurements run it with ‘--help’ like below (and scroll up), note that
measurements are classified by context.

$ astmkcatalog --help

So let's select the properties we want to measure in this tutorial.
First of all, we need to know which measurement belongs to which object
or clump, so we will start with the ‘--ids’ (read as: IDs(1)). We also
want to measure (in this order) the Right Ascension (with ‘--ra’),
Declination (‘--dec’), magnitude (‘--magnitude’), and signal-to-noise
ratio (‘--sn’) of the objects and clumps. Furthermore, as mentioned
above, we also want measurements on clumps, so we also need to call
‘--clumpscat’. The following command will make these measurements on
Segment's F160W output and write them in a catalog for each object and
clump in a FITS table. For more on the zero point, see *note Brightness
flux magnitude::.

$ mkdir cat
$ astmkcatalog seg/xdf-f160w.fits --ids --ra --dec --magnitude --sn \
--zeropoint=25.94 --clumpscat --output=cat/xdf-f160w.fits

From the printed statements on the command-line, you see that
MakeCatalog read all the extensions in Segment's output for the various
measurements it needed. Let's look at the output of the command above:

$ astfits cat/xdf-f160w.fits

You will see that the output of the MakeCatalog has two extensions.
The first extension shows the measurements over the ‘OBJECTS’, and the
second extension shows the measurements over the clumps ‘CLUMPS’.

To calculate colors, we also need magnitude measurements on the other
filters. So let's repeat the command above on them, just changing the
file names and zero point (which we got from the XDF survey web page):

$ astmkcatalog seg/xdf-f125w.fits --ids --ra --dec --magnitude --sn \
--zeropoint=26.23 --clumpscat --output=cat/xdf-f125w.fits

$ astmkcatalog seg/xdf-f105w.fits --ids --ra --dec --magnitude --sn \
--zeropoint=26.27 --clumpscat --output=cat/xdf-f105w.fits

However, the galaxy properties might differ between the filters
(which is the whole purpose behind observing in different filters!).
Also, the noise properties and depth of the datasets differ. You can
see the effect of these factors in the resulting clump catalogs, with
Gnuastro's Table program. We will go deep into working with tables in
the next section, but in summary: the ‘-i’ option will print information
about the columns and number of rows. To see the column values, just
remove the ‘-i’ option. In the output of each command below, look at
the ‘Number of rows:’, and note that they are different.

$ asttable cat/xdf-f105w.fits -hCLUMPS -i
$ asttable cat/xdf-f125w.fits -hCLUMPS -i
$ asttable cat/xdf-f160w.fits -hCLUMPS -i

Matching the catalogs is possible (for example, with *note Match::).
However, the measurements of each column are also done on different
pixels: the clump labels can/will differ from one filter to another for
one object. Please open them and focus on one object to see for
yourself. This can bias the result, if you match catalogs.

An accurate color calculation can only be done when magnitudes are
measured from the same pixels on all images and this can be done easily
with MakeCatalog. In fact this is one of the reasons that NoiseChisel
or Segment do not generate a catalog like most other
detection/segmentation software. This gives you the freedom of
selecting the pixels for measurement in any way you like (from other
filters, other software, manually, etc.). Fortunately in these images,
the Point spread function (PSF) is very similar, allowing us to use a
single labeled image output for all filters(2).

The F160W image is deeper, thus providing better
detection/segmentation, and redder, thus observing smaller/older stars
and representing more of the mass in the galaxies. We will thus use the
F160W filter as a reference and use its segment labels to identify which
pixels to use for which objects/clumps. But we will do the measurements
on the sky-subtracted F105W and F125W images (using MakeCatalog's
‘--valuesfile’ option) as shown below: Notice that the only difference
between these calls and the call to generate the raw F160W catalog
(excluding the zero point and the output name) is the ‘--valuesfile’.

$ astmkcatalog seg/xdf-f160w.fits --ids --ra --dec --magnitude --sn \
--valuesfile=nc/xdf-f125w.fits --zeropoint=26.23 \
--clumpscat --output=cat/xdf-f125w-on-f160w-lab.fits

$ astmkcatalog seg/xdf-f160w.fits --ids --ra --dec --magnitude --sn \
--valuesfile=nc/xdf-f105w.fits --zeropoint=26.27 \
--clumpscat --output=cat/xdf-f105w-on-f160w-lab.fits

After running the commands above, look into what MakeCatalog printed
on the command-line. You can see that (as requested) the object and
clump pixel labels in both were taken from the respective extensions in
‘seg/xdf-f160w.fits’. However, the pixel values and pixel Sky standard
deviation were respectively taken from ‘nc/xdf-f105w.fits’ and
‘nc/xdf-f125w.fits’. Since we used the same labeled image on all
filters, the number of rows in both catalogs are now identical. Let's
have a look:

$ asttable cat/xdf-f105w-on-f160w-lab.fits -hCLUMPS -i
$ asttable cat/xdf-f125w-on-f160w-lab.fits -hCLUMPS -i
$ asttable cat/xdf-f160w.fits -hCLUMPS -i

Finally, MakeCatalog also does basic calculations on the full dataset
(independent of each labeled region but related to whole data), for
example, pixel area or per-pixel surface brightness limit. They are
stored as keywords in the FITS headers (or lines starting with ‘#’ in
plain text). This (and other ways to measure the limits of your
dataset) are discussed in the next section: *note Measuring the dataset
limits::.

---------- Footnotes ----------

(1) This option is plural because we need two ID columns for
identifying "clumps" in the clumps catalog/table: the first column will
be the ID of the host "object", and the second one will be the ID of the
clump within that object. In the "objects" catalog/table, only a single
column will be returned for this option.

(2) When the PSFs between two images differ largely, you would have
to PSF-match the images before using the same pixels for measurements.

File: gnuastro.info, Node: Measuring the dataset limits, Next: Working with catalogs estimating colors, Prev: Segmentation and making a catalog, Up: General program usage tutorial

2.1.14 Measuring the dataset limits
-----------------------------------

In *note Segmentation and making a catalog::, we created a catalog of
the different objects with the image. Before measuring colors, or doing
any other kind of analysis on the catalogs (and detected objects), it is
very important to understand the limitations of the dataset. Without
understanding the limitations of your dataset, you cannot make any
physical interpretation of your results. The theory behind the
calculations discussed here is thoroughly introduced in *note
Quantifying measurement limits::.

For example, with the command below, let's sort all the detected
clumps in the image by magnitude (with ‘--sort=magnitude’) and and print
the magnitude and signal-to-noise ratio (S/N; with ‘-cmagnitude,sn’):

$ asttable cat/xdf-f160w.fits -hclumps -cmagnitude,sn \
--sort=magnitude --noblank=magnitude

As you see, we have clumps with a total magnitude of almost 32! This
is _extremely faint_! Are these things trustable? Let's have a look at
all of those with a magnitude between 31 and 32 with the command below.
We are first using Table to only keep the relevant columns rows, and
using Gnuastro's DS9 region file creation script
(‘astscript-ds9-region’) to generate DS9 region files, and open DS9:

$ asttable cat/xdf-f160w.fits -hclumps -cra,dec \
--range=magnitude,31:32 \
| astscript-ds9-region -c1,2 --radius=0.5 \
--command="ds9 -mecube seg/xdf-f160w.fits -zscale"

Zoom-out a little and you will see some green circles (DS9 region
files) in some regions of the image. There actually does seem to be a
true peak under the selected regions, but as you see, they are very
small, diffuse and noisy. How reliable are the measured magnitudes?
Using the S/N column from the first command above, you can see that such
objects only have a signal to noise of about 2.6 (which is indeed too
low for most analysis purposes)

$ asttable cat/xdf-f160w.fits -hclumps -csn \
--range=magnitude,31:32 | aststatistics

This brings us to the first method of quantifying your dataset's
_magnitude limit_, which is also sometimes called _detection limit_ (see
*note Magnitude limit of image::). To estimate the $5\sigma$ detection
limit of your dataset, you simply report the median magnitude of the
objects that have a signal to noise of (approximately) five. This is
very easy to calculate with the command below:

$ asttable cat/xdf-f160w.fits -hclumps --range=sn,4.8:5.2 -cmagnitude \
| aststatistics --median
29.9949

Let's have a look at these objects, to get a feeling of what these
clump looks like:

$ asttable cat/xdf-f160w.fits -hclumps --range=sn,4.8:5.2 \
-cra,dec,magnitude \
| astscript-ds9-region -c1,2 --namecol=3 \
--width=2 --radius=0.5 \
--command="ds9 -mecube seg/xdf-f160w.fits -zscale"

The number you see on top of each region is the clump's magnitude.
Please go over the objects and have a close look at them! It is very
important to have a feeling of what your dataset looks like, and how to
interpret the numbers to associate an image with them.

Generally, they look very small with different levels of
diffuse-ness! Those that are sharper make more visual sense (to be
$5\sigma$ detections), but the more diffuse ones extend over a larger
area. Furthermore, the noise is measured on individual pixel
measurements. However, during the reduction many exposures are co-added
and stacked, mixing the pixels like a small convolution (creating
"correlated noise"). Therefore you clearly see two main issues with the
detection limit as defined above: it depends on the morphology, and it
does not take into account the correlated noise.

A more realistic way to estimate the significance of the detection is
to take its footprint, randomly place it in thousands of undetected
regions of the image and use that distribution as a reference. This is
technically known as upper-limit measurements. For a full discussion,
see *note Upper limit magnitude of each detection::).

Since it is for each separate object, the upper-limit measurements
should be requested as extra columns in MakeCatalog's output. For
example, with the command below, let's generate a new catalog of the
F160W filter, but with two extra columns compared to the one in ‘cat/’:
the upper-limit magnitude and the upper-limit multiple of sigma.

$ astmkcatalog seg/xdf-f160w.fits --ids --ra --dec --magnitude --sn \
--zeropoint=25.94 --clumpscat --upnsigma=3 \
--upperlimit-mag --upperlimit-sigma \
--output=xdf-f160w.fits

Let's compare the upper-limit magnitude with the measured magnitude of
each clump:

$ asttable xdf-f160w.fits -hclumps -cmagnitude,upperlimit_mag

As you see, in almost all of the cases, the measured magnitude is
sufficiently higher than the upper-limit magnitude. Let's subtract the
latter from the former to better see this difference in a third column:

$ asttable xdf-f160w.fits -hclumps -cmagnitude,upperlimit_mag \
-c'arith upperlimit_mag magnitude -'

The ones with a positive third column (difference) show that the
clump has sufficiently higher brightness than the noisy background to be
usable. Let's use Table's *note Column arithmetic:: to find only those
that have a negative difference:

$ asttable xdf-f160w.fits -hclumps -cra,dec --noblankend=3 \
-c'arith upperlimit_mag magnitude - set-d d d 0 gt nan where'

From more than 3500 clumps, this command only gave $\sim150$ rows (this
number may slightly change on different runs due to the random nature of
the upper-limit sampling(1))! Let's have a look at them:

$ asttable xdf-f160w.fits -hclumps -cra,dec --noblankend=3 \
-c'arith upperlimit_mag magnitude - set-d d d 0 gt nan where' \
| astscript-ds9-region -c1,2 --namecol=3 --width=2 \
--radius=0.5 \
--command="ds9 -mecube seg/xdf-f160w.fits -zscale"

You see that they are all extremely faint and diffuse/small peaks.
Therefore, if an object's magnitude is fainter than its upper-limit
magnitude, you should not use the magnitude: it is not accurate! You
should use the upper-limit magnitude instead (with an arrow in your
plots to mark which ones are upper-limits).

But the main point (in relation to the magnitude limit) with the
upper-limit, is the ‘UPPERLIMIT_SIGMA’ column. you can think of this as
a _realistic_ S/N for extremely faint/diffuse/small objects). The raw
S/N column is simply calculated on a pixel-by-pixel basis, however, the
upper-limit sigma is produced by actually taking the label's footprint,
and randomly placing it thousands of time over un-detected parts of the
image and measuring the brightness of the sky. The clump's brightness
is then divided by the standard deviation of the resulting distribution
to give you exactly how significant it is (accounting for inter-pixel
issues like correlated noise, which are strong in this dataset). You
can actually compare the two values with the command below:

$ asttable xdf-f160w.fits -hclumps -csn,upperlimit_sigma

As you see, the second column (upper-limit sigma) is almost always
less than the S/N. This clearly shows the effect of correlated noise!
If you now use this column as the reference for deriving the magnitude
limit, you will see that it will shift by almost 0.5 magnitudes brighter
and is now more reasonable:

$ asttable xdf-f160w.fits -hclumps --range=upperlimit_sigma,4.8:5.2 \
-cmagnitude | aststatistics --median
29.6257

We see that the $5\sigma$ detection limit is $\sim29.6$! This is
extremely deep! For example, in the Legacy Survey(2), the $5\sigma$
detection limit for _point sources_ is approximately 24.5 (5 magnitudes,
or 100 times, shallower than this image).

As mentioned above, an important caveat in this simple calculation is
that we should only be looking at point-like objects, not simply
everything. This is because the shape or radial slope of the profile
has an important effect on this measurement: at the same total
magnitude, a sharper object will have a higher S/N. To be more precise,
we should first perform star-galaxy separation, then do this only for
the objects that are classified as stars. A crude, first-order, method
is to use the ‘--axis-ratio’ option so MakeCatalog also measures the
axis ratio, then call Table with ‘--range=upperlimit_sigma,,4.8:5.2’ and
‘--range=axis_ratio,0.95:1’ (in one command). Please do this for
yourself as an exercise to see the difference with the result above.

Before continuing, let's remove this temporarily produced catalog:

$ rm xdf-f160w.fits

Another measure of the dataset's limit is the completeness limit
(*note Completeness limit of each detection::). This is necessary when
you are looking at populations of objects over the image. You want to
know until what magnitude you can be sure that you have detected an
object (if it was present). As described in *note Completeness limit of
each detection::, the best way to do this is with mock images. But a
crude, first order result can be obtained from the actual image: by
simply plotting the histogram of the magnitudes:

$ aststatistics cat/xdf-f160w.fits -hclumps -cmagnitude
...
Histogram:
| *
| ** ****
| ***********
| *************
| ****************
| *******************
| **********************
| **************************
| *********************************
| *********************************************
|* * ** ** **********************************************************
|----------------------------------------------------------------------

This plot (the histogram of magnitudes; where fainter magnitudes are
towards the right) is technically called the dataset's _number count_
plot. You see that the number of objects increases with magnitude as
the magnitudes get fainter (to the right). However, beyond a certain
magnitude, you see it becomes flat, and soon afterwards, the numbers
suddenly drop.

Once you have your catalog, you can easily find this point with the
two commands below. First we generate a histogram with fewer bins (to
have more numbers in each bin). We then use AWK to find the magnitude
bin where the number of points decrease compared to the previous bin.
But we only do this for bins that have more than 50 items (to avoid
scatter in the bright end). Finally, in Statistics, we have manually
set the magnitude range and number of bins so each bin is roughly 0.5
magnitudes thick (with ‘--greaterequal=20’, ‘--lessthan=32’ and
‘--numbins=24’)

$ aststatistics cat/xdf-f160w.fits -hclumps -cmagnitude --histogram \
--greaterequal=20 --lessthan=32 --numbins=24 \
--output=f160w-hist.txt
$ asttable f160w-hist.txt \
| awk '$2>50 && $2<prev{print prevbin; exit} \
{prev=$2; prevbin=$1}'
28.932122667631

Therefore, to first order (and very crudely!) we can say that if an
object is in our field of view and has a magnitude of $\sim29$ or
brighter, we can be highly confident that we have detected it. But
before continuing, let's clean up behind ourselves:

$ rm f160w-hist.txt

Another important limiting parameter in a processed dataset is the
surface brightness limit (*note Surface brightness limit of image::).
The surface brightness limit of a dataset is an important measure for
extended structures (for example, when you want to look at the outskirts
of galaxies). In the next tutorial, we have thoroughly described the
derivation of the surface brightness limit of a dataset. So we will
just show the final result here, and encourage you to follow up with
that tutorial after finishing this tutorial (see *note Image surface
brightness limit::)

By default, MakeCatalog will estimate the surface brightness limit of
a given dataset, and put it in the keywords of the output (all keywords
starting with ‘SBL’, which is short for surface brightness limit):

$ astfits cat/xdf-f160w.fits -h1 | grep SBL

As you see, the only one with a unit of ‘mag/arcsec^2’ is ‘SBLMAG’.
It contains the surface brightness limit of the input dataset over
‘SBLAREA’ arcsec$^2$ with ‘SBLNSIG’ multiples of $\sigma$. In the
current version of Gnuastro, ‘SBLAREA=100’ and ‘SBLNSIG=3’, so the
surface brightness limit of this image is 32.66 mag/arcsec$^2$
($3\sigma$, over 100 arcsec$^2$). Therefore, if this default area and
multiple of sigma are fine for you(3) (these are the most commonly used
values), you can simply read the image surface brightness limit from the
catalogs produced by MakeCatalog with this command:

$ astfits cat/*.fits -h1 --keyvalue=SBLMAG

---------- Footnotes ----------

(1) You can fix the random number generator seed, so you always get
the same sampling, see *note Generating random numbers::.

(2) <https://www.legacysurvey.org/dr9/description>

(3) You can change these values with the ‘--sfmagarea’ and
‘--sfmagnsigma’

File: gnuastro.info, Node: Working with catalogs estimating colors, Next: Column statistics color-magnitude diagram, Prev: Measuring the dataset limits, Up: General program usage tutorial

2.1.15 Working with catalogs (estimating colors)
------------------------------------------------

In the previous step we generated catalogs of objects and clumps over
our dataset (see *note Segmentation and making a catalog::). The
catalogs are available in the two extensions of the single FITS file(1).
Let's see the extensions and their basic properties with the Fits
program:

$ astfits cat/xdf-f160w.fits # Extension information

Let's inspect the table in each extension with Gnuastro's Table
program (see *note Table::). We should have used ‘-hOBJECTS’ and
‘-hCLUMPS’ instead of ‘-h1’ and ‘-h2’ respectively. The numbers are
just used here to convey that both names or numbers are possible, in the
next commands, we will just use names.

$ asttable cat/xdf-f160w.fits -h1 --info # Objects catalog info.
$ asttable cat/xdf-f160w.fits -h1 # Objects catalog columns.
$ asttable cat/xdf-f160w.fits -h2 -i # Clumps catalog info.
$ asttable cat/xdf-f160w.fits -h2 # Clumps catalog columns.

As you see above, when given a specific table (file name and extension),
Table will print the full contents of all the columns. To see the basic
metadata about each column (for example, name, units and comments),
simply append a ‘--info’ (or ‘-i’) to the command.

To print the contents of special column(s), just give the column
number(s) (counting from ‘1’) or the column name(s) (if they have one)
to the ‘--column’ (or ‘-c’) option. For example, if you just want the
magnitude and signal-to-noise ratio of the clumps (in the clumps
catalog), you can get it with any of the following commands

$ asttable cat/xdf-f160w.fits -hCLUMPS --column=5,6
$ asttable cat/xdf-f160w.fits -hCLUMPS -c5,SN
$ asttable cat/xdf-f160w.fits -hCLUMPS -c5 -c6
$ asttable cat/xdf-f160w.fits -hCLUMPS -cMAGNITUDE -cSN

Similar to HDUs, when the columns have names, always use the name: it is
so common to mis-write numbers or forget the order later! Using column
names instead of numbers has many advantages:
1. You do not have to worry about the order of columns in the table.
2. It acts as a documentation in the script.
3. Column meta-data (including a name) are not just limited to FITS
tables and can also be used in plain text tables, see *note
Gnuastro text table format::.

Table also has tools to limit the displayed rows. For example, with the
first command below only rows with a magnitude in the range of 29 to 30
will be shown. With the second command, you can further limit the
displayed rows to rows with an S/N larger than 10 (a range between 10 to
infinity). You can further sort the output rows, only show the top (or
bottom) N rows, etc., see *note Table:: for more.

$ asttable cat/xdf-f160w.fits -hCLUMPS --range=MAGNITUDE,28:29
$ asttable cat/xdf-f160w.fits -hCLUMPS \
--range=MAGNITUDE,28:29 --range=SN,10:inf

Now that you are comfortable in viewing table columns and rows, let's
look into merging columns of multiple tables into one table (which is
necessary for measuring the color of the clumps). Since
‘cat/xdf-f160w.fits’ and ‘cat/xdf-f105w-on-f160w-lab.fits’ have exactly
the same number of rows and the rows correspond to the same clump, let's
merge them to have one table with magnitudes in both filters.

We can merge columns with the ‘--catcolumnfile’ option like below.
You give this option a file name (which is assumed to be a table that
has the same number of rows as the main input), and all the table's
columns will be concatenated/appended to the main table. Now, try it
out with the commands below. We will first look at the metadata of the
first table (only the ‘CLUMPS’ extension). With the second command, we
will concatenate the two tables and write them in, ‘two-in-one.fits’ and
finally, we will check the new catalog's metadata.

$ asttable cat/xdf-f160w.fits -i -hCLUMPS
$ asttable cat/xdf-f160w.fits -hCLUMPS --output=two-in-one.fits \
--catcolumnfile=cat/xdf-f125w-on-f160w-lab.fits \
--catcolumnhdu=CLUMPS
$ asttable two-in-one.fits -i

By comparing the two metadata, we see that both tables have the same
number of rows. But what might have attracted your attention more, is
that ‘two-in-one.fits’ has double the number of columns (as expected,
after all, you merged both tables into one file, and did not ask for any
specific column). In fact you can concatenate any number of other
tables in one command, for example:

$ asttable cat/xdf-f160w.fits -hCLUMPS --output=three-in-one.fits \
--catcolumnfile=cat/xdf-f125w-on-f160w-lab.fits \
--catcolumnfile=cat/xdf-f105w-on-f160w-lab.fits \
--catcolumnhdu=CLUMPS --catcolumnhdu=CLUMPS
$ asttable three-in-one.fits -i

As you see, to avoid confusion in column names, Table has
intentionally appended a ‘-1’ to the column names of the first
concatenated table if the column names are already present in the
original table. For example, we have the original ‘RA’ column, and
another one called ‘RA-1’). Similarly a ‘-2’ has been added for the
columns of the second concatenated table.

However, this example clearly shows a problem with this full
concatenation: some columns are identical (for example, ‘HOST_OBJ_ID’
and ‘HOST_OBJ_ID-1’), or not needed (for example, ‘RA-1’ and ‘DEC-1’
which are not necessary here). In such cases, you can use
‘--catcolumns’ to only concatenate certain columns, not the whole table.
For example, this command:

$ asttable cat/xdf-f160w.fits -hCLUMPS --output=two-in-one-2.fits \
--catcolumnfile=cat/xdf-f125w-on-f160w-lab.fits \
--catcolumnhdu=CLUMPS --catcolumns=MAGNITUDE
$ asttable two-in-one-2.fits -i

You see that we have now only appended the ‘MAGNITUDE’ column of
‘cat/xdf-f125w-on-f160w-lab.fits’. This is what we needed to be able to
later subtract the magnitudes. Let's go ahead and add the F105W
magnitudes also with the command below. Note how we need to call
‘--catcolumnhdu’ once for every table that should be appended, but we
only call ‘--catcolumn’ once (assuming all the tables that should be
appended have this column).

$ asttable cat/xdf-f160w.fits -hCLUMPS --output=three-in-one-2.fits \
--catcolumnfile=cat/xdf-f125w-on-f160w-lab.fits \
--catcolumnfile=cat/xdf-f105w-on-f160w-lab.fits \
--catcolumnhdu=CLUMPS --catcolumnhdu=CLUMPS \
--catcolumns=MAGNITUDE
$ asttable three-in-one-2.fits -i

But we are not finished yet! There is a very big problem: it is not
immediately clear which one of ‘MAGNITUDE’, ‘MAGNITUDE-1’ or
‘MAGNITUDE-2’ columns belong to which filter! Right now, you know this
because you just ran this command. But in one hour, you'll start
doubting yourself and will be forced to go through your command history,
trying to figure out if you added F105W first, or F125W. You should
never torture your future-self (or your colleagues) like this! So,
let's rename these confusing columns in the matched catalog.

Fortunately, with the ‘--colmetadata’ option, you can correct the
column metadata of the final table (just before it is written). It
takes four values: 1) the original column name or number, 2) the new
column name, 3) the column unit and 4) the column comments. Since the
comments are usually human-friendly sentences and contain space
characters, you should put them in double quotations like below. For
example, by adding three calls of this option to the previous command,
we write the filter name in the magnitude column name and description.

$ asttable cat/xdf-f160w.fits -hCLUMPS --output=three-in-one-3.fits \
--catcolumnfile=cat/xdf-f125w-on-f160w-lab.fits \
--catcolumnfile=cat/xdf-f105w-on-f160w-lab.fits \
--catcolumnhdu=CLUMPS --catcolumnhdu=CLUMPS \
--catcolumns=MAGNITUDE \
--colmetadata=MAGNITUDE,MAG-F160W,log,"Magnitude in F160W." \
--colmetadata=MAGNITUDE-1,MAG-F125W,log,"Magnitude in F125W." \
--colmetadata=MAGNITUDE-2,MAG-F105W,log,"Magnitude in F105W."
$ asttable three-in-one-3.fits -i

We now have all three magnitudes in one table and can start doing
arithmetic on them (to estimate colors, which are just a subtraction of
magnitudes). To use column arithmetic, simply call the column selection
option (‘--column’ or ‘-c’), put the value in single quotations and
start the value with ‘arith’ (followed by a space) like the example
below. Column arithmetic uses the same "reverse polish notation" as the
Arithmetic program (see *note Reverse polish notation::), with almost
all the same operators (see *note Arithmetic operators::), and some
column-specific operators (that are not available for images). In
column-arithmetic, you can identify columns by number (prefixed with a
‘$’) or name, for more see *note Column arithmetic::.

So let's estimate one color from ‘three-in-one-3.fits’ using column
arithmetic. All the commands below will produce the same output, try
them each and focus on the differences. Note that column arithmetic can
be mixed with other ways to choose output columns (the ‘-c’ option).

$ asttable three-in-one-3.fits -ocolor-cat.fits \
-c1,2,3,4,'arith $5 $7 -'

$ asttable three-in-one-3.fits -ocolor-cat.fits \
-c1,2,RA,DEC,'arith MAG-F125W MAG-F160W -'

$ asttable three-in-one-3.fits -ocolor-cat.fits -c1,2 \
-cRA,DEC --column='arith MAG-F105W MAG-F160W -'

This example again highlights the important point on using column
names: if you do not know the commands before, you have no way of making
sense of the first command: what is in column 5 and 7? why not subtract
columns 3 and 4 from each other? Do you see how cryptic the first one
is? Then look at the last one: even if you have no idea how this table
was created, you immediately understand the desired operation. *When
you have column names, please use them.* If your table does not have
column names, give them names with the ‘--colmetadata’ (described above)
as you are creating them. But how about the metadata for the column you
just created with column arithmetic? Have a look at the column metadata
of the table produced above:

$ asttable color-cat.fits -i

The name of the column produced by arithmetic column is ‘ARITH_1’!
This is natural: Arithmetic has no idea what the modified column is!
You could have multiplied two columns, or done much more complex
transformations with many columns. _Metadata cannot be set
automatically, your (the human) input is necessary._ To add metadata,
you can use ‘--colmetadata’ like before:

$ asttable three-in-one-3.fits -ocolor-cat.fits -c1,2,RA,DEC \
--column='arith MAG-F105W MAG-F160W -' \
--colmetadata=ARITH_1,F105W-F160W,log,"Magnitude difference"
$ asttable color-cat.fits -i

Sometimes, because of a particular way of storing data, you might
need to take all input columns. If there are many columns (for example
hundreds!), listing them (like above) will become annoying, buggy and
time-consuming. In such cases, you can give ‘-c_all’. Upon execution,
‘_all’ will be replaced with a comma-separated list of all the input
columns. This allows you to add new columns easily, without having to
worry about the number of input columns that you want anyway. A
lower-level but more customizable method is to use the ‘seq’ (sequence)
command with the ‘-s’ (separator) option set to ‘','’). For example, if
you have 216 columns and only want to return columns 1 and 2 as well as
all the columns between 12 to 58 (inclusive), you can use the command
below:

$ asttable table.fits -c1,2,$(seq -s',' 12 58)

We are now ready to make our final table. We want it to have the
magnitudes in all three filters, as well as the three possible colors.
Recall that by convention in astronomy colors are defined by subtracting
the bluer magnitude from the redder magnitude. In this way a larger
color value corresponds to a redder object. So from the three
magnitudes, we can produce three colors (as shown below). Also, because
this is the final table we are creating here and want to use it later,
we will store it in ‘cat/’ and we will also give it a clear name and use
the ‘--range’ option to only print columns with a signal-to-noise ratio
(‘SN’ column, from the F160W filter) above 5.

$ asttable three-in-one-3.fits --range=SN,5,inf -c1,2,RA,DEC,SN \
-cMAG-F160W,MAG-F125W,MAG-F105W \
-c'arith MAG-F125W MAG-F160W -' \
-c'arith MAG-F105W MAG-F125W -' \
-c'arith MAG-F105W MAG-F160W -' \
--colmetadata=SN,SN-F160W,ratio,"F160W signal to noise ratio" \
--colmetadata=ARITH_1,F125W-F160W,log,"Color F125W-F160W." \
--colmetadata=ARITH_2,F105W-F125W,log,"Color F105W-F125W." \
--colmetadata=ARITH_3,F105W-F160W,log,"Color F105W-F160W." \
--output=cat/mags-with-color.fits
$ asttable cat/mags-with-color.fits -i

The table now has all the columns we need and it has the proper
metadata to let us safely use it later (without frustrating over column
orders!) or passing it to colleagues.

Let's finish this section of the tutorial with a useful tip on
modifying column metadata. Above, updating/changing column metadata was
done with the ‘--colmetadata’ in the same command that produced the
newly created Table file. But in many situations, the table is already
made and you just want to update the metadata of one column. In such
cases using ‘--colmetadata’ is over-kill (wasting CPU/RAM energy or time
if the table is large) because it will load the full table data and
metadata into memory, just change the metadata and write it back into a
file.

In scenarios when the table's data does not need to be changed and
you just want to set or update the metadata, it is much more efficient
to use basic FITS keyword editing. For example, in the FITS standard,
column names are stored in the ‘TTYPE’ header keywords, so let's have a
look:

$ asttable two-in-one.fits -i
$ astfits two-in-one.fits -h1 | grep TTYPE

Changing/updating the column names is as easy as updating the values
to these keywords. You do not need to touch the actual data! With the
command below, we will just update the ‘MAGNITUDE’ and ‘MAGNITUDE-1’
columns (which are respectively stored in the ‘TTYPE5’ and ‘TTYPE11’
keywords) by modifying the keyword values and checking the effect by
listing the column metadata again:

$ astfits two-in-one.fits -h1 \
--update=TTYPE5,MAG-F160W \
--update=TTYPE11,MAG-F125W
$ asttable two-in-one.fits -i

You can see that the column names have indeed been changed without
touching any of the data. You can do the same for the column units or
comments by modifying the keywords starting with ‘TUNIT’ or ‘TCOMM’.

Generally, Gnuastro's table is a very useful program in data analysis
and what you have seen so far is just the tip of the iceberg. But to
avoid making the tutorial even longer, we will stop reviewing the
features here, for more, please see *note Table::. Before continuing,
let's just delete all the temporary FITS tables we placed in the top
project directory:

rm *.fits

---------- Footnotes ----------

(1) MakeCatalog can also output plain text tables. However, in the
plain text format you can only have one table per file. Therefore, if
you also request measurements on clumps, two plain text tables will be
created (suffixed with ‘_o.txt’ and ‘_c.txt’).

File: gnuastro.info, Node: Column statistics color-magnitude diagram, Next: Aperture photometry, Prev: Working with catalogs estimating colors, Up: General program usage tutorial

2.1.16 Column statistics (color-magnitude diagram)
--------------------------------------------------

In *note Working with catalogs estimating colors:: we created a single
catalog containing the magnitudes of our desired clumps in all three
filters, and their colors. To start with, let's inspect the
distribution of three colors with the Statistics program.

$ aststatistics cat/mags-with-color.fits -cF105W-F125W
$ aststatistics cat/mags-with-color.fits -cF105W-F160W
$ aststatistics cat/mags-with-color.fits -cF125W-F160W

This tiny and cute ASCII histogram (and the general information
printed above it) gives you a crude (but very useful and fast) feeling
on the distribution. You can later use Gnuastro's Statistics program
with the ‘--histogram’ option to build a much more fine-grained
histogram as a table to feed into your favorite plotting program for a
much more accurate/appealing plot (for example, with PGFPlots in LaTeX).
If you just want a specific measure, for example, the mean, median and
standard deviation, you can ask for them specifically, like below:

$ aststatistics cat/mags-with-color.fits -cF105W-F160W \
--mean --median --std

The basic statistics we measured above were just on one column. In
many scenarios this is fine, but things get much more exciting if you
look at the correlation of two columns with each other. For example,
let's create the color-magnitude diagram for our measured targets.

In many papers, the color-magnitude diagram is usually plotted as a
scatter plot. However, scatter plots have a major limitation when there
are a lot of points and they cluster together in one region of the plot:
the possible correlation in that dense region is lost (because the
points fall over each other). In such cases, it is much better to use a
2D histogram. In a 2D histogram, the full range in both columns is
divided into discrete 2D bins (or pixels!) and we count how many
objects fall in that 2D bin.

Since a 2D histogram is a pixelated space, we can simply save it as a
FITS image and view it in a FITS viewer. Let's do this in the command
below. As is common with color-magnitude plots, we will put the redder
magnitude on the horizontal axis and the color on the vertical axis. We
will set both dimensions to have 100 bins (with ‘--numbins’ for the
horizontal and ‘--numbins2’ for the vertical). Also, to avoid strong
outliers in any of the dimensions, we will manually set the range of
each dimension with the ‘--greaterequal’, ‘--greaterequal2’,
‘--lessthan’ and ‘--lessthan2’ options.

$ aststatistics cat/mags-with-color.fits -cMAG-F160W,F105W-F160W \
--histogram2d=image --manualbinrange \
--numbins=100 --greaterequal=22 --lessthan=30 \
--numbins2=100 --greaterequal2=-1 --lessthan2=3 \
--manualbinrange --output=cmd.fits

You can now open this FITS file as a normal FITS image, for example,
with the command below. Try hovering/zooming over the pixels: not only
will you see the number of objects in catalog that fall in each
bin/pixel, but you also see the ‘F160W’ magnitude and color of that
pixel also (in the same place you usually see RA and Dec when hovering
over an astronomical image).

$ astscript-fits-view cmd.fits --ds9scale=minmax

Having a 2D histogram as a FITS image with WCS has many great
advantages. For example, just like FITS images of the night sky, you
can "match" many 2D histograms that were created independently. You can
add two histograms with each other, or you can use advanced features of
FITS viewers to find structure in the correlation of your columns.

With the first command below, you can activate the grid feature of DS9
to actually see the coordinate grid, as well as values on each line.
With the second command, DS9 will even read the labels of the axes and
use them to generate an almost publication-ready plot.

$ astscript-fits-view cmd.fits --ds9scale=minmax --ds9extra="-grid yes"
$ astscript-fits-view cmd.fits --ds9scale=minmax \
--ds9extra="-grid yes -grid type publication"

If you are happy with the grid and coloring and the rest, you can
also use ds9 to save this as a JPEG image to directly use in your
documents/slides with these extra DS9 options (DS9 will write the image
to ‘cmd-2d.jpeg’ and quit immediately afterwards):

$ astscript-fits-view cmd.fits --ds9scale=minmax \
--ds9extra="-grid yes -grid type publication" \
--ds9extra="-saveimage cmd-2d.jpeg -quit"

This is good for a fast progress update. But for your paper or more
official report, you want to show something with higher quality. For
that, you can use the PGFPlots package in LaTeX to add axes in the same
font as your text, sharp grids and many other elegant/powerful features
(like over-plotting interesting points and lines). But to load the 2D
histogram into PGFPlots first you need to convert the FITS image into a
more standard format, for example, PDF. We will use Gnuastro's *note
ConvertType:: for this, and use the ‘sls-inverse’ color map (which will
map the pixels with a value of zero to white):

$ astconvertt cmd.fits --colormap=sls-inverse --borderwidth=0 -ocmd.pdf

Open the resulting ‘cmd.pdf’ and see the PDF. Below you can see a
minimally working example of how to add axis numbers, labels and a grid
to the PDF generated above. First, let's create a new ‘report’
directory to keep the LaTeX outputs, then put the minimal report's
source in a file called ‘report.tex’. Notice the ‘xmin’, ‘xmax’,
‘ymin’, ‘ymax’ values and how they are the same as the range specified
above.

$ mkdir report-cmd
$ mv cmd.pdf report-cmd/
$ cat report-cmd/report.tex
\documentclass{article}
\usepackage{pgfplots}
\dimendef\prevdepth=0
\begin{document}

You can write all you want here...

\begin{tikzpicture}
\begin{axis}[
enlargelimits=false,
grid,
axis on top,
width=\linewidth,
height=\linewidth,
xlabel={Magnitude (F160W)},
ylabel={Color (F105W-F160W)}]

\addplot graphics[xmin=22, xmax=30, ymin=-1, ymax=3] {cmd.pdf};
\end{axis}
\end{tikzpicture}
\end{document}

Run this command to build your PDF (assuming you have LaTeX and
PGFPlots).

$ cd report-cmd
$ pdflatex report.tex

Open the newly created ‘report.pdf’ and enjoy the exquisite quality.
The improved quality, blending in with the text, vector-graphics
resolution and other features make this plot pleasing to the eye, and
let your readers focus on the main point of your scientific argument.
PGFPlots can also built the PDF of the plot separately from the rest of
the paper/report, see *note 2D histogram as a table for plotting:: for
the necessary changes in the preamble.

We will not go much deeper into the Statistics program here, but
there is so much more you can do with it. After finishing the tutorial,
see *note Statistics::.

File: gnuastro.info, Node: Aperture photometry, Next: Matching catalogs, Prev: Column statistics color-magnitude diagram, Up: General program usage tutorial

2.1.17 Aperture photometry
--------------------------

The colors we calculated in *note Working with catalogs estimating
colors:: used a different segmentation map for each object. This might
not satisfy some science cases that need the flux within a fixed
area/aperture. Fortunately Gnuastro's modular programs make it very
easy do this type of measurement (photometry). To do this, we can
ignore the labeled images of NoiseChisel of Segment, we can just built
our own labeled image! That labeled image can then be given to
MakeCatalog

To generate the apertures catalog we will use Gnuastro's MakeProfiles
(see *note MakeProfiles::). But first we need a list of positions
(aperture photometry needs a-priori knowledge of your target positions).
So we will first read the clump positions from the F160W catalog, then
use AWK to set the other parameters of each profile to be a fixed circle
of radius 5 pixels (recall that we want all apertures to have an
identical size/area in this scenario).

$ rm *.fits *.txt
$ asttable cat/xdf-f160w.fits -hCLUMPS -cRA,DEC \
| awk '!/^#/{print NR, $1, $2, 5, 5, 0, 0, 1, NR, 1}' \
> apertures.txt
$ cat apertures.txt

We can now feed this catalog into MakeProfiles using the command
below to build the apertures over the image. The most important option
for this particular job is ‘--mforflatpix’, it tells MakeProfiles that
the values in the magnitude column should be used for each pixel of a
flat profile. Without it, MakeProfiles would build the profiles such
that the _sum_ of the pixels of each profile would have a _magnitude_
(in log-scale) of the value given in that column (what you would expect
when simulating a galaxy for example). See *note Invoking astmkprof::
for details on the options.

$ astmkprof apertures.txt --background=flat-ir/xdf-f160w.fits \
--clearcanvas --replace --type=int16 --mforflatpix \
--mode=wcs --output=apertures.fits

Open ‘apertures.fits’ with a FITS image viewer (like SAO DS9) and
look around at the circles placed over the targets. Also open the input
image and Segment's clumps image and compare them with the positions of
these circles. Where the apertures overlap, you will notice that one
label has replaced the other (because of the ‘--replace’ option). In
the future, MakeCatalog will be able to work with overlapping labels,
but currently it does not. If you are interested, please join us in
completing Gnuastro with added improvements like this (see task 14750
(1)).

We can now feed the ‘apertures.fits’ labeled image into MakeCatalog
instead of Segment's output as shown below. In comparison with the
previous MakeCatalog call, you will notice that there is no more
‘--clumpscat’ option, since there is no more separate "clump" image now,
each aperture is treated as a separate "object".

$ astmkcatalog apertures.fits -h1 --zeropoint=26.27 \
--valuesfile=nc/xdf-f105w.fits \
--ids --ra --dec --magnitude --sn \
--output=cat/xdf-f105w-aper.fits

This catalog has the same number of rows as the catalog produced from
clumps in *note Working with catalogs estimating colors::. Therefore
similar to how we found colors, you can compare the aperture and clump
magnitudes for example.

You can also change the filter name and zero point magnitudes and run
this command again to have the fixed aperture magnitude in the F160W
filter and measure colors on apertures.

---------- Footnotes ----------

(1) <https://savannah.gnu.org/task/index.php?14750>

File: gnuastro.info, Node: Matching catalogs, Next: Reddest clumps cutouts and parallelization, Prev: Aperture photometry, Up: General program usage tutorial

2.1.18 Matching catalogs
------------------------

In the example above, we had the luxury to generate the catalogs
ourselves, and where thus able to generate them in a way that the rows
match. But this is not generally the case. In many situations, you
need to use catalogs from many different telescopes, or catalogs with
high-level calculations that you cannot simply regenerate with the same
pixels without spending a lot of time or using heavy computation. In
such cases, when each catalog has the coordinates of its own objects,
you can use the coordinates to match the rows with Gnuastro's Match
program (see *note Match::).

As the name suggests, Gnuastro's Match program will match rows based
on distance (or aperture in 2D) in one, two, or three columns. For this
tutorial, let's try matching the two catalogs that were not created from
the same labeled images, recall how each has a different number of rows:

$ asttable cat/xdf-f105w.fits -hCLUMPS -i
$ asttable cat/xdf-f160w.fits -hCLUMPS -i

You give Match two catalogs (from the two different filters we
derived above) as argument, and the HDUs containing them (if they are
FITS files) with the ‘--hdu’ and ‘--hdu2’ options. The ‘--ccol1’ and
‘--ccol2’ options specify the coordinate-columns which should be matched
with which in the two catalogs. With ‘--aperture’ you specify the
acceptable error (radius in 2D), in the same units as the columns.

$ astmatch cat/xdf-f160w.fits cat/xdf-f105w.fits \
--hdu=CLUMPS --hdu2=CLUMPS \
--ccol1=RA,DEC --ccol2=RA,DEC \
--aperture=0.5/3600 \
--output=matched.fits
$ astfits matched.fits

From the second command, you see that the output has two extensions
and that both have the same number of rows. The rows in each extension
are the matched rows of the respective input table: those in the first
HDU come from the first input and those in the second HDU come from the
second. However, their order may be different from the input tables
because the rows match: the first row in the first HDU matches with the
first row in the second HDU, etc. You can also see which objects did
not match with the ‘--notmatched’, like below. Note how each extension
of now has a different number of rows.

$ astmatch cat/xdf-f160w.fits cat/xdf-f105w.fits \
--hdu=CLUMPS --hdu2=CLUMPS \
--ccol1=RA,DEC --ccol2=RA,DEC \
--aperture=0.5/3600 \
--output=not-matched.fits --notmatched
$ astfits not-matched.fits

The ‘--outcols’ of Match is a very convenient feature: you can use it
to specify which columns from the two catalogs you want in the output
(merge two input catalogs into one). If the first character is an
'<a>', the respective matched column (number or name, similar to Table
above) in the first catalog will be written in the output table. When
the first character is a '<b>', the respective column from the second
catalog will be written in the output. Also, if the first character is
followed by ‘_all’, then all the columns from the respective catalog
will be put in the output.

$ astmatch cat/xdf-f160w.fits cat/xdf-f105w.fits \
--hdu=CLUMPS --hdu2=CLUMPS \
--ccol1=RA,DEC --ccol2=RA,DEC \
--aperture=0.35/3600 \
--outcols=a_all,bMAGNITUDE,bSN \
--output=matched.fits
$ astfits matched.fits

File: gnuastro.info, Node: Reddest clumps cutouts and parallelization, Next: FITS images in a publication, Prev: Matching catalogs, Up: General program usage tutorial

2.1.19 Reddest clumps, cutouts and parallelization
--------------------------------------------------

As a final step, let's go back to the original clumps-based color
measurement we generated in *note Working with catalogs estimating
colors::. We will find the objects with the strongest color and make a
cutout to inspect them visually and finally, we will see how they are
located on the image. With the command below, we will select the
reddest objects (those with a color larger than 1.5):

$ asttable cat/mags-with-color.fits --range=F105W-F160W,1.5,inf

You can see how many they are by piping it to ‘wc -l’:

$ asttable cat/mags-with-color.fits --range=F105W-F160W,1.5,inf | wc -l

Let's crop the F160W image around each of these objects, but we first
need a unique identifier for them. We will define this identifier using
the object and clump labels (with an underscore between them) and feed
the output of the command above to AWK to generate a catalog. Note that
since we are making a plain text table, we will define the necessary
(for the string-type first column) metadata manually (see *note Gnuastro
text table format::).

$ echo "# Column 1: ID [name, str10] Object ID" > cat/reddest.txt
$ asttable cat/mags-with-color.fits --range=F105W-F160W,1.5,inf \
| awk '{printf("%d_%-10d %f %f\n", $1, $2, $3, $4)}' \
>> cat/reddest.txt

Let's see how these objects are positioned over the dataset. DS9 has
the "Region"s concept for this purpose. And you build such regions
easily from a table using Gnuastro's ‘astscript-ds9-region’ installed
script, using the command below:

$ astscript-ds9-region cat/reddest.txt -c2,3 --mode=wcs \
--command="ds9 flat-ir/xdf-f160w.fits -zscale"

We can now feed ‘cat/reddest.txt’ into Gnuastro's Crop program to get
separate postage stamps for each object. To keep things clean, we will
make a directory called ‘crop-red’ and ask Crop to save the crops in
this directory. We will also add a ‘-f160w.fits’ suffix to the crops
(to remind us which filter they came from). The width of the crops will
be 15 arc-seconds (or 15/3600 degrees, which is the units of the WCS).

$ mkdir crop-red
$ astcrop flat-ir/xdf-f160w.fits --mode=wcs --namecol=ID \
--catalog=cat/reddest.txt --width=15/3600,15/3600 \
--suffix=-f160w.fits --output=crop-red

Like the MakeProfiles command in *note Aperture photometry::, if you
look at the order of the crops, you will notice that the crops are not
made in order! This is because each crop is independent of the rest,
therefore crops are done in parallel, and parallel operations are
asynchronous. So the order can differ in each run, but the final output
is the same! In the command above, you can change ‘f160w’ to ‘f105w’ to
make the crops in both filters. You can see all the cropped FITS files
in the ‘crop-red’ directory with this command:

$ astscript-fits-view crop-red/*.fits

To view the crops more easily (not having to open ds9 for each
image), you can convert the FITS crops into the JPEG format with a shell
loop like below.

$ cd crop-red
$ for f in *.fits; do \
astconvertt $f --fluxlow=-0.001 --fluxhigh=0.005 --invert -ojpg; \
done
$ cd ..
$ ls crop-red/

You can now use your general graphic user interface image viewer to
flip through the images more easily, or import them into your
papers/reports.

The ‘for’ loop above to convert the images will do the job in series:
each file is converted only after the previous one is complete. But
like the crops, each JPEG image is independent, so let's parallelize it.
In other words, we want to run more than one instance of the command at
any moment. To do that, we will use Make
(https://en.wikipedia.org/wiki/Make_(software)). Make is a very
wonderful pipeline management system, and the most common and powerful
implementation is GNU Make (https://www.gnu.org/software/make), which
has a complete manual just like this one. We cannot go into the details
of Make here, for a hands-on video tutorial, see this video introduction
(https://peertube.stream/w/iJitjS3r232Z8UPMxKo6jq). To do the process
above in Make, please copy the contents below into a plain-text file
called ‘Makefile’. Just replace the ‘__[TAB]__’ part at the start of
the line with a single '<TAB>' button on your keyboard.

jpgs=$(subst .fits,.jpg,$(wildcard *.fits))
all: $(jpgs)
$(jpgs): %.jpg: %.fits
__[TAB]__astconvertt $< --fluxlow=-0.001 --fluxhigh=0.005 \
__[TAB]__ --invert -o$

Now that the ‘Makefile’ is ready, you can run Make on 12 threads
using the commands below. Feel free to replace the 12 with any number
of threads you have on your system (you can find out by running the
‘nproc’ command on GNU/Linux operating systems):

$ make -j12

Did you notice how much faster this one was? When possible, it is
always very helpful to do your analysis in parallel. You can build very
complex workflows with Make, for example, see Akhlaghi 2021
(https://arxiv.org/abs/2006.03018) so it is worth spending some time to
master.

File: gnuastro.info, Node: FITS images in a publication, Next: Marking objects for publication, Prev: Reddest clumps cutouts and parallelization, Up: General program usage tutorial

2.1.20 FITS images in a publication
-----------------------------------

In the previous section (*note Reddest clumps cutouts and
parallelization::), we visually inspected the positions of the reddest
objects using DS9. That is very good for an interactive inspection of
the objects: you can zoom-in and out, you can do measurements, etc.
Once the experimentation phase of your project is complete, you want to
show these objects over the whole image in a report, paper or slides.

One solution is to use DS9 itself! For example, run the
‘astscript-fits-view’ command of the previous section to open DS9 with
the regions over-plotted. Click on the "File" menu and select "Save
Image". In the side-menu that opens, you have multiple formats to
select from. Usually for publications, we want to show the regions and
text (in the colorbar) in vector graphics, so it is best to export to
EPS. Once you have made the EPS, you can then convert it to PDF with the
‘epspdf’ command.

Another solution is to use Gnuastro's ConvertType program. The main
difference is that DS9 is a Graphic User Interface (GUI) program, so it
takes relatively long (about a second) to load, and it requires many
dependencies. This will slow-down automatic conversion of many files,
and will make your code hard to move to another operating system. DS9
does have a command-line interface that you can use to automate the
creation of each file, however, it has a very peculiar command-line
interface and formats (like the "region" files). However, in
ConvertType, there is no graphic interface, so it has very few
dependencies, it is fast, and finally, it takes normal tables (in
plain-text or FITS) as input. So in this concluding step of the
analysis, let's build a nice publication-ready plot, showing the
positions of the reddest objects in the image for our paper.

In *note Reddest clumps cutouts and parallelization::, we already
used ConvertType to make JPEG postage stamps. Here, we will use it to
make a PDF image of the whole deep region. To start, let's simply run
ConvertType on the F160W image:

$ astconvertt flat-ir/xdf-f160w.fits -oxdf.pdf

Open the output in a PDF viewer. You see that it is almost fully
black! Let's see why this happens! First, with the two commands below,
let's calculate the maximum value, and the standard deviation of the sky
in this image (using NoiseChisel's output, which we found at the end of
*note NoiseChisel optimization for detection::). Note that NoiseChisel
writes the median sky standard deviation _before_ interpolation in the
‘MEDSTD’ keyword of the ‘SKY_STD’ HDU. This is more robust than the
median of the Sky standard deviation image (which has gone through
interpolation).

$ max=$(aststatistics nc/xdf-f160w.fits -hINPUT-NO-SKY --maximum)
$ skystd=$(astfits nc/xdf-f160w.fits -hSKY_STD --keyvalue=MEDSTD -q)

$ echo $max $skystd
58.8292 0.000410282

$ echo $max $skystd | awk '{print $1/$2}'
143387

In the last command above, we divided the maximum by the sky standard
deviation. You see that the maximum value is more than $140000$ times
larger than the noise level! On the other hand common monitors or
printers, usually have a maximum dynamic range of 8-bits, only allowing
for $2^8=256$ layers. This is therefore the maximum number of "layers"
you can have in a common display formats like JPEG, PDF or PNG! Dividing
the result above by 256, we get a layer spacing of

$ echo $max $skystd | awk '{print $1/$2/256}'
560.106

In other words, the first layer (which is black) will contain all the
pixel values below $\sim560$! So all pixels with a signal-to-noise
ratio lower than $\sim560$ will have a black color since they fall in
the first layer of an 8-bit PDF (or JPEG) image. This happens because
by default we are assuming a linear mapping from floating point to 8-bit
integers.

To fix this, we should move to a different mapping. A good,
physically motivated, mapping is Surface Brightness (which is in
log-scale, see *note Brightness flux magnitude::). Fortunately this is
very easy to do with Gnuastro's Arithmetic program, as shown in the
commands below (using the known zero point(1), and after calculating the
pixel area in units of arcsec$^2$):

$ zeropoint=25.94
$ pixarcsec2=$(astfits nc/xdf-f160w.fits --pixelareaarcsec2)
$ astarithmetic nc/xdf-f160w.fits $zeropoint $pixarcsec2 counts-to-sb \
--output=xdf-f160w-sb.fits

With the two commands below, first, let's look at the dynamic range of
the image now (dividing the maximum by the minimum), and then let's open
the image and have a look at it:

$ aststatistics xdf-f160w-sb.fits --minimum --maximum
$ astscript-fits-view xdf-f160w-sb.fits

The good news is that the dynamic range has now decreased to about 2!
In other words, we can distribute the 256 layers of an 8-bit display
over a much smaller range of values, and therefore better visualize the
data. However, there are two important points to consider from the
output of the first command and a visual inspection of the second.
• The largest pixel value (faintest surface brightness level) in the
image is $\sim43$! This is far too low to be realistic, and is
just due to noise. As discussed in *note Measuring the dataset
limits::, the $3\sigma$ surface brightness limit of this image,
over 100 arcsec$^2$ is roughly 32.66 mag/arcsec$^2$.
• You see many NaN pixels in between the galaxies! These are due to
the fact that the magnitude is defined on a logarithmic scale and
the logarithm of a negative number is not defined.

In other words, we should replace all NaN pixels, and pixels with a
surface brightness value fainter than the image surface brightness limit
to this limit. With the first command below, we will first extract the
surface brightness limit from the catalog headers that we calculated
before, and then call Arithmetic to use this limit.

$ sblimit=$(astfits cat/xdf-f160w.fits --keyvalue=SBLMAG -q)
$ astarithmetic nc/xdf-f160w.fits $zeropoint $pixarcsec2 \
counts-to-sb set-sb \
sb sb $sblimit gt sb isblank or $sblimit where \
--output=xdf-f160w-sb.fits

Let's convert this image into a PDF with the command below:

$ astconvertt xdf-f160w-sb.fits --output=xdf-f160w-sb.pdf

It is much better now and we can visualize many features of the FITS
file (from the central structures of the galaxies and stars, to a little
into the noise and their low surface brightness features. However, the
image generally looks a little too gray! This is because of that bright
star in the bottom half of the image! Stars are very sharp! So let's
manually tell ConvertType to set any pixel with a value less than
(brighter than) 20 to black (and not use the minimum). We do this with
the ‘--fluxlow’ option:

$ astconvertt xdf-f160w-sb.fits --output=xdf-f160w-sb.pdf --fluxlow=20

We are still missing some of the diffuse flux in this PDF. This is
because of those negative pixels that were set to NaN. To better show
these structures, we should warp the image to larger pixels. So let's
warp it to a pixel grid where the new pixels are $4\times4$ larger than
the original pixels. But be careful that warping should be done on the
original image, not on the surface brightness image. We should
re-calculate the surface brightness image after the warping is one.
This is because $log(a+b)\ne log(a)+log(b)$. Recall that surface
brightness calculation involves a logarithm, and warping involves
addition of pixel values.

$ astwarp nc/xdf-f160w.fits --scale=1/4 --centeroncorner \
--output=xdf-f160w-warped.fits

$ pixarcsec2=$(astfits xdf-f160w-warped.fits --pixelareaarcsec2)

$ astarithmetic xdf-f160w-warped.fits $zeropoint $pixarcsec2 \
counts-to-sb set-sb \
sb sb $sblimit gt sb isblank or $sblimit where \
--output=xdf-f160w-sb.fits

$ astconvertt xdf-f160w-sb.fits --output=xdf-f160w-sb.pdf --fluxlow=20

Above, we needed to re-calculate the pixel area of the warpped image,
but we did not need to re-calculate the surface brightness limit! The
reason is that the surface brightness limit is independent of the pixel
area (in its derivation, the pixel area has been accounted for). As a
side-effect of the warping, the number of pixels in the image also
dramatically decreased, therefore the volume of the output PDF (in
bytes) is also smaller, making your paper/report easier to
upload/download or send by email. This visual resolution is still more
than enough for including on top of a column in your paper!

*I do not have the zero point of my image:* The absolute value of the
zero point is irrelevant for the finally produced PDF. We used it here
because it was available and makes the numbers physically
understandable. If you do not have the zero point, just set it to zero
(which is also the default zero point used by MakeCatalog when it
estimates the surface brightness limit). For the value to ‘--fluxlow’
above, you can simply subtract $\sim10$ from the surface brightness
limit.

To summarize, and to keep the image for the next section in a separate
directory, here are the necessary commands:

$ zeropoint=25.94
$ mkdir report-image
$ cd report-image
$ sblimit=$(astfits cat/xdf-f160w.fits --keyvalue=SBLMAG -q)
$ astwarp nc/xdf-f160w.fits --scale=1/4 --centeroncorner \
--output=warped.fits
$ pixarcsec2=$(astfits warped.fits --pixelareaarcsec2)
$ astarithmetic warped.fits $zeropoint $pixarcsec2 \
counts-to-sb set-sb \
sb sb $sblimit gt sb isblank or $sblimit where \
--output=sb.fits
$ astconvertt sb.fits --output=sb.pdf --fluxlow=20

Finally, let's remove all the temporary files we built in the top-level
tutorial directory:

$ rm *.fits *.pdf

*Color images:* In this tutorial we just used one of the filters and
showed the surface brightness image of that single filter as a grayscale
image. But the image can also be in color (using three filters) to
better convey the physical properties of the objects in your image. To
create an image that shows the full dynamic range of your data, see this
dedicated tutorial *note Color images with full dynamic range::.

---------- Footnotes ----------

(1) <https://archive.stsci.edu/prepds/xdf/#science-images>

File: gnuastro.info, Node: Marking objects for publication, Next: Writing scripts to automate the steps, Prev: FITS images in a publication, Up: General program usage tutorial

2.1.21 Marking objects for publication
--------------------------------------

In *note FITS images in a publication:: we created a ready-to-print
visualization of the FITS image used in this tutorial. However, you
rarely want to show a naked image like that! You usually want to
highlight some objects (that are the target of your science) over the
image and show different marks for the various types of objects you are
studying. In this tutorial, we will do just that: select a sub-set of
the full catalog of clumps, and show them with different marks shapes
and colors, while also adding some text under each mark. To add
coordinates on the edges of the figure in your paper, see *note
Annotations for figure in paper::.

To start with, let's put a red plus sign over the sub-sample of
reddest clumps similar to *note Reddest clumps cutouts and
parallelization::. First, we will need to make the table of marks. We
will choose those with a color stronger than 1.5 magnitudes and a
signal-to-noise ratio (in F160W) larger than 5. We also only need the
RA, Dec, color and magnitude (in F160W) columns (recall that at the end
of the previous section we were already in the ‘report-image/’
directory):

$ asttable cat/mags-with-color.fits --range=F105W-F160W,1.5:inf \
--range=sn-f160w,5:inf -cRA,DEC,MAG-F160w,F105W-F160W \
--output=reddest-cat.fits

Gnuastro's ConvertType program also has features to add marks over
the finally produced PDF. Below, we will start with the same
‘astconvertt’ command of the previous section. The positions of the
marks should be given as a table to the ‘--marks’ option. Two other
options are also mandatory: ‘--markcoords’ identifies the columns that
contain the coordinates of each mark and ‘--mode’ specifies if the
coordinates are in image or WCS coordinates.

$ astconvertt sb.fits --output=reddest.pdf --fluxlow=20 \
--marks=reddest-cat.fits --mode=wcs \
--markcoords=RA,DEC

Open the output ‘reddest.pdf’ and see the result. You will see
relatively thick red circles placed over the given coordinates. In your
PDF browser, zoom-in to one of the regions, you will see that while the
pixels of the background image become larger, the lines of these regions
do not degrade! This is the concept/power of Vector Graphics: ideal for
publication! For more on raster (pixelated) and vector
(infinite-resolution) graphics, see *note Raster and Vector graphics::.

We had planned to put a plus-sign on each object. However, because
we did not explicitly ask for a certain shape, ConvertType put a circle.
Each mark can have its own separate shape. Shapes can be given by a
name or a code. The full list of available shapes names and codes is
given in the description of ‘--markshape’ option of *note Drawing with
vector graphics::.

To use a different shape, we need to add a new column to the base
table, containing the identifier of the desired shape for each mark.
For example, the code for the plus sign is ‘2’. With the commands
below, we will add a new column with this fixed value. With the first
AWK command we will make a single-column file, where all the rows have
the same value. We pipe our base table into AWK, so it has the same
number of rows. With the second command, we concatenate (or append) the
new column with Table, and give this new column the name ‘SHAPE’ (to
easily refer to it later and not have to count). With the third
command, we clean-up behind our selves (deleting the extra ‘params.txt’
file). Finally, we use the ‘--markshape’ option to tell ConvertType
which column to use for the shape identifier.

$ asttable reddest-cat.fits | awk '{print 2}' > params.txt

$ asttable reddest-cat.fits --catcolumnfile=params.txt \
--colmetadata=5,SHAPE,id,"Shape of mark" \
--output=reddest-marks.fits
$ rm params.txt

$ astconvertt sb.fits --output=reddest.pdf --fluxlow=20 \
--marks=reddest-marks.fits --mode=wcs \
--markcoords=RA,DEC --markshape=SHAPE

Open the PDF and have a look! You do see red signs over the
coordinates, but the thick plus-signs only become visible after you
zoom-in multiple times! To make them larger, you can give another
column to specify the size of each mark. Let's set the full width of
the plus sign to extend 3 arcseconds. The commands are similar to
above, try to follow the difference (in particular, how we use
‘--sizeinarcsec’).

$ asttable reddest-cat.fits | awk '{print 2, 3}' > params.txt

$ asttable reddest-cat.fits --catcolumnfile=params.txt \
--colmetadata=5,SHAPE,id,"Shape of mark" \
--colmetadata=6,SIZE,arcsec,"Size in arcseconds" \
--output=reddest-marks.fits
$ rm params.txt

$ astconvertt sb.fits --output=reddest.pdf --fluxlow=20 \
--marks=reddest-marks.fits --mode=wcs \
--markcoords=RA,DEC --markshape=SHAPE \
--marksize=SIZE --sizeinarcsec

The power of this methodology is that each mark can be completely
different! For example, let's show the objects with a color less than 2
magnitudes with a circle, and those with a stronger color with a plus
(recall that the code for a circle was ‘1’ and that of a plus was ‘2’).
You only need to replace the first command above with the one below.
Afterwards, run the rest of the commands in the last code-block.

$ asttable reddest-cat.fits -cF105W-F160W \
| awk '{if($1<2) shape=1; else shape=2; print shape, 3}' \
> params.txt

Have a look at the resulting ‘reddest.pdf’. You see that the circles
are much larger than the plus signs. This is because the "size" of a
cross is defined to be its full width, but for a circle, the value in
the size column is the radius. The way each shape interprets the value
of the size column is fully described under ‘--markshape’ of *note
Drawing with vector graphics::. To make them more comparable, let's set
the circle sizes to be half of the cross sizes.

$ asttable reddest-cat.fits -cF105W-F160W \
| awk '{if($1<2) {shape=1; size=1.5} \
else {shape=2; size=3} \
print shape, size}' \
> params.txt

Let's make things a little more complex (and show more information in
the visualization) by using color. Gnuastro recognizes the full
extended web colors
(https://en.wikipedia.org/wiki/Web_colors#Extended_colors), for their
full list (containing names and codes) see *note Vector graphics
colors::. But like everything else, an even easier way to view and
select the color for your figure is on the command-line! If your
terminal supports 24-bit true-color, you can see all the colors by
running this command (supported on modern GNU/Linux distributions):

$ astconvertt --listcolors

we will give a "Sienna" color for the objects that are fainter than
29th magnitude and a "deeppink" color to the brighter ones (while
keeping the same shapes definition as before) Since there are many
colors, using their codes can make the table hard to read by a human!
So let's use the color names instead of the color codes in the example
below (this is useful in other columns require strings-only, like the
font name).

The only intricacy is in the making of ‘params.txt’. Recall that
string columns need column metadata (*note Gnuastro text table
format::). In this particular case, since the string column is the last
one, we can safely use AWK's ‘print’ command. But if you have multiple
string columns, to be safe it is better to use AWK's ‘printf’ and
explicitly specify the number of characters in the string columns.

$ asttable reddest-cat.fits -cF105W-F160W,MAG-F160W \
| awk 'BEGIN{print "# Column 3: COLOR [name, str8]"}\
{if($1<2) {shape=1; size=1.5} \
else {shape=2; size=3} \
if($2>29) {color="sienna"} \
else {color="deeppink"} \
print shape, size, color}' \
> params.txt

$ asttable reddest-cat.fits --catcolumnfile=params.txt \
--colmetadata=5,SHAPE,id,"Shape of mark" \
--colmetadata=6,SIZE,arcsec,"Size in arcseconds" \
--output=reddest-marks.fits
$ rm params.txt

$ astconvertt sb.fits --output=reddest.pdf --fluxlow=20 \
--marks=reddest-marks.fits --mode=wcs \
--markcoords=RA,DEC --markshape=SHAPE \
--marksize=SIZE --sizeinarcsec --markcolor=COLOR

As one final example, let's write the magnitude of each object under
it. Since the magnitude is already in the ‘marks.fits’ that we produced
above, it is very easy to add it (just add ‘--marktext’ option to
ConvertType):

$ astconvertt sb.fits --output=reddest.pdf --fluxlow=20 \
--marks=reddest-marks.fits --mode=wcs \
--markcoords=RA,DEC --markshape=SHAPE \
--marksize=SIZE --sizeinarcsec \
--markcolor=COLOR --marktext=MAG-F160W

Open the final PDF (‘reddest.pdf’) and you will see the magnitudes
written under each mark in the same color. In the case of magnitudes
(where the magnitude error is usually much larger than 0.01 magnitudes,
four decimals is not too meaningful. By default, for printing floating
point columns, we use the compiler's default precision (which is about 4
digits for 32-bit floating point numbers). But you can over-write this
(to only show two digits after the decimal point) with the
‘--marktextprecision=2’ option.

You can customize the written text by specifying a different
line-width (for the text, different from the main mark), or even
specifying a different font for each mark! You can see the full list of
available fonts for the text under a mark with the first command below
and with the second, you can actually see them in a custom PDF (to only
show the fonts).

$ astconvertt --listfonts
$ astconvertt --showfonts

As you see, there are many ways you can customize each mark! The
above examples were just the tip of the iceburg! But this section has
already become long so we will stop it here (see the box at the end of
this section for yet another useful example). Like above, each feature
of a mark can be controlled with a column in the table of mark
information. Please see in *note Drawing with vector graphics:: for the
full list of columns/features that you can use.

*Drawing ellipses:* With the commands below, you can measure the
elliptical properties of the objects and visualized them in a
ready-to-publish PDF (we will only show the ellipses of the largest
clumps):
$ astmkcatalog ../seg/xdf-f160w.fits --ra --dec --semi-major \
--axis-ratio --position-angle --clumpscat \
--output=ellipseinfo.fits
$ asttable ellipseinfo.fits -hCLUMPS | awk '{print 4}' > params.txt
$ asttable ellipseinfo.fits -hCLUMPS --catcolumnfile=params.txt \
--range=SEMI_MAJOR,10,inf -oellipse-marks.fits \
--colmetadata=6,SHAPE,id,"Shape of mark"
$ astconvertt sb.fits --output=ellipse.pdf --fluxlow=20 \
--marks=ellipse-marks.fits --mode=wcs \
--markcoords=RA,DEC --markshape=SHAPE \
--marksize=SEMI_MAJOR,AXIS_RATIO --sizeinpix \
--markrotate=POSITION_ANGLE

To conclude this section, let us highlight an important factor to
consider in vector graphics. In ConvertType, things like line width or
font size are defined in units of _points_. In vector graphics
standards, 72 points correspond to one inch. Therefore, one way you can
change these factors for all the objects is to assign a larger or
smaller print size to the image. The print size is just a meta-data
entry, and will not affect the file's volume in bytes! You can do this
with the ‘--widthincm’ option. Try adding this option and giving it
very different values like ‘5’ or ‘30’.

File: gnuastro.info, Node: Writing scripts to automate the steps, Next: Citing and acknowledging Gnuastro, Prev: Marking objects for publication, Up: General program usage tutorial

2.1.22 Writing scripts to automate the steps
--------------------------------------------

In the previous sub-sections, we went through a series of steps like
downloading the necessary datasets (in *note Setup and data download::),
detecting the objects in the image, and finally selecting a particular
subset of them to inspect visually (in *note Reddest clumps cutouts and
parallelization::). To benefit most effectively from this subsection,
please go through the previous sub-sections, and if you have not
actually done them, we recommended to do/run them before continuing
here.

Each sub-section/step of the sub-sections above involved several
commands on the command-line. Therefore, if you want to reproduce the
previous results (for example, to only change one part, and see its
effect), you'll have to go through all the sections above and read
through them again. If you have ran the commands recently, you may also
have them in the history of your shell (command-line environment). You
can see many of your previous commands on the shell (even if you have
closed the terminal) with the ‘history’ command, like this:

$ history

Try it in your teminal to see for yourself. By default in GNU Bash,
it shows the last 500 commands. You can also save this "history" of
previous commands to a file using shell redirection (to have it after
your next 500 commands), with this command

$ history > my-previous-commands.txt

This is a good way to temporarily keep track of every single command
you ran. But in the middle of all the useful commands, you will have
many extra commands, like tests that you did before/after the good
output of a step (that you decided to continue working on), or an
unrelated job you had to do in the middle of this project. Because of
these impurities, after a few days (that you have forgot the context:
tests you did not end-up using, or unrelated jobs) reading this full
history will be very frustrating.

Keeping the final commands that were used in each step of an analysis
is a common problem for anyone who is doing something serious with the
computer. But simply keeping the most important commands in a text file
is not enough, the small steps in the middle (like making a directory to
keep the outputs of one step) are also important. In other words, the
only way you can be sure that you are under control of your processing
(and actually understand how you produced your final result) is to run
the commands automatically.

Fortunately, typing commands interactively with your fingers is not
the only way to operate the shell. The shell can also take its
orders/commands from a plain-text file, which is called a _script_.
When given a script, the shell will read it line-by-line as if you have
actually typed it manually.

Let's continue with an example: try typing the commands below in your
shell. With these commands we are making a text file (‘a.txt’)
containing a simple $3\times3$ matrix, converting it to a FITS image and
computing its basic statistics. After the first three commands open
‘a.txt’ with a text editor to actually see the values we wrote in it,
and after the fourth, open the FITS file to see the matrix as an image.
‘a.txt’ is created through the shell's redirection feature: '‘>’'
overwrites the existing contents of a file, and '‘>>’' appends the new
contents after the old contents.

$ echo "1 1 1" > a.txt
$ echo "1 2 1" >> a.txt
$ echo "1 1 1" >> a.txt
$ astconvertt a.txt --output=a.fits
$ aststatistics a.fits

To automate these series of commands, you should put them in a text
file. But that text file must have two special features: 1) It should
tell the shell what program should interpret the script. 2) The
operating system should know that the file can be directly executed.

For the first, Unix-like operating systems define the _shebang_
concept (also known as _sha-bang_ or _hashbang_). In the shebang
convention, the first two characters of a file should be '‘#!’'. When
confronted with these characters, the script will be interpreted with
the program that follows them. In this case, we want to write a shell
script and the most common shell program is GNU Bash which is installed
in ‘/bin/bash’. So the first line of your script should be
'‘#!/bin/bash’'(1).

It may happen (rarely) that GNU Bash is in another location on your
system. In other cases, you may prefer to use a non-standard version of
Bash installed in another location (that has higher priority in your
‘PATH’, see *note Installation directory::). In such cases, you can use
the '‘#!/usr/bin/env bash’' shebang instead. Through the ‘env’ program,
this shebang will look in your ‘PATH’ and use the first ‘bash’ it finds
to run your script. But for simplicity in the rest of the tutorial, we
will continue with the '‘#!/bin/bash’' shebang.

Using your favorite text editor, make a new empty file, let's call it
‘my-first-script.sh’. Write the GNU Bash shebang (above) as its first
line. After the shebang, copy the series of commands we ran above.
Just note that the '‘$’' sign at the start of every line above is the
prompt of the interactive shell (you never actually typed it,
remember?). Therefore, commands in a shell script should not start with
a '‘$’'. Once you add the commands, close the text editor and run the
‘cat’ command to confirm its contents. It should look like the example
below. Recall that you should only type the line that starts with a
'‘$’', the lines without a '‘$’', are printed automatically on the
command-line (they are the contents of your script).

$ cat my-first-script.sh
#!/bin/bash
echo "1 1 1" > a.txt
echo "1 2 1" >> a.txt
echo "1 1 1" >> a.txt
astconvertt a.txt --output=a.fits
aststatistics a.fits

The script contents are now ready, but to run it, you should activate
the script file's _executable flag_. In Unix-like operating systems,
every file has three types of flags: _read_ (or ‘r’), _write_ (or ‘w’)
and _execute_ (or ‘x’). To toggle a file's flags, you should use the
‘chmod’ (for "change mode") command. To activate a flag, you put a
'‘+’' before the flag character (for example, ‘+x’). To deactivate it,
you put a '‘-’' (for example, ‘-x’). In this case, you want to activate
the script's executable flag, so you should run

$ chmod +x my-first-script.sh

Your script is now ready to run/execute the series of commands. To
run it, you should call it while specifying its location in the file
system. Since you are currently in the same directory as the script, it
is easiest to use relative addressing like below (where '‘./’' means the
current directory). But before running your script, first delete the
two ‘a.txt’ and ‘a.fits’ files that were created when you interactively
ran the commands.

$ rm a.txt a.fits
$ ls
$ ./my-first-script.sh
$ ls

The script immediately prints the statistics while doing all the
previous steps in the background. With the last ‘ls’, you see that it
automatically re-built the ‘a.txt’ and ‘a.fits’ files, open them and
have a look at their contents.

An extremely useful feature of shell scripts is that the shell will
ignore anything after a '‘#’' character. You can thus add
descriptions/comments to the commands and make them much more useful for
the future. For example, after adding comments, your script might look
like this:

$ cat my-first-script.sh
#!/bin/bash

# This script is my first attempt at learning to write shell scripts.
# As a simple series of commands, I am just building a small FITS
# image, and calculating its basic statistics.

# Write the matrix into a file.
echo "1 1 1" > a.txt
echo "1 2 1" >> a.txt
echo "1 1 1" >> a.txt

# Convert the matrix to a FITS image.
astconvertt a.txt --output=a.fits

# Calculate the statistics of the FITS image.
aststatistics a.fits

Is Not this much more easier to read now? Comments help to provide
human-friendly context to the raw commands. At the time you make a
script, comments may seem like an extra effort and slow you down. But
in one year, you will forget almost everything about your script and you
will appreciate the effort so much! Think of the comments as an email
to your future-self and always put a well-written description of the
context/purpose (most importantly, things that are not directly clear by
reading the commands) in your scripts.

The example above was very basic and mostly redundant series of
commands, to show the basic concepts behind scripts. You can put any
(arbitrarily long and complex) series of commands in a script by
following the two rules: 1) add a shebang, and 2) enable the executable
flag. In fact, as you continue your own research projects, you will
find that any time you are dealing with more than two or three commands,
keeping them in a script (and modifying that script, and running it) is
much more easier, and future-proof, then typing the commands directly on
the command-line and relying on things like ‘history’. Here are some
tips that will come in handy when you are writing your scripts:

As a more realistic example, let's have a look at a script that will
do the steps of *note Setup and data download:: and *note Dataset
inspection and cropping::. In particular note how often we are using
variables to avoid repeating fixed strings of characters (usually
file/directory names). This greatly helps in scaling up your project,
and avoiding hard-to-find bugs that are caused by typos in those fixed
strings.

$ cat gnuastro-tutorial-1.sh
#!/bin/bash

# Download the input datasets
# ---------------------------
#
# The default file names have this format (where `FILTER' differs for
# each filter):
# hlsp_xdf_hst_wfc3ir-60mas_hudf_FILTER_v1_sci.fits
# To make the script easier to read, a prefix and suffix variable are
# used to sandwich the filter name into one short line.
dldir=download
xdfsuffix=_v1_sci.fits
xdfprefix=hlsp_xdf_hst_wfc3ir-60mas_hudf_
xdfurl=http://archive.stsci.edu/pub/hlsp/xdf

# The file name and full URLs of the input data.
f105w_in=$xdfprefix"f105w"$xdfsuffix
f160w_in=$xdfprefix"f160w"$xdfsuffix
f105w_url=$xdfurl/$f105w_in
f160w_url=$xdfurl/$f160w_in

# Go into the download directory and download the images there,
# then come back up to the top running directory.
mkdir $dldir
cd $dldir
wget $f105w_url
wget $f160w_url
cd ..

# Only work on the deep region
# ----------------------------
#
# To help in readability, each vertice of the deep/flat field is stored
# as a separate variable. They are then merged into one variable to
# define the polygon.
flatdir=flat-ir
vertice1="53.187414,-27.779152"
vertice2="53.159507,-27.759633"
vertice3="53.134517,-27.787144"
vertice4="53.161906,-27.807208"
f105w_flat=$flatdir/xdf-f105w.fits
f160w_flat=$flatdir/xdf-f160w.fits
deep_polygon="$vertice1:$vertice2:$vertice3:$vertice4"

mkdir $flatdir
astcrop --mode=wcs -h0 --output=$f105w_flat \
--polygon=$deep_polygon $dldir/$f105w_in
astcrop --mode=wcs -h0 --output=$f160w_flat \
--polygon=$deep_polygon $dldirdir/$f160w_in

The first thing you may notice is that even if you already have the
downloaded input images, this script will always try to re-download
them. Also, if you re-run the script, you will notice that ‘mkdir’
prints an error message that the download directory already exists.
Therefore, the script above is not too useful and some modifications are
necessary to make it more generally useful. Here are some general tips
that are often very useful when writing scripts:

*Stop script if a command crashes*
By default, if a command in a script crashes (aborts and fails to
do what it was meant to do), the script will continue onto the next
command. In GNU Bash, you can tell the shell to stop a script in
the case of a crash by adding this line at the start of your
script:

set -e

*Check if a file/directory exists to avoid re-creating it*
Conditionals are a very useful feature in scripts. One common
conditional is to check if a file exists or not. Assuming the
file's name is ‘FILENAME’, you can check its existance (to avoid
re-doing the commands that build it) like this:
if [ -f FILENAME ]; then
echo "FILENAME exists"
else
# Some commands to generate the file
echo "done" > FILENAME
fi
To check the existance of a directory instead of a file, use ‘-d’
instead of ‘-f’. To negate a conditional, use '‘!’' and note that
conditionals can be written in one line also (useful for when it is
short).

One common scenario that you'll need to check the existance of
directories is when you are making them: the default ‘mkdir’
command will crash if the desired directory already exists. On
some systems (including GNU/Linux distributions), ‘mkdir’ has
options to deal with such cases. But if you want your script to be
portable, it is best to check yourself like below:

if ! [ -d DIRNAME ]; then mkdir DIRNAME; fi

*Avoid changing directories (with '‘cd’') within the script*
You can directly read and write files within other directories.
Therefore using ‘cd’ to enter a directory (like what we did above,
around the ‘wget’ commands), running command there and coming out
is extra, and not good practice. This is because the running
directory is part of the environment of a command. You can simply
give the directory name before the input and output file names to
use them from anywhere on the file system. See the same ‘wget’
commands below for an example.

*Copyright notice:* A very important thing to put _at the top_ of your
script is a one-line description of what it does and its copyright
information (see the example below). Here, we specify who is the
author(s) of this script, in which years, and under what license others
are allowed to use this file. Without it, your script does not
credibility or identity, and others cannot trust, use or acknowledge
your work on it. Since Gnuastro is itself licensed under a copyleft
(https://en.wikipedia.org/wiki/Copyleft) license (see *note Your
rights:: and *note GNU General Public License:: or GNU GPL, the license
finishes with a template on how to add it), any script that uses
Gnuastro should also have a copyleft license: we recommend the same GNU
GPL v3+ like below.

Taking the above points into consideration, we can write a better
version of the script above. Please compare this script with the
previous one carefully to spot the differences. These are very
important points that you will definitely encouter during your own
research, and knowing them can greatly help your productiveity, so pay
close attention (even in the comments).

#!/bin/bash
# Script to download and keep the deep region of the XDF survey.
#
# Copyright (C) 2024 Your Name <yourname@email.company>
# Copyright (C) 2021-2024 Initial Author <incase@there-is.any>
#
# This script is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This script is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Gnuastro. If not, see <http://www.gnu.org/licenses/>.

# Abort the script in case of an error.
set -e

# The file name and full URLs of the input data.
f105w_in=$xdfprefix"f105w"$xdfsuffix
f160w_in=$xdfprefix"f160w"$xdfsuffix
f105w_url=$xdfurl/$f105w_in
f160w_url=$xdfurl/$f160w_in

# Make sure the download directory exists, and download the images.
if ! [ -d $dldir ]; then mkdir $dldir; fi
if ! [ -f $f105w_in ]; then wget $f105w_url -O $dldir/$f105w_in; fi
if ! [ -f $f160w_in ]; then wget $f160w_url -O $dldir/$f160w_in; fi

# Crop out the deep region
# ------------------------
#
# To help in readability, each vertice of the deep/flat field is stored
# as a separate variable. They are then merged into one variable to
# define the polygon.
flatdir=flat-ir
vertice1="53.187414,-27.779152"
vertice2="53.159507,-27.759633"
vertice3="53.134517,-27.787144"
vertice4="53.161906,-27.807208"
f105w_flat=$flatdir/xdf-f105w.fits
f160w_flat=$flatdir/xdf-f160w.fits
deep_polygon="$vertice1:$vertice2:$vertice3:$vertice4"

if ! [ -d $flatdir ]; then mkdir $flatdir; fi
if ! [ -f $f105w_flat ]; then
astcrop --mode=wcs -h0 --output=$f105w_flat \
--polygon=$deep_polygon $dldir/$f105w_in
fi
if ! [ -f $f160w_flat ]; then
astcrop --mode=wcs -h0 --output=$f160w_flat \
--polygon=$deep_polygon $dldir/$f160w_in
fi

---------- Footnotes ----------

(1) When the script is to be run by the same shell that is calling it
(like this script), the shebang is optional. But it is still
recommended, because it ensures that even if the user is not using GNU
Bash, the script will be run in GNU Bash: given the differences between
various shells, writing truly portable shell scripts, that can be run by
many shell programs/implementations, is not easy (sometimes not
possible!).