gnuastro (0.22)
This is gnuastro.info, produced by makeinfo version 7.1 from
gnuastro.texi.
This book documents version 0.22 of the GNU Astronomy Utilities
(Gnuastro). Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.
Copyright © 2015-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License,
Version 1.3 or any later version published by the Free Software
Foundation; with no Invariant Sections, no Front-Cover Texts, and
no Back-Cover Texts. A copy of the license is included in the
section entitled "GNU Free Documentation License".
INFO-DIR-SECTION Astronomy
START-INFO-DIR-ENTRY
* Gnuastro: (gnuastro). GNU Astronomy Utilities.
* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc.
* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help.
* bug-gnuastro: (gnuastro)Report a bug. How to report bugs
* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels.
* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic.
* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro's library.
* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram.
* ConvertType: (gnuastro)ConvertType. Convert different file types.
* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType.
* Convolve: (gnuastro)Convolve. Convolve an input file with kernel.
* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve.
* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params.
* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator.
* Crop: (gnuastro)Crop. Crop region(s) from image(s).
* astcrop: (gnuastro)Invoking astcrop. Options to Crop.
* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords.
* astfits: (gnuastro)Invoking astfits. Options to Fits.
* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image.
* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog.
* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles.
* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles.
* Match: (gnuastro)Match. Match two separate catalogs.
* astmatch: (gnuastro)Invoking astmatch. Options to Match.
* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise.
* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel.
* Segment: (gnuastro)Segment. Segment detections based on signal structure.
* astsegment: (gnuastro)Invoking astsegment. Options to Segment.
* Query: (gnuastro)Query. Access remote databases for downloading data.
* astquery: (gnuastro)Invoking astquery. Options to Query.
* Statistics: (gnuastro)Statistics. Get image Statistics.
* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics.
* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables.
* asttable: (gnuastro)Invoking asttable. Options to Table.
* Warp: (gnuastro)Warp. Warp a dataset to a new grid.
* astwarp: (gnuastro)Invoking astwarp. Options to Warp.
* astscript: (gnuastro)Installed scripts. Gnuastro's installed scripts.
* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script
* astscript-fits-view: (gnuastro)Invoking astscript-fits-view. Options to this script
* astscript-pointing-simulate: (gnuastro)Invoking astscript-pointing-simulate. Options to this script
* astscript-psf-scale-factor: (gnuastro)Invoking astscript-psf-scale-factor. Options to this script
* astscript-psf-select-stars: (gnuastro)Invoking astscript-psf-select-stars. Options to this script
* astscript-psf-stamp: (gnuastro)Invoking astscript-psf-stamp. Options to this script
* astscript-psf-subtract: (gnuastro)Invoking astscript-psf-subtract. Options to this script
* astscript-psf-unite: (gnuastro)Invoking astscript-psf-unite. Options to this script
* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script
* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script
* astscript-zeropoint: (gnuastro)Invoking astscript-zeropoint. Options to this script
END-INFO-DIR-ENTRY
File: gnuastro.info, Node: Pointing pattern design, Next: Moire pattern in stacking and its correction, Prev: Zero point of an image, Up: Tutorials
2.8 Pointing pattern design
===========================
A dataset that is ready for scientific analysis is usually composed of
many separate exposures and how they are taken is usually known as
"observing strategy". This tutorial describes Gnuastro's tools to
simplify the process of deciding the pointing pattern of your observing
strategy.
A "pointing" is the location on the sky that each exposure is aimed
at. Each exposure's pointing is usually moved (on the sky) compared to
the previous exposure. This is done for reasons like improving
calibration, increasing resolution, expending the area of the
observation and etc. Therefore, deciding a suitable pointing pattern is
one of the most important steps when planning your observation strategy.
There are commonly two types of pointings: "dither" and "offset".
These are sometimes used interchangeably with "pointing" (especially
when the final stack is roughly the same area as the field of view.
Alternatively, "dither" and "offset" are used to distinguish pointings
with large or small (on the scale of the field of view) movement
compared to a previous one. When a pointing has a large distance to the
previous pointing, it is known as an "offset", while pointings with a
small displacement are known as a "dither". This distinction originates
from the mechanics and optics of most modern telescopes: the overhead
(for example the need to re-focus the camera) to make small movements is
usually less than large movements.
In this tutorial, let's simulate a hypothetical pointing pattern
using Gnuastro's ‘astscript-pointing-simulate’ installed script (see
*note Pointing pattern simulation::). Since we will be testing very
different displacements between pointings, we'll ignore the difference
between offset and dither here, and only use the term pointing.
Let's assume you want to observe M94
(https://en.wikipedia.org/wiki/Messier_94) in the H-alpha and rSDSS
filters (to study the extended star formation in the outer rings of this
beautiful galaxy!). Including the outer parts of the rings, the galaxy
is half a degree in diameter! This is very large, and you want to
design a pointing pattern that will allow you to cover as much area,
while not loosing your ability to calibrate properly.
*Do not start with this tutorial:* If you are new to Gnuastro and have
not already completed *note General program usage tutorial::, we
recommend going through that tutorial before starting this one. Basic
features like access to this book on the command-line, the configuration
files of Gnuastro's programs, benefiting from the modular nature of the
programs, viewing multi-extension FITS files, and many others are
discussed in more detail there.
* Menu:
* Preparing input and generating exposure map:: Download and image and build exposure map.
* Area of non-blank pixels on sky:: Account for the curved area on the sky.
* Script with pointing simulation steps so far:: Summary of steps for easy testing.
* Larger steps sizes for better calibration:: The initial small dither is not enough.
* Pointings that account for sky curvature:: Sky curvature will cause problems!
* Accounting for non-exposed pixels:: Parts of the detector do not get exposed to light.
File: gnuastro.info, Node: Preparing input and generating exposure map, Next: Area of non-blank pixels on sky, Prev: Pointing pattern design, Up: Pointing pattern design
2.8.1 Preparing input and generating exposure map
-------------------------------------------------
As mentioned in *note Pointing pattern design::, the assumed goal here
is to plan an observations strategy for M94. Let's assume that after
some searching, you decide to write a proposal for the JAST80 telescope
(https://oaj.cefca.es/telescopes/jast80) at the Observatorio Astrofísico
de Javalambre (https://oaj.cefca.es), OAJ(1), in Teruel (Spain). The
field of view of this telescope's camera is almost 1.4 degrees wide,
nicely fitting M94! It also has these two filters that you need(2).
Before we start, as described in *note Pointing pattern simulation::,
it is just important to remember that the ideal pointing pattern depends
primarily on your scientific objective, as well as the limitations of
the instrument you are observing with. Therefore, there is no single
pointing pattern for all purposes. However, the tools, methods,
criteria or logic to check if your pointing pattern satisfies your
scientific requirement are similar. Therefore, you can use the same
methods, tools or logic here to simulate or verify that your pointing
pattern will produce the products you expect after the observation.
To start simulating a pointing pattern for a certain telescope, you
just need a single-exposure image of that telescope with WCS
information. In other words, after astrometry, but before warping into
any other pixel grid (to combine into a deeper stack). The image will
give us the default number of the camera's pixels, its pixel scale
(width of pixel in arcseconds) and the camera distortion. These are
reference parameters that are independent of the position of the image
on the sky.
Because the actual position of the reference image is irrelevant,
let's assume that in a previous project, presumably on NGC 4395
(https://en.wikipedia.org/wiki/NGC_4395), you already had the download
command of the following single exposure image. With the last command,
please take a look at this image before continuing and explore it.
$ mkdir pointing-tutorial
$ cd pointing-tutorial
$ mkdir input
$ siapurl=https://archive.cefca.es/catalogues/vo/siap
$ wget $siapurl/jplus-dr3/reduced/get_fits?id=1050345 \
-O input/jplus-1050345.fits.fz
$ astscript-fits-view input/jplus-1050345.fits.fz
*This is the first time I am using an instrument:* In case you haven't
already used images from your desired instrument (to use as reference),
you can find such images from their public archives; or contacting them.
A single exposure images is rarely of any scientific value
(post-processing and stacking is necessary to make high-level and
science-ready products). Therefore, they become publicly available very
soon after the observation date; furthermore, calibration images are
usually public immediately.
As you see from the image above, the T80Cam images are large (9216 by
9232 pixels). Therefore, to speed up the pointing testing, let's
down-sample the image by a factor of 10. This step is optional and you
can safely use the full resolution, which will give you a more precise
stack. But it will be much slower (maybe good after you have an almost
final solution on the down-sampled image). We will call the output
‘ref.fits’ (since it is the "reference" for our test). We are putting
these two "input" files (to the script) in a dedicated directory to keep
the running directory clean (and be able to easily delete temporary/test
files for a fresh start with a '‘rm *.fits’').
$ astwarp input/jplus-1050345.fits.fz --scale=1/10 -oinput/ref.fits
For a first trial, let's create a cross-shaped pointing pattern with
5 points around M94, which is centered at its center on the RA and Dec
of 192.721250, 41.120556. We'll center one exposure on the center of
the galaxy, and include 4 more exposures that are each 1 arc-minute away
along the RA and Dec axes. To simplify the actual command later(3),
let's also include the column names in ‘pointing.txt’ through two lines
of metadata. Also note that the ‘pointing.txt’ file can be made in any
manner you like, for example, by writing the coordinates manually on
your favorite text editor, or through another programming language or
logic, or etc. Here, we are using AWK because it is sufficiently
powerful for this job, and it is a very small program that is available
on any Unix-based operating system (allowing you to easily run your
programs on any computer).
$ step_arcmin=1
$ center_ra=192.721250
$ center_dec=41.120556
$ echo "# Column 1: RA [deg, f64] Right Ascension" > pointing.txt
$ echo "# Column 2: Dec [deg, f64] Declination" >> pointing.txt
$ echo $center_ra $center_dec \
| awk '{s='$step_arcmin'/60; fmt="%-10.6f %-10.6f\n"; \
printf fmt, $1, $2; \
printf fmt, $1+s, $2; \
printf fmt, $1, $2+s; \
printf fmt, $1-s, $2; \
printf fmt, $1, $2-s}' \
>> pointing.txt
With the commands below, let's have a look at the produced file,
first as plain-text, then with TOPCAT (which needs conversion to FITS).
After TOPCAT is opened, in the "Graphics" menu, select "Plane plot" to
see the five points in a flat RA, Dec plot.
$ cat pointing.txt
# Column 1: RA [deg, f64] Right Ascension
# Column 2: Dec [deg, f64] Declination
192.721250 41.120556
192.737917 41.120556
192.721250 41.137223
192.704583 41.120556
192.721250 41.103889
$ asttable pointing.txt -opointing.fits
$ astscript-fits-view pointing.fits
$ rm pointing.fits
We are now ready to generate the exposure map of the pointing pattern
above using the reference image that we downloaded before. Let's put
the center of our final stack to be on the center of the galaxy, and
we'll assume the stack has a size of 2 degrees. With the second
command, you can see the exposure map of the final stack. Recall that
in this image, each pixel shows the number of input images that went
into it.
$ astscript-pointing-simulate pointing.txt --output=stack.fits \
--img=input/ref.fits --center=$center_ra,$center_dec \
--width=2
$ astscript-fits-view stack.fits
You will see that except for a thin boundary, we have a depth of 5
exposures over the area of the single exposure. Let's see what the
width of the deepest part of the image is. First, we'll use Arithmetic
to set all pixels that contain less than 5 exposures (the outer pixels)
to NaN (Not a Number). In the same Arithmetic command, let's trim all
the blank rows and columns, so the output only contains the pixels that
are exposed 5 times. With the next command, let's view the deep region
and with the last command below, let's use the ‘--skycoverage’ option of
the Fits program to see the coverage of deep part on the sky.
$ deep_thresh=5
$ astarithmetic stack.fits set-s s s $deep_thresh lt nan where trim \
--output=deep.fits
$ astscript-fits-view deep.fits
$ astfits deep.fits --skycoverage
Input file: deep.fits (hdu: 1)
Sky coverage by center and (full) width:
Center: 192.72125 41.120556
Width: 1.880835157 1.392461166
Sky coverage by range along dimensions:
RA 191.7808324 193.6616676
DEC 40.42058203 41.81304319
As we see, in declination, the width of this deep field is about 1.4
degrees. Recall that RA is only defined on the equator and actual
coverage in RA depends on the declination due to the spherical nature of
the sky. This area therefore nicely covers the expected outer parts of
M94. On first thought, it may seem that we are now finished, but that
is not the case unfortunately!
There is a problem: with a step size of 1 arc-minute, the brighter
central parts of this large galaxy will always be on very similar
pixels; making it hard to calibrate those pixels properly. If you are
interested in the low surface brightness parts of this galaxy, it is
even worse: the outer parts of the galaxy will always cover similar
parts of the detector in all the exposures; and they cover a large area
on your image. To be able to accurately calibrate the image (in
particular to estimate the flat field pattern and subtract the sky), you
do not want this to happen! You want each exposure to cover very
different sources of astrophysical signal, so you can accurately
calibrate the artifacts created by the instrument or environment (for
example flat field) or of natural causes (for example the Sky).
For an example of how these calibration issues can ruin low surface
brightness science, please see the image of M94 in the Legacy Survey
interactive viewer (https://www.legacysurvey.org/viewer). After it is
loaded, at the bottom-left corner of the window, write "M94" in the box
of "Jump to object" and press ENTER. At first, M94 looks good with a
black background, but as you increase the "Brightness" (by scrolling it
to the right and seeing what is under the originally black pixels), you
will see the calibration artifacts clearly.
---------- Footnotes ----------
(1) For full disclosure, Gnuastro is being developed at CEFCA (Centro
de Estudios de Física del Cosmos de Aragón); which also hosts OAJ.
(2) For the full list of available filters, see the T80Cam
description (https://oaj.cefca.es/telescopes/t80cam).
(3) Instead of this, later, when you called
‘astscript-pointing-simulate’, you could pass the ‘--racol=1’ and
‘--deccol=2’ options. But having metadata is always preferred (will
avoid many bugs/frustrations in the long-run!).
File: gnuastro.info, Node: Area of non-blank pixels on sky, Next: Script with pointing simulation steps so far, Prev: Preparing input and generating exposure map, Up: Pointing pattern design
2.8.2 Area of non-blank pixels on sky
-------------------------------------
In *note Preparing input and generating exposure map:: we generated a
pointing pattern with very small steps, showing how this can cause
calibration problems. Later (in *note Larger steps sizes for better
calibration::) using larger steps is discussed. In this section, let's
see how we can get an accurate measure of the area that is covered in a
certain depth.
A first thought would be to simply multiply the widths along RA and
Dec reported before: $1.8808\times1.3924=2.6189$ degrees squared. But
there are several problems with this:
• It ignores the fact that RA only has units of degrees on the
equator: at different declinations, differences in RA should be
converted to degrees. This is discussed further in this tutorial:
*note Pointings that account for sky curvature::.
• It doesn't take into account the thin rows/columns of blank pixels
(NaN) that are on the four edges of the ‘deep.fits’ image.
• The differing area of the pixels on the spherical sky in relation
to those blank values can result in wrong estimations of the area.
Let's get a very accurate estimation of the area that will not be
affected by the issues above. With the first command below, we'll use
the ‘--pixelareaonwcs’ option of the Fits program that will return the
area of each pixel (in pixel units of degrees squared). After running
the second command, please have a look at the produced image.
$ astfits deep.fits --pixelareaonwcs --output=deep-pix-area.fits
$ astfits deep.fits --pixelscale
Basic info. for --pixelscale (remove extra info with '--quiet' or '-q')
Input: deep.fits (hdu 1) has 2 dimensions.
Pixel scale in each FITS dimension:
1: 0.00154403 (deg/pixel) = 5.5585 (arcsec/pixel)
2: 0.00154403 (deg/pixel) = 5.5585 (arcsec/pixel)
Pixel area:
2.38402e-06 (deg^2) = 30.8969 (arcsec^2)
$ astscript-fits-view deep-pix-area.fits
You see a donut-like shape in DS9. Move your mouse over the central
(white) region of the region and look at the values. You will see that
the pixel area (in degrees squared) is exactly the same as we saw in the
output of ‘--pixelscale’. As you move your mouse away to other colors,
you will notice that the area covered by each pixel (its value in this
image) deceases very slightly (in the 5th decimal!). This is the effect
of the Gnomonic projection
(https://en.wikipedia.org/wiki/Gnomonic_projection); summarized as ‘TAN’
(for "tangential") in the FITS WCS standard, the most commonly used in
optical astronomical surveys and the default in this script.
Having ‘deep-pix-area.fits’, we can now use Arithmetic to set the
areas of all the pixels that were NaN in ‘deep.fits’ and sum all the
values to get an accurate estimate of the area we get from this pointing
pattern:
$ astarithmetic deep-pix-area.fits deep.fits isblank nan where -g1 \
sumvalue --quiet
1.93836806631634e+00
Therefore, the actual area that is covered is less than the simple
multiplication above. At these declinations, the dominant cause of this
difference is the first point above (that RA needs correction), this
will be discussed in more detail later in this tutorial (see *note
Pointings that account for sky curvature::). Generally, using this
method to measure the area of your non-NAN pixels in an image is very
easy and robust (automatically takes into account the curvature,
coordinate system, projection and blank pixels of the image).
File: gnuastro.info, Node: Script with pointing simulation steps so far, Next: Larger steps sizes for better calibration, Prev: Area of non-blank pixels on sky, Up: Pointing pattern design
2.8.3 Script with pointing simulation steps so far
--------------------------------------------------
In *note Preparing input and generating exposure map:: and *note Area of
non-blank pixels on sky::, the basic steps to simulate a pointing
pattern's exposure map and measure the final output area on the sky
where described in detail. From this point on in the tutorial, we will
be experimenting with the shell variables that were set above, but the
actual commands will not be changed regularly. If a change is necessary
in a command, it is clearly mentioned in the text.
Therefore, it is better to write the steps above (after downloading
the reference image) as a script. In this way, you can simply change
those variables and see the final result fast by running your script.
For more on writing scripts, see as described in *note Writing scripts
to automate the steps::.
Here is a summary of some points to remember when transferring the
code in the sections before into a script:
• Where the commands are edited/changed, please also update them in
your script.
• Keep all the variables at the top, even if they are used later.
This allows to easily view or changed them without digging into the
script.
• You do not need to include visual check commands like the
‘astscript-fits-view’ or ‘cat’ commands above. Those can be run
interactively after your script is finished; recall that a script
is for batch (non-interactive) processing.
• Put all your intermediate products inside a "build" directory.
Here is the script that summarizes the steps in *note Preparing input
and generating exposure map:: (after download) and *note Area of
non-blank pixels on sky:::
#!/bin/bash
#
# Copyright (C) 2024-2024 Mohammad Akhlaghi <mohammad@akhlaghi.org>
#
# Copying and distribution of this file, with or without modification,
# are permitted in any medium under the GNU GPL v3+, without royalty
# provided the copyright notice and this notice are preserved. This
# file is offered as-is, without any warranty.
# Parameters of the script
deep_thresh=5
step_arcmin=1
center_ra=192.721250
center_dec=41.120556
# Input and build directories (can be anywhere in your file system)
indir=input
bdir=build
# Abort the script in case of an error.
set -e
# Make the build directory if it doesn't already exist.
if ! [ -d $bdir ]; then mkdir $bdir; fi
# Build the 5-pointing pointing pattern (with the step size above).
pointingcat=$bdir/pointing.txt
echo "# Column 1: RA [deg, f64] Right Ascension" > $pointingcat
echo "# Column 2: Dec [deg, f64] Declination" >> $pointingcat
echo $center_ra $center_dec \
| awk '{s='$step_arcmin'/60; fmt="%-10.6f %-10.6f\n"; \
printf fmt, $1, $2; \
printf fmt, $1+s, $2; \
printf fmt, $1, $2+s; \
printf fmt, $1-s, $2; \
printf fmt, $1, $2-s}' \
>> $pointingcat
# Simulate the pointing pattern.
stack=$bdir/stack.fits
astscript-pointing-simulate $pointingcat --output=$stack \
--img=input/ref.fits --center=$center_ra,$center_dec \
--width=2
# Trim the regions shallower than the threshold.
deep=$bdir/deep.fits
astarithmetic $stack set-s s s $deep_thresh lt nan where trim \
--output=$deep
# Calculate the area of each pixel on the curved celestial sphere:
pixarea=$bdir/deep-pix-area.fits
astfits $deep --pixelareaonwcs --output=$pixarea
# Report the final area (the empty 'echo's are for visual help in outputs)
echo; echo
echo "Area with step of $step_arcmin arcminutes, at $deep_thresh depth:"
astarithmetic $pixarea $deep isblank nan where -g1 \
sumvalue --quiet
For a description of how to make it executable and how to run it, see
*note Writing scripts to automate the steps::. Note that as you start
adding your own text to the script, be sure to add your name (and year
that you modified) in the copyright notice at the start of the script
(this is very important!).
File: gnuastro.info, Node: Larger steps sizes for better calibration, Next: Pointings that account for sky curvature, Prev: Script with pointing simulation steps so far, Up: Pointing pattern design
2.8.4 Larger steps sizes for better calibration
-----------------------------------------------
In *note Preparing input and generating exposure map:: we saw that a
small pointing pattern is not good for the reduction of data from a
large object like M94! M94 is about half a degree in diameter; so let's
set ‘step_arcmin=15’. This is one quarter of a degree and will put the
center of the four exposures on the four corners of the M94's main ring.
Furthermore, *note Script with pointing simulation steps so far::, the
steps were summarized into a script to allow easy changing of variables
without manually re-entering the individual/separate commands.
After you change ‘step_arcmin=15’ and re-run the script, you will get
a total area (from counting of per-pixel areas) of approximately 0.96
degrees squared. This is just roughly half the previous area and will
barely fit M94! To understand the cause, let's have a look at the full
stack (not just the deepest area):
$ astscript-fits-view build/stack.fits
Compared to the first run (with ‘step_arcmin=1’), we clearly see how
there are indeed fewer pixels that get photons in all 5 exposures. As
the area of the deepest part has decreased, the areas with fewer
exposures have also grown. Let's define our deep region to be the
pixels with 3 or more exposures. Please set ‘deep_thresh=3’ in the
script and re-run it. You will see that the "deep" area is now almost
2.02 degrees squared! This is (slightly) larger than the first run
(with ‘step_arcmin=1’)!
The difference between 3 exposures and 5 exposures seems a lot at
first. But let's calculate how much it actually affects the achieved
signal-to-noise ratio and the surface brightness limit. The surface
brightness limit (or upper-limit surface brightness) are both calculated
by applying the definition of magnitude to the standard deviation of the
background. So we should first calculate how much this difference in
depth affects the sky standard deviation. For a complete discussion on
the definition of the surface brightness limit, see *note Quantifying
measurement limits::.
Deep images will usually be dominated by *note Photon counting
noise:: (or Poisson noise). Therefore, if a single exposure image has a
sky standard deviation of $\sigma_s$, and we combine $N$ such exposures
by taking their mean, the final/stacked sky standard deviation
($\sigma$) will be $\sigma=\sigma_s/\sqrt{N}$. As a result, the surface
brightness limit between the regions with $N$ exposures and $M$
exposures differs by $2.5\times log_{10}(\sqrt{N/M}) = 1.25\times
log_{10}(N/M)$ magnitudes. If we set $N=3$ and $M=5$, we get a surface
brightness magnitude difference of 0.28!
This is a very small difference; given all the other sources of error
that will be present; but how much it improves the calibration
artifacts. Therefore at the cost of decreasing our surface brightness
limit by 0.28 magnitudes, we are now able to calibrate the individual
exposures much better, and even cover a larger area!
The argument above didn't involve any image and was primarily
theoretical. For the more visually-inclined readers, let's add raw
Gaussian noise (with a $\sigma$ of 100 counts) over each simulated
exposure. We will then instruct ‘astscript-pointing-simulate’ to stack
them as we would stack actual data (by taking the sigma-clipped mean).
The command below is identical to the previous call to the pointing
simulation script with the following differences. Note that this is
just for demonstration, so you should not include this in your script
(unless you want to see the noisy stack every time; at double the
processing time).
‘--output’
We are using a different output name, so we can compare the output
of the new command with the previous one.
‘--stack-operator’
This should be one of the Arithmetic program's *note Stacking
operators::. By default the value is ‘sum’; because by default,
each pixel of each exposure is given a value of 1. When stacking
is defined through the summation operator, we can obtain the
exposure map that you have already seen above.
But in this run, we are adding noise to each input exposure
(through the hook that is described below) and stacking them (as we
would stack actual science images). Since the purpose differs
here, we are using this option to change the operator.
‘--hook-warp-after’
This is the most visible difference of this command the previous
one. Through a "hook", you can give any arbitrarily long (series
of) command(s) that will be added to the processing of this script
at a certain location. This particular hook gets applied "after"
the "warp"ing phase of each exposure (when the pixels of each
exposure are mapped to the final pixel grid; but not yet stacked).
Since the script runs in parallel (the actual work-horse is a
Makefile!), you can't assume any fixed file name for the input(s)
and output. Therefore the inputs to, and output(s) of, hooks are
some pre-defined shell variables that you should use in the
command(s) that you hook into the processing. They are written in
full-caps to be clear and separate from your own variables. In
this case, they are the ‘$WARPED’ (input file of the hook) and
‘$TARGET’ (output name that next steps in the script will operate
on). As you see from the command below, through this hook we are
calling the Arithmetic program to add noise to all non-zero pixels
in the warped image. For more on the noise-adding operators, see
*note Random number generators::.
$ center_ra=192.721250
$ center_dec=41.120556
$ astscript-pointing-simulate build/pointing.txt --img=input/ref.fits \
--center=$center_ra,$center_dec \
--width=2 --stack-operator="3 0.2 sigclip-mean" \
--output=build/stack-noised.fits \
--hook-warp-after='astarithmetic $WARPED set-i \
i i 0 uint8 eq nan where \
100 mknoise-sigma \
--output=$TARGET'
$ astscript-fits-view build/stack.fits build/stack-noised.fits
When you visually compare the two images of the last command above,
you will see that (at least by eye) it is almost impossible to
distinguish the differing noise pattern in the regions with 3 exposures
from the regions with 5 exposures. But the regions with a single
exposure are clearly visible! This is because the surface brightness
limit in the single-exposure regions is $1.25\times\log_{10}(1/5)=-0.87$
magnitudes brighter. This almost one magnitude difference in surface
brightness is significant and clearly visible in the stacked image
(recall that magnitudes are measured in a logarithmic scale).
Thanks to the argument above, we can now have a sufficiently large
area with a usable depth. However, each the center of each pointing
will still contain the central part of the galaxy. In other words, M94
will be present in all the exposures while doing the calibrations. Even
in not-too-deep observations, we already see a large ring around this
galaxy. When we do a low surface brightness optimized reduction, there
is a good chance that the size of the galaxy is much larger than that
ring. This very extended structure will make it hard to do the
calibrations on very accurate scales. Accurate calibration is necessary
if you do not want to loose the faint photons that have been recorded in
your exposures.
*Calibration is very important:* Better calibration can result in a
fainter surface brightness limit than more exposures with poor
calibration; especially for very low surface brightness signal that
covers a large area and is systematically affected by calibration
issues.
Ideally, you want your target to be on the four edges/corners of each
image. This will make sure that a large fraction of each exposure will
not be covered by your final target in each exposure, allowing you to
calibrate much more accurately.
File: gnuastro.info, Node: Pointings that account for sky curvature, Next: Accounting for non-exposed pixels, Prev: Larger steps sizes for better calibration, Up: Pointing pattern design
2.8.5 Pointings that account for sky curvature
----------------------------------------------
In *note Larger steps sizes for better calibration::, we saw how a small
loss in surface brightness limit can allow better calibration and even a
larger area. Let's extend this by setting ‘step_arcmin=40’ (almost half
the width of the detector) inside your script (see *note Script with
pointing simulation steps so far::). After running the script with this
change, take a look at ‘build/deep.fits’:
$ astscript-fits-view build/deep.fits --ds9scale=minmax
You will see that the region with 5 exposure depth is a horizontally
elongated rectangle now! Also, the vertical component of the cross with
four exposures is much thicker than the horizontal component! Where
does this asymmetry come from? All the steps in our pointing strategy
had the same (fixed) size of 40 arc minutes.
This happens because the same change in RA and Dec (defined on the
curvature of a sphere) will result in different absolute changes on the
equator. To visually see this, let's look at the pointing positions in
TOPCAT:
$ cat build/pointing.txt
# Column 1: RA [deg, f64] Right Ascension
# Column 2: Dec [deg, f64] Declination
192.721250 41.120556
193.387917 41.120556
192.721250 41.787223
192.054583 41.120556
192.721250 40.453889
$ asttable build/pointing.txt -obuild/pointing.fits
$ astscript-fits-view build/pointing.fits
After TOPCAT opens, under the "graphics" window, select "Plane Plot".
In the newly opened window, click on the "Axes" item on the bottom-left
list of items. Then activate the "Aspect lock" box so the vertical and
horizontal axes have the same scaling. You will see what you expect
from the numbers: we have a beautifully symmetric set of 5 points shaped
like a '+' sign.
Keep the previous window, and let's go back to the original TOPCAT
window. In the first TOPCAT window, click on "Graphics" again, but this
time, select "Sky plot". You will notice that the vertical component of
the cross is now longer than the horizontal component! If you zoom-out
(by scrolling your mouse over the plot) a lot, you will see that this is
actually on the spherical surface of the sky! In other words, as you
see here, on the sky, the horizontal points are closer to each other
than the vertical points; causing a larger overlap between them, making
the vertical overlap thicker in ‘build/pointing.fits’.
On the celestial sphere, only the declination is measured in degrees.
In other words, the difference in declination of two points can be
calculated only with their declination. However, except for points that
are on the equator, differences in right ascension depend on the
declination. Therefore, the origin of this problem is that we done the
additions and subtractions for defining the pointing points in a flat
space: based on the step size in arc minutes that was applied similarly
on RA and Dec (in *note Preparing input and generating exposure map::).
To fix this problem, we need to convert our points from the flat
RA/Dec into the spherical RA/Dec. In the FITS standard, we have the
"World Coordinate System" (WCS) that defines this type of conversion,
using pre-defined projections in the ‘CTYPEi’ keyword (short for for
"Coordinate TYPE in dimension i"). Let's have a look at the stack to
see the default projection of our final stack:
$ astfits build/stack.fits -h1 | grep CTYPE
CTYPE1 = 'RA---TAN' / Right ascension, gnomonic projection
CTYPE2 = 'DEC--TAN' / Declination, gnomonic projection
We therefore see that the default projection of our final stack is
the ‘TAN’ (short for "tangential") projection, which is more formally
known as the Gnomonic projection
(https://en.wikipedia.org/wiki/Gnomonic_projection). This is the most
commonly used projection in optical astronomy. Now that we know the
final projection, we can do this conversion using Table's column
arithmetic operator ‘eq-j2000-from-flat’ like below:
$ pointingcat=build/pointing.txt
$ pointingonsky=build/pointing-on-sky.fits
$ asttable $pointingcat --output=$pointingonsky \
-c'arith RA set-r \
DEC set-d \
r meanvalue set-ref-r \
d meanvalue set-ref-d \
r d ref-r ref-d TAN eq-j2000-from-flat' \
--colmetadata=1,RA,deg,"Right ascension" \
--colmetadata=2,Dec,deg,"Declination"
$ astscript-fits-view build/pointing-on-sky.fits
Here is a break-down of the first command above: to do the
flat-to-sky conversion, we need a reference point (where the two are
equal). We have used the mean RA and mean Dec (through the ‘meanvalue’
operator in Arithmetic) as our reference point (which are placed in the
‘ref-r’ and ‘red-d’ variables. After calling the ‘eq-j2000-from-flat’
operator, we have just added metadata to the two columns.
To confirm that this operator done the job correctly, after the
second command above, repeat the same experiment as before with TOPCAT
(where you viewed the pointing positions on a flat and spherical
coordinate system). You will see that indeed, on the sphere you have a
'+' shape, but on the flat plot, it looks stretched.
*Script update 1:* you should now add the ‘pointingonsky’ definition and
the ‘asttable’ command above into the script of *note Script with
pointing simulation steps so far::. They should be placed before the
call to ‘astscript-pointing-simulate’. Also, in the call to
‘astscript-pointing-simulate’, ‘$pointingcat’ should be replaced with
‘$pointingonsky’ (so it doesn't use the flat RA, Dec pointings).
After implementing this change in your script and running it, open
‘deep.fits’ and you will see that the widths of both the horizontal and
vertical regions are much more similar. The top of the vertical overlap
is slightly wider than the bottom, but that is something you can't fix
by just pointing (your camera's field of view is fixed on the sky!). It
can be correctly by slightly rotating some of the exposures, but that
will result in different PSFs from one exposure to another; and this can
cause more important problems for your final science.
*Plotting the spherical RA and Dec in your papers:* The inverse of the
‘eq-j2000-from-flat’ operator above is the ‘eq-j2000-to-flat’.
‘eq-j2000-to-flat’ can be used when you want to plot a set points with
spherical RA and Dec in a paper. When the minimum and maximum RA and
Dec differ by larger than half a degree, you'll clearly see the
difference. For more, see the description of these operators in *note
Column arithmetic::.
Try to slightly increase ‘step_arcmin’ to make the cross-like region
with 4 exposures as thin as possible. For example, set it to
‘step_arcmin=42’. When you open ‘deep.fits’, you will see that the
depth across this image is almost contiguous (which is another positive
factor!). Try increasing it to 43 arc minutes to see that the central
cross will become almost fully NaN in ‘deep.fits’ (which is bad!).
You will notice that the vertical region of 4 exposure depth is
thinner in the bottom than on the top. This is due to the RA/Dec change
above, but across the width of the image. We can't therefore change
this by just changing the position of the pointings, we need to rotate
some of the exposures if we want it to be removed. But rotation is not
yet implemented in this script.
You can construct any complex pointing pattern (with more than 5
points and in any shape) based on the logic and reasoning above to help
extract the most science from the valuable telescope time that you will
be getting. Since the output is a FITS file, you can easily download
another FITS file of your target, open it with DS9 (and "lock" the
"WCS") with the stack produced by this simulation to make sure that the
deep parts correspond to the area of interest for your science case.
Factors like the optimal exposure time are also critical for the
final result(1), but is was beyond the scope of this tutorial. One
relevant factor however is the effect of vignetting: the pixels on the
outer extremes of the field of view that are not exposed to light and
should be removed from your final stack. They effect your pointing
pattern: by decreasing your total area, they act like a larger spacing
between your points, causing similar shallow crosses as you saw when you
set ‘step_arcmin’ to 43 arc minutes. In *note Accounting for
non-exposed pixels::, we will show how this can be done within the same
test concept that we done here.
---------- Footnotes ----------
(1) The exposure time will determine the Signal-to-noise ration on a
single exposure level.
File: gnuastro.info, Node: Accounting for non-exposed pixels, Prev: Pointings that account for sky curvature, Up: Pointing pattern design
2.8.6 Accounting for non-exposed pixels
---------------------------------------
At the end of *note Pointings that account for sky curvature:: we were
able to maximize the region of same depth in our stack. But we noticed
that issues like strong vignetting
(https://en.wikipedia.org/wiki/Vignetting) can create discontinuity in
our final stacked data product. In this section, we'll review the steps
to account for such effects. Generally, the full area of a detector is
not usually used in the final stack. Vignetting is one cause, it can be
due to other problems also. For example due to baffles in the optical
path (to block stray light), or large regions of bad (unusable or
"dead") pixels that may be in any place on the detector(1).
Without accounting for these pixels that do not receive any light,
the deep area we measured in the sections above will be over-estimated.
In this sub-section, let's review the necessary additions to account for
such artifacts. Therefore, before continuing, please make sure that you
have already read and applied the steps of the previous sections (this
sub-section builds upon that section).
Vignetting strongly depends on the optical design of the instrument
you are using. It can be a constant number of pixels on all the edges
the detector, or it can have a more complex shape. For example on
cameras that have multiple detectors on the field of view, in this case,
the regions to exclude on each detector can be very different and will
not be symmetric!
Therefore, within Gnuastro's ‘astscript-pointing-simulate’ script
there is no parameter for pre-defined vignetting shapes. Instead, you
should define a mask that you can apply on each exposure through the
provided hook (‘--hook-warp-before’; recall that we previously used
another hook in *note Larger steps sizes for better calibration::).
Through the mask, you are free to set any vignetted or bad pixel to NaN
(thus ignoring them in the stack) and applying it in any way that best
suites your instrument and detector.
The mask image should be same size as the reference image, but only
containing two values: 0 or 1. Pixels in each exposure that have a
value of 1 in the mask will be set to NaN before the stacking process
and will not contribute to the final stack. Ideally, you can use the
master flat field image of the previous reductions to create this mask:
any pixel that has a low sensitivity in the master flat (for any reason)
can be set to 1, and the rest of the pixels to 0.
Let's build a simple mask by assuming that we only have strong
vignetting that is affecting the outer 30 arc seconds of the individual
exposures. To mask the outer edges of an image we can use Gnuastro's
Arithmetic program; and in particular, the ‘indexonly’ operator. To
learn more about this operator, see *note Size and position operators::.
But before doing that, we need convert this angular distance to
pixels on the detector. In *note Pointing pattern design::, we used an
undersampled version of the input image, so we should do this conversion
on that image:
$ margin_arcsec=30
$ margin_pix=$(astfits input/ref.fits --pixelscale --quiet \
| awk '{print int('$margin_arcsec'/($1*3600))}')
$ echo $margin_pix
5
To build the mask, we can now follow the recipe under "Image: masking
margins" of the ‘index’ operator in Arithmetic (for a full description
of what this command is doing(2), see *note Size and position
operators::). Finally, in the last command, let's look at the mask
image in the "red" color map of DS9 (which will shows the thin 1-valued
pixels to mask on the border more clearly).
$ width=$(astfits input/ref.fits --keyvalue=NAXIS1 -q)
$ height=$(astfits input/ref.fits --keyvalue=NAXIS2 -q)
$ astarithmetic input/ref.fits indexonly set-i \
$width uint16 set-w \
$height uint16 set-h \
$margin_pix uint16 set-m \
i w % uint16 set-X \
i w / uint16 set-Y \
X m lt X w m - gt or \
Y m lt Y h m - gt or \
or --output=build/mask.fits
$ astscript-fits-view build/mask.fits --ds9extra="-cmap red"
We are now ready to run the main pointing simulate script. With the
command below, we will use the ‘--hook-warp-before’ to apply this mask
on the image of each exposure just before warping. The concept of this
hook is very similar to that of ‘--hook-warp-after’ in *note Pointing
pattern design::. As the name suggests, this hook is applied "before"
the warping. The input to the command given to this hook should be
called with ‘$EXPOSURE’ and the output should be called with ‘$TOWARP’.
With the second command, let's compare the two outputs:
$ astscript-pointing-simulate build/pointing-on-sky.fits \
--output=build/stack-with-trim.fits --img=input/ref.fits \
--center=$center_ra,$center_dec --width=2 \
--hook-warp-before='astarithmetic $EXPOSURE build/mask.fits \
nan where -g1 -o$TOWARP'
$ astscript-fits-view build/stack.fits build/stack-with-trim.fits
As expected, due to the smaller area of the detector that is exposed
to photons, the regions with 4 exposures have become much thinner and on
the bottom, it has been removed. To have contiguous depth in the deeper
region, use this new call in your script and decrease the
‘step_arcmin=41’.
You can use the same command on a mask that is created in any way and
as realistic as possible. More generically, you can use the before and
after hooks for any other operation; for example to insert objects from
a catalog using *note MakeProfiles:: as well as adding noise as we did
in *note Pointing pattern design::.
Therefore it is also good to add the mask and its application in your
script. This should be pretty easy by now (following *note Script with
pointing simulation steps so far:: and the "Script update 1" box of
*note Pointings that account for sky curvature::). So we will leave
this as an exercise.
---------- Footnotes ----------
(1) For an example of bad pixels over the detector, see Figures 4 and
6 of Instrument Science Report WFC3 2019-03
(https://www.stsci.edu/files/live/sites/www/files/home/hst/instrumentation/wfc3/documentation/instrument-science-reports-isrs/_documents/2019/WFC3-2019-03.pdf)
by the Space Telescope Science Institute.
(2) By learning how this command works, you can customize it. For
example, to mask different widths along each separate edge: it often
happens that the left/right or top/bottom edges are affected differently
by vignetting.
File: gnuastro.info, Node: Moire pattern in stacking and its correction, Next: Clipping outliers, Prev: Pointing pattern design, Up: Tutorials
2.9 Moiré pattern in stacking and its correction
================================================
After warping some images with the default mode of Warp (see *note Align
pixels with WCS considering distortions::) you may notice that the
background noise is no longer flat. Some regions will be smoother and
some will be sharper; depending on the orientation and distortion of the
input/output pixel grids. This is due to the Moiré pattern
(https://en.wikipedia.org/wiki/Moir%C3%A9_pattern), which is especially
noticeable/significant when two slightly different grids are
super-imposed.
With the commands below, we'll download a single exposure image from
the J-PLUS survey (https://www.j-plus.es) and run Warp (on a $8\times8$
arcmin$^2$ region to speed it up the demos here). Finally, we'll open
the image to visually see the artificial Moiré pattern on the warped
image.
## Download the image (73.7 MB containing an 9216x9232 pixel image)
$ jplusdr2=http://archive.cefca.es/catalogues/vo/siap/jplus-dr2/reduced
$ wget $jplusdr2/get_fits?id=771463 -Ojplus-exp1.fits.fz
## Align a small part of it with the sky coordinates.
$ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
--width=8/60 -ojplus-e1.fits
## Open the aligned region with DS9
$ astscript-fits-view jplus-e1.fits
In the opened DS9 window, you can see the Moiré pattern as wave-like
patterns in the noise: some parts of the noise are more smooth and some
parts are more sharp. Right in the center of the image is a blob of
sharp noise. Warp has the ‘--checkmaxfrac’ option for direct inspection
of the Moiré pattern (described with the other options in *note Align
pixels with WCS considering distortions::). When run with this option,
an extra HDU (called ‘MAX-FRAC’) will be added to the output. The image
in this HDU has the same size as the output. However, each output pixel
will contain the largest (maximum) fraction of area that it covered over
the input pixel grid. So if an output pixel has a value of 0.9, this
shows that it covered $90\%$ of an input pixel. Let's run Warp with
‘--checkmaxfrac’ and see the output (after DS9 opens, in the "Cube"
window, flip between the first and second HDUs):
$ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
--width=8/60 -ojplus-e1.fits --checkmaxfrac
$ astscript-fits-view jplus-e1.fits
By comparing the first and second HDUs/extensions, you will clearly
see that the regions with a sharp noise pattern fall exactly on parts of
the ‘MAX-FRAC’ extension with values larger than 0.5. In other words,
output pixels where one input pixel contributed more than half of the
its value. As this fraction increases, the sharpness also increases
because a single input pixel's value dominates the value of the output
pixel. On the other hand, when this value is small, we see that many
input pixels contribute to that output pixel. Since many input pixels
contribute to an output pixel, it acts like a convolution, hence that
output pixel becomes smoother (see *note Spatial domain convolution::).
Let's have a look at the distribution of the ‘MAX-FRAC’ pixel values:
$ aststatistics jplus-e1.fits -hMAX-FRAC
Statistics (GNU Astronomy Utilities) 0.22
-------
Input: jplus-e1.fits (hdu: MAX-FRAC)
-------
Number of elements: 744769
Minimum: 0.250213461
Maximum: 0.9987495374
Mode: 0.5034223567
Mode quantile: 0.3773819498
Median: 0.5520805544
Mean: 0.5693956458
Standard deviation: 0.1554693738
-------
Histogram:
| ***
| **********
| *****************
| ************************
| *******************************
| **************************************
| *********************************************
| ****************************************************
| ***********************************************************
| ******************************************************************
|**********************************************************************
|----------------------------------------------------------------------
The smallest value is 0.25 (=1/4), showing that 4 input pixels
contributed to the output pixels value. While the maximum is almost
1.0, showing that a single input pixel defined the output pixel value.
You can also see that the most probable value (the mode) is 0.5, and
that the distribution is positively skewed.
This is a well-known problem in astronomical imaging and professional
photography. If you only have a single image (that is already taken!),
you can undersample the input: set the angular size of the output pixels
to be larger than the input. This will decrease the resolution of your
image, but will ensure that pixel-mixing will always happen. In the
example below we are setting the output pixel scale (which is known as
‘CDELT’ in the FITS standard) to $1/0.5=2$ of the input's. In other
words each output pixel edge will cover double the input pixel's edge on
the sky, and the output's number of pixels in each dimension will be
half of the previous output.
$ cdelt=$(astfits jplus-exp1.fits.fz --pixelscale -q \
| awk '{print $1}')
$ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
--width=8/60 -ojplus-e1.fits --cdelt=$cdelt/0.5 \
--checkmaxfrac
In the first extension, you can hardly see any Moiré pattern in the
noise. When you go to the next (‘MAX-FRAC’) extension, you will see
that almost all the pixels have a value of 1. Of course, decreasing the
resolution by half is a little too drastic. Depending on your image,
you may be able to reach a sufficiently good result without such a
drastic degrading of the input image. For example, if you want an
output pixel scale that is just 1.5 times larger than the input, you can
divide the original coordinate-delta (or "cdelt") by $1/1.5=0.6666$ and
try again. In the ‘MAX-FRAC’ extension, you will see that the range of
pixel values is now between 0.56 to 1.0 (recall that originally, this
was between 0.25 and 1.0). This shows that the pixels are more
similarly mixed and in fact, when you look at the actual warped image,
you can hardly distinguish any Moiré pattern in the noise.
However, deep astronomical data are usually built by several
exposures (images), not a single one. Each image is also taken by
(slightly) shifting the telescope compared to the previous exposure.
This shift is known as "dithering" or a "pointing pattern", see *note
Pointing pattern design::. We do this for many reasons (for example
tracking errors in the telescope, high background values, removing the
effect of bad pixels or those affected by cosmic rays, robust flat
pattern measurement, etc.(1)). One of those "etc." reasons is to
correct the Moiré pattern in the final coadded deep image.
The Moiré pattern is fixed to the grid of the image, slightly
shifting the telescope will result in the pattern appearing in different
parts of the sky. Therefore when we later stack, or coadd, the separate
exposures into a deep image, the Moiré pattern will be decreased there.
However, dithering has possible drawbacks based on the scientific goal.
For example when observing time-variable phenomena where cutting the
exposures to several shorter ones is not feasible. If this is not the
case for you (for example in galaxy evolution), continue with the rest
of this section.
Because we have multiple exposures that are slightly (sub-pixel)
shifted, we can also increase the spatial resolution of the output. For
example, let's set the output coordinate-delta (‘--cdelt’, or pixel
scale) to be 1/2 of the input. In other words, the number of pixels in
each dimension of the output is double the first Warp command of this
section:
$ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
--width=8/60 -ojplus-e1.fits --cdelt=$cdelt/2 \
--checkmaxfrac
$ aststatistics jplus-e1.fits -hMAX-FRAC --minimum --maximum
6.26360438764095e-02 2.50680270139128e-01
$ astscript-fits-view jplus-e1.fits
From the last command, you see that like the previous change in
‘--cdelt’, the range of ‘MAX-FRAC’ has decreased. However, when you
look at the warped image and the ‘MAX-FRAC’ image with the last command,
you still visually see the Moiré pattern in the noise (although it has
significantly decreased compared to the original resolution). It is
still present because 2 is an exact multiple of 1. Let's try increasing
the resolution (oversampling) by a factor of 1.25 (which isn't an exact
multiple of 1):
$ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
--width=8/60 -ojplus-e1.fits --cdelt=$cdelt/1.25 \
--checkmaxfrac
$ astscript-fits-view jplus-e1.fits
You don't see any Moiré pattern in the noise any more, but when you
look at the ‘MAX-FRAC’ extension, you see it is very different from the
ones you had seen before. In the previous ‘MAX-FRAC’ image, you could
see large blobs of similar values. But here, you see that the variation
is almost on a pixel scale, and the difference between one pixel to the
next is not significant. This is why you don't see any Moiré pattern in
the warped image.
In J-PLUS, each part of the sky was observed with a three-point
pointing pattern (very small shifts in each pointing). Let's download
the other two exposures and warp the same region of the sky to the same
pixel grid (using the ‘--gridfile’ feature). Then, let's open all three
warped images in one DS9 instance:
$ wget $jplusdr2/get_fits?id=771465 -Ojplus-exp2.fits.fz
$ wget $jplusdr2/get_fits?id=771467 -Ojplus-exp3.fits.fz
$ astwarp jplus-exp2.fits.fz --gridfile jplus-e1.fits \
-o jplus-e2.fits --checkmaxfrac
$ astwarp jplus-exp3.fits.fz --gridfile jplus-e1.fits \
-o jplus-e3.fits --checkmaxfrac
$ astscript-fits-view jplus-e*.fits
In the three warped images, you don't see any Moiré pattern, so far so
good... now, take the following steps:
1. In the small "Cube" window, click the "Next" button so you see the
‘MAX-FRAC’ extension/HDU.
2. Click on the "Frame" button (in the top row of buttons just on top
of the image), and select the "Single" button in the bottom row.
3. Open the "Zoom" menu (not button), and select "Zoom 16".
4. Press the <TAB> key to flip through each exposure.
5. Focus your eyes on the pixels with the largest value (white colored
pixels), while pressing <TAB> to flip between the exposures. You
will see that in each exposure they cover different pixels (nicely
getting averaged out after stacking).
The exercise above shows that the Moiré pattern (that had already
decreased significantly) will be further decreased after we stack the
images. So let's stack these three images with the commands below.
First, we need to remove the sky-level from each image using *note
NoiseChisel::, then we'll stack the ‘INPUT-NO-SKY’ extensions using
filled MAD-clipping (to reject outliers, and especially diffuse
outliers, robustly, see *note Clipping outliers::).
$ for i in $(seq 3); do \
astnoisechisel jplus-e$i.fits -ojplus-nc$i.fits; \
done
$ astarithmetic jplus-nc*.fits 3 5 0.2 sigclip-mean \
-gINPUT-NO-SKY -ojplus-stack.fits
$ astscript-fits-view jplus-nc*.fits jplus-stack.fits
After opening the individual exposures and the final stack with the last
command, take the following steps to see the comparisons properly:
1. Click on the stack image so it is selected.
2. Go to the "Frame" menu, then the "Lock" item, then activate "Scale
and Limits".
3. Scroll your mouse or touchpad to zoom into the image.
You clearly see that the stacked image is deeper and that there is no
Moiré pattern, while you have slightly _improved_ the spatial resolution
of the output compared to the input. In case you want the stack to have
the original pixel resolution, you just need one more warp:
$ astwarp jplus-stack.fits --cdelt=$cdelt -ojplus-stack-origres.fits
For optimal results, the oversampling should be determined by the
dithering pattern of the observation: For example if you only have two
dither points, you want the pixels with maximum value in the ‘MAX-FRAC’
image of one exposure to fall on those with a minimum value in the other
exposure. Ideally, many more dither points should be chosen when you
are planning your observation (not just for the Moiré pattern, but also
for all the other reasons mentioned above). Based on the dithering
pattern, you want to select the increased resolution such that the
maximum ‘MAX-FRAC’ values fall on every different pixel of the output
grid for each exposure. Note that this discussion is on small shifts
between pointings (dithers), not large ones like offsets); see *note
Pointing pattern design::.
---------- Footnotes ----------
(1) E.g.,
<https://www.stsci.edu/hst/instrumentation/wfc3/proposing/dithering-strategies>
File: gnuastro.info, Node: Clipping outliers, Prev: Moire pattern in stacking and its correction, Up: Tutorials
2.10 Clipping outliers
======================
Outliers occur often in data sets. For example cosmic rays in
astronomical imaging: the image of your target galaxy can be affected by
a cosmic ray in one of the five exposures you took in one night. As a
result, when you compare the measured magnitude of your target galaxy in
all the exposures, you will get measurements like this (all in
magnitudes) 19.8, 20.1, 20.5, 17.0, 19.9 (all fluctuating around
magnitude 20, except the much brighter 17th magnitude measurement).
Normally, you would simply take the mean of these measurements to
estimate the magnitude of your target with more precision. However, the
17th magnitude measurement above is clearly wrong and will significantly
affect the mean: without it, the mean magnitude is 20.07, but with it,
the mean is 19.46:
$ echo " 19.8 20.1 20.5 17 19.9" \
| tr ' ' '\n' \
| aststatistics --mean
1.94600000000000e+01
$ echo " 19.8 20.1 20.5 19.9" \
| tr ' ' '\n' \
| aststatistics --mean
2.00750000000000e+01
This difference of 0.61 magnitudes (or roughly 1.75 times) is
significant (for the definition of magnitudes in astronomy, see *note
Brightness flux magnitude::). In the simple example above, you can
visually identify the "outlier" and manually remove it. But in most
common situations you will not be so lucky! For example when you want
to stack the five images of the five exposures above, and each image has
$4000\times4000$ (or 16 million!) pixels and not possible by hand in a
reasonable time (an average human's lifetime!).
This tutorial reviews the effect of outliers and different available
ways to remove them. In particular, we will be looking at stacking of
multiple datasets and collapsing one dataset along one of its
dimensions. But the concepts and methods are applicable to any analysis
that is affected by outliers.
* Menu:
* Building inputs and analysis without clipping:: Building a dataset for demonstration below.
* Sigma clipping:: Standard deviation (STD) clipping.
* MAD clipping:: Median Absolute Deviation (MAD) clipping.
* Filled re-clipping:: Two clips with holes filled in the middle.
File: gnuastro.info, Node: Building inputs and analysis without clipping, Next: Sigma clipping, Prev: Clipping outliers, Up: Clipping outliers
2.10.1 Building inputs and analysis without clipping
----------------------------------------------------
As described in *note Clipping outliers::, the goal of this tutorial is
to demonstrate the effects of outliers and show how to "clip" them from
basic statistics measurements. This is best done on an actual dataset
(rather than pure theory). In this section we will build nine noisy
images with the script below, such that one of the images has a circle
in the middle. We will then stack the 9 images into one final image
based on different statistical measurements: the mean, median, standard
deviation (STD), median absolute deviation (MAD) and number of inputs
used in each pixel. We will then analyze the resulting stacks to
demonstrate the problem with outliers.
Put the script below into a plain-text file (assuming it is called
‘script.sh’), and run it with ‘bash ./script.sh’. For more on writing
and good practices in shell scripts, see *note Writing scripts to
automate the steps::. The last command of the script above calls DS9 to
visualize the five output stacked images mentioned above.
# Constants
list=""
sigma=10
number=9
radius=30
width=201
bdir=build
profsum=3e5
background=10
random_seed=1699270427
# Clipping parameters (will be set later when we start clipping).
# clip_multiple: 3 for sigma; 4.48 for MAD
# clip_tolerance: 0.1 for sigma; 0.01 for MAD
clip_operator=""
clip_multiple=""
clip_tolerance=""
# Stop if there is any error.
set -e
# If the build directory does not exist, build it.
if ! [ -d $bdir ]; then mkdir $bdir; fi
# The final image (with largest number) will contain the outlier:
# we'll put a flat circle in the center of the image as the outlier
# structure.
outlier=$bdir/in-$number.fits
nn=$bdir/$number-no-noise.fits
export GSL_RNG_SEED=$random_seed
center=$(echo $width | awk '{print int($1/2)+1}')
echo "1 $center $center 5 $radius 0 0 1 $profsum 1" \
| astmkprof --mode=img --mergedsize=$width,$width \
--oversample=1 --output=$nn --mcolissum
astarithmetic $nn $background + $sigma mknoise-sigma \
--envseed -o$outlier
# Build pure noise and add elements to the list of images to stack.
list=$outlier
numnoise=$(echo $number | awk '{print $1-1}')
for i in $(seq 1 $numnoise); do
img="$bdir/in-$i.fits"
if ! [ -f $img ]; then
export GSL_RNG_SEED=$(echo $random_seed | awk '{print $1+'$i'}')
astarithmetic $width $width 2 makenew float32 $background + \
$sigma mknoise-sigma --envseed --output=$img
fi
list="$list $img"
done
# Stack the images,
for op in mean median std mad number; do
if [ x"$clip_operator" = x ]; then
out=$bdir/stack-$op.fits
astarithmetic $list $number $op -g1 --output=$out
else
operator=$clip_operator-$op
out=$bdir/stack-$operator.fits
astarithmetic $list $number $clip_multiple $clip_tolerance \
$operator -g1 --output=$out
fi
done
# Collapse the first and last image along the 2nd dimension.
for i in 1 $number; do
if [ x"$clip_operator" = x ]; then
out=$bdir/collapsed-$i.fits
astarithmetic $bdir/in-$i.fits 2 collapse-median counter \
--writeall --output=$out
else
out=$bdir/collapsed-$clip_operator-$i.fits
astarithmetic $bdir/in-$i.fits $clip_multiple $clip_tolerance \
2 collapse-$clip_operator-median counter \
--writeall --output=$out
fi
done
After the script finishes, you can see the generated input images with
the first command below. The second command shows the stacked images.
$ astscript-fits-view build/in-*.fits --ds9extra="-lock scalelimits yes"
$ astscript-fits-view build/stack-*.fits
Color-blind readers may not clearly see the issue in the opened
images with this color bar. In this case, please choose the "color"
menu at the top of the DS9 and select "gray" or any other color that
makes the circle most visible.
The effect of an outlier on the different measurements above can be
visually seen (and quantitatively measured) through the visibility of
the circle (that was only present in one image, of nine). Let's look at
them one by one (from the one that is most affected to the least):
‘std.fits’
The standard deviation (third image in DS9) is the most strongly
affected statistic by an outlier. This is so strong that the edge
of the circle is also clearly visible! The standard deviation is
calculated by first finding the mean, and estimating the difference
of each element from the mean. Those differences are then taken to
the power of two and finally the square root is taken (after a
division by the number). It is the power-of-two component that
amplifies the effect of the single outlier as you see here.
‘mean.fits’
The mean (first image in DS9) is also affected by the outlier in
such a way that the circle footprint is clearly visible. This is
because the nine images have the same importance in the combination
with a simple mean. Therefore, the outlier value pushes the result
to higher values and the circle is printed.
‘median.fits’
The median (second image in DS9) is also affected by the outlier;
although much less significantly than the standard deviation or
mean. At first sight the circle may not be too visible! To see it
more clearly, click on the "Analysis" menu in DS9 and then the
"smooth" item. After smoothing, you will see how the single
outlier has leaked into the median stack.
Intuitively, we would think that since the median is calculated
from the middle element after sorting, the outlier goes to the end
and won't affect the result. However, this is not the case as we
see here: with 9 elements, the "central" element is the 5th
(counting from 1; after sorting). Since the pixels covered by the
circle only have 8 pure noise elements; the "true" median should
have been the average of the 4th and 5th elements (after sorting).
By definition, the 5th element is always larger than the mean of
the 4th and 5th (because the 4th element after sorting has a
smaller value than the 5th element). Therefore, using the 5th
element (after sorting), we are systematically choosing higher
noise values in regions that are covered by the circle!
With larger datasets, the difference between the central elements
will be less. However, the improved precision (in the elements
without an outlier) will also be more. A detailed analysis of the
effect of a single outlier on the median based on the number of
inputs can be done as an exercise; but in general, as this argument
shows, the median is not immune to outliers; especially when you
care about low signal-to-noise regimes (as we do in astronomy: low
surface brightness science).
‘mad.fits’
The median absolute deviation (fourth image in DS9) is affected by
outliers in a similar fashion to the median.
‘number.fits’
The number image (last image in DS9) shows the number of images
that went into each pixel. Since we haven't rejected any outliers
(yet!), all the pixels in this image have a value of 9.
The example above included a single outlier. But we are not usually
that lucky: there are usually more outliers! For example, the last
command in the script above collapsed ‘1.fits’ (that was pure noise,
without the circle) and ‘9.fits’ (with the circle) along their second
dimension (the vertical). Collapsing was done by taking the median
along all the pixels in the vertical dimension. The output of
collapsing has one less dimension; in this case, producing a 1D table
(with the same number of rows as the image's horizontal axis). To
easily plot the output afterwards, we have also used the ‘counter’
operator. With the command below, you can open both tables and compare
them:
$ astscript-fits-view build/collapsed-*.fits
The last command opens TOPCAT. In the "Graphics" menu, select plane
plot and you will see all the values fluctuating around 10 (with a
maximum/minimum around $\pm2$). Afterwards, click on the "Layers" menu
and click on "Add position control". In the opened tab at the bottom
(where the scroll bar in front of "Table" is empty), select the other
table. In the regions that there was no circle in any of the vertical
axes, the two match nicely (the noise level is the same). However, you
see that the image columns that were partly covered by the outlying
circle gradually get more affected as the width of the circle in that
column increases (the full diameter of the circle was in the middle of
the image). This shows how the median is biased by outliers as their
number increases.
To see the problem more prominently, use the ‘collapse-mean’ operator
instead of the median. The reason that the mean is more strongly
affected by the outlier is exactly the same as above for the stacking of
the input images. In the subsections below, we will describe some of
the basic ways to reject the effect of these outliers (and have better
stacks or collapses). But the methodology is not limited to these types
of data and can be generically applied; unless specified explicitly.
File: gnuastro.info, Node: Sigma clipping, Next: MAD clipping, Prev: Building inputs and analysis without clipping, Up: Clipping outliers
2.10.2 Sigma clipping
---------------------
Let's assume that you have pure noise (centered on zero) with a clear
Gaussian distribution
(https://en.wikipedia.org/wiki/Normal_distribution), or see *note Photon
counting noise::. Now let's assume you add very bright objects (signal)
on the image which have a very sharp boundary. By a sharp boundary, we
mean that there is a clear cutoff (from the noise) at the pixels the
objects finish. In other words, at their boundaries, the objects do not
fade away into the noise.
In optical astronomical imaging, cosmic rays (when they collide at a
near normal incidence angle) are a very good example of such outliers.
The tracks they leave behind in the image are perfectly immune to the
blurring caused by the atmosphere on images of stars or galaxies and
they have a very high signal-to-noise ratio. They are also very
energetic and so their borders are usually clearly separated from the
surrounding noise. See Figure 15 in Akhlaghi and Ichikawa, 2015
(https://arxiv.org/abs/1505.01664).
In such a case, when you plot the histogram (see *note Histogram and
Cumulative Frequency Plot::) of the distribution, the pixels relating to
those objects will be clearly separate from pixels that belong to parts
of the image that did not have any signal (were just noise). In the
cumulative frequency plot, after a steady rise (due to the noise), you
would observe a long flat region were for a certain range of data
(horizontal axis), there is no increase in the index (vertical axis).
In the previous section (*note Building inputs and analysis without
clipping::) we created one such dataset (‘9.fits’). With the command
below, let's have a look at its histogram and cumulative frequency plot
(in simple ASCII format; we are decreasing the default number of bins
with ‘--numasciibins’ to show them easily within the width of the print
version of this manual; feel free to change this).
$ aststatistics build/in-9.fits --asciihist --asciicfp \
--numasciibins=65
ASCII Histogram:
Number: 40401
Y: (linear: 0 to 4191)
X: (linear: -31.9714 -- 150.323, in 65 bins)
| **
| ****
| ******
| ******
| ********
| ********
| **********
| ************
| **************
| ****************** ******
|******************************* *************************
|-----------------------------------------------------------------
ASCII Cumulative frequency plot:
Y: (linear: 0 to 40401)
X: (linear: -31.9714 -- 150.323, in 65 bins)
| ***************
| **********************************************
| ***********************************************
| *************************************************
| **************************************************
| ***************************************************
| ****************************************************
| *****************************************************
| ******************************************************
| ********************************************************
|*****************************************************************
|-----------------------------------------------------------------
Outliers like the example above can significantly bias the
measurement of the background noise statistics. For example let's
compare the median, mean and standard deviation of the image above with
‘1.fits’:
$ aststatistics build/in-1.fits --median --mean --std
9.90529778313248e+00 9.96143102101206e+00 1.00137568561776e+01
$ aststatistics build/in-9.fits --median --mean --std
1.09305819367634e+01 1.74470443173776e+01 2.88895986970341e+01
The effect of the outliers is obvious in all three measures: the
median has become 1.10 times larger, the mean 1.75 times and the
standard deviation about 2.88 times! The differing effect of outliers
in different statistics was already discussed in *note Building inputs
and analysis without clipping::; also see *note Quantifying signal in a
tile::.
$\sigma$-clipping is one way to remove/clip the effect of such very
strong outliers in measures like the above. $\sigma$-clipping is
defined as the very simple iteration below. In each iteration, the
range of input data might decrease. When the outliers are as strong as
above, the outliers will be removed through this iteration.
1. Calculate the standard deviation ($\sigma$) and median ($m$) of a
distribution. The median is used because, as shown above, the mean
is too significantly affected by the presence of outliers.
2. Remove all points that are smaller or larger than
$m\pm\alpha\sigma$.
3. Go back to step 1, unless the selected exit criteria is reached.
There are commonly two types of exit criteria (to stop the
$\sigma$-clipping iteration). Within Gnuastro's programs that use
sigma-clipping, the exit criteria is the second value to the
‘--sclipparams’ option (the first value is the $\alpha$ above):
• When a certain number of iterations has taken place (exit
criteria is an integer, larger than 1).
• When the new measured standard deviation is within a certain
tolerance level of the previous iteration (exit criteria is
floating point and less than 1.0). The tolerance level is
defined by:
$$\sigma_{old}-\sigma_{new} \over \sigma_{new}$$
In each clipping, the dispersion in the distribution is either
less or equal. So $\sigma_{old}\geq\sigma_{new}$.
Let's see the algorithm in practice with the ‘--sigmaclip’ option of
Gnuastro's Statistics program (using the default configuration of
$3\sigma$ clipping and tolerance of 0.1):
$ aststatistics build/in-9.fits --sigmaclip
Statistics (GNU Astronomy Utilities) 0.22
-------
Input: build/in-9.fits (hdu: 1)
-------
3-sigma clipping steps until relative change in STD is less than 0.1:
round number median STD
1 40401 1.09306e+01 2.88896e+01
2 37660 1.00306e+01 1.07153e+01
3 37539 1.00080e+01 9.93741e+00
-------
Statistics (after clipping):
Number of input elements: 40401
Number of clips: 2
Final number of elements: 37539
Median: 1.000803e+01
Mean: 1.001822e+01
Standard deviation: 9.937410e+00
Median Absolute Deviation: 6.772760e+00
After the basic information about the input and settings, the
Statistics program has printed the information for each round
(iteration) of clipping. Initially, there was 40401 elements (the image
is $201\times201$ pixels). After the first round of clipping, only
37660 elements remained and because the difference in standard deviation
was larger than the tolerance level, a third clipping was one. But the
change in standard deviation after the third clip was smaller than the
tolerance level, so the exit criteria was activated and the clipping
finished with 37539 elements. In the end, we see that the final median,
mean and standard deviation are very similar to the data without any
outlier (‘build/1.fits’ in the example above).
The example above provided a single statistic from a single dataset.
Other scenarios where sigma-clipping becomes necessary are stacking and
collapsing (that was the main goal of the script in *note Building
inputs and analysis without clipping::). To generate $\sigma$-clipped
stacks and collapsed tables, you just need to change the values of the
three variables of the script (shown below). After making this change
in your favorite text editor, have a look at the outputs:
$ grep ^clip_ script.sh
clip_operator=sigclip # These variables will be used more
clip_multiple=3 # effectively with the clipping
clip_tolerance=0.1 # operators of the next sections.
$ bash ./script.sh
$ astscript-fits-view build/stack-std.fits \
build/stack-sigclip-std.fits \
build/stack-*mean.fits \
build/stack-*median.fits \
build/stack-*number.fits \
--ds9extra="-tile grid layout 2 4 -scale minmax"
It is clear that the $\sigma$-clipped results have significantly
improved in all four measures (images in the right column in DS9). The
reason is clear in the ‘stack-sigclip-number.fits’ image (which show how
many images were used for each pixel): almost all of the outlying circle
has been accounted for (the pixel values are 8, not 9, showing 8 images
went into those). It is the leaked holes in the
‘stack-sigclip-number.fits’ image (with value of 9) that keep the circle
in the final stack of the other measures (at various levels). See *note
Filled re-clipping:: for stacking operators that can account for this.
So far, $\sigma$-clipping has preformed nicely. However, there are
important caveats to $\sigma$-clipping that are listed in the box below
and further elaborated (with examples) afterwards.
*Caveats of $\sigma$-clipping*: There are some important caveats to
$\sigma$-clipping:
• The standard deviation is itself heavily influenced by the presence
of outliers. Therefore a sufficiently small number of outliers can
expand the standard deviation such that they stay within the
boundaries.
• When the outliers do not constitute a clearly distinct distribution
like the example here, sigma-clipping will not be able to separate
them like here.
To demonstrate how weaker outliers will not be clipped in sigma
clipping, let's decrease the total sum of values in the outlying circle,
then re-run the script:
$ grep ^profsum script.sh
profsum=1e5
$ bash ./script.sh
Let's have a look at the new outlying circle with the first command
below. With the second command, let's view its pixel value histogram
(recall that previously, the circle had a clearly separate
distribution):
$ astscript-fits-view build/in-9.fits
$ aststatistics build/in-9.fits --asciihist --numasciibins=65
ASCII Histogram:
Number: 40401
Y: (linear: 0 to 2654)
X: (linear: -31.9714 -- 79.4266, in 65 bins)
| **
| *****
| *********
| **********
| *************
| **************
| *****************
| *******************
| ***********************
| ****************************************
|*****************************************************************
|-----------------------------------------------------------------
We see that even tough the circle is still clearly visible in the
noise, the histogram is not longer separate; it has blended into the
noise, and just caused a skewness in the otherwise symmetric noise
distribution. Let's try running the ‘--sigmaclip’ option as above:
$ aststatistics build/in-9.fits --sigmaclip
Statistics (GNU Astronomy Utilities) 0.22
-------
Input: build/in-9.fits (hdu: 1)
-------
3-sigma clipping steps until relative change in STD is less than 0.1:
round number median STD
1 40401 1.09295e+01 1.34784e+01
2 39618 1.06762e+01 1.19852e+01
3 39126 1.05265e+01 1.12983e+01
-------
Statistics (after clipping):
Number of input elements: 40401
Number of clips: 2
Final number of elements: 39126
Median: 1.052652e+01
Mean: 1.114819e+01
Standard deviation: 1.129831e+01
Median Absolute Deviation: 7.106166e+00
We see that the median, mean and standard deviation are over
estimated (each worse than the previous!). Let's see how the
$\sigma$-clipping stacking on this outlying flat circle:
$ astscript-fits-view build/stack-std.fits \
build/stack-sigclip-std.fits \
build/stack-*mean.fits \
build/stack-*median.fits \
build/stack-*number.fits \
--ds9extra="-tile grid layout 2 4 -scale minmax"
Compared to the previous run (where the outlying circle was
brighter), we see that $\sigma$-clipping is now less successful in
removing the outlying circle from the stacks; or in the single value
measurements. This is particularly visible in the
‘stack-sigclip-number.fits’ image: the circle barely visible any more:
there is only a very weak clustering of pixels with a value of 8 over
the circle's pixels. This has happened because the outliers have biased
the standard deviation itself to a level that includes them with this
multiple of the standard deviation.
To gauge if $\sigma$-clipping will be useful for your dataset, look
at the histogram (see *note Histogram and Cumulative Frequency Plot::).
The ASCII histogram that is printed on the command-line with
‘--asciihist’ (like above) is good enough in most cases. But you can't
do this manually in every case (as in the stacking which involved more
than forty thousand pixels)! Clipping outliers should be based on a
measure of scatter that is less affected by outliers! Therefore, in
Gnuastro we also have median absolute deviation (MAD) clipping which is
described in the next section (*note MAD clipping::).
File: gnuastro.info, Node: MAD clipping, Next: Filled re-clipping, Prev: Sigma clipping, Up: Clipping outliers
2.10.3 MAD clipping
-------------------
When clipping outliers, it is important that the used measure of
dispersion is itself not strongly affected by the outliers. Previously
(in *note Sigma clipping::), we saw that the standard deviation is not a
good measure of dispersion because of its strong dependency on outliers.
In this section, we'll introduce clipping operators that are based on
the median absolute deviation
(https://en.wikipedia.org/wiki/Median_absolute_deviation) (MAD).
The median absolute deviation is defined as the median of the
differences of each element from the median of the elements. As
mathematically derived in the Wikipedia page above, for a pure Gaussian
distribution, the median absolute deviation will be roughly
$0.67449\sigma$. We can confirm this numerically from the images with
pure noise that we created previously in *note Building inputs and
analysis without clipping::. With the first command below we can see
the raw standard deviation and median absolute deviation values and the
second command shows their division:
$ aststatistics build/in-1.fits --std --mad
1.00137568561776e+01 6.74662296703343e+00
$ aststatistics build/in-1.fits --std --mad | awk '{print $2/$1}'
0.673735
The algorithm of MAD-clipping is identical to $\sigma$-clipping,
except that instead of $\sigma$, it uses the median absolute deviation.
Since the median absolute deviation is smaller than the standard
deviation by roughly 0.67, if you regularly use $3\sigma$ there, you
should use $(3/0.67)\rm{MAD}=(4.48)\rm{MAD}$ when doing MAD-clipping.
The usual tolerance should also be changed due to the differing nature
of the median absolute deviation (based on sorted differences) in
relation to the standard deviation (based on the sum of squared
differences). A tolerance of 0.01 is better suited to the termination
criteria of MAD-clipping.
To demonstrate the steps in practice, let's assume you have the
original script in *note Building inputs and analysis without clipping::
with the changes shown in the first command below. With the second
command we'll execute the script, and with the third command we'll do
the iterations of MAD-clipping:
$ grep '^clip_\|^profsum' script.sh
profsum=1e5
clip_operator=madclip
clip_multiple=4.48
clip_tolerance=0.01
$ bash ./script.sh
$ aststatistics build/in-9.fits --madclip
Statistics (GNU Astronomy Utilities) 0.21.6-28a1
-------
Input: build/in-9.fits (hdu: 1)
-------
4.48-MAD clipping steps until relative change in MAD
(median absolute deviation) is less than 0.01:
round number median MAD
1 40401 1.09295e+01 7.38609e+00
2 38789 1.04261e+01 7.03508e+00
3 38549 1.03469e+01 6.97927e+00
-------
Statistics (after clipping):
Number of input elements: 40401
Number of clips: 2
Final number of elements: 38549
Median: 1.034690e+01
Mean: 1.068946e+01
Standard deviation: 1.062083e+01
Median Absolute Deviation: 6.979274e+00
We see that the median, mean and standard deviation after
MAD-clipping is much better than the basic $\sigma$-clipping (see *note
Sigma clipping::): the median is now 10.3 (was 10.5 in
$\sigma$-clipping), mean is 10.7 (was 10.11) and the standard deviation
is 10.6 (was 10.12).
Let's compare the MAD-clipped stacks with the results of the previous
section. Since we want the images shown in a certain order, we'll first
construct the list of images (with a ‘for’ loop that will fill the
‘imgs’ variable). Note that this assumes you have ran and carefully
read/understand all the commands in the previous sections (*note
Building inputs and analysis without clipping:: and *note Sigma
clipping::). Tip: the three ‘--ds9extra’ options ensure that the bottom
row (showing the number of images used in each pixel) has the same scale
and limits in all three columns.
$ imgs=""
$ p=build/stack # 'p' is short for "prefix"
$ for m in std mean median mad number; do \
imgs="$imgs $p-$m.fits $p-sigclip-$m.fits $p-madclip-$m.fits"; \
done
$ astscript-fits-view $imgs --ds9extra="-tile grid layout 3 5" \
--ds9extra="-scale limits 1 9" \
--ds9extra="-frame move back -scale limits 1 9" \
--ds9extra="-frame move back -scale limits 1 9"
The third column shows the newly created MAD-clipped stacks. We see
that the outlying circle is much more weaker in the MAD-clipped stacks
than in the $\sigma$-clipped stacks in all measures (except for the
"number" measure where the circle should be stronger).
However, unfortunately even MAD-clipping is not perfect and we still
see the circle in all four cases, even with the MAD-clipped median (more
clearly: after smoothing/blocking). The reason is similar to what was
described in $\sigma$-clipping (using the original ‘profsum=3e5’: the
leaked holes in the numbers image. Because the circle is not too
distant from the noise some of its elements do not get clipped, and
their stacked value gets systematically higher than the rest of the
image. In Gnuastro, we have a fix for this that is described fully in
the next section (*note Filled re-clipping::).
File: gnuastro.info, Node: Filled re-clipping, Prev: MAD clipping, Up: Clipping outliers
2.10.4 Filled re-clipping
-------------------------
When source of the outlier covers more than one element, and its flux is
close to the noise level, not all of its elements will be clipped:
because of noise, some of its elements will remain un-clipped; and thus
affecting the output. Examples of this were created and thoroughly
discussed in previous sections with $\sigma$-clipping and MAD-clipping
(see *note Sigma clipping:: and *note MAD clipping::).
To avoid this problem, in Gnuastro we have an extra set of clipping
operators that will do two rounds of clips and with some basic
operations in the middle.
1. The requested clipping is first applied. This will the return the
median and dispersion measure (MAD or STD).
2. A mask is created for each input image (in stacking) or 1D array
(in collapsing). Any pixel that is outside the requested clip
range is set to 1; the rest are set to 0.
3. Isolated regions are found:
• For 2D images (were each pixel has 8 neighbors) the mask
pixels are first dilated (so the edges of the regions are
closed off from the surrounding noise).
• For 1D arrays (where each element only has two neighbors), the
mask is first eroded. This is necessary because the next step
(where the holes are filled), two pixels that have been
clipped purely due to noise with a large distance between them
can wrongly mask a very large range of the input data.
4. Any 0-valued pixel in the masks that are fully surrounded by 1s (or
"holes") are filled (given a value of 1).
5. All the pixels that have a value of 1 in the mask are set to NaN in
the respective input data (that the mask corresponds to).
6. The requested clipping is repeated on the newly masked inputs.
Through this process, the less significant outliers (which do not get
clipped independently) are clipped based on their surrounding elements.
The filled re-clipping operators have an extra ‘-fill’ in their names.
For example the filled MAD-clipped mean is called ‘madclip-fill-mean’
(while the simple MAD-clipped mean operator was called ‘madclip-mean’).
Let's run our script with the filled $\sigma$-clipping and
$MAD$-clipping (before each run, make sure the values shown under the
‘grep’ command are correct).
With the last command, we'll view all the outputs generated so far
(in this section and the previous ones (*note Building inputs and
analysis without clipping::, *note Sigma clipping:: and *note MAD
clipping::):
$ grep '^clip_\|^profsum' script.sh
profsum=1e5
clip_operator=madclip-fill
clip_multiple=4.48
clip_tolerance=0.01
$ bash ./script
$ $ grep '^clip_\|^profsum' script.sh
profsum=1e5
clip_operator=sigclip-fill
clip_multiple=3
clip_tolerance=0.1
$ bash ./script
$ imgs=""
$ for m in std mean median mad number; do \
imgs="$imgs $p-$m.fits $p-sigclip-$m.fits $p-sigclip-fill-$m.fits" \
imgs="$p-madclip-$m.fits $p-madclip-fill-$m.fits"; \
done
$ astscript-fits-view $imgs --ds9extra="-tile grid layout 5 6" \
--ds9extra="-scale limits 1 9" \
--ds9extra="-frame move back -scale limits 1 9" \
--ds9extra="-frame move back -scale limits 1 9" \
--ds9extra="-frame move back -scale limits 1 9" \
--ds9extra="-frame move back -scale limits 1 9"
The last column (belonging to the ‘madclip-fill-*’ operators) is
_finally_ free of the outlying circle (that was present in only one of
nine inputs). The filling operation did not affect the ‘sigclip-fill-*’
operators (third column DS9 from the last command above)! The reason is
clear from the bottom row (showing the number of images used in each
pixel). The weak over density of clipped pixels over the circle is
barely visible and was not strong enough for defining "holes" (that will
be filled). On the contrary, when comparing the ‘madclip-number.fits’
and ‘madclip-fill-number.fits’, the filled holes within the circle are
clearly visible.
But the script also generated collapsed columns of ‘build/in-9.fits’
(to a 1D table). In this case, for each column, the number of outliers
increase as we enter the circle and reach a maximum in the middle of the
image. Let's have a look at those outputs:
$ astscript-fits-view build/collapsed-madclip-9.fits \
build/collapsed-madclip-fill-9.fits
When comparing the two in TOPCAT (following the same process
described in *note Building inputs and analysis without clipping::) you
will notice that the difference is only in the edges of the circle. The
4.48 multiple of MAD-clipping (corresponding to 3 sigma), was not
successful in removing the many outlying pixels due to the circle in the
central pixels of the image.
This is a relatively high threshold and was used because for the
images, we only had 9 elements in each clipping for every pixel. But
for the collapsing, we have many more pixels in each vertical direction
of the image (201 pixels). Let's decrease the threshold to 3 and
calculate the collapsed mean after MAD-clipping, once with filled
re-clipping and once without it:
$ for m in mean number; do \
for clip in madclip madclip-fill; do \
astarithmetic build/in-9.fits 3 0.01 2 collapse-$clip-$m \
counter --writeall -ocollapse-$clip-$m.fits; \
done; \
done
The two loops above created four tables. First, with the command
below, let's look at the two measured mean values (one with filling and
the other without it):
$ astscript-fits-view collapse-*-mean.fits
In the table without filled re-clipping, you see a small shift in the
center of the image (around 100 in the horizontal axis). Let's have a
look at the final number of pixels used in each clipping:
$ astscript-fits-view collapse-*-number.fits
The difference is now clearly visible when you plot both in one
"Plane plot" window. In the filled re-clipping case, we see a clear dip
in the number of pixels that very nicely corresponds to the number of
pixels associated to the circle. But the dip is much more noisy in the
simple MAD-clipping.
File: gnuastro.info, Node: Installation, Next: Common program behavior, Prev: Tutorials, Up: Top
3 Installation
**************
The latest released version of Gnuastro source code is always available
at the following URL:
<http://ftpmirror.gnu.org/gnuastro/gnuastro-latest.tar.gz>
*note Quick start:: describes the commands necessary to configure,
build, and install Gnuastro on your system. This chapter will be useful
in cases where the simple procedure above is not sufficient, for
example, your system lacks a mandatory/optional dependency (in other
words, you cannot pass the ‘$ ./configure’ step), or you want greater
customization, or you want to build and install Gnuastro from other
random points in its history, or you want a higher level of control on
the installation. Thus if you were happy with downloading the tarball
and following *note Quick start::, then you can safely ignore this
chapter and come back to it in the future if you need more
customization.
*note Dependencies:: describes the mandatory, optional and
bootstrapping dependencies of Gnuastro. Only the first group are
required/mandatory when you are building Gnuastro using a tarball (see
*note Release tarball::), they are very basic and low-level tools used
in most astronomical software, so you might already have them installed,
if not they are very easy to install as described for each. *note
Downloading the source:: discusses the two methods you can obtain the
source code: as a tarball (a significant snapshot in Gnuastro's
history), or the full history(1). The latter allows you to build
Gnuastro at any random point in its history (for example, to get bug
fixes or new features that are not released as a tarball yet).
The building and installation of Gnuastro is heavily customizable, to
learn more about them, see *note Build and install::. This section is
essentially a thorough explanation of the steps in *note Quick start::.
It discusses ways you can influence the building and installation. If
you encounter any problems in the installation process, it is probably
already explained in *note Known issues::. In *note Other useful
software:: the installation and usage of some other free software that
are not directly required by Gnuastro but might be useful in conjunction
with it is discussed.
* Menu:
* Dependencies:: Necessary packages for Gnuastro.
* Downloading the source:: Ways to download the source code.
* Build and install:: Configure, build and install Gnuastro.
---------- Footnotes ----------
(1) *note Bootstrapping dependencies:: are required if you clone the
full history.
File: gnuastro.info, Node: Dependencies, Next: Downloading the source, Prev: Installation, Up: Installation
3.1 Dependencies
================
A minimal set of dependencies are mandatory for building Gnuastro from
the standard tarball release. If they are not present you cannot pass
Gnuastro's configuration step. The mandatory dependencies are therefore
very basic (low-level) tools which are easy to obtain, build and
install, see *note Mandatory dependencies:: for a full discussion.
If you have the packages of *note Optional dependencies::, Gnuastro
will have additional functionality (for example, converting FITS images
to JPEG or PDF). If you are installing from a tarball as explained in
*note Quick start::, you can stop reading after this section. If you
are cloning the version controlled source (see *note Version controlled
source::), an additional bootstrapping step is required before
configuration and its dependencies are explained in *note Bootstrapping
dependencies::.
Your operating system's package manager is an easy and convenient way
to download and install the dependencies that are already pre-built for
your operating system. In *note Dependencies from package managers::,
we will list some common operating system package manager commands to
install the optional and mandatory dependencies.
* Menu:
* Mandatory dependencies:: Gnuastro will not install without these.
* Optional dependencies:: Adding more functionality.
* Bootstrapping dependencies:: If you have the version controlled source.
* Dependencies from package managers:: Installing from OS package managers.
File: gnuastro.info, Node: Mandatory dependencies, Next: Optional dependencies, Prev: Dependencies, Up: Dependencies
3.1.1 Mandatory dependencies
----------------------------
The mandatory Gnuastro dependencies are very basic and low-level tools.
They all follow the same basic GNU based build system (like that shown
in *note Quick start::), so even if you do not have them, installing
them should be pretty straightforward. In this section we explain each
program and any specific note that might be necessary in the
installation.
* Menu:
* GNU Scientific Library:: Installing GSL.
* CFITSIO:: C interface to the FITS standard.
* WCSLIB:: C interface to the WCS standard of FITS.
File: gnuastro.info, Node: GNU Scientific Library, Next: CFITSIO, Prev: Mandatory dependencies, Up: Mandatory dependencies
3.1.1.1 GNU Scientific Library
..............................
The GNU Scientific Library (http://www.gnu.org/software/gsl/), or GSL,
is a large collection of functions that are very useful in scientific
applications, for example, integration, random number generation, and
Fast Fourier Transform among many others. To download and install GSL
from source, you can run the following commands.
$ wget https://ftp.gnu.org/gnu/gsl/gsl-latest.tar.gz
$ tar -xf gsl-latest.tar.gz
$ cd gsl-X.X # Replace X.X with version number.
$ ./configure CFLAGS="$CFLAGS -g0 -O3"
$ make -j8 # Replace 8 with no. CPU threads.
$ make check
$ sudo make install
File: gnuastro.info, Node: CFITSIO, Next: WCSLIB, Prev: GNU Scientific Library, Up: Mandatory dependencies
3.1.1.2 CFITSIO
...............
CFITSIO (http://heasarc.gsfc.nasa.gov/fitsio/) is the closest you can
get to the pixels in a FITS image while remaining faithful to the FITS
standard (http://fits.gsfc.nasa.gov/fits_standard.html). It is written
by William Pence, the principal author of the FITS standard(1), and is
regularly updated. Setting the definitions for all other software
packages using FITS images.
Some GNU/Linux distributions have CFITSIO in their package managers,
if it is available and updated, you can use it. One problem that might
occur is that CFITSIO might not be configured with the
‘--enable-reentrant’ option by the distribution. This option allows
CFITSIO to open a file in multiple threads, it can thus provide great
speed improvements. If CFITSIO was not configured with this option, any
program which needs this capability will warn you and abort when you ask
for multiple threads (see *note Multi-threaded operations::).
To install CFITSIO from source, we strongly recommend that you have a
look through Chapter 2 (Creating the CFITSIO library) of the CFITSIO
manual and understand the options you can pass to ‘$ ./configure’ (they
are not too much). This is a very basic package for most astronomical
software and it is best that you configure it nicely with your system.
Once you download the source and unpack it, the following configure
script should be enough for most purposes. Do not forget to read
chapter two of the manual though, for example, the second option is only
for 64bit systems. The manual also explains how to check if it has been
installed correctly.
CFITSIO comes with two executable files called ‘fpack’ and ‘funpack’.
From their manual: they "are standalone programs for compressing and
uncompressing images and tables that are stored in the FITS (Flexible
Image Transport System) data format. They are analogous to the gzip and
gunzip compression programs except that they are optimized for the types
of astronomical images that are often stored in FITS format". The
commands below will compile and install them on your system along with
CFITSIO. They are not essential for Gnuastro, since they are just
wrappers for functions within CFITSIO, but they can come in handy. The
‘make utils’ command is only available for versions above 3.39, it will
build these executable files along with several other executable test
files which are deleted in the following commands before the
installation (otherwise the test files will also be installed).
The commands necessary to download the source, decompress, build and
install CFITSIO from source are described below.
$ urlbase=http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c
$ wget $urlbase/cfitsio_latest.tar.gz
$ tar -xf cfitsio_latest.tar.gz
$ cd cfitsio-X.XX # Replace X.XX with version
$ ./configure --prefix=/usr/local --enable-sse2 --enable-reentrant \
CFLAGS="$CFLAGS -g0 -O3"
$ make
$ make utils
$ ./testprog > testprog.lis # See below if this has an error
$ diff testprog.lis testprog.out # Should have no output
$ cmp testprog.fit testprog.std # Should have no output
$ rm cookbook fitscopy imcopy smem speed testprog
$ sudo make install
In the ‘./testprog > testprog.lis’ step, you may confront an error,
complaining that it cannot find ‘libcfitsio.so.AAA’ (where ‘AAA’ is an
integer). This is the library that you just built and have not yet
installed. But unfortunately some versions of CFITSIO do not account
for this on some OSs. To fix the problem, you need to tell your OS to
also look into current CFITSIO build directory with the first command
below, afterwards, the problematic command (second below) should run
properly.
$ export LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH"
$ ./testprog > testprog.lis
Recall that the modification above is ONLY NECESSARY FOR THIS STEP.
_Do not_ put the ‘LD_LIBRARY_PATH’ modification command in a permanent
place (like your bash startup file). After installing CFITSIO, close
your terminal and continue working on a new terminal (so
‘LD_LIBRARY_PATH’ has its default value). For more on
‘LD_LIBRARY_PATH’, see *note Installation directory::.
---------- Footnotes ----------
(1) Pence, W.D. et al. Definition of the Flexible Image Transport
System (FITS), version 3.0. (2010) Astronomy and Astrophysics, Volume
524, id.A42, 40 pp.
File: gnuastro.info, Node: WCSLIB, Prev: CFITSIO, Up: Mandatory dependencies
3.1.1.3 WCSLIB
..............
WCSLIB (http://www.atnf.csiro.au/people/mcalabre/WCS/) is written and
maintained by one of the authors of the World Coordinate System (WCS)
definition in the FITS standard
(http://fits.gsfc.nasa.gov/fits_standard.html)(1), Mark Calabretta. It
might be already built and ready in your distribution's package
management system. However, here the installation from source is
explained, for the advantages of installation from source please see
*note Mandatory dependencies::. To install WCSLIB you will need to have
CFITSIO already installed, see *note CFITSIO::.
WCSLIB also has plotting capabilities which use PGPLOT (a plotting
library for C). If you wan to use those capabilities in WCSLIB, *note
PGPLOT:: provides the PGPLOT installation instructions. However PGPLOT
is old(2), so its installation is not easy, there are also many great
modern WCS plotting tools (mostly in written in Python). Hence, if you
will not be using those plotting functions in WCSLIB, you can configure
it with the ‘--without-pgplot’ option as shown below.
If you have the cURL library (3) on your system and you installed
CFITSIO version 3.42 or later, you will need to also link with the cURL
library at configure time (through the ‘-lcurl’ option as shown below).
CFITSIO uses the cURL library for its HTTPS (or HTTP Secure(4)) support
and if it is present on your system, CFITSIO will depend on it.
Therefore, if ‘./configure’ command below fails (you do not have the
cURL library), then remove this option and rerun it.
To download, configure, build, check and install WCSLIB from source,
you can follow the steps below.
## Download and unpack the source tarball
$ wget ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2
$ tar -xf wcslib.tar.bz2
## In the `cd' command, replace `X.X' with version number.
$ cd wcslib-X.X
## If `./configure' fails, remove `-lcurl' and run again.
$ ./configure LIBS="-pthread -lcurl -lm" --without-pgplot \
--disable-fortran CFLAGS="$CFLAGS -g0 -O3"
$ make
$ make check
$ sudo make install
---------- Footnotes ----------
(1) Greisen E.W., Calabretta M.R. (2002) Representation of world
coordinates in FITS. Astronomy and Astrophysics, 395, 1061-1075.
(2) As of early June 2016, its most recent version was uploaded in
February 2001.
(3) <https://curl.haxx.se>
(4) <https://en.wikipedia.org/wiki/HTTPS>
File: gnuastro.info, Node: Optional dependencies, Next: Bootstrapping dependencies, Prev: Mandatory dependencies, Up: Dependencies
3.1.2 Optional dependencies
---------------------------
The libraries listed here are only used for very specific applications,
therefore they are optional and Gnuastro can be built without them (with
only those specific features disabled). Since these are pretty
low-level tools, they are not too hard to install from source, but you
can also use your operating system's package manager to easily install
all of them. For more, see *note Dependencies from package managers::.
If the ‘./configure’ script cannot find any of these optional
dependencies, it will notify you of the operation(s) you cannot do due
to not having them. If you continue the build and request an operation
that uses a missing library, Gnuastro's programs will warn that the
optional library was missing at build-time and abort. Since Gnuastro
was built without that library, installing the library afterwards will
not help. The only way is to rebuild Gnuastro from scratch (after the
library has been installed). However, for program dependencies (like
cURL or Ghostscript) things are easier: you can install them after
building Gnuastro also. This is because libraries are used to build the
internal structure of Gnuastro's executables. However, a program
dependency is called by Gnuastro's programs at run-time and has no
effect on their internal structure. So if a dependency program becomes
available later, it will be used next time it is requested.
GNU Libtool
Libtool is a program to simplify managing of the libraries to build
an executable (a program). GNU Libtool has some added
functionality compared to other implementations. If GNU Libtool is
not present on your system at configuration time, a warning will be
printed and *note BuildProgram:: will not be built or installed.
The configure script will look into your search path (‘PATH’) for
GNU Libtool through the following executable names: ‘libtool’
(acceptable only if it is the GNU implementation) or ‘glibtool’.
See *note Installation directory:: for more on ‘PATH’.
GNU Libtool (the binary/executable file) is a low-level program
that is probably already present on your system, and if not, is
available in your operating system package manager(1). If you want
to install GNU Libtool's latest version from source, please visit
its web page (https://www.gnu.org/software/libtool/).
Gnuastro's tarball is shipped with an internal implementation of
GNU Libtool. Even if you have GNU Libtool, Gnuastro's internal
implementation is used for the building and installation of
Gnuastro. As a result, you can still build, install and use
Gnuastro even if you do not have GNU Libtool installed on your
system. However, this internal Libtool does not get installed.
Therefore, after Gnuastro's installation, if you want to use *note
BuildProgram:: to compile and link your own C source code which
uses the *note Gnuastro library::, you need to have GNU Libtool
available on your system (independent of Gnuastro). See *note
Review of library fundamentals:: to learn more about libraries.
GNU Make extension headers
GNU Make is a workflow management system that can be used to run a
series of commands in a specific order, and in parallel if you
want. GNU Make offers special features to extend it with custom
functions within a dynamic library. They are defined in the
‘gnumake.h’ header. If ‘gnumake.h’ can be found on your system at
configuration time, Gnuastro will build a custom library that GNU
Make can use for extended functionality in (astronomical) data
analysis scenarios.
libgit2
Git is one of the most common version control systems (see *note
Version controlled source::). When ‘libgit2’ is present, and
Gnuastro's programs are run within a version controlled directory,
outputs will contain the version number of the working directory's
repository for future reproducibility. See the ‘COMMIT’ keyword
header in *note Output FITS files:: for a discussion.
libjpeg
libjpeg is only used by ConvertType to read from and write to JPEG
images, see *note Recognized file formats::. libjpeg
(http://www.ijg.org/) is a very basic library that provides tools
to read and write JPEG images, most Unix-like graphic programs and
libraries use it. Therefore you most probably already have it
installed. libjpeg-turbo (http://libjpeg-turbo.virtualgl.org/) is
an alternative to libjpeg. It uses Single instruction, multiple
data (SIMD) instructions for ARM based systems that significantly
decreases the processing time of JPEG compression and decompression
algorithms.
libtiff
libtiff is used by ConvertType and the libraries to read TIFF
images, see *note Recognized file formats::. libtiff
(http://www.simplesystems.org/libtiff/) is a very basic library
that provides tools to read and write TIFF images, most Unix-like
operating system graphic programs and libraries use it. Therefore
even if you do not have it installed, it must be easily available
in your package manager.
cURL
cURL's executable (‘curl’) is called by *note Query:: for
submitting queries to remote datasets and retrieving the results.
It is not necessary for the build of Gnuastro from source (only a
warning will be printed if it cannot be found at configure time),
so if you do not have it at build-time there is no problem. Just
be sure to have it when you run ‘astquery’, otherwise you'll get an
error about not finding ‘curl’.
GPL Ghostscript
GPL Ghostscript's executable (‘gs’) is called by ConvertType to
compile a PDF file from a source PostScript file, see *note
ConvertType::. Therefore its headers (and libraries) are not
needed.
Python3 with Numpy
Python is a high-level programming language and Numpy is the most
commonly used library within Python to add multi-dimensional arrays
and matrices. If you configure Gnuastro with ‘--with-python’ _and_
version 3 of Python is available with a corresponding Numpy
Library, Gnuastro's library will be built with some Python-related
helper functions. Python wrappers for Gnuastro's library (for
example, 'pyGnuastro') can use these functions when being built
from source. For more on Gnuastro's Python helper functions, see
*note Python interface::.
This Python interface is only relevant if you want to build the
Python wrappers (like 'pyGnuastro') from source. If you install
the Gnuastro Python wrapper from a pre-built repository like PyPI,
this feature of your Gnuastro library won't be used. Pre-built
libraries contain the full Gnuastro library that they need within
them (you don't even need to have Gnuastro at all!).
*Can't find the Python3 and Numpy of a virtual environment:* make
sure to set the ‘$PYTHON’ variable to point to the ‘python3’
command of the virtual environment before running ‘./configure’.
Note that you don't need to activate the virtual env, just point
‘PYTHON’ to its Python3 executable, like the example below:
$ python3 -m venv test-env # Setting up the virtual env.
$ export PYTHON="$(pwd)/test-env/bin/python3"
$ ./configure # Gnuastro's configure script.
SAO DS9
SAO DS9 (‘ds9’) is a visualization tool for FITS images.
Gnuastro's ‘astscript-fits-view’ program calls DS9 to visualize
FITS images. We have a full appendix on it and how to install it
in *note SAO DS9::. Since it is a run-time dependency, it can be
installed at any later time (after building and installing
Gnuastro).
TOPCAT
TOPCAT (‘topcat’) is a visualization tool for astronomical tables
(most commonly: plotting). Gnuastro's ‘astscript-fits-view’
program calls TOPCAT it to visualize tables. We have a full
appendix on it and how to install it in *note TOPCAT::. Since it
is a run-time dependency, it can be installed at any later time
(after building and installing Gnuastro).
---------- Footnotes ----------
(1) Note that we want the binary/executable Libtool program which can
be run on the command-line. In Debian-based operating systems which
separate various parts of a package, you want want ‘libtool-bin’, the
‘libtool’ package will not contain the executable program.
File: gnuastro.info, Node: Bootstrapping dependencies, Next: Dependencies from package managers, Prev: Optional dependencies, Up: Dependencies
3.1.3 Bootstrapping dependencies
--------------------------------
Bootstrapping is only necessary if you have decided to obtain the full
version controlled history of Gnuastro, see *note Version controlled
source:: and *note Bootstrapping::. Using the version controlled source
enables you to always be up to date with the most recent development
work of Gnuastro (bug fixes, new functionalities, improved algorithms,
etc.). If you have downloaded a tarball (see *note Downloading the
source::), then you can ignore this subsection.
To successfully run the bootstrapping process, there are some
additional dependencies to those discussed in the previous subsections.
These are low level tools that are used by a large collection of
Unix-like operating systems programs, therefore they are most probably
already available in your system. If they are not already installed,
you should be able to easily find them in any GNU/Linux distribution
package management system (‘apt-get’, ‘yum’, ‘pacman’, etc.). The short
names in parenthesis in ‘typewriter’ font after the package name can be
used to search for them in your package manager. For the GNU
Portability Library, GNU Autoconf Archive and TeX Live, it is
recommended to use the instructions here, not your operating system's
package manager.
GNU Portability Library (Gnulib)
To ensure portability for a wider range of operating systems (those
that do not include GNU C library, namely glibc), Gnuastro depends
on the GNU portability library, or Gnulib. Gnulib keeps a copy of
all the functions in glibc, implemented (as much as possible) to be
portable to other operating systems. The ‘bootstrap’ script can
automatically clone Gnulib (as a ‘gnulib/’ directory inside
Gnuastro), however, as described in *note Bootstrapping:: this is
not recommended.
The recommended way to bootstrap Gnuastro is to first clone Gnulib
and the Autoconf archives (see below) into a local directory
outside of Gnuastro. Let's call it ‘DEVDIR’(1) (which you can set
to any directory; preferentially where you keep your other
development projects). Currently in Gnuastro, both Gnulib and
Autoconf archives have to be cloned in the same top directory(2)
like the case here(3):
$ DEVDIR=/home/yourname/Development ## Select any location.
$ mkdir $DEVDIR ## If it doesn't exist!
$ cd $DEVDIR
$ git clone https://git.sv.gnu.org/git/gnulib.git
$ git clone https://git.sv.gnu.org/git/autoconf-archive.git
Gnulib is a source-based dependency of Gnuastro's bootstrapping
process, so simply having it is enough on your computer, there is
no need to install, and thus check anything.
You now have the full version controlled source of these two
repositories in separate directories. Both these packages are
regularly updated, so every once in a while, you can run ‘$ git
pull’ within them to get any possible updates.
GNU Automake (‘automake’)
GNU Automake will build the ‘Makefile.in’ files in each
sub-directory using the (hand-written) ‘Makefile.am’ files. The
‘Makefile.in’s are subsequently used to generate the ‘Makefile’s
when the user runs ‘./configure’ before building.
To check that you have a working GNU Automake in your system, you
can try this command:
$ automake --version
GNU Autoconf (‘autoconf’)
GNU Autoconf will build the ‘configure’ script using the
configurations we have defined (hand-written) in ‘configure.ac’.
To check that you have a working GNU Autoconf in your system, you
can try this command:
$ autoconf --version
GNU Autoconf Archive
These are a large collection of tests that can be called to run at
‘./configure’ time. See the explanation under GNU Portability
Library (Gnulib) above for instructions on obtaining it and keeping
it up to date.
GNU Autoconf Archive is a source-based dependency of Gnuastro's
bootstrapping process, so simply having it is enough on your
computer, there is no need to install, and thus check anything.
Just do not forget that it has to be in the same directory as
Gnulib (described above).
GNU Texinfo (‘texinfo’)
GNU Texinfo is the tool that formats this manual into the various
output formats. To bootstrap Gnuastro you need all of Texinfo's
command-line programs. However, some operating systems package
them separately, for example, in Fedora, ‘makeinfo’ is packaged in
the ‘texinfo-tex’ package.
To check that you have a working GNU Texinfo in your system, you
can try this command:
$ makeinfo --version
GNU Libtool (‘libtool’)
GNU Libtool is in charge of building all the libraries in Gnuastro.
The libraries contain functions that are used by more than one
program and are installed for use in other programs. They are thus
put in a separate directory (‘lib/’).
To check that you have a working GNU Libtool in your system, you
can try this command (and from the output, make sure it is GNU's
libtool)
$ libtool --version
GNU help2man (‘help2man’)
GNU help2man is used to convert the output of the ‘--help’ option
(*note --help::) to the traditional Man page (*note Man pages::).
To check that you have a working GNU Help2man in your system, you
can try this command:
$ help2man --version
LaTeX and some TeX packages
Some of the figures in this book are built by LaTeX (using the
PGF/TikZ package). The LaTeX source for those figures is version
controlled for easy maintenance not the actual figures. So the
‘./boostrap’ script will run LaTeX to build the figures. The best
way to install LaTeX and all the necessary packages is through TeX
live (https://www.tug.org/texlive/) which is a package manager for
TeX related tools that is independent of any operating system. It
is thus preferred to the TeX Live versions distributed by your
operating system.
To install TeX Live, go to the web page and download the
appropriate installer by following the "download" link. Note that
by default the full package repository will be downloaded and
installed (around 4 Gigabytes) which can take _very_ long to
download and to update later. However, most packages are not
needed by everyone, it is easier, faster and better to install only
the "Basic scheme" (consisting of only the most basic TeX and LaTeX
packages, which is less than 200 Mega bytes)(4).
After the installation, be sure to set the environment variables as
suggested in the end of the outputs. Any time you confront (need)
a package you do not have, simply install it with a command like
below (similar to how you install software from your operating
system's package manager)(5). To install all the necessary TeX
packages for a successful Gnuastro bootstrap, run this command:
$ sudo su
# tlmgr install epsf jknapltx caption biblatex biber iftex \
etoolbox logreq xstring xkeyval pgf ms \
xcolor pgfplots times rsfs ps2eps epspdf
To check that you have a working LaTeX executable in your system,
you can try this command (this just checks if LaTeX exists, as
described above, if you have a missing package, you can easily
identify it from the output and install it with ‘tlmgr’):
$ latex --version
ImageMagick (‘imagemagick’)
ImageMagick is a wonderful and robust program for image
manipulation on the command-line. ‘bootstrap’ uses it to convert
the book images into the formats necessary for the various book
formats.
Since ImageMagick version 7, it is necessary to edit the policy
file (‘/etc/ImageMagick-7/policy.xml’) to have the following line
(it maybe present, but commented, in this case un-comment it):
<policy domain="coder" rights="read|write" pattern="{PS,PDF,XPS}"/>
If the following line is present, it is also necessary to
comment/remove it.
<policy domain="delegate" rights="none" pattern="gs" />
To learn more about the ImageMagick security policy please see:
<https://imagemagick.org/script/security-policy.php>.
To check that you have a working ImageMagick in your system, you
can try this command:
$ convert --version
---------- Footnotes ----------
(1) If you are not a developer in Gnulib or Autoconf archives,
‘DEVDIR’ can be a directory that you do not backup. In this way the
large number of files in these projects will not slow down your backup
process or take bandwidth (if you backup to a remote server).
(2) If you already have the Autoconf archives in a separate
directory, or cannot clone it in the same directory as Gnulib, or you
have it with another directory name (not ‘autoconf-archive/’), you can
follow this short step. Set ‘AUTOCONFARCHIVES’ to your desired address.
Then define a symbolic link in ‘DEVDIR’ with the following command so
Gnuastro's bootstrap script can find it:
‘$ ln -s $AUTOCONFARCHIVES $DEVDIR/autoconf-archive’.
(3) If your internet connection is active, but Git complains about
the network, it might be due to your network setup not recognizing the
git protocol. In that case use the following URL for the HTTP protocol
instead (for Autoconf archives, replace the name):
‘http://git.sv.gnu.org/r/gnulib.git’
(4) You can also download the DVD iso file at a later time to keep as
a backup for when you do not have internet connection if you need a
package.
(5) After running TeX, or LaTeX, you might get a warning complaining
about a ‘missingfile’. Run '‘tlmgr info missingfile’' to see the
package(s) containing that file which you can install.
File: gnuastro.info, Node: Dependencies from package managers, Prev: Bootstrapping dependencies, Up: Dependencies
3.1.4 Dependencies from package managers
----------------------------------------
The most basic way to install a package on your system is to build the
packages from source yourself. Alternatively, you can use your
operating system's package manager to download pre-compiled files and
install them. The latter choice is easier and faster. However, we
recommend that you build the *note Mandatory dependencies:: yourself
from source (all necessary commands and links are given in the
respective section). Here are some basic reasons behind this
recommendation.
1. Your operating system's pre-built software might not be the most
recent release. For example, Gnuastro itself is also packaged in
some package managers. For the list see:
<https://repology.org/project/gnuastro/versions>. You will notice
that Gnuastro's version in some operating systems is more than 10
versions old! It is the same for all the dependencies of Gnuastro.
2. For each package, Gnuastro might preform better (or require)
certain configuration options that your distribution's package
managers did not add for you. If present, these configuration
options are explained during the installation of each in the
sections below (for example, in *note CFITSIO::). When the proper
configuration has not been set, the programs should complain and
inform you.
3. For the libraries, they might separate the binary file from the
header files which can cause confusion, see *note Known issues::.
4. Like any other tool, the science you derive from Gnuastro's tools
highly depend on these lower level dependencies, so generally it is
much better to have a close connection with them. By reading their
manuals, installing them and staying up to date with changes/bugs
in them, your scientific results and understanding (of what is
going on, and thus how you interpret your scientific results) will
also correspondingly improve.
Based on your package manager, you can use any of the following
commands to install the mandatory and optional dependencies. If your
package manager is not included in the list below, please send us the
respective command, so we add it. For better archivability and
compression ratios, Gnuastro's recommended tarball compression format is
with the Lzip (http://lzip.nongnu.org/lzip.html) program, see *note
Release tarball::. Therefore, the package manager commands below also
contain Lzip.
‘apt-get’ (Debian-based OSs: Debian, Ubuntu, Linux Mint, etc.)
Debian (https://en.wikipedia.org/wiki/Debian) is one of the oldest
GNU/Linux distributions(1). It thus has a very extended user
community and a robust internal structure and standards. All of it
is free software and based on the work of volunteers around the
world. Many distributions are thus derived from it, for example,
Ubuntu and Linux Mint. This arguably makes Debian-based OSs the
largest, and most used, class of GNU/Linux distributions. All of
them use Debian's Advanced Packaging Tool (APT, for example,
‘apt-get’) for managing packages.
Development features (Ubuntu or derivatives)
By default, a newly installed Ubuntu does not contain the
low-level tools that are necessary for building a software
from source. Therefore, if you are using Ubuntu, please run
the following command.
$ sudo apt-get install gcc make zlib1g-dev lzip
Mandatory dependencies
Without these, Gnuastro cannot be built, they are necessary
for input/output and low-level mathematics (see *note
Mandatory dependencies::)!
$ sudo apt-get install libgsl-dev libcfitsio-dev \
wcslib-dev
Optional dependencies
If present, these libraries can be used in Gnuastro's build
for extra features, see *note Optional dependencies::.
$ sudo apt-get install ghostscript libtool-bin \
libjpeg-dev libtiff-dev \
libgit2-dev curl
Programs to view FITS images or tables
These are not used in Gnuastro's build. They can just help in
viewing the inputs/outputs independent of Gnuastro!
$ sudo apt-get install saods9 topcat
Gnuastro is packaged (https://tracker.debian.org/pkg/gnuastro) in
Debian (and thus some of its derivate operating systems). Just
make sure it is the most recent version.
‘dnf’
‘yum’ (Red Hat-based OSs: Red Hat, Fedora, CentOS, Scientific Linux, etc.)
Red Hat Enterprise Linux (https://en.wikipedia.org/wiki/Red_Hat)
(RHEL) is released by Red Hat Inc. RHEL requires paid
subscriptions for use of its binaries and support. But since it is
free software, many other teams use its code to spin-off their own
distributions based on RHEL. Red Hat-based GNU/Linux distributions
initially used the "Yellowdog Updated, Modifier" (YUM) package
manager, which has been replaced by "Dandified yum" (DNF). If the
latter is not available on your system, you can use ‘yum’ instead
of ‘dnf’ in the command below.
Mandatory dependencies
Without these, Gnuastro cannot be built, they are necessary
for input/output and low-level mathematics (see *note
Mandatory dependencies::)!
$ sudo dnf install gsl-devel cfitsio-devel \
wcslib-devel
Optional dependencies
If present, these libraries can be used in Gnuastro's build
for extra features, see *note Optional dependencies::.
$ sudo dnf install ghostscript libtool \
libjpeg-devel libtiff-devel \
libgit2-devel lzip curl
Programs to view FITS images or tables
These are not used in Gnuastro's build. They can just help in
viewing the inputs/outputs independent of Gnuastro!
$ sudo dnf install saods9 topcat
‘brew’ (macOS)
macOS (https://en.wikipedia.org/wiki/MacOS) is the operating system
used on Apple devices. macOS does not come with a package manager
pre-installed, but several widely used, third-party package
managers exist, such as Homebrew or MacPorts. Both are free
software. Currently we have only tested Gnuastro's installation
with Homebrew as described below. If not already installed, first
obtain Homebrew by following the instructions at <https://brew.sh>.
Mandatory dependencies
Without these, Gnuastro cannot be built, they are necessary
for input/output and low-level mathematics (see *note
Mandatory dependencies::)!
Homebrew manages packages in different 'taps'. To install
WCSLIB via Homebrew you will need to ‘tap’ into
‘brewsci/science’ first (the tap may change in the future, but
can be found by calling ‘brew search wcslib’).
$ brew tap brewsci/science
$ brew install wcslib gsl cfitsio
Optional dependencies
If present, these libraries can be used in Gnuastro's build
for extra features, see *note Optional dependencies::.
$ brew install ghostscript libtool libjpeg \
libtiff libgit2 curl lzip
Programs to view FITS images or tables
These are not used in Gnuastro's build. They can just help in
viewing the inputs/outputs independent of Gnuastro!
$ brew install saoimageds9 topcat
‘pacman’ (Arch Linux)
Arch Linux (https://en.wikipedia.org/wiki/Arch_Linux) is a smaller
GNU/Linux distribution, which follows the KISS principle ("keep it
simple, stupid") as a general guideline. It "focuses on elegance,
code correctness, minimalism and simplicity, and expects the user
to be willing to make some effort to understand the system's
operation". Arch GNU/Linux uses "Package manager" (Pacman) to
manage its packages/components.
Mandatory dependencies
Without these, Gnuastro cannot be built, they are necessary
for input/output and low-level mathematics (see *note
Mandatory dependencies::)!
$ sudo pacman -S gsl cfitsio wcslib
Optional dependencies
If present, these libraries can be used in Gnuastro's build
for extra features, see *note Optional dependencies::.
$ sudo pacman -S ghostscript libtool libjpeg \
libtiff libgit2 curl lzip
Programs to view FITS images or tables
These are not used in Gnuastro's build. They can just help in
viewing the inputs/outputs independent of Gnuastro!
SAO DS9 and TOPCAT are not available in the standard Arch
GNU/Linux repositories. However, installing and using both is
very easy from their own web pages, as described in *note SAO
DS9:: and *note TOPCAT::.
‘zypper’ (openSUSE and SUSE Linux Enterprise Server)
SUSE Linux Enterprise Server(2) (SLES) is the commercial offering
which shares code and tools. Many additional packages are offered
in the Build Service(3). openSUSE and SLES use ‘zypper’ (cli) and
YaST (GUI) for managing repositories and packages.
Configuration
When building Gnuastro, run the configure script with the
following ‘CPPFLAGS’ environment variable:
$ ./configure CPPFLAGS="-I/usr/include/cfitsio"
Mandatory dependencies
Without these, Gnuastro cannot be built, they are necessary
for input/output and low-level mathematics (see *note
Mandatory dependencies::)!
$ sudo zypper install gsl-devel cfitsio-devel \
wcslib-devel
Optional dependencies
If present, these libraries can be used in Gnuastro's build
for extra features, see *note Optional dependencies::.
$ sudo zypper install ghostscript_any libtool \
pkgconfig libcurl-devel \
libgit2-devel \
libjpeg62-devel \
libtiff-devel curl
Programs to view FITS images or tables
These are not used in Gnuastro's build. They can just help in
viewing the inputs/outputs independent of Gnuastro!
$ sudo zypper install ds9 topcat
Usually, when libraries are installed by operating system package
managers, there should be no problems when configuring and building
other programs from source (that depend on the libraries: Gnuastro in
this case). However, in some special conditions, problems may pop-up
during the configuration, building, or checking/running any of
Gnuastro's programs. The most common of such problems and their
solution are discussed below.
*Not finding library during configuration:* If a library is installed,
but during Gnuastro's ‘configure’ step the library is not found, then
configure Gnuastro like the command below (correcting ‘/path/to/lib’).
For more, see *note Known issues:: and *note Installation directory::.
$ ./configure LDFLAGS="-L/path/to/lib"
*Not finding header (.h) files while building:* If a library is
installed, but during Gnuastro's ‘make’ step, the library's header (file
with a ‘.h’ suffix) is not found, then configure Gnuastro like the
command below (correcting ‘/path/to/include’). For more, see *note
Known issues:: and *note Installation directory::.
$ ./configure CPPFLAGS="-I/path/to/include"
*Gnuastro's programs do not run during check or after install:* If a
library is installed, but the programs do not run due to linking
problems, set the ‘LD_LIBRARY_PATH’ variable like below (assuming
Gnuastro is installed in ‘/path/to/installed’). For more, see *note
Known issues:: and *note Installation directory::.
$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/to/installed/lib"
---------- Footnotes ----------
(1)
<https://en.wikipedia.org/wiki/List_of_Linux_distributions#Debian-based>
(2) <https://www.suse.com/products/server>
(3) <https://build.opensuse.org>
File: gnuastro.info, Node: Downloading the source, Next: Build and install, Prev: Dependencies, Up: Installation
3.2 Downloading the source
==========================
Gnuastro's source code can be downloaded in two ways. As a tarball,
ready to be configured and installed on your system (as described in
*note Quick start::), see *note Release tarball::. If you want official
releases of stable versions this is the best, easiest and most common
option. Alternatively, you can clone the version controlled history of
Gnuastro, run one extra bootstrapping step and then follow the same
steps as the tarball. This will give you access to all the most recent
work that will be included in the next release along with the full
project history. The process is thoroughly introduced in *note Version
controlled source::.
* Menu:
* Release tarball:: Download a stable official release.
* Version controlled source:: Get and use the version controlled source.
File: gnuastro.info, Node: Release tarball, Next: Version controlled source, Prev: Downloading the source, Up: Downloading the source
3.2.1 Release tarball
---------------------
A release tarball (commonly compressed) is the most common way of
obtaining free and open source software. A tarball is a snapshot of one
particular moment in the Gnuastro development history along with all the
necessary files to configure, build, and install Gnuastro easily (see
*note Quick start::). It is very straightforward and needs the least
set of dependencies (see *note Mandatory dependencies::). Gnuastro has
tarballs for official stable releases and pre-releases for testing. See
*note Version numbering:: for more on the two types of releases and the
formats of the version numbers. The URLs for each type of release are
given below.
Official stable releases (<http://ftp.gnu.org/gnu/gnuastro>):
This URL hosts the official stable releases of Gnuastro. Always
use the most recent version (see *note Version numbering::). By
clicking on the "Last modified" title of the second column, the
files will be sorted by their date which you can also use to find
the latest version. It is recommended to use a mirror to download
these tarballs, please visit <http://ftpmirror.gnu.org/gnuastro/>
and see below.
Pre-release tarballs (<http://alpha.gnu.org/gnu/gnuastro>):
This URL contains unofficial pre-release versions of Gnuastro. The
pre-release versions of Gnuastro here are for enthusiasts to try
out before an official release. If there are problems, or bugs
then the testers will inform the developers to fix before the next
official release. See *note Version numbering:: to understand how
the version numbers here are formatted. If you want to remain even
more up-to-date with the developing activities, please clone the
version controlled source as described in *note Version controlled
source::.
Gnuastro's official/stable tarball is released with two formats: Gzip
(with suffix ‘.tar.gz’) and Lzip (with suffix ‘.tar.lz’). The
pre-release tarballs (after version 0.3) are released only as an Lzip
tarball. Gzip is a very well-known and widely used compression program
created by GNU and available in most systems. However, Lzip provides a
better compression ratio and more robust archival capacity. For
example, Gnuastro 0.3's tarball was 2.9MB and 4.3MB with Lzip and Gzip
respectively, see the Lzip web page
(http://www.nongnu.org/lzip/lzip.html) for more. Lzip might not be
pre-installed in your operating system, if so, installing it from your
operating system's package manager or from source is very easy and fast
(it is a very small program).
The GNU FTP server is mirrored (has backups) in various locations on
the globe (<http://www.gnu.org/order/ftp.html>). You can use the
closest mirror to your location for a more faster download. Note that
only some mirrors keep track of the pre-release (alpha) tarballs. Also
note that if you want to download immediately after and announcement
(see *note Announcements::), the mirrors might need some time to
synchronize with the main GNU FTP server.
File: gnuastro.info, Node: Version controlled source, Prev: Release tarball, Up: Downloading the source
3.2.2 Version controlled source
-------------------------------
The publicly distributed Gnuastro tarball (for example,
‘gnuastro-X.X.tar.gz’) does not contain the revision history, it is only
a snapshot of the source code at one significant instant of Gnuastro's
history (specified by the version number, see *note Version
numbering::), ready to be configured and built. To be able to develop
successfully, the revision history of the code can be very useful to
track when something was added or changed, also some updates that are
not yet officially released might be in it.
We use Git for the version control of Gnuastro. For those who are
not familiar with it, we recommend the ProGit book
(https://git-scm.com/book/en). The whole book is publicly available for
online reading and downloading and does a wonderful job at explaining
the concepts and best practices.
Let's assume you want to keep Gnuastro in the ‘TOPGNUASTRO’ directory
(can be any directory, change the value below). The full version
controlled history of Gnuastro can be cloned in ‘TOPGNUASTRO/gnuastro’
by running the following commands(1):
$ TOPGNUASTRO=/home/yourname/Research/projects/
$ cd $TOPGNUASTRO
$ git clone git://git.sv.gnu.org/gnuastro.git
The ‘$TOPGNUASTRO/gnuastro’ directory will contain hand-written (version
controlled) source code for Gnuastro's programs, libraries, this book
and the tests. All are divided into sub-directories with standard and
very descriptive names. The version controlled files in the top cloned
directory are either mainly in capital letters (for example, ‘THANKS’
and ‘README’) or mainly written in small-caps (for example,
‘configure.ac’ and ‘Makefile.am’). The former are non-programming,
standard writing for human readers containing high-level information
about the whole package. The latter are instructions to customize the
GNU build system for Gnuastro. For more on Gnuastro's source code
structure, please see *note Developing::. We will not go any deeper
here.
The cloned Gnuastro source cannot immediately be configured,
compiled, or installed since it only contains hand-written files, not
automatically generated or imported files which do all the hard work of
the build process. See *note Bootstrapping:: for the process of
generating and importing those files (it is not too hard!). Once you
have bootstrapped Gnuastro, you can run the standard procedures (in
*note Quick start::). Very soon after you have cloned it, Gnuastro's
main ‘master’ branch will be updated on the main repository (since the
developers are actively working on Gnuastro), for the best practices in
keeping your local history in sync with the main repository see *note
Synchronizing::.
* Menu:
* Bootstrapping:: Adding all the automatically generated files.
* Synchronizing:: Keep your local clone up to date.
---------- Footnotes ----------
(1) If your internet connection is active, but Git complains about
the network, it might be due to your network setup not recognizing the
Git protocol. In that case use the following URL which uses the HTTP
protocol instead: ‘http://git.sv.gnu.org/r/gnuastro.git’
File: gnuastro.info, Node: Bootstrapping, Next: Synchronizing, Prev: Version controlled source, Up: Version controlled source
3.2.2.1 Bootstrapping
.....................
The version controlled source code lacks the source files that we have
not written or are automatically built. These automatically generated
files are included in the distributed tarball for each distribution (for
example, ‘gnuastro-X.X.tar.gz’, see *note Version numbering::) and make
it easy to immediately configure, build, and install Gnuastro. However
from the perspective of version control, they are just bloatware and
sources of confusion (since they are not changed by Gnuastro
developers).
The process of automatically building and importing necessary files
into the cloned directory is known as _bootstrapping_. After
bootstrapping is done you are ready to follow the default GNU build
steps that you normally run on the tarball (‘./configure && make’ for
example, described more in *note Quick start::). Some known issues with
bootstrapping may occur during the process, to see how to fix them,
please see *note Known issues::.
All the instructions for an automatic bootstrapping are available in
‘bootstrap’ and configured using ‘bootstrap.conf’. ‘bootstrap’ and
‘COPYING’ (which contains the software copyright notice) are the only
files not written by Gnuastro developers but under version control to
enable simple bootstrapping and legal information on usage immediately
after cloning. ‘bootstrap.conf’ is maintained by the GNU Portability
Library (Gnulib) and this file is an identical copy, so do not make any
changes in this file since it will be replaced when Gnulib releases an
update. Make all your changes in ‘bootstrap.conf’.
The bootstrapping process has its own separate set of dependencies,
the full list is given in *note Bootstrapping dependencies::. They are
generally very low-level and used by a very large set of commonly used
programs, so they are probably already installed on your system. The
simplest way to bootstrap Gnuastro is to simply run the bootstrap script
within your cloned Gnuastro directory as shown below. However, please
read the next paragraph before doing so (see *note Version controlled
source:: for ‘TOPGNUASTRO’).
$ cd TOPGNUASTRO/gnuastro
$ ./bootstrap # Requires internet connection
Without any options, ‘bootstrap’ will clone Gnulib within your cloned
Gnuastro directory (‘TOPGNUASTRO/gnuastro/gnulib’) and download the
necessary Autoconf archives macros. So if you run bootstrap like this,
you will need an internet connection every time you decide to bootstrap.
Also, Gnulib is a large package and cloning it can be slow. It will
also keep the full Gnulib repository within your Gnuastro repository, so
if another one of your projects also needs Gnulib, and you insist on
running bootstrap like this, you will have two copies. In case you
regularly backup your important files, Gnulib will also slow down the
backup process. Therefore while the simple invocation above can be used
with no problem, it is not recommended. To do better, see the next
paragraph.
The recommended way to get these two packages is thoroughly discussed
in *note Bootstrapping dependencies:: (in short: clone them in the
separate ‘DEVDIR/’ directory). The following commands will take you
into the cloned Gnuastro directory and run the ‘bootstrap’ script, while
telling it to copy some files (instead of making symbolic links, with
the ‘--copy’ option, this is not mandatory(1)) and where to look for
Gnulib (with the ‘--gnulib-srcdir’ option). Please note that the
address given to ‘--gnulib-srcdir’ has to be an absolute address (so do
not use ‘~’ or ‘../’ for example).
$ cd $TOPGNUASTRO/gnuastro
$ ./bootstrap --copy --gnulib-srcdir=$DEVDIR/gnulib
Since Gnulib and Autoconf archives are now available in your local
directories, you do not need an internet connection every time you
decide to remove all un-tracked files and redo the bootstrap (see box
below). You can also use the same command on any other project that
uses Gnulib. All the necessary GNU C library functions, Autoconf macros
and Automake inputs are now available along with the book figures. The
standard GNU build system (*note Quick start::) will do the rest of the
job.
*Undoing the bootstrap:* During the development, it might happen that
you want to remove all the automatically generated and imported files.
In other words, you might want to reverse the bootstrap process.
Fortunately Git has a good program for this job: ‘git clean’. Run the
following command and every file that is not version controlled will be
removed.
git clean -fxd
It is best to commit any recent change before running this command. You
might have created new files since the last commit and if they have not
been committed, they will all be gone forever (using ‘rm’). To get a
list of the non-version controlled files instead of deleting them, add
the ‘n’ option to ‘git clean’, so it becomes ‘-fxdn’.
Besides the ‘bootstrap’ and ‘bootstrap.conf’, the ‘bootstrapped/’
directory and ‘README-hacking’ file are also related to the
bootstrapping process. The former hosts all the imported (bootstrapped)
directories. Thus, in the version controlled source, it only contains a
‘README’ file, but in the distributed tarball it also contains
sub-directories filled with all bootstrapped files. ‘README-hacking’
contains a summary of the bootstrapping process discussed in this
section. It is a necessary reference when you have not built this book
yet. It is thus not distributed in the Gnuastro tarball.
---------- Footnotes ----------
(1) The ‘--copy’ option is recommended because some backup systems
might do strange things with symbolic links.
File: gnuastro.info, Node: Synchronizing, Prev: Bootstrapping, Up: Version controlled source
3.2.2.2 Synchronizing
.....................
The bootstrapping script (see *note Bootstrapping::) is not regularly
needed: you mainly need it after you have cloned Gnuastro (once) and
whenever you want to re-import the files from Gnulib, or Autoconf
archives(1) (not too common). However, Gnuastro developers are
constantly working on Gnuastro and are pushing their changes to the
official repository. Therefore, your local Gnuastro clone will soon be
out-dated. Gnuastro has two mailing lists dedicated to its developing
activities (see *note Developing mailing lists::). Subscribing to them
can help you decide when to synchronize with the official repository.
To pull all the most recent work in Gnuastro, run the following
command from the top Gnuastro directory. If you do not already have a
built system, ignore ‘make distclean’. The separate steps are described
in detail afterwards.
$ make distclean && git pull && autoreconf -f
You can also run the commands separately:
$ make distclean
$ git pull
$ autoreconf -f
If Gnuastro was already built in this directory, you do not want some
outputs from the previous version being mixed with outputs from the
newly pulled work. Therefore, the first step is to clean/delete all the
built files with ‘make distclean’. Fortunately the GNU build system
allows the separation of source and built files (in separate
directories). This is a great feature to keep your source directory
clean and you can use it to avoid the cleaning step. Gnuastro comes
with a script with some useful options for this job. It is useful if
you regularly pull recent changes, see *note Separate build and source
directories::.
After the pull, we must re-configure Gnuastro with ‘autoreconf -f’
(part of GNU Autoconf). It will update the ‘./configure’ script and all
the ‘Makefile.in’(2) files based on the hand-written configurations (in
‘configure.ac’ and the ‘Makefile.am’ files). After running ‘autoreconf
-f’, a warning about ‘TEXI2DVI’ might show up, you can ignore that.
The most important reason for rebuilding Gnuastro's build system is
to generate/update the version number for your updated Gnuastro
snapshot. This generated version number will include the commit
information (see *note Version numbering::). The version number is
included in nearly all outputs of Gnuastro's programs, therefore it is
vital for reproducing an old result.
As a summary, be sure to run '‘autoreconf -f’' after every change in
the Git history. This includes synchronization with the main server or
even a commit you have made yourself.
If you would like to see what has changed since you last synchronized
your local clone, you can take the following steps instead of the simple
command above (do not type anything after ‘#’):
$ git checkout master # Confirm if you are on master.
$ git fetch origin # Fetch all new commits from server.
$ git log master..origin/master # See all the new commit messages.
$ git merge origin/master # Update your master branch.
$ autoreconf -f # Update the build system.
By default ‘git log’ prints the most recent commit first, add the
‘--reverse’ option to see the changes chronologically. To see exactly
what has been changed in the source code along with the commit message,
add a ‘-p’ option to the ‘git log’.
If you want to make changes in the code, have a look at *note
Developing:: to get started easily. Be sure to commit your changes in a
separate branch (keep your ‘master’ branch to follow the official
repository) and re-run ‘autoreconf -f’ after the commit. If you intend
to send your work to us, you can safely use your commit since it will be
ultimately recorded in Gnuastro's official history. If not, please
upload your separate branch to a public hosting service, for example,
Codeberg (https://codeberg.org), and link to it in your report/paper.
Alternatively, run ‘make distcheck’ and upload the output
‘gnuastro-X.X.X.XXXX.tar.gz’ to a publicly accessible web page so your
results can be considered scientific (reproducible) later.
---------- Footnotes ----------
(1) <https://savannah.gnu.org/task/index.php?13993> is defined for
you to check if significant (for Gnuastro) updates are made in these
repositories, since the last time you pulled from them.
(2) In the GNU build system, ‘./configure’ will use the ‘Makefile.in’
files to create the necessary ‘Makefile’ files that are later read by
‘make’ to build the package.
File: gnuastro.info, Node: Build and install, Prev: Downloading the source, Up: Installation
3.3 Build and install
=====================
This section is basically a longer explanation to the sequence of
commands given in *note Quick start::. If you did not have any problems
during the *note Quick start:: steps, you want to have all the programs
of Gnuastro installed in your system, you do not want to change the
executable names during or after installation, you have root access to
install the programs in the default system wide directory, the Letter
paper size of the print book is fine for you or as a summary you do not
feel like going into the details when everything is working, you can
safely skip this section.
If you have any of the above problems or you want to understand the
details for a better control over your build and install, read along.
The dependencies which you will need prior to configuring, building and
installing Gnuastro are explained in *note Dependencies::. The first
three steps in *note Quick start:: need no extra explanation, so we will
skip them and start with an explanation of Gnuastro specific
configuration options and a discussion on the installation directory in
*note Configuring::, followed by some smaller subsections: *note
Tests::, *note A4 print book::, and *note Known issues:: which explains
the solutions to known problems you might encounter in the installation
steps and ways you can solve them.
* Menu:
* Configuring:: Configure Gnuastro
* Separate build and source directories:: Keeping derivate/build files separate.
* Tests:: Run tests to see if it is working.
* A4 print book:: Customize the print book.
* Known issues:: Issues you might encounter.
File: gnuastro.info, Node: Configuring, Next: Separate build and source directories, Prev: Build and install, Up: Build and install
3.3.1 Configuring
-----------------
The ‘$ ./configure’ step is the most important step in the build and
install process. All the required packages, libraries, headers and
environment variables are checked in this step. The behaviors of make
and make install can also be set through command-line options to this
command.
The configure script accepts various arguments and options which
enable the final user to highly customize whatever she is building. The
options to configure are generally very similar to normal program
options explained in *note Arguments and options::. Similar to all GNU
programs, you can get a full list of the options along with a short
explanation by running
$ ./configure --help
A complete explanation is also included in the ‘INSTALL’ file. Note
that this file was written by the authors of GNU Autoconf (which builds
the ‘configure’ script), therefore it is common for all programs which
use the ‘$ ./configure’ script for building and installing, not just
Gnuastro. Here we only discuss cases where you do not have superuser
access to the system and if you want to change the executable names.
But before that, a review of the options to configure that are
particular to Gnuastro are discussed.
* Menu:
* Gnuastro configure options:: Configure options particular to Gnuastro.
* Installation directory:: Specify the directory to install.
* Executable names:: Changing executable names.
* Configure and build in RAM:: For minimal use of HDD or SSD, and clean source.
File: gnuastro.info, Node: Gnuastro configure options, Next: Installation directory, Prev: Configuring, Up: Configuring
3.3.1.1 Gnuastro configure options
..................................
Most of the options to configure (which are to do with building) are
similar for every program which uses this script. Here the options that
are particular to Gnuastro are discussed. The next topics explain the
usage of other configure options which can be applied to any program
using the GNU build system (through the configure script).
‘--enable-debug’
Compile/build Gnuastro with debugging information, no optimization
and without shared libraries.
In order to allow more efficient programs when using Gnuastro
(after the installation), by default Gnuastro is built with a 3rd
level (a very high level) optimization and no debugging
information. By default, libraries are also built for static _and_
shared linking (see *note Linking::). However, when there are
crashes or unexpected behavior, these three features can hinder the
process of localizing the problem. This configuration option is
identical to manually calling the configuration script with
‘CFLAGS="-g -O0" --disable-shared’.
In the (rare) situations where you need to do your debugging on the
shared libraries, do not use this option. Instead run the
configure script by explicitly setting ‘CFLAGS’ like this:
$ ./configure CFLAGS="-g -O0"
‘--enable-check-with-valgrind’
Do the ‘make check’ tests through Valgrind. Therefore, if any
crashes or memory-related issues (segmentation faults in
particular) occur in the tests, the output of Valgrind will also be
put in the ‘tests/test-suite.log’ file without having to manually
modify the check scripts. This option will also activate
Gnuastro's debug mode (see the ‘--enable-debug’ configure-time
option described above).
Valgrind is free software. It is a program for easy checking of
memory-related issues in programs. It runs a program within its
own controlled environment and can thus identify the exact
line-number in the program's source where a memory-related issue
occurs. However, it can significantly slow-down the tests. So
this option is only useful when a segmentation fault is found
during ‘make check’.
‘--enable-progname’
Only build and install ‘progname’ along with any other program that
is enabled in this fashion. ‘progname’ is the name of the
executable without the ‘ast’, for example, ‘crop’ for Crop (with
the executable name of ‘astcrop’).
Note that by default all the programs will be installed. This
option (and the ‘--disable-progname’ options) are only relevant
when you do not want to install all the programs. Therefore, if
this option is called for any of the programs in Gnuastro, any
program which is not explicitly enabled will not be built or
installed.
‘--disable-progname’
‘--enable-progname=no’
Do not build or install the program named ‘progname’. This is very
similar to the ‘--enable-progname’, but will build and install all
the other programs except this one.
*Note:* If some programs are enabled and some are disabled, it is
equivalent to simply enabling those that were enabled. Listing the
disabled programs is redundant.
‘--enable-gnulibcheck’
Enable checks on the GNU Portability Library (Gnulib). Gnulib is
used by Gnuastro to enable users of non-GNU based operating systems
(that do not use GNU C library or glibc) to compile and use the
advanced features that this library provides. We make extensive
use of such functions. If you give this option to ‘$ ./configure’,
when you run ‘$ make check’, first the functions in Gnulib will be
tested, then the Gnuastro executables. If your operating system
does not support glibc or has an older version of it and you have
problems in the build process (‘$ make’), you can give this flag to
configure to see if the problem is caused by Gnulib not supporting
your operating system or Gnuastro, see *note Known issues::.
‘--disable-guide-message’
‘--enable-guide-message=no’
Do not print a guiding message during the GNU Build process of
*note Quick start::. By default, after each step, a message is
printed guiding the user what the next command should be.
Therefore, after ‘./configure’, it will suggest running ‘make’.
After ‘make’, it will suggest running ‘make check’ and so on. If
Gnuastro is configured with this option, for example
$ ./configure --disable-guide-message
Then these messages will not be printed after any step (like most
programs). For people who are not yet fully accustomed to this
build system, these guidelines can be very useful and encouraging.
However, if you find those messages annoying, use this option.
‘--without-libgit2’
Build Gnuastro without libgit2 (for including Git commit hashes in
output files), see *note Optional dependencies::. libgit2 is an
optional dependency, with this option, Gnuastro will ignore any
possibly existing libgit2 that may already be on the system.
‘--without-libjpeg’
Build Gnuastro without libjpeg (for reading/writing to JPEG files),
see *note Optional dependencies::. libjpeg is an optional
dependency, with this option, Gnuastro will ignore any possibly
existing libjpeg that may already be on the system.
‘--without-libtiff’
Build Gnuastro without libtiff (for reading/writing to TIFF files),
see *note Optional dependencies::. libtiff is an optional
dependency, with this option, Gnuastro will ignore any possibly
existing libtiff that may already be on the system.
‘--with-python’
Build the Python interface within Gnuastro's dynamic library. This
interface can be used for easy communication with Python wrappers
(for example, the pyGnuastro package).
When you install the pyGnuastro package from PyPI, the correct
configuration of the Gnuastro Library is already packaged with it
(with the Python interface) and that is independent of your
Gnuastro installation. The Python interface is only necessary if
you want to build pyGnuastro from source (which is only necessary
for developers). Therefore it has to be explicitly activated at
configure time with this option. For more on the interface
functions, see *note Python interface::.
The tests of some programs might depend on the outputs of the tests
of other programs. For example, MakeProfiles is one the first programs
to be tested when you run ‘$ make check’. MakeProfiles' test outputs
(FITS images) are inputs to many other programs (which in turn provide
inputs for other programs). Therefore, if you do not install
MakeProfiles for example, the tests for many the other programs will be
skipped. To avoid this, in one run, you can install all the programs
and run the tests but not install. If everything is working correctly,
you can run configure again with only the programs you want. However,
do not run the tests and directly install after building.
File: gnuastro.info, Node: Installation directory, Next: Executable names, Prev: Gnuastro configure options, Up: Configuring
3.3.1.2 Installation directory
..............................
One of the most commonly used options to ‘./configure’ is ‘--prefix’, it
is used to define the directory that will host all the installed files
(or the "prefix" in their final absolute file name). For example, when
you are using a server and you do not have administrator or root access.
In this example scenario, if you do not use the ‘--prefix’ option, you
will not be able to install the built files and thus access them from
anywhere without having to worry about where they are installed.
However, once you prepare your startup file to look into the proper
place (as discussed thoroughly below), you will be able to easily use
this option and benefit from any software you want to install without
having to ask the system administrators or install and use a different
version of a software that is already installed on the server.
The most basic way to run an executable is to explicitly write its
full file name (including all the directory information) and run it.
One example is running the configuration script with the ‘$ ./configure’
command (see *note Quick start::). By giving a specific directory (the
current directory or ‘./’), we are explicitly telling the shell to look
in the current directory for an executable file named '‘configure’'.
Directly specifying the directory is thus useful for executables in the
current (or nearby) directories. However, when the program (an
executable file) is to be used a lot, specifying all those directories
will become a significant burden. For example, the ‘ls’ executable
lists the contents in a given directory and it is (usually) installed in
the ‘/usr/bin/’ directory by the operating system maintainers.
Therefore, if using the full address was the only way to access an
executable, each time you wanted a listing of a directory, you would
have to run the following command (which is very inconvenient, both in
writing and in remembering the various directories).
$ /usr/bin/ls
To address this problem, we have the ‘PATH’ environment variable. To
understand it better, we will start with a short introduction to the
shell variables. Shell variable values are basically treated as strings
of characters. For example, it does not matter if the value is a name
(string of _alphabetic_ characters), or a number (string of _numeric_
characters), or both. You can define a variable and a value for it by
running
$ myvariable1=a_test_value
$ myvariable2="a test value"
As you see above, if the value contains white space characters, you have
to put the whole value (including white space characters) in double
quotes (<">). You can see the value it represents by running
$ echo $myvariable1
$ echo $myvariable2
If a variable has no value or it was not defined, the last command will
only print an empty line. A variable defined like this will be known as
long as this shell or terminal is running. Other terminals will have no
idea it existed. The main advantage of shell variables is that if they
are exported(1), subsequent programs that are run within that shell can
access their value. So by changing their value, you can change the
"environment" of a program which uses them. The shell variables which
are accessed by programs are therefore known as "environment
variables"(2). You can see the full list of exported variables that
your shell recognizes by running:
$ printenv
‘HOME’ is one commonly used environment variable, it is any user's
(the one that is logged in) top directory. Try finding it in the
command above. It is used so often that the shell has a special
expansion (alternative) for it: '‘~’'. Whenever you see file names
starting with the tilde sign, it actually represents the value to the
‘HOME’ environment variable, so ‘~/doc’ is the same as ‘$HOME/doc’.
Another one of the most commonly used environment variables is
‘PATH’, it is a list of directories to search for executable names. Its
value is a list of directories (separated by a colon, or '<:>'). When
the address of the executable is not explicitly given (like
‘./configure’ above), the system will look for the executable in the
directories specified by ‘PATH’. If you have a computer nearby, try
running the following command to see which directories your system will
look into when it is searching for executable (binary) files, one
example is printed here (notice how ‘/usr/bin’, in the ‘ls’ example
above, is one of the directories in ‘PATH’):
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/bin
By default ‘PATH’ usually contains system-wide directories, which are
readable (but not writable) by all users, like the above example.
Therefore if you do not have root (or administrator) access, you need to
add another directory to ‘PATH’ which you actually have write access to.
The standard directory where you can keep installed files (not just
executables) for your own user is the ‘~/.local/’ directory. The names
of hidden files start with a '<.>' (dot), so it will not show up in your
common command-line listings, or on the graphical user interface. You
can use any other directory, but this is the most recognized.
The top installation directory will be used to keep all the package's
components: programs (executables), libraries, include (header) files,
shared data (like manuals), or configuration files (see *note Review of
library fundamentals:: for a thorough introduction to headers and
linking). So it commonly has some of the following sub-directories for
each class of installed components respectively: ‘bin/’, ‘lib/’,
‘include/’ ‘man/’, ‘share/’, ‘etc/’. Since the ‘PATH’ variable is only
used for executables, you can add the ‘~/.local/bin’ directory (which
keeps the executables/programs or more generally, "binary" files) to
‘PATH’ with the following command. As defined below, first the existing
value of ‘PATH’ is used, then your given directory is added to its end
and the combined value is put back in ‘PATH’ (run '‘$ echo $PATH’'
afterwards to check if it was added).
$ PATH=$PATH:~/.local/bin
Any executable that you installed in ‘~/.local/bin’ will now be
usable without having to remember and write its full address. However,
as soon as you leave/close your current terminal session, this modified
‘PATH’ variable will be forgotten. Adding the directories which contain
executables to the ‘PATH’ environment variable each time you start a
terminal is also very inconvenient and prone to errors. Fortunately,
there are standard 'startup files' defined by your shell precisely for
this (and other) purposes. There is a special startup file for every
significant starting step:
‘/etc/profile’ and everything in ‘/etc/profile.d/’
These startup scripts are called when your whole system starts (for
example, after you turn on your computer). Therefore you need
administrator or root privileges to access or modify them.
‘~/.bash_profile’
If you are using (GNU) Bash as your shell, the commands in this
file are run, when you log in to your account _through Bash_. Most
commonly when you login through the virtual console (where there is
no graphic user interface).
‘~/.bashrc’
If you are using (GNU) Bash as your shell, the commands here will
be run each time you start a terminal and are already logged in.
For example, when you open your terminal emulator in the graphic
user interface.
For security reasons, it is highly recommended to directly type in
your ‘HOME’ directory value by hand in startup files instead of using
variables. So in the following, let's assume your user name is '‘name’'
(so ‘~’ may be replaced with ‘/home/name’). To add ‘~/.local/bin’ to
your ‘PATH’ automatically on any startup file, you have to "export" the
new value of ‘PATH’ in the startup file that is most relevant to you by
adding this line:
export PATH=$PATH:/home/name/.local/bin
Now that you know your system will look into ‘~/.local/bin’ for
executables, you can tell Gnuastro's configure script to install
everything in the top ‘~/.local’ directory using the ‘--prefix’ option.
When you subsequently run ‘$ make install’, all the install-able files
will be put in their respective directory under ‘~/.local/’ (the
executables in ‘~/.local/bin’, the compiled library files in
‘~/.local/lib’, the library header files in ‘~/.local/include’ and so
on, to learn more about these different files, please see *note Review
of library fundamentals::). Note that tilde ('<~>') expansion will not
happen if you put a '<=>' between ‘--prefix’ and ‘~/.local’(3), so we
have avoided the <=> character here which is optional in GNU-style
options, see *note Options::.
$ ./configure --prefix ~/.local
You can install everything (including libraries like GSL, CFITSIO, or
WCSLIB which are Gnuastro's mandatory dependencies, see *note Mandatory
dependencies::) locally by configuring them as above. However, recall
that ‘PATH’ is only for executable files, not libraries and that
libraries can also depend on other libraries. For example, WCSLIB
depends on CFITSIO and Gnuastro needs both. Therefore, when you
installed a library in a non-recognized directory, you have to guide the
program that depends on them to look into the necessary library and
header file directories. To do that, you have to define the ‘LDFLAGS’
and ‘CPPFLAGS’ environment variables respectively. This can be done
while calling ‘./configure’ as shown below:
$ ./configure LDFLAGS=-L/home/name/.local/lib \
CPPFLAGS=-I/home/name/.local/include \
--prefix ~/.local
It can be annoying/buggy to do this when configuring every software
that depends on such libraries. Hence, you can define these two
variables in the most relevant startup file (discussed above). The
convention on using these variables does not include a colon to separate
values (as ‘PATH’-like variables do). They use white space characters
and each value is prefixed with a compiler option(4). Note the ‘-L’ and
‘-I’ above (see *note Options::), for ‘-I’ see *note Headers::, and for
‘-L’, see *note Linking::. Therefore we have to keep the value in
double quotation signs to keep the white space characters and adding the
following two lines to the startup file of choice:
export LDFLAGS="$LDFLAGS -L/home/name/.local/lib"
export CPPFLAGS="$CPPFLAGS -I/home/name/.local/include"
Dynamic libraries are linked to the executable every time you run a
program that depends on them (see *note Linking:: to fully understand
this important concept). Hence dynamic libraries also require a special
path variable called ‘LD_LIBRARY_PATH’ (same formatting as ‘PATH’). To
use programs that depend on these libraries, you need to add
‘~/.local/lib’ to your ‘LD_LIBRARY_PATH’ environment variable by adding
the following line to the relevant start-up file:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/name/.local/lib
If you also want to access the Info (see *note Info::) and man pages
(see *note Man pages::) documentations add ‘~/.local/share/info’ and
‘~/.local/share/man’ to your ‘INFOPATH’(5) and ‘MANPATH’ environment
variables respectively.
A final note is that order matters in the directories that are
searched for all the variables discussed above. In the examples above,
the new directory was added after the system specified directories. So
if the program, library or manuals are found in the system wide
directories, the user directory is no longer searched. If you want to
search your local installation first, put the new directory before the
already existing list, like the example below.
export LD_LIBRARY_PATH=/home/name/.local/lib:$LD_LIBRARY_PATH
This is good when a library, for example, CFITSIO, is already present on
the system, but the system-wide install was not configured with the
correct configuration flags (see *note CFITSIO::), or you want to use a
newer version and you do not have administrator or root access to update
it on the whole system/server. If you update ‘LD_LIBRARY_PATH’ by
placing ‘~/.local/lib’ first (like above), the linker will first find
the CFITSIO you installed for yourself and link with it. It thus will
never reach the system-wide installation.
There are important security problems with using local installations
first: all important system-wide executables and libraries (important
executables like ‘ls’ and ‘cp’, or libraries like the C library) can be
replaced by non-secure versions with the same file names and put in the
customized directory (‘~/.local’ in this example). So if you choose to
search in your customized directory first, please _be sure_ to keep it
clean from executables or libraries with the same names as important
system programs or libraries.
*Summary:* When you are using a server which does not give you
administrator/root access AND you would like to give priority to your
own built programs and libraries, not the version that is (possibly
already) present on the server, add these lines to your startup file.
See above for which startup file is best for your case and for a
detailed explanation on each. Do not forget to replace
'‘/YOUR-HOME-DIR’' with your home directory (for example,
'‘/home/your-id’'):
export PATH="/YOUR-HOME-DIR/.local/bin:$PATH"
export LDFLAGS="-L/YOUR-HOME-DIR/.local/lib $LDFLAGS"
export MANPATH="/YOUR-HOME-DIR/.local/share/man/:$MANPATH"
export CPPFLAGS="-I/YOUR-HOME-DIR/.local/include $CPPFLAGS"
export INFOPATH="/YOUR-HOME-DIR/.local/share/info/:$INFOPATH"
export LD_LIBRARY_PATH="/YOUR-HOME-DIR/.local/lib:$LD_LIBRARY_PATH"
Afterwards, you just need to add an extra
‘--prefix=/YOUR-HOME-DIR/.local’ to the ‘./configure’ command of the
software that you intend to install. Everything else will be the same
as a standard build and install, see *note Quick start::.
---------- Footnotes ----------
(1) By running ‘$ export myvariable=a_test_value’ instead of the
simpler case in the text
(2) You can use shell variables for other actions too, for example,
to temporarily keep some names or run loops on some files.
(3) If you insist on using '<=>', you can use
‘--prefix=$HOME/.local’.
(4) These variables are ultimately used as options while building the
programs. Therefore every value has be an option name followed be a
value as discussed in *note Options::.
(5) Info has the following convention: "If the value of ‘INFOPATH’
ends with a colon [or it is not defined] ..., the initial list of
directories is constructed by appending the build-time default to the
value of ‘INFOPATH’." So when installing in a non-standard directory
and if ‘INFOPATH’ was not initially defined, add a colon to the end of
‘INFOPATH’ as shown below. Otherwise Info will not be able to find
system-wide installed documentation:
‘echo 'export INFOPATH=$INFOPATH:/home/name/.local/share/info:' >>
~/.bashrc’
Note that this is only an internal convention of Info: do not use it for
other ‘*PATH’ variables.
File: gnuastro.info, Node: Executable names, Next: Configure and build in RAM, Prev: Installation directory, Up: Configuring
3.3.1.3 Executable names
........................
At first sight, the names of the executables for each program might seem
to be uncommonly long, for example, ‘astnoisechisel’ or ‘astcrop’. We
could have chosen terse (and cryptic) names like most programs do. We
chose this complete naming convention (something like the commands in
TeX) so you do not have to spend too much time remembering what the name
of a specific program was. Such complete names also enable you to
easily search for the programs.
To facilitate typing the names in, we suggest using the shell
auto-complete. With this facility you can find the executable you want
very easily. It is very similar to file name completion in the shell.
For example, simply by typing the letters below (where <[TAB]> stands
for the Tab key on your keyboard)
$ ast[TAB][TAB]
you will get the list of all the available executables that start with
‘ast’ in your ‘PATH’ environment variable directories. So, all the
Gnuastro executables installed on your system will be listed. Typing
the next letter for the specific program you want along with a Tab, will
limit this list until you get to your desired program.
In case all of this does not convince you and you still want to type
short names, some suggestions are given below. You should have in mind
though, that if you are writing a shell script that you might want to
pass on to others, it is best to use the standard name because other
users might not have adopted the same customization. The long names
also serve as a form of documentation in such scripts. A similar
reasoning can be given for option names in scripts: it is good practice
to always use the long formats of the options in shell scripts, see
*note Options::.
The simplest solution is making a symbolic link to the actual
executable. For example, let's assume you want to type ‘ic’ to run Crop
instead of ‘astcrop’. Assuming you installed Gnuastro executables in
‘/usr/local/bin’ (default) you can do this simply by running the
following command as root:
# ln -s /usr/local/bin/astcrop /usr/local/bin/ic
In case you update Gnuastro and a new version of Crop is installed, the
default executable name is the same, so your custom symbolic link still
works.
The installed executable names can also be set using options to ‘$
./configure’, see *note Configuring::. GNU Autoconf (which configures
Gnuastro for your particular system), allows the builder to change the
name of programs with the three options ‘--program-prefix’,
‘--program-suffix’ and ‘--program-transform-name’. The first two are
for adding a fixed prefix or suffix to all the programs that will be
installed. This will actually make all the names longer! You can use
it to add versions of program names to the programs in order to
simultaneously have two executable versions of a program.
The third configure option allows you to set the executable name at
install time using the SED program. SED is a very useful 'stream
editor'. There are various resources on the internet to use it
effectively. However, we should caution that using configure options
will change the actual executable name of the installed program and on
every re-install (an update for example), you have to also add this
option to keep the old executable name updated. Also note that the
documentation or configuration files do not change from their standard
names either.
For example, let's assume that typing ‘ast’ on every invocation of
every program is really annoying you! You can remove this prefix from
all the executables at configure time by adding this option:
$ ./configure --program-transform-name='s/ast/ /'
File: gnuastro.info, Node: Configure and build in RAM, Prev: Executable names, Up: Configuring
3.3.1.4 Configure and build in RAM
..................................
Gnuastro's configure and build process (the GNU build system) involves
the creation, reading, and modification of a large number of files
(input/output, or I/O). Therefore file I/O issues can directly affect
the work of developers who need to configure and build Gnuastro numerous
times. Some of these issues are listed below:
• I/O will cause wear and tear on both the HDDs (mechanical failures)
and SSDs (decreasing the lifetime).
• Having the built files mixed with the source files can greatly
affect backing up (synchronization) of source files (since it
involves the management of a large number of small files that are
regularly changed. Backup software can of course be configured to
ignore the built files and directories. However, since the built
files are mixed with the source files and can have a large variety,
this will require a high level of customization.
One solution to address both these problems is to use the tmpfs file
system (https://en.wikipedia.org/wiki/Tmpfs). Any file in tmpfs is
actually stored in the RAM (and possibly SWAP), not on HDDs or SSDs.
The RAM is built for extensive and fast I/O. Therefore the large number
of file I/Os associated with configuring and building will not harm the
HDDs or SSDs. Due to the volatile nature of RAM, files in the tmpfs
file-system will be permanently lost after a power-off. Since all
configured and built files are derivative files (not files that have
been directly written by hand) there is no problem in this and this
feature can be considered as an automatic cleanup.
The modern GNU C library (and thus the Linux kernel) defines the
‘/dev/shm’ directory for this purpose in the RAM (POSIX shared memory).
To build in it, you can use the GNU build system's ability to build in a
separate directory (not necessarily in the source directory) as shown
below. Just set ‘SRCDIR’ as the address of Gnuastro's top source
directory (for example, where there is the unpacked tarball).
$ SRCDIR=/home/username/gnuastro
$ mkdir /dev/shm/tmp-gnuastro-build
$ cd /dev/shm/tmp-gnuastro-build
$ $SRCDIR/configure --srcdir=$SRCDIR
$ make
Gnuastro comes with a script to simplify this process of configuring
and building in a different directory (a "clean" build), for more see
*note Separate build and source directories::.
File: gnuastro.info, Node: Separate build and source directories, Next: Tests, Prev: Configuring, Up: Build and install
3.3.2 Separate build and source directories
-------------------------------------------
The simple steps of *note Quick start:: will mix the source and built
files. This can cause inconvenience for developers or enthusiasts
following the most recent work (see *note Version controlled source::).
The current section is mainly focused on this later group of Gnuastro
users. If you just install Gnuastro on major releases (following *note
Announcements::), you can safely ignore this section.
When it is necessary to keep the source (which is under version
control), but not the derivative (built) files (after checking or
installing), the best solution is to keep the source and the built files
in separate directories. One application of this is already discussed
in *note Configure and build in RAM::.
To facilitate this process of configuring and building in a separate
directory, Gnuastro comes with the ‘developer-build’ script. It is
available in the top source directory and is _not_ installed. It will
make a directory under a given top-level directory (given to
‘--top-build-dir’) and build Gnuastro there. It thus keeps the source
completely separated from the built files. For easy access to the built
files, it also makes a symbolic link to the built directory in the top
source files called ‘build’.
When running the developer-build script without any options in the
Gnuastro's top source directory, default values will be used for its
configuration. As with Gnuastro's programs, you can inspect the default
values with ‘-P’ (or ‘--printparams’, the output just looks a little
different here). The default top-level build directory is ‘/dev/shm’:
the shared memory directory in RAM on GNU/Linux systems as described in
*note Configure and build in RAM::.
Besides these, it also has some features to facilitate the job of
developers or bleeding edge users like the ‘--debug’ option to do a fast
build, with debug information, no optimization, and no shared libraries.
Here is the full list of options you can feed to this script to
configure its operations.
*Not all Gnuastro's common program behavior usable here:*
‘developer-build’ is just a non-installed script with a very limited
scope as described above. It thus does not have all the common option
behaviors or configuration files for example.
*White space between option and value:* ‘developer-build’ does not
accept an <=> sign between the options and their values. It also needs
at least one character between the option and its value. Therefore ‘-n
4’ or ‘--numthreads 4’ are acceptable, while ‘-n4’, ‘-n=4’, or
‘--numthreads=4’ are not. Finally multiple short option names cannot be
merged: for example, you can say ‘-c -n 4’, but unlike Gnuastro's
programs, ‘-cn4’ is not acceptable.
*Reusable for other packages:* This script can be used in any software
which is configured and built using the GNU Build System. Just copy it
in the top source directory of that software and run it from there.
*Example usage:* See *note Forking tutorial:: for an example usage of
this script in some scenarios.
‘-b STR’
‘--top-build-dir STR’
The top build directory to make a directory for the build. If this
option is not called, the top build directory is ‘/dev/shm’ (only
available in GNU/Linux operating systems, see *note Configure and
build in RAM::).
‘-V’
‘--version’
Print the version string of Gnuastro that will be used in the
build. This string will be appended to the directory name
containing the built files.
‘-a’
‘--autoreconf’
Run ‘autoreconf -f’ before building the package. In Gnuastro, this
is necessary when a new commit has been made to the project
history. In Gnuastro's build system, the Git description will be
used as the version, see *note Version numbering:: and *note
Synchronizing::.
‘-c’
‘--clean’
Delete the contents of the build directory (clean it) before
starting the configuration and building of this run.
This is useful when you have recently pulled changes from the main
Git repository, or committed a change yourself and ran ‘autoreconf
-f’, see *note Synchronizing::. After running GNU Autoconf, the
version will be updated and you need to do a clean build.
‘-d’
‘--debug’
Build with debugging flags (for example, to use in GNU Debugger,
also known as GDB, or Valgrind), disable optimization and also the
building of shared libraries. Similar to running the configure
script of below
$ ./configure --enable-debug
Besides all the debugging advantages of building with this option,
it will also be significantly speed up the build (at the cost of
slower built programs). So when you are testing something small or
working on the build system itself, it will be much faster to test
your work with this option.
‘-v’
‘--valgrind’
Build all ‘make check’ tests within Valgrind. For more, see the
description of ‘--enable-check-with-valgrind’ in *note Gnuastro
configure options::.
‘-j INT’
‘--jobs INT’
The maximum number of threads/jobs for Make to build at any moment.
As the name suggests (Make has an identical option), the number
given to this option is directly passed on to any call of Make with
its ‘-j’ option.
‘-C’
‘--check’
After finishing the build, also run ‘make check’. By default,
‘make check’ is not run because the developer usually has their own
checks to work on (for example, defined in ‘tests/during-dev.sh’).
‘-i’
‘--install’
After finishing the build, also run ‘make install’.
‘-D’
‘--dist’
Run ‘make dist-lzip pdf’ to build a distribution tarball (in
‘.tar.lz’ format) and a PDF manual. This can be useful for
archiving, or sending to colleagues who do not use Git for an easy
build and manual.
‘-u STR’
‘--upload STR’
Activate the ‘--dist’ (‘-D’) option, then use secure copy (‘scp’,
part of the SSH tools) to copy the tarball and PDF to the ‘src’ and
‘pdf’ sub-directories of the specified server and its directory
(value to this option). For example, ‘--upload my-server:dir’,
will copy the tarball in the ‘dir/src’, and the PDF manual in
‘dir/pdf’ of ‘my-server’ server. It will then make a symbolic link
in the top server directory to the tarball that is called
‘gnuastro-latest.tar.lz’.
‘-p STR’
‘--publish=STR’
Clean, bootstrap, build, check and upload the checked tarball and
PDF of the book to the URL given as ‘STR’. This option is just a
wrapper for ‘--autoreconf --clean --debug --check --upload STR’.
‘--debug’ is added because it will greatly speed up the build.
‘--debug’ will have no effect on the produced tarball (people who
later download will be building with the default optimized, and
non-debug mode). This option is good when you have made a commit
and are ready to publish it on your server (if nothing crashes).
Recall that if any of the previous steps fail the script aborts.
‘-I’
‘--install-archive’
Short for ‘--autoreconf --clean --check --install --dist’. This is
useful when you actually want to install the commit you just made
(if the build and checks succeed). It will also produce a
distribution tarball and PDF manual for easy access to the
installed tarball on your system at a later time.
Ideally, Gnuastro's Git version history makes it easy for a
prepared system to revert back to a different point in history.
But Gnuastro also needs to bootstrap files and also your
collaborators might (usually do!) find it too much of a burden to
do the bootstrapping themselves. So it is convenient to have a
tarball and PDF manual of the version you have installed (and are
using in your research) handily available.
‘-h’
‘--help’
‘-P’
‘--printparams’
Print a description of this script along with all the options and
their current values.
File: gnuastro.info, Node: Tests, Next: A4 print book, Prev: Separate build and source directories, Up: Build and install
3.3.3 Tests
-----------
After successfully building (compiling) the programs with the ‘$ make’
command you can check the installation before installing. To run the
tests, run
$ make check
For every program some tests are designed to check some possible
operations. Running the command above will run those tests and give you
a final report. If everything is OK and you have built all the
programs, all the tests should pass. In case any of the tests fail,
please have a look at *note Known issues:: and if that still does not
fix your problem, look that the ‘./tests/test-suite.log’ file to see if
the source of the error is something particular to your system or more
general. If you feel it is general, please contact us because it might
be a bug. Note that the tests of some programs depend on the outputs of
other program's tests, so if you have not installed them they might be
skipped or fail. Prior to releasing every distribution all these tests
are checked. If you have a reasonably modern terminal, the outputs of
the successful tests will be colored green and the failed ones will be
colored red.
These scripts can also act as a good set of examples for you to see
how the programs are run. All the tests are in the ‘tests/’ directory.
The tests for each program are shell scripts (ending with ‘.sh’) in a
sub-directory of this directory with the same name as the program. See
*note Test scripts:: for more detailed information about these scripts
in case you want to inspect them.
File: gnuastro.info, Node: A4 print book, Next: Known issues, Prev: Tests, Up: Build and install
3.3.4 A4 print book
-------------------
The default print version of this book is provided in the letter paper
size. If you would like to have the print version of this book on paper
and you are living in a country which uses A4, then you can rebuild the
book. The great thing about the GNU build system is that the book
source code which is in Texinfo is also distributed with the program
source code, enabling you to do such customization (hacking).
In order to change the paper size, you will need to have GNU Texinfo
installed. Open ‘doc/gnuastro.texi’ with any text editor. This is the
source file that created this book. In the first few lines you will see
this line:
@c@afourpaper
In Texinfo, a line is commented with ‘@c’. Therefore, un-comment this
line by deleting the first two characters such that it changes to:
@afourpaper
Save the file and close it. You can now run the following command
$ make pdf
and the new PDF book will be available in ‘SRCdir/doc/gnuastro.pdf’. By
changing the ‘pdf’ in ‘$ make pdf’ to ‘ps’ or ‘dvi’ you can have the
book in those formats. Note that you can do this for any book that is
in Texinfo format, they might not have ‘@afourpaper’ line, so you can
add it close to the top of the Texinfo source file.
File: gnuastro.info, Node: Known issues, Prev: A4 print book, Up: Build and install
3.3.5 Known issues
------------------
Depending on your operating system and the version of the compiler you
are using, you might confront some known problems during the
configuration (‘$ ./configure’), compilation (‘$ make’) and tests (‘$
make check’). Here, their solutions are discussed.
• ‘$ ./configure’: _Configure complains about not finding a library
even though you have installed it._ The possible solution is based
on how you installed the package:
• From your distribution's package manager. Most probably this
is because your distribution has separated the header files of
a library from the library parts. Please also install the
'development' packages for those libraries too. Just add a
‘-dev’ or ‘-devel’ to the end of the package name and re-run
the package manager. This will not happen if you install the
libraries from source. When installed from source, the
headers are also installed.
• From source. Then your linker is not looking where you
installed the library. If you followed the instructions in
this chapter, all the libraries will be installed in
‘/usr/local/lib’. So you have to tell your linker to look in
this directory. To do so, configure Gnuastro like this:
$ ./configure LDFLAGS="-L/usr/local/lib"
If you want to use the libraries for your other programming
projects, then export this environment variable in a start-up
script similar to the case for ‘LD_LIBRARY_PATH’ explained
below, also see *note Installation directory::.
• ‘$ make’: _Complains about an unknown function on a non-GNU based
operating system._ In this case, please run ‘$ ./configure’ with
the ‘--enable-gnulibcheck’ option to see if the problem is from the
GNU Portability Library (Gnulib) not supporting your system or if
there is a problem in Gnuastro, see *note Gnuastro configure
options::. If the problem is not in Gnulib and after all its tests
you get the same complaint from ‘make’, then please contact us at
‘bug-gnuastro@gnu.org’. The cause is probably that a function that
we have used is not supported by your operating system and we did
not included it along with the source tarball. If the function is
available in Gnulib, it can be fixed immediately.
• ‘$ make’: _Cannot find the headers (.h files) of installed
libraries._ Your C preprocessor (CPP) is not looking in the right
place. To fix this, configure Gnuastro with an additional
‘CPPFLAGS’ like below (assuming the library is installed in
‘/usr/local/include’:
$ ./configure CPPFLAGS="-I/usr/local/include"
If you want to use the libraries for your other programming
projects, then export this environment variable in a start-up
script similar to the case for ‘LD_LIBRARY_PATH’ explained below,
also see *note Installation directory::.
• ‘$ make check’: _Only the first couple of tests pass, all the rest
fail or get skipped._ It is highly likely that when searching for
shared libraries, your system does not look into the
‘/usr/local/lib’ directory (or wherever you installed Gnuastro or
its dependencies). To make sure it is added to the list of
directories, add the following line to your ‘~/.bashrc’ file and
restart your terminal. Do not forget to change ‘/usr/local/lib’ if
the libraries are installed in other (non-standard) directories.
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
You can also add more directories by using a colon '‘:’' to
separate them. See *note Installation directory:: and *note
Linking:: to learn more on the ‘PATH’ variables and dynamic linking
respectively.
• ‘$ make check’: _The tests relying on external programs (for
example, ‘fitstopdf.sh’ fail.)_ This is probably due to the fact
that the version number of the external programs is too old for the
tests we have preformed. Please update the program to a more
recent version. For example, to create a PDF image, you will need
GPL Ghostscript, but older versions do not work, we have
successfully tested it on version 9.15. Older versions might cause
a failure in the test result.
• ‘$ make pdf’: _The PDF book cannot be made._ To make a PDF book,
you need to have the GNU Texinfo program (like any program, the
more recent the better). A working TeX program is also necessary,
which you can get from Tex Live(1).
• After ‘make check’: do not copy the programs' executables to
another (for example, the installation) directory manually (using
‘cp’, or ‘mv’ for example). In the default configuration(2), the
program binaries need to link with Gnuastro's shared library which
is also built and installed with the programs. Therefore, to run
successfully before and after installation, linking modifications
need to be made by GNU Libtool at installation time. ‘make
install’ does this internally, but a simple copy might give linking
errors when you run it. If you need to copy the executables, you
can do so after installation.
• ‘$ make’ (when bootstrapping): After you have bootstrapped Gnuastro
from the version-controlled source, you may confront the following
(or a similar) error when converting images (for more on
bootstrapping, see *note Bootstrapping::):
convert: attempt to perform an operation not allowed by the
security policy `gs' error/delegate.c/ExternalDelegateCommand/378.
This error is a known issue(3) with ‘ImageMagick’ security policies
in some operating systems. In short, ‘imagemagick’ uses
Ghostscript for PDF, EPS, PS and XPS parsing. However, because
some security vulnerabilities have been found in Ghostscript(4), by
default, ImageMagick may be compiled without Ghostscript library.
In such cases, if allowed, ImageMagick will fall back to the
external ‘gs’ command instead of the library. But this may be
disabled with the following (or a similar) lines in
‘/etc/ImageMagick-7/policy.xml’ (anything related to PDF, PS, or
Ghostscript).
<policy domain="delegate" rights="none" pattern="gs" />
<policy domain="module" rights="none" pattern="{PS,PDF,XPS}" />
To fix this problem, simply comment such lines (by placing a ‘<!--’
before each statement/line and ‘-->’ at the end of that
statement/line).
If your problem was not listed above, please file a bug report (*note
Report a bug::).
---------- Footnotes ----------
(1) <https://www.tug.org/texlive/>
(2) If you configure Gnuastro with the ‘--disable-shared’ option,
then the libraries will be statically linked to the programs and this
problem will not exist, see *note Linking::.
(3) <https://wiki.archlinux.org/title/ImageMagick>
(4) <https://security.archlinux.org/package/ghostscript>
File: gnuastro.info, Node: Common program behavior, Next: Data containers, Prev: Installation, Up: Top
4 Common program behavior
*************************
All the programs in Gnuastro share a set of common behavior mainly to do
with user interaction to facilitate their usage and development. This
includes how to feed input datasets into the programs, how to configure
them, specifying the outputs, numerical data types, treating columns of
information in tables, etc. This chapter is devoted to describing this
common behavior in all programs. Because the behaviors discussed here
are common to several programs, they are not repeated in each program's
description.
In *note Command-line::, a very general description of running the
programs on the command-line is discussed, like difference between
arguments and options, as well as options that are common/shared between
all programs. None of Gnuastro's programs keep any internal
configuration value (values for their different operational steps), they
read their configuration primarily from the command-line, then from
specific files in directory, user, or system-wide settings. Using these
configuration files can greatly help reproducible and robust usage of
Gnuastro, see *note Configuration files:: for more.
It is not possible to always have the different options and
configurations of each program on the top of your head. It is very
natural to forget the options of a program, their current default
values, or how it should be run and what it did. Gnuastro's programs
have multiple ways to help you refresh your memory in multiple levels
(just an option name, a short description, or fast access to the
relevant section of the manual. See *note Getting help:: for more for
more on benefiting from this very convenient feature.
Many of the programs use the multi-threaded character of modern CPUs,
in *note Multi-threaded operations:: we will discuss how you can
configure this behavior, along with some tips on making best use of
them. In *note Numeric data types::, we will review the various types
to store numbers in your datasets: setting the proper type for the usage
context(1) can greatly improve the file size and also speed of reading,
writing or processing them.
We will then look into the recognized table formats in *note Tables::
and how large datasets are broken into tiles, or mesh grid in *note
Tessellation::. Finally, we will take a look at the behavior regarding
output files: *note Automatic output:: describes how the programs set a
default name for their output when you do not give one explicitly (using
‘--output’). When the output is a FITS file, all the programs also
store some very useful information in the header that is discussed in
*note Output FITS files::.
* Menu:
* Command-line:: How to use the command-line.
* Configuration files:: Values for unspecified variables.
* Getting help:: Getting more information on the go.
* Multi-threaded operations:: How threads are managed in Gnuastro.
* Numeric data types:: Different types and how to specify them.
* Memory management:: How memory is allocated (in RAM or HDD/SSD).
* Tables:: Recognized table formats.
* Tessellation:: Tile the dataset into non-overlapping bins.
* Automatic output:: About automatic output names.
* Output FITS files:: Common properties when outputs are FITS.
* Numeric locale:: Decimal point printed like 0.5 instead of 0,5.
---------- Footnotes ----------
(1) For example, if the values in your dataset can only be integers
between 0 or 65000, store them in a unsigned 16-bit type, not 64-bit
floating point type (which is the default in most systems). It takes
four times less space and is much faster to process.
File: gnuastro.info, Node: Command-line, Next: Configuration files, Prev: Common program behavior, Up: Common program behavior
4.1 Command-line
================
Gnuastro's programs are customized through the standard Unix-like
command-line environment and GNU style command-line options. Both are
very common in many Unix-like operating system programs. In *note
Arguments and options:: we will start with the difference between
arguments and options and elaborate on the GNU style of options.
Afterwards, in *note Common options::, we will go into the detailed list
of all the options that are common to all the programs in Gnuastro.
* Menu:
* Arguments and options:: Different ways to specify inputs and configuration.
* Common options:: Options that are shared between all programs.
* Shell TAB completion:: Customized TAB completion in Gnuastro.
* Standard input:: Using output of another program as input.
* Shell tips:: Useful tips and tricks for program usage.
File: gnuastro.info, Node: Arguments and options, Next: Common options, Prev: Command-line, Up: Command-line
4.1.1 Arguments and options
---------------------------
When you type a command on the command-line, it is passed onto the shell
(a generic name for the program that manages the command-line) as a
string of characters. As an example, see the "Invoking ProgramName"
sections in this manual for some examples of commands with each program,
like *note Invoking asttable::, *note Invoking astfits::, or *note
Invoking aststatistics::.
The shell then brakes up your string into separate _tokens_ or
_words_ using any _metacharacters_ (like white-space, tab, ‘|’, ‘>’ or
‘;’) that are in the string. On the command-line, the first thing you
usually enter is the name of the program you want to run. After that,
you can specify two types of tokens: _arguments_ and _options_. In the
GNU-style, arguments are those tokens that are not preceded by any
hyphens (‘-’, see *note Arguments::). Here is one example:
$ astcrop --center=53.162551,-27.789676 -w10/3600 --mode=wcs udf.fits
In the example above, we are running *note Crop:: to crop a region of
width 10 arc-seconds centered at the given RA and Dec from the input
Hubble Ultra-Deep Field (UDF) FITS image. Here, the argument is
‘udf.fits’. Arguments are most commonly the input file names containing
your data. Options start with one or two hyphens, followed by an
identifier for the option (the option's name, for example, ‘--center’,
‘-w’, ‘--mode’ in the example above) and its value (anything after the
option name, or the optional <=> character). Through options you can
configure how the program runs (interprets the data you provided).
Arguments can be mandatory and optional and unlike options, they do
not have any identifiers. Hence, when there multiple arguments, their
order might also matter (for example, in ‘cp’ which is used for copying
one file to another location). The outputs of ‘--usage’ and ‘--help’
shows which arguments are optional and which are mandatory, see *note
--usage::.
As their name suggests, _options_ can be considered to be optional
and most of the time, you do not have to worry about what order you
specify them in. When the order does matter, or the option can be
invoked multiple times, it is explicitly mentioned in the "Invoking
ProgramName" section of each program (this is a very important aspect of
an option).
If there is only one such character, you can use a backslash (‘\’)
before it. If there are multiple, it might be easier to simply put your
whole argument or option value inside of double quotes (‘"’). In such
cases, everything inside the double quotes will be seen as one token or
word.
For example, let's say you want to specify the header data unit (HDU)
of your FITS file using a complex expression like '‘3; images(exposure >
100)’'. If you simply add these after the ‘--hdu’ (‘-h’) option, the
programs in Gnuastro will read the value to the HDU option as '‘3’' and
run. Then, the shell will attempt to run a separate command
'‘images(exposure > 100)’' and complain about a syntax error. This is
because the semicolon (‘;’) is an 'end of command' character in the
shell. To solve this problem you can simply put double quotes around
the whole string you want to pass to ‘--hdu’ as seen below:
$ astcrop --hdu="3; images(exposure > 100)" image.fits
* Menu:
* Arguments:: For specifying the main input files/operations.
* Options:: For configuring the behavior of the program.
File: gnuastro.info, Node: Arguments, Next: Options, Prev: Arguments and options, Up: Arguments and options
4.1.1.1 Arguments
.................
In Gnuastro, arguments are almost exclusively used as the input data
file names. Please consult the first few paragraph of the "Invoking
ProgramName" section for each program for a description of what it
expects as input, how many arguments, or input data, it accepts, or in
what order. Everything particular about how a program treats arguments,
is explained under the "Invoking ProgramName" section for that program.
Generally, if there is a standard file name suffix for a particular
format, that filename extension is checked to identify their format. In
astronomy (and thus Gnuastro), FITS is the preferred format for inputs
and outputs, so the focus here and throughout this book is on FITS.
However, other formats are also accepted in special cases, for example,
*note ConvertType:: also accepts JPEG or TIFF inputs, and writes JPEG,
EPS or PDF files. The recognized suffixes for these formats are listed
there.
The list below shows the recognized suffixes for FITS data files in
Gnuastro's programs. However, in some scenarios FITS writers may not
append a suffix to the file, or use a non-recognized suffix (not in the
list below). Therefore if a FITS file is expected, but it does not have
any of these suffixes, Gnuastro programs will look into the contents of
the file and if it does conform with the FITS standard, the file will be
used. Just note that checking about 5 characters at the end of a name
string is much more efficient than opening and checking the contents of
a file, so it is generally recommended to have a recognized FITS suffix.
• ‘.fits’: The standard file name ending of a FITS image.
• ‘.fit’: Alternative (3 character) FITS suffix.
• ‘.fits.Z’: A FITS image compressed with ‘compress’.
• ‘.fits.gz’: A FITS image compressed with GNU zip (gzip).
• ‘.fits.fz’: A FITS image compressed with ‘fpack’.
• ‘.imh’: IRAF format image file.
Throughout this book and in the command-line outputs, whenever we
want to generalize all such astronomical data formats in a text
place-holder, we will use ‘ASTRdata’ and assume that the extension is
also part of this name. Any file ending with these names is directly
passed on to CFITSIO to read. Therefore you do not necessarily have to
have these files on your computer, they can also be located on an FTP or
HTTP server too, see the CFITSIO manual for more information.
CFITSIO has its own error reporting techniques, if your input file(s)
cannot be opened, or read, those errors will be printed prior to the
final error by Gnuastro.
File: gnuastro.info, Node: Options, Prev: Arguments, Up: Arguments and options
4.1.1.2 Options
...............
Command-line options allow configuring the behavior of a program in all
GNU/Linux applications for each particular execution on a particular
input data. A single option can be called in two ways: _long_ or
_short_. All options in Gnuastro accept the long format which has two
hyphens an can have many characters (for example, ‘--hdu’). Short
options only have one hyphen (<->) followed by one character (for
example, ‘-h’). You can see some examples in the list of options in
*note Common options:: or those for each program's "Invoking
ProgramName" section. Both formats are shown for those which support
both. First the short is shown then the long.
Usually, the short options are handy when you are writing on the
command-line and want to save keystrokes and time. The long options are
good for shell scripts, where you are not usually rushing. Long options
provide a level of documentation, since they are more descriptive and
less cryptic. Usually after a few months of not running a program, the
short options will be forgotten and reading your previously written
script will not be easy.
Some options need to be given a value if they are called and some do
not. You can think of the latter type of options as on/off options.
These two types of options can be distinguished using the output of the
‘--help’ and ‘--usage’ options, which are common to all GNU software,
see *note Getting help::. In Gnuastro we use the following strings to
specify when the option needs a value and what format that value should
be in. More specific tests will be done in the program and if the
values are out of range (for example, negative when the program only
wants a positive value), an error will be reported.
‘INT’
The value is read as an integer.
‘FLT’
The value is read as a float. There are generally two types,
depending on the context. If they are for fractions, they will
have to be less than or equal to unity.
‘STR’
The value is read as a string of characters. For example, column
names in a table, or HDU names in a multi-extension FITS file.
Other examples include human-readable settings by some programs
like the ‘--domain’ option of the Convolve program that can be
either ‘spatial’ or ‘frequency’ (to specify the type of
convolution, see *note Convolve::).
‘FITS or FITS/TXT’
The value should be a file (most commonly FITS). In many cases,
other formats may also be accepted (for example, input tables can
be FITS or plain-text, see *note Recognized table formats::).
To specify a value in the short format, simply put the value after the
option. Note that since the short options are only one character long,
you do not have to type anything between the option and its value. For
the long option you either need white space or an ‘=’ sign, for example,
‘-h2’, ‘-h 2’, ‘--hdu 2’ or ‘--hdu=2’ are all equivalent.
The short format of on/off options (those that do not need values)
can be concatenated for example, these two hypothetical sequences of
options are equivalent: ‘-a -b -c4’ and ‘-abc4’. As an example,
consider the following command to run Crop:
$ astcrop -Dr3 --wwidth 3 catalog.txt --deccol=4 ASTRdata
The ‘$’ is the shell prompt, ‘astcrop’ is the program name. There are
two arguments (‘catalog.txt’ and ‘ASTRdata’) and four options, two of
them given in short format (‘-D’, ‘-r’) and two in long format
(‘--width’ and ‘--deccol’). Three of them require a value and one
(‘-D’) is an on/off option.
If an abbreviation is unique between all the options of a program,
the long option names can be abbreviated. For example, instead of
typing ‘--printparams’, typing ‘--print’ or maybe even ‘--pri’ will be
enough, if there are conflicts, the program will warn you and show you
the alternatives. Finally, if you want the argument parser to stop
parsing arguments beyond a certain point, you can use two dashes: ‘--’.
No text on the command-line beyond these two dashes will be parsed.
Gnuastro has two types of options with values, those that only take a
single value are the most common type. If these options are repeated or
called more than once on the command-line, the value of the last time it
was called will be assigned to it. This is very useful when you are
testing/experimenting. Let's say you want to make a small modification
to one option value. You can simply type the option with a new value in
the end of the command and see how the script works. If you are
satisfied with the change, you can remove the original option for human
readability. If the change was not satisfactory, you can remove the one
you just added and not worry about forgetting the original value.
Without this capability, you would have to memorize or save the original
value somewhere else, run the command and then change the value again
which is not at all convenient and is potentially cause lots of bugs.
On the other hand, some options can be called multiple times in one
run of a program and can thus take multiple values (for example, see the
‘--column’ option in *note Invoking asttable::. In these cases, the
order of stored values is the same order that you specified on the
command-line.
Gnuastro's programs do not keep any internal default values, so some
options are mandatory and if they do not have a value, the program will
complain and abort. Most programs have many such options and typing
them by hand on every call is impractical. To facilitate the user
experience, after parsing the command-line, Gnuastro's programs read
special configuration files to get the necessary values for the options
you have not identified on the command-line. These configuration files
are fully described in *note Configuration files::.
*CAUTION:* In specifying a file address, if you want to use the shell's
tilde expansion (‘~’) to specify your home directory, leave at least one
space between the option name and your value. For example, use ‘-o
~/test’, ‘--output ~/test’ or ‘--output= ~/test’. Calling them with
‘-o~/test’ or ‘--output=~/test’ will disable shell expansion.
*CAUTION:* If you forget to specify a value for an option which requires
one, and that option is the last one, Gnuastro will warn you. But if it
is in the middle of the command, it will take the text of the next
option or argument as the value which can cause undefined behavior.
*NOTE:* In some contexts Gnuastro's counting starts from 0 and in others
1. You can assume by default that counting starts from 1, if it starts
from 0 for a special option, it will be explicitly mentioned.
File: gnuastro.info, Node: Common options, Next: Shell TAB completion, Prev: Arguments and options, Up: Command-line
4.1.2 Common options
--------------------
To facilitate the job of the users and developers, all the programs in
Gnuastro share some basic command-line options for the options that are
common to many of the programs. The full list is classified as *note
Input output options::, *note Processing options::, and *note Operating
mode options::. In some programs, some of the options are irrelevant,
but still recognized (you will not get an unrecognized option error, but
the value is not used). Unless otherwise mentioned, these options are
identical between all programs.
* Menu:
* Input output options:: Common input/output options.
* Processing options:: Options for common processing steps.
* Operating mode options:: Common operating mode options.
File: gnuastro.info, Node: Input output options, Next: Processing options, Prev: Common options, Up: Common options
4.1.2.1 Input/Output options
............................
These options are to do with the input and outputs of the various
programs.
‘--stdintimeout’
Number of micro-seconds to wait for writing/typing in the _first
line_ of standard input from the command-line (see *note Standard
input::). This is only relevant for programs that also accept
input from the standard input, _and_ you want to manually
write/type the contents on the terminal. When the standard input
is already connected to a pipe (output of another program), there
will not be any waiting (hence no timeout, thus making this option
redundant).
If the first line-break (for example, with the <ENTER> key) is not
provided before the timeout, the program will abort with an error
that no input was given. Note that this time interval is _only_
for the first line that you type. Once the first line is given,
the program will assume that more data will come and accept rest of
your inputs without any time limit. You need to specify the ending
of the standard input, for example, by pressing <CTRL-D> after a
new line.
Note that any input you write/type into a program on the
command-line with Standard input will be discarded (lost) once the
program is finished. It is only recoverable manually from your
command-line (where you actually typed) as long as the terminal is
open. So only use this feature when you are sure that you do not
need the dataset (or have a copy of it somewhere else).
‘-h STR/INT’
‘--hdu=STR/INT’
The name or number of the desired Header Data Unit, or HDU, in the
FITS image. A FITS file can store multiple HDUs or extensions,
each with either an image or a table or nothing at all (only a
header). Note that counting of the extensions starts from 0(zero),
not 1(one). Counting from 0 is forced on us by CFITSIO which
directly reads the value you give with this option (see *note
CFITSIO::). When specifying the name, case is not important so
‘IMAGE’, ‘image’ or ‘ImAgE’ are equivalent.
CFITSIO has many capabilities to help you find the extension you
want, far beyond the simple extension number and name. See CFITSIO
manual's "HDU Location Specification" section for a very complete
explanation with several examples. A ‘#’ is appended to the string
you specify for the HDU(1) and the result is put in square brackets
and appended to the FITS file name before calling CFITSIO to read
the contents of the HDU for all the programs in Gnuastro.
*Default HDU is HDU number 1 (counting from 0):* by default,
Gnuastro’s programs assume that their (main/first) input is in HDU
number 1 (counting from zero). So if you don’t specify the HDU
number, the program will read the input from this HDU. For programs
that can take multiple FITS datasets as input (like *note
Arithmetic::) this default HDU applies to the first input, you
still need to call ‘--hdu’ for the other inputs. Generally, all
Gnuastro's programs write their outputs in HDU number 1 (HDU 0 is
reserved for metadata like the configuration parameters that the
program was run with). For more on this, see *note Fits::.
‘-s STR’
‘--searchin=STR’
Where to match/search for columns when the column identifier was
not a number, see *note Selecting table columns::. The acceptable
values are ‘name’, ‘unit’, or ‘comment’. This option is only
relevant for programs that take table columns as input.
‘-I’
‘--ignorecase’
Ignore case while matching/searching column meta-data (in the field
specified by the ‘--searchin’). The FITS standard suggests to
treat the column names as case insensitive, which is strongly
recommended here also but is not enforced. This option is only
relevant for programs that take table columns as input.
This option is not relevant to *note BuildProgram::, hence in that
program the short option ‘-I’ is used for include directories, not
to ignore case.
‘-o STR’
‘--output=STR’
The name of the output file or directory. With this option the
automatic output names explained in *note Automatic output:: are
ignored.
‘-T STR’
‘--type=STR’
The data type of the output depending on the program context. This
option is not applicable to some programs like *note Fits:: and
will be ignored by them. The different acceptable values to this
option are fully described in *note Numeric data types::.
‘-D’
‘--dontdelete’
By default, if the output file already exists, Gnuastro's programs
will silently delete it and put their own outputs in its place.
When this option is activated, if the output file already exists,
the programs will not delete it, will warn you, and will abort.
‘-K’
‘--keepinputdir’
In automatic output names, do not remove the directory information
of the input file names. As explained in *note Automatic output::,
if no output name is specified (with ‘--output’), then the output
name will be made in the existing directory based on your input's
file name (ignoring the directory of the input). If you call this
option, the directory information of the input will be kept and the
automatically generated output name will be in the same directory
as the input (usually with a suffix added). Note that his is only
relevant if you are running the program in a different directory
than the input data.
‘-t STR’
‘--tableformat=STR’
The output table's type. This option is only relevant when the
output is a table and its format cannot be deduced from its
filename. For example, if a name ending in ‘.fits’ was given to
‘--output’, then the program knows you want a FITS table. But
there are two types of FITS tables: FITS ASCII, and FITS binary.
Thus, with this option, the program is able to identify which type
you want. The currently recognized values to this option are:
‘--wcslinearmatrix=STR’
Select the linear transformation matrix of the output's WCS. This
option only takes two values: ‘pc’ (for the ‘PCi_j’ formalism) and
‘cd’ (for ‘CDi_j’). For more on the different formalisms, please
see Section 8.1 of the FITS standard(2), version 4.0.
In short, in the ‘PCi_j’ formalism, we only keep the linear
rotation matrix in these keywords and put the scaling factor (or
the pixel scale in astronomical imaging) in the ‘CDELTi’ keywords.
In the ‘CDi_j’ formalism, we blend the scaling into the rotation
into a single matrix and keep that matrix in these FITS keywords.
By default, Gnuastro uses the ‘PCi_j’ formalism, because it greatly
helps in human readability of the raw keywords and is also the
default mode of WCSLIB. However, in some circumstances it may be
necessary to have the keywords in the CD format; for example, when
you need to feed the outputs into other software that do not follow
the full FITS standard and only recognize the ‘CDi_j’ formalism.
‘txt’
A plain text table with white-space characters between the
columns (see *note Gnuastro text table format::).
‘fits-ascii’
A FITS ASCII table (see *note Recognized table formats::).
‘fits-binary’
A FITS binary table (see *note Recognized table formats::).
‘--outfitsnoconfig’
Do not write any of the program's metadata (option values or
versions and dates) into the 0-th HDU of the output FITS file, see
*note Output FITS files::.
‘--outfitsnodate’
Do not write the ‘DATE’ or ‘DATEUTC’ keywords into the 0-th HDU of
the output FITS file, see *note Output FITS files::.
‘--outfitsnocommit’
Do not write the ‘COMMIT’ keyword into the 0-th HDU of the output
FITS file, see *note Output FITS files::.
‘--outfitsnoversions’
Do not write the versions of any dependency software into the 0-th
HDU of the output FITS file, see *note Output FITS files::.
---------- Footnotes ----------
(1) With the ‘#’ character, CFITSIO will only read the desired HDU
into your memory, not all the existing HDUs in the fits file.
(2) <https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf>
File: gnuastro.info, Node: Processing options, Next: Operating mode options, Prev: Input output options, Up: Common options
4.1.2.2 Processing options
..........................
Some processing steps are common to several programs, so they are
defined as common options to all programs. Note that this class of
common options is thus necessarily less common between all the programs
than those described in *note Input output options::, or *note Operating
mode options:: options. Also, if they are irrelevant for a program,
these options will not display in the ‘--help’ output of the program.
‘--minmapsize=INT’
The minimum size (in bytes) to memory-map a processing/internal
array as a file (on the non-volatile HDD/SSD), and not use the
system's RAM. Before using this option, please read *note Memory
management::. By default processing arrays will only be
memory-mapped to a file when the RAM is full. With this option,
you can force the memory-mapping, even when there is enough RAM. To
ensure this default behavior, the pre-defined value to this option
is an extremely large value (larger than any existing RAM).
Please note that using a non-volatile file (in the HDD/SDD) instead
of RAM can significantly increase the program's running time,
especially on HDDs (where read/write is slower). Also, note that
the number of memory-mapped files that your kernel can support is
limited. So when this option is necessary, it is best to give it
values larger than 1 megabyte (‘--minmapsize=1000000’). You can
then decrease it for a specific program's invocation on a large
input after you see memory issues arise (for example, an error, or
the program not aborting and fully consuming your memory). If you
see randomly named files remaining in this directory when the
program finishes normally, please send us a bug report so we
address the problem, see *note Report a bug::.
*Limited number of memory-mapped files:* The operating system
kernels usually support a limited number of memory-mapped files.
Therefore never set ‘--minmapsize’ to zero or a small number of
bytes (so too many files are created). If the kernel capacity is
exceeded, the program will crash.
‘--quietmmap’
Do not print any message when an array is stored in non-volatile
memory (HDD/SSD) and not RAM, see the description of ‘--minmapsize’
(above) for more.
‘-Z INT[,INT[,...]]’
‘--tilesize=[,INT[,...]]’
The size of regular tiles for tessellation, see *note
Tessellation::. For each dimension an integer length (in units of
data-elements or pixels) is necessary. If the number of input
dimensions is different from the number of values given to this
option, the program will stop with an error. Values must be
separated by commas (<,>) and can also be fractions (for example,
‘4/2’). If they are fractions, the result must be an integer,
otherwise an error will be printed.
‘-M INT[,INT[,...]]’
‘--numchannels=INT[,INT[,...]]’
The number of channels for larger input tessellation, see *note
Tessellation::. The number and types of acceptable values are
similar to ‘--tilesize’. The only difference is that instead of
length, the integers values given to this option represent the
_number_ of channels, not their size.
‘-F FLT’
‘--remainderfrac=FLT’
The fraction of remainder size along all dimensions to add to the
first tile. See *note Tessellation:: for a complete description.
This option is only relevant if ‘--tilesize’ is not exactly
divisible by the input dataset's size in a dimension. If the
remainder size is larger than this fraction (compared to
‘--tilesize’), then the remainder size will be added with one
regular tile size and divided between two tiles at the start and
end of the given dimension.
‘--workoverch’
Ignore the channel borders for the high-level job of the given
application. As a result, while the channel borders are respected
in defining the small tiles (such that no tile will cross a channel
border), the higher-level program operation will ignore them, see
*note Tessellation::.
‘--checktiles’
Make a FITS file with the same dimensions as the input but each
pixel is replaced with the ID of the tile that it is associated
with. Note that the tile IDs start from 0. See *note
Tessellation:: for more on Tiling an image in Gnuastro.
‘--oneelempertile’
When showing the tile values (for example, with ‘--checktiles’, or
when the program's output is tessellated) only use one element for
each tile. This can be useful when only the relative values given
to each tile compared to the rest are important or need to be
checked. Since the tiles usually have a large number of pixels
within them the output will be much smaller, and so easier to read,
write, store, or send.
Note that when the full input size in any dimension is not exactly
divisible by the given ‘--tilesize’ in that dimension, the edge
tile(s) will have different sizes (in units of the input's size),
see ‘--remainderfrac’. But with this option, all displayed values
are going to have the (same) size of one data-element. Hence, in
such cases, the image proportions are going to be slightly
different with this option.
If your input image is not exactly divisible by the tile size and
you want one value per tile for some higher-level processing, all
is not lost though. You can see how many pixels were within each
tile (for example, to weight the values or discard some for later
processing) with Gnuastro's Statistics (see *note Statistics::) as
shown below. The output FITS file is going to have two extensions,
one with the median calculated on each tile and one with the number
of elements that each tile covers. You can then use the ‘where’
operator in *note Arithmetic:: to set the values of all tiles that
do not have the regular area to a blank value.
$ aststatistics --median --number --ontile input.fits \
--oneelempertile --output=o.fits
$ REGULAR_AREA=1600 # Check second extension of `o.fits'.
$ astarithmetic o.fits o.fits $REGULAR_AREA ne nan where \
-h1 -h2
Note that if ‘input.fits’ also has blank values, then the median on
tiles with blank values will also be ignored with the command above
(which is desirable).
‘--inteponlyblank’
When values are to be interpolated, only change the values of the
blank elements, keep the non-blank elements untouched.
‘--interpmetric=STR’
The metric to use for finding nearest neighbors. Currently it only
accepts the Manhattan (or taxicab) metric with ‘manhattan’, or the
radial metric with ‘radial’.
The Manhattan distance between two points is defined with
$|\Delta{x}|+|\Delta{y}|$. Thus the Manhattan metric has the
advantage of being fast, but at the expense of being less accurate.
The radial distance is the standard definition of distance in a
Euclidean space: $\sqrt{\Delta{x}^2+\Delta{y}^2}$. It is accurate,
but the multiplication and square root can slow down the
processing.
‘--interpnumngb=INT’
The number of nearby non-blank neighbors to use for interpolation.
File: gnuastro.info, Node: Operating mode options, Prev: Processing options, Up: Common options
4.1.2.3 Operating mode options
..............................
Another group of options that are common to all the programs in Gnuastro
are those to do with the general operation of the programs. The
explanation for those that are not only limited to Gnuastro but are
common to all GNU programs start with (GNU option).
‘--’
(GNU option) Stop parsing the command-line. This option can be
useful in scripts or when using the shell history. Suppose you
have a long list of options, and want to see if removing some of
them (to read from configuration files, see *note Configuration
files::) can give a better result. If the ones you want to remove
are the last ones on the command-line, you do not have to delete
them, you can just add ‘--’ before them and if you do not get what
you want, you can remove the ‘--’ and get the same initial result.
‘--usage’
(GNU option) Only print the options and arguments and abort. This
is very useful for when you know the what the options do, and have
just forgot their long/short identifiers, see *note --usage::.
‘-?’
‘--help’
(GNU option) Print all options with an explanation and abort.
Adding this option will print all the options in their short and
long formats, also displaying which ones need a value if they are
called (with an ‘=’ after the long format followed by a string
specifying the format, see *note Options::). A short explanation
is also given for what the option is for. The program will quit
immediately after the message is printed and will not do any form
of processing, see *note --help::.
‘-V’
‘--version’
(GNU option) Print a short message, showing the full name, version,
copyright information and program authors and abort. On the first
line, it will print the official name (not executable name) and
version number of the program. Following this is a blank line and
a copyright information. The program will not run.
‘-q’
‘--quiet’
Do not report steps. All the programs in Gnuastro that have
multiple major steps will report their steps for you to follow
while they are operating. If you do not want to see these reports,
you can call this option and only error/warning messages will be
printed. If the steps are done very fast (depending on the
properties of your input) disabling these reports will also
decrease running time.
‘--cite’
Print all necessary information to cite and acknowledge Gnuastro in
your published papers. With this option, the programs will print
the BibTeX entry to include in your paper for Gnuastro in general,
and the particular program's paper (if that program comes with a
separate paper). It will also print the necessary acknowledgment
statement to add in the respective section of your paper and it
will abort. For a more complete explanation, please see *note
Acknowledgments::.
Citations and acknowledgments are vital for the continued work on
Gnuastro. Gnuastro started, and is continued, based on separate
research projects. So if you find any of the tools offered in
Gnuastro to be useful in your research, please use the output of
this command to cite and acknowledge the program (and Gnuastro) in
your research paper. Thank you.
Gnuastro is still new, there is no separate paper only devoted to
Gnuastro yet. Therefore currently the paper to cite for Gnuastro
is the paper for NoiseChisel which is the first published paper
introducing Gnuastro to the astronomical community. Upon reaching
a certain point, a paper completely devoted to describing
Gnuastro's many functionalities will be published, see *note GNU
Astronomy Utilities 1.0::.
‘-P’
‘--printparams’
With this option, Gnuastro's programs will read your command-line
options and all the configuration files. If there is no problem
(like a missing parameter or a value in the wrong format or range)
and immediately before actually running, the programs will print
the full list of option names, values and descriptions, sorted and
grouped by context and abort. They will also report the version
number, the date they were configured on your system and the time
they were reported.
As an example, you can give your full command-line options and even
the input and output file names and finally just add ‘-P’ to check
if all the parameters are finely set. If everything is OK, you can
just run the same command (easily retrieved from the shell history,
with the top arrow key) and simply remove the last two characters
that showed this option.
No program will actually start its processing when this option is
called. The otherwise mandatory arguments for each program (for
example, input image or catalog files) are no longer required when
you call this option.
‘--config=STR’
Parse ‘STR’ as a configuration file name, immediately when this
option is confronted (see *note Configuration files::). The
‘--config’ option can be called multiple times in one run of any
Gnuastro program on the command-line or in the configuration files.
In any case, it will be immediately read (before parsing the rest
of the options on the command-line, or lines in a configuration
file). If the given file does not exist or cannot be read for any
reason, the program will print a warning and continue its
processing. The warning can be suppressed with ‘--quiet’.
Note that by definition, options on the command-line still take
precedence over those in any configuration file, including the
file(s) given to this option if they are called before it. Also
see ‘--lastconfig’ and ‘--onlyversion’ on how this option can be
used for reproducible results. You can use ‘--checkconfig’ (below)
to check/confirm the parsing of configuration files.
‘--checkconfig’
Print options and their values, within the command-line or
configuration files, as they are parsed (see *note Configuration
file precedence::). If an option has already been set, or is
ignored by the program, this option will also inform you with
special values like ‘--ALREADY-SET--’. Only options that are
parsed after this option are printed, so to see the parsing of all
input options, it is recommended to put this option immediately
after the program name before any other options.
This is a very good option to confirm where the value of each
option is has been defined in scenarios where there are multiple
configuration files (for debugging).
‘--config-prefix=STR’
Accept option names in configuration files that start with the
given prefix. Since order matters when reading custom
configuration files, this option should be called *before* the
‘--config’ option(s) that contain options with the given prefix.
This option does not affect the options within configuration files
that have the standard name (without a prefix).
This gives unique features to Gnuastro's configuration files,
especially in large pipelines. Let's demonstrate this with the
simple scenario below. You have multiple configuration files for
different instances of one program (let's assume ‘nc-a.conf’ and
‘nc-b.conf’). At the same time, want to load all the option
names/values into your shell as environment variables (for example
with ‘source’). This happens when you want to use the options as
shell variables in other parts of the your pipeline.
If the two configuration files have different values for the same
option (as shown below), and you don't use ‘--config-prefix’, the
shell will over-write the common option values between the
configuration files. But thanks to ‘--config-prefix’, you can give
a different prefix for the different instances of the same option
in different configuration files.
$ cat nc-a.conf
a_tilesize=20,20
$ cat nc-b.conf
b_tilesize=40,40
## Load configuration files as shell scripts (to define the
## option name and values as shell variables with values).
## Just note that 'source' only takes one file at a time.
$ for c in nc-*.conf; do source $c; done
$ astnoisechisel img.fits \
--config=nc-a.conf --config-prefix=a_
$ echo "NoiseChisel run with --tilesize=$a_tilesize"
$ astnoisechisel img.fits \
--config=nc-b.conf --config-prefix=b_
$ echo "NoiseChisel run with --tilesize=$b_tilesize"
‘-S’
‘--setdirconf’
Update the current directory configuration file for the Gnuastro
program and quit. The full set of command-line and configuration
file options will be parsed and options with a value will be
written in the current directory configuration file for this
program (see *note Configuration files::). If the configuration
file or its directory does not exist, it will be created. If a
configuration file exists it will be replaced (after it, and all
other configuration files have been read). In any case, the
program will not run.
This is the recommended method(1) to edit/set the configuration
file for all future calls to Gnuastro's programs. It will
internally check if your values are in the correct range and type
and save them according to the configuration file format, see *note
Configuration file format::. So if there are unreasonable values
to some options, the program will notify you and abort before
writing the final configuration file.
When this option is called, the otherwise mandatory arguments, for
example input image or catalog file(s), are no longer mandatory
(since the program will not run).
‘-U’
‘--setusrconf’
Update the user configuration file and quit (see *note
Configuration files::). See explanation under ‘--setdirconf’ for
more details.
‘--lastconfig’
This is the last configuration file that must be read. When this
option is confronted in any stage of reading the options (on the
command-line or in a configuration file), no other configuration
file will be parsed, see *note Configuration file precedence:: and
*note Current directory and User wide::. Like all on/off options,
on the command-line, this option does not take any values. But in
a configuration file, it takes the values of ‘0’ or ‘1’, see *note
Configuration file format::. If it is present in a configuration
file with a value of ‘0’, then all later occurrences of this option
will be ignored.
‘--onlyversion=STR’
Only run the program if Gnuastro's version is exactly equal to
‘STR’ (see *note Version numbering::). Note that it is not
compared as a number, but as a string of characters, so ‘0’, or
‘0.0’ and ‘0.00’ are different. If the running Gnuastro version is
different, then this option will report an error and abort as soon
as it is confronted on the command-line or in a configuration file.
If the running Gnuastro version is the same as ‘STR’, then the
program will run as if this option was not called.
This is useful if you want your results to be exactly reproducible
and not mistakenly run with an updated/newer or older version of
the program. Besides internal algorithmic/behavior changes in
programs, the existence of options or their names might change
between versions (especially in these earlier versions of
Gnuastro).
Hence, when using this option (probably in a script or in a
configuration file), be sure to call it before other options. The
benefit is that, when the version differs, the other options will
not be parsed and you, or your collaborators/users, will not get
errors saying an option in your configuration does not exist in the
running version of the program.
Here is one example of how this option can be used in conjunction
with the ‘--lastconfig’ option. Let's assume that you were
satisfied with the results of this command: ‘astnoisechisel
image.fits --snquant=0.95’ (along with various options set in
various configuration files). You can save the state of
NoiseChisel and reproduce that exact result on ‘image.fits’ later
by following these steps (the extra spaces, and <\>, are only for
easy readability, if you want to try it out, only one space between
each token is enough).
$ echo "onlyversion X.XX" > reproducible.conf
$ echo "lastconfig 1" >> reproducible.conf
$ astnoisechisel image.fits --snquant=0.95 -P \
>> reproducible.conf
‘--onlyversion’ was available from Gnuastro 0.0, so putting it
immediately at the start of a configuration file will ensure that
later, you (or others using different version) will not get a
non-recognized option error in case an option was added/removed.
‘--lastconfig’ will inform the installed NoiseChisel to not parse
any other configuration files. This is done because we do not want
the user's user-wide or system wide option values affecting our
results. Finally, with the third command, which has a ‘-P’ (short
for ‘--printparams’), NoiseChisel will print all the option values
visible to it (in all the configuration files) and the shell will
append them to ‘reproduce.conf’. Hence, you do not have to worry
about remembering the (possibly) different options in the different
configuration files.
Afterwards, if you run NoiseChisel as shown below (telling it to
read this configuration file with the ‘--config’ option). You can
be sure that there will either be an error (for version mismatch)
or it will produce exactly the same result that you got before.
$ astnoisechisel --config=reproducible.conf
‘--log’
Some programs can generate extra information about their outputs in
a log file. When this option is called in those programs, the log
file will also be printed. If the program does not generate a log
file, this option is ignored.
*‘--log’ is not thread-safe*: The log file usually has a fixed
name. Therefore if two simultaneous calls (with ‘--log’) of a
program are made in the same directory, the program will try to
write to he same file. This will cause problems like unreasonable
log file, undefined behavior, or a crash.
‘-N INT’
‘--numthreads=INT’
Use ‘INT’ CPU threads when running a Gnuastro program (see *note
Multi-threaded operations::). If the value is zero (‘0’), or this
option is not given on the command-line or any configuration file,
the value will be determined at run-time: the maximum number of
threads available to the system when you run a Gnuastro program.
Note that multi-threaded programming is only relevant to some
programs. In others, this option will be ignored.
---------- Footnotes ----------
(1) Alternatively, you can use your favorite text editor.
File: gnuastro.info, Node: Shell TAB completion, Next: Standard input, Prev: Common options, Up: Command-line
4.1.3 Shell TAB completion (highly customized)
----------------------------------------------
*Under development:* Gnuastro's TAB completion in Bash already greatly
improves usage of Gnuastro on the command-line, but still under
development and not yet complete. If you are interested to try it out,
please go ahead and activate it (as described below), we encourage this.
But please have in mind that there are known issues(1) and you may find
new issues. If you do, please get in touch with us as described in
*note Report a bug::. TAB completion is currently only implemented in
the following programs: Arithmetic, BuildProgram, ConvertType, Convolve,
CosmicCalculator, Crop, Fits and Table. For progress on this task,
please see Task 15799(2).
Bash provides a built-in feature called _programmable completion_(3)
to help increase interactive workflow efficiency and minimize the number
of keystrokes _and_ the need to memorize things. It is also known as
TAB completion, bash completion, auto-completion, or word completion.
Completion is activated by pressing <[TAB]> while you are typing a
command. For file arguments this is the default behavior already and
you have probably used it a lot with any command-line program.
Besides this simple/default mode, Bash also enables a high level of
customization features for its completion. These features have been
extensively used in Gnuastro to improve your work efficiency(4). For
example, if you are running ‘asttable’ (which only accepts files
containing a table), and you press <[TAB]>, it will only suggest files
containing tables. As another example, if an option needs image HDUs
within a FITS file, pressing <[TAB]> will only suggest the image HDUs
(and not other possibly existing HDUs that contain tables, or just
metadata). Just note that the file name has to be already given on the
command-line before reaching such options (that look into the contents
of a file).
But TAB completion is not limited to file types or contents.
Arguments/Options that take certain fixed string values will directly
suggest those strings with TAB, and completely ignore the file structure
(for example, spectral line names in *note Invoking astcosmiccal::)! As
another example, the option ‘--numthreads’ option (to specify the number
of threads to use by the program), will find the number of available
threads on the system, and suggest the possible numbers with a TAB!
To activate Gnuastro's custom TAB completion in Bash, you need to put
the following line in one of your Bash startup files (for example,
‘~/.bashrc’). If you installed Gnuastro using the steps of *note Quick
start::, you should have already done this (the command just after ‘sudo
make install’). For a list of (and discussion on) Bash startup files
and installation directories see *note Installation directory::. Of
course, if Gnuastro was installed in a custom location, replace the
'‘/usr/local’' part of the line below to the value that was given to
‘--prefix’ during Gnuastro's configuration(5).
# Enable Gnuastro's TAB completion
source /usr/local/share/gnuastro/completion.bash
After adding the line above in a Bash startup file, TAB completion
will always be activated in any new terminal. To see if it has been
activated, try it out with ‘asttable [TAB][TAB]’ and ‘astarithmetic
[TAB][TAB]’ in a directory that contains tables and images. The first
will only suggest the files with a table, and the second, only those
with an image.
*TAB completion only works with long option names:* As described above,
short options are much more complex to generalize, therefore TAB
completion is only available for long options. But do not worry! TAB
completion also involves option names, so if you just type
‘--a[TAB][TAB]’, you will get the list of options that start with an
‘--a’. Therefore as a side-effect of TAB completion, your commands will
be far more human-readable with minimal key strokes.
---------- Footnotes ----------
(1)
<http://savannah.gnu.org/bugs/index.php?group=gnuastro&category_id=128>
(2) <https://savannah.gnu.org/task/?15799>
(3)
<https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html>
(4) To learn how Gnuastro implements TAB completion in Bash, see
*note Bash programmable completion::.
(5) In case you do not know the installation directory of Gnuastro on
your system, you can find out with this command: ‘which astfits | sed
-e"s|/bin/astfits||"’
File: gnuastro.info, Node: Standard input, Next: Shell tips, Prev: Shell TAB completion, Up: Command-line
4.1.4 Standard input
--------------------
The most common way to feed the primary/first input dataset into a
program is to give its filename as an argument (discussed in *note
Arguments::). When you want to run a series of programs in sequence,
this means that each will have to keep the output of each program in a
separate file and re-type that file's name in the next command. This
can be very slow and frustrating (mis-typing a file's name).
To solve the problem, the founders of Unix defined pipes to directly
feed the output of one program (its "Standard output" stream) into the
"standard input" of a next program. This removes the need to make
temporary files between separate processes and became one of the best
demonstrations of the Unix-way, or Unix philosophy.
Every program has three streams identifying where it reads/writes
non-file inputs/outputs: _Standard input_, _Standard output_, and
_Standard error_. When a program is called alone, all three are
directed to the terminal that you are using. If it needs an input, it
will prompt you for one and you can type it in. Or, it prints its
results in the terminal for you to see.
For example, say you have a FITS table/catalog containing the B and V
band magnitudes (‘MAG_B’ and ‘MAG_V’ columns) of a selection of galaxies
along with many other columns. If you want to see only these two
columns in your terminal, can use Gnuastro's *note Table:: program like
below:
$ asttable cat.fits -cMAG_B,MAG_V
Through the Unix pipe mechanism, when the shell confronts the pipe
character (<|>), it connects the standard output of the program before
the pipe, to the standard input of the program after it. So it is
literally a "pipe": everything that you would see printed by the first
program on the command (without any pipe), is now passed to the second
program (and not seen by you).
To continue the previous example, let's say you want to see the B-V
color. To do this, you can pipe Table's output to AWK (a wonderful tool
for processing things like plain text tables):
$ asttable cat.fits -cMAG_B,MAG_V | awk '{print $1-$2}'
But understanding the distribution by visually seeing all the numbers
under each other is not too useful! You can therefore feed this single
column information into *note Statistics:: to give you a general feeling
of the distribution with the same command:
$ asttable cat.fits -cMAG_B,MAG_V | awk '{print $1-$2}' | aststatistics
Gnuastro's programs that accept input from standard input, only look
into the Standard input stream if there is no first argument. In other
words, arguments take precedence over Standard input. When no argument
is provided, the programs check if the standard input stream is already
full or not (output from another program is waiting to be used). If
data is present in the standard input stream, it is used.
When the standard input is empty, the program will wait
‘--stdintimeout’ micro-seconds for you to manually enter the first line
(ending with a new-line character, or the <ENTER> key, see *note Input
output options::). If it detects the first line in this time, there is
no more time limit, and you can manually write/type all the lines for as
long as it takes. To inform the program that Standard input has
finished, press <CTRL-D> after a new line. If the program does not
catch the first line before the time-out finishes, it will abort with an
error saying that no input was provided.
*Manual input in Standard input is discarded:* Be careful that when you
manually fill the Standard input, the data will be discarded once the
program finishes and reproducing the result will be impossible.
Therefore this form of providing input is only good for temporary tests.
*Standard input currently only for plain text:* Currently Standard input
only works for plain text inputs like the example above. We will later
allow FITS files into the programs through standard input also.
File: gnuastro.info, Node: Shell tips, Prev: Standard input, Up: Command-line
4.1.5 Shell tips
----------------
Gnuastro's programs are primarily meant to be run on the command-line
shell environment. In this section, we will review some useful tips and
tricks that can be helpful in the pipelines that you run.
* Menu:
* Separate shell variables for multiple outputs:: When you get values from one command.
* Truncating start of long string FITS keyword values:: When the end of the string matters.
File: gnuastro.info, Node: Separate shell variables for multiple outputs, Next: Truncating start of long string FITS keyword values, Prev: Shell tips, Up: Shell tips
4.1.5.1 Separate shell variables for multiple outputs
.....................................................
Sometimes your commands print multiple values and you want to use them
as different shell variables. Let's describe the problem (shown in the
box below) with an example (that you can reproduce without any external
data).
With the commands below, we'll first make a noisy ($\sigma=5$) image
($100\times100$ pixels) using *note Arithmetic::. Then, we'll
measure(1) its mean and standard deviation using *note Statistics::.
$ astarithmetic 100 100 2 makenew 5 mknoise-sigma -oimg.fits
$ aststatistics img.fits --mean --std
-3.10938611484039e-03 4.99607077069093e+00
*THE PROBLEM:* you want the first number printed above to be stored in a
shell variable called ‘my_mean’ and the second number to be stored as
the ‘my_std’ shell variable (you are free to choose any name!).
The first thing that may come to mind is to run Statistics two times,
and write the output into separate variables like below:
$ my_std=$(aststatistics img.fits --std) ## NOT SOLUTION! ##
$ my_mean=$(aststatistics img.fits --mean) ## NOT SOLUTION! ##
But this is not a good solution because as ‘img.fits’ becomes larger
(more pixels), the time it takes for Statistics to simply load the data
into memory can be significant. This will slow down your pipeline and
besides wasting your time, it contributes to global warming (by spending
energy on an un-necessary action; take this seriously because your
pipeline may scale up to involve thousands of large datasets)!
Furthermore, besides loading of the input data, Statistics (and Gnuastro
in general) is designed to do multiple measurements in one pass over the
data as much as possible (to further decrease Gnuastro's carbon
footprint). So when given ‘--mean --std’, it will measure both in one
pass over the pixels (not two passes!). In other words, in this case,
you get the two measurements for the cost of one.
How do you separate the values from the first ‘aststatistics’ command
above? One ugly way is to write the two-number output string into a
single shell variable and then separate, or tokenize, the string with
two subsequent commands like below:
$ meanstd=$(aststatistics img.fits --mean --std) ## NOT SOLUTION! ##
$ my_mean=$(echo $meanstd | awk '{print $1}') ## NOT SOLUTION! ##
$ my_std=$(echo $meanstd | awk '{print $2}') ## NOT SOLUTION! ##
*SOLUTION:* The solution is to formatted-print (‘printf’) the numbers as
shell variables definitions in a string, and evaluate (‘eval’) that
string as a command:
$ eval "$(aststatistics img.fits --mean --std \
| xargs printf "my_mean=%s; my_std=%s")"
Let's review the solution (in more detail):
1. We pipe the output into ‘xargs’(2) (extended arguments) which puts
the two numbers it gets from the pipe, as arguments for ‘printf’
(formatted print; because ‘printf’ doesn't take input from pipes).
2. Within the ‘printf’ call, we write the values after putting a
variable name and equal-sign, and in between them we put a <;> (as
if it was a shell command). The ‘%s’ tells ‘printf’ to print each
input as a string (not to interpret it as a number and loose
precision). Here is the output of this phase:
$ aststatistics img.fits --mean --std \
| xargs printf "my_mean=%s; my_std=%s"
my_mean=-3.10938611484039e-03; my_std=4.99607077069093e+00
3. But the output above is a string! To evaluate this string as a
command, we give it to the eval command like above.
After the solution above, you will have the two ‘my_mean’ and ‘my_std’
variables to use separately in your pipeline:
$ echo $my_mean
-3.10938611484039e-03
$ echo $my_std
4.99607077069093e+00
This ‘eval’-based solution has been tested in in GNU Bash, Dash and
Zsh and it works nicely in them (is "portable"). This is because the
constructs used here are pretty low-level (and widely available).
For examples usages of this technique, see the following sections:
*note Extracting a single spectrum and plotting it:: and *note Pseudo
narrow-band images::.
---------- Footnotes ----------
(1) The actual printed values by ‘aststatistics’ may slightly differ
for you. This is because of a different random number generator seed
used in ‘astarithmetic’. To get an exactly reproducible result, see
*note Generating random numbers::
(2) For more on ‘xargs’, see <https://en.wikipedia.org/wiki/Xargs>.
It will take the standard input (from the pipe in this scenario) and put
it as arguments of the next program (‘printf’ in this scenario). In
other words, it is good for programs that don't take input from standard
input (‘printf’ in this case; but also includes others like ‘cp’, ‘rm’,
or ‘echo’).
File: gnuastro.info, Node: Truncating start of long string FITS keyword values, Prev: Separate shell variables for multiple outputs, Up: Shell tips
4.1.5.2 Truncating start of long string FITS keyword values
...........................................................
When you want to put a string (not a number, for example a file name)
into the keyword value, if it is longer than 68 characters, CFITSIO is
going to truncate the end of the string. The number 68 is the maximum
allowable sting keyword length in the FITS standard(1). A robust way to
solve this problem is to break the keyword into multiple keywords and
continue the file name there. However, especially when dealing with
file names, it is usually the last few characters that you want to
preserve (the first ones are usually just basic operating system
locations).
Below, you can see the three necessary commands to optionally (when
the length is too long) truncate such long strings in GNU Bash. When
truncation is necessary, to inform the reader that the value has been
truncated, we'll put '‘...’' at the start of the string.
$ fname="/a/very/long/file/location"
$ if [ ${#fname} -gt 68 ]; then value="...${fname: -65}"; \
else value=$fname; \
fi
$ astfits image.fits --write=KEYNAME,"$value"
Here are the core handy constructs of Bash that we are using here:
‘${#fname}’
Returns the length of the value given to the ‘fname’ variable.
‘${fname: -65}’
Returns the last 65 characters in the value of the ‘fname’
variable.
---------- Footnotes ----------
(1) In the FITS standard, the full length of a keyword (including its
name) is 80 characters. The keyword name occupies 8 characters, which
is followed by an <=> (1 character). For strings, we need one SPACE
after the <=>, and the string should be enclosed in two single quotes.
Accounting for all of these, we get $80-8-1-1-2=68$ available
characters.
File: gnuastro.info, Node: Configuration files, Next: Getting help, Prev: Command-line, Up: Common program behavior
4.2 Configuration files
=======================
Each program needs a certain number of parameters to run. Supplying all
the necessary parameters each time you run the program is very
frustrating and prone to errors. Therefore all the programs read the
values for the necessary options you have not given in the command-line
from one of several plain text files (which you can view and edit with
any text editor). These files are known as configuration files and are
usually kept in a directory named ‘etc/’ according to the file system
hierarchy standard(1).
The thing to have in mind is that none of the programs in Gnuastro
keep any internal default value. All the values must either be stored
in one of the configuration files or explicitly called in the
command-line. In case the necessary parameters are not given through
any of these methods, the program will print a missing option error and
abort. The only exception to this is ‘--numthreads’, whose default
value is determined at run-time using the number of threads available to
your system, see *note Multi-threaded operations::. Of course, you can
still provide a default value for the number of threads at any of the
levels below, but if you do not, the program will not abort. Also note
that through automatic output name generation, the value to the
‘--output’ option is also not mandatory on the command-line or in the
configuration files for all programs which do not rely on that value as
an input(2), see *note Automatic output::.
* Menu:
* Configuration file format:: ASCII format of configuration file.
* Configuration file precedence:: Precedence of configuration files.
* Current directory and User wide:: Local and user configuration files.
* System wide:: System wide configuration files.
---------- Footnotes ----------
(1) <http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard>
(2) One example of a program that uses the value given to ‘--output’
as an input is ConvertType, this value specifies the type of the output
through the value to ‘--output’, see *note Invoking astconvertt::.
File: gnuastro.info, Node: Configuration file format, Next: Configuration file precedence, Prev: Configuration files, Up: Configuration files
4.2.1 Configuration file format
-------------------------------
The configuration files for each program have the standard program
executable name with a '‘.conf’' suffix. When you download the source
code, you can find them in the same directory as the source code of each
program, see *note Program source::.
Any line in the configuration file whose first non-white character is
a <#> is considered to be a comment and is ignored. An empty line is
also similarly ignored. The long name of the option should be used as
an identifier. The option name and option value should be separated by
any number of 'white-space' characters (space, tab or vertical tab) or
an equal (<=>). By default several space characters are used. If the
value of an option has space characters (most commonly for the ‘hdu’
option), then the full value can be enclosed in double quotation signs
(<">, similar to the example in *note Arguments and options::). If it
is an option without a value in the ‘--help’ output (on/off option, see
*note Options::), then the value should be ‘1’ if it is to be 'on' and
‘0’ otherwise.
In each non-commented and non-blank line, any text after the first
two words (option identifier and value) is ignored. If an option
identifier is not recognized in the configuration file, the name of the
file, the line number of the unrecognized option, and the unrecognized
identifier name will be reported and the program will abort. If a
parameter is repeated more more than once in the configuration files,
accepts only one value, and is not set on the command-line, then only
the first value will be used, the rest will be ignored.
You can build or edit any of the directories and the configuration
files yourself using any text editor. However, it is recommended to use
the ‘--setdirconf’ and ‘--setusrconf’ options to set default values for
the current directory or this user, see *note Operating mode options::.
With these options, the values you give will be checked before writing
in the configuration file. They will also print a set of commented
lines guiding the reader and will also classify the options based on
their context and write them in their logical order to be more
understandable.
File: gnuastro.info, Node: Configuration file precedence, Next: Current directory and User wide, Prev: Configuration file format, Up: Configuration files
4.2.2 Configuration file precedence
-----------------------------------
The option values in all the programs of Gnuastro will be filled in the
following order. If an option only takes one value which is given in an
earlier step, any value for that option in a later step will be ignored.
Note that if the ‘lastconfig’ option is specified in any step below, no
other configuration files will be parsed (see *note Operating mode
options::).
1. Command-line options, for a particular run of ProgramName.
2. ‘.gnuastro/astprogname.conf’ is parsed by ProgramName in the
current directory.
3. ‘.gnuastro/gnuastro.conf’ is parsed by all Gnuastro programs in the
current directory.
4. ‘$HOME/.local/etc/astprogname.conf’ is parsed by ProgramName in the
user's home directory (see *note Current directory and User
wide::).
5. ‘$HOME/.local/etc/gnuastro.conf’ is parsed by all Gnuastro programs
in the user's home directory (see *note Current directory and User
wide::).
6. ‘prefix/etc/astprogname.conf’ is parsed by ProgramName in the
system-wide installation directory (see *note System wide:: for
‘prefix’).
7. ‘prefix/etc/gnuastro.conf’ is parsed by all Gnuastro programs in
the system-wide installation directory (see *note System wide:: for
‘prefix’).
The basic idea behind setting this progressive state of checking for
parameter values is that separate users of a computer or separate
folders in a user's file system might need different values for some
parameters.
*Checking the order:* You can confirm/check the order of parsing
configuration files using the ‘--checkconfig’ option with any Gnuastro
program, see *note Operating mode options::. Just be sure to place this
option immediately after the program name, before any other option.
As you see above, there can also be a configuration file containing
the common options in all the programs: ‘gnuastro.conf’ (see *note
Common options::). If options specific to one program are specified in
this file, there will be unrecognized option errors, or unexpected
behavior if the option has different behavior in another program. On
the other hand, there is no problem with ‘astprogname.conf’ containing
common options(1).
*Manipulating the order:* You can manipulate this order or add new files
with the following two options which are fully described in *note
Operating mode options:::
‘--config’
Allows you to define any file to be parsed as a configuration file
on the command-line or within the any other configuration file.
Recall that the file given to ‘--config’ is parsed immediately when
this option is confronted (on the command-line or in a
configuration file).
‘--lastconfig’
Allows you to stop the parsing of subsequent configuration files.
Note that if this option is given in a configuration file, it will
be fully read, so its position in the configuration does not matter
(unlike ‘--config’).
One example of benefiting from these configuration files can be this:
raw telescope images usually have their main image extension in the
second FITS extension, while processed FITS images usually only have one
extension. If your system-wide default input extension is 0 (the
first), then when you want to work with the former group of data you
have to explicitly mention it to the programs every time. With this
progressive state of default values to check, you can set different
default values for the different directories that you would like to run
Gnuastro in for your different purposes, so you will not have to worry
about this issue any more.
The same can be said about the ‘gnuastro.conf’ files: by specifying a
behavior in this single file, all Gnuastro programs in the respective
directory, user, or system-wide steps will behave similarly. For
example, to keep the input's directory when no specific output is given
(see *note Automatic output::), or to not delete an existing file if it
has the same name as a given output (see *note Input output options::).
---------- Footnotes ----------
(1) As an example, the ‘--setdirconf’ and ‘--setusrconf’ options will
also write the common options they have read in their produced
‘astprogname.conf’.
File: gnuastro.info, Node: Current directory and User wide, Next: System wide, Prev: Configuration file precedence, Up: Configuration files
4.2.3 Current directory and User wide
-------------------------------------
For the current (local) and user-wide directories, the configuration
files are stored in the hidden sub-directories named ‘.gnuastro/’ and
‘$HOME/.local/etc/’ respectively. Unless you have changed it, the
‘$HOME’ environment variable should point to your home directory. You
can check it by running ‘$ echo $HOME’. Each time you run any of the
programs in Gnuastro, this environment variable is read and placed in
the above address. So if you suddenly see that your home configuration
files are not being read, probably you (or some other program) has
changed the value of this environment variable.
Although it might cause confusions like above, this dependence on the
‘HOME’ environment variable enables you to temporarily use a different
directory as your home directory. This can come in handy in complicated
situations. To set the user or current directory configuration files
based on your command-line input, you can use the ‘--setdirconf’ or
‘--setusrconf’, see *note Operating mode options::.
File: gnuastro.info, Node: System wide, Prev: Current directory and User wide, Up: Configuration files
4.2.4 System wide
-----------------
When Gnuastro is installed, the configuration files that are shipped
with the distribution are copied into the (possibly system wide)
‘prefix/etc/’ directory. For more details on ‘prefix’, see *note
Installation directory:: (by default it is: ‘/usr/local’). This
directory is the final place (with the lowest priority) that the
programs in Gnuastro will check to retrieve parameter values.
If you remove an option and its value from the system wide
configuration files, you either have to specify it in more immediate
configuration files or set it each time in the command-line. Recall
that none of the programs in Gnuastro keep any internal default values
and will abort if they do not find a value for the necessary parameters
(except the number of threads and output file name). So even though you
might never expect to use an optional option, it safe to have it
available in this system-wide configuration file even if you do not
intend to use it frequently.
Note that in case you install Gnuastro from your distribution's
repositories, ‘prefix’ will either be set to ‘/’ (the root directory) or
‘/usr’, so you can find the system wide configuration variables in
‘/etc/’ or ‘/usr/etc/’. The prefix of ‘/usr/local/’ is conventionally
used for programs you install from source by yourself as in *note Quick
start::.
File: gnuastro.info, Node: Getting help, Next: Multi-threaded operations, Prev: Configuration files, Up: Common program behavior
4.3 Getting help
================
Probably the first time you read this book, it is either in the PDF or
HTML formats. These two formats are very convenient for when you are
not actually working, but when you are only reading. Later on, when you
start to use the programs and you are deep in the middle of your work,
some of the details will inevitably be forgotten. Going to find the PDF
file (printed or digital) or the HTML web page is a major distraction.
GNU software have a very unique set of tools for aiding your memory
on the command-line, where you are working, depending how much of it you
need to remember. In the past, such command-line help was known as
"online" help, because they were literally provided to you 'on' the
command 'line'. However, nowadays the word "online" refers to something
on the internet, so that term will not be used. With this type of help,
you can resume your exciting research without taking your hands off the
keyboard.
Another major advantage of such command-line based help routines is
that they are installed with the software in your computer, therefore
they are always in sync with the executable you are actually running.
Three of them are actually part of the executable. You do not have to
worry about the version of the book or program. If you rely on external
help (a PDF in your personal print or digital archive or HTML from the
official web page) you have to check to see if their versions fit with
your installed program.
If you only need to remember the short or long names of the options,
‘--usage’ is advised. If it is what the options do, then ‘--help’ is a
great tool. Man pages are also provided for those who are use to this
older system of documentation. This full book is also available to you
on the command-line in Info format. If none of these seems to resolve
the problems, there is a mailing list which enables you to get in touch
with experienced Gnuastro users. In the subsections below each of these
methods are reviewed.
* Menu:
* --usage:: View option names and value formats.
* --help:: List all options with description.
* Man pages:: Man pages generated from -help.
* Info:: View complete book in terminal.
* help-gnuastro mailing list:: Contacting experienced users.
Browse
Usage
Versions