gnuastro (0.22)
This is gnuastro.info, produced by makeinfo version 7.1 from
gnuastro.texi.
This book documents version 0.22 of the GNU Astronomy Utilities
(Gnuastro). Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.
Copyright © 2015-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License,
Version 1.3 or any later version published by the Free Software
Foundation; with no Invariant Sections, no Front-Cover Texts, and
no Back-Cover Texts. A copy of the license is included in the
section entitled "GNU Free Documentation License".
INFO-DIR-SECTION Astronomy
START-INFO-DIR-ENTRY
* Gnuastro: (gnuastro). GNU Astronomy Utilities.
* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc.
* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help.
* bug-gnuastro: (gnuastro)Report a bug. How to report bugs
* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels.
* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic.
* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro's library.
* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram.
* ConvertType: (gnuastro)ConvertType. Convert different file types.
* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType.
* Convolve: (gnuastro)Convolve. Convolve an input file with kernel.
* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve.
* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params.
* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator.
* Crop: (gnuastro)Crop. Crop region(s) from image(s).
* astcrop: (gnuastro)Invoking astcrop. Options to Crop.
* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords.
* astfits: (gnuastro)Invoking astfits. Options to Fits.
* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image.
* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog.
* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles.
* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles.
* Match: (gnuastro)Match. Match two separate catalogs.
* astmatch: (gnuastro)Invoking astmatch. Options to Match.
* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise.
* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel.
* Segment: (gnuastro)Segment. Segment detections based on signal structure.
* astsegment: (gnuastro)Invoking astsegment. Options to Segment.
* Query: (gnuastro)Query. Access remote databases for downloading data.
* astquery: (gnuastro)Invoking astquery. Options to Query.
* Statistics: (gnuastro)Statistics. Get image Statistics.
* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics.
* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables.
* asttable: (gnuastro)Invoking asttable. Options to Table.
* Warp: (gnuastro)Warp. Warp a dataset to a new grid.
* astwarp: (gnuastro)Invoking astwarp. Options to Warp.
* astscript: (gnuastro)Installed scripts. Gnuastro's installed scripts.
* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script
* astscript-fits-view: (gnuastro)Invoking astscript-fits-view. Options to this script
* astscript-pointing-simulate: (gnuastro)Invoking astscript-pointing-simulate. Options to this script
* astscript-psf-scale-factor: (gnuastro)Invoking astscript-psf-scale-factor. Options to this script
* astscript-psf-select-stars: (gnuastro)Invoking astscript-psf-select-stars. Options to this script
* astscript-psf-stamp: (gnuastro)Invoking astscript-psf-stamp. Options to this script
* astscript-psf-subtract: (gnuastro)Invoking astscript-psf-subtract. Options to this script
* astscript-psf-unite: (gnuastro)Invoking astscript-psf-unite. Options to this script
* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script
* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script
* astscript-zeropoint: (gnuastro)Invoking astscript-zeropoint. Options to this script
END-INFO-DIR-ENTRY
File: gnuastro.info, Node: --usage, Next: --help, Prev: Getting help, Up: Getting help
4.3.1 ‘--usage’
---------------
If you give this option, the program will not run. It will only print a
very concise message showing the options and arguments. Everything
within square brackets (‘[]’) is optional. For example, here are the
first and last two lines of Crop's ‘--usage’ is shown:
$ astcrop --usage
Usage: astcrop [-Do?IPqSVW] [-d INT] [-h INT] [-r INT] [-w INT]
[-x INT] [-y INT] [-c INT] [-p STR] [-N INT] [--deccol=INT]
....
[--setusrconf] [--usage] [--version] [--wcsmode]
[ASCIIcatalog] FITSimage(s).fits
There are no explanations on the options, just their short and long
names shown separately. After the program name, the short format of all
the options that do not require a value (on/off options) is displayed.
Those that do require a value then follow in separate brackets, each
displaying the format of the input they want, see *note Options::.
Since all options are optional, they are shown in square brackets, but
arguments can also be optional. For example, in this example, a catalog
name is optional and is only required in some modes. This is a standard
method of displaying optional arguments for all GNU software.
File: gnuastro.info, Node: --help, Next: Man pages, Prev: --usage, Up: Getting help
4.3.2 ‘--help’
--------------
If the command-line includes this option, the program will not be run.
It will print a complete list of all available options along with a
short explanation. The options are also grouped by their context.
Within each context, the options are sorted alphabetically. Since the
options are shown in detail afterwards, the first line of the ‘--help’
output shows the arguments and if they are optional or not, similar to
*note --usage::.
In the ‘--help’ output of all programs in Gnuastro, the options for
each program are classified based on context. The first two contexts
are always options to do with the input and output respectively. For
example, input image extensions or supplementary input files for the
inputs. The last class of options is also fixed in all of Gnuastro, it
shows operating mode options. Most of these options are already
explained in *note Operating mode options::.
The help message will sometimes be longer than the vertical size of
your terminal. If you are using a graphical user interface terminal
emulator, you can scroll the terminal with your mouse, but we promised
no mice distractions! So here are some suggestions:
• <Shift + PageUP> to scroll up and <Shift + PageDown> to scroll
down. For most help output this should be enough. The problem is
that it is limited by the number of lines that your terminal keeps
in memory and that you cannot scroll by lines, only by whole
screens.
• Pipe to ‘less’. A pipe is a form of shell re-direction. The
‘less’ tool in Unix-like systems was made exactly for such outputs
of any length. You can pipe (‘|’) the output of any program that
is longer than the screen to it and then you can scroll through (up
and down) with its many tools. For example:
$ astnoisechisel --help | less
Once you have gone through the text, you can quit ‘less’ by
pressing the <q> key.
• Redirect to a file. This is a less convenient way, because you
will then have to open the file in a text editor! You can do this
with the shell redirection tool (‘>’):
$ astnoisechisel --help > filename.txt
In case you have a special keyword you are looking for in the help,
you do not have to go through the full list. GNU Grep is made for this
job. For example, if you only want the list of options whose ‘--help’
output contains the word "axis" in Crop, you can run the following
command:
$ astcrop --help | grep axis
If the output of this option does not fit nicely within the confines
of your terminal, GNU does enable you to customize its output through
the environment variable ‘ARGP_HELP_FMT’, you can set various parameters
which specify the formatting of the help messages. For example, if your
terminals are wider than 70 spaces (say 100) and you feel there is too
much empty space between the long options and the short explanation, you
can change these formats by giving values to this environment variable
before running the program with the ‘--help’ output. You can define
this environment variable in this manner:
$ export ARGP_HELP_FMT=rmargin=100,opt-doc-col=20
This will affect all GNU programs using GNU C library's ‘argp.h’
facilities as long as the environment variable is in memory. You can
see the full list of these formatting parameters in the "Argp User
Customization" part of the GNU C library manual. If you are more
comfortable to read the ‘--help’ outputs of all GNU software in your
customized format, you can add your customization (similar to the line
above, without the ‘$’ sign) to your ‘~/.bashrc’ file. This is a
standard option for all GNU software.
File: gnuastro.info, Node: Man pages, Next: Info, Prev: --help, Up: Getting help
4.3.3 Man pages
---------------
Man pages were the Unix method of providing command-line documentation
to a program. With GNU Info, see *note Info:: the usage of this method
of documentation is highly discouraged. This is because Info provides a
much more easier to navigate and read environment.
However, some operating systems require a man page for packages that
are installed and some people are still used to this method of
command-line help. So the programs in Gnuastro also have Man pages
which are automatically generated from the outputs of ‘--version’ and
‘--help’ using the GNU help2man program. So if you run
$ man programname
You will be provided with a man page listing the options in the standard
manner.
File: gnuastro.info, Node: Info, Next: help-gnuastro mailing list, Prev: Man pages, Up: Getting help
4.3.4 Info
----------
Info is the standard documentation format for all GNU software. It is a
very useful command-line document viewing format, fully equipped with
links between the various pages and menus and search capabilities. As
explained before, the best thing about it is that it is available for
you the moment you need to refresh your memory on any command-line tool
in the middle of your work without having to take your hands off the
keyboard. This complete book is available in Info format and can be
accessed from anywhere on the command-line.
To open the Info format of any installed programs or library on your
system which has an Info format book, you can simply run the command
below (change ‘executablename’ to the executable name of the program or
library):
$ info executablename
In case you are not already familiar with it, run ‘$ info info’. It
does a fantastic job in explaining all its capabilities itself. It is
very short and you will become sufficiently fluent in about half an
hour. Since all GNU software documentation is also provided in Info,
your whole GNU/Linux life will significantly improve.
Once you've become an efficient navigator in Info, you can go to any
part of this book or any other GNU software or library manual, no matter
how long it is, in a matter of seconds. It also blends nicely with GNU
Emacs (a text editor) and you can search manuals while you are writing
your document or programs without taking your hands off the keyboard,
this is most useful for libraries like the GNU C library. To be able to
access all the Info manuals installed in your GNU/Linux within Emacs,
type <Ctrl-H + i>.
To see this whole book from the beginning in Info, you can run
$ info gnuastro
If you run Info with the particular program executable name, for example
‘astcrop’ or ‘astnoisechisel’:
$ info astprogramname
you will be taken to the section titled "Invoking ProgramName" which
explains the inputs and outputs along with the command-line options for
that program. Finally, if you run Info with the official program name,
for example, Crop or NoiseChisel:
$ info ProgramName
you will be taken to the top section which introduces the program. Note
that in all cases, Info is not case sensitive.
File: gnuastro.info, Node: help-gnuastro mailing list, Prev: Info, Up: Getting help
4.3.5 help-gnuastro mailing list
--------------------------------
Gnuastro maintains the help-gnuastro mailing list for users to ask any
questions related to Gnuastro. The experienced Gnuastro users and some
of its developers are subscribed to this mailing list and your email
will be sent to them immediately. However, when contacting this mailing
list please have in mind that they are possibly very busy and might not
be able to answer immediately.
To ask a question from this mailing list, send a mail to
‘help-gnuastro@gnu.org’. Anyone can view the mailing list archives at
<http://lists.gnu.org/archive/html/help-gnuastro/>. It is best that
before sending a mail, you search the archives to see if anyone has
asked a question similar to yours. If you want to make a suggestion or
report a bug, please do not send a mail to this mailing list. We have
other mailing lists and tools for those purposes, see *note Report a
bug:: or *note Suggest new feature::.
File: gnuastro.info, Node: Multi-threaded operations, Next: Numeric data types, Prev: Getting help, Up: Common program behavior
4.4 Multi-threaded operations
=============================
Some of the programs benefit significantly when you use all the threads
your computer's CPU has to offer to your operating system. The number
of threads available can be larger than the number of physical
(hardware) cores in the CPU (also known as Simultaneous multithreading).
For example, in Intel's CPUs (those that implement its Hyper-threading
technology) the number of threads is usually double the number of
physical cores in your CPU. On a GNU/Linux system, the number of threads
available can be found with the command ‘$ nproc’ command (part of GNU
Coreutils).
Gnuastro's programs can find the number of threads available to your
system internally at run-time (when you execute the program). However,
if a value is given to the ‘--numthreads’ option, the given number will
be used, see *note Operating mode options:: and *note Configuration
files:: for ways to use this option. Thus ‘--numthreads’ is the only
common option in Gnuastro's programs with a value that does not have to
be specified anywhere on the command-line or in the configuration files.
* Menu:
* A note on threads:: Caution and suggestion on using threads.
* How to run simultaneous operations:: How to run things simultaneously.
File: gnuastro.info, Node: A note on threads, Next: How to run simultaneous operations, Prev: Multi-threaded operations, Up: Multi-threaded operations
4.4.1 A note on threads
-----------------------
Spinning off threads is not necessarily the most efficient way to run an
application. Creating a new thread is not a cheap operation for the
operating system. It is most useful when the input data are fixed and
you want the same operation to be done on parts of it. For example, one
input image to Crop and multiple crops from various parts of it. In
this fashion, the image is loaded into memory once, all the crops are
divided between the number of threads internally and each thread cuts
out those parts which are assigned to it from the same image. On the
other hand, if you have multiple images and you want to crop the same
region(s) out of all of them, it is much more efficient to set
‘--numthreads=1’ (so no threads spin off) and run Crop multiple times
simultaneously, see *note How to run simultaneous operations::.
You can check the boost in speed by first running a program on one of
the data sets with the maximum number of threads and another time (with
everything else the same) and only using one thread. You will notice
that the wall-clock time (reported by most programs at their end) in the
former is longer than the latter divided by number of physical CPU cores
(not threads) available to your operating system. Asymptotically these
two times can be equal (most of the time they are not). So limiting the
programs to use only one thread and running them independently on the
number of available threads will be more efficient.
Note that the operating system keeps a cache of recently processed
data, so usually, the second time you process an identical data set
(independent of the number of threads used), you will get faster
results. In order to make an unbiased comparison, you have to first
clean the system's cache with the following command between the two
runs.
$ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
*SUMMARY: Should I use multiple threads?* Depends:
• If you only have *one* data set (image in most cases!), then yes,
the more threads you use (with a maximum of the number of threads
available to your OS) the faster you will get your results.
• If you want to run the same operation on *multiple* data sets, it
is best to set the number of threads to 1 and use Make, or GNU
Parallel, as explained in *note How to run simultaneous
operations::.
File: gnuastro.info, Node: How to run simultaneous operations, Prev: A note on threads, Up: Multi-threaded operations
4.4.2 How to run simultaneous operations
----------------------------------------
There are two(1) approaches to simultaneously execute a program: using
GNU Parallel or Make (GNU Make is the most common implementation). The
first is very useful when you only want to do one job multiple times and
want to get back to your work without actually keeping the command you
ran. The second is usually for more important operations, with lots of
dependencies between the different products (for example, a full
scientific research).
GNU Parallel
When you only want to run multiple instances of a command on
different threads and get on with the rest of your work, the best
method is to use GNU parallel. Surprisingly GNU Parallel is one of
the few GNU packages that has no Info documentation but only a Man
page, see *note Info::. So to see the documentation after
installing it please run
$ man parallel
As an example, let's assume we want to crop a region fixed on the
pixels (500, 600) with the default width from all the FITS images
in the ‘./data’ directory ending with ‘sci.fits’ to the current
directory. To do this, you can run:
$ parallel astcrop --numthreads=1 --xc=500 --yc=600 ::: \
./data/*sci.fits
GNU Parallel can help in many more conditions, this is one of the
simplest, see the man page for lots of other examples. For
absolute beginners: the backslash (‘\’) is only a line breaker to
fit nicely in the page. If you type the whole command in one line,
you should remove it.
Make
Make is a program for building "targets" (e.g., files) using
"recipes" (a set of operations) when their known "prerequisites"
(other files) have been updated. It elegantly allows you to define
dependency structures for building your final output and updating
it efficiently when the inputs change. It is the most common
infra-structure to build software today.
Scientific research methodology is very similar to software
development: you start by testing a hypothesis on a small sample of
objects/targets with a simple set of steps. As you are able to get
promising results, you improve the method and use it on a larger,
more general, sample. In the process, you will confront many
issues that have to be corrected (bugs in software development
jargon). Make is a wonderful tool to manage this style of
development.
Besides the raw data analysis pipeline, Make has been used to for
producing reproducible papers, for example, see the reproduction
pipeline (https://gitlab.com/makhlaghi/NoiseChisel-paper) of the
paper introducing *note NoiseChisel:: (one of Gnuastro's programs).
In fact the NoiseChisel paper's Make-based workflow was the
foundation of a parallel project called Maneage
(http://maneage.org) (_Man_aging data lin_eage_):
<http://maneage.org> that is described more fully in Akhlaghi et
al. 2021 (https://arxiv.org/abs/2006.03018). Therefore, it is a
very useful tool for complex scientific workflows.
GNU Make(2) is the most common implementation which (similar to
nearly all GNU programs, comes with a wonderful manual(3)). Make
is very basic and simple, and thus the manual is short (the most
important parts are in the first roughly 100 pages) and easy to
read/understand.
Make comes with a ‘--jobs’ (‘-j’) option which allows you to
specify the maximum number of jobs that can be done simultaneously.
For example, if you have 8 threads available to your operating
system. You can run:
$ make -j8
With this command, Make will process your ‘Makefile’ and create all
the targets (can be thousands of FITS images for example)
simultaneously on 8 threads, while fully respecting their
dependencies (only building a file/target when its prerequisites
are successfully built). Make is thus strongly recommended for
managing scientific research where robustness, archiving,
reproducibility and speed(4) are important.
---------- Footnotes ----------
(1) A third way would be to open multiple terminal emulator windows
in your GUI, type the commands separately on each and press <Enter> once
on each terminal, but this is far too frustrating, tedious and prone to
errors. It's therefore not a realistic solution when tens, hundreds or
thousands of operations (your research targets, multiplied by the
operations you do on each) are to be done.
(2) <https://www.gnu.org/software/make/>
(3) <https://www.gnu.org/software/make/manual/>
(4) Besides its multi-threaded capabilities, Make will only rebuild
those targets that depend on a change you have made, not the whole work.
For example, if you have set the prerequisites properly, you can easily
test the changing of a parameter on your paper's results without having
to re-do everything (which is much faster). This allows you to be much
more productive in easily checking various ideas/assumptions of the
different stages of your research and thus produce a more robust result
for your exciting science.
File: gnuastro.info, Node: Numeric data types, Next: Memory management, Prev: Multi-threaded operations, Up: Common program behavior
4.5 Numeric data types
======================
At the lowest level, the computer stores everything in terms of ‘1’ or
‘0’. For example, each program in Gnuastro, or each astronomical image
you take with the telescope is actually a string of millions of these
zeros and ones. The space required to keep a zero or one is the
smallest unit of storage, and is known as a _bit_. However,
understanding and manipulating this string of bits is extremely hard for
most people. Therefore, different standards are defined to package the
bits into separate _type_s with a fixed interpretation of the bits in
each package.
To store numbers, the most basic standard/type is for integers ($...,
-2, -1, 0, 1, 2, ...$). The common integer types are 8, 16, 32, and 64
bits wide (more bits will give larger limits). Each bit corresponds to
a power of 2 and they are summed to create the final number. In the
integer types, for each width there are two standards for reading the
bits: signed and unsigned. In the 'signed' convention, one bit is
reserved for the sign (stating that the integer is positive or
negative). The 'unsigned' integers use that bit in the actual number
and thus contain only positive numbers (starting from zero).
Therefore, at the same number of bits, both signed and unsigned
integers can allow the same number of integers, but the positive limit
of the ‘unsigned’ types is double their ‘signed’ counterparts with the
same width (at the expense of not having negative numbers). When the
context of your work does not involve negative numbers (for example,
counting, where negative is not defined), it is best to use the
‘unsigned’ types. For the full numerical range of all integer types,
see below.
Another standard of converting a given number of bits to numbers is
the floating point standard, this standard can _approximately_ store any
real number with a given precision. There are two common floating point
types: 32-bit and 64-bit, for single and double precision floating point
numbers respectively. The former is sufficient for data with less than
8 significant decimal digits (most astronomical data), while the latter
is good for less than 16 significant decimal digits. The representation
of real numbers as bits is much more complex than integers. If you are
interested to learn more about it, you can start with the Wikipedia
article (https://en.wikipedia.org/wiki/Floating_point).
Practically, you can use Gnuastro's Arithmetic program to
convert/change the type of an image/datacube (see *note Arithmetic::),
or Gnuastro Table program to convert a table column's data type (see
*note Column arithmetic::). Conversion of a dataset's type is necessary
in some contexts. For example, the program/library, that you intend to
feed the data into, only accepts floating point values, but you have an
integer image/column. Another situation that conversion can be helpful
is when you know that your data only has values that fit within ‘int8’
or ‘uint16’. However it is currently formatted in the ‘float64’ type.
The important thing to consider is that operations involving wider,
floating point, or signed types can be significantly slower than
smaller-width, integer, or unsigned types respectively. Note that
besides speed, a wider type also requires much more storage space (by 4
or 8 times). Therefore, when you confront such situations that can be
optimized and want to store/archive/transfer the data, it is best to use
the most efficient type. For example, if your dataset (image or table
column) only has positive integers less than 65535, store it as an
unsigned 16-bit integer for faster processing, faster transfer, and less
storage space.
The short and long names for the recognized numeric data types in
Gnuastro are listed below. Both short and long names can be used when
you want to specify a type. For example, as a value to the common
option ‘--type’ (see *note Input output options::), or in the
information comment lines of *note Gnuastro text table format::. The
ranges listed below are inclusive.
‘u8’
‘uint8’
8-bit unsigned integers, range:
$[0\rm{\ to\ }2^8-1]$ or $[0\rm{\ to\ }255]$.
‘i8’
‘int8’
8-bit signed integers, range:
$[-2^7\rm{\ to\ }2^7-1]$ or $[-128\rm{\ to\ }127]$.
‘u16’
‘uint16’
16-bit unsigned integers, range:
$[0\rm{\ to\ }2^{16}-1]$ or $[0\rm{\ to\ }65535]$.
‘i16’
‘int16’
16-bit signed integers, range:
$[-2^{15}\rm{\ to\ }2^{15}-1]$ or $[-32768\rm{\ to\ }32767]$.
‘u32’
‘uint32’
32-bit unsigned integers, range:
$[0\rm{\ to\ }2^{32}-1]$ or $[0\rm{\ to\ }4294967295]$.
‘i32’
‘int32’
32-bit signed integers, range:
$[-2^{31}\rm{\ to\ }2^{31}-1]$ or $[-2147483648\rm{\ to\
}2147483647]$.
‘u64’
‘uint64’
64-bit unsigned integers, range
$[0\rm{\ to\ }2^{64}-1]$ or $[0\rm{\ to\ }18446744073709551615]$.
‘i64’
‘int64’
64-bit signed integers, range:
$[-2^{63}\rm{\ to\ }2^{63}-1]$ or $[-9223372036854775808\rm{\ to\
}9223372036854775807]$.
‘f32’
‘float32’
32-bit (single-precision) floating point types. The maximum
(minimum is its negative) possible value is
$3.402823\times10^{38}$. Single-precision floating points can
accurately represent a floating point number up to $\sim7.2$
significant decimals. Given the heavy noise in astronomical data,
this is usually more than sufficient for storing results. For
more, see *note Printing floating point numbers::.
‘f64’
‘float64’
64-bit (double-precision) floating point types. The maximum
(minimum is its negative) possible value is $\sim10^{308}$.
Double-precision floating points can accurately represent a
floating point number $\sim15.9$ significant decimals. This is
usually good for processing (mixing) the data internally, for
example, a sum of single precision data (and later storing the
result as ‘float32’). For more, see *note Printing floating point
numbers::.
*Some file formats do not recognize all types.* for example, the FITS
standard (see *note Fits::) does not define ‘uint64’ in binary tables or
images. When a type is not acceptable for output into a given file
format, the respective Gnuastro program or library will let you know and
abort. On the command-line, you can convert the numerical type of an
image, or table column into another type with *note Arithmetic:: or
*note Table:: respectively. If you are writing your own program, you
can use the ‘gal_data_copy_to_new_type()’ function in Gnuastro's
library, see *note Copying datasets::.
File: gnuastro.info, Node: Memory management, Next: Tables, Prev: Numeric data types, Up: Common program behavior
4.6 Memory management
=====================
In this section we will review how Gnuastro manages your input data in
your system's memory. Knowing this can help you optimize your usage (in
speed and memory consumption) when the data volume is large and
approaches, or exceeds, your available RAM (usually in various calls to
multiple programs simultaneously). But before diving into the details,
let's have a short basic introduction to memory in general and in
particular the types of memory most relevant to this discussion.
Input datasets (that are later fed into programs for analysis) are
commonly first stored in _non-volatile memory_. This is a type of
memory that does not need a constant power supply to keep the data and
is therefore primarily aimed for long-term storage, like HDDs or SSDs.
So data in this type of storage is preserved when you turn off your
computer. But by its nature, non-volatile memory is much slower, in
reading or writing, than the speeds that CPUs can process the data.
Thus relying on this type of memory alone would create a bad bottleneck
in the input/output (I/O) phase of any processing.
The first step to decrease this bottleneck is to have a faster
storage space, but with a much limited storage volume. For this type of
storage, computers have a Random Access Memory (or RAM). RAM is
classified as a _volatile memory_ because it needs a constant flow of
electricity to keep the information. In other words, the moment power
is cut-off, all the stored information in your RAM is gone (hence the
"volatile" name). But thanks to that constant supply of power, it can
access any random address with equal (and very high!) speed.
Hence, the general/simplistic way that programs deal with memory is
the following (this is general to almost all programs, not just
Gnuastro's): 1) Load/copy the input data from the non-volatile memory
into RAM. 2) Use the copy of the data in RAM as input for all the
internal processing as well as the intermediate data that is necessary
during the processing. 3) Finally, when the analysis is complete, write
the final output data back into non-volatile memory, and free/delete all
the used space in the RAM (the initial copy and all the intermediate
data). Usually the RAM is most important for the data of the
intermediate steps (that you never see as a user of a program!).
When the input dataset(s) to a program are small (compared to the
available space in your system's RAM at the moment it is run) Gnuastro's
programs and libraries follow the standard series of steps above. The
only exception is that deleting the intermediate data is not only done
at the end of the program. As soon as an intermediate dataset is no
longer necessary for the next internal steps, the space it occupied is
deleted/freed. This allows Gnuastro programs to minimize their usage of
your system's RAM over the full running time.
The situation gets complicated when the datasets are large (compared
to your available RAM when the program is run). For example, if a
dataset is half the size of your system's available RAM, and the
program's internal analysis needs three or more intermediately processed
copies of it at one moment in its analysis. There will not be enough
RAM to keep those higher-level intermediate data. In such cases,
programs that do not do any memory management will crash. But
fortunately Gnuastro's programs do have a memory management plans for
such situations.
When the necessary amount of space for an intermediate dataset cannot
be allocated in the RAM, Gnuastro's programs will not use the RAM at
all. They will use the "memory-mapped file" concept in modern operating
systems to create a randomly-named file in your non-volatile memory and
use that instead of the RAM. That file will have the exact size (in
bytes) of that intermediate dataset. Any time the program needs that
intermediate dataset, the operating system will directly go to that
file, and bypass your RAM. As soon as that file is no longer necessary
for the analysis, it will be deleted. But as mentioned above,
non-volatile memory has much slower I/O speed than the RAM. Hence in
such situations, the programs will become noticeably slower (sometimes
by factors of 10 times slower, depending on your non-volatile memory
speed).
Because of the drop in I/O speed (and thus the speed of your running
program), the moment that any to-be-allocated dataset is memory-mapped,
Gnuastro's programs and libraries will notify you with a descriptive
statement like below (can happen in any phase of their analysis). It
shows the location of the memory-mapped file, its size, complemented
with a small description of the cause, a pointer to this section of the
book for more information on how to deal with it (if necessary), and
what to do to suppress it.
astarithmetic: ./gnuastro_mmap/Fu7Dhs: temporary memory-mapped file
(XXXXXXXXXXX bytes) created for intermediate data that is not stored
in RAM (see the "Memory management" section of Gnuastro's manual for
optimizing your project's memory management, and thus speed). To
disable this warning, please use the option '--quiet-mmap'
Finally, when the intermediate dataset is no longer necessary, the
program will automatically delete it and notify you with a statement
like this:
astarithmetic: ./gnuastro_mmap/Fu7Dhs: deleted
To disable these messages, you can run the program with ‘--quietmmap’,
or set the ‘quietmmap’ variable in the allocating library function to be
non-zero.
An important component of these messages is the name of the
memory-mapped file. Knowing that the file has been deleted is important
for the user if the program crashes for any reason: internally (for
example, a parameter is given wrongly) or externally (for example, you
mistakenly kill the running job). In the event of a crash, the
memory-mapped files will not be deleted and you have to manually delete
them because they are usually large and they may soon fill your full
storage if not deleted in a long time due to successive crashes.
This brings us to managing the memory-mapped files in your
non-volatile memory. In other words: knowing where they are saved, or
intentionally placing them in different places of your file system, or
deleting them when necessary. As the examples above show, memory-mapped
files are stored in a sub-directory of the running directory called
‘gnuastro_mmap’. If this directory does not exist, Gnuastro will
automatically create it when memory mapping becomes necessary.
Alternatively, it may happen that the ‘gnuastro_mmap’ sub-directory
exists and is not writable, or it cannot be created. In such cases, the
memory-mapped file for each dataset will be created in the running
directory with a ‘gnuastro_mmap_’ prefix.
Therefore one easy way to delete all memory-mapped files in case of a
crash, is to delete everything within the sub-directory (first command
below), or all files stating with this prefix:
rm -f gnuastro_mmap/*
rm -f gnuastro_mmap_*
A much more common issue when dealing with memory-mapped files is
their location. For example, you may be running a program in a
partition that is hosted by an HDD. But you also have another partition
on an SSD (which has much faster I/O). So you want your memory-mapped
files to be created in the SSD to speed up your processing. In this
scenario, you want your project source directory to only contain your
plain-text scripts and you want your project's built products (even the
temporary memory-mapped files) to be built in a different location
because they are large; thus I/O speed becomes important.
To host the memory-mapped files in another location (with fast I/O),
you can set (‘gnuastro_mmap’) to be a symbolic link to it. For example,
let's assume you want your memory-mapped files to be stored in
‘/path/to/dir/for/mmap’. All you have to do is to run the following
command before your Gnuastro analysis command(s).
ln -s /path/to/dir/for/mmap gnuastro_mmap
The programs will delete a memory-mapped file when it is no longer
needed, but they will not delete the ‘gnuastro_mmap’ directory that
hosts them. So if your project involves many Gnuastro programs
(possibly called in parallel) and you want your memory-mapped files to
be in a different location, you just have to make the symbolic link
above once at the start, and all the programs will use it if necessary.
Another memory-management scenario that may happen is this: you do
not want a Gnuastro program to allocate internal datasets in the RAM at
all. For example, the speed of your Gnuastro-related project does not
matter at that moment, and you have higher-priority jobs that are being
run at the same time which need to have RAM available. In such cases,
you can use the ‘--minmapsize’ option that is available in all Gnuastro
programs (see *note Processing options::). Any intermediate dataset
that has a size larger than the value of this option will be
memory-mapped, even if there is space available in your RAM. For
example, if you want any dataset larger than 100 megabytes to be
memory-mapped, use ‘--minmapsize=100000000’ (8 zeros!).
You should not set the value of ‘--minmapsize’ to be too small,
otherwise even small intermediate values (that are usually very
numerous) in the program will be memory-mapped. However the kernel can
only host a limited number of memory-mapped files at every moment (by
all running programs combined). For example, in the default(1) Linux
kernel on GNU/Linux operating systems this limit is roughly 64000. If
the total number of memory-mapped files exceeds this number, all the
programs using them will crash. Gnuastro's programs will warn you if
your given value is too small and may cause a problem later.
Actually, the default behavior for Gnuastro's programs (to only use
memory-mapped files when there is not enough RAM) is a side-effect of
‘--minmapsize’. The pre-defined value to this option is an extremely
large value in the lowest-level Gnuastro configuration file (the
installed ‘gnuastro.conf’ described in *note Configuration file
precedence::). This value is larger than the largest possible available
RAM. You can check by running any Gnuastro program with a ‘-P’ option.
Because no dataset will be larger than this, by default the programs
will first attempt to use the RAM for temporary storage. But if writing
in the RAM fails (for any reason, mainly due to lack of available
space), then a memory-mapped file will be created.
---------- Footnotes ----------
(1) If you need to host more memory-mapped files at one moment, you
need to build your own customized Linux kernel.
File: gnuastro.info, Node: Tables, Next: Tessellation, Prev: Memory management, Up: Common program behavior
4.7 Tables
==========
"A table is a collection of related data held in a structured format
within a database. It consists of columns, and rows." (from
Wikipedia). Each column in the table contains the values of one
property and each row is a collection of properties (columns) for one
target object. For example, let's assume you have just ran MakeCatalog
(see *note MakeCatalog::) on an image to measure some properties for the
labeled regions (which might be detected galaxies for example) in the
image. For each labeled region (detected galaxy), there will be a _row_
which groups its measured properties as _columns_, one column for each
property. One such property can be the object's magnitude, which is the
sum of pixels with that label, or its center can be defined as the
light-weighted average value of those pixels. Many such properties can
be derived from the raw pixel values and their position, see *note
Invoking astmkcatalog:: for a long list.
As a summary, for each labeled region (or, galaxy) we have one _row_
and for each measured property we have one _column_. This high-level
structure is usually the first step for higher-level analysis, for
example, finding the stellar mass or photometric redshift from
magnitudes in multiple colors. Thus, tables are not just outputs of
programs, in fact it is much more common for tables to be inputs of
programs. For example, to make a mock galaxy image, you need to feed in
the properties of each galaxy into *note MakeProfiles:: for it do the
inverse of the process above and make a simulated image from a catalog,
see *note Sufi simulates a detection::. In other cases, you can feed a
table into *note Crop:: and it will crop out regions centered on the
positions within the table, see *note Reddest clumps cutouts and
parallelization::. So to end this relatively long introduction, tables
play a very important role in astronomy, or generally all branches of
data analysis.
In *note Recognized table formats:: the currently recognized table
formats in Gnuastro are discussed. You can use any of these tables as
input or ask for them to be built as output. The most common type of
table format is a simple plain text file with each row on one line and
columns separated by white space characters, this format is easy to
read/write by eye/hand. To give it the full functionality of more
specific table types like the FITS tables, Gnuastro has a special
convention which you can use to give each column a name, type, unit, and
comments, while still being readable by other plain text table readers.
This convention is described in *note Gnuastro text table format::.
When tables are input to a program, the program reading it needs to
know which column(s) it should use for its desired purposes. Gnuastro's
programs all follow a similar convention, on the way you can select
columns in a table. They are thoroughly discussed in *note Selecting
table columns::.
* Menu:
* Recognized table formats:: Table formats that are recognized in Gnuastro.
* Gnuastro text table format:: Gnuastro's convention plain text tables.
* Selecting table columns:: Identify/select certain columns from a table
File: gnuastro.info, Node: Recognized table formats, Next: Gnuastro text table format, Prev: Tables, Up: Tables
4.7.1 Recognized table formats
------------------------------
The list of table formats that Gnuastro can currently read from and
write to are described below. Each has their own advantage and
disadvantages, so a short review of the format is also provided to help
you make the best choice based on how you want to define your input
tables or later use your output tables.
Plain text table
This is the most basic and simplest way to create, view, or edit
the table by hand on a text editor. The other formats described
below are less eye-friendly and have a more formal structure (for
easier computer readability). It is fully described in *note
Gnuastro text table format::.
FITS ASCII tables
The FITS ASCII table extension is fully in ASCII encoding and thus
easily readable on any text editor (assuming it is the only
extension in the FITS file). If the FITS file also contains binary
extensions (for example, an image or binary table extensions), then
there will be many hard to print characters. The FITS ASCII format
does not have new line characters to separate rows. In the FITS
ASCII table standard, each row is defined as a fixed number of
characters (value to the ‘NAXIS1’ keyword), so to visually inspect
it properly, you would have to adjust your text editor's width to
this value. All columns start at given character positions and
have a fixed width (number of characters).
Numbers in a FITS ASCII table are printed into ASCII format, they
are not in binary (that the CPU uses). Hence, they can take a
larger space in memory, loose their precision, and take longer to
read into memory. If you are dealing with integer type columns
(see *note Numeric data types::), another issue with FITS ASCII
tables is that the type information for the column will be lost
(there is only one integer type in FITS ASCII tables). One problem
with the binary format on the other hand is that it is not portable
(different CPUs/compilers) have different standards for translating
the zeros and ones. But since ASCII characters are defined on a
byte and are well recognized, they are better for portability on
those various systems. Gnuastro's plain text table format
described below is much more portable and easier to
read/write/interpret by humans manually.
Generally, as the name implies, this format is useful for when your
table mainly contains ASCII columns (for example, file names, or
descriptions). They can be useful when you need to include columns
with structured ASCII information along with other extensions in
one FITS file. In such cases, you can also consider header
keywords (see *note Fits::).
FITS binary tables
The FITS binary table is the FITS standard's solution to the issues
discussed with keeping numbers in ASCII format as described under
the FITS ASCII table title above. Only columns defined as a string
type (a string of ASCII characters) are readable in a text editor.
The portability problem with binary formats discussed above is
mostly solved thanks to the portability of CFITSIO (see *note
CFITSIO::) and the very long history of the FITS format which has
been widely used since the 1970s.
In the case of most numbers, storing them in binary format is more
memory efficient than ASCII format. For example, to store
‘-25.72034’ in ASCII format, you need 9 bytes/characters. But if
you keep this same number (to the approximate precision possible)
as a 4-byte (32-bit) floating point number, you can keep/transmit
it with less than half the amount of memory. When catalogs contain
thousands/millions of rows in tens/hundreds of columns, this can
lead to significant improvements in memory/band-width usage.
Moreover, since the CPU does its operations in the binary formats,
reading the table in and writing it out is also much faster than an
ASCII table.
When you are dealing with integer numbers, the compression ratio
can be even better, for example, if you know all of the values in a
column are positive and less than ‘255’, you can use the ‘unsigned
char’ type which only takes one byte! If they are between ‘-128’
and ‘127’, then you can use the (signed) ‘char’ type. So if you
are thoughtful about the limits of your integer columns, you can
greatly reduce the size of your file and also the speed at which it
is read/written. This can be very useful when sharing your results
with collaborators or publishing them. To decrease the file size
even more you can name your output as ending in ‘.fits.gz’ so it is
also compressed after creation. Just note that
compression/decompressing is CPU intensive and can slow down the
writing/reading of the file.
Fortunately the FITS Binary table format also accepts ASCII strings
as column types (along with the various numerical types). So your
dataset can also contain non-numerical columns.
* Menu:
* Gnuastro text table format:: Reading plain text tables
File: gnuastro.info, Node: Gnuastro text table format, Next: Selecting table columns, Prev: Recognized table formats, Up: Tables
4.7.2 Gnuastro text table format
--------------------------------
Plain text files are the most generic, portable, and easiest way to
(manually) create, (visually) inspect, or (manually) edit a table. In
this format, the ending of a row is defined by the new-line character (a
line on a text editor). So when you view it on a text editor, every row
will occupy one line. The delimiters (or characters separating the
columns) are white space characters (space, horizontal tab, vertical
tab) and a comma (<,>). The only further requirement is that all
rows/lines must have the same number of columns.
The columns do not have to be exactly under each other and the rows
can be arbitrarily long with different lengths. For example, the
following contents in a file would be interpreted as a table with 4
columns and 2 rows, with each element interpreted as a 64-bit floating
point type (see *note Numeric data types::).
1 2.234948 128 39.8923e8
2 , 4.454 792 72.98348e7
However, the example above has no other information about the columns
(it is just raw data, with no meta-data). To use this table, you have
to remember what the numbers in each column represent. Also, when you
want to select columns, you have to count their position within the
table. This can become frustrating and prone to bad errors (getting the
columns wrong in your scientific project!) especially as the number of
columns increase. It is also bad for sending to a colleague, because
they will find it hard to remember/use the columns properly.
To solve these problems in Gnuastro's programs/libraries you are not
limited to using the column's number, see *note Selecting table
columns::. If the columns have names, units, or comments you can also
select your columns based on searches/matches in these fields, for
example, see *note Table::. Also, in this manner, you cannot guide the
program reading the table on how to read the numbers. As an example,
the first and third columns above can be read as integer types: the
first column might be an ID and the third can be the number of pixels an
object occupies in an image. So there is no need to read these to
columns as a 64-bit floating point type (which takes more memory, and is
slower).
In the bare-minimum example above, you also cannot use strings of
characters, for example, the names of filters, or some other identifier
that includes non-numerical characters. In the absence of any
information, only numbers can be read robustly. Assuming we read
columns with non-numerical characters as string, there would still be
the problem that the strings might contain space (or any delimiter)
character for some rows. So, each 'word' in the string will be
interpreted as a column and the program will abort with an error that
the rows do not have the same number of columns.
To correct for these limitations, Gnuastro defines the following
convention for storing the table meta-data along with the raw data in
one plain text file. The format is primarily designed for ease of
reading/writing by eye/fingers, but is also structured enough to be read
by a program.
When the first non-white character in a line is <#>, or there are no
non-white characters in it, then the line will not be considered as a
row of data in the table (this is a pretty standard convention in many
programs, and higher level languages). In the first case (when the
first character of the line is <#>), the line is interpreted as a
_comment_.
If the comment line starts with '‘# Column N:’', then it is assumed
to contain information about column ‘N’ (a number, counting from 1).
Comment lines that do not start with this pattern are ignored and you
can use them to include any further information you want to store with
the table in the text file. The most generic column information comment
line has the following format:
# Column N: NAME [UNIT, TYPE(NUM), BLANK] COMMENT
Any sequence of characters between '<:>' and '<[>' will be interpreted
as the column name (so it can contain anything except the '<[>'
character). Anything between the '<]>' and the end of the line is
defined as a comment. Within the brackets, anything before the first
'<,>' is the units (physical units, for example, km/s, or erg/s),
anything before the second '<,>' is the short type identifier (see
below, and *note Numeric data types::).
If the type identifier is not recognized, the default 64-bit floating
point type will be used. The type identifier can optionally be followed
by an integer within parenthesis. If the parenthesis is present and the
integer is larger than 1, the column is assumed to be a "vector column"
(which can have multiple values, for more see *note Vector columns::).
Finally (still within the brackets), any non-white characters after
the second '<,>' are interpreted as the blank value for that column (see
*note Blank pixels::). The blank value can either be in the same type
as the column (for example, ‘-99’ for a signed integer column), or any
string (for example, ‘NaN’ in that same column). In both cases, the
values will be stored in memory as Gnuastro's fixed blank values for
each type. For floating point types, Gnuastro's internal blank value is
IEEE NaN (Not-a-Number). For signed integers, it is the smallest
possible value and for unsigned integers its the largest possible value.
When a formatting problem occurs, or when the column was already
given meta-data in a previous comment, or when the column number is
larger than the actual number of columns in the table (the non-commented
or empty lines), then the comment information line will be ignored.
When a comment information line can be used, the leading and trailing
white space characters will be stripped from all of the elements. For
example, in this line:
# Column 5: column name [km/s, f32,-99] Redshift as speed
The ‘NAME’ field will be '‘column name’' and the ‘TYPE’ field will be
'‘f32’'. Note how all the white space characters before and after
strings are not used, but those in the middle remained. Also, white
space characters are not mandatory. Hence, in the example above, the
‘BLANK’ field will be given the value of '‘-99’'.
Except for the column number (‘N’), the rest of the fields are
optional. Also, the column information comments do not have to be in
order. In other words, the information for column $N+m$ ($m>0$) can be
given in a line before column $N$. Furthermore, you do not have to
specify information for all columns. Those columns that do not have
this information will be interpreted with the default settings (like the
case above: values are double precision floating point, and the column
has no name, unit, or comment). So these lines are all acceptable for
any table (the first one, with nothing but the column number is
redundant):
# Column 5:
# Column 1: ID [,i8] The Clump ID.
# Column 3: mag_f160w [AB mag, f32] Magnitude from the F160W filter
The data type of the column should be specified with one of the
following values:
• For a numeric column, you can use any of the numeric types (and
their recognized identifiers) described in *note Numeric data
types::.
• '‘strN’': for strings. The ‘N’ value identifies the length of the
string (how many characters it has). The start of the string on
each row is the first non-delimiter character of the column that
has the string type. The next ‘N’ characters will be interpreted
as a string and all leading and trailing white space will be
removed.
If the next column's characters, are closer than ‘N’ characters to
the start of the string column in that line/row, they will be
considered part of the string column. If there is a new-line
character before the ending of the space given to the string column
(in other words, the string column is the last column), then
reading of the string will stop, even if the ‘N’ characters are not
complete yet. See ‘tests/table/table.txt’ for one example.
Therefore, the only time you have to pay attention to the
positioning and spaces given to the string column is when it is not
the last column in the table.
The only limitation in this format is that trailing and leading
white space characters will be removed from the columns that are
read. In most cases, this is the desired behavior, but if trailing
and leading white-spaces are critically important to your analysis,
define your own starting and ending characters and remove them
after the table has been read. For example, in the sample table
below, the two '<|>' characters (which are arbitrary) will remain
in the value of the second column and you can remove them manually
later. If only one of the leading or trailing white spaces is
important for your work, you can only use one of the '<|>'s.
# Column 1: ID [label, u8]
# Column 2: Notes [no unit, str50]
1 leading and trailing white space is ignored here 2.3442e10
2 | but they will be preserved here | 8.2964e11
Note that the FITS binary table standard does not define the
‘unsigned int’ and ‘unsigned long’ types, so if you want to convert your
tables to FITS binary tables, use other types. Also, note that in the
FITS ASCII table, there is only one integer type (‘long’). So if you
convert a Gnuastro plain text table to a FITS ASCII table with the *note
Table:: program, the type information for integers will be lost.
Conversely if integer types are important for you, you have to manually
set them when reading a FITS ASCII table (for example, with the Table
program when reading/converting into a file, or with the
‘gnuastro/table.h’ library functions when reading into memory).
File: gnuastro.info, Node: Selecting table columns, Prev: Gnuastro text table format, Up: Tables
4.7.3 Selecting table columns
-----------------------------
At the lowest level, the only defining aspect of a column in a table is
its number, or position. But selecting columns purely by number is not
very convenient and, especially when the tables are large it can be very
frustrating and prone to errors. Hence, table file formats (for
example, see *note Recognized table formats::) have ways to store
additional information about the columns (meta-data). Some of the most
common pieces of information about each column are its _name_, the
_units_ of data in it, and a _comment_ for longer/informal description
of the column's data.
To facilitate research with Gnuastro, you can select columns by
matching, or searching in these three fields, besides the low-level
column number. To view the full list of information on the columns in
the table, you can use the Table program (see *note Table::) with the
command below (replace ‘table-file’ with the filename of your table, if
its FITS, you might also need to specify the HDU/extension which
contains the table):
$ asttable --information table-file
Gnuastro's programs need the columns for different purposes, for
example, in Crop, you specify the columns containing the central
coordinates of the crop centers with the ‘--coordcol’ option (see *note
Crop options::). On the other hand, in MakeProfiles, to specify the
column containing the profile position angles, you must use the ‘--pcol’
option (see *note MakeProfiles catalog::). Thus, there can be no
unified common option name to select columns for all programs (different
columns have different purposes). However, when the program expects a
column for a specific context, the option names end in the ‘col’ suffix
like the examples above. These options accept values in integer (column
number), or string (metadata match/search) format.
If the value can be parsed as a positive integer, it will be seen as
the low-level column number. Note that column counting starts from 1,
so if you ask for column 0, the respective program will abort with an
error. When the value cannot be interpreted as an a integer number, it
will be seen as a string of characters which will be used to
match/search in the table's meta-data. The meta-data field which the
value will be compared with can be selected through the ‘--searchin’
option, see *note Input output options::. ‘--searchin’ can take three
values: ‘name’, ‘unit’, ‘comment’. The matching will be done following
this convention:
• If the value is enclosed in two slashes (for example, ‘-x/RA_/’, or
‘--coordcol=/RA_/’, see *note Crop options::), then it is assumed
to be a regular expression with the same convention as GNU AWK. GNU
AWK has a very well written chapter
(https://www.gnu.org/software/gawk/manual/html_node/Regexp.html)
describing regular expressions, so we will not continue discussing
them here. Regular expressions are a very powerful tool in
matching text and useful in many contexts. We thus strongly
encourage reviewing this chapter for greatly improving the quality
of your work in many cases, not just for searching column meta-data
in Gnuastro.
• When the string is not enclosed between '</>'s, any column that
exactly matches the given value in the given field will be
selected.
Note that in both cases, you can ignore the case of alphabetic
characters with the ‘--ignorecase’ option, see *note Input output
options::. Also, in both cases, multiple columns may be selected with
one call to this function. In this case, the order of the selected
columns (with one call) will be the same order as they appear in the
table.
File: gnuastro.info, Node: Tessellation, Next: Automatic output, Prev: Tables, Up: Common program behavior
4.8 Tessellation
================
It is sometimes necessary to classify the elements in a dataset (for
example, pixels in an image) into a grid of individual, non-overlapping
tiles. For example, when background sky gradients are present in an
image, you can define a tile grid over the image. When the tile sizes
are set properly, the background's variation over each tile will be
negligible, allowing you to measure (and subtract) it. In other cases
(for example, spatial domain convolution in Gnuastro, see *note
Convolve::), it might simply be for speed of processing: each tile can
be processed independently on a separate CPU thread. In the arts and
mathematics, this process is formally known as tessellation
(https://en.wikipedia.org/wiki/Tessellation).
The size of the regular tiles (in units of data-elements, or pixels
in an image) can be defined with the ‘--tilesize’ option. It takes
multiple numbers (separated by a comma) which will be the length along
the respective dimension (in FORTRAN/FITS dimension order). Divisions
are also acceptable, but must result in an integer. For example,
‘--tilesize=30,40’ can be used for an image (a 2D dataset). The regular
tile size along the first FITS axis (horizontal when viewed in SAO DS9)
will be 30 pixels and along the second it will be 40 pixels. Ideally,
‘--tilesize’ should be selected such that all tiles in the image have
exactly the same size. In other words, that the dataset length in each
dimension is divisible by the tile size in that dimension.
However, this is not always possible: the dataset can be any size and
every pixel in it is valuable. In such cases, Gnuastro will look at the
significance of the remainder length, if it is not significant (for
example, one or two pixels), then it will just increase the size of the
first tile in the respective dimension and allow the rest of the tiles
to have the required size. When the remainder is significant (for
example, one pixel less than the size along that dimension), the
remainder will be added to one regular tile's size and the large tile
will be cut in half and put in the two ends of the grid/tessellation.
In this way, all the tiles in the central regions of the dataset will
have the regular tile sizes and the tiles on the edge will be slightly
larger/smaller depending on the remainder significance. The fraction
which defines the remainder significance along all dimensions can be set
through ‘--remainderfrac’.
The best tile size is directly related to the spatial properties of
the property you want to study (for example, gradient on the image). In
practice we assume that the gradient is not present over each tile. So
if there is a strong gradient (for example, in long wavelength ground
based images) or the image is of a crowded area where there is not too
much blank area, you have to choose a smaller tile size. A larger mesh
will give more pixels and so the scatter in the results will be less
(better statistics).
For raw image processing, a single tessellation/grid is not
sufficient. Raw images are the unprocessed outputs of the camera
detectors. Modern detectors usually have multiple readout channels each
with its own amplifier. For example, the Hubble Space Telescope
Advanced Camera for Surveys (ACS) has four amplifiers over its full
detector area dividing the square field of view to four smaller squares.
Ground based image detectors are not exempt, for example, each CCD of
Subaru Telescope's Hyper Suprime-Cam camera (which has 104 CCDs) has
four amplifiers, but they have the same height of the CCD and divide the
width by four parts.
The bias current on each amplifier is different, and initial bias
subtraction is not perfect. So even after subtracting the measured bias
current, you can usually still identify the boundaries of different
amplifiers by eye. See Figure 11(a) in Akhlaghi and Ichikawa (2015) for
an example. This results in the final reduced data to have non-uniform
amplifier-shaped regions with higher or lower background flux values.
Such systematic biases will then propagate to all subsequent
measurements we do on the data (for example, photometry and subsequent
stellar mass and star formation rate measurements in the case of
galaxies).
Therefore an accurate analysis requires a two layer tessellation: the
top layer contains larger tiles, each covering one amplifier channel.
For clarity we will call these larger tiles "channels". The number of
channels along each dimension is defined through the ‘--numchannels’.
Each channel is then covered by its own individual smaller tessellation
(with tile sizes determined by the ‘--tilesize’ option). This will
allow independent analysis of two adjacent pixels from different
channels if necessary. If the image is processed or the detector only
has one amplifier, you can set the number of channels in both dimension
to 1.
The final tessellation can be inspected on the image with the
‘--checktiles’ option that is available to all programs which use
tessellation for localized operations. When this option is called, a
FITS file with a ‘_tiled.fits’ suffix will be created along with the
outputs, see *note Automatic output::. Each pixel in this image has the
number of the tile that covers it. If the number of channels in any
dimension are larger than unity, you will notice that the tile IDs are
defined such that the first channels is covered first, then the second
and so on. For the full list of processing-related common options
(including tessellation options), please see *note Processing options::.
File: gnuastro.info, Node: Automatic output, Next: Output FITS files, Prev: Tessellation, Up: Common program behavior
4.9 Automatic output
====================
All the programs in Gnuastro are designed such that specifying an output
file or directory (based on the program context) is optional. When no
output name is explicitly given (with ‘--output’, see *note Input output
options::), the programs will automatically set an output name based on
the input name(s) and what the program does. For example, when you are
using ConvertType to save FITS image named ‘dataset.fits’ to a JPEG
image and do not specify a name for it, the JPEG output file will be
name ‘dataset.jpg’. When the input is from the standard input (for
example, a pipe, see *note Standard input::), and ‘--output’ is not
given, the output name will be the program's name (for example,
‘converttype.jpg’).
Another very important part of the automatic output generation is
that all the directory information of the input file name is stripped
off of it. This feature can be disabled with the ‘--keepinputdir’
option, see *note Input output options::. It is the default because
astronomical data are usually very large and organized specially with
special file names. In some cases, the user might not have write
permissions in those directories(1).
Let's assume that we are working on a report and want to process the
FITS images from two projects (ABC and DEF), which are stored in the
sub-directories named ‘ABCproject/’ and ‘DEFproject/’ of our top data
directory (‘/mnt/data’). The following shell commands show how one
image from the former is first converted to a JPEG image through
ConvertType and then the objects from an image in the latter project are
detected using NoiseChisel. The text after the ‘#’ sign are comments
(not typed!).
$ pwd # Current location
/home/usrname/research/report
$ ls # List directory contents
ABC01.jpg
$ ls /mnt/data/ABCproject # Archive 1
ABC01.fits ABC02.fits ABC03.fits
$ ls /mnt/data/DEFproject # Archive 2
DEF01.fits DEF02.fits DEF03.fits
$ astconvertt /mnt/data/ABCproject/ABC02.fits --output=jpg # Prog 1
$ ls
ABC01.jpg ABC02.jpg
$ astnoisechisel /mnt/data/DEFproject/DEF01.fits # Prog 2
$ ls
ABC01.jpg ABC02.jpg DEF01_detected.fits
---------- Footnotes ----------
(1) In fact, even if the data is stored on your own computer, it is
advised to only grant write permissions to the super user or root. This
way, you will not accidentally delete or modify your valuable data!
File: gnuastro.info, Node: Output FITS files, Next: Numeric locale, Prev: Automatic output, Up: Common program behavior
4.10 Output FITS files
======================
The output of many of Gnuastro's programs are (or can be) FITS files.
The FITS format has many useful features for storing scientific datasets
(cubes, images and tables) along with a robust features for
archivability. For more on this standard, please see *note Fits::.
As a community convention described in *note Fits::, the first
extension of all FITS files produced by Gnuastro's programs only
contains the meta-data that is intended for the file's extension(s).
For a Gnuastro program, this generic meta-data (that is stored as FITS
keyword records) is its configuration when it produced this dataset:
file name(s) of input(s) and option names, values and comments. You can
use the ‘--outfitsnoconfig’ option to stop the programs from writing
these keywords into the first extension of their output.
When the configuration is too trivial (only input filename, for
example, the program *note Table::) no meta-data is written in this
extension. FITS keywords have the following limitations in regards to
generic option names and values which are described below:
• If a keyword (option name) is longer than 8 characters, the first
word in the record (80 character line) is ‘HIERARCH’ which is
followed by the keyword name.
• Values can be at most 75 characters, but for strings, this changes
to 73 (because of the two extra <'> characters that are necessary).
However, if the value is a file name, containing slash (</>)
characters to separate directories, Gnuastro will break the value
into multiple keywords.
• Keyword names ignore case, therefore they are all in capital
letters. Therefore, if you want to use Grep to inspect these
keywords, use the ‘-i’ option, like the example below.
$ astfits image_detected.fits -h0 | grep -i snquant
The keywords above are classified (separated by an empty line and
title) as a group titled "ProgramName configuration". This meta-data
extension also contains a final group of keywords to keep the basic date
and version information of Gnuastro, its dependencies and the pipeline
that is using Gnuastro (if it is under version control); they are listed
below.
‘DATE’
The creation time of the FITS file. This date is written directly
by CFITSIO and is in UT format.
While the date can be a good metadata in most scenarios, it does
have a caveat: when everything else in your output is the same
between multiple runs, the date will be different! If exact
reproducibility is important for you, this can be annoying! To
stop any Gnuastro program from writing the ‘DATE’ keyword, you can
use the ‘--outfitsnodate’ (see *note Input output options::).
‘DATEUTC’
If the date in the ‘DATE’ keyword is in UTC
(https://en.wikipedia.org/wiki/Coordinated_Universal_Time), this
keyword will have a value of 1; otherwise, it will have a value of
0. If ‘DATE’ is not written, this is also ignored.
‘COMMIT’
Git's commit description from the running directory of Gnuastro's
programs. If the running directory is not version controlled or
‘libgit2’ is not installed (see *note Optional dependencies::) then
this keyword will not be present. The printed value is equivalent
to the output of the following command:
git describe --dirty --always
If the running directory contains non-committed work, then the
stored value will have a '‘-dirty’' suffix. This can be very
helpful to let you know that the data is not ready to be shared
with collaborators or submitted to a journal. You should only
share results that are produced after all your work is committed
(safely stored in the version controlled history and thus
reproducible).
At first sight, version control appears to be mainly a tool for
software developers. However progress in a scientific research is
almost identical to progress in software development: first you
have a rough idea that starts with handful of easy steps. But as
the first results appear to be promising, you will have to extend,
or generalize, it to make it more robust and work in all the
situations your research covers, not just your first test samples.
Slowly you will find wrong assumptions or bad implementations that
need to be fixed ('bugs' in software development parlance).
Finally, when you submit the research to your collaborators or a
journal, many comments and suggestions will come in, and you have
to address them.
Software developers have created version control systems precisely
for this kind of activity. Each significant moment in the
project's history is called a "commit", see *note Version
controlled source::. A snapshot of the project in each "commit" is
safely stored away, so you can revert back to it at a later time,
or check changes/progress. This way, you can be sure that your
work is reproducible and track the progress and history. With
version control, experimentation in the project's analysis is
greatly facilitated, since you can easily revert back if a
brainstorm test procedure fails.
One important feature of version control is that the research
result (FITS image, table, report or paper) can be stamped with the
unique commit information that produced it. This information will
enable you to exactly reproduce that same result later, even if you
have made changes/progress. For one example of a research paper's
reproduction pipeline, please see the reproduction pipeline
(https://gitlab.com/makhlaghi/NoiseChisel-paper) of Akhlaghi and
Ichikawa 2015 (https://arxiv.org/abs/1505.01664) describing *note
NoiseChisel::.
In case you don't want the ‘COMMIT’ keyword in the first extension
of your output FITS file, you can use the ‘--outfitsnocommit’
option (see *note Input output options::).
‘CFITSIO’
The version of CFITSIO used (see *note CFITSIO::). This can be
disabled with ‘--outfitsnoversions’ (see *note Input output
options::).
‘WCSLIB’
The version of WCSLIB used (see *note WCSLIB::). Note that older
versions of WCSLIB do not report the version internally. So this
is only available if you are using more recent WCSLIB versions.
This can be disabled with ‘--outfitsnoversions’ (see *note Input
output options::).
‘GSL’
The version of GNU Scientific Library that was used, see *note GNU
Scientific Library::. This can be disabled with
‘--outfitsnoversions’ (see *note Input output options::).
‘GNUASTRO’
The version of Gnuastro used (see *note Version numbering::). This
can be disabled with ‘--outfitsnoversions’ (see *note Input output
options::).
File: gnuastro.info, Node: Numeric locale, Prev: Output FITS files, Up: Common program behavior
4.11 Numeric locale
===================
If your system locale
(https://en.wikipedia.org/wiki/Locale_(computer_software)) is not
English, it may happen that the '.' is not used as the decimal
separator of basic command-line tools for input or output. For example,
in Spanish and some other languages the decimal separator (symbol used
to separate the integer and fractional part of a number), is a comma.
Therefore in such systems, some programs may print $0.5$ as as '‘0,5’'
(instead of '‘0.5’'). This mainly happens in some core operating system
tools like ‘awk’ or ‘seq’ depend on the locale. This can cause problems
for other programs (like those in Gnuastro that expect a '<.>' as the
decimal separator).
To see the effect, please try the commands below. The first one will
print $0.5$ in your default locale's format. The second set will use
the Spanish locale for printing numbers (which will put a comma between
the 0 and the 5). The third will use the English (US) locale for
printing numbers (which will put a point between the 0 and the 5).
$ seq 0.5 1
$ export LC_NUMERIC=es_ES.utf8
$ seq 0.5 1
$ export LC_NUMERIC=en_US.utf8
$ seq 0.5 1
With the simple command below, you can check your current locale
environment variables for specifying the formats of various things like
date, time, monetary, telephone, numbers, etc. You can change any of
these, by simply giving different values to the respective variable like
above. For a more complete explanation on each variable, see
<https://www.baeldung.com/linux/locale-environment-variables>.
$ locale
To avoid these kinds of locale-specific problems (for example,
another program not being able to read '‘0,5’' as half of unity), you
can change the locale by giving the value of ‘C’ to the ‘LC_NUMERIC’
environment variable (or the lower-level/generic ‘LC_ALL’). You will
notice that ‘C’ is not a human-language and country identifier like
‘en_US’, it is the programming locale, which is well recognized by
programmers in all countries and is available on all Unix-like operating
systems (others may not be pre-defined and may need installation). You
can set the ‘LC_NUMERIC’ only for a single command (the first one below:
simply defining the variable in the same line), or all commands within
the running session (the second command below, or "exporting" it to all
subsequent commands):
## Change the numeric locale, only for this 'seq' command.
$ LC_NUMERIC=C seq 0.5 1
## Change the locale to the standard, for all commands after it.
$ export LC_NUMERIC=C
If you want to change it generally for all future sessions, you can
put the second command in your shell's startup file. For more on
startup files, please see *note Installation directory::.
File: gnuastro.info, Node: Data containers, Next: Data manipulation, Prev: Common program behavior, Up: Top
5 Data containers
*****************
The most low-level and basic property of a dataset is how it is stored.
To process, archive and transmit the data, you need a container to store
it first. From the start of the computer age, different formats have
been defined to store data, optimized for particular applications. One
format/container can never be useful for all applications: the storage
defines the application and vice-versa. In astronomy, the Flexible
Image Transport System (FITS) standard has become the most common format
of data storage and transmission. It has many useful features, for
example, multiple sub-containers (also known as extensions or header
data units, HDUs) within one file, or support for tables as well as
images. Each HDU can store an independent dataset and its corresponding
meta-data. Therefore, Gnuastro has one program (see *note Fits::)
specifically designed to manipulate FITS HDUs and the meta-data (header
keywords) in each HDU.
Your astronomical research does not just involve data analysis (where
the FITS format is very useful). For example, you want to demonstrate
your raw and processed FITS images or spectra as figures within slides,
reports, or papers. The FITS format is not defined for such
applications. Thus, Gnuastro also comes with the ConvertType program
(see *note ConvertType::) which can be used to convert a FITS image to
and from (where possible) other formats like plain text and JPEG (which
allow two way conversion), along with EPS and PDF (which can only be
created from FITS, not the other way round).
Finally, the FITS format is not just for images, it can also store
tables. Binary tables in particular can be very efficient in storing
catalogs that have more than a few tens of columns and rows. However,
unlike images (where all elements/pixels have one data type), tables
contain multiple columns and each column can have different properties:
independent data types (see *note Numeric data types::) and meta-data.
In practice, each column can be viewed as a separate container that is
grouped with others in the table. The only shared property of the
columns in a table is thus the number of elements they contain. To
allow easy inspection/manipulation of table columns, Gnuastro has the
Table program (see *note Table::). It can be used to select certain
table columns in a FITS table and see them as a human readable output on
the command-line, or to save them into another plain text or FITS table.
* Menu:
* Fits:: View and manipulate extensions and keywords.
* ConvertType:: Convert data to various formats.
* Table:: Read and Write FITS tables to plain text.
* Query:: Import data from external databases.
File: gnuastro.info, Node: Fits, Next: ConvertType, Prev: Data containers, Up: Data containers
5.1 Fits
========
The "Flexible Image Transport System", or FITS, is by far the most
common data container format in astronomy and in constant use since the
1970s. Archiving (future usage, simplicity) has been one of the primary
design principles of this format. In the last few decades it has proved
so useful and robust that the Vatican Library has also chosen FITS for
its "long-term digital preservation" project(1).
Although the full name of the standard invokes the idea that it is
only for images, it also contains complete and robust features for
tables. It started off in the 1970s and was formally published as a
standard in 1981, it was adopted by the International Astronomical Union
(IAU) in 1982 and an IAU working group to maintain its future was
defined in 1988. The FITS 2.0 and 3.0 standards were approved in 2000
and 2008 respectively, and the 4.0 draft has also been released
recently, please see the FITS standard document web page
(https://fits.gsfc.nasa.gov/fits_standard.html) for the full text of all
versions. Also see the FITS 3.0 standard paper
(https://doi.org/10.1051/0004-6361/201015362) for a nice introduction
and history along with the full standard.
Many common image formats, for example, a JPEG, only have one
image/dataset per file, however one great advantage of the FITS standard
is that it allows you to keep multiple datasets (images or tables along
with their separate meta-data) in one file. In the FITS standard, each
data + metadata is known as an extension, or more formally a header data
unit or HDU. The HDUs in a file can be completely independent: you can
have multiple images of different dimensions/sizes or tables as separate
extensions in one file. However, while the standard does not impose any
constraints on the relation between the datasets, it is strongly
encouraged to group data that are contextually related with each other
in one file. For example, an image and the table/catalog of objects and
their measured properties in that image. Other examples can be images
of one patch of sky in different colors (filters), or one raw telescope
image along with its calibration data (tables or images).
As discussed above, the extensions in a FITS file can be completely
independent. To keep some information (meta-data) about the group of
extensions in the FITS file, the community has adopted the following
convention: put no data in the first extension, so it is just meta-data.
This extension can thus be used to store Meta-data regarding the whole
file (grouping of extensions). Subsequent extensions may contain data
along with their own separate meta-data. All of Gnuastro's programs
also follow this convention: the main output dataset(s) are placed in
the second (or later) extension(s). The first extension contains no
data the program's configuration (input file name, along with all its
option values) are stored as its meta-data, see *note Output FITS
files::.
The meta-data contain information about the data, for example, which
region of the sky an image corresponds to, the units of the data, what
telescope, camera, and filter the data were taken with, it observation
date, or the software that produced it and its configuration. Without
the meta-data, the raw dataset is practically just a collection of
numbers and really hard to understand, or connect with the real world
(other datasets). It is thus strongly encouraged to supplement your
data (at any level of processing) with as much meta-data about your
processing/science as possible.
The meta-data of a FITS file is in ASCII format, which can be easily
viewed or edited with a text editor or on the command-line. Each
meta-data element (known as a keyword generally) is composed of a name,
value, units and comments (the last two are optional). For example,
below you can see three FITS meta-data keywords for specifying the world
coordinate system (WCS, or its location in the sky) of a dataset:
LATPOLE = -27.805089 / [deg] Native latitude of celestial pole
RADESYS = 'FK5' / Equatorial coordinate system
EQUINOX = 2000.0 / [yr] Equinox of equatorial coordinates
However, there are some limitations which discourage viewing/editing
the keywords with text editors. For example, there is a fixed length of
80 characters for each keyword (its name, value, units and comments) and
there are no new-line characters, so on a text editor all the keywords
are seen in one line. Also, the meta-data keywords are immediately
followed by the data which are commonly in binary format and will show
up as strange looking characters on a text editor, and significantly
slowing down the processor.
Gnuastro's Fits program was designed to allow easy manipulation of
FITS extensions and meta-data keywords on the command-line while
conforming fully with the FITS standard. For example, you can copy or
cut (copy and remove) HDUs/extensions from one FITS file to another, or
completely delete them. It also has features to delete, add, or edit
meta-data keywords within one HDU.
* Menu:
* Invoking astfits:: Arguments and options to Header.
---------- Footnotes ----------
(1) <https://www.vaticanlibrary.va/home.php?pag=progettodigit>
File: gnuastro.info, Node: Invoking astfits, Prev: Fits, Up: Fits
5.1.1 Invoking Fits
-------------------
Fits can print or manipulate the FITS file HDUs (extensions), meta-data
keywords in a given HDU. The executable name is ‘astfits’ with the
following general template
$ astfits [OPTION...] ASTRdata
One line examples:
## View general information about every extension:
$ astfits image.fits
## Print the header keywords in the second HDU (counting from 0):
$ astfits image.fits -h1
## Only print header keywords that contain `NAXIS':
$ astfits image.fits -h1 | grep NAXIS
## Only print the WCS standard PC matrix elements
$ astfits image.fits -h1 | grep 'PC._.'
## Copy a HDU from input.fits to out.fits:
$ astfits input.fits --copy=hdu-name --output=out.fits
## Update the OLDKEY keyword value to 153.034:
$ astfits --update=OLDKEY,153.034,"Old keyword comment"
## Delete one COMMENT keyword and add a new one:
$ astfits --delete=COMMENT --comment="Anything you like ;-)."
## Write two new keywords with different values and comments:
$ astfits --write=MYKEY1,20.00,"An example keyword" --write=MYKEY2,fd
## Inspect individual pixel area taken based on its WCS (in degree^2).
## Then convert the area to arcsec^2 with the Arithmetic program.
$ astfits input.fits --pixelareaonwcs -o pixarea.fits
$ astarithmetic pixarea.fits 3600 3600 x x -o pixarea_arcsec2.fits
When no action is requested (and only a file name is given), Fits
will print a list of information about the extension(s) in the file.
This information includes the HDU number, HDU name (‘EXTNAME’ keyword),
type of data (see *note Numeric data types::, and the number of data
elements it contains (size along each dimension for images and table
rows and columns). Optionally, a comment column is printed for special
situations (like a 2D HEALPix grid that is usually stored as a 1D
dataset/table). You can use this to get a general idea of the contents
of the FITS file and what HDU to use for further processing, either with
the Fits program or any other Gnuastro program.
Here is one example of information about a FITS file with four
extensions: the first extension has no data, it is a purely meta-data
HDU (commonly used to keep meta-data about the whole file, or grouping
of extensions, see *note Fits::). The second extension is an image with
name ‘IMAGE’ and single precision floating point type (‘float32’, see
*note Numeric data types::), it has 4287 pixels along its first
(horizontal) axis and 4286 pixels along its second (vertical) axis. The
third extension is also an image with name ‘MASK’. It is in 2-byte
integer format (‘int16’) which is commonly used to keep information
about pixels (for example, to identify which ones were saturated, or
which ones had cosmic rays and so on), note how it has the same size as
the ‘IMAGE’ extension. The third extension is a binary table called
‘CATALOG’ which has 12371 rows and 5 columns (it probably contains
information about the sources in the image).
GNU Astronomy Utilities X.X
Run on Day Month DD HH:MM:SS YYYY
-----
HDU (extension) information: `image.fits'.
Column 1: Index (counting from 0).
Column 2: Name (`EXTNAME' in FITS standard).
Column 3: Image data type or `table' format (ASCII or binary).
Column 4: Size of data in HDU.
-----
0 n/a uint8 0
1 IMAGE float32 4287x4286
2 MASK int16 4287x4286
3 CATALOG table_binary 12371x5
If a specific HDU is identified on the command-line with the ‘--hdu’
(or ‘-h’ option) and no operation requested, then the full list of
header keywords in that HDU will be printed (as if the ‘--printallkeys’
was called, see below). It is important to remember that this only
occurs when ‘--hdu’ is given on the command-line. The ‘--hdu’ value
given in a configuration file will only be used when a specific
operation on keywords requested. Therefore as described in the
paragraphs above, when no explicit call to the ‘--hdu’ option is made on
the command-line and no operation is requested (on the command-line or
configuration files), the basic information of each HDU/extension is
printed.
The operating mode and input/output options to Fits are similar to
the other programs and fully described in *note Common options::. The
options particular to Fits can be divided into three groups: 1) those
related to modifying HDUs or extensions (see *note HDU information and
manipulation::), and 2) those related to viewing/modifying meta-data
keywords (see *note Keyword inspection and manipulation::). 3) those
related to creating meta-images where each pixel shows values for a
specific property of the image (see *note Pixel information images::).
These three classes of options cannot be called together in one run: you
can either work on the extensions, meta-data keywords in any instance of
Fits, or create meta-images where each pixel shows a particular
information about the image itself.
* Menu:
* HDU information and manipulation:: Learn about the HDUs and move them.
* Keyword inspection and manipulation:: Manipulate metadata keywords in a HDU.
* Pixel information images:: Pixel values contain information on the pixels.
File: gnuastro.info, Node: HDU information and manipulation, Next: Keyword inspection and manipulation, Prev: Invoking astfits, Up: Invoking astfits
5.1.1.1 HDU information and manipulation
........................................
Each FITS file header data unit, or HDU (also known as an extension) is
an independent dataset (data + meta-data). Multiple HDUs can be stored
in one FITS file, see *note Fits::. The general HDU-related options to
the Fits program are listed below as two general classes: the first
group below focus on HDU information while the latter focus on
manipulating (moving or deleting) the HDUs.
The options below print information about the given HDU on the
command-line. Thus they cannot be called together in one command (each
has its own independent output).
‘-n’
‘--numhdus’
Print the number of extensions/HDUs in the given file. Note that
this option must be called alone and will only print a single
number. It is thus useful in scripts, for example, when you need
to do check the number of extensions in a FITS file.
For a complete list of basic meta-data on the extensions in a FITS
file, do not use any of the options in this section or in *note
Keyword inspection and manipulation::. For more, see *note
Invoking astfits::.
‘--hastablehdu’
Print ‘1’ (on standard output) if at least one table HDU (ASCII or
binary) exists in the FITS file. Otherwise (when no table HDU
exists in the file), print ‘0’.
‘--listtablehdus’
Print the names or numbers (when a name does not exist, counting
from zero) of HDUs that contain a table (ASCII or Binary) on
standard output, one per line. Otherwise (when no table HDU exists
in the file) nothing will be printed.
‘--hasimagehdu’
Print ‘1’ (on standard output) if at least one image HDU exists in
the FITS file. Otherwise (when no image HDU exists in the file),
print ‘0’.
In the FITS standard, any array with any dimensions is called an
"image", therefore this option includes 1, 3 and 4 dimensional
arrays too. However, an image HDU with zero dimensions (which is
usually the first extension and only contains metadata) is not
counted here.
‘--listimagehdus’
Print the names or numbers (when a name does not exist, counting
from zero) of HDUs that contain an image on standard output, one
per line. Otherwise (when no image HDU exists in the file) nothing
will be printed.
In the FITS standard, any array with any dimensions is called an
"image", therefore this option includes 1, 3 and 4 dimensional
arrays too. However, an image HDU with zero dimensions (which is
usually the first extension and only contains metadata) is not
counted here.
‘--listallhdus’
Print the names or numbers (when a name does not exist, counting
from zero) of all HDUs within the input file on the standard
output, one per line.
‘--pixelscale’
Print the HDU's pixel-scale (change in world coordinate for one
pixel along each dimension) and pixel area or voxel volume.
Without the ‘--quiet’ option, the output of ‘--pixelscale’ has
multiple lines and explanations, thus being more human-friendly.
It prints the file/HDU name, number of dimensions, and the units
along with the actual pixel scales. Also, when any of the units
are in degrees, the pixel scales and area/volume are also printed
in units of arc-seconds. For 3D datasets, the pixel area (on each
2D slice of the 3D cube) is printed as well as the voxel volume.
If you only want the pixel area of a 2D image in units of
arcsec$^2$ you can use ‘--pixelareaarcsec2’ described below.
However, in scripts (that are to be run automatically), this
human-friendly format is annoying, so when called with the
‘--quiet’ option, only the pixel-scale value(s) along each
dimension is(are) printed in one line. These numbers are followed
by the pixel area (in the raw WCS units). For 3D datasets, this
will be area on each 2D slice. Finally, for 3D datasets, a final
number (the voxel volume) is printed. As a summary, in ‘--quiet’
mode, for 2D datasets three numbers are printed and for 3D
datasets, 5 numbers are printed. If the dataset has more than 3
dimensions, only the pixel-scale values are printed (no area or
volume will be printed).
‘--pixelareaarcsec2’
Print the HDU's pixel area in units of arcsec$^2$. This option
only works on 2D images, that have WCS coordinates in units of
degrees. For lower-level information about the pixel scale in each
dimension, see ‘--pixelscale’ (described above).
‘--skycoverage’
Print the rectangular area (or 3D cube) covered by the given
image/datacube HDU over the Sky in the WCS units. The covered area
is reported in two ways: 1) the center and full width in each
dimension, 2) the minimum and maximum sky coordinates in each
dimension. This is option is thus useful when you want to get a
general feeling of a new image/dataset, or prepare the inputs to
query external databases in the region of the image (for example,
with *note Query::).
If run without the ‘--quiet’ option, the values are given with a
human-friendly description. For example, here is the output of
this option on an image taken near the star Castor:
$ astfits castor.fits --skycoverage
Input file: castor.fits (hdu: 1)
Sky coverage by center and (full) width:
Center: 113.9149075 31.93759664
Width: 2.41762045 2.67945253
Sky coverage by range along dimensions:
RA 112.7235592 115.1411797
DEC 30.59262123 33.27207376
With the ‘--quiet’ option, the values are more machine-friendly
(easy to parse). It has two lines, where the first line contains
the center/width values and the second line shows the coordinate
ranges in each dimension.
$ astfits castor.fits --skycoverage --quiet
113.9149075 31.93759664 2.41762045 2.67945253
112.7235592 115.1411797 30.59262123 33.27207376
Note that this is a simple rectangle (cube in 3D) definition, so if
the image is rotated in relation to the celestial coordinates a
general polygon is necessary to exactly describe the coverage.
Hence when there is rotation, the reported area will be larger than
the actual area containing data, you can visually see the area with
the ‘--pixelareaonwcs’ option of *note Fits::.
Currently this option only supports images that are less than 180
degrees in width (which is usually the case!). This requirement
has been necessary to account for images that cross the RA=0 hour
circle on the sky. Please get in touch with us at
<mailto:bug-gnuastro@gnu.org> if you have an image that is larger
than 180 degrees so we try to find a solution based on need.
‘--datasum’
Calculate and print the given HDU's "datasum" to stdout. The given
HDU is specified with the ‘--hdu’ (or ‘-h’) option. This number is
calculated by parsing all the bytes of the given HDU's data records
(excluding keywords). This option ignores any possibly existing
‘DATASUM’ keyword in the HDU. For more on ‘DATASUM’ in the FITS
standard, see *note Keyword inspection and manipulation:: (under
the ‘checksum’ component of ‘--write’).
You can use this option to confirm that the data in two different
HDUs (possibly with different keywords) is identical. Its
advantage over ‘--write=datasum’ (which writes the ‘DATASUM’
keyword into the given HDU) is that it does not require write
permissions.
‘--datasum-encoded’
Similar to ‘--datasum’, except that the output will be an encoded
string of numbers and small-caps alphabetic characters. This is
the same encoding algorithm that is used for the ‘CHECKSUM’
keyword, but applied to the value of the ‘DATASUM’ result. In some
scenarios, this string can be more useful than the raw integer.
The following options manipulate (move/delete) the HDUs in one FITS
file or to another FITS file. These options may be called multiple
times in one run. If so, the extensions will be copied from the input
FITS file to the output FITS file in the given order (on the
command-line and also in configuration files, see *note Configuration
file precedence::). If the separate classes are called together in one
run of Fits, then first ‘--copy’ is run (on all specified HDUs),
followed by ‘--cut’ (again on all specified HDUs), and then ‘--remove’
(on all specified HDUs).
The ‘--copy’ and ‘--cut’ options need an output FITS file (specified
with the ‘--output’ option). If the output file exists, then the
specified HDU will be copied following the last extension of the output
file (the existing HDUs in it will be untouched). Thus, after Fits
finishes, the copied HDU will be the last HDU of the output file. If no
output file name is given, then automatic output will be used to store
the HDUs given to this option (see *note Automatic output::).
‘-C STR’
‘--copy=STR’
Copy the specified extension into the output file, see explanations
above.
‘-k STR’
‘--cut=STR’
Cut (copy to output, remove from input) the specified extension
into the output file, see explanations above.
‘-R STR’
‘--remove=STR’
Remove the specified HDU from the input file.
The first (zero-th) HDU cannot be removed with this option.
Consider using ‘--copy’ or ‘--cut’ in combination with
‘primaryimghdu’ to not have an empty zero-th HDU. From CFITSIO: "In
the case of deleting the primary array (the first HDU in the file)
then [it] will be replaced by a null primary array containing the
minimum set of required keywords and no data.". So in practice,
any existing data (array) and meta-data in the first extension will
be removed, but the number of extensions in the file will not
change. This is because of the unique position the first FITS
extension has in the FITS standard (for example, it cannot be used
to store tables).
‘--primaryimghdu’
Copy or cut an image HDU to the zero-th HDU/extension a file that
does not yet exist. This option is thus irrelevant if the output
file already exists or the copied/cut extension is a FITS table.
For example, with the commands below, first we make sure that
‘out.fits’ does not exist, then we copy the first extension of
‘in.fits’ to the zero-th extension of ‘out.fits’.
$ rm -f out.fits
$ astfits in.fits --copy=1 --primaryimghdu --output=out.fits
If we had not used ‘--primaryimghdu’, then the zero-th extension of
‘out.fits’ would have no data, and its second extension would host
the copied image (just like any other output of Gnuastro).
File: gnuastro.info, Node: Keyword inspection and manipulation, Next: Pixel information images, Prev: HDU information and manipulation, Up: Invoking astfits
5.1.1.2 Keyword inspection and manipulation
...........................................
The meta-data in each header data unit, or HDU (also known as extension,
see *note Fits::) is stored as "keyword"s. Each keyword consists of a
name, value, unit, and comments. The Fits program (see *note Fits::)
options related to viewing and manipulating keywords in a FITS HDU are
described below.
First, let's review the ‘--keyvalue’ option which should be called
separately from the rest of the options described in this section.
Also, unlike the rest of the options in this section, with ‘--keyvalue’,
you can give more than one input file.
‘-l STR[,STR[,...]’
‘--keyvalue=STR[,STR[,...]’
Only print the value of the requested keyword(s): the ‘STR’s.
‘--keyvalue’ can be called multiple times, and each call can
contain multiple comma-separated keywords. If more than one file
is given, this option uses the same HDU/extension for all of them
(value to ‘--hdu’). For example, you can get the number of
dimensions of the three FITS files in the running directory, as
well as the length along each dimension, with this command:
$ astfits *.fits --keyvalue=NAXIS,NAXIS1 --keyvalue=NAXIS2
image-a.fits 2 774 672
image-b.fits 2 774 672
image-c.fits 2 387 336
If only one input is given, and the ‘--quiet’ option is activated,
the file name is not printed on the first column, only the values
of the requested keywords.
$ astfits image-a.fits --keyvalue=NAXIS,NAXIS1 \
--keyvalue=NAXIS2 --quiet
2 774 672
*Argument list too long:* if the list of input files are too long,
the shell is going to complain with the ‘Argument list too long’
error! To avoid this problem, you can put the list of files in a
plain-text file and give that plain-text file to the Fits program
through the ‘--arguments’ option discussed below.
The output is internally stored (and finally printed) as a table
(with one column per keyword). Therefore just like the Table
program, you can use ‘--colinfoinstdout’ to print the metadata like
the example below (also see *note Invoking asttable::). The
keyword metadata (comments and units) are extracted from the
comments and units of the keyword in the input files (first file
that has a comment or unit). Hence if the keyword does not have
units or comments in any of the input files, they will be empty.
For more on Gnuastro's plain-text metadata format, see *note
Gnuastro text table format::.
$ astfits *.fits --keyvalue=NAXIS,NAXIS1,NAXIS2 \
--colinfoinstdout
# Column 1: FILENAME [name,str10,] Name of input file.
# Column 2: NAXIS [ ,u8 ,] number of data axes
# Column 3: NAXIS1 [ ,u16 ,] length of data axis 1
# Column 4: NAXIS2 [ ,u16 ,] length of data axis 2
image-a.fits 2 774 672
image-b.fits 2 774 672
image-c.fits 2 387 336
Another advantage of a table output is that you can directly write
the table to a file. For example, if you add
‘--output=fileinfo.fits’, the information above will be printed
into a FITS table. You can also pipe it into *note Table:: to
select files based on certain properties, to sort them based on
another property, or any other operation that can be done with
Table (including *note Column arithmetic::). For example, with the
command below, you can select all the files that have a size larger
than 500 pixels in both dimensions.
$ astfits *.fits --keyvalue=NAXIS,NAXIS1,NAXIS2 \
--colinfoinstdout \
| asttable --range=NAXIS1,500,inf \
--range=NAXIS2,500,inf -cFILENAME
image-a.fits
image-b.fits
Note that ‘--colinfoinstdout’ is necessary to use column names when
piping to other programs (like ‘asttable’ above). Also, with the
‘-cFILENAME’ option, we are asking Table to only print the final
file names (we do not need the sizes any more).
The commands with multiple files above used ‘*.fits’, which is only
useful when all your FITS files are in the same directory.
However, in many cases, your FITS files will be scattered in
multiple sub-directories of a certain top-level directory, or you
may only want those with more particular file name patterns. A
more powerful way to list the input files to ‘--keyvalue’ is to use
the ‘find’ program in Unix-like operating systems. For example,
with the command below you can search all the FITS files in all the
sub-directories of ‘/TOP/DIR’.
astfits $(find /TOP/DIR/ -name "*.fits") --keyvalue=NAXIS2
‘--arguments=STR’
A plain-text file containing the list of input files that will be
used in ‘--keyvalue’. Each word (group of characters separated by
SPACE or new-line) is assumed to be the name of the separate input
file. This option is only relevant when no input files are given
as arguments on the command-line: if any arguments are given, this
option is ignored.
This is necessary when the list of input files are very long;
causing the shell to abort with an ‘Argument list too long’ error.
In such cases, you can put the list into a plain-text file and use
this option like below:
$ ls $(path)/*.fits > list.txt
$ astfits --arguments=list.txt --keyvalue=NAXIS1
‘-O’
‘--colinfoinstdout’
Print column information (or metadata) above the column values when
writing keyword values to standard output with ‘--keyvalue’. You
can read this option as column-information-in-standard-output.
Below we will discuss the options that can be used to manipulate
keywords. To see the full list of keywords in a FITS HDU, you can use
the ‘--printallkeys’ option. If any of the keyword modification options
below are requested (for example, ‘--update’), the headers of the input
file/HDU will be changed first, then printed. Keyword modification is
done within the input file. Therefore, if you want to keep the original
FITS file or HDU intact, it is easiest to create a copy of the file/HDU
first and then run Fits on that (for copying a HDU to another file, see
*note HDU information and manipulation::. In the FITS standard,
keywords are always uppercase. So case does not matter in the input or
output keyword names you specify.
*‘CHECKSUM’ automatically updated, when present:* the keyword
modification options will change the contents of the HDU. Therefore, if
a ‘CHECKSUM’ is present in the HDU, after all the keyword modification
options have been complete, Fits will also update ‘CHECKSUM’ before
closing the file.
Most of the options can accept multiple instances in one command.
For example, you can add multiple keywords to delete by calling
‘--delete’ multiple times, since repeated keywords are allowed, you can
even delete the same keyword multiple times. The action of such options
will start from the top most keyword.
The precedence of operations are described below. Note that while
the order within each class of actions is preserved, the order of
individual actions is not. So irrespective of what order you called
‘--delete’ and ‘--update’. First, all the delete operations are going
to take effect then the update operations.
1. ‘--delete’
2. ‘--rename’
3. ‘--update’
4. ‘--write’
5. ‘--asis’
6. ‘--history’
7. ‘--comment’
8. ‘--date’
9. ‘--printallkeys’
10. ‘--verify’
11. ‘--copykeys’
All possible syntax errors will be reported before the keywords are
actually written. FITS errors during any of these actions will be
reported, but Fits will not stop until all the operations are complete.
If ‘--quitonerror’ is called, then Fits will immediately stop upon the
first error.
If you want to inspect only a certain set of header keywords, it is
easiest to pipe the output of the Fits program to GNU Grep. Grep is a
very powerful and advanced tool to search strings which is precisely
made for such situations. for example, if you only want to check the
size of an image FITS HDU, you can run:
$ astfits input.fits | grep NAXIS
*FITS STANDARD KEYWORDS:* Some header keywords are necessary for later
operations on a FITS file, for example, BITPIX or NAXIS, see the FITS
standard for their full list. If you modify (for example, remove or
rename) such keywords, the FITS file extension might not be usable any
more. Also be careful for the world coordinate system keywords, if you
modify or change their values, any future world coordinate system (like
RA and Dec) measurements on the image will also change.
The keyword related options to the Fits program are fully described
below.
‘-d STR’
‘--delete=STR’
Delete one instance of the ‘STR’ keyword from the FITS header.
Multiple instances of ‘--delete’ can be given (possibly even for
the same keyword, when its repeated in the meta-data). All
keywords given will be removed from the headers in the same given
order. If the keyword does not exist, Fits will give a warning and
return with a non-zero value, but will not stop. To stop as soon
as an error occurs, run with ‘--quitonerror’.
‘-r STR,STR’
‘--rename=STR,STR’
Rename a keyword to a new value (for example,
‘--rename=OLDNAME,NEWNAME’. ‘STR’ contains both the existing and
new names, which should be separated by either a comma (<,>) or a
space character. Note that if you use a space character, you have
to put the value to this option within double quotation marks (<">)
so the space character is not interpreted as an option separator.
Multiple instances of ‘--rename’ can be given in one command. The
keywords will be renamed in the specified order. If the keyword
does not exist, Fits will give a warning and return with a non-zero
value, but will not stop. To stop as soon as an error occurs, run
with ‘--quitonerror’.
‘-u STR’
‘--update=STR’
Update a keyword, its value, its comments and its units in the
format described below. If there are multiple instances of the
keyword in the header, they will be changed from top to bottom
(with multiple ‘--update’ options).
The format of the values to this option can best be specified with
an example:
--update=KEYWORD,value,"comments for this keyword",unit
If there is a writing error, Fits will give a warning and return
with a non-zero value, but will not stop. To stop as soon as an
error occurs, run with ‘--quitonerror’.
The value can be any numerical or string value(1). Other than the
‘KEYWORD’, all the other values are optional. To leave a given
token empty, follow the preceding comma (<,>) immediately with the
next. If any space character is present around the commas, it will
be considered part of the respective token. So if more than one
token has space characters within it, the safest method to specify
a value to this option is to put double quotation marks around each
individual token that needs it. Note that without double quotation
marks, space characters will be seen as option separators and can
lead to undefined behavior.
‘-w STR’
‘--write=STR’
Write a keyword to the header. For the possible value input
formats, comments and units for the keyword, see the ‘--update’
option above. The special names (first string) below will cause a
special behavior:
‘/’
Write a "title" to the list of keywords. A title consists of
one blank line and another which is blank for several spaces
and starts with a slash (</>). The second string given to
this option is the "title" or string printed after the slash.
For example, with the command below you can add a "title" of
'My keywords' after the existing keywords and add the
subsequent ‘K1’ and ‘K2’ keywords under it (note that keyword
names are not case sensitive).
$ astfits test.fits -h1 --write=/,"My keywords" \
--write=k1,1.23,"My first keyword" \
--write=k2,4.56,"My second keyword"
$ astfits test.fits -h1
[[[ ... truncated ... ]]]
/ My keywords
K1 = 1.23 / My first keyword
K2 = 4.56 / My second keyword
END
Adding a "title" before each contextually separate group of
header keywords greatly helps in readability and visual
inspection of the keywords. So generally, when you want to
add new FITS keywords, it is good practice to also add a title
before them.
The reason you need to use </> as the keyword name for setting
a title is that </> is the first non-white character.
The title(s) is(are) written into the FITS with the same order
that ‘--write’ is called. Therefore in one run of the Fits
program, you can specify many different titles (with their own
keywords under them). For example, the command below that
builds on the previous example and adds another group of
keywords named ‘A1’ and ‘A2’.
$ astfits test.fits -h1 --write=/,"My keywords" \
--write=k1,1.23,"My first keyword" \
--write=k2,4.56,"My second keyword" \
--write=/,"My second group of keywords" \
--write=a1,7.89,"First keyword" \
--write=a2,0.12,"Second keyword"
‘checksum’
When nothing is given afterwards, the header integrity
keywords ‘DATASUM’ and ‘CHECKSUM’ will be calculated and
written/updated. The calculation and writing is done fully by
CFITSIO, therefore they comply with the FITS standard 4.0(2)
that defines these keywords (its Appendix J).
If a value is given (e.g., ‘--write=checksum,MyOwnCheckSum’),
then CFITSIO will not be called to calculate these two
keywords and the value (as well as possible comment and unit)
will be written just like any other keyword. This is
generally not recommended since ‘CHECKSUM’ is a reserved FITS
standard keyword. If you want to calculate the checksum with
another hashing standard manually and write it into the
header, it is recommended to use another keyword name.
In the FITS standard, ‘CHECKSUM’ depends on the HDU's data
_and_ header keywords, it will therefore not be valid if you
make any further changes to the header after writing the
‘CHECKSUM’ keyword. This includes any further keyword
modification options in the same call to the Fits program.
However, ‘DATASUM’ only depends on the data section of the
HDU/extension, so it is not changed when you add, remove or
update the header keywords. Therefore, it is recommended to
write these keywords as the last keywords that are
written/modified in the extension. You can use the ‘--verify’
option (described below) to verify the values of these two
keywords.
‘datasum’
Similar to ‘checksum’, but only write the ‘DATASUM’ keyword
(that does not depend on the header keywords, only the data).
‘-a STR’
‘--asis=STR’
Write the given ‘STR’ _exactly_ as it is, into the given FITS file
header with no modifications. If the contents of ‘STR’ does not
conform to the FITS standard for keywords, then it may (most
probably: it will!) corrupt your file and you may not be able to
open it any more. So please be *very careful* with this option
(its your responsibility to make sure that the string conforms with
the FITS standard for keywords).
If you want to define the keyword from scratch, it is best to use
the ‘--write’ option (see below) and let CFITSIO worry about
complying with the FITS standard. Also, you want to copy keywords
from one FITS file to another, you can use ‘--copykeys’ that is
described below. Through these high-level instances, you don't
have to worry about low-level issues.
One common usage of ‘--asis’ occurs when you are given the contents
of a FITS header (many keywords) as a plain-text file (so the
format of each keyword line conforms with the FITS standard, just
the file is plain-text, and you have one keyword per line when you
open it in a plain-text editor). In that case, Gnuastro's Fits
program won't be able to parse it (it doesn't conform to the FITS
standard, which doesn't have a new-line character!). With the
command below, you can insert those headers in ‘headers.txt’ into
‘img.fits’ (its HDU number 1, the default; you can change the HDU
to modify with ‘--hdu’).
$ cat headers.txt \
| while read line; do \
astfits img.fits --asis="$line"; \
done
*Don't forget a title:* Since the newly added headers in the
example above weren't originally in the file, they are probably
some form of high-level metadata. The raw example above will just
append the new keywords after the last one. Making it hard for
human readability (its not clear what this new group of keywords
signify, where they start, and where this group of keywords end).
To help the human readability of the header, add a title for this
group of keywords before writing them. To do that, run the
following command before the ‘cat ...’ command above (replace
‘Imported keys’ with any title that best describes this group of
new keywords based on their context):
$ astfits img.fits --write=/,"Imported keys"
‘-H STR’
‘--history STR’
Add a ‘HISTORY’ keyword to the header with the given value. A new
‘HISTORY’ keyword will be created for every instance of this
option. If the string given to this option is longer than 70
characters, it will be separated into multiple keyword cards. If
there is an error, Fits will give a warning and return with a
non-zero value, but will not stop. To stop as soon as an error
occurs, run with ‘--quitonerror’.
‘-c STR’
‘--comment STR’
Add a ‘COMMENT’ keyword to the header with the given value.
Similar to the explanation for ‘--history’ above.
‘-t’
‘--date’
Put the current date and time in the header. If the ‘DATE’ keyword
already exists in the header, it will be updated. If there is a
writing error, Fits will give a warning and return with a non-zero
value, but will not stop. To stop as soon as an error occurs, run
with ‘--quitonerror’.
‘-p’
‘--printallkeys’
Print the full metadata (keywords, values, units and comments) in
the specified FITS extension (HDU). If this option is called along
with any of the other keyword editing commands, as described above,
all other editing commands take precedence to this. Therefore, it
will print the final keywords after all the editing has been done.
‘--printkeynames’
Print only the keyword names of the specified FITS extension (HDU),
one line per name. This option must be called alone.
‘-v’
‘--verify’
Verify the ‘DATASUM’ and ‘CHECKSUM’ data integrity keywords of the
FITS standard. See the description under the ‘checksum’ (under
‘--write’, above) for more on these keywords.
This option will print ‘Verified’ for both keywords if they can be
verified. Otherwise, if they do not exist in the given
HDU/extension, it will print ‘NOT-PRESENT’, and if they cannot be
verified it will print ‘INCORRECT’. In the latter case (when the
keyword values exist but cannot be verified), the Fits program will
also return with a failure.
By default this function will also print a short description of the
‘DATASUM’ AND ‘CHECKSUM’ keywords. You can suppress this extra
information with ‘--quiet’ option.
‘--copykeys=INT:INT/STR,STR[,STR]’
Copy the desired set of the input's keyword records, to the to the
output (specified with the ‘--output’ and ‘--outhdu’ for the
filename and HDU/extension respectively). The keywords to copy can
be given either as a range (in the format of ‘INT:INT’, inclusive)
or a list of keyword names as comma-separated strings (‘STR,STR’),
the list can have any number of keyword names. More details and
examples of the two forms are given below:
Range
The given string to this option must be two integers separated
by a colon (<:>). The first integer must be positive
(counting of the keyword records starts from 1). The second
integer may be negative (zero is not acceptable) or an integer
larger than the first.
A negative second integer means counting from the end. So
‘-1’ is the last copy-able keyword (not including the ‘END’
keyword).
To see the header keywords of the input with a number before
them, you can pipe the output of the Fits program (when it
prints all the keywords in an extension) into the ‘cat’
program like below:
$ astfits input.fits -h1 | cat -n
List of names
The given string to this option must be a comma separated list
of keyword names. For example, see the command below:
$ astfits input.fits -h1 --copykeys=KEY1,KEY2 \
--output=output.fits --outhdu=1
Please consider the notes below when copying keywords with
names:
• If the number of characters in the name is more than 8,
CFITSIO will place a ‘HIERARCH’ before it. In this case
simply give the name and do not give the ‘HIERARCH’
(which is a constant and not considered part of the
keyword name).
• If your keyword name is composed only of digits, do not
give it as the first name given to ‘--copykeys’.
Otherwise, it will be confused with the range format
above. You can safely give an only-digit keyword name as
the second, or third requested keywords.
• If the keyword is repeated more than once in the header,
currently only the first instance will be copied. In
other words, even if you call ‘--copykeys’ multiple times
with the same keyword name, its first instance will be
copied. If you need to copy multiple instances of the
same keyword, please get in touch with us at
‘bug-gnuastro@gnu.org’.
‘--outhdu’
The HDU/extension to write the output keywords of ‘--copykeys’.
‘-Q’
‘--quitonerror’
Quit if any of the operations above are not successful. By default
if an error occurs, Fits will warn the user of the faulty keyword
and continue with the rest of actions.
‘-s STR’
‘--datetosec STR’
Interpret the value of the given keyword in the FITS date format
(most generally: ‘YYYY-MM-DDThh:mm:ss.ddd...’) and return the
corresponding Unix epoch time (number of seconds that have passed
since 00:00:00 Thursday, January 1st, 1970). The
‘Thh:mm:ss.ddd...’ section (specifying the time of day), and also
the ‘.ddd...’ (specifying the fraction of a second) are optional.
The value to this option must be the FITS keyword name that
contains the requested date, for example, ‘--datetosec=DATE-OBS’.
This option can also interpret the older FITS date format
(‘DD/MM/YYThh:mm:ss.ddd...’) where only two characters are given to
the year. In this case (following the GNU C Library), this option
will make the following assumption: values 68 to 99 correspond to
the years 1969 to 1999, and values 0 to 68 as the years 2000 to
2068.
This is a very useful option for operations on the FITS date
values, for example, sorting FITS files by their dates, or finding
the time difference between two FITS files. The advantage of
working with the Unix epoch time is that you do not have to worry
about calendar details (for example, the number of days in
different months, or leap years).
‘--wcscoordsys=STR’
Convert the coordinate system of the image's world coordinate
system (WCS) to the given coordinate system (‘STR’) and write it
into the file given to ‘--output’ (or an automatically named file
if no ‘--output’ has been given).
For example, with the command below, ‘img-eq.fits’ will have an
identical dataset (pixel values) as ‘image.fits’. However, the WCS
coordinate system of ‘img-eq.fits’ will be the equatorial
coordinate system in the Julian calendar epoch 2000 (which is the
most common epoch used today). Fits will automatically extract the
current coordinate system of ‘image.fits’ and as long as it is one
of the recognized coordinate systems listed below, it will do the
conversion.
$ astfits image.fits --wcscoordsys=eq-j2000 \
--output=img-eq.fits
The currently recognized coordinate systems are listed below (the
most common one today is ‘eq-j2000’):
‘eq-j2000’
2000.0 (Julian-year) equatorial coordinates. This is also
known as FK5 (short for "Fundamental Katalog No 5" which was
the source of the star coordinates used to define it).
This coordinate system is based on the motion of the Sun and
has epochs when the mean equator was used (for example
‘eq-b1950’ below). Furthermore, the definition of year is
different: either the Besselian year in 1950.0, or the Julian
year in 2000. For more on their difference and links for
further reading about epochs in astronomy, please see the
description in Wikipedia
(https://en.wikipedia.org/wiki/Epoch_(astronomy)).
Because of these difficulties, the equatorial J2000.0
coordinate system has been deprecated by the IAU in favor of
International Celestial Refernece System (ICRS) but is still
used extensively. ICRS is defined based on extra-galactic
quasars, so it does not depend on the dynamics of the solar
system any more. But to enable historical continuity, ICRS
has been defined to be equivalent to the equatorial J2000.0
within its accepted error bars of the latter (tens of
milli-arcseconds). This justifies the reason that moving to
ICRS has been relatively slow.
‘eq-b1950’
1950.0 (Besselian-year) equatorial coordinates.
‘ec-j2000’
2000.0 (Julian-year) ecliptic coordinates.
‘ec-b1950’
1950.0 (Besselian-year) ecliptic coordinates.
‘galactic’
Galactic coordinates.
‘supergalactic’
Supergalactic coordinates.
‘--wcsdistortion=STR’
If the argument has a WCS distortion, the output (file given with
the ‘--output’ option) will have the distortion given to this
option (for example, ‘SIP’, ‘TPV’). The output will be a new file
(with a copy of the image, and the new WCS), so if it already
exists, the file will be delete (unless you use the ‘--dontdelete’
option, see *note Input output options::).
With this option, the Fits program will read the minimal set of
keywords from the input HDU and the HDU data. It will then write
them into the file given to the ‘--output’ option but with a newly
created set of WCS-related keywords corresponding to the desired
distortion standard.
If no ‘--output’ file is specified, an automatically generated
output name will be used which is composed of the input's name but
with the ‘-DDD.fits’ suffix, see *note Automatic output::. Where
‘DDD’ is the value given to this option (desired output
distortion).
Note that all possible conversions between all standards are not
yet supported. If the requested conversion is not supported, an
informative error message will be printed. If this happens, please
let us know and we will try our best to add the respective
conversions.
For example, with the command below, you can be sure that if
‘in.fits’ has a distortion in its WCS, the distortion of ‘out.fits’
will be in the SIP standard.
$ astfits in.fits --wcsdistortion=SIP --output=out.fits
---------- Footnotes ----------
(1) Some tricky situations arise with values like '‘87095e5’', if
this was intended to be a number it will be kept in the header as
‘8709500000’ and there is no problem. But this can also be a shortened
Git commit hash. In the latter case, it should be treated as a string
and stored as it is written. Commit hashes are very important in
keeping the history of a file during your research and such values might
arise without you noticing them in your reproduction pipeline. One
solution is to use ‘git describe’ instead of the short hash alone. A
less recommended solution is to add a space after the commit hash and
Fits will write the value as '‘87095e5 ’' in the header. If you later
compare the strings on the shell, the space character will be ignored by
the shell in the latter solution and there will be no problem.
(2) <https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf>
File: gnuastro.info, Node: Pixel information images, Prev: Keyword inspection and manipulation, Up: Invoking astfits
5.1.1.3 Pixel information images
................................
In *note Keyword inspection and manipulation:: options like
‘--pixelscale’ were introduced for information on the pixels from the
keywords. But that only provides a single value for all the pixels!
This will not be sufficient in some scenarios; for example due to
distortion, different regions of the image will have different pixel
areas when projected onto the sky.
The options in this section provide such "meta" images: images where
the pixel values are information about the pixel itself. Such images
can be useful in understanding the underlying pixel grid with the same
tools that you study the astronomical objects within the image (like
*note SAO DS9::). After all, nothing beats visual inspection with tools
you are familiar with.
‘--pixelareaonwcs’
Create a meta-image where each pixel's value shows its area in the
WCS units (usually degrees squared). The output is therefore the
same size as the input.
This option uses the same "pixel mixing" or "area resampling"
concept that is described in *note Resampling:: (as part of the
Warp program). Similar to Warp, its sampling can be tuned with the
‘--edgesampling’ that is described below.
One scenario where this option becomes handy is when you are
debugging aligned images using the Warp program (see *note Warp::).
You may observe gradients after warping and can check if they
caused by the distortion of the instrument or not. Such gradients
can happen due to distortions because the detectors pixels are
measuring photons from different areas on the sky (or the type of
projection you're seeing). This effect is more pronounced in
images covering larger portions of the sky, for instance, the TESS
images(1).
Here is an example usage of the ‘--pixelareaonwcs’ option:
# Check the area each 'input.fits' pixel takes in sky
$ astfits input.fits -h1 --pixelareaonwcs -o pixarea.fits
# Convert each pixel's area to arcsec^2
$ astarithmetic pixarea.fits 3600 3600 x x \
--output=pixarea_arcsec2.fits
# Compare area relative to the actual reported pixel scale
$ pixarea=$(astfits input.fits --pixelscale -q \
| awk '{print $3}')
$ astarithmetic pixarea.fits $pixarea / -o pixarea_rel.fits
‘--edgesampling=INT’
Extra sampling along the pixel edges for ‘--pixelareaonwcs’. The
default value is 0, meaning that only the pixel vertices are used.
Values greater than zero improve the accuracy in the expense of
greater time and memory consumption. With that said, the default
value of zero usually has a good precision unless the given image
has extreme distortions that produce irregular pixel shapes. For
more, see *note Align pixels with WCS considering distortions::).
*Caution:* This option does not "oversample" the output image!
Rather, it makes Warp use more points to calculate the _input_
pixel area. To oversample the output image, set a reasonable
‘--cdelt’ value.
---------- Footnotes ----------
(1) <https://www.nasa.gov/tess-transiting-exoplanet-survey-satellite>
File: gnuastro.info, Node: ConvertType, Next: Table, Prev: Fits, Up: Data containers
5.2 ConvertType
===============
The FITS format used in astronomy was defined mainly for archiving,
transmission, and processing. In other situations, the data might be
useful in other formats. For example, when you are writing a paper or
report, or if you are making slides for a talk, you cannot use a FITS
image. Other image formats should be used. In other cases you might
want your pixel values in a table format as plain text for input to
other programs that do not recognize FITS. ConvertType is created for
such situations. The various types will increase with future updates
and based on need.
The conversion is not only one way (from FITS to other formats), but
two ways (except the EPS and PDF formats(1)). So you can also convert a
JPEG image or text file into a FITS image. Basically, other than
EPS/PDF, you can use any of the recognized formats as different color
channel inputs to get any of the recognized outputs.
Before explaining the options and arguments (in *note Invoking
astconvertt::), we will start with a short discussion on the difference
between raster and vector graphics in *note Raster and Vector
graphics::. In ConvertType, vector graphics are used to add markers
over your originally rasterized data, producing high quality images,
ready to be used in your exciting papers. We will continue with a
description of the recognized files types in *note Recognized file
formats::, followed a short introduction to digital color in *note
Color::. A tutorial on how to add markers over an image is then given
in *note Marking objects for publication:: and we conclude with a LaTeX
based solution to add coordinates over an image.
* Menu:
* Raster and Vector graphics:: Images coming from nature, and the abstract.
* Recognized file formats:: Recognized file formats
* Color:: Some explanations on color.
* Annotations for figure in paper:: Adding coordinates or physical scale.
* Invoking astconvertt:: Options and arguments to ConvertType.
---------- Footnotes ----------
(1) Because EPS and PDF are vector, not raster/pixelated formats
File: gnuastro.info, Node: Raster and Vector graphics, Next: Recognized file formats, Prev: ConvertType, Up: ConvertType
5.2.1 Raster and Vector graphics
--------------------------------
Images that are produced by a hardware (for example, the camera in your
phone, or the camera connected to a telescope) provide pixelated data.
Such data are therefore stored in a Raster graphics
(https://en.wikipedia.org/wiki/Raster_graphics) format which has
discrete, independent, equally spaced data elements. For example, this
is the format used FITS (see *note Fits::), JPEG, TIFF, PNG and other
image formats.
On the other hand, when something is generated by the computer (for
example, a diagram, plot or even adding a cross over a camera image to
highlight something there), there is no "observation" or connection with
nature! Everything is abstract! For such things, it is much easier to
draw a mathematical line (with infinite resolution). Therefore, no
matter how much you zoom-in, it will never get pixelated. This is the
realm of Vector graphics
(https://en.wikipedia.org/wiki/Vector_graphics). If you open the
Gnuastro manual in PDF format
(https://www.gnu.org/software/gnuastro/manual/gnuastro.pdf) You can see
such graphics in the Gnuastro manual, for example, in *note Circles and
the complex plane:: or *note Distance on a 2D curved space::. The most
common vector graphics format is PDF for document sharing or SVG for
web-based applications.
The pixels of a raster image can be shown as vector-based squares
with different shades, so vector graphics can generally also support
raster graphics. This is very useful when you want to add some graphics
over an image to help your discussion (for example a $+$ over your
object of interest). However, vector graphics is not optimized for
rasterized data (which are usually also noisy!), and can either not
display nicely, or result in much larger file volume (in bytes).
Therefore, if it is not necessary to add any marks over a FITS image,
for example, it may be better to store it in a rasterized format.
The distinction between the vector and raster graphics is also the
primary theme behind Gnuastro's logo, see *note Logo of Gnuastro::.
File: gnuastro.info, Node: Recognized file formats, Next: Color, Prev: Raster and Vector graphics, Up: ConvertType
5.2.2 Recognized file formats
-----------------------------
The various standards and the file name extensions recognized by
ConvertType are listed below. For a review on the difference between
Raster and Vector graphics, see *note Raster and Vector graphics::. For
a review on the concept of color and channels, see *note Color::.
Currently, except for the FITS format, Gnuastro uses the file name's
suffix to identify the format, so if the file's name does not end with
one of the suffixes mentioned below, it will not be recognized.
FITS or IMH
Astronomical data are commonly stored in the FITS format (or the
older data IRAF ‘.imh’ format), a list of file name suffixes which
indicate that the file is in this format is given in *note
Arguments::. FITS is a raster graphics format.
Each image extension of a FITS file only has one value per
pixel/element. Therefore, when used as input, each input FITS
image contributes as one color channel. If you want multiple
extensions in one FITS file for different color channels, you have
to repeat the file name multiple times and use the ‘--hdu’,
‘--hdu2’, ‘--hdu3’ or ‘--hdu4’ options to specify the different
extensions.
JPEG
The JPEG standard was created by the Joint photographic experts
group. It is currently one of the most commonly used image
formats. Its major advantage is the compression algorithm that is
defined by the standard. Like the FITS standard, this is a raster
graphics format, which means that it is pixelated.
A JPEG file can have 1 (for gray-scale), 3 (for RGB) and 4 (for
CMYK) color channels. If you only want to convert one JPEG image
into other formats, there is no problem, however, if you want to
use it in combination with other input files, make sure that the
final number of color channels does not exceed four. If it does,
then ConvertType will abort and notify you.
The file name endings that are recognized as a JPEG file for input
are: ‘.jpg’, ‘.JPG’, ‘.jpeg’, ‘.JPEG’, ‘.jpe’, ‘.jif’, ‘.jfif’ and
‘.jfi’.
TIFF
TIFF (or Tagged Image File Format) was originally designed as a
common format for scanners in the early 90s and since then it has
grown to become very general. In many aspects, the TIFF standard
is similar to the FITS image standard: it can allow data of many
types (see *note Numeric data types::), and also allows multiple
images to be stored in a single file (like a FITS extension: each
image in the file is called a 'directory' in the TIFF standard).
However, unlike FITS, it can only store images, it has no
constructs for tables. Also unlike FITS, each 'directory' of a
TIFF file can have a multi-channel (e.g., RGB) image. Another
(inconvenient) difference with the FITS standard is that keyword
names are stored as numbers, not human-readable text.
However, outside of astronomy, because of its support of different
numeric data types, many fields use TIFF images for accurate (for
example, 16-bit integer or floating point for example) imaging
data.
EPS
The Encapsulated PostScript (EPS) format is essentially a one page
PostScript file which has a specified size. Postscript is used to
store a full document like this whole Gnuastro book. PostScript
therefore also includes non-image data, for example, lines and
texts. It is a fully functional programming language to describe a
document. A PostScript file is a plain text file that can be
edited like any program source with any plain-text editor.
Therefore in ConvertType, EPS is only an output format and cannot
be used as input. Contrary to the FITS or JPEG formats, PostScript
is not a raster format, but is categorized as vector graphics.
With these features in mind, you can see that when you are
compiling a document with TeX or LaTeX, using an EPS file is much
more low level than a JPEG and thus you have much greater control
and therefore quality. Since it also includes vector graphic lines
we also use such lines to make a thin border around the image to
make its appearance in the document much better. Furthermore,
through EPS, you can add marks over the image in many shapes and
colors. No matter the resolution of the display or printer, these
lines will always be clear and not pixelated. However, this can be
done better with tools within TeX or LaTeX such as PGF/Tikz(1).
If the final input image (possibly after all operations on the flux
explained below) is a binary image or only has two colors of black
and white (in segmentation maps for example), then PostScript has
another great advantage compared to other formats. It allows for 1
bit pixels (pixels with a value of 0 or 1), this can decrease the
output file size by 8 times. So if a gray-scale image is binary,
ConvertType will exploit this property in the EPS and PDF (see
below) outputs.
The standard formats for an EPS file are ‘.eps’, ‘.EPS’, ‘.epsf’
and ‘.epsi’. The EPS outputs of ConvertType have the ‘.eps’
suffix.
PDF
The Portable Document Format (PDF) is currently the most common
format for documents. It is a vector graphics format, allowing
abstract constructs like marks or borders.
The PDF format is based on Postscript, so it shares all the
features mentioned above for EPS. To be able to display it is
programmed content or print, a Postscript file needs to pass
through a processor or compiler. A PDF file can be thought of as
the processed output of the PostScript compiler. PostScript, EPS
and PDF were created and are registered by Adobe Systems.
As explained under EPS above, a PDF document is a static document
description format, viewing its result is therefore much faster and
more efficient than PostScript. To create a PDF output,
ConvertType will make an EPS file and convert that to PDF using GPL
Ghostscript. The suffixes recognized for a PDF file are: ‘.pdf’,
‘.PDF’. If GPL Ghostscript cannot be run on the PostScript file,
The EPS will remain and a warning will be printed (see *note
Optional dependencies::).
‘blank’
This is not actually a file type! But can be used to fill one
color channel with a blank value. If this argument is given for
any color channel, that channel will not be used in the output.
Plain text
The value of each pixel in a 2D image can be written as a 2D matrix
in a plain-text file. Therefore, for the purpose of ConvertType,
plain-text files are a single-channel raster graphics file format.
Plain text files have the advantage that they can be viewed with
any text editor or on the command-line. Most programs also support
input as plain text files. As input, each plain text file is
considered to contain one color channel.
In ConvertType, the recognized extensions for plain text files are
‘.txt’ and ‘.dat’. As described in *note Invoking astconvertt::,
if you just give these extensions, (and not a full filename) as
output, then automatic output will be preformed to determine the
final output name (see *note Automatic output::). Besides these,
when the format of a file cannot be recognized from its name,
ConvertType will fall back to plain text mode. So you can use any
name (even without an extension) for a plain text input or output.
Just note that when the suffix is not recognized, automatic output
will not be preformed.
The basic input/output on plain text images is very similar to how
tables are read/written as described in *note Gnuastro text table
format::. Simply put, the restrictions are very loose, and there
is a convention to define a name, units, data type (see *note
Numeric data types::), and comments for the data in a commented
line. The only difference is that as a table, a text file can
contain many datasets (columns), but as a 2D image, it can only
contain one dataset. As a result, only one information comment
line is necessary for a 2D image, and instead of the starting '‘#
Column N’' (‘N’ is the column number), the information line for a
2D image must start with '‘# Image 1’'. When ConvertType is asked
to output to plain text file, this information comment line is
written before the image pixel values.
When converting an image to plain text, consider the fact that if
the image is large, the number of columns in each line will become
very large, possibly making it very hard to open in some text
editors.
Standard output (command-line)
This is very similar to the plain text output, but instead of
creating a file to keep the printed values, they are printed on the
command-line. This can be very useful when you want to redirect
the results directly to another program in one command with no
intermediate file. The only difference is that only the pixel
values are printed (with no information comment line). To print to
the standard output, set the output name to '‘stdout’'.
---------- Footnotes ----------
(1) <http://sourceforge.net/projects/pgf/>
File: gnuastro.info, Node: Color, Next: Annotations for figure in paper, Prev: Recognized file formats, Up: ConvertType
5.2.3 Color
-----------
Color is generally defined after mixing various data "channels". The
values for each channel usually come a filter that is placed in the
optical path. Filters, only allow a certain window of the spectrum to
pass (for example, the SDSS _r_ filter only allows light from about 5500
to 7000 Angstroms). In digital monitors or common digital cameras, a
different set of filters are used: Red, Green and Blue (commonly known
as RGB) that are more optimized to the eye's perception. On the other
hand, when printing on paper, standard printers use the cyan, magenta,
yellow and key (CMYK, key=black) color space.
* Menu:
* Pixel colors:: Multiple filters in each pixel.
* Colormaps for single-channel pixels:: Better display of single-filter images.
* Vector graphics colors::
File: gnuastro.info, Node: Pixel colors, Next: Colormaps for single-channel pixels, Prev: Color, Up: Color
5.2.3.1 Pixel colors
....................
As discussed in *note Color::, for each displayed/printed pixel of a
color image, the dataset/image has three or four values. To store/show
the three values for each pixel, cameras and monitors allocate a certain
fraction of each pixel's area to red, green and blue filters. These
three filters are thus built into the hardware at the pixel level.
However, because measurement accuracy is very important in scientific
instruments, and we want to do measurements (take images) with
various/custom filters (without having to order a new expensive
detector!), scientific detectors use the full area of the pixel to store
one value for it in a single/mono channel dataset. To make measurements
in different filters, we just place a filter in the light path before
the detector. Therefore, the FITS format that is used to store
astronomical datasets is inherently a mono-channel format (see *note
Recognized file formats:: or *note Fits::).
When a subject has been imaged in multiple filters, you can feed each
different filter into the red, green and blue channels of your monitor
and obtain a false-colored visualization. The reason we say
"false-color" (or pseudo color) is that generally, the three data
channels you provide are not from the same Red, Green and Blue filters
of your monitor! So the observed color on your monitor does not
correspond the physical "color" that you would have seen if you looked
at the object by eye. Nevertheless, it is good (and sometimes
necessary) for visualization (of special features).
In ConvertType, you can do this by giving each separate
single-channel dataset (for example, in the FITS image format) as an
argument (in the proper order), then asking for the output in a format
that supports multi-channel datasets (for example, see the command
below, or *note ConvertType input and output::).
$ astconvertt r.fits g.fits b.fits --output=color.jpg
File: gnuastro.info, Node: Colormaps for single-channel pixels, Next: Vector graphics colors, Prev: Pixel colors, Up: Color
5.2.3.2 Colormaps for single-channel pixels
...........................................
As discussed in *note Pixel colors::, color is not defined when a
dataset/image contains a single value for each pixel. However, we
interact with scientific datasets through monitors or printers. They
allow multiple channels (independent values) per pixel and produce color
with them (on monitors, this is usually with three channels: Red, Green
and Blue). As a result, there is a lot of freedom in visualizing a
single-channel dataset.
The mapping of single-channel values to multi-channel colors is
called called a "color map". Since more information can be put in
multiple channels, this usually results in better visualizing the
dynamic range of your single-channel data. In ConvertType, you can use
the ‘--colormap’ option to choose between different mappings of
mono-channel inputs, see *note Invoking astconvertt::. Below, we will
review two of the basic color maps, please see the description of
‘--colormap’ in *note Invoking astconvertt:: for the full list.
• The most basic colormap is shades of black (because of its strong
contrast with white). This scheme is called Grayscale
(https://en.wikipedia.org/wiki/Grayscale). But ultimately, the
black is just one color, so with Grayscale, you are not using the
full dynamic range of the three-channel monitor effectively. To
help in visualization, more complex mappings can be defined.
• A slightly more complex color map can be defined when you scale the
values to a range of 0 to 360, and use as it as the "Hue" term of
the Hue-Saturation-Value
(https://en.wikipedia.org/wiki/HSL_and_HSV) (HSV) color space
(while fixing the "Saturation" and "Value" terms). The increased
usage of the monitor's 3-channel color space is indeed better, but
the resulting images can be un-"natural" to the eye.
Since grayscale is a commonly used mapping of single-valued datasets,
we will continue with a closer look at how it is stored. One way to
represent a gray-scale image in different color spaces is to use the
same proportions of the primary colors in each pixel. This is the
common way most FITS image viewers work: for each pixel, they fill all
the channels with the single value. While this is necessary for
displaying a dataset, there are downsides when storing/saving this type
of grayscale visualization (for example, in a paper).
• Three (for RGB) or four (for CMYK) values have to be stored for
every pixel, this makes the output file very heavy (in terms of
bytes).
• If printing, the printing errors of each color channel can make the
printed image slightly more blurred than it actually is.
To solve both these problems when storing grayscale visualization,
the best way is to save a single-channel dataset into the black channel
of the CMYK color space. The JPEG standard is the only common standard
that accepts CMYK color space.
The JPEG and EPS standards set two sizes for the number of bits in
each channel: 8-bit and 12-bit. The former is by far the most common
and is what is used in ConvertType. Therefore, each channel should have
values between 0 to 2^8-1=255. From this we see how each pixel in a
gray-scale image is one byte (8 bits) long, in an RGB image, it is 3
bytes long and in CMYK it is 4 bytes long. But thanks to the JPEG
compression algorithms, when all the pixels of one channel have the same
value, that channel is compressed to one pixel. Therefore a Grayscale
image and a CMYK image that has only the K-channel filled are
approximately the same file size.
File: gnuastro.info, Node: Vector graphics colors, Prev: Colormaps for single-channel pixels, Up: Color
5.2.3.3 Vector graphics colors
..............................
When creating vector graphics, ConvertType recognizes the extended web
colors (https://en.wikipedia.org/wiki/Web_colors#Extended_colors) that
are the result of merging the colors in the HTML 4.01, CSS 2.0, SVG 1.0
and CSS3 standards. They are all shown with their standard name in
*note Figure 5.1: colornames. The names are not case sensitive so you
can use them in any form (for example, ‘turquoise’ is the same as
‘Turquoise’ or ‘TURQUOISE’).
On the command-line, you can also get the list of colors with the
‘--listcolors’ option to CovertType, like below. In particular, if your
terminal is 24-bit or "true color", in the last column, you will see
each color. This greatly helps in selecting the best color for our
purpose easily on the command-line (without taking your hands off the
keyboard and getting distracted).
$ astconvertt --listcolors
[image src="gnuastro-figures/color-names.png" text="../gnuastro-figures//color-names.eps" ]
Figure 5.1: Recognized color names in Gnuastro, shown with their
numerical identifiers.
File: gnuastro.info, Node: Annotations for figure in paper, Next: Invoking astconvertt, Prev: Color, Up: ConvertType
5.2.4 Annotations for figure in paper
-------------------------------------
To make a nice figure from your FITS images, it is important to show
more than merely the raw image (converted to a printer friendly format
like PDF or JPEG). Annotations (or visual metadata) over the raw image
greatly help the readers clearly see your argument and put the
image/result in a larger context. Examples include:
• Coordinates (Right Ascension and Declination) on the edges of the
image, so viewers of your paper or presentation slides can get a
physical feeling of the field's sky coverage.
• Thick line that has a fixed tangential size (for example, in kilo
parsecs) at the redshift/distance of interest.
• Contours over the image to show radio/X-ray emission, over an
optical image for example.
• Text, arrows, etc., over certain parts of the image.
Because of the modular philosophy of Gnuastro, ConvertType is only
focused on converting your FITS images to printer friendly formats like
JPEG or PDF. But to present your results in a slide or paper, you will
often need to annotate the raw JPEG or PDF with some of the features
above. The good news is that there are many powerful plotting programs
that you can use to add such annotations. As a result, there is no
point in making a new one, specific to Gnuastro. In this section, we
will demonstrate this using the very powerful PGFPlots(1) package of
LaTeX.
*Single script for easy running:* In this section we are reviewing the
reason and details of every step which is good for educational purposes.
But when you know the steps already, these separate code blocks can be
annoying. Therefore the full script (except for the data download step)
is available in *note Full script of annotations on figure::.
PGFPlots uses the same LaTeX graphic engine that typesets your
paper/slide. Therefore when you build your plots and figures using
PGFPlots (and its underlying package PGF/TiKZ(2)) your plots will blend
beautifully within your text: same fonts, same colors, same line
properties, etc. Since most papers (and presentation slides(3)) are
made with LaTeX, PGFPlots is therefore the best tool for those who use
LaTeX to create documents. PGFPlots also does not need any extra
dependencies beyond a basic/minimal TeX-live installation, so it is much
more reliable than tools like Matplotlib in Python that have hundreds of
fast-evolving dependencies(4).
To demonstrate this, we will create a surface brightness image of a
galaxy in the F160W filter of the ABYSS survey(5). In the code-block
below, let's make a "build" directory to keep intermediate files and
avoid populating the source. Afterwards, we will download the full
image and crop out a 20 arcmin wide image around the galaxy with the
commands below. You can run these commands in an empty directory.
$ mkdir build
$ wget http://cdsarc.u-strasbg.fr/ftp/J/A+A/621/A133/fits/ah_f160w.fits
$ astcrop ah_f160w.fits --center=53.1616278,-27.7802446 --mode=wcs \
--width=20/3600 --output=build/crop.fits
To better show the low surface brightness (LSB) outskirts, we will
warp the image, then convert the pixel units to surface brightness with
the commands below. It is very important that the warping is done
_before_ the conversion to surface brightness (in units of
mag/arcsec$^2$), because the definition of surface brightness is
non-linear. For more, see the surface brightness topic of *note
Brightness flux magnitude::, and for a more complete tutorial, see *note
FITS images in a publication::.
$ zeropoint=25.94
$ astwarp build/crop.fits --centeroncorner --scale=1/3 \
--output=build/scaled.fits
$ pixarea=$(astfits build/scaled.fits --pixelareaarcsec2)
$ astarithmetic build/scaled.fits $zeropoint $pixarea counts-to-sb \
--output=build/sb.fits
We are now ready to convert the surface brightness image into a PDF.
To better show the LSB features, we will also limit the color range with
the ‘--fluxlow’ and ‘--fluxhigh’ options: all pixels with a surface
brightness brighter than 22 mag/arcsec$^2$ will be shown as black, and
all pixels with a surface brightness fainter than 30 mag/arcsec$^2$ will
be white. These thresholds are being defined as variables, because we
will also need them below (to pass into PGFPlots). We will also set
‘--borderwidth=0’, because the coordinate system we will add over the
image will effectively be a border for the image (separating it from the
background).
$ sblow=22
$ sbhigh=30
$ astconvertt build/sb.fits --colormap=gray --borderwidth=0 \
--fluxhigh=$sbhigh --fluxlow=$sblow --output=build/sb.pdf
Please open ‘sb.pdf’ and have a look. Also, please open ‘sb.fits’ in
DS9 (or any other FITS viewer) and play with the color range. Can the
surface brightness limits be changed to better show the LSB structure?
If so, you are free to change the limits above.
We now have the printable PDF representation of the image, but as
discussed above, it is not enough for a paper. We will add 1) a thick
line showing the size of 20 kpc (kilo parsecs) at the redshift of the
central galaxy, 2) coordinates and 3) a color bar, showing the surface
brightness level of each grayscale level.
To get the first job done, we first need to know the redshift of the
central galaxy. To do this, we can use Gnuastro's Query program to look
into all the objects in NED within this image (only asking for the RA,
Dec and redshift columns). We will then use the Match program to find
the NED entry that corresponds to our galaxy.
$ astquery ned --dataset=objdir --overlapwith=build/sb.fits \
--column=ra,dec,z --output=ned.fits
$ astmatch ned.fits -h1 --coord=53.1616278,-27.7802446 \
--ccol1=RA,Dec --aperture=1/3600
$ redshift=$(asttable ned_matched.fits -cz)
$ echo $redshift
Now that we know the redshift of the central object, we can define
the coordinates of the thick line that will show the length of 20 kpc at
that redshift. It will be a horizontal line (fixed Declination) across
a range of RA. The start of this thick line will be located at the top
edge of the image (at the 95-percent of the width and height of the
image). With the commands below we will find the three necessary
parameters (one declination and two RAs). Just note that in
astronomical images, RA increases to the left/east, which is the reason
we are using the minimum and ‘+’ to find the RA starting point.
$ scalelineinkpc=20
$ coverage=$(astfits build/sb.fits --skycoverage --quiet | awk 'NR==2')
$ scalelinedec=$(echo $coverage | awk '{print $4-($4-$3)*0.05}')
$ scalelinerastart=$(echo $coverage | awk '{print $1+($2-$1)*0.05}')
$ scalelineraend=$(astcosmiccal --redshift=$redshift --arcsectandist \
| awk '{start='$scalelinerastart'; \
width='$scalelineinkpc'/$1/3600; \
print start+width}')
To draw coordinates over the image, we need to feed these values into
PGFPlots. But manually entering numbers into the PGFPlots source will
be very frustrating and prone to many errors! Fortunately there is an
easy way to do this: LaTeX macros. New macros are defined by this LaTeX
command:
\newcommand{\macroname}{value}
Anywhere that LaTeX confronts ‘\macroname’, it will replace ‘value’ when
building the output. We will have one file called ‘macros.tex’ in the
build directory and define macros based on those values. We will use
the shell's ‘printf’ command to write these macro definition lines into
the macro file. We just have to use double backslashes in the ‘printf’
command, because backslash is a meaningful character for ‘printf’, but
we want to keep one of them. Also, we put a ‘\n’ at the end of each
line, otherwise, all the commands will go into a single line of the
macro file. We will also place the random '‘ma’' string at the start of
all our LaTeX macros to help identify the macros for this plot.
$ macros=build/macros.tex
$ printf '\\newcommand{\\maScaleDec}'"{$scalelinedec}\n" > $macros
$ printf '\\newcommand{\\maScaleRAa}'"{$scalelinerastart}\n" >> $macros
$ printf '\\newcommand{\\maScaleRAb}'"{$scalelineraend}\n" >> $macros
$ printf '\\newcommand{\\maScaleKpc}'"{$scalelineinkpc}\n" >> $macros
$ printf '\\newcommand{\\maCenterZ}'"{$redshift}\n" >> $macros
Please open the macros file after these commands and have a look to
see if they do conform to the expected format above. Another set of
macros we will need to feed into PGFPlots is the coordinates of the
image corners. Fortunately the ‘coverage’ variable found above is also
useful here. We just need to extract each item before feeding it into
the macros. To do this, we will use AWK and keep each value with the
temporary shell variable '‘v’'.
$ v=$(echo $coverage | awk '{print $1}')
$ printf '\\newcommand{\\maCropRAMin}'"{$v}\n" >> $macros
$ v=$(echo $coverage | awk '{print $2}')
$ printf '\\newcommand{\\maCropRAMax}'"{$v}\n" >> $macros
$ v=$(echo $coverage | awk '{print $3}')
$ printf '\\newcommand{\\maCropDecMin}'"{$v}\n" >> $macros
$ v=$(echo $coverage | awk '{print $4}')
$ printf '\\newcommand{\\maCropDecMax}'"{$v}\n" >> $macros
Finally, we also need to pass some other numbers to PGFPlots: 1) the
major tick distance (in the coordinate axes that will be printed on the
edge of the image). We will assume 7 ticks for this image. 2) The
minimum and maximum surface brightness values that we gave to
ConvertType when making the PDF; PGFPlots will define its color-bar
based on these two values.
$ v=$(echo $coverage | awk '{print ($2-$1)/7}')
$ printf '\\newcommand{\\maTickDist}'"{$v}\n" >> $macros
$ printf '\\newcommand{\\maSBlow}'"{$sblow}\n" >> $macros
$ printf '\\newcommand{\\maSBhigh}'"{$sbhigh}\n" >> $macros
All the necessary numbers are now ready. Please copy the contents
below into a file called ‘my-figure.tex’. This is the PGFPlots source
for this particular plot. Besides the coordinates and scale-line, we
will also add some text over the image and an orange arrow pointing to
the central object with its redshift printed over it. The parameters
are generally human-readable, so you should be able to get a good
feeling of every line. There are also comments which will show up as a
different color when you copy this into a plain-text editor.
\begin{tikzpicture}
%% Define the coordinates and colorbar
\begin{axis}[
at={(0,0)},
axis on top,
x dir=reverse,
scale only axis,
width=\linewidth,
height=\linewidth,
minor tick num=10,
xmin=\maCropRAMin,
xmax=\maCropRAMax,
ymin=\maCropDecMin,
ymax=\maCropDecMax,
enlargelimits=false,
every tick/.style={black},
xtick distance=\maTickDist,
ytick distance=\maTickDist,
yticklabel style={rotate=90},
ylabel={Declination (degrees)},
xlabel={Right Ascension (degrees)},
ticklabel style={font=\small,
/pgf/number format/.cd, precision=4,/tikz/.cd},
x label style={at={(axis description cs:0.5,0.02)},
anchor=north,font=\small},
y label style={at={(axis description cs:0.07,0.5)},
anchor=south,font=\small},
colorbar,
colormap name=gray,
point meta min=\maSBlow,
point meta max=\maSBhigh,
colorbar style={
at={(1.01,1)},
ylabel={Surface brightness (mag/arcsec$^2$)},
yticklabel style={
/pgf/number format/.cd, precision=1, /tikz/.cd},
y label style={at={(axis description cs:5.3,0.5)},
anchor=south,font=\small},
},
]
%% Put the image in the proper positions of the plot.
\addplot graphics[ xmin=\maCropRAMin, xmax=\maCropRAMax,
ymin=\maCropDecMin, ymax=\maCropDecMax]
{sb.pdf};
%% Draw the scale factor.
\addplot[black, line width=5, name=scaleline] coordinates
{(\maScaleRAa,\maScaleDec) (\maScaleRAb,\maScaleDec)}
node [anchor=north west] {\large $\maScaleKpc$ kpc};
\end{axis}
%% Add some text anywhere over the plot. The text is added two
%% times: the first time with a white background (that with a
%% certain opacity), the second time just the text with opacity.
\node[anchor=south west, fill=white, opacity=0.5]
at (0.01\linewidth,0.01\linewidth)
{(a) Text can be added here};
\node[anchor=south west]
at (0.01\linewidth,0.01\linewidth)
{(a) Text can be added here};
%% Add an arrow to highlight certain structures.
\draw [->, red!70!yellow, line width=5]
(0.35\linewidth,0.35\linewidth)
-- node [anchor=south, rotate=45]{$z=\maCenterZ$}
(0.45\linewidth,0.45\linewidth);
\end{tikzpicture}
Finally, we need another simple LaTeX source for the main PDF
"report" that will host this figure. This can actually be your paper or
slides for example. Here, we will suffice to the minimal working
example.
\documentclass{article}
%% Import the TiKZ package and activate its "external" feature.
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize
%% PGFPlots (which uses TiKZ).
\usepackage{pgfplots}
\pgfplotsset{axis line style={thick}}
\pgfplotsset{
/pgfplots/colormap={gray}{rgb255=(0,0,0) rgb255=(255,255,255)}
}
%% Import the macros.
\input{macros.tex}
%% Start document.
\begin{document}
You can write anything here.
%% Add the figure and its caption.
\begin{figure}
\input{my-figure.tex}
\caption{A demo image.}
\end{figure}
%% Finish the document.
\end{document}
You are now ready to create the PDF. But LaTeX creates many temporary
files, so to avoid populating our top-level directory, we will copy the
two ‘.tex’ files into the build directory, go there and run LaTeX.
Before running it though, we will first delete all the files that have
the name pattern ‘*-figure0*’, these are "external" files created by
TiKZ+PGFPlots, including the actual PDF of the figure.
$ cp report.tex my-figure.tex build
$ cd build
$ rm -f *-figure0*
$ pdflatex -shell-escape -halt-on-error report.tex
You now have the full "report" in ‘report.pdf’. Try adding some
extra text on top of the figure, or in the caption and re-running the
last four commands. Also try changing the 20kpc scale line length to
50kpc, or try changing the redshift, to see how the length and text of
the thick scale-line will automatically change. But the good news is
that you also have the raw PDF of the figure that you can use in other
places. You can see that file in ‘report-figure0.pdf’.
In a larger paper, you can add multiple such figures (with different
‘.tex’ files that are placed in different ‘figure’ environments with
different captions). Each figure will get a number in the build
directory. TiKZ also allows setting a file name for each "external"
figure (to avoid such numbers that can be annoying if the image orders
are changed). PGFPlots is also highly customizable, you can make a lot
of changes and customizations. Both TiKZ(6) and PGFPLots(7) have
wonderful manuals, so have a look trough them.
* Menu:
* Full script of annotations on figure:: All the steps in one script
---------- Footnotes ----------
(1)
<http://mirrors.ctan.org/graphics/pgf/contrib/pgfplots/doc/pgfplots.pdf>
(2) <http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf>
(3) To build slides, LaTeX has packages like Beamer, see
<http://mirrors.ctan.org/macros/latex/contrib/beamer/doc/beameruserguide.pdf>
(4) See Figure 1 of Alliez et al. 2019
(https://arxiv.org/abs/1905.11123).
(5) <http://research.iac.es/proyecto/abyss>
(6) <http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf>
(7)
<http://mirrors.ctan.org/graphics/pgf/contrib/pgfplots/doc/pgfplots.pdf>
File: gnuastro.info, Node: Full script of annotations on figure, Prev: Annotations for figure in paper, Up: Annotations for figure in paper
5.2.4.1 Full script of annotations on figure
............................................
In *note Annotations for figure in paper::, we each one of the steps to
add annotations over an image were described in detail. So if you have
understood the steps, but need to add annotations over an image,
repeating those steps individually will be annoying. Therefore in this
section, we will summarize all the steps in a single script that you can
simply copy-paste into a text editor, configure, and run.
*Necessary files:* To run this script, you will need an image to crop
your object from (here assuming it is called ‘ah_f160w.fits’ with a
certain zero point) and two ‘my-figure.tex’ and ‘report.tex’ files that
were fully included in *note Annotations for figure in paper::. Also,
we have brought the redshift as a parameter here. But if the center of
your image always points to your main object, you can also include the
Query command to automatically find the object's redshift from NED.
Alternatively, your image may already be cropped, in this case, you can
remove the cropping step and
# Parameters.
sblow=22 # Minimum surface brightness.
sbhigh=30 # Maximum surface brightness.
bdir=build # Build directory location on filesystem.
numticks=7 # Number of major ticks in each axis.
redshift=0.619 # Redshift of object of interest.
zeropoint=25.94 # Zero point of input image.
scalelineinkpc=20 # Length of scale-line (in kilo parsecs).
input=ah_f160w.fits # Name of input (to crop).
# Stop the script in case of a crash.
set -e
# Build directory
if ! [ -d $bdir ]; then mkdir $bdir; fi
# Crop out the desired region.
crop=$bdir/crop.fits
astcrop $input --center=53.1616278,-27.7802446 --mode=wcs \
--width=20/3600 --output=$crop
# Warp the image to larger pixels to show surface brightness better.
scaled=$bdir/scaled.fits
astwarp $crop --centeroncorner --scale=1/3 --output=$scaled
# Calculate the pixel area and convert image to Surface brightness.
sb=$bdir/sb.fits
pixarea=$(astfits $scaled --pixelareaarcsec2)
astarithmetic $scaled $zeropoint $pixarea counts-to-sb \
--output=$sb
# Convert the surface brightness image into PDF.
sbpdf=$bdir/sb.pdf
astconvertt $sb --colormap=gray --borderwidth=0 \
--fluxhigh=$sbhigh --fluxlow=$sblow --output=$sbpdf
# Specify the coordinates of the scale line (specifying a certain
# width in kpc). We will put it on the top-right side of the image (5%
# of the full width of the image away from the edge).
coverage=$(astfits $sb --skycoverage --quiet | awk 'NR==2')
scalelinedec=$(echo $coverage | awk '{print $4-($4-$3)*0.05}')
scalelinerastart=$(echo $coverage | awk '{print $1+($2-$1)*0.05}')
scalelineraend=$(astcosmiccal --redshift=$redshift --arcsectandist \
| awk '{start='$scalelinerastart'; \
width='$scalelineinkpc'/$1/3600; \
print start+width}')
# Write the LaTeX macros to use in plot. Start with the thick line
# showing tangential distance.
macros=$bdir/macros.tex
printf '\\newcommand{\\maScaleDec}'"{$scalelinedec}\n" > $macros
printf '\\newcommand{\\maScaleRAa}'"{$scalelinerastart}\n" >> $macros
printf '\\newcommand{\\maScaleRAb}'"{$scalelineraend}\n" >> $macros
printf '\\newcommand{\\maScaleKpc}'"{$scalelineinkpc}\n" >> $macros
printf '\\newcommand{\\maCenterZ}'"{$redshift}\n" >> $macros
# Add image extrema for the coordinates.
v=$(echo $coverage | awk '{print $1}')
printf '\\newcommand{\maCropRAMin}'"{$v}\n" >> $macros
v=$(echo $coverage | awk '{print $2}')
printf '\\newcommand{\maCropRAMax}'"{$v}\n" >> $macros
v=$(echo $coverage | awk '{print $3}')
printf '\\newcommand{\maCropDecMin}'"{$v}\n" >> $macros
v=$(echo $coverage | awk '{print $4}')
printf '\\newcommand{\maCropDecMax}'"{$v}\n" >> $macros
# Distance between each tick value.
v=$(echo $coverage | awk '{print ($2-$1)/'$numticks'}')
printf '\\newcommand{\maTickDist}'"{$v}\n" >> $macros
printf '\\newcommand{\maSBlow}'"{$sblow}\n" >> $macros
printf '\\newcommand{\maSBhigh}'"{$sbhigh}\n" >> $macros
# Copy the LaTeX source into the build directory and go there to run
# it and have all the temporary LaTeX files there.
cp report.tex my-figure.tex $bdir
cd $bdir
rm -f *-figure0*
pdflatex -shell-escape -halt-on-error report.tex
File: gnuastro.info, Node: Invoking astconvertt, Prev: Annotations for figure in paper, Up: ConvertType
5.2.5 Invoking ConvertType
--------------------------
ConvertType will convert any recognized input file type to any specified
output type. The executable name is ‘astconvertt’ with the following
general template
$ astconvertt [OPTION...] InputFile [InputFile2] ... [InputFile4]
One line examples:
## Convert an image in FITS to PDF:
$ astconvertt image.fits --output=pdf
## Similar to before, but use the Viridis color map:
$ astconvertt image.fits --colormap=viridis --output=pdf
## Add markers to to highlight parts of the image
## ('marks.fits' is a table containing coordinates)
$ astconvertt image.fits --marks=marks.fits --output=pdf
## Convert an image in JPEG to FITS (with multiple extensions
## if it has color):
$ astconvertt image.jpg -oimage.fits
## Use three 2D arrays to create an RGB JPEG output (two are
## plain-text, the third is FITS, but all have the same size).
$ astconvertt f1.txt f2.txt f3.fits -o.jpg
## Use two images and one blank for an RGB EPS output:
$ astconvertt M31_r.fits M31_g.fits blank -oeps
## Directly pass input from output of another program through Standard
## input (not a file).
$ cat 2darray.txt | astconvertt -oimg.fits
In the sub-sections below various options that are specific to
ConvertType are grouped in different categories. Please see those
sections for a detailed discussion on each group and its options.
Besides those, ConvertType also shares the *note Common options:: with
other Gnuastro programs. The common options are not repeated here.
* Menu:
* ConvertType input and output:: Input/output file names and formats.
* Pixel visualization:: Visualizing the pixels in the output.
* Drawing with vector graphics:: Adding marks in many shapes and colors over the pixels.
File: gnuastro.info, Node: ConvertType input and output, Next: Pixel visualization, Prev: Invoking astconvertt, Up: Invoking astconvertt
5.2.5.1 ConvertType input and output
....................................
At most four input files (one for each color channel for formats that
allow it) are allowed in ConvertType. The first input dataset can
either be a file, or come from Standard input (see *note Standard
input:: and *note Recognized file formats::).
The order of multiple input files is important. After reading the
input file(s) the number of color channels in all the inputs will be
used to define which color space to use for the outputs and how each
color channel is interpreted: 1 (for grayscale), 3 (for RGB) and 4 (for
CMYK) input channels. For more on pixel color channels, see *note Pixel
colors::. Depending on the format of the input(s), the number of input
files can differ.
For example, if you plan to build an RGB PDF and your three channels
are in the first HDU of ‘r.fits’, ‘g.fits’ and ‘b.fits’, then you can
simply call MakeProfiles like this:
$ astconvertt r.fits g.fits b.fits -g1 --output=rgb.pdf
However, if the three color channels are in three extensions (assuming
the HDUs are respectively named ‘R’, ‘G’ and ‘B’) of a single file
(assuming ‘channels.fits’), you should run it like this:
$ astconvertt channels.fits -hR -hG -hB --output=rgb.pdf
On the other hand, if the channels are already in a multi-channel format
(like JPEG), you can simply provide that file:
$ astconvertt image.jpg --output=rgb.pdf
If multiple channels are given as input, and the output format does not
support multiple color channels (for example, FITS), ConvertType will
put the channels in different HDUs, like the example below. After
running the ‘astfits’ command, if your JPEG file was not grayscale
(single channel), you will see multiple HDUs in ‘channels.fits’.
$ astconvertt image.jpg --output=channels.fits
$ astfits channels.fits
As shown above, the output's file format will be interpreted from the
name given to the ‘--output’ option (as a common option to all Gnuastro
programs, for the description of ‘--output’, see *note Input output
options::). It can either be given on the command-line or in any of the
configuration files (see *note Configuration files::). When the output
suffix is not recognized, it will default to plain text format, see
*note Recognized file formats::.
If there is one input dataset (color channel) the output will be
gray-scale. When three input datasets (color channels) are given, they
are respectively considered to be the red, green and blue color
channels. Finally, if there are four color channels they will be cyan,
magenta, yellow and black (CMYK colors).
The value to ‘--output’ (or ‘-o’) can be either a full file name or
just the suffix of the desired output format. In the former case (full
name), it will be directly used for the output's file name. In the
latter case, the name of the output file will be set based on the
automatic output guidelines, see *note Automatic output::. Note that
the suffix name can optionally start with a ‘.’ (dot), so for example,
‘--output=.jpg’ and ‘--output=jpg’ are equivalent. See *note Recognized
file formats::.
The relevant options for input/output formats are described below:
‘-h STR/INT’
‘--hdu=STR/INT’
Input HDU name or counter (counting from 0) for each input FITS
file. If the same HDU should be used from all the FITS files, you
can use the ‘--globalhdu’ option described below. In ConvertType,
it is possible to call the HDU option multiple times for the
different input FITS or TIFF files in the same order that they are
called on the command-line. Note that in the TIFF standard, one
'directory' (similar to a FITS HDU) may contain multiple color
channels (for example, when the image is in RGB).
Except for the fact that multiple calls are possible, this option
is identical to the common ‘--hdu’ in *note Input output options::.
The number of calls to this option cannot be less than the number
of input FITS or TIFF files, but if there are more, the extra HDUs
will be ignored, note that they will be read in the order described
in *note Configuration file precedence::.
Unlike CFITSIO, libtiff (which is used to read TIFF files) only
recognizes numbers (counting from zero, similar to CFITSIO) for
'directory' identification. Hence the concept of names is not
defined for the directories and the values to this option for TIFF
files must be numbers.
‘-g STR/INT’
‘--globalhdu=STR/INT’
Use the value given to this option (a HDU name or a counter,
starting from 0) for the HDU identifier of all the input FITS
files. This is useful when all the inputs are distributed in
different files, but have the same HDU in those files.
‘-w FLT’
‘--widthincm=FLT’
The width of the output in centimeters. This is only relevant for
those formats that accept such a width as metadata (not FITS or
plain-text for example), see *note Recognized file formats::. For
most digital purposes, the number of pixels is far more important
than the value to this parameter because you can adjust the
absolute width (in inches or centimeters) in your document
preparation program.
‘-x’
‘--hex’
Use Hexadecimal encoding in creating EPS output. By default the
ASCII85 encoding is used which provides a much better compression
ratio. When converted to PDF (or included in TeX or LaTeX which is
finally saved as a PDF file), an efficient binary encoding is used
which is far more efficient than both of them. The choice of EPS
encoding will thus have no effect on the final PDF.
So if you want to transfer your EPS files (for example, if you want
to submit your paper to arXiv or journals in PostScript), their
storage might become important if you have large images or lots of
small ones. By default ASCII85 encoding is used which offers a
much better compression ratio (nearly 40 percent) compared to
Hexadecimal encoding.
‘-u INT’
‘--quality=INT’
The quality (compression) of the output JPEG file with values from
0 to 100 (inclusive). For other formats the value to this option
is ignored. Note that only in gray-scale (when one input color
channel is given) will this actually be the exact quality (each
pixel will correspond to one input value). If it is in color mode,
some degradation will occur. While the JPEG standard does support
loss-less graphics, it is not commonly supported.
File: gnuastro.info, Node: Pixel visualization, Next: Drawing with vector graphics, Prev: ConvertType input and output, Up: Invoking astconvertt
5.2.5.2 Pixel visualization
...........................
The main goal of ConvertType is to visualize pixels to/from print or web
friendly formats.
Astronomical data usually have a very large dynamic range (difference
between maximum and minimum value) and different subjects might be
better demonstrated with a limited flux range.
‘--colormap=STR[,FLT,...]’
The color map to visualize a single channel. The first value given
to this option is the name of the color map, which is shown below.
Some color maps can be configured. In this case, the configuration
parameters are optionally given as numbers following the name of
the color map for example, see ‘hsv’. The table below contains the
usable names of the color maps that are currently supported:
‘gray’
‘grey’
Grayscale color map. This color map does not have any
parameters. The full dataset range will be scaled to 0 and
$2^8-1=255$ to be stored in the requested format.
‘hsv’
Hue, Saturation, Value(1) color map. If no values are given
after the name (‘--colormap=hsv’), the dataset will be scaled
to 0 and 360 for hue covering the full spectrum of colors.
However, you can limit the range of hue (to show only a
special color range) by explicitly requesting them after the
name (for example, ‘--colormap=hsv,20,240’).
The mapping of a single-channel dataset to HSV is done through
the Hue and Value elements: Lower dataset elements have lower
"value" _and_ lower "hue". This creates darker colors for
fainter parts, while also respecting the range of colors.
‘viridis’
Viridis is the default colormap of the popular Matplotlib
module of Python and available in many other visualization
tools like PGFPlots.
‘sls’
The SLS color range, taken from the commonly used SAO DS9
(http://ds9.si.edu). The advantage of this color range is
that it starts with black, going into dark blue and finishes
with the brighter colors of red and white. So unlike the HSV
color range, it includes black and white and brighter colors
(like yellow, red) show the larger values.
‘sls-inverse’
The inverse of the SLS color map (see above), where the lowest
value corresponds to white and the highest value is black.
While SLS is good for visualizing on the monitor, SLS-inverse
is good for printing.
‘--rgbtohsv’
When there are three input channels and the output is in the FITS
format, interpret the three input channels as red, green and blue
channels (RGB) and convert them to the hue, saturation, value (HSV)
color space.
The currently supported output formats of ConvertType do not have
native support for HSV. Therefore this option is only supported
when the output is in FITS format and each of the hue, saturation
and value arrays can be saved as one FITS extension in the output
for further analysis (for example, to select a certain color).
‘-c STR’
‘--change=STR’
(‘=STR’) Change pixel values with the following format ‘"from1:to1,
from2:to2,..."’. This option is very useful in displaying labeled
pixels (not actual data images which have noise) like segmentation
maps. In labeled images, usually a group of pixels have a fixed
integer value. With this option, you can manipulate the labels
before the image is displayed to get a better output for print or
to emphasize on a particular set of labels and ignore the rest.
The labels in the images will be changed in the same order given.
By default first the pixel values will be converted then the pixel
values will be truncated (see ‘--fluxlow’ and ‘--fluxhigh’).
You can use any number for the values irrespective of your final
output, your given values are stored and used in the double
precision floating point format. So for example, if your input
image has labels from 1 to 20000 and you only want to display those
with labels 957 and 11342 then you can run ConvertType with these
options:
$ astconvertt --change=957:50000,11342:50001 --fluxlow=5e4 \
--fluxhigh=1e5 segmentationmap.fits --output=jpg
While the output JPEG format is only 8 bit, this operation is done
in an intermediate step which is stored in double precision
floating point. The pixel values are converted to 8-bit after all
operations on the input fluxes have been complete. By placing the
value in double quotes you can use as many spaces as you like for
better readability.
‘-C’
‘--changeaftertrunc’
Change pixel values (with ‘--change’) after truncation of the flux
values, by default it is the opposite.
‘-L FLT’
‘--fluxlow=FLT’
The minimum flux (pixel value) to display in the output image, any
pixel value below this value will be set to this value in the
output. If the value to this option is the same as ‘--fluxhigh’,
then no flux truncation will be applied. Note that when multiple
channels are given, this value is used for all the color channels.
‘-H FLT’
‘--fluxhigh=FLT’
The maximum flux (pixel value) to display in the output image, see
‘--fluxlow’.
‘-m INT’
‘--maxbyte=INT’
This is only used for the JPEG and EPS output formats which have an
8-bit space for each channel of each pixel. The maximum value in
each pixel can therefore be $2^8-1=255$. With this option you can
change (decrease) the maximum value. By doing so you will decrease
the dynamic range. It can be useful if you plan to use those
values for other purposes.
‘-A’
‘--forcemin’
Enforce the value of ‘--fluxlow’ (when it is given), even if it is
smaller than the minimum of the dataset and the output is format
supporting color. This is particularly useful when you are
converting a number of images to a common image format like JPEG or
PDF with a single command and want them all to have the same range
of colors, independent of the contents of the dataset. Note that
if the minimum value is smaller than ‘--fluxlow’, then this option
is redundant.
By default, when the dataset only has two values, _and_ the output
format is PDF or EPS, ConvertType will use the PostScript
optimization that allows setting the pixel values per bit, not byte
(*note Recognized file formats::). This can greatly help reduce
the file size. However, when ‘--fluxlow’ or ‘--fluxhigh’ are
called, this optimization is disabled: even though there are only
two values (is binary), the difference between them does not
correspond to the full contrast of black and white.
‘-B’
‘--forcemax’
Similar to ‘--forcemin’, but for the maximum.
‘-i’
‘--invert’
For 8-bit output types (JPEG, EPS, and PDF for example) the final
value that is stored is inverted so white becomes black and vice
versa. The reason for this is that astronomical images usually
have a very large area of blank sky in them. The result will be
that a large are of the image will be black. Note that this
behavior is ideal for gray-scale images, if you want a color image,
the colors are going to be mixed up.
---------- Footnotes ----------
(1) <https://en.wikipedia.org/wiki/HSL_and_HSV>
File: gnuastro.info, Node: Drawing with vector graphics, Prev: Pixel visualization, Up: Invoking astconvertt
5.2.5.3 Drawing with vector graphics
....................................
With the options described in this section, you can draw marks over your
to-be-published images (for example, in PDF). Each mark can be highly
customized so they can have different shapes, colors, line widths, text,
text size, etc. The properties of the marks should be stored in a table
that is given to the ‘--marks’ option described below. A fully working
demo on adding marks is provided in *note Marking objects for
publication::.
An important factor to consider when drawing vector graphics is that
vector graphics standards (the PostScript standard in this case) use a
"point" as the primary unit of line thickness or font size. Such that
72 points correspond to 1 inch (or 2.54 centimeters). In other words,
there are roughly 3 PostScript points in every millimeter. On the other
hand, the pixels of the images you plan to show as the background do not
have any real size! Pixels are abstract and can be associated with any
print-size.
In ConvertType, the print-size of your final image is set with the
‘--widthincm’ option (see *note ConvertType input and output::). The
value to ‘--widthincm’ is the to-be width of the image in centimeters.
It therefore defines the thickness of lines or font sizes for your
vector graphics features (like the image border or marks). Just recall
that we are not talking about resolution! Vector graphics have infinite
resolution! We are talking about the relative thickness of the lines
(or font sizes) in relation to the pixels in your background image.
‘-b INT’
‘--borderwidth=INT’
The width of the border to be put around the EPS and PDF outputs in
units of PostScript points. If you are planning on adding a
border, its thickness in relation to your image pixel sizes is
highly correlated with the value you give to the ‘--widthincm’
parameter. See the description at the start of this section for
more.
Unfortunately in the document structuring convention of the
PostScript language, the "bounding box" has to be in units of
PostScript points with no fractions allowed. So the border values
only have to be specified in integers. To have a final border that
is thinner than one PostScript point in your document, you can ask
for a larger width in ConvertType and then scale down the output
EPS or PDF file in your document preparation program. For example,
by setting ‘width’ in your ‘includegraphics’ command in TeX or
LaTeX to be larger than the value to ‘--widthincm’. Since it is
vector graphics, the changes of size have no effect on the quality
of your output (pixels do not get different values).
‘--bordercolor=STR’
The name of the color to use for border that will be put around the
EPS and PDF outputs. The list of available colors, along with
their name and an example can be seen with the following command
(also see *note Vector graphics colors::):
$ astconvertt --listcolors
This option only accepts the name of the color, not the numeric
identifier.
‘--marks=STR’
Draw vector graphics (infinite resolution) marks over the image.
The value to this option should be the file name of a table
containing the mark information. The table given to this option
can have various properties for each mark in each column. You can
specify which column contains which property of the marks using the
options below that start with ‘--mark’. Only two property columns
are mandatory (‘--markcoords’), the rest are optional.
The table can be in any of the Gnuastro's *note Recognized table
formats::. For more on the difference between vector and raster
graphics, see *note Raster and Vector graphics::. For example, if
your table with mark information is called ‘my-marks.fits’, you can
use the command below to draw red circles of radius 5 pixels over
the coordinates.
$ astconvertt image.fits --output=image.pdf \
--marks=marks.fits --mode=wcs \
--markcoords=RA,DEC
You can highly customize each mark with different columns in
‘marks.fits’ using the ‘--mark*’ options below (for example, using
different colors, different shapes, different sizes, text, and the
rest on each mark).
‘--markshdu=STR/INT’
The HDU (or extension) name or number of the table containing mark
properties (file given to ‘--marks’). This is only relevant if the
table is in the FITS format and there is more than one HDU in the
FITS file.
‘-r STR,STR’
‘--markcoords=STR,STR’
The column names (or numbers) containing the coordinates of each
mark (in table given to ‘--marks’). Only two values should be
given to this option (one for each coordinate). They can either be
given to one call (‘--markcoords=RA,DEC’) or in separate calls
(‘--markcoords=RA --markcoords=DEC’).
When ‘--mode=image’ the columns will be associated to the
horizontal/vertical coordinates of the image, and interpreted in
units of pixels. In ‘--mode=wcs’, the columns will be associated
to the WCS coordinates (typically Right Ascension and Declination,
in units of degrees).
‘-O STR’
‘--mode=STR’
The coordinate mode for interpreting the values in the columns
given to the ‘--markcoord1’ and ‘--markcoord2’ options. The
acceptable values are either ‘img’ (for image or pixel
coordinates), and ‘wcs’ for World Coordinate System (typically RA
and Dec). For the WCS-mode, the input image should have the
necessary WCS keywords, otherwise ConvertType will crash.
‘--markshape=STR/INT’
The column name(s), or number(s), containing the shapes of each
mark (in table given to ‘--marks’). The shapes can either be
identified by their name, or their numerical identifier. If
identifying them by name in a plain-text table, you need to define
a string column (see *note Gnuastro text table format::). The full
list of names is shown below, with their numerical identifier in
parenthesis afterwards. For each shape, you can also specify
properties such as the size, line width, rotation, and color. See
the description of the relevant ‘--mark*’ option below.
‘circle (1)’
A circular circumference. It's _radius_ is defined by a
single size element (the first column given to ‘--marksize’).
Any value in the second size column (if given for other shapes
in the same call) are ignored by this shape.
‘plus (2)’
The plus sign ($+$). The _length of its lines_ is defined by
a single size element (the first column given to
‘--marksize’). Such that the intersection of its lines is on
the central coordinate of the mark. Any value in the second
size column (if given for other shapes in the same call) are
ignored by this shape.
‘cross (3)’
A multiplication sign ($\times$). The _length of its lines_
is defined by a single size element (the first column given to
‘--marksize’). Such that the intersection of its lines is on
the central coordinate of the mark. Any value in the second
size column (if given for other shapes in the same call) are
ignored by this shape.
‘ellipse (4)’
An elliptical circumference. Its major axis radius is defined
by the first size element (first column given to
‘--marksize’), and its axis ratio is defined through the
second size element (second column given to ‘--marksize’).
‘point (5)’
A point (or a filled circle). Its _radius_ is defined by a
single size element (the first column given to ‘--marksize’).
Any value in the second size column (if given for other shapes
in the same call) are ignored by this shape.
This filled circle mark is defined as a "point" because it is
usually relevant as a small size (or point in the whole
image). But there is no limit on its size, so it can be
arbitrarily large.
‘square (6)’
A square circumference. Its _edge length_ is defined by a
single size element (the first column given to ‘--marksize’).
Any value in the second size column (if given for other shapes
in the same call) are ignored by this shape.
‘rectangle (7)’
A rectangular circumference. Its length along the horizontal
image axis is defined by first size element (first column
given to ‘--marksize’), and its length along the vertical
image axis is defined through the second size element (second
column given to ‘--marksize’).
‘line (8)’
A line. The line's _length_ is defined by a single size
element (the first column given to ‘--marksize’. The line
will be centered on the given coordinate. Like all shapes,
you can rotate the line about its center using the
‘--markrotate’ column. Any value in the second size column
(if given for other shapes in the same call) are ignored by
this shape.
‘--markrotate=STR/INT’
Column name or number that contains the mark's rotation angle. The
rotation angle should be in degrees and be relative to the
horizontal axis of the image.
‘--marksize=STR[,STR]’
The column name(s), or number(s), containing the size(s) of each
mark (in table given to ‘--marks’). All shapes need at least one
"size" parameter and some need two. For the interpretation of the
size column(s) for each shape, see the ‘--markshape’ option's
description. Since the size column(s) is (are) optional, when not
specified, default values will be used (which may be too small in
larger images, so you need to change them).
By default, the values in the size column are assumed to be in the
same units as the coordinates (defined by the ‘--mode’ option,
described above). However, when the coordinates are in WCS-mode,
some special cases may occur for the size.
• The native WCS units (usually degrees) can be too large, and
it may be more convenient for the values in the size column(s)
to be in arc-seconds. In this case, you can use the
‘--sizeinarcsec’ option.
• Similar to above, but in units of arc-minutes. In this case,
you can use the ‘--sizeinarcmin’ option.
• Your sizes may be in units of pixels, not the WCS units. In
this case, you can use the ‘--sizeinpix’ option.
‘--sizeinpix’
In WCS-mode, assume that the sizes are in units of pixels. By
default, when in WCS-mode, the sizes are assumed to be in the units
of the WCS coordinates (usually degrees).
‘--sizeinarcsec’
In WCS-mode, assume that the sizes are in units of arc-seconds. By
default, when in WCS-mode, the sizes are assumed to be in the units
of the WCS coordinates (usually degrees).
‘--sizeinarcmin’
In WCS-mode, assume that the sizes are in units of arc-seconds. By
default, when in WCS-mode, the sizes are assumed to be in the units
of the WCS coordinates (usually degrees).
‘--marklinewidth=STR/INT’
Column containing the width (thickness) of the line to draw each
mark. The line width is measured in units of "points" (where 72
points is one inch), and it can be any positive floating point
number. Therefore, the thickness (in relation to the pixels of
your image) depends on ‘--widthincm’ option. For more, see the
description at the start of this section.
‘--markcolor=STR/INT’
Column containing the color of the mark. This column can be either
a string or an integer. As a string, the color name can be written
directly in your table (this greatly helps in human readability).
For more on string columns see *note Gnuastro text table format::.
As an integer, you can simply use the numerical identifier of the
column. You can see the list of colors with their names and
numerical identifiers in Gnuastro by running ConvertType with
‘--listcolors’, or see *note Vector graphics colors::.
‘--listcolors’
The list of acceptable color names, their codes and their
representation can be seen with the ‘--listcolors’ option. By
"representation" we mean that the color will be shown on the
terminal as the background in that column. But this will only be
properly visible with "true color" or 24-bit terminals, see ANSI
escape sequence standard
(https://en.wikipedia.org/wiki/ANSI_escape_code). Most modern
GNU/Linux terminals support 24-bit colors natively, and no
modification is necessary. For macOS, see the box below.
The printed text in standard output is in the *note Gnuastro text
table format::, so if you want to store this table, you can simply
pipe the output to Gnuastro's Table program and store it as a FITS
table:
$ astconvertt --listcolors | astttable -ocolors.fits
*macOS terminal colors*: as of August 2022, the default macOS
terminal (iTerm) does not support 24-bit colors! The output of
‘--listlines’ therefore does not display the actual colors (you can
only use the color names). One tested solution is to install and
use iTerm2 (https://iterm2.com), which is free software and
available in Homebrew (https://formulae.brew.sh/cask/iterm2).
iTerm2 is described as a successor for iTerm and works on macOS
10.14 (released in September 2018) or newer.
‘--marktext=STR/INT’
Column name or number that contains the text that should be printed
under the mark. If the column is numeric, the number will be
printed under the mark (for example, if you want to write the
magnitude or redshift of the object under the mark showing it).
For the precision of writing floating point columns, see
‘--marktextprecision’. But if the column has a string format (for
example, the name of the object like an NGC1234), you need to
define the column as a string column (see *note Gnuastro text table
format::).
For text with different lengths, set the length in the definition
of the column to the maximum length of the strings to be printed.
If there are some rows or marks that don't require text, set the
string in this column to ‘n/a’ (not applicable; the blank value for
strings in Gnuastro). When having strings with different lengths,
make sure to have enough white spaces (for the shorter strings) so
the adjacent columns are not taken as part of the string (see *note
Gnuastro text table format::).
‘--marktextprecision=INT’
The number of decimal digits to print after the floating point.
This is only relevant when ‘--marktext’ is given, and the selected
column has a floating point format.
‘--markfont=STR/INT’
Column name or number that contains the font for the displayed text
under the mark. This is only relevant if ‘--marktext’ is called.
The font should be accessible by Ghostscript.
If you are not familiar with the available fonts on your system's
Ghostscript, you can use the ‘--showfonts’ option to see all the
fonts in a custom PDF file (one page per font). If you are already
familiar with the font you want, but just want to make sure about
its presence (or spelling!), you can get a list (on standard
output) of all the available fonts with the ‘--listfonts’ option.
Both are described below.
It is possible to add custom fonts to Ghostscript as described in
the Fonts section (https://ghostscript.com/doc/current/Fonts.htm)
of the Ghostscript manual.
‘--markfontsize=STR/INT’
Column name or number that contains the font size to use. This is
only relevant if a text column has been defined (with ‘--marktext’,
described above). The font size is in units of "point"s, see
description at the start of this section for more.
‘--showfonts’
Create a special PDF file that shows the name and shape of all
available fonts in your system's Ghostscript. You can use this for
selecting the best font to put in the ‘--markfonts’ column. The
available fonts can differ from one system to another (depending on
how Ghostscript was configured in that system). The PDF file's
name is constructed by appending a ‘-fonts.pdf’ to the file name
given to the ‘--output’ option.
The PDF file will have one page for each font, and the sizes of the
pages are customized for showing the fonts (each page is
horizontally elongated). This helps to better check the files by
disable "continuous" mode in your PDF viewer, and setting the zoom
such that the width of the page corresponds to the width of your
PDF viewer. Simply pressing the left/right keys will then nicely
show each fonts separately.
‘--listfonts’
Print (to standard output) the names of all available fonts in
Ghostscript that you can use for the ‘--markfonts’ column. The
available fonts can differ from one system to another (depending on
how Ghostscript was configured in that system). If you are not
already familiar with the shape of each font, please use
‘--showfonts’ (described above).
File: gnuastro.info, Node: Table, Next: Query, Prev: ConvertType, Up: Data containers
5.3 Table
=========
Tables are the high-level products of processing on low-leveler data
like images or spectra. For example, in Gnuastro, MakeCatalog will
process the pixels over an object and produce a catalog (or table) with
the properties of each object such as magnitudes and positions (see
*note MakeCatalog::). Each one of these properties is a column in its
output catalog (or table) and for each input object, we have a row.
When there are only a small number of objects (rows) and not too many
properties (columns), then a simple plain text file is mainly enough to
store, transfer, or even use the produced data. However, to be more
efficient, astronomers have defined the FITS binary table standard to
store data in a binary format (which cannot be seen in a text editor
text). This can offer major advantages: the file size will be greatly
reduced and the reading and writing will also be faster (because the RAM
and CPU also work in binary). The acceptable table formats are fully
described in *note Tables::.
Binary tables are not easily readable with basic plain-text editors.
There is no fixed/unified standard on how the zero and ones should be
interpreted. Unix-like operating systems have flourished because of a
simple fact: communication between the various tools is based on human
readable characters(1). So while the FITS table standards are very
beneficial for the tools that recognize them, they are hard to use in
the vast majority of available software. This creates limitations for
their generic use.
Table is Gnuastro's solution to this problem. Table has a large set
of operations that you can directly do on any recognized table (such as
selecting certain rows and doing arithmetic on the columns). For
operations that Table does not do internally, FITS tables (ASCII or
binary) are directly accessible to the users of Unix-like operating
systems (in particular those working the command-line or shell, see
*note Command-line interface::). With Table, a FITS table (in binary or
ASCII formats) is only one command away from AWK (or any other tool you
want to use). Just like a plain text file that you read with the ‘cat’
command. You can pipe the output of Table into any other tool for
higher-level processing, see the examples in *note Invoking asttable::
for some simple examples.
In the sections below we describe how to effectively use the Table
program. We start with *note Column arithmetic::, where the basic
concept and methods of applying arithmetic operations on one or more
columns are discussed. Afterwards, in *note Operation precedence in
Table::, we review the various types of operations available and their
precedence in an instance of calling Table. This is a good place to get
a general feeling of all the things you can do with Table. Finally, in
*note Invoking asttable::, we give some examples and describe each
option in Table.
* Menu:
* Printing floating point numbers:: Optimal storage of floating point types.
* Vector columns:: How to keep more than one value in each column.
* Column arithmetic:: How to do operations on table columns.
* Operation precedence in Table:: Order of running options in Table.
* Invoking asttable:: Options and arguments to Table.
---------- Footnotes ----------
(1) In "The art of Unix programming", Eric Raymond makes this
suggestion to programmers: "When you feel the urge to design a complex
binary file format, or a complex binary application protocol, it is
generally wise to lie down until the feeling passes.". This is a great
book and strongly recommended, give it a look if you want to truly enjoy
your work/life in this environment.
File: gnuastro.info, Node: Printing floating point numbers, Next: Vector columns, Prev: Table, Up: Table
5.3.1 Printing floating point numbers
-------------------------------------
Many of the columns containing astronomical data will contain floating
point numbers (those that aren't an integer, like $1.23$ or
$4.56\times10^{-7}$). However, printing (for human readability) of
floating point numbers has some intricacies that we will explain in this
section. For a basic introduction to different types of integers or
floating points, see *note Numeric data types::.
It may be tempting to simply use 64-bit floating points all the time
and avoid this section over all. But have in mind that compared to
32-bit floating point type, a 64-bit floating point type will consume
double the storage, double the RAM and will take almost double the time
for processing. So when the statistical precision of your numbers is
less than that offered by 32-bit floating point precision, it is much
better to store them in this format.
Within almost all commonly used CPUs of today, numbers (including
integers or floating points) are stored in binary base-2 format (where
the only digits that can be used to represent the number are 0 and 1).
However, we (humans) are use to numbers in base-10 (where we have 10
digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9). For integers, there is a
one-to-one correspondence between a base-2 and base-10 representation.
Therefore, converting a base-10 integer (that you will be giving as an
option value when running a Gnuastro program, for example) to base-2
(that the computer will store in memory), or vice-versa, will not cause
any loss of information for integers.
The problem is that floating point numbers don't have such a
one-to-one correspondence between the two notations. The full
discussion on how floating point numbers are stored in binary format is
beyond the scope of this book. But please have a look at the
corresponding Wikipedia article
(https://en.wikipedia.org/wiki/Floating-point_arithmetic) to get a rough
feeling about the complexity. Of course, if you are interested in the
details, that Wikipedia article should be a good starting point for
further reading.
The most common convention for storing floating point numbers in
digital storage is IEEE Standard for Floating-Point Arithmetic; IEEE 754
(https://en.wikipedia.org/wiki/IEEE_754). In short, the full width (in
bits) assigned to that type (for example the 32 bits allocated for
32-bit floating point types) is divided into separate components: The
first bit is the "sign" (specifying if the number is negative or
positive). In 32-bit floats, the next 8 bits are the "exponent" and
finally (again, in 32-bit floats), the "fraction" is stored in the next
23 bits. For example see this image on Wikipedia
(https://commons.wikimedia.org/wiki/File:Float_example.svg).
In IEEE 754, around zero, the base-2 and base-10 representations
approximately match. However, as we go away from 0, you will loose
precision. The important concept in understanding the precision of
floating point numbers is "decimal digits", or the number of digits in
the number, independent of where the decimal point is. For example
$1.23$ has three decimal digits and $4.5678\times10^9$ has 5 decimal
digits. According to IEEE 754(1), 32-bit and 64-bit floating point
numbers can accurately (statistically) represent a floating point with
7.22 and 15.95 decimal digits respectively.
*Should I store my columns as 32-bit or 64-bit floating point type?* If
your floating point numbers have 7 decimal digits or less (for example
noisy image pixel values, measured star or galaxy magnitudes, and
anything that is derived from them like galaxy mass and etc), you can
safely use 32-bit precision (the statistical error on the measurements
is usually significantly larger than 7 digits!). However, some columns
require more digits; thus 64-bit precision. For example, RA or Dec with
more than one arcsecond accuracy: the degrees can have 3 digits, and 1
arcsecond is $1/3600\sim0.0003$ of a degree, requiring 4 more digits).
You can use the *note Numerical type conversion operators:: of *note
Column arithmetic:: to convert your columns to a certain type for
storage.
The discussion above was for the storage of floating point numbers.
When printing floating point numbers in a human-friendly format (for
example, in a plain-text file or on standard output in the
command-line), the computer has to convert its internal base-2
representation to a base-10 representation. This second conversion may
cause a small discrepancy between the stored and printed values.
*Use FITS tables as output of measurement programs:* When you are doing
a measurement to produce a catalog (for example with *note
MakeCatalog::) set the output to be a FITS table (for example
‘--output=mycatalog.fits’). A FITS binary table will store the same the
base-2 number that was measured by the CPU. However, if you choose to
store the output table as a plain-text table, you risk loosing
information due to the human friendly base-10 floating point conversion
(which is necessary in a plain-text output).
To customize how columns containing floating point values are printed
(in a plain-text output file, or in the standard output in your
terminal), Table has four options for the two different types:
‘--txtf32format’, ‘--txtf32precision’, ‘--txtf64format’ and
‘--txtf64precision’. They are fully described in *note Invoking
asttable::.
*Summary:* it is therefore recommended to always store your tables as
FITS (binary) tables. To view the contents of the table on the
command-line or to feed it to a program that doesn't recognize FITS
tables, you can use the four options above for a custom base-10
conversion that will not cause any loss of data.
---------- Footnotes ----------
(1)
<https://en.wikipedia.org/wiki/IEEE_754#Basic_and_interchange_formats>
File: gnuastro.info, Node: Vector columns, Next: Column arithmetic, Prev: Printing floating point numbers, Up: Table
5.3.2 Vector columns
--------------------
In its most common format, each column of a table only has a single
value in each row. For example, we usually have one column for the
magnitude, another column for the RA (Right Ascension) and yet another
column for the DEC (Declination) of a set of galaxies/stars (where each
galaxy is represented by one row in the table). This common
single-valued column format is sufficient in many scenarios. However,
in some situations (like those below) it would help to have multiple
values for each row in each column, not just one.
• Conceptually: the various numbers are "connected" to each other.
In other words, their order and position in relation to each other
matters. Common examples in astronomy are the radial profiles of
each galaxy in your catalog, or their spectrum. For example, each
MUSE(1) spectra has 3681 points (with a sampling of of 1.25
Angstroms).
Dealing with this many separate measurements as separate columns in
your table is very annoying and prone to error: you don't want to
forget moving some of them in an output table for further analysis,
mistakenly change their order, or do some operation only on a
sub-set of them.
• Technically: in the FITS standard, you can only store a maximum of
999 columns in a FITS table. Therefore, if you have more than 999
data points for each galaxy (like the MUSE spectra example above),
it is impossible to store each point in one table as separate
columns.
To address these problems, the FITS standard has defined the concept
of "vector" columns in its Binary table format (ASCII FITS tables don't
support vector columns, but Gnuastro's plain-text format does, as
described here). Within each row of a single vector column, we can
store any number of data points (like the MUSE spectra above or the full
radial profile of each galaxy). All the values in a vector column have
to have the same *note Numeric data types::, and the number of elements
within each vector column is the same for all rows.
By grouping conceptually similar data points (like a spectrum) in one
vector column, we can significantly reduce the number of columns and
make it much more manageable, without loosing any information! To
demonstrate the vector column features of Gnuastro's Table program,
let's start with a randomly generated small (5 rows and 3 columns)
catalog. This will allows us to show the outputs of each step here, but
you can apply the same concept to vectors with any number of columns.
With the command below, we use ‘seq’ to generate a single-column
table that is piped to Gnuastro's Table program. Table then uses column
arithmetic to generate three columns with random values from that column
(for more, see *note Column arithmetic::). Each column becomes noisy,
with standard deviations of 2, 5 and 10. Finally, we will add metadata
to each column, giving each a different name (using names is always the
best way to work with columns):
$ seq 1 5 \
| asttable -c'arith $1 2 mknoise-sigma f32' \
-c'arith $1 5 mknoise-sigma f32' \
-c'arith $1 10 mknoise-sigma f32' \
--colmetadata=1,abc,none,"First column." \
--colmetadata=2,def,none,"Second column." \
--colmetadata=3,ghi,none,"Third column." \
--output=table.fits
With the command below, let's have a look at the table. When you run
it, you will have a different random number generator seed, so the
numbers will be slightly different. For making reproducible random
numbers, see *note Generating random numbers::. The ‘-Y’ option is used
for more easily readable numbers (without it, floating point numbers are
written in scientific notation, for more see *note Printing floating
point numbers::) and with the ‘-O’ we are asking Table to also print the
metadata. For more on Table's options, see *note Invoking asttable::
and for seeing how the short options can be merged (such that ‘-Y -O’ is
identical to ‘-YO’), see *note Options::.
$ asttable table.fits -YO
# Column 1: abc [none,f32,] First column.
# Column 2: def [none,f32,] Second column.
# Column 3: ghi [none,f32,] Third column.
1.074 5.535 -4.464
0.606 -2.011 15.397
1.475 1.811 5.687
2.248 7.663 -7.789
6.355 17.374 6.767
We see that indeed, it has three columns, with our given names. Now,
let's assume that you want to make a two-element vector column from the
values in the ‘def’ and ‘ghi’ columns. To do that, you can use the
‘--tovector’ option like below. As the name suggests, ‘--tovector’ will
merge the rows of the two columns into one vector column with multiple
values in each row.
$ asttable table.fits -YO --tovector=def,ghi
# Column 1: abc [none,f32 ,] First column.
# Column 2: def-VECTOR [none,f32(2),] Vector by merging multiple cols.
1.074 5.535 -4.464
0.606 -2.011 15.397
1.475 1.811 5.687
2.248 7.663 -7.789
6.355 17.374 6.767
If you ignore the metadata, this doesn't seem to have changed
anything! You see that each line of numbers still has three "tokens"
(to distinguish them from "columns"). But once you look at the
metadata, you only see metadata for two columns, not three. If you look
closely, the numeric data type of the newly added fourth column is
'‘f32(2)’' (look above; previously it was ‘f32’). The ‘(2)’ shows that
the second column contains two numbers/tokens not one. If your vector
column consisted of 3681 numbers, this would be ‘f32(3681)’. Looking
again at the metadata, we see that ‘--tovector’ has also created a new
name and comments for the new column. This is done all the time to
avoid confusion with the old columns.
Let's confirm that the newly added column is indeed a single column
but with two values. To do this, with the command below, we'll write
the output into a FITS table. In the same command, let's also give a
more suitable name for the new merged/vector column). We can get a
first confirmation by looking at the table's metadata in the second
command below:
$ asttable table.fits -YO --tovector=def,ghi --output=vec.fits \
--colmetadata=2,vector,nounits,"New vector column."
$ asttable vec.fits -i
--------
vec.fits (hdu: 1)
------- ----- ---- -------
No.Name Units Type Comment
------- ----- ---- -------
1 abc none float32 First column.
2 vector nounits float32(2) New vector column.
--------
Number of rows: 5
--------
A more robust confirmation would be to print the values in the newly
added ‘vector’ column. As expected, asking for a single column with
‘--column’ (or ‘-c’) will given us two numbers per row/line (instead of
one!).
$ asttable vec.fits -c vector -YO
# Column 1: vector [nounits,f32(2),] New vector column.
5.535 -4.464
-2.011 15.397
1.811 5.687
7.663 -7.789
17.374 6.767
If you want to keep the original single-valued columns that went into
the vector column, you can use the ‘--keepvectfin’ option (read it as
"KEEP VECtor To/From Inputs"):
$ asttable table.fits -YO --tovector=def,ghi --keepvectfin \
--colmetadata=4,vector,nounits,"New vector column."
# Column 1: abc [none ,f32 ,] First column.
# Column 2: def [none ,f32 ,] Second column.
# Column 3: ghi [none ,f32 ,] Third column.
# Column 4: vector [nounits,f32(2),] New vector column.
1.074 5.535 -4.464 5.535 -4.464
0.606 -2.011 15.397 -2.011 15.397
1.475 1.811 5.687 1.811 5.687
2.248 7.663 -7.789 7.663 -7.789
6.355 17.374 6.767 17.374 6.767
Now that you know how to create vector columns, let's assume you have
the inverse scenario: you want to extract one of the values of a vector
column into a separate single-valued column. To do this, you can use
the ‘--fromvector’ option. The ‘--fromvector’ option takes the name (or
counter) of a vector column, followed by any number of integer counters
(counting from 1). It will extract those elements into separate
single-valued columns. For example, let's assume you want to extract
the second element of the ‘defghi’ column in the file you made before:
$ asttable vec.fits --fromvector=vector,2 -YO
# Column 1: abc [none ,f32,] First column.
# Column 2: vector-2 [nounits,f32,] New vector column.
1.074 -4.464
0.606 15.397
1.475 5.687
2.248 -7.789
6.355 6.767
Just like the case with ‘--tovector’ above, if you want to keep the
input vector column, use ‘--keepvectfin’. This feature is useful in
scenarios where you want to select some rows based on a single element
(or multiple) of the vector column.
*Vector columns and FITS ASCII tables:* As mentioned above, the FITS
standard only recognizes vector columns in its Binary table format (the
default FITS table format in Gnuastro). You can still use the
‘--tableformat=fits-ascii’ option to write your tables in the FITS ASCII
format (see *note Input output options::). In this case, if a vector
column is present, it will be written as separate single-element columns
to avoid loosing information (as if you run called ‘--fromvector’ on all
the elements of the vector column). A warning is printed if this
occurs.
For an application of the vector column concepts introduced here on
MUSE data, see the 3D data cube tutorial and in particular these two
sections: *note 3D measurements and spectra:: and *note Extracting a
single spectrum and plotting it::.
---------- Footnotes ----------
(1)
<https://www.eso.org/sci/facilities/develop/instruments/muse.html>
File: gnuastro.info, Node: Column arithmetic, Next: Operation precedence in Table, Prev: Vector columns, Up: Table
5.3.3 Column arithmetic
-----------------------
In many scenarios, you want to apply some kind of operation on the
columns and save them in another table or feed them into another
program. With Table you can do a rich set of operations on the contents
of one or more columns in a table, and save the resulting values as new
column(s) in the output table. For seeing the precedence of Column
arithmetic in relation to other Table operators, see *note Operation
precedence in Table::.
To enable column arithmetic, the first 6 characters of the value to
‘--column’ (‘-c’) should be the activation word '‘arith ’' (note the
space character in the end, after '‘arith’'). After the activation
word, you can use reverse polish notation to identify the operators and
their operands, see *note Reverse polish notation::. Just note that
white-space characters are used between the tokens of the arithmetic
expression and that they are meaningful to the command-line environment.
Therefore the whole expression (including the activation word) has to be
quoted on the command-line or in a shell script (see the examples
below).
To identify a column you can directly use its name, or specify its
number (counting from one, see *note Selecting table columns::). When
you are giving a column number, it is necessary to prefix the number
with a ‘$’, similar to AWK. Otherwise the number is not distinguishable
from a constant number to use in the arithmetic operation.
For example, with the command below, the first two columns of
‘table.fits’ will be printed along with a third column that is the
result of multiplying the first column with $10^{10}$ (for example, to
convert wavelength from Meters to Angstroms). Note that without the
'<$>', it is not possible to distinguish between "1" as a
column-counter, or "1" as a constant number to use in the arithmetic
operation. Also note that because of the significance of <$> for the
command-line environment, the single-quotes are the recommended quoting
method (as in an AWK expression), not double-quotes (for the
significance of using single quotes see the box below).
$ asttable table.fits -c1,2 -c'arith $1 1e10 x'
*Single quotes when string contains <$>*: On the command-line, or in
shell-scripts, <$> is used to expand variables, for example, ‘echo
$PATH’ prints the value (a string of characters) in the variable ‘PATH’,
it will not simply print ‘$PATH’. This operation is also permitted
within double quotes, so ‘echo "$PATH"’ will produce the same output.
This is good when printing values, for example, in the command below,
‘$PATH’ will expand to the value within it.
$ echo "My path is: $PATH"
If you actually want to return the literal string ‘$PATH’, not the
value in the ‘PATH’ variable (like the scenario here in column
arithmetic), you should put it in single quotes like below. The printed
value here will include the ‘$’, please try it to see for yourself and
compare to above.
$ echo 'My path is: $PATH'
Therefore, when your column arithmetic involves the <$> sign (to
specify columns by number), quote your ‘arith ’ string with a single
quotation mark. Otherwise you can use both single or double quotes.
*Manipulate all columns in one call using <$_all>*: Usually we
manipulate one column in one call of column arithmetic. For instance,
with the command below the elements of the ‘AWAV’ column will be sumed.
$ asttable table.fits -c'arith AWAV sumvalue'
But sometimes, we want to manipulate more than one column with the
same expression. For example we want to sum all the elements of all the
columns. In this case we could use the following command (assuming that
the table has four different ‘AWAV-*’ columns):
$ asttable table.fits -c'arith AWAV-1 sumvalue' \
-c'arith AWAV-2 sumvalue' \
-c'arith AWAV-3 sumvalue' \
-c'arith AWAV-4 sumvalue'
To avoid repetition and mistakes, instead of using column arithmetic
many times, we can use the ‘$_all’ identifier. When column arithmetic
confronts this special string, it will repeat the expression for all the
columns in the input table. Therefore the command above can be written
as:
$ asttable table.fits -c'arith $_all sumvalue'
Alternatively, if the columns have meta-data and the first two are
respectively called ‘AWAV’ and ‘SPECTRUM’, the command above is
equivalent to the command below. Note that the character '<$>' is no
longer necessary in this scenario (because names will not be confused
with numbers):
$ asttable table.fits -cAWAV,SPECTRUM -c'arith AWAV 1e10 x'
Comparison of the two commands above clearly shows why it is
recommended to use column names instead of numbers. When the columns
have descriptive names, the command/script actually becomes much more
readable, describing the intent of the operation. It is also
independent of the low-level table structure: for the second command,
the column numbers of the ‘AWAV’ and ‘SPECTRUM’ columns in ‘table.fits’
is irrelevant.
Column arithmetic changes the values of the data within the column.
So the old column metadata cannot be used any more. By default the
output column of the arithmetic operation will be given a generic
metadata (for example, its name will be ‘ARITH_1’, which is hardly
useful!). But metadata are critically important and it is good practice
to always have short, but descriptive, names for each columns, units and
also some comments for more explanation. To add metadata to a column,
you can use the ‘--colmetadata’ option that is described in *note
Invoking asttable:: and *note Operation precedence in Table::.
Since the arithmetic expressions are a value to ‘--column’, it does
not necessarily have to be a separate option, so the commands above are
also identical to the command below (note that this only has one ‘-c’
option). Just be very careful with the quoting! With the
‘--colmetadata’ option, we are also giving a name, units and a comment
to the third column.
$ asttable table.fits -cAWAV,SPECTRUM,'arith AWAV 1e10 x' \
--colmetadata=3,AWAV_A,angstrom,"Wavelength (in Angstroms)"
In case you need to append columns from other tables (with
‘--catcolumnfile’), you can use those extra columns in column arithmetic
also. The easiest, and most robust, way is that your columns of
interest (in all files whose columns are to be merged) have different
names. In this scenario, you can simply use the names of the columns
you plan to append. If there are similar names, note that by default
Table appends a ‘-N’ to similar names (where ‘N’ is the file counter
given to ‘--catcolumnfile’, see the description of ‘--catcolumnfile’ for
more). Using column numbers can get complicated: if the number is
smaller than the main input's number of columns, the main input's column
will be used. Otherwise (when the requested column number is larger
than the main input's number of columns), the final output (after
appending all the columns from all the possible files) column number
will be used.
Almost all the arithmetic operators of *note Arithmetic operators::
are also supported for column arithmetic in Table. In particular, the
few that are not present in the Gnuastro library(1) are not yet
supported for column arithmetic. Besides the operators in *note
Arithmetic operators::, several operators are only available in Table to
use on table columns.
‘wcs-to-img’
Convert the given WCS positions to image/dataset coordinates based
on the number of dimensions in the WCS structure of ‘--wcshdu’
extension/HDU in ‘--wcsfile’. It will output the same number of
columns. The first popped operand is the last FITS dimension.
For example, the two commands below (which have the same output)
will produce 5 columns. The first three columns are the input
table's ID, RA and Dec columns. The fourth and fifth columns will
be the pixel positions in ‘image.fits’ that correspond to each RA
and Dec.
$ asttable table.fits -cID,RA,DEC,'arith RA DEC wcs-to-img' \
--wcsfile=image.fits
$ asttable table.fits -cID,RA -cDEC \
-c'arith RA DEC wcs-to-img' --wcsfile=image.fits
‘img-to-wcs’
Similar to ‘wcs-to-img’, except that image/dataset coordinates are
converted to WCS coordinates.
‘eq-j2000-to-flat’
Convert spherical RA and Dec (in Julian year 2000.0 equatorial
coordinates; which are the most common) into RA and Dec on a flat
surface based on the given reference point and projection. The
full details of the operands to this operator are given below, but
let's start with a practical example to show the concept.
At (or very near) the reference point the output of this operator
will be the same as the input. But as you move away from the
reference point, distortions due to the particular projection will
gradually cause changes in the output (when compared to the input).
For example if you simply plot RA and Dec without this operator, a
circular annulus on the sky will become elongated as the
declination of its center goes farther from the equator. For a
demonstration of the difference between curved and flat RA and
Decs, see *note Pointings that account for sky curvature:: in the
Tutorials chapter.
Let's assume you want to plot a set of RA and Dec points (defined
on a spherical surface) in a paper (a flat surface) and that
‘table.fits’ contains the RA and Dec in columns that are called
‘RA’ and ‘DEC’. With the command below, the points will be
converted to flat-RA and flat-Dec using the Gnomonic projection
(which is known as ‘TAN’ in the FITS WSC standard; see the
description of the first popped operand below):
$ asttable table.fits \
-c'arith RA set-r DEC set-d \
r d r meanvalue d meanvalue TAN \
eq-j2000-to-flat'
As you see, the RA and Dec (‘r’ and ‘d’) are the last two operators
that are popped. Before them, the reference point's coordinates
are calculated from the mean of the RA and Decs ('‘r meanvalue’'
and '‘d meanvalue’'), and the first popped operand is the
projection (‘TAN’). We are using the mean RA and Dec as the
reference point since we are assuming that this is the only set of
points you want to convert. In case you have to plot multiple sets
of points in the same plot, you should give the same reference
point in each separate conversion; like the example below:
$ ref_ra=123.45
$ ref_dec=-6.789
$ asttable table-1.fits --output=flat-1.txt \
-c'arith RA DEC '$ref_ra' '$ref_dec' TAN \
eq-j2000-to-flat'
$ asttable table-2.fits --output=flat-2.txt \
-c'arith RA DEC '$ref_ra' '$ref_dec' TAN \
eq-j2000-to-flat'
This operator takes 5 operands:
1. The _first_ popped operand (closest to the operator) is the
standard FITS WCS projection to use; and should contain a
single element (not a column). The full list of projections
can be seen in the description of the ‘--ctype’ option in
*note Align pixels with WCS considering distortions::. The
most common projection for smaller fields of view is ‘TAN’
(Gnomonic), but when your input catalog contains large
portions of the sky, projections like ‘MOL’ (Mollweide) should
be used. This is because distortions caused by the ‘TAN’
projection can become very significant after a couple of
degrees from the reference point.
2. The _second_ popped operand is the reference point's
declination; and should contain a single value (not a column).
3. The _third_ popped operand is the reference point's right
ascension; and should contain a single value (not a column).
4. The _fourth_ popped operand is the declination column of your
input table (the points that will be converted).
5. The _fifth_ popped operand is the right ascension column of
your input table (the points that will be converted).
‘eq-j2000-from-flat’
The inverse of ‘eq-j2000-to-flat’. In other words, you have a set
of points defined on the flat RA and Dec (after the projection from
spherical to flat), but you want to map them to spherical RA and
Dec. For an example, see *note Pointings that account for sky
curvature:: in the Gnuastro tutorials.
‘distance-flat’
Return the distance between two points assuming they are on a flat
surface. Note that each point needs two coordinates, so this
operator needs four operands (currently it only works for 2D
spaces). The first and second popped operands are considered to
belong to one point and the third and fourth popped operands to the
second point.
Each of the input points can be a single coordinate or a full table
column (containing many points). In other words, the following
commands are all valid:
$ asttable table.fits \
-c'arith X1 Y1 X2 Y2 distance-flat'
$ asttable table.fits \
-c'arith X Y 12.345 6.789 distance-flat'
$ asttable table.fits \
-c'arith 12.345 6.789 X Y distance-flat'
In the first case we are assuming that ‘table.fits’ has the
following four columns ‘X1’, ‘Y1’, ‘X2’, ‘Y2’. The returned column
by this operator will be the difference between two points in each
row with coordinates like the following (‘X1’, ‘Y1’) and (‘X2’,
‘Y2’). In other words, for each row, the distance between
different points is calculated. In the second and third cases
(which are identical), it is assumed that ‘table.fits’ has the two
columns ‘X’ and ‘Y’. The returned column by this operator will be
the difference of each row with the fixed point at (12.345, 6.789).
‘distance-on-sphere’
Return the spherical angular distance (along a great circle, in
degrees) between the given two points. Note that each point needs
two coordinates (in degrees), so this operator needs four operands.
The first and second popped operands are considered to belong to
one point and the third and fourth popped operands to the second
point.
Each of the input points can be a single coordinate or a full table
column (containing many points). In other words, the following
commands are all valid:
$ asttable table.fits \
-c'arith RA1 DEC1 RA2 DEC2 distance-on-sphere'
$ asttable table.fits \
-c'arith RA DEC 9.876 5.432 distance-on-sphere'
$ asttable table.fits \
-c'arith 9.876 5.432 RA DEC distance-on-sphere'
In the first case we are assuming that ‘table.fits’ has the
following four columns ‘RA1’, ‘DEC1’, ‘RA2’, ‘DEC2’. The returned
column by this operator will be the difference between two points
in each row with coordinates like the following (‘RA1’, ‘DEC1’) and
(‘RA2’, ‘DEC2’). In other words, for each row, the angular
distance between different points is calculated. In the second and
third cases (which are identical), it is assumed that ‘table.fits’
has the two columns ‘RA’ and ‘DEC’. The returned column by this
operator will be the difference of each row with the fixed point at
(9.876, 5.432).
The distance (along a great circle) on a sphere between two points
is calculated with the equation below, where $r_1$, $r_2$, $d_1$
and $d_2$ are the right ascensions and declinations of points 1 and
2.
$$\cos(d)=\sin(d_1)\sin(d_2)+\cos(d_1)\cos(d_2)\cos(r_1-r_2)$$
‘ra-to-degree’
Convert the hour-wise Right Ascension (RA) string, in the
sexagesimal format of ‘_h_m_s’ or ‘_:_:_’, to degrees. Note that
the input column has to have a string format. In FITS tables,
string columns are well-defined. For plain-text tables, please
follow the standards defined in *note Gnuastro text table format::,
otherwise the string column will not be read.
$ asttable catalog.fits -c'arith RA ra-to-degree'
$ asttable catalog.fits -c'arith $5 ra-to-degree'
‘dec-to-degree’
Convert the sexagesimal Declination (Dec) string, in the format of
‘_d_m_s’ or ‘_:_:_’, to degrees (a single floating point number).
For more details please see the ‘ra-to-degree’ operator.
‘degree-to-ra’
Convert degrees (a column with a single floating point number) to
the Right Ascension, RA, string (in the sexagesimal format hours,
minutes and seconds, written as ‘_h_m_s’). The output will be a
string column so no further mathematical operations can be done on
it. The output file can be in any format (for example, FITS or
plain-text). If it is plain-text, the string column will be
written following the standards described in *note Gnuastro text
table format::.
‘degree-to-dec’
Convert degrees (a column with a single floating point number) to
the Declination, Dec, string (in the format of ‘_d_m_s’). See the
‘degree-to-ra’ for more on the format of the output.
‘date-to-sec’
Return the number of seconds from the Unix epoch time (00:00:00
Thursday, January 1st, 1970). The input (popped) operand should be
a string column in the FITS date format (most generally:
‘YYYY-MM-DDThh:mm:ss.ddd...’).
The returned operand will be named ‘UNIXSEC’ (short for
Unix-seconds) and will be a 64-bit, signed integer, see *note
Numeric data types::. If the input string has sub-second
precision, it will be ignored because floating point numbers cannot
accurately store numbers with many significant digits. To preserve
sub-second precision, please use ‘date-to-millisec’.
For example, in the example below we are using this operator, in
combination with the ‘--keyvalue’ option of the Fits program, to
sort your desired FITS files by observation date (value in the
‘DATE-OBS’ keyword in example below):
$ astfits *.fits --keyvalue=DATE-OBS --colinfoinstdout \
| asttable -cFILENAME,'arith DATE-OBS date-to-sec' \
--colinfoinstdout \
| asttable --sort=UNIXSEC
If you do not need to see the Unix-seconds any more, you can add a
‘-cFILENAME’ (short for ‘--column=FILENAME’) at the end. For more
on ‘--keyvalue’, see *note Keyword inspection and manipulation::.
‘date-to-millisec’
Return the number of milli-seconds from the Unix epoch time
(00:00:00 Thursday, January 1st, 1970). The input (popped) operand
should be a string column in the FITS date format (most generally:
‘YYYY-MM-DDThh:mm:ss.ddd...’, where ‘.ddd’ is the optional
sub-second component).
The returned operand will be named ‘UNIXMILLISEC’ (short for Unix
milli-seconds) and will be a 64-bit, signed integer, see *note
Numeric data types::. The returned value is not a floating point
type because for large numbers, floating point data types loose
single-digit precision (which is important here).
Other than the units of the output, this operator behaves similarly
to ‘date-to-sec’. See the description of that operator for an
example.
‘sorted-to-interval’
Given a single column (which must be already sorted and have a
numeric data type), return two columns: the first returned column
is the minimum and the second returned column is the maximum value
of the interval of each row row. The maximum of each row is equal
to the minimum of the previous row; thus creating a contiguous
interval coverage of the input column's range in all rows.
The minimum value of the first row and maximum of the last row will
be smaller/larger than the respective row of the input (based on
the distance to the next/previous element). This is done to ensure
that if your input has a fixed interval length between all
elements, the first and last intervals also have that fixed length.
For example, with the command below, we'll use this operator on a
hypothetical radial profile. Note how the intervals are contiguous
even though the radius values are not equally distant (if the row
with a radius of 2.5 didn't exist, the intervals would all be the
same length). For another example of the usage of this operator,
see the example in the description of ‘--customtable’ in *note
MakeProfiles profile settings::.
$ cat radial-profile.txt
# Column 1: RADIUS [pix,f32,] Distance to center in pixels.
# Column 2: MEAN [ADU,f32,] Mean value at that radius.
0 100
1 40
2 30
2.5 25
3 20
$ asttable radial-profile.txt --txtf32f=fixed --txtf32p=3 \
-c'arith RADIUS sorted-to-interval',MEAN
-0.500 0.500 100.000
0.500 1.500 40.000
1.500 2.250 30.000
2.250 2.750 25.000
2.750 3.250 20.000
Such intervals can be useful in scenarios like generating the input
to ‘--customtable’ in MakeProfiles (see *note MakeProfiles profile
settings::) from a radial profile (see *note Generate radial
profile::).
---------- Footnotes ----------
(1) For a list of the Gnuastro library arithmetic operators, please
see the macros starting with ‘GAL_ARITHMETIC_OP’ and ending with the
operator name in *note Arithmetic on datasets::.
File: gnuastro.info, Node: Operation precedence in Table, Next: Invoking asttable, Prev: Column arithmetic, Up: Table
5.3.4 Operation precedence in Table
-----------------------------------
The Table program can do many operations on the rows and columns of the
input tables and they are not always applied in the order you call the
operation on the command-line. In this section we will describe which
operation is done before/after which operation. Knowing this precedence
table is important to avoid confusion when you ask for more than one
operation. For a description of each option, please see *note Invoking
asttable::. By default, column-based operations will be done first.
You can ask for switching to row-based operations to be done first,
using the ‘--rowfirst’ option.
*Pipes for different precedence:* It may happen that your desired series
of operations cannot be done with the precedence mentioned below (in one
command). In this case, you can pipe the output of one call to
‘asttable’ to another ‘asttable’. Just don't forget to give ‘-O’ (or
‘--colinfoinstdout’) to the first instance (so the column metadata are
also passed to the next instance). Without metadata, all numbers will
be read as double-precision (see *note Gnuastro text table format::;
recall that piping is done in plain text format), vector columns will be
broken into single-valued columns, and column names, units and comments
will be lost. At the end of this section, there is an example of doing
this.
Input table information
The first set of operations that will be preformed (if requested)
are the printing of the input table information. Therefore, when
the following options are called, the column data are not read at
all. Table simply reads the main input's column metadata (name,
units, numeric data type and comments), and the number of rows and
prints them. Table then terminates and no other operation is done.
These can therefore be called at the end of an arbitrarily long
Table command. When you have forgot some information about the
input table. You can then delete these options and continue
writing the command (using the shell's history to retrieve the
previous command with an up-arrow key).
At any time only a single one of the options in this category may
be called. The order of checking for these options is therefore
important: in the same order that they are described below:
Column and row information (‘--information’ or ‘-i’)
Print the list of input columns and the metadata of each
column in a single row. This includes the column name,
numeric data type, units and comments of each column within a
separate row of the output. Finally, print the number of
rows.
Number of columns (‘--info-num-cols’)
Print the number of columns in the input table. Only a single
integer (number of columns) is printed before Table
terminates.
Number of rows (‘--info-num-rows’)
Print the number of rows in the input table. Only a single
integer (number of rows) is printed before Table terminates.
Column selection (‘--column’)
When this option is given, only the columns given to this option
(from the main input) will be used for all future steps. When
‘--column’ (or ‘-c’) is not given, then all the main input's
columns will be used in the next steps.
Column-based operations
By default the following column-based operations will be done
before the row-based operations in the next item. If you need to
give precedence to row-based operations, use ‘--rowfirst’.
Column(s) from other file(s): ‘--catcolumnfile’
When column concatenation (addition) is requested, columns
from other tables (in other files, or other HDUs of the same
FITS file) will be added after the existing columns are read
from the main input. In one command, you can call
‘--catcolumnfile’ multiple times to allow addition of columns
from many files.
Therefore you can merge the columns of various tables into one
table in this step (at the start), then start adding/limiting
the rows, or building vector columns, . If any of the
row-based operations below are requested in the same
‘asttable’ command, they will also be applied to the rows of
the added columns. However, the conditions to keep/reject
rows can only be applied to the rows of the columns in main
input table (not the columns that are added with these
options).
Extracting single-valued columns from vectors (‘--fromvector’)
Once all the input columns are read into memory, if any of
them are vectors, you can extract a single-valued column from
the vector columns at this stage. For more on vector columns,
see *note Vector columns::.
Creating vector columns (‘--tovector’)
After column arithmetic, there is no other way to add new
columns so the ‘--tovector’ operator is applied at this stage.
You can use it to merge multiple columns that are available in
this stage to a single vector column. For more, see *note
Vector columns::.
Column arithmetic
Once the final rows are selected in the requested order,
column arithmetic is done (if requested). For more on column
arithmetic, see *note Column arithmetic::.
Row-based operations
Row-based operations only work within the rows of existing columns
when they are activated. By default row-based operations are
activated after column-based operations (which are mentioned
above). If you need to give precedence to row-based operations,
use ‘--rowfirst’.
Rows from other file(s) (‘--catrowfile’)
With this feature, you can import rows from other tables (in
other files, or other HDUs of the same FITS file). The same
column selection of ‘--column’ is applied to the tables given
to this option. The column metadata (name, units and
comments) will be taken from the main input. Two conditions
are mandatory for adding rows:
• The number of columns used from the new tables must be
equal to the number of columns in memory, by the time
control reaches here.
• The data type of each column (see *note Numeric data
types::) should be the same as the respective column in
memory by the time control reaches here. If the data
types are different, you can use the type conversion
operators of column arithmetic which has higher
precedence (and will therefore be applied before this by
default). For more on type conversion, see *note
Numerical type conversion operators:: and *note Column
arithmetic::).
Row selection by value in a column
The following operations select rows based on the values in
them. A more complete description of each of these options is
given in *note Invoking asttable::.
• ‘--range’: only keep rows where the value in the given
column is within a certain interval.
• ‘--inpolygon’: only keep rows where the value is within
the polygon of ‘--polygon’.
• ‘--outpolygon’: only keep rows outside the polygon of
‘--polygon’.
• ‘--equal’: only keep rows with an specified value in
given column.
• ‘--notequal’: only keep rows without specified value in
given column.
• ‘--noblank’: only keep rows that are not blank in the
given column(s).
These options can be called any number of times (to limit the
final rows based on values in different columns for example).
Since these are row-rejection operations, their internal order
is irrelevant. In other words, it makes no difference if
‘--equal’ is called before or after ‘--range’ for example.
As a side-effect, because NaN/blank values are defined to fail
on any condition, these operations will also remove rows with
NaN/blank values in the specified column they are checking.
Also, the columns that are used for these operations do not
necessarily have to be in the final output table (you may not
need the column after doing the selection based on it).
By default, these options are applied after merging columns
from other tables. However, currently, the column given to
these options can only come from the main input table. If you
need to apply these operations on columns from
‘--catcolumnfile’, pipe the output of one instance of Table
with ‘--catcolumnfile’ into another instance of Table as
suggested in the box above this list.
These row-based operations options are applied first because
the speed of later operations can be greatly affected by the
number of rows. For example, if you also call the ‘--sort’
option, and your row selection will result in 50 rows (from an
input of 10000 rows), limiting the number of rows first will
greatly speed up the sorting in your final output.
Sorting (‘--sort’)
Sort of the rows based on values in a certain column. The
column to sort by can only come from the main input table
columns (not columns that may have been added with
‘--catcolumnfile’).
Row selection (by position)
• ‘--head’: keep only requested number of top rows.
• ‘--tail’: keep only requested number of bottom rows.
• ‘--rowrandom’: keep only a random number of rows.
• ‘--rowrange’: keep only rows within a certain positional
interval.
These options limit/select rows based on their position within
the table (not their value in any certain column).
Transpose vector columns (‘--transpose’)
Transposing vector columns will not affect the number or
metadata of columns, it will just re-arrange them in their 2D
structure. As a result, after transposing, the number of rows
changes, as well as the number of elements in each vector
column. See the description of this option in *note Invoking
asttable:: for more (with an example).
Column metadata (‘--colmetadata’)
Once the structure of the final table is set, you can set the
column metadata just before finishing.
Output row selection (‘--noblankend’)
Only keep the output rows that do not have a blank value in the
given column(s). For example, you may need to apply arithmetic
operations on the columns (through *note Column arithmetic::)
before rejecting the undesired rows. After the arithmetic
operation is done, you can use the ‘where’ operator to set the
non-desired columns to NaN/blank and use ‘--noblankend’ option to
remove them just before writing the output. In other scenarios,
you may want to remove blank values based on columns in another
table. To help in readability, you can also use the final column
names that you set with ‘--colmetadata’! See the example below for
applying any generic value-based row selection based on
‘--noblankend’.
As an example, let's review how Table interprets the command below.
We are assuming that ‘table.fits’ contains at least three columns: ‘RA’,
‘DEC’ and ‘PARAM’ and you only want the RA and Dec of the rows where
$p\times 2<5$ ($p$ is the value of each row in the ‘PARAM’ column).
$ asttable table.fits -cRA,DEC --noblankend=MULTIP \
-c'arith PARAM 2 x set-i i i 5 gt nan where' \
--colmetadata=3,MULTIP,unit,"Description of column"
Due to the precedence described in this section, Table does these
operations (which are independent of the order of the operations written
on the command-line):
1. At the start (with ‘-cRA,DEC’), Table reads the ‘RA’ and ‘DEC’
columns.
2. In between all the operations in the command above, Column
arithmetic (with ‘-c'arith ...'’) has the highest precedence. So
the arithmetic operation is done and stored as a new (third)
column. In this arithmetic operation, we multiply all the values
of the ‘PARAM’ column by 2, then set all those with a value larger
than 5 to NaN (for more on understanding this operation, see the
'‘set-’' and '‘where’' operators in *note Arithmetic operators::).
3. Updating column metadata (with ‘--colmetadata’) is then done to
give a name (‘MULTIP’) to the newly calculated (third) column.
During the process, besides a name, we also set a unit and
description for the new column. These metadata entries are _very
important_, so always be sure to add metadata after doing column
arithmetic.
4. The lowest precedence operation is ‘--noblankend=MULTIP’. So only
rows that are not blank/NaN in the ‘MULTIP’ column are kept.
5. Finally, the output table (with three columns) is written to the
command-line. If you also want to print the column metadata, you
can use the ‘-O’ (or ‘--colinfoinstdout’) option. Alternatively,
if you want the output in a file, you can use the ‘--output’ option
to save the table in FITS or plain-text format.
It may happen that your desired operation needs a separate
precedence. In this case you can pipe the output of Table into another
call of Table and use the ‘-O’ (or ‘--colinfoinstdout’) option to
preserve the metadata between the two calls.
For example, let's assume that you want to sort the output table from
the example command above based on the new ‘MULTIP’ column. Since
sorting is done prior to column arithmetic, you cannot do it in one
command, but you can circumvent this limitation by simply piping the
output (including metadata) to another call to Table:
asttable table.fits -cRA,DEC --noblankend=MULTIP --colinfoinstdout \
-c'arith PARAM 2 x set-i i i 5 gt nan where' \
--colmetadata=3,MULTIP,unit,"Description of column" \
| asttable --sort=MULTIP --output=selected.fits
File: gnuastro.info, Node: Invoking asttable, Prev: Operation precedence in Table, Up: Table
5.3.5 Invoking Table
--------------------
Table will read/write, select, modify, or show the information of the
rows and columns in recognized Table formats (including FITS binary,
FITS ASCII, and plain text table files, see *note Tables::). Output
columns can also be determined by number or regular expression matching
of column names, units, or comments. The executable name is ‘asttable’
with the following general template
$ asttable [OPTION...] InputFile
One line examples:
## Get the table column information (name, units, or data type), and
## the number of rows:
$ asttable table.fits --information
## Print columns named RA and DEC, followed by all the columns where
## the name starts with "MAG_":
$ asttable table.fits --column=RA --column=DEC --column=/^MAG_/
## Similar to the above, but with one call to `--column' (or `-c'),
## also sort the rows by the input's photometric redshift (`Z_PHOT')
## column. To confirm the sort, you can add `Z_PHOT' to the columns
## to print.
$ asttable table.fits -cRA,DEC,/^MAG_/ --sort=Z_PHOT
## Similar to the above, but only print rows that have a photometric
## redshift between 2 and 3.
$ asttable table.fits -cRA,DEC,/^MAG_/ --range=Z_PHOT,2:3
## Only print rows with a value in the 10th column above 100000:
$ asttable table.txt --range=10,10e5,inf
## Only print the 2nd column, and the third column multiplied by 5,
## Save the resulting two columns in `table.txt'
$ asttable table.fits -c2,'arith $2 5 x' -otable.fits
## Sort the output columns by the third column, save output:
$ asttable table.fits --sort=3 -ooutput.txt
## Subtract the first column from the second in `cat.txt' (can also
## be a FITS table) and keep the third and fourth columns.
$ asttable cat.txt -c'arith $2 $1 -',3,4 -ocat.fits
## Convert sexagesimal coordinates to degrees (same can be done in a
## large table given as argument).
$ echo "7h34m35.5498 31d53m14.352s" | asttable
## Convert RA and Dec in degrees to sexagesimal (same can be done in a
## large table given as argument).
$ echo "113.64812416667 31.88732" \
| asttable -c'arith $1 degree-to-ra $2 degree-to-dec'
## Extract columns 1 and 2, as well as all those between 12 to 58:
$ asttable table.fits -c1,2,$(seq -s',' 12 58)
Table's input dataset can be given either as a file or from Standard
input (piped from another program, see *note Standard input::). In the
absence of selected columns, all the input's columns and rows will be
written to the output. The full set of operations Table can do are
described in detail below, but for a more high-level introduction to the
various operations, and their precedence, see *note Operation precedence
in Table::.
If any output file is explicitly requested (with ‘--output’) the
output table will be written in it. When no output file is explicitly
requested the output table will be written to the standard output. If
the specified output is a FITS file, the type of FITS table (binary or
ASCII) will be determined from the ‘--tabletype’ option. If the output
is not a FITS file, it will be printed as a plain text table (with space
characters between the columns). When the output is not binary (for
example standard output or a plain-text), the ‘--txtf32*’ or ‘--txtf64*’
options can be used for the formatting of floating point columns (see
*note Printing floating point numbers::). When the columns are
accompanied by meta-data (like column name, units, or comments), this
information will also printed in the plain text file before the table,
as described in *note Gnuastro text table format::.
For the full list of options common to all Gnuastro programs please
see *note Common options::. Options can also be stored in directory,
user or system-wide configuration files to avoid repeating on the
command-line, see *note Configuration files::. Table does not follow
Automatic output that is common in most Gnuastro programs, see *note
Automatic output::. Thus, in the absence of an output file, the
selected columns will be printed on the command-line with no column
information, ready for redirecting to other tools like ‘awk’.
*Sexagesimal coordinates as floats in plain-text tables:* When a column
is determined to be a floating point type (32-bit or 64-bit) in a
plain-text table, it can contain sexagesimal values in the format of
'‘_h_m_s’' (for RA) and '‘_d_m_s’' (for Dec), where the '‘_’'s are
place-holders for numbers. In this case, the string will be immediately
converted to a single floating point number (in units of degrees) and
stored in memory with the rest of the column or table. Besides being
useful in large tables, with this feature, conversion to sexagesimal
coordinates to degrees becomes very easy, for example:
echo "7h34m35.5498 31d53m14.352s" | asttable
The inverse can also be done with the more general column arithmetic
operators:
echo "113.64812416667 31.88732" \
| asttable -c'arith $1 degree-to-ra $2 degree-to-dec'
If you want to preserve the sexagesimal contents of a column, you should
store that column as a string, see *note Gnuastro text table format::.
‘-i’
‘--information’
Only print the column information in the specified table on the
command-line and exit. Each column's information (number, name,
units, data type, and comments) will be printed as a row on the
command-line. If the column is a multi-value (vector) a ‘[N]’ is
printed after the type, where ‘N’ is the number of elements within
that vector.
Note that the FITS standard only requires the data type (see *note
Numeric data types::), and in plain text tables, no
meta-data/information is mandatory. Gnuastro has its own
convention in the comments of a plain text table to store and
transfer this information as described in *note Gnuastro text table
format::.
This option will take precedence over all other operations in
Table, so when it is called along with other operations, they will
be ignored, see *note Operation precedence in Table::. This can be
useful if you forget the identifier of a column after you have
already typed some on the command-line. You can simply add a ‘-i’
to your already-written command (without changing anything) and run
Table, to see the whole list of column names and information. Then
you can use the shell history (with the up arrow key on the
keyboard), and retrieve the last command with all the previously
typed columns present, delete ‘-i’ and add the identifier you had
forgot.
‘--info-num-cols’
Similar to ‘--information’, but only the number of the input
table's columns will be printed as a single integer (useful in
scripts for example).
‘--info-num-rows’
Similar to ‘--information’, but only the number of the input
table's rows will be printed as a single integer (useful in scripts
for example).
‘-c STR/INT’
‘--column=STR/INT’
Set the output columns either by specifying the column number, or
name. For more on selecting columns, see *note Selecting table
columns::. If a value of this option starts with '‘arith ’',
column arithmetic will be activated, allowing you to
edit/manipulate column contents. For more on column arithmetic see
*note Column arithmetic::.
To ask for multiple columns this option can be used in two ways: 1)
multiple calls to this option, 2) using a comma between each column
specifier in one call to this option. These different solutions
may be mixed in one call to Table: for example, '‘-cRA,DEC,MAG’',
or '‘-cRA,DEC -cMAG’' are both equivalent to '‘-cRA -cDEC -cMAG’'.
The order of the output columns will be the same order given to the
option or in the configuration files (see *note Configuration file
precedence::).
This option is not mandatory, if no specific columns are requested,
all the input table columns are output. When this option is called
multiple times, it is possible to output one column more than once.
*Sequence of columns:* when dealing with a large number catalogs
(hundreds for example!), it will be frustrating, annoying and buggy
to insert the columns manually. If you want to read all the input
columns, you can use the special ‘_all’ value to ‘--column’ option.
A more generic solution (for example if you want every second one,
or all the columns within a special range) is to use the ‘seq’
command's features with an extra ‘-s','’ (so a comma is used as the
"separator"). For example if you want columns 1, 2 and all columns
between 12 to 58 (inclusive), you can use the following command:
$ asttable table.fits -c1,2,$(seq -s',' 12 58)
‘-w FITS’
‘--wcsfile=FITS’
FITS file that contains the WCS to be used in the ‘wcs-to-img’ and
‘img-to-wcs’ operators of *note Column arithmetic::. The extension
name/number within the FITS file can be specified with ‘--wcshdu’.
If the value to this option is '‘none’', no WCS will be written in
the output.
‘-W STR’
‘--wcshdu=STR’
FITS extension/HDU in the FITS file given to ‘--wcsfile’ (see the
description of ‘--wcsfile’ for more).
‘-L FITS/TXT’
‘--catcolumnfile=FITS/TXT’
Concatenate (or add, or append) the columns of this option's value
(a filename) to the output columns. This option may be called
multiple times (to add columns from more than one file into the
final output), the columns from each file will be added in the same
order that this option is called. The number of rows in the
file(s) given to this option has to be the same as the input table
(before any type of row-selection), see *note Operation precedence
in Table::.
By default all the columns of the given file will be appended, if
you only want certain columns to be appended, use the
‘--catcolumns’ option to specify their name or number (see *note
Selecting table columns::). Note that the columns given to
‘--catcolumns’ must be present in all the given files (if this
option is called more than once with more than one file).
If the file given to this option is a FITS file, it is necessary to
also define the corresponding HDU/extension with ‘--catcolumnhdu’.
Also note that no operation (such as row selection and arithmetic)
is applied to the table given to this option.
If the appended columns have a name, and their name is already
present in the table before adding those columns, the column names
of each file will be appended with a ‘-N’, where ‘N’ is a counter
starting from 1 for each appended table. Just note that in the
FITS standard (and thus in Gnuastro), column names are not
case-sensitive.
This is done because when concatenating columns from multiple
tables (more than two) into one, they may have the same name, and
it is not good practice to have multiple columns with the same
name. You can disable this feature with ‘--catcolumnrawname’.
Generally, you can use the ‘--colmetadata’ option to update column
metadata in the same command, after all the columns have been
concatenated.
For example, let's assume you have two catalogs of the same objects
(same number of rows) in different filters. Such that
‘f160w-cat.fits’ has a ‘MAGNITUDE’ column that has the magnitude of
each object in the ‘F160W’ filter and similarly ‘f105w-cat.fits’,
also has a ‘MAGNITUDE’ column, but for the ‘F105W’ filter. You can
use column concatenation like below to import the ‘MAGNITUDE’
column from the ‘F105W’ catalog into the ‘F160W’ catalog, while
giving each magnitude column a different name:
asttable f160w-cat.fits --output=both.fits \
--catcolumnfile=f105w-cat.fits --catcolumns=MAGNITUDE \
--colmetadata=MAGNITUDE,MAG-F160W,log,"Magnitude in F160W" \
--colmetadata=MAGNITUDE-1,MAG-F105W,log,"Magnitude in F105W"
For a more complete example, see *note Working with catalogs
estimating colors::.
*Loading external columns with Arithmetic:* an alternative way to
load external columns into your output is to use column arithmetic
(*note Column arithmetic::) In particular the ‘load-col-’ operator
described in *note Loading external columns::. But this operator
will load only one column per file/HDU every time it is called. So
if you have many columns to insert, it is much faster to use
‘--catcolumnfile’. Because ‘--catcolumnfile’ will load all the
columns in one opening of the file, and possibly even read them all
into memory in parallel!
‘-u STR/INT’
‘--catcolumnhdu=STR/INT’
The HDU/extension of the FITS file(s) that should be concatenated,
or appended, by column with ‘--catcolumnfile’. If ‘--catcolumn’ is
called more than once with more than one FITS file, it is necessary
to call this option more than once. The HDUs will be loaded in the
same order as the FITS files given to ‘--catcolumnfile’.
‘-C STR/INT’
‘--catcolumns=STR/INT’
The column(s) in the file(s) given to ‘--catcolumnfile’ to append.
When this option is not given, all the columns will be
concatenated. See ‘--catcolumnfile’ for more.
‘--catcolumnrawname’
Do not modify the names of the concatenated (appended) columns, see
description in ‘--catcolumnfile’.
‘--transpose’
Transpose (as in a matrix) the given vector column(s) individually.
When this operation is done (see *note Operation precedence in
Table::), only vector columns of the same data type and with the
same number of elements should exist in the table. A usage of this
operator is presented in the IFU spectroscopy tutorial in *note
Extracting a single spectrum and plotting it::.
As a generic example, see the commands below. The ‘in.txt’ table
below has two vector columns (each with three elements) in two
rows. After running ‘asttable’ with ‘--transpose’, you can see how
the vector columns have two elements per row (‘u8(3)’ has been
replaced by ‘u8(2)’), and that the table now has three rows.
$ cat in.txt
# Column 1: abc [nounits,u8(3),] First vector column.
# Column 2: def [nounits,u8(3),] Second vector column.
111 112 113 211 212 213
121 122 123 221 222 223
$ asttable in.txt --transpose -O
# Column 1: abc [nounits,u8(2),] First vector column.
# Column 2: def [nounits,u8(2),] Second vector column.
111 121 211 221
112 122 212 222
113 123 213 223
‘--fromvector=STR,INT[,INT[,INT]]’
Extract the given tokens/elements from the given vector column into
separate single-valued columns. The input vector column can be
identified by its name or counter, see *note Selecting table
columns::. After the columns are extracted, the input vector is
deleted by default. To preserve the input vector column, you can
use ‘--keepvectfin’ described below. For a complete usage scenario
see *note Vector columns::.
‘--tovector=STR/INT,STR/INT[,STR/INT]’
Move the given columns into a newly created vector column. The
given columns can be identified by their name or counter, see *note
Selecting table columns::. After the columns are copied, they are
deleted by default. To preserve the inputs, you can use
‘--keepvectfin’ described below. For a complete usage scenario see
*note Vector columns::.
‘-k’
‘--keepvectfin’
Do not delete the input column(s) when using ‘--fromvector’ or
‘--tovector’.
‘-R FITS/TXT’
‘--catrowfile=FITS/TXT’
Add the rows of the given file to the output table. The selected
columns in the tables given to this option should have the same
number and datatype and the rows before control reaches this phase
(after column selection and column concatenation), for more see
*note Operation precedence in Table::.
For example, if ‘a.fits’, ‘b.fits’ and ‘c.fits’ have the columns
‘RA’, ‘DEC’ and ‘MAGNITUDE’ (possibly in different column-numbers
in their respective table, along with many more columns), the
command below will add their rows into the final output that will
only have these three columns:
$ asttable a.fits --catrowfile=b.fits --catrowhdu=1 \
--catrowfile=c.fits --catrowhdu=1 \
-cRA,DEC,MAGNITUDE --output=allrows.fits
*Provenance of each row:* When merging rows from separate catalogs,
it is important to keep track of the source catalog of each row
(its provenance). To do this, you can use ‘--catrowfile’ in
combination with the ‘constant’ operator and *note Column
arithmetic::. For a working example of this scenario, see the
example within the documentation of the ‘constant’ operator in
*note Building new dataset and stack management::.
*How to avoid repetition when adding rows:* this option will simply
add the rows of multiple tables into one, it does not check their
contents! Therefore if you use this option on multiple catalogs
that may have some shared physical objects in some of their rows,
those rows/objects will be repeated in the final table. In such
scenarios, to avoid potential repetition, it is better to use *note
Match:: (with ‘--notmatched’ and ‘--outcols=AAA,BBB’) instead of
Table. For more on using Match for this scenario, see the
description of ‘--outcols’ in *note Invoking astmatch::.
‘-X STR’
‘--catrowhdu=STR’
The HDU/extension of the FITS file(s) that should be concatenated,
or appended, by rows with ‘--catrowfile’. If ‘--catrowfile’ is
called more than once with more than one FITS file, it is necessary
to call this option more than once also (once for every FITS table
given to ‘--catrowfile’). The HDUs will be loaded in the same
order as the FITS files given to ‘--catrowfile’.
‘-O’
‘--colinfoinstdout’
Add column metadata when the output is printed in the standard
output. Usually the standard output is used for a fast visual
check, or to pipe into other metadata-agnostic programs (like AWK)
for further processing. So by default meta-data are not included.
But when piping to other Gnuastro programs (where metadata can be
interpreted and used) it is recommended to use this option and use
column names in the next program.
‘-r STR,FLT:FLT’
‘--range=STR,FLT:FLT’
Only output rows that have a value within the given range in the
‘STR’ column (can be a name or counter). Note that the range is
only inclusive in the lower-limit. For example, with
‘--range=sn,5:20’ the output's columns will only contain rows that
have a value in the ‘sn’ column (not case-sensitive) that is
greater or equal to 5, and less than 20. Also you can use the
comma for separating the values such as this ‘--range=sn,5,20’.
For the precedence of this operation in relation to others, see
*note Operation precedence in Table::.
This option can be called multiple times (different ranges for
different columns) in one run of the Table program. This is very
useful for selecting the final rows from multiple criteria/columns.
The chosen column does not have to be in the output columns. This
is good when you just want to select using one column's values, but
do not need that column anymore afterwards.
For one example of using this option, see the example under
‘--sigclip-median’ in *note Invoking aststatistics::.
‘--inpolygon=STR1,STR2’
Only return rows where the given coordinates are inside the polygon
specified by the ‘--polygon’ option. The coordinate columns are
the given ‘STR1’ and ‘STR2’ columns, they can be a column name or
counter (see *note Selecting table columns::). For the precedence
of this operation in relation to others, see *note Operation
precedence in Table::.
Note that the chosen columns does not have to be in the output
columns (which are specified by the ‘--column’ option). For
example, if we want to select rows in the polygon specified in
*note Dataset inspection and cropping::, this option can be used
like this (you can remove the double quotations and write them all
in one line if you remove the white-spaces around the colon
separating the column vertices):
asttable table.fits --inpolygon=RA,DEC \
--polygon="53.187414,-27.779152 \
: 53.159507,-27.759633 \
: 53.134517,-27.787144 \
: 53.161906,-27.807208" \
*Flat/Euclidean space: * The ‘--inpolygon’ option assumes a
flat/Euclidean space so it is only correct for RA and Dec when the
polygon size is very small like the example above. If your polygon
is a degree or larger, it may not return correct results. Please
get in touch if you need such a feature (see *note Suggest new
feature::).
‘--outpolygon=STR1,STR2’
Only return rows where the given coordinates are outside the
polygon specified by the ‘--polygon’ option. This option is very
similar to the ‘--inpolygon’ option, so see the description there
for more.
‘--polygon=STR’
‘--polygon=FLT,FLT:FLT,FLT:...’
The polygon to use for the ‘--inpolygon’ and ‘--outpolygon’
options. This option is parsed in an identical way to the same
option in the Crop program, so for more information on how to use
it, see *note Crop options::.
‘-e STR,INT/FLT,...’
‘--equal=STR,INT/FLT,...’
Only output rows that are equal to the given number(s) in the given
column. The first argument is the column identifier (name or
number, see *note Selecting table columns::), after that you can
specify any number of values. For the precedence of this operation
in relation to others, see *note Operation precedence in Table::.
For example, ‘--equal=ID,5,6,8’ will only print the rows that have
a value of 5, 6, or 8 in the ‘ID’ column. This option can also be
called multiple times, so ‘--equal=ID,4,5 --equal=ID,6,7’ has the
same effect as ‘--equal=4,5,6,7’.
*Equality and floating point numbers:* Floating point numbers are
only approximate values (see *note Numeric data types::). In this
context, their equality depends on how the input table was
originally stored (as a plain text table or as an ASCII/binary FITS
table). If you want to select floating point numbers, it is
strongly recommended to use the ‘--range’ option and set a very
small interval around your desired number, do not use ‘--equal’ or
‘--notequal’.
The ‘--equal’ and ‘--notequal’ options also work when the given
column has a string type. In this case the given value to the
option will also be parsed as a string, not as a number. When
dealing with string columns, be careful with trailing white space
characters (the actual value maybe adjusted to the right, left, or
center of the column's width). If you need to account for such
white spaces, you can use shell quoting. For example,
‘--equal=NAME," myname "’.
*Strings with a comma (,):* When your desired column values contain
a comma, you need to put a '‘\’' before the internal comma (within
the value). Otherwise, the comma will be interpreted as a
delimiter between multiple values, and anything after it will be
interpreted as a separate string. For example, assume column ‘AB’
of your ‘table.fits’ contains this value: '‘cd,ef’' in your desired
rows. To extract those rows, you should use the command below:
$ asttable table.fits --equal=AB,cd\,ef
‘-n STR,INT/FLT,...’
‘--notequal=STR,INT/FLT,...’
Only output rows that are _not_ equal to the given number(s) in the
given column. The first argument is the column identifier (name or
number, see *note Selecting table columns::), after that you can
specify any number of values. For example, ‘--notequal=ID,5,6,8’
will only print the rows where the ‘ID’ column does not have value
of 5, 6, or 8. This option can also be called multiple times, so
‘--notequal=ID,4,5 --notequal=ID,6,7’ has the same effect as
‘--notequal=4,5,6,7’.
Be very careful if you want to use the non-equality with floating
point numbers, see the special note under ‘--equal’ for more. This
option also works when the given column has a string type, see the
description under ‘--equal’ (above) for more.
‘-b STR[,STR[,STR]]’
‘--noblank=STR[,STR[,STR]]’
Only output rows that are _not_ blank in the given column of the
_input_ table. Like above, the columns can be specified by their
name or number (counting from 1). This option can be called
multiple times, so ‘--noblank=MAG --noblank=PHOTOZ’ is equivalent
to ‘--noblank=MAG,PHOTOZ’. For the precedence of this operation in
relation to others, see *note Operation precedence in Table::.
For example, if ‘table.fits’ has blank values (NaN in floating
point types) in the ‘magnitude’ and ‘sn’ columns, with
‘--noblank=magnitude,sn’, the output will not contain any rows with
blank values in these two columns.
If you want _all_ columns to be checked, simply set the value to
‘_all’ (in other words: ‘--noblank=_all’). This mode is useful
when there are many columns in the table and you want a "clean"
output table (with no blank values in any column): entering their
name or number one-by-one can be buggy and frustrating. In this
mode, no other column name should be given. For example, if you
give ‘--noblank=_all,magnitude’, then Table will assume that your
table actually has a column named ‘_all’ and ‘magnitude’, and if it
does not, it will abort with an error.
If you want to change column values using *note Column arithmetic::
(and set some to blank, to later remove), or you want to select
rows based on columns that you have imported from other tables, you
should use the ‘--noblankend’ option described below. Also, see
*note Operation precedence in Table::.
‘-s STR’
‘--sort=STR’
Sort the output rows based on the values in the ‘STR’ column (can
be a column name or number). By default the sort is done in
ascending/increasing order, to sort in a descending order, use
‘--descending’. For the precedence of this operation in relation
to others, see *note Operation precedence in Table::.
The chosen column does not have to be in the output columns. This
is good when you just want to sort using one column's values, but
do not need that column anymore afterwards.
‘-d’
‘--descending’
When called with ‘--sort’, rows will be sorted in descending order.
‘-H INT’
‘--head=INT’
Only print the given number of rows from the _top_ of the final
table. Note that this option only affects the _output_ table. For
example, if you use ‘--sort’, or ‘--range’, the printed rows are
the first _after_ applying the sort sorting, or selecting a range
of the full input. This option cannot be called with ‘--tail’,
‘--rowrange’ or ‘--rowrandom’. For the precedence of this
operation in relation to others, see *note Operation precedence in
Table::.
If the given value to ‘--head’ is 0, the output columns will not
have any rows and if it is larger than the number of rows in the
input table, all the rows are printed (this option is effectively
ignored). This behavior is taken from the ‘head’ program in GNU
Coreutils.
‘-t INT’
‘--tail=INT’
Only print the given number of rows from the _bottom_ of the final
table. See ‘--head’ for more. This option cannot be called with
‘--head’, ‘--rowrange’ or ‘--rowrandom’.
‘--rowrange=INT,INT’
Only return the rows within the requested positional range
(inclusive on both sides). Therefore, ‘--rowrange=5,7’ will return
3 of the input rows, row 5, 6 and 7. This option will abort if any
of the given values is larger than the total number of rows in the
table. For the precedence of this operation in relation to others,
see *note Operation precedence in Table::.
With the ‘--head’ or ‘--tail’ options you can only see the top or
bottom few rows. However, with this option, you can limit the
returned rows to a contiguous set of rows in the middle of the
table. Therefore this option cannot be called with ‘--head’,
‘--tail’, or ‘--rowrandom’.
‘--rowrandom=INT’
Select ‘INT’ rows from the input table by random (assuming a
uniform distribution). This option is applied _after_ the
value-based selection options (such as ‘--sort’, ‘--range’, and
‘--polygon’). On the other hand, only the row counters are
randomly selected, this option does not change the order.
Therefore, if ‘--rowrandom’ is called together with ‘--sort’, the
returned rows are still sorted. This option cannot be called with
‘--head’, ‘--tail’, or ‘--rowrange’. For the precedence of this
operation in relation to others, see *note Operation precedence in
Table::.
This option will only have an effect if ‘INT’ is larger than the
number of rows when it is activated (after the value-based
selection options have been applied). When there are fewer rows, a
warning is printed, saying that this option has no effect. The
warning can be disabled with the ‘--quiet’ option.
Due to its nature (to be random), the output of this option differs
in each run. Therefore 5 calls to Table with ‘--rowrandom’ on the
same input table will generate 5 different outputs. If you want a
reproducible random selection, set the ‘GSL_RNG_SEED’ environment
variable and also use the ‘--envseed’ option, for more see *note
Generating random numbers::.
‘--envseed’
Read the random number generator seed from the ‘GSL_RNG_SEED’
environment variable for ‘--rowrandom’ (instead of generating a
different seed internally on every run). This is useful if you
want a reproducible random selection of the input rows. For more,
see *note Generating random numbers::.
‘-E STR[,STR[,STR]]’
‘--noblankend=STR[,STR[,STR]]’
Remove all rows in the requested _output_ columns that have a blank
value. Like above, the columns can be specified by their name or
number (counting from 1). This option can be called multiple
times, so ‘--noblank=MAG --noblank=PHOTOZ’ is equivalent to
‘--noblank=MAG,PHOTOZ’. For the precedence of this operation in
relation to others, see *note Operation precedence in Table::.
for example, if your final output table (possibly after column
arithmetic, or adding new columns) has blank values (NaN in
floating point types) in the ‘magnitude’ and ‘sn’ columns, with
‘--noblankend=magnitude,sn’, the output will not contain any rows
with blank values in these two columns.
If you want blank values to be removed from the main input table
_before_ any further processing (like adding columns, sorting or
column arithmetic), you should use the ‘--noblank’ option. With
the ‘--noblank’ option, the column(s) that is(are) given does not
necessarily have to be in the output (it is just temporarily used
for reading the inputs and selecting rows, but does not necessarily
need to be present in the output). However, the column(s) given to
this option should exist in the output.
If you want _all_ columns to be checked, simply set the value to
‘_all’ (in other words: ‘--noblankend=_all’). This mode is useful
when there are many columns in the table and you want a "clean"
output table (with no blank values in any column): entering their
name or number one-by-one can be buggy and frustrating. In this
mode, no other column name should be given. For example, if you
give ‘--noblankend=_all,magnitude’, then Table will assume that
your table actually has a column named ‘_all’ and ‘magnitude’, and
if it does not, it will abort with an error.
This option is applied just before writing the final table (after
‘--colmetadata’ has finished). So in case you changed the column
metadata, or added new columns, you can use the new names, or the
newly defined column numbers. For the precedence of this operation
in relation to others, see *note Operation precedence in Table::.
‘-m STR/INT,STR[,STR[,STR]]’
‘--colmetadata=STR/INT,STR[,STR[,STR]]’
Update the specified column metadata in the output table. This
option is applied after all other column-related operations are
complete, for example, column arithmetic, or column concatenation.
For the precedence of this operation in relation to others, see
*note Operation precedence in Table::.
The first value (before the first comma) given to this option is
the column's identifier. It can either be a counter (positive
integer, counting from 1), or a name (the column's name in the
output if this option was not called).
After the to-be-updated column is identified, at least one other
string should be given, with a maximum of three strings. The first
string after the original name will the selected column's new name.
The next (optional) string will be the selected column's unit and
the third (optional) will be its comments. If the two optional
strings are not given, the original column's units or comments will
remain unchanged.
If any of the values contains a comma, you should place a '‘\’'
before the comma to avoid it getting confused with a delimiter.
For example, see the command below for a column description that
contains a comma:
$ asttable table.fits \
--colmetadata=NAME,UNIT,"Comments\, with a comma"
Generally, since the comma is commonly used as a delimiter in many
scenarios, to avoid complicating your future analysis with the
table, it is best to avoid using a comma in the column name and
units.
Some examples of this option are available in the tutorials, in
particular *note Working with catalogs estimating colors::. Here
are some more specific examples:
‘--colmetadata=MAGNITUDE,MAG_F160W’
This will convert name of the original ‘MAGNITUDE’ column to
‘MAG_F160W’, leaving the unit and comments unchanged.
‘--colmetadata=3,MAG_F160W,mag’
This will convert name of the third column of the final output
to ‘MAG_F160W’ and the units to ‘mag’, while leaving the
comments untouched.
‘--colmetadata=MAGNITUDE,MAG_F160W,mag,"Magnitude in F160W filter"’
This will convert name of the original ‘MAGNITUDE’ column to
‘MAG_F160W’, and the units to ‘mag’ and the comments to
‘Magnitude in F160W filter’. Note the double quotations
around the comment string, they are necessary to preserve the
white-space characters within the column comment from the
command-line, into the program (otherwise, upon reaching a
white-space character, the shell will consider this option to
be finished and cause un-expected behavior).
If your table is large and generated by a script, you can first do
all your operations on your table's data and write it into a
temporary file (maybe called ‘temp.fits’). Then, look into that
file's metadata (with ‘asttable temp.fits -i’) to see the exact
column positions and possible names, then add the necessary calls
to this option to your previous call to ‘asttable’, so it writes
proper metadata in the same run (for example, in a script or
Makefile). Recall that when a name is given, this option will
update the metadata of the first column that matches, so if you
have multiple columns with the same name, you can call this options
multiple times with the same first argument to change them all to
different names.
Finally, if you already have a FITS table by other means (for
example, by downloading) and you merely want to update the column
metadata and leave the data intact, it is much more efficient to
directly modify the respective FITS header keywords with ‘astfits’,
using the keyword manipulation features described in *note Keyword
inspection and manipulation::. ‘--colmetadata’ is mainly intended
for scenarios where you want to edit the data so it will always
load the full/partial dataset into memory, then write out the
resulting datasets with updated/corrected metadata.
‘-f STR’
‘--txtf32format=STR’
The plain-text format of 32-bit floating point columns when output
is not binary (this option is ignored for binary outputs like FITS
tables, see *note Printing floating point numbers::). The
acceptable values are listed below. This is just the format of the
plain-text outputs; see ‘--txtf32precision’ for customizing their
precision.
‘fixed’
Fixed-point notation (for example ‘123.4567’).
‘exp’
Exponential notation (for example ‘1.234567e+02’).
The default mode is ‘exp’ since it is the most generic and will not
cause any loss of data. Be very cautious if you set it to ‘fixed’.
As a rule of thumb, the fixed-point notation is only good if the
numbers are larger than 1.0, but not too large! Given that the
total number of accurate decimal digits is fixed the more digits
you have on the left of the decimal point (integer part), the more
un-accurate digits will be printed on the right of the decimal
point.
‘-p STR’
‘--txtf32precision=INT’
Number of digits after (to the right side of) the decimal point
(precision) for columns with a 32-bit floating point datatype (this
option is ignored for binary outputs like FITS tables, see *note
Printing floating point numbers::). This can take any positive
integer (including 0). When given a value of zero, the floating
point number will be rounded to the nearest integer.
The default value to this option is 6. This is because according
to IEEE 754, 32-bit floating point numbers can be accurately
presented to 7.22 decimal digits (see *note Printing floating point
numbers::). Since we only have an integer number of digits in a
number, we'll round it to 7 decimal digits. Furthermore, the
precision is only defined to the right side of the decimal point.
In exponential notation (default of ‘--txtf32format’), one decimal
digit will be printed on the left of the decimal point. So the
default value to this option is $7-1=6$.
‘-A STR’
‘--txtf64format=STR’
The plain-text format of 64-bit floating point columns when output
is not binary (this option is ignored for binary outputs like FITS
tables, see *note Printing floating point numbers::). The
acceptable values are listed below. This is just the format of the
plain-text outputs; see ‘--txtf64precision’ for customizing their
precision.
‘fixed’
Fixed-point notation (for example ‘12345.6789012345’).
‘exp’
Exponential notation (for example ‘1.23456789012345e4’).
The default mode is ‘exp’ since it is the most generic and will not
cause any loss of data. Be very cautious if you set it to ‘fixed’.
As a rule of thumb, the fixed-point notation is only good if the
numbers are larger than 1.0, but not too large! Given that the
total number of accurate decimal digits is fixed the more digits
you have on the left of the decimal point (integer part), the more
un-accurate digits will be printed on the right of the decimal
point.
‘-B STR’
‘--txtf64precision=INT’
Number of digits after the decimal point (precision) for columns
with a 64-bit floating point datatype (this option is ignored for
binary outputs like FITS tables, see *note Printing floating point
numbers::). This can take any positive integer (including 0).
When given a value of zero, the floating point number will be
rounded to the nearest integer.
The default value to this option is 15. This is because according
to IEEE 754, 64-bit floating point numbers can be accurately
presented to 15.95 decimal digits (see *note Printing floating
point numbers::). Since we only have an integer number of digits
in a number, we'll round it to 16 decimal digits. Furthermore, the
precision is only defined to the right side of the decimal point.
In exponential notation (default of ‘--txtf64format’), one decimal
digit will be printed on the left of the decimal point. So the
default value to this option is $16-1=15$.
‘-Y’
‘--txteasy’
When output is a plain-text file or just gets printed on standard
output (the terminal), all floating point columns are printed in
fixed point notation (as in ‘123.456’) instead of the default
exponential notation (as in ‘1.23456e+02’). For 32-bit floating
points, this option will use a precision of 3 digits (see
‘--txtf32precision’) and for 64-bit floating points use a precision
of 6 digits (see ‘--txtf64precision’). This can be useful for
human readability, but be careful with some scenarios (for example
‘1.23e-120’, which will show only as ‘0.0’!). When this option is
called any value given the following options is ignored:
‘--txtf32format’, ‘--txtf32precision’, ‘--txtf64format’ and
‘--txtf64precision’. For example below you can see the output of
table with and without this option:
$ asttable table.fits --head=5 -O
# Column 1: OBJNAME [name ,str23, ] Name in HyperLeda.
# Column 2: RAJ2000 [deg ,f64 , ] Right Ascension.
# Column 3: DEJ2000 [deg ,f64 , ] Declination.
# Column 4: RADIUS [arcmin,f32 , ] Major axis radius.
NGC0884 2.3736267000000e+00 5.7138753300000e+01 8.994357e+00
NGC1629 4.4935191000000e+00 -7.1838322400000e+01 5.000000e-01
NGC1673 4.7109672000000e+00 -6.9820892700000e+01 3.499210e-01
NGC1842 5.1216920000000e+00 -6.7273195300000e+01 3.999171e-01
$ asttable table.fits --head=5 -O -Y
# Column 1: OBJNAME [name ,str23, ] Name in HyperLeda.
# Column 2: RAJ2000 [deg ,f64 , ] Right Ascension.
# Column 3: DEJ2000 [deg ,f64 , ] Declination.
# Column 4: RADIUS [arcmin,f32 , ] Major axis radius.
NGC0884 2.373627 57.138753 8.994
NGC1629 4.493519 -71.838322 0.500
NGC1673 4.710967 -69.820893 0.350
NGC1842 5.121692 -67.273195 0.400
This is also useful when you want to make outputs of other programs
more "easy" to read, for example:
$ echo 123.45678 | asttable
1.234567800000000e+02
$ echo 123.45678 | asttable -Y
123.456780
*Can result in loss of information*: be very careful with this
option! It can loose precision or generally the full value if the
value is not within a "good" range like this example. Such cases
are the reason that this is not the default format of plain-text
outputs.
$ echo 123.4e-9 | asttable -Y
0.000000
Browse
Usage
Versions