The National Academy of Sciences Decadal Survey for Heliophysics strongly recommendeds that we should follow the lead of other NASA disciplines in providing software tools of general utility that will facilitate the use of HP data. This page is a work-in-progress aimed at meeting this recommendation. Both Solar Physics and Astronomy have IDL libraries that have proven very useful; the initial set of tools here are in that spirit and include pointers to those libraries.


Using Standard Formats

There is great utility in using standard formats for data, and HP has finally come to fairly general agreement on FITS (Solar Data), CDF (Space Physics data), and NetCDF (some ITM data). HDF may eventually be used by HP, but its use is not yet common. Below are some pointers to how to use FITS and CDF formats; NetCDF documentation is in preparation. Note that without the proper tools these formats seem difficult, but they allow the development of powerful tools of broad applicability. Most people find it hard to go back to ASCII once they have learned to use some of these applications.

FITS (Solar Data)

The Flexible Image Transport System (FITS) now provides the standard data format used in Astronomy, and thus they maintain the software. The Solar Physics community has adopted this format as well, which enables them to take advantage of all the Astronomical efforts. The primary FITS site provides a wide variety of support. Particularly useful are FITS image viewers. One widely used viewer, available on all platforms, is SAO Image DS9. Many general IDL routines are available from the IDL Astronomy User’s Library; the routine mrdfits.pro provides a useful way to load FITS files into IDL for analysis (be sure to get the “Other ASTRON” routines required also as indicated in the file). For serious analysis of many solar datasets, a widely used (in some cases practically required) set of IDL routines is provided by SolarSoft. Although it requires some work by the user to get going, this package provides a very rich set of reading (often direct from the Web) and analysis routines, and it is continually updated by the solar community.

CDF (mostly Space Physics)

Judging by recent mission efforts, the NASA Space Physics community has adopted the ISTP Common Data Format (ISTP CDF) as its standard for public data access. It is the format used by the primary access route to the Final Archive at the Space Physics Data Facility (CDAWeb), which contains much of the data from current and past Space Physics missions. The SPDF developed and maintains CDF, and works with providers to assure new data products are accommodated. CDF files are available from various missions (e.g., Cluster) as well as from SPDF, either through simple http or ftp transfer, through CDAWeb, or through Web Services. Any CDF files that are in CDAWeb already can be plotted, downloaded, and converted to ASCII with the CDAWeb interface (//cdaweb.gsfc.nasa.gov). This is often a good place to start; in addition to previews of a given dataset, many sets can be combined into one displayed graph, thus allowing quick qualitative correlative analysis.

Making graphs of individual CDF files, whether on your machine or on the web, is easy: use the open source application Autoplot. You can download a file (in many formats), launch the application from the internet or locally (once downloaded), and open the file, specifying a local or remote address. A folder icon at the right of the header bar allows the user to pick variables to plot either on one set of axes or on multiple panels. Files may be referenced directly from the internet. Autoplot has many other capabilities, including making sets of png overview files and combining plots from multiple instruments and missions; see the Autoplot Website for details.

Using CDF with IDL

The “IDL” commercial data analysis application contains a number of routines to read and make CDF files. SPDF also has a number of additional routines that are not distributed with the standard IDL releases. The easiest way to get these is to download the whole library as a “saveset” called spdfcdas.sav. This can then be brought into IDL with the command:

> restore, ‘/your_IDL_command_path/spdfcdas.sav’

Some of the routines available at that point are documented here, one of the most useful being ”read_mycdf” that will bring an entire CDF file from a local disk into IDL as a “structure.” The equivalent command to bring CDAWeb data directly from the Web into IDL is ”spdfgetdata.”

Loading data from CDAWeb into IDL with specified variables and cadence

IDL CDF tools can be used to read CDAWeb data varibles over the Internet directly into IDL. The variables can be renamed, despiked, rebinned, and interpolated in one command. To use this capability, just download the save file with all the required code as well as the zip file of demos. Unzip the demos, put everything in a location in your IDL path, and start up IDL (it helps with path troubles to start fresh). Type at the prompt:

> .r demo_ace_2sets

Within a short time you should see a plot of four time series from three ACE instruments (MAG, SWEPAM, and SWICS). The IDL session reports the variables that are produced along with other information. A second demo, demo_polar_3sets, shows how more complex data can be read in and plotted. The demo files can be modified, and there is a separate page with a listing of creaders that provides some potentially useful cases. A set of nearly all the individual creaders, identified by the CDAWeb ID, can be dowloaded as a zip file. More details about this access to CDAWeb CDF data, including how to write your own creaders, can be found here. Note that versions of IDL prior to 7.1 will not work with these routines, and 8.2.1+ is highly recommended. Some of the most recent CDF files use a time format not understood by the older IDL versions, although a patch is available to fix this. Please report any bugs. If you wish source code, the programs can be retrieved along with demos and example files by downloading a zip file containing all the routines. This includes the required subset of the above CDAWeb IDL library of routines.

IDL Routines of Note

Various sets of IDL routines are of potentially great general interest to IDL users of Heliophysics data. The first is David Fanning’s ”Coyote Library” of routines that make IDL do what you often hoped it would, such as making the default for graphics be black lines on white and allowing a line color choice by simply stating “color = ‘red’.” Fanning provides a wonderful array of routines that make life easier.

Another set of free IDL routines that can be helpful are provided by the Astronomy IDL library mentioned above. For example, “multiplot.pro” provides a simple way of making multipanel plots with easily controllable space between plots, and many options for labels and other features.

There is a set of routines that has been developed in conjunction with a number of missions (Cluster, Wind, Polar, FAST, and THEMIS) called the Space Physics Environment Data Analysis Software (SPEDAS). It is useful for much more than just the original target datasets, and it now provides access to much of CDAWeb. The basic procedure is to download the set of routines to a place in your path, and then use commands from the set to load and process variables and make plots with both command line and gui input. The main web page points to many videos, a User Manual, and very helpful “crib sheets.” The routines include a capable plotting GUI and the abilitiy to make such things as power spectra of time series and cuts of particle distribution functions. This Suite is becoming more poplular, and is in use by many groups such as some of the Van Allen Probes teams. The creader routines mentioned above can be used to read CDAWeb variables into SPEDAS using the “/tdas” (an older name for the software) flag.