<!DOCTYPE linuxdoc system>
<article>

<title>Mapserver + NEXRAD Composite HOWTO</title>
<author>
  <name>Daryl Herzmann</name>
</author>

<abstract>
This HOWTO explains how to create "unprojected" geo-referenced NEXRAD
imagery ready for use within GIS.  As an example,  <url url="http://mapserver.gis.umn.edu" name="Mapserver"> is used to illustrate web mapping with this 
raster file.

Updated on 28 Jul 2004 with various fixes to make the document current.
</abstract>

<toc>

<sect>
 <heading>Introduction</heading>
<p>
Almost every weather website on the Internet has some sort of <url url="http://www.nws.noaa.gov/radar_tab.html" name="NEXRAD data display">. Unfortunately almost all of these sites are only fulfilling a single purpose, allowing the user to view an image.  Geographical Information Systems (GIS) goes beyond this single purpose model to allow the user to view the image, but then also process it within a GIS system.  Processing could involve working with the RASTER data, reprojecting the image, querying the image, or many other things.
</p>

<p>
This HOWTO covers a mechanism to produce geo-referenced NEXRAD imagery from <url url="http://my.unidata.ucar.edu/content/software/gempak/index.html" name="GEMPAK">. As a GIS web mapping example, we use <url url="http://mapserver.gis.umn.edu" name="Mapserver"> to create a simple, but
powerfull web application. This HOWTO assumes that you are running a 
modern version of Linux.  Other UNIXs with <url url="http://www.gnu.org" name="GNU software"> installed 
should work as well.
</p>

<p>
This HOWTO is geared for those in the <url url="http://my.unidata.ucar.edu" name="UNIDATA community"> with an active <url url="http://my.unidata.ucar.edu/content/software/idd/index.html" name="Internet
Data Distribution"> feed and already have GEMPAK installed.  If you don't have 
either of these, you will probably just be interested in the Mapserver sections.
The <it>radmapserver-files.tar.gz</it> contains an example RADAR image ready
for use with Mapserver or any other raster supporting GIS.
</p>

<sect1>411 on abbreviations
<p>
Some of the abbreviations used in this HOWTO.
<table>
<tabular ca="ll">
IEM|Iowa Environmental Mesonet@
UNIDATA|University meteorological research group@
NEXRAD|Network of Next Generation (WSR-88D) RADAR used by the National Weather Service@
NWS|National Weather Service@
GEMPAK|General Meteorological Application Package.  Useful for dealing with NWS produced formats
</tabular>
<caption>Abbreviations</caption>
</table>
</p>
</sect1>

<sect1>Reading this HOWTO
<p>
Some of you reading this may have no interest in generating the composites, 
but wish to use the composites.  Currently, I am not aware of a group 
generating this data source for free.  I have started generating a national
base composite reflectivity as a proof of concept.
</p>
</sect1>

<sect1>The Quick and Dirty version
<p>
The files used in this howto are available here: <url url="https://mesonet.agron.iastate.edu/docs/radmapserver/radmapserver-files.tar.gz">.
</p>
<p>
<itemize>
<item>Download and install Proj.4 from <url url="http://www.remotesensing.org/proj/"> .
<item>Download and compile Mapserver with proj, TIFF and PNG support from <url url="http://mapserver.gis.umn.edu">  . Place the resulting 'mapserv' binary in your
webserver's CGI-BIN directory.
<item> Extract the <it>radmapserver-files.tar.gz</it> archive into your web root.
<item> Point your browser at <url url="http://your.host.name/cgi-bin/mapserv?map=/var/www/htdocs/radmapserver/radmapserver.map&amp;layers=states"> .
</itemize>
</p>
</sect1>

<sect1>Links
<p>
<itemize>
  <item><url url="http://my.unidata.ucar.edu" name="UNIDATA">
  <item><url url="http://mapserver.gis.umn.edu" name="Mapserver">
  <itemize>
    <item><url url="http://mapserver.gis.umn.edu/docs/" name="Mapserver Docs">
    <item><url url="http://mapserver.gis.umn.edu/cgi-bin/wiki.pl" name="Mapserver WIKI">
  </itemize>
</itemize>
</p>
</sect1>

</sect>
<!-- Requirements -->
<sect>
 <heading>Requirements</heading>
<sect1>Software and Data Access
<p>
The following items must be installed and configured before any of this HOWTO
could potentially work for you.  Describing the installation of any of these 
packages is beyond the scope of this document.
<itemize>
 <item> You should be using the Linux operating system, but there is no reason
 this won't work on Solaris or a BSD variant.
 <item> GEMPAK, for converting the NIDS file format into RASTER
 <item> Some mechanism for NIDS data access
 <itemize>
   <item>Realtime access via UNIDATA IDD using LDM
   <item>Or FTP access to NIDS files via NOAA
   <item>Or locally available NIDS files on your hard-drive.
 </itemize>
 <item> A webserver, which should be Apache.
 <item> ImageMagick: used to <it>convert</it> the GIF files to TIFF.
</itemize>
</p>

<p>
The following items will be installed during this howto.  The installation of
Proj.4 will require privileged access to your system.  The Mapserver components
will require access to your webserver's CGI-BIN directory. 
<itemize>
 <item>Proj.4
 <item>Mapserver
 <item>Simple Mapserver application
</itemize>
</p>

<sect1>Hardware
<p>
Hardware requirements is a tricky topic.  I have seen 300 MHz machines outperform 1 GHz machines when properly configured.  Common sense applies here.  If you want to generate nationwide composites every 5 minutes, you will need a healthy machine.  If you are supporting a small intranet, a modest machine will do. 
</p>

<p>
For our purposes, Mapserver will not introduce a performance bottleneck
on your system.  It is extremely fast and robust!  Any slowness typically comes
when you start reprojecting massive raster layers, but we aren't going to 
do that!
</p>

<sect1>System Environment
<p>
You are going to need root privileges on your system.  So either know your 
root password or be very kind to the system administrator! We will be installing
Proj.4 into /usr/local, which is a standard place to put 3rd party software.
Mapserver will be installed into the cgi-bin directory of your webserver.
</p>

</sect>

<!-- Building the composites -->
<sect>
 <heading>Building the composites with GEMPAK</heading>
<p>
Probably the easiest component of this HOWTO is building the composite 
image within GEMPAK!  If you are running a version of GEMPAK before 5.6.j, you
will need to download and install the nex2img program.  If you already have
nex2img, then you can skip the next subsection.
</p>

<p>
GEMPAK is publicly available from the <url url="http://my.unidata.ucar.edu/content/software/gempak/index.html" name="UNIDATA website">.  Installing GEMPAK 
is way outside the scope of this HOWTO.
</p>

<p>
Nex2img was written by Steve Chiswell at the UNIDATA Program Center.  His 
program is the key to making this whole system work.  Before nex2img, it was
extremely tricky to get the geo-referencing of RASTERs, produced in GEMPAK, correct for use 
within GIS.
</p>

<sect1>Building nex2img
<p>
Building nex2img is very straightforward with GEMPAK already installed.
<tscreen><code>
$ cd $NAWIPS
$ wget https://mesonet.agron.iastate.edu/docs/radmapserver/nex2img.tar.gz
$ tar -zxvf nex2img.tar.gz
$ cd unidata/programs/nex2img
$ make clean
$ make all
$ make install
$ make clean
$ rehash
$ which nex2img
</code></tscreen>
</p>

<p>
With the nex2img program built, you are now ready to generate the composites.
</p>

<sect1>Scripting the generation of the composite
<p>
This simple script will crank out an image.  I am specifying a GRDAREA for
Iowa and extended. You will want to modify for your own needs.  Keep in mind
that this application composites individual site NEXRADs,  you can't build a nationwide 
composite without having all the individual sites. 
<tscreen><code>
#!/bin/csh

# Replace to fit your environment!
source /home/nawips/Gemenviron

set gtime=`date -u +'%y%m%d/%H%M'`

rm -f radar.gif radar.tif >& /dev/null

nex2img << EOF > nex2img.log
 GRDAREA  = 37.00;-104.00;48.99;-87.01
 PROJ     = CED
 KXKY     = 1700;1200
 CPYFIL   =  
 GFUNC    = n0r
 RADTIM   = ${gtime}
 RADDUR   = 15
 RADFRQ   = 
 STNFIL   = nexrad.tbl
 RADMODE  = P
 RADFIL   = radar.gif
 LUTFIL   = upc_rad24.tbl
 list
 run

 exit
EOF

if (-e radar.gif) then
  convert radar.gif radar.tif
  cp radar.tif /var/www/htdocs/radmapserver/gisdata
endif

</code></tscreen>
</p>

<p>
The nex2img program will generate an image called "radar.gif".  In order to 
work with most GIS systems, you will probably want to convert this file
to another format namely TIFF, but you can choose JPG or PNG.  We then place
the radar.tif file in the webserver directory for later use by mapserver.
</p>

<p>
Without getting too long winded, lets discuss image formats.  The easiest 
format for use within GIS is probably TIFF.  Although very large, uncompressed
tiff files are extremely fast within GIS.  The reason is that the application 
can "seek" the file to only pull needed data.  With compressed formats, an
entire set of data needs to be uncompressed and then processed.  Also, most 
people will probably try loading the image in ESRI's ArcView.  ArcView likes
TIFF files and has built in support for them.
</p>

<sect1>Geo-referencing the composite image
<p>
Now that we have an image, we need to georeference it.  Georeferencing of 
images is done with a world file.  A world file is simply a file with 6 lines
of meta data describing the navigation of an image.  The format is for 
example:
<tscreen><code>
0.010 (size of pixel in x direction)
0.000000000000 (rotation term for row) *Typically zero
0.000000000000 (rotation term for column) *Typically zero
-0.010 (size of pixel in y direction)
-95.0000 (x coordinate of centre of upper left pixel in map units)
42.300 (y coordinate of centre of upper left pixel in map units) 
</code></tscreen>
Note that the units of the geo-referencing are in the same units of the
projection.  Since we are in lat-lon space, the units are decimal degrees.
If you were in UTM space, it would probably be meters.
For the image generated from my example of nex2img, my world file 
would look like
<tscreen><code>
   0.01000
   0.00000
   0.00000
  -0.01000
-104.00000
  49.00000
</code></tscreen>
</p>

<sect2>Naming the world file
<p>
Depending on your GIS, the naming convention of your world file may be 
different.  Many systems use a generic <it>.wld</it> to describe any world 
file.  Others use variants on the image format nomenclature, for instance, the world file
for TIF formated files is <it>.tfw</it> and for PNG is <it>.pngw</it>. For our
purposes, we will use <it>.wld</it>.
</p>
</sect2>

</sect1>

<sect1>Lets pause for a second
<p>
If you have made it this far, you are probably wondering what to do with the
RASTER image with no political borders or other denotations and this silly
6 line world file.  The next section shows an example of taking this image
and using the powerful Mapserver to produce a Web Map application.  Trust me, 
this is not the only application of this data.
</p>

</sect1>

</sect>

<!-- Building Mapserver -->
<sect>
  <heading>Building Proj.4 & Mapserver</heading>

<p>
Like many Open Source packages, Mapserver relies on external libraries to handle
much of the heavy lifting.  This is both a good thing and a bad thing.  The good thing is that the Mapserver folks don't have to program all the needed functionality, the bad thing is that you can reach dependency-hell with this setup.  For our purposes, we will only build with Proj.4 support, but you can rebuild mapserver later 
to link with other libraries for more functionality.
</p>

<sect1>Building Proj.4
<p>
 <url url="http://www.remotesensing.org/proj/" name="Proj.4"> provides projection and reprojection support for Mapserver.  Building and installing is hopefully
simple, if you have ever built an open source package before.  Make sure
that /usr/local/lib is included in /etc/ld.so.conf, so that Mapserver can
link to that library without special configuration.
<tscreen><code>
$ wget ftp://ftp.remotesensing.org/pub/proj/proj-4.4.8.tar.gz
$ tar -zxvf proj-4.4.8.tar.gz
$ cd proj-4.4.8
$ ./configure; make
$ su -l
# make install   #(as root, to install in /usr/local!)
# /sbin/ldconfig
# exit
$ make clean
</code></tscreen>
</p>

<p>
If everything went well, you should notice a libproj in /usr/local/lib that 
mapserver can link to.
</sect1>

<sect1>Building Mapserver
<p>
As I write this (28 Jul 2004), Mapserver is at version 4.2.1 .  It is preferable that you be running this version for this HOWTO, but it is not required.
For our purposes, the
only difference between the current version 4.2.1 and the old 3.6.x versions
is the 'OFFSITE' keyword in the .map file.  In Mapserver 4.x, this is a RGB value and in Mapserver 3.x, this is a color map index value. 
<tscreen><code>
$ wget http://cvs.gis.umn.edu/dist/mapserver-4.2.1.tar.gz
$ tar -zxvf mapserver-4.2.1.tar.gz
$ cd mapserver-4.2.1
$ ./configure --with-proj --with-png 
$ make
$ cp -i mapserv /var/www/cgi-bin/mapserv
</code></tscreen>
</p>

<p>
Now, if the build process worked, you should have a binary called 'mapserv'
if your current working directory.  Verify that the mapserv binary is okay 
by executing
<tscreen><code>
$ ./mapserv -v
</code></tscreen>
This command will spit out the capabilities of Mapserver.  For this demo, 
we need support for SHAPEFILES and the TIFF file format.  The output from the
above command should contain <it>OUTPUT=PNG INPUT=TIFF INPUT=SHAPEFILE</it>.
</p>
</sect1>

<sect1>Lets pause again!
<p>
If you got this far with these instructions, you are much smarter than I am. 
Since I couldn't even get this to work following my own instructions.  Anyway,
we now have a CGI application that we can install on our webserver and begin 
building a web mapping application!  Yeah!
</p>
</sect1>


<sect>Building your Web Mapping Application
<sect1>Getting Started
<p>
It is somewhat hard to keep this generic, when everyone does things differently.
I will assume a base RPM install of apache on Red Hat Linux.  If your 
environment is different, you should know how it is different!
</p>

<p>
Assuming that your web directory root is <it>/var/www/htdocs</it> and that
your CGI-BIN directory is <it>/var/www/cgi-bin</it>, you will want to issue the
following commands as a privileged user to these directories.
<tscreen><code>
$ cd /tmp
$ wget https://mesonet.agron.iastate.edu/docs/radmapserver/radmapserver-files.tar.gz
$ cd /var/www/htdocs
$ mkdir mstmp
$ chown apache mstmp
$ tar -xzvf /tmp/radmapserver-files.tar.gz 
</code></tscreen>
The mstmp directory is needed by mapserver to write temporary files to.  The
radmapserver directory is where we will build our application interface at.
</p>

</sect1>

<sect1>The Mapserver .map file
<p>
Fundamental to the Mapserver application, is a .map configuration file.  For 
our demo, we will build a simple map file called radmapserver.map in our
<it>/var/www/htdocs/radmapserver</it> directory.  The file is as follows.
<tscreen><code>
# Start of Mapfile (radmapserver.map)
NAME radmapserver
STATUS ON
SIZE 450 350
EXTENT -104 37 -87 49
UNITS DD
IMAGETYPE PNG

# Map projection definition
PROJECTION
  "proj=epsg:4326"
END


# Map interface definition
WEB
  LOG "/var/www/htdocs/mstmp/radmapserver.log"
  TEMPLATE radmapserver.html
  IMAGEPATH "/var/www/htdocs/mstmp/"
  IMAGEURL "/mstmp/"
END

# Our Geo-referenced RADAR layer
LAYER
  TYPE RASTER
  STATUS ON
  NAME radar
  DATA "/var/www/htdocs/radmapserver/gisdata/radar.tif"
  PROJECTION
   "proj=epsg:4326"
  END
  # For Mapserver 3.x, we would use
  # OFFSITE 0
  OFFSITE 0 0 0
END

# Our states shapefile, just to get some political boundaries
LAYER
  TYPE POLYGON
  STATUS ON
  NAME states
  DATA "/var/www/htdocs/radmapserver/gisdata/states.shp"
  PROJECTION
    "proj=epsg:4326"
  END
  CLASS
    OUTLINECOLOR 255 0 0
  END
END

END # End of mapfile radmapserver.map
</code></tscreen>
</p>

<p>
In the Web definition of our mapfile, we referenced a file called <it>radmapserver.html</it>.
This file is a HTML template mapserver uses to generate an interface.  Our
example radmapserver.html file is as follows.
<tscreen><code>
&lt;html&gt;
&lt;head&gt;
  &lt;title&gt;RadMapserver Test&lt;/title&gt;
&lt;/head&gt;

&lt;form method="GET" action="/cgi-bin/mapserv" name="mapserv"&gt;
&lt;input type="hidden" value="[mapext]" name="imgext"&gt;
&lt;!-- imgxy is set to half of your display image size --&gt;
&lt;input type="hidden" value="225 175" name="imgxy"&gt;
&lt;input type="hidden" value="[map]" name="map"&gt;
&lt;input type="hidden" value="browse" name="mode"&gt;

&lt;table border="1"&gt;
&lt;tr&gt;&lt;td&gt;

&lt;input name="img" type="image" src="[img]" border="0"&gt;

&lt;/td&gt;&lt;td&gt;

&lt;p&gt; &lt;b&gt; Select layers to display &lt;/b&gt; 
&lt;br&gt;&lt;input type="checkbox" value="radar" name="layer" [radar_check] &gt; RADAR layer
&lt;br&gt;&lt;input type="checkbox" value="states" name="layer" [states_check] &gt; States layer

&lt;p&gt;  Set your zoom option:
&lt;br&gt;  
&lt;select name="zoom" size="1"&gt;  
  &lt;option value="4" [zoom_4_select]&gt; Zoom in 4 times 
  &lt;option value="3" [zoom_3_select]&gt; Zoom in 3 times 
  &lt;option value="2" [zoom_2_select]&gt; Zoom in 2 times 
  &lt;option value="1" [zoom_1_select]&gt; Recenter Map 
  &lt;option value="-2" [zoom_-2_select]&gt; Zoom out 2 times 
  &lt;option value="-3" [zoom_-3_select]&gt; Zoom out 3 times 
  &lt;option value="-4" [zoom_-4_select]&gt; Zoom out 4 times 
&lt;/select&gt;

&lt;p&gt;
&lt;input type="submit" value="Make Map!"&gt;
&lt;form&gt;
&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;

&lt;html&gt;
</code></tscreen>
</p>

<p>
Believe it or not, we are ready to rock and roll!  Get your favorite Open Source
web browser loaded up and point it at your website with the following URL
<code>http://your.web.site/cgi-bin/mapserv?map=/var/www/htdocs/radmapserver/radmapserver.map&amp;layers=states</code>
</p>
</sect>

<sect>GIS-Ready! NEXRAD information
<p>
This section contains links to folks already generating GIS-Ready products 
from NEXRAD data.
</p>

<sect1>Iowa Environmental Mesonet
<p>
The IEM is generating a nationwide composite of base reflectivity every 5
minutes.  This product is available in this directory:
<code>https://mesonet.agron.iastate.edu/data/gis/images/unproj/USCOMP/
</code>
The IEM is generating a PNG and TIFF formatted image.  There is also a 
compressed TIFF image available in that directory as well.  The file prefix
<it>n0r_0</it> contains the current data.
</p>

<sect2>Scripting an automated download
<p>
Here is an example script which would allow you to automatically download files
from the IEM website and then save them to a local directory
<tscreen><code>
#!/bin/csh

set DATADIR="/set/me/to/something"

cd /tmp

wget -q https://mesonet.agron.iastate.edu/data/gis/images/unproj/USCOMP/n0r_0.tif.Z
uncompress n0r_0.tif.Z

cd ${DATADIR}

foreach i (9 8 7 6 5 4 3 2 1 0)
  cp -f n0r_${num}.tif n0r_`echo ${num} + 1 | bc`.tif
end

mv /tmp/n0r_0.tif n0r_0.tif

exit
</code></tscreen>
</p>
</sect2>

</sect1>

</sect>

<sect>Conclusions 
<p>
This HOWTO demonstrated a very primitive component of GIS, web mapping.  There
is just so much more that can be done.  For those of you familiar with PHP, 
Mapserver has an outstanding PHP interface to the Mapserver API.  You can 
create some of the neatest applications using the Mapserver API and the power
of PHP.  The Iowa Environmental Mesonet has implemented most of its Mapserver
interfaces using the PHP mapscript module.  The source code for these applications is licensed under the GPL, but I just don't place the code on the website to download.  Just
email me if you would like a copy...
</p>

<sect1>Future Work
<p>
I would suspect that most people will not have the ability to generate nationwide NEXRAD composites.  One of the neat things Mapserver can do is act as a 
Web Map Service (WMS) server and client.  Potentially a WMS server could be
set up somewhere on the Internet and then your mapserver could make a remote
WMS call to it get the image.
</p>
<p>
Another option, will be for either UNIDATA, the IEM, or somebody else to generate the nationwide composite (in 'unprojected' coordinates) and make that available on the IDD.  This is probably
the easiest way to get this data to the community. 
</p>
<p>
Although never covered in this HOWTO, timestamps are a big issue with this 
system.  I have built hacks using DBF files in order to get timestamps on the 
generated images.  Here is an example of output using this 'hack'.  At some 
point, I will add this hack to the HOWTO.
<tscreen>
<url url="https://mesonet.agron.iastate.edu/data/mwcomp.png">
</tscreen>
</p>
</sect1>

<sect1>Contact info
<p>
If you have questions, shoot me an email (akrherz@iastate.edu) or give me a call (515.294.5978).  Good luck!
</p>
</sect1>

</sect>

</article>
