  Mapserver + NEXRAD Composite HOWTO
  Daryl Herzmann


  This HOWTO explains how to create "unprojected" geo-referenced NEXRAD
  imagery ready for use within GIS.  As an example,  Mapserver
  <http://mapserver.gis.umn.edu> is used to illustrate web mapping with
  this raster file.  Updated on 28 Jul 2004 with various fixes to make
  the document current.
  ______________________________________________________________________

  Table of Contents


  1. Introduction
     1.1 411 on abbreviations
     1.2 Reading this HOWTO
     1.3 The Quick and Dirty version
     1.4 Links

  2. Requirements
     2.1 Software and Data Access
     2.2 Hardware
     2.3 System Environment

  3. Building the composites with GEMPAK
     3.1 Building nex2img
     3.2 Scripting the generation of the composite
     3.3 Geo-referencing the composite image
        3.3.1 Naming the world file
     3.4 Lets pause for a second

  4. Building Proj.4 & Mapserver
     4.1 Building Proj.4
     4.2 Building Mapserver
     4.3 Lets pause again!

  5. Building your Web Mapping Application
     5.1 Getting Started
     5.2 The Mapserver .map file

  6. GIS-Ready! NEXRAD information
     6.1 Iowa Environmental Mesonet
        6.1.1 Scripting an automated download

  7. Conclusions
     7.1 Future Work
     7.2 Contact info


  ______________________________________________________________________

  [1m1.  Introduction[0m

  Almost every weather website on the Internet has some sort of NEXRAD
  data display <http://www.nws.noaa.gov/radar_tab.html>. Unfortunately
  almost all of these sites are only fulfilling a single purpose,
  allowing the user to view an image.  Geographical Information Systems
  (GIS) goes beyond this single purpose model to allow the user to view
  the image, but then also process it within a GIS system.  Processing
  could involve working with the RASTER data, reprojecting the image,
  querying the image, or many other things.

  This HOWTO covers a mechanism to produce geo-referenced NEXRAD imagery
  from GEMPAK
  <http://my.unidata.ucar.edu/content/software/gempak/index.html>. As a
  GIS web mapping example, we use Mapserver
  <http://mapserver.gis.umn.edu> to create a simple, but powerfull web
  application. This HOWTO assumes that you are running a modern version
  of Linux.  Other UNIXs with GNU software <http://www.gnu.org>
  installed should work as well.

  This HOWTO is geared for those in the UNIDATA community
  <http://my.unidata.ucar.edu> with an active Internet Data Distribution
  <http://my.unidata.ucar.edu/content/software/idd/index.html> feed and
  already have GEMPAK installed.  If you don't have either of these, you
  will probably just be interested in the Mapserver sections.  The
  [4mradmapserver-files.tar.gz[24m contains an example RADAR image ready for
  use with Mapserver or any other raster supporting GIS.

  [1m1.1.  411 on abbreviations[0m

  Some of the abbreviations used in this HOWTO.

  IEM       Iowa Environmental Mesonet
  UNIDATA   University meteorological research group
  NEXRAD    Network of Next Generation (WSR-88D) RADAR used by the National Weather Service
  NWS       National Weather Service
  GEMPAK    General Meteorological Application Package. Useful for dealing with NWS produced formats


                              Abbreviations

  [1m1.2.  Reading this HOWTO[0m

  Some of you reading this may have no interest in generating the
  composites, but wish to use the composites.  Currently, I am not aware
  of a group generating this data source for free.  I have started
  generating a national base composite reflectivity as a proof of
  concept.

  [1m1.3.  The Quick and Dirty version[0m

  The files used in this howto are available here:
  <https://mesonet.agron.iastate.edu/docs/radmapserver/radmapserver-
  files.tar.gz>.


  +o  Download and install Proj.4 from
     <http://www.remotesensing.org/proj/> .

  +o  Download and compile Mapserver with proj, TIFF and PNG support from
     <http://mapserver.gis.umn.edu>  . Place the resulting 'mapserv'
     binary in your webserver's CGI-BIN directory.

  +o  Extract the [4mradmapserver-files.tar.gz[24m archive into your web root.

  +o  Point your browser at  <http://your.host.name/cgi-
     bin/mapserv?map=/var/www/htdocs/radmapserver/radmapserver.map&layers=states>
     .

  [1m1.4.  Links[0m


  +o  UNIDATA <http://my.unidata.ucar.edu>

  +o  Mapserver <http://mapserver.gis.umn.edu>

     +o  Mapserver Docs <http://mapserver.gis.umn.edu/docs/>

     +o  Mapserver WIKI <http://mapserver.gis.umn.edu/cgi-bin/wiki.pl>

  [1m2.  Requirements[0m

  [1m2.1.  Software and Data Access[0m

  The following items must be installed and configured before any of
  this HOWTO could potentially work for you.  Describing the
  installation of any of these packages is beyond the scope of this
  document.

  +o  You should be using the Linux operating system, but there is no
     reason this won't work on Solaris or a BSD variant.

  +o  GEMPAK, for converting the NIDS file format into RASTER

  +o  Some mechanism for NIDS data access

     +o  Realtime access via UNIDATA IDD using LDM

     +o  Or FTP access to NIDS files via NOAA

     +o  Or locally available NIDS files on your hard-drive.

  +o  A webserver, which should be Apache.

  +o  ImageMagick: used to [4mconvert[24m the GIF files to TIFF.

  The following items will be installed during this howto.  The
  installation of Proj.4 will require privileged access to your system.
  The Mapserver components will require access to your webserver's CGI-
  BIN directory.

  +o  Proj.4

  +o  Mapserver

  +o  Simple Mapserver application

  [1m2.2.  Hardware[0m

  Hardware requirements is a tricky topic.  I have seen 300 MHz machines
  outperform 1 GHz machines when properly configured.  Common sense
  applies here.  If you want to generate nationwide composites every 5
  minutes, you will need a healthy machine.  If you are supporting a
  small intranet, a modest machine will do.

  For our purposes, Mapserver will not introduce a performance
  bottleneck on your system.  It is extremely fast and robust!  Any
  slowness typically comes when you start reprojecting massive raster
  layers, but we aren't going to do that!

  [1m2.3.  System Environment[0m

  You are going to need root privileges on your system.  So either know
  your root password or be very kind to the system administrator! We
  will be installing Proj.4 into /usr/local, which is a standard place
  to put 3rd party software.  Mapserver will be installed into the cgi-
  bin directory of your webserver.

  [1m3.  Building the composites with GEMPAK[0m

  Probably the easiest component of this HOWTO is building the composite
  image within GEMPAK!  If you are running a version of GEMPAK before
  5.6.j, you will need to download and install the nex2img program.  If
  you already have nex2img, then you can skip the next subsection.


  GEMPAK is publicly available from the UNIDATA website
  <http://my.unidata.ucar.edu/content/software/gempak/index.html>.
  Installing GEMPAK is way outside the scope of this HOWTO.

  Nex2img was written by Steve Chiswell at the UNIDATA Program Center.
  His program is the key to making this whole system work.  Before
  nex2img, it was extremely tricky to get the geo-referencing of
  RASTERs, produced in GEMPAK, correct for use within GIS.

  [1m3.1.  Building nex2img[0m

  Building nex2img is very straightforward with GEMPAK already
  installed.


       ______________________________________________________________________
       $ cd $NAWIPS
       $ wget https://mesonet.agron.iastate.edu/docs/radmapserver/nex2img.tar.gz
       $ tar -zxvf nex2img.tar.gz
       $ cd unidata/programs/nex2img
       $ make clean
       $ make all
       $ make install
       $ make clean
       $ rehash
       $ which nex2img
       ______________________________________________________________________



  With the nex2img program built, you are now ready to generate the
  composites.

  [1m3.2.  Scripting the generation of the composite[0m

  This simple script will crank out an image.  I am specifying a GRDAREA
  for Iowa and extended. You will want to modify for your own needs.
  Keep in mind that this application composites individual site NEXRADs,
  you can't build a nationwide composite without having all the
  individual sites.



  ______________________________________________________________________
  #!/bin/csh

  # Replace to fit your environment!
  source /home/nawips/Gemenviron

  set gtime=`date -u +'%y%m%d/%H%M'`

  rm -f radar.gif radar.tif >& /dev/null

  nex2img << EOF > nex2img.log
   GRDAREA  = 37.00;-104.00;48.99;-87.01
   PROJ     = CED
   KXKY     = 1700;1200
   CPYFIL   =
   GFUNC    = n0r
   RADTIM   = ${gtime}
   RADDUR   = 15
   RADFRQ   =
   STNFIL   = nexrad.tbl
   RADMODE  = P
   RADFIL   = radar.gif
   LUTFIL   = upc_rad24.tbl
   list
   run

   exit
  EOF

  if (-e radar.gif) then
    convert radar.gif radar.tif
    cp radar.tif /var/www/htdocs/radmapserver/gisdata
  endif
  ______________________________________________________________________



  The nex2img program will generate an image called "radar.gif".  In
  order to work with most GIS systems, you will probably want to convert
  this file to another format namely TIFF, but you can choose JPG or
  PNG.  We then place the radar.tif file in the webserver directory for
  later use by mapserver.

  Without getting too long winded, lets discuss image formats.  The
  easiest format for use within GIS is probably TIFF.  Although very
  large, uncompressed tiff files are extremely fast within GIS.  The
  reason is that the application can "seek" the file to only pull needed
  data.  With compressed formats, an entire set of data needs to be
  uncompressed and then processed.  Also, most people will probably try
  loading the image in ESRI's ArcView.  ArcView likes TIFF files and has
  built in support for them.

  [1m3.3.  Geo-referencing the composite image[0m

  Now that we have an image, we need to georeference it.  Georeferencing
  of images is done with a world file.  A world file is simply a file
  with 6 lines of meta data describing the navigation of an image.  The
  format is for example:



  ______________________________________________________________________
  0.010 (size of pixel in x direction)
  0.000000000000 (rotation term for row) *Typically zero
  0.000000000000 (rotation term for column) *Typically zero
  -0.010 (size of pixel in y direction)
  -95.0000 (x coordinate of centre of upper left pixel in map units)
  42.300 (y coordinate of centre of upper left pixel in map units)
  ______________________________________________________________________



  Note that the units of the geo-referencing are in the same units of
  the projection.  Since we are in lat-lon space, the units are decimal
  degrees.  If you were in UTM space, it would probably be meters.  For
  the image generated from my example of nex2img, my world file would
  look like


       ______________________________________________________________________
          0.01000
          0.00000
          0.00000
         -0.01000
       -104.00000
         49.00000
       ______________________________________________________________________



  [1m3.3.1.  Naming the world file[0m

  Depending on your GIS, the naming convention of your world file may be
  different.  Many systems use a generic [4m.wld[24m to describe any world
  file.  Others use variants on the image format nomenclature, for
  instance, the world file for TIF formated files is [4m.tfw[24m and for PNG is
  [4m.pngw[24m. For our purposes, we will use [4m.wld[24m.

  [1m3.4.  Lets pause for a second[0m

  If you have made it this far, you are probably wondering what to do
  with the RASTER image with no political borders or other denotations
  and this silly 6 line world file.  The next section shows an example
  of taking this image and using the powerful Mapserver to produce a Web
  Map application.  Trust me, this is not the only application of this
  data.

  [1m4.  Building Proj.4 & Mapserver[0m

  Like many Open Source packages, Mapserver relies on external libraries
  to handle much of the heavy lifting.  This is both a good thing and a
  bad thing.  The good thing is that the Mapserver folks don't have to
  program all the needed functionality, the bad thing is that you can
  reach dependency-hell with this setup.  For our purposes, we will only
  build with Proj.4 support, but you can rebuild mapserver later to link
  with other libraries for more functionality.

  [1m4.1.  Building Proj.4[0m

  Proj.4 <http://www.remotesensing.org/proj/> provides projection and
  reprojection support for Mapserver.  Building and installing is
  hopefully simple, if you have ever built an open source package
  before.  Make sure that /usr/local/lib is included in /etc/ld.so.conf,
  so that Mapserver can link to that library without special
  configuration.
       ______________________________________________________________________
       $ wget ftp://ftp.remotesensing.org/pub/proj/proj-4.4.8.tar.gz
       $ tar -zxvf proj-4.4.8.tar.gz
       $ cd proj-4.4.8
       $ ./configure; make
       $ su -l
       # make install   #(as root, to install in /usr/local!)
       # /sbin/ldconfig
       # exit
       $ make clean
       ______________________________________________________________________



  If everything went well, you should notice a libproj in /usr/local/lib
  that mapserver can link to.

  [1m4.2.  Building Mapserver[0m

  As I write this (28 Jul 2004), Mapserver is at version 4.2.1 .  It is
  preferable that you be running this version for this HOWTO, but it is
  not required.  For our purposes, the only difference between the
  current version 4.2.1 and the old 3.6.x versions is the 'OFFSITE'
  keyword in the .map file.  In Mapserver 4.x, this is a RGB value and
  in Mapserver 3.x, this is a color map index value.


       ______________________________________________________________________
       $ wget http://cvs.gis.umn.edu/dist/mapserver-4.2.1.tar.gz
       $ tar -zxvf mapserver-4.2.1.tar.gz
       $ cd mapserver-4.2.1
       $ ./configure --with-proj --with-png
       $ make
       $ cp -i mapserv /var/www/cgi-bin/mapserv
       ______________________________________________________________________



  Now, if the build process worked, you should have a binary called
  'mapserv' if your current working directory.  Verify that the mapserv
  binary is okay by executing


       ______________________________________________________________________
       $ ./mapserv -v
       ______________________________________________________________________



  This command will spit out the capabilities of Mapserver.  For this
  demo, we need support for SHAPEFILES and the TIFF file format.  The
  output from the above command should contain [4mOUTPUT=PNG[24m [4mINPUT=TIFF[0m
  [4mINPUT=SHAPEFILE[24m.

  [1m4.3.  Lets pause again![0m

  If you got this far with these instructions, you are much smarter than
  I am.  Since I couldn't even get this to work following my own
  instructions.  Anyway, we now have a CGI application that we can
  install on our webserver and begin building a web mapping application!
  Yeah!


  [1m5.  Building your Web Mapping Application[0m

  [1m5.1.  Getting Started[0m

  It is somewhat hard to keep this generic, when everyone does things
  differently.  I will assume a base RPM install of apache on Red Hat
  Linux.  If your environment is different, you should know how it is
  different!

  Assuming that your web directory root is [4m/var/www/htdocs[24m and that your
  CGI-BIN directory is [4m/var/www/cgi-bin[24m, you will want to issue the
  following commands as a privileged user to these directories.


       ______________________________________________________________________
       $ cd /tmp
       $ wget https://mesonet.agron.iastate.edu/docs/radmapserver/radmapserver-files.tar.gz
       $ cd /var/www/htdocs
       $ mkdir mstmp
       $ chown apache mstmp
       $ tar -xzvf /tmp/radmapserver-files.tar.gz
       ______________________________________________________________________



  The mstmp directory is needed by mapserver to write temporary files
  to.  The radmapserver directory is where we will build our application
  interface at.

  [1m5.2.  The Mapserver .map file[0m

  Fundamental to the Mapserver application, is a .map configuration
  file.  For our demo, we will build a simple map file called
  radmapserver.map in our [4m/var/www/htdocs/radmapserver[24m directory.  The
  file is as follows.



  ______________________________________________________________________
  # Start of Mapfile (radmapserver.map)
  NAME radmapserver
  STATUS ON
  SIZE 450 350
  EXTENT -104 37 -87 49
  UNITS DD
  IMAGETYPE PNG

  # Map projection definition
  PROJECTION
    "proj=epsg:4326"
  END


  # Map interface definition
  WEB
    LOG "/var/www/htdocs/mstmp/radmapserver.log"
    TEMPLATE radmapserver.html
    IMAGEPATH "/var/www/htdocs/mstmp/"
    IMAGEURL "/mstmp/"
  END

  # Our Geo-referenced RADAR layer
  LAYER
    TYPE RASTER
    STATUS ON
    NAME radar
    DATA "/var/www/htdocs/radmapserver/gisdata/radar.tif"
    PROJECTION
     "proj=epsg:4326"
    END
    # For Mapserver 3.x, we would use
    # OFFSITE 0
    OFFSITE 0 0 0
  END

  # Our states shapefile, just to get some political boundaries
  LAYER
    TYPE POLYGON
    STATUS ON
    NAME states
    DATA "/var/www/htdocs/radmapserver/gisdata/states.shp"
    PROJECTION
      "proj=epsg:4326"
    END
    CLASS
      OUTLINECOLOR 255 0 0
    END
  END

  END # End of mapfile radmapserver.map
  ______________________________________________________________________



  In the Web definition of our mapfile, we referenced a file called
  [4mradmapserver.html[24m.  This file is a HTML template mapserver uses to
  generate an interface.  Our example radmapserver.html file is as
  follows.



  ______________________________________________________________________
  <html>
  <head>
    <title>RadMapserver Test</title>
  </head>

  <form method="GET" action="/cgi-bin/mapserv" name="mapserv">
  <input type="hidden" value="[mapext]" name="imgext">
  <!-- imgxy is set to half of your display image size -->
  <input type="hidden" value="225 175" name="imgxy">
  <input type="hidden" value="[map]" name="map">
  <input type="hidden" value="browse" name="mode">

  <table border="1">
  <tr><td>

  <input name="img" type="image" src="[img]" border="0">

  </td><td>

  <p> <b> Select layers to display </b>
  <br><input type="checkbox" value="radar" name="layer" [radar_check] > RADAR layer
  <br><input type="checkbox" value="states" name="layer" [states_check] > States layer

  <p>  Set your zoom option:
  <br>
  <select name="zoom" size="1">
    <option value="4" [zoom_4_select]> Zoom in 4 times
    <option value="3" [zoom_3_select]> Zoom in 3 times
    <option value="2" [zoom_2_select]> Zoom in 2 times
    <option value="1" [zoom_1_select]> Recenter Map
    <option value="-2" [zoom_-2_select]> Zoom out 2 times
    <option value="-3" [zoom_-3_select]> Zoom out 3 times
    <option value="-4" [zoom_-4_select]> Zoom out 4 times
  </select>

  <p>
  <input type="submit" value="Make Map!">
  <form>
  </td></tr></table>

  <html>
  ______________________________________________________________________



  Believe it or not, we are ready to rock and roll!  Get your favorite
  Open Source web browser loaded up and point it at your website with
  the following URL

  ______________________________________________________________________
  http://your.web.site/cgi-bin/mapserv?map=/var/www/htdocs/radmapserver/radmapserver.map&layers=states
  ______________________________________________________________________



  [1m6.  GIS-Ready! NEXRAD information[0m

  This section contains links to folks already generating GIS-Ready
  products from NEXRAD data.

  [1m6.1.  Iowa Environmental Mesonet[0m

  The IEM is generating a nationwide composite of base reflectivity
  every 5 minutes.  This product is available in this directory:
  ______________________________________________________________________
  https://mesonet.agron.iastate.edu/data/gis/images/unproj/USCOMP/
  ______________________________________________________________________


  The IEM is generating a PNG and TIFF formatted image.  There is also a
  compressed TIFF image available in that directory as well.  The file
  prefix [4mn0r_0[24m contains the current data.

  [1m6.1.1.  Scripting an automated download[0m

  Here is an example script which would allow you to automatically
  download files from the IEM website and then save them to a local
  directory


       ______________________________________________________________________
       #!/bin/csh

       set DATADIR="/set/me/to/something"

       cd /tmp

       wget -q https://mesonet.agron.iastate.edu/data/gis/images/unproj/USCOMP/n0r_0.tif.Z
       uncompress n0r_0.tif.Z

       cd ${DATADIR}

       foreach i (9 8 7 6 5 4 3 2 1 0)
         cp -f n0r_${num}.tif n0r_`echo ${num} + 1 | bc`.tif
       end

       mv /tmp/n0r_0.tif n0r_0.tif

       exit
       ______________________________________________________________________



  [1m7.  Conclusions[0m

  This HOWTO demonstrated a very primitive component of GIS, web
  mapping.  There is just so much more that can be done.  For those of
  you familiar with PHP, Mapserver has an outstanding PHP interface to
  the Mapserver API.  You can create some of the neatest applications
  using the Mapserver API and the power of PHP.  The Iowa Environmental
  Mesonet has implemented most of its Mapserver interfaces using the PHP
  mapscript module.  The source code for these applications is licensed
  under the GPL, but I just don't place the code on the website to
  download.  Just email me if you would like a copy...

  [1m7.1.  Future Work[0m

  I would suspect that most people will not have the ability to generate
  nationwide NEXRAD composites.  One of the neat things Mapserver can do
  is act as a Web Map Service (WMS) server and client.  Potentially a
  WMS server could be set up somewhere on the Internet and then your
  mapserver could make a remote WMS call to it get the image.

  Another option, will be for either UNIDATA, the IEM, or somebody else
  to generate the nationwide composite (in 'unprojected' coordinates)
  and make that available on the IDD.  This is probably the easiest way
  to get this data to the community.


  Although never covered in this HOWTO, timestamps are a big issue with
  this system.  I have built hacks using DBF files in order to get
  timestamps on the generated images.  Here is an example of output
  using this 'hack'.  At some point, I will add this hack to the HOWTO.

       <https://mesonet.agron.iastate.edu/data/mwcomp.png>


  [1m7.2.  Contact info[0m

  If you have questions, shoot me an email (akrherz@iastate.edu) or give
  me a call (515.294.5978).  Good luck!



