4.0 Ingesting Data Sets
The Ingest Engine (IE) provides the functions necessary to perform ingest and archive of user provided data sets. The process of ingest is complex and care taken during user entry is crucial for success. The first choice is which data type is subject to ingest. Currently, with MyEarth v1.0, there are two data types available: Hierarchical Data Format (HDF) Version 4 {hdf4} and ESRI Shapefile {shp}. The process is to provide a data set, with the file extension .hdf, or .shp, (and the accompanying .shx and .dbf files) and a browse image file. For HDF this is usually a .jpg image file. For ESRI shapefile, a .kml file will be required.
In MyEarthAdmin, select the “IE” pane and enter the appropriate data set filenames, including the full path, and select from the “Pull-Down” menu, either {hdf4, shp}. Then, press the “Ingest” button. The successful ingest will accomplish the placement of the data files in the “Archive” and the update of the spatial database with spatial extent of the data sets. The data may then be available for search and retrieval from the archive at a later time.
Figure 13 – MyEarthAdmin Tool with Ingest Engine Pane visible. IE pane shows a pull-down menu for the user to select File Type (top) and text boxes below for entry of the File Name (middle) and Browse Image (bottom).
Working with File Types
The data types provided with MyEarth are a representative sample set. The HDF file type is a versatile complex file, hierarchically structured to allow a number of data tiles to be included within one file. These are commonly used for satellite data representation, and a number of large collections are available for download from the Internet.
Similarly, the ESRI Shapefile is a commonly used data type, especially popular with GIS communities, where it is considered the standard.
Creating Browse Images
Browse images are necessary for the Visualization Environment, since they provide a visual representation of the data set. A proprietary imaging program is needed to generate the browse image, or alternatively the browse images accompany the data set when it is issued by the data provider. When creating a browse image, consider the projection it will be generated in. Orthographic, or Miller projections work well with the Google Earth visualization environment. If you notice small geo-referencing errors, when the browse images are retrieved from the archive, this will be due to the wrong choice of projection.
If possible, the best choice is to obtain the browse image from the data provider, but in the case where the user needs to purchase a proprietary imaging program, the HDF group provide a free tool, HDFLook, which runs natively in the X11 environment, but can be used on Mac OS X, and Windows. A more sophisticated tool is MS Sphinx, which is licensed software. This allows more options for manipulating HDF file for the generation of the browse image, including RGB, a variety of projections and tighter control over the compression schemes used in the image, e.g. JPEG 2000, etc. For generation of browse images for shp files, use a proprietary conversion tool such as shp2kml, which runs natively in the Windows environment, and is available from sourceforge.net. Note, for retention of full feature selection, remember to use the checkboxes provided in the conversion tool to select the features required during the conversion process.
