Contact Info

Crumbtrail

eye4software.com » HydroMagic » Documentation » Articles and Howto's » A description of the Hydromagic raw data file records


Raw Data Storage Format

Hydromagic stores all incoming data to raw data files when data is being recorded. The files are written as human readable text so you are able to see what data is recorded. This is also very useful in case of problems so that you can look for the problem yourself.

Raw data files are stored in the "RawData" folder in the project tree. The following text is a fragment of a RAW data file recorded in Hydromagic:

NAM RAW00021
CMT 
MOD 0
VIS 1
TZI -60 "West-Europa (standaardtijd)"
HUN 9001
VUN 9001
PRJ 28992
DTM 4289
ELL 7004
PRM 8901
GEO 119 "rdnap2018.geo"
ANT 0.200
DFT 0.200
VER 10.0
BLD 10.0.64.614
KEY 130-3387548503
FTR 0x00000001
MNT 1811918876
DEV 1 5 "Echologger Plugin for Hydromagic" "Mode=2;"
DEV 2 0 "Hydromagic NMEA0183 plugin #1" "sentences=1000419;"
DPT 00000008 00000000 1655211104.900 0.000 0.000 0.000
DPT 00000004 00000000 1655211104.899 0.000 0.000 0.000
TRC 00000000 00000002 1655211104.770 0.000 $GNGGA,125145.00,5248.2116756,N,00555.6773863,E,4,23,0.7,-0.525,M,42.039,M,2.0,0000*7E
POS 00000007 00000002 1655211104.770 0.000 191466.672 535277.542 41.514 -0.525 5.927956438 52.803527927 125145.000
GPS 0000000F 00000002 1655211104.770 0.000 5 23 0 2
DOP 00000001 00000002 1655211104.770 0.000 0.70 0.00 0.00
TRC 00000000 00000002 1655211104.771 0.000 $GNVTG,92.81,T,,M,0.72,N,1.33,K,D*10
SPD 00000000 00000002 1655211104.771 0.000 1.33
HDG 00000001 00000002 1655211104.771 0.000 92.81 0.00
TRC 00000000 00000002 1655211104.771 0.000 $GNGST,125145.00,19.000,,,,0.012,0.010,0.022*5B
ERR 00000007 00000002 1655211104.771 0.000 0.010 0.012 0.022
DPT 00000001 00000001 1655211104.899 0.000 0.771 0.000
SEN 00000001 00000001 1655211104.899 0.000 22.66884 0.00000 0.00000 0.00000 0.00000
MOT 00000006 00000001 1655211104.899 0.000 0.000 0.700 -6.100
DPT 00000002 00000001 1655211104.900 0.000 0.000 0.834
SEN 00000001 00000001 1655211104.900 0.000 22.64677 0.00000 0.00000 0.00000 0.00000
MOT 00000006 00000001 1655211104.900 0.000 0.000 1.200 -6.200
TRC 00000000 00000002 1655211104.975 0.000 $GNGGA,125145.20,5248.2116842,N,00555.6774430,E,4,23,0.7,-0.532,M,42.039,M,1.2,0000*7C
POS 00000007 00000002 1655211104.975 0.000 191466.735 535277.558 41.507 -0.532 5.927957383 52.803528070 125145.200
GPS 0000000F 00000002 1655211104.975 0.000 5 23 0 1
DOP 00000001 00000002 1655211104.975 0.000 0.70 0.00 0.00
TRC 00000000 00000002 1655211104.975 0.000 $GNVTG,92.81,T,,M,0.64,N,1.19,K,D*1F
SPD 00000000 00000002 1655211104.975 0.000 1.19
HDG 00000001 00000002 1655211104.975 0.000 92.81 0.00
TRC 00000000 00000002 1655211104.976 0.000 $GNGST,125145.20,15.000,,,,0.012,0.010,0.022*55
ERR 00000007 00000002 1655211104.976 0.000 0.010 0.012 0.022
		
Example Hydromagic RAW data file (header and body).


RAW data file header fields

The header contains information on both the raw data file and the equipment that was used during the recording of the data. Also offsets such as GNSS antenna height and transducer offset are stored. We will explain each of the record headers and how they can be read. The KEY, FTR and MNT records are not document since they do not contain usable data for clients.

NAM - Raw data file name

The name of the raw data set. Since version 10.1 this value will also been written to your project file. You can not alter this value to change the name of your data set. To do so you have to rename both this record and the file name.

CMT - Comment for raw data file

This fields is no longer used since version 10.1. It was used to store the comment added to a raw data file in the "Project Explorer". Since version 10.1 this comment is stored in your project file.

MOD - Modified flag

Indicates while this file has been modified by the user. After modifying a raw data file by either the "Echogram Digitizer" or the "Sounding Wizard", the modified version of the raw data file will be copied to the "Modified" folder of your project tree and the modified flag will be set to one. When the modified flag has been set to zero, you are looking at original data in either the "RawData" or "Backup" folders.

VIS - Visibility flag

In versions prior to version 10.1 this settings was used to determine whether the raw data was visible (being drawn) in the map view. Since version 10.1 this flag is stored in your project file. This means that this value is still being written, but modifying it in the raw data file has no effect anymore.

TZI - Time Zone Information

Stores the time zone the computer was in while recording the data. While the data is recorded with GMT/UTC timestamps, this might become in handy, when for instance tide files are created in your local time. Our support staff can then convert them to be able to reproduce data when needed.

HUN - Horizontal units

Horizontal units used to record the data. This includes the units used in projected coordinates and offsets. The horizontal units are configured when choosing a map projection. Unless the project has been converted the units are the same as used in the map projection. The units are stored as EPSG id's (9001: Meters, 9002: International Feet, 9003: US Survey Feet, etc...).

VUN - Vertical units

Vertical units used to record the data. This includes the units used in depth, elevations and offsets. The vertical units are configured when choosing a map projection. Unless the project has been converted the units are the same as used in the map projection. The units are stored as EPSG id's (9001: Meters, 9002: International Feet, 9003: US Survey Feet, etc...).

PRJ - Projection

The EPSG or ESRI map projection or coordinate reference system id used for the projected coordinates in the raw data file. To find the projection name for a given coordinate system id, you can look it up in the "Select Map Grid" dialog from the "Coordinate Calculator".

DTM - Horizontal map datum

The EPSG or ESRI datum id for the selected map projection or coordinate reference system as stored in the "PRJ" record discussed above.

ELL - Ellipsoid

The EPSG or ESRI ellipsoid id for the selected map datum as stored in the "DTM" record discussed above.

PRM - Prime Meridian

The EPSG or ESRI prime meridian id for the selected map datum as stored in the "DTM" record discussed above. In most cases this will be the Greenwich prime meridian (EPSG8901).

GEO - Geoid Model

The geoid model that was selected when recording or processing the raw data file. The geoid model is used to calculate the separation between the ellipsoid specified in the "ELL" record and the local vertical datum. You can find more information about the selected ellipsoid in the Geoid Manager. Add "8000" to the geoid identifier used in this record.

ANT - Antenna Height

The height of the antenna phase center above the water line. This value is used to calculate the height of the waterline by subtracting this value from the orthometric height (ellipsoidal height - geoid separation). This value is configured in the "RTK settings tab".

DFT - Transducer Draft

The transducer draft (or static draft) is the vertical offset between the bottom of the echo sounder transducer and the water line. This value is added to the sounding values reported by the echo sounder. This values is already added to the "DPT" record values in the raw data file.

VER - Version

Stores the version of the software that was used to record the data. This allows our support staff to see which software was used in order to reproduce problems.

BLD - Build

Stores the build number of the software that was used to record the data. This allows our support staff to see which software was used in order to reproduce problems.


Raw data file data records

Each raw data file starts with a three letter ID identifying the record. All possible record types are outlines below. After the ID, there are four fields that are common for every record in the data section of the raw data file:

  1. mask field - Contains bit flags telling the software which fields contain valid data. When set to zero you can ignore this field.
  2. channel field - Contains the channel number of the plugin. When this value is set to zero the data was imported or generated by the user;
  3. timestamp field - Contains the time the data was received and processed by a plugin. This time is encoded as seconds (and milliseconds) since 1-Jan-1970.
  4. latency field - latency value entered by the user during the configuration of a plugin. This field is used to correct latency values later.


BAT - Bathymetric data (multi beam)

The bathymetric data record is used to store multi beam swath data in a Hydromagic raw data file. The actual swath (beam) data is stored in a separate file at the file offset specified by the offset field. The record also contains the sound velocity value and beam count for the recorded swath. The mask fields specifies which data is available in the separate (binary) file.

BAT 000006D7 00000000 1666799731.608 0.000 800 00000000 1450.670 000000000001E780
BAT <mask> <channel> <timestamp> <latency> <beams> <flags> <velocity> <offset>
Example bathymetric data and field layout

Bitmask ID Description
0x0001 MASK_BATHY_ALONG_TRACK Along track distances are included
0x0002 MASK_BATHY_ACROSS_TRACK Across track distances are included
0x0004 MASK_BATHY_DEPTHS Processed depth information is included
0x0008 MASK_BATHY_BEAM_ANGLES Across track (beam) angles are included
0x0010 MASK_BATHY_BEAM_FLAGS Corrected across track (beam) angles are included
0x0020 MASK_BATHY_ALONG_ANGLE Along track angles are included
0x0040 MASK_BATHY_ACROSS_ANGLE Across track angles are included
0x0080 MASK_BATHY_ACROSS_ANGLE Beam receiver amplitude values are included
0x0100 MASK_BATHY_TIMESTAMP Beam timestamp values are included
0x0200 MASK_BATHY_TWOWAYTRAVELTIME Two way travel time values are included
0x0400 MASK_BATHY_UNCERTAINTY Uncertainty values are included
Possible bitmask values for the mask field


BTM - Bottom classification parameters

The bottom classification record is only written when an echo sounder is connected which is sending information about signal reflectivity, bottom porosity or bottom density. An example of such an echo sounder is the Unabara Hydro-2F. Depending on the mask field set, parameters can be read from this record. A mask value of 0x000000FF means that all the values are valid.

BTM 000000FF 00000001 1579191590.890 0.000 7.000 18.000 45.000 13.000 2.340 1.310 28.000 83.000
BTM <mask> <channel> <timestamp> <latency> <BL (L)> <BL (H)> <Refl (L)> <Refl (H)> <Dens (L)> <Dens (H)> <P (L)> <P (H)>
Example bottom classification record and field layout

Bitmask ID Description
0x0001 MASK_BOTTOM_LO_BOTTOMLOSS Low frequency bottom loss value is valid
0x0002 MASK_BOTTON_LO_REFLECT Low frequency reflectivity value is valid
0x0004 MASK_BOTTOM_LO_DENSITY Low frequency density value is valid
0x0004 MASK_BOTTOM_LO_POROSITY Low frequency porosity value is valid
0x0010 MASK_BOTTOM_HI_BOTTOMLOSS High frequency bottom loss value is valid
0x0020 MASK_BOTTON_HI_REFLECT High frequency reflectivity value is valid
0x0040 MASK_BOTTOM_HI_DENSITY High frequency density value is valid
0x0080 MASK_BOTTOM_HI_POROSITY High frequency porosity value is valid
Possible bitmask values for the mask field


DOP - Dilution of precision

The DOP or dilution of precision's give an indication of the quality of the determined position. This information is stored in the raw data files. This values can be used to filter data for invalid positions. Depending on the NMEA0183 sentences enabled in your GNSS receiver, one or more fields contain valid values. To check which values are valid, you have to check the mask field.

DOP 00000001 00000002 1655211104.770 0.000 0.70 0.00 0.00
DOP <mask> <channel> <timestamp> <latency> <HDOP> <VDOP> <PDOP>
Example DOP record and field layout

Bitmask ID Description
0x0001 MASK_DOP_HDOP The horizontal DOP value is valid (HDOP)
0x0002 MASK_DOP_VDOP The vertical DOP value is valid (VDOP)
0x0004 MASK_DOP_HDOP The 3D position DOP value is valid (PDOP)
Possible bitmask values for the mask field


DPT - Depth

The depth record is used to store raw depth data from your single beam echo sounder. It can also contain the nadir depth of your side scan sonar or multi beam echo sounder. The mask field indicates whether the low frequency, high frequency or both values are valid. The depth stored in the raw data file is already corrected with the static draft (see DFT header record).

DPT 00000002 00000001 1655211104.900 0.000 0.000 0.834
DPT <mask> <channel> <timestamp> <latency> <depth lo> <depth hi>
Example single beam depth record and field layout

Bitmask ID Description
0x0001 MASK_DEPTH_LO Low frequency depth value is valid
0x0002 MASK_DEPTH_HI High frequency depth values is valid
Possible bitmask values for the mask field


ERR - GNSS Error estimates

When the GNSS receiver has the GST NMEA0183 sentence enabled the error estimates will be logged into the raw data files. This values can be used to filter data for invalid positions.

ERR 00000007 00000002 1655211104.771 0.000 0.010 0.012 0.022
ERR <mask> <channel> <timestamp> <latency> <longitude error> <latitude error> <elevation error>
Example GNSS error estimates record and field layout

Bitmask ID Description
0x0001 MASK_POSITION_ERROR_LON Longitude error value is valid
0x0002 MASK_POSITION_ERROR_LAT Latitude error value is valid
0x0004 MASK_POSITION_ERROR_ELE Elevation error value is valid
Possible bitmask values for the mask field


GPS - GNSS information

The GPS record contains information reported by your GNSS receiver to analyze the quality of the received position and elevation information. This includes the number of satellites used in the position calculation, the current fix quality (or mode), and when using correction signals like WAAS, EGNOS or RTK, a beacon id and the age of the correction data. All this information is used by Hydromagic to process and filter data. Again, analyze the mask field to determine which fields hold valid data, since not all information might be available at all times (this also depends on the NMEA0183 sentences outputted by the GNSS receiver).

GPS 0000000F 00000002 1655211104.770 0.000 5 23 0 2
GPS <mask> <channel> <timestamp> <latency> <quality> <satellites> <beacon id> <correction age>
Example GNSS information record and field layout

Bitmask ID Description
0x0001 MASK_GPS_QUALITY The GNSS quality indicator has been set
0x0002 MASK_GPS_SATS The number of GNSS satellites used has been set
0x0004 MASK_GPS_BEACON The beacon id has been set (corrected GNSS data only)
0x0008 MASK_GPS_AGE The age of correction data has been set (corrected GNSS data only)
Possible bitmask values for the mask field

Quality Description
0 The GNSS receiver hasn't calculated a position fix(yet)
1 The GNSS receiver has calculated an autonomous position fix
2 The GNSS position has been corrected with WAAS/EGNOS/SBAS
4 The GNSS receiver is establishing a RTK FIX (float)
5 The GNSS receiver has calculated a RTK FIX
Possible values for the quality field


HDG - Heading

This record is used to store heading data received from either your GNSS receiver (course over ground), gyro, magnetic compass or motion sensor. The mask field indicates whether the true or magnetic heading is stored in the record. You can switch between true and magnetic heading in the software in the "Calibration" tab.

HDG 00000001 00000002 1655211104.771 0.000 92.81 0.00
HDG <mask> <channel> <timestamp> <latency> <true heading> <magnetic heading>
Example heading record (containing true heading) and field layout

Bitmask ID Description
0x0001 MASK_HEADING_TRUE The record contains a true heading (course over ground)
0x0002 MASK_HEADING_MAGNETIC The record contains a magnetic heading (gyro, compass)
Possible bitmask values for the mask field


MOT - Motion data

The motion data record is used to store data (heave, pitch and roll) from a motion sensor. Motion data can also be imported from third party software developer's raw data files. When an echo sounder has a built in motion sensor, the motion data will be stored in this record as well. The mask field contains bit flags which indicate what information is present in the record. The pitch and roll angels are always stored in degrees.

MOT 00000007 00000000 1666799731.521 0.000 -0.020 3.520 0.820
MOT <mask> <channel> <timestamp> <latency> <heave> <pitch> <roll>
Example motion data (heave,pitch and roll) record and field layout

Bitmask ID Description
0x0001 MASK_MOTION_HEAVE The motion sensor outputs heave
0x0002 MASK_MOTION_PITCH The motion sensor outputs pitch
0x0004 MASK_MOTION_ROLL The motion sensor outputs roll
Possible bitmask values for the mask field


POS - Position data and altitude

Position information as well as elevations like ellipsoidal and MSL height are stored in this record. When configuring your GNSS receiver, make sure that the GGA NMEA0183 sentence is included, since it contains not only position, but also elevation information. If possible, use the GGA sentence rather then GLL or RMC. The mask field contains bit flags which indicate whether elevation data is included or not.

POS 00000007 00000002 1655211104.770 0.000 191466.672 535277.542 41.514 -0.525 5.927956438 52.803527927 125145.000
POS <mask> <channel> <timestamp> <latency> <easting> <northing> <elevation> <MSL> <lon> <lat> <UTC>
Example position data (WGS84, projected position and elevations) record and field layout

Bitmask ID Description
0x0001 MASK_POSITION_WGS84 The record contains WGS84 geographic positions
0x0002 MASK_POSITION_PROJECTED The record contains projected coordinates
0x0004 MASK_POSITION_ELEVATION The record contains the ellipsoidal and/or MSL heights
Possible bitmask values for the mask field


SEN - Sensor Values

The sensor is a generic field to store values other then depth, motion or position. At this moment this field is used to store water and air temperatures, but using a dedicated plugin, it could also be used to store other values like wind, water quality or magnetometer data.

SEN 00000001 00000001 1655211104.899 0.000 22.66884 0.00000 0.00000 0.00000 0.00000
SEN <mask> <channel> <timestamp> <latency> <value1> <value2> <value3> <value4> <value5>
Example sensor data (water temperature) record and field layout


SPD - Speed

Although speed is not required for data processing, it is stored to the raw data file for logging purposes. The speed is always stored in kilometers per hour. Speed is normally obtained directly from the GNSS receiver (speed over ground). For this record the mask field is unused and set to zero.

SPD 00000000 00000002 1655211104.771 0.000 1.33
SPD <mask> <channel> <timestamp> <latency> <speed in kmh>
Example speed data record and field layout


SVS - Sound velocity measurements

When using a sound velocity profiler or the Hydromagic Manual Sound Velocity plugin, there is no need to specify a sound velocity file while processing the raw data files. In this case the sound velocity value from the raw data file will be used. This record can also be created by Hydromagic when importing raw data from 3rd party software.

SVS 00000000 00000000 1666799731.589 0.000 1450.89
SVS <mask> <channel> <timestamp> <latency> <sound velocity (m/s)>
Example sound velocity record and field layout


TID - Tide observations

When using a tide receiver or the "Hydromagic Manual Tide plugin" there is no need to specify a tide file while processing the raw data files. In this case the tide value from the raw data file will be used. This record can also be created when importing raw data from 3rd party software.

TID 00000000 00000000 1666799731.589 0.000 -419.690
TID <mask> <channel> <timestamp> <latency> <tide level>
Example tide observation data record and field layout


TRC - Trace data

The trace record contains raw serial or network data received from the device (for instance NMEA0183 sentences). Having this data stored with the measurement can help you in case of troubleshooting or to rebuild the data from the original incoming data sentences. The maximum length of received data which is stored in this record is 256 bytes. To reduce file size and increase processing speed, this record is not written to modified raw data files. For this record the mask field is unused and set to zero.

TRC 00000000 00000002 1655211104.770 0.000 $GNGGA,125145.00,5248.2116756,N,00555.6773863,E,4,23,0.7,-0.525,M,42.039,M,2.0,0000*7E
TRC <mask> <channel> <timestamp> <latency> <received data>
Example tracing record and field layout