Raw Data Storage Format
Hydromagic stores all incoming data to raw data files when data is being recorded. The files are written as human readable text so you are able to see what data is recorded. This is also very useful in case of problems so that you can look for the problem yourself.
Raw data files are stored in the "RawData" folder in the project tree. The following text is a fragment of a RAW data file recorded in Hydromagic:
NAM RAW00021 CMT MOD 0 VIS 1 TZI -60 "West-Europa (standaardtijd)" HUN 9001 VUN 9001 PRJ 28992 DTM 4289 ELL 7004 PRM 8901 GEO 119 "rdnap2018.geo" ANT 0.200 DFT 0.200 VER 10.0 BLD 10.0.64.614 KEY 130-3387548503 FTR 0x00000001 MNT 1811918876 DEV 1 5 "Echologger Plugin for Hydromagic" "Mode=2;" DEV 2 0 "Hydromagic NMEA0183 plugin #1" "sentences=1000419;" DPT 00000008 00000000 1655211104.900 0.000 0.000 0.000 DPT 00000004 00000000 1655211104.899 0.000 0.000 0.000 TRC 00000000 00000002 1655211104.770 0.000 $GNGGA,125145.00,5248.2116756,N,00555.6773863,E,4,23,0.7,-0.525,M,42.039,M,2.0,0000*7E POS 00000007 00000002 1655211104.770 0.000 191466.672 535277.542 41.514 -0.525 5.927956438 52.803527927 125145.000 GPS 0000000F 00000002 1655211104.770 0.000 5 23 0 2 DOP 00000001 00000002 1655211104.770 0.000 0.70 0.00 0.00 TRC 00000000 00000002 1655211104.771 0.000 $GNVTG,92.81,T,,M,0.72,N,1.33,K,D*10 SPD 00000000 00000002 1655211104.771 0.000 1.33 HDG 00000001 00000002 1655211104.771 0.000 92.81 0.00 TRC 00000000 00000002 1655211104.771 0.000 $GNGST,125145.00,19.000,,,,0.012,0.010,0.022*5B ERR 00000007 00000002 1655211104.771 0.000 0.010 0.012 0.022 DPT 00000001 00000001 1655211104.899 0.000 0.771 0.000 SEN 00000001 00000001 1655211104.899 0.000 22.66884 0.00000 0.00000 0.00000 0.00000 MOT 00000006 00000001 1655211104.899 0.000 0.000 0.700 -6.100 DPT 00000002 00000001 1655211104.900 0.000 0.000 0.834 SEN 00000001 00000001 1655211104.900 0.000 22.64677 0.00000 0.00000 0.00000 0.00000 MOT 00000006 00000001 1655211104.900 0.000 0.000 1.200 -6.200 TRC 00000000 00000002 1655211104.975 0.000 $GNGGA,125145.20,5248.2116842,N,00555.6774430,E,4,23,0.7,-0.532,M,42.039,M,1.2,0000*7C POS 00000007 00000002 1655211104.975 0.000 191466.735 535277.558 41.507 -0.532 5.927957383 52.803528070 125145.200 GPS 0000000F 00000002 1655211104.975 0.000 5 23 0 1 DOP 00000001 00000002 1655211104.975 0.000 0.70 0.00 0.00 TRC 00000000 00000002 1655211104.975 0.000 $GNVTG,92.81,T,,M,0.64,N,1.19,K,D*1F SPD 00000000 00000002 1655211104.975 0.000 1.19 HDG 00000001 00000002 1655211104.975 0.000 92.81 0.00 TRC 00000000 00000002 1655211104.976 0.000 $GNGST,125145.20,15.000,,,,0.012,0.010,0.022*55 ERR 00000007 00000002 1655211104.976 0.000 0.010 0.012 0.022Example Hydromagic RAW data file (header and body).
RAW data file header fields
The header contains information on both the raw data file and the equipment that was used during the recording of the data. Also offsets such as GNSS antenna height and transducer offset are stored. We will explain each of the record headers and how they can be read. The KEY, FTR and MNT records are not document since they do not contain usable data for clients.
NAM - Raw data file name
The name of the raw data set. Since version 10.1 this value will also been written to your project file. You can not alter this value to change the name of your data set. To do so you have to rename both this record and the file name.
CMT - Comment for raw data file
This fields is no longer used since version 10.1. It was used to store the comment added to a raw data file in the "Project Explorer". Since version 10.1 this comment is stored in your project file.
MOD - Modified flag
Indicates while this file has been modified by the user. After modifying a raw data file by either the "Echogram Digitizer" or the "Sounding Wizard", the modified version of the raw data file will be copied to the "Modified" folder of your project tree and the modified flag will be set to one. When the modified flag has been set to zero, you are looking at original data in either the "RawData" or "Backup" folders.
VIS - Visibility flag
In versions prior to version 10.1 this settings was used to determine whether the raw data was visible (being drawn) in the map view. Since version 10.1 this flag is stored in your project file. This means that this value is still being written, but modifying it in the raw data file has no effect anymore.
TZI - Time Zone Information
Stores the time zone the computer was in while recording the data. While the data is recorded with GMT/UTC timestamps, this might become in handy, when for instance tide files are created in your local time. Our support staff can then convert them to be able to reproduce data when needed.
HUN - Horizontal units
Horizontal units used to record the data. This includes the units used in projected coordinates and offsets. The horizontal units are configured when choosing a map projection. Unless the project has been converted the units are the same as used in the map projection. The units are stored as EPSG id's (9001: Meters, 9002: International Feet, 9003: US Survey Feet, etc...).
VUN - Vertical units
Vertical units used to record the data. This includes the units used in depth, elevations and offsets. The vertical units are configured when choosing a map projection. Unless the project has been converted the units are the same as used in the map projection. The units are stored as EPSG id's (9001: Meters, 9002: International Feet, 9003: US Survey Feet, etc...).
PRJ - Projection
The EPSG or ESRI map projection or coordinate reference system id used for the projected coordinates in the raw data file. To find the projection name for a given coordinate system id, you can look it up in the "Select Map Grid" dialog from the "Coordinate Calculator".
DTM - Horizontal map datum
The EPSG or ESRI datum id for the selected map projection or coordinate reference system as stored in the "PRJ" record discussed above.
ELL - Ellipsoid
The EPSG or ESRI ellipsoid id for the selected map datum as stored in the "DTM" record discussed above.
PRM - Prime Meridian
The EPSG or ESRI prime meridian id for the selected map datum as stored in the "DTM" record discussed above. In most cases this will be the Greenwich prime meridian (EPSG8901).
GEO - Geoid Model
The geoid model that was selected when recording or processing the raw data file. The geoid model is used to calculate the separation between the ellipsoid specified in the "ELL" record and the local vertical datum. You can find more information about the selected ellipsoid in the Geoid Manager. Add "8000" to the geoid identifier used in this record.
ANT - Antenna Height
The height of the antenna phase center above the water line. This value is used to calculate the height of the waterline by subtracting this value from the orthometric height (ellipsoidal height - geoid separation). This value is configured in the "RTK settings tab".
DFT - Transducer Draft
The transducer draft (or static draft) is the vertical offset between the bottom of the echo sounder transducer and the water line. This value is added to the sounding values reported by the echo sounder. This values is already added to the "DPT" record values in the raw data file.
VER - Version
Stores the version of the software that was used to record the data. This allows our support staff to see which software was used in order to reproduce problems.
BLD - Build
Stores the build number of the software that was used to record the data. This allows our support staff to see which software was used in order to reproduce problems.
Raw data file data records
Each raw data file starts with a three letter ID identifying the record. All possible record types are outlines below. After the ID, there are four fields that are common for every record in the data section of the raw data file:
- mask field - Contains bit flags telling the software which fields contain valid data. When set to zero you can ignore this field.
- channel field - Contains the channel number of the plugin. When this value is set to zero the data was imported or generated by the user;
- timestamp field - Contains the time the data was received and processed by a plugin. This time is encoded as seconds (and milliseconds) since 1-Jan-1970.
- latency field - latency value entered by the user during the configuration of a plugin. This field is used to correct latency values later.
BAT - Bathymetric data (multi beam)
The bathymetric data record is used to store multi beam swath data in a Hydromagic raw data file. The actual swath (beam) data is stored in a separate file at the file offset specified by the offset field. The record also contains the sound velocity value and beam count for the recorded swath. The mask fields specifies which data is available in the separate (binary) file.
BAT 000006D7 00000000 1666799731.608 0.000 800 00000000 1450.670 000000000001E780 BAT <mask> <channel> <timestamp> <latency> <beams> <flags> <velocity> <offset>Example bathymetric data and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_BATHY_ALONG_TRACK | Along track distances are included |
| 0x0002 | MASK_BATHY_ACROSS_TRACK | Across track distances are included |
| 0x0004 | MASK_BATHY_DEPTHS | Processed depth information is included |
| 0x0008 | MASK_BATHY_BEAM_ANGLES | Across track (beam) angles are included |
| 0x0010 | MASK_BATHY_BEAM_FLAGS | Corrected across track (beam) angles are included |
| 0x0020 | MASK_BATHY_ALONG_ANGLE | Along track angles are included |
| 0x0040 | MASK_BATHY_ACROSS_ANGLE | Across track angles are included |
| 0x0080 | MASK_BATHY_ACROSS_ANGLE | Beam receiver amplitude values are included |
| 0x0100 | MASK_BATHY_TIMESTAMP | Beam timestamp values are included |
| 0x0200 | MASK_BATHY_TWOWAYTRAVELTIME | Two way travel time values are included |
| 0x0400 | MASK_BATHY_UNCERTAINTY | Uncertainty values are included |
BTM - Bottom classification parameters
The bottom classification record is only written when an echo sounder is connected which is sending information about signal reflectivity, bottom porosity or bottom density. An example of such an echo sounder is the Unabara Hydro-2F. Depending on the mask field set, parameters can be read from this record. A mask value of 0x000000FF means that all the values are valid.
BTM 000000FF 00000001 1579191590.890 0.000 7.000 18.000 45.000 13.000 2.340 1.310 28.000 83.000 BTM <mask> <channel> <timestamp> <latency> <BL (L)> <BL (H)> <Refl (L)> <Refl (H)> <Dens (L)> <Dens (H)> <P (L)> <P (H)>Example bottom classification record and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_BOTTOM_LO_BOTTOMLOSS | Low frequency bottom loss value is valid |
| 0x0002 | MASK_BOTTON_LO_REFLECT | Low frequency reflectivity value is valid |
| 0x0004 | MASK_BOTTOM_LO_DENSITY | Low frequency density value is valid |
| 0x0004 | MASK_BOTTOM_LO_POROSITY | Low frequency porosity value is valid |
| 0x0010 | MASK_BOTTOM_HI_BOTTOMLOSS | High frequency bottom loss value is valid |
| 0x0020 | MASK_BOTTON_HI_REFLECT | High frequency reflectivity value is valid |
| 0x0040 | MASK_BOTTOM_HI_DENSITY | High frequency density value is valid |
| 0x0080 | MASK_BOTTOM_HI_POROSITY | High frequency porosity value is valid |
DOP - Dilution of precision
The DOP or dilution of precision's give an indication of the quality of the determined position. This information is stored in the raw data files. This values can be used to filter data for invalid positions. Depending on the NMEA0183 sentences enabled in your GNSS receiver, one or more fields contain valid values. To check which values are valid, you have to check the mask field.
DOP 00000001 00000002 1655211104.770 0.000 0.70 0.00 0.00 DOP <mask> <channel> <timestamp> <latency> <HDOP> <VDOP> <PDOP>Example DOP record and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_DOP_HDOP | The horizontal DOP value is valid (HDOP) |
| 0x0002 | MASK_DOP_VDOP | The vertical DOP value is valid (VDOP) |
| 0x0004 | MASK_DOP_HDOP | The 3D position DOP value is valid (PDOP) |
DPT - Depth
The depth record is used to store raw depth data from your single beam echo sounder. It can also contain the nadir depth of your side scan sonar or multi beam echo sounder. The mask field indicates whether the low frequency, high frequency or both values are valid. The depth stored in the raw data file is already corrected with the static draft (see DFT header record).
DPT 00000002 00000001 1655211104.900 0.000 0.000 0.834 DPT <mask> <channel> <timestamp> <latency> <depth lo> <depth hi>Example single beam depth record and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_DEPTH_LO | Low frequency depth value is valid |
| 0x0002 | MASK_DEPTH_HI | High frequency depth values is valid |
ERR - GNSS Error estimates
When the GNSS receiver has the GST NMEA0183 sentence enabled the error estimates will be logged into the raw data files. This values can be used to filter data for invalid positions.
ERR 00000007 00000002 1655211104.771 0.000 0.010 0.012 0.022 ERR <mask> <channel> <timestamp> <latency> <longitude error> <latitude error> <elevation error>Example GNSS error estimates record and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_POSITION_ERROR_LON | Longitude error value is valid |
| 0x0002 | MASK_POSITION_ERROR_LAT | Latitude error value is valid |
| 0x0004 | MASK_POSITION_ERROR_ELE | Elevation error value is valid |
GPS - GNSS information
The GPS record contains information reported by your GNSS receiver to analyze the quality of the received position and elevation information. This includes the number of satellites used in the position calculation, the current fix quality (or mode), and when using correction signals like WAAS, EGNOS or RTK, a beacon id and the age of the correction data. All this information is used by Hydromagic to process and filter data. Again, analyze the mask field to determine which fields hold valid data, since not all information might be available at all times (this also depends on the NMEA0183 sentences outputted by the GNSS receiver).
GPS 0000000F 00000002 1655211104.770 0.000 5 23 0 2 GPS <mask> <channel> <timestamp> <latency> <quality> <satellites> <beacon id> <correction age>Example GNSS information record and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_GPS_QUALITY | The GNSS quality indicator has been set |
| 0x0002 | MASK_GPS_SATS | The number of GNSS satellites used has been set |
| 0x0004 | MASK_GPS_BEACON | The beacon id has been set (corrected GNSS data only) |
| 0x0008 | MASK_GPS_AGE | The age of correction data has been set (corrected GNSS data only) |
| Quality | Description |
|---|---|
| 0 | The GNSS receiver hasn't calculated a position fix(yet) |
| 1 | The GNSS receiver has calculated an autonomous position fix |
| 2 | The GNSS position has been corrected with WAAS/EGNOS/SBAS |
| 4 | The GNSS receiver is establishing a RTK FIX (float) |
| 5 | The GNSS receiver has calculated a RTK FIX |
HDG - Heading
This record is used to store heading data received from either your GNSS receiver (course over ground), gyro, magnetic compass or motion sensor. The mask field indicates whether the true or magnetic heading is stored in the record. You can switch between true and magnetic heading in the software in the "Calibration" tab.
HDG 00000001 00000002 1655211104.771 0.000 92.81 0.00 HDG <mask> <channel> <timestamp> <latency> <true heading> <magnetic heading>Example heading record (containing true heading) and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_HEADING_TRUE | The record contains a true heading (course over ground) |
| 0x0002 | MASK_HEADING_MAGNETIC | The record contains a magnetic heading (gyro, compass) |
MOT - Motion data
The motion data record is used to store data (heave, pitch and roll) from a motion sensor. Motion data can also be imported from third party software developer's raw data files. When an echo sounder has a built in motion sensor, the motion data will be stored in this record as well. The mask field contains bit flags which indicate what information is present in the record. The pitch and roll angels are always stored in degrees.
MOT 00000007 00000000 1666799731.521 0.000 -0.020 3.520 0.820 MOT <mask> <channel> <timestamp> <latency> <heave> <pitch> <roll>Example motion data (heave,pitch and roll) record and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_MOTION_HEAVE | The motion sensor outputs heave |
| 0x0002 | MASK_MOTION_PITCH | The motion sensor outputs pitch |
| 0x0004 | MASK_MOTION_ROLL | The motion sensor outputs roll |
POS - Position data and altitude
Position information as well as elevations like ellipsoidal and MSL height are stored in this record. When configuring your GNSS receiver, make sure that the GGA NMEA0183 sentence is included, since it contains not only position, but also elevation information. If possible, use the GGA sentence rather then GLL or RMC. The mask field contains bit flags which indicate whether elevation data is included or not.
POS 00000007 00000002 1655211104.770 0.000 191466.672 535277.542 41.514 -0.525 5.927956438 52.803527927 125145.000 POS <mask> <channel> <timestamp> <latency> <easting> <northing> <elevation> <MSL> <lon> <lat> <UTC>Example position data (WGS84, projected position and elevations) record and field layout
| Bitmask | ID | Description |
|---|---|---|
| 0x0001 | MASK_POSITION_WGS84 | The record contains WGS84 geographic positions |
| 0x0002 | MASK_POSITION_PROJECTED | The record contains projected coordinates |
| 0x0004 | MASK_POSITION_ELEVATION | The record contains the ellipsoidal and/or MSL heights |
SEN - Sensor Values
The sensor is a generic field to store values other then depth, motion or position. At this moment this field is used to store water and air temperatures, but using a dedicated plugin, it could also be used to store other values like wind, water quality or magnetometer data.
SEN 00000001 00000001 1655211104.899 0.000 22.66884 0.00000 0.00000 0.00000 0.00000 SEN <mask> <channel> <timestamp> <latency> <value1> <value2> <value3> <value4> <value5>Example sensor data (water temperature) record and field layout
SPD - Speed
Although speed is not required for data processing, it is stored to the raw data file for logging purposes. The speed is always stored in kilometers per hour. Speed is normally obtained directly from the GNSS receiver (speed over ground). For this record the mask field is unused and set to zero.
SPD 00000000 00000002 1655211104.771 0.000 1.33 SPD <mask> <channel> <timestamp> <latency> <speed in kmh>Example speed data record and field layout
SVS - Sound velocity measurements
When using a sound velocity profiler or the Hydromagic Manual Sound Velocity plugin, there is no need to specify a sound velocity file while processing the raw data files. In this case the sound velocity value from the raw data file will be used. This record can also be created by Hydromagic when importing raw data from 3rd party software.
SVS 00000000 00000000 1666799731.589 0.000 1450.89 SVS <mask> <channel> <timestamp> <latency> <sound velocity (m/s)>Example sound velocity record and field layout
TID - Tide observations
When using a tide receiver or the "Hydromagic Manual Tide plugin" there is no need to specify a tide file while processing the raw data files. In this case the tide value from the raw data file will be used. This record can also be created when importing raw data from 3rd party software.
TID 00000000 00000000 1666799731.589 0.000 -419.690 TID <mask> <channel> <timestamp> <latency> <tide level>Example tide observation data record and field layout
TRC - Trace data
The trace record contains raw serial or network data received from the device (for instance NMEA0183 sentences). Having this data stored with the measurement can help you in case of troubleshooting or to rebuild the data from the original incoming data sentences. The maximum length of received data which is stored in this record is 256 bytes. To reduce file size and increase processing speed, this record is not written to modified raw data files. For this record the mask field is unused and set to zero.
TRC 00000000 00000002 1655211104.770 0.000 $GNGGA,125145.00,5248.2116756,N,00555.6773863,E,4,23,0.7,-0.525,M,42.039,M,2.0,0000*7E TRC <mask> <channel> <timestamp> <latency> <received data>Example tracing record and field layout