Transcript
2015 Release Notes 1 December 2015 Deep Links for sharing data searches and plots It is now possible to share your data searches and plots with someone else, even if you created them without being logged in. Regardless of whether you are logged in or not you can share the URL of your search; we call this a "deep link" URL. When this URL is pasted into a browser it will open the search tree to the same branch as the original, and in some cases that will also include the date range you selected. This applies to the following web pages: Data Search - applies to Step 1 of Data Search so does not include a time range or data product selection. Data Preview - includes the data display tab selected (Day, Month, etc). Plotting Utility - includes the sensor and time range. Note that if you have more than one plot, the deep link will only apply to the most recent plot that you added. As a logged-in user it is still possible to save your plots, then share the "reference link", which will include all of your plots. Hydrophone Viewer - includes the device and date.
Data Preview Improved the performance of loading the Data Preview page.
Data Search and Data Products For searches involving complex data products (ie. anything non-scalar) the search status messages that are visible briefly in Step 3 are now saved to a log called searchStatusUpdatesLog.txt. This log is visible in the FTP directory in a folder named with the search ID. See the "More" menu. The spectrogram data product is now available for the icListen AF 2504 hydrophone. The AWAC ADCP at FORCE (Atlantic observatory) now has a Daily Currents plot available.
Plotting Utility A new "Sort by" option has been added, called "Variables by Location". Some refer to this as a "primary sensor" search. You first select a location, then a variable (for example an ocean property such as Salinity).If there is more than one sensor at that location that can measure the same property, the sensor defined as the "primary sensor" will be used. This is the sensor that is the most accurate of the sensors available. It is possible that the primary sensor can change over the time range of a plot and this is indicated by different colours. This feature will soon also be available in Data Preview and Data Search.
QAQC (Quality Assurance / Quality Control) "Study area" type QAQC tests were used only for the VENUS network and are no longer being used. All existing study area tests have been migrated to the newer "Station" tests, which are similar but more flexible. A new type of QAQC test was developed which is called a Region test. This allows quality tests to be defined that apply to all similar sensors in the same "region" - for example in the same part of the ocean with a similar depth.
WARN (Web-enabled Awareness Research Network) Increased the earthquake detection grid by 50% to detect more distant events. There have been a number of small earthquakes slightly to the northwest of the original grid that would have been excluded had they been large enough to be detected. Improved the logging to make it easier to find detections from the same potential earthquake from multiple sensors in the same narrow time window.
oceannetworks.ca Website The Mill Bay observatory web page now shows the most recent data available, with a warning that it may be quite old (equipment has been offline for some time). http://www.oceannetworks.ca/learning/ocean-sense/mill-bay
Miscellaneous
The scalardata web service now also works for sensors that have no data rating. In other words sensors that do not send data on a regular basis such as camera lights and earthquake detection sensors. A new website was created for advertising the INCISE symposium that ONC is coordinating next summer. See http://incise2016.oceann etworks.ca/ Made improvements to the driver for the RBRduo Temperature Depth Logger used in the Atlantic observatory. The seismometer parser has been improved to provide State of Health information such as whether the clock is drifting. Improved the Device Details page to show the device attribute history, not just the current values.
4 November 2015 Login-free Access Without logging in, it is now possible to make a plot in Plotting Utility, copy the URL, send it to someone else who can paste the URL into their browser and see the same plot. This includes the same date range.
Data Preview Added links to Plotting Utility for each sensor (at the Instrument level in the Summary tab). These links open Plotting Utility, open the tree to the same sensor, and make a plot using the "all available" timeframe. If no data is available to be displayed, an empty plot will now be shown with the words "No data found for the selected time range" in the middle of the plot area.
Data Search Complex data products can now be obtained on the same day the data was collected. It used to be necessary to wait until the data was archived at the end of the day. This applies to instruments like ADCPs, ASL echosounders, Biosonics echosounder, Satlantic radiometers and nitrate sensors, LISST, and Imagenex rotary sonar. Greater reduced the time to generate hydrophone spectral density plots using the "all channels" option.
ONC Website (oceannetworks.ca) Added data plots for the FAST-2 Nortek AWAC and FAST-2 Sea-Bird Microcat CTD at the Bay of Fundy observatory (Installations >> Observatories >> Atlantic). The earthquake data dashboard developed last month is now visible in the Data and Tools page.
Instrument Support RDI ADCP data now incorporates the heading, including magnetic declination. Wrote parser and calibration code for the RBR Paroscientific Tilt Meter. Made improvements to the BARS (Benthic and Resistivity Sensors) driver and parser. Updated the Imagenex DeltaT 837 software so that accurate pitch/roll values are used. Created an FTP job for hydrophone files, and a postprocess job to convert the data into WAV format. Wrote a parser for the Suunto Zoop dive watch used at Cambridge Bay. Wrote a parser for the NRL biofuel cells and the Starmon mini temperature sensor. Wrote a parser for the Garmin GPS used by the divers at Cambridge Bay. Wrote a parser for the SeaBird CTD used on the Jason ROV. Wrote a parser for the MV Dorado Discovery ship navigation system.
WARN (Web-enabled Awareness Research Network) Wrote a web service (EventDefinitionService) that an external device can call to initiate an earthquake or tsunami test for demonstration purposes. Updated the Nanometrics Titan driver to 1) make it much more sensitive to detecting earthquakes and 2) to start outputting peak displacement (Pd) and predominant period (Tau) continuously for diagnostic purposes.
Miscellaneous
Automatic QAQC (quality control) tests for derived sensors are now based on the sensors they depend on. Wrote an externally-accessible web service ("rawdata") to return raw data. This was developed primarily for AIS (Automatic Identification System data for ship tracking) but it can be used for other types of raw data.
2 October 2015 New Data Preview feature - no login required This is now the "landing page" - ie. the page you will arrive at when you access Oceans 2.0. This new feature lets you quickly view recent data on most devices and sensor. Plots of data for yesterday and the most recent 30 days are pre-generated so they are available for viewing immediately. Note that Data Search lists a few devices that are intentionally not available in Data Preview.
How Data Preview works: You may or may not be logged in when you arrive at this page; login is not necessary. If you select a location in the tree, you will see a description of that site, and for major locations, a State of the Environment plot showing data for key sensors over all time. If you select a device you see a description of that device and a list of the sensors it contains on the Summary tab. The 24 Hr and Month tabs will show a plot of data for all the sensors with up to 9 sensors combined on each plot. If you select a sensor you will see a plot on the 24 Hr and Month tabs of data for that sensor. More time options will be added later. This data is pre-generated so the plots will appear immediately. If you click on an image an enlarged version will appear, and right-clicking on that image will allow you to save it. Results are updated daily and relate to calendar days regardless of when you view them. At this time, the following restrictions apply; more options will be added later: Only searchable by location (“Instruments by Location”) Limited to Daily and Monthly previews. Engineering devices such as Junction Boxes are hidden from the search tree. Devices and sensors that are offline (ie. not "neptune-searchable") will not appear. Devices in the "Salish Sea Marine Survival Project" (ie. Pacific Salmon Foundation devices) will not have previews at this time. If you wish to download the data you must use the Data Search feature which is visible on the menu, although you are able to save the plot image in Data Preview. Includes all the data plots previously seen in the VENUS Data Plots. This page in venus.uvic.ca will soon be shut down. Support for primary sensor tree browsing (called "Variables by Location") has been added but will not be enabled until the content has been reviewed.
Login-free access The project to make certain pages accessible without logging in is now finished. Login-free access now applies to the following pages. Data Search Data Preview Plotting Utility SeaTube Device Details Annotation Search plus several other pages in the More menu. Some features on these pages will continue to require logging in first in order to use them, such as creating an annotation, creating a video playlist, saving and recalling a plot, and saving your search results (search results will be saved without logging in, but only if you continue to use the same browser on the same computer, have cookies enabled, and do not clear your cache). As before, search results will only be saved for 7 days, regardless of whether you are logged in. Tools menu has been renamed to Admin. All features that require login are in the Admin menu which will be hidden if you are not logged in. Some menu items have moved from Admin to More so that they will be available without logging in.
New Earthquake Display Page A new page will be added soon to oceannetworks.ca to display earthquake activity around the world. All events above magnitude 6 are shown, as well as those above magnitude 4 in the southern British Columbia region. The user is able to interact with the map and the table of values. Seismic data collected by our underwater seismometers is displayed for most events. It will probably be added to the Data & Tools section.
WARN (Web-enabled Awareness Research Network) Added capability of parsing and archiving the continuous Tau and Pd values coming from the Nanometrics Titan accelerometers for diagnostic purposes, once the on-site driver has been updated. Added a new "status" web service to report the status of WARN - ie. the ability to detect earthquakes and tsunamis at this instant (snapshot).
Instrument Support Wrote a driver for the new RBR Tilt Meter. Parser and calibration formulae will be added later. Added support to start acquiring and archiving data from the JASCO AMAR hydrophone array adapter deployed in the Strait of Georgia for the ECHO vessel noise project. For the Pacific Salmon Foundation devices (Salish Sea Marine Survival Project), the corrected GPS data is now parsed so it can be downloaded like other scalar data. Added a parser for the Valeport miniCTD and the navigation sensor on the Zeus II ROV. Made improvements to the driver for the RBRduo Temperature-Depth Logger for the Atlantic observatory. Added support for new instruments at Cambridge Bay including a CO2 sensor and a SAMI pH sensor. Added support for new camera system for Mill Bay. Added support for Coastguard AIS instruments for the new Smart Oceans observatories.
Miscellaneous "DateFrom" is now an editable field for device attributes. Developed a web service to deliver raw AIS (Automatic Identification System) data for ship locations. Created an FTP job for hydrophone files. Added data products for the Kistler accelerometer.
2 September 2015
Data Search and Data Products Did some development of a new feature to quickly preview data on most devices. Will be enabled in a future release when it's complete. Added standard bin-mapping for RDI and Nortek ADCPs to match manufacturer's methods. In ADCP complex data products, removed the daily averaging on pitch and roll values. Improved RDI ADCP manufacturer’s raw files (RDI format), with heading/pitch/roll data from Oceans 2.0 replacing the internal sensor data where appropriate. Added support for the data products for the LISST (Sequoia Laser In-Situ Scattering and Transmissometry) device on the VENUS network. Data products for mobile devices now show the bounding coordinates of the device's location corresponding to the date range of the search. Added a device level time series scalar plot for all scalar devices. This plot displays up to 5 sensors at a time on a common time axis. If there are more than 5 sensors on the device, several plots will be generated. Added a data product for CODAR animation. All data products from the VENUS website have now been integrated into Oceans 2.0. You must use Oceans 2.0 from now on to download data from all devices on the VENUS network; the VENUS data download website is no longer available. Added a new spectral probability density plot data product for hydrophones. This plot displays a week of spectral data as a state of health and event detection summary. Also added a related spectral MATLAB data product which contains one minute spectra for each hydrophone source file (.wav or .hyd). Added corrected GPS data for Salish Sea Marine Survival Project (PSF) instruments.
Login Free Access Removed the login requirement from the remaining tabs on the Device Details page.
Instruments Made improvements to the driver for the RBRduo Temperature Depth Logger for FORCE Wrote a driver and parser for the SAMI pH sensor to be deployed at Cambridge Bay Now have the ability to acquire and parse navigation data from the Oceanic Explorer ROV
WARN (Web-enabled Awareness Research Network) Out of range values for Pd and Tau, the two parameters sent by the accelerometers that are used to compute the epicentre location, are now automatically rejected.
Miscellaneous SeaScribe and SeaTube now support multiple cameras. Created the scalardata web service so external users can access all scalar data. Added the ability to monitor and control Serial Control Units (SCUs) using Device Console. Plus SCU alarms now send SNMP traps. Created Wiring the Abyss pages on oceannetworks.ca for the upcoming dives. Modified the scalardata web service to use sensorcodename rather than sensorcode. For more details see the internal release notes (restricted access): 2015-09-02 Release Notes
5 August 2015 Data Search and Data Products Added interactivity between the search tree and the map when Sort by "Variables by Location" has been selected. Ie. the map now shows the location of the primary sensor when a sensor is selected on the search tree, and vice versa. Data search failures automatically generate tracking "tickets" so we can investigate the cause. Tickets generated by search failures outside the production environment (ie. QA and Development) now contain the word "TEST" to distinguish them from production issues. Added data products for the scalar sensors on the ASL AZFP Echosounders, such as tilt and temperature.
Login Free Access Sea Tube is now accessible without logging in. For instance directly via http://dmas.uvic.ca/SeaTube. The following features within SeaTube will prompt you to log in first: Adding an annotation Editing existing annotations Deleting an annotation Recording to create a playlist Playing public playlists The Device Details (DeviceListing) page is now accessible without logging in. For example if you make a plot using Plotting Utility without logging in, you are able to click on the Details link in the legend to bring up the device details. Or you can go directly there via http://dmas. uvic.ca/DeviceListing. Clicking on a Site link in the Site tab will require login.
oceannetworks.ca website The Wiring the Abyss pages now support multiple simultaneous cruises and dives.
WARN (Web-enabled Awareness Research Network) The CAP messages for earthquake and tsunami notification now include a digital signature. The client API verifies that the signature is present and valid before it will act on notifications. The API and sample client now display the distance from the user to the epicentre, in km.
Miscellaneous SeaScribe and SeaTube now support video feeds from multiple cameras on the same ROV dive. Did some initial design work and prototyping towards a tool that will allow previewing of data for a pre-defined set of instruments and recent time ranges. Created a "scalardata" web service to access any data for scalar sensors for any date range. The two types of Thermosalinographs have been merged into one type. Wrote a parser for the ZEUS II ROV Navigation device so that we can have georeferenced dive data.
2 July 2015 Data Search and Data Products On the Data Search step 1 page, in the "Sort By" selector, "Location" has been renamed to "Instruments by Location", and "Instrument" has been renamed to "Instruments by Category". This is related to an upcoming feature where users will be able to search "Variables by Location". This will replicate and improve upon the "Search by Water Property" function on the VENUS data download page. Added station names to plot titles for complex data such as ADCP. MATLAB data products for mobile devices (ie. all except CSV) now state the range of positioning data (latitude, longitude, depth) for the applicable data, instead of 'NaN'. When using Sort by Location, complex data products now have filenames that contain the location name, similar to scalar data products. Added data products for Imagenex profiling and imaging sonars (primarily on the VENUS network, may be temporarily disabled pending review). Added data products for Sequoia LISST particle analyzers (primarily on the VENUS network, may be temporarily disabled pending review). Improved the status messages when no data is found to aid users in understanding why there's no data and to help find data. Nortek raw binary data products (AQD, PRF, VNO, VEC) will no longer be unavailable when the configuration is unavailable (a warning will be issued in that case, but users will still be able to the get the data).
SeaTube Made most features available without having to log in. This can be accessed from oceannetworks.ca via the Video page on Sights & Sounds. The following features still require logging in: adding, deleting or editing an annotation creating a playlist taking snapshots
WARN (Web-enabled Awareness Research Network) Created a Java API for users who wish to subscribe to and receive earthquake and/or tsunami notifications on their PC to compute time-to-shaking and shaking intensity. Wrote client software that uses the API and displays notifications on a console, as an example of how the API can be used by clients. Improved the earthquake notification messages to add the number of sensors that contributed to each detection and notification.
Miscellaneous Added sensor units to the list of variables returned by the DeviceInfoService web service. Did some initial work on designing a web page to quickly preview data for fixed time ranges.
1 June 2015
Data Search and Data Products Added a new search feature to Data Search (and Plotting Utility) called "Sort by: Primary Sensor" (currently disabled). This is useful when you want to search for data by property but you aren't sure what instruments measure that property. Or there may be more than one instrument but you don't know which one is the best one to use. To use Sort by: Primary Sensor you first select the location you want, then you select the water (or air) property without being concerned about what instrument is measuring that property. If there is more than one instrument, this feature will automatically select the one that we consider to be the best one to use. The software has been written but this feature will not be enabled until testing is completed. For data products for the ZAP (Zooplankton Acoustic Profiler), improved the Config field to show which fields are instrument configuration and which are post process information. Hydrophone data products can now make use of NAXYS calibration files.
Plotting Utility Added a new feature - "Sort by: Primary Sensor". See Data Search above (currently disabled). Developed a new version of Plotting Utility where most features can be accessed without needing to have an account or having to log in. This is linked to the oceannetworks.ca website from the Preview & Download page (under Data & Tools). All the usual features will work except for the following: Details, Docs, and Download Data links Saving plots
WARN (Web-enabled Awareness Research Network) Finished thoroughly testing the Earthquake Correlator and fixing all issues that were found.
Miscellaneous Improved the reprocessing service to handle piggyback devices that are moved to a new parent device on the same day. Work was done to support live divelog video feeds from simultaneous cruises, for the Wiring the Abyss webpage and SeaTube.
4 May 2015 Data Search and Data Products Created data products for the CODAR in the Salish Sea including RUV, TUV, and PNG formats. This device is listed under Marine Radar System. Data products have been created for the British Columbia ferry instruments located in the Salish Sea. Added MAT, PNG, and PDF data products for ZAP (Zooplankton Acoustic Profiler).
QA/QC (quality control) Modified the way data can be manually flagged as good or bad - what we call the "manual QA/QC process". The process is now simpler and more foolproof in that if you need to change the flags on data that already has QA/QC flags, you merely have to state what new flags you want and what time ranges they apply to. You don't have to concern yourself about the state of existing flags when creating your manual QA/QC file. It is now possible to flag data as bad as soon as the data is received - ie. automatically. This is useful if it is known that the instrument is not performing correctly.
WARN (Web-enabled Awareness Research Network) Completed the web service support for CANARIE. This was the final piece to complete the NEP-105 CANARIE-supported project
(however additional WARN development is taking place under a new project). The web service confirms to CANARIE that WARN is operational and reports the number of earthquakes and tsunamis reported in the past 24 hours. It also provides CANARIE with links to documentation on the project. Implemented a server "push" notification method to notify external clients of earthquakes and tsunamis. Later an example client application will be developed to demonstrate how this system works, and to provide a template for users who wish to develop their own client application.
Miscellaneous Added a link to the File Upload feature on the Tools menu. This is an existing feature to upload annotations but had been a hidden page up to now. Now support multiple resolutions of time lapse video in SeaTube for the FORCE (Atlantic) observatory. Added support for primary sensors in ScalarDataAPIService, DeviceService, and in Plotting Utility. This will eventually be used to support expanded preview plots in the oceannetworks.ca website.
Ocean Networks website (oceannetworks.ca) Created "state of the ocean" preview plots for key locations in the NEPTUNE, VENUS, and Cambridge Bay (Arctic) observatories. These show key ocean parameters such as salinity and temperature over a period of several years. They do not require logging into an account in order to view them. See "Data and Tools" followed by "Preview and Download", then "State of the Ocean".
1 April 2015 Data Search and Data Products We now handle null site positions and mobile position offsets in data products.
Instruments Upgraded to handle new sensors and new data format for Ocean Sonics AF and HF hydrophones. New calibration formulas for the second generation SIIMs. Now parsing the State of Health (SOH) data from seismometers.
Miscellaneous Now tracking activity on the login and registration pages with Google Analytics so we can learn about problems our users may be having in these areas. A special web page was added to Oceans 2.0 to display live data from the Strait of Georgia for the Vancouver Aquarium.
26 February 2015 Data Search and Data Products You are now able to zoom in on the Data Search map as far as Google Maps allows. Improved the performance of obtaining device information once you have selected a device in Data Search. Completely overhauled the header of CSV files to give more information and to make it consistent regardless of which network the instrument is located on. Also modified the format of the timestamp in the data area to make it more readable.
Digital Fishers Modified the code to return to the default video playback behaviour in preparation for a new campaign. The only exception is that clips in the new campaign, like the old one, play for 60 seconds rather than 15 seconds. Modified the tutorial graphics at all levels to show specific sea life examples for the new campaign. Modified the annotation choices for the new campaign. Deployed the new campaign called "Feeding Deep Sea Critters" on February 26, 2015.
OPeNDAP Parsed RDI ADCP data is now available using ERDDAP. See OPeNDAP Web Services on the Tools menu. Made is easier for ONC data stewards to configure ERDDAP data sets. Made it easier to maintain ONC sensor mappings to CF standards.
Miscellaneous The hydrophone viewer now has additional choices for annotating marine mammals - northern right whale dolphin, Baird's beaked whale, and Cuvier's beaked whale Made the Citizen Scientist app more reliable. Merged the SoH (State of Health) seismometer FTP jobs and made the job configurable so that one job could be used for all current and future SoH seismometers. In Device Console, made the list of Junction Boxes and SIIMs more descriptive, added Oceanworks devices, and sorted the list by site. Created an FTP job for downloading data from the Queen of Oak Bay ferry. Implemented a server notification method for WARN (Web-enabled Awareness Research Network) in order to notify external users of earthquakes and tsunamis.
3 February 2015 Data Search and Data Products It is now possible to select an instrument for searching data by clicking on it on the Data Search map or in the usual way using the search tree. Modified the metadata reports to show the full path for hyperlinks so this information would not be lost if the report was printed. Removed the Network Topology "Sort by" option from Data Search. Made the initial changes to support a new header format for the CSV data product. Improvements for Nortek ADCP data products: more information within the .mat file products on what was done to the data, improved handling of mobile data, particularly headings and rotations.
WARN (Web-enabled Awareness Research Network) Added a web service so that CANARIE can confirm that WARN is operational. Completed most of the work for the earthquake detection correlator.
Miscellaneous Increased the number of daily "casts" (down-up transits) of the Buoy Profiling System in Saanich Inlet from 2 to 4. Added the ability to control the new camera system at Brentwood College (Mill Bay site). Digital Fishers code was modified to support changes for the upcoming campaign titled "Feeding Critters".
Internal Added support for Google Universal Analytics Modified the way quarter scalar data is generated in order to improve reliability and speed. Wrote a driver for the Sea-Bird Microcat SBE37SMP-ODO and Nortek AWAC which will be deployed at the FORCE observatory in the Bay of Fundy. Added support for VENUS SIIM devices on the Junction Box Alarm Settings page. Wrote a parser for the RBR Concerto device used in the Citizen Scientist project so that data can be viewed with Plotting Utility.
