API Reference¶
Survey Classes (Equipment)¶
Survey¶
Bases: object
Super class from which specific equipment surveys are created
__init__(*, filepath='', site_name=None, site_id=None, date=None, time=None, latitude=np.nan, longitude=np.nan, survey_type='Unknown', equipment_class='Unknown', **kwargs)
¶
Attributes:
| Name | Type | Description |
|---|---|---|
filepath |
str
|
path to raw survey folders or files |
site_name |
str
|
name of location surveyed |
site_id |
str
|
id of location surveyed |
date |
TimeStamp
|
date on which survey was conducted |
time |
TimeStamp
|
time at which survey began |
latitude |
float
|
latitudinal coordinate of survey site |
longitude |
float
|
longitudinal coordinate of survey site |
survey_type |
str
|
category of instrument used to record the survey |
equipment_class |
str
|
name of the instrument used, usually including manufacturer and model number |
identify_unique_file_type(file_type)
¶
Identify name of first unique file of specified type in survey folder
Returns:
| Type | Description |
|---|---|
str
|
file path for file with specified file type suffix |
require_site_id()
¶
Request user input for sites without an ID
Returns:
| Type | Description |
|---|---|
str
|
user entered string for survey site id |
return_coordinates()
¶
Return location as shapely Point geometry
Returns:
| Type | Description |
|---|---|
Point
|
pair of coordinates in EPSG:4326 |
Sontek RS5 ADCP¶
Bases: Survey
Survey object with additional attributes and methods to capture data unique to the Sontek RS5 ADCP
__init__(**kwargs)
¶
Attributes:
| Name | Type | Description |
|---|---|---|
transects |
list
|
store for individual Transect objects that contribute to the survey |
metadata |
DataFrame
|
contains basic survey metadata captured from multiple files generated during the survey |
dis |
DataFrame
|
contains transect level metadata from the generated .dis file |
extract_dis()
¶
Extract survey metadata and transect summary data from first .dis file
Returns:
| Type | Description |
|---|---|
TimeStamp
|
date on which survey was conducted |
str
|
name of location surveyed |
TimeStamp
|
time at which survey began |
float
|
survey discharge in m3/s |
float
|
mean survey depth in m |
Dataframe
|
survey metadata information |
Dataframe
|
transect metadata information, one row per transect |
extract_kml_geodata()
¶
Extract transect geotracking data as geodataframes
Returns:
| Type | Description |
|---|---|
list
|
names of each layer in a .kml file |
extract_kml_metadata()
¶
Extract transect metadata specific to the .kml file
Returns:
| Type | Description |
|---|---|
list
|
listed dictionaries of specific metadata values |
str
|
file path for file with .kml suffix |
generate_transects(kml_metadata, kml_layers)
¶
Create transects for survey as transect class using metadata from .kml file
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kml_metadata
|
list
|
contains dictionaries with unique metadata for each transect extracted from the kml files |
required |
kml_layers
|
list
|
contains geospatial information for each type of tracking used on the Transects extracted from the kml files |
required |
Returns:
| Type | Description |
|---|---|
Transect
|
appends a Transect object to self.transect for each cross-section conducted in the survey |
get_location()
¶
Use first geo-location reading of first transect to gather survey location
Returns:
| Type | Description |
|---|---|
float
|
latitudinal coordinate in EPSG:4326 |
float
|
longitudinal coordinate in EPSG:4326 |
identify_unique_file_type(file_type)
¶
Identify name of first unique file of specified type in survey folder
Returns:
| Type | Description |
|---|---|
str
|
file path for file with specified file type suffix |
ingest_survey()
¶
Top level function to ingest an ADCP survey
require_site_id()
¶
Request user input for sites without an ID
Returns:
| Type | Description |
|---|---|
str
|
user entered string for survey site id |
return_coordinates()
¶
Return location as shapely Point geometry
Returns:
| Type | Description |
|---|---|
Point
|
pair of coordinates in EPSG:4326 |
Sontek RS5 ADCP – Transect¶
Bases: object
Unique to Sontek ADCP, a transect is a single cross-section that forms part of a complete survey, it is almost comparable to an entire survey for a FlowStick
__init__(survey, raw_id, processed_id, type, track_reference, depth_reference, start_edge, geopath)
¶
Attributes:
| Name | Type | Description |
|---|---|---|
survey |
SontekADCPSurvey
|
survey which the transect is a part of |
raw_id |
str
|
id automatically assigned to the raw transect, before is checked and either accepted or discarded |
processed_id |
str
|
id assigned to the transect after being accepted |
type |
str
|
shape of the transect conducted, usually linear cross-section or loop |
datetime |
TimeStamp
|
start time of the transect |
track_refernce |
str
|
which of the geo-tracking methods was used for the final output |
depth_reference |
str
|
which of the depth sensors was used for the final output |
start_edge |
str
|
which bank of the water body did the survey begin, based on looking downstream |
metadata |
DataFrame
|
contains transect metadata from survey level .dis file |
geopath |
str
|
name of the layer in the .kml that stores the final geotracking information for the transect |
geodata |
GeoDataFrame
|
contains geospatial information extract from the .kml file |
sum |
DataFrame
|
contains summary information from ensemble (vertical column) that makes up each transect, extracted from .sum file |
vel |
DataFrame
|
contains velocity information from each cell (horizontal slice of an ensemble), extracted from .ENU.vel file |
snr |
DataFrame
|
contains sensor information from each of the four sensors from each cell (horizontal slice of an ensemble), extracted from .snr file |
extract_snr()
¶
Extract sensor data from transect .snr file
Returns:
| Type | Description |
|---|---|
DataFrame
|
transect sensor data |
extract_sum()
¶
Extract summary data from transect .sum file
Returns:
| Type | Description |
|---|---|
DataFrame
|
transect summary data |
extract_vel()
¶
Extract velocity data from transect .vel file
Returns:
| Type | Description |
|---|---|
DataFrame
|
transect velocity data |
get_datetime()
¶
Get transect datetime using reading from first data collection
Returns:
| Type | Description |
|---|---|
TimeStamp
|
datetime transect began |
get_geodata(kml_path)
¶
Pull transect geodata from survey .kml data
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kml_path
|
str
|
path to survey kml file |
required |
Returns:
| Type | Description |
|---|---|
GeoDataFrame
|
contains transect geotracking data |
get_metadata(survey)
¶
Pull transect metadata from survey .dis data
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
Survey
|
survey transect belongs to |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
single row dataframe with transect metadata |
get_processed_id(survey)
¶
Check survey .dis file to see if transect was processed and given new ID
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
Survey
|
survey transect belongs to |
required |
Returns:
| Type | Description |
|---|---|
str
|
transect id if passed post-survey checks by user |
identify_transect_file_type(file_type)
¶
Identify name of the file in survey folder belonging to specific transect of a specified file type
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_type
|
str
|
suffix of the file to be retrieved |
required |
Returns:
| Type | Description |
|---|---|
str
|
path to specified file type |
plot_geodata()
¶
Plot comparison of three transect geotracking methods for specific transect
Returns:
| Type | Description |
|---|---|
plot
|
plot of the three onboard geotracking methods |
Sontek FlowTracker2¶
Bases: Survey
Survey object with additional attributes and methods to capture data unique to the Sontek FlowTracker2
__init__(**kwargs)
¶
Attributes:
| Name | Type | Description |
|---|---|---|
metadata |
DataFrame
|
contains basic survey metadata generated during the survey |
dis |
DataFrame
|
contains vertical level metadata from the generated .dis file |
sum |
DataFrame
|
contains summary information from each vertical that makes the survey, extracted from .sum file |
dat |
DataFrame
|
contains summary information from each vertical that makes the survey, extracted from .dat file |
ctl |
DataFrame
|
contains information for equipment configuration during the survey, extracted from .ctl file |
extract_ctl()
¶
Extract survey controller data from first .ctl file
Returns:
| Type | Description |
|---|---|
DataFrame
|
survey equipment configuration metadata |
extract_dat()
¶
Extract survey data from first .dat file
Returns:
| Type | Description |
|---|---|
DataFrame
|
vertical summary data from specific variables including temperature |
extract_dis()
¶
Extract survey metadata and transect summary data from first .dis file
Returns:
| Type | Description |
|---|---|
DataFrame
|
vertical level metadata from the generated .dis file |
str
|
name of location surveyed |
TimeStamp
|
date on which survey was conducted |
TimeStamp
|
time at which survey began |
float
|
latitudinal value of survey location |
float
|
longitudinal value of survey location |
float
|
survey discharge in m3/s |
float
|
survey temperature in degC |
float
|
mean survey depth in m |
DataFrame
|
basic survey metadata generated during the survey |
extract_sum()
¶
Extract survey summary data from first .sum file
Returns:
| Type | Description |
|---|---|
DataFrame
|
survey summary data |
get_trusted_site_id()
¶
If we trust the source of the site ID, extract it
Returns:
| Type | Description |
|---|---|
str
|
site id with blank spaces removed |
identify_unique_file_type(file_type)
¶
Identify name of first unique file of specified type in survey folder
Returns:
| Type | Description |
|---|---|
str
|
file path for file with specified file type suffix |
ingest_survey(trust_id=False)
¶
Top level function to ingest a FlowTracker survey
require_site_id()
¶
Request user input for sites without an ID
Returns:
| Type | Description |
|---|---|
str
|
user entered string for survey site id |
return_coordinates()
¶
Return location as shapely Point geometry
Returns:
| Type | Description |
|---|---|
Point
|
pair of coordinates in EPSG:4326 |
Nivus FlowStick¶
Bases: Survey
Survey object with additional attributes and methods to capture data unique to the Nivus FlowStick
__init__(**kwargs)
¶
Attributes:
| Name | Type | Description |
|---|---|---|
metadata |
DataFrame
|
contains basic survey metadata generated during the survey |
sections |
DataFrame
|
contains informations about measurements calculated for the area between two verticals (points) |
points |
DataFrame
|
contains information from each vertical measured as part of the survey |
extract_csv()
¶
Extract survey metadata and data from survey .csv file
Returns:
| Type | Description |
|---|---|
DataFrame
|
basic survey metadata generated during the survey |
DataFrame
|
informations about measurements calculated for the area between two verticals (points) |
DataFrame
|
information from each vertical measured as part of the survey |
str
|
name of location surveyed |
TimeStamp
|
date on which survey was conducted |
TimeStamp
|
time at which survey began |
float
|
survey discharge in m3/s |
float
|
survey temperature in degC |
float
|
mean survey depth in m |
identify_unique_file_type(file_type)
¶
Identify name of first unique file of specified type in survey folder
Returns:
| Type | Description |
|---|---|
str
|
file path for file with specified file type suffix |
ingest_survey()
¶
Top level function to ingest a FlowStick survey
require_site_id()
¶
Request user input for sites without an ID
Returns:
| Type | Description |
|---|---|
str
|
user entered string for survey site id |
return_coordinates()
¶
Return location as shapely Point geometry
Returns:
| Type | Description |
|---|---|
Point
|
pair of coordinates in EPSG:4326 |
OTT MF Pro¶
Bases: Survey
Survey object with additional attributes and methods to capture data unique to the Nivus FlowStick
__init__(**kwargs)
¶
Attributes:
| Name | Type | Description |
|---|---|---|
metadata |
DataFrame
|
contains basic survey metadata generated during the survey |
data |
DataFrame
|
contains information gathered at each vertical during the survey |
convert_flow_to_m3s()
¶
Convert metadata value from litres/second to m^3/second
Returns:
| Type | Description |
|---|---|
DataFrame
|
survey metadata with flow in m^3/s |
DataFrame
|
survey data with flow in m^3/s |
convert_to_m()
¶
Convert data columns from metres to centimetres
Returns:
| Type | Description |
|---|---|
DataFrame
|
survey data with distance values in m |
extract_from_tsv()
¶
Extract survey metadata and data from survey .tsv file
Returns:
| Type | Description |
|---|---|
TimeStamp
|
date on which survey was conducted |
TimeStamp
|
time at which survey began |
DataFrame
|
basic survey metadata generated during the survey |
DataFrame
|
contains information gathered at each vertical during the survey |
identify_unique_file_type(file_type)
¶
Identify name of first unique file of specified type in survey folder
Returns:
| Type | Description |
|---|---|
str
|
file path for file with specified file type suffix |
ingest_survey()
¶
Top level function to ingest an OTT MF Pro survey
require_site_id()
¶
Request user input for sites without an ID
Returns:
| Type | Description |
|---|---|
str
|
user entered string for survey site id |
return_coordinates()
¶
Return location as shapely Point geometry
Returns:
| Type | Description |
|---|---|
Point
|
pair of coordinates in EPSG:4326 |
update_metadata()
¶
Update class attributes using values from metadata
Returns:
| Type | Description |
|---|---|
str
|
site name pulled from metadata |
float
|
total discharge pulled from metadata |
float
|
mean depth pulled from metadata, converted to m if required |
Extract Surveys¶
extract_nivusflowstick_surveys(path)
¶
Extract Nivu[s] FlowStick data from the raw files
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
path to where raw survey data folders are contained |
required |
Returns:
| Type | Description |
|---|---|
list
|
all NivuFlowStick objects successfully extracted from the raw files |
extract_ottmfpro_surveys(path)
¶
Extract OTT MF Pro data from the raw files
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
path to where raw survey data files are contained |
required |
Returns:
| Type | Description |
|---|---|
list
|
all OTTMFProSurvey objects successfully extracted from the raw files |
extract_sontekadcp_surveys(path, plot_transects=False)
¶
Extract Sontek RS5 ADCP survey level and transect level data from the raw files
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
path to where raw survey data folders are contained |
required |
plot_transects
|
bool
|
whether transect geotracks should be shown when being imported |
False
|
Returns:
| Type | Description |
|---|---|
list
|
all SontekADCPSurvey objects successfully extracted from the raw files |
extract_sontekflowtracker_surveys(path, trust_id=False)
¶
Extract Sontek FlowTracker2 data from the raw files
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
path to where raw survey data folders are contained |
required |
Returns:
| Type | Description |
|---|---|
list
|
all SontekFlowTrackerSurvey objects successfully extracted from the raw files |
Plot Surveys¶
add_bar_legend(ax, pc, label, profile_type_colours)
¶
Add custom combined legend if beam type bar included in plot
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ax
|
axis
|
plot containing the cross-section |
required |
pc
|
PatchCollection
|
cross-section cells including fill colour |
required |
label
|
str
|
title for the colour bar legend including sensor number and units |
required |
profile_type_colours
|
dict
|
maps beam type to the colour of the bar section |
required |
Returns:
| Type | Description |
|---|---|
axis
|
plot with added colour bar at bottom of plot |
create_bar_sections(df, widths)
¶
Create bar showing each vertical of the transect, with each section filled according to the beam type used
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
transect velocity / sensor and location information |
required |
widths
|
list
|
contains width of each vertical in the cross-section |
required |
Returns:
| Type | Description |
|---|---|
list
|
contains sub-lists for start point and width of each bar section |
list
|
colours to be used on each section of the bar |
dict
|
maps beam type to the colour of the bar section |
create_plot_df(transect)
¶
Adds required data for cross-section plot into a single dataframe
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
transect
|
TransectADCP
|
transect to be plotted |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
dataframe containing velocity and location data for the chosen transect |
crop_transect_df(df)
¶
Crops start of dataframe to remove rows where the ADCP goes back on itself (happens at start of most ADCP surveys)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
transect velocity / sensor and location information |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
cropped dataframe from last timestep after ADCP travels backwards |
generate_dB_cells(df, snr_df, widths, s_tpl, loc_tpl, Sx)
¶
Generate the cells and corresponding values for a transect
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
transect velocity and location information |
required |
snr_df
|
DataFrame
|
transect sensor data |
required |
widths
|
list
|
contains width of each vertical in the cross-section |
required |
s_tpl
|
str
|
template name ready for infilling with specific cell being looked at (used to get df column) i.e. Cell© S{Sx} (dB) |
required |
loc_tpl
|
str
|
template name ready for infilling with specific cell being looked at (used to get df column) i.e. Cell{c} Loc (m) |
required |
Sx
|
str
|
which of the 4 sensors decibel measurements to plot |
required |
Returns:
| Type | Description |
|---|---|
list
|
all cell co-ordinates for the transect |
list
|
all cell decibel values for the transect |
generate_velocity_cells(df, widths, v_tpl, loc_tpl, Vx)
¶
Generate the cells and corresponding values for a transect
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
transect velocity / sensor and location information |
required |
widths
|
list
|
contains width of each vertical in the cross-section |
required |
v_tpl
|
str
|
template name ready for infilling with specific cell being looked at (used to get df column) i.e. Cell© V{Vx} (m/s) |
required |
loc_tpl
|
str
|
template name ready for infilling with specific cell being looked at (used to get df column) i.e. Cell{c} Loc (m) |
required |
Vx
|
str
|
which of the 4 sensors velocity measurements to plot |
required |
Returns:
| Type | Description |
|---|---|
list
|
all cell co-ordinates for the transect |
list
|
all cell velocity values for the transect |
plot_bar(ax, bar_segments, bar_y, bar_height, bar_colours)
¶
Plot the bar
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ax
|
axis
|
plot containing the cross-section |
required |
bar_segments
|
list
|
contains sub-lists for start point and width of each bar section |
required |
bar_y
|
float
|
y-axis co-ordinate for where to plot the bar |
required |
bar_height
|
float
|
final height of the bar based on depth of cross-section |
required |
bar_colours
|
list
|
colours to be used on each section of the bar |
required |
Returns:
| Type | Description |
|---|---|
axis
|
plot with added colour bar at bottom of plot |
plot_transect_cross_section(transect, crop_start=True, include_PCBB_depth=False, Vx='V1', show_beam_type=False, gradient='BuPu')
¶
Plot a single transect velocity values (cross-section)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
transect
|
TransectADCP
|
chosen transect object for plotting |
required |
crop_start
|
bool
|
whether to remove the portion of the transect where the ADCP is let to drift and then pulled back before the 'official' crossing begins |
True
|
include_PCBB_depth
|
bool
|
whether to plot the depth including the maximum depth reached by the PCBB type beam (PC is standard and desired for a vertical), usually these record depths significantly below the river bed so can hinder the y-axis of the plot |
False
|
Vx
|
str
|
which of the 4 sensors velocity measurements to plot (V1 is standard as it is most directly in the direction of downstream flow) |
'V1'
|
show_beam_type
|
bool
|
whether to include bar showing which beam type (PC / PCBB / BB) measured each vertical |
False
|
gradient
|
str
|
matplotlib colormap to use for the plot |
'BuPu'
|
Returns:
| Type | Description |
|---|---|
axis
|
complete transect (cross-section) plot |
plot_transect_sensor_cross_section(transect, crop_start=True, include_PCBB_depth=True, Vx='V1', show_beam_type=True, gradient='BuPu', Sx='SNR1')
¶
Plot a single transect sensor decibel values (cross-section)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
transect
|
TransectADCP
|
chosen transect object for plotting |
required |
crop_start
|
bool
|
whether to remove the portion of the transect where the ADCP is let to drift and then pulled back before the 'official' crossing begins |
True
|
include_PCBB_depth
|
bool
|
whether to plot the depth including the maximum depth reached by the PCBB type beam (PC is standard and desired for a vertical), usually these record depths significantly below the river bed so can hinder the y-axis of the plot |
True
|
Vx
|
str
|
which of the 4 sensors velocity measurements to use for calculating the mask depth (if include_PCBB_depth=False this ensures the plot is correctly proportioned on the y-axis) |
'V1'
|
show_beam_type
|
bool
|
whether to include bar showing which beam type (PC / PCBB / BB) measured each vertical |
True
|
gradient
|
str
|
matplotlib colormap to use for the plot |
'BuPu'
|
Sx
|
str
|
which of the 4 sensors decibel measurements to plot |
'SNR1'
|
Returns:
| Type | Description |
|---|---|
axis
|
complete transect (cross-section) plot |
set_bar_shape(y_max, cell_half_height_global)
¶
Set the height of the complete bar for adding to the plot
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_max
|
float
|
maximum depth value recorded across any vertical |
required |
cell_half_height_global
|
float
|
half the median height of a cell in the cross-section |
required |
Returns:
| Type | Description |
|---|---|
float
|
y_max update to account for the height of the bar |
float
|
y-axis co-ordinate for where to plot the bar |
float
|
final height of the bar based on depth of cross-section |
Export to Timeseries¶
export_timeseries(timeseries_df, export_path)
¶
Export a single site timeseries as parquet file (.csv to be added)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeseries_df
|
DataFrame
|
dataframe contaning timeseries information for all surveyed sites |
required |
export_path
|
str
|
location of the top level output folder |
required |
Returns:
| Type | Description |
|---|---|
parquet
|
n files, one per site, named after the site, with date, flow and sensor metadata |
get_timeseries_row(survey)
¶
Extract timestamp and discharge information from a survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
Survey
|
any survey type object e.g. Sontek ADCP, Sontek FT2, OTT MF Pro |
required |
Returns:
| Type | Description |
|---|---|
dict
|
timeseries metadata including site id |
handle_datetime_formats(df)
¶
Temporary function
surveys_to_fdri_timeseries(surveys, export_path)
¶
Export to UKCEH Observatory¶
check_site_exists(site_id, export_path)
¶
export_sites(surveys, catchment, export_path)
¶
Export sites surveyed in Observatory csv format
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
catchment
|
NRFA ID for the catchment e.g. Chess = 39088 |
required | |
dataframe
|
name of the class attribute containing the dataframe for export |
required | |
export_path
|
location of the top level output folder |
required |
Returns:
| Type | Description |
|---|---|
csv file
|
contains site name, id, location and survey date information |
export_timeseries(timeseries_df, export_path)
¶
Export a single site timeseries in Observatory csv format
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeseries_df
|
dataframe contaning timeseries information for all surveyed sites |
required | |
export_path
|
location of the top level output folder |
required |
Returns:
| Type | Description |
|---|---|
csv file
|
n files, one per site, named after the site, with date, flow and sensor metadata |
get_data_rows(survey, data_df, cols_to_keep=[], col_naming_map={})
¶
Extract data information from a survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
any survey type object e.g. Sontek ADCP, Sontek FT2, OTT MF Pro |
required | |
data_df
|
contains any and all information from the raw survey data file |
required | |
cols_to_keep
|
columns to be shown on the Observatory, requested by GN to be maximum of 20 columns |
[]
|
|
col_naming_map
|
map for raw column names to Observatory format names |
{}
|
Returns:
| Type | Description |
|---|---|
dict
|
basic survey data with the addition of site id, name and datetime information |
get_metadata_row(survey, metadata_df, cols_to_keep=[], col_naming_map={})
¶
Extract metadata information from a survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
Survey
|
any survey type object e.g. Sontek ADCP, Sontek FT2, OTT MF Pro |
required |
metadata_df
|
DataFrame
|
contains any and all information from the raw survey metadata file |
required |
cols_to_keep
|
list
|
columns to be shown on the Observatory, requested by GN to be maximum of 20 columns |
[]
|
col_naming_map
|
dict
|
map for raw column names to Observatory format names |
{}
|
Returns:
| Type | Description |
|---|---|
dict
|
basic survey metadata including site id, name and datetime information |
get_timeseries_row(survey)
¶
Extract timestamp and discharge information from a survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
Survey
|
any survey type object e.g. Sontek ADCP, Sontek FT2, OTT MF Pro |
required |
Returns:
| Type | Description |
|---|---|
dict
|
timeseries metadata including site id |
nivusflowstick_to_observatory(surveys, export_path, catchment)
¶
Top level function for converting Nivu(s) Flowstick surveys into a set of csv files in Observatory format, outputs from nested functions are: - sites.csv - timeseries - 1x .csv per site - metadata - 1x .csv per survey - data - 1x .csv per survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
export_path
|
location of the top level output folder |
required | |
catchment
|
NRFA ID for the catchment e.g. Chess = 39088 |
required |
ott_mf_pro_to_observatory(surveys, export_path, catchment)
¶
Top level function for converting OTT MF Pro surveys into a set of csv files in Observatory format, outputs from nested functions are: - sites.csv - timeseries - 1x .csv per site - metadata - 1x .csv per survey - data - 1x .csv per survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
export_path
|
location of the top level output folder |
required | |
catchment
|
NRFA ID for the catchment e.g. Tweed = 21014 |
required |
sites_to_observatory(surveys, catchment, existing_sites, ids)
¶
Convert sites surveyed in dataframe format ready for export to csv
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
catchment
|
NRFA ID for the catchment e.g. Chess = 39088 |
required | |
existing_sites
|
metadata for sites with previously ingested survey |
required | |
ids
|
list of site ids already in site metadata export |
required |
Returns:
| Type | Description |
|---|---|
Dataframe
|
contains metadata for all sites surveyed |
sontek_adcp_to_observatory(surveys, export_path, catchment)
¶
Top level function for converting Sontek ADCP surveys into a set of csv files in Observatory format, outputs from nested functions are: - sites.csv - timeseries - 1x .csv per site - metadata - 1x .csv per survey - data - 1x .csv per survey - ensemble data - 1x .csv per survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
export_path
|
location of the top level output folder |
required | |
catchment
|
NRFA ID for the catchment e.g. Chess = 39088 |
required |
sontek_flowtracker_to_observatory(surveys, export_path, catchment)
¶
Top level function for converting Sontek FlowTracker2 surveys into a set of csv files in Observatory format, outputs from nested functions are: - sites.csv - timeseries - 1x .csv per site - metadata - 1x .csv per survey - data - 1x .csv per survey
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
export_path
|
location of the top level output folder |
required | |
catchment
|
NRFA ID for the catchment e.g. Chess = 39088 |
required |
Utils¶
add_survey_id_columns(survey, dataframe)
¶
Add metadata columns to a survey related dataframe
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
survey object which the transect belongs to |
required | |
dataframe
|
dataframe metadata columns to be added to |
required |
Returns:
| Type | Description |
|---|---|
Dataframe
|
updated class attribute dataframe with added columns |
add_transect_id_columns(survey, transect, dataframe)
¶
Add metadata columns to a transect related dataframe
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
survey
|
survey object which the transect belongs to |
required | |
transect
|
transect object containing the relevant dataframe |
required | |
dataframe
|
name of the class attribute containing the dataframe for export |
required |
Returns:
| Type | Description |
|---|---|
Dataframe
|
updated class attribute dataframe with added columns |
export_group_survey_data(surveys, survey_type, dataframe, export_path, file_out_name, add_metadata, file_type)
¶
Export survey data and metadata to csv files
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
survey_type
|
type of surveys in list, used for appropriately storing output files |
required | |
dataframe
|
name of the class attribute containing the dataframe for export |
required | |
file_out_name
|
name to be given to the output file |
required | |
add_metadata
|
does this data need added metadata columns to identify the survey when grouped? |
required |
Returns:
| Type | Description |
|---|---|
csv
|
file containing combined data for all surveys in list |
export_group_transect_data(surveys, dataframe, export_path, file_out_name, file_type, survey_type='Sontek RS5 ADCP')
¶
Export survey data and metadata to csv files
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
surveys
|
contains survey class objects |
required | |
survey_type
|
type of surveys in list, used for appropriately storing output files |
'Sontek RS5 ADCP'
|
|
dataframe
|
name of the class attribute containing the dataframe for export |
required | |
file_out_name
|
name to be given to the output file |
required |
Returns:
| Type | Description |
|---|---|
file(csv / parquet)
|
file containing combined data for all surveys in list with the file type being determined on input |
export_to_csv(export_path, dataframe, survey_type, file_out_name)
¶
Export data to a csv file
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataframe
|
data for export |
required | |
survey_type
|
type of surveys in list, used for appropriately storing output files |
required | |
file_out_name
|
name to be given to the output file |
required |
Returns:
| Type | Description |
|---|---|
csv
|
file containing data |
export_to_parquet(export_path, dataframe, survey_type, file_out_name)
¶
Export data to a parquet file
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataframe
|
data for export |
required | |
survey_type
|
type of surveys in list, used for appropriately storing output files |
required | |
file_out_name
|
name to be given to the output file |
required |
Returns:
| Type | Description |
|---|---|
parquet
|
file containing data |
extract_transect_metadata_from_kml(kml_path)
¶
Function to extract specific metadata from a .kml file using xml.etree
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kml_path
|
location of the .kml file |
required |
Returns:
| Type | Description |
|---|---|
list
|
list dictionaries of specific metadata values |