oncvideo package

Funtions exported by the package

oncvideo.didson_info(source, output='DIDSON_info.csv')[source]

Get information from DIDSON file

Include file timestamp, duration, fps, number of beams and samples, window start (distance from DIDSON to first sample) and window length (distance from first sample to the last).

Parameters:

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *)

  • output (str, default 'DIDSON_info.csv') – Name of the csv file to save video information

oncvideo.download_files(source, output='output', trim=False)[source]

Download files from the table provided by source

Parameters:

  • source (str or pandas.DataFrame) – A pandas DataFrame, or a path to .csv file

  • output (str, default 'output') – Name of the output folder to save converted videos

  • trim (bool, default False) – Trim video files to match the initial seach query

oncvideo.download_st(onc, url, ext='mov')[source]

Generate download link from Seatube

Generate download link for the correspoding video based on the Seatube link provided. Useful for downloading the high-res video that is not avaiable in Seatube. Only supports Seatube V3.

Parameters:

  • onc (onc.ONC) – ONC class object

  • url (str) – The link generated by Seatube V3. E.g. https://data.oceannetworks.ca/SeaTubeV3?resourceTypeId=600&resourceId=4371&time=2023-09-13T20:43:09.000Z or csv file with ‘seatube_link’ column.

  • ext (str, default mov) – Especify a extension of the video to be downloaded.

oncvideo.download_ts(onc, source, category_code, output='output', options='fixed', nworkers=16)[source]

Donwload timeseries data for video files

Based on the filenames that are passed in the source, this function will download time series scalar data (tssd) that corresponds to the same time period as the filenames.

Parameters:

  • onc (onc.ONC) – ONC class object

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *). If a DataFrame or a .csv file, it must have a column ‘filename’ that follow the ONC convention or columns ‘timestamp’ and ‘deviceCode’.

  • category_code (str or list) – Category Code of data to download. E.g. NAV, CTD, OXYSENSOR, etc.

  • output (str, default 'output') – Name of the output folder to save files.

  • options (str, default 'fixed') – Set options for search querry. If ‘fixed’, return clean resampled data for every minute, and maximum gap of one day between queries. If ‘rov’, return raw not resampled data, and set a maximum gap of one hour between queries.

  • nworkers (int, default 16) – Number of simultaneous API calls to make

oncvideo.extract_fov(source, timestamps=None, duration=None, output='fovs', deinterlace=False)[source]

Extract FOVs from videos

Extract framegrabs or clips for each video at given timestamps

Parameters:

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *)

  • timestamps (str, float, or list) – A timestamp or a list of timestaps where the framegrabs will be extracted. Can be a float indicating the number of seconds or a string in the format mm:ss.f, from the start of the video. If None (the default), it will try to get the timestamps from the column ‘fovs’ in the source.

  • duration (float) – If provided, the function will get clips instead of framegrabs, with duration given in seconds. The start of the clips are given by ‘timestamps’.

  • output (str, default 'fovs') – Name of the output folder to save images/videos.

  • deinterlace (bool, default False) – Deinterlace video before getting the frames. This argument is ignored for clips, since the stream is copyied from the original video.

oncvideo.extract_frame(source, interval, output='frames', trim=False, deinterlace=False, rounding_near=False)[source]

Extract frames at a given interval

Parameters:

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *)

  • interval (float) – Interval, in seconds, to extract frames (e.g. a value of 10 will extract one frame every 10 seconds).

  • output (str, default 'output') – Name of the output folder to save converted videos

  • trim (bool, default False) – Trim video files to match the initial seach query

  • deinterlace (bool, default False) – Deinterlace video before getting the frames.

  • rounding_near (bool, default False) – Grab frames at the middle of each interval (default ffmpeg behaviour). If False, will grab frames at the start of each interval.

oncvideo.get_dives(onc, cache=True)[source]

Return all dives from Oceans3.0

Parameters:

  • onc (onc.ONC) – ONC class object

  • cache (bool, default True) – Save file at home folder and use it for future calls if the file is recent (less the 30 days).

Returns:

A DataFrame that includes dives code, start and end times for the dive, deviceCode and locationCode.

Return type:

pandas.DataFrame

Generate Seatube link

Generate a Seatube link from filenames following the Oceans 3 naming convention (deviceCode_timestamp.ext). For now, only supports video avaiable in Seatube V3 (videos from ROV cameras).

Parameters:

  • onc (onc.ONC) – ONC class object

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *). If a DataFrame or a .csv file, it must have a column ‘filename’ that follow the ONC convention or columns ‘timestamp’ and ‘deviceCode’.

  • dive (bool, default False) – Include a column in the output with dives where the video is from

Returns:

The dataFrame from source, with new column url with corresponding Seatube links.

Return type:

pandas.DataFrame

oncvideo.list_file(onc, deviceCode=None, deviceId=None, locationCode=None, dive=None, deviceCategoryCode='VIDEOCAM', dateFrom=None, dateTo=None, quality='ask', extension='mp4', statistics=True)[source]

Get list of files archived in Oceans 3.0

Search archived files based on one of the following criterias: decideCode, deviceId, locationCode and dive number. One of these parameters need to be suplied to the function.

Parameters:

  • onc (onc.ONC) – ONC class object

  • deviceCode (str) – Devide code to seach files

  • deviceId (str or int) – Devide Id to seach files

  • locationCode (str) – Location code to seach files

  • dive (str) – Dive number to seach files

  • deviceCategoryCode (str, default VIDEOCAM) – Device category code to search files. Only used when locationCode is suplied. Usually ‘VIDEOCAM’ for fixed cameras and ‘ROV_CAMERA’ for ROVs. ‘ask’ will list avaiable options and ask user to choose one.

  • dateFrom (str or datetime) – Return videos after specified datetime. Can be any format that is parsed by pandas.to_datetime. If None, will search all videos since the device was first deployed.

  • dateFrom – Return videos before specified datetime. Can be any format that is parsed by pandas.to_datetime. If None, will search all videos until the current date.

  • quality (str, default ask) – Especify a quality to filter videos. Usually should be LOW, standard, 1500, 5000, UHD. ‘ask’ will list avaiable options and ask user to choose one. ‘all’ will get all avaiable videos. Accepts multiple values as comma separated.

  • extension (str, default mp4) – Especify a extension to filter videos. ‘ask’ will list avaiable options and ask user to choose one. ‘all’ will get all avaiable videos. Accepts multiple values as comma separated.

  • statistics (bool, default True) – Also save video durations and file sizes.

Returns:

A DataFrame with the filenames, and videos duration and file sizes if statistics is True

Return type:

pandas.DataFrame

oncvideo.list_file_batch(onc, csvfile, quality='ask', extension='mp4', statistics=True, keep_cols=False)[source]

Batch list of files archived in Oceans 3.0 List video files based on parameters stored in a csv file

This function will execute multiple search for archived files based on parameters stored in a csv file. Usefull to get videos from multiple sites and/or dives. Column names must be one of decideCode, deviceId, locationCode or dive. Csv may also include parameters dateFrom, dateTo, and deviceCategoryCode. Details on these parameters can be found in ‘list_file’ function. A column named ‘group’ can also be used to distingshed each query in the final table.

Parameters:

  • onc (onc.ONC) – ONC class object

  • csvfile (str, path object or file-like object) – Location to the input csv file.

  • quality (str, default ask) – Especify a quality to filter videos. Usually should be LOW, standard, 1500, 5000, UHD. ‘ask’ will list avaiable options and ask user to choose one. ‘all’ will get all avaiable videos. Accepts multiple values as comma separated.

  • extension (str, default mp4) – Especify a extension to filter videos. ‘ask’ will list avaiable options and ask user to choose one. ‘all’ will get all avaiable videos. Accepts multiple values as comma separated.

  • statistics (bool, default True) – Also save video durations and file sizes.

  • keep_cols (bool, default False) – Keep other columns from csvfile into the output.

Returns:

A DataFrame with the filenames, and videos duration and file sizes if statistics is True

Return type:

pandas.DataFrame

oncvideo.make_timelapse(folder='fovs', time_display='elapsed', time_format=None, time_offset=0, fps=10, fontScale=1, logo=False, caption=None, time_xy=None, caption_xy=None)[source]

Generate timelapse video from images

Parameters:

  • folder (str, default 'fovs') – Path to a folder where .jpg images are stored.

  • time_display ({'elapsed', 'current', 'none'}) – How to print the time on the frame. ‘elapsed’ will display as elapsed time since first frame, offset by ‘time_offset’. ‘current’ will display the current real time of the frame. ‘none’ will not display time.

  • time_format (str, default '%Y/%m/%d %Hh' if time_display='current', and '%d days %{H}h' if time_display='elapsed') – Format how the timestamp will be writen on the video. For time_display=’current’, check formating options for ‘strftime’. For time_display=’elapsed’, options are %y %m %w %d %H %M %S for years, months, weeks, days, hours, minutes, seconds.

  • time_offset (str, timedelta or float, default 0) – Offset the time displayed in the frame if time_display=’elapsed’. Passed to pd.to_timedelta, check it’s documentation for options.

  • fps (float, default 10) – Timelapse video FPS.

  • fontScale (float, default 1) – Font scale for the timestamp. Increase values for a larger font size.

  • logo (bool, default False) – Include ONC logo on the video?

  • caption (str, default None) – Insert a caption at the bottom of the screen. You can break lines with <br> tag.

  • time_xy (tuple of 2 ints, default None) – Coordinates of the bottom-left corner of the time text. X is the distance (in pixels) from the left edge and Y is the distance from the top edge of the image. Default will draw in the top-left corner.

  • caption_xy (tuple of 2 int, default None) – Coordinates of the bottom-left corner of the first line of the caption. Default will draw in the bottom corner.

oncvideo.merge_ts(source, ts_data, tolerance=15, units=True)[source]

Merge timeseries data with timestamps

This function will get the timestamps from source and retrive the closest data avaiable inside the ts_data folder. If source is a DataFrame or a .csv file, it should have a column timestamp or filename, from which timestamps will be derived (filenames must follow Oceans naming convention)

Parameters:

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *). If a DataFrame or a .csv file, must have a column ‘filename’ that follow the ONC convention or columns ‘timestamp’ and ‘deviceCode’.

  • ts_data (str) – Folder containg csv files downloaded from Oceans 3

  • tolerance (float) – Tolarance, in seconds, for timestamps to be merged. If the nearest data avaiable from a given timestamp is higher than the tolarance, then a NaN is returned instead.

  • units (bool, default True) – Include units of the vairables in the column names. If False, units are removed.

Returns:

The dataFrame from source, with variables within ts_data merged based on the timestamps

Return type:

pandas.DataFrame

oncvideo.name_to_timestamp(filename)[source]

Get timestamp from a filename

Parameters:

filename (str) – A filename that follows the ONC convention (deviceCode_timestampUTC.ext).

Returns:

The timestamp object corresponding to the filename. The object also has attributes ‘dc’ (deviceCode) and ‘ext’ (file extension) for convenience.

Return type:

pandas.Timestamp

oncvideo.name_to_timestamp_dc(filenames)[source]

Get timestamps and deviceCode from filenames

Wrapper for name_to_timestamp to return deviceCode and timestamp from a pandas Series.

Parameters:

filenames (pandas.Series) – A str series with filenames that follow the ONC convention (deviceCode_timestampUTC.ext).

Returns:

A DataFrame with timestamps and deviceCodes

Return type:

pandas.DataFrame

oncvideo.onc(token=None)[source]

Create an ONC class object

Create an ONC class object, but try to use environment variables to get the API token. The token must stored under ‘ONC_API_TOKEN’.

Parameters:

token (str) – ONC API token

Returns:

ONC class object

Return type:

onc.ONC

oncvideo.read_ddf(urlfile)[source]

Get data from DIDSON file

Parameters:

urlfile (str) – Path to a file or a URL.

Returns:

  • numpy.array – A 1d array with the timestamp for each frame.

  • numpy.array – A 3d array as [frame, Beams, Samples].

oncvideo.read_ts(file, units=True)[source]

Read time series files from ONC and convert to a dataFrame

Parameters:

  • file (str) – Path to a .csv file of time series scalar data (tssd) returned from Oceans.

  • units (bool, default True) – Include units of the vairables in the column names. If False, units are removed.

Returns:

A dataFrame from the csv file.

Return type:

pandas.DataFrame

oncvideo.rename_st(filename)[source]

Rename framegrabs from Seatube to correct timestamp

Correct the timestamp based in the offest of the image that is captured in Seatube. E.g. a file like deviceCode_20230913T200203.000Z-003.jpeg is renamed to deviceCode_20230913T200201.000Z.jpeg.

Parameters:

filename (str) – Filename to be corrected.

Returns:

If possible, the corrected filename, or else it will return the same filename.

Return type:

str

oncvideo.to_mp4(source, output='output', trim=False, deinterlace=False, crf=None, keep_audio=False, yuv420=False, h265=False)[source]

Convert video to mp4

Videos are encoded to mp4 (h264) constrained by quality (Constant Rate Factor), with a faststart option for web video, and constant framerate.

Parameters:

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *)

  • output (str, default 'output') – Name of the output folder to save converted videos

  • trim (bool, default False) – Trim video files to match the initial seach query

  • deinterlace (bool, default False) – Deinterlace video.

  • crf (int, defalt None) – Set CRF (quality level) in ffmpeg. The default will use the default value from ffmpeg.

  • keep_audio (bool, default False) – Keep audio in the video? Default is to remove de audio.

  • yuv420 (bool, defalt False) – Force YUV planar color space with 4:2:0 chroma subsampling. May be needed to display video in some players/browsers.

  • h265 (bool, defalt False) – Use H.265 encoding instead of H.264. H.265 offers a higher compression, but may not be supported by some players/browsers.

oncvideo.video_info(source, output='video_info.csv', check_interlaced=False)[source]

Get video information

Include video codec, pixel format, bitrate, fps, fps mode (constant or variable), scan type ( progressive or interlaced), aspect ratio, width, height, duration and file size. Note that scan type is based in the video metadata, unless check_interlaced is set to True, where idet filter is used to check scan type. Video duration is based from the file, and may differ from the duration returned by ‘list’ or ‘blist’, which is based on the API call.

Parameters:

  • source (str or pandas.DataFrame) – A pandas DataFrame, a path to .csv file, or a Glob pattern to match multiple files (use *)

  • output (str, default 'video_info.csv') – Name of the csv file to save video information

  • check_interlaced (bool, default False) – Use the idet filter from ffmpeg to check if video is interlaced.