Ag Productivity API Guide

The Ag Productivity data is hosted on the Eratos Senaps cloud platform, Senaps hosts and automated scientific workflow to recalculate Ag Productivity within 24 hours of the latest MODIS NDVI data becoming available. This document describes how to find and download Ag Productivity data from the Senaps platform.

Ag Productivity Data is made available as GeoTIFF file format with the following details:

Output Spatial Grid

Resolution	0.0025 degrees in both longitude and latitude, approx 250m
Datum	WGS84
Extents	continental Australia

Output Data Structure

Format	Description

Format

Description

Catalogue

To allow the client to discover which MODIS periods have been processed and check for new Ag Productivity data, a catalogue service is provided. The catalogue is provided as a URL to an XML file which will index every available Ag Productivity data grid file. The client can use file modified timestamps and/or filename pattern (defined below) to determine new files available for download.

The catalog is available in THREDDS format at the following URLs:

Sample data catalogue: https://senaps.eratos.com/thredds/catalog/org_catalogs/csiro-agdatashop/npp/npp-sample/catalog.html

Subscriber data catalogue: https://senaps.eratos.com/thredds/catalog/org_catalogs/csiro-agdatashop/npp/npp/catalog.html

GeoTIFF

The client will be provided access to Ag Productivity data grid as a GeoTIFF which will be available for download via an authenticated HTTPS service within the Senaps platform.

Individual GeoTIFF files will be generated for each available MODIS 16-day period.

National files will be named as follows:

CSPv<version>_<year>_<doy>.tif

State based files will be named as follows:

CSPv<version>_<year>_<doy>_<state>.tif

version: Integer used to discriminant between future versions of modelling methodology

year: Year with century eg. "2019", "2020"

doy: Zero padded day of year. eg. "001","018"

state: Identified the spatial region of the file. 'NSW_ACT' covers both NSW and ACT and Jervis bay territory.

Data type: Float32

No data value: -9

Identifying and downloading datasets programmatically

An XML representation of the Ag Productivity file catalogue can be obtained through the following URLs

Sample data : https://senaps.eratos.com/thredds/catalog/org_catalogs/csiro-agdatashop/npp/npp-sample/catalog.xml

Subscriber data: https://senaps.eratos.com/thredds/catalog/org_catalogs/csiro-agdatashop/npp/npp/catalog.xml

The XML has a service description component and every dataset available will be described in a 'dataset' XML element, below is an example of the sample dataset catalogue.

<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" name="npp" version="1.2">
<service name="grid-file" serviceType="Compound" base="https://senaps.eratos.com">
<service name="grid-all" serviceType="Compound" base="https://senaps.eratos.com">
<service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/"/>
<service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
<service name="wms" serviceType="WMS" base="/thredds/wms/"/>
<service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/"/>
<service name="ncml" serviceType="NCML" base="/thredds/ncml/"/>
<service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/>
</service>
<service name="http" serviceType="HTTPServer" base="/thredds/fileServer/"/>
</service>

<dataset name="CSPv21_NPP_2018.193.tif" ID="5E0728F994277545A2425349EA9658A1" urlPath="csiro-agdatashop/npp/npp/CSPv21_NPP_2018.193.tif" restrictAccess="csiro-agdatashop:npp-subscriber-data-australia">
<authority>csiro-agdatashop</authority>
<dataType>Grid</dataType>
<dataFormat>NetCDF</dataFormat>
<dataSize units="Mbytes">871.8</dataSize>
<date type="modified">2020-11-19T13:00Z</date>
<metadata inherited="true">
<serviceName>grid-file</serviceName>
</metadata>
</dataset>
<dataset name="CSPv21_NPP_2018.193.VIC.tif" ID="82793F97C1E5026CFD4502B414448656" urlPath="csiro-agdatashop/npp/npp/CSPv21_NPP_2018.193.VIC.tif" restrictAccess="csiro-agdatashop:npp-subscriber-data-vic">
<authority>csiro-agdatashop</authority>
<dataType>Grid</dataType>
<dataFormat>NetCDF</dataFormat>
<dataSize units="Mbytes">28.5</dataSize>
<date type="modified">2020-11-19T13:07Z</date>
<metadata inherited="true">
<serviceName>grid-file</serviceName>
</metadata>
</dataset>

To download the file described by the above catalogue entry the urlPath attribute can be concatenated with the 'http' service base path available in the service element in the catalogue XML. The host name will be 'senaps.eratos.com' and the service must be accessed via the secure HTTPS protocol.

Authentication

All Ag Climate Data Shop API products are authenticated using the process described in Authentication for AgData Shop product APIs. After this process you will have obtained an Access Token which must be applied to every API request as a HTTP Authorization Bearer token. Please note tokens expire as per standard OAuth2 Client Credentials flow.

Authorization: Bearer <INSERT TOKEN>

In the example below the requests library and the requests-oauth2client library is used to support AgData Shop authentication.

Python Example

A valid sample data HTTP download URL is https://senaps.eratos.com/thredds/fileServer/csiro-agdatashop/npp/npp-sample/CSPv21_NPP_2018.193.VIC.tif

The subscriber dataset urlPaths are slightly different, for example:

https://senaps.eratos.com/thredds/fileServer/csiro-agdatashop/npp/npp/CSPv21_NPP_2018.193.VIC.tif

A Python example code using the popular requests library is shown below. Note this example is designed to handle the large files by writing them to disk in chunks. The example downloads an Victorian AgProductivity GeoTIFF from the sample data catalogue and then opens and plots the dataset using rasterio and matplotlib.

The above example requires requests and rasterio be installed in your python environment.

Please refer to https://rasterio.readthedocs.io/en/latest/installation.html for detail about installing rasterio on your system, particularly for Windows users.

If you just wish to test the download component of the example; you can remove the last five lines to verify the HTTP download works correctly.

MacOS X users can install rasterio using pip. Linux users need to have the gdal-bin package installed and can then install rasterio using pip.

The working example will render the NPP data in a plot, eg:

Alternatively, once the GeoTIFF is downloaded a suitable tool can be used to visualise it, for example the open source QGIS (www.qgis.org). In QGIS from the Layer → Add Layer → Add Raster Layer menu item:

Full Data Set

The above example demonstrates how to access the Sample Ag Productivity data.
With an active Ag Productivity subscription you will have access to the datasets in the following catalogues:

html catalog: https://senaps.eratos.com/thredds/catalog/org_catalogs/csiro-agdatashop/npp/npp/catalog.html

xml catalog: https://senaps.eratos.com/thredds/catalog/org_catalogs/csiro-agdatashop/npp/npp/catalog.xml

API acceptable usage limits

To maintain quality cost effective service this product includes a acceptable use policy on HTTP API usage.

A single subscription of this product entitles the subscriber to retrieve up to 5GB per month of Ag Productivity data via the HTTP API described in the document.

Please contact the AgData Shop to discus options for higher usage.