New REST web services
A suite of new ‘RESTful’ NBN web services was launched in October. These web services are fully functional and are powering the new NBN Gateway. The API is available to view and download from the NBN Gateway Documentation page. Next year we aim to produce further documentation on how to implement the API, together with some coded examples as new implementations are developed. We had intended to work with a small group of users to help them develop an implementation of the new services in order to refine and document the services prior to wider release, but this was not possible in the timeframe.
We are very sorry that we have not had the capacity to respond to queries about using the new API that have arisen on the forum and via e-mail recently. This is due to staff shortages and the need to prioritise data loading and critical improvements to the new Gateway. We are collating all queries that have arisen and will dedicate staff time to responding to these in January or sooner if possible, giving priority to helping current users of SOAP services migrate to the new REST API.
The REST services are not exact equivalents of the SOAP services so there is no quick ‘translation’ from one to the other – developers will need to write new code to migrate their current systems from SOAP to REST. The REST services deliver all the functionality of the old SOAP services, but please note the following:
- It is not possible to define a point and buffer or site and buffer to call species observations. You can still perform this search via the REST API, but have to do it by creating a user-defined polygon.
- User-defined polygons can only be created in WGS84 format. We aim to offer the facility to define polygons in other projection systems, e.g. OSGB63, but this will not be available until next financial year at the earliest.
If you find that you can’t use the REST services to perform a function that you previously obtained via the SOAP services please do let us know, and we will update the REST API to give you this functionality if necessary.
The performance of the REST services should be good overall, although we are aware that some taxon observation calls using spatial filters are running slowly and users are experiencing timeouts. These spatial queries require intensive processing, so their performance is unlikely to improve in the near future.
SOAP web services
We advised readers in April that the old SOAP web services would continue to run off the old NBN Gateway for as long as required, although the data accessible through them would not be updated. This is still the case. Unfortunately, a necessary change was made to the hardware at CEH which affected the old NBN Gateway and the SOAP services, resulting in failure of these services since late November. The NBN technical team were not notified in advance of this planned change, and have been working since then to try to fix the problem.
We are very sorry that this has taken so long, and that we did not communicate sooner about this. We are aware that several users base their business operations on the SOAP web services. We have been working with partners at CEH to bring the service back online and are pleased to report that they are now working again.
EasiMap
EasiMap is a tool created by the Biological Records Centre to embed a grid map in an iFrame, and is used on websites such as the UK Ladybird Survey and the Highland Biological Recording Group. This tool currently runs off the SOAP services, so it was also affected by the recent problems, for which we apologise. EasiMap is currently being rewritten to run off the new REST web services, so EasiMap users will not need to take any action to migrate their grid maps from SOAP to REST services. This change is expected to take place in early January.
Data security via web services
Some queries have been raised about the security of data that is accessible via web services, both relating to species occurrence data and personal details of Gateway users. Users need to log in to access data via the REST web services, and can only access species occurrence data at the level of resolution they have been granted.
The administration controls on the NBN Gateway require a REST API endpoint that makes user’s details visible to the user themselves and to dataset administrators. User’s account details consist of user ID, first name, last name, password, e-mail address, organisation membership (where applicable) and phone number (not compulsory). The REST API enables:
- Logged in dataset administrators and organisation administrators to view user names, e-mail addresses and organisation membership. They need this information to manage access to their datasets and organisations.
- Logged in users to view and edit all their own account details.
This is in accordance with the NBN Gateway Privacy Policy.
Financial support for the adoption of REST web services
There is funding available via the Defra/NBN contract to help users adopt the new REST web services. The aim is to use this resource to develop generic implementations of web services as Drupal modules, particularly focussing on supporting use by local environmental records centres.
It was previously suggested that the Defra Fund for Biodiversity Recording in the Voluntary Sector administered by Natural England might be an appropriate source of funding for developing implementations of the REST web services, as the fund’s objectives include increasing the quality and relevance of biodiversity data available to national and local users and assisting Local Environmental Records Centres in the provision of a more effective and efficient service to data users. However, it has recently been clarified that the fund cannot be used for any kind of website development work. We apologise for any confusion caused.
Feedback on data downloaded via web services
The download log available to dataset administrators on the NBN Gateway currently only includes records downloaded as CSV files via the Gateway, it does not include records accessed via the REST web services API. This is because the Gateway is itself run off the REST API, so this would mean that every time record details are viewed on screen it would count as a ‘download’ and we felt this would swamp the download log with background noise, distracting dataset administrators from the ‘real’ downloads.
However, it is important for dataset administrators to have as much information as possible on the use of their datasets, so they can demonstrate to recorders and funders how much the data is being used, and also so they can monitor which users and organisations in particular are using their data. We have therefore reviewed this and can offer the following:
- We will provide a log of all downloads of raw taxon records made via the API. Dataset administrators will be able to see this on their ‘My Account’ page.
- This will be on a separate tab to the ‘real’ downloads of data as CSV files via the Gateway.
- The log will not include views of squares on maps or species lists for sites. It will only be triggered when record details are viewed on screen (e.g. when a user selects and queries squares on the interactive map) or when records are downloaded.
- The log will show: the user’s name (with link to e-mail address, and organisation membership if applicable), the date and time of data access, number of records accessed from your dataset, the total number of records accessed, the filters used (e.g. taxonomic or geographic) and the originating IP address.
- It will not show the reason for use – this is only captured when users download data as a CSV via the Gateway.
- Logs will contain a huge amount of data, so this can only be kept for a limited period (6 months initially) but dataset administrators will be able to download raw data from the log as a CSV to produce their own summaries and can store the raw data as long as they want.
- Downloads via the API will never trigger notification e-mails. Dataset administrators will only receive notification emails when someone downloads records from their datasets as a CSV file via the Gateway (if they have opted to receive them – this can be turned off or on at any time via the My Account page).
We aim to have this set up by mid-late January, but we will keep you informed of progress.
Click here to view our article in April eNews about the launch of the new web services.