NBN Gateway Data Access Controls

A new system of data access controls for the NBN Gateway is being rolled out in 2013.  In summary, the main changes are:

• Removal of the “view only” control
• Improved data download functionality including whole dataset downloads
• Only two possible levels of access – ‘public’ and ‘full’
• Greater flexibility in granting enhanced access
• Better feedback for data providers

This web page explains what the changes are, how they will affect you and whether you need to do to anything to prepare for them.  You can also download the information as a pdf if it would be helpful to print the information out.

If you cannot find the answer to your question here or if you would like to discuss the implications of the new access controls, please contact us on access@nbn.org.uk

Important information about the changes

• What are the key changes to the access controls?
• As a data provider, what do I need to do to prepare for the change?
• As a data user, what do I need to do to prepare for the change?
• What help is available from the NBN Team?

Additional Information

• How will current access positions be transferred to the new system?
• What is ‘intermediate’ access and will its removal affect me?
• What are ‘sensitive records’ and will there be changes in the way they are handled?
• How will web services be affected?

Background information

• What are access controls and why do we need them?
• Why are the access controls changing?
• Who was consulted and what was the outcome?

Important information about the changes

What are the key changes to the access controls?

The key changes are:

• Removal of the ‘view only’ control

All records except for those flagged as sensitive will be available to download at the level of public access defined by the dataset administrator.Users will need to be logged in to download data, and dataset administrators will receive a detailed log of exactly who has downloaded their data and for what purpose.Use of data downloaded from the Gateway will continue to be governed by the NBN Gateway Terms and Conditions.

• Only two possible levels of access – ‘public’ and ‘full’

Under the new system there will be only two possible levels of access to each record – public access or full access.When granting enhanced access to individual users or organisations, it will no longer be possible to give intermediate levels of access  i.e. better than public access but not full access.

• Ability to download whole datasets

New export functionality will allow download of a complete species dataset, except for sensitive records, at the resolution at which public access has been defined by the dataset administrator. Data access requests and the data download log will specify what proportion of the dataset is covered by the request/download.

• Improved download facilities

In the current system it is only possible to download data by 10km square, designated site boundary or a single species.In the new system, the user will be able to use any combination of geographic, taxonomic, date and dataset filters to download precisely the records they need.Records from different sources will be combined in a single file for ease of use, with links to the metadata for each source dataset.

• Greater flexibility in granting enhanced access

Enhanced access can be granted to individual records rather than to the whole dataset, using any combination of geographic, taxonomic and date filters.

• Better feedback for data providers

Users will need to log in to download data or to view the details of records in a table.Users who are not logged in will only be able to see squares on the map.Dataset administrators will see a log of everyone who has accessed their data, when and for what purpose, and what percentage of their dataset was downloaded.The log will be updated daily and will include e-mail addresses for all users that have downloaded data.

Click here for a full list of the changes, which also shows how the original proposal was revised as a result of the consultation.

Click here to view a Powerpoint presentation explaining how the new system of access controls can be applied by dataset administrators and how it will affect what is seen by data users.

As a data provider, what do I need to do to prepare for the change?

In December, all species datasets on the NBN Gateway transferred automatically to the new system.  Click here for information on how current access positions will be mapped to the new framework.

All dataset administrators were sent detailed guidance on how to use the existing administration controls on the Gateway to ensure a smooth transition to the new access system with no loss of access for approved users.  

If you changed the level of public access to your dataset, please update the access constraints statement in the dataset’s metadata to take account of this.

Important! If you are an organisation administrator, please check that membership of your organisation is up-to-date by logging into the Gateway, clicking on ‘my account’ in the top right corner and selecting ‘organisations-you manage’ from the menu on the left.  Click on ‘current members’ under ‘manage membership’ and if anyone has left your organisation, click ‘revoke access’ to remove them.

As a data user, what do I need to do to prepare for the change?

If you have been granted enhanced access to some datasets, your access might be affected by these changes.  This is especially likely if you have been granted ‘intermediate’ access to datasets, i.e. better than public access but not full resolution download access with recorder names and attributes.  All individual users with intermediate access to datasets have been contacted via e-mail to advise them to apply for full access to datasets if they still require the data.

Please note that any outstanding data access requests that were not dealt with by the October cut off point have been removed from the system.

Once the new system has been launched, you will need to log in if you want to download data.  You will also need to log in if you want to see the details behind the records, such as record date, site name, recorder name and comments if available.  If you are not logged in, you will only see squares on the map.

Important! Please check that the e-mail address associated with your NBN Gateway account is correct by logging into the Gateway and clicking on ‘my account’ in the top right corner of the screen. 

Finally, please note that if your level of access does change, it will be a much simpler process to re-apply for access under the new system.

What help is available from the NBN Team?

If you have any questions at all about the changes to the data access controls and how they will affect you, please contact us on access@nbn.org.uk

There is also a forum for posting questions or discussing issues related to the changes.

We have issued guidance to all data providers on how they can use the existing access controls to change the level of public access to their data if they wish to do so, and we are available to provide help and advice.

Several users were concerned about the potential impact of the loss of intermediate access levels, so we carried out a detailed review of all the intermediate access agreements on the Gateway.  We contacted all individual users by e-mail to advise them that their intermediate access agreements will revert to public access in October unless they obtain full access to the datasets before then.   We are also contacting data providers who have a large number of intermediate access agreements, to ensure that they are aware of the situation and prepared to handle requests from users to upgrade their intermediate access to full access.

The NBN Gateway Users’ Guidebook will be updated in due course to take account of the new access controls and other changes to the Gateway functionality.

We have discontinued the prototype system as it is now being used for load testing and for function testing purposes.

Additional Information

How will current access positions be transferred to the new framework?

When the new system of access controls is implemented, all of the data and the current access positions will be migrated across to the new framework.  The tables below summarise the automatic translation that will be implemented, and highlights in red where current access levels will change as a result of migration to the new system.   

Public Access

The ability to limit access to ‘view only’ will be removed.  All public access positions that grant view access will be migrated to allow download.

Enhanced access for registered users or organisations

Under the new system there will be only two possible levels of access to each record – public access or full access.  It will no longer be possible to support intermediate levels of access, i.e. access which is better than public access but not full access (i.e. download access at full resolution with recorder names and attributes, if present).  Users with ‘intermediate’ access will find that their access reverts to public access when the new system is launched, unless they apply for and obtain full access.

What is ‘intermediate’ access and will its removal affect me?

One of the key features of the new system of data access controls is the fact that there will be only two possible levels of access to each record.  The two possible levels of access are Public Access and Full Access:

Under the current system of data access controls, it is possible to give individual users and organisations ‘intermediate’ levels of access, i.e. better than public access but not full access (download access at full resolution with recorder names and attributes, if present).  This will no longer be possible under the new system. 

We carried out a detailed review of all the current intermediate access agreements on the Gateway, and contacted data users and data providers to try to ensure that users who had intermediate access did not experience a reduction in access on transition to the new system.  When the new data access controls launched, any existing ‘intermediate’ access levels automatically reverted to the public level of access.

What are ‘sensitive records’ and will there be changes in the way they are handled?

Species records may be flagged as sensitive by the dataset administrator according to their judgement of potential risk of damage to the environment. Access to these records will be restricted in accordance with NBN Exchange Principle 2: “Making biodiversity data available should reduce the risk of damage to the environment. If it is likely to have the opposite effect, availability may need to be controlled”.  

For further information, see the NBN Guidance on sharing environmental sensitive data, which includes advice on applying a standardised criteria-driven assessment of environmental sensitivity.

There may be some benefit in agreeing a standard list of “sensitive” species nationally (some progress has already been made on this), but this could never be rigidly applied – for example there may be variation in policy in different parts of the UK or a species may be sensitive at some times of the year (e.g. the breeding season) but not at others.  A record should not be flagged as sensitive to protect intellectual property rights or commercial interest in the data.

Under the new system of data access controls, dataset administrators will continue to be able to flag records as sensitive.  Records flagged as sensitive in existing datasets will remain so when the datasets are migrated to the new system.

Public access will never include access to sensitive records.

Full access may include or exclude access to sensitive records, at the dataset administrator’s discretion.

In response to requests by data providers, we plan to develop new functionality to enable dataset administrators to specify whether the sensitive control should make records invisible to the public or simply blur their resolution, but this feature will not be available in the first release of the new system.

How will web services be affected?

There are many different types of NBN web services.  Some, such as the grid map service, can deliver ‘view only’ records from the NBN Gateway and display them as squares on a map.  Others, however, can only deliver data which are publicly downloadable.  We therefore hope that the removal of the ‘view only’ control will increase the amount of data that is available via web services.

Public-facing web services deliver data at the level of public access granted to users of the NBN Gateway.  Password-controlled web services for registered users can deliver data to which enhanced access has been granted to those users.  This will not change in the new system.

However, any changes to the levels of public or enhanced access set by the dataset administrators will have an effect on web services.  If public access to a dataset is withdrawn, any web service that retrieves and displays public data would be affected.

We hope that most of the existing data access agreements will be transferred to the revised NBN Gateway access controls and that web services delivery will generally not be affected. It is unlikely that web service developers will need to update any of their code or parameters as a result of the proposed changes. However, because we are not aware of all instances of web service use we cannot guarantee this – please contact the NBN team if you have any specific queries.

Background information

What are access controls and why do we need them?

Data access controls allow data providers to restrict public access to species records if necessary and to grant enhanced access to approved individuals or organisations.

Data providers can use the access controls to:

• Display records at ‘full’ resolution, i.e. 100m squares, or blur the resolution to 1km, 2km or 10km squares
• Display or hide recorder and determiner names
• Display or hide attribute fields (e.g. abundance, life stage)
• Include or exclude sensitive records when giving enhanced access to approved users

It is a fundamental feature of the NBN Gateway that data providers retain ownership and control of their data.  The data access controls have been crucial to developing the Network and encouraging data sharing over the last 10 years, reassuring data providers that sharing data will not result in environmental harm or affect their ability to maintain the services they provide.

Click here to view a Powerpoint presentation about the new data access controls.

Why are the access controls changing?

The main reasons for reviewing and updating the access controls are:

• To increase the use of biodiversity data in accordance with the NBN Strategy 2010-2020. The current controls include a ‘view only’ concept which allows users to see records as squares on a map but not to see the detail behind them or to download them without permission from the data provider.  Removing the ‘view only’ control will increase the usefulness of the Gateway as a research tool and make more data available to consumers of NBN web services.
• To improve clarity over data access – the current ‘view only’ control leads to confusion on the part of the user as to why some records apparently ‘disappear’ when they look for more information.
• To improve data access and exchange by offering a more flexible and user-friendly system for managing access requests, enabling data providers to grant access to a taxonomic or geographic subset of their data for a limited time period, and enabling users to apply for enhanced access to multiple datasets with a single request.
• To offer better feedback to data providers on how their data are being used and by whom, providing useful statistics for reports to partners, funders and the recording community.
• To improve performance of the NBN Gateway and web services – the NBN Gateway now provides access to almost 80 million records.  The current system of access controls already impacts on the performance of more advanced features such as the new Interactive Map, and will cause a significant degradation of performance as the volume of data increases.

Who was consulted and what was the outcome?

From the 18^th October to the 30^th November 2011 there was a programme of consultation with data providers and data users, consisting of a detailed online survey supported by one-to-one communications and meetings with individual organisations.

There was an excellent level of participation in the online survey – we received 93 responses from 85 organisations, including national recording schemes and societies, local record centres, conservation NGOs, local natural history societies, independent researchers, ecological consultants, statutory agencies and academic institutions.  Seventy-seven respondents were current data providers.

In the main, the responses indicated an appreciation that the NBN Gateway must evolve in order to provide acceptable performance and functionality for both data providers and data users.  Virtually all of the proposals were supported by the majority of respondents.  Nevertheless, three elements of the proposals caused significant concern amongst some data providers:

• Introduction of a facility to download a whole dataset

(27% disapproved, while 63% approved, 43% of whom strongly approved).

• Only two levels of access: public and full.

(34% disapproved and 50% approved)

• Removal of the ‘view only’ control, enabling all data to be downloaded.

(35% disapproved, while 52% approved; 28% of whom strongly approved).

The Gateway Technical Steering Group reviewed the response to the first round of consultation in December and recommended that a second round of consultation should take place supported by a forum, screen shots and an interactive prototype of the new system to enable user to visualise how the new controls would work in practice. (The prototype has now been discontinued to allow its use for testing pusposes).

The second round of consultation ran from February to April 2012.  This round of consultation did not include an online survey, but consisted of forum posts, correspondence and meetings with data providers and users.

The proposed changes to the access controls were modified significantly in response to both rounds of consultation.  Click here to download the complete list of changes to the access controls, which also highlights how the original proposal was revised as a result of the consultation.

Finally, meetings were held in June 2012 with representative data providers and users from each sector to gather feedback to inform the development of the user interface for the new system.

The NBN Board of Trustees and Strategy Implementation Group were updated and consulted at key stages throughout the process of reviewing the data access controls.