Imagine this scenario: Circumstances in a foreign nation have become increasingly unstable, requiring safe passage for British nationals out of the crisis region.
The challenge: We have received aerial imagery data that allows us to identify friendly and suspicious vehicle types. Now, we need you to detect and classify vehicles in these images so that we can assess potential risks along key convoy routes.
The Challenge Guide provides details on the data specification, scoring metric and guidance for attempting the challenge. It has been summarised below.
The data for this challenge is high resolution aerial imagery. For the purpose of this challenge, a UK city has been used as the geographical location to represent part of a foreign nation.
The datasets consist of:
- Training images (training.zip): A set of 600 images (JPEG) at 5cm resolution, each one covering an area of 100m by 100m. [580MB]
- Training observations (trainingObservations.csv): The label and location of the vehicles of interest corresponding to the training images. [252.9KB]
- Test images (test.zip): A separate set of 600 images (JPEG) at 5cm resolution, each one covering an area of 100m by 100m. [567MB]
- Sample submission (sampleSubmission.csv): A sample submission file with the correct format, but with random detections. [145.5KB]
The data is available to download in the Data Download page once you have entered the challenge.
You are required to detect and classify the vehicles of interest within high-resolution aerial images. Your submission must:
- Find and record all vehicles in the test images that match one of the 9 vehicle classes of interest. Vehicles cut off by the edge of the image can be ignored.
- Record the image, vehicle class, and centre pixel position of the vehicle for each observation of interest in the submission CSV format. The exact centre pixel position is not vital, but it must be within the ground truth acceptance boundary.
The training images and training observations can be used in any way you decide (subject to the data terms in the Official Rules) to help find the vehicles of interest in the test images.
The submitted results file will be scored using the Jaccard Index, defined as:
- TP are the true positives
- FP are the false positives
- FN are the false negatives
The submission file format is similar to the training observations format. It is a CSV formatted file (standardised for upload to the website), but only requires the id and detection columns. These columns are formatted to be the same as those in the training observations and are defined as:
- Id: The unique id for this row. This is a combination of the image name (e.g. TQ2378_0_0.jpg) with file extension (.jpg) removed, an underscore (_), and the vehicle class (e.g. A). For this example the Id would be TQ2378_0_0_A.
- Detections: The complete set of detections for the image and class combination. Each detection is the centre pixel position of the vehicle for the observation, separated by a colon (:), e.g. xPixel:yPixel, and a pipe (|) is used to separate detections. When there are no detections, “None” is used. Pixel coordinates are integers and defined using standard convention (i.e. with the image origin at top left).
Submissions will need to be ordered by unique ‘id’ (image name then class).
All combinations of image name and vehicle classes need to be included, even if the images do not contain any vehicles of interest. ‘None’ is a valid and possible correct result.
The number of submissions will be limited to 3 submissions per day.
Public/Private Leader board
The public leader board will show results during the competition for a subset (33%) of the test images.
The private leader board will calculate the score for the remaining (67%) test images and will be used to assess the competition winner.
This competition will start on Monday 3rd April 2017 and runs for 6 weeks.
Participants with the top score on the private leader board are candidates to be awarded:
- 1st Place – £20,000
- 2nd Place – £12,000
- 3rd Place – £8,000
Note that the cash prizes will only be paid out to a bank account which is not in a country with a score of 37 or less according to Transparency International’s Corruption Perceptions Index 2014.
Similarly, they will not be paid out to an individual who is a national/located in one of these countries too or whom the UK Government is not reasonably satisfied as to the potential recipient’s identity. Please see the Official Rules.
There are guidance notes in the The Challenge Guide that can be downloaded once you have signed up to the challenge.
The Challenge Master is our expert for the challenge and will be available to offer guidance through the challenge forum.
With regard to the use of data, the following points should be considered:
- There are no restrictions on how you use the training data. You may use all of it, a sample of it, or even none of it.
- You may augment the provided training data with your own training data if you wish, providing you have permission to do so from the data owner.
- There are no preferred techniques for this challenge. For example, traditional computer vision techniques and/or machine learning approaches can be used.
- Basic solutions could run on a standard PC. However, when using machine and deep learning techniques the increased performance gained by using GPU clusters may prove beneficial.
The following instructions for your approach must be followed:
- Your submissions must use only the provided test images (test.zip) for detecting and classifying vehicles and not on any auxiliary information e.g. map data. (The use of additional data sources is permitted for training but not test).
- You must not manually solve the challenge by simply viewing the images yourself and/or crowdsourcing topics from others. An automated software-based solution must be developed.
- You must be able to demonstrate that the solution can be run without any human intervention.
- Your submissions are assessed against the ground truth (as we define it). The judges’ decision shall be final and no correspondence shall be entered into with regard to it.
- Your final solution must operate in an environment without internet access. This is so it can be independently validated as a discrete component (without use of other online services).
- You must not cheat. Cheating is strictly prohibited and any attempts to deliberately enter the competition to disrupt it, is against the rules and the spirit of the competition. We reserve the right to disqualify any participant that does not comply with the above instructions or the Official Rules.
Our Challenge Masters will be available on the challenge forum if you wish to discuss your approach for solving the challenge.
The candidate winner(s) (1st, 2nd & 3rd) may be asked to host their solution on an independent environment (e.g. Amazon Web Services), re-train it and then re-run against the test data to reproduce the results obtained during the competition.
The winning solution will also be executed against a hold-out dataset (100 images, each one 100m by 100m). This is to validate the candidate winning solution and ensure challenge rules have not been violated. Compared to the test data, the hold-out data:
- comes from the same source
- is in the same format
- contains the same vehicle classes.
The solution must ingest this hold out dataset and produce the list of detected observations in the same format as the submission file.
The aim is to obtain results that are consistent with previous results from the test dataset. Solution evaluation will be conducted by a judging panel led by the Challenge Master and includes other experienced data science SMEs. If there is a major discrepancy then this will be discussed with the participant and if it cannot be explained to the satisfaction of the panel, then this may lead to disqualification. The decision of the judging panel is final and no further correspondence on the result will be entered into.
Following successful evaluation, the winning solution shall be delivered to the challenge sponsor in the form of object code and source code (unless restricted by software licence terms). The winning solution must also be accompanied by documentation which describes the approach, resources required and instructions necessary to build and run the solution successfully. Please see Official Rules for further details.
Frequently Asked Questions
Each submission will be ranked by score on the public leader board. For two submissions with equal scores, the one that was first submitted will be ranked higher.
Top 10 entries
|1||gbarbadillo||0.8366||24 Apr 2017, 6:24PM BST||15|
|2||kit1||0.7646||12 Apr 2017, 10:51AM BST||1|
|3||cogitae||0.7564||22 Apr 2017, 9:32AM BST||5|
|4||ALarM||0.4883||24 Apr 2017, 5:24PM BST||5|
|5||Kyle||0.3546||14 Apr 2017, 12:32AM BST||3|
|6||kohei||0.3205||12 Apr 2017, 8:30AM BST||13|
|7||voglinio||0.0335||20 Apr 2017, 4:03PM BST||3|
|8||jane.ostin||0.0017||03 Apr 2017, 9:12PM BST||1|
|9||Chippy||0.0017||03 Apr 2017, 9:16PM BST||1|
|10||alezenonos||0.0017||04 Apr 2017, 2:53PM BST||3|
Definitions and Interpretation
- “BAE Systems”
- means BAE Systems Applied Intelligence Limited (company number 1337451) whose registered address is Surrey Research Park, Guildford, Surrey GU2 7YP.
- “Data Challenge”
- means a data challenge competition held on the Website.
- “Challenge Materials”
- means the images and data provided to Competitors as part of a Data Challenge as updated from time to time.
- means the party uploading an Entry to a Data Challenge via the Website.
- “COTS IPR”
- means Intellectual Property Rights that are commonly used and provided in a standard form and generally made commercially available on standard licence terms which are not typically negotiated by the licensor.
- “COTS Software”
- means software (including open source software) that is commonly used and provided in a standard form and generally made commercially available on standard licence terms which are not typically negotiated by the licensor.
- means data uploaded to the Website by a Competitor describing that Competitor's response to a Data Challenge.
- “Intellectual Property Rights”
- means (a) copyright, rights related to or affording protection similar to copyright, rights in databases, patents and rights in inventions, semi-conductor topography rights, trade marks, rights in internet domain names and website addresses and other rights in trade names, designs, know-how, trade secrets and other rights in confidential information; (b) applications for registration, and the right to apply for registration, for any of the rights listed at (a) that are capable of being registered in any country or jurisdiction; and (c) all other rights having equivalent or similar effect in any country or jurisdiction.
- “Non-COTS IPR”
- means Intellectual Property Rights that are not COTS IPR.
- “Non-COTS Software”
- means software that it not COTS Software.
- means the software used to create an Entry.
- “Sponsoring Agencies”
- means the Defence Science and Technology Laboratory (Dstl), the Government Office for Science, MI5 and SIS.
- “UK Government”
- means the government of the United Kingdom acting through the Sponsoring Agencies.
- means the website found at www.datasciencechallenge.org
Data Challenges are hosted, run and judged by BAE Systems as a supplier of services to UK Government. BAE Systems shall be UK Government's authorised representative for these purposes.
These Official Rules govern the relationship between UK Government and each Competitor and are applicable to all Data Challenges.
Additional, specific terms and rules of participation will apply to individual Data Challenges. Competitors should ensure that they are familiar with all of the terms and rules of participation that apply to a particular Data Challenge.
Failure to adhere to these Official Rules and any specific terms and rules of participation applicable to a particular Data Challenge may result in disqualification.
Eligibility to take part in Data Challenges
Data Challenges are open to individuals aged 18 and over. Entries made by or on behalf of corporate entities will not be accepted.
Officers, directors, employees and their immediate families of the Sponsoring Agencies, BAE Systems, Capgemini UK PLC, Roke Manor Research Limited and their respective group companies, contractors and agents may not participate in Data Challenges.
No payment shall be made (whether directly or via a third party/country) to:
any bank account registered and maintained in any country with a score of 37 or less according to Transparency International's Corruption Perceptions Index 2014; or
an individual who is a national and/or resident of, or located in, any country with a score of 37 or less according to Transparency International's Corruption Perceptions Index 2014, or
an individual who UK Government knows or has reason to suspect (or UK Government's authorised representative knows or has reason to suspect) appears on:
the sanctions list maintained by the United Kingdom Foreign and Commonwealth Office (as amended from time to time); or
the Consolidated List of persons, groups and entities subject to EU financial sanctions, as maintained by the European External Action Service (as amended from time to time); or
the Consolidated Screening List as maintained by the United States Government (as amended from to time).
If UK Government is not (or its authorised representative is not) reasonably satisfied as to the potential recipient's identity, no payment shall be made to that person.
UK Government reserves the right (with or without notice) to update Data Challenges and Challenge Materials during their running. Competitors should regularly check the Website for such updates in order to ensure that they remain familiar with the Data Challenge and are using the latest Challenge Materials. UK Government accepts no liability for any failure on the part of a Competitor in this regard.
Assessment of Entries and Solutions
A maximum of 3 Entries per day per Competitor will be assessed; any Entries in excess of this limit will be disregarded. Entries will be assessed in the order in which they are submitted.
Entries shall be assessed electronically against UK Government's model response (referred to as the “ground truth”) for the relevant Data Challenge.
A Competitor whose Entry is selected as a potential winning Entry shall at its own expense install, configure and make available the Solution in a non-internet facing environment (such that the Solution runs without access to online resources) on an Amazon Web Services Inc. cloud platform (or such other PaaS as UK Government or its authorised representative may approve) for evaluation by UK Government or its authorised representative.
Solutions shall be assessed by, inter alia, their ability to automatically respond to previously unseen data sets and the proximity of their responses to the UK Government's ground truth for the relevant data set.
The winning Competitor will be notified within four weeks of the closing date of the relevant Data Challenge. The judges' decision shall be final and no correspondence shall be entered into with regard to it.
Delivery of Winning Solution
Following successful evaluation of a Solution pursuant to Clause 4.1 above, and to the extent that the same does not comprise COTS Software, the winning Competitor shall deliver a copy of the winning Solution in source code and object code form to UK Government or its authorised representative, together with a description of the resources required and the instructions necessary to build and run the Solution successfully.
As a condition of the award of a prize, the delivered Solution must be capable of being built and run by UK Government or its authorised representative in a non-internet facing environment and generating the winning Entry.
Ownership of and IP in Entries and Solutions
Entries to Data Challenges shall comprise CSV files describing the Competitor's response to the relevant Data Challenge. Once uploaded to the Website, Entries shall become the property of UK Government.
Competitors warrant that their Entries and Solutions are their own original work, or where third party material is incorporated this is used with the necessary permissions, licences or consents, in which case the relevant material and third parties shall be clearly identified.
In respect of any COTS Software and COTS IPR used by a Competitor in a Solution:
the Competitor warrants that it has title to the same or the licences necessary to lawfully use the same for that purpose; and
in respect of a winning Entry, the Competitor shall provide UK Government or its authorised representative with a list of all such COTS Software and COTS IPR together with evidence of its title to or rights to lawfully use the same for that purpose.
In respect of any Non-COTS Software and Non-COTS IPR used by a Competitor in a Solution:
in respect of a winning Entry:
the Competitor shall provide UK Government or its authorised representative with a list of all such Non-COTS Software and Non-COTS IPR; and
the Competitor shall make all such Non-COTS Software and Non-COTS IPR available either generally on an MIT open source licence or by granting UK Government a worldwide, perpetual, irrevocable, royalty-free, non-exclusive, sub-licensable licence to use, modify, adapt, enhance, create derivative works of and exploit all such Non-COTS Software and Non-COTS IPR for any purpose relating to the exercise of UK Government's (or any central UK Government body's) business or function.
Intellectual Property in Data Challenges and Challenge Materials
Intellectual property rights in Data Challenges and Challenge Materials belong to UK Government, its contractors and their respective licensors. Competitors are hereby authorised to download and use the Data Challenge Materials for the purposes of taking part in Data Challenges only.
Competitors shall not reproduce, publish, resell or distribute the Challenge Materials and shall delete the same at the end of the Data Challenge.
Competitors shall not use any Challenge Materials in a defamatory or deceptive context, or in a manner that could be considered libellous, obscene or illegal, or give rise to a claim for unfair competition.
Competitors shall use suitable measures to prevent persons who have not agreed to these Official Rules from gaining access to the Challenge Materials.
Aside from the limited rights described in Clause 7.1 above, participation in a Data Challenge shall not be construed as granting or conferring on the Competitor any title, rights or interests in the Challenge Materials.
Third Party IPR Indemnity
A Competitor shall at all times on written demand indemnify UK Government and BAE Systems (each an “Indemnified Person”) and keep them indemnified against all losses, liabilities, damages, costs and expenses (including legal fees) incurred by or awarded against them arising from any claim or action by a third party that:
the relevant Indemnified Person’s possession and/or use of an Entry or a Solution originating from the Competitor infringes the intellectual property rights of a third party; or
the Competitor’s use of the Challenge Materials is contrary to Clause 7 of these Official Rules.
Exclusion of Warranties and Liability
Subject always to Clauses 9.2:
The Website, the Data Challenges and the Challenge Materials are provided “as is” and without warranty as to accuracy, completeness, availability, suitability, or fitness for any particular purpose. All implied conditions, warranties and representations in relation to the provision of the Website, the Data Challenges and the Challenge Materials are hereby excluded.
UK Government excludes all liability, whether in contract, tort (including negligence) breach of statutory duty or otherwise, even if foreseeable, arising under or in connection with use of, inability to use, or reliance on, the Website or any of its content, one or more Data Challenges, or any Challenge Materials.
Nothing in these Official Rules excludes or limits UK Government’s liability:
for death or personal injury caused by its negligence or the negligence of its employees, agents or subcontractors;
for fraud or fraudulent misrepresentation; or
where it would be unlawful to do so.
Third Party Rights
The indemnity at Clause 8 shall be enforceable against a Competitor by BAE Systems. Apart from that, a person who is not a party to this agreement shall not have any rights under the Contracts (Rights of Third Parties) Act 1999 to enforce any term of this agreement, but this does not affect any right or remedy of a third party which exists, or is available, apart from that Act.
Governing law and jurisdiction
These Official Rules shall be governed by and construed in accordance with the laws of England and the courts of England and Wales shall have exclusive jurisdiction in respect of any dispute or claim that arises out of or in connection with them, provided always that where a Competitor is a consumer (that is, an individual acting for purposes which are wholly or mainly outside their trade, business, craft or profession) resident in Northern Ireland, Scotland or another EU Member State they may also bring proceedings in their home jurisdiction.