MIWP-Action #2127

2016.3: Validation and conformity testing (previously MIWP-5)

Added by Lorena Hernandez Quiros about 5 years ago. Updated over 1 year ago.

Status:ClosedStart date:01 Apr 2014
Priority:NormalDue date:30 Jun 2017
Assignee:Michael Lutz% Done:

100%


Description

Issue:

As INSPIRE is coming into a practical implementation phase there is a great need of tools for validation (metadata, service and data). There is a validation service (web service) available at the EU-portal and some countries have also developed tools for validation of metadata and services, for instance in the Netherlands and Germany. These validators might include slightly different interpretations of standards. To ensure that result from a tests of conformity are identical, a common, officially approved, validator should be accessible from INSPIRE web.

Software vendors claim that their products are INSPIRE-compliant without having undergone a certification process.

The abstract test suites in INSPIRE data specifications define the set of tests to be applied but there is no reference implementation of those abstract test suites.

Proposed change or action:

  • Develop a commonly agreed European validator for data, metadata and network services (incl. performance testing)
    • Testing should focus on interoperability of applications and services
    • legal compliance cannot be checked based on conformity with TG
    • The validation rules should be made explicit so that data providers in Members States know what is validated upon exactly and how is validated
    • the MIG should jointly agree on the tests to be included in the validator
    • Investigate feasibility of executable tests and/or tools or services for checking conformance of datasets with the various DS
  • Establish rule that all new TG need to ATS and executable tests
  • Discuss the possibilities for setting up a compliance certification facility and process similar to the OGC

Organisational set-up:

This action is carried out by a MIG sub-group including MIG representatives and experts from the pool of experts, with support from the ARE3NA ISA action for the implementation of the commonly agreed European validator. 


Related issues

Related to MIG-T - MIWP-Action #2440: MIWP-7b: Extension of Download Service Technical Guidelin... Closed 01 May 2015

History

#1 Updated by Michael Lutz about 5 years ago

  • Assignee changed from Michael Lutz to Daniela Hogrebe

#2 Updated by Michael Lutz about 5 years ago

  • Description updated (diff)

#3 Updated by Daniela Hogrebe almost 5 years ago

  • Assignee changed from Daniela Hogrebe to Carlo Cipolloni

#4 Updated by Michael Lutz almost 5 years ago

  • Tracker changed from Task to MIWP-Action

#5 Updated by Carlo Cipolloni over 4 years ago

  • File ToR_MIG_sub-group_Validation_conformity_testing.docx added
  • File MIWP5_projectplan_annex1.docx added

#6 Updated by Carlo Cipolloni over 4 years ago

The draft Terms of reference (ToR) for this sub-group and the annex project plan are attached. Please add your comments to the ToR as comments to this issue (before next MIG Meeting -28 August).

#7 Updated by Michael Lutz over 4 years ago

  • Status changed from Submitted to Work in progress
  • Start date changed from 10 Mar 2014 to 01 Apr 2014
  • % Done changed from 0 to 10
Status 2014-09-04:
  • This task is documented on the MIWP-5 wiki
  • An initial workshop was held on 15-16 May to define the scope and workplan for this task
  • A workplan and terms of reference for a temporary MIG sub-group have been developed.
  • The work of the sub-group is planned to start in Septemeber 2014.
  • A draft for a change control process for the validation suite (based on the IT Infrastructure Library (ITIL) change management process) has been proposed.
  • A technical workshop on comparing existing conformance tests (in MS and at European level) for metadata and network services and agreeing on common test rules is planned for November/December 2014.

#8 Updated by Michael Lutz over 4 years ago

  • Status changed from Work in progress to Endorsed

#9 Updated by Carlo Cipolloni about 4 years ago

  • Description updated (diff)

Status 28-04-2015

Progress since the last meeting:

Subtask 1.2: it is started to collect comment or revision on the Change management process document endorsed.

Subtask 1.3: it is started to look at existing OGC documents related to certification policies, potentially applicable to the purposes of our task.

Subtask 2.1: it is created a template to collect use case and a revision on that is started.

Subtask 2.3: it has updated the list.

Subtask 3.3: a repository for all the view services ATS has been created an the task start to address all the issues relate to those ATS not takes in account by the expert group.

Subtask 3.4: a specific invitation to BGS to participate and coordinate the subtask has been done; BGS agrees to take part to subtask but not to drive it.

Subtask 3.5: start the analysis of ATS implementation regarding the NRZ as well start to analyse ATS that are common with PS to just replicate effort. Upgrading the Validation Service to new release of OGC GML 3.2 Test Suite and Providing additional files to the Schematron library.

Tasks/deliverables planned for the next month(s):

The task 2.1 will propose the first use cases, a Use Case of a “common validator”, enabling an “integrated” validation of the 3 components (metadata, datasets, services) will be proposed in may and discussed as implication during the GWF/INSPIRE Conference

The task 3.1 will analyse the ATS document produced by expert group to evaluate and vote if it is implementable.

The task 3.2 a repository to address the ATS document provides by T. ?ezník will be created.

The task 3.3 planned to finish ATS description document for middle of May.

The task 3.5 started the ATS analysis and ETS implementation for NRZ, maybe some first example will be available for GWF/INSPIRE Conference.

Overall status:  ahead of schedule|on schedule|behind schedule (plus if necessary some comments)

Risks or issues for discussion in the MIG-T:

How we manage the subtasks that haven’t a person to drive (i.e. subtask 3.2 and 3.4? We collapse with other or we investigate resources to create external expert group financed.

#10 Updated by Carlo Cipolloni almost 4 years ago

Progress since the last meeting:

  • A new work plan has been redesigned;
  • The Metadata ATSs document and their comparison with IR Requirement has been transmitted to MIWP-8;
  • A use case for a common validator has been proposed;
  • First eENVplus/EEA results in the schematron production on PS theme.

Tasks/deliverables planned for the next month(s):

  • We have planed to concentrate on two main development testsuite tasks: metadata and view service;
  • We will decide which part of work we have to put in a procurement;
  • we will build a schematron library exploiting the competences in and the collaborative environment of the Thematic Clusters in relation with Task 3.5.

Overall status:

  • In late on the schedule time

Risks or issues for discussion in the MIG-T:

  • External procurement

#11 Updated by Carlo Cipolloni over 3 years ago

Progress since the last meeting:

  • The second version of Change Management process has been created and uploaded on the web
  • A common use case has been descripted To validate metadata/dataset/services in a common validator and after that a positive discussion about other singol case are proposed to validate the single part as metadata, service and dataset.
  • The MIWP-8 has sent back comments on the comparison on IR and TG requirements table.
  • To solve ATS lack effort and have a good step foward JRC launched a contract with Spatineo regarding ATS subtask 3.x.

Tasks/deliverables planned for the next month(s):

  • Use case design document will be created and descripted
  • Action regarding ATS waiting that new documents are provided to ask comments
  • ETS JRC will launch a request for offer through our ABC-III framework contract (Freddy or Michael can explain better)

Overall status: behind schedule

  • Behind scheduled, but we have made up for delay.
  • Most of the documents are working in progress and are opened to be implemented (use case, change management, etc).
  • the 1st draft of the TG should have been delivered end of May

Risks or issues for discussion in the MIG-T:

  • Small interaction between MIWP-5 and contractor of ATS and ETS activities, we need to ensure that MIWP-5 people provide comments in a short time and contractor send us also documentation in progress.
  • Misalianments between ETS and Use cases proposed.

#12 Updated by Carlo Cipolloni over 3 years ago

Progress since the last meeting:

  • ATSs subtask 3.x are in progress and developed by Ilkka Rinne with a specific contract

Tasks/deliverables planned for the next month(s):

  • Use case design documents will be improved
  • Comments to ATS will be provided
  • ETS JRC will launch a request for offer through our ABC-III framework contract (Freddy or Michael can explain better)

Overall status: behind schedule

  • Behind scheduled, but we have made up for delay.
  • Most of the documents are working in progress and opened could be implemented (use case, change management, etc) in a flexible way.

Risks or issues for discussion in the MIG-T:

  • Lack of interaction between MIWP-5 and contractor of ATS and ETS activities, we need to ensure that MIWP-5 people provide comments in a short time and contractor send us also documentation in progress.
  • Misalignments between ETSs and Use cases proposed and/or existing validation tools to re-use and reduce as much possible the effort.

#13 Updated by Michael Lutz over 3 years ago

Progress since the last meeting:

  • Joint kick-off meeting between the contractor (PwC and interactive instruments) and MIWP-5 for the ARE3NA contract for implementing the INSPIRE reference validator
  • Comments (by 2 sub-group members) on draft ATSs (see table

Tasks/deliverables planned for the next month(s):

  • MIG-T/P consultation on the ATS
    • Timetable
      • Incorporate MIWP-5 comments into ATSs and launch MS consultation (JRC): 29/01/2016
      • MS review (5 weeks) (MS): 04/03/2015
      • MIWP-5 comment resolution meeting (MIWP-5): week 14-18/03/2016
      • Resolution of comments using the Github issue tracker (4 weeks) (PwC/ii, JRC & MIWP-5 members): 31/03/2016
    • The ATS for SDS will be updated to reflect the proposed updated guidelines. It will be submitted into consultation when available.
    • The ATS for MD will be based on the current TG version. An additional ATS should be created for the proposed update of the TG (when available).
    • ii/PwC will review the ATSs from the viewpoints of 
      • clarity (are the descriptions clear enough to allow implementation?) and
      • feasibility (are the proposed tests implementable?)
    • The ATSs will be submitted to both MIG-T and MIG-P in parallel. The MIG representatives will be encouraged to involve the implementers of national validators (where available) in the review.

Overall status: n/a (the schedule needs to be re-defined as agreed between the MIWP-5 sub-group, JRC and the ARe3NA contractors

Risks or issues for discussion in the MIG-T:

  • Lack of interaction between MIWP-5 and contractor
  • Lack of reviews from MS during the consultation

#14 Updated by Michael Lutz over 3 years ago

Progress since the last meeting:

  • ATS sent out for MS consultation (#2685)
  • PwC/ii have started work on the design report for the INSPIRE testing framework, based on the outcomes of the December workshop (v0.1)

Tasks/deliverables planned for the next month(s):

  • MIG-T/P consultation on the ATS
    • Timetable
      • MS review (5 weeks) (MS): 11/03/2015
      • MIWP-5 comment resolution meeting (MIWP-5): 16/03/2016 10:00-12:00 CET
      • Resolution of comments using the Github issue tracker (4 weeks) (PwC/ii, JRC & MIWP-5 members): 08/04/2016
      • The ATS for SDS will be updated to reflect the proposed updated guidelines. It will be submitted into consultation when available.
      • The ATS for MD will be based on the current TG version. An additional ATS should be created for the proposed update of the TG (when available).
    • ii/PwC will review the ATSs from the viewpoints of 
      • clarity (are the descriptions clear enough to allow implementation?) and
      • feasibility (are the proposed tests implementable?)
    • The ATSs will be submitted to both MIG-T and MIG-P in parallel. The MIG representatives will be encouraged to involve the implementers of national validators (where available) in the review.
  • INSPIRE testing framework - design report
    • ii to update the document to v0.2
    • JRC to distribute v0.2 to MIWP-5 architecture sub-team with a request for comments: 22/02/2016

Overall status: n/a (the schedule needs to be re-defined as agreed between the MIWP-5 sub-group, JRC and the ARe3NA contractors)

Risks or issues for discussion in the MIG-T:

  • Procedure for MS review -- Question from DE:

As you asked the MIG P-members like me I am wondering how to proceed best. 

 The ATS is a very technical matter. I could completely transfer my feedback to Daniela Hogrebe, so she is doing it not only as a MIG T-member, but on behalf of me as  german MIG P-member, too.

On the other hand - if there will be another review by the chair of MIG P - I could consider to keep quiet now and give an opinion later. Do you know if there will be a later review or even voting by MIG P? I am not so sure after having a look in the terms of reference of the MIG-P as it is a very technical matter.

  • Lack of interaction between MIWP-5 and contractor
  • Lack of reviews from MS during the consultation

#15 Updated by Michael Lutz over 3 years ago

  • Status changed from Endorsed to Work in progress

#16 Updated by Michael Lutz about 3 years ago

Progress since the last meeting:

  • 350+ comments received during the MS consultation of the ATS (#2685)
    • around 100 from MS and 250+ from JRC and the ARE3NA contractors (PwC/ii) developing the INSPIRE testing framework
  • The comment resolution process was discussed in a MIWP-5 meeting on 16 March
  • Only one minor comment received from the MIWP-5 architecture team on the design report for the INSPIRE testing framework

Tasks/deliverables planned for the next month(s):

  • Resolution of ATS comments
    • Process
      • PwC/ii to propose resolutions for MS and JRC comments
      • MIWP-5 and JRC to propose resolutions for PwC/ii comments
      • Github will be used for the comment resolution process
      • Issues that need further discussion will be addressed in a comment resolution web-conference
    • Timetable
      • Resolution of comments using the Github issue tracker (PwC/ii, JRC & MIWP-5 members): 08/04/2016
      • Comment resolution web-conference: week of 11-15/04/2016
      • The ATS for SDS will be updated to reflect the proposed updated guidelines. It will be submitted into consultation when available.
      • The ATS for MD will be based on the current TG version. An additional ATS should be created for the proposed update of the TG (when available).

Overall status: n/a (the schedule needs to be re-defined as agreed between the MIWP-5 sub-group, JRC and the ARe3NA contractors)

Risks or issues for discussion in the MIG-T:

  • Is another endorsement step (by MIG-P) needed for the ATS?
  • Lack of interaction between MIWP-5 and contractor

#17 Updated by Michael Lutz almost 3 years ago

  • % Done changed from 10 to 40

Progress since the last meeting:

  • Remaining open issues were discussed in a comment resolution web-conference on 28 April.

Tasks/deliverables planned for the next month(s):

  • Resolution of ATS comments
    • Update issues based on the decisions
    • Implement agreed changes in the ATSs
  • Develop updated ATSs for DS TG
  • Develop ATS for MD TG v2.0 (with MIWP-8)

Overall status: n/a (the schedule needs to be re-defined as agreed between the MIWP-5 sub-group, JRC and the ARe3NA contractors)

Risks or issues for discussion in the MIG-T:

 

#18 Updated by Michael Lutz almost 3 years ago

  • Subject changed from MIWP-5: Validation and conformity testing to 2016.3: Validation and conformity testing (previously MIWP-5)
  • Description updated (diff)
  • Due date set to 30 Jun 2017

Progress since the last meeting:

  • Implementation of agreed changes to ATS ongoing
  • The action description has been slightly revised in the new MIWP 2016-2020 to reflect the real organisational set-up incl. considerable support through the ARE3NA ISA action.

Tasks/deliverables planned for the next month(s):

  • Complete implementation of agreed changes in the ATSs
  • Start development of the INSPIRE test framework and first ETSs (to be demo'ed at the INSPIRE conference)
  • Develop updated ATSs for DS TG
  • Develop ATS for MD TG v2.0 (with MIWP-8)

Overall status: ARE3NA work is proceeding according to plan

Risks or issues for discussion in the MIG-T: none

#19 Updated by Michael Lutz over 2 years ago

Progress since the last meeting:

  • Implementation of agreed changes to ATS for NS and MD has been completed, including the agreed restucturing of the ATS
  • Development of ATS for DS has started, focusing on cross-cutting requirements (from the framework TGs and the DS template) and Annex I DS (HY, PS, AU, CP)
  • Initial version of the INSPIRE test framework and first ETSs presented and demonstrated at the INSPIRE conference

Tasks/deliverables planned for the next month(s):

  • Release initial version of the INSPIRE test framework and first ETSs for internal testing
  • Continue development of updated ATSs for DS TG
  • Develop ATS for MD TG v2.0 (with MIWP-8)

Overall status: ARE3NA work is proceeding according to plan

Risks or issues for discussion in the MIG-T: 

  • Is there a need for an MIWP-5 meeting? If so, what should the discussion focus on?

#20 Updated by Michael Lutz over 2 years ago

  • Project changed from MIG-T to MIWP 2016-2020

#21 Updated by Michael Lutz over 2 years ago

  • % Done changed from 40 to 70

Progress since the last meeting

  • The development version of the INSPIRE validator has been made available to volunteer testers at http://35.157.17.37/etf-webapp/ (please write to are3na@jrc.ec.europa.eu to get the credentials for accessing the instance).
  • Draft Executable Test Suites for interoperabilty metadata and for the data themes Addresses, Administrative Units, Cadastral Parcels, Geographic Names and Transport Networks have been added. The Executable Test Suites for metadata and for spatial object types of the annex I data themes are now complete.
  • Some issues in the other Executable Test Suites have been fixed. 
  • The development on the next version of the software (beta version) with additional capabilities as well as the Executable Test Suite for the "WFS pre-defined" download services will be complete in January.

Tasks/deliverables planned for the next month(s)

  • Add Executive Test Suites for "Pre-defined ATOM" and "Direct WFS" (the latter is trivial as there is no requirement in addition to the OGC conformance classes)
  • Add test driver for OGC Team Engine (to support WFS 2.0 and GML 3.2 CITE tests of OGC)
  • Documentation (for users, for developers (API), for ETS developers, for administrators)
  • Address known open issues, various smaller improvements
  • Host validator at JRC
  • Another demo could be organised in February/March 2017 to introduce the beta release (potentially to a wider audience)

Overall status

  • ARE3NA work is proceeding according to plan

Risks or issues for discussion in the MIG-T

  • It will be important to develop any missing ATS and ETS jointly in the MIG - put the planning (incl. resources necessary and available) as a topic in the next MIG-T face-to-face meeting in March 2017.

#22 Updated by Michael Lutz about 2 years ago

Progress since the last meeting

  • Continued development and testing of the INSPIRE validator under the ARE3NA ISA action.
  • Proposed new governance structure for action 2016.3 presented and principally agreed at 38th MIG-T meeting.

Tasks/deliverables planned for the next month(s)

  • Complete developments under current ARE3NA contract
  • Host validator at JRC
  • Organise a webinar with TC for developing ATS and ETS
  • Update action mandate for action 2016.3 (in particular for new governance structure) and present to MIG in June meeting
  • Discuss options and funding sources for further development of the validator (ATS, ETS, testing framework)

Overall status

  • ARE3NA work is proceeding according to plan

Risks or issues for discussion in the MIG-T

 

#23 Updated by Michael Lutz almost 2 years ago

Progress since the last meeting

Tasks/deliverables planned for the next month(s)

  • Complete developments under current ARE3NA contract
  • Host validator at JRC
  • Organise a webinar with TC for developing ATS and ETS (end of June/beginning of July)
  • Discuss options and funding sources for further development of the validator (ATS, ETS, testing framework)

Overall status

  • ARE3NA work is proceeding according to plan
  • Action MIWP-5/2016.3 to be formally closed at the June MIG meeting and then restarted according to the new action mandate

Risks or issues for discussion in the MIG-T

 

#24 Updated by Michael Lutz almost 2 years ago

  • Assignee changed from Carlo Cipolloni to Michael Lutz
  • % Done changed from 70 to 90

Progress since the last meeting

Tasks/deliverables planned for the next month(s)

  • Host validator at JRC
  • Webinar with TC for developing ATS and ETS: 3/7, 14:30-16:00 CEST
  • Call for participation in new 2016.3 sub-group

Overall status

  • ARE3NA work is proceeding according to plan
  • Action MIWP-5/2016.3 formally completed at the June MIG meeting and will be restarted according to the new action mandate

Risks or issues for discussion in the MIG-T

#25 Updated by Michael Lutz over 1 year ago

  • Status changed from Work in progress to Closed
  • % Done changed from 90 to 100

The action was formally closed in the MIG meeting in June 2017.

Also available in: Atom PDF