Support #3573

SK: Missing donwloadable datasets metadata in Geoportal console

Added by Martin Tuchyna over 1 year ago. Updated 5 months ago.

Status:ClosedStart date:22 Apr 2019
Priority:HighDue date:
Assignee:Angelo Quaglia% Done:

0%

Category:Harvesting results
Target version:-
Submitting Organisation:SK Knowledge-Base relevant?:No
Proactive:No Keyword #1:
Country:SK - Slovakia Keyword #2:
Originating UI: Keyword #3:

Description

Dear Geoportal Team,

in connection to the harvesting results:

"Question related to harvest results with id INSPIRE-da77b119-9d6e-11e7-b5a7-52540023a883_20190419-015028"

We would like to clarify, why there were indicated 2 missing downloadable datasets (snap 20190422_01.png + 20190422_00.png), whilst based on our check the endpoints for missing 2 downloadable datasets are available.

These two missing records are:

1.INSPIRE - Hydrografia:
https://rpi.gov.sk/rpi_csw/service.svc/get?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&outputschema=http://www.isotc211.org/2005/gmd&Id=https://data.gov.sk/set/rpi/gmd/17316219/SK_UGKK_ZBGIS_INSPIRE_HY

2.INSPIRE - Parcely katastra nehnuteľností (KN):
https://rpi.gov.sk/rpi_csw/service.svc/get?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&outputschema=http://www.isotc211.org/2005/gmd&Id=https://data.gov.sk/set/rpi/gmd/17316219/SK_UGKK_ESKN_INSPIRE_CP(KN)

We have encountered missing first one already during the previous harvesting (20190418_01.png).

Could you please let us know, why these metadata records dissapeared, whilst they were there before (and are still available in currently published harvesting - snap 20190422_02.png )?

Thanks in advance,

Best regards,

Martin

20190422_00.PNG (82.6 KB) Martin Tuchyna, 22 Apr 2019 08:09 pm

20190422_02.PNG (84.8 KB) Martin Tuchyna, 22 Apr 2019 08:09 pm

20190422_01.PNG (113 KB) Martin Tuchyna, 22 Apr 2019 08:09 pm

20190418_01.PNG (158 KB) Martin Tuchyna, 22 Apr 2019 08:11 pm

2601
2602
2603
2604

History

#1 Updated by Angelo Quaglia over 1 year ago

  • Category set to Harvesting results
  • Status changed from New to Assigned
  • Assignee set to Angelo Quaglia
  • Submitting Organisation set to SK

#2 Updated by Angelo Quaglia over 1 year ago

  • Status changed from Assigned to Feedback

Dear Martin,

for the first dataset the problem is due to this missing aspect:

  • DATA_DOWNLOAD_LINK_IS_AVAILABLE

And that problem was triggered by this issue, as reported in the resource report:

Indeed, if I try now this request:

https://zbgisws.skgeodesy.sk/inspire_hydrography_wfs/service.svc/get?request=GetFeature&Language=eng&CRS=http://www.opengis.net/def/crs/EPSG/0/4258&DataSetIdNamespace=https://data.gov.sk/set/rpi/dat/17316219/&service=WFS&count=10&STOREDQUERY_ID=http://inspire.ec.europa.eu/operation/download/GetSpatialDataSet&version=2.0.0&DataSetIdCode=https://data.gov.sk/set/rpi/dat/17316219/SK_UGKK_ZBGIS_INSPIRE_HY

I get an answer but very well the after the minimum initial response time required by the Network Service Regulation.

Best regards,

Angelo

#3 Updated by Angelo Quaglia over 1 year ago

For the second dataset, I suggest you first fix the issues with WFS Capabilities:

http://inspire-geoportal.ec.europa.eu/sandbox/resources/INSPIRE-da77b119-9d6e-11e7-b5a7-52540023a883_20190419-015028/services/1/PullResults/141-160/services/20/resourceLocator1/download/services/1/resourceReport/

 

You seem to have chose Scenario 2 for the Extended Capabilities. However, you decided to instantiate also the MetadataURL element but assigning to it a wrong URL, It shall point to the service metadata record.

Once you have fixed that, I will investigate about why the INSPIRE Geoportal can exstablish no download link.

 

In summary, the metadata records are correctly processed but do not have a download link.

Best regards,

Angelo

 

#4 Updated by Angelo Quaglia over 1 year ago

  • Subject changed from Missing donwloadable datasets metadata in Geoportal console to SK: Missing donwloadable datasets metadata in Geoportal console

#5 Updated by Martin Tuchyna over 1 year ago

Angelo Quaglia wrote:

Dear Martin, for the first dataset the problem is due to this missing aspect: DATA_DOWNLOAD_LINK_IS_AVAILABLE And that problem was triggered by this issue, as reported in the resource report: The predefined stored query "http://inspire.ec.europa.eu/operation/download/GetSpatialDataSet", for the request "https://zbgisws.skgeodesy.sk/inspire_hydrography_wfs/service.svc/get?request=GetFeature&Language=eng&CRS=http://www.opengis.net/def/crs/EPSG/0/4258&DataSetIdNamespace=https://data.gov.sk/set/rpi/dat/17316219/&service=WFS&count=10&STOREDQUERY_ID=http://inspire.ec.europa.eu/operation/download/GetSpatialDataSet&version=2.0.0&DataSetIdCode=https://data.gov.sk/set/rpi/dat/17316219/SK_UGKK_ZBGIS_INSPIRE_HY" returned the following error: "The request to the HTTP resource at url https://zbgisws.skgeodesy.sk/inspire_hydrography_wfs/service.svc/get?request=GetFeature&Language=eng&CRS=http://www.opengis.net/def/crs/EPSG/0/4258&DataSetIdNamespace=https://data.gov.sk/set/rpi/dat/17316219/&service=WFS&count=10&STOREDQUERY_ID=http://inspire.ec.europa.eu/operation/download/GetSpatialDataSet&version=2.0.0&DataSetIdCode=https://data.gov.sk/set/rpi/dat/17316219/SK_UGKK_ZBGIS_INSPIRE_HY was unsuccesful because The resource url: "https://zbgisws.skgeodesy.sk/inspire_hydrography_wfs/service.svc/get?request=GetFeature&Language=eng&CRS=http://www.opengis.net/def/crs/EPSG/0/4258&DataSetIdNamespace=https://data.gov.sk/set/rpi/dat/17316219/&service=WFS&count=10&STOREDQUERY_ID=http://inspire.ec.europa.eu/operation/download/GetSpatialDataSet&version=2.0.0&DataSetIdCode=https://data.gov.sk/set/rpi/dat/17316219/SK_UGKK_ZBGIS_INSPIRE_HY" did not respond within 30000ms"   Indeed, if I try now this request: https://zbgisws.skgeodesy.sk/inspire_hydrography_wfs/service.svc/get?request=GetFeature&Language=eng&CRS=http://www.opengis.net/def/crs/EPSG/0/4258&DataSetIdNamespace=https://data.gov.sk/set/rpi/dat/17316219/&service=WFS&count=10&STOREDQUERY_ID=http://inspire.ec.europa.eu/operation/download/GetSpatialDataSet&version=2.0.0&DataSetIdCode=https://data.gov.sk/set/rpi/dat/17316219/SK_UGKK_ZBGIS_INSPIRE_HY I get an answer but very well the after the minimum initial response time required by the Network Service Regulation. Best regards, Angelo

Thank you Angelo.

Does that mean, that during the validation of the resources you already check the requirements for the quality of service"?

If so, despite the the legal base, this is important information, which of course we will stress in communication to providers, but would be great to provide detailed documentation of the documentation for the Verification Tool as already requested via Support #3549.

For the query you provided, we would like to check, whether the part of this verification logic

is this one requesting 10 features, or the whole dataset, which might have consequences.

We tried curl on the stored query which failed your validation and achieved response times around 5-6 seconds. 
Example:
 
Total: 5.649039s
Total: 5.331205s
Total: 5.189408s

Thanks in advance.

Best regards,

Martin and Tomas

#6 Updated by Angelo Quaglia over 1 year ago

Dear Martin,

I confirm you that the URL for which a timeout was reported, today appears to be working fine.

Once more I need to say that the INSPIRE Geoportal is not a validator but an INSPIRE Client, the main INSPIRE Client in a manner of speaking.

As such, the INSPIRE Geoportal must, of course, set a timeout on each connection, otherwise a harvesting could last forever.

Instead of using a arbitrary timeout, the INSPIRE Geoportal relies on the minimum performance requirements described in the Network Service Regulation, in which every Operation (Get Network Service Metadata, Discover Metadata, Get Spatial Data Set, etc.) has, potentially, a different maximum initial response time. 

If the Network Service performances are very near to the limit, the "physiological network impedance" does matter but it is quite a rare situation in my experience.

Should the need arise, I could certainly add a tolerance but the entity of the tolerance would then be arbitrary, so I am not very much in favour of that.

However, if the INSPIRE Geoportal receives the initial response within the timeout, you can be confident that your Network Service meets the INSPIRE requirements.

This consitutes a good end-to-end test.

 

Yes, the INSPIRE Geoportal requests maximum 10 features in the case of a WFS, because common WFS implementations are not suited for downloading a complete dataset.

Also in the case of an Atom based Download Service, the download is forcibly interrupted after 1 MByte is downloaded, just enough to give time statistics a reasonable meaningfulness.

As you can see, the INSPIRE Geoportal gives useful information about the performances:

Best regards,

Angelo

#7 Updated by Tomas Kliment over 1 year ago

Ciao Angelo,

I checked the WFS GetCapabilities for which you reported the following: 

The Service Metadata URL "https://zbgisws.skgeodesy.sk/zbgiscsw/service.svc/get?REQUEST=GetRecordById&SERVICE=CSW&VERSION=2.0.2&OUTPUTSCHEMA=http://www.isotc211.org/2005/gmd&ELEMENTSETNAME=full&Id=https://data.gov.sk/set/rpi/gmd/17316219/SK_UGKK_ESKN_INSPIRE_CP(KN)" is invalid because: "the resource found does not match the resource type"  

You seem to have chose Scenario 2 for the Extended Capabilities. However, you decided to instantiate also the MetadataURL element but assigning to it a wrong URL, It shall point to the service metadata record.

Once you have fixed that, I will investigate about why the INSPIRE Geoportal can exstablish no download link.

 
By checking the MetadataURL element it seems to be assigning a good URL to the own WFS service INSPIRE metadata.
 

Regading the mashup of both scenarios, shall we advice the service provider to decide and use only one scenario, or using both is still fine and does not exclude the network service to be sucessfully processed by the main INSPIRE Geoportal client?

thanks a lot,

Tomas

 

 

Angelo Quaglia wrote:

For the second dataset, I suggest you first fix the issues with WFS Capabilities: http://inspire-geoportal.ec.europa.eu/sandbox/resources/INSPIRE-da77b119-9d6e-11e7-b5a7-52540023a883_20190419-015028/services/1/PullResults/141-160/services/20/resourceLocator1/download/services/1/resourceReport/ The Service Metadata URL "https://zbgisws.skgeodesy.sk/zbgiscsw/service.svc/get?REQUEST=GetRecordById&SERVICE=CSW&VERSION=2.0.2&OUTPUTSCHEMA=http://www.isotc211.org/2005/gmd&ELEMENTSETNAME=full&Id=https://data.gov.sk/set/rpi/gmd/17316219/SK_UGKK_ESKN_INSPIRE_CP(KN)" is invalid because: "the resource found does not match the resource type"   You seem to have chose Scenario 2 for the Extended Capabilities. However, you decided to instantiate also the MetadataURL element but assigning to it a wrong URL, It shall point to the service metadata record. Once you have fixed that, I will investigate about why the INSPIRE Geoportal can exstablish no download link.   In summary, the metadata records are correctly processed but do not have a download link. Best regards, Angelo  

 

#8 Updated by Angelo Quaglia over 1 year ago

Dear Tomas,

I think we are talking about different things.

if I open this WFS URL:

https://inspire.skgeodesy.sk/eskn/rest/services/INSPIREWFS/kn_wfs_inspire/GeoDataServer/exts/InspireFeatureDownload/service?SERVICE=WFS&REQUEST=GetCapabilities

 

and I open the MetadataURL:

https://zbgisws.skgeodesy.sk/zbgiscsw/service.svc/get?REQUEST=GetRecordById&SERVICE=CSW&VERSION=2.0.2&OUTPUTSCHEMA=http://www.isotc211.org/2005/gmd&ELEMENTSETNAME=full&Id=https://data.gov.sk/set/rpi/gmd/17316219/SK_UGKK_ESKN_INSPIRE_CP(KN)

I obtain a metadata describing a dataset:

The INSPIRE Geoportal complains exactly about this problem.

Yes, they can keep using this scenario.

Best regards,

Angelo

#9 Updated by Angelo Quaglia over 1 year ago

So, the problem with the second dataset (INSPIRE - Parcely katastra nehnuteľností (KN)) is indeed due to the wrong MetadataUrl link in the Extended Capabilities.

The INSPIRE Geoportal expects to get the dataset metadata from the service coupled resources but it does not find them.

If you put the link to the service metadata, it will show up as downloadable on the INSPIRE Geoportal.

Best regards,

Angelo

 

#10 Updated by Tomas Kliment over 1 year ago

Dear Angelo,

You are right, I mixed it up with the view service linking, now I see the problem and gonna communicate its fix to the provider.

Thanks a lot,

Tomas

 

#11 Updated by Tomas Kliment 5 months ago

  • Status changed from Feedback to Closed

Also available in: Atom PDF