Support #3285

NO: Question regarding the INSPIRE Geoportal Discovery Service Register

Added by Angelo Quaglia about 2 years ago. Updated about 2 years ago.

Status:FeedbackStart date:05 Jun 2018
Priority:NormalDue date:
Assignee:Angelo Quaglia% Done:

0%

Category:Harvesting process
Target version:-
Submitting Organisation:NO Knowledge-Base relevant?:
Proactive:No Keyword #1:
Country:NO - Norway Keyword #2:
Originating UI: Keyword #3:

Description

From: Lars Inge Arnevik
Sent: 05 June 2018 14:04:23 (UTC+01:00) Brussels, Copenhagen, Madrid, Paris
To: inspire-geoportal@jrc.ec.europa.eu
Subject: Question regarding the INSPIRE Geoportal Discovery Service Register
 

 

Hi!

 

I wonder when the next harvesting from Geonorge will take place? I think last harvest was 31. of May and I guess it was interrupted because of our monthly maintenance. We are working on improving the metadata and it is motivating to see the results from the error report J

 

 

Best regards

Lars-Inge

History

#1 Updated by Angelo Quaglia about 2 years ago

  • Status changed from Assigned to Feedback

Dear Lars,

I have just kicked it off.

I had to halt the automatic harvestings in the last few days for maintenance.

Best regards,

Angelo

#2 Updated by Angelo Quaglia about 2 years ago

Dear Lars,

the INSPIRE Geoportal is having big trouble with the following Atom based Download Service:

https://nedlasting.geonorge.no/geonorge/Tjenestefeed.xml

 

1) It is declaring 422 Spatial Data Sets but many entries point to the same metadata, for example 

<inspire_dls:spatial_dataset_identifier_code>041f1e6e-bdbc-4091-b48f-8a5990f3cc5b</inspire_dls:spatial_dataset_identifier_code>

<inspire_dls:spatial_dataset_identifier_namespace>http://www.geonorge.no/</inspire_dls:spatial_dataset_identifier_namespace>

is present in 3 different entries.

 

2) While processing it, the INSPIRE Geoportal gets back hundreds of errors, like this one:

https://nedlasting.geonorge.no/api/download/file//8b4304ea-4fb0-479c-a24d-fa225e2c6e97/21AB2A0C-9CBC-4165-93CE-930849D2CFF3"

refused the connection for the following reason:

"Internal Server Error"

 

Could you please look urgently into this?

 

Kind regards,

Angelo

 

 

#3 Updated by Lars-Inge Arnevik about 2 years ago

Dear Angelo!

1) I have to come back to this, but at the moment we are enhancing the metadata in our catalogue, and to add correct namespace and dataset identifier is one of the tasks.

2) I guess this is because the Atom Feed registered in our catalogue also contains datasets that have access restrictions. These datasets are not included in the harvesting from our portal, but I guess the validation tool are checking all the entries in the Atom Feed. In the harvesting, dated 29. May, this seemed to work well, even though the datasets was restricted from access at that time as well. I have now excluded the Atom Feed Service from our CSW-endpoint to see if the report can be produced.

We have to come back to a solution how to fix this, but it would be nice if you could satr a new harvesting.

Best regards,

Lars-Inge

#4 Updated by Angelo Quaglia about 2 years ago

Dear Lars-Inge,

many thanks for your prompt intervention.

I have just started a new harvesting.

Yes, I have already made some modifications to the server in order for it to be able to handle that service but I will need to restart the web server to make them effective and see if they really work.

The server is under an extremely heavy load at the moment, so the harvesting will take some time to complete.

Best regards,

Angelo

 

 

#5 Updated by Angelo Quaglia about 2 years ago

Comment Edit

Dear Lars-Inge,

the new harvesting results are available:

http://inspire-geoportal.ec.europa.eu/INSPIREWebServices/resources/INSPIREResource/INSPIRE-ccf3ad04-9003-11e3-aef9-52540004b857_20180606-170533/services/1/PullResults

The linkage between INSPIRE Resources is improving:

Best regards,

Angelo

#6 Updated by Lars-Inge Arnevik about 2 years ago

Thank you Angelo!

This is a nice tool and give us a good overwiev of the status. It also give us motivation for furher enhancements on the metadata.

 

Best regards

Lars-Inge

#7 Updated by Lars-Inge Arnevik about 2 years ago

Looks like the validation has failed. Do you see any errors that we can solve, to get the report up and working again?

#8 Updated by Angelo Quaglia about 2 years ago

Dear Lars,

a new harvesting for Norway started this morning at 8.15, so I would expect it to become available shortly.

Best regards,

Angelo

 

#9 Updated by Lars-Inge Arnevik about 2 years ago

Dear Angelo!

Is it a way to request the data from the report in a way where we can get information about the validation for each dataset. Then we could include the validation status  and link to error report in our metadata-klient and flag the status of the metadata for the data owners.

 

Best regards, Lars-Inge

#10 Updated by Angelo Quaglia about 2 years ago

Dear Lars,

the latest harvesting completed successfully.

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-ccf3ad04-9003-11e3-aef9-52540004b857_20180606-170533/services/1/PullResults/

However, the number of metadata records decreased from 490 to 468, so the INSPIRE Geoportal parked the results.

Since there were no errors, I assume it is OK, so I have promoted the latest results to production.

The average interoperability indicator gained two points: from 23.70% to 25.85%

 

Yes, all resource representations and evaluation reports are generated in xml according to this schema:

http://inspire-geoportal.ec.europa.eu/schemas/inspire/geoportal/1.0/geoportal.xsd

From the xml other representations are generated (json, jsonp, html, Solr xml).

In addition, all resources are indexed in a Solr instance, accessible from the outside.

For example, if you wanted to get the Solr representation of the dataset "N50 Kartdata", you could write something like"

http://inspire-geoportal.ec.europa.eu/solr/select?fq=memberStateCountryCode:no&fq=sourceMetadataResourceLocator:\/*&fq=uriCode:"N50 Kartdata"&q=*:*&facet=true&wt=json

You get back this:
{
"responseHeader": {

...

"interoperabilityAspect": [
"DOWNLOAD_MATCHING_DATA_IS_AVAILABLE",
"ALL_DOWNLOADS_MATCHING_DATA_HAVE_COMPLETE_METADATA"
],
"parentResourceMetadataResourceLocator": "/INSPIRE-ccf3ad04-9003-11e3-aef9-52540004b857_20180612-081519/services/1/PullResults/421-440",
"relatedResourceMetadataResourceLocator": [
"/INSPIRE-ccf3ad04-9003-11e3-aef9-52540004b857_20180612-081519/services/1/PullResults/81-100/services/13/resourceLocator1/download/services/1/downloadDatasets/0"
],
"missingInteroperabilityAspect": [
"ALL_LAYERS_MATCHING_DATA_HAVE_COMPLETE_METADATA",
"LAYER_MATCHING_DATA_IS_AVAILABLE",
"RESOURCE_PREVIEW_IMAGE_IS_AVAILABLE",
"RESOURCE_METADATA_IS_COMPLETE",
"INSPIRE_SPATIAL_OBJECT_IS_AVAILABLE"
],
"id": "/INSPIRE-ccf3ad04-9003-11e3-aef9-52540004b857_20180612-081519/services/1/PullResults/421-440/datasets/9",

Then if you open this URL setting the HTTP header Accept:application/xml:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-ccf3ad04-9003-11e3-aef9-52540004b857_20180612-081519/services/1/PullResults/421-440/datasets/9

You get this:

</ns2:OriginalRepresentation>
<ns2:GeoportalErrorCode>error.geoportal.discovery.metadata.xmlvalidation</ns2:GeoportalErrorCode>
<ns2:GeoportalErrorCode>error.geoportal.proxy.error.sourcevalidation</ns2:GeoportalErrorCode>
<ns2:GeoportalWarningCode>
error.geoportal.proxy.metadata.element.inspirespatialdatatheme.missing
</ns2:GeoportalWarningCode>
<ns2:CompletenessIndicator>94.44444444444444</ns2:CompletenessIndicator>
<ns2:InteroperabilityIndicator>28.57142857142857</ns2:InteroperabilityIndicator>
<ns2:SpatialDataCoverageLevel>national</ns2:SpatialDataCoverageLevel>
<ns2:InteroperabilityAspect>DOWNLOAD_MATCHING_DATA_IS_AVAILABLE</ns2:InteroperabilityAspect>
<ns2:InteroperabilityAspect>ALL_DOWNLOADS_MATCHING_DATA_HAVE_COMPLETE_METADATA</ns2:InteroperabilityAspect>
<ns2:MissingInteroperabilityAspect>ALL_LAYERS_MATCHING_DATA_HAVE_COMPLETE_METADATA</ns2:MissingInteroperabilityAspect>
<ns2:MissingInteroperabilityAspect>LAYER_MATCHING_DATA_IS_AVAILABLE</ns2:MissingInteroperabilityAspect>
<ns2:MissingInteroperabilityAspect>RESOURCE_PREVIEW_IMAGE_IS_AVAILABLE</ns2:MissingInteroperabilityAspect>
<ns2:MissingInteroperabilityAspect>RESOURCE_METADATA_IS_COMPLETE</ns2:MissingInteroperabilityAspect>
<ns2:MissingInteroperabilityAspect>INSPIRE_SPATIAL_OBJECT_IS_AVAILABLE</ns2:MissingInteroperabilityAspect>
</ns2:Resource>
 
Just tell me if you need more information.
 
Best regards,
Angelo

 

#11 Updated by Lars-Inge Arnevik about 2 years ago

Dear Angelo, this is good!

We then can use the identifier for the resource (for datasets) to get the validation reports. But we don't have a identifier for the services. 

I try to use the "sourceMetadataResourceLocator", but I don't get any hits. I guess this parameter should contain the fileIdentifier of the metadata?

This request should also return "N50 Kartdata": http://inspire-geoportal.ec.europa.eu/solr/select?fq=memberStateCountryCode:no&fq=sourceMetadataResourceLocator:%22ea192681-d039-42ec-b1bc-f3ce04c189ac%22&fq=uriCode:\/*&q=*:*&facet=true&wt=json

Best regards

Lars-Inge

 

 

 

#12 Updated by Angelo Quaglia about 2 years ago

You can use the field remoteMetadataIdentifier that maps to the fileIdentifier in the ISO19139 metadata

#13 Updated by Angelo Quaglia about 2 years ago

By the way, you can use any field used for tag clouds in the Resource Browser:

http://inspire-geoportal.ec.europa.eu/proxybrowser/#q=*%3A*

#14 Updated by Lars-Inge Arnevik about 2 years ago

Thanks, this is great :-)

#15 Updated by Lars-Inge Arnevik about 2 years ago

Dear Angelo!

I'm sorry to say that we have an issue with some harvested datasets from norwegian laser and ortophoto projects. These datasets have some errors in the metadata, and get a tag that registers them as Inspire datasets. We are working together with the owner of these datasets to change this. At the moment we are using manual methods to remove thes metadata from the Inspire csw-endpoint, with the result that the number of metadata records will vary from time to time. I think this is a check you are running, to secure that the harvest went well... And maybe this is the reason why the two last validation reports are withdrawn.

Regards

Lars-Inge

#16 Updated by Angelo Quaglia about 2 years ago

Der Lars,

many thanks for the notice.

Yes, that's the reason.

I am promoting the latest good one available to production.

Bert regards,

Angelo

Also available in: Atom PDF