Support #3650

BE: Stability of Harvesting

Added by Bart Verbeeck about 1 year ago. Updated about 1 year ago.

Status:FeedbackStart date:17 Jul 2019
Priority:NormalDue date:
Assignee:Angelo Quaglia% Done:

0%

Category:Harvesting process
Target version:-
Submitting Organisation:BE Knowledge-Base relevant?:No
Proactive:No Keyword #1:
Country:BE - Belgium Keyword #2:
Originating UI: Keyword #3:

Description

Goodmorning


We did a harvest on July 11 and on July 16.

For the data below, the viewservice metadata was not harvested on the 11th, but it was harvested the 16th
Gebouwen
http://inspire-geoportal.ec.europa.eu/download_details.html?view=downloadDetails&resourceId=%2FINSPIRE-f0c91711-ece0-11e8-a08e-52540023a883_20190716-144320%2Fservices%2F1%2FPullResults%2F261-280%2Fdatasets%2F14&expandedSection=metadata


For the services below, the downloadservice metadata was found the 11th, but it was not found yesterday.
Watersystemen
http://inspire-geoportal.ec.europa.eu/download_details.html?view=downloadDetails&resourceId=%2FINSPIRE-f0c91711-ece0-11e8-a08e-52540023a883_20190716-144320%2Fservices%2F1%2FPullResults%2F181-200%2Fdatasets%2F19&expandedSection=metadata

Biologische Waarderingskaart
http://inspire-geoportal.ec.europa.eu/download_details.html?view=downloadDetails&resourceId=%2FINSPIRE-f0c91711-ece0-11e8-a08e-52540023a883_20190716-144320%2Fservices%2F1%2FPullResults%2F261-280%2Fdatasets%2F12&expandedSection=metadata


From our side, I think the services metadata has not been changed

Is there
* a problem with the metadata or
* a problem with our csw or
* a problem with the inspire harvester or
* another problem

Thanks

Bart Verbeeck

History

#1 Updated by Angelo Quaglia about 1 year ago

  • Status changed from New to Assigned
  • Assignee set to Davide Artasensi

Dear Bart,

I have assigned this issue to my colleague Davide who will answer.

Best regards,
Angelo

#2 Updated by Angelo Quaglia about 1 year ago

  • Subject changed from Stability of Harvesting to BE: Stability of Harvesting
  • Submitting Organisation set to BE
  • Country set to BE - Belgium

#3 Updated by Daniele Francioli about 1 year ago

  • Assignee changed from Davide Artasensi to Angelo Quaglia

#4 Updated by Angelo Quaglia about 1 year ago

Dear Bart,

1) nothing has changed in the INSPIRE Geoportal between July 11th and July 16th.

I also see that the latest harvesting report (16th July) is perfectly fine:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-f0c91711-ece0-11e8-a08e-52540023a883_20190716-144320/services/1/PullResults

and it says:

Result of the interaction with the Discovery Service

Resources available for discovery: 343, Expected Resource Count: 343, Actual Resource Count : 343

In addition, all records have been correctly processed and indexed:

 

2) When you need to ascertain the issues with linkages between a dataset and its Network Services, please use the Linkage Checker:

http://inspire-geoportal.ec.europa.eu/linkagechecker.html

 

For each dataset you want to check you will need:

1) The Data Set metadata or its fileIdentifier

2) The View Service metadata or its fileIdentifier

3) The Download Service metadata or its fileIdentifier

 

You have listed three datasets:

For the data below, the viewservice metadata was not harvested on the 11th, but it was harvested the 16th
Gebouwen

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-f0c91711-ece0-11e8-a08e-52540023a883_20190716-144320/services/1/PullResults/261-280/14.iso19139.xml

<?xml version="1.0" encoding="UTF-8"?><gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:gml="http://www.opengis.net/gml" xmlns:geonet="http://www.fao.org/geonetwork">
         <gmd:fileIdentifier>
            <gco:CharacterString>bfad8c70-e8f7-4f89-a141-8feba4a1fb8e</gco:CharacterString>
         </gmd:fileIdentifier>

So, this is fine.

 

For the services below, the downloadservice metadata was found the 11th, but it was not found yesterday.
Watersystemen

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-f0c91711-ece0-11e8-a08e-52540023a883_20190716-144320/services/1/PullResults/181-200/19.iso19139.xml

<gmd:fileIdentifier>
<gco:CharacterString>8ac7a847-c322-4a60-ac77-42c3064d1392</gco:CharacterString>
</gmd:fileIdentifier>

OK, could you please identify the metadata of the Download Service?

 

Biologische Waarderingskaart

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-f0c91711-ece0-11e8-a08e-52540023a883_20190716-144320/services/1/PullResults/261-280/12.iso19139.xml

<gmd:fileIdentifier>
<gco:CharacterString>bf31d5c7-e97d-4f71-a453-5584371e7559</gco:CharacterString>
</gmd:fileIdentifier>

OK, could you please identify the metadata of the Download Service?

Best regards,

Angelo

 

#5 Updated by Angelo Quaglia about 1 year ago

  • Status changed from Assigned to Feedback

#6 Updated by Bart Verbeeck about 1 year ago

Thank you Angelo

 

For Gebouwen everything is indeed ok now

For Biologische Waarderingskaart, there was a problem with the storedQuery. Sorry, my mistake.

 

But for Watersystemen, I dont understand what the problem is.

The metadatarecords for data/view/download are

http://geonetwork.geopunt.be/zoekdienst/srv/dut/csw?REQUEST=GetRecordById&SERVICE=CSW&VERSION=2.0.2&ELEMENTSETNAME=full&ID=8ac7a847-c322-4a60-ac77-42c3064d1392&OUTPUTSCHEMA=http://www.isotc211.org/2005/gmd
http://geonetwork.geopunt.be/zoekdienst/srv/dut/csw?REQUEST=GetRecordById&SERVICE=CSW&VERSION=2.0.2&ELEMENTSETNAME=full&ID=4d361174-860d-ee21-4b3e-cafc-a2fc-2151-d3f42167&OUTPUTSCHEMA=http://www.isotc211.org/2005/gmd
http://geonetwork.geopunt.be/zoekdienst/srv/dut/csw?REQUEST=GetRecordById&SERVICE=CSW&VERSION=2.0.2&ELEMENTSETNAME=full&ID=c9f59bd7-f9dc-06ca-3676-eaf0-bc98-221a-8a6bb16a&OUTPUTSCHEMA=http://www.isotc211.org/2005/gmd

 

When I use the Linkage checker, evrything is green, only the "Download Service metadata exists in the INSPIRE Geoportal", is red. And this is actually my question: do you have any idea why the metadata of the downloadservice ihasn't been harvested?

 

Best regards

 

Bart

#7 Updated by Angelo Quaglia about 1 year ago

Hi Bart,
that should be an easy one.
 
The Linkage Checker is correct (which is not surprising as the Linkage Checker merely invokes a Web Service exposed by the INSPIRE Geoportal):
that metadata document is not among the 343 declared to be discoverable by your Discovery Service.
 
I think the missing metadata has not been stamped with a keyword that matches the filter specified by Bart Cosyn when he registered the Discovery Service on the INSPIRE Geoportal:

<ns2:OGCFilter>
    <ogc:Filter xmlns:ogc="http://www.opengis.net/ogc">
        <ogc:PropertyIsLike escapeChar="\" singleChar="_" wildCard="%">
            <ogc:PropertyName>keyword</ogc:PropertyName>
            <ogc:Literal>%Lijst%M%R%INSPIRE%</ogc:Literal>
        </ogc:PropertyIsLike>
    </ogc:Filter>
</ns2:OGCFilter>

For example:

-<gmd:keyword>
-<gco:CharacterString>
Lijst M&R INSPIRE
</gco:CharacterString>
</gmd:keyword>
 

Best regards,

Angelo

#8 Updated by Bart Verbeeck about 1 year ago

Thank you Angelo.

 

The service metadata has that keyword, so there has to be another problem.

Maybe there is a valditation problem with the metadata (a "ë").

I solved that and I will await the harvesting results tomorrow or next week.

 

Just 1 more question (actually 2). Are there plans to adapt the linkage checker (and harvesting) to validate WCS services as download service? Or do you know of a dataset that I can check that is served using a WCS service and that is downloadable in the inspire geoportal?

 

Bart

#9 Updated by Angelo Quaglia about 1 year ago

You are welcome Bart.

1) If you mean an issue in GeoNetwork, preventing the records from being emitted, probably.

Instead, there is no validation issue on the INSPIRE Geoportal, as the record was never emitted by the Discovery Service and never received.

I have just tried.

The service now declares to be serving 342 records instead of 343:

<csw:GetRecordsResponse xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd">
    <csw:SearchStatus timestamp="2019-07-18T13:18:55" />
    <csw:SearchResults numberOfRecordsMatched="342" numberOfRecordsReturned="20" elementSet="brief" nextRecord="21">
 

However, if I ask for exactly those three identifiers:

                    <ogc:Filter xmlns:ogc="http://www.opengis.net/ogc">

                           <ogc:And>

                                 <ogc:PropertyIsLike escapeChar="\" singleChar="_" wildCard="%">

                                        <ogc:PropertyName>keyword</ogc:PropertyName>

                                        <ogc:Literal>%Lijst%M%R%INSPIRE%</ogc:Literal>

                                 </ogc:PropertyIsLike>

                                 <ogc:Or>

                                        <ogc:PropertyIsLike escapeChar="\" singleChar="_" wildCard="%">

                                               <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>

                                               <ogc:Literal>8ac7a847-c322-4a60-ac77-42c3064d1392</ogc:Literal>

                                        </ogc:PropertyIsLike>

                                        <ogc:PropertyIsLike escapeChar="\" singleChar="_" wildCard="%">

                                               <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>

                                               <ogc:Literal>4d361174-860d-ee21-4b3e-cafc-a2fc-2151-d3f42167</ogc:Literal>

                                        </ogc:PropertyIsLike>

                                        <ogc:PropertyIsLike escapeChar="\" singleChar="_" wildCard="%">

                                               <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>

                                               <ogc:Literal>c9f59bd7-f9dc-06ca-3676-eaf0-bc98-221a-8a6bb16a</ogc:Literal>

                                        </ogc:PropertyIsLike>

                                 </ogc:Or>

                           </ogc:And>

                    </ogc:Filter>

I get them back!

<csw:GetRecordsResponse xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd">
    <csw:SearchStatus timestamp="2019-07-18T13:20:47" />
    <csw:SearchResults numberOfRecordsMatched="3" numberOfRecordsReturned="3" elementSet="brief" nextRecord="0">

 

So, now there are two records less that are returned but at least our three records are.

 

2) I am working on full WCS and SOS support.

 

Best regards,

Angelo

Also available in: Atom PDF