Support #3658

DE: INSPIRE CSW Interface - needed filters to resolve coupling between datasets and services

Added by Armin Retterath 3 months ago. Updated about 1 month ago.

Status:ClosedStart date:26 Jul 2019
Priority:NormalDue date:
Assignee:Angelo Quaglia% Done:

100%

Category:Discovery
Target version:-
Submitting Organisation:GDI-DE Knowledge-Base relevant?:No
Proactive:No Keyword #1:
Country:DE - Germany Keyword #2:
Originating UI:Discovery Keyword #3:

Description

i have a question:

is it possible to also support bbox filters, <and> combined text filters and operatesOn filters?


we integrated the german csw with success and it will be a very good sign, if the european csw may be used in a same way .
here are some example filters which we use to support a combined dataset/service search:

two anytext strings with <and>:

<?xml version="1.0" encoding="UTF-8"?><csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.0.0"><ogc:Filter><ogc:And><ogc:PropertyIsEqualTo><ogc:PropertyName>Type</ogc:PropertyName><ogc:Literal>dataset</ogc:Literal></ogc:PropertyIsEqualTo><ogc:And><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>wald*</ogc:Literal></ogc:PropertyIsLike><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>baum*</ogc:Literal></ogc:PropertyIsLike></ogc:And></ogc:And></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

two anytext strings and one box:
<?xml version="1.0" encoding="UTF-8"?><csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.0.0"><ogc:Filter><ogc:And><ogc:PropertyIsEqualTo><ogc:PropertyName>Type</ogc:PropertyName><ogc:Literal>dataset</ogc:Literal></ogc:PropertyIsEqualTo><ogc:And><ogc:And><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>bplan*</ogc:Literal></ogc:PropertyIsLike><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>wa*</ogc:Literal></ogc:PropertyIsLike></ogc:And><ogc:BBOX><ogc:PropertyName>BoundingBox</ogc:PropertyName><gml:Box xmlns:gml="http://www.opengis.net/gml" srsName="EPSG:4326"><gml:coordinates decimal="." cs="," ts=" ">7.18,50.28 7.26,50.35</gml:coordinates></gml:Box></ogc:BBOX></ogc:And></ogc:And></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

operateson -  datasetid <or> fileidentifier
<?xml version="1.0" encoding="UTF-8"?><csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.0.0"><ogc:Filter><ogc:Or><ogc:PropertyIsEqualTo><ogc:PropertyName>OperatesOn</ogc:PropertyName><ogc:Literal>https://gdi-catalog.bmel.de/gdi-geonetwork/srv/ger//xml_iso19139?uuid=71b31668-746b-40a2-975a-172a87318e45</ogc:Literal></ogc:PropertyIsEqualTo><ogc:PropertyIsEqualTo><ogc:PropertyName>OperatesOn</ogc:PropertyName><ogc:Literal>71b31668-746b-40a2-975a-172a87318e45</ogc:Literal></ogc:PropertyIsEqualTo></ogc:Or></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

this filters allow a search for metadata and after it the search for the related datasets (async via ajax).

you can see an example on:
https://opendata.geoportal.rlp.de


* select englisch language (right)
* search for 'kruft'
* select Germany - 'Search in' - facet on the left side (german csw)
* Open 'Datasets (36) - you see the dataset record and below the corresponding services (view/download)


we also have two webservices for a simple json based metadata search (for searching the inspire csw you have to set the parameter catalogueId=5)

only datasets:
https://opendata.geoportal.rlp.de/mapbender/php/mod_callCswMetadata.php?catalogueId=4&searchText=bplan,wa&searchResources=dataset&searchPages=1&searchBbox=7.18,50.28,7.26,50.35&searchTypeBbox=intersects


datasets and services:
https://opendata.geoportal.rlp.de/mapbender/php/mod_callCswMetadata.php?catalogueId=4&searchText=bplan,wa&searchResources=dataset,service&searchPages=1,1&searchBbox=7.18,50.28,7.26,50.35&searchTypeBbox=intersects


resolver for coupled resources:
https://opendata.geoportal.rlp.de/mapbender/php/mod_getCoupledResourcesForDataset.php?getRecordByIdUrl=https%3A%2F%2Fgdk.gdi-de.org%2Fgdi-de%2Fsrv%2Feng%2Fcsw%3Frequest%3DGetRecordById%26service%3DCSW%26version%3D2.0.2%26Id%3D71b31668-746b-40a2-975a-172a87318e45%26ElementSetName%3Dfull%26OUTPUTSCHEMA%3Dhttp%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd

this webservices get the metadata from catalogue and does a service search with an operateson filter - see the example above


thanx in advance and best regards from germany,

armin

History

#1 Updated by Daniele Francioli 3 months ago

  • Status changed from New to Assigned

#2 Updated by Angelo Quaglia 3 months ago

  • Status changed from Assigned to Feedback

Hi Armin,

it is nice to hear from you.

Yes, of course. The work has actually been already done for the most part.

I cannot set a priority on this myself but I am quite sure it will be implemented.

Best regards,

Angelo

 

 

 

 

#3 Updated by Angelo Quaglia 2 months ago

Dear Armin,

I have had a look.

For the bbox the code is actually mostly there but there is a problem with this syntax:

                        <ogc:BBOX>
                            <ogc:PropertyName>BoundingBox</ogc:PropertyName>
                            <gml:Box xmlns:gml="http://www.opengis.net/gml" srsName="EPSG:4326">
                                <gml:coordinates decimal="." cs="," ts=" ">7.18,50.28 7.26,50.35</gml:coordinates>
                            </gml:Box>
                        </ogc:BBOX>

 

Using:

<csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd">

 

 

This is the expected syntax :

<ogc:BBOX>
  <ogc:PropertyName>BoundingBox</ogc:PropertyName>
  <gml:Envelope  srsName="EPSG:4326">
    <gml:coordinates decimal="." cs="," ts=" ">7.18,50.28 7.26,50.35</gml:coordinates>
  </gml:Envelope>
</ogc:BBOX>
 
There is still some work to be done to parse the gml:coordinates string.
 
While instead this is already supported:
 
<ogc:BBOX>
  <ogc:PropertyName>BoundingBox</ogc:PropertyName>
    <gml:Envelope srsName="EPSG:4326">
      <gml:lowerCorner>7.18,50.28</gml:lowerCorner>
      <gml:upperCorner>7.26,50.35</gml:upperCorner>
    </gml:Envelope>
</ogc:BBOX>

 

 

 

Concerning the operatesOn, the INSPIRE Geoportal does compare only with the dataset Unique Resouce Identifier Code(s).

So, the fileIdentifier must be among the dataset Unique Resouce Identifier Codes for the filter to work

This appears to be inline with the AP ISO 1.0

 

 
Best regards,
Angelo
 

#4 Updated by Armin Retterath 2 months ago

hi angelo,

thanx for the information, i succeeded with following spatial filer:

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" service="CSW" version="2.0.2" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.1.0"><ogc:Filter><ogc:And><ogc:PropertyIsEqualTo><ogc:PropertyName>Type</ogc:PropertyName><ogc:Literal>dataset</ogc:Literal></ogc:PropertyIsEqualTo><ogc:BBOX><ogc:PropertyName>BoundingBox</ogc:PropertyName><gml:Envelope srsName="EPSG:4326"><gml:lowerCorner>7.18159618172 50.2823608933</gml:lowerCorner><gml:upperCorner>7.26750846535 50.3502633407</gml:upperCorner></gml:Envelope></ogc:BBOX></ogc:And></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

either your csw and well as geonetwork support this syntax ;-) .

https://opendata.geoportal.rlp.de/mapbender/php/mod_callCswMetadata.php?catalogueId=7&searchText=*&searchResources=dataset&searchPages=1&searchBbox=6,48,8,51&searchTypeBbox=intersects

but i had to switch from <csw:Constraint version="1.0.0"> to <csw:Constraint version="1.1.0">

and the separator for the coordinates is blank not comma

i will do some further test ;-)

 

 

 

#5 Updated by Angelo Quaglia 2 months ago

Hi Armin,

many thanks for the testing and the feedback: much appreciated!

 

Concerning the operatesOn  it was a matter of adding two lines of code, which I added, so it will be supported in the next update.

 

Concerning the spatial filter, The INSPIRE Geoportal is based on ISO AP 1.0, which requires version 1.1.0 for the constraint language as a common base for interoperability between catalogues:

OGC 07-045
8.3 Query facilities

OGC Filter Encoding is an XML based encoding of the OGC_Common Query Language. This query language (Version 1.1.0) must be supported by all catalogue instances of this profile in order to support search interoperability. In addition, the plain textual encoding of CQL version 2.0 may be supported.

Using the more "modern" lower and upper corner syntax does not require further work on the INSPIRE Geoportal, even though only EPSG:4326 is currently supported.

 

Best regards,

Angelo

#6 Updated by Angelo Quaglia about 1 month ago

Hi Armin,

 

I have added the code to support the coupledResources and the correponding unit test.

Please note that the query will only match the MD_Identifier.code value(s) found  in the dataset metadata, after resolving the xlink:href present in the operatesOn elements of the service metadata.

So, neither the uuidref nor the xlink:href values will be matched unless they are also added as a MD_Identifier.code in the target dataset metadata.

 

             <gmd:identifier>
                  <gmd:MD_Identifier>
                     <gmd:code>
                        <gco:CharacterString>https://gdi-catalog.bmel.de/gdi-geonetwork/srv/ger/xml_iso19139?uuid=71b31668-746b-40a2-975a-172a87318e45</gco:CharacterString>
                     </gmd:code>
                  </gmd:MD_Identifier>
               </gmd:identifier>
So , the following filter will not find the record:
 

 

            <ogc:Filter>
                <ogc:Or>
                    <ogc:PropertyIsEqualTo>
                        <ogc:PropertyName>OperatesOn</ogc:PropertyName>
                        <ogc:Literal>https://urldefense.proofpoint.com/v2/url?u=https-3A__gdi-2Dcatalog.bmel.de_gdi-2Dgeonetwork_srv_ger__xml-5Fiso19139-3Fuuid-3D71b31668-2D746b-2D40a2-2D975a-2D172a87318e45&d=DwIFaQ&c=8NwulVB6ucrjuSGiwL_ckQ&r=rkGYKLPaOUqxct2w5TSZOxnUk1JFsyFLvragbFFbO6k&m=2NKNcpt9YfJvspVbTr7c0X8jnsK_1eC4U2o9i2lEzjI&s=YzgiAB3u7ipzBeBdxf4PcrY66SWpwdxrS9V4DJvVCc0&e= </ogc:Literal>
                    </ogc:PropertyIsEqualTo>
                    <ogc:PropertyIsEqualTo>
                        <ogc:PropertyName>OperatesOn</ogc:PropertyName>
                        <ogc:Literal>71b31668-746b-40a2-975a-172a87318e45</ogc:Literal>
                    </ogc:PropertyIsEqualTo>
                </ogc:Or>
            </ogc:Filter>
 
 

 

For the bounding box, only this syntax is supported:

 

                        <ogc:BBOX>
                            <ogc:PropertyName>BoundingBox</ogc:PropertyName>
                            <gml:Envelope srsName="EPSG:4326">
                                <gml:lowerCorner>7.18 50.28</gml:lowerCorner>
                                <gml:upperCorner>7.26 50.35</gml:upperCorner>
                            </gml:Envelope>
                        </ogc:BBOX>
 

The changes will be published sometimes next week.

 

Best regards,

Angelo

 

#7 Updated by Armin Retterath about 1 month ago

hi angelo,

this will be very good ;-) .

one other problem is the logical filter option - the bbox filter has no problem, but if you combine textfilter with a spatial filter, the csw will return an error, which cannot be evaluated (html).

i think it may be a easy to fix.

here are the example request for the inspire catalogue:

spatial filter - ok:

<?xml version="1.0" encoding="UTF-8"?><csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.1.0"><ogc:Filter><ogc:And><ogc:PropertyIsEqualTo><ogc:PropertyName>Type</ogc:PropertyName><ogc:Literal>dataset</ogc:Literal></ogc:PropertyIsEqualTo><ogc:BBOX><ogc:PropertyName>BoundingBox</ogc:PropertyName><gml:Envelope srsName="EPSG:4326"><gml:lowerCorner>7.18159618172 50.2823608933</gml:lowerCorner><gml:upperCorner>7.26750846535 50.3502633407</gml:upperCorner></gml:Envelope></ogc:BBOX></ogc:And></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

textual filter - ok (one text):

<?xml version="1.0" encoding="UTF-8"?><csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.1.0"><ogc:Filter><ogc:And><ogc:PropertyIsEqualTo><ogc:PropertyName>Type</ogc:PropertyName><ogc:Literal>dataset</ogc:Literal></ogc:PropertyIsEqualTo><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>forest*</ogc:Literal></ogc:PropertyIsLike></ogc:And></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

textual filter with errors (two words - forest & environment):

<?xml version="1.0" encoding="UTF-8"?><csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.1.0"><ogc:Filter><ogc:And><ogc:PropertyIsEqualTo><ogc:PropertyName>Type</ogc:PropertyName><ogc:Literal>dataset</ogc:Literal></ogc:PropertyIsEqualTo><ogc:And><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>forest*</ogc:Literal></ogc:PropertyIsLike><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>environment*</ogc:Literal></ogc:PropertyIsLike></ogc:And></ogc:And></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

spatial filter and textual filter with errors:

<?xml version="1.0" encoding="UTF-8"?><csw:GetRecords service="CSW" version="2.0.2" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" maxRecords="10" startPosition="1" outputSchema="http://www.isotc211.org/2005/gmd" resultType="results"><csw:Query typeNames="csw:Record"><csw:ElementSetName>full</csw:ElementSetName><csw:Constraint version="1.1.0"><ogc:Filter><ogc:And><ogc:PropertyIsEqualTo><ogc:PropertyName>Type</ogc:PropertyName><ogc:Literal>dataset</ogc:Literal></ogc:PropertyIsEqualTo><ogc:And><ogc:PropertyIsLike wildCard="*" singleChar="_" escapeChar=""><ogc:PropertyName>AnyText</ogc:PropertyName><ogc:Literal>wald*</ogc:Literal></ogc:PropertyIsLike><ogc:BBOX><ogc:PropertyName>BoundingBox</ogc:PropertyName><gml:Envelope srsName="EPSG:4326"><gml:lowerCorner>7.18159618172 50.2823608933</gml:lowerCorner><gml:upperCorner>7.26750846535 50.3502633407</gml:upperCorner></gml:Envelope></ogc:BBOX></ogc:And></ogc:And></ogc:Filter></csw:Constraint></csw:Query></csw:GetRecords>

maybe the problem is the logical filter - do you have some hints? (geonetwork evaluates the filter)

thanx in advance and best regards from germany ;-)

armin

 

#8 Updated by Angelo Quaglia about 1 month ago

Hi Armin,

many thanks for the additional testcases.

I tried them out and, thanks to the fixes I have previously implemented, they are correctly processed.

Best regards,

Angelo

#9 Updated by Angelo Quaglia about 1 month ago

Hi Amrin,

due to the number of other changes, I encountered a few issues to address during the build.

I hope to be able to deploy tonight.

Best regards,

Angelo

#10 Updated by Armin Retterath about 1 month ago

Hi Angelo,

that would be very nice ;-) - thanx a lot.

Another question: Is there a possibility to use the old validator2 from

https://inspire-geoportal.ec.europa.eu/GeoportalProxyWebServices/resources/INSPIREResourceTester or should we now use the etf validator?

best regards from germany,

Armin

#11 Updated by Angelo Quaglia about 1 month ago

Yes, you should switch to the etf, as it is the reference tool for the validation of INSPIRE Resources.

 

 

 

#13 Updated by Armin Retterath about 1 month ago

 thanx  a lot for the work ;-) - i thinks, that ims... is an old address - maybe  https://gdk.gdi-de.org/gdi-de/srv/eng/csw?request=GetCapabilities&service=CSW&version=2.0.2& is the newer one. i will aks anja and sara if the old address should be used any more

our collegues updated the german catalog from geonetwork 2.8 to 3.6 one week ago - it seems to be a little bit faster than before ;-)

actually i work on our new wfs 3.0 proxy to do the next step in the direction to linked open data ;-) - but there is also much to do

https://www.geoportal.rlp.de/mapbender/php/mod_linkedDataProxy.php?wfsid=472&collection=ms%3Aglascontainer&items=all&f=html&limit=200

https://www.geoportal.rlp.de/mapbender/php/mod_linkedDataProxy.php?wfsid=486&collection=ms%3Aunesco_welterbe&items=all&f=html

https://www.geoportal.rlp.de/linked_open_data/

 

#14 Updated by Angelo Quaglia about 1 month ago

That's cool, thanks! I will forward this to a colleague of mine who is working on WFS 3.0.

 

I have published the changes to the OGC CSW 2.0.2 adapter.

1) HTTPS is now advertised:

 

2) The filters you wrote in the opening statement of this issue now work.

3) OperatesOn is supported:

Please note that only the Coupled Resource Unique Resource Identifier Code is matched:

 

4) In order to facilitate investigation about the operatesOn filter, I have added to the Resource Browser the tag clouds for Coupled Resources.

 

Best regards,

Angelo

#15 Updated by Armin Retterath about 1 month ago

hi angelo - all the things we needed work very well ;-) - thanx a lot!!!

we are now able to do pan-european searches with multiple searchtexts and combined with spatial a search filter. the couplings are resolved as we expected and i think this was a very big step for the implementation of inspire ;-) .

now we have persuasiate the others to fill in the right fields in their metadata (hahaha...)

best regards from germany and have a nice weekend - you deserved it! - trink some beer in the sun ;-)

armin

#16 Updated by Armin Retterath about 1 month ago

  • Status changed from Feedback to Closed
  • % Done changed from 0 to 100

#17 Updated by Angelo Quaglia about 1 month ago

Hi Armin,

That's great!

Many thanks for your positive feedback.

A nice week-end for you, too, and best regards,


Angelo

Also available in: Atom PDF