Support #2338

PT -Direcção de Serviços de Geodesia e Informação Geográfica: Switch to the National Discovery Service

Added by Angelo Quaglia over 5 years ago. Updated over 3 years ago.

Status:ResolvedStart date:27 Feb 2015
Priority:NormalDue date:
Assignee:Angelo Quaglia% Done:

0%

Category:Discovery Service Register
Target version:-
Submitting Organisation:PT -Direcção de Serviços de Geodesia e Informação Geográfica Knowledge-Base relevant?:
Proactive:No Keyword #1:
Country:PT - Portugal Keyword #2:
Originating UI: Keyword #3:

Description

 

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 25 February 2015 14:17
To: 'Henrique Silva'
Cc: 'Rui Alves Cavaco'; 'inspire-geoportal@jrc.ec.europa.eu'; 'Freddy Fierens'; 'Michael Lutz'
Subject: RE: INSPIRE Discovery Service for Portugal
Importance: High

 

Dear Henrique,

 

I think it would be useful to clarify the current situation.

 

The metadata the INSPIRE Geoportal has been harvesting in the last months has been read from:

http://cgpr.igeo.pt/catalogo-inspire-pt/services/csw

 

That service contains records with the same fileIdentifier, something which is against the AP ISO 1.0.

 

The National Discovery Service of Portugal is instead:

http://snig.dgterritorio.pt/geoportal/csw/discovery

 

It contains many more records 19,907 (as of this morning).

 

If I understand well,  you would like to filter them because not all of them are relevant for INSPIRE.

 

 

The 11,446 resources of “unkown” type are actually tiles.

 

To filter them out you just need to use the following filter:

 

                                    <ogc:Filter xmlns:ogc="http://www.opengis.net/ogc">

                                                <ogc:Or>

                                                            <ogc:PropertyIsEqualTo>

                                                                        <ogc:PropertyName>apiso:type</ogc:PropertyName>

                                                                        <ogc:Literal>dataset</ogc:Literal>

                                                            </ogc:PropertyIsEqualTo>

                                                            <ogc:PropertyIsEqualTo>

                                                                        <ogc:PropertyName>apiso:type</ogc:PropertyName>

                                                                        <ogc:Literal>series</ogc:Literal>

                                                            </ogc:PropertyIsEqualTo>

                                                            <ogc:PropertyIsEqualTo>

                                                                        <ogc:PropertyName>apiso:type</ogc:PropertyName>

                                                                        <ogc:Literal>service</ogc:Literal>

                                                            </ogc:PropertyIsEqualTo>

                                                </ogc:Or>

                                    </ogc:Filter>

 

 

If I apply that filter,

This is what I get:

 

 

 

Metadata quality is also quite good:

 

Could you please confirm if my understanding is correct?

 

 

If that is the case, then I suggest you register http://snig.dgterritorio.pt/geoportal/csw/discovery specifying the simple filter given above.

 

 

How to reregister a service that has been deactivated in the geoportal?

 

I have tested on a local instance and I am able to reregister the same service if I simply change the URL slightly like, for example:

http://snig.dgterritorio.pt/geoportal/csw/discovery?REQUEST=GetCapabilities&SERVICE=CSW&VERSION=2.0.2

http://snig.dgterritorio.pt/geoportal/csw/discovery?REQUEST=GetCapabilities&SERVICE=CSW

http://snig.dgterritorio.pt/geoportal/csw/discovery?REQUEST=GetCapabilities

http://cgpr.igeo.pt/catalogo-inspire-pt/services/csw

 

 

Best regards,

Angelo

 

 

 

Ing. Angelo Quaglia

External Consultant

European Commission, DG Joint Research Centre
Institute for Environment and Sustainability

Digital Earth and Reference Data Unit, T.P. 262

Via E. Fermi, 2749.
I-21027 Ispra (VA)
Italy

Tel: +39 347 78 88 492
Fax: +39 0332 78 6325
e-mail: mailto:angelo.quaglia@ext.jrc.ec.europa.eu

URL: http://ies.jrc.ec.europa.eu/SDI/sdi-about-us/staff-profiles/angelo-quaglia.html

 

The views expressed are purely those of the writer and may not in any circumstances be regarded as stating an official position of the European Commission.


Related issues

Related to Geoportal Helpdesk - Support #2693: PT -Direcção de Serviços de Geodesia e Informação Geográf... Closed 16 Feb 2016

History

#1 Updated by Angelo Quaglia over 5 years ago

  • Subject changed from Portugal: National Discovery Service to Portugal: Switch to the National Discovery Service

#2 Updated by Angelo Quaglia over 5 years ago

From: Henrique Silva [mailto:hsilva@dgterritorio.pt]
Sent: 26 February 2015 13:38
To: Angelo Quaglia
Cc: Rui Cavaco; inspire-geoportal@jrc.ec.europa.eu; 'Freddy Fierens'; 'Michael Lutz'; Mário Caetano
Subject: Re: INSPIRE Discovery Service for Portugal

 

Dear Angelo,
thank you for you careful explanation.

A question:
Do you think that exposing all the metadata  catalog (except tiles)  to the INSPIRE Geoportal will have further obligations, regarding harmonization and so on?

Because, at the moment, our INSPIRE domain it's around 1000 records, far from whole records in SNIG geoportal.

With time, we will include a specific keyword, and filter the metadada to be harvest the to INSPIRE geoportal.

Thanks a lot.

Best regards,
Henrique

#3 Updated by Angelo Quaglia over 5 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 26 February 2015 14:47
To: 'Henrique Silva'
Cc: 'Rui Cavaco'; 'inspire-geoportal@jrc.ec.europa.eu'; 'Freddy Fierens'; 'Michael Lutz'; 'Mário Caetano'; 'Michel Millot'
Subject: RE: INSPIRE Discovery Service for Portugal

 

Dear Henrique,

 

INSPIRE obligations encompass datasets, series and services which belong to the INSPIRE Spatial Data Themes.

 

Here is what the INSPIRE Directive says:

 

 

For those objects, Member States must make available for harvesting by the INSPIRE Geoportal, metadata that conforms to the INSPIRE requirements.

 

 

Having said that, I think it is safe to say also that:

 

  • data harmonization obligations do not depend on whether metadata are published on the INSPIRE geoportal;

 

  • datasets not relating to the INSPIRE themes do not share the same obligations and this regardless of whether their metadata is published on the INSPIRE geoportal;

 

 

 

Considering also that results from yesterday’s harvesting looked very good in terms of metadata conformity, I think you can go ahead with the registration of the SNIG service:

 

 

 

 

You will always be free to include filters once your metadata documents have been updated.

 

One important aspect is that metadata data are maintained and also that there is no duplication in the fileIdentifier values.

 

Best regards,

Angelo

 

#4 Updated by Angelo Quaglia over 5 years ago

Login details registered and sent to Henrique Silva.

#5 Updated by Angelo Quaglia over 5 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 28 April 2015 16:12
To: 'Henrique Silva'
Cc: 'Rui Cavaco'; 'Mário Caetano'
Subject: RE: INSPIRE Discovery Service for Portugal
Importance: High

 

Dear Henrique,

 

I have seen that you finally registered http://snig.dgterritorio.pt/geoportal/csw/discovery

 

Unfortunately, you decided also to keep this very long filter:

 

<ogc:Filter>

               <ogc:Or>

                              <ogc:PropertyIsEqualTo>

                                             <ogc:PropertyName>Identifier</ogc:PropertyName>

                                             <ogc:Literal>2fd5909e-2a7a-416f-a268-b97006311c24</ogc:Literal>

                              </ogc:PropertyIsEqualTo>

…(other 1485 similar entries)...

                              <ogc:PropertyIsEqualTo>

                                             <ogc:PropertyName>Identifier</ogc:PropertyName>

                                             <ogc:Literal>4730a5899e4641a8aef9ae360a0a360d</ogc:Literal>

                              </ogc:PropertyIsEqualTo>

               </ogc:Or>

</ogc:Filter>

 

 

In addition to being difficult to manage, it is also the likely cause of the slowness with which the service responds.

Even allowing 30 seconds, it times out. I tested it also from Firefox:

 

 

 

 

 

 

The INSPIRE Network Service Regulation requires an initial response time of maximum 3 seconds:

 

See: http://inspire-geoportal.ec.europa.eu/documentation/legal-and-tgs/ns/CELEX_02009R0976-20101228_EN_TXT/?zoomToText=3%20seconds

 

 

Please try to address this issue as soon as possible as all requests fail:

 

 

See: http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20150424-192112/services/1/PullResults/

 

 

Best regards,
Angelo

 

 

Ing. Angelo Quaglia

External Consultant

European Commission, DG Joint Research Centre
Institute for Environment and Sustainability

Digital Earth and Reference Data Unit, T.P. 262

Via E. Fermi, 2749.
I-21027 Ispra (VA)
Italy

Tel: +39 347 78 88 492
Fax: +39 0332 78 6325
e-mail: mailto:angelo.quaglia@ext.jrc.ec.europa.eu

URL: http://ies.jrc.ec.europa.eu/SDI/sdi-about-us/staff-profiles/angelo-quaglia.html

 

The views expressed are purely those of the writer and may not in any circumstances be regarded as stating an official position of the European Commission.

#6 Updated by Angelo Quaglia over 5 years ago

From: Henrique Silva [mailto:hsilva321@gmail.com]
Sent: 28 April 2015 20:59
To: Angelo Quaglia
Cc: Henrique Silva; Rui Cavaco; Mário Caetano
Subject: Re: INSPIRE Discovery Service for Portugal

 

Dear Angelo,

we decided not to expose all the catalog because it has lots of information that it's not relevant to the INSPIRE Geoportal. 

 

The solution with the long filter is temporary and the harvesting to be done only monthly.

 

It's not possible to increase the timeout response?

Internally, the service is rather fast (5 seconds) to process the request, of course externally will be much longer...

 

Thanks a lot.

Best regards.  

 

#7 Updated by Angelo Quaglia over 5 years ago

Dear Henrique,

 

I have encouraging news.

 

I have found some issues with the registration of the new Discovery Service on the INSPIRE Geoportal.

 

Some values look inappropriate to me and are probably the cause of the malfunctioning.

 

Max parallel requests is set to 10 which might be too much for your service.

The maximum value I have seen is 5 max parallel request but for much more sizeable catalogues (100,000 records or more).

 

I have forced value 1 and kicked off the harvesting again.

I have also forced value 20 for the maxrecords which was 15, but put it back to 15.

 

 

It worked much better but four batches (in red, below) failed because the connection was reset by your service, could you please check on the server?

 

Here is the report:

 

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20150429-101102/services/1/PullResults/

 

 

For those 4 batches the error was:

 

 

 

In addition, “Recaching frequency” is set to daily but my understanding was that it was monthly.

 

Could you please check the values and amend update the registration?

You do not need to deactivate the current, just fill in the new values and click on “Apply Modifications”

https://ies-svn.jrc.ec.europa.eu/projects/mig-inspire/wiki/DiscoveryServiceRegister

 

 

 

 

 

Best regards,
Angelo

 

Ing. Angelo Quaglia

External Consultant

European Commission, DG Joint Research Centre
Institute for Environment and Sustainability

Digital Earth and Reference Data Unit, T.P. 262

Via E. Fermi, 2749.
I-21027 Ispra (VA)
Italy

Tel: +39 347 78 88 492
Fax: +39 0332 78 6325
e-mail: mailto:angelo.quaglia@ext.jrc.ec.europa.eu

URL: http://ies.jrc.ec.europa.eu/SDI/sdi-about-us/staff-profiles/angelo-quaglia.html

 

The views expressed are purely those of the writer and may not in any circumstances be regarded as stating an official position of the European Commission.

#8 Updated by Angelo Quaglia over 5 years ago

From: Henrique Silva [mailto:hsilva321@gmail.com]
Sent: 29 April 2015 15:31
To: Angelo Quaglia
Cc: Henrique Silva; Rui Cavaco; Mário Caetano
Subject: Re: INSPIRE Discovery Service for Portugal

 

Dear Angelo,

very good news!

 

Thanks a lot for your help.

 

For the 4 batches that didn't work, I will see in detail next week.

 

I already updated the registry to monthly.

 

Best regards.

#9 Updated by Angelo Quaglia over 5 years ago

  • Status changed from Assigned to Resolved

#10 Updated by Angelo Quaglia about 5 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 15 June 2015 09:15
To: 'Henrique Silva'
Cc: 'Rui Cavaco'; 'Mário Caetano'; 'Henrique Silva'
Subject: RE: INSPIRE Discovery Service for Portugal
Importance: High

 

Dear Henrique,

The National Discovery Service http://snig.dgterritorio.pt/geoportal/csw/discovery is having serious troubles responding to GetRecords requests.

 

Could you please have a look?

 

 

Best regards,

Angelo

 

#11 Updated by Angelo Quaglia about 5 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 15 June 2015 17:02
To: 'Henrique Silva'
Cc: 'Rui Cavaco'; 'Mário Caetano'; 'Henrique Silva'
Subject: RE: INSPIRE Discovery Service for Portugal

 

Dear Henrique,

I have disabled the harvesting from your service until further notice from your part.

 

Best regards,

Angelo

 

#12 Updated by Angelo Quaglia about 5 years ago

From: Henrique Silva [mailto:hsilva321@gmail.com]
Sent: 16 June 2015 14:10
To: Angelo Quaglia
Cc: Henrique Silva; Rui Cavaco; Mário Caetano
Subject: Re: INSPIRE Discovery Service for Portugal

 

Dear Angelo,

I tested the service from outside just now and it worked well.

It took 14 sec. to complete the request.

 

May be it was a temporary problem.

 

Best regards,

#13 Updated by Angelo Quaglia about 5 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 16 June 2015 16:13
To: 'Henrique Silva'
Cc: 'Henrique Silva'; 'Rui Cavaco'; 'Mário Caetano'
Subject: RE: INSPIRE Discovery Service for Portugal

 

Dear Henrique,
I do not know if you tried with the full query because even from my browser 30 seconds are not enough:

 

Regards,

Angelo

 

 

 

#14 Updated by Angelo Quaglia about 5 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 16 June 2015 17:12
To: 'Henrique Silva'
Cc: 'Henrique Silva'; 'Rui Cavaco'; 'Mário Caetano'
Subject: RE: INSPIRE Discovery Service for Portugal

 

Dear Henrique,

I forced the minimum timeout  to 60 seconds (20 times the maximum set by the law).

All record could be fetched:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20150616-165346/services/1/PullResults/

 

Best regards,
Angelo

 

#15 Updated by Angelo Quaglia about 5 years ago

From: Henrique Silva [mailto:hsilva321@gmail.com]
Sent: 16 June 2015 19:02
To: Angelo Quaglia
Cc: Henrique Silva; Rui Cavaco; Mário Caetano; Paulo Patrício
Subject: Re: INSPIRE Discovery Service for Portugal

 

Dear Angelo,

thanks a lot. This is a temporary measure.

 

I also try with the full query just now.

This time took only 12 sec.

 

Regards,

#16 Updated by Angelo Quaglia about 5 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 18 June 2015 09:46
To: 'Henrique Silva'
Cc: 'Henrique Silva'; 'Rui Cavaco'; 'Mário Caetano'; 'Paulo Patrício'
Subject: RE: INSPIRE Discovery Service for Portugal

 

Dear Henrique,

 

Indeed, the report at http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20150616-165346/services/1/PullResults/

shows that the initial response time ranged from 7.985 seconds up to 27.666, for the same number of records (50).

 

For the time being, I will leave the exceptional 60 seconds timeout but please tell us as soon as you have managed to normalize the situation with your National Discovery Service.

 

Best regards,

Angelo

#17 Updated by Angelo Quaglia over 4 years ago

  • Subject changed from Portugal: Switch to the National Discovery Service to PT -Direcção de Serviços de Geodesia e Informação Geográfica: Switch to the National Discovery Service
  • Status changed from Resolved to Feedback
  • Submitting Organisation set to PT -Direcção de Serviços de Geodesia e Informação Geográfica

Dear Henrique,

it seems that the performance problems with Portugal’s National Discovery Service have not been successfully addressed, yet.

All requests time out unless I specify a very high initial time out.

Similarly, the issue about the very long filter has not been addressed, either even though it was supposed to be a temporary measure.

Could you please explain why this is so?

#19 Updated by Henrique Silva over 4 years ago

Dear Angelo,
there is an agreement from PT entities to put all the INSPIRE metadata with the predefined keyword till the end of this year.

So I asked if it's possible to maintain this solution a whlie longer.

#20 Updated by Angelo Quaglia over 4 years ago

It is not to me to say if it is OK or not but many thanks for the update.

#21 Updated by Angelo Quaglia almost 4 years ago

  • Category set to Discovery Service Register

#22 Updated by Angelo Quaglia almost 4 years ago

  • Country set to PT - Portugal

#23 Updated by Angelo Quaglia over 3 years ago

  • Proactive set to No

Dear Henrique,

very often, the INSPIRE Geoportal is unable to harvest all the records because, for several batches, the initial response time gets higher than 100 seconds (5 seconds is the maximum allowed by the INSPIRE Network Service regulation for the 99% of the time):

The interaction with the remote service at "http://snig.dgterritorio.pt/geoportal/csw/discovery?service=CSW&request=GetCapabilities" ended with the following error "The resource url: "http://snig.dgterritorio.pt/geoportal/csw/discovery" did not respond within 107500ms"

There are several examples of that>:

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20161122-001407/services/1/PullResults/

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20161123-080354/services/1/PullResults/

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20161124-075404/services/1/PullResults/

 

Luckily, on one occasion, the maximum response time was 101.759 and that was enough to complete the harvesting:

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20161123-042353/services/1/PullResults/

 

I hope the activity of switching to the National Discovery Service is right on track.

Do you have any news about that?

 

Best regards,

Angelo

#24 Updated by Angelo Quaglia over 3 years ago

From: Henrique Silva [mailto:hsilva@dgterritorio.pt]
Sent: 05 January 2017 12:22
To: angelo.quaglia@ext.jrc.ec.europa.eu
Subject: ECAS new login

 

Dear Angelo,
I'm going to update the filter, as agreed,
but now I see that are a new ECAS login.

I suppose I have to create a new acount, it is not?


Thanks and have a happy new year!

Best regards,

--

Henrique Silva

#25 Updated by Angelo Quaglia over 3 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 05 January 2017 14:39
To: 'Henrique Silva' <hsilva@dgterritorio.pt>
Subject: RE: ECAS new login

 

Dear Henrique,

 

That is great news!

 

I think you already have an ECAS login, which now has been renamed EU login but it is the same thing.

 

The only caveat is that for the MIG Collaboration Environment you need to login using hsilva@dgterritorio.pt as user name instead of the old ECAS user name while the password is as before.

 

That should work also for the Discovery Service Register.

 

If you really need to create a new ECAS account, you can do that at:

https://ecas.ec.europa.eu/cas/eim/external/register.cgi

 

 

 

Best regards,

Angelo

#26 Updated by Angelo Quaglia over 3 years ago

From: Henrique Silva [mailto:hsilva@dgterritorio.pt]
Sent: 05 January 2017 14:46
To: Angelo Quaglia <angelo.quaglia@ext.jrc.ec.europa.eu>
Subject: Re: ECAS new login

 

Dear Angelo,

That email is not working for the Discovery Service Register.

Error message: User not found

Could you check that?

Thanks a lot,
Henrique
 

#27 Updated by Angelo Quaglia over 3 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 05 January 2017 15:00
To: 'Henrique Silva' <hsilva@dgterritorio.pt>
Subject: RE: ECAS new login

 

Dear Henrique,

It does not work because your registered email there is:

hsilva@igeo.pt

 

Can you use that account?

 

Angelo

 

#28 Updated by Angelo Quaglia over 3 years ago

From: Henrique Silva [mailto:hsilva@dgterritorio.pt]
Sent: 05 January 2017 15:12
To: Angelo Quaglia <angelo.quaglia@ext.jrc.ec.europa.eu>
Subject: Re: ECAS new login

 

Ok, it's done!

With the small, nice filter :)

Thanks a lot and best regards,
Henrique

#29 Updated by Angelo Quaglia over 3 years ago

Dear Henrique,

the updated filter works really fine, many thanks!

Result of the interaction with the Discovery Service
Resources available for discovery: 1675, Expected Resource Count: 1675, Actual Resource Count : 1675
 
PERFORMANCE  
Initial response (seconds) 0.743
Transmission time (seconds) 45.347
Bytes transferred 40,675,790
Average MBytes/sec 0.855

 

The report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-d60bf7f3-ea96-11e4-a2c7-52540004b857_20170105-151942/services/1/PullResults/

 

Best regards,

Angelo

 

#30 Updated by Angelo Quaglia over 3 years ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF