Support #2836

FR - BRGM: Some batches fail

Added by Angelo Quaglia almost 4 years ago. Updated almost 3 years ago.

Status:FeedbackStart date:
Priority:ImmediateDue date:
Assignee:Angelo Quaglia% Done:

0%

Category:Harvesting process
Target version:-
Submitting Organisation:FR - BRGM Knowledge-Base relevant?:
Proactive:Yes Keyword #1:
Country:FR - France Keyword #2:
Originating UI: Keyword #3:

Description

Dear Zoe, 250 records could not be retrieved during the last harvesting due to a new problem:

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@568b4887[exception={net.opengis.ows.v_1_0_0.ExceptionType@7d8905b1[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412193]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]

The report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20160829-010128/services/1/PullResults/

Could you please investigate this issue?

fr.mp4 (2.09 MB) Angelo Quaglia, 24 Nov 2016 11:17 am

fr2.mp4 (2.13 MB) Angelo Quaglia, 29 Nov 2016 10:06 am


Related issues

Related to Geoportal Helpdesk - Support #2405: FR - PNEIG: Several requests coming from the INSPIRE Geop... Closed 07 Apr 2015
Copied from Geoportal Helpdesk - Support #2805: FR - BRGM: Invalid date format in 745 metadata documents Feedback 17 Mar 2016

History

#1 Updated by Angelo Quaglia almost 4 years ago

  • Status changed from New to Assigned

#2 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

the error occurred again last night for 5 batches (corresponding to 250 metadata records that could not be retrieved):

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@8e2a102[exception={net.opengis.ows.v_1_0_0.ExceptionType@7de85b6[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412190]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]" 

It seems to indicate some index or data corruption.

THe full report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20160905-011154/services/1/PullResults/

The problematic batches are the following ones:

#3 Updated by Angelo Quaglia almost 4 years ago

  • Estimated time set to 0.25

#4 Updated by Angelo Quaglia almost 4 years ago

  • Estimated time deleted (0.25)

#5 Updated by Angelo Quaglia almost 4 years ago

  • Proactive set to Yes

#6 Updated by Angelo Quaglia almost 4 years ago

  • Priority changed from Normal to High

Dear Zoe,

the problem is still occurring on several batches:

On Monday:

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20160912-022655/services/1/PullResults/

 

The problem is easily reproducible:

 

Please give us some feedback on this.

#7 Updated by Angelo Quaglia almost 4 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 14 September 2016 15:41
To: Petty Zoe <Z.Petty@brgm.fr>
Cc: 'Marie Lambois' <Marie.Lambois@ign.fr>; 'Taffoureau Etienne' <e.taffoureau@brgm.fr>; Marc.Leobet@developpement-durable.gouv.fr
Subject: several batches fail
Importance: High

 

Dear Zoe,

Twelve days ago I reported the following issue:

Issue https://ies-svn.jrc.ec.europa.eu/issues/2836

 

Sever batches fail with the following error:

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@568b4887[exception={net.opengis.ows.v_1_0_0.ExceptionType@7d8905b1[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412193]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]

 

It is easy to reproduce with a browser:

 

I kindly ask you to update the issue.

 

Best regards,

Angelo

#8 Updated by Zoè Petty almost 4 years ago

Hello Angelo,

Sorry for the delay, I'm on it but i haven't yet found a solution, we have been investigating since and have several options to check.

I hope to resolve this for next week and will give you feedback immediatly.

Best regards,

zoe

#9 Updated by Angelo Quaglia almost 4 years ago

  • Status changed from Assigned to Feedback

Hi Zoe,

many thanks for the prompt update.

We would like to have a fresh harvesting in time for the INSPIRE Conference.

Best regards,

Angelo

 

#10 Updated by Angelo Quaglia almost 4 years ago

  • Priority changed from High to Urgent

Dear Zoe,

the problem is getting worse and affects more and more batches:

 

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20160914-180007/services/1/PullResults/

Result of the interaction with the Discovery Service

Resources available for discovery33814Expected Resource Count33814Actual Resource Count : 19914

 

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20160915-180014/services/1/PullResults/

Result of the interaction with the Discovery Service

Resources available for discovery33954Expected Resource Count33954Actual Resource Count : 20740

 
Now there is a mix of the following errors:

 

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@474d0c54[exception={net.opengis.ows.v_1_0_0.ExceptionType@37906ef6[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#471294]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]

It seems some sort of  index corruption. Can't you rebuild them?

 

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "HTTP connection failed at url: "http://www.geocatalogue.fr/api-public/inspire/servicesRest" because Internal Server Error"

Please check your server logs.

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "HTTP connection failed at url: "http://www.geocatalogue.fr/api-public/inspire/servicesRest" because Service Temporarily Unavailable"

Please check your server logs.

 

 

Result of the interaction with the Discovery Service

Resources available for discovery33954Expected Resource Count33954Actual Resource Count : 20740

#11 Updated by Angelo Quaglia almost 4 years ago

I have disabled the harvesting for France, waiting for your feedback.

#12 Updated by Zoè Petty almost 4 years ago

Dear Angelo,

Our team managed to solve this issue, could you please confirm?

Sorry once again for the delay.

Best regards

zoe

#13 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

many thanks for your update.

I will try now on a test instance.

 

Best regards,

Angelo

#14 Updated by Angelo Quaglia almost 4 years ago

The harvesting of the frist 5,000 was successful.

I will now trigger a full harvesting.

#15 Updated by Zoè Petty almost 4 years ago

Dear angelo,

good news, thank you for the feedback.

Best regards

zoe

#16 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

8 batches still failed with the same error:

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@1291437d[exception={net.opengis.ows.v_1_0_0.ExceptionType@742f1695[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412190]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]" 

The batches are:

The full report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20160920-132617/services/1/PullResults/

Please tell me when a complete and stable fix is in place so that I can rerun a harvesting.

Best regards,

Angelo

#17 Updated by Zoè Petty almost 4 years ago

Dear Angelo,

Through these reports i cannot see which Metadata exactly are linked to these failures. Is there a way to see a list of the incriminated metadata of these batches.

Thank you in advance

zoe

#18 Updated by Angelo Quaglia almost 4 years ago

Even just one defective record causes a failure of all whole batch.

Once I stumble upon a failed batch, I query one by one.

I can do it for you.

 

#19 Updated by Zoè Petty almost 4 years ago

That would be great if you could!

Thank you

zoe

#20 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

I have noticed that since yesterday the records have increased in number because now I get numberOfRecordsMatched=34092

While in the report I read: Resources available for discovery: 33989

So, the defective records, if they are still there, have surely shifted.

I can kick a new harvesting now, if you agree?

#21 Updated by Zoè Petty almost 4 years ago

Please do so

zoe

#22 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

the new harvesting has just completed. ( http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20160921-113652/services/1/PullResults/ )

The INSPIRE Geoportal could not retrieve the following batches:

7001-7050          startPosition="7037"
8101-8150          startPosition="8139"
15301-15350      startPosition="15337"
22051-22100      startPosition="22079"
27051-27100      startPosition="27076"
30151-30200      startPosition="30191"
32201-32250      startPosition="32202"
33651-33700      startPosition="33666"

 

The GetRecordRequest are like this one, just replace the value of startPosition:

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2" maxRecords="1" startPosition="7037" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">
  <csw:Query typeNames="gmd:MD_Metadata">
    <csw:ElementSetName>full</csw:ElementSetName>
    <ogc:SortBy>
      <ogc:SortProperty>
        <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>
        <ogc:SortOrder>ASC</ogc:SortOrder>
      </ogc:SortProperty>
    </ogc:SortBy>
  </csw:Query>
</csw:GetRecords>

The result you get will be:

 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="1.0.0" language="en" xmlns="http://www.w3.org/2001/SMIL20/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:ogc="http://www.opengis.net/ogc" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#411348]</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

Of course, you let time pass, your service will likely add/remove records, reindex and the values of startPost will not match.

#23 Updated by Angelo Quaglia almost 4 years ago

  • Country set to FR - France

#24 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

please tell me if you manager to fix the problems so that I can run the harvesting again.

Best regards,

Angelo

#25 Updated by Zoè Petty almost 4 years ago

Hello Angelo,

First of all i was very happy to finally meet you last week, it's nice to put a face behind the messages!

Secondly, could you please reharvest FR and tell me exactly at what time you did it so we could have a very precise look into the logs, eventually at what time does it stop, any detail is welcome.

Thank you in advance

Best regards,

zoe

#26 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

yes, it was indeed great to meet face-to-face, at last.

Sure, I would expect it to run tonight and I will let you know the results.

Best regards,

Angelo

 

#27 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

This issue is about the fact that France's  National Discovery Service is suffering from what appears to be a recurrent index or record corruption.

It does not happen only at a specific time or during a full harvesting.

For example, it is happening now:

However, if you let too much time pass, your service will likely add/remove records, reindex, and the values of startPos will not match the example:

 

 

You just need to send the following request to http://www.geocatalogue.fr/api-public/inspire/servicesRest

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2" maxRecords="1" startPosition="33209" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">
  <csw:Query typeNames="gmd:MD_Metadata">
    <csw:ElementSetName>full</csw:ElementSetName>
    <ogc:SortBy>
      <ogc:SortProperty>
        <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>
        <ogc:SortOrder>ASC</ogc:SortOrder>
      </ogc:SortProperty>
    </ogc:SortBy>
  </csw:Query>
</csw:GetRecords>

 

You get back the following answer:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="1.0.0" language="en" xmlns="http://www.w3.org/2001/SMIL20/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ns11="http://www.opengis.net/gml/3.2" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:ogc="http://www.opengis.net/ogc" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412191]</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

 

 

 

#28 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe, the test case above is still working.

#29 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe, the test case above is still working.

Please tell me when you plan to be able to work on this issue because we need some coordination between ourselves.

As it is, the National Discovery Service of France is definitely not INSPIRE compliant.

Best regards,

Angelo

#30 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe, the test case above is still working.

#31 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

after last night the records have shifted, so you can reproduce the error NOW with this request:

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2" maxRecords="1" startPosition="6929" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">
  <csw:Query typeNames="gmd:MD_Metadata">
    <csw:ElementSetName>full</csw:ElementSetName>
    <ogc:SortBy>
      <ogc:SortProperty>
        <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>
        <ogc:SortOrder>ASC</ogc:SortOrder>
      </ogc:SortProperty>
    </ogc:SortBy>
  </csw:Query>
</csw:GetRecords>

 

and you get

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="1.0.0" language="en" xmlns="http://www.w3.org/2001/SMIL20/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:ogc="http://www.opengis.net/ogc" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#411348]</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

 

I would appreciate if you could ask someone to look at this issue shortly while the test case is still accurate.

#32 Updated by Angelo Quaglia almost 4 years ago

From: Angelo Quaglia [mailto:angelo.quaglia@ext.jrc.ec.europa.eu]
Sent: 05 October 2016 12:19
To: Petty Zoe <Z.Petty@brgm.fr>
Cc: 'Taffoureau Etienne' <e.taffoureau@brgm.fr>; 'Marie Lambois' <Marie.Lambois@ign.fr>; Marc.Leobet@developpement-durable.gouv.fr; 'Michael Lutz' (michael.lutz@jrc.ec.europa.eu) <michael.lutz@jrc.ec.europa.eu>
Subject: failing batches
Importance: High

 

Dear Zoe,

The documented malfunctionings of the French National Discovery Service are quite serious.

 

This issue is the most urgent and blocking as it does not allow the INSPIRE Geoportal to perform a full harvest:

https://ies-svn.jrc.ec.europa.eu/issues/2836

 

This will become blocking after the next imminent INSPIRE Geoportal update:

https://ies-svn.jrc.ec.europa.eu/issues/2851  

 

Both issues are very easy to reproduce at any time but since your Discovery Service seems to perform reindexing, the actual startPosition values of the test case shift every day so please have a look at it ASAP.

 

 

 

Best regards,

Angelo

 

#33 Updated by Angelo Quaglia almost 4 years ago

From: Petty Zoe [mailto:Z.Petty@brgm.fr]
Sent: 05 October 2016 16:46
To: Angelo Quaglia <angelo.quaglia@ext.jrc.ec.europa.eu>
Cc: Taffoureau Etienne <e.taffoureau@brgm.fr>; 'Marie Lambois' <Marie.Lambois@ign.fr>; Marc.Leobet@developpement-durable.gouv.fr; 'Michael Lutz' <michael.lutz@jrc.ec.europa.eu>
Subject: RE: failing batches

 

Dear Angelo,

We are recovering the logs of the indexation to see where the problem lies, we haven’t changed our config.  We are now looking into every metadata with an error.

Best regards

#34 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

that's great.

Note that this request is still failing now, it might be worth checking the logs while the request is being executed:

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2" maxRecords="1" startPosition="6929" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">
  <csw:Query typeNames="gmd:MD_Metadata">
    <csw:ElementSetName>full</csw:ElementSetName>
    <ogc:SortBy>
      <ogc:SortProperty>
        <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>
        <ogc:SortOrder>ASC</ogc:SortOrder>
      </ogc:SortProperty>
    </ogc:SortBy>
  </csw:Query>
</csw:GetRecords>

#35 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

last night the harvesting ran again, here is the report:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161006-004130/services/1/PullResults/

The error happened for the following batches:

 

 

 

 

 

 

 

 

So, once more, if you use the following request, with startPosition = 6929, 8018, 21764, 26700. 29768, 31749, 33194:

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2" maxRecords="1" startPosition="6929" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">
  <csw:Query typeNames="gmd:MD_Metadata">
    <csw:ElementSetName>full</csw:ElementSetName>
    <ogc:SortBy>
      <ogc:SortProperty>
        <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>
        <ogc:SortOrder>ASC</ogc:SortOrder>
      </ogc:SortProperty>
    </ogc:SortBy>
  </csw:Query>
</csw:GetRecords>

You get back the error:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="1.0.0" language="en" xmlns="http://www.w3.org/2001/SMIL20/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:ogc="http://www.opengis.net/ogc" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412191]</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

Could you please collect the information on your system ASAP with these testcases? It is time consuming for me to make you available the numbers every day.

Best regards,

Angelo

 

#36 Updated by Angelo Quaglia almost 4 years ago

  • Priority changed from Urgent to Immediate

Dear Zoe,

I am very surprised that I have received no update at all from you about this issue since the 5th October.

I am very happy to help you but I see no real effort on your side.

The French National Discovery Service has not been able to consistently satisfy the requirements of the INSPIRE Network Service Regulation, in terms of functionality and capacity, for quite some time now.

The INSPIRE Geoportal tried twice to harvest from it yesterday with very bad, if not disastrous results.

There is always this functionality issue that affects at least 6 batches: 

1) The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@1dc6c836[exception={net.opengis.ows.v_1_0_0.ExceptionType@2e8d41e7[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412193]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

In addiiton, there are capacity issues:

2) The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "HTTP connection failed at url: "http://www.geocatalogue.fr/api-public/inspire/servicesRest" because Service Temporarily Unavailable"

3)The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "HTTP connection failed at url: "http://www.geocatalogue.fr/api-public/inspire/servicesRest" because Internal Server Error" 

 

The harvesting reports are available here:

 

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161013-180016/services/1/PullResults/

Result of the interaction with the Discovery Service

Resources available for discovery34372Expected Resource Count34372Actual Resource Count : 15850

 

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161014-012013/services/1/PullResults/

Result of the interaction with the Discovery Service

Resources available for discovery34372Expected Resource Count34372Actual Resource Count : 32022

 

The batches for which issue 1 has been detected are: 

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@1dc6c836[exception={net.opengis.ows.v_1_0_0.ExceptionType@2e8d41e7[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412193]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

 Best regards,

Angelo

#37 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

the latest harvesting report shows that the error occured for 8 batches:

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@5da96741[exception={net.opengis.ows.v_1_0_0.ExceptionType@2412d42e[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412190]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

 

 

 

 

 

 

 

 

It is clearly a corruption is spreading across your system.

Kind regards,

Angelo

 

P.S.: I have just tried to send a GetRecords request to your Discovery Service and I got an error:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
    <head>
        <title>503 Service Temporarily Unavailable</title>
    </head>
    <body>
        <h1>Service Temporarily Unavailable</h1>
        <p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
    </body>
</html>

#38 Updated by Zoè Petty almost 4 years ago

Angelo,

Our servers are down, hence the "503 Service Temporarily Unavailable" our IT team is on it. Concerning the MD we are changing our DB, i really hope it will solve the harvesting problem we have encountered.

Kind regards

zoe

#39 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

many thanks for the update.

Angelo

#40 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

again in last night harvesting 8 batches (i.e. 400 records) could not be retrieved because of the usual error message: 

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@e8bbdbf[exception={net.opengis.ows.v_1_0_0.ExceptionType@35eeb52a[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#411348]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

The batches in error were the following ones:

When do you expect to have a fix for the problem? 

 

The full harvesting report is available here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161024-013202/services/1/PullResults/

 

Best regards,

Angelo

#41 Updated by Zoè Petty almost 4 years ago

Dear Angelo,

At noon we have launched our new search/csw engine, could you please harvest again and send us the feedback to see if everything has been well set.

Best regards,

zoe

#42 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

many thanks for the update.

I have just kicked it off.

Best regards,

Angelo

#43 Updated by Angelo Quaglia almost 4 years ago

  • Status changed from Feedback to Assigned

#44 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

You said:

"At noon we have launched our new search/csw engine, could you please harvest again and send us the feedback to see if everything has been well set."

I assume the endpoint is the same?

 

The results were even worse, if possible:

Resources available for discovery: 70799, Expected Resource Count: 70799, Actual Resource Count : 43700

The full harvesting report is here:

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161024-185248/services/1/PullResults/

 

The failed batches were:

#45 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

I think that a more timely feedback from you would simplify everybody's work.

Best regards,

Angelo

 

#46 Updated by Zoè Petty almost 4 years ago

Dear Angelo,

Sorry for not keeping you updated but as you have said it isn't good, we have been analyzing the issue and it figures out new developments were necessary. They will be put into action this friday around noon.

This is a major technical change for the plateform and i do hope you understand the situation, we are doing our best to clear out this situation.

Best regards,

zoe

 

#47 Updated by Angelo Quaglia almost 4 years ago

  • Status changed from Assigned to Feedback

Dear Zoe,

many thanks for keeping me up-to-date.

I will therefore suspend the harvesting of France for the moment.

I would greatly appeciate if you could keep me posted about your progress.

Best regards,

Angelo

 

#48 Updated by Zoè Petty almost 4 years ago

Dear Angelo,

Could you please try the harvest for the Geocatalogue, i believe we have identified the probleme and made the correct modifications.

Thank you in advance for your feedback

Best regards

zoe

#49 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

you are very welcome.

That's great, many thanks.

it is running. I will keep you posted.

Best regards,

Angelo

 

 

#50 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

unfortunately, the modifications you implemented seem to have worsened the problem.

I have also noticed that there has been a big increase in the number of metadata records .

Result of the interaction with the Discovery Service

Resources available for discovery70921Expected Resource Count70921Actual Resource Count : 67771

The problem is aways the same and now affects 63 batches:

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@1f2a65b4[exception={net.opengis.ows.v_1_0_0.ExceptionType@19e34392[exceptionText={Error fr.brgm.geocat.service.search.ErrorUUIDNotFound}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

The harvesting report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161031-014033/services/1/PullResults/

Here are the faulty batches:

 

 

 

By the way, The Netherlands asked to launch a survey about which implementations are in use for INSPIRE Discovery Services.

That is being documented in Issue #2869.

Could you please tell me what is your implementation?

 

Best regards,

Angelo

 

#51 Updated by Zoè Petty almost 4 years ago

Dear Angelo,

Thank you for the feed back, i am very suprised because on our side we get a total of 32564MD in the complete CSW as you can see here :

http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetRecords&service=CSW&version=2.0.2&TYPENAMES=gmd:MD_Metadata

given this we need to investigate further because the differences are unclear to me, if you have any advice you can call me 00.33 2 38 64 34 61.

May i ask, just to be sure, at what time was the harvesting done exactly?

Thank you in advance,

Best regards,

Zoe

#52 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

it is indeed extremely weird and I am sorry to say that the situation starts looking a bit out of control on your side:

Here is the GetRecordsResponse of batch 1-50 exactly as it was received from your service.

You can determine the exact time on your server when that response was received and the number of records matched:

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161031-014033/services/1/PullResults/1-50/downloaded

<csw:SearchStatus timestamp="2016-10-31T01:46:51.668+01:00"/>
<csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="70921" numberOfRecordsReturned="50" nextRecord="51">

 

 

 

#53 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

could you please tell me what your CSW implementation is (e.g. GeoNetwork) and the version?

You can only diagnose this issue on your side, looking at the logs.

For sure, it looks like a very serious problem.

I am not sure it helps to keep harvesting from it until you have a clear understanding of what is happening on the server.

Best regards,

Angelo

 

#54 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

since I have not received any updates from you, I have reverted the INSPIRE Geoportal to a previous harvesting for France dated 24th October 2016.

The report of the abnormal harvesting is still available at this URL:

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161031-014033/services/1/PullResults/

 

I am looking forward to receiving assurance that the following errors are being investigated and addressed:

{Error fr.brgm.geocat.service.search.ErrorUUIDNotFound} 

- duplication of metadata records (

<csw:SearchStatus timestamp="2016-10-31T01:46:51.668+01:00"/>

<csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="70921" numberOfRecordsReturned="50" nextRecord="51">

)

 

I would like to know

-when I can reharvest

-what is your current CSW implementation and version

 

Best regards,

Angelo

#55 Updated by Angelo Quaglia almost 4 years ago

Dear Zoe,

given the very low quality of service provided by the National Discovery Service of France, I cannot hide my disappointment at what I can only perceive as a lack of interest, effort and responsiveness on your part.

The new problem about the doubling of metadata records are again caused by a bug on your side.

If you request the capabilities from the endpoint registered on the INSPIRE Geoportal, i.e.:

http://www.geocatalogue.fr/api-public/inspire/servicesRest?version=2.0.2&service=CSW&REQUEST=GetCapabilities

you will see the HTTP bindings are wrong and point to the non-INSPIRE service:

<ows:OperationsMetadata>
<ows:Operation name="GetRecords">
<ows:DCP>
<ows:HTTP>
<ows:Constraint name="PostEncoding">
<ows:Value>SOAP</ows:Value>
</ows:Constraint>
</ows:Post>
</ows:HTTP>
</ows:DCP>
<ows:DCP>
<ows:HTTP>
<ows:Constraint name="PostEncoding">
<ows:Value>XML</ows:Value>
</ows:Constraint>
</ows:Post>
</ows:HTTP>
</ows:DCP>
 
If you query against the endpoint http://www.geocatalogue.fr/api-public/servicesRest you get back 70921 records:
 
 
If you query against what I can infer the INSPIRE endpoint is, i.e., http://www.geocatalogue.fr/api-public/inspire/servicesRest you get back 32564 records:
 
 

I kindly urge you to fix the capabilities as soon as possible.

I also kindly ask you to give me a timely feedback on this and the other outstanding issue that are preventing the INSPIRE Geoportal form performing a clean harvesting of the French National Discovery Service.

Kind regards,

Angelo

#56 Updated by Thierry Vilmus almost 4 years ago

Dear Angelo,

Thank you for your feed-back. Of course we have already seen the getCapabilities problem, pointing to wrong links.

We have worked on our side to fix some redirections but we've seen that it's not sufficient and that some work from our sub-contractor was needed.

This work has been done now, we implement it and test it today, hoping it will solve the problem. Of course we give you some feed back.

We apologize for the lack of feed back recently, but we were working hard to fix the problem, have found some solutions which were note fully efficient, so there was no good news to tell you and no relevant answer to give you on feed back.

We assure you there is no lack of interest or effort !

Best regards,

Thierry Vilmus (I'll be the new Geocatalogue project manager in  a few days, replacing Zoé who is moving for a new job).

+33 2 38 64 30 57

 

#57 Updated by Angelo Quaglia almost 4 years ago

Dear Thierry,

many thanks for your swift and to-the-point reply.

Congratulations on your new position.

The reason why I ask you to keep me up-to-date is that I need to know in a timely manner whether the harvesting job for France needs to be suspended or whether a harvesting result shall be discarded or accepted to be published on the INSPIRE Geoportal.

 

Following a request from another Member State, I am collecting a list of CSW 2.0.2 software implementations running National INSPIRE Discovery Services.

The list is maintained at Issue #2869.

I seem to remember France was running a custom implementation based on GeoServer, but I would appreciate if you could share some more details about it.

 

Best regards,

Angelo

(Thanks for sharing your office number. Mine is +39 347 78 88 492. You can also Skype me at angelo.quaglia )

 

#58 Updated by Thierry Vilmus almost 4 years ago

Dear Angelo,

 

We deploy a new version of the Geocatalogue (v 4.0.1) to correct the harvesting bug. We need to test this new version till monday afternoon at least, so please do not harvest before that.

I keep you updated when harvesting can be done, hopefully monday in the afternoon.

Sorry for the delay...

 

Yes we are running a custom implementation, I'll give you more technical details soon in the issue #2869.

Thanks for your office number !

 

Best regards,

Thierry

 

 

#59 Updated by Angelo Quaglia almost 4 years ago

Dear Thierry,

many thanks for the update.

I did not manage to arrive back to the office in time to stop the harvesting it but it will run to the end and then it will not try again.

That's great to hear. If you would like me to run some preliminary tests just tell me.

Best regards,

Angelo

 

 

#60 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Today we had some infrastructure problems that prevent us to test correctly the new version of the Geocatalogue.

A new batch of tests is scheduled tomorrow at 14:00.

I'll keep you informed.

All my apologies for this new delay, but I think that all will be Ok tomorrow evening.

Best regards,

Thierry

 

#61 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

many thanks for keeping me up-to-date on this.

Best regards,

Angelo

 

#62 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Finally it's working fine in our test environnement !

We change our production environment tomorrow morning, we test it and you should be able to harvest normally from tomorrow afternoon.

I give you the GO as soon as you can harvest again from our side.

I'm sorry to say 'tomorrow' every day, but this time we have solid hopes, as it works in the test environment...

Best regards,

Thierry

 

#63 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

that's great news.

That is not a problem, it happens to me all the time.

Take the time you need.

Best regards,

Angelo

 

 

 

 

#64 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Thank you very much for understanding !

This is another update without an end of the problem...

We have infrastructure contraints with our servers that prevent us to install the tested solution into our production environment this week.

We know we are on the right path to solve the harvesting problem since we have succesfully tested the new version in the test environment.

It's difficult to give you another period when harvesting should work properly ; all I can tell you is that we have scheduled to run the new version in the production environment on Monday 14th, when the infrastructure team has solved the server problem.

I keep you informed as soon as possible and hope this time all will be Ok at the beginning of next week.

Best regards,

Thierry

 

#65 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

understood.

I will wait for your final confirmation.

Best regards,

Angelo

#66 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

can I proceed with the harvesting?

 

Best regards,

Angelo

#67 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Please not yet...

The CSW API returns now the correct XML file but there are still problems with the new indexer engine (solr) that we are investigating now.

I keep you updated.

Best regards,

Thierry

 

#68 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

 

Could you please proceed with a harvesting as a test ?

I'm afraid it will have remaining errors as the indexer engine is still not automatically synchronizing with the database but we'll see if there is some improvement or not...

Thanks in advance,

Kind regards,

Thierry

 

#69 Updated by Thierry Vilmus over 3 years ago

I'm sorry Angelo, in fact we must wait for a few minutes as the solr index is re-building...

#70 Updated by Angelo Quaglia over 3 years ago

Indeed,

the service declares 0 records:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<csw:GetRecordsResponse xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <csw:SearchStatus timestamp="2016-11-15T11:03:02.972+01:00"/>
    <csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="0" numberOfRecordsReturned="0" nextRecord="0"/>
</csw:GetRecordsResponse>

 

#71 Updated by Angelo Quaglia over 3 years ago

No problem. Just tell me when it is ready.

#72 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

You may proceed now, you should see 32623 records.

Best regards,

Thierry

 

#73 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

I actually see 32600 records:

    <csw:SearchStatus timestamp="2016-11-15T11:33:50.638+01:00"/>

    <csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="32600" numberOfRecordsReturned="1" nextRecord="2">

Shall I proceed?

Best regards,

Angelo

#74 Updated by Angelo Quaglia over 3 years ago

I have started it anyway.

#75 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Yes, 32600 records is Ok.

#76 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

the harvesting went well except for one batch which still failed with the known error.

The failed batch was 9051-9100 (Page modified15 Nov 2016, 21:10:41 GMT)

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@171596e[exception={net.opengis.ows.v_1_0_0.ExceptionType@611a820b[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#471293]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

Now the records have reshuffled so the batch at the same startPosition would work.
If you need more help for nailing down the error, please ask.
 
Because of the error, 50 records were not received by the INSPIRE Geoportal:
 
Result of the interaction with the Discovery Service

Resources available for discovery32600Expected Resource Count32600Actual Resource Count : 32550

Performances were good and the highest initial response time was below 5 seconds, which is the limit set by the Network Service Regulation:

PERFORMANCE  
Initial response (seconds) 4.479
Transmission time (seconds) 592.613
Bytes transferred 716,302,814
Average MBytes/sec 1.153

 

The full report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161115-152417/services/1/PullResults/

 

Best regards,

Angelo

 

#77 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

 

Thank you for your detailed feedback.

When you query our CSW, you receive answers both from the search engine index (powered by Solr) and from our database.

An automatic update of the search engine index is run every hour, so it's possible to have a few errors if someone has deleted records in the database some minutes ago and the index is not yet updated.

However, it seems that the index sometimes doesn't update as scheduled, we must investigate on this side.

 

Do you harvest at the same time every day ? If yes, please give me the time and I'll manage to run the index update a few minutes before.

 

Best regards,

Thierry

 

#78 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

The current harvesting frequency of your Discovery Service was set to weekly by your predecessor Etienne.

Depending on the system load, the time of harvesting might vary.

However, I can set the harvesting to run before or after a specific time or only on specific days.

The harvesting lasts several hours (a bit more than 8 hours, currently) because, while harvesting, it validates the Network Services and the resources they serve.

It is also an indication that the 99% availability required by the Network Services Regulation is there.

In addition, the GetRecords requests are spread across a long period of time so that they should affect less the other users of your service.

If we could agree on specific times or days, that would be great.

Best regards,

Angelo

 

#79 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Thanks for your flexibility...

I think the weekly frequency for harvesting is the best compromise between load and accuracy. We should keep this frequency.

Given the time needed by the harvesting process, a small amount of errors is likely to occur.

Maybe the best time to harvest is every Monday evening at say 6:00 pm ; it allows us to check indexes and correct any problems on mondays.

Best regards,

Thierry

 

 

#80 Updated by Angelo Quaglia over 3 years ago

#82 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

I am still able to reproduce the same intermittent problem explained above.

If I had to make a wild guess, I would say it is related to the load balancer, if you have one.

It seems that about half of the times the request targets an instance where some records are not found, hence the error message 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="1.0.0" language="en" xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#412190]</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

 

 

#83 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Many thanks for the video which is self explanatory.

You guess right, we have a load balancer switching between 2 instances.

I see what we can do about that.

Best regards,

Thierry

#84 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

1) now, after about three or four requests I get:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="1.0.0" language="en" xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>Error fr.brgm.geocat.service.search.ErrorUUIDNotFound</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

 

2) Several batches failed last night withthe same error message:

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@1e830dff[exception={net.opengis.ows.v_1_0_0.ExceptionType@6946d37d[exceptionText={Error fr.brgm.geocat.service.search.ErrorUUIDNotFound}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]" 

So, is the problem with the load balancer?

 

My impression is that the root cause of the issues with France's National Discovery Service has not really been identified, so the problems come back after a few days.

 

Best regards,

Angelo

 

 

 

#85 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

I don't  have the feedback from the infrastructure team yet regarding the load balancer.

For uuid errors it should be the index from the Solr search engine which is inconsistent with the database because it still needs a manual update. We work to run it automatically every hour.

The index ran last friday morning. I have run it again a few minutes ago, it should be ok in #5-10 minutes.

I'm away this week on meetings but I keep you informed.

Best regards

Thierry

#86 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

We have identified a problem with the load balancing chain : some "zookeeper" nodes have too small timeouts. We have to find how to change that and hopefully this afternoon it will be Ok.

Best regards

Thierry

#87 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

many thanks for the update.

I do not know if this helps but I am still getting the intermittent ErrorUUIDNotFound message even now that the index should be aligned:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="1.0.0" language="en" xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>Error fr.brgm.geocat.service.search.ErrorUUIDNotFound</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

Best regards,

Angelo

#88 Updated by Thierry Vilmus over 3 years ago

Thank you Angelo,

In fact the updating process of the index has failed, because of the zookeeper problem.

I keep you informed when it is solved.

Best regards,

Thierry

 

#89 Updated by Angelo Quaglia over 3 years ago

DearThierry,

I am not getting the ErrorUUIDNotFound any more but I have noticed that, depending presumably on which node serves the request, I am getting different results or at least a different sort order for the same request:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<csw:GetRecordsResponse xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <csw:SearchStatus timestamp="2016-11-28T16:18:29.905+01:00"/>
    <csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="33058" numberOfRecordsReturned="50" nextRecord="8201">
        <gmd:MD_Metadata>
            <gmd:fileIdentifier>
                <gco:CharacterString>d653df21-2b93-40d0-804c-c7f1c51a3da0</gco:CharacterString>
            </gmd:fileIdentifier>

 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<csw:GetRecordsResponse xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <csw:SearchStatus timestamp="2016-11-28T16:18:57.034+01:00"/>
    <csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="33058" numberOfRecordsReturned="50" nextRecord="8201">
        <gmd:MD_Metadata>
            <gmd:fileIdentifier>
                <gco:CharacterString>fr-120066022-jdd-03421c2e-d51c-478b-b03b-8d1d9187cdf6</gco:CharacterString>
            </gmd:fileIdentifier>

 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<csw:GetRecordsResponse xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <csw:SearchStatus timestamp="2016-11-28T16:19:12.575+01:00"/>
    <csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="33058" numberOfRecordsReturned="50" nextRecord="8201">
        <gmd:MD_Metadata>
            <gmd:fileIdentifier>
                <gco:CharacterString>d653df21-2b93-40d0-804c-c7f1c51a3da0</gco:CharacterString>
            </gmd:fileIdentifier>
            <gmd:language>
                <gco:CharacterString>fre</gco:CharacterString>
            </gmd:language>

 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<csw:GetRecordsResponse xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
    <csw:SearchStatus timestamp="2016-11-28T16:19:23.002+01:00"/>
    <csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="33058" numberOfRecordsReturned="50" nextRecord="8201">
        <gmd:MD_Metadata>
            <gmd:fileIdentifier>
                <gco:CharacterString>fr-120066022-jdd-03421c2e-d51c-478b-b03b-8d1d9187cdf6</gco:CharacterString>
            </gmd:fileIdentifier>

 

 

The request is always the same and contains a sorting clause:

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2" maxRecords="50" startPosition="8151" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">
  <csw:Query typeNames="gmd:MD_Metadata">
    <csw:ElementSetName>full</csw:ElementSetName>
    <ogc:SortBy>
      <ogc:SortProperty>
        <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>
        <ogc:SortOrder>ASC</ogc:SortOrder>
      </ogc:SortProperty>
    </ogc:SortBy>
  </csw:Query>
</csw:GetRecords>

 

This is a problem because it makes the results of metadata collection unpredictable.

 

 

#90 Updated by Angelo Quaglia over 3 years ago

I have disabled the harvesting until this new issue is clarified.

#91 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

I have uploaded a new video that ilustrates the unexpected and repeatable behaviour.

For the same request(*), which contains a SortBy clause, I get two different answers depending, I presume, on the node which is serving the request.

One node responds repeatedly with this record as the first one:

<gmd:fileIdentifier>
                <gco:CharacterString>d653df21-2b93-40d0-804c-c7f1c51a3da0</gco:CharacterString>
</gmd:fileIdentifier>

 

The other node responds repeatedly with this record as the first one:

<gmd:fileIdentifier>
                <gco:CharacterString>fr-120066022-jdd-03421c2e-d51c-478b-b03b-8d1d9187cdf6</gco:CharacterString>
</gmd:fileIdentifier>

 

I do not find the first record in the second answer and viceversa.

The two nodes declare to have the same number of records indexed.

Another wild guess I could make is that the sortBy clause is not applied or the index is corrupt on at least one of the two nodes.

 

In order to shed some more light, I changed the startPosition from 8151 to 1 in the request(*):

Sometimes I get the usual error:

<ows:ExceptionText>Error fr.brgm.geocat.service.search.ErrorUUIDNotFound</ows:ExceptionText>

some other times I get back a record: 

          <gmd:fileIdentifier>
                <gco:CharacterString>fr-120066022-jdd-4f89ae6e-1910-45b5-aa06-f5961c0b4de6</gco:CharacterString>
            </gmd:fileIdentifier>

 

This cannot be the first record if the SortBy clause apiso:Identifier, ASC is correctly applied.

So, I see at least two problems:

On one node the index is corrupt.

On the other node the SortBy clause is not applied correctly or the index is also corrupt.

 

Please fix this issue as soon as possible because the two nodes are giving inconsistent answers.

I suggest you disable the load balancing until you are able to guarantee that both nodes are giving reliable answers.

 

Best regards,

Angelo

 

 

 

(*)

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2" maxRecords="50" startPosition="8151" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">

  <csw:Query typeNames="gmd:MD_Metadata">
    <csw:ElementSetName>full</csw:ElementSetName>
    <ogc:SortBy>
      <ogc:SortProperty>
        <ogc:PropertyName>apiso:Identifier</ogc:PropertyName>
        <ogc:SortOrder>ASC</ogc:SortOrder>
      </ogc:SortProperty>
    </ogc:SortBy>
  </csw:Query>
</csw:GetRecords>

#92 Updated by Angelo Quaglia over 3 years ago

The first request ( startPosition="1") I sent today, I got back the usual error:

<ows:ExceptionText>Error fr.brgm.geocat.service.search.ErrorUUIDNotFound</ows:ExceptionText>

The second time I got back a record:

<gmd:fileIdentifier>
      <gco:CharacterString>fr-120066022-jdd-4f89ae6e-1910-45b5-aa06-f5961c0b4de6</gco:CharacterString>
</gmd:fileIdentifier>

 

 

#93 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

I'm back to the office. Thank you for your detailed feedback that will help very much to find the problems.

We try to solve what we can this afternoon and the remaining will be done on monday.

Index is now rebuilt. We don't see inconsistent data for the moment but tests are not complete.

Best regards,

Thierry

#94 Updated by Angelo Quaglia over 3 years ago

Dear Thierry, 

you are welcome.

Indeed, I do not seem to get back the Error fr.brgm.geocat.service.search.ErrorUUIDNotFound error anymore.

However, the discrepancy between the responses of the two nodes persists:

Node X:

            <gmd:fileIdentifier>
                <gco:CharacterString>ffd880a6-39f8-4651-acaa-fb622154c506</gco:CharacterString>
            </gmd:fileIdentifier>

Node Y:

            <gmd:fileIdentifier>

                <gco:CharacterString>FR-120066022-MDWFS-DDT_12-CARTELIE_POS_12096</gco:CharacterString>
            </gmd:fileIdentifier>

 

I was wondering about whether you could, as a temporary workaround, set up your load balancer so that all requests coming from the INSPIRE Geoportal (IP 139.191.1.236) are directed always to the same node. Even better if the node selected is the one that has the most up-to-date index.

 

Best regards,

Angelo

 

#95 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

We reproduce the discrepancy you mentioned. Unfortunately, the way the system is set up, we cannot direct a particular IP on a single node.

We have checked indexes synchronization and it's Ok so the problem is elsewhere.

We're working on it...

I keep you informed.

Best regards,

Thierry

 

#96 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Our sub-contractor has identified the problem, at the configuration level of our search engine, which is balanced between two nodes.

I come back to you when it is solved.

Best regards,

Thierry

#97 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

many thanks for the udpate.

Best regards,

Angelo

#98 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

any news from your sub-contractor?

Best regards,

Angelo

#99 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Unfortunately, the solution (sending an 'optimize' request on the 2 solr cores) does not solved the problem. Solr cores are synchronized and have the same data but they continue to sort queries in a different way.

This is really a very annoying problem, as harvesting with a sequence of GetRecords queries is not possible anymore.

I'm waiting for a new solution on tomorrow.

Best regards,

Thierry

 

#100 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Sorry for all these days without an answer.

Meanwhile we have tried many solutions with our sub-contactors, in vain, and came to the conclusion that using only one solr shard (with some replicas to ensure sustanaibility) is the only solution.

We have to assess that one shard is enough to support our database and indexes load and weight. We are carrying tests right now.

I keep you informed.

Best regards,

Thierry

 

#101 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

many thanks for your timely updates.

I am sorry to hear you are facing so much trouble and I will be waiting for news.

Best regards,

Angelo

#102 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

 

I'm very pleased to inform you that we have implemented a temporary solution which should allow you to harvest our geocatalogue...

As we expect less load for the end of the year, we have cut off the search engine replicas to use only one. So there is no more inconsistency in our results.

Of course this is temporary : at the beginning of the new year we'll go back to the replicas system. At that time, all queries will be internally sorted by Id to ensure that all search engine replicas retrieve results in the same order.

 

Thanks for your patience...

Warmest season's greetings,

Thierry

#103 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

many thanks for the excellent news.

I have therefore reenabled the harvesting for the French catalogue.

My best wishes to you, too!

 

Best regards,

Angelo

#104 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

the harvesting ran yesterday.

Result of the interaction with the Discovery Service

Resources available for discovery33977Expected Resource Count33977Actual Resource Count : 33377

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20161222-180211/services/1/PullResults/

The 12 batches below failed with the usual error:

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@2ec8f51c[exception={net.opengis.ows.v_1_0_0.ExceptionType@3039ede7[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#352439]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

Best regards,

Angelo

 

#105 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

 

Thank you for the feedback.

Today we have implemented a new indexation method : it's an incremental indexation (only changes since last indexation are processed) and not a full one anymore.

This leads to 2 improvements :

  - it's faster so can be launched every hour (instead of every day) ;

  - it keeps the index running while re-indexing, so there is no more maintenance downtime.

 

So next harvesting should lead to much fewer errors.

Best regards,

Thierry

 

 

#106 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

this looks very promising!

There isn't enough time today to start a new harvesting and see the end of it.

I propose to leave it as it is now and wait for next week harvesting.

The JRC will be closed next week.

I wish you a merry Christmas and a happy new year.

Best regards,

Angelo

 

#107 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

the problem is still there.

The custom implementation you are using is extremely fragile and prone to data corruption:

Even now if I query:

 maxRecords="1" startPosition="6044"

I get the usual error:

"net.opengis.ows.v_1_0_0.ExceptionReport@1ec04977[exception={net.opengis.ows.v_1_0_0.ExceptionType@42408954[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#471292]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

At this point, I am not sure your software provider has clearly identified the root cause of the problem, therefore it is unable to fix it.

 

 

The harvesting ran again today and several batches failed with that error:

The full report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20170103-103843/services/1/PullResults/

The batches that failed were:

Best regards,

Angelo 

#108 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Do you still run into errors ?

If you agree, we have to discuss together how to improve the harvesting: it may lead to change thinks on our end, but also the way you are querying.

Your queries are spread through a long time: this is good for not overloading but, as metadata are added and deleted quite frequently in the Geocatalogue, Ids you are using after several hours are very likely to be obsolete.

So maybe we can think of queries in one go, to retrieve all data at one time. Our system is supposed to support such a heavy load.

What do you think?

Best regards,

Thierry

 

PS. I'm going to the 4th meeting of MIG 2016.2 sub-workgroup on Monitoring and Reporting 2019, will you be there too ?

#109 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

Something prevented the harvesting from completing in the past few days.

It is running at the moment. I will keep you posted.

By the way, I have one issue #2902 about a service with almost 1000 copuled resources to most of which access is forbidden to the INSPIRE Geoportal. 

 

It is definitley possible to harvest all in one shot or even split the work in 4 or 5 big batches.

However, I have just tried to harvest 10000 records and I got back this answer:

net.opengis.ows.v_1_0_0.ExceptionReport@587611b[exception={net.opengis.ows.v_1_0_0.ExceptionType@6968bfb[exceptionText={parameter value must be <= 1000}, exceptionCode=InvalidParameterValue, locator=MaxRecord]}, version=1.0.0, language=en]

Can you remove that limitation?

 

Best regards,

Agelo

#110 Updated by Angelo Quaglia over 3 years ago

I will be here on those days and I will likely take part to at least part of the meeting.

#111 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

I will be glad to meet you and maybe we can find some time to talk about our harvesting problems.

I see what I can do for the 1000 records limitation.

Best regards,

Thierry

 

#112 Updated by Angelo Quaglia over 3 years ago

We will definitely find the opportunity to discuss that.

#113 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

it was very nice meeting you in person yesterday.

Let me summarize our exchange here:

1)  you told me that your system currently cannot return more than 1000 records and for best performance 500 would be preferable.

I expressed my surprise because your solution is based on Apache Solr like the INSPIRE Geoportal Operational Pilot, therefore I know that Apache Solr has no issue streaming even all of its content in a single request. You can try to request 10,000 records and it will simply require more time (which increases linearly with the numer of requested records).

So, there must at least one component in your system which is not implementing the streaming correctly.

You told me you would investigate.

 

2) You also told me that you are going to investigate the issue about the index corruption that is causing a whole batch to fail.

It really seems to me, that your contractor has not even tried to reproduce and understand the error.

It is really critical, even more if we switch to larger batches since it means to lose 500 records for every occurrence of the error.

 

3) You showed interest in a way to have your Discovery Service publish a list of changes in metadata so that a client can stay up-to-date consuming the only changes without requesting all the content each time.

There is a standardised way called "Resource Sync" that you can access at: http://www.openarchives.org/rs/1.0/resourcesync

From the web site:

"This ResourceSync specification describes a synchronization framework for the web consisting of various capabilities that allow third-party systems to remain synchronized with a server's evolving resources. The capabilities may be combined in a modular manner to meet local or community requirements. This specification also describes how a server should advertise the synchronization capabilities it supports and how third-party systems may discover this information. The specification repurposes the document formats defined by the Sitemap protocol and introduces extensions for them. "

I implemented that in the Pilot some time ago for another project, even though the feature is not enabled in the production Geoportal.

 

Best regards,

Angelo

#114 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

last night's harvesting went much better.

The ErrorUUIDNotFound error was confined to the last few batches:

 

 

 

 

 

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@6010a16e[exception={net.opengis.ows.v_1_0_0.ExceptionType@1fc03c64[exceptionText={Error fr.brgm.geocat.service.search.ErrorUUIDNotFound}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"

 

Could you please tell me the number of resources you expect the INSPIRE Geoportal to have harvested?

Result of the interaction with the Discovery Service

Resources available for discovery: 33884, Expected Resource Count: 33884, Actual Resource Count : 33634

Best regards,

Angelo
 

PERFORMANCE  
Initial response (seconds) 7.018
Transmission time (seconds) 883.774
Bytes transferred 746,862,016
Average MBytes/sec 0.806

#115 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

last night's harvesting was also good:

Result of the interaction with the Discovery Service
Resources available for discovery: 33897, Expected Resource Count: 33897, Actual Resource Count : 33647
 
PERFORMANCE  
Initial response (seconds) 13.512
Transmission time (seconds) 1,841.641
Bytes transferred 747,622,409
Average MBytes/sec 0.387

The critical error

The interaction with the remote service at "http://www.geocatalogue.fr/api-public/inspire/servicesRest?request=GetCapabilities&service=CSW&version=2.0.2" ended with the following error "net.opengis.ows.v_1_0_0.ExceptionReport@8a41f4c[exception={net.opengis.ows.v_1_0_0.ExceptionType@6c01b442[exceptionText={Error fr.brgm.geocat.service.search.ErrorUUIDNotFound}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]"  

occurred for the following batches:

 

Best regards,

Angelo

#116 Updated by Thierry Vilmus over 3 years ago

Dear Angelo,

Thank you for your kind feedback !

Can you please warn be the day before your next harvesting ?

I will manually delete record links we suspect are causing the UUIDnotFound problems.

Then, if harvesting goes without error, we'll implement an automatic solution, to finally allow you to harvest smoothly !

Best regards,

Thierry

 

#117 Updated by Angelo Quaglia over 3 years ago

Dear Thierry,

excellent, I will do that.

The next harvesting is expected to start one week from 03 May 2017, 06:10:16 GMT, that is after Wednesday 10 May 2017, 06:10:16 GMT.

So, the cleanup should be done next Tuesday. The harvester starts at 6pm local time but the job will be picked up after 04:10:16 local time.

It might happen that the queue is full and the harvesting is delayed to the next day after 6pm local time.

I will keep you posted.

Best regards,

Angelo

 

 

#118 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

the harvesting will likely run tonight, so if you would like to perform the cleanup, today is the day.

 

Thanks in advance and best regards,

Angelo

#119 Updated by Thierry Vilmus about 3 years ago

Thank you Angelo,

Cleaning has been done...

Let's see what happen tonight !

Best regards,

Thierry

 

#120 Updated by Thierry Vilmus about 3 years ago

Dear Angelo,

Data in Géocatalogue should be stable up to tomorrow the 11th at 1 am, when the next harvesting will take place.

Do you think your harvesting will be finished at that time ?

Kind regards,

Thierry

 

#121 Updated by Angelo Quaglia about 3 years ago

Many thanks Thierry.

Last time it started at 8pm local time and finished at 10am:

Page created03 May 2017, 06:10:16 GMT     Page modified04 May 2017, 08:03:41 GMT

 

#122 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

I am sorry but the geoportal was quite busy last night and did not manage to start the harvesting for France.

 

Best regards,

Angelo

 

 

#123 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

the harvesting ran last night but a few batches were returned with no record inside:

This is the response for startPosition = 24301

-<csw:GetRecordsResponse xmlns="http://www.w3.org/2001/SMIL20/" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:gml="http://www.opengis.net/gml" xmlns:gts="http://www.isotc211.org/2005/gts" xmlns:ows="http://www.opengis.net/ows" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:dct="http://purl.org/dc/terms/" xmlns:ns13="http://www.opengis.net/gml/3.2" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ns15="http://www.w3.org/2001/SMIL20/Language" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gmd="http://www.isotc211.org/2005/gmd">
<csw:SearchStatus timestamp="2017-05-12T04:08:49.093+02:00" />
<csw:SearchResults recordSchema="http://www.isotc211.org/2005/gmd" numberOfRecordsMatched="0" numberOfRecordsReturned="0" nextRecord="0" />
 
</csw:GetRecordsResponse>

Best regards,

Angelo


#124 Updated by Thierry Vilmus about 3 years ago

Dear Angelo,

We already started this discussion when we meet in Ispra, our current system is not fully compatible with the time period required for your harvesting.

Everyday at 4am (GMT+2), we run our daily indexing, which takes about 15 minutes to complete. During this period (4:00-#4:15am), our system is unreachable...

So at 04:08:49 you encountered an error due to this indexing process.

Between 1am and 3am, we harvest nearly all the distant catalogues which are feeding the Géocatalogue. So time between 1am and 4:15am is not advised to harvest our catalogue.

As we already told about, one solution could be to reduce the time required for your harvesting, I think our infrastructure is able to support a heavy load. So we could test that.

A second solution, maybe better, is to keep your current harvesting procedure, but to run it at 4:30am. At that time, you will not meet re-indexing problem, and the database should be stable.

Is it possible to configure your harvest system to start at 4:30am instead of 8pm ?

 

On our side, we need to have a better indexing method (not a daily full one as now but an hourly incremental one which will not made the system unreachable).

Best regards,

Thierry

 

#125 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

yes, I remember that we talked about that but I was not sure about the exact timing of your procedures.

The new harvesting procedure will first harvest the metadata and then process the Network Services and that should alleviate the problem.

For the time being I will configure the harvesting to start after 4.30AM (6.30am GMT) which sounds quite good to me.

Many thanks and best regards,

Angelo

 

 

#126 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

last night the harvesting was one of the best ever.

Result of the interaction with the Discovery Service

Resources available for discovery: 34299, Expected Resource Count: 34299, Actual Resource Count : 34249

The geoportal managed to kick it off at 28 May 2017, 03:44:05 GMT:

 Page created28 May 2017, 03:44:05 GMT     Page modified28 May 2017, 15:24:17 GMT

 

The error {Error fr.brgm.geocat.service.search.ErrorUUIDNotFound} occurred only in the very last batch:

The report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20170528-054348/services/1/PullResults/

 

Best regards,

Angelo

#127 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

my understanding was that you would be able to have this longstanding issue fixed.

What is the situation?

Best regards,

Angelo

#128 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

last night the harvesting ran and the well-known error:

net.opengis.ows.v_1_0_0.ExceptionReport@1ae80a32[exception={net.opengis.ows.v_1_0_0.ExceptionType@77539a30[exceptionText={No row with the given identifier exists: [fr.brgm.geocat.model.AbstractMetadata#536495]}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]

occurred for the following four batches:

1151-1200 18701-18750 31001-31050 33801-33850

As a result, 200 metadata docuemnts could not be retrieved:

Resources available for discovery: 33969, Expected Resource Count: 33969, Actual Resource Count : 33769

 
The performances were good and well within the minimum required by Network Service Regulation:
 
 
PERFORMANCE  
Initial response (seconds) 3.894
Transmission time (seconds) 640.102
Bytes transferred 745,472,988
Average MBytes/sec 1.111
 

 

All the details are available at :

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20170621-045203/services/1/PullResults/

 

Best regards,

Angelo

#129 Updated by Thierry Vilmus about 3 years ago

Dear Angelo,

Thanks for your feed-back.

Yes the problem is well known now... We have seen that it is beyond database links and requires changes in our Java code.

Best regards,

Thierry

#130 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

in the harvesting that started early this morning three batches were affected by the usual error:

13001-13050
26001-26050
34101-34150

 

The full report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20170628-050039/services/1/PullResults/

 

Best regards,

Angelo

#131 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

in the latest harvesting ( Page created05 Jul 2017, 04:00:16 GMT     Page modified05 Jul 2017, 17:10:48 GMT) four batches were affected by the usual error:

The full report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20170705-060006/services/1/PullResults/

Performance are remarkable. The initial response time shown in the first page of the report is the HIGHEST among all batches of 50 records:

PERFORMANCE  
Initial response (seconds) 2.096
Transmission time (seconds) 954.098
Bytes transferred 756,277,725
Average MBytes/sec 0.756

Best regards,

Angelo

 

 

#132 Updated by Angelo Quaglia about 3 years ago

Dear Thierry,

in the latest harvesting (14 Jul 2017, 03:02:37 GMT     Page modified14 Jul 2017, 16:58:54 GMT) seven batches were affected by the usual error:

 

The full report is here:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20170714-050220/services/1/PullResults/

Performance were good. The initial response time shown in the first page of the report is the HIGHEST among all batches of 50 records:

PERFORMANCE  
Initial response (seconds) 5.766
Transmission time (seconds) 1,209.038
Bytes transferred 763,376,160
Average MBytes/sec 0.602
 

Best regards,

Angelo

#133 Updated by Angelo Quaglia almost 3 years ago

Dear Thierry,

without your manual cleanup the known problem keeps getting worse:

net.opengis.ows.v_1_0_0.ExceptionReport@5cdd0262[exception={net.opengis.ows.v_1_0_0.ExceptionType@24093b46[exceptionText={Error fr.brgm.geocat.service.search.ErrorUUIDNotFound}, exceptionCode=NoApplicableCode, locator=<null>]}, version=1.0.0, language=en]

It now affects 18 batches (listed below and in the harvesting report at http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-5145fa60-0067-11e5-9ea6-52540004b857_20170808-043224/services/1/PullResults/)

As a result, several records could not be retrieved but I must promote this harvesting to production in order to have fresh metadata,

Result of the interaction with the Discovery Service

Resources available for discovery: 34715, Expected Resource Count: 34715, Actual Resource Count : 33965

 

Best regards,

Angelo

 

#134 Updated by Angelo Quaglia almost 3 years ago

Dear Thierry,

The service at http://catalogue.geo-ide.developpement-durable.gouv.fr/catalogue/srv/fre/csw-moissonnable-ds-8

is again not serving requests  (See  Issue #2405)

Could you please help?

 

Best regards,

Angelo

 

Also available in: Atom PDF