Support #3819

MT: Harvesting Failure

Added by Rene Agius 8 months ago. Updated 3 months ago.

Status:FeedbackStart date:05 Feb 2020
Priority:NormalDue date:
Assignee:Davide Artasensi% Done:

0%

Category:Harvesting Console
Target version:-
Submitting Organisation:MITA Knowledge-Base relevant?:No
Proactive:No Keyword #1:
Country:MT - Malta Keyword #2:
Originating UI: Keyword #3:

Description

Dear Daniele,

I am trying to start a new harvest but for some reason it is being scheduled but it seems it is failing while it is running and no changes have been made since the last successfull harvest. I am attaching a screenshot for your referral. Your assistance is much appreciated.

 

Regards,

Rene

Harvest05-02-2020.PNG (31.2 KB) Rene Agius, 05 Feb 2020 04:02 pm

Harvesting.zip - Latest Harvesting Results (712 KB) Rene Agius, 16 Apr 2020 01:02 pm

2697

History

#1 Updated by Rene Agius 6 months ago

Dear Daniele,

I hope you are all doing well due to what is going around in Europe at the moment, if upon returning to normality there are any updates regarding this issue we'e appreciate them as it is still persisting.

Regards,

Rene

 

#2 Updated by Davide Artasensi 6 months ago

  • Status changed from New to Assigned
  • Assignee changed from Daniele Francioli to Davide Artasensi

Dear Rene,

due to the current COVID-19 emergency here in Italy, our team is forced to telework, that unfortunately implies an additional hindrance on our activities.

I would like to thank you in advance for your patience.

 

I'm currently updating the core backend system. As soon as I completed, your harvest session will be scheduled to start.

Then, we will monitor the results and we will keep you posted.

 

Davide on behalf of JRC INSPIRE Support team

#3 Updated by Davide Artasensi 6 months ago

Dear Rene,

 

the harvest session was completed successfully. Nevertheless, I noticed a significant drop in the downloadable count, so please let me investigate if everything is fine.

I will let you know as soon as possible.

 

Davide on behalf of the JRC INSPIRE Support team

#4 Updated by Rene Agius 6 months ago

Thank you for the update Davide, much appreciated even here in Malta we are striving with teleworking due to COVID19

#5 Updated by Rene Agius 6 months ago

Hi! Davide,

I have a small question, In the harvesting portal sometimes there are a number of queued and running jobs does this affect the harvesting process? as we are still facing the same problem that the harvesting process when started from our end is failing for some reason.

Thanks in advance for your help

Regards,

Rene

#6 Updated by Davide Artasensi 5 months ago

  • Subject changed from Harvesting Failure to MT: Harvesting Failure
  • Status changed from Assigned to Feedback

Dear Rene,

regarding your question about unexpected behaviors between queued and running processes, you are right and there could always be the possibility of a bug or hitting some resource limits that could affect the harvest.

Nonetheless, at this moment we are not encountering this kind of problem.

 

In your case, last week I tried to run several tentative of harvest and I found there was an error on the results from your discovery service.

We were not able to successfully complete the harvest due to the discrepancy on the number of resources, between the declared/expected and the actually retrieved count.

In particular, after the batch 181-200, we were not be able anymore to retrieve any metadata.

Last successfull result from the discovery service:

<?xml version="1.0" encoding="UTF-8"?>
<csw:GetRecordsResponse xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd">
  <csw:SearchStatus timestamp="2020-04-10T17:15:59" />
  <csw:SearchResults numberOfRecordsMatched="327" numberOfRecordsReturned="20" elementSet="full" nextRecord="201">

    <gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:geonet="http://www.fao.org/geonetwork" xsi:schemaLocation="http://www.isotc211.org/2005/gmd http://schemas.opengis.net/iso/19139/20070417/gmd/gmd.xsd http://www.isotc211.org/2005/gmx http://schemas.opengis.net/iso/19139/20070417/gmx/gmx.xsd">
      <gmd:fileIdentifier>
        <gco:CharacterString>7eae6311-09be-49f6-bfa1-9962ed97b176</gco:CharacterString>
      </gmd:fileIdentifier>

After this, any subsequent request returns simply an empty <csw:SearchResults/> element.

<?xml version="1.0" encoding="UTF-8"?>
<csw:GetRecordsResponse xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd">
  <csw:SearchStatus timestamp="2020-04-10T17:24:34" />
  <csw:SearchResults numberOfRecordsMatched="327" numberOfRecordsReturned="20" elementSet="full" nextRecord="221">

  </csw:SearchResults>
</csw:GetRecordsResponse>

 

As now, we completed successfully a tentative of harvest (328/328) and it seems fixed.

On our side, we had in place a fixed threshold that automatically discard that harvest result as error/corrupted, so we have now disabled it for this kind of mismatch.

 

Best regards,

Davide on behalf of JRC INSPIRE Support

#7 Updated by Rene Agius 5 months ago

Thank you for your reply Daniele, much appreciated.

 

Regards,

Rene

#8 Updated by Rene Agius 5 months ago

Dear Daniele,

The harvesting process now is successfully completed. One thing I am noticing is that with every harvest the amount of data being caught by the harvest is different each time. I attached a zip file containing some screenshots portraying this issue.

 

Thank you for you help and Regards,

Rene

#9 Updated by Rene Agius 4 months ago

Dear Davide,

 

I trust this message finds you well, I tried to perform another harvest and noticed that it is starting to run but it is failing again. Is there a possibility that it is related to the latest changes done on the harvesting process? As even the harvest performed from your side and mentioned in the email seems to have failed.

 

Regards,

Rene

#10 Updated by Davide Artasensi 4 months ago

Dear Rene,

Thank you for your message. First, about the changes on the setting we discussed during our last meeting, I didn't apply yet on the sandbox environment (the one used by HarvestConsole) so everything is "behaving" as usual. Regarding the harvest failures, I run some additional tests on your services, but without significant improvements, so I will start doing another batch of test soon, and I will keep you posted so we could address the issue together.

KR,

Davide

 

#11 Updated by Rene Agius 3 months ago

Dear Davide,

 

Thank You for your help.

 

Regards,

Rene

Also available in: Atom PDF