MT: Harvesting Failure
|Status:||Feedback||Start date:||05 Feb 2020|
|Assignee:||Davide Artasensi||% Done:|
|Submitting Organisation:||MITA||Knowledge-Base relevant?:||No|
|Country:||MT - Malta||Keyword #2:|
|Originating UI:||Keyword #3:|
I am trying to start a new harvest but for some reason it is being scheduled but it seems it is failing while it is running and no changes have been made since the last successfull harvest. I am attaching a screenshot for your referral. Your assistance is much appreciated.
#2 Updated by Davide Artasensi 6 months ago
- Status changed from New to Assigned
- Assignee changed from Daniele Francioli to Davide Artasensi
due to the current COVID-19 emergency here in Italy, our team is forced to telework, that unfortunately implies an additional hindrance on our activities.
I would like to thank you in advance for your patience.
I'm currently updating the core backend system. As soon as I completed, your harvest session will be scheduled to start.
Then, we will monitor the results and we will keep you posted.
Davide on behalf of JRC INSPIRE Support team
#3 Updated by Davide Artasensi 6 months ago
the harvest session was completed successfully. Nevertheless, I noticed a significant drop in the downloadable count, so please let me investigate if everything is fine.
I will let you know as soon as possible.
Davide on behalf of the JRC INSPIRE Support team
#5 Updated by Rene Agius 6 months ago
I have a small question, In the harvesting portal sometimes there are a number of queued and running jobs does this affect the harvesting process? as we are still facing the same problem that the harvesting process when started from our end is failing for some reason.
Thanks in advance for your help
#6 Updated by Davide Artasensi 5 months ago
- Subject changed from Harvesting Failure to MT: Harvesting Failure
- Status changed from Assigned to Feedback
regarding your question about unexpected behaviors between queued and running processes, you are right and there could always be the possibility of a bug or hitting some resource limits that could affect the harvest.
Nonetheless, at this moment we are not encountering this kind of problem.
In your case, last week I tried to run several tentative of harvest and I found there was an error on the results from your discovery service.
We were not able to successfully complete the harvest due to the discrepancy on the number of resources, between the declared/expected and the actually retrieved count.
In particular, after the batch 181-200, we were not be able anymore to retrieve any metadata.
Last successfull result from the discovery service:
After this, any subsequent request returns simply an empty <csw:SearchResults/> element.
As now, we completed successfully a tentative of harvest (328/328) and it seems fixed.
On our side, we had in place a fixed threshold that automatically discard that harvest result as error/corrupted, so we have now disabled it for this kind of mismatch.
Davide on behalf of JRC INSPIRE Support
#8 Updated by Rene Agius 5 months ago
- File Harvesting.zip added
The harvesting process now is successfully completed. One thing I am noticing is that with every harvest the amount of data being caught by the harvest is different each time. I attached a zip file containing some screenshots portraying this issue.
Thank you for you help and Regards,
#9 Updated by Rene Agius 4 months ago
I trust this message finds you well, I tried to perform another harvest and noticed that it is starting to run but it is failing again. Is there a possibility that it is related to the latest changes done on the harvesting process? As even the harvest performed from your side and mentioned in the email seems to have failed.
#10 Updated by Davide Artasensi 4 months ago
Thank you for your message. First, about the changes on the setting we discussed during our last meeting, I didn't apply yet on the sandbox environment (the one used by HarvestConsole) so everything is "behaving" as usual. Regarding the harvest failures, I run some additional tests on your services, but without significant improvements, so I will start doing another batch of test soon, and I will keep you posted so we could address the issue together.