Support #3678

SE: Handling multiple on-line links

Added by Michael Östling 10 months ago. Updated about 1 month ago.

Status:ClosedStart date:10 Sep 2019
Priority:HighDue date:
Assignee:Michael Östling% Done:

0%

Category:Linkage Checker
Target version:-
Submitting Organisation:Lantmäteriet Knowledge-Base relevant?:No
Proactive:No Keyword #1:
Country:SE - Sweden Keyword #2:
Originating UI: Keyword #3:

Description

We have  hadr We have in our national profile a recommendations to add a online-links to webpages with additional information about a resource.
So in our metadata for a network-service we normally have a link to the network-service itself as well as a link to a webpage with information.

While validating a service-metadata record we don't fully understand if that additional online link is allowed.

We check eg the following three records in linkage-checker


Dataset
https://www.geodata.se/geodataportalen/srv/en/csw-inspire?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&id=a2acb721-a900-42e7-9386-90f712836e64&outputSchema=csw:IsoRecord


View service
https://www.geodata.se/geodataportalen/srv/en/csw-inspire?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&id=a99ee0ef-9e9c-4bab-85ba-df5e86ce7605&outputSchema=csw:IsoRecord

 

Download service
https://www.geodata.se/geodataportalen/srv/en/csw-inspire?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&id=353facbc-3f7d-4273-8ca6-b1a7f3265556&outputSchema=csw:IsoRecord

 

When checking these all seems fine except for the download-service
Here we get the following error

The necessary resource locator (URL) of the Data Set from the Download service could not be determined.


But when inspecting the metadata it has multiple online resource locators (where one link points to ATOM-feed)
While validating this metadata record in the ETF it is reported as Passed, with comments.

In the ETF validation the rules are explained. Here it seems that it IS allowed to have multiple links.

This test checks each resource locator URL to see if it is syntactically correct and if the resource it references can be accessed, in order to determine its type. If the referenced resource is recognized as a Network Service, it checks whether the linkage to the dataset is declared and implemented.

This test case only applies to records with a hierarchyLevel value 'service'.

The test checks if a linkage is provided. If none is given, the test will complete successfully. If one or more are provided, for each linkage the test checks:

BUt the validator only inspects the first URL. Here it shown an error for the link to the webpage.

The metadata record set has 1 record(s) with errors for this assertion.
XML document 'csw-inspire.xml', record '353facbc-3f7d-4273-8ca6-b1a7f3265556': The metadata record has a linkage property, which references a resource with media type 'text/html; charset=utf-8' at 'https://www.lantmateriet.se/sv/Kartor-och-geografisk-information/geodatatjanster/hydrografi-nedladdning/' which is not one of the media types indicating an INSPIRE network service. Please review the resource to check that it meets the linkage requirements; specifically, if the resource is a webpage, check that it provides further instructions or a client application that directly accesses the service.

 

The download service have a link to a ATOM FEED.
This ATOM-feed has a link to service-metadata record and also at each ENTRY a link to the dataset-metadata.
But the subfeeds (dataset-feeds) do not have links to dataset-metadata.
Will the linkage checker also check metadata down to this level ?

 



Kind Regards
Michael Östling

 

 


 

 

History

#1 Updated by Angelo Quaglia 10 months ago

Dear Michael,

it is really much simpler than that, but the description used in the Linkage Checker does not help.

Simply put, the INSPIRE Geoportal is unable to download the data:

 

#2 Updated by Michael Östling 10 months ago

But I think this should be accepted anyhow.
This is a pay-service so data will never be accesible without a license.

We have understood that for WFS it is enough that the Capabibilities can be accessed and for a ATOM-feed that the ATOM-feeds are accesible
But acces to data is not needed for metadata to be accepted.

But the strange thing is that this have been a approved Priotized data in earlier harvesting session. But now after they added a HTML-online link it is no longer accepted  

 

 

An other strange thing:

We have previosly had ca 11 download services and 30 view services.
But in our last run in Harvesting console we only 1 accepted download service and 10 view services. And we believe very little have been done on these metadata.
Have there been any updates in the harvesting console last 1-2 months?
If so are there any documenation on that ?   

 

Kind Regards
Michael

 

#3 Updated by Angelo Quaglia 10 months ago

Dear Michael,

I agree with you.

The problem is that I keep receving conflicting requirements: one is to only present to the user download links that actually work, i.e. that actually return some meaningful content.

I have already explained many times that in INSPIRE, download links are allowed to be access-protected , but then, the INSPIRE Geoportal cannot possibly check that the links actually work.

As a result, the DOWNLOAD_MATCHING_DATA_IS_AVAILABLE is correctly awarded but the DATA_DOWNLOAD_LINK_IS_AVAILABLE is not.

 

Angelo

#4 Updated by Michael Östling 10 months ago

  • Assignee set to Angelo Quaglia

Dear Angelo,

I got your point;

I guess in future it should be a checkbox in Geoportal like "Show only Open data" that only has working links to data.

But in our case we are (in this moment) mainly trying to fulfil the directive and not end users needs.

So with current rules in Harvester these protected services will not be accepted ?

That is a change since they where approved in last harvesting session in june

I will ask Inspire-team to lift this in MIGt to get some clarification.

An other question, how did you validate the service so you could see the old validator 2 results like you showed on screenshot above?

In Linkage checker we only see the simplified error reports and detailed reports can be only been seen in in Harvesting console for records harvested from our CSW.
But If I want to check just one random record I don't know what tool to use. I have of course the Reference valdator but that give different results than old validator2
or in the resourcebrowser so i still see a need for a tool that gives the messages you showed.

Regards
Michael 
 



 

#5 Updated by Angelo Quaglia 10 months ago

Dear Michael,

I believe that I could add an aspect that says: DATA_DOWNLOAD_LINK_VERIFIED

So, to that to your services the DATA_DOWNLOAD_LINK_IS_AVAILABLE could still be awarded but not the DATA_DOWNLOAD_LINK_VERIFIED one.

In my initial idea, the DATA_DOWNLOAD_LINK_IS_AVAILABLE aspect meant simply that the INSPIRE Geoportal could find/form a link.

 

One additional problem about protected services, is that there has not yet been an agreement in INSPIRE about the ways to protect the services so the Geoportal can only guess.

One other possibility, would be to give us credentials that would allow the INSPIRE Geoportal to download even just a sample of the data.

 

From the Linkage Checker, you can still access the Validator 2 reports, clicking on "View the Evaluation Report", as depicted in the picture below.

I instisted to have that link added because it gives essential insight about the results.

Angelo

#6 Updated by Michael Östling 10 months ago

Hi Angelo,

I did not get any response on the question below. I see also Lars and Ine from Denmark/Netherlands have problems with harvesting results as we have.
We have have trying to find issues in our own catalogue but it seems to maybe be a general issue in the harvesting module in the INSPIRE Geoportal

 

An other strange thing:

We have previosly had ca 11 download services and 30 view services.
But in our last run in Harvesting console we only 1 accepted download service and 10 view services. And we believe very little have been done on these metadata.
Have there been any updates in the harvesting console last 1-2 months?
If so are there any documenation on that ?   

 

/Michael

#7 Updated by Angelo Quaglia 10 months ago

Hi Michael.

I believe I answered you: the iNSPIRE Geoportal now checks that the download links are actually working.

Angelo

#8 Updated by Angelo Quaglia 10 months ago

Regarding the View Services you said:

An other strange thing:

We have previosly had ca 11 download services and 30 view services.
But in our last run in Harvesting console we only 1 accepted download service and 10 view services. And we believe very little have been done on these metadata.
Have there been any updates in the harvesting console last 1-2 months?
If so are there any documenation on that ?   

 

Could you please clarify where you got those numbers? In the sandbox, the INSPIRE Geoportal contacted 99 View Services:

#9 Updated by Angelo Quaglia 10 months ago

Hi Michael,

Since your service correctly advertises the need for authentication, it is easy for me to add a condition:

 

                            if (downloadResourceLocator.getBytesTransferred().compareTo(BigInteger.ZERO) == 0) {
                                if (mostSpecificException instanceof ResourceLocatorResourceUnauthorizedException) {
                                } else {
                                    downloadResourceLocator.setDownloadedContentDeemedMeaningful(false);
                                }
                            }

 

However, I still have to clarify this with Robert.

 

#10 Updated by Angelo Quaglia 10 months ago

Robert told me these issues will be discussed in Prague.

#11 Updated by Michael Östling 10 months ago

Hi Angelo.

There was also one additional issuse we neded to get clarified

We have for many of our downloadservices multiple online-links

Eg for this record

https://www.geodata.se/geodataportalen/srv/eng/csw-inspire?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&outputSchema=csw:IsoRecord&id=c9940808-9276-4bfb-a717-bf6d2cbb7325

We have a link both to direct download point where a user can download a shapefile and then in addition to that there is a proper link to a WFS service.
It seems the refrerence validator do not accept that

And in Linkage checker we get same error that the download link could not be established

But  the WFS do actually work When we test to call it returns data.

https://opendata-view.smhi.se/SMHI_vatten/smhi_kustvatten_och_havsomraden_SVAR_2012_2/wfs?service=WFS&request=GetFeature&typenames=SMHI_vatten:smhi_kustvatten_och_havsomraden_SVAR_2012_2

 

When reading the TG we understand it that multiple online-links can exist and one of the links must point
to a valid download services

We also wonder if maybe the order of links matter. That the first link must must be a valid download service

Regards
Michael

#12 Updated by Angelo Quaglia 10 months ago

Dear Michael,

the INSPIRE Geoportal is happy about finding more than one link in a service metadata.

The only consequence is that it might discover more than one WFS service.

The INSPIRE Geoportal is capable of probing an OGCS endpoint replacing the existing query parameters with those needed for the GetCapabilities request.

In your case it finds two as explained this in Issue #3684 .

From I can on the reports, neither WFS makes available a suitable Stored Query implementing the requires Get Spatial Data Set operation.

Best regards,

Angelo

 

 

#13 Updated by Daniele Francioli 2 months ago

  • Subject changed from Handling multiple on-line links to SE: Handling multiple on-line links
  • Status changed from New to Feedback
  • Assignee changed from Angelo Quaglia to Michael Östling

Dear Michael,

We are going through the issues that are still not addressed.

We would linke to understand if this issue is still valid for you. Can we close it?

Thank you in advance,

Daniele on behalf of the JRC INSPIRE Support team

#14 Updated by Michael Östling about 1 month ago

  • Status changed from Feedback to Closed

Also available in: Atom PDF