HTTP Conditional GET
A conditional GET is an HTTP GET request that returns an HTTP 304 response (instead of HTTP 200). An HTTP 304 response indicates that the resource has not been modified since the previous GET, and so the resource is not returned to the client in such a response.
There are at least two (not completely independent) approaches to conditional GET:
Last-Modified
/If-Modified-Since
ETag
/If-None-Match
In both cases, the value of a response header is used as the value of a subsequent request header. For example, note the Last-Modified
and ETag
headers in the response to this HEAD request for InCommon metadata:
$ curl --silent --head \ http://wayf.incommonfederation.org/InCommon/InCommon-metadata.xml HTTP/1.1 200 OK Date: Mon, 22 Aug 2011 00:18:37 GMT Accept-Ranges: bytes ETag: W/"3752903-1313780996000" Last-Modified: Fri, 19 Aug 2011 19:09:56 GMT Content-Type: application/xml Content-Length: 3752903 Connection: close
If we take the value of the Last-Modified
header from the previous response as the value of the If-Modified-Since
header in the following request, we receive a 304 response (and no content) from the server:
$ curl --silent --head \ --header 'If-Modified-Since: Fri, 19 Aug 2011 19:09:56 GMT' \ http://wayf.incommonfederation.org/InCommon/InCommon-metadata.xml HTTP/1.1 304 Not Modified Date: Mon, 22 Aug 2011 00:35:48 GMT Connection: close ETag: W/"3752903-1313780996000"
Similarly, if we take the value of the ETag
header from the previous response as the value of the If-None-Match
header in the following request, we again receive a 304 response:
$ curl --silent --head \ --header 'If-None-Match: W/"3752903-1313780996000"' \ http://wayf.incommonfederation.org/InCommon/InCommon-metadata.xml HTTP/1.1 304 Not Modified Date: Mon, 22 Aug 2011 00:37:04 GMT Connection: close ETag: W/"3752903-1313780996000"
The use of conditional GET has significant benefits, on both the client and the server (and the intervening network as well). On the wayf.incommonfederation.org server, roughly 3/4 of all metadata requests result in HTTP 304. That translates into more than 15K metadata requests per day that conveniently avoid the unnecessary overhead of metadata refresh. For a file whose size is more than 4MB and growing, that represents a significant cost savings.
Conditional GET has security benefits as well. Since requests that result in HTTP 304 are issued virtually without penalty, a client can request metadata more frequently than absolutely necessary. In the case of InCommon metadata, which is produced daily, hourly requests will result in just one HTTP 200 response in a typical 24-hour period. If, however, InCommon Operations signs metadata more than once per day (which happens on occasion), or more importantly, a key in metadata is compromised, necessitating an immediate production run, the fact that clients are attempting to refresh metadata hourly has significant potential benefit.
Tools and Tips
The command-line examples above suggest a tool based on curl is possible. A tool that implements HTTP conditional GET is attached to this wiki page. It's a bash script that caches the HTTP response header along with the resource. In this way, subsequent requests can provide the appropriate request headers. If the server supports conditional GET, and the resource has not changed since the previous GET (as indicated by HTTP 304), the script accesses the resource from cache. Here’s how you might use the script to intelligently fetch InCommon metadata:
$ echo $url http://wayf.incommonfederation.org/InCommon/InCommon-metadata.xml $ ./smart_get.sh -q $url 4070346 $ echo $? 0 $ ./smart_get.sh -q $url 0 $ echo $? 0
Option -q
tells the script to run quietly, in which case it returns the integer number of bytes returned from the server. This can be used in a simple cron script (included with the attached tool) such as the following:
url=http://wayf.incommonfederation.org/InCommon/InCommon-metadata.xml bytes=`./smart_get.sh -q $url` if [ $bytes -eq 0 ]; then # do nothing echo "no metadata to process" else # process metadata echo "$bytes bytes of metadata to process" fi
Later versions of Shibboleth (at least IdP 2.2 and SP 2.4) implement HTTP conditional GET (and more) so the above script is not particularly useful unless you're running something else. For instance, simpleSAMLphp does everything except HTTP conditional GET, so users of simpleSAMLphp can easily modify the above cron script by calling SSP's builtin metarefresh module if the number of bytes returned is nonzero.