Contribute to this Wiki Page!
Protect Against Failed Metadata Processes
All Shibboleth IdP deployers in the InCommon Federation are strongly advised to take the following precautions to protect themselves against possible failed metadata processes:
Allocate at least 1500MB of heap space in the JVM
Enable DEBUG-level logging on the following Java classes:
See the sections below for details.
Heap Allocation in the JVM
As of February 18, 2016, the Shibboleth Project recommends at least 1500MB of heap space for deployments that rely on metadata files larger than 25MB. This recommendation applies to both Shibboleth IdP V3 and Shibboleth IdP V2:
The InCommon metadata aggregate is large and growing!
Actually, we have reason to believe that metadata refresh failures are occurring with metadata files smaller than 25MB. Over the last few months, Shibboleth IdP deployers have reported numerous failed metadata processes. In the system log, it appears as though a refresh is attempted but no new metadata is found so the IdP continues to rely on the original metadata. This eventually results in expired metadata and failed user logins. The reasons for these failures are not entirely understood since a known logging issue is preventing the root cause from being logged (see the next section for a workaround to this issue). Strong evidence suggests that insufficient memory is the cause of these failures.
Allocate sufficient heap space in the JVM
For some Shibboleth IdP deployments, even that may not be enough. The following advice was overheard on the Shibboleth mailing list:
Use a 64-bit OS, hand it 3G or more, and stop wasting valuable person time trying to save money on RAM.
Please take this issue very seriously!
Working Around a Known Logging Issue
Since a metadata refresh process thread reportedly fails without logging an exception, the Shibboleth Project team investigated the issue:
Shib IdP V2: https://issues.shibboleth.net/jira/browse/JOST-243 (closed)
Shib IdP V3: https://issues.shibboleth.net/jira/browse/OSJ-125 (closed)
Apparently the JVM is running out of memory but the error message is not being logged. There is a workaround, however. The failure can be observed in logs when DEBUG mode logging is enabled on the relevant Java class.
Work around a known logging issue
To work around this issue in Shibboleth IdP V2, add the following DEBUG logger to /opt/shibboleth-idp/conf/logging.xml:
To work around this issue in Shibboleth IdP V3, add the following DEBUG logger to /opt/shibboleth-idp/conf/logback.xml:
Note: The logging issue will be fixed in Shibboleth IdP V2.4.5 and Shibboleth IdP V3.2.0, respectively.
All simpleSAMLphp deployers in the InCommon Federation are strongly encouraged to upgrade to simpleSAMLphp v1.13.2 (or later). Previous versions of simpleSAMLphp are known to have performance issues when processing large metadata files. See the simpleSAMLphp mailing list archives for details regarding this performance issue.