This is a revised Federation Testing Environment Working Group Charter, updated in September 2021

Background

The InCommon community has been asking for an easier, more tangible way to validate whether services will interoperate seamlessly when integrated into the Federation. In particular, a federation test environment has long been a frequently requested feature.

In 2018, InCommon Operations began drafting requirements for a “test federation”. Since that time, the Kantara SAML2 Deployment Profile interoperability standard (a.k.a., the Deployment Profile) emerged and InCommon transitioned to Baseline Expectations. These new developments will inform the design of federation integration testing tools aimed at easing deployment as well as producing a sustainable operating environment.

The Federation Testing Environment Working Group coincides with the InCommon Technical Advisory Committee’s (TAC’s) work to prepare InCommon to adopt the Deployment Profile. The working group will support this by working through the specifics of how to allow InCommon, deployers, and implementers measure against the testable statements in the Deployment Profile. In order to clearly measure an entity’s success in meeting the requirements of the statements, there needs to be a set of reference / compliance testing tools on which the community can rely. 

Charter

The InCommon Technical Advisory Committee (TAC) convenes the InCommon Federation Test Environment Working Group to turn the Deployment Profile requirements into specific, implementable testing criteria and prioritize tests of specific functions while developing any needed compliance details. The result will allow InCommon to build testing tools to be used to measure an entity's compliance in cases where:

• an existing or prospective InCommon Participant integrates services (Identity Provider and Service Provider) via the InCommon Federation
• the InCommon federation operator introduces changes to Federation infrastructure

The Federation Test Environment Working Group should:

1. Develop the testable Deployment Profile requirements into specific, implementable testing criteria
2. Consider conformance validation opportunities for Deployment Profile statements in terms of
   1. Timing
      1. Testable statements are a concern when either a deployment is registered in InCommon or software is implemented
   2. Purpose
      1. Compliance enforcement
      2. Federation interoperability testing
      3. Software conformance testing
3. Develop any needed compliance testing details so that
   1. InCommon can build testing tools
   2. Participants / prospective participants / InCommon can use them to measure an entity's compliance with the testable statements in the Deployment Profile
4. Consider how testing and / or validation tooling for the statements reduces barriers to entering InCommon and increases the overall value of participation in the Federation. Where relevant, capture any value statements for InCommon.
5. Where possible, identify existing tools around the global federation community

Work Products / Deliverables

The InCommon Federation Test Environment Working Group shall conclude its work no more than six months from its initial convening date. The group will produce a written report at the conclusion of its proceedings.

The final report should provide the recommendations referenced in charter items 1, 3, and 4. This includes:

• The specific, implementable testing criteria for the testable Deployment Profile statements - This is a set of prescriptive test specifications to make the criteria clear to InCommon for implementation. The tools will need to present results to testers in the form of information about what statements their entities met and did not meet and why.
• Tests of specific functions presented according to priority
• Any additional details necessary to ensure compliance with the statements
• Any memorable statements of the value that testing / validation tooling for the Deployment Profile requirements will bring to InCommon Participants and prospective participants (e.g., interoperability, etc.)

The working group should not overly constrain itself with immediate implementation challenges. It will be left to the implementers to develop strategies accordingly.

Should unforeseen reasons prevent the working group from completing its work at the end of the six-month period, it should still produce a written report describing its work to date as well as recommendations for any follow-up actions. 

Membership

Membership in the InCommon Federation Test Environment Working Group is open to all interested parties. The solicitation will take place on the InCommon Participants list. TAC leadership may explicitly name key stakeholders to participate in the working group to ensure balanced representation from IdP, SP, and Federation operators. Members with software development experience will be critical to the success of the group. Members join the Working Group by subscribing to the mailing list and Slack channel, participating in the calls, and otherwise actively engaging in the work of the group.

References

• InCommon Federation Adopts SAML V2.0 Deployment Profile
• Draft InCommon Test Federation Requirements from Nicole Roy - https://docs.google.com/document/d/1vQ_jk7ApSpuClTiQCTqcmbjpGXRpPT0VLbfwD0MEaOI/edit#heading=h.5xox0fk8w6i 
• Test Federation User Stories and Planning Sub Group notes - 
  https://docs.google.com/document/d/1SW_I2hgLift1XirpAJwf2ZYAG4b22mv6sNqDXCqm0YU/edit#heading=h.xbd497z1biub
• Working draft location of the working group charter - 
  https://docs.google.com/document/d/1_8hwYmjrmsW_c-VBZC4cGqEHfHTwOh5rXihh2P-J-lo/edit#