Multi-Factor Authentication Solution Evaluation Criteria
This document outlines criteria that should be consider when evaluating multi-factor authentication products and services. Much of the content in this document is based on material from The Quest to Replace Passwords: A Framework for Comparative Evaluation of Web Authentication Schemes by Joseph Bonneau, Cormac Herley, Paul C. van Oorschot, and Frank Stajano, March, 2012.
Deployment Environment
- Is the solution compatible with existing server and client platforms? Examples for higher education include:
- Operating Systems
- Windows
- Macintosh
- Linux
- Android
- iOS
- Browsers
- Firefox
- Chrome
- Safari
- Internet Explorer
- Web application platforms
- PHP
- .NET
- Java
- Ruby
- Python
- Node.js
- Microsoft AD
- Globus
- Unix shell
- VPN
- Mobile
- CAS
- Shibboleth
- Radius
- Operating Systems
- What features does the solution have to facilitate broad deployment throughout a university community?
- administration tools
- user support tools
- integration with local SSO
- ability to delegate of admin rights
- What features does the solution have to be deployed in a BYOD environment?
- Is the solution scalable and flexible enough to be adapted to potential future applications?
- Does the solution support capabilities other than authentication, such as encryption and digital signature?
- Is the solution cloud-based or on-premise? Where are the potential failure points?
- Are capabilities provided that can assist migration to another solution at some time in the future?
Usability and Accessibility
- Are users required to remember secrets? How many? What is the required complexity of those secrets?
- Does adding more accounts for the user increase the burden on the user?
- Are users required to carry an additional object (to be used as an authentication token)?
- What physical effort is required of the user?
- How easy is the solution to learn?
- How much time is required to perform an authentication? To associate a new account?
- How reliable is the authentication process (i.e., are false negatives common)?
- How easy is it to recover from a lost token or forgotten credentials?
- How accessible is the solution?
- Do certain disabilities preclude its use?
- Are multiple technologies supported to accommodate to specific disabilities?
- Overall, what strategies are employed to address accessibility issues?
Security
- Is the solution resistant to physical observation?
- Is the solution resistant to targeted impersonation (e.g., by an acquaintance)?
- Is the solution resistant to throttled guessing?
- Is the solution resistant to unthrottled guessing?
- Is the solution resistant to internal observation (e.g., by intercepting traffic across a network or within an endpoint device?
- Is the solution resistant to leaks from other verifiers?
- Is the solution resistant to phishing?
- Is the solution resistant to theft?
- Does the solution require a trusted third party?
- Is explicit user consent required to complete authentication?
- Can authentication with one verifier be linked with another?
- Does the solution require secrets to be stored on a server?
- Is the technology mature? Has it been reviewed by a sufficiently large community?
- Is the solution proprietary? Can the implementation be assessed independently by users?
User Support
- What documentation is provided? Is there an FAQ?
- Are tools available to assist help desk personnel?
- Are training materials and/or classes available?
Product Support
- What support services are a provided by the vendor? By third parties?
- What management tools are available?
- Are add-ons available from third parties? What other vendors have integrated the solution with their products?
- What are the communication channels with the provider?
- Is there a viable user community?
Compliance
- Does the solution satisfy the authentication criteria for well-known assurance profiles?
- Does the solution satisfy the authentication criteria for compliance with FERPA, HIPAA, PCI, and other requirements for higher education?
- Does the service agreement share liability appropriately?
- Does the solution conform with applicable standards, particularly FIPS 140-2 and NIST 800-63-2? Are there plans for alignment with the Fido Alliance?
Costs
- Pricing
- What is the purchase cost of the solution?
- What are the support costs of the solution?
- Which portions of the cost are incremental (per user), which grow as the number of users grow (although perhaps for each user), and which are fixed?
- Life-cycle costs
- What is the startup cost?
- What are the ongoing operational costs? What are the staffing requirements?
- What is the fixed cost per user?
- Are there variable "per user" costs?
- What is the potential cost of migration to a new solution in the future?
- Costs that may be born by end users
- Does the solution require telephony services that may have associated costs?