- Friday 17 March 2023
- Presenters / Facilitators
- Ashish Pandit, University of California San Diego
- Poll / Survey
- Google Slides presentation (including responses to the poll noted above).
2. Role of Architecture in Microservice (and API) Design and Strategy
2.1. Background & Context
- A microservice and API strategy fits well with UCSD’s user-centric approach and focus on customer experience.
- The broader goal of having a solid architecture in this area is to help deliver this customer experience.
- Ashish will share aspects of the API and microservices journey at UCSD and ideas from the ongoing Itana API/Governance working group.
2.2. Basic API Definition and History
- An API (application programming interface) is a software intermediary that allows two processes or applications to interact with each other.
- APIs often allow for either retrieving or sharing data.
- Many APIs at UCSD still tend to be related to data and not necessarily about improving business processes and user experiences.
- There are missed opportunities here.
- Ashish appreciates the way MuleSoft defines APIs and classifies them into three categories:
- Less re-usability - similar to a “verb”
- Most re-usable - like a “noun”
It can be helpful (natural) to align API security policy with these three areas.
- History of Architectural Patterns
- SOAP, REST, etc.
- SOAP is old but still works and can be very solid in certain circumstances.
- Some transition is taking place to newer technologies:
- An example is JSON-RPC (may lend itself to message based processes)
- There is always a danger of creating "monolithic" APIs
- For example, an API representation of an entire database
- APIs like this are hard to update and change
- A good microservices architecture involves small, autonomous services that are modeled around a single business process within a particular business domain.
- Microservices should be self-contained and ideally implement a single business capability.
- Microservices may sometimes include data transformations to suit particular business needs.
- Netflix is a pioneer in microservices
- While microservices offer a lot of flexibility, they don’t necessarily provide a great user experience.
- Netflix has been introducing GraphQL as a way to improve the user experience of accessing different microservices.
- Some key elements of microservices:
2.3. "Product" approach to APIs and microservices
- Ultimately the promise of API and microservice approaches is to improve business capabilities.
- Align technology with business needs that create real value and improve user experience.
- Ashish feels that a "product" approach to APIs and microservices is more likely to be successful than traditional “project” approaches.
- This means that solutions should be treated as a product that is assessed and improved over time (as opposed to a time-boxed, one-time solution for a specific need)
- There should be a focus on publishing and "externalizing" solutions - making them available to developers and other teams.
- This MuleSoft article on products vs. projects contains numerous links to resources on this topic.
2.4. API Survey Results
- There were 12 responses to the API survey (linked at the top of this document)
- 2/3 of institutions have been using APIs for over 5 years.
- Much API work is still ad hoc and solutions are not published for broader use.
- While there are many different drivers for API adoption, few institutions prioritize API work based on business value or strategic alignment.
- API work is often ad hoc, reactive, or project driven.
- Project based work isn’t necessarily a bad thing as long as the project aligns with business goals and delivering value.
- Ideally, solutions created out of a project will then be managed as long-lived products with a lifecycle and a customer focus.
- Without the right people and broad enough thinking, we often miss chances to make APIs reusable and valuable to others.
- REST and SOAP APIs are by far the most common among respondents.
- Some of the most significant barriers to API adoption include:
- Resistance to change and cultural support.
- Problems with legacy processes and technical debt.
- Security and privacy considerations.
- Governance and coordination is often limited.
- See slides for full, detailed survey results.
2.5. Challenges & Opportunities
- Support model for APIs and microservices
- Once projects are done and solutions are implemented, who supports the API going forward?
- Risk of duplication of effort
- Often, numerous teams or individuals have access to the same data.
- Existing business and process needs/gaps are often a great way to jump start an API initiative.
- To truly realize the potential of APIs and microservices usually requires close collaboration and coordination among different teams.
- Can we really work in an API-centric manner?
- Can we do this across large IT organizations?
- Vendors, SaaS solutions, etc.
- A question came up about this and it led to some interesting discussion.
- External vendor solutions can introduce challenges.
- Vendor APIs and capabilities may be lacking and not fit nicely in this model.
- This can definitely be a point of tension.
- However, APIs are probably just as important as accessibility!
- You can’t poison your “digital DNA” with products that don’t interoperate in modern ways. (Jeff Kennedy)
- However, sometimes applications and vendors are “forced” upon us.
- Making APIs and microservices central to vendor conversations is key.
- Often there is a focus on how to “get data in” but just as important is how to “get data out”.
- Most solutions and applications produce data that can be very valuable.
- What happens with that data and how can we use it?
- Exposing and publishing data helps foster a product culture.
- Vendors will often throw around the term “API", but support and technology is shaky.
- IT teams often need to swoop in to fill these gaps.
- This can require significant resources.
2.6. Governance, Strategy, and Maturity
- Governance tends to be mostly ad hoc and informal (see survey results in slides).
- UW noted that they are starting to work towards API governance but there is uncertainty in terms of what this means and how to implement it.
- API "turnkey" solutions from vendors
- Tricky territory
- Still may require significant staff investment to realize potential.
- Without the right architecture, cultural support, and mindset, these solutions are not likely to help.
- Gartner strongly connects maturity to APIs aligning with business value:
- "People don’t want drills, they want to make holes." (Jeff Kennedy)
- This approach doesn’t lend itself to a business value / business aligned approach.
- Higher Ed needs a rich “platform” of APIs and microservices that can be leveraged to deliver seamless user experiences. (Jim Phelps)
- When done well, it can empower developers and other service teams to "hide the details" from users and bundle various business functions into something functional and beautiful.
2.7. Role of Architecture
- Business focus is key - start with business needs as the why.
- Architecture spans various domains:
- Should “solution architects” be included here?
- Solution architects play an important role in implementation decisions at some institutions.
- There is value in maintaining an API and microservices strategy and “product” that lives outside the typical stream of IT projects.
- When done well, projects can tap into an existing API and microservices framework when these needs arise within project work.
- Who represents the end-user experience, though?
- This varies a great deal.
- Sometimes, someone is looking out for the student experience in a broad sense.
- Often there is a lack of coordination, however.
- This represents more missed opportunities.
- Can enterprise architecture help bring these various teams and ideas together to create a coherent API and microservices strategy?
2.8. Next Steps at UCSD
- Looking at the CAUDIT Higher Education Data Reference Model.
- Broadly, the desire is to align teaching, learning, and business capabilities with API and micoservices strategy.
- Thinking about the three types of APIs (system, process, experience)
- System APIs are closest to university systems of record
- Focus on aligning process APIs with business capabilities.
- Ideally, explore GraphQL to help tie things together to produce experience APIs.
- Closest to users, most impactful on user experience.
- Stress the importance of single purpose APIs.
3. Itana Business
4. Further Information
4.2. Zoom Chat Transcript
12:17:42 From Alberto Mendoza to Everyone:
do these Microservices do any data transformation to suit the business need?
12:19:01 From jeff kennedy to Everyone:
Impressive seeing one institution with more than a hundred APIs available for consumption by people in the wider community!
12:19:20 From Jim Phelps (UW) to Everyone:
Reacted to "Impressive seeing on..." with 👍
12:20:59 From jeff kennedy to Everyone:
...evidently, a few years ago now, "By 2015, Netflix’s API was receiving two billion API edge case requests each day across over 500 microservices. This increased to 700 microservices by 2017."
12:25:49 From Glenn Donaldson (Ohio State) to Everyone:
We just had to give stats to our leadership so just sharing info: DIS+A: Enterprise Integration Platform & Other Services:
275 - 300 Integrations(Web Services/APIs/micro-services) - representing integrations across many core systems including Workday, PeopleSoft Student, Space Information System….
95 -100 Consuming Units/Applications representing multiple individual users in College/Departments/External Vendors/Systems
Provided/still providing data for COVID testing. etc.
12:26:22 From jeff kennedy to Everyone:
Replying to "do these Microservic..."
Thinking of the MuleSoft three-layer deployment model for APIs, and noting that microservices are self-contained and fully autonomous, does that mean a microservices API is always a _system_ API? If so, there is presumably quite an industry required around the governance of how contracts and vocabulary and authentication and authorisation are managed across the microservices fleet (especially when there can be hundreds of them!).
12:26:36 From jeff kennedy to Everyone:
Reacted to "We just had to give ..." with 👍
12:27:45 From jeff kennedy to Everyone:
¿ This might be a leap, but is a microservices approach particularly well-suited for institutions that have adopted a data mesh architecture?
12:36:14 From jeff kennedy to Everyone:
+1 for the "bat cave"! 🦇
12:39:58 From jeff kennedy to Everyone:
¿ How different does "API Governance" actually need to be from other types of architecture + design + delivery governance?
12:41:21 From Glenn Donaldson (Ohio State) to Everyone:
That is just too logical Jim!
12:41:52 From Alberto Mendoza to Everyone:
"I don't like the way you built it so I'm going to build my own" <-- I'm sure this resonates with this group a lot 😆
12:42:03 From Jim Phelps (UW) to Everyone:
Reacted to ""I don't like the wa..." with 👍
12:42:24 From Dave Goldhammer (CU Boulder) to Everyone:
What I’ve observed is a general lack of coordinated strategy, both in terms of data and business processes. This tends to lead to tightly coupled integrations instead of more reusable, shared, autonomous functions. A few teams have taken some initiative to develop APIs in a proof-of-concept manner - hopefully this can demonstrate value and create more buy in.
12:44:12 From jeff kennedy to Everyone:
#lol is there anywhere ChatGPT isn't popping up? Overnight, this DZone piece on getting ChatGPT to create a RESTful API (the example is actually for "Student") turned up, and opens a whole other can of worms about the role of AI in software development = https://dzone.com/articles/create-a-rest-api-in-c-using-chatgpt
12:46:47 From jeff kennedy to Everyone:
Reacted to "What I’ve observed i..." with 👍
12:49:40 From jeff kennedy to Everyone:
@Jim = absolutely, we must protect our customers against having to learn and navigate our internal organsation structures when they engage with our institutions (positive statement here for business capabilities at the heart of our value streams and as the basis of our API scaffolding!).
12:49:53 From jeff kennedy to Everyone:
Replying to "@Jim = absolutely, w..."
1967, Conway's Law = “Organizations, who design systems, are constrained to produce designs which are copies of the communication structures of these organizations.”
12:52:43 From Alberto Mendoza to Everyone:
What shocks me is how fast ChatGPT is rolling out everywhere... Yesterday MS unveiled ChatGPT / Copilot as part of Office 365... PowerPoint will use ChatGPT to create entire slideshows for you.
It just reminds me of Amazon rolling out Alexa, putting it on Thermostats and what not, and only but recently they disbanded that program, and IBM selling out Watson Health. I wonder if GPT will meet the same fate as these other AI/ML/NLP tools.
12:54:25 From jeff kennedy to Everyone:
Reacted to "What shocks me is ho..." with 👍
12:56:56 From Dave Goldhammer (CU Boulder) to Everyone:
Reacted to "@Jim = absolutely, w..." with ❤️
12:58:59 From Beth A Schaefer to Everyone:
This was so informative today, Appreciate your work, Ashish!