Page tree
Skip to end of metadata
Go to start of metadata

Welcome to the NET+ Google Cloud Platform (GCP) wiki

Many Internet2 member institutions take advantage of this service offering. If your institution is one of them then this wiki will provide details on how to make the most of your participation of the programming and interact with peers across Internet2 member institutions. 

This program is open to all US higher education institutions. There are additional access fees for institutions that are not members of Internet2.  For details on how to join the program, please email netplus@internet2.edu. 

You can also find out more about the Internet2 Cloud Connect offering for GCP Partner Interconnect.

Service Documentation and Resources

Accessibility:

Identity:

  • GCP identities are closely tied to the GSuite environment - schools currently use Google Cloud Directory Sync to sync users and to populate Google Groups
  • If you are interested in working on a recommendation for dynamic group population and Role mapping for authorization management - email netplus@internet2.edu

Information Security:

Contract and Pricing:

Community Resources

Participate in our Subscriber Community:

Institutions participating in the NET+ GCP program may take advantage of our email discussion list and Slack channel to receive curated program updates and participate in other activities and events.

The NET+ GCP Service Advisory Board hosts regular subscriber calls where campus cloud teams meet to discuss their challenges, share lessons learned and collaborate to find the best answers for their institutions' GCP deployment. We regularly bring in Google engineers or product managers to discuss services and give feedback on how GCP features could best serve the unique needs of higher education institutions.

Please contact Bob Flynn bflynn@internet2.edu to be added.

Join the GCP Community Forum (Open to all community members):

Users of GCP are encouraged to join the #google channel in the EDUCAUSE Cloud Community Group Slack. See the Higher Ed Cloud Community page on the Cloud Wiki for instructions to join.

Collaborate on the Cloud Wiki:

Speaking of community, did you know about the Cloud Wiki? This was created specifically for YOU, members of the higher education community to collaborate with each other. Log in to see a Cloud Job descriptions page and contribute your knowledge!

Contribute Code:

Looking to share your latest Terraform config? Add it to the Cloud Wiki Helpful GitHub Repos list or email sjeanes@internet2.edu to request access and create a repo in the Community Cloud Config GitHub organization.

Questions on Offers, Distributors and Resellers, Agreement Structure:

Find answers to frequently asked question in these Knowledge Base articles.

Key Program Updates

Subscribers may review our mailing list archives for monthly program and GCP updates.

Here at Internet2, we are fortunate to be working with a wonderful group of students from Notre Dame's Master of Science in Business Analytics program. The group is working to help us gain insight from detailed usage data we get from the NET+ AWS and GCP programs. Our hopes are that we will be able to use that data to observe emerging patterns of cloud infrastructure in higher education and research, and to use that knowledge to help the community support effective cloud use.

In order to provide analytic access to the data, which is kept in Google Big Query tables, we wanted to provide the students with a Jupyter notebook environment where they would not need to download or store the data on their own personal laptops while they work with us. This post documents how we are providing that environment using Managed Notebooks in GCP's Vertex AI Workbench.

We have set up a Google Group for the class project, containing the members of the class as well as the Internet2 staff working on the project with them. In order to allow the group the ability to create notebooks, we added the Notebooks Admin role for the group within our GCP project (as described in (https://cloud.google.com/vertex-ai/docs/workbench/user-managed/iam). Open question: Would Notebooks Runner be adequate for our purposes?

For our purposes, as we only have four students in our group, we used the GCP Console to manually create the notebooks. The process could be automated  for larger repeated use (or one could use Google's Rad Lab Data Science repo).

The process for creating Managed Notebooks is documented here: https://cloud.google.com/vertex-ai/docs/workbench/managed/create-managed-notebooks-instance

At present Managed Notebooks are only available for a single user, so we created an individual instance for each student, naming each notebook with the student's email identifier. Each notebook can be assigned a single owner (at the bottom of the Advanced Settings screen), which is where you assign the notebook to the student's email address.

To help in managing costs, we reduced the size of the instances from the default n1-standard-4 to n1-standard-2, and reduced the idle timeout period from 180 minutes to 60 minutes.

The result of creating notebooks manually in the console is a running notebook process, viewable in the Vertex AI Workbench screen in the console. We then stop those processes, as we will rely on the students to start them up when they want to use a notebook.

To give the notebooks access to our Big Query tables required assigning the BigQuery Read Session User role to our group. The group already had the BigQuery Data Viewer and BigQuery Job User roles assigned within our project.

The process for accessing Big Query data from a Jupyter notebook is documented here: https://cloud.google.com/bigquery/docs/visualize-jupyter

Because we are using GCP Managed Notebooks, all the necessary pieces for accessing Big Query are pre-installed (as are the usual Python data science modules), so the notebooks are ready to go once started.

We anticipate very low costs for using this service: Managed Notebooks are currently in Preview, and there is no management fee for managed notebooks while in Preview. The instance costs for the n1-standard-2 machines are $0.10 per hour. There can be costs for queries submitted to Big Query, but we anticipate that our uses will remain well within the free tier of Big Query usage.

Many thanks to Maddie Howe for helping to test and troubleshoot this process!

We sent out the following instructions to the students to let them know how to access their notebooks.

I’ve set you each up with a Jupyter environment in our GCP organization for work on the capstone project. 
To get to the environment, follow these instructions:
  1. Go to the Managed Notebooks page in the GCP console: 
    https://console.cloud.google.com/vertex-ai/workbench/list/managed?_ga=2.66336813.283589364.1646256329-1869828962.1513966007
  2. You should see a notebook named with your email id – e.g. nd-capstone-jdoe
  3. Click in the checkbox next to your notebook name and then click on the Start icon up on the Workbench 
    menu line at the top of the page.
    (if you don’t see the Start icon, click on the three dots there and you will).
    It takes 5-10 minutes to spin up the instance.
  4. Once your instance is running, click on Open Jupyterlab and you’ll get a new tab with 
    Jupyterlab – that can also take a few minutes.
  5. You can then start a new notebook.
  6. You should be able to access our Big Query tables as documented here:

    https://cloud.google.com/bigquery/docs/visualize-jupyter
A sample query to test:

%%bigquery testdf
SELECT distinct Product_Name FROM `projectname.datasetname.tablename`
order by Product_Name

That will put the result of the query in a pandas dataframe called testdf. To verify:
print(testdf)

A few notes:
- When you’re done using Jupyter, please go back into the console and stop your instance.
- The instances time out after 60 minutes of no use, so it’s not the end of the world
if you don’t stop it, but it’s a good practice to get into.
- The instances are not huge – 2 CPU, 7.5 GB of RAM, no GPU, 100 GB of disk. If you need more power, please let me know.

Update: March 9, 2022

Aaron Gussman from Google sent along an example of using the notebooks API to create a managed notebook instance, which doesn't appear to be in Google's documentation anywhere yet.

Here is the API example to create a Managed Notebooks runtime with Idle Shutdown settings:

BASE_ADDRESS="notebooks.googleapis.com"

LOCATION="us-central1"

PROJECT_ID="YOUR_PROJECT_ID"

AUTH_TOKEN=$(gcloud auth application-default print-access-token)

RUNTIME_ID="my-runtime"

OWNER_EMAIL="YOUR_EMAIL"

RUNTIME_BODY="{

  'access_config': {

    'access_type': 'SINGLE_USER',

    'runtime_owner': '${OWNER_EMAIL}'

  },

  'software_config': {

    'idle_shutdown': true,

    'idle_shutdown_timeout': 180

  }

}"


curl -X POST https://${BASE_ADDRESS}/v1/projects/$PROJECT_ID/locations/$LOCATION/runtimes?runtime_id=${RUNTIME_ID} -d "${RUNTIME_BODY}" \

 -H "Content-Type: application/json" \

 -H "Authorization: Bearer $AUTH_TOKEN" -v






The new year is underway and already there is a lot going on. I’m going to order this edition’s content based on how soon the deadline is approaching and how urgent it is for your attention.

Three Alarm

Kion Infoshare

Over the past year we have been hearing from schools in the higher education cloud community that they are looking for some help sorting through the plethora of 3rd-party tools swirling around the cloud infrastructure space. The top needs expressed were:


  1. account automation and maintenance
  2. security monitoring and remediation
  3. cost tracking management and savings 

One product specifically suggested and already being piloted by a few schools is Kion (formerly known as cloudtamer.io). We asked Kion to show the community around their tools and answer your questions. 

Research Innovators Program

Google Cloud is committed to supporting researchers across the globe who are solving complex challenges. As part of our efforts, we are relaunching the Google Cloud Research Innovators program. Applications are open until January 14th, 2022. To apply, please fill out this short form. For additional information about eligibility for the program, please visit the Research Innovators overview.


Two Alarm

Funding opportunity from NSF + Google

For the second year in a row, Google has partnered with the National Science Foundation (NSF) to offer cloud funding and training through CISE-MSI solicitation 22-518. The CISE-MSI program offers $7M in funding for PIs at Minority-Serving Institutions (see list here). Applicants who request funding for Google Cloud via NSF 22-518 are eligible for a 50% match in cloud funds from Google, expanding their total possible NSF award amount by the matched number. Awardees will also gain access to free, live, instructor-led workshops and classroom tools from Google Cloud. Applications are due February 11th, and details for how to apply can be found in the solicitation documentation.

Program Satisfaction Survey

One of the vital roles of the NET+ team is to bring the community together, making and sharing opportunities to learn from each other and from the cloud vendors. NET+ Program Development Manager Tara Gyenis has put together a brief survey to get your take on how well we are doing with community engagement. We also want your feedback on vendor (GCP) and resellers’ performance. The survey is designed to get input from everyone from the cloud team to procurement to management. We ask that you share it with others at your institution. Please respond by January 31, 2022.



Speaking of engagement…

One of our most powerful engagement tools are our meetings. Yeah, yeah, I know, meetings… But no, seriously. When we bring you, the members of the community together, magic happens. The sharing of challenges, successes, approaches, obstacles, plans, etc. sparks some of the best conversations and most productive networking and collaboration. 

We are sprucing up the NET+ GCP engagement calendar for 2022. Here’s what we have planned. Calendar invitations to come out soon.


    • NET+ GCP Technology – We are going to spin up something new for the NET+ GCP program that has already been very successful in the NET+ AWS program – regular “open mic” technical discussions. These calls are for your GCP architects, admins, etc. to discuss technical questions and challenges you might encounter when managing Google Cloud at your institutions. We’ll share the questions, answers, lessons learned and challenges that remain. (monthly to start)
  • NET+ GCP Subscribers – By adding the technical call, the Subscribers meeting will be able to focus more on strategic discussion. The target audience will be the cloud managers, infrastructure directors, etc. who drive cloud enablement and infrastructure stategy. Twice over the course of the year we’ll ask you to invite your leadership and we’ll invite GCP and Google Edu leadership to attend as well. (quarterly)
  • NET+ GCP Town Halls – We will bring the community together to learn from Google (or other presenter/panel) about a big picture topic/approach/strategy/trend. Attendees come away from these sessions with greater understanding of the topic and how they might take advantage of this knowledge within their institution, where to go to learn more, etc. Community interest will drive the topics. If you have ideas, please send them to bflynn@internet2.edu

We will be sending calendar invitations to all of these to the NET+ contacts we have. Please accept the invitations for the meetings that interest you and decline the rest. And please do forward them on to others in your organization for whom they may be appropriate. 

Make a difference in your community. Join the NET+ GCP SAB


While NET+ Cloud IPS Manager Bob Flynn provides the staff time to address program mechanics and keep things running, it is the NET+ GCP Service Advisory Board (SAB) that provides the leadership and guidance to move the program forward. We are seeking new voices, new schools and new perspectives on the SAB. Make 2022 the year you get more involved in this community. Help make a difference to schools all across the country. Submit your name to be on the NET+ GCP SAB by emailing netplus@internet2.edu


One Alarm

ICYMI – For higher education community, choosing Google Workspace for Education is a team effort 

GCP administrators have always had to be cognizant of the connection between GCP and Google Workspace. Increasingly Google Workspace users are leveraging GCP in both subtle and sophisticated ways. In a nod toward keeping your GCP teams abreast of developments with Google Workplace for Education, I recommend this blog post by Steven Butschi, Head of Education for Google. It recounts the hard work done by many of you in this community to bring Google to the table to work out a path to grow with Google’s cloud and productivity platforms.

Credits

The images in this newsletter come from a variety of sources. Credit where credit is due.

Requests?

Do you have ideas or requests for a future newsletter or engagement event? Email bflynn@internet2.edu



NET+ GCP Service Advisory Board Membership

  • John Bailey, Washington University of St Louis (Chair)
  • Atul Pala, San Jose State
  • Brian Pasquini, University of Pittsburgh
  • Damian Doyle, University of Maryland, Baltimore County
  • Erik Lundberg, University of Washington
  • James Bennett, Indiana University
  • Jim Thomas, Indiana University
  • Joshua Hyman, University of Pittsburgh
  • Maneesha Aggarwal, Columbia University
  • Nick Young, University of North Carolina, Greensboro
  • Rikus Gosen, Masachusetts Institute of Technology
  • Eric Rau, Michigan State University
  • Sam Porter, University of Maryland
  • Stratos Efstathiadis, New York University
  • Rick Rhoades, Penn State University
  • Oren Sreebny, Internet2, Staff Liaison
  • Bob Flynn, Internet2, Staff Liaison

To Contact the Service Advisory Board

Questions?

Send Feedback or Submit a Feature Request:

The NET+ GCP program is managed by an Internet2 program manager with the support of the NET+ GCP Service Advisory Board. 

The NET+ GCP Service Advisory Board reviews and priorities community feature requests on a periodic basis. Feature requests may be submitted to netplus@internet2.edu.




 




  • No labels