This is My Architecture: Speed HPC Data Sets Quickly, Securely

This is My Architecture: Speed HPC Data Sets Quickly, Securely

By Flux7 Labs
February 19, 2020

Each week AWS features innovative cloud architectures from its partners and customers in a video feature it calls, This is My Architecture. We’re excited to share that this week’s installment features our very own DevOps Engineer, Arthur Mandel, and David Fluck from Toyota Research Institute. In the episode, they highlight how two technologies — AWS P3 GPU instances and AWS Service Catalog — enable TRI data scientists to more quickly progress their research.

Toyota Research Institute: On-Demand Self-Service Portal for Data Scientists to Process Data Sets

Specifically, Mandel and Fluck share how they: 

  • Deployed AWS P3 GPU instances to deliver high performance compute in the cloud to increase the number of simulations completed for TRI’s high performance computing. Amazon reports that these instance types can increase simulations completed by 3-4x and have been proven to reduce machine learning training times from days to minutes.
  • Created an on-demand self-service portal for TRI data scientists with AWS Service Catalog, Amazon’s service for creating and managing catalogs of IT services. TRI’s data scientists use the Service Catalog-powered portal to create and process data sets both quickly and securely.

In the episode, Fluck and Mandel walk viewers through the process from the user’s perspective, illustrating how Python Scripts, Service Catalog, S3 buckets, FSX for Lustre, GPU clusters, Machine Learning models and more all work together to help TRI autonomous vehicle efforts. Watch the episode today to learn how to use AWS Service Catalog products and trigger them on-demand to create P3 compute clusters to process Machine Learning data sets.

 

Interested in additional HPC Resources? Check out our HPC resource page or contact us today to learn how our HPC Consulting Practice can help address your technology and business needs.

Technology is always changing. Stay in the loop with the Flux7 Blog

Written by Flux7 Labs

Flux7, an NTT DATA Company, is the only Sherpa on the DevOps journey that assesses, designs, and teaches while implementing a holistic solution for its enterprise customers, thus giving its clients the skills needed to manage and expand on the technology moving forward. Not a reseller or an MSP, Flux7 recommendations are 100% focused on customer requirements and creating the most efficient infrastructure possible that automates operations, streamlines and enhances development, and supports specific business goals.