The SK&A Group Migrates Data to the Cloud with AWS S3 & Lambda

Challenge:

The SK&A Group (structural engineering, testing, and inspection services) had archive data sitting on-premises but needed to migrate 1 terabyte (TB) of data into the cloud. They needed to preserve the existing file structure to minimize user acclimation. For accessibility, the engineering firm also needed to do a bulk import of users and assign each user to a group.

Solution:

SK&A expressed an interest in testing AWS’ capabilities due to its reputation as the number one cloud solution platform and ease of user administration with Amazon’s Identity Access Management. Amazon’s Simple Storage Service (S3) seemed like a clear choice when evaluating cloud storage solutions due to its 11 nines durability and efficient administration.

During an introductory analysis of SK&A’s needs and on-premises file store, SHI’s AWS architects decided that most of the engagement could be scripted. The project was split into two separate phases: one for importing and creating users and a second phase for migrating data.

SHI’s AWS experts utilized AWS Lambda to create and run a Python script that accepted a csv (provided by SK&A) from an S3 bucket, parsed it, interacted with IAM to create AWS Console users for each specified user in the file, and placed each new user in their appropriate IAM group. By utilizing Lambda in Phase 1, SHI greatly reduced the amount of time it would take to provision and configure each user account manually.

In Phase 2, SHI installed the AWS Command Line Interface (CLI) on SK&A’s host machine. The CLI was then configured with an IAM user that only had programmatic access and the least amount of privileges it would require to move data into S3. Since preserving SK&A’s file hierarchy was a top priority, SHI Solutions architects wrote scripts to move data into S3 incrementally. The migration schedule minimized the impact on SK&A’s business operations, avoiding heavy network saturation.

The data was migrated into two parent folders belonging to one S3 bucket which allowed SK&A to keep data logically separate for each of their geographic sectors. By running this type of migration script, SK&A could do any “out-of-scope” migrations themselves without learning the intricacies of AWS.

Both during and after the engagement, SHI Solutions architects helped SK&A learn more about S3 – educating the IT staff about Glacier and S3IA, and discussed whether or not their data was a good candidate for either.

Benefits:

Moving the required data archives to AWS allowed SK&A to confidently and securely store and access their data. By leveraging S3, SK&A can implement native control mechanisms to access data (Bucket Policies and Access Control Lists), and Server-Side Encryption to encrypt data stored at rest. SK&A no longer needs to back up the archives due to S3’s inherent 99.999999999% durability. A native AWS solutions enabled SK&A to save money by eliminating the use of a third party software, and provided the ability to only pay for the storage capacity in use, which lowered overall costs. The new solution will save SK&A thousands of dollars in years to come!