Mass Data Migration from Egnyte to Microsoft
Our client is an international advertising agency who were using Egnyte to store data for all of their offices globally. Due to the high running and renewal costs, Support Partners were approached to migrate all existing Egnyte assets to the Microsoft Platform.
Solution
Scope
Multiple Egnyte Instances across multiple business entities
Identified Files
Over 100 Million
Total Storage
800-900TB
Utilising the Microsoft Azure platform Support Partners designed and developed a migration engine that not only reduced costs but also improved compliance
Due to the scale of the problem the client faced Support Partners divided the project into 2 phases. Firstly an analysis phase of the content stored in Egynte and secondly to sort and migrate the content to Microsoft Sharepoint, Microsoft OneDrive and Azure Blob storage.
The analysis enabled both Support Partners and the client understand the volume and scale of content. This included details of the type of content stored by users, how and why. By using this data, a rules engine was developed that ensured content could be migrated efficiently, securely and to the most appropriate destination .
For example, the analysis showed that there was a large number of video files (production broadcast videos) being stored on a global scale. Using the analysis data we were able to identify these files and build a rule to move them to lower cost Blob storage while moving other collaborative assets to Sharepoint maintaining the value of sharing that Egnyte offered.
A second example was centred around compliance. The analysis highlighted a large amounts of Technical files (Systems files, Disk Images) that had being stored within Egnyte. These files posed a risk to corporate compliance and security. Through the analysis, non compliant files were removed from the migration.
Total Files by Size and Number
Analysis
Using Azure native services, Support Partner built a series of Azure Functions to list content in Egnyte using the API. Capturing all the metadata including the file, type, permissions and comments. Storage of this data was put into Microsoft SQL server and was used to build the migration engine. To ensure the database was always accurate a Egnyte listener was also created to pick up updates.
Analysis dashboards created included:
Summary data of the total storage by file type and category
Egnyte analysis progress showing how many records were processed (on average we would process 880,000 records per day with peaks around 3 millions records)
Zero size folders (Client were able to remove 16000 empty top level folders)
Top level folder analysis showing the breakdown of a top level folder and the sub contents (This was used to identify owners of these folders and the migration approach)
81,415,266
# Files
11,708,177
# Folders
15,309
CommentsNo.
231,104,092
PermissionsNo.
105,592
UserAccountsNo.
302
GroupsNo.
Approach
When presented with the challenge from the client and the brief to “migrate from Egnyte” the scale of the amount of content was not know. Initial research showed a large amount of content but there was no real understanding of what needed to be migrated and how. Support Partners therefore initiated a project to analyse Egnyte and extract all asset data into an easy to interpret dashboard.
Our project approach followed these steps:
Discovery phase – an investigation to the scope and definition of the requirements.
Development phase 1 – analysis engine build and testing
Analysis phase – report and dashboard development and analysis working closely with the client into the insight. Scope and definition of the requirements for the migration engine
Development phase 2 – migration engine build and test
Release to production with ongoing support
All the time ensuring that anything developed could be repurposed or would have value downstream.
Real-time analysis data
Analysis of file types
rules engine UI
rules list
Migration
The data presented in analysis enabled Support Partners to build a lightweight policy engine on the Azure Platform to facilitate the migration. The engine was built using React components within an Azure Web Application utilising the analysis data in the SQL database to generate policies that could be applied to migrated content.
The features of the application include:
Ability to define rules on any criteria stored within the database, for example by file category, extension, size, name, permission with field-level logic.
Ability to choose different endpoint by rule (this could be Blob storage, Sharepoint or OneDrive)
Ability to apply rules to specific top-level folder for migration
Ability to copy comments to Sharepoint
Ability to create stub file in Sharepoint for content migrated to Blob
Ability to track the progress of each migration, identify errors, rerun failed files.
Ability to use Microsoft Delegated Permissions from a user to access their OneDrive to migrate content to
All migrations were tracked in the database and reports were generated showing completed migrations, amount of content migrated and the time to migrate. Enabling the client to have a real-time view of the migration status.
Technology used
Azure Functions
Azure SQL Database
Blob Storage
Service Bus
Azure App Services
Azure Application Insights
Microsoft Graph API
Egnyte API
Impact
It has enabled our client to understand how collaborative platforms are being used. In this case from Backups to Work in Progress repositories to an Ad hoc storage for unstructured data
Migrating their assets out of Egnyte and onto Microsoft it saved them hundreds of thousand dollars per year in renewal and consumption costs.
The core development for the migration engine is being repurposed as a policy-based archiving solution for SharePoint
Able to leverage lower-cost storage for older content and large files
Comparable migration solutions (Off the shelf) were priced per TB, by designing and developing this engine we were able to deliver this 40% saving on the initial investment
It also ensured that our clients’ global offices and users were compliant with security policies around data storage
The client is now able to leverage their investment in Sharepoint and Microsoft teams for collaboration. From an end-user perspective, assets were easier to access and it was easier to store and secure newly created assets.
Ready to Connect with Us?
Contact us
