Storage Assessment Beginners Guide
We have come across customers who are looking for a starting point or a guideline on assessing their on-premises storage for migration or optimizations. This document has been created as a beginners guide to start the process.
Storage Assessment Guideline document
Contents
Objective
Task List for Storage Assessment
Capacity Planning in Existing Infrastructure
Network Considerations
Target Storage Types in Azure
Target Data Classification and Categorization
Testing and Validation
Timeline and Milestones
Cost Estimation
Pre Migration Risks
Post Migration Risks
Conclusion
Objective
The primary objective is to conduct a comprehensive assessment of the current on-premises storage solutions which are hosting data. This strategic initiative may be driven by upcoming hardware refresh or end of life, a need for optimization by getting rid of unwanted data, increase availability of storage services, improvement in disaster recovery capabilities, and reduce the overall Total Cost of Ownership (TCO)
Task List for Storage Assessment:
Task list to complete the storage assessment.
- Gather the existing landscape information for storage assessment and planning.
- Collect data using different tools or through manual discovery.
- Assessment result analysis for identified data
- Data Preparation.
- Regulatory and Compliance checks on data hosting.
- Solution Architecture and Design.
- Design approval and implementation planning.
- Preparing Test Cases for Data checks
- Testing the migration.
- Calculate TCO
- Seek Approval for Production Deployment
Capacity Planning in Existing Infrastructure
Tools for Assessment
Companies providing storage solutions in the market have their own storage assessment tools
such as HP, Dell EMC, IBM. Check with the customer on the currently available subscriptions. When solution assessment tools were not available manual assessment strategy can be used.
Understanding Current Usage
Using the applications, databases and data centers in scope understand current usage trends across the estate. Sample data provided below.
|
Workload |
No of Virtual |
Allocated (TB) B |
Data to be deleted (TB) |
Onprem (TB) D |
Data move (TB) |
Total Required storage |
Storage reduction from current allocated space (%) G= (C/F*100) |
Data Migration To cloud |
|
Application 1 |
|
|
|
|
|
|
|
|
|
Application 2 |
|
|
|
|
|
|
|
|
|
Application 3 |
|
|
|
|
|
|
|
|
|
Application 4 |
|
|
|
|
|
|
|
|
Storage information collected at Data Centre Level
|
DC Location |
Storage Count in Locations A |
Total Size (TB) |
Allocated Capacity (TB) |
Used Capacity (TB) |
Shared Disk assigned to multiple VMs (TB) |
Data marked for deletion from Used Capacity (TB) |
Data to be migrated to Cloud (TB) |
Data to be kept OnPrem (TB) |
|
DC 1 |
|
|
|
|
|
|
|
|
|
DC 2 |
|
|
|
|
|
|
|
|
|
DC 3 |
|
|
|
|
|
|
|
|
Forecast Future Demand:
Analyze historical data to forecast future demand accurately. Consider factors such as business growth, seasonal variations, and upcoming projects that may impact resource requirements.
Come up with future growth estimate considering some percentage for year-on-year growth in data. This will help estimate growth in storage on cloud.
Data Migration Strategy:
There are two main methods to migrate data from OnPremise to Azure Cloud.
- Online mode
Data is transferred across the network using a VPN or an ExpressRoute.
Detailed information on VPN and Express Route can be found on the provided links.
- Offline mode
Data is migrated by shipping data using services like Azure databox, data disk and data box heavy.
Azure data box features and specifications
Data Disk Features and Specifications
Data Box heavy Features and Specifications
Since Data box was the preferred choice. This migration describes databox in further detail.
Azure Data Box Service:
Azure Data Box will serve as the chosen solution due to its hybrid capabilities of seamless import of on-premises data into Azure in offline mode. The overall approach involves transferring data to a Microsoft-provided storage device of 80 TB usable capacity, followed by secure shipment to Azure Data Centers for final upload to the linked storage accounts. The decision to opt for Azure Data Box was taken due to the following considerations:
- Given the substantial size of Archive and Cold datasets, coupled with a significant number of files, Azure Data Box will help in off the network transfer of data which clogging any network bandwidth.
- Azure Data Box ensures a secure and compliant migration process due to several built-in features such as passkey, rugged casing, logging mechanisms.
- Since the data to be migrated will be infrequently accessed there is no immediate need for data to be available immediately for an application.
Once the initial migration is over, incremental data can be migrated to Azure over the network pipe in off peak hours using command line tools such as AZ Copy
*Source of image is internet
The following diagram indicates the flow of data through the Azure Data Box solution from on-premises to Azure and the various security features in place as the data flows through the solution. This flow is for an import order for your Data Box.
Azure Data Box specifications
- Azure Data Box Size - 80 TB Usable
- Data Protection / Security - AES 256-bit BitLocker Encryption Supported.
- Network Interface
- 2 X 1-GbE interface – Management
- 2 X 10-GbE interface - DATA 1, DATA 2 (Both are for data, can be configured as dynamic (default) or static.
- Data Box Supported Operating System - Microsoft Windows Server 2016 /2019 or later.
- File Type Supported
- SMB 3.0
- NFS All Version Including 4.1
Diagram source Microsoft Azure Data Box security overview | Microsoft Learn
Network Considerations
Connect on-premises infrastructure and Azure Cloud using ExpressRoute on VPN gateway.
There are several important network factors to consider ensuring a secure, reliable, and optimized connection. ExpressRoute provides a dedicated private connection between your on-premises data center and Azure, bypassing the public internet. VPN will create encrypted tunnels, but data will traverse through the internet.
Bandwidth Planning:
For online transfers calculate the bandwidth capacity based on data generated and frequency of transfer. The data generated can be in the form of files. Frequency of transfer can be daily, weekly or monthly based on t business decisions.
You may use the bandwidth calculator free online tool to estimate the targeting bandwidth.
Security and Compliance
Online mode data transfer security
In Online mode data will be transferred either through express route or through VPN.
Express Route provides private dedicated pipe between the source and destination. Traffic can be further filtered using firewalls and NSGs.
VPN provides an encrypted tunnel over the internet for data movement.
If moving data over the internet is not approved by security teams, then express route is the preferred option for dedicated private connectivity.
Security Baseline for ExpressRoute
Offline mode data transfer security
Data box device protection has several features such as passkey, rugged casing, tampering detection, databox specific software, locked boot state, trusted platform module. This ensures only authorized entities can view and access the data.
Data box-related events can be logged for inspection on connections, import, export operations.
Target Storage Types in Azure
Azure offers many storage solution types as referenced in the link below:
Typical migration from OnPrem scenarios involve below storage types.
Azure Blob Storage
Allows unstructured data to be stored and accessed at a massive scale in block blobs.
Available in hot, cool and archive tier.
Azure File Storage
Fully managed cloud file shares that you can access from anywhere using NFS or SMB.
Azure Elastic SAN
Fully integrated solution that simplifies deploying, scaling, managing, and configuring a SAN.
Azure managed Disks
Block-level storage volumes for Azure VMs
Target Data Classification and Categorization:
Based on the assessment done with different stakeholders of identified data below data classification has been done for the data assets:
|
Classification of Data for Cloud Movement |
|||
|
DC Location |
Application |
Data Type on Cloud |
Data size to be moved to Cloud (TB) |
|
<DC Location Name> |
< Name of Application> |
<BLOB, FILE, SAN, HOT, COOL, ARCHIVE> |
Size of data |
Testing and Validation:
The plan covers various aspects such as data integrity checks, performance testing, user acceptance testing and contingency plans.
-
Data Validation:
- Manually compare sample data between the source and destination.
- Automate validation checks using scripts for larger data files.
- Verify metadata of transferred files through visual inspection or automated methods.
-
Access and Retrieval Testing:
- Manually access the transferred files to inspect and compare contents.
- Download files to check for any corruption.
- Monitor the time it takes to upload and download specific files.
-
Search Validation:
- Manually test search features to locate specific items.
-
Testing with Users:
- Test the data transferred through the application.
- Mimic real-time use activities by testing all use cases.
-
Security and Access Control Validation:
- Manually verify user access controls by attempting to access restricted data.
- Review audit logs to ensure correct security event logging.
- Simulate unauthorized access attempts to validate permissions.
-
Regulatory Compliance Checks:
- Consult with legal or compliance experts to validate adherence.
- Manually review relevant regulations and documentation.
- Explore using Azure Policy to automate checks against regulatory requirements.
-
Error Handling and Logging:
- Simulate error scenarios and observe system response.
- Manually review error logs for completeness and detail.
- Test the system’s ability to recover from errors and log relevant information.
-
Documentation Review:
- Prepare documentation for the data transferred.
- Verify that documentation aligns with the latest state of transfer.
-
Stakeholder Communication:
- Set up regular cadence and project management meetings for migration.
- Share updates on important milestones throughout the end-to-end migration.
-
Final Approval and Transition to Production:
- Obtain formal sign-off from stakeholders following the internal approval process.
- Plan for the production rollout.
Timeline and Milestones:
Insert your detailed project plan for the storage assessment here indicating all the key milestones
Cost Estimation:
The link from Microsoft provides an estimation of pricing for critical products like Storage Account, Express Route, VPN Gateway, Azure data box and their SKU types.
Total Projected Savings (TCO)
Savings Projection as per Assessment (Date)
|
|
|
Calculation of Savings |
|
|
|
|
Description |
Total Assessed Storage A |
Storage retained OnPrem |
Target Storage Type on Cloud |
Target Cloud Storage Cost |
Total storage Released D |
|
Storage Capacity (TB) |
|
|
|
|
|
|
Price per Unit( INR) |
|
|
|
|
|
|
Annual Cost (INR) |
|
|
|
|
|
|
Savings |
A-(B+C) |
|
|
|
|
Pre Migration Risks
Migration Readiness : Evaluate the complexity of the migration process and its impact to business.
Cost : Evaluate the costs of migration, including unexpected expenses due to delays or issues.
Business Continuity Planning: Develop and test business continuity plans to ensure quick recovery in case of unexpected events.
Communication to Stakeholders: Communicate changes to end-users to minimize disruptions during and after migration.
Identity and Access Management: Identify risks related to data security during migration due to unauthorized access or data breaches.
Dependency Mapping: Recognize the risks associated with integrating on-premises systems with Azure services.
Downtime and Service Disruptions: Assess the risk of downtime during migration and loss of service.
Data Sensitivity: Assess the sensitivity of data and classify it accordingly to determine appropriate security measures
Backup and Recovery: Develop a backup and recovery strategy to overcome challenges during migration.
Phased Migration: Consider a phased approach to migrating data. Once the first phase is success migrate the next phase.
Data Encryption: Plan for data encryption at rest or in transit.
During and Post migration Risks
Performance Testing: Conduct performance testing to ensure that Azure resources meet the required performance benchmarks.
Training: Provide training to IT staff and end-users on Azure services, security best practices, and compliance requirements.
Monitoring: Use Azure Monitor to continuously track performance, detect issues, and optimize resources.
Governance: Use Azure Policy to enforce organizational standards and compliance requirements.
Identity and Access Management (IAM): Implement strong IAM policies to control access to Azure resources.
Data Encryption: Implement encryption mechanisms for data both in transit and at rest to enhance security.
Threat Monitoring: Leverage Microsoft Defender for Cloud for continuous monitoring, threat detection, and security recommendations.
Conclusion:
Communicate assessment results and TCO to higher management.