This is the year where migrating to Office 365 is no longer impossible for those with a lot of data to migrate. Don’t get me wrong; moving to Office 365 was always possible with the help of tools like Sharegate. The reality however, is that Microsoft quickly began throttling migrations as they’re using the same APIs used by regular services and thus, if left uncheck, would slow down multiple tenants at a time.
Today, Microsoft unveiled a new Office 365 Migration API that boosts the speed of migration of files by leveraging Azure. The resulting speed is insane.
High level overview of the Office 365 Migration API
If you just want to quickly understand how it works without diving into the technicalities of the migration, well you’re lucky, as it’s fairly straightforward.
Essentially, you export your content into a migration package that gets sent to Azure Storage. When the timer job runs in Azure, they’ll take that package and put the content in your SharePoint or OneDrive for Business based on your package settings.
Do understand that I have over-simplified it to give you that high level overview of the Office 365 Migration API. It’s in fact, a little more complicated to do than it seems, but the results are incredible… we are now talking multiple GBs per hour!
Add this infographic to your site by copying this code:
Other Articles to Learn Office 365
What is Office 365 – The Grand Tour [Webinar Recording] 
- Co-Authoring Documents
- Yammer
- Sharing & External Users
- Video Portals
- Office 365 Migration API
The “geekier” summary:
Before you begin, you must have an Azure subscription, as you will need to use an Azure Storage Account and incur additional costs. The costs for storage are minimal, but the requirement is there nonetheless.
I strongly recommend that you create a new Storage Account just for the migration to get a separate Azure Storage Access Key and limit any potential problems to this migration. Also, when you create a Storage Account, you can choose where it’ll physically be located. Great news for those large-scale migrations to Office 365 in different regions.
Once this is ready, you may begin.
- Create a storage container in Azure to store the content to be migrated
- Upload your content to the storage container for the content
- Take a Snapshot of the BLOB in the container
- Create a new storage container in the same Azure storage account for the migration package manifest files
- Generate all required manifest files following the expected schema
- Upload the manifest files to your Azure storage container created for these files
- Take a Snapshot of each file uploaded in the manifest container on Azure.
- Call a CSOM method called CreateMigrationJob to tell Microsoft to come pick up your package and content and put your migration job in queue.
- You can get real-time updates by specifying an Azure Queue that will receive event messages.
- A log file is deposited in the manifest storage container once the entire operation is done letting you know how it went.
The most difficult part is to actually create the manifest files that tell the migration job how to import your content. Things like the ID, the metadata (though not all is supported) and any other information.
Important: This new Office 365 Migration API does not migrate the structure nor does it create it. So you’ll have create it manually before or use 3rd party tools like Sharegate that will leverage this new API, but also migrate everything else using existing APIs.
Familiarities with the Content Migration Deployment APIs from previous SharePoint
For those of you that have been around SharePoint since 2007, I’m sorry. Actually, you can’t count me in those “lucky” SharePoint people and it’s why it’s easier for me to understand this Office 365 Migration API. It’s actually not really new and uses the good old stsam import and export commands and more recently Export-SPWeb and Import-SPWeb.
You’ll find the information on creating these uncompressed migration packages on MSDN and frankly, it works exactly the same way as it used to. The difference is that now, you send this package to Azure in the method described above and Microsoft will take care of importing it to your Office 365 SharePoint or OneDrive for Business using the information in your manifest file.
Let me give you an example:
If you plan to migrate your File Shares or On-Premises SharePoint to Office 365 using this new migration API, here’s how it would work. I’m going to assume you already have an Azure Subscription and that within it, a new Storage Account. I’ll also assume that you have it ready with the two required containers, one for the content and one for the manifest files.
- Somehow find a way to upload your content into the Azure Storage Container. You’ll have to use the custom code with their SDK or PowerShell. Another way is to find 3rd party tools that will do it for you. Then take a snapshot, otherwise it won’t work.
- Create the content migration package with all the xml files required and detailed in the MSDN link above: ExportSettings.xml, LookupListMap.xml, Manifest.xml, Requirements.xml, RootObjectMap.xml, SystemData.xml and UserGroup.xmlThe most important file is the manifest that references each object you want to import and tells the story on how to migrate it to the destination. Once again, you’ll need to snapshot these in Azure.
- Call the referenced CSOM method CreateMigrationJob with the parameters telling it where to get the Content Container and the Manifests.
- Wait for the timer to pass and pick up your migration job and import it into your Office 365.
What you should know about the new Office 365 Migration API
Besides it being a long and tedious effort to create these packages, and especially the manifests file, there are a few things you should know for your migration package. One we mentioned already, there’s no easy way to import your content in the Azure Storage Containers without coding or using tools.
The size of the package
Each package sent should stay relatively small or below 25GB to ensure a blazing fast migration and limit potential problems.
The file size of content being migrated
In its first release, this new migration pipeline supports files no larger than 2GB, regardless of the file size limit in SharePoint or OneDrive for Business. This is expected to change in the future.
Only use un-compressed packages
For those with experience with the Import/Export commands, you may want to create a .cmp file. However, this is not supported.
Support for content versioning and unique identifiers in lists or libraries
This is huge! Not only does this migration API allow you to keep version history, it also allows you to keep the ID value of your list items and documents respectively.
Delete your storage container after migration
Since you’re actually paying to store your data in Azure, it’s important that after your migration is successful, you delete what is no longer needed. It’s only 2 to 5 cents per GB, but still.
The results are insane speeds when migrating to Office 365
How do I know so much about this new Office 365 Migration API on the first day of its release? Sharegate has been working closely with the folks over at Microsoft, and together we were able to get Sharegate with its new “Insane Mode” for blazing fast speeds when migrating to Office 365.
It’ll help you avoid the complicated steps, but also allows you to migrate your existing structure and manipulate it while doing so. The new migration pipeline is strictly for content and security objects. Sharegate will help you assign metadata on the fly and even let you use Excel to do it so you don’t have to explain manifest.xml to anyone.
Overall, this is great news for Microsoft and will definitely help more organizations move to the cloud with Office 365, SharePoint Online and OneDrive for Business. What used to take days will now be done in mere hours.
