For a long time, the biggest challenge with Office 365 was to move your content there. Tools were the only way to migrate to Office 365 and even with them, the speed limited many larger migrations and for some, even stopped it.
To solve this, Microsoft introduced the Office 365 migration API and more recently, the Office 365 Import Service to help you leverage it. I’ve been getting questions on the Office 365 Import Service and its limitations, so I decided to dive deeper.
My initial thoughts: it’s clear that it’s targeting developers and very simple migrations with a few destinations (few Document Libraries). The steps I had to go through to get a simple list migrated, or documents in libraries using this method are too much of an effort.
You could say I’m biased since I work at Sharegate, but I invite you to read on and do your own tests to see for yourself.
Basics of the Office 365 Migration API Pipeline
Since this new Office 365 Import Service leverages the API heavily, I’d recommend you first read up on What is the Office 365 Migration API. There are things supported and others that aren’t with this new migration pipeline and they’re important to know before you continue.
Only content in lists and libraries are supported, all structure must already exist.
If we try summarizing it, you’ll essentially use the Content Deployment packages that have been available in SharePoint for many years now. You create a package that contains XML files with the information needed then import the content somewhere else.
You then upload these packages to an Azure Storage Account and create an import job to say that it’s ready to be picked up and moved to your Office 365. This way, you can upload the content very quickly to the cloud in Azure and then the migration import is done almost locally in their datacenters.
Discovering the Office 365 Import Service for My Migration
It’s advertised that I can use this new Office 365 Import Service to migrate my file shares, SharePoint list items and documents from my On-Premises to the cloud. I thought I’d test it out.
You’ll notice a new Import menu in your tenant Admin center.
This will prompt a window with a step by step we should follow.
The first step with the TechNet guide is the most important one, because most of the work is done manually with PowerShell. The documentation is well written and helps us get through it without wondering what the next step is. However, there are some things that didn’t work for me.
When I tried to import a SharePoint list, it’s a guaranteed error and no log available to figure out what happened. But I’ll talk about that further in this article.
Continuing the guide to import data to Office 365, I needed my keys to connect to the Azure storage I would upload the packages to.
I know it says to download the Azure AzCopy tool, but I have no idea why since we don’t use it at all in the steps. You can skip that download, but you’ll need the codeplex tool Azure Storage Explorer if you plan on migrating any SharePoint lists…
Migrating from File Shares to SharePoint Using Office 365 Import Service
Probably the best fit for the Office 365 Import is with a File Shares migration to SharePoint. It introduced a new PowerShell commandlet in the SharePoint Online Management Shell, which you have to download if it wasn’t done already.
One of the new commandlets is for you to create a Package from a file share source. So I followed the steps and first connected to my Office 365 destination.
Then proceeded to create a package with a folder on my desktop.
Essentially, it looks at the source and creates a package in the folder of my choosing. There are additional parameters available to keep the permissions as well.
The thing about this step is that it basically creates the package for you, which is great but not enough. Whether you’re migrating from Files Shares or SharePoint, you’ll need to enter all the GUIDS and URLs of the destination to replace what’s in these XML files.
Look at it this way, the method used by this Office 365 Import Service to do a migration was originally meant as an Export/Import feature in SharePoint. So when you export from SharePoint or with this new commandlet for File Shares, it creates a package with the information as is.
If you try to import this somewhere else, it won’t work because it won’t find the same document library to put the files back in. It’s even worse coming from File Share, since there was none.
The next commandlet allows us to connect to the desired destination and go back to our package files to rewrite them with the correct destination information. That’s why your structure at the destination is required, you must create all Sites, Lists and Libraries before you start using the Office 365 Import.
Here are some of the actions that were done to my XML files in my original package.
This is the magic of the Office 365 Import Service.
You’ll notice that it goes through and replaces IDs and URLs in these XML files to work with that Document Library mentioned in the PowerShell command.
At this point, you can go in the XML files to make any modifications you’d like to the data before it gets uploaded and imported to Office 365. I already detailed them out when I covered the Migration API and Microsoft has more on their TechNet pages as well.
Our last step before going back to the Office 365 Import interface is to upload the package and associated files to Azure.
-Note: Make sure that you create a new folder in Azure for the package container name parameter each time if you plan to do multiple migrations with different destinations.
I went back to the Office 365 Import interface in my Admin Center now that I was ready to start the migration job. There, I continued the steps and checked the boxes that asked if I had uploaded my files and had access to my mapping file.
Yeah I know, I didn’t know what this mapping file was but assumed I had access to it or that it got generated somewhere in the last step and had not seen it yet.
Well it turns out you have to create it yourself… And there isn’t much to do in there either.
Basically, it’s the file where you can tell the Office 365 Import job that you may have multiple packages to come and pick up and send them to different destinations. Site by Site, List by List and Library by Library.
I went back again to the Office 365 Import interface and named my migration task and pointed to my CSV mapping file. And the migration finally happened, allowing me to keep my data and modified dates as well as created date.
Migrating from SharePoint On-Premises to Office 365 with the Import Service
I won’t repeat the steps or write what’s in the TechNet guide again. Everything is relatively the same, the only difference is when creating the package with the content to be migrated to Office 365.
You’ll have to use the Export-SPWeb PowerShell command to get your list or library exported and intro an uncompressed package.
Once again, there are additional parameters to includes things like permissions and versioning on items and files as you export them in a package.
I then converted it so the XML files adapt to the destination.
And imported it to Azure Storage so that I can start the migration back in the Office 365 Import interface. Note that I had to change the Package Path as the guide on TechNet doesn’t give you the right directions for this.
Though these steps worked when I tested a migration of a SharePoint document library to Office 365, I couldn’t get this list imported. Even though there were no logs or errors, just an Import Failed message.
Failed to migrate a SharePoint list to Office 365 with the Import Service:
Thankfully, my awesome Sharegate colleagues who ran into the same issue building the Insane Mode in Sharegate, told me what the problem was. In the last step where we upload to Azure, we create a Files Container and a Package Container. But since it’s a SharePoint list, nothing is actually uploaded to the files container because it’s not needed.
Turns out, the Office 365 Import Service, or rather the migration API, requires a file to be in the files container. Wow! So I had to download a free tool that allows me to explore my Azure Storage account and upload a fake file in the container for this specific package. This means I have to do this for every single list I’d want to migrate. That fixed it.
What you should know before using the Office 365 Import Service
The migration API was targeted mainly to vendors like us at Sharegate to help Microsoft customers move their files to Office 365 a whole lot faster without slowing down the SharePoint servers.
Now, the Office 365 Import Service makes it a bit easier because you don’t have to manually modify the GUIDS, IDs and URLs but still targets a very developer oriented audience. The number of steps to migrate to just one destination such as a list of library is too long and complex.
And of course, it doesn’t do everything – here are some things you should look out for:
Everything needs to already exist
Create your Site Collections, Lists, Libraries as well as Content Types and Site Columns and anything in the list or library that will be needed. This is content migration and you can modify XML files package after package to edit the metadata and users.
Buggy Managed Metadata support
It looks like your Terms tagged to your documents or list items migrate successfully, but when you click on edit on your content it’s in red and not properly linked.
Fragile Lookup Columns support
If you migrate documents or list items that have a lookup column, you’ll have to make sure that the list or library they’re looking up exists at the destination AND that the IDs match exactly or it won’t work.
Modifying Metadata and User Mapping
Though you can tweak the Manifest and UserMapping XML files to get what you need, it’s not easy to do nor quick. In terms of metadata, there are many limitations: Internal Name needs to be exactly the same is one that will bite you a few times as users renamed their columns.
Free during preview, but will incur storage costs
Though it’s free to use the Azure Storage now, it’ll soon switch over and you’ll need to pay for the storage. It’s normally a minimal cost, but some partners received information that it may be 8$/GB. This has not yet been officially announced however, stay tuned.
Can use another storage container but will lose features
If you decide to use your own storage container, you’ll have to call the migration job, or import job rather, yourself. Basically, the Office 365 Import Service interface cannot be used. You’d only be leveraging the new PowerShell commands with the Migration API.
Not optimized for performance
Though it’ll be a lot faster than regular migrations, Microsoft have themselves recommended we create packages that are no larger than 2GB to launch multiple import jobs at the same time. This does a multi-threaded migration and thus boosts your migration time and is what Sharegate does behind the scenes.
The way the Office 365 Import is set up, it creates one big migration package.
If you’re migrating a relatively small File Share to very few document libraries in SharePoint, then the Office 365 Import Service is awesome. I’ll be honest though, for the rest of the scenarios, it seems quite long and complicated. I know… I work at Sharegate so you could argue that I wouldn’t want you to use this.
Best you can do is look at all the steps and make your own decision, you could of course test it out yourself. However, I don’t see this new service replacing the need for a third-party tool to simplify the migration of your SharePoint content over to Office 365. What do you think?
