New Post has been published on o365info.com
New Post has been published on http://o365info.com/mail-migration-office-365-factors-impact-mail-migration-throughput-part-24/
Mail migration to Office 365 | Factors that impact mail Migration performance | Part 2/4
When we implement a project of mail migration to the cloud (Exchange Online), one of the most impotent measurement values that we use is the “throughput value” of the mail migration. In simple words, we want and need to know, how much time it will take us to copy data (mailbox content) from the organization to the “cloud” (Exchange Online).
This “value” is important to us because based on this value, we can provide an estimation for How long it will take to finish the project, what are the costs that involved in the project (a longer project will cost more than shorter time project) and so on
However, before we can provide the required estimation, we need to know what are the “entity” or the factors that are involved in the mail migration process and how this factors impact the mail migration process.
Note – In the next article (Office 365 mail migration | Optimizing the throughput | Part 3/4 ) of this series, we will review these factors and the options that are available for use for improving and optimize these factors.
A quick reference for the article series
Office 365 Mail migration | Migration methods | Part 1/4 The first article on this series includes a description of the different mail migration options that are available for us for migrating existing mail infrastructure to Office 365 (Exchange Online) and focus on the features and the characters of the different mail migration methods that relate to the mail migration. Mail migration to Office 365| Optimizing the Mail Migration throughput | Part 3/4 The third article recommendation and best practices for improving and optimize the performance of the mail migration throughput. Mail migration to Office 365 | Measure and estimate Mail Migration throughputs | Part 4/4 The last article in this series is dedicated to information about the “expected mail migration throughput” and includes a nice Excel based utility (Office 365 – Multiple and Single Mailbox Migration throughput calculator) that will help us to provide an estimation for the expected data transfer rate. Based on this information we can provide a reasonable “end date” for the completion of the mail migration project.
General mail migration questions
Before we begin, I would like to ask would like to ask two questions (that will be answered in the current article and in the next articles in this series). Q1: Is there any kind of a formal answer about a “fixed number” that we can use, when we need to estimate the throughput of mail migration (the transfer rate of data)? A1: Yes, it is true that there is not much information on the subject but, there is a Microsoft official article that provides “numbers” and estimation for a mail migration throughput.
Q2: Why do we need to use an “estimation” or “average range of values” for the mail migration throughput? Why we cannot use a simple a mathematical formula for calculating the required time for transfer information from point A to point B? A2: The answer is that – when we migrate a mailbox from point A to point B, there are many components that are involved in the process. Each of these “components” have a specific character that impact the result of the mail migration throughput.
When we want to migrate the content of a user’s mailbox to the cloud (Point B), we need to get through many elements. Each of these “elements” have the ability to “hold” the process or just let the data get through.
Each of the “elements” or the factor have resource’s limitations that affect the performance. For example, in case that the “element” is the local Exchange On-Premise server, the mail migration throughput is depended or based on the Exchange On-Premise server resources such as: CPU, RAM, Hard disk and so on.
Additional factors are the type of the mail migration that we use, limitation of the communications line, Throttling policy, the size of the mailbox, the number of mail items and much more.
Factors that impact mail migration throughput
In the following section, we will review some of the factors that impact the throughputs (transfer rate) of the mail migration. Additional and important information about the performance factors that is not included in the current articles, appears in the article: Exchange Online Migration Performance and Best Practices
Network infrastructure
1. Communication line
One of the most important factors that relates directly to the result of the mail migration transfer rate is obvious, the existing communication line.
1.1 The “speed” (The bandwidth) of the organizational communication line Regarding the rate or the speed of the communication line, don’t forget that many times the communication line technology is providing different transfer rate for “upload’” verses “download” (asynchronies). When we migrate existing mail infrastructure to the cloud, the operation is considered as “upload” (moving data from the local network to the public network) and many times, the transfer rate for “upload” is slower or less than the download rate.
1.2 The free or available percentage of the organization communication line A very important factor is the “load” that exists on the communication line or the “free percentage bandwidth” of the communication line. For example, it’s not enough to say that the existing communication line bandwidth is 20MB, but instead, the real question is how much percentages of the bandwidth are free or available for the task of the mail migration project.
1.3 Working hours and weekends The additional element that affects the communication line “free or available bandwidth” is – the factor of working hours and weekends. The basic assumption is that during the “working hours,” the amount of the “free or available bandwidth” of the communication line, is lower than the rate that we can get after the working hours and especially at weekends.
2. Geographic location of the organization’s network
Office 365 data centers and mail infrastructure are physically located in different geographical location worldwide. There are three “main point” in which Office 365 Data centers are located. Here is a quotation from an article that I have found:
The way I’ve always answered this question is that Microsoft divides the world into three regions: North & South America, Europe & Africa, and Asia & Australia. We have an undisclosed number of datacenters in each of those regions, and we select a primary datacenter within the region where the organization’s headquarters address is located
[Source of information: Where is my Office 365 data located? ]
Additional important question that we can ask is: in case that our organization has multiple sites that are located worldwide, can we assume that each location is configured to access the nearest Office 365 data center?
The answer is: “No”. All the organizational resources such as user’s mailboxes will be located in one data center and will not be “distributed” between the various Office 365 data centers worldwide.
Microsoft has a regionalized data center strategy. The customer’s country or region, which the customer’s administrator inputs during the initial setup of the services, determines the primary storage location for that customer’s data.
For specific details about where are the Office 365 data center geographically located, you can read more information in the article: Where is my data?
The point of this information is that- there is an impact for the geographic “distance” between the organization’s network and the Exchange Online data center. (I will provide more information about how to check the quality of the communication and provide a recommendation in the last article of this serious Office 365 mail migration | Optimizing the throughput | Part 3/4 )
3. Network devices
Network devices such as a Firewall or Proxy servers can impact the results of the throughput (data transfer rate) in a couple of ways:
The load factor- The load or the performance of these devices: for example, in case that the existing Firewall or Proxy server has very high utilization, this could lead to a reduction of the mail migration throughput.
Number of session restrictions- Another factor could be: the feature of Firewall and especially Proxy server that limits or restricts the number of sessions in the communication process.
For example: the process of mail migration could create tens or hundreds of sessions, between the Exchange on-Premises server and Exchange Online. In case that the Firewall or the Proxy server is configured with a maximum number of session’s value, and the mail migration process reaches this limit, the throughput of the mail migration is reduced.
Exchange On-Premise infrastructure
1. Data source (Exchange Server) – Performance of the local Exchange On-Premise servers.
Mail migration is based on the concept in which the mailbox data is copied or moved to the cloud (Exchange Online). In a scenario of multiple mailboxes, the amount of the data that is transferred can easily become huge and the operation of accessing to each of the mail server mailboxes and copy the data creates a considerable load on the Exchange on-Premises server.
In a scenario of Hybrid configuration, the server load could impact the Exchange on-Premises server mailbox server who hosts the user’s mailboxes or to the Exchange on-Premises server hybrid server, which serve as a router between the internal Exchange on-Premises servers and the cloud.
2. Exchange sites
Mid and large organization, will usually have more than one Exchange site. The subject of Exchange sites can impact the data transfer rate of the mail migration from two perspectives:
The geographic location of the Exchange site versus the geographic location of the Exchange Online data center. In case that the Exchange site is “closer” to the Exchange Online data center, we would probably have better results.
Performing mail migration by using multiple Exchange sites In case that the mail migration project is implemented by performing the mail migration from multiple Exchange sites, we can migrate more mailboxes in a specific time window and the result of the mail migration throughput is significantly improved.
3. Hybrid migration & MRSProxy
When using the option of Hybrid migration, the Exchange on-Premises server component that is responsible for – implementing and managing the move mailbox operations is the MRSProxy. The task of moving mailbox from source server to the destination server described as: MRSProxy connections. In Exchange on-Premises server, the default value for the MRSProxy for the maximum number of connections is 100. In case that we want to increase this value, we can change the default value by using a PowerShell command (we will review the exact syntax in the Office 365 mail migration | Optimizing the throughput | Part 3/4 )
4. Exchange Mailbox charters
Most of the time we mention the term “mailbox,” we use this term in general. In the reality, the term “mailbox” is used for describing many different types of mailboxes such as:
Small-sized mailbox with a small amount of mail items
Average mailbox with the average number of mail items
Small-sized mail box with a large number of mail items
Large-sized mailbox that usually characterizes as a mailbox with large-size + large amount of mail items.
So what is my point?
My point is – that most of the time we relate only to the “mailbox size” and we understand that logically, it will take more time to migrate Large-sized mailbox versus a Small-sized mailbox. An important factor that is not mentioned most of the time that has a tremendous impact on the mail migration throughput is: the number of mail items that exists in the user mailbox.
Common migration performance factors One 4GB mailbox with 400 items, each with 10 megabytes (MB) of attachments, will migrate faster than one 4-GB mailbox with 100,000 smaller items.
[Data source: Exchange Online Migration Performance and Best Practices ]
Number of mail items and third party mail migration tools In the article: Exchange Online Migration Performance and Best Practices, We can find a data table that focuses on the mail migration throughput results in a scenario on a mailbox with an average amount of mail items versus a mailbox with a lot or above average of mail items when using an RPC mail migration.
The little “catch” is that the data table relates to a result of mail migration that was created when using a third party mail migration tools.
(I must admit that I’m wondering why the article includes information about third party mail migration tools and doesn’t include a related information to the Office 365 built-in migration options)
Although the table compares the migration throughput in a scenario of third party mail migration products think that we can use the information as a derivative for all of Office 365 mail migration options such as: Cutover and Stage migration because, these methods are also based on an RPC connection that is created between the Exchange on-Premises server and the Exchange Online server.
So what is the data telling us? If we take a closer look at the data that appear in the table, we can observe very inserting issue: when we migrate an Exchange mailbox (small size mailbox in our scenario) that have multiple numbers of mail items the process will create a huge amount of RPC transactions.
To emphasize the conclusion form the data that appear in the table, I have created the following diagram.
In this scenario, we can see information about the mail migration throughput for two mailboxes. One mailbox is quite small (size of 249 MB), and the other mailbox is larger (377 MB). The interesting fact is that – the time that we need to move the smaller mailbox is significantly longer (11~ hours for the smaller mailbox verse 4.5~ hours for the larger mailbox).
The answer for this “strange result” is that the smaller mailbox, includes 13,000~ mail items versus the larger mailbox that includes 4,100~ mail items. The conclusion is that the factor of the number of mail items in a mailbox, has a huge impact of the mail migration throughput.
5. Exchange on-Premises server throttling policy
Additional factor that we can relate to is: the Exchange on-Premises throttling policy. In case that there is an implementation of: Exchange on-Premises throttling policy, the throttling policy could impact the performance of the mail migration process. You can read additional information in the following articles:
Throttling Policy Associations in Exchange 2010 SP1
Exchange Online Throttling and Limits FAQ
Office 365\Exchange Online factors
1. Throttling policy\Throttling mechanism
The term “Throttling policy” is a term that describes a similar concept as the term QOS (Quality Of Service). A Throttling policy is basically a predefined restriction or a limitation which is enforced by the administrator on a specific system.
For example: in the Exchange server environment, we can use Throttling policy for: limiting the number of concurrent PowerShell remote sessions, the maximum amount of data that can be accepted or sent and so on. The question that relates to our scenario of mail migration to Office 365 is: Does Office 365 use Throttling policy that restricts the amount of data that can be “transferred” from the on-Premises mail server to Exchange Online? And the answer is Yes and No.
In Office 365, there is an implementation of three types of throttling policy that relates to male migrations:
Office 365 migration-service throttling
Office 365 user throttling
Resource health-base throttling
Office 365 migration-service throttling
The migration-service throttling define the maximum number of concurrent mailbox move to the Exchange Online server. For example:
The default value for concurrent mailboxes “moves” when using Cutover, Stage or Hybrid migration is 20.
The default value for concurrent mailboxes “moves” when using IMAP migration is 4.
Q1: can I change this default value? A1: Yes, very easily from the GUI interface or by using a PowerShell command.
Q2: can I assume that the obvious conclusion is: if I double the default value from 20 concurrent moves to 40 concurrent moves, I will double the mail migration throughput? A1: In theory the answer is: “yes.” We use the term “In theory,” because when we increase the number of maximum mailbox move on the Exchange Online server side (and in the Exchange on-Premises server side), the results could be: overwhelming of the Exchange on-Premises server because, the task of “move mailbox” is consuming resources from the server who “deliver” the mailboxes (Exchange on-Premises server) and the server who “accept” the mailboxes (Exchange Online).
My point is that we can assume that Exchange Online has an unlimited amount of resources, but we cannot apply the same conclusion to the Exchange on-Premises server because the basic assumption is that the Exchange on-Premises server has limited hardware resources.
Office 365 user throttling
The term “user throttling” describe an Office 365 Throttling policy, which is implemented or “enforced” in two scenarios: scenario 1: when using the option of “PST Migration”Scenario 2: when using Third party MAPI Migration (client-uploading migration method. That is implemented by using RPC over HTTPS).
Q1: What is the reason for using the user throttling policy? A1: I could not find a formal answer for this question, but I can assume that the reason for implementing the user throttling policy when using PST migration (client-uploading migration) is that this type of migration, is not very effective and cause a high load on the side that “export the data” (The client side) and the side the “accept” the data (Exchange Online).
The Office 365 user throttling policy was designed for “protecting” the Exchange Online from a scenario in which we implement tens or hundreds of PST migration and by doing so, “overwhelming” the Exchange Online servers.
User throttling affects most third-party migration tools and the client-uploading migration method. These migration methods use client access protocols, such as RPC over HTTP, to migrate mailbox data to Exchange Online mailboxes. These tools are usually used to migrate data from platforms such as IBM Lotus Domino and Novell GroupWise. User throttling is the most restrictive throttling method in Office 365. Because user throttling is set up to work against an individual end user, any application-level usage will easily exceed the throttling policy and result in slower data migration.
[Data source: Exchange Online Migration Performance and Best Practices ]
Resource health-base throttling
The Resource health-base throttling was created to protect the health of the Exchange Online server in special scenarios. The use of the Resource health-base throttling will be implanted only on special occasions, when the monitoring system identifies a special load or specific problem that relates to the Exchange Online server infrastructure.
All migration methods are subject to the governance of availability throttling, but Office 365 service throttling doesn’t affect Office 365 migrations as much as the other types of throttling described in the previous sections. Resource health-based throttling is the least aggressive throttling method and occurs only when there is a service availability issue that affects end users and critical service operations.
[Data source: Exchange Online Migration Performance and Best Practices ]
Summery
The following screenshot was taken from the Microsoft article: Exchange Online Migration Performance and Best Practices
The table describes the “matrix” of the Office 365 Throttling policies and in which of the migration scenario a specific Throttling policy is implemented.
In the following diagram, we can see a different representation of the information that appears in the table: we can see that the Resource health-base throttling is implemented for all the mail migration types without exception. When “server to server migration” such as: Stage or Cutover migration, the Office 365 migration-service throttling is used to define a maximum number of concurrent mailbox move.
A quick reference for the article series
Office 365 Mail migration | Migration methods | Part 1/4 The first article on this series includes a description of the different mail migration options that are available for us for migrating existing mail infrastructure to Office 365 (Exchange Online) and focus on the features and the characters of the different mail migration methods that relate to the mail migration. Mail migration to Office 365| Optimizing the Mail Migration throughput | Part 3/4 The third article recommendation and best practices for improving and optimize the performance of the mail migration throughput. Mail migration to Office 365 | Measure and estimate Mail Migration throughputs | Part 4/4 The last article in this series is dedicated to information about the “expected mail migration throughput” and includes a nice Excel based utility (Office 365 – Multiple and Single Mailbox Migration throughput calculator) that will help us to provide an estimation for the expected data transfer rate. Based on this information we can provide a reasonable “end date” for the completion of the mail migration project.
Additional reading
Exchange Online Migration Performance and Best Practices
Exchange Online Throttling and Limits FAQ
EWS throttling in Exchange
Now it’s Your Turn! We really want to know what you think about the article