How To Change Source Mac Address In Outgoing Packets Originating From A Hypver V Vcirtual Machine

For years, I'd never heard of this problem. Then, all of a sudden, I'thou seeing it everywhere. It'due south not easy to precisely outline a symptom tree for yous. Networked applications will behave oddly. Remote desktop sessions may skip or hang. Some network traffic will not laissez passer at all. Other traffic volition bear erratically. Rather than effort to give you a thorough symptom tree, we'll only describe the setup that can be addressed with the contents of this commodity: you're using Hyper-V with a tertiary-party network load balancer and experiencing network-related problems.

Acknowledgements

Before I ever encountered it, the problem was described to me by one my readers. Check out our Complete Guide to Hyper-5 Networking article and wait in the comments section for Jahn's input. I had a different experience, but that chat helped me reach a resolution much more speedily.

Problem Reproduction Instructions

The problem may appear under other conditions, but should always occur under these:

• The network adapters that host the Hyper-Five virtual switch are configured in a team
  • Load-balancing algorithm: Dynamic
  • Teaming mode: Switch Contained (likely occurs with switch-embedded teaming as well)
• Traffic to/from affected virtual machines passes through a tertiary-party load-balancer
  • Load balancer uses a MAC-based system for load balancing and source verification
    • Citrix Netscaler calls its feature "MAC based forwarding"
    • F5 load balancers call it "auto terminal hop"
  • The load balancer'southward "internal" IP address is on the aforementioned subnet as the virtual automobile's
• Sufficient traffic must exist exiting the virtual machine for Hyper-V to load residue some of it to a unlike physical adapter

I'll go into more than detail subsequently. This listing should aid yous decide if you're looking at an article that tin can aid you.

Resolution

Fixing the problem is very piece of cake, and tin can be done without downtime. I'll bear witness the options in preference order. I'll explain the impacting differences later.

Option i: Alter the Load-Balancing Algorithm

Your best bet is to change the load-balancing algorithm to "Hyper-V port". Y'all can change information technology in the lbfoadmin.exe graphical interface if your direction operating system is GUI-mode Windows Server. To alter information technology with PowerShell (assuming merely one team):

Become-NetLbfoTeam | Gear up-NetLbfoTeam -LoadBalancingAlgorithm HyperVPort

There volition be a brief interruption of networking while the change is made. It won't be every bit bad as the network bug that yous're already experiencing.

Option two: Modify the Teaming Mode

Your second pick is to modify your teaming mode. Information technology's more involved considering you lot'll also need to update your physical infrastructure to friction match. I've ever been able to practice that without downtime every bit long as I changed the physical switch first, but I can't hope the aforementioned for anyone else.

Decide if you want to utilize Static teaming or LACP teaming. Configure your physical switch accordingly.

Modify your Hyper-V host to use the aforementioned mode. If your Hyper-5 system's direction operating system is Windows Server GUI, you can use lbfoadmin.exe. To change it in PowerShell (assuming merely one squad):

Go-NetLbfoTeam | Set up-NetLbfoTeam -TeamingMode Static

or

Get-NetLbfoTeam | Set up-NetLbfoTeam -TeamingMode Lacp

In this context, it makes no difference whether you pick static or LACP. If you lot desire more data, read our commodity on the teaming modes.

Selection 3: Disable the Characteristic on the Load Balancer

You could tell the load balancer to cease trying to be clever. In full general, I would choose that choice last.

An Investigation of the Problem

And so, what's going on? What acquired all this? If you've got an environment that matches the one that I described, then you lot've unintentionally created the perfect weather for a storm.

Whose fault is it? In this case, I don't really think that it's off-white to assign error. Everyone involved is trying to make your network traffic go faster. They sometimes do that by playing fast and loose in that gray expanse between Ethernet and TCP/IP. We have lots of standards that govern each individually, merely not and then many that apply to the means that they tin collaborate. The problem arises because Microsoft is playing one game while your load balancer plays another. The games have different rules, and neither side is aware that another game is itinerant.

Traffic Leaving the Virtual Motorcar

We'll showtime on the Windows guest side (also applies to Linux). Your awarding inside your virtual machine wants to send some data to another computer. That goes something like this:

1. Application: "Network, ship this data to calculator world wide web.altaro.com on port 443".
2. Network: "DNS server, get me the IP for www.altaro.com"
3. Network: "IP layer, determine if the IP address for www.altaro.com is on the same subnet"
4. Network: "IP layer, send this packet to the gateway"
5. IP layer passes downward for packaging in an Ethernet frame
6. Ethernet layer transfers the frame

The role to understand: your application and your operating arrangement don't actually care about the Ethernet role. Whatever happens down there merely happens. Especially, information technology doesn't care at all nigh the source MAC.

Traffic Crossing the Hyper-V Virtual Switch

Considering this detail Ethernet frame is coming out of a Hyper-Five virtual machine, the first thing that it encounters is the Hyper-V virtual switch. In our scenario, the Hyper-Five virtual switch rests atop a team of network adapters. Equally y'all'll remember, that team is configured to utilise the Dynamic load balancing algorithm in Switch Independent way. The algorithm decides if load balancing can be practical. The teaming mode decides which pathway to utilize and if it needs to repackage the outbound frame.

Switch independent mode means that the physical switch doesn't know anything about a team. It just knows virtually two or more Ethernet endpoints connected in standard access mode. A port in that style tin "host" any number of MAC addresses;the physical switch's capability defines the limit. Yet, the same MAC addresscannot appear on multiple access ports simultaneously. Assuasive that would cause all sorts of bug.

So, if the squad wants to load remainder traffic coming out of a virtual car, it needs to ensure that the traffic has a source MAC address that won't crusade the concrete switch to panic. For traffic going out anything other than the primary adapter, it uses the MAC address of the physical adapter.

So, no matter how many physical adapters the team owns, one of two things will happen for each outbound frame:

• The team volition choose to employ the physical adapter that the virtual machine's network adapter is registered on. The Ethernet frame will travel as-is. That means that its source MAC accost will be exactly the same as the virtual network adapter's (significant, not repackaged)
• The team will cull to use an adapter other than the ane that the virtual motorcar'south network adapter is registered on. The Ethernet frame will be altered. The source MAC address volition exist replaced with the MAC accost of the concrete adapter

Note : The visualization does non embrace all scenarios. A virtual network adapter might be affinitized to the second physical adapter. If then, its load balanced packets would travel out of the shown "pNIC1" and use that concrete adapter'due south MAC as a source.

Traffic Crossing the Load Balancer

And then, our frame arrives at the load balancer. The load balancer has a really crummy job. It needs to make traffic go faster, not slower. And, it acts like a TCP/IP router. Routers need to unpackage inbound Ethernet frames, look at their IP information, and make decisions on how to transmit them. That requires compute power and time.

If information technology needs too much fourth dimension to exercise all this, and so people would prefer to live without the load balancer. That means that the load balancer's manufacturer doesn't sell any units, doesn't brand any coin, and goes out of business. So, they come with all sorts of tricks to brand traffic faster. I way to practice that is by not doing quite and so much piece of work on the Ethernet frame. This is a gross oversimplification, but you lot get the idea:

Essentially, the load balancer only needs to recollect which MAC address sent which frame, and then information technology doesn't demand to worry and then much about all that IP nonsense (it's actually more complicated than that, just this is close enough).

The Hyper-Five/Load Balancer Collision

At present nosotros've arrived at the core of the problem: Hyper-Five sends traffic from virtual machines using source MAC addresses that don't belong to those virtual machines. The MAC addresses vest to the physical NIC. When the load balancer tries to associate that traffic with the MAC accost of the physical NIC, everything breaks.

Trying to exist helpful (remember that), the load balancer attempts to render what it deems as "response" traffic to the MAC that initiated the chat. The MAC, in this case, belongs directly to that second physical NIC. It wasn't expecting the traffic that's at present coming in, so information technology silently discards the frame.

That happens because:

• The Windows Server network teaming load balancing algorithms aresend only; they volition not perform reverse translations. There are lots of reasons for that and they are all expert, so don't get upset with Microsoft. Besides, it's not similar anyone else does things differently.
• Because the entering Ethernet frame is not contrary-translated, its destination MAC belongs to a concrete NIC. The Hyper-V virtual switch volition not send any Ethernet frame to a virtual network adapter unless it owns the destination MAC
• In typical system-to-system communications, the "responding" system would have sent its traffic to the IP accost of the virtual machine. Through the normal class of typical networking, that traffic's destination MAC would always belong to the virtual machine. It'south only because your load balancer is trying to speed things along that the frame is being sent to the concrete NIC's MAC address. Otherwise, the source MAC of the original frame would take been little more than than trivia.

Stated a scrap more simply: Windows Server network teaming doesn't know that anyone cares near its frames' source MAC addresses and the load balancer doesn't know that anyone is lying about their MAC addresses.

Why Hyper-V Port Mode Fixes the Problem

When yous select the Hyper-Five port load balancing algorithm in combination with the switch independent teaming style, each virtual network adapter'southward MAC address is registered on a unmarried concrete network adapter. That's the same behavior that Dynamic uses. However, no load balancing is done for any given virtual network adapter; all traffic inbound and exiting whatsoever given virtual adapter willalways use the same physical adapter. The team achieves load balancing by placing each virtual network adapter across its concrete members in a circular-robin mode.

Source MACs will always be those of their respective virtual adapters, so at that place'southward nothing to go confused nearly.

I like this mode as a solution because it does a good chore addressing the outcome without making any other changes to your infrastructure. The drawback would be if you only had a few virtual network adapters and weren't getting the best distribution. For a 10GbE system, I wouldn't worry.

Why Static and LACP Gear up the Problem

Static and LACP teaming involve your Windows Server system and the physical switch like-minded on a single logical pathway that consists of multiple physical pathways. All MAC addresses are registered on that logical pathway. Therefore, the Windows Server squad has no need of performing whatever source MAC commutation regardless of the load balancing algorithm that you cull.

Since no MAC substitution occurs here, the load balancer won't get anything dislocated.

I don't like this method equally much. It means modifying your physical infrastructure. I've noticed that some physical switches don't like the LACP failover process very much. I've encountered some that demand a minute or more to discover that a concrete link was downwardly and react accordingly. With every concrete switch that I've used or heard of, the switch contained mode fails over nigh instantly.

That said, using a static or LACP team will permit you to continue using the Dynamic load balancing algorithm. All else being equal, you'll become a more than even load balancing distribution with Dynamic than y'all will with Hyper-V port manner.

Why You Should Permit the Load Balancer Exercise Its Job

The third listed resolution suggests disabling the related feature on your load balancer. I don't like that selection, personally. I don't have much feel with the Citrix product, merely I know that the F5 buries their "Auto Last Hop" feature fairly deeply. Also, these 2 manufacturers enable the feature by default. Information technology won't be obvious to a maintainer that you've made the change.

Nevertheless, your situation might dictate that disabling the load balancer's feature causes fewer problems than irresolute the Hyper-V or concrete switch configuration. Do what works best for you.

Using a Unlike Internal Router Too Addresses the Upshot

In all of these scenarios, the load balancer performs routing. Really, these types of load balancersalways perform routing, because they nowadays a unmarried IP address for the service to the outside globe and translate internally to the back-end systems.

All the same, cipher states that the internal source IP accost of the load balancermust exist in the same subnet every bit the back-cease virtual machines. You might do that for performance reasons; every bit I said above, routing incurs overhead. However, this all a known quantity and modern routers are pretty good at what they do. If any router is present betwixt the load balancer and the back-terminate virtual machines, and then the MAC address issue will sort itself out regardless of your load balancing and teaming mode selections.

Accept You Experienced this Phenomenon?

If so, I'd love to hear from you. What system did y'all experience it happening? How did y'all resolve the situation (if you were able)? Perhaps y'all've merely encountered it and arrived hither to get a solution – if so permit me know if this explanation was helpful or if y'all need any further help regarding your particular environment. The comment section beneath awaits.

Source: https://www.altaro.com/hyper-v/erratic-behavior-hyper-v-network-load-balancers/

Posted by: grenierundon1941.blogspot.com

Share this post