I've been working around this issue by building a utility machine that is in the same subnet as the brokers which does work. I get mixed results depending on how I run Server Manager.
1. Run as Administrator
When I run Server Manager as administrator here are the result:
Run as a regular user with admin credentials entered in Manage As for each RDS server including the 2 brokers listed:
I get the following message
As I said running them in either scenario on servers in the same subnet works without issue.
Some things that have been done recently
System Center Endpoint Protection has been enabled on all clients
Subnets have added to the closest sites in AD Sites and Services. Both the client and brokers are in the same AD Site.
Any help is appreciated.