Replication Group ID: B8242CE2-F5EB-47DA-BA5B-1DD2F7EE3AB9
This would cause a break in replication which wasn’t desirable during production hours. The strange thing was, it occurred every 5 minutes like clockwork, for all our servers separated by VPN.
I eventually discovered it was a problem with our Sonicwall devices providing the VPN connection. There was a 5 minute timeout value for TCP connections, which was being enforced on the DFSR connections for some reason.
While not an ideal solution, we have worked around this error by setting the value to a sufficiently high number.
UPDATE Sept 2011: I realized that the majority of this post was describing the problem and not the solution, so I’ve updated with clear instructions on what I’ve done to resolve this.
To start I only created these rules on my hub firewall at our head office. Doing them on each branch office wasn’t necessary.
I created address objects for each of my DFS servers, and placed them into two groups – one for local (from the firewall’s perspective) and one for servers across a VPN link.
Then using the firewall rules matrix, I create two rules, one in each of the indicated sections:
The two rules I created look like this:
On the properties for each rule, on the Advanced tab, increase the TCP connection timeout to some large value:
This was necessary for my Sonicwall Pro 4060 running SonicOS Enhanced 126.96.36.199-51e. In a couple of days we are replacing this with an NSA 2400 on SonicOS 5.8.x, so I’ll disable these rules to see if the issue still occurs on new hardware.
Over Christmas I deployed a two node Hyper-V Failover Cluster with a Dell MD3220i SAN back end. Its been running for almost a month with no issues, and I’m finally finishing the documentation.
My apologies if the documentation appears “jumpy” or incomplete, as half was done during the setup, and the other half after the fact. If you’d like clarification or have any questions, just leave a comment.
The MD3220i needs to be configured with appropriate access and network information.
Before powering on the MD3220i, see if you can find the MAC Addresses for the managment ports. If so, create a static DHCP assignment for those MAC’s aligning with the IP configuration you have designed.
Otherwise, the default IP’s are 192.168.128.101 and 192.168.128.102
The MD Storage Manager software needs to be installed to manage the array. You can download the latest version from Dell.
Once installed, do an automatic search for the array so configuration can begin.
Ensure that email notifications are set up to the appropriate personnel.
Premium Feature Enable
We have purchased the High Performance premium feature. To enable:
In the MD Storage Manager, click Storage Array > Premium Features
Select the High Performance feature and click Enable
Navigate to where the key file is saved, and choose it.
Disk Group/Virtual Disk Creation
Below is an image of our disk group, virtual disk, and CSV design. What works for us may not be most suitable for everyone else.
Each virtual disk maps to a virtual machine’s drive letter.
My only concern with this setup is the 2 TB limit for a VHD. By putting our DFS shares into a VHD, we will eventually approach that limit and need to find some resolution. At the moment I decided this was still a better solution than direct iscsi disks.
MD3220i ISCSI Configuration
Configure iSCSI Host ports
In the Array Manager, click “Storage Array” > iSCSI > Configure Host Ports…
In the iSCSI host port list, select the RAID controller and host port, and assign IP addresses according to your design
For every port, choose “Advanced Port Settings”
Turn Jumbo Frames on for every iSCSI port
Create Host Mappings for Disk access
In the Array Manager, choose the “Mappings” tab
Click “Default Group” and select Define > New Group
Name this: Hyper-V-Cluster
Within that group, add two new hosts. Here’s how to get the Host initiator ID:
Log into hyper-v host, go to command prompt, type: iscsicpl
On the Configuration tab, copy and paste “initiator name” within the MD Storage software.
Hyper-V ISCSI Configuration
Remote into the Hyper-V hosts.
In the command line, type and press enter (case sensitive):
start /w ocsetup MultipathIo
On the second tab, check to enable iscsi support.
Follow the MPIO driver install instructions I previously wrote about here: https://faultbucket.ca/2010/12/md3220i-mpio-driver-install-on-hyper-v/
Reboot the server after that.
Again on each Hyper-V host, from the command line, type iscsicpl. If prompted to start service, choose yes.
When the iSCSI window appears, enter any IP address of the MD3220i controller, and click QuickConnect.
A discovered target should appear there, with a status of “Connected”.
Highlight that target, and select “properties”. The “Sessions” window will appear, with one session listed (I know the screenshot is wrong).
Check that session, and click “Disconnect”, then click OK.
On the main ISCSI window (where you clicked QuickConnect), select “Connect”
Then check off “enable multipath”, and click Advanced
Select the Microsoft iSCSI initiator, and then set up the appropriate source and target IP, according to the iscsi config here:
Do this for each storage NIC on each server. There should be 4 connections per server.
Then click the “Volumes and devices” tab, and select Auto-Configure”. You should see one entry for each disk group you made.
Now we should be able to go to disk management of a single server, create quorum witness disk and your simple volumes.
If you haven’t performed the steps in the Remote Management & Tools section, do so now.
Create an mmc with Disk Management control for one Hyper-V host
You will see your 3 disks within this control, as offline and unallocated.
You want to initialize them as GPT devices, and create a simple volume with all the space used.
Name the 2GB one (which was created during disk group setup on the MD3220i) as Quorum.
Those steps only need to be applied to a single server, since its shared storage.
Further disk setup happens after the Failover Cluster has been created.
Storage Network Config and Performance changes
To enable jumbo frames, I followed the instructions found here:
Other than what I discovered through the setup process and have included in the documentation, there were no real issues found.
Oddly enough, as I was gathering screenshots for this post, remoting into the servers and using the MMC control, one of the Hyper-V hosts restarted itself. I haven’t looked into why yet, but the live migration of the VM’s to the other host was successful, without interrupting the OS or client access at all!
Nothing like trial by fire to get the blood pumping.
A quick review of the Engenious 5611P that we have been using for a few months to connect two offices.
My company recently leased space in a building adjacent to our head office. These two buildings are separated by 250 feet of parking lot, with clear line of sight.
We wanted network connectivity for data, as well as Voice Over IP, so that an additional phone system and receptionist aren’t required.
A wireless link is the obvious solution in this case, and we started with a Cisco WAP4410N. The interface for these devices was good, and setup was quick. We used the draft-N protocol and setup a WPA2 secured WDS bridge.
For ease of install we placed the two devices within the building behind the window.
Performance was acceptable, and after a few tests we deemed the link good. However, after about a week of activity in the office, there were severe performance issues, with packets dropping frequently and the Cisco devices locking up.
Ultimately we sourced the issue as being:
Cisco WAP4410N being a crappy device
Interference in the B/G/N network range within our area
Based on that, we purchased two Engenious EOC-5611P devices, to set up an 802.11a network.
Here’s a couple screenshots of the web interface:
Since setting this up, the link has been rock solid, through hot weather (35 Celsius), cold weather (-40 Celsius), rain and heavy snow. (again, these are indoor behind a window). Our VOIP equipment has zero problems and we haven’t had any issues with the software on the devices.
The only downside that we have found of the Engenious devices is the lack of WPA2 encryption while using the bridge mode. Currently only WEP is supported for that mode.
I would immediately recommend these devices for anyone looking for a line of sight link, especially with 802.11a protocol. We purchased from NCIX here:
Just solved a particularily troublesome issue that wasn’t obvious at first but makes sense now.
We have multiple Internet connections from multiple providers; this displays what they’re plugged into on our Sonicwall 4060:
The WiBand connection is from a wireless ISP, connecting to a basestation about 1.5KM away.
There is a client of ours across the street who is trying to access a website who’s DNS entry refers to the X3 interface provided by a Shaw IP address. They receive a strange “Oops, we could not reach that website” error page within Internet Explorer.
I did an nslookup to make sure the DNS A record for the site was still correct, which it was. I used our external dial-up line to ensure that the site was up and available, which it was.
Based on this, I suggested to my company contact dealing with the client that perhaps they are using a custom DNS provider who has an incorrect A record for our site, who is providing that custom error page. I then forget about the issue.
The next day I get a call from the client’s IT department that the problem still exists. We run through some DNS troubleshooting, and determine that site is getting the right IP, but still getting the custom error page.
I decide to check the error logs on my firewall, and the only thing of note is an “IP Spoof Detected” error. After asking what the source IP is from the problem site, it is confirmed that its the same IP as the ‘spoof’.
The client site has this IP address (close enough):
Alarm bells start going off as I realize this is very similar to our WiBand IP on X2. Our IP for that link is (changed for privacy):
Turns out the client across the street from us is also using WiBand for an ISP, and we’re connecting to the same basestation, in the same subnet.
The HTTP request is coming in on X3, but the response can’t leave X3 destined for the client IP, since that range is on X2. So our firewall drops the packet.
Our current work-around is a static route that forces the return traffic out the correct interface. It looks a little like this:
Source Destination Service Gateway Interface
Any Client IP HTTP(all) Shaw Gateway IP X3
I had the gateway on this route originally set to OUR Shaw IP, but this was incorrect.
I suppose next step is to find out why WiBand has us on the same subnet, and whether they could use VLAN’s or something else to segregate us. It’s a little disappointing that we will be the guinea pigs for this, as I would have thought an ISP would have resolved these type of issues by now.