Does your vendor treat you well?

Jeff Miles Leave a comment

There’s a few projects I’m managing lately related to storage expansion and conversion to backup-to-disk (which I’ll hopefully post on in the future). Part of this process has been the acquisition of equipment and software.
The order was split between two vendors, and was quite large by our standards. Something unexpected happened that surprised me, and made me think about the client/vendor relationship my company has with others.

Part of the purchase was with CDW Canada, and our account rep added a not-insignificant gift item to the order. It was a really nice touch, and something that while not immediately practical, really puts CDW in a good light.

I spend much more money with other vendors over CDW for a variety of reasons, and yet none treat me as good as they do. None of my other vendors have such a consistent quality online ordering system, or stock checks, or shipping speed. None of my other vendors send me a big tin of Christmas cookies every year, like CDW does.

This isn’t to say I don’t appreciate my other vendors; I wouldn’t put money their way if that was the case. But it’s the little things like what CDW has or does that make it worthwhile for us to keep using them.

 

I could be off base here, and some may think that a vendor shouldn’t be required to woo with gifts and tokens. I disagree with that though; it’s a competitive marketplace, and those things (along with great customer service and support) go a long way in making the decision of where to spend my money easier.

How well does your vendor treat you? If they’re a primary one and you spend lots of money there, perhaps its time to talk to your account rep about what they can do for you; what advantages they provide over others.

Sonicwall Global VPN disconnecting repeatedly

Jeff Miles 1 Comment

For a while now I’ve had my Sonicwall Global VPN policy on the firewall set as a “route all” connection. This means that all traffic for the VPN client goes through the Sonicwall directly, and blocks access on the client’s end to local devices.

 

Yesterday I came upon a situation where I needed to enable a client access to both the VPN and local devices at the same time. This called for Split Tunnels!

However, I didn’t want to enable split tunnels universally for all my VPN clients. Luckily I found this Sonicwall documentation on setting up a single WanGroupVPN with two different policies based on user group.

The premise is that you set up your Wan GroupVPN as a split tunnel, but then give certain users access only to a specific address object and use a specific NAT Policy (I won’t regurgitate the entire document here).

This was working great, but I shortly found that when testing as the split tunnel user, I would get connected and then disconnected within 10 seconds. Typically the connection would last for 3 successful pings.

After a bit of Googling I found this article which explained it being caused by an incorrect address object within the “VPN Access” tab for the user.

I checked that out, and strangely enough, only the correct item was listed:

I looked at both the “Everyone” and “Trusted Users” group and it looked the same.

 

After a lot of head scratching, I finally discovered that in fact the “Everyone” group did have “All Interface IP” object applied to it, by viewing a logged in user’s status here:

Click for large view

Somehow that was still selected for the “Everyone” group, but it just wasn’t displaying when viewing the “VPN Access” tab. So I clicked “remove all”, and then re-added the appropriate objects, and problem solved!

 

Processor & RAM upgrade on Dell R410

Jeff Miles Leave a comment

Have I said before that I love virtualization? Because I really, really do.

In my original Hyper-V implementation, I used two Dell R410’s, each with 32 GB of RAM (4 sticks) and 1 Xeon 5630 processor. It’s been a little bit of time since then, with some additional VM’s brought online for various services. My benchmarks showed it was time to upgrade the cluster, mostly for RAM failover amounts; I can’t go below 50% available RAM otherwise all the VM’s won’t be able to run on one host.

So I called up my Dell rep, ordered 2 x Xeon 5630 and 8 x 8 GB of RAM, and today installed them.

Intel Xeon

 

The install went very smoothly, and because of the Hyper-V cluster and Live Migration, occurred in the middle of the day without downtime or interruption.

This is the process I used:

  • Manually drained a host (Windows Server 2012 will have this as a feature, which is nice).
  • Performed Windows Updates and a BIOS update from Dell
  • Restarted the server, and entered BIOS setup to ensure latest version applied successfully
  • Turned off server, slide out from the rack (Man do I love the RapidRails).
  • Opened up the chassis, removed the shroud covering processors and RAM
  • Added 4 sticks of RAM
  • Removed the CPU filler, and inserted the new processor.
  • Attached the passive heatsink, and screwed it into the mounts.
  • Turned on the server.

And that’s it! Just like that, I’ve doubled the capacity of my infrastructure, and it took under an hour.

 

Unicast Flooding on PowerConnect 5548

Jeff Miles 4 Comments

Yesterday I learned something new; it’s possible for a switch to stop operating as a switch, and start flooding all unicast packets out every interface. This is something I just solved on a Dell PowerConnect 5548 switch.

In retrospect, this happened a few months ago too, but at the time I couldn’t spend any time troubleshooting, and rebooting the switch resolved the problem. This time I wanted to get to the source of the problem.

I first noticed a problem when accessing network resources was a bit slower than normal. I took a quick look at our network weathermap (combination of Cacti and weathermap plugin) and noticed that all ports coming out of our 5548 were pushing ~90 Mbps of traffic, which is definitely not normal

Weathermap plugin for Cacti
Weathermap Output (taken during normal traffic)

 

I logged into our Cacti interface and took a look at the graph for one of the interfaces on that switch:

Based on that graph, I could see that the traffic started Tuesday morning, and was pretty consistent.The interesting thing is that this was happening on ALL the interfaces on the switch, including the Link Aggregated groups.

PowerConnect monitoring graph
PowerConnect monitoring graph

 

At this point I spent quite a bit of time trying to figure out whether it was a reporting problem (since the devices on the other end of those connections weren’t reporting high traffic) or an actual traffic issue. I hadn’t heard of unicast flooding before, so I didn’t immediately start looking there.

I started a Port Mirror from one of the ports that should have almost zero traffic, and Wireshark gave me hundreds of thousands of packets within a few seconds, all for devices not actually on the port I was mirroring. At this point I understood what was happening, but not why.

A quick google led me to the term “Unicast Flood” and some probably causes, but none of them really applied. My network topology is flat, a single VLAN with no STP. CPU utilization was low, and the address table only had 8 entries in it.

Wait, 8 entries? A core switch should have hundreds of entries in it’s address table right? I was experiencing a unicast flood because the switch wasn’t properly storing MAC addresses in it’s table, causing almost all the traffic to be pushed out every interface.

Back to google, and I eventually came across the following in release notes from firmware in October 2011:

Description User Impact Resolution

Devices stop to learn MAC addresses after 49.7 days

After 49.7 days of operation, the device stops re-learning MAC addresses. These MACs which were previously learned will not appear in MAC address table. As a result traffic streams sent to previously learned MAC addresses are treated as unknown-unicast traffic and flooded within the VLAN.

MAC address learning mechanism was fixed so that both learning new addresses and re-learning existing addresses are updating the MAC Address database.

That’s one mighty big bug to be on a core switch. Turns out that I hadn’t updated the switch to the latest firmware when I first received it in February 2012 (nor was it shipped with current firmware) which is a very uncharacteristic thing for me to do.

Today I updated the firmware to the latest, and we’ll see what happens in 49.7 days.

 

FTP software alternatives

Jeff Miles 2 Comments

I’m looking for an alternative to FTP, because to be honest, FTP really sucks. Right now I’m using it for:

  • one-time transfers between individuals
  • long term sites for repeated transfers
  • data uploads for internal procedures

 

The big features that I need are security (through SSL), ease of use, and email notifications. The other requirement is that it must be on-premise; some of these long term sites have GB of data that changes daily, and I can’t have my users waiting to transfer that data to the cloud before its available for our clients.

So far I’ve found 3 potential solutions:

 

Ajaxplorer is attractive because its open-source, and offers a familiar interface through a web browser. It doesn’t require any add-ons or plugins to access. I ended up setting up a test site with ajaxplorer to further evaluate, and have determined that despite all of its strengths, the weaknesses are just too big to fully replace FTP within my environment. Notably:

  • Email notifications: they are possible, but aren’t very configurable, and are only per-file. There isn’t a way to make an email summary per connection. If someone uploads 3000 files, the recipient is going to get 3000 emails.
  • Active Directory integration: again, this is possible, but you can’t mix AD users and local user accounts. We want both so that our internal users don’t need another account, but external clients don’t need to be in our AD.

 

Citrix Sharefile was introduced to me through a cold call, but it happened to be a day after I started looking for an FTP replacement. Sharefile is a fully featured, cloud based file transfer application that can be used through a Windows client, mobile client, or web browser. Its really nice actually, and very easy for the average user to pick up. The client for Windows (and add-ons for Microsoft Office) add a lot of value, and give the capability to auto-upload to Sharefile or generate links to send out.

External clients receiving files can do so straight through their web browser, and depending on the link generated for them they may not need to log in at all.

Sharefile is licensed per named user per month, and seems to be pretty competitive. The major downside of Sharefile right now is that there is no on-premise option. According to the sales rep I was speaking to we can expect this installation type later this year. Whether I wait for that depends on whether I find something else that can do the same job in the mean time.

 

Which brings me to RES HyperDrive. This is very similar to Citrix Sharefile, except that it is on-premise from the start. One problem: it hasn’t been released yet. I’ve got a demo sitting on my desktop to try out which I’m very excited about but even if I love it, it will be difficult to trust the first release of some software. I was very pleased when downloading their VM for the demo that it was pre-packaged in a Hyper-V format. This is the first vendor that I’ve seen doing so.

I suppose this post is a little premature due to having not tested RES, but if it performs well and has all the features I need, a follow-up post will be coming with my installation and setup experience.
At the moment I’m not sure how RES Hyperdrive is licensed, or potential costs.

 

If anyone is using any of these applications and have an experience to share, please do in the comments!