This week’s episode features two segments, each an interview with a Wireless LAN Professional. First is Jared Griffith of CinergyWiFi, a Wireless LAN Value Added Reseller (WLAN VAR) – and the second is a story about troubleshooting from Tim Dennehy of the University of Kansas.
Both of these interviews were an attempt to help share a little insight into the wide variety of positions available for Wireless LAN Professionals.
Jared Griffith
Cinergy Wi-Fi, Inc
[email protected]
http://cinergywifi.com/
http://RuckusGear.com/
“Our MacIntosh issue”….
We’re a large public university – 25,000 students and several thousand staff members. The main WLAN is “Jayhawk”, and is almost a coffee shop style SSID with no restrictions and a simple webauth for access. Of course we have other SSIDs on campus – for athletics, the bookstore, guests and even the parking department. Students and faculty all have accounts in Active Directory, so that is how they get online for our main Jayhawk WLAN. From a network administration standpoint, it is extremely difficult to control what clients come on our wireless network. There is no “standard machine” issued — you come to university with whatever you walked in the door with.
We have 1800+ Cisco radios on campus spread across 79 buildings, all with two fully redundant WiSM farms in two different locations. Any controller can fail and the secondary or tertiary controller (or any other controller for that matter) can easily pickup the load so the user doesn’t experience an outage other than the time the access point needs to find another controller. It’s a lot of work to setup on the backend, but the payoff is if any controller fails at 3am, my phone won’t ring. In fact, the night shift might even not notice if a controller failed since the help desk probably wouldn’t get any calls.
Last school year ended in May. In June, after the students left, we upgraded all of our controller code from 5.2 to 6.x and then to 7.x so we could start implementing the new Cisco 3502 series access points. Upgrading during the school year, as you can imagine, is extremely difficult to get as we need a change window just like every other 24x7x365 organization. The upgrade went well — at least it did for the next three months, anyway.
The Kansas Jayhawks play football — and of course during the football games the press shows up. The pressbox is only used for the football stadium, so if the wireless network got broken during an upgrade, it would probably go unnoticed until the next time a game was played.
Sure enough, about four weeks ago, we had our first football game. We have four WLANs being broadcasted in the 8th floor of our pressbox. Our main SSID, Jayhawk, the ticket scanner WLAN, the Athletic’s statistics WLAN and guest WLAN. Three hours before the game, my phone rang with “the press is coming to the 8th floor of the pressbox and we’re having issues getting their Macintoshes online. As a last ditch effort to them online before the game, and employee used his credentials to get them on Jayhawk.
The press is supposed to use the guest (KU-Passport) WLAN, but for whatever reason they were first getting on the WLAN called Jayhawk. A call to Apple confirmed that the Macintosh computers will first search for the strongest signal (all four wlans should have the same signal strength) and then if all SSIDs are of the same signal strength, will search alphabetically. Jayhawk is before KU-Passport in the alphabet, so the press would try to get online and end up on the wrong SSID.
The press would enter in the username and password that was given to them (they didn’t know they were on the wrong SSID) for that specific game — we would create usernames and passwords for the press using Cisco’s Lobby Ambassador that was built in to Cisco’s Wireless Control Server. They would not get online and immediately call for technical assistance.
The technical assistance representative for the game would assist them and inform them they are the wrong SSID, and they need to associate to KU-Passport and enter in your username and password. This is where the trouble began. The “press user” would navigate to the correct SSID and attempt to associate, only to not get an IP address or the web authentication page. The technical representative would then assume the username and password was “bad” and get them back on Jayhawk with his credentials. This would only aggravate the issue — they would try to get back on Jayhawk and this time they would not get an IP address and of course, not get redirected to the web authentication page. The user would end up rebooting their Macintosh and would either get online with the technical respresentative’s credentials on Jayhawk, or maybe they would get lucky and end up on the KU-Passport WLAN and get online like they should have.
After hours and hours and hours (probably more like days…) of Googling, I found that I am not the only one with Macintosh issues. Other educational institutions are having the same exact issues, only I din’t find any solutions to the problems. I called Apple, and they said “we don’t see this issue anywhere else — it must be your wireless network”. Of course five minutes’ worth of Googling shows we’re not alone.
I decided to call Cisco’s Technical Assistance Center. Jeff called me back within the hour asking me if I had a wireless sniff of the clients in question. I had to tell him I didn’t own a Macintosh, and there’s no way I can’t ask a member of the press to give me his/her Macintosh so I can install Wireshark on it either. He asked me to run some debugs on the wireless lan controller, so I did. The result was that I could not see a Macintosh laptop sending out a DHCP Request. I came to the conclusion I needed to find out if the Macintoshes were actually sending out DHCP requests, and the only way to do this was to run Wireshark on a local laptop that is having issues or have a laptop running a protocol analyzer in the press box with a promiscuous wlan adapter.
We own AirMagnet WiFi Analyzer, so we tried to use it to sniff for DHCP requests to no avail. I called Fluke and opened up a technical assistance case with them. They called me back and agreed that they couldn’t seem to capture a DHCP request either for whatever reason. No word back yet….
Next step… I downloaded Wireshark and put it on my WindowsXP laptop, but did not have a promiscuous adapter, so could not see all the traffic – hence no protocol analysis traces to send to Cisco. I contacted CACE Technologies and looked at their applications. A promiscuous adapter is about 800 dollars, so I wrote it up and submitted the request to management and have not heard back yet. Turns out if I buy three of their adapters, I can monitor all three channels in promiscuous mode and actually get traces of what is actually going on!
After looking around (and doing a lot of begging) I was able to borrow a Macintosh laptop running 10.6.x. I attempted to install Wireshark on the machine to no avail — I am not a Mac user and was literate enough to get to get the application downloaded, but unable to set the permissions in such a fashion to actually get it to work.
I then received a phone call from Cisco’s TAC. Jeff said that I could try enabling “fast SSID change”. He said enabling this would allow my clients to change between SSIDs faster than normal, and it might help. Of course I jumped right on it and turned that on.
We went right back to testing! We started testing with the Macintosh I borrowed – we tried changing between SSIDs quickly and could replicate the problem. We then enabled “Fast SSID change” on the controller and our preliminary results look promising. We have a lot more testing to do (and one more game this weekend for the ultimate challenge) before we call this a success, but for right now we’re happy with the results. Why our WinXP boxes and my Ubuntu laptop have no issues whatsoever and why the Macintosh comptuers have problems is beyond me. But if “fast SSID change” fixes our Macintosh computers, I’m a happy Jayhawk…
The day before the game, we tested everything again. We backed out all the configuration changes that have happened in the last 12 months, which was the last time we didn’t have any problems. Same results – the Macs would not change SSIDs and get an IP address without having “fast SSID change” enabled. Twelve months ago we did not have this setting enabled, and after backing out all of the changes in the last year we have proved the problem still exists. The only thing we could not back out easily was downgrading the code from 7.0 to 5.2. At this juncture we can only assume that upgrading the code caused this issue since that is the only change we could not “undo” easily without affecting the entire campus.
Game day. A complete success! We arrived 3 hours before game time and I personally walked in with a Win7, XP and two Macintosh laptops. All four machines associated to the network easily and we changed SSIDs quickly as expected. We then turned off the “fast SSID change” and tried again, which brought back all the issues we were seeing. We quickly turned turned it back on everything was okay. The press showed up an hour later and had no problems at all.
Tim Dennehy – [email protected]
Thanks for your patronage. If you’d like to be included in our upcoming monthly newsletter, be sure to register with our ‘double-opt-in’ mailing list in the top right corner of the website.
Thanks for listening.
We’d love to have you subscribe to our RSS feed – just click the button in the upper right corner of the web page. Until next week, thanks for listening!
If you have any feedback on the show – please drop an e-mail to [email protected].
Subscribe To The Wireless LAN Weekly Podcast:
