At the WLPC event in Phoenix of February 2018, Jerry Olla and Scott McDermott taught a Deep Dive class about mobile WLAN testing. The main tool they used as part of the class was the NanoPi NEO2.
The NanoPi NEO2 ended up being the perfect foundation to build a reliable, light weight, powerful, mobile WLAN testing tool.
In this interview Keith and Jerry discuss how this tool came to be, why Jerry wanted to create this tool in the first place, and the different uses for the tool.
If you want to purchase your own pre-built, pre-loaded kit with the NanoPi NEO2 you can get one from the store.
Following is the full conversation between Keith (KP) and Jerry (JO) – Jussi Kiviniemi (JK) from Ekahau pops in at the beginning to say “hi”.
Note – The transcripts have been edited and condensed for clarity. You can listen to the full interview on episode 140 of the Wireless LAN Professionals podcast HERE.
KP: Welcome back to Wireless LAN Professionals Podcast. Today I have with me Jerry Olla. Jerry how are you?
JO: Hey Keith yeah doing doing fantastic. Hanging out with my my buddy Jussi in Florida
KP: In Florida, you guys are a BICSI right?
JO: Yeah we just actually wrapped up the BICSI event here and are now catching up on some things here before heading home.
KP: Did you convince them all to do Wi-Fi correctly?
JO: Yeah I think we made some progress. What do you think Jussi?
JK: Yeah for sure. Hey Keith good to hear from you again.
We had a speaker’s slot here at BICSI where we presented to like more than a hundred people there about Wi-Fi.
Thirty minutes before the speak I told Jerry he should come on stage with me. He told me, “Well I’ve never seen your presentation. I have no idea what you’re going to talk about.”
I think that made the presentation even better. We tag-teamed it and it was awesome. It was a full room, standing room only – people had to stand at the back. I think the presentation was well-received. Jerry did an awesome job. He brought a real expertise into the house.
JO: I tried to balance things out.
KP: Right, so he was he was the counterpoint to your humor?
JK: To my marketing spiel.
KP: Jerry was there to say, “Jussi said ‘this’ but let me tell you how it really works.”
JK: As Keith always says about me, “He’s a VP so he doesn’t understand technology.” Right?
KP: oh I know you understand the technology, it’s just fun to tease you.
Well on to this podcast today. Thanks for for kicking in there Jussi.
With Jerry years ago, before you were with Ekahau, we were at a conference I think was in Minneapolis
JO: That’s right.
A Few Tries “Pre NanoPi”
KP: We were there with with Blake as well. You had this cool little toy that you showed off you could do some WLAN testing with. Blake and I, and a couple others, kind of followed along with what you did and it turned into something we decided to include at WLPC in Budapest. We’ve since tried it across all the subsequent WLPCs. Now we are introducing a new little, better, cooler one at WLPC in Phoenix.
I thought this would be a good chance for our audience to listen in and kind of hear the history of how you developed this whole thing. This is your baby and it’s come a long way. You good with that?
JO: Yeah absolutely. I totally forgot about that event. That was definitely a big step in the right direction of the the small portable throughput testing device. That’s really where the whole idea was born.
I had been at a company working with voice over Wi-Fi type deployments. Anybody that’s worked with voice over Wi-Fi knows the challenges that are associated with those type of deployments. I needed to test the readiness of a network for voice over Wi-Fi and the resiliency of the network to make sure that it was going to support that type of traffic.
I needed to be able to go to different sites and quickly deploy this throughput testing device. It needed to be able test the throughput as well as the consistency of the network. Measuring things, particularly like jitter, on the network to see if the networking was fluctuating. Was the network going fast-slow-fast-slow or was it running at a consistent speed end to end? I needed to not only test the Wi-Fi but also the the wired piece where the head end of the voice server would be.
I needed to be able to go to different sites and quickly deploy this throughput testing device.
KP: You could have used just a laptop running iperf on one end. That that’s the first thing most of us would think of. “Just get out my spare laptop and attach it someplace.”
JO: Yeah absolutely. I certainly could have. However, I was traveling a lot, and anybody who’s done Wi-Fi surveys can relate. The smaller and more portable and battery powered our equipment can be the better. So, lugging around a second laptop wasn’t going to be a good solution for me. I wanted to come up with a really portable, really lightweight solution I could travel around with. Also, on occasion, I needed to move this thing around to different parts of the network. So, to be able to have the tool battery powered and be able to unplug it from one switch into another switch and a different part of the network or different building is what I was after.
The smaller and more portable and battery powered our equipment can be the better.
KP: Did you think of a little raspberry pie is an alternative
JO: I briefly looked at a raspberry pie and then quickly realized that was not the right solution for a real true robust throughput testing device. The biggest limitation on the Raspberry Pi is the 10/100 only Ethernet port. You’re going to have a big bottleneck there, especially if you really want to “drag race test” an 802.11ac network and see what the speed is. For jitter and being able to test a few certain things the Raspberry Pi would have worked fine. But I knew it was not going to be the ideal solution for what I was looking for.
KP: So, what was the single board computer you went with instead?
JO: So I would even back up a little bit further before the Odroid. I did kick around a couple of other pieces of hardware. The very first one that i actually used was actually an open source wireless router that was a consumer grade with a couple of gigabit ethernet ports. It was open source running an embedded Linux. I was able to use that to run an iperf server. But it was just really difficult to work with. It was really limited on what Linux versions it would support and what packages I could install.
KP: Was it like a regular Linux or was it more like a WRT?
JO: It was a DD-WRT firmware running on there. I was really limited of which versions the hardware would support. Also, it was a little bigger because it was a wireless router. I couldn’t really battery power it and I needed a brick charger for it. It wasn’t really much more convenient than lugging around an extra laptop.
KP: And then?
The ODROID – One Step Closer To The NanoPi
JO: And then I discovered the ODROID. I kicked around a couple of different single board computers prior to the ODROID which are not really worth mentioning because they just weren’t very good for support and Linux support. I ended up finding the ODROID made by hard kernel and had really good support from the community. Not as vast as the Raspberry Pi community is. Good enough for my purposes of supporting different Linux kernels and it was up to date, supporting all the packages and things like that. Works with the Debian and Ubuntu distros. It really checked all the boxes I needed. It had the nice same form factor as the the Raspberry Pi. It was nice and small and really low-power so could be easily powered by a battery.
KP: How did it test out from a throughput standpoint? Was it the bottleneck or could it outrun your Wi-Fi?
JO: Good question. A couple of the other single board computers I did try out that were similar form factors would max out just because of the pure kind of horsepower of the hardware. They just couldn’t handle more than maybe like five or six hundred megabits per second and the hard kernel, the Odroid C2 that we used, could easily do over 900 megabits a second of up and down. It could saturate the whole Gigabit Ethernet connection.
the Odroid C2 that we used, could easily do over 900 megabits a second of up and down. It could saturate the whole Gigabit Ethernet connection.
KP: And ODROID had some others that are even more powerful but then they draw more power.
JO: Exactly. It was kind of the perfect balance – size of horsepower that it had capable of processing but then didn’t draw very much power. It really met all the needs. It also had plenty of options of accessories like an LCD screen and different cases and ways we could configure it for the throughput testing device needs.
KP: The first time I saw you had one of these it was kind of difficult to figure out. It was “headless” at that time – meaning there was no screen at all. What was the process you had to go through in that first cycle to figure out your IP address?
JO: That definitely was the pain point. Up until then everything I had been doing was all headless. I would be plugging into a network and then I would have to ecstatically configure an IP and hopefully be able to route to that IP address. Or I’d have it grab a DHCP address and reverse engineer going to the DHCP server and plugging in the MAC address and looking up that IP address or use some type of IP scanner.
It wasn’t a pretty process and wasn’t consistent. From a user standpoint it wasn’t very ideal.
We had to figure out a better solution of telling us what the IP address was.
One of the initial ideas I kicked around was an email script. It was a Python script that I had found and kind of reversed for these purposes. When it would grab an IP address it would actually email me what the IP address was.
This “fix” filled that gap a little bit but whatever you were plugging the ODROID into needed to be able to route to the internet to be able to send an email. That wasn’t always an option for me. And some of these customer settings and some of these voice only networks didn’t need to route to the internet so yeah, that kind of shot that one in the foot pretty quickly.
From a user standpoint it wasn’t very ideal.
KP: Another issue we ran across in Budapest, that obviously has been solved since then, was the idea that we needed to get a bigger memory card. The thinking was, if you have a bigger memory card you can save lots of data. Unfortunately, there was no way at the time that ODROID could make copies of the memory at the scale we needed for WLPC, which meant each ODROID had to be imaged manually.
JO: Yeah that was a very painful process. You’re bringing back all sorts of fun memories now. I blocked most of that stuff out but I’m glad you remember it.
KP: It was the “bad years”.
JO: Yeah, we had overcome a few obstacles getting it this far.
We tried to do a 32GB memory card thinking the main advantage was you if we’re going to use this thing potentially down the road as some kind of remote packet capture device or something we’d want to have some decent storage to be able to store those kind of files and stuff on.
But the pain point was for imaging these things, since that’s also the hard drive, is then we have to image 32 gigs of an image file and be able to transfer that image file around. We decided to go a different direction where every time we installed this thing it ran an install script and pulled down all the packages. We tried to think this thing through a few different ways. It definitely had some scalability issues.
We decided to go a different direction where every time we installed this thing it ran an install script and pulled down all the packages.
KP: So there’s a hundred people or so in Budapest – who can attest to that whole process. I mean we got it to work. There was a local drive. Once they got their ODROIDs to work and because they were headless they would take them and plug it in and eventually all the packages came down and a rebooted. It worked and then they had control over it.
But now, using a smaller 8GB drive you can just do a straight up image – copy – boom – done.
JO: That really was the ideal solution. It fixed a lot of the issues we found. With this new project we’re working on I actually found I can squeeze it down into a 4GB memory card and even make the process easier to move around. I’ve kind of continued to fine-tune this whole process to be able to scale this thing.
KP: So the next phase after the crazy headless process was we went, “What if we get a screen?”
We played with a little two line LCD for a while. This worked initially, since all we needed was the IP address. If we could get the two line LCD to show up and display the IP address then you could gain remote control.
But better than that, and what we used last year at all three WLPCs was an LCD screen on top of the Odroid with an image, and that process went along much much better.
But, we still had issues with cases. It’s kind of a single-board computer so what do you mount it in? How do you power it? You have to have power. So the original ODROID worked well for a year. Between the three WLPCs I think we went through maybe 400 ODROIDs. But now tell us a little bit about the NanoPi.
The NanoPi NEO2 Is Introduced
JO: So that’s that’s what we’ll be unveiling at this year’s WLPC. We found a new hardware platform that’s still the same concept but it’s now built off this NanoPi infrastructure and the board we are using is called the NEO2. It has a Gigabit Ethernet port. It’s capable of doing 800 – 900+ megabit per second throughput making it great for testing things like the latest and greatest 802.11ac, and APs. The NanoPi provides all the functionality we needed but in an even smaller form factor.
It is a tiny little cube and has a small LCD screen on there that’s actually OLED based making it a nice clean crisp text output. It’s nice and bright. It is fairly small but still easy to read. It even has some buttons built in. We get a lot of same functionality that we had in the ODROID but with even smaller form factor and in a more robust, metal case. You can throw it in your bag and not have to worry about something crushing it or it having any problems down the road.
KP: And it is not using eMMC memory. It’s using just an SD card. Have you found any changes with this new set up worth worrying about?
JO: One of the things that has been a problem in the past is the micro SD cards we were using. We had some corruption issues with these consumer grade cards if you didn’t shut the unit down properly, you just disconnected the power while it was in the middle of maybe reading and writing some information.
The NanoPi provides a professional or I think they call it “industrial” micro SD card with some some read/write verification capabilities. So there is some protection built in which should help prevent those kind of issues from occurring. So far I have not seen any of those kind of problems we saw in the past so I’m hoping that this is going to be a good reliable solution for long-term.
The NanoPi provides a professional or I think they call it “industrial” micro SD card with some some read/write verification capabilities.
KP: Good. Since we went through the little history of the hardware let’s go back and do a history of the services or software that was on your version of this this little test tool.
In the beginning you just had iperf and iperf 2 or 3?
JO: That’s a good question. I don’t remember if it was iPerf2 or 3. Probably would have been iPerf2. This is way back when I first started with DD-WRT and one of the big problems is it only would support iPerf2. I couldn’t put the latest iPerf3 version, I couldn’t put on the ePerf the Ekahau version of iPerf3 customized build of that for Wi-Fi.
KP: A good little side note – “What is ePerf and why do we need it?”
JO: Yeah good question. So ePerf is Ekahau’s own internal build of the iPerf 3 package. It allows us to do the same kind of throughput testing that we do with iPerf3 but using it in conjunction with the Ekahau software for performing a site survey. The reason you would want to use that over using iPerf2 or iPerf 3 is because the iPerf3 software native code wasn’t necessarily built specifically for Wi-Fi performance testing. It especially wasn’t designed for the way we use it – physically moving around a site, roaming between access points, potentially roaming off the network to the point of being disconnected during a throughput test – which can crash that iPerf server. The server wasn’t really built for the resiliency of Wi-Fi and that kind of testing. The ePerf enhances that and is built specifically for that so if you do get disconnected during a throughput test it will automatically resume itself as soon as you’re reassociated to the network and not crash in the process.
So ePerf is Ekahau’s own internal build of the iPerf 3 package. It allows us to do the same kind of throughput testing that we do with iPerf3 but using it in conjunction with the Ekahau software for performing a site survey.
KP: so you don’t necessarily have to have an Ekahau ESS ePerf client, any iPerf3 client will work right?
JO: Um, yes and no. It’s based off of the iPerf 3 code but it’s not identical. It doesn’t have the full feature set of like iPerf3 so it kind of depends on what you’re testing. Certain things will work with it. iPerf3 can be a little finicky depending on your clients and what versions of iPerf3 they support.
KP: So the recommendation is if you’re using a regular iPerf3 client use iPerf3 server, and if you’re using the Ekahau use the ePerf server.
JO: Yep exactly. That’s the recommendation. That’s why on the NanoPi and the ODROID it’s running both the native iPerf3 code as well as the ePerf code. They’re just configured to run in two different ports so that way you can run those tests as needed.
KP: And so with even with this little OLED display or when we had ODROIDS in the past, you can click a button and it shows up your current IP address. What are the things you can hit with that? Obviously you can SSH into it and have control over the Linux box itself. In addition what other ports is it listening on?
JO: Yeah good question. So it’s got the iPerf2 running on its native port. ePerf is running on the native iPerf3 port and then you also have iPerf3 – I don’t want to say the wrong port, I think it’s a 5202 if I remember right. I think the default port is like 5201 so I incremented it by one. And then you also have the zap protocol running on there – the zap package made by ruckus which uses the speed flex app. If anybody’s familiar with that you know you that’s normally running on ruckus APs and controllers. This device can also run those types of throughput tests so if you happen to not have a ruckus Network to test this thing on you can plug this into any other vendor’s equipment to do those SpeedFlex tests as well
KP: And if you don’t have any clients at all?
JO: If you don’t have any clients to do the tests your saying?
KP: Yeah so it also runs HTML
JO: Yes, one of the cool additions from suggestions at the WLPCs was around the idea of enhancing this by adding a web server and doing some cool things with that. One of the things we did was tried to emulate speedtest.net that everybody’s so familiar with to test their “Wi-Fi networks”. I kind of put that in quotes because we know as wireless engineers you’re not really testing the Wi-Fi network. You’re testing a lot of other things when you’re doing speedtest.net. If we want to test just the Wi-Fi we need to have a closed loop. So we put a web server on the the NanoPi running an html5 version of a speed test similar to speedtest.net. You can run those upload and download speed tests on a closed loop to test the actual Wi-Fi throughput. Any client will obviously be able to do that whether you’re on an iPhone, or an Android device, or Windows, or a Mac. You just punch in the IP address of the NanoPi device and you’re able to do that speed test.
KP: What I like it for is when your customers want to see the speed test. You can plug it in, tell them the IP address, they hit it on their phone, laptop, tablet whatever they’ve got and they can run their own tests. And obviously it’s only testing the wireless because we put it right directly behind wherever their AP is.
So that was a handful of throughput testing; iPerf2, iPerf3, ePerf, ZAP, and html5. Five different versions of doing throughput tests what else can it do?
So that was a handful of throughput testing; iPerf2, iPerf3, ePerf, ZAP, and html5. Five different versions of doing throughput tests what else can it do?
JO: There are a couple of things we can do besides just the throughput testing. It has a USB port on there so we can connect some stuff up to that. One of the things we provide is a USB based Wi-Fi adapter. It’s the same kind of client wireless adapter that you could plug into your computer and use it to connect with networks. But with the Odroid and NanoPi running Linux we can do some other cool things. Packet analysis type of stuff for one. We could use it as a client device and we could take that Wi-Fi adapter and connect this thing to a wireless network if we wanted to be able to access it or over the Wi-Fi. To be a little more useful you can plug this thing into the wired connection and then utilize the wireless adapter as some type of monitoring device.
Right now I’m traveling and I can connect via the VPN into my house and access a NanoPi in my house with a wireless adapter. I can link up to Wi-Fi Explorer Pro and use that as a remote sensor to check the Wi-Fi in my house and see how its is performing. I can see what other networks maybe overlapping with mine. I can use that as if I was in my house with Wi-Fi Explorer Pro running but remotely from anywhere in the world.
So that’s one of the functions you could use the adapter for. I don’t know if you want to comment on that at all but I’ll talk about another thing as well .
What Else Was Loaded Into This NanoPi?
KP: I think Adrian’s tool is wonderful we use it all the time. For those people who have Macs and Wi-Fi it’s like the number one go-to tool. And then with his pro edition you get the ability to access a remote session. It could be another laptop that’s talking back but in this case we just have this little NanoPi that does all those features. It’s a wonderful thing.
In addition you have kismet and horst. Let’s talk a little bit about those two options.
JO: Yeah, so kismet is a you know…
KP: and again for both these you do need to have that Wi-Fi NIC connected.
JO: Yep. So kismet has been around for quite a number of years. Most people when they hear kismet they think of this terminal, old-school Linux kind of interface, a text based kind of thing. Well it’s come a long way and they’ve got a pretty cool web-based version of kismet that I’ve been toying around with. It’s still in pretty active development but it’s come a long way. Pretty useful from a wireless engineer’s standpoint. We can put that adapter into a monitor mode, log into a web interface and be able to see all sorts of information not only about the APs in the environment but the clients as well. We can get a lot of information – some statistics about packets and retries. We can see this on an AP level as well as a client level and be able to even gather some information about which clients are associated with which APs and on which channels. We can get an overview of the spectrum of how many client devices are on each channel on the 2.4 and the 5 gigahertz bands.
KP: To recap – your laptop would be connected to the access point that you’re using. Behind the access point is the NanoPi hardwired into its Ethernet port. Then the NanoPi has a Wi-Fi NIC in its USB port and kismet is running on the NanoPi using the wireless NIC and you’re accessing it via the Ethernet. So you’re not coming in on the same NIC that it’s trying to do its work.
JO: Correct.
KP: And horst…
JO: Horst is an interesting one. So The horst acronym means a couple of different things but I always use the Highly Optimized Radio Scanning Tool. It’s a pretty slick one. It is a terminal base so you’ll need to use something like an SSH client to log into the NanoPi. From there it will automatically put the adapter into a monitor mode and then you can analyze all sorts of live packet analysis type stuff. We can see things like retries. We can see both AP and client information. We can filter down to a specific BSSID or a specific device MAC address and we can see how that device is performing, what data rates that client is using and what the retry percentages are. Again, this can all be done remote. So this thing could be deployed as a remote sensor on a network and you could SSH into it and be able to see all sorts of statistics about the RF environment from a frame and packet analysis live packet analysis level.
KP: Very very cool stuff. The NanoPi gives you a little, teeny handheld tool sitting in your pocket, a battery-operated device. It started out just to solve a problem that you had and has come a long way. We definitely want to thank you for the amount of hard work you’ve done in in building it. I remember oh maybe two years ago when I first got started on this with you. I couldn’t do any of the linux stuff. I kept asking, “Oh Jerry how do I do this?” It’s come a long way. We’ll be having a session of this. There’s a deep dive that Jerry’s doing with Scott McDermott on mobile testing and if you’re at WLPC be able to see these little test tools. If you’re interested we’ll put some links in the show notes about what they look like and where you can pick one up.
Thank you very much Jerry for your time today.
JO: Thanks Keith, Thanks for all the support and making all this happen. We’ve come a long ways in it and definitely would not have made it this far with it without all your help and support and getting this happening.
KP: Well we’ll just keep continuing on making it better. Who knows what we’re going to see next year? Maybe it’ll be the size of a thumb drive. But this one is small, light, easy, and very robust. And we learned this time around we’re not having the attendees assemble them at all. It will be preassembled and they’ll just work. We’ll treat it like a little baby appliance.
Well thanks for your time. Enjoy Florida it where it’s warm before you head back home to Wisconsin where it’s probably a little chilly right now.
JO: Yes it is. All right, thanks very much Keith
KP: Thanks
