Colleen Szymanik
Twitter: @wifi_unicorn
Linkedin: https://www.linkedin.com/in/colleenszymanik/
Video Transcription:
All right. So. Colleen Szymanik. Joining me on stage also is Firas Shaari we work for new hardware certification for Comcast. And we wanted to kind of share a little bit of our experience of.
The last year we’ve started testing all the different chipsets that support 11ax. And we kind of wanted to walk through our process and a little bit about our journey and some of the things we learned so we could share with you.
Hopefully, we can get a lot of these issues to interrupt things that we’re seeing fixed sooner than later.
So we’ve been hearing a trend about a lot of data. So the data that I collected is where do you start when you try to boil the ocean with something like this. So we look at the popular applications that are on our network. And you will want to take a guess or fathom what number one is one?
Netflix. So I don’t actually have a breakdown of streaming versus download it or anything like that but that’s number one. Second YouTube. thank you, YouTube.
The interesting stat that I learned from that one was that half of the YouTube traffic is not just the TCP traffic it’s quick. So UDP matters.
So as I’m trying to build in my head what modeling, what environments some setting up. I’m looking at that like well I should have a lot of obviously DNS is on there much further down smaller packets very just small transactional stuff.
But UDP will be important for us to test to make sure we know that it works well and works concurrently. Right. So that’s. Interesting. There’s obviously a lot of video traffic or service provider and a cable company.
Clients. So very popular fruit company. Is probably the majority of the popular clients that are out there. Some of the other statistics so it’s not very surprising or very fun but some of the other statistics that we started building our environment on that I think is kind of interesting is the concurrency. Because the charter of 11ax is the high density.
So how many concurrent devices are connected in a typical home? This audience would not be what I’d model. Because I mean most likely not everyone’s home is going to look like the homes that most of us have. So the interesting stat is that I get.
About ten, eleven devices in like the eighty-five percentile that are concurrently connected about four on to four and about six or seven that are on five gigahertz.
So that gives me somewhere to start. One of the other statistics that I find interesting that I’ve actually been following for a while is I have some. Telemetry data out there that actually gives me the interesting distribution of what the packet sizes are out there.
So while the majority of traffic is in the 15 hundred by packet range. The amount of small packets that are under network has done nothing but grow in the last three or four years.
Which I find interesting. So it went from about twenty-seven percent which I thought was a pretty high number, to begin with. And that’s sixty-four to one hundred and twenty-seven by packet range to thirty-nine percent. That is crazy. And usually we don’t market or we don’t manage that kind of stuff. But that’s something we’re like wow.
So now we’re going to have high efficiency. And what’s the charter it’s supposed to help that bottom 10 percent. So those small packets take those resource units interesting.
I’m setting myself up for the disappointment of expectations but I like to think we have empowerment that might really kind of play well for this new technology. So, of course, we also have to state what we want. So we wrote a technical requirement document, we put a whole bunch of you know table stakes in there about what’s supported today, some thresholds on service and.
How many devices are connected? What kind of service level we’re looking at? What kind of bandwidth requirements do we need? As well as little things like OFDMA, Linkedin, downlink some of the differences that we see with that actually I think that’s more interesting are.
The fact that. Some vendors are chipsets only support 4 some support 8 some support 30 to concurrently OFDMA. Have I tested that yet? No, I have not. But trust the verify. So if it’s where I think eventually we’ll come along the way. But it’s something that I think is interesting that there are differences.
So actually my deep dive which I love so far is on the API side of things. And one of my instructors said yesterday that all the interesting things are happening in software. And as hardware engineers hey now I will say yes. There are a lot of interesting things happening in software.
But if you have poor hardware. Your software is not necessarily gonna fix that poor hardware. So we’re going to get this right. We’re going to do as best we can.
So how am I setting up the test and what do we doing. So we have the popular clients, the popular applications. We have turntables. We have all the things that basically we normalize power.
We normalize Antennas. We kind of just set everything up to say OK this is the same as we could possibly make it for the same track. I got it. Let’s see what the differences are.
So we get some charts and some charts and some charts and some charts and charts and charts and charts and charts and charts.
What does that tell us? Well, some are better at one thing and better at others. So doesn’t really help and I must say that one of the things that I really hated for the longest time was the fact that people would measure success from just bandwidth. Well, I don’t care necessarily about that with if I’m not being asked for a gig of traffic. Doesn’t mean my network’s failing, it’s OK.
I should be able to serve what I’m asked for. How do I do that intelligently? But we over the course of the last year it has a lot of conversations with a lot of QA and chipset vendors and stuff.
And they actually brought up the best scenario that I thought that actually justifies this in my head for being OK. Which is bandwidth is the canary that sings. So it’s not going to exactly tell you that there’s something wrong or that there’s like what the problem is or that it’s failing. But if you get a dip Or something happens.
It’s singing and you better go and dig and have the right tools and telemetry and everything set up that basically shows you like what’s going on.
So you dig and we dug here and one of the things that we found was that this client was dropping so the band was going down pretty poorly. This is a cumulative distribution function and we weren’t getting any NPD to use. We did some packet capture as this stuff turns out that there is an Interrupt problem with MFP. So earlier today we heard from AB and SAE that oh that MFP is mandatory.
Well, it’s mandatory of course for the Wi-Fi certification. But doesn’t mean it’s going to come out enabled and it doesn’t mean that it’s going to work. So these are the trials and tribulations that we’re going through. Let’s. Talk through a couple more of them.
Firas Shaari:
To go through this and really quick. So basically what we see here is actually the endurance test. So like what Colleen said is like true but tests are not the goal. There the mean to get to the head and issues with. So what we see here is actually in AB under endurance test in which we were like pushing 900 almost to the third level.
Nine hundred Mbps on the uplink declined was three by three an 80 megahertz VHT. And for some reason that like we collected the logs and the data and sent back to the vendor. But for some reason, the AB will continue working fine until a certain point in time when you see the drop between the offered load and the what is actually received and the interesting thing that we saw here is basically that the AP kind of you lose all connectivity to the A/B that point.
So like you can cessation to it, you can’t tell that, you can do it, you can log on to the GUI like you basically the AB kind alike it’s frozen.
And one can go back to the original like to a normal state until you stop the load. I mean we’re not sure what’s going on and like we sent everything back to the render but I would if I would take an educated guess I would say it’s not a Wi-Fi issue it’s a more of something like eating up the processing power on this thing.
The thing is like our testing is not only about Wi-Fi we’re not looking at like getting the best chipset only, but like we’re looking at a whole solution that like will be integrated and in our next products.
And it’s very hard to test something without having the right tools for it. And like we don’t have any clients for x we don’t have any tools to look at the back at. So what we can look at it. So what we come up with is like the beacon. Everyone can look at the Beacon even in a client look at the Beacon. So like what we did is actually we capture the beacon for different clients for different ABs and we basically run the analysis for it.
So we looked at like What is supported from a hardware perspective, what is supported from like a roadmap from a software perspective, what the vendor is not interesting and supporting it.
And actually one of the interesting things we find like some of the features that like those ABs were advertising they were advertising incorrectly. I mean I’m not sure if it’s clear but some of the Greenfields with the S on them they have a star on them.
So basically those means that whatever whenever you see those stars means that like we actually saw that the AB was incorrectly advertising this feature this element. And when we actually faced the problem with it, with one of our own a certain AB on VHT. No matter what we did we couldn’t get the client to connect to 160. I know you guys hate 160 but bear with me here.
So 160 short guard interval so no matter what we did, we put them in and we put them in a clean environment and nothing works. And then when I was surprised we went back and we saw actually the AB was advertising that it does not support this short guard interval even though it did like and the vendor said it does.
One of the other things that we so like that we concluded through the testing is actually like the small back aggregation rate. So like around 40 or 45 percent of the traffic on our network is like small packets. So like it’s very important for us to have an AP that is capable of aggregating like a high volume of packets with a small size.
So like this plays a lot on our decision choosing what solution. If you can see like at a certain point the throughput is capped by the Internet because like we’re running traffic like to downlink from the Ethernet to the Wi-Fi. So like what kept them like 900 something this is where all the like if you can see all the columns just like almost they have the same value because of the Internet search.
We had to include that heat map here. Everybody loves the heat map.
So some of the vendors actually claim that like even with the new AX we will see an improved VHT performance around like 10 percent and we were trying to quantify this improvement. So what we did is actually we took the new APIs with the AX and compared them to the old solutions that we have deployed in the field and we run some. I mean our home test solutions we run some site surveys to get like to get our size to get like uplink and downlink through.
But to be fair we saw an increase but like nothing that like we can actually like to see on user experience like it didn’t really like how much of an improvement with comes through these our experience but from the abstracts point of view yes it did look like you can see that like the new chipsets had better performance when it comes to VHT.
This is a very interesting case we had where the actual AB had only an issue with a single client 80 megahertz VHT clients like it were it will work fine with every any type of client but like if you specifically like connect a client with that one special stream 80 megahertz VHT around the vicinity of minus fifty-seven DBM you will see a spike in the back at data rates. And for some reason we still don’t know what’s causing it and what’s the issue, we’re trying to figure out what is.
This is coming from one of our guest’s former distinguished engineers they’ve ever been and we were talking about it and he was thinking that it could be a problem with the LNA the low noise amplifier. So basically what do you have here like when you have an input like to an input signal there is a point where like the signal drops under a certain threshold where like instead of bypassing the low noise amplifier you go through it.
And from his experience, he thinks that this could be an issue like transitioning from that like transitioning from pipe passing the amplifier to going through the amplifier that could be the issue. We still know we’re looking at it and collecting the data and send it to the vendor.
Other tests we’re doing is the RF testing we use the exia chambers where vera waves. So we look at the AB as complying with FCC but I mean we don’t really like we think like vendors do a really good job on this but we’ll just make sure we look at that stuff just double check, we look at like the spectrum mask, we look at the EVM of course EVM is very important for us. It’s a part of our TRD. So we look at the constellations and all different MCS rates.
So we also look at what’s called spectral fatness. So this is an interesting case where you can see this is a three by three special three by three I think VHT 80 megahertz. So if you can see here like the on antenna like the AB best as these bits of flatness tests but it doesn’t actually do that antenna B and C like you can see a little spike going outside the borders here this is where the limits.
Because we have like the average between the sort of the difference between the average power and the power for each subcarrier on the of the M should be around for a fourth DBM plus or minus and then 60 BM at the ends, so like actually, in this case, it kind of like a passes that. Important lessons for a lesson that all things we learned from here like tests and test and test you were here with the AB with every single device you can put your hands on because like you never know which one of them will trigger an issue with the AB.
RF testing is very important making sure that look the physical layer is solid. Before like going to the Mac layer, this is very important.
Thank you. Sorry. I mean we’re ready already way behind our time. Thank you.
