We released the WLAN Pros Toolbox to the public on July 7, a free set of 170+ tools for Wi-Fi professionals. Four days later, we set out to prove it wrong. We went through the whole app, tool by tool, and we found real problems. Here is the good, the bad, and the ugly, along with what we did about every bit of it.
The one that stings
Ping Sweep scans a network segment and tells you which devices are alive and answering. It was calling dead ones alive. A device that was actually down and unreachable came back as responsive.
Think about what you do with that. You trust the tool, you cross the device off the list, you skip the investigation, and you go looking somewhere else while the thing you needed sits offline. A tool meant to keep you out of trouble could have walked you straight into it.
The fixed tool reports how many hosts answered on the port it probed, and when none of them do, it says so and refuses to draw the conclusion for you. Hosts that are ICMP-only, or that firewall the port, will not appear. Silent hosts may still be up. A scanner that cannot see a host has no business calling it alive, and it has no business calling it dead either.
The app argued with itself about your signal
This one is quieter and it goes deeper.
Our analyzer graded your RSSI on one set of thresholds. Our own Signal Thresholds reference screen showed you a different set. The same dBm reading got two different grades depending on which screen you happened to open.
At -73 dBm, the reference screen told you the signal was fair and usable. The analyzer, looking at the identical number, told you the signal was weak, that you were at the edge of coverage, and that no faster internet plan was going to touch it. One of those answers sends an engineer up a ladder to move an Access Point that was fine where it was.
The reference screen carried values I had reviewed and corrected myself. The grading constants behind the verdict engine were never updated to match. Nothing in the code tied the two together, so nothing ever complained. Both surfaces now read from one shared source, and a test fails the build if they ever drift apart again.
That is the finding I would want a working engineer to take from all of this. An app that contradicts itself about the same measurement cannot be used as a reference, and being a reference was the entire job.
We showed our work, and our work was wrong
A handful of tables in the app named their sources right there on the screen. Those turned out to be among the most wrong things in the product. One printed its standards document under the data, and its numbers did not match the document it cited.
A citation nobody pins to the value it justifies does not make the data right. It makes the data look right, and it stops the next person from checking. Ours had exactly that effect for the life of the product.
The mechanism is the part worth taking back to your own work. The numbers were not invented. Every one of them was a real published value, correct for a different input than the one printed beside it. Every sanity check we knew how to run came back clean, and internal consistency was preserved by the very nature of the bug.
A number can be self-consistent and still be a lie.
The only check that catches that is a comparison against an external primary source at a named input. It was the one check we did not have. We have it now, and we went through the rest of the reference tables and corrected what we found there.
The tests were green, and they could never have caught it
We had a large test suite and it was all green.
Our tests were generated from the code. They proved the port was faithful, that the app computed what its own source data said it should. They could not prove the source data was ever right, and the source data was wrong.
The signal-grading tests were the purest version of the problem. They derived the dBm values they expected from the very constants they were testing. A test built that way cannot fail. It can only agree with you.
I was double-checking. I was checking the wrong thing.
The new tests assert against the standards documents and the manufacturer datasheets, never against us. And one of them now checks the two signal screens against each other, so the app can no longer disagree with itself in silence.
The good, and the platform that is still broken
We published exactly what was wrong. No burying it in a vague “performance improvements” note, no spin. If a number was broken, we said so, and we said by how much.
The corrections are live on iPhone, Mac, and Android.
They are not on Windows. Our Windows build machine was reimaged and the build key went with it, and we cannot rebuild that platform until I am physically back at the machine. Windows users are running the pre-audit build right now: the scanner that calls dead hosts alive, the grading that argues with itself, all of it. There is no version of that sentence that sounds good, and I am not going to dress it up. We are telling you instead of letting you find out.
Why we are opening the whole thing up
We are going a step further. We are relicensing the entire app under AGPL-3.0 and opening the source. That decision is made and the license is already committed to the code. What is left is cleaning up the repository history, and then it goes public. Two reasons, both about you.
First, you should never have to just take our word for it. Once the source is public, anyone can look inside and check our numbers for themselves. Trust you can verify is worth more than trust you are asked to assume.
Second, AGPL keeps the Toolbox free and open for the working professionals who use it, including in their paid work. A “non-commercial” license would have locked out our own audience, the exact people this was built for. And it makes sure any improvement anyone makes stays open for everyone else too.
The one we missed
Then, on the night before this article was meant to publish, I found a worse one. On my own phone.
I was on cellular. No Wi-Fi at all. I opened Test My Connection, the tool at the front door of the app, the one somebody reaches for first when their internet feels slow. It showed me a Wi-Fi signal card with a green LIVE badge. It showed a Wi-Fi data rate of 29 Mbps. And it handed me this verdict:
“Your internet can carry more than your Wi-Fi link is passing. Boost the Wi-Fi signal to raise the ceiling.”
It told a man with no Wi-Fi to go improve his Wi-Fi.
The 29 Mbps was a real number once. It was the last reading that phone ever took, from the last time it had been on a network. The app kept showing the last number it had ever seen, and it put a LIVE badge on it.
The cause was a gate. The honest “You are not connected to Wi-Fi” state only appeared for someone who had never captured a Wi-Fi reading at all. Once you had seen one, dropping to cellular did not clear it. The intent was kind. Do not blank out a user’s data over a brief signal drop. The effect was dishonest. The app presented a number it did not have.
Two things about it are worse than the bug.
It was not new. It shipped in earlier releases, which means it was live on our own front page the whole time we spent auditing ourselves and feeling good about it.
And our own audit missed it. The audit swept reference data, the tables and formulas sitting behind every number the app prints. This was a state bug, a different animal. A state bug is about what the app does when the thing it is measuring is not there. Our audit asked whether our numbers were right. It never asked the harder question: is the app honest when it has no number at all?
We fixed it. And we held this article until the corrected build was live, because publishing “we audited ourselves and fixed it” on top of a live bug on our own front page would have been the exact behavior this piece argues against.
What this comes down to
We say we care about accuracy, and we say we care about this community. Anyone can say that. This is what it looks like when you mean it and nobody is forcing your hand. You audit your own work harder than anyone else would. You admit what you find. You fix it in public. And you hand people the tools to check you.
The Toolbox is not perfect today, and we are not going to pretend it is. An entire platform is still running the broken build and we just told you which one. The audit was not a thing we did once and finished. We are still looking, and we expect to keep finding.
When a tool hands you a result, how do you know you can trust it?