The filter list was manually generated with an external tool, which takes input from the full official IEEE OUI vendor list, and filters all OUIs which are local or group adresses (bit 40+41 of MAC) and all non mobile devices like wifi routers.
This is of course a wild estimation. Could be improved. The tool should be placed in the repository in separate dir /tools if license grants this. To be done.
Seed value for initializing the pseudorandom generator is now taken from 2,4 GHz RF noise of ESP32, so should be true random
New version 1.2.91 out now!
Thanks again to @Charles for contributing:
- enhanced RGB LED blink-blink for incoming Wifi/BLE packets (see updated readme)
- serial EEPROM MCP 24AA02E64 support (deveui reading)
- counter now shows sum of seen unique Wifi+BLE devices, filtering out double (hashed!) MACs so that same devices found on Wifi + BLE are counted once only
And, never ending story, some minor fixes…
2 Likes
@Charles There must be a bug came in with the new code. Counter display now stucks at 16. Device with older software counts much more.
Humm, we changed lot of things, including Hash & Salt of MAC address. I’m currently trying to understand what all is doing. I’m going to push a new salt & hash, waiting for your comment once done.
Anyway, may be we could have new define to avoid hash & salt and store full MAC (I know not good for privacy but we need to have a working reference to compare). Let me 1H between lunch to see what I can do.
Hey on your picture, what’s the 142 BLE devices seen around and fixed to 16, that’s strange, may be we have a real bug
Let me finish my change please (1H), it’s really a mess Github Fork&Sync for my repo while we work together everyday on source code
take your time, i keep my fingers off the code today!
1 Like
The 142 BLE could be true, i was in an environment which is known for having ~800 beacons from a marketing company installed.
Hmm, i re-thinked this. It could be correct, if the vendor filter filtered out the most of that 142 BLEs because they are stationary BLE beacons, not mobile devices. That would be the desired result.
But then we still miss the Wifi counts which the other device showed up. In worst case reason for this is, that the hashing consumes to much time and we have packet loss on the device with hashing.
I will go to the busy area again and make a second test.
I tried to optimize hash & salt.
Would you mind take a look to the new PR?
I’ll check it.
Meanwhile i did a second test in a very densy area. Definately a bug: Counter seems to stuck at 16.
Yeah I changed things on display and done proper casting of counters. should works better, thanks for testing
May be enable/disable filtering could be a config option instead of a compilation one?
Merge done, now i see counts > 16 again!
Did not understand the bug, but we have better code now
Thanks again for contributing, you’re much faster in C/C++ programming than me.
BLE scan seems pretty fast now, i a high dense area the counter is running like hell.
Wifi scanning takes much more time.
So having both is our USP.
Yes, filtering could be a remote configurable option, but then the huge vendor array table will always cost flash, even if not needed. So, maybe having both is best:
- keep vendor filtering as compile option (to have small code if flash memory is limited)
- remote configurable on/off (e.g. does nothing if filtering is not compiled).
Thanks, and you’re welcome
Would be interesting to see what the output log in the dense area to see if it’s a display bug or other.
Logs will show all counters values in realtime
I will try to write a log tomorrorow.
We need to check two things:
-
no packet loss in dense areas -> by comparision with device which has the old software without hashing, but needs 2nd same hardware board with same RF / Antenna performance
-
check that hashing works without collisions (= does not give same hash for different MACs) even in dense areas -> otherwise we count too less
Numbers seem to be correct now on display. But still have to go a test in a more dense area.
But the display garbage problem still persist, see picture.
funny, this odd bug seems not to produce pixel garbage, but characters.
Looks like something with u8x8 lib, maybe the SSD1306 driver is buggy or not fully compatible with the display controller.
Or data transmission issue on I2C bus?
One more thing: I’m not 100% sure but i believe my LoPy RGB LED is flashing red or orange, not magenta, while scanning. But the BLE scan runs pretty fast, flashing is very short. Maybe it just looks like not magenta, but is.
Would you mind trying new PR? I think the bug is gone
Avoid display in Wifi Callback, in BLE looks like no problem.
aah, that looks good, working still with u8x8.
I merged the PR to branch test and put in on my TTGOv2.
Will test today in dense environment.