ISP Health scoring seems different after newer versions

[KittyFarts]

Hi TJ,

I had a question about how ISP Health is scored/measured.

A couple versions ago I was usually seeing my ISP Health around 82-84, but now it sits around 50-54 and does not really recover.

Previous Live View:

Current V1.21.0 Live View:

The part that is confusing me is that nothing major changed on my network. Same ISP, same gateway, same general setup, and from what I can tell the targets/paths are mostly the same too.

The main Live View stats also do not look terrible to me:

• ISP RTT is still around 9-10 ms
• ISP loss is 0%
• Transit RTT is around 46 ms
• Transit loss is 0%

I tried running discovery again a few times. I also removed/cleared the paths/targets and ran discovery again, but the ISP Health score still ends up back in the low/mid 50s.

So I’m wondering:

• Did the ISP Health scoring logic change in the newer versions?
• Is the score now weighted more heavily against discovered targets/transit health/congestion findings?
• Could the same targets that previously gave me ~82 now score around ~54 because the formula changed?

I’m just trying to understand why it changed so much when my network itself has not really changed.

Thanks!!

[tvancott42]

You're kind of burying the lede here haha. What does the full ISP Health report show? Click the score card or check the Monitoring -> ISP Health tab. Also you don't want to delete and readd targets usually as that will cause you to orphan data. If certain targets are falsely lossy due to ICMP deprioritization, just disable them with the toggles in Latency Targets and that prevents them from being added again in upstream discovery.

The full ISP Health report will show you what's up though.

[elevul]

Weirdly enough mine went from "Poor" to "Fair" since the last update 🤔

[tvancott42]

Weirdly enough mine went from "Poor" to "Fair" since the last update 🤔

This makes more sense with the revisions I made. Or it's incidental. It is based upon the last 48 hr snapshot. I'm working on some features to call out and score total outages too, and also detect what layer the outage was at.

[KittyFarts]

You're kind of burying the lede here haha. What does the full ISP Health report show? Click the score card or check the Monitoring -> ISP Health tab. Also you don't want to delete and readd targets usually as that will cause you to orphan data. If certain targets are falsely lossy due to ICMP deprioritization, just disable them with the toggles in Latency Targets and that prevents them from being added again in upstream discovery.

The full ISP Health report will show you what's up though.

Oops, my bad LOL I didn’t want to start off strong by dumping a ton of giant screenshots into the issue 🤣

I grabbed the full ISP Health report like you mentioned. But since the v1.21.1 update - Live View is now showing ISP Health = 0, but the live/idle stats still show around 10 ms ISP RTT, 0% ISP loss, around 32 ms Transit RTT, and 0% Transit loss.

I can see the findings for loaded packet loss/congestion, so that part makes sense now. What I’m mainly trying to understand is why the overall score is going all the way to 0 when the report sections themselves are not zero. Access Layer is still in the 80s, and ISP Network / Transit Health are in the high 40s.

I had no idea removing those could cause orphan data. I already deleted/rebuilt targets a few times before seeing your reply, so that might be playing a part to the weirdness.

Is there a way to remove/reset that orphaned data, or clean up the latency/path history or would this be a 'wipe the whole app' type of situation?

Here's a screenshot of my ISP Health reports:

(network was down for about an hour on 6/16 as I was power cycling the entire network - idk what those other outages are that are logged)

[tvancott42]

Oh it thinks you had a bunch of outages when those targets were disabled and came back up. I need to make sure to time box those and also throw out outage candidates if the target wasn't on. Once you get 48 hours of data it'll all go back to normal.

Those Comcast targets are wild. You can probably disable as many of those as you want unless you find them interesting. I'm really surprised that many routers turned up in the trace.

[tvancott42]

Oh I see it better now. I think the targets that got readded after the outage caused the outage to be registered in several chunks instead of just one event. I'll look into this.

[tvancott42]

@KittyFarts this should be cleaned up in the release that's about to drop. But do note the reason for your overall score drop are the distant hops on the Comcast ASN. The scoring isn't lying, the delta between your local access network and some of those metros is really poor. But, if your real internet host latency is under control, then the best thing to do is just turn off the poor scoring Comcast targets. The reason why they score low is they're high latency versus your local IX targets. Which correctly indicates Comcast's own backbone kind of sucks compared to Tier 1 providers.

[KittyFarts]

Awesome thank you TJ!

I just installed the latest and greatest and I see ISP score of 15 but I'll let it sit for a while and not touch anything:

Also, I notice on the bottom 'Per-Network RTT (48h) that some of the targets are cut off:

Thank you for the super quick remedy, now I'll go hunt the poor Comcast target.

I am already ticked off with Xfinity, I ordered 2G/2G symmetric download speeds recently but they are trying to find out why I'm only getting 2G/250Mbps.... :sigh:

[tvancott42]

@KittyFarts I've got some major improvements coming here, that will help you hunt down flaky targets, and it reshapes how I score on DOCSIS and on larger intra-ISP backbones like Comcast.

Score should be coming up and becoming more meaningful soon.

You can also rename targets during Upstream Discovery if they're ridiculously long. Renames will adjust existing targets, keyed on their IP.

[KittyFarts]

@KittyFarts I've got some major improvements coming here, that will help you hunt down flaky targets, and it reshapes how I score on DOCSIS and on larger intra-ISP backbones like Comcast.

Score should be coming up and becoming more meaningful soon.

You can also rename targets during Upstream Discovery if they're ridiculously long. Renames will adjust existing targets, keyed on their IP.

You are simply the best, thank you kindly!!

Idk if you want this ticket closed out or not so I'll leave it to you!🤪

[tvancott42]

Let's keep it open for tuning this, your WAN is definitely an excellent case for me to continue adjusting things if needed.

I have a bunch of GPON/XGS-PON sites, but nobody on Xfinity anymore.

New release, ETA ~10 min will have all of the new features I was mentioning.

[tvancott42]

Also @KittyFarts you'll see that 1.1.1.1 Cloudflare DNS target classified as ISP/Access layer disappear automatically, re-run the upstream discovery to get your real first hop. This is due to UniFi Network WAN SLA's ping targets populating the ARP table with 1.1.1.1 and 8.8.8.8, so upstream discovery was mistakenly picking those up.

[elevul]

Stupid question: does the ISP monitoring specifically send out through WAN1 or if the Gateway is in Load Balancing mode the packets are sent out of the interface the gateway decides?

I wonder if my ISP Health changes are because I changed the gateway to load balancing and the VDSL on WAN2, while MUCH slower than the DOCSIS on WAN1, has better latency and stability.

[tvancott42]

Stupid question: does the ISP monitoring specifically send out through WAN1 or if the Gateway is in Load Balancing mode the packets are sent out of the interface the gateway decides?

I wonder if my ISP Health changes are because I changed the gateway to load balancing and the VDSL on WAN2, while MUCH slower than the DOCSIS on WAN1, has better latency and stability.

Not at all. Currently it will monitor with probes however it routes out of your Network Optimizer box. But the correlation to throughput stats comes from the WAN interface that's at the top of your load balancing list in UniFi Network.

You can effectively tell which connection is chosen as the UniFi Network expected ISP speeds get pulled in and displayed in ISP Health.

I'd suggest a PBR to stick your NMS traffic to one or the other and then make sure the top WAN connection is the one you want to monitor.

Multi-WAN support is coming but it'll be a feature that requires an multiple IPs to be bound to the NO box so probes can be routed using PBRs to the desired WAN. But I'll automate that. It'll be the first time NO modifies anything in UniFi Network (and not just gateway customization layers) so I'm not in a particular rush on that.