LTE or DSL?

Which Router?

I do not think it matter that much.

This should be a problem on O2 side and I do think they shouls whitelist webex.com, *.webex.com. Other colleagues have the same problem.

Anyway: DSL, Fritz Box 7590.

uPnP active?

The following option is active.

Transmit status information over UPnP

Universal Plug & Play (UPnP) is used to provide connected computers with status information about the FRITZ!Box in the home network. It has no influence on the security settings of the FRITZ!Box.

Is any O2 technical representative here?

I also faced the same issue and spent hours trying to fix this. My company IT help desk also suggested (After spending more than 2 hours to find the root cause, we started uninstalling webex meeting application and installing it again which not solved the issue. )

You have to call your Internet provider and ask them to whitelist webex.com, *.webex.com and open ports 443, 45 and 9000 for TCP and UDP services. here are the references if you need them.
Ports needed.
https://help.webex.com/en-us/WBX264/How-Do-I-Allow-Webex-Meetings-Traffic-on-My-Network#id_135400
Domains that need to be whitelisted
https://help.webex.com/en-us/WBX264/How-Do-I-Allow-Webex-Meetings-Traffic-on-My-Network#id_135010
Just in case, IP addresses needed
https://help.webex.com/en-us/WBX264/How-Do-I-Allow-Webex-Meetings-Traffic-on-My-Network#id_135011

O2 technical team please help.

Joining the webex meeting is one of the most important part of my job.

I have the exact same issue. Cannot use my DSL connection to connect for webex. Are those services throttled?

+1 with the same problem through DSL and LTE.
Some more information:
my company webex link leads me to 

xxxxxxxxxx.webex.com.       600     IN      CNAME   nebulaie.webex.com.
nebulaie.webex.com.     7       IN      CNAME   global-nebulaie.webex.com.
global-nebulaie.webex.com. 10   IN      A       66.114.168.199

I have massive packet loss to this server

--- 66.114.168.199 ping statistics ---
197 packets transmitted, 91 received, 53% packet loss, time 196625ms
rtt min/avg/max/mdev = 191.816/194.253/199.427/1.315 ms

The loss seems to start a bit earlier already:
 Host                                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. fritz.box                                                                              0.0%    12    0.4   0.5   0.4   0.5   0.0
 2. loopback1.0004.acln.04.muc.de.net.telefonica.de                                        0.0%    12   17.0  17.6   9.5  32.9   8.0
 3. bundle-ether11.0001.dbrx.04.muc.de.net.telefonica.de                                   0.0%    12   10.0  10.1   9.8  10.6   0.2
 4. ae2-0.0001.corx.04.muc.de.net.telefonica.de                                            0.0%    12   16.9  16.9  16.6  17.2   0.2
 5. ae20-0.0002.corx.02.fra.de.net.telefonica.de                                           0.0%    12   23.8  18.6  16.6  28.2   3.7
 6. bundle-ether2.0002.dbrx.01.off.de.net.telefonica.de                                    0.0%    12   31.7  18.7  17.1  31.7   4.2
 7. bundle-ether2.0001.prrx.01.off.de.net.telefonica.de                                    0.0%    12   17.1  17.0  16.8  17.2   0.1
 8. 176.52.252.32                                                                          0.0%    12   16.7  17.2  16.6  19.6   0.8
 9. 176.52.248.139                                                                         0.0%    12   29.6  29.3  29.0  29.6   0.2
10. xe-10-2-1.edge6.London1.Level3.net                                                     0.0%    12   28.5  28.5  28.4  28.7   0.1
11. ae-1-3501.ear1.NewYork6.Level3.net                                                    90.9%    12  104.1 104.1 104.1 104.1   0.0
12. CISCO-SYSTE.ear1.NewYork6.Level3.net                                                  72.7%    12  120.5 120.6 120.5 120.9   0.2
13. jfk10-wxbb-crt02-bu60.webex.com                                                       33.3%    12  120.4 119.7 119.4 120.4   0.4
14. jfk10-wxbb-crt02-bu60.webex.com                                                       27.3%    12  117.5 119.2 117.5 120.0   0.8
15. ord10-wxbb-crt02-te0-5-0-2.webex.com                                                  63.6%    12  140.4 140.4 140.0 140.8   0.4
16. ord10-wxbb-crt01-bu10.webex.com                                                       54.5%    11  137.3 138.6 137.3 139.7   0.9
17. sjc02-wxbb-crt01-bu101.webex.com                                                      45.5%    11  191.5 191.7 190.7 192.6   0.6
18. sjc02-wxbb-pe02-bu12.webex.com                                                        72.7%    11  194.4 193.5 192.8 194.4   0.8
19. sjc02-wxp00-gw02-po522-100.webex.com                                                  45.5%    11  195.6 195.5 195.1 195.9   0.3
20. ???
21. global-nebulaie.webex.com                                                             63.6%    11  192.7 192.6 192.2 192.8   0.3
 

Hello @Nunzio Visciano, @ArunBalakrishna, @AugustinB and @dnsroland, welcome or welcome back to our o2 community :-)

I am sorry to hear there are issues using Cisco WebEx from home, thankfully you already provided valuable information with the attached traces to the server.

If the connection to WebEx works, no matter how slow and with how many disconnects that means that within the whole route from your computer to the destinated server every setting is correct. If there were any blocked ports or blacklisted hosts you would not be able to reach the destination in any case at all.

As you can see in the provided trace, up to hop 10 (xe-10-2-1.edge6.London1.Level3.net in London) the traffic flows just fine as it should. At this point the traffic is already out of our network and is transported by a Tier1 provider, Level 3 Communications. 

It is on hop 11 where things start to go south. As soon as the traffic arrives on the East Coast (New York) you see a spike in packet loss and high latency. This continues up to the traffics destination.

I just ran a trace to the destination and still recieve a high latency but the packet loss itself has stopped (except on two gateways that don’t seem to answer to every ping, but that’s okay).

I dug a little deeper and found this information pointing to an outage of Level 3 that affected internet traffic globally to any service that relied on this provider.

I hope that the situation has stabilized and you are able to use your WebEx meetins again from our side there was nothing that we could have done to improve this, luckily it seems to be resolved :-)

Regards,

Lars

Hi,

I can confirm that it works for me again (mobile and fixed), also the packet loss is gone if I retry what I did yesterday.

THANKS Lars!!

br
Roland

This is just to remark an experience I had yesterday: everything was workig fine in the morning. I nerverthless agree with the analysis about the packages lost outside of the O2 network. This problem could be definetly closed. Thanks to both of ou Lars and Roland.

It is anyway better to wait this afternoon to declare this really closed.

Hi Lars,

the problem is back right now. WebEx is almost unusable. Soon It will start to be completely unusale. If I connect to my meetings with my company mobile (we do not use O2) everything works perfectly.

I would consider the problem still open and the lost packets a coincidence.

Regards,

  Nunzio

yes. I also just noticed that the problem between ?Spain? and L3 or L3 in the UK and the US is back again….
@o2_Lars Maybe it’s worth bugging your upstream provider for an alternative route?

 Host                                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. fritz.box                                                                              0.0%    20    0.4   0.4   0.4   0.5   0.0
 2. loopback1.0003.acln.04.muc.de.net.telefonica.de                                        0.0%    20    9.3  27.1   9.1  62.0  17.5
 3. bundle-ether10.0002.dbrx.04.muc.de.net.telefonica.de                                   0.0%    20    9.6   9.7   9.2  10.4   0.3
 4. ae3-0.0001.corx.04.muc.de.net.telefonica.de                                            0.0%    20   28.7  22.5  15.8  36.5   5.6
 5. ae20-0.0002.corx.02.fra.de.net.telefonica.de                                           0.0%    20   16.2  16.3  15.8  17.8   0.5
 6. bundle-ether1.0003.dbrx.02.fra.de.net.telefonica.de                                    0.0%    20   16.3  16.6  16.3  17.0   0.2
 7. bundle-ether1.0004.prrx.02.fra.de.net.telefonica.de                                    0.0%    20   16.3  16.5  16.1  16.8   0.2
 8. 176.52.252.28                                                                          0.0%    20   16.2  16.7  15.8  18.9   0.8
 9. 176.52.248.121                                                                         0.0%    20   28.5  28.7  28.1  29.3   0.3
10. xe-10-2-1.edge6.London1.Level3.net                                                     0.0%    20   27.7  28.0  27.6  28.8   0.3
11. ae-1-3501.ear1.NewYork6.Level3.net                                                    89.5%    19   99.6  99.8  99.6 100.1   0.4
12. CISCO-SYSTE.ear1.NewYork6.Level3.net                                                  61.1%    19  122.8 122.6 121.6 123.6   0.7
13. jfk10-wxbb-crt02-bu60.webex.com                                                       72.2%    19  122.3 123.2 122.3 123.6   0.5
14. jfk10-wxbb-crt02-bu60.webex.com                                                       63.2%    19  122.7 122.9 122.3 123.5   0.5
15. ord10-wxbb-crt02-te0-5-0-2.webex.com                                                  73.7%    19  139.8 139.2 138.6 139.8   0.5
16. ord10-wxbb-crt01-bu10.webex.com                                                       63.2%    19  135.3 135.5 134.9 136.3   0.5
17. sjc02-wxbb-crt01-bu101.webex.com                                                      57.9%    19  193.0 192.7 191.9 193.4   0.5
18. sjc02-wxbb-pe02-bu12.webex.com                                                        72.2%    19  190.9 191.9 190.9 193.1   0.8
19. sjc02-wxp00-gw02-po522-100.webex.com                                                  77.8%    19  193.0 192.4 191.7 193.0   0.5
20. ???
21. sjc02-nebulaie.webex.com                                                              68.4%    19  191.3 190.5 189.8 191.3   0.6
 

to add:

this is one from the competition.
It looks like they would get a better service ;-)

 

 Host                                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. easy.box                                                                          0.0%    54    0.6   0.6   0.5   0.8   0.1
 2. dslb-084-059-211-001.084.059.pools.vodafone-ip.de                                 0.0%    54    9.6   9.5   9.0  10.0   0.2
 3. 92.79.245.104                                                                     0.0%    54    9.8   9.9   9.5  10.4   0.2
 4. 88.79.10.176                                                                      0.0%    54   11.3  11.3  10.8  12.0   0.3
 5. 92.79.202.90                                                                      0.0%    54   11.4  12.4  10.6  73.9   8.5
 6. 145.254.2.181                                                                     0.0%    54   11.8  16.3  11.3  94.1  15.4
 7. ae4-100-xcr1.mue.cw.net                                                           0.0%    54   11.7  11.7  11.0  13.8   0.4
 8. ae10.pcr1.fis.cw.net                                                              0.0%    54  104.5  98.6  97.0 115.9   3.2
 9. ae37-pcr1.ptl.cw.net                                                              0.0%    54  108.6  97.3  96.4 108.6   1.7
10. et-7-1-0-xcr1.nyh.cw.net                                                          0.0%    54   96.9  96.7  95.7  98.2   0.5
11. ae13-xcr2.nyk.cw.net                                                              0.0%    54   96.8  97.5  96.5 104.3   1.4
12. linn-products-pri-gw.mrs.uk.cw.net                                               30.2%    54   97.7  97.7  97.1  98.4   0.3
13. CISCO-SYSTE.ear1.NewYork6.Level3.net                                              0.0%    53   97.5  97.6  96.9 100.0   0.4
14. jfk10-wxbb-crt02-bu60.webex.com                                                   0.0%    53   97.6  97.7  97.0  98.3   0.3
15. jfk10-wxbb-crt02-bu60.webex.com                                                   0.0%    53   97.7  97.7  97.1 101.6   0.6
16. ord10-wxbb-crt02-te0-5-0-2.webex.com                                              0.0%    53  141.3 141.0 140.5 142.7   0.3
17. ord10-wxbb-crt01-bu10.webex.com                                                   0.0%    53  117.9 117.7 117.1 118.4   0.3
18. sjc02-wxbb-crt01-bu101.webex.com                                                  0.0%    53  166.4 166.1 165.3 167.0   0.3
19. sjc02-wxbb-pe01-bu11.webex.com                                                    0.0%    53  166.4 166.2 165.8 167.0   0.3
20. sjc02-wxp00-gw02-po512-100.webex.com                                              0.0%    53  167.7 167.3 166.9 168.3   0.3
21. ???
22. sjc02-nebulaie.webex.com                                                          0.0%    53  167.5 167.5 167.0 169.3   0.4

I am more focused on algorithms less on networks. Is a throttling visible out this this analysis. Are we sure nothing changed on O2 side having noticed the significative bandwidth use by this service as many of us work from home?

To me it looks more like a problem at o2’s upstream provider (or the upstream-provider’s upstream-provider).

VF does not have the problem, but they use a different route.

I do not think this is a traffic shaping on purpose (and it is also not within o2’s network).
 

as the problem is gone this morning again but the route doesn’t seem different, it could also be some congestion in the evening our TZ? (at L3 in London → NY? OR Spain → London)

Hello all,

going through this thread it seems like issues start around 2 to 3 PM which would translate to 8 to 9 AM East Coast time. I can imagine that at this time many working days start with a meeting which may lead to a congested network an the WebEx side.

Does anyone of you ,maybe has a traceroute at hand from a point in time were it was working fine? I wonder if the traffic was always routed to the US, since WebEx also provides two European WebEx locations.

@dnsroland could you also provide a traceroute over our competitors line as soon as you notice a degredation in service using our DSL connection?

Regards,

Lars

the webex instance of my employer is in the US so it won’t use the european ones unfortunately.

I’ve started a cron-job to have the above recorded every 10min and will look into it in the afternoon.

I can also see to get a trace or at least ping to that server from the US to the US.

Again, thanks for looking into it @o2_Lars 

yesterday everything was fine except a smaller glitch around 1pm.

I’ll leave the jobs active throughout the weekend and Monday.

But fingers crossed; it might be gone.

for me everything is working today as well, I don’t see any bigger delay.
I’ll keep watching that but for me all is fine for now.

br
Roland

I was too optimistic…

@o2_Lars below are the tests when I first noticed that problem today again:

o2:

VF:

btw: it is not up to 100% in my samples today, but still ongoing

Hello @dnsroland,

I had a look at the traces you provided, first of all thanks for that :-)

All traces had the hop CISCO-SYSTE.ear1.NewYork6.Level3.net in it so it seems there is no way around Level3. I just ran a new trace to global-nebulaie.webex.com and found that the route changed. Instead going from London to New York it now runs from London to Washington. So it seems Level3 noticed issues with the route over New York.  It seems also the route inside the Cisco network has changed significantly.

In the old traces it ran from New York (JFK) over Chicago (ORD) to San Jose (SJC), now it’s running over Washington (IAD), Dallas (DFW), Los Angeles (LAX) to San Jose (SJC). It seems Cisco likes using the IATA airport codes of the city where the router is located in. :-)

Could you have a look at the stability of the WebEx sessions? Did they get any better regarding video and audio quality and stabilty?

Regards,

Lars

looks good so far, thanks for that detailed look and for your help