Best practices from billions of calls

so I want to talk to you about the best practices from billions of calls some lessons I learned from basically reverse engineering some major VoIP services I work friend yet I walk occasionally at Weber DC hacks so you’ll find a lot of country then today for me so I’m a mechanical engineer I like to build things but actually I prefer deconstructing some reverse engineering them a lot more so back in May 2014 chat did a article on Amazon made a on the web RTC hacks you can read that out there it basically looked at how Amazon made a worked and it was pretty important at that point that we saw that WebRTC was ready for production and used the weapon shield library and following that I did these construction of hangouts when hangout started using WebRTC and I looked at Firefox hello and at some point I got an email from Sarah saying oh you’re done great tear downs and he wanted to look at this shift to mobile we’ve seen with a party seat back late 2014 early 2015 and we wanted to identify some best practices and so I spent a couple of weeks looking at various services I wrote 80 pages of reports on that a couple of blog posts lots of nodes and some tooling for that and basically looking at services that have billions of calls whatsapp facetime facebook Messenger wire viber and skype and we learned some lessons about adoption to the mobile platform about codec tuning about encryption on those heavy loss 2g and 3G networks and about ice so I want to talk about that so Chad used Wireshark in his deconstruction of mayday and vert shock is one of the best tools out there for decoding Network packets for analyzing stuff and if you know how to use it it’s very powerful however sometimes you don’t notice things so Amazon made a used stun server and that information has been in this dump we had since then and nobody noticed that it uses this net net OS a sing which turned out to be an acne session border controller and also Wireshark as good as it is it is not very good at visualizing things so I’ll come back to this graph later and sometimes you need better tools that can generate nice graphs that people love so I looked at things like FaceTime which is from Apple it does voice chat and video chat it works on iOS and OS X and no surprise it’s not using WebRTC however it is rumored to be big at least that’s what people tell me and they have a couple of years of deployment experience so it’s been around since 2010 and before that it was ichat AV so lots of experience so they should know what they doing well we found that they using h.264 they support h.265 however they don’t use h.265 when being on Wi-Fi that’s something where you can see that h.265 it is better in terms of bandwidth however it consumes more battery so if you have enough bandwidth don’t you use h.265 and they negotiate the image size between devices so they send the optimal resolution doing that with your SD p and of course they use turn service run by ahmed distributed a bump around the globe for low latency and we found that they’re using a very fast mechanism for the candidate a location which skips some round trip times and that reduces the call set-up time and as we’ve heard earlier this call set-up time the time it takes to establish the call is very important for user experience so I also looked at whatsapp which does voice calls and it is pretty big as well so they’re not using WebRTC they’re using an old library called pj sip they don’t do this modern ice stuff it don’t use detailers instead to use the older security description encryption scheme and looking at the binaries we found that they might use some WebRTC components like the echo canceller however they don’t tell us if they do in those license would require them so the audio codec has been a mystery for quite some while so he said oh it’s opus but I showed him this graph and basically shows how the packet lengths are distributed and you can see on in black is chrome on our party sees sending opus and what’s up is definitely looking different so recently is someone commented on where / DC hacks and said oh yes i did a man-in-the-middle attack on the signal link traffic and it’s using opus at a different sampling rate which explains why this looks different and one of the most interesting things I found was that the call is always relate for the first few seconds Emil told me about that that this will be a great idea and I found it to be true for whatsapp and if they can switch to peer-to-peer they go to peer-to-peer to take the load off see turn of the service they have so they’re using service they call a conference bridge 50-something in ireland and again this reduces a call set-up time so I also looked at viber which is doing voice and video chat it’s not using repartee it might be using the older mobile IP solution stuff and it’s not using this RTP protocol so it’s completely encrypted so when sash asks about it I asked him do you really want me to look at that because I wasn’t sure what I could find in that and it turns out that you can still see all those traffic patterns like this graph which shows that basically is a switch from relay service to peer-to-peer which we’ve seen with what’s up before I also looked at skype encrypted as well however at least they show some debug information so we can see it’s using silk white band as an audio codec it uses h.264 video and it was summer in Sweden such had lots of time we did some calls and we were testing on simulated 2g 3G and 4G networks and we found that skype really dynamically adapt sings like the sampling rate the packet frequency and the bitrate to those network conditions that’s very impressive what we also found that despite those of us being in Europe it used relay service in redland which doesn’t make sense to optimize one thing and not the other we also looked at wire which does voice calls text chat and picture sharing 11 Android app an iOS app and they work in chrome and firefox and they use opus and we were interested in how they adapt opus to the mobile platform so the browser interop is using WebRTC they’re using details for encryption with very strong encryption mechanisms and they really partially using WebRTC dork but not for networking like Firefox doesn’t use it for networking easy you’re using the media engine and their own fixed version of lib opus and you can see that they have killed the audio codec massively so the bitrate in chrome black and put the app in blue so that you can see the bitrate is a lot lower in the app also you see that in this packet distribution saying so it’s shifted to the left and when testing going to simulate the 3g network we saw a drop from the usual 50 kilobit per second traffic to 25 and going to 2g it dropped again to like 18 kilo bits per second so this is heavily tuned for a good audio experience and this is very hard to do however they know what they’re doing they were involved with the I said coded with opus oh these are people who know what they’re doing really so I also looked at facebook Messenger which is audience video chat ten percent of mobile voice over IP and it is pretty apparent that it uses with WebRTC ugh libraries they have a nice notice file showing that they use it works on android iOS Chrome Firefox Opera and in the browser chat asked me to look at that back in January and I told him it’s boring what do you want me to look at that it’s pretty common however on mobile it was completely different that’s heavily tune their app from well usage and you can see that in the distribution you can see that they send lots of small packets and lots of large packets so adapting to those network conditions they see and they use a lot of different audio codecs and negotiate that through SDP for example they use the Isaac low-complexity codec which is virtually unknown outside google see I said codec when talking to chrome and opus tuned for monal when talking to Firefox all handle through ftp and for encryption this is a complicated story so when talking to browsers which must implement DTLS they used ET les but between their apps a used the older security description mechanism and that allows retroactively decryption which is a bad thing however it is faster than detail s in some conditions so in summary we learned quite a lot of best practices from those services and also things that are not so good if we are really interested in all the gory details although 60 80 pages go to where / DC hacks so we saw a lot of adoption to the mobile platform which is networks with high packet loss and delay devices with constraints on cpu and battery usage and we found that audio is way more important than video and that 11 sessions are more important than multi-party and encryption on heavy lost networks is a problem because if there is packet loss the Detailers handshake takes a lot of time however while you can work around that using Esther’s please don’t do that we’ve also seen a lot of codec tuning that is using different codecs lowering the bitrate increase the frame size of the frame sent using motor instead of stereo and we’ve seen some hardware acceleration for example with facetime however facebook Messenger uses vp8 on iOS and software one of the interesting things we saw was that with ice a lot of people go for the relay first so the turn server is not the last resort that’s the first thing you go to that reduces the call set-up time then they switch to peer-to-peer if they have successful that way you could also hide this dtls latency in the ring latency that is the time until the user picks up the call so thank you if you want to follow me on twitter i’m home speak on flower or you go to WebRTC hacks to read about stuff right there thanks at all do we have any questions yes I think you mentioned the beginning that you need better tools than a wire shark for looking at the stuff did you write something all yourself for year it was I was writing staff based on note pcap then generating those graphs with the highcharts library it is a small set of tools but it is very useful other questions search any suggestions on the next tool ripple and I should test well we can compare browsers ah that should be internet go I have a quick question of all the things that you found what was the most surprising to you well I think cee at the what’s up went for this relay service first I mean Emil had told me about that quite a while ago and when I found it I immediately thought oh yes he said that and it was good to see this use in practice well FaceTime is different they use a very complicated architecture combination of SIP and XMPP and they establish a data Channel and then do a sip session over that and it must be some very Akane reasons to do that right very old architecture you probably know more about that but you don’t want to talk about

Read More: Side Part Hairstyle Tutorial | Classic Men’s Side Part | How I Style My Hair