Anatomy of a CyanogenMod OTA

by: Bogdan PetrovanJanuary 22, 2014

cyanogenmod nexus 5 boot screen aa 2

The CyanogenMod team explains how over-the-air (OTA) updates work and why it can take a lot of time to patch bugs on CM devices.

Starting from an issue with the O-Click not pairing with the CyanogenMod edition of the Oppo N1, community manager Abhisek Devkota explains in a blog post how the team fixed the bug, put it through quality assurance, obtained Google’s approval, and finally rolled out the fix to anxious users.

While the bug required just a “simple fix”, as Abhisek put it, the process of validating and rolling out the fix was lengthy, even with the help of the “incremental OTA” update that the team built into the CM Installer app and the CM Oppo N1. This system allows developers to update just the parts of the software that need changes, saving time and reducing the size of OTA packages.

Code > QA > CTS > OEM > OTA

After the fix was applied to the CM code, the QA team took over to ensure the problem was solved and that no new bugs made it to the build. After the QA green light, the patched build went through Google’s Compatibility Test Suite, which is a set of automatic tests that determine if the build is compatible with the Play Store and offers a quality user experience. Because the test suit takes nine hours to complete and is quite “fickle”, it can take days for a given build to pass CTS. As a reminder, the Oppo N1 was the first CM device to pass the CTS and obtain Google’s official approval to run its package of licensed apps.

After QA and CTS, the OEM (in this case Oppo) and sometime Google need to green light the update, following which the OTA can finally roll out to users. According to Abhisek, the patch took three weeks to move from code to devices, which may seem a lot, until you consider all the complex and time-consuming procedures that need to be completed.

Even so, Abhisek Devkota says that managing support and updates for the Oppo N1 is a challenge that the team “enjoys”. The experience and new skills should come in handy once that pure CM smartphone from OnePlus launches later this year.

  • Luka Mlinar

    Google was saying how they wanted things like CM to happen and how they are looking forward to it. This is what they said when Facebook rolled out it’s POS so i just assume they are no hypocrites. Anyway, both Google and CM need to work on speeding this process up. It needs to work like clockwork. With that they need more testers working around the clock in 3 8 hour shifts. testing it by the book from top to bottom and in every scenario. Car manufacturers got really god at this trough out they years. Now i don’t know how their testing process worked but apparently there are holes in their system since the O-Click didn’t work from the start. There are things you can’t half ass and mistakes shouldn’t be tolerated, especially when you are on the verge of starting a new movement (not sure how else to say it).

    • Luka Mlinar

      So just to clarify. It needs to be “by the book” any by saying that i mean a long process of check lists. Not by a “what comes to mind system”.

      • bob

        google fucked up from the very beginning catering to everyones needs, both carriers and advertisers. thats why we now have half backed, infringement-burdened (oracle, likely to win), inefficient virtual machine os which even google is trying to get rid off while pushing chrome os, not to mention samsung and others…

        4 cores at 2.3 ghz with 3 gb of ram aint enough to run android smooth and efficient. try running windows inside windows, thats how android works…

        what were they thinking?

        let me tell you, smth along this lines: lets crawl into everyones ass so we can have a huge platform for selling ads and peoples privacy.


        and through the updates problem, this becomes crystal clear… why did google cater to carriers while apple didnt? why apple rolls ota updates to every costumer in the world in a few minutes?

        because google doesnt care about the end product, they just want market share to sell adds.

        to me, that is not acceptable. i like to buy from companies that consider me a costumer, a buyer, not just end user.

        btw, hello to my fellow croatian, or smth else, certainly ex-yu :D

        • MrMagoo

          So what you’re saying is, if you had a business, you wouldn’t try to everything in your power to help something succeed when everywhere you looked people were telling you you couldn’t and it wouldn’t work. That if you had a vision, a goal, and a passion, you would do what you had to do to make people happy and make a shit ton of money? OOOOOOh and I forgot about your comment on the roll out to the world in minutes!!!! hahahahahahahahaha You do remember that didn’t go as planned right. people waited days cause they couldn’t get it to download. People I work with that have them had all kinds of problems, not to mention IOS 7 has been patched numerous times since release…. on ONE device syle! Google has Hundreds of different devices to worry about. AND YOU SLY DOG! YOU’VE GOT ME MONOLOGUING! ugh, if you hate android so much….why are you here?!?!

          • bob

            if i had a business, first i would make smth i think is the best. if that worked out, i would be apple. if it didnt, i would just copy somebody who is sucessfull, like google did.

            but the point is, i wouldnt suck everybodys dick just too enlarge my marketshare. i would do what i think is the best.

            similar as people behave. i dont like people who want everybody to like them. i like genuine people. and i like genuine companies. companies that stand for smth they think is the best. not like samsung, lets throw a billion products out, lets see what sticks. i cant have respect for such companies.

            yeah, minutes, every time. maybe there were some server problems…

            android users have to wait for half year for updates

            google says devices should be supported for 18 months. ios 7 came out for iphone 4 too, thats 2010, 36 months?
            and you still respond to me. get real. google is fucking you in the ass while catering to advertisers. why isnt youtube in html 5 yet?

            i want to pay with money not privacy

          • MrMagoo

            So what you’re saying is, you troll sites of a system you despise just to throw out you’re your words that make you feel superior. You come to a site that has nothing to do with your love and you shit all over it! Oh and you like to suck dick!

            Meanwhile, Google doesn’t rape everyone for their devices and gives you a venue to produce whatever you want because it’s open source. Google gives the ability to be free and get software from wherever you want. Get music from where you want. Have devises available to you that have storage, and when they don’t, they don’t rape you for $100-200 for and additional 16-32 gigs of storage that costs pennies in comparison. You hate Google and I hate the take it or leave it attitude of Apple. The only difference is that I stay off the Apple blogs and you come piss all over ours.

            Have a good day dick head!! Go troll somewhere else!!

          • bob

            and google isnt take it or leave it?

            ‘we’ll gather and remember for years every action you take on our services to better serve the needs of our costumers (advertising companies)’

            the problem is, i can evade iphone, but i cant evade google search

            i find great comfort that the biggest internet company you just cannot evade if you want to use the internet makes money solely of my actions, interests etc. on internet.

            i have to pay with my privacy because the rest of the world are cheap or poor fucks

            at least give me the option to evade google. and apple tried with ‘do not track’ option, but google evaded it and is now in a legal process, likely to lose.

            the only argument you can give me is money – its cheaper!

            well, let me tell you smth. price of a product doesnt make a product better or worse, it only tells you if you can buy it or not

            so overpriced hardware products ftw. i would rather lose my money than my privacy. i can always make more money

            and im not here to troll, im here to read the tech news, which are sadly nonexistent on this site, apart from old/new snapdragon 805 which is being crushed by a7, half the clock, half the cores, the iris scanning and 2k displays on mobile

            google and oems, a happy crowd, selling our data because they are aware that their so-called costumers dont wanna pay real money for anything. open source haha that ought to be good


          • MrMagoo

            Anyone would be a fool to not understand where each company makes it’s money. Apple makes it off the hardware I know this. Apple makes amazing hardware, and there is no dispute in that. I also understand that they do whatever they can to keep their products incompatible with others. I understand that in doing so, if you want compatibility in the household you need to purchase all Apple products. Sounds like a bit of a tie down to me. Everyone has friends with i products. I was with some this weekend. They had a video on their iPad. We wanted to watch it on the tv. It was such an amazing hassle to make happen it almost wasn’t worth it. They ended up uploading to a cloud so I could play it on my phone and miricast it from my phone. Next day I come to work and all the guys are like “you need an Apple TV to do that, that’s all”. Why? Cause Apple chooses to make sure you spend more money with “them” to make it convenient. Privacy nowadays is a joke. You talk about privacy like there is a person in corporate Google sitting there watching what you do. Nobody cares what porn you masturbate to! It’s all algorithms! People give up their privacy all day long on Facebook and other social cites around the world and what they don’t give up the government is taking anyway. Google’s trying to make the future of the world happen today, just like Apple, and make it so when it does happen, they were a part of it, just like Apple! I look to Google to put the ability of enjoying the finer things in life without having to pay out of my “broke as Fuck ass” for it.

          • bob

            im not sure thats the case

            they make their product incompatible to implement their technology they think is better

            lightning connector is miles ahead of mini usb. for on thing, double sided

            apple tv streaming, they were practicaly the first to implement this kind of thing

            facetime, miles ahead, picture quality and bandwidth

            imessage, mp4, etc

            what youre proposing is that they just give their technology, technology they invented and patented to other for free so everyone could use it. that just cant happened. they need to diferentiate themselfs, both through software and hardware, and they cant do that if they are giving their technology to others.

            google can do that, because google isnt selling to us, they are selling our data to advertisers, so their only interest is large marketshare. ofcourse theyll be open source. but i just cant respect that business model. i dont like to buy from companies that dont consider me as a costumer

            so, apple sells differentiated hardware to YOU, software is free and if you have everything apple, everything will work great

            google couldnt do that. they needed to conquer marketshare quickly, so they just gave everything to everybody for free. and what you get in the end?

            you get a gazillion oems, everyone with a different version of android, you get old tech, like mini usb for the sake of inter-oem compatibility, you get 1 year of waiting for updates, you get the OS that isnt optimised to run on certain devices, but an os that needs to run on every device (hence java). basically you get windows, and thats crap

            company that doesnt have vertical integration cant make great products

            thats why samsung is developing tizen. they need their own os so they can tailor it to their devices

            and theres just no other ecosystem like apples. if you have everything apple, everything works great. but its a little expensive

            the point i was talking about is that google makes money solely on my privacy. do you understand that? and in that regard theyll probably be more inovative than in android. i just don like that

            remember, nothing is for free. and if im paying, than im paying with money, nothing else

            and people just need to get out of their love affair with money. money is for spending. if you dont have enough, mobile phones can last for more than a year. buy better but less frequent.

          • Azeem

            There is no point in updating a phone that can’t even run an os, which is what Apple did. They screwed over those iPhone 4 owners. Go back to your unreasonable ways. If you don’t like Google or Android, you aren’t welcome here.

          • bob

            it runs fine with 7.0

            it runs great with 7.1

            try putting android 4.3 on 2010 flagship. thats your OS

            fucking java virtual machine

            if people here got a clue how inefficient android really is, and what a mess it really is, no one would defend it

        • Luka Mlinar

          “because Google doesn’t care about the end product, they just want market share to sell adds” this seams to be true :/
          And on about having a hard time running Android. Tell me about it; my machine has a hard enough time running bluestacks. I was surprised how much it took. Every time i turn it on, my CPU cooler switches to max power.
          Lol yea, a Croat. Pozdrav i tebi druže :D

          • AssToast

            Because android isn’t made to run on desktop processors and neither the apps are to

    • MrMagoo

      Well I’m not sure how else to say it…. but I’m pretty sure it’s not like you just did. To think rolling software out for a “phone” is put at such a high priority to have people working around the clock, 24 hrs a day, is going a lil far. In the end, it’s not stopping anyone from dying, it’s not fixing world hunger, it’s not giving us world peace, and it’s not necessary for life. I think people should be able to go home and sleep at night. It’s just O-Click dude. It’s for entertainment, that’s it.

      • Luka Mlinar

        First of “rolling software out” is what they do. It’s the only thing they do. Secondly; I’m not saying people should work 24/7, I’m saying they should work fast to fix a fuck up.
        Hey man, not everyone knows how to run a company.

  • Alan Shearer

    Three weeks?? THREE WEEKS?? lol They still beat all oems by more than half!