The Price of Secrecy

05 May 2018 / Andrew Vaziri / Comment

You may notice the gap in posts on this website between March 2016 and May 2018. Such a gap is common among personal blogs. People get busy, and they move on. However in this case, my absence was not my decision for the majority of that time. I was asked by my (former) employer to refrain from posting on this blog, uploading hobby videos to Youtube, or appearing on the non-profit podcast I volunteer with.

In May 2016 I started working for Apple's Special Projects Group. I left that role at the beginning of this year. But don't get your hopes up, this is not a tell-all by a disgruntled employee. First off, I am not disgruntled. I left on good terms, and if I had a time machine, I would do it all over again. Secondly, there is nothing secret in this post. And finally, the purpose of this post is to help others avoid pitfalls related to secrecy that Apple had to navigate during my time there. When I was hired, my project had a broken secrecy culture which ultimately required a series of corrective actions. I divide the improvements to the secrecy policy into three buckets:

Focus
Balance
Clarity

Focus - Asking hundreds of people to keep total secrecy on projects that span many years lowers adherence. I saw that first hand. Because my project was a particularly poorly kept secret it was very common for non-Apple engineers to ask point blank questions to Apple engineers. In this situation you are supposed to say "No, no, no. I work on the new Jet-Pack" or "I can't comment". But often, people don't want to be fake to their friends and they decide to acknowledge the project. With that admission they violate the official policy of total secrecy. As the conversation continues they decide where their personal secrecy threshold is. This normalizes the behavior, and the next time their personal secrecy threshold becomes a little more lax. Apple addressed this in two strokes. First, they relaxed the secrecy guidelines for things that truly don't need to be secret. Secondly, they made a clear list of what few things were the "crown jewel secrets" of the project.

Balance - Some things have moderate business value as a secret, but are quite valuable when shared openly. One such thing is academic research, especially in Machine Learning. This is one area where Apple made a particularly big improvement over my time there. At the beginning, Apple did not publish papers publicly. Furthermore, many internal machine learning teams were completely unaware of each other's work.

To unify Apple's internal machine learning community they held an Apple only machine learning conference which drew a few hundred engineers from across the globe to Cupertino. I was scheduled to present a 15-minute sanitized presentation that did not identify my team. At the very beginning of the day's events my boss's boss's boss's boss, the keynote speaker, made an impassioned speech about when the need for collaboration trumps the need for secrecy. During that speech he announced the existence of my team for the first time to employees outside the project. To paraphrase he said "Today marks the beginning of a new transparency-first approach. No more secrets for the sake of keeping secrets". Ironically, following that speech the event organizer instructed me to continue to conceal my team affiliation unless I had a personal confirmation from the keynote speaker to be transparent. That anecdote aside, the conference had the desired effect and there was a much more robust discussion between ML teams afterwards.

Regarding external academic research, Apple took its first steps by hiring Russlan Salakhutdinov as AI Director and publishing a few very well received papers. Those papers provides potential employees with some expectation of the quality of work going on behind the scenes, and it keeps current employees steeped in the creative ideas they need to push their fields forward.

Codification - Although very few people read this blog, or watch my videos, such things are important to me. They provide an opportunity for me to practice communicating technical topics in simple terms. Nowhere in my contract did it explicitly bar me from posting on my personal blog, or making hobby videos on Youtube. However, there were some rather broadly worded guidelines about public statements, and being a rule follower I asked for official clarification.

I was frustrated when the news came back that my only path forward was to get approval from many levels up the chain of command. I went to my boss to start the process and she told me to "pick my battles". Despite that, I persisted and continued the process which went part way up the chain of command before stalling. I went to my boss again to try and get the process rolling. It was during that conversation that my boss succeeded in convincing me that I should really abandon my request. The phrase "at-will employee" flashed across my mind. That's how a vaguely defined policy becomes a "culture of fear". By the end of my time at Apple there were much more well defined concessions regarding what should be allowed. People were able to update their LinkedIn accounts with fairly straightforward information about the skills they used in their work, as long as they avoided saying what the product was. Apple even circulated an official list of phrases you could use to describe the technology and it's applicable markets.

So why did I leave if they addressed the issues? Well, there are many reasons to decide that a career change is in order. Early on at Apple, I had the opportunity to work on Reinforcement Learning and I had grown to love it. However, over time my role drifted. Near the end, spending quality time on RL became the exception rather than the rule. I felt that I needed to prioritize RL or put it aside. I did not see any open RL roles within Apple, and finding an external opportunity would be a challenge. The entirety of my career in RL had been conducted as part of a secret project. I had helped write a paper at Apple but it was never published. The source code was complete, and the paper was 70% written, but no one had enough time to collect all the experimental results. After two years of work I had no published papers, no projects I could discuss, no patents and no source code I could show.

Ultimately I decided to focus on reinforcement learning. I applied to a handful of companies that see reinforcement learning as the core of their business. I poured a month of nights and weekends into a project that used 100% open source DRL frameworks to demonstrate my abilities without using any Apple resources. During interviews I asked if I could present the research of others since I had none of my own.

Luckily my fears were unfounded and I did not have a hard time demonstrating my RL chops. I received multiple offers and I have been with bons.ai for five months now. I have been able to spend 100% of my time focusing on making a scalable, and easy to use AI platform with Deep RL at it's core. I am really excited by the success we have had in solving real business problems for customers. That level of practicality is not something you hear about often in reinforcement learning. I think bons.ai is very unique in that regard, and I will expand on that idea in my next post.