After 6 years and 6 months my last day at Shopify was May 10th 2024. My parting was bittersweet. I’d made a lot of friends over six years at Shopify and I was the second most tenured employee on the database platform team so I think some of the less senior employee’s looked up to me as a bit of a “old timer”. On the day I left I had worked for Shopify longer then 91% of the other employees at the company.

Working at Shopify was a great experience for me and I’d do it again in a second. I joined a company that was freshly public and had under 2,000 employees. I rode the growth train to over 13000 employees and got to live through the post COVID hangover were the company shrank back down to around 9,000 employees.

The most fun project I worked on at Shopify was the build out of the “Shopify Canada” region. It was basically a specialized instance of Shopify deployed in Google Cloud’s “northamerica-northeast1” region to host the stores that were going to sell cannabis online in Canada. It was interesting because this was 2018 and Jeff Sessions and the Trump administration were pushing against cannabis legalization in the United States and here I am helping build out one of the largest legalization experiments in the world. Its a bit of a funny memory.

When I first joined Shopify I was shocked at how accessible the executive leadership was. The weekly all hands was often lead by the CEO (Tobi) and he would answer re-time AMA questions with coherence. I was impressed with how accessible the leadership was and I was really excited by the work.

I had four really good years at Shopify and then two really crappy ones. 2018 - 2022 I worked on numerous database projects that I thought were interesting and engaging. Then things went badly starting in 2022.

At the end of 2021 the NoSQL database project I was working on was cancelled by executive fiat. I was given the opportunity to move to a new team but I choose to move back to my former MySQL database team. This was a pretty big mistake as I just didn’t gel with the departments leadership or direction. I spent the next two years kind of running in place. It didn’t help that these two years were pretty terrible for Shopify. A 10% layoff in July 2022 was the warning shot. Then the massive 20% layoff and sale of the logistics business in May 2023 was the big hammer. In my infrastructure organization there was a 37% layoff. I think I barely survived it. After the May 2023 layoff it took the organization at least six months to recover from the shell shock. I’ve learned layoffs are hugely demotivating to the workforce and Tobi’s leadership through these layoffs began to show some real weakness.

After its 2020-2021 boom years Shopify struggled. The executive team became a clown car of chaos:

• At the end of 2020 Craig Miller the long time CPO/Chief Marketing officer clearly had a falling out with Tobi and abruptly left the company.
• In early 2021 nearly the entire C-Suite was cleaned out (CTO - JML, General Console - Joe, and VP of HR -Britney).
• Tobi then hired a new CTO (Allen Leinwan) at the end of 2021. Allen was gone by January 2023.
• Amy Shapero (CFO) left the company in Fall 2022

Internally there was a lot of noise the logistics business was a mess. I read internal documents that indicated that to compete with amazon we were going have to spend $5-10 Billion dollars in shipping subsidies to offset our high per package cost. Clearly this didn’t work out.

It’s clear that Tobi is hard to work with. It seems that very few employee’s can tolerate working for him very long. I felt that Shopify had a lot of internally developed engineering talent when I joined but it struggled to retain this talent because eventually top flight engineers would get the eye of Tobi on them and very few would stay with the company long after this. The second hand feedback I heard was that Tobi was demanding, impatient and had a lot of preconceived notions on how things should work but was difficult to move off those positions. This lead to Shopify hiring a lot of “free agent” Principle+ Engineer types to backfill the internally developed talent that would leave. While these individuals were probably pretty talented they would struggle to assess “the battle ground” that they had just parachuted into and often struggle to make significant contributions.

I remember meeting a new employee in May 2021 who remarked that I had been at Shopify through “hyper growth”. I truth I never really noticed this.
The system architecture of the Shopify Core application laid out in 2014 more or less scaled out with fairly shallow changes over the next 10 years. Sure we moved the application from co-location data centers to Kubernetes in GCP but fundamentally the system was not re-architected in this time.

From 2020 on at Shopify the company developed a lot of anti-patterns.
I saw a lot of managers doing a lot of hiring for hirings sake. One of the easy ways to become a director was to hire as many people as you could and then make the case that you should be be a director now.

Tobi would become deeply involved in fairly low level technical decisions. For example I remember Tobi complaining about how we changed schema in MySQL, which in my opinion was actually a fairly inconsequential thing.

After the sale of logistics Tobi focused on trying to build a very top down organization where all projects were reviewed by Tobi himself. It was somewhat crazy that Tobi was reviewing a project to update the version of our database proxy. This lead to so many anti-patterns where engineers would either avoid submitting new projects or just do work without review. A lot of the teams became fairly dysfunctional because you felt like you could trust your managers judgement about what to work on. It felt as if all innovation needed to come form Tobi. Unfortunately I got the sense there was a executive team around Tobi that was focused on enabling this behavior rather then helping modify it.

In my own little database world we built out a Database as a Service platform called “KateSQL”. It is basically a Rails application that runs an Ansible like task framework called “tomas” to orchestrate a number of K8S resources for running MySQL on Kuberenets. I never bought into the KateSQL architecture or purpose. I felt like having a web API to build MySQL was redundant to the services offered by Google Cloud and a risky spend of engineering time as eventually executive leadership would find the KateSQL system redundant to services provided by some third party (Probably Google). Building and maintaining KateSQL was a huge cost to Shopify. I estimate we spent around $15 million in engineering staffing costs building KateSQL and it was going to cost around $7 million in engineering staff annually to maintain. This doesn’t incorporate all the opportunity costs we had as we were re-inventing something we could have purchased from our cloud supplier. I’ll be interested to see how KateSQL plays out for Shopify. I suspect they can make it successful, but I’m betting the negative on this one.

For me Shopify was a challenging place to work. I really had a to level up my skills a lot of feel like I belonged. There were a lot of really impressively smart people there. I’m very appreciative of my old Boss (Sergio Roysen) who for some reason gave a random sysadmin a “production engineer” job at a growing tech company. This opportunity changed my life and I’m thankful for my time at Shopify. Working at Shopify made me feel great. Like I was successful, but oddly it did not make me feel happy.