Once upon a time in a business far far away, my job was to take assist in the development of a major release for a SaaS enterprise platform. This platform already had a non-trivial amount of users and so far, it had been developed sporadically by a consultant/contracting company.

Naiveté

I show up on day one expecting access to some type of VCS somewhere, maybe a short call with a core engineer about how to build and run the environment and the service. Who knows, if they were really nice, they might even introduce me to some of the interesting concepts in the domain and walk me through all the major building blocks that one should know when working.

Instead… silence

Offensive Spell #1 - Bureaucracy

Well, turns out, I was not authorized to access the code. Sorry - let me make a correction - I was allowed to access the code, but I was not allowed access to the repository.

But wait, what? Wasn’t my company paying these people to write code… for them? Well, yeah, but that code was just a wrapper. The system was a Java/React application, built on-top of an ERP/CMS platform that the consultant company customized for their customers. The implementation layer that my company owned and paid for was a wrapper around this CMS, built on top of its data model and underlying functionality which relied heavily on the use of Oracle database functions to do the actual dirty work.

All of that is fine and all. Personally, I would not have made the decision to build my system using these pieces of technology but they probably got it cheap and quick at the time thanks to piggybacking on their solution. But the partnership was structured in a way where it was essentially impossible for my employer to boost the velocity of development without going directly through the consulting company.

I got around this for a while by getting my manager, who had access to the repo, clone the code onto his machine and send it over to me. Then I got around this “for real” but pestering the consultant project manager every day, at every stand-up, for around a month about my access to the repository. Eventually, I got it - after already having made several contributions to the code base.

I am sure many reading this relate this incident to typical IT access requests inside a large corporation, where getting simple access to something requires navigating large amounts of red tape. This situation is a bit different however - my employer at this time was their customer and we already had one person in our organization with access to the repo. The specific thing that the consulting company wanted to avoid was having a developer having access, one who could make contributions and push code.

Offensive Spell #2 - We have the keys to the car

Great, now I can clone the codebase and inspect at what my employer had paid for. But what I cannot do is actually run the damn thing. The core of the system, the ERP/CMS that the product was built on top of, was a closed-source proprietary JAR that we had to download from their artifact repository. However, we were not actually allowed to download it.. this meant that a core component required to run the system, was literally not in our hands at all. This made running a local environment impossible, let alone having our own staging/test environment hosted somewhere.

Once again I began beating at the proverbial door of the consultant project manager in charge of working with us. Every day, every stand-up, the same blocker and the same request: “I cannot run the system locally because I cannot download OhSoCoolProprietaryTech” - “When can I download OhSoCoolProprietaryTech” - “Still blocked by not being able to run OhSoCoolProprietaryTech”.

The developers on the other side of the fence though, they were great. Eventually after some back and forth between the managers, I managed to get my hands on a copy of an older version of the JAR that I could use to bootstrap the system.

Oooh yeah, finally, I’m going to be able to run this bad boy.

Offensive Spell #3 - Oracle DB

So, the system used Oracle DB as the database. When trying to setup the system, I was surprised to learn that getting an Oracle DB license cost something crazy, like > $30,000. For my employer, a small company, this was a completely unfeasible cost. Talking to their solution architects for help with this problem resulted in me entering a meeting with 3 very senior and very competent engineers who tried to pitch us an expensive, isolated deployment of the application that we would be able to access via a point-to-site VPN that they would manage for us. Again, very expensive and not exactly what we needed. I was the only developer at the time after all.

Luckily though, Oracle provided the same database via their cloud hosting solution. Same database, much lower price. So I did some hacking and I got their platform working against a cloud-hosted instance of the database. My local environment could connect to this database - run all the bootstrapping stuff (scripts to setup the database schema, initialize some seed data, etc) - and then we could go ahead and finally get working.

Offensive Spell #4 - We cannot let you commit to our codebase

As the consulting company was responsible for the operation of the platform (it was hosted on their infrastructure, with their stack, and developed by their engineers after all), they initially were quite negative about letting in-house developers contribute code to the codebase. To some extent this was the most relatable and reasonable outcry they had expressed; theoretically I could make some terrible changes that would ruin the night of whoever was responsible for ops that day.

In practice though, this was almost impossible. We had no access to their dev/test/staging environments, and we also did not have any authorization to control the deployment systems. The consultants basically had all the opportunity to review, test and validate changes before they arrived in production.

Changing the narrative

Eventually I realised that what the consultants really feared most was that we would take over the entire development of the software platform. To be fair, this fear was not unfounded - we actually were planning on doing just that. But this should have been viewed as an inevitability by the customer success manager from the consultants, after all, if they worked with us to make sure the handover and migration went as smooth as possible, we would likely reach out to them when we needed to bolster our development.

With this realisation, I proposed a release pipeline that would surely generate billable hours for the consultants. In a nutshell, we write code locally in our environment, and upon completion we would hand over the code to them. At this point, they could take all the time they needed to go through every line of code and make sure that these changes would not blow up the data center. They were happy to accept this as they were able to allocate any of their engineers to review and validate our changes. This was great as they had the flexibility to allocate whoever was available at the time to review incoming changes, and we finally had a way to get our code to production.

Moral of the story

This post is mostly a big vent from me. At the time it felt very frustrating that the biggest obstacles between me and delivering to production were not technical, but organizational and political. It did however take all of the technical knowledge I had to be able to understand what was feasible and what was not - if my non-technical manager had to deal with their offensive spells, I believe every single one would have been effective.

The biggest takeaway from me was understanding the underlying motivation behind why these organizational blockers were being put in place. Once I grokked that they wanted more billable hours, I was able to create a workflow and process that suited their needs. Due to these actions however, my organization quickly came to the understanding that we should take ownership of the technical stack and own the keys to the car ourselves.

My advice to the consultants: work with your customers and not against them. The relationship between my organisation and theirs suffered greatly due to the artificial constraints that they applied to reduce development velocity and create more billable hours. To be clear, the engineers on their end were great and very competent - this was purely a managerial decision that caused us a lot of pain and bad vibes.