Category Archives: .NET

SD&D 2022

I had the chance to attend the Software Design & Development conference at the Barbican Centre this week. It was my first conference since the plague so it was a new experience. I will here coalesce some of the main points that I gathered. I may go into specifics in a couple of topics that stood out to me, but this is mostly so that I have some notes to aid my memory.

Overall

The conference is very well organised, you can tell it’s not their first rodeo. From a practicality point of view the Barbican is equally easy/cumbersome to get to from all directions, which is about as good as you can get in London where usually you will end up favouring proximity to a subset of main line termini, thus making the location cumbersome to get to for at least half the population (since there are airports in every direction out from the city). There were a number of timeslots where you truly had FOMO for choosing one track over another, which is a design goal of a program committee – so, well done SD&D! There was some unfortunate setting in the AV equipment which interfered with shortcuts in Visual Studio in interesting ways, but surely that can be addressed somehow.

Kevlin Henney

The first law of developer conferences states Always Catch a Talk by Kevlin Henney if Available. It doesn’t teach you anything about a specific new thing, but it puts the entire universe into context, and always inspires you to go ahead and look up papers from the olden days that describe modern phenomena.

C# features

My biggest bafflement came not from the talks that showcased new C# features, such as the latest iteration of pattern matching and the like – as I usually get introduced to them when they show up as options in ReSharper – but by the old abomination that is default implementations on interfaces in C#8. This talk by Jeremy Clark was an eye opener. It is so jank you wonder how it could ever be released into production. My guess is that the compatibility argument from MAUI was the big reason, and it makes sense, but basically my instinct to stay away from it was right, but my guess is that you will have weird bugs because of it at some point in the future.

I also caught the C# Channels talk. I seem to recall the gestation of Channels, as if I remember correctly it was publicly brought into being through discussions on David Fowler’s twitter account (unless I misremember). Anyway, it is a highly civilised way of communicating between two async tasks with back pressure. Intuitive to use and safe. Like a concurrent queue but a lot nicer.

Micro services

Allen Holub had a number of talks on Microservices and if you can I’m sure you should see more of them, due to the abundance of other brilliant talks I only caught his test driven architecture talk which I will mention below. Other than him though there were talks by Juval Löwy and Neal Ford that covered architecture and micro services. I caught Software architecture foundations: identifying characteristics by Neal Ford and Sander Hoogendoorns’s talk on migrating to microservices in small steps, both worth watching.

Security

There were a couple of security talks focussing on automating security as well as the updated OWASP top 10 chart of threats as well as a couple of talks by Scott Brady about – I am guessing – – Identity Server. The dizzying array of choices meant I didn’t go to the identity server talks, but they are definitely on my to watch-list if there is ever video from this.

The security automation ones though Continuous Security by Kim van Wilgen and Add Security into your Agile Process by Cecilia Wirén took you through what you need to really improve the security posture of your development process. Kim v Wilgen was more in-depth on tooling whilst Cecilia Wiren was more about the whole process and what to consider where. The only sustainable way forward is to automate things like dependency vetting/tracking and as much static code analyisis you can, to bring these concerns as far left as you can, menaing early in the process. Rewriting a function even before you commit it is easy. Having an automated fuzzing tool discover you have a buffer overrun vulnerability when you thought you were done is worse from a cost-of-remediation point of view, but of course letting a bad vulnerability out into the wild is an order of magnitude worse, so whilst catching them in the dev cycle is preferable, attempting to catch stuff late through more heavy handed automation that is too time consuming to run on every commit is still worth considering running periodically as it’s better than the alternative.

Tests drive everything

How to stop testing and break your codebase by Clare Sudbery was an amazing talk that really hit home. It was like an experiment of what if I just skip test-first for a bit and see what happens? brought out of time crunch, tiredness and a sense that it would be a safe trade-off because I know what I’m doing and – in her case – I have acceptance tests . Like I have noticed in various side projects, when you let go of the discipline the drawbacks come at you hard and fast – almost cartoonishly so. It was one of the most relatable talks I have ever attended.

Allen Holub’s DbC (Design by Coding):applying TDD principles to architecture was fascinating. It started out by being a bit “old man yells at clouds” about how real agile is index cards on a board, not Jira, and although yes, index cards or post-it’s on a physical board is preferable to an electronic board, the electronic board prevents me from having to commute four days out of the five, so – no. The points raised about authoring million detailed tickets ahead of time being a waste though, hard agree, and the rest of the diatribe against big design up-front I was all aboard with and probably to some extent already doing. The interesting bit was to come though.
He presented a hypothetical technical problem that needed architecting. Instead of drawing a diagram or writing specs, he whipped out an editor with his favourite version of junit and wrote some java code through TDD – all in one file – that implemented tests and classes that symbolised microservices and their endpoints, as well as the interactions between them. Light weight, easy to read for developers, pleasant tooling and at least as useful as a diagram. In both cases you start writing the code from scratch, but you have a design document that makes sense. I’m not 100% sold on the concept but it’s worth considering.

Various highlights

Tuesday morning keynote was about quantum computing, and reluctantly I must concede that it probably is the future, and the community seems to be looking for converts, but I just can’t go from quantum entanglement to taking data from a web page and shoving it into a database, so I guess I have to wait those three years before it hits the mainstream and it will become digestable for the likes of me. If you are into cryptography as in encryption, not the various ponzi schemes, you should probably get into it now.

I caught a Kate Gregory talk – on naming – after only being a fan off of her YouTube talks on C++ vs C, definitely worth seeing. She proposed the strategy of “just mash the keyboard if you can’t think of a name and go back to it later” was also brought forward elsewhere on the conference, the point being: don’t get stuck trying to come up with a name, start writing the code and as you start talking about what the thing you just wrote is and what it is responsible for, a great name will eventually become evident and then you use that. The effort of coming up with a good name is worth it, and with refactoring tools it’s worth just moving past the instant rather than making a bad decision. Once you have finished the feature and proceeded, the caveats and difference between the name and the actual implementation will fade into obscurity and when you come back to the same code in three weeks you will have forgotten all about it and can be misled by bad names as easily as if you hadn’t written the code yourself.

Conclusion

I really enjoyed this conference. Again, with a past in a program committee I really admired the work they put in to cause so much anxiety when picking talks. Obviously the QE2 conference centre in Westminster is newer so NDC London benefits from that, and if you want to see Troy Hunt and the asp.net core guys Fowler and Edwards you would be better off over to NDC, but SD&D had all the core things right and a wider array of breakout sessions, and infrastructure like the food was a lot less chaotic at SD&D than it usually is at NDC. Compared to BuildStuff and Øredev – those conferences I only attended as visitor to the city, so I didn’t have to commute, meaning of course that’s nice.

If you like the environment you’ll be pleased that there was no shilling or abundance of obscure t-shirts handed out that would have drained natural resources, but I suspect that SD&D would have enjoyed more sponsorships and an expo floor which allegedly they have had before . Since this conference was actually SD&D2020 postponed several times over the course of two years, it is possible that SD&D 2023 will have pre-pandemic levels of shilling. All I know is I enjoy free t-shirts to clothe the child. Regardless I strongly recommend attending this conference.

Life is friction

Life is just people and things working together to make things difficult for you. Like on a rainy windy day where you can just lean into the wall of oncoming air and water and just push through.

Most of these things you cannot really do anything about, and there is no point to complaining about it, but then there are small wins, like going around the corner of a big building and it taking a few seconds for the wind to change direction and blast you in the face again. Those few seconds are golden.

Anyway – one of those breaks in the rain is that I’ve switched off comments on my blog. There are two people on average that read a post, and rarely do they want anything from me. A handful of posts have over the decades accumulated hundreds of views. Among humans my writing has the attention it deserves.

The bots though are big, unrelenting fans and have an insatiable appetite for communicating all kinds of offers through commenting on my posts (that they can’t have read according to the page statistics).

I pay for a service that is supposed to deal with my popularly in the bot scene. An inbox zero-as-a-service, basically. Well those guys were annoyed that I sent too much traffic. Again two (2) readers per day generates enough spam bots that I either have to get an even more ludicrously expensive anti spam tier, buy a higher tier blog hosting to be allowed to add a captcha, or lastly self-host with expenses of both money and time.

I don’t want to do any of those as they cost money, and if you have seen the rest of the blog you’ll see why I’d rather not be spending any money on it. So I’m shutting the comments. I know this may lead to reduced “engagement” but the thing is, people that reach this page know how to reach me, so nothing is really lost, except friction.

I get that brief respite from the rain that you get at a large building site where the hoarding and scaffolding are overbuilt into a luxurious chip board arcade with strip lights and trip hazard warning tape everywhere. You get in out of the direct rain, but big drops from 70m up the scaffolding hit you directly on your skull through a gap in the chip board instead. It’s a win, but you’re never allowed to be too elated.

Anyway, if you need me, you know where to find me.

Happy New Year?

The current state of affairs

In 2020 I learned the meaning of the English expression Busman’s Holiday, and it generally applies to software developers that write code on their free time, but especially so during a pandemic with abundant remote working. Putting that aside, I will make some predictions of what will be happening over the coming year.

Predictions

The Pestilence

Given how popular the Omicron strain has proven, my guess is that everybody will have had Covid, and the patience for government measures will have grown thin, especially given the attitudes with which they flout the rules within the government itself. If Labour takes power, of course this can all change and we can be heading for more lockdowns.

The Industry

Despite lockdowns and the inevitable destruction of the service industry (yes, for good reasons in managing the spread of the virus, but let us be honest with the consequences) the IT industry has fared reasonably well. As long as I have lived in this country there has been a general election every two-and-bit years, and we could be looking at one of those again, and in a run-up to that, Rishi Sunak will want to keep money pumped into the system, meaning IT people will most likely still do quite well for a bit longer.

The Great Resignation

From the discussions around recruitment before the above variant gained popularity, there seemed to be two main streams, people that want to work remote full time, and people who want to work in a hybrid capacity, where you do meetings and collaboration in the office, and focussed work remote – if not at home at least in a co-working space closer to your home. The crutch used by weak leaders to manage people – counting bums in seats – will probably need to be replaced by some kind of outcome-based measurement. Luckily that ought to align quite well with company targets. No company has a slide in an AGM saying “well revenue is down, profits are down but luckily we have 99.5% occupancy of our desks“, the goal is to make money, and with the right type of goals within an organisation you can have department and team goals that in some way works towards the overall business goals, but of course measuring the right thing is key, so – yes – it is harder than just counting empty desks.

My thinking is that if the pandemic calms down, we will se a subset of organisations that are unashamedly on-prem only, and those that look for work that is on-prem only will go there, but I suspect that it will be harder to hire for those positions.

The Continuous Delivery

People insist with this Agile malarkey, and even though “Scrum, but…” remains the dominant methodology, companies are starting to read Accelerate and realise that they need to move faster, so gradually obstacles are being dismantled. Management structures are tweaked, project management and budgeting is being replaced with product and portfolio management. Coordination exists in companies already. Organisations that are famously agile say they struggle to coordinate cross-cutting changes across an organisation, but in old enterprises, that coordination work is the thing they do well, because in their current day-to-day even the most trivial piece of work cuts across several teams and needs careful planning and second-guessing to be delivered safely. The big differentiator is to change the internal structure so that for 80% of changes, a single team can plan, test, construct and monitor features completely independently, while leaving some version of the existing structure to deal with the subset of changes where you still need to coordinate. If you achieve that, you are in a vastly better place than before.

The Hardware Shortage

Have you tried buying a graphics card? A car? Well, you may have noticed that there is a supply chain crisis in the world. US container ports are struggling now and what originally started with the double whammy of Chinese New Year and OG Covid shutting down electronics suppliers, got worse as there was a supply shock when the pessimistic demand prognoses turned out to have not accounted for stimulus checks inducing demand globally, and more recently there i are geopolitical issues when one of the main semiconductor suppliers globally, Taiwan Semiconductor Manufacturing Company (TSMC) is situated in region on the brink of war while at the same time Intel are struggling to produce any advanced processor nodes in their own fabs, even though they now are producing a competitive line of processors again.

My prediction is grim here, but let’s pretend like things will go well. I don’t think you should buy anytihng in 2022 if you can avoid it, which has been my advice from March 2020 onwards, that hasn’t changed.,

The Crypto Scams

Just like with drug trafficking and modern slavery, you can make a lot of money with cryptocurrencies and NFTs, and you can already see that the biggest profits are made when people are robbed of their coins.

As you dream up your practical use cases that will finally be the problem that crypto solves, just remember this: Like with all applications of cryptographic signing, the time it takes to encrypt or decrypt something is part of why it works, why it is secure. You will never have a world where these transactions are fast and secure. All exchanges for cryptocurrencies that trade fast circumvent a number of supposed features of a distributed ledger. There is no “it will be faster, eventually” unless you are prepared to sacrifice some of the key selling points.

Luckily China has decided that crypto currencies are inherently decadent and are clamping down on miners, and if western utilities start going after those that steal electricity with more zeal, we could start to see positive change.

Don’t forget that NFTs, Bitcoin and Eth singlehandedly is destroying the Paris Accord on climate change, You can heat a typical American home for six weeks on the energy required for one (1) bitcoin transaction. As computers become faster, this will gradually be worse as well.

Conclusion

As with any arbitrary point in time, the time immediately after will not be drastically different than the time immediately preceding it, so there will be much of the same next year, but I have still tried to make some statements that are specific enough that we can go back in a year to see what I got right and what I got wrong. Happy New Year!

Async Enumerable in C# / .NET 6

Background

In recent times Microsoft have begun to performance test their web platforms. Whilst previous generations of their .NET framework and ASP.NET web platform had prioritised ease of development over performance quite dramatically, the latest generation ASP.NET Core performs quite well, on Linux no less.

After inventing the async/await model of abstracting away callback hell when writing asynchronous code, the New Microsoft, the Ones That Care About Performance realised that people will just allocate all the RAM in the universe if you let them, and that whilst engaging in the now very common practice of using ASP.NET Core to create web APIs that produce data as json payloads, users would mercilessly just serialise massive payloads of List<T> into one massive string that they would shove out onto the network, or have server endpoints that would accept arbitrarily large strings off of the Internet attempting to coerce into a List<T>, meaning ASP.NET Core services could be knocked offline by supplying a ludicrously large payload, and performance could be a bit erratic at times, depending in the size of data the user was requesting.

So what do they do? Well, a couple of things, but one of them is to introduce the concept of IAsyncEnumerable<T>, an asynchronous enumerable, that supports cancellation, clean exception handling and stable performance for handling variably sized payloads without suffering unpredictable performance impact.

The goal today is to successfully serve a payload in ASP.NET Core 6.0, and to deserialise it in a client application, also in .NET 6, serialising onto streams, deserialising off of streams, processing data without allocating massive payloads, also – beginning to receive data right away rather than to wait before the full payload has been buffered in its entirety in various services along the way before eventually reaching the end user.

Physics and leaky abstractions

Just to preface this – just like the async /await doesn’t fundamentally change physics, i e there is no getting away from the fact that you first kick off an operation and basically schedule code to be run when that operation has finished, leaving you to do other things. I.e. since your code will actually return to the caller directly after you’re scheduled the first async operation, the code has to return something in addition to the normal return value, it has to return a handle through which you can access the state of the function, and – once the operation has completed – the return value. This way the surrounding code has a chance to deal with the asynchrony but most of the time just pretend that the code is synchronous.

You see, if the human squishy brain cannot fathom mulithreading, don’t let me get started on asynchrony.

So with a normal asynchonous function that returns a scalar, the caller receives a System.Threading.Task that encalsulates the asynchonous state and eventually the return value. The async keyword lets you pretend it isn’t there and write synchronous code , as long as you put an await in before the asynchronous call is made.

Contagion

You’ll notice though, like with monads, that when you’ve started wrapping your return values in Task<T>, it’ll go all the way across to the other side of the application, i e if your database code is asynchronous, the repository or other database access layer topology you have will be asynchronous too, and then you turn around and then you find that it has spread all the way to your ASP.NET controller. On the plus side, the ASP.NET controller automagically injects a CancellationToken that you can send all the way down to the database and get automagic cancellation support for long running queries if people refresh their page, but that’s an aside.

The point here is the contagion. You can attempt to force things to be different with GetAwaiter().GetResult() to block a thread while it’s evaluating, but that is very dangerous performance-wise, better to just let it spread, except for in places where Microsoft have been lazy, such as in Validation and Configuration, cause clearly when it would mean work for them it’s “not necessary” but when it’s eons of work for us they are fine with it. Our time is free for them.

Anyway, I mean it makes sense that the abstraction must leak in some cases, and IAsyncEnumerable is no different. Any one return value would fly in the face of the whole streaming thing. So awaiting a task doesn’t really make sense. Instead it’s iterators all the way down. Everywhere. Each level yield returns to the next, all the way down the chain.

Dapper allegedly comes with support for IAsyncEnumerable, but at the time of writing there is zero documentation supporting that allegation.

You can simulate that by writing this bit of code:

    public static async IAsyncEnumerable<T> QueryIncrementally<T>(this SqlConnection conn, CommandDefinition commandDefinition, CommandBehavior behaviour = CommandBehavior.CloseConnection)
    {
        await using var reader = await conn.ExecuteReaderAsync(commandDefinition, behaviour);
        var rowParser = reader.GetRowParser<T>();

        while (await reader.ReadAsync())
        {
            yield return rowParser(reader);
        }
    }

From that you can then pass the payload up iterator style, yield returning all the way up, until you get to the controller where you can declare the controller to return IAsyncEnumerable and the framework will handle it correctly.

Obviously as you cross the network boundary you have a choice in how to proceed, do you want to receive the data incrementally as well, or do you want to wait for all of it to arrive?

Since you made such a fuss in the first API, we will assume you want the consuming side to be as much work.

    private static async Task<Stream> GetStream(HttpClient client, string endpoint)
    {
        var response = await client.GetAsync(endpoint, HttpCompletionOption.ResponseHeadersRead);
        var responseStream = await response.Content.ReadAsStreamAsync();
        return responseStream;
    }

    public static async IAsyncEnumerable<T> HandleGetIncremental<T>(this HttpClient client, string endpoint)
    {
        var stream = await GetStream(client, endpoint);
        var items = JsonSerializer.DeserializeAsyncEnumerable<T>(stream, CreateSerializerOptions());
        await foreach (var item in items)
            yield return item;
    }

And then, of course, you yield return all the way up to the next network boundary.

Is this ready for prime time? Well, in the sense that Jay Leno was ready for prime time when he ceded the Tonight Show to Conan O’Brien, but everybody would probably like some more pace and less awkwardness.
Apparently letting lambdas yield return is on its way, and hopefully that can make it easier to pipe an IAsyncEnumerable through one API to the next, easily adding some filter or transformation mid flight rather than the incessant await foreaching that is now necessary.

Best is the enemy – stick with good

Working life, like any series of events, can be compared to other stories, such as those in cinema. Is your workday like Avengers Endgame when all your coworkers show up out of thin air and swarm to solve a difficult problem? Are you Malcolm Tucker of In the Thick of It helping your co-workers by providing astute observations and giving gentle constructive criticism? Or is it more like the middle of a 70s social realist movie when the alcoholic father / engineering manager promises you that although sure it’s bad now – mistakes have been made – but you know it’ll be great, we’ll stand up kubernetes and we’ll never deploy manually again?

Obviously – when they make a big budget movie out of the Phoenix Project, we can just look at that, but until then – what film are you living now? Which one would you like to be living?

You want to believe assurances of a bright future, but deep down, you know you’ve heard it before. Perhaps problems being indiscreetly alluded to in an early act comes back in the final act to cause a massive predictable calamity making you think your movie has a poorly crafted arc. Perhaps you give your social realism engineering manager the “you need to cut down, think of the kids” speech – but it’s met with denial. “Our network guys are diligent [to be fair -they probably are, regular John C Reillys the lot of them], it takes 2 minutes to make a configuration change – why would I take hours out of their day to write scripts to do things they complete in half an hour including the red tape we have imposed upon them? Do you realise how busy they are?”.

What if you want to switch franchises – so to speak? Get into a better movie? Let’s say your film is the social realism one, and after a few accidents in the workplace, the union is shutting the site, and the owners are threatening to move production overseas. Car factory, sounds Birmingham-based on the accents. Lovely soundtrack with early seventies Black Sabbath. Your character has to stop the mayhem on the factory floor so that the union will allow production to start before the owners scrap the factory for good, your budget is £0 but you happen to have massive rolls of black and yellow adhesive tape, some PPE and a loudhailer. Basically, you can turn your film around, you can do it – but you do have to literally start doing something.

I’m writing this to continue on a ball of yarn I’ve been unravelling in other posts. Basically I want to state that DevOps doesn’t need to mean shiny and new. Any type of automation that does the job is fine. You don’t have to change platforms , you can – and I personally mean should – start by automating the existing stuff and not by building a new feature complete platform. Take the first step! Stop dreaming about a service mesh and kubernetes. It won’t happen soon enough.

This next bit will be very marvel oriented, by the way, but feel free to translate this to your own cinematic universe. It’s like – you can manage to automate and ship software reliably, but you may not be ready to be Tony Stark. You are still be part of the MCU, but you won’t be an arms dealer billionaire or even an Australian Norse space God, or even a fighter pilot with accidental alien super powers and amnesia. The best you can hope for is to write PowerShell or bash. PowerShell and bash, perhaps. Cobble together some automation with whatever CLI you have laying around. Automate the simple things. Even if you are disrespected in the office like Agent Carter you can eventually save the day. The big first step is to figure out how all your hand crafted bespoke servers are really built and figure out how to build them from scratch with scripting. This is the painful, tedious first step that you have to take. How can I create my production environment using only scripting and free or affordable tools that my people already know how to use?

In too many companies the deployment automation is:

  1. Download packaged tested software from archive
  2. Disable monitoring to avoid scaring your on-call people
  3. Divert network traffic from node
  4. Decompress archive and copy files in place
  5. Restart services
  6. Re-enable traffic.
  7. Repeat 2 -6 for all other sides in the load balancer.
  8. Re-enable monitoring

This is not enough. There may be any number of unknown things that just live on your VMs without which things just wouldn’t work. Crucial OS settings that were made once that nobody remembers anymore. Such hidden things are the potentially big surprises that derail containerisation projects or cloud migrations. You need to Agent Carter the Whole Thing.

  1. Define networking. You have some leeway here – use a wild card cert, generate a new short-lived cert- create a load balancer or just a rule for a central load balancer. This depends on what you have in your infrastructure and what tools you know how to use, but basically – if the starting state is nothingness, after the automation is run, there should be a way for the outside world to find your service and know if it is healthy. If things already exist, your scripting should only make expected changes to it and be able to run multiple times without accidentally causing mayhem. Make sure any WAF rules or similar to enable access to dependent services are also set up here. If you can’t reach a necessary service at all, this should be immediately obvious from tooling without even digging into logs.
  2. Define virtual servers. If all you have is VMWare CLI, then create a VM based off of a suitable template. If you have some fancy cloud provider, use the highest abstraction level you can get away with. Azure Webapp, AWS ECS or Lambda. Stay away from raw VMs if you’re running in cloud, they are expensive.
  3. Install your servers to their desired infrastructure state and patch level. Ideally you use Ansible or even Powershell Desired State Configuration. There are so many non-trendy options that you already probably have a few installed. Chef or Puppet works too, if you have guys that know that stuff already. Find out what people already know and pick the simplest technology. The specific technology you choose isn’t key here, the big idea is learning how to take empty metal and get your stuff working there without having to do any manual intervention whatsoever. All of the infrastructure must be code.
  4. Now you’re at the point where the previous list is relevant. Of course depending on your choice of technology you may not need to repoint load balancers as some tools like chef and puppet support in-place upgrade. A central brain/source of truth will announce that new software exists, and you have to manage in place upgrades through ruby scripting if you’re unlucky, but it works . Either way, only here are we at what the previous CD solution thought was all of it.

You aren’t done until you can spawn a service as easily as your users shout “Another!”. You can get to this point with tools your guys already know. It may not as sexy is just flying straight through an enemy star destroyer using helm whilst your mechanical keyboard glows in addressable LED colours, but the point is your organisation most likely possesses the skill to do this already. You must take the first step.

Whichever cinematic universe your life’s film belongs to, should be proud at what you have achieved in the face of such adversity.

The Power of Sample Code

What is wrong with OOP?

In the culture wars between “Object Oriented Programming” and Functional Programming, you will find proponents of OOP that argue that we are doing fine – why should we change? and proponents of FP that lists a litany of inherent problems with what we are doing today and point to the ways FP solves them. After I once was at an Object Bootcamp with Fred George I believe the two main schools of thought are both wrong. All the problems listed by the FP peeps are correct, but they are not inherent in OOP, actually OOP addresses a few of them, but we are as an industry not doing OOP.

I may feel Fred George is the Messiah, but he is not alone in his views. Greg Young has similar concerns.

Inheritance is not the Big Deal

I am old enough to remember Borland C++ ads from the 90s. It focused a lot on inheritance, and reuse through inheritance became the USP for object oriented languages.

As soon as you have written some code though, you realise inheritance is the worst, as it creates undue coupling, making changes very hard to implement.

When Borland made those ads about how Porsche Turbo inherited the Carrera but implemented a big fat rear wing, they had begun their foray into C++ because it offered a way to handle the substantial boilerplate involved in writing a program in Windows. It was relatively straightforward to implement the basics and create a usable abstraction on top of the raw Windows API that made the developer experience much more pleasant.

As visual designers became a thing, they wanted a way to map properties with code, so that UI components (those things implemented as objects we mentioned above) could be manipulated by a developer in design mode. “Property” setters, basically syntactic sugar disguising normal functions, allowed the UI designers to read settings from the object, and replace them with what the developer types in. With this work, Borland and Microsoft were working to catch up with InterfaceBuilder from NeXT Computer (the same thing that lives on today in the Apple MacOS/iOS SDK) that had bolted a different type system on top of C and called it Objective C – but that had a world leading visual designer at the time. Anyway, I think they were in a hurry and didn’t think things through.

Approaches to deal with big programs

In a large codebase, the big problem is achieving low coupling but high cohesion. This means, you want all the code that belongs together to live together but you don’t want to have to make changes in seemingly unrelated code to modify a piece of functionality.

In large problems of old, you could call any subroutine from anywhere else, and many resources were shared, meaning between the time you set a value in a variable and you read from it, some piece of code in between could have modified the value, and you would not be automatically able to know where this access is made and how to prevent it.

In FP, we use modules for scoping, meaning you group functions into modules to aid readability, but the key concept, the Big Idea is immutability. After a value is created, it exists globally, but since they are read-only once created, the drawbacks of global state go away. There is no way to change something that somebody else relies on. You can transform it into a new thing that you need, but the original value hangs around until it’s no longer needed. It is harder to accidentally break other code with changes you are making

The Big Idea in Object oriented development is Encapsulation. You put the data with the code and manipulate abstractions. This means that if you get your abstractions right, you can change or replace these abstractions without needing to make sweeping changes in the codebase.

The original concept of object orientation relied on independent small sub programs that communicated by message passing, implicitly imagining like an “in tray” of messages that the object could process at its own pace and then send a response when the work was completed. However – objects were in C++, Java and C# was implemented as special dynamically allocated structs to which you made function calls, i e they became decidedly more synchronous than they were in Smalltalk or Simula. You would recognise Erlang Processes and Actors as looking more like OG objects. You also see that what made objects useful were that they shared properties we today associate with the term micro services, but on a smaller scale.

So what’s the problem, and what’s up with the title of this blog post?

Java, and C# arguably even more so, took the Big Idea and tossed it out the window. Property Get/Property Set to support novelties like graphical designers and visual components are a clear violation of encapsulation. Why are we letting objects access data that lives in other objects? The need to do that is a huge red flag that your model is incorrect. Both the bible, i.e. Refactoring, by Fowler and the actual Bible condemn this, this feature envy.

But why did these properties survive the nineties and live on into modern day? Why have they made things worse with auto properties?

Sample code

When you learn a new language, or to code in general, the main threshold is getting to the point where you write idiomatic code in that language. I e you use familiar phrases. You indent the code in a certain way, you name things according to a certain standard and you use familiar ways to do things like open a database connection, make a HTTP request et c, that a seasoned programmer would be familiar with. Unfortunately- in C# at least, these antipatterns are canon at this point, so to write properly encapsulated code would maybe cause a casual reviewer to ask WTF and be sceptical.

What is canon comes from the publicly available body of work that a beginner can reasonably access. Meaning, effectively Microsoft sets the bar when they announce features, document them and create samples.

There are some issues here. If you look at a large piece of sample code, you may notice how difficult it is to identify the key concept being demoed as the logging code or error handling bulk up the code in a way that is distracting, so brevity must be allowed to remain a priority, clearly.

At the edges where the code starts interacting with network and storage, this type of organisation isn’t inherently despicable either, so a blanket ban is perhaps not the way forward either.

How do we make it clear to new OO devs that when they fill that empty Models folder their project template creates for them with code they would be better off thinking OO proper?

By that I mean making classes that are extremely small, use value objects, prefer private fields, avoid properties et cetera. My suspicion is that any attempt at conveying this programming style through the medium of sample code in templates or documentation is doomed. The bulk of code necessary to not only prove the concept but to in fact make it part of the vernacular would require a large number of people making quite a lot of good code public do that new learners can assimilate the knowledge.

I think good OO code is scarce. Getting the abstractions right is just too hard, you will have compromises in various places, and all the tools tempt you with ways to stray from the narrow path of righteousness, but with modern refactoring tools you should be able to address some of the issues amd continually strive to make the code better.

Incidentally, with properly sized objects you can unit test without cheating (using internal helper methods, or by using mocks), so there is scope to brighten up the tests as well.

The Powershell and the Glory

In which I add a custom prompt by making a hack in the PowerShell profile.

As I have mentioned in previous posts, I use Oh My Posh to set the theme in Powershell. While working with Pulumi to create deployment stacks, I thought I could use a way to see which stack is the current one, i.e. to effectively have the output of pulumi stack --show-name appear in the prompt automatically.

Back in the old world, the agnoster theme was the prettiest. In my terminal at least, it looked quite a lot worse after upgrading to Oh-My-Posh 3, so I did exactly what they say in the documentation, I used Get-PoshThemes to look at all of them, exported the one I liked best into a json file and went to work.

Command

The naïve implementation would be to add a new segment in the prompt, using the segment type seemed to be “command”, which does what it says on the tin, it allows you to call a command and display the output, like it works in Bash.

        {
          "type": "command",
          "style": "powerline",
          "foreground": "#000000",
          "background": "#ffff00",
          "properties": {
            "shell": "powershell",
            "command": "pulumi stack --show-name"
          }
        },  

They do warn you that there will be performance implications, and – yes- on my 16 core desktop it still takes forever to start a process in PowerShell, so that didn’t seem to be a workable way forward. The suggested approach is to “abuse environment variables”, so… let’s?

Environment variable

I have previously made hacks to set window titles in cmder to work around iffy built-in support for showing the path as the tab name. The idea was to replace the built-in “cd” alias with a PowerShell function that also does dodgy stuff on the side apart from changing directory. In this case I would test if a pulumi.yaml file exists in the new directory, and in that case set the variable PULUMI_STACK to the output of pulumi stack --show-name, or set the variable to empty.

# --- other stuff
function Change-Directory() {
    param(
        [string]
        $directory
    )
    Set-Location $directory
    $env:PULUMI_STACK = ""
    if (Test-Path "pulumi.yaml") {
        $env:PULUMI_STACK = & pulumi stack --show-name
    }
}
# --- other stuff
Set-Alias -name cd -Value Change-Directory -Option AllScope

I of course don’t want to globally change this variable, I explicitly only care about the current terminal session, so hence I’m not trying to update the registry or anything like that. To read this variable and show a prompt, we then modify the theme json file to leverage the envvar block and to contain the following:

{
    "type": "envvar",
    "style": "powerline",
    "foreground": "#000000",
    "background": "#ffff00",
    "properties": {
       "var_name": "PULUMI_STACK"
    }
},  

After this work, the prompt is much faster, beyond acceptable, maybe even pleasant.

You can have nice things

I have come across a few things that are legitimately pleasant to use, so I thought I should collate them here to aid my aging memory. Dear reader, I am not attempting to copy Scott Hanselman’s tools list, I am stealing the concept.

Github Actions

Yea, not something revolutionary I just uncovered that you never heard of before, but still. It’s pretty great. Out of all the yet-another-yet-another-markup-language-configuration-file-to-configure-a-thing tools that exist that help you orchestrate builds, I personally find Github Actions the least weirdly magical and easy to live with, but then I’ve only tried CircleCI, Azure DevOps/TFS and TeamCity.

Pulumi – Infrastructure as code

Write your infrastructure code in C# using Pulumi.It supports Azure, AWS, Google Cloud and Kubernetes, but – as I’ve ranted about before, this shouldn’t be taken as a way to support multi-cloud, the object hierarchy is still very bespoke to each cloud provider. That said, you can mix and match providers in a stack, let’s say you have your DNS hosted in DNSimple but your cloud compute bits in Azure. You would be stuck doing a lot of bash scripting to make it work otherwise, but Pulumi lets you write one C# file that describes all of your infra, mostly.
You will recognise the feel of using it from chef, basically you write code that describes the infrastructure, but the actual construction isn’t happening in the code, first the description is made, the desired state is then compared to the actual running state, and adjustments are made. It is a thin wrapper over terraform, but it does what it says on the tin.

MinVer – automagic versioning for .NET Core

At some point you will write your build chain hack to populate some attributes on your Assembly to stamp a brand on a binary so you can display a version on your site that you can track back to a specific commit. The simplest way of doing this, without needing to change branching strategy or write custom code, is MinVer.

It literally browses through your commits to find your version tags and then increments that version with how many commits there are from that commit. It is what I dreamed would be out there when I started looking. It is genius.

A couple of gotchas: It relies – duh- on having access to the git history, so you need to remember to remove .git from your .dockerignore file, or else your dotnet publish inside docker build will fail to locate any version information. Obviously, unless you intended to release all versions of your source code in the docker image, make sure you have a staged docker build – this is the default in recent Visual Studio templates – but still. I encourage you in any case to mount your finished docker image using docker run -it --entrypoint sh imagename:tag to have a look that your docker image contains what you expect.

Also, in your GitHub Actions you will need to allow for a deeper fetch depth for your script to have enough data to calculate the version number, but that is mentioned in the documentation. I already used a tag prefix ‘v’ for my versions, so I had to add that to my project files. No problems, it just worked. Very impressed.

A cloud strategy

I’m going to rehash some learnings that I have made over the last decade and a bit of doing cloud in one way or another. I have recently read thought leaders report similar things – only better written and backed by more experience of course – which made me think “oh, I’m not totally crazy then” and to set about writing this down.

Basically, it is my medium tempered take on the whole cloud thing, in terms of getting on it.

Why cloud is cool

In the bad old days, if you had an idea for some software, you had to start by buying a server. I mean it was never on the scale of “I have an idea for a consumer product, let’s build a factory”, but still it was definitely a barrier to getting started.

The impetus for building out cloud was that Amazon needed compute for their little bookshop website (remember!?) and thought it was prohibitively expensive to buy high-end servers, so they decided to buy a metric faecal ton of low-end computers instead, and use software to provision this aggregated computing horsepower, and basically letting people choose between a bunch of virtual machine sized depending on the oompgh a certain department needed for their application.

This was of course extremely complicated, the software bit, but once they were done, they had accidentally created cloud compute and could make more money selling cloud compute than selling books. The ability of using relatively simple APIs to create and provision VMs allowed startups to acquire really pathetic servers for nearly nothing, which was amazing for trialling ideas and fed the software boom we have seen over the last decades. Other services were built on top, and competitors came around.

Why cloud is cool is the tight APIs that lets you create and destroy infrastructure in an automated fashion. Yes, you can autoscale, but also the rapid prototyping potential is really beneficial and arguably even more significant.

What to worry about

Security

Yes, of course. Public cloud, you can tell from the name. It is not the same as having your servers at home, at least psychologically. On the other hand, assuming you are secure because your servers are inside your own building is false as well, you are still on the internet. There is no getting around this, you already have network people and security people, and with all cloud providers there are ways of securing your network that they will be familiar with how to operate sensibly. I e your network and security people will know what to do.

Cost

Yes, it’s a big one. Not all cloud things are free or near free. Basically, running a VM 24/7 and block storage (as in a scalable pretend hard drive mapped to a VM) are usually the most expensive things you can do in a public cloud. Sadly, a VM tucked away in a virtual private network with elastic storage mapped to it seem like such an easy migration path if you are currently running your apps in VM You will need to migrate your apps over to cloudier solutions such as AWS Fargate/Lambda or Azure App Services to reduce cost eventually. For your in-house LOB apps you can in most cases (but not all) trivially replace file system storage with cloud native blob storage such as AWS S3 or Azure BlobStorage for your files storage needs, but it does require code changes, even if they are small. As the cloud bill start to come in, it seems a good way to spend developer resources as the returns in terms of cost savings can be quite significant. Be wary of giving developers the ability to create resources at will, as the odd developer accidentally leaving a VM running will quickly accumulate. There are ways of dealing with this kind of stuff, but do consider it.

What not to worry about

Multi cloud

There are many tools that provide abstractions over cloud APIs, and many tools that promise that they can offer you independence and warn of vendor lock-in. That is for most people just a waste.

You will need to choose one provider for your app stack. You can still have Google Apps for email and use AWS for cloud, or use DNSimple for DNS, Office 365 for email and AWS for apps, those are mostly orthogonal concerns. You will suffer outages. You will not – without incurring unfathomable cost – be able to load balance across cloud providers. If you really are that uptime sensitive, it would be cheaper for you to have georedundant datacentres and give the cloud thing a miss.

The problem with attempting to stay cloud agnostic is that you can only use the lowest common denominator of the tools you have available rather than throwing yourself in feet first into all the opportunities that exist with a given cloud provider.
Worst case, if the CEO gets angry enough at something and wants to switch just to make a point, it still will not be completely impossible to rewrite code in the seams. For instance, if you change your code to use AWS S3, it would be relatively trivial to change the code that calls S3 to use Azure BlobStorage in a pinch. No need to go choose a platform for it. Just like with ORMs and database providers (“with NHibernate it’s so easy, you can switch to Oracle much more easily”) people very rarely switch cloud providers. There would have to be a very compelling economic argument anyway.

Why go through with it?

Rapid prototyping

You should test in production anyway, but if you insist on creating test environments, being able to copy/paste your prod environment exactly and test your changes is only possible in an environment where you aren’t poking at real metal. It would be ludicrous to buy overbuilt on-prem hardware “because sometimes I like to spawn up a few extra copies of prod”. The powers that be would be livid at the massive capital expense that would go underutilised most of the time. With cloud however you spawn, test and destroy in minutes. Merely a blip on the radar in terms of cost. To mitigate the risk of developers leaving stray instances around you can just use governance like you do anywhere else in a workplace, but ideally the concept of ephemeral instances should lend itself to clean up nicely.

Modern software development

Bringing the organisation to a place where it has autonomous engineering teams that can bring feature from idea to production without hand-offs is the key driver for organisational performance. Moving to the cloud is going to make that happen. You could achieve this with on prem as well, but it would probably mean buying more hardware than you will really use more than in short bursts. If that trade off is worthwhile to you, then who am I to deny you your wish, but for most people cloud is the way to go.

What to do?

You probably need to get some help with this. Everybody in your organisation is already doing things. Taking on a cloud migration is going to be a massive effort for everybody, and you are still probably going to need an external experienced consultancy to help you. There are many out there that offer to architect a cloud migration for you. Not everyone is a charlatan, but, given that the selection process for these types of gigs is “who did the CxO meet at a conference/play golf with/…” I think the most important take-away is that the individual firm probably doesn’t really matter. It also probably doesn’t really matter which cloud provider you go with. Let a bunch of ops and devs benchmark the tools and APIs of various providers and come back with their feedback. There are probably going to be some budgets that can be negotiated between your consultancy (who most likely also has a VAR agreement with some cloud providers) , the provider and yourself that will determine some kind of benefits for one over another, but that’s still only speculation at this point.

Eventually one provider will be declared the winner and work will start. It doesn’t matter, really, which one was chosen. Even If the engineers say there really is a show-stopper, do investigate, but most problems can probably be avoided through some development. If you are running some weird VM somewhere that needs specific hypervisor features or some curious networking then of course you will have a challenge. Not necessarily an impossible one, though. This is not going to be done in an afternoon anyway. There is time to make changes to code, and there will be a need to do so in order to fully leverage the public cloud as mentioned above. Obviously non-blockers can be deferred, but unlike traditional tech debt there will be direct cost implications of course.

Enterprise IT, a tragedy?

In the beginning of time, Grace Hopper invented business software and Cobol at IBM. This meant that in addition to calculating the trajectory of an intercontinental ballistic missile, large computers could now be installed, literally built into the very lower basement of international enterprises and be used to process payroll and do forecasting, gradually obsoleting armies of calculators and typists.

Earliest business applications

At this point, nobody knew how to actually or maintain write large software systems. I mean, granted, current paradigms like object oriented programming and functional programming were formalised around that time, but still, people had not written as much software back then as you now have to write to make a car set fire to petrol at an opportune time given crank angle, engine speed and temperature (I exaggerate, but not really).

From terminals to personal computers

Eventually Microsoft began it’s journey to global domination in the office by piggybacking onto IBM’s good name and getting installed into offices by default since nobody got fired for buying IBM. Unfortunately, IBM’s own operating system guys that wrote time sharing operating systems for mainframes had no influence over the people that created the CP/M knockoff that was to be PC-DOS (and Microsoft’s MS DOS), so it had no multi-user or networking security features built it as, well you were supposed to have it on your desk, and buy another computer for somebody else’s desk. No need for passwords, amirite? I mean UNIX existed and was fairly wide-spread in the corporate environment at the time, and it had decent security built-in, at least a fundamental understanding that even as a power user, you don’t want to have that power all the time.

The beginnings of Enterprise IT was therefore to maintain and write software for mainframe computers, and that was so alien of work to most organisations that it quickly became outsourced to suppliers that could lower maintenance costs, but overcharged for development work, thus lowering OPEX but moving software to CAPEX. This had one of the first negative incentives driving the business to make fewer changes, making enterprise IT infamously slow to react and become a department of No.

As technology changed, the IT department stayed in the cellars and basements where the business rarely ventured. As mainframes were phased out and replaced with banks of personal computers, the white coats disappeared. Gradually, typing stuff into computers at the IT department stopped being work exclusively for women, and men started taking over; salaries increased (not saying there is a causal relationship, just saying), but still no natural daylight.

Forces conspire to make organisations write and maintain code

There is a syndrome that happens to computer people called “Not Invented Here”, which is a form of exceptionalism that means that no outsider could possibly understand our Very Special Requirements, so we need to write our own X, where X can be anything. While certain pieces of software became standardised, like payroll and accounting for small businesses, there is still big money in helping people shoot themselves in the foot by developing and maintaining their own adaptations of commercial ERP systems like SAP and on a lesser scale Dynamics NAV or GP. Microsoft Excel is the most successful way of combining both, as in letting people buy a commercial off the shelf application but then making excel sheets with bespoke maths that can both be the lifeblood of new business and be its cause of death when the bespoke template turns out to have a bug and nobody knows how to fix it anymore.

An accounting quirk (see the bit about CAPEX above) where you count code written as value created rather than a pure expense, and book development costs as having added capital value to your code base despite, objectively, few if any of your competitors would buy your bespoke mess of a back-office system if you offered it on the open market. Also they most often don’t count the depreciation that comes naturally unless you make the code maintainable and easy to refactor if requirements change in future.

At this point we have an alienated IT department that has only two key metrics, to cut cost and to add features to internal software products, but no real way to directly discuss requirements with those that use the software.

Negative incentives

In the beginning you only have developers. As in, they develop, profess to test and deploy their code without any supervision. Then you have an outage that embarrasses somebody in management but no singular scapegoat could be found and all of a sudden you get ops guys that are there to protect the business from the devs. After another outage that embarrasses the management further, you may get a QA department. You can imagine how IT security comes to exist as a function within an IT department?

You now have one team that’s there to make changes that implement business requirements they have captured some way or another, one side that’s there to make sure those changes are valid and one side that’s there to make sure there are no outages. The incentives become to make few releases so that the QA guys want to be able to make a full regression before they green-light the release, and the QA guys are the ones getting a bollocking when bugs get out in the wild. The ops guys have enough to contend with without f^!£$g developers making changes ON PURPOSE, how are you ever going to maintain a stable system if you keep poking it with new software all the time?! So yea at best one release per month, anything quicker would be irresponsible and you would be working your QA department to the bone.

From a business perspective this means you are never getting your change in. As more things go wrong, process/red tape is added, lead times longer, change freezes are introduced periodically. Longer release cycles and bigger releases cause bigger problems.

Technically speaking, after mainframes people bought servers. At first they were just computers, beige like all the rest of them, and a server room was just a cupboard where they were shoved. You bought a physical computer from a supplier where you got a decent price, and you set it up, installed your OS and then you installed your software one it. Or you bought the computer and shipped it to your software vendor and let them install their software on it. 19″ racks became a thing, and servers became distinct from PCs in that they became loud and flat. You were going to stick them in a room away from humans anyway, so noise was no longer a concern. You did want many computers per rack, and you wanted simple but effective cooling, so you would fit powerful high-RPM fans.

As things progressed, people realised that it is hard to find office space that allows you to fit redundant power, automatic fire suppression and redundant network connections, so instead of trying to fit that into your basement, they would go to a third party that offered co-located datacentres. There you could mount your rack servers and they would give you ways to remotely manage them, so you wouldn’t have to physically interact with the servers to run them. All the patching and other maintenance could be done over the internet, and the data centre would make sure no villains could get at your hardware.

After a while people realised that you could just buy a couple of massively overpowered servers and then divvy up the computer horsepower onto virtual machines, pretend servers that would behave like separate physical machines. Carving out a bit of virtual compute and create a new “server” was a lot faster than buying a physical server and having your colo provider plug it into your rack. You would still have to install the OS and configure the networking, but there would be templates and automation. Heck, you could even write command line scripts to new up new servers.

Point is, in Enterprise IT there is no time to write scripts, and far be it from the mind of any ops person to collaborate with developers to explore things like version control and automated testing, I mean the developers are the enemy, the cause of all our problems, why would we collaborate? VMs are therefore largely artisan creations with very bespoke installs, apart from possibly sharing a raw template with some antivirus or monitoring, not even instance 1 and 2 of a load balanced pair have the same software on them.

As the blows keep coming with outages, bugs in productions and near misses that cause leadership to go on long-term sick leave, decrees can go out to create test environments that are “the same as production”.

Servers are bought (VM hosts, or course) and VMs are configured. Obviously, it is prohibitively expensive to make it exactly the same as production given that the load requirements will be different, so some corners are cut, but, it should be close enough. As the old meme would go – narrator: It was not close enough. Also, since the server operating systems are still hand crafted, there are multiple potential places where differences between production and test can creep in.

Great expectations

In various countries, leaps were made at various times that caused people to adopt electronics at vastly greater rate than before. In Sweden there was a push in the late nineties, early noughties where you would get a quite substantial tax rebate if you bought a modern computer at a certain cost, causing a lot of people to all of a sudden possess a modern computer. With various Covid stimulus cheques it seems a lot of Americans put money straight into a gaming computer (thus worsening the current silicon supply chain constraint situation). There have been similar schemes all over the world at one time or another to encourage adoption of technology to promote familiarity with new technology, I just can’t be bothered to google more examples. Basically, people have used the internet and are able to see what commercial software development can do. Another cause of people becoming aware of the wider world is how various Apple products have had a marked impact on Enterprise IT, and BYOD basically becomes a thing in businesses because the CFO one days comes back to the office with a MacBook Air and stares the IT department down until they “make it work” with their existing corporate software that only runs on a specific type of dell laptop they have imaged two years ago, requiring IE6.

The exposure to what computers can really do and the daily torture of using enterprise software to do their jobs, dissatisfaction among the people on the business side is rife. Eventually, some middle manager just takes their corporate expense account and hires some consultant off the books and builds some quick win software to solve a specific problem. If we are unlucky, this is an all out win. It works, it generates business and was executed in a timely fashion on, or just over, budget. A dangerous precedent is set, another wedge is driven in between the business and the IT department. Shadow IT is born.

Recap

To bring us to the final bit of the story, let’s assess where we are.
Enterprise IT is disgraced, underfunded and distinct from the core business, literally moved away from the rest of the organisation. A troll under a bridge or an ogre in a swamp. Sure, the CTO may report to the Board of Directors, but there is no real correlation between desired business outcomes and the metrics that the IT department measures its services against, and no work is done to ensure that IT services directly benefit the key outcomes that the business as a whole needs to achieve. Control of IT spend is instead largely project based, as in, the business has an idea, it needs some IT support, a project is created with a budget and a final ship date before anybody with technical know-how has even assessed it, and then work commences. Probably contractors are brought in if there is sizeable chunk of development, but normally the department is kept lean. Service desk and “maintenance development” presumably outsourced to a country in a different timezone.

What Enterprise IT wants is to be a force for good within the organisation, but since the beginnings of time IT has often been deemed a non-core support function, and that increased conceptual distance has made it more difficult to effectively be of use to the organisation. Cultural differences between IT and other parts of the organisation, and perhaps ineffective communication between stakeholders and the developer organisation as caused upper management to silo the organisations further apart rather than agree on an effective way of working closer together. In cases where IT is not involved in other aspects of the business, obviously there will be no “osmotic” assimilation of domain knowledge, so the business may be shocked of how little the IT department actually knows about the bread-and-butter business that keeps the lights on.

Roadmap

How do we turn things around? What are the most important things moving forward? I will leave out the Practices bit, because I have rarely seen bad practitioners in organisations I have worked with. People will write automated tests and implement CI and CD if they are given circumstances not directly hostile to professional software development. The bigger lego pieces are usually where the problems lie, and why management buy-in is crucial.

Increase automation

The VMs are not family or even pets, you should delete them all and start over. Not in one go, or to cause another outage, but replace at pace the VMs you are currently running in production with new ones created through automation. When you can automatically deploy your core business application from nothing to a running instance without any manual intervention you are done. Thinking you could replace your running VMs with automation but not actually having proven it has no value. Ideally make the deployment process be some version of “stand up new VMs with an app on them, run tests to make sure it’s working, route traffic to the new instances, destroy the old instances”, so that you can deploy without causing any loss of availability. This is crucial in building trust with the rest of the organisation. When I write VM, I’m not making a dig against containers or containerisation, just saying you can achieve this without moving to kubernetes and a service mesh, you can do it with bog standard VMs and a bit of scripting.

Decouple systems

In order to be able to fearlessly deploy, you will have to decouple systems so that you can deploy small changes often, with small blast radius and short time to recover.

Organise your IT department after what you are supporting. Yes, after business functions but also after the systems that you maintain. If you have a monolith that support all business functions, then you need to split it up. The output of different teams needs to be independently deployable. This is not easy, but it is the only way to enable teams to make predictable progress. Consider it an investment in predictable outcomes.

Maintain products, don’t run projects

When you have finally managed to put together teams that match your customers and your work, then don’t disband them again after you have delivered a certain milestone and the “project is done”. Budget for products rather than projects, so see your own software as a driver to increase profits and reduce cost. Set targets for what you want to achieve and measure outcomes. Let developers prototype things and show the business what’s possible. If you have built the right platform for people, it will be possible to securely bring new ideas to life and test them with real clients and get true feedback. This way you can delight customers faster, and there is lesser risk you get saddled with maintaining some Shadow-IT piece of software that suddenly became core IT after it was successful and the consultant that wrote it moved on to greener pastures.

Goal

By introducing a wider interface between the business and IT, and making teams that have the autonomy and platform support to independently iterate of features quickly, you will both delight the business, delight the customers, make more money and have happier teams. There will be times when you have to organise and coordinate, but the lion’s share of work can be done independently.