#dataflow

[ayeforscotland]

What is Dataflow?

This post is inspired by another post about the Crowd Strike IT disaster and a bunch of people being interested in what I mean by Dataflow. Dataflow is my absolute jam and I'm happy to answer as many questions as you like on it. I even put referential pictures in like I'm writing an article, what fun!

I'll probably split this into multiple parts because it'll be a huge post otherwise but here we go!

A Brief History

Our world is dependent on the flow of data. It exists in almost every aspect of our lives and has done so arguably for hundreds if not thousands of years.

At the end of the day, the flow of data is the flow of knowledge and information. Normally most of us refer to data in the context of computing technology (our phones, PCs, tablets etc) but, if we want to get historical about it, the invention of writing and the invention of the Printing Press were great leaps forward in how we increased the flow of information.

Modern Day IT exists for one reason - To support the flow of data.

Whether it's buying something at a shop, sitting staring at an excel sheet at work, or watching Netflix - All of the technology you interact with is to support the flow of data.

Understanding and managing the flow of data is as important to getting us to where we are right now as when we first learned to control and manage water to provide irrigation for early farming and settlement.

Engineering Rigor

When the majority of us turn on the tap to have a drink or take a shower, we expect water to come out. We trust that the water is clean, and we trust that our homes can receive a steady supply of water.

Most of us trust our central heating (insert boiler joke here) and the plugs/sockets in our homes to provide gas and electricity. The reason we trust all of these flows is because there's been rigorous engineering standards built up over decades and centuries.

For example, Scottish Water will understand every component part that makes up their water pipelines. Those pipes, valves, fitting etc will comply with a national, or in some cases international, standard. These companies have diagrams that clearly map all of this out, mostly because they have to legally but also because it also vital for disaster recovery and other compliance issues.

Modern IT

And this is where modern day IT has problems. I'm not saying that modern day tech is a pile of shit. We all have great phones, our PCs can play good games, but it's one thing to craft well-designed products and another thing entirely to think about they all work together.

Because that is what's happened over the past few decades of IT. Organisations have piled on the latest plug-and-play technology (Software or Hardware) and they've built up complex legacy systems that no one really knows how they all work together. They've lost track of how data flows across their organisation which makes the work of cybersecurity, disaster recovery, compliance and general business transformation teams a nightmare.

Some of these systems are entirely dependent on other systems to operate. But that dependency isn't documented. The vast majority of digital transformation projects fail because they get halfway through and realise they hadn't factored in a system that they thought was nothing but was vital to the organisation running.

And this isn't just for-profit organisations, this is the health services, this is national infrastructure, it's everyone.

There's not yet a single standard that says "This is how organisations should control, manage and govern their flows of data."

Why is that relevant to the companies that were affected by Crowd Strike? Would it have stopped it?

Maybe, maybe not. But considering the global impact, it doesn't look like many organisations were prepared for the possibility of a huge chunk of their IT infrastructure going down.

Understanding dataflows help with the preparation for events like this, so organisations can move to mitigate them, and also the recovery side when they do happen. Organisations need to understand which systems are a priority to get back operational and which can be left.

The problem I'm seeing from a lot of organisations at the moment is that they don't know which systems to recover first, and are losing money and reputation while they fight to get things back online. A lot of them are just winging it.

Conclusion of Part 1

Next time I can totally go into diagramming if any of you are interested in that.

How can any organisation actually map their dataflow and what things need to be considered to do so. It'll come across like common sense, but that's why an actual standard is so desperately needed!

[fuzzyghost]

[landscape-stories]

LS 27 | Machine INFLUENCES

Henrik Spohler - Dataflow

magazine.landscapestories.net

[irr3alism]

F L 0 W

[ayeforscotland]

What is Dataflow? Part 2: Diagrams

This is the second part of a couple of posts about Dataflow, particularly why it's important for the world going forward and relating to the Crowd Strike IT disaster.

Read the first part here.

Before I get into this one today, I wanted to address a couple of things.

Firstly, Dataflow is something that nearly every single person can understand. You do NOT:

Need to have a degree in Computing Science

Need to work in IT

Need to be a data analyst / Spreadsheet master

If any of you see the word 'Data' and feel your eyes glazing over, try and snap out of it because, if you're anything like me, Dataflow is much more approachable as a concept.

Secondly, what do I mean by IT?

Traditionally in most of our media the all-encompassing 'IT department' handles everything to do with technology. But every business works differently and there are many job titles with lots of crossover.

For example, you can be an infrastructure engineer where your focus is on building and maintaining the IT infrastructure that connects your organisation internally and externally. This is a completely different role from an Application Portfolio Manager who is tasked with looking after the Applications used in business processes.

Both are technical people and come under the banner of 'IT' - but their roles are focused in different areas. So just bear that in mind!

Now that's out of the way, let's begin! This one will be a little bit deeper, and questions welcome!

An Intro to Diagrams

You probably do not need a history of why pictures are important to the human race but to cover our bases, ever since we put traced our hands on a cave wall we have been using pictures to communicate.

Jump forward in time and you have engineers like Leonardo Da Vinci drafting engineering schematics.

You get the idea, humans have been creating diagrams (Pictures) for thousands of years. Centuries of refinement and we have much more modern variations.

And there's one main reason why diagrams are important: They are a Common Language.

In this context, a Common Language helps bridge a language gap between disciplines as well as a linguistic gap. A Spanish electrician and a German electrician should be able to refer to the same diagram and understand each other, even if they don't know each other's language.

The reason they can do this is because they're are international standards which govern how electrical diagrams are created.

A Common Language for Digital?

Here's an image I've shown to clients from governments and institutions to global organisations.

Everything around us, from the products we use to the bridges we drive over and the buildings we live, work, enjoy and shop in had diagrams backing them.

You would not build a skyscraper without a structural engineering diagram, you would not build an extension on your house if an architect couldn't produce a blueprint.

Why is there not an equivalent for the Digital World and for Dataflow?

Where is the Digital Common Language?

This is the bit where the lightbulb goes on in a lot of people's heads. Because, as I mentioned in Part 1, the flow of data is the flow of information and knowledge. And the common mistake is that people think of dataflow, and only ever think about the technology.

Dataflow is the flow of information between People, Business Processes *and* Technology Assets.

It is not reserved to Technology specialists. When you look at the flow of data, you need to understand the People (Stakeholders) at the top, the processes that they perform (and the processes which use the data) and the technology assets that support that data.

The reason why this is important is because it puts the entire organisation in context.

It is something that modern businesses fail to do. They might have flow charts and network diagrams, and these are 'alright' in specific contexts, but they fall to pieces when they lack the context of the full organisation.

For example, here is a Network Diagram. It is probably of *some* value to technical personnel who work in infrastructure. Worth bearing in mind, some organisations don't even have something like this.

To be absolutely clear, this diagram will hold some value for some people within the organisation. I'm not saying it's completely useless. But for almost everyone else, it is entirely out of context, especially for any non-technical people.

So it doesn't help non-technical people understand why all of these assets are important, and it doesn't help infrastructure teams articulate the importance of any of these assets.

What happens if one of those switches or routers fails? What's the impact on the organisation? Who is affected? The diagram above does not answer those questions.

On the other side of the business we have process diagrams (aka workflow diagrams) which look like this.

Again we run into the same problem - This is maybe useful for some people working up at the process layer, but even then it doesn't provide context for the stakeholders involved (Are there multiple people/departments involved throughout) and it doesn't provide any context for technical personnel who are responsible for maintaining the technology that supports this process.

In short, nobody has the big picture because there is not a common language between Business & IT.

Conclusion

So what do we do? Well we need to have a Common Language between Business & IT. While we need people with cross-functional knowledge, we also need a common language (or common framework) for both sides of the organisation to actually understand each other.

Otherwise you get massively siloed departments completely winging their disaster recovery strategies when things like Crowd Strike goes down.

Senior Management will be asked questions about what needs to be prioritised and they won't have answers because they aren't thinking in terms of Dataflow.

It's not just 'We need to turn on everything again' - It's a question of priorities.

Thing is, there's a relatively simple way to do it, in a way that looking at any engineering diagram feels simple but actually has had decades/centuries of thought behind it. It almost feels like complete common sense.

I'll save it for Part 3 if you're interested in me continuing and I'll make a diagram of my blog.

The important thing is mapping out all the connections and dependencies, and there's not some magic button you press that does it all.

But rigorous engineering work is exactly that, you can't fudge it with a half-arsed attempt. You need to be proactive, instead of reacting whenever disaster strikes.

[ayeforscotland]

Tumblr’s totally not the place for it but damn I could talk about how to properly map data flow for hours.

Like it’s genuinely exciting non-dystopian digital future stuff when you think how it could help organisations.

It fucking runs circles around businesses crowing about how they’re ’implementing AI’ (they are not).

Would genuinely create a sideblog or something to teach folk but again, tumblr probably not the platform for it.

[ayeforscotland]

With INEOS closing down Scotland’s last oil refinery in Grangemouth, we really need to be having a conversation about a new national industry.

Scotland has engineers who’ve specialised in process control and optimising the flow of oil, and those skills can be applied to optimising the flow of data by using the same engineering principals.

We should have started preparing for this a decade ago, the next best time to do it is now.

[ayeforscotland]

What is Dataflow? Part 3: Doing the Practical

Apologies for the delay in getting this next section up - past few weeks have been super busy and then, hilariously, I was ill last week.

Read Part 1 here.

Read Part 2 here.

In Part 2 I wrote about how important diagrams have been throughout history. Understanding the 'big picture' has been important for every triumph of engineering. From bridges to skyscrapers to oil rigs and wind turbines, all of these have had diagrams backed by international standards which enabled them to be built.

The digital world hasn't quite managed that yet. In the other posts I've tried to drill home the point that modern digital businesses are often extremely siloed, communication and documentation isn't there and there is a lack of a common language between 'Business' and 'IT'.

This lack of understanding means organisations do not understand how data flows through their business and their supply chain.

It's the understanding of dataflow that's important here because it enables organisations to focus on optimising, securing and maintaining flows across an organisation rather than siloed teams patching things up where they can and not understand the upstream and downstream impact on the business.

Method and Layers

Going to preface this by saying that this may come across as complete common sense, and to some extent you'll be completely correct!

This is an example of how to create a very basic dataflow. But I will first start with understanding all of the People, Processes and Technology that I use to post on Tumblr.

So I start with six layers:

Ownership

Business Process

Application

System

Hardware

Infrastructure

What is important to remember here is that you do not have to be a specialist in every single layer.

A Business Analyst will feel much more at home in the Business Process Layer, while an Infrastructure Manager will be much more knowledgeable about the Infrastructure layer.

The important thing is that this Business & IT Diagram allows them to communicate more efficiently.

Let's Build a Dataflow!

In this example - There's an 'AyeforScotland' Element (the rectangle!) at the top. I'm the owner of everything below that element. The black lines are 'connections' representing the connectivity between the different elements.

Following the example, I'm responsible for' managing my blog 'Blog Management' which breaks down into smaller processes: Draft posts, schedule posts, answer anon abuse, and reblog shitposts.

Coming down to the Application Layer (red) - You can see that I draft and schedule posts using Tumblr Desktop and I'm using Firefox Web Browser for that.

But for answering anon abuse and reblogging shitposts, I'm using the Tumblr App.

In the Systems layer you can see I'm using Windows 11 on my PC (Hardware) and I'm using iOS on my iPhone.

Both my PC and iPhone connect to my BT Router.

And that's it for this Business & IT Diagram. I've shown clearly how I'm responsible for the processes and how I use the technology to perform those processes. I don't necessarily need to show everything on a single diagram because it would lose clarity.

This next Business & IT Diagram is much smaller, and establishes the relationships and dependencies on Tumblr to provide the service. And that's because we're complying with the laws and rules of a methodology.

In this diagram (probably need to zoom in to see it) I'm at the top left as 'AyeforScotland' and my 'BT Router' is spatially below me. Following the rules and laws of the method, that maintains the relationship that I have with the BT Router, I own it.

But I don't own the small 'Internet' that's next to it horizontally. I've simplified the concept of the internet for this example.

There's also two owners - 'Automattic' which owns and operates 'Tumblr' below it, with Tumblr being responsible for the 'Provision of Tumblr Services'.

Now naturally 'provision of Tumblr services' will break out into loads of sub-processes. Tumblr could map out their entire organisation (and if they need a hand, they can DM me!) But for this dataflow it's not really required.

Now both diagrams above are not dataflows. But close your eyes for a second and you can visualise what they are.

But because we've created our two diagrams, we understand the connectivity and using the software we can create the dataflow.

Now again, this is very basic. But when you put things into a dataflow context, you can put this down in front of a wide range of people from different business disciplines and they can start to optimise how the business works.

Here's a much larger Dataflow example, that you won't be able to read because it exceeds A0 printing size, but it should convey the scale.

If any of the connections or elements fail along this dataflow - The dataflow stops.

This costs organisations time and/or money.

So understanding dataflows allows IT people to articulate to business people "Hi boss, if this server goes down it will bring down this dataflow and cost the business $10,000 an hour" - Suddenly it's in a language they understand.

It helps with strategic decision-making, it helps with communication, it helps document how things *actually* work as opposed to how people think they work, and once you switch to thinking in terms of 'dataflow' it's hard to stop.

Conclusion

I can't wait to answer all the questions on the back of this.

Also one area I didn't go into is that each of the elements (rectangles) can also hold data (Financial data, Technical Specs, Risk & Cybersecurity metrics, Governance documentation etc).

It's also really easy to get started with it. You can start in any of the layers based on your area of work.