Maturity Models and genAI

I've just had a notification from LinkeDin exhorting me to keep up with cybersecurity and artificial intelligence frameworks and maturity models.

I assume that when they say artificial intelligence, they really mean generative artificial intelligence, since the world, at large, seems to have forgotten the many other approaches to artificial intelligence, such as expert systems, game theory, and pattern recognition.  (Computers, at least until we get quantum computers, seem to be particularly bad at pattern recognition.  I tend to tell people that this is because computers have no natural predators.)

I have no problems with frameworks.  I have been teaching about cybersecurity frameworks for a quarter of a century now.  Since I've been teaching about them, I have also had to explore, in considerable depth, frameworks in regard to capital risk (from the finance industry), business analysis breakdown frameworks, checklist security frameworks, cyclical business improvement and enhancement frameworks, and a number of others.  I've got a specialty presentation on the topic for conferences.  I include maturity models.  In a fair amount of detail.  It's an important model within the field of frameworks.  It not only tells you are where you are, but in strategic terms, what type of steps to take next, in terms of improving your overall business operations.

But a capability and maturity model?  For a technology, and even an industry, that didn't even exist four years ago?

Okay, let's set aside, for a moment, the fact that the entire industry is only four years old.  We needn't argue about that.  I've got a much stronger case to make that this is a really stupid idea.

Capability maturity models, in general, have five steps.  (Yes, I know, there are some people who add a sixth step, and sometimes even a 7th, usually in between the existing steps.)  But let's just stick with the basic maturity model model.

The first step is usually "chaotic."  Some models now call this first step "initial," rather than "chaotic," since nobody thinks that they work in a chaotic industry.  But, let's face it: when a new industry starts up, it's chaos.  You really don't know what you're doing.  If you are really lucky, you succeed, in that you make enough revenue, or you have patient enough investors, to continue on until you find out what you are doing, and how to make enough revenue to survive, by doing it.  That's chaotic.  It doesn't mean that you aren't working hard.  It doesn't mean that you don't have at least some idea of what you are doing, and the technology, or the business model, that you are working with.  But, that's just the nature of a startup.  You don't have a really good idea of what you are doing.  You don't have a really good idea of what the market is.  You may have some idea of what your customers are like, but you don't have an awful lot of hard information about that.  It's basically chaos.

That's basically where generative artificial intelligence is right now.

Building upon the idea of neural networks, which is a been around for eighty years (and was deeply flawed even to begin with), about a dozen companies have been able to build large language models.  These LLMs have been able to pass the Turing test.  If you're chatting with a chatbot, you're not really sure whether you're chatting with a chatbot, or some really boring person who happens to be able to call up dictionary entries really quickly.  We know enough about neural networks, and Markov chain analysis, and Bayesian analysis, to have a very rough idea of how to build these models, and how they operate.  But we still don't really know how they are coming up with what they're coming up with.  We haven't been able to figure out how not to get them to just simply make stuff up, and tell us wildly wrong "facts."  We haven't been able, sufficiently reliably, to tell them not to tell us stuff that's really, really dangerous.  We try to put guard rails on them, but we keep on getting surprised by how often they present us with particularly dangerous text, in ways we never expected.

We don't know what we're doing.  Not really.  So it's chaotic.

We don't really know what we're doing.  So, we don't really know, quite yet, how to make money off of what we're doing.  Yes, some businesses have been able to find specific niches where the currently available functions of large language models can be rented, and then packaged, to provide useful help in some specific fields.  Some companies that are on the edges of this idea of genAI are able to rent LLM capabilities from the few companies that have built large language models, and have been able to find particular tasks, which they can then perform for businesses, and get enough revenue to survive.  And yes, through low rank adaptation, either the major large language model companies, or some companies that are renting basic functions from them, are able to produce specialty generative AI functions, and make businesses out of them.  But the industry as a whole, overall, is still spending an awful lot more money building the large language model models then the industry, as a whole, is making in revenue.  So we still don't know how generative artificial intelligence works, and we still haven't figured out how to make money from it.  It's chaotic.

But another point about capability maturity models is that the second step is "repeatable."  The initial step, chaotic, is where you don't know what you're doing.  The second step is when you know that you can do it again (even if you *still* don't know what you're doing).

And even the companies, the relatively few companies, who have actually built large language models from scratch, haven't done it again.

Oh yes, I know.  The companies that have made large language models keep on changing the version numbers.  And each version comes out with new features, or functions, and becomes a bit better than the one with the version number before it.

The thing is, you will notice that they still keep the same basic name for their product.  That's because, really, this is still the same basic large language model.  It's just that the company has thrown more hardware at it, and more memory storage, and possibly even built data centres in different locations, and shoveled in more, and more, and more data for the large language model to munch on, and extend it's statistical database further and further.  Nobody has built another, and completely different, large language model, after they have built the first one.

In the first place, it's bloody expensive.  You have to build an enormous computer, with an enormous number of processing cores, and an enormous number of specialty statistical processing units, and enormous amounts of memory to store all of the data that your large language model is crunching on, and it requires enormous amounts of energy to run it all, and it requires enormous amounts of energy, and probably an awful lot of water, to take the waste heat away from your computers so that they don't fry themselves.

And you've now got competitors, chomping at your heels, and you can't waste time risking enormous amounts of money, even if you can get a lot of investors eager to give you that money, trying a new, and unproven, approach to building large language models, when you already have a large language model which is working, even if you don't know how well it's working.  So nobody is going to repeat all the work that they did in the first place, when they've got all this competition that they have to keep ahead of.  When they have a large language model, which they really don't understand, and they are trying desperately to figure out what the large language model is doing, so that they can fix some of the bugs in it, and make it work better.  Even if they don't really know how it works.

Okay, yes, you can probably argue that the competitors are, in fact, repeating what you're doing.  Except that they don't know what *they're* doing, either.  All of these companies have the generative artificial intelligence tiger by the tail, and they aren't really in charge of it.  Not until they can figure out what the heck it is doing.

I'm not sure that that counts as the "repeatable" stage of a maturity model.

And the third stage is "documented."  At the "documented" stage, you definitely *do* have to understand what you're doing, so that you can document what you are doing.  And yes, all of the general artificial intelligence companies are looking, as deeply as they can, as far as they can, into the large language model that they have produced, and are continuing, constantly, to enhance.  The thing is, while, yes, they are producing some documentation in this regard, it's definitely not the whole model that is completely documented.  Yes, they are starting to find out some interesting things about the large language models.  They are starting to find out, by analyzing the statistical model that the large language models are producing, what might be useful, and what might be creating problems.  But nobody's got a really good handle on this.  (The way you can tell that people really don't have a good handle on this, is that the large language model companies are spending so much money, all over the world, lobbying governments to try and prevent the governments from creating regulations to regulate generative artificial intelligence. If the genAI companies knew what they were doing, they would have some ideas on what kind of regulations are helpful, and what kind of regulations would help make the industry safer, and what kind of business and revenue regulations might affect.  But they don't actually know what they're doing, and therefore they are terrified that the governments might [probably accidentally] cut off a profitable revenue stream, or even just a potentially useful function for generative artificial intelligence.)

So, no.  You can't have an artificial intelligence capability maturity model.  Yet.  Because we don't know what generative artificial intelligence is.  Yet.