All Things Techie With Huge, Unstructured, Intuitive Leaps

Ding Dong, The Mouse is Dead -- Extreme UX

The computer mouse was one of the most innovative advanced inventions for inputing data and instructions to a computer. It was a stroke of genius that enabled the acceleration of the dominance of personal computers in our lives. But, like all things, the mouse is almost dead.

Prior to the mouse, one had to remember the entire command lexicon of the operating system, and then type in command with the appropriate switches and parameters. UNIX gurus still do this today, but they are born on a different planet of nerdom. I know, because I am one of them. The mouse enabled one to drop down one of twenty menus, fan out the sub-menus and make a selection. The mouse was a precision instrument, and the need for a scroll wheel was proof of the need to go through a plethora of menus.

The thin edge of the wedge for the death knell of the mouse was the iPhone. The thumb and forefinger became the chief method of selection of menu items. The iPad and plethora of smart phones sealed the deal for the death of the mouse. This is a trend that isn't going away anytime soon. As a matter of fact, there is now a phone that is your laptop.

So what does this mean for UX or User Experience. We have to be all thumbs -- or mouseless. This is a huge challenge.

A thumb-driven menu takes up a lot of screen real estate. And it is a mortal sin to give up functionality, just because you are using an application or a web page on your portable device. What is now needed is extreme UIX techniques, especially when it comes to menus.

We have to ditch the massive hierarchical menu model and be a lot smarter. Gone are the days when all you had to do was add the following to your code:
MenuItem *myMenuItemFile = new MenuItem(S"&File");    MenuItem *myMenuItemNew = new MenuItem(S"&New");

Tests in UX show that there are definite hot spots on the application or web screen where the human eye is drawn to first. Humans are highly adaptable and quickly settle into patterns that reflect brain function. Wireframes and testing shows that users look at screens in an "F" pattern. The first thing we look at is the horizontal bar of the F. In other words, we look at the very first bit of content. This does not include or mean that we look at the little menu items at the top of the page. It doesn't take long for our subconscious mind to figure out that those are bitty unimportant menu items that aren't necessary until we really need them.

If we are designing a User Experience that is engaging, we should let the content rule on the top bar of the F. After all content is king. People don't come for the menus.

However if actions are required on your web page or applications, the menu is an important function. So, original web designers got it right when putting a menu on the left hand side of the page. Thousands of views of a screen have taught our minds to look there.

As to the menu itself, we do not have the luxury of putting a gazillion menu items, drop down menus filled with options and choices galore on a menu. Even the concept of displaying only the menus you need is not productive in a thumb-driven environment, because the user doesn't know what she or he is going to need until they use it.

Psychologists tell us that humans are best with 4 to seven choices. So in Extreme UX, that is all the top menu or main menu will have. I can hear the gasps now. Imagine a program like Microsoft Word with only four top menu items. You can't, because Word is written by dinosaurs, exploiting the last bit of the 1980's that they can wring out for monetary gain.

So how would an extreme programming do it? With brilliance and envelope-pushing code. I would like to introduce a term to you: "Application Use Lifecycle". Huh?

Just like software development has a life cycle, the use of an application has a life cycle. When an application is invoked, and you are starting a new output of that application, you require a very different set of commands than if you open an object that has already had work done on it in that application.

Mapping out the stages of the Application Use Lifecycle will give you a good idea of what menu items are need when. The next step is to write some supervisor code to figure out where you are in the Application Use Lifecycle. By determining this, you will know what menu items are appropriate for that use.

If you want a tutorial on how to do this, you will have to wait for my book to come out on this -- or you will have to hire me to teach your programmers. However, I can guarantee you that with the death of the mouse, you will have to adopt something like this to accommodate the thumb-driven menus.

There are positives with the death of a mouse. If you see a dead mouse, then you know that it is safe to go ahead and score the cheese.

More on this topic to come.

Making Your Computer Into a Parallel You

I have previously introduced my concept of gramlets which is a whole bunch of single function programs that do one thing only. They are standalone programs of a tiny size, and can be invoked to operate on anything and do what they do. One example is the Addition Gramlet For Integers that will add any integers given to it as parameters and return the sum. This is but a trivial example.

Where gramlets will come to shine, is in the evolution of the personal computer. Personal computers have come a long way, and soon your phone will be your personal computer. Or it might be in the fabric of your jacket, or stuck in your ear -- it doesn't matter. The personal computer will continue to evolve.

It will become smaller and smaller, and can even off-load large processing tasks to the cloud. Douglas Adams may not have been far off the mark, when he said in the Hitchhiker's Guide to the Galaxy that the Earth was one huge computer.

The operating system for the personal computer must evolve as well. Gone will be the days of the dumb desktop. Your personal computer will be a learning machine that will learn all about you, anticipate your needs and handle most of the digital aspects of your life autonomously.

To become a parallel you, the operating system of the personal computer will have to ditch the ancient Microsoft paradigm and turn into a massively parallel system, just like the human brain.

Input will be fed into your personal computer via any means (web, data entry by scanner, camera etc) and the computer will take care of it. The input will be fed into all of the gramlets. They will fire if they recognize that the data is meant for them and process it.

For example, an image is fed in. The word processor and speech processor gramlet ignore it. The Filing Gramlet puts it away with the rest of the pictures, but first it passes it to the ident unit which analyses the picture to see if it is a picture of you, friends, family or whatever. Then the image is passed to the ImageQualityGramlet which adjusts the white balance, removes the red eye and passes it back for filing in the appropriate place. In the process, every Gramlet saw this input, and if the threshold was met, the gramlet recognized the data and did its job on it.

A parallel system of gramlets that learn by watching you will be the hallmark of the new personal computer operating system.

Anyone dealing with you will not be able to tell if it is really you, or the parallel you embedded in your personal computer. This will be like the graphic at the top of this post. Are the lines parallel or not? The answer will surprise you.

Solving The "ish" Function

Quick question: "What is sort of like a dog, belongs to the dog family, has the features of a dog, sort of looks like a dog, but isn't really a dog, but is real "dog-ish"? Can you pick out the answer in the photo below?

What the human brain is very good at, is pattern recognition, or in pseudo techno talk, solving the "ish" function. Our minds can make huge intuitive leaps on very little information. We correctly surmise a correct conclusion from imperfect clues. It serves us well in the real world, because that is a necessity in many situations, including survival situations.

Computers are not so good at the "ish" function. We carbon units have neural nets that are superb at pattern recognition. When you are reading these words, your are not constructing them in your head from the assemblage of letters, rather your brain is instantly recognizing the pattern of the word. That is why you can udenrtnasd tihs snetncee if jsut the frsit and lsat ltetres are itnact.

What got me thinking about the ish function is in the following photo below (click on it for a larger image):

This comes from the actual FBI website. It is a code that they have been trying to crack for 10 years and have failed. Finally they put it up on their website and they are asking the publics's help to solve the puzzle. The URL is

The back story is that a man was murdered and the only thing in his pocket, were two pages of writing written in some sort of code. The best codebreakers in the world have been toiling for 10 years trying to break this code. After putting up the story on their website, they have gotten over 2,000 tips and comments, including one from me.

What I did, was let my own ish circuits process the code. A couple of things jumped out at me. There are repetitions, but I don't think this is cryptography. Cryptography is where you substitute one letter for another, for example an E would be substituted with a J and L would be substitued with a Z. In this way the word "eel" would be written as JJZ.

Rather, I think this is shorthand, where just a few letters represent the whole word. And my contention is that this is bookie shorthand. For example, one can see the word "CBET". Everyone knows that a C note is a 100, so a CBET is code for a hundred dollar bet. You can also make out "TOTE". A tote bet is for win, place or show on horse race.

So this note would be eminently solvable quickly with a built-in ish function in an operating system. One would give the ish function a glossary of betting terms, and the computer would spew out the solutions. One quick ish function that my brain did, was assign the sequence "NCBE" as a Nickel Bet or a $500 bet.

To write an ish function is harder than it looks. Let's suppose that you had to pick out the word GLANDULAR from this sequence FRGGNGBIGGLNDLREGO. There are no white spaces to help you out. What you have to do in general terms is to first look for the general wildcard symbol G*R. In this case it would return 5 possibilities, and then you have have to eliminate the ones that didn't fit. The human brain doesn't have much trouble seeing NiCkelBEt in NCBE, but the ish function would require a lot of flops to tentatively figure it out.

However the ish function would be mighty handy. It would greatly improve OCR or optical character recognition. It would include ROM or Rough Order of Magnitude. When you and I are working on a problem, and our frame of reference tells us that the possible range of answers lie say between 46 and 59, if we get an answer close to that range, we know that we are on the right track.

In the old days, computers were assigned tasks to determine exact values. Quantum physics and fuzzy logic has shown us the shortcomings of exact values. The human brain doesn't operate with exact values, and if we want computers to emulate the human brain, then we need a real good ish function for words, math, pictures, images, meanings and memes. This list isn't comprehensive, but it is ish!

Application for Emotion-Sensing Software

In the blog entry below, I discuss how to go about writing software that senses emotion. After I wrote the blog article, a superb application came to -- customer service help lines.

These days, a lot of companies initially answer help lines with an automated answering system and a computerized kiosk helps the caller navigate to the right person.

If you had a software robot assessing the calls, and that software robot could tell if the caller was angry or frustrated, then the software would route that person immediately to a human. This kind of software could be valuable to companies who handle a large amount of customer calls per day.

Then the cynical side of me pipes up and says, maybe companies would rather have an irate customer dealt with by the computer. It saves emotional wear and tear on the customer service agent, and computers are much cheaper than humans.

Whatever way it goes, it won't be long before this emotional awareness is built into software.

Software that Emotes -- Going to the Dogs

Capturing Emotion Using Software

Hanging in a bilingual country has taught me a lot language and communication -- especially intra-species communication. For example, on my walks and hikes, I often run into dogs that do not understand English commands. They were trained totally in the French language. They have run ahead of their masters on the trail and run into me. Of course, I don't know that they are members of French speaking family, but it really doesn't matter. Dogs are good at getting the gist and meaning of commands from humans. I can get a well-behaved dog to sit and wait for its master, even though it doesn't understand English.

How does a dog do this? A dog can gain a lot of clues from the way the commands sound. If I firmly tell the dog to "Sit" in a sharp voice, it usually does. A dog can judge the type of command by the timbre of the voice. A dog figures out the command by the qualities of the sound of the spoken word.

Can a computer be trained to do the same? I realized of course, that if we are ever to get a true pass on a Turing test where a person doesn't know if they are talking to a computer or human, we have to have software that emotes. To have emotion, one must be able of recognizing emotion. To get an easy algorithm, I went to the example of the French speaking dogs.

When I analyze what is going on, I realize that what dogs respond to is the various components of the sound of a command. The primary components of the sound wave are attack, sustain and decay. A command to sit or stay has a fast attack, a short sustain and a fast decay. A soothing sound has a slow attack, a long sustain and a long decay.

Attack is how fast or the rate that the sound reaches its peak amplitude. Sustain is the body of how long the sound lasts after it reaches peak amplitude, and decay is how it fades away. A sound that scares you has a fast attack. It also grabs your attention.

So the algorithm for a software program to determine emotion, is to break the sound wave down into Attack, Sustain and Decay. The real neat part about doing it this way, is that it is language independent. Whether you say "Cara mia, ti voglio bene", or "Ma Belle, je t'aime" or "My darling, I love you", there will be no fast attacks and short decays.

Individual words that convey emotion usually demonstrate Onomatopoeia, or sounding like the source of the sound. For example the word hiccup sounds like a hiccup, and a smooth sibilant stream is peaceful. Hence by rating each word for ASD or Attack/Sustain/Decay, one can rate a sentence for the emotion it has. One can do it mathematically or let a series of artificial neural nets be trained to recognize various ASD patterns. The bottom line is that while one cannot get the full range of emotion from just the components of the sound, one can use that as a good starting point as to what emotions are being expressed.

So there you have it space cadets -- a methodology for software that emotes. Source code to follow. Status of project for serious application -- requires funding.

New Paradigm -- Crap Software

I am amazed at how flaky Twitter is, and yet it is a major force to be reckoned with in social media. Often times when I try to log it, it says "Twitter is busy now. Try again later."

Just this morning, I tried logging and saw this message:

SSL connection error
Unable to make a secure connection to the server. This may be a problem with the server, or it may be requiring a client authentication certificate that you don't have.
Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.

I thought that it was something wrong with my machine because the error was thrown by which is localhost or my computer. So I tried logging into Facebook, and there was no problem. I immediately knew that it was another Twitter issue.

It blows me away that there is a new paradigm in software -- putting out crap or buggy software, just to occupy a market space. The bugs get fixed later. If Twitter were production software in a company, it wouldn't cut the mustard.

In a similar vein, I use GIMP a lot to manipulate images. For the most part, it is OK, but when you do a series of transforms on the graphic, it has a tendency to crash. Regularly. GIMP is free, so you can't complain too much, but it still represents the concept of a less-than-perfect element thrown out to the public.

I'm willing to bet that what we will see, is software becoming obsolete before all of the bugs are fixed.

Is there anything to be done about it? I doubt it because although Twitter is a private company, and hasn't made money, it has huge potential to transform into super currency like Facebook did. As long as there is no financial incentive to put out good software, we will get buggy crap in our virtual worlds.

Morning Gramlets

Gramlets are programlets. Take a software program and bust it into pieces making each method a standalone program. Can you imagine how memory efficient and fast a program like that would be? It is a programming methodology whose time has come. Read the blog entry immediately below this one for my introduction to gramlets.

To make gramlets work, you would need a persistent supervisor of menus (or gramlet supermen) program that was the menu director. It would discover all of the gramlets that you have loaded or available online and give you menu choices to call them.

Incorporated into the gramlet supermen would be the ability to create macros that strings gramlets together to form functions.

The neat thing is that there could be a gramlet exchange much like a stock exchange where one could buy individual gramlets or gramlets strung together by macros or a connector gramlet.

Half-Baked Gramlets

Warning: This idea is just half-baked, but I wanted to capture the meme before I lost it.

I was shocked with nostalgia to see a sample of BASIC programming lately. It was a very simple example and looked something like this:

10 REM BASIC Program
20 REM
40 FOR I=1 TO 10
80 END

This was way before Object Oriented Programming. It is a trivial example of the language. In BASIC programs that actually did real work there were sub-routines, goto statements and various loops and such all in one monolithic program.

These sorts of programs were the original layer of abstraction from programming a computer with wires. The programming of computers with a language sort of led us down the alternate path of programming in the same way that Alexander Graham Bell diverted us from our digital future with an analog telephone for all those years. I am not saying that programming with a language instead of wires was a mistake, just saying that with my newly-hatched idea, monolithic programs like BASIC heavily influences us today, and in perhaps the wrong way.

The huge monolithic programs like BASIC, COBOL and even AS400 were bulky and unwieldy. Subroutines provided the first reusable code, and perhaps are the beginnings of object oriented programming. Just maybe though, object oriented programming is not enough.

The legacy of the original monolithic programs still haunts us. For example, have you opened Microsoft Word lately. It is, I am sure, an object oriented program, and yet it is a huge accretion of objects that make it indistinguishable from the old huge BASIC programs of long ago. There are functions to spell-check. There are functions for mail-merge. There are functions to generate table of contents. When the change came from Word2003 to Word2007, I had to re-learn how to find the things that I wanted. It is so bulky, that if I want to capture some text, I just use Notepad, whereas before I used to use Word.

Then I started developing with Windows Mobile. Of course, you can't use the same memory-hogging techniques that go into programs like Word. With devices getting smaller and smaller, even with Moores Law still having room to expand, having processing power in all places is the goal. To do that, you have to have programs that are super memory efficient. Just like cars running on less gas, you have to have programs running on less memory.

That is where I was struck with the idea of Programlets, or Gramlets. (Like I say, this idea is half-baked, and I haven't yet Googled to see if anyone is doing it). Gramlets are mini-programs broken down into the smallest logical components. And each Gramlet has a method to invoke a Runner which invokes another Gramlet.

Just like Neural Nets, instead of firing up a large monolithic program like Word just to do one function, you fire up a program that does just one function. The stack and heap footprints would be very small, and the program would exit once you have finished the function. Here is an example in kiddie-scripting C# for the runner program:

class Runner
static void Main(string[] args)

These are very trivial examples, but if you had to do mathematical addition, you would call the runner to call the adder and feed it two numbers:

class AdderInt
static int Main(int[] args)
return args[0] + args[1];

A more useful example, is lets say you have a gramlet to catch byte stream to create a document. The byte stream capture calls the runner to open a Notepad gramlet. Once the byte stream is captured, it exits. If you wanted to do a spellcheck, you would call a spellcheck gramlet. The spellcheck gramlet could be used anywhere -- in a browser, in a spreadsheet -- where ever it was needed.

Each gramlet is a standalone executable. This way, one could program by linking gramlets. This would be the ultimate manifestation Agile Developments.

This half-baked idea is going back into the oven for some more baking.

RugbyMetrics Performance Analysis Themes

Data mining and predictive analytics is a wonderful thing. It gives objective insight into whatever comes under the data microscope. In this case it is rugby. The insights are fascinating. For example, take a look at this equation:

It's called a Hurst exponent, and it was derived by a mathematics God named Benoit Mandelbrot. It is a Mandelbroatian math element of fractal geometry. It was originally developed to determine how big a dam to build on the Nile River. What does it have to do with rugby?

Let's look at the analogy of the Nile River. In the case of the Nile River, one expects to find ebbs and flows of the amount of water flowing through the river based on rain and drought. The series is infinite as long as the Nile does not run dry. The Hurst exponent is used to estimate variability of the flow over time.

In rugby there are ebbs and flows during the game in terms of meters gained on the pitch by a particular team. Of course, the time is not infinite, but it is 80 minutes. During those 80 minutes the game flows back and forth. If I calculate the variability of meters gained per play during a game using the Hurst exponent, it infers different things about the teams.

The Hurst exponent is defined in terms of the asymptotic behaviour of the rescaled range as a function of the time span of a time series.

Let's suppose that I analyze a video of a rugby game and just for fun, determine the Hurst exponent of the opposing team. Let's suppose that their variability in meters gained on the field is higher than my team. There are a few reasons why a team is highly variable in terms of meters gained in play. Finding that reason shows a vulnerability and something for the opposition to exploit.

If I take this same concept and apply it to a finer degree on granularity at the player level, I can determine by comparative analysis if a player is ready to play or still not up to snuff after an injury.

Analysis and number crunching of this kind yields an amazing amount of objective knowledge about the game that was previously unknown. And this is the type of knowledge that gives teams an incredible advantage over mere human coaching.

As for software, the neat thing about this stuff, is that SQL stored procedures and views are the input from the data mart to determine these things. One needs the game dissected very finely and then non-jagged data for the math transforms to operate on the returned cursors from the data cubes. Data can be made to spill its guts.

Multi-Layer-Perceptrons On Silicon

I having a mental blast playing with multi-layer perceptrons, artificial intelligence and neural nets. I have built a few applications with MLP's. If you don't know what MLP's are, here is a LINK. ( ).

I have built them in Java, Visual Basic and C# .NET and have given them fairly interesting things to do. I once had a website using them to predict the Dow Jones average (until September 11 killed it). I now give them mental problems to do.

The most amazing thing about artificial intelligence, is when you first realize that a machine made an independent decision that was not programmed in.

I was musing on AI and MLP's especially and hearkened back to the day when I used to draw out FPLA's or Field Programmable Logic Arrays. Essentially they were silicon chips full of logic arrays (ANDs, NANDs, NORs, FlipFlops, Exclusive ORs and all of the Boolean gates of logic) all connected with internal fuses. You connected up what you wanted internally and blew out the rest of the connections in the same way that you would bomb a PROM.

So, the big Eureka moment is this. I want to put multilayer perceptrons into silicon. Should be child's play to do with a combination of FLPA and R/W PLA's. The thresholds of each of the nodes would hold the intelligence of the neural nets so that the training epochs result would be held in memory.

Imagine a single chip that would learn to recognize a driver in a car and react to that driver to set up the car parameters for that driver. Then when the car was sold, the chip would go into a learning mode to recognize the new driver. This would work well with fly-by-wire systems of all kinds.

This could be a big thing.

RugbyMetrics Performance Analysis Software

(click on pic for a larger image)

I am about to test my new rugby video and performance analysis software on an Aviva Premiership professional rugby team.

This entire exercise is akin to Sabremetrics in North American professional baseball where one uses data-mining and intense statistical analysis to discover scientific, objective knowledge about the game that was previously unknown.

For example, in baseball a Seattle-based team with a limited budget of $40 million had to compete with a team like the New York Yankees who could afford marquee players with a salary budget of $125 million. It was found that conventional wisdom about assembling a team of good players was neither conventional, nor wisdom.

In the old convention, all of the scouts looked for a young pitcher with a blistering fastball and a high strikeout count. Through data analysis, it was found that not only was a pitcher with a high ground out ratio more valuable, but he was also cheaper on the open market.

This intensive analysis has been done with baseball, football and basketball. No one has yet done it with rugby. Rugby is a very dynamic game and stats abound, but they are all conventional stats that really do not measure what a player does minute by minute and apply that to the greater context of the play. Some software packages give you a minute by minute report, but it is meaningless unless you take it in the context of inputs, outputs and results, both on a micro and macro level. There is tons of minute-by-minute data, but very little pattern knowledge of what that data means.

I developed the piece of video analysis and annotation software that models and digitizes the entire game second by second. It is all put into a database in a proprietary format that allows extensive data-mining and statistical analysis of every second of every game.

There are a few software packages that give you a real wow-factor re-simulation of the game, and a whole pile of individual statistics about every facet of the game, but these are not very helpful in real life. What is helpful is garnering new knowledge -- objective knowledge about the sport and about your team. The way that it is done, is by taking statistics, facts, data and factors, and integrating that data into knowledge by making inferences, testing those inferences and endless cutting and re-dimensioning of data until statistically significant knowledge about the team and the game pops out. Nobody is doing this out there, and I will be the first. Like Sabremetrics, this will add objective knowledge about the sport of rugby on the macro level.

The most important aspect of this, is that the tool can isolate and chart characteristic on-field behaviour of the opposition, thus allowing the planning of a defense for a formidable opponent. Teams that are well coached consist of human beings with ingrained play patterns, and it is a valuable tactical advantage for an opposing team to know this. If you know the attack vector that is coming, you can plan for it. But it is not enough to know the attack vector, you have to know the probabilities and how it will play out, and how it changes with different parameters.

A model is being developed using Bayesian Predictive Analytics that will predict and spot the future Dan Carter superstars of rugby while they are pimply-faced teenagers.

This is a quantum leap forward compared to something like Opta stats, which dissects every game into component parts but just posts flat statistics that do not show why these events that they have dissected happen the way they do. RugbyMetrics is taking rugby performance analysis to the next level.