Human centered analytics

Engineers, computer scientists and technicians alike have affinity with mathematics, numbers, formulas. They are the kind of people who prefer to know How it works than What it delivers. They are the ones who disassemble an old watch and try to fix it.
5920883-disassembled-wrist-watch-lies-on-table-Stock-Photo.jpg
We try to understand why it works. Normal people just wear it.
One of my common mistakes in my first years in the corporate world was that I faced it like it was a college examination: I tried to explain to the executives the hypothesis, the developments, the formulas, and expected then to ask about some technical stuff, like why I used a Chi-Squared distribution instead of a Normal one, as if they were academic teachers.

Surely it was not the case. In the corporate world, and in the real world in general, people want answers. Just it, answers. And they want you to explain the solution in simple, easy-to-follow and reasonable steps. They are not stupid and of course they don’t want to be treated like that. They will check the big numbers and look for contradictions. And they want to know, in human words, why is your solution the correct one. You have to tell a history. It is your task to explain what you mean in simple ways, and to defend your position using the same domain of words (not formulas).

shakespeare-words.jpg
We need to tell histories with words, like Shakespeare. Not formulas.

 


The optimal solution for the wrong problem
In the same line of thinking, I spent a lot of years in my quest to always find the optimal solution with the best up-to-date method for analytical problems.
Wrong. Completely wrong.
An optimal solution for the wrong problem solves nothing.
FindX.png
People in the real world don’t need the optimal solution. They need a solution that works effectively. And the only way to find an effective solution is in the Questions, not in the method of resolution of the model. The problem is in the formulation of the problem.
If you want to solve a real world problem, it will not come with a formulation “Use integer programming to solve this set of equations”. It will come in diffuse information pieces, with dozens of people telling different points of will, with spurious sparse databases. And you will have to formulate the questions.
The model can be as simple as a single linear equation, as long as it provides the effective solution.

Challenges of Analytics
I see a lot of excitement and buzz words: Big Data, Artificial intelligence, Watson, R-Phyton or whatever.
bigdata.jpg

These are only tools. They don’t solve problems alone. They will not solve all the problems of the world.

In a couple of years, these buzz words will fade, some will be forgotten, and new buzz names will appear, promising the same things.

 

For me, better than trying to provide the magical solution, is to make the basics. Simple and effective solutions. And these solutions must:

  • Provide information, guidance, clarity
  • Support decision
  • Empower analysis
That is, provide ways to make questions.
Arnaldo Gunzi
Feb 2016
“The release of atom power has changed everything except our way of thinking. The solution to this problem lies in the heart of mankind. If only I had known, I would have become a watchmaker”
Albert Einstein

Running time of variables – AIMMS

How to find problems in calculation of variables?

AIMMS is extremely powerful, but sometimes it can be very tricky.

AIMMS uses sparse matrix multiplication methods. It makes a lot of sense, since the matrices in linear or integer programming are usually giant, but sparse. But sparse arithmetics is difficult to understand and there are sometimes troubles caused by this.


Real life problem example

I had a model, whose behavior was very very strange.

It was taking a lot of time, about 10 minutes, in the preprocessing stage. After the long preprocessing, the actual running of the solver was fast, one minute in average.

I checked the formulation dozen of times, but there was no evident mistake. 


AIMMS profiler

In order to investigate where the problem was, I used the AIMMS profiler tool.

The profiler can be found in Tools -> Diagnostic Tools -> Profiler.

Profiler.png
Profiler menu
After one more running of the model, I could find the results in Profiler -> Results Overview.

 

ProfilerResultsOverview.png
Overview of the profiler results (just illustrative)
In this case, it showed clearly that one specific variable – “vNroCac”  – was taking too long to be processed. It had clearly a running time several times greater than other variables. I’ve found the offender!

This specific variable had a structure like this: it calculated a lot of things, using other variables as inputs, and some of these variables also used other variables and parameters. I guess the sparse multiplication was taking too long to be processed, due to these several subcalculations…

I simply eliminated this variable and reformulated the model.

Now,  the preprocessing time was fine!


 

Conclusion

The profiler is a great tool for investigating running time calculation problems in Aimms.

 

Arnaldo Gunzi

Feb. 2016

 

 

How to use all cores of the computer in Aimms

Introduction
Running combinatorial optimization problems in Aimms is, most of times, a hard task (if the problem is easy, you don’t need to pay a lot of money in the Aimms license – use COIN-OR’s CBC, Excel or something else).

But, when you run Aimms in default mode, it uses only one core of the computer. It sees to be a waste of resources, since your company probably bought a mighty computer to run Aimms.

One frequent question is: how to use all cores of the computer?

 


 

Barrier
The way to answer the question is simple. Just set the LP method to “Barrier”, and set the parallel configurations as following instructions:

LP_Method.png
In Settings -> Project options -> (your solver) -> General -> LP Method := change it to barrier
Other configuration:

LP_Method_config.png

In Settings -> Project options -> (your solver) -> Parallel -> Global thread limit := set with the number of cores of your computer
In Settings -> Project options -> (your solver) -> Parallel -> Parallel mode := set with “opportunistic”

And voilá, now the computer is using all cores of the computer.

AllCores.png
In this example, the CPU is in 100% – using the 4 cores of the computer

A word of caution here: the solver must allow this barrier parallel configuration, otherwise there’s no deal.

How to know which solver  is running? Just check it in Settings -> Solver configuration

 

Solver.png


 

What the hell is barrier?

In default mode, Aimms use the famous Simplex method (or similar variant). The Simplex begin with a feasible solution, then improves it by searching the corners of the Simplex polyhedron.

Simplex.PNG
Illustration of the feasible region of a problem
The Barrier method is an Interior Point method. It begins in a point, then improves it following the direction of most gain. Since there are lots of ways to follow this interior path, it makes sense to do this in parallel.

barrier.PNG
Barrier method

 

This textbook of Robert Vanderbei is the source of these two images, and also an excellent introduction to the barrier method.

 


 

Does this parallelization  really works?

It can not be said surely for every case that it will work, since it depends on the model.

I can say that in every model I tested, for my work, it improved slightly the solution. The objective function was 2% – 5% better using the parallel barrier method (to the same limited solution time).

 


 

Conclusion

 

The use of barrier parallel method can improve the solution, but not much. Anyway, it is an interesting feature to be explored.

 

Arnaldo Gunzi

Feb 2016