Engineers, computer scientists and technicians alike have affinity with mathematics, numbers, formulas. They are the kind of people who prefer to know How it works than What it delivers. They are the ones who disassemble an old watch and try to fix it.
One of my common mistakes in my first years in the corporate world was that I faced it like it was a college examination: I tried to explain to the executives the hypothesis, the developments, the formulas, and expected then to ask about some technical stuff, like why I used a Chi-Squared distribution instead of a Normal one, as if they were academic teachers.
Surely it was not the case. In the corporate world, and in the real world in general, people want answers. Just it, answers. And they want you to explain the solution in simple, easy-to-follow and reasonable steps. They are not stupid and of course they don’t want to be treated like that. They will check the big numbers and look for contradictions. And they want to know, in human words, why is your solution the correct one. You have to tell a history. It is your task to explain what you mean in simple ways, and to defend your position using the same domain of words (not formulas).
The optimal solution for the wrong problem
In the same line of thinking, I spent a lot of years in my quest to always find the optimal solution with the best up-to-date method for analytical problems.
Wrong. Completely wrong.
An optimal solution for the wrong problem solves nothing.
People in the real world don’t need the optimal solution. They need a solution that works effectively. And the only way to find an effective solution is in the Questions, not in the method of resolution of the model. The problem is in the formulation of the problem.
If you want to solve a real world problem, it will not come with a formulation “Use integer programming to solve this set of equations”. It will come in diffuse information pieces, with dozens of people telling different points of will, with spurious sparse databases. And you will have to formulate the questions.
The model can be as simple as a single linear equation, as long as it provides the effective solution.
Challenges of Analytics
I see a lot of excitement and buzz words: Big Data, Artificial intelligence, Watson, R-Phyton or whatever.
These are only tools. They don’t solve problems alone. They will not solve all the problems of the world.
In a couple of years, these buzz words will fade, some will be forgotten, and new buzz names will appear, promising the same things.
For me, better than trying to provide the magical solution, is to make the basics. Simple and effective solutions. And these solutions must:
Provide information, guidance, clarity
That is, provide ways to make questions.
“The release of atom power has changed everything except our way of thinking. The solution to this problem lies in the heart of mankind. If only I had known, I would have become a watchmaker”
AIMMS is extremely powerful, but sometimes it can be very tricky.
AIMMS uses sparse matrix multiplication methods. It makes a lot of sense, since the matrices in linear or integer programming are usually giant, but sparse. But sparse arithmetics is difficult to understand and there are sometimes troubles caused by this.
Real life problem example
I had a model, whose behavior was very very strange.
It was taking a lot of time, about 10 minutes, in the preprocessing stage. After the long preprocessing, the actual running of the solver was fast, one minute in average.
I checked the formulation dozen of times, but there was no evident mistake.
In order to investigate where the problem was, I used the AIMMS profiler tool.
The profiler can be found in Tools -> Diagnostic Tools -> Profiler.
After one more running of the model, I could find the results in Profiler -> Results Overview.
In this case, it showed clearly that one specific variable – “vNroCac” – was taking too long to be processed. It had clearly a running time several times greater than other variables. I’ve found the offender!
This specific variable had a structure like this: it calculated a lot of things, using other variables as inputs, and some of these variables also used other variables and parameters. I guess the sparse multiplication was taking too long to be processed, due to these several subcalculations…
I simply eliminated this variable and reformulated the model.
Now, the preprocessing time was fine!
The profiler is a great tool for investigating running time calculation problems in Aimms.
Running combinatorial optimization problems in Aimms is, most of times, a hard task (if the problem is easy, you don’t need to pay a lot of money in the Aimms license – use COIN-OR’s CBC, Excel or something else).
But, when you run Aimms in default mode, it uses only one core of the computer. It sees to be a waste of resources, since your company probably bought a mighty computer to run Aimms.
One frequent question is: how to use all cores of the computer?
The way to answer the question is simple. Just set the LP method to “Barrier”, and set the parallel configurations as following instructions:
And voilá, now the computer is using all cores of the computer.
A word of caution here: the solver must allow this barrier parallel configuration, otherwise there’s no deal.
How to know which solver is running? Just check it in Settings -> Solver configuration
What the hell is barrier?
In default mode, Aimms use the famous Simplex method (or similar variant). The Simplex begin with a feasible solution, then improves it by searching the corners of the Simplex polyhedron.
The Barrier method is an Interior Point method. It begins in a point, then improves it following the direction of most gain. Since there are lots of ways to follow this interior path, it makes sense to do this in parallel.
This textbook of Robert Vanderbei is the source of these two images, and also an excellent introduction to the barrier method.
Does this parallelization really works?
It can not be said surely for every case that it will work, since it depends on the model.
I can say that in every model I tested, for my work, it improved slightly the solution. The objective function was 2% – 5% better using the parallel barrier method (to the same limited solution time).
The use of barrier parallel method can improve the solution, but not much. Anyway, it is an interesting feature to be explored.
Analytics and Combinatorial Optimization are vast and wonderful areas, and also very difficult ones.
As a consultant on these areas, I’ve worked a lot with several tools and techniques. But the applications were always in very big companies, that can afford to pay a consulting company and can buy expensive software licenses.
I believe Small Businesses also have problems that can be solved with powerful analytical tools. But these tools have to be well designed, free of charge and effective.
Mission of the blog:
– Empowerment of the user, with practical, easy to use and powerful analytical tools for small businesses.
– Practical applications, from literature and from the author’s experience.
There will have a range of softwares used here:
Microsoft Excel, because it’s a ubiquitous and powerful software.
AIMMS: the powerful optimization platform for very hard problems.
CBC: the free solver from Coin-operated, excellent too.
Arnaldo Satoru Gunzi
The title of this blog is inspired in the beautiful poem “To See a World…”, by William Blake.
“To See a World…”
To see a World in a Grain of Sand And a Heaven in a Wild Flower, Hold Infinity in the palm of your hand And Eternity in an hour.