Posts for the month of October 2012

Real-time!

Some pearls of convention wisdom says that we cannot make things less crappy by adding more crap, or, say, solve a dept problem adding more dept.

On the contrary, we can read, in market speak, that piling up more Java could solve its problems, and even provide real-time querying. Real-time! In Java!

Lets take a look for this mirracle:

https://github.com/metamx/druid/tree/master/realtime/src/main/java/com/metamx/druid/realtime

package com.metamx.druid.realtime;

import com.metamx.common.lifecycle.Lifecycle;
import com.metamx.common.logger.Logger;
import com.metamx.druid.log.LogLevelAdjuster;

/**
 */
public class RealtimeMain
{
  private static final Logger log = new Logger(RealtimeMain.class);

  public static void main(String[] args) throws Exception
  {
    LogLevelAdjuster.register();

    Lifecycle lifecycle = new Lifecycle();

    lifecycle.addManagedInstance(
        RealtimeNode.builder().build()
    );

    try {
      lifecycle.start();
    }
    catch (Throwable t) {
      log.info(t, "Throwable caught at startup, committing seppuku");
      System.exit(2);
    }

    lifecycle.join();
  }
}
  • John, do you see any real-time?
  • Nope.
  • Oh, I found real-time! Look!
package com.metamx.druid.realtime;

import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.metamx.common.ISE;
import com.metamx.common.concurrent.ScheduledExecutorFactory;
import com.metamx.common.concurrent.ScheduledExecutors;
import com.metamx.common.config.Config;
import com.metamx.common.lifecycle.Lifecycle;
import com.metamx.common.lifecycle.LifecycleStart;
import com.metamx.common.lifecycle.LifecycleStop;
import com.metamx.common.logger.Logger;
import com.metamx.druid.client.ClientConfig;
import com.metamx.druid.client.ClientInventoryManager;
import com.metamx.druid.client.MutableServerView;
import com.metamx.druid.client.OnlyNewSegmentWatcherServerView;
import com.metamx.druid.client.ServerView;
import com.metamx.druid.collect.StupidPool;
import com.metamx.druid.db.DbConnector;
import com.metamx.druid.db.DbConnectorConfig;
import com.metamx.druid.http.QueryServlet;
import com.metamx.druid.http.RequestLogger;
import com.metamx.druid.http.StatusServlet;
import com.metamx.druid.initialization.Initialization;
import com.metamx.druid.initialization.ServerConfig;
import com.metamx.druid.initialization.ServerInit;
import com.metamx.druid.initialization.ZkClientConfig;
import com.metamx.druid.jackson.DefaultObjectMapper;
import com.metamx.druid.query.DefaultQueryRunnerFactoryConglomerate;
import com.metamx.druid.query.QueryRunnerFactoryConglomerate;
import com.metamx.druid.utils.PropUtils;
import com.metamx.emitter.EmittingLogger;
import com.metamx.emitter.core.Emitters;
import com.metamx.emitter.service.ServiceEmitter;
import com.metamx.http.client.HttpClient;
import com.metamx.http.client.HttpClientConfig;
import com.metamx.http.client.HttpClientInit;
import com.metamx.metrics.JvmMonitor;
import com.metamx.metrics.Monitor;
import com.metamx.metrics.MonitorScheduler;
import com.metamx.metrics.MonitorSchedulerConfig;
import com.metamx.metrics.SysMonitor;
import com.metamx.phonebook.PhoneBook;
import org.I0Itec.zkclient.ZkClient;
import org.codehaus.jackson.map.BeanProperty;
import org.codehaus.jackson.map.DeserializationContext;
import org.codehaus.jackson.map.InjectableValues;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.jsontype.NamedType;
import org.codehaus.jackson.smile.SmileFactory;
import org.codehaus.jackson.type.TypeReference;
import org.jets3t.service.S3ServiceException;
import org.jets3t.service.impl.rest.httpclient.RestS3Service;
import org.jets3t.service.security.AWSCredentials;
import org.mortbay.jetty.Server;
import org.mortbay.jetty.servlet.Context;
import org.mortbay.jetty.servlet.ServletHolder;
import org.skife.config.ConfigurationObjectFactory;

import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ScheduledExecutorService;
...

Yeah.. That's it..

Enterprise software Next.

Enterprise software, unlike hot internet startups in Valley (overpriced rapid Rails coding) is not sexy, but there is still tons of money here.

There is a deep crisis in in-house development because of the Java fiasco. Instead of getting results on schedule and within budget all they got instead is piles and piles of unmaintainable Java crap, which doesn't work as expected, late for years and still doesn't meet the requirements, consumes all available resources and crashes unexpectedly every week. What is even more important - several times overspending without even coming close to meet the specs.

This is reality of today's Enterprise Development with Java. No wonder that budgets were cut out and enthusiasm vanished.

But don't you see, no one in Valley are using Java or PHP for their startups. A few, however, believes that if they will pile up another layer on top of JVM then it will magically become more efficient, stable, predictable and less resource consuming. Call it Scala or Clojure - it doesn't matter. They want to believe.

Btw, have you ever ask yourself, why there is NDK in Andoroid? Are you aware how many open-source shared libraries, written in C and C++ included in Android installation? Have you ever considered how Google Chrome was implemented.

The answer is - Chrome reuses tens of open-source libraries. No one usually rewrites complex specific code, such as cryptography functions of hardware-optimized math functions, they just dlload and call the code. This is why we have NDK and JNI in Android.

Wait, there is another story. Openssl is now included in NodeJS source tree to be compiled and linked with Node's binary. Guess what? It is because calling openssl's functions from inside V8 is very inefficient - it is slow has huge overhead, so it is reasonable to make a local wrapper and use it.

The bigger idea? VMs are bad when you need to interact with underlying OS. Re-using specially optimized functions provided by OS or open source libraries is a much better idea.

There is a very short list of open source packages engineer reuse very often - openssl, pcre, expat, sqlite, berkeley db, postgresql, gmp, curl, etc.

There are binding for almost every scripting language to make dynamic calls to these libraries. Why? Because re-using best quality, community-tested code is much more efficient.

Hey, but what about Enterprise Software Development? Well, look at Android, Google Chrome or even MacOSX. When a company needs some software product it does not going to buy a bloatware from, say, SAP, IBM or Oracle. It is legacy. They bootstrap the product themselves, reusing open-source protocols and software packages.

People are building internet startups very the same way, the very same 80/20 rule is here - they reuse ideas, they reuse protocols, they reuse code - 80% and they write their own code - 20%.

Those who are smart enough even make their own code reusable for the open source community, so that other people may test it, fix and even improve it.

Why open source model works and people do contribute, spent their time writing or fixing publicly available code? The answer is quite simple - because they find it useful for themselves. They themselves reusing the code. So, the secret is - make something useful for other people.

Now, if by some divine miracle we can eliminate all those bone-headed managers striving to keep their positions, there is a very clear way to make business software - the very same way the internet community do. Take the best ideas, take the best protocols, take the best tools and packages. This is the simple part.

The difficult part, which bone-headed crowd is unable to grasp - is about structuring your effort and code to make products that other people will reuse and enjoy. They will do it only when several criteria were meet:

  • it is doing an important thing
  • it doing it efficiently
  • it reuses familiar packages
  • it uses appropriate tools

This is how nginx and redis and postfix and hundreds of other software engineering masterpieces were made. This is how Google Chrome or Android were made.

Now look. Chrome is No.1 browser. nginx is No.1 web server. Android is No.1 smartphone platform. Is there any SAP, Oracle or IBM? Nope. All those projects were bootstrapped and based on open-source.

Are these idea new or original? Nope. The only problem we have is the ignorant and stagnant minds of the Enterprise folks. Their main concern is not innovations or efficiency which could be attained using a new software product, their main concern is their position and their money, their small very limited and very fixed world of ass-covering and responsibility avoiding.

So, putting them aside, there is no obstacles for making the next level of the enterprise, or B2B or whatever software, the very same way all those internet services we use every day, were made.

Fix for Ubuntu 12.10

There is a quick fix for Ubuntu 12.10. (for experts)

switch to a text console (Ctrl+Alt+F1) and log in, then do this

$ sudo apt-get remove unity* oneconf* lightdm nautilus
Y
$ sudo apt-get install gdm gnome-desktop-data gnome-panel

follow the dependencies.

Packers vs. Mappers

There was a thought provocative essay, called "Programmers Stone" written in 90s. It was rather long, but its opening was outstanding. Author discussed two different mind-sets - packers and mapper. No we can call them coders and programmers.)

The wast majority belongs to the first category, so they set rules and influence fashion. That is why almost everything eventually becomes Java - explicit typing, redundancy, forced description of every single little detail - all the packer's mindset likes and welcomes.

Even a guidelines for Common Lisp are heavily influenced by packers (corporations are hiring them to code from 9 to 5).

The all small marvels which together create the pure artistic sense of using Lisp packers want to throw away and replace with strict typing, ugly constructs, fixed rules, concrete recipes.

Don't use dynamic lists, they say, use fixed structures. Don't do recursion, use loops. Don't use nil, it is tricky. Do not return multiple values, it is confusing, and so on.. Everything with is unique and liberating, gives you more freedom of expression, leads to more compact, more natural idioms considered harmful.

They want it back to the realms of strictly controlled, imperative packing world, where they feel safe and confident, performing their restricted, repetitive tasks according to fixed rules.

On the other hand, going into unknown, exploring, observing, figuring-out, describing and prototyping on the go, this very moment - the attitude that is alien to packers and hated by them.

The very essence of an art is an intense awareness, attempting to catch this very moment and act in a harmony with it on the spot. It is about flowing, being flexible and spontaneous. It has nothing to do with restricted, fixed, repetitive behavior, which characterizing stupidity.

To learn the art of programming you should learn the artistic Lisp - its Scheme dialect, with remarkable general ideas, wise conventions, and very few, carefully selected special forms.

After you will *realize* the beauty of one general application rule *together* with prefix notation *and* common underlying list structure, after you *internalize* that having *less* pre-defined special forms is much better than having lots of confusing special cases, that you could extend this core language with any function you wish. After you will *realize* that code is data, represented as lists, and together with the list-aware reader it gives you almost algebraic transformations for free, that the-empty-list is a nice thing and recursion *is* much more natural way to think, then, you will look at world of packers very differently.

Scheme language is a natural programming language for an artists, people with taste and drives, with the passion for exploration and going into unknown, open-ended problems, for people with different mindset.

For safe packing with primitive, fixed hierarchies there is imperative Java.

The Google Effect.

Everyone was told in a elementary school that being a copycat is not not good and will cause lots of problems in later life.

Why? Because by copying a ready solution, produced by someone else, we gain nothing, even developing a wrong habit of thinking, as if all we need is an ability to find a ready solution and copy-paste it.

Now Google provides us a really easy way to find a solution to copy-paste, and this is what crowds do. Searching is the new normal.

Instead of trying to identify underlying ideas, concepts and principles, to understand how it works, and why it is so, people are searching for a ready solution, produced by someone else, that matches their assumptions and fulfill their expectations.

Snap-judgments on quickly matched features against expectations without any attempt to understand, leave alone analyze, of what is inside is the cause of later problems.

Professionally advertised features, a promise of easy live and tons of ready solutions is the major selling strategy. This is why such crap like php, mysql (with myisam and table-level locking) mongodb, nodejs, clojure or JVM gained such popularity. They promised an easy live without understanding of what is inside.

The bandwagon and the peer effects, of course, are in a play. Everyone are using php and mysql (now rails and nodejs), so it must be good. Everyone are using Java and Javascript, so they must be good.

But we know, that exactly the opposite is the case. Everyone are eating a cheap, processed fast-food, so, it must be good. Everyone are drinking booze, so, it is OK. Everyone are getting credits having no means to pay back - so we shall do.

There are different kinds of solutions, which was not a result of marketing efforts to sell crap to suckers, but an evolved results of a bottom-up processes of producing a solution by themselves, for themselves. There are Linux kernel, FreeBSD, postgresql, MIT Scheme and SBCL, nginx and redis, Go and python3, even Rails at the time of 1.x versions.

As long as people will continue to search for a ready solution instead of spending time to master the concepts and underlying principles, and then select the appropriate tools, we will have piles of lowest quality crap for sale, and even open-source solutions will be bloated and ill-designed, put together in a hurry without any understanding of underlying abstraction layers, guided only by the strategy how to sell.

Lisp-like syntax on top of JVM - Wow! Side-effects, underlying general List structure? Tail-call optimized recursion? Who cares?! Lets marry V8 with event-driven development concept - wow! Server-side Javascript. Underlying OS? Event notification model? Calling conventions? Who cares?!

This tendency is not limited to Information Technology field, it is in finance and in investing, in every bubble, produced by a wave of mass hysteria.

We could call it the bubble of copy-pasters, the bubble of mediocre, the age of incompetent. The Google effect.

Just watch this..

What programming really is? Watch this. http://youtu.be/efhh0Cf6sT8

Make sure everyone can understand what you have done". Brilliant.