My new super duper server

For the hardware geeks among you lets start with the specs.
Dell R810 platform
4x Intel Xeon E7-8837 2.67GHz 8 core processor
128GB RAM
6x300GB 10k RPM SAS drives (RAID5 using 1 as a hot spare)

This server is for running simulations.  It is not serving up webpages, or in fact for any sort of internet application.  I needed plenty of parallel power and this is what would fit the budget and requirements best.

So what did I do with it?

Before I start, one of the reasons I am writing this blog is to get input.  If while reading this you see something that looks wrong, or a mistake, or that there is a better way to do it then feel free to drop me a line.  Let me know where I have gone wrong, and feel free to suggest improvements.

This beast chews a large amount of electricity and makes a lot of noise.  It also pumps out a huge amount of heat.  My office is small so we elected to host it at a server farm near by and let them worry about all that.  When it arrive I went down there and installed it.  Spending a day in a server room is not fun if you don't have warm enough clothes, I can also recommend some fingerless gloves.

Annoyingly when I started to install the OS I noticed something wrong with the Harddrive space.  I stopped the install and rebooted in the RAID management system.  Someone hadn't seen my note about having one hot spare.  I had to rebuild the RAID which took a while because the initialization took forever.  Having sorted that little problem out I proceeded to install Ubuntu Server 11.10.

The simulation program that I have created is a written in Java and has a GUI front end for configuring the specifics of each individual simulation.  That meant some sort of graphical window manager.  I installed unity-2d ( I am pretty sure that this was a mistake).  Once installed I added GNU Screen, the Oracle version of Java, Eclipse, Firefox, Webmin and FreeNX.

Reasons for my decisions:

Ubuntu
I am most familiar with this distribution, I use it at home and for other systems at work.  I feel comfortable with it and under the time constraints felt I needed to chose something that required no time for me to learn.  Debian could have worked too as they are pretty similar and both use apt.  Given that they are similar I reasoned that there would be no difference either way, so I stuck to Debian.  My Other reasonable option was CentOS.  Not being familiar with that or with yum, I decided that I did not want to spend the whole day looking up how to install and operating system that I had never used before.  What I should have done was done some test installations on VMWare instances on my old server.

Java
The simulations are optimization and testing of a live/ real time system that runs elsewhere, ie not on this particular server.  The simulations are far more computer intensive that the live system.  The original system that I created depended on a Java API.  There were no other options at the time.  As a result the software that I have written has remained in Java.  As time has gone on we have moved away from the original API and now we primarily use QuickFIX/J.  Another Java API.  QuickFIX has other APIs but considering how far I had come with the software already written in Java it seemed like it would take too much effort on my part to rework everything into another language like C++.  I picked the Oracle JRE  as I have had some trouble in the past with the ICEDTEA version.

Unity-2d
Unity itself is a controversial window manager, Ubuntu created it as an alternative to Gnome3.  As a change from Gnome2 the previous default window manager its pretty drastic, and there are many people left feeling that the choices that they used to have with Linux are disappearing in a world of constrained window managers.  To be honest I do not agree, Ubuntu is aimed at the entire desktop market, their goal is to create an easy to use Linux desktop (or laptop) that you don't have to be a Linux guru to understand.  There are options if you want them chose something else.  I digress, I chose Unity-2d as there is only a standard graphics card in the server and I really don't need the full version.  I believe this was a mistake.  I should have opted for something even more lightweight and simple.  I am not sure whether this will cause me any problems but I could certainly have used something else.  I did however have to chose something.  As I said the application that I will be using is controlled by a GUI written in Java, I therefore must have X and a window manager.
My major annoyance with unity is that when I have many windows open I just want a regular taskbar.  The unity panel is annoying and can really slow me down.  Simple solution to this was using gnome-panel.  I now have a taskbar at the bottom which makes life much easier switching between running applications. There was another options tint2 but I prefered the more simple UI of gnome-panel.
For those of you wanting to know more about configuring unity in general have a look here.  And more on using unity's features here.
Everything is working fine now as far as the window manager is concerned.  I still have my doubts though, it is possible that unity is slowing things down a little.  Possibly there is no escape while I need a GUI I must use something and this might be just as good or bad as any.

Eclipse
I use eclipse for my Java coding, so it is useful to have it available should I need to do something directly on the server.  I don't envisage needing it very often at all but better to have it available just incase.

Webmin
I love Webmin, it makes many sysadmin tasks very easy.  Additionally I have made it available over the internet so that I can view it from my office.  I especially like if for configuring the firewall.  Iptables can be a bit fiddly and Webmin makes life much easier.

FreeNX
I am using my FreeNX a free port of NoMachine NX server for my remote desktop session.  The advantage of this compared to exporting the display is that it is easy to keep the current session alive, like most remote desktop software.  I have had trouble using VNC on Linux in the past and I had heard about FreeNX before.  Having searched around the internet I found that this was recommended.  An additional bonus is that it opens a separate session to that which is viewable directly on the server.  Instructions for Ubuntu installation can be found here.  I have not been able to set it up avaiable over the web on a browser but there are instructions here.  So far so good, configuration was easy and the NoMachine NX client for windows connects, disconnects and reconnects to existing sessions with no problems.  The interface is nice and fast over my Internet connection and so far I am happy with the results.  I can use my software in a nice full screen GUI over the Internet as if I was sat in front of the server myself.

Thats my setup.  I will follow this post up with any gripes and problems that I have with my system as I begin to put it through its paces.  Let me know if you have any questions about what I did what I installed and if you have problems with any of these bits of software let me know, maybe I can help.

JVisualVM - Java's hidden monitor and profiling tool

There are a few different ways of profiling your Java application.  One is using the -prof switch when calling running a Java app.  This will output a profile file when the app is terminated.  Another is a third party app like Yourkit.  Yourkit is great, but you have to add a something to the commandline and after 30 days full free trial is its super expensive.

Finally in recent editions of the JDK there is jvisualvm.exe.  Make sure you have the JDK and not just the JRE.  locate the install directory and find the bin directory.  There you should find the executable.  Once you open it up you can go to tools and add some additional plugins.  On the left hand side you will see the available JVMs that are connected.  The good thing about this profiler/monitor is that you don't need to add anything to the commandline like you do with Yourkit.  In your kit you have to add something to connect to that yourkit agent.  Here it just connects and you can see all the JVMs running.  Each instance of Java should run in a new JVM so you can run and view many different programs at once.  It gives you a basic view of the memory and processor usage along details about calls to GC and the number of threads and classes being used.  Additionally you can run a profiler to which will sample the application and give details of either cpu or memory usage.  This is extremely useful in finding memory leaks or in my case optimizing your code.  I do a large amount of number crunching which tends to use a lot of memory and cpu, using jvisualvm I have reduced the amount of memory used and made the whole process more efficient.  There are other features that let you view information about classes.

I came across and a problem when running jvisualvm the other day.  The executable ran and but it did not detect my running application. I ran an old version of my application and found that it was detected.  In the end I have discovered that for some strange reason the java executable has to be running with code located on the same drive.  Its very strange but seems to be the only solution I am aware of.  If anyone else has seen this problem let me know, especially if you solved it in another way.  As usual drop me line if you have questions that I haven't answered in this post.
Happy optimizing.

Excpetions - if you haven't caught on then you should try to

I am sure this has been said before but I came across an error that occurred yesterday.  I was told the code was working.  For the most part I could see that in the staging environment it was working just fine.  Sunday comes around.  My install day.  I check the staging environment.  I find that there is a null pointer exception.  The code crashed at a strange point.  With some deduction I realize where the error is coming from.  A method was called on an object that was clearly null, hence causing the exception.  The object that caused the exception was instantiated on the line before.  So no doubt there.  I the method called on the line before returns the object in question so I look at the method, which happened to be in a different class.  I see:

public SomeObject badMethod(InputObject obj){
  try {
       <code some of which that could cause an exception..
          including declaration of SomeObject 
          and return of the instantiated object>
  }
  catch (Exception e) {
     return null;
  }
}

Two big problems with this, possibly three.  The minor of all of these is that so much code is in the try block.  I am not sure if this is wrong, but at the same time it doesn't look right mainly from a maintenance point of view.  I had to delete the try and catch lines to find out which line of code require the exception handling, thanks to Eclipse that bit was simple.

I found out that the exception was actually a ParseException.

TIP #1: When using try catch blocks to catch exceptions, catch the specific exception.  Always be more explicit when you can.  Code is more understandable and you have the specific exception object available to you in the catch block.

The really big problem was how this was handled.  Theoretically I have no problem with returning null from a method.  However if you use a method that could return null that value must be handled.  Sometimes it is better if the method throws an exception instead.

TIP #2: If the method could return null you must test for it.  Otherwise you will end up with a nasty NullPointerException

In my case I ended up with an unhandled NullPointerException instead of a ParseException.  I had no further information about this as well.  This brings me to another point, at the very least you want to have the line

e.printStackTrace();

in the catch block.  At least then you can track the exception properly.  Better still use a logger, and add a little message of your own perhaps with some variable values printed so that you have an idea of what went wrong in your log or on your console. With most loggers you can add the the exception itself as an argument and the logger will handle outputting the relevant information from it.

TIP #3: Use a logger to print a sensible message from the catch block.

In this example the there was no real exception handling in the code.  The catch block just returns a null which is not handled in the calling code.  My solution in this case was to add a sensible logging line and throw the exception again.  This way the exception is passed up the stack.  The key here is that it forces the calling method to handle this exception.  In this case that works, I needed an extra try catch block, but the error will be handled better.  Now the parse exception will be passed up through the calling stack and the method will now handle this error correctly.

TIP #4: Handle the exception, its not enough just to print the stack trace or log the error.

Conclusion:
The tips I have presented here are very important when dealing with exceptions in Java.  It is too easy, especially in Eclipse which does so much for you, to leave the printStackTrace() in there and not do anything else.  But they must be handled.  Learn to use them problem and exceptions will be your friend.  Errors will be handled correctly and the code will run more smoothly and hopefully be a little more readable.

Happy coding.

Fixing errors you cannot see and how I tried {to} catch (them){}

For those of you following my twitter @ctoisrael then you will know I was bug fixing today.  My system had a bug, the bug produced a tremendous amount of output.  I can only presume that what ever went wrong produced some sort of uncaught exception.  Uncaught exceptions are not written to the log, this is not usually a huge problem.  I run my system using multiple "windows" on a gnu screen.  If you use the linux commandline a lot, particularly remotely, this is one of the most useful tool there are out there.  I will save screen for another post. I actually have it set to scrollback a large amount of lines, but today even 10000 lines was too much.  There was just too much output.  Anything output by System.out or System.err was gone for good.  The log was not revealing.  I could see that there must have been an error from a malformed log line.  It didn't look right but I couldn't tell where it came from.

Some of the most critical errors in Java are NullPointerExceptions.  They are not caught by default, and the IDEs obviously do not enforce you to surround code with try and catch blocks.  So I am guessing that somewhere outside of my scrollback was a NullPointer or ConcurrentModification exception that was not caught and though did not crash the whole system, resulted in some malformed data that produced a glitch that got me in to a big mess, but thankfully not a costly one.  The symptoms were resolved and no long term damage done.  I am still not further on with the root issue.

I cannot find the root issue it is impossible, I checked the code where it could have happened but I cannot see anything wrong.  I have been through the log, I can see the result but still not the cause.  The next time this happened I must be prepared.  In order to do this I have to catch all possible exceptions.  This it turns out is not as easy as you might think.  Ideally I need to catch all exceptions, handle them if necessary and write them to the log.

I have looked in to this in the past.  I have found two suggested solutions:

1. Surround everything with try/catch blocks, catching Exception, so that all unhandled  / uncaught exceptions will caught, and then handled.
2. Create a Class that implements Thread.UncaughtExceptionHandler and add it to the Thread

The try catch method is a simple solution but a flexible one.  One must remember that every thread needs to have a try catch block around it.  My suggestion would be to put it in the run method.

public void run(){

  try {

    <all code to run the thread>

  } catch (Exception e){

    <handle exception>

  }

}


This can be a huge pain to implement if your existing code has made different threads.  Additionally I find that too many try catch blocks can look ugly and longer than it should be.  The one advantage of this is that it allows the coder to have control over each potential exception in every place that it could occur.  Clever use or more specific exceptions that can go uncaught can allow handling in different ways.

Despite this the second method is far better.  I created an abstract class:

public abstract class Log4JUncaughtExceptionHandler implements Thread.UncaughtExceptionHandler {
  private static Logger log = Logger.getLogger("log");
}

because it is abstract and subclass must implement the method uncaughtException(Thread t, Exception ex);
I created a subclass:

public class Log4JCatchAllExceptions extends Log4JUncaughtExceptionHandler{

  @Override

  public class uncaughtException(Thread t, Exception ex){

    log.error(t.getName(),ex);

  }

}

I used them as follows.  In any class with a main method I use the class to catch all exceptions with the line

Thread.setDefaultUncaughtExceptionHandler(new Log4JCatchAllExceptions());

This only needs to go in once for the entire program.  All uncaught exceptions will be caught and handled by the Log4JCatchAllExceptions class.  Its pretty neat, no need to add lots of try catch blocks to the code and has pretty much the same effect.  What you don't have is the possibility to handle exceptions differently in different cases.  The reason is because it has been added as a static property of the Thread class, hence every new thread will have this property.  This can be overridden in the thread instances by using:

Thread t = new Thread();

t.setUncaughtExcpetionHandler(new Log4JUncaughtExceptionHandler(){

  @Override

  public class uncaughtException(Thread t, Exception ex){

    log.error(t.getName(),ex);

  }

};

t.start();

This will set the handler of the specific thread to the anonymous class that is declared here.  So any time that there needs to be a specific handling of and error it can be done like this.  Here I have just printed a log in the same way that I did with the default class.  But in my real code I restart threads so that if they die thanks to a null pointer exception then they will respawn, reducing the effects of the error while recording it at the same time.

And don't forget that there are times where going back to regular try catch blocks will fill in any holes that are not handled by these two classes.

Shutdown Hooks Eclipsed by terminate button

We are creating a new module for our system to receive data from another source. Everytime we terminate the test application in Eclipse it causes and unexpected termination at the suppliers end. I get notified and they assume something is wrong. Solution add a Shutdown hook to end the connection gracefully.

Nothing is ever that easy.

Something About J: Shutdown Hook and Eclipse

turns out that a known bug in Eclipse is that clicking on the red terminate button does not kill anything nicely.

No solutions only workarounds:

1. Run in a console. A bit annoying when working in dev, you need to constantly stop and start the app and it can slow you down a bit.

2. Relatively safe workaround if you are using threads. It works well but you have to remember to use it and not click the red button!

Thanks to both for providing the most concise and useful explanations and suggestion.

Java TimeZones (Daylight savings time not included)

Quick background.  We receive data with different timestamps.  The easiest way to work with the data is to use SimpleDateFormat and work with Date.  Date holds a reference to the date as number of milliseconds since the Epoch (Jan 1 1970).
For some reason our suppliers give us formated dates form different timezones.  No problem we just set the dateformatter to use a different timezone when parsing the date string.  The API doc states that it "also figures out daylight savings". However it does not state that you must use the long code ("Europe/London") for the timezone and not the 3 letter code ("GMT").  List of short codes. Now we are using the long code we are getting the correct dates with daylight savings taken account of.

The benefits of getting things right in OO

For anyone experienced in writing good OO code, the design is critical.  Planning ahead is essential in creating reusable objects.  I have been working on some old code that I wrote when I first started my current job and at the same time was very new to Java.  Its horrible.  Everything thrown in together.  Control, view and models all in one class; kludges, hacks and workarounds thrown in instead of proper refactoring.  Right now I am adding a new feature to the system.  I had two choices glue another piece of code to the mess or refactor the whole code base while adding in this change.  The result an extended project but more maintainable code, better code generally. Additionally many things that I am aware could be added or an issue have been taken care of, so that they should be easier to deal with later.

So why was it such a mess in the first place?

There are many reasons, but it comes down to perceived amount of time to finish a task.  If I think I can get something done with a little hack or a quick kludge then thats the way its going to get done.  I know its wrong and I know in the future it might cost me but my environment can sometimes dictate that it must be done now.  I am the CTO, I am the bridge to the technical staff. My position is to get the technology working based on the requirements of my non technical bosses.  The situation where my bosses think something simple in their heads should be easy and quick to implement happens time and time again

The non technical management want working systems or products.  My products are two separate systems.  One system is an offline system.  They run it, it does something, and produces a result.  The other is a live realtime system that runs 24/5 communicating with the outside world. That is what they want to see, it is my job to make sure they are produced, working and reliable.  Occasionally they request changes or we have routine upgrades.  This is where things get complicated.  They want it done.  They want it done soon.  They have no understanding of unit testing, refactoring and documentation.

What is the solution?
Get it done.  Many could argue getting it done means unit tests and refactoring as you go.  They would say that doing testing and keep code maintained is actually efficient in the short run as well.  This maybe the case but some times I am just too short sighted to see it like that.  I am under pressure, I have to get it done, why would I write some more code that isnt really "relevant" right now.

I am in a good position right now.  I have two working systems and though there is pressure to get this new feature into production I have more time, and I am able to fend off my bosses so long enough to refactor a large chunk of my system.

Conclusion.  My code looks a lot nicer.  I have implemented the MVC pattern nicely, well considerably better than before.  I have reusable objects that have meant creating the two different view layers considerably easier.  Small changes to the model in the future will require only making changes in one and not two spots.  Its obvious now, but I am glad to see how much I have learned since I started.
So I got it done and now I have got it right.  It would have been nice to get it right first time, but they next project will benefit from what I have learned.  I will not necessarily get it right but it wont be as wrong as the code that I have just rewritten was.


Edit: This post was written concerning refactoring the offline system.  A change was made to the live system recently that used the refactoring of the offline system.  The change was smooth and easy because it simply reused previously tested and working components from the offline system. go me!