Thursday, June 30, 2011

Problem with reporting of disk space

I ran out of room on a drive.  It was reporting that there was 0 space left on the drive.  I delete some files and then after running df -h it was still reporting 0 space left on drive.  I found this
http://serverfault.com/questions/158524/disk-space-not-freed-on-ext3-raid1-after-deleting
which explains quite nicely the problem.

In short the solution was to type

sudo tune2fs -m 0 /dev/<partitionname>

and all was solved.

Sunday, June 26, 2011

Start up pitches

http://mashable.com/2011/06/24/startup-pitch-presentation/

Whether you are just getting started or about to raise capital this is a really good breakdown of what to present to investors.  It is good to be thinking about this from the beginning.  As you develop you company and get ready to present it might change, but by the time you are presenting have it down pat.  And don't forget to have an elevator pitch too.  Sometimes you might only get 30 seconds with someone, and you better have a good answer for the inevitable "and what do you do?" question.  Get it right and make it count.

happy pitching

Wednesday, June 22, 2011

Automating password authenticated commands

On my linux server I have an IPSEC VPN set up to an external connection.  This was a stipulation from the counterparty to allow us to send secure data, from a verified user.  I set that up before I started this blog, so at some point I will do a retrospective blog on that.  For now I want to talk about something a little more general, I am just framing the problem.

I have a FIX connection running over this VPN, using the QuickFix/J API.  For some reason that I cannot quite figure out I lose the session and it will not automatically reconnect.  This is usually solved be resetting the VPN. I wrote a short script to take down and bring back up the connection, called restartVPN:

#! /bin/bash
sudo ipsec auto --verbose --down Test
sudo ipsec auto --verbose --up Test

This works fine.  As you can see I use sudo to give me admin rights to do this.  This is fine for a manual operation.  However I have had this problem occur and not been notified quickly enough to do this manually without any consequences therefore I wish to detect when my session has been down for too long and automatically reset it.

This should be no problem I make a call from my code that is watching the session status to the bash script above and hey presto.  However this will not work, sudo requires password authentication for non root users be default.  A call from another system will get stuck waiting for the password and as its supposed to automated there isn't much I can do about that.  

WARNING: Messing around with the sudoers file is dangerous and should be done very carefully and allowing only the least possible permissions to make your scripts run automatically.

The solution is allowing sudo to run ipsec without requiring a password to get the admin rights.  Once done ipsec must still be run with sudo, so there is a small layer of security but the password layer is gone, so if you do type sudo ipsec make sure you know what you are doing.
Before you do this it is worth checking out the sudo and visudo man pages.
Bear in mind that my user is called ctoisrael and the command I am running is ipsec.  You will need to replace this with your own specifics.
First run visudo to edit the visudo file.

~/sudo visudo

It will open in your default editor.  Make sure you are familiar with editing in which ever you have chosen.
add a line like this at the bottom:

ctoisrael ALL = NOPASSWD: /usr/sbin/ipsec

the first ALL means for all servers, in this case it doesn't matter because I am only setting this on this server and not copying it to other servers.  If I wanted to be specific ie that it can only be done on one server I would replace this first ALL with the name of the server.  The rest should be self explanatory.  I have specified that no password is needed when running the ipsec command, but only for this user.
Save the file.  It will be default save sudoers.tmp.  The visudo program will check to make sure it can be saved at the real sudoers file and then save it.

Now when I run restartVPN it does not prompt me for the password which means it can be run from a script that will automatically reset my VPN when it goes down.

I found a pretty good guide to setting up IPSEC VPNs if you are interested, here.

Feel free to write something in the comments if you have any questions about this or anything else I blog about, happy to help.

Tuesday, June 14, 2011

Fixing errors you cannot see and how I tried {to} catch (them){}

For those of you following my twitter @ctoisrael then you will know I was bug fixing today.  My system had a bug, the bug produced a tremendous amount of output.  I can only presume that what ever went wrong produced some sort of uncaught exception.  Uncaught exceptions are not written to the log, this is not usually a huge problem.  I run my system using multiple "windows" on a gnu screen.  If you use the linux commandline a lot, particularly remotely, this is one of the most useful tool there are out there.  I will save screen for another post. I actually have it set to scrollback a large amount of lines, but today even 10000 lines was too much.  There was just too much output.  Anything output by System.out or System.err was gone for good.  The log was not revealing.  I could see that there must have been an error from a malformed log line.  It didn't look right but I couldn't tell where it came from.

Some of the most critical errors in Java are NullPointerExceptions.  They are not caught by default, and the IDEs obviously do not enforce you to surround code with try and catch blocks.  So I am guessing that somewhere outside of my scrollback was a NullPointer or ConcurrentModification exception that was not caught and though did not crash the whole system, resulted in some malformed data that produced a glitch that got me in to a big mess, but thankfully not a costly one.  The symptoms were resolved and no long term damage done.  I am still not further on with the root issue.

I cannot find the root issue it is impossible, I checked the code where it could have happened but I cannot see anything wrong.  I have been through the log, I can see the result but still not the cause.  The next time this happened I must be prepared.  In order to do this I have to catch all possible exceptions.  This it turns out is not as easy as you might think.  Ideally I need to catch all exceptions, handle them if necessary and write them to the log.

I have looked in to this in the past.  I have found two suggested solutions:

  1. Surround everything with try/catch blocks, catching Exception, so that all unhandled  / uncaught exceptions will caught, and then handled.
  2. Create a Class that implements Thread.UncaughtExceptionHandler and add it to the Thread
The try catch method is a simple solution but a flexible one.  One must remember that every thread needs to have a try catch block around it.  My suggestion would be to put it in the run method.

public void run(){
  try {
    <all code to run the thread>
  } catch (Exception e){
    <handle exception>
  }
}


This can be a huge pain to implement if your existing code has made different threads.  Additionally I find that too many try catch blocks can look ugly and longer than it should be.  The one advantage of this is that it allows the coder to have control over each potential exception in every place that it could occur.  Clever use or more specific exceptions that can go uncaught can allow handling in different ways.

Despite this the second method is far better.  I created an abstract class:

public abstract class Log4JUncaughtExceptionHandler implements Thread.UncaughtExceptionHandler {
  private static Logger log = Logger.getLogger("log");
}

because it is abstract and subclass must implement the method uncaughtException(Thread t, Exception ex);
I created a subclass:

public class Log4JCatchAllExceptions extends Log4JUncaughtExceptionHandler{


  @Override
  public class uncaughtException(Thread t, Exception ex){
    log.error(t.getName(),ex);
  }
}

I used them as follows.  In any class with a main method I use the class to catch all exceptions with the line

Thread.setDefaultUncaughtExceptionHandler(new Log4JCatchAllExceptions());

This only needs to go in once for the entire program.  All uncaught exceptions will be caught and handled by the Log4JCatchAllExceptions class.  Its pretty neat, no need to add lots of try catch blocks to the code and has pretty much the same effect.  What you don't have is the possibility to handle exceptions differently in different cases.  The reason is because it has been added as a static property of the Thread class, hence every new thread will have this property.  This can be overridden in the thread instances by using:

Thread t = new Thread();
t.setUncaughtExcpetionHandler(new Log4JUncaughtExceptionHandler(){
  @Override
  public class uncaughtException(Thread t, Exception ex){
    log.error(t.getName(),ex);
  }
};
t.start();

This will set the handler of the specific thread to the anonymous class that is declared here.  So any time that there needs to be a specific handling of and error it can be done like this.  Here I have just printed a log in the same way that I did with the default class.  But in my real code I restart threads so that if they die thanks to a null pointer exception then they will respawn, reducing the effects of the error while recording it at the same time.
And don't forget that there are times where going back to regular try catch blocks will fill in any holes that are not handled by these two classes.

Saturday, June 11, 2011

Programming tests: is there a shortcut

For those of you not following my tweets, I tweeted the other day about http://www.codeeval.com/.  This is a great service that basically allows you to create job applications online (no big deal) with the bar to sending it in as passing the programming problem that it presents (very big deal).  This could be a big help in theory.  You can see how organised they are about coding, for a good enough problem it will show their problem solving skills and their thought processes.


I have a problem with this though, there seems to be no way to block the applicant looking the answers up.  There are timed problems, but copy paste is pretty quick once you have found the algorithm on the web.  In addition it has plagiarism detector so you can see if candidates are sharing their answers.  This could actually be used by submitting answers that you find on the web.  Once a candidate uses the same answers from searching the web the plagiarism detector will flag them.  


Considering all of this I would still want to do a reasonable live test as well.  My main reason being efficiency of programmers is quite important to me.  I cannot stand sitting with someone who is repetitively doing the same slow process over and over again.  I saw recently a description of programmers as being "Inherently lazy with just the right amount of motivation."  This description presents us with someone that is too lazy to keep doing something that takes 2 minutes over and over again, but will be motivated to spend an hour finding or writing a solution that will make this process automatic.  I love this, its true about me, and about many other coders.  I remember hearing a quote by Richard Stallman, Founder of the GNU project,  who said that he would have never got it done if he wasn't lazy.  I cannot find any reference for that right now so don't take me at my word.  But I am pretty sure that I read it in Rebel Code.  I digress...


I want to see not just efficient solutions to problems but efficient working method to get there.  There is nothing more painful than watching someone click on the file menu and select copy and paste, even using right click in an editor, everyone should know ctrl^c ctrl^v.  Thats ok, I doubt you would come across many programmers that don't, but there are more examples of things that people can do to work quicker.  Selecting, single words and lines are very easy to select by double or triple clickling, no need to be exact about highlighting all the letters.  The less a mouse is used the better.  Users of VI and EMACS will be proficient at this and will probably be mouse free in other environments too.  Additionally X windows users will know that if you highlight something it is copied to the clipboard and can be pasted using a single middle click.  Saving too, most of the time its just ctrl^s but not doing it will mean you get prompted to save a few seconds are required of a mouse click, these things do add up.  It sounds like I am talking about some pretty irrelevant or small things but trust me when you watch some one who doesn't use these things its slow.  


I am not the best at these things, but I definitely notice when they are not being done.  When I spent more time in EMACS and VI I taped a list of common commands on the wall behind my monitor.  The more I used them the less I had to refer to the list and the faster I got at using these wonderful tools.  


One weird thing that I have is about searching through code.  I get it if you don't know the code, but there are many shortcuts especially in the IDEs to make it quicker.  I much prefer ctrl^j to ctrl^f in Eclipse, but what I prefer the most is being familiar enough with the code to know where to go.  Don't search just go straight there.  Someone who can type quick enough and knows the shortcuts to search should be able to beat me in a large file.  However its the familiarity with the code that I am looking for.  If you are my programmer and you don't know the code well enough how can you solve bugs or make feature changes.  I am not talking here about new code, I am talking about code the programmer has written themselves.  Searching is fine, I would never say to someone that they cannot do it, but I want to see some indication that the code they have spent  a few weeks writing is somehow bouncing around their heads.

Wednesday, June 8, 2011

I love whiteboards

... but not as much as these guys. http://bit.ly/jAxznB

I love this idea, I think its great.  I have had whiteboards from my undergraduate degree and my post grad.  My supervisor meetings revolved around us working through ideas on the white board together.  If it was important and I didn't have time to write it down I would bring my camera and take a picture of it to refer to later.

I cannot recommend them enough.  I have one white board in my office, its full there is almost no space left for temporary things.  I got it when I was lost in my notepads; ideas, todo lists, diagrams, passwords and notes.  Things were getting difficult to keep track off.  Now I have my whiteboard, it has not just everything I need to do, but everything that needs done.  My task, my staff's tasks, even my boss's tasks.  Its colour coded too, to indicate priority, that way I can add things to lists without worrying about the other.  Larger tasks are broken down in separate sections in to a group of smaller tasks.  Where possible time estimates are written next to tasks to give an idea of the scale of what needs to be done.

This is great when things get done they are erased, when things do not get done they stay, much better than being lost in the depths of early pages of a notepad.  My notepads are for doodles and daily tasks, or notes from the evening to remind me to do something the following day.  Once the page is turned in the notepad I need not worry about the previous pages, all the important stuff is on the whiteboard.

Its great for planning and working in a team.  Particularly if anybody can easily see who is assigned what.  My bosses know that I have plenty to do and can see by the turn over of tasks that I am at least getting something accomplished every day.

The problem I have is that I only have one right now.  Its almost full of tasks and plans for future projects.  There is little room for diagrams, pseudo code and doodles.  I find that I spend some time every day explaining something to someone, this is best done with a space on the whiteboard and a dry marker in hand.

I must add a task to the whiteboard:
Get more whiteboards   CTO 2hours

Monday, June 6, 2011

More on Hiring: or how to avoid moron hiring

I have written about hiring before and I felt there was more I could say about it.  I do not have a huge amount of experience of searching for jobs elsewhere but here in Israel I feel the whole process is lacking.  I am originally from the UK and though I spent most of my time since high school studying at university I did apply for some jobs.  One experience jumps to mind.  I had a phone interview with a big consulting firm.  At the time I was not so confident with my programming ability.  I was young and naive and I guess what ever I said came off as, "I don't like programming, I don't want to do it."  I was told later by HR from this company that I did not progress because I said that I did not like programming.  (things have changed considerably for me).  There was no attitude of 'lets interview him some more and see how it goes.'  Later interviews, here in Israel, with other big companies have resulted in my not progressing for good reason, failing to pass whatever puzzles, tests and problems that were set out for me (Google, Bloomberg, and Brevin Howard), and being offered a job by Intel because I did pass these tests.  These are big companies, with HR that has been developed over many years.  They are efficient and are able to take on the best candidates.

So why here do we have this attitude of wasting interviewers' and candidates' time by interviewing them when they are inappropriate.  Additionally at my previous company, for all the positives that they gained from taking a chance and hiring me, there were many people who were hired who were let go in 3 months or less because they were just unsuitable for the job.  The turn over of employees in my time there was ridiculous.  I suggested that they paid more to new candidates in order to attract better people, they didn't believe me and stated that there just isn't anyone out there.  This I find hard to believe.

Testing is the way forward.  As I stated in my last post on this topic it should be not just one question but a number designed to demonstrate many different abilities and not to prejudice a decision by the candidate failing at one thing.  There are certain things that I want to know from a candidate that I should implement for the next interview process.  I work with linux and the commandline a lot.  I use Cygwin on my windows machines to interface with my linux boxes and more importantly to gain the power of bash to manipulate files in windows.  Where am I going with this.  If a candidate claims knowledge of linux I have an easy question to test something basic.  Given the directory C:\src\ (or whatever) find all the java files in that directory and its sub directories that have classes that implement a certain interface.  Lets say the action listener interface.  I could even sit them at my keyboard and get them to do it.  If the candidate has no understanding of linux this would be impossible.  If they do but are unfamiliar with the syntax of grep or find for example I would be happy for them to spend 5 mins reading the man pages.  I would expect an answer like:

grep -Ri ActionListener * | grep -v svn | grep implements


I would expect to see them remove the svn directories from the result and refine the use of ActionListener to only when it is implemented in the class and not used as an anonymous class.
This is an ideal list of things a candidate should know.  Its use is explained in a link near the bottom.  A good score would certainly make a an ideal sounding candidate.  But these things especially the in depth knowledge of computer science things like sorting and search algorithms do not always translate well into the best employees. Discussed in a further article about the hiring process, it seems different approaches can be taken and are valid.


Another thing that I would like to see is their participation in something like StackOverflow, Experts exchange or some other forum of this type.  Alternatively involvement in an open source project.  


Regular expressions are such a useful tool.  Everyone should know about them and how to use them.  Again its easy to set up a simple test to discover the number of words of a certain type in a give text file.  I am reminded of this wonderful xkcd cartoon.


Finally here is another article on some positive traits to look out for in a new candidate.

Happy hiring!

Wednesday, June 1, 2011

Congratulations Linus - 20 years of a job well done

Linux kernel 3.0 released

Very amusing comments from him too.

Keep up the good work.

Shutdown Hooks Eclipsed by terminate button

We are creating a new module for our system to receive data from another source. Everytime we terminate the test application in Eclipse it causes and unexpected termination at the suppliers end. I get notified and they assume something is wrong. Solution add a Shutdown hook to end the connection gracefully.

Nothing is ever that easy.

Something About J: Shutdown Hook and Eclipse

turns out that a known bug in Eclipse is that clicking on the red terminate button does not kill anything nicely.

No solutions only workarounds:

1. Run in a console. A bit annoying when working in dev, you need to constantly stop and start the app and it can slow you down a bit.

2. Relatively safe workaround if you are using threads. It works well but you have to remember to use it and not click the red button!

Thanks to both for providing the most concise and useful explanations and suggestion.