What to do if the Hook API is not enough for you ? You want for example to write a code which handles a new command like "YOURCOOLCOMMAND: whatever@example".
For this kind of needs you should implement the CommandHandler interface. This gives you a lower-level API to handle this kind of tasks. If you want to support a custom Hook in your CommandHandler its the best to just extend AbstractHookableCmdHandler.
Be aware that your implementation needs to be thread-safe as it will get used as singleton. If you need to store temporary informations within the SMTP Transaction/Session you should use the SMTPSession.getState().put(...) and SMTPSession.getConnectionState().put(...) methods.
For a more explanation see the apidocs for the protocols library which is used by James SMTP Server.
Just create a jar file which contains your code and put it in /path/to/james/conf/lib/ folder. The jar will get picked up by the ClassLoader automatic then.
After that open the smtpserver.xml file and add your hook to the <handlers></handlers> block in the order you want to have it executed. So if you have 2 MailHook implementation make sure the first one which is listed in smtpserver.xml is the one you want to call first.