I am not sure if it's ok to open a new topic in the forum to file a bug report but I couldn't open a new issue on the tracker.
The bug is when you specify a wrong qrels file (in my case the topics file).
The bug occurs in org.terrier.evaluation.TRECQrelsInMemory.
Basically, in the method loadQrelsFile() (line 374), the catch block logs fatally an error:
FATAL - java.lang.NumberFormatException: For input string: ""
but the program doesn't not exit. (It's actually quite hard to know which line fired the exception since the try block is 30-line long...)
Then, in the method queryExistInQrels(String queryid) (line 459), the attribute qrelsPerQuery which has not been initialized (thus is null) is asked for its length which throws a java.lang.NullPointerException.
None of the logs is clear about being fed a wrong qrel file. And usually, when logging with fatal, the program is expected to exit isn't it (since it's fatal)?