Troubleshooting file leaks in java applications

Recently I was diagnosing a production issue related file handle leaks. The setup was as follows – There were two applications. Application A was an ETL process that loaded the input file into the system and Application B that deleted these input files after A completes. Application B randomly failed with an error that stated that the file was still being used by another process.

Initially we thought it to be an issue with application scheduling. Probably Application B was starting before Application A completed but looking at the execution times we ruled out that possibility. At the first look it didn’t seem obvious for Application A to not close the file handle since the failure was random in nature. Application A being a ETL process it opened and closed the input file multiple times during its processing. Reviewing code to figure out all the code snippets that did file i/o was the last we might want to do.

Looking out for a solution I came across this nice utility called the file leak detector that turned out to be the right debugger for the problem in hand. It is a java agent that records file open/close operations. It can be run as a http server so that the information could be accessed from the browser.

The java agent pinpointed us (printed stacktrace) to the piece of code that left the file handle open. It was quite funny to look at the code after we found it . The code look similar to

   public void validateFile(String fileName) {
		try {
			new InputStreamReader(new FileInputStream(fileName));
		} catch (Exception e) {
			logger.error("Validation failed", e);
		}
	}

The code never closed the file handle after opening it and relied on garbage collection to do that, hence the failure was random.

Looking into the source, it seems that the tool uses java instrumentation api to alter the byte code of the file i/o classes namely FileInputStream, FileOutputStream, RandomAccessFile & ZipFile.

This ended the journey of troubleshooting a dreaded production bug !

Advertisement