org.apache.hadoop.util
Class MRAsyncDiskService

java.lang.Object
  extended by org.apache.hadoop.util.MRAsyncDiskService

public class MRAsyncDiskService
extends Object

This class is a container of multiple thread pools, each for a volume, so that we can schedule async disk operations easily. Examples of async disk operations are deletion of files. We can move the files to a "toBeDeleted" folder before asychronously deleting it, to make sure the caller can run it faster. Users should not write files into the "toBeDeleted" folder, otherwise the files can be gone any time we restart the MRAsyncDiskService. This class also contains all operations that will be performed by the thread pools.


Field Summary
static org.apache.commons.logging.Log LOG
           
static String TOBEDELETED
           
 
Constructor Summary
MRAsyncDiskService(FileSystem localFileSystem, String[] nonCanonicalVols)
          Create a AsyncDiskServices with a set of volumes (specified by their root directories).
MRAsyncDiskService(JobConf conf)
          Initialize MRAsyncDiskService based on conf.
 
Method Summary
 boolean awaitTermination(long milliseconds)
          Wait for the termination of the thread pools.
 void cleanupAllVolumes()
          Move all files/directories inside volume into TOBEDELETED, and then delete them.
 boolean moveAndDeleteAbsolutePath(String absolutePathName)
          Move the path name to a temporary location and then delete it.
 boolean moveAndDeleteFromEachVolume(String pathName)
          Move the path name on each volume to a temporary location and then delete them.
 boolean moveAndDeleteRelativePath(String volume, String pathName)
          Move the path name on one volume to a temporary location and then delete them.
 void shutdown()
          Gracefully start the shut down of all ThreadPools.
 List<Runnable> shutdownNow()
          Shut down all ThreadPools immediately.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

TOBEDELETED

public static final String TOBEDELETED
See Also:
Constant Field Values
Constructor Detail

MRAsyncDiskService

public MRAsyncDiskService(FileSystem localFileSystem,
                          String[] nonCanonicalVols)
                   throws IOException
Create a AsyncDiskServices with a set of volumes (specified by their root directories). The AsyncDiskServices uses one ThreadPool per volume to do the async disk operations.

Parameters:
localFileSystem - The localFileSystem used for deletions.
nonCanonicalVols - The roots of the file system volumes, which may be absolte paths, or paths relative to the ${user.dir} system property ("cwd").
Throws:
IOException

MRAsyncDiskService

public MRAsyncDiskService(JobConf conf)
                   throws IOException
Initialize MRAsyncDiskService based on conf.

Parameters:
conf - local file system and local dirs will be read from conf
Throws:
IOException
Method Detail

shutdown

public void shutdown()
Gracefully start the shut down of all ThreadPools.


shutdownNow

public List<Runnable> shutdownNow()
Shut down all ThreadPools immediately.


awaitTermination

public boolean awaitTermination(long milliseconds)
                         throws InterruptedException
Wait for the termination of the thread pools.

Parameters:
milliseconds - The number of milliseconds to wait
Returns:
true if all thread pools are terminated within time limit
Throws:
InterruptedException

moveAndDeleteRelativePath

public boolean moveAndDeleteRelativePath(String volume,
                                         String pathName)
                                  throws IOException
Move the path name on one volume to a temporary location and then delete them. This functions returns when the moves are done, but not necessarily all deletions are done. This is usually good enough because applications won't see the path name under the old name anyway after the move.

Parameters:
volume - The disk volume
pathName - The path name relative to volume root.
Returns:
false if the file is not found
Throws:
IOException - If the move failed

moveAndDeleteFromEachVolume

public boolean moveAndDeleteFromEachVolume(String pathName)
                                    throws IOException
Move the path name on each volume to a temporary location and then delete them. This functions returns when the moves are done, but not necessarily all deletions are done. This is usually good enough because applications won't see the path name under the old name anyway after the move.

Parameters:
pathName - The path name relative to each volume root
Returns:
false If any of the target pathName did not exist, note that the operation is still done on all volumes.
Throws:
IOException - If any of the move failed

cleanupAllVolumes

public void cleanupAllVolumes()
                       throws IOException
Move all files/directories inside volume into TOBEDELETED, and then delete them. The TOBEDELETED directory itself is ignored.

Throws:
IOException

moveAndDeleteAbsolutePath

public boolean moveAndDeleteAbsolutePath(String absolutePathName)
                                  throws IOException
Move the path name to a temporary location and then delete it. Note that if there is no volume that contains this path, the path will stay as it is, and the function will return false. This functions returns when the moves are done, but not necessarily all deletions are done. This is usually good enough because applications won't see the path name under the old name anyway after the move.

Parameters:
absolutePathName - The path name from root "/"
Returns:
false if we are unable to move the path name
Throws:
IOException - If the move failed


Copyright © 2009 The Apache Software Foundation