New Java Features¶
I decided to refresh my Java knowledge (last version I used was java 1.6), and because I learn by coding (much better than I learn by reading) here are code samples I prepared.
Note
This might be updated.
Java 1.7¶
Note
All examples were actually made on Java 1.8 JVM.
I used this as a reference list of changes ( Oracle comparison is way to comprehensive).
java.nio.path package¶
Java File class is awful, but a very nice api was introduced in this
version of Java.
Package java.nio.path has a very nice tutorial by Oracle, so
I won’t describe it here.
Here is my (very simple) example, it works similarly to
Disk Usage Analyzer.
or du command on Linux, that is: it summarizes disk usage for a folder.
To do this I just needed to implement a FileVisitor instance, and then
pass it to Files.walkFileTree.
Code Highlights¶
Most of the logic is in Visitor that subclasses FileVisitor, this
class is used to traverse whole directory tree. Inside this instance we keep
track of where in the directory tree we are, by using a stack.
Queue<Long> fileSizes = Collections.asLifoQueue(new ArrayDeque<>());
Each entry in the stack corresponds to a parent directory of currently processed path, and each contains a total size of that directory.
To add size of current object to size of the parent following code is used:
private void pushSize(long size){
long lastSize = fileSizes.poll();
lastSize+=size;
fileSizes.add(lastSize);
}
@Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
fileSizes.add(0L);
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
long dirSize = fileSizes.poll();
pushSize(dirSize);
if (maxDepthToDisplay<0 || maxDepthToDisplay >= fileSizes.size()) {
System.out.println(level(fileSizes.size()) + dir + " " + humanReadableByteCount(dirSize, true));
}
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (Files.isRegularFile(file)) {
pushSize(Files.size(file));
}
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
return FileVisitResult.CONTINUE;
}
Complete example¶
import java.io.IOException;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayDeque;
import java.util.Collections;
import java.util.Queue;
public class PathExamples {
private static class Visitor extends SimpleFileVisitor<Path>{
long maxDepthToDisplay;
Queue<Long> fileSizes = Collections.asLifoQueue(new ArrayDeque<>());
protected Visitor(long maxDepthToDisplay) {
super();
this.maxDepthToDisplay = maxDepthToDisplay;
fileSizes.add(0L);
}
private void pushSize(long size){
long lastSize = fileSizes.poll();
lastSize+=size;
fileSizes.add(lastSize);
}
private String level(int depth){
StringBuilder sbr = new StringBuilder();
for (int ii = 0; ii < depth; ii++) {
sbr.append(" ");
}
return sbr.toString();
}
/**
* http://stackoverflow.com/a/3758880
*/
public static String humanReadableByteCount(long bytes, boolean si) {
int unit = si ? 1000 : 1024;
if (bytes < unit) return bytes + " B";
int exp = (int) (Math.log(bytes) / Math.log(unit));
String pre = (si ? "kMGTPE" : "KMGTPE").charAt(exp-1) + (si ? "" : "i");
return String.format("%.1f %sB", bytes / Math.pow(unit, exp), pre);
}
@Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
fileSizes.add(0L);
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
long dirSize = fileSizes.poll();
pushSize(dirSize);
if (maxDepthToDisplay<0 || maxDepthToDisplay >= fileSizes.size()) {
System.out.println(level(fileSizes.size()) + dir + " " + humanReadableByteCount(dirSize, true));
}
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (Files.isRegularFile(file)) {
pushSize(Files.size(file));
}
return FileVisitResult.CONTINUE;
}
}
public static void main(String[] args) throws IOException {
Path path = Paths.get(args[0]);
Files.walkFileTree(path, new Visitor(3));
}
}
Fork Join Framework¶
Java 1.7 has very nice Fork-Join Framework, that allows one to dynamically split between cores, but here is the catch: we don’t know the amount of work needed upfront.
I have decided to try this framework, to (once again) summarize size of a directory tree.
This framework is nicely explained in the tutorials.
Overall I’m surprised with the performance of both naive and parralel implementation, naive version takes 6 seconds (when ran on my 150GB home directory), while parralel takes 3sec.
Code Highlights¶
Task result is a POJO object, containing path, it’s size, information whether this path is a directory, and sub directories (if any). Here is the definition:
private static class WalkFileResult{
public final Path dirPath;
public final boolean isDir;
public final long dirSize;
public final List<WalkFileResult> subdirs;
public WalkFileResult(Path dirPath, long dirSize) {
this(dirPath, dirSize, Collections.emptyList());
}
public WalkFileResult(Path dirPath, long dirSize, List<WalkFileResult> subdirs) {
super();
this.dirPath=dirPath;
this.dirSize=dirSize;
this.isDir=Files.isDirectory(dirPath);
this.subdirs=subdirs;
}
}
Single task has following logic:
- If we are looking at a file, calculate file size and return it.
- If we are looking at a directory, create task for each child of the directory, execute these tasks in parrarel and then calculate the size.
In Java it is:
@Override
protected WalkFileResult compute() {
try {
if (Files.isRegularFile(currentPath)) {
return new WalkFileResult(currentPath, Files.size(currentPath));
} else if(Files.isDirectory(currentPath)) {
List<WalkFileTask> subTasks = getSubtasks();
return joinOnSubtasks(subTasks);
}
}catch (IOException | InterruptedException e){
throw new RuntimeException(e);
}catch (ExecutionException e){
throw new RuntimeException(e.getCause());
}
return new WalkFileResult(currentPath, 0L);
}
private List<WalkFileTask> getSubtasks() throws IOException{
// This visitor just returns immediate children of current path
Visitor v = new Visitor(currentPath);
Files.walkFileTree(currentPath, v);
return v.subtasks;
}
private WalkFileResult joinOnSubtasks(List<WalkFileTask> subTasks) throws ExecutionException, InterruptedException {
long size = 0;
List<WalkFileResult> subDirs = new ArrayList<>();
for (WalkFileTask res: invokeAll(subTasks)){
WalkFileResult wfr = res.get();
size+=wfr.dirSize;
if (wfr.isDir){
subDirs.add(wfr);
}
}
return new WalkFileResult(currentPath, size, subDirs);
}
Complete example¶
package examples;
import javax.sound.midi.SysexMessage;
import java.io.IOException;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.ForkJoinTask;
import java.util.concurrent.RecursiveTask;
public class ForkJoinPath{
private static class WalkFileResult{
public final Path dirPath;
public final boolean isDir;
public final long dirSize;
public final List<WalkFileResult> subdirs;
public WalkFileResult(Path dirPath, long dirSize) {
this(dirPath, dirSize, Collections.emptyList());
}
public WalkFileResult(Path dirPath, long dirSize, List<WalkFileResult> subdirs) {
super();
this.dirPath=dirPath;
this.dirSize=dirSize;
this.isDir=Files.isDirectory(dirPath);
this.subdirs=subdirs;
}
}
private static class WalkFileTask extends RecursiveTask<WalkFileResult>{
private static class Visitor extends SimpleFileVisitor<Path>{
public final Path root;
public List<WalkFileTask> subtasks = new ArrayList<>();
protected Visitor(Path root) {
super();
this.root = root;
}
@Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
if(Files.isSameFile(dir, root)){
return FileVisitResult.CONTINUE;
}
if (Files.isReadable(dir)) {
subtasks.add(new WalkFileTask(dir));
}
return FileVisitResult.SKIP_SUBTREE;
}
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (Files.isReadable(file) && Files.isRegularFile(file)) {
subtasks.add(new WalkFileTask(file));
}
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
return FileVisitResult.CONTINUE;
}
}
private final Path currentPath;
public WalkFileTask(Path currentPath) {
super();
this.currentPath=currentPath;
}
private List<WalkFileTask> getSubtasks() throws IOException{
// This visitor just returns immediate children of current path
Visitor v = new Visitor(currentPath);
Files.walkFileTree(currentPath, v);
return v.subtasks;
}
private WalkFileResult joinOnSubtasks(List<WalkFileTask> subTasks) throws ExecutionException, InterruptedException {
long size = 0;
List<WalkFileResult> subDirs = new ArrayList<>();
for (WalkFileTask res: invokeAll(subTasks)){
WalkFileResult wfr = res.get();
size+=wfr.dirSize;
if (wfr.isDir){
subDirs.add(wfr);
}
}
return new WalkFileResult(currentPath, size, subDirs);
}
@Override
protected WalkFileResult compute() {
try {
if (Files.isRegularFile(currentPath)) {
return new WalkFileResult(currentPath, Files.size(currentPath));
} else if(Files.isDirectory(currentPath)) {
List<WalkFileTask> subTasks = getSubtasks();
return joinOnSubtasks(subTasks);
}
}catch (IOException | InterruptedException e){
throw new RuntimeException(e);
}catch (ExecutionException e){
throw new RuntimeException(e.getCause());
}
return new WalkFileResult(currentPath, 0L);
}
}
private static String level(int depth){
StringBuilder sbr = new StringBuilder();
for (int ii = 0; ii < depth; ii++) {
sbr.append(" ");
}
return sbr.toString();
}
/**
* http://stackoverflow.com/a/3758880
*/
public static String humanReadableByteCount(long bytes, boolean si) {
int unit = si ? 1000 : 1024;
if (bytes < unit) return bytes + " B";
int exp = (int) (Math.log(bytes) / Math.log(unit));
String pre = (si ? "kMGTPE" : "KMGTPE").charAt(exp-1) + (si ? "" : "i");
return String.format("%.1f %sB", bytes / Math.pow(unit, exp), pre);
}
private static void printResult(WalkFileResult wfr, int depth){
if (depth >= 3){
return;
}
System.out.println(level(depth) + wfr.dirPath + " " + humanReadableByteCount(wfr.dirSize, false));
for (WalkFileResult child: wfr.subdirs){
printResult(child, depth+1);
}
}
public static void main(String[] args) throws ExecutionException, InterruptedException {
long start = System.nanoTime();
Path path = Paths.get(args[0]);
ForkJoinPool pool = new ForkJoinPool();
WalkFileTask task = new WalkFileTask(path);
pool.execute(task);
printResult(task.get(), 0);
double duration = (System.nanoTime() - start) * 1E-9;
System.out.println(duration);
}
}
Notable mentions¶
There is also very nice WatchService, that allows to monitor filesysem
for file changes.
Java 1.8¶
Streams and Lambdas¶
Third attempt to do the same task: to summarize size of a directory tree.
This times using Streams and Lambdas. Solution is most concise, but
least readable IMO. Also, while other solutions transparently handle unreadable
directories, this one explodes with AccessDenied exception.
Code Highlights¶
Result POJO:
A function that can throw an exception:
@FunctionalInterface
public interface CheckedFunction<T, R> {
R apply(T t) throws IOException;
}
A lambda that calculates file size:
CheckedFunction<Path, DirSize> mapper = (Path p) -> new DirSize(p,
Files.walk(p).parallel()
.filter(Files::isReadable)
.mapToLong(StreamExamples::safeSize).sum());
A stream that walks over FS calculating size of each directory:
Files.walk(path, 3)
.parallel().filter(Files::isDirectory).filter(Files::isReadable).map(
(Path p) -> {
try {
return mapper.apply(p);
} catch (IOException e) {
return new DirSize(p, -1);
}
}).forEach(
(DirSize d) ->
System.out.println(d.path + " " + humanReadableByteCount(d.size, false)));
Complete example¶
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
/**
* Created by jb on 12/3/15.
*/
public class StreamExamples {
private static class DirSize{
public final Path path;
public final long size;
public DirSize(Path path, long size) {
this.path = path;
this.size = size;
}
}
@FunctionalInterface
public interface CheckedFunction<T, R> {
R apply(T t) throws IOException;
}
public static long safeSize(Path p){
try {
return Files.size(p);
} catch (IOException e) {
return 0;
}
}
public static String humanReadableByteCount(long bytes, boolean si) {
int unit = si ? 1000 : 1024;
if (bytes < unit) return bytes + " B";
int exp = (int) (Math.log(bytes) / Math.log(unit));
String pre = (si ? "kMGTPE" : "KMGTPE").charAt(exp-1) + (si ? "" : "i");
return String.format("%.1f %sB", bytes / Math.pow(unit, exp), pre);
}
public static void main(String[] args) throws IOException {
long start = System.nanoTime();
Path path = Paths.get(args[0]);
CheckedFunction<Path, DirSize> mapper = (Path p) -> new DirSize(p,
Files.walk(p, Integer.MAX_VALUE).parallel().filter(Files::isReadable).mapToLong(StreamExamples::safeSize).sum());
Files.walk(path, 3)
.parallel().filter(Files::isDirectory).filter(Files::isReadable).map(
(Path p) -> {
try {
return mapper.apply(p);
} catch (IOException e) {
return new DirSize(p, -1);
}
}).forEach((DirSize d) -> System.out.println(d.path + " " + humanReadableByteCount(d.size, false)));
double duration = (System.nanoTime() - start) * 1E-9;
System.out.println(duration);
}
}