Java on Ideas on Software Engineering

Selective setAccessible permissions using Byte Buddy and the Java SecurityManager

Tue, 19 Jul 2016 21:25:08 +0000

The Java SecurityManager provides a lot of good permissions for restricted what untrusted code can access. The downside of the standard permissions is that many of them are too broad, making it hard to enable permissions around commonly used functionality without defeating the purpose of the restricted security. The permissions around reflections are one of these cases, where the permissions around the setAccessible reflections method is either all on or all off. It is possible to work around this default behavior, providing more selective fine-grained permissions, by using bytecode weaving. However, for any of this this to work the Java SecurityManager must be used along with loading the code that these permissions will be applied to from a separate jar, probably using a custom or at least isolated instance of a URLClassLoader. This is a prerequisite which has already been well documented by Will Sargent in his post Self-Protecting Sandbox Using SecurityManager as well as my own example sandbox runtime environment sandbox-runtime.

The main permission in question around the setAccessible method is ReflectPermission("suppressAccessChecks"). When this permission is disabled it disallows changing the accessibility of fields, methods, and constructors. By default the SecurityManager is not enabled, which means all permission checks are disabled. Since this is the standard way Java is run, many common libraries, and even many alternative JVM language are dependent on being able to use reflection to access private fields and methods. If you do need to run code with a SecurityManager, then the ReflectPermission("suppressAccessChecks") permission defeats all security provided by the SecurityManager. Untrusted code can make use of setAccessible to work around and disable the SecurityManager entirely.

It may appear that the only two choices are between having a secure environment and disallowing untrusted code from using libraries or languages that require reflection, or allowing such access by having an unsecure environment. However, there is an alternative if you are willing to put in the work. Libraries like Byte Buddy make it possible to replace methods of existing classes with new implementations, at runtime, through the use of a Java agent.

I have implemented a custom, more fine-grained, permission around access to the setAccessible method. In my examples I have chosen to use the code source or the ClassLoader as the determining factor; code that attempts to call setAccessible will only be granted permission if both itself and the class it is attempting to change the accessibility of are from the same class loader. In other words, this reflections method can only be used on classes from the same jar as the untrusted code; it cannot be used to access private fields of standard Java classes. This is just what I chose, as it fit best for the security settings I’m targeting. The determining factor could be setup in many different ways, including a package or class whitelist or blacklist.

The first piece that is needed is the new custom Permission. This permission uses the class loader of the caller and callee to determine whether permissions should be granted or not.

public class UserSetAccessiblePermission extends Permission {
  private final ClassLoader loader;  

  public UserSetAccessiblePermission(ClassLoader loader) {
    super("userSetAccessible");

    this.loader = loader;
  }  

  @Override
  public boolean implies(Permission permission) {
    if (!(permission instanceof UserSetAccessiblePermission)) {
      return false;
    }

    UserSetAccessiblePermission that = (UserSetAccessiblePermission) permission;

    return that.loader == this.loader;
  }  

  // equals and hashCode omitted  

  @Override
  public String getActions() {
    return "";
  }
}

With the permission class in place, next we need a place to actually check the permission. This will happen in our custom implementation of the setAccessible method that will replace the standard setAccessible method. This can be done using Byte Buddy by creating our custom implementation as a static method with method parameters compatible with the standard setAccessible.

This class has a few interesting parts. It uses the @This annotation from Byte Buddy to get the object the original setAccessible method was called on. Access to this object can be used to retrieve the class loader of the object the accessibility is attempting to be changed on (callee). If the class loader is not an instance of the user class loader, then we just check the standard permission, but if it is a user class loader, then we continue with checking the custom permission. To check the permission we create an instance of the new permission with the class loader of the callee. If the caller has the same permission with the same instance of the user class loader, then the permissions is granted (a SecurityException is not thrown).

public class AccessibleObjectStub {
  private final static Permission STANDARD_ACCESS_PERMISSION =
      new ReflectPermission("suppressAccessChecks");

  public static void setAccessible(@This AccessibleObject ao, boolean flag)
      throws SecurityException {
    SecurityManager sm = System.getSecurityManager();
    if (sm != null) {
      Permission permission = STANDARD_ACCESS_PERMISSION;
      if (isFromUserLoader(ao)) {
        try {
          permission = getUserAccessPermission(ao);
        } catch (Exception e) {
          // Ignore. Use standard permission.
        }
      }

      sm.checkPermission(permission);
    }
  }

  private static Permission getUserAccessPermission(AccessibleObject ao)
      throws IllegalAccessException, InvocationTargetException, InstantiationException,
      NoSuchMethodException, ClassNotFoundException {
    ClassLoader aoClassLoader = getAccessibleObjectLoader(ao);
    return new UserSetAccessiblePermission(aoClassLoader);
  }

  private static ClassLoader getAccessibleObjectLoader(AccessibleObject ao) {
    return AccessController.doPrivileged(new PrivilegedAction<ClassLoader>() {
      @Override
      public ClassLoader run() {
        if (ao instanceof Executable) {
          return ((Executable) ao).getDeclaringClass().getClassLoader();
        } else if (ao instanceof Field) {
          return ((Field) ao).getDeclaringClass().getClassLoader();
        }

        throw new IllegalStateException("Unknown AccessibleObject type: " + ao.getClass());
      }
    });
  }

  private static boolean isFromUserLoader(AccessibleObject ao) {
    ClassLoader loader = getAccessibleObjectLoader(ao);

    if (loader == null) {
      return false;
    }

    return UserClassLoaders.isUserClassLoader(loader);
  }
}

This gets us most of the way, but not quite all the way there. This replacement implementation only checks the permission, but we still need to actually modify the accessibility of the object after the permission is granted. This cannot be done directly by the replacement as the accessibility is controlled by a private field on the AccessibleObject. We can, however, accomplish this using the method chaining feature of Byte Buddy, which can be used to chain private methods. The private field on the AccessibleObject class is controlled by a private method named setAccessible0 which we can chain as part of our Byte Buddy transformation using the andThen method.

First create a Byte Buddy type pool that includes the bootstrap classloader and your jar that includes the stubs.

final TypePool bootstrapTypePool = TypePool.Default.of(
    new ClassFileLocator.Compound(
        new ClassFileLocator.ForJarFile(jarFile),
        ClassFileLocator.ForClassLoader.of(null)));

Next get the AccessibleObject.setAccessible0 method using reflections.

Method setAccessible0Method;
try {
  String setAccessible0MethodName = "setAccessible0";
  Class[] paramTypes = new Class[2];
  paramTypes[0] = AccessibleObject.class;
  paramTypes[1] = boolean.class;
  setAccessible0Method = AccessibleObject.class
      .getDeclaredMethod(setAccessible0MethodName, paramTypes);
} catch (NoSuchMethodException e) {
  throw new RuntimeException(e);
}

With those two pieces we can now build the Byte Buddy transformer to match on the setAccessible method and to delegate to the stub and chain to the setAccessible0 method.

AgentBuilder.Transformer transformer = new AgentBuilder.Transformer() {
  @Override
  public DynamicType.Builder<?> transform(
      DynamicType.Builder<?> builder,
      TypeDescription typeDescription, ClassLoader classLoader) {
    return builder.method(
        ElementMatchers.named("setAccessible")
            .and(ElementMatchers.takesArguments(boolean.class)))
        .intercept(MethodDelegation.to(
            bootstrapTypePool.describe(
                "com.leacox.sandbox.security.stub.java.lang.reflect.AccessibleObjectStub")
                .resolve())
            .andThen(MethodCall.invoke(setAccessible0Method).withThis().withAllArguments()));
  }
}

That will take care of replacing the default implementation of setAccessible with our custom implementation and chain to the setAccessible0 method if permission is granted. If an exception occurs, for instance due to permissions not being granted, the chained transformation will exit early avoiding the call to actually change the accessibility.

The final step is to install Byte Buddy as a Java agent and actually perform the transformation. With a SecurityManager installed and this transformation executed, you can now run untrusted code with permissions to modify the accessibility of its own classes and methods, but not standard Java, your own runtime jar or other isolated code.

Instrumentation instrumentation = ByteBuddyAgent.install();
// Append the jar containing the stub replacement to the bootstrap classpath
instrumentation.appendToBootstrapClassLoaderSearch(jarFile);

AgentBuilder agentBuilder = new AgentBuilder.Default()
       .disableClassFormatChanges()
       .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
       .ignore(none()); // disable default ignores so we can transform Java classes
       .type(ElementMatchers.named("java.lang.reflect.AccessibleObject"))
       .transform(transformer)
       .installOnByteBuddyAgent();

In addition to everything above, you would also want to transform the other setAccessible method, setAccessible(AccessibleObject[] array, boolean flag). For more information on this and a full implementation of a simple sandboxed runtime with this transformation have a look at my sandbox-runtime on Github.

Implementing Scala-like Pattern Matching in Java 8

Thu, 13 Aug 2015 00:00:00 +0000

I’ve been writing Scala off and on for the last several years. I took both of the Scala courses on Coursera and really enjoyed them. However, I still write a lot of Java. There are a lot of Scala features that I miss while working with Java, and as Java 8 was coming out in early 2014, I thought about feature parity between the two languages. The addition of lambdas to Java is great, but one of my favorite features of Scala is pattern matching, and Java 8 does not include that. At the time I started to wonder if pattern matching could be implemented as a Java library. I wasn’t sure if it would even be possible, but I wrote up a quick Github gist with what some different patterns might look like.

My gist from early 2014 includes Guava Optional matching (I hadn’t yet noticed that Java 8 included it’s own Optional type), Integer matching as an example of matching on exact values, Cons like matching on a list, and the concept of an orElse case. I wrote these examples directly in the gist editor, I’m quite sure they don’t compile, but I wanted to get the concept out of my head and stored somewhere. Once I got the idea out of my head, I promptly forgot about it for just over a year.

In May 2015 I was starting to use Java 8 on some real projects, and one day that old gist popped into my head. I started thinking about it more and decided it might be worth giving a more in-depth look. Initially, I did some searches to see if any libraries had already been created for this. I found a couple that had some pattern matching features, but they were not exactly what I had imagined. Javaslang, provides a lot of nice functional features that are missing from Java 8, but pattern matching isn’t the focus of this library, and it only has support for matching on type and value. Benji Weber gives examples of pattern matching on his blog and implemented them in his expressions library.

The expressions library pattern matching was closer to what I had in mind. It can extract top-level and nested values, but it requires implementing a ‘constructor’ method for anything you want to match on and doesn’t built-in matching for things like Optional or List. At this point I decided I’d give a shot at implementing pattern matching in Java 8 that is as close to the Scala pattern matching as possible. I like to try to come up with a good name for my projects, so I spent some time with a thesaurus looking for words related to ‘matching’ and ‘patterns’, and I eventually landed on motif.

Before I go into some details about the process of implementing motif, below are some examples of what code using motif might look like.

Matching on Optional:

Optional<Person> personOpt = getPerson();
match(personOpt)
    .when(some(any())).then(person -> doStuff(person))
    .when(none()).then(() -> System.out.println("Person not found"))
    .doMatch();

Nested matching:

Optional<Tuple2<String, String>> opt = Optional.of(Tuple2.of("first", "second"));
match(opt)
    .when(some(tuple2(eq("third"), any()))).then(b -> doStuff(b))
    .when(some(tuple2(any(), eq("second")))).then(a -> doStuff(a))
    .when(none()).then(() -> System.out.println("Tuple not found"))
    .doMatch();

Checkout the Github page or Github repo for additional details. Version 0.1 of motif is available from maven central. It is a little more verbose (horizontally) than Scala, but it provides much of the same functionality. Currently, it supports extracting a maximum of 3 parameters, whereas Scala supports up to 22 in 2.10 and possibly unlimited in 2.11+? I’ve rarely found myself needing to extract more than 3 parameters in Scala, so I thought this would be an OK limitation for now. Support for additional parameters could be added in the future, but it will require writing additional code generation logic.

Motif provides many useful pattern matchers out of the box, but it is also very extensible. A Guava extension could be created, for instance, that provides matchers for Guava’s Optional and any other Guava concepts that would be useful to match on. The motif-generator project provides builders for generating matcher cases for extensions. OptionalCasesGenerator is a good example for using the generator.

The implementation of the Motif pattern matching happened in three phases. The initial phase was a varargs method that accepted the matching cases, and automatically executed the matching. This ran into ambiguity issues for consumers of the code, so I ended up scrapping it and converting to a fluent interface. By using a fluent interface, I could use intermediate classes to track types and avoid ambiguity. This second phase led to what is essentially the final API, except without nested extraction. Once I saw that the fluent interface was working well, I added nested extraction as a third phase. The nested extraction leads to a bit of combinatorial explosion of the API, so I added code generation during this phase to reduce the likelyhood of errors and to avoid writing very similar code multiple times. The code generation portion, is a separate library, and makes it easy for third party extensions to generate all of the necessary methods needed.

Motif has been released as version 0.1 and is available in Maven central. The API is pretty solid, and I don’t expect any dramatic changes in the future. Give it a try in your Java 8 code.

Differentiate Java Processes by Working Directory

Sun, 17 Feb 2013 00:00:00 +0000

A lot of server software runs on Java, but the different Java processes are listed as simply Java via commands like top or ps -ef. One feature of the Stackify agent is tracking the running processes on a server. I needed a way to identify what each Java process really was in a fairly simple and concise way, but there isn’t a clear winner in the output of the top or ps commands.

Running ps -ef | grep java will output something nasty, similar to what follows. You can likely figure out what the Java process is actually running from the args — the example below obviously being tomcat — however, it’s not the most readable and doesn’t give a clear and concise value to display to a user in the Stackify web site. The user of the process below also indicates that it is running tomcat, but that won’t be the case for all Java processes.

tomcat7 41298 1 5 18:05 ? 00:00:03 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

I decided it might work to differentiate the Java processes by working directory, but I wasn’t sure if there was a good way to find the working directory. Through some Google searches I discovered the pwdx command. The pwdx command takes a list of pids as parameters and outputs a row with the pid and the working directory for each pid. The working directory doesn’t work for all situations, but I’ve found it generally gives a concise way of determining what a Java process is actually running. I’m certainly open to suggestions if anyone has a better idea.

Running the pwdx command for the output above — sudo pwdx 41298 — leads to the following output. By combining this with the process name I can give the Java processes a more informative name when displayed in the Stackify website. This Java process would be displayed as Java (/var/lib/tomcat7).

41298: /var/lib/tomcat7

A Safer Way To Create Java Subprocesses

Mon, 11 Feb 2013 00:00:00 +0000

If you have ever had to start a subprocess from a Java application, then you likely know how painful it can be to use the Process class via Runtime.exec or ProcessBuilder. There are numerous posts on stackoverflow as well as a multitude of blog posts on the correct usage of the Process class. I found a blog post on the five common pitfalls of the Process class to be the most helpful when I began using subprocesses. Being aware of the pitfalls helps, but if you are running a lot of subprocesses, the additional code needed in each instance can become tedious and you increase the odds of making a mistake. I ended up creating the process-warden library for a safer way to create Java subprocesses that is reusable.

I make use of subprocesses in the Stackify monitoring agent for executing shell commands as well as for updating a running agent – which I hope to write a separate blog post on at some point. It quickly became clear that having a common way of providing a timeout for the Process.waitFor method and making sure that the streams are closed would be very beneficial. I created the FinalizedProcess and FinalizedProcessBuilder classes that wrap the usage of the Process class and take care of many of the common pitfalls.

FinalizedProcess implements Closable and in its close method it makes sure that all of the subprocess streams are closed as well as optionally destroying the subprocess. It also has a waitFor method that takes a timeout in milliseconds instead of a no parameter waitFor. If the subprocess execution takes longer than the timeout, then an InterruptedException will be thrown and the interrupted flag will be cleared when InterruptedExceptions are thrown. With these additions three of the five common pitfalls are remedied.

For the two remaining pitfalls the problem of process grandchildren not being kill when a subprocess is killed cannot be remedied via code, so that will still require user knowledge to avoid. As for the prompt consumption of subprocess output, I didn’t have an immediate need for discarding the output, so the initial version of process-warden doesn’t provide any functionality for consuming the output. In a future release I hope to add a class that will silently consume the output and error streams along with a method on FinalizedProcessBuilder to enable the silent consumption.

So how do you use process-warden? It’s easy.

Add it to your maven project:

<dependency>
	<groupId>com.leacox.process</groupId>
	<artifactId>process−warden<artifactId>
	<version>1.0.0</version>
</dependency>

Use FinalizedProcessBuilder to create a FinalizedProcess to execute:

// With Java 7
FinalizedProcessBuilder pb = new FinalizedProcessBuilder("myCommand", "myArg");
try (FinalizedProcess process = pb.start()) {
  int returnVal = process.waitFor(5000); // 5 second timeout
}

// With Java 6
FinalizedProcessBuilder pb = new FinalizedProcessBuilder("myCommand", "myArg");
FinalizedProcess process = pb.start();
try {
  int returnVal = process.waitFor(5000); // 5 second timeout
} finally {
  process.close();
}

If you are currently using Process and ProcessBuilder, give process-warden a shot. It should help simplify your code. I appreciate any feedback or suggestions. Feel free to log issues or fork the project on github.

Writing a Good Linux Init Script

Fri, 28 Dec 2012 16:18:45 +0000

A good Linux init script that works across multiple distros is hard to find. Most distros have a sample/skeleton init script to base your own init scripts on, but that requires having a separate init script for each distro. For the Stackify Linux agent I wanted to have a single init script that would work on all common Linux distros. I had a decent init script that worked pretty well for ‘start’ and ‘stop’ commands, but not so much for ‘status’. The ‘status’ command worked on my development machine (Ubuntu) but didn’t work on the other distros I was testing on (CentOS/Redhat/Amazon).

Since one of the things the Stackify Linux agent does is provide information on the different installed services and their statuses, it is important that the status of our own service is correct. As we prepare for the first beta release of the Linux agent, I decided it was time to rewrite the init script. I started with some Google searches on how to create a cross-distro init script, but they mostly led me back to the skeleton init scripts. While having lunch with a former coworker at Fuzzy’s Taco Shop – which I highly recommend – we got to discussing init scripts and he mentioned seeing an “init script to rule them all.”

After lunch, I promptly searched for this legendary script. My search lead me to the One Java init script to rule them all, which is an init script for Java application servers (e.g. Tomcat and JBoss) that will work on most Linux distros. Turns out this script was really close to what I needed since the Stackify Linux agent is a Java application. Many Linux distros have started implementing the Linux Standard Base (LSB) which provides logging methods and a standard way to implement init script statuses. However, since not all versions support LSB, I cannot use those logging methods for my init script.

The nice thing about the Java init script to rule them all is that it checks for the existence of the LSB methods and if they do not exist it provides its own implementation. I was able to base the Stackify agent init script largely on this existing script. The script now works on all of the distros previously mentioned and should also work on most other distros. The status returns the correct status on these distros and returns the correct status code for systems that do support LSB.