[]

Image Animation With Procedural Textures

What we are building

In 1630, Dutch painter named Matthias Storm created a beautiful painting of a young man reading by candlelight. Little did he know that in 2017, his painting would be brought to life with the help of procedural textures and WebGL.

After watching this Blizzard tech talk, I started to see the potential for procedural noise to create cool effects. The technique is simple: multiply two textures together, and then multiply the product by 2.0. This technique is how Blizzard built all their spell effects in the game Diablo III. Turns out that it’s a common game technique.

Applying game programming techniques to the web

When I saw this tech talk, first I started wishing I was a game programmer. Then I started wondering, could I use this same technique on the web? Adding in some fiery, smokey, or magical effects could be a great way to spice up static images and make them into a compelling experience.

Breakdown

Here are some of the layers in the effect built up step-by-step:

  • I took 2 layers of turbulent Perlin noise, multiplied them together and then by 2.0
  • I scrolled both layers of noise upwards at different speeds
  • I multipled this noise by a mask that I made in Photoshop for the flame, to make sure the flame only showed where I wanted it
  • I used the mask noise to ramp between yellow and orange for the flame
  • I used the screen blend mode to put the flame over the image
  • I used 2 more layers of noise and another mask for the rest of the lighting in the scene. The noise was lower frequency and non-turbulent to be more subtle.
  • I pulsed the whole effect by mixing with the original image using offset sine waves

Implementation

Code On Github

I used React along with gl-react for the WebGL implementation, and glslify to include the noise from glsl-noise.

Mar 31

Advanced Android Animation in Scala

On mobile, it’s really important for animations to feel tactile and physical. When doing complex animations, it’s common to chain together animations, or even interleave animations with other operations, like changing text. Neither Android nor iOS handle multi-stage animations gracefully. We are going to build a new animation approach that is cleaner and more powerful.

My belief is that with a more concise animation system, you will be encouraged to use animations more, whereas with verbose and clunky animation systems, you will be tempted to skimp on animations.

What we are building

async {
  await(refreshButton.animateAlpha(0.5f, 500 millis))
  onUiThread { refreshButton.setEnabled(false) }

  val fakeHttpRequest = getBalance()

  await(progress.animateAlpha(1f, 500 millis))

  // asynchronously wait for the new balance from the server
  val newBalance: Int = await(fakeHttpRequest)

  await(accountBalance.animateScale(0.8f, 0.8f, 300 millis))

  // animate the dollar amount
  val numberAnimation = ViewHelper.valueAnimator(
    startBalance,
    newBalance, // the new balance from the server
    1.5 seconds,
    new DecelerateInterpolator(1),
    new IntEvaluator()
  ) {
    v => accountBalance.setText("$" + v.toString)
  }

  await(numberAnimation)
  await(accountBalance.animateScale(1f, 1f, 500 millis, new OvershootInterpolator(3f)))
  await(progress.animateAlpha(0f, 500 millis))
  await(refreshButton.animateAlpha(1f, 500 millis))

  onUiThread { refreshButton.setEnabled(true) }
}

This probably looks like no Android code that you’ve seen before. It’s a combination of Scala’s Futures and animatior helpers that we will write, which lets us chain the animations in a straightforward and linear way.

The refresh animation is deceptively simple: it’s actually built from 8 animations along with an HTTP request and other UI changes. This is incredibly compact.

Why do I do this?

Android has a zillion animation APIs but they all feel like they are lacking something.

  • View Animation and ViewPropertyAnimator: Can only animate certain properties. Cannot animate text color, drawable properties like background color, or LayoutParams.

  • Property Animation: ValueAnimator and ObjectAnimator are powerful, can animate arbitary objects unlike iOS. Low-level, verbose, and annoying to compose. Specifying both the start value and end value gets tiring really quickly.

  • AnimationSet: Can compose multiple animations into a chain, but cannot compose animations with non-animations, like changing views or other asynchronous operations.

  • XML-based animations: Inflexible due to hard-coded magic numbers.

The biggest problem with all these animation approaches is their composability. It’s awkward to chain animations with other animations or with things that are not animations.

Composability, composability, composability

When you are only chaining two animations, traditional Android animation works pretty well. For example, here is ViewPropertyAnimator in Java:

textView.animate().alpha(0f).setDuration(500).withEndAction(new Runnable() {
  @Override
  public void run() {
    textView.setText("REFRESHED TEXT");
    textView.animate().alpha(1f).setDuration(500);
  }
});

Looks fine. So far. But if we try to add on a couple more animations, you encounter callback hell:

textView.animate().alpha(0f).setDuration(500).withEndAction(new Runnable() {
  @Override
  public void run() {
    textView.setText("REFRESHED TEXT");

    textView.animate().alpha(1f).setDuration(500).withEndAction(new Runnable() {
      @Override
      public void run() {
        textView.animate().scaleX(1.03f).scaleY(1.03f).setDuration(300).withEndAction(new Runnable() {
          @Override
          public void run() {
            textView.animate().scaleX(1f).scaleY(1f).setDuration(500);
          }
        });
      }
    });
  }
});

If you keep going like this, you are going to hit editor-window overflow.

ValueAnimator/ObjectAnimator have similar issues with nesting.

To make things even more complicated, what if you wanted to compose your refresh animation with an HTTP request? If we wanted to do something like (1) fade out, (2) run HTTP request to retrieve the new resource, and (3) fade in with new text, then it would be a total mess of callbacks.

Let’s make our own animation helper that is composable. The goal:

  • Chain animations in sequence
  • Run animations in parallel
  • Avoid needing to specify start values
  • Animate anything
  • Concise syntax
  • Animations can be composed with non-animations and with asynchronous operations like HTTP requests
  • No nesting in multi-stage animations

Animation with Futures

My previous article about Scala on Android introduced Scala’s Future abstraction for dealing with asynchronous operations. When doing animations, I reached for Futures because they are asynchronous. If your animations are in Futures, and your HTTP requests are also in Futures, then you can compose animations and HTTP requests! I got this idea from trying to comprehend the Macroid Scala Android framework.

We are going to make some animation helpers by wrapping ValueAnimators inside Futures. To make things even more succint, we are going to patch our animation methods onto Views.

Example: Fade out a button, then disable it

button.animateAlpha(0f, 500 millis) onSuccess(case _ => button.setEnabled(false))

The Implementation

We will give Views our animation helper methods by using Scala’s implicit wrapper class construct:

object ViewHelper {
  implicit class AnimatedView(v: View) {
    def animateAlpha(alpha: Float, duration: FiniteDuration): Future[Unit] = {
      //...
    }
  }
}

// in Activity, import the wrapper conversion into scope
import com.emptyarray.scala.android.util.ViewHelper._

// all `Views` are now implicitly wrapped in AnimatedViews,
// and now have `animateAlpha`
view.animateAlpha(0f, 500 millis)

ValueAnimator, first try

ValueAnimators are very powerful and general, but they have a lot of boilerplate. Let’s make a helper that gives us a ValueAnimator in one shot, runs a closure from an AnimatorUpdateListener and returns a Future that completes when the animation is done.

object ViewHelper extends UiThreadHelper {
  def valueAnimator(start: Float, 
                    stop: Float, 
                    duration: FiniteDuration)(f: Float => Unit): Future[Unit] = {

    val animator = ValueAnimator.ofFloat(start, stop)

    animator.setDuration(duration.toMillis)

    animator.addUpdateListener(new AnimatorUpdateListener {
      override def onAnimationUpdate(animator: ValueAnimator): Unit = {
        f(animator.getAnimatedValue.asInstanceOf[Float])
      }
    })

    runAnimator(animator) // we will write this soon
  }
}

Things to notice:

  • FiniteDuration is Scala’s time duration class which let’s you write stuff like 1.5 seconds instead of only using milliseconds as a Long
  • You might notice that there are two parameter sets in parentheses! That’s because it’s a curried function. This is nicer for syntax.
  • f: Float => Unit means a function from a Float to Unit. Unit is Scala’s void. This function will run for every update of the animation, and we will access the Float as the animated value.
  • Returns a Future

But this version is too simplistic. What if we want to animate an Int instead of a Float? What if we want to use a different Interpolator or Evaluator?

ValueAnimator, second try

object ViewHelper extends UiThreadHelper {
  def valueAnimator[A, T](start: A, stop: A, duration: FiniteDuration,
                       interpolator: TimeInterpolator,
                       evaluator: TypeEvaluator[T])
                      (f: A => Unit): Future[Unit] = {

    // pattern match handles the casting                  
    val animator = (start, stop) match {
      case (s: Float, e: Float) => ValueAnimator.ofFloat(s, e)
      case (s: Int, e: Int) => ValueAnimator.ofInt(s, e)
    }

    animator.setDuration(duration.toMillis)
    animator.setInterpolator(interpolator)
    animator.setEvaluator(evaluator)

    animator.addUpdateListener(new AnimatorUpdateListener {
      override def onAnimationUpdate(animator: ValueAnimator): Unit = {
        f(animator.getAnimatedValue.asInstanceOf[A])
      }
    })

    runAnimator(animator)
  }
}

Now valueAnimator is more generic thanks to the type parameters: one for the type of the ValueAnimator, and the other for the type of the Evaluator.

Next we will actually run the ValueAnimator inside a Future.

ValueAnimator in a Future

object ViewHelper {
  def runAnimator(animator: Animator): Future[Unit] = {
    val p = Promise[Unit]()

    animator.addListener(new AnimatorListenerAdapter {
      override def onAnimationEnd(animator: Animator): Unit = p.success(())
      override def onAnimationCancel(animtor: Animator): Unit = p.success(())
    })

    onUiThread(animator.start())

    p.future
  }
}

We start a Promise, and run the animator (on the UI thread), completing the promise when the animation is ended or canceled. We return the Promise's Future, which can be observed by consumers. There is no way to cancel an animation right now, but this approach could be extended to add one.

With our helpers, we can now animate any function we want. Well, any function that takes a Float and returns nothing.

animateAlpha, the whole story

implicit class AnimatedView(v: View) {

  def animateAlpha(alpha: Float, duration: FiniteDuration): Future[Unit] = {

    getFromUiThread(v.getAlpha).flatMap {
      startAlpha =>

        ViewHelper.valueAnimator(startAlpha, alpha, duration,
          new AccelerateDecelerateInterpolator(), new FloatEvaluator())(v.setAlpha)
    }
  }
}

OK, I admit that this is starting to get a little complicated. But I’ll explain what’s going on here:

  • The key part is that we are now using valueAnimator to animate a function
  • The function we are animating is setAlpha on the View that wrapped in our implicit class. v.setAlpha is a short-hand for a function invocation like { x => v.setAlpha(x) }, where x is passed from our ValueAnimator's getAnimatedValue
  • I don’t want to want to have to specify the start alpha: it should be the View's current alpha. The problem is that I’m being a bit mean to myself and I’m not willing to assume that I’m on the UI thread.
  • To get the View's initial alpha, I run getAlpha in a Future that runs on the UI thread. That’s what getFromUiThread does, and you can see the implementation on Github.
  • What is flatMap? That’s the question that all beginning Scala developers ask. flatMap is an idea from functional programming. In the context of a Future, what you need to know is that flatMap lets you chain Futures together.

The payoff for this complexity is that we can have animations without a start value, and we can start them from any thread. This will make our code much cleaner in other places.

Animating Numbers

Since we can animate functions, we can animate the numbers with the valueAnimator helper also:

ViewHelper.valueAnimator(
  startBalance,
  newBalance,
  1.5 seconds,
  new DecelerateInterpolator(1),
  new IntEvaluator()
) {
  v => accountBalance.setText("$" + v.toString)
}

v => accountBalance.setText("$" + v.toString) is the animation function that we are passing in. This returns a Future that we can chain with other animations.

So how do you chain animations?

Normally in Scala, you would chain Futures like using for comprehensions:

// fade a view in, then out, then make it gone
for {
  a <- progress.animateAlpha(1f, 500 millis)
  b <- progress.animateAlpha(0f, 500 millis)
} yield onUiThread { progress.setVisibility(View.GONE) }

This syntax isn’t nested, but it is a little awkard. for comprehensions require extracting the value of the Future into the variables on the left, but we don’t need that because our animation helpers return Unit (void).

Luckily, scala-async supplies macros that let us wait for Futures to complete without blocking:

import scala.async.Async.{async, await}

async {
  await(progress.animateAlpha(1f, 500 millis))
  await(progress.animateAlpha(0f, 500 millis))
  onUiThread { progress.setVisibility(View.GONE) }
}

Much more intuitive. And note the ease of composing Futures with other operations, like changing the UI.

The full animation

Now that chaining animations is easy, let’s go to town and write a really complex refresh animation, interleaved with an HTTP request and with enabling/disabling the refresh button. Imagine how much more code this would take in a normal Java Android approach!

async {
  await(refreshButton.animateAlpha(0.5f, 500 millis))
  onUiThread { refreshButton.setEnabled(false) }

  val fakeHttpRequest = getBalance()

  await(progress.animateAlpha(1f, 500 millis))

  // asynchronously wait for the new balance from the server
  val newBalance: Int = await(fakeHttpRequest)

  await(accountBalance.animateScale(0.8f, 0.8f, 300 millis))

  val numberAnimation = ViewHelper.valueAnimator(
    startBalance,
    newBalance,
    1.5 seconds,
    new DecelerateInterpolator(1),
    new IntEvaluator()
  ) {
    v => accountBalance.setText("$" + v.toString)
  }

  await(numberAnimation)
  await(accountBalance.animateScale(1f, 1f, 500 millis, new OvershootInterpolator(3f)))
  await(progress.animateAlpha(0f, 500 millis))
  await(refreshButton.animateAlpha(1f, 500 millis))

  onUiThread { refreshButton.setEnabled(true) }
}

We need onUiThread because the async block is running in the thread pool, not on the main thread. See my previous article for more explanation about threading. Find this whole project on Github with the full implementation:

Project On Github

Jul 11

Scala Takes Over Android

Scala is becoming an increasingly popular choice on Android. Why?

  • concise, flexible syntax
  • higher-level abstractions
  • functional programming suport
  • type safety
  • libraries and ecosystems
  • interoperability with Java

As mobile gets bigger, there is an increasing need for abstraction for solving common problems. From the vantage-point of web development, mobile development seems very verbose and low-level.

Yet mobile development is moving in an increasingly abstract direction. Apple’s new Swift demonstrates a recognition of higher-level programming on iOS. And even before Swift, RubyMotion on iOS (and now Android) provided native mobile implementations of Ruby, spawning a large ecosystem of frameworks.

Scala allows high-level programming in Android. And it turns out that Swift is very similar to Scala.

Let’s take a brief look at what Scala offers for Android.

Lambda functions

When I first came to Android from Ruby, I was very confused by the lack of lambda functions in Java. I soon learned that Java typically uses anonymous classes for Runnables, Listeners, and Callbacks:

button.setOnClickListener(new View.OnClickListener {
  def onClick(v:View) {
    button.setText("Clicked");
  }
});

new Handler().post(new Runnable() {
  @Override
  public void run() {
    button.setText:
  }
});

This is pretty clunky. Java 8 has lambdas, but Android doesn’t support them without effort. How do anonymous functions look in Scala?

button.onClick(button.setText("Clicked"))
runOnUiThread(button.setText("Clicked"))

And a 5-liner becomes a one-liner. These examples depend on the Scaloid framework, but it’s not hard to implement similar functionality yourself. Since functions are first-class, you can pass them around.

Imagine you had a custom Seekbar control with a TextView indicating the current progress. You could pass your control a function from the progress integer to the fully formatted text. So 2 would become “Width 2 cm.” This way you have full flexibility over the text and you can use this same control with multiple units or any arbitrary suffixes or prefixes.

customSeekbarWithText.textFunction = {
  text => s"Width: $text cm." // the s before the quotes make it string interpolation
}

Less Boilerplate

public class LoginActivity extends Activity  {
  Button loginButton;
  EditText username, password;

  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    loginButton = (Button)findViewById(R.id.loginButton);
    username = (EditText)findViewById(R.id.username);
    password = (EditText)findViewById(R.id.password);

    loginButton.setOnClickListener(new View.OnClickListener() {
      @Override
      public void onClick(View v) {
        // ... validate form
        Intent intent = new Intent(this, AnotherActivity.class);
        startActivity(intent);
      }
    }
  }
}
class LoginActivity extends SActivity with TypedFindView  {
  private lazy val loginButton = findView(TR.loginButton)
  private lazy val email = findView(TR.email)
  private lazy val password = findView(TR.password)

  onCreate {
    setContentView(R.layout.main)

    loginButton.onClick(startActivity[WelcomeActivity])
  }
}

I’m not going to explain how this all works, because this is the post where I’m trying to convince you to care, not the post where I get down into the nitty-gritty. For now we’ll just say that this conciseness depends on android-sdk-plugin and Scaloid doing some stuff behind the scenes to thin out all that code. When using Scala, you will also have the ability to be concise in your own code.

Let’s break down what is going on here:

  • We are using the Scaloid framework: import org.scaloid.common._
  • SActivity is Scaloid’s base activity trait. A trait is like a more powerful interface.
  • TypedFindVew gives us the findView method, from android-sdk-plugin. TR is the generated typed resources created by the plugin, so there is no need to constantly cast views to their correct type
  • lazy means that the variable isn’t evaluated until it’s used
  • val is an immutable variable
  • onCreate is Scaloid’s syntactic sugar
  • onClick is Scaloid monkey-patching a method for Views with an implicit conversion. Don’t worry about understanding implicit conversions right now.
  • startActivity is a generic method that shortens Activity boilerplate

Asynchronous logic

Scala’s Future is an abstraction for asynchronous logic that is powerful and concise.

On Android, asynchronous programming typically uses AsyncTask. An asynchronous operation will be a subclass of AsyncTask with methods for callbacks. AsyncTasks have the same problem of all callback approaches: they are poorly composable, which means it’s hard to combine them or order them.

For example, say you want multiple sequential HTTP requests. Traditionally in Android, you would use an AsyncTask, and then fire off the next request in the onPostExecute callback. This approach nests your logic. There are frameworks like Bolts that let you write sequential tasks, but Scala comes with Futures out of the box.

A Future is a type that represents an asynchronous operation that is either successful or a failure. A Promise is a container for a future that can be manually completed or failed. A single Future might look like this:

val userFuture = User.findByEmail("a@b.com")

userFuture.onSuccess {
  case user => Log.v(LOG_TAG, user.id.toString)
}
userFuture.onFailure {
  case e => Log.v(LOG_TAG, e.getMessage)
}

So far, all we have is a success callback. But Futures really shine when you have multiple asynchronous operations.

Sequential asynchronous operations

If you wrap your HTTP requests in Promises and have them return Futures, then you can can execute them in sequence. Don’t worry about the Promise part now, for now we will just look at the Future side:


for {
  user <- User.findByEmail("a@b.com") // findByEmail returns a Future
  posts <- user.findPosts()
} yield {
  Log.v(LOG_TAG, s"User: ${user.id}, first post: ${posts(0).title}")
}

The for isn’t a “for loop,” it’s a for comprehension, which is Scala’s syntactic sugar for chaining operations.

Parallel asynchronous operations

What if instead of running Futures in sequence, you want to run them in parallel and do something when they are all done? It’s easy to run two AsyncTasks in parallel, but doing something when they are both done would be annoying, because each callback would have check whether the other callback was completed using shared mutable state.

What we really need is a tool that will take a bunch of Futures and then notifies us when they are all done.

// Transform an array of posts into a sequence of post-deletion requests
// post.delete() returns a Future that completes when the request returns
val deletions: Seq[Future[Int]] = posts.map(_.delete()).toSeq

// assuming all deletions succeed, you will get a Seq of post id's of the deleted posts
Future.sequence(deletions) onSuccess {
  case d: Seq[Int] => Log.v(LOG_TAG, d.mkString(", "))
}

What the heck is a Seq[Future[Int]]? It’s just what it sound like: a Seq of Futures of Ints, or a sequence of operations that will give you an Int in the future. Future.sequence takes a Seq[Future[Int]] and gives you a Future[Seq[Int]]. It gives you a Future of the Seq formed by the results of all the Futures that you pass it. It’s really nice how Scala gives you abstractions like this for really common asynchronous tasks.

I know I’m glossing over a few things here, like handling failure of Futures (easy), and canceling Futures (Scala’s standard Future cannot be canceled, but there are other implementations like Twitter’s that can be). I’m also not going to write out how these examples would look with AsyncTask because it would be long and I have much better stuff to do with my life.

Threading

You might be wondering, what thread are these Futures running on to avoid blocking? Futures in Scala run in an implicit ExecutionContext. Scala has a concept of “implicit” parameters, which are defined with the implicit keyword and don’t have to be manually passed in. Implicits get pretty complicated, but the main thing you need is an implicit ExecutionContext in scope that Futures will grab. Scala’s standard ExecutionContext is usually imported like this: import ExecutionContext.Implicits.global, but we will use Android’s ThreadPoolExecutor instead.

// this needs to be in scope somewhere or imported
implicit val execContext = ExecutionContext.fromExecutor(AsyncTask.THREAD_POOL_EXECUTOR)

Future { // this Future grabs the implicit execContext and uses it
  doSomethingInThreadPool()
} onSuccess {
  case result => runOnUiThread( updateUi() )
}

// equivalent to passing in an explicit parameter like this:
val execContext = ExecutionContext.fromExecutor(AsyncTask.THREAD_POOL_EXECUTOR)

Future {
  doSomethingInThreadPool()
}(execContext) onSuccess {
  case result => runOnUiThread( updateUi() )
}

Use a Future to jump onto your thread pool, and then Scaloid’s runOnUiThread to jump back onto the main/UI thread so you can update the UI. With Macroid, there are extensions to Futures like onSuccessUi which runs the success callback on the UI thread:

implicit val execContext = ExecutionContext.fromExecutor(AsyncTask.THREAD_POOL_EXECUTOR)

Future {
  doSomethingInThreadPool()
} onSuccessUi {
  case result => updateUi()
}

Resources

These are only a few of the goodies that Scala gives you on Android. Here are some additional resources for Scala on Android:

Jun 24