Scala Programming – A Skeptic’s Journey

Skedulo prides itself on designing and building an exceptional product for our customers. Meet Dean Povey, Team Lead for Backend Development at Skedulo, as he discusses his experiences using Scala.

Like many developers, I cut my teeth on C/C++ and Java. I’ve written a lot of working software, and the vast majority of it was imperative or object-oriented. Sure I liked lambdas, and I had seen the difference that making stuff immutable had done for my multi-threaded code. But for me, functional programming seemed to be something for self-absorbed purists more concerned with the beauty of their code, and not something that I needed to help me get my job done.

So when I started my first job in Scala, I brought healthy skepticism. The syntax seemed nice – it had a lot of syntactic sugar like case classes, operator overloading, pattern matching and type inference. But I didn’t foresee it changing the way I developed software. So here I am twenty-something months later, and I wanted to ask myself a few questions about my journey and where I see myself going with Scala in the future.

While Scala has much to offer that I won’t cover (e.g., type classes and functional programming constructs for managing side-effects), I think it is useful to consider what are those things that are most accessible to developers coming from a Java/C++ ecosystem.

Where Did all the Bugs Go?

About six months in, I was stumped. I couldn’t work out how we were not getting the level of bug reports on our backend systems that I had seen before. I mean we had acceptable unit and integration test coverage, but not much better than I had on other projects. I should have been seeing a reasonable amount of coding errors popping up, but nope. The odd logic error trickled in, but the usual steady flow of bugs was nowhere to be seen. Was no one using our system?

After deciding that this was not the case, I sat down and thought about it. I realized that many of the bugs I was no longer seeing were usually the result of simple programming errors:

• Null pointer exceptions.
• Basic type errors – like passing unchecked values to functions, or swapping the order of parameters of the same type.
• Concurrency issues – shared mutable state leading to race conditions and other errors in concurrent code.
• Uncaught exceptions.

In the production Java system I most recently worked on, these accounted for about 50% of the errors in our codebase (a number I just made up but seems about right). A significant amount of our issues weren’t business logic bugs, but simple programming errors that could be avoided. Was Scala just better at some of this stuff?

No More NPEs!

One study of 712 Open Source Software Projects (Mining Frequent Bug-Fix Code Changes) found that about 5% of bug fixes were to address issues with missing null checks. Furthermore, this is by far the most common repeated error, appearing in nearly 50% of the projects they surveyed. In my Java project, we had at least this number of similar errors despite using static analysis tools and annotating code with @Nonnull and @Nullable annotations and generally having good unit test coverage. It’s just too easy to screw up, and there is too much legacy code and libraries that are not null safe to deal with entirely.

Furthermore, an awful lot of your code ends up being writing checks for null. A study (Tracking Null Checks in Open-Source Java Systems) determined that something like 35% of conditionals in the code they surveyed included null checks.

This bug does not exist in idiomatic Scala (or it does not exist to a first approximation). You don’t allow instantiation of null instances of your types, and you use the Option type instead. By absolutely overwhelming convention in Scala, if it’s not Option, then it will never be null. This seemingly small difference is incredibly liberating. I recently had to review some Java code, and I spent an inordinate amount of time tracing through multiple API layers checking for the null safety of various functions and parameters. If you are considering a move from Java or C++ to Scala, then do it for no other reason than never having to write if (x == null) in your code again.

Type Safety

Once you get exposed to the benefits of type safety through experiences like Option, you quickly become hooked. When the language supports static typing in a way that is a joy to use, you realize that this isn’t a burden on you, but instead a bunch of tests on your code run by the compiler and not unit tests you have to write. A good example of this is using type-safe constructors to ensure that you can produce instances of types which are guaranteed to have certain invariants. Much like the Option type enables us to enforce the invariant that any instance we receive in a Scala program can never be null, so too can other types of invariant be enforced. In Java, we would do this by making a private constructor and a factory method that would throw an exception on failure. However, in Scala, we would prefer to use Option or Either (which allows a Left branch for reporting errors and a Right branch for reporting success) to wrap the resulting type and have the caller deal with it explicitly.

At first glance, this seems unwieldy. But when you realize the power of the other machinery Scala provides such as “for comprehensions” for chaining these checks together, or you become aware of the Applicative Validated type from the cats library which allows you to collapse multiple wrapped values into a single combined valid (or invalid) value; it changes your approach to how you build your programs. Suddenly when you see a variable of that type in the guts of your program, you don’t have to wonder (or write code to check) if it’s valid or not, since you have already guaranteed it is by virtue of the type safe constructor.

For lighter weight types such as String, Scala provides a simple way of wrapping that is (mostly) free at runtime.

class EmailAddress(val value: String) extends AnyVal
object EmailAddress {
    def apply(value: String): Either[Exception, EmailAddress] = {
      if (validEmailAddress(value)) {
        Right(new EmailAddress(value))
      } else {
        Left(new IllegalArgumentException(s"Invalid email address: $value"))
      }
    }
}

def sendEmail(address: EmailAddress, msg: String): Unit = ???

EmailAddress("[email protected]") match {
  case Right(email) => sendEmail(email, msg)
  case Left(err)    => logError("Could not send email", err)
}

To the compiler, this will appear like a String (unless we do certain things that require reflection at runtime).

Scala’s library ecosystem also provides many sophisticated mechanisms for making your code more type-safe including Refined types and Heterogeneous lists.

Immutability

If you are doing any form of multithreaded or concurrent programming, you quickly learn the advantages of immutable data structures. They prevent race conditions occurring as modifications to shared state are no longer possible. In Java, we can mark variables as final, but we have to take careful steps to make sure the instances that these variables are bound to are immutable as well (manually copy the values when returning from getters for example). In Scala, we have the general purpose case class which provides this for us for data values. Also, Scala has a range of collections classes which provide efficient implementations of immutable data structures such as lists, sets, vectors, and hash tables.

Handling Exceptions

If you are wrapping Java libraries in Scala, you will occasionally still be having to deal with exceptions. Scala provides the Try type that allows us to convert an exception to a type we can more usefully deal with:

val t = Try {
   throw new RuntimeException("Boom!")
}

At first glance, this doesn’t seem to provide us much over the try/catch approach we are familiar with in Java. But Try also allows us to deal with our errors, or convert them into other types to make them easier to manage. For example, if we turn all our results into an Either using the cats toEither extension method we can use a “for comprehension” to gather all the data and only run the result if all of the items succeed:

for {
  email <- EmailAddress(address)
  msg   <- Try { horribleJavaLibraryMethodThatThrowsExceptions }.toEither
} yield {
  sendEmail(email, msg)
}

Remembering that our EmailAddress apply method returned an Either[Exception, EmailAddress]. Because of the way Either is defined we will only to continue evaluating the next line if this succeeds and the method returns a Right(EmailAddress). Furthermore, we will only send an email if both the values are valid. No more nested if statements to screw up.

Was it all Sweetness and Light?

Well no, Scala has its rough spots. It has no decent support for union types or enums. The Enumeration type is broken to a great extent. You end up needing to use a sealed trait which is not awful but boilerplate heavy. The enumeratum library provides some helpful macros to assist you.

Compile times are a common criticism, but for the most part, this doesn’t affect you in practice. Mostly code is incrementally compiled, and you don’t notice much lag.

IDE support is pretty annoying. You get lots of situations where your code has squiggly red lines in IntelliJ, and you have to go into SBT to see if it should have compiled anyway – this is due mainly to the support for Scala macros in libraries which have to be explicitly added to the plugin. Refactoring is there but limited to what you get compared to Java (admittedly you are very spoiled if you use IntelliJ for refactoring Java code).

Type inference makes the code hard to read sometimes. Often you are looking at a “for comprehension,” and you don’t know what the types are of the individual components. IntelliJ does provide a helpful inspection that will tell you any inferred type which mostly works, but in general, I find this painful trying to read code that I didn’t write.

Sometimes you find yourself fighting the compiler more than you would like. If you haven’t worked out exactly which type or implicit you need, it can take a while to untangle it. “For comprehensions” tend to confuse you with this until you get the hang of them. The good news is that once it compiles it almost always works how you expected.

And What’s Your Feeling About Going Back to Java/C++ Now?

I used to see talks about functional programming and type safety where people would complain that they couldn’t use <insert imperative language here> because it didn’t provide all the guarantees with which they were familiar. I always hated this attitude, because you should always strive to be able to solve problems well in any language. However, I will confess that recently digging into some Java library and trying to make it work without having the same compile-time safety guarantees I have become used to, I gave up and rewrote it in Scala.

There is a thing I have always believed about developing software, which is that speed comes from confidence. When you write one line of code it is usually easy to convince yourself that that code is correct, and you can feel more productive if you write that one line without bothering to write tests. But as your code base grows you are always concerned with whether your change might break some other bit of code. If you have good test coverage, you can run the tests to be sure and not overthink it. If you don’t, you end up staring at a bunch of code, trying to remember how it works before deciding if you broke it or not.

The language features of Scala such as type safety and immutability extend my comfort zone, giving me confidence that my code will work. The best thing is all I have to do to get this added benefit is to make it compile!

In short, it’s not just a better Java (although you can use it that way) – if you embrace it, it’s a better way of writing code altogether.