I have been working at Skedulo for 6 months now as a scala backend engineer. This is my first experience working professionally with Scala (and functional programming). Therefore I would like to take some time to reflect on my experience over the past 6 months, as well as my thoughts on the language and ecosystem.

Where I Started

At the very end of my university life I stumbled upon Haskell, which really inspired me for how different and expressive it is. I then proceed to spend the next few months studying and tinkering with Haskell. By the time I started chasing other shiny things, I was comfortable with the basic concepts (up to monads).

When I started my new job, armed with my Haskell and Java experiences, Scala felt like the sibling I never met. For the first month or so I was focused on finding out what parts of Hasekll/Java Scala has inherited, and applying them as I go.

This post is mostly based on my experience working on a project that performs scheduling, job allocations, notifications and various other CRUD operations.

The Language

The Good

Scala is a pragmatic, expressive and powerful language - You can produce some readable and correct code easily with just a few sprinkles of functional style:

List(1, 2, 3, 4)
  .map(_ * 4)            // List(4, 8, 12, 16)
  .filter(_ % 8 == 0)    // List (8, 16)
  .map(_.toString)       // List("8", "16")
  .mkString(",")         // "8,16"

Though the example above is just a toy example, I was often surprised at how succinct Scala can be at encoding business logic (and invariants) due to the following language features:

• Case Classes - Clean syntax for constructing type-safe records representing business objects
• For Comprehensions - Boilerplate-free program control flow
• Sealed Traits - Encoding invariants in business logic and making sure all cases are handled (or explicitly ignored)
• Pattern Matching - Works very well with sealed traits and case classes for matching and extracting data

There are many other helpful features that are rare or not found in other mainstream languages such as implicits and type inference. On the ecosystem/standard library side, immutability by default and Option type instead of nulls makes reasoning about code simple. In general, Scala serves its purpose very, very well when you use it as a “Better Java”.

One of the practical benefits, as I have experienced first-hand, is that your code can be refactored and evolved very aggressively. For a codebase that was written by novice Scala developers making a MVP, there was certainly a lot of ugly code written in the name of tight deadlines. However, the strong static typing means I am able to evolve the code and be confident of its correctness. With just Immutability and no-nulls by default means that I already save who-knows-how-many hours compared to Java or Python.

The Bad

Type Inference

The Type Inference isn’t perfect - it mostly works until it breaks on more complex types (which normally involes higher-kinded types). There are often cases where IntelliJ would break but not scalac, which in turn breaks type inspection and autocomplete (not to mention code being highlighted red and make you doubt yourself). Here is an example of a code that scalac gladly compiles but IntelliJ chokes on:

Generally, just sprinkling some type annotations solves the problem. Though it certainly feels like spending half your paycheck on a full-course meal only to be told that you cannot have the dessert unless you do 20 push-ups and pay an extra 10 dollars.

The Bugs

Here is a simple example of a compiler bug I encountered while trying to destructure the tuple on the right side of a \/[SomeError, (Int, Int)]:

  sealed trait SomeError
  case class ErrorOne() extends SomeError
  case class ErrorTwo() extends SomeError

  // attempt 1: desugar the tuple of the either directly
  for {
    (first, second) <- (1, 1).right
  } yield first

  // [error] Main.scala: diverging implicit expansion for type scalaz.Monoid[A]
  // [error] starting with object stringInstance in trait StringInstances
  // [error]     (first, second) <- (1, 1).right
  //                                       ^

  // Huh? Ok Google tells me that diverging implicit normally means that we are trying to find an implicit
  // on a Nothing. 

  // attempt 2: Let's fix the type on the left side of \/

  for {
    (first, second) <- (1, 1).right[SomeError]
  } yield first

  // [error] Main.scala: could not find implicit value for parameter M: scalaz.Monoid[SomeError]
  // [error]     (first, second) <- (1, 1).right[SomeError]
  //                                       ^

  // Huh why is it trying to find the Monoid instance for SomeError?
  // Monoid instance shouldn't be involved?!

  // attempt 3: Let's bind the tuple to a name and destructure it on the next line

  for {
    result <- (1, 1).right
    (first, second) = result
  } yield first

  // it compiles now?! Wut?

So it turns out that there is some bug with for comprehension desugaring, and it has been around for a while. Yikes.

I probably only encountered 2~3 compiler bugs and they are mostly annoyances with ugly workarounds than correctness issues.

Type System Limitations

Maybe due to Scala’s focus on uniting OO and FP, encoding FP concepts are way more verbose than its Haskell counterpart. Two examples of this are Typeclasses (using implicits/context bounds) and Abstract Data Types (with sealed traits).

An example of this is attempting to use multiple-level monad transformers involving EitherT. Since Scala type constructors cannot be partially applied, You will need to employ Type Lambdas hacks to get it to work. which is both verbose, confusing and killing type inference.

This is quite scary for me as someone who is constantly learning and applying functional concepts at work, because it feels like if I try any advanced functional constructs, I am trading away maintainability, simplicity and productivity.

While I am being really critical in this section, I am by no means implying that the people involved in Scala tooling or compilers are incompetant or lazy. No other language has tried doing what Scala is doing, and the people working on Scala has done a great job. However it is a bit sad that some style of programming is much more painful (and probably not worth it because you are trading off conciseness and maintainability).

This is probably the only real fundamental issue with Scala that discourages me from exploring advanced functional techniques in Scala.

Tools and Ecosystem

The Good

Using Java libraries in Scala is a total breeze. I have never had any problem (and expectely so since Java-interoperability is one of the main selling points of Scala). In addition to that, there are an abundance of high quality libraries and frameworks you can choose to solve your problems in Scala, with many of them being backed commercially.

At work we use Play Framework as our backend, Slick as our SQL access layer, Akka for concurrent processing, ScalaTest / ScalaCheck for testing and Scalaz for functional-goodiness.

On the tooling front, Play Frameworks’s various SBT plugins makes the development experience relatively smooth. Due to incremental compilation, compile time is generally not a problem but could be improved (0 - 5 seconds typically).

I spend most of my coding time in Intellij with vim emulation (Ideavim). While I do not make use of many of its more advanced features, the navigation, refactoring, autocomplete and code analysis is invaluable for getting work done.

The Bad

Similar to the ecosystems of other statically typed functional languages, documentation seems to be quite weak, maybe because it is considered not necessary (which to a degree is true because you can figure most things out by just looking at the types). However, I do think this is gradually changing, as many newer projects have excellent documentation and examples.

Scala’s Current and Future

There are a lot of awesome projects in Scala that excites me:

• Scala.js seems like a very good option for frontend development.
• Dotty, the research project aiming to “try out new language concepts and compiler technologies for Scala” will likely power the next generation of Scala language in terms of new features, better type inference and faster compile times.
• There are work going on to compile Scala to run as native binaries. Along with Scala.js, this opens many new doors for Scala where it previously was not a practical choice (e.g. scripting or CLI tools, due to slow JVM startup time)
• Development in the Scala compiler itself, bringing better compile times and better scala-to-java interopt.

On Productivity, Correctness and Maintainability

Like most developers, I am constantly on the lookout for better ways to program. There are many facets as to what consitutes as “better” - Productivity? Expressiveness? Correctness? Maintainability? The relationship between these facets are very complex and the right balance for each person is different, hence the constant static vs dynamic typing debate.

For me personally it has definitely been a constant struggle between all of these factors. There are times I spend days making sure my code is “correct”, when simple due diligence and documentation can probably prevent mistakes made by future edits. (And that type-safe solution you spent working on? It is invalidated by a new business requirement). There is a little voice that constantly questions myself whether the next person that comes in will be able to understand my code and maintain it, which has been a major consideration when I am deciding how “type-safe” I want to be.

As for now, I have settled on being “Simply Correct”. This means to make full use of the language’s first class features to maximize Correctness and Productivity without sacrificing Maintainability. Beyond that, every extra bit of correctness or expressiveness constructs I implement will need to founded by real requirments.
It turns out YAGNI (“You Aint Gonna Need It”) principle applies nicely here too.

Please don’t leave behind a trail of single character variable names and undocumented mess because I should “just follow the types”.

Conclusion

Scala is a great language for Getting Things Done, whether you are fresh off the boat from Javaland or an experienced FP programmer.

At this moment, if I were to be asked to deliver a project ASAP while making it maintainable, Scala will be the language I choose for its pragmatism, expressiveness and comprehensive ecosystem. However, Scala is definitely far from perfect; The compiler bugs you will find when attempting to write advanced, type-safe code, as well some language design trade-offs means that it is siginificantly harder to reach the expressive level of other “purer” languages like Haskell.

Personally, I aim to explore Haskell more and attempt to write it for real business applications, so I can learn more about how to design and implement purely functional architecture. (instead of the functional-on-the-outside, imperative-on-the-inside design I always fallback to because Scala makes it easy)