Scala Native for cross-platform binaries

🦀🛠 Rustaceans have been hard at work [rewriting classic unix tools for the modern age](https://zaiste.net/posts/shell-commands-rust/). With its static type system and committment to creating zero-cost abstractions, rust is a great tool for publishing command line apps~ Now Scala can too! We'll be looking at reimplimenting a gnu util [`csplit`]([https://man7.org/linux/man-pages/man1/csplit.1.html](https://man7.org/linux/man-pages/man1/csplit.1.html)) in Scala as a project called [`resplit`](https://github.com/AesaKamar/resplit)! You can follow along here with in-code examples, or [head directly to GitHub](https://github.com/AesaKamar/resplit) ## 🔧 How do I get Scala to build a binary and ditch my JVM? All you need is a simple `build.sbt` file that looks something like this: <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 1 name := "resplit" 2 3 scalaVersion := "3.2.2" 4 5 // This tells our project 6 enablePlugins(ScalaNativePlugin) 7 8 // defaults set with common options shown 9 nativeConfig ~= { configuration => 10 configuration 11 // https://scala-native.org/en/stable/user/sbt.html#garbage-collectors 12 .withLTO(scala.scalanative.build.LTO.none) 13 // https://scala-native.org/en/stable/user/sbt.html#compilation-modes 14 .withMode(scala.scalanative.build.Mode.debug) 15 // https://scala-native.org/en/stable/user/sbt.html#garbage-collectors 16 .withGC(scala.scalanative.build.GC.commix) 17 } 18 19 libraryDependencies ++= Seq( 20 "co.fs2" %%% "fs2-core" % "3.6.1", 21 "co.fs2" %%% "fs2-io" % "3.6.1", 22 "org.typelevel" %%% "cats-effect" % "3.4.8", 23 "org.typelevel" %%% "cats-core" % "2.9.0", 24 "com.github.scopt" %%% "scopt" % "4.1.0", 25 "com.lihaoyi" %%% "pprint" % "0.8.1", 26 "org.typelevel" %%% "munit-cats-effect" % "2.0.0-M3" % Test 27 ) 28 29 // I often declare and do not use things while developing 30 tpolecatExcludeOptions ++= ScalacOptions.warnUnusedOptions 31 32 // set to Debug for compilation details (Info is default) 33 logLevel := Level.Info 34 </pre> This `enablePlugins(ScalaNativePlugin` directive will give you access to the `sbt nativeLink` task which will produce a precompiled binary for you! ## 📜🪓 A program to split large files by a regex #### Given this Input file: ``` cat1 cat2 cat3 dog1 cat4 cat5 cat6 dog2 cat7 cat8 cat9 ``` #### When I run ``` > cat testfile | resplit '(dog)\d' '$1' ``` #### Then I should get these 3 output files `000_` ``` cat1 cat2 cat3 ``` `001_dog` ``` dog1 cat4 cat5 cat6 ``` `002_dog` ``` dog2 cat7 cat8 cat9 ``` Here is a sneak-peek at the final product Streaming "The Adventures of Sherlock Holmes" with `wget` and spliting by chapter regex into unique files <img src="https://user-images.githubusercontent.com/4334491/224520721-9500f632-12a6-41f0-859a-d5f84b4d6b01.gif"> ## 🧑‍💻 Making a user-friendly cli ### Our `Main` file This is pretty small, we use `IOApp` from `cats.effect` to let us hook into the `run` method <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 1 import cats.effect.{ExitCode, IO, IOApp} 2 import scopt.OParser 3 4 object Main extends IOApp { 5 6 def run(args: List[String]): IO[ExitCode] = 7 OParser 8 .parse( 9 parser = InputArgs.commandLineParser, 10 args = args, 11 init = InputArgs() 12 ) 13 .fold(ifEmpty = IO(ExitCode.Error)) { parsedInputs => 14 Resplit 15 .resplit(args = parsedInputs) 16 .compile 17 .drain 18 .map(_ => ExitCode.Success) 19 } 20 } </pre> We want to get this nice user-friendly help message... -- but how do we set up all of that without spending all of our time mired in command line parsing? 😵‍💫 ```sh > resplit --help Usage: resplit [options] regexToMatch [regexToSub] Splits a file based on a regex. split files will be prefixed by digits, and named by the contents of the matched regular expression. Outputs names of files created to stdout regexToMatch A regular expression to split the file on regexToSub A regular expression substitution expression to use to format the output filenames -n, --digits <value> Number of digits to left-pad the split filenames with -d, --directory <value> Directory to write the split files into -f, --file <value> Read from the specified file instead of stdin --suppressMatched Include the line that matched the regexMatch arg as the first line in the split files -s, --quiet Quiet --help prints this usage text ``` ### `scopt` for command line argument parsing We set up a simple `case class` to hold onto all of the inputs we'd like to "gather" from the user <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 1 import scopt.{OParser, OParserBuilder} 2 3 import java.io.File 4 import scala.util.matching.Regex 5 6 case class InputArgs( 7 regexToMatch: Regex = "".r, 8 regexToSub: Option[String] = None, 9 filenamePaddingDigits: Int = 3, 10 outputDirectory: Option[File] = None, 11 inputFileInsteadOfStdin: Option[File] = None, 12 suppressMatched: Boolean = false, 13 silentMode: Boolean = false 14 ) </pre> We use `scopt`'s easy-to-use DSL to specify how to gather all of the arguments, options, and flags <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 16 /** Use [[OParserBuilder]] API to specify how to take input args from the 17 * command line and make them user friendly 18 */ 19 object InputArgs { 20 import scala.util.chaining._ 21 private val builder: OParserBuilder[InputArgs] = OParser.builder[InputArgs] 22 23 val commandLineParser: OParser[Unit, InputArgs] = { 24 import builder._ 25 OParser.sequence( 26 programName("resplit"), 27 note( 28 """Splits input based on a regex. Split files will be prefixed by digits, 29 |and named by the contents of the matched regular expression. 30 | 31 |Outputs names of files created to stdout 32 |""".stripMargin 33 ), 34 arg[String]("regexToMatch") 35 .required() 36 .text("A regular expression to split the file on ") 37 .action((arg, conf) => conf.copy(regexToMatch = arg.r)), 38 arg[Option[String]]("regexToSub") 39 .optional() 40 .text( 41 "A regular expression substitution expression to use to format the output filenames" 42 ) 43 .action((arg, conf) => conf.copy(regexToSub = arg)), 44 opt[Int]('n', "digits") 45 .optional() 46 .text("Number of digits to left-pad the split filenames with") 47 .validate(n => 48 (n > 0 && n <= 128) 49 .pipe(Either.cond(_, (), "Not a valid number of digits")) 50 ) 51 .action((arg, conf) => conf.copy(filenamePaddingDigits = arg)), 52 opt[File]('d', "directory") 53 .text("Directory to write the split files into") 54 .optional() 55 .action((arg, conf) => conf.copy(outputDirectory = Some(arg))), 56 opt[File]('f', "file") 57 .text("Read from the specified file instead of stdin") 58 .optional() 59 .validate(_.isFile.pipe(Either.cond(_, (), "Not a valid file"))) 60 .action((arg, conf) => conf.copy(inputFileInsteadOfStdin = Some(arg))), 61 opt[Unit]("suppressMatched") 62 .optional() 63 .text( 64 "Include the line that matched the regexMatch arg as the first line in the split files" 65 ) 66 .action((_, conf) => conf.copy(suppressMatched = true)), 67 opt[Unit]('s', "quiet") 68 .optional() 69 .text("Quiet") 70 .action((_, conf) => conf.copy(silentMode = true)), 71 help("help").text("prints this usage text") 72 ) 73 } 74 } </pre> We can even use it to bind in a `--help` option 😉 ## 📂 Setting up the filesystem Before we can write to an output directory, we've got to make sure it exists! <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 17 Stream 18 .eval(createDirectoryOrDoNothing(args.outputDirectory)) </pre> We use `Stream.eval` to run an effect, `IO[SomeType]` into a `Stream[IO, SomeType]` <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 48 def createDirectoryOrDoNothing(outputDirectory: Option[File]): IO[Unit] = 49 outputDirectory 50 .map(_.getPath) 51 .map(Path.apply) 52 .fold(ifEmpty = IO.unit) { dir => 53 fs2.io.file 54 .Files[IO] 55 .exists(dir) 56 .flatMap { exists => 57 if (exists) IO.unit 58 else 59 fs2.io.file 60 .Files[IO] 61 .createDirectory(dir) 62 } 63 } </pre> ## 📥 Streaming the inputs from either `stdin` or a file ### `fs2` stands for streaming We leverage `fs2`'s powerful and succint API for helping us deal with the complexities of streaming data <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 16 def resplit(args: InputArgs): fs2.Stream[IO, (Path, Chunk[String])] = 17 Stream 18 .eval(createDirectoryOrDoNothing(args.outputDirectory)) 19 .productR { 20 args.inputFileInsteadOfStdin 21 .fold(ifEmpty = fs2.io.stdin[IO](bufSize = 1024)) { 22 _.getPath 23 .pipe(Path.apply) 24 .pipe(fs2.io.file.Files[IO].readAll) 25 } 26 } </pre> The `resplit` function will be called by out `Main` class when the cli app is invoked Firstly, it checks the arguments if an inputFile is passed in to use instead of stdin (We've luckily leveraged `scopt` previously to parse it in under the `-f` or `--file` option, so we don't need to specify the details here ☺️) Since both a file stream and `stdin` are bytestreams all the same, we'll need to pipe them `through` 2 more functions to get them into a `Stream[IO, String]` By the way, `productR` is equivalent to `flatMap`, where we discard the argument to the lambda ### More well-typed helper functions <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 24 .through { inputStream => 25 if (args.suppressMatched) 26 inputStream.split(thisLine => args.regexToMatch.matches(thisLine)) 27 else 28 splitInclusive(inputStream)(thisLine => 29 args.regexToMatch.matches(thisLine) 30 ) 31 } </pre> We decide if the matched line should be included in the file, or disincluded Here's the spec of how that `splitInclusive` function should behave <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 12 test("splitInclusive") { 13 val value: fs2.Stream[Pure, Int] = 14 fs2.Stream.range(0, 10) 15 16 val predicate: Int => Boolean = 17 _ % 4 == 0 18 19 val split: fs2.Stream[Pure, Chunk[Int]] = 20 Resplit.splitInclusive(value)(predicate) 21 22 val allBeginWithPredicate: Boolean = split 23 .forall(_.head.forall(predicate)) 24 .compile 25 .toList 26 .head 27 28 assert(allBeginWithPredicate) 29 } </pre> ## 📤 Streaming the outputs to a file ### Pulling the first line out from each file <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 36 .zipWithIndex 37 .map { (chunkOfLines, chunkNumber) => 38 ( 39 inferPathFromFirstMatchedLineOfChunk(args, chunkOfLines, chunkNumber), 40 chunkOfLines 41 ) 42 } </pre> Hopefully these strong type signatures and declarative names are mostly self-explanatory! <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 85 // TODO this is technically impure because is has a console printing effect in it 86 def inferPathFromFirstMatchedLineOfChunk( 87 args: InputArgs, 88 c: Chunk[String], 89 i: Long 90 ): Path = { 91 val fileContext: String = c.head 92 .flatMap(args.regexToMatch.findFirstIn(_)) 93 .map { matchedChars => 94 args.regexToSub.fold(ifEmpty = matchedChars) { providedRegexSub => 95 try 96 matchedChars.replaceFirst(args.regexToMatch.regex, providedRegexSub) 97 catch 98 case _ => 99 if (!args.silentMode) 100 (s"invalid regex: on lt;/span>matchedChars " + 101 s"for lt;/span>{args.regexToMatch.regex} " + 102 s"with substitution lt;/span>providedRegexSub") 103 .pipe(Console.err.println) 104 "" 105 } 106 } 107 .getOrElse("") 108 val iii: String = leftPad(i.toString, args.filenamePaddingDigits, '0') 109 args.outputDirectory 110 .fold(ifEmpty = Path.apply(s"lt;/span>{iii}_lt;/span>fileContext")) { dir => 111 Path.apply(s"lt;/span>dir/lt;/span>{iii}_lt;/span>fileContext") 112 } 113 } </pre> ### Actually writing the file Lastly, here's the function responsible for writing the file, making heavy use of `fs2` <pre style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 44 .evalTap { (path, chunkOfLines) => 45 writeChunkToPathAndPrint(chunkOfLines, path, args.silentMode) 46 } </pre> <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 65 def writeChunkToPathAndPrint( 66 c: Chunk[String], 67 path: Path, 68 isSilent: Boolean = false 69 ): IO[Unit] = 70 fs2.Stream 71 .chunk(c) 72 .intersperse("\n") 73 .append(fs2.Stream.emit("\n")) 74 .through(fs2.text.utf8.encode) 75 .through(fs2.io.file.Files[IO].writeAll(path)(_)) 76 .compile 77 .drain 78 .flatTap { _ => 79 if (isSilent) IO.unit 80 else IO(Console.println(s"lt;/span>path\t")) 81 } </pre> Similarly to how we needed to pipe our input `through` `text.utf8.decode` and `text.lines`, we need to first `intersperse` some newline characters back in and then pipe it `through` `text.utf8.encode` If you've been wondering what `.compile.drain` are doing, that's tjust the way to instruct `fs2` to finalize this stream pipeline to get ready to run as an `IO` effect 🏎 ### Final spec Here is a description of how this `resplit` funciton should behave <pre class="intellij" style="color:rgb(0,0,0);font-weight:normal;font-style:normal;font-family:monospace;"> 56 test("resplit") { 57 for { 58 output <- Resplit 59 .resplit( 60 InputArgs( 61 regexToMatch = "dog\\d".r, 62 regexToSub = None, 63 filenamePaddingDigits = 3, 64 outputDirectory = Some(new File("tmp/")), 65 inputFileInsteadOfStdin = Some(new File("testfile")), 66 suppressMatched = false, 67 silentMode = true 68 ) 69 ) 70 .compile 71 .toList 72 expectedOutput = List( 73 (Path("tmp/000_"), Chunk("cat1", "cat2", "cat3")), 74 (Path("tmp/001_dog1"), Chunk("dog1", "cat4", "cat5", "cat6")), 75 (Path("tmp/002_dog2"), Chunk("dog2", "cat7", "cat8", "cat9")) 76 ) 77 } yield { 78 79 assertEquals(output, expectedOutput) 80 } 81 } </pre> ## Conclusion Hopefully this has convinced you that using ScalaNative isn't so hard! It affords us the ability to use all the powerful typing and wonderful libraries we know and love in Scala and get the instant startup and small binary sizes on natively compiled applications. Happy CLI hacking! 🎉