Avro4s Fork

This repository is an actively maintained fork of the original avro4s. Our fork focuses on supporting modern requirements—namely Java 21, Scala 3, and Nix—and may include changes that are not backward compatible with earlier Java or Scala versions.

Disclaimer

You are free to copy any features or improvements from this fork back into the original avro4s project. Please note that this repository is optimized for our own production needs, so it may diverge substantially from the original.

Why This Fork?

We created this fork to meet specific enterprise and client requirements that demand:

Java 21 Support (production-ready)
Scala 3 Support (production-ready)
Nix Support for reproducible builds and environments

These updates may introduce breaking changes for projects using older Java or Scala versions.

Usage

To integrate Avro4s Fork into your Scala project, add the following dependency to your build.sbt:

libraryDependencies += "com.natural-transformation" %% "avro4s-core" % "5.1.0"

If you need to work with cats, you also need to add avro4s-cats to your dependencies.

Example

import com.sksamuel.avro4s._
import java.io.ByteArrayOutputStream
import java.io.OutputStream
import java.nio.ByteBuffer
import org.apache.avro.Schema
import org.apache.avro.util.ByteBufferInputStream
import scala.jdk.CollectionConverters.SeqHasAsJava
import scala.util.Using

object AvroUtil {

  def toBin[T: Encoder: SchemaFor](o: T): Array[Byte] =
    Using.resource(new ByteArrayOutputStream) { output =>
      Using.resource(AvroOutputStream.binary[T].to(output).build()) { avro =>
        avro.write(o)
      }
      output.toByteArray
    }

  def fromBin[T: Decoder: SchemaFor](bytes: Array[Byte], writerSchema: Schema): T =
    Using.resource(
      AvroInputStream.binary[T].from(bytes).build(writerSchema, SchemaFor[T].schema)
    ) { is =>
      is.iterator.next() // can throw
    }

  def fromBin[T: Decoder: SchemaFor](bytes: Array[Byte]): T =
    Using.resource(
      AvroInputStream.binary[T].from(bytes).build(SchemaFor[T].schema, SchemaFor[T].schema)
    ) { is =>
      is.iterator.next() // can throw
    }

  def writeByteBuffer[T: Encoder: SchemaFor](o: T, buf: ByteBuffer): Unit =
    Using.resource(AvroOutputStream.binary[T].to(new ByteBufferBackedOutputStream(buf)).build()) { avro =>
      avro.write(o) // can throw
    }

  def readByteBuffer[T: Decoder: SchemaFor](buf: ByteBuffer, writerSchema: Schema): T =
    Using.resource(
      AvroInputStream
        .binary[T]
        .from(new ByteBufferInputStream(List(buf).asJava))
        .build(writerSchema, SchemaFor[T].schema)
    ) { is =>
      is.iterator.next() // can throw
    }

  private class ByteBufferBackedOutputStream(buf: ByteBuffer) extends OutputStream {
    def write(b: Int): Unit =
      buf.put(b.toByte) // can throw

    override def write(bytes: Array[Byte], off: Int, len: Int): Unit =
      buf.put(bytes, off, len) // can throw

  }

}

import cats.data.{NonEmptyList, Validated, ValidatedNel}
import com.sksamuel.avro4s.SchemaFor
import com.sksamuel.avro4s.cats.{given, _}
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpecLike

class AvroUtilSpec extends AnyWordSpecLike with Matchers {

  "AvroUtil" should {
    "serialize and deserialize a case class with a Validated field" in {
      case class CatsValidatedTest(f1: Validated[String, Int], f2: Validated[String, Long])

      val schemaFor: SchemaFor[CatsValidatedTest] = summon[SchemaFor[CatsValidatedTest]]
      println(s"schema for CatsValidatedTest is ${schemaFor.schema.toString(true)}")

      val serialized    = AvroUtil.toBin(CatsValidatedTest(Validated.Invalid("error1"), Validated.Valid(1L)))
      val derserialized = AvroUtil.fromBin[CatsValidatedTest](serialized)

      derserialized shouldEqual CatsValidatedTest(Validated.Invalid("error1"), Validated.Valid(1L))

    }

    "serialize and deserialize a case class with a ValidatedNel field" in {
      case class CatsValidatedNelTest(errorsOrNum: ValidatedNel[String, Int])
      val schemaFor: SchemaFor[CatsValidatedNelTest] = summon[SchemaFor[CatsValidatedNelTest]]
      println(s"schema for CatsValidatedNelTest is ${schemaFor.schema.toString(true)}")

      val serialized    = AvroUtil.toBin(CatsValidatedNelTest(Validated.Invalid(NonEmptyList.one("error1"))))
      val derserialized = AvroUtil.fromBin[CatsValidatedNelTest](serialized)

      derserialized shouldEqual CatsValidatedNelTest(Validated.Invalid(NonEmptyList.one("error1")))

    }
  }

}

Note: Although the library is published under the group ID com.natural-transformation, the package imports remain the same as the original Avro4s library (com.sksamuel.avro4s). This ensures backward compatibility with existing codebases, requiring no changes to import statements.

Commercial Support

If your company is considering this fork for production, we can assist with both adoption and migration. Because each use case varies, pricing is arranged on an individual basis. For inquiries, please contact Natural Transformation.

Reproducibility and Security

We provide a strong guarantee of reproducibility and security by utilizing Nix for both local development and GitHub Actions CI. This approach ensures that every environment, from your local machine to our CI pipelines, uses exactly the same dependencies. As a result, we achieve the highest levels of reproducibility and security, eliminating discrepancies across different environments.

Contributing and Issue Reporting

We welcome all contributions—bug reports, feature requests, and pull requests. Before contributing, please:

Check existing issues and pull requests to avoid duplication
Ensure any changes align with this project’s focus on Java 21, Scala 3, and Nix support

We appreciate your feedback and help in making this fork even better!

natural-transformation / avro4s 5.1.0