File Storage For Scala
A common interface for doing basic file storage operations on multiple platforms
// Choose any you want to use libraryDependencies += "com.deliganli" %% "fstore4s-s3" % "0.1.0" libraryDependencies += "com.deliganli" %% "fstore4s-gcs" % "0.1.0" libraryDependencies += "com.deliganli" %% "fstore4s-local" % "0.1.0" // You can add the core library if you want to implement your own file store // Other modules above use core inherently libraryDependencies += "com.deliganli" %% "fstore4s-core" % "0.1.0"
Current supported file storage platforms are;
- Amazon Web Services (AWS) S3
- Google Cloud Storage
- Local file storage
The methods for manipulating data are same across all platforms.
store.uri(address) // Generates signed uri valid over configured time store.list(prefix) // Returns a stream with files under the prefix . .
Platform can be changed easily by using another instance of
S3Store.create[IO, SimpleDomainFile](config, credentialProvider) GCSStore.create[IO, SimpleDomainFile](config) LocalFileStore.create[IO, SimpleDomainFile](config)
create methods are just helpers, one can create the
FileStore directly by providing necessary underlying store instance
Unifies storage providers'
Iterator and other java streaming based functionality under
What is Address?
It is basically a case class that has an implementation for
FileCodec[T] is used for naming the file from structured fields and vice versa
SimpleDomainFile is a basic implementation that used across tests, simply it names the stored files with 3 properties; version, category, name. For example;
Address("test-dir", SimpleDomainFile("V1", "attachment", "some-file.txt"))
This structured data can be used to encode the file to the filestore with the filename of
Address class can be constructed from the filename as well so when we read filenames from the store directly we can have our structured data back
With this way we can provide
case class -> filename and
filename -> case class transformations that guarantees each unique case class corresponds to a unique file
This library is NOT
- Sophisticated scala wrapper for particular file storage to use all possible functionality
- Battle tested, production ready
This library might be useful at
- Using one of the cloud file stores for production and using local store for tests
- Quickly switch between file stores with minimal code change
- Streaming applications that makes use of fs2
This library is very crude at the moment, not even sure there would be a demand since this is an abstraction there will be the trade-offs between customizability and unification. If developed further, one probably will always find themselves deciding between providing optional configuration that may not work with other vendors and not allowing any vendor specific usage
The drawbacks of 'melting everything in one pot' approach may discourage users with higher volume and performance demands due to the lack of vendor-specific configuration. Though I tried to make everything streaming to make life easier for them as well.