An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.

aka.ms/hyperspace

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

Please review our contribution guide.

This repository contains symbolic links which don't work properly on Windows. To build this project on Windows, you can use our provided Git aliases to replace symbolic links with junctions.

$ git config --local include.path ../dev/.gitconfig
$ git replace-symlinks # replace symlinks with junctions
$ #git restore-symlinks # use this to restore symlinks if you need

You can use the built-in sbt shell in IntelliJ without any problems. However, the built-in "Build Project" command may not work. To fix the issue, go to Project Structure -> Project Settings -> Modules and follow these steps:

• Mark src/main/scala and src/main/scala-spark2 as "Sources" and src/test/scala and src/test/scala-spark2 as "Tests" for the spark2_4 module.
• Mark src/main/scala and src/main/scala-spark3 as "Sources" and src/test/scala and src/test/scala-spark3 as "Tests" for the spark3_0 module.
• Remove the root and hyperspace-sources modules.
• An example of Project Structure

Additionally, you might have to run sbt buildInfo if you encounter an error like object BuildInfo is not a member of package com.microsoft.hyperspace for the first build.

This project would not have been possible without the outstanding work from the following communities:

• Apache Spark: Unified Analytics Engine for Big Data, the engine that Hyperspace builds on top of.
• Delta Lake: Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads. Hyperspace derives quite a bit of inspiration from the way the Delta Lake community operates and pioneering of some surrounding ideas in the context of data lakes (e.g., their novel use of optimistic concurrency).
• Databricks: Unified analytics platform. Many thanks to all the inspiration they have provided us.
• .NET for Apache Spark™: Hyperspace offers .NET bindings for developers, thanks to the efforts of this team in collaborating and releasing the bindings just-in-time.
• Minimal Mistakes: The awesome theme behind Hyperspace documentation.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Apache License 2.0, see LICENSE.