.NET for Apache Spark provides high-performance APIs for using Apache Spark from C# and F#. Use DataFrames and Spark SQL to work with structured data, and Structured Streaming to process streaming data.
Microsoft.Spark targets .NET Standard 2.0 and 2.1. Running an application also requires a compatible .NET Worker, Apache Spark runtime, and Scala bridge. The Worker runs on Windows, Linux, and macOS using .NET 8, or on Windows using .NET Framework.
Deployment guides are available for Azure HDInsight Spark, Amazon EMR Spark, and Databricks on AWS and Azure. Check the target platform's runtime and Worker compatibility before following a guide; these guides do not validate every managed-service runtime.
For background on the proposal for upstream .NET bindings, see SPIP: .NET bindings for Apache Spark.
- Supported Apache Spark
- Releases
- Get Started
- Building from Source
- Samples
- Test Status
- Contributing
- Inspiration and Special Thanks
- How to Engage, Contribute and Provide Feedback
- Support
- .NET Foundation
- Code of Conduct
- License
The next release removes Apache Spark 2.x support and retains the Spark 3.0 through 3.5 runtime lines. See the migration guide before upgrading an existing application.
The getting-started examples use Spark 3.5.3 with Scala 2.12. Spark 4.0 integration requires the Scala 2.13 bridge and JDK 17; it does not imply full API or deployment parity with Spark 3.x. Match the bridge JAR to the Spark runtime and the Microsoft.Spark/Worker release, and validate the APIs and deployment configuration your application uses.
For Spark 4.0 semi-structured data, see Variant schemas and Spark SQL; native .NET Variant values and Variant UDFs are not supported.
The published v2.3.1 release supports Spark 2.4 (except 2.4.2) and Spark 3.0 through 3.5, not Spark 4.0. See the release compatibility notes for that release; they do not describe the current source tree.
.NET for Apache Spark releases are available here and NuGet packages are available here.
These instructions will show you how to run a .NET for Apache Spark app using .NET 8.
Follow the platform-specific instructions to install prerequisites, build the bridge and .NET components, and run the tests.
| Platform | Instructions |
|---|---|
| Windows | .NET 8 · .NET Framework 4.8 |
| Ubuntu | .NET 8 |
Browse the sample index for DataFrame and Structured Streaming examples. See Building from Source for setup instructions and the benchmark guide for running TPC-H queries.
| Scenario | Description | Code |
|---|---|---|
| DataFrames and Spark SQL | Basic queries and transformations | C# · F# |
| Streaming word count | Count words from a network stream | C# · F# |
| Windowed word count | Aggregate streaming data over time windows | C# · F# |
| Kafka word count | Process streaming data from Apache Kafka | C# · F# |
| TPC-H queries | Express benchmark queries using DataFrame APIs or SQL | DataFrame APIs · SQL |
End-to-end test status for 
; updates may be delayed.
| Spark | Windows | Linux |
|---|---|---|
| 3.0.0 |  |
 |
| 3.0.1 |  |
 |
| 3.0.2 |  |
 |
| 3.1.1 |  |
 |
| 3.1.2 |  |
 |
| 3.2.1 |  |
 |
| 3.2.2 |  |
 |
| 3.2.3 |  |
 |
| 3.3.0 |  |
 |
| 3.3.1 |  |
 |
| 3.3.2 |  |
 |
| 3.3.3 |  |
 |
| 3.3.4 |  |
 |
| 3.4.0 |  |
 |
| 3.4.1 |  |
 |
| 3.4.2 |  |
 |
| 3.4.3 |  |
 |
| 3.4.4 |  |
 |
| 3.5.0 |  |
 |
| 3.5.1 |  |
 |
| 3.5.2 |  |
 |
| 3.5.3 |  |
 |
| 4.0.0 |  |
 |
| 4.0.1 |  |
 |
| 4.0.2 |  |
 |
| 4.0.3 |  |
 |
| 4.0.4 |  |
 |
We welcome contributions! Please review our contribution guide.
This project would not have been possible without the outstanding work from the following communities:
- Apache Spark: Unified Analytics Engine for Big Data, the underlying backend execution engine for .NET for Apache Spark
- Mobius: C# and F# language binding and extensions to Apache Spark, a pre-cursor project to .NET for Apache Spark from the same Microsoft group.
- PySpark: Python bindings for Apache Spark, one of the implementations .NET for Apache Spark derives inspiration from.
- sparkR: one of the implementations .NET for Apache Spark derives inspiration from.
- Apache Arrow: A cross-language development platform for in-memory data. This library provides .NET for Apache Spark with efficient ways to transfer column major data between the JVM and .NET CLR.
- Pyrolite - Java and .NET interface to Python's pickle and Pyro protocols. This library provides .NET for Apache Spark with efficient ways to transfer row major data between the JVM and .NET CLR.
- Databricks: Unified analytics platform. Many thanks to all the suggestions from them towards making .NET for Apache Spark run on Azure and AWS Databricks.
The .NET for Apache Spark team encourages contributions, both issues and PRs. The first step is finding an existing issue you want to contribute to or, if you cannot find any, opening an issue.
.NET for Apache Spark is an open source project under the .NET Foundation and does not come with Microsoft Support unless otherwise noted by the specific product. For issues with or questions about .NET for Apache Spark, please create an issue.
The .NET for Apache Spark project is part of the .NET Foundation.
This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.
.NET for Apache Spark is licensed under the MIT license.