---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 95c6f39716470a23fd681d30173aba0e7ed366be7ccfdc2e7b876a0d9acaa24f
  pageDirectory: concepts
  sources:
    - combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md
  confidence: 0.95
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - ray-core-to-spark-integration-patterns
    - RCTSIP
    - ray-core-to-spark-data-exchange-patterns
    - RCTSDEP
  citations:
    - file: combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md
title: Ray Core to Spark Integration Patterns
description: "Three patterns for writing data from Ray Core applications to Spark: persist to temporary location, connect via Spark Connect, or use third-party libraries like deltalake or deltaray."
tags:
  - databricks
  - ray-core
  - spark
  - integration-patterns
timestamp: "2026-06-19T17:46:46.296Z"
---

# Ray Core to Spark Integration Patterns

**Ray Core to Spark Integration Patterns** refers to the methods and techniques for exchanging data between Ray Core applications and Apache Spark within the same Databricks environment. These patterns allow developers to leverage the strengths of both distributed computing engines — Ray for flexible task execution and reinforcement learning, and Spark for large-scale data processing and analytics.

## Overview

Databricks supports running Ray and Spark operations in the same execution environment. Ray and Spark integration is supported by Delta Lake and Unity Catalog, which provide robust data management, secure access, and lineage tracking. When Ray Core tasks need to write data back to Spark, there are three main integration patterns, each suited to different workload characteristics and data management needs. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

Data exchange between Ray and Spark is also possible using higher-level APIs. For instance, you can create a distributed Ray dataset from a Spark DataFrame using `ray.data.from_spark()`, and write Ray data to Spark using `ray.data.Dataset.write_databricks_table()` for Unity Catalog-enabled workspaces. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

## Pattern 1: Persist Output in a Temporary Location

The most common pattern for writing data from Ray to Spark is to store the output data in a temporary location, such as Unity Catalog volumes or DBFS. After storing the data, the Ray driver thread reads each part of the files on the worker nodes and consolidates them into a final DataFrame for further processing. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

This approach works best when the output data is in tabular form, such as a Pandas DataFrame generated by a Ray Core task. Use this method when the output from Ray tasks is too large to fit in the memory of the driver node or the shared object-store. If you need to handle large datasets without persisting data to storage, consider increasing the memory allocated to the driver node in your Databricks cluster to improve performance. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

```python
import os
import uuid
import numpy as np
import pandas as pd

@ray.remote
def write_example(task_id, path_prefix):
    num_rows = 100
    df = pd.DataFrame({
        'foo': np.random.rand(num_rows),
        'bar': np.random.rand(num_rows)
    })
    df.to_csv(os.path.join(path_prefix, f"result_part_{task_id}.csv"))

n_tasks = 10
dbfs_prefix = f"/dbfs/<USERNAME>"
path_prefix = os.path.join(dbfs_prefix, f"/ray_tmp/write_task_{uuid.uuid4()}")
tasks = ray.get([write_example.remote(i, path_prefix) for i in range(n_tasks)])

df = spark.read.csv(path_prefix.replace("/dbfs", "dbfs:"), header=True, inferSchema=True)
```

^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

## Pattern 2: Connect Using Spark Connect

Another way for Ray Core tasks to interact with Spark within the remote task is to use Spark Connect. This allows you to set up the Spark context on the Ray worker to point to the Spark cluster running from the driver node. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

To set this up, you must configure Ray cluster resources to allocate space for Spark. For example, if a worker node has 8 CPUs, set `num_cpus_worker_node` to 7, leaving 1 CPU for Spark. For larger Spark tasks, it is recommended to allocate a larger share of resources. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

```python
from databricks.connect import [[databrickssession|DatabricksSession]]
import ray

@ray.remote
class SparkHandler(object):
    def __init__(self, access_token=None, cluster_id=None, host_url=None):
        self.spark = ([[databrickssession|DatabricksSession]]
                      .builder
                      .remote(host=host_url,
                              token=access_token,
                              cluster_id=cluster_id)
                      .getOrCreate()
                      )

    def test(self):
        df = self.spark.sql("select * from samples.nyctaxi.trips")
        df.write.format("delta").mode("overwrite").saveAsTable("catalog.schema.taxi_trips")
        return df.count()

access_token = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()
cluster_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().clusterId().get()
host_url = f"https://{dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().get('browserHostName').get()}"

sh = SparkHandler.remote(access_token=access_token,
                         cluster_id=cluster_id,
                         host_url=host_url)
print(ray.get(sh.test.remote()))
```

^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

This example uses the notebook-generated token. However, Databricks recommends that production use cases use an access token stored in Databricks secrets. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

Since this process calls a single Spark driver, it creates a threading lock which causes all tasks to wait for the preceding Spark tasks to complete. Therefore, it is recommended to use this when there are not many concurrent tasks since they all will have sequential behavior as the Spark tasks complete. For these situations, it is better to persist the output and then combine into a single Spark dataframe at the end and then write out to an output table. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

## Pattern 3: Third-Party Libraries

Another option is using third-party libraries that interact with Delta Lake and Spark. Databricks does not officially support these third-party libraries. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

An example is the `deltalake` library from the `delta-rs` project. This approach currently only works with Hive [Metastore](/concepts/metastore.md) tables, not Unity Catalog tables. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

```python
from deltalake import DeltaTable, write_deltalake
import pandas as pd
import numpy as np
import ray

@ray.remote
def write_test(table_name):
    random_df_id_vals = [int(np.random.randint(1000)), int(np.random.randint(1000))]
    pdf = pd.DataFrame({"id": random_df_id_vals, "value": ["foo", "bar"]})
    write_deltalake(table_name, pdf, mode="append")

def main():
    table_name = "database.mytable"
    ray.get([write_test.remote(table_name) for _ in range(100)])
```

^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

Another third-party library available is the `deltaray` library from the Delta Incubator project. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

```python
import pathlib
import deltaray
import deltalake as dl
import pandas as pd

cwd = pathlib.Path().resolve()
table_uri = f'{cwd}/tmp/delta-table'
df = pd.DataFrame({'id': [0, 1, 2, 3, 4, ]})
dl.write_deltalake(table_uri, df)
ds = deltaray.read_delta(table_uri)
ds.show()
```

^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

## Best Practices

- Always use the techniques described in the Ray cluster best practice guide to ensure the cluster is fully utilized. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]
- Consider using Unity Catalog volumes to store output data in a non-tabular format and provide governance. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]
- Ensure that the `num_cpus_worker_node` configuration is set so that the number of CPU cores matches that of the Spark worker node. Similarly, set `num_gpus_worker_node` to the number of GPUs per Spark worker node. In this configuration, each Spark worker node launches one Ray worker node that fully utilizes the resources of the Spark worker node. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

## Limitations

Unity Catalog currently does not share credentials for writing to tables from non-Spark writers. Therefore, all data being written to a Unity Catalog table from a Ray Core task will require that the data be persisted and then read with Spark, or Databricks Connect must be set up within the Ray task. ^[combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md]

## Related Concepts

- Ray Data from Spark DataFrame – Creating Ray datasets from Spark DataFrames using `ray.data.from_spark()`.
- Ray to Unity Catalog Integration – Writing Ray data to Unity Catalog tables.
- [Delta Lake on Databricks](/concepts/delta-lake-on-databricks.md) – The storage layer that supports both Ray and Spark.
- [Unity Catalog](/concepts/unity-catalog.md) – The governance layer for data management and access control.
- [Spark Connect](/concepts/spark-connect.md) – The protocol used in Pattern 2 for connecting Ray tasks to Spark.
- DBFS – Databricks File System used for temporary storage in Pattern 1.

## Sources

- combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md

# Citations

1. [combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws.md](/references/combine-ray-and-spark-in-the-same-environment-on-databricks-databricks-on-aws-47bcb6c6.md)
