Apache Spark supports processing various types of data. Not all expressions support all data types. The RAPIDS Accelerator for Apache Spark has further restrictions on what types are supported for processing. This tries to document what operations are supported and what data types each operation supports. Because Apache Spark is under active development too and this document was generated against version 3.0.1 of Spark. Most of this should still apply to other versions of Spark, but there may be slight changes.

General limitations

Decimal

The Decimal type in Spark supports a precision up to 38 digits (128-bits). The RAPIDS Accelerator in most cases stores values up to 64-bits and will support 128-bit in the future. As such the accelerator currently only supports a precision up to 18 digits. Note that decimals are disabled by default in the plugin, because it is supported by a relatively small number of operations presently. This can result in a lot of data movement to and from the GPU, slowing down processing in some cases. Result Decimal precision and scale follow the same rule as CPU mode in Apache Spark:

 * In particular, if we have expressions e1 and e2 with precision/scale p1/s1 and p2/s2
 * respectively, then the following operations have the following precision / scale:
 *
 *   Operation    Result Precision                        Result Scale
 *   ------------------------------------------------------------------------
 *   e1 + e2      max(s1, s2) + max(p1-s1, p2-s2) + 1     max(s1, s2)
 *   e1 - e2      max(s1, s2) + max(p1-s1, p2-s2) + 1     max(s1, s2)
 *   e1 * e2      p1 + p2 + 1                             s1 + s2
 *   e1 / e2      p1 - s1 + s2 + max(6, s1 + p2 + 1)      max(6, s1 + p2 + 1)
 *   e1 % e2      min(p1-s1, p2-s2) + max(s1, s2)         max(s1, s2)
 *   e1 union e2  max(s1, s2) + max(p1-s1, p2-s2)         max(s1, s2)

However, Spark inserts PromotePrecision to CAST both sides to the same type. GPU mode may fall back to CPU even if the result Decimal precision is within 18 digits. For example, Decimal(8,2) x Decimal(6,3) resulting in Decimal (15,5) runs on CPU, because due to PromotePrecision, GPU mode assumes the result is Decimal(19,6). There are even extreme cases where Spark can temporarily return a Decimal value larger than what can be stored in 128-bits and then uses the CheckOverflow operator to round it to a desired precision and scale. This means that even when the accelerator supports 128-bit decimal, we might not be able to support all operations that Spark can support.

Timestamp

Timestamps in Spark will all be converted to the local time zone before processing and are often converted to UTC before being stored, like in Parquet or ORC. The RAPIDS Accelerator only supports UTC as the time zone for timestamps.

CalendarInterval

In Spark CalendarIntervals store three values, months, days, and microseconds. Support for this type is still very limited in the accelerator. In some cases only a a subset of the type is supported, like window ranges only support days currently.

Configuration

There are lots of different configuration values that can impact if an operation is supported or not. Some of these are a part of the RAPIDS Accelerator and cover the level of compatibility with Apache Spark. Those are covered here. Others are a part of Apache Spark itself and those are a bit harder to document. The work of updating this to cover that support is still ongoing.

In general though if you ever have any question about why an operation is not running on the GPU you may set spark.rapids.sql.explain to ALL and it will try to give all of the reasons why this particular operator or expression is on the CPU or GPU.

Key

Types

Support

SparkPlan or Executor Nodes

Apache Spark uses a Directed Acyclic Graph(DAG) of processing to build a query. The nodes in this graph are instances of SparkPlan and represent various high level operations like doing a filter or project. The operations that the RAPIDS Accelerator supports are described below.

Expression and SQL Functions

Inside each node in the DAG there can be one or more trees of expressions that describe various types of processing that happens in that part of the plan. These can be things like adding two numbers together or checking for null. These expressions can have multiple input parameters and one output value. These expressions also can happen in different contexts. Because of how the accelerator works different contexts have different levels of support.

The most common expression context is project. In this context values from a single input row go through the expression and the result will also be use to produce something in the same row. Be aware that even in the case of aggregation and window operations most of the processing is still done in the project context either before or after the other processing happens.

Aggregation operations like count or sum can take place in either the aggregation, reduction, or window context. aggregation is when the operation was done while grouping the data by one or more keys. reduction is when there is no group by and there is a single result for an entire column. window is for window operations.

The final expression context is AST or Abstract Syntax Tree. Before explaining AST we first need to explain in detail how project context operations work. Generally for a project context operation the plan Spark developed is read on the CPU and an appropriate set of GPU kernels are selected to do those operations. For example a >= b + 1. Would result in calling a GPU kernel to add 1 to b, followed by another kernel that is called to compare a to that result. The interpretation is happening on the CPU, and the GPU is used to do the processing. For AST the interpretation for some reason cannot happen on the CPU and instead must be done in the GPU kernel itself. An example of this is conditional joins. If you want to join on A.a >= B.b + 1 where A and B are separate tables or data frames, the + and >= operations cannot run as separate independent kernels because it is done on a combination of rows in both A and B. Instead part of the plan that Spark developed is turned into an abstract syntax tree and sent to the GPU where it can be interpreted. The number and types of operations supported in this are limited.