class WindowSpec extends AnyRef
A window specification that defines the partitioning, ordering, and frame boundaries.
Use the static methods in Window to create a WindowSpec.
- Annotations
- @Stable()
- Source
- WindowSpec.scala
- Since
1.4.0
- Alphabetic
- By Inheritance
- WindowSpec
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
orderBy(cols: Column*): WindowSpec
Defines the ordering columns in a WindowSpec.
Defines the ordering columns in a WindowSpec.
- Annotations
- @varargs()
- Since
1.4.0
-
def
orderBy(colName: String, colNames: String*): WindowSpec
Defines the ordering columns in a WindowSpec.
Defines the ordering columns in a WindowSpec.
- Annotations
- @varargs()
- Since
1.4.0
-
def
partitionBy(cols: Column*): WindowSpec
Defines the partitioning columns in a WindowSpec.
Defines the partitioning columns in a WindowSpec.
- Annotations
- @varargs()
- Since
1.4.0
-
def
partitionBy(colName: String, colNames: String*): WindowSpec
Defines the partitioning columns in a WindowSpec.
Defines the partitioning columns in a WindowSpec.
- Annotations
- @varargs()
- Since
1.4.0
-
def
rangeBetween(start: Long, end: Long): WindowSpec
Defines the frame boundaries, from
start
(inclusive) toend
(inclusive).Defines the frame boundaries, from
start
(inclusive) toend
(inclusive).Both
start
andend
are relative from the current row. For example, "0" means "current row", while "-1" means one off before the current row, and "5" means the five off after the current row.We recommend users use
Window.unboundedPreceding
,Window.unboundedFollowing
, andWindow.currentRow
to specify special boundary values, rather than using long values directly.A range-based boundary is based on the actual value of the ORDER BY expression(s). An offset is used to alter the value of the ORDER BY expression, for instance if the current order by expression has a value of 10 and the lower bound offset is -3, the resulting lower bound for the current row will be 10 - 3 = 7. This however puts a number of constraints on the ORDER BY expressions: there can be only one expression and this expression must have a numerical data type. An exception can be made when the offset is unbounded, because no value modification is needed, in this case multiple and non-numeric ORDER BY expression are allowed.
import org.apache.spark.sql.expressions.Window val df = Seq((1, "a"), (1, "a"), (2, "a"), (1, "b"), (2, "b"), (3, "b")) .toDF("id", "category") val byCategoryOrderedById = Window.partitionBy($"category").orderBy($"id").rangeBetween(Window.currentRow, 1) df.withColumn("sum", sum($"id") over byCategoryOrderedById).show() +---+--------+---+ | id|category|sum| +---+--------+---+ | 1| b| 3| | 2| b| 5| | 3| b| 3| | 1| a| 4| | 1| a| 4| | 2| a| 2| +---+--------+---+
- start
boundary start, inclusive. The frame is unbounded if this is the minimum long value (
Window.unboundedPreceding
).- end
boundary end, inclusive. The frame is unbounded if this is the maximum long value (
Window.unboundedFollowing
).
- Since
1.4.0
-
def
rowsBetween(start: Long, end: Long): WindowSpec
Defines the frame boundaries, from
start
(inclusive) toend
(inclusive).Defines the frame boundaries, from
start
(inclusive) toend
(inclusive).Both
start
andend
are relative positions from the current row. For example, "0" means "current row", while "-1" means the row before the current row, and "5" means the fifth row after the current row.We recommend users use
Window.unboundedPreceding
,Window.unboundedFollowing
, andWindow.currentRow
to specify special boundary values, rather than using integral values directly.A row based boundary is based on the position of the row within the partition. An offset indicates the number of rows above or below the current row, the frame for the current row starts or ends. For instance, given a row based sliding frame with a lower bound offset of -1 and a upper bound offset of +2. The frame for row with index 5 would range from index 4 to index 7.
import org.apache.spark.sql.expressions.Window val df = Seq((1, "a"), (1, "a"), (2, "a"), (1, "b"), (2, "b"), (3, "b")) .toDF("id", "category") val byCategoryOrderedById = Window.partitionBy($"category").orderBy($"id").rowsBetween(Window.currentRow, 1) df.withColumn("sum", sum($"id") over byCategoryOrderedById).show() +---+--------+---+ | id|category|sum| +---+--------+---+ | 1| b| 3| | 2| b| 5| | 3| b| 3| | 1| a| 2| | 1| a| 3| | 2| a| 2| +---+--------+---+
- start
boundary start, inclusive. The frame is unbounded if this is the minimum long value (
Window.unboundedPreceding
).- end
boundary end, inclusive. The frame is unbounded if this is the maximum long value (
Window.unboundedFollowing
).
- Since
1.4.0
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()