Packages

package worker

Package Members

  1. package core

Type Members

  1. final class Common extends GeneratedFile
    Annotations
    @Generated()
  2. final class DirectWorker extends GeneratedMessage with DirectWorkerOrBuilder

    The worker that will be created to process UDFs


    The worker that will be created to process UDFs

    Protobuf type org.apache.spark.udf.worker.DirectWorker

    Annotations
    @Generated()
  3. trait DirectWorkerOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  4. final class LocalTcpConnection extends GeneratedMessage with LocalTcpConnectionOrBuilder

    Communication between the engine and worker
is done using a localhost TCP connection.

On [[DirectWorker]] creation, a PORT
is passed as a connection parameter.

It is expected that the worker binds to this
port on both IPv4 and IPv6 localhost interfaces.
E.g. the worker server should be reachable via
127.0.0.1:PORT and [::1]:PORT.

Examples:
8080
1234


    Communication between the engine and worker
is done using a localhost TCP connection.

On [[DirectWorker]] creation, a PORT
is passed as a connection parameter.

It is expected that the worker binds to this
port on both IPv4 and IPv6 localhost interfaces.
E.g. the worker server should be reachable via
127.0.0.1:PORT and [::1]:PORT.

Examples:
8080
1234

    Protobuf type org.apache.spark.udf.worker.LocalTcpConnection

    Annotations
    @Generated()
  5. trait LocalTcpConnectionOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  6. final class ProcessCallable extends GeneratedMessage with ProcessCallableOrBuilder

    Protobuf type org.apache.spark.udf.worker.ProcessCallable

    Protobuf type org.apache.spark.udf.worker.ProcessCallable

    Annotations
    @Generated()
  7. trait ProcessCallableOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  8. sealed final class UDFProtoCommunicationPattern extends Enum[UDFProtoCommunicationPattern] with ProtocolMessageEnum

    The UDF execution type/shape.


    The UDF execution type/shape.

    Protobuf enum org.apache.spark.udf.worker.UDFProtoCommunicationPattern

    Annotations
    @Generated()
  9. sealed final class UDFWorkerDataFormat extends Enum[UDFWorkerDataFormat] with ProtocolMessageEnum

    The UDF in & output data format.


    The UDF in & output data format.

    Protobuf enum org.apache.spark.udf.worker.UDFWorkerDataFormat

    Annotations
    @Generated()
  10. final class UDFWorkerProperties extends GeneratedMessage with UDFWorkerPropertiesOrBuilder

    Protobuf type org.apache.spark.udf.worker.UDFWorkerProperties

    Protobuf type org.apache.spark.udf.worker.UDFWorkerProperties

    Annotations
    @Generated()
  11. trait UDFWorkerPropertiesOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  12. final class UDFWorkerSpecification extends GeneratedMessage with UDFWorkerSpecificationOrBuilder

    /
/ Worker specification
/


    /
/ Worker specification
/

    Protobuf type org.apache.spark.udf.worker.UDFWorkerSpecification

    Annotations
    @Generated()
  13. trait UDFWorkerSpecificationOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  14. final class UnixDomainSocket extends GeneratedMessage with UnixDomainSocketOrBuilder

    Communication between the engine and worker
is done using a UNIX domain socket.

On [[DirectWorker]] creation, a path to a socket
to listen on is passed as an argument.
Examples:
/tmp/channel-uuid.sock
/some/system/path/channel-1234.sock


    Communication between the engine and worker
is done using a UNIX domain socket.

On [[DirectWorker]] creation, a path to a socket
to listen on is passed as an argument.
Examples:
/tmp/channel-uuid.sock
/some/system/path/channel-1234.sock

    Protobuf type org.apache.spark.udf.worker.UnixDomainSocket

    Annotations
    @Generated()
  15. trait UnixDomainSocketOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  16. final class WorkerCapabilities extends GeneratedMessage with WorkerCapabilitiesOrBuilder

    Capabilities used for query planning
and running the worker during query execution.


    Capabilities used for query planning
and running the worker during query execution.

    Protobuf type org.apache.spark.udf.worker.WorkerCapabilities

    Annotations
    @Generated()
  17. trait WorkerCapabilitiesOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  18. final class WorkerConnection extends GeneratedMessage with WorkerConnectionOrBuilder

    Protobuf type org.apache.spark.udf.worker.WorkerConnection

    Protobuf type org.apache.spark.udf.worker.WorkerConnection

    Annotations
    @Generated()
  19. trait WorkerConnectionOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  20. final class WorkerEnvironment extends GeneratedMessage with WorkerEnvironmentOrBuilder

    Set of callables that can be used to setup, verify,
and cleanup the worker environment before the worker
callable is invoked.

These callables should be specified if the worker
setup cannot be assumed to be pre-installed on the
target cluster.

If defined, the below callables will be invoked once
per Spark executor in the following order (top -> bottom):

┌──────────────┐
│              │
│ Verification │
│              │
└──┬────────┬──┘
│        │
Failed │        │ Succeeded
│        │
┌──────▼───────┐│
│              ││
│ Installation ││
│              ││
└──────┬───────┘│
│        │
▼        ▼
...
UDF worker creation
...
│
│
┌────▼────┐
│         │
│ Cleanup │
│         │
└─────────┘

All scripts are optional.
However, if a verification script is supplied, an installation
script needs to be supplied as well.


    Set of callables that can be used to setup, verify,
and cleanup the worker environment before the worker
callable is invoked.

These callables should be specified if the worker
setup cannot be assumed to be pre-installed on the
target cluster.

If defined, the below callables will be invoked once
per Spark executor in the following order (top -> bottom):

┌──────────────┐
│              │
│ Verification │
│              │
└──┬────────┬──┘
│        │
Failed │        │ Succeeded
│        │
┌──────▼───────┐│
│              ││
│ Installation ││
│              ││
└──────┬───────┘│
│        │
▼        ▼
...
UDF worker creation
...
│
│
┌────▼────┐
│         │
│ Cleanup │
│         │
└─────────┘

All scripts are optional.
However, if a verification script is supplied, an installation
script needs to be supplied as well.

    Protobuf type org.apache.spark.udf.worker.WorkerEnvironment

    Annotations
    @Generated()
  21. trait WorkerEnvironmentOrBuilder extends MessageOrBuilder
    Annotations
    @Generated()
  22. final class WorkerSpec extends GeneratedFile
    Annotations
    @Generated()
  23. class WorkerSpecification extends AnyRef

    :: Experimental :: Typed Scala wrapper around the protobuf UDFWorkerSpecification.

    :: Experimental :: Typed Scala wrapper around the protobuf UDFWorkerSpecification.

    Annotations
    @Experimental()

Ungrouped