Packages

c

org.apache.spark.udf.worker

WorkerEnvironment

final class WorkerEnvironment extends GeneratedMessage with WorkerEnvironmentOrBuilder

Set of callables that can be used to setup, verify,
and cleanup the worker environment before the worker
callable is invoked.

These callables should be specified if the worker
setup cannot be assumed to be pre-installed on the
target cluster.

If defined, the below callables will be invoked once
per Spark executor in the following order (top -> bottom):

┌──────────────┐
│              │
│ Verification │
│              │
└──┬────────┬──┘
│        │
Failed │        │ Succeeded
│        │
┌──────▼───────┐│
│              ││
│ Installation ││
│              ││
└──────┬───────┘│
│        │
▼        ▼
...
UDF worker creation
...
│
│
┌────▼────┐
│         │
│ Cleanup │
│         │
└─────────┘

All scripts are optional.
However, if a verification script is supplied, an installation
script needs to be supplied as well.

Protobuf type org.apache.spark.udf.worker.WorkerEnvironment

Annotations
@Generated()
Source
WorkerEnvironment.java
Linear Supertypes
WorkerEnvironmentOrBuilder, GeneratedMessage, Serializable, AbstractMessage, Message, MessageOrBuilder, AbstractMessageLite[MessageType, BuilderType], MessageLite, MessageLiteOrBuilder, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WorkerEnvironment
  2. WorkerEnvironmentOrBuilder
  3. GeneratedMessage
  4. Serializable
  5. AbstractMessage
  6. Message
  7. MessageOrBuilder
  8. AbstractMessageLite
  9. MessageLite
  10. MessageLiteOrBuilder
  11. AnyRef
  12. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(obj: AnyRef): Boolean
    Definition Classes
    WorkerEnvironment → AbstractMessage → Message → AnyRef → Any
    Annotations
    @Override()
  8. def findInitializationErrors(): List[String]
    Definition Classes
    AbstractMessage → MessageOrBuilder
  9. def getAllFields(): Map[FieldDescriptor, AnyRef]
    Definition Classes
    GeneratedMessage → MessageOrBuilder
  10. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. def getDefaultInstanceForType(): WorkerEnvironment
    Definition Classes
    WorkerEnvironment → MessageOrBuilder → MessageLiteOrBuilder
    Annotations
    @Override()
  12. def getDescriptorForType(): Descriptor
    Definition Classes
    GeneratedMessage → MessageOrBuilder
  13. def getEnvironmentCleanup(): ProcessCallable

    Callable, which is invoked after all workers have been terminated.
This can be used to cleanup dependencies or temporary resources.

(Optional)


    Callable, which is invoked after all workers have been terminated.
This can be used to cleanup dependencies or temporary resources.

(Optional)

    optional .org.apache.spark.udf.worker.ProcessCallable environment_cleanup = 3;

    returns

    The environmentCleanup.

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  14. def getEnvironmentCleanupOrBuilder(): ProcessCallableOrBuilder

    Callable, which is invoked after all workers have been terminated.
This can be used to cleanup dependencies or temporary resources.

(Optional)


    Callable, which is invoked after all workers have been terminated.
This can be used to cleanup dependencies or temporary resources.

(Optional)

    optional .org.apache.spark.udf.worker.ProcessCallable environment_cleanup = 3;

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  15. def getEnvironmentVerification(): ProcessCallable

    Callable, which is called to verify that an environment
is suitable to start the UDF worker. This callable
needs to verify that
- The worker code itself is present
- Any needed dependencies are present

(Optional)


    Callable, which is called to verify that an environment
is suitable to start the UDF worker. This callable
needs to verify that
- The worker code itself is present
- Any needed dependencies are present

(Optional)

    optional .org.apache.spark.udf.worker.ProcessCallable environment_verification = 2;

    returns

    The environmentVerification.

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  16. def getEnvironmentVerificationOrBuilder(): ProcessCallableOrBuilder

    Callable, which is called to verify that an environment
is suitable to start the UDF worker. This callable
needs to verify that
- The worker code itself is present
- Any needed dependencies are present

(Optional)


    Callable, which is called to verify that an environment
is suitable to start the UDF worker. This callable
needs to verify that
- The worker code itself is present
- Any needed dependencies are present

(Optional)

    optional .org.apache.spark.udf.worker.ProcessCallable environment_verification = 2;

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  17. def getField(field: FieldDescriptor): AnyRef
    Definition Classes
    GeneratedMessage → MessageOrBuilder
  18. def getInitializationErrorString(): String
    Definition Classes
    AbstractMessage → MessageOrBuilder
  19. def getInstallation(): ProcessCallable

    A callable that prepares cluster nodes by installing
the required runtime, dependencies, and UDF worker binaries
at the expected locations. This step ensures that
the worker starting callable
can subsequently locate and launch the worker process.

(Required if environment_verification is given)


    A callable that prepares cluster nodes by installing
the required runtime, dependencies, and UDF worker binaries
at the expected locations. This step ensures that
the worker starting callable
can subsequently locate and launch the worker process.

(Required if environment_verification is given)

    optional .org.apache.spark.udf.worker.ProcessCallable installation = 1;

    returns

    The installation.

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  20. def getInstallationOrBuilder(): ProcessCallableOrBuilder

    A callable that prepares cluster nodes by installing
the required runtime, dependencies, and UDF worker binaries
at the expected locations. This step ensures that
the worker starting callable
can subsequently locate and launch the worker process.

(Required if environment_verification is given)


    A callable that prepares cluster nodes by installing
the required runtime, dependencies, and UDF worker binaries
at the expected locations. This step ensures that
the worker starting callable
can subsequently locate and launch the worker process.

(Required if environment_verification is given)

    optional .org.apache.spark.udf.worker.ProcessCallable installation = 1;

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  21. def getOneofFieldDescriptor(oneof: OneofDescriptor): FieldDescriptor
    Definition Classes
    GeneratedMessage → AbstractMessage → MessageOrBuilder
  22. def getParserForType(): Parser[WorkerEnvironment]
    Definition Classes
    WorkerEnvironment → GeneratedMessage → Message → MessageLite
    Annotations
    @Override()
  23. def getRepeatedField(field: FieldDescriptor, index: Int): AnyRef
    Definition Classes
    GeneratedMessage → MessageOrBuilder
  24. def getRepeatedFieldCount(field: FieldDescriptor): Int
    Definition Classes
    GeneratedMessage → MessageOrBuilder
  25. def getSerializedSize(): Int
    Definition Classes
    WorkerEnvironment → GeneratedMessage → AbstractMessage → MessageLite
    Annotations
    @Override()
  26. def getUnknownFields(): UnknownFieldSet
    Definition Classes
    GeneratedMessage → MessageOrBuilder
  27. def hasEnvironmentCleanup(): Boolean

    Callable, which is invoked after all workers have been terminated.
This can be used to cleanup dependencies or temporary resources.

(Optional)


    Callable, which is invoked after all workers have been terminated.
This can be used to cleanup dependencies or temporary resources.

(Optional)

    optional .org.apache.spark.udf.worker.ProcessCallable environment_cleanup = 3;

    returns

    Whether the environmentCleanup field is set.

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  28. def hasEnvironmentVerification(): Boolean

    Callable, which is called to verify that an environment
is suitable to start the UDF worker. This callable
needs to verify that
- The worker code itself is present
- Any needed dependencies are present

(Optional)


    Callable, which is called to verify that an environment
is suitable to start the UDF worker. This callable
needs to verify that
- The worker code itself is present
- Any needed dependencies are present

(Optional)

    optional .org.apache.spark.udf.worker.ProcessCallable environment_verification = 2;

    returns

    Whether the environmentVerification field is set.

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  29. def hasField(field: FieldDescriptor): Boolean
    Definition Classes
    GeneratedMessage → MessageOrBuilder
  30. def hasInstallation(): Boolean

    A callable that prepares cluster nodes by installing
the required runtime, dependencies, and UDF worker binaries
at the expected locations. This step ensures that
the worker starting callable
can subsequently locate and launch the worker process.

(Required if environment_verification is given)


    A callable that prepares cluster nodes by installing
the required runtime, dependencies, and UDF worker binaries
at the expected locations. This step ensures that
the worker starting callable
can subsequently locate and launch the worker process.

(Required if environment_verification is given)

    optional .org.apache.spark.udf.worker.ProcessCallable installation = 1;

    returns

    Whether the installation field is set.

    Definition Classes
    WorkerEnvironment → WorkerEnvironmentOrBuilder
    Annotations
    @Override()
  31. def hasOneof(oneof: OneofDescriptor): Boolean
    Definition Classes
    GeneratedMessage → AbstractMessage → MessageOrBuilder
  32. def hashCode(): Int
    Definition Classes
    WorkerEnvironment → AbstractMessage → Message → AnyRef → Any
    Annotations
    @Override()
  33. def internalGetFieldAccessorTable(): FieldAccessorTable
    Attributes
    protected[worker]
    Definition Classes
    WorkerEnvironment → GeneratedMessage
    Annotations
    @Override()
  34. def internalGetMapFieldReflection(fieldNumber: Int): MapFieldReflectionAccessor
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
  35. final def isInitialized(): Boolean
    Definition Classes
    WorkerEnvironment → GeneratedMessage → AbstractMessage → MessageLiteOrBuilder
    Annotations
    @Override()
  36. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  37. def makeExtensionsImmutable(): Unit
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
  38. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  39. def newBuilderForType(parent: BuilderParent): Builder
    Attributes
    protected[worker]
    Definition Classes
    WorkerEnvironment → AbstractMessage
    Annotations
    @Override()
  40. def newBuilderForType(): Builder
    Definition Classes
    WorkerEnvironment → Message → MessageLite
    Annotations
    @Override()
  41. def newInstance(unused: UnusedPrivateParameter): AnyRef
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
  42. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  43. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  44. def parseUnknownField(input: CodedInputStream, unknownFields: Builder, extensionRegistry: ExtensionRegistryLite, tag: Int): Boolean
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
    Annotations
    @throws(classOf[java.io.IOException])
  45. def parseUnknownFieldProto3(input: CodedInputStream, unknownFields: Builder, extensionRegistry: ExtensionRegistryLite, tag: Int): Boolean
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
    Annotations
    @throws(classOf[java.io.IOException])
  46. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  47. def toBuilder(): Builder
    Definition Classes
    WorkerEnvironment → Message → MessageLite
    Annotations
    @Override()
  48. def toByteArray(): Array[Byte]
    Definition Classes
    AbstractMessageLite → MessageLite
  49. def toByteString(): ByteString
    Definition Classes
    AbstractMessageLite → MessageLite
  50. final def toString(): String
    Definition Classes
    AbstractMessage → Message → AnyRef → Any
  51. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  52. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  53. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  54. def writeDelimitedTo(output: OutputStream): Unit
    Definition Classes
    AbstractMessageLite → MessageLite
    Annotations
    @throws(classOf[java.io.IOException])
  55. def writeReplace(): AnyRef
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
    Annotations
    @throws(classOf[java.io.ObjectStreamException])
  56. def writeTo(output: CodedOutputStream): Unit
    Definition Classes
    WorkerEnvironment → GeneratedMessage → AbstractMessage → MessageLite
    Annotations
    @Override()
  57. def writeTo(output: OutputStream): Unit
    Definition Classes
    AbstractMessageLite → MessageLite
    Annotations
    @throws(classOf[java.io.IOException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

  2. def internalGetMapField(fieldNumber: Int): MapField[_ <: AnyRef, _ <: AnyRef]
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
    Annotations
    @Deprecated
    Deprecated
  3. def mergeFromAndMakeImmutableInternal(input: CodedInputStream, extensionRegistry: ExtensionRegistryLite): Unit
    Attributes
    protected[protobuf]
    Definition Classes
    GeneratedMessage
    Annotations
    @throws(classOf[com.google.protobuf.InvalidProtocolBufferException]) @Deprecated
    Deprecated

Inherited from WorkerEnvironmentOrBuilder

Inherited from GeneratedMessage

Inherited from Serializable

Inherited from AbstractMessage

Inherited from Message

Inherited from MessageOrBuilder

Inherited from AbstractMessageLite[MessageType, BuilderType]

Inherited from MessageLite

Inherited from MessageLiteOrBuilder

Inherited from AnyRef

Inherited from Any

Ungrouped