I am creating an API that can access data stored using an Avro schema. The
API can only know the Avro schema at runtime when it is passed as a parm by
a user of the API. I need to initialize a custom serializer with the Avro
schema on remote worker and driver processes. I've tried to set the schema
in an environment variable using SparkConf.setExecutorEnv("app.schema",
schema) but that environment variable does not appear to be available when
I try to access it in the serializer. I've tried using SPARK_YARN_USER_ENV
as well but no luck. Is there a way to pass initialization parameters to
the serializer at runtime?
Thanks,
Anthony
