Hello Denes,
Thanks for responding.
I do not have a load balancer yet. I am going directly against
oozie-server. This is for testing.
I have attached oozie-site. Only Error I see in the oozie logs is
DEBUG UserGroupInformation:1875 - SERVER[myhost0.mydomain.com]
PrivilegedAction [as: oozie/myhost0.mydomain....@myservice.mydomain.com
(auth:KERBEROS)][action: org.apache.hadoop.ipc.Client$Connection$2@33c92320]
java.lang.Exception
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:839)
at
org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1677)
at org.apache.hadoop.ipc.Client.call(Client.java:1502)
at org.apache.hadoop.ipc.Client.call(Client.java:1455)
jetty.out file does not have anything relevant. How do we enable DEBUG
logging for jetty?
On Thu, Feb 2, 2023 at 11:21 PM Dénes Bodó <dionu...@apache.org> wrote:
> Hey Anup,
>
>
> May you please share your oozie-site.xml and the related Oozie server logs
> and Jetty logs somehow? Have you also set up a load balancer? Have you goz
> 503 from both Oozie instance?
>
> If you got 503 you must have sumething suspicios in oozie logs or jetty
> logs.
>
> Regards,
> Denes
>
> On Fri, 3 Feb 2023, 01:08 anup ahire, <ahirea...@gmail.com> wrote:
>
> > Hello,
> >
> > I followed steps from
> > https://oozie.apache.org/docs/5.0.0/AG_Install.html#High_Availability_HA
> > to
> > configure HA in Kerberized cluster.
> >
> > After enabling HA, I am seeing that clients are not able to communicate
> to
> > oozie server and get 503 service unavailable. . The embedded oozie server
> > process is up and listening to the required port. Logs are not showing
> > anything relevant that can explain the issue.
> > After removing zookeepr config, oozie server again becomes accessible.
> >
> > Any idea what might be going wrong ?
> >
> > Thanks.
> >
>
<configuration xmlns:xi="http://www.w3.org/2001/XInclude";>
<property>
<name>credentialStoreClassPath</name>
<value>/var/lib/ambari-agent/cred/lib/*</value>
</property>
<property>
<name>hadoop.security.credential.provider.path</name>
<value>localjceks://file/usr/current/oozie-server/conf/oozie-site.jceks</value>
</property>
<property>
<name>local.realm</name>
<value>MYSERVICE.MYDOMAIN.COM</value>
</property>
<property>
<name>oozie.action.retry.interval</name>
<value>30</value>
</property>
<property>
<name>oozie.action.sharelib.for.spark.exclude</name>
<value>oozie/jackson.*</value>
</property>
<property>
<name>oozie.authentication.authentication.provider.url</name>
<value></value>
</property>
<property>
<name>oozie.authentication.expected.jwt.audiences</name>
<value></value>
</property>
<property>
<name>oozie.authentication.jwt.cookie</name>
<value>hadoop-jwt</value>
</property>
<property>
<name>oozie.authentication.kerberos.keytab</name>
<value>/etc/security/keytabs/oozie.ha.keytab</value>
</property>
<property>
<name>oozie.authentication.kerberos.name.rules</name>
<value>RULE:[1:$1@$0](ambari...@myservice.mydomain.com)s/.*/ambari-qa/
RULE:[1:$1@$0](datastu...@myservice.mydomain.com)s/.*/datastudio/
RULE:[1:$1@$0](h...@myservice.mydomain.com)s/.*/hdfs/
RULE:[1:$1@$0](jupyter...@myservice.mydomain.com)s/.*/jupyterhub/
RULE:[1:$1@$0](sp...@myservice.mydomain.com)s/.*/spark/
RULE:[1:$1@$0](yarn-...@myservice.mydomain.com)s/.*/yarn-ats/
RULE:[1:$1@$0](.*@MYSERVICE.MYDOMAIN.COM)s/@.*//
RULE:[2:$1@$0](amshb...@myservice.mydomain.com)s/.*/ams/
RULE:[2:$1@$0](ams...@myservice.mydomain.com)s/.*/ams/
RULE:[2:$1@$0](am...@myservice.mydomain.com)s/.*/ams/
RULE:[2:$1@$0](d...@myservice.mydomain.com)s/.*/hdfs/
RULE:[2:$1@$0](h...@myservice.mydomain.com)s/.*/datastudioadmin/
RULE:[2:$1@$0](h...@myservice.mydomain.com)s/.*/hive/
RULE:[2:$1@$0](j...@myservice.mydomain.com)s/.*/mapred/
RULE:[2:$1@$0](j...@myservice.mydomain.com)s/.*/hdfs/
RULE:[2:$1@$0](l...@myservice.mydomain.com)s/.*/livy/
RULE:[2:$1@$0](n...@myservice.mydomain.com)s/.*/yarn/
RULE:[2:$1@$0](n...@myservice.mydomain.com)s/.*/hdfs/
RULE:[2:$1@$0](oo...@myservice.mydomain.com)s/.*/oozie/
RULE:[2:$1@$0](rangerad...@myservice.mydomain.com)s/.*/ranger/
RULE:[2:$1@$0](rangertags...@myservice.mydomain.com)s/.*/rangertagsync/
RULE:[2:$1@$0](rangerusers...@myservice.mydomain.com)s/.*/rangerusersync/
RULE:[2:$1@$0](r...@myservice.mydomain.com)s/.*/yarn/
RULE:[2:$1@$0](sp...@myservice.mydomain.com)s/.*/spark/
RULE:[2:$1@$0](y...@myservice.mydomain.com)s/.*/yarn/
RULE:[2:$1@$0](yarn-ats-hb...@myservice.mydomain.com)s/.*/yarn-ats/
DEFAULT</value>
</property>
<property>
<name>oozie.authentication.kerberos.principal</name>
<value>*</value>
</property>
<property>
<name>oozie.authentication.public.key.pem</name>
<value></value>
</property>
<property>
<name>oozie.authentication.simple.anonymous.allowed</name>
<value>true</value>
</property>
<property>
<name>oozie.authentication.type</name>
<value>kerberos</value>
</property>
<property>
<name>oozie.base.url</name>
<value>http://myhost0.mydomain.com:11000/oozie</value>
</property>
<property>
<name>oozie.credentials.credentialclasses</name>
<value>hcat=org.apache.oozie.action.hadoop.HCatCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials</value>
</property>
<property>
<name>oozie.db.schema.name</name>
<value>oozie</value>
</property>
<property>
<name>oozie.ha.authentication.kerberos.keytab</name>
<value>/etc/security/keytabs/oozie.ha.keytab</value>
</property>
<property>
<name>oozie.ha.authentication.kerberos.principal</name>
<value>*</value>
</property>
<property>
<name>oozie.https.keystore.file</name>
<value>/etc/security/serverKeys/keystore.jks</value>
</property>
<property>
<name>oozie.https.keystore.type</name>
<value>jks</value>
</property>
<property>
<name>oozie.https.truststore.file</name>
<value>/etc/security/serverKeys/truststore.jks</value>
</property>
<property>
<name>oozie.service.ActionService.executor.ext.classes</name>
<value>
org.apache.oozie.action.email.EmailActionExecutor,
org.apache.oozie.action.hadoop.ShellActionExecutor,
org.apache.oozie.action.hadoop.SqoopActionExecutor,
org.apache.oozie.action.hadoop.DistcpActionExecutor</value>
</property>
<property>
<name>oozie.service.AuthorizationService.authorization.enabled</name>
<value>true</value>
</property>
<property>
<name>oozie.service.AuthorizationService.security.enabled</name>
<value>true</value>
</property>
<property>
<name>oozie.service.CallableQueueService.callable.concurrency</name>
<value>3</value>
</property>
<property>
<name>oozie.service.CallableQueueService.queue.size</name>
<value>1000</value>
</property>
<property>
<name>oozie.service.CallableQueueService.threads</name>
<value>10</value>
</property>
<property>
<name>oozie.service.coord.normal.default.timeout</name>
<value>120</value>
</property>
<property>
<name>oozie.service.coord.push.check.requeue.interval</name>
<value>30000</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/etc/hadoop/conf</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.kerberos.enabled</name>
<value>true</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.kerberos.principal</name>
<value>oozie/_h...@myservice.mydomain.com</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.keytab.file</name>
<value>/etc/security/keytabs/oozie.service.keytab</value>
</property>
<property>
<name>oozie.service.JPAService.create.db.schema</name>
<value>false</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://myhost0.mydomain.com/oozie</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.username</name>
<value>oozie</value>
</property>
<property>
<name>oozie.service.JPAService.pool.max.active.conn</name>
<value>10</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>
<value>*</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>oozie.service.PurgeService.older.than</name>
<value>30</value>
</property>
<property>
<name>oozie.service.PurgeService.purge.interval</name>
<value>3600</value>
</property>
<property>
<name>oozie.service.SchemaService.wf.ext.schemas</name>
<value>shell-action-0.1.xsd,email-action-0.1.xsd,hive-action-0.2.xsd,sqoop-action-0.2.xsd,ssh-action-0.1.xsd,distcp-action-0.1.xsd,shell-action-0.2.xsd,oozie-sla-0.1.xsd,oozie-sla-0.2.xsd,hive-action-0.3.xsd</value>
</property>
<property>
<name>oozie.service.SparkConfigurationService.spark.configurations</name>
<value>*=/usr/current/spark3-client/conf</value>
</property>
<property>
<name>oozie.service.URIHandlerService.uri.handlers</name>
<value>org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler</value>
</property>
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/${user.name}/share/lib</value>
</property>
<property>
<name>oozie.services</name>
<value>
org.apache.oozie.service.SchedulerService,
org.apache.oozie.service.MetricsInstrumentationService,
org.apache.oozie.service.MemoryLocksService,
org.apache.oozie.service.UUIDService,
org.apache.oozie.service.ELService,
org.apache.oozie.service.AuthorizationService,
org.apache.oozie.service.UserGroupInformationService,
org.apache.oozie.service.HadoopAccessorService,
org.apache.oozie.service.JobsConcurrencyService,
org.apache.oozie.service.URIHandlerService,
org.apache.oozie.service.DagXLogInfoService,
org.apache.oozie.service.SchemaService,
org.apache.oozie.service.LiteWorkflowAppService,
org.apache.oozie.service.JPAService,
org.apache.oozie.service.StoreService,
org.apache.oozie.service.DBLiteWorkflowStoreService,
org.apache.oozie.service.CallbackService,
org.apache.oozie.service.ActionService,
org.apache.oozie.service.ShareLibService,
org.apache.oozie.service.CallableQueueService,
org.apache.oozie.service.ActionCheckerService,
org.apache.oozie.service.RecoveryService,
org.apache.oozie.service.PurgeService,
org.apache.oozie.service.CoordinatorEngineService,
org.apache.oozie.service.BundleEngineService,
org.apache.oozie.service.DagEngineService,
org.apache.oozie.service.CoordMaterializeTriggerService,
org.apache.oozie.service.StatusTransitService,
org.apache.oozie.service.PauseTransitService,
org.apache.oozie.service.GroupsService,
org.apache.oozie.service.ProxyUserService,
org.apache.oozie.service.XLogStreamingService,
org.apache.oozie.service.JvmPauseMonitorService,
org.apache.oozie.service.SparkConfigurationService,
org.apache.oozie.service.SchemaCheckerService</value>
</property>
<property>
<name>oozie.services.ext</name>
<value>org.apache.oozie.service.JMSAccessorService,org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService,org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService, org.apache.oozie.service.ZKJobsConcurrencyService,org.apache.oozie.service.ZKUUIDService</value>
</property>
<property>
<name>oozie.system.id</name>
<value>oozie-${user.name}</value>
</property>
<property>
<name>oozie.systemmode</name>
<value>NORMAL</value>
</property>
<property>
<name>oozie.zookeeper.connection.string</name>
<value>myhost0.mydomain.com:2181,myhost1.mydomain.com:2181,myhost1.mydomain.com:2181</value>
</property>
<property>
<name>oozie.zookeeper.namespace</name>
<value>oozie</value>
</property>
<property>
<name>oozie.zookeeper.secure</name>
<value>true</value>
</property>
<property>
<name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
<value>false</value>
</property>
</configuration>
