Hi, I am trying to debug hive-exec (3.0.0) code running in local mode with Tez as an execution engine. So far I managed to successfully create tez session and complete compile phase.
2018-07-30T17:50:05,773 INFO [main] common.TezYARNUtils: Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true 2018-07-30T17:50:05,987 INFO [main] client.LocalClient: DAGAppMaster thread has been created 2018-07-30T17:50:05,987 INFO [main] client.LocalClient: DAGAppMaster is not created wait for 100ms... 2018-07-30T17:50:05,988 INFO [DAGAppMaster Thread] client.LocalClient: Using working directory: /tmp/hive/_tez_session_dir/84a05842-702d-4a8d-b5c6-bdedfbd8d203/.tez/application_1532969405620_0001_wd 2018-07-30T17:50:06,087 INFO [main] client.LocalClient: DAGAppMaster is not created wait for 100ms... 2018-07-30T17:50:06,145 INFO [DAGAppMaster Thread] app.DAGAppMaster: Created DAGAppMaster for application appattempt_1532969405620_0001_000000, versionInfo=[ component=tez-dag, version=0.9.1, revision=23b58b2b996eee255aab1a045412de00677ca2f1, SCM-URL=scm:git: https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2017-12-13T00:06:01Z ] 2018-07-30T17:50:06,157 INFO [DAGAppMaster Thread] shim.HadoopShimsLoader: Trying to locate HadoopShimProvider for hadoopVersion=3.1.0, majorVersion=3, minorVersion=1 2018-07-30T17:50:06,158 INFO [DAGAppMaster Thread] shim.HadoopShimsLoader: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=3.1.0, majorVersion=3, minorVersion=1 2018-07-30T17:50:06,172 INFO [DAGAppMaster Thread] app.DAGAppMaster: AM Level configured TaskSchedulers: [0:TezUber:null] 2018-07-30T17:50:06,172 INFO [DAGAppMaster Thread] app.DAGAppMaster: AM Level configured ContainerLaunchers: [0:TezUber:null] 2018-07-30T17:50:06,172 INFO [DAGAppMaster Thread] app.DAGAppMaster: AM Level configured TaskCommunicators: [0:TezUber:null] 2018-07-30T17:50:06,172 INFO [DAGAppMaster Thread] app.DAGAppMaster: Comparing client version with AM version, clientVersion=0.9.1, AMVersion=0.9.1 2018-07-30T17:50:06,187 INFO [main] client.LocalClient: DAGAppMaster state: NEW 2018-07-30T17:50:06,187 INFO [main] client.LocalClient: DAGAppMaster is not started wait for 100ms... 2018-07-30T17:50:06,187 INFO [main] client.LocalClient: DAGAppMaster is not created wait for 100ms... 2018-07-30T17:50:06,205 INFO [DAGAppMaster Thread] app.TaskCommunicatorManager: Creating Default Local Task Communicator 2018-07-30T17:50:06,228 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap 2018-07-30T17:50:06,231 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker 2018-07-30T17:50:06,233 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler 2018-07-30T17:50:06,233 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher 2018-07-30T17:50:06,233 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher 2018-07-30T17:50:06,233 INFO [DAGAppMaster Thread] app.DAGAppMaster: Using concurrent dispatcher: false 2018-07-30T17:50:06,234 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher 2018-07-30T17:50:06,234 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher 2018-07-30T17:50:06,235 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler 2018-07-30T17:50:06,235 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler 2018-07-30T17:50:06,243 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager 2018-07-30T17:50:06,251 INFO [DAGAppMaster Thread] launcher.TezContainerLauncherImpl: Creating LocalContainerLauncher 2018-07-30T17:50:06,282 INFO [DAGAppMaster Thread] common.AsyncDispatcher: Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager 2018-07-30T17:50:06,287 INFO [main] client.LocalClient: DAGAppMaster state: NEW 2018-07-30T17:50:06,287 INFO [main] client.LocalClient: DAGAppMaster is not started wait for 100ms... 2018-07-30T17:50:06,287 INFO [main] client.LocalClient: DAGAppMaster is not created wait for 100ms... 2018-07-30T17:50:06,288 INFO [DAGAppMaster Thread] node.AMNodeTracker: blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 3, nodeUpdatesRescheduleEnabled: false 2018-07-30T17:50:06,288 INFO [DAGAppMaster Thread] history.HistoryEventHandler: Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService 2018-07-30T17:50:06,294 INFO [DAGAppMaster Thread] impl.SimpleHistoryLoggingService: Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/tmp/hive/_tez_session_dir/84a05842-702d-4a8d-b5c6-bdedfbd8d203/.tez/application_1532969405620_0001_wd/localmode-log-dir 2018-07-30T17:50:06,295 INFO [DAGAppMaster Thread] impl.SimpleHistoryLoggingService: Initializing SimpleHistoryLoggingService, logFileLocation=/tmp/hive/_tez_session_dir/84a05842-702d-4a8d-b5c6-bdedfbd8d203/.tez/application_1532969405620_0001_wd/localmode-log-dir/history.txt.appattempt_1532969405620_0001_000000, maxErrors=10 2018-07-30T17:50:06,295 INFO [DAGAppMaster Thread] recovery.RecoveryService: RecoveryService initialized with recoveryPath=file:/tmp/hive/_tez_session_dir/84a05842-702d-4a8d-b5c6-bdedfbd8d203/.tez/application_1532969405620_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100 2018-07-30T17:50:06,296 INFO [DAGAppMaster Thread] HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1532969405620_0001_000000, appSubmitTime=1532969406115, launchTime=1532969406144 2018-07-30T17:50:06,301 INFO [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] impl.SimpleHistoryLoggingService: Starting SimpleHistoryLoggingService 2018-07-30T17:50:06,387 INFO [main] client.LocalClient: DAGAppMaster state: INITED 2018-07-30T17:50:06,387 INFO [main] client.LocalClient: DAGAppMaster is not startetd wait for 100ms... 2018-07-30T17:50:06,387 INFO [main] client.LocalClient: DAGAppMaster is not created wait for 100ms... 2018-07-30T17:50:06,402 INFO [ServiceThread:DAGClientRPCServer] client.DAGClientServer: Instantiated DAGClientRPCServer at / 192.168.0.10:50839 2018-07-30T17:50:06,403 INFO [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1532969405620 2018-07-30T17:50:06,412 INFO [DAGAppMaster Thread] HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1532969405620_0001_000000, startTime=1532969406411 2018-07-30T17:50:06,412 INFO [DAGAppMaster Thread] app.DAGAppMaster: In Session mode. Waiting for DAG over RPC 2018-07-30T17:50:06,487 INFO [main] client.LocalClient: DAGAppMaster state: IDLE 2018-07-30T17:50:06,519 INFO [main] client.TezClient: The url to track the Tez Session: N/A But during the run phase I get the following error on windows 7: 2018-07-30T17:41:46,744 INFO [LocalContainerLauncher-SubTaskRunner] launcher.TezLocalCacheManager: Creating symlink: file:/E:/code/apache-hive-3.0.0-src/tez-local-cache6231610932283994633/tez-conf.pb1808822955231457846/tez-conf.pb <- file:/E:/code/apache-hive-3.0.0-src/tez-conf.pb 2018-07-30T17:41:46,745 ERROR [LocalContainerLauncher-SubTaskRunner] launcher.LocalContainerLauncher: TezSubTaskRunner failed due to exception java.nio.file.InvalidPathException: Illegal char <:> at index 2: /E:/code/apache-hive-3.0.0-src/tez-conf.pb at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) ~[?:1.8.0_161] at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) ~[?:1.8.0_161] at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) ~[?:1.8.0_161] at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94) ~[?:1.8.0_161] at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255) ~[?:1.8.0_161] at java.nio.file.Paths.get(Paths.java:84) ~[?:1.8.0_161] at org.apache.tez.dag.app.launcher.TezLocalCacheManager.createSymlink(TezLocalCacheManager.java:173) ~[tez-dag-0.9.1.jar:0.9.1] at org.apache.tez.dag.app.launcher.TezLocalCacheManager.localize(TezLocalCacheManager.java:126) ~[tez-dag-0.9.1.jar:0.9.1] at org.apache.tez.dag.app.launcher.LocalContainerLauncher.launch(LocalContainerLauncher.java:263) ~[tez-dag-0.9.1.jar:0.9.1] at org.apache.tez.dag.app.launcher.LocalContainerLauncher.access$300(LocalContainerLauncher.java:82) ~[tez-dag-0.9.1.jar:0.9.1] at org.apache.tez.dag.app.launcher.LocalContainerLauncher$TezSubTaskRunner.run(LocalContainerLauncher.java:207) [tez-dag-0.9.1.jar:0.9.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] are there any other additional config items apart from the following, to run tez on windows in local mode? conf.set("fs.defaultFS", "file:///"); conf.set("mapred.job.tracker", "local"); conf.set("mapreduce.framework.name", "local"); conf.setBoolean(TezConfiguration.TEZ_LOCAL_MODE, true); conf.set("fs.default.name", "file:///"); conf.setBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH, true); conf.set("hive.execution.engine", "tez"); conf.set("hive.llap.execution.mode", "none"); conf.setBoolean("hive.in.ide.test", true); cheers!
