[jira] [Created] (HDFS-11589) Unable to remove dead node after datanode decommission
Chang Zong created HDFS-11589: - Summary: Unable to remove dead node after datanode decommission Key: HDFS-11589 URL: https://issues.apache.org/jira/browse/HDFS-11589 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Chang Zong After successfully decommissioned a datanode, the state of that node in web ui(http://:50070) is changed to "decommissioned". Then I stopped the datanode service in that host and the state is changed to "Dead", and the number of last contact keeps increasing. I tried to remove the hostname from the dfs.exclude and run "hdfs dfsadmin -refreshNodes" again, but that dead node is still there. Any way to solve that problem without restarting the whole cluster? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Can we update protobuf's version on trunk?
On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang wrote: > > > > > If unknown fields are dropped, then applications proxying tokens and > > other > > >> data between servers will effectively corrupt those messages, unless > we > > >> make everything opaque bytes, which- absent the convenient, > prenominate > > >> semantics managing the conversion- obviate the compatibility machinery > > that > > >> is the whole point of PB. Google is removing the features that > justified > > >> choosing PB over its alternatives. Since we can't require that our > > >> applications compile (or link) against our updated schema, this > creates > > a > > >> problem that PB was supposed to solve. > > > > > > > > > This is scary, and it potentially affects services outside of the > Hadoop > > > codebase. This makes it difficult to assess the impact. > > > > Stack mentioned a compatibility mode that uses the proto2 semantics. > > If that carries unknown fields through intermediate handlers, then > > this objection goes away. -C > > > Did some more googling, found this: > > https://groups.google.com/d/msg/protobuf/Z6pNo81FiEQ/fHkdcNtdAwAJ > > Feng Xiao appears to be a Google engineer, and suggests workarounds like > packing the fields into a byte type. No mention of a PB2 compatibility > mode. Also here: > > https://groups.google.com/d/msg/protobuf/bO2L6-_t91Q/-zIaJAR9AAAJ > > Participants say that unknown fields were dropped for automatic JSON > encoding, since you can't losslessly convert to JSON without knowing the > type. > > Unfortunately, it sounds like these are intrinsic differences with PB3. > > As I read it Andrew, the field-dropping happens when pb3 is running in proto3 'mode'. Let me try it... St.Ack > Best, > Andrew >
[jira] [Resolved] (HDFS-10640) Modify LOG statements to use sl4j templates.
[ https://issues.apache.org/jira/browse/HDFS-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas resolved HDFS-10640. -- Resolution: Later Closing this, given attention to solving this across the project in HADOOP-12956 > Modify LOG statements to use sl4j templates. > - > > Key: HDFS-10640 > URL: https://issues.apache.org/jira/browse/HDFS-10640 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-9809) Abstract implementation-specific details from the datanode
[ https://issues.apache.org/jira/browse/HDFS-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas resolved HDFS-9809. - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 > Abstract implementation-specific details from the datanode > -- > > Key: HDFS-9809 > URL: https://issues.apache.org/jira/browse/HDFS-9809 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Fix For: 3.0.0-alpha2 > > Attachments: DFSIO_benchmark_HDFS-9809.docx, HDFS-9809.001.patch, > HDFS-9809.002.patch, HDFS-9809.003.patch, HDFS-9809.004.patch > > > Multiple parts of the Datanode (FsVolumeSpi, ReplicaInfo, FSVolumeImpl etc.) > implicitly assume that blocks are stored in java.io.File(s) and that volumes > are divided into directories. We propose to abstract these details, which > would help in supporting other storages. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/ [Mar 28, 2017 2:33:26 PM] (jlowe) YARN-6359. TestRM#testApplicationKillAtAcceptedState fails rarely due to [Mar 28, 2017 3:02:07 PM] (yqlin) HDFS-11577. Combine the old and the new chooseRandom for better [Mar 28, 2017 8:23:20 PM] (varunsaxena) YARN-5368. Memory leak in timeline server (Jonathan Eagles via Varun [Mar 28, 2017 10:18:03 PM] (varunsaxena) YARN-6357. Implement putEntitiesAsync API in TimelineCollector (Haibo [Mar 29, 2017 1:36:24 AM] (aajisaka) YARN-6329. Remove unnecessary TODO comment from [Mar 29, 2017 5:14:03 AM] (wang) HDFS-10971. Distcp should not copy replication factor if source file is [Mar 29, 2017 6:11:48 AM] (rakeshr) HDFS-11541. Call RawErasureEncoder and RawErasureDecoder release() -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestShellBasedUnixGroupsMapping hadoop.metrics2.impl.TestMetricsSourceAdapter hadoop.security.TestRaceWhenRelogin hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.mapred.TestMRTimelineEventHandling Timed out junit tests : org.apache.hadoop.hdfs.server.namenode.TestEditLog org.apache.hadoop.hdfs.server.namenode.TestQuotaByStorageType org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/diff-compile-javac-root.txt [184K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [364K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [36K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/360/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_h
[jira] [Created] (HDFS-11591) Block Storage: fix TestCBlockServer test cleanup
Xiaoyu Yao created HDFS-11591: - Summary: Block Storage: fix TestCBlockServer test cleanup Key: HDFS-11591 URL: https://issues.apache.org/jira/browse/HDFS-11591 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: HDFS-7240 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor TestCBlockServer needs to clean up the CBlockManager after the test. Otherwise the RPC port bind exception will be thrown as reported in recent [Jenkins|https://builds.apache.org/job/PreCommit-HDFS-Build/18830/testReport/org.apache.hadoop.cblock/TestCBlockServer/org_apache_hadoop_cblock_TestCBlockServer/]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/ [Mar 28, 2017 2:33:26 PM] (jlowe) YARN-6359. TestRM#testApplicationKillAtAcceptedState fails rarely due to [Mar 28, 2017 3:02:07 PM] (yqlin) HDFS-11577. Combine the old and the new chooseRandom for better [Mar 28, 2017 8:23:20 PM] (varunsaxena) YARN-5368. Memory leak in timeline server (Jonathan Eagles via Varun [Mar 28, 2017 10:18:03 PM] (varunsaxena) YARN-6357. Implement putEntitiesAsync API in TimelineCollector (Haibo [Mar 29, 2017 1:36:24 AM] (aajisaka) YARN-6329. Remove unnecessary TODO comment from [Mar 29, 2017 5:14:03 AM] (wang) HDFS-10971. Distcp should not copy replication factor if source file is [Mar 29, 2017 6:11:48 AM] (rakeshr) HDFS-11541. Call RawErasureEncoder and RawErasureDecoder release() -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.server.diskbalancer.TestDiskBalancer hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-compile-root.txt [136K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-compile-root.txt [136K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-compile-root.txt [136K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [500K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [72K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/272/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K] https://builds.apache.org/job/h
Re: Can we update protobuf's version on trunk?
Is the below evidence enough that pb3 in proto2 syntax mode does not drop
'unknown' fields? (Maybe you want evidence that java tooling behaves the
same?)
To be clear, when we say proxy above, are we expecting that a pb message
deserialized by a process down-the-line that happens to have a crimped
proto definition that is absent a couple of fields somehow can re-serialize
and at the end of the line, all fields are present? Or are we talking
pass-through of the message without rewrite?
Thanks,
St.Ack
# Using the protoc v3.0.2 tool
$ protoc --version
libprotoc 3.0.2
# I have a simple proto definition with two fields in it
$ more pb.proto
message Test {
optional string one = 1;
optional string two = 2;
}
# This is a text-encoded instance of a 'Test' proto message:
$ more pb.txt
one: "one"
two: "two"
# Now I encode the above as a pb binary
$ protoc --encode=Test pb.proto < pb.txt > pb.bin
[libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
specified for the proto file: pb.proto. Please use 'syntax = "proto2";' or
'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
syntax.)
# Here is a dump of the binary
$ od -xc pb.bin
000 030a6e6f126574036f77
\n 003 o n e 022 003 t w o
012
# Here is a proto definition file that has a Test Message minus the 'two'
field.
$ more pb_drops_two.proto
message Test {
optional string one = 1;
}
# Use it to decode the bin file:
$ protoc --decode=Test pb_drops_two.proto < pb.bin
[libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
specified for the proto file: pb_drops_two.proto. Please use 'syntax =
"proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
to proto2 syntax.)
one: "one"
2: "two"
Note how the second field is preserved (absent a field name). It is not
dropped.
If I change the syntax of pb_drops_two.proto to be proto3, the field IS
dropped.
# Here proto file with proto3 syntax specified (had to drop the 'optional'
qualifier -- not allowed in proto3):
$ more pb_drops_two.proto
syntax = "proto3";
message Test {
string one = 1;
}
$ protoc --decode=Test pb_drops_two.proto < pb.bin > pb_drops_two.txt
$ more pb_drops_two.txt
one: "one"
I cannot reencode the text output using pb_drops_two.proto. It complains:
$ protoc --encode=Test pb_drops_two.proto < pb_drops_two.txt >
pb_drops_two.bin
[libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
specified for the proto file: pb_drops_two.proto. Please use 'syntax =
"proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
to proto2 syntax.)
input:2:1: Expected identifier, got: 2
Proto 2.5 does same:
$ ~/bin/protobuf-2.5.0/src/protoc --encode=Test pb_drops_two.proto <
pb_drops_two.txt > pb_drops_two.bin
input:2:1: Expected identifier.
Failed to parse input.
St.Ack
On Wed, Mar 29, 2017 at 10:14 AM, Stack wrote:
> On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang
> wrote:
>
>> >
>> > > If unknown fields are dropped, then applications proxying tokens and
>> > other
>> > >> data between servers will effectively corrupt those messages, unless
>> we
>> > >> make everything opaque bytes, which- absent the convenient,
>> prenominate
>> > >> semantics managing the conversion- obviate the compatibility
>> machinery
>> > that
>> > >> is the whole point of PB. Google is removing the features that
>> justified
>> > >> choosing PB over its alternatives. Since we can't require that our
>> > >> applications compile (or link) against our updated schema, this
>> creates
>> > a
>> > >> problem that PB was supposed to solve.
>> > >
>> > >
>> > > This is scary, and it potentially affects services outside of the
>> Hadoop
>> > > codebase. This makes it difficult to assess the impact.
>> >
>> > Stack mentioned a compatibility mode that uses the proto2 semantics.
>> > If that carries unknown fields through intermediate handlers, then
>> > this objection goes away. -C
>>
>>
>> Did some more googling, found this:
>>
>> https://groups.google.com/d/msg/protobuf/Z6pNo81FiEQ/fHkdcNtdAwAJ
>>
>> Feng Xiao appears to be a Google engineer, and suggests workarounds like
>> packing the fields into a byte type. No mention of a PB2 compatibility
>> mode. Also here:
>>
>> https://groups.google.com/d/msg/protobuf/bO2L6-_t91Q/-zIaJAR9AAAJ
>>
>> Participants say that unknown fields were dropped for automatic JSON
>> encoding, since you can't losslessly convert to JSON without knowing the
>> type.
>>
>> Unfortunately, it sounds like these are intrinsic differences with PB3.
>>
>>
> As I read it Andrew, the field-dropping happens when pb3 is running in
> proto3 'mode'. Let me try it...
>
> St.Ack
>
>
>
>> Best,
>> Andrew
>>
>
>
[jira] [Created] (HDFS-11592) Closing a file has a wasteful preconditions
Eric Badger created HDFS-11592:
--
Summary: Closing a file has a wasteful preconditions
Key: HDFS-11592
URL: https://issues.apache.org/jira/browse/HDFS-11592
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Eric Badger
Assignee: Eric Badger
When a file is closed, the NN checks if all the blocks are complete. Instead of
a simple 'if (!complete) throw new IllegalState(expensive-err-string)" it
invokes "Preconditions.checkStatus(complete, expensive-err-string)". The check
is done in a loop for all blocks, so more blocks = more penalty. The expensive
string should only be computed when an error actually occurs. A telltale sign
is seeing this in a stacktrace:
{noformat}
at java.lang.Class.getEnclosingMethod0(Native Method)
at java.lang.Class.getEnclosingMethodInfo(Class.java:1072)
at java.lang.Class.getEnclosingClass(Class.java:1272)
at java.lang.Class.getSimpleBinaryName(Class.java:1443)
at java.lang.Class.getSimpleName(Class.java:1309)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.assertAllBlocksComplete(INodeFile.java:246)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11593) Change SimpleHttpProxyHandler#exceptionCaught log level from info to debug
Xiaoyu Yao created HDFS-11593:
-
Summary: Change SimpleHttpProxyHandler#exceptionCaught log level
from info to debug
Key: HDFS-11593
URL: https://issues.apache.org/jira/browse/HDFS-11593
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Xiaoyu Yao
Assignee: Xiaobing Zhou
Priority: Minor
A busy datanode may have many client disconnect exception logged with stack
like below, which does not provide much useful information. Propose to reduce
the log level from info to debug.
{code}
2017-03-29 20:28:55,225 INFO web.DatanodeHttpServer
(SimpleHttpProxyHandler.java:exceptionCaught(147)) - Proxy for / failed. cause:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:446)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Can we update protobuf's version on trunk?
On Wed, Mar 29, 2017 at 1:13 PM, Stack wrote:
> Is the below evidence enough that pb3 in proto2 syntax mode does not drop
> 'unknown' fields? (Maybe you want evidence that java tooling behaves the
> same?)
I reproduced your example with the Java tooling, including changing
some of the fields in the intermediate representation. As long as the
syntax is "proto2", it seems to have compatible semantics.
> To be clear, when we say proxy above, are we expecting that a pb message
> deserialized by a process down-the-line that happens to have a crimped proto
> definition that is absent a couple of fields somehow can re-serialize and at
> the end of the line, all fields are present? Or are we talking pass-through
> of the message without rewrite?
The former; an intermediate handler decoding, [modifying,] and
encoding the record without losing unknown fields.
This looks fine. -C
> Thanks,
> St.Ack
>
>
> # Using the protoc v3.0.2 tool
> $ protoc --version
> libprotoc 3.0.2
>
> # I have a simple proto definition with two fields in it
> $ more pb.proto
> message Test {
> optional string one = 1;
> optional string two = 2;
> }
>
> # This is a text-encoded instance of a 'Test' proto message:
> $ more pb.txt
> one: "one"
> two: "two"
>
> # Now I encode the above as a pb binary
> $ protoc --encode=Test pb.proto < pb.txt > pb.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb.proto. Please use 'syntax = "proto2";' or
> 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
> syntax.)
>
> # Here is a dump of the binary
> $ od -xc pb.bin
> 000 030a6e6f126574036f77
> \n 003 o n e 022 003 t w o
> 012
>
> # Here is a proto definition file that has a Test Message minus the 'two'
> field.
> $ more pb_drops_two.proto
> message Test {
> optional string one = 1;
> }
>
> # Use it to decode the bin file:
> $ protoc --decode=Test pb_drops_two.proto < pb.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> one: "one"
> 2: "two"
>
> Note how the second field is preserved (absent a field name). It is not
> dropped.
>
> If I change the syntax of pb_drops_two.proto to be proto3, the field IS
> dropped.
>
> # Here proto file with proto3 syntax specified (had to drop the 'optional'
> qualifier -- not allowed in proto3):
> $ more pb_drops_two.proto
> syntax = "proto3";
> message Test {
> string one = 1;
> }
>
> $ protoc --decode=Test pb_drops_two.proto < pb.bin > pb_drops_two.txt
> $ more pb_drops_two.txt
> one: "one"
>
>
> I cannot reencode the text output using pb_drops_two.proto. It complains:
>
> $ protoc --encode=Test pb_drops_two.proto < pb_drops_two.txt >
> pb_drops_two.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> input:2:1: Expected identifier, got: 2
>
> Proto 2.5 does same:
>
> $ ~/bin/protobuf-2.5.0/src/protoc --encode=Test pb_drops_two.proto <
> pb_drops_two.txt > pb_drops_two.bin
> input:2:1: Expected identifier.
> Failed to parse input.
>
> St.Ack
>
>
>
>
>
>
> On Wed, Mar 29, 2017 at 10:14 AM, Stack wrote:
>>
>> On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang
>> wrote:
>>>
>>> >
>>> > > If unknown fields are dropped, then applications proxying tokens and
>>> > other
>>> > >> data between servers will effectively corrupt those messages, unless
>>> > >> we
>>> > >> make everything opaque bytes, which- absent the convenient,
>>> > >> prenominate
>>> > >> semantics managing the conversion- obviate the compatibility
>>> > >> machinery
>>> > that
>>> > >> is the whole point of PB. Google is removing the features that
>>> > >> justified
>>> > >> choosing PB over its alternatives. Since we can't require that our
>>> > >> applications compile (or link) against our updated schema, this
>>> > >> creates
>>> > a
>>> > >> problem that PB was supposed to solve.
>>> > >
>>> > >
>>> > > This is scary, and it potentially affects services outside of the
>>> > > Hadoop
>>> > > codebase. This makes it difficult to assess the impact.
>>> >
>>> > Stack mentioned a compatibility mode that uses the proto2 semantics.
>>> > If that carries unknown fields through intermediate handlers, then
>>> > this objection goes away. -C
>>>
>>>
>>> Did some more googling, found this:
>>>
>>> https://groups.google.com/d/msg/protobuf/Z6pNo81FiEQ/fHkdcNtdAwAJ
>>>
>>> Feng Xiao appears to be a Google engineer, and suggests workarounds like
>>> packing the fields into a byte type. No mention of a PB2 compatibility
>>> mode. Also here:
>>>
>>> https://groups.google.com/d/msg/protobuf/bO2L6-_t91Q/-zIaJAR9AAAJ
>>>
>>> Participants say that un
[jira] [Created] (HDFS-11594) Ozone: close container should call compactDB
Anu Engineer created HDFS-11594: --- Summary: Ozone: close container should call compactDB Key: HDFS-11594 URL: https://issues.apache.org/jira/browse/HDFS-11594 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Anu Engineer Assignee: Anu Engineer Closing a container should call compactDB to compact the LevelDB. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Can we update protobuf's version on trunk?
On Wed, Mar 29, 2017 at 3:12 PM, Chris Douglas
wrote:
> On Wed, Mar 29, 2017 at 1:13 PM, Stack wrote:
> > Is the below evidence enough that pb3 in proto2 syntax mode does not drop
> > 'unknown' fields? (Maybe you want evidence that java tooling behaves the
> > same?)
>
> I reproduced your example with the Java tooling, including changing
> some of the fields in the intermediate representation. As long as the
> syntax is "proto2", it seems to have compatible semantics.
>
>
Thanks.
> > To be clear, when we say proxy above, are we expecting that a pb message
> > deserialized by a process down-the-line that happens to have a crimped
> proto
> > definition that is absent a couple of fields somehow can re-serialize
> and at
> > the end of the line, all fields are present? Or are we talking
> pass-through
> > of the message without rewrite?
>
> The former; an intermediate handler decoding, [modifying,] and
> encoding the record without losing unknown fields.
>
>
I did not try this. Did you? Otherwise I can.
St.Ack
> This looks fine. -C
>
> > Thanks,
> > St.Ack
> >
> >
> > # Using the protoc v3.0.2 tool
> > $ protoc --version
> > libprotoc 3.0.2
> >
> > # I have a simple proto definition with two fields in it
> > $ more pb.proto
> > message Test {
> > optional string one = 1;
> > optional string two = 2;
> > }
> >
> > # This is a text-encoded instance of a 'Test' proto message:
> > $ more pb.txt
> > one: "one"
> > two: "two"
> >
> > # Now I encode the above as a pb binary
> > $ protoc --encode=Test pb.proto < pb.txt > pb.bin
> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> > specified for the proto file: pb.proto. Please use 'syntax = "proto2";'
> or
> > 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
> > syntax.)
> >
> > # Here is a dump of the binary
> > $ od -xc pb.bin
> > 000 030a6e6f126574036f77
> > \n 003 o n e 022 003 t w o
> > 012
> >
> > # Here is a proto definition file that has a Test Message minus the 'two'
> > field.
> > $ more pb_drops_two.proto
> > message Test {
> > optional string one = 1;
> > }
> >
> > # Use it to decode the bin file:
> > $ protoc --decode=Test pb_drops_two.proto < pb.bin
> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> > specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> > "proto2";' or 'syntax = "proto3";' to specify a syntax version.
> (Defaulted
> > to proto2 syntax.)
> > one: "one"
> > 2: "two"
> >
> > Note how the second field is preserved (absent a field name). It is not
> > dropped.
> >
> > If I change the syntax of pb_drops_two.proto to be proto3, the field IS
> > dropped.
> >
> > # Here proto file with proto3 syntax specified (had to drop the
> 'optional'
> > qualifier -- not allowed in proto3):
> > $ more pb_drops_two.proto
> > syntax = "proto3";
> > message Test {
> > string one = 1;
> > }
> >
> > $ protoc --decode=Test pb_drops_two.proto < pb.bin > pb_drops_two.txt
> > $ more pb_drops_two.txt
> > one: "one"
> >
> >
> > I cannot reencode the text output using pb_drops_two.proto. It complains:
> >
> > $ protoc --encode=Test pb_drops_two.proto < pb_drops_two.txt >
> > pb_drops_two.bin
> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> > specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> > "proto2";' or 'syntax = "proto3";' to specify a syntax version.
> (Defaulted
> > to proto2 syntax.)
> > input:2:1: Expected identifier, got: 2
> >
> > Proto 2.5 does same:
> >
> > $ ~/bin/protobuf-2.5.0/src/protoc --encode=Test pb_drops_two.proto <
> > pb_drops_two.txt > pb_drops_two.bin
> > input:2:1: Expected identifier.
> > Failed to parse input.
> >
> > St.Ack
> >
> >
> >
> >
> >
> >
> > On Wed, Mar 29, 2017 at 10:14 AM, Stack wrote:
> >>
> >> On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang
> >> wrote:
> >>>
> >>> >
> >>> > > If unknown fields are dropped, then applications proxying tokens
> and
> >>> > other
> >>> > >> data between servers will effectively corrupt those messages,
> unless
> >>> > >> we
> >>> > >> make everything opaque bytes, which- absent the convenient,
> >>> > >> prenominate
> >>> > >> semantics managing the conversion- obviate the compatibility
> >>> > >> machinery
> >>> > that
> >>> > >> is the whole point of PB. Google is removing the features that
> >>> > >> justified
> >>> > >> choosing PB over its alternatives. Since we can't require that our
> >>> > >> applications compile (or link) against our updated schema, this
> >>> > >> creates
> >>> > a
> >>> > >> problem that PB was supposed to solve.
> >>> > >
> >>> > >
> >>> > > This is scary, and it potentially affects services outside of the
> >>> > > Hadoop
> >>> > > codebase. This makes it difficult to assess the impact.
> >>> >
> >>> > Stack mentioned a compatibility mode that uses the proto2 semantics.
> >>> > If that carries unknown fields through intermediate handlers, then
> >>> > this obje
[jira] [Created] (HDFS-11595) Clean up all the create() overloads in DFSClient
SammiChen created HDFS-11595: Summary: Clean up all the create() overloads in DFSClient Key: HDFS-11595 URL: https://issues.apache.org/jira/browse/HDFS-11595 Project: Hadoop HDFS Issue Type: Improvement Reporter: SammiChen A follow on of HDFS-10996 discussion. Clean up all the create() overloads in DFSClient, they seem unnecessary since DistributedFileSystem also has very similar overloads for filling in defaults. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11596) hadoop-hdfs-client jar is in the wrong directory in release tarball
Andrew Wang created HDFS-11596:
--
Summary: hadoop-hdfs-client jar is in the wrong directory in
release tarball
Key: HDFS-11596
URL: https://issues.apache.org/jira/browse/HDFS-11596
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Affects Versions: 3.0.0-alpha2, 2.8.0
Reporter: Andrew Wang
Priority: Critical
Mentioned by [~aw] on HDFS-11356. The hdfs-client jar is in the lib directory
rather than with the other hadoop jars:
>From the alpha2 artifacts:
{noformat}
-> % find . -name "*hdfs-client*.jar"
./share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/hadoop-hdfs-client-3.0.0-alpha2.jar
./share/hadoop/hdfs/sources/hadoop-hdfs-client-3.0.0-alpha2-sources.jar
./share/hadoop/hdfs/sources/hadoop-hdfs-client-3.0.0-alpha2-test-sources.jar
./share/hadoop/hdfs/lib/hadoop-hdfs-client-3.0.0-alpha2.jar
./share/hadoop/hdfs/hadoop-hdfs-client-3.0.0-alpha2-tests.jar
{noformat}
Strangely enough, the tests jar is in the right place.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11597) Ozone: Add Ratis management API
Tsz Wo Nicholas Sze created HDFS-11597: -- Summary: Ozone: Add Ratis management API Key: HDFS-11597 URL: https://issues.apache.org/jira/browse/HDFS-11597 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze We need an API to manage raft clusters, e.g. - RaftClusterId createRaftCluster(MembershipConfiguration) - void closeRaftCluster(RaftClusterId) - MembershipConfiguration getMembers(RaftClusterId) - void changeMembership(RaftClusterId, newMembershipConfiguration) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
