[jira] [Resolved] (HDFS-15976) Make mkdtemp cross platform
[ https://issues.apache.org/jira/browse/HDFS-15976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15976. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make mkdtemp cross platform > --- > > Key: HDFS-15976 > URL: https://issues.apache.org/jira/browse/HDFS-15976 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > mkdtemp is used for creating temporary directory, adhering to the given > pattern. It's not available on Visual C++. Need to make this cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16174) Refactor TempFile and TempDir in libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-16174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16174. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Refactor TempFile and TempDir in libhdfs++ > -- > > Key: HDFS-16174 > URL: https://issues.apache.org/jira/browse/HDFS-16174 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > In C++, we generally do the declaration in the header files and the > corresponding implementation in the .cc files. Here we see that the > implementation of TempFile and TempDir are done in configuration_test.h > itself. This offers no benefit and the compilation of TempFile and TempDir > classes are duplicated for every #include of the configuration_test.h header. > Thus, we need to implement it in separate cc files to avoid this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16178) Make recursive rmdir in libhdfs++ cross platform
[ https://issues.apache.org/jira/browse/HDFS-16178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16178. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make recursive rmdir in libhdfs++ cross platform > > > Key: HDFS-16178 > URL: https://issues.apache.org/jira/browse/HDFS-16178 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The *TempDir* class in libhdfs++ is currently using nftw API provided by > *ftw.h*, which is only present in Linux and not on Windows. Need to use the > APIs from C++17 *std::filesystem* to make this cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16205) Make hdfs_allowSnapshot tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16205. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make hdfs_allowSnapshot tool cross platform > --- > > Key: HDFS-16205 > URL: https://issues.apache.org/jira/browse/HDFS-16205 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > The source files for hdfs_allowSnapshot uses *getopt* for parsing the command > line arguments. getopt is available only on Linux and thus, isn't cross > platform. We need to replace getopt with *boost::program_options* to make > this cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16250) Refactor AllowSnapshotMock using GMock
[ https://issues.apache.org/jira/browse/HDFS-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16250. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Refactor AllowSnapshotMock using GMock > -- > > Key: HDFS-16250 > URL: https://issues.apache.org/jira/browse/HDFS-16250 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently, the > [implementation|https://github.com/apache/hadoop/blob/35a8d48872a13438d4c4199b6ef5b902105e2eb2/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/tools/hdfs-allow-snapshot-mock.cc] > of AllowSnapshotMock is quite basic. Need to replace this with GMock so that > we can tap into the benefits offered by GMock. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16254) Cleanup protobuf on exit of hdfs_allowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16254. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Cleanup protobuf on exit of hdfs_allowSnapshot > -- > > Key: HDFS-16254 > URL: https://issues.apache.org/jira/browse/HDFS-16254 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Need to move the call google::protobuf::ShutdownProtobufLibrary() to main > method instead of > [AllowSnapshot::HandlePath|https://github.com/apache/hadoop/blob/35a8d48872a13438d4c4199b6ef5b902105e2eb2/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-allow-snapshot/hdfs-allow-snapshot.cc#L116-L117] > since we want the clean-up tasks to run only when the program exits. > The current implementation doesn't cause any issues since > AllowSnapshot::HandlePath is called only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16251) Make hdfs_cat tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16251. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make hdfs_cat tool cross platform > - > > Key: HDFS-16251 > URL: https://issues.apache.org/jira/browse/HDFS-16251 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The source files for hdfs_cat uses *getopt* for parsing the command line > arguments. getopt is available only on Linux and thus, isn't cross platform. > We need to replace getopt with *boost::program_options* to make this cross > platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16263) Add CMakeLists for hdfs_allowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16263. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add CMakeLists for hdfs_allowSnapshot > - > > Key: HDFS-16263 > URL: https://issues.apache.org/jira/browse/HDFS-16263 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, hdfs_allowSnapshot is built in it's [parent directory's > CMakeLists.txt|https://github.com/apache/hadoop/blob/95b537ee6a9ff3082c9ad9bc773f86fd4be04e50/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/CMakeLists.txt#L83-L89]. > Need to move this into a separate CMakeLists.txt file under > hdfs-allow-snapshot so that it's more modular. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16265) Refactor HDFS tool tests for better reuse
[ https://issues.apache.org/jira/browse/HDFS-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16265. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Refactor HDFS tool tests for better reuse > - > > Key: HDFS-16265 > URL: https://issues.apache.org/jira/browse/HDFS-16265 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Blocker > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently, the test cases written in hdfs-tool-test.h isn't easy to reuse. > Primarily because the expectations are different for each HDFS tool. I > realized this while I was creating the PR for HDFS-16260. For instance, > passing more than one argument is erroneous to hdfs_allowSnapshot while it's > the only valid scenario for hdfs_deleteSnapshot. > Thus, it won't be possible to reuse the test cases without decoupling the > expectations from the test case definitions. The solution here is to move the > expectations to the corresponding mock classes and invoke the call to set > them up in the test cases after the creation of mock instances. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16260) Make hdfs_deleteSnapshot tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16260. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make hdfs_deleteSnapshot tool cross platform > > > Key: HDFS-16260 > URL: https://issues.apache.org/jira/browse/HDFS-16260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The source files for hdfs_deleteSnapshot uses *getopt* for parsing the > command line arguments. getopt is available only on Linux and thus, isn't > cross platform. We need to replace getopt with *boost::program_options* to > make this cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16267) Make hdfs_df tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16267. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make hdfs_df tool cross platform > > > Key: HDFS-16267 > URL: https://issues.apache.org/jira/browse/HDFS-16267 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: libhdfscpp, pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The source files for hdfs_df uses *getopt* for parsing the command line > arguments. getopt is available only on Linux and thus, isn't cross platform. > We need to replace getopt with *boost::program_options* to make this cross > platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16278) Make HDFS snapshot tools cross platform
[ https://issues.apache.org/jira/browse/HDFS-16278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16278. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make HDFS snapshot tools cross platform > --- > > Key: HDFS-16278 > URL: https://issues.apache.org/jira/browse/HDFS-16278 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: libhdfscpp, pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The source files for *hdfs_createSnapshot*, *hdfs_disallowSnapshot* and > *hdfs_renameSnapshot* uses getopt for parsing the command line arguments. > getopt is available only on Linux and thus, isn't cross platform. We need to > replace getopt with boost::program_options to make these tools cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16300) Use libcrypto in Windows for libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-16300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16300. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Use libcrypto in Windows for libhdfspp > -- > > Key: HDFS-16300 > URL: https://issues.apache.org/jira/browse/HDFS-16300 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 > Environment: Windows >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Blocker > Labels: libhdfscpp, pull-request-available > Fix For: 3.4.0 > > Attachments: build-log-hdfs-nacl-windows-10.log > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, eay32 is the library that's used in libhdfspp for Windows. > Whereas, we use libcrypto for the rest of the platforms. As per the following > mail thread, the OpenSSL library was renamed from eay32 to libcrypto from > OpenSSL version 1.1.0 onwards - > https://mta.openssl.org/pipermail/openssl-dev/2016-August/008351.html. > Thus, we need to use libcrypto on Windows as well to ensure that we > standardize the version of the OpenSSL libraries used across platforms. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability
[ https://issues.apache.org/jira/browse/HDFS-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16307. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve HdfsBlockPlacementPolicies docs readability > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16411) RBF: RouterId is NULL when set dfs.federation.router.rpc.enable=false
[ https://issues.apache.org/jira/browse/HDFS-16411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16411. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: Íñigo Goiri Resolution: Fixed > RBF: RouterId is NULL when set dfs.federation.router.rpc.enable=false > - > > Key: HDFS-16411 > URL: https://issues.apache.org/jira/browse/HDFS-16411 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: YulongZ >Assignee: Íñigo Goiri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HDFS-16411.000.patch, HDFS-16411.001.patch, > HDFS-16411.002.patch, HDFS-16411.003.patch, HDFS-16411.004.patch, > HDFS-16411.005.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > When dfs.federation.router.rpc.enable=false, routerid is null, but > RouterHeartbeatService need updateStateStore() with routerId. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16451) RBF: Add search box for Router's tab-mounttable web page
[ https://issues.apache.org/jira/browse/HDFS-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16451. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Add search box for Router's tab-mounttable web page > > > Key: HDFS-16451 > URL: https://issues.apache.org/jira/browse/HDFS-16451 > Project: Hadoop HDFS > Issue Type: New Feature > Components: rbf >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2022-02-09-18-17-53-498.png, > image-2022-02-09-18-18-29-262.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > In our cluster, we have mount many paths in HDFS Router and it may lead to > take some time to load the mount-table page of Router when we open it in the > browser. > In order to use the mount-table page more conveniently, maybe we should add a > search box style, just like the screenshot below > !image-2022-02-09-18-17-53-498.png! > !image-2022-02-09-18-18-29-262.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16447) RBF: Registry HDFS Router's RPCServer & RPCClient metrics for PrometheusSink
[ https://issues.apache.org/jira/browse/HDFS-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16447. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Registry HDFS Router's RPCServer & RPCClient metrics for PrometheusSink > > > Key: HDFS-16447 > URL: https://issues.apache.org/jira/browse/HDFS-16447 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When we enable PrometheusSink for HDFS Router, Router' prometheus sink miss > some metrics, for example `RpcClientNumActiveConnections` and so on. > > We need registry some Router's rpcserver & rpcclient metrics for > PrometheusSink. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16440) RBF: Support router get HAServiceStatus with Lifeline RPC address
[ https://issues.apache.org/jira/browse/HDFS-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16440. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Support router get HAServiceStatus with Lifeline RPC address > - > > Key: HDFS-16440 > URL: https://issues.apache.org/jira/browse/HDFS-16440 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: YulongZ >Assignee: YulongZ >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HDFS-16440.001.patch, HDFS-16440.003.patch, > HDFS-16440.004.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > NamenodeHeartbeatService gets HAServiceStatus using > NNHAServiceTarget.getProxy. When we set a special > dfs.namenode.lifeline.rpc-address , NamenodeHeartbeatService may get > HAServiceStatus using NNHAServiceTarget.getHealthMonitorProxy. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16459) RBF: register RBFMetrics in MetricsSystem for promethuessink
[ https://issues.apache.org/jira/browse/HDFS-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16459. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: register RBFMetrics in MetricsSystem for promethuessink > > > Key: HDFS-16459 > URL: https://issues.apache.org/jira/browse/HDFS-16459 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Router' RBFMetrics was not register in MetricsSystem. We can't find these > metrics from PrometheusSink. Maybe we should fix it. > > After fix it , some RBFMetrics will export like this > {code:java} > # HELP rbf_metrics_current_tokens_count Number of router's current tokens > # TYPE rbf_metrics_current_tokens_count gauge > rbf_metrics_current_tokens_count{processname="Router",context="dfs",hostname=""} > 2 {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16577) Let administrator override connection details when registering datanodes
Teo Klestrup Röijezon created HDFS-16577: Summary: Let administrator override connection details when registering datanodes Key: HDFS-16577 URL: https://issues.apache.org/jira/browse/HDFS-16577 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 3.2.2 Reporter: Teo Klestrup Röijezon Currently (as of 3.2.2, but reading through the release notes this doesn't seem to have changed since then) DataNodes use the same properties for deciding which port to bind each service to, as for deciding which ports are included in the `DatanodeRegistration` sent to the NameNode. Further, NameNodes overwrite the DataNode's IP address with the incoming address during registration. Both of these prevent external users from connecting to DataNodes that are hosted behind some sort of NAT (such as Kubernetes). I have created a spike branch ([https://github.com/stackabletech/hadoop/tree/spike/override-datanode-id,] based on v3.2.2) that I have confirmed solves this problem for us. There's clearly some work to be done integrating this properly (such as using the regular Hadoop config system and falling back to the old behaviour if no override is configured). I'd be happy to take that on to the best of my ability (with the caveats that I'm not super familiar with the Hadoop codebase, and that my Java is quite rusty at this point) if the overall direction seems acceptable. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16640) RBF: Show datanode IP list when click DN histogram in Router
[ https://issues.apache.org/jira/browse/HDFS-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16640. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Show datanode IP list when click DN histogram in Router > > > Key: HDFS-16640 > URL: https://issues.apache.org/jira/browse/HDFS-16640 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Show datanode IP list when click DN histogram in Router. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16605) Improve Code With Lambda in hadoop-hdfs-rbf moudle
[ https://issues.apache.org/jira/browse/HDFS-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16605. Fix Version/s: 3.4.0 Hadoop Flags: Incompatible change Resolution: Fixed > Improve Code With Lambda in hadoop-hdfs-rbf moudle > -- > > Key: HDFS-16605 > URL: https://issues.apache.org/jira/browse/HDFS-16605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16660) Improve Code With Lambda in IPCLoggerChannel class
[ https://issues.apache.org/jira/browse/HDFS-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16660. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve Code With Lambda in IPCLoggerChannel class > -- > > Key: HDFS-16660 > URL: https://issues.apache.org/jira/browse/HDFS-16660 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Improve Code With Lambda in IPCLoggerChannel class -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16687) RouterFsckServlet replicates code from DfsServlet base class
[ https://issues.apache.org/jira/browse/HDFS-16687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16687. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RouterFsckServlet replicates code from DfsServlet base class > > > Key: HDFS-16687 > URL: https://issues.apache.org/jira/browse/HDFS-16687 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Reporter: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > RouterFsckServlet replicates the method "getUGI(HttpServletRequest request, > Configuration conf)" from DfsServlet instead of just extending DfsServlet. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets
[ https://issues.apache.org/jira/browse/HDFS-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-13274. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Extend RouterRpcClient to use multiple sockets > --- > > Key: HDFS-13274 > URL: https://issues.apache.org/jira/browse/HDFS-13274 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > HADOOP-13144 introduces the ability to create multiple connections for the > same user and use different sockets. The RouterRpcClient should use this > approach to get a better throughput. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16678) RBF supports disable getNodeUsage() in RBFMetrics
[ https://issues.apache.org/jira/browse/HDFS-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16678. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF supports disable getNodeUsage() in RBFMetrics > - > > Key: HDFS-16678 > URL: https://issues.apache.org/jira/browse/HDFS-16678 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > In our prod environment, we try to collect RBF metrics every 15s through > jmx_exporter. And we found that collection task often failed. > After tracing and found that the collection task is blocked at getNodeUsage() > in RBFMetrics, because it will collection all datanode's usage from > downstream nameservices. This is a very expensive and almost useless > operation. Because in most scenarios, each NameSerivce contains almost the > same DNs. We can get the data usage's from any one nameservices, not from RBF. > So I feel that RBF should supports disable getNodeUsage() in RBFMetrics. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16705) RBF: Support healthMonitor timeout configurable and cache NN and client proxy in NamenodeHeartbeatService
[ https://issues.apache.org/jira/browse/HDFS-16705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16705. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Support healthMonitor timeout configurable and cache NN and client proxy > in NamenodeHeartbeatService > - > > Key: HDFS-16705 > URL: https://issues.apache.org/jira/browse/HDFS-16705 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > When I read NamenodeHeartbeatService.class of RBF, I feel that there are > somethings we can do for NamenodeHeartbeatService.class. > * Cache NameNode Protocol and Client Protocol to avoid creating a new proxy > every time > * Supports healthMonitorTimeout configuration > * Format code of getNamenodeStatusReport to make it clearer -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16724) RBF should support get the information about ancestor mount points
[ https://issues.apache.org/jira/browse/HDFS-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16724. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF should support get the information about ancestor mount points > -- > > Key: HDFS-16724 > URL: https://issues.apache.org/jira/browse/HDFS-16724 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Suppose RBF cluster have 2 nameservices and to mount point as below: > * /user/ns1 -> ns1 -> /user/ns1 > * /usre/ns2 -> ns2 -> /user/ns2 > Suppose we disable default nameservice of the RBF cluster and try to > getFileInfo of the path /user, RBF will throw one IOException to client due > to can not find locations for path /user. > But as this case, RBF should should return one valid response to client, > because /user has two sub mount point ns1 and ns2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16736) Link to Boost library in libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16736. Hadoop Flags: Reviewed Resolution: Fixed > Link to Boost library in libhdfspp > -- > > Key: HDFS-16736 > URL: https://issues.apache.org/jira/browse/HDFS-16736 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 > Environment: Windows 10 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: libhdfscpp, pull-request-available > > The compilation of HDFS Native Client fails on Windows 10 due to the > following error - > {code} > [exec] > "H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj" > (default target) (105) -> > [exec] rpc.lib(rpc_engine.obj) : error LNK2019: unresolved external symbol > "__declspec(dllimport) public: __cdecl > boost::gregorian::greg_month::greg_month(unsigned short)" > (__imp_??0greg_month@gregorian@boost@@QEAA@G@Z) referenced in function > "private: static class boost::posix_time::ptime __cdecl > boost::date_time::microsec_clock boost::posix_time::ptime>::create_time(struct tm * (__cdecl*)(__int64 const > *,struct tm *))" > (?create_time@?$microsec_clock@Vptime@posix_time@boost@@@date_time@boost@@CA?AVptime@posix_time@3@P6APEAUtm@@PEB_JPEAU6@@Z@Z) > > [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj] > [exec] rpc.lib(request.obj) : error LNK2001: unresolved external symbol > "__declspec(dllimport) public: __cdecl > boost::gregorian::greg_month::greg_month(unsigned short)" > (__imp_??0greg_month@gregorian@boost@@QEAA@G@Z) > [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj] > [exec] > H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\RelWithDebInfo\logging_test.exe > : fatal error LNK1120: 1 unresolved externals > [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj] > {code} > Thus, we need to link against the Boost library to resolve this error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16734) RBF: fix some bugs when handling getContentSummary RPC
[ https://issues.apache.org/jira/browse/HDFS-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16734. Fix Version/s: 3.4.0 Resolution: Fixed > RBF: fix some bugs when handling getContentSummary RPC > -- > > Key: HDFS-16734 > URL: https://issues.apache.org/jira/browse/HDFS-16734 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Suppose there are some mount points as bellows in RBF without default > namespace. > ||Source Path||NameSpace||Destination Path || > |/a/b|ns0|/a/b| > |/a/b/c|ns0|/a/b/c| > |/a/b/c/d|ns1|/a/b/c/d| > Suppose there a file /a/b/c/file1 with 10MB data in ns0 and a file > /a/b/c/d/file2 with 20MB data in ns1. > There are bugs during handling some cases: > ||Case Number||Case||Current Result||Expected Result|| > |1|getContentSummary('/a')|Throw RouterResolveException |2files and 30MB data| > |2|getContentSummary('/a/b')|2files and 40MB data|3files and 40MB data| > Bugs for these cases: > Case1: If can't find any locations for the path, RBF should try to do it > with sub mount points. > Case2: RBF shouldn't repeatedly get content summary from one same namespace > with same ancestors path, such as from ns0 with /a/b and from ns0 with /a/b/c. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16750) NameNode should use NameNode.getRemoteUser() to log audit event to avoid possible NPE
[ https://issues.apache.org/jira/browse/HDFS-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16750. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > NameNode should use NameNode.getRemoteUser() to log audit event to avoid > possible NPE > -- > > Key: HDFS-16750 > URL: https://issues.apache.org/jira/browse/HDFS-16750 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > NameNode should use NameNode.getRemoteUser() to log audit event to avoid > possible NPE. > The relate code is: > {code:java} > private void logAuditEvent(boolean succeeded, String cmd, String src, > String dst, FileStatus stat) throws IOException { > if (isAuditEnabled() && isExternalInvocation()) { > logAuditEvent(succeeded, Server.getRemoteUser(), Server.getRemoteIp(), > cmd, src, dst, stat); > } > } > // the ugi may be null. > private void logAuditEvent(boolean succeeded, > UserGroupInformation ugi, InetAddress addr, String cmd, String src, > String dst, FileStatus status) { > final String ugiStr = ugi.toString(); > ... > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16770) [Documentation] RBF: Duplicate statement to be removed for better readabilty
[ https://issues.apache.org/jira/browse/HDFS-16770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16770. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Documentation] RBF: Duplicate statement to be removed for better readabilty > > > Key: HDFS-16770 > URL: https://issues.apache.org/jira/browse/HDFS-16770 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Both the below 2 statements gives the same meaning, later one can be removed. > The Router monitors the local NameNode and its state and heartbeats to the > State Store. > The Router monitors the local NameNode and heartbeats the state to the State > Store. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16772) refreshHostsReader should use the new configuration
[ https://issues.apache.org/jira/browse/HDFS-16772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16772. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > refreshHostsReader should use the new configuration > --- > > Key: HDFS-16772 > URL: https://issues.apache.org/jira/browse/HDFS-16772 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > `refreshHostsReader` should use the latest configuration. > And the current code as bellow: > {code:java} > /** Reread include/exclude files. */ > private void refreshHostsReader(Configuration conf) throws IOException { > if (conf == null) { > conf = new HdfsConfiguration(); > // BUG here > this.hostConfigManager.setConf(conf); > } > this.hostConfigManager.refresh(); > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16777) datatables@1.10.17 sonatype-2020-0988 vulnerability
[ https://issues.apache.org/jira/browse/HDFS-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16777. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > datatables@1.10.17 sonatype-2020-0988 vulnerability > > > Key: HDFS-16777 > URL: https://issues.apache.org/jira/browse/HDFS-16777 > Project: Hadoop HDFS > Issue Type: Bug > Components: ui >Affects Versions: 3.3.4 >Reporter: Eugene Shinn (Truveta) >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Our static analysis security tool detected that HDFS's UI currently includes > a vulnerable version of datatables detected by Sonatype (sonatype-2020-0988). > From the vulnerability description: > _"The `datatables.net` package is vulnerable to Prototype Pollution. The > `setData` function in `jquery.dataTables.js` fails to protect prototype > attributes when objects are created during the application's execution. A > remote attacker can exploit this to modify the behavior of object prototypes > which, depending on their use in the application, may result in a Denial of > Service (DoS), Remote Code Execution (RCE), or other unexpected execution > flow."_ > This issue was addressed in v 1.11.5 (ref: [Fix: Protect developers from > inadvertantely introducing prototype pol… · > DataTables/Dist-DataTables@e2e19ea > (github.com)).|https://github.com/DataTables/Dist-DataTables/commit/e2e19eac7e5a6f140d7eefca5c7deba165b357eb#diff-e7d8309f017dd2ef6385fa8cdc1539a2R2765] > N.B. this issue was also detected within the YARN UI as well. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl
[ https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16848. Hadoop Flags: Reviewed Resolution: Fixed > RBF: Improve StateStoreZookeeperImpl > - > > Key: HDFS-16848 > URL: https://issues.apache.org/jira/browse/HDFS-16848 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Sun Hao >Assignee: Sun Hao >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Currently, router is getting/updating state from zk sequentially. It will > slowdown router load/update state cache especially for a large cluster or a > multi region cluster. > We propose adding a threadpool to deal with zk state synchronization。 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16900) Method DataNode#isWrite seems not working in DataTransfer constructor method
[ https://issues.apache.org/jira/browse/HDFS-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16900. Resolution: Won't Fix > Method DataNode#isWrite seems not working in DataTransfer constructor method > > > Key: HDFS-16900 > URL: https://issues.apache.org/jira/browse/HDFS-16900 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.3.4 >Reporter: ZhangHB >Priority: Major > > In constructor method of DataTransfer, there is codes below: > {code:java} > if (isTransfer(stage, clientname)) { > this.throttler = xserver.getTransferThrottler(); > } else if(isWrite(stage)) { > this.throttler = xserver.getWriteThrottler(); > } {code} > the stage is a parameter of DataTransfer Constructor. Let us see where > instantiate DataTransfer object. > In method transferReplicaForPipelineRecovery, codes like below: > {code:java} > final DataTransfer dataTransferTask = new DataTransfer(targets, > targetStorageTypes, targetStorageIds, b, stage, client); {code} > but the stage can never be PIPELINE_SETUP_STREAMING_RECOVERY or > PIPELINE_SETUP_APPEND_RECOVERY. > It can only be TRANSFER_RBW or TRANSFER_FINALIZED. So I think the method > isWrite is not working. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16924) Add libhdfs APIs for createFile
Zoltán Borók-Nagy created HDFS-16924: Summary: Add libhdfs APIs for createFile Key: HDFS-16924 URL: https://issues.apache.org/jira/browse/HDFS-16924 Project: Hadoop HDFS Issue Type: Bug Components: fs Reporter: Zoltán Borók-Nagy HDFS-14478 introduces builder-based APIs for openFile() based on HADOOP-15229. We should also add builder-based APIs for createFile() based on HADOOP-14365. This would be especially useful for object stores to tune performance of file writes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16953) RBF: Mount table store APIs should update cache only if state store record is successfully updated
[ https://issues.apache.org/jira/browse/HDFS-16953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16953. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Mount table store APIs should update cache only if state store record is > successfully updated > -- > > Key: HDFS-16953 > URL: https://issues.apache.org/jira/browse/HDFS-16953 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > RBF Mount table state store APIs addMountTableEntry, updateMountTableEntry > and removeMountTableEntry performs cache refresh for all routers regardless > of the actual record update result. If the record fails to get updated on > zookeeper/file based store impl, reloading the cache for all routers would be > unnecessary. > > For instance, simultaneously adding new mount point could lead to failure for > the second call if first call has not added new entry by the time second call > retrieves mount table entry from getMountTableEntries before attempting to > call addMountTableEntry. > {code:java} > DEBUG [{cluster}/{ip}:8111] ipc.Client - IPC Client (1826699684) connection > to nn-0-{ns}.{cluster}/{ip}:8111 from {user}IPC Client (1826699684) > connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user} sending #1 > org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol.addMountTableEntry > DEBUG [{cluster}/{ip}:8111 from {user}] ipc.Client - IPC Client (1826699684) > connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user} got value #1 > DEBUG [main] ipc.ProtobufRpcEngine2 - Call: addMountTableEntry took 24ms > DEBUG [{cluster}/{ip}:8111 from {user}] ipc.Client - IPC Client (1826699684) > connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user}: closed > DEBUG [{cluster}/{ip}:8111 from {user}] ipc.Client - IPC Client (1826699684) > connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user}: stopped, remaining > connections 0 > TRACE [main] ipc.ProtobufRpcEngine2 - 1: Response <- > nn-0-{ns}.{cluster}/{ip}:8111: addMountTableEntry {status: false} > Cannot add mount point /data503 {code} > The failure to write new record: > {code:java} > INFO [IPC Server handler 0 on default port 8111] > impl.StateStoreZooKeeperImpl - Cannot write record > "/hdfs-federation/MountTable/0SLASH0data503", it already exists {code} > Since the successful call has already refreshed cache for all routers, second > call that failed should not have refreshed cache for all routers again as > everyone already has updated records in cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16959) RBF: State store cache loading metrics
[ https://issues.apache.org/jira/browse/HDFS-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16959. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: State store cache loading metrics > -- > > Key: HDFS-16959 > URL: https://issues.apache.org/jira/browse/HDFS-16959 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > With increasing num of state store records (like mount points), it would be > good to be able to get the cache loading metrics like avg time for cache load > during refresh, num of times cache is loaded etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16973) RBF: MountTableResolver cache size lookup should take read lock
[ https://issues.apache.org/jira/browse/HDFS-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16973. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: MountTableResolver cache size lookup should take read lock > --- > > Key: HDFS-16973 > URL: https://issues.apache.org/jira/browse/HDFS-16973 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Mount table resolver location cache gets invalidated by taking write lock as > part of addEntry/removeEntry/refreshEntries calls. Since the write lock > exclusively updates the cache, getDestinationForPath already takes read lock > before accessing the cache. Similarly, retrieval of the cache size should > also take the read lock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16949) Update ReadTransferRate to ReadLatencyPerGB for effective percentile metrics
[ https://issues.apache.org/jira/browse/HDFS-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16949. Hadoop Flags: Reviewed Resolution: Fixed > Update ReadTransferRate to ReadLatencyPerGB for effective percentile metrics > > > Key: HDFS-16949 > URL: https://issues.apache.org/jira/browse/HDFS-16949 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Ravindra Dingankar >Assignee: Ravindra Dingankar >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.0 > > > HDFS-16917 added ReadTransferRate quantiles to calculate the rate which data > is read per unit of time. > With percentiles the values are sorted in ascending order and hence for the > transfer rate p90 gives us the value where 90 percent rates are lower > (worse), p99 gives us the value where 99 percent values are lower (worse). > Note that value(p90) < p(99) thus p99 is a better transfer rate as compared > to p90. > However as the percentile increases the value should become worse in order to > know how good our system is. > Hence instead of calculating the data read transfer rate, we should calculate > it's inverse. We will instead calculate the time taken for a GB of data to be > read. ( seconds / GB ) > After this the p90 value will give us 90 percentage of total values where the > time taken is less than value(p90), similarly for p99 and others. > Also p(90) < p(99) and here p(99) will become a worse value (taking more time > each byte) as compared to p(90) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16971) Add read time metrics for remote reads in Statistics
[ https://issues.apache.org/jira/browse/HDFS-16971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16971. Fix Version/s: (was: 3.3.0) Hadoop Flags: Reviewed Resolution: Fixed > Add read time metrics for remote reads in Statistics > > > Key: HDFS-16971 > URL: https://issues.apache.org/jira/browse/HDFS-16971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Melissa You >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > In Filesystem.java, currently it only collects bytes read not the > corresponding timing of reads. > In particular, we're interested in time spent on remote reads(not local > reads). Adding timing info will help us understand it better and further > analyze how locality of DN behaves. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16982) Use the right Quantiles Array for Inverse Quantiles snapshot
[ https://issues.apache.org/jira/browse/HDFS-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16982. Fix Version/s: (was: 3.3.0) Hadoop Flags: Reviewed Resolution: Fixed > Use the right Quantiles Array for Inverse Quantiles snapshot > - > > Key: HDFS-16982 > URL: https://issues.apache.org/jira/browse/HDFS-16982 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, metrics >Reporter: Ravindra Dingankar >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > HDFS-16949 introduced InverseQuantiles. However during snapshot for Inverse > Quantiles we were still trying to access values from previous snapshot based > on the Quantile Array declared in MutableQuantiles. ( Quantile(.50, .050), > Quantile(.75, .025), Quantile(.90, .010), Quantile(.95, .005), Quantile(.99, > .001) ) > For InverseQuantiles we wont have these values ( except for Quantile(.50, > .050) ) thus except for 50 Percentile snapshot wont return any value for the > remaining quantiles. > Fix is to use the correct Quantiles Array to retrieve values during snapshot. > The new UTs verify this behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16998) RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity
[ https://issues.apache.org/jira/browse/HDFS-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16998. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity > -- > > Key: HDFS-16998 > URL: https://issues.apache.org/jira/browse/HDFS-16998 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17008) Fix RBF JDK 11 javadoc warnings
[ https://issues.apache.org/jira/browse/HDFS-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17008. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix RBF JDK 11 javadoc warnings > --- > > Key: HDFS-17008 > URL: https://issues.apache.org/jira/browse/HDFS-17008 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HDFS-16978 excluded proto packages from maven-javadoc-plugin for rbf, hence > now we have JDK 11 javadoc warnings (e.g. > [here|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5554/14/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt]). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17015) Typos in HDFS Documents
[ https://issues.apache.org/jira/browse/HDFS-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17015. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Typos in HDFS Documents > --- > > Key: HDFS-17015 > URL: https://issues.apache.org/jira/browse/HDFS-17015 > Project: Hadoop HDFS > Issue Type: Improvement > Components: configuration >Affects Versions: 3.3.5 >Reporter: Liang Yan >Priority: Minor > Fix For: 3.4.0 > > > There are some typos in HDFS documents. I will submit PR to fix these typos. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17009) RBF: state store putAll should also return failed records
[ https://issues.apache.org/jira/browse/HDFS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17009. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: state store putAll should also return failed records > - > > Key: HDFS-17009 > URL: https://issues.apache.org/jira/browse/HDFS-17009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.4.0 > > > State store implementations allow adding/updating multiple records using > putAll. The implementation returns whether all records were successfully > added or updated. We should also allow the implementation to return which > records failed to get updated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17020) RBF: mount table addAll should print failed records in std error
[ https://issues.apache.org/jira/browse/HDFS-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17020. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: mount table addAll should print failed records in std error > > > Key: HDFS-17020 > URL: https://issues.apache.org/jira/browse/HDFS-17020 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Now that state store putAll supports returning failed records keys, addAll > command for mount entries should also support printing failed records in the > standard error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17027) RBF: Add supports for observer.auto-msync-period when using routers
[ https://issues.apache.org/jira/browse/HDFS-17027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17027. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Add supports for observer.auto-msync-period when using routers > --- > > Key: HDFS-17027 > URL: https://issues.apache.org/jira/browse/HDFS-17027 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > None-RBF clients that use observer reads have the option to set > *dfs.client.failover.observer.auto-msync-period.* . This config > makes the client automatically do an msync, allowing clients to use the > observer reads feature without any code change. > To use observer reads with RBF, clients set > *dfs.client.rbf.observer.read.enable*. The way this flag is implemented does > not allow clients to use the *auto-msync-period* config. So with RBF, clients > either have to > # Not use observer reads > # Use observer reads with the risk of stale reads > # Make code changes to explicitly call msync. > We should add support for > *dfs.client.failover.observer.auto-msync-period.*. This can be > done by adding a ProxyProvider, in a similar manner to the > ObserverReadProxyProvider. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17041) RBF: Fix putAll impl for mysql and file based state stores
[ https://issues.apache.org/jira/browse/HDFS-17041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17041. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Fix putAll impl for mysql and file based state stores > -- > > Key: HDFS-17041 > URL: https://issues.apache.org/jira/browse/HDFS-17041 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Only zookeeper based state store allows all records to be inserted even > though only few of them already exists and "errorIfExists" is true, however > file/fs as well as mysql based putAll fails the whole putAll operation > immediately after encountering single record that already exists in the > records and "errorIfExists" is provided true (which is the case while > inserting records for the first time). > For all implementations, we should allow inserts of the records that do not > already exist and report any record as failure that already exists, rather > than failing the whole operation and not trying to insert valid records. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17030) Limit wait time for getHAServiceState in ObserverReaderProxy
[ https://issues.apache.org/jira/browse/HDFS-17030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17030. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Limit wait time for getHAServiceState in ObserverReaderProxy > > > Key: HDFS-17030 > URL: https://issues.apache.org/jira/browse/HDFS-17030 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > When namenode HA is enabled and a standby NN is not responsible, we have > observed it would take a long time to serve a request, even though we have a > healthy observer or active NN. > Basically, when a standby is down, the RPC client would (re)try to create > socket connection to that standby for _ipc.client.connect.timeout_ _* > ipc.client.connect.max.retries.on.timeouts_ before giving up. When we take a > heap dump at a standby, the NN still accepts the socket connection but it > won't send responses to these RPC requests and we would timeout after > _ipc.client.rpc-timeout.ms._ This adds a significantly latency. For clusters > at Linkedin, we set _ipc.client.rpc-timeout.ms_ to 120 seconds and thus a > request takes more than 2 mins to complete when we take a heap dump at a > standby. This has been causing user job failures. > We could set _ipc.client.rpc-timeout.ms to_ a smaller value when sending > getHAServiceState requests in ObserverReaderProxy (for user rpc requests, we > still use the original value from the config). However, that would double the > socket connection between clients and the NN (which is a deal-breaker). > The proposal is to add a timeout on getHAServiceState() calls in > ObserverReaderProxy and we will only wait for the timeout for an NN to > respond its HA state. Once we pass that timeout, we will move on to probe the > next NN. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17042) Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode
[ https://issues.apache.org/jira/browse/HDFS-17042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17042. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode > > > Key: HDFS-17042 > URL: https://issues.apache.org/jira/browse/HDFS-17042 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0, 3.3.9 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We'd like to add two new types of metrics to the existing NN > RpcMetrics/RpcDetailedMetrics. These two metrics can then be used as part of > SLA/SLO for the HDFS service. > * {_}RpcCallSuccesses{_}: it measures the number of RPC requests where they > are successfully processed by a NN (e.g., with a response with an RpcStatus > {_}RpcStatusProto.SUCCESS){_}{_}.{_} Then, together with {_}RpcQueueNumOps > ({_}which refers the total number of RPC requests{_}){_}, we can derive the > RpcErrorRate for our NN, as (RpcQueueNumOps - RpcCallSuccesses) / > RpcQueueNumOps. > * OverallRpcProcessingTime for each RPC method: this metric measures the > overall RPC processing time for each RPC method at the NN. It covers the time > from when a request arrives at the NN to when a response is sent back. We are > already emitting processingTime for each RPC method today in > RpcDetailedMetrics. We want to extend it to emit overallRpcProcessingTime for > each RPC method, which includes enqueueTime, queueTime, processingTime, > responseTime, and handlerTime. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17057) RBF: Add DataNode maintenance states to Federation UI
[ https://issues.apache.org/jira/browse/HDFS-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17057. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Add DataNode maintenance states to Federation UI > -- > > Key: HDFS-17057 > URL: https://issues.apache.org/jira/browse/HDFS-17057 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Add DataNode maintenance states to Federation UI -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17055) Export HAState as a metric from Namenode for monitoring
[ https://issues.apache.org/jira/browse/HDFS-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17055. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Export HAState as a metric from Namenode for monitoring > --- > > Key: HDFS-17055 > URL: https://issues.apache.org/jira/browse/HDFS-17055 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0, 3.3.9 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > We'd like measure the uptime for Namenodes: percentage of time when we have > the active/standby/observer node available (up and running). We could monitor > the namenode from an external service, such as ZKFC. But that would require > the external service to be available 100% itself. And when this third-party > external monitoring service is down, we won't have info on whether our > Namenodes are still up. > We propose to take a different approach: we will emit Namenode state directly > from namenode itself. Whenever we miss a data point for this metric, we > consider the corresponding namenode to be down/not available. In other words, > we assume the metric collection/monitoring infrastructure to be 100% reliable. > One implementation detail: in hadoop, we have the _NameNodeMetrics_ class, > which is currently used to emit all metrics for {_}NameNode.java{_}. However, > we don't think that is a good place to emit NameNode HAState. HAState is > stored in NameNode.java and we should directly emit it from NameNode.java. > Otherwise, we basically duplicate this info in two classes and we would have > to keep them in sync. Besides, _NameNodeMetrics_ class does not have a > reference to the _NameNode_ object which it belongs to. An _NameNodeMetrics_ > is created by a _static_ function _initMetrics()_ in {_}NameNode.java{_}. > We shouldn't emit HA state from FSNameSystem.java either, as it is > initialized from NameNode.java and all state transitions are implemented in > NameNode.java. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17065) Fix typos in hadoop-hdfs-project
[ https://issues.apache.org/jira/browse/HDFS-17065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17065. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix typos in hadoop-hdfs-project > > > Key: HDFS-17065 > URL: https://issues.apache.org/jira/browse/HDFS-17065 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17067) Use BlockingThreadPoolExecutorService for nnProbingThreadPool in ObserverReadProxy
[ https://issues.apache.org/jira/browse/HDFS-17067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17067. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Use BlockingThreadPoolExecutorService for nnProbingThreadPool in > ObserverReadProxy > -- > > Key: HDFS-17067 > URL: https://issues.apache.org/jira/browse/HDFS-17067 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > In HDFS-17030, we introduced an ExecutorService, to submit > getHAServiceState() requests. We constructed the ExecutorService directly > from a basic ThreadPoolExecutor, without setting _allowCoreThreadTimeOut_ to > true. Then, the core thread will be kept up and running even when the main > thread exits. To fix it, one could set _allowCoreThreadTimeOut_ to true. > However, in this PR, we decide to directly use an existing executorService > implementation (_BlockingThreadPoolExecutorService_) in hadoop instead. It > takes care of setting _allowCoreThreadTimeOut_ and also allows setting the > prefix for thread names. > {code:java} > private final ExecutorService nnProbingThreadPool = > new ThreadPoolExecutor(1, 4, 1L, TimeUnit.MINUTES, > new ArrayBlockingQueue(1024)); > {code} > A second minor issue is we did not shutdown the executorService in close(). > It is a minor issue as close() will only be called when the garbage collector > starts to reclaim an ObserverReadProxyProvider object, not when there is no > reference to the ObserverReadProxyProvider object. The time between when an > ObserverReadProxyProvider becomes dereferenced and when the garage collector > actually starts to reclaim that object is out of control/under-defined > (unless the program is shutdown with an explicit System.exit(1)). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17118) Fix minor checkstyle warnings in TestObserverReadProxyProvider
[ https://issues.apache.org/jira/browse/HDFS-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17118. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix minor checkstyle warnings in TestObserverReadProxyProvider > -- > > Key: HDFS-17118 > URL: https://issues.apache.org/jira/browse/HDFS-17118 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > > We noticed a few checkstyle warnings when backporting HDFS-17030 from trunk > to branch-3.3. The yetus build was not stable at that time and we did not > notice the newly added checkstyle warnings. > PR for HDFS-17030 which has been merged into trunk: > [https://github.com/apache/hadoop/pull/5700] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17148. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16924) Add libhdfs APIs for createFile
[ https://issues.apache.org/jira/browse/HDFS-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy resolved HDFS-16924. -- Resolution: Fixed Yes, I think we can resolve this. > Add libhdfs APIs for createFile > --- > > Key: HDFS-16924 > URL: https://issues.apache.org/jira/browse/HDFS-16924 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Zoltán Borók-Nagy >Assignee: Uma Maheswara Rao G >Priority: Major > > HDFS-14478 introduces builder-based APIs for openFile() based on HADOOP-15229. > We should also add builder-based APIs for createFile() based on HADOOP-14365. > This would be especially useful for object stores to tune performance of file > writes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17162) RBF: Add missing comments in StateStoreService
[ https://issues.apache.org/jira/browse/HDFS-17162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17162. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Add missing comments in StateStoreService > -- > > Key: HDFS-17162 > URL: https://issues.apache.org/jira/browse/HDFS-17162 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: TIsNotT >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > StateStoreService lacks an introduction to StateStoreFileSystemImpl and > StateStoreMySQLImpl [link > HDFS-16943|https://issues.apache.org/jira/projects/HDFS/issues/HDFS-16943?filter=allissues]. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17178) BootstrapStandby needs to handle RollingUpgrade
[ https://issues.apache.org/jira/browse/HDFS-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17178. Hadoop Flags: Reviewed Resolution: Fixed > BootstrapStandby needs to handle RollingUpgrade > > > Key: HDFS-17178 > URL: https://issues.apache.org/jira/browse/HDFS-17178 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > During rollingUpgrade, bootstrapStandby will fail with an exception due to > different NameNodeLayoutVersions. We can ignore this safely during > RollingUpgrade because different NameNodeLayoutVersions are expected. > * NameNodes will not be able to recover with BootstrapStandby if they go > through destructive repair before the rollingUpgrade has been finalized. > Error during BootstrapStandby before change: > {code:java} > = > About to bootstrap Standby ID nn2 from: >Nameservice ID: MTPrime-MWHE01-0 > Other Namenode ID: nn1 > Other NN's HTTP address: https://MWHEEEAP002D9A2:81 > Other NN's IPC address: MWHEEEAP002D9A2.ap.gbl/10.59.208.18:8020 > Namespace ID: 895912530 > Block pool ID: BP-1556042256-10.99.154.61-1663325602669 >Cluster ID: MWHE01 >Layout version: -64 >isUpgradeFinalized: true > = > 2023-08-28T19:35:06,940 ERROR [main] namenode.NameNode: Failed to start > namenode. > java.io.IOException: java.lang.RuntimeException: > org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: > Image transfer servlet at > https://MWHEEEAP002D9A2:81/imagetransfer?getimage=1&txid=25683470&storageInfo=-64:895912530:1663325602669:MWHE01&bootstrapstandby=true > failed with status code 403 > Response message: > This namenode has storage info -63:895912530:1663325602669:MWHE01 but the > secondary expected -64:895912530:1663325602669:MWHE01 > at > org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:583) > ~[hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?] > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1717) > ~[hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?] > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1819) > [hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?] > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: > Image transfer servlet at https://MWHEEEAP002D9A2:81{code} > This is caused because the namespaceInfo sent from the proxy node does not > include the effective layout version, which causes BootstrapStandby to send a > request with a storageinfo param using the service layout version. This > causes the proxy node to refuse the request, because it compares the > storageinfo param against its storage info, which uses the effective layout > version, not the service layout version. > To fix this we can modify the proxy.versionRequest() call stack to set the > layout version using the effective layout version on the proxy node. We can > then add logic to BootstrapStandby to properly handle the case where the > proxy node is in rolling upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working
[ https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17138. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: We changed the hadoop.security.auth_to_local configuration of one > router, the other routers stopped working > > > Key: HDFS-17138 > URL: https://issues.apache.org/jira/browse/HDFS-17138 > Project: Hadoop HDFS > Issue Type: Bug > Environment: hadoop 3.3.0 >Reporter: Xiping Zhang >Assignee: Xiping Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-08-02-16-20-34-454.png, > image-2023-08-03-10-32-03-457.png > > > other routers error log: > !image-2023-08-02-16-20-34-454.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14943) RBF: Add Namespace to the Overview page in the Web UI
Íñigo Goiri created HDFS-14943: -- Summary: RBF: Add Namespace to the Overview page in the Web UI Key: HDFS-14943 URL: https://issues.apache.org/jira/browse/HDFS-14943 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri The Namenode shows the Namespace field which can be used to access it through HDFS. The Router should also report its namespace. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14961) TestDFSZKFailoverController fails consistently
Íñigo Goiri created HDFS-14961: -- Summary: TestDFSZKFailoverController fails consistently Key: HDFS-14961 URL: https://issues.apache.org/jira/browse/HDFS-14961 Project: Hadoop HDFS Issue Type: Bug Reporter: Íñigo Goiri TestDFSZKFailoverController has been consistently failing with a time out waiting in testManualFailoverWithDFSHAAdmin(). In particular {{waitForHAState(1, HAServiceState.OBSERVER);}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14974) RBF: TestRouterSecurityManager#testCreateCredentials should use :0 for port
Íñigo Goiri created HDFS-14974: -- Summary: RBF: TestRouterSecurityManager#testCreateCredentials should use :0 for port Key: HDFS-14974 URL: https://issues.apache.org/jira/browse/HDFS-14974 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri Currently, {{TestRouterSecurityManager#testCreateCredentials}} create a Router with the default ports. However, these ports might be used. We should set it to :0 for it to be assigned dynamically. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15016) RBF: getDatanodeReport() should return the latest update
Íñigo Goiri created HDFS-15016: -- Summary: RBF: getDatanodeReport() should return the latest update Key: HDFS-15016 URL: https://issues.apache.org/jira/browse/HDFS-15016 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri Currently, when the Router calls getDatanodeReport() (or getDatanodeStorageReport()) and the DN is in multiple clusters, it just takes the one that comes first. It should consider the latest update. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15075) Remove process command timing from BPServiceActor
Íñigo Goiri created HDFS-15075: -- Summary: Remove process command timing from BPServiceActor Key: HDFS-15075 URL: https://issues.apache.org/jira/browse/HDFS-15075 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri HDFS-14997 moved the command processing into async. Right now, we are checking the time to add to a queue. We should remove this one and maybe move the timing within the thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15062) Add LOG when sendIBRs failed
[ https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15062. Resolution: Fixed > Add LOG when sendIBRs failed > > > Key: HDFS-15062 > URL: https://issues.apache.org/jira/browse/HDFS-15062 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.0.3, 3.2.1, 3.1.3 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch, > HDFS-15062.003.patch > > > {code} > /** Send IBRs to namenode. */ > void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration, > String bpid, String nnRpcLatencySuffix) throws IOException { > // Generate a list of the pending reports for each storage under the lock > final StorageReceivedDeletedBlocks[] reports = generateIBRs(); > if (reports.length == 0) { > // Nothing new to report. > return; > } > // Send incremental block reports to the Namenode outside the lock > if (LOG.isDebugEnabled()) { > LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports)); > } > boolean success = false; > final long startTime = monotonicNow(); > try { > namenode.blockReceivedAndDeleted(registration, bpid, reports); > success = true; > } finally { > if (success) { > dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime, > nnRpcLatencySuffix); > lastIBR = startTime; > } else { > // If we didn't succeed in sending the report, put all of the > // blocks back onto our queue, but only in the case where we > // didn't put something newer in the meantime. > putMissing(reports); > } > } > } > {code} > When call namenode.blockReceivedAndDelete failed, will put reports to > pendingIBRs. Maybe we should add log for failed case. It is helpful for > trouble shooting -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15112) RBF: do not return FileNotFoundException when a subcluster is unavailable
Íñigo Goiri created HDFS-15112: -- Summary: RBF: do not return FileNotFoundException when a subcluster is unavailable Key: HDFS-15112 URL: https://issues.apache.org/jira/browse/HDFS-15112 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri If we have a mount point using HASH_ALL across two subclusters and one of them is down, we may return FileNotFoundException while the file is just in the unavailable subcluster. We should not return FileNotFoundException but something that shows that the subcluster is unavailable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15127) RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points.
Íñigo Goiri created HDFS-15127: -- Summary: RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points. Key: HDFS-15127 URL: https://issues.apache.org/jira/browse/HDFS-15127 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri A HASH_ALL mount point should not allow creating new files if one subcluster is down. If the file already existed in the past, this could lead to inconsistencies. We should return an unavailable exception. {{TestRouterFaultTolerant#testWriteWithFailedSubcluster()}} needs to be changed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15163) hdfs-2.10.0-webapps-secondary-status.html miss moment.js
谢波 created HDFS-15163: - Summary: hdfs-2.10.0-webapps-secondary-status.html miss moment.js Key: HDFS-15163 URL: https://issues.apache.org/jira/browse/HDFS-15163 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 2.10.0 Reporter: 谢波 Fix For: 2.10.1 Attachments: 微信截图_20200212183444.png hdfs-2.10.0-webapps-secondary-status.html miss moment.js -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15184) Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned:
任建亭 created HDFS-15184: -- Summary: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1 Key: HDFS-15184 URL: https://issues.apache.org/jira/browse/HDFS-15184 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.2.1 Environment: windows 10 JDK 1.8 maven3.6.1 ProtocolBuffer 2.5.0 CMake 3.1.3 git 2.25.0 zlib 1.2.5 Visual Studio 2010 Professional Reporter: 任建亭 Fix For: 3.2.1 When I build hadoop 3.2.1 on windows10, it failed. My command is 'mvn clean package -Pdist,native-win -DskipTests -Dtar'. {code:java} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1 [ERROR] around Ant part .. @ 9:122 in D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11439) testGenerationStampInFuture UT fails
[ https://issues.apache.org/jira/browse/HDFS-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-11439. Resolution: Duplicate > testGenerationStampInFuture UT fails > > > Key: HDFS-11439 > URL: https://issues.apache.org/jira/browse/HDFS-11439 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yesha Vora >Priority: Major > Attachments: testGenerationStampInFuture.log > > > testGenerationStampInFuture UT fails as below. > {code} > Error Message > expected:<18> but was:<0> > Stacktrace > java.lang.AssertionError: expected:<18> but was:<0> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency.testGenerationStampInFuture(TestNameNodeMetadataConsistency.java:125){code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15215) The Timestamp for longest write/read lock held log is wrong
[ https://issues.apache.org/jira/browse/HDFS-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15215. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed > The Timestamp for longest write/read lock held log is wrong > --- > > Key: HDFS-15215 > URL: https://issues.apache.org/jira/browse/HDFS-15215 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.3.0 > > > I found the Timestamp for longest write/read lock held log is wrong in trunk: > {code} > 2020-03-10 16:01:26,585 [main] INFO namenode.FSNamesystem > (FSNamesystemLock.java:writeUnlock(281)) - Number of suppressed > write-lock reports: 0 > Longest write-lock held at 1970-01-03 07:07:40,841+0900 for 3ms via > java.lang.Thread.getStackTrace(Thread.java:1559) > ... > {code} > Looking at the code, it looks like the timestamp comes from System.nanoTime() > that returns the current value of the running Java Virtual Machine's > high-resolution time source and this method can only be used to measure > elapsed time: > https://docs.oracle.com/javase/8/docs/api/java/lang/System.html#nanoTime-- > We need to make the timestamp from System.currentTimeMillis(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15239) Add button to go to the parent directory in the explorer
Íñigo Goiri created HDFS-15239: -- Summary: Add button to go to the parent directory in the explorer Key: HDFS-15239 URL: https://issues.apache.org/jira/browse/HDFS-15239 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri Currently, when using the HDFS explorer page, it is easy to go into a folder. However, to go back one has to use the browser back button (if one is coming from that folder) or to edit the path by hand. It would be nice to have the typical button to go to the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15217) Add more information to longest write/read lock held log
[ https://issues.apache.org/jira/browse/HDFS-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15217. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add more information to longest write/read lock held log > > > Key: HDFS-15217 > URL: https://issues.apache.org/jira/browse/HDFS-15217 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.4.0 > > > Currently, we can see the stack trace in the longest write/read lock held > log, but sometimes we need more information, for example, a target path of > deletion: > {code:java} > 2020-03-10 21:51:21,116 [main] INFO namenode.FSNamesystem > (FSNamesystemLock.java:writeUnlock(276)) - Number of suppressed > write-lock reports: 0 > Longest write-lock held at 2020-03-10 21:51:21,107+0900 for 6ms via > java.lang.Thread.getStackTrace(Thread.java:1559) > org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1058) > org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:257) > org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:233) > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1706) > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3188) > ... > {code} > Adding more information (opName, path, etc.) to the log is useful to > troubleshoot. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15345) RBF: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
[ https://issues.apache.org/jira/browse/HDFS-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15345. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups > after HADOOP-13442 > > > Key: HDFS-15345 > URL: https://issues.apache.org/jira/browse/HDFS-15345 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.5 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.4.0 > > > HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This > ticket is opened to change RouterPermissionChecker#checkSuperuserPrivilege > to use UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15184) Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned
[ https://issues.apache.org/jira/browse/HDFS-15184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 任建亭 resolved HDFS-15184. Resolution: Not A Problem > Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run > (make) on project hadoop-hdfs-native-client: An Ant BuildException has > occured: exec returned: 1 > > > Key: HDFS-15184 > URL: https://issues.apache.org/jira/browse/HDFS-15184 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.2.1 > Environment: windows 10 > JDK 1.8 > maven3.6.1 > ProtocolBuffer 2.5.0 > CMake 3.1.3 > git 2.25.0 > zlib 1.2.5 > Visual Studio 2010 Professional >Reporter: 任建亭 >Priority: Major > Fix For: 3.2.1 > > > When I build hadoop 3.2.1 on windows10, it failed. My command is 'mvn clean > package -Pdist,native-win -DskipTests -Dtar'. > {code:java} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project > hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1 > [ERROR] around Ant part ... dir="D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" > executable="msbuild">... @ 9:122 in > D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15383) RBF: Disable watch in ZKDelegationSecretManager for performance
[ https://issues.apache.org/jira/browse/HDFS-15383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15383. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Disable watch in ZKDelegationSecretManager for performance > --- > > Key: HDFS-15383 > URL: https://issues.apache.org/jira/browse/HDFS-15383 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.4.0 > > > Based on the current design for delegation token in secure Router, the total > number of watches for tokens is the product of number of routers and number > of tokens, this is due to ZKDelegationTokenManager is using PathChildrenCache > from curator, which automatically sets the watch and ZK will push the sync > information to each router. There are some evaluations about the number of > watches in Zookeeper has negative performance impact to Zookeeper server. > In our practice when the number of watches exceeds 1.2 Million in a single ZK > server there will be significant ZK performance degradation. Thus this ticket > is to rewrite ZKDelegationTokenManagerImpl.java to explicitly disable the > PathChildrenCache and have Routers sync periodically from Zookeeper. This has > been working fine at the scale of 10 Routers with 2 million tokens. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15312) Apply umask when creating directory by WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15312. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Apply umask when creating directory by WebHDFS > -- > > Key: HDFS-15312 > URL: https://issues.apache.org/jira/browse/HDFS-15312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: Ye Ni >Assignee: Ye Ni >Priority: Minor > Fix For: 3.4.0 > > > WebHDFS methods for creating file/directories were always creating it with > 755 permissions as default for both files and directories. > The configured *fs.permissions.umask-mode* is intentionally ignored. > This Jira is to apply this setting in such scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15600) TestRouterQuota fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15600. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > TestRouterQuota fails in trunk > -- > > Key: HDFS-15600 > URL: https://issues.apache.org/jira/browse/HDFS-15600 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The test is failing due to addition of a new storage type {{NVDIMM}} in > middle. > Ref : > https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/204/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterQuota/testStorageTypeQuota/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12449) TestReconstructStripedFile.testNNSendsErasureCodingTasks randomly cannot finish in 60s
[ https://issues.apache.org/jira/browse/HDFS-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-12449. Resolution: Fixed > TestReconstructStripedFile.testNNSendsErasureCodingTasks randomly cannot > finish in 60s > -- > > Key: HDFS-12449 > URL: https://issues.apache.org/jira/browse/HDFS-12449 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha4 >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: flaky-test > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12449.001.patch > > > TestReconstructStripedFile.testNNSendsErasureCodingTasks randomly cannot > finish in 60s > reduce the file size and loop count -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15656) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails occasionally
Íñigo Goiri created HDFS-15656: -- Summary: TestReconstructStripedFile.testNNSendsErasureCodingTasks fails occasionally Key: HDFS-15656 URL: https://issues.apache.org/jira/browse/HDFS-15656 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri HDFS-12449 tried to fix this issue but this is still failing intermittently. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15654) TestBPOfferService#testMissBlocksWhenReregister fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-15654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15654. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > TestBPOfferService#testMissBlocksWhenReregister fails intermittently > > > Key: HDFS-15654 > URL: https://issues.apache.org/jira/browse/HDFS-15654 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > {{TestBPOfferService.testMissBlocksWhenReregister}} is flaky. It fails > randomly when the > following expression is not true: > {code:java} > assertTrue(fullBlockReportCount == totalTestBlocks || > incrBlockReportCount == totalTestBlocks); > {code} > There is a race condition here that relies once more on "time" to synchronize > between concurrent threads. The code below is is causing the > non-deterministic execution. > On a slow server, {{addNewBlockThread}} may not be done by the time the main > thread reach the assertion call. > {code:java} > // Verify FBR/IBR count is equal to generate number. > assertTrue(fullBlockReportCount == totalTestBlocks || > incrBlockReportCount == totalTestBlocks); > } finally { > addNewBlockThread.join(); > bpos.stop(); > bpos.join(); > {code} > Therefore, the correct implementation should wait for the thread to finish > {code:java} > // the thread finished execution. > addNewBlockThread.join(); > // Verify FBR/IBR count is equal to generate number. > assertTrue(fullBlockReportCount == totalTestBlocks || > incrBlockReportCount == totalTestBlocks); > } finally { > bpos.stop(); > bpos.join(); > {code} > {{DataNodeFaultInjector}} needs to have a longer wait_time too. 1 second is > not enough to satisfy the condition. > {code:java} > DataNodeFaultInjector.set(new DataNodeFaultInjector() { > public void blockUtilSendFullBlockReport() { > try { > GenericTestUtils.waitFor(() -> { > if(count.get() > 2000) { > return true; > } > return false; > }, 100, 1); // increase that waiting time to 10 seconds. > } catch (Exception e) { > e.printStackTrace(); > } > } > }); > {code} > {code:bash} > Stacktrace > java.lang.AssertionError > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister(TestBPOfferService.java:350) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Prov
[jira] [Created] (HDFS-15692) Improve furse_dfs read performace
István Fajth created HDFS-15692: --- Summary: Improve furse_dfs read performace Key: HDFS-15692 URL: https://issues.apache.org/jira/browse/HDFS-15692 Project: Hadoop HDFS Issue Type: Improvement Components: fuse-dfs Reporter: István Fajth Currently fuse_dfs uses a prefetch buffer to read from HDFS via libhdfs' pread method. The algorithm inside fuse_read.c in short does the following: if the rdbuffer size is less then the buffer provided then reads directly to the buffer else grab lock if the preftch buffer does not have more data then fills the prefetch buffer endif fills the supplied buffer via memcpy from the prefetch buffer release lock endif It would be nice to have a background thread and double prefetch buffers, so while one buffer serves the reads coming from the local client, the other can prefetch the data, with that we can improve the read speed, especially with EC encoded files. According to some measurements I did, if I increase the read buffer, there is a significant change in runtime, with 64MB the runtime is really closer to HDFS by a large margin. Interestingly 128MB as the buffer size does not perform well, but 256MB is even more closer to what the dfs client can provide. (16 vs 18 seconds with rep3 files, while in par with ec encoded files dfs vs fuse) So it seems it is worth to stream continuously a larger chunk of data, at least with pread, but in case we have a separate fetching thread and double buffering, we don't even need positioned reads, simply just continuous streaming of data with read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15693) Improve native code's performance when writing to HDFS
István Fajth created HDFS-15693: --- Summary: Improve native code's performance when writing to HDFS Key: HDFS-15693 URL: https://issues.apache.org/jira/browse/HDFS-15693 Project: Hadoop HDFS Issue Type: Improvement Components: fuse-dfs, native Reporter: István Fajth For reads, we introduced direct buffers in order to more efficiently communicate between the JVM and the native code, and we have readDirect and pReadDirect in hdfs.c implemented. Writes on the other hand still use the putByteArrayRegion call, which results in a copy of the buffer in memory. This Jira is to explore what has to be done in order to start to use direct buffers. A short initial list I see at the moment: - add a new StreamCapability for streams wanting to support writes via direct buffer - implement this capability in the DFSOutputStream and DFSStripedOutputStream - implement a writeDirect method on the native side fuse_dfs can benefit from this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15163) hdfs-2.10.0-webapps-secondary-status.html miss moment.js
[ https://issues.apache.org/jira/browse/HDFS-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 resolved HDFS-15163. --- Resolution: Fixed > hdfs-2.10.0-webapps-secondary-status.html miss moment.js > > > Key: HDFS-15163 > URL: https://issues.apache.org/jira/browse/HDFS-15163 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.10.0 >Reporter: 谢波 >Priority: Minor > Attachments: 微信截图_20200212183444.png > > Original Estimate: 96h > Remaining Estimate: 96h > > hdfs-2.10.0-webapps-secondary-status.html miss moment.js > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15843) [libhdfs++] Make write cross platform
[ https://issues.apache.org/jira/browse/HDFS-15843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15843. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [libhdfs++] Make write cross platform > - > > Key: HDFS-15843 > URL: https://issues.apache.org/jira/browse/HDFS-15843 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.2.2 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We're currently using the *write* function from unistd.h which isn't > cross-platform. We need to replace this with *std::cout.write* instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15903) Refactor X-Platform library
[ https://issues.apache.org/jira/browse/HDFS-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15903. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Refactor X-Platform library > --- > > Key: HDFS-15903 > URL: https://issues.apache.org/jira/browse/HDFS-15903 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.2.2 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > X-Platform started out as a utility to help in writing cross platform code in > Hadoop. As its scope expanding to cover various scenarios, it is necessary to > refactor it in early stages to provide proper organization and growth of the > X-Platform library. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15913) Remove useless NULL checks before instanceof
[ https://issues.apache.org/jira/browse/HDFS-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15913. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Remove useless NULL checks before instanceof > > > Key: HDFS-15913 > URL: https://issues.apache.org/jira/browse/HDFS-15913 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Jiajun Jiang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HDFS-15913.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Remove useless NULL checks before instanceof in several classes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15910) Replace bzero with explicit_bzero for better safety
[ https://issues.apache.org/jira/browse/HDFS-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15910. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Replace bzero with explicit_bzero for better safety > --- > > Key: HDFS-15910 > URL: https://issues.apache.org/jira/browse/HDFS-15910 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.2.2 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > It is better to always use explicit_bzero since it guarantees that the buffer > will be cleared irrespective of the compiler optimizations - > https://man7.org/linux/man-pages/man3/bzero.3.html. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15918) Replace RAND_pseudo_bytes in sasl_digest_md5.cc
[ https://issues.apache.org/jira/browse/HDFS-15918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15918. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Replace RAND_pseudo_bytes in sasl_digest_md5.cc > --- > > Key: HDFS-15918 > URL: https://issues.apache.org/jira/browse/HDFS-15918 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following > warning during compilation that it's deprecated - > {code} > [WARNING] > /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:97:74: > warning: 'int RAND_pseudo_bytes(unsigned char*, int)' is deprecated > [-Wdeprecated-declarations] > [WARNING] from > /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:20: > [WARNING] /usr/include/openssl/rand.h:44:1: note: declared here > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15922) Use memcpy for copying non-null terminated string in jni_helper.c
[ https://issues.apache.org/jira/browse/HDFS-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15922. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Use memcpy for copying non-null terminated string in jni_helper.c > - > > Key: HDFS-15922 > URL: https://issues.apache.org/jira/browse/HDFS-15922 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > We currently get a warning while compiling HDFS native client - > {code} > [WARNING] inlined from 'wildcard_expandPath' at > /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:427:21, > [WARNING] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: > warning: '__builtin_strncpy' output truncated before terminating nul copying > as many bytes from a string as its length [-Wstringop-truncation] > [WARNING] > /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:402:43: > note: length computed here > {code} > The scenario here is such that the copied string is deliberately not null > terminated, since we want to insert a PATH_SEPARATOR ourselves. The warning > reported by strncpy is valid, but not applicable in this scenario. Thus, we > need to use memcpy which doesn't mind if the string is null terminated or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15929) Replace RAND_pseudo_bytes in util.cc
[ https://issues.apache.org/jira/browse/HDFS-15929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15929. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Replace RAND_pseudo_bytes in util.cc > > > Key: HDFS-15929 > URL: https://issues.apache.org/jira/browse/HDFS-15929 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following > warning during compilation that it's deprecated - > {code} > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc: > In function ‘std::string hdfs::GetRandomClientName()’: > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31: > warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated > [-Wdeprecated-declarations] >78 | RAND_pseudo_bytes(&buf[0], 8); > | ^ > In file included from /usr/include/openssl/e_os2.h:13, > from /usr/include/openssl/ossl_typ.h:19, > from /usr/include/openssl/rand.h:14, > from > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h:29, > from > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:19: > /usr/include/openssl/rand.h:44:1: note: declared here >44 | DEPRECATEDIN_1_1_0(int RAND_pseudo_bytes(unsigned char *buf, int num)) > | ^~ > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31: > warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated > [-Wdeprecated-declarations] > 78 | RAND_pseudo_bytes(&buf[0], 8); > | ^ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15927) Catch polymorphic type by reference
[ https://issues.apache.org/jira/browse/HDFS-15927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15927. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Catch polymorphic type by reference > --- > > Key: HDFS-15927 > URL: https://issues.apache.org/jira/browse/HDFS-15927 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Need to catch polymorphic exception types by reference in order to realize > the polymorphic usage, if any. Otherwise, the functionality of the caught > object is restricted to only that of the base class. > Also, the following warning gets reported since it's currently caught by > value - > {code:java} > [WARNING] > /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:138:22: > warning: catching polymorphic type 'const class hdfs::uri_parse_error' by > value [-Wcatch-value=] > [WARNING] > /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:151:27: > warning: catching polymorphic type 'struct hdfs::ha_parse_error' by value > [-Wcatch-value=] > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15944) Prevent truncation by snprintf
[ https://issues.apache.org/jira/browse/HDFS-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15944. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Prevent truncation by snprintf > -- > > Key: HDFS-15944 > URL: https://issues.apache.org/jira/browse/HDFS-15944 > Project: Hadoop HDFS > Issue Type: Bug > Components: fuse-dfs, libhdfs >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > There are some areas of code in libhdfs and fuse-dfs components where the > destination buffer is smaller than the source that's trying to write. This > would cause truncation. Thus we need to ensure that the source that's being > written doesn't exceed the destination buffer size. > The following warnings are reported for this issue - > {code} > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c: > In function ‘doTestHdfsOperations.isra.0’: > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:226:39: > warning: ‘/many_files_’ directive output may be truncated writing 12 bytes > into a region of size between 1 and 4096 [-Wformat-truncation=] > 226 | snprintf(filename, PATH_MAX, "%s/many_files_%d", listDirTest, > nFile); > | ^~~~ > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:226:36: > note: directive argument in the range [0, ] > 226 | snprintf(filename, PATH_MAX, "%s/many_files_%d", listDirTest, > nFile); > |^~ > In file included from /usr/include/stdio.h:867, > from > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/expect.h:23, > from > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:19: > /usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: > ‘__builtin___snprintf_chk’ output between 14 and 4112 bytes into a > destination of size 4096 >67 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, > | ^~~~ >68 |__bos (__s), __fmt, __va_arg_pack ()); > |~ > {code} > {code} > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/test/fuse_workload.c:255:33: > warning: ‘/a’ directive output may be truncated writing 2 bytes into a > region of size between 1 and 4096 [-Wformat-truncation=] > 255 | snprintf(tmp, sizeof(tmp), "%s/a", base); > | ^~ > In file included from /usr/include/stdio.h:867, > from > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/expect.h:23, > from > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/test/fuse_workload.c:22: > /usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: > ‘__builtin___snprintf_chk’ output between 3 and 4098 bytes into a destination > of size 4096 >67 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, > | ^~~~ >68 |__bos (__s), __fmt, __va_arg_pack ()); > |~ > /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/test/fuse_workload.c:263:33: > warning: ‘/b’ directive output may be truncated writing 2 bytes into a > region of size between 1 and 4096 [-Wformat-truncation=] > 263 | snprintf(tmp, sizeof(tmp), "%s/b", base); > | ^~ > In file included from /usr/include/stdio.h:867, > from > /mnt/c/Users/Gautham/pr
[jira] [Resolved] (HDFS-15947) Replace deprecated protobuf APIs
[ https://issues.apache.org/jira/browse/HDFS-15947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15947. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Replace deprecated protobuf APIs > > > Key: HDFS-15947 > URL: https://issues.apache.org/jira/browse/HDFS-15947 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Some protobuf APIs are soon going to get deprecated and must be replaced with > newer ones. One of the warnings are reported due to the issue is as follows - > {code} > [ 48%] Building CXX object > main/native/libhdfspp/tests/CMakeFiles/rpc_engine_test.dir/rpc_engine_test.cc.o > /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc: > In function ‘std::pair std::__cxx11::basic_string > RpcResponse(const > hadoop::common::RpcResponseHeaderProto&, const string&, const > boost::system::error_code&)’: > /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc:92:56: > warning: ‘int google::protobuf::MessageLite::ByteSize() const’ is > deprecated: Please use ByteSizeLong() instead [-Wdeprecated-declarations] >92 | pbio::CodedOutputStream::VarintSize32(h.ByteSize()) + > |^ > In file included from > /usr/local/include/google/protobuf/generated_enum_util.h:36, > from /usr/local/include/google/protobuf/map.h:49, > from > /usr/local/include/google/protobuf/generated_message_table_driven.h:34, > from > /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/tests/test.pb.h:26, > from > /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc:22: > /usr/local/include/google/protobuf/message_lite.h:408:7: note: declared here > 408 | int ByteSize() const { return internal::ToIntSize(ByteSizeLong()); } > | ^~~~ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15909) Make fnmatch cross platform
[ https://issues.apache.org/jira/browse/HDFS-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15909. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make fnmatch cross platform > --- > > Key: HDFS-15909 > URL: https://issues.apache.org/jira/browse/HDFS-15909 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.2.2 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > The function fnmatch isn't available in Visual C++. Need to make this cross > platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15955) Make explicit_bzero cross platform
[ https://issues.apache.org/jira/browse/HDFS-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15955. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Make explicit_bzero cross platform > -- > > Key: HDFS-15955 > URL: https://issues.apache.org/jira/browse/HDFS-15955 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs++ >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The function explicit_bzero isn't available in Visual C++. Need to make this > cross platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15423) RBF: WebHDFS create shouldn't choose DN from all sub-clusters
[ https://issues.apache.org/jira/browse/HDFS-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-15423. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: WebHDFS create shouldn't choose DN from all sub-clusters > - > > Key: HDFS-15423 > URL: https://issues.apache.org/jira/browse/HDFS-15423 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf, webhdfs >Reporter: Chao Sun >Assignee: Fengnan Li >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h > Remaining Estimate: 0h > > In {{RouterWebHdfsMethods}} and for a {{CREATE}} call, {{chooseDatanode}} > first gets all DNs via {{getDatanodeReport}}, and then randomly pick one from > the list via {{getRandomDatanode}}. This logic doesn't seem correct as it > should pick a DN for the specific cluster(s) of the input {{path}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org