[jira] [Resolved] (HDFS-15976) Make mkdtemp cross platform

2021-08-10 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15976.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make mkdtemp cross platform
> ---
>
> Key: HDFS-15976
> URL: https://issues.apache.org/jira/browse/HDFS-15976
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> mkdtemp is used for creating temporary directory, adhering to the given 
> pattern. It's not available on Visual C++. Need to make this cross platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16174) Refactor TempFile and TempDir in libhdfs++

2021-08-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16174.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Refactor TempFile and TempDir in libhdfs++
> --
>
> Key: HDFS-16174
> URL: https://issues.apache.org/jira/browse/HDFS-16174
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In C++, we generally do the declaration in the header files and the 
> corresponding implementation in the .cc files. Here we see that the 
> implementation of TempFile and TempDir are done in configuration_test.h 
> itself. This offers no benefit and the compilation of TempFile and TempDir 
> classes are duplicated for every #include of the configuration_test.h header. 
> Thus, we need to implement it in separate cc files to avoid this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16178) Make recursive rmdir in libhdfs++ cross platform

2021-08-20 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16178.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make recursive rmdir in libhdfs++ cross platform
> 
>
> Key: HDFS-16178
> URL: https://issues.apache.org/jira/browse/HDFS-16178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The *TempDir* class in libhdfs++ is currently using nftw API provided by 
> *ftw.h*, which is only present in Linux and not on Windows. Need to use the 
> APIs from C++17 *std::filesystem* to make this cross platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16205) Make hdfs_allowSnapshot tool cross platform

2021-09-22 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16205.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make hdfs_allowSnapshot tool cross platform
> ---
>
> Key: HDFS-16205
> URL: https://issues.apache.org/jira/browse/HDFS-16205
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The source files for hdfs_allowSnapshot uses *getopt* for parsing the command 
> line arguments. getopt is available only on Linux and thus, isn't cross 
> platform. We need to replace getopt with *boost::program_options* to make 
> this cross platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16250) Refactor AllowSnapshotMock using GMock

2021-10-05 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16250.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Refactor AllowSnapshotMock using GMock
> --
>
> Key: HDFS-16250
> URL: https://issues.apache.org/jira/browse/HDFS-16250
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, the 
> [implementation|https://github.com/apache/hadoop/blob/35a8d48872a13438d4c4199b6ef5b902105e2eb2/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/tools/hdfs-allow-snapshot-mock.cc]
>  of AllowSnapshotMock is quite basic. Need to replace this with GMock so that 
> we can tap into the benefits offered by GMock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16254) Cleanup protobuf on exit of hdfs_allowSnapshot

2021-10-06 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16254.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Cleanup protobuf on exit of hdfs_allowSnapshot
> --
>
> Key: HDFS-16254
> URL: https://issues.apache.org/jira/browse/HDFS-16254
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Need to move the call google::protobuf::ShutdownProtobufLibrary() to main 
> method instead of 
> [AllowSnapshot::HandlePath|https://github.com/apache/hadoop/blob/35a8d48872a13438d4c4199b6ef5b902105e2eb2/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-allow-snapshot/hdfs-allow-snapshot.cc#L116-L117]
>  since we want the clean-up tasks to run only when the program exits.
> The current implementation doesn't cause any issues since 
> AllowSnapshot::HandlePath is called only once.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16251) Make hdfs_cat tool cross platform

2021-10-07 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16251.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make hdfs_cat tool cross platform
> -
>
> Key: HDFS-16251
> URL: https://issues.apache.org/jira/browse/HDFS-16251
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The source files for hdfs_cat uses *getopt* for parsing the command line 
> arguments. getopt is available only on Linux and thus, isn't cross platform. 
> We need to replace getopt with *boost::program_options* to make this cross 
> platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16263) Add CMakeLists for hdfs_allowSnapshot

2021-10-08 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16263.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Add CMakeLists for hdfs_allowSnapshot
> -
>
> Key: HDFS-16263
> URL: https://issues.apache.org/jira/browse/HDFS-16263
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently, hdfs_allowSnapshot is built in it's [parent directory's 
> CMakeLists.txt|https://github.com/apache/hadoop/blob/95b537ee6a9ff3082c9ad9bc773f86fd4be04e50/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/CMakeLists.txt#L83-L89].
>  Need to move this into a separate CMakeLists.txt file under 
> hdfs-allow-snapshot so that it's more modular.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16265) Refactor HDFS tool tests for better reuse

2021-10-09 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16265.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Refactor HDFS tool tests for better reuse
> -
>
> Key: HDFS-16265
> URL: https://issues.apache.org/jira/browse/HDFS-16265
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, the test cases written in hdfs-tool-test.h isn't easy to reuse. 
> Primarily because the expectations are different for each HDFS tool. I 
> realized this while I was creating the PR for HDFS-16260. For instance, 
> passing more than one argument is erroneous to hdfs_allowSnapshot while it's 
> the only valid scenario for hdfs_deleteSnapshot.
> Thus, it won't be possible to reuse the test cases without decoupling the 
> expectations from the test case definitions. The solution here is to move the 
> expectations to the corresponding mock classes and invoke the call to set 
> them up in the test cases after the creation of mock instances.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16260) Make hdfs_deleteSnapshot tool cross platform

2021-10-11 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16260.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make hdfs_deleteSnapshot tool cross platform
> 
>
> Key: HDFS-16260
> URL: https://issues.apache.org/jira/browse/HDFS-16260
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The source files for hdfs_deleteSnapshot uses *getopt* for parsing the 
> command line arguments. getopt is available only on Linux and thus, isn't 
> cross platform. We need to replace getopt with *boost::program_options* to 
> make this cross platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16267) Make hdfs_df tool cross platform

2021-10-13 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16267.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make hdfs_df tool cross platform
> 
>
> Key: HDFS-16267
> URL: https://issues.apache.org/jira/browse/HDFS-16267
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The source files for hdfs_df uses *getopt* for parsing the command line 
> arguments. getopt is available only on Linux and thus, isn't cross platform. 
> We need to replace getopt with *boost::program_options* to make this cross 
> platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16278) Make HDFS snapshot tools cross platform

2021-10-21 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16278.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make HDFS snapshot tools cross platform
> ---
>
> Key: HDFS-16278
> URL: https://issues.apache.org/jira/browse/HDFS-16278
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The source files for *hdfs_createSnapshot*, *hdfs_disallowSnapshot* and 
> *hdfs_renameSnapshot* uses getopt for parsing the command line arguments. 
> getopt is available only on Linux and thus, isn't cross platform. We need to 
> replace getopt with boost::program_options to make these tools cross platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16300) Use libcrypto in Windows for libhdfspp

2021-11-04 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16300.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Use libcrypto in Windows for libhdfspp
> --
>
> Key: HDFS-16300
> URL: https://issues.apache.org/jira/browse/HDFS-16300
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
> Environment: Windows
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
>  Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
> Attachments: build-log-hdfs-nacl-windows-10.log
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, eay32 is the library that's used in libhdfspp for Windows. 
> Whereas, we use libcrypto for the rest of the platforms. As per the following 
> mail thread, the OpenSSL library was renamed from eay32 to libcrypto from 
> OpenSSL version 1.1.0 onwards - 
> https://mta.openssl.org/pipermail/openssl-dev/2016-August/008351.html.
> Thus, we need to use libcrypto on Windows as well to ensure that we 
> standardize the version of the OpenSSL libraries used across platforms.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability

2021-11-08 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16307.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Improve HdfsBlockPlacementPolicies docs readability
> ---
>
> Key: HDFS-16307
> URL: https://issues.apache.org/jira/browse/HDFS-16307
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 3.3.1
>Reporter: guo
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> improve HdfsBlockPlacementPolicies docs readability



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16411) RBF: RouterId is NULL when set dfs.federation.router.rpc.enable=false

2022-02-08 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16411.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
 Assignee: Íñigo Goiri
   Resolution: Fixed

> RBF: RouterId is NULL when set dfs.federation.router.rpc.enable=false
> -
>
> Key: HDFS-16411
> URL: https://issues.apache.org/jira/browse/HDFS-16411
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: rbf
>Reporter: YulongZ
>Assignee: Íñigo Goiri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-16411.000.patch, HDFS-16411.001.patch, 
> HDFS-16411.002.patch, HDFS-16411.003.patch, HDFS-16411.004.patch, 
> HDFS-16411.005.patch
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> When dfs.federation.router.rpc.enable=false, routerid is null, but 
> RouterHeartbeatService need updateStateStore() with routerId.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16451) RBF: Add search box for Router's tab-mounttable web page

2022-02-09 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16451.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Add search box for Router's tab-mounttable web page
> 
>
> Key: HDFS-16451
> URL: https://issues.apache.org/jira/browse/HDFS-16451
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: rbf
>Affects Versions: 3.4.0
>Reporter: Max  Xie
>Assignee: Max  Xie
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: image-2022-02-09-18-17-53-498.png, 
> image-2022-02-09-18-18-29-262.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In our cluster, we have mount many paths in HDFS Router and it may lead to 
> take some time to load the mount-table page of Router when we open it  in the 
> browser.
> In order to use the mount-table page more conveniently, maybe we should add a 
> search box style, just like the screenshot below
> !image-2022-02-09-18-17-53-498.png!
> !image-2022-02-09-18-18-29-262.png!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16447) RBF: Registry HDFS Router's RPCServer & RPCClient metrics for PrometheusSink

2022-02-10 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16447.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Registry HDFS Router's RPCServer & RPCClient metrics for PrometheusSink
> 
>
> Key: HDFS-16447
> URL: https://issues.apache.org/jira/browse/HDFS-16447
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Affects Versions: 3.4.0
>Reporter: Max  Xie
>Assignee: Max  Xie
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When we enable PrometheusSink for HDFS Router,  Router' prometheus sink miss 
> some metrics, for example `RpcClientNumActiveConnections` and so on.
>  
> We need  registry some  Router's rpcserver & rpcclient metrics for 
> PrometheusSink.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16440) RBF: Support router get HAServiceStatus with Lifeline RPC address

2022-02-15 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16440.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Support router get HAServiceStatus with Lifeline RPC address
> -
>
> Key: HDFS-16440
> URL: https://issues.apache.org/jira/browse/HDFS-16440
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Reporter: YulongZ
>Assignee: YulongZ
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-16440.001.patch, HDFS-16440.003.patch, 
> HDFS-16440.004.patch
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> NamenodeHeartbeatService gets HAServiceStatus using 
> NNHAServiceTarget.getProxy. When we set a special 
> dfs.namenode.lifeline.rpc-address , NamenodeHeartbeatService may get 
> HAServiceStatus using NNHAServiceTarget.getHealthMonitorProxy.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16459) RBF: register RBFMetrics in MetricsSystem for promethuessink

2022-02-21 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16459.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: register RBFMetrics in MetricsSystem for promethuessink
> 
>
> Key: HDFS-16459
> URL: https://issues.apache.org/jira/browse/HDFS-16459
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Affects Versions: 3.4.0
>Reporter: Max  Xie
>Assignee: Max  Xie
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Router' RBFMetrics was not register  in MetricsSystem. We can't find these 
> metrics from PrometheusSink. Maybe we should fix it. 
>  
> After fix it ,  some  RBFMetrics will export like this 
> {code:java}
> # HELP rbf_metrics_current_tokens_count Number of router's current tokens
> # TYPE rbf_metrics_current_tokens_count gauge
> rbf_metrics_current_tokens_count{processname="Router",context="dfs",hostname=""}
>  2 {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-16577) Let administrator override connection details when registering datanodes

2022-05-11 Thread Jira
Teo Klestrup Röijezon created HDFS-16577:


 Summary: Let administrator override connection details when 
registering datanodes
 Key: HDFS-16577
 URL: https://issues.apache.org/jira/browse/HDFS-16577
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: datanode, namenode
Affects Versions: 3.2.2
Reporter: Teo Klestrup Röijezon


Currently (as of 3.2.2, but reading through the release notes this doesn't seem 
to have changed since then) DataNodes use the same properties for deciding 
which port to bind each service to, as for deciding which ports are included in 
the `DatanodeRegistration` sent to the NameNode. Further, NameNodes overwrite 
the DataNode's IP address with the incoming address during registration.

Both of these prevent external users from connecting to DataNodes that are 
hosted behind some sort of NAT (such as Kubernetes).

I have created a spike branch 
([https://github.com/stackabletech/hadoop/tree/spike/override-datanode-id,] 
based on v3.2.2) that I have confirmed solves this problem for us. There's 
clearly some work to be done integrating this properly (such as using the 
regular Hadoop config system and falling back to the old behaviour if no 
override is configured). I'd be happy to take that on to the best of my ability 
(with the caveats that I'm not super familiar with the Hadoop codebase, and 
that my Java is quite rusty at this point) if the overall direction seems 
acceptable.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16640) RBF: Show datanode IP list when click DN histogram in Router

2022-07-21 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16640.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Show datanode IP list when click DN histogram in Router
> 
>
> Key: HDFS-16640
> URL: https://issues.apache.org/jira/browse/HDFS-16640
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: wangzhaohui
>Assignee: wangzhaohui
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Show datanode IP list when click DN histogram in Router.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16605) Improve Code With Lambda in hadoop-hdfs-rbf moudle

2022-07-21 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16605.

Fix Version/s: 3.4.0
 Hadoop Flags: Incompatible change
   Resolution: Fixed

> Improve Code With Lambda in hadoop-hdfs-rbf moudle
> --
>
> Key: HDFS-16605
> URL: https://issues.apache.org/jira/browse/HDFS-16605
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16660) Improve Code With Lambda in IPCLoggerChannel class

2022-07-27 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16660.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Improve Code With Lambda in IPCLoggerChannel class
> --
>
> Key: HDFS-16660
> URL: https://issues.apache.org/jira/browse/HDFS-16660
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Improve Code With Lambda in IPCLoggerChannel class



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16687) RouterFsckServlet replicates code from DfsServlet base class

2022-08-03 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16687.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RouterFsckServlet replicates code from DfsServlet base class
> 
>
> Key: HDFS-16687
> URL: https://issues.apache.org/jira/browse/HDFS-16687
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: federation
>Reporter: Steve Vaughan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> RouterFsckServlet replicates the method "getUGI(HttpServletRequest request, 
> Configuration conf)" from DfsServlet instead of just extending DfsServlet.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets

2022-08-11 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-13274.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Extend RouterRpcClient to use multiple sockets
> ---
>
> Key: HDFS-13274
> URL: https://issues.apache.org/jira/browse/HDFS-13274
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> HADOOP-13144 introduces the ability to create multiple connections for the 
> same user and use different sockets. The RouterRpcClient should use this 
> approach to get a better throughput.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16678) RBF supports disable getNodeUsage() in RBFMetrics

2022-08-12 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16678.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF supports disable getNodeUsage() in RBFMetrics
> -
>
> Key: HDFS-16678
> URL: https://issues.apache.org/jira/browse/HDFS-16678
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> In our prod environment, we try to collect RBF metrics every 15s through 
> jmx_exporter. And we found that collection task often failed. 
> After tracing and found that the collection task is blocked at getNodeUsage() 
> in RBFMetrics, because it will collection all datanode's usage from 
> downstream nameservices.  This is a very expensive and almost useless 
> operation. Because in most scenarios, each NameSerivce contains almost the 
> same DNs. We can get the data usage's from any one nameservices, not from RBF.
> So I feel that RBF should supports disable getNodeUsage() in RBFMetrics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16705) RBF: Support healthMonitor timeout configurable and cache NN and client proxy in NamenodeHeartbeatService

2022-08-15 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16705.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Support healthMonitor timeout configurable and cache NN and client proxy 
> in NamenodeHeartbeatService
> -
>
> Key: HDFS-16705
> URL: https://issues.apache.org/jira/browse/HDFS-16705
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> When I read NamenodeHeartbeatService.class of RBF, I feel that there are 
> somethings we can do for NamenodeHeartbeatService.class.
>  * Cache NameNode Protocol and Client Protocol to avoid creating a new proxy 
> every time
>  * Supports healthMonitorTimeout configuration
>  * Format code of getNamenodeStatusReport to make it clearer



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16724) RBF should support get the information about ancestor mount points

2022-08-23 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16724.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF should support get the information about ancestor mount points
> --
>
> Key: HDFS-16724
> URL: https://issues.apache.org/jira/browse/HDFS-16724
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Suppose RBF cluster have 2 nameservices and to mount point as below:
>  * /user/ns1 -> ns1 -> /user/ns1
>  * /usre/ns2 -> ns2 -> /user/ns2
> Suppose we disable default nameservice of the RBF cluster and try to 
> getFileInfo of the path /user, RBF will throw one IOException to client due 
> to can not find locations for path /user. 
> But as this case, RBF should should return one valid response to client, 
> because /user has two sub mount point ns1 and ns2.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16736) Link to Boost library in libhdfspp

2022-08-26 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16736.

Hadoop Flags: Reviewed
  Resolution: Fixed

> Link to Boost library in libhdfspp
> --
>
> Key: HDFS-16736
> URL: https://issues.apache.org/jira/browse/HDFS-16736
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
> Environment: Windows 10
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: libhdfscpp, pull-request-available
>
> The compilation of HDFS Native Client fails on Windows 10 due to the 
> following error -
> {code}
> [exec] 
> "H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj"
>  (default target) (105) ->
> [exec]   rpc.lib(rpc_engine.obj) : error LNK2019: unresolved external symbol 
> "__declspec(dllimport) public: __cdecl 
> boost::gregorian::greg_month::greg_month(unsigned short)" 
> (__imp_??0greg_month@gregorian@boost@@QEAA@G@Z) referenced in function 
> "private: static class boost::posix_time::ptime __cdecl 
> boost::date_time::microsec_clock boost::posix_time::ptime>::create_time(struct tm * (__cdecl*)(__int64 const 
> *,struct tm *))" 
> (?create_time@?$microsec_clock@Vptime@posix_time@boost@@@date_time@boost@@CA?AVptime@posix_time@3@P6APEAUtm@@PEB_JPEAU6@@Z@Z)
>  
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj]
> [exec]   rpc.lib(request.obj) : error LNK2001: unresolved external symbol 
> "__declspec(dllimport) public: __cdecl 
> boost::gregorian::greg_month::greg_month(unsigned short)" 
> (__imp_??0greg_month@gregorian@boost@@QEAA@G@Z) 
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj]
> [exec]   
> H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\RelWithDebInfo\logging_test.exe
>  : fatal error LNK1120: 1 unresolved externals 
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj]
> {code}
> Thus, we need to link against the Boost library to resolve this error.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16734) RBF: fix some bugs when handling getContentSummary RPC

2022-08-26 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16734.

Fix Version/s: 3.4.0
   Resolution: Fixed

> RBF: fix some bugs when handling getContentSummary RPC
> --
>
> Key: HDFS-16734
> URL: https://issues.apache.org/jira/browse/HDFS-16734
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Suppose there are some mount points as bellows in RBF without default 
> namespace.
> ||Source Path||NameSpace||Destination Path ||
> |/a/b|ns0|/a/b|
> |/a/b/c|ns0|/a/b/c|
> |/a/b/c/d|ns1|/a/b/c/d|
> Suppose there a file /a/b/c/file1 with 10MB data in ns0 and a file 
> /a/b/c/d/file2 with 20MB data in ns1.
> There are bugs during handling some cases:
> ||Case Number||Case||Current Result||Expected Result||
> |1|getContentSummary('/a')|Throw RouterResolveException |2files and 30MB data|
> |2|getContentSummary('/a/b')|2files and 40MB data|3files and 40MB data|
> Bugs for these cases:
> Case1: If can't find any locations for the path,  RBF should try to do it 
> with sub mount points.
> Case2: RBF shouldn't repeatedly get content summary from one same namespace 
> with same ancestors path, such as from ns0 with /a/b and from ns0 with /a/b/c.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16750) NameNode should use NameNode.getRemoteUser() to log audit event to avoid possible NPE

2022-09-02 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16750.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> NameNode should use NameNode.getRemoteUser() to log audit event to avoid 
> possible NPE 
> --
>
> Key: HDFS-16750
> URL: https://issues.apache.org/jira/browse/HDFS-16750
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> NameNode should use NameNode.getRemoteUser() to log audit event to avoid 
> possible NPE.
> The relate code is: 
> {code:java}
> private void logAuditEvent(boolean succeeded, String cmd, String src,
> String dst, FileStatus stat) throws IOException {
>   if (isAuditEnabled() && isExternalInvocation()) {
> logAuditEvent(succeeded, Server.getRemoteUser(), Server.getRemoteIp(),
> cmd, src, dst, stat);
>   }
> }
> // the ugi may be null.
> private void logAuditEvent(boolean succeeded,
> UserGroupInformation ugi, InetAddress addr, String cmd, String src,
> String dst, FileStatus status) {
>   final String ugiStr = ugi.toString();
>   ...
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16770) [Documentation] RBF: Duplicate statement to be removed for better readabilty

2022-09-13 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16770.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> [Documentation] RBF: Duplicate statement to be removed for better readabilty
> 
>
> Key: HDFS-16770
> URL: https://issues.apache.org/jira/browse/HDFS-16770
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Both the below 2 statements gives the same meaning, later one can be removed.
> The Router monitors the local NameNode and its state and heartbeats to the 
> State Store.
> The Router monitors the local NameNode and heartbeats the state to the State 
> Store.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16772) refreshHostsReader should use the new configuration

2022-09-19 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16772.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> refreshHostsReader should use the new configuration
> ---
>
> Key: HDFS-16772
> URL: https://issues.apache.org/jira/browse/HDFS-16772
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> `refreshHostsReader` should use the latest configuration.
> And the current code as bellow:
> {code:java}
> /** Reread include/exclude files. */
> private void refreshHostsReader(Configuration conf) throws IOException {
>   if (conf == null) {
> conf = new HdfsConfiguration();
> // BUG here
> this.hostConfigManager.setConf(conf);
>   }
>   this.hostConfigManager.refresh();
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16777) datatables@1.10.17 sonatype-2020-0988 vulnerability

2022-10-12 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16777.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> datatables@1.10.17  sonatype-2020-0988 vulnerability
> 
>
> Key: HDFS-16777
> URL: https://issues.apache.org/jira/browse/HDFS-16777
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 3.3.4
>Reporter: Eugene Shinn (Truveta)
>Assignee: Ashutosh Gupta
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Our static analysis security tool detected that HDFS's UI currently includes 
> a vulnerable version of datatables detected by Sonatype (sonatype-2020-0988). 
> From the vulnerability description:
> _"The `datatables.net` package is vulnerable to Prototype Pollution. The 
> `setData` function in `jquery.dataTables.js` fails to protect prototype 
> attributes when objects are created during the application's execution. A 
> remote attacker can exploit this to modify the behavior of object prototypes 
> which, depending on their use in the application, may result in a Denial of 
> Service (DoS), Remote Code Execution (RCE), or other unexpected execution 
> flow."_
> This issue was addressed in v 1.11.5 (ref: [Fix: Protect developers from 
> inadvertantely introducing prototype pol… · 
> DataTables/Dist-DataTables@e2e19ea 
> (github.com)).|https://github.com/DataTables/Dist-DataTables/commit/e2e19eac7e5a6f140d7eefca5c7deba165b357eb#diff-e7d8309f017dd2ef6385fa8cdc1539a2R2765]
> N.B. this issue was also detected within the YARN UI as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl

2023-02-04 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16848.

Hadoop Flags: Reviewed
  Resolution: Fixed

> RBF: Improve StateStoreZookeeperImpl 
> -
>
> Key: HDFS-16848
> URL: https://issues.apache.org/jira/browse/HDFS-16848
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Reporter: Sun Hao
>Assignee: Sun Hao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Currently, router is getting/updating state from zk sequentially. It will 
> slowdown router load/update state cache especially for a large cluster or a 
> multi region cluster.
> We propose adding a threadpool to deal with zk state synchronization。



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16900) Method DataNode#isWrite seems not working in DataTransfer constructor method

2023-02-06 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16900.

Resolution: Won't Fix

> Method DataNode#isWrite seems not working in DataTransfer constructor method
> 
>
> Key: HDFS-16900
> URL: https://issues.apache.org/jira/browse/HDFS-16900
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 3.3.4
>Reporter: ZhangHB
>Priority: Major
>
> In constructor method of DataTransfer, there is codes below:
> {code:java}
> if (isTransfer(stage, clientname)) {
>   this.throttler = xserver.getTransferThrottler();
> } else if(isWrite(stage)) {
>   this.throttler = xserver.getWriteThrottler();
> } {code}
> the stage is a parameter of DataTransfer Constructor. Let us see where 
> instantiate DataTransfer object.
> In method transferReplicaForPipelineRecovery, codes like below:
> {code:java}
> final DataTransfer dataTransferTask = new DataTransfer(targets,
> targetStorageTypes, targetStorageIds, b, stage, client); {code}
> but the stage can never be PIPELINE_SETUP_STREAMING_RECOVERY or 
> PIPELINE_SETUP_APPEND_RECOVERY.
> It can only be TRANSFER_RBW or TRANSFER_FINALIZED.  So I think the method 
> isWrite is not working.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-16924) Add libhdfs APIs for createFile

2023-02-15 Thread Jira
Zoltán Borók-Nagy created HDFS-16924:


 Summary: Add libhdfs APIs for createFile
 Key: HDFS-16924
 URL: https://issues.apache.org/jira/browse/HDFS-16924
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: fs
Reporter: Zoltán Borók-Nagy


HDFS-14478 introduces builder-based APIs for openFile() based on HADOOP-15229.

We should also add builder-based APIs for createFile() based on HADOOP-14365.

This would be especially useful for object stores to tune performance of file 
writes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16953) RBF: Mount table store APIs should update cache only if state store record is successfully updated

2023-03-18 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16953.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Mount table store APIs should update cache only if state store record is 
> successfully updated
> --
>
> Key: HDFS-16953
> URL: https://issues.apache.org/jira/browse/HDFS-16953
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> RBF Mount table state store APIs addMountTableEntry, updateMountTableEntry 
> and removeMountTableEntry performs cache refresh for all routers regardless 
> of the actual record update result. If the record fails to get updated on 
> zookeeper/file based store impl, reloading the cache for all routers would be 
> unnecessary.
>  
> For instance, simultaneously adding new mount point could lead to failure for 
> the second call if first call has not added new entry by the time second call 
> retrieves mount table entry from getMountTableEntries before attempting to 
> call addMountTableEntry.
> {code:java}
> DEBUG [{cluster}/{ip}:8111] ipc.Client - IPC Client (1826699684) connection 
> to nn-0-{ns}.{cluster}/{ip}:8111 from {user}IPC Client (1826699684) 
> connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user} sending #1 
> org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol.addMountTableEntry
> DEBUG [{cluster}/{ip}:8111 from {user}] ipc.Client - IPC Client (1826699684) 
> connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user} got value #1
> DEBUG [main] ipc.ProtobufRpcEngine2 - Call: addMountTableEntry took 24ms
> DEBUG [{cluster}/{ip}:8111 from {user}] ipc.Client - IPC Client (1826699684) 
> connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user}: closed
> DEBUG [{cluster}/{ip}:8111 from {user}] ipc.Client - IPC Client (1826699684) 
> connection to nn-0-{ns}.{cluster}/{ip}:8111 from {user}: stopped, remaining 
> connections 0
> TRACE [main] ipc.ProtobufRpcEngine2 - 1: Response <- 
> nn-0-{ns}.{cluster}/{ip}:8111: addMountTableEntry {status: false}
> Cannot add mount point /data503 {code}
> The failure to write new record:
> {code:java}
> INFO  [IPC Server handler 0 on default port 8111] 
> impl.StateStoreZooKeeperImpl - Cannot write record 
> "/hdfs-federation/MountTable/0SLASH0data503", it already exists {code}
> Since the successful call has already refreshed cache for all routers, second 
> call that failed should not have refreshed cache for all routers again as 
> everyone already has updated records in cache.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16959) RBF: State store cache loading metrics

2023-03-29 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16959.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: State store cache loading metrics
> --
>
> Key: HDFS-16959
> URL: https://issues.apache.org/jira/browse/HDFS-16959
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> With increasing num of state store records (like mount points), it would be 
> good to be able to get the cache loading metrics like avg time for cache load 
> during refresh, num of times cache is loaded etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16973) RBF: MountTableResolver cache size lookup should take read lock

2023-04-05 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16973.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: MountTableResolver cache size lookup should take read lock
> ---
>
> Key: HDFS-16973
> URL: https://issues.apache.org/jira/browse/HDFS-16973
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Mount table resolver location cache gets invalidated by taking write lock as 
> part of addEntry/removeEntry/refreshEntries calls. Since the write lock 
> exclusively updates the cache, getDestinationForPath already takes read lock 
> before accessing the cache. Similarly, retrieval of the cache size should 
> also take the read lock.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16949) Update ReadTransferRate to ReadLatencyPerGB for effective percentile metrics

2023-04-10 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16949.

Hadoop Flags: Reviewed
  Resolution: Fixed

> Update ReadTransferRate to ReadLatencyPerGB for effective percentile metrics
> 
>
> Key: HDFS-16949
> URL: https://issues.apache.org/jira/browse/HDFS-16949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Reporter: Ravindra Dingankar
>Assignee: Ravindra Dingankar
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.0
>
>
> HDFS-16917 added ReadTransferRate quantiles to calculate the rate which data 
> is read per unit of time.
> With percentiles the values are sorted in ascending order and hence for the 
> transfer rate p90 gives us the value where 90 percent rates are lower 
> (worse), p99 gives us the value where 99 percent values are lower (worse).
> Note that value(p90) < p(99) thus p99 is a better transfer rate as compared 
> to p90.
> However as the percentile increases the value should become worse in order to 
> know how good our system is.
> Hence instead of calculating the data read transfer rate, we should calculate 
> it's inverse. We will instead calculate the time taken for a GB of data to be 
> read. ( seconds / GB )
> After this the p90 value will give us 90 percentage of total values where the 
> time taken is less than value(p90), similarly for p99 and others.
> Also p(90) < p(99) and here p(99) will become a worse value (taking more time 
> each byte) as compared to p(90)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16971) Add read time metrics for remote reads in Statistics

2023-04-13 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16971.

Fix Version/s: (was: 3.3.0)
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Add read time metrics for remote reads in Statistics
> 
>
> Key: HDFS-16971
> URL: https://issues.apache.org/jira/browse/HDFS-16971
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Reporter: Melissa You
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> In Filesystem.java, currently it only collects bytes read not the 
> corresponding timing of reads.
> In particular, we're interested in time spent on remote reads(not local 
> reads). Adding timing info will help us understand it better and further 
> analyze how locality of DN behaves. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16982) Use the right Quantiles Array for Inverse Quantiles snapshot

2023-04-18 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16982.

Fix Version/s: (was: 3.3.0)
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Use the right Quantiles Array for Inverse Quantiles snapshot 
> -
>
> Key: HDFS-16982
> URL: https://issues.apache.org/jira/browse/HDFS-16982
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode, metrics
>Reporter: Ravindra Dingankar
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> HDFS-16949 introduced InverseQuantiles. However during snapshot for Inverse 
> Quantiles we were still trying to access values from previous snapshot based 
> on the Quantile Array declared in MutableQuantiles. ( Quantile(.50, .050), 
> Quantile(.75, .025), Quantile(.90, .010), Quantile(.95, .005), Quantile(.99, 
> .001) )
> For InverseQuantiles we wont have these values ( except for Quantile(.50, 
> .050) ) thus except for 50 Percentile snapshot wont return any value for the 
> remaining quantiles.
> Fix is to use the correct Quantiles Array to retrieve values during snapshot. 
> The new UTs verify this behavior.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16998) RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity

2023-05-04 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16998.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity
> --
>
> Key: HDFS-16998
> URL: https://issues.apache.org/jira/browse/HDFS-16998
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17008) Fix RBF JDK 11 javadoc warnings

2023-05-12 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17008.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix RBF JDK 11 javadoc warnings
> ---
>
> Key: HDFS-17008
> URL: https://issues.apache.org/jira/browse/HDFS-17008
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> HDFS-16978 excluded proto packages from maven-javadoc-plugin for rbf, hence 
> now we have JDK 11 javadoc warnings (e.g. 
> [here|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5554/14/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt]).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17015) Typos in HDFS Documents

2023-05-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17015.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Typos in HDFS Documents
> ---
>
> Key: HDFS-17015
> URL: https://issues.apache.org/jira/browse/HDFS-17015
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: configuration
>Affects Versions: 3.3.5
>Reporter: Liang Yan
>Priority: Minor
> Fix For: 3.4.0
>
>
> There are some typos in HDFS documents. I will submit PR to fix these typos.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17009) RBF: state store putAll should also return failed records

2023-05-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17009.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: state store putAll should also return failed records
> -
>
> Key: HDFS-17009
> URL: https://issues.apache.org/jira/browse/HDFS-17009
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.4.0
>
>
> State store implementations allow adding/updating multiple records using 
> putAll. The implementation returns whether all records were successfully 
> added or updated. We should also allow the implementation to return which 
> records failed to get updated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17020) RBF: mount table addAll should print failed records in std error

2023-05-22 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17020.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: mount table addAll should print failed records in std error
> 
>
> Key: HDFS-17020
> URL: https://issues.apache.org/jira/browse/HDFS-17020
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Now that state store putAll supports returning failed records keys, addAll 
> command for mount entries should also support printing failed records in the 
> standard error.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17027) RBF: Add supports for observer.auto-msync-period when using routers

2023-05-31 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17027.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Add supports for observer.auto-msync-period when using routers
> ---
>
> Key: HDFS-17027
> URL: https://issues.apache.org/jira/browse/HDFS-17027
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> None-RBF clients that use observer reads have the option to set 
> *dfs.client.failover.observer.auto-msync-period.* . This config 
> makes the client automatically do an msync, allowing clients to use the 
> observer reads feature without any code change.
> To use observer reads with RBF, clients set 
> *dfs.client.rbf.observer.read.enable*. The way this flag is implemented does 
> not allow clients to use the *auto-msync-period* config. So with RBF, clients 
> either have to 
> # Not use observer reads
> # Use observer reads with the risk of stale reads
> # Make code changes to explicitly call msync.
> We should add support for 
> *dfs.client.failover.observer.auto-msync-period.*. This can be 
> done by adding a ProxyProvider, in a similar manner to the 
> ObserverReadProxyProvider.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17041) RBF: Fix putAll impl for mysql and file based state stores

2023-06-13 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17041.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Fix putAll impl for mysql and file based state stores
> --
>
> Key: HDFS-17041
> URL: https://issues.apache.org/jira/browse/HDFS-17041
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Only zookeeper based state store allows all records to be inserted even 
> though only few of them already exists and "errorIfExists" is true, however 
> file/fs as well as mysql based putAll fails the whole putAll operation 
> immediately after encountering single record that already exists in the 
> records and "errorIfExists" is provided true (which is the case while 
> inserting records for the first time).
> For all implementations, we should allow inserts of the records that do not 
> already exist and report any record as failure that already exists, rather 
> than failing the whole operation and not trying to insert valid records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17030) Limit wait time for getHAServiceState in ObserverReaderProxy

2023-06-14 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17030.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Limit wait time for getHAServiceState in ObserverReaderProxy
> 
>
> Key: HDFS-17030
> URL: https://issues.apache.org/jira/browse/HDFS-17030
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> When namenode HA is enabled and a standby NN is not responsible, we have 
> observed it would take a long time to serve a request, even though we have a 
> healthy observer or active NN. 
> Basically, when a standby is down, the RPC client would (re)try to create 
> socket connection to that standby for _ipc.client.connect.timeout_ _* 
> ipc.client.connect.max.retries.on.timeouts_ before giving up. When we take a 
> heap dump at a standby, the NN still accepts the socket connection but it 
> won't send responses to these RPC requests and we would timeout after 
> _ipc.client.rpc-timeout.ms._ This adds a significantly latency. For clusters 
> at Linkedin, we set _ipc.client.rpc-timeout.ms_ to 120 seconds and thus a 
> request takes more than 2 mins to complete when we take a heap dump at a 
> standby. This has been causing user job failures. 
> We could set _ipc.client.rpc-timeout.ms to_ a smaller value when sending 
> getHAServiceState requests in ObserverReaderProxy (for user rpc requests, we 
> still use the original value from the config). However, that would double the 
> socket connection between clients and the NN (which is a deal-breaker). 
> The proposal is to add a timeout on getHAServiceState() calls in 
> ObserverReaderProxy and we will only wait for the timeout for an NN to 
> respond its HA state. Once we pass that timeout, we will move on to probe the 
> next NN. 
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17042) Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode

2023-06-15 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17042.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode
> 
>
> Key: HDFS-17042
> URL: https://issues.apache.org/jira/browse/HDFS-17042
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.4.0, 3.3.9
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We'd like to add two new types of metrics to the existing NN 
> RpcMetrics/RpcDetailedMetrics. These two metrics can then be used as part of 
> SLA/SLO for the HDFS service.
>  * {_}RpcCallSuccesses{_}: it measures the number of RPC requests where they 
> are successfully processed by a NN (e.g., with a response with an RpcStatus 
> {_}RpcStatusProto.SUCCESS){_}{_}.{_} Then, together with {_}RpcQueueNumOps 
> ({_}which refers the total number of RPC requests{_}){_}, we can derive the 
> RpcErrorRate for our NN, as (RpcQueueNumOps - RpcCallSuccesses) / 
> RpcQueueNumOps. 
>  * OverallRpcProcessingTime for each RPC method: this metric measures the 
> overall RPC processing time for each RPC method at the NN. It covers the time 
> from when a request arrives at the NN to when a response is sent back. We are 
> already emitting processingTime for each RPC method today in 
> RpcDetailedMetrics. We want to extend it to emit overallRpcProcessingTime for 
> each RPC method, which includes enqueueTime, queueTime, processingTime, 
> responseTime, and handlerTime.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17057) RBF: Add DataNode maintenance states to Federation UI

2023-06-26 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17057.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Add DataNode maintenance states to Federation UI 
> --
>
> Key: HDFS-17057
> URL: https://issues.apache.org/jira/browse/HDFS-17057
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Add DataNode maintenance states to Federation UI 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17055) Export HAState as a metric from Namenode for monitoring

2023-06-26 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17055.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Export HAState as a metric from Namenode for monitoring
> ---
>
> Key: HDFS-17055
> URL: https://issues.apache.org/jira/browse/HDFS-17055
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.4.0, 3.3.9
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We'd like measure the uptime for Namenodes: percentage of time when we have 
> the active/standby/observer node available (up and running). We could monitor 
> the namenode from an external service, such as ZKFC. But that would require 
> the external service to be available 100% itself. And when this third-party 
> external monitoring service is down, we won't have info on whether our 
> Namenodes are still up.
> We propose to take a different approach: we will emit Namenode state directly 
> from namenode itself. Whenever we miss a data point for this metric, we 
> consider the corresponding namenode to be down/not available. In other words, 
> we assume the metric collection/monitoring infrastructure to be 100% reliable.
> One implementation detail: in hadoop, we have the _NameNodeMetrics_ class, 
> which is currently used to emit all metrics for {_}NameNode.java{_}. However, 
> we don't think that is a good place to emit NameNode HAState. HAState is 
> stored in NameNode.java and we should directly emit it from NameNode.java. 
> Otherwise, we basically duplicate this info in two classes and we would have 
> to keep them in sync. Besides, _NameNodeMetrics_ class does not have a 
> reference to the _NameNode_ object which it belongs to. An _NameNodeMetrics_ 
> is created by a _static_ function _initMetrics()_ in {_}NameNode.java{_}.
> We shouldn't emit HA state from FSNameSystem.java either, as it is 
> initialized from NameNode.java and all state transitions are implemented in 
> NameNode.java.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17065) Fix typos in hadoop-hdfs-project

2023-07-03 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17065.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix typos in hadoop-hdfs-project
> 
>
> Key: HDFS-17065
> URL: https://issues.apache.org/jira/browse/HDFS-17065
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhaohui Wang
>Assignee: Zhaohui Wang
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17067) Use BlockingThreadPoolExecutorService for nnProbingThreadPool in ObserverReadProxy

2023-07-20 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17067.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Use BlockingThreadPoolExecutorService for nnProbingThreadPool in 
> ObserverReadProxy
> --
>
> Key: HDFS-17067
> URL: https://issues.apache.org/jira/browse/HDFS-17067
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> In HDFS-17030, we introduced an ExecutorService, to submit 
> getHAServiceState() requests. We constructed the ExecutorService directly 
> from a basic ThreadPoolExecutor, without setting _allowCoreThreadTimeOut_ to 
> true. Then, the core thread will be kept up and running even when the main 
> thread exits. To fix it, one could set _allowCoreThreadTimeOut_ to true. 
> However, in this PR, we decide to directly use an existing executorService 
> implementation (_BlockingThreadPoolExecutorService_) in hadoop instead. It 
> takes care of setting _allowCoreThreadTimeOut_ and also allows setting the 
> prefix for thread names.
> {code:java}
>   private final ExecutorService nnProbingThreadPool =
>   new ThreadPoolExecutor(1, 4, 1L, TimeUnit.MINUTES,
>   new ArrayBlockingQueue(1024));
> {code}
> A second minor issue is we did not shutdown the executorService in close(). 
> It is a minor issue as close() will only be called when the garbage collector 
> starts to reclaim an ObserverReadProxyProvider object, not when there is no 
> reference to the ObserverReadProxyProvider object. The time between when an 
> ObserverReadProxyProvider becomes dereferenced and when the garage collector 
> actually starts to reclaim that object is out of control/under-defined 
> (unless the program is shutdown with an explicit System.exit(1)).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17118) Fix minor checkstyle warnings in TestObserverReadProxyProvider

2023-08-05 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17118.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix minor checkstyle warnings in TestObserverReadProxyProvider
> --
>
> Key: HDFS-17118
> URL: https://issues.apache.org/jira/browse/HDFS-17118
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Trivial
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We noticed a few checkstyle warnings when backporting HDFS-17030 from trunk 
> to branch-3.3. The yetus build was not stable at that time and we did not 
> notice the newly added checkstyle warnings.
> PR for HDFS-17030 which has been merged into trunk: 
> [https://github.com/apache/hadoop/pull/5700]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL

2023-08-11 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17148.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> ---
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Reporter: Hector Sandoval Chaverri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them 
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in 
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired 
> tokens from the cache, but most tokens have been evicted automatically per 
> the TTL configuration. This leads to many expired tokens in the SQL database 
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead 
> of in the memory cache when running the periodic cleanup.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16924) Add libhdfs APIs for createFile

2023-08-15 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltán Borók-Nagy resolved HDFS-16924.
--
Resolution: Fixed

Yes, I think we can resolve this.

> Add libhdfs APIs for createFile
> ---
>
> Key: HDFS-16924
> URL: https://issues.apache.org/jira/browse/HDFS-16924
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Reporter: Zoltán Borók-Nagy
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> HDFS-14478 introduces builder-based APIs for openFile() based on HADOOP-15229.
> We should also add builder-based APIs for createFile() based on HADOOP-14365.
> This would be especially useful for object stores to tune performance of file 
> writes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17162) RBF: Add missing comments in StateStoreService

2023-08-23 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17162.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Add missing comments in StateStoreService
> --
>
> Key: HDFS-17162
> URL: https://issues.apache.org/jira/browse/HDFS-17162
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: rbf
>Reporter: TIsNotT
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> StateStoreService lacks an introduction to StateStoreFileSystemImpl and 
> StateStoreMySQLImpl [link 
> HDFS-16943|https://issues.apache.org/jira/projects/HDFS/issues/HDFS-16943?filter=allissues].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17178) BootstrapStandby needs to handle RollingUpgrade

2023-09-12 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17178.

Hadoop Flags: Reviewed
  Resolution: Fixed

> BootstrapStandby needs to handle RollingUpgrade 
> 
>
> Key: HDFS-17178
> URL: https://issues.apache.org/jira/browse/HDFS-17178
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Danny Becker
>Assignee: Danny Becker
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> During rollingUpgrade, bootstrapStandby will fail with an exception due to 
> different NameNodeLayoutVersions. We can ignore this safely during 
> RollingUpgrade because different NameNodeLayoutVersions are expected.
>  * NameNodes will not be able to recover with BootstrapStandby if they go 
> through destructive repair before the rollingUpgrade has been finalized.
> Error during BootstrapStandby before change:
> {code:java}
> =
> About to bootstrap Standby ID nn2 from:
>Nameservice ID: MTPrime-MWHE01-0
> Other Namenode ID: nn1
>   Other NN's HTTP address: https://MWHEEEAP002D9A2:81
>   Other NN's IPC  address: MWHEEEAP002D9A2.ap.gbl/10.59.208.18:8020
>  Namespace ID: 895912530
> Block pool ID: BP-1556042256-10.99.154.61-1663325602669
>Cluster ID: MWHE01
>Layout version: -64
>isUpgradeFinalized: true
> =
> 2023-08-28T19:35:06,940 ERROR [main] namenode.NameNode: Failed to start 
> namenode.
> java.io.IOException: java.lang.RuntimeException: 
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException:
>  Image transfer servlet at 
> https://MWHEEEAP002D9A2:81/imagetransfer?getimage=1&txid=25683470&storageInfo=-64:895912530:1663325602669:MWHE01&bootstrapstandby=true
>  failed with status code 403
> Response message:
> This namenode has storage info -63:895912530:1663325602669:MWHE01 but the 
> secondary expected -64:895912530:1663325602669:MWHE01
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:583)
>  ~[hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?]
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1717)
>  ~[hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?]
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1819) 
> [hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?]
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException:
>  Image transfer servlet at https://MWHEEEAP002D9A2:81{code}
> This is caused because the namespaceInfo sent from the proxy node does not 
> include the effective layout version, which causes BootstrapStandby to send a 
> request with a storageinfo param using the service layout version. This 
> causes the proxy node to refuse the request, because it compares the 
> storageinfo param against its storage info, which uses the effective layout 
> version, not the service layout version. 
> To fix this we can modify the proxy.versionRequest() call stack to set the 
> layout version using the effective layout version on the proxy node. We can 
> then add logic to BootstrapStandby to properly handle the case where the 
> proxy node is in rolling upgrade.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working

2023-09-18 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-17138.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: We changed the hadoop.security.auth_to_local configuration of one 
> router, the other routers stopped working
> 
>
> Key: HDFS-17138
> URL: https://issues.apache.org/jira/browse/HDFS-17138
> Project: Hadoop HDFS
>  Issue Type: Bug
> Environment: hadoop 3.3.0
>Reporter: Xiping Zhang
>Assignee: Xiping Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: image-2023-08-02-16-20-34-454.png, 
> image-2023-08-03-10-32-03-457.png
>
>
> other routers  error log:
> !image-2023-08-02-16-20-34-454.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-14943) RBF: Add Namespace to the Overview page in the Web UI

2019-10-30 Thread Jira
Íñigo Goiri created HDFS-14943:
--

 Summary: RBF: Add Namespace to the Overview page in the Web UI
 Key: HDFS-14943
 URL: https://issues.apache.org/jira/browse/HDFS-14943
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri


The Namenode shows the Namespace field which can be used to access it through 
HDFS.
The Router should also report its namespace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-14961) TestDFSZKFailoverController fails consistently

2019-11-06 Thread Jira
Íñigo Goiri created HDFS-14961:
--

 Summary: TestDFSZKFailoverController fails consistently
 Key: HDFS-14961
 URL: https://issues.apache.org/jira/browse/HDFS-14961
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Íñigo Goiri


TestDFSZKFailoverController has been consistently failing with a time out 
waiting in testManualFailoverWithDFSHAAdmin(). In particular 
{{waitForHAState(1, HAServiceState.OBSERVER);}}.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-14974) RBF: TestRouterSecurityManager#testCreateCredentials should use :0 for port

2019-11-08 Thread Jira
Íñigo Goiri created HDFS-14974:
--

 Summary: RBF: TestRouterSecurityManager#testCreateCredentials 
should use :0 for port
 Key: HDFS-14974
 URL: https://issues.apache.org/jira/browse/HDFS-14974
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri


Currently, {{TestRouterSecurityManager#testCreateCredentials}} create a Router 
with the default ports. However, these ports might be used. We should set it to 
:0 for it to be assigned dynamically.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15016) RBF: getDatanodeReport() should return the latest update

2019-11-26 Thread Jira
Íñigo Goiri created HDFS-15016:
--

 Summary: RBF: getDatanodeReport() should return the latest update
 Key: HDFS-15016
 URL: https://issues.apache.org/jira/browse/HDFS-15016
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri


Currently, when the Router calls getDatanodeReport() (or 
getDatanodeStorageReport()) and the DN is in multiple clusters, it just takes 
the one that comes first. It should consider the latest update.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15075) Remove process command timing from BPServiceActor

2019-12-19 Thread Jira
Íñigo Goiri created HDFS-15075:
--

 Summary: Remove process command timing from BPServiceActor
 Key: HDFS-15075
 URL: https://issues.apache.org/jira/browse/HDFS-15075
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri


HDFS-14997 moved the command processing into async.
Right now, we are checking the time to add to a queue.
We should remove this one and maybe move the timing within the thread.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15062) Add LOG when sendIBRs failed

2019-12-20 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15062.

Resolution: Fixed

> Add LOG when sendIBRs failed
> 
>
> Key: HDFS-15062
> URL: https://issues.apache.org/jira/browse/HDFS-15062
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.3, 3.2.1, 3.1.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-15062.001.patch, HDFS-15062.002.patch, 
> HDFS-15062.003.patch
>
>
> {code}
>   /** Send IBRs to namenode. */
>   void sendIBRs(DatanodeProtocol namenode, DatanodeRegistration registration,
>   String bpid, String nnRpcLatencySuffix) throws IOException {
> // Generate a list of the pending reports for each storage under the lock
> final StorageReceivedDeletedBlocks[] reports = generateIBRs();
> if (reports.length == 0) {
>   // Nothing new to report.
>   return;
> }
> // Send incremental block reports to the Namenode outside the lock
> if (LOG.isDebugEnabled()) {
>   LOG.debug("call blockReceivedAndDeleted: " + Arrays.toString(reports));
> }
> boolean success = false;
> final long startTime = monotonicNow();
> try {
>   namenode.blockReceivedAndDeleted(registration, bpid, reports);
>   success = true;
> } finally {
>   if (success) {
> dnMetrics.addIncrementalBlockReport(monotonicNow() - startTime,
> nnRpcLatencySuffix);
> lastIBR = startTime;
>   } else {
> // If we didn't succeed in sending the report, put all of the
> // blocks back onto our queue, but only in the case where we
> // didn't put something newer in the meantime.
> putMissing(reports);
>   }
> }
>   }
> {code}
> When call namenode.blockReceivedAndDelete failed, will put reports to 
> pendingIBRs. Maybe we should add log for failed case. It is helpful for 
> trouble shooting



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15112) RBF: do not return FileNotFoundException when a subcluster is unavailable

2020-01-10 Thread Jira
Íñigo Goiri created HDFS-15112:
--

 Summary: RBF: do not return FileNotFoundException when a 
subcluster is unavailable 
 Key: HDFS-15112
 URL: https://issues.apache.org/jira/browse/HDFS-15112
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri


If we have a mount point using HASH_ALL across two subclusters and one of them 
is down, we may return FileNotFoundException while the file is just in the 
unavailable subcluster.
We should not return FileNotFoundException but something that shows that the 
subcluster is unavailable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15127) RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points.

2020-01-16 Thread Jira
Íñigo Goiri created HDFS-15127:
--

 Summary: RBF: Do not allow writes when a subcluster is unavailable 
for HASH_ALL mount points.
 Key: HDFS-15127
 URL: https://issues.apache.org/jira/browse/HDFS-15127
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri


A HASH_ALL mount point should not allow creating new files if one subcluster is 
down.
If the file already existed in the past, this could lead to inconsistencies.
We should return an unavailable exception.
{{TestRouterFaultTolerant#testWriteWithFailedSubcluster()}} needs to be changed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15163) hdfs-2.10.0-webapps-secondary-status.html miss moment.js

2020-02-12 Thread Jira
谢波 created HDFS-15163:
-

 Summary: hdfs-2.10.0-webapps-secondary-status.html miss moment.js
 Key: HDFS-15163
 URL: https://issues.apache.org/jira/browse/HDFS-15163
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 2.10.0
Reporter: 谢波
 Fix For: 2.10.1
 Attachments: 微信截图_20200212183444.png

hdfs-2.10.0-webapps-secondary-status.html miss moment.js

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15184) Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned:

2020-02-19 Thread Jira
任建亭 created HDFS-15184:
--

 Summary: Failed to execute goal 
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project 
hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
 Key: HDFS-15184
 URL: https://issues.apache.org/jira/browse/HDFS-15184
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 3.2.1
 Environment: windows 10

JDK 1.8

maven3.6.1

ProtocolBuffer 2.5.0

CMake 3.1.3

git 2.25.0

zlib 1.2.5

Visual Studio 2010 Professional
Reporter: 任建亭
 Fix For: 3.2.1


When I build hadoop 3.2.1 on windows10, it failed. My command is 'mvn clean 
package -Pdist,native-win -DskipTests -Dtar'.
{code:java}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project 
hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part .. @ 9:122 in 
D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-11439) testGenerationStampInFuture UT fails

2020-03-13 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-11439.

Resolution: Duplicate

> testGenerationStampInFuture UT fails
> 
>
> Key: HDFS-11439
> URL: https://issues.apache.org/jira/browse/HDFS-11439
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Yesha Vora
>Priority: Major
> Attachments: testGenerationStampInFuture.log
>
>
> testGenerationStampInFuture UT fails as below.
> {code}
> Error Message
> expected:<18> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<18> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency.testGenerationStampInFuture(TestNameNodeMetadataConsistency.java:125){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15215) The Timestamp for longest write/read lock held log is wrong

2020-03-24 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15215.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> The Timestamp for longest write/read lock held log is wrong
> ---
>
> Key: HDFS-15215
> URL: https://issues.apache.org/jira/browse/HDFS-15215
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.3.0
>
>
> I found the Timestamp for longest write/read lock held log is wrong in trunk:
> {code}
> 2020-03-10 16:01:26,585 [main] INFO  namenode.FSNamesystem 
> (FSNamesystemLock.java:writeUnlock(281)) - Number of suppressed 
> write-lock reports: 0
>   Longest write-lock held at 1970-01-03 07:07:40,841+0900 for 3ms via 
> java.lang.Thread.getStackTrace(Thread.java:1559)
> ...
> {code}
> Looking at the code, it looks like the timestamp comes from System.nanoTime() 
> that returns the current value of the running Java Virtual Machine's 
> high-resolution time source and this method can only be used to measure 
> elapsed time:
> https://docs.oracle.com/javase/8/docs/api/java/lang/System.html#nanoTime--
> We need to make the timestamp from System.currentTimeMillis().



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15239) Add button to go to the parent directory in the explorer

2020-03-24 Thread Jira
Íñigo Goiri created HDFS-15239:
--

 Summary: Add button to go to the parent directory in the explorer
 Key: HDFS-15239
 URL: https://issues.apache.org/jira/browse/HDFS-15239
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri


Currently, when using the HDFS explorer page, it is easy to go into a folder.
However, to go back one has to use the browser back button (if one is coming 
from that folder) or to edit the path by hand.
It would be nice to have the typical button to go to the parent.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15217) Add more information to longest write/read lock held log

2020-04-18 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15217.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Add more information to longest write/read lock held log
> 
>
> Key: HDFS-15217
> URL: https://issues.apache.org/jira/browse/HDFS-15217
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.4.0
>
>
> Currently, we can see the stack trace in the longest write/read lock held 
> log, but sometimes we need more information, for example, a target path of 
> deletion:
> {code:java}
> 2020-03-10 21:51:21,116 [main] INFO  namenode.FSNamesystem 
> (FSNamesystemLock.java:writeUnlock(276)) - Number of suppressed 
> write-lock reports: 0
>   Longest write-lock held at 2020-03-10 21:51:21,107+0900 for 6ms via 
> java.lang.Thread.getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1058)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:257)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:233)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1706)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3188)
> ...
> {code}
> Adding more information (opName, path, etc.) to the log is useful to 
> troubleshoot.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15345) RBF: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442

2020-05-12 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15345.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups 
> after HADOOP-13442
> 
>
> Key: HDFS-15345
> URL: https://issues.apache.org/jira/browse/HDFS-15345
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 2.7.5
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Fix For: 3.4.0
>
>
> HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This 
> ticket is opened to change  RouterPermissionChecker#checkSuperuserPrivilege 
> to use UGI#getGroups. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15184) Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned

2020-06-06 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

任建亭 resolved HDFS-15184.

Resolution: Not A Problem

> Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run 
> (make) on project hadoop-hdfs-native-client: An Ant BuildException has 
> occured: exec returned: 1
> 
>
> Key: HDFS-15184
> URL: https://issues.apache.org/jira/browse/HDFS-15184
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.2.1
> Environment: windows 10
> JDK 1.8
> maven3.6.1
> ProtocolBuffer 2.5.0
> CMake 3.1.3
> git 2.25.0
> zlib 1.2.5
> Visual Studio 2010 Professional
>Reporter: 任建亭
>Priority: Major
> Fix For: 3.2.1
>
>
> When I build hadoop 3.2.1 on windows10, it failed. My command is 'mvn clean 
> package -Pdist,native-win -DskipTests -Dtar'.
> {code:java}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project 
> hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
> [ERROR] around Ant part ... dir="D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" 
> executable="msbuild">... @ 9:122 in 
> D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15383) RBF: Disable watch in ZKDelegationSecretManager for performance

2020-06-23 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15383.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Disable watch in ZKDelegationSecretManager for performance
> ---
>
> Key: HDFS-15383
> URL: https://issues.apache.org/jira/browse/HDFS-15383
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Fix For: 3.4.0
>
>
> Based on the current design for delegation token in secure Router, the total 
> number of watches for tokens is the product of number of routers and number 
> of tokens, this is due to ZKDelegationTokenManager is using PathChildrenCache 
> from curator, which automatically sets the watch and ZK will push the sync 
> information to each router. There are some evaluations about the number of 
> watches in Zookeeper has negative performance impact to Zookeeper server.
> In our practice when the number of watches exceeds 1.2 Million in a single ZK 
> server there will be significant ZK performance degradation. Thus this ticket 
> is to rewrite ZKDelegationTokenManagerImpl.java to explicitly disable the 
> PathChildrenCache and have Routers sync periodically from Zookeeper. This has 
> been working fine at the scale of 10 Routers with 2 million tokens. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15312) Apply umask when creating directory by WebHDFS

2020-07-06 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15312.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Apply umask when creating directory by WebHDFS
> --
>
> Key: HDFS-15312
> URL: https://issues.apache.org/jira/browse/HDFS-15312
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Ye Ni
>Assignee: Ye Ni
>Priority: Minor
> Fix For: 3.4.0
>
>
> WebHDFS methods for creating file/directories were always creating it with 
> 755 permissions as default for both files and directories.
> The configured *fs.permissions.umask-mode* is intentionally ignored.
> This Jira is to apply this setting in such scenario.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15600) TestRouterQuota fails in trunk

2020-09-29 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15600.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> TestRouterQuota fails in trunk
> --
>
> Key: HDFS-15600
> URL: https://issues.apache.org/jira/browse/HDFS-15600
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: rbf
>Reporter: Ayush Saxena
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The test is failing due to addition of a new storage type {{NVDIMM}} in 
> middle.
> Ref :
> https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/204/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterQuota/testStorageTypeQuota/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-12449) TestReconstructStripedFile.testNNSendsErasureCodingTasks randomly cannot finish in 60s

2020-10-27 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-12449.

Resolution: Fixed

> TestReconstructStripedFile.testNNSendsErasureCodingTasks randomly cannot 
> finish in 60s
> --
>
> Key: HDFS-12449
> URL: https://issues.apache.org/jira/browse/HDFS-12449
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha4
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
>  Labels: flaky-test
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12449.001.patch
>
>
> TestReconstructStripedFile.testNNSendsErasureCodingTasks randomly cannot 
> finish in 60s
> reduce the file size and loop count



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15656) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails occasionally

2020-10-27 Thread Jira
Íñigo Goiri created HDFS-15656:
--

 Summary: TestReconstructStripedFile.testNNSendsErasureCodingTasks 
fails occasionally
 Key: HDFS-15656
 URL: https://issues.apache.org/jira/browse/HDFS-15656
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Íñigo Goiri


HDFS-12449 tried to fix this issue but this is still failing intermittently.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15654) TestBPOfferService#testMissBlocksWhenReregister fails intermittently

2020-10-28 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15654.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> TestBPOfferService#testMissBlocksWhenReregister fails intermittently
> 
>
> Key: HDFS-15654
> URL: https://issues.apache.org/jira/browse/HDFS-15654
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Ahmed Hussein
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> {{TestBPOfferService.testMissBlocksWhenReregister}}  is flaky. It fails 
> randomly when the 
> following expression is not true:
> {code:java}
>   assertTrue(fullBlockReportCount == totalTestBlocks ||
>   incrBlockReportCount == totalTestBlocks);
> {code}
> There is a race condition here that relies once more on "time" to synchronize 
> between concurrent threads. The code below is is causing the 
> non-deterministic execution.
> On a slow server, {{addNewBlockThread}} may not be done by the time the main 
> thread reach the assertion call.
> {code:java}
>   // Verify FBR/IBR count is equal to generate number.
>   assertTrue(fullBlockReportCount == totalTestBlocks ||
>   incrBlockReportCount == totalTestBlocks);
> } finally {
>   addNewBlockThread.join();
>   bpos.stop();
>   bpos.join();
> {code}
> Therefore, the correct implementation should wait for the thread to finish
> {code:java}
>  // the thread finished execution.
>  addNewBlockThread.join();
>   // Verify FBR/IBR count is equal to generate number.
>   assertTrue(fullBlockReportCount == totalTestBlocks ||
>   incrBlockReportCount == totalTestBlocks);
> } finally {
>   bpos.stop();
>   bpos.join();
> {code}
> {{DataNodeFaultInjector}} needs to have a longer wait_time too. 1 second is 
> not enough to satisfy the condition.
> {code:java}
>   DataNodeFaultInjector.set(new DataNodeFaultInjector() {
> public void blockUtilSendFullBlockReport() {
>   try {
> GenericTestUtils.waitFor(() -> {
>   if(count.get() > 2000) {
> return true;
>   }
>   return false;
> }, 100, 1); // increase that waiting time to 10 seconds.
>   } catch (Exception e) {
> e.printStackTrace();
>   }
> }
>   });
> {code}
> {code:bash}
> Stacktrace
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister(TestBPOfferService.java:350)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Prov

[jira] [Created] (HDFS-15692) Improve furse_dfs read performace

2020-11-23 Thread Jira
István Fajth created HDFS-15692:
---

 Summary: Improve furse_dfs read performace
 Key: HDFS-15692
 URL: https://issues.apache.org/jira/browse/HDFS-15692
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: fuse-dfs
Reporter: István Fajth


Currently fuse_dfs uses a prefetch buffer to read from HDFS via libhdfs' pread 
method.

The algorithm inside fuse_read.c in short does the following:
 if the rdbuffer size is less then the buffer provided
 then
  reads directly to the buffer
 else
  grab lock
    if the preftch buffer does not have more data
    then
      fills the prefetch buffer
    endif
    fills the supplied buffer via memcpy from the prefetch buffer
  release lock
endif

It would be nice to have a background thread and double prefetch buffers, so 
while one buffer serves the reads coming from the local client, the other can 
prefetch the data, with that we can improve the read speed, especially with EC 
encoded files.

According to some measurements I did, if I increase the read buffer, there is a 
significant change in runtime, with 64MB the runtime is really closer to HDFS 
by a large margin. Interestingly 128MB as the buffer size does not perform 
well, but 256MB is even more closer to what the dfs client can provide. (16 vs 
18 seconds with rep3 files, while in par with ec encoded files dfs vs fuse)

So it seems it is worth to stream continuously a larger chunk of data, at least 
with pread, but in case we have a separate fetching thread and double 
buffering, we don't even need positioned reads, simply just continuous 
streaming of data with read.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-15693) Improve native code's performance when writing to HDFS

2020-11-23 Thread Jira
István Fajth created HDFS-15693:
---

 Summary: Improve native code's performance when writing to HDFS
 Key: HDFS-15693
 URL: https://issues.apache.org/jira/browse/HDFS-15693
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: fuse-dfs, native
Reporter: István Fajth


For reads, we introduced direct buffers in order to more efficiently 
communicate between the JVM and the native code, and we have readDirect and 
pReadDirect in hdfs.c implemented.

Writes on the other hand still use the putByteArrayRegion call, which results 
in a copy of the buffer in memory.

This Jira is to explore what has to be done in order to start to use direct 
buffers.
A short initial list I see at the moment:
- add a new StreamCapability for streams wanting to support writes via direct 
buffer
- implement this capability in the DFSOutputStream and DFSStripedOutputStream
- implement a writeDirect method on the native side

fuse_dfs can benefit from this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15163) hdfs-2.10.0-webapps-secondary-status.html miss moment.js

2020-11-30 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

谢波 resolved HDFS-15163.
---
Resolution: Fixed

> hdfs-2.10.0-webapps-secondary-status.html miss moment.js
> 
>
> Key: HDFS-15163
> URL: https://issues.apache.org/jira/browse/HDFS-15163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.10.0
>Reporter: 谢波
>Priority: Minor
> Attachments: 微信截图_20200212183444.png
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> hdfs-2.10.0-webapps-secondary-status.html miss moment.js
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15843) [libhdfs++] Make write cross platform

2021-03-16 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15843.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> [libhdfs++] Make write cross platform
> -
>
> Key: HDFS-15843
> URL: https://issues.apache.org/jira/browse/HDFS-15843
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: libhdfs++
>Affects Versions: 3.2.2
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We're currently using the *write* function from unistd.h which isn't 
> cross-platform. We need to replace this with *std::cout.write* instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15903) Refactor X-Platform library

2021-03-22 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15903.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Refactor X-Platform library
> ---
>
> Key: HDFS-15903
> URL: https://issues.apache.org/jira/browse/HDFS-15903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: libhdfs++
>Affects Versions: 3.2.2
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> X-Platform started out as a utility to help in writing cross platform code in 
> Hadoop. As its scope expanding to cover various scenarios, it is necessary to 
> refactor it in early stages to provide proper organization and growth of the 
> X-Platform library.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15913) Remove useless NULL checks before instanceof

2021-03-23 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15913.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Remove useless NULL checks before instanceof
> 
>
> Key: HDFS-15913
> URL: https://issues.apache.org/jira/browse/HDFS-15913
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Reporter: Jiajun Jiang
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-15913.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Remove useless NULL checks before instanceof in several classes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15910) Replace bzero with explicit_bzero for better safety

2021-03-23 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15910.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Replace bzero with explicit_bzero for better safety
> ---
>
> Key: HDFS-15910
> URL: https://issues.apache.org/jira/browse/HDFS-15910
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.2.2
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> It is better to always use explicit_bzero since it guarantees that the buffer 
> will be cleared irrespective of the compiler optimizations - 
> https://man7.org/linux/man-pages/man3/bzero.3.html.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15918) Replace RAND_pseudo_bytes in sasl_digest_md5.cc

2021-03-24 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15918.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Replace RAND_pseudo_bytes in sasl_digest_md5.cc
> ---
>
> Key: HDFS-15918
> URL: https://issues.apache.org/jira/browse/HDFS-15918
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following 
> warning during compilation that it's deprecated -
> {code}
> [WARNING] 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:97:74:
>  warning: 'int RAND_pseudo_bytes(unsigned char*, int)' is deprecated 
> [-Wdeprecated-declarations]
> [WARNING]  from 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:20:
> [WARNING] /usr/include/openssl/rand.h:44:1: note: declared here
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15922) Use memcpy for copying non-null terminated string in jni_helper.c

2021-03-26 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15922.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Use memcpy for copying non-null terminated string in jni_helper.c
> -
>
> Key: HDFS-15922
> URL: https://issues.apache.org/jira/browse/HDFS-15922
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We currently get a warning while compiling HDFS native client -
> {code}
> [WARNING] inlined from 'wildcard_expandPath' at 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:427:21,
> [WARNING] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: 
> warning: '__builtin_strncpy' output truncated before terminating nul copying 
> as many bytes from a string as its length [-Wstringop-truncation]
> [WARNING] 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:402:43:
>  note: length computed here
> {code}
> The scenario here is such that the copied string is deliberately not null 
> terminated, since we want to insert a PATH_SEPARATOR ourselves. The warning 
> reported by strncpy is valid, but not applicable in this scenario. Thus, we 
> need to use memcpy which doesn't mind if the string is null terminated or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15929) Replace RAND_pseudo_bytes in util.cc

2021-03-30 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15929.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Replace RAND_pseudo_bytes in util.cc
> 
>
> Key: HDFS-15929
> URL: https://issues.apache.org/jira/browse/HDFS-15929
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> RAND_pseudo_bytes was deprecated in OpenSSL 1.1.1. We get the following 
> warning during compilation that it's deprecated -
> {code}
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:
>  In function ‘std::string hdfs::GetRandomClientName()’:
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
>  warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated 
> [-Wdeprecated-declarations]
>78 |   RAND_pseudo_bytes(&buf[0], 8);
>   |   ^
> In file included from /usr/include/openssl/e_os2.h:13,
>  from /usr/include/openssl/ossl_typ.h:19,
>  from /usr/include/openssl/rand.h:14,
>  from 
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h:29,
>  from 
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:19:
> /usr/include/openssl/rand.h:44:1: note: declared here
>44 | DEPRECATEDIN_1_1_0(int RAND_pseudo_bytes(unsigned char *buf, int num))
>   | ^~
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:31:
>  warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated 
> [-Wdeprecated-declarations]
>    78 |   RAND_pseudo_bytes(&buf[0], 8);
>   |   ^
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15927) Catch polymorphic type by reference

2021-03-30 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15927.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Catch polymorphic type by reference
> ---
>
> Key: HDFS-15927
> URL: https://issues.apache.org/jira/browse/HDFS-15927
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Need to catch polymorphic exception types by reference in order to realize 
> the polymorphic usage, if any. Otherwise, the functionality of the caught 
> object is restricted to only that of the base class.
> Also, the following warning gets reported since it's currently caught by 
> value -
> {code:java}
> [WARNING] 
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:138:22:
>  warning: catching polymorphic type 'const class hdfs::uri_parse_error' by 
> value [-Wcatch-value=]
> [WARNING] 
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2792/src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc:151:27:
>  warning: catching polymorphic type 'struct hdfs::ha_parse_error' by value 
> [-Wcatch-value=]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15944) Prevent truncation by snprintf

2021-04-02 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15944.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Prevent truncation by snprintf
> --
>
> Key: HDFS-15944
> URL: https://issues.apache.org/jira/browse/HDFS-15944
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs, libhdfs
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> There are some areas of code in libhdfs and fuse-dfs components where the 
> destination buffer is smaller than the source that's trying to write. This 
> would cause truncation. Thus we need to ensure that the source that's being 
> written doesn't exceed the destination buffer size.
> The following warnings are reported for this issue -
> {code}
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:
>  In function ‘doTestHdfsOperations.isra.0’:
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:226:39:
>  warning: ‘/many_files_’ directive output may be truncated writing 12 bytes 
> into a region of size between 1 and 4096 [-Wformat-truncation=]
>   226 |   snprintf(filename, PATH_MAX, "%s/many_files_%d", listDirTest, 
> nFile);
>   |   ^~~~
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:226:36:
>  note: directive argument in the range [0, ]
>   226 |   snprintf(filename, PATH_MAX, "%s/many_files_%d", listDirTest, 
> nFile);
>   |^~
> In file included from /usr/include/stdio.h:867,
>  from 
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/expect.h:23,
>  from 
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:19:
> /usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: 
> ‘__builtin___snprintf_chk’ output between 14 and 4112 bytes into a 
> destination of size 4096
>67 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
>   |  ^~~~
>68 |__bos (__s), __fmt, __va_arg_pack ());
>   |~
> {code}
> {code}
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/test/fuse_workload.c:255:33:
>  warning: ‘/a’ directive output may be truncated writing 2 bytes into a 
> region of size between 1 and 4096 [-Wformat-truncation=]
>   255 |   snprintf(tmp, sizeof(tmp), "%s/a", base);
>   | ^~
> In file included from /usr/include/stdio.h:867,
>  from 
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/expect.h:23,
>  from 
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/test/fuse_workload.c:22:
> /usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: 
> ‘__builtin___snprintf_chk’ output between 3 and 4098 bytes into a destination 
> of size 4096
>67 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
>   |  ^~~~
>68 |__bos (__s), __fmt, __va_arg_pack ());
>   |~
> /mnt/c/Users/Gautham/projects/apache/wsl/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/test/fuse_workload.c:263:33:
>  warning: ‘/b’ directive output may be truncated writing 2 bytes into a 
> region of size between 1 and 4096 [-Wformat-truncation=]
>   263 |   snprintf(tmp, sizeof(tmp), "%s/b", base);
>   | ^~
> In file included from /usr/include/stdio.h:867,
>  from 
> /mnt/c/Users/Gautham/pr

[jira] [Resolved] (HDFS-15947) Replace deprecated protobuf APIs

2021-04-05 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15947.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Replace deprecated protobuf APIs
> 
>
> Key: HDFS-15947
> URL: https://issues.apache.org/jira/browse/HDFS-15947
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Some protobuf APIs are soon going to get deprecated and must be replaced with 
> newer ones. One of the warnings are reported due to the issue is as follows -
> {code}
> [ 48%] Building CXX object 
> main/native/libhdfspp/tests/CMakeFiles/rpc_engine_test.dir/rpc_engine_test.cc.o
> /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc:
>  In function ‘std::pair std::__cxx11::basic_string > RpcResponse(const 
> hadoop::common::RpcResponseHeaderProto&, const string&, const 
> boost::system::error_code&)’:
> /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc:92:56:
>  warning: ‘int google::protobuf::MessageLite::ByteSize() const’ is 
> deprecated: Please use ByteSizeLong() instead [-Wdeprecated-declarations]
>92 |   pbio::CodedOutputStream::VarintSize32(h.ByteSize()) +
>   |^
> In file included from 
> /usr/local/include/google/protobuf/generated_enum_util.h:36,
>  from /usr/local/include/google/protobuf/map.h:49,
>  from 
> /usr/local/include/google/protobuf/generated_message_table_driven.h:34,
>  from 
> /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/tests/test.pb.h:26,
>  from 
> /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc:22:
> /usr/local/include/google/protobuf/message_lite.h:408:7: note: declared here
>   408 |   int ByteSize() const { return internal::ToIntSize(ByteSizeLong()); }
>   |   ^~~~
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15909) Make fnmatch cross platform

2021-04-06 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15909.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make fnmatch cross platform
> ---
>
> Key: HDFS-15909
> URL: https://issues.apache.org/jira/browse/HDFS-15909
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: libhdfs++
>Affects Versions: 3.2.2
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The function fnmatch isn't available in Visual C++. Need to make this cross 
> platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15955) Make explicit_bzero cross platform

2021-04-08 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15955.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make explicit_bzero cross platform
> --
>
> Key: HDFS-15955
> URL: https://issues.apache.org/jira/browse/HDFS-15955
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The function explicit_bzero isn't available in Visual C++. Need to make this 
> cross platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-15423) RBF: WebHDFS create shouldn't choose DN from all sub-clusters

2021-04-12 Thread Jira


 [ 
https://issues.apache.org/jira/browse/HDFS-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-15423.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: WebHDFS create shouldn't choose DN from all sub-clusters
> -
>
> Key: HDFS-15423
> URL: https://issues.apache.org/jira/browse/HDFS-15423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: rbf, webhdfs
>Reporter: Chao Sun
>Assignee: Fengnan Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> In {{RouterWebHdfsMethods}} and for a {{CREATE}} call, {{chooseDatanode}} 
> first gets all DNs via {{getDatanodeReport}}, and then randomly pick one from 
> the list via {{getRandomDatanode}}. This logic doesn't seem correct as it 
> should pick a DN for the specific cluster(s) of the input {{path}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



  1   2   3   4   5   6   7   8   9   10   >