Thanks for the comments Nilotpal, I have created a Jira to track commands with missing JSON output, and added one Jira to it already:
https://issues.apache.org/jira/browse/HDDS-6637 If you, or anyone, comes across a command that is missing the JSON option, please add a Jira to that epic and we endeavour to resolve it. For existing tests that are not in Apache, I would recommend starting the process to move them to use JSON, as this will be an ongoing effort. I know we had one internal test failing related to `ozone admin container info` - that command already has JSON output available, and hence it can be modified already to use it. If you have others that cannot be modified (due to missing JSON), please file Jiras against the above epic and we will take care of it. Thanks, Stephen. On Fri, Apr 22, 2022 at 1:00 PM Nilotpal Nandi <nilotpalna...@apache.org> wrote: > Hi Stephen , > I am +1 on having both human readable output and json output (with an > extra argument) for CLI commands. > Please make sure, for ALL the CLI commands/sub-commands , json outputs are > present before actually making changes in human readable CLI outputs. > Otherwise, tests would still rely on human readable CLI outputs and it > would require regular maintenance in tests. > Also, whenever there is any modification done in json formats, we need to > ensure backward compatibility of json outputs (i.e, key names should not be > changed). > I would request to create a task JIRA to track these efforts. > Once these tasks are completed, internal tests will be modified > accordingly to parse json outputs only and human readable outputs would not > be parsed anymore. > > Thanks, > Nilotpal Nandi > > On 2022/04/21 11:23:13 Stephen O'Donnell wrote: > > Thanks for the comments Pifta. > > > > I guess my suggestion "for JSON the existing field names and structures > > should remain the same", was a little vague. I agree that with JSON, > having: > > > > { > > f1: v1 > > f2: v2 > > } > > > > Is semantically the same whether f1 or f2 comes first. The point I was > > trying to make is that we should not move the nesting of values, eg going > > to something like this, would not be compatible: > > > > { > > wrapper: { > > f1: v1 > > f2: v2 > > } > > } > > > > I also agree we should document all compatibility areas - and also define > > what happens to something which is deprecated. Eg should it be removed > > after some number of releases, or remain forever? However I feel it is > > better to break down the discussion into small areas (like this one on > just > > the CLI command), as otherwise the discussion could go off in many > > directions for some time. Better we reach agreement on small pieces and > > build up the full picture I think. > > > > On Thu, Apr 21, 2022 at 12:13 PM István Fajth <fapi...@gmail.com> wrote: > > > > > Hi Stephen, > > > > > > thank you for bringing this up, I strongly agree with you, we should > get a > > > document about compatibility for sure. > > > On the CLI part, I also agree with all the 4 points in general, though > let > > > me add a few things: > > > For #1. JSON as it is defined at: https://www.json.org/json-en.html > does > > > not require the location to be the same within an object to mean the > same. > > > ("An object is an unordered set of name/value pairs."), though change > of > > > the order can be considered as a different output for arrays. ("An > array is > > > an ordered collection of values."). With that I think if we provide a > real > > > correct JSON output, then object members order should/may change at any > > > point in time. Also adding new fields in objects should not take care > of > > > any ordering. In testing or processing the output, every usages of the > CLI > > > JSON output should parse the full JSON and should not rely on any > custom > > > parsing or regex matching, unless that matching is not sensitive to > > > ordering of object fields... > > > For #2-4 I think it is clear, and I would not add much more here... To > > > reflect on the suggestion from Attila, I think we should also have > tests > > > that are checking the textual output of CLI commands with all the > possible > > > options, so that if there is a change for any reason, we would have a > > > change in that test, so that it would be easier to document what and > how > > > has been changed in the CLI, if we ever want to add such a separate > section > > > into the changelog. > > > > > > I also agree on having documented rules about compatibility and how we > > > maintain it, not just on the CLI level but as a whole thing covering > > > protocol, API, CLI, REST and any other publicly available access > points, > > > and for all internal or external communication interfaces that Ozone > has. > > > CLI is a good start, but we should not just stop there. > > > > > > Pifta > > > > > > Stephen O'Donnell <sodonn...@cloudera.com.invalid> ezt írta (időpont: > > > 2022. > > > ápr. 20., Sze, 11:00): > > > > > > > Discussing compatibility with some team members, we realised that we > > > don't > > > > have anything written down about compatibility guarantees in general. > > > There > > > > are many areas in compatibility, but for this discussion, I would > like to > > > > focus on CLI command output. > > > > > > > > Ozone is still relatively immature, and as such we often find a need > to > > > add > > > > to or modify the CLI command output to provide more information to > help > > > > users of the system, or as features are added and enhanced. > > > > > > > > There are two ways the CLI commands can produce output: > > > > > > > > 1. Human readable format. > > > > 2. JSON format. > > > > > > > > The default output is "Human Readable", with JSON enabled via the > > > "--json" > > > > flag. Not all commands support JSON at the moment, but most do, and > we > > > > should endeavour to ensure they all do, and fill the gaps ASAP. I am > also > > > > mainly focused on admin type commands (eg ozone admin container info > etc) > > > > rather than "key listing" commands, similar to the "ls" output we > know > > > from > > > > filesystems and hadoop. However I believe "ozone sh key list" etc > already > > > > provide their output in JSON format by default. > > > > > > > > I would like the community to agree on a set of compatibility > principals > > > > for CLI command output, and as a starting point for discussion, I > would > > > > like to propose the following: > > > > > > > > 1. We should ensure that information is never removed from command > > > output, > > > > unless it is deprecated and removed gracefully after some number of > > > > releases. For JSON the existing field names and location in the JSON > > > > structure should remain the same. > > > > > > > > 2. Information can be added to command output, and the output is > still > > > > considered compatible as all existing information is still present. > > > > > > > > 3. Following from (2), this means the format of the "Human Readable" > > > output > > > > may change if, for example, some new field or information is added > on an > > > > existing line, or new lines of output are added. > > > > > > > > 4. Due to (3), the "Human Readable" output is not intended to be > consumed > > > > programmatically. We recommend only JSON be consumed > programmatically, as > > > > existing structures will remain even if new information is added. > > > > > > > > Please let me know your thoughts on the above proposals, or any other > > > > suggestions in the scope of "CLI Output Compatibility" and I will > bring > > > the > > > > suggestions together into something we can publish in the docs when > we > > > > reach agreement. > > > > > > > > Thanks, > > > > > > > > Stephen. > > > > > > > > > > > > > -- > > > Pifta > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org > For additional commands, e-mail: dev-h...@ozone.apache.org > >
