Hi All, With this change, will I be able to perform a dynamic attach if I opt in using the command line switch in JDK 25 (which I believe is the next LTS).
Kind regards, Kirk > On May 18, 2023, at 9:31 AM, Ron Pressler <ron.press...@oracle.com> wrote: > > First, you will need to accept that no one is as invested in the user > experience of the Java ecosystem as a whole as much as OpenJDK’s maintainers, > and no one is as incentivised to work toward its continued success. If you > don’t believe it, you will still need to accept it axiomatically if you wish > to participate in constructive discussions on the OpenJtrDK mailing lists. > > Now, nothing can impact Java’s direction more than user reports about > problems they’re experiencing. We then talk to them try to understand the > core of the problem and try to fix it at the right place. What you’ve done is > not reported a problem but said that you know of others that may experience > problems for unspecified reasons. That is not what we mean by user reports, > and I hope you can see how unhelpful it is from the perspective of product > management. If we don’t know why people do things, there is simply no way for > us to address the issues they experience. Similarly, when asked for reports > about common uses for dynamically attached agents (that cannot be loaded at > startup), saying “there are people who need to dynamically attach an agent” > is also unhelpful. We are obviously aware that the need is not zero, and such > a statement doesn’t help us in any way asses the frequency or severity of the > issue. > > Given that people who cannot control the command line and yet wish to deploy > the most sophisticated Java applications on the newest releases are in for a > world of pain regardless of this JEP, you can help us help them by getting > them to talk to us and explain their experience. We very much want to help > them, but we cannot solve a problem if we don’t know what it is. > > — Ron > >> On 18 May 2023, at 16:49, Jack Shirazi <ja...@fasterj.com> wrote: >> >> You asked for reports of common uses of dynamically loaded agents for >> serviceability and difficulties setting a flag. I have provided several. You >> have ignored most of the examples, and categorized the remainder as >> "incorrect uses of Java" (by which I assume you mean incorrect uses of >> JVMs), as if you are the judge of what is correct and incorrect rather than >> business priorities. >> >> As a sponsor of the JEP you have a built-in incentive to simply ignore and >> dismiss objections, and I'm seeing that applied here. That's not really >> acceptable. I'm sure you have the technical ability to proceed to force >> through an unpopular JEP while ignoring objections. That seems unwise, but >> that's up to you >> >> >> On 18/05/2023 16:27, Ron Pressler wrote: >>> It is simply untenable to accommodate many incorrect uses of Java on top of >>> the correct ones, and, for example, allowing the dynamic loading of agents >>> but disallowing loading them at startup is an incorrect use of Java. The >>> solution to people using Java incorrectly is to educate them — which this >>> JEP will hopefully help do — not to punish those who use Java correctly. >>> >>> Similarly, given the number of command line options that are added in each >>> release, those who feel they have a need to choose what subset of them they >>> allow, must be committed to review and amend their allowed options for each >>> release. >>> >>> Those who actually do the kind of things mentioned in the two previous >>> paragraphs are invited to discuss them with us. But while I understand that >>> you are personally particularly interested in agents, the people I >>> mentioned in the previous two paragraphs will have been affected by other >>> JEPs more than by this one, so I think this JEP is too small and too niche >>> to discuss the importance that the command line has had for Java >>> applications in the last five years. Maybe when the informational JEP is >>> accepted it could serve as a basis for discussion/user education. >>> >>> — Ron >>> >>>> On 18 May 2023, at 15:30, Jack Shirazi <ja...@fasterj.com> wrote: >>>> >>>> That's a strange analysis, two out of the six examples of well known real >>>> world applications, systems and platforms has that claim. And these are >>>> well known. I'm not sure where most of your comments come from, but let me >>>> assume this isn't just a blanket dismissal because you're the JEP sponsor >>>> and so incentivized to do that. >>>> >>>> - All the scenarios are well known real world ones. If you don't know >>>> about them, ask the community. >>>> >>>> - The business processing platforms, as I said "can only configure the >>>> application and any JVM parameters that have been explicitly exported by >>>> the 3rd party" do indeed allow the heap size to adjusted, because they >>>> explicitly make that parameter configurable - in the application >>>> properties. That's pretty common. Most startup scripts for most JVM >>>> applications out there choose a set of JVM parameters that are >>>> independently configurable via properties/variables/options without >>>> explictly setting the JVM command-line. That's the most common way to >>>> start JVMs! I'm not making the claim that all these can't change the >>>> command line, just explaining how some specific chosen parameters can be >>>> set but not the whole command line, as you don't seem to be aware of that >>>> very common practice. So yes, this is not only viable but common. The >>>> applications/systems/platforms that restrict access to changing the >>>> startup scripts are a subset. They do so for their own reasons. >>>> >>>> - I fail to see how a new release matters here. The business model is what >>>> drives this, not your assumptions. >>>> >>>> - You seem to have mis-categorized all the scenarios as being of one type, >>>> which they most certainly are not. >>>> >>>> >>>> On 16/05/2023 18:22, Ron Pressler wrote: >>>>> At the core of your arguments is the claim — that we’ve heard told >>>>> second-hand but rarely if ever reported first-hand — that the inability >>>>> to control the command line is common. This claim is very important >>>>> because its implications go well beyond the relatively niche issue of >>>>> dynamically loaded agents, so I think it merits further discussion4. >>>>> Certainly since the discontinuation of the centralised JRE deployment >>>>> model in JDK 11, it’s been a deep assumption of Java’s design that >>>>> deploying a Java application requires control of the command line. If you >>>>> cannot control the command line, there are things you simply cannot do, >>>>> including much more basic things than loading agents. If your application >>>>> needs APM, surely it also needs control over the heap and GC with >>>>> potentially new options. >>>>> >>>>> Those who restrict access to the command line will need to explain their >>>>> constraints because that approach is already not viable. Deploying a Java >>>>> application, certainly one that supports super-advanced uses such as the >>>>> dynamic loading of agents for code manipulation, without control of the >>>>> command line is not a model that the platform has supported for a while. >>>>> Java’s deployment model changed some years ago, and policies that applied >>>>> to the retired model do not apply for the new one and have to change >>>>> (again, nothing to do with agents). >>>>> >>>>> Since we’re talking about changes that only take place in new releases >>>>> anyway, I find the notion that adding a command line flag is harder than >>>>> adopting a new runtime version to be somewhat suspect, but if there’s a >>>>> good reason for that, someone will need to present it. >>>>> >>>>> A service vendor that wishes to allow a Java program to use a new runtime >>>>> and to create a child process that injects a native library into the >>>>> parent process will need to explain why they cannot also allow that >>>>> program to set command line flags. If you want to support JDK 11 and >>>>> upward to enjoy new features and performance enhancements, you need to >>>>> also support the changes to the deployment model that accompany these new >>>>> developments. >>>>> >>>>> — Ron >>>>> >>>>>> On 16 May 2023, at 13:36, Jack Shirazi <ja...@fasterj.com> wrote: >>>>>> >>>>>> Here I refer to anyone who is operating the JVM as a customer. Scenarios >>>>>> where this JEP will (when moved from deprecation to fully applied) >>>>>> onerously adversely impact customers include: >>>>>> >>>>>> 1. Most obviously, those customers who have running JVMs, decide they >>>>>> want to add an observability agent, but are unaware of the >>>>>> DisableAttachMechanism JVM option - most operators are unaware of the >>>>>> majority of JVM options. These customers would only be able to add >>>>>> observability agents after >>>>>> 1a. investigating why the attach is failing >>>>>> 1b. resetting their command-line >>>>>> 1c. waiting until they can restart >>>>>> These customers would normally be able to immediately gain APM data, and >>>>>> troubleshoot an existing running JVM when it has a problem, by attaching >>>>>> an agent. The requirement to reset the command-line would mean that >>>>>> future invocations would be observable and can use an agent for >>>>>> troubleshooting, but often the choice to apply a troubleshooting agent >>>>>> is their first encounter with agent technology - in other words the >>>>>> first time they need it, they can't use it. It's hardly ideal to insist >>>>>> that only sophisticated or experienced customers can use agent >>>>>> troubleshooting technologies >>>>>> >>>>>> 2. Many customers have a long process between proposing command-line >>>>>> changes and allowing them to be applied in production (including changes >>>>>> via environment variables). Allowing an agent to be attached provides >>>>>> immediate capability until that process is complete. The same long >>>>>> process means there is no prospect of switching on remote agent >>>>>> attachment without that same process being followed. These customers >>>>>> currently accept agent attachment before the changes have been passed as >>>>>> the technology is proven, robust and typically they use an agent from a >>>>>> vendor that guarantees (and with support contract) that it is valid to >>>>>> attach the agent to a production JVM >>>>>> >>>>>> 3. There are types of customers who run 3rd party JVM applications that >>>>>> they configure - but they can only configure the application and any JVM >>>>>> parameters that have been explicitly exported by the 3rd party. (This is >>>>>> very common for bought-in business processing applications). Changes to >>>>>> the JVM command-line are rarely allowed in this scenario (it violates >>>>>> the contract). Typically if the customer wants a change they make an >>>>>> enhancement request and even where these are prioritized, they often >>>>>> have to pay more for the changes so they are reluctant to make requests >>>>>> that are not directly business enhancing. Currently these types of >>>>>> customers can attach APM agents with no issue, providing excellent >>>>>> observability in to the request flow in the system to identify problems. >>>>>> These customers will be disadvantaged >>>>>> >>>>>> 4. In complete opposition to the whole thrust of the JEP, I've seen >>>>>> customers who will not allow agents attached via the command-line but >>>>>> will allow them if started by a Java library. Go figure. It's not >>>>>> logical but that's our industry for you. This is the case for several >>>>>> managed platforms (where the developer does not have direct access to >>>>>> JVM environment variables and arguments), and also where the person in >>>>>> charge of the deployment delegates the responsibility of agents to the >>>>>> dev team, who need to do this programmatically. Perhaps these >>>>>> customers/platforms will accept adding the flipped option to the >>>>>> command-line, perhaps not or perhaps only after finding that they now >>>>>> have an agent attachment issue - in any case it imposes additional >>>>>> process to something that works well at the moment >>>>>> >>>>>> 5. Injecting agents to containers running JVMs is a minefield. >>>>>> Attachment via a script after startup is often easier. Changing the >>>>>> command-line involves setting the JAVA_TOOL_OPTIONS environment variable >>>>>> before starting the container, but if the container already uses that >>>>>> variable then the conflict usually causes loss of one or the other >>>>>> setting, so it doesn't work in that case. >>>>>> >>>>>> 6. For k8s, using JAVA_TOOL_OPTIONS is currently the preferred mechanism >>>>>> (eg via mutating webhooks) and works well so this JEP shouldn't matter. >>>>>> But there are cases where that doesn't work (eg per container conflicts >>>>>> as above; or where security roles restrict this; etc) and in those the >>>>>> only alternative is to attach to the pod and attach the agent . This >>>>>> would require rebuilding images (either with the >>>>>> -XX:-DisableAttachMechanism option or with -javaagent/agentlib). Of >>>>>> course the observability community will attempt to pre-empt the problem >>>>>> by telling everyone to build their images with >>>>>> -XX:-DisableAttachMechanism - but that already points to the JEP being >>>>>> an anti-pattern >>>>>> >>>>>> >>>>>> On 12/05/2023 19:28, Ron Pressler wrote: >>>>>>> (Moving to the appropriate mailing lists for the discussion of this JEP) >>>>>>> >>>>>>> We want reports of common uses of dynamically loaded agents for >>>>>>> serviceability and difficulties setting a flag. Our judgment will sway >>>>>>> if we learn that the use of dynamically loaded agents for >>>>>>> serviceability is very common and that setting a command line flag is >>>>>>> onerous. Such reports of “I use dynamically loaded agents for X and >>>>>>> it’s hard for me to set a flag because Y” should be made here, i.e. >>>>>>> jigsaw-...@openjdk.org, serviceability-dev@openjdk.org. >>>>>>> >>>>>>> Saying “I don’t like this (because I can think of cases where it may >>>>>>> inconvenience me a little)” is not a report of a problem. A JDK feature >>>>>>> that is disliked by only 1% of users will still be disliked by tens of >>>>>>> thousands of people, and pretty much every JDK feature or lack of a >>>>>>> feature is disliked by some Java developers; some features even >>>>>>> inconvenience some minority of users. By physical necessity we >>>>>>> sometimes inconvenience some users because users have contradictory >>>>>>> requirements. What we’re trying to estimate is just *how much* of an >>>>>>> inconvenience will be caused by feature X or the lack of X when >>>>>>> integrated over the entire ecosystem. >>>>>>> >>>>>>> — Ron >>>>>>> >>>>>>>> On 12 May 2023, at 12:37, Jack Shirazi <ja...@fasterj.com> wrote: >>>>>>>> >>>>>>>> Thanks, this is going in circles. You want reports, I'm fine with >>>>>>>> that, I will provide a report. But my one report is not going to be >>>>>>>> sufficient to move your judgement. So I'll ask once again where should >>>>>>>> further such reports go, and at what point does your judgement sway? >>>>>>>> >>>>>>>> >>>>>>>> On 12/05/2023 16:46, Ron Pressler wrote: >>>>>>>>> Let’s start with you describing the particular use-cases of >>>>>>>>> dynamically loaded agents that you’re concerned about and why you >>>>>>>>> think a command-line flag to enable the functionality is onerous. In >>>>>>>>> other words, describe the nature and severity of a *problem*. >>>>>>>>> Remember that the goal of JDK maintainers is to serve the ecosystem >>>>>>>>> as a whole, which means accommodating the conflicting desires by >>>>>>>>> different classes of users. Because different people’s requirements >>>>>>>>> are sometimes in contradiction with one another, we need to make a >>>>>>>>> judgment. As JEP 451 says, this judgment is based on the assumptions >>>>>>>>> that: 1. The need for dynamically loaded agent is not very common, >>>>>>>>> and 2. When needed, adding a flag is not onerous. >>>>>>>>> >>>>>>>>> Stating you don’t like a policy that’s been discussed for roughly a >>>>>>>>> decade and started to be put into effect five years ago is not >>>>>>>>> enough. However, if you have questions regarding the informational >>>>>>>>> JEP that attempts to summarise past discussions >>>>>>>>> (https://openjdk.org/jeps/8305968) I’ll gladly try and answer them. >>>>>>>>> >>>>>>>>> — Ron >>>>>>>>> >>>>>>>>>> On 12 May 2023, at 10:05, Jack Shirazi <ja...@fasterj.com> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Integrity must be opt out, and cannot be opt in, and so opting in >>>>>>>>>>> is not a solution that will give us integrity*by default*. >>>>>>>>>>> Seehttps://openjdk.org/jeps/8305968#Strong-Encapsulation-by-Default >>>>>>>>>> This is an opinion, not a statement of fact. It needs to be >>>>>>>>>> justified, not assumed. Integrity is a goal, and there is a balance >>>>>>>>>> between what is useful and what can be limited. For full integrity, >>>>>>>>>> don't use the JVM at all. I for one prefer to continue using it. >>>>>>>>>> >>>>>>>>>>> The only information of relevance would be reports showing that >>>>>>>>>>> dynamically loading agents are a commonly-needed functionality and >>>>>>>>>>> that adding a command-line option to allow it is onerous. >>>>>>>>>> I'm fine with that. I'm reporting exactly that here. I encourage >>>>>>>>>> others interested in this to also report that. I'll mention it in my >>>>>>>>>> next newsletter - where do you want the reports sent? My readers >>>>>>>>>> won't want to signup to this email list just to send a comment. At >>>>>>>>>> what point does the reporting mean the JEP is dropped? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 12/05/2023 14:44, Ron Pressler wrote: >>>>>>>>>>>> On 12 May 2023, at 05:26, Jack Shirazi <ja...@fasterj.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Thank you for your reply. This makes it clear that the JEP has a >>>>>>>>>>>> single specific tradeoff. So we have two capabilities at issue here >>>>>>>>>>>> >>>>>>>>>>>> A) Currently libraries can turn themselves into agents >>>>>>>>>>>> >>>>>>>>>>>> B) Currently agents can remotely attach >>>>>>>>>>>> >>>>>>>>>>>> The JEP has decided for the community that each of these are a bad >>>>>>>>>>>> thing and should be disabled by default (though enableable by >>>>>>>>>>>> setting an option). >>>>>>>>>>> No, the JEP says: >>>>>>>>>>> >>>>>>>>>>> "To assure integrity, we need stronger measures to prevent the >>>>>>>>>>> misuse by libraries of dynamically loaded agents. Unfortunately, we >>>>>>>>>>> have not found a simple and automatic way to distinguish between a >>>>>>>>>>> serviceability tool that dynamically loads an agent and a library >>>>>>>>>>> that dynamically loads an agent.” >>>>>>>>>>> >>>>>>>>>>> The only problem is libraries, but because there’s no simple way to >>>>>>>>>>> distinguish between the two, and because dynamically loaded agents >>>>>>>>>>> are not needed in most serviceability uses, disabling them by >>>>>>>>>>> default is reasonable. BTW, this was already decided in 2017 in JEP >>>>>>>>>>> 261: https://openjdk.org/jeps/261 >>>>>>>>>>> >>>>>>>>>>> As the JEP also says, in the future we may be able to distinguish >>>>>>>>>>> between tools and libraries via a more complex mechanism that could >>>>>>>>>>> allow tools to load agents dynamically without the flag. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> My involvement in community discussions over the years has been >>>>>>>>>>>> that no one complains about (A), it has not been used maliciously, >>>>>>>>>>>> and there is a small niche who use it. (B) is used quite a lot and >>>>>>>>>>>> enhances JVM serviceability with a capability that is a clear >>>>>>>>>>>> advantage over other runtimes. It seems a shame to eliminate that >>>>>>>>>>>> competitive advantage. >>>>>>>>>>> Malicious use is not a concern *at all*. What this JEP addresses is >>>>>>>>>>> integrity by default. See https://openjdk.org/jeps/8305968 >>>>>>>>>>> >>>>>>>>>>>> The JEP clearly points out that anyone concerned by these can >>>>>>>>>>>> disable the ability with a simple command-line option, so there is >>>>>>>>>>>> a simple solution for this minority. >>>>>>>>>>> Integrity must be opt out, and cannot be opt in, and so opting in >>>>>>>>>>> is not a solution that will give us integrity *by default*. See >>>>>>>>>>> https://openjdk.org/jeps/8305968#Strong-Encapsulation-by-Default >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> The fundamental error is really that the attaching agent is >>>>>>>>>>>> read-write rather than read-only. If we could change that, it >>>>>>>>>>>> would be ideal, but sadly I don't think that's easily doable. >>>>>>>>>>> Perhaps, but most uses of dynamically loaded agents (and nearly all >>>>>>>>>>> uses of dynamically loaded *Java* agents) are for “write.” The most >>>>>>>>>>> common use-case for “read-only” is dynamically attached advanced >>>>>>>>>>> profilers that use JVM TI. The solution there, as the JEP says, is >>>>>>>>>>> not to separate agent capabilities but to improve JFR’s >>>>>>>>>>> capabilities — which do not require an agent at all — and JFR can >>>>>>>>>>> obtain profiles far more efficiently than anything JVM TI could >>>>>>>>>>> ever hope to achieve. >>>>>>>>>>> >>>>>>>>>>>> I and many in the monitoring community believe this JEP is NOT an >>>>>>>>>>>> enhancement to the JDK. The proposers believe it is. Is there a >>>>>>>>>>>> mechanism other than this email discussion list to gain wider >>>>>>>>>>>> community feedback so we can ascertain if there is really a strong >>>>>>>>>>>> community preference either way? >>>>>>>>>>>> >>>>>>>>>>> The only information of relevance would be reports showing that >>>>>>>>>>> dynamically loading agents are a commonly-needed functionality and >>>>>>>>>>> that adding a command-line option to allow it is onerous. >>>>>>>>>>> >>>>>>>>>>> — Ron >
