... thanks Manoj for reviewing the proposal. The additional features you mentioned can be of course added... as soon as I have a moment I'll have a stab at this and will provide an example in a unit test that demonstrates it.
Cheers On Thu, Jan 30, 2025 at 7:36 PM Manoj Mohanan <mano...@apache.org> wrote: > > Hi Aleks, > > Thank you for your initiative in proposing this system enhancement. I > strongly support this change, as it addresses a critical need in our > current architecture. > > The existing synchronous command handler facilitates the maker-checker > workflow. It allowed direct storage of serialized commands (including > request payload JSON) in the database, enabling easy retrieval and replay > during checker approval. As we transition to the new approach, Are there > proposed modifications to the maker-checker workflow itself? > > Additionally, the existing implementation includes a permission check > (authorization) prior to invoking the handler, as well as a Hook Event > Processor integrated into the control flow to invoke external APIs > asynchronously. To ensure continuity of these functionalities, can we > include these in the proposed command executors? > > This will help maintain critical auditability standards while adopting > the updated architecture. > > Regards, > Manoj > > On 2025/01/26 09:38:46 Aleksandar Vidakovic wrote: > > On Sun, Jan 26, 2025 at 6:59 AM VICTOR MANUEL ROMERO RODRIGUEZ < > > victor.rom...@fintecheando.mx> wrote: > > > > > Hello, > > > > > > I have been reading the discussion, the FISP and the GitHub PR... this > > > must receive more feedback from the community. > > > > > > > That's why we are here... thanks again for participating. > > > > > > > > > > It causes me some noise that the FSIP is focused on the API Rest layer > and > > > the PR is introducing a new header that could also be used for > > > Idempotency... We can understand the reasons for having it in the Rest > > > layer. But what about the idempotency that is being used in the > batch/event > > > processing? > > > > > > > Sorry for the noise... > > I'm not really seeing the "focus" on the REST API layer... the proposal > is > > kept explicitly free of any assumption which web stack you are using... > > this stuff could even work with Webflux. > > Introducing a new header? There is not even an implementation for > > idempotency in this proposal... maybe you are referring to the example in > > the unit test? That stuff (everything in the "sample" package) is purely > > for demonstration. > > Could you maybe elaborate what problem you see concerning the batch > > processing? > > > > > > > > > > I have found the implementation for sync and async commands... What > about > > > streaming? Also in the same package there are pipeline, executor, > router .. > > > What about notifications? My questions are for getting feedback from > you if > > > they are expected/discarded on this new proposed infrastructure. > > > > > > > ... I would add to your list "what about non-blocking"? Well, the reason > > why it's not in my proposal is, because right now we have a classic > > synchronous blocking execution path in place upstream... and the first > > step would be to get the whole thing gradually to a more performant > > solution with clearly structured internal API. Another important thought > > here is to avoid/reduce any additional learning curve; the community is > > used to the classic "I call a function and get a result" paradigm of > > programming; introducing more complex paradigms would need more time for > > adoption. That being said: if the community decided tomorrow to go all in > > for non-blocking, reactive, GRPC, [place your preference here] and > > bulldozer things then I would happily participate... but my feeling is > that > > this is very unlikely. And before anything like this happens we would > need > > all these hard-wired references to JSON data structures from the business > > logic services anyway. > > > > As for notifications: noted, good point. > > And concerning feedback: I have no claim here that what I wrote in the PR > > is set in stone... that's why we discuss it here (I hope more people > join). > > Things that I didn't have on the radar and that you and others find > > indispensable can be of course added. > > > > > > > > > > I think that Apache Camel is a good tool... I don't see it as a vendor > > > lock-in... well If do that (to see it as a vendor lock-in)... then what > > > about spring boot itself? i.e. nowadays we have Quarkus and > Micronaut... > > > and I think it is more complex to move some Apache Fineract code to > these > > > frameworks or add plugins developed in these frameworks to Apache > Fineract > > > runtime. > > > > > > > Again, the idea here is to keep this proposal as self contained as > possible > > to avoid having to decide on too many fundamental changes... hence, you > see > > no Apache Camel. That being said: I can write you an adapter in 10min > that > > runs the whole thing over Camel... but doesn't leak any details about > Camel > > being used to the rest of the code base. Don't see what the flaw would be > > here to properly abstract implementation details? > > > > Is there a ready to use solution in Spring Boot for command processing > > and/or CQRS? I think not... but let me know if I am missing something > here. > > What is out there is Axon which is a complete framework implemented ON > TOP > > OF Spring/Boot... but that thing would require us to jump on their > internal > > APIs and how they think things should be processed... personally I think > > these guys thought about this subject a bit longer and have a way more > > complete solution... but again, given the need for backwards > compatibility > > and the requirement to be not (too) disruptive to any other upstream > > development: very unlikely that such a fundamental change would happen > and > > cause a major refactoring fest. > > > > I'm not so sure why Quarkus and Micronaut are mentioned here... yes, very > > nice and capable modern frameworks... but they also have no specific > notion > > of command processing, but are generic frameworks like Spring/Boot... my > > intention is not to change any fundamental underlying frameworks (give > and > > take they have similar features)... again, I'd like to keep this as small > > and self contained as possible. > > > > > > > > > > Happy to read your feedback. > > > > > > Regards > > > > > > Victor > > > > > > > > > > > > > > > El jue, 23 ene 2025 a las 23:40, James Dailey (<jdai...@apache.org>) > > > escribió: > > > > > >> On Thu, Jan 23, 2025 at 9:20 AM Aleksandar Vidakovic > > >> <chee...@monkeysintown.com> wrote: > > >> > > > >> > ... thanks James for the input... I'll try to answer your last > couple > > >> of questions from my perspective (read: opinionated... take with a > pinch of > > >> salt): > > >> > > >> JD: Aleks - thank you. I always learn something from this back and > > >> forth with you. > > >> > > > >> > too clever: the current implementation I suggest that anyone tries > to > > >> draw a sequence diagram that explains the flow of execution and make > it fit > > >> on one page vs the new proposal will most likely contain less than a > > >> handful of lines. You can apply the same if you take lines of code as > a > > >> metric... overall the new proposal has less than 50 lines of code > that are > > >> relevant (I don't know the number for upstream, but I think it's safe > to > > >> say it's more). If we assume that we can achieve the same results > with less > > >> code then I think the answer is easy here > > >> > maintainability: well, see above... the current solution is not > > >> documented at all and I am pretty sure I am not alone when I saw "I > really > > >> can't explain all the steps" (doesn't mean they are not necessary); > what I > > >> want to say is that the existing solution would really need a lot more > > >> explanation than just "CQRS", I think that would be a fair > requirement. > > >> Admittedly, the new proposal also has no documentation (other than > the wiki > > >> page and what I wrote in this message). But: I think if I did write > it it > > >> can fit on one page (with diagrams), this module (it's a real one) has > > >> (almost) no external dependencies (other than the frameworks that we > use > > >> anyway), it makes no assumption about any of the business logic that > might > > >> or might be passing through (existing implementation fails already > there... > > >> see CommandWrapper and the various entity IDs that are buried > there... this > > >> wrapper class should not be aware of anything it transports)... which > > >> brings me back to the point of less code which is I think from a > > >> maintenance point of view preferrable > > >> > > >> JD: When I say "overly clever" that is in contrast to simplicity > > >> through elegant design. A favorite quote "There are two ways of > > >> constructing a software design: One way is to make it so simple that > > >> there are obviously no deficiencies, and the other way is to make it > > >> so complicated that there are no obvious deficiencies. The first > > >> method is far more difficult." I think if you are aiming for > > >> something simple enough to have obviously no (or much fewer) > > >> deficiencies, that is, an improvement. But, could you write some > > >> documentation about the concept? It should be simple to describe "on > > >> paper", yes? > > >> > > >> > > > >> > Apache Camel: ... disclaimer, I really like that framework and used > it > > >> on a ton of occasions. That being said: choosing a framework is a > > >> commitment pretty much like a vendor lock-in. Depending on how you > > >> integrate a framework like Camel (this will be more than a JAR file > and you > > >> can either hide the fact you use Camel from the rest of your app or > you > > >> fully expose it...) upstream means if for some reason it turns out > that > > >> Camel is not a good choice or the community doesn't want yet another > > >> dependency then we might find ourselves in a refactoring fest to > revert > > >> things. If you look closely in the proposed sources you will see that > first > > >> of all there are Java interfaces that propose a contract on how to > wire > > >> things together... and there not many... which leaves a lot of room > for > > >> actual implementations (Camel or something else). In fact, 3 or 4 > years ago > > >> I actually created a drop-in replacement for the upstream > > >> SynchronousCommandProcessing service and ran Camel behind the scenes > and > > >> was actually very happey with the outcome. When I did this there were > > >> basically 2 relevant functions that needed to be taken care of. Today > there > > >> is a lot more going on there and I am not so sure if you could just > drop-in > > >> Camel effortlessly with the current incarnation of the command > processing > > >> service. > > >> > > >> JD: Ok. I can buy not wanting another dependency, but only if our > > >> level of effort is relatively small ongoing. Otherwise we are taking > > >> on code maintenance for our "own thing" when a perfectly suited > > >> solution is in the same software foundation. > > >> > > >> > Asking Apache Camel's community for opinion: well, can't hurt... > they > > >> do stuff like this literally every day, so I am pretty sure whatever > we'll > > >> exchange with them will be very informative. But that doesn't relieve > us > > >> from deciding if you want to go all in on Apache Camel it would be > anyway a > > >> good practice to abstract these implementation details away (aka hide > to > > >> the rest of Fineract that you are using Camel). If that is the case > then we > > >> need a contract (aka Java interface). The one that is there won't do > it > > >> anymore... without major rework... and that is the point. The proposal > > >> intends to ensure a gradual non disruptive migration (not open heart > > >> surgery) > > >> > > >> JD: Sure, that makes sense, you need to new Java interface... but > > >> wouldn't it be better to spend a bit of time in design and validation > > >> at this early stage. I think we're talking about a pretty significant > > >> optimization from its location in the stack. Who should reach out? > > >> > > >> > whitepapers, alternatives: I think the first thing that Google or > > >> ChatGPT searches will tell you is "use an existing CQRS framework"... > and > > >> this will most likely show you AxonIQ (a CQRS framework implemented > with > > >> Spring/Boot)... but that is then even more of a vendor lock in than > using a > > >> more generic solution like Camel... Axon will force us to use their > > >> contracts (internal APIs, Java interfaces etc.), in short: refactor > fest, > > >> disruptive. There are other low level "solutions" (like LMAX > Disruptor) > > >> that are somewhat in the vicinity of this type of application, but > require > > >> work, to my knowledge there is nothing out there we could just > magically > > >> drop and use without any refactoring. Disclaimer: in one of the 3 > drop-in > > >> implementations of the proposed command processing I am actually > using LMAX > > >> Disruptor... its implementation details just don't leak into the rest > of > > >> the system > > >> > Spring Boot 3 compliance: yes (buzzword drop: "auto-configuration") > > >> > > >> JD: Excellent > > >> > > >> > cutting edge: not sure how to read this here... is this meant as a > > >> requirement or as an argument against the adoption of the proposal as > in > > >> "too experimental"... as I've written the code I am obviously biased > so I > > >> leave that to the community to decide and come up with improvements > and/or > > >> alternatives/arguments if someone doesn't agree > > >> > > >> JD: Yep. The ambiguity is on purpose - cutting edge can be great in > > >> getting results, or it can make you bleed. > > >> > > >> > > > >> > Let me know if I skipped something, made an error or was not clear > > >> enough. > > >> > > >> JD: Very clear. Now, before this code is committed, I would also > > >> like to be sure we have a sensible way of documenting the progress so > > >> that if we are doing a release, we make note of how much of the code > > >> base is using the new methods. I also think we should discuss this in > > >> context of the next release. (coming up soon). > > >> > > >> > > > >> > On Thu, Jan 23, 2025 at 5:00 PM <jdai...@apache.org> wrote: > > >> >> > > >> >> Thanks for bringing this to the list. It looks to be a very low > level > > >> (in the stack) and therefore, highly impactful. I was there when the > > >> decision was made to adopt this pattern and > > >> SynchronousCommandProcessingService as a flexible improvement to the > > >> existing CQRS. I remember asking some questions, but this was and is, > > >> beyond my direct experience. > > >> >> > > >> >> > > >> > https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5:+New+command+processing+infrastructure > > >> >> > > >> >> What I do know is that we should be deliberate with this process, > and > > >> I appreciate your write up on wiki. Definitely other architects here > > >> should take a look. > > >> >> > > >> >> At times over the past decisions - it feels to me that we try to be > > >> "too clever", and this creates a problem with maintainability. I'd > like to > > >> make sure we understand the alternatives as we dig into this. You > raised > > >> Apache Camel as an option - would it be worth it to ask someone over > in > > >> that project to comment on this? Is there some whitepaper or > comparison > > >> out there between the alternatives available? Is this consistent with > > >> Spring Boot 3 ? Is this on the cutting edge? > > >> > > > > > >
