Alexander - Do you want to continue this discussion and get to a vote? https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5%3A+New+command+processing+infrastructure
Really appreciate the conversation thus far. It seems it faces no opposition (clarification yes), but I don't know if there's a real appetite to fully implement across the entire code base... so I guess that's my concern, that once started it gets completed. Not a reason to drop it but something to consider when doing implementation. Thanks, On Mon, Mar 10, 2025 at 6:28 AM Aleksandar Vidakovic < chee...@monkeysintown.com> wrote: > Hi Terence, > > ... first: thanks for your encouraging and insightful comments. > > The current status of this proposal: > > - I've added the capability to save incoming write requests (aka > commands) to the database (brings us a bit closer to the initial idea of > CQRS replayable commands to restore the system at any given point in time); > there is some error during the build process that I still have to track > down > - Manoj made some comments around maker-checker functionality in the > current production version of Fineract. I didn't reply extensively to that, > but my view is that command processing and maker-checker are two different > problem domains that should not be mixed together. I'd like to address the > maker-checker concern in a different proposal that would (maybe) help us on > multiple fronts: > - maker-checker is more related to human tasks/interactions in a > workflow... maybe (!) we can address this in a way that we can later > optionally (!) drop in a proper workflow engine... > - having maker-checker as a separate component should have the same > benefits as command processing (type safety, clear readable execution > path, > extensible/replaceable, better documentation) > - maker-checker is tightly related to our security model; an > improved implementation should help also with a long standing effort to > modularize the security of Fineract; we have a classic RBAC model baked > into Fineract that can't really be replaced without major refactoring; > even > adding roles/permissions to adjust to the deployment environment is not > really easy... let alone using some more advanced concepts like Open > Policy > Agent or OpenFGA > - again, this would be subject to a separate proposal... don't want > to overload > > The first services that will be migrated to the new command processing are > free of maker-checker requirements and are NOT critical ones (like loan, > general ledger etc.). I think that way we won't create a monster pull > request and validate the two improved concepts (command processing, > maker-checker) step by step. > > Cheers, > > Aleks > > On Mon, Mar 10, 2025 at 11:14 AM Terence Monteiro < > terence.plac...@gmail.com> wrote: > >> Hi Aleks and fellow Fineract Devs, >> >> Just wanted to bump this thread and share some thoughts.. >> 1. I'm certain of the benefit such accurate typing and class usage for >> data interchange will bring. From my experience with both strongly typed >> (C++, Java, Python) and weakly typed languages (C, Perl5, PHP) the >> flexibility of the latter and not using classes becomes a challenge when >> the project scales up to around 10000+ lines of code. I think Fineract is >> at the stage of its evolution where this kind of work will bring a sea >> change in ease of ongoing Development and auto-generated Documentation. >> 2. I like the bullet point summary of changes expressed in the >> fineract-command module README from the PR. From an architecture >> standpoint, I see a far reaching impact such a change can have. I feel the >> questions raised after the initial mail by Aleks have been satisfactorily >> answered so I'd like to know whether we have progressed beyond the PR >> shared in the initial mail. >> 3. Practically having worked with developers of varying degrees of >> experience and IDE (Eclipse, Intellij-Idea, VS Code) adoption and >> utilization, this will yield, if done properly, much more accurate >> auto-generated Class names for function signatures and accurate >> implementation stubs. Circling back to the earlier point about the stage we >> are at with Fineract having all these lines of code, I feel this FSIP can >> lay the groundwork for a lot of contributions to >> a). Clean, refactor current code >> b). introduce new features >> c). Inject more quality API consuming projects like Web app and mobile >> app from Mifos, as well as the dozens of open-source and >> proprietary consumer apps >> >> Given this promise and opening of possibilities, I strongly support this >> proposal. I'd be glad to assist in any way possible working you Aleks on >> fleshing out required parts of making this happen as long as I can get some >> more native Java eyes to review (I think in algorithms preferably through >> an OOP lens, but don't always think in Java unless deep into a Java >> project). >> >> Regards, >> Terence Monteiro. >> >> >> On Fri, Jan 31, 2025 at 12:21 AM Aleksandar Vidakovic < >> chee...@monkeysintown.com> wrote: >> >>> ... thanks Manoj for reviewing the proposal. The additional features you >>> mentioned can be of course added... as soon as I have a moment I'll have a >>> stab at this and will provide an example in a unit test that demonstrates >>> it. >>> >>> Cheers >>> >>> On Thu, Jan 30, 2025 at 7:36 PM Manoj Mohanan <mano...@apache.org> >>> wrote: >>> >>>> >>>> Hi Aleks, >>>> >>>> Thank you for your initiative in proposing this system enhancement. I >>>> strongly support this change, as it addresses a critical need in our >>>> current architecture. >>>> >>>> The existing synchronous command handler facilitates the maker-checker >>>> workflow. It allowed direct storage of serialized commands (including >>>> request payload JSON) in the database, enabling easy retrieval and replay >>>> during checker approval. As we transition to the new approach, Are there >>>> proposed modifications to the maker-checker workflow itself? >>>> >>>> Additionally, the existing implementation includes a permission check >>>> (authorization) prior to invoking the handler, as well as a Hook Event >>>> Processor integrated into the control flow to invoke external APIs >>>> asynchronously. To ensure continuity of these functionalities, can we >>>> include these in the proposed command executors? >>>> >>>> This will help maintain critical auditability standards while adopting >>>> the updated architecture. >>>> >>>> Regards, >>>> Manoj >>>> >>>> On 2025/01/26 09:38:46 Aleksandar Vidakovic wrote: >>>> > On Sun, Jan 26, 2025 at 6:59 AM VICTOR MANUEL ROMERO RODRIGUEZ < >>>> > victor.rom...@fintecheando.mx> wrote: >>>> > >>>> > > Hello, >>>> > > >>>> > > I have been reading the discussion, the FISP and the GitHub PR... >>>> this >>>> > > must receive more feedback from the community. >>>> > > >>>> > >>>> > That's why we are here... thanks again for participating. >>>> > >>>> > >>>> > > >>>> > > It causes me some noise that the FSIP is focused on the API Rest >>>> layer and >>>> > > the PR is introducing a new header that could also be used for >>>> > > Idempotency... We can understand the reasons for having it in the >>>> Rest >>>> > > layer. But what about the idempotency that is being used in the >>>> batch/event >>>> > > processing? >>>> > > >>>> > >>>> > Sorry for the noise... >>>> > I'm not really seeing the "focus" on the REST API layer... the >>>> proposal is >>>> > kept explicitly free of any assumption which web stack you are >>>> using... >>>> > this stuff could even work with Webflux. >>>> > Introducing a new header? There is not even an implementation for >>>> > idempotency in this proposal... maybe you are referring to the >>>> example in >>>> > the unit test? That stuff (everything in the "sample" package) is >>>> purely >>>> > for demonstration. >>>> > Could you maybe elaborate what problem you see concerning the batch >>>> > processing? >>>> > >>>> > >>>> > > >>>> > > I have found the implementation for sync and async commands... What >>>> about >>>> > > streaming? Also in the same package there are pipeline, executor, >>>> router .. >>>> > > What about notifications? My questions are for getting feedback >>>> from you if >>>> > > they are expected/discarded on this new proposed infrastructure. >>>> > > >>>> > >>>> > ... I would add to your list "what about non-blocking"? Well, the >>>> reason >>>> > why it's not in my proposal is, because right now we have a classic >>>> > synchronous blocking execution path in place upstream... and the >>>> first >>>> > step would be to get the whole thing gradually to a more performant >>>> > solution with clearly structured internal API. Another important >>>> thought >>>> > here is to avoid/reduce any additional learning curve; the community >>>> is >>>> > used to the classic "I call a function and get a result" paradigm of >>>> > programming; introducing more complex paradigms would need more time >>>> for >>>> > adoption. That being said: if the community decided tomorrow to go >>>> all in >>>> > for non-blocking, reactive, GRPC, [place your preference here] and >>>> > bulldozer things then I would happily participate... but my feeling >>>> is that >>>> > this is very unlikely. And before anything like this happens we would >>>> need >>>> > all these hard-wired references to JSON data structures from the >>>> business >>>> > logic services anyway. >>>> > >>>> > As for notifications: noted, good point. >>>> > And concerning feedback: I have no claim here that what I wrote in >>>> the PR >>>> > is set in stone... that's why we discuss it here (I hope more people >>>> join). >>>> > Things that I didn't have on the radar and that you and others find >>>> > indispensable can be of course added. >>>> > >>>> > >>>> > > >>>> > > I think that Apache Camel is a good tool... I don't see it as a >>>> vendor >>>> > > lock-in... well If do that (to see it as a vendor lock-in)... then >>>> what >>>> > > about spring boot itself? i.e. nowadays we have Quarkus and >>>> Micronaut... >>>> > > and I think it is more complex to move some Apache Fineract code to >>>> these >>>> > > frameworks or add plugins developed in these frameworks to Apache >>>> Fineract >>>> > > runtime. >>>> > > >>>> > >>>> > Again, the idea here is to keep this proposal as self contained as >>>> possible >>>> > to avoid having to decide on too many fundamental changes... hence, >>>> you see >>>> > no Apache Camel. That being said: I can write you an adapter in 10min >>>> that >>>> > runs the whole thing over Camel... but doesn't leak any details about >>>> Camel >>>> > being used to the rest of the code base. Don't see what the flaw >>>> would be >>>> > here to properly abstract implementation details? >>>> > >>>> > Is there a ready to use solution in Spring Boot for command processing >>>> > and/or CQRS? I think not... but let me know if I am missing something >>>> here. >>>> > What is out there is Axon which is a complete framework implemented >>>> ON TOP >>>> > OF Spring/Boot... but that thing would require us to jump on their >>>> internal >>>> > APIs and how they think things should be processed... personally I >>>> think >>>> > these guys thought about this subject a bit longer and have a way more >>>> > complete solution... but again, given the need for backwards >>>> compatibility >>>> > and the requirement to be not (too) disruptive to any other upstream >>>> > development: very unlikely that such a fundamental change would >>>> happen and >>>> > cause a major refactoring fest. >>>> > >>>> > I'm not so sure why Quarkus and Micronaut are mentioned here... yes, >>>> very >>>> > nice and capable modern frameworks... but they also have no specific >>>> notion >>>> > of command processing, but are generic frameworks like Spring/Boot... >>>> my >>>> > intention is not to change any fundamental underlying frameworks >>>> (give and >>>> > take they have similar features)... again, I'd like to keep this as >>>> small >>>> > and self contained as possible. >>>> > >>>> > >>>> > > >>>> > > Happy to read your feedback. >>>> > > >>>> > > Regards >>>> > > >>>> > > Victor >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > El jue, 23 ene 2025 a las 23:40, James Dailey (<jdai...@apache.org >>>> >) >>>> > > escribió: >>>> > > >>>> > >> On Thu, Jan 23, 2025 at 9:20 AM Aleksandar Vidakovic >>>> > >> <chee...@monkeysintown.com> wrote: >>>> > >> > >>>> > >> > ... thanks James for the input... I'll try to answer your last >>>> couple >>>> > >> of questions from my perspective (read: opinionated... take with a >>>> pinch of >>>> > >> salt): >>>> > >> >>>> > >> JD: Aleks - thank you. I always learn something from this back >>>> and >>>> > >> forth with you. >>>> > >> > >>>> > >> > too clever: the current implementation I suggest that anyone >>>> tries to >>>> > >> draw a sequence diagram that explains the flow of execution and >>>> make it fit >>>> > >> on one page vs the new proposal will most likely contain less than >>>> a >>>> > >> handful of lines. You can apply the same if you take lines of code >>>> as a >>>> > >> metric... overall the new proposal has less than 50 lines of code >>>> that are >>>> > >> relevant (I don't know the number for upstream, but I think it's >>>> safe to >>>> > >> say it's more). If we assume that we can achieve the same results >>>> with less >>>> > >> code then I think the answer is easy here >>>> > >> > maintainability: well, see above... the current solution is not >>>> > >> documented at all and I am pretty sure I am not alone when I saw >>>> "I really >>>> > >> can't explain all the steps" (doesn't mean they are not >>>> necessary); what I >>>> > >> want to say is that the existing solution would really need a lot >>>> more >>>> > >> explanation than just "CQRS", I think that would be a fair >>>> requirement. >>>> > >> Admittedly, the new proposal also has no documentation (other than >>>> the wiki >>>> > >> page and what I wrote in this message). But: I think if I did >>>> write it it >>>> > >> can fit on one page (with diagrams), this module (it's a real one) >>>> has >>>> > >> (almost) no external dependencies (other than the frameworks that >>>> we use >>>> > >> anyway), it makes no assumption about any of the business logic >>>> that might >>>> > >> or might be passing through (existing implementation fails already >>>> there... >>>> > >> see CommandWrapper and the various entity IDs that are buried >>>> there... this >>>> > >> wrapper class should not be aware of anything it transports)... >>>> which >>>> > >> brings me back to the point of less code which is I think from a >>>> > >> maintenance point of view preferrable >>>> > >> >>>> > >> JD: When I say "overly clever" that is in contrast to simplicity >>>> > >> through elegant design. A favorite quote "There are two ways of >>>> > >> constructing a software design: One way is to make it so simple >>>> that >>>> > >> there are obviously no deficiencies, and the other way is to make >>>> it >>>> > >> so complicated that there are no obvious deficiencies. The first >>>> > >> method is far more difficult." I think if you are aiming for >>>> > >> something simple enough to have obviously no (or much fewer) >>>> > >> deficiencies, that is, an improvement. But, could you write some >>>> > >> documentation about the concept? It should be simple to describe >>>> "on >>>> > >> paper", yes? >>>> > >> >>>> > >> > >>>> > >> > Apache Camel: ... disclaimer, I really like that framework and >>>> used it >>>> > >> on a ton of occasions. That being said: choosing a framework is a >>>> > >> commitment pretty much like a vendor lock-in. Depending on how you >>>> > >> integrate a framework like Camel (this will be more than a JAR >>>> file and you >>>> > >> can either hide the fact you use Camel from the rest of your app >>>> or you >>>> > >> fully expose it...) upstream means if for some reason it turns out >>>> that >>>> > >> Camel is not a good choice or the community doesn't want yet >>>> another >>>> > >> dependency then we might find ourselves in a refactoring fest to >>>> revert >>>> > >> things. If you look closely in the proposed sources you will see >>>> that first >>>> > >> of all there are Java interfaces that propose a contract on how to >>>> wire >>>> > >> things together... and there not many... which leaves a lot of >>>> room for >>>> > >> actual implementations (Camel or something else). In fact, 3 or 4 >>>> years ago >>>> > >> I actually created a drop-in replacement for the upstream >>>> > >> SynchronousCommandProcessing service and ran Camel behind the >>>> scenes and >>>> > >> was actually very happey with the outcome. When I did this there >>>> were >>>> > >> basically 2 relevant functions that needed to be taken care of. >>>> Today there >>>> > >> is a lot more going on there and I am not so sure if you could >>>> just drop-in >>>> > >> Camel effortlessly with the current incarnation of the command >>>> processing >>>> > >> service. >>>> > >> >>>> > >> JD: Ok. I can buy not wanting another dependency, but only if our >>>> > >> level of effort is relatively small ongoing. Otherwise we are >>>> taking >>>> > >> on code maintenance for our "own thing" when a perfectly suited >>>> > >> solution is in the same software foundation. >>>> > >> >>>> > >> > Asking Apache Camel's community for opinion: well, can't hurt... >>>> they >>>> > >> do stuff like this literally every day, so I am pretty sure >>>> whatever we'll >>>> > >> exchange with them will be very informative. But that doesn't >>>> relieve us >>>> > >> from deciding if you want to go all in on Apache Camel it would be >>>> anyway a >>>> > >> good practice to abstract these implementation details away (aka >>>> hide to >>>> > >> the rest of Fineract that you are using Camel). If that is the >>>> case then we >>>> > >> need a contract (aka Java interface). The one that is there won't >>>> do it >>>> > >> anymore... without major rework... and that is the point. The >>>> proposal >>>> > >> intends to ensure a gradual non disruptive migration (not open >>>> heart >>>> > >> surgery) >>>> > >> >>>> > >> JD: Sure, that makes sense, you need to new Java interface... but >>>> > >> wouldn't it be better to spend a bit of time in design and >>>> validation >>>> > >> at this early stage. I think we're talking about a pretty >>>> significant >>>> > >> optimization from its location in the stack. Who should reach out? >>>> > >> >>>> > >> > whitepapers, alternatives: I think the first thing that Google or >>>> > >> ChatGPT searches will tell you is "use an existing CQRS >>>> framework"... and >>>> > >> this will most likely show you AxonIQ (a CQRS framework >>>> implemented with >>>> > >> Spring/Boot)... but that is then even more of a vendor lock in >>>> than using a >>>> > >> more generic solution like Camel... Axon will force us to use their >>>> > >> contracts (internal APIs, Java interfaces etc.), in short: >>>> refactor fest, >>>> > >> disruptive. There are other low level "solutions" (like LMAX >>>> Disruptor) >>>> > >> that are somewhat in the vicinity of this type of application, but >>>> require >>>> > >> work, to my knowledge there is nothing out there we could just >>>> magically >>>> > >> drop and use without any refactoring. Disclaimer: in one of the 3 >>>> drop-in >>>> > >> implementations of the proposed command processing I am actually >>>> using LMAX >>>> > >> Disruptor... its implementation details just don't leak into the >>>> rest of >>>> > >> the system >>>> > >> > Spring Boot 3 compliance: yes (buzzword drop: >>>> "auto-configuration") >>>> > >> >>>> > >> JD: Excellent >>>> > >> >>>> > >> > cutting edge: not sure how to read this here... is this meant as >>>> a >>>> > >> requirement or as an argument against the adoption of the proposal >>>> as in >>>> > >> "too experimental"... as I've written the code I am obviously >>>> biased so I >>>> > >> leave that to the community to decide and come up with >>>> improvements and/or >>>> > >> alternatives/arguments if someone doesn't agree >>>> > >> >>>> > >> JD: Yep. The ambiguity is on purpose - cutting edge can be great >>>> in >>>> > >> getting results, or it can make you bleed. >>>> > >> >>>> > >> > >>>> > >> > Let me know if I skipped something, made an error or was not >>>> clear >>>> > >> enough. >>>> > >> >>>> > >> JD: Very clear. Now, before this code is committed, I would also >>>> > >> like to be sure we have a sensible way of documenting the progress >>>> so >>>> > >> that if we are doing a release, we make note of how much of the >>>> code >>>> > >> base is using the new methods. I also think we should discuss this >>>> in >>>> > >> context of the next release. (coming up soon). >>>> > >> >>>> > >> > >>>> > >> > On Thu, Jan 23, 2025 at 5:00 PM <jdai...@apache.org> wrote: >>>> > >> >> >>>> > >> >> Thanks for bringing this to the list. It looks to be a very low >>>> level >>>> > >> (in the stack) and therefore, highly impactful. I was there when >>>> the >>>> > >> decision was made to adopt this pattern and >>>> > >> SynchronousCommandProcessingService as a flexible improvement to >>>> the >>>> > >> existing CQRS. I remember asking some questions, but this was and >>>> is, >>>> > >> beyond my direct experience. >>>> > >> >> >>>> > >> >> >>>> > >> >>>> https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5:+New+command+processing+infrastructure >>>> > >> >> >>>> > >> >> What I do know is that we should be deliberate with this >>>> process, and >>>> > >> I appreciate your write up on wiki. Definitely other architects >>>> here >>>> > >> should take a look. >>>> > >> >> >>>> > >> >> At times over the past decisions - it feels to me that we try >>>> to be >>>> > >> "too clever", and this creates a problem with maintainability. >>>> I'd like to >>>> > >> make sure we understand the alternatives as we dig into this. You >>>> raised >>>> > >> Apache Camel as an option - would it be worth it to ask someone >>>> over in >>>> > >> that project to comment on this? Is there some whitepaper or >>>> comparison >>>> > >> out there between the alternatives available? Is this consistent >>>> with >>>> > >> Spring Boot 3 ? Is this on the cutting edge? >>>> > >> >>>> > > >>>> > >>>> >>>