Hi Aleks and fellow Fineract Devs, Just wanted to bump this thread and share some thoughts.. 1. I'm certain of the benefit such accurate typing and class usage for data interchange will bring. From my experience with both strongly typed (C++, Java, Python) and weakly typed languages (C, Perl5, PHP) the flexibility of the latter and not using classes becomes a challenge when the project scales up to around 10000+ lines of code. I think Fineract is at the stage of its evolution where this kind of work will bring a sea change in ease of ongoing Development and auto-generated Documentation. 2. I like the bullet point summary of changes expressed in the fineract-command module README from the PR. From an architecture standpoint, I see a far reaching impact such a change can have. I feel the questions raised after the initial mail by Aleks have been satisfactorily answered so I'd like to know whether we have progressed beyond the PR shared in the initial mail. 3. Practically having worked with developers of varying degrees of experience and IDE (Eclipse, Intellij-Idea, VS Code) adoption and utilization, this will yield, if done properly, much more accurate auto-generated Class names for function signatures and accurate implementation stubs. Circling back to the earlier point about the stage we are at with Fineract having all these lines of code, I feel this FSIP can lay the groundwork for a lot of contributions to a). Clean, refactor current code b). introduce new features c). Inject more quality API consuming projects like Web app and mobile app from Mifos, as well as the dozens of open-source and proprietary consumer apps
Given this promise and opening of possibilities, I strongly support this proposal. I'd be glad to assist in any way possible working you Aleks on fleshing out required parts of making this happen as long as I can get some more native Java eyes to review (I think in algorithms preferably through an OOP lens, but don't always think in Java unless deep into a Java project). Regards, Terence Monteiro. On Fri, Jan 31, 2025 at 12:21 AM Aleksandar Vidakovic < chee...@monkeysintown.com> wrote: > ... thanks Manoj for reviewing the proposal. The additional features you > mentioned can be of course added... as soon as I have a moment I'll have a > stab at this and will provide an example in a unit test that demonstrates > it. > > Cheers > > On Thu, Jan 30, 2025 at 7:36 PM Manoj Mohanan <mano...@apache.org> wrote: > >> >> Hi Aleks, >> >> Thank you for your initiative in proposing this system enhancement. I >> strongly support this change, as it addresses a critical need in our >> current architecture. >> >> The existing synchronous command handler facilitates the maker-checker >> workflow. It allowed direct storage of serialized commands (including >> request payload JSON) in the database, enabling easy retrieval and replay >> during checker approval. As we transition to the new approach, Are there >> proposed modifications to the maker-checker workflow itself? >> >> Additionally, the existing implementation includes a permission check >> (authorization) prior to invoking the handler, as well as a Hook Event >> Processor integrated into the control flow to invoke external APIs >> asynchronously. To ensure continuity of these functionalities, can we >> include these in the proposed command executors? >> >> This will help maintain critical auditability standards while adopting >> the updated architecture. >> >> Regards, >> Manoj >> >> On 2025/01/26 09:38:46 Aleksandar Vidakovic wrote: >> > On Sun, Jan 26, 2025 at 6:59 AM VICTOR MANUEL ROMERO RODRIGUEZ < >> > victor.rom...@fintecheando.mx> wrote: >> > >> > > Hello, >> > > >> > > I have been reading the discussion, the FISP and the GitHub PR... this >> > > must receive more feedback from the community. >> > > >> > >> > That's why we are here... thanks again for participating. >> > >> > >> > > >> > > It causes me some noise that the FSIP is focused on the API Rest >> layer and >> > > the PR is introducing a new header that could also be used for >> > > Idempotency... We can understand the reasons for having it in the Rest >> > > layer. But what about the idempotency that is being used in the >> batch/event >> > > processing? >> > > >> > >> > Sorry for the noise... >> > I'm not really seeing the "focus" on the REST API layer... the proposal >> is >> > kept explicitly free of any assumption which web stack you are using... >> > this stuff could even work with Webflux. >> > Introducing a new header? There is not even an implementation for >> > idempotency in this proposal... maybe you are referring to the example >> in >> > the unit test? That stuff (everything in the "sample" package) is purely >> > for demonstration. >> > Could you maybe elaborate what problem you see concerning the batch >> > processing? >> > >> > >> > > >> > > I have found the implementation for sync and async commands... What >> about >> > > streaming? Also in the same package there are pipeline, executor, >> router .. >> > > What about notifications? My questions are for getting feedback from >> you if >> > > they are expected/discarded on this new proposed infrastructure. >> > > >> > >> > ... I would add to your list "what about non-blocking"? Well, the reason >> > why it's not in my proposal is, because right now we have a classic >> > synchronous blocking execution path in place upstream... and the first >> > step would be to get the whole thing gradually to a more performant >> > solution with clearly structured internal API. Another important thought >> > here is to avoid/reduce any additional learning curve; the community is >> > used to the classic "I call a function and get a result" paradigm of >> > programming; introducing more complex paradigms would need more time for >> > adoption. That being said: if the community decided tomorrow to go all >> in >> > for non-blocking, reactive, GRPC, [place your preference here] and >> > bulldozer things then I would happily participate... but my feeling is >> that >> > this is very unlikely. And before anything like this happens we would >> need >> > all these hard-wired references to JSON data structures from the >> business >> > logic services anyway. >> > >> > As for notifications: noted, good point. >> > And concerning feedback: I have no claim here that what I wrote in the >> PR >> > is set in stone... that's why we discuss it here (I hope more people >> join). >> > Things that I didn't have on the radar and that you and others find >> > indispensable can be of course added. >> > >> > >> > > >> > > I think that Apache Camel is a good tool... I don't see it as a vendor >> > > lock-in... well If do that (to see it as a vendor lock-in)... then >> what >> > > about spring boot itself? i.e. nowadays we have Quarkus and >> Micronaut... >> > > and I think it is more complex to move some Apache Fineract code to >> these >> > > frameworks or add plugins developed in these frameworks to Apache >> Fineract >> > > runtime. >> > > >> > >> > Again, the idea here is to keep this proposal as self contained as >> possible >> > to avoid having to decide on too many fundamental changes... hence, you >> see >> > no Apache Camel. That being said: I can write you an adapter in 10min >> that >> > runs the whole thing over Camel... but doesn't leak any details about >> Camel >> > being used to the rest of the code base. Don't see what the flaw would >> be >> > here to properly abstract implementation details? >> > >> > Is there a ready to use solution in Spring Boot for command processing >> > and/or CQRS? I think not... but let me know if I am missing something >> here. >> > What is out there is Axon which is a complete framework implemented ON >> TOP >> > OF Spring/Boot... but that thing would require us to jump on their >> internal >> > APIs and how they think things should be processed... personally I think >> > these guys thought about this subject a bit longer and have a way more >> > complete solution... but again, given the need for backwards >> compatibility >> > and the requirement to be not (too) disruptive to any other upstream >> > development: very unlikely that such a fundamental change would happen >> and >> > cause a major refactoring fest. >> > >> > I'm not so sure why Quarkus and Micronaut are mentioned here... yes, >> very >> > nice and capable modern frameworks... but they also have no specific >> notion >> > of command processing, but are generic frameworks like Spring/Boot... my >> > intention is not to change any fundamental underlying frameworks (give >> and >> > take they have similar features)... again, I'd like to keep this as >> small >> > and self contained as possible. >> > >> > >> > > >> > > Happy to read your feedback. >> > > >> > > Regards >> > > >> > > Victor >> > > >> > > >> > > >> > > >> > > El jue, 23 ene 2025 a las 23:40, James Dailey (<jdai...@apache.org>) >> > > escribió: >> > > >> > >> On Thu, Jan 23, 2025 at 9:20 AM Aleksandar Vidakovic >> > >> <chee...@monkeysintown.com> wrote: >> > >> > >> > >> > ... thanks James for the input... I'll try to answer your last >> couple >> > >> of questions from my perspective (read: opinionated... take with a >> pinch of >> > >> salt): >> > >> >> > >> JD: Aleks - thank you. I always learn something from this back and >> > >> forth with you. >> > >> > >> > >> > too clever: the current implementation I suggest that anyone tries >> to >> > >> draw a sequence diagram that explains the flow of execution and make >> it fit >> > >> on one page vs the new proposal will most likely contain less than a >> > >> handful of lines. You can apply the same if you take lines of code >> as a >> > >> metric... overall the new proposal has less than 50 lines of code >> that are >> > >> relevant (I don't know the number for upstream, but I think it's >> safe to >> > >> say it's more). If we assume that we can achieve the same results >> with less >> > >> code then I think the answer is easy here >> > >> > maintainability: well, see above... the current solution is not >> > >> documented at all and I am pretty sure I am not alone when I saw "I >> really >> > >> can't explain all the steps" (doesn't mean they are not necessary); >> what I >> > >> want to say is that the existing solution would really need a lot >> more >> > >> explanation than just "CQRS", I think that would be a fair >> requirement. >> > >> Admittedly, the new proposal also has no documentation (other than >> the wiki >> > >> page and what I wrote in this message). But: I think if I did write >> it it >> > >> can fit on one page (with diagrams), this module (it's a real one) >> has >> > >> (almost) no external dependencies (other than the frameworks that we >> use >> > >> anyway), it makes no assumption about any of the business logic that >> might >> > >> or might be passing through (existing implementation fails already >> there... >> > >> see CommandWrapper and the various entity IDs that are buried >> there... this >> > >> wrapper class should not be aware of anything it transports)... which >> > >> brings me back to the point of less code which is I think from a >> > >> maintenance point of view preferrable >> > >> >> > >> JD: When I say "overly clever" that is in contrast to simplicity >> > >> through elegant design. A favorite quote "There are two ways of >> > >> constructing a software design: One way is to make it so simple that >> > >> there are obviously no deficiencies, and the other way is to make it >> > >> so complicated that there are no obvious deficiencies. The first >> > >> method is far more difficult." I think if you are aiming for >> > >> something simple enough to have obviously no (or much fewer) >> > >> deficiencies, that is, an improvement. But, could you write some >> > >> documentation about the concept? It should be simple to describe "on >> > >> paper", yes? >> > >> >> > >> > >> > >> > Apache Camel: ... disclaimer, I really like that framework and >> used it >> > >> on a ton of occasions. That being said: choosing a framework is a >> > >> commitment pretty much like a vendor lock-in. Depending on how you >> > >> integrate a framework like Camel (this will be more than a JAR file >> and you >> > >> can either hide the fact you use Camel from the rest of your app or >> you >> > >> fully expose it...) upstream means if for some reason it turns out >> that >> > >> Camel is not a good choice or the community doesn't want yet another >> > >> dependency then we might find ourselves in a refactoring fest to >> revert >> > >> things. If you look closely in the proposed sources you will see >> that first >> > >> of all there are Java interfaces that propose a contract on how to >> wire >> > >> things together... and there not many... which leaves a lot of room >> for >> > >> actual implementations (Camel or something else). In fact, 3 or 4 >> years ago >> > >> I actually created a drop-in replacement for the upstream >> > >> SynchronousCommandProcessing service and ran Camel behind the scenes >> and >> > >> was actually very happey with the outcome. When I did this there were >> > >> basically 2 relevant functions that needed to be taken care of. >> Today there >> > >> is a lot more going on there and I am not so sure if you could just >> drop-in >> > >> Camel effortlessly with the current incarnation of the command >> processing >> > >> service. >> > >> >> > >> JD: Ok. I can buy not wanting another dependency, but only if our >> > >> level of effort is relatively small ongoing. Otherwise we are taking >> > >> on code maintenance for our "own thing" when a perfectly suited >> > >> solution is in the same software foundation. >> > >> >> > >> > Asking Apache Camel's community for opinion: well, can't hurt... >> they >> > >> do stuff like this literally every day, so I am pretty sure whatever >> we'll >> > >> exchange with them will be very informative. But that doesn't >> relieve us >> > >> from deciding if you want to go all in on Apache Camel it would be >> anyway a >> > >> good practice to abstract these implementation details away (aka >> hide to >> > >> the rest of Fineract that you are using Camel). If that is the case >> then we >> > >> need a contract (aka Java interface). The one that is there won't do >> it >> > >> anymore... without major rework... and that is the point. The >> proposal >> > >> intends to ensure a gradual non disruptive migration (not open heart >> > >> surgery) >> > >> >> > >> JD: Sure, that makes sense, you need to new Java interface... but >> > >> wouldn't it be better to spend a bit of time in design and validation >> > >> at this early stage. I think we're talking about a pretty >> significant >> > >> optimization from its location in the stack. Who should reach out? >> > >> >> > >> > whitepapers, alternatives: I think the first thing that Google or >> > >> ChatGPT searches will tell you is "use an existing CQRS >> framework"... and >> > >> this will most likely show you AxonIQ (a CQRS framework implemented >> with >> > >> Spring/Boot)... but that is then even more of a vendor lock in than >> using a >> > >> more generic solution like Camel... Axon will force us to use their >> > >> contracts (internal APIs, Java interfaces etc.), in short: refactor >> fest, >> > >> disruptive. There are other low level "solutions" (like LMAX >> Disruptor) >> > >> that are somewhat in the vicinity of this type of application, but >> require >> > >> work, to my knowledge there is nothing out there we could just >> magically >> > >> drop and use without any refactoring. Disclaimer: in one of the 3 >> drop-in >> > >> implementations of the proposed command processing I am actually >> using LMAX >> > >> Disruptor... its implementation details just don't leak into the >> rest of >> > >> the system >> > >> > Spring Boot 3 compliance: yes (buzzword drop: "auto-configuration") >> > >> >> > >> JD: Excellent >> > >> >> > >> > cutting edge: not sure how to read this here... is this meant as a >> > >> requirement or as an argument against the adoption of the proposal >> as in >> > >> "too experimental"... as I've written the code I am obviously biased >> so I >> > >> leave that to the community to decide and come up with improvements >> and/or >> > >> alternatives/arguments if someone doesn't agree >> > >> >> > >> JD: Yep. The ambiguity is on purpose - cutting edge can be great in >> > >> getting results, or it can make you bleed. >> > >> >> > >> > >> > >> > Let me know if I skipped something, made an error or was not clear >> > >> enough. >> > >> >> > >> JD: Very clear. Now, before this code is committed, I would also >> > >> like to be sure we have a sensible way of documenting the progress so >> > >> that if we are doing a release, we make note of how much of the code >> > >> base is using the new methods. I also think we should discuss this in >> > >> context of the next release. (coming up soon). >> > >> >> > >> > >> > >> > On Thu, Jan 23, 2025 at 5:00 PM <jdai...@apache.org> wrote: >> > >> >> >> > >> >> Thanks for bringing this to the list. It looks to be a very low >> level >> > >> (in the stack) and therefore, highly impactful. I was there when the >> > >> decision was made to adopt this pattern and >> > >> SynchronousCommandProcessingService as a flexible improvement to the >> > >> existing CQRS. I remember asking some questions, but this was and is, >> > >> beyond my direct experience. >> > >> >> >> > >> >> >> > >> >> https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5:+New+command+processing+infrastructure >> > >> >> >> > >> >> What I do know is that we should be deliberate with this process, >> and >> > >> I appreciate your write up on wiki. Definitely other architects here >> > >> should take a look. >> > >> >> >> > >> >> At times over the past decisions - it feels to me that we try to >> be >> > >> "too clever", and this creates a problem with maintainability. I'd >> like to >> > >> make sure we understand the alternatives as we dig into this. You >> raised >> > >> Apache Camel as an option - would it be worth it to ask someone over >> in >> > >> that project to comment on this? Is there some whitepaper or >> comparison >> > >> out there between the alternatives available? Is this consistent >> with >> > >> Spring Boot 3 ? Is this on the cutting edge? >> > >> >> > > >> > >> >
