Alexander -  Do you want to continue this discussion and get to a vote?

https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5%3A+New+command+processing+infrastructure


Really appreciate the conversation thus far.

It seems it faces no opposition (clarification yes), but I don't know if
there's a real appetite to fully implement across the entire code base...
so I guess that's my concern, that once started it gets completed.   Not a
reason to drop it but something to consider when doing implementation.

Thanks,



On Mon, Mar 10, 2025 at 6:28 AM Aleksandar Vidakovic <
chee...@monkeysintown.com> wrote:

> Hi Terence,
>
> ... first: thanks for your encouraging and insightful comments.
>
> The current status of this proposal:
>
>    - I've added the capability to save incoming write requests (aka
>    commands) to the database (brings us a bit closer to the initial idea of
>    CQRS replayable commands to restore the system at any given point in time);
>    there is some error during the build process that I still have to track 
> down
>    - Manoj made some comments around maker-checker functionality in the
>    current production version of Fineract. I didn't reply extensively to that,
>    but my view is that command processing and maker-checker are two different
>    problem domains that should not be mixed together. I'd like to address the
>    maker-checker concern in a different proposal that would (maybe) help us on
>    multiple fronts:
>       - maker-checker is more related to human tasks/interactions in a
>       workflow... maybe (!) we can address this in a way that we can later
>       optionally (!) drop in a proper workflow engine...
>       - having maker-checker as a separate component should have the same
>       benefits as command processing (type safety, clear readable execution 
> path,
>       extensible/replaceable, better documentation)
>       - maker-checker is tightly related to our security model; an
>       improved implementation should help also with a long standing effort to
>       modularize the security of Fineract; we have a classic RBAC model baked
>       into Fineract that can't really be replaced without major refactoring; 
> even
>       adding roles/permissions to adjust to the deployment environment is not
>       really easy... let alone using some more advanced concepts like Open 
> Policy
>       Agent or OpenFGA
>       - again, this would be subject to a separate proposal... don't want
>       to overload
>
> The first services that will be migrated to the new command processing are
> free of maker-checker requirements and are NOT critical ones (like loan,
> general ledger etc.). I think that way we won't create a monster pull
> request and validate the two improved concepts (command processing,
> maker-checker) step by step.
>
> Cheers,
>
> Aleks
>
> On Mon, Mar 10, 2025 at 11:14 AM Terence Monteiro <
> terence.plac...@gmail.com> wrote:
>
>> Hi Aleks and fellow Fineract Devs,
>>
>> Just wanted to bump this thread and share some thoughts..
>> 1. I'm certain of the benefit such accurate typing and class usage for
>> data interchange will bring. From my experience with both strongly typed
>> (C++, Java, Python) and weakly typed languages (C, Perl5, PHP) the
>> flexibility of the latter and not using classes becomes a challenge when
>> the project scales up to around 10000+ lines of code. I think Fineract is
>> at the stage of its evolution where this kind of work will bring a sea
>> change in ease of ongoing Development and auto-generated Documentation.
>> 2. I like the bullet point summary of changes expressed in the
>> fineract-command module README from the PR. From an architecture
>> standpoint, I see a far reaching impact such a change can have. I feel the
>> questions raised after the initial mail by Aleks have been satisfactorily
>> answered so I'd like to know whether we have progressed beyond the PR
>> shared in the initial mail.
>> 3. Practically having worked with developers of varying degrees of
>> experience and IDE (Eclipse, Intellij-Idea, VS Code) adoption and
>> utilization, this will yield, if done properly, much more accurate
>> auto-generated Class names for function signatures and accurate
>> implementation stubs. Circling back to the earlier point about the stage we
>> are at with Fineract having all these lines of code, I feel this FSIP can
>> lay the groundwork for a lot of contributions to
>>    a). Clean, refactor current code
>>    b). introduce new features
>>    c). Inject more quality API consuming projects like Web app and mobile
>> app from Mifos, as well as the dozens of open-source and
>> proprietary consumer apps
>>
>> Given this promise and opening of possibilities, I strongly support this
>> proposal. I'd be glad to assist in any way possible working you Aleks on
>> fleshing out required parts of making this happen as long as I can get some
>> more native Java eyes to review (I think in algorithms preferably through
>> an OOP lens, but don't always think in Java unless deep into a Java
>> project).
>>
>> Regards,
>> Terence Monteiro.
>>
>>
>> On Fri, Jan 31, 2025 at 12:21 AM Aleksandar Vidakovic <
>> chee...@monkeysintown.com> wrote:
>>
>>> ... thanks Manoj for reviewing the proposal. The additional features you
>>> mentioned can be of course added... as soon as I have a moment I'll have a
>>> stab at this and will provide an example in a unit test that demonstrates
>>> it.
>>>
>>> Cheers
>>>
>>> On Thu, Jan 30, 2025 at 7:36 PM Manoj Mohanan <mano...@apache.org>
>>> wrote:
>>>
>>>>
>>>> Hi Aleks,
>>>>
>>>> Thank you for your initiative in proposing this system enhancement. I
>>>> strongly support this change, as it addresses a critical need in our
>>>> current architecture.
>>>>
>>>> The existing synchronous command handler facilitates the maker-checker
>>>> workflow. It allowed direct storage of serialized commands (including
>>>> request payload JSON) in the database, enabling easy retrieval and replay
>>>> during checker approval. As we transition to the new approach, Are there
>>>> proposed modifications to the maker-checker workflow itself?
>>>>
>>>> Additionally, the existing implementation includes a permission check
>>>> (authorization) prior to invoking the handler, as well as a Hook Event
>>>> Processor integrated into the control flow to invoke external APIs
>>>> asynchronously. To ensure continuity of these functionalities, can we
>>>> include these in the proposed command executors?
>>>>
>>>> This  will help maintain critical auditability standards while adopting
>>>> the updated architecture.
>>>>
>>>> Regards,
>>>> Manoj
>>>>
>>>> On 2025/01/26 09:38:46 Aleksandar Vidakovic wrote:
>>>> > On Sun, Jan 26, 2025 at 6:59 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>>> > victor.rom...@fintecheando.mx> wrote:
>>>> >
>>>> > > Hello,
>>>> > >
>>>> > > I have been reading the discussion, the FISP and the GitHub PR...
>>>> this
>>>> > > must receive more feedback from the community.
>>>> > >
>>>> >
>>>> > That's why we are here... thanks again for participating.
>>>> >
>>>> >
>>>> > >
>>>> > > It causes me some noise that the FSIP is focused on the API Rest
>>>> layer and
>>>> > > the PR is introducing a new header that could also be used for
>>>> > > Idempotency... We can understand the reasons for having it in the
>>>> Rest
>>>> > > layer. But what about the idempotency that is being used in the
>>>> batch/event
>>>> > > processing?
>>>> > >
>>>> >
>>>> > Sorry for the noise...
>>>> > I'm not really seeing the "focus" on the REST API layer... the
>>>> proposal is
>>>> > kept explicitly free of any assumption which web stack you are
>>>> using...
>>>> > this stuff could even work with Webflux.
>>>> > Introducing a new header? There is not even an implementation for
>>>> > idempotency in this proposal... maybe you are referring to the
>>>> example in
>>>> > the unit test? That stuff (everything in the "sample" package) is
>>>> purely
>>>> > for demonstration.
>>>> > Could you maybe elaborate what problem you see concerning the batch
>>>> > processing?
>>>> >
>>>> >
>>>> > >
>>>> > > I have found the implementation for sync and async commands... What
>>>> about
>>>> > > streaming? Also in the same package there are pipeline, executor,
>>>> router ..
>>>> > > What about notifications? My questions are for getting feedback
>>>> from you if
>>>> > > they are expected/discarded on this new proposed infrastructure.
>>>> > >
>>>> >
>>>> > ... I would add to your list "what about non-blocking"? Well, the
>>>> reason
>>>> > why it's not in my proposal is, because right now we have a classic
>>>> > synchronous blocking execution path in place upstream... and the
>>>> first
>>>> > step would be to get the whole thing gradually to a more performant
>>>> > solution with clearly structured internal API. Another important
>>>> thought
>>>> > here is to avoid/reduce any additional learning curve; the community
>>>> is
>>>> > used to the classic "I call a function and get a result" paradigm of
>>>> > programming; introducing more complex paradigms would need more time
>>>> for
>>>> > adoption. That being said: if the community decided tomorrow to go
>>>> all in
>>>> > for non-blocking, reactive, GRPC, [place your preference here] and
>>>> > bulldozer things then I would happily participate... but my feeling
>>>> is that
>>>> > this is very unlikely. And before anything like this happens we would
>>>> need
>>>> > all these hard-wired references to JSON data structures from the
>>>> business
>>>> > logic services anyway.
>>>> >
>>>> > As for notifications: noted, good point.
>>>> > And concerning feedback: I have no claim here that what I wrote in
>>>> the PR
>>>> > is set in stone... that's why we discuss it here (I hope more people
>>>> join).
>>>> > Things that I didn't have on the radar and that you and others find
>>>> > indispensable can be of course added.
>>>> >
>>>> >
>>>> > >
>>>> > > I think that Apache Camel is a good tool... I don't see it as a
>>>> vendor
>>>> > > lock-in... well If do that (to see it as a vendor lock-in)... then
>>>> what
>>>> > > about spring boot itself? i.e. nowadays we have Quarkus and
>>>> Micronaut...
>>>> > > and I think it is more complex to move some Apache Fineract code to
>>>> these
>>>> > > frameworks or add plugins developed in these frameworks to Apache
>>>> Fineract
>>>> > > runtime.
>>>> > >
>>>> >
>>>> > Again, the idea here is to keep this proposal as self contained as
>>>> possible
>>>> > to avoid having to decide on too many fundamental changes... hence,
>>>> you see
>>>> > no Apache Camel. That being said: I can write you an adapter in 10min
>>>> that
>>>> > runs the whole thing over Camel... but doesn't leak any details about
>>>> Camel
>>>> > being used to the rest of the code base. Don't see what the flaw
>>>> would be
>>>> > here to properly abstract implementation details?
>>>> >
>>>> > Is there a ready to use solution in Spring Boot for command processing
>>>> > and/or CQRS? I think not... but let me know if I am missing something
>>>> here.
>>>> > What is out there is Axon which is a complete framework implemented
>>>> ON TOP
>>>> > OF Spring/Boot... but that thing would require us to jump on their
>>>> internal
>>>> > APIs and how they think things should be processed... personally I
>>>> think
>>>> > these guys thought about this subject a bit longer and have a way more
>>>> > complete solution... but again, given the need for backwards
>>>> compatibility
>>>> > and the requirement to be not (too) disruptive to any other upstream
>>>> > development: very unlikely that such a fundamental change would
>>>> happen and
>>>> > cause a major refactoring fest.
>>>> >
>>>> > I'm not so sure why Quarkus and Micronaut are mentioned here... yes,
>>>> very
>>>> > nice and capable modern frameworks... but they also have no specific
>>>> notion
>>>> > of command processing, but are generic frameworks like Spring/Boot...
>>>> my
>>>> > intention is not to change any fundamental underlying frameworks
>>>> (give and
>>>> > take they have similar features)... again, I'd like to keep this as
>>>> small
>>>> > and self contained as possible.
>>>> >
>>>> >
>>>> > >
>>>> > > Happy to read your feedback.
>>>> > >
>>>> > > Regards
>>>> > >
>>>> > > Victor
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > El jue, 23 ene 2025 a las 23:40, James Dailey (<jdai...@apache.org
>>>> >)
>>>> > > escribió:
>>>> > >
>>>> > >> On Thu, Jan 23, 2025 at 9:20 AM Aleksandar Vidakovic
>>>> > >> <chee...@monkeysintown.com> wrote:
>>>> > >> >
>>>> > >> > ... thanks James for the input... I'll try to answer your last
>>>> couple
>>>> > >> of questions from my perspective (read: opinionated... take with a
>>>> pinch of
>>>> > >> salt):
>>>> > >>
>>>> > >> JD:   Aleks - thank you.  I always learn something from this back
>>>> and
>>>> > >> forth with you.
>>>> > >> >
>>>> > >> > too clever: the current implementation I suggest that anyone
>>>> tries to
>>>> > >> draw a sequence diagram that explains the flow of execution and
>>>> make it fit
>>>> > >> on one page vs the new proposal will most likely contain less than
>>>> a
>>>> > >> handful of lines. You can apply the same if you take lines of code
>>>> as a
>>>> > >> metric... overall the new proposal has less than 50 lines of code
>>>> that are
>>>> > >> relevant (I don't know the number for upstream, but I think it's
>>>> safe to
>>>> > >> say it's more). If we assume that we can achieve the same results
>>>> with less
>>>> > >> code then I think the answer is easy here
>>>> > >> > maintainability: well, see above... the current solution is not
>>>> > >> documented at all and I am pretty sure I am not alone when I saw
>>>> "I really
>>>> > >> can't explain all the steps" (doesn't mean they are not
>>>> necessary); what I
>>>> > >> want to say is that the existing solution would really need a lot
>>>> more
>>>> > >> explanation than just "CQRS", I think that would be a fair
>>>> requirement.
>>>> > >> Admittedly, the new proposal also has no documentation (other than
>>>> the wiki
>>>> > >> page and what I wrote in this message). But: I think if I did
>>>> write it it
>>>> > >> can fit on one page (with diagrams), this module (it's a real one)
>>>> has
>>>> > >> (almost) no external dependencies (other than the frameworks that
>>>> we use
>>>> > >> anyway), it makes no assumption about any of the business logic
>>>> that might
>>>> > >> or might be passing through (existing implementation fails already
>>>> there...
>>>> > >> see CommandWrapper and the various entity IDs that are buried
>>>> there... this
>>>> > >> wrapper class should not be aware of anything it transports)...
>>>> which
>>>> > >> brings me back to the point of less code which is I think from a
>>>> > >> maintenance point of view preferrable
>>>> > >>
>>>> > >> JD:  When I say "overly clever" that is in contrast to simplicity
>>>> > >> through elegant design.  A favorite quote "There are two ways of
>>>> > >> constructing a software design: One way is to make it so simple
>>>> that
>>>> > >> there are obviously no deficiencies, and the other way is to make
>>>> it
>>>> > >> so complicated that there are no obvious deficiencies. The first
>>>> > >> method is far more difficult."    I think if you are aiming for
>>>> > >> something simple enough to have obviously no (or much fewer)
>>>> > >> deficiencies, that is, an improvement.  But, could you write some
>>>> > >> documentation about the concept? It should be simple to describe
>>>> "on
>>>> > >> paper", yes?
>>>> > >>
>>>> > >> >
>>>> > >> > Apache Camel: ... disclaimer, I really like that framework and
>>>> used it
>>>> > >> on a ton of occasions. That being said: choosing a framework is a
>>>> > >> commitment pretty much like a vendor lock-in. Depending on how you
>>>> > >> integrate a framework like Camel (this will be more than a JAR
>>>> file and you
>>>> > >> can either hide the fact you use Camel from the rest of your app
>>>> or you
>>>> > >> fully expose it...) upstream means if for some reason it turns out
>>>> that
>>>> > >> Camel is not a good choice or the community doesn't want yet
>>>> another
>>>> > >> dependency then we might find ourselves in a refactoring fest to
>>>> revert
>>>> > >> things. If you look closely in the proposed sources you will see
>>>> that first
>>>> > >> of all there are Java interfaces that propose a contract on how to
>>>> wire
>>>> > >> things together... and there not many... which leaves a lot of
>>>> room for
>>>> > >> actual implementations (Camel or something else). In fact, 3 or 4
>>>> years ago
>>>> > >> I actually created a drop-in replacement for the upstream
>>>> > >> SynchronousCommandProcessing service and ran Camel behind the
>>>> scenes and
>>>> > >> was actually very happey with the outcome. When I did this there
>>>> were
>>>> > >> basically 2 relevant functions that needed to be taken care of.
>>>> Today there
>>>> > >> is a lot more going on there and I am not so sure if you could
>>>> just drop-in
>>>> > >> Camel effortlessly with the current incarnation of the command
>>>> processing
>>>> > >> service.
>>>> > >>
>>>> > >> JD: Ok. I can buy not wanting another dependency, but only if our
>>>> > >> level of effort is relatively small ongoing. Otherwise we are
>>>> taking
>>>> > >> on code maintenance for our "own thing" when a perfectly suited
>>>> > >> solution is in the same software foundation.
>>>> > >>
>>>> > >> > Asking Apache Camel's community for opinion: well, can't hurt...
>>>> they
>>>> > >> do stuff like this literally every day, so I am pretty sure
>>>> whatever we'll
>>>> > >> exchange with them will be very informative. But that doesn't
>>>> relieve us
>>>> > >> from deciding if you want to go all in on Apache Camel it would be
>>>> anyway a
>>>> > >> good practice to abstract these implementation details away (aka
>>>> hide to
>>>> > >> the rest of Fineract that you are using Camel). If that is the
>>>> case then we
>>>> > >> need a contract (aka Java interface). The one that is there won't
>>>> do it
>>>> > >> anymore... without major rework... and that is the point. The
>>>> proposal
>>>> > >> intends to ensure a gradual non disruptive migration (not open
>>>> heart
>>>> > >> surgery)
>>>> > >>
>>>> > >> JD: Sure, that makes sense, you need to new Java interface...  but
>>>> > >> wouldn't it be better to spend a bit of time in design and
>>>> validation
>>>> > >> at this early stage.  I think we're talking about a pretty
>>>> significant
>>>> > >> optimization from its location in the stack.  Who should reach out?
>>>> > >>
>>>> > >> > whitepapers, alternatives: I think the first thing that Google or
>>>> > >> ChatGPT searches will tell you is "use an existing CQRS
>>>> framework"... and
>>>> > >> this will most likely show you AxonIQ (a CQRS framework
>>>> implemented with
>>>> > >> Spring/Boot)... but that is then even more of a vendor lock in
>>>> than using a
>>>> > >> more generic solution like Camel... Axon will force us to use their
>>>> > >> contracts (internal APIs, Java interfaces etc.), in short:
>>>> refactor fest,
>>>> > >> disruptive. There are other low level "solutions" (like LMAX
>>>> Disruptor)
>>>> > >> that are somewhat in the vicinity of this type of application, but
>>>> require
>>>> > >> work, to my knowledge there is nothing out there we could just
>>>> magically
>>>> > >> drop and use without any refactoring. Disclaimer: in one of the 3
>>>> drop-in
>>>> > >> implementations of the proposed command processing I am actually
>>>> using LMAX
>>>> > >> Disruptor... its implementation details just don't leak into the
>>>> rest of
>>>> > >> the system
>>>> > >> > Spring Boot 3 compliance: yes (buzzword drop:
>>>> "auto-configuration")
>>>> > >>
>>>> > >> JD: Excellent
>>>> > >>
>>>> > >> > cutting edge: not sure how to read this here... is this meant as
>>>> a
>>>> > >> requirement or as an argument against the adoption of the proposal
>>>> as in
>>>> > >> "too experimental"... as I've written the code I am obviously
>>>> biased so I
>>>> > >> leave that to the community to decide and come up with
>>>> improvements and/or
>>>> > >> alternatives/arguments if someone doesn't agree
>>>> > >>
>>>> > >> JD:  Yep. The ambiguity is on purpose - cutting edge can be great
>>>> in
>>>> > >> getting results, or it can make you bleed.
>>>> > >>
>>>> > >> >
>>>> > >> > Let me know if I skipped something, made an error or was not
>>>> clear
>>>> > >> enough.
>>>> > >>
>>>> > >> JD:  Very clear.  Now, before this code is committed, I would also
>>>> > >> like to be sure we have a sensible way of documenting the progress
>>>> so
>>>> > >> that if we are doing a release, we make note of how much of the
>>>> code
>>>> > >> base is using the new methods. I also think we should discuss this
>>>> in
>>>> > >> context of the next release.  (coming up soon).
>>>> > >>
>>>> > >> >
>>>> > >> > On Thu, Jan 23, 2025 at 5:00 PM <jdai...@apache.org> wrote:
>>>> > >> >>
>>>> > >> >> Thanks for bringing this to the list. It looks to be a very low
>>>> level
>>>> > >> (in the stack) and therefore, highly impactful. I was there when
>>>> the
>>>> > >> decision was made to adopt this pattern and
>>>> > >> SynchronousCommandProcessingService as a flexible improvement to
>>>> the
>>>> > >> existing CQRS. I remember asking some questions, but this was and
>>>> is,
>>>> > >> beyond my direct experience.
>>>> > >> >>
>>>> > >> >>
>>>> > >>
>>>> https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5:+New+command+processing+infrastructure
>>>> > >> >>
>>>> > >> >> What I do know is that we should be deliberate with this
>>>> process, and
>>>> > >> I appreciate your write up on wiki.  Definitely other architects
>>>> here
>>>> > >> should take a look.
>>>> > >> >>
>>>> > >> >> At times over the past decisions - it feels to me that we try
>>>> to be
>>>> > >> "too clever", and this creates a problem with maintainability.
>>>> I'd like to
>>>> > >> make sure we understand the alternatives as we dig into this. You
>>>> raised
>>>> > >> Apache Camel as an option - would it be worth it to ask someone
>>>> over in
>>>> > >> that project to comment on this?  Is there some whitepaper or
>>>> comparison
>>>> > >> out there between the alternatives available?  Is this consistent
>>>> with
>>>> > >> Spring Boot 3 ?  Is this on the cutting edge?
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>

Reply via email to