Hi, I have addressed the invalid flow json issue: https://issues.apache.org/jira/browse/NIFI-14149
Regards, Peter Kedvessy Péter <pkedve...@gmail.com> ezt írta (időpont: 2025. jan. 10., P, 10:49): > Hi Robert, > > Have you checked your minifi agent logs? Specifically the minifi-app.log. > There should be an error describing the reason for the failed > connection/communication with the remote NiFi instance... Is > https://odc-nifi-test.nci.org.au:444/nifi the right URI of your remote > NiFi instance where you are expecting flow files from the minifi agent? > > Regards, > Peter > > Robert Cohen <robert.co...@anu.edu.au.invalid> ezt írta (időpont: 2025. > jan. 8., Sze, 5:20): > >> OK, I fixed the validation. I had an error in my bootstrap.conf. >> >> Its not ideal that the toolkit transform-nifi generates a invalid flow >> file. >> >> >> Now I can get minifi to start. But the code isnt working. >> >> This is a trivial "ping" . It just uses a GenFlowFile to make a flow. >> And sends it to a remote process group. The remote process group lives >> in another nifi instance, and just logs and discards the flows. >> >> >> I had to move the code in the source nifi into sub group. And I exported >> that. And ran it through the converter. >> >> Inside nifi, its working fine. >> >> But in the minifi, its generating the flows. I can see them piling up in >> the WAL. But they arent being transmitted. Theres nothing being received >> by the remote nifi. >> >> OK, I fixed the validation. I had an error in my bootstrap.conf. >> >> Its not ideal that the toolkit transform-nifi generates a invalid flow >> file. >> >> >> Now I can get minifi to start. But the code isnt working. >> >> This is a trivial "ping" . It just uses a GenFlowFile to make a flow. >> And sends it to a remote process group. The remote process group lives >> in another nifi instance, and just logs and discards the flows. >> >> >> I had to move the code in the source nifi into sub group. And I exported >> that. And ran it through the converter. >> >> Inside nifi, its working fine. >> >> But in the minifi, its generating the flows. I can see them piling up in >> the WAL. But they arent being transmitted. Theres nothing being received >> by the remote nifi. >> >> >> >> On 8/1/2025 2:33 pm, Robert Cohen wrote: >> > I googled the error message. >> > >> > I found a thread here which seems the same >> > >> > >> https://community.cloudera.com/t5/Support-Questions/Unable-to-start-MiNiFi-because-of-an-error-caused-due-to-the/m-p/393219 >> > >> > >> > >> > One thing it suggests is editing the flow file and changing the >> > following setting from 0. >> > >> > |"maxTimerDrivenThreadCount":5 But when I do that minifi fails to >> > start as it says the flow file fails validation Does the flow file >> > contain a checksum? | >> > >> > >> > On 8/1/2025 1:27 pm, Robert Cohen wrote: >> >> Thanks for that. It clears some things up. >> >> >> >> Im not sure if the doco is supposed to be suitable for nifi 1/2 or >> >> just nifi 2. >> >> >> >> But you might want to clarify in the doco whether config files are >> >> just used by nifi 1. Just used by nifi 2. Or supported in nifi 2 for >> >> legacy. But not recommended. >> >> >> >> >> >> Anyway, Ive put my code from a nifi 2.1.0 into minifi 2.1.0. >> >> >> >> I get the following error >> >> >> >> 2025-01-08 13:16:09,065 ERROR [MiNiFi logging handler] >> >> org.apache.nifi.minifi.StdErr Failed to start flow service: Unable to >> >> load flow due to: >> >> org.apache.nifi.controller.serialization.FlowSynchronizationException: >> >> java.lang.IllegalArgumentException: Cannot set max number of threads >> >> to less than 1 >> >> >> >> I attach the converted flow file >> >> >> >> I can't see any obvious parameter related to max number of threads. >> >> >> >> >> >> >> >> On 7/1/2025 6:56 pm, Kedvessy Péter wrote: >> >>> [You don't often get email from pkedve...@gmail.com. Learn why this >> >>> is important at https://aka.ms/LearnAboutSenderIdentification ] >> >>> >> >>> Hi Robert, >> >>> >> >>> Thanks for reporting the documentation issues. I filed a JIRA to fix >> >>> them: >> >>> https://issues.apache.org/jira/browse/NIFI-14093 >> >>> >> >>> >> >>> In terms of security configuration, you can follow the following >> guide: >> >>> >> https://github.com/apache/nifi/blob/main/minifi/minifi-docs/src/main/markdown/System_Admin_Guide.md#security-properties-in-bootstrapconf >> >>> >> >>> >> >>> About the different configuration files you mentioned: >> >>> - bootstrap.conf is the main configuration file what we should update >> >>> - minifi.properties is a generated properties file based on the >> >>> bootstrap.conf >> >>> - nifi.properties is the legacy configuration file of minifi >> >>> - config.yaml is the legacy flow configuration, now we have the json >> >>> one >> >>> what you have tried to import previously >> >>> >> >>> I hope it helps to set up your minfi agent. >> >>> >> >>> Regards, >> >>> Peter >> >>> >> >>> Robert Cohen <robert.co...@anu.edu.au.invalid> ezt írta (időpont: >> 2025. >> >>> jan. 7., K, 6:19): >> >>> >> >>>> I'm having a certain amount of trouble getting good info on >> >>>> administration of minifi 2. >> >>>> >> >>>> The README file says thats documentation can be found at >> >>>> >> >>>> https://nifi.apache.org/minifi >> >>>> and >> >>>> https://cwiki.apache.org/confluence/display/NIFI/MiNiFi >> >>>> >> >>>> >> >>>> neither of which URL works. >> >>>> >> >>>> >> >>>> Im assuming the real documentation is at >> >>>> >> >>>> >> >>>> >> https://github.com/apache/nifi/blob/main/minifi/minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file >> >>>> >> >>>> >> >>>> >> >>>> But Im having some Im finding it a bit confusing. >> >>>> >> >>>> I want to configure the keystore/truststore and sensitive prop >> >>>> passwords. >> >>>> >> >>>> It appears I can configure them either in bootstrap.conf or >> >>>> minifi.properties. Is either to be prefered over the other? >> >>>> >> >>>> The documention mentions nifi.properties and config.yml. But doesnt >> >>>> mention minifi.properties. >> >>>> >> >>>> Im assuming the mentions of config.yml are leftover from minifi 1 >> >>>> documentation. Or does minifi 2 use it for backwards compatability. >> >>>> >> >>>> And I assume nifi.properties has been renamed to minifi.properties >> and >> >>>> the documentation hasnt caught up. >> >>>> >> >>>> >> >>>> Actually dumping strings out of the jar files, I can find instances >> of >> >>>> nifi.properties. But I cant find any references to minifi.properties. >> >>>> >> >>>> However the tarball contains a minifi.properties file but not a >> >>>> nifi.properties file. >> >>>> >> >>>> So Im really not sure what is going on. >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On 6/1/2025 4:40 pm, Robert Cohen wrote: >> >>>>> Ahh, it appears the functionality is there. Its just that the README >> >>>>> file hasnt been updated, >> >>>>> >> >>>>> Theres now a transform-nifi command in addition to the transform-yml >> >>>>> command. >> >>>>> >> >>>>> >> >>>>> On 6/1/2025 2:59 pm, Robert Cohen wrote: >> >>>>>> [You don't often get email from robert.co...@anu.edu.au.invalid. >> >>>>>> Learn why this is important at >> >>>>>> https://aka.ms/LearnAboutSenderIdentification ] >> >>>>>> >> >>>>>> I had a look at the minifi toolkit. >> >>>>>> >> >>>>>> However, it only appears to contain functionality for converting >> >>>>>> legacy >> >>>>>> minifi config to minifi 2 config >> >>>>>> >> >>>>>> Heres an excerpt from the minifi-tookit 2.1.0 README file >> >>>>>> >> >>>>>> Valid commands include: >> >>>>>> transform-yml: Transforms legacy MiNiFi flow config YAML >> into >> >>>>>> MiNiFi flow config JSON >> >>>>>> >> >>>>>> ## Example >> >>>>>> - You have an older version of MiNiFi located in >> >>>>>> <legacy_minifi_directory>. >> >>>>>> - You would like upgrade to the latest version of MiNiFi. You >> >>>>>> downloaded >> >>>>>> and extracted the latest MiNiFi into <latest_minifi_directory>. >> >>>>>> - Run the following command to migrate the flow and the bootstrap >> >>>>>> config >> >>>>>> ``` >> >>>>>> ./config.sh transform-yml <legacy_minifi_directory>/conf/config.yml >> >>>>>> <legacy_minifi_directory>/conf/bootstrap.conf >> >>>>>> <latest_minifi_directory>/conf/flow.json.raw >> >>>>>> <latest_minifi_directory>/conf/bootstrap.conf >> >>>>>> ``` >> >>>>>> >> >>>>>> >> >>>>>> Is there functionality somewhere to convert a nifi 2.x flow file >> >>>>>> to a >> >>>>>> minifi 2 flow file, >> >>>>>> >> >>>>>> >> >>>>>> On 12/12/2024 6:48 pm, Kedvessy Péter wrote: >> >>>>>>> [You don't often get email from pkedve...@gmail.com. Learn why >> this >> >>>>>>> is important at https://aka.ms/LearnAboutSenderIdentification ] >> >>>>>>> >> >>>>>>> Hi Robert, >> >>>>>>> >> >>>>>>> Seems like the documentation has not been updated. I'm going to >> >>>>>>> file >> >>>>>>> a JIRA >> >>>>>>> ticket to apply doc update about how we can import NiFi 2 >> >>>>>>> exported flow >> >>>>>>> files into MiNiFi. >> >>>>>>> In a nutshell: before moving the json file into minifi, we should >> >>>>>>> convert >> >>>>>>> it with minifi toolkit. Related ticket for more details: >> >>>>>>> https://issues.apache.org/jira/browse/NIFI-12644 >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Regards, >> >>>>>>> Peter >> >>>>>>> >> >>>>>>> Robert Cohen <robert.co...@anu.edu.au.invalid> ezt írta (időpont: >> >>>> 2024. >> >>>>>>> dec. 12., Cs, 2:11): >> >>>>>>> >> >>>>>>>> Im having trouble getting code into minifi 2 from nifi 2. >> >>>>>>>> >> >>>>>>>> The documentation here >> >>>>>>>> >> >>>>>>>> >> >>>> >> https://github.com/apache/nifi/blob/main/minifi/minifi-docs/src/main/markdown/minifi-java-agent-quick-start.md >> >>>> >> >>>>>>>> >> >>>>>>>> says >> >>>>>>>> >> >>>>>>>> # Export the dataflow in JSON format. >> >>>>>>>> # Move your new .json file to |minifi/conf|. >> >>>>>>>> # Rename your .json file /flow.json.raw/. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Im using the option in nifi to "download flow definition" >> >>>>>>>> >> >>>>>>>> However when I load that into minifi, I get an error message. >> >>>>>>>> >> >>>>>>>> java.lang.Exception: Unable to load flow due to: >> >>>>>>>> >> org.apache.nifi.controller.serialization.FlowSerializationException: >> >>>>>>>> >> >>>>>>>> Could not parse flow as a VersionedDataflow >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> When I compare the file to the dummy flow file that ships with >> >>>>>>>> minifi. >> >>>>>>>> >> >>>>>>>> The file that ships with minifi starts with >> >>>>>>>> >> >>>>>>>> { >> >>>>>>>> "encodingVersion": { >> >>>>>>>> "majorVersion": 2, >> >>>>>>>> "minorVersion": 0 >> >>>>>>>> }, >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> But the export from nifi has >> >>>>>>>> >> >>>>>>>> >> {"flowContents":{"identifier":"a2ac574c-635d-3df5-baf0-6cbf4dd5bdec", >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> So the 2 files dont appear to be the same format. >> >>>>>>>> >> >>>>>>>> Is there some other method of exporting code from nifi that Im >> >>>>>>>> missing? >> >>>>>>>> >> >>>>>>>> >> >
