[DISCUSS] Google Summer of Code 2018

2018-02-27 Thread Matt Burgess
If you haven't heard yet, the Apache Software Foundation was selected as an organization for this year's Google Summer of Code [1]. I've seen activity on other Apache projects' mailing lists requesting ideas for issues, features, components, etc. that could be good proposals/ideas for GSoC, and I'd

Re: A question to copy files to other folders

2018-02-28 Thread Matt Burgess
Looking at the code, it appears that PutFile should support the creation of arbitrary directories if the Create Missing Directories property is set to true. With that and setting the Directory property to ${path}, I would think that would create the subdirectories properly. If not, what error are y

Re: [DISCUSS] Google Summer of Code 2018

2018-02-28 Thread Matt Burgess
[1] http://s.apache.org/gsoc2018ideas On Wed, Feb 28, 2018 at 12:12 AM, Joe Witt wrote: > Matt > > Did you have some ideas/features/enhancements in mind you think would > be good to propose? > > Thanks > Joe > > On Tue, Feb 27, 2018 at 6:56 PM, Matt Burgess wrote: >>

Re: Hive backward compatibily

2018-03-05 Thread Matt Burgess
Milan, Unfortunately no, without code changes the Hive processors are not compatible with Hive 1.1.x. There were API changes both to the Java code and the Thrift code (the latter adds the client_protocol field you mentioned), so you can't change the version of the Hive dependencies and get the NA

Re: NiFi build failed, but got reported as a success

2018-03-10 Thread Matt Burgess
The Atlas profile is intentionally not included in the default assembly, I believe due to its size. Sent from my iPhone > On Mar 10, 2018, at 10:03 AM, Pierre Villard > wrote: > > Hi Mike, > > I see where the problem is. It's been introduced with NIFI-4936. > I also noticed that the atlas pr

Re: [VOTE] Establish Fluid Design System, a sub-project of Apache NiFi

2018-03-11 Thread Matt Burgess
+1 On Sun, Mar 11, 2018 at 1:00 PM, Jeff wrote: > +1 > > On Sat, Mar 10, 2018 at 8:42 PM Joe Skora wrote: > >> +1 >> >> >> On Fri, Mar 9, 2018, 3:10 PM Scott Aslan wrote: >> >> > All, >> > >> > Following a solid discussion for the past couple of weeks [1] regarding >> the >> > establishment of

Re: NiFi SplitAvro processor giving extra records

2018-03-12 Thread Matt Burgess
Rohit, Are you routing both the "split" and "original" relationships to the downstream processor? If so, and you don't want the 6th flowfile containing all 5 records, then do not route the original relationship to the downstream processor. You can route it to some other processor or self-terminate

Re: HiveConnectionPool URL with trustStorePasswrd

2018-03-14 Thread Matt Burgess
Milan, You could subclass the existing HiveConnectionPool, add a property for key/truststore password that is sensitive, then build the URL manually during a call to getConnection(). Then the rest of the URL would not have to remain sensitive, if that is preferable. You might even consider an SSL

Re: ExecuteHQL failed to reading table from Hive

2018-03-14 Thread Matt Burgess
Rohit, Do you mean the SelectHiveQL processor? If so can you share more information about your configuration, the query, the table, etc.? If you are using a custom processor, can you share its location? Perhaps the author or other users of it can address this. Thanks, Matt On Wed, Mar 14, 2018

Re: "Method not supported" error facing GenerateTableFetch and QueryDatabaseTable processor

2018-03-20 Thread Matt Burgess
Rohit, Hive is not supported by the database fetch processors yet, there is a Jira [1] and some PRs [2],[3] under review to add this capability. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-3093 [2] https://github.com/apache/nifi/pull/1281 [3] https://github.com/apache/nifi/pull/

Re: FlattenJson

2018-03-20 Thread Matt Burgess
I think Bryan is asking about what happens AFTER this part of the flow. For example, if you are doing routing you can use QueryRecord (and you won't need the SplitJson), if you are doing transformations you can use JoltTransformJSON (often without SplitJson as well), etc. Regards, Matt On Tue, Ma

Re: FlattenJson

2018-03-20 Thread Matt Burgess
gt;> >> From there on we use a lot of routeOnAttritutes and use that values on >> sql queries to other tables like select * from someTable where >> id=${myExtractedAttribute} >> >> To be honest I tryed JoltTransformJSON but I could not get it working :) >> >> >

Re: ExecuteHQL failed to reading table from Hive

2018-03-21 Thread Matt Burgess
This sounds like a bug in HiveJdbcCommon, we should be able to support a Hive SMALLINT into an Avro integer or something. On Tue, Mar 20, 2018 at 11:53 AM, Rohit wrote: > Hi Matt, > > Issue got resolved after casting smallint to int, as avro not supported > smallint datatype. > > Thanks. > > > >

Re: Class Loading Conflicts - Different JAR Versions

2018-03-26 Thread Matt Burgess
Bri/yan, IMO I think we'd be better off with moving the nifi-mongodb-client-service-api and nifi-mongodb-services bundles into the nifi-mongodb-bundle. That's something I missed while reviewing the PR that put them in standard services. I don't see a direct dependency on nifi-standard-services, an

Re: Class Loading Conflicts - Different JAR Versions

2018-03-26 Thread Matt Burgess
stry will be a powerful capability. > > Also, +2 for the greeting, though Bryan may choose to put the "y" first. :-) > > Thanks, > Brian > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess wrote: > >> Bri/yan, >> >> IMO I think we'd be better

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

2018-03-27 Thread Matt Burgess
+1 (binding) Release this package as nifi-1.6.0 Verified checksums, commit, L&N, etc. Ran full build with unit tests and tried various flows, all looks well! On Mon, Mar 26, 2018 at 11:34 PM, Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache > Ni

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

2018-04-04 Thread Matt Burgess
+1 (binding), ran through the verification helper on Mac OS X 10.13.4 with full build, tried various flows and validated a couple fixes. Thanks for handling the release Joe! On Tue, Apr 3, 2018 at 7:49 PM, Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release o

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

2018-04-06 Thread Matt Burgess
Mike, Is there a workaround for this? Like setting the query key(s) for insert even if they are ignored? Or will that cause other issues? Thanks, Matt On Fri, Apr 6, 2018 at 8:56 AM, Mike Thomsen wrote: > Sorry to be the bearer of bad news, but this morning I found a bug in > PutMongo that bre

Re: How can I pass a flowfile attribute to a controller service?

2018-04-10 Thread Matt Burgess
+1 to Andy's answer. Note that DBCPConnectionPool isn't a connection but a connection pool, which keeps some number of connections available for use by multiple consumers (processors, e.g.). I wouldn't want to have to keep track of X number of configurations in a single controller service, which wo

Re: Calling getLogger() from @OnScheduled, @OnStopped, etc.

2018-04-12 Thread Matt Burgess
I’m AFK right now but I’m thinking there is a point in the lifecycle where you can expect a logger, and before that (like when the processor is instantiated as you have it coded), you may not be able to assume there is a logger (either in real NiFi or the mock-verse :) My guess is the rule of t

Re: Apache nifi isuues-while ingesting data to hdfs of aws instance

2018-04-19 Thread Matt Burgess
If that's the case, you may be able to fix that by adding the following property to your hdfs-site.xml: dfs.client.use.datanode.hostname = true It should cause the hostnames of the data nodes to be returned to the client, rather than private/inaccessible IPs. Regards, Matt On Thu, Apr 19, 2018

Re: No controller service types found that are applicable for this property

2018-04-19 Thread Matt Burgess
You may also need to add a NAR dependency to your processor's NAR's POM if you haven't already: org.apache.nifi nifi-standard-services-api-nar 1.7.0-SNAPSHOT nar On Thu, Apr 19, 2018 at 12:08 PM, rishabprasad005 wrote: > Yes, I have added nifi-dbcpservice-api in processor pom

Re: ExecuteGroovyScript processor unable to resolve IOUtils

2018-04-23 Thread Matt Burgess
Max, ExecuteGroovyScript does not (yet) automatically include some Apache Commons libraries such as commons-lang3, commons-io, commons-net, etc. However ExecuteScript does already include those, so you can put your Groovy script in there and not have to @Grab the library. Of course, you won't get

Re: Graph database support w/ NiFi

2018-05-12 Thread Matt Burgess
All, As Joe implied, I'm very happy that we are discussing graph tech in relation to NiFi! NiFi and Graph theory/tech/analytics are passions of mine. Mike, the examples you list are great, I would add Titan (and its fork Janusgraph as Kay-Uwe mentioned) and Azure CosmosDB (these and others are at

Re: NiFi code re-use

2018-05-13 Thread Matt Burgess
IMO the concept of "functions" is different enough to warrant its own set of components, rather than trying to shoehorn them into a wormhole / PG discussion. Perhaps a Function component which may feel like a PG (but with UI distinction such as rounded corners?), but would be referred to by name (u

Re: [DISCUSS] Change Apache NiFi Fluid Design System naming

2018-05-17 Thread Matt Burgess
Makes sense to me, I'm a +1 as well. On Wed, May 16, 2018 at 4:52 PM, Scott Aslan wrote: > Good timing as we're close to ready for a first nifi-fds release. > > I would definitely favor us keeping the 'nifi-fds' naming as that means I > dont have to change a bunch of code so Apache NiFi Flow Desi

Re: [VOTE] Change the name Apache NiFi Fluid Design System to Apache NiFi Flow Design System

2018-05-18 Thread Matt Burgess
+1 (binding) On Fri, May 18, 2018 at 10:26 AM, Bryan Bende wrote: > +1 (binding) > > On Fri, May 18, 2018 at 10:18 AM, James Wing wrote: >> +1 (binding) >> >>> On May 18, 2018, at 6:30 AM, Rob Moran wrote: >>> >>> Following positive response discussing the name change of nifi-fds [1], I'd >>> l

Re: How to capture failure bulletins in custom processor

2018-05-25 Thread Matt Burgess
You can use a SiteToSiteBulletinReportingTask [1], it will send flow files containing bulletins to an Input Port on the root process group (aka top-level canvas), then you can send to whichever processors you like for further operations. Regards, Matt [1] https://nifi.apache.org/docs/nifi-docs/c

Re: Using EL classes with just java Map objects

2018-06-02 Thread Matt Burgess
Mike, IIRC the "top-level" EL evaluator will go through a string finding EL constructs and pass them into Query I think. Also ReplaceText (for some reason) is the only place I know of where you can quote something and (if EL is present), the result is treated as a string literal. Otherwise in NiF

Re: [ANNOUNCE] New Apache NiFi Committer Sivaprasanna Sethuraman

2018-06-06 Thread Matt Burgess
Congratulations and welcome aboard Sivaprasanna! On Tue, Jun 5, 2018 at 10:09 AM, Tony Kurc wrote: > On behalf of the Apache NiFI PMC, I am very pleased to announce that > Sivaprasanna has accepted the PMC's invitation to become a committer on the > Apache NiFi project. We greatly appreciate all

Re: Executing Stored Procedure only once

2018-06-06 Thread Matt Burgess
Anil, Are you using SplitRecord because something times out in PutDatabaseRecord otherwise? If not, you shouldn't need the SplitRecord, and without it, things should work the way you expect. If you do need to split into smaller flow files, you should be able to use Wait/Notify with the parent flo

Re: [ANNOUNCE] New NiFi PMC member Mike Thomsen

2018-06-06 Thread Matt Burgess
Congratulations Mike! Great to have you aboard, I appreciate your commitment and work ethic and looking forward to continued great things! On Wed, Jun 6, 2018 at 6:22 PM, Tony Kurc wrote: > NiFi community, > On behalf of the Apache NiFi PMC, I am pleased to announce that Mike > Thomsen has accep

Re: [DISCUSS] Release candidate timeframe for Apache NiFi 1.7.0

2018-06-10 Thread Matt Burgess
essSession.read resulting in outer stream > > being > > > >>> closed — Mark Payne > > > >>> * NIFI-5209 Remove toolkit migration without password functionality — > > > Andy > > > >>> LoPresto > > > >>> * NIFI-5166 C

Re: Adding new data anonymization processor bundle

2018-06-20 Thread Matt Burgess
I think is a great idea, I filed a Jira [1] a while ago in case someone wanted to start working on it (or in case I got a chance). It mentions ARX but any Apache-friendly implementation is of course welcome. I think it should be in its own bundle as it is functionality separate from all our other b

Re: [VOTE] Release Apache NiFi 1.7.0

2018-06-20 Thread Matt Burgess
+1 (binding) Verified all artifacts, full build with contrib-check, verified the Hive 3 NAR is not in the assembly unless the include-hive3 profile is activated, also ran through various flows to exercise Hive 3 and PutORC functionality (and their associated Record Readers, Writers, and intermedia

Re: [VOTE] Release Apache NiFi 1.7.0

2018-06-21 Thread Matt Burgess
> '${project.basedir}'. > > Thanks. > > On Wed, Jun 20, 2018 at 11:15 PM, Matt Burgess wrote: > > > +1 (binding) > > > > Verified all artifacts, full build with contrib-check, verified the > > Hive 3 NAR is not in the assembly unless the i

Re: validation changes in 1.7.0

2018-07-02 Thread Matt Burgess
Do your processors work against 1.6.0? I’m wondering if there’s something with the new Expression Language scope stuff, I believe that can cause unit test failures even if it still works (albeit deprecated) on a live NiFi. Regards, Matt > On Jul 2, 2018, at 4:52 PM, Mark Payne wrote: > > Hey

Re: Bug in drop downs?

2018-07-07 Thread Matt Burgess
I would say it’s “expected”, properties with allowable values shouldn’t support EL per se, although there are properties in NiFi where one of the allowable values is “Use x.y.z attribute” which implies Flow File scope but isn’t documented via annotations or anything but the description. Regards

Re: [discuss] should we do a nifi 1.7.1 release?

2018-07-09 Thread Matt Burgess
This Jira search [1] for Bugs resolved in 1.8.0 has 10 items, there are some I don't see in this list (NIFI-5278 [2] and NIFI-5349 [3]), should these all be included or on a case-by-case basis? Thanks, Matt [1] https://issues.apache.org/jira/browse/NIFI-5394?jql=project%20%3D%20NIFI%20AND%20fixV

Re: Status history graphs over 24hrs (v1.3.0)

2018-08-03 Thread Matt Burgess
Netflix uses Atlas [1], the standalone JAR is only 8 KB [2] but it's written in Scala, not sure what the overall footprint of the Scala libraries and other dependencies are... Regards, Matt [1] https://github.com/netflix/atlas/wiki [2] https://mvnrepository.com/artifact/com.netflix.atlas/atlas-st

Re: [CaptureChangeMysql] Cannot get real value of primitive type when column type is unsigned.

2018-08-08 Thread Matt Burgess
Looks like they've identified the onus being on the client not the library. We do read column types when a Distributed Map Cache Client is specified, so we should probably be able to do these conversions. Please feel free to write a Jira [1] to cover this. Thanks, Matt [1] https://issues.apache.o

Re: Redis making licensing changes - we need to be mindful of this

2018-08-22 Thread Matt Burgess
Joe, Thanks for bringing this to our attention, definitely something to keep aware of. I did some initial digging, it appears that the "normal" Redis stuff is still Apache 2.0 licensed, and it appears that will remain; also AFAICT we are only using the "normal" Redis bits via spring-data-redis, bu

Re: ExecuteSQL failed to process due to illegal character

2018-09-07 Thread Matt Burgess
ExecuteSQL outputs Avro with an embedded schema, it is the Avro schema that doesn't allow parentheses in the field names. You can avoid the error by setting Normalize names to true as you found, and also you could alias the field names in the SQL itself, such as: SELECT RANGE(flowenddate) AS flowE

Re: ExecuteSQL Avro schema: all fields are nullable

2018-09-13 Thread Matt Burgess
Christophe, IIRC the isNullable() method gives inconsistent results across JDBC drivers (see [1], [2] for examples) and in such cases might cause more harm than good. Having said that, we could perhaps add a property to the relevant components such as "Honor Non-Nullable Fields" or something like

Re: how to hide or show some nifi property when i change drop-down box value ?

2018-09-17 Thread Matt Burgess
I think the general capability you are referring to is called "dependent properties" and it has not yet been implemented [1]. To some extent you can do this programmatically by overriding customValidate() or onPropertyModified() for example, and updating the list of PropertyDescriptors that are re

Re: POJO to Record

2018-09-28 Thread Matt Burgess
Moving users list to BCC and adding devs list, as this question is better suited for the devs. Ed, Does XMLReader not parse your XML? I guess you'd need to know the schema so if you don't (especially if it is dynamic) then you might be faced with a custom processor as you described. Take a look

Re: Issue with ValidateRecord processor and Avro map data type

2018-10-10 Thread Matt Burgess
Maxime, I believe this is a bug, specifically at [1]. It should support MapRecord entries as well. I have written NIFI-5678 [2] to address this. The workaround is the one you and Bryan mentioned, using a record instead of a map. Regards, Matt [1] https://github.com/apache/nifi/blob/master/nifi-

Re: Graph database support w/ NiFi

2018-10-15 Thread Matt Burgess
ertex with label "Token" and attribute "name"="Mary"). If the >>>>> vertex does not exist, the vertex has to be created. >>>>> Since I want to save e.g. Wikipedia to my graph I want to avoid the >>>>> supernode problem for t

Re: [VOTE] Release Apache NiFi 1.8.0

2018-10-19 Thread Matt Burgess
+1 (binding), verified via release helper, ran full build with unit tests, tried various flows on a standalone instance. Great work everyone! Thanks Jeff for RM'ing! On Wed, Oct 17, 2018 at 11:59 PM Jeff wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache N

Re: Need help with Controller Service implementation

2018-10-21 Thread Matt Burgess
I’m not at my computer but I think your ControllerService file (for ServiceLoader) might need to be in META-INF/services instead of META-INF.services? Sent from my iPhone > On Oct 21, 2018, at 1:56 PM, Sivaprasanna wrote: > > Team, > > I'm working on a controller service implementation (NIFI

Re: [VOTE] Release Apache NiFi 1.8.0 (RC3)

2018-10-23 Thread Matt Burgess
+1 (binding), ran through release helper, tested various flows including record-based processors and Hive 3 processors (although the Hive 3 NAR is not in the assembly). On Tue, Oct 23, 2018 at 1:56 AM Jeff wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache

Re: CSVreader incorrectly parse RFC4180

2018-10-24 Thread Matt Burgess
I think you're running into NIFI-5525 [1], it should be fixed in master as well as the upcoming 1.8.0 release. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-5525 On Wed, Oct 24, 2018 at 7:14 AM Nalinrat Laekawan wrote: > Hi nifi team, > > I found some issue with CSVreader in NI

Re: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

2018-10-25 Thread Matt Burgess
NiFi (as of the last couple releases I think) has the ability to set auto-terminating relationships; this IMO is one of those use cases (for NiFi 1.x). If new relationships are added, they could default to auto-terminate; then the existing processors should remain valid. However we might want an "o

Re: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

2018-10-25 Thread Matt Burgess
I do not think most users > would appreciate it being a condition silently not handled by the normal > failure path. > > Thanks, > Peter > > -Original Message- > From: Matt Burgess [mailto:mattyb...@apache.org] > Sent: Friday, October 26, 2018 10:18 AM >

Re: [PutElasticSearchProcessor] - Put Elastic search Processor with External version

2018-11-06 Thread Matt Burgess
Ulisses, There is a Jira [1] along with a patch [2] to add this feature, but it appears to have languished for most of the year. I will leave a comment on the PR to see if the author is interested in continuing the work; otherwise, perhaps someone else from the community will pick it up. Regards,

Re: Help with loading a file into a cache

2018-11-30 Thread Matt Burgess
Dave, Depending on the processor you're going to use to store these records into cache (see Mike's reply), if you want to convert each of the lines to JSON objects, you can use ReplaceText: Search Value: ^([^:]+):(.*) Replacement Value: {"$1":$2} Replacement Strategy: Line By Line This creates a

Re: Contributions and getting started

2019-01-23 Thread Matt Burgess
Ryan, I'm glad to join my NiFi community colleagues by saying "welcome aboard!!" and we'll do our best to get you up and running :) For the Jira part, Apache's Jira lets you create a case without being a contributor for that particular project. For that reason, there are a few NIFI cases out there

Re: JIRA Ticket Help Needed

2019-01-30 Thread Matt Burgess
Alex, Your interest in contributing is most welcome, thanks in advance! What is your Apache Jira username? I should be able to take care of this for you. Regards, Matt On Wed, Jan 30, 2019 at 4:59 PM Alex Aversa wrote: > > Hello, > > I had created a ticket in JIRA and it appears I am unable to

Re: [EXT] [discuss] release apache nifi 1.9.0

2019-02-13 Thread Matt Burgess
. > > > > > > > > On Thu, Feb 7, 2019 at 1:17 AM Andy LoPresto > > > wrote: > > > > > > > > > Ah, sorry, I misunderstood. You are not the PR author; you want those > > > > same > > > > > comments to be addressed. >

Re: [VOTE] Release Apache NiFi 1.9.0 (rc2)

2019-02-19 Thread Matt Burgess
+1 (binding) Release this package as nifi-1.9.0 Ran through release helper, tried various flows using several components, everything looks good. Thanks Joe for RM'ing! On Sat, Feb 16, 2019 at 10:50 PM Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of

Re: NIFI Unit Test Error with @OnScheduled annotation

2019-02-27 Thread Matt Burgess
+1 to Bryan's comments. If I had to guess I'd say the error is probably in the line: kuduTable = clientService.getKuduClient().openTable(tableName); That's the only one that does heavy lifting vs standard NiFi API calls, and should be in a try/catch for that reason. Also IIRC setting up external

Re: Issue with json.decode in jython 2.7.1

2019-03-13 Thread Matt Burgess
Kevin, I feel your pain, that's a particularly nasty bug, and in fact there were a few bugs to be included in the 2.7.2 release. However, AFAIK Jython 2.7.2 was never released. The latest in Maven is 2.7.1 [1], and even in Github there's only a 2.7.2 alpha tag [2]. Do you know of a place to get a

Re: [VOTE] Release Apache NiFi 1.9.1 (rc1)

2019-03-13 Thread Matt Burgess
+1 (binding) Release this package as nifi-1.9.1 Ran the release helper and various flows including one exercising one of the changes in this release. On Wed, Mar 13, 2019 at 1:49 AM Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.9.1.

Re: Support Gradle for building NAR

2019-04-18 Thread Matt Burgess
Franklin, What kind of support do you mean? Maybe an "official" NiFi Gradle plugin to build NARs, like the NAR MOJO plugin for Maven? What are the concerns with using the third-party library (I heard it works fine)? I also wrote my own inline script [1] (rather than a plugin) you could use in your

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.0

2019-05-03 Thread Matt Burgess
+1 (binding) Release this package as nifi-nar-maven-plugin-1.3.0 Ran through release helper, verified artifacts. Thanks for RM'ing Bryan! On Tue, Apr 30, 2019 at 12:03 PM Bryan Bende wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache > NiFi NAR Maven Plug

Re: Adding schema inference

2019-05-21 Thread Matt Burgess
Mike, Check AbstractRouteRecord, it uses the read-first-get-schema-read-rest pattern. However for that snippet, it is the RecordReader that is possibly updating the schema (currently the only thing that does this is schema inference), then the RecordSetWriter is created using the (possibly updated

Re: Adding schema inference

2019-05-23 Thread Matt Burgess
multiple schema levels that don't exist. How should that be handled? An > exception or generating empty record declarations that have one field? > > On Tue, May 21, 2019 at 2:07 PM Matt Burgess wrote: > > > Mike, > > > > Check AbstractRouteRecord, it uses the read-fi

Re: Json conversion query : Help

2019-05-24 Thread Matt Burgess
Try this spec: [ { "operation": "shift", "spec": { "somedate": "somedate.\\$date" } } ] I haven't tried this in NiFi (only in http://jolt-demo.appspot.com), if there's an issue with Expression Language you might need to escape more of the $ somehow. Regards, Matt On Fri, M

Re: Adding Searchable to components

2019-07-08 Thread Matt Burgess
Mike, I definitely like the idea. If the list is long we could create an Epic case and people can keep biting off chunks of it into Jira cases underneath. Or if they're easily categorizable (like extending your recent PR [1] into all scriptable components), we could have a Jira for each "area", as

Re: [discuss] approaching a NiFi 1.10.0 release

2019-08-29 Thread Matt Burgess
Mike, I’ll review those two graph PRs tonight or tomorrow (if they’re still open by then) > On Aug 29, 2019, at 3:43 PM, Mike Thomsen wrote: > > I have two open graph-related PR's that are really small and are needed to > close some bugs that will be bad for early adopters: > > https://github

Re: Jira contributor access request

2019-10-16 Thread Matt Burgess
Andrew, Welcome! I have added you as a Contributor to all the NiFi projects. Looking forward to your contributions! Regards, Matt On Wed, Oct 16, 2019 at 11:31 AM Andrew Rodriguez wrote: > > Hello, > > I was hoping to gain access to contribute, as I'm able. My Jira username is > atrodriguez. >

Re: [ANNOUNCE] New Apache NiFi Committer Peter Turcsanyi

2019-10-28 Thread Matt Burgess
Congratulations Peter! On Sun, Oct 27, 2019 at 9:05 PM Aldrin Piri wrote: > > Apache NiFi community, > > On behalf of the Apache NiFI PMC, I am very pleased to announce that Peter > has accepted the PMC's invitation to become a committer on the Apache NiFi > project. We greatly appreciate all of

Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)

2019-11-01 Thread Matt Burgess
+1 (binding) Ran thru the release helper and tried a number of flows with various controller services and processors, LGTM Thanks for RM'ing Joe! On Tue, Oct 29, 2019 at 1:32 PM Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > nifi-1.10

Re: jira contributer access request

2019-11-12 Thread Matt Burgess
Nissim, I have added you as a contributor on the 3 NiFi projects in Jira. Looking forward to your contributions! Regards, Matt On Tue, Nov 12, 2019 at 12:29 PM Nissim Shiman wrote: > > Hello Apache Nifi team, > I would like to request jira contributer access to apache nifi. I would like > to

Re: New Distributed Map Cache Impementations

2019-11-14 Thread Matt Burgess
Shawn, There are also Redis and Couchbase distributed map cache clients already in NiFi. I don't see any Jiras or PRs related to a DynamoDB or JDBC ones. I thought about making ones for JDBC, Hazelcast and/or Nitrite [1] (with or without a DataGate server), but never got around to it. I think Dyna

Re: Jira contributor access

2019-12-04 Thread Matt Burgess
Roger, I have added you as a Contributor to the NIFI, NIFIREG, and NIFILIBS projects in Jira, looking forward to your contributions! I'm interested to know what you have in mind for native Vertica support vs the RDBMS processors using the Vertica JDBC driver, are you thinking bulk loading capabili

Re: Error with doc generation for a custom NiFi NAR

2019-12-11 Thread Matt Burgess
Alex, This same thing happened to me a day or two ago. I'm not sure if it's a bug or works as intended, but the problem stems from trying to generate doc for an interface that extends ControllerService rather than a class implementation of ControllerService. I believe it has something to do with b

Re: add me to Jira contributor access for Apache NIFI

2019-12-16 Thread Matt Burgess
Juan, I have added you as a Contributor to the 3 NiFi Jira projects, looking forward to your contributions! Regards, Matt On Mon, Dec 16, 2019 at 9:54 PM Juan Sequeiros wrote: > > Hello all, > > I am requesting "Jira contributor access" and my username is > digitalplumber. > > Thanks > -- > Ju

Re: Jira contributor access

2019-12-16 Thread Matt Burgess
Daniel, I have added you to the 3 NiFi projects in Apache Jira as a Contributor, looking forward to your contributions! Regards, Matt On Sat, Dec 14, 2019 at 3:51 PM Daniel Magyar wrote: > > Hi, > > I'd love to start contributing to Nifi, could you please grant > me contributor access on the ji

Re: Apache NiFi Jan 2020 Board Report

2020-01-07 Thread Matt Burgess
Looks good to me, thanks for putting this together Joe! On Tue, Jan 7, 2020 at 1:32 PM Joe Witt wrote: > > Team, > > Here is the content of the board report I submitted for us this quarter. > Again great progress! > > Thanks > Joe > > > ## Description: > The mission of NiFi is the creation and ma

Re: Dynamic property for controller service

2020-01-09 Thread Matt Burgess
Mark, Good timing; Shawn, Bryan and I were talking about that on the Apache NiFi Slack just today. It's been captured in NIFI-6852 [1] and Shawn has drafted a PR [2] and is working on a unit test for it. My overall thought is that you really shouldn't be binding to the generic ControllerService, r

Re: Jira contributor access

2020-01-15 Thread Matt Burgess
Richie, I have added you as a Contributor to the three NiFi projects in Jira. Welcome, looking forward to your contributions! Regards, Matt On Wed, Jan 15, 2020 at 11:34 AM Richie Howell wrote: > > Hi all, > I'd like to request Jira contributor access, my username is rhowell > > Thanks, > Richi

Re: NiFi using HiveQL in secured kerberos environment with Zookeeper connection string - failing

2020-01-16 Thread Matt Burgess
Erik, It's been fixed in Hive 3, and even though the Hive 3 components don't come with the distribution of Apache NiFi (due to size concerns), they can be added manually from the Apache repository [1] [2]. The issue was fixed in Hive 2 but NiFi does not have a Hive 2 bundle, so if you are running

Re: [VOTE] Release Apache NiFi 1.11.0 (rc3)

2020-01-21 Thread Matt Burgess
+1 (binding) Ran through release helper, tried various flows including some new features (PutRecord, e.g.) PutRecord might not have made the improvements list, changelog, release notes, etc. for 1.11 due to an omission in Jira, but the Jira has since been corrected [1] -Matt [1] https://issues.a

master is now 1.12.0-SNAPSHOT

2020-01-23 Thread Matt Burgess
This is a friendly reminder that if you have PRs/patches open against SNAPSHOT components (or perhaps adding your own) that you should update the versions to 1.12.0-SNAPSHOT and re-submit, this will help expedite the review/merge process, please and thanks!

Re: Jira contributor access

2020-02-04 Thread Matt Burgess
Xun, I have added you as a contributor to the NiFi projects in Jira. Welcome! Looking forward to your contributions. Regards, Matt On Tue, Feb 4, 2020 at 7:13 AM Xun REN wrote: > > Hello, > > Could you please give me the Jira contributor access for NiFi ? > > My Jira username is: renxunsaky > >

Re: Requesting contributor access

2020-02-10 Thread Matt Burgess
Bence, I have added you as a contributor to the NiFi projects in Jira. Welcome aboard, looking forward to your contributions! Regards, Matt On Mon, Feb 10, 2020 at 11:45 AM Bence Simon wrote: > > Dear NiFi devs, > > I would like to ask you to grant me contributor access. My username is > simonb

Re: Adding Failure Information to AbstractRecordProcessor

2020-02-12 Thread Matt Burgess
Shawn, We've done this ad hoc in other places when it made sense to do so. Maybe we'd want to avoid adding any record values to the error message, although since the attribute is associated with the flow file content I'm not sure that's necessary from a security perspective. The only other thing I

Re: QueryDatabaseTable custom query generates invalid Oracle queries

2020-02-13 Thread Matt Burgess
Can you share your configuration for QueryDatabaseTable? Is the "Database type" property set to the right version of Oracle? Also I believe you'll need the table name to be capitalized. Regards, Matt On Thu, Feb 13, 2020 at 10:20 AM aboubakry wane wrote: > > Hello,I use nifi to collect data from

Re: [VOTE] Release Apache NiFi 1.11.2 (rc1)

2020-02-18 Thread Matt Burgess
+1 (binding) ran through release helper and various flows. Thanks Joe for RM'ing! On Fri, Feb 14, 2020 at 10:15 PM Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.11.2. > > The source zip, including signatures, digests, etc. can be fou

Re: [VOTE] Release Apache NiFi 1.11.3

2020-02-22 Thread Matt Burgess
+1 (binding), ran through release helper and verified the file leaks are fixed for DistributedMapCacheClientService and the SiteToSite Reporting Tasks. Thanks for RM'ing Joe! On Fri, Feb 21, 2020 at 10:21 PM Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source releas

Re: [DISCUSS] Advanced search capabilities

2020-02-25 Thread Matt Burgess
What about an in-memory (perhaps with disk persistence) representation of the flow as a graph using Apache Tinkerpop? That would alleviate the need for a separate graph DB while still allowing you to do graph searches, at the cost of more memory usage. Not sure what the footprint would look like fo

Re: nifi connection with hive

2020-03-04 Thread Matt Burgess
Aya, The Hive components included with Apache NiFi are not compatible with Hive 2.x, they are built with Hive 1.2.x. There is a Jira to add Hive 2 support [1] but it is not yet in NiFi. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-6456 On Tue, Mar 3, 2020 at 6:35 AM Aya Asfour

Re: [VOTE] Release Apache NiFi 1.11.4

2020-03-19 Thread Matt Burgess
+1 (binding), ran through release helper and tried various flows (verifying some of the changes in this release). Thanks for RM'ing Joe! On Wed, Mar 18, 2020 at 1:16 PM Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.11.4. > > The sou

Re: Upgrade to Avro 1.9.X

2020-03-24 Thread Matt Burgess
Looking at the Jiras that were released between 1.8.1 (our current version) and 1.9.2 (the current GA version of Avro) [1], it appears that if we were to upgrade Avro, we definitely should go to the latest (there are a number of regressions in 1.8.2 and 1.9.0 for example). However I'm a little leer

Re: Upgrade to Avro 1.9.X

2020-03-24 Thread Matt Burgess
age of the ObjectMapper class to what it is in Databind 2.x. I can't find any other reference to the old Databind (or its ObjectMapper) yet, but still on the case :) On Tue, Mar 24, 2020 at 1:25 PM Matt Burgess wrote: > > Looking at the Jiras that were released between 1.8.1 (our current >

Re: Upgrade to Avro 1.9.X

2020-03-24 Thread Matt Burgess
w the JSON nodes are referenced. I cannot > remember what the exact issue was, but it wasn't a quick drop and replace > with maybe a handful of edits. > > Maybe I was looking at the wrong thing though... > > On Tue, Mar 24, 2020 at 1:26 PM Matt Burgess wrote: > > > Looki

Re: question about RouteOnAttribute 1.8.0 processor

2020-03-25 Thread Matt Burgess
Cody, ExecuteSQL outputs data in Avro format and embeds the schema in the content of the flow file (which Avro supports), leading to a non-zero flowfile size. As of NiFi 1.8.0, you can instead use ExecuteSQLRecord to output the data in any supported format (via a RecordSetWriter controller service

Re: [VOTE] Release Apache NiFi Registry 0.6.0

2020-04-03 Thread Matt Burgess
+1 (binding) Ran through release helper, verified artifacts, ran through Drew's "Getting Started With Apache NiFi Registry" tutorial (which is great btw) to connect NiFi to the Registry and run a few scenarios. Thanks for RM'ing Arpad! On Thu, Apr 2, 2020 at 11:54 AM Arpad Boda wrote: > > Hello

Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3

2024-05-01 Thread Matt Burgess
One thing that was mentioned was an included Jira for "providing a clearer distinction between framework and extensions," This involved moving extensions to a new module and for community folks this is causing problems with any new improvements in-flight before that. Seems like it needed more anno

  1   2   3   4   5   6   >