PutElasticsearchHttpRecord writing null values

2018-02-26 Thread Mike Thomsen
I have a client that uses that processor and doesn't like the fact that null record fields get sent to ElasticSearch instead of excluded from the payload. Does anyone know if there is a good reason to keep that behavior? It doesn't seem to add anything except line noise when looking at the results

Re: PutElasticsearchHttpRecord writing null values

2018-02-27 Thread Mike Thomsen
ave the option to > > supress nulls. I've been meaning to share this back. This way one can > set > > how to treat nulls just like the jsonrecordsetwriter. > > > > Robert > > > > > > On Mon, Feb 26, 2018, 07:31 Mike Thomsen wrote: > > > >

Re: A question to copy files to other folders

2018-02-28 Thread Mike Thomsen
That seems like a pretty easy thing to fix with PutFile. Could be done with a patch to add an attribute that provides a relative path. On Wed, Feb 28, 2018 at 7:00 AM, wrote: > Hi, team, > > I'm writing a data flow template to copy files from directory A to other > two directories B and C > > Th

Re: PutElasticsearchHttpRecord writing null values

2018-03-01 Thread Mike Thomsen
t > > how to treat nulls just like the jsonrecordsetwriter. > > > > Robert > > > > > > On Mon, Feb 26, 2018, 07:31 Mike Thomsen wrote: > > > >> I have a client that uses that processor and doesn't like the fact that > >> null record fields get sent to E

Re: Opinion wanted: Cyber orchestration "distro"

2018-03-06 Thread Mike Thomsen
I'm also free to review the bundle if you need help. I also don't mind helping with the grunt work of reshaping it into a bundle that can be used with NiFi proper instead of as a special distro. On Tue, Mar 6, 2018 at 12:17 PM, Joe Witt wrote: > Andre, > > You're definitely free to do such a thi

Re: Is it possible to deploy NiFi to an WildFly app server ?

2018-03-07 Thread Mike Thomsen
Most of NiFi isn't a JavaEE application, so I don't see how you could do more than maybe get part of the web UI working in WildFly. On Wed, Mar 7, 2018 at 6:36 AM, Александр wrote: > We have a requirement to run NiFi in an only approved WildFly app > server. Is it possible to do that? > > Thanks

Patch for HBase visibility labels

2018-03-07 Thread Mike Thomsen
It's a big patch, but this should be a big boost for NiFi's HBase integration. https://github.com/apache/nifi/pull/2518 Any reviews would be appreciated. It also adds a new processor, DeleteHBaseCells to enable granular deletes (a requirement with visibility labels in HBase). No setup is require

Re: Custom controller service

2018-03-08 Thread Mike Thomsen
Varun, You didn't say what type of database is it. Can you share that info? On Thu, Mar 8, 2018 at 1:15 AM, varun yadav wrote: > Hii, > > I am trying to create a custom controller service for connection to a local > database. > I am getting so many issues in that. If you can help with a few doc

NiFi build failed, but got reported as a success

2018-03-09 Thread Mike Thomsen
Log: https://api.travis-ci.org/v3/job/351552597/log.txt Build list: https://travis-ci.org/apache/nifi/builds/351552596?utm_source=github_status&utm_medium=notification They're all green, but the build failed.

Re: [discuss] nifi 1.6.0 release

2018-03-12 Thread Mike Thomsen
If we could get this patch for HBase visibility labels merged before, I'd appreciate it. https://github.com/apache/nifi/pull/2518 If anyone else has patches they'd like to see merged, feel to ping me on GitHub, and I'll help out with a code review. Thanks, Mike On Mon, Mar 12, 2018 at 12:56 PM

Re: [discuss] nifi 1.6.0 release

2018-03-12 Thread Mike Thomsen
tributer / review agreement. > > > > -- Another Mike > > > > On Mon, Mar 12, 2018 at 1:28 PM, Mike Thomsen > > wrote: > > > >> If we could get this patch for HBase visibility labels merged before, > I'd > >> appreciate it. > >> &g

Re: PutMongo 1.3.0 - Upsert issue

2018-03-16 Thread Mike Thomsen
I noticed that your update document didn't have an update key in it. The default lookup key is _id so you'd need something like this: { "_id": "ID_HERE", "$set": { "new_field": 1 } } On Wed, Jan 10, 2018 at 6:05 AM, fabe_bdx wrote: > Dear Pierre, > > First all many thnaks for

Re: Finding Performance bottleneck issue

2018-03-16 Thread Mike Thomsen
Milan, Can you share some details about where you are running into problems? Like a basic description of what it's trying to do? On Fri, Mar 16, 2018 at 10:39 AM, Milan Das wrote: > I have a custom processor, it works as expected. But I feel there is some > performance measure need to be done.

Re: Finding Performance bottleneck issue

2018-03-16 Thread Mike Thomsen
relate this event with > the corresponding user logon event as well as to any other security audit > events generated during this logon session. > > > > Regards, > Milan Das > > > On 3/16/18, 10:56 AM, "Mike Thomsen" wrote: > > Milan, > >

Re: [ANNOUNCE] New Apache NiFi Committer Mike Thomsen

2018-03-21 Thread Mike Thomsen
Thanks everyone! On Wed, Mar 21, 2018 at 9:55 AM, Joe Witt wrote: > Mike > > Thanks for all the great contributions and reviews and discussions and > congratulations on the well deserved commit bit! > > Thanks > Joe > > On Wed, Mar 21, 2018 at 1:53 PM, Kevin Doran wrote: > > Congrats, Mike! > >

Java 10

2018-03-21 Thread Mike Thomsen
Looks like Oracle just dropped Java 10 on us: http://www.oracle.com/technetwork/java/javase/downloads/jdk10-downloads-4416644.html Announcement for anyone interested: http://mail.openjdk.java.net/pipermail/announce/2018-March/000247.html

Re: SOAP Service through InvokeHTTP

2018-03-24 Thread Mike Thomsen
Been a while since I've done anything with SOAP, but the way I remember it is it's just a POST operation with a XML payload at the end of the day. If you have configured the endpoint properly in the processor's URL property and set it to do a POST, the *flowfile content* should be where you put the

Weird issue w/ Redis pool service in an integration test

2018-03-26 Thread Mike Thomsen
Over the weekend I started playing around with a new processor called PutRedisHash based on a request from the user list. I set up a really simple IT and hit a problem pretty quickly. This passes validation: @Test public void testStandalone() throws Exception { final String attrName = "key.nam

Re: Weird issue w/ Redis pool service in an integration test

2018-03-26 Thread Mike Thomsen
https://github.com/MikeThomsen/nifi/commit/dad71af803a9ead5bcbead3ce65671c4c2430605#diff-09d6acf9b469f7d2491ab7dbdab93fea On Mon, Mar 26, 2018 at 11:38 AM, Bryan Bende wrote: > Can you share the code for your AbstractRedisProcessor? > > > On Mon, Mar 26, 2018 at 9:52 AM, Mike Thom

Re: Weird issue w/ Redis pool service in an integration test

2018-03-26 Thread Mike Thomsen
s to create > a connection, rather than using the RedisConnectionPool to obtain a > connection. > > On Mon, Mar 26, 2018 at 11:38 AM, Bryan Bende wrote: > > Can you share the code for your AbstractRedisProcessor? > > > > > > On Mon, Mar 26, 2018 at 9:52 AM,

Re: Class Loading Conflicts - Different JAR Versions

2018-03-26 Thread Mike Thomsen
I was just following a convention at the time. So unless something breaks because of the move, I don't see any reason why it would be an issue to move it. In fact, with NIFI-4325 the only reason I put the new ElasticSearch service/api in the standard-services segment of the code base was I ran int

Re: Class Loading Conflicts - Different JAR Versions

2018-03-27 Thread Mike Thomsen
uery performance issues and concurrency issues > for > >> the data flow if we're trying to overwrite an entire collection at once. > >> That said, if I'm missing something, I'm happy to learn! > >> > >> Thanks again, > >> Brian > >&

Re: Read processor property in init()

2018-03-28 Thread Mike Thomsen
Just do... @OnScheduled public void setup(ProcessContext context) { //Read properties and do setup. } On Wed, Mar 28, 2018 at 8:57 AM, Jeff Zemerick wrote: > Hi everyone, > > Is there a recommended method for making user-configurable property > values available to a processor's init()? I wo

Anything like this been implemented yet?

2018-03-28 Thread Mike Thomsen
I have a client that would benefit from being able to run certain queries periodically. Like half a dozen or more. Is there any processor where you can associate a bunch of strings (like JSON) and send them out individually as flowfiles? Thanks, Mike

Re: Anything like this been implemented yet?

2018-03-28 Thread Mike Thomsen
ike a database. On Wed, Mar 28, 2018 at 5:02 PM, Mike Thomsen wrote: > I have a client that would benefit from being able to run certain queries > periodically. Like half a dozen or more. Is there any processor where you > can associate a bunch of strings (like JSON) and send them out ind

Re: Firefox - Slower movements when moving around the canvas

2018-04-03 Thread Mike Thomsen
I experienced it yesterday as well. I'm about to install a LTS version of Firefox so I can keep going there. While someone from the UI side figures this out, I think it should be filed as a bug report to Mozilla. On Sat, Mar 31, 2018 at 5:04 PM, Pierre Villard wrote: > Just confirmed that behav

Re: Firefox - Slower movements when moving around the canvas

2018-04-03 Thread Mike Thomsen
anything really relevant... > > [1] https://www.mozilla.org/en-US/firefox/59.0/releasenotes/ > [2] https://mozillagfx.wordpress.com/2017/12/05/off-main-thread-painting/ > > > > 2018-04-03 16:00 GMT+02:00 Mike Thomsen : > > > I experienced it yesterday as well. I'm about

Re: Firefox - Slower movements when moving around the canvas

2018-04-03 Thread Mike Thomsen
Found on Reddit that you can toggle OMTP using this: layers.omtp.enabled Disabled it and restarted. Didn't seem to do any good. On Tue, Apr 3, 2018 at 3:08 PM, Mike Thomsen wrote: > Yeah, I'm using a Mac. > > On Tue, Apr 3, 2018 at 2:34 PM, Pierre Villard < > pierre.vi

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

2018-04-06 Thread Mike Thomsen
Sorry to be the bearer of bad news, but this morning I found a bug in PutMongo that breaks insert mode. PR here: https://github.com/apache/nifi/pull/2609 On Fri, Apr 6, 2018 at 5:14 AM, Koji Kawamura wrote: > +1 (binding) > > Ran through the release helper steps. > Confirmed example flows inclu

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

2018-04-06 Thread Mike Thomsen
gess wrote: > Mike, > > Is there a workaround for this? Like setting the query key(s) for > insert even if they are ignored? Or will that cause other issues? > > Thanks, > Matt > > > On Fri, Apr 6, 2018 at 8:56 AM, Mike Thomsen > wrote: > > Sorry to b

Twitter API changes

2018-04-06 Thread Mike Thomsen
Anyone know if this impacts GetTwitter? https://9to5mac.com/2018/04/06/twitter-api-breaking-tweetbot-twitterrific-features/ Thanks, Mike

Programmatically building a nested schema

2018-04-08 Thread Mike Thomsen
Can someone tell me how to build a nested schema using RecordSchema and RecordField objects? The use case here is to make MongoDBLookupService able to roughly detect the schema from the returned bson Document. Thanks, Mike

Re: [ANNOUNCE] Apache NiFi 1.6.0 release

2018-04-09 Thread Mike Thomsen
Any word on when the Docker repository is going to be updated? Thanks, Mike On Sun, Apr 8, 2018 at 10:46 PM, Joe Witt wrote: > Hello > > The Apache NiFi team would like to announce the release of Apache NiFi > 1.6.0. > > Apache NiFi is an easy to use, powerful, and reliable system to > process

Re: Host a web service on Nifi to be able to consume by third party systems

2018-04-09 Thread Mike Thomsen
Not really. The embedded Jetty container is meant to host NiFi web components, not third party apps. There's no real reason to deploy a WAR file there because you could just spin up Tomcat, drop the SOAP service there and have them both accessible through a reverse proxy. Thanks, Mike On Mon, Ap

Re: [ANNOUNCE] Apache NiFi 1.6.0 release

2018-04-09 Thread Mike Thomsen
I think the latest tag is not updated. I just switched from pull apache/nifi:latest to nifi:1.6.0 and got it. On Mon, Apr 9, 2018 at 8:15 AM, Pierre Villard wrote: > Mike, > > It's up on my side: https://hub.docker.com/r/apache/nifi/builds/ > > Pierre > > 2018-04-09 12

Re: Signing AWS Elasticsearch Requests?

2018-04-09 Thread Mike Thomsen
Jon, In 1.6, we added a new client service for ES. There's only one processor that uses it for now, JsonQueryElasticSearch, but I have more under way that I hope will make it into 1.7. If you take a look at the existing version of the client service implementation that uses the official ElasticSea

Re: Signing AWS Elasticsearch Requests?

2018-04-09 Thread Mike Thomsen
Related, interesting take on AWS ES: https://code972.com/blog/2017/12/111-why-you-shouldnt-use-aws-elasticsearch-service I saw some other critical posts that went into detail on other issues. Considering how easy it is to stand up ES yourself, I think it would be worthwhile to strongly consider g

Re: beginner

2018-04-10 Thread Mike Thomsen
It adds an attribute to the flowfile called text.count.line for line counts. Thanks, Mike On Tue, Apr 10, 2018 at 10:01 AM, vivek wrote: > while working with the line count processer it was not performming the > task > it is ternsfering the file to the destination can any one help me to do >

Re: Signing AWS Elasticsearch Requests?

2018-04-10 Thread Mike Thomsen
ier to use than the previous one. > Is this going to eventually be used by the existing processors (ex. > FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a > TransportClient implementation and merge the FetchEs FetchEsHttp and > FetchEs5 (and put, etc) processor

Re: Data migration in sequence

2018-04-12 Thread Mike Thomsen
It'd be very helpful to know what the input and output processors are for each of the flows because they have different capabilities for triggering them. On Thu, Apr 12, 2018 at 12:03 PM Anil Rai wrote: > sorry, sent email in between. > This can be achieved by putting few flags in cache and chec

Re: Data migration in sequence

2018-04-12 Thread Mike Thomsen
is easy. > > If it is #2 then I believe you'd want a series of Wait/Notify > > processors where B waits the notification signal from A, C from B, and > > so on. > > > > Thanks > > > > On Thu, Apr 12, 2018 at 4:16 PM, Mike Thomsen > > wrote: >

Re: Ongoing support for a contribution

2018-04-12 Thread Mike Thomsen
Anthony, I think monitoring those lists would be a great idea. In the interim, if you could get a Docker image up on Docker Hub that allows members of the community to casually spin up a working MarkLogic instance suitable for kicking the tires with your processor(s) that would be a big help. Als

Proposal: standard record metadata attributes for data sources

2018-04-13 Thread Mike Thomsen
I'd like to propose that all non-deprecated (or likely to be deprecated) Get/Fetch/Query processors get a standard convention for attributes that describe things like: 1. Source system. 2. Database/table/index/collection/etc. 3. The lookup criteria that was used (similar to the "query attribute" s

Re: Error with getSolr

2018-04-14 Thread Mike Thomsen
Can you look at the logs and provide a stacktrace? Thanks, Mike On Sat, Apr 14, 2018 at 8:25 AM misterpilou < pierrelouis.chevallie...@gmail.com> wrote: > Hi everyone, > > I'm discovering nifi and using GetSolr, i fix some errors but one remain: > > org.apache.nifi.schema.access.SchemaNotFoundE

Re: Unit test for database Put processor

2018-04-16 Thread Mike Thomsen
The PutMongo tests fail because the Mongo processors haven't been fully migrated to the new method of declaring EL support. With that said, the Mongo tests aren't almost entirely integration tests, not unit tests because Mongo is kinda painful to mock. On Mon, Apr 16, 2018 at 5:50 PM Anthony Roach

Re: Unit test for database Put processor

2018-04-16 Thread Mike Thomsen
test with the way Maven is configured here. The integration tests will not run unless you do *mvn integration-test -Pintegration-tests*. Thanks, Mike On Mon, Apr 16, 2018 at 6:00 PM Mike Thomsen wrote: > The PutMongo tests fail because the Mongo processors haven't been fully > mig

Re: The import org.apache.nifi.dbcp.DBCPService cannot be resolved

2018-04-17 Thread Mike Thomsen
Did you declare a dependency on nifi-dbcp-service-api? If not, you have to do that and set the scope to "provided. On Tue, Apr 17, 2018 at 7:05 AM Rishab Prasad wrote: > Hi, > > I am trying to create a custom processor equivalent to that of > ListDatabaseTable. I copied and pasted the source cod

Re: Nifi Custom Processor Import Error

2018-04-17 Thread Mike Thomsen
It was added after 1.5. Might be on 1.6. Definitely in 1.7.0-Snapshot. On Tue, Apr 17, 2018 at 4:56 PM Brendan Rebuck wrote: > I am currently working on creating a custom processor. I have library > downloaded and referenced for nifi-api-1.5.0.jar. > > When I use the "import org.apache.nifi.expre

Re: The import org.apache.nifi.expression.ExpressionLanguageScope cannot be resolved.

2018-04-18 Thread Mike Thomsen
That particularly class is in 1.7.0-SNAPSHOT, not 1.5.0. You'd have to download the code from GitHub and build it to get that version. If you're writing a custom NAR, you don't need to worry about that until you move to 1.7.0. On Wed, Apr 18, 2018 at 10:07 AM RITIKA NIWAS wrote: > I am getting a

FoundationDB and local jars

2018-04-21 Thread Mike Thomsen
Yesterday, I started looking at building CRUD processors for FoundationDB. Only problem I found is that their Java client is not mavenized. So my question to the rest of the team is, should I hold off until they mavenize it or should I just build a FoundationDB bundle and exclude it from the normal

Re: How can I pass a flowfile attribute to a controller service?

2018-04-23 Thread Mike Thomsen
Rishab, > Also, is there a way where we can postpone the evaluation of expression language used til the processor starts running? No because the controller service is a dependency of the processor and evaluating the expression language in the CS property descriptors must be done when enabling the

Re: Recommended hardware specs to run sandbox environments on workstation

2018-04-23 Thread Mike Thomsen
Depending on what you want to do with them, you can run NiFi, Kafka and Storm together with about 4GB-5GB of RAM and get good results locally. For NiFi and Kafka, the Docker images will be the safest bet. You'll also need to factor in the memory for running ZooKeeper if you want that, but about 256

Re: Custom Controller Service

2018-04-25 Thread Mike Thomsen
Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad wrote: > Hi, >

Re: Is there a configuration to limit the size of nifi's flowfile repository

2018-04-25 Thread Mike Thomsen
I have a feeling that what Ben meant was how to limit the content repository size. On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard wrote: > Hi Ben, > > Since the flow file repository contains the information of the flow files > currently being processed by NiFi, you don't want to limit that repos

Re: JSON Path Expression

2018-04-27 Thread Mike Thomsen
The JsonPath processor uses an older version of the Java library. I think it's v2.0. Have you tried that against the latest version of the Java lib? I know there's at least one JsonPath feature the version we use doesn't support. On Fri, Apr 27, 2018 at 3:14 PM Anil Rai wrote: > Experts, > > Inp

Re: JSON Path Expression

2018-04-27 Thread Mike Thomsen
r than > evaluatejsonpath?) > > Thanks > Anil > > > On Fri, Apr 27, 2018 at 3:37 PM, Mike Thomsen > wrote: > > > The JsonPath processor uses an older version of the Java library. I think > > it's v2.0. Have you tried that against the latest version of t

Re: Updating / Adding LICENSE and NOTICE entries for new processors

2018-04-27 Thread Mike Thomsen
Anthony, Related to that, have you verified that your dependencies are all using licenses that can be merged into an ASF-sponsored project? The link below contains the ASF's legal guidance on the subject: https://www.apache.org/legal/resolved.html I strongly recommend doing a deep dive through y

Re: Nifi Development Question - Asynchronous Writes

2018-05-01 Thread Mike Thomsen
Vivek, To add to what Bryan said, if you send a batch to MarkLogic and there is a failure the standard pattern would be to send all of the flowfiles that were part of the batch to a failure relationship. Depending on the error, you might want to also support a "retry" relationship for situations l

Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

2018-05-02 Thread Mike Thomsen
GetMongo can also use the body of a flowfile for the query. That is its new default behavior if you leave the query field blank and have an incoming connection from another processor. That would be a good way to integrate the flow with another application. For example, you could add FetchKafka to t

Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

2018-05-02 Thread Mike Thomsen
me? > > > Brajendra Mishra > Persistent Systems Ltd. > > -Original Message- > From: Mike Thomsen > Sent: Wednesday, May 02, 2018 5:58 PM > To: dev@nifi.apache.org > Subject: Re: GetMongoDB : How to pass parameters as input to GetMongoDB > processor > > GetMongo ca

Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

2018-05-03 Thread Mike Thomsen
; > Persistent Systems Ltd. > > > > *From:* Mike Thomsen > *Sent:* Wednesday, May 02, 2018 6:24 PM > > > *To:* dev@nifi.apache.org > *Subject:* Re: GetMongoDB : How to pass parameters as input to GetMongoDB > processor > > > > That might require 1.6.0.

Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

2018-05-03 Thread Mike Thomsen
; > I did attach the same in my previous mail. Well reattaching it again. > > Well error is at GetMongoDB Processor and error text is : "Upstream > > Connections is invalid because Processor does not allow upstream > > connections but currently has 1" > > >

MarkLogic code review

2018-05-05 Thread Mike Thomsen
https://github.com/apache/nifi/pull/2671 If anyone else wants to jump in, their PR is up for review now. Instructions on how to build a dockerized MarkLogic node: http://developer.marklogic.com/blog/building-a-marklogic-docker-container

Re: NiFi REST APIs : How to create flows through 'nifi-api'

2018-05-08 Thread Mike Thomsen
Brajendra, The NiFi API is typically called from external clients, not from within a flow. With that said, as far as I understand your use case I don't think it's going to work out that well for you. What you seem to want to do here is programatically create new processors to handle new situations

ASF -> GitHub sync

2018-05-11 Thread Mike Thomsen
I approved one of Matt's PRs and merged, but it seems to be taking unusually long for the sync to GitHub. Is something broken?

Re: ASF -> GitHub sync

2018-05-11 Thread Mike Thomsen
No, it was probably 5k-10kb of updates. On Fri, May 11, 2018 at 8:59 AM Marc Parisi wrote: > I've seen it get stuck only to unstick with another merge > > On Fri, May 11, 2018, 8:50 AM Mike Thomsen wrote: > > > I approved one of Matt's PRs and merged, but it seem

Graph database support w/ NiFi

2018-05-12 Thread Mike Thomsen
I was wondering if anyone on the dev list had given much thought to graph database support in NiFi. There are a lot of graph databases out there, and many of them seem to be half-baked or barely supported. Narrowing it down, it looks like the best candidates for a no fuss, decent sized graph that w

Re: Proposal: standard record metadata attributes for data sources

2018-05-13 Thread Mike Thomsen
like when a data set was loaded into NiFi, how many records went through a terminating processor, etc. IMO this could help batch-oriented organizations feel more at ease with something stream-oriented like NiFi. On Fri, Apr 13, 2018 at 4:01 PM Mike Thomsen wrote: > I'd like to propose that

Re: Proposal: standard record metadata attributes for data sources

2018-05-14 Thread Mike Thomsen
> Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On May 13, 2018, at 11:48 AM, Mike Thomsen wrote: > > @Joe @Matt > > This is kinda related to the point that Joe made in the graph DB thread

Can we close this one?

2018-05-14 Thread Mike Thomsen
https://github.com/apache/nifi/pull/2672 The test API doesn't like evaluateExpressionLanguage(FlowFile) getting passed a null flowfile, but the framework doesn't care if that happens when NiFi is running. It handles the evaluation just fine. Any objections to me closing this out? Thanks, Mike

Re: Handling an error and sys.exit from python in ExecuteScript

2018-05-16 Thread Mike Thomsen
I think what you'd want to do is something like this: session.yield() return On Wed, May 16, 2018 at 11:03 AM James McMahon wrote: > Hello. I am using an ExecuteScript processor to run a python script in the > NiFi jython environment. Early in my script I try to get user and group > info from m

Sync is stuck again

2018-05-16 Thread Mike Thomsen
Mark pushed a big commit and it looks like GitHub is stuck on the sync. Wanted to let anyone doing reviews know.

Re: Sync is stuck again

2018-05-16 Thread Mike Thomsen
I reviewed NIFI-2966 and that cleared it up. On Wed, May 16, 2018 at 4:45 PM Suneel Marthi wrote: > Try making a commit with a white space - that should usually normalize > things ? > > On Wed, May 16, 2018 at 4:17 PM, Mike Thomsen > wrote: > > > Mark pushed a big

A user's experience with a bake off

2018-05-21 Thread Mike Thomsen
https://statsbot.co/blog/open-source-etl/ Someone on my team shared that. Overall, it's a very fair take. I think it has some direct action items, particularly on the UX side that could be considered for 1.8 to close some gaps. (Not trying to start a flame war here, as once again I think it's a f

Re: Put data to Elastic with static settings or index template

2018-05-22 Thread Mike Thomsen
Bobby, You need to use ElasticSearch templates for this. https://www.elastic.co/guide/en/elasticsearch/reference/5.4/indices-templates.html That's the official Elastic-sanctioned way of doing this. On Tue, May 22, 2018 at 4:23 AM Koji Kawamura wrote: > Hi Bobby, > > Elasticsearch creates inde

Re: A user's experience with a bake off

2018-05-22 Thread Mike Thomsen
> well. > > Thanks > > On Mon, May 21, 2018 at 4:17 PM, Mike Thomsen > wrote: > > https://statsbot.co/blog/open-source-etl/ > > > > Someone on my team shared that. Overall, it's a very fair take. I think > it > > has some direct action items, particu

Re: Proposal: standard record metadata attributes for data sources

2018-05-24 Thread Mike Thomsen
RK event could be used and a new flowfile > with > > the relevant attributes/content could be created. The flowfiles would be > > linked, but the “sensitive” information wouldn’t travel with the > original. > > > > Andy LoPresto > > alopre...@apache.org > &

Re: Accumulo processors

2018-05-27 Thread Mike Thomsen
Dave, I don't know how far along Mark's code is, but you might find some useful code you can borrow from the HBase commit(s) that added visibility label support. For example, there is code for handling visibility labels w/ PutHBaseRecord that might be reusable if you want to create a put record pr

Re: Accumulo processors

2018-05-27 Thread Mike Thomsen
; I'm not really familiar with Accumulo, I am currently reading the docs on > the apache web site. > Dave > > > On Sunday, 27 May 2018, 21:25, Mike Thomsen > wrote: > > > Dave, > > I don't know how far along Mark's code is, but you might find

Re: rat failures afte taking master?

2018-05-31 Thread Mike Thomsen
Should be resolved now. It got added to the exclude list for standard-processors. On Thu, May 31, 2018 at 8:56 AM Otto Fowler wrote: > [WARNING] Files with unapproved licenses: > > /home/travis/build/ottobackwards/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resou

Using EL classes with just java Map objects

2018-06-02 Thread Mike Thomsen
I tried working with the EL package's Query object to try building something like this: def evaluate(String query, Map coordinates) { def compiled = Query.compile(query) compiled.evaluate(coordinates) } Which for [ name: "John Smith" ] and query '${name}-${name:length()}' I expected would

Re: Using EL classes with just java Map objects

2018-06-02 Thread Mike Thomsen
gt; I didn't try this out, and it's very possible my assumptions are not > spot-on, so if these don't work let me know and I'll take a closer > look. > > Regards, > Matt > > > > > On Sat, Jun 2, 2018 at 6:30 PM, Mike Thomsen > wrote: > > I

Re: Using EL classes with just java Map objects

2018-06-02 Thread Mike Thomsen
.compile("${name}").evaluate(coordinates) + "-" + >> Query.compile("${name:length()}") >> >> I didn't try this out, and it's very possible my assumptions are not >> spot-on, so if these don't work let me know and I'll take a

Travis is broken?

2018-06-04 Thread Mike Thomsen
It says it cannot find repo apache/nifi. Anyone know what's happening?

Re: [ANNOUNCE] New Apache NiFi Committer Sivaprasanna Sethuraman

2018-06-05 Thread Mike Thomsen
Congratulations, Sivaprasanna! On Tue, Jun 5, 2018 at 10:22 AM Bryan Bende wrote: > Congrats! and thank you for your contributions to the NiFi community. > > On Tue, Jun 5, 2018 at 10:16 AM, Kevin Doran wrote: > > Congrats, Sivaprasanna! > > > > On 6/5/18, 10:09, "Tony Kurc" wrote: > > > >

Re: [ANNOUNCE] New NiFi PMC member Mike Thomsen

2018-06-07 Thread Mike Thomsen
gt; > > > Congratulations Mike! Great to have you aboard, I appreciate your > > > > > commitment and work ethic and looking forward to continued great > > > > > things! > > > > > > > > > > On Wed, Jun 6, 2018 at 6:22 PM, Tony Kurc >

Re: Java 10

2018-06-07 Thread Mike Thomsen
Once that is done, it should run on Java 10. However, under the new Oracle version rules Java 10 is a short term iteration. Java 11 will probably be the next version officially supported by NiFi because it's supposed to be Java's next LTS release. Java 10 features like the var key will also be rej

Re: Java 10

2018-06-07 Thread Mike Thomsen
h Java 9 features be rejected until then? (ie. preserving > Java-8 compatibility) Is there a plan for supporting multiple Java > versions? > > On Thu, Jun 7, 2018 at 10:18 AM, Mike Thomsen > wrote: > > > Once that is done, it should run on Java 10. However, under the new > Or

LookupService + flowfile attributes

2018-06-08 Thread Mike Thomsen
On the RestLookupService PR I think Koji mentioned the idea of expanding the lookup capability to include flowfile attributes. That sort of thing would be immensely useful on two PRs I have already open for lookup service changes for ES and Mongo. Koji, add your thoughts, but what I'm thinking woul

Re: LookupService + flowfile attributes

2018-06-09 Thread Mike Thomsen
ld be FlowFile attributes, then dynamic properties, > so that user can overwrite attribute values when necessary. > > Koji > > > On Sat, Jun 9, 2018 at 1:06 AM, Mike Thomsen > wrote: > > On the RestLookupService PR I think Koji mentioned the idea of expanding > > the

Re: [DISCUSS] Release candidate timeframe for Apache NiFi 1.7.0

2018-06-09 Thread Mike Thomsen
some comments) > > >>> * NIFI-5268 JoltTransformJSON specification property fails with EL — > > Koji > > >>> Kawamura > > >>> * NIFI-5266 PutElasticsearchHttp processors should sanitize > parameters > > — > > >>> Matt Burgess >

Should the elastic search client service impl get renamed before 1.7?

2018-06-10 Thread Mike Thomsen
The current implementation uses the last stable release of the 5.X client from Elastic, and 6.X is already mature so we'll need to be able to have room to create a new implementation copy that uses that client if there are things we have to change between them. So does it make sense to throw in a n

Re: Should the elastic search client service impl get renamed before 1.7?

2018-06-10 Thread Mike Thomsen
7;t introduce any breaking > changes, we can continue with the name that we are having now. > > - > Sivaprasanna > > On Sun, Jun 10, 2018 at 5:16 PM, Mike Thomsen > wrote: > > > The current implementation uses the last stable release of the 5.X client > > from Elast

Re: Why "ExecuteSQL" processor serving DELETE and UPDATE SQL queries.

2018-06-11 Thread Mike Thomsen
If you really need to restrict the commands that a particular user can execute, it should be done via the database users that NiFi's services are using. In 1.7, there's a new database service coming along that wraps two or more database pools. It could probably be used in scenarios like this to ass

Re: Should the elastic search client service impl get renamed before 1.7?

2018-06-11 Thread Mike Thomsen
7 that deals with 7. It makes it look like the simply > named ElasticSearchClient is generic and will work with any version. This > is what we did with the PublishKafka* processors and it has only caused > confusion. > > Sent from my iPhone > > > On Jun 10, 2018, at 8:20

Re: Should the elastic search client service impl get renamed before 1.7?

2018-06-11 Thread Mike Thomsen
Should mention that I did not check the L&N for the ES 6.X version. That needs to be reviewed as part of the PR. On Mon, Jun 11, 2018 at 7:54 AM Mike Thomsen wrote: > https://github.com/apache/nifi/pull/2782 > > Renamed the service and added a clone for 6.X. The integration tests

Re: LookupService + flowfile attributes

2018-06-11 Thread Mike Thomsen
riteria by default. I'll update the PR accordingly and make the new method default to the existing one in all of the lookup services that are already there. On Sat, Jun 9, 2018 at 8:44 AM Mike Thomsen wrote: > https://issues.apache.org/jira/browse/NIFI-5287 > > On Sat, Jun 9, 2018

Re: LookupService + flowfile attributes

2018-06-12 Thread Mike Thomsen
com/apache/nifi/pull/2777#issuecomment-396512384 > > Thanks, > Koji > > On Tue, Jun 12, 2018 at 1:56 AM, Mike Thomsen > wrote: > > Koji, > > > > After reading Mark's comments on GitHub, it occurred to me that the > MongoDB > > lookup service and the

Transition Mongo processors to use a controller service

2018-06-15 Thread Mike Thomsen
I created a ticket for moving the Mongo processors to use the Mongo client service as their configuration source. https://issues.apache.org/jira/browse/NIFI-5239 For at least one release, it should be optional, but I'm not sure the best way to do that. Should I make the client service and the req

Re: Unable to bring up NiFi user interface

2018-06-15 Thread Mike Thomsen
Yeah, Java 9 support is slated for NiFi 1.7.0. On Fri, Jun 15, 2018 at 6:44 AM Otto Fowler wrote: > Running with Java 9 is not supported with 1.6.0 I believe. > > > > On June 15, 2018 at 06:32:27, Dave (dave7sw...@gmail.com) wrote: > > Hi, > > I am not sure if I can post my query here. If I am n

  1   2   3   4   5   6   7   >