Hello Phoenix-Users, I was wondering what recommendations there are around continuously pushing data into Phoenix from a Java-program. We have highly varying ingestion rates, often < 10 / second but sometimes spiking up considerably to ~ 10k-15k for short periods of time.
Hence our current approach in our loading-application is to use a buffer and then by time and amount (so like max 1sec or 1000 UPSERT's, whichever comes first) send a commit. So in essence we're doing PreparedStatements and execute() all the time and manual commit control. This proved much faster than auto-commit (which should in theory be the way to go given the nature of the implementation) which only performed at a few hundred upserts per second peak. The current approach is in the thousands but never 5-digit against 3 datanodes backed by SSD's (!!). To me it seems-- after digging in the Phoenix-Code -- it is stacking up the modifications in-memory "again" (in contrast to just sending it off) and then doing a batch-execution which makes sure the transaction level in [1] is met. [1] http://phoenix.apache.org/index.html#transactions Any recommendations how to improve this? Would JDBC Transaction-Batch be better? Regards Dominik
