@Slim I have copied the image to Google Docs and hope to work fine. https://docs.google.com/document/d/1K_JE_a47bu1I7va1GwUK36vdOKZGqGFWTt4qPuTRShg/edit?usp=sharing
Slim Bouguerra <slim.bougue...@gmail.com> 于2019年4月26日周五 上午12:13写道： > Hey sorry your image is not showing? Not sure why. > > On Wed, Apr 24, 2019 at 6:53 AM PengHui Li <codelipeng...@gmail.com> > wrote: > > > Sorry for so long to reply, > > > > I drew a simple picture, hope can help for the question. > > The main point is to reduce the read of messages from unnecessary topics > > while read data from partitioned table of hive. > > [image: image.png] > > > > Slim Bouguerra <bs...@apache.org> 于2019年4月20日周六 上午12:16写道： > > > >> Hi am not sure am getting the question 100% Can you share a design doc > or > >> outline the big picture in your mind? FYI am not very familiar with > Pulsar > >> thus please account for that :D > >> But let me point out that Hive does not have the notion of partitions > for > >> tables backed by storage handlers, that is because by definition the > table > >> is not stored by Hive therefore can not control the layout. > >> > >> Will be happy to look at any POC. > >> looking forward to hear from you. > >> > >> On Wed, Apr 17, 2019 at 7:25 PM PengHui Li <codelipeng...@gmail.com> > >> wrote: > >> > >> > @Slim > >> > > >> > I want to use different pulsar topic to store data for different hive > >> > partition. Is there a way to do this, or does this idea make sense? > >> > > >> > Can you give me some advice? > >> > > >> > > >> > 李鹏辉gmail <codelipeng...@gmail.com> 于2019年4月15日周一 下午6:22写道： > >> > > >> > > I already have a simple implementation that can write data and query > >> > data. > >> > > I read the design document and implementation of kafka. > >> > > There are some differences of table partition with what I think. > >> > > > >> > > I want hive table partition locations work with pulsar topics. > >> Different > >> > > table partitions correspond to different topics. > >> > > But i can’t get the partition where the data will be written. > >> > > > >> > > I know that the drawback of doing this is that it will lose the > order > >> of > >> > > the stream data itself. > >> > > But can reduce unnecessary data reading when querying. > >> > > > >> > > Best Regards > >> > > > >> > > Penghui > >> > > Beijing,China > >> > > > >> > > > >> > > > >> > > > 在 2019年4月13日，21:43，Jörn Franke <jornfra...@gmail.com> 写道： > >> > > > > >> > > > I think you need to develop a custom hiveserde + custom > >> > > Hadoopinputformat + custom Hiveoutputformat > >> > > > > >> > > >> Am 12.04.2019 um 17:35 schrieb 李鹏辉gmail <codelipeng...@gmail.com > >: > >> > > >> > >> > > >> Hi guys, > >> > > >> > >> > > >> I’m working on integration of hive and pulsar recently. But now i > >> have > >> > > encountered some problems and hope to get help here. > >> > > >> > >> > > >> First of all, i simply describe the motivation. > >> > > >> > >> > > >> Pulsar can be used as infinite streams for keeping both historic > >> data > >> > > and streaming data, So we want to use pulsar as a storage extension > >> for > >> > > hive. > >> > > >> In this way, hive can read the data in pulsar naturally, and can > >> also > >> > > write data into pulsar. > >> > > >> We will benefit from the same data that provides both interactive > >> > query > >> > > and streaming capabilities. > >> > > >> > >> > > >> As an improvement, support data partitioning can make the query > >> more > >> > > efficient(e.g. partition by date or any other field). > >> > > >> > >> > > >> But > >> > > >> > >> > > >> - how to get hive table partition definition? > >> > > >> - While user inert data to hive table, how to get partition the > >> data > >> > > should be store? > >> > > >> - While use select data from hive table, how to determine data is > >> in > >> > > that partition? > >> > > >> > >> > > >> If hive already expose some mechanism to support, please show me > >> how > >> > to > >> > > use it. > >> > > >> > >> > > >> Best regards > >> > > >> > >> > > >> Penghui > >> > > >> Beijing, China > >> > > >> > >> > > >> > >> > > >> > >> > > > >> > > > >> > > >> > > -- > > B-Slim > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______ >
