[GitHub] kangpinghuang opened a new pull request #572: update rowset state to visible when set version

[GitBox]

kangpinghuang opened a new pull request #572: update rowset state to visible 
when set version
URL: https://github.com/apache/incubator-doris/pull/572
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chenhao7253886 opened a new pull request #573: Fix bug that recvr thread update sub plan's QueryStatistics when it i…

[GitBox]

chenhao7253886 opened a new pull request #573: Fix bug that recvr thread update 
sub plan's QueryStatistics when it i…
URL: https://github.com/apache/incubator-doris/pull/573
 
 
   …s destructed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] morningman opened a new pull request #574: Fix bug that heartbeat error msg may be null

[GitBox]

morningman opened a new pull request #574: Fix bug that heartbeat error msg may 
be null
URL: https://github.com/apache/incubator-doris/pull/574
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay merged pull request #573: Fix bug that recvr thread update sub plan's destructed QueryStatistics

[GitBox]

imay merged pull request #573: Fix bug that recvr thread update sub plan's 
destructed QueryStatistics
URL: https://github.com/apache/incubator-doris/pull/573
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] morningman merged pull request #574: Fix bug that heartbeat error msg may be null

[GitBox]

morningman merged pull request #574: Fix bug that heartbeat error msg may be 
null
URL: https://github.com/apache/incubator-doris/pull/574
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250110250
 
 

 ##
 File path: fe/src/main/java/org/apache/doris/http/meta/ColocateMetaService.java
 ##
 @@ -0,0 +1,281 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.meta;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.ColocateTableIndex;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.http.ActionController;
+import org.apache.doris.http.BaseRequest;
+import org.apache.doris.http.BaseResponse;
+import org.apache.doris.http.IllegalArgException;
+import org.apache.doris.http.rest.RestBaseAction;
+import org.apache.doris.http.rest.RestBaseResult;
+import org.apache.doris.http.rest.RestResult;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.persist.ColocatePersistInfo;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.lang.reflect.Type;
+
+/*
+ * the colocate meta define in {@link ColocateTableIndex}
+ */
+public class ColocateMetaService {
+private static final Logger LOG = 
LogManager.getLogger(ColocateMetaService.class);
+private static final String GROUP_ID = "group_id";
+private static final String TABLE_ID = "table_id";
+private static final String DB_ID = "db_id";
+
+private static ColocateTableIndex colocateIndex = 
Catalog.getCurrentColocateIndex();
+
+private static long checkAndGetGroupId(BaseRequest request) throws 
DdlException {
+long groupId = 
Long.valueOf(request.getSingleParameter(GROUP_ID).trim());
+LOG.info("groupId is {}", groupId);
 
 Review comment:
   remove this debug log?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250110434
 
 

 ##
 File path: fe/src/main/java/org/apache/doris/http/meta/ColocateMetaService.java
 ##
 @@ -0,0 +1,281 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.meta;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.ColocateTableIndex;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.http.ActionController;
+import org.apache.doris.http.BaseRequest;
+import org.apache.doris.http.BaseResponse;
+import org.apache.doris.http.IllegalArgException;
+import org.apache.doris.http.rest.RestBaseAction;
+import org.apache.doris.http.rest.RestBaseResult;
+import org.apache.doris.http.rest.RestResult;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.persist.ColocatePersistInfo;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.lang.reflect.Type;
+
+/*
+ * the colocate meta define in {@link ColocateTableIndex}
+ */
+public class ColocateMetaService {
+private static final Logger LOG = 
LogManager.getLogger(ColocateMetaService.class);
+private static final String GROUP_ID = "group_id";
+private static final String TABLE_ID = "table_id";
+private static final String DB_ID = "db_id";
+
+private static ColocateTableIndex colocateIndex = 
Catalog.getCurrentColocateIndex();
+
+private static long checkAndGetGroupId(BaseRequest request) throws 
DdlException {
+long groupId = 
Long.valueOf(request.getSingleParameter(GROUP_ID).trim());
+LOG.info("groupId is {}", groupId);
+if (!colocateIndex.isGroupExist(groupId)) {
+throw new DdlException("the group " + groupId + "isn't  exist");
+}
+return groupId;
+}
+
+private static long getTableId(BaseRequest request) throws DdlException {
+long tableId = 
Long.valueOf(request.getSingleParameter(TABLE_ID).trim());
+LOG.info("tableId is {}", tableId);
 
 Review comment:
   remove this log?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250110058
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   I'm not sure why we can't handle chunked encoding. "100-continue" has 
problem with `LoadAction`, what about chunked


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] morningman commented on issue #481: Doris' function should not be passed to mysql server

[GitBox]

morningman commented on issue #481: Doris' function should not be passed to 
mysql server
URL: https://github.com/apache/incubator-doris/issues/481#issuecomment-456730838
 
 
   And we also has a problem that when we are trying to query a mysql table 
using mysql builtin function, Doris will return "xxx can't support"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] morningman commented on issue #554: ERROR 1064 (HY000): convert_tzcan't support

[GitBox]

morningman commented on issue #554: ERROR 1064 (HY000): convert_tzcan't support
URL: https://github.com/apache/incubator-doris/issues/554#issuecomment-456730955
 
 
   Are you trying to query a mysql table using Doris?
   This may related to issue #481 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

kangkaisen commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250127735
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   `HttpObjectAggregator.beginAggregation` will remove chunked 
transfer-encoding header.
   
   `HttpObjectAggregator.finishAggregation`  will set content-length header.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

kangkaisen commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250128152
 
 

 ##
 File path: fe/src/main/java/org/apache/doris/http/meta/ColocateMetaService.java
 ##
 @@ -0,0 +1,281 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.meta;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.ColocateTableIndex;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.http.ActionController;
+import org.apache.doris.http.BaseRequest;
+import org.apache.doris.http.BaseResponse;
+import org.apache.doris.http.IllegalArgException;
+import org.apache.doris.http.rest.RestBaseAction;
+import org.apache.doris.http.rest.RestBaseResult;
+import org.apache.doris.http.rest.RestResult;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.persist.ColocatePersistInfo;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.lang.reflect.Type;
+
+/*
+ * the colocate meta define in {@link ColocateTableIndex}
+ */
+public class ColocateMetaService {
+private static final Logger LOG = 
LogManager.getLogger(ColocateMetaService.class);
+private static final String GROUP_ID = "group_id";
+private static final String TABLE_ID = "table_id";
+private static final String DB_ID = "db_id";
+
+private static ColocateTableIndex colocateIndex = 
Catalog.getCurrentColocateIndex();
+
+private static long checkAndGetGroupId(BaseRequest request) throws 
DdlException {
+long groupId = 
Long.valueOf(request.getSingleParameter(GROUP_ID).trim());
+LOG.info("groupId is {}", groupId);
+if (!colocateIndex.isGroupExist(groupId)) {
+throw new DdlException("the group " + groupId + "isn't  exist");
+}
+return groupId;
+}
+
+private static long getTableId(BaseRequest request) throws DdlException {
+long tableId = 
Long.valueOf(request.getSingleParameter(TABLE_ID).trim());
+LOG.info("tableId is {}", tableId);
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

kangkaisen commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250128131
 
 

 ##
 File path: fe/src/main/java/org/apache/doris/http/meta/ColocateMetaService.java
 ##
 @@ -0,0 +1,281 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.meta;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.ColocateTableIndex;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.http.ActionController;
+import org.apache.doris.http.BaseRequest;
+import org.apache.doris.http.BaseResponse;
+import org.apache.doris.http.IllegalArgException;
+import org.apache.doris.http.rest.RestBaseAction;
+import org.apache.doris.http.rest.RestBaseResult;
+import org.apache.doris.http.rest.RestResult;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.persist.ColocatePersistInfo;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.lang.reflect.Type;
+
+/*
+ * the colocate meta define in {@link ColocateTableIndex}
+ */
+public class ColocateMetaService {
+private static final Logger LOG = 
LogManager.getLogger(ColocateMetaService.class);
+private static final String GROUP_ID = "group_id";
+private static final String TABLE_ID = "table_id";
+private static final String DB_ID = "db_id";
+
+private static ColocateTableIndex colocateIndex = 
Catalog.getCurrentColocateIndex();
+
+private static long checkAndGetGroupId(BaseRequest request) throws 
DdlException {
+long groupId = 
Long.valueOf(request.getSingleParameter(GROUP_ID).trim());
+LOG.info("groupId is {}", groupId);
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangpinghuang opened a new pull request #575: realize rowset remove api

[GitBox]

kangpinghuang opened a new pull request #575: realize rowset remove api
URL: https://github.com/apache/incubator-doris/pull/575
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250137167
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   what problem will it cause that aggregator replaces chucked with 
content-length? 
   I think request body is aggregated and pass to handler, and it is OK.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

kangkaisen commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250144542
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   For stream load, there should be chunked transfer-encoding header, and the 
header should deliver to BE, we shouldn't remove the chunked transfer-encoding 
header.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250146626
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   I think if there is 100-continue header, FE returns redirect to client, then 
client will send request to BE, and the headers which be received is sent from 
client not from FE.
   
   And if there is no 100-continue header, FE will reject this request. 
   
   If all loading data is routed by FE, it is a huge load for it. So we need 
client must add "100-continue" header to request's header, and then FE can 
redirect the client to some BE to handle this load request


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

kangkaisen commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250159152
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   Yes. If we want to get request body, we need to aggregate it. but for the 
`LoadAction`, we shouldn't let FE handle the Load data. which is conflicting. 
So, I think we could make `DorisHttpObjectAggregator` only handle post request. 
what do you think of it ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250162919
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   The followings are what I want to figure out for current 
`DorisHttpObjectAggregator`
   
   1. If request has 100-continue header,  does it receive request body before 
call action's handler? If this is true, I think in normal case for 
`LoadAction`, FE won't receive load data.
   
   2. If request has no 100-continue header and request body is exceed max body 
size, will it return an error response to client ? If this is true, I think 
it's OK to receive relatively small data, however it is difficult to let client 
known that request misses 100-continue header, because aggregator is a common 
path for all input HTTP request.
   
   Making `DorisHttpObjectAggregator` only apply POST request seems OK.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli merged pull request #572: update rowset state to visible when set version

[GitBox]

chaoyli merged pull request #572: update rowset state to visible when set 
version
URL: https://github.com/apache/incubator-doris/pull/572
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli opened a new pull request #576: Fix data race of header

[GitBox]

chaoyli opened a new pull request #576: Fix data race of header
URL: https://github.com/apache/incubator-doris/pull/576
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] morningman merged pull request #576: Fix data race of header

[GitBox]

morningman merged pull request #576: Fix data race of header
URL: https://github.com/apache/incubator-doris/pull/576
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

kangkaisen commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250166332
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   1 Yes.  `DorisHttpObjectAggregator`  receive all request body before call  
`HttpServerHandler` action's handler. 
   
   2 Yes.  `DorisHttpObjectAggregator`  will response `413 Request Entity Too 
Large`.  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli opened a new pull request #577: Refactor tablet schema

[GitBox]

chaoyli opened a new pull request #577: Refactor tablet schema
URL: https://github.com/apache/incubator-doris/pull/577
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250437314
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   OK, I see
   
   It's OK to only apply this aggregator to POST request


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

imay commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250437314
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   OK, I see
   
   I think enabling chunked encoding in this aggregator is acceptable.
   
   1. If you apply this aggregator to all HTTP method, when client send `Load` 
request with "Content-Length" header and without "100-continue", this 
aggregator would also receive data. There is no difference between chunked or 
non chunked.
   
   2. If you only apply this aggregator to POST method. it's OK to receive 
chunked encoding body to POST action.  This aggregator won't receive body for 
`Load` action.
   
   So, I think you can let the aggregator to receive chunked body
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen commented on a change in pull request #562: Add ColocateMetaService

[GitBox]

kangkaisen commented on a change in pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562#discussion_r250451151
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/http/common/DorisHttpObjectAggregator.java
 ##
 @@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.common;
+
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.http.HttpContent;
+import io.netty.handler.codec.http.HttpMessage;
+import io.netty.handler.codec.http.HttpObject;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.handler.codec.http.HttpUtil;
+
+/*
+ * don't handle 100-continue and chunked transfer-encoding http header
+ */
+public class DorisHttpObjectAggregator extends HttpObjectAggregator {
+// the flag for aggregator whether has started
+// in order not to handle chunked transfer-encoding header in {@link 
isContentMessage} method
 
 Review comment:
   OK. I see.  I agree with you. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli merged pull request #577: Refactor tablet schema

[GitBox]

chaoyli merged pull request #577: Refactor tablet schema
URL: https://github.com/apache/incubator-doris/pull/577
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay merged pull request #562: Add ColocateMetaService

[GitBox]

imay merged pull request #562: Add ColocateMetaService
URL: https://github.com/apache/incubator-doris/pull/562
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli merged pull request #567: Add publish version task

[GitBox]

chaoyli merged pull request #567: Add publish version task
URL: https://github.com/apache/incubator-doris/pull/567
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen opened a new issue #578: Refactor CreateTableTest

[GitBox]

kangkaisen opened a new issue #578: Refactor CreateTableTest
URL: https://github.com/apache/incubator-doris/issues/578
 
 
   Why refactor CreateTableTest?
   
   1 Most of tests in CreateTableTest write wrong, couldn't work as we expect. 
For example, although many tests throw DdlException, but the specific error 
isn't we expected.
   
   2 Use jmockit instead of EasyMock.
   
   3 We should not anly assert the exception type ,but also assert the 
exception message.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] kangkaisen opened a new pull request #579: Refactor CreateTableTest

[GitBox]

kangkaisen opened a new pull request #579: Refactor CreateTableTest
URL: https://github.com/apache/incubator-doris/pull/579
 
 
   For issue [578](https://github.com/apache/incubator-doris/issues/578)
   
   why remove hamcrest-core in pom?
   
   If we use ExpectedException, there will be `java.lang.NoClassDefFoundError: 
org/hamcrest/TypeSafeMatcher`.
   
   The junit has a dependency on  hamcrest-core, So I think we can remove 
hamcrest-core dependency.
   ```
   [INFO] +- junit:junit:jar:4.12:test
   [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
   ```
   All FE UT passed.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] yiguolei opened a new pull request #580: Add prepare txn, commit txn, publish txn

[GitBox]

yiguolei opened a new pull request #580: Add prepare txn, commit txn, publish 
txn
URL: https://github.com/apache/incubator-doris/pull/580
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250470742
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -663,66 +675,18 @@ SchemaChangeDirectly::~SchemaChangeDirectly() {
 SAFE_DELETE(_dst_cursor);
 }
 
-bool SchemaChangeDirectly::_write_row_block(ColumnDataWriter* writer, 
RowBlock* row_block) {
+bool SchemaChangeDirectly::_write_row_block(RowsetBuilderSharedPtr 
rowset_builder, RowBlock* row_block) {
 for (uint32_t i = 0; i < row_block->row_block_info().row_num; i++) {
-if (OLAP_SUCCESS != writer->attached_by(_dst_cursor)) {
-OLAP_LOG_WARNING("fail to attach writer");
+if (OLAP_SUCCESS != rowset_builder->add_row(_dst_cursor)) {
 
 Review comment:
   where content of _dst_cursor comes from? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250469872
 
 

 ##
 File path: be/src/olap/rowset/alpha_rowset_writer.cpp
 ##
 @@ -70,22 +71,54 @@ OLAPStatus AlphaRowsetWriter::add_row(const char* row, 
Schema* schema) {
 if (status != OLAP_SUCCESS) {
 std::string error_msg = "add row failed";
 LOG(WARNING) << error_msg;
-return status; 
+return status;
 }
 _column_data_writer->next(row, schema);
+_is_pending_rowset++;
+return OLAP_SUCCESS;
+}
+
+OLAPStatus AlphaRowsetWriter::add_row_block(RowBlock* row_block) {
+size_t pos = 0;
+row_block->set_pos(pos);
+RowCursor row_cursor;
+row_cursor.init(_rowset_builder_context.tablet_schema);
+while (pos < row_block->limit()) {
+row_block->get_row(pos, &row_cursor);
+add_row(&row_cursor);
+row_block->pos_inc();
+pos = row_block->pos();
+_is_pending_rowset++;
+}
+return OLAP_SUCCESS;
+}
+
+OLAPStatus AlphaRowsetWriter::add_rowset(RowsetSharedPtr rowset) {
 
 Review comment:
   This name is confused. add_rowset_for_linked_schema_change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250465708
 
 

 ##
 File path: be/src/olap/rowset/segment_group.cpp
 ##
 @@ -732,6 +735,35 @@ bool 
SegmentGroup::remove_old_files(std::vector* removed_links) {
 return true;
 }
 
+bool SegmentGroup::copy_segments_to_path(const std::string& dest_path) {
+if (dest_path == "" || dest_path == _rowset_path_prefix) {
 
 Review comment:
   dest_path.empty()


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250471849
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -764,10 +728,10 @@ bool SchemaChangeDirectly::process(ColumnData* 
olap_data, SegmentGroup* new_segm
 RowBlock* ref_row_block = NULL;
 bool need_create_empty_version = false;
 OLAPStatus res = OLAP_SUCCESS;
-if (!olap_data->empty()) {
-res = olap_data->get_first_row_block(&ref_row_block);
+if (!rowset_reader->has_next()) {
 
 Review comment:
   has_next() is the same as empty() ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250472225
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -930,29 +874,22 @@ bool SchemaChangeWithSorting::process(ColumnData* 
olap_data, SegmentGroup* new_s
 RowBlock* ref_row_block = NULL;
 bool need_create_empty_version = false;
 OLAPStatus res = OLAP_SUCCESS;
-if (!olap_data->empty()) {
-res = olap_data->get_first_row_block(&ref_row_block);
+if (rowset_reader->has_next()) {
 
 Review comment:
   has_next() is the same as empty() ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250467986
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -2176,7 +2040,9 @@ OLAPStatus 
SchemaChangeHandler::_alter_tablet(SchemaChangeParams* sc_params) {
 << "status=" << 
sc_params->ref_tablet->schema_change_status().status;
 }
 
-sc_params->ref_tablet->release_rs_readers(&(sc_params->ref_olap_data_arr));
+for (auto rs_reader : sc_params->ref_rowset_readers) {
 
 Review comment:
   auto& rs_reader


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250466836
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -1115,121 +1057,109 @@ bool SchemaChangeWithSorting::process(ColumnData* 
olap_data, SegmentGroup* new_s
 }
 
 bool SchemaChangeWithSorting::_internal_sorting(const vector& 
row_block_arr,
-const Version& 
temp_delta_versions,
-SegmentGroup** 
temp_segment_group) {
-ColumnDataWriter* writer = NULL;
+const Version& version,
+const VersionHash version_hash,
+RowsetSharedPtr* rowset) {
 uint64_t merged_rows = 0;
 RowBlockMerger merger(_tablet);
 
-(*temp_segment_group) =
-new(nothrow) SegmentGroup(_tablet->tablet_id(),
-  0,
-  _tablet->tablet_schema(),
-  _tablet->num_key_fields(),
-  _tablet->num_short_key_fields(),
-  _tablet->num_rows_per_row_block(),
-  _tablet->rowset_path_prefix(),
-  temp_delta_versions,
-  rand(),
-  false,
-  0, 0);
-if (NULL == (*temp_segment_group)) {
-OLAP_LOG_WARNING("failed to malloc SegmentGroup. [size=%ld]", 
sizeof(SegmentGroup));
-goto INTERNAL_SORTING_ERR;
+RowsetBuilderContextBuilderSharedPtr context_builder(new 
RowsetBuilderContextBuilder());
+RowsetId rowset_id = 0;
+OLAPStatus status = 
RowsetIdGenerator::instance()->get_next_id(_tablet->data_dir(), &rowset_id);
+if (status == OLAP_SUCCESS) {
+LOG(WARNING) << "get next rowset id failed";
+return false;
 }
-
-VLOG(3) << "init writer. tablet=" << _tablet->full_name()
-<< ", block_row_size=" << _tablet->num_rows_per_row_block();
-writer = ColumnDataWriter::create(*temp_segment_group, false,
-  _tablet->compress_kind(), 
_tablet->bloom_filter_fpp());
-if (NULL == writer) {
-OLAP_LOG_WARNING("failed to create writer.");
-goto INTERNAL_SORTING_ERR;
+context_builder->set_rowset_id(rowset_id)
+   .set_tablet_id(_tablet->tablet_id())
+   .set_partition_id(_tablet->partition_id())
+   .set_tablet_schema_hash(_tablet->schema_hash())
+   .set_rowset_type(ALPHA_ROWSET)
+   .set_rowset_path_prefix(_tablet->tablet_path())
+   .set_tablet_schema(_tablet->tablet_schema())
+   .set_num_key_fields(_tablet->num_key_fields())
+   .set_num_short_key_fields(_tablet->num_short_key_fields())
+   
.set_num_rows_per_row_block(_tablet->num_rows_per_row_block())
+   .set_compress_kind(_tablet->compress_kind())
+   .set_bloom_filter_fpp(_tablet->bloom_filter_fpp())
+   .set_rowset_state(VISIBLE)
+   .set_version(version)
+   .set_version_hash(version_hash);
+
+RowsetBuilderContext context = context_builder->build();
+RowsetBuilderSharedPtr rowset_builder(new AlphaRowsetBuilder());
+if (rowset_builder == nullptr) {
+LOG(WARNING) << "new rowset builder failed";
+return false;
 }
+VLOG(3) << "init rowset builder. tablet=" << _tablet->full_name()
+<< ", block_row_size=" << _tablet->num_rows_per_row_block();
+rowset_builder->init(context);
 
-if (!merger.merge(row_block_arr, writer, &merged_rows)) {
-OLAP_LOG_WARNING("failed to merge row blocks.");
+if (!merger.merge(row_block_arr, rowset_builder, &merged_rows)) {
+LOG(WARNING) << "failed to merge row blocks.";
 goto INTERNAL_SORTING_ERR;
 }
 add_merged_rows(merged_rows);
 
-if (OLAP_SUCCESS != (*temp_segment_group)->load()) {
-OLAP_LOG_WARNING("failed to reload olap index.");
+*rowset = rowset_builder->build();
+
+if (OLAP_SUCCESS != (*rowset)->init()) {
+LOG(WARNING) << "failed to reload olap index.";
 goto INTERNAL_SORTING_ERR;
 }
 
-SAFE_DELETE(writer);
 return true;
 
 INTERNAL_SORTING_ERR:
-SAFE_DELETE(writer);
-
-(*temp_segment_group)->delete_all_files();
-SAFE_DELETE(*temp_segment_group);
+rowset_builder->release();
 return false;
 }
 
 bool SchemaChangeWithSorting::_external_sorting(
-vector& src_segment_groups,
-SegmentGroup* dest_segment_group,
+vector& src_rowsets,
+RowsetBuilderSharedPtr rowset_builder,
 TabletSharedPtr tablet)

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r249622581
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -1756,97 +1638,99 @@ OLAPStatus SchemaChangeHandler::schema_version_convert(
 }
 
 // c. 转换数据
-ColumnData* olap_data = NULL;
-for (vector::iterator it = ref_segment_groups->begin();
-it != ref_segment_groups->end(); ++it) {
-ColumnData* olap_data = ColumnData::create(*it);
-if (NULL == olap_data) {
-OLAP_LOG_WARNING("fail to create ColumnData.");
-res = OLAP_ERR_MALLOC_ERROR;
-goto SCHEMA_VERSION_CONVERT_ERR;
-}
-
-olap_data->init();
-
-SegmentGroup* new_segment_group = nullptr;
-if ((*it)->transaction_id() == 0) {
-new_segment_group = new SegmentGroup(dest_tablet->tablet_id(),
- 0,
- dest_tablet->tablet_schema(),
- dest_tablet->num_key_fields(),
- 
dest_tablet->num_short_key_fields(),
- 
dest_tablet->num_rows_per_row_block(),
- 
dest_tablet->rowset_path_prefix(),
- olap_data->version(),
- olap_data->version_hash(),
- olap_data->delete_flag(),
- (*it)->segment_group_id(), 0);
+for (vector::iterator it = ref_rowsets->begin();
+it != ref_rowsets->end(); ++it) {
+RowsetReaderSharedPtr rowset_reader = (*it)->create_reader();
+rowset_reader->init(nullptr);
+
+RowsetBuilderContextBuilder context_builder;
+RowsetId rowset_id = 0;
+RowsetIdGenerator::instance()->get_next_id(dest_tablet->data_dir(), 
&rowset_id);
+if ((*it)->is_pending()) {
+PUniqueId load_id;
+load_id.set_hi(0);
+load_id.set_lo(0);
+context_builder.set_rowset_id(rowset_id)
+   .set_tablet_id(dest_tablet->tablet_id())
+   .set_partition_id(dest_tablet->partition_id())
+   .set_tablet_schema_hash(dest_tablet->schema_hash())
+   .set_rowset_type(ALPHA_ROWSET)
+   .set_rowset_path_prefix(dest_tablet->tablet_path())
+   .set_tablet_schema(dest_tablet->tablet_schema())
+   .set_num_key_fields(dest_tablet->num_key_fields())
+   
.set_num_short_key_fields(dest_tablet->num_short_key_fields())
+   
.set_num_rows_per_row_block(dest_tablet->num_rows_per_row_block())
+   .set_compress_kind(dest_tablet->compress_kind())
+   
.set_bloom_filter_fpp(dest_tablet->bloom_filter_fpp())
+   .set_rowset_state(PREPARING)
+   .set_txn_id((*it)->txn_id())
+   .set_load_id(load_id);
 } else {
-new_segment_group = new SegmentGroup(dest_tablet->tablet_id(),
- 0,
- dest_tablet->tablet_schema(),
- dest_tablet->num_key_fields(),
- 
dest_tablet->num_short_key_fields(),
- 
dest_tablet->num_rows_per_row_block(),
- 
dest_tablet->rowset_path_prefix(),
- olap_data->delete_flag(),
- (*it)->segment_group_id(), 0,
- (*it)->is_pending(),
- (*it)->partition_id(),
- (*it)->transaction_id());
-}
-
-if (NULL == new_segment_group) {
-LOG(FATAL) << "failed to malloc SegmentGroup. size=" << 
sizeof(SegmentGroup);
-res = OLAP_ERR_MALLOC_ERROR;
-goto SCHEMA_VERSION_CONVERT_ERR;
-}
-
-new_segment_groups->push_back(new_segment_group);
-
-if (!sc_procedure->process(olap_data, new_segment_group, dest_tablet)) 
{
+context_builder.set_rowset_id(rowset_id)
+   .set_tablet_id(dest_tablet->tablet_id())
+   .set_partition_id(dest_tablet->partition_id())
+   .set_tablet_schema_hash(dest_tablet->schema_hash())
+   .set_rows

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250472139
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -843,52 +795,42 @@ bool SchemaChangeDirectly::process(ColumnData* 
olap_data, SegmentGroup* new_segm
 }
 add_filted_rows(filted_rows);
 
-if (!_write_row_block(writer, new_row_block)) {
+if (!_write_row_block(rowset_builder, new_row_block)) {
 OLAP_LOG_WARNING("failed to write row block.");
 result = false;
 goto DIRECTLY_PROCESS_ERR;
 }
 
-olap_data->get_next_row_block(&ref_row_block);
-
-}
+rowset_reader->next_block(&ref_row_block);
 
-if (OLAP_SUCCESS != writer->finalize()) {
-result = false;
-goto DIRECTLY_PROCESS_ERR;
 }
 
-if (OLAP_SUCCESS != new_segment_group->load()) {
 
 Review comment:
   where to load rowset?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250466397
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -1115,121 +1057,109 @@ bool SchemaChangeWithSorting::process(ColumnData* 
olap_data, SegmentGroup* new_s
 }
 
 bool SchemaChangeWithSorting::_internal_sorting(const vector& 
row_block_arr,
-const Version& 
temp_delta_versions,
-SegmentGroup** 
temp_segment_group) {
-ColumnDataWriter* writer = NULL;
+const Version& version,
+const VersionHash version_hash,
+RowsetSharedPtr* rowset) {
 uint64_t merged_rows = 0;
 RowBlockMerger merger(_tablet);
 
-(*temp_segment_group) =
-new(nothrow) SegmentGroup(_tablet->tablet_id(),
-  0,
-  _tablet->tablet_schema(),
-  _tablet->num_key_fields(),
-  _tablet->num_short_key_fields(),
-  _tablet->num_rows_per_row_block(),
-  _tablet->rowset_path_prefix(),
-  temp_delta_versions,
-  rand(),
-  false,
-  0, 0);
-if (NULL == (*temp_segment_group)) {
-OLAP_LOG_WARNING("failed to malloc SegmentGroup. [size=%ld]", 
sizeof(SegmentGroup));
-goto INTERNAL_SORTING_ERR;
+RowsetBuilderContextBuilderSharedPtr context_builder(new 
RowsetBuilderContextBuilder());
 
 Review comment:
   RowsetBuilderContextBuilderSharedPtr -> RowsetWriterContextBuilderSharedPtr


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250467924
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -2347,56 +2213,26 @@ OLAPStatus SchemaChange::create_initial_rowset(
 SchemaHash schema_hash,
 Version version,
 VersionHash version_hash,
-SegmentGroup* segment_group) {
+RowsetBuilderSharedPtr rowset_builder) {
 VLOG(3) << "begin to create init version. "
 << "begin=" << version.first << ", end=" << version.second;
 
-TabletSharedPtr tablet;
-ColumnDataWriter* writer = NULL;
 OLAPStatus res = OLAP_SUCCESS;
 
 do {
 if (version.first > version.second) {
-OLAP_LOG_WARNING("begin should not larger than end. [begin=%d 
end=%d]",
- version.first, version.second);
+LOG(WARNING) << "begin should not larger than end. "
+ << " [begin=" << version.first
 
 Review comment:
   bracket is not necessary.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250465200
 
 

 ##
 File path: be/src/olap/rowset/alpha_rowset.cpp
 ##
 @@ -181,17 +189,65 @@ OLAPStatus AlphaRowset::_init_segment_groups() {
 << "version=" << version.first << "-"
 << version.second << ", "
 << "version_hash=" << version_hash;
-return res; 
+return res;
 }
 }
 }
 _segment_group_size = _segment_groups.size();
 if (_is_cumulative_rowset && _segment_group_size > 1) {
 LOG(WARNING) << "invalid segment group meta for cumulative rowset. 
segment group size:"
 << _segment_group_size;
-return OLAP_ERR_ENGINE_LOAD_INDEX_TABLE_ERROR; 
+return OLAP_ERR_ENGINE_LOAD_INDEX_TABLE_ERROR;
+}
+return OLAP_SUCCESS;
+}
+
+OLAPStatus AlphaRowset::_init_pending_segment_groups() {
+std::vector pending_segment_group_metas;
+AlphaRowsetMeta* _alpha_rowset_meta = (AlphaRowsetMeta*)_rowset_meta.get();
+
_alpha_rowset_meta->get_pending_segment_groups(&pending_segment_group_metas);
+for (auto& pending_segment_group_meta : pending_segment_group_metas) {
+Version version = _rowset_meta->version();
+int64_t version_hash = _rowset_meta->version_hash();
+int64_t txn_id = _rowset_meta->txn_id();
+int64_t partition_id = _rowset_meta->partition_id();
+std::shared_ptr segment_group(new 
SegmentGroup(_rowset_meta->tablet_id(),
+_rowset_meta->rowset_id(), _tablet_schema, _num_key_fields, 
_num_short_key_fields,
+_num_rows_per_row_block, _rowset_path, false, 
pending_segment_group_meta.pending_segment_group_id(),
+pending_segment_group_meta.num_segments(), true, partition_id, 
txn_id));
+if (segment_group.get() == nullptr) {
 
 Review comment:
   segment_group == nullptr is OK.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250467548
 
 

 ##
 File path: be/src/olap/schema_change.h
 ##
 @@ -273,8 +279,8 @@ class SchemaChangeHandler {
 
 OLAPStatus schema_version_convert(TabletSharedPtr ref_tablet,
   TabletSharedPtr new_tablet,
-  std::vector* 
ref_segment_groups,
-  std::vector* 
new_segment_groups);
+  std::vector* 
ref_rowsets,
 
 Review comment:
   new_rowsets may be corresponds to old_rowsets. So old_rowsets may be a 
better name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli commented on a change in pull request #563: Modify schema change to use rowset

[GitBox]

chaoyli commented on a change in pull request #563: Modify schema change to use 
rowset
URL: https://github.com/apache/incubator-doris/pull/563#discussion_r250472635
 
 

 ##
 File path: be/src/olap/schema_change.cpp
 ##
 @@ -1700,10 +1582,10 @@ OLAPStatus SchemaChangeHandler::_create_new_tablet(
 OLAPStatus SchemaChangeHandler::schema_version_convert(
 
 Review comment:
   If you have duplicated rowset to handle ingestion when schema change, this 
function is redundant.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay merged pull request #579: Refactor CreateTableTest

[GitBox]

imay merged pull request #579: Refactor CreateTableTest
URL: https://github.com/apache/incubator-doris/pull/579
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] lenmom opened a new issue #581: install document issue

[GitBox]

lenmom opened a new issue #581: install document issue
URL: https://github.com/apache/incubator-doris/issues/581
 
 
   The build document in compile document should to be refine:
   In CentOS, install gcc5.3 is using 
   yum install devtoolset-4 report no package found which lead to the 
installation faield.
   
   Provide a set of command to install gcc v5.2+
   **sudo yum install centos-release-scl**
   **sudo yum install devtoolset-4-gcc***
   
   to use gcc 5.3, type  the following command in the terminal:
   **scl enable devtoolset-4 bash**
   
   we have three ways to make gcc5.3 avalible permanantly: 
   1st.  **sudo ln -s [path_to_gcc_5.3] /usr/bin**
   2nd.  
   **vim ~/.bashrc or vim  ~/.bash_profile
   source scl_source enable devtoolset-4**
   3rd.
   **echo "source /opt/rh/devtoolset-3/enable" >> /etc/bashrc
   source /etc/bashrc**


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli merged pull request #580: Add prepare txn, commit txn, publish txn

[GitBox]

chaoyli merged pull request #580: Add prepare txn, commit txn, publish txn
URL: https://github.com/apache/incubator-doris/pull/580
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] yiguolei opened a new pull request #582: rename lastest_version to rowset_with_max_version

[GitBox]

yiguolei opened a new pull request #582: rename lastest_version to 
rowset_with_max_version
URL: https://github.com/apache/incubator-doris/pull/582
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] chaoyli merged pull request #582: rename lastest_version to rowset_with_max_version

[GitBox]

chaoyli merged pull request #582: rename lastest_version to 
rowset_with_max_version
URL: https://github.com/apache/incubator-doris/pull/582
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] morningman opened a new pull request #583: Allow repair VERSION_IMCOMPLETE tablet when altering table

[GitBox]

morningman opened a new pull request #583: Allow repair VERSION_IMCOMPLETE 
tablet when altering table
URL: https://github.com/apache/incubator-doris/pull/583
 
 
   Previously we do not allow repair tablet if the table it belongs
   to is under ALTER process. But it will possibly let the alter job
   failed due to some replica's failure of load.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] imay merged pull request #583: Allow repair VERSION_IMCOMPLETE tablet when altering table

[GitBox]

imay merged pull request #583: Allow repair VERSION_IMCOMPLETE tablet when 
altering table
URL: https://github.com/apache/incubator-doris/pull/583
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

[GitHub] morningman commented on issue #581: install document issue

[GitBox]

morningman commented on issue #581: install document issue
URL: https://github.com/apache/incubator-doris/issues/581#issuecomment-457097886
 
 
   Thanks. We will merge it into INSTALL wiki


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org