[GitHub] [incubator-doris] spaces-X opened a new pull request #8046: [Bug] fix incorrect serialized_size of TDigest object
spaces-X opened a new pull request #8046: URL: https://github.com/apache/incubator-doris/pull/8046 # Proposed changes No issue ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] annotated tag 1.13.5-2.12-1.0.0 updated (f72f32d -> ff4234d)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag 1.13.5-2.12-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. *** WARNING: tag 1.13.5-2.12-1.0.0 was modified! *** from f72f32d (commit) to ff4234d (tag) tagging f72f32dd4d746a830a4a765bf419546ddbe6e4b6 (commit) by morningman on Mon Feb 14 16:06:33 2022 +0800 - Log - [maven-release-plugin] copy for tag 1.13.5-2.12-1.0.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] morningman opened a new pull request #3: [chore] add maven-gpg-plugin
morningman opened a new pull request #3: URL: https://github.com/apache/incubator-doris-flink-connector/pull/3 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] morningman merged pull request #3: [chore] add maven-gpg-plugin
morningman merged pull request #3: URL: https://github.com/apache/incubator-doris-flink-connector/pull/3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] branch master updated (c53bffc -> 2ecca7e)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. from c53bffc [chore] change the repo name (#2) add 2ecca7e [chore] add maven-gpg-plugin (#3) No new revisions were added by this update. Summary of changes: flink-doris-connector/pom.xml | 16 +++- 1 file changed, 15 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] branch branch-1.0.0 created (now 2ecca7e)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. at 2ecca7e [chore] add maven-gpg-plugin (#3) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] annotated tag 1.13.5-2.12-1.0.0 updated (64424f4 -> 0a5a9e7)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag 1.13.5-2.12-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. *** WARNING: tag 1.13.5-2.12-1.0.0 was modified! *** from 64424f4 (commit) to 0a5a9e7 (tag) tagging 64424f45443f47898cf8bfbc7a904641d763925f (commit) by morningman on Mon Feb 14 16:20:28 2022 +0800 - Log - [maven-release-plugin] copy for tag 1.13.5-2.12-1.0.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] annotated tag 1.13.5-2.12-1.0.0 updated (52b56ae -> 2fa4395)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag 1.13.5-2.12-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. *** WARNING: tag 1.13.5-2.12-1.0.0 was modified! *** from 52b56ae (commit) to 2fa4395 (tag) tagging 52b56ae1e967129244b686ea8e14273e74c5b20e (commit) by morningman on Mon Feb 14 16:23:18 2022 +0800 - Log - [maven-release-plugin] copy for tag 1.13.5-2.12-1.0.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] annotated tag 1.13.5-2.12-1.0.0 updated (796df5b -> efa28e3)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag 1.13.5-2.12-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. *** WARNING: tag 1.13.5-2.12-1.0.0 was modified! *** from 796df5b (commit) to efa28e3 (tag) tagging 796df5b0d94bd43ce45e4c7a640dfa69cbca8b7c (commit) by morningman on Mon Feb 14 16:27:04 2022 +0800 - Log - [maven-release-plugin] copy for tag 1.13.5-2.12-1.0.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] annotated tag 1.12.7-2.12-1.0.0 updated (04f06e4 -> 06be4b3)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag 1.12.7-2.12-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. *** WARNING: tag 1.12.7-2.12-1.0.0 was modified! *** from 04f06e4 (commit) to 06be4b3 (tag) tagging 04f06e44d5eb3a0b1415e2c4c819f335afa6965a (commit) by morningman on Mon Feb 14 16:30:24 2022 +0800 - Log - [maven-release-plugin] copy for tag 1.12.7-2.12-1.0.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] annotated tag 1.11.6-2.12-1.0.0 updated (fc21f36 -> be6a811)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to annotated tag 1.11.6-2.12-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git. *** WARNING: tag 1.11.6-2.12-1.0.0 was modified! *** from fc21f36 (commit) to be6a811 (tag) tagging fc21f36e1964ac44fb433ad5ca90d2806b94dfc5 (commit) by morningman on Mon Feb 14 16:33:26 2022 +0800 - Log - [maven-release-plugin] copy for tag 1.11.6-2.12-1.0.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new issue #8047: [Vectorized][Bug] Some multi arg funtion use in DATETIME not effective in DATE type
HappenLee opened a new issue #8047: URL: https://github.com/apache/incubator-doris/issues/8047 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version vectorized ### What's Wrong? ``` mysql> select yearweek(k10) from baseall_mysql order by k1; ERROR 1105 (HY000): errCode = 2, detailMessage = Function yearweek is not implemented ``` ### What You Expected? get the right exe in DATE type ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of query and load in system level
tianhui5 commented on pull request #7474: URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814296 > Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 removed a comment on pull request #7474: [Feature] Limit frequency of query and load in system level
tianhui5 removed a comment on pull request #7474: URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814296 > Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of query and load in system level
tianhui5 commented on pull request #7474: URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814985 > Hi @tianhui5 > > Although we had some private discussions about this feature, we eventually needed a complete design and usage document to help all developers and users understand how this feature works and how to use it. > > It is recommended that the final design be fully described in the issue #7129 (and the previous design be marked as Deprecated). > > Also, I suggest a separate user document describing the functionality from a usage perspective, including the followings: > > 1. the purpose of the feature (Motivation) > 2. what scenarios the feature is applicable to, how to use it, and what impact it will have. > 3. whether there are shortcomings for further improvement. > > In the previous Doris development, we also had a lot of features that were difficult to maintain due to lack of documentation, and I hope to be able to improve the process in this area in future projects. Thanks! Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on pull request #7474: [Feature] Limit frequency of query and load in system level
tianhui5 edited a comment on pull request #7474: URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814985 > Hi @tianhui5 > > Although we had some private discussions about this feature, we eventually needed a complete design and usage document to help all developers and users understand how this feature works and how to use it. > > It is recommended that the final design be fully described in the issue #7129 (and the previous design be marked as Deprecated). > > Also, I suggest a separate user document describing the functionality from a usage perspective, including the followings: > > 1. the purpose of the feature (Motivation) > 2. what scenarios the feature is applicable to, how to use it, and what impact it will have. > 3. whether there are shortcomings for further improvement. > > In the previous Doris development, we also had a lot of features that were difficult to maintain due to lack of documentation, and I hope to be able to improve the process in this area in future projects. Thanks! Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r52554 - in /dev/incubator/doris/flink-connector: ./ 1.0.0/
Author: morningman Date: Mon Feb 14 08:52:53 2022 New Revision: 52554 Log: Add flink connector 1.0.0 for flink 1.11/1.12/1.13 Added: dev/incubator/doris/flink-connector/ dev/incubator/doris/flink-connector/1.0.0/ dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz (with props) dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.asc dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.sha512 dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz (with props) dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz.asc dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz.sha512 dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz (with props) dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.asc dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.13.5-2.12-1.0.0-incubating-src.tar.gz.sha512 Added: dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz == Binary file - no diff available. Propchange: dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.asc == --- dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.asc (added) +++ dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.asc Mon Feb 14 08:52:53 2022 @@ -0,0 +1,14 @@ +-BEGIN PGP SIGNATURE- + +iQHKBAABCAA0FiEEfDkqHpxmyczPOIYY8QYm42JFg8sFAmIKFxYWHGNoZW5taW5n +eXVAYXBhY2hlLm9yZwAKCRDxBibjYkWDy5iHC/97VlbuyEXjZVVZdQKmVPy2dcj8 +zK5cGV089U5xntntA/V+rtx45PbCQeFSf6r4j/U74sBDDUFf/ChqWvdSgCMjs/wR ++9DDbP0exQCdoCUodeEyn8gIJrNZ3OUaxdmfx8/24hJfBC1hRS2N0g09VynxlRl8 +IQuO0zjts9Xpi1CIJHoFnpsdUnu6M7+DcLxuAWywOzV4Mkb1yU/CPsTxamKI6SFF +oCt57Ta8tyI0kaY9OMhdpfBh2X+yWqK6nhvgUlEHPc36WlqM6Env6kFgRhdx3yoE +TzndHwpUxQEdVbsz6er/OZn+9DUI7hQOyEy2jACt7Va7DKRlfp1fWWGjqd0ZOO+6 +suVY8fCM5B3/yUb1KGycyg7E8L8G1oOQ4F2kdrGXA1Tmn2qABjZfj2Sc7UTSPaqh +/j5SOsEAJQV3J+Y3zLFOkAmoSR2JxcgRhtD+3CJg4QpCj2XobHezu4hLzLg3r3q6 +HmTO0acf55/uAT6KVq0d2iTpOZ5KxuMAHeUCJ5w= +=8I8Y +-END PGP SIGNATURE- Added: dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.sha512 == --- dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.sha512 (added) +++ dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz.sha512 Mon Feb 14 08:52:53 2022 @@ -0,0 +1 @@ +643683356eacf450a3da8d0b52f73de68b8777d8d370223c3970113002e75b2f apache-doris-flink-connector-1.11.6-2.12-1.0.0-incubating-src.tar.gz Added: dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz == Binary file - no diff available. Propchange: dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz.asc == --- dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz.asc (added) +++ dev/incubator/doris/flink-connector/1.0.0/apache-doris-flink-connector-1.12.7-2.12-1.0.0-incubating-src.tar.gz.asc Mon Feb 14 08:52:53 2022 @@ -0,0 +1,14 @@ +-BEGIN PGP SIGNATURE- + +iQHKBAABCAA0FiEEfDkqHpxmyczPOIYY8QYm42JFg8sFAmIKF7MWHGNoZW5taW5n +eXVAYXBhY2hlLm9yZwAKCRDxBibjYkWDyxoUC/949vjb+hO8Aw/FvZNx5XblqLhk +e2YrYEKvOum9AazKgEkOEdfA+FN4eSznE0AhbgWqxGFfjwHxxQ4sr5oU7m6Zla3y +5CEoG13KR6n567N4BVoOYyL5uR+62SuqDgp1wRT12VPIoMc9hui/Feudzg9mP72x +KV2luvn+0HIqGME+RECKRA1mNx8yPrbtl4ktzjTUEipFGXD+yEZTkh+Kie
[GitHub] [incubator-doris] bluemasion commented on issue #6530: Build failed in centos
bluemasion commented on issue #6530: URL: https://github.com/apache/incubator-doris/issues/6530#issuecomment-1038815800 > upgrade gcc version hi ,I met the same issue ,and also updated the gcc to version 8,but still got the issue, cloud give me some advise to solve it ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 opened a new pull request #8048: [Doc]Added be -238 error code description
hf200012 opened a new pull request #8048: URL: https://github.com/apache/incubator-doris/pull/8048 Added be -238 error code description # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] crazy2323 commented on issue #6530: Build failed in centos
crazy2323 commented on issue #6530: URL: https://github.com/apache/incubator-doris/issues/6530#issuecomment-1038818022 > use docker build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on issue #8025: [Feature] Support load binlog from MySQL directly instead of Canal
tianhui5 edited a comment on issue #8025: URL: https://github.com/apache/incubator-doris/issues/8025#issuecomment-1038834008 > How to read and parse binlog There is a mature library called Debezium, which is used by Flink CDC. It can connect MySQL and convert binlog to defined record. I think we can add the dependency of Debezium to solve the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on issue #8025: [Feature] Support load binlog from MySQL directly instead of Canal
tianhui5 commented on issue #8025: URL: https://github.com/apache/incubator-doris/issues/8025#issuecomment-1038834008 > How to read and parse binlog There is a mature library called Debezium, which is used by Flink CDC. It can connect MySQL and convert binlog to defined record. I think we can add the dependency of Debezium to solve the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #7972: [vectorized] [join] eliminate branch prediction & opt hash join performance
zuochunwei commented on a change in pull request #7972:
URL: https://github.com/apache/incubator-doris/pull/7972#discussion_r805629745
##
File path: be/src/vec/exec/join/join_op.h
##
@@ -26,15 +26,15 @@ namespace doris::vectorized {
struct RowRef {
using SizeT = uint32_t; /// Do not use size_t cause of memory economy
-const Block* block = nullptr;
+// const Block* block = nullptr;
SizeT row_num = 0;
// Use in right join to mark row is visited
// TODO: opt the varaible to use it only need
bool visited = false;
Review comment:
use bitfield to make visited as the highest bit of row_num? maybe useful.
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -292,8 +318,9 @@ struct ProcessHashTableProbe {
std::vector same_to_prev;
same_to_prev.reserve(1.2 * _batch_size);
-int right_col_idx = _left_table_data_types.size();
-int right_col_len = _right_table_data_types.size();
+std::vector _build_index;
+_build_index.reserve(1.2 * _batch_size);
Review comment:
consider using stack array instead?
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -217,49 +249,44 @@ struct ProcessHashTableProbe {
// after probe data eof
if (!_join_node->_is_right_semi_anti) {
++repeat_count;
-for (size_t j = 0; j < right_col_len; ++j) {
-auto& column =
*mapped.block->get_by_position(j).column;
-mcol[j + right_col_idx]->insert_from(column,
mapped.row_num);
-}
+_build_index.emplace_back(mapped.row_num);
}
} else {
for (auto it = mapped.begin(); it.ok(); ++it) {
// right semi/anti join should dispose the data in
hash table
// after probe data eof
if (!_join_node->_is_right_semi_anti) {
++repeat_count;
-for (size_t j = 0; j < right_col_len; ++j) {
-auto& column =
*it->block->get_by_position(j).column;
-// TODO: interface insert from cause
serious performance problems
-// when column is nullable. Try to make
more effective way
-mcol[j +
right_col_idx]->insert_from(column, it->row_num);
-}
+_build_index.emplace_back(it->row_num);
}
it->visited = true;
}
}
-}
-} else if (_join_node->_match_all_probe ||
- _join_node->_join_op == TJoinOp::LEFT_ANTI_JOIN) {
-++repeat_count;
-// only full outer / left outer need insert the data of right
table
-if (_join_node->_match_all_probe) {
-for (size_t j = 0; j < right_col_len; ++j) {
-DCHECK(mcol[j + right_col_idx]->is_nullable());
-mcol[j + right_col_idx]->insert_data(nullptr, 0);
+} else if (_join_node->_match_all_probe) {
+// only full outer / left outer need insert the data of
right table
+++repeat_count;
+for (size_t j = 0; j < _right_col_len; ++j) {
+DCHECK(mcol[j + _right_col_idx]->is_nullable());
+mcol[j + _right_col_idx]->insert_data(nullptr, 0);
}
}
}
-
items_counts[_probe_index++] = repeat_count;
current_offset += repeat_count;
-
if (current_offset >= _batch_size) {
break;
}
}
-
-for (int i = 0; i < right_col_idx; ++i) {
+
+// insert all match build rows
+for (int i = 0; i < _right_col_len; i++) {
+auto &column = *_build_block.get_by_position(i).column;
+for (int j = 0; j < _build_index.size(); j++) {
Review comment:
use insert_indices_from() instead
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -804,6 +838,10 @@ Status HashJoinNode::_hash_table_build(RuntimeState*
state) {
RETURN_IF_ERROR(child(1)->open(state));
SCOPED_TIMER(_build_timer);
Block block;
+MutableColumns columns(_right_table_data_types.size());
+for (int i = 0; i < _right_table_data_types.size(); i++) {
+columns[i] = _right_table_data_types[i]->create_column();
Review comment:
call reserve() here to avoid exlarging multi-times?
##
File path: be/src
[GitHub] [incubator-doris] caiconghui commented on issue #8025: [Feature] Support load binlog from MySQL directly instead of Canal
caiconghui commented on issue #8025: URL: https://github.com/apache/incubator-doris/issues/8025#issuecomment-1038842641 > Debezium > > How to read and parse binlog > > There is a mature library called Debezium, which is used by Flink CDC. It can connect MySQL and convert binlog to defined record. I think we can add the dependency of Debezium to solve the problem. Based on past experience, you should run the binglog synchronization job on the be, and fe is only used to start the binlog synchronization job -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #7972: [vectorized] [join] eliminate branch prediction & opt hash join performance
zuochunwei commented on a change in pull request #7972:
URL: https://github.com/apache/incubator-doris/pull/7972#discussion_r805648869
##
File path: be/src/vec/exec/join/join_op.h
##
@@ -26,15 +26,15 @@ namespace doris::vectorized {
struct RowRef {
using SizeT = uint32_t; /// Do not use size_t cause of memory economy
-const Block* block = nullptr;
+// const Block* block = nullptr;
SizeT row_num = 0;
// Use in right join to mark row is visited
// TODO: opt the varaible to use it only need
bool visited = false;
RowRef() {}
-RowRef(const Block* block_ptr, size_t row_num_count, bool is_visited =
false)
-: block(block_ptr), row_num(row_num_count), visited(is_visited) {}
Review comment:
i am not sure whether placing member data 'row_count' before 'next' will
save memory, try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on issue #8025: [Feature] Support load binlog from MySQL directly instead of Canal
tianhui5 commented on issue #8025: URL: https://github.com/apache/incubator-doris/issues/8025#issuecomment-1038864398 > fault tolerance design > > Debezium > > > > How to read and parse binlog > > > > > > There is a mature library called Debezium, which is used by Flink CDC. It can connect MySQL and convert binlog to defined record. I think we can add the dependency of Debezium to solve the problem. > > Based on past experience, you should run the binglog synchronization job on the be, and fe is only used to start the binlog synchronization job I came up this idea because the current Canal design is running synchronization job on FE, and send data to BE. Or do you know some cpp lib that similar with Debezium? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #7972: [vectorized] [join] eliminate branch prediction & opt hash join performance
zuochunwei commented on a change in pull request #7972:
URL: https://github.com/apache/incubator-doris/pull/7972#discussion_r805656136
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -177,89 +180,113 @@ struct ProcessHashTableProbe {
std::vector items_counts(_probe_rows);
auto& mcol = mutable_block.mutable_columns();
-
-int right_col_idx = _join_node->_is_right_semi_anti ? 0 :
_left_table_data_types.size();
-int right_col_len = _right_table_data_types.size();
int current_offset = 0;
+std::vector _build_index;
+_build_index.reserve(1.2 * _batch_size);
for (; _probe_index < _probe_rows;) {
-// ignore null rows
if constexpr (ignore_null) {
if ((*null_map)[_probe_index]) {
items_counts[_probe_index++] = 0;
continue;
}
}
-
int repeat_count = 0;
-auto find_result =
-(*null_map)[_probe_index]
+if constexpr (JoinOpType::value == TJoinOp::INNER_JOIN) {
+if (!(*null_map)[_probe_index]) {
+auto find_result =
key_getter.find_key(hash_table_ctx.hash_table, _probe_index, _arena);
+
+if (find_result.is_found()) {
+auto& mapped = find_result.get_mapped();
+
+if (mapped.get_row_count() == 1) {
+++repeat_count;
+_build_index.emplace_back(mapped.row_num);
+} else {
+if (_probe_index + 2 < _probe_rows)
+key_getter.prefetch(hash_table_ctx.hash_table,
_probe_index + 2, _arena);
+for (auto it = mapped.begin(); it.ok(); ++it) {
+++repeat_count;
Review comment:
increase repeat_count value out of the loop
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -217,49 +249,44 @@ struct ProcessHashTableProbe {
// after probe data eof
if (!_join_node->_is_right_semi_anti) {
++repeat_count;
-for (size_t j = 0; j < right_col_len; ++j) {
-auto& column =
*mapped.block->get_by_position(j).column;
-mcol[j + right_col_idx]->insert_from(column,
mapped.row_num);
-}
+_build_index.emplace_back(mapped.row_num);
}
} else {
for (auto it = mapped.begin(); it.ok(); ++it) {
// right semi/anti join should dispose the data in
hash table
// after probe data eof
if (!_join_node->_is_right_semi_anti) {
++repeat_count;
-for (size_t j = 0; j < right_col_len; ++j) {
-auto& column =
*it->block->get_by_position(j).column;
-// TODO: interface insert from cause
serious performance problems
-// when column is nullable. Try to make
more effective way
-mcol[j +
right_col_idx]->insert_from(column, it->row_num);
-}
+_build_index.emplace_back(it->row_num);
}
it->visited = true;
}
}
-}
-} else if (_join_node->_match_all_probe ||
- _join_node->_join_op == TJoinOp::LEFT_ANTI_JOIN) {
-++repeat_count;
-// only full outer / left outer need insert the data of right
table
-if (_join_node->_match_all_probe) {
-for (size_t j = 0; j < right_col_len; ++j) {
-DCHECK(mcol[j + right_col_idx]->is_nullable());
-mcol[j + right_col_idx]->insert_data(nullptr, 0);
+} else if (_join_node->_match_all_probe) {
+// only full outer / left outer need insert the data of
right table
+++repeat_count;
+for (size_t j = 0; j < _right_col_len; ++j) {
+DCHECK(mcol[j + _right_col_idx]->is_nullable());
+mcol[j + _right_col_idx]->insert_data(nullptr, 0);
}
}
}
-
items_counts[_probe_index++] = repeat_count;
current_offset += repeat_count;
-
if (current_offset >= _batch_size) {
break;
}
}
-
-for (int i = 0; i < right_col_
[GitHub] [incubator-doris] tianhui5 edited a comment on issue #8025: [Feature] Support load binlog from MySQL directly instead of Canal
tianhui5 edited a comment on issue #8025: URL: https://github.com/apache/incubator-doris/issues/8025#issuecomment-1038864398 > > Debezium > > > > How to read and parse binlog > > > > > > There is a mature library called Debezium, which is used by Flink CDC. It can connect MySQL and convert binlog to defined record. I think we can add the dependency of Debezium to solve the problem. > > Based on past experience, you should run the binglog synchronization job on the be, and fe is only used to start the binlog synchronization job I came up this idea because the current Canal design is running synchronization job on FE, and send data to BE. Or do you know some cpp lib that similar with Debezium? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #8025: [Feature] Support load binlog from MySQL directly instead of Canal
caiconghui commented on issue #8025: URL: https://github.com/apache/incubator-doris/issues/8025#issuecomment-1038876966 > > How to read and parse binlog > > There is a mature library called Debezium, which is used by Flink CDC. It can connect MySQL and convert binlog to defined record. I think we can add the dependency of Debezium to solve the problem. > > > Debezium > > > > > > > > How to read and parse binlog > > > > > > > > > There is a mature library called Debezium, which is used by Flink CDC. It can connect MySQL and convert binlog to defined record. I think we can add the dependency of Debezium to solve the problem. > > > > > > Based on past experience, you should run the binglog synchronization job on the be, and fe is only used to start the binlog synchronization job > > I came up this idea because the current Canal design is running synchronization job on FE, and send data to BE. Or do you know some cpp lib that similar with Debezium? no idea about cpp lib that parsing binlog. I think there are 3 solutions 1. continue to find cpp library to parse binlog 2. write code to parse binlog 3. parse binlog in broker so we can still use Debezium by pulling data from broker? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7979: [Vectorized][Feature] Support mysql external table insert into stm
github-actions[bot] commented on pull request #7979: URL: https://github.com/apache/incubator-doris/pull/7979#issuecomment-1038920979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xiangtao opened a new issue #8049: [Bug] be crash when upgrade be from 0.12.21 to 0.13.14
xiangtao opened a new issue #8049: URL: https://github.com/apache/incubator-doris/issues/8049 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 0.12.21 ### What's Wrong? upgrade be to 0.13.14 , but occure crash : *** Aborted at 1644833981 (unix time) try "date -d @1644833981" if you are using GNU date *** PC: @ 0x1141060 doris::FunctionContextImpl::allocate_local() *** SIGSEGV (@0x) received by PID 267620 (TID 0x7ff2e036f700) from PID 18446744073709551615; stack trace: *** @ 0x7ff3299ed2f0 (unknown) @ 0x1141060 doris::FunctionContextImpl::allocate_local() @ 0x11421e1 doris_udf::StringVal::StringVal() @ 0x114228e doris_udf::StringVal::copy_from() @ 0xe0ac2a doris::AggregateFunctions::string_concat_finalize() @ 0x16a87c7 doris::NewAggFnEvaluator::SerializeOrFinalize() @ 0x15fa365 doris::PartitionedAggregationNode::GetOutputTuple() @ 0x1600d66 doris::PartitionedAggregationNode::GetRowsFromPartition() @ 0x160125e doris::PartitionedAggregationNode::GetNextInternal() @ 0x160144f doris::PartitionedAggregationNode::get_next() @ 0x10f666f doris::PlanFragmentExecutor::get_next_internal() @ 0x10f7b31 doris::PlanFragmentExecutor::open_internal() @ 0x10f83e0 doris::PlanFragmentExecutor::open() @ 0x1065b0e doris::FragmentExecState::execute() @ 0x1068e76 doris::FragmentMgr::_exec_actual() @ 0x106eeec std::_Function_handler<>::_M_invoke() @ 0x11f4a62 doris::ThreadPool::dispatch_thread() @ 0x11ec8c8 doris::Thread::supervise_thread() @ 0x7ff3297a2e25 start_thread @ 0x7ff329ab5bad __clone ### What You Expected? no crash ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new pull request #8050: [Function][Vectorized] Fix multi args function call the DATETIME type not effective in DATE type and add the alias function
HappenLee opened a new pull request #8050: URL: https://github.com/apache/incubator-doris/pull/8050 1. Support some function alias of mod/fmod, adddate/add_data 2. Support some function of multi args: week, yearweek 3. Fix bug of multi args function call the DATETIME type not effective in DATE type # Proposed changes Issue Number: close #8047 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (Yes) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (Yes) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8044: [Feature][Vectorized] support aggregate function ndv()/approx_count_distinct()
HappenLee commented on a change in pull request #8044:
URL: https://github.com/apache/incubator-doris/pull/8044#discussion_r805719613
##
File path:
be/src/vec/aggregate_functions/aggregate_function_approx_count_distinct.h
##
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "exprs/anyval_util.h"
+#include "olap/hll.h"
+#include "udf/udf.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+struct AggregateFunctionApproxCountDistinctData {
+HyperLogLog hll_data;
+
+void add(StringRef value) {
+StringVal sv = value.to_string_val();
+uint64_t hash_value = AnyValUtil::hash64_murmur(sv,
HashUtil::MURMUR_SEED);
+if (hash_value != 0) {
+hll_data.update(hash_value);
+}
+}
+
+void merge(const AggregateFunctionApproxCountDistinctData& rhs) {
+hll_data.merge(rhs.hll_data);
+}
+
+void write(BufferWritable& buf) const {
+std::string result;
+result.resize(hll_data.max_serialized_size());
+int size = hll_data.serialize((uint8_t*)result.data());
+result.resize(size);
+write_binary(result, buf);
+}
+
+void read(BufferReadable& buf) {
+std::string result;
Review comment:
use StringRef to prevent mem alloc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] awakeljw opened a new pull request #8051: Doris
awakeljw opened a new pull request #8051: URL: https://github.com/apache/incubator-doris/pull/8051 # Proposed changes Issue Number: close #xxx ## Problem Summary: eliminate branch prediction ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on a change in pull request #7473: [Feature][Transaction] Support two phase commit (2PC) for stream load
weizuo93 commented on a change in pull request #7473:
URL: https://github.com/apache/incubator-doris/pull/7473#discussion_r805733884
##
File path: be/src/http/action/stream_load_2pc.cpp
##
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "http/action/stream_load_2pc.h"
+
+#include
+#include
+
+#include "common/status.h"
+#include "http/http_channel.h"
+#include "http/http_headers.h"
+#include "http/http_request.h"
+#include "http/http_status.h"
+#include "http/utils.h"
+#include "runtime/stream_load/stream_load_context.h"
+#include "runtime/stream_load/stream_load_executor.h"
+#include "util/json_util.h"
+
+namespace doris {
+
+const static std::string HEADER_JSON = "application/json";
+
+StreamLoad2PCAction::StreamLoad2PCAction(ExecEnv* exec_env) :
_exec_env(exec_env) {}
+
+void StreamLoad2PCAction::handle(HttpRequest* req) {
+Status status = Status::OK();
+std::string status_result;
+
+if (config::disable_stream_load_2pc) {
+status = Status::InternalError("Two phase commit (2PC) for stream load
was disabled");
+status_result = to_json(status);
+HttpChannel::send_reply(req, HttpStatus::OK, status_result);
+return;
+}
+
+StreamLoadContext* ctx = new StreamLoadContext(_exec_env);
+ctx->ref();
+req->set_handler_ctx(ctx);
+ctx->db = req->param(HTTP_DB_KEY);
+std::string req_txn_id = req->header(HTTP_TXN_ID_KEY);
+try {
+ctx->txn_id = std::stoull(req_txn_id);
+} catch (const std::exception& e) {
+status = Status::InternalError("convert txn_id [" + req_txn_id + "]
failed");
+status_result = to_json(status);
+HttpChannel::send_reply(req, HttpStatus::OK, status_result);
+return;
+}
+ctx->txn_operation = req->header(HTTP_TXN_OPERATION_KEY);
Review comment:
> check invalid parameter?
OK, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on a change in pull request #7473: [Feature][Transaction] Support two phase commit (2PC) for stream load
weizuo93 commented on a change in pull request #7473: URL: https://github.com/apache/incubator-doris/pull/7473#discussion_r805734082 ## File path: be/src/common/config.h ## @@ -353,6 +353,7 @@ CONF_mInt32(stream_load_record_batch_size, "50"); CONF_Int32(stream_load_record_expire_time_secs, "28800"); // time interval to clean expired stream load records CONF_mInt64(clean_stream_load_record_interval_secs, "1800"); +CONF_mBool(disable_stream_load_2pc, "false"); Review comment: > I think we can set default value to true. And set it true after we complete the flink connector and everything is ok, OK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] steadyBoy opened a new pull request #8052: [docs] fix document date-time-functions typo
steadyBoy opened a new pull request #8052: URL: https://github.com/apache/incubator-doris/pull/8052 # Proposed changes Issue Number: close #xxx ## Problem Summary: 1. en: add convert_tz.md 2. en: fix curdate.md, the fix is to modify the title 3. en: add curtime.md 4. en: fix str_to_date.md, the fix is to modify the title 5. zh-CN: fix convert_tz.md, the fix is to modify the description 6. zh-CN: fix curdate.md, the fix is to modify the title ## Checklist(Required) 1. Does it affect the original behavior: (I Don't know) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (Yes) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] steadyBoy closed pull request #8052: [docs] fix document date-time-functions typo
steadyBoy closed pull request #8052: URL: https://github.com/apache/incubator-doris/pull/8052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] steadyBoy opened a new pull request #8053: [docs] fix document date-time-functions typo
steadyBoy opened a new pull request #8053: URL: https://github.com/apache/incubator-doris/pull/8053 1. en: add convert_tz.md 2. en: fix curdate.md, the fix is to modify the title 3. en: add curtime.md 4. en: fix str_to_date.md, the fix is to modify the title 5. zh-CN: fix convert_tz.md, the fix is to modify the description 6. zh-CN: fix curdate.md, the fix is to modify the title # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (I Don't know) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (Yes) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #8054: [Enhancement] add aclocal/automake version check
BiteThet opened a new issue #8054: URL: https://github.com/apache/incubator-doris/issues/8054 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description if aclocal version not equal to automake version, some thirdpatry library will build fail. So i add a precheck at start stage of build thirdpatry. ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #8055: [Enhancement] add aclocal/automake version check
BiteThet opened a new pull request #8055: URL: https://github.com/apache/incubator-doris/pull/8055 # Proposed changes Issue Number: close #8054 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zuochunwei opened a new pull request #8056: (unitTest) resolve unit test failed problem for VGenericIteratorsTest
zuochunwei opened a new pull request #8056: URL: https://github.com/apache/incubator-doris/pull/8056 # Proposed changes change vgeneric_iterators cause unit test failed, this PR fix it Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a change in pull request #8050: [Function][Vectorized] Fix multi args function call the DATETIME type not effective in DATE type and add the alias
BiteThet commented on a change in pull request #8050: URL: https://github.com/apache/incubator-doris/pull/8050#discussion_r805781599 ## File path: run-be-ut.sh ## @@ -75,7 +75,7 @@ fi eval set -- "$OPTS" -PARALLEL=$[$(nproc)/5+1] +PARALLEL=$[$(nproc)/4+1] Review comment: better not modify this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 opened a new pull request #8057: [Doc]Add version upgrade instructions
hf200012 opened a new pull request #8057: URL: https://github.com/apache/incubator-doris/pull/8057 Add version upgrade instructions # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8053: [docs] fix document date-time-functions typo
github-actions[bot] commented on pull request #8053: URL: https://github.com/apache/incubator-doris/pull/8053#issuecomment-1039163579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-manager] hf200012 merged pull request #1: [community] move GitBox mail to commits@doris
hf200012 merged pull request #1: URL: https://github.com/apache/incubator-doris-manager/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-manager] branch master updated: [community] move GitBox mail to commits@doris (#1)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris-manager.git The following commit(s) were added to refs/heads/master by this push: new 62f77b9 [community] move GitBox mail to commits@doris (#1) 62f77b9 is described below commit 62f77b95f6af1c32d4ed44824d841572f1f543e2 Author: Mingyu Chen AuthorDate: Tue Feb 15 10:21:10 2022 +0800 [community] move GitBox mail to commits@doris (#1) move GitBox mail to commits@doris Co-authored-by: morningman --- .asf.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.asf.yaml b/.asf.yaml index 46f0303..681abdc 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -16,7 +16,7 @@ # github: - description: Flink/Spark Connectors for Apache Doris(Incubating) + description: Cluster manager for Apache Doris(Incubating) homepage: https://doris.apache.org/ labels: - data-warehousing @@ -37,4 +37,4 @@ github: dismiss_stale_reviews: true required_approving_review_count: 1 notifications: -pullrequests_status: d...@doris.apache.org +pullrequests_status: commits@doris.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yinzhijian commented on a change in pull request #7994: [Feature](ThreadPool) Support thread pool per disk for scanners
yinzhijian commented on a change in pull request #7994:
URL: https://github.com/apache/incubator-doris/pull/7994#discussion_r806397427
##
File path: be/src/util/priority_thread_pool.hpp
##
@@ -39,6 +39,7 @@ class PriorityThreadPool {
public:
int priority;
WorkFunction work_function;
+int queue_id;
Review comment:
task_id is not mean which disk this task belongs to, so queue_id maybe
better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yinzhijian commented on a change in pull request #7994: [Feature](ThreadPool) Support thread pool per disk for scanners
yinzhijian commented on a change in pull request #7994:
URL: https://github.com/apache/incubator-doris/pull/7994#discussion_r806397427
##
File path: be/src/util/priority_thread_pool.hpp
##
@@ -39,6 +39,7 @@ class PriorityThreadPool {
public:
int priority;
WorkFunction work_function;
+int queue_id;
Review comment:
task_id doesn't mean which disk this task belongs to, so queue_id maybe
better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] qidaye commented on a change in pull request #8053: [docs] fix document date-time-functions typo
qidaye commented on a change in pull request #8053:
URL: https://github.com/apache/incubator-doris/pull/8053#discussion_r806401270
##
File path:
docs/en/sql-reference/sql-functions/aggregate-functions/group_concat.md
##
@@ -0,0 +1,63 @@
+---
+{
+"title": "group_concat",
Review comment:
`group_concat` already exists in string-functions subdirectory.
##
File path:
docs/zh-CN/sql-reference/sql-functions/date-time-functions/convert_tz.md
##
@@ -25,14 +25,12 @@ under the License.
-->
# convert_tz
-
+## description
## Syntax
Review comment:
```suggestion
### Syntax
```
##
File path:
docs/zh-CN/sql-reference/sql-functions/aggregate-functions/group_concat.md
##
@@ -0,0 +1,63 @@
+---
+{
+"title": "group_concat",
Review comment:
Same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] qidaye commented on pull request #8057: [Doc]Add version upgrade instructions
qidaye commented on pull request #8057: URL: https://github.com/apache/incubator-doris/pull/8057#issuecomment-1039809442 The three-digit version is not the official release. Is it appropriate to mention it in the official documentation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8000: [fix](compatibility) Fix compatibility issue of PRowBatch and some tablet sink bugs
morningman merged pull request #8000: URL: https://github.com/apache/incubator-doris/pull/8000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (fa7cd94 -> 884fddb)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from fa7cd94 [typo](docs) fix typos in multi-tenant.md(zh-CN) (#8032) add 884fddb [fix](compatibility) Fix compatibility issue of PRowBatch and some tablet sink bugs (#8000) No new revisions were added by this update. Summary of changes: be/src/exec/tablet_sink.cpp| 58 -- be/src/exec/tablet_sink.h | 14 +++--- be/src/olap/tablet_meta.cpp| 4 +- be/src/runtime/row_batch.cpp | 12 - docs/en/administrator-guide/config/fe_config.md| 4 +- docs/zh-CN/administrator-guide/config/fe_config.md | 2 +- .../main/java/org/apache/doris/common/Config.java | 2 +- .../doris/load/routineload/RoutineLoadManager.java | 52 +-- .../java/org/apache/doris/qe/QeProcessorImpl.java | 2 +- .../doris/clone/TabletReplicaTooSlowTest.java | 6 +-- 10 files changed, 83 insertions(+), 73 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8029: [community] add collaborators in .asf.yaml
morningman merged pull request #8029: URL: https://github.com/apache/incubator-doris/pull/8029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (884fddb -> f190806)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 884fddb [fix](compatibility) Fix compatibility issue of PRowBatch and some tablet sink bugs (#8000) add f190806 [community] add collaborators in .asf.yaml (#8029) No new revisions were added by this update. Summary of changes: .asf.yaml | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new issue #8058: [Bug] BE crashed when calling IndexCahnnel::mark_as_failed
morningman opened a new issue #8058: URL: https://github.com/apache/incubator-doris/issues/8058 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version trunk ### What's Wrong? ``` *** Aborted at 164499 (unix time) try "date -d @164499" if you are using GNU date *** PC: @ 0x56338a754d63 doris::stream_load::IndexChannel::mark_as_failed() *** SIGSEGV (@0x0) received by PID 54060 (TID 0x7f1f82eac700) from PID 0; stack trace: *** @ 0x56338b98d4f2 google::(anonymous namespace)::FailureSignalHandler() @ 0x7f2058480920 (unknown) @ 0x56338a754d63 doris::stream_load::IndexChannel::mark_as_failed() @ 0x56338a75bc5a _ZZN5doris11stream_load11NodeChannel9open_waitEvENKUlbE_clEb @ 0x56338a7633bb doris::stream_load::ReusableClosure<>::Run() @ 0x56338bb107fb brpc::Controller::EndRPC() @ 0x56338bb10aa0 brpc::Controller::RunEndRPC() @ 0x56338bca3fef bthread::TaskGroup::task_runner() @ 0x56338bc94bf1 bthread_make_fcontext ``` ### What You Expected? Not crash ### How to Reproduce? Not very common. Seem to happen when rpc timeout for a long time. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #8059: [Feature][Vectorized] support function `convert_tz()`
BiteThet opened a new issue #8059:
URL: https://github.com/apache/incubator-doris/issues/8059
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Description
support function `convert_tz()`
### Use case
mysql> select convert_tz('2019-08-01 13:21:03', 'Asia/Shanghai',
'America/Los_Angeles');
+---+
| convert_tz('2019-08-01 13:21:03', 'Asia/Shanghai', 'America/Los_Angeles') |
+---+
| 2019-07-31 22:21:03 |
+---+
mysql> select convert_tz('2019-08-01 13:21:03', '+08:00',
'America/Los_Angeles');
++
| convert_tz('2019-08-01 13:21:03', '+08:00', 'America/Los_Angeles') |
++
| 2019-07-31 22:21:03|
++
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #8060: [Feature][Vectorized] support function `convert_tz()`
BiteThet opened a new pull request #8060: URL: https://github.com/apache/incubator-doris/pull/8060 # Proposed changes Issue Number: close #8059 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Henry2SS opened a new issue #8061: [Enhancement] rewrite OR to In in some cases
Henry2SS opened a new issue #8061: URL: https://github.com/apache/incubator-doris/issues/8061 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description close to #7120. According to the comment, implement it in ExtractCommonFactorsRule.java. ### Solution According to the comment, implement it in ExtractCommonFactorsRule.java. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Henry2SS opened a new pull request #8062: [Rewrite](OrToIn) implement rewrite rule OrToIn in ExtractCommonFactorsRule
Henry2SS opened a new pull request #8062: URL: https://github.com/apache/incubator-doris/pull/8062 # Proposed changes Issue Number: close #8061, close #7120 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 2. Has unit tests been added: (Yes) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #7984: [Docs] add rpc function document
stalary commented on a change in pull request #7984:
URL: https://github.com/apache/incubator-doris/pull/7984#discussion_r806427694
##
File path:
samples/doris-demo/udf-demo/src/main/java/org/apache/doris/udf/FunctionGrpc.java
##
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.udf;
+
+import io.grpc.stub.StreamObserver;
+import net.devh.boot.grpc.server.service.GrpcService;
+import org.apache.doris.proto.FunctionService;
+import org.apache.doris.proto.PFunctionServiceGrpc;
+import org.apache.doris.proto.Types;
+
+import java.util.List;
+
+/**
+ * FunctionGrpc
+ *
+ * @author lirongqian
+ * @since 2022/02/08
+ */
+@GrpcService
+public class FunctionGrpc extends
PFunctionServiceGrpc.PFunctionServiceImplBase {
+
+@Override
+public void fnCall(FunctionService.PFunctionCallRequest request,
StreamObserver responseObserver) {
+String functionName = request.getFunctionName();
+FunctionService.PFunctionCallResponse res;
+if ("grpc_add".equals(functionName)) {
+List argsList = request.getArgsList();
+int sum = 0;
+for (Types.PValues pValues : argsList) {
+sum += pValues.getInt32Value(0) % 10;
Review comment:
I was using it for testing, so I deleted it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #7984: [Docs] add rpc function document
stalary commented on a change in pull request #7984:
URL: https://github.com/apache/incubator-doris/pull/7984#discussion_r806427735
##
File path:
samples/doris-demo/udf-demo/src/main/java/org/apache/doris/udf/FunctionGrpc.java
##
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.udf;
+
+import io.grpc.stub.StreamObserver;
+import net.devh.boot.grpc.server.service.GrpcService;
+import org.apache.doris.proto.FunctionService;
+import org.apache.doris.proto.PFunctionServiceGrpc;
+import org.apache.doris.proto.Types;
+
+import java.util.List;
+
+/**
+ * FunctionGrpc
+ *
+ * @author lirongqian
+ * @since 2022/02/08
+ */
+@GrpcService
+public class FunctionGrpc extends
PFunctionServiceGrpc.PFunctionServiceImplBase {
+
+@Override
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #7984: [Docs] add rpc function document
stalary commented on a change in pull request #7984: URL: https://github.com/apache/incubator-doris/pull/7984#discussion_r806428159 ## File path: docs/.vuepress/sidebar/en.js ## @@ -255,7 +255,8 @@ module.exports = [ directoryPath: "udf/", children: [ "contribute-udf", - "user-defined-function", + "user-defined-function-cpp", + "user-defined-function-rpc", Review comment: done ## File path: docs/.vuepress/sidebar/en.js ## @@ -255,7 +255,8 @@ module.exports = [ directoryPath: "udf/", children: [ "contribute-udf", - "user-defined-function", + "user-defined-function-cpp", Review comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #7984: [Docs] add rpc function document
stalary commented on a change in pull request #7984:
URL: https://github.com/apache/incubator-doris/pull/7984#discussion_r806428781
##
File path: docs/zh-CN/extending-doris/udf/user-defined-function-rpc.md
##
@@ -0,0 +1,98 @@
+---
+{
+"title": "User Defined Function Rpc",
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #7984: [Docs] add rpc function document
stalary commented on a change in pull request #7984:
URL: https://github.com/apache/incubator-doris/pull/7984#discussion_r806429209
##
File path: docs/zh-CN/extending-doris/udf/user-defined-function-rpc.md
##
@@ -0,0 +1,98 @@
+---
+{
+"title": "User Defined Function Rpc",
+"language": "zh-CN"
+}
+---
+
+
+
+# User Defined Function Rpc
+
+可以通过 Rpc 的方式调用函数逻辑，通过 protobuf 进行数据传输，支持
Java/C++/Python/Ruby/Go/PHP/JavaScript 等多种语言
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #7984: [Docs] add rpc function document
stalary commented on a change in pull request #7984:
URL: https://github.com/apache/incubator-doris/pull/7984#discussion_r806430083
##
File path: samples/doris-demo/udf-demo/src/main/resources/application.yml
##
@@ -0,0 +1,3 @@
+grpc:
Review comment:
done
##
File path: docs/zh-CN/extending-doris/udf/user-defined-function-rpc.md
##
@@ -0,0 +1,98 @@
+---
+{
+"title": "User Defined Function Rpc",
+"language": "zh-CN"
+}
+---
+
+
+
+# User Defined Function Rpc
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zbtzbtzbt commented on pull request #6367: [Bug][Bloom Filter] Fix bug of bloom filter null value flag not be reset
zbtzbtzbt commented on pull request #6367: URL: https://github.com/apache/incubator-doris/pull/6367#issuecomment-1039836809 we meet a same bug in meituan,your pr is useful,thanks @weizuo93 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8045: [Fix bug] should remove decommission job type from enum
github-actions[bot] commented on pull request #8045: URL: https://github.com/apache/incubator-doris/pull/8045#issuecomment-1039840410 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8045: [Fix bug] should remove decommission job type from enum
morningman commented on pull request #8045: URL: https://github.com/apache/incubator-doris/pull/8045#issuecomment-1039840630 Hi @yiguolei , please pay attention to your PR title: http://doris.incubator.apache.org/community/commit-format-specification.html#commit-format-specification -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8042: [Improvement] check simd instructions before start
morningman commented on a change in pull request #8042:
URL: https://github.com/apache/incubator-doris/pull/8042#discussion_r806436781
##
File path: be/src/service/doris_main.cpp
##
@@ -74,6 +79,199 @@ static void thrift_output(const char* x) {
} // namespace doris
+// These code is referenced from clickhouse
Review comment:
Can these code be moved to a separated file. like
`util/instruction_utils.hpp`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a change in pull request #8042: [Improvement] check simd instructions before start
yiguolei commented on a change in pull request #8042:
URL: https://github.com/apache/incubator-doris/pull/8042#discussion_r806439574
##
File path: be/src/service/doris_main.cpp
##
@@ -74,6 +79,199 @@ static void thrift_output(const char* x) {
} // namespace doris
+// These code is referenced from clickhouse
Review comment:
1. These code is not utils, they will be only used in the main function
and could not in other functions. There maybe some signal catching problem if
running in other functions I have not test it.
2. These code should run at the beginning of the program so I add
__attribute__(priority = 101) to checker. If other functions running before
these code meet SIMD code and could not find the SSE instruction the program
may crash we could not catch the signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8042: [Improvement] check simd instructions before start
morningman commented on a change in pull request #8042:
URL: https://github.com/apache/incubator-doris/pull/8042#discussion_r806445798
##
File path: be/src/service/doris_main.cpp
##
@@ -74,6 +79,199 @@ static void thrift_output(const char* x) {
} // namespace doris
+// These code is referenced from clickhouse
+// It is used to check the SIMD instructions
+enum class InstructionFail {
+NONE = 0,
+SSE3 = 1,
+SSSE3 = 2,
+SSE4_1 = 3,
+SSE4_2 = 4,
+POPCNT = 5,
+AVX = 6,
+AVX2 = 7,
+AVX512 = 8,
+ARM_NEON = 9
+};
+
+auto instruction_fail_to_string(InstructionFail fail)
+{
+switch (fail)
+{
+#define ret(x) return std::make_tuple(STDERR_FILENO, x, ARRAY_SIZE(x) - 1)
+case InstructionFail::NONE:
+ret("NONE");
+case InstructionFail::SSE3:
+ret("SSE3");
+case InstructionFail::SSSE3:
+ret("SSSE3");
+case InstructionFail::SSE4_1:
+ret("SSE4.1");
+case InstructionFail::SSE4_2:
+ret("SSE4.2");
+case InstructionFail::POPCNT:
+ret("POPCNT");
+case InstructionFail::AVX:
+ret("AVX");
+case InstructionFail::AVX2:
+ret("AVX2");
+case InstructionFail::AVX512:
+ret("AVX512");
+case InstructionFail::ARM_NEON:
+ret("ARM_NEON");
+}
+__builtin_unreachable();
+}
+
+
+sigjmp_buf jmpbuf;
+
+void sig_ill_check_handler(int, siginfo_t *, void *)
+{
+siglongjmp(jmpbuf, 1);
+}
+
+/// Check if necessary SSE extensions are available by trying to execute some
sse instructions.
+/// If instruction is unavailable, SIGILL will be sent by kernel.
+void check_required_instructions_impl(volatile InstructionFail & fail)
+{
+#if defined(__SSE3__)
+fail = InstructionFail::SSE3;
+__asm__ volatile ("addsubpd %%xmm0, %%xmm0" : : : "xmm0");
+#endif
+
+#if defined(__SSSE3__)
+fail = InstructionFail::SSSE3;
+__asm__ volatile ("pabsw %%xmm0, %%xmm0" : : : "xmm0");
+
+#endif
+
+#if defined(__SSE4_1__)
+fail = InstructionFail::SSE4_1;
+__asm__ volatile ("pmaxud %%xmm0, %%xmm0" : : : "xmm0");
+#endif
+
+#if defined(__SSE4_2__)
+fail = InstructionFail::SSE4_2;
+__asm__ volatile ("pcmpgtq %%xmm0, %%xmm0" : : : "xmm0");
+#endif
+
+/// Defined by -msse4.2
+#if defined(__POPCNT__)
+fail = InstructionFail::POPCNT;
+{
+uint64_t a = 0;
+uint64_t b = 0;
+__asm__ volatile ("popcnt %1, %0" : "=r"(a) :"r"(b) :);
+}
+#endif
+
+#if defined(__AVX__)
+fail = InstructionFail::AVX;
+__asm__ volatile ("vaddpd %%ymm0, %%ymm0, %%ymm0" : : : "ymm0");
+#endif
+
+#if defined(__AVX2__)
+fail = InstructionFail::AVX2;
+__asm__ volatile ("vpabsw %%ymm0, %%ymm0" : : : "ymm0");
+#endif
+
+#if defined(__AVX512__)
+fail = InstructionFail::AVX512;
+__asm__ volatile ("vpabsw %%zmm0, %%zmm0" : : : "zmm0");
+#endif
+
+#if defined(__ARM_NEON__)
+fail = InstructionFail::ARM_NEON;
+__asm__ volatile ("vadd.i32 q8, q8, q8" : : : "q8");
+#endif
+
+fail = InstructionFail::NONE;
+}
+
+bool write_retry(int fd, const char * data, size_t size)
+{
+if (!size)
+size = strlen(data);
+
+while (size != 0)
+{
+ssize_t res = ::write(fd, data, size);
+
+if ((-1 == res || 0 == res) && errno != EINTR)
+return false;
+
+if (res > 0)
+{
+data += res;
+size -= res;
+}
+}
+
+return true;
+}
+
+/// Macros to avoid using strlen(), since it may fail if SSE is not supported.
+#define WRITE_ERROR(data) do \
+{ \
+static_assert(__builtin_constant_p(data)); \
+if (!write_retry(STDERR_FILENO, data, ARRAY_SIZE(data) - 1)) \
+_Exit(1); \
+} while (false)
+
+/// Check SSE and others instructions availability. Calls exit on fail.
+/// This function must be called as early as possible, even before main,
because static initializers may use unavailable instructions.
+void check_required_instructions()
+{
+struct sigaction sa{};
+struct sigaction sa_old{};
+sa.sa_sigaction = sig_ill_check_handler;
+sa.sa_flags = SA_SIGINFO;
+auto signal = SIGILL;
+if (sigemptyset(&sa.sa_mask) != 0
+|| sigaddset(&sa.sa_mask, signal) != 0
+|| sigaction(signal, &sa, &sa_old) != 0)
+{
+/// You may wonder about strlen.
+/// Typical implementation of strlen is using SSE4.2 or AVX2.
+/// But this is not the case because it's compiler builtin and is
executed at compile time.
+
+WRITE_ERROR("Can not set signal handler\n");
+_Exit(1);
+}
+
+volatile InstructionFail fail = InstructionFail::NONE;
+
+if (sigsetjmp(jmpbuf, 1))
+{
+WRITE_ERROR("Instruction check fail. The CPU does not support ");
+if (!std::apply(write_retry, instruction_fail_to_string(fail)))
+_Exit(1);
+WRITE_ERROR(" instruction set.\n");
[GitHub] [incubator-doris] morningman commented on a change in pull request #8042: [Improvement] check simd instructions before start
morningman commented on a change in pull request #8042:
URL: https://github.com/apache/incubator-doris/pull/8042#discussion_r806446003
##
File path: be/src/service/doris_main.cpp
##
@@ -74,6 +79,199 @@ static void thrift_output(const char* x) {
} // namespace doris
+// These code is referenced from clickhouse
Review comment:
Ok I see.
I've checked and it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8042: [Improvement] check simd instructions before start
morningman commented on a change in pull request #8042:
URL: https://github.com/apache/incubator-doris/pull/8042#discussion_r806445798
##
File path: be/src/service/doris_main.cpp
##
@@ -74,6 +79,199 @@ static void thrift_output(const char* x) {
} // namespace doris
+// These code is referenced from clickhouse
+// It is used to check the SIMD instructions
+enum class InstructionFail {
+NONE = 0,
+SSE3 = 1,
+SSSE3 = 2,
+SSE4_1 = 3,
+SSE4_2 = 4,
+POPCNT = 5,
+AVX = 6,
+AVX2 = 7,
+AVX512 = 8,
+ARM_NEON = 9
+};
+
+auto instruction_fail_to_string(InstructionFail fail)
+{
+switch (fail)
+{
+#define ret(x) return std::make_tuple(STDERR_FILENO, x, ARRAY_SIZE(x) - 1)
+case InstructionFail::NONE:
+ret("NONE");
+case InstructionFail::SSE3:
+ret("SSE3");
+case InstructionFail::SSSE3:
+ret("SSSE3");
+case InstructionFail::SSE4_1:
+ret("SSE4.1");
+case InstructionFail::SSE4_2:
+ret("SSE4.2");
+case InstructionFail::POPCNT:
+ret("POPCNT");
+case InstructionFail::AVX:
+ret("AVX");
+case InstructionFail::AVX2:
+ret("AVX2");
+case InstructionFail::AVX512:
+ret("AVX512");
+case InstructionFail::ARM_NEON:
+ret("ARM_NEON");
+}
+__builtin_unreachable();
+}
+
+
+sigjmp_buf jmpbuf;
+
+void sig_ill_check_handler(int, siginfo_t *, void *)
+{
+siglongjmp(jmpbuf, 1);
+}
+
+/// Check if necessary SSE extensions are available by trying to execute some
sse instructions.
+/// If instruction is unavailable, SIGILL will be sent by kernel.
+void check_required_instructions_impl(volatile InstructionFail & fail)
+{
+#if defined(__SSE3__)
+fail = InstructionFail::SSE3;
+__asm__ volatile ("addsubpd %%xmm0, %%xmm0" : : : "xmm0");
+#endif
+
+#if defined(__SSSE3__)
+fail = InstructionFail::SSSE3;
+__asm__ volatile ("pabsw %%xmm0, %%xmm0" : : : "xmm0");
+
+#endif
+
+#if defined(__SSE4_1__)
+fail = InstructionFail::SSE4_1;
+__asm__ volatile ("pmaxud %%xmm0, %%xmm0" : : : "xmm0");
+#endif
+
+#if defined(__SSE4_2__)
+fail = InstructionFail::SSE4_2;
+__asm__ volatile ("pcmpgtq %%xmm0, %%xmm0" : : : "xmm0");
+#endif
+
+/// Defined by -msse4.2
+#if defined(__POPCNT__)
+fail = InstructionFail::POPCNT;
+{
+uint64_t a = 0;
+uint64_t b = 0;
+__asm__ volatile ("popcnt %1, %0" : "=r"(a) :"r"(b) :);
+}
+#endif
+
+#if defined(__AVX__)
+fail = InstructionFail::AVX;
+__asm__ volatile ("vaddpd %%ymm0, %%ymm0, %%ymm0" : : : "ymm0");
+#endif
+
+#if defined(__AVX2__)
+fail = InstructionFail::AVX2;
+__asm__ volatile ("vpabsw %%ymm0, %%ymm0" : : : "ymm0");
+#endif
+
+#if defined(__AVX512__)
+fail = InstructionFail::AVX512;
+__asm__ volatile ("vpabsw %%zmm0, %%zmm0" : : : "zmm0");
+#endif
+
+#if defined(__ARM_NEON__)
+fail = InstructionFail::ARM_NEON;
+__asm__ volatile ("vadd.i32 q8, q8, q8" : : : "q8");
+#endif
+
+fail = InstructionFail::NONE;
+}
+
+bool write_retry(int fd, const char * data, size_t size)
+{
+if (!size)
+size = strlen(data);
+
+while (size != 0)
+{
+ssize_t res = ::write(fd, data, size);
+
+if ((-1 == res || 0 == res) && errno != EINTR)
+return false;
+
+if (res > 0)
+{
+data += res;
+size -= res;
+}
+}
+
+return true;
+}
+
+/// Macros to avoid using strlen(), since it may fail if SSE is not supported.
+#define WRITE_ERROR(data) do \
+{ \
+static_assert(__builtin_constant_p(data)); \
+if (!write_retry(STDERR_FILENO, data, ARRAY_SIZE(data) - 1)) \
+_Exit(1); \
+} while (false)
+
+/// Check SSE and others instructions availability. Calls exit on fail.
+/// This function must be called as early as possible, even before main,
because static initializers may use unavailable instructions.
+void check_required_instructions()
+{
+struct sigaction sa{};
+struct sigaction sa_old{};
+sa.sa_sigaction = sig_ill_check_handler;
+sa.sa_flags = SA_SIGINFO;
+auto signal = SIGILL;
+if (sigemptyset(&sa.sa_mask) != 0
+|| sigaddset(&sa.sa_mask, signal) != 0
+|| sigaction(signal, &sa, &sa_old) != 0)
+{
+/// You may wonder about strlen.
+/// Typical implementation of strlen is using SSE4.2 or AVX2.
+/// But this is not the case because it's compiler builtin and is
executed at compile time.
+
+WRITE_ERROR("Can not set signal handler\n");
+_Exit(1);
+}
+
+volatile InstructionFail fail = InstructionFail::NONE;
+
+if (sigsetjmp(jmpbuf, 1))
+{
+WRITE_ERROR("Instruction check fail. The CPU does not support ");
+if (!std::apply(write_retry, instruction_fail_to_string(fail)))
+_Exit(1);
+WRITE_ERROR(" instruction set.\n");
[GitHub] [incubator-doris] wangbo opened a new issue #8063: [refactor] refactor some interfaces of storage layer column
wangbo opened a new issue #8063: URL: https://github.com/apache/incubator-doris/issues/8063 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description _No response_ ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo opened a new pull request #8064: (#8063) refactor some interfaces of storage layer column
wangbo opened a new pull request #8064: URL: https://github.com/apache/incubator-doris/pull/8064 # Proposed changes Issue Number: close #8063 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8044: [Feature][Vectorized] support aggregate function ndv()/approx_count_distinct()
HappenLee commented on a change in pull request #8044:
URL: https://github.com/apache/incubator-doris/pull/8044#discussion_r806499008
##
File path:
be/src/vec/aggregate_functions/aggregate_function_approx_count_distinct.cpp
##
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/aggregate_functions/aggregate_function_approx_count_distinct.h"
+
+#include "vec/columns/column_string.h"
+#include "vec/columns/columns_number.h"
+
+namespace doris::vectorized {
+
+AggregateFunctionPtr create_aggregate_function_approx_count_distinct(
+const std::string& name, const DataTypes& argument_types, const Array&
parameters,
+const bool result_is_nullable) {
+AggregateFunctionPtr res = nullptr;
+WhichDataType which(argument_types[0]->is_nullable()
+? reinterpret_cast(argument_types[0].get())
+ ->get_nested_type()
+: argument_types[0]);
+
+// TODO: use template traits here.
+if (which.is_uint8()) {
+res.reset(new
AggregateFunctionApproxCountDistinct(argument_types));
+} else if (which.is_int8()) {
Review comment:
support `Int128` and use `create_with_numeric_type` function to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee closed issue #7978: [Vectorized][Feature] Support mysql external table insert into stm
HappenLee closed issue #7978: URL: https://github.com/apache/incubator-doris/issues/7978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee merged pull request #7979: [Vectorized][Feature] Support mysql external table insert into stm
HappenLee merged pull request #7979: URL: https://github.com/apache/incubator-doris/pull/7979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (f190806 -> 25d6477)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from f190806 [community] add collaborators in .asf.yaml (#8029)
add 25d6477 [Vectorized][Feature] Support mysql external table insert
into stm (#7979)
No new revisions were added by this update.
Summary of changes:
be/src/exec/data_sink.cpp | 13 +-
be/src/runtime/mysql_table_writer.cpp | 148 -
be/src/runtime/mysql_table_writer.h| 13 +-
be/src/vec/CMakeLists.txt | 1 +
.../sink/vmysql_table_writer.cpp} | 44 +++---
.../sink/vmysql_table_writer.h}| 34 +++--
6 files changed, 209 insertions(+), 44 deletions(-)
copy be/src/{runtime/mysql_table_sink.cpp => vec/sink/vmysql_table_writer.cpp}
(61%)
copy be/src/{runtime/mysql_table_sink.h => vec/sink/vmysql_table_writer.h}
(69%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
