Hi Yibo We’ve found flakiness with protobuf_ep when building in parallel with GNU Make. There are comments about this in ThirdpartyToolchain.cmake I think and prior JIRA issues related to this.
For faster builds we generally are focused on using -GNinja so recommend using the ninja build tool instead of make. Wes On Fri, Jan 3, 2020 at 8:28 AM Yibo Cai <yibo....@arm.com> wrote: > Maybe just install protobuf dev package is the simplest solution. > > On 1/2/20 1:55 PM, Yibo Cai wrote: > > I noticed a fresh build always stuck at compiling protobuf for a long > time. We've decided to use single job building for each third party module > [1], partly because different thirty party modules are built concurrently > (protobuf is built concurrently with jemalloc, but protobuf itself is built > with only one job). > > > > The problem is that protobuf takes much more time than other modules to > finish, which blocks the whole building process. I tried adding "-j4" > manually when compiling protobuf [2], it significantly reduced time for a > fresh build. Below is my testing. > > > > test setting > > ------------ > > cpu: Intel E5-2650, 20 logical cores > > memory: 64G > > > > test command > > ------------ > > cmake -DCMAKE_BUILD_TYPE=Release -DARROW_BUILD_TESTS=ON > -DARROW_COMPUTE=ON -DARROW_GANDIVA=ON .. > > make -j8 > > > > test result > > ----------- > > build protobuf with single job(default): 10min 32sec > > build protobuf with four jobs(add -j4): 6min 23sec > > > > Build time dropped 40% from 632s to 383s. Even bigger gap is observed on > Arm platform. > > > > I would suggest enabling multi job build for protobuf, maybe set half > total jobs. Say, it we launch arrow build with "make -j8", we compile > protobuf with "-j4". Code may be kind of ugly and not consistent, but > deserves the effort IMHO. Comments? > > > > [1] https://github.com/apache/arrow/pull/2779 > > [2] > https://github.com/apache/arrow/blob/master/cpp/cmake_modules/ThirdpartyToolchain.cmake#L1163 >
