2020-02-17 10:12:07 UTC - Eugen: I'm currently getting `Space left on device ...` errors like this when running/debugging pulsar-broker tests, with about 24G available: > 19:01:41.493 [LedgerDirsMonitorThread:org.apache.bookkeeper.util.DiskChecker@156] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /tmp/bk0test7169448334269197941/current : 25039376384, Used space fraction: 0.95016724 > threshold 0.95. Has someone encountered these errors before? ---- 2020-02-17 16:41:33 UTC - David Kjerrumgaard: @Eugen These are normal and indicate that you have exceed the maximum disk usage threshold for bookies, which by default is 95% of the total disk. ---- 2020-02-17 22:03:08 UTC - Eugen: I wonder if this is a good setting for dev... ---- 2020-02-17 22:13:20 UTC - David Kjerrumgaard: I would think so. How much total disk space do you have in your dev environment? ---- 2020-02-17 22:20:11 UTC - Eugen: Never enough: ```$ df -h|grep /$ /dev/nvme0n1p2 468G 421G 24G 95% /``` ---- 2020-02-17 22:21:36 UTC - Eugen: Is it technically necessary to require 25G on a 500G disk to run tests? ---- 2020-02-17 22:22:52 UTC - Eugen: for PROD, this is perhaps a good default setting, for DEV (not only tests, but also when starting up pulsar in standalone), it seems unnecessary ---- 2020-02-17 22:33:09 UTC - David Kjerrumgaard: You can increase that threshold if you like, but I don't know if it is "safe" due to a lack of experience in adjusting it to some other setting. ---- 2020-02-17 22:55:23 UTC - Eugen: It seems I'm the only developer with disk utilization around 95%... will cope! ---- 2020-02-18 05:34:43 UTC - Devin G. Bost: @Eugen You probably need to purge your docker files. ---- 2020-02-18 05:35:40 UTC - Devin G. Bost: ```$ docker system prune -a --volumes``` Just be careful to back up any important volumes. This command is unforgiving. ---- 2020-02-18 05:35:59 UTC - Devin G. Bost: Running Pulsar tests seems to fill up docker disk space rather quickly. ---- 2020-02-18 05:36:18 UTC - Eugen: @Devin G. Bost :slightly_smiling_face: my docker is always very clean for exactly this reason... ---- 2020-02-18 05:36:27 UTC - Devin G. Bost: Gotcha. ---- 2020-02-18 05:36:37 UTC - Eugen: but thanks! ---- 2020-02-18 05:36:38 UTC - Devin G. Bost: Very smart. :slightly_smiling_face: ---- 2020-02-18 05:37:29 UTC - Devin G. Bost: It doesn’t take many Pulsar builds to fill up your disk space. ---- 2020-02-18 05:38:01 UTC - Eugen: ```$ docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 0 0 0B 0B Containers 0 0 0B 0B Local Volumes 0 0 0B 0B Build Cache 0 0 0B 0B``` ---- 2020-02-18 05:38:13 UTC - Devin G. Bost: hmm ---- 2020-02-18 05:38:19 UTC - Devin G. Bost: And, you’re still getting the error? ---- 2020-02-18 05:38:38 UTC - Eugen: because my disk really is 95% full ---- 2020-02-18 05:39:17 UTC - Eugen: imo 25G should be enough to run tests, regardless if it's only 5% of disk space +1 : Devin G. Bost ---- 2020-02-18 05:39:38 UTC - Eugen: but anyways, I always figure out something to delete... ---- 2020-02-18 05:40:48 UTC - Devin G. Bost: I’ve been having trouble working on the very annoying `ProxyParserTest.textRegexSubscription()` test that seems to fail very frequently. When I run the test locally, it just repeatedly gives me this:
```22:34:04.261 [pulsar-external-listener-46-1:org.apache.pulsar.client.impl.PulsarClientImpl@686] WARN org.apache.pulsar.client.impl.PulsarClientImpl - [topic: <persistent://sample/test/local/topic1>] Could not get connection while getPartitionedTopicMetadata -- Will try again in 100 ms 22:34:04.278 [pulsar-client-io-45-5:org.apache.pulsar.client.impl.ConnectionPool@201] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to OCPC-LM31977:56437 : <http://java.net|java.net>.UnknownHostException: failed to resolve 'OCPC-LM31977' after 3 queries ``` Has anyone seen this before? ---- 2020-02-18 05:41:25 UTC - Devin G. Bost: It seems like it’s not translating my hostname to 127.0.0.1 ---- 2020-02-18 05:42:04 UTC - Devin G. Bost: oh gotcha… ---- 2020-02-18 05:42:34 UTC - Devin G. Bost: Yeah, I’ve run into this… I think I actually created a Github Issue a while ago about it haha. ---- 2020-02-18 05:43:29 UTC - Devin G. Bost: It could be argued though that a disk performs best when at least 10% of it is available, but that’s not always feasible. I know that on the mac I get from my work, I’m always pushing up against the 256 GB I have. ---- 2020-02-18 06:49:39 UTC - Devin G. Bost: When I connect to my VPN, I get the same warning loop but with a different hostname. ---- 2020-02-18 06:50:34 UTC - Eugen: Thinkpad here, but same here with 500G - will go with 1T next time ---- 2020-02-18 06:51:15 UTC - Devin G. Bost: Have you tried running a disk audit tool to identify what in your filesystem is taking up most of the space? ---- 2020-02-18 06:51:35 UTC - Devin G. Bost: On Windows, I’ve used WinDirStat. On Mac, I’ve used Disk Inventory X. ---- 2020-02-18 06:52:03 UTC - Devin G. Bost: What’s your OS? ---- 2020-02-18 06:52:20 UTC - Eugen: What, Linux of course :slightly_smiling_face: ---- 2020-02-18 06:52:41 UTC - Devin G. Bost: Ubuntu? Fedora? CentOS? BSD? :wink: ---- 2020-02-18 06:52:51 UTC - Eugen: Have been using Debian for ages, but currently giving Ubuntu a spin ---- 2020-02-18 06:53:01 UTC - Devin G. Bost: Cool. ---- 2020-02-18 06:53:12 UTC - Devin G. Bost: Ubuntu has become a lot more useful in recent years. ---- 2020-02-18 06:53:28 UTC - Eugen: I've always loved debian for their great package management ---- 2020-02-18 06:54:30 UTC - Eugen: the reason I am trying Ubuntu is that zoom did not support desktop sharing on Debian testing... (although other Debian versions were supported) ---- 2020-02-18 06:55:02 UTC - Eugen: one of these days, I want to take a look at BSD and see if there is anything to those bold claims of those noisy BSD guys.. :slightly_smiling_face: ---- 2020-02-18 06:55:17 UTC - Eugen: they are bashing Linux all the time - and technically maybe rightly so ---- 2020-02-18 06:55:36 UTC - Eugen: "Worse is better" laughing : Devin G. Bost ---- 2020-02-18 06:55:40 UTC - Devin G. Bost: Good to know. I usually found myself in the System V camp (Fedora, Red Hat, SUSE, CentOS), but I started using Ubuntu recently since they had better support for ML drivers. ---- 2020-02-18 06:56:06 UTC - Eugen: I'm coming from SuSE (2 decades ago) ---- 2020-02-18 06:56:13 UTC - Eugen: reveals my German heritage ---- 2020-02-18 06:56:13 UTC - Devin G. Bost: Cool ---- 2020-02-18 06:56:22 UTC - Devin G. Bost: haha 2 decades ago. ---- 2020-02-18 06:56:27 UTC - Eugen: but never liked their rpm package management ---- 2020-02-18 06:56:33 UTC - Devin G. Bost: What didn’t you like about it? ---- 2020-02-18 06:56:36 UTC - Eugen: had to reinstall the whole system more than once ---- 2020-02-18 06:56:41 UTC - Eugen: because it got out of whack ---- 2020-02-18 06:56:46 UTC - Devin G. Bost: I never had issues with it. ---- 2020-02-18 06:56:51 UTC - Eugen: long time ago, so maybe it's gotten better ---- 2020-02-18 06:56:57 UTC - Eugen: but deb never failed me ---- 2020-02-18 06:57:00 UTC - Devin G. Bost: Were you creating RPMs or using Yum? ---- 2020-02-18 06:57:15 UTC - Eugen: back in the day I was only using it ---- 2020-02-18 06:57:25 UTC - Eugen: and there was no yum back then, I believe ---- 2020-02-18 06:57:46 UTC - Eugen: now, with docker, it's a different age anyway +1 : Devin G. Bost ---- 2020-02-18 06:58:02 UTC - Eugen: (no need to install everything on your host) ---- 2020-02-18 06:58:13 UTC - Devin G. Bost: Ah, that’s probably the issue. Raw RPMs are like running Python setup.py files. It’s the easiest way to make a big mess. ---- 2020-02-18 06:58:18 UTC - Eugen: btw I know WinDirStat +1 : Devin G. Bost ---- 2020-02-18 06:58:31 UTC - Eugen: because I knew KWinDirStat first ---- 2020-02-18 06:58:39 UTC - Devin G. Bost: Didn’t even know about it. ---- 2020-02-18 06:58:46 UTC - Eugen: that's where it comes from, a KDE tool on Linux ---- 2020-02-18 06:58:58 UTC - Devin G. Bost: Makes sense. The K gives the name away. ---- 2020-02-18 06:59:12 UTC - Eugen: KDirStat ---- 2020-02-18 06:59:21 UTC - Devin G. Bost: I was just about to comment on that haha. ---- 2020-02-18 06:59:25 UTC - Devin G. Bost: KWindows ---- 2020-02-18 06:59:31 UTC - Devin G. Bost: KDos ---- 2020-02-18 06:59:36 UTC - Eugen: :slightly_smiling_face: ---- 2020-02-18 06:59:59 UTC - Devin G. Bost: Or, the Gnome version, Gindows. ---- 2020-02-18 07:00:17 UTC - Devin G. Bost: (I just made that up.) ---- 2020-02-18 07:01:21 UTC - Devin G. Bost: How long have you been working with Pulsar? ---- 2020-02-18 07:02:57 UTC - Eugen: discovered Pulsar in January ---- 2020-02-18 07:03:27 UTC - Eugen: have been implementing a distributed db on top of Kafka before that (hobby project, unpublished, because unfinished) ---- 2020-02-18 07:03:47 UTC - Eugen: currently trying to convince our customer to go with Pulsar for a new project, it's gonna be Pulsar or Kafka ---- 2020-02-18 07:04:11 UTC - Devin G. Bost: Interesting. We have combined Pulsar with Apache Ignite for a lot of our customer use cases. They’re a very good combination. ---- 2020-02-18 07:04:32 UTC - Devin G. Bost: Ignite is a very stable technology from our experience so far, and the learning curve wasn’t too bad. ---- 2020-02-18 07:04:38 UTC - Devin G. Bost: It’s extremely fast. ---- 2020-02-18 07:04:55 UTC - Eugen: interesting. how are you using ignite? ---- 2020-02-18 07:05:43 UTC - Devin G. Bost: A lot of our users need fast lookups for stream data enrichment. We have Pulsar function flows that populate Ignite with data, and we have function flows that provide that data to other applications. ---- 2020-02-18 07:06:11 UTC - Devin G. Bost: For batch cases (when dealing with legacy clients), we have applications that regularly dump data into Pulsar topics to get picked up by our function flows. ---- 2020-02-18 07:06:51 UTC - Devin G. Bost: We also have backfill paths that have been really useful when data-impacting bugs ended up in production. ---- 2020-02-18 07:07:27 UTC - Eugen: you mentioned ML - is that part of what you are doing? ---- 2020-02-18 07:08:22 UTC - Devin G. Bost: I did a lot of ML in my previous role. During the last year, I’ve been more focused around opening doors since ML needs data. ---- 2020-02-18 07:10:26 UTC - Eugen: Makes sense ---- 2020-02-18 07:10:59 UTC - Devin G. Bost: @Eugen What happens when you try to run the test? ---- 2020-02-18 07:12:09 UTC - Eugen: I like Pulsar so far, but for the immediate use case, we may get away with using Kafka, so we need to think of something to convince the client. "no need to rebalance" may do the trick ---- 2020-02-18 07:12:29 UTC - Devin G. Bost: Yeah, that’s a big one. ---- 2020-02-18 07:12:54 UTC - Devin G. Bost: Did you ever see the video I created about Pulsar? It was intended to convince the business folks. ---- 2020-02-18 07:13:15 UTC - Devin G. Bost: <https://www.youtube.com/watch?v=qsz-FruOGoo&feature=youtu.be> ---- 2020-02-18 07:13:20 UTC - Eugen: thx, will have a look ---- 2020-02-18 07:13:31 UTC - Devin G. Bost: It should help :wink: ---- 2020-02-18 07:20:00 UTC - Eugen: works fine for me ---- 2020-02-18 07:20:21 UTC - Eugen: (my last git pull was some time yesterday) ---- 2020-02-18 07:20:56 UTC - Eugen: I'm not seeing your warning messages ---- 2020-02-18 07:26:08 UTC - Devin G. Bost: Weird… ---- 2020-02-18 07:26:17 UTC - Devin G. Bost: Mine is current as well. ---- 2020-02-18 07:50:46 UTC - Eugen: @Devin G. Bost No idea if it is related to your issue, but perhaps turning off ipv6 helps? ---- 2020-02-18 07:50:57 UTC - Devin G. Bost: I’ll try that. Thanks. ---- 2020-02-18 07:59:21 UTC - Devin G. Bost: Didn’t help. ---- 2020-02-18 08:00:54 UTC - Devin G. Bost: I get the same issue even when not connected to the internet. ---- 2020-02-18 08:04:08 UTC - Devin G. Bost: Maybe I can hack my hosts file to fix it. ---- 2020-02-18 08:05:17 UTC - Devin G. Bost: That worked. ---- 2020-02-18 08:05:30 UTC - Devin G. Bost: Hacking the hosts file fixed it. ----
