Re: [lttng-dev] A question on protocols
On Mon, Nov 19, 2018 at 7:46 PM Jonathan Rajotte-Julien < jonathan.rajotte-jul...@efficios.com> wrote: > Hi Patrick, > > > The security between consumerd and relayd isn't an issue as we'd be > forced > > to keep both on the device. This because we have no way of transporting > the > > consumerd <-> relayd communication over the HTTP proxy, which is our only > > choice. We have already talked to the team in charge of the network > paths. > > In that case using lttng-relayd yield no advantage unless you are able to > have > a device running only lttng-relayd and gathering data from other devices > and > then sending the data via http. > Unfortunately, we're don't have any external device that can run the lttng-relayd component. > > > > > > The relayd side have to be on the device. The device has very limited > > amounts of free flash. Even if there were flash available it would wear > > down the flash faster than what we'd like. > > Not sure I follow here. The whole point of lttng-relayd is to be off the > device > being traced and to relay trace data using network only (skipping disk). > Sure, so given this I cannot use relayd for our purposes. The only other option would be to set LTTng to local tracing, which we don't have flash for. My hope was that we could somehow stream the CTF records somewhere else, over that HTTP proxy, ot be analyzed. As you explained earlier, the relayd can be viewed as a FTP server of sorts. So that will most likely not work. > > > A RAM-based disc might have been a solution if we would have had any > > mentionable amounts of RAM to spare for this. Unfortunately we don't have > > that. > > In all cases you will need some RAM somewhere if using LTTng (and I think > any > other monitoring/tracing tools) since we need to allocate tracing buffer. > We have RAM for the tracing buffer, no issues there. The tracing buffer is just a temporary medium until you get the trace on disc, possibly via relayd. Correct? I interpretted your RAM disc suggestion as a place where I could store the trace instead of using flash. > > > > > > > Keep in mind that trace production is normally much more quicker than > trace > > > reading/analysis. A buffering scheme is mostly always necessary. > > > > > > > Very valid point, I'll for sure keep that in mind. > > > > This speaks strongly against replacing relayd with something homebrewed, > as > > was the original plan. We'd probably not be able to shuffle the data fast > > enough over the network to keep up with the pace. > > If you are unable to "reserve" RAM/network bandwidth and/or use FLASH this > is > slowly becoming a "you can't have your cake and eat it too" situation. > This must > be frustrating... > > > > > > > You can also use the --trace-file-size and --trace-file-count to limit > the > > > disk > > > footprint of each live session. Make sure to have enough buffer for > live > > > reading if still using live. > > > > > > > This isn't an option as our disc is flash based and we'd like to limit > the > > wear due to collecting metrics. > > I was under the impression that the lttng-relayd process was on another > device. > I misunderstood the situation. > Perhaps i could have been more clear on the situation as well. We have LTTng tracing capabilites on our devices. During development we can use LTTng on the device and relayd on our dev machines and everything works great. In production, the situation is different. The only means we have of transporting anything from the device is a HTTP proxy. We're trying to find a way where we can keep a conitnuous LTTng trace on the device, tunnel it through the HTTP proxy and analyze the data remotely. The devices are in the embedded realm, limited amounts of RAM, CPU, and flash. This limits what we can do on the device when it comes to storage for example. Hopefully I've made our situation a bit more clear. Based on our discussion so far, I have to say that LTTng is starting to look less and less as a viable way forward for us. Regards, Patrik ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] A question on protocols
Hi Patrick, > Sure, so given this I cannot use relayd for our purposes. The only other > option would be to set LTTng to local tracing, which we don't have flash > for. My hope was that we could somehow stream the CTF records somewhere > else, over that HTTP proxy, ot be analyzed. As you explained earlier, the > relayd can be viewed as a FTP server of sorts. So that will most likely not > work. A possibility would be to spare some RAM (not easy I know), mount a tmpfs, configure the session to output to the tmpfs, use session rotation based on size, send the data, remove the data, rinse and repeat. Now the question is what is the amount of tracing data generated and the amount of RAM available. > We have RAM for the tracing buffer, no issues there. The tracing buffer is > just a temporary medium until you get the trace on disc, possibly via > relayd. Correct? Yes. The consumerd is responsible for fetching the data from the buffers, then it output it locally or send it over the network to lttng-relayd. You can take a look at the interaction here: https://lttng.org/docs/v2.10/#doc-plumbing > > I interpretted your RAM disc suggestion as a place where I could store the > trace instead of using flash. This was the correct interpretation. > > Perhaps i could have been more clear on the situation as well. > > We have LTTng tracing capabilites on our devices. During development we can > use LTTng on the device and relayd on our dev machines and everything works > great. Glad to hear it. > > In production, the situation is different. > > The only means we have of transporting anything from the device is a HTTP > proxy. We're trying to find a way where we can keep a conitnuous LTTng > trace on the device, tunnel it through the HTTP proxy and analyze the data > remotely. > > The devices are in the embedded realm, limited amounts of RAM, CPU, and > flash. This limits what we can do on the device when it comes to storage > for example. > > Hopefully I've made our situation a bit more clear. > > Based on our discussion so far, I have to say that LTTng is starting to > look less and less as a viable way forward for us. It would be quite disappointing if you would not be able to reuse the work done for instrumenting your applications. Cheers -- Jonathan Rajotte-Julien EfficiOS ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] A question on protocols
On Tue, Nov 20, 2018 at 6:16 PM Jonathan Rajotte-Julien < jonathan.rajotte-jul...@efficios.com> wrote: > Hi Patrick, > > > Sure, so given this I cannot use relayd for our purposes. The only other > > option would be to set LTTng to local tracing, which we don't have flash > > for. My hope was that we could somehow stream the CTF records somewhere > > else, over that HTTP proxy, ot be analyzed. As you explained earlier, the > > relayd can be viewed as a FTP server of sorts. So that will most likely > not > > work. > > A possibility would be to spare some RAM (not easy I know), mount a tmpfs, > configure the session to output to the tmpfs, use session rotation based on > size, send the data, remove the data, rinse and repeat. Now the question > is what > is the amount of tracing data generated and the amount of RAM available. > > > We have RAM for the tracing buffer, no issues there. The tracing buffer > is > > just a temporary medium until you get the trace on disc, possibly via > > relayd. Correct? > > Yes. > > The consumerd is responsible for fetching the data from the buffers, then > it > output it locally or send it over the network to lttng-relayd. > > You can take a look at the interaction here: > > https://lttng.org/docs/v2.10/#doc-plumbing Is the contents of the RAM buffer already proper CTF records ready to be consumed, or is there some transformation that needs to take place as well? If so, can i find that in some library or is it part of the consumerd code base? If i were to read the ring buffer instead of consumerd, then i would perhaps be in a situation where I could stream the ring buffer to another machine over that HTTP proxy? That way I wouldn't need any RAM disc or anything, the data is already stored. The consumerd logic that reads the ring buffer, is that part of some library i could take advantage of? > > > > > > I interpretted your RAM disc suggestion as a place where I could store > the > > trace instead of using flash. > > This was the correct interpretation. > > > > > Perhaps i could have been more clear on the situation as well. > > > > We have LTTng tracing capabilites on our devices. During development we > can > > use LTTng on the device and relayd on our dev machines and everything > works > > great. > > Glad to hear it. > We're very pleased with LTTng. > > > > > In production, the situation is different. > > > > The only means we have of transporting anything from the device is a HTTP > > proxy. We're trying to find a way where we can keep a conitnuous LTTng > > trace on the device, tunnel it through the HTTP proxy and analyze the > data > > remotely. > > > > The devices are in the embedded realm, limited amounts of RAM, CPU, and > > flash. This limits what we can do on the device when it comes to storage > > for example. > > > > Hopefully I've made our situation a bit more clear. > > > > Based on our discussion so far, I have to say that LTTng is starting to > > look less and less as a viable way forward for us. > > It would be quite disappointing if you would not be able to reuse the work > done > for instrumenting your applications. > Quite so, which is why we're trying so hard to find a viable way forward. Regards, Patrik ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
