Ah! I think I have the kernel of a usable idea here. Thanks.
How does this sound: template 1 outputs template 2. Input to template 1
kicks off the goroutines and places {{ get_results token_xyz }} as the
output, but also has an output channel that we can wait on for all of the
answers Then, template 2 is executed with a function that pulls the output
from the channel. Sort of hard to explain, but seems on the face to be
workable. I'll post results after I prototype it.
--
Michael
On Wednesday, May 31, 2017 at 1:23:40 PM UTC-5, Egon wrote:
>
> On Wednesday, 31 May 2017 20:34:56 UTC+3, Michael Brown wrote:
>>
>> This is almost certainly going to be a case where you have the correct
>> solution and I don't have the go experience to properly understand it. I
>> wasn't quite understanding how your code was "patching" the results and how
>> the template package knows to wait for it.
>>
>
> Let's say you have a template
>
> {{ sleep }} xxx {{ sleep }}
>
> Once you render with that approach you get something like this...
>
> <<token1>> xxx <<token2>>
>
> So instead of returning the actual value you replace them with a token.
> And spawn a go routine to fetch the answers.
>
> You wait for the results or just some, doesn't matter.
>
> Once you get a results, you replace it in the buffer...
>
> token1 -> "alpha"
> token2 -> "beta"
>
> "alpha" xxx "beta"
>
> Or whatever the result is.
>
> However this approach is pretty limited when you want to range over the
> results.
>
> Doing two passes over the template, might work better, but needs somewhat
> better logic to handle lookup; but that could work for handling multiple
> results...
>
> {{ sleep }}
>
> In the first pass you start the goroutines and each output goes into an
> array of channels
>
> var rpcs = []func() interface{}
>
> var funcMap = template.FuncMap { "sleep": func() { rpcs = append(rpcs,
> sleep) }, }
>
> // now you schedule goroutines in whatever way you need for the rpcs and
> write the results to a channel
> var results = []chan interface{}
>
> the second pass would do it as
> current := 0
>
> var funcMap = template.FuncMap { "sleep": func() interface{} {
>
> result := <-results[current]
>
> current++
>
> return result
>
>
> // ...
>
>
> Without knowing the actual template, it is hard to recommend something
> better simpler or easier.
>
> Let me describe my problem.
>>
>> I have a REST interface running on an embedded system. The REST api is
>> mandated by another entity and I have zero control over the output, I have
>> to produce specific conformant output. The data that I need to build the
>> interface is on the system in a variety of other processes, and I can get
>> that data via a couple of different RPC mechanisms, however sometimes these
>> RPC mechanisms can be slow. (I'm in progress building go bindings for them).
>>
>> The current code which creates the REST interface is a huge morass of C
>> code that generates the JSON output, so the exact structure of the JSON
>> output is hardcoded in the code.
>>
>> I have an opportunity here, and that is that it appears to me that, with
>> a little work, I can completely templatize the rest output. That is, I can
>> have zero page-specific code and render almost all of the output by
>> providing a small number of generic data access functions. The issue I ran
>> into is the serial nature of the substitution kills the performance in my
>> prototype.
>>
>> Yes, I can make a specific go function per page that gathers all the data
>> I need and provides it to the template. But that would mean that I'd need
>> to maintain both the templates and the data access functions. This is still
>> an improvement on the old way of doing things, but I was hoping to jump
>> straight to a fully templatized system, which according to my initial
>> analysis, would be considerably less code and would not really be abusing
>> the template system (most of the function calls are very straightforward
>> and there is no heavy processing of the output).
>>
>> --
>> Michael
>>
>> On Wednesday, May 31, 2017 at 10:56:57 AM UTC-5, Egon wrote:
>>>
>>> Both of my described approaches run the funcs serially however it does
>>> not wait for the response and later patxhes the results.
>>>
>>> Can you describe the whole thing you are building? Piecing the
>>> requirements and purpose together from comments is difficult.
>>>
>>> I.e. how much memory, how big is request latency, who gets the output,
>>> where do the templates come from...
>>>
>>> On May 31, 2017 6:40 PM, "Michael Brown" <michael...@gmail.com> wrote:
>>>
>>> The best thing I can think of is to modify text/template to add futures
>>> support, and then do multi-pass rendering. The place to add this looks
>>> relatively simple, however the implementation looks complicated (and I just
>>> wrote my first program in go a couple weeks ago...)
>>>
>>> The problem that I see with the solution below is that text/template
>>> execution tries to instantiate the values of each function serially and
>>> essentially waits for completion on each, serially. I've spent the last
>>> couple hours examining the text/template implementation.
>>> --
>>> Michael
>>>
>>>
>>> On Wednesday, May 31, 2017 at 8:59:20 AM UTC-5, Egon wrote:
>>>>
>>>> The best idea I can think of (without digging and modifying
>>>> text/template) is to use special tokens and replace them afterwards...
>>>> Of course this approach has limitations in what it can do.
>>>>
>>>> *// note: code untested and incomplete*
>>>>
>>>> type Token string
>>>> type Queries struct {
>>>> pending sync.WaitGroup
>>>> mu sync.Mutex
>>>> responses map[Token]string
>>>> }
>>>>
>>>> func (q *Queries) createToken() Token {
>>>> return unique token
>>>> }
>>>>
>>>> func (q *Queries) Do(fn func() string) Token {
>>>> token := q.createToken()
>>>> q.pending.Add(1)
>>>> go func(){
>>>> defer q.pending.Done()
>>>> result := fn()
>>>> q.mu.Lock()
>>>> q.responses[token] = result
>>>> q.mu.Unlock()
>>>> }()
>>>> return token
>>>> }
>>>>
>>>> func (q *Queries) Wait(){ q.pending.Wait() }
>>>> func (q *Queries) Patch(data []byte) []byte {
>>>> // replace tokens with responess
>>>> }
>>>>
>>>> func main() {
>>>>
>>>> q := NewQueries()
>>>>
>>>> var funcMap = template.FuncMap {
>>>>
>>>> "sleep": func() Token { return q.Do(func() string {
>>>> time.Sleep(1 * time.Second); return "slept" }) },
>>>>
>>>> }
>>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}}
>>>> {{sleep}} {{sleep}}")
>>>> var buf bytes.Buffer
>>>>
>>>> tmpl.Execute(&buf, nil)
>>>>
>>>> q.Wait()
>>>>
>>>> os.Stdout.Write(q.Patch(buf.Bytes))
>>>>
>>>> }
>>>>
>>>> The other approach would be to do multiple passes:
>>>>
>>>> 1. execute template
>>>> 2. collect funcs that haven't been run yet
>>>> 2.1. no funcs left --> output
>>>> 3. execute these funcs, cache the func values
>>>> 4. goto step 1 using the cache
>>>>
>>>> On Wednesday, 31 May 2017 16:26:15 UTC+3, Michael Brown wrote:
>>>>>
>>>>> I am designing a system that will heavily use text/template processing
>>>>> and I've run into one issue that is going to be a show stopper for me if
>>>>> I
>>>>> can't figure out a way around it.
>>>>>
>>>>> Execute() on a template will run all of the functions in the template
>>>>> serially.
>>>>>
>>>>> For example, when you run the code below, you can see it output
>>>>> "slept" once every second until it completes after 3 seconds. In this
>>>>> example, the sleep is simulating an RPC call to another process that may
>>>>> take some considerable time (few tenths of a second), but there will be a
>>>>> large number of these calls that could all theoretically run in parallel
>>>>> (ie. there are no data dependencies between them). I'd really like to
>>>>> know
>>>>> a way that I could have the templating engine run all of the functions at
>>>>> once and collect the output, ie. in the example below, the entire program
>>>>> should run in 1 second.
>>>>>
>>>>> package main
>>>>>
>>>>>
>>>>> import (
>>>>>
>>>>> "text/template"
>>>>>
>>>>> "os"
>>>>>
>>>>> "time"
>>>>>
>>>>> )
>>>>>
>>>>>
>>>>> var funcMap = template.FuncMap {
>>>>>
>>>>> "sleep": func() string { time.Sleep(1 * time.Second); return
>>>>> "slept" },
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> func main() {
>>>>>
>>>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}}
>>>>> {{sleep}} {{sleep}}")
>>>>>
>>>>> tmpl.Execute(os.Stdout, nil)
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "golang-nuts" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/golang-nuts/j0WwjQE11rw/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> golang-nuts...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
