Glad it was of any help.
I'll just leave my fix here for others who might get confused by my
previously screwed up example:
https://play.golang.org/p/slzcEm0Zka
On 16/06/17 10:53, James Pettyjohn wrote:
Thanks for the follow-up!
That ended up being the approach I took - map two levels deep (because
that's how the keys come through) and load everything at start up and
use as an immutable structure. While in dev it is using the old code
essentially and that's fine.
I also noted that internal structure in the old code using a struct {x
string, y string} for its first level key which I thought odd. Not sure
what effects that would have on things.
Best,
James
On Thursday, June 15, 2017 at 8:17:47 AM UTC-7, ksug wrote:
There were mistakes with the code I posted. Oh well, in a hurry and not
the point.
On 15/06/17 22:41, Kiki Sugiaman wrote:
> I think I misunderstood what you were trying to accomplish. Let
me try
> to rephrase and see if I'm getting closer:
>
> - to accommodate template changes during development, you introduced
> read-locking to your app.
> - as a result, your reads in production are slow. Never mind the
fact
> that locking is pointless in production because the templates
don't change.
> - ideally, you want your reads to yield to template changes during
> development, but go lockless in production.
>
> If the above are correct, then the solution is not data structure
> change, but code swapping during init() depending on the environment
> variable $DEVMODE (or whatever you want to call it).
>
> Example:
> https://play.golang.org/p/wqb9lVIPLa
<https://play.golang.org/p/wqb9lVIPLa>
> (repeat several times and change the random seed to randomize the
> behavior a bit)
>
> Of course, optimizing the data structure doesn't hurt. And to answer
> your question, yes map can handle 10's of thousands entries with
100MB
> raw data with *relative* ease.
>
>
>
> On 15/06/17 15:45, James Pettyjohn wrote:
>> I looked at doing boltdb but ended up with a slightly simpler
>> approach. I don't need the disk persistence in this case.
>>
>> I tried radix tree implementation but found it slower than my
current
>> nested map with locks. In that interest I rewrote the nested map
>> approach considering the key partitioning noted by Robert.
>>
>> The primary differences in my approach this time were:
>> 1) Read the whole dataset once treat it as immutable, no locking
>> 2) Align the API to no require any manipulation of the keys
after the
>> first load, so every lookup is now 0 allocation.
>>
>> The structure is a simplistic map[string]map[string]string -
language
>> -> key. There was a breakdown of the second map further in the
>> previous implementation, probably bringing it from a few
thousand to a
>> few hundred each. Don't know what the threshold on the benefits
there,
>> based on my experience the map implementation can handle 10s of
>> thousands with ease.
>>
>> This brought the lookups down by about 50%:
>>
>> BenchmarkLookupLooseLazy-8 3000000 458 ns/op 32 B/op 1 allocs/op
>> BenchmarkLookupLooseMap-8 5000000 265 ns/op 0 B/op 0 allocs/op
>>
>> Let me know if see anything else worth adjusting.
>>
>> Best,
>> James
>>
>>
>> On Wednesday, June 14, 2017 at 2:01:44 PM UTC-7, Robert
Johnstone wrote:
>>
>> I'm surprised that the memory overhead is significant - 100
MB is
>> not that much.
>>
>> Assuming that you don't need atomic updates to the entire KV
store,
>> partition the keys.
>>
>> Does the periodic reload involve changing the keys? If not,
you
>> could map the dataset into nested structs. However, you
will still
>> need to synchronise access if you want to to reload without
stopping
>> the server, but that would just be the leaves. Switching
just the
>> top-tier to a struct could help with the contention.
>>
>>
>> On Wednesday, 14 June 2017 14:45:25 UTC-4, James Pettyjohn
wrote:
>>
>> I have an application which has all of it's text, multiple
>> languages, stored in XML on disk that is merged into
templates
>> on the fly. About 100MB. Templates use dozens of strings
for
>> each render.
>>
>> Currently this is loaded in full into memory in a bunch
of tier
>> hash maps. They are lazy loaded and using multiple locks to
>> perform reads but, unless in dev mode, actually don't
change
>> throughout the lifetime of the application and should be
>> considered immutable.
>>
>> While workable at a smaller scale, it's slow at scale.
The most
>> important factor is concurrent lookup speed, secondary
concern
>> is memory overhead. And it cannot preclude periodic
reload while
>> doing dev.
>>
>> Is there a data structure or lib that suits this
scenarios more
>> than others?
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from
it, send
>> an email to golang-nuts...@googlegroups.com <javascript:>
>> <mailto:golang-nuts+unsubscr...@googlegroups.com <javascript:>>.
>> For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to golang-nuts+unsubscr...@googlegroups.com
<mailto:golang-nuts+unsubscr...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
