Extremely, sorry forgot to add attachment.
On Mon, Sep 16, 2019 at 11:11 AM nilsocket <nilsoc...@gmail.com> wrote:
> We want to export data from PG to Kafka,
> We can't rely on extension which we have written as there could be any
> problems which we are not aware of and PG might break.
> We don't want our master to go down because of the extension we have
> written.
>
> So, we are okay with having a new PG instance whose work is just to export
> data, as slave.
>
> What we thought of doing is to pause recovery (start-up) process, on any
> vacuum changes on system catalog tables and resume once, our
> logical-decoding has caught up.
> That way, we aren't bloating our master.
>
> Our problem is, we aren't able to exactly identify what WAL records are
> causing Vacuum changes, as far as our understanding goes, `HEAP_2 (rmgr),
> CLEAN` and `BTREE (rmgr), VACUUM` records.
>
> Inorder to see our understanding is right or not, IPC (inter-process
> communication) between WAL_SENDER and START_UP process, is not efficient
> enough.
>
> I have seen Latches, but I'm not sure exactly how to use them, as from
> comments, my understanding is START_UP process is not available in PGPROC
> array.
>
> For efficient inter-process communication, what would be ideal way to
> communicate between two processes.
>
> I'm new to PostgreSQL, and C - world.
>
> What we are trying to achieve is something similar to this:
>
> START_UP process goes to sleep, as soon as it sees any vacuum on catalog.
> WAL_SENDER process will resume recovery (wakeup), as soon as it caught up
> and goes to sleep.
> START_UP process will wake up when THERE is something for WAL_SENDER to
> send.
>
> Basically, IPC, between two processes, where one process generates work,
> and other consumes it.
> producer should go to sleep, until consumer caught up.
> consumer should signal producer of it's completion and wake up producer.
> cycle goes on...
>
> As I have indicated before, I new to C and PostgreSQL, I familiar with
> GoLang, and I have written a sample program in Go (attached below).
>
> Any suggestions and pointers would be greatly helpful.
>
> --
> Thank you
>
--
Thank you
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
// just trying to simulate communication
// between goroutines, to test similar one in c with latches
var wg = new(sync.WaitGroup)
var tick = time.Tick(time.Second * 20)
func main() {
wakeUpT1 := make(chan struct{}, 1)
wakeUpT2 := make(chan struct{}, 1)
closeCh := make(chan struct{}, 1)
wg.Add(2)
go thread(1, wakeUpT1, wakeUpT2, closeCh)
go thread(2, wakeUpT2, wakeUpT1, closeCh)
// boot strapping,
wakeUpT1 <- struct{}{}
wg.Wait()
}
func thread(id int, wakeMeUp, wakeHimup, closeCh chan struct{}) {
defer wg.Done()
end:
for {
select {
case <-wakeMeUp:
t := time.Millisecond * time.Duration(rand.Intn(1000))
fmt.Println(id, "woke up and doing work for:", t)
<-time.Tick(t)
wakeHimup <- struct{}{}
case <-closeCh:
fmt.Println("timeout, quitting", id)
break end
case <-tick:
close(closeCh)
}
}
}
