Hi,
I didn't want to create a new thread with same topic so though of making a
reply here.
I made a solution for the same problem https://play.golang.org/p/GP8qVB9FPfx
without
the use of lock and somehow it is working, but ends up going in the
deadlock! What is going wrong here?
func Crawl(root string, r_depth int, fetcher Fetcher) {
ch := make(chan struct {
string
int
})
var wg sync.WaitGroup
crawl := func(url string, depth int) {
defer wg.Done()
if depth <= 0 {
return
}
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("found: %s %q\n", url, body)
for _, u := range urls {
ch <- struct {
string
int
}{u, depth - 1}
}
}
urlMap := map[string]bool{root: true}
wg.Add(1)
go crawl(root, r_depth)
for url := range ch {
if _, ok := urlMap[url.string]; !ok {
wg.Add(1)
go crawl(url.string, url.int)
urlMap[url.string] = true
}
}
wg.Wait()
}
Regards,
Vaibhav
On Tuesday, August 28, 2012 at 3:07:37 AM UTC+5:30, Kyle Lemons wrote:
>
> Looks pretty good.
>
> On Mon, Aug 27, 2012 at 2:57 AM, Thomas Gawehns <thomas...@gmail.com
> <javascript:>> wrote:
>
>> Hi folks,
>> I'm new to go and must admit i like it. But somehow my solution to the
>> web crawler exercise looks to simple:
>>
>> func Crawl(url string, depth int, fetcher Fetcher) {
>>
>> visitedUrls := make(map[string]bool, 100)
>>
> This is accessed concurrently without a lock. With GOMAXPROCS=1, this
> probably won't blow up, but it's technically a data race and thus not
> "correct."
>
>
>> var InnerCrawl func(url string, depth int,
>> fetcher Fetcher, finish chan bool)
>> InnerCrawl = func (url string, depth int,
>> fetcher Fetcher, finish chan bool) {
>> defer func() { finish<- true } ()
>> done := make(chan bool)
>> if depth <= 0 || visitedUrls[url] {
>> return
>> }
>> visitedUrls[url] = true
>> body, urls, err := fetcher.Fetch(url)
>> if err != nil {
>> fmt.Println(err)
>> return
>> }
>> fmt.Printf("found: %s %q\n", url, body)
>> for _, u := range urls {
>> go InnerCrawl(u, depth-1, fetcher, done)
>> }
>> for _ = range urls {
>> <-done
>> }
>> }
>> done := make(chan bool)
>> go InnerCrawl(url, depth, fetcher, done)
>> <-done
>> return
>> }
>>
>> Is there some flaw in it???
>> rgds Thomas
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/a0aebb00-cb03-4bdb-a409-c24f723868d5%40googlegroups.com.
