Colly Max Depth and encoding/json - null

Question

我已经完成了围棋之旅，现在正在学习一些Colly教程。我了解最大深度，并一直试图在这样的围棋程序中实现它：package mainimport (    "encoding/json"    "log"    "net/http"    "github.com/gocolly/colly")func ping(w http.ResponseWriter, r *http.Request) {    log.Println("Ping")    w.Write([]byte("ping"))}func getData(w http.ResponseWriter, r *http.Request) {    //Verify the param "URL" exists    URL := r.URL.Query().Get("url")    if URL == "" {        log.Println("missing URL argument")        return    }    log.Println("visiting", URL)    //Create a new collector which will be in charge of collect the data from HTML    c := colly.NewCollector(        // MaxDepth is 2, so only the links on the scraped page        // and links on those pages are visited        colly.MaxDepth(2),        colly.Async(true),    )    // Limit the maximum parallelism to 2    // This is necessary if the goroutines are dynamically    // created to control the limit of simultaneous requests.    //    // Parallelism can be controlled also by spawning fixed    // number of go routines.    c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 2})    //Slices to store the data    var response []string    //onHTML function allows the collector to use a callback function when the specific HTML tag is reached    //in this case whenever our collector finds an    //anchor tag with href it will call the anonymous function    // specified below which will get the info from the href and append it to our slice    c.OnHTML("a[href]", func(e *colly.HTMLElement) {        link := e.Request.AbsoluteURL(e.Attr("href"))        if link != "" {            response = append(response, link)        }    })执行此操作时，响应为 null。取出 MaxDepth 和 Async 行会导致预期的响应（仅使用顶级链接）。任何帮助是值得赞赏的！

Answer 1

在异步模式下运行时，将在实际发出请求之前返回（请参阅此处）;并行演示中演示了正确的过程。在您的情况下，这意味着：c.Visit

c.Visit(URL)
c.Wait()

仅使用 async 在仅发出一个请求时不是很有用。查看reddit示例，了解如何在一次操作中访问多个URL。

注意：您确实应该检查这些函数返回的错误值，添加错误处理程序也是很好的做法。