50000 个并行请求的点击 URL

Question

我正在尝试访问任何示例 url 来并行 50000 个请求。但我收到各种错误。有人可以让我知道我做错了什么吗？或者应该采取什么策略？func MakeRequests(url string, ch chan<- int, wg *sync.WaitGroup) {    time.Sleep(time.Duration(rand.Intn(10)) * time.Millisecond)    resp, err := http.Get(url)    if err != nil {        fmt.Println(err)    }    ch <- resp.StatusCode    wg.Done()}func main() {    var waitGroup sync.WaitGroup     count := 0    ch := make(chan int)    url := "http://some-dummy-url/ping"    totalHits, _ := strconv.Atoi("50000")    waitGroup.Add(totalHits)    for i := 0; i < totalHits; i++ {        go MakeRequests(url, ch, &waitGroup)    }    for i := 0; i < totalHits; i++ {        if <-ch == 200 {            count++        }    }    waitGroup.Wait()    fmt.Printf("Total number of successfull request :" + strconv.Itoa(count))}错误Get http://some-dummy-url/ping: dial tcp 10.120.0.45:80: connect: can't assign requested addresspanic: runtime error: invalid memory address or nil pointer dereference[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x123da32]goroutine 13425 [running]:main.MakeRequests(0x7ffeefbff868, 0x22, 0xc0000880c0, 0xc0000ae290)        /Users/kshitij/go/src/github.com/pingService/healthCheck.go:19 +0xe2created by main.main        /Users/kshitij/go/src/github.com/pingService/healthCheck.go:33 +0x136exit status 2

Answer 1

你必须resp.Body靠近MakeRequests。

导致错误的另一个问题是您由于错误而尝试访问 StatusCodewhen is nil。resp

这是固定的代码。

func MakeRequests(url string, ch chan<- int, wg *sync.WaitGroup) {

    time.Sleep(time.Duration(rand.Intn(10)) * time.Millisecond)

    resp, err := http.Get(url)

    if resp != nil {

        ch <- resp.StatusCode

        defer func() {

            _, err = io.Copy(ioutil.Discard, resp.Body)

            resp.Body.Close()

        }()

    } else {

        ch <- -1 // because main expect exactly totalHit values in ch

    }

    if err != nil {

        fmt.Println(err)

    }

    wg.Done()

}

Answer 2

您可以使用以下库：

Requests：一个 Go 库，用于减少发出 HTTP 请求时的麻烦（20k/s 请求）

https://github.com/alessiosavi/Requests

这个想法是分配一个请求列表，然后使用可配置的“并行”因子发送它们，该因子允许一次仅运行“N”个请求。

// This array will contains the list of request

var reqs []requests.Request

// N is the number of request to run in parallel, in order to avoid "TO MANY OPEN FILES. N have to be lower than ulimit threshold"

var N int = 12

// Create the list of request

for i := 0; i < 1000; i++ {

    // In this case, we init 1000 request with same URL,METHOD,BODY,HEADERS 

    req, err := requests.InitRequest("https://127.0.0.1:5000", "GET", nil, nil, true) 

    if err != nil {

        // Request is not compliant, and will not be add to the list

        log.Println("Skipping request [", i, "]. Error: ", err)

    } else {

        // If no error occurs, we can append the request created to the list of request that we need to send

        reqs = append(reqs, *req)

    }

}

此时，我们有一个包含必须发送的请求的列表。让我们并行发送它们吧！

// This array will contains the response from the givens request

var response []datastructure.Response

// send the request using N request to send in parallel

response = requests.ParallelRequest(reqs, N)

// Print the response

for i := range response {

    // Dump is a method that print every information related to the response

    log.Println("Request [", i, "] -> ", response[i].Dump())

    // Or use the data present in the response

    log.Println("Headers: ", response[i].Headers)

    log.Println("Status code: ", response[i].StatusCode)

    log.Println("Time elapsed: ", response[i].Time)

    log.Println("Error: ", response[i].Error)

    log.Println("Body: ", string(response[i].Body))

}

您可以在存储库的示例文件夹中找到示例用法。