tornado的mongo驱动选择,pymongo,motor,asyncmongo还是其他?

Question

背景我用apache的abtest在公司的两台虚拟机上面测试,发现用pymongo的速度最快,asyncmongo其次,最后才是motor库.囧機器配置server端ubuntu12.04Intel(R)Core(TM)2DuoCPUE7500@2.93GHz內存500M基本的server配置全部打開˜(比如maxsoconn设到很大)client端freebsd(机子在公司,暂时空缺=.=)測試工具abtest测试用例我这里只贴了asyncmongo的用例,其他两个的代码结构类似,但跟其他业务结合得比较紧,所以就不贴了.大概的用例描述:客户端发起一个json格式的post请求,tornado这边根据player_id跟mongo要数据,只是读请求,不存在锁的问题.删了一些敏感的信息.#!/usr/bin/envpython#encoding:utf-8importasyncmongoimporttornado.webfromtornadoimportwebimporttornado.ioloopfromtornado.ioloopimportIOLoopfromtornado.httpserverimportHTTPServerclassRankHandler(tornado.web.RequestHandler):def__init__(self,application,request,**kwargs):super(RankHandler,self).__init__(application,request,**kwargs)self.set_header('Content-Type','application/json')@propertydefdb(self):[e-mail](self):r={}##decodemsgbodytry:d=tornado.escape.json_decode(self.request.body)exceptValueError,e:self.log.error('decodetrackdataerror.e=%s'%e)r['status_code']=500r['status_txt']='decodejsonerror'self.write(tornado.escape.json_encode(r))self.finish()returnevent=d.get('event')ifnotevent:self.log.error('trackargsmissingargevent.')r['status_code']=500r['status_txt']='missing_arg_event'self.write(tornado.escape.json_encode(r))self.finish()returnevent_data=d.get('data')ifevent_dataandnotisinstance(event_data,dict):self.log.error('trackargsbadargdata.')r['status_code']=500r['status_txt']='bad_arg_data'self.write(tornado.escape.json_encode(r))self.finish()returnif(event=="u_add"):passelif(event=="u_group"):passelif(event=="u_update"):passelif(event=="u_get_point"):self.db.ranking_list.find_one({"_id":event_data["player_id"]},callback=self._on_response)def_on_response(self,response,error):r={}iferror:raisetornado.web.HTTPError(500)result={"data":{"_id":response['_id'],"rank_point":response["rank_point"]}}r.update(result)ifnotr.get('status_code',None):r['status_code']=200r['status_txt']='OK'self.write(tornado.escape.json_encode(r))self.finish()returnclassApplication(web.Application):def__init__(self):""""""handlers=[(r"/api/xxx",RankHandler),]settings=dict(debug=True,autoescape=None,)super(Application,self).__init__(handlers,**settings)self.db=asyncmongo.Client(pool_id='mydb',host='0.0.0.0',port=27017,maxcached=10,maxconnections=20000,dbname='xxx')defmain():http_server=HTTPServer(Application(),xheaders=True)http_server.bind(8880,'127.0.0.1')http_server.start()IOLoop.instance().start()if__name__=="__main__":main()测试结果同步mongo庫➜testgit:(master)✗ab-n10000-c3000-pdata-get-user-rank_point.xml-T'application/json''http://192.168.0.201:8880/api/xxx'ThisisApacheBench,Version2.3Copyright1996AdamTwiss,ZeusTechnologyLtd,http://www.zeustech.net/LicensedtoTheApacheSoftwareFoundation,http://www.apache.org/Benchmarking192.168.0.201(bepatient)Completed1000requestsCompleted2000requestsCompleted3000requestsCompleted4000requestsCompleted5000requestsCompleted6000requestsCompleted7000requestsCompleted8000requestsCompleted9000requestsCompleted10000requestsFinished10000requestsServerSoftware:TornadoServer/3.1.1ServerHostname:192.168.0.201ServerPort:8880DocumentPath:/api/xxxDocumentLength:80bytesConcurrencyLevel:3000Timetakenfortests:23.551secondsCompleterequests:10000Failedrequests:0Writeerrors:0Totaltransferred:2170000bytesTotalbodysent:1990000HTMLtransferred:800000bytesRequestspersecond:424.61[#/sec](mean)Timeperrequest:7065.317[ms](mean)Timeperrequest:2.355[ms](mean,acrossallconcurrentrequests)Transferrate:89.98[Kbytes/sec]received82.52kb/ssent172.50kb/stotalConnectionTimes(ms)minmean[+/-sd]medianmaxConnect:118062222.9106110825Processing:26511302042.653920975Waiting:25510402018.951520972Total:28229362824.0293020976Percentageoftherequestsservedwithinacertaintime(ms)50%293066%340275%352680%359290%667095%682398%996199%15001100%20976(longestrequest)異步mongo庫(q2_rank)➜testgit:(master)✗ab-n10000-c3000-pdata-get-user-rank_point.xml-T'application/json''http://192.168.0.201:8880/api/xxx'ThisisApacheBench,Version2.3Copyright1996AdamTwiss,ZeusTechnologyLtd,http://www.zeustech.net/LicensedtoTheApacheSoftwareFoundation,http://www.apache.org/Benchmarking192.168.0.201(bepatient)Completed1000requestsCompleted2000requestsCompleted3000requestsCompleted4000requestsCompleted5000requestsCompleted6000requestsCompleted7000requestsCompleted8000requestsCompleted9000requestsCompleted10000requestsFinished10000requestsServerSoftware:TornadoServer/3.1.1ServerHostname:192.168.0.201ServerPort:8880DocumentPath:/api/xxxDocumentLength:80bytesConcurrencyLevel:3000Timetakenfortests:24.629secondsCompleterequests:10000Failedrequests:0Writeerrors:0Totaltransferred:2170000bytesTotalbodysent:1990000HTMLtransferred:800000bytesRequestspersecond:406.02[#/sec](mean)Timeperrequest:7388.749[ms](mean)Timeperrequest:2.463[ms](mean,acrossallconcurrentrequests)Transferrate:86.04[Kbytes/sec]received78.90kb/ssent164.95kb/stotalConnectionTimes(ms)minmean[+/-sd]medianmaxConnect:0412794.5176205Processing:102462861793.2708810475Waiting:83662561843.9708310468Total:103266981894.1719914014Percentageoftherequestsservedwithinacertaintime(ms)50%719966%770075%782580%787590%824495%916198%1036699%10763100%14014(longestrequest)98%6677結果分析理論上來說,異步的mongo應該可以處理更多的併發連接,但實際測試來看,併發連接數相差不大.我想到的原因:-测试机子太挫-測試的數據量不夠大,如果增大返回的數據大小,或者延長查詢的時間,這樣理論上motor性能會更好看點-我查看了mongod的日志,发现异步的驱动,每次都会跟mongo新生成一个连接,频繁的连接建立与删除,造成了性能上的损耗?而pymongo则从头到尾都是一个连接.最后有没童鞋做过这方面的性能测试呢,可否把测试结果分享下=.=?然后,大家能否推荐下一些比较好的结合方式?比如我在知乎上面看到过有人问类似的问题.