基于Golang 高并發(fā)問題的解決方案
Golang 高并發(fā)問題的解決
Golang在高并發(fā)問題上,由于協(xié)程的使用,相對于其他編程語言,已經(jīng)有了很大的優(yōu)勢,即相同的配置上,Golang可以以更低的代價處理更多的線程,同樣的線程數(shù),占用更低的資源!及時這樣,只是解決了一部分問題而已,因為在每個協(xié)程里,處理邏輯還是會有問題。
高并發(fā)時,還是要考慮服務器所能承受的最大壓力,數(shù)據(jù)庫讀取時的io問題,連接數(shù)問題,帶寬問題等等
研究了一下并發(fā)解決方案,在此記錄一下
參考文章:Handling 1 Million Requests per Minute with Go
地址:http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/
代碼如下:
//================================== // * Name:Jerry // * Tel:18017448610 // * DateTime:2019/2/24 14:02 //================================== package main import ( "github.com/lunny/log" "runtime" "sync" "time" ) //工廠模型 type Factory struct { Wg *sync.WaitGroup //任務監(jiān)控系統(tǒng) MaxWorker int //最大機器數(shù) MaxJobs int //最大工作數(shù)量 JobQueue chan int //工作隊列管道 Quit chan bool //是否關(guān)閉機器 } //創(chuàng)建工廠模型 func NewFactory(maxWorker int, wg *sync.WaitGroup) Factory { return Factory{ Wg: wg, //引用任務監(jiān)控系統(tǒng) MaxWorker: maxWorker, //機器數(shù)量(數(shù)量多少,根據(jù)服務器性能而定) JobQueue: make(chan int, maxWorker), //工作管道,數(shù)量大于等于機器數(shù) Quit: make(chan bool), } } //設置最大訂單數(shù)量 func (f *Factory) SetMaxJobs(taskNum int) { f.MaxJobs = taskNum } //開始上班 func (f *Factory) Start() { //機器開機,MaxWorker for i := 0; i < f.MaxWorker; i++ { //每一臺機器開啟后,去工作吧 go func() { //等待下發(fā)命令 for { select { case i := <-f.JobQueue: //接到工作,開工! f.doWork(i) case <-f.Quit: log.Println("機器關(guān)機") return } } }() } } //分配每個任務到管道中 func (f *Factory) AddTask(taskNum int) { //系統(tǒng)監(jiān)控任務 +1 f.Wg.Add(1) //分配任務到管道中 f.JobQueue <- taskNum } //模擬耗時工作 func (f *Factory) doWork(taskNum int) { //生產(chǎn)產(chǎn)品的工作 time.Sleep(200 * time.Millisecond) //完成工作報告 f.Wg.Done() //log.Println("完工:", taskNum) } //創(chuàng)建工廠 func Begin() { //配置工作核數(shù) gomaxprocs := runtime.GOMAXPROCS(runtime.NumCPU()) log.Println("核數(shù):", gomaxprocs) //配置監(jiān)控系統(tǒng) wg := new(sync.WaitGroup) //開工廠 factory := NewFactory(1000, wg) //訂單量 factory.SetMaxJobs(10000) //開始上班 factory.Start() log.Println("開始生產(chǎn)") //講所有的訂單,添加到任務隊列 for i := 0; i < factory.MaxJobs; i++ { factory.AddTask(i) } factory.Wg.Wait() log.Println("所有訂單任務生產(chǎn)完成") }
測試代碼及結(jié)果
上面代碼中,MaxWorker的數(shù)量很重要,取決于服務器所能承受的壓力,當然也不能無限增大,合理數(shù)值效率最高(具體多少合適,自己測試)
代碼:
func Benchmark_Begin(b *testing.B) { Begin() }
結(jié)果:
1000臺機器(協(xié)程),10000的工作量,我的個人PC測試結(jié)果如下:
2019/02/26 16:42:31 核數(shù): 4
2019/02/26 16:42:31 開始生產(chǎn)
2019/02/26 16:42:33 所有訂單任務生產(chǎn)完成
goos: windows
goarch: amd64
pkg: day11
Benchmark_hight2-4 1 2035574000 ns/op
PASS
Process finished with exit code 0
總結(jié):
此方法僅僅是在代碼層面解決一定的問題,高并發(fā) 產(chǎn)生的原因還包括其他原因,如帶寬,數(shù)據(jù)庫讀取速度等等,還需加大帶寬,多級數(shù)據(jù)庫,優(yōu)化數(shù)據(jù)的檢索等等方法
補充:golang 高并發(fā)任務處理方案
這個主要用golang 的chan 和routine屬性做的,比很多語言方便多了,可以參考參考
//任務的請求 type MtaskRequest struct { Ceshi int // [redacted] } //job隊列+work池 var ( MaxWorker = os.Getenv("MAX_WORKERS") MaxQueue = os.Getenv("MAX_QUEUE") ) // Job represents the job to be run type Job struct { MtaskRequest MtaskRequest } // A buffered channel that we can send work requests on. // var JobQueue chan Job ---這樣申明會卡主,沒有初始化 var JobQueue = make(chan Job) // Worker represents the worker that executes the job type Worker struct { WorkerPool chan chan Job JobChannel chan Job quit chan bool } func NewWorker(workerPool chan chan Job) Worker { return Worker{ WorkerPool: workerPool, JobChannel: make(chan Job), quit: make(chan bool)} } // Stop signals the worker to stop listening for work requests. func (w Worker) Stop() { go func() { w.quit <- true }() } type Dispatcher struct { // A pool of workers channels that are registered with the dispatcher WorkerPool chan chan Job maxWorkers int } func NewDispatcher(maxWorkers int) *Dispatcher { pool := make(chan chan Job, maxWorkers) return &Dispatcher{WorkerPool: pool, maxWorkers: maxWorkers} } // Start method starts the run loop for the worker, listening for a quit channel in // case we need to stop it func (w Worker) Start() { go func() { for { // register the current worker into the worker queue. w.WorkerPool <- w.JobChannel select { case <-w.JobChannel: time.Sleep(5 * time.Second) // we have received a work request. fmt.Println("調(diào)起worker") case <-w.quit: // we have received a signal to stop return //不能寫default } } }() } func (d *Dispatcher) Run() { //啟動一定數(shù)量的worker fmt.Println("啟動一定數(shù)量的worker") for i := 0; i < d.maxWorkers; i++ { worker := NewWorker(d.WorkerPool) worker.Start() } go d.dispatch() } //分派任務 func (d *Dispatcher) dispatch() { for { select { case job := <-JobQueue: //接收一個job請求 fmt.Println("JobQueue 收到請求") go func(job Job) { // try to obtain a worker job channel that is available. // this will block until a worker is idle jobChannel := <-d.WorkerPool // dispatch the job to the worker job channel jobChannel <- job }(job) } } } //接收到紅包數(shù)據(jù) func (this *TaskRedbao) UserGetRedbao(red_id, uid, shop_id, rand_arr, Amoney string) error { fmt.Println("收到 接收到紅包數(shù)據(jù) http請求") mtaskRequest := MtaskRequest{67} work := Job{MtaskRequest: mtaskRequest} JobQueue <- work return nil }
以上為個人經(jīng)驗,希望能給大家一個參考,也希望大家多多支持腳本之家。如有錯誤或未考慮完全的地方,望不吝賜教。
相關(guān)文章
Golang中omitempty關(guān)鍵字的具體實現(xiàn)
本文主要介紹了Golang中omitempty關(guān)鍵字的具體實現(xiàn),文中通過示例代碼介紹的非常詳細,具有一定的參考價值,感興趣的小伙伴們可以參考一下2022-01-01