Spring?Boot實(shí)戰(zhàn)解決高并發(fā)數(shù)據(jù)入庫(kù)之?Redis?緩存+MySQL?批量入庫(kù)問(wèn)題
前言
最近在做閱讀類的業(yè)務(wù),需要記錄用戶的PV,UV;
項(xiàng)目狀況:前期嘗試業(yè)務(wù)階段;
特點(diǎn):
快速實(shí)現(xiàn)(不需要做太重,滿足初期推廣運(yùn)營(yíng)即可)快速投入市場(chǎng)去運(yùn)營(yíng)收集用戶的原始數(shù)據(jù),三要素:
誰(shuí)在什么時(shí)間閱讀哪篇文章提到PV,UV腦海中首先浮現(xiàn)特點(diǎn):
需要考慮性能(每個(gè)客戶每打開一篇文章進(jìn)行記錄)允許數(shù)據(jù)有較小誤差(少部分?jǐn)?shù)據(jù)丟失)
架構(gòu)設(shè)計(jì)
架構(gòu)圖:
時(shí)序圖
記錄基礎(chǔ)數(shù)據(jù)MySQL表結(jié)構(gòu)
CREATE TABLE `zh_article_count` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `bu_no` varchar(32) DEFAULT NULL COMMENT '業(yè)務(wù)編碼', `customer_id` varchar(32) DEFAULT NULL COMMENT '用戶編碼', `type` int(2) DEFAULT '0' COMMENT '統(tǒng)計(jì)類型:0APP內(nèi)文章閱讀', `article_no` varchar(32) DEFAULT NULL COMMENT '文章編碼', `read_time` datetime DEFAULT NULL COMMENT '閱讀時(shí)間', `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '創(chuàng)建時(shí)間', `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新時(shí)間', `param1` int(2) DEFAULT NULL COMMENT '預(yù)留字段1', `param2` int(4) DEFAULT NULL COMMENT '預(yù)留字段2', `param3` int(11) DEFAULT NULL COMMENT '預(yù)留字段3', `param4` varchar(20) DEFAULT NULL COMMENT '預(yù)留字段4', `param5` varchar(32) DEFAULT NULL COMMENT '預(yù)留字段5', `param6` varchar(64) DEFAULT NULL COMMENT '預(yù)留字段6', PRIMARY KEY (`id`) USING BTREE, UNIQUE KEY `uk_zh_article_count_buno` (`bu_no`), KEY `key_zh_article_count_csign` (`customer_id`), KEY `key_zh_article_count_ano` (`article_no`), KEY `key_zh_article_count_rtime` (`read_time`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='文章閱讀統(tǒng)計(jì)表';
技術(shù)實(shí)現(xiàn)方案
SpringBoot
Redis
MySQL
代碼實(shí)現(xiàn)
完整代碼(GitHub,歡迎大家Star,Fork,Watch)
https://github.com/dangnianchuntian/springboot
主要代碼展示
Controller
/* * Copyright (c) 2020. zhanghan_java@163.com All Rights Reserved. * 項(xiàng)目名稱:Spring Boot實(shí)戰(zhàn)解決高并發(fā)數(shù)據(jù)入庫(kù): Redis 緩存+MySQL 批量入庫(kù) * 類名稱:ArticleCountController.java * 創(chuàng)建人:張晗 * 聯(lián)系方式:zhanghan_java@163.com * 開源地址: https://github.com/dangnianchuntian/springboot * 博客地址: https://zhanghan.blog.csdn.net */ package com.zhanghan.zhredistodb.controller; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.validation.annotation.Validated; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.RestController; import com.zhanghan.zhredistodb.controller.request.PostArticleViewsRequest; import com.zhanghan.zhredistodb.service.ArticleCountService; @RestController public class ArticleCountController { @Autowired private ArticleCountService articleCountService; /** * 記錄用戶訪問(wèn)記錄 */ @RequestMapping(value = "/post/article/views", method = RequestMethod.POST) public Object postArticleViews(@RequestBody @Validated PostArticleViewsRequest postArticleViewsRequest) { return articleCountService.postArticleViews(postArticleViewsRequest); } /** * 批量將緩存中的數(shù)據(jù)同步到MySQL(模擬定時(shí)任務(wù)操作) */ @RequestMapping(value = "/post/batch", method = RequestMethod.POST) public Object postBatch() { return articleCountService.postBatchRedisToDb(); }
Service
/* * Copyright (c) 2020. zhanghan_java@163.com All Rights Reserved. * 項(xiàng)目名稱:Spring Boot實(shí)戰(zhàn)解決高并發(fā)數(shù)據(jù)入庫(kù): Redis 緩存+MySQL 批量入庫(kù) * 類名稱:ArticleCountServiceImpl.java * 創(chuàng)建人:張晗 * 聯(lián)系方式:zhanghan_java@163.com * 開源地址: https://github.com/dangnianchuntian/springboot * 博客地址: https://zhanghan.blog.csdn.net */ package com.zhanghan.zhredistodb.service.impl; import java.util.ArrayList; import java.util.Date; import java.util.List; import java.util.stream.Collectors; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.stereotype.Service; import org.springframework.util.CollectionUtils; import com.alibaba.fastjson.JSON; import com.zhanghan.zhredistodb.controller.request.PostArticleViewsRequest; import com.zhanghan.zhredistodb.dto.ArticleCountDto; import com.zhanghan.zhredistodb.mybatis.mapper.XArticleCountMapper; import com.zhanghan.zhredistodb.service.ArticleCountService; import com.zhanghan.zhredistodb.util.wrapper.WrapMapper; import cn.hutool.core.util.IdUtil; @Service public class ArticleCountServiceImpl implements ArticleCountService { private static Logger logger = LoggerFactory.getLogger(ArticleCountServiceImpl.class); @Autowired private RedisTemplate<String, String> strRedisTemplate; private XArticleCountMapper xArticleCountMapper; @Value("${zh.article.count.redis.key:zh}") private String zhArticleCountRedisKey; @Value("#{T(java.lang.Integer).parseInt('${zh..article.read.num:3}')}") private Integer articleReadNum; /** * 記錄用戶訪問(wèn)記錄 */ @Override public Object postArticleViews(PostArticleViewsRequest postArticleViewsRequest) { ArticleCountDto articleCountDto = new ArticleCountDto(); articleCountDto.setBuNo(IdUtil.simpleUUID()); articleCountDto.setCustomerId(postArticleViewsRequest.getCustomerId()); articleCountDto.setArticleNo(postArticleViewsRequest.getArticleNo()); articleCountDto.setReadTime(new Date()); String strArticleCountDto = JSON.toJSONString(articleCountDto); strRedisTemplate.opsForList().rightPush(zhArticleCountRedisKey, strArticleCountDto); return WrapMapper.ok(); } * 批量將緩存中的數(shù)據(jù)同步到MySQL public Object postBatchRedisToDb() { Date now = new Date(); while (true) { List<String> strArticleCountList = strRedisTemplate.opsForList().range(zhArticleCountRedisKey, 0, articleReadNum); if (CollectionUtils.isEmpty(strArticleCountList)) { return WrapMapper.ok(); } List<ArticleCountDto> articleCountDtoList = new ArrayList<>(); strArticleCountList.stream().forEach(x -> { ArticleCountDto articleCountDto = JSON.parseObject(x, ArticleCountDto.class); articleCountDtoList.add(articleCountDto); }); //過(guò)濾出本次定時(shí)任務(wù)之前的緩存中數(shù)據(jù),防止死循環(huán) List<ArticleCountDto> beforeArticleCountDtoList = articleCountDtoList.stream().filter(x -> x.getReadTime() .before(now)).collect(Collectors.toList()); if (CollectionUtils.isEmpty(beforeArticleCountDtoList)) { xArticleCountMapper.batchAdd(beforeArticleCountDtoList); Integer delSize = beforeArticleCountDtoList.size(); strRedisTemplate.opsForList().trim(zhArticleCountRedisKey, delSize, -1L); } }
測(cè)試
模擬用戶請(qǐng)求訪問(wèn)后臺(tái)(多次請(qǐng)求)
查看緩存中訪問(wèn)數(shù)據(jù)
模擬定時(shí)任務(wù)將緩存中數(shù)據(jù)同步到DB中
這時(shí)查看緩存中的數(shù)據(jù)已經(jīng)沒(méi)了
查看數(shù)據(jù)庫(kù)表結(jié)構(gòu)
總結(jié)
- 項(xiàng)目中定時(shí)任務(wù)
- 問(wèn)演示方便用http代替定時(shí)任務(wù)調(diào)度;實(shí)際項(xiàng)目中用XXL-job,參考:定時(shí)任務(wù)的選型及改造
- 定時(shí)任務(wù)項(xiàng)目中用redis鎖防止并發(fā)(定時(shí)任務(wù)調(diào)度端多次調(diào)度等),參考:Redis實(shí)現(xiàn)計(jì)數(shù)器—接口防刷—升級(jí)版(Redis+Lua)
- 后期運(yùn)營(yíng)數(shù)據(jù)可以從閱讀記錄表中拉數(shù)據(jù)進(jìn)行相關(guān)分析
- 訪問(wèn)量大:可以將MySQL中的閱讀記錄表定時(shí)遷移走(MySQL建歷史表,MongoDB等)
到此這篇關(guān)于Spring Boot實(shí)戰(zhàn)解決高并發(fā)數(shù)據(jù)入庫(kù)之 Redis 緩存+MySQL 批量入庫(kù)的文章就介紹到這了,更多相關(guān)Spring Boot高并發(fā)數(shù)據(jù)入庫(kù)內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家!
相關(guān)文章
Redis對(duì)象與redisObject超詳細(xì)分析源碼層
這篇文章主要介紹了Redis對(duì)象與redisObject源碼層的分析,文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)吧2022-11-11基于Redis6.2.6版本部署Redis?Cluster集群的問(wèn)題
這篇文章主要介紹了基于Redis6.2.6版本部署Redis?Cluster集群,本文給大家介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值,需要的朋友可以參考下2022-04-04Redis在項(xiàng)目中常見的12種使用場(chǎng)景示例和說(shuō)明
Redis是一個(gè)開源的高性能鍵值對(duì)數(shù)據(jù)庫(kù),它以其內(nèi)存中數(shù)據(jù)存儲(chǔ)、鍵過(guò)期策略、持久化、事務(wù)、豐富的數(shù)據(jù)類型支持以及原子操作等特性,在許多項(xiàng)目中扮演著關(guān)鍵角色,以下是整理的12個(gè)Redis在項(xiàng)目中常見的使用場(chǎng)景舉例說(shuō)明和解釋2024-06-06