Java使用Sharding-JDBC分庫(kù)分表進(jìn)行操作

更新時(shí)間：2021年08月17日 16:41:32 作者：輻射工兵

Sharding-JDBC 是無侵入式的 MySQL 分庫(kù)分表操作工具，本文主要介紹了Java使用Sharding-JDBC分庫(kù)分表進(jìn)行操作，感興趣的可以了解一下

Sharding-JDBC 是無侵入式的 MySQL 分庫(kù)分表操作工具，所有庫(kù)表設(shè)置僅需要在配置文件中配置即可，無須修改任何代碼。
本文寫了一個(gè) Demo，使用的是 SpringBoot 框架，通過 Docker 進(jìn)行 MySQL 實(shí)例管理，分庫(kù)分表結(jié)構(gòu)如下圖，同時(shí)所有的庫(kù)都進(jìn)行了主從復(fù)制：

主從庫(kù)搭建

Docker 項(xiàng)目結(jié)構(gòu)：

docker
├── docker-compose.yml
├── master
│   ├── data
│   ├── log
│   │   └── error.log
│   ├── my.cnf
│   └── mysql-files  # Win 需要，Linux 不需要
├── README.md
└── slave
    ├── data
    ├── log
    │   └── error.log
    ├── my.cnf
    └── mysql-files

Compose File

version: '3'

networks:
  sharding-jdbc-demo:
    driver: bridge
    ipam:
      config:
        - subnet: 172.25.0.0/24

services:
  master:
    image: mysql
    container_name: sharding-jdbc-demo-master
    ports:
      - "3307:3306"
    volumes:
      - "./master/data:/var/lib/mysql"
      - "./master/mysql-files:/var/lib/mysql-files"         # win 下的 MySQL8 需要，Linux 不需要
      - "./master/log/error.log:/var/log/mysql/error.log"
      - "./master/my.cnf:/etc/mysql/my.cnf"
    environment:
      MYSQL_ROOT_PASSWORD: 123456
    entrypoint: bash -c "chown -R mysql:mysql /var/log/mysql && chmod 644 /etc/mysql/my.cnf && exec /entrypoint.sh mysqld"
    restart: unless-stopped
    networks:
      sharding-jdbc-demo:
        ipv4_address: 172.25.0.101

  slave:
    image: mysql
    container_name: sharding-jdbc-demo-slave
    ports:
      - "3308:3306"
    volumes:
      - "./slave/data:/var/lib/mysql"
      - "./slave/mysql-files:/var/lib/mysql-files"
      - "./slave/log/error.log:/var/log/mysql/error.log"
      - "./slave/my.cnf:/etc/mysql/my.cnf"
    environment:
      MYSQL_ROOT_PASSWORD: 123456
    entrypoint: bash -c "chown -R mysql:mysql /var/log/mysql && chmod 644 /etc/mysql/my.cnf  && exec /entrypoint.sh mysqld"
    restart: unless-stopped
    networks:
      sharding-jdbc-demo:
        ipv4_address: 172.25.0.102

Master 配置

[mysqld]
pid-file         = /var/run/mysqld/mysqld.pid
socket           = /var/run/mysqld/mysqld.sock
datadir          = /var/lib/mysql
log-error        = /var/log/mysql/error.log
bind-address     = 0.0.0.0
secure-file-priv = NULL
max_connections  = 16384

character-set-server = utf8mb4
collation-server     = utf8mb4_general_ci
init_connect         ='SET NAMES utf8mb4'
skip-name-resolve

server_id = 1
log-bin = mysql-bin
binlog-do-db = db_order_1 # 復(fù)制 db_order_1
binlog-do-db = db_order_2 # 復(fù)制 db_order_2
binlog-do-db = db_user    # 復(fù)制 db_user
log-slave-updates
sync_binlog = 1
auto_increment_offset = 1
auto_increment_increment = 1
expire_logs_days = 7
log_bin_trust_function_creators = 1

# Custom config should go here
!includedir /etc/mysql/conf.d/

Slave 配置

[mysqld]
pid-file         = /var/run/mysqld/mysqld.pid
socket           = /var/run/mysqld/mysqld.sock
datadir          = /var/lib/mysql
log-error        = /var/log/mysql/error.log
bind-address     = 0.0.0.0
secure-file-priv = NULL
max_connections  = 16384

character-set-server = utf8mb4
collation-server     = utf8mb4_general_ci
init_connect         ='SET NAMES utf8mb4'
skip-name-resolve
skip-host-cache

server_id = 2
log-bin = mysql-bin
log-slave-updates
sync_binlog = 0
innodb_flush_log_at_trx_commit = 0  # 提交策略
replicate-do-db = db_order_1        # 復(fù)制 db_order_1
replicate-do-db = db_order_2        # 復(fù)制 db_order_2
replicate-do-db = db_user           # 復(fù)制 db_user
slave-net-timeout = 60              # 重連時(shí)間
log_bin_trust_function_creators = 1

# Custom config should go here
!includedir /etc/mysql/conf.d/

主從配置

啟動(dòng)容器 docker compose up -d;
登錄 Master mysql -uroot -h 127.0.0.1 -P 3307 -p ;
查看 master 狀態(tài)。

mysql> show master status\G
*************************** 1. row ***************************
File: mysql-bin.000004                            # 記住 Bin log 當(dāng)前文件名稱
Position: 156                                     # 記住 Bin log 當(dāng)前偏移量
Binlog_Do_DB: db_order_1,db_order_2,db_user       # 確認(rèn)復(fù)制數(shù)據(jù)庫(kù)是否正確
Binlog_Ignore_DB:
Executed_Gtid_Set:

登錄 Slave mysql -uroot -h 127.0.0.1 -P 3308 -p
設(shè)置 Master 連接，注意 host 與 port 是內(nèi)網(wǎng)的地址和端口。

mysql> change master to master_host='172.25.0.101',
    master_user='root',
    master_password='123456',
    master_port=3306,
    master_log_file='mysql-bin.000004',
    master_log_pos=156;

啟動(dòng)同步

mysql> start slave;

查看 Slave 狀態(tài)，若 Slave_IO 與 Slave_SQL 都在運(yùn)行為 YES 即成功。

mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for source to send event
Master_Host: 172.25.0.101
Master_User: root
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000004
Read_Master_Log_Pos: 156
Relay_Log_File: d2a706a02933-relay-bin.000002
Relay_Log_Pos: 324
Relay_Master_Log_File: mysql-bin.000004
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB: db_order_1,db_order_2,db_user

創(chuàng)建分庫(kù)分表

登錄 Master，創(chuàng)建數(shù)據(jù)庫(kù)：

CREATE DATABASE db_order_1;
CREATE DATABASE db_order_2;
CREATE DATABASE db_user;

此時(shí)從庫(kù)也會(huì)創(chuàng)建數(shù)據(jù)庫(kù)，若沒有，則是主從配置失敗了。
此時(shí)已完成垂直分庫(kù)和水平分庫(kù)。接下來創(chuàng)建數(shù)據(jù)表：

Order 1 庫(kù)

先 USE db_order_1;，再分別創(chuàng)建 t_dict 全局表、t_order_1 和 t_order_2 水平分表。

DROP TABLE IF EXISTS `t_dict`;
CREATE TABLE `t_dict`
(
    `id`         int                                    NOT NULL AUTO_INCREMENT,
    `type`       int                                    NOT NULL,
    `enum_value` int                                    NOT NULL,
    `name`       varchar(64) COLLATE utf8mb4_general_ci NOT NULL DEFAULT '',
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  AUTO_INCREMENT = 7
  DEFAULT CHARSET = utf8mb4
  COLLATE = utf8mb4_general_ci;
LOCK TABLES `t_dict` WRITE;
INSERT INTO `t_dict` VALUES (1,1,0,'未定義'),(2,1,1,'未付款'),(3,1,2,'已付款'),(4,1,3,'退款中'),(5,1,4,'已退款'),(6,1,5,'已完成'),(7,2,0,'未定義'),(8,2,1,'已創(chuàng)建'),(9,2,2,'已驗(yàn)證'),(10,2,3,'已凍結(jié)'),(11,2,4,'已注銷'),(12,2,5,'已刪除');
UNLOCK TABLES;

DROP TABLE IF EXISTS `t_order_1`;
CREATE TABLE `t_order_1`
(
    `id`      bigint         NOT NULL,
    `user_id` bigint         NOT NULL,
    `price`   decimal(10, 2) NOT NULL,
    `status`  int            NOT NULL DEFAULT '1',
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  DEFAULT CHARSET = utf8mb4
  COLLATE = utf8mb4_general_ci;

DROP TABLE IF EXISTS `t_order_2`;
CREATE TABLE `t_order_2`
(
    `id`      bigint         NOT NULL,
    `user_id` bigint         NOT NULL,
    `price`   decimal(10, 2) NOT NULL,
    `status`  int            NOT NULL DEFAULT '1',
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  DEFAULT CHARSET = utf8mb4
  COLLATE = utf8mb4_general_ci;

Order 2 庫(kù)

先 USE db_order_2;，再分別創(chuàng)建 t_dict 全局表、t_order_1 和 t_order_2 水平分表。所執(zhí)行 SQL 與 db_order_1 一致。

User 庫(kù)

先 USE db_user;，再分別創(chuàng)建 t_dict 全局表和 t_user 表，此處就不進(jìn)行水平或垂直分表了。垂直分表 sharding-jdbc 不會(huì)去處理，因?yàn)榇怪狈直碇缶褪钱惐懋悩?gòu)了，執(zhí)行 Join 操作就可以了，或者代碼進(jìn)行多次查詢實(shí)現(xiàn)。

DROP TABLE IF EXISTS `t_dict`;
CREATE TABLE `t_dict`
(
    `id`         int                                    NOT NULL AUTO_INCREMENT,
    `type`       int                                    NOT NULL,
    `enum_value` int                                    NOT NULL,
    `name`       varchar(64) COLLATE utf8mb4_general_ci NOT NULL DEFAULT '',
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  AUTO_INCREMENT = 7
  DEFAULT CHARSET = utf8mb4
  COLLATE = utf8mb4_general_ci;
LOCK TABLES `t_dict` WRITE;
INSERT INTO `t_dict` VALUES (1,1,0,'未定義'),(2,1,1,'未付款'),(3,1,2,'已付款'),(4,1,3,'退款中'),(5,1,4,'已退款'),(6,1,5,'已完成'),(7,2,0,'未定義'),(8,2,1,'已創(chuàng)建'),(9,2,2,'已驗(yàn)證'),(10,2,3,'已凍結(jié)'),(11,2,4,'已注銷'),(12,2,5,'已刪除');
UNLOCK TABLES;


DROP TABLE IF EXISTS `t_user`;
CREATE TABLE `t_user`
(
    `id`   bigint                                  NOT NULL AUTO_INCREMENT,
    `name` varchar(128) COLLATE utf8mb4_general_ci NOT NULL DEFAULT '',
    `type` int                                     NOT NULL DEFAULT '1',
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  AUTO_INCREMENT = 1426999086541635586
  DEFAULT CHARSET = utf8mb4
  COLLATE = utf8mb4_general_ci;

Sharding-JDBC 引入

Sharding-JDBC maven 包：

<!-- Sharding-jdbc -->
<dependency>
        <groupId>org.apache.shardingsphere</groupId>
        <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
        <version>4.1.1</version>
</dependency>

本 Demo 其他用到的依賴，分別是 Junit 測(cè)試、Lombok、MyBatis Plus、Druid 連接池、MySQL 驅(qū)動(dòng)、Java Faker 數(shù)據(jù)生成器：

<dependencies>
        <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-test</artifactId>
                <scope>test</scope>
        </dependency>
        <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.13.2</version>
                <scope>test</scope>
        </dependency>
        <dependency>
                <groupId>org.projectlombok</groupId>
                <artifactId>lombok</artifactId>
                <optional>true</optional>
        </dependency>
        <!-- MyBatis Plus -->
        <dependency>
                <groupId>com.baomidou</groupId>
                <artifactId>mybatis-plus-boot-starter</artifactId>
                <version>3.4.3.1</version>
        </dependency>
        <!-- Druid -->
        <dependency>
                <groupId>com.alibaba</groupId>
                <artifactId>druid</artifactId>
                <version>1.2.6</version>
        </dependency>
        <!-- MySQL -->
        <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
        </dependency>
        <!-- Sharding-jdbc -->
        <dependency>
                <groupId>org.apache.shardingsphere</groupId>
                <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
                <version>4.1.1</version>
        </dependency>
        <!-- Data Faker -->
        <dependency>
                <groupId>com.github.javafaker</groupId>
                <artifactId>javafaker</artifactId>
                <version>1.0.2</version>
        </dependency>
</dependencies>

Sharding-JDBC 配置

可選配置

啟用 SQL 打?。?br />

spring.shardingsphere.props.sql.show = true

數(shù)據(jù)源配置

總共有 t_order_1、t_order_2 和 t_user 三個(gè)庫(kù)，加上單主單從的復(fù)制，因此有 6 個(gè)數(shù)據(jù)庫(kù)，需要配置六個(gè)數(shù)據(jù)源：

# Datasource Define
spring.shardingsphere.datasource.names = o1-master,o2-master,o1-slave,o2-slave,u-master,u-slave
# datasource o1-master
spring.shardingsphere.datasource.o1-master.type = com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.o1-master.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.o1-master.url = jdbc:mysql://localhost:3307/db_order_1?useUnicode=true
spring.shardingsphere.datasource.o1-master.username = root
spring.shardingsphere.datasource.o1-master.password = 123456
# datasource o1-slave
spring.shardingsphere.datasource.o1-slave.type = com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.o1-slave.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.o1-slave.url = jdbc:mysql://localhost:3308/db_order_1?useUnicode=true
spring.shardingsphere.datasource.o1-slave.username = root
spring.shardingsphere.datasource.o1-slave.password = 123456
# datasource o2-master
spring.shardingsphere.datasource.o2-master.type = com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.o2-master.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.o2-master.url = jdbc:mysql://localhost:3307/db_order_2?useUnicode=true
spring.shardingsphere.datasource.o2-master.username = root
spring.shardingsphere.datasource.o2-master.password = 123456
# datasource o2-slave
spring.shardingsphere.datasource.o2-slave.type = com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.o2-slave.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.o2-slave.url = jdbc:mysql://localhost:3308/db_order_2?useUnicode=true
spring.shardingsphere.datasource.o2-slave.username = root
spring.shardingsphere.datasource.o2-slave.password = 123456
# datasource u-master
spring.shardingsphere.datasource.u-master.type = com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.u-master.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.u-master.url = jdbc:mysql://localhost:3307/db_user?useUnicode=true
spring.shardingsphere.datasource.u-master.username = root
spring.shardingsphere.datasource.u-master.password = 123456
# datasource u-slave
spring.shardingsphere.datasource.u-slave.type = com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.u-slave.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.u-slave.url = jdbc:mysql://localhost:3308/db_user?useUnicode=true
spring.shardingsphere.datasource.u-slave.username = root
spring.shardingsphere.datasource.u-slave.password = 123456

主從復(fù)制配置

主從配置不需要聲明，在定義時(shí)會(huì)自動(dòng)讀取 key 中的主從配置庫(kù)作為邏輯庫(kù)，如下面的 db-order-1。

# Replication Define
spring.shardingsphere.sharding.master-slave-rules.db-order-1.master-data-source-name=o1-master
spring.shardingsphere.sharding.master-slave-rules.db-order-1.slave-data-source-names=o1-slave
spring.shardingsphere.sharding.master-slave-rules.db-order-2.master-data-source-name=o2-master
spring.shardingsphere.sharding.master-slave-rules.db-order-2.slave-data-source-names=o2-slave
spring.shardingsphere.sharding.master-slave-rules.db-user.master-data-source-name=u-master
spring.shardingsphere.sharding.master-slave-rules.db-user.slave-data-source-names=u-slave

數(shù)據(jù)節(jié)點(diǎn)配置

數(shù)據(jù)節(jié)點(diǎn)，指的是每張數(shù)據(jù)表，由于存在分庫(kù)、分表、全局的不同類型，因此數(shù)據(jù)節(jié)點(diǎn)也有不同類型。注意，由于我們進(jìn)行了主從復(fù)制，因此這里的數(shù)據(jù)庫(kù)不能直接填數(shù)據(jù)源的名稱，應(yīng)該填在主從復(fù)制配置的 Key 中定義的名稱，如 db-user 而不是 u-master 或 u-slave。

全局表：

# BroadCast Table
spring.shardingsphere.sharding.broadcast-tables = t_dict

單庫(kù)單表：
key-generator.column 設(shè)置主鍵列。
key-generator.type 設(shè)置主鍵生成類型，這里使用雪花算法，其實(shí)沒必要因?yàn)椴皇欠直淼?，但是不填也?huì)默認(rèn)使用這個(gè)。

# Data Node t_user
spring.shardingsphere.sharding.tables.t_user.actual-data-nodes = db-user.t_user
spring.shardingsphere.sharding.tables.t_user.key-generator.column = id
spring.shardingsphere.sharding.tables.t_user.key-generator.type = SNOWFLAKE

分庫(kù)分表：
在 actual-data-nodes 中使用 groovy 表達(dá)式進(jìn)行設(shè)置。
在 database-strategy 中設(shè)置切分方式，具體自查，暫沒時(shí)間寫。

# Data Node t_order, If there is not master-salve-replication, use datasource name like "o$-master->{1..2}.t_order_$->{1..2}"
spring.shardingsphere.sharding.tables.t_order.actual-data-nodes = db-order-$->{1..2}.t_order_$->{1..2}
spring.shardingsphere.sharding.tables.t_order.key-generator.column = id
spring.shardingsphere.sharding.tables.t_order.key-generator.type = SNOWFLAKE
# database sharding strategy
spring.shardingsphere.sharding.tables.t_order.database-strategy.inline.sharding-column = user_id
spring.shardingsphere.sharding.tables.t_order.database-strategy.inline.algorithm-expression = db-order-$->{user_id % 2 + 1}
# table sharding strategy
spring.shardingsphere.sharding.tables.t_order.table-strategy.inline.sharding-column = id
spring.shardingsphere.sharding.tables.t_order.table-strategy.inline.algorithm-expression = t_order_$->{id % 2 + 1}

Demo 程序

參見：zoharyips/sharding-jdbc-demo (github.com)

到此這篇關(guān)于Java使用Sharding-JDBC分庫(kù)分表進(jìn)行操作的文章就介紹到這了,更多相關(guān)Java Sharding-JDBC分庫(kù)分表內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！

您可能感興趣的文章: