Sharing-jdbc Configuration Analysis

Sharing core configuration is mainly as follows (official website):

  • Fragmentation rule

    Total entry for fragmentation rule configuration. Contains data source configuration, table configuration, binding table configuration, read-write separation configuration, and so on.

  • Data source configuration

    List of Real Data Sources

  • Table configuration

    Configuration of Logical Table Name, Data Node and Tabulation Rules

  • Data Node Configuration

    Used to configure the mapping relationship between logical tables and real tables. It can be divided into two forms: uniform distribution and self-defined distribution.

  • Fragmentation policy configuration

    For fragmentation strategy, there are two dimensions: data source fragmentation strategy and table fragmentation strategy.

    • Data Source Fragmentation Strategy:

      It corresponds to the Database Sharing Strategy. Target data sources for configuring data allocation

    • Table Fragmentation Strategy
      It corresponds to the TableSharing Strategy. A target table for configuring the allocated data that exists within the target data source of the data. So table fragmentation strategy is dependent on the results of data source fragmentation strategy
  • Self-increasing Primary Key Generation Strategy

    Distributed primary keys are not duplicated by generating self-increasing primary keys in client instead of original self-increasing primary keys in database.

 

Next, the core configurations are analyzed.

  1. Take multi-master and multi-slave read-write separation and table fragmentation as examples
    public final class ShardingMasterSlaveConfigurationPrecise implements ExampleConfiguration {
        
        @Override
        public DataSource getDataSource() throws SQLException {
            ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();
            //Fragmentation table rule configuration
            shardingRuleConfig.getTableRuleConfigs().add(getOrderTableRuleConfiguration());
            shardingRuleConfig.getTableRuleConfigs().add(getOrderItemTableRuleConfiguration());
            shardingRuleConfig.getBindingTableGroups().add("t_order, t_order_item");
            //Setting Default Data Source Fragmentation Policy
            shardingRuleConfig.setDefaultDatabaseShardingStrategyConfig(new StandardShardingStrategyConfiguration("user_id", new PreciseModuloShardingDatabaseAlgorithm()));
            //Setting default table fragmentation policy
            shardingRuleConfig.setDefaultTableShardingStrategyConfig(new StandardShardingStrategyConfiguration("order_id", new PreciseModuloShardingTableAlgorithm()));
            //Master-slave configuration to support multi-master and multi-slave
            shardingRuleConfig.setMasterSlaveRuleConfigs(getMasterSlaveRuleConfigurations());
            //Create Sharing Data Source Data Source
            return ShardingDataSourceFactory.createDataSource(createDataSourceMap(), shardingRuleConfig, new Properties());
        }
        
        private static TableRuleConfiguration getOrderTableRuleConfiguration() {
            //Piecewise Table Configuration
            TableRuleConfiguration result = new TableRuleConfiguration(/*Logic table*/"t_order", /*Data source names. True tables*/"ds_${0..1}.t_order_${[0, 1]}");
            //Custom Primary Key Generation Configuration
            result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_id", getProperties()));
            return result;
        }
        
        private static TableRuleConfiguration getOrderItemTableRuleConfiguration() {
            TableRuleConfiguration result = new TableRuleConfiguration("t_order_item", "ds_${0..1}.t_order_item_${[0, 1]}");
            result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_item_id", getProperties()));
            return result;
        }
        
        private static List<MasterSlaveRuleConfiguration> getMasterSlaveRuleConfigurations() {
            MasterSlaveRuleConfiguration masterSlaveRuleConfig1 = new MasterSlaveRuleConfiguration("ds_0", "demo_ds_master_0", Arrays.asList("demo_ds_master_0_slave_0", "demo_ds_master_0_slave_1"));
            MasterSlaveRuleConfiguration masterSlaveRuleConfig2 = new MasterSlaveRuleConfiguration("ds_1", "demo_ds_master_1", Arrays.asList("demo_ds_master_1_slave_0", "demo_ds_master_1_slave_1"));
            return Lists.newArrayList(masterSlaveRuleConfig1, masterSlaveRuleConfig2);
        }
        
        private static Map<String, DataSource> createDataSourceMap() {
            final Map<String, DataSource> result = new HashMap<>();
            result.put("demo_ds_master_0", DataSourceUtil.createDataSource("demo_ds_master_0"));
            result.put("demo_ds_master_0_slave_0", DataSourceUtil.createDataSource("demo_ds_master_0_slave_0"));
            result.put("demo_ds_master_0_slave_1", DataSourceUtil.createDataSource("demo_ds_master_0_slave_1"));
            result.put("demo_ds_master_1", DataSourceUtil.createDataSource("demo_ds_master_1"));
            result.put("demo_ds_master_1_slave_0", DataSourceUtil.createDataSource("demo_ds_master_1_slave_0"));
            result.put("demo_ds_master_1_slave_1", DataSourceUtil.createDataSource("demo_ds_master_1_slave_1"));
            return result;
        }
        
        private static Properties getProperties() {
            Properties result = new Properties();
            result.setProperty("worker.id", "123");
            return result;
        }
    }
  2. Sharing Rule Configuration Fragmentation Rule Core Configuration
    @Getter
    @Setter
    public final class ShardingRuleConfiguration implements RuleConfiguration {
        //Table Rule Configuration
        private Collection<TableRuleConfiguration> tableRuleConfigs = new LinkedList<>();
    
        //Binding table configuration
        private Collection<String> bindingTableGroups = new LinkedList<>();
    
        //Broadcast table configuration
        private Collection<String> broadcastTables = new LinkedList<>();
    
        //Default data source name
        private String defaultDataSourceName;
    
        //Default repository policy
        private ShardingStrategyConfiguration defaultDatabaseShardingStrategyConfig;
    
        //Default fragmentation strategy
        private ShardingStrategyConfiguration defaultTableShardingStrategyConfig;
    
        //Default Primary Key Generation Tool Class
        private KeyGeneratorConfiguration defaultKeyGeneratorConfig;
    
        //Master-slave rule configuration
        private Collection<MasterSlaveRuleConfiguration> masterSlaveRuleConfigs = new LinkedList<>();
    
        //Data Desensitization Rule Configuration
        private EncryptRuleConfiguration encryptRuleConfig;
    }

     

  3. Analysis of Sharing Rule Configuration in Sharing Data Source Factory #createDataSource
    /**
     * Sharding data source factory.
     * 
     * @author zhangliang 
     */
    @NoArgsConstructor(access = AccessLevel.PRIVATE)
    public final class ShardingDataSourceFactory {
        
        /**
         * Create sharding data source.
         *
         * @param dataSourceMap data source map
         * @param shardingRuleConfig rule configuration for databases and tables sharding
         * @param props properties for data source
         * @return sharding data source
         * @throws SQLException SQL exception
         */
        public static DataSource createDataSource(
                final Map<String, DataSource> dataSourceMap, final ShardingRuleConfiguration shardingRuleConfig, final Properties props) throws SQLException {
            //Create Sharing Data Source Data Source and Sharing Rule Configuration
            return new ShardingDataSource(dataSourceMap, new ShardingRule(shardingRuleConfig, dataSourceMap.keySet()), props);
        }
    }
    

     
  4. Sharding Rule Fragmentation Rule
    public ShardingRule(final ShardingRuleConfiguration shardingRuleConfig, final Collection<String> dataSourceNames) {
            Preconditions.checkArgument(null != shardingRuleConfig, "ShardingRuleConfig cannot be null.");
            Preconditions.checkArgument(null != dataSourceNames && !dataSourceNames.isEmpty(), "Data sources cannot be empty.");
            this.shardingRuleConfig = shardingRuleConfig;
            //Sharing data source name
            shardingDataSourceNames = new ShardingDataSourceNames(shardingRuleConfig, dataSourceNames);
            //Create a set of table rules
            tableRules = createTableRules(shardingRuleConfig);
            //Create group binding tables
            //1. Find from tableRules
            //2. Whether broadcasting table, if broadcasting table, dataSourceNames. size () == 1? DataSourceNames. iterator (). next (): shardingRuleConfig. getDefaultDataSourceName ();
            //3. If both cases are not satisfied, the default data source is used to create tableRule.
            bindingTableRules = createBindingTableRules(shardingRuleConfig.getBindingTableGroups());
            //Broadcast table
            broadcastTables = shardingRuleConfig.getBroadcastTables();
            //Default repository policy
            defaultDatabaseShardingStrategy = createDefaultShardingStrategy(shardingRuleConfig.getDefaultDatabaseShardingStrategyConfig());
            //Default table fragmentation strategy
            defaultTableShardingStrategy = createDefaultShardingStrategy(shardingRuleConfig.getDefaultTableShardingStrategyConfig());
            //Default Primary Key Generation Tool Class
            defaultShardingKeyGenerator = createDefaultKeyGenerator(shardingRuleConfig.getDefaultKeyGeneratorConfig());
            //Create master-slave rules
            masterSlaveRules = createMasterSlaveRules(shardingRuleConfig.getMasterSlaveRuleConfigs());
            //Data desensitization rules
            encryptRule = createEncryptRule(shardingRuleConfig.getEncryptRuleConfig());
        }
        

     

  5. TableRule table rules
    public TableRule(final TableRuleConfiguration tableRuleConfig, final ShardingDataSourceNames shardingDataSourceNames, final String defaultGenerateKeyColumn) {
            //Get the logical table
            logicTable = tableRuleConfig.getLogicTable().toLowerCase();
            //The inline expression resolves the real table, such as ds_${0.1}. t_order_${[0,1]}.
            //It can be interpreted as follows:
            //ds_0.t_order_0
            //ds_0.t_order_1
            //ds_1.t_order_0
            //ds_1.t_order_1
            List<String> dataNodes = new InlineExpressionParser(tableRuleConfig.getActualDataNodes()).splitAndEvaluate();
    
            dataNodeIndexMap = new HashMap<>(dataNodes.size(), 1);
    
            //True Table Data Node
            //1. If no real tables are configured, the corresponding real table data nodes are generated according to logical tables and data sources.
            //2. Otherwise, the real data nodes are generated from the real tables parsed from the inline expression.
            actualDataNodes = isEmptyDataNodes(dataNodes)
                ? generateDataNodes(tableRuleConfig.getLogicTable(), shardingDataSourceNames.getDataSourceNames()) : generateDataNodes(dataNodes, shardingDataSourceNames.getDataSourceNames());
            //True table
            actualTables = getActualTables();
            //Data Source - > Fragmentation Strategy
            databaseShardingStrategy = null == tableRuleConfig.getDatabaseShardingStrategyConfig() ? null : ShardingStrategyFactory.newInstance(tableRuleConfig.getDatabaseShardingStrategyConfig());
            //Table - > Fragmentation Strategy
            tableShardingStrategy = null == tableRuleConfig.getTableShardingStrategyConfig() ? null : ShardingStrategyFactory.newInstance(tableRuleConfig.getTableShardingStrategyConfig());
            //PhotoID
            generateKeyColumn = getGenerateKeyColumn(tableRuleConfig.getKeyGeneratorConfig(), defaultGenerateKeyColumn);
            //Generate primary key tool classes, such as SNOWFLAKE
            shardingKeyGenerator = containsKeyGeneratorConfiguration(tableRuleConfig)
                    ? new ShardingKeyGeneratorServiceLoader().newService(tableRuleConfig.getKeyGeneratorConfig().getType(), tableRuleConfig.getKeyGeneratorConfig().getProperties()) : null;
        }

     

Tags: Programming Database SQL

Posted on Thu, 29 Aug 2019 19:51:48 -0700 by bastien