[Redis] HBase到Redis实时同步(管道复制Pipe)

HBase到Redis实时同步,个人觉得最好的当然是管道复制了。这里不便给出核心代码,仅给个可用的样例代码

关于性能:本身由于通过内存来做中转,考虑到实际所用数据的数据结构复杂度以及数据量,可能非常损耗CPU,请认真考虑多并发、以及分布式(多机器并发)等方法来提高性能。

实际使用,我们将并发稳定在中等速度  2亿+ hits/分钟 (机器比较多且性能很强)

示例:

    private int currentInPipe = 0;
    private int clusterSize = 0;
    private ClusterBatch pipeline = null;
    private static HashSet<HostAndPort> redisHosts = new HashSet<HostAndPort>();
    private static int connMode = 1;

    public String strSetUsePipe(String key, String value) throws RecommenderException {
        try {
            if(pipeline == null){
                pipeline = clientAdapter.getPipeline();
            }
            pipeline.set(key, value);
            currentInPipe++;
            if(currentInPipe == clusterSize*10){
                pipeline.sync();
                currentInPipe = 0;
            }
            return "";
        } catch (Exception e) {
            throw new RecommenderException("Failed to set key value");
        }
    }

    public void sync() throws RecommenderException {
        try {
            if(pipeline != null){
                pipeline.sync();
                currentInPipe = 0;
            }
        } catch (Exception e) {
            throw new RecommenderException("redis sync error!");
        }
    }

//调用的main方法
    public static void main(String[] args) throws Exception {
        String cfgFile = "E:\\workspace\\divinerconfig\\redis-site.properties";
        RedisConnectorImpl.initClientCfg(cfgFile);
        RedisConnectorImpl rConn = RedisConnectorImpl.getInstance();
        for (int i = 0; i < 100; i++) {
            rConn.strSetUsePipe("ttt" + i, "ttt" + i);
        }
        rConn.sync();
        String val = rConn.strGet("ttt" + 90);
        System.out.println(val);
    }
分类上一篇:     分类下一篇:

Leave a Reply