AWS CLIを使ってマルチAZなElastiCache Redisのリードレプリカを作成する

aws_logoredis

ゴール

AWS ElastiCache は RDS と同じくマルチ AZ に対応している。

Q: Does Amazon ElastiCache for Redis support Multi-AZ operation?

Yes, with Amazon ElastiCache for Redis you can create a read replica in another AWS Availability Zone. Upon a failure of the primary node, we will provision a new primary node. In scenarios where the primary node cannot be provisioned, you can decide which read replica to promote to be the new primary. For more details on how to handle node failures see here.

http://aws.amazon.com/elasticache/faqs/

VPC 内に AWS ElastiCache Redis クラスタを作成し、リードレプリカをマルチAZに配置する方法をメモ。
諸般の事情により、構築はコマンドラインツールの AWS CLI から行う。

AWS-ElastiCache-Redis-Multi-AZ

ElastiCache の機能については以下の記事を参考にすること。

作業の流れ

  1. VPC & サブネットの作成
  2. ElastiCache向けサブネットの作成
  3. Redis クラスタの作成
  4. レプリケーショングループの作成
  5. リードレプリカの追加
  6. リードレプリカのマスター昇格
  7. レプリケーショングループから特定のクラスタを削除

1. VPC & サブネットの作成

VPC を作成し、各AZにサブネットを作成。

今回は 10.6.0.0/12 な VPC に対して

  • “ap-northeast-1c” : 10.6.0.0/24
    “ap-northeast-1a” : 10.6.1.0/24

の2つのサブネットを用意。

$ aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-xxx
{
    "Subnets": [
        {
            "VpcId": "vpc-xxx",
            "Tags": [
                {
                    "Value": "subnet-1c",
                    "Key": "Name"
                }
            ],
            "CidrBlock": "10.6.0.0/24",
            "MapPublicIpOnLaunch": false,
            "DefaultForAz": false,
            "State": "available",
            "AvailabilityZone": "ap-northeast-1c",
            "SubnetId": "subnet-bbb",
            "AvailableIpAddressCount": 249
        },
        {
            "VpcId": "vpc-xxx",
            "Tags": [
                {
                    "Value": "subnet-1a",
                    "Key": "Name"
                }
            ],
            "CidrBlock": "10.6.1.0/24",
            "MapPublicIpOnLaunch": false,
            "DefaultForAz": false,
            "State": "available",
            "AvailabilityZone": "ap-northeast-1a",
            "SubnetId": "subnet-aaa",
            "AvailableIpAddressCount": 251
        }
    ]
}

2. ElastiCache向けサブネットの作成

Redis クラスタに設定するサブネット群(Cache Subnet Group)を作成。

マルチAZにしたいので

  • ap-northeast-1a
  • ap-northeast-1c

それぞれの VPC サブネットを指定する。

$ # http://docs.aws.amazon.com/cli/latest/reference/elasticache/create-cache-subnet-group.html
$ aws elasticache create-cache-subnet-group \
  --cache-subnet-group-name my-cache-subnet-group \
  --cache-subnet-group-description "cache subnet group"  \
  --subnet-ids subnet-aaa subnet-bbb

{
    "CacheSubnetGroup": {
        "VpcId": "vpc-xxx",
        "CacheSubnetGroupDescription": "cache subnet group",
        "Subnets": [
            {
                "SubnetIdentifier": "subnet-bbb",
                "SubnetAvailabilityZone": {
                    "Name": "ap-northeast-1c"
                }
            },
            {
                "SubnetIdentifier": "subnet-aaa",
                "SubnetAvailabilityZone": {
                    "Name": "ap-northeast-1a"
                }
            }
        ],
        "CacheSubnetGroupName": "my-cache-subnet-group"
    }
}

確認
cache-subnet-group

$ aws elasticache describe-cache-subnet-groups --cache-subnet-group-name my-cache-subnet-group
{
    "CacheSubnetGroups": [
        {
            "VpcId": "vpc-xxx",
            "CacheSubnetGroupDescription": "cache subnet group",
            "Subnets": [
                {
                    "SubnetIdentifier": "subnet-bbb",
                    "SubnetAvailabilityZone": {
                        "Name": "ap-northeast-1c"
                    }
                },
                {
                    "SubnetIdentifier": "subnet-aaa",
                    "SubnetAvailabilityZone": {
                        "Name": "ap-northeast-1a"
                    }
                }
            ],
            "CacheSubnetGroupName": "my-cache-subnet-group"
        }
    ]
}

3. Redis クラスタの作成

Redis クラスタを my-redis-primary という名前で作成。
キャッシュエンジンに Redis を指定した場合、クラスタと言いつつも1ノード固定となる。

詳細は次のドキュメントを参照

Creating a Cache Cluster in an Amazon VPC

http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ManagingVPC.CreatingCacheCluster.html

$ # http://docs.aws.amazon.com/cli/latest/reference/elasticache/create-cache-cluster.html
$ aws elasticache create-cache-cluster \
  --cache-cluster-id my-redis-primary \
  --engine-version 2.8.6 \
  --cache-subnet-group-name my-cache-subnet-group \
  --num-cache-nodes 1 \
  --cache-node-type cache.t2.micro \
  --engine redis \
  --port 6379
{
    "CacheCluster": {
        "Engine": "redis",
        "CacheParameterGroup": {
            "CacheNodeIdsToReboot": [],
            "CacheParameterGroupName": "default.redis2.8",
            "ParameterApplyStatus": "in-sync"
        },
        "CacheClusterId": "my-redis-primary",
        "CacheSecurityGroups": [],
        "NumCacheNodes": 1,
        "AutoMinorVersionUpgrade": true,
        "CacheClusterStatus": "creating",
        "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
        "CacheSubnetGroupName": "my-cache-subnet-group",
        "EngineVersion": "2.8.6",
        "PendingModifiedValues": {},
        "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
        "CacheNodeType": "cache.t2.micro"
    }
}

クラスタの確認

elasticache cache-clusters

elasticache describe-cache-clusters コマンドに --show-cache-node-info オプションをつけて、ノード情報(出力の CacheNodes のブロック)も表示させる。

$ # http://docs.aws.amazon.com/cli/latest/reference/elasticache/describe-cache-clusters.html
$ aws elasticache describe-cache-clusters --show-cache-node-info
{
    "CacheClusters": [
        {
            "Engine": "redis",
            "CacheNodes": [
                {
                    "CacheNodeId": "0001",
                    "Endpoint": {
                        "Port": 6379,
                        "Address": "my-redis-primary.asdf.0001.apne1.cache.amazonaws.com"
                    },
                    "ParameterGroupStatus": "in-sync",
                    "CacheNodeCreateTime": "2014-09-26T07:45:08.599Z",
                    "CacheNodeStatus": "available"
                }
            ],
            "CacheParameterGroup": {
                "CacheNodeIdsToReboot": [],
                "CacheParameterGroupName": "default.redis2.8",
                "ParameterApplyStatus": "in-sync"
            },
            "CacheClusterId": "my-redis-primary",
            "CacheSecurityGroups": [],
            "NumCacheNodes": 1,
            "CacheClusterCreateTime": "2014-09-26T07:45:08.599Z",
            "AutoMinorVersionUpgrade": true,
            "CacheClusterStatus": "available",
            "PreferredAvailabilityZone": "ap-northeast-1c",
            "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
            "CacheSubnetGroupName": "my-cache-subnet-group",
            "EngineVersion": "2.8.6",
            "PendingModifiedValues": {},
            "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
            "CacheNodeType": "cache.t2.micro"
        }
    ]
}

Redis への接続確認

クラスタの作成が完了したタイミングを見計らって Redis に接続してみる。

$ redis-cli -h redis-primary.asdf.0001.apne1.cache.amazonaws.com ping
PONG
$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com set foo bar
OK
$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com get foo
"bar"
$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com info
# Server
redis_version:2.8.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:37f1586c49770bbe
redis_mode:standalone
os:Amazon ElastiCache
arch_bits:64
multiplexing_api:epoll
gcc_version:0.0.0
process_id:1
run_id:e88e73750084b95d9206e19d13a468bc8b4890a5
tcp_port:6379
uptime_in_seconds:358
uptime_in_days:0
hz:10
lru_clock:662599
config_file:/etc/redis.conf

# Clients
connected_clients:4
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:3730192
used_memory_human:3.56M
used_memory_rss:11718656
used_memory_peak:3782088
used_memory_peak_human:3.61M
used_memory_lua:33792
mem_fragmentation_ratio:3.14
mem_allocator:jemalloc-3.2.0

# Persistence
loading:0
rdb_changes_since_last_save:353
rdb_bgsave_in_progress:0
rdb_last_save_time:1411717478
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:8
total_commands_processed:815
instantaneous_ops_per_sec:2
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:1
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.04
used_cpu_user:0.04
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Keyspace
db0:keys=2,expires=0,avg_ttl=0

4. レプリケーショングループの作成

master-slave を束ねるレプリケーショングループを my-repgroup という名前で作成。
--primary-cluster-id として先ほど作成したクラスタの my-redis-primary を指定
詳細は次のURLを参照

http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ManagingReplication.html

$ # http://docs.aws.amazon.com/cli/latest/reference/elasticache/create-replication-group.html
$ aws elasticache create-replication-group \
  --replication-group-id  my-repgroup \
  --primary-cluster-id my-redis-primary \
  --replication-group-description "My replication group"
{
    "ReplicationGroup": {
        "Status": "creating",
        "MemberClusters": [
            "my-redis-primary"
        ],
        "Description": "My replication group",
        "ReplicationGroupId": "my-repgroup",
        "PendingModifiedValues": {}
    }
}

レプリケーショングループを確認

$ # http://docs.aws.amazon.com/cli/latest/reference/elasticache/describe-replication-groups.html
$ aws elasticache describe-replication-groups
{
    "ReplicationGroups": [
        {
            "Status": "available",
            "Description": "My replication group",
            "NodeGroups": [
                {
                    "Status": "available",
                    "NodeGroupMembers": [
                        {
                            "CurrentRole": "primary",
                            "PreferredAvailabilityZone": "ap-northeast-1c",
                            "CacheNodeId": "0001",
                            "ReadEndpoint": {
                                "Port": 6379,
                                "Address": "my-redis-primary.asdf.0001.apne1.cache.amazonaws.com"
                            },
                            "CacheClusterId": "my-redis-primary"
                        }
                    ],
                    "NodeGroupId": "0001",
                    "PrimaryEndpoint": {
                        "Port": 6379,
                        "Address": "my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com"
                    }
                }
            ],
            "ReplicationGroupId": "my-repgroup",
            "MemberClusters": [
                "my-redis-primary"
            ],
            "PendingModifiedValues": {}
        }
    ]
}

DNS レコードを確認

クラスタのエンドポイント

$ dig my-redis-primary.asdf.0001.apne1.cache.amazonaws.com +short
10.6.0.194

レプリケーショングループのエンドポイント

$ dig my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13498
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com. IN A

;; ANSWER SECTION:
my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com. 54 IN CNAME my-redis-primary.asdf.0001.apne1.cache.amazonaws.com.
my-redis-primary.asdf.0001.apne1.cache.amazonaws.com. 54 IN A 10.6.0.194

;; Query time: 0 msec
;; SERVER: 10.6.0.2#53(10.6.0.2)
;; WHEN: Fri Sep 26 07:52:31 2014
;; MSG SIZE  rcvd: 124

レプリケーショングループのプライマリーエンドポイントに Redis で接続

$ redis-cli -h my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com PING
PONG
$ redis-cli -h my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com GET foo
"bar"

Redis レプリケーション情報を確認。

$ redis-cli -h my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com INFO Replication
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

role:master でスレーブはなし(connected_slaves:0)

$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com INFO Replication
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

5. リードレプリカの追加

詳細は次の URL を参照

Adding a Read Replica To A Replication Group
http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ManagingReplication.html#ManagingReplication.AddingReplica

  • ap-northeast-1a
  • ap-northeast-1c

の各 AZ にリードレプリカを追加する。

--preferred-availability-zone で AZ を指定する。

$ aws elasticache create-cache-cluster \
  --cache-cluster-id my-replica-1a \
  --replication-group-id my-repgroup \
  --preferred-availability-zone ap-northeast-1a
{
    "CacheCluster": {
        "Engine": "redis",
        "CacheParameterGroup": {
            "CacheNodeIdsToReboot": [],
            "CacheParameterGroupName": "default.redis2.8",
            "ParameterApplyStatus": "in-sync"
        },
        "CacheClusterId": "my-replica-1a",
        "CacheSecurityGroups": [],
        "NumCacheNodes": 1,
        "ReplicationGroupId": "my-repgroup",
        "AutoMinorVersionUpgrade": true,
        "CacheClusterStatus": "creating",
        "PreferredAvailabilityZone": "ap-northeast-1a",
        "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
        "CacheSubnetGroupName": "my-cache-subnet-group",
        "EngineVersion": "2.8.6",
        "PendingModifiedValues": {},
        "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
        "CacheNodeType": "cache.t2.micro"
    }
}

$ aws elasticache create-cache-cluster \
  --cache-cluster-id my-replica-1c \
  --replication-group-id my-repgroup \
  --preferred-availability-zone ap-northeast-1c
{
    "CacheCluster": {
        "Engine": "redis",
        "CacheParameterGroup": {
            "CacheNodeIdsToReboot": [],
            "CacheParameterGroupName": "default.redis2.8",
            "ParameterApplyStatus": "in-sync"
        },
        "CacheClusterId": "my-replica-1c",
        "CacheSecurityGroups": [],
        "NumCacheNodes": 1,
        "ReplicationGroupId": "my-repgroup",
        "AutoMinorVersionUpgrade": true,
        "CacheClusterStatus": "creating",
        "PreferredAvailabilityZone": "ap-northeast-1c",
        "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
        "CacheSubnetGroupName": "my-cache-subnet-group",
        "EngineVersion": "2.8.6",
        "PendingModifiedValues": {},
        "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
        "CacheNodeType": "cache.t2.micro"
    }
}

確認

replication-groups

$ aws elasticache describe-replication-groups
{
    "ReplicationGroups": [
        {
            "Status": "available",
            "Description": "My replication group",
            "NodeGroups": [
                {
                    "Status": "available",
                    "NodeGroupMembers": [
                        {
                            "CurrentRole": "primary",
                            "PreferredAvailabilityZone": "ap-northeast-1c",
                            "CacheNodeId": "0001",
                            "ReadEndpoint": {
                                "Port": 6379,
                                "Address": "my-redis-primary.asdf.0001.apne1.cache.amazonaws.com"
                            },
                            "CacheClusterId": "my-redis-primary"
                        }
                    ],
                    "NodeGroupId": "0001",
                    "PrimaryEndpoint": {
                        "Port": 6379,
                        "Address": "my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com"
                    }
                }
            ],
            "ReplicationGroupId": "my-repgroup",
            "MemberClusters": [
                "my-redis-primary",
                "my-replica-1a",
                "my-replica-1c"
            ],
            "PendingModifiedValues": {}
        }
    ]
}

MemberClusters に

  • “my-redis-primary”,
  • “my-replica-1a”,
  • “my-replica-1c”

と3つのクラスタが並んでいる。

ノード一覧を確認

$ aws elasticache describe-cache-clusters --show-cache-node-info
{
    "CacheClusters": [
        {
            "Engine": "redis",
            "CacheNodes": [
                {
                    "CacheNodeId": "0001",
                    "Endpoint": {
                        "Port": 6379,
                        "Address": "my-redis-primary.asdf.0001.apne1.cache.amazonaws.com"
                    },
                    "ParameterGroupStatus": "in-sync",
                    "CacheNodeCreateTime": "2014-09-26T07:45:08.599Z",
                    "CacheNodeStatus": "available"
                }
            ],
            "CacheParameterGroup": {
                "CacheNodeIdsToReboot": [],
                "CacheParameterGroupName": "default.redis2.8",
                "ParameterApplyStatus": "in-sync"
            },
            "CacheClusterId": "my-redis-primary",
            "CacheSecurityGroups": [],
            "NumCacheNodes": 1,
            "CacheClusterCreateTime": "2014-09-26T07:45:08.599Z",
            "ReplicationGroupId": "my-repgroup",
            "AutoMinorVersionUpgrade": true,
            "CacheClusterStatus": "available",
            "PreferredAvailabilityZone": "ap-northeast-1c",
            "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
            "CacheSubnetGroupName": "my-cache-subnet-group",
            "EngineVersion": "2.8.6",
            "PendingModifiedValues": {},
            "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
            "CacheNodeType": "cache.t2.micro"
        },
        {
            "Engine": "redis",
            "CacheNodes": [
                {
                    "CacheNodeId": "0001",
                    "Endpoint": {
                        "Port": 6379,
                        "Address": "my-replica-1a.asdf.0001.apne1.cache.amazonaws.com"
                    },
                    "ParameterGroupStatus": "in-sync",
                    "CacheNodeCreateTime": "2014-09-26T08:05:45.556Z",
                    "CacheNodeStatus": "available"
                }
            ],
            "CacheParameterGroup": {
                "CacheNodeIdsToReboot": [],
                "CacheParameterGroupName": "default.redis2.8",
                "ParameterApplyStatus": "in-sync"
            },
            "CacheClusterId": "my-replica-1a",
            "CacheSecurityGroups": [],
            "NumCacheNodes": 1,
            "CacheClusterCreateTime": "2014-09-26T08:05:45.556Z",
            "ReplicationGroupId": "my-repgroup",
            "AutoMinorVersionUpgrade": true,
            "CacheClusterStatus": "available",
            "PreferredAvailabilityZone": "ap-northeast-1a",
            "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
            "CacheSubnetGroupName": "my-cache-subnet-group",
            "EngineVersion": "2.8.6",
            "PendingModifiedValues": {},
            "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
            "CacheNodeType": "cache.t2.micro"
        },
        {
            "Engine": "redis",
            "CacheNodes": [
                {
                    "CacheNodeId": "0001",
                    "Endpoint": {
                        "Port": 6379,
                        "Address": "my-replica-1c.asdf.0001.apne1.cache.amazonaws.com"
                    },
                    "ParameterGroupStatus": "in-sync",
                    "CacheNodeCreateTime": "2014-09-26T08:06:19.412Z",
                    "CacheNodeStatus": "available"
                }
            ],
            "CacheParameterGroup": {
                "CacheNodeIdsToReboot": [],
                "CacheParameterGroupName": "default.redis2.8",
                "ParameterApplyStatus": "in-sync"
            },
            "CacheClusterId": "my-replica-1c",
            "CacheSecurityGroups": [],
            "NumCacheNodes": 1,
            "CacheClusterCreateTime": "2014-09-26T08:06:19.412Z",
            "ReplicationGroupId": "my-repgroup",
            "AutoMinorVersionUpgrade": true,
            "CacheClusterStatus": "available",
            "PreferredAvailabilityZone": "ap-northeast-1c",
            "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
            "CacheSubnetGroupName": "my-cache-subnet-group",
            "EngineVersion": "2.8.6",
            "PendingModifiedValues": {},
            "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
            "CacheNodeType": "cache.t2.micro"
        }
    ]
}

各クラスタのエンドポイントに接続

レプリケーションエンドポイント

$ redis-cli -h my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com INFO replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.16.2.226,port=6379,state=online,offset=54039,lag=1
slave1:ip=172.16.1.38,port=6379,state=online,offset=54039,lag=1
master_repl_offset:54127
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:54126
connected_slaves:2
slave0:ip=172.16.2.226,port=6379,state=online,offset=54039,lag=1
slave1:ip=172.16.1.38,port=6379,state=online,offset=54039,lag=1

から slave が2台つながっているのがわかる。

master(my-redis-primary)エンドポイント

$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com INFO replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.16.2.220,port=6379,state=online,offset=26427,lag=1
slave1:ip=172.16.1.227,port=6379,state=online,offset=26427,lag=1
master_repl_offset:26515
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:26514

更新も可能

$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com set foo bar
OK

slave(my-replicas-1a)エンドポイント

$ redis-cli -h my-replica-1a.asdf.0001.apne1.cache.amazonaws.com INFO replication
# Replication
role:slave
master_host:my-repgroup.asdf.0001.internal.apne1.cache.amazonaws.com
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:21415
repl_sync_enabled:1
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

$ redis-cli -h my-replica-1a.asdf.0001.apne1.cache.amazonaws.com set foo bar
(error) READONLY You can't write against a read only slave.

role:slave とスレーブで slave_read_only:1 と更新は不可。
(Redis 自体は設定次第で slave に write を許可することもできる)

slave(my-replicas-1c)エンドポイント

$ redis-cli -h my-replica-1c.asdf.0001.apne1.cache.amazonaws.com INFO replication
# Replication
role:slave
master_host:my-repgroup.asdf.0001.internal.apne1.cache.amazonaws.com
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:21869
repl_sync_enabled:1
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

$ redis-cli -h my-replica-1c.asdf.0001.apne1.cache.amazonaws.com set foo bar
(error) READONLY You can't write against a read only slave.

role:slave とスレーブで slave_read_only:1 と更新は不可。

6.リードレプリカのマスター昇格

modify-replication-group コマンドでレプリケーショングループの設定を変更する。
--primary-cluster-id で master にするクラスタIDを指定する

--apply-immediately 引数も渡して、即時に設定変更させる。このオプションを外すと(デフォルトはオフ)、次のメンテナンスやリスタートまで待たなければいけない。

If true , this parameter causes the modifications in this request and any pending modifications to be applied, asynchronously and as soon as possible, regardless of the –preferred-maintenance-window setting for the replication group.
If false , then changes to the nodes in the replication group are applied on the next maintenance reboot, or the next failure reboot, whichever occurs first.
Valid values: true | false
Default: false

http://docs.aws.amazon.com/cli/latest/reference/elasticache/modify-replication-group.html

$ # http://docs.aws.amazon.com/cli/latest/reference/elasticache/modify-replication-group.html
$ aws elasticache modify-replication-group \
  --replication-group-id my-repgroup \
  --primary-cluster-id=my-replica-1c \
  --apply-immediately
{
    "ReplicationGroup": {
        "Status": "modifying",
        "Description": "My replication group",
        "NodeGroups": [
            {
                "Status": "modifying",
                "NodeGroupMembers": [
                    {
                        "CurrentRole": "primary",
                        "PreferredAvailabilityZone": "ap-northeast-1c",
                        "CacheNodeId": "0001",
                        "ReadEndpoint": {
                            "Port": 6379,
                            "Address": "my-redis-primary.asdf.0001.apne1.cache.amazonaws.com"
                        },
                        "CacheClusterId": "my-redis-primary"
                    },
                    {
                        "CurrentRole": "replica",
                        "PreferredAvailabilityZone": "ap-northeast-1a",
                        "CacheNodeId": "0001",
                        "ReadEndpoint": {
                            "Port": 6379,
                            "Address": "my-replica-1a.asdf.0001.apne1.cache.amazonaws.com"
                        },
                        "CacheClusterId": "my-replica-1a"
                    },
                    {
                        "CurrentRole": "replica",
                        "PreferredAvailabilityZone": "ap-northeast-1c",
                        "CacheNodeId": "0001",
                        "ReadEndpoint": {
                            "Port": 6379,
                            "Address": "my-replica-1c.asdf.0001.apne1.cache.amazonaws.com"
                        },
                        "CacheClusterId": "my-replica-1c"
                    }
                ],
                "NodeGroupId": "0001",
                "PrimaryEndpoint": {
                    "Port": 6379,
                    "Address": "my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com"
                }
            }
        ],
        "ReplicationGroupId": "my-repgroup",
        "MemberClusters": [
            "my-redis-primary",
            "my-replica-1a",
            "my-replica-1c"
        ],
        "PendingModifiedValues": {
            "PrimaryClusterId": "my-replica-1c"
        }
    }
}

DNS レコードを確認

切り替えに伴い、レプリケーショングループのマスターの向きさきが redis-primary から my-replica-1cに代わる。

master 切り替え前の DNS

$ dig +short redis-primary.asdf.0001.apne1.cache.amazonaws.com
10.6.1.129
$ dig +short replica-1c.asdf.ng.0001.apne1.cache.amazonaws.com
10.6.0.165
$ dig +short my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com
redis-primary.asdf.0001.apne1.cache.amazonaws.com.
10.6.1.129

master 切り替え後の DNS

$ dig  my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com +short
my-replica-1c.asdf.0001.apne1.cache.amazonaws.com.
10.6.0.165

redis のreplication 確認

my-redis-primary はマスターからスレーブに変わっている。

$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com info replication
# Replication
role:slave
master_host:my-repgroup.asdf.0001.internal.apne1.cache.amazonaws.com
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:7440
repl_sync_enabled:1
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:60517

$ redis-cli -h my-redis-primary.asdf.0001.apne1.cache.amazonaws.com set foo bar
(error) READONLY You can't write against a read only slave.

クラスタ my-replica-1c はスレーブからマスターに変わっている。

$ redis-cli -h my-replica-1c.asdf.0001.apne1.cache.amazonaws.com info replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.16.1.173,port=6379,state=online,offset=6722,lag=0
slave1:ip=172.16.2.220,port=6379,state=online,offset=6634,lag=1
master_repl_offset:6722
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:6721

$ redis-cli -h my-replica-1c.asdf.0001.apne1.cache.amazonaws.com set foo bar
OK

MONITORコマンドを確認

マスターが切り替わる前後で MONITOR コマンドを実行しておくと、次のように SLAVEOF コマンドでスレーブ操作が行われているのがわかる。

“my-redis-primary”

1411719360.002993 [0 127.0.0.1:25927] "set" "ElastiCacheMasterReplicationTimestamp" "2014-09-26T08:16:00.001Z"
1411719360.013549 [0 127.0.0.1:25929] "ping"
1411719360.189183 [0 127.0.0.1:25927] "ping"
1411719360.189329 [0 127.0.0.1:25927] "slaveof" "my-repgroup.asdf.0001.internal.apne1.cache.amazonaws.com" "6379"
1411719360.553870 [0 172.16.2.220:18599] "ping"
1411719360.556116 [0 172.16.2.220:18599] "replconf" "listening-port" "6379"
1411719360.558395 [0 172.16.2.220:18599] "psync" "e88e73750084b95d9206e19d13a468bc8b4890a5" "60519"
1411719360.560730 [0 172.16.2.220:18599] "sync"

"slaveof" "my-repgroup.asdf.0001.internal.apne1.cache.amazonaws.com" "6379"my-repgroup.asdf.0001.internal.apne1.cache.amazonaws.com:6379 のスレーブになる。

“my-replica-1c”

1411719360.027778 [0 172.16.1.173:6379] "set" "ElastiCacheMasterReplicationTimestamp" "2014-09-26T08:16:00.001Z"
1411719360.681949 [0 127.0.0.1:46232] "ping"
1411719360.682848 [0 127.0.0.1:46232] "slaveof" "no" "one"
1411719361.000970 [0 127.0.0.1:46232] "ping"
1411719361.001050 [0 127.0.0.1:46232] "set" "ElastiCacheMasterReplicationTimestamp" "2014-09-26T08:16:01.000Z"

"slaveof" "no" "one" で以前のスレーブ設定が無効化される。

7. レプリケーショングループから特定のクラスタを削除

レプリケーショングループのクラスタ群が

  • master : my-replica-1c@1c
  • slave : my-replica-1a@1a
  • slave : my-redis-primary@1a

となっている状態で、クラスタ my-replica-1a を削除する。
コマンドは delete-cache-cluster を使う。

$ # http://docs.aws.amazon.com/cli/latest/reference/elasticache/delete-cache-cluster.html
$ aws elasticache delete-cache-cluster --cache-cluster-id my-replica-1a
{
    "CacheCluster": {
        "Engine": "redis",
        "CacheParameterGroup": {
            "CacheNodeIdsToReboot": [],
            "CacheParameterGroupName": "default.redis2.8",
            "ParameterApplyStatus": "in-sync"
        },
        "CacheClusterId": "my-replica-1a",
        "CacheSecurityGroups": [],
        "NumCacheNodes": 1,
        "CacheClusterCreateTime": "2014-09-26T08:05:45.556Z",
        "ReplicationGroupId": "my-repgroup",
        "AutoMinorVersionUpgrade": true,
        "CacheClusterStatus": "deleting",
        "PreferredAvailabilityZone": "ap-northeast-1a",
        "ClientDownloadLandingPage": "https://console.aws.amazon.com/elasticache/home#client-download:",
        "CacheSubnetGroupName": "my-cache-subnet-group",
        "EngineVersion": "2.8.6",
        "PendingModifiedValues": {},
        "PreferredMaintenanceWindow": "sun:20:00-sun:21:00",
        "CacheNodeType": "cache.t2.micro"
    }
}

確認

$ aws elasticache describe-replication-groups
{
    "ReplicationGroups": [
        {
            "Status": "available",
            "Description": "My replication group",
            "NodeGroups": [
                {
                    "Status": "available",
                    "NodeGroupMembers": [
                        {
                            "CurrentRole": "replica",
                            "PreferredAvailabilityZone": "ap-northeast-1c",
                            "CacheNodeId": "0001",
                            "ReadEndpoint": {
                                "Port": 6379,
                                "Address": "my-redis-primary.asdf.0001.apne1.cache.amazonaws.com"
                            },
                            "CacheClusterId": "my-redis-primary"
                        },
                        {
                            "CurrentRole": "primary",
                            "PreferredAvailabilityZone": "ap-northeast-1c",
                            "CacheNodeId": "0001",
                            "ReadEndpoint": {
                                "Port": 6379,
                                "Address": "my-replica-1c.asdf.0001.apne1.cache.amazonaws.com"
                            },
                            "CacheClusterId": "my-replica-1c"
                        }
                    ],
                    "NodeGroupId": "0001",
                    "PrimaryEndpoint": {
                        "Port": 6379,
                        "Address": "my-repgroup.asdf.ng.0001.apne1.cache.amazonaws.com"
                    }
                }
            ],
            "ReplicationGroupId": "my-repgroup",
            "MemberClusters": [
                "my-redis-primary",
                "my-replica-1c"
            ],
            "PendingModifiedValues": {}
        }
    ]
}

MemberClusters の出力から、 my-replica-1a が削除されたことがわかる。

Advertisements
Tagged with: , , ,
Posted in aws, database

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Archives
  • RT @__apf__: How to write a research paper: a guide for software engineers & practitioners. docs.google.com/presentation/d… /cc @inwyrd 4 months ago
  • RT @HayatoChiba: 昔、自然と対話しながら数学に打ち込んだら何かを悟れるのではと思いたち、専門書1つだけ持ってパワースポットで名高い奈良の山奥に1週間籠ったことがある。しかし泊まった民宿にドカベンが全巻揃っていたため、水島新司と対話しただけで1週間過ぎた。 それ… 5 months ago
  • RT @googlecloud: Ever wonder what underwater fiber optic internet cables look like? Look no further than this deep dive w/ @NatAndLo: https… 5 months ago
  • @ijin UTC+01:00 な時間帯で生活しています、、、 10 months ago
  • RT @mattcutts: Google's world-class Site Reliability Engineering team wrote a new book: amazon.com/Site-Reliabili… It's about managing produc… 1 year ago
%d bloggers like this: