Amazon Machine LearningのチュートリアルをAWS CLIから実行してみる

Summary

“Amazon Machine Learning Developer Guide” には “Tutorial: Using Amazon ML to Predict Responses to a Marketing Offer” というこのサービスの初心者向けのチュートリアルが含まれている。

Tutorial: Using Amazon ML to Predict Responses to a Marketing Offer
http://docs.aws.amazon.com/machine-learning/latest/mlconcepts/mlconcepts.html

チュートリアルはマネージドコンソールから操作しているので、将来のスクリプト化を見据えて AWS CLI から操作してみる。

チュートリアルの流れ

University of California, Irvine の Machine Learning Repository にある Bank Marketing Data Set を利用して、顧客が定期預金を契約するかどうかの 2 値分類を行う。

作業の流れは以下

  • Step 1: Download, Edit, and Upload Data
  • Step 2: Create a Datasource
  • Step 3: Create an ML Model
  • Step 4: Review the ML Model’s Performance and Set a Score Threshold
  • Step 5: Use the ML Model to Generate Batch Predictions
  • Step 6: Clean Up

学習・評価用データとバッチ予測結果は S3 に置く。

Step 0: AWS CLI の設定

2015/06/01 時点では Amazon Marketing Learning(以下 ML) は US East(Northern Virginia)でしか利用できないため、デフォルトのリージョンを一時的に変更しておく。

$ cat ~/.aws/config
[default]
output=json
region=us-east-1

Step 1: Download, Edit, and Upload Data

新規に S3 バケットを用意し、学習データ(banking.csv)と評価データ(banking-batch.csv)をアップロードします。

$ BUCKET=b6b6
$ aws s3 mb s3://$BUCKET
make_bucket: s3://b6b6/
$ aws s3 cp banking.csv s3://$BUCKET/
upload: ./banking.csv to s3://b6b6/banking.csv
$ aws s3 cp banking-batch.csv s3://$BUCKET/
upload: ./banking-batch.csv to s3://b6b6/banking-batch.csv

AWS ML では以下の S3 操作を行います

  • ML から S3 にある学習データの GetObject
  • ML から S3 にある評価データの GetObject
  • ML S3 にバッチ予測結果データをバケットのルート直下に PutObject

このためのバケットポリシーを設定します

詳細は次の URL を参照 http://docs.aws.amazon.com/machine-learning/latest/dg/reference.html

$ cat ml_bucket_policy.json
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AmazonML_s3:ListBucket",
			"Effect": "Allow",
			"Principal": {
				"Service": "machinelearning.amazonaws.com"
			},
			"Action": "s3:ListBucket",
			"Resource": "arn:aws:s3:::b6b6",
			"Condition": {
				"StringLike": {
					"s3:prefix": [
						"*",
						"banking.csv*",
						"banking-batch.csv*"
					]
				}
			}
		},
		{
			"Sid": "AmazonML_s3:GetObject",
			"Effect": "Allow",
			"Principal": {
				"Service": "machinelearning.amazonaws.com"
			},
			"Action": "s3:GetObject",
			"Resource": [
				"arn:aws:s3:::b6b6/*",
				"arn:aws:s3:::b6b6/banking.csv*",
				"arn:aws:s3:::b6b6/banking-batch.csv*"
			]
		},
		{
			"Sid": "AmazonML_s3:PutObject",
			"Effect": "Allow",
			"Principal": {
				"Service": "machinelearning.amazonaws.com"
			},
			"Action": "s3:PutObject",
			"Resource": "arn:aws:s3:::b6b6/*"
		}
	]
}
$ aws s3api put-bucket-policy --bucket $BUCKET --policy file://ml_bucket_policy.json

Step 2: Create a Datasource

次に S3 にある学習データを ML データソースに設定します。

API は create-data-source-from-s3

API get-data-source でレスポンスの StatusMessage からステータス確認し、 COMPLETED になれば OK

$ DATA_SOURCE=ds-test
$ cat data-spec.json
{
  "DataLocationS3":"s3://b6b6/banking.csv",
  "DataSchema" : "{\"version\":\"1.0\",\"rowId\":null,\"rowWeight\":null,\"targetAttributeName\":\"y\",\"dataFormat\":\"CSV\",\"dataFileContainsHeader\":true,\"attributes\":[{\"attributeName\":\"age\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"job\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"marital\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"education\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"default\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"housing\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"loan\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"contact\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"month\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"day_of_week\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"duration\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"campaign\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"pdays\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"previous\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"poutcome\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"emp_var_rate\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"cons_price_idx\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"cons_conf_idx\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"euribor3m\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"nr_employed\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"y\",\"attributeType\":\"BINARY\"}],\"excludedAttributeNames\":[]}"
}
$ aws machinelearning create-data-source-from-s3 \
  --data-source-id $DATA_SOURCE \
  --data-source-name "Banking Data 1" \
  --data-spec file://data-spec.json \
  --compute-statistics
{
    "DataSourceId": "ds-test"
}

$ aws machinelearning get-data-source --data-source-id $DATA_SOURCE
{
    "Status": "INPROGRESS",
    "NumberOfFiles": 1,
    "Name": "Banking Data 1",
    "DataLocationS3": "s3://b6b6/banking.csv",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "DataSizeInBytes": 4767561,
    "ComputeStatistics": true,
    "LastUpdatedAt": 1433031333.088,
    "DataSourceId": "ds-test",
    "Message": "Current Step: BASIC_STATISTICS (1/2) 50%",
    "CreatedAt": 1433031254.229
}
$ aws machinelearning get-data-source --data-source-id $DATA_SOURCE --verbose
{
    "Status": "COMPLETED",
    "NumberOfFiles": 1,
    "Name": "Banking Data 1",
    "DataLocationS3": "s3://b6b6/banking.csv",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "DataSizeInBytes": 4767561,
    "ComputeStatistics": true,
    "LastUpdatedAt": 1433031580.255,
    "DataSourceId": "ds-test",
    "LogUri": "https://eml-prod-emr.s3.amazonaws.com/1234567890-pr-ml-test/userlog/...",
    "DataSourceSchema": "{\"version\":\"1.0\",\"rowId\":null,\"rowWeight\":null,\"targetAttributeName\":\"y\",\"dataFormat\":\"CSV\",\"dataFileContainsHeader\":true,\"attributes\":[{\"attributeName\":\"age\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"job\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"marital\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"education\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"default\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"housing\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"loan\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"contact\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"month\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"day_of_week\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"duration\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"campaign\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"pdays\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"previous\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"poutcome\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"emp_var_rate\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"cons_price_idx\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"cons_conf_idx\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"euribor3m\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"nr_employed\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"y\",\"attributeType\":\"BINARY\"}],\"excludedAttributeNames\":[]}",
    "CreatedAt": 1433031254.229
}

ログ(LogUri)から内部処理に Elastic MapReduce (EMR) を使っていることがわかる

データのフォーマット指定

API では --data-spec file://data-spec.json で指定するデータ・フォーマットの指定している。

JSON ファイルでは

  • データの場所("DataLocationS3":"s3://b6b6/banking.csv")
  • 目的変数("targetAttributeName":"y")
  • CSV フォーマットであること("dataFormat":"CSV")
  • 各カラムのデータ型("attributeName":"age","attributeType":"NUMERIC"})

などを定義しており、このファイルの作成が面倒。

get-data-source API に --verbose オプションをつけると、データ定義も取得できる。最初は管理画面から設定して、細かいところは用途に応じてあとから修正するのが楽と思われる。

学習データのデータソースには--compute-statisticsフラグを有効にすること

CreateDataSoureFromS3 の API ドキュメントから

The compute statistics for a DataSource. The statistics are generated from the observation data referenced by a DataSource. Amazon ML uses the statistics internally during an MLModel training. This parameter must be set to true if the DataSource needs to be used for MLModel training

http://docs.aws.amazon.com/machine-learning/latest/APIReference/API_CreateDataSourceFromS3.html#API_CreateDataSourceFromS3_RequestSyntax

実際、このフラグを無効にしたデータソースを次のモデル作成のタイミングで指定すると、次の様なエラーが発生する

$ aws machinelearning create-ml-model ...
A client error (InvalidInputException) occurred when calling the CreateMLModel operation: FAILURES (1): Statistics were not requested for the training data source, 'dummy'. CreateMLModel() requests can only use data sources that were created with the 'ComputeStatistics' flag set.

Step 3: Create an ML Model

次に予測モデル設定します。

API は create-ml-model

二値分類なので --ml-model-type "BINARY" とする

API get-ml-model でレスポンスの Status からステータス確認し、StatusCOMPLETED になれば OK

$ ML_MODEL=ml-test
$ aws machinelearning create-ml-model \
  --ml-model-id $ML_MODEL \
  --ml-model-name "Subscription propensity model" \
  --ml-model-type "BINARY" \
  --training-data-source-id  $DATA_SOURCE
{
    "MLModelId": "ml-test"
}

# http://docs.aws.amazon.com/cli/latest/reference/machinelearning/get-ml-model.html
# http://docs.aws.amazon.com/machine-learning/latest/APIReference/API_GetMLModel.html
$ aws machinelearning get-ml-model --ml-model-id $ML_MODEL
{
    "Status": "INPROGRESS",
    "Name": "Subscription propensity model",
    "TrainingParameters": {
        "sgd.l2RegularizationAmount": "1E-6",
        "sgd.maxMLModelSizeInBytes": "100000000",
        "sgd.maxPasses": "10",
        "algorithm": "sgd",
        "sgd.l1RegularizationAmount": "0.0"
    },
    "MLModelType": "BINARY",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "EndpointInfo": {
        "PeakRequestsPerSecond": 0,
        "EndpointStatus": "NONE"
    },
    "MLModelId": "ml-test",
    "InputDataLocationS3": "s3://b6b6/banking.csv",
    "LastUpdatedAt": 1433031823.1,
    "TrainingDataSourceId": "ds-test",
    "CreatedAt": 1433031820.012
}
$ aws machinelearning get-ml-model --ml-model-id $ML_MODEL
{
    "Status": "COMPLETED",
    "SizeInBytes": 429625,
    "Name": "Subscription propensity model",
    "TrainingParameters": {
        "sgd.l2RegularizationAmount": "1E-6",
        "sgd.maxMLModelSizeInBytes": "100000000",
        "sgd.maxPasses": "10",
        "algorithm": "sgd",
        "sgd.l1RegularizationAmount": "0.0"
    },
    "MLModelType": "BINARY",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "EndpointInfo": {
        "PeakRequestsPerSecond": 0,
        "EndpointStatus": "NONE"
    },
    "MLModelId": "ml-test",
    "InputDataLocationS3": "s3://b6b6/banking.csv",
    "LastUpdatedAt": 1433032063.415,
    "TrainingDataSourceId": "ds-test",
    "LogUri": "https://eml-prod-emr.s3.amazonaws.com/1234567890-pr-ml-test/userlog/...",
    "CreatedAt": 1433031820.012
}

Step 4: Review the ML Model’s Performance and Set a Score Threshold

次に作成したモデルでデータを評価します。

API は create-evaluation

API get-evaluation でレスポンスの Status からステータス確認し、StatusCOMPLETED になれば OK

難しいので、対応する Developer Guide も目を通しておいたほうが良い http://docs.aws.amazon.com/machine-learning/latest/dg/evaluating_models.html

評価後のしきい値の調整は管理画面からで。

$ EVALUATION=ev-test
$ aws machinelearning create-evaluation \
  --evaluation-id $EVALUATION \
  --evaluation-name "Subscription propensity evaluation" \
  --ml-model-id $ML_MODEL \
  --evaluation-data-source-id $DATA_SOURCE
{
    "EvaluationId": "ev-test"
}
# http://docs.aws.amazon.com/cli/latest/reference/machinelearning/get-evaluation.html
# http://docs.aws.amazon.com/machine-learning/latest/APIReference/API_GetEvaluation.html
$ aws machinelearning get-evaluation --evaluation-id $EVALUATION
{
    "EvaluationDataSourceId": "ds-test",
    "Status": "INPROGRESS",
    "Name": "Subscription propensity evaluation",
    "InputDataLocationS3": "s3://b6b6/banking.csv",
    "EvaluationId": "ev-test",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "MLModelId": "ml-test",
    "LastUpdatedAt": 1433032120.971,
    "PerformanceMetrics": {
        "Properties": {}
    },
    "CreatedAt": 1433032117.915
}

$ aws machinelearning get-evaluation --evaluation-id $EVALUATION
{
    "EvaluationDataSourceId": "ds-test",
    "Status": "COMPLETED",
    "Name": "Subscription propensity evaluation",
    "InputDataLocationS3": "s3://b6b6/banking.csv",
    "EvaluationId": "ev-test",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "MLModelId": "ml-test",
    "LastUpdatedAt": 1433032365.509,
    "LogUri": "https://eml-prod-emr.s3.amazonaws.com/1234567890-pr-ml-test/userlog/...",
    "PerformanceMetrics": {
        "Properties": {
            "BinaryAUC": "0.9395119502741789"
        }
    },
    "CreatedAt": 1433032117.915
}

Step 5: Use the ML Model to Generate Batch Predictions

評価用データソースを指定

学習データではないので --no-compute-statistics とする。(デフォルトfalse/noなので、このオプションは不要)

$ DATA_SOURCE_BATCH=ds-test-batch
$ cat data-spec-batch.json
{
  "DataLocationS3":"s3://b6b6/banking-batch.csv",
  "DataSchema": "{\"version\":\"1.0\",\"rowId\":null,\"rowWeight\":null,\"targetAttributeName\":null,\"dataFormat\":\"CSV\",\"dataFileContainsHeader\":true,\"attributes\":[{\"attributeName\":\"age\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"job\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"marital\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"education\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"default\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"housing\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"loan\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"contact\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"month\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"day_of_week\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"duration\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"campaign\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"pdays\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"previous\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"poutcome\",\"attributeType\":\"CATEGORICAL\"},{\"attributeName\":\"emp_var_rate\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"cons_price_idx\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"cons_conf_idx\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"euribor3m\",\"attributeType\":\"NUMERIC\"},{\"attributeName\":\"nr_employed\",\"attributeType\":\"NUMERIC\"}],\"excludedAttributeNames\":[]}"
}
$ aws machinelearning create-data-source-from-s3 \
    --data-source-id $DATA_SOURCE_BATCH \
    --data-source-name "Banking Data 2" \
    --data-spec file://data-spec-batch.json \
    --no-compute-statistics
{
    "DataSourceId": "ds-test-batch"
}
$ aws machinelearning get-data-source --data-source-id $DATA_SOURCE_BATCH
{
    "Status": "COMPLETED",
    "NumberOfFiles": 1,
    "Name": "Banking Data 2",
    "DataLocationS3": "s3://b6b6/banking-batch.csv",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "DataSizeInBytes": 468846,
    "ComputeStatistics": false,
    "LastUpdatedAt": 1433032648.258,
    "DataSourceId": "ds-test-batch",
    "CreatedAt": 1433032648.258
}

データのフォーマット指定

API では --data-spec file://data-batch-spec.json で指定するデータ・フォーマットの指定している。

基本的なフォーマットは STEP2 で利用した学習データと同じだが、目的変数("targetAttributeName":null) となっていたり、 分類結果である {"attributeName":"y","attributeType":"BINARY"} の列が評価データにはない、といったこまかな違いは存在する。

get-data-source API に --verbose オプションをつけると、データ定義も取得できる。最初は管理画面から設定して、細かいところは用途に応じてあとから修正するのが楽と思われる。

バッチ予測を実行

メインとなるバッチ予測の実行。

API は create-batch-prediction

API get-batch-prediction でレスポンスの Status からステータス確認し、 COMPLETED になれば OK

$ BATCH_PREDICTION=bp-test
$ aws machinelearning create-batch-prediction \
    --batch-prediction-id $BATCH_PREDICTION \
    --batch-prediction-name "Subscription propensity Predictions" \
    --ml-model-id $ML_MODEL \
    --batch-prediction-data-source-id $DATA_SOURCE_BATCH \
    --output-uri s3://$BUCKET/
{
    "BatchPredictionId": "bp-test"
}
# http://docs.aws.amazon.com/machine-learning/latest/APIReference/API_GetBatchPrediction.html
# http://docs.aws.amazon.com/cli/latest/reference/machinelearning/get-batch-prediction.html
$ aws machinelearning get-batch-prediction --batch-prediction-id $BATCH_PREDICTION
{
    "Status": "INPROGRESS",
    "Name": "Subscription propensity Predictions",
    "InputDataLocationS3": "s3://b6b6/banking-batch.csv",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "MLModelId": "ml-test",
    "LastUpdatedAt": 1433032861.346,
    "BatchPredictionDataSourceId": "ds-test-batch",
    "OutputUri": "s3://b6b6/",
    "BatchPredictionId": "bp-test",
    "Message": "Current Step: BATCH_PREDICTION (1/1) 0%",
    "CreatedAt": 1433032818.172
}
$ aws machinelearning get-batch-prediction --batch-prediction-id $BATCH_PREDICTION
{
    "Status": "COMPLETED",
    "Name": "Subscription propensity Predictions",
    "InputDataLocationS3": "s3://b6b6/banking-batch.csv",
    "CreatedByIamUser": "arn:aws:iam::1234567890:user/awscli",
    "MLModelId": "ml-test",
    "LastUpdatedAt": 1433033003.939,
    "BatchPredictionDataSourceId": "ds-test-batch",
    "OutputUri": "s3://b6b6/",
    "BatchPredictionId": "bp-test",
    "LogUri": "https://eml-prod-emr.s3.amazonaws.com/1234567890-pr-ml-test/userlog/...",
    "CreatedAt": 1433032818.172
}

バッチ予測結果を取得

バッチ予測結果は --output-uri で指定した S3 パスに result というディレクトリで作成されます。

$ aws s3 ls s3://$BUCKET/
                           PRE batch-prediction/
2015-05-31 09:40:20          0 .writePermissionCheck.tmp
2015-05-31 09:12:10     468846 banking-batch.csv
2015-05-31 09:12:06    4767561 banking.csv
$ aws s3 ls s3://$BUCKET/batch-prediction/
                           PRE result/
2015-05-31 09:42:02         96 bp-test.manifest
$ aws s3 sync s3://$BUCKET/batch-prediction batch-prediction
warning: Skipping file /Users/george/Dev/aws-machine-learning/batch-prediction/. File does not exist.
download: s3://b6b6/batch-prediction/bp-test.manifest to batch-prediction/bp-test.manifest
download: s3://b6b6/batch-prediction/result/bp-test-banking-batch.csv.gz to batch-prediction/result/bp-test-banking-batch.csv.gz
$ gzip -d batch-prediction/result/bp-test-banking-batch.csv.gz
$ head -n 20 batch-prediction/result/bp-test-banking-batch.csv
bestAnswer,score
0,8.652008E-2
0,1.342608E-2
0,1.971196E-2
0,1.625213E-3
0,2.575607E-3
0,9.214219E-2
0,4.216536E-1
0,1.793497E-3
0,2.575086E-3
0,4.006434E-3
0,4.669604E-2
0,1.50039E-2
0,2.411173E-3
0,8.510052E-3
0,4.313236E-1
0,2.866098E-1
0,2.3708E-2
0,1.281799E-3
0,1.571054E-3

これらで最低限のCLI操作はできるハズ。

Advertisements
Tagged with: ,
Posted in aws

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Archives
  • RT @__apf__: How to write a research paper: a guide for software engineers & practitioners. docs.google.com/presentation/d… /cc @inwyrd 4 months ago
  • RT @HayatoChiba: 昔、自然と対話しながら数学に打ち込んだら何かを悟れるのではと思いたち、専門書1つだけ持ってパワースポットで名高い奈良の山奥に1週間籠ったことがある。しかし泊まった民宿にドカベンが全巻揃っていたため、水島新司と対話しただけで1週間過ぎた。 それ… 4 months ago
  • RT @googlecloud: Ever wonder what underwater fiber optic internet cables look like? Look no further than this deep dive w/ @NatAndLo: https… 4 months ago
  • @ijin UTC+01:00 な時間帯で生活しています、、、 10 months ago
  • RT @mattcutts: Google's world-class Site Reliability Engineering team wrote a new book: amazon.com/Site-Reliabili… It's about managing produc… 1 year ago
%d bloggers like this: