Alluxio使用——TensorFlow篇

1.安装部署

TensorFlow安装部署
Alluxio安装部署

SSH免密登陆

2.使用

1).创建alluxio根目录

[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs mkdir /training-data Successfully created directory /training-data

2).创建本地目录，并挂载到alluxio根目录

a).创建本地目录

mkdir -p /home/bigdata/data

b).挂载到alluxio根目录

[bigdata@carbondata alluxio-2.0.0]$ ./integration/fuse/bin/alluxio-fuse mount /home/bigdata/data /training-data Starting alluxio-fuse on local host. Successfully mounted Alluxio path "/training-data" to /home/bigdata/data. See /home/bigdata/alluxio-2.0.0/logs/fuse.log for logging messages

c).验证挂载状态

[bigdata@carbondata alluxio-2.0.0]$ ./integration/fuse/bin/alluxio-fuse stat pid mount_point alluxio_path 11074 /home/bigdata/data /training-data

3).准备测试数据

a).下载测试数据

inception-2015-12-05

wget http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz

b).测试数据上传至alluxio

## 创建数据目录 [bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs mkdir /training-data/imagenet Successfully created directory /training-data/imagenet ## 上传数据 [bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs copyFromLocal /home/bigdata/inception-2015-12-05.tgz /training-data/imagenet Failed to cache: There is no worker with enough space for a new block of size 536,870,912 [bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs copyFromLocal /home/bigdata/inception-2015-12-05.tgz /training-data/imagenet Copied file:///home/bigdata/inception-2015-12-05.tgz to /training-data/imagenet

4).图像识别测试

下载脚本

curl -o classify_image.py -L https://raw.githubusercontent.com/tensorflow/models/master/tutorials/image/imagenet/classify_image.py

运行脚本

(tensorflow) [bigdata@carbondata tensorflow_data]$ python classify_image.py --model_dir /home/bigdata/data/imagenet WARNING:tensorflow:From classify_image.py:227: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead. WARNING:tensorflow:From classify_image.py:139: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead. W0829 21:25:37.012851 139730996795200 deprecation_wrapper.py:119] From classify_image.py:139: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead. WARNING:tensorflow:From classify_image.py:141: __init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version. Instructions for updating: Use tf.gfile.GFile. W0829 21:25:39.488382 139730996795200 deprecation.py:323] From classify_image.py:141: __init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version. Instructions for updating: Use tf.gfile.GFile. WARNING:tensorflow:From classify_image.py:125: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead. W0829 21:25:41.580853 139730996795200 deprecation_wrapper.py:119] From classify_image.py:125: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead. 2019-08-29 21:26:31.840733: W tensorflow/core/framework/op_def_util.cc:357] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization(). WARNING:tensorflow:From classify_image.py:146: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. W0829 21:27:04.229350 139730996795200 deprecation_wrapper.py:119] From classify_image.py:146: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. 2019-08-29 21:27:04.889623: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-08-29 21:27:05.865288: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 2019-08-29 21:27:05.888049: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557e570 executing computations on platform Host. Devices: 2019-08-29 21:27:05.888110: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined> 2019-08-29 21:27:07.301608: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory. 2019-08-29 21:27:07.357415: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory. 2019-08-29 21:27:07.359446: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory. 2019-08-29 21:27:07.431136: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory. 2019-08-29 21:27:07.593467: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory. 2019-08-29 21:27:08.113432: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412](One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. WARNING:tensorflow:From classify_image.py:85: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead. W0829 21:27:11.583446 139730996795200 deprecation_wrapper.py:119] From classify_image.py:85: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead. giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca (score = 0.89107) indri, indris, Indri indri, Indri brevicaudatus (score = 0.00779) lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens (score = 0.00296) custard apple (score = 0.00147) earthstar (score = 0.00117)