tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported

[tiogoGit]

I've looked into question #235 But that couldn't solve this problem. I once succeeded in testing Cryptonets. However, after that I will always fail to do
NGRAPH_ENCRYPT_DATA=1 \ NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \ NGRAPH_TF_BACKEND=HE_SEAL \ python test.py --batch_size=4096
I've tried rebuilding from master and changing HE_SEAL to HE_SEAL_CKKS. I also tried restarting the PC. None of them worked.
The result of failure is
`(venv-tf-py3) mugi@mugi-Precision-5820-Tower:~/he-transformer/examples/MNIST/Cryptonets$ NGRAPH_ENCRYPT_DATA=1 \

NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json
NGRAPH_TF_BACKEND=HE_SEAL
python test.py --batch_size=4096
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2019-08-27 15:06:01.623635: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2019-08-27 15:06:01.624328: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x19d0b90 executing computations on platform Host. Devices:
2019-08-27 15:06:01.624366: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
Traceback (most recent call last):
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 90, in
test_mnist_cnn(FLAGS)
File "test.py", line 58, in test_mnist_cnn
(x_train, y_train, x_test, y_test) = load_mnist_data()
File "/home/mugi/he-transformer/examples/MNIST/mnist_util.py", line 29, in load_mnist_data
y_test = tf.one_hot(y_test, 10).eval()
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 731, in eval
return _eval_using_default_session(self, feed_dict, self.graph, session)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5579, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/mugi/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported`

I've built from master and also built python after that. The version message of my PC is:

Operating system: Ubuntu 18.04.
CMake = 3.10.2
Compiler: g++ version = 8.3.0, clang = 6.0.0
python3.6
virtualenv v16.1.0
bazel v0.25.2

[fboemer]

Does the make install step succeed? We expect libhe_seal_backend.so to appear in
$HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib
and
$HE_TRANSFORMER/build/external/lib

[tiogoGit]

Does the make install step succeed? We expect he_seal_backend.so to appear in
$HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib
and
$HE_TRANSFORMER/build/external/lib

No, there is only libhe_seal_backend.so. And I installed it again but there is still no he_seal_backend.so. However the install seemed ok. The make install ends with this:
[ 90%] Linking CXX executable unit-test [ 91%] Built target unit-test [100%] Built target pybind11 Install the project... -- Install configuration: "" -- Installing: /home/mugi/he-transformer/build/external/lib/libhe_seal_backend.so -- Set runtime path of "/home/mugi/he-transformer/build/external/lib/libhe_seal_backend.so" to "$ORIGIN"

What's strange is that, this time only $HE_TRANSFORMER/build/external/lib contains he_seal_backend.so
this time. And $HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib contains no .so file.
I can run NGRAPH_ENCRYPT_DATA=1 \ NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \ NGRAPH_TF_BACKEND=HE_SEAL \ python test.py --batch_size=4096
While the result is still unacceptable.
| Encryption parameters : | scheme: CKKS | poly_modulus_degree: 8192 | coeff_modulus size: 180 (30 + 24 + 24 + 24 + 24 + 24 + 30) bits \ [INFO] 2019-08-28T04:09:15z src/seal/he_seal_backend.cpp 87 Scale 1.67608e+07 [INFO] 2019-08-28T04:09:15z src/seal/he_seal_executable.cpp 462 Encrypting data [INFO] 2019-08-28T04:09:15z src/seal/he_seal_executable.cpp 465 Batching data with batch size 4096 total time(s) 10.469 Error count: 3679 of 4096 elements. Accuracy: 0.101806640625

At the same time, Crypptonets-ReLU is alright:
/ | Encryption parameters : | scheme: CKKS | poly_modulus_degree: 2048 | coeff_modulus size: 54 (54) bits \ [INFO] 2019-08-28T04:32:34z src/seal/he_seal_backend.cpp 87 Scale 8.38861e+06 [INFO] 2019-08-28T04:32:34z src/seal/he_seal_executable.cpp 462 Encrypting data [INFO] 2019-08-28T04:32:34z src/seal/he_seal_executable.cpp 465 Batching data with batch size 1024 [WARN] 2019-08-28T04:32:34z src/seal/he_seal_executable.cpp 1384 Performing Relu without client is not privacy-preserving [WARN] 2019-08-28T04:32:35z src/seal/he_seal_executable.cpp 1384 Performing Relu without client is not privacy-preserving total time(s) 0.754 y_pred [7 2 1 ... 7 5 4] Error count 84 of 1024 elements. Accuracy: 0.917969
Just like #266 . I think there is something wrong in make install, but don't know how to correct it.
Could you tell me why?

[fboemer]

So there are two orthogonal issues here:

1. The install; if the CryptoNets-Relu example is working, it seems the install worked okay. After the install, we expect
   *$HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib to contain libhe_seal_backend.so

• $HE_TRANSFORMER/build/external/lib to contain libhe_seal_backend.so

2. Accuracy. Try larger parameters, e.g. he_seal_ckks_config_N13_L8.json. I do see high accuracy (~97.5%) with the currents weights using the current master branch.

[fboemer]

As it turns out, HE_SEAL is not supported seems to be thrown whenever he-transformer throws an error, rather than printing the more descriptive reason. This is a reported issue with the ngraph-bridge.

Could you confirm NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json exists? (i.e. $HE_TRANSFORMER is defined)

[tiogoGit]

Yes, it exists.

[cjbumgardner]

I'll chime in and say I'm have the same problem as the poster. I also have HE_TRANSFORMER defined, and the configs file mentioned exists. Also, make install succeeded. Although, even after a few make install runs, I don't get "libhe_seal_backend.so" in the virtual environment bin; it is in "external/lib", though. There is no "he_seal_backend.so" anywhere. Here's output for folder contents:

ubuntu@******:~/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib$ ls
python2.7 python3.6

ubuntu@******:~/he-transformer/build/external/lib$ ls
cmake libiomp5.so libngraph_bridge.so libprotobuf-lite.la libprotobuf.a libprotobuf.so.20.0.1 libprotoc.so.20 libtbb.so.2
libcpu_backend.so libmkldnn.so libngraph_test_util.a libprotobuf-lite.so libprotobuf.la libprotoc.a libprotoc.so.20.0.1 pkgconfig
libhe_seal_backend.so libmklml_intel.so libnop_backend.so libprotobuf-lite.so.20 libprotobuf.so libprotoc.la libseal.a
libinterpreter_backend.so libngraph.so libprotobuf-lite.a libprotobuf-lite.so.20.0.1 libprotobuf.so.20 libprotoc.so libtbb.so

[fboemer]

Thank you, @tiogoGit , @cjbumgardner for your reports. Here's more details on what the installation looks like at each step. Note, there is no he_seal_backend.so, just libhe_seal_backend.so, and build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib should be empty after all. Please let me know which, if any, of the below steps fails.

1. cmake ..
2. make install

• $HE_TRANSFORMER/build/external/lib/libhe_seal_backend.so should exist
• $HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/lib/libhe_seal_backend.so should exist, and be a soft link to $HE_TRANSFORMER/build/external/lib/libhe_seal_backend.so

3. You should be able to run ./test/unit-test
4. source external/venv-tf-py3/bin/activate
5. NGRAPH_TF_BACKEND=HE_SEAL python $HE_TRANSFORMER/examples/ax.py should work and print NGraph using backend: HE_SEAL This indicates the installation worked as expected
6. cd $HE_TRANSFORMER/examples/MNIST/Cryptonets
7. NGRAPH_VOPS=all NGRAPH_TF_BACKEND=HE_SEAL python test.py runs inference on unencrypted data using HE_SEAL backend
8. NGRAPH_ENCRYPT_DATA=1 \ NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \ NGRAPH_TF_BACKEND=HE_SEAL \ python test.py --batch_size=4096 NGRAPH_VOPS=all NGRAPH_TF_BACKEND=HE_SEAL python test.py should result in ~89% accuracy (higher accuracy can be found with larger parameters, or perhaps re-training the model)

[cjbumgardner]

Hi Fabian, Thank you for your email. I started with fresh ubuntu 18.04 amazon cpu only instance, and did an install from scratch. I still ran into the same problems. I also made a docker container from an image of he-transformer, and I included some comments about that experience that could be relevant at the end. Below, I give some of the terminal outputs. I appreciate all the feedback. Thanks again! And hope you're having a great weekend. Best, Christopher

1. cmake .. 2. make install - $HE_TRANSFORMER/build/external/lib/libhe_seal_backend.so should exist - $HE_TRANSFORMER/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/lib/libhe_seal_backend.so should exist, and be a soft link to $HE_TRANSFORMER/build/external/lib/libhe_seal_backend.so

Everything up this point seemed fine, and the .so files/softlinks exist.

1. You should be able to run ./test/unit-test Last lines of the unit tests.

[----------] Global test environment tear-down [==========] 241 tests from 7 test cases ran. (39225 ms total) [ PASSED ] 241 tests. YOU HAVE 5 DISABLED TESTS

1. source external/venv-tf-py3/bin/activate 2. NGRAPH_TF_BACKEND=HE_SEAL python $HE_TRANSFORMER/examples/ax.py should work and print NGraph using backend: HE_SEAL This indicates the installation worked as expected

:~/he-transformer/build$ NGRAPH_TF_BACKEND=HE_SEAL python $HE_TRANSFORMER/examples/ax.py /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) 2019-09-21 17:50:30.249827: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2394495000 Hz 2019-09-21 17:50:30.250166: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d1a930 executing computations on platform Host. Devices: 2019-09-21 17:50:30.250197: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined> Traceback (most recent call last): File "/home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call return fn(*args) File "/home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/he-transformer/examples/ax.py", line 26, in <module> f_val = sess.run(f, feed_dict={b: np.ones((1, 4))}) File "/home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run run_metadata_ptr) File "/home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "/home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run run_metadata) File "/home/ubuntu/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported All of the following tests have the same problem with HE_SEAL not supported.

1. cd $HE_TRANSFORMER/examples/MNIST/Cryptonets 2. NGRAPH_VOPS=all NGRAPH_TF_BACKEND=HE_SEAL python test.py runs inference on unencrypted data using HE_SEAL backend 3. NGRAPH_ENCRYPT_DATA=1 \ NGRAPH_HE_SEAL_CONFIG=$HE_TRANSFORMER/configs/he_seal_ckks_config_N13_L7.json \ NGRAPH_TF_BACKEND=HE_SEAL \ python test.py --batch_size=4096 NGRAPH_VOPS=all NGRAPH_TF_BACKEND=HE_SEAL python test.py should result in ~89% accuracy (higher accuracy can be found with larger parameters, or perhaps re-training the model) —

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Since there could be underlying package conflicts that I have no idea about, I searched for a docker image. I bring this up only because I got a little further with testing, but still had problems. Below is from a docker image on dockerhub of he-transformer: *tataucloud/cryptonet*. I'm not sure of this version of the he-transformer, but I can see there isn't MobileNetV2 as in the he-transformer git repository. Very, very strangely, someone else I knew ran tests from the same docker image, and we had conflicting results. Both run from most recent mac os (not that this should matter for docker), both on CPU. We could both successfully run: GRAPH_TF_BACKEND=HE_SEAL_CKKS python $HE_TRANSFORMER/examples/axpy.py GRAPH_TF_BACKEND=HE_SEAL_BFV python $HE_TRANSFORMER/examples/axpy.py (slightly different file name in the docker, but its like ax.py). I'll note that I ran these same tests on the amazon instance where I did the 'make install' process, and it had the same error outputs as before except changing the backend names. But only he could run the following on his docker image: GRAPH_TF_BACKEND=HE_SEAL python $HE_TRANSFORMER/examples/axpy.py (venv-tf-py3) root@4aa96af9df6f:/he-transformer/examples# NGRAPH_TF_BACKEND=HE_SEAL_CKKS python axpy.py TensorFlow version installed: 1.12.0 (v1.12.0-0-ga6d8ffa) nGraph bridge built with: 1.12.0 (v1.12.0-0-ga6d8ffa) [INFO] 2019-09-21T18:25:16z src/seal/bfv/he_seal_bfv_backend.cpp 64 Using SEAL BFV default parameters [INFO] 2019-09-21T18:25:16z src/seal/ckks/he_seal_ckks_backend.cpp 82 Using SEAL CKKS default parameters [INFO] 2019-09-21T18:25:16z src/seal/ckks/he_seal_ckks_backend.cpp 86 Error config_path is NULL [INFO] 2019-09-21T18:25:16z src/seal/ckks/he_seal_ckks_backend.cpp 87 Error using NGRAPH_HE_SEAL_CONFIG. Using default [WARN] 2019-09-21T18:25:16z src/seal/ckks/he_seal_ckks_backend.cpp 220 Custom coefficient modulus has total bit count 120 which is greater than the default bit count 27, resulting in lower security [INFO] 2019-09-21T18:25:16z src/seal/he_seal_util.hpp 47 / Encryption parameters: | scheme: HE:SEAL:CKKS | poly_modulus: 1024 | coeff_modulus size: 120 bits \ noise_standard_deviation: 3.2 [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 331 [ Parameter_1 ] [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 336 Parameter shape {} [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 331 [ Parameter_0 ] [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 336 Parameter shape {2, 2} [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 331 [ Constant_2 ] [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 341 Constant shape {2, 2} [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 467 Inputs: [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 475 Outputs: Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 413 Constant_2 took 1ms [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 331 [ Broadcast_4 ] [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 467 Inputs: Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 475 Outputs: Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 413 Broadcast_4 took 2ms [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 331 [ Add_3 ] [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 467 Inputs: Plain, Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 475 Outputs: Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 413 Add_3 took 3ms [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 331 [ Multiply_5 ] [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 467 Inputs: Plain, Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 475 Outputs: Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 413 Multiply_5 took 4ms [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 331 [ Result_6 ] [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 467 Inputs: Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 475 Outputs: Plain [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 413 Result_6 took 0ms [INFO] 2019-09-21T18:25:16z src/he_backend.cpp 421 Total time 10 (ms) Result: [[10. 15.] [20. 25.]] (venv-tf-py3) root@4aa96af9df6f:/he-transformer/examples# NGRAPH_TF_BACKEND=HE_SEAL python axpy.py TensorFlow version installed: 1.12.0 (v1.12.0-0-ga6d8ffa) nGraph bridge built with: 1.12.0 (v1.12.0-0-ga6d8ffa) Traceback (most recent call last): File "/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported During handling of the above exception, another exception occurred: Traceback (most recent call last): File "axpy.py", line 27, in <module> f_val = sess.run(f, feed_dict={b: np.ones((2, 2)), c: np.array(5, )}) File "/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build/venv-tf-py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InternalError: NGRAPH_TF_BACKEND: HE_SEAL is not supported

…

---------------------------------------------------------------------------------------------------------- Below is also from the same Docker image. I trained the cryptonet with python train.py. This may be some other unrelated issue. Everything seemed to work and then the process was killed. Killed (venv-tf-py3) root@4aa96af9df6f:/he-transformer/examples/MNIST-Cryptonets# NGRAPH_ENCRYPT_DATA=1 NGRAPH_BATCH_DATA=1 NGRAPH_BATCH_TF=1 NGRAPH_HE_SEAL_CONFIG=../../test/model/he_seal_ckks_config_13.json NGRAPH_TF_BACKEND=HE_SEAL_CKKS python test.py --batch_size=4096 --report_accuracy=1 TensorFlow version installed: 1.12.0 (v1.12.0-0-ga6d8ffa) nGraph bridge built with: 1.12.0 (v1.12.0-0-ga6d8ffa) Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz [INFO] 2019-09-21T03:24:02z src/seal/ckks/he_seal_ckks_backend.cpp 66 Using SEAL CKKS config with 7 30-bit coefficients [INFO] 2019-09-21T03:24:04z src/seal/he_seal_util.hpp 47 / Encryption parameters: | scheme: HE:SEAL:CKKS | poly_modulus: 8192 | coeff_modulus size: 210 bits \ noise_standard_deviation: 3.2 [INFO] 2019-09-21T03:24:04z src/seal/ckks/he_seal_ckks_backend.cpp 284 Creating batched plain tensor with shape 4096, 784 [INFO] 2019-09-21T03:24:04z src/seal/ckks/he_seal_ckks_backend.cpp 284 Creating batched plain tensor with shape 4096, 10 [INFO] 2019-09-21T03:24:04z src/he_backend.cpp 252 Encrypting data [INFO] 2019-09-21T03:24:04z src/he_backend.cpp 255 Batching data [INFO] 2019-09-21T03:24:04z src/he_backend.cpp 290 Encrypting parameter 0 [INFO] 2019-09-21T03:24:04z src/he_backend.cpp 298 plain_input->get_batched_element_count() 784 [INFO] 2019-09-21T03:24:06z src/he_backend.cpp 305 Done encrypting parameter 0 [INFO] 2019-09-21T03:24:06z src/he_backend.cpp 331 [ Parameter_0 ] [INFO] 2019-09-21T03:24:06z src/he_backend.cpp 336 Parameter shape {4096, 784} [INFO] 2019-09-21T03:24:06z src/he_backend.cpp 331 [ Constant_17 ] [INFO] 2019-09-21T03:24:06z src/he_backend.cpp 341 Constant shape {100, 10} [INFO] 2019-09-21T03:24:06z src/he_backend.cpp 467 Inputs: [INFO] 2019-09-21T03:24:06z src/he_backend.cpp 475 Outputs: Plain [INFO] 2019-09-21T03:24:08z src/he_backend.cpp 413 Constant_17 took 1555ms [INFO] 2019-09-21T03:24:08z src/he_backend.cpp 331 [ Constant_14 ] [INFO] 2019-09-21T03:24:08z src/he_backend.cpp 341 Constant shape {845, 100} Killed

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#271?email_source=notifications&email_token=AFMYZMA42PSMSNSLI7HNYYDQKOXK3A5CNFSM4IP7APW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7EF6UA#issuecomment-533225296>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFMYZMBBMOWQOCE4SZSNO3TQKOXK3ANCNFSM4IP7APWQ> .

-- Christopher Bumgardner, PhD Department of Mathematics University of California, Davis

[fboemer]

Thanks @cjbumgardner for your detailed report again. A few notes:

1. Once merged, shrestha/backend_error_message_fixes tensorflow/ngraph-bridge#229 will hopefully return a more useful error message
2. I wouldn't recommend using older versions. There have been many improvements in the functionality and runtime in recent months. We used to support two backends, HE_SEAL_CKKS, and HE_SEAL_BFV. The former has been renamed HE_SEAL, the latter is no longer supported.
3. TensorFlow version installed: 1.12.0 suggests you may be using an older version. We expect TensorFlow version 1.14
4. Would you also try

python -c "import ngraph_bridge; print(ngraph_bridge.list_backends())"

We expect HE_SEAL to appear in the list. If so, this suggests the package is installed correctly, there is just some error when creating backend instance.

[cjbumgardner]

Hi Fabian, Thank you fro your reply. I set up an ubuntu16.04 docker image from the he-transformer github. Everything runs fine. I wish I could point to what the problem was, but obviously it's system dependent. I will mention that at least with Docker v19.03 community edition, the make_docker_image.sh file throws an error due to punctuation in the name of the docker image. It's probably just the + sign in the date appended to the image name, but I changed every non alpha-numberic character to an underscore (and that fixed it). Thank you for all you feedback. The cryptonets (after training) gives a test score of over 90% accuracy (this was the unencrypted test), but when using test.py on encrypted input, it yields around 10% accuracy as others have noted. I haven't investigated this and I don't know if noise is monitored when experimenting with encryption settings, but a place I plan to investigate (given everything seems to work ok) is possible noise build up past useful decryption (this is probably stated somewhere but I was under the impression that the SEAL library doesn't support bootstrapping yet). Thank you again, Christopher

…

On Tue, Sep 24, 2019 at 7:02 AM Fabian Boemer ***@***.***> wrote: Thanks @cjbumgardner <https://github.com/cjbumgardner> for your detailed report again. A few notes: 1. Once merged, tensorflow/ngraph-bridge#229 <tensorflow/ngraph-bridge#229> will hopefully return a more useful error message 2. I wouldn't recommend using older versions. There have been many improvements in the functionality and runtime in recent months. We used to support two backends, HE_SEAL_CKKS, and HE_SEAL_BFV. The former has been renamed HE_SEAL, the latter is no longer supported. 3. TensorFlow version installed: 1.12.0 4. Would you also try python -c "import ngraph_bridge; print(ngraph_bridge.list_backends())" We expect HE_SEAL to appear in the list. If so, this suggests the package is installed correctly, there is just some error when creating backend instance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#271?email_source=notifications&email_token=AFMYZMFW2R2LMR4FFEW2XW3QLIMW5A5CNFSM4IP7APW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7OO7DA#issuecomment-534572940>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFMYZMEK5H5VIL2355OUI2DQLIMW5ANCNFSM4IP7APWQ> .

-- Christopher Bumgardner, PhD Department of Mathematics University of California, Davis