Java / Python Data Exchange with the Redis Memory Database

As the majority of my code base is in Java, integrating my Python machine learning algorithms can sometimes be a challenge.

For my Fido project (a home intrusion detection system based on my Intruder Net that generates a image a second from an IP camera and will alert me when an "intruder" appears in the image), I had the following requirements:

• Be able to transfer an image from Java to Python (i.e., ability to send binary data)
• Be able to transfer data from Python to Java
• Image transfer and Python result need to occur in <1 second
• File system should never be touched (for efficiency and speed everything should reside in memory)
• Java and Python code bases should maintain independence
• Java, Python, middleware should be able to communicate via TCP/IP (assuming memory / network data transfer is fast)
• Message exchange should be simple

I was curious if a memory only database would work and have been very please with the Redis database.

Proof of Concept

My PoC code is include in the REDIS_POC.zip file below.

You will also need the latest release of the Redis database (redis-5.0.8.tar.gz).

This test code does the following, using Redis as the linking mechanism between my Java and Python code:

Code base	Description	Code
Java	Converts image to byte array	byte[] byteArrayImage = IOUtils.toByteArray(in);
Java	Uses the Redis increment feature to update a counter for the image	jedis.incr("counter");
Java	Get the new counter number from Redis	String count = jedis.get("counter");
Java	Uses the Redis set feature to upload the byte array image with the ID of the counter (the image ID is prefixed with "img:")	binJedis.set(("img:" + count).getBytes(), byteArrayImage);
Java	Loop until Python has returned an answer "infer:" of the same counter ID	pythonResponse = jedis.get("infer:" + count);
Python	Loop until counter appears or is updated	counter = r.get("counter")
Python	Create the Redis key for image retrieval	img_counter = "img:" + counter.decode("utf-8")
Python	Retrieve the image byte array from Redis	image = r.get(img_counter.encode())
Python	For debug purposes, show the retrieved image	show_image(image)
Python	For PoC purposes, instead of using tensorflow to infer the image, generate a random code to send to Java	infer_intruder = random.randint(0, 1)
Python	Delete the image from Redis	r.unlink(img_counter)
Python	Send the infer result to Java (in this case either a 0 or a 1) (the infer ID is prefixed with "infer:")	r.set("infer:" + counter.decode("utf-8"), infer_intruder)

 

My environment uses the following:

Java Environment	JDK 11 IDE: Eclipse 2019-09 Choose RedisPOC (Java Workspace) as your Eclipse Workspace Be sure to load the libraries in the project lib folder
Python Environment	Python 3.7 IDE: PyCharm Community 2019.3 Install the redis library via pip or conda Choose RedisPOC (python pycharm) as your project
Redis	Install most recent release (redis-5.0.8.tar.gz)

 

Update the TestPythonDataExchangeViaRedis.java with the IP Address of the Redis database:

static String Redis_IPAdd = "192.168.159.39";

Update the test_image_from_redis.py with the IP address of the Redis database:

redis_ip = "192.168.159.39"

I made the following changes to the redis.conf (for debugging I enable Redis to save to the filesystem):

bind 127.0.0.1 protected-mode yes loglevel debug save 900 1 save 300 10 save 60 10000 aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes*