⚠️ Please migrate to lukasmartinelli/pipecat. This repository is no longer maintained by Lukas Martinelli.

redis-pipe allows you to treat Redis Lists as if they were Unix pipes. It connects stdin and stdout with LPUSH and LPOP. For AMQP based messages queues like RabbitMQ or ActiveMQ check out the successor project pipecat.

Set the REDIS_HOST and REDIS_PORT environment variables for easy configuration or pass --host and --port arguments.

Pipe in value to redis-pipe and it will LPUSH them to the Redis List.

echo "hi there" | redis-pipe greetings

If you call redis-pipe with a tty attached it will LPOP all values from the Redis List and write them to stdout.

redis-pipe greetings

You can also limit the amount of values popped from the list.

redis-pipe --count 100 greetings

Support for blocking mode with BLPOP is not supported yet.

In this sample we pipe the syslog to a Redis List called logs.

tail -f /var/log/syslog | redis-pipe logs

You can now easily collect all the syslogs of your machines on a single server.

redis-pipe logs > logs.txt

Create jobs and store them.

cat jobs.txt | redis-pipe jobs

Process jobs on several workers and store the results.

redis-pipe --count 10 jobs | python do-work.py | redis-pipe results

Collect the results.

redis-pipe results > results.txt

Simply download the release and extract it. Add the binary install path to your ~/.bash_rc or ~/.bash_profile file e.g

wget https://github.com/lukasmartinelli/redis-pipe/releases/download/v1.4.1/redis-pipe_darwin_amd64.zip
unzip redis-pipe_darwin_amd64.zip
cd redis-pipe_darwin_amd64
cat << $ >> ~/.bash_profile
> export PATH = "$(pwd):\$PATH"
> $
$ source ~/.bash_profile
redis-pipe --help

wget https://github.com/lukasmartinelli/redis-pipe/releases/download/v1.4.1/redis-pipe_linux_amd64.tar.gz
tar -xvzf redis-pipe_linux_amd64.tar
cd redis-pipe_linux_amd64
cat << $ >> ~/.bash_rc
> export PATH = "$(pwd):\$PATH"
> $
$ source ~/.bash_rc
redis-pipe --help

$ go get github.com/lukasmartinelli/redis-pipe

To be able to use the the built binary in any shell, make sure to have your $GOPATH properly set in your ~/.bash_profile (on OS X) or ~/.bash_rc (on Linux) file e.g:

$ cat << ! >> ~/.bash_rc
> export GOPATH="\$HOME/gopath"
> export PATH="\$GOPATH:\$GOPATH/bin:\$PATH"
> !
$ source ~/.bash_rc