Friday, November 16, 2007

Cool Unix Tools

Today I was faced with dumping a postgres database, compressing it, downloading it, uncompressing it and restoring it on my local dev machine. It's a painfully slow process, which I've always had to supervise.

Surely, there must be a better way to do this...

Some small research time later I discovered netcat, or nc for short. nc lets you create a pipe over a network. As its man page states, it's a TCP/IP swiss army knife.

This is how I ended up automating most of the process.

On my local machine:
nc -l -p 1979 | bunzip2 | psql database_name

On the remote machine:
pg_dump database_name | bzip2 | nc my_local_ip 1979

Voila! The database dumped its data, piped it through bzip2 then over the network, to my waiting nc process which received the data, then piped it through bunzip2 and then into psql.

Cool. Another demonstration showing that Unix is such a great environment.


Anonymous said...

How about this:

ssh -C remote-machine pg_dump database_name | psql database_name

cowmix said...

You might want to check out 'socat' as a 'nc' replacement. Its functionality is a superset of 'nc'.

Simon Wittber said...

ssh -C remote-machine pg_dump database_name | psql database_name

This is cool, but I don't know of anyway to get ssh to compress its traffic. I'm transferring 2GB over a rather slow connection, so bzip2 saves me hours!

cowmix said...

"-C" does compress the traffic.. not as much as bzip2.. but good enough...

Simon Wittber said...

Ah cool. It pays to read man pages :)

-C Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11 and TCP connections). The compression algorithm is the same used by gzip(1).

Marinus said...

mmmm echo drop database;| bzip2 | nc my_local_ip 1979


Simon Wittber said...

Hahaha! A simple and irrefutable argument to use ssh instead of nc.

Popular Posts