Reconsidering my foolishness

I took a few minutes to try option B on this serial-to-network problem.

Here’s a much simpler option.

On the Ministation, run a script like (this is the lightly tarted up version):

#!/bin/sh

PORT=/dev/ttyACM0
SERVER=localhost
SERVER_PORT=1234

stty -F $PORT 57600

while [ true ];
do
cat $PORT | nc $SERVER $SERVER_PORT
sleep 1
done

The client netcat (nc) will die if it can’t connect to the server, hence the while loop. The sleep prevents flapping in that case. [Though as I discovered below, this only works as long as bits are coming into the serial port…]

And on the other end:

#!/bin/sh

PORT=1234
OFILE=/tmp/ofile.txt

nc -kl $PORT >> $OFILE

Time to try it out, I think…

Update:

Here’s an even more tarted up version which lets you send data back out the serial port by poking it to the server through a FIFO.

Client:

#!/bin/sh

PORT=/dev/ttyUSB0
SERVER=localhost
SERVER_PORT=1234

stty -F $PORT 57600

while [ true ];
do
cat $PORT | nc $SERVER $SERVER_PORT > $PORT
sleep 1
done

Server:

#!/bin/sh

PORT=1234
OFILE=/tmp/ofile.txt
OUTFIFO=/tmp/foofifo

if [ ! -p $OUTFIFO ]
then
mkfifo $OUTFIFO
fi
tail -f $OUTFIFO | nc -kl $PORT >> $OFILE

This seems pretty robust to most combinations of server or client problems except if the client can’t reach the server and there’s no data coming in the serial port. The client’s nc will die but the associated cat won’t die till it tries to write to the broken pipe.

Does this matter? Yes. If the client starts and can’t connect to the server, the nc will die, and if the widget is mis-configured to not output any data on the serial port, the (FIFO)->(server’s nc)->(client’s nc)->(serial port) redirection won’t work because the client’s nc is dead, but because the cat is still running, the while(1) won’t cycle. There’s probably a clever way around this, but I’ve definitely reached the limits of my shell scripting abilities.

Of course, if you have a network connection you could always SSH to the client and poke the serial port manually.

Ah, and these scripts will require old-skool logfile rotation:

cp ofile.txt ofile.txt.2 && cat /dev/null > ofile.txt

I assume logrotate can handle that elegantly.

2nd side-note. The Ministation SDK doesn’t build busybox’s stty by default. You’ll probably want that for setting the baud rate on the serial port. It’s easily added by changing the relevant line in conf/xs2/busybox.config to:

CONFIG_STTY=y

ser2syslog

For the weather buoy project we need a way to take data in over our Ministation’s serial port and push it upstream over the network. As the ministation is pretty limited in flash and ram, I assumed a compiled (as opposed to scripted) solution would be best.

I think my ideal system would be store-and-forward, with the serial port as a “push,” something along the lines of:

  1. Open the serial port.
  2. Forever:
    1. Read an buffer the data.
    2. When you have one “line” of data, try to poke it out over a network interface.
    3. If you succeed, forget that line of data.
    4. If you fail, save it to try next time. When you reach some maximum amount of backlog, start dropping the older messages.

Where the other end of the conversation would be a network-to-disk server:

  1. Open a network port
  2. As data comes in, write it to a file.

It sounds like a simple brief, but I couldn’t find a canned solution which does the job.

One obvious choice is ser2net, though it’s actually designed the other way around. As a “server,” it listens on a network connection (many, actually), opening serial ports as necessary when a connection comes in.

I’m a bit rusty on Linux network programming, so as an interim, I thought I’d try a serial-to-syslog gateway, operating as above but pushing data to syslog (who can then handle the network forwarding).

It’s not a terrible solution, with a couple of caveats:

  • It only really works because our data is read-only, and it’s NMEA-like strings, so it’s ASCII, nicely delimited into lines. Syslog wouldn’t work with binary data, I don’t think.
  • The Ministation SDK runs busybox’s syslogd, which only supports UDP forwarding, not TCP. OK, so it would only provide one additional ACK of security, not true store and forward, but it’s would be a start.

On the other end of the connection, I’ll have a full-fat computer running rsyslogd which can handle a bit of store and forward to servers off in the cloud.

In any case, I wrote a ser2syslog. I won’t pretend it’s polished, but works for me. As per the README, I was originally going to build it from the framework of ser2net, but given ser2net is a mature, featureful product, and arranged backwards (as above), I stripped 95% of the ser2net code out. I believe (hope!) the copyright and attribution are correct.

Next up is testing it on a Ministation.

Update:

Hm. I just realized I could do the same thing very simply on the command line. Something like:

$ cat /dev/ttyS0 | logger -p local7.info -t “/dev/ttyS0”

And I could probably achieve what I need network-wise with netcat. Sigh….

PhD Proposal

As part of my studies, I’m required to submit a research proposal at about 6 months into my program. I did that long ago, then subsequently binned that topic.

Since January, I’ve been working on a new proposal, which I’ve posted here.

I’ll fully admit it’s too vague, and too long. But it’s also submitted and out of my hair.

Building custom Ubiquiti firmwares

As noted previously, I’m mucking about with a Ubiquiti Ministation2 as a networked SBC for a project. The Ministation2 is Linux-based, and Ubiquiti provide an SDK for rolling your own firmware. I wanted a mechanism for introducing my own customizations while also keeping the tarball downloads from Ubiquiti at arms length.

After a bit of iteration, I ended up with the a project I’ve posted at Github. It’s a kind of reverse-patch-and-overlay environment, and no, I’m not a shell and Makefile genius, so I accept there’s lots of room for improvement.

Basically, you check out the the Remix,

$ git clone git://github.com/amarburg/Ubiquiti-SDK-Remix.git

then:

$ cd Ubiquiti-SDK-Remix
$ ./bootstrap.sh

This will download the SDK from Ubiquiti (the version is currently hardcoded in bootstrap.sh), expand it on top of itself, and patch the SDK’s files as appropriate. You end up with a hybrid of the original SDK with the Remix’s improvements.

The Remix won’t check that you have the relevant tools, so make sure you’ve installed the MIPS toolchain and other dependencies. Then:

$ make clean xs2

as before.

The remix currently includes just one new package — ser2net-2.7 (it builds, but totally untested on the Ministation), but it provides a good template for how to include other packages in apps/local/.

Custom firmware on the Ubiquiti Ministation


We’re using a Ubiquiti Ministation as wifi-enabled SBC for a weather station project. There’s a lot to like, and a few things lacking. It has a single 802.11b/g interface, a single ethernet port, and a single console/serial port. Like? Most everything. Dislike? The single serial port, as we want to use it as a serial-to-network gateway, which means sacrificing the console. It also runs a custom Linux installation, which we can hack it to our devious needs.

Instructions for rebuilding the system are out there, but pretty brief. Here’s my experiences.

Ubiquiti provide the SDK as a tarball, as well as a MIPS toolchain as a i386 deb. That’s fine with me as I’m running Ubuntu, though sadly I’m running 64-bit, so I built up a small VirtualBox machine for development. Not optimal, but nothing wrong with it. It’s noted here that you could also build the MIPS compiler with buildroot. your mileage may vary.

So:

  1. Download the SDK and toolchain from here, selecting the Ministation2 as the model. As of right now, the current firmware is v3.6.1.
  2. Install the toolchain on a i386 Debian/Ubuntu system.
  3. You’ll also need a few other packages. I got this list from here, so I can’t verify it’s correctness.

    $ sudo apt-get install sharutils fakeroot zlib1g-dev patch lzma flex bison

    If a package is missing, you’ll notice pretty quickly when the compile blows up.

  4. Unpack the SDK someplace convenient:

    $ mkdir ministation
    $ cd ministation
    $ tar -xjvf ~/SDK.UBNT.v3.6.1.4873.tar.bz2
    $ cd SDK.UBNT.v3.6.1.4873

First, let’s see how the SDK works out of the box. Note, I already have my ministation up, running, and configured properly. So what happens when I “upgrade” my ministation with a filesystem I’ve built myself.

First, build it:

$ make xs2

And go get a refreshing drink.

The resulting package will end up in a directory with a name like rootfs/XS2.ar2316.v3.6.1.unknown.110707.1210 which gives the Ubiquiti model (XS2), the chip (ar2316), version number (v3.6.1) and a date/timestamp (110707.1210). The unkown is where Ubiquiti puts a patch number (like 4873 in this case), not sure why that doesn’t make it through.

Conveniently, they also symlink in rootfs/XS2.ar2316.v3.6.1.latest which points to the latest build.

In that directory are a number of intermediate results, as well as a few .bin packages:

$ ls -al
total 38976
drwxr-xr-x 2 aaron aaron 4096 2011-07-14 16:17 .
drwxr-xr-x 14 aaron aaron 4096 2011-07-14 16:22 ..
-rw-r–r– 1 aaron aaron 678439 2011-07-14 16:17 bzImage
-rwxr-xr-x 1 aaron aaron 2605056 2011-07-14 16:17 squashfs.bin
-rwxr-xr-x 1 aaron aaron 493554 2011-07-14 16:17 vmlinux.lzma
-rwxr-xr-x 1 aaron aaron 25055859 2011-07-14 16:17 vmlinux.notstripped
-rw-r–r– 1 aaron aaron 3099018 2011-07-14 16:17 XS2.ar2316.v3.6.1.unknown.110714.1614-8M.bin
-rw-r–r– 1 aaron aaron 3099018 2011-07-14 16:17 XS2.ar2316.v3.6.1.unknown.110714.1614.bin
-rw-r–r– 1 aaron aaron 4858567 2011-07-14 16:17 XS2.ar2316.v3.6.1.unknown.110714.1614.debug_binaries.notstripped.tgz

Comparing XS2.ar2316.v3.6.1.unknown.110714.1614.bin to the “stock” Ubiquiti filesystem:

$ ls -l
total 9856
-rw-r–r– 1 amarburg amarburg 3244841 2011-07-19 10:50 MiniStation2-v3.6.1.build4866.bin

my version is a bit smaller. Hmm. Well, never mind.

Install the new version of the s/w through the the web interace. Luckily, the Ministation warns me I’m about to break things:



Well, let’s go. I monitored the process on the serial console (which is a separate topic).

Writing ‘FIS directory ‘ to /dev/mtd4(FIS directory ) … [[1300398.770000] Restarting system.
+Ethernet eth0: MAC address 00:15:6d:a4:14:ed
IP: 0.0.0.0/255.255.255.0, Gateway: 0.0.0.0
Default server: 0.0.0.0

RedBoot(tm) bootstrap and debug environment [ROMRAM]
Ubiquiti Networks certified release, version 0.9.00483.1103151313 – built 13:14:44, Mar 15 2011

Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc.

Board: Ubiquiti Ubiquiti AR2315/6/7/8 based board (Ubiquiti PicoStation2 detected)

Arch: ar2316
RAM: 0x80000000-0x82000000, [0x80040b80-0x80fe1000] available
Flash: 0xbfc00000-0xbfff0000, in 64 blocks of 0x00010000 bytes each
== Executing boot script in 1.000 seconds – enter ^C to abort
RedBoot> cache off
RedBoot> fis load -d -e kernel
Trying LZMA decompression…
Image loaded from 0x80041000-0x801df23c
Entry point: 0x80196040, address range: 0x80042000-0x801bf000
RedBoot> go
[ 0.000000] CPU revision is: 00019064
[ 0.000000] Primary instruction cache 16kB, physically tagged, 4-way, linesize 16 bytes.
[ 0.000000] Primary data cache 16kB 4-way, linesize 16 bytes.
[ 0.000000] Linux version 2.4.27-ubnt0 (aaron@moa) (gcc version 3.3.3) #2 Tue Jul 19 10:55:22 NZST 2011

Much to my surprise, it worked. And, much to my surprise, my configuration has been maintained. Sweet.

Experiments with MogileFS

OK, OK, I’m a server nerd.    More to the point, I’m a storage nerd, and I have a big man crush on Isilon.  The whole idea just seems right — everytime you add disks, you also add CPUs and network interfaces to keep your chakras in balance.   And all your nodes form a massive Infiniband cabal of data and metadata sharing, providing plausible deniability in the event of hardware failure (can you tell I’ve been reading Among the Truthers?)

Anyway, I’m drawn to every project which claims to offer similar properties in an open-source format.  Today’s example: MogileFS.

I successfully installed Mogile on a low-end Ubuntu machine, so I’ve decided to set up a fresh install on handy, much gruntier CentOS 5.5 machine.   I’ll start with a fairly simple installation using the Mogile tracker (mandatory), and their backend storage server (optional, could be any WebDAV server, I think).    Just one node at first, as both tracker and storage, with room for expansion, obviously.  No other load balancers or proxies, as this is mostly for distributing data during processing and I expect a lot of random access.  MySQL database.

I’m trying to install as normal as much as possible rather a system-level user.  Obviously I’m using root privileges to install new RPMs..

I’m basically tracking the HOWTO from the MogileFS Google Code wiki.

I assume all of the necessary package dependencies are taken care of (like a Mysql server, Perl, and the Perl-Mysql libraries).

Pull the latest code from CPAN:

sudo perl -MCPAN -e ‘install “MogileFS:Server”;’ -e ‘install “MogileFS:Utils”‘

On this CentOS 5.5 system, the default system CPAN/Perl configuration didn’t seem to be current enough to find the packages. I had to do a massive CPAN package update to get this to work.

Once the Perl is all loaded, create the necessary Mysql entries.

$ mysql -p
Enter password:

Type ‘help;’ or ‘h’ for help. Type ‘c’ to clear the buffer.

mysql> CREATE DATABASE mogilefs;
mysql> GRANT ALL ON mogilefs.* TO ‘mogile’@’%’;
mysql> SET PASSWORD FOR ‘mogile’@’%’ = OLD_PASSWORD( ‘sekrit’ );
mysql> FLUSH PRIVILEGES;
mysql> quit

And get mogile to setup its databases:

$ mogdbsetup –dbname=mogilefs –dbuser=mogile –dbpassword=foobar

With a clean installation, the first step is to set up one or more trackers. First, create a configuration file (I’m using ~/usr/etc/mogilefsd.conf)

db_dsn = DBI:mysql:mogilefs;port=3306;mysql_connect_timeout=5
db_user = mogile
db_pass = barfoo
conf_port = 7001
listener_jobs = 5
node_timeout = 5
rebalance_ignore_missing = 1

Relative to the HOWTO, I’ve omitted the bit in db_dsn which specifies the database server, as it’s running on the same machine.

Then start the tracker with:

$ mogilefsd -c ~/usr/etc/mogilefsd

Without the “–daemonize” flag, it will run in the foreground for testing.

To simplify later steps, store the IP addresses of my trackers in a conf file:

$ echo “trackers = 127.0.0.1:7001” > ~/.mogilefs.conf

Next configure a storage server (again, on the same machine .. it’s just a test setup).

Create a configuration file (~/usr/etc/mogstored.conf for me)

httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/home/myhome/usr/var/mogdata

Before starting the daemon, add the storage server to the database, and the devices to the server:

$ mogadm host add wrcws –ip=132.181.86.104
$ mogadm device add wrcws 1

I manually specified the IP address for the host because I didn’t want to inadvertently decide the server was at ‘127.0.0.1’ — which would break access from other machines.

You can check your hosts and devices with:

$ mogadm host list
wrcws [1]: down
IP: 132.181.86.104:7500

$ mogadm device list
wrcws [1]: down
used(G) free(G) total(G)
dev1: alive 0.000 0.000 0.000

Of course, wrcws shows as down because I haven’t started the server yet. Do that with:

$ mogstoraged -c ~/usr/etc/mogstored.conf

Again no “–daemonize” so we get lovely debugging in the foreground.

Hmm. That didn’t fix things immediately.

Ah, forgot Centos comes with SELinux on full-attack by default. Add access through firewall to ports 3306 (MySQL), 7500 and 7501 (Mogilefs storage), 7001 (Mogilefs tracker).

Restarting the daemons provided a bit of success.

Ahha. Also, I need to remember that the devices (“dev1”) must be created as a directory under your “docroot” (or, a whole disk drive could be mounted there, for example). Forgot to do that and got slightly cryptic messages:

[Fri Jul 15 00:35:02 2011] [monitor(20561)] Port 7500 not listening on 132.181.86.104 (http://132.181.86.104:7500/dev1/usage)? Error was: 404 Not Found

Actually creating the dev directories:

$ mkdir ~/usr/var/mogdata/dev1

leads to something like:

$ mogadm check
Checking trackers…
132.181.86.104:7001 … OK

Checking hosts…
[ 1] wrcws … OK

Checking devices…
host device size(G) used(G) free(G) use% ob state I/O%
—- ———— ———- ———- ———- —— ———- —–
[ 1] dev1 376.754 0.421 376.333 0.11% writeable 0.0
—- ———— ———- ———- ———- ——
total: 376.754 0.421 376.333 0.11%

Neat.

Now a quick test. Create a domain and class:

$ mogadm domain add testdomain
$ mogadm class add testdomain testclass

And run a quick test, sending the file /etc/services in the cluster, then getting it back out again.

$ mogupload –domain=testdomain –key=’/etc/services’ –file=’/etc/services’
$ mogfetch –domain=testdomain –key=’/etc/services’ –file=”-”
# /etc/services:
# $Id: services,v 1.42 2006/02/23 13:09:23 pknirsch Exp $
#
# Network services, Internet style
#

Great.

As a second step, I wanted to take my original Ubuntu-based server and add it to my new cluster.

On the Ubuntu machine (whose setup otherwise mirrors the one described here), modified mogilefsd.conf to point to the Mysql database. Also change my ~/.mogilefs.conf files to reflect both trackers.

trackers = wrcws.canterbury.ac.nz:7001,open.grc.canterbury.ac.nz:7001

And add the new host and devices to the system:

$ mogadm host add open –ip=132.181.86.22
$ mogadm device add open 2

Apparently device numbers need to be unique across the whole system?

And you end up with:

$ mogadm check
Checking trackers…
wrcws.canterbury.ac.nz:7001 … OK
open.grc.canterbury.ac.nz:7001 … OK

Checking hosts…
[ 1] wrcws … OK
[ 2] open … OK

Checking devices…
host device size(G) used(G) free(G) use% ob state I/O%
—- ———— ———- ———- ———- —— ———- —–
[ 1] dev1 376.754 0.423 376.331 0.11% writeable 0.0
[ 2] dev3 213.182 138.408 74.774 64.92% writeable 0.0
—- ———— ———- ———- ———- ——
total: 589.937 138.831 451.106 23.53%

Awesome.