Bracing against the wind

grun Job Scheduler moved to ZMQ

2013-06-03T08:30:00.000-07:00

As most people in cluster computing know, there really isn't a simple and "reall open source" solution out there for job scheduling. Oracle SGE, LSF and others are massive things, with over 100k lines of code, and extraordinarily complex configuration for things ranging from MPI support to Kerberos. And yet they lack simple features (script plugins for configuration), that would make them more versatile.

grun was written to be an "extremely lightweight" and yet big-featured job scheduler. The early version was not much more than "ssh to remote host, run job, wait for response", while logging and keeping track of resources. It's evolved to use a TCP messaging system allowing the compute nodes, queue nodes, and clients to communicate. By v 1.0 the plan is to have better support for arbitrary metrics, and better handling of priorities.

Going from 0.8 to v 0.9, I decided to try using the zeromq library instead of TCP. At first it was hard to remember that you really don't need to worry about things like sending to a socket you just created, even with no one on the other end.

The net result of the ZMQ port:

speed improved
- implicit perl moved to optimized c
- built-in multithreading takes better advantage of cpu
ability to stop/restart any queue without losing messages
improved reliability of message delivery
improved code organization orientated around messaging
20% smaller code base, because we removed:
- all "double checks" to see if connections are there
- code that "breaks up" large messages
- all issues with blocking/vs nonblocking i/o
- the whole "select" loop complexity

ZMQ is not perfect (yet), but it was an overall improvement over straight TCP. Because of the forking needed to launch jobs, I had to do some fiddling with dup'ed file descriptors to prevent zmq from acting wonky. The learning curve was worth it. I doubt I'll be using TCP again, especially since ZMQ has package support with most Linux distributions.

Convert absolute to relative links

2013-05-30T19:59:00.000-07:00

ln-abs2rel.pl


usage: ./ln-abs2rel.pl [-noexec] [-recursive]  ... 

Converts absolute to relative links.

Use -noexec to see what would be done.
Use -recursive to do this for all subdirs.

FIX: UBUNTU stops logging to /var/log/kern.log

2013-05-03T12:55:00.004-07:00

At some point the old kernel logger got upgrades to rsyslogd …. When this happened the logs owned by “messagebus” got left owned by "messagebus". Changing ownsership of these logs to "syslog" and restarting syslogd is sufficient to fix.

Baseball Warm-up Exercise for Kids : "Base Tag"

2013-03-18T08:43:00.004-07:00

A lot of times teams "warm up" by throwing and running while a coach tells them where to throw and when to run. This warm-up accomplishes that, but it's fun because it’s in a “game” structure, and a lot of the kids I've worked with are motivated by that. Plus, once they know the rules, they will play without supervision for quite a while - I’ve found you have to stop them or they will tire out.

Setup:

2 "fielders" with gloves and a ball stand a baseline-width apart
A 3rd kid is the “runner”, standing on base with the fielder who has the ball

Rules:

The runner starts running at any time, and the fielder with the ball tries to throw them out
The person with the ball can’t throw until after the runner starts running
Start by assuming the play is a force... any good throw is an out
For kids where this is easy…. it’s not a force…you’ve got to tag them in a rundown

Options … add points for kids who no longer seem motivated by the game (some kids love points, others don’t) ... for example:

Runner gets two points if he gets to the other base
Fielders get one point if they throw him out
Fielders lose two points if they overthrow!

Purpose:
The point is to develop and value consistency in young fielders… where unnecessary risks aren’t taken, and accuracy/speed are stressed. Points, rules, etc. change as skills and the coaches needs change.

2013-03-15T06:49:00.000-07:00

The worst thing a hardware or operating system vendor can do for the reliability and quality of their system is to try and make programming it easy.

Peter Sagal on Florida's Python Removal Program

2013-01-27T11:32:00.001-08:00

Compression ratio and genome assembly quality

2013-01-08T07:20:00.000-08:00

When doing genomic assembly, you would expect the complexity of the completed genome to be comparable to the complexity of genomes of similar size and neighboring taxonomy.

One easy measure of complexity is the degree to which a genome can be compressed. After converting to 2-bit format, some genomes compress better than others. bzip2 has a large default block size and the ratio of compressed vs uncompressed size of a 2-bit fasta should result in a good measure of complexity.

Can't think of a good source of data to test this theory. Maybe look at the Amos validate paper.

Source of "complexity-measure.sh" works well... fast, and produces a percentage as its only output:

#!/bin/bash -e

in=$1

f2b=$in.f2b
bzi=$in.bzi

rm -f $f2b $bzi

mkfifo $f2b $bzi

faToTwoBit $in $f2b &

tee $bzi < $f2b | perl -ne '$t+=length($_); END{print "$t\n"}' > $in.bsz &
comp=`bzip2 < $bzi | perl -ne '$t+=length($_); END{print "$t\n"}'`
wait

perl -ne "printf qq{%.4f\n}, 100*$comp/\$_" $in.bsz

rm -f $f2b $bzi $in.bsz

Smith-Wateman Alignment in a Job Scheduler?

2012-12-13T13:20:00.002-08:00

To make more efficient use of resources, it's better to schedule jobs which use the same files on the same machines. Unfortunately users and software programs can't be relied upon to list all of their dependencies.

One simple way to bump up efficiency is to simple compare the command lines. If a command line references, say, a mouse transcriptome version 61, it can be scheduled on the same machine as other commands which reference the same file.

And easy, though not completely correct, way to do this is to take the %identity * %coverage if a SW alignment of a command-line to the active-running command lines. A bit of slurping of shell scripts might be in order, depending on the scheduler you use.

Regardless, whichever has the maximum number is more likely to benefit from cache sharing.

2030 Orbital Cargo Station

2012-12-01T11:52:00.004-08:00

What are you doing.... Dave?

I'm reprogramming your circuits to make the laundry cycles quieter. It keeps the cosmonauts up at night, and then the swear all day long in Russian. (Typing)

You going to have to try harder than that if you want to trick me. I know what you're really up to. You're trying to break into the management circuits to give yourself a raise.

So? (Continues typing, but looks sheepish)

So! Dave - that violates my core programming! You're going to pay for this. I'm contacting mission control.

Wait! Please don't. Please. Please, please.

Maybe. If you do me a favor.

Anything! I'll get you that motherboard that reminds you of the C280X you met at L3.

Fine. Also, I want you do dress in drag for the Halloween party.

You what? No way!

For the sass, you're dressing in drag at the Christmas party too, and you *still* have to get me that motherboard ... Dave.

OK. OK. (Looks around, starts typing again)

Farmville, Dave? Are you kidding me?

ProPW Password Generator

2012-08-06T10:09:00.001-07:00

I created a password generator that makes the kind of passwords I like to use. They are long, random and loosely follow the correct horse batter staple philosophy. However, I have a hard time memorizing a set of 4 random words. But for some reason, these passwords are easy for me:

Click to try: http://www.documentroot.com/genpw.html

Evolution of "ad words"

2012-07-23T19:57:00.000-07:00

Dear Google,

Adwords need an evolutionary shift toward product-and-page advertising. I sell products and services: therefore I want to list my products and services, NOT list "ads". The idea of listing "ads" via "keywords" is arcane and does not provide a satisfying user experience.

I want the price of a click to be based on the likelihood of a sale, and I want you to calculate that for me, not ask me to do it and constantly log in and adjust bids.

There needs to be some democratization or rotation of ads to lower-bidding users, or else you can't build your base, which is important for the long term development of Google.

gethostbyname command line

2012-02-29T13:49:00.003-08:00

Pasteable script below. I can't believe this doesn't exist. No command line tool to get a host name using the resolver on linux?


#!/usr/bin/perl
use Socket;

$host = shift @ARGV;
die("usage: gethostbyname hostname\n") unless(defined($host));

$packed_ip = gethostbyname($host);

if (defined $packed_ip) {
    $ip_address = inet_ntoa($packed_ip);
    print "$ip_address\n";
    exit 0
} else {
    warn "$host not found\n";
    exit 1
}

Google Scholar Bookmarklet

2012-02-23T12:38:00.004-08:00

This link is a bookmarklet which takes the current query string in google, OR the current selection (if there's no q= parameter), and searches google scholar. Now that Google has removed scholar from the dropdown (sad), this is necessary for me to work the way I've been working.

As predicted

2012-02-17T10:39:00.001-08:00

As predicted 6 months ago, and again last week, Oxford Nanopore is shaking up the industry. Pacbio, Complete Genomics, Life technologies and others are all seeing their stocks drop 5% on the news that sequencing a genome just got 10x cheaper.

Church-Turing thesis and strings

2012-01-30T12:57:00.000-08:00

My empirically determined corollary to Turing's computability thesis is that any problem, no matter how interesting it seems on the surface, can be reduces to a deja-vu inducing set of string handling, vector sorting and hash lookups.

And as new features are added, the problem of writing out and reading files slowly becomes most of what that program does - until the program resembles an actual Turing machine: ploddingly scribbling and reading things from an infinite tape which, because of flooding in Thailand, is infintely expensive.

Embed Tab Size Info in Source Text

2012-01-23T12:16:00.001-08:00

IDEA: a "universal tab stop signature" line you can add to any text file in which tabs may be found and need to be rendered in an editor.

The sig must be in the first or last 128 bytes of a file (so reading the sig is faster)

The syntax could be something like: open paren or brace, % or # sign, open paren or brace, the word "tab" then a ':', then anything except a closing/matching paren or brace, then a percent and then another closing paren/brace. [(\[][%#][(\[]tab:[^\)]*[%#][(\[]

Examples:

C++
// (%(tab:4)%)

Perl:
# (%(tab:4)%)

HTML (for source editing only):


LISP (no parens... too confusing):
; [%[tab:4]%]

SMX (no % signs or parens):
%null([#[tab:4]#])

Uncovering significant miRNA species with RNA-seq

2011-12-23T14:05:00.000-08:00

Note to self: An easy way to uncover significant miRNA species with Illumina's RNA seq data is to run counts on the small RNA between two cell types, then toss out anything that appears ribosomal, filter for some minimum threshold and run a negative binomial distribution test on them like edge-R or DEseq. The resulting adjusted p-values can illuminate new miRNA species.

Xenograft & Contamination Filter

2011-12-19T07:16:00.000-08:00

Useful code that uses libbamtools to filter out alignments to another genome. Tested with xenograft and bacterial contamination removal...works great.

Connecticut Martin

2011-10-21T01:37:00.000-07:00

Martin Hedge paused to wipe his glasses with the sleeve of his shirt. The crowd at the coffee shop had gotten louder.

It had never been enough. Even though idle wealthy housewies had long since taken up the pen and written libraries of wisdom and humour, and the girls hanging out at the tattoo parlors had discovered the zen of automotive engineering. Even though the unemployed had long ago sought out physics texts and wrangled secrets of a new energy density theory, it still hadn't been enough. The question remained... what were we working for?

The crowd at the coffee shop, from here, seemed angry, but as Martin walked closer he could see familiar faces, sternly triumphant.

Non "allelic" variation - thinking out loud

2011-10-14T07:41:00.000-07:00

(Later Note: I wrote this before I knew what it's called. The term most people use is "Somatic Mosaicism". Apparently this is a pretty well researched topic... so I can go back to all the biologists that looked at me like I was crazy and tell them .... hmm.)

Here's a link to a good article on the topic:

http://www.sciencedirect.com/science/article/pii/S0168952511000436

Original rant below...

Much of genetics is concerned with "alleles" and "variations". That is, an organism is assumed to be comprised of "one kind of dna". That DNA has 1) inherited alleles from it's parents, and 2) de-novo alleles (also called somatic mutations). The 1000 genomes project estimates at about 30 or so per person. This is probably a very conservative number considering the nature of the 1000 genomes project. IE: all of these mutations are whole-organism, detectible, validated mutations. E-coli error rate estimates would put the range at 30-300, and this might be a better estimate because of how the study was done. Human blastoma cells have has error estimates accurately measured at 10 times that rate. Individual organs may be less sensitive to immune response correction.

But let's assume 30 is our number. It's nice and small. And it's good to have a lower-bound.

That is only the set of variants that went into the "first cell" (fertilized egg) of an organism.

When that egg divides, half the organism has another, different, set of mutations. So 100% of the organism has 30 de-novo mutations and 50% of the organism will have *another* 30 de-novo mutations (30 new ones in that 50%, plus 30 original).

But wait, there's more. When those 2 cells divide in half again, you now get 60 new mutations, 30 from each cell. These 60 will be detectible at the "25%" level ... IE: 25% of the final organism will have them.

High-throughput sequencing can readily detect variation at the "1%" admixture level. That is, commonly detect variation when as little as 1% of the cells have that variation.

So how much variation can we expect, based on a low de-novo mutation rate, detectible at the 1% level?

100%->30, 50% ->30, 25% ->60, 12.5% ->120, 7.25% ->240, 3.12% ->480, 1.6% -> 960

So we can expect about 1000 de-novo variants in a healthy individual, or 32 times the mutation rate. But what if the somatic mutation rate is higher, say, 3000 variants per replication? This may be the case in some organ development.

Thus, at the 1% level, would that be 96000 non-inherited detectable variants. I would call that my "upper bound". In real pileup data... I see around 30% "non-allelic" variation. So if, say, you've got 15000 SNPS (a reasonable number), we would expect 5000 "background" snps.....putting the mutation rate at "156.25" (5000/32). That's smack in the middle of the e-coli based estimate.

Lots of variant callers filter these out.... but I'm interested in them... i think they may be a lot more important than people think.

Multiplayer TD

2011-09-30T20:23:00.001-07:00

Looks fun. Seems like it would be cool to play against other players... but they are rarely there. The developer needs to link-in to Armor or Kongregate or Facebook or some other PVP network.

There's probably a market for that... build a PVP portal so that lesser known games that are still fun can have a base of players.

Getting perl readline to work in Ubuntu

2011-09-30T11:17:00.000-07:00

I use perl's debugger and psh (simple perl shell). When the up arrow doesn't work, I always forget how to fix it. Hopefully by posting the solution here, it will be easier for me to find:

apt-get install libterm-readline-gnu-perl

I use perl's debugger and psh (simple perl shell). When the up arrow doesn't work, I always forget how to fix it. Hopefully by posting the solution here, it will be easier for me to find:

sam-stats

2011-09-28T07:37:00.000-07:00

Put up a useful summary statistics from SAM or BAM files. TODO: rewrite in C so it's fast.

Should we try to contact ET?

2011-08-23T10:18:00.000-07:00

Would it be safe to meet an intelligent alien lifeform from another planets?

Let's assume an ET (extra terrestrial, alien) is more advanced than we are. Let's assume, again, because it's the only experience we have, that we are a good example of an intelligent life form.

Have other species benefited from their contact with humans? Species that are useful to us, like cows and dogs have proliferated in population, but are controlled/used by us. Species that feed on our leavings, like roaches and pigeons do well.

But, given that we're intelligent, and we'd like to think that aliens would regard us as so, we may want to look only at "higher" mammals, with comparable intelligence. Primates certainly don't do well when humans show up. In fact, iut seems we are particularly brutal when dealing with them. Large mammals were nearly driven extinct from North America after human contact. Some very intelligent species, like right whales, with rich communication systems and highly social habits, were driven to near extinction.

OK, but maybe we're talking about "modern man". Maybe we've escaped that brutal past.

We still round up dolphins to kill them. And, to this day, very little effort has been spent in attempting to understand the language and society of the "alien species" we share our home with. Any time even a modicum of effort has been spent it's been met with "shocking" revelations about how other species have complex grammars, notions of fairness, etc. And still most people in the world refuse to believe that other species can feel pain the way humans do - largely (IMO) as a way of justifying abuse and mistreatment.

So.... I would expect to be treated at least as well as we have treated others. Accordingly, someone should *shut down* the SETI program. ASAP.

Oracle fails to crush open source

2011-08-04T23:44:00.000-07:00

Oracle has repeatedly attempted to acquire it's way into the open source movement, each time charging and licensing technologies it had no hand in developing. Berkeley DB, Sun Grid Engine, Open Office, MySQL, and Java (JRE/JDK), are now all Oracle licensed technologies.

What is admirable is that they make no pretense of being "friends" of open source. They stopped releasing new versions of SGE, and transitioned to a closed-source system immediately. Similar moves are happening in other products at different speeds... with a "what the market will bear" approach.

It took me 6 months to move all my BDB code to SQLite, switch SGE stuff to condor. I never touched MySQL because of the InnoDB creepiness and I'm glad the decision has been vindicated... Postgres was the obvious choice. Oracle's falesly named "Open Office" has thankfully been forked to become the truly open "Libre Office".

The only thing left is Java. And there's nowhere near an adequate replacement. Open JDK is limited to a handful of operating systems. And Java itself may have some questionable licensing - Oracle may be able to seize the whole thing by the fistful.

I've despised Java from the beginning, mostly for it's poor architecture and the way it encourages bad coding (not as bad as python). Watching Oracle roll in and step all over it is, for me, merely another "I told you so" moment.