Bracing against the wind

Church-Turing thesis and strings

2012-01-30T12:57:00.000-08:00

My empirically determined corollary to Turing's computability thesis is that any problem, no matter how interesting it seems on the surface, can be reduces to a deja-vu inducing set of string handling, vector sorting and hash lookups.

And as new features are added, the problem of writing out and reading files slowly becomes most of what that program does - until the program resembles an actual Turing machine: ploddingly scribbling and reading things from an infinite tape which, because of flooding in Thailand, is infintely expensive.

Embed Tab Size Info in Source Text

2012-01-23T12:16:00.001-08:00

IDEA: a "universal tab stop signature" line you can add to any text file in which tabs may be found and need to be rendered in an editor.

The sig must be in the first or last 128 bytes of a file (so reading the sig is faster)

The syntax could be something like: open paren or brace, % or # sign, open paren or brace, the word "tab" then a ':', then anything except a closing/matching paren or brace, then a percent and then another closing paren/brace. [(\[][%#][(\[]tab:[^\)]*[%#][(\[]

Examples:

C++
// (%(tab:4)%)

Perl:
# (%(tab:4)%)

HTML (for source editing only):


LISP (no parens... too confusing):
; [%[tab:4]%]

SMX (no % signs or parens):
%null([#[tab:4]#])

Uncovering significant miRNA species with RNA-seq

2011-12-23T14:05:00.000-08:00

Note to self: An easy way to uncover significant miRNA species with Illumina's RNA seq data is to run counts on the small RNA between two cell types, then toss out anything that appears ribosomal, filter for some minimum threshold and run a negative binomial distribution test on them like edge-R or DEseq. The resulting adjusted p-values can illuminate new miRNA species.

Xenograft & Contamination Filter

2011-12-19T07:16:00.000-08:00

Useful code that uses libbamtools to filter out alignments to another genome. Tested with xenograft and bacterial contamination removal...works great.

Connecticut Martin

2011-10-21T01:37:00.000-07:00

Martin Hedge paused to wipe his glasses with the sleeve of his shirt. The crowd at the coffee shop had gotten louder.

It had never been enough. Even though idle wealthy housewies had long since taken up the pen and written libraries of wisdom and humour, and the girls hanging out at the tattoo parlors had discovered the zen of automotive engineering. Even though the unemployed had long ago sought out physics texts and wrangled secrets of a new energy density theory, it still hadn't been enough. The question remained... what were we working for?

The crowd at the coffee shop, from here, seemed angry, but as Martin walked closer he could see familiar faces, sternly triumphant.

Non "allelic" variation - thinking out loud

2011-10-14T07:41:00.000-07:00

(Later Note: I wrote this before I knew what it's called. The term most people use is "Somatic Mosaicism". Apparently this is a pretty well researched topic... so I can go back to all the biologists that looked at me like I was crazy and tell them .... hmm.)

Here's a link to a good article on the topic:

http://www.sciencedirect.com/science/article/pii/S0168952511000436

Original rant below...

Much of genetics is concerned with "alleles" and "variations". That is, an organism is assumed to be comprised of "one kind of dna". That DNA has 1) inherited alleles from it's parents, and 2) de-novo alleles (also called somatic mutations). The 1000 genomes project estimates at about 30 or so per person. This is probably a very conservative number considering the nature of the 1000 genomes project. IE: all of these mutations are whole-organism, detectible, validated mutations. E-coli error rate estimates would put the range at 30-300, and this might be a better estimate because of how the study was done. Human blastoma cells have has error estimates accurately measured at 10 times that rate. Individual organs may be less sensitive to immune response correction.

But let's assume 30 is our number. It's nice and small. And it's good to have a lower-bound.

That is only the set of variants that went into the "first cell" (fertilized egg) of an organism.

When that egg divides, half the organism has another, different, set of mutations. So 100% of the organism has 30 de-novo mutations and 50% of the organism will have *another* 30 de-novo mutations (30 new ones in that 50%, plus 30 original).

But wait, there's more. When those 2 cells divide in half again, you now get 60 new mutations, 30 from each cell. These 60 will be detectible at the "25%" level ... IE: 25% of the final organism will have them.

High-throughput sequencing can readily detect variation at the "1%" admixture level. That is, commonly detect variation when as little as 1% of the cells have that variation.

So how much variation can we expect, based on a low de-novo mutation rate, detectible at the 1% level?

100%->30, 50% ->30, 25% ->60, 12.5% ->120, 7.25% ->240, 3.12% ->480, 1.6% -> 960

So we can expect about 1000 de-novo variants in a healthy individual, or 32 times the mutation rate. But what if the somatic mutation rate is higher, say, 3000 variants per replication? This may be the case in some organ development.

Thus, at the 1% level, would that be 96000 non-inherited detectable variants. I would call that my "upper bound". In real pileup data... I see around 30% "non-allelic" variation. So if, say, you've got 15000 SNPS (a reasonable number), we would expect 5000 "background" snps.....putting the mutation rate at "156.25" (5000/32). That's smack in the middle of the e-coli based estimate.

Lots of variant callers filter these out.... but I'm interested in them... i think they may be a lot more important than people think.

Multiplayer TD

2011-09-30T20:23:00.001-07:00

Looks fun. Seems like it would be cool to play against other players... but they are rarely there. The developer needs to link-in to Armor or Kongregate or Facebook or some other PVP network.

There's probably a market for that... build a PVP portal so that lesser known games that are still fun can have a base of players.

Getting perl readline to work in Ubuntu

2011-09-30T11:17:00.000-07:00

I use perl's debugger and psh (simple perl shell). When the up arrow doesn't work, I always forget how to fix it. Hopefully by posting the solution here, it will be easier for me to find:

apt-get install libterm-readline-gnu-perl

I use perl's debugger and psh (simple perl shell). When the up arrow doesn't work, I always forget how to fix it. Hopefully by posting the solution here, it will be easier for me to find:

sam-stats

2011-09-28T07:37:00.000-07:00

Put up a useful summary statistics from SAM or BAM files. TODO: rewrite in C so it's fast.

Should we try to contact ET?

2011-08-23T10:18:00.000-07:00

Would it be safe to meet an intelligent alien lifeform from another planets?

Let's assume an ET (extra terrestrial, alien) is more advanced than we are. Let's assume, again, because it's the only experience we have, that we are a good example of an intelligent life form.

Have other species benefited from their contact with humans? Species that are useful to us, like cows and dogs have proliferated in population, but are controlled/used by us. Species that feed on our leavings, like roaches and pigeons do well.

But, given that we're intelligent, and we'd like to think that aliens would regard us as so, we may want to look only at "higher" mammals, with comparable intelligence. Primates certainly don't do well when humans show up. In fact, iut seems we are particularly brutal when dealing with them. Large mammals were nearly driven extinct from North America after human contact. Some very intelligent species, like right whales, with rich communication systems and highly social habits, were driven to near extinction.

OK, but maybe we're talking about "modern man". Maybe we've escaped that brutal past.

We still round up dolphins to kill them. And, to this day, very little effort has been spent in attempting to understand the language and society of the "alien species" we share our home with. Any time even a modicum of effort has been spent it's been met with "shocking" revelations about how other species have complex grammars, notions of fairness, etc. And still most people in the world refuse to believe that other species can feel pain the way humans do - largely (IMO) as a way of justifying abuse and mistreatment.

So.... I would expect to be treated at least as well as we have treated others. Accordingly, someone should *shut down* the SETI program. ASAP.

Oracle fails to crush open source

2011-08-04T23:44:00.000-07:00

Oracle has repeatedly attempted to acquire it's way into the open source movement, each time charging and licensing technologies it had no hand in developing. Berkeley DB, Sun Grid Engine, Open Office, MySQL, and Java (JRE/JDK), are now all Oracle licensed technologies.

What is admirable is that they make no pretense of being "friends" of open source. They stopped releasing new versions of SGE, and transitioned to a closed-source system immediately. Similar moves are happening in other products at different speeds... with a "what the market will bear" approach.

It took me 6 months to move all my BDB code to SQLite, switch SGE stuff to condor. I never touched MySQL because of the InnoDB creepiness and I'm glad the decision has been vindicated... Postgres was the obvious choice. Oracle's falesly named "Open Office" has thankfully been forked to become the truly open "Libre Office".

The only thing left is Java. And there's nowhere near an adequate replacement. Open JDK is limited to a handful of operating systems. And Java itself may have some questionable licensing - Oracle may be able to seize the whole thing by the fistful.

I've despised Java from the beginning, mostly for it's poor architecture and the way it encourages bad coding (not as bad as python). Watching Oracle roll in and step all over it is, for me, merely another "I told you so" moment.

View the internet as if you are Google

2011-07-29T08:36:00.000-07:00

Many companies present a different face to search engines than they do to users. It's sometimes interesting to see the internet as if you are a search engine:

To change the Chrome user agent string, add the –user-agent option switch to the chrome.exe program shortcut:

chrome.exe --user-agent=" Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Correlation Heat Map

2011-06-21T20:44:00.001-07:00

Illumina Quality Score 'B'

2011-06-10T12:12:00.001-07:00

When analyzing about 100 PhiX control lanes, by aligning to PhiX and calling any reference mismatch an "error", the quality score 'B' performs higher than 'C' on average. The 'B' score is approximately abot 3 phred-scores lower than the mean score for the whole run. I think the 'B' score represents the equivalent of "I don't know" on the Phred scale. For programs that rely on quality scores to indicate actual likelihoods of error, the 'B' score should be treated as a special case.

This is in addition to the known issue that reported phred scores have higher ovserved error rates overall, especially at the top end of the scale (Q-40's perform like Q-37's should, etc.)

Lightweight Job Queuing Engine and GitHub

2011-05-14T20:08:00.000-07:00

I wrote a job queuing daemon. It sits on a head node and any number of compute nodes. Good with EC2. Replacement for condor, OGE (formerly SGE), torque, etc. So far, it's good about crash recovery, very good about latency. Other job queuing systems seem to be sooo slow with latency, you never want to use them. And they all spend so much time worrying about copying i/o files all over the place, when all I ever use is NFS anyway.

https://github.com/earonesty/grun

Written because those other systems are administratively difficult, and have more features than I need. Some day I may choose to go back to condor, or SGE, but for now, it's nice to have a smaller, faster, easier one.

I'm trying github instead of google code. The developers and gangla got me to try it out. I think I might like it. Not sure why it makes you "add" and "commit" separately.

I based the socket system off of ppcgid, since it worked so well in the past.

Dealing with FASTQ's

2011-04-28T02:59:00.000-07:00

I posted some code for dealing with FASTq's. They're all written in C++, and each one replaces a tool that's between 3 and 50 times slower. The scoring metric they use for "anchored sequence alignment" has been improved. Also, they contain lots of "do what I mean" features, allowing the user to focus on the results, and not on the mechanics of processing sequencing data.

The essential tasks they accomplish are sequence quality filtering, adapter/primer removal, demultiplexing and paired-end joining (stitching):

They have to be compiled, but they are each "monolithic" ... and require no libraries, headers, etc. I tested fastq-join on Linux and Windows, so I know it compiles easily. The other 2 tools have only been tested on Linux.

See: https://code.google.com/p/ea-utils/

New aligner

2011-03-15T10:02:00.000-07:00

I'm working on a new aligner i have that doesn't use BTREE. It's fun. Not hard, and finally working.

F-Test for Comparing Variability

2011-03-06T02:41:00.001-08:00

I left my textbook at home, and had to search for a while to find a decent description of how to use an F-test (even though it's somewhat controversial), so I'm posting it for future reference. Wikipedia stuff on statistics appears to be written by people who forget it's supposed to be human-readable.

GATCGGAAGAGCGGTTCAGCAGGAATG

2011-02-23T11:38:00.000-08:00

I knew it was some kind of primer, adapter or barcode - but I wasn't sure which, whether to trim, or if the barcode came before or after.

BLAS*T searches turned up nothing, but Joel answered, within a second, "it's an adapter sequence". He said "I just Googled it".

Reminder to self... Google searching DNA sequences works (NOTE: Bing didn't).

Approximate Line Count for Very Large Files

2011-02-10T02:06:00.000-08:00

When dealing with very large files, the unix tool "wc" can be extremely slow. The alternative, byte size, is often not what I want to look at, especially when trying to estimate the number of reads in a fastq file.

A good estimate (2 sig figs) is, 90% of the time, what I need.

alc is my "approximate" line count tool. It counts the number of lines in a file, just like wc, except it only "samples" the file in a series of segments. By seeking and reading 200K from a dozen places in the file, rather than reading the whole thing, I get a good representative sample, and an accurate-enough count.

A True Teflon Alternative

2011-02-04T18:00:00.000-08:00

We went looking for a nonstick solution that wasn't coated in Teflon or PFTE, or whatever people try to sell it as. Apparently it causes developmental disorders in laboratory animals (click for EPA advisory). I can only wonder what it does to human kids.

The first thing we found was Swiss Diamond... which is a complete scam. Glad to have avoided that before making an expensive mistake, I scoured reviews and sites and we found the Aeternum and Green Earth line of ceramic coated pans (like porcelain, it seems).

I bought the Aeternum and it works exactly as well as advertised. Nonstick, and not made with creepy chemicals. If it lasts or breaks, I'll report it here.

Uncuddled Else

2010-12-19T11:39:00.000-08:00

I don't care how cute the name is. It wastes precious vertical space while adding nothing to readability. Worse yet, the uncuddled else substracts from the implication that it is a part of the prior "if" statement... which it is.

Migrate from Flickr to Picasa

2010-12-09T22:20:00.000-08:00

There are other migration tools out there but nothing works as well as Nathan Van Geem's script... preserves your organiation, titles, descriptions... and moves it all to picasa in one clean step.

Of course... you'll need to have python installed on your machine.

But the advantage is that it uses almost no space... downloading and uploading at the same time, so it doesn't chew up your drive with "yet another copy" of those pictures.

I thought of giving people access to it as a service... but really, his instructions aren't too hard

HOWTO: Condor on an Intranet

2010-12-08T12:37:00.000-08:00

Here's what I had to do to make it work. So far it's fine.

ADMIN:

1. If you install Condor as an RPM, it will default to running everything as the user "condor". This will prevent the condor machinery from smartly forking jobs as "the user who submitted them". Especially, if you're on a private network, this makes no sense though, so open it up with CONDOR_UIDS=0.0

2. Lots of machines mounted to the same share? This one was useful:

LOCAL_CONFIG_FILE = /mnt/blah/condor/$(HOSTNAME).local

Now I can keep configs for each machine, without having to worry about condor_config -set. I use symbolic links to create "machine classes" that I add machines to when I bring up new nodes.

3. condor_config_val -set ... seems nice, but don't bother. You really need to keep your configs well groomed. I wasted time getting it to work... and then deleted everything I set. Must be nice on heterogeneous public networks where the config file isn't on some shared mount. (In fact I just disabled it).

4. Trust everyone... let the firewall protect you. TRUST_UID_DOMAIN = True. This is good for getting started, so you're not bumping into roadblocks all the time. Afterwards, tinker with password auth. Kerberos and the like are a bit much, basic password encryption is fine, IMO, and will work everywhere.

5. GET RID of stuff that the RPM stuffs in your local config. Things like "Suspend=False" and "Start=True" are in there... and they are bad. Since your machines don't have a keyboard, ditch the Keyboard stuff and go with "START=$CPUIdle". Since you don't want your jobs to get randomly killed, go with "SUSPEND=$CPUBusy". (You can just delete the stuff in the local config and go with the UWCS defaults... they are really good enough for most things.). Your global config should be the SAME on all boxes. Your local config should be minimal... just the stuff for that machine.

USING

1. condor_run has to be run *from a shared directory* or it won't work. It says that in the docs, but I didn't read it.

2. Likewise, for vanilla jobs, make sure the paths to all your files are the same on every machine you run on. Trying to sort it out in a script is a mess. Fix paths first. Then run.

Better than Textpad

2010-12-08T11:42:00.000-08:00

For the longest time, I used textbad because of teh block-editing, regex, and extremely large file support.

Notepad++ wasn't even a contender years ago, but it has cought up... and passed with flying colors:

http://notepad-plus-plus.org/

Switched all my scripts over today. Not looking back.