The band was formed in 1996 when Dave McDonnell (credited as The Diminisher), Jeremy Barnes (as "Marta Tennae"), and Griffin Rodriguez ("Blue Hawaii") teamed up to write and improvise music. Their almost-entirely-improvised album In A Different City was recorded in Chicago during this time, and later released in 1999. In a unique twist, the band had an audience paint vinyl LP covers while they performed, which were later collected as "painticons" and distributed as the actual album covers for the LP version of the album.
The Orange Tapered Moon EP followed in 2000, recorded during a late night marathon session at WFMU. The LP covers were again created in the same way the In a Different City covers were.
