Andrew Montalenti

This post is password protected. To view it please enter your password below:

Password:

I recently watched Oliver Stone’s Wall Street again. It really is amazing how relevant this movie is in 2011, ~25 years after its original release in 1987.

This speech, in particular, is a knockout, given the recent Occupy Wall Street movement:

Bud: How much is enough, Gordon? When does it all end, huh? How many yachts can you water-ski behind? How much is enough, huh?
Gekko: It’s not a question of enough, pal. It’s a Zero Sum game – somebody wins, somebody loses. Money itself isn’t lost or made, it’s simply transferred – from one perception to another. Like magic. This painting here? I bought it ten years ago for sixty thousand dollars. I could sell it today for six hundred. The illusion has become real, and the more real it becomes, the more desperately they want it. Capitalism at its finest.
Bud: How much is enough, Gordon?
Gekko: The richest one percent of this country owns half our country’s wealth, five trillion dollars. One third of that comes from hard work, two thirds comes from inheritance, interest on interest accumulating to widows and idiot sons – and what I do, stock and real estate speculation. It’s bullshit. You got ninety percent of the American public out there with little or no net worth. I create nothing. I own. We make the rules, pal. The news, war, peace, famine, upheaval, the price per paper clip. We pick that rabbit out of the hat while everybody sits out there wondering how the hell we did it. Now, you’re not naive enough to think we’re living in a democracy, are you, buddy? It’s the free market. And you’re a part of it.

Watch the full speech on YouTube here.

On Fred Wilson’s blog post about Raise Cache and HackNY, someone asked a very legitimate question:

Why are we raising money to benefit CS students from top programs around the country? Why are we raising money to help companies like Business Insider and bit.ly hire interns?

The event looks like fun but I’ve been trying to understand hackNY and don’t understand why it’s a charitable cause.

And, I wrote a response — providing some anecdotal information about how Parse.ly benefited from HackNY, and why it matters in a city flush with gold-plated Wall St. internships.

Let me give you a hint why HackNY is somewhat charitable: Wall Street firms pay between $15-$30k/summer for technical interns in NYC. Most startups — especially early-stage ones — simply can’t compete with that.

You may not think startups like bit.ly and Business Insider need any help (and with their $13-15M in capital funding raised, maybe you are right), but in 2010, when Parse.ly sought a summer intern, we had no funding and a

That said, it'd probably be good to have more truly early-stage (pre-funding) companies on the roster -- one of the problems with this, though, is that a lot of pre-funding companies are in such a fragile state that the internship might be over a couple weeks into the summer. Also, pre-funding companies aren't typically "buzzworthy". The first two years of HackNY has been partially about creating some buzz about NYC tech among university students, something at which it has succeeded spectacularly.

Both of our HackNY interns (2010 + 2011) have commented about how one of the interesting parts of the program is that it simply raises awareness of the different stages of companies. Since the HackNY interns all live together in university housing, they share stories -- and you'll have folks working on >100-person teams at places like Gilt Group and folks working on <10-person teams at places like Parse.ly. Also, the program’s lecture series does a good job of encouraging students to consider entrepreneurship or startup work as a post-graduation option. This is a countervailing force to the professional HR/recruiting teams employed by Wall Street and other Fortune 500s to market their positions to top students.

from Raise Cache at AVC.com.

I came across this wonderful piece of historical retelling by David Beazley, one of my favorite Pythonistas and the author of Python Essential Reference. Here is a man who conquered C++ in just about every way, but ultimately found himself trapped in its byzantine complexity, only to escape by way of Python.

Swig grew a fully compatible C++ preprocessor that fully supported macros. A complete C++ type system was implemented including support for namespaces, templates, and even such things as template partial specialization. Swig evolved into a multi-pass compiler that was doing all sorts of global analysis of the interface. Just to give you an idea, Swig would do things such as automatically detect/wrap C++ smart pointers.It could wrap overloaded C++ methods/function. Also, if you had a C++ class with virtual methods, it would only make one Python wrapper function and then reuse across all wrapped subclasses.

Under the covers of all of this, the implementation basically evolved into a sophisticated macro preprocessor coupled with a pattern matching engine built on top of the C++ type system [...] This whole pattern matching approach had a huge power if you knew what you were doing [...]

In hindsight however, I think the complexity of Swig has exceeded anyone’s ability to fully understand it (including my own). For example, to even make sense of what’s happening, you have to have a pretty solid grasp of the C/C++ type system (easier said than done). Couple that with all sorts of crazy pattern matching, low-level code fragments, and a ton of macro definitions, your head will literally explode if you try to figure out what’s happening. So far as I know, recent versions of Swig have even combined all of this type-pattern matching with regular expressions. I can’t even fathom it.

Sadly, my involvement was Swig was an unfortunate casualty of my academic career biting the dust. By 2005, I was so burned out of working on it and so sick of what I was doing, I quite literally put all of my computer stuff aside to go play in a band for a few years. After a few years, I came back to programming (obviously), but not to keep working on the same stuff. In particularly, I will die quite happy if I never have to look at another line of C++ code ever again. No, I would much rather fling my toddlers, ride my bike, play piano, or do just about anything than ever do that again.

From this Swig mailing list entry.

It’s hard to find me gushing more unapologetically than when I talk about the virtues of my favorite programming language, Python.

Indeed, my life for the last 3 years has been dominated by the language. In many ways, pursuing a startup and enduring the associated financial hardship was partially because I had become frustrated with using Java in my full-time work and wanted to convert hobby projects I was building outside of work hours into full-fledged projects.

Something else I noticed in the last three years is that my programming life has become very zen-like. I now rarely discuss or debate things like programming language features, strange constructs like generics, or which framework to use or ignore. Instead, I spend most of my time building a product that people love. My colleagues and I communicate with code. And what better language to communicate in than arguably the world’s most readable? What better language to deliver value in than one that simply gets out of your way?

I therefore get a great amount of joy of showing other people what the Zen of Python can mean in their lives. Last year, I gave a training course to a 20-person team of government employees who were using dated languages like Fortran and COBOL to build important government systems. A bright manager in the organization realized how much more productive their team could be if they stopped worrying about compiler versions and IDEs and started thinking in code. But the key was to understand the value of Python, not necessarily as a language with a certain set of features, but as a way of doing things, as a cultural influence. This is a culture that says, “the language should fade away”, similarly to how Edward Tufte argues that the chrome and administrative debris should fade away when displaying content.

Since then, I have used these slides countless times to espouse the virtues of my favorite interpreter. This has included giving short seminars in NYC, training Parse.ly interns and new hires (here’s a shot of me and two Parse.ly engineers, Michael and Zach), holding Python office hours at HackNY’s hackathon (indeed, here’s an action shot on Flickr to prove it!), and introducing my friends to the joy of building software.

Code and Slides: The Zen of Python in 3 days

It’s therefore with elation that today I am able to provide the slides and materials for this talk free to the world on my Github account. Simply click here:

https://github.com/Parsely/python-adv-slides

Or, if you’re not interested in the code behind the slides (because the slides are actually created with the help of Python itself), you can go here to simply view them.

http://pixelmonkey.org/pub/python-training/

(note, modern web browser like Chrome or Firefox recommended for speed and smoothness)

This may be my first “open source presentation”, in that the slide presentation itself is provided as code that is freely available, and reproducing the slide presentation is a matter of running a code generator.

A touch of NLP with Python

I mentioned that one of the reasons I love Python is because it gets out of your way. Nowhere is this more evident than Python’s ability to prototype algorithms to natural language processing and corpus linguistics problems, something I do everyday. The Natural Language Toolkit (NLTK) provides an excellent starting point from learning about this aspect of computer science.

Recently, Parse.ly has gotten some new engineers on the team who are great Python programmers but without any background in NLP. To give them a taste of how well-suited Python is for this task, I gave a seminar entitled, “Just Enough NLP with Python”. Just as with the prior slides, you can view these online on Github:

https://github.com/Parsely/python-nlp-slides

And you can view the compiled slides here:

http://pixelmonkey.org/pub/nlp-training/

Between these two pieces, you can learn the Zen of Python (PEP 20) very rapidly, and incorporate it into your everyday software life.

If you have any comments or feedback, feel free to reach out to me on Twitter. I’d love to hear what you think. And if you, too, are bitten by the Python bug (as one of our Parse.ly interns recently was in a big, almost romantic way) then you should reach out to us, since we’re hiring.

And, sadly, our top engineering graduates don’t always become engineers. They move into finance or management consulting — both of which pay far higher salaries than engineering. I have seen the dilemma that my engineering students at at Duke University have faced. Do they take a job in civil engineering that pays $70,000, or join big Wall Street financial firm and make $120,000? With the hefty student loans that hang over their heads, most have made the financially sensible decision. In some years, half of our graduates have ended up taking jobs outside of engineering. Instead of developing new types of medical devices, renewable energy sources and ways to sustain the environment, my most brilliant students are designing new ways to help our investment banks engineer the financial system.

[...] We also need to make the engineering profession “cool” again, with the same sense of excitement and urgency in engineering and science that we saw during the Sputnik days. Back then, engineering was considered essential to the nation’s survival. Engineers and scientists were national heroes. It’s not that we don’t have problems to solve. The economy is in dire straits. Natural resources such as food, water, and crude oil are becoming scarce. Drug-resistant bacteria threaten us with doomsday plagues. But we’re not offering our best minds incentive to solve them.

From Mr. President, there is no engineer shortage.

Luckily this is happening already in high tech in NYC, thanks to awesome programs like HackNY and collabraCode (both of which my startup Parse.ly formally supported). As much as it pains me to say it, I also think The Social Network may be seen as a cultural catalyst for software engineers becoming “cool”.

But high tech is only a small piece of the puzzle — we need the same active marketing for students’ minds in biotech, education, medical research, civil engineering, etc.

Today, I turn 27. Even though I was deep in the middle of a project late last night, I peeled myself away from my monitors, went to sleep, and woke up late to enjoy a day of reading outside.

Parse.ly has an official “take your birthday off” policy, so I made sure to set a good example.

I remember when I was younger, I used to look forward to birthdays very eagerly. Birthdays were when I got a new videogame or programming book. Birthdays were about stuff, and taking the day to play with new toys.

Now, over a decade later, my birthday is much less about stuff. I don’t play videogames anymore, and I already know how to program. I am fortunate to live comfortably and don’t long for stuff any longer. My Nintendo Wii gathers dust (like everyone else’s, it seems). My computer is no longer used to amuse me, but to allow me to work on my passions — building software, building a company, staying informed, informing others. I have a seemingly endless queue of books I’d like to read, movies I’d like to watch, things I’d like to write, software I’d like to build. I’ve come to realize that birthdays, at my age, are more about time.

In my ruthlessly efficient worldview — where I regularly talk of cost-benefit analysis, backlog prioritization, and productivity — my birthday has become about taking a moment to flip my prioritized world on its head. Let’s not pick an item from the top of the prioritized backlog. Instead, let me take something from the backburner, for once. Let me behave — if only for a day — as if I had all the time in the world.

I don’t need stuff. I just need time. Of course, that’s the bittersweet part of one’s birthday. That even as you come to realize the importance of time, the day acts as a reminder of how our time on this earth is limited. 1 day passes, and only n-1 left to make a difference.

I’ve been quite busy with work lately, so haven’t had time to send a few posts toward my blog. However, I have been working on some spare time and work-related projects that I’d love to share with everyone here.

Among them:

• Lifehacking through standing desks. I have created a standing desk setup for my home office, and my investors in Parse.ly have created a healthy startup office in NYC. I have some thoughts about this as it applies engineering and hacking to the thing many of us do most: sit in a desk chair all day.
• 2-day Python Training Course. I have created a 2-day Python training course that helps existing programmers learn Python by comparing it very directly to languages they may already know, like C and Java. I gave this course to a group of government employees a few months ago. It has some interesting characteristics: I designed the whole course using ReStructured Text (ReST) and compiled it into a web-based presentation. This means that the entire course has the potential to be “open source” — exercises, slides, and all. I plan to release this to the public. For now, I am just clearing a few of the images I used in the course to make sure I don’t inadvertently infringe copyright. After that, I will open to the public.
• Groovy/JavaScript articles go public domain. Some articles I wrote for GroovyMag and JSMag last year are now able to be published in the public domain. These include one about RESTful services with Groovy, one that talks about functional programming with JavaScript, and finally one that discusses a design for “metagrids” in ExtJS 3.x. I will put all three articles up on this blog once they are reformatted.

Stay tuned!

I was a bona fide Java programmer for 5 years before I started working on Aleph Point and Parse.ly. I truly believe that Python and JavaScript are fundamentally better languages than Java for a variety of reasons born out of experience with each of them. (Note: Before this gets marked as flamebait, please notice that not only was I Java programmer for more than 5 years, but I was also a Java open source contributor!) I have enormous respect for the Java open source community, which has produced some of the highest quality modules available anywhere.

Now, don’t get me wrong — Python also has batteries included, and usually, when I think that I’m missing a great module I used to use in Java, it already exists in a much more powerful form in Python’s Standard Library or the wealth of modules on PyPI, GitHub, and Bitbucket. However, I believe in not reinventing the wheel, and so if a great open source tool exists in Java, I will want to interact with it.

One of these modules which we use extensively at Parse.ly is Apache Solr, and its surrounding Lucene project modules. Lucene is an extremely mature framework for document indexing, and Solr is a powerful server-ization of that technology that fits well into complex, mixed language distributed systems. I know there are efforts — like Whoosh — to build fast search engines atop the Python language. And I applaud these efforts — more projects means more competition, and more competition means better products. However, I still believe that you go with the best of breed tools available for production software, and you try not to let religious arguments about programming language get in the way.

Lately, I have come across more and more Java open source projects that have no equivalent in Python, and which I would like to access. Knowing that I wanted to feel comfortable incorporating Java open source projects — beyond Solr, which was already nicely wrapped as a web service — I, at first, thought that I’d be forced to still live among the weeds of complex class and interface definitions, cumbersome Java IDEs, XML configuration files, and (IMO) time-wasting rabbit holes like dependency injection, configuration management, and classpath hell. And then I found Groovy.

The most under-rated language ever

Groovy was released as one of the earliest dynamic languages written atop the JVM, back in 2003. I remember hearing about it back then and not “taking it seriously”, as we programmers often do with important, new technologies. I had the same ill feelings toward JavaScript until as late as 2007, but now consider myself more a JavaScript programmer than a Java one!

In truth, had I read James Strachan’s blog post discussing why he created Groovy, I would have probably paid more attention:

So I’ve been musing a little while if its time the Java platform had its own dynamic language designed from the ground up to work real nice with existing code; creating/extending objects normal Java can use and vice versa. Python/Jython’s a pretty good base – add the nice stuff from Ruby and maybe sprinkle on some AOP features and we could have a really Groovy new language for scripting Java objects, writing test cases and who knows, even doing real development in it.

I had thoughts like these back in my Java programming days, too. I would see features that were standard in other languages — concise list/map syntax, “bare” functions (that could live outside of class definitions), optional/dynamic typing, metaprogramming, a proper REPL, first-class functions — and think to myself, “Aren’t we missing a whole lot by lacking these features in Java?” The more I started to code Python and Java side-by-side, the more I started to realize that many “core” Java technologies were basically created to cope with inadequacies of the language itself.

For example, unit testing “frameworks” and formal IDE/debuggers are more popular in Java than other languages because of the lack of the REPL — dynamic language programmers tend to test their code from an interactive shell as a regular part of programming. Utility libraries like Apache Common Utils were created to cope with syntactic deficiencies around core types like lists and maps. And on and on.

Groovy is a “post-modern language” dynamic language. By that, I mean it is clearly inspired by and learns from the lessons of both Python and Ruby. It is also the world’s first programming language that was written to target an existing community of programmers. Let me explain. Python was not written “to target” C programmers or Visual Basic programmers. It was written as a new language altogether, not targeted at anyone in particular. However, Groovy was written “to target” Java programmers — it is the dynamic language Sun should have built for the JVM as dynamic languages gained popularity and the Java language stagnated.

Groovy is respectful of and cooperative with Java itself. One of its primary design goals is to live alongside existing Java code, even while Groovy’s syntax far surpasses that of Java. In this respect, Groovy plays a very similar role in the Java ecosystem that Python plays in the C ecosystem. However, unlike Python and C’s relationship, Groovy’s relationship with Java is bidirectional. What that means is that one can easily write Java libraries that execute Groovy code (provided the Groovy Language is on the classpath as a JAR), and Groovy libraries can easily execute Java code. In fact, you can even have complex relationships among these two languages, such as Groovy classes serving as base classes for Java classes and vice versa.

However, what’s perhaps most astonishing about Groovy is that it truly got a whole lot of things right. That is, the language simply is extremely well designed, and specifically greased for programmer productivity. It learned from the mistakes of existing dynamic languages, and made a dynamic language that is truly the “best of both worlds”. When I introduced the language to an engineer on my team at Parse.ly, he uttered some words that really stuck with me: “This language was clearly designed by very, very lazy programmers.”

So true. Typing is a chore — let’s do less of it to code without sacrificing readability. That’s the Groovy way.

Over the last several years, it has matured to the point where I feel extremely comfortable writing production code in Groovy, especially thanks to the surrounding ecosystem of the Grails project.

Python is still my muse, but Groovy is my Winston Wolf

A famous character in the movie Pulp Fiction is Harvey Keitel’s “Winston Wolf”. He’s the underworld problem solver — a guy who gets things done with a no-nonsense attitude.

Winston Wolf: If I’m curt with you, it’s because time is a factor. I think fast, I talk fast, and I need you two guys to act fast if you want to get out of this.

Python is a beautiful language and will continue to be my language of choice for production code. However, I simply can’t ignore the enormous wealth of production-quality Java code that has been written for which there is no good Python equivalent. I also can’t ignore the unbelievable engineering effort that went into making the JVM a powerful, scalable, and stable platform for production software. I thought I’d have to throw this community away (for myself and my companies) due to the deficiencies of the Java language. However, I have come to realize that Groovy has proven itself the worthy successor to Java. (Yes, successor — as in, you would be a fool to write your code directly in Java these days.) And personally, I will never work on a Java project again without Groovy. Java is a mess, but Groovy is my clean-up man.

I have also observed that the “ported” JVM languages have failed. This was a surprise to me as it was happening, but in retrospect, is not as surprising. Jython and JRuby are simply not viable long-term open source projects. The target community is only those programmers who know both Python and Java or both Ruby and Java. And there was no great impetus to truly “learn” those environments, because Python programmers felt that they already “knew” Jython, since the language was the same. Is the benefit of having access to Java libraries worth the downside of using a non-standard VM for my primary programming language? Usually not.

Now, while I wasn’t even paying attention — slowly learning Groovy in my spare time and using it for small projects dependent on Java — it seems that the world of independent dynamic JVM languages has taken off. Groovy has not had nearly as much popularity as languages like Scala (used at Twitter and Foursquare) or Clojure. And Groovy lacks “sex appeal”, possibly because no significant large-scale projects have been documented atop Groovy and because the language is an incremental improvement on successful mixed-paradigm languages.

However, unlike Clojure and Scala, the Groovy language can be learned in a weekend by an existing Java programmer with dynamic language experience, and can be learned in a couple weeks by anyone else. And unlike those other languages, it does not try to be a fundamentally new programming language with new programming models. Instead, it simply dramatically improves the Java language itself using a syntax that becomes quickly familiar to dynamic programmers.

When you consider the cost/benefit analysis of learning Groovy — for a couple weeks of my time, I gain access to being able to easily script — and, with Grails, wrap as web services — an almost limitless supply of Java open source code, you realize you’d be a fool not to learn the language.

I like new stuff as much as the next guy, but Java is a language that is still used by millions of professional programmers across the world, and has proven its ability to be the basis for large projects. And Groovy is a 2X or 3X productivity improvement over Java, with full bi-directional compatibility with existing Java code? I’ll take that over a new-fangled “revolutionary” language any day!

So, what’s so groovy about Groovy?

Oh, nothing… just things like this:

["Rob", "Christopher", "Joe", "John"].findAll{ 
  it.size() < = 4 
}.each { 
  println it 
}
// ==> prints "Rob", "Joe" and "John"

Yes, we’re really not in Java anymore. Let’s walk through the pain that is Java step by step to arrive at our Groovy salvation.

First, Java loves boilerplate. It really is an affliction of the language and the community. A great example of this is Java’s take on module imports.

Even though Java has a great standard library that is extremely well-designed and complete (for the most part), it fails to make this standard library easily accessible to you. That’s because all of Java’s best libraries are hidden in obtuse, deeply-nested library locations, and one can never remember where they are. IDEs solve this problem by providing an “Organize Imports” facility, which expert Java programmers use instead of managing imports themselves. This problem, alone, puts Java IDEs on significantly more productive footing than simple text editors like vim.

Groovy says, “this is nonsense — stop the madness”. In Groovy, a slew of commonly-used Java modules are simply imported automatically. If you run into a namespace conflict, that’s OK — we’re all adults here.

// these are all automatic in every Groovy module you write
import java.lang.*;
import java.util.*;
import java.net.*;
import java.io.*;
import java.math.BigInteger;
import java.math.BigDecimal;
import groovy.lang.*;
import groovy.util.*;

Note that groovy.lang and groovy.util add significant utilities above and beyond Java.

Next up are the major data structures used by any modern programmer — lists and maps (aka dictionaries). If you haven’t been doing Java in awhile, you won’t miss this kind of code.

List<Integer> someItems = Arrays.asList(new Integer[] {1, 2, 3, 4});
for (Integer item : someItems) {
    System.out.println(item);
}
Map<String, String> someMapping = new HashMap<String , String>() {{
    put("ST", "started);
    put("IP", "in progress");
    put("DN", "done");
}}
for (Map.Entry<String, String> entry : someMapping.entrySet()) {
    String key = entry.getKey();
    String value = entry.getValue();
    System.out.println(key + " => " + value);
}

Note that the above is about the most concise possible code you can write for these operations using standard Java facilities. In fact, I have even tapped into some obscure features, using Arrays.asList to initialize the ArrayList (leveraging the fact that arrays, but not lists, have a concise initialization syntax) and leveraging double curly brace initialization for the Map, which is an extremely non-standard and perhaps even overly inefficient technique. (The alternative, though, is to write out someMapping.put(key, val) for each entry in the Map — ick).

Let’s look at this same syntax, but in Groovy.

someItems = [1, 2, 3, 4]
someItems.each {
  println it
}
someMapping = ["ST": "started", "IP": "in progress", "DN": "done"]
someMapping.each { key, val ->
  println "${key} => ${val}"
}

Not only is this code more concise, it is also much more readable. The signal-to-noise ratio is much better.

Groovy has great support for Functional Programming and Domain-Specific Languages.

// first-class functions/methods and DSL syntax
clean = { text ->
  text.replaceAll("[^A-Za-z]*", "")
}
clean "my-text-*foo"
remove = someItems.&remove
remove 3
someItems

It retains support for loved Java features like switch statements, but these become much more powerful and concise, e.g. in this factory function that actually returns cleaner functions depending on the passed-in type.

// proper closure, switches and variable interpolation
cleanerFactory = { type -> 
  switch (type) {
    case "text":
        return clean
    case "numbers":
        return { text ->
            text.replaceAll("[^0-9]*", "")
        }
    default:
        throw new IllegalArgumentException("invalid type ${type}")
  }
}

Using the cleanerFactory is a breeze, simply call it and you get back a function which you may then call. Yes, this is basically impossible in Java. To get something close, you’d have to make the factory return “Runnable” instances and then call .run() on the results.

// try/catch and bean syntax shorthand
test = "my-text-*foo-1234"
println cleanerFactory("text")(test)
println cleanerFactory("numbers")(test)
try {
  println cleanerFactory("foo")(test)
} catch (IllegalArgumentException ex) {
  println "cleanerFactory returned exception: ${ex.class.simpleName}, ${ex.message}"
}

Just like Java, Groovy supports classes. But it improves them by adding a concise syntax for declaring proper JavaBeans. The following class has getters, setters, and a keyword constructor autogenerated for you.

// classes and types, if you need them
class Article {
  String title
  String summary
  String content
  Date date
}
article = new Article(
    title: "Parse.ly uses Groovy, oh noes!", 
    summary: "Didier resigns in protest", 
    content: "Keith goes back to .Net, Toms returns to Ireland.  Ahhhh!",
    date: new Date(2011, 3, 7)
)
article.properties.each{println it}

Also notice the special “.properties” property, which allows you to iterate over a JavaBean’s property without using Java’s obtuse reflection API.

// inheritance if you need it
class WSJArticle extends Article {
    List topics
}
wsj = new WSJArticle(topics: ["US", "Finance", "World News"])
wsj.topics.each{ println it }

Inheritance works exactly as expected, and follows Java rules. However, inheritance gets even better in Groovy because it loosens stupid rules that Java has, such as “final” classes.

// and sure, even call and inherit from Java code (even final classes!)
class BetterDate extends Date {
    def getTimeString = {
        return "{this.time}"
    }
}
println new BetterDate().timeString

It’s simply not possible to subclass Date in Java, but Groovy has no problem with that. We’re all adults here.

// and sure, use Groovy scripting constructs with existing Java APIs!
lotsaItems = [1, 1, 2, 2, 2, 3, 3, 3, 3, 3]
println Collections.frequency(lotsaItems, 3)
Collections.shuffle(lotsaItems)
println lotsaItems

The above is an example of using existing Java APIs. Collections is surely a nice static class. Groovy lets you use it to your heart’s content.

The API for the Collections.binarySearch method is written as:

static <T> int binarySearch(List<? extends T> list, T key, Comparator<? super T> c)

Yes, that is “documentation” So basically, I need to create a class that implements the Comparator interface, and then I need to pass that comparator to this method along with my list. That will require a lot of angle brackets to get the generics right. Hmm, who cares, let’s just do it the easy way: in Groovy.

// even existing, complex APIs work fine in Groovy, and with less code!
comparator = [
  compare: { a, b -> a.equals(b) ? 0 : Math.abs(a) < Math.abs(b) ? -1 : 1 }
] as Comparator
println Collections.binarySearch([0, 2, 5, 7, 8], 5, comparator)

This shows off the ability of Groovy to easily coerce values using some very intelligent rules. Comparator is an interface that expects a “compare” method — therefore, we can coerce a map with a “compare” key and a function value into an implementation of that interface.

// convenience features abound, like heredocs and safe pointer access
myDocument = """
  This is a long document...
"""
foo = [myDoc: null]
println foo.myDoc?.size() // returns null rather than an exception

If I had a dime for every time I noticed Java programmers using StringBuilder to work around its lack of multi-line strings, or adding extra methods just to do null checks... well, I'd be very rich indeed Groovy says -- no need to complicate your life with that noise.

// and type coercion that actually works!
set = [1, 2, 3, 4, 5] as Set
5 in set
// ==> true!

Yes, the in keyword. Yes, lists can be converted to sets. Because that makes sense!

// this is a feature I wish Python had -- JavaScript-like map syntax
map = [:]
map.item1 = "foo"
map.item2 = "bar"
map.each { println it }
// ==> "item1"="foo", "item2"="bar"
println "done"

Groovy also learns a little from JavaScript -- maps are accessible using either key index or dot syntax. I personally like this, I know some Python people won't.

// nice range and slice syntax
items = [1, 2, 3, 4, 5]
println items[0..2]
for (i in 0..5) { println i }

Similar to Python's slice syntax, Groovy supports a "Range" type using two dots. It works where you expect it to: array indexing, for loops.

Using Grails, the standard web framework for Groovy, you can write simple RESTful, JSON web services in just tens of lines of code. e.g., here's a Grails controller that renders the result of a function as JSON.

package service
import com.thirdparty.library.ContentExtractor
import grails.converters.JSON
 
class ParseController {
  def contentExtractor = new ContentExtractor()
  def index = {
    def article = contentExtractor.extractContent(params.url)
    render article as JSON
  }
}

This will respond to a service like http://service:8080/parse?url=http://google.com with a JSON document containing the Java properties from the article object.

Seriously, how much easier can it get to wrap up a Java library as a JSON web service? Not much.

// in short, a pretty language worth looking at
println "DONE."

Download Groovy and Grails to start playing around!

Early on during this startup adventure, a person I trust told me, “Watch out — startups aren’t for the faint of heart.” Looking back on my personal net income graph from 2009 to present, I can see what he meant.

May 2009 is when I entered Dreamit Ventures to begin working on what would become Parse.ly. That’s when I plunged my “savings buffer” into the company. The few months after that had me frantically trying to recover from the realization that startup progress is measured in months and years, not days and weeks.

Sachin and I switched gears from targeting consumers with a free product to targeting large online content properties with a paid product, and bootstrapped the company with side consulting gigs. We didn’t tell anyone we did the side consulting work (unless they specifically asked). We watched other entrepreneurs go into credit card debt and borrow money from trusting friends and relatives. We didn’t believe in that, so we took the hard road of “earning our survival”.

However, our costs were going up, not down, as we pursued a more ambitious product with more demanding clients. Also, my expenses skyrocketed as COBRA disappeared for my health insurance and I had to pay for horribly overpriced sole proprietorship plans. (Fact: America’s broken healthcare system is harmful to entrepreneurs.) I knew I needed to do something to “stop the bleeding” on my financial situation — so, I took on more consulting gigs…

My consulting gigs only started earning me enough to stabilize my personal expenses late last year. That’s when you see my net income turn into a very nice flat line. Those months felt more secure than any of the preceding ones. It was around then that I realized a lot of people took advantage of my desperate position and underpaid me. I raised my prices and started to round up talented engineers to work with me as a distributed team. That got things under control.

Mid-to-end of 2010, I felt like I could conquer the world. My finances stable, I didn’t need me no stinkin’ investors. We could bootstrap this company to success ourselves. After a year of hard work, I had plentiful consulting clients, and more knocking down my door. There was a stable 50/50 split between Parse.ly time and paid consulting time. We were also on the verge of closing big contracts for Parse.ly. I could cover my expenses well, and saw lots of personal gain in my future. My consulting company even seemed like a nicely profitable and happy company in its own right.

Of course, it was around this time that investors finally decided to notice Parse.ly and that we closed financing. Note to entrepreneurs — no one ever gives you money when you actually need it!

Mid-January was first time in 2 years that I had health insurance paid by someone else and my first “real paycheck” in 2 years. January and February have been amazing, productive, ambitious, and life-changing months. So proud of Parse.ly team and what’s coming up next for us.

In all, the past 18 months has been a crazy ride, but I survived! Personal relationships were strained, but not soured. Financial pressure applied, but not to the breaking point.

This graph above is also quite an interesting side effect of this process. It shows the net income not for me personally, but for my consulting company, Aleph Point, Inc. In the span of 10 months, I took my consulting company from a 1-man shop to a profitable, legitimate company. That required me to wear a “CEO hat” and focus on things like profit margins, contract negotiation, and customer acquisition. What I like about this graph is that it’s a hockey stick profit chart. True, consulting work is not scalable, but it sure is profitable. Of course, due to Parse.ly’s funding round, I now need a bigger hockey stick.

“Don’t do other things?”

I wanted to write about this experience because it directly contradicts some advice one of my online mentors, Paul Graham, gave in his essay, “How Not to Die”. He wrote:

The number one thing not to do is other things. If you find yourself saying a sentence that ends with “but we’re going to keep working on the startup,” you are in big trouble. Bob’s going to grad school, but we’re going to keep working on the startup. We’re moving back to Minnesota, but we’re going to keep working on the startup. We’re taking on some consulting projects, but we’re going to keep working on the startup. You may as well just translate these to “we’re giving up on the startup, but we’re not willing to admit that to ourselves,” because that’s what it means most of the time. A startup is so hard that working on it can’t be preceded by “but.”

If I hadn’t worked on my consulting projects last year, Parse.ly would have died. PERIOD. PG is often right, but he’s wrong on this one. I agree that startups require your undivided attention. But surviving isn’t a compromise — it’s a necessity. Do it however you possibly can.

In any event, I learned a lot from this experience. On to the next! It’s time to pump some green for Parse.ly!

Hacker News

Oh, wow. I ended up on the front page of Hacker News. Check out the great comments and discussion going on there. Also, feel free to check out some other posts I’ve written about startups, such as:

• Ideas and Execution
• What One Does
• The Startup Diet
• It’s easier to play the option than the bet

I made an interesting discovery today.

“Free Market” vs. “Labor Union” in Google Ngram Book Viewer

Explains why no one has heard of labor unions and everyone is raving about the free market

(by the way, you can download the entire dataset behind this neat little Google Labs project)

Let’s say that you want to make some money at the horse tracks. The guy behind the booth gives you the following choice:

1. The Bet. You can place a bet now, before the race starts, on the horse with, in your view, the best odds of winning.
2. The Option. Instead of betting now, he will sell you an option. The option gives you the right to bet on any horse during the race, up until 5 seconds before the first horse crosses the finish line. However, there’s a small catch: everyone else who owns this option has the same right, and only 3 gamblers are allowed per horse during the race. If all the spots fill up, you simply get your money back. Further, each other gambler’s choice will be made publicly available to others during the race.

Most people, even if they don’t know much about horse betting, will rightfully think The Option is the way to go. No matter how much information you might have about each of the horses and riders before the race, the information you gather during the race is much more valuable. A top-ranked horse might break a leg, or its rider might trip up on the start. Even if you pick a favorite before the race begins, once it gets going and you see him lagging behind, the economic reality put in front of you will override your speculative capacities.

In short, it’s easier to play the option than the bet. It’s less risky. It’s more fun. There are less ways to lose. So, it’s no surprise most horse betting does not work this way.

However, let’s analyze the interesting crowd behavior that emerges during a race where The Option is available.

Some Examples

Let’s say one horse suddenly pulls ahead of the pack. Those who are paying attention and who hold the option will quickly exercise it. Many other gamblers will miss their opportunity, creating an oversubscribed horse.

Take another example. Since the bets are public, other gamblers will be inclined to pay attention to signals from the gamblers with the best past record of picking horses. If a well-known gambler with an excellent record picks a horse in the middle of the pack, its two other spots will likely fill up very quickly.

Some gamblers will avoid making picks based on sudden movements by the horses or other gamblers. Instead, these less emotional players will wait until a reasonable time period before the end of the race — say 30 or 45 seconds. They will wait until they have all the trending information necessary to make a call. They won’t mind picking a #2 or #3 winner — they just want minimal risk of picking a loser.

Finally, some horses will go unpicked altogether. That’s not to say they aren’t strong horses with a lot of potential. They just have a lot of perceived risk. And gamblers don’t like taking on risk when The Option gives them a way to altogether mitigate it.

Changing the Rules

A horse race with The Option at play seems complicated enough, but let’s change the rules a little more. Let’s say the race doesn’t end — doesn’t have a set start time and end time. Let’s say that once a horse gets 3 options exercised on it, it’s removed from the race. Let’s say new horses enter the race every day, and that horses only tend to run for between 2-5 minutes each. After 5 minutes, most riders remove their horse from the race on their own.

In this new race, small signals have a big impact. Charge ahead suddenly and you might get your 3 spots filled. Convince a top-tier gambler to go to bat for you, and you’re all set — your other two spots will fill up quickly. If you’ve been in the race before and had your spots filled up quickly, you’ll likely get them filled up quickly when you enter the race again.

Managing which options to exercise also becomes a little trickier. With new horses entering all the time, the biggest gamblers will start to have a horse flow management problem. They’ll delegate some of the initial analysis and tracking work to junior gamblers. Riders and horse owners will start to do outrageous things to get noticed by the gambling ecosystem.

As but one horse of hundreds or thousands, it will therefore become possible for you to run a steady race without any of these signals, and basically go unnoticed in the crowd. Occasionally, you will get considered by the unemotional, data-oriented gamblers. However, they will wait until the last possible minute before pulling the trigger on your option, because it is in their interest to do so.

Do the Gamblers Change the Game?

One has to wonder what effect The Option has on the race. For example, if the goal is to leave the race via The Option — rather than to cross the finish line — will some riders start slow and then sprint quickly ahead to attract the fast-acting trend-followers? Will some riders try to network with the top-tier gamblers before they enter the race, to win their favor outside the game itself?

Would any gambler even choose The Old-Fashioned Bet anymore? It seems to me the only time someone would is if the rider and horse had such a great past record, that they were a favorite to win. No-name rider and newly-trained horses will never garner Bets, only Options.

And what about those hundreds of mid-range horses that never make sudden forward motions? Instead, they slowly and steadily continue to beat their best lap time. They aren’t flashy and never catch a glance from the top-tier gamblers, or even the stadium audience. For them, there is but one choice: Keep riding. Keep pushing. A gambler may notice you at the last moment, just when you think you haven’t any stamina or wherewithal left.

Sure, it won’t seem fair, to see all these other horses leaving the race early. But don’t look left and right — just look ahead. Before you know it, you may not even care about The Option, since you will have crossed the finish line all on your own.

So, next time you hear a “maybe” when you ask someone to bet on your horse, remember: it’s easier to play the option than the bet. The reason you entered the race in the first place was to ride like hell and win for yourself, not to be someone else’s winning bet.

Pythonic isn’t just idiomatic Python — it’s tasteful Python. It’s less an objective property of code, more a compliment bestowed onto especially nice Python code. The reason Pythonistas have their own word for this is because Python is a language that encourages good taste; Python programmers with poor taste tend to write un-Pythonic code.

This is highly subjective, but can be easily understood by Pythonistas who have been with the language for awhile.

Here’s some un-Pythonic code:

def xform(item):
    data = {}
    data["title"] = item["title"].encode("utf-8", "ignore")
    data["summary"] = item["summary"].encode("utf-8", "ignore")
    return data

This code is both un-Pythonic and unidiomatic. There’s some code duplication which can very easily be factored out. The programmer hasn’t used concise, readability-enhancing facilities that are available to him by the language. Even lazy programmers will recognize this code’s clear downsides.

Here’s another version that is more idiomatic but is nonetheless still un-Pythonic:

class ItemTransformer(object):
    def __init__(self, item):
        self.item = item
 
    def encode(self, key):
        return self.item[key].encode("utf-8", "ignore")
 
    def transform(self):
        return dict(
            title=self.encode("title"), 
            summary=self.encode("summary"))

Nothing about this code is particularly unidiomatic. I might even see code like this in many popular open source projects. But it’s in poor taste. It’s un-Pythonic.

What is the code doing? It’s just taking an incoming dictionary, encoding its values using utf-8, and returning a new dictionary with those encoded values. There is no need to introduce an ItemTransformer object — it’s an extra abstraction and is just making the signal-to-noise ratio poorer. People coming from Java often write un-Pythonic code because Java is a language that does not reward good taste. The Pythonic view is: programming is hard enough — let’s not make it harder for ourselves.

Here’s a more Pythonic version:

def encode_news_item(item):
    def encoded(*keys):
        for key in keys: 
            yield (key, item[key].encode("utf-8", "ignore"))
    return dict(encoded("title", "summary"))

This code shows comfort with Python’s features, but does not abuse this comfort by obfuscating the code with mind-bending constructions. The programmer has reduced the problem to two comprehensible subproblems: creating a stream of tuples (key, encoded_value) and constructing the new dictionary from that stream. This leverages the elegant fact that in Python, dictionaries can be easily constructed from (key, value) tuples.

This version avoids code duplication while also making the last line (the return statement) a rough description of the entire function. “return a dictionary of the encoded values for the keys title and summary” Idiomatic, yes. But also tasteful, and thus Pythonic.

Even Pythonic code can be improved:

def encode_news_item(item):
    encoded = lambda key: item[key].encode("utf-8", "ignore")
    keys = ("title", "summary")
    return dict((key, encoded(key)) for key in keys)

And though Pythonic code is often smaller than its un-Pythonic counterparts, the experienced Pythonista knows the road to hell is paved with good intentions:

def encode_news_item(item):
    keys = ("title", "summary")
    return dict(zip(keys, map(lambda key: item[key].encode("utf-8", "ignore"), keys)))

Ick… time to hg revert this idiomatic Python code to the more Pythonic version

Last summer, we got our company, Parse.ly, off the ground at DreamIt Ventures incubator program in Philadelphia. Since then, we’ve talked to a lot of founders about our experience in the program. Many founders are data-driven people who are looking for concrete advice about how to optimize their experience at these programs. One of the most successful runway-extending pieces of advice we have given has been to keep food costs low. We were able to get our food cost down to $4/person/day through some simple planning during that summer, and each of us also lost 10-15 pounds in the process. We felt great, were productive, and made our DreamIt investment last. I think this might be one of the core reasons for our company’s survival and success. This is the story behind “The Startup Diet”.

DreamIt Ventures had just cut us a check for $20K to get our startup off the ground. But my cofounder Sachin and I were worried. $20K seems like a lot of money, but it’s actually not that much. Not when you’re using it for both living expenses and to hire other people to get your company off the ground. So we started planning our spend and rationing the money immediately.

We knew we’d use some of the money for our living expenses. We had just arrived in Philadelphia, and we were living in a startup house with Matt and Burak, the founders of Tidal Labs, and Jack, one of the founders of SeatGeek. It turns out that rent wasn’t that expensive in Philly, especially in this arrangement. Instead, our number one cost, we determined, was going to be food.

The Next Meal

One of the founders of modern management, Peter F. Drucker, once wrote, “In all recorded history there has not been one economist who has had to worry about where the next meal would come from.” How true. But every startup entrepreneur I talked to obsessed with one thing and one thing only: “runway”. In other words, where the next meal would come from once the small amount of funding we got ran out.

With some good take-out and food truck options in University City (where DreamIt’s offices were), we were already spending as little as $10/person/day. But actually getting this food wasted valuable time, and it usually wasn’t very healthy. We heard reports of the “Startup 15″ — the 15 pounds many founders gain when beginning work on their companies. We also knew that when you eat like crap, it affects your productivity negatively. Finally, we knew healthier takeout options were prohibitively expensive. This presented a serious problem. Thinking like founders, we decided to hack this situation to our advantage.

Most companies at DreamIt figured it couldn’t get any better than the food trucks, and stuck to low-cost takeout most every day. We wondered — was there a way we could cut our food cost down, eat healthily, have positive productivity, save time, and hell, even lose weight in the process?

The Birth of a Good Idea

Thus was born The Startup Diet. What we needed was a diet with the following properties:

• Healthy, energy-filled food to enhance productivity
• Low daily calorie intake to prevent weight gain
• Idiot-proof and simple meal preparation
• Extremely cheap per-person cost
• Ability to purchase in bulk to reduce time wasted at supermarket

Sachin went online researching diet options and came across Tim Ferriss’s blog. Ferriss takes a scientific approach to living, and is best known for his popular book, The 4-Hour Work Week. The relevant article Sachin found was called How to Lose 20 lbs. of Fat in 30 Days… Without Doing Any Exercise. This provided an excellent starting point.

The Four Rules

I encourage you to read his post, but for the impatient, here are his rules:

• Rule #1: Avoid “white” carbohydrates
• Rule #2: Eat the same few meals over and over again
• Rule #3: Don’t drink calories
• Rule #4: Take one day off per week

The diet we came up with was basically a fork of Ferriss’s. We followed his four rules. The most important of these ended up being rules #2 and #4. By turning our meals into a routine, it became much easier to make them efficient and reduce the time spent buying, prepping, and cooking food. By taking one day off per week, we didn’t feel that we were “missing out” on important business opportunities as a result of our diet — e.g., lunch appointments with mentors or a night out at the bar to blow off steam with the team and other founders. Our “cheat day” also had an interesting other efficiency effect: we were just forced to schedule all of our breakfast/lunch/dinner meetings for a single day, and thus didn’t interrupt our work week with distractions.

The Core Insight: Beans as a Base

The goals of our diet were different than Ferriss’s. But like him, we used beans/legumes as a base. Beans are an excellent base food for a startup diet because they meet all of our core requirements above. But the best thing about beans is their startup economics. If you consider that our goal is to ingest the cheapest/healthiest possible calories, you can’t find a more flexible foodstuff than canned black beans. You can buy organic canned black beans for as little as $1.50/can. One can of beans is around 600 calories.

If you look at black beans from an efficiency standpoint, you get 4 calories per $0.01. Compare a slice of pizza which is 200 calories and will usually run you $2.50; that’s only about 0.8 calories per $0.01, making beans about 5X more efficient. But beans aren’t just more energy-efficient, they’re also loaded with protein and fiber (about 7g of each per serving). Most of the cost in a diet comes from eating expensive proteins like chicken and beef, and beans have the ability to provide both efficient calories and efficient proteins at the same time. Finally, you can buy canned beans in bulk and store them.

For other nutrition, we would lean on four more sources: eggs; salad (mixed greens, romaine lettuce, tomatoes, and peppers); prepackaged vegetarian protein sources like tofu, seitan, and wheat gluten which we would use sparingly; and canned soups.

The Trial Run

We went to the supermarket to run our first trial. We knew this would need some coordination. Our first trick was to make an up-front investment in lots of differently-sized Ziploc containers and canvas carrying bags, so that we could flexibly relocate food where we needed it (home, office).

We wanted to prep food for the week ahead of time and bring it into the office to ensure we didn’t cheat on the diet. The next trick was to come up with a meal plan using the four rules and our beans-as-a-base theory. Here is the meal plan we came up with:

• Breakfast: alternate between 2 eggs per person or vegetarian saussage links; add hot sauce and side of mixed greens.
• Lunch: alternate between mixed salad with one can of soup (we liked Split Pea and Lentil) or salad with black beans on top. We would prep the salad for the week in zip lock containers, and leave the rest of the things we needed (e.g. salad dressing or the canned beans) at the office.
• Dinner: a combination of black beans, vegetarian protein (usually “chicken” strips or “beef” rips), hot sauce, onions, and jalapeño peppers. We would eat this “core” food like a burrito stuffing, but instead of putting it in a tortilla, we would eat it with romaine lettuce wrappings. (Think “Lettuce Wraps” that you can get at fancy New American / Asian restaurants.) The great thing about this as our dinner food is that we could make the filling ahead of time for 2-3 days and then just reheat it. Fresh Romaine Lettuce has a great crunch and lets you eat the food with your bare hands, even minimizing dish cleanup time. We just had to keep a regular supply of Romaine Lettuce on-hand.

An Agile-Friendly Diet

I mentioned Matt from Tidal Labs. He was also our landlord. His Spruce Hill House was the perfect atmosphere for getting some serious summer hacking done. But one of the nicest — and eventually, most useful — things about the house was the huge kitchen and huge Sub Zero refrigerator. Matt later told me bought the refrigerator off eBay for a $1 (it wasn’t functional and the owner just wanted to get rid of it), and then Matt fixed it himself. He is scrappy and awesome that way.

Sachin and I treated this Sub Zero fridge as our diet command center. We would prep our meals before and after our long days at the office. One of us would prep breakfast for the day while the other prepped lunch/dinner materials for the next 2 days. We’d collaborate on prepping dinner. Every couple days, we’d swap.

We cherished this time for its management value to the company. In the morning, we’d have a founder standup: we’d figure out the things we wanted to accomplish for the day while prepping and eating breakfast. We’d evaluate the day over dinner, a kind of mini retrospective. The Startup Diet became a kind of management technique.

Making It Work

Perhaps by now you’re convinced that this diet has a lot of cost and efficiency benefits. You can’t beat living on $4/person/day for food. You can’t beat saving (by our estimate) 1-2 hours a day on prep and meal time.

But, the main question we get about this was, didn’t that taste horrible? Surprisingly, no. We used a lot of obvious flavor enhancers to keep things tasting pretty good. We invested in good olive oil, sea salt (in a grinder), black peppercorns (in a grinder), and threw in a liberal helping of spices, hot sauce, and herbs.

The second common question we get was, wasn’t it mind-numbingly boring to keep eating the same thing over and over again? Again, surprisingly, no. The first week you wonder how you’ll go a whole summer like this. The second week, it just becomes the routine.

Remember, startups are hard in the early stages. Every day requires your fullest attention. Therefore, turning meals into a routine actually made a lot of sense — it reduced the degree to which we were interrupted in getting the company off the ground. Our body got used to it, and we hardly even noticed that we were eating the same meals week by week. Rule #4, the cheat day, was the real savior in this aspect. We used to look forward to our cheat days, and we made them count. Cheat days also acted as a kind of reward for a week of focused work.

Making It Your Own

I am not a doctor or a nutrition expert, and so all the regular caveats apply. But this was how we were able to keep our costs down while Parse.ly was being incubated at DreamIt Ventures.

Learn to love black beans and other legumes — they are a healthy, flexible and efficient food. Plan your approach. Turn eating into a routine, and prepare most of your meals ahead of time. Cut out the carbs, beer and soda — drink lots of water instead. Use meal prep time to manage your company. Use your cheat day to manage external relationships and reward yourself.

Stay focused on your startup. Lose weight. Gain productivity. Extend your runway.

The rest is up to you.

One of America’s greatest strengths is social mobility. There are several cases of an individual starting with nothing and persevering to become rich, powerful, and influential. Success stories of this kind have become an important part of American business mythology, especially in the world of entrepreneurship. They are strong motivators for individuals embarking on companies of their own.

For those of us who start companies, we see the company as a vehicle to creating something valuable and lasting for society, while also advancing our personal goals. This isn’t usually hubris or ego, though sometimes it may be. Instead, it’s usually an attempt to make your time worthwhile: to yourself, to those close to you, and — if you’re lucky and persistent enough — to the entire world.

The problem with social mobility is that not every individual begins at the same starting line. In fact, the range is huge. Those who start with an influential family or significant capital resources have a much easier time getting to the top. For those who don’t have this head start, things are a lot harder.

Though America is not entirely merit-based, it can reward individuals for hard work. I’ve experienced the benefit both of an advantageous starting point and hard work in my 26 years on this earth. I also believe that with each step and milestone in my life, my potential to create enduring value for society has increased significantly.

Beginnings

The inspiration for this essay was a comment I read online about a successful young businessman who was the son of a successful businessman. “I’d [like to] read a story about a 25-yo [who] made good on the same scale[,] who went to a state college, had screwed up parents who were too busy fighting with each other or gettiing drunk to even have a clue what he was doing, isn’t childhood friends with a celebrity… Just happened to be smart and hardworking and optimistic even despite all those factors. That would actually be an interesting story.”

My parents weren’t screwed up, but they did fight a lot — my Mom and Dad separated when I was in elementary school and divorced shortly after that. I’ve not been childhood friends with a celebrity and I don’t have a trust fund.

I’m not saying I came from a disadvantaged upbringing — in fact, quite the opposite. I went to public high school in New York. To a New Yorker, that may not sound like a huge step up in the world, but I recognize that public school in New York represents one of the top educations you can get.

I grew up in a nice house in a quiet suburban neighborhood. I had good, encouraging teachers. My parents were liberal and a positive influence. I didn’t have a silver spoon in my mouth, but I also didn’t have any serious handicap in my upbringing. Probably my biggest step up in the world, given my current trajectory, was that when I was 10 or 11 years old, my Dad noticed an interest I had in computers. And so, he bought me a PC (running DOS / Windows 3.1) and set it up for me on Christmas Day. From that point forward, I was enchanted by the machine. And once I got a 28K modem and dial-up access to the web (on one of the first ISPs, The Pipeline), I became a citizen of the world before I had even hit puberty.

This I do know — though I had a head start, I also worked hard. I was a geek — as I got older, I built my own servers in my basement, taught myself to program, and discovered Linux and Free Software. But I also kept ahead in “the real world”. I did feel a little disconnected from my peers in my private pursuits; reflecting on my childhood, I realize I “grew up” a little more quickly than my peers.

My First “Startups”

At 15, I had built and launched somewhat popular websites for a couple of personal projects. Nothing you would remember — each one was related in some way to an interest I had in multiplayer online video games, an interest I aged out of a year later. Each website had between a few hundred and several thousand visitors a week.

But the sites looked nice. When I told my family friends and teachers in my local suburb that “I built websites”, they were all interested in having me work for them. Recognizing the opportunity, I turned this into a business.

By 2010 standards, the websites I built were boring and simple — static designs, gaudy flash “intros”, contact forms and quickly-outdated information portals. Yet in 1999, this was cutting-edge. So cutting-edge that people were willing to pay to be on the web. I printed up business cards and called my company a “digital online identity firm”. Might seem like an obvious thing now, but I felt proud to deliver that value to my early-adopter customers. This was my first “B2B startup”. During this period, my friends would ask me to hang out on the weekends, but I’d be working on HTML/CSS, JavaScript and Macromedia Flash, building up my online portfolio.

At 16, I was approached by the editors of a technology publishing company called Friends of ED to write a couple chapters for a book of theirs called Flash 5 Dynamic Content Studio. They found me on the web — my first “digital introduction”, before the days of LinkedIn. On the weekends, I regularly answered people’s Actionscript programming questions in a listserv called flashpro. They liked that I had both a Flash and programming background, so they contacted me to contribute to their book — one of the first that discussed building backend systems and tying them to Flash frontends.

I corresponded with them entirely via e-mail. I wrote the chapters and edited about 80% of the book during my junior year of high school, while also prepping for SATs and acting as editor of my school newspaper. I probably got 3 hours of sleep a night. My friends thought I was crazy.

When the book finally went to press and started showing up on Amazon.com and in Barnes & Noble stores, it was one of my proudest moments in my early adulthood. I just thought about how the work I did was being seen and read by thousands of people across the world. Each of those individuals would go on and use that knowledge to build web applications for other individuals. There was seemingly no limit to the impact of my work. I may have been naive, but I thought, what could be better than this?

I still remember when the editors in the UK found out I was only 16 (it was while we were negotiating payment). They were shocked, they actually didn’t believe me. They had to check with lawyers to see if they could even pay me. Then they paid me the bunch of money they owed me for my hourly editing rate and IP rights, and I had all the savings I needed to live throughout my first couple college years without asking my parents for support.

By 16, I had some unique experiences. I created a market in my skills. I built working software for companies small and large. I contributed to a project that had global impact. I made some real money. And, in my junior and senior years of high school, I got a taste of the competing demands of personal and work life.

College: Creating Value for Myself

When I got into NYU on scholarship, I was thrilled. My grades in high school let me go to a good college without breaking the bank. I would be able to attend a top-tier school without the financial stress typically associated with private schools. I’d get to stay in New York, one of my true loves. I had also been rejected by just about every other top-tier university in the country. Come to think of it, the ups and downs of the college application process was good training for startup life — especially the rejections.

I worked my butt off in college. I really got into it. Not just the coursework, but also just pursuing my passions within the student body. I held talks that tried to stoke up my peers about technology, economics and open source software, all passions of mine. My friends rarely saw me because I was often working on these and other spare time projects.

College was interesting. I worked hard, but rarely got paid for my work. In some sense, working hard in college is good preparation for a startup lifestyle. You work hard because you believe in yourself and in your own potential, not because you have a paycheck coming.

It was also in college where I came to an important realization: I am a software engineer and computer scientist. Though there are many ways for me to add value, my preparation and passions make me especially well-suited to building software. And so, that’s where I should have an impact.

Internship: Creating Value for Others

I took an internship in summer of my sophomore year where I got paid a flat $3000 to build a web application for a NYC-based non-profit, The Unemployment Action Center of NY. I probably spent 750 hours on the thing (60 hours a week), not making it a very profitable proposition. My friends again thought I was crazy — lots of them had taken well-paying and prestigious internships at software companies and banks, but I didn’t even apply to those.

This was my first experience with building a real product that people actually used. I worked entirely from home. I used my own development environment and hosting environment. I gathered requirements directly from the end-users. I made all the technical decisions and implemented all the code. That was good experience for startup life. Built useful stuff, solved real problems. Made lots of little decisions.

The case management application I built is still running today, relatively bug-free, and has helped thousands of unemployed people get legal representation for free, so that they can get justice for wrongful termination and other cases. I’m really proud of that application. I didn’t care that I got paid $4/hour for it. Looking back, I would have done it for free. What a phenomenal organization and what a great cause to apply technology toward.

Post-College: Creating Value for Money

The rest of my story is told pretty well by this NY Observer article.

I worked at Morgan Stanley for a couple of years while living at home on Long Island to save money. Working at the bank was a huge change from the rest of my life. The primary focus of my job became my paycheck. Technology decisions were made for me, by external committees. I was expected to follow all sorts of processes and procedures. I was expected to become a conventional programmer. I found myself building products that had an unclear end-user.

I did learn things there, too. I learned what it was like to work with engineers on a daily basis. I learned a lot from my managers and colleagues, many of whom were just astoundingly intelligent individuals. I learned how big companies operated. But mostly, I learned a lot about myself.

I learned that work meant more to me than a paycheck. Work should be about solving problems, helping people, and creating enduring value. Money is important, but it wasn’t what attracted me to technology in the first place. And that’s why I knew I had to leave that firm.

Startup Life: Creating Lasting Value for Myself, for Others, and for Money

I accepted this compromise: I would work at the bank to save the money I needed to build my own company. I was “realistic”, and this seemed a fair compromise.

I anticipated that I’d need at least a $20K buffer to start my startup (it ended up being more than that). And in March of last year, I quit Morgan Stanley to embark on this path. My awesome, like-minded friend from NYU, Sachin Kamdar, quit his job too. Together, we tried to build something of value, even though we didn’t know what that would be initially. We worked out of cafes, brainstorming ideas and building prototypes. We consulted on the side — I, through the boutique software engineering firm I founded, Aleph Point. Early on, my girlfriend prophetically referred to my financial strategy in all this as “putting a bandaid on a hemorrhaging femoral artery”.

Our friends thought we were crazy. Chuck a steady paycheck to work on an ill-defined “startup”? For some reason, all we could think was, hell yes we would.

Our first stroke of luck was getting into DreamIt Ventures. NYC didn’t have seed programs like Tech Stars or Seed Start yet. We moved to Philadelphia for the summer, leaving our NYC apartment leases and girlfriends behind. They thought we were crazy. The $20K from DreamIt basically covered our living expenses in Philly, but it did lift some financial pressure for a few months. We came up with a $4/person/day diet — which we called “the startup diet” — that involved eating beans, soup, romaine lettuce, and veggie burgers basically every day. We lived in a “startup house” with the founders of SeatGeek and Tidal Labs. We worked in an office with the founders of Postling, NoteHall, and other fledgling companies. We started hacking on what would eventually become Parse.ly. And for the first time in our lives, we were surrounded by individuals who also felt that what one does is more important than how one benefits.

Our second stroke of luck was finding our exceptionally talented lead engineer, Didier Deshommes, who has been with us ever since we incubated Parse.ly in Philadelphia. Ever since DreamIt, we operate on this maxim: we got our share of luck — now, let’s make our own.

Creating Your Own Luck

Perhaps our startup founding story isn’t remarkable. But lately, reflecting on the last year — where we bootstrapped and self-financed this startup, overworking ourselves and straining our personal relationships — I’ve started to feel like it is pretty remarkable.

We made our own luck. And we’ve brought all our skills and experience to bear. I’ve been preparing for running Parse.ly since the age of 15. I’ve been preparing, basically, my whole life. And I’m so glad we did this together. With all the ups and downs, if I had done this on my own, I would have given up. But we just kept pushing each other. Every time we hit a roadblock, we just said, “we can still do this.” And we did.

I’m really proud of this company and our team. We’ve built a real, valuable technology of lasting value. We’ve learned so much. And we’ve done it all against the odds. We are a small NYC startup with two no-name founders. Neither of us have “prior exits” or “a solid track record”. Neither of us have built and launched startups before. But none of that matters. We discovered that people want their content filtered and prioritized and that online publishers want their content optimized, and we used elbow grease, hustle, and dedicated hacking to make the rest happen.

Our startup is three hard-working and optimistic guys who believe that we can change the world. We put it all on the line to create something from nothing. Our work is still in progress, and there are still challenges ahead. There are easier things we could be doing, but we don’t care. You may think we’re crazy, and you may be right.

But that’s the final lesson of startups, isn’t it? You have to be a little crazy.

—

Thanks to Sachin Kamdar for editing this essay. And thanks to our awesome friends, significant others, and families for still cheering us on even as they rightfully thought we were crazy.

Nobody moves to New York because they think they’re just like everybody else. A young kid, fueled by a toxic blend of bravado and wicked insecurity, can expend a truly terrifying amount of energy trying to prove her exceptionalism, prove that she is different (read: better) than the dull hometown peers she left behind, who go to uncreative jobs in uncreative clothes, eat at Chili’s, practice monogamy.

Describes many non-native New Yorkers really well. No offense non-natives, but some of you are definitely trying to prove something! From Dirty pictures I didn’t want taken

My good friends at HiiDef just launched a new app that has been in beta for awhile, Flavors.me. This is an excellent tool that has a great, simple, and usable design.

What’s the value preposition of Flavors.me? It’s to unify your various “online identities” into a single, dynamic, automatically-updated, and elegant website.

What do I mean by that? OK — so, like most people on the web, you spread public information about yourself in multiple places. You might run one or two blogs (personal and work?). You might have a Facebook account, a Twitter account. You may share your favorite books at GoodReads, your favorite movies at Netflix, and your favorite music on Last.fm.

Flavors.me lets you take all that information and put it together in a single website to serve as your “online identity”. All your publicly shared information, aggregated in one place, and displayed beautifully.

I’ve been running a Flavors.me site for some time that you can see here: http://flavors.me/pixelmonkey

Now, that’s the end product. All the content gets pulled dynamically from your various online feeds. The real magic with Flavors.me is how easy it is to get there. You can drastically change the look and feel of this site using a dynamic, “WYSIWYG” interface. You can do one or two clicks to add a service, reorder it, rename it. Another couple of clicks and you change font sizes, colors, and even the overall layout.

As I was writing this post, I noticed that Flavors.me had added LinkedIn support, which wasn’t available earlier in the beta. So I went ahead and added it. I simply logged in and pulled up the design panel.

From there, I could navigate over to “Services”, click the “LinkedIn” logo, and Flavors.me would guide me through the authorization process to make LinkedIn data available to them for display on my page. You simply get redirected to LinkedIn.com, log in there (if not already logged in), and then get redirected back to the Flavors control panel.

Next, I could simply drag-and-drop the service box in the panel in order to reorder it on the page. Yay, drag-and-drop! There is also in-line editing, and editing the title of the LinkedIn “popout” dynamically updates my profile page in real-time! Nice touch.

Finally, I can go to the “Design” panel to tweak fonts, layout, sizes, etc. Look at how Flavors.me displays fonts. It’s amazing — this is the web, but it’s easier to change my Flavors.me web design font styles than it is to change font styles in a Microsoft Word document!

Now, depending on where, exactly, you store the bits that make up your online identity, you may find yourself disappointed by a lack of this-or-that service. They already support quite a few, and they are adding more in the future, I hear. The one I sorely missed was Delicious, so I could share my bookmarks with the world.

But wait! For those services that still don’t have first-class support, Flavors.me gracefully supports RSS feeds of any variety. I simply popped the RSS feed for my Delicious bookmarks into Flavors.me services panel, and, voila! — my bookmarks are now publicly shared!

It’s this kind of simplicity, design sense, and user-centric approach that makes me love the web as a place to develop, deploy, and use software. So, what are you waiting for? Sign up for Flavors.me today: it’s free!

Andrew Leonard of Salon.com has written an article about switching from Chase to a local community bank, in response to HuffPo’s MoveYourMoney campaign.

I’ve written on this blog multiple times about my frustration with Chase bank, but it’s interesting to see someone with as big a readership as Andrew Leonard writing about it. Are commercial big banks’ days numbered?

I recently re-read Douglas Crockford’s JavaScript: The Good Parts. I have been writing more and more JavaScript lately, especially object-oriented JavaScript plugging into existing frameworks. Re-reading the book has definitely been a useful exercise — I think when I first read it approximately 6 months ago, I didn’t fully understand it. But now, I do.

I also found it very interesting to hear Crockford wax poetic about the virtue of simplicity in all forms of software design. The following passage concludes the book.

When I started thinking about this[...], I wanted to take the subset idea further, to show how to take an existing [product] and make significant improvements to it by making no changes except to exclude the low-value features.

We see a lot of feature-driven product design in which the cost of features is not properly accounted. Features can have a negative value to consumers because they make the products more difficult to understand and use. We are finding that people like products that just work. It turns out that designs that just work are much harder to produce than designs that assemble long lists of features.

Features have a specification cost, a design cost, and a development cost. There is a testing cost and a reliability cost. The more features there are, the more likely one will develop problems or will interact badly with another. In software systems, there is a storage cost, which was becoming negligible, but in mobile applications is becoming significant again. There are ascending performance costs because Moore’s Law doesn’t apply to batteries.

Features have a documentation cost. Every feature adds pages to the manual, increasing training costs. Features that offer value to a minority of users impose a cost on all users. So, in designing products[...], we want to get the core features—the good parts—right because that is where we create most of the value.

We all find the good parts in the products that we use. We value simplicity, and when simplicity isn’t offered to us, we make it ourselves. My microwave oven has tons of features, but the only ones I use are cook and the clock. And setting the clock is a struggle. We cope with the complexity of feature-driven design by finding and sticking with the good parts.

It would be nice if products[...] were designed to have only good parts.

I removed direct references to the main subject of Crockford’s discussion — namely, the JavaScript language itself. The truth is, this advice is much more valuable for the design of all software products. Perhaps one day someone will write the much needed book, Startups: The Good Parts.

I’ll start off this post with a somewhat controversial claim: I invented Dropbox.

I’ll show why this claim doesn’t matter later, but for now, I’ll assure you that it’s true.

How many of you out there use Dropbox? If you don’t, you should — it’s an excellent tool. In its free version, it provides you with 2GB of storage “in the cloud”, using a new kind of folder called a “Dropbox”. What distinguishes a Dropbox from other folders on your computer? The following:

• Every file put in your Dropbox is automatically (and securely) uploaded to Dropbox’s servers, ensuring you have an offsite backup of all data therein.
• Multiple computers can gain access to a Dropbox, ensuring files are automatically synchronized across computers without having to use complication version control systems.
• All files in your Dropbox are versioned, ensuring you can always recover an older version of a file in case you accidentally overwrite a good version.

Dropbox is supported on Windows, Mac OS X, and Linux, and now even has mobile applications, as well. Further, I have a special place in my heart for this service because I started using it almost 2 years ago, and it has acted as a file sharing and project management tool for my own startup’s internal operations at Parse.ly. I was therefore more than ecstatic to discover that this excellent tool and its smart founders had also made it through all of the hurdles necessary to get an early-stage company the financing it needs: they’ve raised over $7 million in financing and have over 3 million users.

But there is another reason I absolutely love Dropbox: because it was my idea. I invented it.

In the summer of 2004, I was really itching to get into Google’s Summer of Code competition. This was the summer I had taken a job working from home as the lead web developer at the Unemployment Action Center of NY. Though the job was great experience — letting me build my first full web application for a real client — I was itching to work on a technically juicy problem, something that affected me in my daily computer use.

And so, I sat down for a day and wrote up a Google Summer of Code proposal for a new system I had invented called Persistent Folders. It wasn’t exactly like Dropbox, but damn close. Even the implementation is close: Dropbox and my system both sync files using rsync, and both use a Python daemon process. The main difference is that since my system was meant to be open source, it did not require the use of a company-maintained service; instead, I proposed that users piggyback existing storage they have via web hosting providers.

Unfortunately, my project wasn’t selected.

Why am I posting this? I recently had a discussion with another engineer after I had discussed some of the technology behind Parse.ly with him. He was surprised at how liberal I was with explaining our internal implementation, architecture, and algorithms. He asked me, “Aren’t you worried that I could steal your idea?”

I responded, “You can steal it all you want; I dare you to try and implement it!” I then explained that to me, ideas don’t matter. I had the idea for a hundred startups that now exist before they started. I know from talking to users and customers of Parse.ly that they had our idea before we implemented it. What matters in software is not an idea, but execution of that idea. Ideas are a dime a dozen.

I began this post with the statement, I invented Dropbox. And now I’m here to tell you that it doesn’t matter one bit, because I never implemented Dropbox. And you can’t own ideas…

If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it. Its peculiar character, too, is that no one possesses the less, because every other possesses the whole of it. He who receives an idea from me, receives instruction himself without lessening mine; as he who lights his taper at mine, receives light without darkening me.
-Thomas Jefferson

Imagine if when I had come up with this idea, I had patented it. Would it really have been fair to force Dropbox to license my technology? To sue them for “infringing” on my patent?

No way — these guys deserve the success they have. How stifling it would have been for me to put any roadblocks in their way because “I had the idea first”! The world is better because Dropbox exists. And, these guys have had — in my opinion — near-flawless execution. So, kudos to them.

Consider the following matrix:

A mediocre idea with a good implementation is worth infinitely more than a carefully-guarded, good idea with no implementation. Of course, the best products are both great ideas and great implementations. And I think my proposal for “Persistent Folders” — written three years before Dropbox even started — proves this to me in a very personal way.

Now, I’m certain the Dropbox guys never read my little proposal, because it was only sent to Google and otherwise sat on my hard drive for years, not viewed by anyone. I had a great idea, but no implementation. And the Dropbox guys took a great idea (one they arrived at on their own, I’m sure), and gave it the implementation it deserved.

For those of you who guard your ideas carefully, I’d suggest you stop wasting your time, get off your butt, and focus on actually building stuff. Because if you don’t, someone else will!

For the curious, below is my proposal to Google Summer of Code 2004, unaltered from its original draft. This is just a relic; for now, I’m glad to keep hacking on my own little idea, hoping one day I can look back and say I executed as well as Dropbox did…

PERSISTENT FOLDERS: A New Metaphor for Data Synchronization
May 13, 2004

The Problem

Computer users are more and more finding themselves with a serious problem: the fragmentation of personal data across multiple physical machines, and even multiple operating systems. Users who find it most comfortable to have a desktop machine at home and a mobile laptop computer “on-the-go” (for business or trips) have to deal with chaotic and often frustrating manual methods to copy that data to the needed places. Some users carry USB memory sticks — which hold their important “working set” of files — and either work directly off those disks (suffering reduced speeds) or make copies of the folders therein on their actual machines, as they become needed, spreading yet more copies and compounding the problem of synchronization. Others abuse ubiquitous technologies for accessing important files, by either e-mailing the files to themselves or by uploading them temporarily to web or FTP servers.

The end result of any of these methods is fragmentation of personal data, with error-prone manual processes for replication, and the inability for a consistent way to search, backup or even just keep track of all the important files and folders spread among the PCs in question.

What is needed to solve this problem, or at least make the problem more manageable, is an intelligent, user-friendly, customizable cross-platform program that allows for transparent synchronization of personal data across multiple computers and systems, either via a LAN or via the Internet.

Introducing the Persistent Folder

Up to this point, most users are used to understanding folders as existing solely in one location. The only way to get a folder’s data onto another machine is to copy that folder, thus making a duplicate. The goal of this project will be to introduce a new high-level data storage mechanism known as a Persistent Folder. Persistent Folders differ from regular folders in that they are meant to be transparently persistent, or synchronized, across different computers.

To explore a hypothetical example, imagine user Joe sits down to do some personal accounting work on Friday. He creates a folder on his desktop called “Personal Accounting,” and begins working on some OpenOffice spreadsheets in that folder. A few minutes later, Joe realizes that some of his accounting work will have to be done while he is away during the weekend, on his laptop computer.

To enable this, he simply right-clicks the folder on his desktop and says “Make this folder persistent across multiple computers.” When he does this, a dialog comes up asking him to type in a description of the persistent folder, and to select computers on which he wishes to make this folder persistent. Joe is presented with a dialog of computers currently available on the LAN. He sees his laptop, “MobileJoe”, and selects it. He then presses OK.

Within seconds, Joe sees the folder “Personal Accounting” appear on his mobile computer’s desktop, even though he hasn’t even touched his mobile computer yet. When he enters that folder, he sees the same spreadsheets that are available on his main PC’s desktop.

These two folders are now treated by Joe as “one persistent folder available across two computers.” He can add files to one folder and they will automatically be propagated to the other. He can modify files and the new versions will then exist in the other. Joe no longer has to worry about pushing his data back and forth across the computers.

Internet Synchronization by way of Ubiquitous Services

One of the major questions one may ask at this point, “that is all well and good for persistence of a folder on a LAN, but what about when I leave my home/office, and need to synchronize over the Internet?”

One approach would be to have a special server application with which all computers that wish to share persistent folders could synchronize. But then the user needs his own server, and needs to install my little server daemon on it. And not many users have their own servers on which they can just install any old application. So for normal users, this, in fact, is no approach at all.

Although most users don’t run their own servers, many modern users do have _server access_. That is, many users do have Web/FTP hosting providers to whom they pay subscriptions, and these servers power their blogs, personal photo stores, etc. The goal of Persistent Folders would be to allow users to piggyback their existing web services to synchronize their files, essentially turning a folder on that server into a Persistent Folder Repository.

A dialog in the properties of a persistent folder would include a checkbox that allows the user to “make this folder available over the Internet.” It would then allow a user to choose a method for making this folder available via the Internet, asking the user to provide a “persistent folder repository” via FTP by default (due to commonality), but equally possible would be scp, rsync, NFS, or even something like cvs/svn (the goal, of course, would be to make the design modular enough that it could support any server type with basic file system operations).

Then, every other visible machine on the LAN who shares that persistent folder would have option enabled automatically. If the machines are offline, the option can be entered manually by the same method explained above.

From that point on, the folders become persistently available and transparently usable, just as before. If the other computers are available on the LAN, then the program utilizes LAN speeds and synchronizes directly. Otherwise, it synchronizes with the Internet Repository.

Lucky Joe is now able to work on his files in the office, leave his laptop there (“the darn thing is so heavy to carry around!”), return home, and continue right on working on those files which are now found in the “same folder” on his desktop machine at home.

Implementation Ideas

I plan on implementing this idea in Python, since one of the major goals of the project is to allow Persistent Folders to exist not just across computers, but also across operating systems (so that a Windows desktop could have a persistent folder that also exists on a Linux laptop, for example). For a user interface, I plan to use PyGTK, since that’s what I know, and since it is also cross-platform. I need something powerful like GTK since there will be times when user intervention will be necessary, but since it is a goal of this project to reduce the number of times the user must intervene, I want to make sure that when he does, he is presented with sane, human-readable, and user-friendly dialogs.

I don’t plan to reinvent the wheel. Most of the magic of Persistent Folders is just making existing synchronization tools work relatively silently and in a way that makes sense with a user’s workflow. The main tool I am thinking of using is rsync, the relatively-ubiquitous UNIX utility for incremental synchronization of directory trees. I considered using the unison, as recommended by Ubuntu’s Wiki entry, but I saw two major problems: (1) unison is no longer under active development and (2) it is written in OCaml, a relatively obscure language which I don’t know. Therefore, any bugs I discover in unison would not be fixed in a timely fashion by its developers, and any features I’d like to add to support my idea would be quite difficult to implement.

As for monitoring folders for changes, I imagine the most elegant solution would be to take advantage of inotify under Linux (like Beagle does) and perhaps handle ChangeNotify events under Windows. I’d really like to avoid polling, since polling is just plain evil for something as potentially neat as this. Regardless, I’ll probably need to code Yet Another Daemon (or Windows service) to keep track of Persistent Folders on the local machine and their equivalents across the network.

Finally, for secure synchronization of files, I plan to use SSH tunneling wherever possible, which is available under Windows under the OpenSSH for Windows Sourceforge project, http://sshwindows.sf.net. Linux distros like Ubuntu, of course, have all the ssh support one needs.

Conclusion

This project aims to introduce a new metaphor users may utilize to share important files across multiple computers: the Persistent Folder. A Persistent Folder is not a shared a folder; rather, it is seen as a single folder that exists locally across multiple computers, and can be treated by the user as such. Changes at any one folder rapidly propagate to the others. Properly implemented, this may provide a better way for users to manage important data that might otherwise be scattered, fragmented, and even lost through the daily shuffle of file transfers across networked PCs.

Appendix A: Project Roadmap

o June 24: Begin Work, with ideas now fully developed in the form of documents. Post these documents to Ubuntu’s Wiki to encourage ideas from community.

o July 5: Have a console synchronization wrapper and some network discovery stuff in Python written, and have UI concepts designed in Glade.

o July 20: Make 0.1 (GUI and basic features working) release, so that Google can show it off at OSCON?

o August 1: Have other great features, like the Internet synchronization and multi-protocol support, in 0.2 release.

o August 10: Consider working on features to allow transparent backup and versioning, to include with bug fixes in a 0.3 release.

o August 20: Make simultaneous 0.4 releases on Linux and
Windows, hammering out as many cross-platform issues as possible. Add inotify/ChangeNotify support if not already there.

o August 30: The big 0.5 release, finished for Google’s deadline. Let Google/Ubuntu make the decision if it’s worthy of being renamed a “1.0″ release.

o September 1: Live a less stressful life, since my files are now neatly synchronized among my PCs! But I’ll keep making it better.

Appendix B: Hold On, Isn’t Samba Good Enough?

One common response to this project may be, “Aren’t Samba and SMB shared folders good enough?” Though Samba is good, I do not believe it is good enough. Here’s why:

(1) Samba does not aim to present the user with a metaphor of a folder existing in multiple locations at the same time. I believe this metaphor would be appreciated as powerful by longtime computer users and subconsciously acknowledged as highly usable by novices.

(2) Samba only allows users to share folders on their drives. Other Samba users may then mount those folders via the LAN. This two-step, asymmetric process already seems complicated and convoluted to end users. But more importantly, synchronization is left up to the user. In theory, users could avoid synchronization altogether and work on the files directly, via the share. In practice, LAN speeds are not adequate, and other issues are raised (such as file locking and write conflicts). This forces users to home-brew their own synchronization protocol to make sure duplicate files at different versions doesn’t result in an accidental loss of data.

(3) Samba does not provide any easy method to access personal files once one leaves the LAN and enters the WAN, short of opening up a bunch of ports on your router and trying to connect in from outside (a very slow and insecure method).

(4) Samba provides no recourse when a computer is no longer available. Persistent Folders, on the other hand, make data available “offline” by design.

(5) Samba does not know whether two duplicate files exist across the network. Therefore, Samba cannot be immediately utilized as a transparent form of backup.

Is the Persistent Folder meant to replace Samba? Of course not. I believe Samba has specific purposes: to allow for easy, fast, one-time transfers of files across a LAN, and to allow networked printer and device sharing. I do not believe Samba is an adequate solution to allow a user to treat a folder as if it existed on multiple computers at once.

Appendix C: Versioning’s the Thing

Upon further contemplation of this project, I realized I had left a question unanswered in this document. What is the right way to deal with version conflicts during the silent synchronization phase among Persistent Folders?

Imagine user Joe works on a file in a Persistent Folder which exists on his laptop and desktop computers. The first version (let’s call it version 0.1) was created on his desktop, and Joe had the laptop on the same network, allowing automatic synchronization to occur. But then Joe’s laptop was disconnected from his desktop, as Joe was without network or even Internet access. He works on the document a bit, which can now be called version 0.2. But then Joe forgets that he worked on the document on his laptop, and when he gets home, he works on his desktop computer, creating a version 0.2 there too. Which version 0.2 should be synchronized?

Normally, the answer would be to pick the most recent one. But we don’t want Joe’s laptop work to be lost, just because Joe forgot about his prior work, do we? Well, what needs to happen is a bit of smart conflict resolution with no data loss.

I propose that Persistent Folders should also be versioned to a sane degree. Perhaps by default 3-5 versions of files are always kept, with the most recent versions visible directly in the persistent folder, and other versions buried (in .dotfiles or hidden folders) behind there. Then, the Persistent Folder monitor should inform Joe (“whisper” to Joe via the notification area/systray) that there was a conflict, and the newest file was chosen. But at any point, Joe can choose to revert a file back to an older version. The interface should be such that Joe can even see where the file was edited, for example:

Personal Accounts.xls – Prior Versions
o version 0.2: edited on MobileJoe
o version 0.2: edited on DesktopJoe
o version 0.1: edited on DesktopJoe

Of course, versioning could be disabled (at risk of data loss to the user), or could be enabled as “smart versioning” to only version files when conflicts occur during synchronization. Some may say that versioning is dangerous because it can drastically increase disk usage, but as I mentioned above, the other benefit is that the user gets redundant network backup of files for free. In higher versions of Persistent Folders, things could get more sophisticated by allowing users to turn off versioning even at the file-level, but I don’t think that’d be necessary in the first few releases.

So, for those not keeping score, Dropbox supports:

• Windows, Linux and Mac OS X syncing, using a scheme similar to that described above, except with a managed set of servers (Dropbox’s) acting as the “repository”
• sync over LAN (in latest beta releases) when PCs are local
• file-level versioning
• Collaboration by sharing folders with other users — a feature I didn’t discuss in my proposal, but that would be a clear next step for a managed service

Go Dropbox!

I recently did some web design work in collaboration with a graphic designer. She introduced me to what has become my latest favorite piece of CSS code: 960.gs.

960.gs is a CSS grid framework, similar in spirit to Blueprint CSS and YUI Grid. However, 960.gs is at once more minimalist than these approaches, and more thorough.

The author has a detailed blog post explaining his motivations for working on 960.gs, so I won’t rehash each of those. Instead, I’ll just dive into what I liked about it.

1. Very simple and comprehensible name choices for CSS classes: You can see their demo page (view its source) for examples, but I can even summarize the approach in a few short sentences. You begin a grid container with a container_XX class, where XX is the number of total grid columns in the container. Every child element has a grid_XX class, which is the number of columns (aka colspan) for that element. All grid_XX elements should add up to the XX amount in the container if you are using all the grid elements available. To use larger gutter widths, you can use prefix_XX and suffix_XX classes. Finally, for any child grid elements, you need to mark the first one with an alpha class and the last one with an omega class — this is necessary to get rid of left/right margins on starting and ending elements.
2. Excellent CSS generator: this online CSS generator tool lets you generate 960.gs CSS that is just right for your website.
3. jQuery-based bookmarklet for double-checking grid layouts: another online tool is a bookmarklet which will “inject” a customizable grid overlay on any existing site, to help with alignment.
4. Templates for every major design tool: I saved the best for last. Nathan’s focus in designing 960.gs was to allow better collaboration between trained graphic designers (whose tools are typically Adobe Photoshop, Fireworks, Illustrator, etc.) and trained web designers (whose tools are things like Rails, Grails, Django). To ease this collaboration, the author provides templates for every major graphic design tool that has “grid overlays” compatible with the 960.gs framework.

Finally, the entire project is hosted on bitbucket and developed in the open. What more could you ask for? This has really simplified my approach to standard CSS designs.

I presented Parse.ly at the NYC Search & Discovery Meetup on Thurs, Oct. 29.  The meetup is organized by Otis Gospodnetic (blog), who is one of the authors of Lucene in Action and the author of the upcoming book, Solr in Action. We make heavy use of Lucene and Solr on Parse.ly, so it was exciting to get an opportunity to present to a community of fellow technologists building systems with these excellent technologies.

In addition to running Parse.ly, I also run a small consulting business, Aleph Point, Inc. In the course of working on client jobs, I sometimes have to make business purchases, which I always pay in full at the end of every month. I have never carried a balance on my credit card and I never intend to.

When I signed up for a business checking account at Chase, the branch manager who I worked with (and who now no longer works there) encouraged me to sign up for a business credit card, as well. I thought, hey, why not — I’m just going to use it for small purchases like monthly hosting fees and the like.

Recently, I made a relatively large purchase at Best Buy for a client, which I was going to be reimbursed for. It was about $200. I already had a balance of $350 on my account, and a few days later my account was closing for the month.

When I looked over my account information a few days later, I found a strange charge. $39 OVERLIMIT FEE. What’s that, I thought?

Well, I went to my branch to find out. The branch manager explained that my credit card had a credit limit of $500. Wow. That’s a low credit limit. I explained that I sometimes make reimbursed purchases of more than $500, so this was quite strange value to pick. Further, I already had multiple other credit cards with much higher limits, so I don’t get why they would choose such a small limit for me. “Oh”, the branch manager said, “that’s because we’re picking $500 limits for all new business customers.” Hmph, fine… seems strange, but fine. (The engineer in me thinks, “Couldn’t they have done an analysis of my cash flow to figure out a more reasonable limit? Of course not — this is a bank, after all. That’s expecting them to be smart.”)

“OK,” I said. “So I have a $500 limit on the account. But why is there an OVERLIMIT FEE? Shouldn’t it just be declined? What is that about?” She says, “Oh, unfortunately, I can’t explain why that was charged. You’ll have to call the number on the back of the card.” I say, “Really? Why?” She says, “Unfortunately, we can’t discuss credit card fees at the branch.” This strikes me as a very strange policy, probably just put in place as a deterrent for people actually contesting their fees. Very sneaky little bastards, these Chase guys. But fine, for now I’ll follow the policy.

A couple days later, I call up the number on the back of the card. I ask about the fee. He says, “Yes, sir, that is a completely valid fee.” I reply, “Valid? Who cares if it is valid? Any fee you guys put on my account is ‘valid’. The question is, why was it put there? And is it justified?” He replied, “It was put there because you went over your $500 limit.” I asked, “Why did you let me go over the $500 limit? If I have a limit, shouldn’t I get TRANSACTION DECLINED when I go over? Isn’t that the whole point of a limit?”

The guy on the phone laughs. Literally, he laughed at me. “No, Chase provides overlimit protection as a convenience to our customers. So that if, for example, you’re taking a client out to dinner, you won’t be embarrassed by going over your limit.”

I didn’t respond for a couple of seconds, because I was parsing his sentence. “So, this isn’t so much a fee, as much as a convenient service Chase is providing me. You guys are saving my embarrassment for the mere cost of $39. I get it now.”

“Yes, sir.”

Apparently, the guy didn’t get my sarcasm. “Here’s what I think,” I continued. “I think you guys are just ripping me off. There is absolutely no reason not to decline the transaction, except that in allowing the transaction to go through, you now assess a fee. It doesn’t cost you anything for me to go $50 over limit, as I did. And I paid down the account in full through my automatic payment system just a few days later. So, I think you guys just figured — hey, here’s an easy way to make free money off our consumers. Here’s another fee we can invent because we are greedy bastards.”

He seemed a little taken aback, and then said, “No, sir. Chase does not rip off its customers. Chase is here to serve its customers.”

“OK,” I say. “I’ve been your customer for more than a decade. So since you’re in the business of serving customers, I suppose you’ll have no problem removing this fee, which is completely unjustified.”

“Unfortunately, sir,” he responds, “I cannot remove the fee.”

“What do you mean? Of course you can remove it,” I say, incredulous. “Just pull up my account and press delete on the line that says, OVERLIMIT FEE.”

“No, sir, I cannot. It is a valid fee. Valid fees cannot be removed.”

“There’s that word again. Here is another example of a valid fee: $500 for transferring money between my checking and savings account on a Wednesday. If you guys had that policy, that would be a valid fee.”

“We don’t have that policy.”

“Listen,” I’m revving now. “It would be one thing if you guys charged $1 for going over my credit limit. But you charge $39. You know how you guys came up with the number $39?”

“This fee is set by the executives of the banks.”

“Wow, I wish I had just recorded that to send to Congress! No, really –”, I interrupt. “I don’t care who came up with the number. But I’ll tell you how $39 was picked. It was picked because after rigorous market testing, they found that if the fee were $40, people would grab their kitchen knife, run out in the street, and kill every fucking Chase banker in sight.”

He was silent.

“So, the executives decided — better make it $39.”

He didn’t laugh this time. No sense of humor. “Listen, just let me speak to a supervisor to get this issue resolved.”

He insists, “Sir, no supervisor can remove this fee. The executives mandate that we cannot remove it.”

“Who can I contact to appeal the fee, AND the policy that does not allow it be removed, AND the appointment of the executive who instituted it?”

“You can write a letter to business services in Delaware,” the representative says.

Frustrated, I end with a rant. It was long-winded and I won’t produce the whole thing here. It talked about my prior blog post, which pointed out how Chase ran an insecure document exchange system. It talked about how there are currently 21 Chase customers who are “mad as hell and not gonna take it anymore”, ranting on my site about how crappy Chase is. And I discussed how even after contacting Chase about this issue, even after pointing them to a significant problem with numerous angry customers, they do nothing. Just like they are doing nothing now. A company that is like a black hole.

“And as a systems engineer,” I conclude, “the fact that you guys can’t remove this fee just makes me so god-damn depressed, I can’t even express it to you. We have managed to put a man on the moon, but JPMorgan Chase bank cannot hit delete on a fucking spreadsheet. How pathetic is that?”

I resignedly ask for the complaint address. I write it down. And I end the conversation with, “Listen, I know you’re just doing your job. But I want to make it clear to you that your company does not deserve a single god-damn dime of my money. Neither the money in my personal checking account, nor in my business checking account, nor any of the money my representatives awarded you in that $25 billion bailout that saved your company’s greedy ass. It’s not your fault — it’s your company’s fault. But for the love of everything good and just in this world, man, why the hell do you work for these clowns?”

That bit of humanity connected with him, just a little. I could hear it in his voice. “I hope things turnaround for you with Chase. I really do. But there’s nothing I can do for you at this time.”

I hope things turn around, too. In the meanwhile, I’ll sharpen my kitchen knife.

A few days ago, I got a call from my girlfriend, Olivia. I was so deep in working on my startup, Parse.ly, that I hadn’t checked my bank account statements in several weeks. We just went into private beta last Thursday, after DreamIt Demo Day. She noticed some suspicious charges, and so I looked into them. Indeed, it looked like I had been a victim of fraud: there were three charges that clearly was not me.

I immediately called Chase Customer Service. In order to confirm the details about my account, the representative needed me to identify the fraudulent charges, but also identify charges that were actually valid. For this latter bit, I needed to identify the time/place of a specific transaction. This card was mostly used for online auto bill payments, so this turned out to be impossible for any of my last 20 valid payments. Yet the customer service rep insisted that I name a time and place. I told her, “The time and place was whenever the server for this system decided to automatically bill my account. I don’t know where their server is, I don’t know what time their cron jobs run.”

“Cron jobs?” she said.

Right, I had been hanging around techies at DreamIt Ventures for too long. “Listen, the transaction didn’t take place physically, it took place digitally. I can identify one transaction, which is about a month old, where I actually used the card in-person to buy something.” She finally understood and let me move on.

Burak from Trendsta said he felt bad for me, for how patient I had to be with this person. But that was the least of it. This little technical misunderstanding was nothing compared to what followed.

I was told that in order to get a credit back from my account, they had to collect from me a signed affidavit indicating the charges were fraudulent. This affadavit would be “securely shared” with me via e-mail. OK, “sounds good” I said. I waited around for the e-mail to come in.

Finally, two e-mails arrived in my inbox. The important bits are in red. First:

Message from Chase Customer Claims Secure Document Exchange

From: chase_customer_claims@secure-dx.com

Welcome to the Chase Customer Claims Secure Document Exchange. You recently contacted Chase regarding your claim number XXXX. Your documents are available for your review.

Per our telephone conversation, you will need to register to our secure website.

Your initial password is: password

Your initial user name has been sent to you in a separate email.

On your first log in, you will be required to select a new password.

Thank you for using Chase Customer Claims Secure Document Exchange.

To contact Chase for claim related questions or to withdraw your claim, please call 1-866-564-2262.

Any geek reading this will immediately identify some key things wrong with this e-mail that make it look like a total phishing expedition. Namely:

1. The e-mail address, rather than being from a chase.com domain, was from a strange domain named “secure-dx.com”.
2. Rather than sending a cryptographically secure, expiring activation link, a default password was sent in plain text.
3. To make matters worse, the password is the same for all users, and thus anyone who can guess my e-mail address can easily impersonate me on this “secure document” website.
4. The default password is “password”. WTF?! I mean, c’mon?

I didn’t quite understand why I needed a “second e-mail” now, but I opened it up. Here it is, excerpted:

Your Chase Customer Claims Secure Document Exchange Electronic Package is available online

From: chase_customer_claims@secure-dx.com

ANDREW MONTALENTI,

Welcome to the Chase Customer Claims Secure Document Exchange.You recently contacted Chase regarding your claim number XXXX. Your documents are available for your review.

Per our telephone conversation, you will need to register to our secure website by clicking on the link below or copy and paste the link into your browser’s address bar.

https://chase.secure-dx.com/consumerdcx-chase_atm

Your user name is my.email@hidden.com

Your initial password has been sent to you in a separate email

On your first log in, you will be required to select a new password. NOTE: This site is different from Chase.com and passwords are not related. Updating your password on Chase Customer Claims Secure Document Exchange will have no impact on established Chase.com passwords.

Once registered, you will be able to access your customer correspondence on our secure website. You may be offered the option to complete and sign the form online if you wish to do so. [...]

To say I was confused would be a major understatement. I was downright depressed.

My guess is that the engineers at Chase thought that by separating the “password e-mail” from the “user e-mail”, that somehow made the whole communication more secure. Two e-mails are better than one, right?

The most important thing to point to is the link. The link where this secure communication will happen is not at the chase.com domain Instead, it is at https://chase.secure-dx.com/consumerdcx-chase_atm. There is no way, NO WAY this is a real Chase site, I think.

I click on the link and in Firefox, I see this:

At this point, my paranoid self turns on. Curious, I click through the link anyway. And I see this:

Now I’m really paranoid. Links off secure-dx.com pointing back to chase.com’s privacy policy. A username and password box and a sort of hokey imitation of the Chase.com web design. I realize, holy shit, I’m being duped! Not just small-time credit card fraud, but someone has managed to really take over my life!

Why am I freaking out? The customer service person I talked to, I realize what must have happened. That wasn’t Chase. Someone stole my credit card information and then set up a call forwarding on my cell phone, somehow, to point Chase’s customer service number to some fraudulent interceptor. This person then diligently took my claim only to send me an e-mail that would get yet more information out of me and take me for even more money. I freaked!

Immediately, I double-checked my call logs and compared them to Chase.com customer service numbers. I made sure to change my DNS server to OpenDNS to make sure no one was somehow intercepting that. Finally, I realized I could look at the number written on the back of my Chase credit cards. It all checked out — the number was good. So I switched phone. I called Chase customer service on both my phone and Olivia’s. I made sure the messages were exactly the same. From Olivia’s phone, I called back Chase again to speak to someone there about this. But then I got even more paranoid — how big could this be? — so I decided to hang up. Instead, I called my local Chase branch in my neighborhood.

With my local branch’s help, I got transferred via a branch office line to the actual Chase customer service. Finally on a secure line, I thought to myself. When they picked up, I was expecting to uncover the scam of the century. I felt like an investigative journalist right on the tail of something truly big.

But then I spoke to the Chase representative, on the secure line, and she explained to me that this is just the normal procedure. secure-dx.com is the website they use for “securely” sharing documents.

I was livid. I explained everything wrong with this setup. I demanded to speak to a supervisor. I spoke to a supervisor. He said he did not know why the system was the way it was. He wasn’t a software guy. He just knew that “with the way the business is changing lately, a lot of systems are in flux.” I said this flux was unacceptable. “I’m a software engineer,” I said. “How can I possibly trust Chase to manage my financial accounts if something as simple as sharing a PDF document is done in the least secure way possible?” What other skeletons might they have in the closet?

I wanted to be forwarded to the department responsible for that. After my explanation to him of what was wrong, he fully understood the problem. To his credit, he admitted it was wrong the way it was set up. He actually tried to track down a supervisor. But there was none that could field IT and software requests.

They promised to call me once they could track someone down to talk about this. No call yet.

My excitement came down a couple of notches. I was not the investigative journalist undercovering an elaborate scam any longer. Instead, I was a software engineer. And some members of my profession have let me down. Big time.

In the meanwhile, I did the research and found the vendor who provided this service to Chase. They are Wolters Kluwer, a “financial services and banking compliance solutions provider”. The product page for “SDX”, Secure Document Exchange, is completely ludicrous. They claim this product includes “industry-leading security, including PKI encryption and multi-level user authentication, to keep communications safe at every step of the process.”

Right, so the password was sent in plain text. The default password is “password”. And, rather than having a chase.com subdomain which points at Wolters Kluwer’s server (e.g. secure-dx.chase.com) and sharing a secure chase.com certificate with them, they decide to host the whole thing outside of the chase.com domain, so that as a user, I have no way of confirming this actually is an e-mail or system originating from Chase. Users are so confused by this that they have already reported it as a phishing scam, even though it is not one.

That’s industry-leading? That’s “safe communication”?

No, that’s a joke. Chase should be ashamed.

This post is only intended for those who actually run GNOME and Linux, just a warning

Will a new, national insurance plan solve the essential problem of the rising cost of health care?  According to Atul Gawande, it won't.  What is needed is nothing short of a complete cultural shift in the community of practicing medical doctors and the organizations/institutions that provide care.  From the article:

Providing health care is like building a house. The task requires experts, expensive equipment and materials, and a huge amount of coördination. Imagine that, instead of paying a contractor to pull a team together and keep them on track, you paid an electrician for every outlet he recommends, a plumber for every faucet, and a carpenter for every cabinet. Would you be surprised if you got a house with a thousand outlets, faucets, and cabinets, at three times the cost you expected, and the whole thing fell apart a couple of years later? Getting the country’s best electrician on the job (he trained at Harvard, somebody tells you) isn’t going to solve this problem. Nor will changing the person who writes him the check.

After my recent upgrade to Pidgin 2.5.5 (on Ubuntu Jaunty), GTalk mysteriously stopped working.  Check out the FAQ entry on the Pidgin developer website for an explanation.  The workaround, not listed there, is to change your "Connect Server" to "talk.google.com".  Pidgin will then prompt you once for a certificate, and after that, it will connect fine.

Sorry for excessive metaphors related to trees, but it just seemed so fitting.

You see, for almost a year, Sachin (the other founder of Cog Tree) and I have spent every moment of our free time to the path of starting this company.  We felt quite nomadic during that time — …