Bingo Card Creator (and etcetera) Year In Review 2011

I’m Patrick McKenzie (patio11 on the Internets) and for the last several years I’ve run a small software company.  My first product was Bingo Card Creator, my current product focus is Appointment Reminder, and I do occasional consulting for a variety of clients, mostly on helping them sell more of their software over the Internet.

Traditionally, right before Christmas every year I release an annual report.  See, for example, 2006, 2007, 2008, 2009, and 2010.  (Crikey, have I really been doing this for that long?)  I’ve also traditionally published live stats for Bingo Card Creator, but not my other lines of business.

Writing the annual report is partially to keep me grounded, partially to talk through my thoughts on the year and goals for next year, and partially to (hopefully) give other folks ideas that they can use in their own businesses.  I hope you find it interesting or, at the very least, mildly amusing.

Obligatory disclaimers: Assume any statistics that I give are “roughly accurate, to the best of my knowledge, at the time this report was written.”  There are still a few weeks left in the year.  Sales are typically low in the last two weeks, but the exact timing of credit card charges can cause a bit of jitter in the December stats.  From past experience, I have a high degree of certainty that there are about $1,000 or $2,000 of expenses (across all lines of business) which aren’t in the bookkeeping  system yet and won’t be until I sit down in March and check things for taxes.

Capsule summary: Best year ever, by a lot.  Broke $100,000 in sales for the first time and increased total profits to ~$70k.  2012 has inflection points coming for life and the business.

The Year In Brief

I put Bingo Card Creator into maintenance mode for approximately 48 weeks out of 2011: I only answered emails and kept systems running, but took no action to improve the product or marketing.  (The other four weeks I tried a few minor things out.)  This was, theoretically, supposed to free me to spend most of my efforts on Appointment Reminder…

… but that didn’t end up happening.  For a variety of reasons, most of my focus business-wise went into consulting.  Although I technically only did about 10 weeks of consulting during the year, I spent quite a bit of overhead time on e.g. arranging deals which ended up falling through, arranging the deals which did actually go through, and doing general promotion activities like speaking at conferences.  (I had the opportunity to speak at about a half dozen conferences this year, and assorted other events.  It is great fun, but since I generally have to fly to America for them, they tend to munch a full week out of my schedule each.  I spent almost three months of the year in the US, doing a combination of family events, consulting, prospecting, speaking, and meeting some Internet buddies to discuss plans for later.)

I also lost two solid months due to dealing with legal issues, mostly centering around Immigration.  I’d love to fill you in on the nitty-gritty, but have been asked not to by people close to the situation.  Suffice it to say that I was a shoe-in for a Japanese visa back when I worked at a large megacorp, was not a shoe-in for a visa when doing my own thing, and had a very hairy experience with getting them to approve me as a “self-employed engineering consultant.”  Tips of the hat to my Japanese clients, particularly Makeleaps / Webnet IT and myGengo, whose support was instrumental in getting Immigration to approve my renewal.

Despite not having nearly as much time to work on Appointment Reminder as I would have liked, I did manage to firm up its technical underpinnings, add new features requested by clients, and do a small amount of work marketing it.  I hope to make that more of my focus in 2012.

Bingo Card Creator

Despite being in maintenance mode, BCC continued performing like a trooper.  People always ask “Could you afford to live on it only?” and the answer is “Yes, but barely, and it would require a lifestyle adjustment, mostly in the don’t-fly-across-the-Pacific-so-often department.”  BCC did not meet the numeric goals that I had for the year.

Stats:

Sales: 1,539 (up 6% from last year’s 1,451)

Refunds: 14 (down from 22 last year, to .9% of sales from 1.5%)

Sales Net Of Refunds: $45,479.93 (up 5% from $43,398.55)

Expenses: $22,560.00 (up from $18,287.93, but largely just due to an accounting issue — I can’t split costs in my homegrown bookkeeping software, so the ~$3,000 I paid for servers for AR is hiding in that number)

Profits: $22,919.93 (see above accounting issue, essentially flat from last year’s $25,904.66)

Wage per hour: Let’s see, ~15 hours of programming, 20 minutes a week on customer support…  about $700 an hour.  Not too bad.

 

Web Stats:

(All stats are from bingocardcreator.com unless otherwise specified.)

Visits: 821k (up from 777k)

Unique visitors: 670k (up from 655k)

Page views: 2.9 million (up from 2.7 million)

Traffic sources of note: Google (46%), AdWords (18%), Binghoo (13%)

Trial signups for online version: 82,000 (up from 72,000)

Approximate online trial to purchase conversion rate: 1.8%

 

Narrative Version:

Aside from kicking up AdWords spend modestly (to no good effect) and running a few A/B tests, nothing really substantial happened with Bingo Card Creator this year.  I lost probably $1,000 to $2,000 of sales when the site crashed right during the middle of the Halloween rush for ~9 hours while I was on an airplane.  That was a little disappointing, but while it broke my candy budget it won’t exactly put me in the poorhouse.

Projections that BCC would continue to grow despite not being actively worked on turned out to be totally wrong.  I forecast 50% growth, reasoning “Hey, most of the systems work pretty much without my intervention, so I think the overall growth of the Internet plus a few A/B test means, oh, 50% or so.”  It mostly tread water.  I’m not hugely disappointed.

 

What Went Right:

  • Not having to work hardly at all for it.
  • Aside from the Halloween crash, the system was largely stable for the year.  I think I got woken up by the automated alarm maybe once.
  • SEO, AdWords, email marketing, and the usual scalable marketing stuff continued to be my bread and butter even when I was too lazy to actually cut and butter bread.

What Didn’t Work So Well:

  • Crashing on the third busiest day of the year, in such a way that it depresses my AdWords campaigns for the first and second busiest days of the year.
  • I integrated Stripe and expected a huge lift in conversions for going from Paypal to a simple CC-based payment system.  I tested this extensively in A/B tests.  I love everything about the Stripe system, but I have no evidence for “Stripe is better than Paypal/Google Checkout”, “Stripe/Paypal/Google Checkout is better than Paypal / Google Checkout”, etc etc.  That said, it might be something as simple as my buttons being ugly.  I’ll probably take a whack at it in the future, or better yet, have my designer take a whack at it.

Consulting

I did a few weeks of consulting this year, for several different clients.  Mostly, I do my engineering / marketing shtick for software companies, although some of my clients have been a wee bit farther afield.  I wrote up a fairly typical engagement with Fog Creek.  That one was a mutual success and we’ll continue to work together in the future.  (To the best of my knowledge, all of my consulting clients are happy with my work.)

One thing I’m going to do differently in the future is to work for less clients.  Don’t get me wrong: I love all my clients.  I was privileged to work with them.  However, it takes approximately X units of work to set up an engagement with a previous satisfied customer, 5X units of work to get a new prospect to the go/don’t-go decision on a new engagement, and I generally have to get three to four prospects to that point to actually wind up with a signed contract.  As my buddy Thomas at Matasano says, “That is life in the big leagues.”  However, since I’m not in a position where 100% utilization is a huge overriding goal of mine, I don’t need to keep the new prospect pipeline totally full… so I’m probably going to cut back on it quite a bit in 2012.  I’ll continue doing follow-up engagements for established clients where it makes mutual sense to do so, and I’m still of course available for interesting projects, but I’m not going to be doing six-week fly-across-America-four-times tours to drum up new business.

The following numbers are approximations only.  NDAs and having the sense God gave a tadpole constrain me from revealing my “going rate.”

Consulting sales: $55,000

Consulting expenses: $13,000  (mostly hotels and airfare for prospecting, which I pay for out of pocket.)

What Went Right:

  • Client selection.  I was, again, privileged to work for people who have interesting businesses, problems that I could make substantial contributions on, and the willingness and ability to pay all invoices in a timely fashion.
  • Raising rates.  My first guesstimate at my rate, back in 2010, was $X.  It turns out that I could do just about as much work as I wanted regardless of whether I charged $X, $2X, or $5X.  As a result, I typically quote fairly high rates and mostly stick with them, unless there is another reason I really, really want an engagement to happen.

What Didn’t Work So Well:

  • Disorganization.  At one point I was juggling something like five simultaneous proposals out while preparing for three conferences, two engagements, and six weeks of travel.  It got so bad that I showed up at a city once and checked at the airport for where I was staying, quickly seeing that I mistimed a conference by three days and thus had no hotel booked, booking a hotel from the taxi, and then arriving at the hotel to recheck my schedule and discover that I had used the previous year’s schedule and was actually simultaneously at a different hotel across Brooklyn.  (Shoutout to the Brooklyn Beta guys for saving me from my own stupidity that week.)  There were multiple points in the year where I found myself wishing for either a boss or a secretary or somebody to just say “Show up to X on Monday and Do Stuff and all the stuff that is not Stuff will be taken care of.”  My occasional slipups in dealing with the demands of a growing business caused me to drop balls in ways that were sometimes client-visible, too.  This is a major part of the motivation for cutting back next year.  (There is Plan B, of course: hire folks to do either the execution or the admin and take whichever part they’re not doing, but I don’t think I’m moving in that direction.)
  • Too much work!  Largely due to overhead and travel, plus the outsize distraction generated by the same, consulting munched a heck of a lot more time than I thought it was going to.  I wanted to have a solid eight months of the year to work on AR.  I think I probably got maybe two.

 

Appointment Reminder

I launched Appointment Reminder last December, with the goal of having approximately 200 customers and $10k in monthly recurring revenue by now.  I had planned on focusing for most of 2011 on marketing and selling it to more businesses.  That largely didn’t happen, but since I got the fundamentals of my SEO strategy in place (while largely ignoring the modestly more advanced content creation / etc that runs BCC and that I usually help clients with), the business grew despite my best efforts at totally neglecting it to focus on consulting and not getting deported.

AR has been hanging around at a crossroads for a while now.  There are two very different trajectories it could go down.  In one, I grow it organically, and it grows into a modestly profitable software business which will provide handsomely for my family and (in the fairly near future) employees.  In two, I take outside investment, and attempt to grow as quickly as possible to $N million a year in revenue, at which point options would include either a) selling to one of the larger players in the small business software space or b) continued operations at scale with a focus on growth.  Luckily, I have  the luxury of waiting on making that decision: my runway is infinite, the market opportunity is only getting bigger, and the perceived value of my involvement with a startup among investors does not appear to be depreciating.

This is one of the reasons I can’t be as open as I would like to be about the current status of the business.  BCC has essentially no secrets, and would not really benefit from having them, as — aside from elementary school English teachers — there is nobody out there who has something I want for BCC.  However, if I hypothetically wanted to take investment, then accredited investors suddenly have something I want very much and having secrets about AR gives me something with which to trade to get it.  (It is similar to not putting prices on an Enterprise Software website.  You can trivially get them, but the price of getting them is giving a salesman permission to give you the spiel.  Similarly, folks who ask about AR’s numbers these days are generally asking in the hopes that they eventually receive a phone call asking them for a check.)

The other reason I can’t talk about AR numbers so much is that I radically underestimated how important the enterprise market would be to the business, and you can’t spell enterprise without NDA.

So: I wanted to have two hundred customers by now.  For the publicly available plans, I currently have a few dozen paying customers.  There are ways to get things from me that don’t involve paying the numbers on the Pricing page.

AR is modestly profitable — it covers all of its own costs.  I plow most of the money it generates back into the business, though, rather than taking distributions.  For example, I’m now about 95% certain that I will have significant contractor or employee involvement on it in 2012.

Revenue: Undisclosed

Expenses: Undisclosed (very modest ongoing expenses, reinvested most profits)

Profits: I took about $5k just to have a number that would minimize disbelief at the tax office.

What Worked Right:

  • Twilio.  The Twilio API and service have been unalloyed epic wins for Appointment Reminder.  I had zero disruptions in service attributable to them, their customer support has been fast, responsive, and technically savvy (even helping me debug my own code at points), and they’ve been very supportive of me.  Plus they have these awesome red track jackets that they keep sending me, which you’ve probably seen if you’ve seen a picture of me doing any talk this year.  (I actually wear them mostly because I love the color red, but apparently I wear them so often that folks at the Fog Creek office thought the Twilio logo was my logo.)
  • Sendgrid: It’s like Twilio, except for email.  Great service.  No red jackets.
  • Unit testing & staging servers.  I am gradually getting more sophisticated in my engineering practices, and have been ramping up my testing activities since starting to code AR.  It has transformed the way that I do development, for the better, and made it easier to respond to customer requests to change things while decreasing the number of problems I have caused.  Total win.  See my presentation at TwilioConf for examples of the specific ways I use it for AR.
  • Exact match domain names.  ”Hey Patrick, how is it that with no marketing budget and nearly no marketing work you rank #1 for [appointment reminder]?”  I told everybody that I was buying the .org specifically because that would happen but apparently folks didn’t believe me.
  • Using the self-service site as lead generation for enterprise sales.  Fairly self explanatory.
  • The service itself: AR solves a clear customer need, and my customers are raving fans of it.  There exist many services businesses which incur hundreds in direct costs and thousands in forgone revenue for a single missed appointment.  (Think, say, an HVAC company which sends a three-man team of tradesmen out to your house to replace your heater, which is a $2,000+ job, only to discover that you aren’t home to let them in.)  One of my customers reports that just the delta in no-shows since starting to use AR would pay for his mortgage and his daughter’s college education.  Many of my other customers report that their office managers, who previously did telephone reminder calls manually, are ecstatic to not have to do them any more.  Customer retention among folks who actually use the system (as opposed to signing up, doing a test call, and forgetting about it) is virtually 100%.
  • Talking to smart people for advice: Since I’ve been going back and forth on the investment question, I talked to a lot of entrepreneurs and investors whose opinions I respect.  I really appreciate their feedback, which ranged from “Are you kidding?  You’d hate it.” to “I want to invest in you, but realistically, you would lose nothing by waiting until you are sure.” to “Best decision I ever made.” and helpfully included a lot of actionable advice on how to do things in the meanwhile such that options remain open.

What Didn’t Work So Well:

  • Catastrophic engineering failures.  I had one combination outage/catastrophic failure in February (the details are recounted in that TwilioConf presentation) and a ~3 day period of sporadically degraded operations after my move to Rackspace, which I finalized over the Thanksgiving holiday.  Both of those were my fault, for architecting the system in a way which did not gracefully handle its multiple moving parts getting out-of-sync with each other.  I’ve since done significant work on making it more stable.  (Overall reliability for the year has been excellent, but those periods were easily the most stressed I’ve ever been about any business issue.)
  • Lack of focus: I’ve been commenting above on this, so I won’t belabor the issue, but I really didn’t get to work on AR as much as I wanted.
  • Enterprise sales: I’m actually fairly decent at Enterprise Sales, and am working with someone in the industry who has a deep Rolodex among folks who would be great candidates for AR, but (partly due to the focus issue and partly due to my own comfort level) I didn’t put nearly enough effort towards it this year.  What I should honestly do is go to a conference some time, prospect like a madman, and then make following up on those leads my only job until I’ve got contracts signed.  (The prices for enterprise SaaS make this very economically viable.)

Goals For 2012

Bingo Card Creator

  • I’d be happy with continued flatness ($~30k profits on $50k sales), maybe.  It isn’t the source of growth for my business anymore.
  • Continue using it as a laboratory for weird ideas I have on conversion optimization.
  • Don’t break it during Halloween.

Consulting

  • Do less work prospecting for new clients.
  • Do more work for existing clients.
  • Modestly increase billings, if that makes sense for where my overall business is.  (If I take external investment in AR, that will likely require shuttering the consulting business.)

Appointment Reminder

  • Figure out whether I want to take investment or not.  If so, do so.
  • Convince Keith (who I do my podcast with) to work with me, if possible.  (Don’t worry, he knows this is on the agenda.  We’re best friends.)
  • See about transferring responsibility for the engineering (particularly front-end) side of things so I can focus on marketing/sales.
  • 10x current sales numbers.  That seems to be a fairly safe bet regardless of whether I shoot for a small business or for a high-growth business.  (1,000x-ing would be another story.)

A personal note: The last 3,300 words ultimately matter much, much less than the next 3: she said yes.  We’re announcing to our family on Christmas, as per our family tradition.

What Now? Episode 4 – “Your Market is Not Worthy”

What Now? Episode 4 - ”Your Market is Not Worthy”

Download this episode, in which Gavin and Andrey talk about free app promotions, joining the Christmas app sale rush, Android app sale progress, comparing app store approval processes, Communist supermarkets, NOOK troubles, silly questionnaires, Andrey’s new app idea, single platform products, Ludum Dare, making games in 48 hours, trying out Moai, family businesses, learning to write code from scratch, choosing your market, developer management, good and bad sales days, android game dev videos, and the joys of recurring revenue.

Startups and Relationships: Can you have both?

Brad Feld thinks so, and he’s ready, willing and able to explain to you what has and has not worked so far in his 18 year marriage with his wife, Amy Batchelor. Pat and I interviewed Brad – noted VC, blogger, early stage investor and cofounder of TechStars – for The Startup Success Podcast (Just posted as Show 127).

If you want to build a great startup, but have no wish to fail your wife, significant other, or yourself, this is one show you should listen to.

You’re reading Startups and Relationships: Can you have both? from: 47 Hats. If you like this post, there’s plenty more! Want more sales for your startup? Stop by and let’s chat, or consider a Microconsult with Bob Walsh.

Show #127: Brad Feld on the Startup Marriage

Brad Feld

Brad Feld

In this show Bob and Pat talk with tech luminary Brad Feld about one of the most important and least discussed aspects of successful entrepreneurism: how do you build a startup and not burn down your marriage?

In this very candid conversation, Brad shares with us tips, practical techniques, frameworks of communication and insights on how you as an entrepreneur can deal with the unique personal costs, challenges and frustrations your passion bring to the key personal relationships in your life.

Brad, with his wife Amy Batchelor, are gathering material for a book they are co-authoring, “Startup Marriage: Balancing Entrepreneurship And Relationship.”

Whether you’ve been married for years, just dating, or have zero personal life, Brad is an enormously successful early stage investor, startup founder, blogger, and incubator creator: you will want to hear what he has to say.

Want 2012 to be the breakout year for your startup? Why not do a MicroConsult with Bob Walsh? Instead of hypotheticals and too much information, Bob will work with you for an hour via Skype developing 8 to 10 specific todos that will get your startup in gear. Details at 47hats.com.

Play it now!

Download Show #127 here: Show #127 Or if you prefer, Subscribe to the podcast in Apple iTunes.

Bob Walsh blogs at 47Hats, is on Twitter at @bobwalsh or you can email him at bob.walsh@47hats.com.

Patrick Foley blogs at PatrickFoley.com, is on Twitter at @patrickfoley or you can email him at patrick.foley@microsoft.com

Relevant Links:

Brad Feld Thoughts.
Startup Marriage: Balancing Entrepreneurship And Relationship..
About Brad Feld.
Foundry Group.
TechStars.

Post to Twitter

Productizing Twilio Applications

This post includes video, slides, and a full-text writeup. I recommend bookmarking it if you’re on an iPhone right now.

I make extensive use of Twilio (a platform company that lets you do telephony with an API) in running Appointment Reminder, my core product focus at the moment.  (Wait around a day or two and I’ll tell you a bit about how it is doing in my annual end-of-the-year wrapup.)

Twilio has a very passionate developer community and fairly good documentation on their website, but I’ve sometimes been frustrated at it, because it teaches you the bare minimum to get phones ringing.  That is truly a wonderful thing and a necessary step to building a telephony application.  However, if you continue developing your application in the way that the Quick Start guides suggest, you will routinely run into problems regarding testing your code, maintaining it, securing it, and generally providing the best possible experience to your customers and the people they are calling.

I have a wee bit more than a year of practical experience with a Twilio application in production, so I went to TwilioConf and did a presentation about how to “productize” Twilio applications: to take them past the “cool weekend hack” stage and make them production-ready.  Twilio has graciously released videos of many of the presentations at TwilioConf, so I thought I’d write up my presentation for the benefit of folks who were not at the conference.

The Video (~30 Minutes)

Twilio Conference 2011: Patrick McKenzie – Productizing Twilio Apps from Twilio on Vimeo.

The Presentation (40 Slides)

The Writeup

 

Why I Think Twilio Will Take Over The World

(This was not actually in the presentation, because I didn’t have enough time for it, but I sincerely believe it and want to publish it somewhere.)

I think Twilio is, far and away, the most exciting technology I’ve ever worked with.  The world needs cat photos, local coupons, and mobifotosocial games, too, but it needs good telephony systems a lot more, as evidenced by companies paying billions of dollars for them.

Additionally, Twilio is the nascent, embryonic form of the first Internet that a billion people are going to have access to, because Twilio turns every phone into a smartphone.  The end-game for Zynga’s take-over-the-world vision is the human race slaved to artificial dopamine treadmills.  The endgame for Twilio’s vision is that every $2 handset in Africa is the moral equivalent of an iPhone.  I know which future I want to support.

Smartphones aren’t smart because of anything on the phones themselves, they’re smart because they speak HTTP and thus get always-on access to a universe of applications which are improving constantly.  Twilio radically reduces the amount of hardware support a phone needs to speak HTTP — it retroactively upgrades every phone in the world to do so.  After that, all you need is the application logic.  And what application logic there is — because the applications live on web servers, they have access to all the wonderful infrastructure built to run the Internet, from APIs that let you get Highly Consequential Data like e.g. weather reports or stock prices or whatever, to easy integration with systems which were never built to have a phone operating as part of them.

You can’t swing a stick in a business without hitting a problem which a phone application makes great sense for.  I filled up three pages of a notebook with them in just a week after being exposed to Twilio for the first time.  Order status checking for phone/fax/mail orders.  Integrated CRMs for phone customer service representatives.  Flight information.  Bank balances.  Server monitoring.  Appointment reminders.  Restaurant reservations.  Local search.  Loyalty programs.  Time card systems.  Retail/service employee support systems.  Shift management.  The list goes on and on and on.

Seriously, start writing Twilio apps.

What This Presentation Will Actually Cover

I’m tremendously optimistic about the futures of Twilio and the eventual futures of companies which make Twilio applications, but I’m pessimistic about your immediate future as an engineer writing a Twilio app, because it is going to be filled with pain.  You’re probably going to make some choices which will cause you and your customers intense amounts of suffering.  I’ve already done several of them, so use me as the inoculatory cowpox and avoid dying.

Crying In A Cold, Dark Room

Back in February 2011, I moved from my previous apartment to my current house.  I unwisely decided to push a trivial code change prior to boxing things up.  This trivial code change did not immediately take down the server, but did cause one component (queue worker processes) to fail some hours later.  The most immediate consequence of this was that outgoing appointment reminder phone calls / SMSes / emails failed to go out.  Since I was busy moving, I did not notice the automated messages about this failure.

When I discovered the failure (8 hours into customer-visible downtime), I panicked.  Rather than reasoning through what had happened, I reverted the code change and pushed reset on the queue worker processes.  This worked, and the queue quickly went from 2,000 pending jobs to 0 pending jobs.  I then went to bed.

At roughly 3 AM, I woke up with a vague feeling of unease about how I had handled the situation, and checked my email.  My customers (small businesses using AR to talk to their clients) had left several incensed messages about how their client had reported receiving dozens of unceasing calls on the behalf of their business, in a row, at 7:30 PM the night before (right after I had restarted the queue workers).

Here was the error: my application assumed that the queue would always be almost clear, because queue workers operate continuously.  A cron job was checking the DB every 5 minutes to see whether a particular client had been contacted about her appointment yet.  If she hadn’t, the cron job pushed another job to the queue to make the phone call / SMS / email.  When the queue came back up, each client received approximately ~100 queued events simultaneously.  These did not themselves check, at the start of the job, whether the job was still valid, because the application assumed that the cron job would only schedule valid reminder requests and not execute 100 times in between queue clearings.

This resulted in approximately 15 people receiving a total of 600 phone calls, 400 SMSes, and 200 emails, in approximately a 5 minute period of time.

There are a variety of ways I could have avoided causing this problem for my customers:

  1. Don’t make code changes prior to planned unavailability, even if they look trivial.
  2. Don’t ever leave your phone that gets emergency messages out of your pocket.
  3. Switch to idempotent queues, so that adding the same job multiple times does not result in multiple copies of the job.
  4. Add per-client rate limits, so that application logic errors don’t cause runaway contact attempts.
  5. Add failsafes for historically unprecedented levels of activity, shutting down the system until I could manually check it for correctness.

Testing Twilio Applications

Unit testing and integration testing are virtually required to produce production-quality Twilio applications, and will make it much less likely for you to create catastrophically bad bugs in production.  Unfortunately, testing Twilio applications is much harder than testing traditional CRUD web applications, because of how TWIML is different than HTML (in terms of how minor syntax errors actually cause business problems), how it is not easy to replicate telephone operation in integration testing, and because Twilio sometimes has poor separation of concerns between the MVC of a web application, the Twilio helper library, and the Twilio service itself.

Twilio testing is inherently dangerous, because non-production environments (testing, staging, development, etc) could conceivably generate actual, real-world phone calls to phone numbers which were in your database but not actually under your control.  The first and most important tip I have for Twilio testing is to make it explicitly impossible to contact anyone not on a whitelist from code when you’re not in production.  I have a quick snippet that I put in a Rails initializer which monkeypatches my Twilio library to force it to only make phone calls or SMSes to whitelisted numbers.  (I don’t suggest actually re-using this code, particularly as you may not be using Rails or the same Twilio library that I am using, but you can reuse the idea of enforcing safety in non-production environments.)

 

 

A lot of Twilio testing will, unfortunately, require manual button-pressing (or scripts which simulate button-pressing on a telephone).  This is easier to accomplish if you can expose your local development machine to the actual Internet.  There are strong security reasons why you don’t want to do this but, if you’re comfortable with doing it, LocalTunnel is a great way to actually accomplish it.

Also see the section below on Modeling Phone Calls, because it will make Twilio phone trees and call logic much more tractable to unit testing.

You Should Have A Staging Server

A staging server is just a copy of the entire production system minus the actual customers.  (You probably shouldn’t put production data on it, because staging systems are designed to break and as a result they may leak data through e.g. SQL injections.  This is an easy way to lose your DB.)  You should use firewalls and/or server rules to make the staging server inaccessible to the world (aside from Twilio and any other APIs which need to access your site for it to work), but assume you will botch this.

Staging servers are virtually mandatory for Twilio applications, because Twilio apps can fail in ways which will not be detected until they are actually accessed over the Internet.  For example, even with unit and integration testing, failing to properly deploy all audio assets (MP3 files, etc) will cause Twilio to throw hard, customer-visible errors in production.  I have automated systems which check for this now, but since that isn’t an exhaustive list of things that can go wrong in production, part of my workflow for deploying all changes on Twilio is to push them to the staging server first, and then having automated scripts exercise the core functionality of the application and ensure that it continues to work.

How To Model Phone Calls

Twilio Quick Start guides generally don’t suggest modeling phone calls explicitly, instead relying on just taking user input and doing if/then or switch statements on it.  This is ineffective for non-trivial use cases, because as the application logic gets more complicated, it will tend to accumulate lots of technical debt, be hard to visually verify for correctness, and be extremely difficult to automatically test.  Instead, you should model Twilio calls as state machines.  I am a big fan of state_machine in the Ruby world.

I’ll skip the CS201 description of what a state machine actually is.  If you didn’t take that course, Google is your friend.

You should model calls such that they start in a predictable state and user input moves the call between states, causing side effects of a) running any business logic required and b) outputting Twiml to Twilio, to continue driving the call.  This lets you replace case statements with a lot of parallel structure with well-defined transition tables within the call models.  Those models are then trivial to unit test.  Additionally, adopting coding conventions such as “the Twiml to be executed at a given state is always state_name.xml and any audio assets go in /foo/bar/state_name/*.mp3 “allows you to write trivial code which will test for their presence, which will save you from having to manually go through the entire phone tree every time on the staging server to verify that refactoring didn’t break anything.

Additionally, state machines are much easier to reason over than masses of spaghetti code which case statements tend to produce.  For example, consider the following code, which attempts to implement the phone prompt “Press 1 to confirm your appointment, press 2 to cancel your appointment, press 3 to ask for us to contact you about your appointment.”  Spot the bug.

There are actually over six bugs in that code, above the trivial ones you probably saw with numbers not lining up to action names:

  • The Twilio API will pass this code params[:Digits] not params[:digits], which will cause an error that won’t be caught until you physically pick up the phone.
  • The comparisons of params[:digits] with integers will fail, because it includes string representations of numbers.
  • There are several mistakes in mapping numbers to actions.
  • One of the action names is spelled improperly.

These are very easy to miss because our brains get lulled into a false sense of security by parallel structure.  Instead, the model should be taking care  of that mapping between user input and state transitions.  This would radically simplify the code and make the controller virtually failure-free, while letting the model exhaustively unit-test possible user input, expected transitions, and business logic side effects.

State machines might seem like an unnecessary complication when you only have three branches in your code, but production Twilio applications can get very, very complicated.  Here is a state diagram from Appointment Reminder.  You do not want to have to test these transitions manually!

Dealing With Answering Machines

Dealing with the case where the phone calls is answered by an answering machine or voicemail system has been the hardest application design problem for me in doing outgoing phone calls in Twilio.  The documentation suggests using an IfMachine feature, which will cause Twilio to listen to a few seconds of the phone call prior to executing your code.  They do some opaque AI magic to determine whether the entity speaking (or not speaking) in that interval is a machine or not, and tell your application whether it is talking to a machine or a human.  In my experience, this has error rates in the 20% region, and many customers intensely dislike the gap of dead air at the start of their phone calls.  Also, if the heuristic improperly detects the beep, your message will start playing early, causing the recording to be cut off in the middle.

There are several ways you could attempt to deal with this:

  • Ignore the issue and treat both machines and humans the same.  This will produce the optimal result for humans, but your system will be virtually unusable when it gets a machine.  (This happens very frequently in my use case.)
  • Force a keypress (“1″) prior to playing your message, then give all users the same message.  This will force most machines to start recording immediately, stopping the cut-off-in-the-middle problem but annoying some clients.
  • Play instructions such as “This is an automated message from $COMPANY.  To hear it, press 1.”  Assume that anyone who doesn’t press 1 in 5 seconds is a machine and play the machine message.  If they interact with the call, play the human message.  This is my preferred solution (although not actually implemented in AR publicly yet, because customers don’t really grok this issue until it bites them personally).

There is one particular problem with recording messages on answering machines: if you give a user instructions such as “Press 1 to confirm your message” and they follow that instruction when listening to their voicemail, that keypress will not be caught by your application, it will be caught by their voicemail system, with unpredictable results (such as deleting the message) and an absolute certainty of not doing what your keypress would normally do.  Users do not understand why this happens.  They expect your instructions to them to work.

Securing Twilio Applications

Twilio applications have a superset of the security issues of web applications.  In addition to the usual SQL injections / XSS issues / etc, use of the telephone has unique security issues associated with it.

One issue is that confidential information is only confidential until you repeat it into a telephone.  Even assuming that the phone call isn’t intercepted (which is, ahem, problematic), there are very common user errors and use cases which will cause that information to be disclosed to third parties.  For example:

  • User error in inputting telephone numbers causes the message to go to the wrong party.
  • The message goes to corporate voicemail, where it will routinely be accessible to third parties.
  • The message is played over a speakerphone / cell phone / etc within earshot of third parties.
  • The message is saved on a physical device which can predictably leave the physical control of authorized parties.
  • etc, etc

Don’t ever put confidential information into an outgoing message, unless you have an automated way to authenticate who you are speaking with.

For incoming phone calls, Caller ID is not sufficient authentication.  It can be trivially spoofed, indeed, your phone company will probably sell you a product whose sole aim is to spoof Caller ID.  Instead, you should use a circumstance where the user is already authenticated and authorized, such as a face-to-face meeting or using a username / password pair in a web application, and then give them one-time PINs to do whatever they need to do on your system.  Alternatively, you can implement an entire password system for your incoming phone calls, but users tend to hate them, so I try to keep to the one-time PIN metaphor.  (When a user does something on the AR site which requires calling the system, such as setting up a recording for a reminder, I tell them “Call 555-555-5555 and put in your Task Code 1234″, which (since it is time-sensitive) both helps me look up what they were doing on a multi-user system and also conclusively demonstrates that they were able to read a web page which already verified their identity.

Not in the presentation because the slide got deleted for some reason: the 4chan rule.  Even if your free trial is discovered by 4chan, the world should not become a darker, more evil place.  There exists tremendous possibilities for abuse of free-form input/output to people’s telephones.  I gate access to my trial by requiring a valid credit card, and demonstration calls and the like have strict rate limits which prevent them from being used to spam someone’s phone to death.  (I should also make it impossible to send demo calls outside of standard work hours.  This is easy to say but a little tricky to implement across multiple time zones while still encouraging legitimate use of demo calls, which is why I haven’t done it yet.)

Twilio Scales Impressively

Twilio and modern web technologies scale impressively well by the standards of traditional businesses.  However, you should probably continue to rate-limit your systems, even though you could theoretically do substantially more volume.  For example, many customers who ask about scaling issues do not sufficiently understand that your application scales several orders of magnitude better than their business processes.  For example, a prospective client asked if my system could handle 10,000 phone calls a month.  I told them that I could handle that in under an hour.  They were quite excited about that, but as we continued to speak about their needs, it developed that actually doing that would have crushed their business.  They would have made 10,000 phone calls in an hour, received over 1,000 callbacks, and their two full-time telephone operators would have been overwhelmed by incoming demand for their time.

Grab Bag of Random Advice

  • Never contact Twilio, or any external API, inside the HTTP request/response cycle.  Doing so imposes an unacceptable delay in performance and slaves your reliability to that of the worst performing API you use.  (Twilio has never had user-visible downtime, but some APIs I rely on have.) Queue the request and tell the browser that you’ve done so.  You can drizzle AJAX magic on your website to make this feel responsive for your users.
  • The Twilio Say verb will have a robot read your message.  This is adequate for development, but for production, people prefer listening to people.  Fiverr.com is great for finding voice actresses for $5.
  • You can’t record too much information about Twilio requests, responses, and errors.  I stuff everything in Redis these days.  I strongly wish I had started doing this earlier, rather than writing “An error happened” to a log file and being unable to determine exactly what the error was or easily figured out whose account it actually affected.
  • When in doubt, don’t make that phone call.  Design your system to fail closed.  This is a continuous discipline, but it will drastically cut down on catastrophic problems.

Wrapup

That’s it for the presentation contents.  I remain very interested in Twilio apps, and am happy to talk to you about them whenever. My contact details are trivially discoverable.

I’m going to attempt to write a more comprehensive guide to developing Twilio applications, eventually. We’ll see what form that takes — I would really like to provide people an (even) easier way to get started, but at the same time I can’t justify dropping two months of my schedule to write a traditional book on it.

When being “first” is not a competitive advantage

“WE STARTED THE TREND”

“Wherever you hear companies talking about Managed WordPress Hosting, they are following our lead.”

This is how one of our competitors attempts to differentiate and compete with us at WP Engine. Is it effective?

Most startups insist they’re “first” at something. As they should — what’s the point of a new company that adds nothing new to the world? But is it a good thing? Does it make you better than your competition?

On the surface being “first” sounds impressive, implying innovation and leadership. But upon reflection, it’s not.

Google wasn’t the first search engine, the iPod wasn’t the first mp3 player, and DropBox wasn’t the first shared filesystem. Yet they are the market leaders. Indeed, their market’s greatest innovators.

Being first does imply innovation, but also faults and weakness. The first electric car, while impressively innovative, was pretty bad by most standards — short-range, sluggish, unattractive, and constantly in the repair shop.

So too with software — “v1.0″ might imply novelty, but it also implies “buggy” and “incomplete.” Who wants their business to depend on a v1.0 product?

Even if you consider WP Engine to be nothing more than an imitator of this competitor (we’re not, but that’s a story for another time), the electric car analogy is apt, because in fact we’re an improvement in almost every way.

This competitor had 40 minutes of downtime last month; we had 0 (according to Pingdom, an unaffiliated 3rd-party). We’re 30% faster at delivering HTML (also Pingdom). Our security service is better — we guarantee cleanup of any hacks while this competitor simply claims they’ve never been hacked (though we’ve found malware in customers of theirs who switched to us). Our support is more responsive and knowledgable because we employ more WordPress experts per 1000 customers than anyone.

Those are tangible competitive advantages; being “first” isn’t.

In fact, being first is a shackle because you’re stuck with legacy support — features you thought were important but aren’t now, grandfathered customers who are no longer profitable, and a reputation that’s hard to revamp even if your company has in fact changed.

Worse, being first means you make all the mistakes in public, for all to see. New competitors get to swoop in afterwards with the hindsight that you created — in pricing, positioning, marketing, features, design, … anything. They get the running start without all your baggage.

Now there are situations where being first is indeed a massive advantage. This exception arises when you’re not only first, but able to expand fast and continue innovate. Early mp3 players didn’t do this — they didn’t get huge traction and weren’t creative in hardware or design, which meant Apple had the space to innovate in design while the market was still largely available.

But McDonalds and Ford were first and kept their leads, because they never stopped innovating in process and product, constantly driving down costs while adding new desirable features, using their lead to fund even stronger growth.

Amazon is the quintessential example in the hosting industry. They didn’t just put “cloud computing” on the map, they’ve never stopped lowering prices while expanding both depth and breadth of service. If they keep it up — and there’s currently no reason to think they won’t — it’s hard to see how anyone could overtake them. And so far no one has come close, though at least two multi-billion-dollar companies are trying.

So yes it can work, and a well-funded startup combined with a stellar team and bit of good fortune can occasionally pull it off. But most little companies aren’t in that position, and it’s often not the best risk/reward position anyway.

What this means is that being “first” isn’t an end in itself — it’s not an advantage, not a feature, not a benefit the customer can experience. It’s a means to one of those ends — it can be levered into market dominance or it can be a manicle that locks your company in a box you can’t get out of.

It’s OK to be proud of being “first” at something, and you should be. But like being “disruptive,” being “first” isn’t necessarily desirable.

In any event, it’s not a competitive advantage. It’s just history.

Let’s continue in the comments — when is “first” a benefit or a drawback?


Sign up for AppSumo's daily deals specifically for web geeks & entrepreneurs.

Go back to school for your startup. Free!


Wouldn’t it be awesome if you could take a class in how to build your startup, from a world known authority, at one of the best colleges in the world, for free?

That’s exactly what you can get from Steve Blank, at Stanford University, starting in February. The Lean Launchpad, also know as Engineering 245, is an online class with free enrollment. That’s right, for the price of your email address and name, you can take from the comfort of your computer an Honest-to-God Stanford University class.

How cool is that? (look for me there, third row on the left.)

Here’s the class description:

In this class you’ll learn how to turn a great idea into a great company.

We now know that startups are not smaller versions of large companies. Large companies execute known business models. They use big company tools – business plans, income statements, revenue models, etc. to help organized their execution. In contrast startups search for a business model. And all the big company tools are irrelevant in the early days of a startup. This class is not about how to write a business plan. It’s not an exercise on how smart you are in a classroom, or how well you use the research library. The end result is not a PowerPoint slide deck for a VC presentation. Instead you will be getting your hands dirty as you encounter the chaos and uncertainty of how a startup actually works.

You’ll learn how to use a business model canvas to brainstorm each part of a company and customer development to get out of the classroom to see whether anyone other than you would want/use your product. Finally, you’ll see how agile development can help you rapidly iterate your product to build something customers will use and buy. Each week will be a new adventure as you test each part of your business model.

And who is Steve Blank, you have to ask?

The Instructor:

Steve Blank is a serial entrepreneur and has been a founder or early employee at 8 startups, including 4 resulting in successful IPOs. For the past 7 years he’s been teaching entrepreneurship to Stanford Engineering students. He’s been awarded a Stanford Undergraduate teaching award, and the San Jose mercury news has called him one of the 10 influencers in Silicon Valley.

Already know everything there is to know about building a successful startup (Ha!)? Well, how about a couple of other classes? Technology Entrepreneurship or Software Engineering for Software as a Service? Same deal: free, online, world-class instructors and information.

(A note re the sign up pages above – the html/css coding is kind of funky, at least on OS X Chrome and Safari. After you enter your name, hit tab, then do your email address, then tab to get to the sign up button.)

You’re reading Go back to school for your startup. Free! from: 47 Hats. If you like this post, there’s plenty more! Want more sales for your startup? Stop by and let’s chat, or consider a Microconsult with Bob Walsh.

What Now? Episode 3 – “Adapt or Die”

What Now? Episode 3 - ”Adapt or Die”

Download this episode, in which Gavin and Andrey talk about Retro Dreamer’s office habits, Antair’s office experience, keeping up appearances, Android porting tribulations, the Android paid apps report, Andrey’s latest gadget buy, customer support fears, Retro Dreamer’s new game, free-to-play development, the race to the bottom hitting $0.10, the Daily Show on IAP, Kindle Fire one-click ordering, designing a game with a little help from the players, subscription revenue, and chargebacks.