Wednesday, December 16, 2015

Introducing Argos Dashboard

If you've seen our previous posts or tech talks, you know that we are huge fans of Hystrix, the resiliency library by Netflix. The Hystrix dashboard has been great for monitoring a single application or service, but as the number of applications and microservices increased, we found ourselves needing a more holistic view of our infrastructure. We currently trend all our Hystrix metrics into Graphite for correlation and forecasting, but we really needed something concise and real-time. Inspired by the usefulness of the Hystrix dashboard, we have created Argos Dashboard.

Argos Dashboard provides a real-time view of multiple Hystrix clusters by aggregating data for each cluster onto a single dashboard. The new dashboard has been extremely useful for monitoring how requests flow through the dependencies of each application. It also helps easily monitor what happens when we do performance and resiliency testing. Today, we are releasing Argos Dashboard on GitHub at We hope you find Argos Dashboard valuable and look forward to feedback and pull requests.

Monday, November 30, 2015

Everything You Need to Know About Bower, Part 1

Here at, we decided to migrate to a private Bower registry to use as our artifact repository.  There were numerous growing pains, and we’re hopeful that if we share them with others, maybe they can be spared some groans.
Semantic Versioning (SemVer)
Before we can dive in, first we need to visit the idea of semantic versioning.  The gist of SemVer is this: a version number consists of a triad of numbers, for example: 1.2.3.  The individual numbers are referred to, from left to right, as MajorMinor, and Patch.  As your project develops, you should adjust different sections of the version number depending on how significant the change was between this release and the previous one:
  • Increment the Major when releasing a feature that breaks the API and is not backwards-compatible.
  • Increment the Minor when releasing a feature that doesn't break the API.
  • Increment the Patch when releasing a bug fix (or a dozen).
And that's more or less semantic versioning.  (By the way, did you notice that the SemVer spec is actually versioned using SemVer?  How meta!)
The Subtle Nuances of Node's and Bower's SemVer
But when used with npm and Bower, SemVer is more complicated than that.  There are several pitfalls in the many exceptions, shorthands, and minutiae of the way node-semver handles ranges.
This information is technically available in the version section of the node-semver README.  Go ahead, read that document; we dare you.  I'll wait...
Ah.  Back already?  Right, because that documentation reads like, well, Git documentation.  It's obtuse and assumes you're a lawyer.
It's simple enough to say, "This bug fix is a patch release, therefore my new version is x.y.z+1."  But when you need to say, "My software depends on library X and library Y, and specifically these versions," Bower can accommodate, by expanding the SemVer spec with some extra syntax.
You can use >, >=<<=, and ^ and ~ to ask for the latest within a range of versions.  For example, >=1.2.3 will resolve to the latest version, as long as it's at least 1.2.3, but under 2.0.0.  >=1.2.3 <1.3.0 will resolve to any 1.2.x version, as long as x is at least 3.  ^ is shorthand for the first example (^1.2.3 will get any 1 release greater than or equal to 1.2.3) and ~ is shorthand for the second example (~1.2.3 will get, etc, but not 1.3.0).  Don't worry about writing this down; there will be an examples section in part two.
At this point, let's cover some of the wrong assumptions you might be making about Bower.

Bower is Just Git

It's helpful to know that Bower is really just a layer between you and Git tags.  Bower "knows" about releases by scanning Git tags for SemVer-like tag names.  When you register a Bower project, you have to give it a name and a Git endpoint–and that's it.  Bower then treats Git tags that look like SemVers as released versions, takes care of resolving the request for you, and performs Git actions on your behalf.
Where this bit us is that Bower is set up to periodically scan Git repos for tags.  It caches the Git tag list, which is reasonable, but when we're working with our own libraries and apps, it can be tedious to wait up to 10 minutes for a fresh version to show up in Bower.  We have since reduced this refresh rate to 1 minute, but it's still possible to be moving fast, catch Bower at the very beginning of that refresh cycle, and try to install your new lib/app but not have it show up because Bower doesn't know about it yet.
To actually do this with your private Bower server, edit the config file property repositoryCache.git.refreshTimeout from the default of 10 to something more reasonable.  
To see if a version is available yet, perform a bower info command.  Simply feed it the name of the project (such as bower info project-name), and, optionally, a version request (such as bower info project-name#>=1.2.3).  Without the version request it will return a full list of available versions, and with the version request you'll see which version it resolves to, which will hopefully be your freshly-released version.

Random Bower-related Git Tip

Say you're curious which branch was used to release a given version of an app – we typically work in descriptive branch names that may not indicate the version number.  You can find a branch related to a tag with the following:
git branch -r --contains 1.2.3

Bower Prefers Stable Versions

This should be tattooed on your forehead, and you should have a mirror at all times so you don't forget it.  Or at least, it should be added to the masthead.
When we say that it "prefers stable versions," let's explain by way of example.  Let's suppose we have these two versions available:
  • 1.2.3
  • 1.2.4-0 (a pre-release version)
And then let's suppose we specify that we want ~1.2.3 of this library. It's reasonable to think that the resolved version would be 1.2.4-0, because it's the latest release and the version is less than 1.3.0, so it fits the description.  However, it actually resolves to 1.2.3–because it prefers stable versions.  It doesn't matter if a newer, fresher pre-release is available; if a stable version matches the request, Bower declares that you roll with that.
But what if there are no stable versions that match?  Then, Bower sighs heavily and looks at you with an exasperated "if that's what you really want" kind of look and gives you the latest pre-release. For example, ~1.2.4 would resolve to 1.2.4-0.
Just to follow this logic through, if 1.2.4 was properly released, and 1.2.5-0 was begun, then:
  • ~1.2.3 resolves to 1.2.4, because it's the latest stable release
  • ~1.2.4 resolves to 1.2.4, because it's the latest stable release
  • ~1.2.5 resolves to 1.2.5-0, because there is no stable release that qualifies, and that's the latest pre-release.

Major Version 0 is Special

Bower treats v0.y.z as initial development, which means that the public API should not be considered stable.  This actually has ramifications when using the caret as a range identifier (see below).

You Don't Need to Specify a Full Triad in Bower

You can specify a version as simply 1 or 1.2.  Specifying a single digit is, in effect, specifying the major version, and saying, "I care not what minor and patch version I get, just get me the latest."  In other words, 1 == ^1.0.0. Using just two digits, such as 1.2, will, as you might expect, be the same as ~1.2.0.  This in itself isn't terrible, but it leads to the following...

^ Behaves Differently Depending on Unexpected Things

We initially explained the caret as a range that is greater than or equal to the specified version, including patch and minor updates but not major updates.  There is a dramatic "but" here–the official node-semver definition of the caret is that it allows changes that do not modify the leftmost nonzero digit.  So if you specified ^0.0.33 is the leftmost nonzero digit, and that's not allowed to be modified when using the caret, so you've basically locked this down to 0.0.3.  And ^0.2.3 is basically the same as ~0.2.3.

~ Has Alternate Behaviors, Too

Not be left out, the tilde also has exceptions to the basic rule.  Generally the tilde allows patch-level changes, but if you have specified a simple ~1, then minor-level changes are allowed.  That is, ~1 == ^1.0.0.  Just to be clear, ~1.2 == ~1.2.0.  That's a slight difference from ~1.2.3, which wouldn't pick up 1.2.0.

Pre-release Tags Factor Into the Version

Let's start with some basic facts:
  • A pre-release is a version that is not considered stable or finished.  SemVer stipulates that a version must not be changed once it is released, but a pre-release is a way to iterate over a single version during development.
  • In SemVer, a pre-release is designated as either:
    • A hyphen and then a digit following the main tuple (such as 1.2.3-0).  The digit gets incremented as pre-releases are made.
    • A hyphen, a string identifier, a dot, and a digit (such as 1.2.3-awesome-feature.0).  The string identifier can only contain alphanumerics and hyphens.  The digit after the dot would get incremented as pre-releases are made.
  • A pre-release tag is the string identifier in the previous example.
Bower seems a little hard to understand when it comes to discerning pre-release tags. Consider the following scenario: Two teams are working on the next release of awesome-project, and differentiate their work with versions 1.2.3-awesome-feature.0 and 1.2.3-killer-feature.0.  Obviously they increment the .0 to .1, and so on, as needed.
Now, let's say you're on the "awesome feature" team.  And you want to request that version, not the "killer feature" version.  So you set awesome-project's version in your app's bower.json to >=1.2.3-awesome-feature.0. And you would reasonably expect Bower to pick up the latest "awesome feature" tag, whether it's .0 or .12.
You can tell where this is going; it doesn't.  It picks up the latest "killer feature" tag.  Why?  Because Bower doesn't look at the pre-release tag so much as an identifier in its own right, but as an indicator that there is a pre-release tag.  Once it sees that you want a pre-release tag, it looks at all pre-release tags and picks the highest one.  As in, alphabetically highest.  Therefore, "killer-feature" will win out over "awesome-feature."
It gets better.  You don't even need to supply an existing pre-release tag to get Bower to look for them.  This request will also resolve to the latest "killer feature" release: >=1.2.3-made-up-stuff.  We kid you not.
The only way to restrict Bower to a single tag is with the following: >=1.2.3-awesome-feature.0 <1.2.3-awesome-feature.100 or some improbably high number.  This is rather unwieldy, but it does work.
As such, pre-release tags may be useful for supplying meaning to a version in development, but not much else.

To Be Continued

In Part Two, I'll take these little bits of Bower logic and share our solution for working with Bower in a way that allows multiple teams to work in parallel, using Bower as a means to distribute our in-house libraries.

References and Further Reading

Official SemVer spec:
Official node-semver spec:
A short and easy-to-read attempt at making the SemVer spec more approachable:

Monday, October 5, 2015

Developer Interviews: Peter Davidson

What do you do and how long have you been at bbcom?

 I'm a principal engineer on a platform api team.  I've been with bbcom since April 2014.

Linux or Mac?


What is a typical day like for you?

 My day is coding, interspersed with a daily scrum, occasional meetings and our daily team basketball game.

What is in your developer tool box?  

 My two favorite tools right now work hand in hand ->  Hystrix and RxJava.  I use Hystrix to simplify threading and add bulk heading and circuit breaking.  I use RxJava to compose data from multiple data sources into a single data stream.  My favorite rx function is

What are your favorite pro tips?

Threading is hard, caching is hard, naming is hard.  Make objects immutable and it makes caching and threading easier.

What did you code up today?  

Fixed a bug with user status being cached with a stale database value by a different thread, after removing from cache but prior to committing the transaction.   
Modified RxJava Observables to emit a fallback value when data is not present in the database. 

What has been the most challenging problem you’ve had to solve?  

We outgrew our current Mongo solution (without sharding) for our news feed.  We needed a solution that met both our scale and performance needs.  Earlier this year, we prototyped, developed and migrated to a news feed based on Cassandra.  The Cassandra news feed improved our performance and will allow us to scale for the future.

Any crazy Bbcom stories?


What is the best thing about working at Bbcom?

Lots of places have challenging problems, but the thing that makes bbcom a great place to work is the people I get to work with.  People at bbcom are passionate about technology and fitness.  We have internal tech meet-ups, daily basketball/football games and a lot of fun.

Friday, September 11, 2015

A coding demo on Grunt, RequireJS, AngularJS and Yeoman

Our Client Side Engineers at have fully embraced a modern front-end tech stack to speed up development, testing, and deploys. We use AngularJS to build and scale our social and eCommerce apps, RequireJS for modularity in our build process, and Grunt and Yeoman for minification, compilation, linking, and more.

Learn how transitioned to these frameworks and tools with live coding demonstrations.

Thursday, September 10, 2015

Improving Fault and Latency Tolerance with Hystrix

The API for serves more than a hundred million API calls everyday across hundreds of servers. In this video you will learn how we use Hystrix to build a distributed system that is both fault and latency tolerant. We will discuss the bulkhead and circuit breaker patterns used by Hystrix to provide a resilient and fast API.

Tuesday, April 21, 2015

Give back to the people that you owe your success to

It’s with great pleasure that I introduce, via the first official blog entry, the engineering team.  Our intent with this blog, and our soon-to-be published open source projects, is to give back to the community from which we've learned so much.  Paying it forward is a core value that’s literally baked into our company’s DNA:
"Give back to the people that you owe your success to.
We will share insight into our experience by covering subjects such as performance, scale, frameworks, and mobile just to name a few.  This post will likely be one of the most vanilla and non-technical, so I'll try to keep it brief.  

If you already know about (aka bbcom), then you can stop here.  For those who are new to the site, the rest of this post is for you.  

On the surface, bbcom may just look like another eCommerce company.  In reality, it’s an engineering powerhouse built on some of the most advanced platforms in existence.  There’s a reason we’re ranked within the top 200 on Alexa - we have a huge digital publication, the largest fitness social network, one of the top forums in the world, and the biggest sports nutrition site to boot.  Shocked? That’s understandable. When I came on board over four years ago, I too was surprised. What we’ve achieved since then is truly impressive.  

The graphic below was put together in 2012 for the “Tech Talk” we hosted in Boise, ID.

While we’ve surpassed these numbers by leaps and bounds, their magnitude alone shows the volume of work we produce and the number of people we reach.  All of this with a team of just over 100 passionate technologists with engineering offices in Boise, Portland and Costa Rica

What about the future?  We have big plans to continue revolutionizing the health and fitness space.  Our near term focus is to help our users create lasting change i.e. sticking to your workout schedule.  To create positive habits that stick with you indefinitely.  This, as it turns out, is a very hard problem to solve.  Here at bbcom we welcome the challenge with arms wide open.  Bring it on!     

Thank you for reading.  Check back often as we will be contributing regularly. Better yet, subscribe.

VP Engineering & Product