Friday, September 2, 2016

Why do we need DevOps?

Engineering Manager, Eugene Dina, discusses dev-ops and how the company is leveraging it to streamline its process, and minimize its time to market.

By empowering its engineers at every step of the development and deployment pipeline, and utilizing automation, infrastructure as code, and automatic provisioning, teams are able to build and deploy software more efficiently, and with fewer defects.

Wednesday, June 8, 2016

Developer Interviews: Justin Podzimek

What do you do and how long have you been at bbcom?

I’m currently a Principal Software Engineer and have been working at for about 8 years now. In addition to normal development, I’m responsible for helping my team of client-side engineers, software engineers, and QA in whatever project we’re currently working on. I also have a helping hand in guiding the technology we use at along with the other technical leaders.

Linux or Mac?

Mac, but only slightly. There are some applications and utilities on my Mac that just aren’t there, or aren’t as good, on Linux. That being said, a good portion of my work is spent in the command line, so as long as it’s not Windows, I’m not super picky.

What is a typical day like for you?

The mornings are fairly consistent, with a daily scrum to discuss what we’ve been working on and what’s coming up. After that, my day is spent in a combination of working with the other members of my team, meetings, and heads down with my headphones on getting some work done.

What is in your developer tool box?  

#1 - An IDE. Depending on what I’m working on, it’s either IntelliJ or Sublime Text.  #2 - Slack. Getting push notifications on my phone when someone needs me and I’m not online is a huge help. #3 - Google Chrome as my preferred browser.  #4 - Spotify. Music is important to my development mojo.  #5 - Terminal.

What are your favorite pro tips?

alias yolo=‘git commit -am “Deal with it” && git push  -f origin master’
Honestly, the best tip I can give is "stay in the flow”. It’s tough to get back in once it’s broken, so do everything you can to minimize distractions.

What did you code up today?  

We’re currently in the process of migrating upwards of 28,000 articles to a new CMS system living in Amazon Web Services (AWS). Today, I continued on with the migration script to download every image from an article and upload it to S3. In addition, tweaking and fine-tuning the process to make sure the cutover is as seamless as possible to our readers.

What has been the most challenging problem you’ve had to solve?  

Last year, we update the wrapper (header, footer, and sidebars) across our entire site. This was a HUGE project. Every technical team was involved in updating around 15 different Java and PHP applications, migrating our front-end code to use a new build system and AngularJS, site-wide feature enhancements like search and a new navigation, and countless other features. This was all completed in just a few months before the holidays. In all my years working at, never have I seen collaboration between such a large group work as smoothly as it did.

Any crazy Bbcom stories?

The only one that comes to mind is the “poo-nami”. I think that one should be left up to the imagination.

What is the best thing about working at Bbcom?

You get to work on the coolest, most cutting edge products, with the coolest, most knowledge people around. There’s a lot to love about working at

Thursday, May 19, 2016

Everything You Need to Know About Bower, Part 2

In Part 1 of this two-part rant on Bower, I discussed many of the arguably non-intuitive quirks of SemVer as implemented by NPM and Bower.  In Part 2, we round that out by sharing how we've dealt with those peculiarities and successfully collaborate using a private bower server.

How to Make it Work

At, different teams are often making distinct changes in the same code base, and we need to be able differentiate between their releases.  Git branches are obviously going to keep the work separate, but in order to consume or deploy the changes, we need to pre-release the app to Bower, and then it's out there for anyone to consume.  Here's where the whole "stable release" restriction and shorthands can cause trouble.

We Need to Be Restrictive

Third-party apps are generally fine to leave open-ended, using the ^ or ~ shorthands.  More often than not, we want the latest bug fix.  But as we're developing our own interdependent apps, we need to lock down the version request.  It's verbose, but in most cases the only way to restrict the version to a set tuple, while allowing pre-releases within that tuple, while also rejecting any version outside of that tuple, is:

>=x.y.z-0 <x.y.z+1 (where -0 is literal text and z+1 is one more than the number represented by z)
Or, if using a pre-release tag:
>=x.y.z-tag.0 <x.y.z-tag.100
Ya gotta tell Bower what ya want.
For example, use >=1.2.3-0 <1.2.4. Note that we need to explicitly use the -0 part of the version; without it, Bower will only look for stable releases, and, in effect, we could have just said =1.2.3.  With the -0, though, we gain several things:
  • Bower will consider pre-releases acceptable, and pick up the latest 1.2.3-x version.
  • Bower will refuse 1.2.4 and higher, pre-release or otherwise, ensuring that the team gets the version they're working on.
  • Bower will consider the 1.2.3 release valid for that range and prefer it over the pre-releases, meaning the app that is requesting the version doesn't necessarily need to update its bower.json in order to release.  Ideally, at release time, bower.json specifies exact version numbers, but if a version was missed then this notation won't break the desired behavior.
Similarly, if a different team uses >=1.2.4-0 <1.2.5, they will ensure that they don't accidentally pick up version 1.2.3 when it becomes an official release.


Note that you can try these with your own private Bower registry.  Create a new git repo, register it with your Bower server, and then iterate over changes to the readme file, while releasing the various tags.  Then you can test with bower info [your-bower-project]#[request version here].  

Further note that you need to either escape the ><, and space characters – for example, bower info test-semver#\>=1.0.0-0\ \<1.0.1 – or else wrap the whole package#version part in quotes, like bower info "test-semver#>=1.0.0-0 <1.0.1".  (For reference: how the command line uses spaces and angle brackets)

Examples of Favoring Stable Releases

The examples below assume the following available versions:
  • 0.0.0
  • 0.0.1
  • 0.0.2-0
  • 0.0.2-1
  • 0.0.2-2
  • 0.0.2
  • 0.0.3-0
  • 0.0.3-1
  • 0.0.3
  • 0.0.4-0
  • 0.0.4-1
  • 0.0.5-0
  • 0.0.5-1

Requested version
Resolved version
^0.0.0 or ~ assumes you want the latest stable release. 0.0.4 and 0.0.5 exist only in pre-release versions, so 0.0.3 is resolved.
0.0.5-1Only pre-release versions are greater than 0.0.3 (because we aren't including 0.0.3 itself), so even though Bower would assume you want the latest stable version, there is no latest stable version in this case, and the latest pre-release is returned.
~ the same as the above example. We want the latest patch greater than or equal to 0.0.4, and since there are no stable releases in that range, we get the latest pre-release.
0.0.5 or =0.0.5noneDeclaring a stable version with no range will not return any version if the stable version doesn't exist; Bower assumes you want the exact stable version requested and will not return the pre-release versions.

Examples of Pre-release Tags

The following examples assume the following available versions (released in order):
  • 1.0.0-alpha.0
  • 1.0.0-alpha.1
  • 1.0.0-beta.0
  • 1.0.0-beta.1
  • 1.0.1-foo.0
  • 1.0.1-foo.1
  • 1.0.1-bar.0
  • 1.0.1-bar.1
  • 1.0.1-z.0
  • 1.0.1-z.1
  • 1.0.1-a.0
  • 1.0.1-a.1

Requested version
Resolved version
Apparently, Bower considers anything with a pre-release tag as simply a marker to get a pre-release. It considers the "latest" to be the highest in an alphabetically sorted list of tags. Therefore, pre-release z is chosen.
>=1.0.0-alpha.0 < range really only limits the numeric triad to 1.0.0, so we filter out all of the 1.0.1s. But the "latest" version is the highest alphabetically, so beta is returned.
>=1.0.0-made-up-stuff1.0.1-z.1Note that the pre-release tag doesn't even exist, and moreover there is no "dot number." Bower sees no foul in this. It muses, "Ah, you want a pre-release. Here's the latest pre-release!"
>=1.0.0-alpha.0 <1.0.0-alpha.1001.0.0-alpha.1This actually works! Cumbersome as heck, but it does enable separation of pre-release tags.

The End (v1.0.0-a.1)

Hopefully this guide to how Bower handles SemVers reduces your aches and pains.  If you'd like to read more, check out the section below.

References and Further Reading

Official SemVer spec:
Official node-semver spec:
A short and easy-to-read attempt at making the SemVer spec more approachable:

Wednesday, February 24, 2016

Building an Upload Microservice using AWS Lambda

One of’s core missions is to help people transform their lives through the adoption of a healthy lifestyle and exercise. One way we motivate people to begin such a journey is through our yearly “transformation challenge.” Over the course of 12 weeks, participants in the challenge are encouraged to adopt a training plan and set new habits that will lead to a fitter, healthier body. Prizes are awarded to the people who make the most profound transformation over the course of the contest. 

In order to participate, contestants are required to upload photos of themselves before and after the challenge. As one might imagine, the time period shortly before and shortly after each challenge tends to produce a spike in photo-upload traffic to our servers. Historically, these periods have been problematic for the scalability of our backend systems. In order to improve our end-user experience (not to mention our own), we decided to re-architect our photo-upload system in late 2015.

This tech talk describes how we created a new microservice-based architecture using Amazon Web Services technologies as well as a Java/Spring stack. Our goal was to greatly improve our ability to handle spikes in image-upload traffic without manual intervention. As a result, we created autoscaling RESTful services using Spring Boot and AWS Elastic Beanstalk. Image-processing logic has been encapsulated into highly scalable Amazon Lambda functions.

We recently launched the 2016 transformation challenge. With our new image-upload infrastructure in place, we have been able to handle peak loads without a hitch.

Wednesday, January 27, 2016

Video: What is RESTful? (2.0.0)

Roughly two years ago, the Commerce Engineering Team set out to deliver native mobile phone applications for iOS and Android that offered the same rich user experience that hundreds of thousands of customers enjoy within our web applications each day. We started with a couple of UI/UX mockups, a firm knowledge that we needed to develop an API, and a strong desire to make that API RESTful. A large, dedicated team spent 18 months of their lives learning and growing to achieve a soft launch of the applications in May of 2015 in the Google Play Store and iTunes. We believe a project of such magnitude must, beyond its primary goals, create useful and reusable assets to be considered a success. By that definition, this project was a huge success. We have a few libraries we plan to open source, a bunch of best practices to document and present, and a ton of knowledge and opinions to share with the development community.

Other than the applications themselves, the most valuable assets developed during the project have been our opinions on what defines a RESTful architecture. Even months into the project, we found ourselves in long debates that led to redefining our views of RESTful architecture. Each of these cycles triggered new breakthroughs and pitfalls in our implementation. Even now, months after the launch, we still find ourselves re-evaluating past decisions and making significant changes in our implementation to align with the newest visions. The pieces of that newest vision were collected together into a presentation named "What Is RESTful?" that was used multiple times internally to achieve a common base understanding from which the more interesting conversations around implementation grew. Even this presentation has gone under multiple revisions and has recently achieved the SemVer status of 2.0.0. This 2.0.0 version was recently presented at the Fall 2015 Tech Talk and is now available for online viewing. We expect to use this presentation in future posts and presentations as a launching point to dive deep into the finer aspects of our implementation and the software technology that grew from our vision. It is certain that we have not gotten everything right, and we encourage feedback, both constructive and critical, in order to develop the 3.0.0 version of this vision.

Wednesday, December 16, 2015

Introducing Argos Dashboard

If you've seen our previous posts or tech talks, you know that we are huge fans of Hystrix, the resiliency library by Netflix. The Hystrix dashboard has been great for monitoring a single application or service, but as the number of applications and microservices increased, we found ourselves needing a more holistic view of our infrastructure. We currently trend all our Hystrix metrics into Graphite for correlation and forecasting, but we really needed something concise and real-time. Inspired by the usefulness of the Hystrix dashboard, we have created Argos Dashboard.

Argos Dashboard provides a real-time view of multiple Hystrix clusters by aggregating data for each cluster onto a single dashboard. The new dashboard has been extremely useful for monitoring how requests flow through the dependencies of each application. It also helps easily monitor what happens when we do performance and resiliency testing. Today, we are releasing Argos Dashboard on GitHub at We hope you find Argos Dashboard valuable and look forward to feedback and pull requests.

Monday, November 30, 2015

Everything You Need to Know About Bower, Part 1

Here at, we decided to migrate to a private Bower registry to use as our artifact repository.  There were numerous growing pains, and we’re hopeful that if we share them with others, maybe they can be spared some groans.
Semantic Versioning (SemVer)
Before we can dive in, first we need to visit the idea of semantic versioning.  The gist of SemVer is this: a version number consists of a triad of numbers, for example: 1.2.3.  The individual numbers are referred to, from left to right, as MajorMinor, and Patch.  As your project develops, you should adjust different sections of the version number depending on how significant the change was between this release and the previous one:
  • Increment the Major when releasing a feature that breaks the API and is not backwards-compatible.
  • Increment the Minor when releasing a feature that doesn't break the API.
  • Increment the Patch when releasing a bug fix (or a dozen).
And that's more or less semantic versioning.  (By the way, did you notice that the SemVer spec is actually versioned using SemVer?  How meta!)
The Subtle Nuances of Node's and Bower's SemVer
But when used with npm and Bower, SemVer is more complicated than that.  There are several pitfalls in the many exceptions, shorthands, and minutiae of the way node-semver handles ranges.
This information is technically available in the version section of the node-semver README.  Go ahead, read that document; we dare you.  I'll wait...
Ah.  Back already?  Right, because that documentation reads like, well, Git documentation.  It's obtuse and assumes you're a lawyer.
It's simple enough to say, "This bug fix is a patch release, therefore my new version is x.y.z+1."  But when you need to say, "My software depends on library X and library Y, and specifically these versions," Bower can accommodate, by expanding the SemVer spec with some extra syntax.
You can use >, >=<<=, and ^ and ~ to ask for the latest within a range of versions.  For example, >=1.2.3 will resolve to the latest version, as long as it's at least 1.2.3, but under 2.0.0.  >=1.2.3 <1.3.0 will resolve to any 1.2.x version, as long as x is at least 3.  ^ is shorthand for the first example (^1.2.3 will get any 1 release greater than or equal to 1.2.3) and ~ is shorthand for the second example (~1.2.3 will get, etc, but not 1.3.0).  Don't worry about writing this down; there will be an examples section in part two.
At this point, let's cover some of the wrong assumptions you might be making about Bower.

Bower is Just Git

It's helpful to know that Bower is really just a layer between you and Git tags.  Bower "knows" about releases by scanning Git tags for SemVer-like tag names.  When you register a Bower project, you have to give it a name and a Git endpoint–and that's it.  Bower then treats Git tags that look like SemVers as released versions, takes care of resolving the request for you, and performs Git actions on your behalf.
Where this bit us is that Bower is set up to periodically scan Git repos for tags.  It caches the Git tag list, which is reasonable, but when we're working with our own libraries and apps, it can be tedious to wait up to 10 minutes for a fresh version to show up in Bower.  We have since reduced this refresh rate to 1 minute, but it's still possible to be moving fast, catch Bower at the very beginning of that refresh cycle, and try to install your new lib/app but not have it show up because Bower doesn't know about it yet.
To actually do this with your private Bower server, edit the config file property repositoryCache.git.refreshTimeout from the default of 10 to something more reasonable.  
To see if a version is available yet, perform a bower info command.  Simply feed it the name of the project (such as bower info project-name), and, optionally, a version request (such as bower info project-name#>=1.2.3).  Without the version request it will return a full list of available versions, and with the version request you'll see which version it resolves to, which will hopefully be your freshly-released version.

Random Bower-related Git Tip

Say you're curious which branch was used to release a given version of an app – we typically work in descriptive branch names that may not indicate the version number.  You can find a branch related to a tag with the following:
git branch -r --contains 1.2.3

Bower Prefers Stable Versions

This should be tattooed on your forehead, and you should have a mirror at all times so you don't forget it.  Or at least, it should be added to the masthead.
When we say that it "prefers stable versions," let's explain by way of example.  Let's suppose we have these two versions available:
  • 1.2.3
  • 1.2.4-0 (a pre-release version)
And then let's suppose we specify that we want ~1.2.3 of this library. It's reasonable to think that the resolved version would be 1.2.4-0, because it's the latest release and the version is less than 1.3.0, so it fits the description.  However, it actually resolves to 1.2.3–because it prefers stable versions.  It doesn't matter if a newer, fresher pre-release is available; if a stable version matches the request, Bower declares that you roll with that.
But what if there are no stable versions that match?  Then, Bower sighs heavily and looks at you with an exasperated "if that's what you really want" kind of look and gives you the latest pre-release. For example, ~1.2.4 would resolve to 1.2.4-0.
Just to follow this logic through, if 1.2.4 was properly released, and 1.2.5-0 was begun, then:
  • ~1.2.3 resolves to 1.2.4, because it's the latest stable release
  • ~1.2.4 resolves to 1.2.4, because it's the latest stable release
  • ~1.2.5 resolves to 1.2.5-0, because there is no stable release that qualifies, and that's the latest pre-release.

Major Version 0 is Special

Bower treats v0.y.z as initial development, which means that the public API should not be considered stable.  This actually has ramifications when using the caret as a range identifier (see below).

You Don't Need to Specify a Full Triad in Bower

You can specify a version as simply 1 or 1.2.  Specifying a single digit is, in effect, specifying the major version, and saying, "I care not what minor and patch version I get, just get me the latest."  In other words, 1 == ^1.0.0. Using just two digits, such as 1.2, will, as you might expect, be the same as ~1.2.0.  This in itself isn't terrible, but it leads to the following...

^ Behaves Differently Depending on Unexpected Things

We initially explained the caret as a range that is greater than or equal to the specified version, including patch and minor updates but not major updates.  There is a dramatic "but" here–the official node-semver definition of the caret is that it allows changes that do not modify the leftmost nonzero digit.  So if you specified ^0.0.33 is the leftmost nonzero digit, and that's not allowed to be modified when using the caret, so you've basically locked this down to 0.0.3.  And ^0.2.3 is basically the same as ~0.2.3.

~ Has Alternate Behaviors, Too

Not be left out, the tilde also has exceptions to the basic rule.  Generally the tilde allows patch-level changes, but if you have specified a simple ~1, then minor-level changes are allowed.  That is, ~1 == ^1.0.0.  Just to be clear, ~1.2 == ~1.2.0.  That's a slight difference from ~1.2.3, which wouldn't pick up 1.2.0.

Pre-release Tags Factor Into the Version

Let's start with some basic facts:
  • A pre-release is a version that is not considered stable or finished.  SemVer stipulates that a version must not be changed once it is released, but a pre-release is a way to iterate over a single version during development.
  • In SemVer, a pre-release is designated as either:
    • A hyphen and then a digit following the main tuple (such as 1.2.3-0).  The digit gets incremented as pre-releases are made.
    • A hyphen, a string identifier, a dot, and a digit (such as 1.2.3-awesome-feature.0).  The string identifier can only contain alphanumerics and hyphens.  The digit after the dot would get incremented as pre-releases are made.
  • A pre-release tag is the string identifier in the previous example.
Bower seems a little hard to understand when it comes to discerning pre-release tags. Consider the following scenario: Two teams are working on the next release of awesome-project, and differentiate their work with versions 1.2.3-awesome-feature.0 and 1.2.3-killer-feature.0.  Obviously they increment the .0 to .1, and so on, as needed.
Now, let's say you're on the "awesome feature" team.  And you want to request that version, not the "killer feature" version.  So you set awesome-project's version in your app's bower.json to >=1.2.3-awesome-feature.0. And you would reasonably expect Bower to pick up the latest "awesome feature" tag, whether it's .0 or .12.
You can tell where this is going; it doesn't.  It picks up the latest "killer feature" tag.  Why?  Because Bower doesn't look at the pre-release tag so much as an identifier in its own right, but as an indicator that there is a pre-release tag.  Once it sees that you want a pre-release tag, it looks at all pre-release tags and picks the highest one.  As in, alphabetically highest.  Therefore, "killer-feature" will win out over "awesome-feature."
It gets better.  You don't even need to supply an existing pre-release tag to get Bower to look for them.  This request will also resolve to the latest "killer feature" release: >=1.2.3-made-up-stuff.  We kid you not.
The only way to restrict Bower to a single tag is with the following: >=1.2.3-awesome-feature.0 <1.2.3-awesome-feature.100 or some improbably high number.  This is rather unwieldy, but it does work.
As such, pre-release tags may be useful for supplying meaning to a version in development, but not much else.

To Be Continued

In Part Two, I'll take these little bits of Bower logic and share our solution for working with Bower in a way that allows multiple teams to work in parallel, using Bower as a means to distribute our in-house libraries.

References and Further Reading

Official SemVer spec:
Official node-semver spec:
A short and easy-to-read attempt at making the SemVer spec more approachable: