Background
This blog should have been written months ago, but I've been busy with a new job and laziness, and I've been distracted by Cloud Foundry. Frankly, I've not really written much Chef code in 5 months now. So, now that I'm feeling some Chef pain again, I'm back to looking for a way to make life better. My frustrations largely center in a few areas of integration between cookbooks, as well as in the management of version-less components of Chef.
- Roles are not versioned, which means I can't change the behavior of a role in one environment without impacting others.
- Environments, nodes, and databags are not versioned.
- Cross-cutting concerns are everywhere. Ports, paths, logs, services, etc. I need a way for my cookbooks to communicate these cross-cutting concerns.
- Dependency management. I like to keep my cookbook dependencies clean, but when you have dozens of cookbooks, that gets pretty challenging.
The Details
Roles
Roles are the forgotten stepchild of Chef. They were never really needed, and were never done right. What's a Role? Well, it's a collection of recipes, other roles, and attributes. What's a cookbook? It's a collection of recipes and attributes (plus some other stuff). In other words, a Role is just a castrated cookbook. They'd be useful if they hadn't been so leaned down as to remove the versioning. With versions missing, there is no way to migrate a change in your roles from develop to staging to production. So people resort to all sorts of odd behavior to allow this. At my current employer, we have roles with name_<environment> type names. It's a hack, and it's ugly, and it makes it really easy to mess something up. Besides, a major reason for Chef is to ensure consistency between environments. My answer, don't use roles. Just use very simple cookbooks that wrap other cookbooks and provide the function of roles, while also providing versions.
NOTE: Yes, I know that this prevents you from doing any searches on roles, and likely hides most of the runlist from doing searches for recipes too. However, you really shouldn't be searching roles and recipes anyway. They are the HOW of a node. You should be searching the WHAT of a node. What it does, or provides. See the section on cross cutting concerns to see why I don't need to search for roles and recipes.
Environments, Nodes, and Databags
Mostly this is a workflow issue as I really only care about having them in version control, not so much about having them version numbered. Unlike Roles, you don't really move environments and nodes from dev to staging, etc. Databags you could argue either way, and I honestly haven't used them enough to really comment. I'm still working on a good solution to this, but for now I keep these items locally on my machine in a git repo. Any changes require a commit before they are upload to the server. Honestly, I wish there were a way to have the Chef server use a git repo for these items.
Cross-Cutting Concerns
I hate dependencies, and try to limit the number of cookbooks that each of my cookbooks require. Largely because I don't want to tell you how to do things. Thus, my cookbook should tell you about the logs it creates, the services it provides, the ports it uses, etc. However, my cookbook should not open ports, process logs, etc. The challenge is a consistent language between cookbooks to ensure that my firewall cookbook can find all of my ports. The folks at Infochimps created the silverware cookbook, which goes a long ways toward bringing an aspect oriented approach to cookbook development, but I found their cookbook to be pretty intense, and I had trouble trying to learn it. So I'm starting development on my own Spicerack cookbook. The intent is to provide a set of small libraries that will allow for the easy sharing of information between cookbooks. Presently, I intend to support ports, services, endpoints, logfiles, and possibly directories. The initial test/use case will be wiring together some web-services with a firewall cookbook of some sort, logstash, and kibana.
Dependency Management
As I already mentioned, I hate dependencies, and do my best to avoid them. But I also believe in DRY compartmentalized code, so I have plenty of dependencies. Spork from Etsy goes a long way to making version management of cookbooks easier. It also makes managing the versions used by each environment easier. What it doesn't do yet, nor does anything else I've found, is handle the dependencies between cookbooks. When I update a library cookbook, I'd love a tool that would give me a list of all cookbooks in my library that depend on that updated cookbook. I could then choose whether to update the dependency (and thus the version # for each of those cookbooks), or leave it alone.
That said, I'm becoming increasingly convinced that this is a smell in my process. I've become quite fond of semantic versioning and ~> constraints. I'm hoping that the combination of these two will go a long way toward making dependency management easier. Nonetheless, knowing which cookbooks need to use my update would be great.
So there you have it. Coming soon, How Docker and Conf.d are Changing my Entire DevOps Approach.
