re:Invent 2017 Debrief

Well, it's been a long month, but I think it's time to finally revisit my re:Invent predictions.  I did better than I had expected.  Predictions in Blue

General Announcements and Themes

Serverless

They'll also announce several new services that are purely serverless in nature.  These services will be around: 1. making serverless more secure, 2. making serverless faster and easier to deploy and test (including a HUGE K8s announcement, 3. serverless monitoring.

I think I nailed this one pretty good.  From the widely expected ECS for K8s, to the new Aurora Serverless database, there was a lot to take in on the serverless front.  However, I think there is still a large hole around security and monitoring for serverless.  It's not that you can't use tools like X-ray and AWS WAF to solve those problems, but I feel like more could be done here.

Security

Security is only going to get hotter over the next few years, and AWS doesn't want you to think about security.  So I expect to see more services (or at least more features for WAF and Inspector) to help make security super easy.  In the spirit of serverless computing, all of these will be fully managed services and many of them will not even require action on the part of users.

I was spot on with this one, and so was AWS.  Watchdog is going to be a super helpful product, and is as easy as it comes.  A few button clicks and you have this massive new security system in place.  I'll be interested to see how well they build the interface.  Those with minimal security understanding will need to be able to grasp watch watchdog does and does not do for them.  It also needs to be crazy easy to understand what actions to take when an event is detected.

Data analytics

I considered machine learning, but I think that's next year.  I do expect to see some very targeted announcements in the poly/rekognition/lex family announced last year, but I think those will mostly be very specific ML tools.  Next year I expect to see some broad ML announcements designed to make custom deep learning models super easy, but I just don't think that's been solved yet.  So this year will focus back on analytics.  Specifically, using serverless computing to process and analyze data super easily.  Among other things, this will involve some new connector services that will help pull together various AWS services more easily.  Expect at least a few of these to be lambda branded.  

Well, I missed the target badly on this one, but I'm glad I did.  AWS overshot my expectations, plain and simple.  SageMaker is not only the coolest named service of all time, but looks to be a HUGE step up in making ML accessible to the masses.  

Specific Announcements

Ok, so I'll probably only get 1-2 of these, but here's what I'm expecting/hoping to see:

New instances types

Specifically, it's time for an M5 instance type and a Machine Learning specific instance type.  Wildcard, maybe a new instance family with fast interconnects specifically targeting super computer users, but I don't think so yet.

Yes M5, no ML specific.  Not fast interconnect yet, but we did get bare metal and a new P class instance.

New Security Services

A new IDS NAT gateway option

A virus scanning something...

So sad there is still no AWS IDS :(

Serverless

A managed kubernetes.  Maybe as a ECS branded thing, but probably it's own thing.

A way to run lambda on your data more directly.... Not exactly sure here, but maybe a lambda tie in with one of the big data offereings?  Redshift maybe.

Meh, it was a stretch.

Blockchain

It's hot, and I can't imagine AWS will miss this train.  Expect a blockchain service that is fully managed.

Maybe my biggest surprise.  AWS is missing this train for now.

Bare Metal

After doing some reading last night, I came across an article by James Hamilton about Oracles pipe dream of competing in the cloud with 1/10th the average investment.  In the comments, he talked a bit about bare metal in the cloud.  Given his comments, I think we'll see this soon.  Maybe not this week, but certainly by next year.  This is basically a service which lets you run your own hypervisor on AWS hardware.  It's the opposite of what AWS wants to do, but would greatly speed cloud migrations for many companies that currently run on VMWare and similar hypervisor systems.

Yup

Multi-Region Aurora

We NEED a multi-region RDBMS.  I don't expect to see this outside Aurora, but I do think we'll see a multi-region option coming to Aurora very soon.  What's not clear is how you overcome the speed of light issues.  How do you ensure 100% data retention while still having millisecond commit time?  My left field prediction is that we will see other database systems (redshift, dynamo, kinesis) also go multi-region.

Well, apparently the speed of light is a problem still ;).  However, multi-region multi-master Dynamo is pretty amazing stuff.  It became more clear to me at re:invent that this just can't be done with Guaranteed consistency models.  But multi-region read-replicas for Aurora is a pretty good step.

Other random stuff

• Yup, 5 venues was a mess, but I don't think it's going to get better
• I only rode the buses one day, but it was ugly, and I hear it didn't get a lot better
• The party was actually not as bad as I expected.  They expanded to 3 tents, but it was still pretty crowded.  They did a good job breaking things out between the tents too.
• Werner failed me.  WORST KEYNOTE EVER!
• No MLB this year.  The NFL came instead! :)   Glad to see them finally catching up with this.
• Well, we have a new certification, but it makes no sense to me.  There was really no need for a "freshman" certification.  I think it de-values the entire cert system.  As a side note, AWS is getting TOO BIG for general certs though.  It may be time to specialize them more into areas like Compute, Data, Network, Serverless/container, etc.
• No checking gift, but lots more random giveaways.  Overall, I got a pair of buttons, a deeplense, a 10" fire tablet, an echo+ with lightbulb, and some sort of internet button thing that I can't find any documentation on.
• The hoodies had NO blue on them, and were certainly not better quality :(
• There were some printer giveaways
• Netflix had some cool new stickers
• Tuesday night was actually a lot better than I expected.  Still not as good as James Hamilton, but pretty great nonetheless.

My re:Invent 2017 Predictions

Re:invent is always full of fun, lessons, tons of swag, and new AWS services.  This will be my 4th re:invent, and I think I'm finally getting into the swing of things.  I'm already too late to bother with a re:inven tips blog post, and let's face it, I have no followers to care, so I'm skipping directly to my predictions.  I'll go with three categories: general themes of the keynotes and announcements, specific service announcements, and random other stuff.  Maybe, just maybe, I'll even follow up with a post to reveal how I did.

General Announcements and Themes

So, let's be straight here, it's really hard to accurately predict the exact new services that will be offered.  I'm still going to try, but lest I look completely incompetent, I'm going to start with some more general predictions.

Serverless

Serverless computing will be a HUGE theme again this year.  AWS has been working to figure out exactly how lambda can play over the last 2 years, and I think they've got the architecture pretty much figured out.  Expect this year to be the year that they declare serverless ready for prime time.  As part of that, not only will they showcase how at least one major company is doing something amazing with serverless, but they'll also announce several new services that are purely serverless in nature.  These services will be around: 1. making serverless more secure, 2. making serverless faster and easier to deploy and test (including a HUGE K8s announcement, 3. serverless monitoring.

Security

Security is only going to get hotter over the next few years, and AWS doesn't want you to think about security.  So I expect to see more services (or at least more features for WAF and Inspector) to help make security super easy.  In the spirit of serverless computing, all of these will be fully managed services and many of them will not even require action on the part of users.

Data analytics

I considered machine learning, but I think that's next year.  I do expect to see some very targeted announcements in the poly/rekognition/lex family announced last year, but I think those will mostly be very specific ML tools.  Next year I expect to see some broad ML announcements designed to make custom deep learning models super easy, but I just don't think that's been solved yet.  So this year will focus back on analytics.  Specifically, using serverless computing to process and analyze data super easily.  Among other things, this will involve some new connector services that will help pull together various AWS services more easily.  Expect at least a few of these to be lambda branded.  

Specific Announcements

Ok, so I'll probably only get 1-2 of these, but here's what I'm expecting/hoping to see:

New instances types

Specifically, it's time for an M5 instance type and a Machine Learning specific instance type.  Wildcard, maybe a new instance family with fast interconnects specifically targeting super computer users, but I don't think so yet.

New Security Services

A new IDS NAT gateway option

A virus scanning something...

Serverless

A managed kubernetes.  Maybe as a ECS branded thing, but probably it's own thing.

A way to run lambda on your data more directly.... Not exactly sure here, but maybe a lambda tie in with one of the big data offereings?  Redshift maybe.

Blockchain

It's hot, and I can't imagine AWS will miss this train.  Expect a blockchain service that is fully managed.

Bare Metal

After doing some reading last night, I came across an article by James Hamilton about Oracles pipe dream of competing in the cloud with 1/10th the average investment.  In the comments, he talked a bit about bare metal in the cloud.  Given his comments, I think we'll see this soon.  Maybe not this week, but certainly by next year.  This is basically a service which lets you run your own hypervisor on AWS hardware.  It's the opposite of what AWS wants to do, but would greatly speed cloud migrations for many companies that currently run on VMWare and similar hypervisor systems.

Multi-Region Aurora

We NEED a multi-region RDBMS.  I don't expect to see this outside Aurora, but I do think we'll see a multi-region option coming to Aurora very soon.  What's not clear is how you overcome the speed of light issues.  How do you ensure 100% data retention while still having millisecond commit time?  My left field prediction is that we will see other database systems (redshift, dynamo, kinesis) also go multi-region.

Other random stuff

• The new format with 5 venues is going to stink
• I've got Monday at the Aria, Tue-Thur I'm in the park at Jam sessions (these are great).  Nonetheless, I still expect it to be a giant nightmare.
• The buss system will be a mess (I'm cheating a bit on this one, my first bus ride this AM stunk)
• The party is going to be too crowded to enjoy.  The first year they moved outside was pretty good, but last year the tents were just not big enough.  I loved that they put the games in one tent and the music in the other, but with the larger crowd, they'll need at least 3 tents this year to even have a chance.
• Werner will outdo himself again.  The snowmobile was just amazing last year.  Don't get me wrong, I didn't hear a soul say "I'm so glad they have that now", and I imagine they've got dozens of customers total for that thing, but the wow factor was great.  It was the buzz of the rest of the day.  I expect them to find a way to do something more amazing this year.
• MLB will be back.  Every year they seem to include MLB in one of their keynotes.  I'm not a baseball fan, but these guys are doing amazing stuff with technology.
• They'll introduce another specialty exam.  Last year they tried analytics, security, and networking, and two of the three were a success.  This year I think they'll introduce a serverless exam.
• Last year we all got a dot at checkin.  This year it will be a pair of echo buttons.
• The hoodies will be better than last year, and have more blue on them.
• At least one vendor will be giving away a 3d printer
• Netflix will have new stickers
• The Tuesday night speaker will not be as cool as James Hamilton.  Seriously, go watch this.  I don't care who you are, you can't top that.

Cloudformation on the Commandline

This post is mostly a reminder for me, as I have this issue often enough to kill me, but rarely enough that I always forget the solution.

When you have a cloud formation template that has a parameter of type `list`, passing a value to that on the commandline is NOT intuitive.

"Parameters": {
  "Subnets": {
    "Type": "List<AWS::EC2::Subnet::Id>
  }
}

Here's the trick.

myprompt>aws cloudformation create-stack --stack-name some-stack --template-url https://s3.amazonaws.com/.....   --parameters ParameterKey=Subnets,ParameterValue=\'subnet-2353264,subnet-6513946131\'

NOTICE that we did not put any spaces in the subnet list and that we had to quote it, but not just quote it, we had to escape those quotes too.

Thanks to https://github.com/aws/aws-cli/issues/870 for getting me pointed in this direction.

Saving some cash - (Multi domain wildcard certificates)

Passion

I love to save money.  It's best when I save my own money, but saving my company money is great too.  It's part of why I love my job.  As a Develop Happiness Engineer, my job is to make developers lives easier, so they can be more productive, and so that we can hire fewer of them to do the same great work.  Really, my job boils down to increasing productivity, reducing server count, and keeping everything repeatable and audit-able.

Certificates

One of the necessary evil costs of running a professional website is encrypting traffic.  When you're a healthcare company, this becomes even more critical.  After all, this is pretty sensitive data.  But certificates are not free.  What's worse, I discovered yesterday that even the expensive ones only cover one level of sub-domains.

You see, I was reviewing our URL structure, and realized it was ugly.  We had things like myapp-staging.mydomain.com in one place, and things like staging-someapp.mydomain.com in others.  To make it worse, our route 53 hosted zone had become way too big, and it was getting hard to manage.  So I decided it was time to cleanup the hierarchy and use subdomains to provide some structure to our URLs.

The Problem

Well, we have a wildcard domain cert through a major certificate provider.  It applies to *.mydomain.com.  When I went to create www.develop.mydomain.com I discovered that our cert was not valid for it.  Yep, it only covers one level of subdomain.  So my grand plan was going to cost us a couple thousand dollars by the time I covered all 7+ subdomains we needed.  Lucky for me, only one of our domains is actually used by external customers.  Everything else is internal to our system, or accessed by our own employees.

Self-signed Certificates to the Rescue?

So that means we can use self-signed certs, right?  Well, yes, but how many of them?  And how to distribute them?  Yuk.  If only I could get ONE certificate that covered ALL of my domains and subdomains across ALL internal infrastructure.

A Better Way

Well, it turns out you can. You just need to create a certificate with Subject Alternative Names.  I found a blog post on self signed SAN certs that did a pretty good job.  I just had to clean it up a little and elaborate by adding wildcard SANs to it.  And without further ado, here is the process.

Creating a multi-domain wildcard certificate

Cautions

First, a quick caution.  When you include ANY SANs in a certificate, you must include ALL of them.  That is, the CN in your cert is ignored if you have even one SAN listed.  So be sure to include your base domain in your san list.

The Extensions File

You  must have a file that declares your certificate extensions.  You can do this in your openssl.cnf file, or you can do it (as I do) in a standalone file.

extensions.cnf

1 2 3 4 5 6 7 8 9 10 11

[ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [ alt_names ] DNS.1 = mydomain.com DNS.2 = *.mydomain.com DNS.3 = myotherdomain.com DNS.4 = *.myotherdomain.com DNS.5 = *.mysubdomain.mydomain.com

Generate the certificate

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

# First create a key file (you'll need to enter your password 2X openssl genrsa -des3 -out mykey.key 2048 # Next convert it to a passwordless key (remember that password?) openssl rsa -in mykey.key -out mykey.rsa # Now you need to create a signing request # This requires you to enter a bunch of info # I include: # Country Name # State # City # Organization Name # Email Address # But leave blank the OU, CN, and Password openssl req -new -key mykey.rsa -out mykey.csr # Finally, we generate the actual certificate openssl x509 -req -extensions v3_req -days 365 -in mykey.csr -signkey mykey.rsa -out mykey.crt -extfile extensions.cnf

And that's it, you now have a certificate you can use to secure any domain you had listed in that extensions.cnf file.

 Good luck!

Musings on Chef

Background

This blog should have been written months ago, but I've been busy with a new job and laziness, and I've been distracted by Cloud Foundry.  Frankly, I've not really written much Chef code in 5 months now.  So, now that I'm feeling some Chef pain again, I'm back to looking for a way to make life better.  My frustrations largely center in a few areas of integration between cookbooks, as well as in the management of version-less components of Chef.

1. Roles are not versioned, which means I can't change the behavior of a role in one environment without impacting others.
2. Environments, nodes, and databags are not versioned.  
3. Cross-cutting concerns are everywhere.  Ports, paths, logs, services, etc. I need a way for my cookbooks to communicate these cross-cutting concerns.
4. Dependency management.  I like to keep my cookbook dependencies clean, but when you have dozens of cookbooks, that gets pretty challenging.  

The Details

Roles

Roles are the forgotten stepchild of Chef.  They were never really needed, and were never done right.  What's a Role?  Well, it's a collection of recipes, other roles, and attributes.  What's a cookbook?  It's a collection of recipes and attributes (plus some other stuff).  In other words, a Role is just a castrated cookbook.  They'd be useful if they hadn't been so leaned down as to remove the versioning.  With versions missing, there is no way to migrate a change in your roles from develop to staging to production.  So people resort to all sorts of odd behavior to allow this.  At my current employer, we have roles with name_<environment> type names.  It's a hack, and it's ugly, and it makes it really easy to mess something up.  Besides, a major reason for Chef is to ensure consistency between environments.  My answer, don't use roles.  Just use very simple cookbooks that wrap other cookbooks and provide the function of roles, while also providing versions.  

NOTE: Yes, I know that this prevents you from doing any searches on roles, and likely hides most of the runlist from doing searches for recipes too.  However, you really shouldn't be searching roles and recipes anyway.  They are the HOW of a node.  You should be searching the WHAT of a node.  What it does, or provides.  See the section on cross cutting concerns to see why I don't need to search for roles and recipes. 

Environments, Nodes, and Databags

Mostly this is a workflow issue as I really only care about having them in version control, not so much about having them version numbered.  Unlike Roles, you don't really move environments and nodes from dev to staging, etc.  Databags you could argue either way, and I honestly haven't used them enough to really comment.  I'm still working on a good solution to this, but for now I keep these items locally on my machine in a git repo.  Any changes require a commit before they are upload to the server.  Honestly, I wish there were a way to have the Chef server use a git repo for these items.

Cross-Cutting Concerns

 I hate dependencies, and try to limit the number of cookbooks that each of my cookbooks require.  Largely because I don't want to tell you how to do things.  Thus, my cookbook should tell you about the logs it creates, the services it provides, the ports it uses, etc.  However, my cookbook should not open ports, process logs, etc.  The challenge is a consistent language between cookbooks to ensure that my firewall cookbook can find all of my ports.  The folks at Infochimps created the silverware cookbook, which goes a long ways toward bringing an aspect oriented approach to cookbook development, but I found their cookbook to be pretty intense, and I had trouble trying to learn it.  So I'm starting development on my own Spicerack cookbook.  The intent is to provide a set of small libraries that will allow for the easy sharing of information between cookbooks.  Presently, I intend to support ports, services, endpoints, logfiles, and possibly directories.  The initial test/use case will be wiring together some web-services with a firewall cookbook of some sort, logstash, and kibana.

Dependency Management

As I already mentioned, I hate dependencies, and do my best to avoid them.  But I also believe in DRY compartmentalized code, so I have plenty of dependencies.  Spork from Etsy goes a long way to making version management of cookbooks easier.  It also makes managing the versions used by each environment easier.  What it doesn't do yet, nor does anything else I've found, is handle the dependencies between cookbooks.  When I update a library cookbook, I'd love a tool that would give me a list of all cookbooks in my library that depend on that updated cookbook.  I could then choose whether to update the dependency (and thus the version # for each of those cookbooks), or leave it alone.  

That said, I'm becoming increasingly convinced that this is a smell in my process.  I've become quite fond of semantic versioning and ~> constraints.  I'm hoping that the combination of these two will go a long way toward making dependency management easier.  Nonetheless, knowing which cookbooks need to use my update would be great.

So there you have it.  Coming soon, How Docker and Conf.d are Changing my Entire DevOps Approach.

Growing Pains (and Joys)

Well, I've been on the job for 2 days now, and I'm feeling the pain of new technologies.  Believe it or not, I just got my first smart phone on Monday.  I'm cheap, and have never been able to bring myself to pay more than $20 a month for cell coverage, so smartphones were out of the question.  I can say that learning to use an android phone has been a challenge, but a wonderful experience.  I feel more connected, and love the ability to access things when I'm away from home.  I'm sure I'll feel the leash that comes with it soon enough, but overall I'm thrilled to finally be carrying a modern phone.
My new laptop, however, is another story.  I got my first mac yesterday, a nice new macbook pro.  I will say that the size and battery life are a wonderful step up from the Dell I had when I worked at Lockheed.  Learning a new OS, however, is killing me.  I feel kinda clueless when I have to use Google or ask a friend just to figure out simple things like, where is my terminal?  How do install apps?  I don't care what they say, the idea that Apple products "just work" is laughable to me.  I'm still trying to find an email client that I'm happy with.  That said, it's been one day, so I'm sure that OS X will grow on me.  The sad thing is that I'm probably a month away from hating Windows and OS X.  I'll be one of those dual users who is always wishing for the features of the OS I'm not using at the moment.
The final challenge, which I will start today, is the new software stack.  CareKinesis is using a lot of tools that I've only read about.  I'm thrilled for the challenge of learning new software, and the joy of finding great ways to use it to make myself more effective, but I'm also very aware of how mentally straining it will be to learn 2 or 3 new tools per day for a few weeks.  Meanwhile, my 2 man team will be making some major decisions in the next few weeks about some additional tech we want to add to the stack.  There will be a lot of brain drain for a while, but it's going to be a fun ride.

A grand new adventure

On Friday the 7th of February, I will officially end my career at Lockheed Martin, after almost 8 years.  It has been a wild ride, but I'm glad to see it end.  perhaps in a week or two I'll write more on why I'm glad to leave, but for today I want to focus on what's next for me.  My grand new adventure.

On the 24th I'll start me new job at a small healthcare company, CareKinesis.  The company is focused on medication management, essentially helping manage the drug regimen for people who take lots of different pills on a daily basis.  We write software to handle the prescription process, have a call center, a pharmacy, etc.  My role will be as a "Developer Happiness Engineer", and it will be awesome.  As we roll things out, I plan to blog about the details, and keep all of you up to date.  I'm particularly excited about this company because they are so open and community friendly.  This new opportunity will let me contribute to open source more, blog more, and generally be involved in the software community, and I'm psyched about it.

What's a Developer Happiness Engineer?  Glad you asked.  The job is simple, do everything you can to ensure the development team is happy, and never has to do anything that isn't fun.  We automate builds, integration, testing, and deployment.  We then layer monitoring and alerting on top of that, with a super awesome dashboard to give you instant insight into all of the above.  The idea is that, eventually, a developer will be able to write new code, push a button, and 5 minutes later, their code will be on the production system.  They can then look at graphs that show the tests passing, the integration work succeeding, and various metrics of system performance, with lines indicating when their code hit the various systems.  They can look to see if their memory bug really got fixed, their compression scheme worked, or if their refresh times really went down.  We've got a long way to go to get to that point, but it will be a fun ride.  If you want to know more about what this sort of system looks like, take a look at codeascraft.com (Etsy's blog).  They're already doing most of this, and are basically the gold standard for it.

Anyhow, that's what's next for me, and I'm super excited to get started.