Picking the right framework

It has all been done before

For someone starting out building applications for web/mobile/etc, it can be absurdly difficult with all the choice. Should you use Ruby? Python? Haskell? Elixir? C? Rust? Go? Erlang? Node? PHP? Java? How about frameworks? Ruby on rails? Flask? OK, perhaps choosing is too difficult. What about a nosql database? 225 and growing, so not easy. Perhaps it would be easier to figure out a deployment model between containers and paas and configuration management tools? Or front-end development frameworks?

Luckily, it is not all that important. New does not mean necessarily better, but you have to have a good overview of what you are doing and why.

For instance, every story has been written and every tale has been told. Did you know that every plot can be broken down into seven basic plots?

• Overcoming the monster
• Rags to riches
• The quest
• Voyage and return
• Comedy
• Tragedy
• Rebirth

We are probably not going to do it better than our predecessors. There are reasons for that - concentration, speed, and memory which anecdotally has been slipping over time¹, but before I digress and start telling tales of my youth where I went to school walking uphill in the snow both ways², the same keeps happenning in an accelerated cycle in the software world. It is quite the microcosm, a universe to itself, if you will.

As far as I can see, there are major cycles and the fundamentals of how to build things stay identical given the context of systems engineering which now has a shiny new name - devops. In the context of product development, systems engineering is referred to as a process of value-stream maps. In fact, I should probably write about systems engineering. However, repackaging does not make something novel, it is just a new application of an older, more fundamental technique³.

Major cycles and trends

• Data size relative to fast memory (shards vs large storage)
• Computation done on the server or client (thin vs rich clients)

The software world keeps oscillating between server vs client architectures and sharded vs monolithic data sets. With the onset of cloud computing, we are currently in the cycle of sharded data sets and thin clients (SAAS). While the capacity of RAM is growing, there is also a large proliferation of clients (IOT). Whether all these devices need to connected to a central mainframe which holds all the data (aka. gmail) is a different story, but as usual, things will change. There is also a fundamental need to account for network speeds across locations and a movement towards privacy.

So what happens when storage devices become as fast as memory or memory becomes as cheap as disk? Why would someone want to use map-reduce algorithms when they can do an in-memory sort?

What will likely stay the same…

Having watched this a few times now, here is what I have gleaned:

• Data structures and storage: Choose the right structure - sets, hashes, etc. And before picking any tools, know your operational aspects, consistency guarantees, latency needs, throughput/capacity requirements, and whether you need to replicate the data or not.
• Caching: A variety of caches are always necessary in different layers.
• SMP: Single thread cpu speeds have flatlined⁴. However, if you are coding in C, you can probably do it single threaded pretty fast. However, between programming with message passing or shared memory/cache coherence, I will take message passing every time.⁵
• API: Communication between processes is important. Whether you are doing rest, soap, rpc, or other message passing.
• Frameworks: There are no good frameworks. Every framework has a tradeoff where you will spend a ludicrous amount of time trying to do something you need that would have been trivial without it. And to figure out the trade-offs, you will need a lot of experience with them. A catch-22.

And translating that to plain english in the context of web development, you probably do not need to deal with petabytes of data or billions of concurrent connections, and it is easier to scale up than out, so the minimum toolset would include:

• A structured data store. Pick an SQL database. Learn SQL properly.
• An object cache. Redis and memcache usually work. Avoid mongo.
• A language that supports good concurrency out of the box to build a REST API. Golang/Elixir come to mind among others. Pick something that is not node.js⁶.
• A library to parse the data to display it for your front end application⁷. Note that I am not saying to use a framework or to build a single page app, but you will need a way to display data that has been fetched from the server. If you want to use a framework, by all means, do so, but avoid the single-page apps as they are pretty tough.
• Common sense as to when to build something monolithic or split it out into a service. This type of split of front and back makes it easier to maintain and manage while retaining flexibility.

The good news is that the languages and technologies do not really matter, it is whether you have a good foundation. The problem is that a good foundation comes with experience. And experience is what you get when you don’t get what you want.

tl;dr - Keep it simple and go out and build things.