Our original Real time Slack Bot Ultron was a AWS Cloudwatch Bot. It served its purpose well and saved us from lot of performance and downtime issues with alerts that are not available via Cloudwatch. All was well but its greed for more compute resource knew no bounds. As our infrastructure grew so did its memory and CPU consumption. It was still manageable to run it as a Cloudwatch slack bot so we didn't think much over rewriting or optimising it. So how did it die for UltronEx to rise from its Ruby ashes in Elixir? We been running real time analytics stream testing of our events for some time and with the number of events that flow in, its quite hard for anyone to follow a particular event or their activity stream. One of the engineer from our mobile team had an idea to track event id and push those msgs to individiuals as a filter. Eventually he left and his project "Big Nose" went away with him. Mobile team after his departure required a similar solution but not limited to just id but any keyword matching and the quickest way was to add the similar functionality to our current RTM bot Ultron. Couple hours of work and it was live. Then its ever increasing hunger kicked in, with the amount of events flowing in real time and each to be searched for matching keywords in the message and download any attachment available to search in took its toll on Ultron. It became sluggish dropping messages freezing up in responses, missing to forward matches as more and more people used it.

As a developer that was quite unacceptable and to see something once so prized be fading under its own load was heartbreaking. For sometime I was looking for something worth building in Elixir to see adoption ease for a Rubyist and evaluate the performance gains that are claimed. Hence UltronEx was born in Elixir over a weekend. Its main focus was the RTM based matching message forwarding. It had to focus on 3 things that Ultron wasn't able to keep up with:

• Ingest all incoming messages
• Look for a match in message body, download any attachment and check for a match
• Forward message with attachment when a match is found.

So that was how Ultron died and UltronEx was born but was UltronEx any better than Ultron ? I will let the stats do the talking

Ultron CPU: 58 Load average: 1.0

UltronEx CPU: 0.0 Load average: 0.0

Once UltronEx was live for a moment I thought I got something wrong as there was no CPU or load average showing up. So Enabled In-depth monitoring on Digital Ocean. As both of them run on a 1GB 1 vCPU droplet for message forwarding RTM purpose and the stats were same as via top. It also shows when UltronEx peaks Ultron died under the same load.

Ultron

UltronEx

Not willing to admit that Ruby could be doing so dismal added New Relic to see If it was really just the bot that was using the CPU and load average and the result was same.

Ultron

UltronEx

Here is a more detailed htop output to make sure there was no additional processes running that could be the differentiating factor between the two.

The BEAM performance was just too good for MRI to keep up. Elixir was eating Ruby up for lunch.

Even when there were no matching jobs set for it to process any incoming messages. At times Ultron had a 2 second delay over UltronEx in responding to commands due to incoming messages.

From language perspective Elixir resembles Ruby quite closely making it easier for someone to write code in. The functional paradigm of the language takes time to get your head around and you won't master it over the weekend. Pattern matching does show its prowess in Elixir even when you don't write much functional code.

Ruby is a great language and coupled with Rails is an amazing web stack. Seeing is believing so I am looking forward to exploring options that better suits use cases than to just use one particular language for everything due to the comfort level.

We are currently running a hackathon to write Ultron versions in Go, Kotlin, Rust, TypeScript to compare performance and adaption rate for Rubyists.

The code for UltronEx is avaialble on Github