Sidekiq (Redis) vs Shoryuken (AWS SQS)

DISCLAIMER: This post is not intended to compare Sidekiq and Shoryuken implementations. Both are very similar, actually Shoryuken started as a shameless copy of Sidekiq <3 open source. The idea of this post is to compare Sidekiq as a Redis based queue system with Shoryuken as a SQS client.

Resque > Sidekiq > Shoryuken

Before Sidekiq, Resque was the main option to go for background jobs for many Rubyists. But Sidekiq changed that, mainly because it uses threads instead of process forking (Resque). No matter how many improvements MRI is receiving on process forking, we can't get the same performance (usage of resources) using forks as we get with threads. You can check some Sidekiq testimonials confirming that.

But the game is changing again - it's Shoryuken time.

With Sidekiq:

With Shoryuken:

Perfomance

WARNING: SUBJECTIVE PERFORMANCE TESTS!!!

Are you wondering about performance? I would be, Redis is super fast.

To test performance, I created two projects:

Test plan:

Result:

Although the jobs consumption result was pretty close, sending jobs with Sidekiq is clearly faster than with Shoryuken.

The jobs where enqueued using Heroku (Amazon EC2, us-east-1) via heroku run rake populate - this Rake task is available on the linked projects above.

Yeah, Redis write performance is amazing. But is 55.63ms much?

Every time you run these tests you will get a different result, but in most cases Sidekiq is slightly faster to consume and clearly faster to send jobs than Shoryuken.

More than once while trying bigger batches with Sidekiq, I got less requests on PutsReq than the jobs sent. Shoryuken was consistent, the number of jobs sent was the number of requests shown on PutsReq.

Pricing

SQS is a paid solution. You can run Sidekiq for free using Heroku with something like Redis To Go.

Nope! SQS can be also free or cheaper than Sidekiq:

Shoryuken

Shoryuken pricing:

SQS pricing:

Sidekiq

Sidekiq pricing:

Sidekiq Pro pricing:

Redis To Go pricing:

From Sidekiq to Shoryuken

Did I convince you?

Have a look at the migration steps: From Sidekiq to Shoryuken.

Conclusion

There's no silver bullet solution. If your workers aren't thread safe workers, maybe you should stick with Resque. If the performance difference between Shoryuken and Sidekiq matters to you, go with Sidekiq.

The point I'm trying to make is that in general for background jobs in Ruby, IMO Shoryuken/SQS is the new main option to go. It's fast, durable, distributed, auto scalable, cheap (or even free) and you don't need to care about the infrastructure, AWS will handle that for you.