On my travels I often see people getting into trouble by assuming that messages will be processed in the same order as they are being sent. If the there was a list of fallacies for asynchronous messaging this one would be:
Messages will always be processed in order
The reason why this is not true mainly comes from the fallacy that the network is reliable. This turns out, often painfully, to not be true and usually leads us to use message queuing systems like MSMQ, RabbitMQ, ActiveMQ, etc.
Queuing systems helps us mitigate the "network is reliable" fallacy by providing us with store and forward capabilities. This basically means that a sent message will be stored locally until the infrastructure can transfer the message safely to the recipient. To do all this in a performant and scalable way queuing systems usually won't supports ordered delivery since that drives the complexity up making the former hard to achieve. You can think of this like a multi lane highway, some cars will travel faster than others, due to size, speed,number of stops,etc. This leads to messages overtaking each other and there for reach their destination in random order. Even if they do arrive in order there can be multiple concurrent threads processing them which leads to even more rearrangement.
But every thing seems to run fine on my machine?
This is what make the whole thing so tricky. Systems under light load will most of the time process messages in order. This means that you will only see this happening when your system is under heavy load making it very hard to debug.
But I use one thread and have a queuing system that guarantees order?
Yes running in this mode will solve give you ordering at the infrastructure level. But unfortunately this is not enough. Communication frameworks like NServiceBus will manage exceptions for you by doing a few retries and then moving the message to a error queue. This means that a message can be retried from the error queue days after it arrived to its original destination. While this is happening you don't want your entire system grind to a halt while that message is waiting for someone to issue a retry. This means that even though the infrastructure might support ordered delivery, messages can still be processed out of order at the application level.
So what should we do then?
There is unfortunately no silver bullet available that will solve all this for you. First of all you need to make sure that you never write code that assumes messages to arrive in a certain order, you can't even assume that they will arrive at all .This leads you to write explicit code to make sure that your application behaves consistently in these situations. This is a good thing!
You can solve this in a few ways, some times the solution is to anchor the message in time so that the receiver can make correct decisions. Publishing price updates is a good example where having a ValidFrom - ValidTo will make processing order irrelevant. Another common way is to use sagas as a way to ensure that message ordering won't matter. I won't go into the details but if you're interested I've done a few talks on this subject:
Repeat after me: Don't assume message ordering when building a message based system