Loadtesting MobileCoin Blockchain

Hey @Josh and team,

Hope this finds everyone well. Since I have a certain amount of cloud resources left over from my last load testing gig, I thought maybe I can interest you in a load test from both a heterogenous and a homogenous server topography - free of charge. I have some experience doing load tests with a range of tools and I miss it quite a lot, since my current role as a CI/CD does not include load/perf testing.

Since this may be of interest to other people, I thought we might use this page as a staging area for ideas around load testing, both scripted and unscripted.

Since I have a machine supporting sgx, we could even generate traffic that normally would occur between servers on some scale, but I think I remember seeing SGX support on the Microsoft Azure Servers

Interesting test scenarios would involve:

  1. “Normal” load ramp-ups

  2. Erratic or malicious behavior by one or more clients generating:
    1.1. payments transactions
    1.2. payment requests

  3. Erratic or malicious behavior by a validator node

  4. fail-over tests / load-balancing tests

  5. soak tests (particularly interesting, since infrastructure must be up without downtime I assume)

  6. Deployment-testing under load

  7. Load-spikes

If it is not a secret: How does the development team approach performance related risks? Can we capture outgoing traffic at all? Is it possible in theory to run multiple clients from the same IP? Do you intend to rely on the mob/crowd and a controlled growth of the community?

Cheers

Andrej

1 Like

Hi Andrej,

The short answer is that we approach performance testing with humility and caution. We definitely want to try to break the machine that we’ve built.

From a high level, I see two sort of vectors for breaking the system:

  1. denial of service
  2. validation errors

DoS has a few different layers. Basically, we want to handle any well-formed request that hits us, and since we have so many different layers of privacy protection built into the system, most of the normal ways that operators throttle unexpected network behavior aren’t available to us. We can’t throttle based on user pattern heuristics since we don’t have those analytics. The only way we can really throttle is based on IP, which is pretty well understood (but worth testing!!). Lastly, there’s a sub-category of DoS that comes from network partitions, which is harder to test but potentially more insidious.

Validation errors are more persnickety. If a user can trick the network into accepting an invalid transaction, that would be really bad. This requires some kind of fuzzing as well as hand-crafted attacks.

I think you’re proposing to help with the first one. Let me sync with @drakeley and the rest of the eng team to see how we can engage with you :).

Cheers,
Joshua

Hi Joshua,

definitely the first kind, although I may try my luck. Let me see if I can capture some traffic first and what I can devise to run a test.

Cheers
Andrej