Scouring the Business of Technology

Business Model of Apple


Apple has just 8 percent of the global phone market, but makes about 80 percent of the industry’s operating profits.

Wall Street analysts expect Apple Inc. to post a profit of $9.2 billion for the January to March quarter when it reports on Tuesday. That’s roughly in line with the profit expected from the world’s largest oil company, Exxon Mobil Corp.

The majority of the profit will come from iPhone sales, especially now that three of the four national US wireless carriers -AT&T, Sprint, and Verizon- sell the phone.

Siri-Controlled Car

Proof of concept of Siri remote car control, courtesy of Brandon Fiquett.

MySQL High Availability


  • Baron Schwartz aka Xaprb dislikes MMM as HA tool
  • Traditional setup: master-slave replication
  • At a specified interval – the heartbeat – the clustering solution will see if your MySQL instance is still running. The default heartbeat is often something like 10 seconds and minimum granularity tends to be one second. Problem: What happens between these seconds the clustering solution is completely unaware of. If you have 5000 trx/sec, you could have fifty-thousand failures before an attempt to fix the error is made.
  • The clustering solution takes small peeks into MySQL from the outside, but other than that MySQL remains a black box. Problem: Say that there are network errors. This causes your transactions to fail, and it also causes replication to fail. So which one failed first? If transactions started failing first, and replication was still working until the end, then you are fine and you can fail over to the other node. But if replication failed first, then you will lose data if you fail over, because not everything was replicated. Your clustering software has no idea, it wasn’t looking when the error happened.
  • Typically the clustering software itself will have some communication going on, which has the benefit that it verifies that the network connection between nodes is ok. This is in itself useful, sure. Problem: But just like above, if the clustering software detects that network has failed, it’s still mostly unaware of the state of MySQL. The failover decision is done blindly.
  • In a typical setup like above, the clustering software is actually just checking whether there is a network connection between the two MySQL nodes. If yes, that makes it happy. Problem: Nobody is really checking whether the application servers can really connect to those MySQL nodes! This is one of the most classic errors to make in software programming: when testing for error conditions, you are not testing the thing you actually want to know the answer to, but testing something else. Kind of like the guy in the joke who was searching for his keys under the lamp where there was light, not where he actually lost the keys.

Henrik Ingo’s favored solution: Galera (synchronous multi-master replication)

  • Thanks to synchronous replication and Galera’s quorum mechanism, no commits are lost anywhere. When the failure happens, it will be detected as part of the replication process. The Galera nodes themselves figure out which node is thrown out of the cluster, and the remaining ones – who “have quorum” – proceed to commit the transaction. Application nodes that connected to the failing node will of course receive errors as their commits fail.
  • There is no need for maintaining master-slave state, virtual ip or to do any failover. The application can connect to any of the Galera nodes and if the transaction succeeds, then it succeeds. If it fails, retry with another node.
  • As a side comment: since the replication is synchronous there is no slave lag as you are familiar from MySQL replication, which can also cause you to lose data. This is not a weakness of clustering frameworks, but a strength of Galera compared to classic MySQL replication that most people out there still are using.

Caveat: This works if you use JDBC (see Oracle blog example)

So in short, when using a Galera cluster, you should use mysql:loadbalance: in front of your JDBC connection string. This allows you to then give a list of MySQL nodes, which are all writeable masters. The JDBC driver will connect to any one of them and commit the transaction. If a node is not available, it will just try another one. (If a transaction was already in progress, it will fail with an exception, you can then retry it and it will just connect to a new node.)

Related links:

MySQL Master HA (MHA) tool – by Yoshinori Matsunobu

SkySQL commercial support for MHA

Three Database Clustering Users – by Josh Berkus


Networking as a Service

Courtesy: Cisco, GigaOM

Courtesy: Mozy

Note: This is a reprint from Open-E Blog

There is  plenty of talk about Bonding and Multipath IO, but it is very difficult to get solid information about either one. Typically what documentation can be found is very bulky and the most important practical questions go unanswered.

As a result the following questions are often heard:

  • When should I use bonding and when should I use multipath?
  • I was expecting better throughput with bonding, why am I not seeing this?
  • My RAID array shows 400MB/sec with local test, how can I get 400/MB sec outside?

Before we answer the above questions, let’s understand first how MPIO and bonding works.

MPIO allows a server with multiple NICs to transmit and receive I/O across all available interfaces to a corresponding MPIO-enabled server.  If a server has two 1Gb NICs and the storage server has two 1Gb NICs, the theoretical maximum throughput would be about 200 MB/s.

Link aggregation (LACP, 802.3ad, etc.) via NIC teaming does not work the same way as MPIO. Link aggregation does not improve the throughput of a single I/O flow.  A single flow will always traverse only one path.  The benefit of link aggregation is seen when several “unique” flows exist, each from different source. Each individual flow will be sent down its own available NIC interface which is determined by a hash algorithm. Thus with more unique flows, more NICs will provide greater aggregate throughput. Link aggregation will not provide improved throughput for iSCSI, although it does provide a degree of redundancy.

Bonding works between a server and switch. Numerous workstations using each using a single NIC connected to the switch will benefit from bonded connections between the switch and storage server.

MPIO works between a storage server and the client server, whether or not there is a switch in the path.

With these basics fact, it will now be easier to answer our questions.

Q: When do I need bonding, and when is multipath appropriate?

A: Bonding works for a NAS server with multiple workstations connected.

MPIO works between hosts and initiators on FC or iSCSI.  An example of MPIO configuration with a performance test showing 200MB/sec using dual Gb NIC’s is demonstrated step-by-step at: How to configure DSS V6 MPIO with Windows 2008 Server .

In short:

  • Bonding works for NAS
  • MPIO works for SAN

Courtesy: Enterprise Irregulars

Point 1:  NoSQL technologies require more investment than Relational to get going with. 

The remarks from Netflix are pretty clear on this.  From the Netflix “Tech” blog:

Adopting the non-relational model in general is not easy, and Netflix has been paying a steep pioneer tax while integrating these rapidly evolving and still maturing NoSQL products. There is a learning curve and an operational overhead.

Or, as Sid Anand says, “How do you translate relational concepts, where there is an entire industry built up on an understanding of those concepts, to NoSQL?’

Companies embarking on NoSQL are dealing with less mature tools, less available talent that is familiar with the tools, and in general fewer available patterns and know-how with which to apply the new technology.  This creates a greater tax on being able to adopt the technology.  That sounds a lot like what we expect to see in premature optimizations to me.

Point 2:  There is no particular advantage to NoSQL until you reach scales that require it.  In fact it is the opposite, given Point 1.

It’s harder to use.  You wind up having to do more in your application layer to make up for what Relational does that NoSQL can’t that you may rely on.  Take consistency, for example.  As Anand says in his video, “Non-relational systems are not consistent.  Some, like Cassandra, will heal the data.  Some will not.  If yours doesn’t, you will spend a lot of time writing consistency checkers to deal with it.”  This is just one of many issues involved with being productive with NoSQL.

Point 3:  If you are fortunate enough to need the scaling, you will have the time to migrate to NoSQL and it isn’t that expensive or painful to do so when the time comes.

The root of premature optimization is engineers hating the thought of rewriting.  Their code has to do everything just exactly right the first time or its crap code.  But what about the idea you don’t even understand the problem well enough to write “good” code at first.  Maybe you need to see how users interact with it, what sorts of bottlenecks exist, and how the code will evolve.  Perhaps your startup will have to pivot a time or two before you’ve even started building the right product.  Wouldn’t it be great to be able to use more productive tools while you go through that process?  Isn’t that how we think about modern programming?

Yes it is, and the only reason not to think that way is if we have reason to believe that a migration will be, to use Stonebreaker’s words, “a fate worse than death.”  The trouble is, it isn’t a fate worse than death.  And yes, it will help to have great engineers, but by the time you get to the volumes that require NoSQL, you’ll be able to afford them, and even then, it isn’t that bad.


Get every new post delivered to your Inbox.

Join 53 other followers