We’ve made the difficult decision to shut down our dedicated server plans on March 1, 2019. As you may know, we have not offered dedicated server plans since last year. While we understand the impact this will have on our customers, shutting down these plans will allow us to focus on improving and adding new features to our latest cloud product, Mojo Stratus.
Our mission has always been to deliver a world-class product for Magento stores. That is why we offered our own hand-picked hardware for years. First with plans using cPanel and then our own server management panel, Mojo Host Manager. Cloud architecture has caught up to bespoke server builds and we are proud to say that Mojo Stratus will half your response times compared to our dedicated hardware. Put another way, your performance will AT LEAST double, guaranteed.
The results speak for themselves, below are graphs from live stores we host showing New Relic average response times before and after the switch, denoted by the red line. After moving to Stratus their sites loaded faster and became more stable, while being able to handle more traffic than ever before.
The Mojo Stratus platform is built entirely on the AWS cloud stack and is hosted on Amazon's hardware and instances. It utilizes containers to automatically scale for a given store as needed. This means that you no longer have to worry about Ram or CPU restrictions.
We've also included Autoscaling, CDN, Web Application Firewall and DDoS protection as part of Mojo Stratus. We use a utility pricing model so that you only pay for what you use. Our Mojo Stratus plans start at $98 per month plus monthly sessions and storage.
We know that you will have a lot of questions about Mojo Stratus. We’ve put together a FAQ document with more information. If you still have further questions or concerns, simply reply to this email and we will get back to you as quickly as possible.
If you wish to sign up today and start the migration process, you can do so at https://magemojo.com/.
Flashback to Imagine 2018 in April. As organizers for Meet Magento New York, we were invited to attend an organizers meeting. The meeting was to learn about the new Magento Association taking over the Meet Magento Association. This was the first time we heard about it and were surprised. At Imagine we learned that 4 community members would be leading it. The meeting was, well, not much. There was very little information provided. Our impression was the Magento Association was transferring ownership of Meet Magento Association.
Since Imagine there have been several more meetings. In these meetings the attendee types started to drift. First organizers from other events. Then Magento Masters. Then community celebrities. Then people from outside the Magento community.
We started hearing Magento Association was more than Meet Magento Association. Magento Association was more than events. But what was Magento Association?
The Magento Association idea started before the Adobe acquisition. The idea came from the fear of losing Meet Magento Association. The M eet Magento Association is the association created by Thomas Goletz in Germany. There's a long history of Meet Magento DE, Meet Magento, and the Meet Magento Association. I won't get into the nitty gritty of it. The important thing to know is that the Meet Magento Association was formed to produce the DE event and to spread more Meet Magento events around the world. But Meet Magento Association never owned the trademark for Meet Magento. Since then Meet Magento Association events have been spread around the globe and in this regard it's been very successful in doing so.
As a member of the Meet Magento Association we pay a yearly fee to the association. The money earned from the DE event (which is the only break even Meet Magento Association event and also earns profits) is used to fund the Meet Magento Association's yearly budget. There has been contention in the DE community for several years. The contention is around having an organizer, one who's not earning a living from Magento, organizing a Magento event. There are opposing pov's here. On one side people question the over-commercialization and revenue generation of MMDE. The organizer yearly fees, where that money goes, and what we as organizers receive back. On the other side is an organization who has full time employees devoted to the Organization and producing MMDE.
Over time, for various reasons, the founder of Meet Magento Association became more involved in China. The contention was growing stronger. The Meet Magento trademark was owned by neither Meet Magento Association nor Magento. Magento saw all this as a risk of losing the Meet Magento events - over 25 worldwide. The German laws are complicated, to say the least. It was easier to form a new association and buy the trademark than it was to take ownership of the existing Meet Magento Association. At this point, the Magento Association was very much about protecting the future of the Meet Magento events. But once you create a non-profit association, how do you manage a community with it?
Magento looked at Acquia and the Drupal Association. They learned that looking back, the Drupal Association really wished they had engaged a professional company to help them. Drupal Association spent years self-teaching themselves lessons that could have been avoided. From these conversations Magento found SmithBuckland. They engaged SmithBuckland to start the Magento Association. Hopefully, this would avoid the wasted time and costly mistakes of the Drupal Association.
Enter SmithBuckland. SmithBuckland has been leading these meetings as a discovery to help form the structure of what Magento Association should be. Up until now Magento only knows they wanted to protect losing the Meet Magento events. Through these meetings they start to realize maybe there's more than just the Meet Magento Association. Maybe they can add more events. Maybe there's potential here for more than just events. That's when more and more unusual people were invited to the meetings. Largely to explore and see what community opportunities there which evolve with every meeting.
Three critical mistakes were made here.
First, they tried to keep this quiet. The reason being they didn't exactly know what they wanted or how to do it. They didn't want the community to form pre-conceived notions about what Magento Association was. Instead, the community was left to draw their own conclusions. Most of us do not have good experiences with Magento's business practices. Nor with their involvement in our events. We were left to draw our own conclusions. Conclusions through the lens of previous negative experiences.
Second, the people invited were not described. Yes, Magento put up a form to fill out if you're interested (which I filled out and never was contacted - another employee was). Yes they reached out to Masters and Event Organizers. But many others appeared and it very much looks like you had to be a friend of certain people to get invited. From a Community POV this is bad optics. A lot of the drift was simply miscommunication between the people inviting attendees and people attending not following the rules. Say what the guidelines are for the people you invited and stick to those guidelines. Enforce your attendance policies at meetings. It's important for the appearance to not show cronyism in order to build confidence and trust in the process.
Third was communication, or lack of. This goes back to keeping things quiet. They should have come out and said clearly what the business goals were driving the decision to create the Magento Association. Just say it. Make sure you announce the form to request your involvement by more than just a tweet. The communication strategy with community needs a major upgrade. As it stands now you pretty much need to hang out on twitter all day to know what's going on (please don't kill me Sherrie - I love you).
SmithBuckland isn't going to be here forever. But before they go they need to figure out who's going to pay for this thing. That's right, it's not Magento, it's us. Surprise! At least it was for me. For some reason I thought with all the unicorns being thrown around that events would be supported and possibly even help funded.
Why would I think that?
Last year at MMNY we didn't receive anything from Magento, not even some stickers. But they wanted free tickets. They wanted private meeting rooms. In fact, we weren't even allowed to say we were an official Magento event. I was personally reprimanded for saying this. I was given a long legal clause to say that sounded like it was the end of a pharmaceutical commercial. It basically said we're not an official event, Magento and Meet Magento Association have a cooperation agreement, and Magento supports the Meet Magento Association. But never, ever, say you're an official event.
Talking with fellow organizers there are many issues with Magento and the Meet Magento events. From stalling on dates, to trying to insert speakers into keynotes only 2 weeks in advance. It's really been unpleasant working with them on the events. In fact, Magento has been downright been obstructing us in some instances.
Here we are with the Magento Association and a decision to make. The Meet Magento Association was collecting dues for basically just the right to use a license to "Meet Magento" (which they never owned). Why would we be interested in participating in Magento Association at all? Events like Titans and Unconference and others have been fine on their own. We've been all been fine on our own, arguably better off without Magento / Meet Magento Association. Why would we want to be involved in this new Magento Association, especially if we're going to be the ones likely to fund it by now sending yearly dues to Magento Association...
Because it's a chance for a fresh start. It's a chance to do this Meet Magento Association thing over. Magento's control should at least in theory be limited due to the 503c status. This limits their controlling interests on the board to a single seat voting/non-voting. The actual structure remains in discussion. What I would like to see is X board seats. The community makes nominations. Nominees accept/decline. Then the community votes on the nominees for the final board seats. It would also be a good idea to decide the distribution of board seats and types to the types of stakeholders. For example in the community - developers, small business merchants, technology partners, etc.
What is Magento Association then. It was the saving of Meet Magento Association. From there it's not sure what it wants to be. In SmithBuckland's own words: "It was also challenging to prioritize stakeholders as it is unclear what is the #1 problem the Association needs to solve or opportunity to capture".
When you hear Magento say "you tell us" they are really serious. They do want to figure it out. They need to figure it out. As an Organizer I'm not sure what they can do for us. We're already established. We already know 90% of what we need to know. If anything, a central calendar of event dates organizers could collab on, to ensure a good distribution of events, would be nice. Some say a simple spreadsheet could suffice. But there does need to be an arbiter and some process to ensure fair play and dispute resolution. Does that need an association? Is it worth paying thousands of dollars per year? Is it worth the hassle of dealing with Magento?
No, we're going to need a heck of a lot more than that if we're going to participate only as a Meet Magento organizer, on our own free will.
But hey, at least right now we have the chance to say what that something more should be. Should it provide efforts focused on building Merchant / Small Business owner communities? Should it provide efforts on security research and protection?
We're on board with all these ideas. We love the community and have always supported community efforts, even at our own expense, and my personal expense. And will continue to do so.
You the community member needs to tell Magento. What is the problem you see that needs solved? What opportunities do you see to be captured? How do you think it should be funded?
The launch and improvement of Mojo Stratus has been a bumpy road. Stratus was launched just before Meet Magento New York. Stratus was our major release and plastered on every wall at MMNYC. We wanted to do something innovative and different with Mojo Stratus.
Stratus is different. Rather than continuing down the path of traditional server offerings – i.e. you get a server, things are installed on it, and you have this big monolithic piece of hardware running whatever you need – we decided to use containers. A long time ago we looked at Docker and containers to use when developing our panel Mojo Host Manager. That was several years ago and containers were an unstable parlor trick. Great for production if you were ok with your production site constantly being on fire.
Support for containers is now widespread especially with Google releasing their Kubernetes technology. We decided to use containers to build the services for Mojo Stratus. All the services your average Magento 2 store would need, and our initial release worked despite several issues. We tweaked, and tweaked, and got through the Thanksgiving sales season unscathed. Then in December major systemic issues began to appear with no obvious explanation.
First, we saw database problems. On Mojo Stratus, Amazon Aurora hosts all the databases. Aurora's main strength is scaling out read replicas. If you have ever tried to set up your own MySQL master-slave setups or other DIY clustering, then you know it is not very fun. Aurora makes this easy and we wanted to have read replicas for future scaling. It is still MySQL, though, and subject to the same problems you might expect from high usage and other bugs in MySQL. What we saw were patterns of locks in MySQL which would freeze all transactions on all stores for a few seconds. Then a huge spike in active connections as all the traffic on Stratus backed up into Aurora. Site alarms would go off, the sky would fall, customers noticed, cats and dogs living together, mass hysteria. We needed deeper insight into Aurora and fast.
Getting insight into Aurora was not easy. We needed something pre-built. Basic stats from AWS or even getting them yourself from the MySQL engine are not useful. That isn’t a fault in Aurora. For an application specific problem you want to see what queries are happening during a failure event. After some trial and error, we came up Vivid Cortex (https://www.vividcortex.com/) and hooked it into Stratus. Vivid Cortex provides tons of information about what queries are running. Vivid Cortex helped us answer questions like:
What queries are running
How often do certain queries run
What databases run certain queries the most
What queries are the most time consuming
We can’t give Vivid Cortex enough love for watching database performance
After gathering a lot of data, we found a pattern. Stratus would lock up during certain types of queries. They would occur during certain actions on particular stores and lock everything up. On top of this, the Magento 2 crons were going haywire. Magento 2 has a bug (https://github.com/magento/magento2/issues/11002) where the cron_schedule table can inflate to infinity. Crons start running all over each other and destroying your server in the process. Certain extensions can have particularly heavy cron tasks within them. Either due to necessity or inefficiencies in the code. And they all run at once.
With a bug causing locks, crons waging war against everything, and bad queries coming in, we had a recipe for poor performance. Even with the massive resources of Amazon Aurora. We used multiple approaches to bring everything under control. First, we notified customers about problem extensions and site code. Second, we started limiting crons ultimately creating our own extension to manage them. We've also pushed forward support for various services like Elasticsearch. Search in Magento should not be the default through MySQL if possible. We are also working on a MySQL reader extension for Magento 2 to take full advantage of Aurora scaling.
Those solutions helped, but we still had issues on the file system side. The file issues left us baffled. We had no issues going through the busiest days of the season Black Friday and Cyber Monday. From the start we have been using ObjectiveFS. An amazing filesystem that lets us store data on S3 and it gets pulled locally. Files are cached to speed up performance, running anything direct off S3 would be very slow. Especially Magento where thousands of files maybe opened and called on a single request.
ObjectiveFS would use a lot of CPU , spike iowait affecting every customer. That issue started in December and was not a problem in the last few months of 2017. The iowait spikes became more frequent and severe, unrelated to specific traffic, and we had to do something. We shopped around for other file system solutions and came up Weka.io. Weka.io is a high performance file sharing solution that offers low latency and high throughput over the network. With most file systems like Weka, you can't get the low latencies needed for an application like Magento.
Weka promised it all with file system latencies in the microsecond range. Well known share tech like CEPH etc all have times in the millisecond. It used its own kernel driver and relied on i3 instances and NVME storage. Again, when loading 1000 plus files per page load, you need low latency, that’s why the switch to SSDs was so important early on for MageMojo. It looked like a drop in replacement file system and we fired it up and got It working.
The initial results were promising. Weka handled 300k request per second for files over the network without issues, and writes were no problem. You could write from one place and see the file nearly instantly from another source. Many frontend and backend parts of Magento write to a file and display it via an ajax request (product image uploads). Where on other systems an image upload would work, but the thumbnail would not appear since the write was not fast enough.
After more testing, we went ahead and moved everyone off ObjectiveFS to the weka.io filesystem. We had a few issues with its configuration, and worked with their team to get everything set up correctly. For a while life was good. But Weka.io added significant latency to the load times, even with its microsecond response over the network. On average about a full second compared to the original ObjectiveFS system. The load time was a trade-off for what we believed to be stability.
In February we had a critical failure on the Weka.io cluster a few weeks after completing our migration to their filesystem. The system was designed to be redundant so that 2 storage nodes can fail without data loss. In our case 3 nodes failed, putting the data in the ephemeral storage at risk of recovery. A bug in the Weka.io software caused the entire cluster to become unresponsive and we were never given the full explanation from the Weka.io team, unfortunately.
We brought the stores online within 24 hours using an older copy of the data. In the following days we restored files as we could and helped bring back stores using their more recent data. We stabilized again on ObjectiveFS. We got back to business . ObjectiveFS was not as bad as we recalled, having fixed some other issues Aurora related. And not long before the Weka failure, we learned about the Meltdown vulnerability.
This is the real kicker on top of it all. Once Meltdown became public knowledge, we learned that Amazon had secretly patched all their systems in mid-December. Meltdown patches coincide with the random systemic issues. We thought they were ObjectiveFS specific. It was not until we went back to ObjectiveFS that we realized there could be a connection. We also had AWS Enterprise support confirm the patching timeline. They were under embargo not to reveal the vulnerability.
In hindsight, that change severely impacted our file system performance and we know the Meltdown patches can hurt the specific load created by Magento especially stat calls, and Magento makes thousands of them per request. Post Black Friday, multiple issues converged to create a sudden unstable system. We failed to identify it correctly and tried to fix it with different technology. In hindsight, that was a major mistake on our part. A lot of sleepless nights paid on that debt.
With the realization about Meltdown and a new look at ObjectiveFS, we resumed testing and making more tweaks. Performance was better but not the best we hoped for. More and more updates gave us incremental improvements. In the first iteration we used multiple ObjectiveFS mounts. They covered many stores on a given physical node, and those mounts existed on all workers in the Stratus cluster. As a store scaled out, the containers already had the files available. Requests would cache the files a container needed on the respective node over time. But with many stores sharing a mount, the cache sizes became very large relative to a store. With such a large cache, any given request needed to fetch a lot of specific files from a large haystack. Testing confirmed it was a major bottleneck.
For Stratus 2.5, the current generation, we moved to having a single ObjectiveFS file mount per store. Each store has its own file cache local to a node running its containers on disk and in memory. We launched Stratus 2.5 2 weeks ago and it has solved every file system issue we’ve received complaints about, especially update slowness in Magento admin. Site performance is faster than ever, according to our New Relic data every store is 30% faster now. Stores with heavy file operations on load show even more improvement.
We’ve also added a lesser known feature called Stratus Cache. Stratus cache directly adds most of your code base into the container images we use for scaling. Stratus caches bypasses the file system for a majority of the system calls and improves performance while making scaling for large sales a breeze. If you are planning a large promotion or traffic influx, please let us know and we help get that working for you.
To contribute back to the community and improve Stratus, we’ve started making our own Magento 2 modules to address specific concerns we have about Magento 2 performance. Our first release was a complete re-work of the cron system in Magento 2. On Github at https://github.com/magemojo/m2-ce-cron . By default the Magento 2 crons can take a server down in the right conditions and they constantly fight each other and run the same task multiple times. Our module eliminates that problem, because it causes issues with stores and vital cron tasks are missed.
Next we have our split DB extension viewable at https://github.com/magemojo/m2-ce-splitdb . Magento 1 CE allowed merchants to easily use a master-slave database setup with a dedicated reader. Stratus uses Aurora which scales by having seamless multiple readers in a cluster. Since M2 CE does not support this at all out of the box, we had to build our solution. We believe Community should be able to scale just as well as Enterprise.
As we near Magento Imagine, we are working on improving the dev experience on Stratus. We provide free dev instances which are the same CDN and stack used by any production Stratus instance. Going forward, we want to include more tools, tests and utilities to make a developer friendly environment. The primary feature will be Live Preview. At the click of a button, customers can create exact copies of their production store, including the database. Then developers can go in and make changes, commit them, run tests, and push to production. Preview sites will be storable so you can save different versions of the site and refer to them as needed. After the initial release of Live Preview, we will be adding tools to perform Selenium and unit tests.
Stratus is now the premiere platform for Magento hosting. Nothing can scale and run your Magento store better. We've come a long way and we are grateful for our customer's patience. Now it's time to get back to business and stop worrying about your server.
Many changes happen on the servers, from nightly yum updates, to our security scanner updates, to our server configuration updates. In order to improve our change management process we promise to do the following:
The second Meet Magento Vietnam conference in 2016 took us to Ho Chi Minh City, Vietnam in October of 2016.
We were grateful for a relatively uneventful trip through Shanghai, but left the airport much lighter than expected - our luggage had not yet arrived! Not that we could clear the customs check though. The airport had a power outage which took all of the equipment down. That guaranteed a quick exit from the airport, without local currency, since ATMs went down with the power out. Yay for cellphones - we were still able to get an Uber to our hotel. The complimentary dressing gown in the closet was a blessing since that was the only clean piece of clothing we had :)
We finally got some Vietnamese dong and were instant millionaires. With the exchange rate at about 22 500 to the dollar we were walking around with more than a million dong! We were all set for a colorful trip to the local markets to hunt for anything worthy of a conference attendee. We would have been happier exploring the city and meeting others in town for the event but it was intriguing to pick up some local traditional stuff for the event the next day. Besides traditional gear was the only practical choice - it seemed that dresses in Vietnam were typically shorter and quite comical on someone of my height. The best items are usually ordered in advance and tailor made to fit. No time for that though. The quick and easy tourist options would have to do. I got all primped the next morning with my Aodai over a pair of jeans, only to get rained on, on the way to the event. All that trouble only to arrive looking disheveled anyway!
It was amazing to meet a whole new subset of the Magento community as well as lots of really keen students. We made great friends who also made for wonderful networkers, tour guides of Ho Chi Minh City, its hottest club spots, dinner haunts and of course coffee shops. Vietnamese filter coffee is not for the faint hearted! This flavorful strong brew is bound to keep you alert for many hours! One of the highlights of this event was getting to know Thomas Goletz and watching him interact with the community. It was truly inspiring. Pictured here is Tra My Nguyen, one of the conference organizers, towering over Thomas Goletz. The conference venue was well set up with enormous screens which were great for large audiences.
Since we commented on the showers in our blog post covering Magetitans UK, we felt it fitting to offer a detailed account of the bathrooms we encountered in Vietnam. They are often appropriately called wet bathrooms. Sure enough, taking a shower gets everything wet. Bath tubs and shower cubicles are often not common place. Many shower areas we encountered doubled as the area with the toilet and basin. The shower head simply protruded from the wall in that room - really efficient use of space.
We made sure we had time to explore the many charms of this amazing country. Apart from museums, street food and cafes, we wandered over to Bui Vien, a densely populated tourist and backpacker area which appeared to be alive all night. It was a great place to hang out outdoors and people watch. It was also a great place to make friends and enjoy local delicacies prepared in front of you in a matter of minutes. We very quickly found our favorite locales for great coffee, meals and cocktails and were always amazed at the enterprising, friendly and hardworking nature of folks who we miss dearly. A warning though, the electrical wiring would make anyone cringe. No, this is not your typical data-center wiring standard!
Hahn at the restaurant a few doors down from our hotel, was always happy to see us and continued to surprise us with amazing dishes that were not even available on her menu. She was even happy to leave her restaurant just to walk with me through winding alley ways off the main drag so we could find fresh fruit.
I was able to establish a routine in the craziness of Bui Vien as if I had always been there. We found a charming hotel managed by 2 young men who filled the roles of concierge, porter, hotel reception, tour and travel office, cleaning staff and all round great guys. Our first floor room was carefully chosen with large windows so we could always feel like we were a part of the daily activities, which were distinctively different at various parts of the day. The breakfast period was perhaps the calmest but by no means quiet. Various store fronts and road side vendors popped up almost everywhere with motorcycles, bicycles, taxis, Ubers, and pedestrians buzzing around with no clear demarkation of sidewalk and roadway. Stores and street vendors set up little plastic stools or a rolling cart on the roadside for a breakfast of banh-mi (sandwiches), soup, rice porridge or Vietnamese omelettes. Many of these vendors vanished until lunchtime and vanished again until dinner, not there there was ever a shortage of delectable food options or a welcoming smile. We once popped our unannounced heads into a language Institute that seemed to be preparing for Halloween. The students welcomed us as if they were expecting guests. We wandered through the scary maze they were setting up left and left after great conversations and lots of pictures.
We were always amazed by the incredible friendliness and perseverance of folks everywhere. One of our favorite things was offering larger than usual tips to various people when the opportunity arose. It was so easy to communicate with as little as a smile and gratitude so it was wonderful to add a token of appreciation.The exchange rate was so heavily in our favor and what is a small token of gratitude in the US, seemed to make someones day in Vietnam. Our oarsman in Ha Long Bay certainly made our day, as did everyone we met in business and social interactions. He was great at finding the hot picture spots and directing us into appropriate poses using hilarious gestures. He would maneuver the boat with an oar in one hand while taking pictures of us with the other. He seemed to be skilled at using absolutely any phone or camera handed to him while still rowing with one arm :) Much to the amusement of us all, he also scolded me for my meek attempt at a smile when he tried to take my picture :D
One rule of survival was to quickly learn how to safely cross the street. I thought I has conquered this as a kid :D Stepping into traffic and maintaining a constant pace was the safest so motorists could anticipate your your next move. The throng of motorbikes, bicycles and larger vehicles simply buzzed around pedestrians and we did not observe a single accident involving a pedestrian. It amazed us, that even larger and fragile items like construction material, floral bouquets, and multiple trays of eggs and were all safely transported by bike. This was perhaps a common theme - things that may have seemed daunting or not possible back home were regular occurrences in daily life. We were sad to leave all this behind and hope to return many times. We keep in touch with friends made along the way and look forward to visiting in the future. Perhaps someday we can host some of them in the US.