Load In The Loop: Episode 11 – SoundApproach

Jordan Griffey

March 20, 2020
Load In The Loop: Episode 11 – SoundApproach

Load In the Loop: Episode 11 is here! Join Eric Hileman and Ivan Chepurnyi as they identify and fix some performance issues for SoundApproach.

Sponsored by blackfire: https://blackfire.io/

Follow us on Twitter! https://twitter.com/loadintheloop

Ivan: ivan.chepurnyi@ecomdev.org, @IvanChepurnyi
Eric: litl@magemojo.com, @ericvhileman


Ivan Chepurnyi: [00:00:00] Hey Eric. So not the long time ago. We just had another recording just two days ago, how is it going for you now?

Eric Hileman: [00:00:08] Good. Good. Yeah. It was just the other day. Got a crazy recording schedule.

Ivan Chepurnyi: [00:00:16] Yeah. I have to spend more time and next week with my kids. because my wife is going for a small trip to Ukraine and I’m going to be a home with the kids, so I cannot do any more recordings next week until she comes back. And, yeah. So we would try to put as much as possible as this week. So, how, how has it going on, with, business in general? I heard you release Blackfire recently.

Eric Hileman: [00:00:45] Yeah, we do. We have support for Blackfire now in the UI. It’s under the PHP settings. we have XD bug in there as well.

So if you go to PHP and you’ll see Blackfire and allows you to put in your key, enable it and get it to go.

Ivan Chepurnyi: [00:00:59] Great. So now we don’t need to be good. And if you are developers anymore to enabled like fire profiling and probably we can now profile any major store result. you know, like having. So a administrator actually taken a look at it.

So, so as, as it says, great news. So now even March, merchants can just enable Blackfire and a grade black fighter profile, bike fight account, and to specify server settings. Enable it and Zen just trunk profile button and see how, how is every Magento sort of perform. So

Eric Hileman: [00:01:36] no ticket needed.

Ivan Chepurnyi: [00:01:37] So the Rollo person to be real scared.


Eric Hileman: [00:01:46] it’s better this better than a Blackfire and if you turn on XD bug, yeah, your performance is gonna. It’s gonna change dramatically. But at least with Blackfire, you’re not interrupting any performance. So it’s much less intrusive.

Ivan Chepurnyi: [00:01:59] Yeah, it was Blackfire what I really liked said, in Blackfire you have, is this a possibility to, you know, just, specifically Dell, Hey, only Ford to ECU is, or four as the session or, you know, I want to profile, so when I’m viewing, I’m running a profile or all of the AZA users, they don’t have any, failings at someone profile.

So website. But for instance, if we talk about new Relic, new Relic has some overhead because it has a, I still have to capture some traces, homes, a execution plan. What was application? We go say a test, some special rules, like if it takes more than specific diamond starts to creating a trace. Of what’s, what’s happening, what was there.

and it still counts number of that way. Square is number of I operations to radius, let’s say, and so on. So neuro Lucas, very intrusive. It, it’s not that a lot of overhead, of course, comparing to, you know, ex Dubai GAM, but. Still? Yeah. Okay, so, Oh, that’s a intro.

Eric Hileman: [00:03:12] The intro thing.

Ivan Chepurnyi: [00:03:13] Yeah. Let’s record. Does the intro okay.


Eric Hileman: [00:03:16] Load in the Loop is sponsored by Blackfire dot. IO. I’m Eric Heilman, CEO and cofounder of MageMojo Magento hosting of New York city purposes program is to take some of what we’ve learned over the last 10 years of Magento specific hosting and evaluate one of our customers stores. We’ll share with you what we find in hopes that it may help improve your Magento’s skills and the performance of your Magento stores.

Along with me on this strip is my cohost Ivan.

Ivan Chepurnyi: [00:03:40] Hello. My name is Ivan. I am a one of a former Magento one core developers. now I’m independent consultant here in the Netherlands. I’m helping with performance and a lot of other crazy stuff like building complex. I seen her new speech be applications for Magento merchants in order to bring performance to the next level.

So, let’s, recap a little bit, on the topics that we were discussing. last episode. So we were talking a little bit about a lot of concurrent orders right in the system.

Eric Hileman: [00:04:17] Yeah. Pushing it up to a thousand, over a thousand. You think you could do 2000, 3000?

Ivan Chepurnyi: [00:04:23] Yes. And no problem. As soon as a, as a operation is, it’s happens on my skill side is not run in spiral.

So basically, as the biggest problem as I was explaining last time, is that my SQL, ever connection, creates a new, threat in operating system. So. Every single connection has basically, own, threat to operate was a data surgery’s data from the file system. It populates a memory. it also locks a memory when it wants to update something.

And let’s say when a lot of frets, try to update the same table, is there is a problem with negotiation of which frat commerce up to reach for it. And. It’s just not very scalable. Right? So my skill needs to fight with itself internally in order to protest more and more records. what works great in my skill, for instance, is there is a possibility.

to have a very high throughput for bright append only operations. So if you don’t need to update the record, it’s very fast process because

Eric Hileman: [00:05:35] you still want answers.

Ivan Chepurnyi: [00:05:36] Yeah. It’s, as soon as you insert a acquiring, a new primary key is a very fast operation. So it’s like just increment on an integer in memory.

And this increment on integer and memory is a very efficient, operation on any CPU. Yeah, well, whatever Fred it is. So this is fun. I work great. but,  the rest of the things are just right, into the file in append only manner and Zen, when you flash a transaction, you just. Updates any index also in the background.

So like Xenu and reskill versions, they are much more, better at dealing with this kind of concurrency, but there’s a problem still exists. when you work with things like stock, when you work with things like Salesforce counters, when you work with things like, Hey, I just graded the new order, but this is the same time, like after.

immune with second. I’m also running an update operation just to change the order status because we go to confirmation, Hey, it’s speed. So I just want to right away to update in order. And in this case, here’s where as the scalability issues are coming in because a Zen, you might. Health issues with an actual database because when a lot of people try to update the same record as it is a high chance of a weight log timeouts and deadlocks in Zandt, so how you can actually achieve much more better throughput.

Yeah. Is to have a worker, is that this responsible for the wool saber operation by itself. So this worker knows about all of the customers who wants to place an order, and Zeus worker can, properly batch my skill operations so it can decide, Hey, I’m gonna. You know, every free milliseconds. I’m going to collect all the pendant orders and I’m going to store them in the database, and then I’m going to defy all of those 3000 customers who as a result of Zeus orders, Sable petition, but I’m not going to do it one by one.

I’m going to do it in

Eric Hileman: [00:07:51] open and connection.

Ivan Chepurnyi: [00:07:53] I already have a connection. I just started. It turns action isn’t just transaction. I’m going to insert also orders in one transaction. So it’s going to be a batch operation for my skill is going to be much more simpler, much more performance, because it’s the same as you.

Just, insert let’s say, and follows on threat courts in, that the way stable, was in a single transaction for my skill. It’s nothing. And it can do it too easily. I remember loser was some software like hundred thousand, records per second that somethings that my scale can

Eric Hileman: [00:08:26] 200,000 or more.

Ivan Chepurnyi: [00:08:28] Yeah. So my is capable of much more as an people. We’d be people just think about it. Was a problem is the application implementation of support.

Eric Hileman: [00:08:38] Exactly. Exactly. That’s why when someone says on Twitter like, Oh, you’re, you’re limited by my SQL, and you’re limited by Retis when you’re using it as a message queue, like, no, you can do 200,000 inserts a second in my SQL.

And consumers, you know, are not consumers with rapid, but a, an a, a message. Broker, like there’s no way that that’s limiting your order through put, it’s the application and how the application is using it. Like stop blaming the, the stack. Like that’s not a thing.

Ivan Chepurnyi: [00:09:08] Yeah. It’s quite funny because, I, I hear some, so many people just deciding, Hey, we’re going to switch your database, let’s say to another one because we think it’s going to allow us more scalability.

But you know, like if you use the same approach, was this another  system? It’s still going to be the same problem. And South. Like you, you might get maybe 10 orders more per second or for exact, but that doesn’t make sense to change the whole stack as it’s you’re not using. Right. It doesn’t make any sense.


Eric Hileman: [00:09:38] What do you think about using Reddis as a, as a message broker? That was, that was new to me. Like, I guess Retis ad pub and sub to it. And now it’s like, to me, Reddis was always a key in value in, in memory key value database that’s used as a cash. So to hear it used as a message broker was new to me.

Ivan Chepurnyi: [00:09:57] it’s, it’s kind of interesting approach. I will say it’s good from point of view of simplicity of your stack, you know, like one less application to manage you in your stack and everyone actually using you have a radius nowadays. So like sounds reasonable. I also, at some point for time, I was, wanted to write a.

A message queue in pure PHP.

Eric Hileman: [00:10:25] Yeah. I suppose you don’t have one in the stash.

Ivan Chepurnyi: [00:10:27] actually how you want to repository might get hub, but, it was created only just as a proof of concept with pure in memory. One, when you have a message queue just in your last insurance BHB process where you, you just do, it was in one single event loop.

I didn’t do it multi-process communication yet, but. And is possible. And if you are using the writers for it, it’s good because. like radius makes sense to use when you write in a synchronous application. If you’ll write a simple applications that that’s like regular PHB, with FBM time to open the radius connection, you know, like send all the data to it closes connection and so on.

Like you said, to rate, is it traffic to radius? Right? So it’s gonna. Like you have to run really a huge network. Coupling it to your radius instance in order to be able to. Protests, a lot of them. And also red is a single threaded, so it makes sense to use radius as, as, as a very lightweight message queue.

It doesn’t make sense to spread it out as a scalable solution because only if you are using, let’s say, a multi threaded version of radius, it might work, but still it’s a little bit preferred risk. so. I would bet there use something like a rabbit MQ or you know, something that is built specifically for scalability and for clustering of the message queue itself.

Eric Hileman: [00:12:00] That’s what my thought as well, like right us, Loretta’s persistence is a polling, so like rabbit, that’s, you send it a message. It’s atomic. Like it guarantees you have the message like Reddis it’s in memory and it’s polling and writing the desk. So you, you don’t have the level of persistence. Right?

Ivan Chepurnyi: [00:12:18] Ah. I think it depends, you know, if you take a look at, so synchronize programming itself, is it a hyper would be looked at when you write to an eventually consistent system?

Is that, is the data’s that you’ve sent might not arrive at the destination and you need to resend it through nozzle destinations and. So

Eric Hileman: [00:12:38] once it’s in there, like you could, you could lose it in theory and Retis, cause it’s a memory of something went wrong versus if something goes wrong and rabbit, you’re not going to lose a couple seconds or what, however often you’re flushing to, to disk.

Ivan Chepurnyi: [00:12:53] So. So, so that’s why you need to have some kind of a feedback operation. So when you were at, let’s say a stock deduction system, and let’s say you have an actual web service, as it receives requests, Hey, I want to reserve this item, or I want to adjust as a quantity in a warehouse for this item. And is this a web service?

A will communicate with the actual worker and said, it’s gonna run an actual deduction, in a data store, and this worker needs to return back a message acknowledging that it’s already protest. Is there a request to deduct it? If you don’t have this feedback, you cannot rely even on a rabbit MQ because you can just send a message to rabbit MQ.

Yes, it’s saved in a rabbit MQ, but if your worker or your workers are there to, you are not processing those messages. So just you just spend your red button message queue. So, so it, it always should be f-ed back operation. You should never think like, Hey, I’ve, I’ve send my message into message queue and it’s done.

No, it’s not a good to, to, to be done

Eric Hileman: [00:14:08] as thinking like, so Shopify and I haven’t watched these videos that I’ve got links to, but Shopify engineering team was talking about, I guess apparently they’re using Radis. As a message broker and their order transaction process, which kind of seems, I don’t know, like if you’re an ESP and you need a message broker and your startup, I don’t know.

It kind of seems okay, I use rattus but if you’re going above that and you need consistency and you’re like, I don’t know, that seems to get a little bit crazy to me. Like you would use Retis for that. Just the level of persistence.

Ivan Chepurnyi: [00:14:41] I think here’s where a good signs that Shopify is not using any as in Hunter’s micro services.

So all of their code is in Caroneous. Definitely because otherwise they would not use a radius for my Cisco. because radius for message queue is kind of a simple solution, but it’s simple solution. If, let’s say you have a. Ruby on rails application. All right.

Eric Hileman: [00:15:11] Yeah, that was the rails thing, and apparently it’s really big and rails community and I stepped into a whole battle zone between PHP and rails or whatever Magento versa.


Ivan Chepurnyi: [00:15:22] really like Ruby language. I wrote fan by show for

Eric Hileman: [00:15:26] orchestration. I think about it. I don’t have an opinion one way or the other,

Ivan Chepurnyi: [00:15:29] like

Eric Hileman: [00:15:30] I could be cool. I don’t know.

Ivan Chepurnyi: [00:15:33] Ruby language is very cool. I really like it. Like it’s, you can change a Zillow language on the fly. Like you can change properties of numbers or strings as a class of objects.

So you can create new methods, you can replace the default behavior of Zillow language, internal structure. Sounds cool. And it’s a lot of fun. It’s interested in language. I was writing a lot of orchestration for, and wild grant environments at some point in chef and chef is based on, a Ruby. But what they really don’t like.

I don’t like a Ruby on rails. So for me is this framework looks a little bit, too much, from the era of, MVC frameworks. And we see in bad connotation because, it doesn’t have strict separation of the domain. It has a separation by. Kind of type this is, that doesn’t bring any business value. So, but it’s just a, you know, my opinion is there’s probably a lot of people who will tell Ruby on rails is just Nicea

Eric Hileman: [00:16:44] we’re going to get all the rails fanboy hate now on Twitter.

And then, and then there’s the multithreading like, you know, something like rabbit or Kafka is going to be multi-threaded and support that versus a redness. That’s single threaded.

Ivan Chepurnyi: [00:17:00] Zero is a red is for Q knows that this move is right at about the Auri to, it’s not most straight out of the books, but if, let’s say you write really us assurance applications of this multi as well, like what you can do in the rust, you already, you don’t need to read this at all because in that case you can send message from one threat to another threat.

And you can tell, Hey, I have would say sorta two core CPU and  to do course are effectively 64, active threats to which you can send to staff and process, concurrently. And it basically means that I just can start processing something on one threatens in, just move through another as threat. And, It’s the same time I can handle, let’s say 200,000 connections. Some are dangerously was all at any, hiccups. So, and this is the only possible if you have completely different partying in application development when you really don’t block on IO, because as soon as you have a soundbar and a block that you will introduce some kind of a message,

Eric Hileman: [00:18:13] sending an email, like sending an email.

Ivan Chepurnyi: [00:18:16] Yeah. well I like sending an email. for instance, there is a good example, of, the place where I see her nurse, or y’all can really, simplify your stack because you can just connect to SMTP server as it for honestly, you don’t need to wait to do it. The response to you, you can just process other requests.

Is it that are happening? You, you just send some bites and you wait till those bites protest, but you don’t wait. You grant some code. Dime and majority of, unfortunately, a majority of PHP projects are Ruby projects. Even PI. Some projects in bison actually is there is some  community. I think there was some objection or was there, but majority of deal for these development communities are never working with system Kronos gold.

So a just wait. It’s tough, man.

Eric Hileman: [00:19:05] Yeah, it is. I, I did a lot of asynchronous with C sharp. years ago, writing an SMTP server and it, it, it’s mind boggling. Like when you get into the details, managing state and multithreading and juggling cores, it’s

Ivan Chepurnyi: [00:19:21] always challenge. Is this one I know for sure. Like, is this why I really like Ross?

Because Ross too. Made it compiled time issue. So you cannot really mess up with your, concurrent memory access. He just, he just spits out at you UK, you cannot, you have to explicitly specify a atomic, multithreaded reference diaper in order to be able to send this value to another threat. So you, and you don’t have a possibility like.

In majority of languages like C and M and a C plus Blas, Java and so on, like every single value, is it to you. A sand soundbar I usually use point or so references, right? And in the rust, it’s very common just to, handovers or ownership was a value. So as soon as you have, are able, you already an opener on it.

As soon as you finished your function, execution, is this value. Is not a corporate somewhere, is it? All their ship of this value is transferred through another piece of code.

Eric Hileman: [00:20:27] That’s very important.

Ivan Chepurnyi: [00:20:30] Yeah. And to in, in Ross, Digi, just on the language design level. So this is completely changes your attitude to development.

Like it’s much more harder to develop as a beginning, but this one is, you understand, logic and why it’s happening and the

Eric Hileman: [00:20:43] code runs, you know, what it, what it’s doing and why it’s doing it.

Ivan Chepurnyi: [00:20:47] Yeah. So it forces you to think about your application structure from point of view of, sending messages sexually from one piece of code to another one.

Because as the message is living in your piece of code, as soon as your code is running, so as soon as your function stopped running, you cannot, access this reference later. It was to, you know, some, special. Language constructs that allows you to do it, but it’s just

Eric Hileman: [00:21:17] shoot yourself in the foot.

Ivan Chepurnyi: [00:21:19] Yeah. So you have to go out of your way. You usually follow something. Is this goal to you? Do a mighty crust? Who are you? Right? Cell phones at this, was out, any shenanigans who is memory? All in memory management. So, so let, let’s start with our, Sound

Eric Hileman: [00:21:37] approach, sound approach right up our alley, man.

All kinds of audio file gears. So I was already on the site looking and shopping around, cause I’m always looking to upgrade stuff like you are, you just upgraded your

Ivan Chepurnyi: [00:21:49] stuff.

Eric Hileman: [00:21:49] Like we’re always, like every, everybody who starts a podcast is always continuously upgrading their stuff.

Ivan Chepurnyi: [00:21:54] And so I actually, it’s a funny story about this microphone because this microphone, I bought it back in the fall wasn’t 11.

Really? Yeah. This is, it’s just.

Eric Hileman: [00:22:08] Like in a drawer, like waiting to be used.

Ivan Chepurnyi: [00:22:11] I was pretty much, I was thinking about starting online training courses at some point, around Magento and I was doing also some video recordings. There is some recordings on, via email, from that era where I was doing online.

Screencasts and I was explaining about TDD and Magento and how to build stuff.

Eric Hileman: [00:22:32] it was such as a decade ago, almost 10 years.

Ivan Chepurnyi: [00:22:34] Yeah, it is. it’s 90 years ago. So it was a long time ago. Like it’s hard to believe. And this microphone actually is Southern. This is the first things that they bought, for specifically for, education.

So, and now he just works pretty well. So. I, I don’t,

Eric Hileman: [00:22:52] I didn’t get your money. And it sounds good. It still sounds good. That’s awesome.

Ivan Chepurnyi: [00:22:56] Yeah, it’s a, it’s a continence or based, so it’s USB contracts are based, I think it was one was the first one. Since that time we see a USB connection. So. And did it works in any operation system still, so I’m pretty hyper.

Was it

Eric Hileman: [00:23:14] eventually you’ll be recording from Linux almost. Almost there.

Ivan Chepurnyi: [00:23:18] Yeah. I still need to buy another camera. I think about buying like magic, four K pocket camera. That’s a nice

Eric Hileman: [00:23:26] one.

Ivan Chepurnyi: [00:23:27] Yeah. It’s specifically made for, we do. So, that sounds more fun because my GoPro unfortunately doesn’t work well under Linux ends.

It’s bidding and under windows as well, like you have to apply a lot of cool little gradings. It’s out of my legal expertise.

Eric Hileman: [00:23:48] Yeah.

Ivan Chepurnyi: [00:23:48] Yeah. Okay. So that’s a review. Our customer today. So, this is some Magento too, right?

Eric Hileman: [00:24:00] Yeah, I think so.

Ivan Chepurnyi: [00:24:01] Yeah. Okay. So let’s, as I’m disabled right away, aggregation and let’s profile a homepage.

Let’s see what’s happening here.

Eric Hileman: [00:24:08] Yeah. Let’s look for some cool stuff to buy. Let’s look for some upgrades.

Ivan Chepurnyi: [00:24:15] I recently just use some

Eric Hileman: [00:24:17] audio file

Ivan Chepurnyi: [00:24:17] headphones I recently. So, eh, do, which is is so Linus deaf tips, YouTube

Eric Hileman: [00:24:23] channel. Yeah. He’s been going for a long time.

Ivan Chepurnyi: [00:24:26] Yeah. It has some nice, so we do about a sorta K a headphone set.

Like, wow,

Eric Hileman: [00:24:36] I can’t imagine they

Ivan Chepurnyi: [00:24:38] like really as a headphones, as it calm was as a wall set to listen to them. Like you have amplifiers you have by bench or for other stuff like fielders and so on. So it just crazy stuff. You need towards this sweet is that we do it. It’s quite interesting.

Eric Hileman: [00:24:58] No, I don’t. My wallet says my wallet and my bank account says no, I doubt.

Ivan Chepurnyi: [00:25:05] Yeah, and okay, so this is five seconds on the homepage. So it was just a silly little bit of a problem, I think for this customer because as a home page shouldn’t take that much to load. Let’s take a look probably, with, as a family interview first. Let’s see. Ah, Oh, actually, let’s go to, that’ll be, it’s clear is because this one I think is the most important one.

So here we have, I loaded in a loop, definitely. So is there some piece of the codes that Lord’s categories, in a loop? Maybe somewhere in the menu? Oh, this is kind of common problem for most of Magento developers. Say, just, instead of adding, an attribute as a select into the menu, three-generation a is a decided, Hey, we, I, I can’t add any of my attributes.

What does it say? It’s already in flipped table, so you can order the axes them. You just need to observe one event in Magento resource model for a flat to know it’s going to ration and it will be available in the final data structure from which has a menu is built. But, Zaid Jesse . I’m just gonna. as soon as I render  we new item, I’m just gonna access this category and a lot of tolls of data related to it.

And Zen, you sees this kind of query like.

Eric Hileman: [00:26:38] So how did they fix that? And you think that’s coming out of an extension or that’s a custom theme

Ivan Chepurnyi: [00:26:43] of we will see probably some, we will see it in a timeline view. So let’s switch to it. I think it’s going to be as a manual rendering process. Let’s take a look. So we see a header, we see sections, container, and talk.

My new year, so it was a document you see here, it takes a one and half seconds. And this customer also using  as well. Right. So here’s also another problem with Magento is that even through on as a top menu is include this ESI include Magento’s tool renders it. However, later on it’s anyway going to render it only for ESI include.

So it will be the waste of time. But in general, I think adjust it. Custom template. I think.

Eric Hileman: [00:27:34] Mega menu. We just saw smart way. Mega menu.

Ivan Chepurnyi: [00:27:37] Yeah, just previous one. So it seems like a very popular extension. So I think this,

this mega menu is doing some calls. So let’s take a look at, There’s a model, Lord. Yeah. So you see here, we can see from the metrics, but as away, is there a reason, if, if you want to get, some right away information about how many models has been lowered within Magento, you can install Blackfire Magento to plugin and Zen and tools show you, is this metrics in a timeline view?

yeah. So in this case, for instance, it does mean, is that. Magento to old Mondo Lord. It tells me is that there is some models that has been loyal to 181 time. So this is quite too much.


Yeah. So here where we see Zillow squares.

So as a very quick fix, is actually replace, is a render notices blocking’s a default handle who is just hardcore that, ESI tag and Zen, create a separate, handle, in which you will render as a full blown menu. Zen, your menu will be cached with varnish. And it will be cashed on Luan since it have been cashed on every page.

Eric Hileman: [00:29:15] Talk a little bit more about that on the, the ESI. So that’s one there. Yeah, there’s some, so there’s something about that, who, who was it, Lightspeed as a drop in replacement for a patchy, and they have a Magento plugin and they say one of the big benefits is that they batch everything up. To their ESI calls.

So whereas Magento and it’s varnish extensions, making individual ESI calls for each include, they batch them up and as is that what’s happening here.

Ivan Chepurnyi: [00:29:51] I think Magento has only one. You said glutes out. Those are books, so I didn’t know why. Yeah, I don’t know why you’re doing it to bash it together. So

Eric Hileman: [00:30:01] I don’t know.

I was, they, they mentioned something about that. Of course, they’re trying to sell us on their, their light MEJ cash or. Whatever it is. And, that was one of the things they were explaining to me is that they get a big performance improvements by only going to varnish wants versus Magento going to varnish multiple times.

Perfect. I’m not sure how they did it. I’ll have to, I’ll have to him refresh my memory and ask him about that and we can talk about that probably in a future episode.

Ivan Chepurnyi: [00:30:29] Yeah, definitely. probably we can, discuss it more in details on some of the examples and maybe we should start a sub show when we talk in deep about technical details.

You know.

Eric Hileman: [00:30:42] Click here to watch our video on,

Ivan Chepurnyi: [00:30:45] on ESA includes right

Eric Hileman: [00:30:47] on ESI includes,

Ivan Chepurnyi: [00:30:50] yeah. It sounds like a good idea. I think a lot of people will find it valuable from developers.

Eric Hileman: [00:30:57] Yeah. I’d be interested in doing it.

Ivan Chepurnyi: [00:30:59] Yeah. So here I see a Nazar problem. if you take a look at the product collection, You can see that as they’re having a search engine running here, so it means that some stuff gets output from.

Mmm. From layered navigation, but I don’t see in all the red navigation on this page. So is this a sounds interesting? Maybe it’s this block is showing some issues, but I think it’s, it couldn’t be automatically generated. I think it’s manually enter it. So, is, is there a, some word navigation, Lord happens that shouldn’t happen on a homepage.

And this takes quite a lot of time, like 500 milliseconds. You see it here. So render the filters before and Zen, it tries to execute an empty search query. Oh, it’s a matter of sweet. Okay.

Eric Hileman: [00:32:08] Frequent flyer on the show.

Ivan Chepurnyi: [00:32:11] Yeah.

So this plugin has some kind of Auron search, a plugin, and inside of it, it actually builds to literally navigation. So I think Zimmer suite, search plugin is actually doesn’t take into account when product collection is loaded, was outside interrogation. And it tries to load letter navigation for the wool store.

So what we see here is actually building all of the facets that are available, plus the products in the store hands. This is pretty, pretty bad for, is this costumer, because, yeah. here you can see as well. I’m like. Does this is this thing, gold store API documents. It just tells us is there some kind of fried operation happening into my scale.

So it, this just right away big? No, no. And, your,

Eric Hileman: [00:33:20] that’s a new one.

Ivan Chepurnyi: [00:33:22] yeah. It’s, some things that happens in standard default, my scale implementation of, alert navigation. So Magento grabbed souls of product ideas and these sources into temperature table and Zen forms at temporary table. It’s going to show you the results on a page, a little bit of overkill in my opinion.

because. You could write much more by queries as in trying to use to improve the tables because it’s going to kill under,

Eric Hileman: [00:33:51] don’t get me started on temporary tables, man.

Ivan Chepurnyi: [00:33:55] Yeah. So you see, is there a scene sort of into search DMP.

And is this one like what I see here, I don’t think it’s used anywhere else afterwards because most probably I like.

let’s do a small thing. Well, that’s a run that we said debug mode enabled. I just want to see what kind of queer use executives are. So we enabled debugging mode. This is where the cool feature boys away as it, you can see every single parameter is a sentence that’ll be scary, like an in show profile.

Usually want to see an agreed to data. When you see just a queries group by tables. Was our tuning parameters, but now like who see every single query, sip but ugly and being executed. So I just interested in does this, that the waste query, is there any product ID as a parameter?

Yeah, it’s the same number of queries. Let’s see. It’s all squares.

Yeah, this is going to open notes a bit slower because he goes, there’s plenty of data. Yeah. And as I said, dental, we close it so well, that’s search for, you know, it’s just. Can I incorporate somehow select,

okay, let’s try was if, something does it, highlight it for me.

Eric Hileman: [00:36:01] You may need to highlight all.

Ivan Chepurnyi: [00:36:06] Search queries. Wow. That’s crazy. Yeah.

Oof. Single product load. You see

quite a lot of things happening here. Why?

Eric Hileman: [00:36:33] Why? Why are all these being loaded though? Like just that  that’s just the navigation dropdown that they’re showing. Where you shop. Categories are shot by brands.

Ivan Chepurnyi: [00:36:46] No, a is this one. What I’m looking right now at is this actually a lot of product?

Eric Hileman: [00:36:54] Yeah. Why is it loading a product.

Ivan Chepurnyi: [00:36:58] And because some attribute is missing and  I’m just going to load the product separately because I don’t want to add an attribute to regional collection select.

Eric Hileman: [00:37:09] Okay,

Ivan Chepurnyi: [00:37:09] so this is probably is a reason.

Eric Hileman: [00:37:13] Can you tell where that’s being. And who the culprit is that’s calling that

Ivan Chepurnyi: [00:37:18] a. So is this a problem with, you know,  was out having an access to zip code because, usually when you have an access to the code, you can, add some, Blackfire profile or marks so you can go into the code piece.

because I, I could tell it if I would be able to see more details what actually gets called there. But. Oh, probably actually was debug mode. Maybe it’s going to be more open to what actually happened. Oh yeah.

well, Elise, I mean, we

Eric Hileman: [00:37:56] could, we could get,

Ivan Chepurnyi: [00:37:57] we have interfered,

Eric Hileman: [00:37:59] but we can’t really change it. We wouldn’t want to show it on the show either. Probably.

Ivan Chepurnyi: [00:38:04] Here it is.

Eric Hileman: [00:38:07] Okay. So who, Oh, daily deals

Ivan Chepurnyi: [00:38:11] marked away. Gotcha. And this one is actually a term, very terrible extension because they can look at, it’s just mess up the cold get first item.

Aziz get first item. So first of all, is they’re using collection in order to access the one product. And Zinzi loads is product again. So like. For every single check for easy to do product a, is there a gun in the Lord collection or products and Zen, is it going to take the first item? Autos is collection of products.

So this is kind of terrible, and this is happening for every single product as AC on this list, and let’s see, are any of those products or here. There was a campfire, any kind of deal label, I don’t see an indie label, so I think could just like probably is this fun, like this product? Probably it’s a dual product, but a

Eric Hileman: [00:39:19] retail, you say real Tam

Ivan Chepurnyi: [00:39:21] price, but does this extension, like you can just compare as a final price and a.

I’m ready to go. Our price was a product and it was, the price is different to just shows a discount like you don’t need an extension for is that you can do it directly in a template or just creating a simple a block cause it will compare it as to where your scent will tell you, Hey, is it a dual product or not?

You don’t need towards the full product model again from a collection in order to achieve it. So was, this is definitely a poor implementation here. so how

Eric Hileman: [00:40:02] much time is that? The

Ivan Chepurnyi: [00:40:04] daily deals, like it’s 20 milliseconds for each product. And you know, like we have pretty much a lot of them, so

Eric Hileman: [00:40:18] yeah. How many are they actually Cheyenne?

They saw. Well, I guess you can. You can. It’s a slider. So

Ivan Chepurnyi: [00:40:24] yeah, so it’s a slider. So is this all leased? So it takes one second in total to render. And I think this one second is mostly contributed by this dual product and apart from is also some standard Magento pricing. Yeah. Final price as well and gets involved, or was there, but not as much as a deal product.

Eric Hileman: [00:40:49] Yeah, there’s only, I think there’s only like 10 or 12 products in that slider

Ivan Chepurnyi: [00:40:54] and it’s too much time spent. So let’s finalize our homepage.  so we can see in the homepage, is it menu is a problem. So, so something is loading a category. Yeah. So our smart way mega menu is loaded in category. For every single categories.

There is a separate database operation to law, to additional data from it happening. and other things that we see here is that, our mirror suite, a search extension, triggers a loading or falls, a lot of navigation aggregations on a home page. Whereas this should not happen at all because, this one is rendering of the block, related to, products like  products that are shown on the homepage.

So like, you’d already knows which products . And here, immuno speed extension doesn’t add any guards to make sure that, Hey, are we actually on alert navigation page? The tweens, those filters, like this is the wrong place to preload the filter values.

and here’s a, just decided, Hey, we just gonna add some fuel throughout your soul’s thesis right away. Just completely redundant piece of the code, because it can save one second. So just,

Eric Hileman: [00:42:19] and a lot IO.

Ivan Chepurnyi: [00:42:21] Yeah. And a lot of PIO, like, because it also prizes to database, not only reads from it. And Zen and Azzurra were on the, is our deal products.

So basically like fixing saucy shoes. it all starts with, is this, yeah. So here we are. basically this customer can have our in one second, render time on the home page. Like Xirrus probably Azara thinks as well, like, Oh yeah. Also not to optimize composer of the Lord as well. So it was a common thing as well.

So optimize composer off to deal with those issues, like, and Zay will have definitely set them to first bite below probably 800 milliseconds. This is a love that. Yeah. And this is very simple fixes. So it was this. Like, just add an attribute into collection instead of floorings and models separately. Zan, North calling a search, a plugin on homepage.

This is simple fixed rate and an Azara simple fix is just, instead of zoos deal, product extension, just grade the check. You’ve final price is different than the regular price. And then you just show, is this blocking, is this be Sophie Shimano? Like in total, I think like it’s a couple of days of developer work.

In order to fix those. However, from point of view of, the actual return on investment, it’s much more, gains for the end merchant to achieve. So let’s go into, Oh, okay. So is there a building store like Amazon style? They have very similar styles. The menu, this one reminds me, Amazon. Like Mica. Tega is by brands.

Let’s see. So let’s go into this one and let’s profile the, they go to punch. How do we think, what’s kind of result we’re going to see here?

Eric Hileman: [00:44:40] This one’s been pretty heavy in our experience, but I don’t know if it’s, if they’re all heavy or if it’s just specific ones that are having. Is that I’ve hit some that are definitely took longer than others.

anything interesting we could upgrade to on this page?

Ivan Chepurnyi: [00:45:07] Yeah. I think you, you know what’s going to be our next suggestion.

Smile last excuse. Right. And definitely a lost six year old is going to shave off a lot of milliseconds from this page because it’s not gonna work your way to temporary table storage. And Zen can just remove mirror Switzer shoe extension, and they just can installs this absolutely free. no need to pay, you know, Forza license or whatever.

Just open source extension. Just go and install, try it out on the dev instance first, of course, to make sure that your design is not completely broken and just adjust your design and it’s going to be a very nice, looking  page was littered navigation. So let’s see what we here, how here. So there is almost 500 database queries.

We have

200 millisecond spent on IO and harmful for your day is actually happening when the database, Zaza health probably happens on Reddis. Or file system. So let’s take a look at the timeline right away. I think we all right away. See here in menu of course. So we just right away skipping it because we already know what there’s a problems there.

okay, so it was a product list. I think it’s the same problem as on a homepage because there was just more products, but I think there is the same deal product.

Yeah. Is deal product happening pair as well. Also, is there this Magento bog as well? It’s kind of for buck, but not ABOG. Magento introduced it actually as a feature in 2.0, in Magento one as the tutor price was calculated as a part of the price index. In Magento too, unfortunately, is it tier price is not.

The calculated is a part as a price index, like you don’t have any more resistance to your price field on which you can rely and show as low as value on the front end. However. I don’t think that this customer is actually utilizing Teeter prices because I don’t see any as little as a labels on these products.

Probably customer needs. Just to check. Oh, wait a second. Xirrus sorry. Right away. As they said, they don’t use it. I right away see a product with a as low as a label, so it was just a, I’ll do a quest, and it has probably some price tiers if you buy it in bulk or no, it’s actually Jessica configurable product.

Eric Hileman: [00:48:09] I wonder why it says as low as

Ivan Chepurnyi: [00:48:11] it’s Magento way of, showing, the price. If we’ll configure a product on a category page because it can change per option. So as they decided to let you show with his Lois, you can change it in the configuration, so you can just show it as a price, but also like your right light showing as low as when you actually have a single price for any type of connector.

Eric Hileman: [00:48:43] Yeah. But as low as disappears, why don’t you change it? That’s interesting.

Ivan Chepurnyi: [00:48:47] Yeah. Because it’s gets updated by JavaScript and as soon as you choose an option, it’s not as well as anymore because you already selected something. So it was, this is, I think all

Eric Hileman: [00:48:58] would be here as low as it goes,

Ivan Chepurnyi: [00:49:02] as loud as it goes.

Oh boy. It’s so expensive. also those USB type C active cables, this, this crazy, my mag Buka USB type C cables got broken. Saw orphan in it. Just terrible quality and

Eric Hileman: [00:49:24] dongle life.

Ivan Chepurnyi: [00:49:25] Yeah. Hashtag

Eric Hileman: [00:49:26] dongle life

Ivan Chepurnyi: [00:49:27] as is. That’s why I really like my, BC build here.

Eric Hileman: [00:49:32] Me too, man. My laptop, I’ve got HTMI and an HTMI poor network.

Ivan Chepurnyi: [00:49:38] Sorry, I’m wrong to call it PC built in just at this store built because, because I, I run. Alenex for my work everyday. So I

Eric Hileman: [00:49:50] got full ports on my, on my laptop, and I love it. I plug into him. I tried to use a Thunderbolt three dock for it, and it’s just, it’s janky and sketchy, so I just go straight to the ports.

Both sides. Ports are nice.

Ivan Chepurnyi: [00:50:07] Yeah. So it looks like. As this page. was there a, some, some improvements can be done on, product surrendering, right? So

Eric Hileman: [00:50:19] yeah, so tier pricing, anything they can do with that or that’s just the way it is.

Ivan Chepurnyi: [00:50:23] One hour Mark.

Eric Hileman: [00:50:25] It’s our one hour reminder audience can’t see it. But Ivan needs to reboot to finish installing windows updates cause he dual boots into windows only when we recall.

Yeah. And I know then I don’t think you ever actually do the updates before you boot back into Lennox. Do you? You just keep closing the box out and then just read.

Ivan Chepurnyi: [00:50:45] So like I was doing some of the dates. before previous recording. So we didn’t see it last time because the ones that they, I looked at  into windows and decided TL, let’s run an update.

And they take and took four, four, two minutes of my time to wait for it to complete. So, so I’m definitely just gonna reboot into Boone two and forget about it. Just forget about it. Next recording. Yeah. So, so definitely the product list is, there is some time can be saved here. And a Zillow or I’d navigation as well.

So here we see is that as a filter, values are a little bit slow. And here we have, I must’ve shot by, so, there was misfit sewers, there were some math to show by. And here’s a seller performance. He sure, right here, like 700 something millisecond to render filters. And there’s also this mirror suite extension is again, alters, is it query from Magento?

And there was some clarity happening and it’d be able to send an alert navigation. So yeah, just switched to elastic suit. And it’s the same thing have been shown. Surgery by Brandt surgery books. This is main store. So, so when you show by extensions here, so may stores show by brand also introduces some performance issues.

So basically like all of this piece of code can be reduced to very simple. That’s the way it’s created. Recent operations, like again below one second is possible. So zero is nothing. Special loans is, could take every page in order to, keep it. It’s surrender time. you know, slow.

Eric Hileman: [00:52:50] So would you replace all those with a  elastic suite or just the shop?

Ivan Chepurnyi: [00:52:56] So, in general, like, so there’s navigation. Does this search and, any kind of additional extensions that are used to generate those filters? Like. I would just invest some time with a developer to build the customization. So, top prefer elastics to use. you see a for installing, an extensions that will, grabs his data outside of philosophy search.

Because I’m more than sure is that she’ll buy brand from main store is not using the same message as, let’s say, shop buys or trillions of data and every, every is this extension probably executes it through a search API of Magento catalog search modules that, you know, works a lot of redundant operations.

So that’s, Oh, it will be a tough problem, I think. Yeah. All right. Sarah. So in general, yeah. Or we can saves is five seconds easily.

Eric Hileman: [00:53:53] That’s not just five seconds. That is just so much load on the server.

Ivan Chepurnyi: [00:53:58] Like. Yeah, it is definitely. So I’m just talking from point of view for the end user who is with the new website, right.

If you could filter, you want to have a stop your website. You don’t want to wait for five seconds to get the response back on your selections

Eric Hileman: [00:54:15] once any traffic comes in, like there. Everything slows down. I mean, Stratos will scale. Yeah. But you know, when you’re hitting the database for all those requests, like

Ivan Chepurnyi: [00:54:25] now my customers, this is the biggest problem here is definitely, the standard Magento search and giant.

so it’s using my skill and to start a search engine is localizing a lot of temporary tables.

Eric Hileman: [00:54:40] Yeah. We’ve been increasing the, the memory for temp tables in my SQL for customers a lot, and across the board even. But still, once you get so many concurrent users, you’re going to disk for it. You can’t allocate that much.

Ivan Chepurnyi: [00:54:54] Okay, so let’s go into broke, right?

Eric Hileman: [00:54:59] Yep.

Ivan Chepurnyi: [00:55:04] So this is some kind of simple product, worse, or it’s a configurable product as well? Probably the configurable, let’s see. Let’s disable aggregation profiles. Is Prague beach

it’s fast. So probably it’s not a configurable product.

Oh, that’s, that’s a lot for a power cable.

Eric Hileman: [00:55:47] that’s, yeah. That’s, proprietary for sure. $1,000.

Ivan Chepurnyi: [00:55:54] And sell on tokens at Apple charges too much for a monitor stent.

Okay, let’s take a look at the timeline. So six seconds. This is not too fast. So definitely menu. So we already discussed it and Zen, related products. So is, there’s probably is the same thing was, it was, deal product and product price itself. Soap Rowley. Is this fun take sometime? Because was that also, what did we have here as well?

You have a lot of small elements.

Oh, it’s getting a little bit late here. I’m just a little bit,

Eric Hileman: [00:56:44] it is later than we normally record.

Ivan Chepurnyi: [00:56:46] Yeah. So here we have, some rendering, but I think here, and we see some, slow rendered times because first of all, who’s a composer is not optimized. And maybe a setup called cash as well as not turned on.

So. It could be a problem as well. and considering at a composer of toilets are not optimized. I highly doubt, I would say disabled. file a validation in, open code cache. . So again,

Eric Hileman: [00:57:23] yeah, FA file validation. If auto scalings on, we usually disable that, but if auto scalings off, we usually leave it enabled so that when customers FTP changes up. there’s a lot of customers who will do that. They’ll just, put code up on the instance and expect to see their changes right away.

Ivan Chepurnyi: [00:57:43] So I’m about to do, do it right now, test enough to scaled an instance or single scale because we are using your Blackfire profile, right?

Eric Hileman: [00:57:55] Yeah. I think auto scalings off for this customer, I think.

Ivan Chepurnyi: [00:58:02] Okay. Well, so here we have, Our, I must’ve shocked by, and here we have composer of the Lord. You see composer a version loading. So is this fun? Is a problem and a gang like gesture move almost to show by. I don’t see why a monster show by is used on a product page.

Like it doesn’t make sense. Do you see anything really that the show by here

Eric Hileman: [00:58:29] go up? No,

Ivan Chepurnyi: [00:58:30] I don’t see anything as well. Like. I’m just following this just related products hands. This one is just, just a slide. General products. You might be interested in. Silver, always a cross sale and bought in general. Yeah.

I don’t think it’s, something valuable. And here again, we see  navigation. Is built here as well. On the product page is it makes it completely redundant, right? You don’t want to have an alert navigation on the product page. So, and sexually counts from this. Oh, so I see this one as this small block here.

You might be interested in this products. Is this one, is a block. is this a SCM as work. And he’s gets probably rendered by, his system. So they have some kind of CMS blow cause that contains, is this, special, template that they use. And inside of this template, all those extensions like immuno suite search and the Mosty show buys a, just add a lot of hooks.

Is that the executed completely right in and stuff. Sorry, I’m going to sneeze.

Okay. so here we have it. there’s a lot of times that can be saved, definitely. So basically, yeah. Again menu and the gain shall by immuno suit search like very simple solution is it can save a lot of cost and resources. And Zen, I would advise this customer to investigate more in detail when Zay removes his big bottlenecks, like talk menu and let it navigation use H on a product page on homepage, on Reddit and places.

Zen, tried to go and do bogs themselves and see what’s else is taken time because then they will see more information. Because right now, like. Because these pieces are not magical contributing. It doesn’t make sense. You know, just taking liquids on right now and work on it because you have bigger problems is it can give much more, return right away.

Eric Hileman: [01:00:58] Yeah. Our team says they disabled auto scaling when they were working on search. I don’t know exactly why they would do that, but probably making changes often or something and they wanted to see them right away. That’s usually what happens and then people forget to turn it back on. Do you remember when PCs, this old PCs used to have that turbo button on the front?

It would actually say turbo and it’d be a separate button .

Ivan Chepurnyi: [01:01:23] And, you know, like it’s completely different. as far as far as their noses, tubal buttons, his actual was doing completely different. Opposite. Yeah. Good.  well, sexually reducing the clock speed of the processor because. Oh is it old? The game scores not dropped backwards compatibility.

Yeah. But

Eric Hileman: [01:01:45] it makes me think though, I kinda, I kinda want to put a turbo button like that in the UI. And when you click it, it does composer auto load optimizations. Cause cause I think every store we looked at hasn’t done that.

Ivan Chepurnyi: [01:01:58] Just create a turbo button and it does a composer autoload optimize. Yeah. But a  set before 2.2.

A Magento composer that Jason was also not containing the write a file pass for, generated the class maps. So after I was doing some performance testing back in 2016, I was doing some benchmarks of Magento one versus Magento two, and I’ve noticed that a lot of, yeah, so there’s, there was this famous benchmark from other company, I won’t name it.

Is that say Wharton, like when Magento two is like four times slower and Magento one and so on. I just found out like cause they completely misconfigured Magentol legs. They didn’t enable composite of TULO. Also, I found out couple of bugs in Magento itself. Is that also a very simple to fix on deployment phase.

She has buy, modify and composer Jason and including all of the files that you want to have been included into afterload class map. Of composer. So if you have a store is that migrated from one version of Magento to another one, you are responsible for changing your composer, DJs, and filing your project.

So if you are running a store from 2.0 era. and you are, we’ll say right now in 2.3, your composite raise from the filed might be out of date from the base package. So you need to adjust it, especially as a location senior component of Jason’s that are related to. Off the Lord sections because after the Lord has been punished, and that

Eric Hileman: [01:03:40] was, it wasn’t fixed until after 2.2

Ivan Chepurnyi: [01:03:46] yeah.

After 2.2 only. So. After my feedback from the Lord test, I’ve submitted some, patches, and they fix it in a GitHub repository and they fix it also in their deployment scripts when they deploy those Magento based packages. But, but when you upgrade, unfortunately, as a base root composer, Jason file is not debatable because it be, update those, this file.

Real AIDS. Yeah. Composer consistency. Because you don’t know what you can update in broad in project composite, recent files that might already have been more defined for this project.

Eric Hileman: [01:04:24] Well, that’s interesting. Okay.

Ivan Chepurnyi: [01:04:27] And is there is already some script that actually can modify existing one? So, definitely, it’s nice when I can just point to get hub repository where it’s already existing.

I think it’s in Y, repository. Yes. Magento load test, Magento to bootstrap. So here’s this, funny sh file you see here. Yeah. So it was, this is from Zim, Magento 2.0 era. Is there a bunch of optimizations that they apply to attends? There is a simple shell script that will include a, is it require it, pass us into after Lord.

But this is for 2.04, 2.2 and 2.3 a. Is this passage different? It’s not to our generation. It’s now generation. Slash code. So it changed a little bit between the releases, but you can adjust this code and just modify as this fun, into the right pass and it will, alphabetical patch. Your composer that Jason file to includes a proper, off to load configuration for your project.

Okay, so I w I will put it into our secret major mojo, a Lord and a loop. Select Chet.

Eric Hileman: [01:05:54] Yeah, we’ll link that up in the show notes

Ivan Chepurnyi: [01:05:56] and I’m definitely going to create a some branch. Actually, let’s do, it’s right away. Can I edit file and place here? Yes, I can do it. So those jets should be generation or generated. Well, let’s, let’s take a look at, Magento. Good. Have Magento, Magento too. And let’s take a quick look at the code and let’s see.

Add the composer that Jason.

Generate goat. So it was, this is how it should look like. And in 2.0 there was snow past slaves that

So here we are.

Eric Hileman: [01:06:57] Awesome. So then you can use that if you were on Magento pre 2.2 and you upgraded to 2.2 your composer Jason, needs this in order for the auto load optimizations to work.

Ivan Chepurnyi: [01:07:10] Yeah,

I’m flipping to save it right away.

And they will call it after 2.2 upgrade, whether it’s only if you have a PREA, release of, yeah. If you were using 2.1 and before, 2.2 was it released. So it’s only applicable in this case. If you started from 2.2, you don’t need this one. And yeah, I’m just going to leave it as a branch for now, and I’m gonna put it into our show notes.

Yeah. Cool. Very fast. Very fast. Online editing was old to checkings as symptoms.

Nothing can go wrong, right?

Eric Hileman: [01:08:14] Never

Ivan Chepurnyi: [01:08:14] met Maria. No, never. Never, ever. Okay, so I think we can erupt. Robot ball were showed today. Right.

Eric Hileman: [01:08:23] Well, thank you to Blackfire Datto for sponsoring us, and providing this great tool that we use on the show. We do have support for that and mojo Stratus. Now in the UI, and our PHP settings, you can enable that yourself and you can find us at, load in the loop on Twitter, L ITL at major mojo.

Dot com and avant. We’ll link up our own Twitters and the a in the show notes. Anything else you want to say before we head out?

Ivan Chepurnyi: [01:08:48] yeah, so just a profile. Your code was black fire, like fire is a very nice tool and. Keep track of your other based queries. keep track of your concurrency and low contention on your production servers.

Pay attention to the way how we use that base. So thank you everyone who. Join our show today and deals. The next recording I think is going to be in two weeks.

Eric Hileman: [01:09:18] Yes. And use you smile. The last disc. Sweet. Yeah.

Ivan Chepurnyi: [01:09:23] Smile. Last succeeded. Big shout out to guys for developing it to make an a tree for everyone. It was great.


Eric Hileman: [01:09:31] right. Thanks everyone.

Ivan Chepurnyi: [01:09:32] Thank you. Bye.


Notify of
Inline Feedbacks
View all comments

Ready to get Started?

Sign up now
Would love your thoughts, please comment.x