Load in the Loop: Episode 3 – Scrip,Inc.

Jordan Griffey

November 18, 2019
Load in the Loop: Episode 3 – Scrip,Inc.

Load In the Loop: Episode 3 is here! Join Eric Hileman and Ivan Chepurnyi as they identify and fix some performance issues for Scrip, Inc.

Follow us on Twitter! @loadintheloop

Sponsorship provided by blackfire.io

Contact info: 
Ivan: ivan.chepurnyi@ecomdev.org, @IvanChepurnyi
Eric: litl@magemojo.com, @ericvhileman


Ivan Chepurnyi: [00:00:00] So what kind of fun stuff are we going to do this time?

Eric Hileman: [00:00:07] Well, we’ve. We’ve had a lot of tickets open for this customer, and they’re, they’ve been struggling with some performance issues for awhile now, and they’re looking to increase their traffic by 3X. So…

Ivan Chepurnyi: [00:00:21] Yeah, that’s a lot. That’s a lot.

Eric Hileman: [00:00:23] This is Load in the Loop sponsored by blackfire.io. I’m Eric the CEO and co founder of MageMojo Magento hosting.

For the last 10 years, we’ve been hosting exclusively Magento and we provide a deep level of support for Magento. On this podcast we review some of the customers and we share what we find to help others. And I’m here with my cohost Ivan.

Ivan Chepurnyi: [00:00:42] Oh, hello everyone. Uh, this is Ivan. Uh, I’m running my own consultancy businesses in the Netherlands, I help merchants to bring Magento performance to better levels and to reduce overhead of application on CPU of hosting. So today, um, me and Eric, we are going to, again take a look at another, uh, MageMojo customer. And Eric tell us about the customer and what the customer does.

Eric Hileman: [00:01:13] Yeah. Customer is Scrip Co, um, they work in the medical industry. They’ve had a few tickets open for a while now with performance issues, and we’ve been helping them out to get to the bottom of it.

They’re multi-store. So they’re dealing with some of that and they’re looking to increase their traffic by like three X. So we’re here today to take a look and see what we can find. And. Reduce those bottlenecks so that they can grow their store and increase the traffic that they

Ivan Chepurnyi: [00:01:40] want. Yeah, it’s interesting.

So let’s see. We’ll discuss. Summer does, so, uh, let me, copy-paste the URL. Just a moment so. And they run… How many stores does they run?

Eric Hileman: [00:02:00] That’s a good question. I know of a handful, but I think they have more, I want to say there are 15 to 20. Let me check.

Ivan Chepurnyi: [00:02:07] 15 to 20 stories, right? Yeah.

And there are fewer Magento 2, um, as a, I remember as their B2B customer. Right?

Eric Hileman: [00:02:22] You know what, I’m not sure. That’s a good question. I think they might. Do both, but I think they’re primarily B2B. Let’s see.

Ivan Chepurnyi: [00:02:30] Yeah, because I see as they do medical supplies, so it looks like, uh, it’s B2B, so, okay, I’ll, let’s, uh,  just dive into it.

Let’s just start profiling, right? And let’s run our, uh, nice Blackfire profile.


So it’s like, yeah, it’s working quite good. I can say, well, you know, like it’s not like cache is disabled or something. It looks like everything is good here. So probably we’re gonna find some interesting issues, not, you know, simple ones.

And this is the homepage, right? So

Eric Hileman: [00:03:41] Yeah of one of the stores.

Ivan Chepurnyi: [00:03:48] Okay. We are right now hitting uncached the homepage and yeah. Seems like, cause…

Eric Hileman: [00:03:57] They use a lot more CPU than they should.

Ivan Chepurnyi: [00:04:01] Yeah. Let’s see. Let’s see what takes so much time. So Blackfire is always getting rates for us and a very nice report. And have you actually seen as a recent article from Blackfire about our show?

Eric Hileman: [00:04:19] Yes, I did. That’s pretty cool.

Ivan Chepurnyi: [00:04:23] Yeah, I really like how they explained how it was a couple of lines of changing of the code. We actually fixed a lot of, a huge actually performance issue for that customer. Yeah, by the way, is that customer already applied our fix or they still a working with developer and doing it?

Eric Hileman: [00:04:44] I think they did. Yeah. Their developer’s really responsive, so he was, he was extremely happy. The last two episodes we did, the developers were extremely happy, and probably reaching out to you to hire you actually.

Ivan Chepurnyi: [00:04:58] And unfortunately, I’m completely booked right now.

Eric Hileman: [00:05:01] Yeah.

Ivan Chepurnyi: [00:05:02] It’s, it’s a little bit of a bummer, but.

We will find, later on some time for them. So let’s take a look. So it was, this is a homepage, right? And the home page, we can see that, uh, yeah, it takes some time. Uh, already, uh, basically is this time I decided to keep original size of the screen. So we, uh, you know, can see right away the numbers, uh, without , you know, clicking through, um, a page.

So it’s a little bit more convenient, in my opinion.

Eric Hileman: [00:05:34] Should still be clear.

Ivan Chepurnyi: [00:05:35] Uh, so, so the homepage, a time to get a rate takes. Yeah, almost seven seconds. And from this seven seconds, most of the time is actually spent on, uh, input output. So most of his time was actually spent in database.

You see,

Eric Hileman: [00:05:51] it says 1.8, right. And then 5.7…

Ivan Chepurnyi: [00:05:55] But it’s, but it’s a lot comparing to other pages. So if you remember, Adam’s Horse Supply . Yeah. As they were having a less IO, uh, issues. And here, yeah, it’s seven seconds to that are spent in PHP as well. But, um, from my experience, majority of the times at the spent in BHB is related to the number of, database operation you do, because you have to process the data that you receive from the database.

Eric Hileman: [00:06:29] So Blackfire counts that waiting. Well, no, it doesn’t. It doesn’t look like it counts. The waiting time for the database.

Ivan Chepurnyi: [00:06:36] It’s the same CPU time. But, uh, we were discussing before, you know, like you Blackfire reports, quite accurately the time actually spent on IO on the database. Uh, so if we, uh, just consider is that amount of classes and methods is that Blackfire has to hook into, in order to measure, uh, zip performance of particular PHP code parts.

We can safely, you know, like reduce it by 40%, and then we have a reasonable CPU time. So if we would use. By 40%…

Eric Hileman: [00:07:12] That’s the one that worries me. Yeah, that 5.73 in the CPU. But if you add that to the IO, that’s the total 6.91 close…

Ivan Chepurnyi: [00:07:22] By 40% it will be a something that would be executed without hooks because you know, like you need, you have some overhead to measure this functionality, but.

If we had a run on profile page, I expect it would be something like four/three seconds, you know, um, being generated the server side. Um, but, uh, the thing here is that from that time, one point. 18 seconds on IO. It’s quite a lot because, um, I, I’ve seen much more, uh, better. Uh, yeah, 1000 database queries, is not good. But I’ve seen thousands of database queries being, uh, let’s say spending maximum 300 milliseconds on database.

So is this is an indication that there is also apart from a number of database queries also probably some heavy queries happening as well.

So let’s take a look right away into the timeline as the best view you can get from, uh, point of view , um, a representation. And let’s see. Let’s see what’s going on here. So here we have something, uh, custom grid.

And the biggest time here is spent I see on sale, get product label or something like that. Get sale product label. So…

Eric Hileman: [00:09:08] Product labels. We saw two product label extensions last week.

Ivan Chepurnyi: [00:09:14] Yeah, it’s, it’s quick, very common issue. But as this one, I think it’s, it’s their getsale product. The label.

I’m not sure. Maybe it’s even not the their problem, because I don’t know, um, what the code looks like.

Eric Hileman: [00:09:30] Yeah. They’re overriding it, but we don’t know what they changed.

Ivan Chepurnyi: [00:09:35] Yeah. So this is just kind of helper, but we don’t know what this helper does. You know, like maybe. Uh, calls, some core method that actually works through all of those products.

But still, you know, like there is this configurable product and probably is there only one configurable product, right? Yeah. So here it is. You see?

Eric Hileman: [00:09:56] Yeah. Yeah.

Ivan Chepurnyi: [00:09:58] Choose option. Oh, actually all…

Eric Hileman: [00:10:00] Yeah. I see some of them…

Ivan Chepurnyi: [00:10:02] Three of them are configurable.

Eric Hileman: [00:10:04] If you choose options, do you go to the product display page or does it overlay something to choose and add.

Ivan Chepurnyi: [00:10:10] I think you are going to the page.

Eric Hileman: [00:10:14] Yeah. So that doesn’t seem like they need to…

Ivan Chepurnyi: [00:10:18] Oh, but I don’t see here drop though.

Eric Hileman: [00:10:23] No…

Ivan Chepurnyi: [00:10:24] Oh, now I see it.

Eric Hileman: [00:10:26] Oh, was that an HX call that was delayed or..?

Ivan Chepurnyi: [00:10:31] Yeah, it looks like, so this is probably some kind of custom made, um, custom made functionality. So here, yeah, we see that on the home page.

Oh, did the block change?

Eric Hileman: [00:10:54] It switches to a different… The feature.

Ivan Chepurnyi: [00:11:03] It’s a cached page. Wait a second. Oh, it’s probably, it’s a cached page because I’m not profiling right now. So…

Or it just changes randomly, maybe.

Eric Hileman: [00:11:20] Yeah. Yeah. Varnish is there.

Ivan Chepurnyi: [00:11:25] Okay. So another thing is here. Let’s see.

Media gallery, get edit. Yeah. It’s almost nothing compared to this one. So the biggest problem here, what I see, is actually is this custom grid and inside of this custom grid is get sale product label gets called, a lot. And as well, we actually sees the standard Magento performance issue as well, uh, related to the price.

You see it here, you know, price like takes 600 millisecond per product. Uh, and this is because Magento walks through every single configurable product. Oh, actually it’s even funnier. Wait a second. Magento, catalogue model resource, model load product. Wait, wait, wait. Where is it called? It’s called on some interceptor, get product by ID.

Wait a second. So within the collection someone calls. Oh, MageDelight.

Okay, so yeah, here we are.

You see this one?

Eric Hileman: [00:13:04] Yeah.

Ivan Chepurnyi: [00:13:06] So MageDelight, it’s, it looks like an extension vendor, right?

Eric Hileman: [00:13:11] Subscribe now?


Ivan Chepurnyi: [00:13:16] On every single product uh, what it does, configures complete load of product itself. So let’s say. This is actually the most common load in the loop you can ever find, you know, like when someone actually loads a separate product by product id.

Eric Hileman: [00:13:39] Okay.

Ivan Chepurnyi: [00:13:40] So here it adds some kind of has… Hasis… Has special price check. Uh, it adds this adjustment to the product price itself. And. It gets some product by ID. So yeah, this is actually the problem. So basically it means that over here it just loads the whole product model instead of actually accessing the data it needs, you know, like if it needs some kind of adjustments, why don’t add to, you know, on collection after load.

Why don’t collect the product IDs in around the database queries specifically for those adjustment values, you know? And then you get to this adjustment values and then you operate with them, uh, by doing this simple database queries instead of, you know, like loading every single product separately.

So this extension definitely is, uh, giving troubles to this customer. And I can tell for sure, like if it’s on every single product for 600 milliseconds on homepage…

Eric Hileman: [00:14:54] Oh, wow.

Ivan Chepurnyi: [00:14:57] So I, I can imagine what can we see in the category page because I am more than sure that the same problem is going to happen category system as well.

Eric Hileman: [00:15:08] Yeah. Uh, so is that doing a lot of calculations in CPU or it’s just database. IO. It’s just…

Ivan Chepurnyi: [00:15:14] Uh, yeah. So, uh, from profile, what we see here is actually, is that loading the product itself takes 600 milliseconds. Because Magento has, um, everything, uh, is that assigned to a product load itself?

Everything gets, uh, uh, triggered. You know, like media gallery, uh, tier prices, uh, product details, information. Maybe MSI maybe, you know, like binds your files or extensions is that are not supposed to be used to for this more functionality because I’m more than sure that as soon as you load the product, uh, the next item. Um. You know, like it’s not going to use this product. This is probably just going to be discarded. So only just for this adjustment value or something. It does this full product load. So it’s completely, uh, you know, not high-quality implementation.

Eric Hileman: [00:16:20] Is there a way to tell Magento 2 when you load a product to limit the parts? No, just the whole full thing.

Ivan Chepurnyi: [00:16:27] Uh, you, you have a repository and does this repository, um, yeah, you just specify ID of the  product you want to load. And here you see it, product repository get by ID. So it’s only a single parameter and then you see the whole, uh, elephant, elephant, to access a coin, you know.

Yeah. It wouldn’t probably… Like you just wanted to order some pizza, but instead you get the all the uh, uh, pizzza delivery.

Eric Hileman: [00:17:11] You get pizza, bread sticks, and French fries and soda, and…

Ivan Chepurnyi: [00:17:15] It’s the same thing. Uh, as we were talking on the first podcast and the second as well, we were talking about developers thinking point of view of objects. Not the data they need to access. They think, hey, Magento already as a subject into which I already store the data, so probably if I want to use it on the front end, I also need to use the same object, but  then it’s wrong assumption about design because you would have to go through the chain of actions in order to access the same object you were using in admin panel and…

Eric Hileman: [00:17:54] It seems like, it seems like that should be parameterized with like a second parameter, which is an object, like a string, I dunno, a JSON object that tells the attributes or the sections or the parts that you want to load. If it’s not there, load the whole thing. But if it is there and then just load the ones that you asked for.

Ivan Chepurnyi: [00:18:15] It’s not about product repositories. So product repository, um, it assumes  you’ll work with API, right? So product repository was meant to be used in API based interactions. Let’s say you access a product by ID through the API, and it’s nice, you know, like it grants you all the data that you need.

If you want to access lists of products, then, you can specify list of attributes you want to receive, you know, from as this list of the product. And it says that’s why it’s, it’s different. And in this case, usually when you write some kind of business logic, it’s also a bad approach to use EAV data to store your, uh, specifics, uh, about the product.

Let’s say if we talk about this, a module probably stores data,in the EAV structure in order to retrieve it later, but usually you have to create your own table, uh, completely isolated from the catalog into which you can build efficient queries in order to access it on the front end, and then just have a service model into which you just feed a list of product IDs and you receive back the data you need instead of, you know…

Eric Hileman: [00:19:34] Yeah. Right.

The monstrous queries that come out from the EAV and using the query builder are just insane. I think we saw one yesterday. It was copied into a text file. It was like six megabytes. It was insane, and it returned one ID. The whole query returned one ID.

Ivan Chepurnyi: [00:19:56] Wow. I want to see this.

Maybe we, you can send that to me later.

Eric Hileman: [00:20:04] I will. We’ll do an episode dedicated to the worst SQL queries we’ve seen. We’ve got some good ones.

Ivan Chepurnyi: [00:20:11] Yeah, it definitely makes sense. So, okay. We have spent quite a lot of time on this simple problem, but I hope it should be clear what needs to be done.

Okay. Now another one is this brand slider. Amasty override. So probably there is some kind of Amasty module that they had to customize a little bit. Okay. And here. Oh, okay. This is interesting.

It looks kind of heavy. Database query 29 milliseconds.

Is it some kind of layered navigation getting built here?

Eric Hileman: [00:21:09] Question?

Ivan Chepurnyi: [00:21:12] On the homepage. That’s strange.

Brand slider get items data. So 600 milliseconds. Now let’s see. What kind of brands we have here. Anything resembling brand slides?

Eric Hileman: [00:21:34] There. Shot by brands. That’s the brand slider. Yeah.

Ivan Chepurnyi: [00:21:39] Okay, so just to show these logos.

Eric Hileman: [00:21:45] Yep.

Ivan Chepurnyi: [00:21:47] They run this database query? Actually not the one, but wait a second.

One, two, three, four, five, six, seven, eight, nine, and each of them is actually a quite heavy database query.

Just a moment.

And then it also uses temporary table to store this data.

Eric Hileman: [00:22:21] Insert in the search? What?

Ivan Chepurnyi: [00:22:25] Yeah. So this is where your, um, MySQL server is dying out? Yeah. Homepage.

Eric Hileman: [00:22:34] Yeah. That’s pretty much what we see on the server side.

Ivan Chepurnyi: [00:22:38] So is this again, uh, our favorite extension vendor Amasty.

Yeah. So basically from these six seconds, three seconds is spent on getting these brand logos.

Eric Hileman: [00:22:55] Wow.

Ivan Chepurnyi: [00:22:58] Just…

Eric Hileman: [00:23:00] Just to show some brands in a slider.

Ivan Chepurnyi: [00:23:04] Yeah. So, yep. And the rest of the things are very minuscule. So it was the biggest problem, our load in the loop in the product grid for adjustments, right? And another one is this, uh …product label?

I don’t know what the code is behind it, what it’s using, but I expect it’s probably triggering the same, the same functionality because you know, uh, iterating all over, all of the simple products of a configurable is always an expensive operation. So…

Eric Hileman: [00:23:42] What’s the insert in the search? I’m really curious what that is, because search is always just, it’s soul crushing for us.

Ivan Chepurnyi: [00:23:52] It’s just Magento. Uh, so Magento, uh, using as a default engine to build lighter navigation it’s using MySQL.

Eric Hileman: [00:24:02] Yeah. But it looked like an insert.

Ivan Chepurnyi: [00:24:05] Yeah. So is, is, is this is a thing, uh, they decided to abstract away. Uh, the part that returns you back the iDs of the products that match the selection from that actual, um, aggregation, uh, of the results.

So let’s say you have some search engine from outside, let’s say elastic search, and you want to build your layered navigation. Right? Uh, or let’s say you have a search engine that doesn’t provide you a lot of navigation out of the book. So for instance, if you use, uh, uh, elastic suite by smile, uh, layered navigation is already provided from elastic search.

But if you would say, use as a default elastic search, Magento would get all the IDs from Elasticsearch stored into the Magento temporary table, and then it will join those IDs in order to generate your layered navigation.

Eric Hileman: [00:25:05] Ohhhh.

Ivan Chepurnyi: [00:25:07] So that’s why Magento inserts, uh, those, uh, documents into database

Eric Hileman: [00:25:15] Into the search table. It’s not still… But is it still full text?

Ivan Chepurnyi: [00:25:21] It’s not full text, but…

Eric Hileman: [00:25:23] The database table type.

Ivan Chepurnyi: [00:25:25] It’s temporary tables it’s created. It means it didn’t work. If couldn’t walk IO a lot, especially when you have a lot of concurrent connections so database server, it gets hit a lot. So usually I just disable this part and replace, you know, it was very lightweight and simple queries specific before this group was, because as they see Amasty, what they decided to do, Hey, we have, um, to retrieve brands that are going to be shown on the category. We want to probably calculate how many total products we have. So we’re going to show, you know, the most sold products or something like that. Or the most products we have on stock, so they just use standard Magento search API in order to find those brands. By most soldquantities instead of actually just writing a very simple database query to obtain all of this data they need.

Eric Hileman: [00:26:24] It seems like you could go straight to the database for that without a problem.

Ivan Chepurnyi: [00:26:30] Yeah. And you can also build a, you know, an index. You can build an index server that would generate for uses data in a very simple format.

Um. And then you also will be able, let’s say, to go into admin panel and build a UI that would allow a merchant actually to prioritize brands as they want to show, you know, on the homepage, if they don’t like the automated functionality. And here it looks like, yeah, just they just take the data right away from, uh, terrible queries, uh, on a database level.

And apart from that, they do a lot of write operations. So just removing this slider…

Eric Hileman: [00:27:13] Half the load time right there, the brand slider, three seconds…

Ivan Chepurnyi: [00:27:18] Just this part of the timeline, like 60% of the page load time.

Yeah, it’s crazy.

Eric Hileman: [00:27:32] That’s a good find.

Ivan Chepurnyi: [00:27:33] And another forty, is customer grid, but I think all of those issues are fixable, quite easy. So let’s see. Uh, we’re still on homepage, right?

Eric Hileman: [00:27:45] Yeah.

Ivan Chepurnyi: [00:27:49] Let’s go, uh, into category page. So here we don’t see any products, right? This one has been loaded in the separate right?

Eric Hileman: [00:28:02] Yeah.

Ivan Chepurnyi: [00:28:03] Actually, let’s take a look how much time it took because it looks like it’s a little bit slow.

Okay. Now we see this product. Okay. Why? I don’t see anything.



Strange. Let’s refresh. Maybe I had some… Oh yeah. Now it shows me the request. Because I was a little bit terrified. Okay, so it’s three seconds. Uncached page.

So it’s even not the cached in Varnish as you see.

Eric Hileman: [00:29:05] Yup.

Ivan Chepurnyi: [00:29:07] Yeah, looks like Oh, a little bit interesting. Like it already knows the product is gonna have, you know, it already tells, Hey, show me, this product.

But why it takes so much time? I can imagine if I just copy over this link.  Copy as… Yeah, just copy link address.

Eric Hileman: [00:29:40] That’s a lot of copies…

Ivan Chepurnyi: [00:29:43] And let’s just run our favorite thing. Blackfire profile.


Eric Hileman: [00:29:56] I mean, that seems to return click, but it looked like there was a lot of other Ajax calls.

Ivan Chepurnyi: [00:30:04] Yeah. But this ajax call is quite heavy as well, I think. yeah. It’s four seconds to render for products.

So yeah, it’s probably as well a problem too. And this is, it’s all the time uncached, so this is even not using, um, varnish in order to cache these ajax calls? Because those probably should be completely cacheable.

Eric Hileman: [00:30:51] Yeah, I wouldn’t imagine best-sellers changing just that often, really.

Ivan Chepurnyi: [00:30:58] Yeah. So this is three seconds and 400 milliseconds on iOS spent.

So definitely, definitely a problem.

And this could have been done. You know, like even within the page, it could be rendered under 20 milliseconds. You know, because it’s only four products, you can build a very simple database query. To get these four products IDs and then just use Magento collection in order to show these top sellers.

Because, as I understand, it just shows top products.

Eric Hileman: [00:31:47] Was it excluded from varnish because of the get parameters on the end of the URL?

Ivan Chepurnyi: [00:31:54] Um, I, I think the just completely not cacheable itself because you have to specify in Magento, you have to specify if your controller is cacheable explicitly. If you don’t specify a cacheable, uh. if your controller is cahceable you know, it’s just gonna bypass Varnish.

Eric Hileman: [00:32:15] We’ve, cause we’ve seen people send out emails and then they have varnish, but the server resources just go wild.

And then when we look…

Ivan Chepurnyi: [00:32:24] Oh, it’s, it’s, it’s, it’s a, it’s about Varnish configuration.

Eric Hileman: [00:32:27] Yeah. They had parameters on the URL that weren’t excluded. And so they didn’t cache it.

Ivan Chepurnyi: [00:32:34] Every single marketing automation software. Uh, adds a lot of unique parameters. So when you set up Varnish you, you don’t need to blindly use default VCL you have been, you have provided from your, uh, vendor like Magento. Uh, so Varnish VCL is not one size fits all. It always needs to be modified. You always need to exclude parameters that are specific to your marketing software.

As well. A lot of people don’t have in their VCLs they don’t have, uh… In Google. When you click on search result, you have to the URL, a parameter being appended, a G click ID. And basically if someone visits you from Google and you don’t have this parameter filtered out in Varnish URL, every single new visitor to your website from Google will receive uncached page result and it will be a much worse experience for them and they will just bounce back and yeah…

Eric Hileman: [00:33:43] We hear that.

We have Varnish. What do you mean? You know, our, our database is, is,… the load is very high. Well, you know, it’s not working.

Ivan Chepurnyi: [00:33:53] It always needs to be tested as well. So you need to test different scenarios. You need to make sure, like if you have new subscriber campaigns, you need to make sure that you try yourself to open the newsletter you send out. See what URLs are there. If is there are new parameters being added or something because your, you know, like software changes all the time and also the time you need to be ready to adjust.

So it’s, it’s, it’s not like you just build something and you forget about it. You always need to maintain, uh, your Varnish configuration. So let, let’s go, um. And profile the category page itself. Right.

So it takes some time.

So I don’t see any, Oh, I went blurry. Blurry a little bit. So, uh, I’m, uh, seeing that ther is not match products actually. There are no products here? But I expect that this, um, thing here on the left side is gonna take some time to generate because this is a layered navigation as far as I understand.

Eric Hileman: [00:35:37] Yeah.

Ivan Chepurnyi: [00:35:37] You know, like products and so on. And probably if I click on it, I’m going to see some products. But let’s see how much time it takes to actually render it. Uh, okay.

Uh, 600, database queries. Why?

Interceptors and the receptors, but what do those interceptors do? I render some container, templates purchase. What is this? Uh, purchase. Templates. Impression. Okay. It looks like some kind of Google tag manager thing. Oh my… One of my favorite extension vendors. Anowave. Hi guys.

Eric Hileman: [00:37:07] What’s this Impression push for?

Ivan Chepurnyi: [00:37:09] I really like your, um, Google tag manager extensions. They’re super slow. So here it is. Here we have, um. A thing like those Google tag manager impressions. We didn’t see any product at all. Right. But this extension actually, yeah, just takes one second of time to render those impressions because it loads some kind of collection.

Okay. Catalog product collection before to HTML observer. Execute. And here’s also our Amasty as well. We’ve shopped by, and this is the guest. It’s, it’s again gonna…

Eric Hileman: [00:38:02] It’s the layer navigation. Yes.

Ivan Chepurnyi: [00:38:13] Yeah. This is just doesn’t look like the right database query, um, built. I was there in this Amasty extension because you know, if, if you would use  elastic suite, you could have got the data from elastic in 30 milliseconds. With all the needed that faceted filters for you. And here, it takes quite a lot of time.

Let’s refresh it again

Here that’s a lot of database operations that are completely unnecessary, in my opinion, and could have been built in much more simpler way. so yeah, first of all, a lot of load in the loops and. Let me guess. It also shows all the products in the category, right? Because if I go here and what if I just opened the page source and I just take a look at.

UTM. Okay. Impression data. So here it is.

Eric Hileman: [00:39:50] What now… What’s the marketing purpose of this?

You’re telling…

Ivan Chepurnyi: [00:39:56] This is bascially to use this data? In, uh, Google, uh, tag manager, but this, Ano, Anowave extension is a huge abuser. And do you see, it also adds a lot of, a lot of, a lot of redundant JavaScript on the page as well. So it’s basically, not the only bad for servers, it’s also bad for front end performance as well. Yeah. I just, well, let’s open it up. Oh yeah, I even can’t select it. Let’s see.

Eric Hileman: [00:40:38] So they’re send…

Ivan Chepurnyi: [00:40:41] So…

Eric Hileman: [00:40:42] They’re sending this back to Google tag manager to say that somebody saw these products. I, I don’t know much about Google tag manager.

Ivan Chepurnyi: [00:40:55] It’s basically an impression of. So they could, but it looks like it’s a little bit broken JSON.

I don’t know why it looks like that, but it looks like that for me is, Oh, it just analyzed in one line so that’s why I’m a little bit confused. Okay, let’s do JSON viewer.

Let’s paste this JSON here. And let’s see, 47 so 47 items are shown. yeah, so basically very simple product information that is completely no need to, you know, even use. You know, all those load in the loops because it’s using price and they would just use the price from price index there is no need to, you know, in order to access this information, like least category ID name was a product brand that was a product price and position it, it basically grabs all the product collection and loads it on the category page.

When there are no products actually shown.

Eric Hileman: [00:42:17] Yeah, well, there’s what I mean, we’re looking at like 10 and it’s got 40 some listed there. What? Oh wait, where’s the other 30 I don’t understand.

Ivan Chepurnyi: [00:42:33] There are no products at all here.

Eric Hileman: [00:42:34] At the bottom? At the bottom? No. Even at the bottom. That still doesn’t add up to…

Ivan Chepurnyi: [00:42:39] As I say, it just shows the products that they’re directly assigned to this modalities category.

Even if the customer actually doesn’t, doesn’t show these products..

Eric Hileman: [00:42:50] Wow. Okay.

Ivan Chepurnyi: [00:42:51] But this Google tag manager extension loads them anyway.

Eric Hileman: [00:42:56] Oh shit, that’s no good.

Ivan Chepurnyi: [00:42:59] Yeah. So, uh, in general, I also work with another customer right now on removing, uh, one of the Google tag manager extension, not on the way but another one. There are plenty of bad Google tag manager extensions from the market.

Eric Hileman: [00:43:13] Which one would you recommend?

Ivan Chepurnyi: [00:43:14] None of them. Because Google tag manager is actually a very straight forward implementation. Or you just can write a couple of lines of JavaScript in order to push the data into Google and tell about the user actions. You don’t even need to output any data from the backend.

You can just use Java script and build CSS selector to find, Hey, which products I see on the page. You can build this data by just accessing your, uh, products by, uh, washing what. Uh. Customer would actually even show, sees in their view. So, uh, as I understand, like from point of view, of impressions, it makes sense to actually see which product your customer actually, so, or for instance, hovered over.

But these extensions. They just said, Hey, we’re just gonna nuke Google tag manager with also the data that we have. Yeah.

Eric Hileman: [00:44:19] Yeah, it looks like they just took everything from the category, even though it’s not even displayed on this page, which is false data that this merchant would be working off of.

Ivan Chepurnyi: [00:44:28] Yeah. So basically, what do you usually should do?

You know, like if customer hovers over, uh, let’s say, Oh, mouse over, we’ll say on this item and you know, Hey, looks like this item is using, uh, this specific CSS class. So probably I can tell, Hey, does this hold and cold therapy category name? I’m, I’m, gonna send this as impression data for Google tag manager.

You know. And so on. Or for instance, when customer scroll. So you can use also intersection observer in the browser in order to see which elements are currently visable in the view of the customer as well. And you can base your Google tag manager data based on this, and it’s just, you know, lazy implementation  of Google tag manager by those extension vendors.

That, um. Just implemented because you know, merchants are looking for proper integrations, but no one tells them, Hey, uh, in order to implement the good, good, good Google tag manager data, you actually need to work with your developer in order to properly integrate it into your theme because extension won’t solve it.

You have always, um, to adopt it to your design, adopt it to your user behavior. You, you shouldn’t just rely on different integration. Like for instance, you can have some default integration. We installed, let’s say for shopping card for. Checkout, you know, to report these kinds of events, but actual view on the category  page or a product page, it’s better to adjust it to your team.

So don’t just install Google tag manager extensions that just does everything for you.

Eric Hileman: [00:46:15] I’d be willing to bet a lot of people don’t even look at that. Google tag manager data.

Ivan Chepurnyi: [00:46:22] A lot of merchants actually look into eCommerce data related to sales. Yeah. But from point of view of viewing the product to details like, Hey, which field our customer has clicked on, or which products customer visited prior to going to shopping cart and actually funneling through all the checkout.

Uh, very rarely customers actually configure Google tag manager in such a way. Okay. So yeah. This one is definitely a problem. So we right away can say is that the load time will be reduced a lot by just removing this Anowave extension. So this, this, this is a problematic one. And yeah.

There are plenty of issues.  Another thing is, uh. Which one is it? Was it, category page? Hmm. This is a widget.

this is it. Yeah. Show by department and also another thing we saw here is related to Amasty.

Where is it?

The filters? Yeah, Shop By. So it’s quite a lot of time, I can tell you for sure spent there. So basically in my experience in Magento one in Magento, two no good extensions for layered navigation are available in the market. You know, it was all trying to achieve everything, um, you know, and not the most efficient way.

So, and here, wait a second, what’s happening here as well? This is interesting as well, like 1.3 seconds and some kind of category or what’s loaded here? Attribute options. For configurable product. What, what it is, get identities.


What is this? Who calls every single product by ID? Magento Core?

Do you see it?

Eric Hileman: [00:49:18] Yeah.

After plugin. It looks like it’s truncated in the name.

Ivan Chepurnyi: [00:49:35] Is it really a bug in Magento core? I have never seen this one before.

So getting identities, so this is. Edit by full page cache module. You see? A full page cache module. I can assent what it does. So it renders the product IDs as, um, identities for, uh. Basing it, uh, into the response headers of the page. So then, you can flush cache, by product IDs instead of doing it by URLs.

And here we see that this catalog product model intercepted and get the identities, get some kind of intercepted over here from configurable product module after get identities. Let’s see. Um, if we can find in the code itself, uh. That is, uh, where unknowns there. Let’s see. Um, look it up right away in Magento code base afterGetIdentities, right?

No, not the right..

Eric Hileman: [00:50:54] Oh wait you spelled it wrong. Go back.

You put an F instead of the T. Identities. Yeah.

Ivan Chepurnyi: [00:51:05] Yeah.

I want the just to have identifies. Yeah. So configurable products extender there. So this is the one. Okay. What did..

Here it is.

Eric Hileman: [00:51:24] Hey, it looks like a loop.

Ivan Chepurnyi: [00:51:29] And this is in. This was added in November 23rd, 2018 so it was, this is quite recent contribution and the commit message is not telling us anything about it.

Eric Hileman: [00:51:54] Uh…

Ivan Chepurnyi: [00:51:55] It just adds.

Eric Hileman: [00:51:58] He added unit tests to it.

Ivan Chepurnyi: [00:52:02] Well, what is the purpose of this functionality? I just, I even don’t see any value in the test as well because, Oh, it looks a little bit strange.

Yeah, so this is definitely a part of the configurable. After getIdentities and here going to be switched to the default. Devo.. Develop. Yeah. It still is the same one. So still the same code base. and the problem is here is that it tries to get all of these simple product IDs of configurable being added to the  cache stacks, but inadvertently, uh, the problem here is that it uses this get configurable type, and I can bet 100% that this gets children IDs, loads product through product repository and loads the single configurable product. Again.

Let’s see. So this is a configurable type and configurable type with a configurable product model product type configurable. So good. We’re in the same module. So we can just go in a product.  Configurable and let’s search for this get children IDs. Okay.

Very nice. We are going down the rabbit hole. Let’s see. Uh, which one is this? I guess it’s resource model product type configurable.

Um, okay. then. Looks strange. So many contributors here.

Eric Hileman: [00:54:09] Yeah, 62.

Ivan Chepurnyi: [00:54:12] Yeah. So let’s go to a resource model product type configurable get children IDs. Okay. Here I see a simple database query. So looks like this problem is not in the most recent version. Maybe it’s a problem with visible only in this one. Do you know which Magento version are they running?

Eric Hileman: [00:54:52] Yeah, I can check.

2.2.6  B2B

Ivan Chepurnyi: [00:55:33] 2.2.6 right? Yeah. Okay, so probably that’s the reason, um, because. Uh, of that issue in core, and probably it was already fixed and they just don’t have the most recent version. So probably if we go into, again, uh, what was it? It was product identities extender. After get identities? Yes, so product identities extender.

Oh wait. Sorry. It’s still there.

I was just opening the wrong. Okay. I just was, taking a look at the broad… At the wrong product. Um, yeah. So after get identities problem is still here. A product repository you get by ID. But strange. I didn’t see it on my project probably because, uh, probably, because I already well already someone from the developers probably already wasn’t a moving it or maybe becasue I’m using it a little bit different version of page cache configuration. So this one a little bit interesting? So definitely is there a problem and I don’t understand why, why it’s doing. So basically what it takes, it takes, variant ids of Oh,

I see. Why I don’t see it. Uh, now I understand. So the problem is, uh, in inverse relation of the products. So if you have, uh, if you are showing simple products independently from configurable products on your category pages, but these simple products are also assigned to configurable products.

It also tries to a, add all the ids of every single configurable product. Simple product is assigned to together with all the simple products of that configurable product. And that’s why this product repository get by ID gets triggered. So this is a huge find in Magento performance in Magento core.

This is definitely a huge performance issue because, yeah, you just saw it, you know, like it’s 1.5 seconds. Just two or three IDs. Basically. It’s the same thing. I mean, it needs to have worse customer service model just to retrieve those IDs, I even don’t know if actually is those IDs are needed in there.

Maybe they are just edited in case because you know, like what can affect, uh, visibility of a simple product. Because if you show a simple product, right, it doesn’t matter. Is it children configurable or not? Is there a status of visibility of this product doesn’t affect, uh, you know, its view on the current category.

So, you know, like changes in configurable product or in simple products of that configurable product also shouldn’t affect, um, it shouldn’t affect the visibility of actual product you are currently viewing.

Eric Hileman: [00:59:53] Is there? No…

Ivan Chepurnyi: [00:59:55] If you disable configurable product, but your simple product is still showing category page separately.

I don’t see why, why it’s actually added here. I think this plugin can be safely disabled. I don’t see any reason in it.

Eric Hileman: [01:00:12] The a plugin itself? I mean, do you really need to load the product repository just to get the parent. There’s no…

Ivan Chepurnyi: [01:00:19] No, I mean like the idea of this plugin itself is wrong.

So that’s what I’m trying to explain. So what it does, so let’s say you have a simple product on the page.

Eric Hileman: [01:00:30] Yeah.

Ivan Chepurnyi: [01:00:32] And then, this a simple product? Uh, what this plugin does, it takes all the parent products of this simple product. So it was this simple product is assigned to a configurable. Uh, then Magento takes all of the IDs of configurable products in order to add them to your current page cache tags. But configurable product doesn’t affect with the view of simple product in any way. So simple product is completely independent. It, it shouldn’t be affected by a configurable product.

So this is completely

Eric Hileman: [01:01:11] an inverse relationship.

Ivan Chepurnyi: [01:01:13] Inverse relationshio that is completely redundant. So, uh, I really don’t understand, uh, this thing here because…

Eric Hileman: [01:01:25] I don’t know, what do we do next? What do we report?

Ivan Chepurnyi: [01:01:30] I think the best pull request here is actually to remove this plug in all together from core.

Eric Hileman: [01:01:40] Try it.

Ivan Chepurnyi: [01:01:45] But, uh, first of all, yeah, let’s put it into the notes for these guys that they actually need to just disable this plug in right away, but then in general, yeah, I think this plug in just makes no sense.

But wait a second. Wait a second. I think we have some history of changes.


Not much history.  Hide from product page option does not work for child products, but how, how does removing this one help to solve this issue? And it doesn’t make any sense. And what is the history of this file when it was introduced? Was it in 2.2 okay. Hide from product page… Again. Absolutely. Absolutely no information. About his ticket, like, looks like completely unrelated to the title of the commit and it was added on February 13, uh, was it before.

Oh, it was still there before in such style.

Hide from product page option does not work for child of configurable product. I never heard about this future hide from product page. Okay, so it was introduced in 2.1. This plugin. And all of the way to 2.3 it’s still there. And if by any chance you are showing simple products from configurable products, you are affected by this performance issue. And we’re over, if you are showing multiple simple products of a configurable product. It means for every single simple product, you will separately load configurable product of that simple product with all the configurable product. So…

Eric Hileman: [01:04:56] That’s why we called this show Load in the Loop.

Ivan Chepurnyi: [01:05:01] Yeah, definitely. Yeah. Okay. We even,didn’t hit page with categories and I think we’re already over overtime was our show. So yeah, it was an interesting experience. I can tell you for sure.

Like we’re already, uh, interesting issues we’ve found. Um, Our favorite extensions right.

Eric Hileman: [01:05:27] Yeah. Amasty, once again, Anowave was in there.

Ivan Chepurnyi: [01:05:33] Recurring theme in our show, like in second episode in a row, we have some issues with Amasty extension and then  we have a new one that I, I’ve met her before. Uh, it’s Anowave, so Anowave is, um, low quality Google tag manager plugin. Um, I advise, just remove it, then just implement it. Very simple. Uh, I remember Jisse, uh, from, uh, Netherlands, uh, have created some kind of open source plugin for Google tag manager, but I didn’t review it so I cannot..

Eric Hileman: [01:06:21] Oh, Jisse? They have like a coalition of extension vendors to write good extensions, what’s it called?

Ivan Chepurnyi: [01:06:28] ExtDN.

Eric Hileman: [01:06:29] They’re going to kill me for not remembering that. E. X. T. yeah. E. X. T. D. N. right. That’s probably good. It’s probably good. They…. Those guys. Write good extensions.

Ivan Chepurnyi: [01:06:40] Yo, let’s take a look. all right. And GT down

Eric Hileman: [01:06:45] Yireo. Yeah. Y I R E O



Ivan Chepurnyi: [01:06:55] Yeah. Easier.

Eric Hileman: [01:07:01] Yeah.

Ivan Chepurnyi: [01:07:02] So is this, uh, okay, so this is the, it’s a layered test, but here’s his Yireo.

Um. Oh, it’s through Ygrek. How they call it in the Netherlands? Uh, Y, uh, here we have…

Eric Hileman: [01:07:24] Yeah, I can’t pronounce any of them..

Ivan Chepurnyi: [01:07:25] Extensions. Yeah. So let’s see. Yes, tag manager. Yeah. This is for Magento one?

Eric Hileman: [01:07:37] Oh, wait, Google tag manager 2. Is that still?

Ivan Chepurnyi: [01:07:41] Oh yeah. Here it is. Let’s see, where is it? Can I find you’d have here it this. Okay, so here is the one. Okay, good, good, good, good. And maybe this one is worth to give it a shot. And if, the customer actually missing something from, from the output, they can always edit.

Ah, okay. What it does? So they take scripts.

And Oh, so yeah, he just adds, at the start of the body he adds some kind of script, yeah. Over there, actually, he adds some kind of a script, uh, with some impressions for such a way. So completely and in ways in point of view or templates. Okay. And let’s see his script. Where is it?

Was it? Let’s just quickly, sorry, Jisse for reviewing your extension code on show. But, uh, we need to know, can we suggest it to the customer? So it’s view model script. So let’s go into the view model and let’s see what’s happening over there and this view model script.

Add product, register current product.

Then current category. So just output some template or some custom stuff. Um Hmm. So far, I don’t see any load in the loop. So let’s take a look at the template.

Oh yeah.

Well, looks it looks reasonable, you know. nothing outrageous.

So yeah.

Eric Hileman: [01:10:57] Cool.

Ivan Chepurnyi: [01:10:59] Should be okay.

Eric Hileman: [01:11:00] Swap out the Anowave.

Ivan Chepurnyi: [01:11:03] Yeah. Just, just, uh, use it instead of Anowave extension. It probably has much more less features, you know, than Anowave, but, uh, at least it’s not gonna..

Eric Hileman: [01:11:21] Give you wrong data? Slow you down?

Ivan Chepurnyi: [01:11:26] Yeah, probably it just has some basic tag managing, you know, like it just shows. Hey, you are visiting this category page, but like impression, sending  impressions to Google tag manager if actually customer don’t see products. It doesn’t make sense.

Eric Hileman: [01:11:42] Yeah. Bad data is worse than no data. Is that the saying?

Ivan Chepurnyi: [01:11:50] So in general, I think there was some quick wins, um, uh, this customer can achieve by removing Anowave extension and, uh,

Eric Hileman: [01:12:03] the brand slider. Yeah. The, the, the Amasty, uh, brand slider was 60% of the home page.

Ivan Chepurnyi: [01:12:11] Yeah. So brand slide slider. Um. They can probably keep this extension itself with Amasty but just grabs the data directly from the database for a service model and render a very lightweight walk. It doesn’t make sense to, uh, build, um, on top of existing… to search API in order to just show brands from their homepage.

It doesn’t make sense.

Eric Hileman: [01:12:38] Uh, speaking of the search and going into layered navigation, they are using elastic search here, but it’s Magento’s… I think they are. Jen says, Oh, she says they have ES, MyuSQL, and external all set up and they’re..

Wait a minute…

Ivan Chepurnyi: [01:13:09] They definitely not using Elastic Suite. I didn’t see it.

Eric Hileman: [01:13:13] Oh, not elastic suite. Elastic search.

Ivan Chepurnyi: [01:13:18] Oh yeah. So they use standard core elastic search? Yeah, it’s, it’s, it’s not performance.

Eric Hileman: [01:13:25] So, okay. So you would recommend using Smile’s ElastiSuite?

Ivan Chepurnyi: [01:13:32] Yeah, it’s a, it’s reasonable, like it’s, it’s not the top, top, top performance, but it’s very, a simple solution for fixing majority of the issues.

Unfortunately, that also comes with a lot of customized front-end. So they replace a lot of core parts of Magento because unfortunately, uh, yeah. You, you have to build really, uh, interesting extension from point of view of a structure to preserve the standard Magento core functionality, but at the same time provide high performance.

So there is no extension on the markets that does it.

Eric Hileman: [01:14:11] But Elastic Suite’s better by like two times, three times faster, better than the built in Magento uses wouldyou say?

Ivan Chepurnyi: [01:14:19] Yeah. Uh, let’s say like, is that all the layered navigation filters? They’re completely outside of MySQL. So if they’re  completely outside of MySQL,  it’s already huge achievement because you just, saw what kind of terrible queries are happening. Were there like 30 milliseconds on every single database query to get filters? Yeah, it’s just…

Eric Hileman: [01:14:45] Were you? Are you a fan of Sphinx? Was that you who was really more into Sphinx than elastic search?

Ivan Chepurnyi: [01:14:53] Yeah. Yeah. And it’s, it’s something that I like much more, like Sphinx is actually right now, is not open source anymore. Uh, but there is an open source fork of Sphinx uh, based on the most recent open source version of Sphinx. And it lives very nice life. It’s called Manticore, it’s called Manticore search.

So it’s kind of very nice. Uh. Uh, you know, uh, ancient, um, mythology, uh, related brand names. So Sphinx search, Manticore search, a lot of interesting stuff, and, um, Manticore is actually very nice and very lightweight. It can run a, alongside your standard PHP application, uh, as a simple binary, it doesn’t require you to install Java to run the huge VM in order to run it.

It’s very simple. Uh, C program, uh, that is very efficient as well. Uh, like where are you, very, very memory efficient so it doesn’t eat  all of your RAM and it’s blazingly fast. Also, it speaks MySQL protocol. So you can build the queries like you would work with my scale bot, uh, to search engine. Also, if you were a very efficient exporter for Sphinx, uh, importing data into Sphinx itself. Uh, is around, uh, 40 K records per minute… Per second,sorry.

Eric Hileman: [01:16:31] Per second?

Ivan Chepurnyi: [01:16:32] Per second, yeah. So if you stream data pipe of CSV data directly into the index tool of Manticore, it takes 40K records per second in order to build their file system based index. So actually you have an index tools that can be run on completely separate server, and then you can, uh, archive those, uh, index files. Deploy them into search server and send the signal to search server. Hey, please reload your in memory index from files.

And then, uh, you have a zero downtimereload of the search data, uh, with very efficient Sphinx server. So we take probably double the RAM, and during this process, but usually Sphinx takes very small amount of RAM.

Eric Hileman: [01:17:25] Yeah. You had me at not having to run Java. I hate running Java.

Ivan Chepurnyi: [01:17:32] And also in, uh, ElasticSearch is based on Lucene and Lucene itself is very legacy, uh, search, uh, API. So, so Solar is based on Lucene. Uh, elastic search is based on Lucene. Sphinx is using completely different, uh, approach. So they have their own index structure, uh, completely in memory. Um, and you can build, actually a scale query, uh, in Sphinx terms, uh, that will return you everything that will return you products will return you layered navigation. And. Will serve you coffee.

And a, I actually was building some proof of concept for, um, as in, as a catalog API based on Manticore search.

And this, um, proof of concept was able to serve, uh 1700 concurrent users per second.

Eric Hileman: [01:18:43] Oh, right. Yeah. I remember talking about that…

Ivan Chepurnyi: [01:18:46] Uh, yeah. This one is actually on… . half million SKUs. So you have index of half a million SKUs in Sphinx. We surround 50 a filterable attributes in layered navigation, and it was directly just concurrently, with hundred concurrent users. It was able to process, uh, 1700 concurrent requests per second.

And, um, it’s just Manticore with a synchronous PHP application. Is that just. runs on a single node.

Eric Hileman: [01:19:32] Yeah. We, we support Sphinx, but we don’t, I think Mirasvit is the only extension that we see really using it. And we’re not real fans of that one. It was it doing, it does. Um, I dunno, there’s part of that in there that we always have problems with.

It’s, um. I can’tthink of it off hand. Are you familiar with that? Mirasvit Sphinx extension?

Ivan Chepurnyi: [01:19:57] Yeah, I don’t use it. Yeah. So basically Sphinx requires to build a special configuration yourself. Like you can actually build it for a custom project very efficiently, but you know. Packaging  this kind of extension as a standalone solution that someone can install from marketplace and just run it in a store is not something that I would suggest to do.

Because, you know, like those kind of extensions, they’re more oriented to the merchants who just want to configure and forget. Right? But, um, this kind of technical solutions they require. Uh, attention of the expert in that area, not just, you know, can the merchant install it.. Then the merchant will mess up the configuration

Eric Hileman: [01:20:48] Mirtasvit, it falls back, I think, to MySQL for full text, and then it just does this expansion thing and it just kills, kills mySQL when it falls back. Yeah.

Ivan Chepurnyi: [01:21:01] Yeah. Um, basically as well, you know, like. You need to do,

Eric Hileman: [01:21:05] Misspell, that’s it.

Ivan Chepurnyi: [01:21:08] Your Sphinx configurations to customer specifics.

You know, like you need to, uh, also properly build an index because in Sphinx, there is two kinds of indexes, uh, actually there is one  kind of index, full text index. And other things are just memory attributes.

So if you, let’s say you build, uh, let’s say your category pages and. Uh, you filter by category ID as an attribute, and you have half a million products.

Uh, it’s going to be very slow search in Sphinx because Sphinx will have to work through every single product to take its category ID attribute, uh, compare it to the value you’re looking for, and then go next, go next to the next. It’s still going to be fasterthan  doing it in MySQL because Sphinx gives us data, uh, in memory, but still it’s going to be painfully slow.

Um, so what do you usually do when you build this Sphinx index? You actually create a fallback search field. With category slug. Oh, and then with this category slug who just applied as a search term for a full text search index and because category have kind of good cardinality, uh, value, uh, it, uh, is very fast, full text search index.

Eric Hileman: [01:22:35] Got it.

Ivan Chepurnyi: [01:22:35] Say, Hey, how will it save 5,000 products with this search term assigned to it? So you can build very effective, uh, category pages, uh, with Sphinx.

Eric Hileman: [01:22:47] It was, it was Mirasvitz Misspell that I was thinking of that hammers the database extremely hard. We usually recommend disabling it, mainly because searches, searches are really big attack point for DDoS. So if you have competitors who are coming at you, or even just bots that enjoy eating search queries, it can easily, easily consume a lot of server resources just hitting your search. Uh, we’re often times…

Ivan Chepurnyi: [01:23:18] So, so, so many Sphinx solutions that completely bypass Magentfull text  and quick search.

Yeah. Uh, so, uh, what I was building it, it was for my Magento one Sphinx implementation. So, uh, I have this kind of end point where I have to build trigrams based on product data. So in all the exhaust suggestions, Oh, okay. Can start, start typing. Uh,

Eric Hileman: [01:23:51] yeah. You, autocomplete..

Ivan Chepurnyi: [01:23:52] You, you just complete. And need, need to get a, you know, off the complete, uh, of your search query.

So what I built, uh, for, uh, that solution, uh, I actually had to, uh, do some, uh, exports of the data from the products, and then I’ve built a special autocomplete suggestion builder. So I’d build let’s say, brand, product name, color, uh, and let’s say season. And then I mix these words differently and then I build from these words I also do trigrams. Trigrams. It’s kind of free characters’ set in every single word. And then, uh, when someone types a query, I also said, but I did by trigrams and then I sorted by trigrams, not by the full keyword,then the most matches of the trigrams form these suggestions will pop up as a source suggestion for a customer.

Eric Hileman: [01:24:57] Yeah, that’s sufficient. You can easily hammer a server just from auto complete. If you get people in there a lot. I’ve seen a lot of bad auto-complete implementations that just kill…

Ivan Chepurnyi: [01:25:09] It requires a very good, uh, implementation behind it.

If, you know, if you just do a database, query, uh, by like. You know, it’s not something that you want to have on your production server.

Do you want to have something uh, implemented properly? You know, it was trigrams. Uh, Elasticsearch has trigram support as far as I remember by default.

But you still need to create some kind of separate index if you want to have high quality up to a suggestion, you know? Yeah. And just in Sphinx, it’s a little bit different. In Sphinx, you have to do a lot of things yourself. So it says, that’s why, uh, when I say is that if someone installs a Sphinx extension from marketplaces, they will never receive, uh, you know, high-performance from Spinx with the most best solution.

You still need to spend some development time on actually adjusting it to your needs because there is no one size fits all, unfortunately.

Eric Hileman: [01:26:08] Yeah. Okay. Well we should probably wrap it up.

Ivan Chepurnyi: [01:26:13] Yeah, it was nice episode today, I think.

Eric Hileman: [01:26:16] Yeah, it was. It was good. It was good. So what are you going to do about that Magento two core?

Uh, you’re going to do a pull request to suggest to remove that plugin?

Ivan Chepurnyi: [01:26:26] I don’t know.

Eric Hileman: [01:26:28] Maybe talk to some of the developers?

Ivan Chepurnyi: [01:26:30] It makes sense just to open an issue, you know? And get some discussions going on, uh, to see what actually was a reason to add this plugin in the first place. Uh, but yes, implementationfor the plugin itself, uh, is questionable in my opinion.

Like, I, I don’t see you reason to have on there. So for this customer, definitely. As a quick fix, developers should just disable it.

Eric Hileman: [01:27:06] Yup. Well, so lots of good suggestions for this customer. Any parting words before we take off how to reach you? We always put that in the show notes, so we’ll link here.


Ivan Chepurnyi: [01:27:17] I think it doesn’t make sense to pronounce it during this and then during this recording, but, uh. You can find our contact details if you want to get your store reviewed. Um, me and Eric’s details are available always in show notes and as well a under YouTube video. Right? Yup, it is. It should be available under YouTube video, so you can contact us and you can always follow us on Twitter.

Also, I think we can open DMs on a Load in the  Loop Twitter account, and then me or Eric can answer your inquiries aabout reviewing your store. And as you said Eric, uh, you have, uh, a lot of customers still, at MageMojo just to continue reviews.

Eric Hileman: [01:28:09] We have no shortage of content.

Ivan Chepurnyi: [01:28:14] Definitely.

So, um, yes, thanks to this customer we found very interesting issue. Uh, I had never seen this core issue before. But now, yeah, I see it. Simple products assigned to configurables at the show category page triggers a very terrible performance degradation issue in Magento.

Eric Hileman: [01:28:40] Cool. I’ll start that discussion on Twitter.

When we, uh, when we tweet the show out, we’ll @ the extension companies and give Jisse some props and, and ask if anybody else understands and knows the reason for that and start a conversation. So good stuff. I think we’ll, we’ll wrap there and thanks to Blackfire for the awesome sponsorship and the cool, awesome tool for us to work with.

Yeah, without`l Blackfire probably it would be much more harder because. You know, uh, it’s very easy to use. Uh, you just have your browser extension, you click profile and it’s done, and you, you can open timeline and you can just watch for all of the issues. Also, uh, the guys at Blackfire, they were telling me like, Hey, you know, like we have this nice feature in Blackfire that can tell you if you have your caches turned off?

So if we open from the last show, if we open one of the profiles,let’s just open this load in a loop, uh, environment. So if we go into, uh, Adam, uh, Adam’s a horse supply, uh, let’s open this 50 seconds, and if we open here.

Uh, found reccomendations tab uh, some composer. Uh, okay, this is different thing, but.

Yeah, there it is.

Ivan Chepurnyi: [01:30:27] Magento two locational, cache should be enabled and full page cache should be enabled. JavaScript bundling should be enabled. Um, and a bunch of other  suggestions just…

Eric Hileman: [01:30:42] Yeah, so they did, they did find the caches.

And then you said about adding the mode two. That was a, a feature suggestion they got from one of our episodes about adding a recommendation for, um, to go into production mode.

Ivan Chepurnyi: [01:30:57] Yeah. It also very good thing. So they are going to add this suggestion as well, uh to blackfire, so it’s win win for, for all of us. We review Magento stores and, uh, they improve their product for Magento, developers.

Eric Hileman: [01:31:14] Cool. Well, thanks for everybody who’s still watching at this point. It’s really long,, so thank you. If you’re still here, I guess that’s a wrap for us today. Thanks, Ivan.

Ivan Chepurnyi: [01:31:26] Thanks everyone! Until the next time.

Notify of
Inline Feedbacks
View all comments

Ready to get Started?

Sign up now
Would love your thoughts, please comment.x